Using Large Language Models to Generate JUnit Tests: An Empirical Study
Authors:
Mohammed Latif Siddiq,
Joanna C. S. Santos,
Ridwanul Hasan Tanvir,
Noshin Ulfat,
Fahmid Al Rifat,
Vinicius Carvalho Lopes
Abstract:
A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well…
▽ More
A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests.
△ Less
Submitted 8 March, 2024; v1 submitted 30 April, 2023;
originally announced May 2023.
Expanding Frontiers: Settling an Understanding of Systems-of-Information Systems
Authors:
Valdemar Vicente Graciano Neto,
Bruno Gabriel Araújo Lebtag,
Paulo Gabriel Teixeira,
Priscilla Batista,
Vinícius Carvalho Lopes,
Jamal El-Hachem,
Jérémy Buisson,
Flavio Oquendo,
Juliana Fernandes,
Francisco Ferreira,
Rodrigo Peireira dos Santos,
Davi Viana,
Everton Cavalcante,
Mohamad Kassab,
Ahmad Mohsin,
Roberto Oliveira,
Vânia Neves,
Maria Istela Cagnin,
Elisa Yumi Nakagawa
Abstract:
System-of-Systems (SoS) has consolidated itself as a special type of software-intensive systems. As such, subtypes of SoS have also emerged, such as Cyber-Physical SoS (CPSoS) that are formed essentially of cyber-physical constituent systems and Systems-of-Information Systems (SoIS) that contain information systems as their constituents. In contrast to CPSoS that have been investigated and covered…
▽ More
System-of-Systems (SoS) has consolidated itself as a special type of software-intensive systems. As such, subtypes of SoS have also emerged, such as Cyber-Physical SoS (CPSoS) that are formed essentially of cyber-physical constituent systems and Systems-of-Information Systems (SoIS) that contain information systems as their constituents. In contrast to CPSoS that have been investigated and covered in the specialized literature, SoIS still lack critical discussion about their fundamentals. The main contribution of this paper is to present those fundamentals to set an understanding of SoIS. By offering a discussion and examining literature cases, we draw an essential settlement on SoIS definition, basics, and practical implications. The discussion herein presented results from research conducted on SoIS over the past years in interinstitutional and multinational research collaborations. The knowledge gathered in this paper arises from several scientific discussion meetings among the authors. As a result, we aim to contribute to the state of the art of SoIS besides paving the research avenues for the forthcoming years.
△ Less
Submitted 25 March, 2021;
originally announced March 2021.
