Automated Unit Test Case generation has been the focus of extensive literature within the research community . Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult  to read or understand for developers . In this paper we propose AthenaTest, an approach that aims at generating unit test cases by learning from real-world , developer-written test cases. Our approach relies on a state-of-the-art sequence-to-sequence transformer model which is able to write useful test cases for a given method under test (i. e., focal method). We also introduce methods2test - the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 630k test cases mined from 70k open-source repositories hosted on GitHub. We use this dataset to train a transformer model to translate focal methods into the corresponding test cases. We evaluate the ability of our model in generating test cases using natural language processing as well as code-specific criteria. First, we assess the quality of the translation compared to the target test case, then we analyze properties of the test case such as syntactic correctness and number and variety of testing APIs (e. g., asserts). We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3 . Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated test cases .
Automated unit test case generation tools facilitate test-driven development and support developers by suggesting tests intended to identify flaws in their code . Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult for developers to read or understand . In this paper we propose AthenaTest, an approach that aims to generate unit test cases by learning from real-world focal methods and developer-written testcases. We formulate unit test case generation as a sequence-to-sequence learning task, adopting a two-step training procedure consisting of denoising pretraining on a large unsupervised Java corpus, and supervised finetuning for a downstream translation task of generating unit tests. We investigate the impact of natural language and source code pretraining, as well as the focal context information surrounding the focal method. Both techniques provide improvements in terms of validation loss, with pretraining yielding 25\% relative improvement and focal context providing additional 11.1\% improvement. We also introduce Methods2Test, the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 780K test cases mined from 91K open-source repositories from GitHub. We evaluate AthenaTest on five defects4j projects, generating 25K passing test cases covering 43.7\% of the focal methods with only 30 attempts. We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3 , finding that our approach outperforms GPT-3 and has comparable coverage w.r.t. EvoSuite. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated tests, showing overwhelmingly preference towards AthenaTest .
R
Unit Test Case generation has been the focus of extensive literature within the research community
unit test case generation tools facilitate test-driven development and support developers by suggesting tests intended to identify flaws in their code
A
for developers
D
for developers
R
at generating
to generate
R
,
focal methods and
R
test cases. Our approach relies on a state-of-the-art
testcases. We formulate unit test case generation as a
R
transformer model which is able to write useful test cases for a given method under test (i. e., focal method).
learning task, adopting a two-step training procedure consisting of denoising pretraining on a large unsupervised Java corpus, and supervised finetuning for a downstream translation task of generating unit tests. We investigate the impact of natural language and source code pretraining, as well as the focal context information surrounding the focal method. Both techniques provide improvements in terms of validation loss, with pretraining yielding 25\% relative improvement and focal context providing additional 11.1\% improvement.
R
methods2test -
Methods2Test,
R
630k
780K
R
70k
91K
R
hosted on
from
R
use this dataset to train a transformer model to translate focal methods into the corresponding test cases. We evaluate the ability of our model in generating test cases using natural language processing as well as code-specific criteria. First, we assess the quality of the translation compared to the target test case, then we analyze properties of the test case such as syntactic correctness and number and variety of testing APIs (e. g., asserts).
evaluate AthenaTest on five defects4j projects, generating 25K passing test cases covering 43.7\% of the focal methods with only 30 attempts.
R
.
, finding that our approach outperforms GPT-3 and has comparable coverage w.r.t. EvoSuite.
R
test cases
tests, showing overwhelmingly preference towards AthenaTest