C on T est: A Unit Test Completion Benchmark featuring Context (2021)

Abstract: We introduce CONTEST, a benchmark for NLP-based unit test completion, the task of predicting a test’s assert statements given its setup and focal method, i.e. the method to be tested. ConTest is large-scale (with 365k datapoints). Besides the test code and tested code, it also features context code called by either. We found context to be crucial for accurately predicting assertions. We also introduce baselines based on transformer encoder-decoders, and study the effects of including syntactic information and context. Overall, our models achieve a BLEU score of 38.2, while only generating unparsable code in 1.92% of cases.

## introduction testing is commonly considered an important part of software development, but it tends to be neglected in practice, as developers find it rather timeconsuming and tedious. this has motivated automated testing: approaches such as evosuite @xcite and randoop @xcite can bootstrap tests with decent code coverage using static code analysis and evolutionary search. however, recent studies @xcite have found the readability of those generated tests to be subpar, and have -more importantly -found that the above approaches struggle with producing "meaningful" checks that truly assert the code to behave as expected. therefore, recent approaches have tackled aibased test completion as a research challenge for nlp-based programming, using encoder-decoder models with bidirectional recurrent networks @xcite or pre-trained transformers @xcite . these models take the test's setup code, together with the targeted method call (focal method) as input and predict the test's assertion. since this assertion is arguably the test's part which requires the most understanding of the target method, it is also the part where heuristic approaches struggle the most. here, an ai-based approach can fill this gap and provide the most valuable addition to existing solutions. we extend this line of research by introducing contest, a new contextual benchmark for automated test completion. contest is based on github data. each of its datapoints features a test method, linked with the tested focal method using a fuzzy name matching. additionally, the test is split into segments using heuristics and static code analysis: we provide the focal method, test setup code executed before the assertions, context methods called within the setup code, and context methods called within the focal method. a manual inspection of samples estimates its ground truth's accuracy of matching the correct focal method at 94%, but we even found the falsely matched methods to be relevant to the test. summarizing, contest offers the following benefits foot_0 : our work targets the fields of automated software testing and natural language processing. automated software testing can be divided into test generation approaches, aiming to generate the complete test @xcite and test completion approaches, aiming to generate meaningful assert statements @xcite . test completion has been tackled by rule based approaches such as agitar @xcite , randoop @xcite , and evosuite @xcite , which can bootstrap tests with decent code coverage using static code analysis and evolutionary search. however, recent studies @xcite have found that these approaches struggle with producing "meaningful" checks that truly assert the code to behave as expected. furthermore these approaches require handcrafted rules in order to generate assert statements. recently, the focus has shifted to approach automatic software testing using natural language processing (nlp) methods. @xcite employ a rnn-based neural machine translation system to generate complete tests, while more recent work @xcite adapts pre-trained transformer models such as bart @xcite . this and most recent work builds on trans-formers @xcite , which have become ubiquitous in natural language processing for sequence-to-sequence tasks. however, the authors report that the model often had difficulties to correctly initialize the object under test, as it was lacking the context information to do so. our benchmark contest includes such context information. recently, test completion has been tackled with machine translation methods by @xcite . the authors propose an rnn-based approach to assert generation called atlas, along with a dataset of the same name. the atlas dataset consists of 158, 096 tests paired with single line assert statements and is, to the best of the authors knowledge, the only available dataset for test completion available. note that the atlas dataset is less than half the size of contest, does not contain any context information, does not provide a project-wise split and in contrast to contest contains only single line assertions. atlas has been used in recent studies of @xcite , where the authors pre-train the transformer-based bart model on a large set of english texts and code artifacts and subsequently fine-tune it to generate assert statements. applying their model on our dataset is an interesting direction for future research. ## dataset java has been the most used programming language in 2020, and its unit test framework junit has become a widely adopted industry standard @xcite . driven by the vast amount of open-source projects available on github that employ junit as a testing framework, we chose to build contest by scraping junit projects from github. by limiting the projects to only one testing framework the test structure will stay more consistent throughout the dataset. furthermore, as junit requires annotating tests with @test, such methods can be located easily. ## approach our approach towards test completion is illustrated in figure 2 . similar to prior successful work on test completion and test generation @xcite , our model utilizes a transformer encoderdecoder architecture. additionally, we utilize context code that is called by either test method or focus method as additional input. also, we investigate two fundamental ways of encoding source code: (1) in form of linearized abstract syntax trees (asts), and (2) a tokenized version of the source code. our model pipeline is illustrated in figure 2 : we process two source files containing the test and the tested code by parsing relevant code parts into asts, which are combined to the so-called source tree describing the whole test. this tree is fed into an encoder-decoder transformer, which produces a assertion's linearized ast. this is compared to the target ast using a the cross entropy loss function. ## experiments we evaluate our transformer-based models on con-test to investigate the usefulness of contextual information and compare our syntax-based ast encoding with a token-only baseline. finally, we investigate the effect of generalizing between training and test projects using contest's project-level split. ## conclusion we have proposed a large-scale benchmark for automatic test completion coined contest. in addition to pairs of test and focal methods, our benchmark uniquely contains context and setup code, offers multiline targets, and defines project-level splits. we have shown in an ablation study that context information appears to be extremely relevant to the task of test completion, and that a sequenceto-sequence transformer baseline struggles with generalizing across projects. future research could aim to improve cross-project generalization, for example by fine-tune large scale pre-trained language models for code @xcite on contest.