The Program Testing Ability of Large Language Models for Code (2024)

Abstract: Recent development of large language models (LLMs) for code like CodeX and CodeT5+ shows promise in achieving code intelligence. Their ability of synthesizing program targeting a pre-defined algorithmic coding task has been intensively tested and verified on datasets including HumanEval and MBPP. Yet, evaluation of these LLMs from more perspectives (than just program synthesis) is also anticipated, considering their broad scope of applications. In this paper, we explore their ability of automatic test cases generation. We show intriguing observations and reveal how the quality of their generated test cases can be improved. Following recent work which uses generated test cases to enhance program synthesis, we further leverage our findings in improving the quality of the synthesized programs and show +11.77% and +4.22% higher code pass rates on HumanEval+ comparing with the GPT-3.5-turbo baseline and the recent state-of-the-art, respectively. Our code is publicly available at https://github.com/asdasxzxcq/TestCaseGen.

## introduction the community has witnessed a surge in the development of large language models (llms), which have achieved incredible ability in understanding and generating not only texts but also code. llms for code (codex @xcite , starcoder @xcite , codet5+ @xcite , etc.) have been widely adopted to a variety of applications to achieve code intelligence, and there is an apparent arms race between these llms. however, current evaluation of these llms mostly focuses on program completion/synthesis, despite the models can also be utilized in other applications, e.g., automatic unit test case generation. the ability of automatically generating proper test cases is of great desire to software engineering, * corresponding author yet challenging. traditional test case generation efforts primarily focus on creating diverse test inputs to identify faults in the code as much as possible via maximizing their coverage, e.g., line coverage and branch coverage @xcite @xcite @xcite , and they lack the ability of determining whether the code adheres to the aim of the function which is represented by input-output relationships. yet, desired test cases should not only show an high coverage but also present a correct understanding of the "true" desired input-output relationships in in the code being tested. being capable of synthesizing correct code implementations given docstrings, machine learning models and (especially) the recent llms for code seem capable of understanding the desired inputoutput relationship (described in natural language) of a function. this strong capability enables llms to generate unit test cases automatically and fulfill the aforementioned aim @xcite . however, the ability of code llms to automatically generate diverse test inputs paired with their correct test outputs, has not been systematically evaluated. @xcite compared codex with two opensource llms in a single setting and showed that the quality of test cases is of importance to the success of their method which improves program synthesis, but gpt-3.5 and advanced open-source llms that emerge afterwards are of course not evaluated. in this paper, we systematically compare the ability of recent llms for code in generating test cases from perspectives focusing on their correctness and diversity, considering that 1) program testing is of great interest in software engineering and software security as have been mentioned, 2) test cases can further be adopted to improve the program understanding @xcite and program synthesis performance @xcite , and 3) the ability of these llms in generating test cases has not yet been investigated systematically, despite the arms race. our analyses focus on algorithmic coding, based on the 164 problems from humaneval+ @xcite and 427 sanitized problems from mbpp @xcite . it is worth noting that, in practice, the model may encounter various scenarios when test cases are required to be generated. it may generate test cases when provided with only natural language descriptions in a docstring and without any specific program implementation targeting an algorithmic coding task, or it could generate test cases when given an "optimal" oracle implementation. in other situations, it may need to test its own imperfect generated program or the program generated by other models. therefore, in contrast to @xcite 's work which focuses on a single setting, we consider 4 different testcase generation settings (i.e., the "self-generated" setting that uses each llm to test programs synthesized by the llm itself, the "cross-generated" setting that lets all llms to test the same set of programs synthesized by a group of four llms, "oracle" which tests an oracle implementation, and the "placeholder" (as shown in figure 1 ), and we consider a collection of 11 llms. we conducted intensive experiments, from which intriguing takeaway messages are delivered. as previously mentioned, several very recent papers @xcite @xcite have shown that appropriate usage of generated test cases can improve the quality of program synthesis. yet, the quality of generated test cases largely impacts the performance of such methods. due to the lack of systematic evaluation of the testing ability of llms for code, it is unclear how to craft test cases that could be potentially more helpful to program synthesis. the studies in this paper also shed light on this. we show that, substantially improved program synthesis performance can be gained by utilizing takeaway messages in our studies. specifically, we can achieve +11.77% higher code pass rate on humaneval+, in comparison with the gpt-3.5-turbo baseline. compared with codet which is a very recent state-of-the-art, our solution gains +4.22% higher code pass rate. ## large language models for code in this section, we outline the evaluated models. we use some "small" models whose numbers of parameters are around 1b (to be more specific, from 770m to 1.3b in our choices) and some larger mod-els that achieve state-of-the-art performance in the task of program synthesis. for small models, we use incoder (1.3b) @xcite , codegen2 (1b) @xcite , codet5+ (770m) @xcite , and santacoder (1.1b) @xcite . as for larger models that achieve state-of-theart program synthesis performance, we use code-gen2 (16b) @xcite , codegen-multi (16b) @xcite , codegen-mono (16b) @xcite , starcoder (15b) @xcite , wizardcoder (15b) @xcite , codegeex2 (6b) @xcite , and gpt-3.5-turbo. we tested pass@1 of all models except gpt-3.5-turbo whose result can be directly collected from liu et al. (2023a)'s paper. by sorting their pass@1 from high to low, they are ranked as: gpt-3.5-turbo (61.7%), wizard-coder (46.23%, 15b), codegeex2 (29.97%, 6b), starcoder (27.9%, 15b), codegen-mono (26.15%, 16b), codegen2 (19.33%, 16b), codegen-multi (15.35%, 16b). the ranks on the mbpp dataset are similar. refer to appendix a.3 for more details. ## programs to be tested for evaluating the test case generation ability of the llms, we need an oracle to express the groundtruth functionality of the tested code. fortunately, huamneval @xcite and mbpp @xcite provide such oracles. in our experiments, we use an amended version of humaneval called humaneval+ @xcite , together with sanitized version of mbpp. these datasets are established to evaluate basic python programming performance of llms, and they contain 164 and 427 problems, respectively. ## main results for test case generation the experiment results of small and large llms on humaneval+ can be found in the above four takeaway messages can all be inferred from tables 1 and 2 . in addition to all these results, we conduct more experiments to further achieve the following takeaway messages. we conducted an experiment by first selecting programming problems in humaneval+, where the code pass rate of an llm is neither 0% nor 100%. then we separate its selfgenerated programs into two groups, with one group only contains programs that are considered as correct and the other only contains incorrect programs. in figure 3 : how the correctness of the test cases changes with their order when being generated. the best correctness and the latterly generated ones are more incorrect. this may be due to the fact that the model tends to first generate content with a high level of confidence (which is also more likely to be correct). ## improving program synthesis using the generated test cases high quality test cases are not only desired in program analyses, but also helpful to program synthesis. previous methods have successfully used generated test cases to improve the performance of llms in synthesizing programs. for instance, li et al. (2023a) designed a special prompt which involves the test cases as an preliminary, if they are available, for generating programs. one step further, chen et al. (2023) proposed codet, which leverages the llm to obtain test cases first and tests all synthesized programs with these test cases by performing a dual execution agreement, and it picks the programs in the largest consensus set (i.e., the consensus set with the most program implementations and test cases) as output to obtain state-ofthe-art program synthesis performance. we encourage interested reader to read the original paper. in the previous section, we have obtained results about many intriguing properties of the program testing performance of llms for code. in this section, we would like to drive the readers to think whether it is possible to utilize these results to improve the program synthesis performance, considering that the test cases (hand-crafted and given or automatically generated in particular) are widely and successfully used in program synthesis. we will show that, by utilizing takeaway messages in section 4, program synthesis performance of previous methods can be improved significantly. taking codet as an example, the method uses a placeholder to generate test cases and treats all the test cases as equally correct as a prior. however, as discussed in our third takeaway message, using selfgenerated programs helps to achieve more powerful ability in generating correct test cases. moreover, if multiple test cases are provided in a single run of generation given an llm, the correctness of the test cases decreases with their generation order, as shown in our sixth point. hence, to obtain superior program synthesis performance, we introduce two simple modifications to it: 1) we employ the "self-generated" setting instead of the "placeholder" setting for generating test cases, which means we used synthesize programs in prompts when generating test cases for each program, 2) we assign different weights to the generated test cases based on their order in each generation result, which means we used the rank of each generated test case to re-weight its contribution to the consensus set it belongs to. note that, inspired by the sixth takeaway codet+sg+rw. for codet, we follow their official implementation and generate 100 × 5 test cases for each problem. for fair comparison, we ensure that our solutions with sg and/or rw generate the same numbers of program implementations and test cases as codet does. hence, for each problem in humaneval+, we synthesize a program together with its 5 test cases for 100 times when sg and/or rw are incorporated, i.e., we have @xmath0, 2, 3, 4, 5}. it can be seen from the table that both sg and wr improves the program synthesis performance considerably on most llms, except for incoder, codegen2-1b, codet5+, and santacoder for which the test cases generated in the placeholder setting show similar or even higher correctness than in the self-generated setting and sg fails with them. for some llms, sg is more powerful, while, on the other models including san-tacoder and starcoder, rw is more powerful. in general, smaller models benefit more from rw than from sg + rw, probably because smaller models generate test cases with higher correctness rates in the placeholder setting than in the self-generated setting. by combining sg and rw, the program synthesis performance of most powerful llms in ## conclusion in this paper, we have performed thorough analyses of recent llms (mostly llms for code) in generating test cases for programs. through comprehensive experiments with 11 llms on programming benchmark datasets including humaneval+ and mbpp (the sanitized version), we have uncovered a range of intriguing characteristics of these llms for program testing. we have illustrated how the capabilities of these llms in generating test cases can be enhanced in comparing intensive empirical results in four different settings. based on our findings, we are also able to improve the performance of state-of-the-art llms in synthesizing programs with test cases of higher quality. we believe our work can provide new research insights and spark new ideas in program synthesis, test-case generation, and llm understanding, and we look forward to future exploration in these directions. we then concatenate the extra prompt with the code and feed the concatenation into each llm, for extracting test cases from the model output. when using humaneval+ and mbpp, we try removing test cases in the docstrings of the function, if there exist any, just to get rid of the broad hints from the docstrings @xcite . the temperature for generating test cases is kept as 0.2. once obtained, the generated test cases are then compiled, and evaluated for their correctness and diversity to report the pass rate p ′ and the coverage rate c. when calculating, for each problem and every set of completions generated, we create a temporary folder.