One-to-many testing for code generation from (just) natural language (2024)

Abstract: MBPP is a popular dataset for evaluating the task of code generation from natural language. Despite its popularity, there are three problems: (1) it relies on providing test cases to generate the right signature, (2) there is poor alignment between instruction and evaluation test cases, and (3) contamination of the exact phrasing being present in training datasets. We adapt MBPP to emphasize on generating code from just natural language by (1) removing ambiguity about the semantics of the task from the descriptions, and (2) evaluating generated code on multiple sets of assertions to account for ambiguity in the syntax. We compare popular open and closed weight models on the original (MBPP) and adapted (MBUPP) datasets.

## introduction code generation from natural language (nl-tocode) is a popular task to evaluate the capabilities of language models @xcite @xcite . one of the most popular nl-to-code datasets is the mostly basic python programs (mbpp) dataset @xcite . in this dataset, each problem contains a natural language description, a code solution and three test cases in the form of assert statements. we identify three main problems with mbpp. first, it heavily relies on test cases to identify syntactic properties of the code to generate, as the provided assertions require a specific signature. second, descriptions sometimes contain instructions that the assertions are not testing for, like asking to sort "using heap queue." third, being a popular dataset distributed on many channels, data contamination is a significant issue @xcite . in this paper, we introduce an adapted code generation benchmark, called mbupp foot_0 , that allows for the description to be underspecified with respect to syntactic properties of code. each problem consists of a text description as input to the model, and a set of assertions to validate the output. we generate both the descriptions and assertions from mbpp problems using a combination of llms, intuition and validation. additionally, we provide results of different open and closed weight models on mbpp and mbupp. we show which assertions are more often picked, indicating data contamination. further, we release the dataset and the model generations to seed further research in this area. we make the following contributions. ## motivating example as an example, let us look at the problem "write a function to find sequences of lowercase letters joined with an underscore using regex" and the associated assertions are (with based on just the text description, it is not clear if the user expects a function @xmath0. the tests also do not evaluate whether the function actually uses a regular expression or not. our adapted benchmark puts all emphasis on the "nl" part of nl-to-code. we assume that a user is not specific about the syntax of the program and does not care about it: they want to obtain any function that does what they describe. the adapted write a function to convert the given binary number to its decimal equivalent. given only the description, any code generator returns a function. instead of providing the signature, which users will not likely do, we match the generated function to the signature of our assertions and then verify if the program satisfies any of the assertion sets. ## mbupp an example of an evaluation in mbupp is shown in figure 2 . the only input to the code generator is a text description. this text description is allowed to be underspecified with respect to syntactic properties of the function, like argument order and types (data structures) used to represent the output, and we provide multiple sets of assertions that capture this underspecification. additionally, if multiple functions are generated to solve the problem, we verify if any of them satisfies the assertions to allow the generator to use helper functions. we adapt benchmarks in two phases: improving the text descriptions and obtaining sets of assertions to capture ambiguity on syntactic properties. write a function that matches the beginning of a string to a word. write a function that matches if there is a word at the beginning of a string. write a function that matches a word at the beginning of a string. write a function that matches if there is a word at the beginning of a string. ## results on mbupp we describe our evaluation setup, main results, and further analysis on behaviour of different models. ## related work evaluating the coding abilities of large language models is a hot topic. mbpp @xcite and humaneval @xcite were among the first benchmarks and are still the most widely used. they have been translated to different languages @xcite @xcite . newer benchmarks like natural-codebench @xcite and pythonsaga @xcite consist of harder programming problems. all of these benchmarks consist of test sets used to evaluate the performance of models on these. evalplus @xcite improves the functional testing of code generation benchmarks by generating more test cases. these benchmarks and evaluations contain a function signature and/or test cases. we focus on code generation from only natural language and adapt the tests to account for any ambiguity. ## conclusion in this paper, we introduce mbupp, an adaptation of mbpp which addresses three main challenges with the original dataset: (1) ambiguity and underspecification in the descriptions, (2) contamination of the dataset by being present in common training corpora of models, (3) poor alignment of the assertions with the description. we show results of popular open and closed weight models on the original and adapted dataset. we also present analysis on different properties of mbupp, diversity and temperature of the generations.