Execution-Based Evaluation for Open-Domain Code Generation (2023)

Abstract: To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling – CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.

## introduction evaluations of nl-to-code generation systems, especially for general-purpose programming languages such as python, have put an increasing emphasis on methods that execute code to verify the results. the predominant approach for creating such test sets is to manually write test cases for canonical code solutions @xcite @xcite . the correctness of model predictions is then evaluated by seeing if generated code passes the test cases @xcite . compared to execution-free metrics such as text match against reference solutions, execution-based methods more rigorously assess the functional correctness of code @xcite . we analyze and highlight three aspects of odex ( §3). first, odex has broad domain coverage of 79 libraries, with 53.4% of the problems employing at least one library. second, odex contains queries in four different languages, with 439, 90, 164, and 252 samples in english, spanish, japanese, and russian, as shown in figure 1 . third, odex addresses three unique challenges in open-domain code execution: irreproducible runs (figure 1 a ), randomized outputs (figure 1 b ), and specialized equivalence checks (figure 2 ). we evaluate two state-of-the-art code llm families, codex and codegen, on odex ( §5). our study shows that larger model sizes and augmented training data improve execution accuracy. meanwhile, we observe satisfactory multilingual capabilities, despite that neither model was specifically designed for multilingual usage. however, we find that models face greater yet varied challenges with open-domain queries compared to closed-domain queries ( §5). specifically, codex achieves higher (3) natural language intents as part of the docstrings (english translations are not included in the actual non-english inputs during inference). gray boxes indicate places for code solutions. as shown on the right, a code lm fills out the gray boxes with code solutions, which are then executed on the unit tests underneath. notably, writing unit tests for open-domain queries is often more challenging: a requires simulated execution due to the difficulty of reproduction; b is verified through approximate equivalence. prior work focuses more on basic assertions, as in c and d . ## the odex dataset in this section, we describe our four-step process of constructing the odex dataset. we first collect resources of natural, open-domain coding queries ( §2.1). next, we establish the annotation standard and procedures for test case creation ( §2.2). we then describe the annotator hiring and working processes ( §2.3). finally, we conduct checks to ensure data quality ( §2.4). ## dataset analysis we analyze odex from three aspects: domain diversity ( §3.1), sample complexity ( §3.2), and execution support ( §3.3). ## experiment setup code llms have achieved strong results on multiple code generation tasks, yet their open-domain proficiency is understudied due to the limited domain settings of past datasets. to examine model capabilities in the open domain, we evaluate two top-performing model families, codex and code-gen, on odex. we perform evaluations using a prompting setting, without finetuning any model. we introduce the baseline models, the prompt settings, and lay out the metrics for evaluation. the codex family at the time of this work, codex had three publicly available models. code-cushman-001 (c1) is a 12b codex model in @xcite . code-davinci-001/002 (d1, d2) are two 175b gpt-3 models. 6 the codegen family codegen @xcite models are auto-regressive models trained on a combination of nl and code corpora, differing in model sizes (350m, 2.7b, 6.1b, 16.1b) and training data. models are progressively trained on thepile (gao et al., 2020), bigquery, foot_5 and bigpython datasets are denoted as nl, multi, and mono. the most powerful codegen-16.1b-mono, performs similarly to code-cushman-001 on the humaneval and mtpb datasets. prompt design for fair comparison, we use the same prompt for both model families. while prompting with few-shot in-context examples may improve, our experiments do not always find this helpful for both models. therefore, we report zeroshot results as baselines and leave few-shot results to §7. creating zero-shot prompts only requires content from the test sample. following chen et al. (2021), we construct prompts by concatenating function context and a docstring. a docstring includes the nl intent and optional unit tests (compared in §7). figure 6 shows an example prompt. evaluation metrics we follow chen et al. ( 2021 ) and measure the execution accuracy using the pass@k metric, by computing the fraction of problems having at least one correct prediction within k samples. we also compare it with a series of execution-free metrics later in §5. ## experiment results we first present the overall performance of two model families on odex ( §5.1). next, given the unique challenges of open-domain code, we study the variances between open-and closed-domain problems ( §5.2), and in individual domains ( §5.3). ## comparing to execution-free metrics in this section, we study the alignment between execution-based evaluation and five execution-free metrics, identifying advantages for both types. ## conclusion we present odex, an open-domain code generation dataset supporting execution-based evaluation via human-written test cases. odex not only supports execution-based evaluation of code using test cases, but also extends the task to the open domain, covering 79 diverse python libraries and four natural languages (english, spanish, japanese, and russian). comparing two state-of-the-art code generation models, codex and codegen, our dataset effectively unveils their varied behaviors between program domains and language contexts. odex serves as a comprehensive nl-to-code benchmark given its open-domain coverage, multi-natural language queries, and multi-metric support. when bringing code execution to open domain scenarios, our explorations also reveal emerging challenges in test creation and reliable execution, which we hope that our dataset will enable future work to tackle.