L 2 CE val: Evaluating Language-to-Code Generation Capabilities of Large Language Models (2024)

Abstract: Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs. Despite promising results, there is a notable lack of a comprehensive evaluation of these models’ language-to-code generation capabilities. Existing studies often focus on specific tasks, model architectures, or learning paradigms, leading to a fragmented understanding of the overall landscape. In this work, we present L2CEval, a systematic evaluation of the language-to-code generation capabilities of LLMs on 7 tasks across the domain spectrum of semantic parsing, math reasoning, and Python programming, analyzing the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition, we assess confidence calibration, and conduct human evaluations to identify typical failures across different tasks and models. L2CEval offers a comprehensive understanding of the capabilities and limitations of LLMs in language-to-code generation. We release the evaluation framework1 and all model outputs, hoping to lay the groundwork for further future research. All future evaluations (e.g., LLaMA-3, StarCoder2, etc) will be updated on the project website: https://l2c-eval.github.io/.

## introduction language-to-code (l2c foot_1 ) is a type of task that aims to automatically map natural language descriptions to programs, which are later executed to satisfy the user's demand @xcite . as illustrated in figure 1 , language-to-code is the foundation of many applications in ai, such as task-oriented dialogue systems @xcite , coding assistant @xcite , language interfaces to databases @xcite , and robotic control @xcite . it has also served as a great testbed for evaluating various language understanding capabilities of nlp systems, such as logical and math reasoning @xcite , grounded language understanding @xcite , and tool use @xcite . recent progress on large language models (llms) @xcite @xcite , especially those specifically trained for coding @xcite @xcite , has shown that llms trained on a mixture of text and code are able to perform language-to-code generation under few-shot or even zero-shot learning settings @xcite . however, the modeling factors that affect the performance of llms for such l2c tasks-including model size, training data mixture, prompting methods, and instruction tuning-are poorly understood. in addition, there lacks a consistent evaluation of different llms on the same spectrum of language-to-code tasks, making it difficult for the users to decide which models to use for certain tasks or if they should resort to finetuning their own models. beyond model performance, model properties such as robustness to prompt and confidence calibration are also crucial for understanding the reliability of llms, but such properties have not been systematically studied for l2c tasks. in this work, we present l2ceval, providing a systematic evaluation of the language-to-code generation capabilities of llms. l2ceval includes a wide range of state-of-the-art models, specifically 56 models from 13 different organizations, all evaluated on three core domains of language-to-code generation tasks: semantic parsing, math reasoning, and python programming. our l2ceval framework includes extensive evaluations of models as small as 1 billion parameters, to significantly larger ones such as falcon-180b, as well as davinci and gpt-4 models from openai. we also benchmark models that are trained on different mixtures of data of varying sizes (35b ∼ 3.5t tokens), as well as models that are instruction-tuned, from both open-source and open-access proprietary categories. our work is the first to conduct extensive and thorough comparisons of llms for language-to-code generation across multiple dimensions of variation. to summarize, we release l2ceval and its main contributions are as follows: through our work, we hope to provide insight into applying llms to l2c applications, as well as building future llms. the main motivation behind l2ceval is to provide a comprehensive evaluation of language-to-code generation capabilities and understand what affects such l2c capabilities. in the following sections, we first discuss the key desiderata in design of l2ceval in § 2.1, then the formulation of l2c in § 2.2, then in § 2.3 we introduce the domains and specific tasks we consider for l2ceval, as well as the reasons for choosing such tasks. finally, in § 2.4, we describe the models included in l2ceval and the model selection process. ## results and analysis we organize the experiment results and analysis as follows. we first discuss different scaling effects in § 3.1, then in § 3.2 and § 3.3, we analyze how pretraining data mixture and instruction-tuning affects the models for l2c tasks. to study model robustness, in § 3.4, we measure the sensitivity of the models on the few-shot demonstrations, and show model confidence calibration in § 3.5. finally, we present an error analysis in § 3.6. ## limitations while we strive to provide a comprehensive and fair evaluation of llms in l2c tasks, here we also discuss the limitations of l2ceval. generation using greedy decoding. in this work, we use greedy decoding to generate a single program for each example as the models' output. while this is the most efficient way of generation and ensures fair comparison for different models as it is not affected by factors like sampling temperature, it is also relatively noisy @xcite . for tasks such as mbpp or python programming in general, sampling k solutions then measure pass@k (any of the k programs being correct) or n@k (i.e., # the k programs being correct) are better as they give the model k tries to generate the correct program to lower the variance. for python programming tasks, such methods are closer to practical use cases as we typically have test cases that can filter out some incorrect programs in the samples. for other tasks, having a better pass@k also provides opportunities for post-generation reranking methods such as @xcite @xcite . however, the cost for evaluating pass@k or n@k is k times of the compute compared with greedy decoding, thus we choose to only evaluate greedy decoding in this work and leave sampling-based evaluation to future work. execution-based evaluation. moreover, we mainly rely on execution-based evaluation (i.e., execution accuracy) for this work. however, such evaluation may produce spurious programs, i.e., false-positive programs that achieve the correct execution result by chance @xcite . in this work, we adopt human evaluation to measure the problem of spuriousness and found non-trivial portion of ''correct'' programs being spurious for spider but not for other datasets. in addition, execution may not always be straightforward in practice, especially when complex dependencies and potentially harmful programs are considered @xcite . ## conclusions in this paper, we present l2ceval, a comprehensive evaluation framework for natural language to code generation, and we evaluate 56 models from 13 organizations, on 7 tasks from 3 core domains. l2ceval investigates models' performance on a variety of axes such as model scale, training data mixture, sensitivity to few-shot exemplars as well as the impact of instruction tuning, inter alia. we also present an analysis on the model calibration and conduct a human evaluation of common error modes across different models. we hope our study will provide useful insights for the community into applying llms for downstream code applications and future model development efforts.