XC ode E val: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval (2024)

Abstract: Recently, pre-trained large language models (LLMs) have shown impressive abilities in generating codes from natural language descriptions, repairing buggy codes, translating codes between languages, and retrieving relevant code segments. However, the evaluation of these models has often been performed in a scattered way on only one or two specific tasks, in a few languages, at a partial granularity (e.g., function) level, and in many cases without proper training data. Even more concerning is that in most cases the evaluation of generated codes has been done in terms of mere lexical overlap with a reference code rather than actual execution. We introduce *xCodeEval*, the largest executable multilingual multitask benchmark to date consisting of 25 M document-level coding examples (16.5 B tokens) from about 7.5 K unique problems covering up to 11 programming languages with execution-level parallelism. It features a total of 7 tasks involving code understanding, generation, translation and retrieval. *xCodeEval* adopts an execution-based evaluation and offers a multilingual code execution engine, *ExecEval* that supports unit test based execution in all the 11 languages. To address the challenge of balancing the distributions of text-code samples over multiple attributes in validation/test sets, we propose a novel data splitting and a data selection schema based on the geometric mean and graph-theoretic principle. Our experiments with OpenAI’s LLMs (zero-shot) and open-LLMs (zero-shot and fine-tuned) on the tasks and languages demonstrate to be quite challenging as per the current advancements in language models.

## introduction automatically generating computer programs to solve complex problems has been a long-standing goal in ai @xcite . in recent * equal contribution years, specifically with the growth of large language models (llms), we have witnessed tremendous progress in synthesizing code that is not just relevant but also fully functional with no further human modification needed @xcite . the progress made in related tasks such as program synthesis @xcite , program repair @xcite , code translation @xcite , and code retrieval @xcite are having a profound impact on increasing developer productivity @xcite and aiding educators @xcite . despite the fact that advancements are expected to be general with benchmarks, their evaluation has been performed in a scattered way on a limited number of languages like python and java, on a partial granularity level such as at the level of a statement @xcite or function @xcite , focusing on only one or two specific tasks like program synthesis and translation, and in many cases without fine-tuning data @xcite or in terms of lexical n-gram based relevance @xcite rather than execution. we present a summary of the characteristics of existing program evaluation test-beds in tables 1 and 6 . to address these limitations, and drive further advancements in the creation of more general-purpose llms for problem solving, we introduce xcodee-val, the largest executable multilingual multitask benchmark to date consisting of 20m coding examples from about 7.5k unique algorithmic problems. it covers up to 17 programming languages with the parallelism of multilingual data. it features a total of 7 tasks involving code understanding, generation, translation and retrieval, and wherever appropriate it employs an execution-based evaluation protocol. a detailed documentation of the dataset can be found in (appendix a). figure 1 shows an example from xcodeeval; it includes a problem description in natural language, a buggy and bug- ## design principles & contribution xcodeeval is a result of a number of crucial design principles and contributes to solving many challenges in programming language evaluation as highlighted below. reasoning. as a genre, problem solving posits a unique set of challenges that require (a) understanding problem description in complex natural language, (b) expertise in data structures, algorithms, (c) complex reasoning that goes beyond memorization, and (d) generating programs of potentially hundreds of lines so that they can pass a comprehensive list of expertly designed tests. given the current progress in llms and their instruction following capability @xcite , competition-level problems that humans find challenging, provide an interesting benchmark to test many aspects of intelligence @xcite . ## xcodeeval: data, execution engine & tasks we construct our dataset from 25m openly available samples from codeforces.com for a total of 7514 distinct problems. each sample s @xmath0, and a problem p i can be solved by employing a set of algorithmic techniques t i ⊂ t , which we refer to as problem tags (e.g., 2-sat, binary search); see figure 8 for a complete list of tags. we created two different types of test sets: (i) compact: this set is intended to quickly test all languages, aimed at evaluating llm apis with a lower pricing structure for llm researchers in academia. (ii) titan: this set is designed to perform robust tests on all languages, especially for use in industry and/or open-weight llm setups. compact and titan test sets. to prevent data leakage, we put aside n h (= 1354) problems as held-out set d ho for the compact and titan test split. it ensures that the problems in the compact and titan test sets are not seen in training and models need to generalize to unseen problems. we then create d compact and d titan splits from d ho , while maintaining a balanced tag distribution and ensuring that all the tags in these two sets also exist in the training data, which could be a requirement for certain tasks (e.g., tag classification). for this, we iterate over a number of seeds and create random splits. let γ be the expected ratio of the number of samples in d compact and d titan , for each random split, we calculate a tag-wise ratio γ t , the ratio of the number of samples in d compact and d titan for each tag @xmath1. the geometric mean of {γ t } @xmath2. we select the split whose score is closest to γ. appendix b-algorithm 1 describes our method, which also ensures that d compact and d titan test sets contain the same tag sets as the training set. ## evaluation and analysis for all tasks except code translation, we evaluate on the compact split. for code translation from source languages, we used the compact small split (appendix d.4). ## evaluation of program synthesis on smaller models for program synthesis tasks, we finetuned starcoderbase-3b model with our trained data. we also evaluated the codellama-7b-instruct-hf and codellama-13b-instruct-hf models with our evaluation data. a 3b fine-tuned model is better than a 7b instruct model but worse than a 13b instruct model. we observe that training a smaller model with the training data performs well on our task rather than using a general-purpose instruct/chat model. however, large instruct models are better than smaller fine-tuned models. so, the impact of our training dataset varies between different scales. comparing the results between tables 3 and 5 also provides a general idea of how challenging our task is. ## conclusion & future work we have introduced xcodeeval , a large-scale multilingual multitask benchmark for code-based large language models. xcodeeval features seven different tasks involving code understanding, generation, translation and retrieval in up to 17 programming languages, and it employs an executionbased evaluation protocol. we have also presented execeval, a multilingual code execution engine that supports all the programming languages in xcodeeval . in summary, the combination of xcodeeval and execeval presents a novel framework that offers a fresh perspective for examining and analyzing large language models, facilitating comprehensive and to some extent highly interpretable investigations. we hope that by utilizing the extensive metadata and execution-based evaluation, there is potential for the discovery of new scaling laws and emergent capabilities.