C ode S cope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLM s on Code Understanding and Generation (2024)

Abstract: Large Language Models (LLMs) have demonstrated remarkable performance on assisting humans in programming and facilitating programming automation. However, existing benchmarks for evaluating the code understanding and generation capacities of LLMs suffer from severe limitations. First, most benchmarks are insufficient as they focus on a narrow range of popular programming languages and specific tasks, whereas real-world software development scenarios show a critical need to implement systems with multilingual and multitask programming environments to satisfy diverse requirements. Second, most benchmarks fail to consider the actual executability and the consistency of execution results of the generated code. To bridge these gaps between existing benchmarks and expectations from practical applications, we introduce **CodeScope**, an execution-based, multilingual, multitask, multidimensional evaluation benchmark for comprehensively measuring LLM capabilities on coding tasks. CodeScope covers **43 programming languages** and **eight coding tasks**. It evaluates the coding performance of LLMs from three dimensions (perspectives): **length**, **difficulty**, and **efficiency**. To facilitate execution-based evaluations of code generation, we develop **MultiCodeEngine**, an automated code execution engine that supports 14 programming languages. Finally, we systematically evaluate and analyze eight mainstream LLMs and demonstrate the superior breadth and challenges of CodeScope for evaluating LLMs on code understanding and generation tasks compared to other benchmarks. The CodeScope benchmark and code are publicly available at https://github.com/WeixiangYAN/CodeScope.

## introduction driven by advances in deep learning and nlp, llms have demonstrated outstanding proficiency in various generation and understanding tasks @xcite . however, existing benchmarks @xcite @xcite for evaluating llms mainly focus on nlp tasks, such as common sense reasoning, academic examination, and authenticity verification. existing evaluation methods are significantly insufficient in terms of evaluating completeness and comprehensiveness for code understanding and generation capabilities of llms. firstly, many code llms, such as codet5+ @xcite , wizardcoder @xcite , and code llama @xcite , employ their own specific single-task evaluation datasets, making it infeasible to comprehensively compare the performance of various llms on code understanding and generation tasks on a unified standard. secondly, existing datasets mostly evaluate llms on code tasks @xcite for a narrow range of popular programming languages, with a focus on python and single program synthesis tasks. however, software development often involves multiple programming 1 5511 languages, each following different programming paradigms such as object-oriented, functional, and procedural. evaluating llms within a multilingual framework can reveal their ability to generalize across various languages and paradigms. moreover, the complementarity between multiple tasks facilitates a comprehensive evaluation of the overall performance of llms, ensuring that an llm is not over-optimized for a specific task and can maintain strong performance across diverse tasks. importantly, multitask settings more accurately simulate the various requirements and challenges faced in real-world software development practices and hence better test the generalizability of llms. thirdly, most studies (e.g., widely used benchmarks @xcite and @xcite ) rely on matching-based evaluation metrics, such as bleu @xcite or @xcite , to measure the quality of generated code. however, these metrics may not reflect the practical applicability of the code, as they only compare the surface form similarity between the generated code and the reference code @xcite . the ultimate goal of code generation is to produce code that can execute correctly and accomplish specific tasks. therefore, execution-based metrics, which evaluate the functionality and correctness of the generated code by running it on test cases or comparing its output with the expected output, are more reliable and informative. to address these limitations, we propose code-scope, a benchmark that evaluates the coding proficiency of llms using execution-based metrics in a multilingual and multitask setting. codescope consists of eight tasks for code understanding and generation, covering 43 programming languages with an average of 13 languages per task. the task descriptions are summarized in to support codescope, we develop a multilingual code execution engine, multicodeengine, which extends the execeval engine @xcite to accommodate 14 programming languages for code generation tasks. we also establish eight strong baselines for each task to facilitate comprehensive comparisons of coding capabilities of llms. we expect these explorations will provide a deep understanding of the strengths and limitations of llms on code understanding and generation tasks and provide valuable guidance for future research directions. our contributions can be summarized as follows: ## related work many existing benchmarks for code understanding and generation tasks do not use execution-based evaluations. for example, @xcite and @xcite only use matching-based metrics, such as bleu or code-bleu, which compare the surface form similarity between the generated code and the reference code. however, these metrics may not capture the practical applicability of the code, as they can be misled by syntactically correct but semantically incorrect code, or by different implementations of the same functionality. previous research has shown that code lexical similarity and execution correctness are weakly correlated @xcite @xcite . a recent benchmark, xcodeeval @xcite , uses executionbased metrics in a multilingual and multitask setting, but some of its tasks are not relevant for llms, such as code retrieval, which requires a large and reliable code knowledge base that is not yet available for llms. furthermore, we found several flaws in the xcodeeval dataset, such as the inclusion of russian language data, which biases the natural language understanding of the instructions; inconsistencies between test cases and actual execution outputs; and the presence of invalid "cheat codes" that users have submitted to the website. in addition, most of the related works @xcite @xcite @xcite evaluate model performance on program synthesis tasks with python as the target language. among the existing datasets for program synthesis, humaneval @xcite is the most popular, with 164 problems and an average of 6.7 unit tests per problem. mbpp @xcite contains 974 entrylevel programming tasks, while @xcite includes 23,914 more advanced programming problems. apps @xcite is designed to pose more challenging programming problems. however, the coverage of programming languages in program synthesis tasks is still limited. some recent studies @xcite have attempted to expand the range of programming languages in program synthesis task, but they only cover a few languages. mbxp @xcite is a dataset that covers ten programming languages, generated by a scalable transformation framework. humaneval-x @xcite is another dataset that covers five programming languages, created by human translation. moreover, some recent research @xcite has pointed out the limitations of humaneval in evaluating the contextual appropriateness of the generated code. @xcite propose the semantics-based codenet benchmark, which significantly increases the variety of supported programming languages, yet the evaluation tasks remain relatively limited as code similarity and classification, and code translation. notably, experts find about half of the solutions in the codenet datasets are incorrect @xcite . @xcite introduce aixbench, which includes 175 java samples. however, due to the absence of unit tests, model performance has to be evaluated manually. @xcite translates the humaneval and mbpp benchmarks into eighteen languages using compiler methods, though the translation accuracy is not guaranteed. classeval @xcite evaluates llms in the complex scenario of classlevel program synthesis, including 100 class-level python program synthesis samples. this study indicates that current llms still face considerable challenges in effectively handling class-level code generation. we provide a more detailed discussion of other code evaluation benchmarks in section a.2 of the appendix. ## the codescope benchmark codescope evaluates the performance of llms on both code understanding and generation tasks. more details on dataset construction for each task are in appendix a. ## comparison with humaneval and mbpp benchmarks the rankings of these llms in code understanding are different from their rankings in humaneval and mbpp. for example, gpt-4, which ranks highest in humaneval and mbpp, is only fifth in code-scope (understanding). this indicates that strong performance in code generation tasks does not necessarily imply a good understanding of complex code. in codescope (generation), we use the same method to calculate the overall score. gpt-4 and gpt-3.5 do much better in code generation than in humaneval and mbpp. this disparity may be attributed to two reasons. first, codescope (generation) tests the general ability of llms to generate code for multiple objectives and languages. unlike humaneval and mbpp, which only test nl-to-pl tasks, codescope tests nl-to-pl, pl-to-pl, and nl&pl-to-pl tasks, examining the correctness, quality, and efficiency of the generated code, as well as the adaptability of llms to different languages. second, codescope (generation) presents more complex and diverse problems, with varying levels of difficulty. in contrast, humaneval and mbpp feature simpler, predefined problems. for instance, the average number of tokens in solutions is 53.8 and 57.6 for humaneval and mbpp, respectively, but 507.6 for codescope (generation). consequently, some llms that perform well in humaneval and mbpp, such as wizardcoder, do not fare as well in codescope (generation). in codescope (overall), the rankings of llms on codescope, humaneval, and mbpp are not consistent. this inconsistency highlights the advantages of codescope in terms of its breadth and challenge. codescope evaluates both code generation and code understanding skills, which are more relevant for real-world programming scenarios. additionally, codescope employs multilingual, multidimensional, multitask, and execution-based evalua-tion methods, enhancing the difficulty and diversity of the evaluation. codescope simulates the actual programming environment better, and provides a more comprehensive and detailed framework for evaluating the coding skills of llms. ## conclusion we present codescope, the first comprehensive benchmark for evaluating llms on coding tasks. codescope covers 43 programming languages, eight coding tasks, and three evaluation dimensions, using a fine-grained, execution-based evaluation method. we evaluate and analyze eight popular llms on codescope, and reveal their strengths and weaknesses on different tasks and settings. we also compare codescope with other benchmarks, and show the importance of codescope in testing llms on real-world programming scenarios with multitasking, multilingual, and multidimensional challenges. we offer a comprehensive resource, tool, and benchmark for evaluating llms on code understanding and generation skills, aiming to advance future research in this area. in future work, we will focus on further augmenting llms' advanced capabilities in processing and generating complex code. future studies can progress along two distinct trajectories. firstly, enhance the programming capabilities of llms to directly solve various challenging problems, aiming to achieve over 90% performance level on codescope. secondly, explore using autonomous agents to achieve a more effective collaborative division of labor, which could help solve complex programming challenges more efficiently. through these avenues, we expect to drive the expansion of the frontiers in the domain of code intelligence with llms. 9 5519