P ython S aga: Redefining the Benchmark to Evaluate Code Generating LLM s (2024)

Abstract: Driven by the surge in code generation using large language models (LLMs), numerous benchmarks have emerged to evaluate these LLMs capabilities. We conducted a large-scale human evaluation of *HumanEval* and *MBPP*, two popular benchmarks for Python code generation, analyzing their diversity and difficulty. Our findings unveil a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. Furthermore, we uncover a worrying prevalence of easy tasks that can inflate model performance estimations. To address these limitations, we propose a novel benchmark, *PythonSaga*, featuring 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels. The robustness of our benchmark is demonstrated by the poor performance of existing Code-LLMs. The code and data set are openly available to the NLP community at this [URL](https://github.com/PythonSaga/PythonSaga).

## introduction the rapid advancement of large language models (llm), such as gemini @xcite , @xcite , llama @xcite and palm @xcite , has achieved near-human or even superhuman performance @xcite in a wide range of nlp tasks. this surge has also led to the development of custom code generation models, such as codex @xcite , starcoder @xcite , codegen @xcite , code llama @xcite , codegeex @xcite . these specialized models, collectively referred to hereafter as "code-llms", harness the capabilities of llms for automated code generation from human descriptions. figure 1 shows a toy example with an input description from a human and an expected python code generated by a code-llm. the prevalence of python as the dominant programming language has significantly influenced the majority of code-llms to showcase their code-generation capabilities on python-specific benchmarks. consequently, humaneval @xcite , mbpp @xcite , apps @xcite , and ds-1000 @xcite have emerged as prominent benchmarks, leveraging data curated from popular coding platforms like @xcite , @xcite , and codeforces @xcite and crowdsourcing efforts. these benchmarks offer a diverse range of programming challenges, with sizes ranging from a few hundred instances in humaneval @xcite ) to several thousand instances in datasets like apps @xcite and mbpp @xcite . code generation benchmarks, like their nlp counterparts @xcite , are reaching saturation, revealing limitations in their ability to evaluate models comprehensively. figure 6 present in appendix reports pass@1 score foot_0 of recent code-llms on two popular benchmarks, hu-maneval @xcite and mbpp @xcite . this performance, approaching human-level capabilities, raises two key questions: in this paper, we introduce a comprehensive hierarchical classification of programming concepts, categorizing them into basic, intermediate, and advance levels (see section 3). we then rigorously evaluate two benchmarks, humaneval @xcite and mbpp @xcite , on two dimensions: diversity of programming concepts and user-perceived difficulty. our findings reveal a significant bias towards a small subset (<53%) of programming concepts, leaving the vast majority underrepresented. additionally, more than 80% of the problems are perceived as easy, raising concerns about the generalizability and effectiveness (see section 4). to address these limitations, in section 5, we propose a novel code generation benchmark, pythonsaga, featuring a balanced representation of 38 programming concepts across three difficulty levels in the form of 185 manually crafted problems. surprisingly, our experiments show poor pass@1 scores by the majority of the existing open (< 4.5%) and closed-source (< 13%) code-llms on pythonsaga. furthermore, a detailed analysis uncovers significant disparities in their capacity to handle different programming concepts and difficulty levels. code generation models: the remarkable surge in the popularity of llms has also been accompanied by significant advancements in code-llms. these models exhibit the capability to generate code in designated programming languages, guided by instructions presented in the form of prompts, functions, or docstrings. prominent examples of such code-llms include but are not limited to, codex @xcite , codegen @xcite , code llama (roziere et al., 2023), starcoder @xcite , codegeex @xcite , and starcoder2 @xcite . these code-llms are largely multilingual, capable of handling multiple programming languages, and their parameter sizes range from 1 billion to 70 billion. their training datasets encompass popular programming websites and code repositories such as github, leetcode, and geeksforgeeks. all popular code-llms primarily focus on python programs due to its widespread usage in ml and ai applications. python-based evaluation benchmarks: recent thrust in python code generation models also led to the development of several benchmark datasets. the py150 dataset @xcite , consisting of 150,000 python source files from github, serves as a valuable tool for llm evaluation. the apps dataset @xcite features 10,000 problems from platforms like codewars, at-coder, kattis, and codeforces. humaneval @xcite comprises 164 handwritten problems. the mbpp dataset @xcite contains 974 entry-level problems. additionally, the mathqa-python dataset @xcite , with 23,914 problems, assesses code synthesis from complex textual descriptions. limitations in existing benchmarks: current datasets for evaluating large language models (llms) often lack transparency and comprehensiveness in problem selection and categorization. this opacity hinders assessments of the generalizability and representativeness of the benchmarks, potentially leading to overestimation of llm performance on code generation tasks. to address this issue, this paper proposes a comprehensive problem categorization by outlining recommended concepts for problem inclusion, aiming to establish a rigorous and transparent benchmarking framework. ## conclusion and future work this study emphasizes the crucial need for a more balanced and comprehensive evaluation framework to ensure a fair and accurate assessment of large language models (llms) capable of generating code from human inputs. we address this gap by proposing an extensive categorization and hierarchy of programming concepts. an analysis of two prominent python code generation benchmarks reveals limited diversity in both programming concepts and difficulty levels. notably, we introduce a novel benchmark characterized by a uniform representation of concepts and difficulty, offering a more robust assessment paradigm. our findings suggest that existing benchmarks potentially overestimate llm performance on code generation tasks. this work lays the groundwork for the future development of diverse and representative python code generation benchmarks, paving the way for similar studies in other programming languages.