N atural C ode B ench: Examining Coding Performance Mismatch on H uman E val and Natural User Queries (2024)

Abstract: Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the best-performing GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github.com/THUDM/NaturalCodeBench.

## introduction large language models (llms) pre-trained on extensive open code repositories @xcite @xcite have demonstrated impressive performance on code synthesis and even achieve performance comparable to average human level in coding competitions @xcite . unlike open * sz, hz, xl, and qz contributed equally. as a result, evaluating code synthesis presents unique challenges in the era of llms. traditional evaluation metrics by token matching @xcite @xcite show a weak correlation with human judgement @xcite and overlook functional correctness of the generated code @xcite . recently, execution-based evaluation has gained increasing popularity, where code generated by models is tested through unit tests to verify its functional correctness. it leads to the development of several benchmarks, including humaneval @xcite , mbpp @xcite , mbxp @xcite , codecontests @xcite , and ds-1000 @xcite . notwithstanding their commendable reliability and accuracy, these benchmarks fall short to sufficiently capture the wide range of needs and complexity found in real-world engineering applications. they are primarily limited to well-defined coding problems in algorithm, program basics, or data science. for example, as shown in figure 1 , a problem from humaneval @xcite tests the implementation of a basic function has_close_elements and takes floatingpoint arguments as inputs. however, in practical applications, user engineering requirements can be much more complex and varied. in figure 1 , we showcase an example adapted from a real user query, where the user asks to read and parse xml files given certain tags. difficult and costly though it is, curating a benchmark composed of such problems is meaningful for evaluating the real user experience of llm code synthesis. contributions. in light of the challenge, we introduce naturalcodebench (ncb), a challenging application-driven dataset for code synthesis evaluation. ncb is dedicated to creating a reliable evaluation environment that is more aligned with real-world applications. we leverage an codegeex @xcite online services to collect real and diverse application-related user queries. after filtering and reprocessing, 402 high-quality python and java problems are compiled, covering 6 domains including software, front-end, system administration, and artificial intelligence, highlighting practical scenarios. beyond basic data structures like lists and numbers, the test inputs for ncb problems include versatile file types and other complex structures, making it more challenging. the challenging nature of ncb necessitates significant human labor in its annotation process to improve construction efficiency, we tailor a semi-automated annotation pipeline to curate highquality, testable, and useful queries with corresponding test cases. specifically, we employ @xcite to generate reference solutions followed by manual correction. subsequently, gpt-4, guided by the problem descriptions and reference solutions, generates multiple test cases, which are also refined with manual correction, for each problem. consequently, the annotators are only required to correct any errors, substantially reducing the time and manpower required. comparative experiments reveal that our semi-automated pipeline can quadruple the construction speed of the evaluation framework, as evidenced by tests involving programming experts with or without the pipeline. based on ncb, we conduct extensive experiments on a variety range of llms, encompassing 39 apis or open models. the results indicate that although certain llms demonstrate comparable performance on established benchmarks like humaneval, they exhibit significant performance disparities when evaluated using ncb. it suggests that there may be inadequate focus on optimizing llms for practical coding applications, or have conducted over-specified optimization on humaneval-style problems. more importantly, even the best-performing gpt-4 only reaches about a pass rate of 53%, demonstrating a large room for llms to improve their coding skills to face realworld coding challenges. to facilitate community research, we pack up the whole ncb testing environment into a docker image and make its development set publicly available. to sum up our contributions: ## benchmark construction the overview of ncb is shown in figure 2 . the pipeline of constructing ncb consists of four steps: 1) collecting and filtering high-quality problems from online services (section 2.1) 2) constructing a complete evaluation framework through a semi-automated pipeline (section 2.2) 3) designing prompts to align different models (section 2.3) 4) translating all problems and instructions to produce bilingual versions (section 2.4). ## dataset statistics we provide more detailed statistics in each problem within the dataset has been carefully curated with a set of test cases to assess the correctness of solutions. on average, there are 9.3 test cases associated with each problem. these cases are strategically designed, with about 60% focused on enhancing statement and branch coverage, and the remaining 40% dedicated to evaluating the robustness of solutions against corner values. the average word count for each problem in the ncb is 78.3. compared with other benchmark. in addition, ncb includes more data types. furthermore, ncb focuses on assessing the model's ability to handle multiple file formats, a type of data that is both very commonly used in daily life and relatively challenging to process. we note that the problems involving files have fewer test cases, since gpt-4 still struggles to fully generate various types of file . this is also more challenging for human annotators to design compared to simpler data types. on the other hand, ncb is also limited by its size due to the high costs of problems collection and the construction of the evaluation framework. we are continuously working on expanding our benchmark. ## conclusion we propose naturalcodebench for evaluating the code generating ability of llms. our benchmark comprises a total of 402 problems selected from coding online services, and it supports automatic evaluation of code generated by llms. we have also proposed a semi-automated pipeline for efficiently constructing the entire benchmark, achieving an efficiency gain of more than 4x compared to manual construction. we hope that ncb can provide a fair environment for the comparison between models, and our pipeline can also provide inspiration to other complex tasks or domains where evaluation costs are high.