D ev E val: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories (2024)

Abstract: How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs.To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,825 testing samples from 115 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs’ coding abilities in real-world code repositories. For example, the highest Pass@1 of gpt-4 only is 53.04% in our experiments. We also analyze LLMs’ failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs’ predictions have been released.

## introduction code generation with large language models (llms) has attracted lots of researchers' attention @xcite @xcite , and some commercial products have been produced, e.g., github copilot @xcite . with more and more llms emerging, how to evaluate llms on code generation remains an open question. conala @xcite concode @xcite humaneval @xcite mbpp @xcite apps @xcite pandaseval @xcite numpyeval @xcite aixbench @xcite classeval @xcite codereval @xcite deveval (ours) context-aware dependencies (i.e., invocations of code elements defined in current repositories). the benchmark should cover both types of code and ensure their ratios are realistic. the number of dependencies should also be consistent with real-world repositories. however, as shown in based on deveval, we propose repository-level code generation, which simulates the developers' coding process in a working repository. the task asks models to write the code based on requirements and a complete repository. we evaluate 8 popular llms (i.e., gpt-4 (ope-nai, 2023b), gpt-3.5 @xcite , starcoder 2 @xcite , @xcite ). these llms exhibit low performance on deveval, especially compared to their performance on previous benchmarks. for example, gpt-4-turbo-1106 achieves a pass@1 score of 80% on hu-maneval, while its highest pass@1 on deveval is only 53.04%. our results reveal the coding abilities of these llms in real-world repositories. we further analyze failed cases and summarize the shortcomings of existing llms in deveval. in summary, our contributions are as follows: intra-class dependency: imapclient.py::imapclient::_command_and_check imapclient.py::imapclient::folder_encode intra-file dependency: imapclient.py::namespace cross-file dependency: imap_utf7.py::decode_utf7 response_parser.py::parse_response def test_namespace(self): self.set_return(b'(("&ap8-." "/")) nil nil') self.assertequal(self.client.namespace(), ((("\xff.", "/"),), none, none)) . . . . def namespace(self): """return the namespace for the imap account as a tuple of three elements: personal, other, and shared. the function should send the namespace command to the server and receive the response. then, it parses the response and converts it into the desired format. :param self: imapclient, an instance of the imapclient class. :return: namespace. the namespace for the account as a tuple of three elements. each element may be none if no namespace of that type exists, or a sequence of (prefix, separator) pairs. """ @xmath0._command_and_check("namespace") @xmath1: if item is none: parts.append(item) else: @xmath2, separator in item: if self.folder_encode: @xmath3.append((prefix, to_unicode(separator))) parts.append(tuple(converted)) return namespace(*parts) we hope deveval can align with the actual experiences of developers during the practical development process. by deveval, practitioners can pick up superior llms and facilitate the application of code generation techniques in real-world repositories. ## benchmark construction as shown in figure 3 , the collection of deveval consists of five stages. stage ❶: repository selection. pypi (pypi) is a rich data source for real code repositories. we identify the top 10 popular programming topics in pypi and select the top 50 repositories under each topic. the selection follows three criteria: open-source licenses, non-fork and non-malicious repositories, and explicit unit tests. we download the latest released versions in november 2023 and finally obtain 500 practical projects (10 topics * 50 projects). stage ❷: function parse. we extract functions from 500 repositories and exclude trivial functions (i.e., empty or initialization functions). we extract each function's signature and function body (i.e., reference code). the other programs with current repositories are considered as repository contexts. finally, this stage obtain 590,365 candidate functions. stage ❸: tests construction. we extract test cases from repositories invoking specific candidate functions. we use a public frameworksetuptools to automatically install running environments and run test cases with a popular testing framework -pytest. candidate functions without executable test cases are excluded. meanwhile, we ensure our test cases succeed in the reference code and fail in the wrong programs. finally, we retain 3,764 candidate functions. stage ❹: human annotation. we engaged 13 developers to manually annotate requirements and reference dependencies for each candidate function. given their countries of residence, all annotators obtain adequate payments. through discussions with annotators, we establish two criteria for requirements. naturalnessensuring the requirement reads like a natural description from the perspective of a real-world developer. functionality-demanding clear descriptions of the code's purposes and input-output parameters. each requirement undergoes a dual-annotation process, with one annotator assigned to its initial drafting and another responsible for a meticulous double-check. trivial functions (e.g., shortcut functions) and functions violating the ethical code (e.g., malware) are excluded. subsequently, the same 13 annotators review the reference code and label its reference dependencies. finally, we retain 2,846 functions with high-quality requirements and reference dependencies. stage ❺: benchmark construction. we select candidate functions to construct based on two criteria: consistent with the data distribution of 500 realworld repositories and including as many functions as possible. finally, we select 1,323 (73%) nonstandalone functions and 502 (27%) standalone functions to construct deveval. ## discussion results on different program types. figure 5 shows pass@1 of gpt-4 on different program types (i.e., standalone and non-standalone). we have two observations from the results. ❶ contexts are crucial to generating non-standalone functions. for example, adding local files improves the pass@1 on non-standalone functions from 11.26 to 54.5. ❷ contexts also benefit standalone functions. this is attributed to the domain knowledge within contexts, aiding llms in understanding requirements. ❸ there is considerable room for improving llms on both programs. how to effectively retrieve relevant contexts is a key problem. results on different dependency types. figure 6 shows the recall@1 of gpt-4 on different dependency types (i.e., intra-class, intra-file, and crossfile). the results yield two insights. ❶ without context, llms can reason about some simple dependencies from requirements (e.g., initialization functions of returned objects), but still exhibit low recall@1 values across three dependency types. ❷ with contexts, llms exhibit an improvement in generating dependencies. nevertheless, llms have yet to grapple with generating dependencies, especially cross-file dependencies. as illustrated in figure 4 , llms often ignore available dependencies defined in contexts. ## conclusion and future work this paper proposes a new code generation benchmark named deveval. collected through a meticulous pipeline, deveval aligns with real-world code repositories in multiple dimensions, e.g., real code distributions, sufficient dependencies, and realscale repositories. we evaluate 8 popular llms in deveval. the results reveal the strengths and weaknesses of llms in real repositories. compared to previous benchmarks, deveval offers a more challenging and practical evaluation scenario. we hope deveval can facilitate the applications of llms in practical repositories. in the future, we will continue to update de-veval, e.g., multilingual testing samples, more projects, and more test cases. besides, we will explore how to improve the performance of llms in context-based code generation, e.g., retrievalaugmented and tool-augmented generation.