P roject E val: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation (2025)

Abstract: Recently, LLM agents have made rapid progress in improving their programming capabilities. However, existing benchmarks lack the ability to automatically evaluate from users’ perspective, and also lack the explainability of the results of LLM agents’ code generation capabilities. Thus, we introduce ProjectEval, a new benchmark for LLM agents project-level code generation’s automated evaluation by simulating user interaction. ProjectEval is constructed by LLM with human reviewing. It has three different level inputs of natural languages or code skeletons. ProjectEval can evaluate the generated projects by user interaction simulation for execution, and by code similarity through existing objective indicators. Through ProjectEval, we find that systematic engineering project code, overall understanding of the project and comprehensive analysis capability are the keys for LLM agents to achieve practical projects. Our findings and benchmark provide valuable insights for developing more effective programming agents that can be deployed in future real-world production.

## introduction the field of programming has seen significant advances with the rise of large language models (llms), today the llm agents can do many programming works without the help of humans @xcite . to evaluate the programming ability of gpt-2, @xcite raised the first programming benchmark named humaneval in 2021. humaneval was also the first of the humanevalbased benchmarks. in the last three years, more than 20 benchmarks had been raised manually or automatically based on llm. after humaneval, mbpp @xcite came out which also was a base for many benchmarks. mbpp concentrating on algorithm realization. in 2023, ds-1000 @xcite was raised and represented a new series of benchmarks, evaluating llm agents programming abilities of accessing the third-party libraries or packages. in 2024, llm-based programming agents developed rapidly, researchers noticed that many of them can do project-level programming @xcite . therefore, project-level benchmarks came out, they were softwaredev @xcite , projectdev @xcite , srdd @xcite , cassd @xcite , and devbench @xcite . these benchmarks provided the methods evaluated agent's code generation capabilities at the granularity level project. however, only devbench achieved automated evaluation using in-project test units, and the test-unit-evaluation was not a realistic method in a production environment used by human users. and, the other benchmarks still relied on natural language tests, which usually required human judgment for correctness and efficiency. moreover, only devbench provided a table of evaluation scores, but it lacked explainability. to bridge this gap, we propose projecteval, a novel benchmark tailored for automated evaluating project-level programming tasks(missions) in this field. projecteval is designed to assess the ability of agents to tackle complex user-driven tasks with precision and adaptability. unlike existing benchmarks, projecteval emphasizes real-world usability by integrating automated test suites of user interaction simulation, parameter analysis, and canonical solutions into a cohesive evaluation pipeline. projecteval contains 20 real-world tasks with totally 284 testcases, and supports two task types: website-based projects and batch/consolebased programs, with theoretical scalability to even more complex, custom ui-based tasks. by leveraging llms for generation, supplemented with manual reviewing, projecteval ensures robust and detailed evaluation metrics. we introduce the core structure and evaluation process of projecteval, highlight the challenges faced by state-of-the-art models such as gemma-2 and gpt-4o, and demonstrate the benchmark's ability to test comprehensive capabilities through pass rate (pass@k) and existing algorithms. our goal is to establish a more effectively, and more explainable new standard for automated evaluating project code quality. we summarize the contributions of this work as follow: ## conclusion we develop a new benchmark projecteval. it fills the gap for the lack of benchmark in the project granularity level code generation field of natural language processing and provides automated evaluation tools for higher-level research in llm agent. we also leverage additional objective metrics to reveal the effectiveness of llm agents at different stages of project generation. these metrics are crucial in revealing the capabilities for improvement in the agents' performance, thus offering a deeper understanding of how these models can be further enhanced. we confirmed that gpt-4o is still the sota in this field. our findings and benchmark provide valuable insights for developing more effective programming agents that can be deployed in real-world production environments.