C ode A gent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges (2024)

Abstract: Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units. However, real-world software development often involves complex code repositories with complex dependencies and extensive documentation. To enable LLMs to handle these realworld repo-level code generation, we present CodeAgent, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. CodeAgent integrates five programming tools, enabling interaction with software artifacts for information retrieval, code implementation, and code testing. We implement four agent strategies to optimize these tools’ usage. To the best of our knowledge, CodeAgent is the first agent tool framework specifically for repo-level code generation. In order to measure the effectiveness of our method at the repository level, we have introduced a benchmark dataset CodAgentBench. The performance on this dataset shows a significant improvement brought by our method, with improvements of pass rate ranging from 2.0 to 15.8. Further tests on the HumanEval benchmark confirm CodeAgent’s adaptability and efficacy across various code generation tasks. Notably, CodeAgent outperforms commercial products like Github Copilot, showcasing superior accuracy and efficiency. These results demonstrate CodeAgent’s robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.

## repo-level code generation task to fill the gap between existing code generation tasks and practical coding scenarios, we formalize the repo-level code generation task. since a code repository generally contains intricate invocation relationships, only with a deep understanding of the code repository can llms generate satisfying programs that not only adhere to requirements but also seamlessly integrate with the current repository. given a code repository, the repo-level code generation task aims to generate code based on all the software artifacts included in the repository, encompassing the documentation, code dependency, runtime environment, which form the task input. here we give a detailed description of its composition format. figure 1 shows an illustration of the repo-level code generation task. documentation it describes the generation targets and is the main input component of repo-level code generation. the documentation provides additional supporting information beyond the nl requirements. it contains class-level (class name, signature, and member function) and function-level (functional description, and params description) information of targets. typically, the correctness of generated programs is verified with the test suite. the generated programs must conform to the inter-face (e.g., the input parameters). thus, the documentation also provides the type and interpretation of input parameters and output values. in addition, considering that requirements usually contain domain-specific terminologies, the documentation explains these terms as well, such as mathematical theorems. as shown in figure 1 , documentation of the project contains rich information, where different elements are highlighted with diverse colors. ## codeagent method we introduce a novel llm-based agent framework codeagent that leverages external tools to enhance the problem-solving abilities of llms in intricate repo-level code generation. codeagent seamlessly pauses generation whenever tools are called and resumes generation by integrating their outputs. these tools can assist llms with the entire code generation process, including information retrieval, code implementation, and code testing as shown in ## experiment we perform extensive experiments to answer three research questions: (1) how much can codeagent improve the advanced code generation llms on repo-level code generation (section 5.2); (2) what is the improvement of our codeagent on classical code generation such as humaneval (section 5.3); ( 3 ) to what extent do our selected tools in the agent system help for repo-level coding (section 5.4). ## conclusion we formalize the repo-level code generation task to evolve real-world coding challenges. to enhance llms to handle repo-level code generation, we propose codeagent, a novel llm-based agent framework. codeagent develops five programming tools, enabling llms to interact with software artifacts, and designs four agent strategies to optimize tools' usage. to evaluate the effectiveness of our codeagent, we construct codeagent-bench, a new benchmark for repo-level code generation that includes rich information about the code repository. experiments on nine llms show that codeagent achieves a significant improvement on diverse programming tasks, highlighting its potential in real-world coding challenges.