Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback (2024)

Abstract: Large Language Models (LLMs) have shown remarkable progress in automated code generation. Yet, LLM-generated code may contain errors in API usage, class, data structure, or missing project-specific information. As much of this project-specific context cannot fit into the prompts of LLMs, we must find ways to allow the model to explore the project-level code context. We present CoCoGen, a new code generation approach that uses compiler feedback to improve the LLM-generated code. CoCoGen first leverages static analysis to identify mismatches between the generated code and the project’s context. It then iteratively aligns and fixes the identified errors using information extracted from the code repository. We integrate CoCoGen with two representative LLMs, i.e., GPT-3.5-Turbo and Code Llama (13B), and apply it to Python code generation. Experimental results show that CoCoGen significantly improves the vanilla LLMs by over 80% in generating code dependent on the project context and consistently outperforms the existing retrieval-based code generation baselines.

## introduction large language models (llms), especially those pre-trained on code, as demonstrated by tools such as github copilot @xcite , amazon's codewhisperer @xcite , and chat-gpt @xcite , are revolutionizing how developers approach programming by automatically generating code for given contexts (e.g., naturallanguage descriptions or surrounding incomplete code). while existing llm-based code generation tools excel in code generation within a small-scale and isolated context (i.e., a single file), integrating * also with national engineering research center for big data technology and system, services computing technology and system lab, cluster and grid computing lab, school of computer science and technology, huazhong university of science and technology, wuhan, 430074, china. enhance the image by first reducing noise and then adjusting brightness and contrast. llm-based code generation into real-world software projects remains challenging @xcite . practical code generation for software repositories is often associated with broader project-level contexts declared in other repository files due to the demands of modularity, structure, and comprehension in software management @xcite . solely providing the task requirements and surrounding incomplete code can lead llms to overlook the complex hierarchies of apis, classes, data structures, or type constraints specific to a software project's repository, risking omitting essential logic during code generation @xcite . as a motivating example, consider the case shown in figure 1 . in this case, we use the ope-nai gpt-3.5-turbo api to generate an imageenhancing function by first reducing the notice and then adjusting brightness and contrast. given the prompt in figure 1a , the llm will produce a code snippet of calling a reduce_noise method as depicted in figure 1b . although the generated code follows a standard workflow specified in the task prompt, it leads to a compilation error in our application context because reduce_noise is not implemented. this is unsurprising as the prompt does not provide enough project-level context for the llm, and can be fixed by providing the projectspecific function denoising.denoise as context, as illustrated in figure 1c . while a carefully engineered prompt may resolve this issue, it is not always possible for the user to generate such prompts, and the project context may be too large to fit into the prompt. as we will empirically show later in (section 2.2), such errors are commonly found when applying llms to repository-level code generation @xcite . this paper investigates a way to effectively integrate llm-based code generation with existing code implementations within a software project. our solution is to leverage project-level contextual information, such as project-specific implementation of classes, methods, and data structures, to reduce compilation errors and improve code quality. directly incorporating the entire project code into a language model is infeasible due to model input sequence length limitations. instead, we use compiler-based analysis to post-process the modelgenerated code by first detecting discrepancies between the generated code and the project's context. we then utilize information extracted from the project code base to rectify the mismatches in using modules, apis, and classes. our approach combines well-established compiler techniques with emerging generative methods, allowing software developers to leverage the power of llms without being overwhelmed and discouraged by the frequent compilation and semantic errors in the model-generated code. we present cocogen, a method to allow an llm to leverage the code repository of a software project to enhance the quality of the generated code. for a given llm-generated code sample, coco-gen first compiles it and identifies context-related errors. it then retrieves the related context from the code repository to fix the errors. this iterative generation and verification process proceeds repetitively until no error is identified in the generated solution. we demonstrate that cocogen enhances the accuracy of generation (as indicated by pass rates) by bridging the gap between the repository context and the intended solution. we evaluate cocogen by applying it to the codereval benchmarking dataset @xcite , which consists of code generation tasks utilizing project-specific context. we test cocogen on two popular code generation models: the gpt-3.5-turbo (openai, 2023) and code llama @xcite . experimental results demonstrate that cocogen significantly improves the repositorylevel code generation performance of different dependency levels, outperforming the baseline by over 80% relative pass rates in generating functions dependent on project-specific contexts. moreover, our iterative method consistently enhances the performance of vanilla retrieval-augmented generation. we also provide a comprehensive analysis of the effectiveness and limitations of cocogen, offering insights for future research. this paper makes the following contributions: ## conclusion in this paper, we show how to use project-specific context information, indexed structural and semantic, to fix compilation errors generated by compilers and improve the quality of code generated by llms. our experimental results show the increased prevalence of errors related to project contexts in project-level code generation compared to function-level code generation. the presented cocogen can effectively fix the compilation errors by retrieving related context from the project, thus significantly improving the native llm baselines on over 80% relative pass rates in generating functions dependent on project-specific contexts.