E vo R : Evolving Retrieval for Code Generation (2024)

Abstract: Recently the retrieval-augmented generation (RAG) has been successfully applied in code generation. However, existing pipelines for retrieval-augmented code generation (RACG) employ static knowledge bases with a single source, limiting the adaptation capabilities of Large Language Models (LLMs) to domains they have insufficient knowledge of. In this work, we develop a novel pipeline, EVOR, that employs the synchronous evolution of both queries and diverse knowledge bases. On two realistic settings where the external knowledge is required to solve code generation tasks, we compile four new datasets associated with frequently updated libraries and long-tail programming languages, named EVOR-BENCH. Extensive experiments demonstrate that EVOR achieves two to four times of execution accuracy compared to other methods such as Reflexion (Shinn et al., 2024), DocPrompting (Zhou et al., 2023), etc. We demonstrate that EVOR is flexible and can be easily combined with them to achieve further improvement. Further analysis reveals that EVOR benefits from the synchronous evolution of queries and documents and the diverse information sources in the knowledge base. We hope that our studies will inspire more insights into the design of advanced RACG pipelines in future research.

## introduction the retrieval-augmented generation (rag) paradigm has raised significant attention due to its efficiency in adapting large language models (llms) without training @xcite @xcite @xcite . recent research has demonstrated its successful applications in code generation. they implement the retrieval-augmented code generation (racg) pipelines either using a given query @xcite , or a rewritten version @xcite write ponyc code to print odd numbers from 0 to n (inclusive) ## evolving retrieval given a question n in natural language, the objective of retrieval-augmented code generation is to first retrieve relevant information k + from external knowledge k and then augment large language models to generate a program p in the target library/programming language, which llm m is not familiar with. distinct from the classical retrieval-augmented generation, which usually focuses on static knowledge bases, we propose synchronous evolution of both queries and diverse knowledge bases. intuitively, this helps the retrieval model identify more relevant information and thus improves the quality of llm generation @xcite . in this section, we present the process of query evolution ( §2.1), the knowledge base construction and evolution ( §2.2), the evor pipeline ( §2.3) and the collection of evor-bench ( §2.4). ## experiment docprompting: @xcite proposed to explicitly leverage code documentation by first retrieving the relevant documentation pieces given a natural language (nl) intent, and then generating code based on the nl intent and the retrieved documentation. it can be viewed as a degraded version of evor where neither queries nor knowledge bases evolve and the retrieval pool encompasses the documentation as a single source. ## related works since the focus of our work is to enhance code generation with retrieval, our work is closely related to code generation and retrieval-augmented code generation. additionally, we are connected to the line of code execution since we also leverage it as an important retrieval source. ## conclusion much recent work illustrated the ability of llms to incorporate external knowledge with retrievalaugmented generation. we propose a novel pipeline, evor, which achieves two to four times execution accuracy compared to existing code generation methods. extensive experiments demonstrate that evor can be easily combined with them to provide further improvements, including the agent-based ones to solve challenging tasks such as repo-level code generation. through an indepth analysis, we further show the complementary strength of synchronous evolution of queries and documents in racg, which enhances the model performance by larger margins with more diverse knowledge sources. we hope that our findings will inspire researchers and practitioners to develop efficient and effective strategies in their customized code-generation tasks with llms.