R epo C oder: Repository-Level Code Completion Through Iterative Retrieval and Generation (2023)

Abstract: The task of repository-level code completion is to continue writing the unfinished code based on a broader context of the repository. While for automated code completion tools, it is difficult to utilize the useful information scattered in different files. We propose RepoCoder, a simple, generic, and effective framework to address the challenge. It streamlines the repository-level code completion process by incorporating a similarity-based retriever and a pre-trained code language model in an iterative retrieval-generation pipeline. RepoCoder makes effective utilization of repository-level information for code completion and has the ability to generate code at various levels of granularity. Moreover, we propose a new benchmark RepoBench, which consists of the latest and high-quality real-world repositories covering line, API invocation, and function body completion scenarios. Experimental results indicate that RepoCoder significantly improves the In-File completion baseline by over 10% in all settings and consistently outperforms the vanilla retrieval-augmented code completion approach. Furthermore, we validate the effectiveness of RepoCoder through comprehensive analysis, providing valuable insights for future research. Our source code and benchmark will be publicly available after the paper review.

## introduction in real-world software production, it is crucial for developers to be aware of other files within the repository during programming. this challenge gives rise to the task of repository-level code completion, where automated tools are expected to utilize the broader context of a repository rather than relying solely on in-file information to complete unfinished code. code files within a repository often exhibit interrelated dependencies, including shared utilities, configurations, and cross-api invocations resulting from modularization (tu et al., 2014). additionally, each repository typically follows customized naming conventions and coding styles @xcite , which contribute to enhanced readability and maintainability. however, developing effective repository-level code completion tools remains an open problem. although approaches relying on static code analysis and heuristic rules @xcite @xcite can reliably parse specific repository context, they have limitations in the completion scenario, limiting capability for varying-length completions anywhere in a file. meanwhile, studies @xcite @xcite tuning language models on labeled data excel in their respective evaluation scenarios but face challenges generalizing to unseen repositories without retraining. in this paper, we propose an approach to leverage off-the-shelf retrievers in order to locate valuable information within a repository and enhance the context for language models. we introduce a novel framework called repocoder that aims to improve code retrieval and completion performance. as depicted in figure 1 , we enhance the conventional in-file code completion method by incorporating the retrieval-augmented generation (rag) technique, which allows us to search for relevant code snippets from the repository to assist in generating the code completion. additionally, we introduce repocoder, which employs an iterative pipeline that utilizes the generated code completion to enhance the retrieval process, thus bridging the gap between the retrieval context and the intended completion target. figure 2 provides an example that illustrates the rationale behind our design. we demonstrate that relying solely on the unfinished code is insufficient to retrieve useful information from the repository. in the example, the model improvises a statement calling the colmap api in the first iteration. the predicted parameters are reasonable yet incorrect. this is because the incomplete code preceding the code completion does not serve as an adequate retrieval query for the intended completion target. however, by performing a subsequent retrieval from the repository using the model's generated completion, we can successfully retrieve the target api signature and complete the code effectively. furthermore, we introduce the repoeval benchmark designed for evaluating the repository-level code completion task, which is constructed using the latest high-quality repositories sourced from github foot_0 . by introducing repoeval, we address the lack of established benchmarks in the repositorylevel scenario. notably, repoeval is the first benchmark that encompasses three levels of code completion granularity: line, api invocation, and function body. we also leverage unit tests present in the repository to enhance the accuracy of evaluation, which overcomes the limitations of similaritybased metrics. to rigorously validate the effectiveness of repocoder, we conduct extensive experiments using different language models of varying sizes, including gpt-3.5-turbo foot_1 and code-gen @xcite . experimental results demonstrate that repocoder achieves significant improvements over in-file completion performance, surpassing the baseline by over 10% across different experimental settings. moreover, our iterative framework consistently enhances the performance of vanilla retrieval-augmented generation. we also provide a comprehensive analysis of the effectiveness and limitations of repocoder, offering insights for future research. our contributions can be summarized as follows: • we propose repocoder, a novel iterative retrieval-generation framework for the repository-level code completion task. ## benchmark construction to facilitate the evaluation of code completion tools in the repository-level scenario, we propose a novel repoeval benchmark. this benchmark is carefully constructed using the latest high-quality repositories sourced from github and encompasses three levels of code completion granularity: line, api invocation, and function body. to assess the correctness of completed functions, we utilize unit tests present in the repository instead of relying solely on similarity-based metrics. each sample in the repoeval benchmark is annotated with the corre-sponding source repository, file path, line numbers, and ground truth completion. for analysis and unit test execution, complete copies of the repositories are archived as of january 2023. to construct repoeval, we first meticulously curate a collection of python repositories from github that satisfy the following criteria: opensource license, created after january 1, 2022 foot_2 , nonfork original repositories, over 100 stars, over 80% of files written in python, and explicit unit tests. furthermore, to mitigate potential biases, we employ a random selection process for the repositories and create three distinct datasets for line completion, api invocation completion, and function body completion. additional details regarding the selected repositories can be found in line completion: in adherence to the conventions of code completion benchmarks @xcite , we implement the line completion scenario. first, according to the above-mentioned criteria, we select 8 repositories that vary in size and cover different domains. then we randomly select 200 lines to complete from each repository, ensuring the lines are non-repetitive, not code comments, and each line contains at least 5 tokens. eventually, a total of 1600 test samples are generated for the line completion dataset. ## analysis in this section, we conduct further analyses on the retrieved code snippets to gain a deeper understanding of repocoder and provide valuable insights for future research. ## conclusion and future work in conclusion, we introduce repocoder, a straightforward and effective framework for the repositorylevel code completion task. by leveraging a retriever and a language model, repocoder effectively utilizes repository-level information. through an iterative process of retrieval and generation, repocoder bridges the gap between retrieval context and the target code, resulting in improved code completion performance. our extensive experiments conducted on the repoeval benchmark demonstrate that repocoder consistently and significantly enhances in-file completion performance, surpassing the vanilla retrieval-augmented generation (rag) approach. furthermore, our analysis provides valuable insights into the rationale and limitations of repocoder. with its simplicity, versatility, and effectiveness, repocoder has the potential to become an indispensable tool in real-world software development, and we aim to further enhance its usability and robustness.