On Improving Repository-Level Code QA for Large Language Models (2024)

Abstract: Large Language Models (LLMs) such as ChatGPT, GitHub Copilot, Llama, or Mistral assist programmers as copilots and knowledge sources to make the coding process faster and more efficient. This paper aims to improve the copilot performance by implementing different self-alignment processes and retrieval-augmented generation (RAG) pipelines, as well as their combination. To test the effectiveness of all approaches, we create a dataset and apply a model-based evaluation, using LLM as a judge. It is designed to check the model’s abilities to understand the source code semantics, the dependency between files, and the overall meta-information about the repository. We also compare our approach with other existing solutions, e.g. ChatGPT-3.5, and evaluate on the existing benchmarks. Code and dataset are available online (https://anonymous.4open.science/r/ma_llm-382D).

## introduction coding assistants @xcite @xcite , are invaluable to any programming team for developing software applications, games, or machine learning models involves writing code using programming languages. commercial ai-assisted programming chatbot like github copilot foot_1 , codeium foot_2 or starcoder @xcite help to understand the code better, to generate some code, and to fix errors faster. however, it is important to note that coding assistants may generate incorrect information, also known as "hallucinations", when requests go beyond the model training data or require additional knowledge @xcite . another drawback of such assistants is the data protection problem: users need to be extremely careful while sharing private code and data with commercial coding assistants. sensitive or proprietary code could be exposed to unintended parties. this could potentially lead to data breaches and intellectual property concerns @xcite . moreover, most coding assistants are of general use and cannot be applied to solve context-specific issues or answer natural questions based on repository-level semantics. to mitigate these limitations, we develop two methods to improve the llms response quality on repository-level programming in a more specific, cost-effective and privacy-focused manner. one promising solution is retrieval-augmented generation (rag) @xcite , incorporating the repository-level data into the generative process, to deliver accurate and relevant responses. the second approach is inspired by @xcite and aims to increase the performance of the models by fine-tuning them with synthetic self-generated data using the self-alignment procedure. finally, we combine a rag pipeline with a fine-tuned model trained on a self-augmented dataset, which can be considered as both cost-effective and privacyfriendly approach that improves the performance of coding assistants on a specific repository. when working on the repository-level programming tasks, selecting the appropriate source is also crucial, as it should represent common repository structures and be big enough to generate training data. therefore, we consider the python spyder ide repository foot_3 at version 5.5 due to its abundance of short functions and extensive documentation. we use the open-source model mistral 7b @xcite as a base and fine-tuned model, connected to rag pipelines. mistral 7b is a pretrained llm that outperforms llama 2 7b, 13b @xcite and codellama 7b @xcite on most benchmarks. regarding the evaluation techniques, we apply the llm-as-a-judge @xcite @xcite @xcite ) method that leverages a superior model to judge other models responses. we utilize it to test whether adding information through fine-tuning or rag pipeline improves the response quality. spydercodeqa, our new evaluation dataset, is used as the test data for as llm-as-a-judge evaluation. additionally, we apply the humaneval @xcite and mbpp @xcite benchmarks to measure the catastrophic forgetting of code generation abilities after fine-tuning. the contributions of the paper are as follows: ## related work this section provides an overview of the existing studies related to the paper: repository-level programming, code-based question answering, and llm evaluation. ## dataset construction in order to measure the performance the performance of the models on repository-level programming, we create a new evaluation dataset named spydercodeqa comprising of 325 samples established on the spyder ide foot_4 . it is based on three dimensions: source code semantics comprehension (subsection 3.1), dependency comprehension (subsection 3.2), and meta-information comprehension (subsection 3.3). figure 1 presents an overview of the three dimensions of the evaluation dataset. the first one aims at understanding the containing text and code elements about the repository source code and being able to answer semantic questions about it. the second dimension evaluates the ability to understand the relationships between files within the repository and between files and imported libraries. the third dimension assesses the ability to understand general information about the repository using readme files (build commands, requirements or legal information of the repository, unrelated to the source code). the following subsections provide an overview of the creation process for each dimension in detail. typical samples for each dimension are shown in appendix a in figure 8 . ## methodology this section describes the methods we implement in the paper. first, we describe the data preprocessing step (subsection 4.1), which is common for all approaches. then we explain the self-alignment approach in subsection 4.2 and our implementation of rag in subsection 4.3. subsection 4.4 explains how both approaches can be combined. ## evaluation this section describes two evaluation strategies applied in the paper: using llms (primarily gpt 3.5/4) as judges @xcite and standard benchmark evaluation using metrics. llm-as-ajudge methods are preferred over bleu @xcite and rouge @xcite , as they can only evaluate the semantic similarity between human and model-generated responses, which might not be related to the correctness of the responses. ## conclusion in this paper we introduce a new manually created dataset -spydercodeqa -which includes 325 question-and-answer pairs (q&a pairs) from the spyder ide repository. we split it into three question dimensions: semantics understanding, dependency understanding, and knowledge of repository meta-information. we also present a series of experiments using self-alignment, rag, and their combination to evaluate llms' performance on repository-level code q&a using the generated dataset. we show that the quality of the system can be significantly improved when applying both approaches together: the llm-as-a-judge win rate is approximately 64%, which is 7% higher than both approaches separately. regarding the models' performance on different dataset dimensions, we can see that they perform exceptionally well for code semantics, which is the human-labeled dimension. in future work, we plan to improve the selfalignment pipeline to create a more diverse dataset that includes q&a pairs mainly focused on code generation to prevent the "catastrophic forgetting" of the model. another possible direction is to perform the human evaluation to better align the model with user needs. it would provide additional insights since humans are the target audience for q&a on repository-level programming, and they often have more knowledge about the repository, allowing them to better judge the model's responses.