Retrieval Augmented Code Generation and Summarization (2021)

Abstract: Software developers write a lot of source code and documentation during software development. Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implementing software or documenting them. To mimic developers’ code or summary generation behavior, we propose a retrieval augmented framework, REDCODER, that retrieves relevant code or summaries from a retrieval database and provides them as a supplement to code generation or summarization models. REDCODER has a couple of uniqueness. First, it extends the state-of-the-art dense retrieval technique to search for relevant code or summaries. Second, it can work with retrieval databases that include unimodal (only code or natural language description) or bimodal instances (code-description pairs). We conduct experiments and extensive analysis on two benchmark datasets of code generation and summarization in Java and Python, and the promising results endorse the effectiveness of our proposed retrieval augmented framework.

## introduction in recent years, automating source code generation and summarization is receiving significant attention due to its potential in increasing programmers' productivity and reducing developers' tedious workload. consequently, various approaches have been explored in the literature to facilitate code generation @xcite and code documentation/summarization @xcite @xcite . despite initial success, most of the generated code still suffers from poor code quality @xcite . therefore, the question remains-how to generate better code from a given summary and vice versa. source code generation and summarization, however, are intrinsically complex and challenging. they involve generating diverse token sequences such as different variables, operators, keywords, classes, and method names @xcite , which requires understanding the programming languages at lexical, syntax, and semantics levels. to combat these issues, recent studies (e.g., @xcite ; @xcite ; @xcite ; @xcite ; xu et al. ( 2020 )) take a learning-based approach-they train representations of code and the associated text by leveraging existing high-quality source code and short text descriptions available in open-source repositories and question answering forums such as github and stack overflow. then fine-tune the representation models on the downstream tasks. although these dataset contains high-quality human-written code and text, since the existing approaches do not directly leverage them during the generation process, the gain achieved by these approaches is still limited, especially when the source code is long. to overcome this, we take advantage of the existing high-quality source code and their description by including them directly in the generation process that are retrieved via information retrieval technique. in this work, we present redcoder, a retrieval augmented code generation and summarization framework. while designing red-coder, we take motivation from how developers take advantage of existing resources. for example, developers often search for relevant code in the code repository, and if found, adapt the retrieved code in their own context. similarly, when an api usage is unclear, they search in question answering forums (e.g., stackoverflow) @xcite . such an additional resource helps developers to increase their development productivity @xcite . we design redcoder as a two-step process (see figure 1 ). in the first step, given the input (nl text for code generation, or code snippet for summarization) a retriever module retrieves relevant source code (for code generation) or summaries (for code summarization) from a database. 1 in the second step, a generator processes the retrieved code/summary along with the original input to generate the target output. in this way, redcoder enhances the generation capability by augmenting the input through retrieval. the two-step process allows us to design a modular and configurable framework for source code and summary generation. various designs of retriever and generator models can be incorporated into this framework. existing cross-encoder code retrievers being computationally expensive, their applicability to retrieve from a large database is limited @xcite . a natural choice would be to use sparse term based retrievers such as tf-idf or bm25 @xcite . however, the retriever module in redcoder should exhibit a good understanding of source code and programmers' natural language, which is a non-trivial task due to the syntactic and semantic structure of the source code @xcite . such an expectation of searching for semantically similar code and summary may not be attainable by a sparse token level code retriever (e.g., bm25). to that end, we design the retriever module in redcoder based on programming languages (pl) and natural languages (nl) understanding models (e.g., graphcodebert @xcite ). this retriever module extends the state-ofthe-art dense retrieval technique @xcite using two different encoders for encoding the query and document. as for the generator, redcoder can handle retrieval databases consisting of both unimodal (only code or natural language description) and bimodal instances (code-description pairs) and makes the best usage of all the auxiliary information that 1 the database could be open source repositories (e.g., github) or developers' forums (e.g., stack overflow). are available. yet, to incorporate information, we augment the retrieved information only in the input level. it does not modify the underlying architecture of the generator module -preserving its model agnostic characteristics. we evaluate the effectiveness of redcoder on two popular programming languages (java and python) on both code generation and code summarization tasks. the empirical results show that, redcoder's concept of retrieval augmented generation elevates the state-of-the-art code generation from an exact match score of 18.6 to 23.4 and the summary generation bleu-4 score from 18.45 to 22.95 even when we forcefully remove the target candidate from the retrieved code or summary. with further experiments, we establish the importance of both the retrieved code and retrieves summary in the generation process. the source code for reproducing our experiments are at https: //github.com/rizwan09/redcoder . ## background we first introduce the problem formulation and discuss the fundamentals of the retriever and generator components that redcoder is built upon. ## proposed framework: redcoder our proposed code generation and summarization framework, redcoder generates the target code or summary by augmenting the input x with relevant code snippets or summaries. we build our retriever module by training a dpr model differently from @xcite . with an intelligent scheme, we then augment the retrieved candidates and their pairs (if available) to provide auxiliary supervision to the generator. we briefly describe the model components in this section. ## experiment setup in order to investigate the effectiveness of our framework, we perform a comprehensive study and analysis on code generation and summarization in two programming languages, java and python. ## analysis in this section, we analyze redcoder's performance on the following points. ## conclusion we propose redcoder to automate developers' writing of code and documentation by reusing what they have written previously. we evaluate red-coder on two benchmark datasets and the results demonstrate a significant performance boost with the help of the retrieved information. in the future, we want to extend redcoder to support other code automation tasks such as code translation.