RACE : Retrieval-augmented Commit Message Generation (2022)

Abstract: Commit messages are important for software development and maintenance. Many neural network-based approaches have been proposed and shown promising results on automatic commit message generation. However, the generated commit messages could be repetitive or redundant. In this paper, we propose RACE, a new retrieval-augmented neural commit message generation method, which treats the retrieved similar commit as an exemplar and leverages it to generate an accurate commit message. As the retrieved commit message may not always accurately describe the content/intent of the current code diff, we also propose an exemplar guider, which learns the semantic similarity between the retrieved and current code diff and then guides the generation of commit message based on the similarity. We conduct extensive experiments on a large public dataset with five programming languages. Experimental results show that RACE can outperform all baselines. Furthermore, RACE can boost the performance of existing Seq2Seq models in commit message generation.

## introduction in software development and maintenance, source code is frequently changed. in practice, code changes are often documented as natural language commit messages, which summarize what (content) the code changes are or why (intent) the code is changed @xcite . high-quality commit messages are essential to help developers understand the evolution of software without diving into implementation details, which can save a large amount of § yanlin wang and hongbin sun are the corresponding authors. over the years, many approaches have been proposed to automatically generate commit messages. early studies @xcite are mainly based on predefined rules or templates, which may not cover all situations or comprehensively infer the intentions behind code changes. later, some studies @xcite @xcite adopt information retrieval (ir) techniques to reuse commit messages of similar code changes. they can take advantage of similar examples, but the reused commit messages might not correctly describe the content/intent of the current code change. recently, some seq2seqbased neural network models @xcite @xcite @xcite have been proposed to understand code diffs and generate the high-quality commit messages. these approaches show promising performance, but they tend to generate high-frequency and repetitive tokens and the generated commit messages have the problem of insufficient information and poor readability @xcite . some studies @xcite also explore the combination of neural-based and ir-based techniques. @xcite propose an approach to rank the retrieved commit message (obtained by a simple ir-based model) and the generated commit message (ob-tained by a neural network model). @xcite propose to use the similar code diff as auxiliary information in the inference stage, while the model is not trained to learn how to effectively utilize the information of retrieval results. therefore, both of them fail to take advantage of the information of retrieved similar results well. in this paper, we propose a novel model race (retrieval-augmented commit message generation), which retrieves a similar commit message as an exemplar, guides the neural model to learn the content of the code diff and the intent behind the code diff, and generates the readable and informative commit message. the key idea of our approach is retrieval and augmentation. specifically, we first train a code diff encoder to learn the semantics of code diffs and encode the code diff into highdimensional semantic space. then, we retrieve the semantically similar code diff paired with the commit message on a large parallel corpus based on the similarity measured by vectors' distance. next, we treat the similar commit message as an exemplar and leverage it to guide the neural-based models to generate an accurate commit message. however, the retrieved commit messages may not accurately describe the content/intent of current code diffs and may even contain wrong or irrelevant information. to avoid the retrieved samples dominating the processing of commit message generation, we propose an exemplar guider, which first learns the semantic similarity between the retrieved and current code diff and then leverages the information of the exemplar based on the learned similarity to guide the commit message generation. to evaluate the effectiveness of race, we conduct experiments on a large-scale dataset mcmd @xcite with five programming language (java, c#, c++, python and javascript) and compare race with 11 state-of-the-art approaches. experimental results show that: (1) race significantly outperforms existing state-ofthe-art approaches in terms of four metrics (blue, meteor, rouge-l and cider) on the commit message generation. (2) race can boost the performance of existing seq2seq models in commit message generation. for example, it can improve the performance of nmtgen @xcite , commitbert (jung, 2021), codet5small @xcite and codet5-base @xcite ) by 43%, 11%, 15%, and 16% on average in terms of bleu, respectively. in addition, we also conduct human evaluation to confirm the effectiveness of race. we summarize the main contributions of this paper as follows: ## related work code intelligence, which leverages machine learning especially deep learning-based method to understand source code, is an emerging topic and has obtained the promising results in many software engineering tasks, such as code summarization @xcite @xcite and code search @xcite @xcite . among them, commit message generation plays an important role in the software evolution. in early work, information retrieval techniques are introduced to commit message generation @xcite @xcite . for instance, changedoc @xcite retrieves the most similar commits according to the syntax or semantics in the code diff and reuses commit messages of similar code diffs. nngen @xcite is a simple yet effective retrieval-based method using the nearest neighbor algorithm. it firstly recalls the top-k similar code diffs in the parallel corpus based on cosine similarity between bag-of-words vectors of code diffs. then select the most similar result based on bleu scores between each of them (topk results) and the input code diff. these approaches can reuse similar examples and the reused commit messages are usually readable and understandable. recently, many neural-based approaches @xcite @xcite @xcite @xcite @xcite have been used to learn the semantic of code diffs and translate them into commit messages. for example, nmtgen @xcite and commit-gen @xcite treat the code diffs as plain texts and adopt the seq2seq neural network with different attention mechanisms to translate them into commit messages. codisum @xcite extracts both code structure and code semantics from code diffs and jointly models them with a multi-layer bidirectional gru to better learn the representations of code diffs. ptrgncmsg @xcite incorporates the pointer-generator network into the seq2seq model to handle out-ofvocabulary (oov) words. commitbert leverage codebert @xcite , a pre-trained language model for source code, to learn the semantic representations of code diffs and adopt a transformer-based @xcite decoder to generate the commit message. these approaches show promising results on the generation of commit messages. recently, introducing retrieved relevant results into the training process has been found useful in most generation tasks @xcite @xcite . some studies @xcite also explore the combination of neural-based models and ir-based techniques to generate commit messages. atom @xcite ensembles the neural-based model and the ir-based technique through the hybrid ranking. specifically, it uses bilstm to encode asts paths extracted from asts of code diffs and adopt a decoder to generate commit messages. it also uses tf-idf technique to represent code diffs as vectors and retrieves the most similar commit message based on cosine similarity. the generated and retrieved commit messages are finally prioritized by a hybrid ranking module. corec @xcite is also a hybrid model and only considers the retrieved result during the inference. specifically, at the training stage, they use an encoderdecoder neural model to encode the input code diffs by an encoder and generate commit messages by a decoder. at the inference stage, they first use the trained encoder to retrieve the most similar code diff from the training set. then they reuse a trained encoder-decoder to encode the input and retrieved code diff, combine the probability distributions (obtained by two decoders) of each word, and generate the final commit message step by step. in summary, atom does not learn to refine the retrieved results or the generated results, and corec is not trained to utilize the information of retrieval results. therefore, both of them fail to take full advantage of the retrieved similar results. in this paper, we treat the retrieved similar commit as an exemplar and train the model to leverage the exemplar to enhance commit message generation. ## proposed approach the overview of race is shown in figure 1 . it includes two modules: retrieval module and generation module. specifically, race firstly retrieves the most semantically similar code diff paired with the commit message from the large parallel training corpus. the semantic similarity between two code diffs is measured by the cosine similarity of vectors obtained by a code diff encoder. next, race treats the retrieved commit message as an example and uses it to guide the neural network to generate an understandable and concise commit message. ## conclusion this paper proposes a new retrieval-augmented neural commit message generation method, which treats the retrieved similar commit message as an exemplar and uses it to guide the neural network model to generate an accurate and readable commit message. extensive experimental results demonstrate that our approach outperforms recent baselines and our framework can significantly boost the performance of four neural network models. our data, source code and appendix are available at https://github.com/ deepsoftwareanalytics/race .