C ommit BERT : Commit Message Generation Using Pre-Trained Programming Language Model (2021)

Abstract: Commit message is a document that summarizes source code changes in natural language. A good commit message clearly shows the source code changes, so this enhances collaboration between developers. Therefore, our work is to develop a model that automatically writes the commit message. To this end, we release 345K datasets consisting of code modification and commit messages in six programming languages (Python, PHP, Go, Java, JavaScript, and Ruby). Similar to the neural machine translation (NMT) model, using our dataset, we feed the code modification to the encoder input and the commit message to the decoder input and measure the result of the generated commit message with BLEU-4. Also, we propose the following two training methods to improve the result of generating the commit message: (1) A method of preprocessing the input to feed the code modification to the encoder input. (2) A method that uses an initial weight suitable for the code domain to reduce the gap in contextual representation between programming language (PL) and natural language (NL).

## introduction commit message is the smallest unit that summarizes source code changes in natural language. figure 1 shows the git diff representing code modification and the corresponding commit message. a message : fix deprecated ref to tokenizer.max len good commit message allows developers to visualize the commit history at a glance, so many teams try to do high quality commits by creating rules for commit messages. for example, conventional commits foot_0 is one of the commit rules to use a verb of a specified type for the first word like 'add' or 'fix' and limit the length of the character. it is very tricky to follow all these rules and write a good quality commit message, so many developers ignore it due to lack of time and motivation. so it would be very efficient if the commit message is automatically written when a code modification is given. similar to text summarization, many studies have been conducted by taking code modification @xmath0, ..., x n ) as encoder input and commit message @xmath1, ..., y m ) as decoder input based on the nmt (neural machine translation) model. @xcite @xcite however, taking the code modification without distinguishing between the added and the deleted part as model input makes it difficult to understand the context of modification in the nmt model. in addition, previous studies tend to train from scratch when training a model, but this method does not show good performance because it creates a large gap in the contextual representation between programming language (pl) and natural language (nl). to overcome the problems in previous studies and train a better commit message generation model, our approach follows two stages: ## related work commit message generation has been studied in various ways. @xcite collect 2m commits from the @xcite and top 1k java projects in github. among the commit messages, only those that keep the format of "verb + object" are filtered, grouped into verb types with similar characteristics, and then the classification model is trained with the naive bayes classifier. @xcite use the commit data collected by @xcite to generate the commit message using an attention-based rnn encoder-decoder nmt model. they filter again in a "verb/direct-object pattern" from 2m data and finally used the 26k commit message data. @xcite uses an nmt model similar to @xcite , but uses git diff and commit pairs collected from 1∼3 repositories of python, java, javascript, and c++ as training data. @xcite propose a retrieval model using @xcite 's 26k commit as training data. code modification is represented by bags of words vector, and the message with the highest cosine similarity is retrieved. @xcite collect only '.java' file format from @xcite and use 509k dataset as training data for nmt. also, to mitigate the problem of out-of-vocabulary (oov) of code domain input, they use generation distribution or copying distribution similar to pointer-generator networks @xcite . van hal et al. (2019) also argues that the @xcite entire data is noise and proposes a pre-processing method that filters the better commit messages. @xcite argue that it is challenging to represent the information required for source code input in the nmt model with a fixed-length. in order to alleviate this, it is suggested that only the added and deleted parts of the code modification be abbreviated as abstract syntax tree (ast) and applied to the bi-lstm model. nieb et al. presented a large gap between the contextual representation between the source code and the natural language when generating commit messages. previous studies have used rnn or lstm model, they use the transformer model, and similarly to other studies, they use @xcite as the training data. to reduce this gap, they try to reduce the two-loss that predict the next code line (explicit code changes) and the randomly masked word in the binary file. ## dataset we collect a 345k code modification dataset and commit message pairs from 52k repositories of six programming languages (python, php, go, java, javascript, and ruby) on github. when using raw git diff as model input, it is difficult to distinguish between added and deleted parts, so unlike @xcite , our dataset focuses only on the added and deleted lines in git diff. the detailed data collection and pre-processing method are shown as a pseudo-code in algorithm 1: to collect only the code that is a re-distributable license, we have listed the github repository name in the codesearchnet dataset. after that, all the repositories are cloned through multi-threading. detailed descriptions of functions that collect the commit hashes in a repository and the code modifi-algorithm 1 code modification parser from the list of repositories. figure 2 : commit message verb type and frequency statistics. only 'upgrade' is not included in the high frequency, but is included in a similar way to 'update'. this refers to the verb group in @xcite . cations in a commit hash are as follows: and the corresponding commit message is obscure and very abstract. also, we check that some code modification or commit message is a meaningless dummy file. to filter these, we create the filtering function and the rules as follows. as a result, we collect 345k code modification and commit message pair datasets from 52k github repositories and split commit data into 80-10-10 train/validation/test sets. this results are shown in ## commitbert we propose the idea of generating a commit message through the codebert model with the our dataset ( § 4). to this end, this section describes how to feed inputs code modification (@xmath2, del 1 ), ..., (add n , del n ))) and commit message (@xmath3, ..., msg n )) to codebert and how to use pre-trained weights more efficiently to reduce the gap in contextual representation between programming language (pl) and natural language (nl). ## experiment to verify the proposal in section 5 in the commit message generation task, we do two experiments. ## conclusion and future work our work presented a model summarizing code modifications to solve the difficulty of humans manually writing commit messages. to this end, this paper proposed a method of collecting data, a method of taking it to a model, and a method of improving performance. as a result, it showed a successful result in generating a commit message using our proposed methods. consequently, our work can help developers who have difficulty writing commit messages even with the application. although it is possible to generate a high-quality commit message with a pre-trained model, future studies to understand the code syntax structure remain in our work. as a solution to this, com-mitbert should be converted to ast (abstract