Co-training for Commit Classification (2021)

Abstract: Commits in version control systems (e.g. Git) track changes in a software project. Commits comprise noisy user-generated natural language and code patches. Automatic commit classification (CC) has been used to determine the type of code maintenance activities performed, as well as to detect bug fixes in code repositories. Much prior work occurs in the fully-supervised setting – a setting that can be a stretch in resource-scarce situations presenting difficulties in labeling commits. In this paper, we apply co-training, a semi-supervised learning method, to take advantage of the two views available – the commit message (natural language) and the code changes (programming language) – to improve commit classification.

## introduction as a society's reliance on software grows, so will its needs for software analysis. commits in version control systems (e.g. git), a way of tracking changes made to code repositories, have been used in a range of analyses, including understanding maintenance activities @xcite @xcite , studying the impact of social factors in developer teams @xcite , and detecting bug-fixing patches @xcite @xcite . despite the usefulness of analyzing commits, there has been a lack of large-scale commit datasets. in 2017, to our knowledge, the largest public commit dataset @xcite contained 1,151 commits categorized into three classes. for perspective, open source projects could see to tens of thousands of commits per year. for example, the linux kernel's repository had 82,300 commits in 2019 @xcite and now comprises more than 1 million commits. the rate at which commits are created coupled with the variety of the kinds of commit message (nl view): improve numerical stability of layernorm ↩→ (#59987) summary: pull request resolved: #59987 similar as groupnorm, improve numerical ↩→ stability of layernorm by welford ↩→ algorithm and pairwise sum. test plan: buck test mode/dev-nosan //caffe2/ ↩→ test:nn --"layernorm" reviewed by: ngimel differential revision: d29115235 fbshipit-source-id: 376 ↩→ dac89a4e14bd340aaaf169fef8d0d4ca4a1c4 code change (pl view, cropped with ellipsis) diff --git a/aten/src/aten/native/cpu/ ↩→ layer_norm_kernel.cpp b/aten/src/aten/ ↩→ native/cpu/layer_norm_kernel.cpp index 95a35571646d..366afe64b72a 100644 ---a/aten/src/aten/native/cpu/ ↩→ layer_norm_kernel.cpp +++ b/aten/src/aten/native/cpu/ a commit consists of a commit message in natural language (nl) and code changes in programming languages (pl) (see figure 1 ). assuming weak dependence between the two views (nl and pl), a known sufficient condition for co-training to succeed @xcite , we apply iterative cotraining @xcite to train classifiers by bootstrapping from freely available unlabeled commit samples. empirically, we show that co-training improves commit classification (cc) over a range of data scarcity scenarios. with large transformer models achieving stateof-the-art performance in nlp tasks, @xcite created codebert, a pretrained bimodal model based on roberta's @xcite architecture (another large language model) for natural language and code to capture the semantic connection between both modalities via an attention mechanism to output representations that can support nl-pl understanding tasks. due to the presence of nl and pl views in commits, we propose finetuning a pretrained codebert for transfer learning on the cc task. in this paper we examine the following questions: rq1: can attention between nl and pl improve classification? rq2: is codebert better than roberta at representing commit messages for cc? rq3: can co-training improve cc? while rq3 is our main contribution, we study rq1 and rq2 to decide on the neural network architecture. with large language models achieving stateof-the-art performance in nlp tasks, @xcite created codebert, a pre-trained bimodal model based on roberta's @xcite architecture (another large language model) for natural language and code in an attempt to capture the semantic connection between both modalities to output representations that can support nl-pl understanding tasks. codebert is pre-trained on code-documentation pairs from github repositories with the masked language modeling objective (used in pre-training bert), as well as the replaced token detection objective (used in pre-training electra @xcite ). codebert is bi-modal in the sense that natural language and code can be concatenated into a single sequence and fed into the model to give rise to attention between code and natural language tokens. in this paper, we adopt a fine-tuning approach to adapt roberta and code-bert models for cc. datasets @xcite combined three datasets @xcite @xcite containing commits from open-source projects that cover a range of domains such as databases, programming languages and integration frameworks to create a dataset with 1,793 labeled commits for classifying types of software maintenance activities. another data set was compiled by @xcite , providing 1,282 commits that fix vulnerabilities in 205 open-source java projects for classifying security commits. to our knowledge, the largest data set (that we denote as ra-data) was compiled by @xcite , aggregating three datasets @xcite @xcite as well as additional scraping to provide 8,057 security-relevant commits and about 110,000 nonsecurity-relevant commits spanning 1,339 opensource projects and 20 programming languages. in this paper, we use ra-data as it is larger and more varied than the other datasets. to our knowledge, our work is the first to attempt cc on this dataset. co-training co-training was first proposed by @xcite and applied to classify university web pages by assuming that the web pages' text and inbound links are two distinct views conditionally independent to the other given the class label, and sufficient for learning by itself. during each co-training iteration, the classifiers' most confident predictions for each view are added to an initially small set of labeled training data, expanding it. in the same iteration, the classifiers are further trained on the expanded dataset. later, @xcite proved that weak dependence between views, a weaker assumption than conditional independence, is sufficient for co-training. next, @xcite proved an even weaker sufficient condition they term expansion assumption. after that, @xcite showed that combinative label propagation over two views is both sufficient and necessary for co-training. ling et al. ( 2009 ) empirically verifies on 32 datasets that improvements by co-training are likelier the more independent the two views are and the more sufficient each view is, publishing results supporting aforementioned sufficient conditions for co-training. in this work, we observed that commits possess a natural view split -commit message (natural language) and code changes (programming language) -that one could reasonably hope for the weak dependence assumption as well as the sufficiency for learning from one view assumption to hold. if true, co-training is a natural fit for learning on commit datasets. ## the 900repo dataset to create a balanced dataset from ra-data, we first noted that about 910 repositories overlap in both classes -security ("positive" class) & non-security ("negative" class). the remaining repositories occur in one class but not the other. we extracted all 3,765 positive samples belonging to these 910 overlapping repositories and randomly sampled twice as many negative samples from the same 910 repositories, giving a total of 10,000 commits. as at august 2021, due to ra-data not having code changes (a.k.a. diffs), we scraped for the code changes via github's api. the result is a dataset with 3,765 positive samples and roughly 6,300 negative samples that we refer to as 900repo. this dataset is publicly available at https://github. com/davidleejy/wnut21-cotrain . rq2. as codebert was pre-trained with the aim of capturing the semantic connection between nl and pl, one might expect codebert to produce better representations for commit messages than the architecturally-identical roberta which was not trained with nl-pl understanding in mind. our experiments in section 4.1 suggest that the representations generated by roberta and code-bert for commit messages offer similar performance outcomes for cc. ## co-training for commit classification in the previous section, a joint representation of the two modalities is obtained by attaching a single mlp to the outputs of the transformer models. however, here the codebert & roberta outputs are directed to separate mlp's. for convenience, we denote as nl the roberta-mlp classifier model on commit messages, and as pl the codebert-mlp classifier model on code changes. to simulate resource-poor conditions, we use either 0.5%, 1%, 2% or 4% of the 900repo dataset as labeled training data with a validation set of the same size, and 15% of the data as test set. the remaining is used as unlabeled data for co-training. the validation data is used by supervised baselines for model selection (i.e. selecting the epoch during training with best performance on validation set). we show our co-training algorithm in algorithm 1. in each iteration, a small amount of confidently labeled commits are added to the training dataset which in turn is used in the next iteration's training. at all times, only the mlp portions are fine-tuned via gradient descent with adam for cc. pre-trained codebert and roberta weights are frozen. in all our experiments, we set @xmath0.001, @xmath1.15, n max ( ) = 3, n max ( ) = 5, n min ( ) = n min ( ) = 1. results: under both supervised & co-training approaches, the nl classifier is observed to perform better than the pl classifier. this could be due to the challenging nature of obtaining good representations for programming languages as well as the length of code changes frequently being much longer than the commit message and codebert's 512 token limit. ## conclusion we found co-training to be helpful for cc in resource-poor settings where there are insufficient labeled commits for supervised approaches. further research directions could explore the use of semi-supervised learning methods to expand the capabilities of semantic search within code repositories. another limitation of our current work is that commits are often too long for roberta and codebert. for future work, we could apply models that allow longer sequences as input, e.g., the longformer @xcite .