Leveraging Context-Aware Prompting for Commit Message Generation (2024)

Abstract: Writing comprehensive commit messages is tedious yet important, because these messages describe changes of code, such as fixing bugs or adding new features. However, most existing methods focus on either only the changed lines or nearest context lines, without considering the effectiveness of selecting useful contexts. On the other hand, it is possible that introducing excessive contexts can lead to noise. To this end, we propose a code model COMMIT (Context-aware prOMpting based comMIt-message generaTion) in conjunction with a code dataset CODEC (COntext and metaData Enhanced Code dataset). Leveraging program slicing, CODEC consolidates code changes along with related contexts via property graph analysis. Further, utilizing CodeT5+ as the backbone model, we train COMMIT via context-aware prompt on CODEC. Experiments show that COMMIT can surpass all compared models including pre-trained language models for code (code-PLMs) such as CommitBART and large language models for code (code-LLMs) such as Code-LlaMa. Besides, we investigate several research questions (RQs), further verifying the effectiveness of our approach. We release the data and code at: https://github.com/Jnunlplab/COMMIT.git.

## introduction commit messages @xcite are communication channels of programmers and part of version control system, which tracks and manages every modification to software code during development. while version control systems help software teams work better by comparing earlier versions of code and turning back the clock to fix mistakes, natural language commit messages summarize what happened in the code change and/or explain why the code change was made @xcite . however, developers, whose time is precious, are often reluctant to write commit messages. five kinds of automatic commit message generation approaches have been proposed. first, information retrieval based models @xcite find the most similar messages for a new commit. second, end-to-end generative models @xcite are trained directly on task corpus. third, hybrid models @xcite incorporate both retrieved and generated messages. fourth, pre-trained language models (plms) are adapted into code tasks @xcite . fifth, large language models (llms) are called via apis or finetuned @xcite . however, existing methods normally have some shortcomings, e.g., focusing on only the changed lines is not enough or including unrelated lines around changes will bring in noise @xcite . as exemplified in fig. 1 , the added and deleted codes are literally the same, leading to the code changes illdefined. also, the association between the changed lines and the unchanged parts of the program is unclear due to lacking program dependency analysis. motivated by the fact that many static analysis tools foot_0 have been proposed based on the theory of code property graphs (cpgs) @xcite , we propose a code model com-mit (context-aware prompting based commitmessage generation) in conjunction with a code dataset codec (context and metadata enhanced code dataset). we construct codec for consolidating the code changes along with their semantically related contexts via using cpgs. further, designing context-enhanced prompt, we implement commit using codet5+ @xcite as the backbone model that is trained on codec. the technique novelty is two-fold. first, we propose new graph representations including deleted context graphs (dcgs), added context graph (acgs), and added-deleted context graph (ad-cgs). both dcgs and acgs can be efficiently derived from cpgs and we can further construct adcgs via combining acgs and dcgs (e.g., identifying the same program statements and collecting the dependency edges). adcgs contain richer information than both dcgs and acgs. based on them, we can efficiently extract useful contexts (indicated by adcg nodes) from code differences (i.e., diffs) via program slicing. second, we select possibly crucial contexts which have a cpgcontained program dependency with the changed codes, and enhance a prompt-based code model (e.g., codet5+) for better message generation. our contributions can be summarized as follows: the rest of this paper is organized as follows. first, after introducing cpgs in sec. 2, the collection process of codec, along with a comparison with seven commonly used datasets, are placed at sec. 3. then, commit is presented in sec. 4. the experiments in sec. 5 include comparisons between various models and discusses on rqs as well as case studies. we conclude this paper with future works in (sec. 6). more details including the experimental settings and the prompt design are posted in appx. a∼d because of the space limit. ## code property graph cpgs play a vital role in extracting associated context of the code change in our method. following definitions from previous studies @xcite , cpgs can be formally defined as follows. definition 1 property graph (pg). the property graph @xmath0, e, λ, µ) is a labeled directed pseudograph with attached properties. v is the set of all vertices, e ⊆ (v × v ) is the edge set, and λ : @xmath1. given all keys of properties k and all values of properties s, µ : (v ∪ e) × @xmath2. e.g., key-value pairs can be (k, x). ## the codec dataset since existing datasets lack supervision signals on context of the code change, we construct our own dataset for the studied task in this paper. we elaborate on main steps of dataset construction and provide a comprehensive comparison with others. ## the commit model implementing commit has four steps: (1) cpgs are built by analyzing source codes before and after changes. (2) dependencies in cpg are first extracted and then merged to adcg. (3) contextaware representation is constructed on adcg via programming slicing. (4) commit-message is generated by context-aware prompting on codet5+. ## conclusion this paper proposes the codec dataset and the commit model. as dependency-sensitive contexts to the code changes are fully considered, codec is featured as context-aware comparing with commonly used datasets. we also propose a dependency extraction algorithm based on the innovative added-deleted context graphs and implement commit using codet5+ as backbone. extensive experiments and case studies demonstrate the effectiveness and superiority of our proposals. future works are as follows. our current dataset collected only java programs. we would like to extend the dataset by including other programming language codes such as c++ and python in future. we will further validate the effective of our method on diverse programming-style codes. besides, we will conduct an in-depth analysis on the contribution of each kind of program dependence used in this paper, to the overall performance of our model.