RNS um: A Large-Scale Dataset for Automatic Release Note Generation via Commit Logs Summarization (2022)

Abstract: A release note is a technical document that describes the latest changes to a software product and is crucial in open source software development. However, it still remains challenging to generate release notes automatically. In this paper, we present a new dataset called RNSum, which contains approximately 82,000 English release notes and the associated commit messages derived from the online repositories in GitHub. Then, we propose classwise extractive-then-abstractive/abstractive summarization approaches to this task, which can employ a modern transformer-based seq2seq network like BART and can be applied to various repositories without specific constraints. The experimental results on the RNSum dataset show that the proposed methods can generate less noisy release notes at higher coverage than the baselines. We also observe that there is a significant gap in the coverage of essential information when compared to human references. Our dataset and the code are publicly available.

## introduction recently, there has been considerable interest in applying natural language processing (nlp) techniques to support software development (iyer et al., 2016; yin and neubig, 2018; panthaplackel et al., 2020). one such task involves the automatic generation of release notes. a release note is a technical document that describes the latest changes to a software product, which is necessary for software developers to adjust their codes accurately for using the updated software. since release notes are timeconsuming to write manually, several studies have been done to explore automatic release note generation. moreno et al. (2014) proposed arena, figure 1 : an example data in rnsum; this example is derived from the release of tag v2.6.4 in https: //github.com/vuejs/vue . an automatic release notes generator, which first extracts and summarizes the changes in the source code and then integrates them with the information provided by version-control and issue-tracking systems. pokorný (2020) developed glyph, which classifies commit messages into predefined releasenote categories (e.g., features, bug fixes) using pre-trained word embeddings and produces the categorized commit messages as the final release note. despite the progress reported in these previous studies, usable release note generators are far from realization. we attribute this difficulty mainly to two problems. first, the existing resources for automatic release note generation are scarce; for example, glyph was trained on only 5,000 commit messages, which is too little for obtaining a sufficiently generalized model. second, the existing methods have limited applicability; for example, arena requires an issue tracker hosted on jira, thus preventing it from being used for most github repositories. also, glyph's predefined release-note categories do not include deprecations and removals, which are often indispensable in release notes. to alleviate the above problems, we introduce rnsum, a new large-scale dataset for automatic release note generation via commit logs summarization. an example data in rnsum is shown in figure 1 . rnsum consists of approximately 82,000 release notes derived from online repositories on github. the contents of each release note are further categorized into four release-note classes: (1) features, (2) improvements, (3) bug fixes, and (4) deprecations+ (deprecations, removals, and breaking changes). the release notes are associated with the commit messages that were used by human maintainers to write the release notes. the difficulty of this task is that there is no explicit alignment between each commit message and the release note categories. for example, in figure 1 , the first commit message ("chore: make documentation clearer (#9450)") is not reflected in the release notes. in contrast, the second commit message ("fix: empty scoped slot should return undefined fix #9452") is reflected as the third release note in the bug fixes class. we propose two approaches to this task: classwise extractive-then-abstractive summarization (ceas) and classwise abstractive summarization (cas) models, which learn to produce categorized release notes given unlabeled commit messages in extractive and abstractive manners, respectively. the two proposed models can leverage modern transformer-based sequence-to-sequence (seq2seq) architectures (e.g., bart (lewis et al., 2020)) and can be used for various repositories without any special constraints. we evaluate the proposed models and the previous systems on the rnsum dataset and report that our approaches generate less noisy release notes at higher coverage than the baselines given only unlabeled commit messages. we also perform human evaluations carefully to assess how well the systems could generate release notes in terms of quality (precision) and coverage (recall), revealing that there still remains a significant gap in the coverage when compared to human references. our dataset and the source code are publicly available. 1 here, we define the automatic release note generation task. the input is a set of commit messages (sentences), @xmath0, . . . , x n }. given the input commit messages x, our goal is to generate labeled release notes y c for each predefined release-note class @xmath1. each labeled release note is a collection of sentences, i.e., y @xmath2,1 , y c,2 , . . . . according to moreno et al. (2014), the major contents of most release notes can be categorized into the following classes: fixed bugs, new features, new code components (cc), modified cc, etc. based on their observations, we define the release-note classes c comprising the following four categories: features (f), improvements (i), bug fixes (b), and deprecations+ (d; deprecations, removals, and breaking changes). @xcite our classes do not include refactoring, document changes, and library updates because most of the maintainers on github omitted these changes in their release notes. our dataset can be interpreted as a collection of quintuples, namely {x, y f , y i , y b , y d }. for simplicity, we also use (x, y) instead of the quintuple representation. it is worth noting that the labeled release notes y c can be empty. for example, in figure 1 , the software update is related to improvements and bug fixes. thus, the release note contains only the improvements and bug fixes classes, and y @xmath3. to address the limitations of these three studies, we built a new large-scale dataset called rnsum, which contains approximately 82,000 release notes with commit messages. we reviewed the release notes carefully and redefined the four classes. we also propose classwise summarization methods to automatic release note generation, which can be applied to all english repositories on github. classwise summarization there are several reported studies that use class information for summarization. cao et al. (2017) and yang et al. (2018) proposed using text categories to improve text summarization. liang et al. (2019) proposed a clinical note extractive summarization system that generates summaries based on specific disease names. in contrast to these studies, we developed classwise summarization methods for release note generation, which we confirm through experiments to be more effective than the baselines. ## proposed methods automatic release note generation can be viewed as a task of summarizing commit messages x into the labeled release notes y c . in this paper, we introduce classwise extractive-then-abstractive summarization (ceas) and classwise abstractive summarization (cas) models, which we instantiate by modern transformer-based sequence-to-sequence (seq2seq) networks and can be universally used in various repositories without any special constraints. ## conclusion in this paper, we presented a new large-scale dataset for automatic generation of release notes. the dataset comprised approximately 82k release notes from over 7k repositories on github. we formulated a task to automatically generate release notes by summarizing the commit messages, which can be applied to all software development projects that use english. we confirmed the validity of the proposed classwise extractive-then-abstractive summarization (ceas) model and the classwise abstractive summarization (cas) model via experiments to generate less noisy release notes at higher coverage than the baselines. however, there are still gaps in the coverage performance compared to manually generated outputs; it is expected that this could be improved by including additional information such as issues and pull requests with the commit logs.