Using Developer Discussions to Guide Fixing Bugs in Software (2022)

Abstract: Automatically fixing software bugs is a challenging task. While recent work showed that natural language context is useful in guiding bug-fixing models, the approach required prompting developers to provide this context, which was simulated through commit messages written after the bug-fixing code changes were made. We instead propose using bug report discussions, which are available before the task is performed and are also naturally occurring, avoiding the need for any additional information from developers. For this, we augment standard bug-fixing datasets with bug report discussions. Using these newly compiled datasets, we demonstrate that various forms of natural language context derived from such discussions can aid bug-fixing, even leading to improved performance over using commit messages corresponding to the oracle bug-fixing commits.

## introduction software defects, or bugs, arise for a number of reasons, including missing or changing specifications, programming errors, poor documentation, and overall complexity @xcite . due to the extensive developer time and effort needed to fix bugs @xcite , there is growing interest in automated bug fixing @xcite @xcite @xcite . most of these approaches only consider the buggy code snippet when generating the fix. however, with such limited context, this is extremely challenging. for instance, in figure 1a , generating the fixed code requires removing .append("\n"), but this is not obvious from inspecting the buggy code alone. to address this, @xcite proposed prompting developers for a natural language description of intent (e.g., "removed trailing newlines...") that can guide a model in performing the task. as a proxy, in their study, they used the commit message corresponding to the oracle commit which fixed the bug. by showing that natural language can aid bugfixing, their study yields promising results. however, we raise two concerns with their approach. first, prompting developers for additional information can be burdensome for them, as it requires time and manual effort. second, and more importantly, it is unrealistic to use the oracle commit message as a proxy. since it is written after the bug is fixed to document the code changes @xcite , it does not accurately reflect information actually available when the task needs to be performed. in reality, there are more appropriate sources of natural language to guide fixing bugs, which are naturally occurring and available before the task is to be performed. namely, many bugs are first reported through issue tracking systems (e.g., github issues), where developers engage in a discussion to collectively understand the problem, investigate the cause, and formulate a solution (before they are fixed) @xcite . content in these discussions are often relevant to generating the fix. for example, in figure 1b , the title suggests that the bug pertains to "trailing newlines" and the last utterance of the discussion recommends "removing the newlines." additionally, using modern techniques @xcite that summarize content relevant towards implementing the solution in a bug report discussion, we can also automatically obtain a natural language description of the solution ("remove trailing newlines..."). note that these sequences provide insight on the intent of the fix, much like the oracle commit message, without requiring any additional input or any context beyond what is naturally available. in this work, we use bug report discussions to facilitate automated bug fixing. while these discussions have been previously used to automate tasks related to bug resolution, such as localizing bugs @xcite and assigning relevant developers @xcite , to our knowledge, they have never been used to directly generate the bug-fixing code. we propose various input context representations, encompassing different natural language components that are tied to the discussions and likely to capture their meaningful aspects. we heuristically derive components from discussions, including the discussion as a whole, the title, and last utterance. we also derive components algorithmically through model-generated solution descriptions and the attended discussion utterances during this generation. we incorporate these representations into large sequence-to-sequence models pretrained on large amounts of source code and technical text @xcite and then do task-specific finetuning. for training and evaluation, we mine bug report discussions from github issues and map them to subsets of @xcite 's bug-fixing patches datasets. 1 results show that when bug report discussions are available, they lead to significant improvements in fixing bugs, even outperforming using the oracle commit message. ## using bug report discussions many bugs are reported with issue tracking systems, through which a user can open a bug report and initiate a discussion with developers. the user first states the problem in the title and typically elaborates in the first utterance. developers then join the discussion and engage in a dialogue with the user as well as other developers. these discussions isolate the problem, diagnose the cause, and prescribe potential solutions @xcite . due to their technical nature, they often span more than just natural language, including system error messages and relevant code snippets @xcite . furthermore, they are readily available before bugs are fixed. so, we consider using these contextually rich discussions to guide the task of bug fixing. we devise various strategies for heuristically and algorithmically deriving context from these discussions. ## data chakraborty and ray (2021) relied on the commonly used bug-fixing patches (bfp) datasets @xcite . this entails bfp small , with examples extracted from java methods spanning fewer than 50 tokens, and bfp medium , with examples extracted from methods spanning 50-100 tokens. in this work, we also focus on these datasets, particularly the preprocessed versions released by @xcite . however, since they do not include the associated bug report discussions, we enrich examples with this information. ## results following chakraborty and ray (2021), we compute how often (%) the generated output exactly matches the target fixed code snippet. we perform statistical significance testing with bootstrap tests @xcite , using 10,000 samples (with sample size 5,000) and @xmath0.05. we provide sample output in appendix b. we present results in we also observe that using context derived from bug report discussions leads to improved performance (1.5-2.8%) over using the oracle commit message during our finetuning with the disc-bfp training sets. however, when chakraborty and ray (2021) originally considered using the oracle commit message, they had finetuned with it as input on significantly more data (i.e., the full bfp training sets). so, we further investigate by initializing plbart parameters from the chakraborty and ray (2021) checkpoint which was finetuned using the oracle commit message (@xmath1. then, we perform finetuning on the disc-bfp training sets with the various finetune/test ctxt disc-bfp small disc-bfp med without nl § 35.5 25.3 oracle msg ¶ 36.2 25.9 whole discussion 34.1 25.6 title 35.2 25.3 last utterance 36.2 25.6 soln desc 33.4 26.5 § soln desc + title 39.2 § ¶ 26.2 § attended seg 36.9 § 24.1 relative to the results presented in overall, the scores and magnitude of improvement tend to be lower for disc-bfp medium . this is likely due to the challenges of generating longer sequences @xcite and the stringent evaluation metric requiring exact match with the reference. the best performance on the disc-bfp small test set comes from using solution description + title. for disc-bfp medium , it is with the last utterance. since both of these are derived from the whole discussion, one may expect using the whole discussion to yield similar or even improved performance; however, this is not the case. including the whole discussion substantially increases the input length, which models like plbart cannot easily handle. this can be partially attributed to the practical challenge of fitting the entire sequence in the model's limited context window, with 12.8-15.8% training examples getting truncated. however, the bigger challenge is drawing meaning from such large amounts of text. we demonstrate the benefits of using more concise sequences, through various natural language elements that are likely to capture critical aspects of the whole discussion. ## conclusion in this work, we investigated the utility of natural language for automated bug fixing. unlike prior work, which leverages an unrealistic source of natural language for this purpose, through oracle commit messages, we consider a naturally occurring source that is often available: bug report discussions. we explore various strategies for deriving natural language context from these discussions, using our newly compiled discussion-augmented, bug-fixing patches datasets. we show that when these discussions are available, they offer useful context for bug fixing, even leading to improved performance over using oracle commit messages. furthermore, bug report discussions are not always available, and our work focuses on those instances in which they are available. because @xcite do not consider bug report discussions in their work, they do not require examples to have bug report discussions in their datasets. however, we do need examples to have these discussions for our study. because we are unable to map many of their examples to bug report discussions, we focus on smaller subsets of their datasets.