Learning to Describe Solutions for Bug Reports Based on Developer Discussions (2022)

Abstract: When a software bug is reported, developers engage in a discussion to collaboratively resolve it. While the solution is likely formulated within the discussion, it is often buried in a large amount of text, making it difficult to comprehend and delaying its implementation. To expedite bug resolution, we propose generating a concise natural language description of the solution by synthesizing relevant content within the discussion, which encompasses both natural language and source code. We build a corpus for this task using a novel technique for obtaining noisy supervision from repository changes linked to bug reports, with which we establish benchmarks. We also design two systems for generating a description during an ongoing discussion by classifying when sufficient context for performing the task emerges in real-time. With automated and human evaluation, we find this task to form an ideal testbed for complex reasoning in long, bimodal dialogue context.

## introduction software bugs in open-source projects are reported through issue tracking systems like github issues. when a bug is reported, a discussion is initiated among developers to collectively resolve it @xcite . the bug resolution process is often strenuous and time-consuming, involving extended deliberations @xcite among multiple participants @xcite , spanning long periods of time @xcite . although a solution often emerges within the discussion @xcite , this can easily get lost in a large amount of text @xcite . wading through a long discussion to determine whether a solution has been suggested, comprehending it, and then implementing it can be daunting, especially for developers who are not closely following the discussion @xcite . consequently, the resolution can be delayed. as developers scan through the long discussion, it is desirable to have an automated system that guides them to more easily absorb information relevant towards implementing the solution. we propose automatically generating a concise natural language description of the solution by synthesizing relevant content as it emerges in the discussion. for example, as the discussion in figure 1 progresses, the cause of the bug is identified as the shutter getting closed "when seeking to an unprepared period" and a solution emerges: "suppress closing the shutter in this case, provided the old and new periods belong to the same window." our task aims to describe this solution: prevent shutter closing for within-window seeks to unprepared periods. to study this task, we build a corpus from bug report discussions on github issues. the changes made within the code base to resolve the bug are often linked to the bug report in the form of a commit or pull request. we develop a novel approach to ob-tain noisy supervision for the solution description from the associated commit message or pull request title which describe the bug-resolving changes in natural language. to control for noise, we apply filtering techniques. the dataset and code are publicly available for research use. 1 with this data, we set benchmarks for generating solution descriptions, conditioned on the discussion. from the long context, a model must learn to tease out and condense information relevant to the solution. handling long context is critical for tasks with dialogue as input, since the input grows rapidly with the number of interactions. additionally, the context entails technical text, with natural language and source code often appearing in the same sentence @xcite . so, deducing information from the context to articulate a meaningful description requires complex reasoning. we explore generation models including transformer models @xcite and plbart @xcite , which was pretrained on large quantities of code and technical text. we evaluate with automated metrics and human evaluation. furthermore, we investigate integrating our task into a real-time setting. an informative description can be generated only if there is sufficient context about the solution, so we must wait until this context is available. in figure 1 , generation should be performed only after utterance #4 is made in the discussion. since the solution is not formulated until that point, there is insufficient context to reliably generate a description before then. we design two methods for integrating a classifier that determines when to generate with a generation model: (1) a pipelined system with independently trained classification and generation models; (2) a joint system that is simultaneously trained for both tasks. by monitoring progress and later chiming into the discussion with a solution description, this combined system lays the groundwork for future work on developing an intelligent dialogue agent which participates in discussions to facilitate more efficient bug resolution. while there is growing interest in building tools to support development activities such as code summarization @xcite , comment updating @xcite , and commit message generation @xcite , dialogue systems have been largely understudied in this domain. we con-sider our work as a step towards building more dialogue-based ai tools for software development. ## problem setting as shown in figure 1 , when a user reports a bug, they state the problem in the title (e.g., "black screen appears when we seek over an adgroup") and initiate a discussion by making the first utterance (u 1 ), which usually elaborates on the problem. other participants join the discussion at later time steps through utterances (u 2 ...u t ), where t is the total number of utterances. throughout the discussion, developers discuss various aspects of the bug, including a potential solution @xcite . we propose the task of generating a concise description of the solution (e.g., "prevent shutter closing for within-window seeks to unprepared periods") by synthesizing relevant content within the title and sequence of utterances (u 1 , u 2 ...). ## data following prior work on other tasks @xcite , we mine issue reports corresponding to open-source java projects from github issues. issue reports can entail feature requests as well as bug reports. in this work, we focus on the latter. we identify bug reports by searching for "bug" in the labels assigned to a report and by using a heuristic for identifying bugrelated commits @xcite . ## generating solution descriptions we first generate informative solution descriptions in a static setting, in which we leverage the oracle context from the discussion (i.e., the title and u 1 ...u tg ). from ## supporting real-time generation generating an informative description requires sufficient context about the solution being available in the discussion. in a real-time setting, this context is likely not immediately available but rather emerges as the discussion progresses, and we must wait until it becomes available to generate a solution description. however, the time step at which it becomes available (t g ) is not known beforehand, so we must instead predict it (t p ) in order to perform generation during ongoing discussions. for this, we consider classifying whether sufficient context is available upon each new utterance. in figure 1 , the solution is formulated in u 4 , so the correct behavior is to predict the negative label at @xmath0, 2, 3 and the positive label at @xmath1. once the positive label is predicted at t p foot_5 , the description is generated, conditioned on the title and u 1 ...u tp . we develop title plbart (f) reference (1) issue with dex: oidc server is not available at the 'quarkus.oidc.auth-server-url' url fix trailing slash in auth -server url strip trailing forward slash from oidc url (2) invaliddatatypeexception: udata contains value larger than integer.max_value ddr issue decoding lookswitch fix bug in byte code dumper when tableswitch instruction precedes tableswitch instruction fix interpretation of switch instructions in byte code dumper (3) worldmap viewport changes when switching between dashboard pages don ' t refresh widget grid when worldmap loses viewport define key prop for map visualization to update map on dimension change (4) workaround comments exist in opengrokindexer/pom.xml file while the related issues are already fixed. fix jflex -de / jflex # 705 ( comment ) use jflex 1.8.2 (5) why subscribe with single action for onnext design to crush if error happened? 1 . x : fix subscription . subscribe ( ) to return observable . empty ( ) 2 . x : fix subscription . subscribe ( ) to return observable . empty ( ) fixed sonar findings ## related work bug report summarization: to help developers gather information from bug reports, there is interest in automatic bug report summarization. approaches for this are designed to generate holistic summaries of bug reports, with a summary being 25% of the length of the bug report @xcite . we instead aim to generate a concise description that captures a specific aspect of the bug report. next, bug report summaries are not widely available, so approaches for this task rely on unsupervised techniques @xcite or supervision from a small amount of data @xcite . our approach for obtaining noisy supervision allows us to train supervised models on a large amount of data. bug report summarization is a post hoc task, done after the bug has been resolved, to help developers address related bug reports in the future. in contrast, our goal is to help resolve the present bug report, so our system must learn when to perform generation during an ongoing discussion. approaches for bug report summarization have been predominantly extractive whereas ours is abstractive. while we are interested in how bug report summarization techniques fair on our task, their implementations are not publicly available. commit message generation: unlike the task of automatically generating commit messages to describe code changes that have already been made @xcite , our system aims to generate natural language descriptions that can drive code changes. response triggering: classifying when to generate a description relates to chatbots learning to respond at an appropriate time @xcite in dyadic conversations. the goal is to avoid interrupting a user who splits up an utterance across multiple turns. we instead consider multi-party dialogue in which an agent should wait until a specific type of content emerges in the discussion. @xcite studied turn-taking decisions in spoken dialogue systems, using audio-visual features, while ours is a text-based system. dialogue + software: we view our work as a step towards building a dialogue agent for streamlining software bug resolution. there has been minimal work in building interactive systems for this domain, with the exception of a few for tasks like query refinement @xcite and code generation @xcite . @xcite recently built a dialogue corpus through a "wizard of oz" experiment to study the potential of a q&a assistant during bug fixing. @xcite developed a dialogue corpus based on ubuntu chat logs to study q&a assistants for technical support. in contrast, our dataset is designed for building a collaborative agent that participates in multi-party conversations rather than one which answers directed questions. ## conclusion we presented the novel task of generating concise natural language solution descriptions to guide developers in absorbing information relevant towards bug resolution from long discussions. we established benchmarks for this task using a dataset that we constructed with supervision derived from commit messages and pull request titles. through automated and human evaluation, we demonstrated the utility of these models and also highlight their shortcomings, to encourage more research in exploring ways to address these challenges. we also simulated a real-time setting through two approaches for combining a generation model with a classification component for determining when sufficient context for generating an informative description emerges in an ongoing discussion. we believe this lays the groundwork for future work on building a dialogue agent that participates in bug report discussions to foster efficient resolution. to build our system, we used data from github, in accordance with its acceptable use policy, and no additional permission was required. namely, the policy states: "researchers may use public, non-personal information from the service for research purposes, only if any publications resulting from that research are open access." 7 we use only publicly available data and use it only for research purposes. additionally, the data we used to train and evaluate models (and publicly release) does not contain personal information (e.g., usernames of users who authored utterances and linked mentions). we require that any future work using our dataset must abide by github's official policy as well. for evaluation, we conducted human evaluation, for which participants willfully volunteered to be part of the study. they were not compensated for their participation.