Improving IR-Based Bug Localization with Context-Aware Query Reformulation

Recent findings suggest that Information Retrieval (IR)-based bug localization techniques do not perform well if the bug report lacks rich structured information (relevant program entity names). Conversely, excessive structured information (stack traces) in the bug report might not always help the automated localization either. In this paper, we propose a novel techniqueBLIZZARD that automatically localizes buggy entities from project source using appropriate query reformulation and effective information retrieval. In particular, our technique determines whether there are excessive program entities or not in a bug report (query), and then applies appropriate reformulations to the query for bug localization. Experiments using 5,139 bug reports show that our technique can localize the buggy source documents with 7%56% higher Hit@10, 6%62% higher MAP@10 and 6%62% higher MRR@10 than the baseline technique. Comparison with the state-of-the-art techniques and their variants report that our technique can improve 19% in MAP@10 and 20% in MRR@10 over the state-of-the-art, and can improve 59% of the noisy queries and 39% of the poor queries.

Introduction
============

Despite numerous attempts for automation @crowddebug [@debugadvisor; @majumdar; @reuse; @zimmer], software debugging is still largely a manual process which costs a significant amount of development time and efforts @whofix [@parnin; @harmful]. One of the three steps of debugging is the identification of the location of a bug in the source code,  bug localization @parnin [@parninireval]. Recent bug localization techniques can be classified into two broad families–*spectra based* and *information retrieval (IR) based* @locombined. While spectra-based techniques rely on execution traces of a software system, IR-based techniques analyse shared vocabulary between a bug report ( query) and the project source for bug localization @buglocator [@stacktrace]. Performances of IR-based techniques are reported to be as good as that of spectra-based techniques, and such performances are achieved using a low cost text analysis @raobug [@parninireval]. Unfortunately, recent qualitative and empirical studies @parninireval [@icse2018] have reported two major limitations. First, IR-based techniques cannot perform well without the presence of rich structured information ( program entity names pointing to defects) in the bug reports. Second, they also might not perform well with a bug report that contains excessive structured information ( stack traces, Table \[table:noisy\]) @parninireval. One possible explanation of these limitations could be that most of the contemporary IR-based techniques @buglocator [@bluir; @kak; @versionhistory; @locombined; @raobug; @ldabug] use almost verbatim texts from a bug report as a *query* for bug localization. That is, they do not perform any meaningful modification to the query except a limited natural language pre-processing ( stop word removal, token splitting, stemming). As a result, their query could be either *noisy* due to excessive structured information ( stack traces) or *poor* due to the lack of relevant structured information ( Table \[table:poor\]). One way to overcome the above challenges is to (a) refine the noisy query ( Table \[table:noisy\]) using appropriate filters and (b) complement the poor query ( Table \[table:poor\]) with relevant search terms. Existing studies @versionhistory [@wembedding; @hyloc; @versionhistoryjsep] that attempt to complement basic IR-based localization with costly data mining or machine learning alternatives can also equally benefit from such query reformulations.

In this paper, we propose a novel technique –BLIZZARD– that locates software bugs from source code by employing *context-aware query reformulation* and information retrieval. Our technique (1) first determines the quality ( prevalence of structured entities or lack thereof) of a bug report ( query) and classifies it as either *noisy*, *rich* or *poor*, (2) then applies appropriate reformulation to the query, and (3) finally uses the improved query for the bug localization with information retrieval. Unlike earlier approaches @buglocator [@bluir; @bluirplus; @versionhistory], it either refines a noisy query or complements a poor query for effective information retrieval. Thus, BLIZZARD has a high potential for improving IR-based bug localization. To illustrate the capability of our technique in improving bug localization, we provide two examples in which it outperforms the baseline. The baseline technique that uses all terms except punctuation marks, stop words and digits from a bug report, returns its first correct result for the noisy query containing stack traces in Table \[table:noisy\] at the 53$^{rd}$ position. On the contrary, our technique refines the same noisy query, and returns the first correct result at the first position of the ranked list which is a significant improvement over the baseline. Similarly, when we use a poor query containing no structured entities such as in Table \[table:poor\], the baseline technique returns the correct result at the 30$^{th}$ position. On the other hand, our technique improves the same poor query, and returns the result again at the first position. BugLocator @buglocator, one of the well cited IR-based techniques, returns such results at the 19$^{th}$ and 26$^{th}$ positions respectively for the noisy and poor queries which are far from ideal.

We evaluate our technique in several different dimensions using four widely used performance metrics and 5,139 bug reports ( queries) from six Java-based subject systems. First, we evaluate in terms of the performance metrics, contrast with the baseline, and BLIZZARD localizes bugs with 7%–56% higher accuracy ( Hit@10), 6%–62% higher precision ( MAP@10) and and 6%–62% higher result ranks ( MRR@10) than the baseline (Section \[sec:result\]). Second, we compare our technique with three bug localization techniques @buglocator [@bluir; @versionhistoryjsep], and our technique can improve 19% in MAP@10 and 20% in MRR@10 over the state-of-the-art @versionhistoryjsep (Section \[sec:comparison\]). Third, we also compare our approach with four state-of-the-art query reformulations techniques, and BLIZZARD improves the result ranks of 59% of the noisy queries and 39% of the poor queries which are 22% and 28% higher respectively than that of the state-of-the-art @saner2017masud (Section \[sec:comparison\]). By incorporating *report quality aspect* and *query reformulation* into IR-based bug localization, we resolve an important issue which was either not addressed properly or otherwise overlooked by earlier studies, which makes our work *novel*. Thus, the paper makes the following contributions:

-   A novel query reformulation technique that filters noise from and adds complementary information to the bug report, and suggests improved queries for bug localization.

-   A novel bug localization technique that locates bugs from the project source by employing quality paradigm of bug reports, query reformulation, and information retrieval.

-   Comprehensive evaluation of the technique using 5,139 bug reports from six open source systems and validation against seven techniques including the state-of-the-art.

-   A working prototype with detailed experimental data for replication and third party reuses.

Graph-Based Term Weighting {#sec:bg}
==========================

Term weighting is a process of determining relative importance of a term within a body of texts ( document). @tfidf first introduced TF-IDF ( term frequency $\times$ inverse document frequency) as a proxy to term importance which had been widely used by information retrieval community for the last couple of decades. Unfortunately, TF-IDF does not consider semantic dependencies among the terms in their importance estimation. @rada later proposed TextRank as a proxy of term importance which was adapted from Google's PageRank @pagerank and was reported to perform better than TF-IDF. In TextRank, a textual document is encoded into a text graph where unique words from the document are denoted as nodes, and meaningful relations among the words are denoted as connecting edges @rada. Such relationships could be statistical ( co-occurrence), syntactic ( grammatical modification) or semantic ( conceptual relevance) in nature @blanco. In this research, we identify important terms using graph-based term weighting from a bug report that might contain structured elements ( stack traces) and unstructured regular texts.

BLIZZARD: Proposed Technique {#sec:blizzard}
============================

Fig. \[fig:sysdiag\] shows the schematic diagram of our proposed technique–BLIZZARD. Furthermore, Algorithm \[algo\] shows the pseudo-code for BLIZZARD. We make use of bug report quality, query reformulation, and information retrieval for localizing bugs in source code from bug reports of any quality as shown in the following sections:

Bug Report Classification {#sec:classification}
-------------------------

Since our primary objective with this work is to overcome the challenges posed by the different kinds of information bug reports may contain, we categorize the reports prior to bug localization. In addition to having natural language texts, a bug report typically may contain different structured elements: (1) stack traces (reported active stack frames during the occurrence of a bug,  Table \[table:noisy\]), and (2) program elements such as method invocations, package names, and source file names. Having consulted with the relevant literature @structcls [@parninireval; @goodbugreport], we classify the bug reports into three board categories (Steps 1, 2a, 2b and 2c, Fig. \[fig:sysdiag\]) as follows:

**BR$\mathbf{_{ST}}$:** $ST$ stands for stack traces. If a bug report contains one or more stack traces besides the regular texts or program elements, it is classified into BR$_{ST}$. Since trace entries contain too much structured information, query generated from such a report is generally considered *noisy*. We apply the following regular expression @stacktrace to locate the trace entries from the report content.

\[frame=single\] (.\*)?(.+)(.+)($(.+)\.java:\d+$|$Unknown Source$ |$Native Method$)

**BR$\mathbf{_{PE}}$:** $PE$ stands for program elements. If a bug report contains one or more program elements ( method invocations, package names, source file name) but no stack traces in the texts, then it is classified into BR$_{PE}$. Queries generated from such report are considered *rich*. We use appropriate regular expressions @rigby to identify the program elements from the texts. **BR$\mathbf{_{NL}}$:** $NL$ stands for natural language. If a bug report contains neither any program elements nor any stack traces, it is classified into BR$_{NL}$. That is, it contains only unstructured natural language description of the bug. Queries generated from such reports are generally considered *poor* in this work.

We adopt a semi-automated approach in classifying the bug reports ( the queries). Once a bug report is provided, we employ each of our regular expressions to determine its class. If the automated step fails due to ill-defined structures of the report, the class is determined based on manual analysis. Given the explicit nature of the structured entities, human developers can identify the class easily. The contents of each bug report are considered as the *initial queries* which are reformulated in the next few steps.

Query Reformulation
-------------------

Once bug reports ( queries) are classified into three classes above based on their structured elements or lack thereof, we apply appropriate reformulations to them. In particular, we analyse either bug report contents or the results retrieved by them, employ graph-based term weighting, and then identify important keywords from them for query reformulation as follows:

**Trace Graph Development from BR$\mathbf{_{ST}}$:**\[sec:tracegraph\] According to existing findings @icse2018 [@parninireval], bug reports containing stack traces are potentially noisy, and performances of the bug localization using such reports ( the queries) are below the average. Hence, important *search keywords* should be *extracted* from the noisy queries for effective bug localization. In this work, we transform the stack traces into a trace graph ( Fig. \[fig:tgraph\]) (Steps 3a, 4a, Fig. \[fig:sysdiag\], Lines 8–10, Algorithm \[algo\]), and identify the important keywords using a graph-based term weighting algorithm namely PageRank @rada [@blanco].

To the best of our knowledge, to date, graph-based term weighting has been employed only on unstructured regular texts @saner2017masud and semi-structured source code @ase2017masud. On the contrary, we deal with stack traces which are structured and should be analysed carefully. Stack traces generally comprise of an error message containing the encountered exception(s), and an ordered list of method invocation entries. Each invocation entry can be considered as a tuple $t\{P,C,M\}$ that contains a package name $P$, a class name $C$, and a method name $M$. While these entities are statically connected within a tuple, they are often hierarchically connected ( caller-callee relationships) to other tuples from the traces as well. @hillicse09 consider method signatures and field signatures as salient entities from the source code, and suggest keywords from them for code search. Similarly, we consider class name and method name from each of the $N$ tuples as the salient items, and represent them as the nodes and their dependencies as the connecting edges in the graph. In stack traces, the topmost entry ( $i=1$) has the highest degree of interest @context which gradually decreases for the entries at the lower positions in the list. That is, if $t_{i}\{P_i,C_i,M_i\}$ is a tuple under analysis, and $t_{j}\{P_j,C_j,M_j\}$ is a neighbouring tuple with greater degree of interest, then the nodes $V_i$ and edges $E_i$ are added to the trace graph $G_{ST}$ as follows: $$\small
\setlength{\abovedisplayskip}{1pt}
\setlength{\belowdisplayskip}{1pt}
\begin{split}
V_i=\{C_i, M_i\},~E_i= \{C_i\leftrightarrow M_i\}\cup\{C_i\rightarrow C_j, M_i\rightarrow M_j\}\mid j=i-1\\
V=\bigcup\limits_{i=1}^ N\{V_{i}\},~E=\bigcup\limits_{i=1}^N\{E_{i}\},~G_{ST}=(V,E) 
\end{split}$$ For the example stack traces in Table \[table:noisy\], the following connecting edges: `JDIValue`$\leftrightarrow$`toString`, `Cast`$\leftrightarrow$`execute`, `Cast`$\rightarrow$`JDIValue`,\
`execute`$\rightarrow$`toString`, `Interpreter`$\leftrightarrow$`execute`, and `Interpreter`\
$\rightarrow$`Cast` are added to the example trace graph in Fig. \[fig:tgraph\].

\[scale=1, auto,swap\] /in <span> <span>(-.1,0)/Cast</span>, <span>(1.1,0)/access</span>, <span>(1.6,-2)/Interpreter</span>, <span>(-.6,-2)/JDIValue</span>, <span>(2.5,0)/toString</span>, <span>(3.9,0)/run</span>, <span>(1,-1)/runEvaluation</span>, <span>(3.5,-1.2)/doEvaluation</span>, <span>(1.6,1)/EvaluationThread</span>, <span>(-1,-1)/execute</span>, <span>(3.6,-2)/JDIThread</span>, <span>(4.9,-.6)/Thread</span></span> () at ;

at (1.6,1) \[fill=gray!30\] <span>EvaluationThread</span>; at (2.5,0) \[inner sep=1pt,fill=gray!30\] <span>toString</span>; at (-.6,-2) \[inner sep=1pt,fill=gray!30\] <span>JDIValue</span>; at (3.9,0) \[inner sep=1pt,fill=gray!30\] <span>run</span>; at (-1,-1) \[inner sep=1pt,fill=gray!30\] <span>execute</span>;

<span>bg</span> / in <span> execute/Interpreter, JDIValue/toString, run/Thread, JDIThread/EvaluationThread, Cast/execute, runEvaluation/run, EvaluationThread/Interpreter, run/access, execute/toString, Thread/JDIThread, Cast/JDIValue, runEvaluation/JDIThread, run/execute, EvaluationThread/access, execute/Cast, EvaluationThread/doEvaluation, access/doEvaluation, Interpreter/execute, EvaluationThread/run, doEvaluation/EvaluationThread, toString/JDIValue, Thread/run, Interpreter/Cast, access/EvaluationThread, doEvaluation/runEvaluation, run/EvaluationThread, JDIThread/runEvaluation</span> () – ();

**Text Graph Development from BR$\mathbf{_{PE}}$:** Bug reports containing relevant program entities ( method names) are found effective as queries for IR-based bug localization @parninireval [@bluir; @icse2018]. However, we believe that *appropriate keyword selection* from such reports can further *boost up* the localization performance. Existing studies employ TextRank and POSRank on natural language texts, and identify search keywords for concept location @saner2017masud and information retrieval @rada [@blanco]. Although bug reports ( from BR$_{PE}$) might contain certain structures such as program entity names ( class name, method name) and code snippets besides natural language texts, the existing techniques could still be applied to them given that these structures are treated appropriately. We thus remove stop words @stopword and programming keywords @keyword from a bug report, *split* the structured tokens using *Samurai* ( a state-of-the-art token splitting tool @samurai), and then transform the preprocessed report ($R_{pp}$) into a set of sentences ($S\in R_{pp}$). We adopt @saner2017masud that exploits *co-occurrences* and *syntactic dependencies* among the terms for identifying important terms from a textual body ( change request). We thus develop two text graphs (Steps 3b, 4b, Fig. \[fig:sysdiag\], Lines 10–11, Algorithm \[algo\]) using co-occurrences and syntactic dependencies among the words from each report as follows:

*(1) Text Graph using Word Co-occurrences:* In natural language texts, the semantics ( senses) of a given word are often determined by its contexts ( surrounding words) @wordsense [@wembedding; @w2vec]. That is, co-occurring words complement the semantics of each other. We thus consider a sliding window of size $K$ ( $K=2$) @rada, capture co-occurring words, and then encode the word co-occurrences within each window into connecting edges $E$ of a text graph @saner2017masud. The individual words ($\forall w_i\in V$) are denoted as nodes in the graph. Thus, for a target word $w_i$, the following node $V_i$ and two edges $E_i$ will be added to the text graph G$_{PE}$ as follows: $$\setlength{\abovedisplayskip}{1pt}
\setlength{\belowdisplayskip}{1pt}
\begin{split}
V_i= \{w_i\},~E_i=\{w_i\leftrightarrow w_{i-1}, w_i\leftrightarrow w_{i+1}\}\mid S=[w_1..w_i..w_N] \\
V=\bigcup\limits_{\forall S\in R_{pp}} \bigcup\limits_{w_i\in S}\{V_{i}\},~E=\bigcup\limits_{\forall S\in R_{pp}}\bigcup\limits_{w_i\in S}\{E_{i}\},~G_{PE}=(V,E)
\end{split}$$ Thus, the example phrase–*"source code directory"*–yields two edges, *"source"*$\leftrightarrow$*"code"* and *"code"*$\leftrightarrow$*"directory"* while extending the text graph with three distinct nodes– *"source", "code"* and *"directory"*.

*(2) Text Graph using POS Dependencies:* According to *Jespersen's Rank* theory @jespersen [@blanco; @saner2017masud], parts of speech (POS) from a sentence can be divided into three ranks– *primary* ( noun), *secondary* ( verb, adjective) and *tertiary* ( adverb)– where words from a higher rank generally define ( modify) the words from the same or lower ranks. That is, a noun modifies only another noun whereas a verb modifies another noun, verb or an adjective. We determine POS tags using Stanford POS tagger @postagger, and encode such syntactic dependencies among words into connecting edges and individual words as nodes in a text graph. For example, the sentence annotated using Penn Treebank tags @postagger–*"Open$_{VB}$ the$_{DT}$ source$_{NN}$ code$_{NN}$ directory$_{NN}"$*–has the following syntactic dependencies: *"source"*$\leftrightarrow$*"code"*, *"code"*$\leftrightarrow$*"directory"*, *"source"*$\leftrightarrow$*"directory"*,\
*"open"*$\leftarrow$*"source"*, *"open"*$\leftarrow$*"code"* and *"open"*$\leftarrow$*"directory"*, and thus adds six connecting edges to the text graph. **Source Term Graph Development for BR$\mathbf{_{NL}}$:** Bug reports containing only natural language texts and no structured entities are found not effective for IR-based bug localization @parninireval [@icse2018]. We believe that such bug reports possibly miss the right keywords for bug localization. Hence, they need to be *complemented* with *appropriate keywords* before using. A recent study @ase2017masud provides improved reformulations to a poor natural language query for concept location by first collecting *pseudo-relevance feedback* and then employing graph-based term weighting. In pseudo-relevance feedback, Top-K result documents, returned by a given query, are naively considered as relevant and hence, are selected for query reformulation @qsurvey [@refoqus]. Since bug reports from BR$_{NL}$ class contain only natural language texts, the above study might directly be applicable to them. We thus adopt their approach for our query reformulation, collect Top-K ( $K=10$) source code documents retrieved by a BR$_{NL}$-based query, and develop a source term graph (Steps 3c, 4c, Fig. \[fig:sysdiag\], Lines 13–15, Algorithm \[algo\]).

@hillicse09 consider method signatures and fields signatures from source code as the salient items, and suggest keywords for code search from them. In the same vein, we also collect these signatures from each of the $K$ feedback documents for query reformulation. In particular, we extract structured tokens from each signature, split them using *Samurai*, and then generate a natural language phrase from each token @hillicse09. For example, the method signature–`getContextClassLoader()`–can be represented as a verbal phrase– *"get Context Class Loader"*. We then analyse such phrases across all the feedback documents, capture co-occurrences of terms within a fixed window ( $K=2$) from each phrase, and develop a source term graph. Thus, the above phrase adds four distinct nodes and three connecting edges – *"get"*$\leftrightarrow$*"context"*, *"context"*$\leftrightarrow$*"class"* and *"class"*$\leftrightarrow$*"loader"* – to the source term graph.

\[!tb\]

\[1\] $Q' \gets$ {} $C_R \gets$ getBugReportClass($R$) $R_{pp}\gets$ preprocess ($R$) $ST \gets$ getStackTraces ($R$) $G_{ST} \gets$ getTraceGraph ($ST$) $G_{PE} \gets$ getTextGraphs ($R_{pp}$) $R_F \gets$ getPseudoRelevanceFeedback ($R_{pp}$) $G_{NL} \gets$ getSourceTermGraph ($R_F$) $PR_{CK}\gets$ getPageRank ($G_{CK}$) $Q[C_R] \gets$ getTopKTerm(sortByWeight($PR_{CK}$)) $N_E \gets$ getExceptionName($R$) $M_E \gets$ getErrorMessage($R$) $Q' \gets$ {$N_E \cup  M_E \cup Q[C_R]$} $Q' \gets Q[C_R]$ $Q' \gets$ {$R_{pp} \cup Q[C_R]$} **return** Lucene($corpus$, $Q'$)

**Term Weighting using PageRank:** Once each body of texts ( stack traces, regular texts, source document) is transformed into a graph, we apply PageRank @pagerank [@rada; @saner2017masud; @ase2017masud] to the graph for identifying important keywords. PageRank was originally designed for web link analysis, and it determines the reputation of a web page based on the votes or recommendations ( hyperlinks) from other reputed pages on the web @pagerank. Similarly, in the context of our developed graphs, the algorithm determines importance of a node ( term) based on incoming links from other important nodes of the graph. In particular, it analyses the connectivity ( connected neighbours and their weights) of each term $V_i$ in the graph recursively, and then calculates the node's weight $TW(V_i)$: $$\label{eq:textrank}
\small
\setlength\abovedisplayskip{1pt}
\setlength\belowdisplayskip{1pt}
TW(V_{i})=(1-\phi)+\phi\sum_{j\epsilon In(V_{i})}\frac{TW(V_{j})}{|Out(V_{j})|}~~ (0 \le \phi \le1)$$ Here, $In(V_{i})$ refers to nodes providing incoming links to $V_i$, $Out(V_{j})$ refers to nodes that $V_{j}$ is connected to through outgoing links, and $\phi$ is the damping factor. @pagerank consider $\phi$ as the probability of staying on the web page and $1-\phi$ as the probability of jumping off the page by a random surfer. They use $\phi=0.85$ which was adopted by later studies @rada [@blanco; @saner2017masud], and we also do the same. We initialize each node in the graph with a value of 0.25 @rada, and recursively calculate their weights unless they converge below a certain threshold ( 0.0001) or the iteration count reaches the maximum ( 100) @rada. Once the calculation is over, we end up with an accumulated weight for each node (Step 5, Fig. \[fig:sysdiag\], Lines 16–20, Algorithm \[algo\]). Such weight of a node is considered as an estimation of relative importance of corresponding term among all the terms ( nodes) from the bug report ( graph).

**Reformulation of the Initial Query:** Once term weights are calculated, we rank the terms based on their weights, and select the Top-K (8$\le K \le$30, Fig. \[fig:query-length\]) terms for query reformulations. Since bug reports ( initial queries) from three classes have different degrees of structured information (or lack thereof), we carefully apply our reformulations to them (Steps 6, 7, Fig. \[fig:sysdiag\], Lines 21–30, Algorithm \[algo\]). In case of BR$_{ST}$ ( noisy query), we replace trace entries with the reformulation terms, extract the error message(s) containing exception name(s), and combine them as the reformulated query. For BR$_{NL}$ ( poor query), we combine preprocessed report texts with the highly weighted source code terms as the reformulated query. In the case of BR$_{PE}$, only Top-K weighted terms from the bug report are used as a reformulated query for bug localization.

Bug Localization
----------------

**Code Search:** Once a reformulated query is constructed, we submit the query to *Lucene* @refoqus [@trconfig]. Lucene is a widely adopted search engine for document search that combines Boolean search and VSM-based search methodologies ( TF-IDF @tfidf). In particular, we employ the Okapi BM25 similarity from the engine, use the reformulated query for the code search, and then collect the results (Step 8, Fig. \[fig:sysdiag\], Lines 31–32, Algorithm \[algo\]). These resultant and potentially buggy source code documents are then presented as a ranked list to the developer for manual analysis.

**Working Examples:** Table \[table:we\] shows our reformulated queries for the showcase bug reports in Table \[table:noisy\] ( BR$_{ST}$), Table \[table:poor\] ( BR$_{NL}$), and another example report from BR$_{PE}$ class. Baseline queries from these reports return their first correct results at the 53$^{rd}$ (for BR$_{ST}$), 27$^{th}$ (for BR$_{PE}$) and 30$^{th}$ (for BR$_{NL}$) positions of their corresponding ranked lists. On the contrary, BLIZZARD refines the noisy query from BR$_{ST}$ report, selects important keywords from BR$_{PE}$ report, and enriches the poor query from BR$_{NL}$ report by adding complementary terms from relevant source code. As a result, all three reformulated queries return their first correct results ( buggy source files) at the topmost ( first) positions, which demonstrate the potential of our technique for bug localization.

Experiment {#sec:experiment}
==========

We evaluate our proposed technique in several different dimensions using four widely used performance metrics and more than 5K bug reports (the queries) from six different subject systems. First, we evaluate in terms of the performance metrics and contrast with the baseline for different classes of bug reports/queries (Section \[sec:result\]). Second, we compare our approach with three state-of-the-art bug localization techniques (Section \[sec:comparison\]). Third, and possibly the most importantly, we also compare our approach with four state-of-the-art query reformulations techniques (Section \[sec:comparison\]). In particular, we answer four research questions using our experiments as follows:

-   **RQ$\mathbf{_1}$:** (a) How does BLIZZARD perform in bug localization, and (b) how do various parameters affect its performance?

-   **RQ$\mathbf{_2}$:** Do our reformulated queries perform better than the baseline search queries from the bug reports?

-   **RQ$\mathbf{_3}$:** Can BLIZZARD outperform the existing bug localization techniques including the state-of-the-art?

-   **RQ$\mathbf{_4}$:** Can BLIZZARD outperform the existing query reformulation techniques targeting concept/feature location and bug localization?

Experimental Dataset {#sec:dataset}
--------------------

**Dataset Collection:** We collect a total of 5,139 bug reports from six open source subject systems for our experiments. The dataset was taken from an earlier empirical study @icse2018. Table \[table:expds\] shows our dataset. First, all the resolved ( marked as RESOLVED) bug reports of each subject system were collected from the BugZilla and JIRA repositories given that they were submitted within a specific time interval (Table \[table:expds\]). Then the version control history of each system at GitHub was consulted to identify the bug-fixing commits @bugid. Such approach was regularly adopted by the relevant literature @twkraft [@buglocator; @stacktrace], and we also follow the same. In order to ensure a fair evaluation, we also discard such bug reports from our dataset for which no source code files ( Java classes) were changed or no relevant source files exist in the collected system snapshot.

**Goldset Development:** We collect *changeset* ( list of changed files) from each of our selected bug-fixing commits, and develop a *goldset*. Multiple changesets for the same bug were merged together. **Replication Package:** Our working prototype and experimental data are publicly available @blizzard for replication and reuse.

Performance Metrics {#sec:pmetrics}
-------------------

We use four performance metrics for the evaluation and comparison of our technique. Since these metrics were frequently used by the relevant literature @buglocator [@bluir; @stacktrace; @versionhistory; @wembedding; @saner2017masud], they are also highly appropriate for our experiments in this work.

**Hit@K:** It is defined as the percentage of queries for which at least one buggy file ( from the goldset) is correctly returned within the Top-K results. It is also called Recall@Top-K @bluir and Top-K Accuracy @saner2017masud in the literature.

**Mean Average Precision@K (MAP@K)**: Unlike regular precision, this metric considers the ranks of correct results within a ranked list. Precision@K calculates precision at the occurrence of each buggy file in the list. Average Precision@K (AP@K) is defined as the average of Precision@K for all the buggy files in a ranked list for a given query. Thus, Mean Average Precision@K is defined as the mean of Average Precision@K (AP@K) of all queries as follows: $$\label{eq:avep}
\setlength\abovedisplayskip{1pt}
\setlength\belowdisplayskip{1pt}
AP@K=\frac{\sum_{k=1}^{D}P_{k}\times buggy(k)}{|S|},~~ MAP@K=\frac{\sum_{q\epsilon Q}AP@K(q)}{|Q|}$$ Here, function $buggy(k)$ determines whether $k^{th}$ file (or result) is buggy ( returns 1) or not ( returns 0), $P_{k}$ provides the precision at $k^{th}$ result, and $D$ refers to the number of total results. $S$ is the gold set for a query, and $Q$ is the set of all queries. The bigger the MAP@K value is, the better a technique is.

**Mean Reciprocal Rank@K (MRR@K)**: Reciprocal Rank@K is defined as the multiplicative inverse of the rank of first correctly returned buggy file ( from gold set) within the Top-K results. Thus, Mean Reciprocal Rank@K (MRR@K) averages such measures for all queries in the dataset as follows: $$\setlength{\abovedisplayskip}{1pt}
\setlength{\belowdisplayskip}{1pt}
MRR@K(Q) =\frac{1}{|Q|}\sum_{q\in Q}{\frac{1}{firstRank(q)}}$$ Here, $firstRank(q)$ provides the rank of first buggy file within a ranked list. MRR@K can take a maximum value of 1 and a minimum value of 0. The bigger the MRR@K value is, the better a bug localization technique is.

**Effectiveness (E)**: It approximates a developer's effort in locating the first buggy file in the result list @stacktrace. That is, the measure returns the rank of first buggy file in the result list. The lower the effectiveness value is, the better a given query is,  the developer needs to check less amount of results from the top before reaching the actual buggy file in the list.

Experimental Results {#sec:result}
--------------------

We first show the performance of our technique in terms of appropriate metrics (RQ$_1$-(a)), then discuss the impacts of different adopted parameters upon the performance (RQ$_1$-(b)), and finally show our comparison with the baseline queries (RQ$_2$) as follows:

**Selection of Baseline Queries, and Establishment of Baseline Technique and Baseline Performance:** Existing studies suggest that text retrieval performances could be affected by query quality @refoqus, underlying retrieval engine @trconfig or even text preprocessing steps @stemming [@kevic]. Hence, we choose the baseline queries and baseline technique pragmatically for our experiments. We conduct a detailed study where three independent variables– bug report field ( title, whole texts), retrieval engine ( Lucene @refoqus, Indri @bluir) and text preprocessing step ( stemming, no stemming)–are alternated, and then we choose the best performing configuration as the baseline approach. In particular, we chose the preprocessed version ( performed stop word and punctuation removal, split complex tokens but avoided stemming) of the whole texts ( *title* + *description*) from a bug report as a baseline query. Lucene was selected as the baseline technique since it outperformed Indri on our dataset. The performance of Lucene with the baseline queries was selected as the baseline performance ( Table \[table:blizzard\]) for IR-based bug localization in this study. In short, our baseline is: (preprocessed whole texts + splitting of complex tokens + Lucene search engine).

**Answering RQ$\mathbf{_1}$(a) – Performance of BLIZZARD:** As shown in Table \[table:blizzard\], on average, our technique–BLIZZARD–localizes 74.52% of the bugs from a dataset of 5,139 bug reports with 47% mean average precision@10 and a mean reciprocal rank@10 of 0.50 which are 12%, 13% and 14% higher respectively than the baseline performance measures. That is, on average, our technique can return the first buggy file at the second position of the ranked list, almost half of returned files are buggy ( true positive) and it succeeds three out of four times in localizing the bugs. Furthermore, while the baseline technique is badly affected by the noisy ( BR$_{ST}$) and poor queries ( BR$_{NL}$), our technique overcomes such challenges with appropriate query reformulations, and provides significantly higher performances. For example, the baseline technique can localize 48% of the bugs from BR$_{ST}$ dataset ( noisy queries) with only 28% precision when Top-10 results are considered. On the contrary, our technique localizes 75% of the bugs with 46% precision in the same context which are 56% and 62% higher respectively than the corresponding baseline measures. Such improvements are about 7% for BR$_{NL}$,  poor queries. In the cases where bug reports contain program entities,  BR$_{PE}$, and the baseline performance measures are already pretty high, our technique further refines the query and provides even higher performances. For example, BLIZZARD improves both baseline MRR@10 and baseline MAP@10 for BR$_{PE}$ dataset by 10% which is promising.

![Comparison of BLIZZARD with baseline technique in terms of (a) MAP@K and (b) MRR@K<span data-label="fig:compare-baseline"></span>](mapk-mrrk-compare-v3){width="3in"}

![Impact of query reformulation length on the MAP@10 of our technique–BLIZZARD<span data-label="fig:query-length"></span>](query-length-check){width="3in"}

![Quality improvement of (a) noisy and (b) poor baseline queries by our technique–BLIZZARD<span data-label="fig:baseline-compare-qe"></span>](qe-changed-v2){width="3.35in"}

Fig. \[fig:compare-baseline\] further demonstrates the comparative analyses between BLIZZARD and the baseline technique for various Top-K results in terms of (a) precision and (b) reciprocal rank in the bug localization. From Fig. \[fig:compare-baseline\]-(a), we see that precision reaches to the maximum pretty quickly ( at $K \approx 4$) for both techniques. While the baseline technique suffers from noisy ( from BR$_{ST}$) and poor ( from BR$_{NL}$) queries, BLIZZARD achieves significantly higher precision than the baseline. Our non-parametric statistical tests–*Mann-Whitney Wilcoxon* and *Cliff's Delta*–reported *p-values*$<0.05$ with a *large* effect size (  $0.77\le \Delta \le 1.00$). Although the baseline precision for BR$_{PE}$ is higher, BLIZZARD offers even higher precision. From Fig. \[fig:compare-baseline\]-(b), we see that mean reciprocal ranks of BLIZZARD have a logarithmic shape and whereas the baseline counterparts look comparatively flat. That is, as more results from the top of the ranked list are considered, more true positives are identified by our technique than the baseline technique does. Statistical tests also reported strong significance ( *p-values*$<$0.001) and a *large* effect size ( 0.62$\le$$\Delta$$ \le$1.00) of our measures over the baseline counterparts. That is, BLIZZARD performs a good job in reformulating the noisy and poor queries, and such reformulations contribute to a significant improvement in the bug localization performances.

**Answering RQ$\mathbf{_1}$(b) –Impact of Parameters and Settings:** We investigate the impacts of different adopted parameters -*query reformulation length*, *word stemming*, and *retrieval engine* - upon our technique, and justify our choices. BLIZZARD reformulates a given query ( bug report) for bug localization, and hence, size of the reformulated query is an important parameter. Fig. \[fig:query-length\] demonstrates how various reformulation lengths can affect the MAP@10 of our technique. We see that precision reaches the maximum for three report classes at different query reformulation lengths ( R$_L$). For BR$_{ST}$, we achieve the maximum precision at R$_L$=11, and for BR$_{NL}$, such maximum is detected with R$_L$ ranging between 8 and 12. On the contrary, precision increases in a logarithmic manner for BR$_{PE}$ bug reports. We investigated up to 30 reformulation terms and found the maximum precision. Given the above empirical findings, we chose R$_L$=11 for BR$_{ST}$, R$_L$=30 for BR$_{PE}$ and R$_L$=8 for R$_{NL}$ as the adopted reformulation lengths and our choices are justified.

We also investigate the impact of stemming and text retrieval engine on our technique. We found that stemming did not improve the performance of BLIZZARD,  reduced localization accuracy. Similar finding was reported by earlier studies as well @stemming [@kevic]. We also found that Lucene performs better than Indri on our dataset. Besides, Lucene has been widely used by relevant literature @refoqus [@trconfig; @stacktrace; @saner2017masud]. Given the above findings and earlier suggestions, our choices on stemming and retrieval engine are also justified.

.3em

BLIZZARD outperforms baseline in accuracy, precision and reciprocal rank by 7%–56%, 6%–62% and 6%–62% respectively across three report groups, and our adopted parameters are also justified.

**Answering RQ$\mathbf{_2}$-Comparison with Baseline Queries:** While Table \[table:blizzard\] contrasts BLIZZARD with the baseline approach for top 1 to 10 results, we further investigate how BLIZZARD performs compared to the baseline when all results of a query are considered. We compare our queries with two baseline queries –*title* ( BL$_T$), *title*+*description* ( BL) – from each of the bug reports. When our query returns the first correct result at a higher position in the result list than that of corresponding baseline query, we call it *query improvement* and vice versa *query worsening*. When result ranks of the reformulated query and the baseline query are the same, then we call it *query preserving*. From Table \[table:baseline-qe\], we see that our applied reformulations improve 59% of the noisy queries ( BR$_{ST}$) and 39%–56% of the poor ( BR$_{NL}$) queries both with $\approx$ 25% worsening ratios. That is, the improvements are more than two times the worsening ratios. Fig. \[fig:baseline-compare-qe\] further demonstrates the potential of our reformulations where improvement, worsening and preserving ratios are plotted for each of the six subject systems. We see that noisy queries get benefited greatly from our reformulations, and on average, their query effectiveness improve up to 122 positions ( MRD of BR$_{ST}$, Table \[table:baseline-qe\]) in the result list. Such improvement of ranks can definitely help the developers in locating the buggy files in the result list more easily. The poor queries also improve due to our reformulations significantly ( *p-value*=0.004$<$0.05, *Cliff's* $\Delta$=0.94 (*large*)), and the correct results can be found 16 positions earlier (than the baseline) in the result list starting from the top. Quantile analysis in Table \[table:compare-qe\] also confirms that noisy and poor queries are significantly improved by our provided reformulations. Besides, the benefits of query reformulations are also demonstrated by our findings in Table \[table:blizzard\] and Fig. \[fig:compare-baseline\]. .5em

Our applied reformulations to the bug localization queries improve 59% of the noisy queries and 39%–56% of the poor queries, and return the buggy files closer to the top of result list. Such improvements can reduce a developer's effort in locating bugs.

\[!t\]

Comparison with Existing Techniques {#sec:comparison}
-----------------------------------

**Answering RQ$\mathbf{3}$ –Comparison with Existing IR-Based Bug Localization Techniques:** Our evaluation of BLIZZARD with four widely used performance metrics shows promising results. The comparison with the best performing baseline shows that our approach outperforms the baselines. However, in order to further gain confidence and to place our work in the literature, we also compared our approach with three IR-based bug localization techniques @buglocator [@bluir; @versionhistoryjsep] including the state-of-the-art @versionhistoryjsep. @buglocator first employ improved Vector Space Model ( rVSM) and bug report similarity for locating buggy source files for a new bug report. @bluir employ structured information retrieval where (1) a bug report is divided into two fields–*title*, *description* and a source document is divided into four fields–*class*, *method*, *variable* and *comments*, and then (2) eight similarity measures between these two groups are accumulated to rank the source document. We collect authors' implementations of both techniques for our experiments.

While the above studies use bug report contents only, the later approaches combine them @bluirplus and add more internal @brtracer or external information sources such as version control history @versionhistory and author information @versionhistoryjsep. In the same vein, @versionhistoryjsep recently combine five internal and external information sources - similar bug report, structured IR, stack traces, version control history and bug reporter's history – for ranking a source document, and outperform five earlier approaches which makes it the state-of-the-art in IR-based bug localization. Given that authors' implementation is not publicly available, we implement this technique ourselves by consulting with the original authors. Since BLIZZARD does not incorporate any external information sources, to ensure a fair comparison, we also implement a variant of the state-of-the-art namely AmaLgam+$_{BRO}$. It combines bug report texts, structured IR and stack traces ( Table \[table:components\]) for source document ranking. From Table \[table:compare\], we see that AmaLgam+ performs the best among the existing techniques. However, its performance comes at a high cost of mining six information contents ( Table \[table:components\]). Besides, for optimal performance, AmaLgam+ needs past bug reports, version control history and author history which might always not be available. Thus, to ensure a fair comparison, we develop two variants of our technique–BLIZZARD$_{BRO}$ and BLIZZARD+. BLIZZARD$_{BRO}$ combines query reformulation with bug report only features whereas BLIZZARD+ combines query reformulation with all ranking components of AmaLgam+ ( details in Table \[table:components\]). We then compare both BLIZZARD and BLIZZARD$_{BRO}$ with AmaLgam+$_{BRO}$, and BLIZZARD+ with AmaLgam+ respectively. As shown in Table \[table:compare\], BLIZZARD outperforms AmaLgam+$_{BRO}$ in terms of all three metrics especially for BR$_{PE}$ reports while performing moderately high with other report groups. For example, BLIZZARD provides 22% higher MRR@10 and 24% higher MAP@10 than AmaLgam+$_{BRO}$ for BR$_{PE}$. When all report only features are complemented with appropriate query reformulations, our technique, BLIZZARD$_{BRO}$ outperforms AmaLgam+$_{BRO}$ in terms of all three metrics–Hit@K, MAP@10 and MRR@10– with each report groups. Such findings suggest that BLIZZARD$_{BRO}$ can better exploit the available resources ( bug report contents) than the state-of-the-art variant, and returns the buggy files at relatively higher positions in the ranked list. Furthermore, BLIZZARD+ outperforms the state-of-the-art, AmaLgam+, by introducing query reformulation paradigm. For example, BLIZZARD+ improves Hit@5 and Hit@10 over AmaLgam+ for each of the three query types,  15% and 12% respectively for noisy queries (BR$_{ST}$). It also should be noted that none of the existing techniques is robust to all three report groups simultaneously. We overcome such issue with appropriate query reformulations, and deliver $\approx$75%–80% Hit@10 irrespective of the bug report quality. From Table \[table:components\], we see that BLIZZARD$_{BRO}$ provides 20% higher MRR@10 than AmaLgam+$_{BRO}$ by consuming equal amount of resources,  bug report only. All these findings above suggest two important points. First, earlier studies might have failed to exploit the report contents and structures properly for bug localization. Second, query reformulation has a high potential for improving the IR-based bug localization.

![Comparison of (a) MAP@K and (b) Hit@K with the state-of-the-art IR-based bug localization techniques<span data-label="fig:topk-mapk-st"></span>](mapk-hitk-compare-v3){width="3in"}

![Comparison of Hit@10 across all subject systems<span data-label="fig:subject-acc"></span>](system-hit10-compare-v2){width="3.35in"}

Fig. \[fig:topk-mapk-st\] demonstrates a comparison of BLIZZARD with the existing techniques in terms of (a) MAP@K and (b) Hit@K for various Top-K results. Our statistical tests report that BLIZZARD, BLIZZARD$_{BRO}$ and BLIZZARD+ outperform AmaLgam+$_{BRO}$ and AmaLgam+ respectively in MAP@K by a significant margin ( *p-values*$\le$0.001) and *large* effect size ( 0.82$\le$$\Delta$$\le$1.00). Similar findings were also achieved for Hit@K.

Fig. \[fig:subject-acc\] and Fig. \[fig:subject-mrrk\] focus on subject system specific performances. From Fig. \[fig:subject-acc\], we see that BLIZZARD outperforms AmaLgam+$_{BRO}$ with four systems in Hit@10, and falls short with two systems. However, BLIZZARD$_{BRO}$ and BLIZZARD+ outperform AmaLgam+$_{BRO}$ and AmaLgam+ respectively for all six systems. As shown in the box plots of Fig. \[fig:subject-mrrk\], BLIZZARD has a higher median in MRR@10 and MAP@10 than AmaLgam+$_{BRO}$ across all subject systems. AmaLgam+ improves both measures especially lauraadams@example.com. However, BLIZZARD+ provides even higher MRR@10 and MAP@10 than any of the existing techniques including the state-of-the-art.

.3em

Our technique outperforms the state-of-the-art from IR-based bug localization in various dimensions. It offers 20% higher precision and reciprocal rank than that of state-of-the-art variant ( AmaLgam+$_{BRO}$) by using only query reformulation rather than costly alternatives,  mining of version control history

![Comparison of (a) MRR@10 and (b) MAP@10 with existing techniques across subject systems<span data-label="fig:subject-mrrk"></span>](mrrk-mapk-compare-boxplot-v3){width="3in"}

**Answering RQ$\mathbf{_4}$ –Comparison with Existing Query Reformulation Techniques:** While we have already showed that our approach outperforms the baselines and the state-of-the-art IR-based bug localization approaches, we also wanted to further evaluate our approach in the context of query reformulation. We thus compared BLIZZARD with four query reformulation techniques @rocchio [@refoqus; @sisman; @saner2017masud] including the state-of-the-art @saner2017masud that were mostly used for concept/feature location. We use authors' implementation of the state-of-the-art, STRICT, and re-implement the remaining three techniques. We collect Query Effectiveness ( rank of the first correct result) of each of the reformulated queries provided by each technique, and compare with ours using quantile analysis. From Table \[table:compare-qe\], we see that 48% of the noisy ( BR$_{ST}$) queries are improved by STRICT, and 32% of the poor ( BR$_{NL}$) queries are improved by @sisman. Neither of these techniques considers bug report quality ( prevalence of structured information or lack thereof) and each technique applies the same reformulation strategy to all reports. On the contrary, BLIZZARD chooses appropriate reformulation based on the class of a bug report, and improves 59% of the noisy queries and 39% of the poor queries which are 22% and 20% higher respectively. When compared using quantile analysis, we see that our quantiles are highly promising compared to the baseline. Our reformulations clearly improve the noisy queries, and 75% of the improved queries return their first correct results within Top-9 ( Q$_3$=9) positions whereas STRICT needs Top-17 positions for the same. In the case of poor queries, quantiles of BLIZZARD are comparable to that of @sisman. However, BLIZZARD worsens less and preserves higher amount of the baseline queries which demonstrate its high potential.

BLIZZARD outperforms the state-of-the-art in query reformulation using context-aware ( responsive to report quality) query reformulation. Whatever improvements are offered to noisy and poor queries by the state-of-the-art, our technique improves 22% more of noisy queries and 20% more of the poor queries.

Threats to Validity {#sec:threat}
===================

Threats to internal validity relate to experimental errors and biases @wordsim. Replication of existing studies and misclassification of the bug reports are possible sources of such threats. We use authors' implementation of three techniques @buglocator [@bluir; @saner2017masud] and re-implement the remaining four. While we cannot rule out the possibility of any implementation errors, we re-implemented them by consulting with the original authors @versionhistoryjsep and their reported settings and parameters @refoqus [@rocchio; @sisman]. While our technique employs appropriate regular expressions for bug report classification, they are limited in certain contexts ( ill-structured stack traces) which require limited manual analysis currently. More sophisticated classification approaches @brcls [@brclslda; @defcat] could be applied in the future work. Threats to external validity relate to generalizability of a technique @wordsim. We conduct experiments using Java systems. However, since we deal with mostly structured items ( stack traces, program entities) from a bug report, our technique can be adapted to other OOP-based systems that have such structured items.

Related Work {#sec:related}
============

**Bug Localization:** Automated bug localization has been an active research area for over two decades @bluir. Existing studies from the literature can be roughly categorized into two broad families–*spectra* based and *information retrieval (IR)* based @locombined [@parninireval]. We deal with IR-based bug localization in this work. Given that spectra based techniques are costly and lack scalability @parninireval [@stacktrace], several studies adopt IR-based methods such as Latent Semantic Indexing (LSI) @lsiPoshyvanyk, Latent Dirichlet Allocation (LDA) @ldabug [@raobug] and Vector Space Model (VSM) @buglocator [@bluir; @kak; @stacktrace; @kimtwophase; @brtracer] for bug localization. They leverage the shared vocabulary between bug reports and source code entities for bug localization. Unfortunately, as existing evidences @parninireval [@icse2018] suggest, they are inherently subject to the quality of bug reports. A number of recent studies complement traditional IR-based localization with spectra based analysis @locombined, machine learning @hyloc [@learn2rank] and mining of various repositories– bug report history @bluirplus, version control history @kak [@versionhistory], code change history @blia [@locus] and bug reporter history @versionhistoryjsep. Recently, @versionhistoryjsep combine bug report contents and three external repositories, and outperform five earlier IR-based bug localization techniques @buglocator [@bluir; @bluirplus; @brtracer; @versionhistory; @kak] which makes it the state-of-the-art. In short, the contemporary studies advocate for combining (1) multiple localization approaches ( dynamic trace analysis @locombined, Deep learning @hyloc, learning to rank @learn2rank [@wembedding]) and (2) multiple external information sources with classic IR-based localization, and thus, improve the localization performances. However, such solutions could be costly ( multiple repository mining) and less scalable ( dependency on external information sources), and hence, could be infeasible to use in practice. In this work, we approach the problem differently, and focus on better leveraging the potential of the resources at hand ( bug report and source code) which might have been *underestimated* by the earlier studies. In particular, we refine the noisy queries ( containing stack traces) and complement the poor queries ( lacks structured items), and offer an effective information retrieval unlike the earlier studies. Thus, issues raised by low quality bug reports @parninireval have been significantly addressed by our technique, and our experimental findings support such conjecture. We compare with three existing studies including the state-of-the-art @versionhistoryjsep, and the detailed comparison can be found in Section \[sec:comparison\] ( RQ$_3$).

A few studies @stacktrace [@brtracer] analyse stack traces from a bug report for bug localization. However, they apply the trace entries to boost up source document ranking, and superfluous trace entries were not discarded from their stack traces. Learning-to-rank @learn2rank [@wembedding] and Deep learning @hyloc based approaches might also suffer from noisy and poor queries since they adopt classic IR without query reformulation in their document ranking. Recent studies @wembedding [@drewbl] employ distributional semantics of words to address limitations of VSM. Since noisy terms in the report could be an issue, our approach can complement these approaches through query reformulation.

**Query Reformulation:** There exist several studies @gayg [@qperf; @saner2017masud; @ase2016masud; @hillicse09; @refoqus; @kevic; @wembedding; @rastkar; @verbose] that support concept/feature/concern location tasks using query reformulation. However, these approaches mostly deal with unstructured natural language texts. Thus, they might not perform well with bug reports containing excessive structured information ( stack traces), and our experimental findings also support this conjecture (Table \[table:compare-qe\]). @sisman first introduce query reformulation in the context of IR-based bug localization. However, their approach cannot remove noise from a query. Recently, @observed identify observed behaviour (OB), expected behaviour (EB) and steps to reproduce (S2R) from a bug report, and then use OB texts as a reformulated query for bug localization. However, they only analyse unstructured texts whereas we deal with both structured and unstructured contents. Since we apply query reformulation, we compare with four recent query reformulation techniques employed for concept location–Rocchio @rocchio, RSV @rsv, STRICT @saner2017masud @ase2017masud and bug localization– SCP @sisman. The detailed comparison can be found in Section \[sec:comparison\] ( RQ$_4$).

In short, existing IR-based techniques suffer from *quality issues* of bug reports whereas traditional query reformulation techniques are *not well-adapted* for the bug reports containing excessive structured information ( stack traces). Our work fills this *gap* of the literature by incorporating context-aware ( report quality aware) query reformulation into the IR-based bug localization. Our technique better exploits resources at hand and delivers equal or higher performance than the state-of-the-art at a relatively lower cost. To the best of our knowledge, such comprehensive solution was not provided by any of the existing studies.

Conclusion and Future Work {#sec:conclusion}
==========================

Traditional IR-based bug localization is inherently subject to the (low) quality of submitted bug reports. In this paper, we propose a novel technique that leverages the quality aspect of bug reports, incorporates context-aware query reformulation into the bug localization, and thus, overcomes such limitation. Experiments using 5,139 bug reports from six open source systems report that BLIZZARD can offer up to 62% and 20% higher precision than the best baseline technique and the state-of-the-art respectively. Our technique also improves 22% more of noisy queries and 20% more of the poor queries than that of state-of-the-art. In future, we plan to apply our learned insights and our technique to further complex activities during debugging such as automatic bug fixing. **Acknowledgement:** This research was supported by Saskatchewan Innovation & Opportunity Scholarship (2017–2018), and the Natural Sciences and Engineering Research Council of Canada (NSERC).

{'source': 'arXiv_out/1808'}