N EURAL P ROGRAM R EPAIR BY J OINTLY L EARNING TO L OCALIZE AND R EPAIR
A BSTRACT
Due to its potential to improve programmer productivity and software quality, automated program repair has been an active topic of research.
Newer techniques harness neural networks to learn directly from examples of buggy programs and their ﬁxes.
In this work, we consider a recently identiﬁed class of bugs called variable-misuse bugs.
The state-of-the-art solution for variable misuse enumerates potential ﬁxes for all possible bug locations in a program, before selecting the best prediction.
We show that it is beneﬁcial to train a model that jointly and directly localizes and repairs variable-misuse bugs.
We present multi-headed pointer networks for this purpose, with one head each for localization and repair.
The experimental results show that the joint model signiﬁcantly outperforms an enumerative solution that uses a pointer based model for repair alone.
I NTRODUCTION
Advances in machine learning and the availability of large corpora of source code have led to growing interest in the development of neural representations of programs for performing program analyses.
In particular, different representations based on token sequences (Gupta et al., 2017; Bhatia et al., 2018), program parse trees (Piech et al., 2015; Mou et al., 2016), program traces (Reed & de Freitas, 2015; Cai et al., 2017; Wang et al., 2018), and graphs (Allamanis et al., 2018) have been proposed for a variety of tasks including repair (Devlin et al., 2017b; Allamanis et al., 2018), optimization (Bunel et al., 2017), and synthesis (Parisotto et al., 2017; Devlin et al., 2017a).
In recent work, Allamanis et al.
(2018) proposed the problem of variable misuse (V AR M ISUSE ): given a program, ﬁnd program locations where variables are used, and predict the correct variables that should be in those locations.
A V AR M ISUSE bug exists when the correct variable differs from the current one at a location.
Allamanis et al.
(2018) show that variable misuses occur in practice, e.g., when a programmer copies some code into a new context, but forgets to rename a variable from the older context, or when two variable names within the same scope are easily confused.
Figure 1a shows an example derived from a real bug.
The programmer copied line 5 to line 6, but forgot to rename object name to subject name .
Figure 1b shows the correct version.
Allamanis et al.
(2018) proposed an enumerative solution to the V AR M ISUSE problem.
They train a model based on graph neural networks that learns to predict a correct variable (among all type-
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] correct variables available in the scope) for each slot in a program.
A slot is a placeholder in a program where a variable is used.
The model is trained on a synthetic dataset containing a training example for each slot in the programs from a corpus of correct source ﬁles, and teaching the model to predict the correct, existing variable for each slot.
At inference time, a program of unknown correctness is turned into n prediction tasks, one for each of its n slots.
Each prediction task is then performed by the trained model and predictions of high probability that differ from the existing variable in the corresponding slot are provided to the programmer as likely V AR M ISUSE bugs.
Unfortunately, this enumerative strategy has some key technical drawbacks.
First, it approximates the repair process for a given program by enumerating over a number of independent prediction problems, where important shared context among the dependent predictions is lost.
Second, in the training process, the synthetic bug is always only at the position of the slot.
If for example, the program in Figure 1b were used for training, then ﬁve training examples, one corresponding to each identiﬁer in a blue box (a variable read, in this case), would be generated.
In each of them, the synthetic bug is exactly at the slot position.
However, during inference, the model generates one prediction problem for each variable use in the program.
In only one of these prediction problems does the slot coincide with the bug location; in the rest, the model now encounters a situation where there is a bug somewhere else, at a location other than the slot .
This differs from the cases it has been trained on.
For example, in Figure 1a, the prediction problem corresponding to the slot on line 5 contains a bug elsewhere (at line 6) and not in the slot.
Only the problem corresponding to the slot on line 6 would match how the model was trained.
This mismatch between training and test distributions hampers the prediction accuracy of the model.
In our experiments, it leads to an accuracy drop of 4% to 14% , even in the non-enumerative setting, i.e., when the exact location of the bug is provided.
Since the enumerative approach uses the prediction of the same variable as the original variable for declaring no bugs at that location, this phenomenon contributes to its worse performance.
Another drawback of the enumerative approach is that it produces one prediction per slot in a program, rather than one prediction per program.
Allamanis et al.
(2018) deal with this by manually selecting a numerical threshold and reporting a bug (and its repair) only if the predicted probability for a repair is higher than that threshold.
Setting a suitable threshold is difﬁcult: too low a threshold can increase false positives and too high a threshold can cause false negatives.
In order to deal with these drawbacks, we present a model that jointly learns to perform: 1) classiﬁcation of the program as either faulty or correct (with respect to V AR M ISUSE bugs), 2) localization of the bug when the program is classiﬁed as faulty, and 3) repair of the localized bug.
One of the key insights of our joint model is the observation that, in a program containing a single V AR M ISUSE bug, a variable token can only be one of the following: 1) a buggy variable (the faulty location), 2) some occurrence of the correct variable that should be copied over the incorrect variable into the faulty location (a repair location), or 3) neither the faulty location nor a repair location.
This arises from the fact that the variable in the fault location cannot contribute to the repair of any other variable – there is only one fault location – and a variable in a repair location cannot be buggy at the same time.
This observation leads us to a pointer model that can point at locations in the input (Vinyals et al., 2015) by learning distributions over input tokens.
The hypothesis that a program that contains a bug at a location likely contains ingredients of the repair elsewhere in the program (Engler et al., 2001) has been used quite effectively in practice (Le Goues et al., 2012).
Mechanisms based on pointer networks can play a useful role to exploit this observation for repairing programs.
We formulate the problem of classiﬁcation as pointing to a special no-fault location in the program.
To solve the joint prediction problem of classiﬁcation, localization, and repair, we lift the usual pointer-network architecture to multi-headed pointer networks, where one pointer head points to the faulty location (including the no-fault location when the program is predicted to be non-faulty) and another to the repair location.
We compare our joint prediction model to an enumerative approach for repair.
Our results show that the joint model not only achieves a higher classiﬁcation, localization, and repair accuracy, but also results in high true positive score.
Furthermore, we study how a pointer network on top of a recurrent neural network compares to the graph neural network used previously by Allamanis et al.
The comparison is performed for program repair given an a priori known bug location, the very same task used by that work.
Limited to only syntactic inputs, our model outperforms the graph-based one by 7 percentage points.
Although encouraging, this comparison is only limited to syntactic inputs; in contrast, the graph model uses both syntax and semantics to achieve state-of-the-art repair accuracy.
In future work we plan to study how jointly predicting bug location and repair might improve the graph model when bug location is unknown, as well as how our pointer-network-based model compares to the graphbased one when given semantics, in addition to syntax; the latter is particularly interesting, given the relatively simpler model architecture compared to message-passing networks (Gilmer et al., 2017).
In summary, this paper makes the following key contributions: 1) it presents a solution to the general variable-misuse problem in which enumerative search is replaced by a neural network that jointly localizes and repairs faults; 2) it shows that pointer networks over program tokens provide a suitable framework for solving the V AR M ISUSE problem; and 3) it presents extensive experimental evaluation over multiple large datasets of programs to empirically validate the claims.
R ELATED W ORK
Allamanis et al.
(2018) proposed an enumerative approach for solving the V AR M ISUSE problem by making individual predictions for each variable use in a program and reporting back all variable discrepancies above a threshold, using a graph neural network on syntactic and semantic information.
We contrast this paper to that work at length in the previous section.
Devlin et al.
(2017b) propose a neural model for semantic code repair where one of the classes of bugs they consider is V AR R EPLACE , which is similar to the V AR M ISUSE problem.
This model also performs an enumerative search as it predicts repairs for all program locations and then computes a scoring of the repairs to select the best one.
As a result, it also suffers from a similar training/test data mismatch issue as Allamanis et al.
Similar to us, they use a pooled pointer model to perform the repair task.
However, our model uses multi-headed pointers to perform classiﬁcation, localization, and repair jointly.
DeepFix (Gupta et al., 2017) and SynFix (Bhatia et al., 2018) repair syntax errors in programs using neural program representations.
DeepFix uses an attention-based sequence-to-sequence model to ﬁrst localize the syntax errors in a C program, and then generates a replacement line as the repair.
SynFix uses a Python compiler to identify error locations and then performs a constraint-based search to compute repairs.
In contrast, we use pointer networks to perform a ﬁne-grained localization to a particular variable use, and to compute the repair.
Additionally, we tackle variable misuses, which are semantic bugs, whereas those systems ﬁx only syntax errors in programs.
The DeepBugs (Pradel & Sen, 2018) paper presents a learning-based approach to identifying namebased bugs.
The main idea is to represent program expressions using a small set of features (e.g., identiﬁer names, types, operators) and then compute their vector representations by concatenating the individual feature embeddings.
By injecting synthetic bugs, the classiﬁer is trained to predict program expressions as buggy or not for three classes of bugs: swapped function arguments, wrong binary operator, and wrong operand in a binary operation.
Similar to previous approaches, it is also an instance of an enumerative approach.
Unlike DeepBugs, which embeds a single expression, our model embeds the full input program (up to a maximum preﬁx length) and performs both localization and repair of the V AR M ISUSE bugs in addition to the classiﬁcation task.
Moreover, our model implements a pointer mechanism for representing repairs that often requires pointing to variable uses in other parts of the program that are not present in the same buggy expression.
Sk p (Pu et al., 2016) is another enumerative neural program repair approach to repair student programs using an encoder-decoder architecture.
Given a program, for each program statement s i , the decoder generates a statement s (cid:48)
i conditioned on an encoding of the preceding statement s i − 1 and the following statement s i +1 .
Unlike our approach, which can generate V AR M ISUSE repairs using a pointer mechanism, the Sk p model would need to predict full program statements for repairing such bugs.
Moreover, similar to the DeepBugs approach, it would be difﬁcult for the model to predict repairs that include variables deﬁned two or more lines above the buggy variable location.
Automated program repair (APR) has been an area of active research in software engineering.
The traditional APR approaches (Gazzola et al., 2018; Monperrus, 2018; Motwani et al., 2018) differ from our work in the following ways: 1)
They require a form of speciﬁcation of correctness to repair a buggy program, usually as a logical formula/assertion, a set of tests, or a reference implementation; 2) They depend on hand-designed search techniques for localization and repair; 3)
The techniques are applied to programs that violate their speciﬁcations (e.g., a program that fails some tests), which means that the programs are already known to contain bugs.
In contrast, a recent line of research in APR is based on end-to-end learning, of which ours is an instance.
Our solution (like some other learning-based repair solutions) has the following contrasting features: 1) It does not require any speciﬁcation of correctness, but learns instead to ﬁx a common class of errors directly from sourcecode examples; 2) It does not perform enumerative search for localization or repair—we train a neural network to perform localization and repair directly; 3) It is capable of ﬁrst classifying whether a program has the speciﬁc type of bug or not, and subsequently localizing and repairing it.
The APR community has also designed some repair bechmarks, such as ManyBugs and IntroClass (Goues et al., 2015), and Defects4J (Just et al., 2014), for test-based program repair techniques.
The bugs in these benchmarks relate to the expected speciﬁcation of individual programs (captured through test cases) and the nature of bugs vary from program to program.
These benchmarks are therefore suitable to evaluate repair techniques guided by test executions.
Learning-based solutions like ours focus on common error types, so it is possible for a model to generalize across programs, and work directly on embeddings of source code.
P OINTER M ODELS FOR L OCALIZATION AND R EPAIR OF V AR M ISUSE
We use pointer-network models to perform joint prediction of both the location and the repair for V AR M ISUSE bugs. We exploit the property of V AR M ISUSE that both the bug and the repair variable must exist in the original program.
The model ﬁrst uses an LSTM (Hochreiter & Schmidhuber, 1997) recurrent neural network as an encoder of the input program tokens.
The encoder states are then used to train two pointers: the ﬁrst pointer corresponds to the location of the bug, and the second pointer corresponds to the location of the repair variable.
The pointers are essentially distributions over program tokens.
The model is trained end-to-end using a dataset consisting of programs assumed to be correct.
From these programs, we create both synthetic buggy examples, in which a variable use is replaced with an incorrect variable, and bug-free examples, in which the program is used as is.
For a buggy training example, we capture the location of the bug and other locations where the original variable is used as the labels for the two pointers.
For a bug-free training example, the location pointer is trained to point to a special, otherwise unused no-fault token location in the original program.
In this paper, we focus on learning to localize a single V AR M ISUSE bug, although the model can naturally generalize to ﬁnding and repairing more bugs than one, by adding more pointer heads.
3. P ROBLEM D EFINITION
We ﬁrst deﬁne our extension to the V AR M ISUSE problem, which we call the V AR M ISUSE R EPAIR problem.
We deﬁne the problem with respect to a whole program’s source code, although it can be deﬁned for different program scopes: functions, loop bodies, etc.
We consider a program f as a sequence of tokens f = (cid:104) t 1 , t 2 , · · · , t n (cid:105) , where tokens come from a vocabulary T , and n is the number of tokens in f .
The token vocabulary T consists of both keywords and identiﬁers.
Let V ⊂ T denote the set of all tokens that correspond to variables (uses, deﬁnitions, function arguments, etc.).
For a program f , we deﬁne V f def ⊆ V as the set of all variables deﬁned in f , including function arguments; this is the set of all variables that can be used within the scope, including as repairs for a putative V AR M ISUSE bug.
Let V f use ⊆
V × L denote the set of all (token, location) pairs corresponding to variable uses, where L denotes the set of all program locations.
Given a program f , the goal in the V AR M ISUSE R EPAIR problem is to either predict if the program is already correct (i.e., contains no V AR M ISUSE bug) or to identify two tokens: 1) the location token ( t i , l i ) ∈
V f use , and 2) a repair token t j ∈ V f def .
The location token corresponds to the location of the V AR M ISUSE bug, whereas the repair token corresponds to any occurrence of the correct variable in the original program (e.g., its deﬁnition or one of its other uses), as illustrated in Figure 2a.
In the example from Figure 2a, T contains all tokens, including variables, literals, keywords; V contains sources , object name , subject name , and result ; V f use contains the uses of variables, e.g., sources at its locations on lines 2 and 3, and V f def is the same as V in this example.
1 def validate_sources(sources): 2 object_name = get_content(sources, ’obj’) 3 subject name =
get_content(sources, ’subj’)
4 result =
Result() 5 result.objects.append(object_name)) 6 result.subjects.append(object name) 7 return result location repair ( def f ( a , b ) : ......
t t t t t t t t tn
Embedding Layer e
e e e e e e e en
LSTM h h
h h
h h h h hn        ......
Program Tokens:
Program Mask:
Attention Mechanism α[0] (location) α[1] (repair)
[Figure]
details
[Figure]
model.
3. M ULTI - HEADED P OINTER M ODELS
We now deﬁne our pointer network model (see Figure 2b).
Given a program token sequence f = (cid:104) t 1 , t 2 , · · · , t n (cid:105) , we embed the tokens t i using a trainable embedding matrix φ : T → R d , where d is the embedding dimension.
We then run an LSTM over the token sequence to obtain hidden states (dimension h ) for each embedded program token.
[Equation]
Let m ∈ { 0 , 1 } n be a binary vector such that m [ i ] =
if the token t i ∈ V f def ∪ V f use , otherwise m [ i ] = 0 .
The vector m acts as a masking vector to only consider hidden states that correspond to states of the variable tokens.
Let H ∈ R h × n denote a matrix consisting of the hidden-state vectors of the LSTM obtained after masking, i.e., H = m (cid:12) [ h 1 · · · h n ] .
We then perform attention over the hidden states using a mechanism similar to that of Rockt ¨ aschel et al.
(2016) as follows:
[Equation]
where W 1 , W 2 ∈ R h × h are trainable projection matrices and 1 n ∈ R n is a vector of ones used to obtain n copies of the ﬁnal hidden state h n .
We then use another trained projection matrix W ∈ R h × 2 to compute the α ∈ R 2 × n as follows:
[Equation]
The attention matrix α corresponds to two probability distributions over the program tokens.
The ﬁrst distribution α (cid:124) [0] ∈ R n corresponds to location token indices and the second distribution α (cid:124) [1] ∈ R n corresponds to repair token indices.
We experiment with and without using the masking vector on the hidden states, and also using masking on the unnormalized attention values.
3. T RAINING THE M ODEL
We train the pointer model on a synthetically generated training corpus consisting of both buggy and non-buggy Python programs.
Starting from a publicly available dataset of Python ﬁles, we construct the training, validation, and evaluation datasets in the following manner.
We ﬁrst collect the source code for each program deﬁnition from the Python source ﬁles.
For each program deﬁnition f , we collect the set of all variable deﬁnitions V f def and variable uses V f use .
For each variable use ( v u , l u ) ∈
V f use , we replace its occurrence by another variable v d ∈ V
f def to obtain an example ensuring the following conditions: 1) v d (cid:54) = v u , 2) | V f def | > 1 , i.e., there are at least 2 possible variables that could ﬁll the slot at l u .
Let i denote the token index in the program f of the variable v u chosen to be replaced by another (incorrect) variable v d in the original program.
We then create two binary vectors Loc ∈ { 0 , 1 } n and Rep ∈ { 0 , 1 } n in the following manner:
[Equation]
Loc is a location vector of length n (program length) which is 1 at the location containing bug, otherwise 0 .
Rep is a repair vector of length n which is 1 at all locations containing the variable v u (correct variable for the location i ), otherwise 0 .
For each buggy training example in our dataset, we also construct a non-buggy example where the replacement is not performed.
This is done to obtain a 50 - 50 balance in our training datasets for buggy and non-buggy programs.
For non-buggy programs, the target location vector Loc has a special token index 0 set to 1 , i.e. Loc [0] = 1 , and the value at all other indices is 0 .
We use the following loss functions for training the location and repair pointer distributions.
[Equation]
The loss function for the repair distribution adds up the probabilities of target pointer locations.
We also experiment with an alternative loss function L (cid:48)
rep that computes the maximum of the probabilities of the repair pointers instead of their addition.
[Equation]
The joint model optimizes the additive loss L joint =
L loc + L rep .
The enumerative solution discussed earlier forms a baseline method.
We specialize the multi-headed pointer model to produce only the repair pointer and use it within the enumerative solution for predicting repairs.
E VALUATION
In our experimental evaluation, we evaluate three research questions.
First, is the joint prediction model V AR M ISUSE R EPAIR effective in ﬁnding V AR M ISUSE bugs in programs and how does it compare against the enumerative solution (Section 4.1)?
Second, how does the presence of as-yetunknown bugs in a program affect the bug-ﬁnding effectiveness of the V AR M ISUSE repair model even in the non-enumerative case (Section 4.2)?
Third, how does the repair pointer model compare with the graph-based repair model by Allamanis et al.
(Section 4.3)?
Benchmarks We use two datasets for our experiments.
Primarily, we use ETH-Py150 2 , a public corpus of GitHub Python ﬁles extensively used in the literature (Raychev et al., 2016; Vechev &
Yahav, 2016).
It consists of 150K Python source ﬁles, already partitioned by its publishers into training and test subsets containing 100K and 50K ﬁles, respectively.
We split the training set into two sets: training (90K) and validation (10K).
We further process each dataset partition by extracting unique top-level functions, resulting in 394K (training), 42K (validation), and 214K (test) unique functions.
For each function, we identify V AR M ISUSE slots and repair candidates.
For each function we generate one bug-free example (without any modiﬁcation) and one buggy example by randomly selecting a location and replacing it with an incorrect variable.
More details about the data generation is presented in the appendix (Section A).
Because of the quadratic complexity of evaluating the enumerative model, we create a smaller evaluation set by sampling 1K ﬁles that results in 12,218 test examples (out of which half are bug-free).
All our evaluation results on the
ETH dataset use a ﬁltered test set (not original).
Note that the inputs to the enumerative model and the joint model are different; the joint model accepts a complete program, while the enumerative model accepts a program with a hole that identiﬁes a slot.
For this reason, training, validation, and test datasets for the enumerative approach are constructed by inserting a hole at variable-use locations.
Our second dataset, MSR-VarMisuse , is the public portion of the dataset used by Allamanis et al.
It consists of 25 C# GitHub projects, split into four partitions: train, validation, seen test, and unseen test, consisting of 3738, 677, 1807, and 1185 ﬁles each.
The seen test partition contains (different) ﬁles from the same projects that appear in the train and validation partitions, whereas the unseen test partition contains entire projects that are disjoint from those in test and validation.
Note the differences between the two datasets: ETH-Py150 contains Python examples with a function-level scope, slots are variable loads, and candidates are variables in the scope of the slot (Python is dynamically typed, so no type information is used); in contrast, MSR-VarMisuse contains C# examples that are entire ﬁles, slots are both load and store uses of variables, and repair candidates are all variables in the slot’s scope with an additional constraint that they are also typecompatible with the slot.
We use the ETH-Py150 dataset for most of our experiments because we are targeting Python, and we use MSR-VarMisuse when comparing to the results of Allamanis et al.
The average number of candidates per slot in the ETH-Py150 dataset is about 9 .
26 , while in MSR-VarMisuse it is about 3 .
4. J OINT M ODEL VS . E NUMERATIVE A PPROACH
We ﬁrst compare the accuracy of the joint model (Section 3.2) to that of an enumerative repair model, similar in spirit (but not in model architecture) to that by Allamanis et al.
For the enumerative approach, we ﬁrst train a pointer network model M r to only predict repairs for a given program and slot.
At test time, given a program P , the enumerative approach ﬁrst creates n variants of P , one per slot.
We then use the trained model M r to predict repairs for each of the n variants and combine them into a single set.
We go through the predictions in decreasing order of probability, until a prediction modiﬁes the original program.
If no modiﬁcations happen, then it means that the model classiﬁes the program under test as a bug-free program.
We deﬁne two additional parameters: 1) τ : a threshold value for probabilities to decide whether to return the predictions, and 2) k : the maximum number of predictions the enumerative approach is allowed to make.
The results for the comparison for different τ and k values are shown in Table 1.
We measure the following metrics: 1) True Positive , the percentage of the bug-free programs in the ground truth classiﬁed as bug free; 2) Classiﬁcation Accuracy , the percentage of total programs in the test set classiﬁed correctly as either bug free or buggy; 3) Localization Accuracy , the percentage of buggy programs for which the bug location is correctly predicted by the model; and 4) Localization+Repair Accuracy , the percentage of buggy programs for which both the location and repair are correctly predicted by the model.
The table lists results in decreasing order of prediction permissiveness.
A higher τ value (and lower k value) reduces the number of model predictions compared to lower τ values (and higher k values).
As expected, higher τ and lower k values enable the enumerative approach to achieve a higher true positive rate, but a lower classiﬁcation accuracy rate.
More importantly for buggy programs, the localization and repair accuracy drop quite sharply.
With lower τ and higher k values, the true positive rate drops dramatically, while the localization and repair accuracy improve signiﬁcantly.
In contrast, our joint model achieves a maximum localization accuracy of 71% and localization+repair accuracy of 65 . 7% , an improvement of about 6 .
4% in localization and about 9 .
9% in localization+repair accuracy, compared to the lowest threshold and highest k values.
Remarkably, the joint model achieves such high accuracy while maintaining a high true-positive rate of 84 .
5% and a high classiﬁcation accuracy of 82 .
This shows that the network is able to perform the localization and repair tasks jointly, efﬁciently, and effectively, without the need of an explicit enumeration.
Performance Comparison:
In addition to getting better accuracy results, the joint model is also more efﬁcient for training and prediction tasks.
During training, the examples for the pointer model are easier to batch compared to the GGNN model in Allamanis et al.
(2018) as different programs lead to different graph structures.
Moreover, as discussed earlier, the enumerative approaches require making O ( n ) predictions at inference time, where n denotes the number of variable-use locations in [Table] a program.
On the other hand, the joint model only performs a single prediction for the two pointers given a program.
4. E FFECT OF I NCORRECT S LOT P LACEMENT
We now turn to quantifying the effect of incorrect slot placement, which occurs frequently in the enumerative approach: n − 1 out of n times for a program with n slots.
We use the same repaironly model from Section 4.1, but instead of constructing an enumerative bug localization and repair procedure out of it, we just look at a single repair prediction.
We apply this repair-only model to a test dataset in which, in addition to creating a prediction problem for a slot, we also randomly select one other variable use in the program (other than the slot) and replace its variable with an incorrect in-scope variable, thereby introducing a V AR M ISUSE bug away from the slot of the prediction problem.
We generate two datasets: AddBugAny , in which the injected V AR M ISUSE bug is at a random location, and AddBugNear , in which the injection happens within two variable-use locations from the slot, and in the ﬁrst 30 program tokens; we consider the latter a tougher, more adversarial case for this experiment.
The corresponding bug-free datasets are NoBugAny and NoBugNear with the latter being a subset of the former.
We refer to two experiments below: Any (comparison between NoBugAny and AddBugAny ) and Near (comparison between NoBugNear and AddBugNear ).
Figure 3 shows our results.
Figure 3a shows that for Any , the model loses signiﬁcant accuracy, dropping about 4 .
3 percentage points for τ = 0 .
The accuracy drop is lower as a higher prediction probability is required by higher τ , but it is already catastrophically low.
Results are even worse for the more adversarial Near .
As shown in Figure 3b, accuracy drops between 8 and 14 . 6 percentage points for different reporting thresholds τ .
These experiments show that a repair prediction performed on an unlikely fault location can signiﬁcantly impair repair, and hence the overall enumerative approach, since it relies on repair predictions for both localization and repair.
Figure 4 shows some repair predictions in the presence of bugs.
4. C OMPARISON OF G RAPH AND P OINTER N ETWORKS
We now compare the repair-only model on MSR-VarMisuse , the dataset used by the state-of-theart V AR M ISUSE localization and repair model by Allamanis et al.
Our approach deviates in three primary ways from that earlier one: 1) it uses a pointer network on top of an RNN encoder rather than a graph neural network, 2) it does separate but joint bug localization and repair rather than using repair-only enumeratively to solve the same task, and 3) it applies to syntactic program information only rather than syntax and semantics.
Allamanis et al.
(2018) reported in their ablation [Figure] [Figure] , user, page): 2 if not user and not self._username: raise TwitterError(’twitter.
Api’)
4 if user: 5 url = ’http://%s.json’ % user 6 else: 7 url = ’http://s.json’ 8 parameters = {} 9 if <slot>: 10 parameters[’page’] = data 11 json = self._FetchUrl(url, parameters=parameters) 12 data = simplejson.loads(json)
Prediction: page (33.7%) (cid:88)
Prediction: data (39.5%) (cid:55) 1 def multisock(self,name,_type): 2 if _type: return MultiSocket(name, <slot>) 4 else: return MultiSocket(name) 1 def multisock(self,name,_type): 2 if _type: return MultiSocket( type, <slot>) 4 else: return MultiSocket(name)
Prediction: type (66.92%) (cid:88)
Prediction: name (35.8%) (cid:55) 1 def closable(self): 2 item = self.dockItem() 3 if item is not None:return <slot>.closable() 5 return True 1 def closable(self): 2 item = self.dockItem() 3 if self is not None:return <slot>.closable() 5 return True
[Figure]
study that their system, on syntax only, achieved test accuracy of 55 .
3% on the “seen” test; on the same test data we achieve 62 .
3% accuracy.
Note that although the test data is identical, we trained on the published training dataset 3 , which is a subset of the unpublished dataset used in that ablation study.
We get better results even though our training dataset is about 30% smaller than their dataset.
4. E VALUATION ON V ARIABLE M ISUSE IN P RACTICE
In order to evaluate the model on realistic scenarios, we collected a dataset from multiple software projects in an industrial setting.
In particular, we identiﬁed pairs of consecutive snapshots of functions from development histories that differ by a single variable use.
Such before-after pairs of function versions indicate likely variable misuses, and several instances of them were explicitly marked as V AR M ISUSE bugs by code reviewers during the manual code review process. More precisely, we ﬁnd two snapshots f and f (cid:48) of the same program for which V f def = V f (cid:48) def , V f use = V f common ∪ ( t i , l i ) , and V f (cid:48) use =
V f common ∪ ( t (cid:48)
i , l i ) , where t i (cid:54) =
t (cid:48) i , and t i , t (cid:48) i ∈ V f def . For each such before-after snapshot pair ( f, f (cid:48) ) , we collected all functions from the same ﬁle in which f was present.
We expect our model to classify all functions other than f as bug-free.
For the function f , we want the model to classify it as buggy, and moreover, localize the bug at l i , and repair by pointing out token t (cid:48) i .
In all, we collected 4592 snapshot pairs.
From these, we generated a test [Table] dataset of 41672 non-buggy examples and 4592 buggy examples.
We trained the pointer model on a training dataset from which we exclude the 4592 ﬁles containing the buggy snapshots.
The results of the joint model and the best localization and repair accuracies achieved by the enumerative baseline approach are shown in Table 2.
The joint model achieved a true positive rate of 67 . 3 %, classiﬁcation accuracy of 66 .
7 %, localization accuracy of 21 .
9 % and localization+repair accuracy of 15 . 8 %.
These are promising results on data collected from real developer histories and in aggregate, our joint model could localize and repair 727 variable misuse instances on this dataset.
On the other hand, the enumerative approach achieved signiﬁcantly lower values of true positive rate of 41 .
7 %, classiﬁcation accuracy of 47 .
2 %, localization accuracy of 6 . 1 %, and localization+repair accuracy of 4 . 5 %.
C ONCLUSION
In this paper, we present an approach that jointly learns to localize and repair bugs.
We use a key insight of the V AR M ISUSE problem that both the bug and repair must exist in the original program to design a multi-headed pointer model over a sequential encoding of program token sequences.
The joint model is shown to signiﬁcantly outperform an enumerative approach using a model that can predict a repair given a potential bug location.
In the future, we want to explore joint localization and repair using other models such as graph models and combinations of pointer and graph models, possibly with using more semantic information about programs.
A T RAINING D ATA G ENERATION
For each Python function in the ETH-Py150 dataset, we identify V AR M ISUSE slots and repair candidates.
We choose as slots only uses of variables in a load context; this includes explicit reads from variables in right-hand side expressions ( a = x + y ), uses as function-call arguments ( func(x, y) ), indices into dictionaries and lists even on left-hand side expressions ( sequence[x] = 13 ), etc.
We deﬁne as repair candidates all variables that are in the scope of a slot, either deﬁned locally, imported globally (with the Python global keyword), or as formal arguments to the enclosing function.
For each combination of function, slots, and candidates, we generate one bug-free example, by leaving the function as is and marking it (by assumption) as correct.
We also generate one buggy example per slot, as long as there are at least two repair candidates for a slot (otherwise, the repair problem would be trivially solved by picking the only eligible candidate); we discard slots and corresponding examples with only trivial repair solutions, and we discard functions and corresponding examples with only trivial slots.
To illustrate, using the correct function in Figure 1b, we would generate one bug-free example (labeling the function, as is, as correct), and one buggy example per underlined slot (inserting an incorrect variable chosen at random), each identifying the current variable in the slot as the correct repair, and the variables sources , object name , subject name , and result as repair candidates.
Although buggy repair examples are deﬁned in terms of candidate variable names , any mention of a candidate in the program tokens can be pointed to by the pointer model; for example, the repair pointer head, when asked to predict a repair for the slot on line 6, could point to the (incorrect) variable sources appearing on lines 1, 2, or 3, and we don’t distinguish among those mentions of a predicted repair variable when it is the correct prediction.
The MSR-VarMisuse dataset consists of 25 C# GitHub projects, split into four partitions: train, validation, seen test, and unseen test, consisting of 3738, 677, 1807, and 1185 ﬁles each.
The seen test partition contains (different) ﬁles from the same projects that appear in the train and validation partitions, whereas the unseen test partition contains entire projects that are disjoint from those in test and validation.
The published dataset contains pre-tokenized token sequences, as well as
V AR M ISUSE repair examples, one per slot present in every ﬁle, as well as associated repair candidates for that slot, and the correct variable for it.
This dataset deﬁnes slots as variable uses in both load and store contexts (e.g., even left-hand side expressions), and candidates are type-compatible with the slot.
Every example has at least two repair candidates.