H OPPITY : L EARNING G RAPH T RANSFORMATIONS TO D ETECT AND F IX B UGS IN P ROGRAMS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a learning-based approach to detect and ﬁx a broad range of bugs in
Javascript programs.
We frame the problem in terms of learning a sequence of graph transformations: given a buggy program modeled by a graph structure, our model makes a sequence of predictions including the position of bug nodes and corresponding graph edits to produce a ﬁx.
Unlike previous works that use deep neural networks, our approach targets bugs that are more complex and semantic in nature (i.e. bugs that require adding or deleting statements to ﬁx).
We have realized our approach in a tool called H OPPITY .
By training on 290,715 Javascript code change commits on Github, H OPPITY correctly detects and ﬁxes bugs in 9,490 out of 36,361 programs in an end-to-end fashion.
Given the bug location and type of the ﬁx, H OPPITY also outperforms the baseline approach by a wide margin.
I NTRODUCTION
The sheer size and complexity of modern codebases makes it impossible for them to be bug-free.
As a result, a more reasonable and effective strategy has emerged, which aims to prevent bugs in production by applying automated tools to detect and even ﬁx them early in the development process.
This trend has gained increasing popularity in recent years.
Examples include Google’s Tricorder (Sadowski et al., 2015), Facebook’s Getaﬁx (Scott et al., 2019) and Zoncolan, and Microsoft’s Visual
Studio IntelliCode.
The techniques underlying these tools can be classiﬁed into broadly two categories: logical, rule-based techniques (Sadowski et al., 2015) and statistical, data-driven techniques (Allamanis et al., 2018; Pradel & Sen, 2018; Vasic et al., 2019).
The former uses manually written rules capturing undesirable code patterns and scans the entire codebase for these classes of bugs.
The latter learns to detect abnormal code from a large code corpus using deep neural networks.
Despite great strides, however, both kinds of tools are limited in generality because they target error patterns in speciﬁc codebases or they target speciﬁc bug types.
For instance, Zoncolan’s rules are designed to be speciﬁcally applicable to Facebook’s codebases, and deep learning models target specialized bugs in variable naming or binary expressions.
Moreover, the patterns are relatively syntactic, allowing them to be speciﬁed by human experts using logical rules or learnt from a corpus of programs.
In this paper, we propose a novel learning-based approach for ﬁnding and ﬁxing bugs in Javascript programs automatically.
Javascript is a scripting language designed for web application development.
It has been the most popular programming language on GitHub since 2013.
Repairing Javascript code presents a unique challenge as bugs manifest in diverse forms due to unusual language features and the lack of tooling support.
Therefore, the primary goal of our approach is generality since it needs to be effective against a board spectrum of programming errors, such as using wrong operators or identiﬁers, accessing undeﬁned properties, mishandling variable scopes, triggering type incompatibilites, and many others.
Another important novel aspect concerns our approach’s ability to deal with bugs that are more complex and semantic in nature, namely, bugs that require adding or removing statements from a program, which are not considered by prior works.
Finally, compared to automated program repair techniques (Le Goues et al., 2019; Scott et al., 2019; Hua et al., 2018; Chen et al., 2018) which require knowledge of bug location, this paper presents an end-to-end approach including localizing bugs, predicting the types of ﬁxes, and generating patches.
We design our model architecture in a similar vein as a Neural Turing Machine (NTM) (Graves et al., 2014).
It consists of an external memory (a Graph Neural Network) for embedding a buggy program and a central controller (an LSTM) that makes a sequence of primitive actions (e.g., predicting type, ﬁnding location, generating patch, etc.) to perform a ﬁx.
The multi-step decision process is implemented by an autoregressive model.
Crucially, our model differs from the standard NTM in how the memory is manipulated: apart from the common read and write operations, the controller can also expand or shrink the memory when adding or deleting nodes in the original graph.
We have realized our approach in a tool called H OPPITY .
By training on 290,715 Javascript code change commits collected from Github, H OPPITY correctly detects and ﬁxes bugs in 9,490 out of 36,361 programs using a beam size of three.
Compared to GGNN (Allamanis et al., 2018), H OPPITY achieves 8.3% better accuracy in ﬁxing bugs, when the type of the ﬁx and the bug location are kept constant for both models.
M OTIVATING E XAMPLES
Javascript is a rather different language compared to other object-oriented languages (e.g. C++, Java, or C#).
Besides the weak, dynamic typing discipline of scripting languages, Javascript supports many peculiar features that do not exist in other languages.
For example, it allows a property (i.e., a ﬁeld) to be added to or removed from an object at runtime.
As another example, Javascript did not support block-level scoping until recently, allowing a variable deﬁned in a block structure such as a for loop to be exposed to the entire function in which the loop occurs.
While the latest ES6 language standard incorporates block-level scoping, developers have been programming without it for decades, resulting in a large body of legacy code.
Finally, Javascript’s eval function, which interprets and executes a string as a code fragment, is widely regarded as a major source of bugs and vulnerabilities.
All of these aspects make programming in Javascript a frustrating and error-prone experience.
[Table]
[Figure]
[Table]
[Figure]
[Table]
(a) InnerHTML should have been innerHTML .
if
[Table]
[Figure]
Static analyzers aim to detect common coding errors in Javascript programs by applying logical rulebased reasoning on source code.
TAJS (Jensen et al., 2009) and ESLint (Zakas, 2013) are prominent examples.
These tools face important challenges to be effective.
We present several examples in
Figure 1 to illustrate their limitations.
Due to the complex nature of client-side web APIs, TAJS and ESLint choose to ignore analyzing built-in libraries for the sake of scalability.
As depicted in
Figure 1a, when developers mistakenly capitalize the ﬁrst letter of innerHTML , a property of class
Element in DOM (Document Object Model), both analyzers fail to catch the error.
Javascript will then silently allow developers to set the previously non-existent property InnerHTML to the empty string.
Later, when developers attempt to access the intended property, innerHTML , the program will crash and potentially cause a security vulnerability or incur a costly debugging experience.
Additionally, static analyzers can never deal with functional bugs (i.e., errors that violate the program speciﬁcation and yet conform to the coding rules).
Figure 1b shows one such example.
The goal is to split a string using regular expressions.
However, the program incorrectly splits the input ' and ' into ['', ' and ', ''] instead of ['', ''] , which is what the developer intended.
Since the error does not cause program to crash, static analyzers are incapable of catching them.
The bugs that static analyzers missed in both cases are in hindsight quite obvious to human programmers.
The criteria they use is very simple: any code snippet that seems to deviate from common code patterns is likely to be buggy.
This is precisely the observation that our approach seeks to mimic.
In
[Figure]
particular, if a model observes a property or an unusual way of splitting strings that never appeared in the training data, it is likely to recognize those abnormal code fragments as potential bugs.
The main advantage of our approach over existing neural-based bug detectors (Allamanis et al., 2018; Pradel &
Sen, 2018; Vasic et al., 2019) is its generality.
Unlike these prior works that target speciﬁc classes of bugs (e.g., variable naming issues or binary expression bugs), we train a single model to deal with a wide range of bug types, encompassing all previously proposed ones.
Compared to past approaches that leverage a graph-based neural network model (Allamanis et al., 2018), our model is capable of more sophisticated transformations such as adding or removing nodes, as shown in Figure 1c and 1d.
Finally, our model not only locates but also ﬁxes bugs,
whereas program repair (Le Goues et al., 2019; Scott et al., 2019; Hua et al., 2018; Chen et al., 2018) or bug localization (Ball et al., 2003;
Jose & Majumdar, 2011; Wang et al., 2019) techniques only solve a single task.
M ODEL
We model the problem of detecting and repairing bugs in programs as a structured prediction problem on a graph-based representation of programs.
Given a graph g bug that represents a buggy program, we wish to predict a graph g fix that represents the ﬁxed program.
Our model aims to capture the structured prediction by a sequence of up to T steps of graph transformations:
[Equation]
The high-level overview of the graph sequence transformation is shown in Figure 2.
Different programs may need a different number of steps T ( g bug ) which is also determined by the model.
We ﬁrst introduce our representation module for programs in Sec 3.1.
We then elucidate each step of the above transformation in Sec 3.2.
Finally, we summarize and present the full model in Sec 3.3.
3. P ROGRAM REPRESENTATION
Programs written in a high-level language have rich structure.
Researchers have proposed graph-based representations to capture this structure (Allamanis et al., 2018).
We start with this approach of representing programs using graphs with certain modiﬁcations for our task.
As shown in the left part of Figure 2, we ﬁrst parse the program’s source code into an abstract syntax tree (AST) form that captures the program’s syntactic structure.
We then connect the leaf nodes with SuccToken edges.
Unlike previous approaches, we additionally add value nodes that store the actual content of the leaf nodes, with special ValueLink edges connecting them together.
The purpose of introducing this additional set of nodes is to provide a name-independent strategy for code representation and modiﬁcation, which we elucidate in the next section.
Hereafter, we use g fix , g bug or g in general to represent either the source code or the corresponding graph structure.
After representing the program as a graph, we use a Graph Neural Network (GNN) (Scarselli et al., 2008) to map the graph into a representation in a ﬁxed dimensional vector space.
Speciﬁcally, given a graph g
=
( V, E ) with set of nodes V and edges E , we need a function f ( g ) (cid:55)→ ( R d , R | V |× d ) to obtain the d -dimensional representation of graph g (denoted as (cid:126)g ), as well as representations of individual nodes v ∈ V (denoted as (cid:126)v ).
To parameterize f ( · ) , we employ the form in GIN (Xu et al.,
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
o n t r o ll e r
[Figure]
, with our adaptation to our multigraph for program representation in the following manner:
[Equation]
where W l,k 1 ∈ R d × d , W l 2 ∈ R dK ×
d are model parameters and σ ( · ) is tanh in this paper.
K is the total number of edge types in this multi-graph representation.
In the end, the node embedding is (cid:126)v =
h ( L ) v , where L is the total number of propagations in the GNN.
N k ( v ) is the set of neighbors of node v that are connected by edge with type k .
Following GIN, the graph representation (cid:126)g is the aggregation of h lv , ∀ l ∈ 0 , 1 , . . .
, L .
We use max pooling to aggregate h lv for each l , and then take the average of these L + 1 vectors to obtain (cid:126)g .
At the beginning, we use the node type as one-hot features to initialize h (0) v , where the types are either obtained from the AST representation, or from the local value table as shown in Figure 2.
Note that we don’t use features like variable names or function names in this graph representation, as different programs may follow different naming conventions.
Instead, we focus on the syntactic structure of the source code, so as to enable naming-agnostic representation across different programs.
3. O NE - STEP GRAPH EDIT
There are ﬁve types of operators to choose from for a single step graph edit, namely, adding a node ( ADD ), deleting a node ( DEL ), replace a node value ( REP_VAL ), replace a node type ( REP_TYPE ) and stop ( NO_OP ).
When combined with multi-step edits, these operators sufﬁce to capture a rich variety of code modiﬁcations.
We notice that these operators share some common low-level primitives, such as ﬁnding the location, predicting value, etc.
So we ﬁrst introduce the individual low-level primitives and then present how to assemble these for each type of graph edit operator.
3.2. L OW - LEVEL PRIMITIVES
Our low-level primitives contain location, type, and value prediction.
These primitives can be combined for different operators later on.
In this section, we assume the availability of a controller, represented as (cid:126)c ∈ R d .
It keeps track of the global state, including the original source code, as well as the edits made so far.
We will elaborate this when we assemble different primitives together.
Location The location primitive locates a speciﬁc position in the source code.
While it corresponds to region selection in the original text representation, with the graph representation, we can easily treat it as a node selection step.
As different programs have different numbers of nodes, we employ a pointer network (Vinyals et al., 2015) into the graph structure.
Speciﬁcally, after obtaining the node embeddings { (cid:126)v } v ∈ V , we select the node via loc ( (cid:126)c, g ) =
arg max v ∈ V (cid:126)v (cid:62) (cid:126)c for simplicity.
Value
The value primitive assigns a value for a leaf node in the AST.
Instead of predicting the replacement value using a language generative model (Chen et al., 2018) or GNN score function (Allamanis et al., 2018), we adopt the attention mechanism to let the model to choose from either the values appearing in the current ﬁle (local value table), or a collection of global values that are common for the speciﬁc language.
Let D val be the global dictionary of commonly used leaf-node values in the language, where each item i v ∈ D val is associated with a vector representation (cid:126)i v ∈ R d .
The local value table is denoted as V val which is a subset of the nodes in current graph.
Then, the value is predicted via val ( (cid:126)c, g ) =
argmax t ∈ D val ∪ V val (cid:126)t (cid:62) (cid:126)c .
Again we use inner product simply for efﬁciency, while more expressive score functions can also be used.
Type The type primitive assigns the type for non-terminal nodes in an AST.
As the total possible number of types is ﬁnite and ﬁxed for a given language, the type prediction is simply a multiclass classiﬁcation problem.
However, we can utilize the AST grammar checker with contextual information to prune the output space.
To predict the type of a given non-terminal node, we can obtain its parent node and current children.
Then, by looping over the valid production rules at the current location, we can obtain a list of all valid types.
The ﬁnal type is only chosen from this set.
3.2. G RAPH EDIT OPERATORS
Each round of edit starts with the current graph g , the corresponding graph embedding, and the ‘macro-context’ embedding (cid:126)c M that captures the edit history so far.
In each round, the edit type e is predicted out of the ﬁve operators.
Then, a ‘micro-context’ embedding (cid:126)c m =
LSTM ( (cid:126)e | c M ) is obtained from the macro embedding updated by the LSTM with operator embedding (cid:126)e .
The micro-context embedding is used as the controller throughout the process of each operator.
ADD This operation adds a new node to the graph.
Unlike in Li et al.
(2018) where the node and corresponding edges are added in separate stages, which would introduce extra complexity, we introduce a simple mechanism that can uniquely add a node and corresponding edges.
As is shown in
Figure 3, this process invokes two location primitives, one value primitive, and one type primitive.
The two location calls determine the parent and left sibling of the node.
With this information, we can uniquely determine the position to insert into the AST.
Finally, the corresponding edges—
SuccToken , ValueLink , and AST edges—can automatically be inferred with the location.
As this process is autoregressive, the micro-context embedding is kept updated with all the primitive calls.
For this speciﬁc operator, the context is updated in the order of: (cid:126)c m 1 =
LSTM ( −−−−→ v parent | (cid:126)c m ) , (cid:126)c m 2 = LSTM
−−−−→ v sibling | (cid:126)c m 1 ) , (cid:126)c m 3 =
LSTM ( val ( (cid:126)c m 2 , g ) | (cid:126)c m 2 ) and (cid:126)c m 4 =
LSTM ( type ( (cid:126)c m 3 , g ) | (cid:126)c m 3 ) .
In the end, (cid:126)c ADD =
(cid:126)c m 4 summarizes the process.
DEL This operator deletes a node and corresponding edges in the graph.
If it is a non-terminal node in the AST, then the corresponding subtree is removed as well.
To locate the node, a single call to the location primitive is needed.
Then, the micro-context embedding is updated by the LSTM via the embedding of the node being deleted.
REP_VAL This operator replaces the value of a leaf (terminal) node in the AST.
This procedure consists of two primitive steps: locating the node and predicting the value.
Tthe leaf node is linked to the new value node in the internal value table via a ValueLink edge.
Also, the micro-context embedding is updated by the LSTM via the embedding of the corresponding node and value.
REP_TYPE This operator changes the type of a non-terminal node, which involves location and type primitive steps.
The micro-context embedding is updated by the LSTM via the embedding of the corresponding node and type.
NO_OP
This op does not change the graph.
It simply denotes the end of the sequence of graph edits.
3. G RAPH TRANSFORMATION
1: Input g bug ∼ D and model parameters θ .
2: Obtain (cid:126)g 0 , { (cid:126)v v ∈ g bug } = f
( g bug ) , let (cid:126)c M 0 be null.
3: for t = 1 to T do 4:
Predict e t with −−→ c (cid:48)
M t = LSTM ( −−→ g t − 1 |−−−→ c M t − 1 ) .
5: if e t = NO_OP then 6: set g T =
g t − 1 and exit the loop.
7: end if 8:
Perform operator e t with −−→ c (cid:48)
M t .
Get new graph g t , update −−→ c M t with −→ c e t .
10: end for 11: Return g T
Our end-to-end model for graph transformation inference is shown in Alg 1.
We denote the buggy graph g bug as g 0 for simplicity.
Then, for the t -th graph edit, we ﬁrst obtain the graph representation −−→ g t −and node embeddings as well.
Next, the macro-context embedding is obtained by −−→ c (cid:48)
M t = LSTM ( −−→ g t − 1 |−−−→ c M t − 1 ) .
After picking the graph edit operator e t , we obtain the corresponding micro-context summary −→ c e t , which is used to update the macro-context
[Table] embedding as
[Equation]
This process continues until it reaches the maximum steps T or the NO_OP operator is selected.
Note that our framework can capture the situation when the input program is bug-free.
In this case, the
NO_OP operator is supposed to be triggered at the ﬁrst step.
Also, each edit step is not limited to a single node level operation.
It can be extended to modify a certain substructure (e.g., replace a tree node with one of its children).
This in turn allows program repair to be performed in fewer edit steps.
L EARNING
Given the dataset D
= { ( g ( i ) bug , g ( i ) fix ) } |D| i =1 which consists of pairs of buggy code and the ﬁxed code, the learning objective max θ E ( g bug ,g fix ) ∼D p ( g fix | g bug ; θ ) maximizes the likelihood of ﬁxes.
Since the probability is factorized according to Eq 1 where a sequence of transformations is performed, we parse the source code using the SHIFT AST format, and utilize a JSON diff toolbox to compile the code differences into a sequence of AST edits.
This serves as the ﬁne-grained supervision mechanism for our graph transformation formulation.
Thus, the MLE objective above is realized with the sum of cross entropy loss at each step of graph edits.
During training, we jointly optimize the graph representation module { f t ( · ) } Tt =1 , each of the operator module and the controller module which is parameterized by LSTM.
We use the Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
99 and initial learning rate of 10 − 3 .
Due to the large size of each sample, we use a small batch size of 10 during training.
Furthermore, to stabilize the training, we apply the gradient clip with the maximum norm of 5.
I NFERENCE
The inference procedure involves searching for the maximum in the combinatorial space: arg max g fix p ( g fix | g bug ; θ ) .
Since the search space is very large, however, we use beam-search to approximately ﬁnd the ﬁxes with highest probabilities.
Speciﬁcally, we maintain a pool of partially ﬁxed programs { ˜ g } , which starts with simply the single buggy program g bug .
The pool size is limited by the beam-search size B .
For each ˜ g , we propose the top B primitives ( location , type , value ) or top B operators, depending on the current stage of the edit ˜ g .
Then the total B 2 solutions are ranked together based on the joint log-likelihood, and the top B solutions with the largest likelihood are kept in the pool for the next round of beam search.
Unlike beam search for language models where the vocabulary size is ﬁxed, in our setting, the available choices or even the steps of inference may vary (e.g., the ADD operator has more steps of primitive calls than the DEL operator).
Our implementation is based on PyTorch with customized
GPU kernels to enable efﬁcient inference on GPUs.
E XPERIMENTS
Dataset Our model is trained and evaluated on a corpus of nearly half a million data points.
We have created a robust system to continuously collect small changes in Javascript programs from Github.
Given a commit, we download the Javascript ﬁle before and after the change: ( src buggy , src fixed ) .
Commits can contain many types of changes such as feature additions, refactorings, bug ﬁxes, etc.
In an attempt to ﬁlter our dataset to only include bug ﬁxes, we use a heuristic based on the number of changes to the AST.
A commit with a smaller number of AST differences is more likely to be a bug ﬁx.
Thus for the experiments, we use three different datasets:
OneDiff with precisely one edit; textttZeroOneDiff with zero and one edit and ZeroOneTwoDiff with zero, one or two edits.
We additionally ﬁlter out data points with ASTs larger than 500 nodes as a parameter in our system.
A detailed overview of our corpus crawler is available in Appendix B. [Table]
6. E VALUATION
We train the model for 3 epochs on the training set until the validation loss converges.
We tried different conﬁgurations of our model with different number of layers and different graph embedding methods besides the generic one in Eq 2.
We report on these ablation studies in Appendix C.
Table 2 shows the evaluation results of our model on a held out test set consisting of samples from
OneDiff dataset.
Additional experiments on ZeroOneDiff and ZeroOneTwoDiff datasets are available in Appendix A. We also provide experimental results with respect to different conﬁgurations.
Accuracy is shown for each graph edit operation type.
Accuracy is measured in a complete discrete graph edit operation step.
For example consider Figure 1a, in which we edit an object property name with the REP_VAL operation.
If the model incorrectly predicts the locatio to be of type DEL , then it will not go on to predict a value .
In this case, the model will be penalized twice in the operation accuracy as well as the value accuracy.
A prediction is considered totally correct only if the entire sequence of graph edit primitives is correct.
Note that top-1 greedy prediction is not always among top-3 when beam search is used.
Additionally, operator prediction is only evaluated on the top prediction as there are 4 types in total.
To demonstrate the magnitude of the search space, we compare H OPPITY to a model that selects uniformly at random, in each step of the graph edit process.
The random model performs well at operation type selection since the search space only has four options ( ADD , REP_VAL , REP_TYPE ,
DEL ).
However, after the operation type is predicted, the random model’s accuracy drops, as there are up to 500 nodes in the buggy AST.
When it predicts value, the accuracy drops even further as our vocabulary contains 5,000 values.
Lastly, type prediction has slightly better accuracy than value prediction because the number of the types of AST nodes in total is smaller than our vocabulary.
6. B ASELINES
As existing approaches cannot be applied for comparison in Table 7, we adapt the baselines to some restricted settings in this section.
As most of the baselines focusing on ﬁxing one bug, we report the results on OneDiff dataset in these experiments.
Note that for all the comparisons we simply provide equal amount of information to our model without retraining it.
GGNN: Allamanis et al.
(2018) (denoted as GGNN) proposed approaches for speciﬁc bug repair, including VARMISUSE and VARNAMING.
Speciﬁcally, for REP_TYPE prediction we have
• GGNN-Rep: we adopt VARMISUSE to replace with candidate node type and modify the graph structure correspondingly; we use their proposed max-margin formulation for training; • GGNN-Cls: we perform multi-class classiﬁcation using the target node and graph embedding. For REP_VAL prediction, we also made two versions of adaptations: • GGNN-Rep: similar to above, here the candidate set is from values in the current graphs plus the top-100 frequent values used for repair in the training set.
• GGNN-RNN: we adopt VARNAMING approach to predict value directly.
Due to the huge vocabulary size, we use char-level language model for predicting the replacement.
Table 3 and 4 show the comparison when buggy node is known.
Regarding the type prediction, as the number of types is large, the likelihood formulation with classiﬁcation objective outperforms the max-margin loss based ones.
As in this limited case GGNN-Cls and H OPPITY are quite similar except for graph representation, the performance is also comparable.
Also for the value prediction,
[Table]
[Table] [Table] [Table] our formulation of pointer on graph is more effective.
We found when the space of decisions is large, it is hard to apply structure prediction method like GGNN-Rep in this case.
Since real-world programs are noisy, the sentences used in different programs vary a lot, which makes it difﬁcult for language models to predict the exact accurate value.
A possible extension is to combine the language model with the graph pointer, which we will explore in future work.
SequenceR: The model proposed by Chen et al.
(2018) is a translation based model, and thus is more generic than GGNN as it directly outputs the text tokens for code repair.
However it still needs the supervision of bug location with a pair of <START_BUG> and <END_BUG> .
We compare our model with this by providing location information as well.
Table 5 summarizes the total accuracy for ﬁxing a single bug.
Note that to make the prediction of SequenceR easier, we let it predict the same information as our model ( i [Table] e. , predict op, value etc.
in a sequential way), rather than the noisy raw text code.
Thus the experiment mainly shows the beneﬁt of formulating code repair with graphs over text tokens.
With the above two baselines, we can see that in the restricted case our model can still yield comparable or even better performances.
Given that our model can go for more edits without location information, we believe this tool is more generic and effective for code repair.
TAJS:
We also compare the bug detection ability of H OPPITY against TAJS (Jensen et al., 2009) which is a well-known static analysis tool for Javascript programs.
Automating the comparison for our entire test set proved to be infeasible.
For example, TAJS only accepts JavaScript ES5 programs, while the vast majority of current JavaScript projects use ES6 or other variants like React JSX.
Another problem is that TAJS does not analyze code that is not invoked, e.g., a library function that is not called by client code.
Moreover, determining the right command-line options of TAJS is non-trivial since it provides many options targeting different JavaScript runtime environments.
Due to these issues, we forgo a large-scale comparison, and instead pick 30 random points in our test set to manually analyze using TAJS.
Table 6 depicts the results (Appendix D provides further details).
We restrict the chosen test points to satisfy a necessary condition for undeﬁned property bugs since
TAJS claims to be proﬁcient in detecting this class of bugs.
In the process, we also pick some functional bugs, as well as cases of refactoring modiﬁcations.
By resolving the numerous issues that prevented us from automating the comparison, we were able to run TAJS manually.
TAJS failed to detect any real bugs in the 30 test points.
While functional bugs and refactoring modiﬁcations are beyond TAJS, however, TAJS also raises many unrelated false alarms due to its failures in locating
NodeJS libraries, importing JSON ﬁles, or recognizing built-in global variables.
These warnings are detrimental because TAJS suspends the analysis as soon as it detects what it preceives to be a bug.
To further aid TAJS, we omitted parts of each program that are unrelated to the bug, in the hope of driving TAJS’s analysis as deep as possible.
After all these measures, TAJS managed to detect two of the undeﬁned property bugs (Bug IDs 4 and 6 in Appendix D).
In contrast, H OPPITY is able to correctly detect 5 bug locations of the 30 testing points within our top 3 predictions.
Moreover, H OPPITY also produces 4 patches that are identical to the developer’s ﬁxes.
Our comparison highlights H OPPITY ’s two important strengths compared to TAJS.
First, H OPPITY relieves developers from the enormous burden of manual conﬁguration.
Second, H OPPITY achieves far better performance in detecting as well as ﬁxing the bugs in Javascript programs.
R ELATED W ORK
Static analysis for bug detection.
Static analyzers such as FindBugs, Error-Prone, and Semmle use syntactic pattern-matching and dataﬂow analysis to ﬁnd common bugs.
Typically, detecting even a single class of bugs can require dozens or even hundreds of patterns.
Coverity (Bessey et al., 2010),
SonarQube, and Clang Static Analyzer check for semantic inconsistencies in code based on more sophisticated path analyses.
Infer (Calcagno et al., 2015) is built upon sound principles and can prove the absence of certain classes of bugs.
TAJS belongs to this category as well.
Due to the undecidability of the problem, however, approximations are inevitable which voids the guarantees in practice.
Compared to all static analysis tools, H OPPITY offers the following advantages: (1) it targets a board range of programming errors; (2) it not only localizes bugs but also ﬁxes them; and (3) it has signiﬁcantly higher signal-to-noise ratio (i.e., detects more bugs with less false alarms).
Learning-based bug detection.
Allamanis et al.
(2018) target variable-misuse errors and present a solution based on a gated graph neural network model to predict the correct variable name given a buggy location.
Vasic et al.
(2019) present a pointer network on top of a RNN which outperforms Allamanis et al.
(2018) on the same task.
DeepBugs (Pradel & Sen, 2018) proposes a name-based bug detection scheme.
Their model is trained to predict three classes of bugs: swapped function arguments, wrong binary operator, and wrong operand in a binary operation.
Compared to these models, our approach is capable of detecting and ﬁxing a wide range of errors in Javascript.
SequenceR (Chen et al., 2018) uses sequence-to-sequence model to translate a buggy code segment into correct one;
Getaﬁx (Scott et al., 2019) produces human-like bug ﬁxes by learning from past ﬁxes.
It employs a hierarchical clustering algorithm that sorts ﬁx patterns according to their generality.
While these approaches are general against different types of bugs, they still need the bug location as input.
Graph learning and optimization.
Our work is closely related to the literature in graph representation learning and optimization.
Our model uses a variant of GNN that is inspired by many representative works (Li et al., 2015; Xu et al., 2018;
Si et al., 2018), with the adaptation of local value table and pointer mechanism.
Our work is also related to auto-regressive graph modeling
Johnson (2016); Li et al.
; Brockschmidt et al.
Dai et al.
(2018), but with more generic operations such as subtree deletion and attribute modiﬁcations.
Some other works model the graph modiﬁcation in latent space (Jin et al., 2018;
Yin et al., 2018), but such frameworks lack ﬁne-grained control over the generative process, and thus are not very suitable for performing code repair.
C ONCLUSION
We proposed an end-to-end learning-based approach to detect and ﬁx bugs in Javascript programs.
We realized the approach in a tool H OPPITY and demonstrated that it correctly predicts 9,490 out of 36,361 code changes in real programs on Github.
In the future, we plan to expand the targeted bugs to include those that are caused by the interdependence among multiple ﬁles or that require multiple steps to ﬁx.
We will also deploy H OPPITY in an IDE to further evaluate its accuracy and utility.
Finally, we plan to extend our learning framework to support other languages.
Due to its language-independence, we believe H OPPITY will beneﬁt developers beyond Javascript as well.
A A DDITIONAL E XPERIMENTS
[Table]
Full experiment results In addition to the evaluation of samples with one edit Table 7, we also evaluate H OPPITY on the following datasets:
• ZeroOneDiff - Includes samples with labels of zero or one edit • ZeroOneTwoDiff - Includes samples with labels of zero, one, or two edits.
We trained models on each dataset for roughly 12 hours on a single GTX 2080Ti GPU.
Accuracy on the ZeroOneDiff is the highest as predicting that an AST is not buggy does not consist of any low level primitive predictions.
This makes it a much easier prediction for the model than say, an ADD operation which the parent location, left sibling, value, and type must all be predicted correctly in order to be considered accurate.
TRUE LABEL
BUGGY
NOT BUGGY
[Table]
False positive/negative study An evaluation of false positives and false positives is available in
Table 8.
In this setting, we treat the problem as a classiﬁcation problem and our model attempts to predict if a given AST is BUGGY / NOT BUGGY.
If the model predicts ADD , REP_VAL , REP_TYPE , or DEL , we consider this a prediction of "BUGGY."
Accordingly, if the model predicts NO_OP , we consider this to be a prediction of "NOT BUGGY."
Accuracy v.s. size of graph To demonstrate the affect of AST size on H OPPITY ’s prediction accuracy, we include Figure 4.
As expected, AST size and accuracy are inversely related.
[Table]
Accuracy v.s. beam search size In Table 9 we compare the performance with different beam sizes.
As we can see, the top-3 accuracy with beam size 3 is signiﬁcantly better than top-1 accuracy with just greedy prediction.
This is expected, as in the decision process there are ’bottleneck’ stages with only a few predictions (e.g., the op prediction).
Thus from beam-1 to beam-3 there’s huge improvement, but further beyond the performance maxed out.
B D ATA C OLLECTION
We have built a robust system to automatically collect millions of bug-ﬁxes in Javascript programs from Github.
Our system continuously crawls Github for commits containing Javascript ﬁles and creates a label consisting of the change to the AST corresponding to each such ﬁle.
Our system consists of three entirely automated parallel steps:
Collect Commits: Our system uses the GH Archive API to easily access Github event data for a speciﬁc hour in time.
After obtaining all data for the hour, we ﬁlter this using the Github API to only include commits that consist of edits to Javascript ﬁles.
Download Files: As we are obtaining a list of valid commits from step 1, we begin downloading the pair: ( src buggy , src fixed ) where src buggy is the ﬁle prior to the commit and src fixed is the ﬁle following the commit that contains the changes made.
Create Label:
For each Javascript ﬁle downloaded, we parse the source code into a JSON format of the AST.
Our system uses the SHIFT AST
Abstract Syntax Tree representations are designed to naturally and intuitively represent the structure of the source code.
Because of this design goal, small changes in the source code can often lead to very large changes in the AST.
We chose the SHIFT AST representation with consideration to our goal of maximizing the number of commits with only one difference between the ASTs.
This component produces a pair of ASTs: ( AST buggy , AST fixed ) , at which point a JSON differencing algorithm, fast json-patch 2 is applied to create a label.
The label includes the operation type and node edited for each difference between AST buggy and AST fixed .
Each step of this process is parallelized in order to grow our corpus as quickly as possible.
Our dataset has the advantage that it is continuously growing without human input.
Our system is language independent and highly extensible and modular.
For example, it can handle any language so long as it can be parsed into a JSON AST.
For each label, we must download two ﬁles src buggy and src fixed .
Additionally, if source ﬁles cannot be parsed into a SHIFT AST, a label cannot be created.
For our learning corpus, we limit the dataset to only include labels with one AST difference.
Additionally, in an attempt to limit graph size, we only include data points in which the AST buggy and AST fixed have less than 500 nodes. [Table]
Figure 5 plots the distribution of number of edits that are recorded in Table 10.
We can see the distribution is long tail, with majority of edits as 1 or 2.
C A BLATION STUDY
We tried different graph representations with corresponding graph embedding methods.
The multi represents the multi-graph deﬁned by different edge types, with the parameterization of message passing function mentioned in Eq 2; the code2inv is the parameterization used in Si et al.
; the single instead uses a single graph with edge types as one-hot edge features.
We found that more layers does not lead to better generalization in our setting, and it becomes slower in terms of convergence.
So we report the results with 4 layers in our main paper.
[Table]
D 30 R ANDOM T ESTING P OINTS FOR TAJS B ASELINE S TUDY
I D
G it hub
L i
nk t
o D i ff
F il e
B uggy
C od e
F i x e d C od e j s - a j a x - h itti ng - a p i s - l a bv - i nd e x .
j s login name r o m a 2h i r a c onv e r t .
j s ( elem ).
value ( elem ).
innerHTML
L e t s R o ll r ou t e r .
j s username userName
M E
AN a r ti c l e s .
s e r v e r .
r ou t e .
j s app .
params app .
param j s - do m - a nd - e v e n t sac ti ng - on - e v e n t s - l a b - vi nd e x .
j s ( elem ).
InnerHTML ( elem ).
innerHTML r n - m it r a i sm b
L i s t
A l bu m s .
j s info .
type info .
album _ type
R eac t - Q u i z C
o m pon e n t
Q u i z
Q u e s ti on .
j s ( obj ).
instructio n_ texts ( obj ).
instruction _ text m u s k e t ee
r- s hop
o r d e r .
j s
DECIMAL
INTEGER
A L F
A C h a r t s c
r
o ss h
a
i r s .
j s
Math .
floor
Math .
round h
ac k c
i n c i nn a ti/ s it e
A dv i s o r s .
j s color .
primary color .
accent r eac t - n a ti v ew it h -r e dux -r eac t - n a v
i g a ti o nv2 - bo il e r p
l
a t e s p l a s h .
j s
COLOR .
DARK
COLOR .
PANTOME
A l o C h a t
C on t a i n e r .
j s
PropTypes .
string
PropTypes .
object p a ndo r av a li d a ti on i nd e x .
j s addAsyncSetup addSyncSetup
O r ca z .
j s erase () explode ()
i o t db - m ongodb c oun t .
j s then make ﬂ o m -r eac t qu e s ti on .
j s
DataTypes .
STRING
DataTypes .
TEXT j 5 - l e d s b li nk .
j s blink strobe
C r a x i d i s p l a y .
j s game .
draw () game .
run () g r a ph - j s po i n t .
j s data dataManager m a ti c .
j s g e t
ET H
F r o m
F a u ce t .
j s
ETHFaucetAddress
ETHFAUCET _
ADDRESS z u li p s t r ea m _ m u ti ng .
j s add _ messages add _ old _ messages
W T FA dv e n t u r e m a n a .
j s this .
_ super this .
super s i m p l e - c
r a f ti ng g a t
h e r .
j s getBehavior
getMeta g ee k
T a l k s i nd e x .
j s checkTalkOwnersh ip checkCommentOwnership p
i zza - t o t a
ll y -r o c k s
F o r m .
j s getRecipe getRecipes koo t b e f o r e _ r ou t e r _ m a t c h .
j s origin originTrue f o_ r e s t i nd e x .
j s testRecipes preference c
r yp tii
R O T 13 .
j s registerSetting ( ) addSetting ()
D i s c o r d W it
h D a t a b
a s e h e l p .
j s print () printSplit () o r g .
c i v i c r m .
c i v
i ca s e
C a s e
D e t a il s F il e
T a b .
j s areAvailable () isAllowed ()
I D
B ug
T yp e
T A J S
H O
PP I T Y
F a l s e
A l a r m
P r e d i c ti on
T op
T op
T op  und e ﬁ n e d p r op e r t y und
e ﬁ n e d p r op e r t y
F a il e d doc .
getElem () und
e ﬁ n e d p r op e r t y
F a il e d i m po r ti ng
li b r a r y util und
e ﬁ n e d p r op e r t y
U nd e ﬁ n e d process app .
params (’.
) (cid:88) und
e ﬁ n e d p r op e r t y
F a il e d doc .
getElem () und
e ﬁ n e d p r op e r t y
U nd e ﬁ n e d process und e ﬁ n e d p r op e r t y f un c ti on a l bug f un c ti on a l bug f un c ti on a l bug f un c ti on a l bug f un c ti on a l bug f un c ti on a l bug serviceProvider .
addAsyncSetup (..
.) (cid:88)
f un c ti on a l bug f un c ti on a l bug
F a il e d i m po r ti ng
li b r a r y events f un c ti on a l bug f un c ti on a l bug led .
strobe ( ) (cid:88) f un c ti on a l bug this .
game .
draw () (cid:88)
r e f ac t
o r i ng r e f ac t o r i ng r e f ac t
o r i ng r e f ac t o r i ng r e f ac t
o r i ng
F a il e d i m po r ti ng
li b r a r y r e f ac t
o r i ng
F a il e d i m po r ti ng
li b r a r y r e f ac t
o r i ng r e f ac t o r i ng r e f ac t
o r i ng exports .
testRecipes = ...
(cid:88) r e f ac t
o r i ng r e f ac t o r i ng r e f ac t
o r i ng
F a il e d i m po r ti ng
li b r a r y