F IX B UGS WITH T RANSFORMER THROUGH A N EURALS YMBOLIC E DIT G RAMMAR
A BSTRACT
We introduce NSEdit (neural-symbolic edit), a novel Transformer-based code repair method.
Given only the source code that contains bugs, NSEdit predicts an editing sequence that can ﬁx the bugs.
The edit grammar is formulated as a regular language, and the Transformer uses it as a neural-symbolic scripting interface to generate editing programs.
We modify the Transformer and add a pointer network to select the edit locations.
An ensemble of rerankers are trained to re-rank the editing sequences generated by beam search.
We ﬁne-tune the rerankers on the validation set to reduce over-ﬁtting.
NSEdit is evaluated on various code repair datasets and achieved a new state-of-the-art accuracy ( 24 . 04% ) on the Tufano small dataset of the CodeXGLUE benchmark.
NSEdit performs robustly when programs vary from packages to packages and when buggy programs are concrete.
We conduct detailed analysis on our methods and demonstrate the effectiveness of each component.
I NTRODUCTION
Neural networks pretrained on source code (Feng et al., 2020; Kanade et al., 2020;
Guo et al., 2020;
Ahmad et al., 2021) are bringing substantial gains on many code understanding tasks such as code classiﬁcation, code search, and code completion (Lu et al., 2021).
However, code repair remains challenging because it requires the model to have robust syntactic and semantic understanding of a given program even when it contains bugs.
The difference between a buggy program and its ﬁxed version often lies in small details that must be ﬁxed exactly, which further requires the model to have precise knowledge about programming.
Code repair with large language models often formulate the problem as a Neural Machine Translation problem (NMT), where the input buggy code is “translated” into its ﬁxed version as the output.
We categorize existing work based on two design factors: the translation target and the buggy code representation .
For the translation target, the model can predict the ﬁxed code directly (Tufano et al., 2019; Phan et al., 2021; Yasunaga & Liang, 2021) or generate some form of edit that can be applied to the buggy code to ﬁx it (Yao et al., 2021;
Chen et al., 2019; Zhu et al., 2021;
Yin et al., 2018).
For the buggy code representation, we can use a tokenized sequence of buggy code for sequence-to-sequence (Seq2Seq) prediction (Chen et al., 2019; Bhatia et al., 2018) or program analysis representations such as abstract syntax tree (AST), data-ﬂow graph (DFG), error messages and build conﬁgurations (Yao et al., 2021; Allamanis et al., 2021;
Berabi et al., 2021; Tarlow et al., 2020).
We propose NSEdit, a Transformer-based (Vaswani et al., 2017) model that predicts the editing sequence given only the source code.
We use both encoder and decoder of the Transformer: the encoder processes the buggy code, and the decoder predicts the editing sequence given an edit grammar.
We design the edit grammar as a regular language, and the Transformer uses it as a domain-speciﬁc language (DSL) to write scripts that can ﬁx the bugs when executed.
The grammar consists of two actions, delete and insert , which are added to the vocabulary of the language model as new tokens.
The decoder has two modes: word/action mode predicts the two action tokens along a r X i v : . 06643v [ c s .
L G ] A p r with word tokens, and location mode selects the location of the edit. A pointer network implements the location selection mode, and we slice the encoder memory as the embedding of the edit location to enable content-based retrieval, instead of representing a location as a static word embedding in the vocabulary.
We use beam search to generate predictions at inference time.
Given diversityversus-quality problem encountered in all beam-search-based sequence generation methods (Kool et al., 2019), we train rerankers to improve the quality of sequences generated (Ng et al., 2019;
Lee et al., 2021; Ramesh et al., 2021).
An overview of the architecture is provided in Figure 1.
We will publish our source code.
We now introduce our decisions and hypotheses with respect to the two design factors of translation target and buggy code representations in the context of existing literature.
Translation target Using the ﬁxed code as translation target is straight-forward to implement, with the added beneﬁt that input and target are both code sequences, which can be easier to model and, compared to edits, more similar to the code corpus that large language models are pre-trained on.
However, as our results show, predicting the ﬁxed code directly may encourage the model to learn the copying behavior which causes overﬁtting and does not reﬂect the goal of editing the code to make changes.
Existing edit prediction approaches often rely on a graph representation of the buggy code (Yao et al., 2021).
This implicitly assumes that there exists a graph representation for the buggy code in the ﬁrst place, which may not be true if the bug causes syntactical errors.
Existing edit prediction approaches may also require signiﬁcant architecture design and involve multiple stages of edit prediction (Zhu et al., 2021;
Hashimoto et al., 2018;
Yin et al., 2018).
Instead, we design an edit grammar and rely on Transformer to do what it does best as a language model: learn the edit grammar and output editing instructions with this grammar.
As to the architectural changes, we add a pointer network to predict the location of an edit, which is an essential modiﬁcation.
Our use of rerankers is an orthogonal change that adds to the performance signiﬁcantly but not the complexity of the main Transformer model.
The edit grammar can be seen as a DSL for edits, given which the
Transformer generates a short program to edit the input sequence.
This places our work alongside the burgeoning neural-symbolic literature to use neural networks to write executable scripts (Chen et al., 2021; Mao et al., 2019), vastly expanding the algorithmic capacity of deep learning systems.
Buggy code representation We decide to train the model end-to-end given only the buggy code and ﬁxed code without any auxiliary program analysis information.
In light of recent ﬁnding that
Transformers are universal sequence-to-sequence function approximators (Yun et al., 2020), we want to prove that Transformers are powerful enough to learn the syntax and data ﬂow through pretraining on large code corpus and can do so robustly even when bugs are present in the input.
When syntactical structures are not given, the Transformer model has to learn the syntax, which may provide additional signal that helps the model learn the semantics of the code corpus (Manning et al., 2020).
When human programmers debug, they look at the code directly and only use AST/DFG as a mental model, and, as our results show, Transformer can learn to edit the code directly as well.
Representing buggy code with program analysis graphs can incorporate important static and dynamic analysis information.
However, for code repair in particular, bugs may cause syntactical errors that prevent extraction of program analysis graphs.
Program analysis tools may impose restrictions on the inputted programs (e.g. language-dependent tools), while our problem formulation is more general and portable.
Lastly, it is a known challenge that neural program models encounter generalizability issues when semantic-preserving program transformations are encountered (Rabin et al., 2021).
The main contributions of our paper are: (1) Our proposed method NSEdit achieves the state-ofthe-art performance on the CodeXGLUE code repair benchmark (Lu et al., 2021).
We show that pre-trained Transformer, given only the buggy code without program analysis representation or auxiliary information, can reach SOTA performance in code repair formulated as a sequence-tosequence neural machine translation problem.
(2) We formulate NSEdit grammar that is a regular language and one of the simplest edit representation in the literature.
The two-mode decoder and ﬁnite state machine together ensure that the model follows the grammar.
The Transformer uses the grammar as a neural-symbolic API to generate executable scripts that edit the code.
We show that predicting editing sequences leads to superior performance, even when programs vary from packages to packages.
(3) We use a pointer network to achieve content-based edit location selection.
We slice the encoder memory to obtain the latent representation of a potential edit location.
(4) We use an ensemble of rerankers to re-order the top- K editing sequences produced by beam search and signiﬁcantly improve all exact match accuracy.
We apply a novel technique to ﬁne-tune the rerankers on validation set which effectively reduces over-ﬁtting of rerankers.
(5) We show that the reranking score can be used to improve the precision of editing sequences with efﬁcient trade-off of recall.
R ELATED WORK
Code repair with deep learning In addition to the code repair methods we discussed in the Introduction, we see a diverse array of methods proposed in recent years.
Getaﬁx (Johannes et al., 2019) presents a novel hierarchical clustering algorithm that summarizes ﬁx patterns into a hierarchy and uses a simple ranking technique based on the context of a code change to select the most appropriate ﬁx for a given bug.
Vasic et al. (2019a) presents multi-headed pointer networks to localize and ﬁx the variable misuse bugs.
Recently, Dinella et al.
(2020) learns a sequence of graph transformations to detect and ﬁx a broad range of bugs in Javascript: given a buggy program modeled by a graph structure, the model makes a sequence of predictions including the position of bug nodes and corresponding graph edits to produce a ﬁx.
DeepDebug (Drain et al., 2021) trains a backtranslation Transformer model and uses various program analysis information obtained from test suites to ﬁne-tune the model.
Neural machine translation with Transformer Transformer and its derivative models such as
BERT (Devlin et al., 2018) and GPT (Radford et al., 2018) form a family of large language models that dominate neural machine translation and deep natural language processing.
The models consists of many parameters, and the performance of the model scales with the size of the model (Brown et al., 2020).
Recently, Transformer has been adapted to domains other than natural language processing, such as image classiﬁcation (Dosovitskiy et al., 2020) and protein structure modeling (Rao et al., 2021), demonstrating Transformer as a general-purpose architecture.
M ETHODS
Problem setup Our code repair dataset contains pairs ( x, y ) of strings, where x is the source code that contains bugs (buggy code), and y is the ﬁxed code.
Given the buggy code as the input sequence x , the goal of NSEdit is to generate the correct sequence of edits e that can transform x into y .
For notations, we use variables such as x to denote strings or constants.
We use bold variables such as x to denote tokens and tensors.
We use hatted variables such as ˆ x or ˆ x to denote model predictions.
We use uppercase bold variables such as X to denote a probability distribution.
Overview of NSEdit Training Training NSEdit consists of a pipeline of stages.
We tokenize the buggy-ﬁxed pair ( x, y ) with pre-trained CodeBERT tokenizer before obtaining ground truth edits e .
We load the pre-trained CodeBERT encoder (Feng et al., 2020) and CodeGPT decoder (Lu et al., 2021).
We ﬁne-tune the NSEdit model f to predict editing sequences with teacher forcing.
After training the model, we use beam search to generate the top-5 editing sequences (hypotheses).
We train two rerankers with different architectures to classify which editing sequence is correct among the beam search hypotheses.
The two rerankers and the original beam search score are combined with an ensemble model to produce the ﬁnal reranking.
Lastly, we ﬁne-tune the rerankers on the beam search hypotheses on the validation set to reduce over-ﬁtting.
The beam search hypotheses reranked by the ﬁne-tuned ensemble are the ﬁnal predictions.
3. T OKENIZATION AND EDITING SEQUENCE GENERATION
We tokenize the both buggy code and ﬁxed code with a Byte Pair Encoding (BPE) (Sennrich et al., 2015) tokenizer that is pre-trained on CodeBERT code corpus.
To compute the editing sequences, we use a variation of Ratcliff-Obershelp algorithm (Ratcliff & Metzener, 1988) implemented in
Python’s difﬂib library.
The editing sequences are computed on tokenized sequences instead of raw strings.
To formulate the NSEdit grammar formally, an editing sequence consists of two types of actions: delete ( i, j ) and insert ( i, s ) .
The delete ( i, j ) action deletes the subsequence in [ i, j ) from the buggy sequence x .
The insert ( i, s ) action inserts a sequence of tokens s before location i in the buggy sequence x .
As a result, NSEdit grammar contains three types of tokens in an editing sequence: action, word and location tokens.
The ﬁnite state machine that describes NSEdit grammar is provided in Figure 2.
An example editing sequence can be [DELETE] [LOC_1] [LOC_2] [INSERT] [LOC_2] @Override public .
More example editing sequences are provided in Appendix E.
ACTION BOS
INS AT
EOS DEL FROM WORD DEL TO [BOS] [ E O S ] [INSERT] [ D ELETE ] [LOC l ] [LOC l ] [LOC l ] [ D ELETE ] [INSERT] [ E O S ] [ w ]
3. T RAINING T RANSFORMER TO PREDICT BUG FIX EDITING SEQUENCES WITH TEACHER FORCING
We use the Transformer model (Vaswani et al., 2017) to perform sequence-to-sequence prediction.
The NSEdit model computes f ( x ) =
ˆ e , where x is the buggy token sequence and ˆ e is the predicted editing sequence.
The encoder processes buggy code x and outputs the encoder memory m , formally shown in Equation 1.
For input x with L tokens and model with h hidden units, the encoder memory has shape ( L, h ) , omitting the batch dimension.
The decoder takes m and the current editing sequence token e i as the input and autoregressively predicts the next token
ˆ e i +1 by maximum likelihood, as shown in Equation 2 and 3, where [ · ] denotes the slicing operator in Python. [Equation] [Equation]
[Equation]
We use teacher forcing as the training procedure (Williams & Zipser, 1989; Lamb et al., 2016).
This means that in Equation 2, the ground truth edit token
e i is inputted into the decoder, but not the predicted token ˆ e i .
For Seq2Seq models, teacher forcing decouples prediction ˆ e i from ˆ e i +1 during back propagation, thus it is more robust against vanish/exploding gradient problems common in recurrent neural networks.
We ﬁne-tune pre-trained CodeBERT (Feng et al., 2020) and CodeGPT
(Lu et al., 2021).
We modify the decoder to have two modes, a word/action mode that predicts edit actions and inserted words, and a location mode that predicts edit locations.
The original CodeBERT tokenizer has 50265 word tokens in the vocabulary, and we add [DELETE] and [INSERT] tokens to the vocabulary.
When predicting words or actions, the decoder outputs a probability vector ˆ w over a set W of 50267 elements by passing the logits output c into the softmax function, shown in Equation 4.
[Equation]
When predicting locations, instead of further expanding the vocabulary to add 513 location tokens and predict them along with words and actions, the decoder uses a pointer network in place of the last layer of the decoder (Figure 1).
The pointer network is a feed forward neural network.
It transforms the output from the penultimate layer of the decoder into a latent representation v (Vinyals et al., 2015; Vasic et al., 2019b).
In order to determine the location of the edit, we compute the dot product between v and m before a softmax function over all edit locations, as shown in equation 5.
As the result, the pointer network outputs a probability vector ˆ L over all edit locations at index 0 , 1 , 2 ...L for a buggy code with L tokens.
[Equation]
Since ground truth is available with teacher forcing, we determine which decoder mode to use given the type of ground truth token
e i +1 .
We slice the encoder memory as the embedding m [ l ] to replace the embedding of a location token [LOC l ] as the input to the decoder in Equation 2.
As the result, the input m [ l ] and output v of the decoder for locations are both content-based representations, rather than a ﬁxed location embedding that does not change when location context changes with the input program.
We use cross entropy loss for both word/action prediction and location prediction and add them together with equal coefﬁcients.
3. G ENERATING BEAM SEARCH HYPOTHESES DURING INFERENCE
During inference, when ground truth is not available, we generate sequence predictions with beam search (Reddy et al., 1977; Graves, 2012; Sutskever et al., 2014).
Every partially generated sequence is assigned a probability Π that is the product of every token probability, and the Top- K most probable editing sequences (hypotheses) are outputted, where K = 5 .
Formal deﬁnitions of our beam search procedure is provided in Appendix A.
The ﬁnite state machine can uniquely determine the next token type based on the previous token given the transition function in Figure 2.
Formally, we modify Equation 2 to use fsm to determine the token type as an input to the decoder
[Equation]
We mask the probability of an invalid token to be zero, thereby ensuring valid NSEdit grammar syntax.
We slice the encoder memory based on the predicted edit location ˆ e i during inference.
Formally, when the current input is an edit location, ˆ e i = arg max j =0 , 1 , 2 ..
L ˆ E i [ j ] = [LOC l ] = m [ l ] in Equation 6.
3. R ERANKING THE BEAM SEARCH HYPOTHESES
Our results show that top-5 accuracy is signiﬁcantly higher than top-1 accuracy (Table 6), meaning that the correct edit can be produced among the 5 beam search hypotheses but not ranked the most probable by the original beam search probability Π in Equation 11.
Beam search with models trained with teacher-forcing can produce a diverse set of hypotheses, and the quality of the hypotheses may be improved (Kool et al., 2019).
To do so, we rerank the beam search hypotheses with rerankers (Ng et al., 2019; Lee et al., 2021).
We formulate this problem as a classiﬁcation problem: given K hypotheses produced by the beam search that have a correct prediction, the objective of the reranker is to classify which of the K hypotheses is the correct one.
The reranking score for a hypothesis ˆ e k is computed as
[Equation]
where T = 0 .
5 is a temperature term that controls the smoothness of the probability distribution (Lee et al., 2021).
Note that each hypothesis ˆ e k goes through the same reranker function where each reranker (ˆ e k ) has a scalar output.
This model can be seen as a special case of Siamese model (Koch et al., 2015).
We use cross entropy loss to train rerankers on the beam search hypotheses that are produced on the training set by the main NSEdit model.
We train two rerankers with different architectures: one with both Transformer encoder and decoder, the same architecture as the main NSEdit model, and the other with encoder only.
The Transformer reranker outputs reranking score at the end of the sequence, and the encoder-only reranker outputs at the [BOS] token.
Formally, for a beam hypotheses ˆ e of length L , we compute the reranking scores reranker
[Equation]
reranker
[Equation]
where ˆ E L − 1 is computed given by Equation 6 and m by Equation 1.
The ff function is a simple one-layer feed-forward neural network.
For every beam search hypotheses ˆ e , we have three ranking scores: the original beam search log probability score Π (Equation 11 ) and two reranking scores.
We use a linear ensemble model to blend the ranking scores:
[Equation]
where c 1 , c 2 are hyperparameters to be tuned (Jahrer et al., 2010; Breiman, 1996).
To search for the best hyperparameters, we train the rerankers on the training set, and we pick the conﬁguration with the highest validation accuracy.
[Table]
Results show that rerankers tend to overﬁt on the training set.
To mitigate reranker’s overﬁtting issue, we ﬁne-tune the rerankers on the validation set for b epochs (Tennenholtz et al., 2018).
To tune hyperparameter b , we re-split the validation set by 75:25, ﬁne-tune the reranker on the 75% split for b epochs, and pick the b with the highest accuracy on the 25% split.
We see that tuning for one epoch ( b = 1 ) performs the best.
To produce the ﬁnal ﬁne-tuned ensemble reranker, we ﬁne-tune both rerankers on 100% of the validation set for one epoch, and combine them with the same ensemble hyperparameters c 1 , c 2 found before ﬁne-tuning.
E XPERIMENTS AND R ESULTS 4. NSE DIT ACHIEVES SOTA PERFORMANCE ON C ODE XGLUE CODE REPAIR BENCHMARK
NSEdit achieved the state-of-the-art (SOTA) performance (24.04%) on the Tufano et al.
(2019) code repair dataset as a part of the CodeXGLUE benchmark (Lu et al., 2021).
We report our results on Tufano datasets in comparison with other code repair methods (Feng et al., 2020; Guo et al., 2020; Ahmad et al., 2021; Phan et al., 2021) currently on the CodeXGLUE benchmark in Table 1.
Compared to other methods, our method NSEdit is the only one that predicts any form of edits, while all others predict ﬁxed programs directly.
Some example bug ﬁxes correctly produced by our
NSEdit model are provided in Appendix E.
In Table 1, the Baseline model is a Transformer with the CodeBERT encoder and a randomly initialized six-layer Transformer decoder.
Compared to the complete NSEdit model, Baseline has four differences: (1) the prediction target is ﬁxed code by default.
(2) the decoder does not have a pointer network for location mode.
(3) CodeGPT is not used to initialize the decoder.
(4) rerankers are not used.
We will reuse this Baseline model in the following experiments.
4. P REDICTING EDITING SEQUENCES PERFORMS BETTER THAN PREDICTING FIXED CODE
We formulate a novel NSEdit grammar to predict editing sequences as the target, rather than the ﬁxed code.
To conﬁrm that predicting editing sequences yields better performance, we initialize two
Baseline models without CodeGPT and rerankers, the same as in Section 4.1.
One Baseline model predicts editing sequences and the other predicts ﬁxed code.
The only difference in the architecture is the pointer network needed to predict locations in editing sequences.
We report the results on
Tufano abstract datasets in Table 2.
Predicting edits has better performance, possibly because editing sequence is shorter than ﬁxed code, and it discourages copying behavior by focusing on the changes.
[Table]
NSE DIT PERFORMS ROBUSTLY AGAINST PACKAGE - TO - PACKAGE VARIATIONS
We note that the Tufano et al.
(2019) training, validation and test sets have overlapping Java packages.
To investigate the effect of package-to-package variations, we curate an in-house dataset from the publicly available 10K Github Java packages (Allamanis & Sutton, 2013), which will be public.
Our dataset generation process resembles Tufano et al.
(2019): we partition the dataset given the buggy program length and normalize variable names, method names and type names in abstract code, while retaining the original names in concrete code.
We implement a strict policy to separate training, validation and test set packages.
Other than the same packages, closely related packages that share same naming preﬁx, e.g. “spring-cloud-stream-samples” and “spring-cloud-stream”, are considered related and also separated in either training, validation or test set.
The dataset consists of 138575/12983/9282 train/valid/test samples.
We report the accuracy of Baseline models in Table 3.
Recall that we hypothesize previously that predicting ﬁxed code directly encourages the model to learn the copying behavior.
We see that when packages are mixed, all models overﬁt with large gap between validation and test accuracy.
Furthermore, mixing packages causes all models to have reduced accuracy, likely because overﬁtting is a signiﬁcant performance bottleneck.
When predicting code, accuracy on concrete code (27.61%, row 3) is slightly better than accuracy on abstract code (26.38%, row 1) on validation set with mixed training/validation packages, but only half at test time [Table] [Table] with unseen packages (3.45% v.s. 9.78%), possibly because for concrete code, the model is given more context and it is easier to copy from similar programs
[Table]
which makes overﬁtting more severe. Predicting edits does not suffer the same performance drop at test time for concrete code (7.11%, row 4) compared to abstract code (9.07%, row 2), which supports our hypothesis that predicting edits discourages copying behavior.
When packages are separate and code is concrete, predicting editing sequences (9.04%, row 8) doubles the test time top-1 accuracy of predicting ﬁxed code directly (3.98%, row 7).
This is the most valuable use case in applications, and predicting edits has even greater advantage than predicting ﬁxed code.
4. NSE DIT IS A GENERAL BUG FIX METHOD IN DIFFERENT LANGUAGES AND SETTINGS
The NSEdit grammar is language agnostic.
To conﬁrm that NSEdit works on languages other than
Java, we evaluate our NSEdit model on ETH Py150 dataset (Kanade et al., 2020; Raychev et al., 2016).
The Python dataset contains ﬁve classiﬁcation tasks, and we experiment on three of them that are related to code repair: variable-misuse classiﬁcation, wrong binary operator, and swapped operand.
We process ETH Py150 according to our problem setup and report the performance of
NSEdit in Table 4.
We conﬁrm that NSEdit is a general method can edit buggy programs in different languages and settings.
4. P RE - TRAINED ENCODER AND DECODER CAN BE FINE - TUNED TOGETHER ON EDITING SEQUENCES
Fine-tuning pre-trained models on code repair contributes to the performance of our model.
To investigate the effect of different backbone models, we change the pre-trained backbones used to initialize the weights of the Transformer before ﬁne-tuning.
We report accuracy on Tufano datasets in Table 5.
The CodeBERT encoder backbone is trained on multiple programming languages, including Java.
Even when the input sequences contain bugs and program analysis auxiliary information is not pro- [Table] vided, CodeBERT can robustly extract program information and improve the performance of NSEdit after ﬁne-tuning.
The CodeGPT decoder backbone also improves the performance of NSEdit.
CodeBERT and CodeGPT are two independently published models pre-trained on different datasets [Table] and our results conﬁrm that even when large language models are pre-trained in different settings, they can be integrated into the same Transformer model.
Notably, CodeGPT is pre-trained on code [Table] and the model can effectively transfer knowledge to predict editing sequences.
4. R ERANKERS BRING SIGNIFICANT PERFORMANCE IMPROVEMENTS
The use of rerankers signiﬁcantly improves the performance of NSEdit to the state-of-the-art level.
We investigate the effect of different settings of rerankers in Table 6.
[Table]
NSEdit [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
The ﬁnal reranker ablation Fine-tuned ensemble achieves the best accuracy among all settings and is presented in the Methods section.
Notably, we see that the use of Transformer reranker has lower accuracy than without reranker.
Ensemble rerankers and validation-set ﬁne-tuning both improved performance separately and in combination.
Further details about reranker ablation experiment settings are provided in Appendix B.
4. E NSEMBLE RERANKER EFFICIENTLY TRADES OFF PRECISION AND RECALL
Rerankers rerank the beam search hypotheses as discussed in Section 3.4.
A beam search hypothesis has higher reranking score if it is predicted to be more likely.
Therefore, we can use the reranking score computed in Equation 10 as a conﬁdence metric to trade off precision and recall of the editing sequence generated by beam search.
If the reranking score is lower than a threshold, we discard the editing sequence predicted (negative), and otherwise we use it (positive).
We sweep the threshold parameter and generate the precision-recall trade-off plot for Tufano small abstract dataset in Figure 3.
Note that for accuracy reported previously, the model makes a prediction every time (always positive) and recall is 100%.
When the model is not required to make a prediction for every buggy sequence, precision can be improved with trade-off of recall by using the reranking score as a conﬁdence metric.
This tradeoff is usually desirable in bug ﬁx applications.
Speciﬁcally, when recall is lowered from 100% to 38.4%, precision can be increased from 24.04% to 50%.
In comparison, we also use beam search’s original sum of log probability score computed in Appendix Equation 11 as the conﬁdence metric, and we see that reranking score is more efﬁcient because its recall is higher at every precision level.
Speciﬁcally, the original score’s recall needs to be lowered to 22.4% in order to reach the same precision level of 50%, which is almost half of the reranking score
’s recall.
This result additionally conﬁrms that our rerankers learn an extrinsic signal that differs from the main model’s intrinsic conﬁdence.
D ISCUSSIONS
NSEdit has achieved a new state-of-the-art performance on the code repair task of the CodeXGLUE benchmark (Lu et al., 2021; Tufano et al., 2019).
For code repair, it is more effective to predict editing sequences than ﬁxed code.
As closing thoughts, we want to draw attention that our edit grammar generates a neural-symbolic interface that is a special case of a general paradigm.
Integration of deep learning systems with traditional symbolic systems can be achieved by formulating a domain-speciﬁc language (DSL) and training a Transformer to use the DSL as a programmable interface.
A B EAM SEARCH FORMAL DEFINITION
During beam search, we maintain K incomplete subsequences (i.e. beams)
B i,k , where B 0 ,k
= [ [BOS] ] for all k = 1 , 2 ...K .
For any B i,k , we compute the probability
Pr(ˆ e i +1 ,k | ˆ e 1 ,k , ˆ e 2 ,k , ...
ˆ e i,k ) =
ˆ E i +1 ,k
[ˆ e i +1 ,k ] given Equation 6.
The probability of the concatenated sequence B i +1 ,k
= B i,k + [ˆ e i +1 ,k ] =
[ˆ e 1 ,k , ˆ e 2 ,k , ...
ˆ e i,k , ˆ e i +1 ,k ] is the product of the probabilities of all tokens as they are generated.
Formally,
[Equation]
We denote this product as Π in Equation 10.
Taking logarithm on both sides, we have
[Equation]
We take the top- K most probable tokens ˆ e i +1 ,k ∈ W for all beam k = 1 , 2 , ...
K , given
[Equation]
where any previous step beam may have multiple concatenated beams in top- K with different tokens.
The selected top- K tokens
ˆ e i +1 ,k form a new set of K beams for the next iteration, until [EOS] is predicted or maximum length is reached.
Note that since arg topK is applied to { 1 , 2 , ...K } × W , it is possible for different tokens to append to the same previous beam and are all included in the top- k beams for the next step.
Also note that beam B i +1 ,k may not contain beam B i,k as a subsequence.
B R ERANKER ABLATION EXPERIMENTS DETAILS
In this section, we provide more details on the settings of the reranker ablation experiments presented in Table 6.
The NSEdit without rerankers version does not use rerankers and directly reports beam search accuracy based on the original beam search score Π in Equation 11.
We compare the reranked accuracy with this baseline.
The Transformer version trains a single Transformer reranker to rerank the beam search hypotheses, ignoring the original beam search score.
We see that the accuracy is lower than the accuracy of NSEdit without rerankers .
The Ensemble version trains both rerankers and blends the three ranking scores by optimizing validation accuracy.
We see that the ensemble reranker improves over Transformer reranker but not better than NSEdit without rerankers .
The Fine-tuned Transformer version trains a single Transformer reranker and tunes it on validation set for one epoch.
We see that it signiﬁcantly outperforms the Transformer version without ﬁnetuning, which suggests that ﬁne-tuning on validation prevents the reranker from overﬁtting.
The accuracy of this ablation outperforms NSEdit without rerankers .
The ﬁnal version Fine-tuned ensemble uses a blended ensemble of two rerankers ﬁne-tuned on the entire validation set, as described in the Methods section.
C T RAINING AND MODEL HYPERPARAMETERS
We implement our Transformer architecture in PyTorch, except the pre-trained CodeBERT and
CodeGPT models, which we load from HuggingFace (Wolf et al., 2019) .
The learning rate is set to be 1e-4 multiplied by the number of GPUs.
When CodeGPT weights are loaded, we halve the learning rate of pre-trained parameters and quadruple the learning rate of randomly initialized parameters.
AdamW optimizer (Loshchilov & Hutter, 2017) with triangular learning rate scheduler is used in all experiments.
The NSEdit main model is trained for at most 60 epochs and early stopping is applied if the accuracy does not improve.
Automatic mixed precision (AMP) is enabled.
Training of the model together with rerankers on Tufano datasets takes around a day on a machine with
V100 Nvidia GPUs.
In beam search, a length penalized score is computed for the partially generated sequences at every step according to Wu et al.
After the scores are computed, the ﬁnite state machine set the invalid tokens to have zero probability according to the edit grammar as described in Section 3.3.
Rerankers are trained on buggy programs for which the beam search produces at least one correct editing sequence.
We train rerankers for 12 epochs, with the same learning rate setup as the main
NSEdit model.
We ﬁne-tune the reranker on validation set for 1 epoch with 1 GPU.
We ﬁnd the best hyperparameters with Ray tune (Liaw et al., 2018) or grid search.
The ensemble reranker on Tufano datasets have coefﬁcients reported in Table 7.
Each coefﬁcient is searched among 10 candidates in logarithmic interval [0 . 01 , 100] , then another 20 candidates in a narrower linear
[Table]
D T UFANO DATASET EDITING SEQUENCE STATISTICS
In Table 8, we summarize statistics about the ground truth editing sequences in Tufano et al.
(2019) datasets.
We see that the editing sequences in Tufano medium, compared to Tufano small dataset, have more number of edits, longer insertion length and longer overall editing sequence length by mean and median.
This suggests that the bug ﬁxes in Tufano medium dataset are overall more difﬁcult to predict correctly.
[Table]
E E XAMPLE BUG FIXES
We provide some examples of bug ﬁx that are correctly produced by NSEdit in Figures 4 to 12.
[Figure]
[Figure]
The predicted editing sequence is [INSERT][LOC_6][][DELETE][LOC_7][LOC_9][INSERT][LOC_9]) @Override public Iterator<?
> downstreams(){ WindowGroupedFlux<T> g=window; + if (g == null) { - if (g == null) return Collections.emptyList().iterator(); ?
+ return Collections.emptyList().iterator(); + } return Collections.singletonList(g).iterator(); }
The predicted editing sequence is [DELETE][LOC_31][LOC_34][INSERT][LOC_34] {return[INSERT][LOC_41]}
[Figure]
Coding style improvement.
The predicted editing sequence is [DELETE][LOC_37][LOC_40][DELETE][LOC_64][LOC_66]
[Figure]
[Figure]
[Figure]
The predicted editing sequence is [DELETE][LOC_1][LOC_2][INSERT][LOC_2]@Override public @Override public boolean apply(PickleEvent pickleEvent){ String picklePath=
pickleEvent.uri; if (!lineFilters.containsKey(picklePath)) { - return true; ?
^^^ + return false; ? ^^^^ } for (
Long line : lineFilters.get(picklePath)) { for (
PickleLocation location : pickleEvent.pickle.getLocations (cid:44) → ())
{ if (line =
= location.getLine()) { return true ; } } } return false ; }
The predicted editing sequence is [DELETE][LOC_44][LOC_45][INSERT][LOC_45] false /** *Returns the preferred fragment size.
* @param format target format * @return the preferred fragment size * @throws IOException if failed to compute size by I/O error *
@throws InterruptedException if interrupted * @throws IllegalArgumentException if some parameters were {@code null (cid:44) → } */ public
[Equation] > format [Equation]
(cid:44) [Equation]
[Figure]
[Equation] [Equation] { if (
[Equation]
{ throw new IllegalArgumentException("format must not be null"); } [Figure] [Figure] - if
[Equation] [Figure]
[Equation] [Figure] getPreferredFragmentSize [Equation]
[Figure]
if [Equation] [Figure] [Equation]
[Figure]
[Equation]
[Figure]
The predicted editing sequence is [DELETE][LOC_140][LOC_141][INSERT][LOC_141] < [Figure] [Figure] ){ if
[Figure]
== null ) return false ; [Figure] [Figure] [Figure]
+
[Figure]
Wrong operator.
The predicted editing sequence is [DELETE][LOC_35][LOC_36][INSERT][LOC_36] && @Override public void run(){ this .ownerThread=
Thread.currentThread(); Log.debug("Starting event loop","name",name); setStatus(LoopStatus.BEFORE_LOOP); try { beforeLoop(); } catch ( Throwable e) { - Log.error("Error occured before loop is started","name",name,"error (cid:44) → ",e); +
Log.error("Error occurred before loop is started","name",name," (cid:44) → error",e); ? + setStatus(LoopStatus. FAILED); return ; }setStatus(LoopStatus.LOOP); while (status == LoopStatus.LOOP) { if (Thread.currentThread().isInterrupted()) { break ; } try { insideLoop(); } catch (
Throwable e) { Log.error("Event loop exception in " + name,e); } } setStatus(LoopStatus. AFTER_LOOP);afterLoop(); setStatus(LoopStatus. STOPPED);Log.debug("Stoppedeventloop","name",name); } Spelling error.
The predicted editing sequence is [DELETE][LOC_68][LOC_70][INSERT][LOC_70] occurred