SampleFix: Learning to Correct Programs by Eﬃcient Sampling of Diverse Fixes
Abstract
Automatic program correction holds the potential of dramatically improving the productivity of programmers.
Recent advances in machine learning and NLP have rekindled the hope to eventually fully automate the process of repairing programs.
A key challenge is ambiguity, as multiple codes – or ﬁxes – can implement the same functionality, and there is uncertainty on the intention of the programmer.
As a consequence, datasets by nature fail to capture the full variance introduced by such ambiguities.
Therefore, we propose a deep generative model to automatically correct programming errors by learning a distribution over potential ﬁxes.
Our model is formulated as a deep conditional variational autoencoder that can eﬃciently sample diverse ﬁxes for a given erroneous program.
In order to account for inherent ambiguity and lack of representative datasets, we propose a novel regularizer to encourage the model to generate diverse ﬁxes.
Our evaluations on common programming errors show strong improvements over the state-of-the-art approaches.
Extended version available at https://arxiv.org/abs/1906.10502 .
Introduction
Software development is a time-consuming and expensive process.
Unfortunately, programs are written by humans typically come with bugs.
Debugging is also typically performed by humans and can contain mistakes.
Therefore, automatically locating and correcting program errors oﬀers the potentials to increase productivity as well as improve the correctness of software.
Advances in deep learning [10–12], computer vision [6, 13] and NLP [2, 15] has dramatically boosted the machine’s ability to automatically learn representations of natural data such as images and language contents.
These advances in deep learning and the advent of large corpora of source code [1] provide new opportunities toward harnessing deep learning methods to understand, generate, or debug programs.
There has been an increasing interest to use deep learning to tackle the “common programming errors” (compile errors), which includes missing delimiters, extraneous symbols, and type errors.
Gupta et al.
[8] propose DeepFix, a sequence-to-sequence model to predict a ﬁx for the erroneous program.
RLAssist [7] repairs the programs by employing reinforcement learning, and Yasunaga and Liang [17] propose DrRepair to resolve the syntax error by introducing a program feedback graph.
In these works, the problem is formulated as a deterministic task, where the model is trained to predict a single location and ﬁx for each error.
However, diﬀerent codes – and their ﬁxes – can express the same functionality.
Besides, there is also uncertainty about the intention of the programmer (Figure 1).
Hence, we propose a generative framework to automatically correct programming errors by learning the distribution of potential ﬁxes.
We investigate diﬀerent solutions to model the distribution of the ﬁxes and sample multiple ﬁxes, including diﬀerent variants of Conditional Variation Autoencoders (CVAE) and beam search decoding.
It turns out CVAE and beam search decoding are complementary (section 3), while CVAE is computationally more eﬃcient in comparison to beam search decoding.
Figure 1 shows an example of the generated diverse and correct ﬁxes using our proposed approach.
To summarize, the contributions of this paper are as follows, 1.
We propose an eﬃcient generative method to automatically correct common programming errors.
We propose a novel regularizer to encourage the model to generate diverse ﬁxes.
Our generative model together with the diversity regularizer shows strong improvement over the state-of-the-art approaches.
SampleFix: Generative Model for Code Fixes
Repairing the common program errors is a challenging task due to the ambiguity in potential corrections.
Therefore, we propose a deep generative framework to automatically localize and repair errors by learning the distribution of potential ﬁxes – rather than a single ﬁx – given the erroneous program.
We address the problem of generating multiple ﬁxes using CVAE and beam search decoding.
In the CVAE formulation, we model the distribution of the ﬁxes using the conditional latent variable.
The distribution of latent variables is learned using pairs of erroneous programs and the corresponding ﬁxes.
Furthermore, to encourage diversity in the ﬁxes, we propose a novel regularizer that encourages the model to generate diverse samples by maximizing the distance among drawn ﬁxes.
In the inference time given an erroneous program, our model draws 𝑇 candidate ﬁxes from the learned distribution.
To select one out of 𝑇 candidates, we employ a compiler.
The compiler evaluates each ﬁx by compiling the updated program.
Among the ﬁxes, we select the ﬁx which resolves the largest number of error messages.
To resolve the remaining error(s), we follow Gupta et al.
[8] and iteratively input the updated program to the model.
In the following, we provide detail of our approach.
2.1 Conditional Variational Autoencoders for Generating Fixes
CVAEs [14] model conditional distributions 𝑝 𝜃 ( y | x ) using latent variables z .
The conditioning introduced through z enables the modelling of complex multi-modal distributions.
As powerful transformations can be learned using neural networks, z itself can have a simple distribution which allows for eﬃcient sampling.
This model allows for sampling from 𝑝 𝜃 ( y | x ) given an input sequence x , by ﬁrst sampling latent variables ̂
z from the prior distribution 𝑝 ( z ) .
During training, amortized variational inference is used and the latent variables z are learned using a recognition network 𝑞 𝜙 ( z | x , y ) , parametrized by 𝜙 .
In detail, the variational lower bound of the model is maximized,
[Equation]
Penalizing the divergence of 𝑞 𝜙 ( z | x , y ) to the prior in Equation 1 allows for sampling from the prior 𝑝 ( z ) during inference.
In practice, the variational lower bound is estimated as follow [14],
[Equation]
where, ̂ z i ∼ 𝑞 𝜙 ( z | x , y ) , and 𝑇 is the number of samples.
In our model, the input x is the erroneous program and y is the ﬁx that contains line of the error with the repaired line.
2.2 Enabling Diverse Samples using a Best of Many Objective
Casting our model in the CVAE framework would enable us to sample a set of candidate ﬁxes for a given erroneous program.
However, the standard variational lower bound objective does not encourage diversity in the candidate ﬁxes [14].
In detail, as the average likelihood is considered, all candidate ﬁxes must explain the “true” ﬁx in training set well.
To encourage diversity, we employ "Best of
Many Samples" (BMS) objective proposed by Bhattacharyya et al.
[Equation]
Compared to Equation 2, this objective (Equation 3) encourages diversity in the model by considering highly likely candidate ﬁxes.
This enables the model to generate diverse and accurate candidates [3].
2.3 DS-SampleFix: Encouraging Diversity with a Diversity-sensitive Regularizer
To increase diversity using Equation 3 , we need to use around 𝑇 = 10 samples during training.
This is computationally prohibitive especially for large models, as it requires 10 times the memory ortimes the number of forward passes.
If 𝑇 is decreased, the objective behaves similarly to the CVAE objective.
Therefore, in order to encourage the model to generate diverse ﬁxes even when high values of 𝑇 are computationally prohibitive, we propose a novel regularizer that explicitly encourages the two closest candidate ﬁxes to have maximum distance (Equation 4).
This penalizes generating similar candidate ﬁxes for a given erroneous program and thus encourages diversity in the set of candidate ﬁxes.
In comparison to Equation 3, we observe considerable gains with the use of only 𝑇 = 2 samples,
[Equation]
Note that the samples { ̂ y i , ̂ y j } can be of diﬀerent lengths.
Therefore, we pad the samples to equalize lengths.
In practice, we ﬁnd that the Euclidean distance performs best for the metric 𝑑 in Equation 4.
2.4 Beam Search Decoding for Generating Fixes
Beam search decoding is classical model to generate multiple outputs from a sequence-to-sequence model [16, 5].
In our generative model, we employ the beam search decoding to sample more diverse ﬁxes.
In detail, to sample multiple ﬁxes we decode with beam width of size 𝐾 for each sample z .
2.5 Model Architecture and Implementation Details
To ensure a fair comparison, our generative model is based on the sequence-to-sequence architecture, similar to Gupta et al.
Figure 2 shows the architecture of our approach in detail.
Note that the recognition network is available to encode the ﬁxes to latent variables z only during training.
All of the networks in our framework consists of 4-layers of LSTM cells with 300 units.
The network is optimized using Adam optimizer [9] with the default setting.
We use 𝑇 =samples to train our models, and 𝑇 = 100 samples during inference.
During inference, the conditioning erroneous program x is input to the encoder, which encodes the program to the vector v .
To generate multiple ﬁxes using our decoder, the code vector v along with a sample of z from the prior 𝑝 ( z ) is input to the decoder.
For simplicity, we use a standard Gaussian  (0 , I ) prior.
The decoder is unrolled in time and output logits ( 𝑝 𝜃 ( y |
̂ z i , x ) ).
To resolve multiple errors we use the iterative repair strategy as introduced by Gupta et al.
Experiments
We evaluate our approach on the task of repairing common programming errors and compare it with the state-of-the-art approaches.
3.1 Dataset
We use the dataset published by Gupta et al.
[8] as it’s sizable and includes real-world data.
It contains
C programs written by students in an introductory programming course.
The programs were collected using a web-based system [4].
These programs contain typographic and missing variable declaration errors.
The dataset contains two sets of data, which are called synthetic and real-world data.
The synthetic data contains the erroneous programs which are synthesized by mutating correct programs.
The real-world data contains 6975 erroneous programs with 16766 error messages written by students.
We use synthetic data to train and validate our model, and we use real-world data to test our model.
[Table]
Speed denotes the computational time for sampling 100 ﬁxes ﬁxed
3.2 Evaluation
We evaluate our approach on real-world set of erroneous programs.
In the real-world data, we don’t have access to the intended ﬁx(es).
However, we can check the correctness of the program using the compiler.
In Table 1 we compare our approaches with the state-of-the-art approaches.
In this table (Table 1) we show the performance of DeepFix+ BS (beam search), SampleFix, and our DSSampleFix.
Furthermore, we show that DS-SampleFix can still take advantage of beam search.
In order to do that, for each sample 𝑧 we decode with beam width of size 5, and to sample 100 ﬁxes we draw 20 samples from 𝑝 ( 𝑧 ) .
We also provide the sampling speed in terms of sampling 100 ﬁxes for a given program using an average over 100 runs.
Table 1 shows that our approaches outperform DeepFix [8], RLAssist [7], and DrRepair [17] in resolving the error messages.
This shows that generating multiple diverse ﬁxes can lead to substantial improvement in performance.
SampleFix, DS-SampleFix, and DS-SampleFix + BS (beam search) resolve 56.3%, 61.0%, and 65.2% of the error messages respectively.
Furthermore, the performance advantage of DS-SampleFix over SampleFix shows the eﬀectiveness of our novel regularizer.
Note that the concurrent work DrRepair [17] has achieved further improvements by relying on the compiler.
While utilizing the compiler output seems to be beneﬁcial, it also limits the generality of the approach.
For a fair comparison, we report the performance of DrRepair without the compiler output, but consider informing our model by the compiler output an interesting avenue of future work.
Qualitative Example.
We illustrate diverse ﬁxes generated by our DS-SampleFix in Figure 3 using an example with typographic errors with the corresponding drawn outputs of DS-SampleFix.
In this example, there is a missing closing curly bracket after line 12.
We can see that DS-SampleFix generates multiple correct ﬁxes to resolve the error in the given program.
This indicates that our approach is capable of handling inherent ambiguity and uncertainty in predicting ﬁxes.
More examples can be found in Appendix A.
Conclusion
We propose a novel approach to correct common programming errors.
We recognize and model the inherent ambiguity and uncertainty with predicting multiple ﬁxes.
In contrast to previous works, our novel approach is able to learn the distribution over candidate ﬁxes rather than the most likely ﬁx.
We propose a novel diversity-sensitive regularizer that helps us overcome the inherent limitations of supervised datasets.
We show that our approach is capable of generating multiple correct ﬁxes.
Furthermore, our evaluations show improvements over state-of-the-art approaches.
Appendix A Additional Qualitative Examples
Here we provide additional examples of repairing erroneous programs using our proposed approach.
Figure 4 and 5 show examples of the generated multiple correct ﬁxes using DS-SampleFix.
In the code example given in Figure 4, there is a missing closing curly bracket.
In this example
DS-SampleFix generates four correct ﬁxes for the given erroneous program.
Among these ﬁxes, the third and fourth ﬁxes express the same functionality.
In the code example given in Figure 5, there is an extraneous closing curly bracket.
In this example, we can see DS-SampleFix generates two correct ﬁxes that express diﬀerent functionalities.
Note that the model outputs a program location along with the corresponding ﬁx.
Figure 6 and 7 show code examples where DS-SampleFix repairs multiple typographic and missing variable declaration errors.
In Figure 6, we can see that DS-SampleFix resolves two typographic errors at line 10 and 16.
In Figure 7, DS-SampleFix resolves missing variable declaration errors by deﬁning variables n , eof , and m .
Figures 8 and 9 show two examples of resolving multiple errors using itrerative repair strategy, here we show 5 samples out of 100 drawn samples.
In Figure 8a we can see that in the ﬁrst iteration our approach resolves a typographic error at line 9, and in the second iteration (Figure 8b), after updating the program, it resolves another typographic error at line 11.
Figure 9 show an example of iteratively resolving multiple missing variable declaration errors.
Figures 9a and 9b show resolving multiple errors by deﬁning variables i and j respectively.
Note that the outputs contain only _eos_ refer to the empty string.
[Figure] [Figure]