Fix-Filter-Fix: Intuitively Connect Any Models for Effective Bug Fixing (2021)

Abstract: Locating and fixing bugs is a time-consuming task. Most neural machine translation (NMT) based approaches for automatically bug fixing lack generality and do not make full use of the rich information in the source code. In NMT-based bug fixing, we find some predicted code identical to the input buggy code (called unchanged fix) in NMT-based approaches due to high similarity between buggy and fixed code (e.g., the difference may only appear in one particular line). Obviously, unchanged fix is not the correct fix because it is the same as the buggy code that needs to be fixed. Based on these, we propose an intuitive yet effective general framework (called Fix-Filter-Fix or Fˆ3) for bug fixing. Fˆ3 connects models with our filter mechanism to filter out the last model’s unchanged fix to the next. We propose an Fˆ3 theory that can quantitatively and accurately calculate the Fˆ3 lifting effect. To evaluate, we implement the Seq2Seq Transformer (ST) and the AST2Seq Transformer (AT) to form some basic Fˆ3 instances, called Fˆ3_ST+AT and Fˆ3_AT+ST. Comparing them with single model approaches and many model connection baselines across four datasets validates the effectiveness and generality of Fˆ3 and corroborates our findings and methodology.

## introduction locating and repairing bugs of programs automatically is important in software engineering. many approaches @xcite @xcite based on the neural machine translation (nmt) have achieved promising performance for semantic bug fixing. the basic idea is to automatically translate a buggy code fragment into a fixed patch. however, there still exist some limitations. most of them (1) do not fully exploit the information of source code and only partly using the textual or structured information; (2) are single model architectures with poor generality. we find that many current nmt-based bug fixing models often predict exactly the same output code as the input buggy codes as figure 1 shows (see red lines), which we call unchanged fix. the input code is buggy, while the unchanged fix does not make any changes to the buggy code, so the unchanged fix is a failed fix obviously. it is because the buggy code and fixed code are always very similar and the vocabularies of the buggy and fixed code are closely overlapped in bug fixing. this phenomenon may also exist in many tasks with similar vocabularies before and after translation, such as automatic post-editing and text style transfer. in fact, this is an unsupervised phenomenon, i.e., without comparing the ground-truth fixed code, it is possible to determine directly whether the prediction is correct by knowing only the result of the model prediction and the input buggy code. this has led to the following question we aim to answer in this paper: "can we filter out those unchanged fixes and then refine the bug fixing process with a different model?". a similar scenario also exists when revising a paper. multiple revisions can always find more errors than single revisions. intuitively, in bug fixing, multiple models in tandem can provide better fixing than a single model does. based on the above observation, we propose a general and intuitive framework for bug fixing (called fix-filter-fix or f 3 ) with high performance and marginal extra cost. f 3 uses a filter mechanism to connect several different learners (individual models for bug fixing). the filter mechanism needs to directly filter out some buggy code fragments that a learner fails to fix without checking the ground-truth fixed code in the dataset, and then feed the filtered buggy code into the next learner. each learner in f 3 should be able to fix a portion of the buggy code that others cannot fix. our filter mechanism in this paper compares a learner's predicted results with the input buggy code, filters out the unchanged fix, and continues the corresponding buggy code to the next learner for processing, but it may not an optimal filter mechanism. an optimal filter mechanism can filter out all the buggy code fragments that a learner fails to fix. it is intuitive that f 3 can improve performance. to make the intuition more precise, we propose a theory to precisely calculate the specific performance improvement of f 3 combined with multiple learners without experimental verification. since source code contains textual and structural information, we apply seq2seq transformer (st) to fix bugs based on the textual representation of code, while the ast2seq transformer (at) is based on abstract syntax tree (ast), the structural information of code. we connect these two learners in different orders to implement f 3 instances, called f 3 st +at and f 3 at +st . we compare its performance with the single model baselines @xcite and model connection baselines on four datasets transformed by bfp and codrep datasets @xcite . experimental results demonstrate that our f 3 st +at outperforms all the baselines at a low cost. then we experimentally investigate the effects when using different orders or number of learners on f 3 as experimental corroboration of our theoretical proof. finally, we analyze the generality and broader impact of f 3 . in summary, the key contributions are as follows: ## unchanged fix issue when applying nmt to the problem of bug fixing, the buggy code is translated into the fixed code for the purpose of fixing. in this process, as shown in figure 1 , it often happens that the sequence predicted by the model, and the sequence of buggy code at the time of input, are exactly the same, a phenomenon that we call unchanged fix. the input code is buggy, and the predicted unchanged fix is exactly the same as the buggy code, which means that the unchanged fix must also be buggy, and therefore it must not be a successful fix. in other words, we do not need to actually test whether the predicted code can run, or know how the code that is actually fixed should look like. just by comparing the sequences predicted by the model, with the input sequences, we can filter out a batch of cases where the fix obviously fails, so unchanged fix contains unsupervised properties. this phenomenon is caused by the fact that the input and output before and after translation are highly similar, or the vocabularies are highly sim-ilar, which may cause the model to "accidentally" generate exactly the same results as the input. according to our tests on the seq2seq transformer, as shown in figure 2 , when the training epoch increases from 1 to 300, the proportion of unchanged fix in the test set increases sharply and then decreases. the proportion is low at the beginning because the initialized sequence is completely chaotic. as the epoch increases, it slowly learns the approximate distribution of the dataset, and thus the phenomenon of unchanged fix starts to appear. after that, the distribution learned by the model becomes more and more accurate, and the number of model prediction errors gradually decreases, thus the number of unchanged fixes decreases. therefore, unchanged fix can be considered as a kind of lapse phenomenon when the model is not in a perfect state, just like a new painting student who wants to draw a tiger but accidentally draws a cat. the observed lapse scenario suggests that generative models, like humans, may learn a general generative logic first and then continuously improve and refine the learned knowledge. unchanged fix implies that the model itself has a general learning of the distribution of the data, but does not have the particularly precise details. this type of model is more common in many complex tasks of nlp based on nmt, which means that the unchanged fix issue may have a generality that is not limited to the bug fixing task. ## preliminaries for the purpose of quantitative proof for the properties of f 3 , we denote all the buggy codes in the test set as t , the multiple learners in f 3 as @xmath0, m 2 , . . . , m |m| }, where |m| is the number of learners. the part of f 3 before the m i (including the m i ) is called f 3 i . in particular, given a buggy program @xmath1, the learner m i will generate a fixed program y. in this paper, we classify y into the following four different sets: the goal of our filter mechanism is to filter out those unchanged fixes from each learner's predicted/generated fixes and feed them into the next learners. we can obtain the following rules for m 1 : similarly, we define c(f 3 i ), u(f 3 i ), cw(f 3 i ), w(f 3 i ) for the f 3 results. ## experiment and analysis our paper is biased towards verifying the theoretical validity of the f 3 , and the experiments are just one of the supporting evidences. f 3 may improve performance in any area where unchanged fixes exist, such as automatic post-editing and text style transfer, which is intuitive and theoretically proven by us. here we just pick the bug fixing as a typical task for experimental validation, and these experiments should also hold for other f 3compliant domains. in the experiments, we focus on investigating the following research questions: ## conclusion we reveal and study the unchanged fix issue in learning-based bug fixing tasks. based on our findings, we propose an intuitive yet effective general framework called f 3 to concatenate different learners with the filter mechanism to filter out unchanged fixes. we demonstrate the considerable performance and generality of f 3 from both theoretical and experimental perspectives. in the future, we will the design better filter mechanism and apply f 3 to different learning tasks.