Improving Transformer-based Program Repair Model through False Behavior Diagnosis (2023)

Abstract: Research on automated program repairs using transformer-based models has recently gained considerable attention. The comprehension of the erroneous behavior of a model enables the identification of its inherent capacity and provides insights for improvement. However, the current landscape of research on program repair models lacks an investigation of their false behavior. Thus, we propose a methodology for diagnosing and treating the false behaviors of transformer-based program repair models. Specifically, we propose 1) a behavior vector that quantifies the behavior of the model when it generates an output, 2) a behavior discriminator (BeDisc) that identifies false behaviors, and 3) two methods for false behavior treatment. Through a large-scale experiment on 55,562 instances employing four datasets and three models, the BeDisc exhibited a balanced accuracy of 86.6% for false behavior classification. The first treatment, namely, early abortion, successfully eliminated 60.4% of false behavior while preserving 97.4% repair accuracy. Furthermore, the second treatment, namely, masked bypassing, resulted in an average improvement of 40.5% in the top-1 repair accuracy. These experimental results demonstrated the importance of investigating false behaviors in program repair models.

## introduction automated program repair (apr), a technique that has attracted substantial attention from both academia and industry, can mitigate the costs associated with bug fixing during software development and maintenance by automatically generating fixing patches. transformers have received enormous attention and have been applied to apr, showing effectiveness in fixing defects, bugs, errors, and vulnerabilities 1 @xcite @xcite @xcite . because existing apr studies have primarily focused on developing and improving the apr model, the internal decision-making process behind patch generation is often overlooked. investigating the inner workings of a model can offer valuable insights by identifying its capabilities and limitations @xcite @xcite @xcite @xcite . specifically, false behaviors of the model can provide a direction for improvement. for example, @xcite bug localization accuracy by eliminating the input tokens that the model considers when it fails to localize @xcite . similarly, an examination of the internal behavior of the model during the generation of incorrect patches can provide direction for improvement. current studies on incorrect patches primarily focus on distinguishing incorrect patches by relying primarily on the embedding vectors of the input buggy code @xcite . however, because of the repeated reformulation of these vectors within the inner layers of the transformer-based model, the embedding vector lacks sufficient information to discern the process that causes incorrect patch generation. moreover, once an incorrect patch is identified, determining directions to improve becomes challenging. therefore, research must be conducted with a focus on identifying and addressing false inner workings (that is, false behaviors), in which the model generates incorrect patches. applying the methods and findings of previous studies that analyzed the model behavior may appear like an intuitive solution. however, directly utilizing them becomes challenging because of the disparity between our target task, program repair, and the tasks investigated in previous studies. code fixing is a generation task that poses difficulties when the results or methodologies typically used in classification tasks are applied @xcite . among the various generation tasks @xcite , code fixing focuses specifically on eliminating bugs from the source code. to ensure consistency in the input/output language and preserve the correct structure and semantics of the code, the majority of the inputs must remain unchanged. furthermore, as most existing studies focus on comprehending the successful or overall behaviors of a model @xcite @xcite , a deficiency exists in understanding false behaviors and identifying their characteristics for automatic detection. therefore, it is essential to conduct research to comprehend false behaviors, along with their identification and subsequent treatment for the apr domain. this study proposes a methodology for diagnosing and treating false behaviors in transformerbased program repair models. we propose a behavior vector to represent the internal behavior of the model. the proposed behavior vector is generated by extracting the attention weights and value vectors from each attention head inside the transformer model. this approach enables us to capture the model behavior in terms of the way it considers or neglects the input tokens during the patch generation steps. additionally, we introduce a behavior discriminator (bedisc) that distinguishes the patterns between successful and false behaviors exhibited by the model. based on this, we introduce two treatments 1) early abortion and 2) masked bypassing. our contributions are as follows: ## conclusion this study is the first to examine the inner workings of program repair model failures and to introduce a behavior vector for diagnosing false behaviors in a transformer-based program repair model. we proposed methods for diagnosing and addressing the false behavior of the program repair model and validated their effectiveness across three models and four datasets. the proposed bedisc and treatment strategies improved the apr performance by 18.6% on average and up to 130.4%. the experimental results demonstrated the necessity for additional investigations into the false behavior of program repair models. in future studies, we intend to expand the proposed method to a wider range of models and bugs. in addition, we intend to investigate false behaviors across broader target tokens to develop an advanced treatment approach.