Detect-Localize-Repair: A Unified Framework for Learning to Debug with C ode T 5 (2022)

Abstract: Automated software debugging is a crucial task for improving the productivity of software developers. Many neural-based techniques have been proven effective for debugging-related tasks such as bug localization and program repair (or bug fixing). However, these techniques often focus only on either one of them or approach them in a stage-wise manner, ignoring the mutual benefits between them. In this work, we propose a novel unified Detect-Localize-Repair framework based on a pretrained programming language model CodeT5 to seamlessly address these tasks, named CodeT5-DLR. Specifically, we propose three objectives to adapt the generic CodeT5 for debugging: a bug detection objective to determine whether a given code snippet is buggy or not, a bug localization objective to identify the buggy lines, and a program repair objective to translate the buggy code to its fixed version. We evaluate it on each of these tasks and their combined setting on two newly collected line-level debugging datasets in Java and Python. Extensive results show that our model significantly outperforms existing baselines from both NLP and software engineering domains.

## introduction program debugging is crucial, yet most costdominating in software development. the goal of program debugging is to localize erroneous lines of a program (bug localization) and fixes this buggy patch (program repair). the majority of debugging tools falls into two categories: program analysisbased and neural-based. to debug a program, program analysis-based techniques employ compilerbased and software engineering theory to build code analysis tools. these methods have a significant disadvantage in that they are not scalable to large and complicated programs. on the other hand, a recent trend is to use neural-based techniques @xcite @xcite @xcite based on the naturalness hypothesis of software code @xcite . they adopt a generic data-driven approach to train neural networks to automatically acquire bug-fix patterns through learning from a massive corpora of previous bug-fixes. however, these techniques suffer from a few major drawbacks. first, they often utilize codespecific or language-specific features such as control flow, data flow, and abstract syntax trees (asts), which requires a significant amount of engineering effort for a careful design of code representations and thus hinders their applicability to more diverse domains or programming languages. second, recent studies have focused on detecting bugs at coarse-grained code granularity such as function level or file level, which has been shown to be impractical in real-world use @xcite . it is also not ideal to localize bugs at too fine-grained level like the token level, which might lead to a large number of false positives @xcite . line-level or statement-level bug localization, on the other hand, has been extensively studied in the domain of program analysis, such as spectrum-based bug localization @xcite @xcite , in which the buggy statements are localized based on the signals of failed test cases, or mutation-based bug localization @xcite @xcite , in which the buggy statements are localized by randomly mutating the statements and measuring against a test suite. however, these traditional techniques necessitate the execution of test cases in order to complete the bug localization process, which has scalability issues. as a result, we propose that line-level bug detection is more reasonable if we can use large-scale bug fixes datasets; and it corresponds to how human developers read and debug programs. finally, these techniques are only intended for either bug localization or program repair, or treat them separately as two stages, which fails to exploit their potential mutual benefits. intuitively, a preciser bug detector is able to inform the repairer with more accurate buggy information to aid the bug fixing, while a good repairer usually has strong code understanding that is also required in bug detection. to address these issues, we propose a unified framework for adapting a general pretrained programming language model for line-level debugging and repair. our framework is based on a key observation about how programmers debug their code. first, he or she must determine whether or not a function is buggy. if it is buggy, the developer must localize the problematic line and provide a patch (repair). inspired from this procedure, we propose three fine-tuning objectives on top of a pretrained language model for debugging. using pretrained language models for code has two advantages. first, by treating code as natural language, it reduces the effort of the code representation engineering process. second, it can leverage the pretrained knowledge gained from large number of source code. we employ codet5 @xcite as the foundation model which has achieved state-of-the-art results on a wide range of code intelligence tasks. codet5 is pretrained on large-scale code corpus collected from github using code-aware objectives, which endows the model with strong code understanding capability. the first objective is the function-level bug detection task, which entails determining whether or not a particular piece of code includes a bug (d). second, in order to give developers with more valuable information at a finer-grained code granularity, we propose a bug localization aim to identify the exact lines of code that include bugs (l). the third ojective is the program repair, which is used to convert the buggy code to the correct code (if applicable) (r). we expect that these tasks will complement one another and culminate in a robust, all-encompassing software debugging tool capable of doing many debugging-related tasks. codet5-dlr is the model created by applying all of the fine-tuning objectives. to evaluate on the whole debugging procedure (d-l-r), we newly collected two large-scale bugfix datasets in java and python programming languages from github commits, which is released to facilitate future research as part of our contribution. we consider two types of bugs: single-line and multi-line. we evaluate our codet5-dlr on three separate debugging-related tasks: function-level bug detection, line-level bug localization, and program repair. our evaluation results show that our model significantly outperforms existing baselines on all of the tasks. we further conduct ablation studies to demonstrate that jointly training with the three objectives yields better performance than separately training on each single task. finally, we design a unified evaluation task to combine all of these tasks that is to mimic how developers localize and fix bugs in real-world scenario, where our model is able to correctly localize 33.93% buggy lines and repair 46.93% for a single line bug fix java dataset. our major contributions are three-fold: ## method as shown in figure 1 , we present a unified framework to jointly address three crucial tasks in program debugging: bug detection (whether a given code snippet contains bugs), bug localization (which lines are buggy), and program repair (how to repair bugs). we first define the input/output formulation of these tasks in § 2.1 and then revisit the foundation model codet5 in § 2.2, followed by introducing each of tasks in § 2.3 in detail. ## datasets we collect two new datasets, one is the single line bug-fixes pair and the other is the multi-line bugfixes pair. the single line bug-fixes (karampatsis and sutton, 2020) have been considered recently as one of the major issues that affects the quality of source code. these bugs can be fixed easily with simple code changes such as changing operators, renaming identifiers, swaping variables and so on. however, these bugs occurred frequently, and current static-based techniques are incapable of detecting them accurately (less than 10% in accuracy). existing datasets (karampatsis and sutton, 2020; richter and wehrheim, 2022), however, are not suitable for our purpose due to three reasons: (1) they contain only the code changes at the file level while our goal is to detect buggy lines at both function level and line level; (2) they does not contain the before and after function-level information of the code changes but only the patches at line-level; and (3) these datasets are mostly for single-line bug fixes, while our goal is to extend for a more realistic setting of multi-line bugs. with such reasons, we decide to collect datasets by ourselves for a comprehensive evaluation. we follow similar steps from (karampatsis and sutton, 2020) to collect two datasets in java and python. concretely, we extract bug-fixes code changes from github commits, we use pydriller @xcite , a tool that mines software repositories from git. to decide if a commit fixes a bug, we follow karampatsis and @xcite to check if its commit message contains at least one of the keywords: error, bug, fix, issue, mistake, incorrect, fault, defect, flaw, and type. this heuristic has been shown to achieve 96% accuracy on a set of 300 manually verified commits @xcite and 97.6% on a set of 384 manually verified commits @xcite . the code changes are made up of three parts of a source file: before changes, after changes, and the difference between the two (patch). however, because we want to localise bugs at the function and line level rather than the file level, we need to perform additional preprocessing to extract the code changes at the function level. we use lizard foot_0 to extract the functions and compare the different between the functions from the before and after version of a source file (obtained from pydriller). we end up with two datasets of different types and languages, one is for single-line bug-fixes in java (sl-java) and the other is the multi-lines bugfixes in python (ml-python). for sl-java, beside the code changes for bug fixes, we also follow karampatsis and sutton (2020) to use treesitter foot_1 to identify 13 bug patterns for the single buggy lines. we do not aim to detect the exact patterns because they can be easily detected using matching rules on asts if we can localize if a line is buggy. however, the patterns is useful in analysing how well our techniques work on each pattern, allowing us to gain a deeper understanding of the debugging process. figure 2 shows a sample in our sl-java dataset. a sample comprises primarily of the buggy code snippet (before) and the fixed code snippet (after) (after). it also includes the line number of the defective line. multiple buggy lines are present in the ml-python samples as opposed to a single buggy line. each instance also include other meta data, such as commit message, commit id (sha1) and project name so that it is easy to trace back the original bug information. ## experiments we employ the codet5-base (220m) foot_2 as the foundation model for our unified codet5-dlr framework. for the purpose of ablation study, we also consider three variants of our model: codet5-d trained with l detect , codet5-l trained with l localize , and codet5-r trained with l repair . we set the maximum source and target sequence lengths to 512. all experiments are performed on nvidia a100 gpus with 40 gb memory. ## related work pretrained language models for code recently, language models in natural language processing has been applied to model source code @xcite @xcite @xcite @xcite . codebert @xcite pretrain a model of code on multiple programming languages by adapting a roberta model @xcite . cubert @xcite ) pretrains a bert model for code using a large dataset of curated python files. in general, most of the techniques treat code similar to texts and adapt the same pretraining strategies as for natural language. some techniques, such as codet5 @xcite , encode source code features such as identifier information, data flow, and function name, among others, to pretrain code models, which may result in better overall performance. ## conclusion we proposed a novel detect-localize-repair framework for jointly detecting bugs, localizing bugs and suggesting program repairs. our model is built on the codet5 foundation model and is fine-tuned jointly to achieve three debugging-related objectives: function-level bug detection, line-level bug localization, and program repair. these three objectives are based on how software developers locate bugs and repair programs in their daily work. our evaluation results show that training these 3 objectives together yields better results than fine-tuning on each objective individually. furthermore, we also contribute to provide two new datasets that can be used for evaluating both bug localization and program repair tasks. our datasets differ from existing datasets in that we provide the exact line that is buggy, as well as the before and after versions of a code snippet. we will make our datasets publicly available to facilitate research on this topic.