Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models (2023)

Abstract: Automatic program repair (APR) has gained increasing attention as an essential technique in software development to reduce manual debugging efforts and boost developers’ productivity. Recent advances in deep learning (DL) based models have demonstrated promising results by learning from large-scale bug-fix examples in a data-driven manner. However, in practical scenarios, software bugs have an imbalanced distribution, and the fixing knowledge learned by APR models often only capture the patterns of frequent error types, making it inapplicable to handle the rare error types. To address this limitation, we investigate a novel task of low-resource APR, and propose Meta-APR, a new meta-learning framework integrated with code pretrained language models to generate fixes for low-resource bugs with limited training samples. Our Meta-APR learns better error-specific knowledge from high-resource bugs through efficient first-order meta-learning optimization, which allows for a faster adaptation to the target low-resource bugs. Besides, while we adopt CodeT5, a pretrained code-aware encoder-decoder Transformer, as the backbone model for Meta-APR, it is a model-agnostic framework that can be integrated with any neural models. Extensive experimental results on three benchmarks in various programming languages verify the superiority of our method over existing DL-based APR approaches.

## introduction program repair is critical to improving the productivity and stability of software development. however, it is resource-consuming and costprohibitive @xcite @xcite . a reliable automatic program repair (apr) system is thus crucial to reduce manual debugging efforts and development time @xcite . figure 1 : illustration of our meta-apr framework with codet5 for low-resource error-specific automatic program repair. we first meta-train the codet5 with highresource bugs ( ), where the backbone model is updated via gradient descent with respect to θ. after that, meta-apr is finetuned on the target low-resource bugs (▲,▲) using few-shot examples (10, 50, 100). with the advances in deep learning (dl) models @xcite and accessibility to large code corpora @xcite , neural approaches to apr have achieved remarkable performance via exploiting existing code patches @xcite . these models are typically trained and evaluated on datasets that comprise a mix of various error types, which are diverse in nature: they vary in terms of the number of bug-fix pairs per error type, and are typically imbalanced. moreover, the performance gaps across different error types are tremendous @xcite , which can significantly impair the apr models' performance. these observations motivate us to consider the idea: rather than training the model jointly on all error types, could we train a model that is quickly adaptable to any low-resource error-specific apr task? inspired by the success of meta-learning on low-resource nlp tasks like machine transla-tion @xcite and dialogue generation @xcite , in this work, we drive pioneering efforts in formalizing low-resource apr, and propose an effective metalearning framework that utilizes a code pretrained model to enhance apr performance. low-resource apr formulation: unlike traditional apr approaches that jointly learn a model from a mix of error types, our formulation considers each rare error type as a low-resource target task. accordingly, we create datasets specifically to support the evaluation of low-resource errorspecific apr, based on three practical apr benchmarks in various programming languages: tfix in javascript @xcite , manysstubs4j in java @xcite , and tssb-3m in python @xcite . we observe diverse and imbalanced error type distributions in these benchmarks, e.g., tfix, manysstubs4j, and tssb-3m respectively have 31, 8, and 9 error types that are of low-resource foot_0 along with 21, 6, and 14 high-resource error types. ## approach fig. 1 illustrates the overview of our proposed meta-apr, a meta-learning framework that lever-algorithm 1: meta-training for apr require: a set of high-resource error types t @xmath0, t2, . . . , tn}, ∀@xmath1, fj)} |d i | @xmath2, a apr model f θ , inner loop learning rate α, outer loop learning rate β, meta update step size m initialize: initialize θ from the apr model f θ output: optimal meta-trained apr model f θ while not done do d @xmath3. forall @xmath4.1 evaluate the inner loop cross entropy loss linner(f θ , b support s ) update error-specific model parameters with gradient descent: update global model parameters with estimation: ## experimental results and analysis in this section, we compare meta-apr with other code pretrained models in different training settings on a set of our curated low-resource error-specific apr tasks from three benchmarks ( §5.1), followed by a detailed analysis on the effects of different error types and token length ( §5.2), and a pilot study to compare with a closed-sourced large language model such as chatgpt in fixing these challenging low-resource bugs ( §5.3). ## conclusion in this work, we present meta-apr, a simple yet effective framework that extends codet5 with meta-learning for low-resource apr. it is a modelagnostic framework that can be integrated with any learning-based models. to the best of our knowledge, we are the first to investigate apr in the lowresource setting and curate error-specific datasets in different low-resource degrees from three apr benchmarks in python, java, and javascript. comprehensive experiments have verified the superiority of meta-apr over other learning strategies with various code pretrained models. more analysis shows that meta-apr can better align the representations of high-resource and low-resource bugs, and fix bugs with various sequence lengths and error types. a pilot comparison with chatgpt further shows that our meta-apr is still more capable of fixing these challenging low-resource bugs.