R e P air: Automated Program Repair with Process-based Feedback (2024)

Abstract: The gap between the trepidation of program reliability and the expense of repairs underscore the indispensability for Automated Program Repair (APR). APR is instrumental in transforming vulnerable programs into more robust ones, bolstering program reliability while simultaneously diminishing the financial burden of manual repairs. Commercial-scale language models (LM) have taken APR to unprecedented levels. However, due to the limitations of model capabilities by parameters, a one-step substantial modification may not achieve the desired effect for models with parameters less than 100B. Moreover, humans interact with the LLM through explicit prompts, which hinders the LLM from receiving feedback from compiler and test cases to automatically optimize its repair policies. Explicit prompts from humans not only increase additional manpower costs, but also pose potential misunderstandings between human’s intent and LMs.Based on the above considerations, we are exploring how to ensure small-scale LM still outperform through process supervision and feedback. We start by constructing a dataset named CodeNet4Repair, replete with multiple repair records, which supervises the fine-tuning of a foundational mode. Building upon the encouraging outcomes of reinforcement learning, we develop a reward model that serves as a critic, providing feedback for the fine-tuned LM’s action, progressively optimizing its policy. During inference, we require the LM to generate solutions iteratively until the repair effect no longer improves or hits the maximum step limit. The experimental results show that this process-based feedback not only outperforms larger outcome-based generation methods, but also nearly matches the performance of closed-source commercial large-scale LMs.

## introduction since the birth of the program, its reliability has been a primary concern. the capacity of large language models to auto-generate code has further problem description compiler memory limit exceed accepted hidden test case hidden test case • [1, 1, 1, 2] • [2, 3, 3] • [1]*10000 memory limit: o(1) time limit: o(n) time limit exceed submission 1 def majorityelement(nums): nums.sort() return nums[len(nums)/2] submission 1 def majorityelement(nums): nums.sort() return nums[len(nums)/2] submission 3 def majorityelement(nums): @xmath0: if @xmath1: @xmath2: @xmath3: if @xmath4: @xmath5: @xmath6: vote[num]=vote.get(num,0)+1 return max(vote,@xmath7.get) submission 2 def majorityelement(nums): @xmath8: vote[num]=vote.get(num,0)+1 return max(vote,@xmath9.get) figure 1 : the general procedure for competitors to refine a solution on programming contest platform. initially, they draft a solution based on the problem description and additional constraints such as time and memory limits. they then progressively improve their solution using feedback from the platform, like exceeding time or memory limits, until they achieve an accepted result. prior works viewed apr as a sequence-tosequence task, then llms, with their massive model parameters and training data, can address this issue through zero-or few-shot learning @xcite @xcite @xcite @xcite @xcite @xcite @xcite . whether it's traditional methods or llms, they achieve the repair by making a single modification. these outcome-based supervision methods pose a stringent challenge to the model due to the substantial edit distance between input and output. simultaneously, the one-step modification approach does not align with human behavioral patterns: to be specific, when facing complex tasks, programmers usually repair the buggy program step by step in a cycle of modification, testing, and feedback. this inspires us to pay attention to the exploration of modification processes in apr tasks. unlike outcome-based supervision, this process-based method requires guidance and supervision at each modification step. figure 1 depicts the typical program repair process in programming competitions, which is guided by compilers and test cases: competitors initially construct a sketch of the program based on the problem's intent and other constraints. they then complete the repair through a continuous process of submission, feedback, and interaction. specifically, they initially attempt to solve "find the majority element" problem by sorting and selecting the middle element. however, when they encounter time constraints, they shift to calculating each element's frequency. but this method still requires o(n) space complexity. after receiving "memory limit exceed" status, they ultimately use the boyer-moore majority vote algorithm with o(1) space complexity. this example illustrates the characteristics of process-based program repair, which necessitates ongoing interaction with the compiler and test cases for feedback-guided repair. however, applying this type of process-based feedback on lms appears unfeasible. first and foremost, the primary obstacle hindering related research is the absence of processbased datasets in practical scenarios. second, the supervision method for intermediate processes in apr tasks are still under exploration. previous work explored process-based supervision in mathematical problem solving and reasoning tasks @xcite , which required the intermediate steps to be correct. however, the intermediate steps in apr tasks serve as incorrect supervision signals and cannot be directly used for training. last, interaction with lms can be achieved through explicit prompt engineering. attempts to use prompt engineering instead of compiler and test cases for feedback hinder the precise construction of human intentions by lms. deepfix review4repair bug2fix codenet4repair language c java java python test cases × × × ✓ repair form single step single step single step multi step problem description × × × ✓ test size 6,971 2,961 58,356,545 10,144 ## data collection as far as we know, there has not been a dataset established for program repair tasks that includes processes. thus we attempt to develop a procedural dataset specifically for these tasks. overall, the dataset includes problem description, memory and time limit of the problem, repair process (a series of programs from error to correctness) and resource usage during execution (memory usage, cpu time, and code size). we derive our data from the large-scale programming competition dataset codenet @xcite . we make every effort to cleanse and filter this data to guarantee its quality. we clarify problem description, clean up the program from preliminary to fine, and collect additional high-quality test cases. finally, we organize the program into a procedural format. in compiler compiler environment test cases case 1 case 2 …… case 1 case 2 …… test cases case 1 case 2 …… compiler environment test cases case 1 case 2 …… @xmath10 ## method our methodology follows that of ouyang et al., which aligned gpt-3 with human. we start from starcoderbase @xcite , a 15.5b code foundation pre-trained lm with 8k context length. it is trained on 1 trillion tokens from the stack @xcite . figure 2 illustrates the techniques used to train repair. ## conclusion in this work, we proposed an automated program repair method that utilizes process-based feedback. this approach is inspired by strategies used in programming competitions and incorporates process supervision into the repair process. due to the unavailability of suitable datasets, we first created a multi-step repair dataset called codenet4repair. this dataset not only includes the initial faulty programs and the final accepted ones but also records the intermediate repair process. on this basis, we implemented supervision over the process with feedback. specifically, we fine-tuned a pre-trained model starcoderbase through to make it understand program repair tasks. then, we empirically defined a partial order relationship for program states and trained reward models using pairwise ranking. this reward model would serve as a critic, assessing each attempt by the lm to polish the program. meanwhile, the lm plays the role of an actor, constantly adjusting its policies for repairing the program based on received rewards. during the inference stage, lm iteratively repaired the program until the exit conditions are met. the experimental results indicated that the process-based feedback method can demonstrate excellent performance. with smaller model sizes, repair still can rival or even surpass closed-source commercial large models. we believe that process-based feedback is currently underexplored, and will continue to explore more universal methods.