INTERVENOR : Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair (2024)

Abstract: This paper introduces INTERVENOR (INTERactiVE chaiN Of Repair), a system designed to emulate the interactive code repair processes observed in humans, encompassing both code diagnosis and code repair. INTERVENOR prompts Large Language Models (LLMs) to play distinct roles during the code repair process, functioning as both a Code Learner and a Code Teacher. Specifically, the Code Learner is tasked with adhering to instructions to generate or repair code, while the Code Teacher is responsible for crafting a Chain-of-Repair (CoR) to serve as guidance for the Code Learner. During generating the CoR, the Code Teacher needs to check the generated codes from Code Learner and reassess how to address code bugs based on error feedback received from compilers. Experimental results demonstrate that INTERVENOR surpasses baseline models, exhibiting improvements of approximately 18% and 4.3% over GPT-3.5 in code generation and code translation tasks, respectively. Our further analyses show that CoR is effective to illuminate the reasons behind bugs and outline solution plans in natural language. With the feedback of code compilers, INTERVENOR can accurately identify syntax errors and assertion errors and provide precise instructions to repair codes. All data and codes are available at [https://github.com/NEUIR/INTERVENOR](https://github.com/NEUIR/INTERVENOR).

## introduction large language models (llms), such as chat-gpt @xcite , have shown remarkable performance on code related tasks @xcite @xcite . this has significantly enhanced the efficiency and productivity in coding and software development @xcite . current approaches for code-based models involve pretraining language models on i am a student to conduct better coding code corpora @xcite @xcite and employing chain-of-thought (cot) to prompt the coding proficiency of llms @xcite @xcite . however, compelling llms to directly generate entirely correct code proves to be exceptionally challenging, even for proficient programmers in real-world scenarios @xcite . recently, researchers focus on improving the code generation ability of llms through self-repair techniques @xcite . these methods leverage llms themselves to execute and repair codes, thereby enhancing the quality of generated code. moreover, multi-agent collaborative coding approaches @xcite have also proven their effectiveness in handling difficult code tasks by prompting llms to play different roles, such as developers and testers. however, lots of bugs are difficult to find due to the cognitive inertia @xcite -overlooking the buggy codes that may not conform to their pre-existing coding thinking of llms. these agent-based code refinement methods @xcite heavily rely on the self-evaluation capabilities of llms, potentially encountering the degeneration-of-thought (dot) problem @xcite . this paper proposes interactive chain of repair (intervenor) to alleviate the dot problem in code repair. the approach incorporates feedback from code compilers to enhance the code repair process. following dong et al. (2023), we develop two agents, namely, code learner and code teacher, to collaboratively repair codes interactively. as illustrated in figure 1 , the code learner is tasked with generating/repairing codes based on provided instructions. to craft specific code repair instructions to guide the code learner @xcite , the code teacher generates the chain-of-repair (cor) to illustrate the bug repair solutions for the code learner. instead of @xcite , code teacher incorporates the bug report from compilers to rethink the reasons of code errors and generate planning on how to repair the bugs. this interactive code repair process will continue until the code learner successfully fixes all code errors or reaches the predetermined maximum number of repair attempts. experimental results demonstrate the effectiveness of intervenor by outperforming previous baseline models. notably, intervenor also achieves about 18% and 4.3% improvements over @xcite in code generation and code translation tasks, showing its ability to improve the quality of generated codes through iterative code repair. besides, we also build the codeerror dataset for evaluating the code repair ability of intervenor by collecting the buggy code snippets from gpt-3.5 and real-world user-submitted codes. intervenor further validates its efficacy by demonstrating a twofold increase in the number of successfully repaired codes. our further analyses illustrate that inter-venor is effective in leveraging the bug messages from code compilers, recognizing the reasons for code errors, and providing correction planning in natural language. thanks to our cor mechanism, intervenor avoids thinking by llms themselves and can accurately diagnose the buggy codes and correct the assertion errors and name/syntax errors even in more difficult code generation scenar-ios. our cor mechanism enables llms to avoid designing complex code generation/repair prompts and achieve the best performance via only threeturn code repair. it also shows the potential to leverage the feedback from environments or rule systems to evolve llms @xcite . ## related work code generation tasks @xcite @xcite aim to generate correct and executable code based on the given natural language description, which has drawn lots of attention from researchers. llms such as chat-gpt (openai, 2022) and gpt-4 (openai, 2023) have shown strong effectiveness in generating code of high quality. to enhance the coding ability of llms, existing work focuses on code-specific pretraining and performs exceptionally well in code generation tasks @xcite @xcite . recently, some models focus on utilizing prompting techniques to enhance the coding capabilities of llms. codecot @xcite is inspired by @xcite and prompts the quality of generated codes using code-cot and self-exam methods. llms are asked to craft the code and design a set of test cases to polish the codes. structured chain-of-thought (scot) @xcite further considers the program structure, such as sequences, branches, and loops, and prompts llms to generate intermediate reasoning steps with program structures. nevertheless, forcing llms to directly generate the completely correct codes is challenging in the code generation task @xcite . to generate more accurate code, existing efforts primarily concentrate on self-refine @xcite and @xcite techniques. the self-refine models aspire to improve the quality of generated code by decoding multiple samples and subsequently selecting the most suitable one based on specific criteria. one strategy to formulate customized criteria involves executing the generated code and selecting the optimal one based on the resulting execution outcomes @xcite @xcite . another approach is to rerank multiple code solutions to determine the final code @xcite @xcite . however, these methods necessitate significant computing re-sources for generating code candidates, rendering them inefficient @xcite . another research avenue involves employing an iterative code repair approach to enhance the quality of generated code @xcite @xcite @xcite . self-debug (chen et al., 2023b) utilizes explanations generated by llms to rectify self-generated code, while @xcite incorporates human-provided feedback for improvement. self-edit @xcite employs error messages to refine generated code, but it necessitates the training of an additional fault-aware editor to generate a new program. instead of directly integrating feedback for code repair, intervenor designs an additional agent to reflect the reasons for coder errors and generate the chain-of-repair (cor). this cor is then employed to instruct the other agent in code repair through natural language. moreover, the work @xcite ) also designs a multi-agent collaborative approach to simulate the software development process and improve the efficiency of code generation. nevertheless, these methods are highly dependent on the self-evaluation ability of llms and may face the degeneration-of-thought (dot) problem @xcite . unlike them, intervenor focuses on the bugfixing process and proposes a simple but effective solution, which utilizes external tools, such as the python interpreter, to execute the code @xcite and use the accurate bug report to facilitate the agent collaboration during interactive code repair. ## methodology in this section, we introduce intervenor, which conducts an interactive program repair process using llm collaboration. we first describe the preliminary of code repair (sec. 3.1) and then introduce our interactive chain-of-repair (cor) mechanism (sec. 3.2). ## evaluation results in this section, we evaluate the overall performance of intervenor. then we conduct ablation studies and also show the effectiveness of interactive cor in different testing scenarios. finally, case studies are presented. ## conclusion this paper proposes intervenor, which regards the feedback of compilers as the inter-venor to facilitate the agent collaboration of code error diagnosis and repair. our experiments show that intervenor outperforms previous code generation/repair models by generating accurate code repair solutions and instructions.