Self-Edit: Fault-Aware Code Editor for Code Generation (2023)

Abstract: Large language models (LLMs) have demonstrated an impressive ability to generate codes on competitive programming tasks. However, with limited sample numbers, LLMs still suffer from poor accuracy. Inspired by the process of human programming, we propose a generate-and-edit approach named Self-Edit that utilizes execution results of the generated code from LLMs to improve the code quality on the competitive programming task. We execute the generated code on the example test case provided in the question and wrap execution results into a supplementary comment. Utilizing this comment as guidance, our fault-aware code editor is employed to correct errors in the generated code. We perform extensive evaluations across two competitive programming datasets with nine different LLMs. Compared to directly generating from LLMs, our approach can improve the average of pass@1 by 89% on APPS-dev, 31% on APPS-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B. Compared to other post-processing methods, our method demonstrates superior accuracy and efficiency.

## introduction large language models (llms) have recently been applied to the competitive programming task. this task requires understanding a complex natural language description of a problem with example test cases and correctly implementing solutions that can span hundreds of lines. solutions are evaluated by executing them on hidden test cases. however, existing llms often have low accuracy and pass rates in this task. for example, on a popular competitive programming benchmark apps-test @xcite , the nearly most powerful model gpt3 @xcite achieves only 7% accuracy when allowed to submit only one program per task (referred to as pass@1). gpt3 output: ## methodology we provide an overview of the self-edit pipeline in figure 2 . given the problem description, we first generate the initial code with llm. then we execute the example test case to obtain test results and construct the supplementary comment. finally, we train a fault-aware code editor model to refine the code based on the problem description, generated code, and supplementary comment. ## experiment we present extensive experiments that span two representative datasets and nine different llms for code generation, whose parameter counts range across four orders of magnitude. the details of the adopted llms are described in section 3.1. we aim to investigate four research questions: (1) how much can fault-aware code editors improve various code generation models on competitive programming (sec. 4.2), (2) the advantages of editor-based methods over existing ranking methods (sec. 4.3), ( 3 ) to what extent does the supplementary comments help to refine the program (sec. 4.4), (4) how does the number of editing rounds affect the final result (sec. 4.5). ## conclusion we propose a generate-and-edit approach named self-edit that utilizes execution results of the generated code from llms to improve the code quality on the competitive programming task. the central component of our approach is the fault-aware code editor, which can edit and optimize the generated code. in-depth evaluations demonstrate our approach significantly improves the quality of llms' output code.