Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024)

Abstract: Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.

## introduction code generation is a critical yet challenging task that has various downstream applications, such as text-to-code generation @xcite @xcite , code translation @xcite , and code autocomple- † corresponding authors. tion @xcite . recent progress in large language models (llms) @xcite @xcite @xcite significantly boosts the performance of code generation and demonstrates a promising potential to be generally applied in different requirements and tasks @xcite @xcite . however, generating correct programs is not a one-time effort. existing works suggest enhancing code generation through multiple sampling @xcite , selfconsistency @xcite @xcite , and candidates ranking @xcite @xcite . despite these advanced approaches, they still fall short on basic programming questions from the hu-maneval and mbpp datasets. this underscores the limitations of single-pass program generation. recognizing this, a series of works have been proposed to refine the programs generated in a single pass, based on feedback from either human annotator @xcite or ## large language model debugger problem formulation. we follow the problem formulation of program debugging in @xcite . in a code generation task, each sample can be represented as a triplet (q, t v , t h ), where q represents the task description, t v and t h are visible and hidden test cases. in the text-to-code generation task, q consists of a brief paragraph in natural language that outlines the intended goal of the task. in the code translation task, q represents a program written in a language other than the target language, and the objective is to translate this program into the target language. a code generator is first provided with q and t v to generate the seed program a 0 . in the debugging stage, a debugger further refines a 0 and finally outputs a new program a * . a * will be tested using the hidden tests t h to evaluate whether it is correct or not. t h is only used in the evaluation stage and not visible during the code generation or debugging stages. ## experiments we evaluate ldb on three code generation benchmarks: humaneval @xcite , transcoder (roziere et al., 2020), and mbpp @xcite . humaneval and mbpp are for text-to-code generation, where the task description is a brief passage outlines the intended functionality of the program to be generated. transcoder is for code translation which requires to translate a program from c++ into python. the task description of transcoder consists of a c++ program to be translated. we compute pass@1 accuracy with hidden test cases for assesment. we conduct experiments with the proprietary llm, gpt-3.5 (turbo-0613) @xcite , and the open-source llms, codellama (34b-instruct) @xcite and starcoder (∼15b) @xcite as backbones. ## related work augmented code language models recent language models based on deep neural networks @xcite @xcite demonstrate great potential for coding tasks. despite their impressive capabilities, these models face challenges such as syntax correctness @xcite , semantic alignment @xcite , code reliability @xcite , and conflict merging @xcite . to address these challenges, some focus on enhancing initial code generation by leveraging multiple candidates @xcite @xcite or refining solutions based on better test cases @xcite and selfconsistency @xcite , while others train verifiers on execution results to predict solution quality @xcite . in contrast, ldb enhances code generation without expanding sampling numbers complementing existing methods, while these existing code generation methods could provide better seeds for our debugger. ## conclusion we present ldb, a debugging framework that help llms refine generated programs with runtime execution information. we empirically show that ldb significantly improves code generation accuracy and achieves state-of-the-art performance in program debugging, by segmenting the programs into basic blocks and tracking the intermediate values. experiments also reveal its unique paradigm of program debugging by using runtime information.