D ebug B ench: Evaluating Debugging Capability of Large Language Models (2024)

Abstract: Large Language Models (LLMs) have demonstrated exceptional coding capability. However, as another critical component of programming proficiency, the debugging capability of LLMs remains relatively unexplored. Previous evaluations of LLMs’ debugging ability are significantly limited by the risk of data leakage, the scale of the dataset, and the variety of tested bugs. To overcome these deficiencies, we introduce ‘DebugBench’, an LLM debugging benchmark consisting of 4,253 instances. It covers four major bug categories and 18 minor types in C++, Java, and Python. To construct DebugBench, we collect code snippets from the LeetCode community, implant bugs into source data with GPT-4, and assure rigorous quality checks. We evaluate two commercial and four open-source models in a zero-shot scenario. We find that (1) while closed-source models exhibit inferior debugging performance compared to humans, open-source models relatively lower pass rate scores; (2) the complexity of debugging notably fluctuates depending on the bug category; (3) incorporating runtime feedback has a clear impact on debugging performance which is not always helpful. As an extension, we also compare LLM debugging and code generation, revealing a strong correlation between them for closed-source models. These findings will benefit the development of LLMs in debugging.

## introduction large language models (llms) have demonstrated exceptional code generation abilities. llm-based coding methods @xcite have achieved human-level performance on benchmarks like humaneval @xcite and mbpp @xcite . llms have also become the core engine of practical programming assistance applications like github @xcite . similar to code generation, debugging foot_0 is also a crucial component in programming, consuming 35-50% of the development duration and 50-75% of the total budget @xcite . however, unlike coding, the debugging abilities of llms remain relatively unexplored. one primary obstacle in code debugging research is the lack of evaluation benchmarks. while some basic evaluations @xcite @xcite verify the effectiveness of llm-based debugging methods, these evaluations have notable limitations that prevent us from comprehensively assessing the debugging capabilities of llms as exhibited in to overcome these deficiencies, we create de-bugbench, a dataset of 4,253 instances for llm debugging evaluation. we first collect code solution snippets from leetcode (2023), a popular programming challenge platform. to reduce the risk of data leakage, we ensure all of the instances in debugbench are released after july 2022, which is beyond the pre-training data cutoff date of tested models. for fine-grained evaluation of various bug types, we develop a bug taxonomy based on @xcite 's classification criteria. the classification encompasses four major bug categories: syntax, reference, logic, and multiples, along with 18 minor types as illustrated in figure 1 . subsequently, we prompt @xcite to implant bugs into the code solutions in pursuit of sufficient data scales for each bug type. we cover snippet-level code in c++, java, and python. to ensure integrity, we conduct automatic filtering and manual inspection. as shown in figure 1 , we evaluate two closedsource language models, gpt-4-0613 (openai, 2022) and gpt-3.5-turbo-0613 (openai, 2023), along with four open-source models: codellama-7b-instruct (rozière et al., 2023), llama-3-8b-instruct (meta, 2024), ## benchmark construction as illustrated in figure 2 , to construct debugbench, we first collect questions, code snippets, and examples from leetcode (2023) community, then employ gpt-4 (openai, 2023) for bug implantation. to ensure the integrity of the benchmark, we conduct automatic filtering and final human inspection. ## conclusion in this work, we presented debugbench, a benchmark specifically designed to evaluate the debugging capabilities of large language models. debug-bench was developed utilizing source data from leetcode (leetcode, 2023) and bug implantation with prompted gpt-4, underpinned by a stringent quality control process. our experiments with debugbench revealed several key findings: (1) in a zero-shot scenario, closed-source models exhibited relatively lower debugging performance compared to humans, while open-source models attained a even lower pass rate score in zero-shot scenarios; (2) multiple and logical errors posed a greater challenge for the repair compared to syntax and reference errors; (3) runtime feedback enhanced the debugging performance for syntax and reference errors but is unhelpful for logical errors; (4) for closed-source llms, debugging syntax or reference errors is easier than code generation while logic or multiple errors can equally hard or even harder. and their capabilities in coding and debugging are correlated. hopefully, these findings will contribute to the advancement of large language models in the field of automatic debugging. the data is open-sourced via apache-2.0 license.