MLD ebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios (2025)

Abstract: Code debugging is a crucial task in software engineering, which attracts increasing attention. While remarkable success has been made in the era of large language models (LLMs), current research still focuses on the simple no-library or single-library setting, ignoring the complex multi-library scenario in real-world applications. To address this limitation, we make the first attempt to introduce MLDebugging (Multi-Library Debugging), a comprehensive benchmark designed to assess debugging challenges within multi-library Python code. Specifically, MLDebugging encompasses 126 distinct Python libraries, covering a wide range of multi-library code issues, categorized into seven distinct types. Furthermore, we conduct a thorough evaluation of MLDebugging using both mainstream open-source and closed-source LLMs and highlight that current LLMs still struggle to correctly perform code debugging across multi-library scenarios. We hope this work can uncover the potential of LLMs in multi-library debugging scenario and offer insights for future research.

## introduction code debugging emerges a significant urge for code review, requiring bug location first and then fixing the bug for correct functionality, which has garnered increasing attention for software engineering @xcite . in light of the need to enhance the efficiency of code debugging and repair, a series of work consider adapting automatic code debugging (acd) techniques to serve as a fast and promising solution to the persistent issue of software defects @xcite @xcite . with the advancement of large language models (llms) @xcite ), * equal contribution † corresponding author debugger type mismatch b64encode only accepts string type arguments a considerable body of research has been dedicated to effective code debugging. specifically, @xcite was the first to reframe the code debugging task as a text-to-text problem. @xcite introduced the first debugging dataset specifically designed for llms, which leveraged code snippets from the leetcode (2025) platform. @xcite proposed multiple debugging sub-tasks, thereby expanding the code debugging task to encompass multi-language and multi-task scenarios. furthermore, @xcite further extended this benchmark to include multilingual debugging contexts, covering 18 programming languages. this extension facilitates a more thorough assessment of llm debugging capabilities across various programming languages. despite its success, as shown in figure 1 (a) , the current research are still limited to the simple no-library or single-library setting, which fails to satisfy the requirements of some complex multilibrary scenario in real-world applications. actually, in real-world software development, the use of multiple libraries is a common practice, as evidenced by research @xcite , which emphasizes the importance of multi-library scenarios for code debugging. unlike the previous no-library or single-library scenarios, as illustrated in figure 1 (b), multi-library debugging natrually introduce two distinct challenges: (1) understanding multiple libraries for bug location and (2) utilizing multiple libraries for bug fixing, which cannot be addressed by previous approaches. driven by this motivation, we propose mldebugging (multi-library debugging) in this study, a benchmark designed to evaluate debugging across 126 libraries, comprising 1,175 samples. as shown in to assess the limitations of current llms, we carry out a comprehensive evaluation of both openand closed-source llms using mldebugging. our experiments reveal the following insights: (1) current llms excel at debugging method class errors but struggle with conceptual mistakes and missing imports. (2) the structured nature of the mldebugging, widespread use of libraries, and access to complete runtime information, such as test cases and feedback, enhance llm performance. (3) in mldebugging, models like @xcite , which are built on distillation techniques, fall short of enhancing task performance. our contributions are summarized as follows: to promote further investigation, the complete dataset can be accessed via https://github. com/hjytsuki/mldebugging . in the context of multi-library code debugging, the task involves generating the correct code ĉr,l from the erroneous code c r,l . this process can be formally expressed as: type count type mismatch (tm) 97 data transfer issues (dti) 127 function parameter errors (fpe) 88 parameter configuration errors (pce) 60 function misuse (fm) 101 requirement misunderstanding (rm) 143 import errors (ie) 23 the resulting corrected code ĉr,l will pass all test cases, satisfying the following condition: ## dataset analysis in this study, we design distribution-based experiments to evaluate the alignment between error distributions in our dataset and real-world debugging scenarios. first, we extract question-answer pairs focused on issues related to multiple libraries from stack @xcite . we then apply textual embeddings to the error descriptions in both the mldebugging and debugbench datasets. to quantify the distributional similarity, we use two key measures: (1) centroid-based comparison we calculate the centroids of bug description embedding vectors for mldebugging, debug-bench, and stackoverflow separately, then evaluate the cosine similarity and euclidean distance between them. (2) real-world proximity for each sample, we measure its distance to the nearest real-world sample from stackoverflow. consider the text embedding vector a from the benchmark (mldebugging, debugbench) and the real bug description embedding vector b from stackoverflow. we retain the b j points closest to a i and compute the sum of their euclidean distances to obtain the the results in ## related work the rise of large language models (llms) has also had a considerable impact on automated code debugging (acd) tasks, as previous datasets have been vulnerable to data leakage risks @xcite . to facilitate a smoother transition from traditional datasets to those suited for llm evaluations, numerous remarkable contributions have surfaced. debugbench @xcite stands as the first dataset designed specifically to assess the debugging capabilities of large models. this work collects data from leetcode, subsequently introducing bugs via model injection. in the realm of apr, xcodeeval @xcite has proposed three distinct sub-tasks, spanning multiple programming languages, to comprehensively evaluate a model's ability to repair code. meanwhile, mdeval @xcite ) adopts a multi-language approach, encompassing a benchmark across 18 programming languages. in addition, the challenges presented by specific realworld scenarios have prompted research into niche areas, with several efforts concentrating on executable code, data processing, and other specialized contexts @xcite @xcite . these advancements have undeniably propelled the field forward. however, the majority of these datasets are sourced from algorithmic competition platforms, often overlooking the need for python multi-library scenarios. therefore, we have constructed a benchmark specifically aimed at evaluating the debugging abilities of models in the context of multiple libraries, providing an in-depth assessment of their proficiency in both static knowledge comprehension and multi-library code interaction. ## conclusion this work introduces mldebugging, a benchmark designed to assess debugging challenges in multilibrary code. we conduct a comprehensive analysis on mldebugging and the experimental results reveal that the current llms still struggle in multilibrary scenario. this work emphasizes the need for further research to improve llm performance in multi-library settings and provides insights to guide future developments in this field.