R epo D ebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models (2025)

Abstract: Large Language Models (LLMs) have exhibited significant proficiency in code debugging, especially in automatic program repair, which may substantially reduce the time consumption of developers and enhance their efficiency. Significant advancements in debugging datasets have been made to promote the development of code debugging. However, these datasets primarily focus on assessing the LLM’s function-level code repair capabilities, neglecting the more complex and realistic repository-level scenarios, which leads to an incomplete understanding of the LLM’s challenges in repository-level debugging. While several repository-level datasets have been proposed, they often suffer from limitations such as limited diversity of tasks, languages, and error types. To mitigate this challenge, this paper introduces RepoDebug, a multi-task and multi-language repository-level code debugging dataset with 22 subtypes of errors that supports 8 commonly used programming languages and 3 debugging tasks. Furthermore, we conduct evaluation experiments on 10 LLMs, where Claude 3.5 Sonnect, the best-performing model, still cannot perform well in repository-level debugging.

## introduction large language model (llm) based code debugging refers to automatically detecting @xcite , locating @xcite , and repairing @xcite @xcite @xcite errors to improve functionality and reliability. it has shown great potential in improving software development efficiency and reducing the time and effort required for software engineering (se) @xcite . to evaluate the code debugging performance of large language models (llms), various benchmarks have been developed that mainly focus on evaluating the automatic program repair capacity of llms @xcite @xcite . notably, debugeval @xcite introduces four debugging-related tasks (bug location, bug identification, code review, and code repair). meanwhile, mdeval @xcite is a multilingual benchmark for three similar tasks. swe-bench @xcite and @xcite are repository-level benchmarks that focus on evaluating the end-to-end ability of llms to fix issues based on github issue reports. however, these studies either center on functionlevel code debugging, overlooking the substantial challenges in evaluating the repository-level code debugging of llms, or lack evaluation diversity on different tasks, languages, and error types. to mitigate the gap, we propose a multi-task and multi-language repository-level debugging dataset, repodebug, which spans 8 programming languages (c, c#, go, java, javascript, python, ruby, and rust) and 3 tasks (bug identification, bug localization, and automatic program repair). following @xcite , repodebug is meticulously constructed with 22 distinct subtypes of bugs systematically classified into 4 types: syntax errors, logic errors, reference errors, and multiple errors. each instance within the repodebug comprises a buggy code file, the subtype of the bug, and the precise location of the bug. specifically, this paper collects data in the repodebug from 63 github repositories, all created after 2022, to mitigate data leakage. of these, 17 repositories are in the test set, and 46 repositories are in the training set. subsequently, this paper introduces 22 subtypes of bugs to the code files in these repositories by constructing abstract syntax trees using the tree-sitter foot_0 and recording their exact locations. to ensure the validity of the bugs, we conduct both automated filtering and manual inspection in the process of collection and construction. to evaluate the ability of llms to debug code errors, this paper utilizes four metrics following @xcite ; @xcite , in which this paper distinguish between the success rate of identifying a single error location and multiple error locations. @xcite , the evaluation experiments involve three closed-source models and six open-source models of varying sizes. finally, the experiment results reveal: (1) existing large language models exhibit limitations in performance on the repodebug dataset. (2) llms perform differently in different languages, and the error in java is easier to detect and repair. (3) errors of different types vary in difficulty, with multiple errors being the most challenging and syntactic errors being the simplest. the main contributions are as follows: ## related work this section provides a comprehensive review of the existing benchmarks relevant to our research. section 2.1 first explores the repository-level benchmarks. subsequently, section 2.2 delves into the domain of automatic program repair, introducing datasets and benchmarks that assess the ability of llms to identify and fix errors in code. ## dataset construction in this section, the construction process of repodebug is described. as shown in figure 2 , the process begins with collecting project information from github and selecting repositories and c c# go java javascript python ruby rust anbjorge/pgqueuer pgqueuer reciples test db.py example.py … readme.md … stars date license pr suffix length tree-sitter example.py example.py manual check example.py error locations error type correct code wrong function name 12 the number of buggy line buggy code 1 of 22 designed error types step1: data collection step2: ast-based bug design step3: manual check files that meet the specified requirements. the code files are then edited using an ast parser to introduce bugs. lastly, manual checks are performed to ensure the validity and quality of the data. ## analysis to comprehensively investigate the factors influencing the repository-level code debugging capa-bilities of large language models, we employ three research questions (rqs) for further analysis: (1) examining the influence of various error types on repository-level code debugging (rq1); and (2) investigating the effect of the number of errors on repository-level code debugging (rq2); and (3) analyzing the impact of different token length on repository-level code debugging (rq3). ## conclusion this work identifies the significant challenge of repository-level code debugging across multiple tasks and languages. to mitigate this challenge, we introduce a novel task focused on error identification, localization, and repair for repositorylevel code. we also present the first multi-task and multi-language dataset for repository-level code debugging, repodebug. to facilitate further research, we conducted comprehensive benchmarking experiments on repodebug, and the results highlight the limitations of llms on repository-level code debugging. we hope that repodebug will serve as a valuable dataset to promote future research in repository-level code debugging.