D epend E val: Benchmarking LLM s for Repository Dependency Understanding (2025)

Abstract: While large language models (LLMs) have shown considerable promise in code generation, real-world software development demands advanced repository-level reasoning. This includes understanding dependencies, project structures, and managing multi-file changes. However, the ability of LLMs to effectively comprehend and handle complex code repositories has yet to be fully explored. To address these challenges, we introduce a hierarchical benchmark designed to evaluate repository dependency understanding(DependEval) for LLMs. The benchmark is based on 2683 repositories collected from real-world websites. It evaluates models on three core tasks: Dependency Recognition, Repository Construction, and Multi-file Editing, across 8 programming languages from actual code repositories. Our evaluation of over 25 LLMs reveals substantial performance gaps and provides valuable insights into repository-level code understanding.

## related work code large language models. the rapid advancement of generative language models has spurred extensive research on ai applications in software engineering @xcite @xcite . @xcite @xcite @xcite @xcite have demonstrated significant performance improvements on various code-related tasks. these advancements have accelerated the development of code assistant tools like copilot foot_0 , tongyi lingma @xcite , and cursor 3 . ## conclusion we introduce dependeval, a benchmark designed to evaluate the repository-level dependency understanding of llms. built on a diverse set of 15,576 real-world repositories, the benchmark covers three key tasks: dependency recognition, repository construction, and multi-file editing, across eight programming languages. our evaluation of over 25 llms reveals significant performance gaps, highlighting the difficulties models face when managing complex code repositories. these insights point to the need for llms to improve their handling of dependencies, project structures, and multi-file modifications. our work sets the stage for future advancements in enhancing the repository-level reasoning capabilities of llms.