DI - BENCH : Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale (2025)

Abstract: Large Language Models have advanced automated software development, however, it remains a challenge to correctly infer dependencies, namely, identifying the internal components and external packages required for a repository to successfully run. Existing studies highlight that dependency-related issues cause over 40% of observed runtime errors on the generated repository. To address this, we introduce DI-BENCH, a large-scale benchmark and evaluation framework specifically designed to assess LLMs’ capability on dependency inference. The benchmark features 581 repositories with testing environments across Python, C#, Rust, and JavaScript. Extensive experiments with textual and execution-based metrics reveal that the current best-performing model achieves only a 48% execution pass rate on Python, indicating significant room for improvement. DI-BENCH establishes a new viewpoint for evaluating LLM performance on repositories, paving the way for more robust end-to-end software synthesis.

## introduction large language models (llms) have revolutionized automated software development, scaling from function-level code completion @xcite to repository-level code synthesis @xcite @xcite . a pivotal yet often overlooked step in this process is ensuring that generated repositories are fully executable. this requires accurate inference and integration of all necessary dependencies, both internal (across project components) and external (from package ecosystems). without robust dependency inference, even the most advanced code generation solutions risk failing at runtime, impeding further iteration, evaluation, and reliable deployment. as illustrated in figure 1 , dependency inference involves understanding the intricate relationships within the codebase and mapping out the external packages required for execution. such dependencies are typically documented in configuration files that may vary from language to language (see appendix a). correctly reconstructing these relationships is a foundational capability: it not only ensures that code generation tools produce functional and self-contained repositories, but it also informs deeper reasoning about project architecture and build systems @xcite . consequently, mastering dependency inference is a critical leap forward for enabling robust, end-toend software synthesis and maintenance. despite the significance of dependency inference, current llm-based approaches struggle in this area. works like chatdev @xcite and devbench @xcite )-pioneers in repository-level generation using multi-agent llm systems-have reported that dependency-related issues (e.g., missing or incorrectly specified modules) account for over 50% of their observed runtime errors. metagpt @xcite also demonstrates that missing or incorrectly generated dependencies represent one of the most significant hallucinations when llms attempt to generate the entire project. these challenges highlight the difficulty that state-of-the-art models face in accurately navigating build systems and package repositories. although existing repository-level benchmarks such as swebench @xcite , repobench @xcite , and devbench @xcite offer valuable insights into a model's ability to handle large contexts and generate code at scale, none focuses on systematically evaluating dependency inference capabilities. to address this critical gap, we introduce di-bench, the first comprehensive repository-level benchmark dedicated to dependency inference. di-bench comprises 581 verified repositories, including 387 regular-sized and 194 large-sized, across four popular programming languages (python, c#, rust, and javascript). each repository is carefully curated to assess a model's ability to identify both internal and external dependencies. we pair this dataset with a rigorous, multi-faceted evaluation framework. beyond measuring textual matching accuracy between model-generated and groundtruth dependencies, we propose a novel ci-based execution evaluation by reusing each repository's intrinsic continuous integration (ci) pipelines as automated test harnesses. this approach enables scalable and objective assessment of end-to-end executability, eliminating the costly and error-prone need for manual environment setup. through comprehensive experiments involving various llms and prompting strategies, we observed that even the best-performing llm achieved only a 48% executability rate on python. this finding highlights significant room for future improvement in this area. our analysis revealed that several factors influence performance, including the dependency amount and repository size. notably, issues such as hallucination and challenges related to dependency metadata emerged as critical bottlenecks that adversely affect model performance. in summary, our contributions are as follows: by spotlighting dependency inference and offering a dedicated benchmark, our work lays the foundation for advancing llms toward robust, end-to-end repository-level software synthesis. ## related works repository-level coding tasks have attracted increasing attention in recent years. many benchmarks @xcite @xcite center on code completion tasks at various granularities -from individual lines and api calls to entire function implementations. swe-bench and its variant @xcite challenge llms and llm-powered systems with real-world scenarios, using issues and pull requests from popular python repositories on github. while these benchmarks focus on assistant-like tasks, recent work explores llms' capabilities in complete project generation. devbench @xcite decomposes the development process into distinct stages and evaluates ai performance at each stage. agent-asa-judge @xcite introduces devai, innovatively employing llm agents as evaluators of development outcomes. however, existing works have not adequately addressed build configuration evaluation: code completion tasks @xcite @xcite do not generate build files, and issue-fixing benchmarks like swe-bench @xcite contain only 1% of patches related to build configurations. in repository generation tasks @xcite , build file generation is merely treated as one subtask without dedicated evaluation. recent works on dependency or version specific code generation @xcite @xcite have import sklearn.metrics import pandas as pd def get_metric(): if task @@xmath2.20.3", "scikit-learn@@xmath3.5.3", "scikit-learn$==1.1.0", ] ground truth 🤕 version mismatch: requires scikit-@xmath4.1.0 deprecated in version 1.2.0 run test attributeerror: module 'sklearn.metrics' has no attribute 'scorers' 🙂 run test test pass! llm may fail to identify dependencies used in its generated code 🧑💻 help me write the code do the following: ... explored code generation tasks based on evolving dependencies and api usage changes. our paper aims to infer the dependencies from existing code, which can be seen as the reverse process. prior research in dependency inference @xcite has predominantly focused on python ecosystems using traditional program analysis techniques, while lacking broader language coverage. our study fills in this gap by providing a benchmark specifically designed for evaluating dependency inference capability across multiple mainstream languages. ## dependency inference although many studies focus on repository code generation with llms recently, there exists a significant gap between the generated code and the executable and operational software, dependency. in this paper, we adapt dependency inference, which aims to generate a list of dependencies based on the source code. as shown in figure 2 , the code generated by llm cannot be executed directly without installing the required dependencies; however, it is non-trivial to identify the correct dependencies using llms. the example shows that the llm generates dependencies with a wrong version ('scikit-@xmath5.1.0' rather than 'scikit-@xmath6.2.2'), resulting in execution failure. automatic and accurate dependency inference makes end-to-end code development possible by installing the inferred dependencies for execution. furthermore, it can enable key scenarios like fullyautomated evaluation and iterative code improvement with execution feedback. besides the repository, dependency inference can be also applied to small code snippets like python notebook, incremental code changes, and etc. formally, the task is formulated as below: given a software repository containing many source code files and build configuration files where dependency-related sections are masked, the dependency inference task aims to generate a list of inferred dependencies to fill into the configuration. formally, we define the task as: ## di-bench focused on the task of dependency inference, we introduce di-bench, a meticulously curated, largescale benchmark dataset and evaluation framework at the repository level. di-bench encompasses 581 real-world, testable repository instances across 4 programming languages, providing a comprehensive platform for assessing llm-based methods in identifying and managing repository dependencies. ## experiment setup this section provides a detailed description of the experimental settings, including the llms, baseline methods, and evaluation metrics. baseline methods. we designed three baseline systems with various prompting strategies to evaluate how llms perform in the dependency inference task, intentionally avoiding complex techniques such as agent-based methods. additionally, we report the human performance in appendix c via recruiting experienced developers. two additional baselines including program analysis approach and retrieval-augmented generation (rag) baseline are presented in appendix e. metrics. we use textual and execution-based metrics, and the fake rate in evaluation. models. since code repositories are usually very long, we choose llms that support at least 128k context windows as the backbone models, including proprietary models: gpt-4o (ope-nai, 2024b), gpt-4o-mini (openai, 2024a), claude 3.5 sonnet @xcite , and gemini 2.0 flash @xcite , and open-source models: qwen-coder-v2.5-instruct @xcite , llama 3.1-instruct @xcite , @xcite and deepseek v3 @xcite . detailed settings on model serving are presented in appendix h.5. ## conclusion we introduce di-bench, the first benchmark dedicated to dependency inference across 581 repositories in four programming languages: python, c#, rust, and javascript. in addition to measuring textual accuracy, we propose a novel ci-based evaluation that incorporates actual tests execution. extensive experiments on various open-source and proprietary llms demonstrate that even the most advanced models struggle to infer dependencies accurately, highlighting opportunities for future advancements. we believe this study lays the groundwork for repository-level code development, with dependency inference serving as a pivotal step toward fully automated code generation.