C odex G raph: Bridging Large Language Models and Code Repositories via Code Graph Databases (2025)

Abstract: Large Language Models (LLMs) excel in stand-alone code tasks like HumanEval and MBPP, but struggle with handling entire code repositories. This challenge has prompted research on enhancing LLM-codebase interaction at a repository scale. Current solutions rely on similarity-based retrieval or manual tools and APIs, each with notable drawbacks. Similarity-based retrieval often has low recall in complex tasks, while manual tools and APIs are typically task-specific and require expert knowledge, reducing their generalizability across diverse code tasks and real-world applications. To mitigate these limitations, we introduce CodexGraph, a system that integrates LLM agents with graph database interfaces extracted from code repositories. By leveraging the structural properties of graph databases and the flexibility of the graph query language, CodexGraph enables the LLM agent to construct and execute queries, allowing for precise, code structure-aware context retrieval and code navigation. We assess CodexGraph using three benchmarks: CrossCodeEval, SWE-bench, and EvoCodeBench. Additionally, we develop five real-world coding applications. With a unified graph database schema, CodexGraph demonstrates competitive performance and potential in both academic and real-world environments, showcasing its versatility and efficacy in software engineering. Our code and demo will be released soon.

## introduction large language models (llms) excel in code tasks, impacting automated software engineering @xcite @xcite . repository-level tasks @xcite @xcite mimic software engineers' work with large codebases @xcite . these tasks require models to handle intricate dependencies and comprehend project structure @xcite . current llms struggle with long-context inputs, limiting their effectiveness with large codebases @xcite and lengthy sequences reasoning @xcite . researchers have proposed methods to enhance llms by retrieving task-relevant code snippets and structures, improving performance in complex software development @xcite @xcite . however, these approaches mainly rely on either similarity-based retrieval @xcite @xcite or manual tools and apis @xcite . similarity-based retrieval methods, common in retrieval-augmented generation (rag) systems @xcite , often struggle with complex reasoning for query formulation @xcite and handling intricate code structures @xcite , leading to low recall rates. meanwhile, existing tool/api-based interfaces that connect codebases and llms are typically task-specific and require extensive expert knowledge @xcite . furthermore, our experimental results in section 5 indicate that the two selected methods lack flexibility and generalizability for diverse repository-level code tasks. recent studies have demonstrated the effectiveness of graph structures in code repositories @xcite . meanwhile, inspired by recent advances in graph-based rag @xcite @xcite and the application of executable code (such as sql, cypher, and python) to consolidate llm agent actions @xcite @xcite , we present codexgraph, as shown in figure 1 (a) . codexgraph alleviates the limitations of existing approaches by bridging code repositories with llms through graph databases. codexgraph utilizes static analysis to extract code graphs from repositories using a task-agnostic schema that defines the nodes and edges within the code graphs. in these graphs, nodes represent source code symbols such as module, class, and function, and each node is enriched with relevant meta-information. the edges between nodes represent the relationships among these symbols, such as contains, inherits, and uses (see figure 2 for an illustrative example). by leveraging the structural properties of graph databases, codexgraph enhances the llm agent's comprehension of code structures. codexgraph leverages repository code information and graph structures for global analysis and multi-hop reasoning, enhancing code task performance. when users provide code-related inputs, the llm agent analyzes the required information from the code graphs, constructs flexible queries using graph query language, and locates relevant nodes or edges. this enables precise and efficient retrieval, allowing for effective scaling to larger repository tasks. to evaluate the effectiveness of the codexgraph, we assess its performance across three challenging and representative repository-level benchmarks: cross-codeeval @xcite , swe-bench @xcite and evocodebench @xcite . our experimental results demonstrate that, by leveraging a unified graph database schema (section 3.1) and a simple workflow design (section 3.2), the codex-graph achieves competitive performance across all academic benchmarks, especially when equipped with more advanced llms. furthermore, as illustrated in figure 1 (b), to address real-world software development needs, we extend codexgraph to the featurerich modelscope-agent @xcite framework. section 6 highlights five real-world application scenarios, including code debugging and writing code comments, showcasing the versatility and efficacy of codexgraph in practical software engineering tasks. our contributions are from three perspectives: we demonstrate codexgraph's competitive performance on three challenging and representative repository-level code benchmarks. ## codexgraph: enable llms to navigate the code repository codexgraph is a system that bridges code repositories and large language models (llms) through code graph database interfaces. it indexes input code repositories using static analysis, storing code symbols and relationships as nodes and edges in a graph database according to a predefined schema. when presented with a coding question, codexgraph leverages the llm agent to generate graph queries, which are executed to retrieve relevant code fragments or code structures from the database. the detailed processes of constructing the code graph database and the llm agent's interactions with it are explained in sections 3.1 and 3.2, respectively. a further explation 3.1 build graph databases from code repositories schema. we abstract code repositories into code graphs where nodes represent symbols in the source code, and edges represent relationships between these symbols. the schema defines the types of nodes and edges, directly determining how code graphs are stored in the graph database. different programming languages typically require different schemas based on their characteristics. in our project, we focus on python and have empirically designed a schema tailored to its features, with node types including module, class, method, function, field, and global variable, and edge types including contains, inherits, has method, has field, and uses. each node type has corresponding attributes to represent its meta-information. for instance, method nodes have attributes such as name, file path, class, code, and signature. for storage efficiency, nodes with a code attribute do not store the code snippet directly in the graph database but rather an index pointing to the corresponding code fragment. figure 2 illustrates a sample code graph derived from our schema, and appendix a.1 shows the details of the schema. phase 1: shallow indexing. the code graph database construction process consists of two phases, beginning with the input of the code repository and schema. the first phase employs a shallow indexing method, inspired by sourcetrail's static analysis process foot_0 , to perform a single-pass scan of the entire repository. during this scan, symbols and relationships are extracted from each python file, processed only once, and stored as nodes and edges in the graph database. concurrently, meta-information for these elements is recorded. this approach ensures speed and efficiency, capturing all nodes and their meta-information in one pass. however, the shallow indexing phase has limitations due to its single-pass nature. some important edges, particularly certain inherits and contains relationships, may be overlooked as they might require context from multiple files. function: "square_root" field: "last_result", "radius" global_variable: "pi" # example edges contains: summary. our code graph database design offers four key advantages for subsequent use. first, it ensures efficient storage by storing code snippets as indexed references rather than directly in the graph database. second, it enables multi-granularity searches, from module-level to variable-level, accommodating diverse analytical needs. third, it facilitates topological analysis of the codebase, revealing crucial insights into hierarchical and dependency structures. last, this schema design supports multiple tasks without requiring modifications, demonstrating its versatility and general applicability. these features collectively enhance the system's capability to handle complex code analysis tasks effectively across various scenarios. regarding the discussion of indexing efficiency, please refer to appendix a.6. ## experimental setting benchmarks. we employ three diverse repositorylevel code benchmarks to evaluate codexgraph: crosscodeeval @xcite , swe-bench @xcite , and evocodebench @xcite . crosscodeeval is a multilingual scope cross-file completion dataset for python, java, typescript, and c#. swe-bench evaluates a model's ability to solve github issues with 2, 294 issue-pull request pairs from 12 python repositories. evocodebench is an evolutionary code generation benchmark with comprehensive annotations and evaluation metrics. we report our primary results on the crosscodeeval lite (python) and swe-bench lite test sets for cross-codeeval and swe-bench, respectively, and on the full test set for evocodebench. crosscodeeval lite (python) and swe-bench lite represent subsets of their respective datasets. crosscodeeval lite (python) consists of 1000 randomly sampled python instances, while swe-bench lite includes 300 instances randomly sampled after filtering out those with poor issue descriptions. remark: during indexing of 43 sympy samples from the swe-bench dataset, we face out-of-memory issues due to numerous files and complex dependencies, leading to their exclusion. similarly, some evocodebench samples are omitted due to test environment configuration issues. thus, swe-bench lite and evocodebench results are based on 257 and 212 samples, respectively. baselines. we evaluate whether codexgraph is a powerful solution for retrieval-augmented code generation (racg) @xcite . we specifically assess how effectively code graph database interfaces aid llms in understanding code repositories, particularly when handling diverse code questions across different benchmarks to test codexgraph 's general applicability. to achieve this, we select resilient racg baselines that can be adapted to various tasks. based on the categories in section 2.2, we choose bm25 @xcite and autocoderover @xcite , which are widely recognized in code tasks @xcite @xcite , along with a no-rag method. besides, since our work focuses on racg methods and their generalizability, we exclude methods that interact with external websites @xcite and runtime environments @xcite , as well as task-specific methods that are not easily adaptable across multiple benchmarks @xcite . these methods fall outside the scope of our project. especially, although @xcite evaluate autocoderover exclusively on swe-bench, we extend its implementation to crosscodeeval and evocodebench, while retaining its core set of 7 codespecific tools for code retrieval. large language models (llms). we evaluate codexgraph on three advanced llms with long text processing, tool use, and code generation capabilities: gpt-4o, deepseek-coder-v2 @xcite , and qwen2-72b-instruct @xcite . for the hyperparameters of the selected large language models, we empirically set the temperature coefficient to 0.0 for both gpt-4o and qwen2-72b-instruct, and to 1.0 for deepseek-coder-v2. all other parameters are kept at their default settings. ## real-world application scenario to showcase codexgraph's practical value, we developed five code agents using the modelscope-agent framework @xcite . these agents address common coding challenges that involve understanding and navigating complex inter-file dependencies: code chat (repository inquiry), code debugger (bug diagnosis and resolution), code unittestor (test generation), code generator (new feature implementation), and code commentor (documentation enhancement). each agent integrates key codexgraph concepts to solve specific production environment issues. examples and details are provided in appendix a.3, with figure 5 illustrating code chat's webui. ## conclusion codexgraph addresses the limitations of existing racg methods, which often struggle with flexibility and generalization across different code tasks. by integrating llms with code graph database interfaces, codexgraph facilitates effective, code structureaware retrieval for diverse repository-level code tasks. our evaluations highlight its competitive performance and broad applicability on academic benchmarks. additionally, we provide several code applications in modelscope-agent, demonstrating codexgraph 's capability to enhance the accuracy and usability of automated software development. the qualitative analysis and the schema explanation have been postponed to appendix a.1 and a.2, respectively.