Retrieval-Augmented Generation with Hierarchical Knowledge (2025)

Abstract: Graph-based Retrieval-Augmented Generation (RAG) methods have significantly enhanced the performance of large language models (LLMs) in domain-specific tasks. However, existing RAG methods do not adequately utilize the naturally inherent hierarchical knowledge in human cognition, which limits the capabilities of RAG systems. In this paper, we introduce a new RAG approach, called HiRAG, which utilizes hierarchical knowledge to enhance the semantic understanding and structure capturing capabilities of RAG systems in the indexing and retrieval processes. Our extensive experiments demonstrate that HiRAG achieves significant performance improvements over the state-of-the-art baseline methods.

## introduction retrieval augmented generation (rag) @xcite @xcite has been introduced to enhance the capabilities of llms in domain-specific or knowledge-intensive tasks. naive rag methods retrieve text chunks that are relevant to a query, which serve as references for llms to generate responses, thus helping address the problem of "hallucination" @xcite . however, naive rag methods usually overlook the relationships among entities in the retrieved text chunks. to address this issue, rag systems with graph structures were proposed @xcite @xcite , which construct knowledge graphs (kgs) to model relationships between entities in the input documents. although existing rag systems integrating graph structures have demonstrated outstanding performance on various tasks, they still have some serious limitations. @xcite introduces communities in indexing using the leiden algorithm @xcite , but the communities only capture the structural proximity of the entities in the kg. kag @xcite indexes with a hierarchical representation of information and knowledge, but their hierarchical structure relies too much on manual annotation and requires a lot of human domain knowledge, which renders their method not scalable to general tasks. ligh-trag @xcite utilizes a dual-level retrieval approach to obtain local and global knowledge as the contexts for a query, but it ignores the knowledge gap between local and global knowledge, that is, local knowledge represented by the retrieved individual entities (i.e., entity-specific details) may not be semantically related to the global knowledge represented in the retrieved community summaries (i.e., broader, aggregated summaries), as these individual entities may not be a part of the retrieved communities for a query. we highlight two critical challenges in existing rag systems that integrate graph structures: challenge (1) occurs because existing methods over-rely on source documents, often resulting in constructing a knowledge graph (kg) with many entities that are not structurally proximate in the kg even though they share semantically similar attributes. for example, in figure 1 , although the entities "big data" and "recommendation system" share semantic relevance under the concept of "data mining", their distant structural relationship in the kg reflects a corpus-driven disconnect. these inconsistencies between semantic relevance and structural proximity are systemic in kgs, undermining their utility in rag systems ## related work in this section, we discuss recent research concerning graph-augmented llms, specifically rag methods with graph structures. gnn-rag (mavromatis and karypis, 2024) employs gnn-based reasoning to retrieve query-related entities. then they find the shortest path between the retrieved entities and candidate answer entities to construct reasoning paths. lightrag @xcite integrates a dual-level retrieval method with graph-enhanced text indexing. they also decrease the computational costs and speed up the adjustment process. grag @xcite leverages a soft pruning approach to minimize the influence of irrelevant entities in retrieved subgraphs. it also implements prompt tuning to help llms comprehend textual and topological information in subgraphs by incorporating graph soft prompts. structrag (li et al., 2024) identifies the most suitable structure for each task, transforms the initial documents into this organized structure, and subsequently generates responses according to the established structure. microsoft graphrag (edge et al., 2024) first retrieves related communities and then let the llm generate the response with the retrieved communities. they also answer a query with global search and local search. kag (liang et al., 2024) introduces a professional domain knowledge service framework and employs knowledge alignment using conceptual semantic reasoning to mitigate the noise issue in openie. kag also constructs domain expert knowledge using human-annotated schemas. reg (zou et al., 2025) uses llms to refine weak supervision signals to align retrievers in fine-grained graph-based rag systems. ## preliminary and definitions in this section, we give a general formulation of an rag system with graph structure referring to the definitions in @xcite and @xcite . in an rag framework m as shown in equation 1, llm is the generation module, r represents the retrieval module, φ denotes the graph indexer, and ψ refers to the graph retriever: when we answer a query, the answer we get from an rag system is represented by a * , which can be formulated as most of the time, we only need to retrieve the most relevant subgraph g from the external graph database g. therefore, here we can approximate m(a|q, g) as follows: ## the hirag framework hirag consists of the two modules, hiindex and hiretrieval, as shown in figure 2 . in the hiindex module, we construct a hierarchical kg with different knowledge granularity in different layers. the summary entities in a higher layer represent more coarse-grained, high-level knowledge but they can enhance the connectivity between semantically similar entities in a lower layer. in the hiretrieval module, we select the most relevant entities from each retrieved community and find the shortest path to connect them, which serve as the bridge-level knowledge to connect the knowledge at both local and global levels. then an llm will generate responses with these three-level knowledge as the context. ## experimental evaluation we report the performance evaluation results of hirag in this section. baseline methods. we compared hirag with state-of-the-art and popular baseline rag methods. naiverag @xcite splits original documents into chunks and retrieves relevant text chunks through vector search. graphrag (edge et al., 2024) utilizes communities and we use the local search mode in our experiments as it retrieves community reports as global knowledge, while their global search mode is known to be too costly and does not use local entity descriptions. lightrag @xcite uses both global and local knowledge to answer a query. fastgraphrag @xcite integrates kg and personalized pagerank as proposed in @xcite . kag @xcite integrates structured reasoning of kg with llms and employs mutual indexing and logicalform-guided reasoning to enhance professional domain knowledge services. ## conclusions we presented a new approach to enhance rag systems by effectively utilizing graph structures with hierarchical knowledge. by developing (1) hiindex which enhances structural and semantic connectivity across hierarchical layers, and (2) hiretrieval which effectively bridges global conceptual abstractions with localized entity descriptions, hi-rag achieves superior performance than existing methods.