Knowledge Graph-Guided Retrieval Augmented Generation (2025)

Abstract: Retrieval-augmented generation (RAG) has emerged as a promising technology for addressing hallucination issues in the responses generated by large language models (LLMs). Existing studies on RAG primarily focus on applying semantic-based approaches to retrieve isolated relevant chunks, which ignore their intrinsic relationships. In this paper, we propose a novel Knowledge Graph-Guided Retrieval Augmented Generation (KG ^2 RAG) framework that utilizes knowledge graphs (KGs) to provide fact-level relationships between chunks, improving the diversity and coherence of the retrieved results. Specifically, after performing a semantic-based retrieval to provide seed chunks, KG ^2 RAG employs a KG-guided chunk expansion process and a KG-based chunk organization process to deliver relevant and important knowledge in well-organized paragraphs. Extensive experiments conducted on the HotpotQA dataset and its variants demonstrate the advantages of KG ^2 RAG compared to existing RAG-based approaches, in terms of both response quality and retrieval quality.

## introduction recently, large language models (llms) @xcite @xcite have achieved remarkable success across a broad range of real-world tasks, including question answering @xcite , writing assistance @xcite , code generation @xcite , and many others @xcite . however, hallucinations @xcite in the generated responses becomes a critical challenge, which often results from containing outdated information or lacking domain-specific knowledge. retrieval-augmented generation (rag) @xcite has emerged as a feasible solution to mitigate hallucinations by retrieving relevant knowledge from provided documents and incorporating it into the prompts of llms for response generation. existing studies in rag @xcite @xcite @xcite , as shown in fig. 1 , employ keyword-based or semantic-based approaches to retrieve documents or chunks having the highest similarities to user queries. however, these retrieved chunks can be homogeneous and redundant, which fails to provide the intrinsic relationships among these chunks and cannot further activate the reasoning abilities of llms. furthermore, the retrieved chunks are often directly concatenated in the order of their similarity scores and fed into llms as part of the prompts. such a practice can lead to isolated pieces of information, limiting the utility of llms in generating comprehensive and reliable responses. knowledge graphs (kgs) @xcite , as structured abstractions of realworld entities and their relations, can be expected to effectively supplement existing semantic-based rag approaches by integrating structured factual knowledge. knowledge within a kg, represented in the form of triplets (head entity, relation, tail entity), is naturally linked through overlapping entities. a simplified workflow for utilizing kgs in rag is shown in fig. 1 , where relevant triplets are retrieved to augment the context for response gen-eration in llms, providing fact-level relationships among chunks and highlighting important facts that may be missed by semantic-based approaches. shed light by such insights, in this paper, we propose a novel knowledge graph-guided retrieval augmented generation framework, called kg 2 rag. specifically, we first perform chunking and kg-chunk association during the offline processing of the provided documents, establishing linkages between chunks and a specific kg to capture the fact-level relationships among these chunks. based on the chunks and the kg, kg 2 rag employs kg-enhanced chunk retrieval, which consists of a semantic-based retrieval and graph-guided expansion. the semantic-based retrieval prepares several seed chunks using embedding and ranking techniques @xcite . these seed chunks are then used to extract a relevant subgraph from the association kg, onto which we can apply graph traversal algorithms to include the chunks containing overlapped or related entities and triplets. such a design of graph-guided expansion provides a greater diversity of retrieved chunks and a comprehensive knowledge network. after that, we incorporate a post-processing stage named kg-based context organization in kg 2 rag. on one hand, the kg-based context organization serves as a filter to retain the most relevant information contained in the subgraph, thereby enhancing the informativeness of the retrieved chunks. on the other hand, it serves as an arranger to organize the chunks into internally coherent paragraphs with the knowledge graph as a skeleton. these semantically coherent and wellorganized chunks are fed into the llms along with user queries for response generation. we conduct a series of experiments on the widely-used hotpotqa @xcite dataset and its newly constructed variants to mitigate the impacts of prior knowledge on llms. we adopt a distractor and a fullwiki setting, comparing kg 2 rag with several rag-based approaches. the experimental results demonstrate that kg 2 rag consistently outperforms baselines in terms of both response quality and retrieval quality. moreover, we conduct an ablation study to highlight the effectiveness of different modules in kg 2 rag. the constructed dataset and source code are released at https://github.com/ nju-websoft/kg2rag to further promote the development and application of kgs in rag. ## methodology an overview of the workflow of kg 2 rag is illustrated in fig. 2 . in the following subsections, we provide more details following the workflow of kg 2 rag, including document offline processing (sec. 2.1), kg-enhanced chunk retrieval (sec. 2.2), and kg-based context organization (sec. 2.3). ## related work retrieval-augmented generation to address the issues of hallucinations @xcite due to a lack of corresponding knowledge or containing outdated knowledge, retrievalaugmented generation (rag) @xcite in a typical rag system @xcite , the documents are first segmented into chunks based on lengths and structures, and then encoded with an embedding model @xcite and indexed for efficient retrieval. inspired by the idea of sliding windows @xcite , sentence window retrieval @xcite fetches the neighboring chunks around the retrieved chunks and concatenates them into a single larger chunk for context enrichment. however, sentence window retrieval only considers the physical proximity of text chunks within the same document. different from existing studies, kg 2 rag performs retrieval expansion based on factual associations among chunks that may be across multiple documents. reranking @xcite is a critical technique in information retrieval @xcite ). in rag systems, feeding the retrieved chunks along with the queries into a deep learning-based cross-encoder @xcite can measure the semantic relevance more precisely, thereby enhancing both the retrieval and generation quality. kg 2 rag organizes the retrieved chunks into paragraphs with kgs as the skeleton, allowing a fine-grained measurement of paragraph-level relevance to queries. llms with knowledge graph llm @xcite is one of the most representative achievements of contemporary artificial intelligence (ai). kgs @xcite , as graphstructured relational databases, serve as a crucial data infrastructure for ai applications. research indicates that llms have the potential to address tasks related to kgs, such as knowledge graph completion @xcite and knowledge graph question answering @xcite . recently, the research community begins to explore how kgs can be used to enhance the generation capability of llms @xcite @xcite . for example, kgp @xcite constructs a docu-ment kg consisting of page and passage nodes, and links passage nodes with tf-idf. the document kg is employed for retrieval expansion. the document kg constructed by kgp is based on sentencelevel text similarity, which essentially functions similarly to simply expanding the context window. graphrag @xcite targets at queryfocused summarization tasks. graphrag extracts kgs automatically from the document base with an llm and analyzes the semantic structure of the dataset before querying, by splitting the kg from different level and detecting linked nodes hierarchically. different from previous studies, kg 2 rag aims to enhance rag with the fact-level structure and factual knowledge of kgs. ## conclusion in this paper, we propose kg 2 rag, a novel framework designed to enhance the performance of rag through the integration of kgs. we introduce linkages between chunks and a specific kg, which help in providing fact-level relationships among these chunks. consequently, kg 2 rag suggests performing the kg-guided chunk expansion and the kg-based context organization based on seed chunks retrieved by semantic-based retrieval approaches. through these processes, the retrieved chunks become diverse, intrinsically related, and self-consistent, forming well-organized paragraphs that can be fed into llms for high-quality response generation. we compare kg 2 rag with existing rag-based approaches, demonstrating its superior performance in both response quality and retrieval quality. an ablation study is also conducted to further confirm the contributions of kg-guided chunk expansion and kg-based context organization, indicating that these two modules collaboratively enhance the effectiveness of kg 2 rag.