KERAG : Knowledge-Enhanced Retrieval-Augmented Generation for Advanced Question Answering (2025)

Abstract: Retrieval-Augmented Generation (RAG) mitigates hallucination in Large Language Models (LLMs) by incorporating external data, with Knowledge Graphs (KGs) offering crucial information for question answering. Traditional Knowledge Graph Question Answering (KGQA) methods rely on semantic parsing, which typically retrieves knowledge strictly necessary for answer generation, thus often suffer from low coverage due to rigid schema requirements and semantic ambiguity. We present KERAG, a novel KG-based RAG pipeline that enhances QA coverage by retrieving a broader subgraph likely to contain relevant information. Our retrieval-filtering-summarization approach, combined with fine-tuned LLMs for Chain-of-Thought reasoning on knowledge sub-graphs, reduces noises and improves QA for both simple and complex questions. Experiments demonstrate that KERAG surpasses state-of-the-art solutions by about 7% in quality and exceeds GPT-4o (Tool) by 10-21%.

## introduction retrieval-augmented generation (rag) has been extensively explored recently in both academia and industry, as a means to mitigate the hallucination issues of large language models (llms) @xcite @xcite @xcite . rag enhances llm outputs by incorporating relevant data from external sources, thereby grounding the generated content with factual knowledge. knowledge graphs serve as a crucial data source, providing efficient access to rich and precise information for question answering @xcite . knowledge graph question answering (kgqa) is a well-established field that has been extensively studied long before the advent of llms. most solutions utilize the technique of semantic parsing, where a natural language question is converted into a structured query, in the form of either a sparql query @xcite @xcite @xcite @xcite , or a retrieval path that first identifies the queried entity and then follows a particular predicate to a neighboring entity or attribute value as the answer @xcite @xcite . when successful, these solutions produce precise and concise answers. however, they require rigorous schemas (ontologies) and have low tolerance to parsing errors (see figure 1 for an example). as a result, whereas kgqa generally provides highly accurate answers, it often suffers from low coverage, missing answers for many questions. with llms emerging, recent works @xcite leverage the reasoning ability of llms to generate multiple retrieval paths; however, the small number of paths (e.g., up to 3 in @xcite ) does not fully address the retrieval recall issue. llms, with their strong summarization and reasoning capabilities, offer a new perspective to pursue kgqa and allow us to push reasoning from retrieval-path generation to the summarization step. at retrieval time, instead of following up to a few retrieval paths, we gather from the neighborhood of the topic entity all information potentially relevant to the question. at answer generation time, we leverage llm's reasoning capabilities to identify relevant information and generate accurate answers. by focusing on the whole neighborhood rather than a few kg paths, the complexity of generating logical forms for kg querying is significantly reduced and the retrieval recall can significantly increase. although the idea appears straightforward, there are three prominent challenges. first, entities, particularly head entities, can be associated with a substantial amount of knowledge (2 million triples per head entity @xcite ). all knowledge may not fit within the context window, and even if it does, the noises can lead to confusion during summarization. second, some questions require information from multi-hop neighbors, complicating the determination of retrieval boundaries, and further increasing the volume of retrieved information. lastly, complex questions that necessitate aggregation and reasoning remain challenging. we proposed kerag, a novel knowledge-enhanced rag pipeline that employs a retrievalfilter-summarization paradigm. the retrieval part gathers information about the topic entity mentioned in the question; the filtering part eliminates information irrelevant to the question; and the summarization part reasons over the remaining information to generate the answer. rather than retrieving the entities and triples strictly necessary for gen-erating an answer, our method retrieves a broader subgraph that is likely to contain relevant information for answer generation. there are two key innovations that ensure highquality answers. first, in our planning phase we interleave multi-hop neighborhood retrieval with filtering according to the kg schema, allowing us to filter at the predicate level rather than overwhelming the summarizer with irrelevant triples, and meanwhile ensuring that all information necessary for question answering is retrieved from the neighborhood. second, we fine-tune the llm for chain-of-thought (cot) reasoning over the neighborhood sub-graph, enabling it to generate accurate answers for both simple questions and complex questions. an additional advantage of our method is its applicability not only to graph databases that support sparql, but also to apibased knowledge-providing systems, which are commonly used in production @xcite . to summarize, we make three contributions. ## related work one major line of research of kgqa is semanticparsing-based (sp-based) kgqa approaches, which primarily transform natural language queries into logical forms and initiate structured queries to access the knowledge graph content. the spbased kgqa approaches can be further divided into multi-step and seq2seq approaches. the multistep approaches generally formulate the semantic parsing problem as a multi-step search problem: identifying the core entity, generating and expanding the query graph based on the relationships between the attributes of the entities and the predicates of the queries @xcite @xcite . the seq2seq approaches primarily consider the semantic parsing problem in a seq2seq manner with the help of fine-tuned language models: directly generating the complete semantic expression based on the query and retrieving the relevant knowledge graph content @xcite @xcite @xcite @xcite . specifically, pangu @xcite presents an innovative framework for grounded language understanding that integrates a symbolic agent with a neural language model. this system enables the stepwise formulation of valid plans while employing the language model to evaluate the viability of these logical plans in an online manner. wikisp is a novel few-shot seq2seq semantic parser developed by @xcite , which fine-tunes a llama model on modified sparql queries. another line of research in kgqa is the information-retrieval-based kgqa approaches, where a subgraph is generated based on the entities and relations in the query, graph matching and retrieval techniques are applied to retrieve relevant graphs from the kg to answer the query @xcite @xcite @xcite . for instance, @xcite introduces a novel approach with a trainable subgraph retriever (sr) designed to operate separately from the reasoning process, identifying relevant subgraphs and prioritizing smaller and more relevant subgraphs. the subgraphs identified by our approach are more relaxed compared to @xcite , which provide a broader context for the summarizer to generate accurate answers. thanks to the strong reasoning ability of language models, an increasing number of works start to seek or analyze the assistance of language models @xcite @xcite @xcite @xcite . the kgqa community also focuses on llm-based approaches recently, which utilize large language models (llms) to perform kgqa @xcite @xcite . as for the llm-based kgqa approaches, several stateof-the-art llms such as llama-3.1s @xcite support tool calling, which provides the opportunity to directly prompt the llms to generate api calls or sparql queries. however, retrieving directly with the tool call plan given by the llm would propagate planning errors into the retrieval process, resulting in insufficient kg content being retrieved. we provide a comparison of performance between our kerag pipeline and the llama-3.1-70b-instruct (tool) model (which can freely plan and initiate multiple rounds of api calls) in tables 1 and 2. beyond naive llm tool calling, tog @xcite introduces beam search on the knowledge graph to iteratively explore reasoning paths for llms. tog-2 @xcite links to external wikipedia pages to get additional context, the retrieval and reasoning are conducted between kg and external pages. structgpt @xcite designs an interface for data retrieval and a unique process that involves invoking, linearizing, and generating outputs. our method is fundamentally different: these approaches retrieve information from kgs based on retrieval paths, whereas kerag retrieves all potentially relevant information in a neighborhood, thus improving on the retrieval recall (~11% over tog, section 5.4). as such, kerag achieves a higher accuracy on all benchmarks, improving over tog and tog-2 by up to 20% and 4% respectively ( ## problem statement knowledge graph: a knowledge graph (kg) k can be defined as @xmath0, r, d), where e is a set of entities, r is a set of relations and optionally d is a set of domains covered by k.