A Framework of Knowledge Graph-Enhanced Large Language Model Based on Question Decomposition and Atomic Retrieval (2024)

Abstract: Knowledge graphs (KGs) can provide explainable reasoning for large language models (LLMs), alleviating their hallucination problem. Knowledge graph question answering (KGQA) is a typical benchmark to evaluate the methods enhancing LLMs with KG. Previous methods on KG-enhanced LLM for KGQA either enhance LLMs with KG retrieval in a single round or perform multi-hop KG reasoning in multiple rounds with LLMs. Both of them conduct retrieving and reasoning based solely on the whole original question, without any processing to the question. To tackle this limitation, we propose a framework of KG-enhanced LLM based on question decomposition and atomic retrieval, called KELDaR. We introduce question decomposition tree as the framework for LLM reasoning. This approach extracts the implicit information of reasoning steps within complex questions, serving as a guide to facilitate atomic retrieval on KG targeting the atomic-level simple questions at leaves of the tree. Additionally, we design strategies for atomic retrieval, which extract and retrieve question-relevant KG subgraphs to assist the few-shot LLM in answering atomic-level questions. Experiments on KGQA datasets demonstrate that our framework outperforms existing reasoning-based baselines. And in a low-cost setting without additional training or fine-tuning, our framework achieves competitive or superior results compared to most existing training-based baselines.

## introduction to tackle challenges such as interpretability and hallucinations @xcite in large language models (llms), some studies @xcite have sought to enhance llm reasoning using knowledge graphs (kgs). these approaches offer domain-specific and real-time knowledge, aiding llms in performing interpretable reasoning and reducing hallucinations. in * corresponding author. the evaluation of kg-enhanced llm methods, knowledge graph question answering (kgqa) is an important task aiming to answer given questions based on factual knowledge in kg. previous works on kg-enhanced llm for kgqa proposed retrieve-then-answer methods aiding llm with kg retrieval in a single round interaction between llm and kg , as shown in figure 1 (top left). they retrieve question-relevant kg facts, which are then fed into the llm all at once as references for answer generating @xcite @xcite . however, these methods rely heavily on the kg retrieval results, using the llm only as a tool for reasoning based on the logic in retrieval results and extracting the final answer from them. this limits the flexibility of llm in reasoning @xcite and doesn't fully leverage its strengths in question reasoning. to enable the llm to play a more active role in reasoning, existing methods enable multi-round llm-kg interaction for llms to perform multihop reasoning on kg, as shown in figure 1 (bottom left). starting from the topic entities of the question, they prompt the llm to select the most relevant next-hop relations or entities on kg. and iterate this process until llm determines that the chain of entities and relations it traversed is sufficient to answer the question @xcite . however, these methods perform only onehop retrieval of relations and entities on kg during each round of llm-kg interaction, resulting in low efficiency. additionally, each hop's retrieval is based solely on the whole original question, without making llm aware of the specific sub-question that needs to be addressed at the current step for more targeted selection. the methods above on kg-enhanced llm for kgqa are all based on original questions without analyzing their logic explicitly during reasoning and kg retrieval. to tackle this limitation, we draw inspiration from a work training small language models on explainable question answering @xcite , which integrate heterogeneous knowledge sources and analyze the semantics of complex questions with question decomposition tree. our idea is to introduce this structure into a framework of kg-enhanced llm. to realize our idea, we need to tackle two key problems. first, we need to design a framework capable of performing question decomposition and enabling atomic retrieval for atomic-level questions. second, considering the shortcoming on efficiency in previous kg retrieval, it's also important to design the kg extracting and retrieving strategy for atomic retrieval. to this end, we propose a framework of kg-enhanced llm based on question decomposition and atomic retrieval (keldar). to excavate the information of reasoning logic in questions, we introduce the question decomposition tree as a framework for llm reasoning, which classifies the questions according to their complexity and allows complex questions to be decomposed into a tree for multi-step reasoning, enabling atomic retrieval on kg for corresponding sub-question of each reasoning step. to improve the efficiency of the atomic retrieval, we further design efficient strategies for extracting and retrieving question-relevant kg subgraph of facts, expanding the candidate subgraph to a pruned two-hop range. to evaluate the effectiveness of our framework, we conducted experiments on two commonly used kgqa datasets @xcite . experimental results demonstrate that our proposed framework significantly enhances the reasoning performance of the llm. our framework outperforms reasoning-based baselines without additional training or fine-tuning like us, and even achieves competitive or superior results compared to most training-based baselines. our main contributions are as follows: ## conclusion this study introduces a framework of kgenhanced llm based on question decomposition and atomic retrieval, called keldar. we introduce the question decomposition tree as a framework for llm reasoning, which allows atomic kg retrieval targeting each reasoning step. additionally, we designed strategies of atomic retrieval to extract and retrieve question-relevant kg facts, expanding the range of candidate subgraph to two hops. experimental results demonstrate that our proposed framework significantly outperforms the state-ofthe-art reasoning-based methods.