Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs (2024)

Abstract: Improving the performance of large language models (LLMs) in complex question-answering (QA) scenarios has always been a research focal point. Recent studies have attempted to enhance LLMs’ performance by combining step-wise planning with external retrieval. While effective for advanced models like GPT-3.5, smaller LLMs face challenges in decomposing complex questions, necessitating supervised fine-tuning. Previous work has relied on manual annotation and knowledge distillation from teacher LLMs, which are time-consuming and not accurate enough. In this paper, we introduce a novel framework for enhancing LLMs’ planning capabilities by using planning data derived from knowledge graphs (KGs). LLMs fine-tuned with this data have improved planning capabilities, better equipping them to handle complex QA tasks that involve retrieval. Evaluations on multiple datasets, including our newly proposed benchmark, highlight the effectiveness of our framework and the benefits of KG-derived planning data.

## introduction the past few years have witnessed significant innovations in llms @xcite @xcite . while llms excel in many natural language processing tasks, they still face challenges, particularly the smaller models, in handling complex question-answering (qa) tasks @xcite @xcite @xcite . to improve the performance of llms on complex qa tasks, past research has tried various methods: (1) employing carefully designed prompt strategies to guide the model in reasoning, such as chain of thought (cot) @xcite q: what sports have fluminense and fran walsh's spouse played in? q1: who is fran walsh's spouse? a1: ans_1 q2: what sports does {ans_1} play? a2: ans_2 q3: what sports does fluminense play? ## related works reasoning and planning with llms in the context of llms, reasoning typically involves decomposing complex questions into sub-questions @xcite . prominent techniques include chain-of-thought (cot) prompting @xcite which elicits rationales that lead to the final answers, and its extension, using self-consistency @xcite or automated demonstration selection @xcite . other methods, such as react @xcite , generate reasoning steps sequentially by integrating planning, with additional strategies like tree of thoughts (tot) @xcite , reasoning via planning (rap) @xcite , and other methods @xcite facilitating complex question decomposition through varied planning approaches. unlike most methods that rely on in-context learning through prompt engineering, our approach generates planning data from kgs to fine-tune llm, thereby enhancing their planning capabilities. retrieval-augmented generation retrieval-augmented generation (rag) can enhance llms by incorporating external data, allowing models to access up-to-date information and factual knowledge to mitigate hallucinations @xcite @xcite . each module in the rag pipeline can be optimized, for instance, through retriever tuning @xcite , self-reflection during retrieval @xcite , or query refinement @xcite . to address multihop questions, iterative rag models @xcite @xcite have been developed, which iteratively conduct retrievalenhanced generation and generation-enhanced retrieval. however, the multiple rag steps in existing methods are not optimized and rely heavily on in-context learning. our approach uses planning data from kgs to facilitate more efficient rag. llms with kgs in the existing realm of llms, kgs are primarily utilized as sources of structured factual knowledge @xcite . for example, think-on-graph @xcite extracts relevant triples from kgs to assist in qa. reasoning on graph (rog) @xcite generates relation-based plans and retrieves corresponding paths from these graphs. while aiding in kgqa tasks where answers are directly sourced from kgs, these graphs also support rationale generation. chain-of-knowledge (cok) @xcite further leverages kgs along with other heterogeneous sources to generate faithful rationales. unlike previous studies, our approach constructs planning data for complex questions from kgs, recognizing that patterns within kgs inherently represent multi-step plans. this data is utilized to enhance the planning capabilities of llms. complex logical query in kgs recent research on complex logic queries in kgs primarily focuses on first-order logical (fol) queries that incorporate operations like conjunctions, disjunctions, negation, and existential quantifiers within incomplete kgs @xcite @xcite @xcite @xcite . these works define diverse patterns to assess the capability of logical operations in vector spaces, specifically targeting logical forms rather than natural language. nonetheless, their methodologies for pattern definition and extraction inspire our approach to deriving complex questions from kgs. ## experiment we aim to answer the following research questions in our experiments: ## conclusion in this paper, we try to enhance the planning ability in retrieval-augmented llms using kgs. specifically, we design a framework for learning to plan from kg (lpkg). the proposed lpkg framework first utilizes the rich patterns in the kgs to construct planning data, then fine-tune planning llms based on such data to enable them to conduct planning on downstream datasets, and ultimately get the final answer through parsing and execution. the experimental results reveal the excellent performance of the lpkg framework and also demonstrate the effectiveness of using kg-sourced data to enhance llms' planning ability. finally, we construct clqa-wiki, providing a more challenging complex qa benchmark for the community.