Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering (2024)

Abstract: Multi-Hop Question Answering (MHQA) task presents a significant challenge for large language models (LLMs) due to the intensive knowledge required. Current solutions, like Retrieval-Augmented Generation, typically retrieve potential documents from an external corpus to read an answer. However, the performance of this retrieve-then-read paradigm is constrained by the retriever and the inevitable noise in the retrieved documents. To mitigate these challenges, we introduce a novel generate-then-ground (GenGround) framework, synergizing the parametric knowledge of LLMs and external documents to solve a multi-hop question. GenGround empowers LLMs to alternate two phases until the final answer is derived: (1) formulate a simpler, single-hop question and directly generate the answer; (2) ground the question-answer pair into retrieved documents, amending any wrong predictions in the answer. We also propose an instructional grounding distillation method to generalize our method into smaller models. Extensive experiments conducted on four datasets illustrate the superiority of our method. To further facilitate future research, we have collected a dataset that traces the reasoning process.

## introduction multi-hop question answering (mhqa) tasks @xcite require multi-hop reasoning using intensive knowledge to derive the answer @xcite . it has been widely employed in various practical scenarios and domains @xcite ). to answer a multi-hop question, most prior work integrates large language models (llms) with information retrieval techniques, following a retrieve-then-read paradigm @xcite . as illustrated in figure 1 , the initial step employs the llms to break down the complex question and formulate a series of simpler, single questions in a step-by-step manner. for each step, the llms are guided to derive an answer from the relevant documents, which are retrieved using the formulated question @xcite . despite the progress of the retrieve-then-read paradigm, it faces two challenges in practice. first, its effectiveness is constrained by the performance of the retriever @xcite . given that the answer can only be derived from the retrieved documents, the inherent world knowledge of llms is overlooked. this limitation is particularly magnified in multi-hop qa tasks that frequently require complex logical reasoning @xcite . the retrievers may struggle to retrieve all necessary documents to answer the question, leading to a performance decline using this paradigm @xcite . second, the retrieved documents inevitably contain irrelevant or plausible-looking statements @xcite . directly incorporating them into the chain-ofreasoning of llms may mislead the llms to generate incorrect or irrelevant responses @xcite . therefore, developing an adaptive method for utilizing the retrieved documents remains an active research area. inspired by the extensive knowledge and powerful deduction capability of llms, e.g., chatgpt or gpt-4, we propose to address these challenges with a novel generate-then-ground (genground) method, as shown in figure 1 . this approach diverges from the retrieve-then-read paradigm by first allowing the llms to generate an immediate answer, and then grounding this answer in evidence to revise it. in this work, we focus on the following research questions: (1) rq1: how does our method synergize the world knowledge of llms and retrieved documents to answer a multihop question? (2) rq2: for llms with different scales of parameters, how do we generalize our method? to address rq1, we enable llms to alternate between answer deduction and instructional knowledge grounding. in the deduction phase, llms form sub-questions from the input question and context. the llms then produce an immediate answer for each sub-question. to prevent nonfactual hallucination by llms @xcite , we guide llms to revise the answer in the grounding phase, using documents retrieved from an external corpus like wikipedia. the llms ground the question-answer pair in evidence by citing relevant content and correcting errors. this revised answer is used for the next iteration's sub-question, continuing till the final hop. we also introduce a batch grounding strategy for efficient document use. to address rq2, we propose an instructional grounding distillation method. despite llms like chatgpt performing well with our method, smaller models may struggle with instructionfollowing in the grounding phase. thus, we use 50k single-hop questions from the natural questions (nq) dataset @xcite , each with ground-truth and noise documents. we guide chatgpt to generate and adjust an answer for each question, then distill chatgpt's process into a student model using instruction tuning. extensive experiments on four commonly-used mhqa datasets demonstrate superior performance over strong baselines (e.g., react and dspy), achieving the best performance overall. we also observe that our instructional grounding distillation empowers the smaller model with strong performance. to sum up, our main contributions are as follows: (1) we propose a novel generatethen-ground framework for retrieval-argument generation technique in multi-hop question tasks, which effectively synergizes the knowledge of llms and retrieved documents. (2) we introduce an instructional grounding distillation method, enabling a smaller model with the generatethen-ground framework. (3) experiments on four datasets are conducted to demonstrate the superiority of our method. ## generate-then-ground with llms this section provides a detailed explanation of genground. as depicted in figure 2 , genground empowers the llms to alternate between two phases over multiple iterations until the final answer a, is derived. these phases include answer deduction (section 3.1) and instructional knowledge grounding (section 3.2). during each iteration, the former guides the llms to generate here is an answer to the question. please cite evidence from the documents list to revise the answer. you should encapsulate the evidence using ''<@xmath0, and the revised answer using ''<@xmath1. if no evidence can be found, just give ''<@xmath2. a simpler, single-hop answer. the latter phase addresses llms' non-factual hallucination @xcite by prompting them to ground the question-answer pair in evidence and correct wrong predictions. the revised answer and sub-question are then integrated into the llms' context for the following iteration's prediction. ## generalization with grounding distillation while llms like chatgpt are skilled and adept at following instructions, they are often considered black boxes @xcite and their extensive parameters can increase latency and inference cost in real-world applications @xcite . thus, we aim to adapt our framework to smaller, open source models with fewer parameters. initial experiments show these smaller models struggle to cite relevant evidence during the knowledge grounding phase. to overcome this, we introduce instructional grounding distillation (idg), which distills the output trajectory of chatgpt into a smaller student model. ## conclusion we present a generate-then-ground (genground) framework for multi-hop question answering tasks, synergizing the parametric knowledge of llms and external documents to solve a multihop question. given a multi-hop question, our genground enable llms to alternate two phases until the final answer is derived: (1) formulate a 0 3000 6000 9000 token consumption 0.0 0.6 1.2 1.8 distribution 1e 3 : 3541.6 : 7806.4 : 8917.8 ours grg w/ decomposition retgen figure 6: the efficiency analysis for different methods, where we count the number of consumed tokens and compute the average consumption µ. simpler, single-hop question and directly generate the answer; (2) ground the question-answer pair in retrieved documents, amending any wrong predictions in the answer. to generalize our framework into smaller models, we also propose an instructional grounding distillation method. extensive experiments conducted on four datasets illustrate the superiority of our framework.