TRACE the Evidence: Constructing Knowledge-Grounded Reasoning Chains for Retrieval-Augmented Generation (2024)

Abstract: Retrieval-augmented generation (RAG) offers an effective approach for addressing question answering (QA) tasks. However, the imperfections of the retrievers in RAG models often result in the retrieval of irrelevant information, which could introduce noise and degrade the performance, especially when handling multi-hop questions that require multiple steps of reasoning. To enhance the multi-hop reasoning ability of RAG models, we propose TRACE. TRACE constructs knowledge-grounded reasoning chains, which are a series of logically connected knowledge triples, to identify and integrate supporting evidence from the retrieved documents for answering questions. Specifically, TRACE employs a KG Generator to create a knowledge graph (KG) from the retrieved documents, and then uses a novel Autoregressive Reasoning Chain Constructor to build reasoning chains. Experimental results on three multi-hop QA datasets show that TRACE achieves an average performance improvement of up to 14.03% compared to using all the retrieved documents. Moreover, the results indicate that using reasoning chains as context, rather than the entire documents, is often sufficient to correctly answer questions.

## introduction retrieval-augmented generation (rag) models have achieved remarkable performance on question answering (qa) task @xcite @xcite . these models employ a retriever-reader architecture @xcite . the retriever is used to retrieve a set of documents relevant to the questions, and the reader generates answers based on these documents. moreover, the reader is often instantiated with large language models (llms) due to their powerful in-context learning capabilities, leading to superior zero-shot performance. in this setting, the retrieved documents are prepended to the question, which is used as input to the llms to generate answers @xcite . however, simply prepending all the documents returned by the retriever can result in suboptimal performance. this is because existing retrievers are not perfect and often include irrelevant documents in the retrieved set. these irrelevant documents introduce noises, which can mislead the reader and degrade performance @xcite . this issue is particularly problematic when answering multi-hop questions, which involve multiple reasoning steps to obtain the correct answers. previous study indicates that irrelevant documents can significantly impair the multi-hop reasoning ability of rag models @xcite . therefore, this work focuses on improving the multi-hop reasoning capability of rag models by enhancing their ability to identify and integrate supporting evidence within documents. the supporting evidence refers to the information within documents that directly contributes to answering the questions @xcite . to this end, we propose trace, which constructs knowledgegrounded reasoning chains to identify and integrate supporting evidence from multiple documents. figure 1 provides an illustration of trace. specifically, in order to identify supporting evidence, trace first transforms the retrieved documents into a knowledge graph (kg), i.e., a set of knowledge triples in the form of ⟨head entity, relation, tail entity⟩ that describe relationships between entities. this is achieved by using in-context learning to prompt an llm instructed as a kg generator to generate knowledge triples from the retrieved documents. the motivation for converting documents into a kg is that, compared with sentences or documents, which contain multiple pieces of information, knowledge triples offer a finer-grained and more concise way to express knowledge, where each triple only conveys a single piece of factual knowledge. leveraging kg triples can reduce the impact of irrelevant data when identifying supporting evidence @xcite , leading to more accurate identification of relevant information. for example, the sentence "albert einstein (14 march 1879-18 april 1955) was a german-born theoretical physicist." contains multiple pieces of information about albert einstein, including his birth and death days, nationality, and profession. one of the triples generated from this sentence could be ⟨albert einstein, date of birth, 14 march 1879⟩, which decouples the birthday information from the sentence. this finer granularity helps in minimising the inclusion of irrelevant information, making it easier to identify supporting evidence when answering questions related to einstein's birthday. notably, the generation of the kg is independent of questions. trace next aims to identify and integrate supporting evidence from the kg to answer multi-hop questions. specifically, trace employs an autoregressive reasoning chain constructor to construct reasoning chains from the kg. each reasoning chain comprises several kg triples that logically connect pieces of supporting evidence to answer the questions. for example, for a multi-hop question "when was the father of albert einstein born?", "⟨albert einstein, father, hermann ein-stein⟩, ⟨hermann einstein, date of birth, 3 july 1814⟩" is a reasoning chain that provides the necessary information to answer the question. these reasoning chains facilitate the integration of dispersed pieces of supporting evidence, thereby enhancing the model's ability to generate correct answers. moreover, the reasoning chains are constructed in an autoregressive manner. at each step, trace uses in-context learning to prompt the constructor to select a triple from the kg based on both the question and the previously selected triples. the objective is to ensure that the selected triple forms a logically coherent reasoning chain with the previously selected triples. the autoregressive way of constructing reasoning chains is inspired by human reasoning, where each piece of information is considered in the context of what has already been understood. this step-by-step reasoning approach is particularly suitable for multi-hop questions, as it can trace the logical connections across multiple pieces of evidence, ensuring an accurate inference process. for example, in the previously mentioned multi-hop question, if trace has already identified one piece of supporting evidence: ⟨albert einstein, father, hermann einstein⟩, it can then focus on finding the next relevant piece of evidence, i.e., ⟨hermann einstein, date of birth, 3 july 1814⟩. consequently, compared to vanilla rag models, trace creates a kg from the retrieved documents and constructs reasoning chains from the kg in an autoregressive manner to identify and integrate supporting evidence. given the reasoning chains, the trace reader either directly uses them as context to generate the answer (trace-triple), or use them to identify a subset of documents that are useful for answering the question (trace-doc). we conduct experiments on three multi-hop qa datasets in a zero-shot setting, our results show that, compared to using all the retrieved documents, trace-triple and trace-doc achieve average improvements of 14.03% and 13.46% in terms of exact match (em), respectively. moreover, our results indicate that, in the rag setting, constructing more condensed reasoning chains (i.e., kg triples) from the retrieved documents as context, rather than using the entire documents, is often sufficient to correctly answer questions. our contributions are summarised as follows: (1) we propose trace, which builds knowledgegrounded reasoning chains to enhance the multihop reasoning ability of rag models; (2) we propose a novel autoregressive method to construct reasoning chains to identify and integrate supporting evidence; (3) experimental results on three multi-hop qa datasets show that trace achieves average improvement of up to 14.03% in terms of em compared to using all the retrieved documents. ## problem formulation this work focuses on tackling multi-hop questions. we denote a multi-hop question and its answer as q and a, respectively. each question is associated with a set of n documents: d @xmath0, d 2 , . . . d n }, which are obtained with a retriever model. following previous work @xcite , the documents are often retrieved from wikipedia, where each document comprises a title and a text. given the question q and the document set d q , the goal is to correctly generate the answer a. ## trace this section begins by outlining the overall framework of trace in § 3.1. next, we explain the details of each component in the following sections: kg generator in § 3.2, reasoning chain constructor in § 3.3 and finally, answer generation in § 3.4. ## conclusion this paper proposes trace to enhance the multihop reasoning ability of rag models. specifically, trace employs a kg generator to create a kg from the retrieved documents and then uses a novel autoregressive reasoning chain constructor to build reasoning chains from the kg. given the reasoning chains, trace either directly uses them as context to generate answers or uses them to identify a subset of relevant documents. experimental results on three multi-hop qa datasets show that, compared to using all the retrieved documents, trace achieves an average performance of up to 14.03% in em. moreover, the reasoning chains can effectively generalise across various reader models.