REANO : Optimising Retrieval-Augmented Reader Models through Knowledge Graph Generation (2024)

Abstract: Open domain question answering (ODQA) aims to answer questions with knowledge from an external corpus. Fusion-in-Decoder (FiD) is an effective retrieval-augmented reader model to address this task. Given that FiD independently encodes passages, which overlooks the semantic relationships between passages, some studies use knowledge graphs (KGs) to establish dependencies among passages. However, they only leverage knowledge triples from existing KGs, which suffer from incompleteness and may lack certain information critical for answering given questions. To this end, in order to capture the dependencies between passages while tacking the issue of incompleteness in existing KGs, we propose to enhance the retrieval-augmented reader model with a knowledge graph generation module (REANO). Specifically, REANO consists of a KG generator and an answer predictor. The KG generator aims to generate KGs from the passages and the answer predictor then generates answers based on the passages and the generated KGs. Experimental results on five ODQA datasets indicate that compared with baselines, REANO can improve the exact match score by up to 2.7% on the EntityQuestion dataset, with an average improvement of 1.8% across all the datasets.

## introduction the open domain question answering (odqa) task @xcite aims to answer questions with knowledge from an external corpus, such as wikipedia. retrieval-augmented models are an effective way of addressing the odqa task @xcite . these models employ a retriever-reader architecture that consists of both a retriever and a reader @xcite . the retriever model retrieves a set of passages that are relevant to the questions and then the reader model extracts or generates answers conditioned on the questions and the passages. the two-stage structure of odqa systems allows for the independent optimisation of the retriever and the reader @xcite . in this paper, we focus on enhancing the performance of the reader model, given that the effectiveness of odqa systems primarily depends on the reader's ability to interpret and process the information within the retrieved passages. there has been remarkable progress in using generative readers to address the odqa task @xcite . for example, @xcite proposed a generative reader called fusion-in-decoder (fid), which uses an encoder-decoder based t5 @xcite model to generate answers. by separately encoding each passage with the encoder and combining the token embeddings of these passages as inputs of the decoder, fid could effectively utilise the knowledge from each passage to generate answers. however, since fid independently encodes each passage, it overlooks the semantic relationships between passages, which are critical for answering questions that require multi-hop reasoning @xcite . to this end, several studies @xcite @xcite proposed to leverage knowledge graphs (kgs) to establish relational dependencies among passages. specifically, kgs store relational information between real-world entities in the form of triples ⟨head entity, relation, tail entity⟩. these knowledge triples are used to construct passage graphs @xcite or entity-passage graphs @xcite to enhance the multihop reasoning ability of the reader models. despite the success of these models, they all directly use the triples from existing kgs such as wikidata @xcite . however, these kgs often suffer from incompleteness @xcite and may lack certain information critical for answering questions that are present within the passages. for example, as illustrated figure 1 , for a question "who is younger, steven spielberg or james cameron?", existing kgs may lack the birthday information of these two individuals, which is crucial to correctly answer the question. this incompleteness can hinder the reader's ability to fully comprehend the passages and answer questions. therefore, in order to capture the dependencies between passages and tackling the issue of incompleteness in existing kgs, we propose to enhance the retrieval-augmented generative readers with a knowledge graph generation module (reano). specifically, reano consists of a kg generator and an answer predictor. the kg generator generates a kg, which consists of a set of knowledge triples, based on the retrieved passages. since the kg is inferred from the passages, it can effectively capture and preserve the critical information contained within these passages. after generating the kg, reano employs the answer predictor to perform inference over the unstructured passages and the structured kg to generate answers. our answer generator is based on the fid model. the key difference is that it leverages a graph neural network (gnn) @xcite to identify and select the top-k triples most relevant to the questions. these triples are concatenated in the order of relevance as an additional passage for answer generation. by combining relevant triples into a passage, we can gather information from multiple passages that are critical to answering the questions, alleviating the reasoning burden of the answer predictor. we conduct experiments on five odqa datasets and the results indicate that compared with state-of-the-art baselines, our reano improves the exact match (em) score by up to 2.7% on the entityquestion dataset, with an average improvement of 1.8% across all the datasets. our contribution can be summarised as follows: @xcite we propose reano, which integrates a kg generation module to enhance the performance of the generative readers; @xcite we use a gnn to identify and select the top-k knowledge triples that are most relevant to the questions from the generated kgs and combine these triples as an additional passage for enhanced answer generation; (3) experimental results on five odqa datasets show that reano can improve the em score by up to 2.7% compared with state-of-the-art baselines. fusion-in-decoder. the fusion-in-decoder (fid) model @xcite ) is a simple yet effective generative reader model for the odqa task. it leverages an encoder-decoder based t5 model @xcite to generate answers: ## reano the overall framework of our reano is shown in figure 2 . this section begins with the probabilistic formulation of reano in § 3.1, followed by its parameterisation details in § 3.2. finally, the training strategies are introduced in § 3.3. ## experiments 4.1 experimental setup datasets. we conduct experiments on five odqa datasets: natural questions (nq) @xcite , triviaqa (tqa) @xcite , entityquestions (eq) @xcite , 2wikimultihopqa (2wqa) @xcite and musique @xcite . for nq and tqa, we use the data splits provided by @xcite . each question in nq and tqa is associated with a set of 100 passages, which are retrieved by the dpr @xcite model from the december 20, 2018 wikipedia dump that contains 21m passages. the same data are also used in our baselines, except rag-seq. for eq, following @xcite , we use the same 21m wikipedia dump as the corpus and employ bm25 @xcite to retrieve top-20 relevant passages for each question. the 2wqa and musique are multi-hop qa datasets that require reasoning over multiple passages to answer questions. each question in 2wqa and musique is associated with 10 and 20 passages respectively provided by the authors. we directly use these passages in our experiments. more details about the datasets are provided in appendix b.1. baselines. since reano is built under the fid framework (izacard and grave, 2021b), we mainly compare it with the fid model and its variants. in particular, we compare with models from each of the following categories: (1) extractive reader: dpr @xcite ; (2) generative reader: rag-seq. @xcite , @xcite ; (3) kg-enhanced reader: kg-fid @xcite , oreolm @xcite , grape @xcite . due to resource constraints, both our reano and baselines use the base versions of the transformer models. evaluation. following izacard and grave (2021b), we use greedy decoding to generate answers and employ the exact match (em) as the evaluation metric, where a predicted answer is considered correct if it matches any answer in a list of gold answers after normalisation @xcite . training and hyperparameter details. as the backbone model and set the number of gnn layers l as 3. due to the resource constraints, we only use the top-50 passages for each question in both nq and tqa datasets and use all the passages per question in the other datasets. throughout the experiments, we set the number of triples selected by the gnn k as 10. we use adamw @xcite with a constant learning rate of 1e-4 as the optimizer and set the batch size as 64. more training details are provided in appendix b.2. ## conclusion this paper proposes reano to address the odqa task. reano aims to capture dependencies among passages with a knowledge graph generation module. specifically, it consists of a kg generator, which generates kgs based on the passages, and an answer predictor, which generates answers based on both the passages and the generated kgs. the answer predictor is based on the fid model, with an additional gnn model to identify and select knowledge triples relevant to questions. experimental results on five odqa datasets indicate that reano can improve the em score by up to 2.7% compared with state-of-the-art baselines.