KG - F i D : Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering (2022)

Abstract: Current Open-Domain Question Answering (ODQA) models typically include a retrieving module and a reading module, where the retriever selects potentially relevant passages from open-source documents for a given question, and the reader produces an answer based on the retrieved passages. The recently proposed Fusion-in-Decoder (FiD) framework is a representative example, which is built on top of a dense passage retriever and a generative reader, achieving the state-of-the-art performance. In this paper we further improve the FiD approach by introducing a knowledge-enhanced version, namely KG-FiD. Our new model uses a knowledge graph to establish the structural relationship among the retrieved passages, and a graph neural network (GNN) to re-rank the passages and select only a top few for further processing. Our experiments on common ODQA benchmark datasets (Natural Questions and TriviaQA) demonstrate that KG-FiD can achieve comparable or better performance in answer prediction than FiD, with less than 40% of the computation cost.

## introduction open-domain question answering (odqa) is the task of answering natural language questions in open domains. a successful odqa model relies on effective acquisition of world knowledge. a popular line of work treats a large collection of open-domain documents (such as wikipedia articles) as the knowledge source, and design a odqa system that consists of a retrieving module and a reading module. the retriever pulls out a small set of potentially relevant passages from the opensource documents for a given question, and the reader produces an answer based on the retrieved passages @xcite @xcite ). an earlier example of this kind is drqa @xcite , which used * work done during internship at microsoft. an traditional search engine based on the bag of words (bow) document representation with tf-idf term weighting, and a neural reader for extracting candidate answers for each query based on the dense embedding of the retrieved passages. with the successful development of pre-trained language models (plms) in neural network research, dense embedding based passage retrieval (dpr) models @xcite have shown superior performance over bow/tf-idf based retrieval models due to utilization of contextualized word embedding in dpr, and generative qa readers @xcite ) usually outperform extraction based readers @xcite due to the capability of the former in capturing lexical variants with a richer flexibility. the recently proposed fusion-in-decoder (fid) model @xcite is representative of those methods with a dpr retriever and a generative reader, achieving the state-of-the-art results on odqa evaluation benchmarks. fid also significantly improved the scalability of the system over previous generative methods by encoding the retrieved passages independently instead of encoding the concatenation of all retrieved passages (which was typical in previous methods). inspired by the success of fid, this paper aims further improvements of the state of the art of odqa in the paradigm with a dpr retriever and a generative reader. specifically, we point out two potential weaknesses or limitations of fid as the rooms for improvements, and we propose a novel solution namely kg-fid to address these issues with fid. the two issues are: issue 1. the independent assumption among passages is not justified. notice that both the dpr retriever and the generative reader of fid perform independent encoding of the retrieved passages, which means that they cannot leverage the semantic relationship among passages for passage embedding and answer generation even if such relational knowledge is available. but we know that rich semantic connections between passages often provide clues for better answering questions @xcite . issue 2. efficiency bottleneck. for each input question, the fid generative reader receives about 100 passages from the dpr module, with a relatively high computational cost. for example, the inference per question takes more than 6 trillion floating-point operations. simply reducing the number of retrieved passages sent to the reader will not be a good solution as it will significantly decrease the model performance @xcite . how to overcome such inefficient computation issue is a challenging question for the success of fid in realistic odqa settings. we propose to address both of the above issues with fid by leveraging an existing knowledge graph (kg) to establish relational dependencies among retrieved passages, and employing graph neural networks (gnns) to re-rank and prune retrieved passages for each query. we name our new approach as kg-fid. specifically, kg-fid employs a two-stage passage reranking by applying gnn to model structural and semantic information of passages. both stages rerank the input passages and only a few top-reranked passages are fed into subsequent modules. the first stage reranks passages returned by the retriever, where we use the passage embeddings generated by dpr as the initial gnn node representation. this allows reranking a much larger set of initial candidate passages to enhance coverage of answers. the second stage performs joint passage reranking and answer generation, where the node embeddings are initialized by the embeddings of passage-question pairs output from the reader encoder. this stage operates on a smaller candidate set but aims for more accurate reranking and passage pruning. to improve the efficiency, in the second-stage reranking, our gnn model adopts representations from the intermediate layer in the reader encoder instead of the final layer to initiate passage node embeddings. then only a few top reranked passages will be passed into the higher layers of encoder and the decoder for answer generation, while other passages will not be further processed. this is coupled with a joint training of passage reranking and answer generation. as shown in section 4.3, these strategies significantly reduce the computation cost while still maintaining a good qa performance. our experiments on odqa benchmark datasets natural questions and triviaqa demonstrate that kg-fid can achieve comparable or better performance in answer prediction than fid, with only 40% of the computation cost of fid. ## related work odqa with text corpus odqa usually assumes that a large external knowledge source is accessible and can be leveraged to help answer prediction. for example, previous works @xcite @xcite mainly use wikipedia as knowledge source which contains millions of text passages. in this case, current odqa models mainly contains a retriever to select related passages and a reader to generate the answer. thus, the follow-up works mainly aim to: (1) improve the retriever: from sparse retrieval based on tf-idf or bm25 @xcite to dense retrieval @xcite based on contextualized embeddings generated by pre-trained language models (plms). moreover, some further improvement are also proposed such as better training strategy @xcite , reranking based on retrieved passages @xcite @xcite , and knowledge distillation from reader to retriever (izacard and grave, 2020); (2) improve the reader: changing from recurrent neural network @xcite to plms such as extractive reader bert @xcite @xcite and generative reader bart and t5 @xcite . besides, some works @xcite @xcite have shown that additional unsupervised pre-training on retrieval-related language modeling tasks can further improve odqa performance. however, none of these methods modeled the relationships among different passages. odqa with knowledge graph besides the unstructured text corpus, world knowledge also exists in knowledge graphs (kgs), which represent entities and relations in a structural way and have been used in a variety of nlp tasks @xcite @xcite . some works @xcite @xcite ) restrict the answer to be entities in the knowledge graph, while our work focus on more general odqa setting where the answer can be any words or phrases. under this setting, some recent efforts have been made to leverage knowledge graphs for odqa @xcite @xcite . for example, unik-qa @xcite transforms kg triplets into text sentences and combine them into text corpus, which loses structure information of kg. other works use kg to build relationship among passages similar to ours. kaqa @xcite use passage graph to propagate passage retrieve scores and answer span scores. graph-retriever @xcite iteratively retrieve passages based on the relationship between passages, and also use passage graph to improve passage selection in an extractive reader. however, applying kg to improve the recent advanced fid framework remains unstudied. ## method in the following sections, we first introduce how to apply kg to build a graph structure among the retrieved passages (section 3.1). then we show how we adopt the graph-based stage-1 reranking with dpr retriever to improve passage retrieval (section 3.2). next we introduce joint stage-2 reranking and answer generation in the reading module (section 3.3). finally we illustrate the improvement of efficiency by using intermediate layer representation for stage-2 reranking (section 3.4). the overview of our framework is illustrated in figure 1 . ## experiment in this section, we conduct extensive experiments on two most commonly-used odqa benchmark datasets: natural questions (nq) @xcite which is based on google search queries, and triviaqa @xcite which contains questions from trivia and quiz-league websites. we follow the same setting as @xcite to preprocess these datasets, which is introduced in appendix a.1. all our experiments are conducted on 8 tesla a100 40gb gpus.