Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2021)

Abstract: Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages. This is evidence that sequence-to-sequence models offers a flexible framework to efficiently aggregate and combine evidence from multiple passages.

## introduction recently, several works have shown that factual information can be extracted from large scale language models trained on vast quantities of data @xcite @xcite . building on that observation and the advances in pretraining of natural language processing models, @xcite introduced a generative model for open domain question answering. without relying on external knowledge, this method obtained competitive results on several benchmarks. however, it requires models containing billions of parameters, since all the information needs to be stored in the weights. this makes models expensive to query and train. in this paper, we investigate how much this method could benefit from having access to an external source of knowledge, such as wikipedia. retrieval based approaches were previously considered in the context of open domain question answering with extractive models @xcite . in that case, systems start by retrieving alan turing was a british computer scientist. born in maida vale, london… where was alan turing born? generative seq2seq model maida vale, london then, a generative encoder-decoder model produces the answer, conditioned on the question and the retrieved passages. this approach scales well with the number of retrieved passages, as the performance keeps improving when retrieving up to one hundred passages. aggregating and combining evidence from multiple passages is not straightforward when using extractive models, and multiple techniques have been proposed to address this limitation @xcite . in this paper, we explore a simple approach having the best of both worlds, by building on the exciting developments in generative modeling and retrieval for open domain question answering. this method proceeds in two steps, by first retrieving supporting passages using either sparse or dense representations. then, a sequence-to-sequence model generates the answer, taking as input the retrieved passages in addition to the question. while conceptually simple, this method sets new state-ofthe-art results on the triviaqa and naturalquestions benchmarks. in particular, we show that the performance of our method significantly improves when the number of retrieved passages increases. we believe that this is evidence that generative models are good at combining evidence from multiple passages, compared to extractive ones. ## related work open domain question answering is the task of answering general domain questions, in which the evidence is not given as input to the system. while being a longstanding problem in natural language processing @xcite , this task has recently regained interest following the work by @xcite . in that version of the problem, strong supervision is available to the learning system, in the form of spans corresponding to answers. @xcite proposed to solve the problem by first retrieving support document from wikipedia, before extracting the answer from the retrieved document. different methods were proposed to tackle the setting where no gold spans are given to the system, but only the correct answer. @xcite proposed to use a global normalization over all the span corresponding to the answer, which was later applied to bert based models @xcite . @xcite introduced a method based on hard expectationmaximization to tackle noisy supervision from this setting. @xcite described a technique to aggregate answers from different paragraphs, using confidence and coverage scores. passage retrieval is an important step in open domain question answering, and is an active area of research to improve qa systems. initially, sparse representations based on tf/idf were used to retrieve support documents @xcite . @xcite introduced a supervised learning method to rerank paragraphs based on bilstm, while @xcite trained a ranking system with reinforcement learning. a second approach to improve the retrieval step of qa systems is to used additional information such as the wikipedia or wikidata graphs @xcite . recently, multiple works show that retrieval systems entirely based on dense representation and approximate nearest neighbors were competitive with traditional approaches. such models can be trained using weak supervision in the form of question-answer pairs @xcite , or pretrained using a cloze task and finetuned end-toend @xcite . generative question answering was mostly considered in previous work for datasets requiring to generate answers, such as narrativeqa @xcite , coqa @xcite or eli5 @xcite . these datasets were generated in a way that answers do not correspond to spans in support documents, thus requiring abstractive models. @xcite showed that generative models are competitive for reading comprehension tasks such as squad @xcite , where answers are spans. @xcite proposed to use large pretrained generative models, without using additional knowledge, for open domain question answering. closest to our work, @xcite and @xcite introduced retrieval augmented generative models for open domain question answering. our approach differs from these works by how the generative model processes the retrieved passages. this allows to scale to large numbers of documents, and to benefit from this large amount of evidence. ## method in this section, we describe our approach to open domain question answering. it proceeds in two steps, first retrieving support passages before processing them with a sequence to sequence model. model nq triviaqa squad open em em em em f1 drqa (chen et al., 2017) ---29.8 -multi-passage bert ( wang et al., 2019) ---53.0 60.9 path retriever @xcite 31.7 --56.5 63.8 graph retriever @xcite 34.7 55.8 ---hard em @xcite 28.8 50.9 ---orqa @xcite 31.3 45.1 -20.2 -realm @xcite 40.4 ----dpr @xcite 41.5 57.9 -36.7 -spanseqgen @xcite 42.5 ----rag @xcite 44.5 56.1 68.0 -- retrieval. for the retrieval of support passages, we consider two methods: bm25 @xcite and dpr @xcite . in bm25, passages are represented as bag of words, and the ranking function is based on term and inverse document frequencies. we use the implementation from apache lucene 1 with default parameters, and tokenize questions and passages with spacy. 2 in dpr, passages and questions are represented as dense vector representations, computed using two bert networks. the ranking function is the dot product between the query and passage representations. retrieval is performed using approximate nearest neighbors with the faiss library. 3 reading. our generative model for open domain qa is based on a sequence-to-sequence network, pretrained on unsupervised data, such as t5 or bart @xcite . the model takes as input the question, as well as the support passages, and generates the answer. more precisely, each retrieved passage and its title are concatenated with the question, and processed independently from other passages by the encoder. we add special tokens question:, title: and context: before the question, title and text of each passage. finally, the decoder performs atten-1 lucene.apache.org 2 spacy.io 3 github.com/facebookresearch/faiss tion over the concatenation of the resulting representations of all the retrieved passages. the model thus performs evidence fusion in the decoder only, and we refer to it as fusion-in-decoder. by processing passages independently in the encoder, but jointly in the decoder, this method differs from @xcite and @xcite . processing passages independently in the encoder allows to scale to large number of contexts, as it only performs self attention over one context at a time. this means that the computation time of the model grows linearly with the number of passages, instead of quadratically. on the other hand, processing passages jointly in the decoder allows to better aggregate evidence from multiple passages. ## experiments in this section, we report empirical evaluations of fusion-in-decoder for open domain qa. datasets. we consider the following datasets, and use the same setting as @xcite : following @xcite we use the validation as test, and keep 10% of the training set for validation. we use the wikipedia dumps from dec. 20, 2018 for nq and triviaqa and from dec. 21, 2016 for squad. we apply the same preprocessing as @xcite ; @xcite , leading to passages of 100 words, which do not overlap. ## conclusion in this paper, we study a simple approach to open domain question answering, which relies on retrieving support passages before processing them with a generative model. we show that while conceptually simple, this approach is competitive with existing methods, and that it scales well with the number of retrieved passages. in future work, we plan to make this model more efficient, in particular when scaling to large number of support passages. we also plan to integrate the retrieval in our model, and to learn the whole system end-to-end.