U nited QA : A Hybrid Approach for Open Domain Question Answering (2021)

Abstract: To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid approach for leveraging the strengths of both models. We apply novel techniques to enhance both extractive and generative readers built upon recent pretrained neural language models, and find that proper training methods can provide large improvement over previous state-of-the-art models. We demonstrate that a simple hybrid approach by combining answers from both readers can efficiently take advantages of extractive and generative answer inference strategies and outperforms single models as well as homogeneous ensembles. Our approach outperforms previous state-of-the-art models by 3.3 and 2.7 points in exact match on NaturalQuestions and TriviaQA respectively.

## introduction open-domain question answering (qa) has been a long standing problem in natural language understanding, information retrieval, and related fields @xcite . an typical open-domain qa system follows the retrieval-reader framework @xcite @xcite , where the relevant passages are first retrieved from a large text corpus, and a reader module then navigates multiple passages for answer inference. in this work, we study two paradigms of reader modules, i.e. extractive @xcite and generative @xcite readers. the extractive reader extracts contiguous spans from the retrieved passages whereas the generative reader sequentially decodes the answer string which might not be contained in the retrieved passages. ## method in this section, we present the overall pipeline of the unitedqa system, which consists of three components: retrieval, reading, and re-ranking. first, the retrieval module fetches a list of relevant passages from a wikipedia dump for a given question. then, the module of hybrid readers produces answer candidates from the set of retrieved passages. last, the re-ranking module combines the answer candidates with linear interpolation and produce the final answer. retrieval following @xcite , we consider two methods, bm25 and dense passage retrieval (dpr), for retrieving the support passages for a given question. for bm25, passages are encoded as bag of words (bow), and inverse document frequencies are used as the ranking function. for dpr, passages and questions are represented as dense vectors based on two bert @xcite models. the relevance score is then computed based on the dot production between the query and passage vectors. in this paper, we adopt the same implementation as @xcite for retrieving passages. specifically, the english wikipedia dump from dec. 20, 2018 is used as the source documents for retrieval, with the removal of semi-structured data, such as tables or lists. each document is split into disjoint 100-word passages as the basic retrieval unit. the top-100 passages are then passed for reading. reading we combine the generative reader and the extractive reader to produce answer candidates over the retrieved passages. here, we only give a highlevel description of our approach. more details regarding our improved extractive and generative models are presented in §2.1 and §2.2 respectively. the generative reader is based on a sequenceto-sequence model pre-trained in a forwardgeneration fashion on a large corpus, i.e. t5 @xcite . similar to @xcite , the model takes the question and its relevant passages as input, and then generates the answer string token by token. specifically, the concatenation of all retrieved passages and the corresponding question is used as the encoder input. then, the decoder performs reasoning over the concatenation of all evidence through an attention mechanism. following state-of-the-art extractive qa models @xcite , our extractive reader is based on a transformer neural network pre-trained with a cloze style selfsupervised objective, i.e. electra @xcite . here, a pair of a given question and a support passage is jointly encoded into neural text representations. these representations are then used to define scores or probabilities of possible answer begin and end positions, which are in turn used to define probabilities over possible answer spans. finally, the answer string probabilities are based on the aggregation over all possible answer spans from the entire set of support passages. ## analysis in this section, we first carry out ablation study on the extractive and generative model improvements. moreover, we aim to take a deeper look and understand the difference between the two models. ## related work open-domain qa open-domain qa requires a system to answer questions based on evidence retrieved from a large corpus such as wikipedia @xcite . recent progress has been made towards improving evidence retrieval through both sparse vector models like tf-idf or bm25 @xcite , and dense vector models based on bert @xcite @xcite . generally, the dense representations complement the sparse vector methods for passage retrieval as they can potentially give high similarity to semantically related text pairs, even without exact lexical overlap. unlike most work focusing on a pipeline model, @xcite propose a pre-training objective for jointly training both the retrieval encoder and reader. it is further extended by @xcite with a dynamic update of the passage index during the training. instead, in this work, we focus on a hybrid reader approach for open-domain qa. by simply combing answer predictions from extractive and generative models, our unitedqa achieves significant improvements over state-of-the-art models. ## conclusion in this study, we propose a hybrid model for opendomain qa, called unitedqa, which combines the strengths of extractive and generative readers. we demonstrate the effectiveness of unitedqa on two popular open-domain qa benchmarks, natu-ralquestions and triviaqa. our results show that the proposed unitedqa model significantly outperforms single extractive and generative models as well as their corresponding homogeneous ensembles, and sets new state-of-the-art on both benchmarks. we also perform a comprehensive empirical study to investigate the relative contributions of different components of our model and the techniques we use to improve the readers. for future work, it would be interesting to explore model compression approaches for reducing the model size of retriever and reader separately or jointly through pruning, quantization, and knowledge distillation.