R2-D2 : A Modular Baseline for Open-Domain Question Answering (2021)

Abstract: This work presents a novel four-stage open-domain QA pipeline R2-D2 (Rank twice, reaD twice). The pipeline is composed of a retriever, passage reranker, extractive reader, generative reader and a mechanism that aggregates the final prediction from all system’s components. We demonstrate its strength across three open-domain QA datasets: NaturalQuestions, TriviaQA and EfficientQA, surpassing state-of-the-art on the first two. Our analysis demonstrates that: (i) combining extractive and generative reader yields absolute improvements up to 5 exact match and it is at least twice as effective as the posterior averaging ensemble of the same models with different parameters, (ii) the extractive reader with fewer parameters can match the performance of the generative reader on extractive QA datasets.

## introduction last year showed rapid progress in neural factoid open-domain question answering based on retriever-reader architecture (open-qa). such open-qa systems @xcite seek evidence for answering the questions inside the knowledge source using the retriever and then extract the answer from the retrieved knowledge using the reader. the knowledge source is often a large corpus of short snippets of natural language, so-called passages (e.g., taken from an encyclopedia). the progress can be attributed to advances in neural retrieval methods @xcite @xcite @xcite , inter alia) that benefit from smarter negative sampling strategies or a better trade-off between complex question-passage interaction and its efficiency. it also can be attributed to reading methods that enable process1 our demo is available at http://r2d2.fit.vutbr.cz/ . code and preprocessed data are available at https://github.com/ knot-fit-but/r2-d2 . this work demonstrates the relative improvement of 23-32% compared to last year's state-ofthe-art dpr system @xcite , while using the same knowledge source and the retriever. we propose a state-of-the-art open-qa baseline composed of retriever, passage reranker, extractive reader, generative reader, and a novel component fusion approach. we follow the practice from information retrieval and show that our moderately sized reranker allows to reduce the passage count needed at the input of large reader models about four times. our readers then take the best from both worlds. the extractive reader proposes a list of salient answer spans. the generative reader reranks these spans, seeing all the passages at once, or generates its own answer. the proposed pipeline is heterogeneous and modular, making it an ideal benchmark. to sum up, our contributions are three-fold: ## open-qa pipeline we propose the r2-d2 (rank twice, read twice), 4-stage pipelined system that can choose whether to generate or to extract an answer. the parameters of each component in pipeline are estimated separately. it is composed of dpr passage retriever @xcite , passage reranker (see subsection 2.1), and two readers. figure 1 shows the diagram of our system. the first reader performs an extractive span-selection similar to @xcite . the second reader is based on fusion-in-decoder (fid) @xcite . formally, given a question @xmath0, p 2 , ..., p n } composed of passages p i , the retriever learns a ranking function rank : q×@xmath1. we assume each passage contains its passage title (e.g., title from the wikipedia article). taking a top-k scoring passages c r ⊂ c, reranker again rescores c r scoring passages by learning a reranking function rerank : q×c @xmath2. note that while rank and rerank have similar signatures, the computational cost of rerank over the same amount of passages is drastically higher, as it computes fine-grained interaction between tokens of question and passage. next, the rescored passages are passed to two readers: the extractive reader reads top-v passages c rr ⊂ c r independently of each other and assigns the probability p e (a e |q, c rr ) to each span a e in the passages (see subsection 2.2). the fid generative reader reads top-v 2 passages c rr ⊂ c r jointly and generates an answer from probability space p g (a g |q, c rr ) via greedy search. finally, r2-d2 aggregates the outputs from all components using two fusions (described in subsection 2.3). ## experimental setup our models are implemented in pytorch @xcite using transformers @xcite . we use 12gb gpu to train the passage reranker, 48gb gpu for the generative reader, and 16x 32gb gpus to train the extractive reader with @xmath3. the inference runs on 12gb gpu. in all experiments, we used adam optimizer with a decoupled weight decay @xcite . our models are evaluated by two metrics: exact match (em) measures the proportion of examples, for which the system prediction matched at least one annotated ground-truth answer. we use the script from @xcite 4 . ## https://cutt.ly/rkznier accuracy@k measures the proportion of examples, for which the ground-truth answer string is present in top-k retrieved passages. we match the string exactly as @xcite 5 . ## results and analysis the effectiveness of our approach is compared with the state-of-the-art in ## related work passage reranking. previous work in qa based on neural nets used bi-lstm encoders @xcite that score each document independently. over time, bi-lstm were replaced by bert-like transformer encoders @xcite . for document ranking, @xcite proposed a multistage architecture. the first stage scores each document independently, and the second estimates the more relevant document from all document pairs. another document ranking approach uses the seq2seq model to generate a true or false answer to the document's relevance to the query @xcite . recent works have often focused on effective reranking. @xcite achieved inference speedup using early exiting, jang and kim (2020) proposed a smaller and faster model, and @xcite came up with a method which uses reader's predictions to rerank the passages. our reranker is most similar to @xcite ; @xcite , except that unlike in ir, we assume there is just one correct passage and thus train our model via categorical cross-entropy. multipassage reading comprehension related work considers generative and extractive approaches towards modeling the reader. the generative reader generates an answer while conditioned on question alone @xcite , or question with relevant passages @xcite . @xcite showed it suffices to concatenate the passages in the decoder of seq2seq model, increasing the amount of top-passages the model can depend on dramatically. the extractive reader used in open-qa assumes that the answer is a continuous span string in located in retrieved paragraphs @xcite . @xcite proposed to aggregate the probabilities of distantly supervised answer matches via maximum marginal likelihood (mml). lin et al. (2018) proposed to denoise distantly supervised answer string matches in mml via paragraph-ranker. cheng et al. (2020a) experimented with different assumptions for mml, showing improvement when marginalizing over components of span probability independently. fajcik et al. (2020) proposed to model joint span probability directly via compound objective, instead of modeling the probability of span's start and end independently. karpukhin et al. (2020) incorporated an independent passage classifier loss to his mml objective. our objective is similar to the last work, except that it uses joint component and also optimizes mml over relevant passages' probabilities. component fusion. yang et al. (2019) also combined bm25 ranker and reader scores via linear combination. our work can be seen as an extension of this idea to combining the scores of all pipeline's components. iyer et al. (2020) proposed a system which directly learns to rerank questionpassage-answer triplets proposed via extractive model. however, reranking answers from their large extractive model via large reranker leads to ~1 em improvement absolute, whereas r2-d2s score aggregation improves 4 to 5 em w.r.t. the extractive reader. concurrently with our work, cheng et al. ( 2021 ) proposed hard voting ensembling scheme to combine the reader predictions. firstly, each model from an ensemble produces its best prediction, then the votes for identical predictions are combined, omitting the scores produced by the individual models. the authors obtained best results using two fid readers and single extractive reader, leading to 1.6 and 2.4 em improvement on tq-open and nq-open, compared to their best single extractive or generative model. ## conclusion this work proposed r2-d2, a novel state-of-the-art pipeline for open-domain qa based on 4 components: retriever, reranker, generative reader and extractive reader. we showed that employing a reranker is a reasonable alternative to using large passage counts at the input of both the extractive and the generative reader. our results on nq-open and efficientqa showed that the extractive and the generative reader could perform equally in open-qa, although the generative reader is twice the size of the extractive reader. on the other hand, we observe the extractive reader underperforms on tq-open. we hypothesize, that the cause is (1) the complexity of trivia questions with many entities, which often require combining evidence from multiple passages -these are impossible to answer for the extractive reader by design -and (2) the expensive hyperparameter search, as we used nq-open hyperparameters also for tq-open. contrary to belief based on the results on different datasets @xcite @xcite , we found the extractive reader can also benefit from larger input sizes, both in training and test time. finally, we proposed a component fusion, which allows merging the complementary behavior of generative and extractive approaches along with the ranking components and found it improves the results significantly. due to its heterogenous and modular nature, our pipeline forms an ideal base for future research of component integration in modern open-qa.