C - MORE : Pretraining to Answer Open-Domain Questions by Consulting Millions of References (2022)

Abstract: We consider the problem of pretraining a two-stage open-domain question answering (QA) system (retriever + reader) with strong transfer capabilities. The key challenge is how to construct a large amount of high-quality question-answer-context triplets without task-specific annotations. Specifically, the triplets should align well with downstream tasks by: (i) covering a wide range of domains (for open-domain applications), (ii) linking a question to its semantically relevant context with supporting evidence (for training the retriever), and (iii) identifying the correct answer in the context (for training the reader). Previous pretraining approaches generally fall short of one or more of these requirements. In this work, we automatically construct a large-scale corpus that meets all three criteria by consulting millions of references cited within Wikipedia. The well-aligned pretraining signals benefit both the retriever and the reader significantly. Our pretrained retriever leads to 2%-10% absolute gains in top-20 accuracy. And with our pretrained reader, the entire system improves by up to 4% in exact match.

## introduction open-domain question answering (qa) aims to extract the answer to a question from a large set of passages. a simple yet powerful approach adopts a two-stage framework @xcite , which first employs a retriever to fetch a small subset of relevant passages from large corpora (i.e., retriever) and then feeds them into a reader to extract an answer (text span) from them. due to its simplicity, a sparse retriever such as tf-idf/bm25 is generally used together with a trainable reader @xcite . however, recent advances show that transformer-based dense retrievers trained on supervised data @xcite can greatly boost the performance, which better captures the semantic relevance between the question and the correct passages. such approaches, albeit promising, are restricted by the limited amount of human annotated training data. inspired by the recent progresses of language models pretraining @xcite @xcite , we would like to address the following central question: can we pretrain a two-stage open-domain qa system (retriever + reader) without task-specific human annotations? unlike general language models, pretraining such a system that has strong transfer capabilities to downstream open-domain qa tasks is challenging. this is mainly due to the lack of well-aligned pretraining supervision signals. in particular, we need the constructed pretraining dataset (in the form of question-answer-context triplets) to: (i) cover a wide range of domains (for open-domain applications), (ii) link a question to its semantically relevant context with supporting evidence (for training the retriever), and (iii) identify the correct answer in the context (for training the reader). there have been several recent attempts in addressing these challenges. orqa @xcite creates pseudo query-passage pairs by randomly sampling a sentence from a paragraph and treating the sampled sentence as the question while the rest sentences as the context. realm @xcite ) adopts a retrieve-then-predict approach, where the context is dynamically retrieved during training and an encoder (reader) predicts question: the boarding crew freed 14 iranian and pakistani fishermen who had been held as hostages for over two months. ## method recall that we want to automatically construct a large-scale open-domain qa pretraining dataset that satisfies three criteria: (i) the dataset should cover a wide range of domains for the open-domain qa purpose. (ii) the context passage is semantically relevant to the question and contains direct supporting evidence for answering the question. (iii) the correct answer span in the context passage for answering the question should also be identified for training the reader. this section first discusses how to extract a large amount of statementreference pairs from the wikipedia and then explain how to construct pseudo question-answer-context triplets for pretraining open-domain qa systems. ## conclusion this paper proposes an effective approach for pretraining open-domain qa systems. specifically, we automatically construct three million pseudo question-answer-context triplets from wikipedia that align well with open-domain qa tasks. extensive experiments show that pretraining a widelyused open-domain qa model (dpr) on our constructed data achieves promising performance gain in both retrieval and qa accuracies. future work includes exploring the effectiveness of the constructed data on more open-domain qa models (e.g., realm) and training strategies (e.g., joint optimizing the retriever and reader).