Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2021)

Abstract: Commonly used information retrieval methods such as TF-IDF in open-domain question answering (QA) systems are insufficient to capture deep semantic matching that goes beyond lexical overlaps. Some recent studies consider the retrieval process as maximum inner product search (MIPS) using dense question and paragraph representations, achieving promising results on several information-seeking QA datasets. However, the pretraining of the dense vector representations is highly resource-demanding, e.g. , requires a very large batch size and lots of training steps. In this work, we propose a sample-efficient method to pretrain the paragraph encoder. First, instead of using heuristically created pseudo question-paragraph pairs for pretraining, we use an existing pretrained sequence-to-sequence model to build a strong question generator that creates high-quality pretraining data. Second, we propose a simple progressive pretraining algorithm to ensure the existence of effective negative samples in each batch. Across three open-domain QA datasets, our method consistently outperforms a strong dense retrieval baseline that uses 6 times more computation for training. On two of the datasets, our method achieves more than 4-point absolute improvement in terms of answer exact match.

## introduction with the promise of making the vast amount of information buried in text easily accessible via userfriendly natural language queries, the area of opendomain qa has attracted lots of attention in recent years. existing open-domain qa systems are typically made of two essential components @xcite . a retrieval module first retrieves a compact set of paragraphs from the whole corpus equal contribution. our code is available at https://github.com/ xwhan/proqa.git . over the past few years, much of the progress in open-domain qa has been focusing on improving the reading module of the system, which only needs to process a small number of retrieved paragraphs. specifically, improvements include stronger reading comprehension models @xcite @xcite and paragraph reranking models @xcite that assign more accurate relevance scores to the retrieved paragraphs. however, the performance is still bounded by the retrieval modules, which simply rely on traditional ir methods such as tf-idf or bm25 @xcite . these methods retrieve text solely based on n-gram lexical overlap and can fail on cases when deep semantic matching is required and when there are no common lexicons between the question and the target paragraph. while neural models have proven effective at learning deep semantic matching between text pairs @xcite @xcite , they usually require computing question-dependent paragraph encodings (i.e., the same paragraph will have different representations when considering different questions), which is formidable considering space constraints and retrieval efficiency in practice. more recent studies @xcite @xcite show that such a dilemma can be resolved with large-scale matching-oriented pretraining. these approaches use separate encoders for questions and paragraphs and simply model the matching between the question and paragraph using inner products of the output vectors. thus, these systems only need to encode all paragraphs in a question-agnostic fashion, and the resulted dense corpus index could be fixed and reused for all possible questions. while achieving significant improvements over the bm25 baseline across a set of information-seeking qa datasets, existing pretraining strategies are highly sample-inefficient and typically require a large batch size (up to thousands), such that diverse and effective negative question-paragraph pairs could be included in each batch. when using a small batch size in our experiments, the model ceases to improve after certain updates. given that a 12g gpu can only store around 10 samples with the bert-base architecture at training time, the wider usage of these methods to corpora with different domains (e.g., nonencyclopedic web documents or scientific publications) is hindered given modest gpu hardware. in this work, we propose a simple and sampleefficient method for pretraining dense corpus representations. we achieve stronger open-domain qa performance compared to an existing method @xcite that requires 6 times more computation at training time. besides, our method uses a much smaller batch size and can be implemented with only a small number of gpus, i.e., we use at most 4 titan rtx gpus for all our experiments. in a nutshell, the proposed method first uses a pretrained sequence-to-sequence model to generate high-quality pretraining data instead of relying on heuristics to create pseudo question-paragraph pairs; for the training algorithm, we use clustering techniques to get effective negative samples for each pair and progressively update the clusters. our method's efficacy is further validated through ablation studies, where we replicate existing methods that use the same amount of resources. for the downstream qa experiments, we carefully investigate different finetuning objectives and show the different configurations of the retrieval and span prediction losses have nontrivial effects on the final performance. we hope this analysis could save the efforts on trying out various finetuning strategies of future research that focus on improving the retrieval component of open-domain qa systems. the main contributions of this work include: ## framework we begin by introducing the network architectures used in our retrieval and reading comprehension models. next, we present how to generate highquality question-paragraph pairs for pretraining and how we progressively train the retrieval model with effective negative instances. finally, we show how to finetune the whole system for qa. ## error analysis to investigate the fundamental differences of the dense and sparse retrieval methods in open-domain qa, we conduct an error analysis using both the proposed method and a baseline system that uses tf-idf and bm25 for retrieval. this baseline uses a similar retrieval pipeline as @xcite and is trained with the finetuning objective defined in eq. 2. this sparse retrieval baseline achieves 29.7 em on the official dev set of naturalquestions-open while our method achieves 36.7 em foot_4 . figure 2 shows the venn diagram of the error sets from both systems. our key findings are summarized in the following paragraphs. the difference retrieval paradigms could complement each other. first, according to the error set differences (shown by the white regions in figure 2 ), a considerable portion of the error cases (9.1% of the devt set) of our dense-retrieval system dense errors (63.3%) sparse errors (70.3%) nq development set shared errors (54.2%) figure 2 : the error sets (from the official dev set of naturalquesitons-open) of our dense retrieval method and a baseline system using sparse retrieval methods like tf-idf. we use circles to represent the error sets of both systems. the percentages show the relative size of each set in terms of the whole dev set. both systems are underestimated. as shown in figure 2 , 54.2% of the questions cannot be correctly answered by either system. however, our manual inspection on 50 of the shared error cases suggests that around 30% of these errors are due to annotation issues (14%) or the ambiguous nature of real-user queries (16%). one obvious annotation issue is the incompleteness of the answer labels. example questions include "when did brazil lose to in 2014 world cup?", to which both "germany" and "netherlands" are correct answers. this issue occurs because the annotators of naturalquestions only have a local view of the knowledge source as they are only asked to label the answer span using one document. in terms of the ambiguous questions, many of them are due to constraints unspecified by the question words, such as "what is the population of new york city?" (the time constraint is implicit) or "when did justice league come out in canada?" (needs entity disambiguation). this kind of questions result from the informationseeking nature of the open-domain qa task where the users usually use the minimal number of words for searching and they are not aware of the potential ambiguous factors. to solve this kind of questions, an interactive qa system might be necessary. in the appendix, we show more ambiguous questions in which other kinds of constraints are missing. the task of answering questions without specifying specific domains has been intensively studied since the earlier trec qa competitions @xcite . studies in the early stage @xcite @xcite mostly rely on highly sophisticated pipelines and heterogeneous resources. built on the recent advances in machine reading comprehension, @xcite show that open-domain qa can be simply formulated as a reading comprehension problem with the help of a standard ir component that provides candidate paragraphs for answer extraction. this two-stage formulation is simple yet effective to achieve competitive performance while using wikipedia as the only knowledge resource. following this formulation, a couple of recent studies have proposed to improve the system using stronger reading comprehension models @xcite , more effective learning objectives @xcite @xcite or paragraph reranking models @xcite @xcite . however, the retrieval components in these systems are still based on traditional inverted index methods, which are efficient but might fail when the target paragraph does not have enough lexicon overlap with the question. in contrast to the sparse term-based features used in tf-idf or bm25, dense paragraph vectors learned by deep neural networks @xcite can capture much richer semantics beyond the n-gram term features. to build effective paragraph encoders tailed for the paragraph retrieval in open-domain qa, more recent studies @xcite @xcite propose to pretrain transformer encoders @xcite with objectives that simulate the semantic matching between questions and paragraphs. for instance, @xcite uses the inverse cloze pretraining task to train a bi-encoder model to match a sentence and the paragraph in which the sentence belongs to. these approaches demonstrate promising performance but require a lot of resources for pretraining. the focus of this paper is to reduce the computational requirements of building an effective corpus index. ## conclusion we propose an efficient method for pretraining the dense corpus index which can replace the tradi-tional ir methods in open-domain qa systems. the proposed approach is powered by a better data generation strategy and a simple yet effective data sampling protocol for pretraining. with careful finetuning, we achieve stronger qa performance than orqa that uses much more computational resources. we hope our method could encourage more energy-efficient pretraining methods in this direction such that the dense retrieval paradigm could be more widely used in different domains.