Dense Passage Retrieval for Open-Domain Question Answering (2020)

Abstract: Open-domain question answering relies on efficient passage retrieval to select candidate contexts, where traditional sparse vector space models, such as TF-IDF or BM25, are the de facto method. In this work, we show that retrieval can be practically implemented using dense representations alone, where embeddings are learned from a small number of questions and passages by a simple dual-encoder framework. When evaluated on a wide range of open-domain QA datasets, our dense retriever outperforms a strong Lucene-BM25 system greatly by 9%-19% absolute in terms of top-20 passage retrieval accuracy, and helps our end-to-end QA system establish new state-of-the-art on multiple open-domain QA benchmarks.

## introduction open-domain question answering (qa) @xcite is a task that answers factoid questions using a large collection of documents. while early qa systems are often complicated and consist of multiple components @xcite ; @xcite , inter alia), the advances of reading comprehension models suggest a much simplified two-stage framework: (1) a context retriever first selects a small subset of passages where some of them contain the answer to the question, and then (2) a machine reader can thoroughly examine the retrieved contexts and identify the correct answer @xcite . although reducing open-domain qa to machine reading is a very reasonable strategy, a huge performance degradation is often observed in practice 2 , indicating the needs of improving retrieval. retrieval in open-domain qa is usually implemented using tf-idf or bm25 @xcite , which matches keywords efficiently with an inverted index and can be seen as representing the question and context in highdimensional, sparse vectors (with weighting). conversely, the dense, latent semantic encoding is complementary to sparse representations by design. for example, synonyms or paraphrases that consist of completely different tokens may still be mapped to vectors close to each other. consider the question "who is the bad guy in lord of the rings?", which can be answered from the context "sala baker is best known for portraying the villain sauron in the lord of the rings trilogy." a term-based system would have difficulty retrieving such a context, while a dense retrieval system would be able to better match "bad guy" with "villain" and fetch the correct context. dense encodings are also learnable by adjusting the embedding functions, which provides additional flexibility to have a task-specific representation. with special in-memory data structures and indexing schemes, retrieval can be done efficiently using maximum inner product search (mips) algorithms (e.g., @xcite ; @xcite ). however, it is generally believed that learning a good dense vector representation needs a large number of labeled pairs of question and contexts. dense retrieval methods have thus never be shown to outperform tf-idf/bm25 for opendomain qa before orqa @xcite , which proposes a sophisticated inverse cloze task (ict) objective, predicting the blocks that contain the masked sentence, for additional pretraining. the question encoder and the reader model are then finetuned using pairs of questions and answers jointly. although orqa successfully demonstrates that dense retrieval can outperform bm25, setting new state-of-the-art results on multiple open-domain qa datasets, it also suffers from two weaknesses. first, ict pretraining is computationally intensive and it is not completely clear that regular sentences are good surrogates of questions in the objective function. second, because the context encoder is not fine-tuned using pairs of questions and answers, the corresponding representations could be suboptimal. in this paper, we address the question: can we train a better dense embedding model using only pairs of questions and passages (or answers), without additional pretraining? by leveraging the now standard bert pretrained model @xcite and a dual-encoder architecture @xcite , we focus on developing the right training scheme using a relatively small number of question and passage pairs. through a series of careful ablation studies, our final solution is surprisingly simple: the embedding is optimized for maximizing inner products of the question and relevant passage vectors, with an objective comparing all pairs of questions and passages in a batch. our dense passage retriever (dpr) is exceptionally strong. it not only outperforms bm25 by a large margin (65.2% vs. 42.9% in top-5 accuracy), but also results in a substantial improvement on the end-to-end qa accuracy compared to orqa (41.5% vs. 33.3%) in the open natural questions setting @xcite . our contributions are twofold. first, we demonstrate that with the proper training setup, simply fine-tuning the question and passage encoders on existing question-passage pairs is sufficient to greatly outperform bm25. our empirical results also suggest that additional pretraining may not be needed. second, we verify that, in the context of open-domain question answering, a higher retrieval precision indeed translates to a higher end-to-end qa accuracy. by applying a modern reader model to the top retrieved passages, we achieve comparable or better results on multiple qa datasets in the open-retrieval setting, compared to several, much complicated systems. ## background the problem of open-domain qa studied in this paper can be described as follows. given a factoid question, such as "who first voiced meg on family guy?" or "where was the 8th dalai lama born?", a system is required to answer it using a large corpus of diversified topics. more specifically, we assume the extractive qa setting, in which the answer is restricted to a span appearing in one or more passages in the corpus. assume that our collection contains d documents, d 1 , d 2 , • • • , d d . we first split each of the documents into text passages of equal lengths as the basic retrieval units foot_0 and get m total passages in our corpus @xmath0, p 2 , . . . , p m }, where each passage p i can be viewed as a sequence of tokens w for a fixed k, a retriever can be evaluated in isolation on top-k retrieval accuracy, which is the fraction of questions for which c f contains a span that answers the question. ## dense passage retriever (dpr) we focus our research in this work on improving the retrieval component in open-domain qa. given a collection of m text passages, the goal of our dense passage retriever (dpr) is to index all the passages in a low-dimensional and continuous space, such that it can retrieve efficiently the top k passages relevant to the input question for the reader at run-time. note that m can be very large (e.g., 21 million passages in our experiments, described in section 4.1) and k is usually small, such as 20-100. ## experimental setup in this section, we describe the data we used for experiments and the basic setup. ## experiments: passage retrieval in this section, we evaluate the retrieval performance of our dense passage retriever (dpr), along with analysis on how its output differs from the dpr model used in our main experiments is trained using the in-batch negative setting (section 3.2) with a batch size of 128 and one additional bm25 negative passage per question. we trained the question and passage encoders for up to 40 epochs for large datasets (nq, triviaqa, squad) and 100 epochs for small datasets (trec, wq), with a learning rate of 10 -5 using adam, linear scheduling with warm-up and dropout rate 0.1. while it is good to have the flexibility to adapt the retriever to each dataset, it would also be desirable to obtain a single retriever that works well across the board. to this end, we train a multidataset encoder by combining training data from all datasets excluding squad. 8 in addition to dpr, we also present the results of bm25, the traditional retrieval method 9 and bm25+dpr, using a linear combination of their scores as the new ranking function. specifically, we obtain two initial sets of top-2000 passages based on bm25 and dpr, respectively, and rerank the union of them using bm25(q,p) + λ • sim(q, p) as the ranking function. we used λ = 1.1 based on the retrieval accuracy in the development set. ## experiments: question answering in this section, we experiment with how different passage retrievers affect the final qa accuracy. ## conclusion in this work, we demonstrated that dense retrieval can outperform and potentially replace the traditional sparse retrieval component in open-domain question answering. while a simple dual-encoder approach can be made to work surprisingly well, we showed that there are some critical ingredients to training a dense retriever successfully. moreover, our empirical analysis and ablation studies indicate that more complex model frameworks or similarity functions do not necessarily provide additional values. as a result of improved retrieval performance, we obtained new state-of-the-art results on multiple open-domain question answering benchmarks.