Learning Dense Representations of Phrases at Scale (2021)

Abstract: Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underperform retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot filling tasks, showing the promise of utilizing DensePhrases as a dense knowledge base for downstream tasks.

## introduction open-domain question answering (qa) aims to provide answers to natural-language questions using a large text corpus @xcite @xcite . while a dominating approach is a two-stage retriever-reader approach @xcite @xcite , we focus on a recent new paradigm solely based on phrase retrieval @xcite . phrase retrieval highlights the use of phrase representations and finds answers purely based on the similarity search in the vector space of phrases. 2 without relying on an expensive reader model for processing text passages, it has demonstrated great runtime efficiency at inference time. despite great promise, it remains a formidable challenge to build vector representations for every single phrase in a large corpus. since phrase representations are decomposed from question representations, they are inherently less expressive than cross-attention models @xcite . moreover, the approach requires retrieving answers correctly out of billions of phrases (e.g., 6 × 10 10 phrases in english wikipedia), making the scale of the learning problem difficult. consequently, existing approaches heavily rely on sparse representations for locating relevant documents and paragraphs while still falling behind retriever-reader models @xcite . in this work, we investigate whether we can build fully dense phrase representations at scale for opendomain qa. first, we aim to learn strong phrase representations from the supervision of reading comprehension tasks. we propose to use data augmentation and knowledge distillation to learn better phrase representations within a single passage. we then adopt negative sampling strategies such as inbatch negatives @xcite , to better discriminate the phrases at a larger scale. here, we present a novel method called pre-batch negatives, which leverages preceding mini-batches as negative examples to compensate the need of large-batch training. lastly, we present a query-side fine-tuning strategy that dras- finally, we envision that densephrases acts as a neural interface for retrieving phrase-level knowledge from a large text corpus. to showcase this possibility, we demonstrate that we can directly use densephrases for fact extraction, without rebuilding the phrase storage. with only fine-tuning the question encoder on a small number of subjectrelation-object triples, we achieve state-of-the-art performance on two slot filling tasks @xcite , using less than 5% of the training data. ## background we first formulate the task of open-domain question answering for a set of k documents @xmath0, . . . , d k }. we follow the recent work @xcite and treat all of english wikipedia as d, hence k ≈ 5 × 10 6 . however, most approaches-including ours-are generic and could be applied to other collections of documents. the task aims to provide an answer â for the input question q based on d. in this work, we focus on the extractive qa setting, where each answer is a segment of text, or a phrase, that can be found in d. denote the set of phrases in d as s(d) and each phrase s @xmath1, . . . , w end(k) in its document d doc(k) . in practice, we consider all the phrases up to @xmath2. an extractive qa system returns a phrase ŝ = argmax @xmath3, q) where f is a scoring function. the system finally maps ŝ to an answer string â: text(ŝ) = â and the evaluation is typically done by comparing the predicted answer â with a gold answer a * . although we focus on the extractive qa setting, recent works propose to use a generative model as the reader @xcite , or learn a closed-book qa model @xcite , which directly predicts answers without using an external knowledge source. the extractive setting provides two advantages: first, the model directly locates the source of the answer, which is more interpretable, and second, phraselevel knowledge retrieval can be uniquely adapted to other nlp tasks as we show in §7.3. ## learning phrase representations in this section, we start by learning dense phrase representations from the supervision of reading comprehension tasks, i.e., a single passage p contains an answer a * to a question q. our goal is to learn strong dense representations of phrases for @xmath4, which can be retrieved by a dense representation of the question and serve as a direct answer ( §4.1). then, we introduce two different negative sampling methods ( §4.2, §4.3), which encourage the phrase representations to be better discriminated at the full wikipedia scale. see figure 1 for an overview of densephrases. ## analysis ablation of phrase representations. model m share split qg distill em denspi bb. 70.2 sb. 68.5 bl. 73.6 dense bb. 70.2 phrases bb. 71.9 sb. 73.2 sb. 76.3 sb. 78.3 effect of batch negatives. we further evaluate the effectiveness of various negative sampling methods introduced in §4.2 and §4.3. since it is computationally expensive to test each setting at the full wikipedia scale, we use a smaller text corpus d small of all the gold passages in the development sets of natural questions, for the ablation study. empirically, we find that results are generally well correlated when we gradually increase the size of |d|. as shown in ## related work learning effective dense representations of words is a long-standing goal in nlp @xcite @xcite @xcite . beyond words, dense representations of many different granularities of text such as sentences @xcite or documents @xcite have been explored. while dense phrase representations have been also studied for statistical machine translation @xcite or syntactic parsing @xcite , our work focuses on learning dense phrase representations for qa and any other knowledge-intensive tasks where phrases can be easily retrieved by performing mips. this type of dense retrieval has been also studied for sentence and passage retrieval @xcite (see @xcite for recent advances in dense retrieval). while densephrases is explicitly designed to retrieve phrases that can be used as an answer to given queries, retrieving phrases also naturally entails retrieving larger units of text, provided the datastore maintains the mapping between each phrase and the sentence and passage in which it occurs. ## conclusion in this study, we show that we can learn dense representations of phrases at the wikipedia scale, which are readily retrievable for open-domain qa and other knowledge-intensive nlp tasks. we learn both phrase and question encoders from the supervision of reading comprehension tasks and introduce two batch-negative techniques to better discriminate phrases at scale. we also introduce query-side fine-tuning that adapts our model to different types of queries. we achieve strong performance on five popular open-domain qa datasets, while reducing the storage footprint and improving latency significantly. we also achieve strong performance on two slot filling datasets using only a small number of training examples, showing the possibility of utilizing our densephrases as a knowledge base.