Learning to Retrieve Passages without Supervision (2022)

Abstract: Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. In this work we ask whether this dependence on labeled data can be reduced via unsupervised pretraining that is geared towards ODQA. We show this is in fact possible, via a novel pretraining scheme designed for retrieval. Our “recurring span retrieval” approach uses recurring spans across passages in a document to create pseudo examples for contrastive learning. Our pretraining scheme directly controls for term overlap across pseudo queries and relevant passages, thus allowing to model both lexical and semantic relations between them. The resulting model, named Spider, performs surprisingly well without any labeled training examples on a wide range of ODQA datasets. Specifically, it significantly outperforms all other pretrained baselines in a zero-shot setting, and is competitive with BM25, a strong sparse baseline. Moreover, a hybrid retriever over Spider and BM25 improves over both, and is often competitive with DPR models, which are trained on tens of thousands of examples. Last, notable gains are observed when using Spider as an initialization for supervised training.

## introduction state-of-the-art models for retrieval in open domain question answering are based on learning dense text representations @xcite @xcite . however, such models rely on large datasets of question-passage pairs for training. these datasets are expensive and sometimes even impractical to collect (e.g., for new languages or domains), and models trained on them often fail to generalize to new question distributions @xcite . the above difficulty motivates the development of retrieval models that do not rely on large annotated training sets, but are instead trained only on unlabeled text. indeed, self-supervision for retrieval has gained considerable attention recently @xcite @xcite . however, when applied in a "zero-shot" manner, such models are still outperformed by sparse retrievers like bm25 @xcite and by supervised models (see @xcite . moreover, models like realm @xcite and mss @xcite ) that train a retriever and a reader jointly (i.e. in an end-to-end fashion), treating retrieval as a latent variable, outperform contrastive models like ict @xcite , but are much more computationally-intensive. in this work we introduce spider (span-based unsupervised dense retriever), a dense model pretrained in a contrastive fashion from self-god at sinai granted aaron the priesthood for himself and his male descendants, and he became the first high priest of the israelites. aaron … the books of exodus, leviticus and numbers maintain that aaron received from god a monopoly over the priesthood for himself and his male descendants … aaron … during the journey in the wilderness, aaron was not always prominent or active … lennon said that much of the song's lyrics and content came from his wife yoko ono, and in 2017 the process to give yoko co-writing credit imagine (john lennon song) … several poems from yoko ono's 1964 book grapefruit inspired lennon to write the lyrics for "imagine"… imagine (john lennon song) … imagine was written during the let it be session. lennon finished composing "imagine" one morning … figure 2 : an example of our pretraining approach: given a document d (e.g. the article "aaron" in wikipedia), we take two passages that contain a recurring span s. one of them is transformed into a short query (left) q using a random window surrounding s, in which s is either kept (top) or removed (bottom). the second passage is then considered the target for retrieval p + , while a random passage from d that does not contain s is considered the negative p -(right). each batch is comprised of multiple such examples, and the pretraining task is to select the passage p + i for each query q i (solid line) from the passages of all examples (in-batch negatives; dashed lines). spider is based on a novel self-supervised scheme: recurring span retrieval. we leverage recurring spans in different passages of the same document (e.g. "yoko ono" in figure 2 ) to create pseudo examples for self-supervised contrastive learning, where one of the passages containing the span is transformed into a short query that (distantly) resembles a natural question, and the other is the target for retrieval. additionally, we randomly choose whether to keep or remove the recurring span from the query to explicitly model cases where there is substantial overlap between a question and its target passage, as well as cases where such overlap is small. we evaluate spider on several odqa benchmarks. spider narrows the gap between unsupervised dense retrievers and dpr on all benchmarks (figure 1 , we further demonstrate the utility of spider as an off-the-shelf retriever via cross-dataset evaluation (i.e., when supervised models are tested against datasets which they were not trained on), a setting that often challenges dense retrievers @xcite . in this setting, spider is competitive with supervised dense retrievers trained on an abundance of training examples. last, spider significantly outperforms other pretrained models when used as an initialization towards dpr training, and also shows strong crossdataset generalization. for example, spider finetuned on triviaqa is, to the best of our knowledge, the first dense model to outperform bm25 on the challenging entityquestions dataset @xcite . taken together, our results demonstrate the potential of pretraining for reducing the reliance of odqa models on training data. ## background in open-domain question answering (odqa), the goal is to find the answer to a given question over a large corpus, e.g. wikipedia @xcite @xcite . this task has gained considerable attention following recent advancement in machine reading comprehension, where models reached human parity in extracting an answer from a paragraph given a question @xcite . due to the high cost of applying such reading comprehension models, or readers, over the entire corpus, state-of-the-art systems for odqa first apply an efficient retriever -either sparse @xcite or dense @xcite -in order to reduce the search space of the reader. recently, dense retrieval models have shown promising results on odqa, even outperforming strong sparse methods that operate on the lexical-level, e.g. bm25. specifically, the dominant approach employs a dual-encoder architecture, where documents and questions are mapped to a shared continuous space such that proximity in that space represents the relevance between pairs of documents and questions. formally, let @xmath0, ..., p m } be a corpus of passages. each passage @xmath1, such that e p (p) ∈ r d . similarly, the question encoder e q is defined such that the representation of a question q is given by e q (q) ∈ r d . then, the relevance of a passage p for q is given by: given a question q, the retriever finds the top-k candidates with respect to s(q, •), i.e. top-k @xmath2, p). in order to perform this operation efficiently at test time, a maximum-inner product search (mips) index @xcite is built over the encoded passages {e p (p 1 ), ..., e p (p m )}. while considerable work has been devoted to create pretraining schemes for dense retrieval @xcite ; inter alia), it generally assumed access to large training datasets after pretraining. in contrast, we seek to improve dense retrieval in the challenging unsupervised setting. our contribution towards this goal is twofold. first, we construct a self-supervised pretraining method based on recurring spans across passages in a document to emulate the training process of dual-encoders for dense retrieval. our pretraining is simpler and cheaper in terms of compute than end-to-end models like realm @xcite and mss @xcite . second, we demonstrate that a simple combination of bm25 with our models leads to a strong hybrid retriever that rivals the performance of models trained with tens of thousands of examples. ## our model: spider we now describe our approach for pretraining dense retrievers, which is based on a new selfsupervised task (section 3.1). our pretraining is based on the notion of recurring spans @xcite within a document: given two paragraphs with the same recurring span, we construct a query from one of the paragraphs, while the other is taken as the target for retrieval (figure 2 ). other paragraphs in the document that do not contain the recurring span are used as negative examples. we train a model from this self-supervision in a contrastive fashion. since sparse lexical methods are known to complement dense retrieval @xcite , we also incorporate a simple hybrid retriever (combining bm25 and spider) in our experiments (section 3.2). ## experimental setup to evaluate how different retrievers work on different settings and given different amounts of supervision, we simulate various scenarios by using existing datasets, with an emphasis on the unsupervised setting. ## results our experiments show that spider significantly improves performance in the challenging unsupervised retrieval setting, even outperforming strong supervised models in many cases. thus, it enables the use of such retrievers when no examples are available. when used for supervised dpr training, we observe significant improvements over the baselines as well. we perform ablation studies that demonstrate the importance of our pretraining design choices. ## related work pretraining for dense retrieval has recently gained considerable attention, following the success of self-supervised models in many tasks @xcite @xcite . while most works focus on fine-tuning such retrievers on large datasets after pretraining @xcite @xcite @xcite , we attempt to bridge the gap between unsupervised dense models and strong sparse (e.g. bm25; @xcite or supervised dense baselines (e.g. dpr; @xcite ). a concurrent work by @xcite presented dpr-paq, which shows strong results on nq after pretraining. however, their approach utilizes paq @xcite , a dataset which was generated using models trained on nq, and is therefore not unsupervised. leveraging recurring spans for self-supervised pretraining has previously been considered for numerous tasks, e.g. coreference resolution and coreferential reasoning @xcite @xcite and question answering @xcite @xcite . @xcite for qa. while we focus in this work on dual-encoder architectures, other architectures for dense retrieval have been introduced recently. @xcite showed that replacing a single representation with multiple vectors per document enjoys favorable theoretical and empirical properties. @xcite introduced late-interaction models, where contextualized representations of query and document tokens are first computed, and a cheap interaction step that models their fine-grained relevance is then applied. phrase-based retrieval @xcite eliminates the need for a reader during inference, as it directly retrieves the answer span given a query. @xcite demonstrated strong end-to-end odqa results with this approach, and @xcite showed that it is also effective for passage retrieval. our pretraining scheme can be seamlessly used for those architectures as well. ## conclusion in this work, we explore learning dense retrievers from unlabeled data. our results demonstrate that existing models struggle in this setup. we introduce a new pretraining scheme for dual-encoders that dramatically improves performance, reaching good results without any labeled examples. our results suggest that careful design of a pretraining task is important for learning unsupervised models that are effective retrievers for odqa. a baselines: further details bm25 @xcite a sparse bag-of-words model that extends tf-idf (i.e. reward rare terms that appear in both q and p) by accounting for document length and term frequency saturation. bert @xcite was pretrained on two self-supervised tasks: masked language modeling (mlm) and next sentence prediction (nsp). we evaluate bert only in the supervised setting, namely as a backbone for fine-tuning, similar to dpr. ict @xcite a dual-encoder model which was pretrained on the inverse cloze task. given a batch of passages, ict masks a sentence from each passage, and trains to predict what is the source passage for each sentence. ict encourages lexical matching by keeping the sentence in the original passage with low probability. note that unlike our approach, ict is trained to produce representations to corrupted passages. in addition, we condenser & cocondenser @xcite condenser is an architecture that aims to produce dense sequence-level (i.e. sentences and passages) representations via a variant of the mlm pretraining task. specifically, to predict a masked token x t , they condition the prediction on two representations: (1) a representation of x t from an earlier layer in the encoder, and (2) a dense sequence-level representation of the [cls] token at the last layer of the network. cocondenser adds a "corpus-aware" loss alongside mlm to create better embeddings by sampling two sub-spans from each sequence and train in a contrastive fashion. mss @xcite an unsupervised model in which a dense retriever and a reader are trained jointly end-to-end. first, salient spans (e.g. entities) are identified using a ner model. then, some of them are masked. the training objective is to predict these missing spans while using retrieved documents as evidence. due to the latent nature of the retrieval process in this model, its training is substantially more expensive than contrastive learning. in addition, it requires frequent updates of the encoded evidence corpus. contriever @xcite a contemporary work. contriever is an unsupervised dense model trained in a contrastive fashion, using random cropping to generate two views of a given input. dpr @xcite a supervised model for odqa based on dual-encoders and trained in a contrastive fashion (see section 2). all dpr models considered in the paper are initialized with a bert-base encoder, and trained on full datasets: dpr-single models are trained on a single dataset, and are also referred to as dpr-x, where x is the name of the dataset. dpr-multi was trained onnq, triviaqa, wq and trec. for dpr-nq and dpr-multi, we use the checkpoints released by the authors. we re-train the other dpr-single models (which were not made public) using the same hyper-parameters as @xcite . we do not train a dpr model on enti-tyqs. the models we trained are consistent with the results of @xcite , except for dpr-squad, where we did not manage to reproduce the original results.