L a P ra D o R : Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval (2022)

Abstract: In this paper, we propose LaPraDoR, a pretrained dual-tower dense retriever that does not require any supervised data for training. Specifically, we first present Iterative Contrastive Learning (ICoL) that iteratively trains the query and document encoders with a cache mechanism. ICoL not only enlarges the number of negative instances but also keeps representations of cached examples in the same hidden space. We then propose Lexicon-Enhanced Dense Retrieval (LEDR) as a simple yet effective way to enhance dense retrieval with lexical matching. We evaluate LaPraDoR on the recently proposed BEIR benchmark, including 18 datasets of 9 zero-shot text retrieval tasks. Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models, and further analysis reveals the effectiveness of our training strategy and objectives. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.

## introduction dense retrieval uses dense vectors to represent documents and retrieve documents by similarity scores between query vectors and document vectors. different from cross-encoders @xcite @xcite or late-interaction models @xcite , which predict a match score for each query-document pair thus are computationally costly, dense retrieval can be run in milliseconds, with the help of an approximate nearest neighbor (ann) retrieval library, e.g., faiss @xcite . as a drawback, dense retrieval models often require large supervised datasets like ms-marco @xcite (533k training examples) or nq @xcite (133k training examples) for training. unfortunately, @xcite empirically show that models trained on one dataset suffer from an out-of-domain (ood) problem when transferring to another. this hinders the applications of dense retrieval systems. on the other hand, creating a large supervised training dataset for dense retrieval is time-consuming and expensive. for many low-resource languages, there is even no existing supervised dataset for retrieval and it can be extremely difficult to construct one. the recently proposed beir benchmark @xcite highlights the generalization ability of text retrieval systems. the benchmark features a setting where models are trained on a large supervised dataset ms-marco @xcite and then tested on 18 heterogeneous datasets of 9 tasks. in this paper, we propose large-scale pretrained dense zero-shot retriever (laprador), a fully unsupervised pretrained retriever for zero-shot text retrieval. while existing dense retrievers need large supervised data and struggle to compete with a lexical matching approach like bm25 @xcite for zero-shot retrieval, we take a different approach by complementing lexical matching with semantic matching. without any supervised data, laprador outperforms all dense retrievers on beir. laprador achieves state-of-the-art performance on beir with a further fine-tuning, outperforming re-ranking, despite being 22.5× and 42× faster on gpu and cpu, respectively. training laprador faces two challenges: (1) training efficiency. for large-scale pretraining, training efficiency can be important. in contrastive learning, more negative instances often lead to better performance @xcite @xcite . however, traditional inbatch negative sampling is bottlenecked by limited gpu memory. to alleviate this problem, we propose iterative contrastive learning (icol), which iteratively trains the query and document encoders with a cache mechanism. compared to existing solutions moco @xcite and xmoco @xcite , icol does not introduce extra encoders and can solve the mismatching between representation spaces, thus demonstrating superior performance. to summarize, our contribution is three-fold: (1) we train laprador, an all-around unsupervised pretrained dense retriever that achieves state-ofthe-art performance on the beir benchmark. (2) we propose iterative contrastive learning (icol) for training a retrieval model effectively. (3) we propose lexicon-enhanced dense retrieval as an efficient way for combining bm25 with a dense retriever, compared to the widely-used re-ranking paradigm. pretraining for retrieval @xcite first propose to pretrain a bi-encoder retriever with an inverse cloze task (ict), which constructs a training pair by randomly selecting a sentence from a passage as the query and leaving the rest as the document. @xcite propose two pretraining tasks for wikipedia and attempt to combine them with ict and masked language modeling (mlm). @xcite pretrain a retriever and a reader together for end-to-end question answering (qa). very recently, dpr-paq @xcite highlight the importance of domain matching by using both synthetic and crawled qa data to pretrain and then fine-tune the model on downstream datasets for dialogue retrieval. condenser @xcite is a new transformer variant for mlm pretraining. it exploits an information bottleneck to facilitate learning for information aggregation. on top of that, cocondenser @xcite adds an unsupervised corpus-level contrastive loss to warm up the passage embedding space. different from these works, laprador is the first pretrained retriever that does not require fine-tuning on a downstream dataset and can perform zero-shot retrieval. ## conclusion and future work in this paper, we introduce laprador, an unsupervised pretrained dense retriever that achieves state-of-the-art performance on the zero-shot text retrieval benchmark beir. we propose iterative contrastive learning (icol) for efficiently training laprador and lexicon-enhanced dense retrieval (ledr) to combine lexical matching with laprador. our experiments verify the effectiveness of both icol and ledr, shedding light on a new paradigm for unsupervised text retrieval. for future work, we plan to extend unsupervised laprador to multilingual and multi-modal retrieval.