Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval (2023)

Abstract: Neural ‘dense’ retrieval models are state of the art for many datasets, however these models often exhibit limited domain transfer ability. Existing approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models. We present \texttt{ABEL} , a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings. Our technique follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever. By iterating this loop, the two components mutually enhance one another’s performance. Experimental results demonstrate that our unsupervised \texttt{ABEL} model outperforms both leading supervised and unsupervised retrievers on the BEIR benchmark. Meanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training. By either fine-tuning \texttt{ABEL} on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-art results.

## introduction remarkable progress has been achieved in neural information retrieval through the adoption of the dual-encoder paradigm @xcite , which enables efficient search over vast collections of passages by factorising the model such that the encoding of queries and passages are decoupled, and calculating the query-passage similarity using dot product. however, the efficacy of training dualencoders heavily relies on the quality of labelled data, and these models struggle to maintain competitive performance on retrieval tasks where dedicated training data is scarce @xcite . various approaches have been proposed to enhance dense retrievers @xcite in zero-shot settings while maintaining the factorised dual-encoder structure, such as pre-training models on web-scale corpus @xcite and learning from cross-encoders through distillation @xcite . other alternatives seek to trade efficiency for performance by using complex model architectures, such as fine-grained token interaction for more expressive representations @xcite and scaling up the model size for better model capacity @xcite . another line of work trains customised dense retrievers on target domains through query generation @xcite . this training paradigm is generally slow and expensive, as it employs large language models to synthesise a substantial number of high-quality queries. in this paper, we present abel, an alternating bootstrapping training framework for unsupervised dense retrieval. our method alternates the distillation process between a dense retriever and a reranker by switching their roles as teachers and students in iterations. on the one hand, the dense retriever allows for efficient retrieval due to its factorised encoding, accompanied by a compromised model performance. on the other hand, a reranker has no factorisation constraint, allowing for more fine-grained and accurate scoring, but at the cost of intractable searches. our work aims to take advantage of both schools by equipping the dense retriever with accurate scoring by the reranker while maintaining search efficiency. specifically, i) the more powerful but slower reranker is used to assist in the training of the less capable but more efficient retriever; ii) the dense retriever is employed to improve the performance of the reranker by providing refined training signals in later iterations. this alternating learning process is repeated to iteratively enhance both modules. compared with conventional bootstrapping approaches @xcite , wherein the well-trained model itself is used to discover additional solutions for subsequent training iterations, our method considers one model (i.e., teacher) as the training data generator to supervise another model (i.e., student), and their roles as teacher and student are switched in every next step. this mechanism naturally creates a mutual-learning paradigm to enable iterative bidirectional knowledge flow between the retriever and the reranker, in contrast to the typical singlestep and unidirectional distillation where a student learns from a fixed teacher @xcite . through extensive experiments on various datasets, we observe that abel demonstrates outstanding performance in zero-shot settings by only using the basic bm25 model as an initiation. additionally, both the retriever and reranker components involved in our approach can be progressively enhanced through the bootstrapping learning process, with the converged model outperforming more than ten prominent supervised retrievers and achieving state-of-the-art performance. meanwhile, abel is efficient in its training process by exclusively employing sentences from raw texts as queries, rather than generating queries from large language models. the use of the simple dual-encoder architecture further contributes to its efficient operation. in summary, our contributions are: ## preliminary given a short text as a query, the passage-retrieval task aims at retrieving a set of passages that in-clude the necessary information. from a collection of passages as the corpus, @xmath0, p 2 , • • • , p n }, a retriever d fetches top-k passages p k d (q) = {p 1 , p 2 , • • • , p k } from p that are most relevant to a specific query q. optionally, a reranker r is also employed to fine-grain the relevance scores for these retrieved passages. ## alternating distillation we propose an unsupervised alternating distillation approach that iteratively boosts the ability of a retriever and a reranker, as depicted in fig. 1 . alg. 1 outlines the proposed method with three major steps. our approach starts with warming up a retriever by imitating bm25. subsequently, a recursive learning paradigm consisting of two steps is conducted: (1) training a reranker based on the labels extracted from the retriever by the last iteration; ## conclusion in this paper, we introduce abel, an unsupervised training framework that iteratively improves both retrievers and rerankers. our method enhances a dense retriever and a cross-encoder reranker in a closed learning loop, by alternating their roles as teachers and students. the empirical results on various tasks demonstrate that this simple technique can significantly improve the capability of dense retrievers without relying on any human-annotated data, surpassing a wide range of competitive sparse and supervised dense retrievers. we believe that abel is a generic framework that could be easily combined with other retrieval augmenting techniques, and benefits a range of downstream tasks.