P air D istill: Pairwise Relevance Distillation for Dense Retrieval (2024)

Abstract: Effective information retrieval (IR) from vast datasets relies on advanced techniques to extract relevant information in response to queries. Recent advancements in dense retrieval have showcased remarkable efficacy compared to traditional sparse retrieval methods. To further enhance retrieval performance, knowledge distillation techniques, often leveraging robust cross-encoder rerankers, have been extensively explored. However, existing approaches primarily distill knowledge from pointwise rerankers, which assign absolute relevance scores to documents, thus facing challenges related to inconsistent comparisons. This paper introduces Pairwise Relevance Distillation (PairDistill) to leverage pairwise reranking, offering fine-grained distinctions between similarly relevant documents to enrich the training of dense retrieval models. Our experiments demonstrate that PairDistill outperforms existing methods, achieving new state-of-the-art results across multiple benchmarks. This highlights the potential of PairDistill in advancing dense retrieval techniques effectively. Our source code and trained models are released at https://github.com/MiuLab/PairDistill

## introduction information retrieval (ir) is the process of retrieving relevant information from vast datasets, such as web pages or documents, based on user queries. recently, deep learning methods, notably the dense passage retriever (dpr) @xcite , have attracted attention for their superior performance compared to traditional sparse retrieval techniques like bm25. these methods, often termed dual-encoder models, encode both queries and documents into high-dimensional representations, facilitating efficient similarity computation and retrieval via nearest neighbor search @xcite ). despite the effectiveness of dense retrievers, their modeling capacity is limited. to enhance retrieval performance, knowledge distillation is commonly employed @xcite . typically, knowledge from a robust cross-encoder reranker is distilled to train the dense retriever, achieving state-of-the-art results on retrieval benchmarks @xcite . the efficacy of knowledge distillation largely relies on the performance of the reranker, which serves as the upper bound for the distilled retriever's performance. however, existing studies primarily utilized pointwise rerankers for knowledge distillation, which an absolute relevance score is assigned for each document. such scores are not trivial to compare due to inconsistent baselines. in contrast, pairwise reranking, an advanced technique comparing pairs of documents to assess their relative relevance to a query, has demonstrated superior reranking performance @xcite . by emphasizing relative comparison, pairwise rerankers can distinguish more finely between similarly relevant documents, yielding more precise relevance scores conducive to better distillation. in this paper, we introduce pairwise relevance distillation (pairdistill), a novel method leveraging the fine-grained training signals provided by pairwise rerankers. pairdistill enriches the training of dense retrieval models by distilling knowledge from pairwise comparisons, enabling the model to learn more nuanced distinctions between closely ranked passages. we conduct extensive experiments and demonstrate that pairdis-till outperforms all baseline of similar size on multiple benchmark, as shown in figure 1 . in addition, we show that pairdistill is effective across difference architectures, i.e., colbert @xcite and dpr @xcite , and in a domain adaptation setting. furthermore, we demonstrate the potential of adopting llm rerankers in pairdistill. our contributions are summarized as follows: various studies have delved into enhancing dense retrieval, including negative example mining techniques like rocketqa @xcite , and diverse data augmentation methods such as dragon @xcite . colbert @xcite introduced the lateinteraction mechanism, offering an alternative architecture for dense retrieval. another line of research is pre-training strategies for dense retrieval. approaches like contriever @xcite , cocondenser (gao and callan, 2022), and coco-dr @xcite have proposed contrastive pre-training techniques tailored for retrieval tasks. concurrently, cot-mae @xcite and retromae @xcite have focused on masked auto-encoding for pre-training. as large language models (llms) continue to evolve, their integration into dense retrieval has become increasingly widespread. gtr @xcite utilized llm encoders, showcasing performance gains with increased model size. similarly, promptagator @xcite and inpars @xcite employed llms to generate synthetic query-document pairs, effectively enhancing the training of dense retrievers. building on a similar concept, @xcite extended the approach to multilingual settings, enabling broader applicability. our contribution is orthogonal to these studies as we concentrate on refining training signals for knowledge distillation. this suggests that our approach holds potential for integration with other methods to achieve further improvements. knowledge distillation for dense retrieval enhancing the performance of dense retrievers often involves employing knowledge distillation techniques. izacard and grave (2020) pioneered the distillation of knowledge from the reader to the retriever, resulting in improved performance in open-domain question answering. rankdistill @xcite presented a distillation framework for top-k ranking. following this, rock-etqav2 @xcite and margin-mse @xcite proposed knowledge distillation from cross-encoder rerankers to enhance dense retrievers, while cl-drd @xcite introduced curriculum learning for crossencoder distillation. further advancements include prod @xcite , which proposed a progressive distillation framework, and abel @xcite , introducing an alternating distillation framework with impressive zero-shot performance. these prior work all performed distillation from pointwise rerankers. on the other hand, our method introduces pairwise relevance distillation, leveraging finer-grained training signals from pairwise rerankers. passage reranking passage reranking serves as a pivotal second-stage process following initial large-scale retrieval efforts. various studies have introduced deep reranking models that assess the relevance of query-document pairs by encoding them and predicting relevance scores @xcite . for instance, monot5 @xcite introduced a generation-based method for passage reranking by fine-tuning llms on ms-marco @xcite , distinguishing relevant from irrelevant documents. duot5 @xcite proposed pairwise reranking, simultaneously comparing two documents to significantly enhance reranking performance. tart @xcite fine-tunes llms via multi-task instruction tuning on diverse retriever datasets. another line of research focuses on zero-shot passage reranking with llms, which removes the need for retrieval supervision. upr @xcite pioneered this approach, proposing to rerank passages by estimating the conditional likelihood of generating the query from the passage using llms. huang and chen (2024) enhanced reranking performance further by employing instructiontuned llms. moreover, @xcite and @xcite introduced listwise passage reranking by incorporating prompts with chatgpt. our method combines the superior performance of pairwise reranking with knowledge distillation, which improves retrieval performance significantly and results in state-of-the-art performance on multiple benchmarks. ## background in this section, we detail two key tasks: dense retrieval and passage reranking. following that, we explore knowledge distillation, a widely adopted technique aimed at bolstering the efficacy of dense retrievers. note that we interchangeably use the terms "passage" and "document" in this paper. ## our method: pairdistill in this section, we introduce our proposed method, pairwise relevance distillation (pairdistill). an illustration of the proposed framework is shown in figure 2 . ## experiments our proposed method, pairwise relevance distillation, can be applied to both supervised datasets and zero-shot domain adaptation tasks. in this section, we conduct extensive experiments on passage retrieval tasks to validate and analyze the effectiveness of the proposed method. ## conclusion in this paper, we introduce pairwise relevance distillation (pairdistill), a novel distillation method for dense retrieval that leverages the finergrained training signal provided by the pairwise rerankers. through extensive experiments, we demonstrate that pairdistill achieves state-ofthe-art performance in both in-domain and out-ofdomain evaluation. further analyses show that the proposed method offers consistent improvements across domains and architectures. we hope this study could provide insights into distillation methods for dense retrieval and prompt more advance distillation techniques.