ADAM : Dense Retrieval Distillation with Adaptive Dark Examples (2024)

Abstract: To improve the performance of the dual-encoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works prepare training instances by pairing each query with one positive and a batch of negatives. However, most hard negatives mined by advanced dense retrieval methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose Adam, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with adaptive dark examples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query by strengthening negatives and masking positives in the discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher’s confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.

## introduction information retrieval (ir) that aims to identify relevant passages for a given query is an important topic for both academic and industrial areas, and has powered many downstream tasks such as opendomain qa @xcite and knowledgegrounded conversation @xcite . typically, ir systems usually follow the retrieve-andre-rank paradigm @xcite @xcite where a fast retriever -10.0 -7.5 -5.0 -2.5 0.0 2.5 5.0 7. @xcite over ms-marco. pos and neg mean the distribution of positive and hard negatives respectively. the hard negatives are provided by rocketqav2 @xcite . though effective, we argue that existing dense retrieval distillation methods may not fully exploit the dark knowledge deeply held by the teacher. in knowledge distillation @xcite , the student learns not just the highestscored class from the soft labels provided by the teacher, but also the entire probability distribution over classes, as this contains comprehensive finegrained information referred to as "dark knowledge". however, we empirically find that for existing distillation methods, the soft labels (i.e., the probability distributions over one positive and multiple negatives for a query) given by the teacher are too "sharp", despite they already adopted hard negatives @xcite . as illustrated in figure 1 , we draw the score distributions of the positive and negative pairs using a pre-trained cross-encoder teacher. it can be observed that the scores for most hard negatives are quite low (concentrated in (-7.5, -2.5)) and distributed far from the positives that have high scores. a similar observation is also drawn by @xcite . this phenomenon indicates that even the hard negatives mined by the dense retriever are still too trivial for a well-trained cross-encoder teacher to distinguish, losing most of the utile dark knowledge. to alleviate this issue, we propose adam, a knowledge distillation framework that can better exploit dark knowledge deeply held in the teacher by distillation with adaptive dark examples. our method originated from the intuition that a good soft label for the retriever to learn should be more smooth, which implies that the provided querypassage pairs should diversely distribute from highly-relevant pairs to loosely-relevant pairs from the view of the teacher. to fill the gap between highly-relevant pairs and loosely-relevant pairs existing in current negative sampling methods, we propose two approaches to construct dark examples that all have moderate relevance to the query. the first approach is to make negatives more relevant to the query by strengthening the negatives with the positive passage. the second approach is to make positives less relevant to the query by replacing some randomly selected tokens with mask tokens. considering that the newly created passages have moderate relevance to the query, we believe it is more appropriate to call them dark examples instead of negatives. with these dark examples added, we successfully make the score distribution smoother as shown in figure 3(b) , so that we can transfer more useful dark knowledge from the teacher. moreover, since the soft label for different query-positive-negatives have different "sharpness" which we consider as an indication of how well the dark knowledge has been exploited, we further propose a self-paced distillation strategy that adaptively selects those examples whose soft labels are sharp to conduct our dark-example-based distillation to better transfer the dark knowledge. we conduct experiments on two benchmarks, including ms-marco @xcite and trec deep learning 2019 @xcite . in both benchmarks, the model is required to select the best response from a candidate pool. evaluation results indicate that our method is significantly better than existing models on two benchmarks. to sum up, our contributions is three-fold: ## related works there are two lines of research related to our work: dense retriever and knowledge distillation. dense retriever. to overcome the vocabulary and semantic mismatch problems existing in conventional term-based approaches such as bm25 @xcite , researchers began to build neural retrievers upon pre-trained language models @xcite . in this way, the whole input text can be represented as a dense vector in a lowdimensional space (e.g., 768) and efficient retrieval can be achieved by approximate nearest neighbor search (ann) algorithms such as faiss @xcite . to learn a good dense retriever, various attempts have been made including hard negative mining @xcite @xcite @xcite , retrieval-oriented pre-training @xcite , knowledge distillation @xcite @xcite , etc. we mainly focus on knowledge distillation in this paper. knowledge distillation. knowledge distillation @xcite aims to transfer the knowledge from a powerful teacher model to a student model to help it learn better. to achieve this goal, the student model is provided with the teacher's outputs as the supervision signal that it is enforced to mimic. there are multiple types of supervision signals for the student to learn, including the teacher's output logits @xcite , intermediate representations @xcite , relations of representations @xcite , etc. in the context of dense retrieval distillation, researchers basically adopt the cross-encoder as the teacher and use the teacher's probability distribution over candidate passages as the supervision signal. on this basis, several studies @xcite @xcite explored on-the-fly distillation to jointly optimize the teacher and the student, zeng et al. (2022) and lin et al. (2023) combined knowledge distillation with curriculum strategies to gradually improve the student. different from existing work, we focus on the quality of knowledge held in the teacher's soft label and propose to distill with adaptive dark examples to better transfer the dark knowledge to the student. ## methodology in this section, we first introduce the preliminaries in dense retrieval distillation, then present our dark example augmentation method and adaptive distillation with dynamic data selection. ## experiments we evaluate our method on two public humanannotated real-world benchmarks, namely ms-marco and trec deep learning 2019. ## conclusion in this paper, we propose a knowledge distillation framework that can better transfer the dark knowledge in the cross-encoder with adaptive dark examples to help the dual-encoder achieve better performance. we propose two approaches to create dark examples that are much harder for the crossencoder teacher to distinguish than typical hard negatives to transfer more dark knowledge. further, we propose a self-paced distillation strategy that transfers the knowledge adaptively with highconfidence training instances. experimental results in two widely-used benchmarks verify the effectiveness of our proposed method.