MTA 4 DPR : Multi-Teaching-Assistants Based Iterative Knowledge Distillation for Dense Passage Retrieval (2024)

Abstract: Although Dense Passage Retrieval (DPR) models have achieved significantly enhanced performance, their widespread application is still hindered by the demanding inference efficiency and high deployment costs. Knowledge distillation is an efficient method to compress models, which transfers knowledge from strong teacher models to weak student models. Previous studies have proved the effectiveness of knowledge distillation in DPR. However, there often remains a significant performance gap between the teacher and the distilled student. To narrow this performance gap, we propose MTA4DPR, a Multi-Teaching-Assistants based iterative knowledge distillation method for Dense Passage Retrieval, which transfers knowledge from the teacher to the student with the help of multiple assistants in an iterative manner; with each iteration, the student learns from more performant assistants and more difficult data. The experimental results show that our 66M student model achieves the state-of-the-art performance among models with same parameters on multiple datasets, and is very competitive when compared with larger, even LLM-based, DPR models.

## introduction although plm/llm-based dense passage retrieval (dpr) models @xcite have superior performance, those models' inference efficiency and deployment costs are still cumbering their wide applications. to obtain an efficient and effective dpr model, researchers are paying more attention to knowledge distillation. previous studies @xcite @xcite have proved the effectiveness of * corresponding author knowledge distillation in dpr. however, the performance gap between the teacher and the distilled student often remains significant, especially when the teacher is a very good one. in this paper, we hypothesize that incorporating assistants into knowledge distillation can help improve students' performance, just as teaching assistants in universities can assist students in learning course content. in addition, inspired by curriculum learning @xcite , we also believe that multiple iterations can further narrow the gap between the teacher and the student since the latter is capable of learning from more challenging data and more effective assistants as the iterations go on. therefore, we introduce mta4dpr, a multi-teaching-assistants based iterative distillation method. specifically, mta4dpr transfers knowledge from the teacher to the student with the help of multiple assistants iteratively. for each iteration, we first use off-the-shelf teacher/assistant dpr models to generate datasets for training and evaluation. then, we use a fusion module to generate a series of fused assistants. after that, we train the student to learn from the teacher with the help of the best assistant selected among all fused and original assistants by our selection module, as illustrated in figure 1 . at the end of each iteration, we evaluate the student's performance and replace the worst-performing assistant with it if it outperforms any existing assistants. what's more, we also incorporate data that the student predicted incorrectly in the previous iteration into the newly constructed dataset, by which the difficulty of each iteration's dataset is increased. in this way, as the training iterates, the student can learn from more performant assistants and more difficult data. the experimental results on ms marco, trec dl 2019 and 2020 and natural questions show the effectiveness of our method. our 66m student model achieves the state-of-the-art performance among models with same parameters on multiple datasets, and is competitive when compared with larger, even llm-based, dpr models. to summarize, our main contributions are: ## conclusion in this paper, we propose mta4dpr, an iterative multi-assistant distillation method for dpr. it distills the student with the help of the teaching assistants in an iterative manner, with each iteration creating more difficult datasets and more performant assistants. the experimental results on ms marco, trec dl 2019 and 2020 and natural questions show the effectiveness of our method. our 66m dpr model can achieve the state-ofthe-art performance among models with same parameters on multiple datasets and is very competitive when compared with larger, even llmbased, dpr models. mta4dpr confirms that the iterative distillation with multiple assistants can improve the distillation performance. since it is orthogonal to existing distillation methods, other distillation pipelines can be combined with mta4dpr to further improve their performance. in addition, mta4dpr is not constrained by model structures and tasks, and can be broadly applicable other fields than dpr, including text classification, question answering and text summarization, etc. 9