PILE : Pairwise Iterative Logits Ensemble for Multi-Teacher Labeled Distillation (2022)

Abstract: Pre-trained language models have become a crucial part of ranking systems and achieved very impressive effects recently. To maintain high performance while keeping efficient computations, knowledge distillation is widely used. In this paper, we focus on two key questions in knowledge distillation for ranking models: 1) how to ensemble knowledge from multi-teacher; 2) how to utilize the label information of data in the distillation process. We propose a unified algorithm called Pairwise Iterative Logits Ensemble (PILE) to tackle these two questions simultaneously. PILE ensembles multi-teacher logits supervised by label information in an iterative way and achieved competitive performance in both offline and online experiments. The proposed method has been deployed in a real-world commercial search system.

## introduction search engines have been an infrastructure in the information age to satisfy people's needs for querying about information. in modern search engines, multi-stage pipelines are usually employed where ranking is usually known as the very final stage. it takes as input the shortlisted candidates of relevant documents (i.e. web pages) retrieved from previous stages, and concentrates on sorting @xcite based on the degree of match between the latent semantics of documents and the search intent of the user's query. with the flourishing of pre-trained language models @xcite @xcite , bert-based models have achieved state-of-the-art performance in a broad range of downstream tasks and text ranking is no exception. pre-trained rankers based on bert show impressive performance in ranking tasks @xcite @xcite . despite the state-of-the-art performance pretrained models yield in laboratories, it is hardly possible to apply them directly in real-world search engines. their large numbers of parameters go against computational efficiency while the online environment is strictly restricted in resources. therefore, before deploying a pre-trained model online, one necessary procedure is to reduce computational costs. knowledge distillation is one of the most commonly used methods to reduce the model size @xcite and accelerate the computation process. in a standard workflow of distillation, a large model (i.e. teacher) is pretrained and finetuned well in advance, and a small model (i.e. student) imitates the teacher model's behaviors. the knowledge learned by the teacher model is then transferred to the student model. one of the risk factors hindering the improvement of the student model is that the knowledge acquired by a single teacher may be insufficient and biased. a straightforward solution is using an ensemble of multiple teachers for knowledge transfer. the ensemble process takes the predictions of multiple teachers into account and provides comprehensive guidance that helps to improve student performance @xcite . however, these teachers sometimes conflict with each other, and the heuristic of treating them equally by taking the mean of their predictions ignores the fact that they vary in confidence and correctness, thus often leading to suboptimal performance @xcite . besides, the valuable label information is ignored. hence, how to ensemble knowledge from multiteacher and how to utilize the label information are two key questions during the distillation process. in this work, we introduce a unified algorithm called pairwise iterative logits ensemble (pile) to tackle these two questions simultane-ously. the key idea of pile is to assign a higher weight to teachers that produce more consistent soft targets with the golden labels. the resulting soft targets not only retain the generalization information transferred from the teacher models but also are integrated with the label information annotated by human experts. we conduct both offline and online experiments and the results validate the effectiveness of pile. the main contributions of this paper can be summarized as follows: text ranking the goal of text ranking is to generate an ordered list of texts in response to a query. conventional learning-to-rank (ltr) techniques @xcite are widely used for text ranking, which plays an important role in a wide range of applications, like search engines and recommender systems. ltr techniques can be roughly categorized into three types: pointwise approach @xcite , pairwise approach @xcite , and listwise approach @xcite . the former two are more widely used in practice as they are easier to optimize. recently, deep learning approaches have been widely adopted in ranking and bert-based ranking models achieve state-of-the-art ranking effectiveness. for example, @xcite use bert-large @xcite as the backbone and feed the concatenation of query and passage text to estimate the relevant scores for passage re-ranking. @xcite formulate the ranking problem as a pointwise and pairwise classification problem and tackle them with two bert models in a multi-stage ranking pipeline. @xcite aggregate sentence-level information to estimate the relevance of the documents and transfer the learned model to capture cross-domain notions of relevance. however, the performance improvement comes at the cost of efficiency, which limits their realworld application. there are several ways to maintain high performance while keeping efficient computations for bert-based models, such as knowledge distillation @xcite , weight sharing @xcite , pruning @xcite , and quantization @xcite . in this paper, we focus on knowledge distillation which has proven a promising way to compress large models while maintaining performance. knowledge distillation the idea of knowledge distillation was first introduced by @xcite to train small and fast models to mimic cumbersome and complex models, without much loss in performance. @xcite developed this idea further by minimizing the difference between their soft target distribution. with the rise of the pre-training and fine-tuning paradigm, various work has later extended this idea to large-scale pretrained models and shown impressive results on multiple nlp tasks @xcite @xcite with a significant gain in training efficiency. @xcite conducted knowledge transfer during the pre-training phase, also known as a task-agnostic way. @xcite proposed an approach to transfer knowledge between intermediate layers in the fine-tuning stage. @xcite additionally uses attentionbased distillation and hidden states-based distillation for students to imitate teachers' behaviors in intermediate layers. @xcite introduced self-attention relation-based transfer and teacher assistants @xcite to further improve the performance of students. ensemble knowledge distillation there is also some other work exploring the issues of multiteacher distillation. for example, @xcite adaptively ensemble knowledge distillation to find a better optimizing direction for the student network. @xcite designed a co-finetuning framework to jointly finetune multiple teachers for better collaborative knowledge distillation. @xcite explored the influence of teacher model adoption which is promising for improving student performance. different from the above work, we investigate the problem of ensemble knowledge distillation in ranking tasks and use the golden label to supervise the ensemble process. ## conclusion in this work, we propose an easy-to-implement approach to multi-teacher distillation for large-scale ranking models. our algorithm ensembles multiteacher logits supervised by human-annotated labels in an iterative way. we conduct the offline experiments as well as deploy our methods in an online commercial search system which demonstrates its superiority.