Dynamic Knowledge Distillation for Pre-trained Language Models (2021)

Abstract: Knowledge distillation (KD) has been proved effective for compressing large-scale pre-trained language models. However, existing methods conduct KD statically, e.g., the student model aligns its output distribution to that of a selected teacher model on the pre-defined training dataset. In this paper, we explore whether a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency, regarding the student performance and learning efficiency. We explore the dynamical adjustments on three aspects: teacher model adoption, data selection, and KD objective adaptation. Experimental results show that (1) proper selection of teacher model can boost the performance of student model; (2) conducting KD with 10% informative instances achieves comparable performance while greatly accelerates the training; (3) the student performance can be boosted by adjusting the supervision contribution of different alignment objective. We find dynamic knowledge distillation is promising and provide discussions on potential future directions towards more efficient KD methods.

## introduction knowledge distillation (kd) @xcite aims to transfer the knowledge from a large teacher model to a small student model. it has been widely used @xcite @xcite to compress large-scale pre-trained language models (plms) like bert @xcite and roberta @xcite in recent years. by knowledge distillation, we can obtain a much smaller model with comparable performance, while greatly reduce the memory usage and accelerate the model inference. although simple and effective, existing methods usually conduct the kd learning procedure statically, e.g., the student model aligns its output probability distribution to that of a selected teacher model on the entire pre-defined corpus. in other words, the following three aspects of kd are specified in advance and remain unchanged during the learning procedure: (1) the teacher model to learn from (learning target); (2) the data used to query the teacher (learning material); (3) the objective functions and the corresponding weights (learning method). however, as the student competency evolves during the training stage, it is particularly unreasonable to pre-define these learning settings and keep them unchanged. conducting kd statically may lead to (1) unqualified learning from a too large teacher, (2) repetitive learning on instances that the student has mastered, and (3) sub-optimal learning on alignments that are unnecessary. this motivates us to explore an interesting problem: whether a dynamic kd framework considering the student competency evolution during training can bring benefits, regarding the student performance and learning efficiency? in this paper, we propose a dynamic knowledge distillation (dynamic kd) framework, which attempts to empower the student to adjust the learning procedure according to its competency. specifically, inspired by the success of active learning @xcite , we take the prediction uncertainty, e.g., the entropy of the predicted classification probability distribution, as a proxy of the student competency. we strive to answer the following research questions: (rq1) which teacher is proper to learn as the student evolves? (rq2) which data are actually useful for student models in the whole kd stage? (rq3) does the optimal learning objective change in the kd process? in particular, we first explore the impact of the teacher size to dynamic knowledge distillation. second, we explore whether dynamically choosing instances that the student is uncertain for kd can lead to a better performance and training efficiency trade-off. third, we explore whether the dynamic adjustment of the supervision from alignment of prediction probability distributions and the alignment of intermediate representations in the whole kd stage can improve the performance. ## background: knowledge distillation given a student model s and a teacher model t , knowledge distillation aims to train the student model by aligning the outputs of the student model to that of the teacher. for example, @xcite utilize the teacher model outputs as soft targets for the student to learn. we denote s(x) and t (x) as the output logit vector of the student and the teacher for input x, respectively. the kd can be conducted by minimizing the kullback-leibler (kl) divergence distance between the student and teacher prediction: ## dynamic knowledge distillation in this section, we introduce the dynamic knowledge distillation. the core idea behind is to empower the student to adjust the learning procedure according to its current state, and we investigate the three aspects illustrated in figure 1 . ## discussions after the preliminary explorations on the three aspects of dynamic kd, we observe that it is promising for improving the efficiency and the distilled student performance. here we provide potential directions for further investigations. ## related work our work relates to recent explorations on applying kd for compressing the plms and active learning. knowledge distillation for plms knowledge distillation @xcite aims to transfer the dark knowledge from a large teacher model to a compact student model, which has been proved effective for obtaining compact variants of plms. those methods can be divided into general distillation @xcite @xcite and task-specific distillation @xcite @xcite @xcite @xcite . the former conducts kd on the general text corpus while the latter trains the student model on the task-specific datasets. in this paper, we focus on the latter one as it is more widely adopted in practice. compared to existing static kd work, we are the first to explore the idea of dynamic kd, making it more flexible, efficient and effective. active learning @xcite , where a learning system is allowed to choose the data from which it learns from, for achieving better performance with fewer labeled data. traditional selection strategies include uncertainty-based methods @xcite , which select the instances that model is most uncertain about, query-by-committee @xcite , which select instances with highest disagreements between a set of classifiers, and methods based on decision theory @xcite . in this paper, inspired by the success of active learning, we introduce dynamic kd that utilizes the different strategies like prediction entropy as a proxy of student competency to adaptively adjust the different aspects of kd. our explorations show that the uncertainty-based strategies are effective for improving the efficiency and performance of kd. ## conclusion in this paper, we introduce dynamic knowledge distillation, and conduct exploratory experiments regarding teacher model adoption, data selection and the supervision adjustment. our experimental results demonstrate that the dynamical adjustments on the three aspects according to the student uncertainty is promising for improving the student performance and learning efficiency. we provide discussions on the potential directions worth exploring in the future, and hope this work can motivate studies towards more environmental-friendly knowledge distillation methods.