Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation (2023)

Abstract: It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledge transfer. In order to enhance the guidance of the teacher training process, we introduce the concept of distillation influence to determine the impact of distillation from each training sample on the student’s generalization ability. In this paper, we propose Learning Good Teacher Matters (LGTM), an efficient training technique for incorporating distillation influence into the teacher’s learning process. By prioritizing samples that are likely to enhance the student’s generalization ability, our LGTM outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark.

## introduction the recent success of natural language processing (nlp) is driven by the adoption of large-scale pretrained language models @xcite @xcite . as these models are scaling up in depth and width, they become increasingly computational and storage intensive, making deployment difficult. to address this issue, different methods have been proposed for crafting efficient models with minimal loss in performance, such as weight pruning @xcite , network quantization @xcite , and knowledge distillation (kd) @xcite @xcite . among these methods, kd has proven to be effective in various nlp applications @xcite and is widely adopted. the idea of kd involves asking a lightweight stu- * equal contribution. † work done while being at amazon. 1 our code is public available at https://github.com/ twinkle0331/lgtm dent model to mimic the output of a large teacher model so as to transfer the knowledge. ideally, a teacher with better performance should be able to transfer more knowledge to the student. therefore in most knowledge distillation algorithms, the teacher network is trained to maximize its own performance. however, multiple studies @xcite have observed that a teacher with higher performance does not necessarily lead to a betterperforming student, and may even cause a performance degradation. @xcite has attributed this inefficiency in knowledge distillation to challenges during optimization. as the model capacity gap between the student and the teacher increases, the optimization process becomes more likely to be trapped in local optima @xcite . one way to address the performance degradation in kd is to update the teacher via feedback from student's performance, also known as learning to teach (l2t) @xcite . l2t allows the teacher model to adjust its "teaching agenda" by interacting with the student. among the l2t algorithms, online distillation @xcite @xcite trains the student and teacher concurrently and enforces similarity between their outputs on the training set. however, online distillation focuses on transferring the knowledge of the teacher to the student on training set without explicitly considering how well the student will perform on validation set. on the other hand, meta distillation @xcite takes the generalization ability of student on the held-out validation set into account, and guides the teacher's learning process to maximize the generalization ability. however, the optimization objective of meta distillation may result in a degraded teacher model, as it only receives supervision from the student model. it is well-known that humans are more efficient learners when their teachers provide guidance on the level of attention they should devote to certain problems based on their current knowledge. similarly, it is possible that a student model could be trained more effectively if it receives such guidance from a teacher. to accomplish this goal, the teacher should prioritize samples that are likely to enhance the student's generalization ability during training, thus allowing the student to perform better on the held-out validation set. in this work, inspired by the concept of influence function @xcite , we propose distillation influence to estimate how distilling on each training sample impacts the student's performance on the validation set. in addition, we are able to interpret existing l2t methods from the perspective of influence function, so as to gain a deeper understanding of their limitations. the optimization process of existing l2t methods are often impacted by outliers, because they assign all training samples in the mini-batch the same weight. hence, we propose our l2t framework, learning good teacher matters (lgtm), which assigns loss weights of the training samples based on their distillation influence. extensive experiments have shown that lgtm enables more effective knowledge transfer. in summary, our contributions are as follows: given a labeled training dataset d train , a batch of b r training samples and their corresponding labels are referred to as z @xmath0, y r ), where r indicates training. we index each sample in the training batch z r as z r i . similarly for validation dataset d val , we define the batch of samples as z @xmath1, y e ), where e indicates validation. in addition, we introduce the notation of the jacobian matrix in the context of working with the chain rule and gradient. in particular, let f : r @xmath2, and let @xmath3. we use the notation ∂f ∂@xmath4, which has dimensions k × n. for simplicity, we annotate ∂f ∂v as ∇ v . we use x ⊺ to denote the transpose of the matrix x. ## revisiting learning to teach in this paper, we focus on task-specific distillation given pre-trained language models. under this setting, the teacher model is already pre-trained in an unsupervised manner and the student model is either derived from part of the teacher model or pre-trained in an unsupervised manner as well. ## experiments in this section, we first describe our experiment setup including datasets and baselines in sec. 5.1. then we compare our proposed lgtm to meta distillation to gain some basic understanding of how to incorporate the student's feedback in sec. 5.2. to further verify the effectiveness of our method, in sec. 5.3 we compare to 10 widely adopted knowledge distillation baselines and show consistently better results. then we demonstrate how distillation influence works in sec. 5.4, followed by ablation studies of lgtm in sec. 5.5. ## conclusion in this paper, we first revisit several learning to teach paradigms in knowledge distillation. then we propose distillation influence to determine how distilling from each training sample impacts the student's generalization ability. by visualizing how the distillation influence of each sample changes during training, we can see that a simple re-weighting using distillation influence is able to help student training, e.g., reduce overfitting. built on top of distillation influence, we propose our learning to teach framework, lgtm, that consistently outperforms existing knowledge distillation methods on text classification tasks in the glue benchmark.