Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective (2023)

Abstract: Knowledge distillation (KD) is a highly promising method for mitigating the computational problems of pre-trained language models (PLMs). Among various KD approaches, Intermediate Layer Distillation (ILD) has been a de facto standard KD method with its performance efficacy in the NLP field. In this paper, we find that existing ILD methods are prone to overfitting to training datasets, although these methods transfer more information than the original KD. Next, we present the simple observations to mitigate the overfitting of ILD: distilling only the last Transformer layer and conducting ILD on supplementary tasks. Based on our two findings, we propose a simple yet effective consistency-regularized ILD (CR-ILD), which prevents the student model from overfitting the training dataset. Substantial experiments on distilling BERT on the GLUE benchmark and several synthetic datasets demonstrate that our proposed ILD method outperforms other KD techniques. Our code is available at https://github.com/jongwooko/CR-ILD .

## introduction recent advances in nlp have shown that using plms such as bert @xcite and roberta @xcite on downstream tasks is effective. although these models achieve stateof-the-art performances in various domains, the promising results of plms require numerous computation and memory costs. deploying such large models on resource-constrained devices such as mobile and wearable devices is impractical. it is thus crucial to train computationally efficient smallsized networks with similar performance to that of large models. kd is promising model compression technique where knowledge is transferred from a large and high-performing model (teacher) to a smaller model (student). kd has been shown to be reliable in reducing the number of parameters and computations while achieving competitive results on downstream tasks. recently, kd has attracted more attention in the nlp field, especially due to large plms. however, it is clear that the original kd @xcite is not performing well in terms of maintaining the performance of compressed plms and that it needs to have additional auxiliary training objectives @xcite . ild methods @xcite , which encourage the student model to extract knowledge from the transformer layers of the teacher network, have demonstrated efficacy in improving student model performance and have become a de facto standard in kd. despite of success of ild methods, many research have been proposed to design layer mapping functions @xcite or new training objective @xcite to transfer the teacher's knowledge better. these ild methods transfer more knowledge to the student model from the intermediate transformer layers of the teacher model. however, we find that the use of ild in fine-tuning may induce performance degradation in some cases. as shown in figure 1 , while existing ild methods such as tinybert @xcite and bert-emd @xcite ) work well on standard glue benchmark @xcite , we observe that these methods have performance degradation compared to original kd on ill-conditioned datasets such as those with few-samples and label noise. because few-sample @xcite or heterogeneous datasets @xcite can be easily found in real-world datasets, the existing ild methods, which show performance reduction in figure 1 , are hard to use in real-world applications. to mitigate such performance degradation, we identify the main problem as that intermediate transformer knowledge can incur overfitting on the training dataset of the student model. we further @xcite ) work well on the standard glue benchmark @xcite , we observe that the existing ild methods are problematic under few-samples training datasets or the presence of label noise. however, our proposed method shows robustly higher performance than the original kd for all datasets. we use bert small @xcite as the student model and bert base @xcite as the teacher model. the detailed descriptions for dataset are in appendix b. ## related works model compression of lms. transformer encodes contextual information for input tokens @xcite . in recent years, from the success of transformer, transformerbased models such as gpt @xcite , bert @xcite , and t5 @xcite have become a new state of the arts, driving out recurrent or convolutional networks on various language tasks. however, the promising results of these models are accompanied by numerous parameters, which necessitate a high computation and memory cost for inference. existing compression techniques can be categorized as low-rank matrix factorization @xcite , quantization @xcite , and kd @xcite . knowledge distillation for lms. kd is one of the most well-known neural model compression techniques. the goal of kd is to enable the student model with fewer parameters to achieve similar performance to that of the teacher model with a large number of parameters. in the recent few years, a wide range of different methods have been developed that apply data augmentation @xcite , adversarial training @xcite , and loss terms re-weighting @xcite to reduce the performance gap between the teacher and the student. in another line in the nlp field, ild-based methods have exhibited higher effectiveness over original kd @xcite methods for compression plms. sun et al. (2019) proposed the bert-pkd to transfer representations of the [cls] token of the teacher model. jiao et al. (2020) proposed tinybert, which performed transformer distillation in both pre-training and fine-tuning. wang et al. (2020) distilled the self-attention module of the last transformer layer of the teacher. li et al. (2020) leveraged earth mover's distance (emd) to determine the optimal layer mapping between the teacher and student networks. park et al. (2021) presented new kd objectives that transfer contextual knowledge via two types of relationships. ## observations: two things everyone should know to mitigate overfitting in this section, we identify that overfitting is the main problem for performance degradation while conducting ild in fine-tuning. this overfitting problem can occur even in the standard glue benchmark. moreover, the ill-conditioned dataset, where overfitting problems can occur more easily, induces a larger performance reduction. furthermore, we investigate that this overfitting problem is able to be reduced by (1) distilling the last transformer layer and (2) conducting ild on supplemen- mha + ir mha ir 85 86 87 88 89 90 (a) mrpc mha + ir mha ir 55 60 65 70 (b) rte mha + ir mha ir 88 89 90 91 92 (c) sst-2 mha + ir mha ir 88 89 90 w/o ild last uniform emd (d) qnli mha + ir mha ir 88 89 90 91 (a) mrpc mha + ir mha ir 60 62 65 68 70 w/o ild last uniform emd (b) rte figure 3: performance distribution box plot across 20 random trials for mrpc and rte when bert small @xcite is used as the student model. well-pretrained student models have more consistent performance to the choice of layer mappings. ## experiments to verify the effectiveness of cr-ild, we compare the performance of ours with previous distillation methods on the standard glue and ill-conditioned glue benchmark. the descriptions for experimental setup are in appendix b and c. ## conclusion this paper introduces a better use of ild that transfer knowledge by using outputs of transformer layers of the teacher and the student models. we found that existing ild methods may lead the student model to overfit the training dataset of target tasks and degenerate the generalizability. furthermore, we investigated that conducting the ild (1) only for the last transformer layer and (2) on supplementary tasks can alleviate the overfitting problems. based on our observations, we proposed consistency-regularized ild that incurs smoother functions and enhance the generalizability of the student models. our proposed method effectively distills the knowledge of teacher models by (1) encouraging the flat minima of function from consistency regularization between original embeddings and mixup embeddings of the student models and