A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models (2023)

Abstract: Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance. Previous studies have shown that DWT can be effective in the vision domain and natural language processing (NLP) pre-training stage. Specifically, DWT shows promise in practical scenarios, such as enhancing new generation or larger models using pre-trained yet older or smaller models and lacking a resource budget. However, the optimal conditions for using DWT have yet to be fully investigated in NLP pre-training. Therefore, this study examines three key factors to optimize DWT, distinct from those used in the vision domain or traditional knowledge distillation. These factors are:(i) the impact of teacher model quality on DWT effectiveness, (ii) guidelines for adjusting the weighting value for DWT loss, and (iii) the impact of parameter remapping as a student model initialization technique for DWT.

## introduction recently, distillation from weak teacher (dwt) @xcite , a reversed knowledge distillation (kd) technique, has gained attention from researchers. unlike the traditional kd @xcite @xcite , which compresses a pre-trained model by transferring its knowledge to a smaller model, dwt distills knowledge from a smaller (or weaker) pre-trained model to a larger model to improve its quality during training. dwt is well-suited for practical real-world scenarios such as: for the above cases, dwt can utilize the existing pre-trained models and improve the learning of new (larger) models. studies @xcite have shown that dwt allows a larger student model to leverage the knowledge of a weaker, smaller pre-trained teacher model in both the computer vision and nlp pre-training stages. while previous research by @xcite has demonstrated the potential of dwt in the nlp domain, it did not fully explore the key aspects of dwt such as the impact of teacher model quality and a student model initialization technique for dwt. however, to truly unlock the potential of dwt for real-world applications, we need a deeper understanding of the key conditions and factors that contribute to its performance. for example, the effect of dwt might differ from traditional kd and potentially harm the student model, depending on the quality of its teacher. therefore, this work conducts in-depth studies and uncovers crucial insights to optimize dwt in the pre-training stage of nlp as follows: we believe that these observations provide useful guidelines to better utilize dwt techniques for real-world applications. ## distillation from weak teacher in this section, we formulate the distillation from weak teacher (dwt) strategy, which involves training the target (student) model using both the teacher's predictions (soft labels) and the ground truth (hard labels). task given a classification task with c classes, for each training instance x and its corresponding ground truth label y, the ground truth distribution over the labels is denoted as q(c|x) (abbreviated as q(c)) where for each label c in the set {1...c}, q(y) = 1 and q(c) = 0 for all c not equal to y. ## experiment we conducted a study to analyze the efficacy of the dwt method and present key observations for optimizing its impact in three core elements: (i) the quality of the teacher model, (ii) the degree of soft knowledge transfer, and (iii) the initialization type (parameter remapping) of the student model. training setting we use a default loss weight ratio of α h : α @xmath0:1 for the hard loss and soft loss during distillation. the learning rate is set to 5e -4, and the models are trained for 20 epochs with the application of quantization, linear warmup (5%), the adam optimizer (kingma and ba, 2014), 16 batch sizes per gpu, and 8 a100 gpus per run. in the pre-training stage, we utilize a reduced dataset of 30 million sentences generated by uniformly selecting one sentence out of every four sentences from the original dataset, which consists of a combination of bookcorpus @xcite and wikipedia (foundation). the performance of the distilled models is evaluated on the dev sets of the glue benchmark @xcite , comprising nine sentence-level classification tasks (please see the supplementary file for more details.). ## conclusion distillation from weak teacher (dwt) is a technique that improves the performance of a larger student model by transferring knowledge from a weaker, smaller teacher model. despite the potential of dwt, the optimal conditions to use dwt have yet to be fully investigated in nlp pre-training. this study investigated three crucial factors for optimizing dwt in nlp pre-training, which differ from those in vision or traditional kd. these factors include the impact of teacher model quality, the use of parameter remapping as an initialization technique for dwt, and guidelines for adjusting the weighting value of the dwt loss.