Multiple Teacher Distillation for Robust and Greener Models (2021)

Abstract: The language models nowadays are in the center of natural language processing progress. These models are mostly of significant size. There are successful attempts to reduce them, but at least some of these attempts rely on randomness. We propose a novel distillation procedure leveraging on multiple teachers usage which alleviates random seed dependency and makes the models more robust. We show that this procedure applied to TinyBERT and DistilBERT models improves their worst case results up to 2% while keeping almost the same best-case ones. The latter fact keeps true with a constraint on computational time, which is important to lessen the carbon footprint. In addition, we present the results of an application of the proposed procedure to a computer vision model ResNet, which shows that the statement keeps true in this totally different domain.

## introduction nowadays the language models became a cornerstone in many natural language processing tasks. their results in the benchmarks show new high scores. but with great power sometimes comes huge size, the current models could have dozens of billions of weights, e.g. turingnlg @xcite , to @xcite with 175 billion parameters, and counting. in many cases the resources of computational memory are limited and there is a demand for small solutions. one of such solutions is a distillation of language models. there were presented several approaches for the specified task, among others these are tinybert @xcite and distilbert @xcite . we analyzed these approaches and found that they share an important flaw -the dependency from the random seed used in the distillation process. 0 work done while artur ilichev was at huawei. during the distillation a student model needs to be trained multiple times with different random seeds to achieve better performance, although it is not guaranteed that there will be "winning numbers" in your seed choice. so we concentrated on the worst case scenario and proposed a technique to improve it. considering the computational resources, the improvement could be achieved with the same computational budget, allowing one to diminish the carbon footprint. we propose the novel technique of multi-teacher distillation called to make the mentioned language models more robust to seed selection. we evaluated our method on a computer vision classification model resnet @xcite and make sure that the proposed technique is applicable to a totally different domain. our contribution is as follows: we present (i) a new distillation method and an experimental evaluation of this method for three models, namely (ii) tinybert and (iii) distilbert, where we modified the distillation procedure adding the taskspecific distillation, for three natural language understanding tasks and (iv) resnet for a computer vision task, showing on the one hand that models learned from multiple teachers are consistently better in the worst case and about the same in the best case, and on the other hand, these models are better with a constraint on computational time. this work is structured as follows: in section 3 we describe the distillation process and our modification (in section 3.5); in section 4 we describe the datasets used in the experiments, which are described in section 5. the section 6 concludes the article and discusses the obtained results. ## related work common techniques for model compression and acceleration can be roughly grouped into three groups. pruning parts of large-scale models allows to re-duce the number of weights and accelerate inference. @xcite drop entire layers from pre-trained transformer models, showing that several top-layers can be dropped, maintaining the performance on downstream tasks. @xcite remove all but one attention head, showing that indeed one head might be sufficient at the test time not only for sentence modeling tasks and but for machine translation also. quantization keeps the network structure unchanged, but quantizes network weights to smaller data types, such as int8. quantization can be performed both post training @xcite or during finetuning @xcite . knowledge distillation @xcite trains the more compact models, students, to reproduce the behavior of a larger model, the teacher. bert-pkd @xcite , tinybert @xcite and distil-bert @xcite , distilled versions of the bert model, are commonly used as strong baselines for bert compression. we describe tiny-bert and distilbert models in more detail in the next section. @xcite show that distilling from a better (larger and more accurate) teacher does not always lead to a better student model. we see this result as a motivation for using multiple teachers instead of trying to pick the best one to get a better score. there are several prior studies considering distillation from multiple teachers for computer vision (cv) or natural language processing (nlp) tasks. it can be applied to a multi-task or multidomain setting. for example, @xcite combine the knowledge of teachers trained on different tasks, @xcite train teachers on different features extracted from video frames, @xcite use domain-specific teachers for domain adaptation, and @xcite obtain multilingual machine translation model using teachers pre-trained for each language pair. some authors apply multiple teachers without significantly modifying the distillation pipeline, which is closer to our work. @xcite propose two ways to utilize multiple teachers in the distillation process: to augment the training data with soft labels provided by different models or to switch the teacher models dynamically at the mini-batch level. @xcite show that averaging the prediction of three teachers trained with different learning rates can improve the score on question answering (qa) and natural language inference tasks. yang et al. (2020) adopted twostage distillation procedure and showed improvement in several qa tasks. @xcite show the improvement on several tasks from the glue benchmark. additionally, @xcite propose to add normally distributed random noise to the logits of the teacher model during distillation, claiming that such procedure is a simulation of learning from multiple teachers. besides prediction averaging, multiple teachers can also be utilized to transfer knowledge contained in hidden states or structural relations between examples. @xcite average soft-labels of multiple teachers and propose to transfer relative dissimilarity among intermediate representations using teacher voting to select the best ordering relationships. @xcite combine soft-labels of multiple teachers with learnable weights, distill structural knowledge between data examples, and transfer intermediate layer representations making each teacher responsible for a specific group of layers in the student network. both papers relate to the computer vision field, both use models with different architectures as teachers, and both show that 5 teachers are better than 3 for their methods (in terms of classification accuracy), but not better for the original knowledge distillation. to the best of our knowledge, there is no study dedicated to the isolated investigation of the effect that multiple teachers distillation has on the model quality and robustness and of how this effect change with the number of teachers. importantly, we use models with exactly the same architecture but fine-tuned with different random seeds as teachers. ## model distillation we briefly describe the formulation of original knowledge distillation procedure @xcite , two approaches to distill bert-like models, and one approach to distill the resnet cv model. then we describe how multiple teachers get involved in the process. ## datasets for evaluation we use a subset of tasks from the glue benchmark @xcite for nlp models. we chose the cola task, since the performance drop is the biggest on this dataset for both considered models. the mrpc and sst tasks were chosen in addition to cola task due to the average performance drop. also, the size of mrpc is comparable to cola, while sst-2 is a much bigger corpus. another important feature is that mrpc and sst-2 corpora have test labels publicly available, while the cola dataset has not, so below all the results are provided on the development set from this dataset. a brief description of the datasets is provided in this section. we summarize information about the datasets in for a computer vision model we chose the classic cifar-10 dataset. for the chosen implementation the results, on this dataset are 93.68% and 91.73% for teacher and student respectively foot_0 . ## experiments in this section we describe the experimental setup and then provide the results and their analysis. all the experiments were performed on a single geforce rtx 2080 ti. figure 2 : results for tinybert. shaded: ± one standard deviation.