One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022)

Abstract: Knowledge distillation is an effective method to transfer knowledge from a large pre-trained teacher model to a compacted student model. However, in previous studies, the distilled student models are still large and remain impractical in highly speed-sensitive systems (e.g., an IR system). In this study, we aim to distill a deep pre-trained model into an extremely compacted shallow model like CNN. Specifically, we propose a novel one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning. Moreover, we leverage large-scale unlabeled data to improve the performance of students. Empirical studies on three sentiment classification tasks demonstrate that our approach achieves better results with much fewer parameters (0.9%-18%) and extremely high speedup ratios (100X-1000X).

## introduction sentiment classification is a task of classifying a text into sentimental orientation categories, such as positive and negative, and this task plays an important role in natural language processing (nlp) and benefits many real applications @xcite @xcite . the past few years have witnessed the prevailing of deep pre-trained models on sentiment classification @xcite @xcite . however, despite their significant improvements over non-pre-trained models like cnn and lstm, their need for a large number of computing resources and relatively long inference time becomes a major bottleneck for realworld applications. to solve this problem, knowledge distillation, which transfers knowledge from a large model (the teacher) to a smaller model (the student), is gaining popularity for reducing the computing and time costs. however, existing distilled models' parameters are still too large for some low-end devices and speed-sensitive applications. for instance, sentiment classification is usually an essential component of an information retrieval (ir) system @xcite , in which users are highly speedsensitive to the responding speed. thus, improving the inference speed of the pre-trained model on sentiment classification becomes a critical element of applying the pre-trained sentiment classification model to ir applications. motivated by the above, in this paper, we aim to propose a novel distillation technique that can distill a hugeparameterized pre-trained model like bert @xcite into a minimal-parameterized non-pre-trained model like cnn or lstm. in principle, compared with traditional shallow models, deep pre-trained models have two major advantages. first, most pre-trained models are architectured in the manner of ensemble learning. in the literature, ensemble learning has been proven to be effective in performance boosting. for instance, bert combines multiple transformer layers under an ensemble architecture. meanwhile, each transformer layer contains the attention ensemble by using multiple selfattention heads @xcite . second, a pre-trained model highly benefits from the knowledge contained in unlabeled data. for instance, the bert-base model is pre-trained using unlabeled data containing 3.3 billion words from wikipedia and bookscorpus. at least, the large scale of unlabeled data enables the pretrained model to handle unknown words in a downstream task more easily. for instance, given a sentence "everything is awesome!", suppose that the word "awesome" is not observed in the training data in a sentiment classification task. thus, non-pre-trained shallow models like cnn-based or lstm-based classifiers cannot easily determine the sentiment orientation of "awesome". on the contrary, however, pre-trained models like bert can infer the meaning of "awesome", which is close to the observed word "excellent". in this paper, inspired by the above, we propose a novel ensemble knowledge distillation approach by leveraging both ensemble learning and unlabeled data. specifically, first, we use multiple shallow models, together with their ensemble model, as student models during distillation in order to take advantage of ensemble learning. thanks to many previous studies on multi-view learning on sentiment classification, multiple student models could be easily obtained by using various kinds of multiple views, such as multiple types of embeddings @xcite and multiple languages @xcite . second, we leverage large-scale unlabeled sentiment classification corpora during distillation. different from most previous studies on one-teacher and one-student knowledge distillation, we propose a one-teacher and multiple-student ensemble distillation framework, which is illustrated in figure 1 . empirical studies on three sentiment classification tasks demonstrate that our approach outperforms the pre-trained teacher models with much fewer parameters (0.9%-18%) and extremely high speedup ratios (100x-1000x). the remainder of this paper is organized as follows. section 2 provides an overview of the related studies on sentiment analysis and knowledge distillation. section 3 explains the details of our approach. section 4 introduces the experimental settings and results. section 5 states the conclusion and the future work. ## methodology in this section, we introduce the details of our approach. ## experiments in this section, we systematically evaluate our oneteacher and multiple-student knowledge distillation approach in three types of sentiment classification tasks, i.e., supervised sentiment classification, zero-shot sentiment classification, and crosslingual sentiment classification. ## conclusion in this study, we propose a novel approach of knowledge distillation, namely one-teacher and multiple-student knowledge distillation, in sentiment classification. our approach is capable of compacting a large model into a minimal ensem-ble model with both ensemble learning and unlabeled data. empirical studies on three sentiment classification tasks demonstrate that the distilled model performs even better than the teacher model with much fewer parameters and a much better operational performance on cpu. in our future work, we aim to improve our approach by carefully selecting a suitable number of unlabeled samples instead of using all of them. in addition, we would like to apply our approach to other nlp tasks besides sentiment classification.