Meta- KD : A Meta Knowledge Distillation Framework for Language Model Compression across Domains (2021)

Abstract: Pre-trained language models have been applied to various NLP tasks with considerable performance gains. However, the large model sizes, together with the long inference time, limit the deployment of such models in real-time applications. One line of model compression approaches considers knowledge distillation to distill large teacher models into small student models. Most of these studies focus on single-domain only, which ignores the transferable knowledge from other domains. We notice that training a teacher with transferable knowledge digested across domains can achieve better generalization capability to help knowledge distillation. Hence we propose a Meta-Knowledge Distillation (Meta-KD) framework to build a meta-teacher model that captures transferable knowledge across domains and passes such knowledge to students. Specifically, we explicitly force the meta-teacher to capture transferable knowledge at both instance-level and feature-level from multiple domains, and then propose a meta-distillation algorithm to learn single-domain student models with guidance from the meta-teacher. Experiments on public multi-domain NLP tasks show the effectiveness and superiority of the proposed Meta-KD framework. Further, we also demonstrate the capability of Meta-KD in the settings where the training data is scarce.

## introduction pre-trained language models (plm) such as bert @xcite and xlnet @xcite have achieved significant success with the two-stage "pre-training and fine-tuning" process. despite the performance gain achieved in various nlp tasks, the large number of model parameters a physics student may learn physics equations better with a powerful all-purpose teacher. and the long inference time have become the bottleneck for plms to be deployed in real-time applications, especially on mobile devices @xcite @xcite . thus, there are increasing needs for plms to reduce the model size and the computational overhead while keeping the prediction accuracy. knowledge distillation (kd) @xcite is one of the promising ways to distill the knowledge from a large "teacher" model to a small "student" model. recent studies show that kd can be applied to compress plms with acceptable performance loss @xcite @xcite @xcite . however, those methods mainly focus on single-domain kd. hence, student models can only learn from their in-domain teachers, paying little attention to acquiring knowledge from other domains. it has been shown that it is beneficial to consider cross-domain information for kd, by either training a teacher using cross-domain data or multiple teachers from multiple domains @xcite @xcite . consider an academic scenario in figure 1 . a typical way for a physics student to learn physics equations is to directly learn from his/her physics teacher. if we have a math teacher to teach him/her basic knowledge of equations, the student can obtain a better understanding of physics equations. this "knowledge transfer" technique in kd has been proved efficient only when two domains are close to each other @xcite . in reality, however, it is highly risky as teachers of other domains may pass non-transferable knowledge to the student model, which is irrelevant to the current domain and hence harms the overall performance @xcite . besides, current studies find multi-task fine-tuning of bert does not necessarily yield better performance across all the tasks @xcite . to address these issues, we leverage the idea of meta-learning to capture transferable knowledge across domains, as recent studies have shown that meta-learning can improve the model generalization ability across domains @xcite @xcite . we further notice that meta-knowledge is also helpful for cross-domain kd. re-consider the example in figure 1 . if we have an "all-purpose teacher" (i.e., the meta-teacher) who has the knowledge of both physics principles and mathematical equations (i.e., the general knowledge of the two courses), the student may learn physics equations better with the teacher, compared to the other two cases. hence, it is necessary to train an "all-purpose teacher" model for domain-specific student models to learn. in this paper, we propose the meta-knowledge distillation (meta-kd) framework, which facilities cross-domain kd. generally speaking, meta-kd consists of two parts, meta-teacher learning and meta-distillation. different from the k-way n-shot problems addressed in traditional metalearning @xcite , we propose to train a "meta-learner" as the meta-teacher, which learns the transferable knowledge across domains so that it can fit new domains easily. the meta-teacher is jointly trained with multi-domain datasets to acquire the instance-level and feature-level metaknowledge. for each domain, the student model learns to solve the task over a domain-specific dataset with guidance from the meta-teacher. to improve the student's distillation ability, the metadistillation module minimizes the distillation loss from both intermediate layers, output layers, and transferable knowledge, combined with domainexpertise weighting techniques. to verify the effectiveness of meta-kd, we conduct extensive experiments on two nlp tasks across multiple domains, namely natural language inference @xcite and sentiment analysis @xcite . experimental results show the effectiveness and superiority of the proposed meta-kd framework. moreover, we find our method performs well especially when i) the in-domain dataset is very small or ii) there is no in-domain dataset during the training of the metateacher. in summary, the contributions of this study can be concluded as follows: the rest of this paper is summarized as follows. section 2 describes the related work. the detailed techniques of the meta-kd framework are presented in section 3. the experiments are reported in section 4. finally, we conclude our work and discuss the future work in section 5. foot_0 ## related work our study is close to the following three lines of studies, introduced below. ## the meta-kd framework in this section, we formally introduce the meta-kd framework. we begin with a brief overview of meta-kd. after that, the techniques are elaborated. ## experiments in this section, we conduct extensive experiments to evaluate the meta-kd framework on two popular text mining tasks across domains. ## conclusion and future work in this work, we propose the meta-kd framework which consists of meta-teacher learning and meta distillation to distill plms across domains. experiments on two widely-adopted public multidomain datasets show that meta-kd can train a meta-teacher to digest knowledge across domains to help better teach in-domain students. quantitative evaluations confirm the effectiveness of meta-kd and also show the capability of meta-kd in the settings where the training data is scarce i.e. there is no or few in-domain data. in the future, we will examine the generalization capability of meta-kd in other application scenarios and apply other meta-learning techniques to kd for plms.