Towards Zero-Shot Knowledge Distillation for Natural Language Processing (2021)

Abstract: Knowledge distillation (KD) is a common knowledge transfer algorithm used for model compression across a variety of deep learning based natural language processing (NLP) solutions. In its regular manifestations, KD requires access to the teacher’s training data for knowledge transfer to the student network. However, privacy concerns, data regulations and proprietary reasons may prevent access to such data. We present, to the best of our knowledge, the first work on Zero-shot Knowledge Distillation for NLP, where the student learns from the much larger teacher without any task specific data. Our solution combines out-of-domain data and adversarial training to learn the teacher’s output distribution. We investigate six tasks from the GLUE benchmark and demonstrate that we can achieve between 75% and 92% of the teacher’s classification score (accuracy or F1) while compressing the model 30 times.

## introduction deep learning based natural learning processing (nlp) systems have become state-of-the-art on many applications such as machine translation (mt) @xcite , natural language understanding (nlu) @xcite and language generation (brown et al., 2020) among others. these models are increasingly trained on huge corpora and with billions of trainable parameters @xcite . this is prohibitive for deploying them on edge devices as well as maintaining them on servers. moreover, training and evaluating them leaves a significant environmental footprint @xcite wherein avoiding the resource hungry training is very challenging @xcite and may be unavoidable @xcite . model com-pression approaches make it feasible to employ current state of the art models on edge devices. model compression @xcite has received a lot of attention in the nlp community due to the aforementioned reasons. some of the algorithms include model pruning @xcite , quantization @xcite , low-rank matrix factorization @xcite and knowledge distillation (kd) @xcite . kd is one of the most commonly used, application and model agnostic, compression and ensembling algorithm. it is one of the most widely researched algorithms for compressing transformer based language models @xcite @xcite @xcite . in kd, the student needs to be trained with the teacher's training data so as to prevent loss of accuracy. however, we can not assume this access for many practical problems. some of the concerns preventing access include data privacy, intellectual property, size and transience @xcite . e.g. a model trained on patient health records might be available but the data itself may be inaccessible due to patient privacy. in computer vision (cv), zero-shot kd (zskd) has been proposed to train a student without using any data. in this context zero-shot refers to training without using data instead of no training at all. @xcite propose generating "data impressions" by updating noise using backpropogation until it generates 'valid' teacher logits and then training the student on these data impressions. @xcite use a generator to produce synthetic images and use the teacher as discriminator, observing that for real images the softmax function of the teacher encourages a unimodal distribution. @xcite use a generator to produce synthetic training samples employing adver-sarial training to improve the quality. @xcite generate synthetic data by conditioning a generator on output samples from the teacher and a low dimensional representation of the generated samples. these works assume that there is no data available whatsoever for training the student but they do not transfer to text due to the discrete input space @xcite . we relax this condition and argue that we can still achieve the goals of zskd if we use easy to access out-of-domain (ood), task agnostic data to aid the process. @xcite put forth a similar argument, albeit for the problem of model extraction, where they use simple heuristic rules to generate training data for a student, of similar or larger size to the teacher, in order to learn the teacher's output distribution. however, they do not put constraints on the size of the student and even propose a student larger than the teacher. we study the problem of zskd for nlp and adapt an ood dataset similar to @xcite . in addition we train a text generator to generate samples which maximize the divergence between the teacher and student output while staying close to the ood distribution. our contributions are as follows: ## methodology we rely on an adversarial text generator as the backbone of our method. however, we still need data to pre-train the generator. since we assume access to a general purpose ood data, we delineate general principles to extract a training set from this source. finally, we apply kd on a combination of the ood training data and the adversarial training data. figure 1 gives a visual illustration of the proposed zskd method. ## conclusion we present the first study on zero-shot knowledge distillation (zskd) for nlp. we present an algorithm based on ood data generation and ad-versarial learning and evaluate on six tasks from the glue benchmark reaching to within 75% of the teacher performance on all tasks while attaining a 30x compression. the next steps are to a) explore a generic methodology for ood data creation and b) study sequence generation tasks such as machine translation and abstractive summarization and achieve compression without the original training data while having access to a teacher.