A dversarial S elf- S upervised D ata- F ree D istillation for T ext C lassification (2020)

Abstract: Large pre-trained transformer-based language models have achieved impressive results on a wide range of NLP tasks. In the past few years, Knowledge Distillation(KD) has become a popular paradigm to compress a computationally expensive model to a resource-efficient lightweight model. However, most KD algorithms, especially in NLP, rely on the accessibility of the original training dataset, which may be unavailable due to privacy issues. To tackle this problem, we propose a novel two-stage data-free distillation method, named Adversarial self-Supervised Data-Free Distillation (AS-DFD), which is designed for compressing large-scale transformer-based models (e.g., BERT). To avoid text generation in discrete space, we introduce a Plug & Play Embedding Guessing method to craft pseudo embeddings from the teacher’s hidden knowledge. Meanwhile, with a self-supervised module to quantify the student’s ability, we adapt the difficulty of pseudo embeddings in an adversarial training manner. To the best of our knowledge, our framework is the first data-free distillation framework designed for NLP tasks. We verify the effectiveness of our method on several text classification datasets.

## introduction recently, pre-trained language models @xcite @xcite have achieved tremendous progress and reached the state-of-the-art performance in various downstream tasks such as text classification @xcite , language inference @xcite and question answering @xcite . these models become an indispensable part of current models for their transferability and generalizability. * corresponding author however, such language models are huge in volume and demand highly in computational resources, making it impractical in deploying them on portable systems with limited resources (e.g., mobile phones, edge devices) without appropriate compression. recent researches @xcite @xcite focus on compressing the large-scale models to a shallow and resource-efficient network via weight pruning @xcite , knowledge distillation @xcite , weight quantization @xcite and parameter sharing @xcite . among them, some methods @xcite draw on the idea of transfer learning, utilizing knowledge distillation @xcite to transfer latent representation information embedded in teachers to students. these knowledge distillation methods share some commonalities: they rely on the training data to achieve high accuracy. it will be intractable if we need to compress a model without publicly accessible data. reasons for that include privacy protection, company assets, safety/security concerns and transmission. representative samples include gpt2 @xcite , which has not released its training data with fears of abuse of language models. google trains a neural machine translation system @xcite using internal datasets owned and protected by the company. deepface @xcite ) is trained on user images under confidential policies for protecting users. further, some datasets, like common crawl dataset used in gpt3 @xcite , contain nearly a trillion words and are difficult to transmit and store. conventional knowledge distillation methods are highly dependent on data. some models or algorithms in computer vision like dafl @xcite , zskd @xcite solve the datafree distillation by generating pseudo images or uti-lizing metadata from teacher models. exploratory researches @xcite also show that gans can synthesize harder and more diversified images by exploiting disagreements between teachers and students. however, these models only make attempts in image tasks, designing for continuous and real-valued images. applying these models to generate sentences is challenging due to the discrete representation of words @xcite . backpropagation on discrete words is not reasonable, and it seems unlikely to pass the gradient through the text to the generator. apart from the discontinuity problem of text, some promotion strategies like layer-wise statistic matching in batch normalization @xcite are not suitable for transformer-based models, which transposes batch normalization into layer normalization to fit with varied sentence length @xcite . to address the above issues and distill without data, we propose a novel data-free distillation framework called "adversarial self-supervised data-free distillation"(as-dfd). we invert bert to perform gradient updates on embeddings and consider parameters of the embedding layer as accessible knowledge for student models. under constraints of constructing "bert-like" vectors, pseudo embeddings extract underlying representations of each category. besides, we employ a self-supervised module to quantify the student's ability and adversarially adjust the difficulty of pseudo samples, alleviating the insufficient supervisory problem controlled by the one-hot target. our main contributions are summarized as follows: ## methods in this section, we present our two-stage distillation framework named adversarial self-supervised data-free distillation (as-dfd). we craft welltrained embedding-level pseudo samples by controllable plug & play embedding guessing with alignment constraints (section 3.1) and adversarially adapt synthetic embeddings under selfsupervision of the student (section 3.2). using these pseudo samples, we transfer knowledge from the teacher to the student (section 3.3). the workflow of as-dfd is illustrated in figure 1 . problem definition knowledge distillation is a compression technique to train a high-performance model with fewer parameters instructed by the teacher model @xcite . let t be a large transformer-based teacher model (12layer bert-base here) and s be a comparatively lightweight student model. for each sentence x, the classification prediction can be formulated as: our goal of data-free knowledge distillation is to train the student parameters θ s with no data x available. in other words, we only have a teacher model t and we need to compress it. ## conclusion in this paper, we propose as-dfd, a novel datafree distillation method applied in text classification tasks. we use plug & play embedding guessing with alignment constraints to solve the problem that gradients cannot update on the discrete text. to dynamically adjust synthetic samples according to students' situations, we involve an adversarial self-supervised module to quantify students' abilities. experimental results on three text datasets demonstrate the effectiveness of as-dfd. however, it's still challenging to ensure the diversity of generated embeddings under the weak supervision signal and we argue that the gap between synthetic and real sentences still exists. in the future, we would like to explore data-free distillation on more complex tasks.