Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation (2020)

Abstract: Recently, BERT has become an essential ingredient of various NLP deep models due to its effectiveness and universal-usability. However, the online deployment of BERT is often blocked by its large-scale parameters and high computational cost. There are plenty of studies showing that the knowledge distillation is efficient in transferring the knowledge from BERT into the model with a smaller size of parameters. Nevertheless, current BERT distillation approaches mainly focus on task-specified distillation, such methodologies lead to the loss of the general semantic knowledge of BERT for universal-usability. In this paper, we propose a sentence representation approximating oriented distillation framework that can distill the pre-trained BERT into a simple LSTM based model without specifying tasks. Consistent with BERT, our distilled model is able to perform transfer learning via fine-tuning to adapt to any sentence-level downstream task. Besides, our model can further cooperate with task-specific distillation procedures. The experimental results on multiple NLP tasks from the GLUE benchmark show that our approach outperforms other task-specific distillation methods or even much larger models, i.e., ELMO, with efficiency well-improved.

## introduction as one of the most important progress in the natural language processing field recently, the bidirectional encoder representation from transformers (bert) @xcite has been proved to be effective in improving the performances of various nlp tasks by providing a powerful pre-trained language model based on large-scale unlabeled corpora. recent studies have shown that bert's capability can be further enhanced by utilizing deeper architectures or performing the pre-training on larger * * equal contribution during the internship at tencent. corpora with appropriate guidance @xcite @xcite . despite its strength in building distributed semantic representations of sentences and supporting various nlp tasks, bert holds a huge amount of parameters that raises the difficulty of conducting online deployment due to its unsatisfying computational efficiency. to address this issue, various studies have been done to utilize the knowledge distillation @xcite for compressing bert and meanwhile keep its semantic modeling capability as much as possible @xcite . the distilling methodologies include simulating bert with a much smaller model (e.g., lstm) @xcite and reducing some of the components, such as transformers, attentions to obtain the smaller bert based model @xcite . nevertheless, the current methods highly rely on a labeled dataset upon a specified task. firstly, bert is fine-tuned on the specified task to get the teaching signal for distillation, and the student model with simpler architectures attempts to fit the task-specified fine-tuned bert afterward. such methodologies can achieve satisfying results by capturing the task-specified biases @xcite @xcite , which are inherited by the tuned bert @xcite . unfortunately, the powerful generalization nature of bert tends to be lost. apparently, distilling bert's original motivation is to obtain a lightweight substitution of bert for online implementations, and bert's general semantic knowledge, which plays a significant role in some nlp tasks like sentence similarity quantification, is expected to be maintained accordingly. meanwhile, for many nlp tasks, manual labeling is quite a high-cost work, and large amounts of annotated data can not be guaranteed to obtain. thus, it is of great necessity to compress bert with the non-task-specific training procedure on unlabeled datasets. for achieving the non-task-specific distillation from bert, this paper proposes a distillation loss function to approximate sentence representations by minimizing the cosine distance between the sentence representation given by the student network and the one from bert. as a result, a student network with a much smaller scale of parameters is produced. since the distilling strategy purely focuses on the simulation of sentence embeddings from bert, which is not directly related to any specific nlp task, the whole training procedure takes only a large amount of sentences without any manual labeling work. similar to bert, the smaller student network can also perform transfer learning to any sentence-level downstream tasks, such as text classification and sentence matching. the proposed methodology is evaluated on the open platform of general language understanding evaluation (glue) @xcite , including the single sentence (sst-2), similarity and paraphrase (qqp and mrpc), and natural language inference (mnli) tasks. the experimental results show that our proposed model outperforms the models distilled from a bert fine-tuned on a specific task. moreover, our model inferences more efficiently than other transformer-based distilled models. ## related works with the propose of elmo @xcite , various studies take the representation given by pretrained language models as additional features to improve the performances. @xcite propose universal language model finetuning (ulmfit), an effective transfer learning method that can be applied to any nlp task and accordingly, using pre-trained language models in downstream tasks became one of the most exciting directions. on this basis, developing with deeper network design and more effective training methods, pre-trained models' performances improved continuously @xcite @xcite . since the release of bert @xcite , the stateof-the-art (sota) results on 11 nlp tasks have been produced consequently. with the improvement in performances, the computing cost increases, and the inference procedure becomes slower accordingly. thus, various stud-ies focused on the model compression upon bert. among the most common model compression techniques, the knowledge distillation @xcite has been proven to be efficient in transferring the knowledge from large-scaled pre-trained language models into another one @xcite @xcite . with the help of proposed distillation loss, @xcite compressed bert into fewer layers by shortening the distance of internal representations between student and teacher berts. for the sentence-pair modeling, @xcite found the cross-attention function across sentences is consuming and tried to remove it with distillation on sentence-pair tasks. different from these studies distilling bert into transformer-based models, @xcite proposed convolutional student architecture to distill gpt for efficient text classification. moreover, focusing on the sequence labeling tasks, @xcite derived a bilstm or minibert from bert via standard distillation procedure to simulate the prediction on each token. besides, @xcite proposed to distill bert into a bilstm based model with penalizing the mean square error between the student's logits and the ones given by bert as the objective on specific tasks, and introduced various data augmentation methods during distillation. ## method as introduced in section 1, our proposed method consists of two procedures. firstly, we distill bert into a smaller student model via approximating the representation of sentences given by bert. afterward, similar to bert, the student model can be fine-tuned on any sentence-level task, such as text classification and sentence matching. ## conclusions in this paper, we have presented a sentence representation approximating oriented method for distilling the pre-trained bert model into a much smaller bilstm without specifying tasks, so as to inherit the general semantic knowledge of bert for better generalization and universal-usability. the experiments conducted based on the glue benchmark have shown that our proposed nontask-specific distillation methodology can improve the performances on multiple sentence-level downstream tasks. from the experimental results, the following conclusions can be drawn: 1) for a specified task, our proposed distillation method can bring the 5% improvement to the pure bilstm model on average; 2) the proposed model can outperform the state-of-the-art bilstm based pre-trained language model, which contains much more parameters; 3) compared to the task-specific distillation, our distilled model is less dependent on the corpus size of the downstream task with satisfying performances guaranteed.