BERT -of-Theseus: Compressing BERT by Progressive Module Replacing (2020)

Abstract: In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact substitutes. Then, we randomly replace the original modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively increase the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the original and compact models. Compared to the previous knowledge distillation approaches for BERT compression, our approach does not introduce any additional loss function. Our approach outperforms existing knowledge distillation approaches on GLUE benchmark, showing a new perspective of model compression.

## introduction with the prevalence of deep learning, many huge neural models have been proposed and achieve state-of-the-art performance in various fields @xcite . specifically, in natural language processing (nlp), pretraining and fine-tuning have become the new norm of most tasks. transformer-based pretrained models @xcite @xcite @xcite have dominated the field of both natural language understanding (nlu) and natural language generation (nlg). these models benefit from their "overparameterized" nature @xcite and contain millions or even billions of parameters, making it computationally expensive and inefficient considering both memory consumption and * equal contribution. work done during these two authors' internship at microsoft research asia. 1 the code and pretrained model are available at https: //github.com/jetrunner/bert-of-theseus high latency. this drawback enormously hinders the applications of these models in production. to resolve this problem, many techniques have been proposed to compress a neural network. generally, these techniques can be categorized into quantization @xcite , weights pruning @xcite and knowledge distillation (kd) @xcite . among them, kd has received much attention for compressing pretrained language models. kd exploits a large teacher model to "teach" a compact student model to mimic the teacher's behavior. in this way, the knowledge embedded in the teacher model can be transferred into the smaller model. however, the retained performance of the student model relies on a welldesigned distillation loss function which forces the student model to behave as the teacher. recent studies on kd @xcite even leverage more sophisticated model-specific distillation loss functions for better performance. different from previous kd studies which explicitly exploit a distillation loss to minimize the distance between the teacher model and the student model, we propose a new genre of model compression. inspired by the famous thought experiment "ship of theseus" foot_0 in philosophy, where all components of a ship are gradually replaced by new ones until no original component exists, we propose theseus compression for bert (bert-of-theseus), which progressively substitutes modules of bert with modules of fewer parameters. we call the original model and compressed model predecessor and successor, in correspondence to the concepts of teacher and student in kd, respectively. as shown in figure 1 , we first specify a substitute (successor module) for each predecessor module (i.e., modules in the predecessor model). then, we randomly replace each predecessor module with its corresponding successor module by a probability and make them work together in the training phase. after convergence, we combine all successor modules to be the successor model for inference. in this way, the large predecessor model can be compressed into a compact successor model. theseus compression shares a similar idea with kd, which encourages the compressed model to behave like the original, but holds many merits. first, we only use the task-specific loss function in the compression process. however, kd-based methods use task-specific loss, together with one or multiple distillation losses as its optimization objective. also, selecting various loss functions and balancing the weights of each loss for different tasks and datasets can be laborious @xcite . second, different from recent work @xcite , theseus compression does not use transformer-specific features for compression thus is potential to compress a wide spectrum of models. third, instead of using the original model only for inference in kd, our approach allows the predecessor model to work in association with the compressed successor model, enabling a possible gradient-level interaction. moreover, the different module permutations mixing both predecessor and successor modules may add extra regularization, similar to dropout @xcite . with a curriculum learning @xcite driven replacement scheduler, our approach achieves promising performance compressing bert @xcite , a large pretrained transformer model. to summarize, our contribution is two-fold: (1) we propose a novel approach, theseus compression, revealing a new pathway to model compression, with no additional loss function. (2) our compressed bert model is 1.94× faster while retaining more than 98% performance of the original model, outperforming other kd-based compression baselines. ## related work model compression model compression aims to reduce the size and computational cost of a large model while retaining as much performance as possible. conventional explanations @xcite claim that the large number of weights is necessary for the training of deep neural network but a high degree of redundancy exists after training. recent work @xcite proposes the lottery ticket hypothesis claiming that dense, randomly initialized and feed-forward networks contain subnetworks that can be recognized and trained to get a comparable test accuracy to the original network. quantization @xcite reduces the number of bits used to represent a number in a model. weights pruning @xcite conducts a binary classification to decide which weights to be trimmed from the model. knowledge distillation (kd) @xcite aims to train a compact model which behaves like the original one. fitnets @xcite demonstrates that "hints" learned by the large model can benefit the distillation process. born-again neural network @xcite reveals that ensembling multiple identical-parameterized students can outperform a teacher model. lit @xcite introduces block-wise intermediate representation training. @xcite distilled knowledge from ensemble models to improve the performance of a single model on nlu tasks. @xcite exploited kd for multi-lingual machine translation. different from kd-based methods, our proposed theseus compression is the first approach to mix the original model and compact model for training. also, no additional loss is used throughout the whole compression procedure, which simplifies the implementation. faster bert very recently, many attempts have been made to speed up a large pretrained language model (e.g., bert @xcite ). @xcite reduced the parameters of a bert model by pruning unnecessary heads in the transformer. @xcite quantized bert to 2-bit using hessian information. also, substantial modification has been made to transformer architecture. @xcite exploited a structure dropping mechanism to train a bert-like model which is resilient to pruning. albert @xcite leverages matrix decomposition and parameter sharing. however, these models cannot exploit ready-made model weights and require a full retraining. @xcite used a bilstm architecture to extract task-specific knowledge from bert. distilbert @xcite applies a naive knowledge distillation on the same corpus used to pretrain bert. patient knowledge distillation (pkd) @xcite designs multiple distillation losses between the module hidden states of the teacher and student models. pretrained distilla-r 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " t d a 4 s p s t 1 k o q c j r l p t n n z w g s j r @xmath0 ## bert-of-theseus in this section, we introduce module replacing, the technique proposed for bert-of-theseus. further, we introduce a curriculum learning driven scheduler to obtain better performance. the workflow is shown in figure 1 . ## experiments in this section, we introduce the experiments of theseus compression for bert @xcite compression. we compare bert-of-theseus with other compression methods and further conduct experiments to analyze the results. ## analysis in this section, we conduct extensive experiments to analyze our bert-of-theseus. ## discussion in this paper, we propose theseus compression, a novel model compression approach. we use this approach to compress bert to a compact model that outperforms other models compressed by knowledge distillation. our work highlights a new genre of model compression and reveals a new path towards model compression. for future work, we would like to explore the possibility of applying theseus compression on heterogeneous network modules. first, many developed in-place substitutes (e.g., shufflenet unit @xcite for resblock @xcite , reformer layer @xcite for transformer layer @xcite ) are natural successor modules that can be directly adopted in theseus compression. also, it is possible to use a feed-forward neural network to map features between the hidden spaces of different sizes @xcite to enable replacement between modules with different input and output sizes. although our model has achieved good performance compressing bert, it would be interesting to explore its possible applications in other neural models. as summarized in