The development of over-parameterized pretrained language models has made a significant contribution toward the success of natural language processing . While over-parameterization of these models is the key to their generalization power , it makes them unsuitable for deployment on low-capacity devices . We push the limits of state-of-the-art Transformer-based pre-trained language model compression using Kronecker decomposition . We present our KroneckerBERT , a compressed version of the BERT BASE model obtained by compressing the embedding layer and the linear mappings in the multi-head attention , and the feed-forward network modules in the Transformer layers . Our KroneckerBERT is trained via a very efficient two-stage knowledge distillation scheme using far fewer data samples than state-of-the-art models like MobileBERT and TinyBERT . We evaluate the performance of KroneckerBERT on well-known NLP benchmarks . We show that our KroneckerBERT with compression factors of 7.7× and 21× outperforms state-of-theart compression methods on the GLUE and SQuAD benchmarks . In particular , using only 13 % of the teacher model parameters , it retain more than 99 % of the accuracy on the majority of GLUE tasks . In recent years , the emergence of Pre-trained Language Models ( PLMs ) has led to a significant breakthrough in Natural Language Processing ( NLP ) . The introduction of Transformers and unsupervised pre-training on enormous unlabeled data are the two main factors that contribute to this success . Transformer-based models ( Devlin et al . , 2018 ; Radford et al . , 2019 ; Yang et al . , 2019 ; Shoeybi et al . , 2019 ) are powerful yet highly overparameterized . The enormous size of these models does not meet the constraints imposed by edge devices on memory , latency , and energy consumption . Therefore there has been a growing interest in developing new methodologies and frameworks for the compression of these large PLMs . Similar to other deep learning models , the main directions for the compression of these models include low-bit quantization ( Gong et al . , 2014 ; Prato et al . , 2019 ) , network pruning ( Han et al . , 2015 ) , matrix decomposition ( Yu et al . , 2017 ; Lioutas et al . , 2020 ) and Knowledge distillation ( KD ) ( Hinton et al . , 2015 ) . These methods are either used in isolation or in combination to improve compression-performance trade-off . Recent works have been relatively successful in compressing Transformer-based PLMs to a certain degree ( Sanh et al . , 2019 ; Sun et al . , 2019 ; Jiao et al . , 2019 ; Sun et al . , 2020 ; Xu et al . , 2020 ; Wang et al . , 2020 ; Kim et al . , 2021 ) ; however , moderate and extreme compression of these models ( compression factors > 5 and 10 resepctively ) is still quite challenging . In particular , several works ( Mao et al . , 2020 ; Zhao et al . , 2019a Zhao et al . , , 2021 ) ) that have tried to go beyond the compression factor of 10 , have done so at the expense of a significant drop in performance . Following the classical assumption that matrices often follow a low-rank structure , low-rank decomposition methods have been used for compression of weight matrices in deep learning models ( Yu et al . , 2017 ; Swaminathan et al . , 2020 ; Winata et al . , 2019 ) and especially Transformer-based models ( Noach and Goldberg , 2020 ; Mao et al . , 2020 ) . However , low-rank decomposition methods only exploit redundancies of the weight matrix in the horizontal and vertical dimensions and thus limit the flexibility of the compressed model . Kronecker decomposition on the other hand exploits redun- dancies in predefined patches and hence allows for more flexibility in their representation . Recent works prove Kronecker product to be more effective in retaining accuracy after compression than SVD ( Thakker et al . , 2019 ) . This work proposes a novel framework that uses Kronecker decomposition for compression of Transformer-based PLMs and provides a very promising compression-performance trade-off for medium and high compression levels , with 13 % and 5 % of the original model parameters respectively . We use Kronecker decomposition for the compression of both Transformer layers and the embedding layer . For Transformer layers , the compression is achieved by representing every weight matrix both in the multi-head attention ( MHA ) and the feed-forward neural network ( FFN ) as a Kronecker product of two smaller matrices . We also propose a Kronecker decomposition for compression of the embedding layer . Previous works have tried different techniques to reduce the enormous memory consumption of this layer ( Khrulkov et al . , 2019 ; Li et al . , 2018 ) . Our Kronecker decomposition method can substantially reduce the amount of required memory while maintaining low computation . Using Kronecker decomposition for large compression factors leads to a reduction in the model expressiveness . This is due to the nature of the Kronecker product and the fact that elements in this representation are tied together . To address this issue , we propose to distill knowledge from the intermediate layers of the original uncompressed network to the Kronecker network during training . Training of the state-of-the art BERT compression models ( Zhao et al . , 2019a , b ; Sun et al . , 2020 Sun et al . , , 2019 ) ) involve an extensive training which requires vast computational resources . For example in ( Sun et al . , 2020 ) , first a specially designed teacher , i.e IB-BERT LARGE is trained from scratch on the en-tire English wikipedia and Book Corpus . The student is then pretrained on the same corpus via KD while undergoing an additional progressive KD phase . Another example is TinyBERT ( Jiao et al . , 2019 ) which requires pretraining on the entire English Wikipedia and also uses extensive data augmentation ( 20× ) for fine-tuning on the downstream tasks . We show that our Kronecker BERT can out perform state-of-the-art with significantly less training requirements . More precisely , our Kronecker-BERT model undergoes a very light pretraining on only 10 % of the English Wikipedia for 3 epochs followed by finetuning on the original downstream data . Note that , while our evaluations in this work are limited to BERT , this proposed compression method can be directly used to compress other Transformer-based NLP models . The main contributions of this paper are as follows : • Compression of the embedding layer using the Kronecker decomposition with very low computational overhead . We introduced a novel method for compressing Transformer-based language models that uses Kronecker decomposition for the compression of the embedding layer and the linear mappings within the Transformer blocks . The proposed framework was used to compress the BERT BASE model . We used a very light two-stage KD method to train the compressed model . We show that the proposed framework can significantly reduce the size and the number of computations while outperforming stateof-the-art . The proposed method can be directly applied for compression of other Transformer-based language models . The combination of the proposed method with other compression techniques such layer truncation , pruning and quantization can be an interesting direction for future work .

Challenge: Moderate or Extreme model compressions with factors of > 5 or 10  for transformer models are challenging without a significant performance drop or massive training.
Approach: They propose a compression method with Kronecker decomposition with a two-staged knowledge distillation from intermediate layers of the original model to avoid losing the expressiveness.
Outcome: Their model outperforms state-of-the-art compressed models on GLUE and SQuAD and retains more than 99% of the accuracy while using 13% of the original parameters.