Large pre-trained language models reach stateof-the-art results on many different NLP tasks when fine-tuned individually ; They also come with a significant memory and computational requirements , calling for methods to reduce model sizes ( green AI ) . We propose a twostage model-compression method to reduce a model 's inference time cost . We first decompose the matrices in the model into smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition . This approach has the benefit of reducing the number of parameters while preserving much of the information within the model . We experimented on BERTbase model with the GLUE benchmark dataset and show that we can reduce the number of parameters by a factor of 0.4x , and increase inference speed by a factor of 1.45x , while maintaining a minimal loss in metric performance . Deep learning models have been demonstrated to achieve state-of-the-art results , but require large parameter storage and computation . It 's estimated that training a Transformer model with a neural architecture search has a CO 2 emissions equivalent to nearly five times the lifetime emissions of the average U.S. car , including its manufacturing ( Strubell et al . , 2019 ) . Alongside the increase in deep learning models complexity , in the NLP domain , there has been a shift in the NLP modeling paradigm from training a randomly initialized model to fine-tuning a large and computational heavy pre-trained language model ( Howard and Ruder , 2018 ; Peters et al . , 2018 ; Devlin et al . , 2018 ; Radford , 2018 ; Radford et al . , 2019 ; Dai et al . , 2019 ; Yang et al . , 2019 ; Lample and Conneau , 2019 ; Liu et al . , 2019b ; Raffel et al . , 2019 ; Lan et al . , 2019 ; Lewis et al . , 2019 ) . While re-using pre-trained models offsets the training costs , inference time costs of the finetuned models remain significant , and are showstoppers in many applications . The main challenge with pre-trained models is how can we reduce their size while saving the information contained within them . Recent work , approached this by keeping some of the layers while removing others ( Sanh et al . , 2019 ; Sun et al . , 2019 ; Xu et al . , 2020 ) . A main drawback of such approach is in its coarse-grained nature : removing entire layers might discard important information contained within the model , and working at the granularity of layers makes the trade-off between compression and accuracy of a model hard to control . Motivated by this , in this work we suggest a more finegrained approach which decomposes each matrix to two smaller matrices and then perform feature distillation on the internal representation to recover from the decomposition . This approach has the benefit of preserving much of the information while reducing the number of parameters . Alongside the advantage of preserving the information within each layer , there is also a memory flexibility advantage compared to removing entire layers ; As a result of decomposing each matrix to two smaller matrices , we can store each of the two matrices in two different memory blocks . This has the benefit of distributing the model matrices in many small memory blocks , which is useful when working in shared CPU-based environments . We evaluated our approach on the General Language Understanding Evaluation ( GLUE ) benchmark dataset ( Wang et al . , 2018 ) and show that our approach is superior or competitive in the different GLUE tasks to previous approaches which remove entire layers . Furthermore , we study the effects of different base models to decompose and show the superiority of decomposing a fine-tuned model compared to a pre-trained model or a ran-domly initialized model . Finally , we demonstrate the trade-off between compression and accuracy of a model . We presented a way to compress pre-trained large language models fine-tuned for specific tasks , while preserving much of the information contained within them , by using matrix decomposition to two small matrices . For future work it might be interesting to combine this approach with another approach such as pruning or quantization to achieve smaller models .

Challenge: Existing coarse-grained approaches for reducing the inference time of pretraining models remove layers, posing a trade-off between compression and the accuracy of a model.
Approach: They propose a model-compression method which decompresses the matrix and performs feature distillation on the internal representations to recover from the decomposition.
Outcome: The proposed method reduces the model size by 0.4x and increases inference speed by 1.45x while keeping the performance degradation minimum on the GLUE benchmark.