Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning (2020)

Abstract: Pretrained large-scale language models have increasingly demonstrated high accuracy on many natural language processing (NLP) tasks. However, the limited weight storage and computational speed on hardware platforms have impeded the popularity of pretrained models, especially in the era of edge computing. In this work, we propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization. Besides the significantly reduced weight storage and computation, the proposed approach achieves high compression rates. Experimental results on different models (BERT, RoBERTa, and DistilBERT) on the General Language Understanding Evaluation (GLUE) benchmark tasks show that we achieve up to 5.0x with zero or minor accuracy degradation on certain task(s). Our proposed method is also orthogonal to existing compact pretrained language models such as DistilBERT using knowledge distillation, since a further 1.79x average compression rate can be achieved on top of DistilBERT with zero or minor accuracy degradation. It is suitable to deploy the final compressed model on resource-constrained edge devices.

## introduction transformer-based language model pre-training has proven to be highly effective in learning universal language representations from large-scale unlabeled data and being fine-tuned to adapt to downstream tasks @xcite . representative works such as bert @xcite , xlnet @xcite , roberta (liu * these authors contributed equally et al., 2019b), mt-dnn @xcite , al-bert @xcite , gpt-2 @xcite , and unilmv2 @xcite have substantially advanced the state-of-the-art across a variety of downstream tasks, such as text classification, natural language inference, and question answering. despite its success in performance improvement in natural language understanding and generation, the computational cost and data storage of transformer-based pre-trained language model are two widely recognized concerns due to transformer's deep architecture and rich parameters. these models typically contain several hundred million parameters. the recent released research models even reach multi-billion parameters, such as megatronlm (8.3 billion parameters) @xcite , turing-nlg (17 billion parameters) (microsoft, 2020) and gpt-3 (175 billion parameters) @xcite , which require more advanced computing facility. hence, it is imperative to reduce the computational cost and model storage of pre-trained transformer-based language models in order to popularize their applications in computer systems, especially in edge devices with limited resources. several works have been developed in the context of model compression, such as knowledge distillation @xcite @xcite , weight pruning @xcite , parameter sharing @xcite and weight quantization @xcite . for computer vision, the information compressed/reduced in image features can be partially retrieved from neighboring pixels since they share similar and uniform characteristics spatially. however, for nlp, the syntax and semantics information of transformer in language/text domain are more sensitive than that of computer vision. a high compression rate for large-scale language models is difficult to achieve on downstream nlp tasks. as a result, there are few works in exploring and optimizing hardware-friendly model compression techniques for state-of-the-art transformer-based pre-trained language models, to reduce the weight storage and computation on computer system while maintaining prediction accuracy. in this work, we propose an efficient transformer-based large-scale language representations using block structured pruning. the contributions of this work are as follows. we evaluate the proposed approach on several glue benchmark tasks @xcite . experimental results show that we achieve high compression rates with zero or minor accuracy degradation. with significant gain in weight storage reduction (up to 5×) and computation efficiency, our approach can maintain comparable accuracy score to original large models including distilbert. the hardware-friendly transformer-based acceleration method is suitable to be deployed on resourceconstrained edge devices. ## related work to address the memory limitation and high computational requirement of commonly seen deep learning platforms such as graphics processing unit (gpu), tensor processing unit (tpu) and fieldprogrammable gate array (fpga) on large-scale pre-trained language models, various of compact nlp models or model compression techniques have been investigated. albert @xcite utilizes parameter sharing technique across encoders to reduce weight parameters and uses the same layer structures as bert. it achieves comparable results on different benchmarks to bert. despite the weight storage reduction, the computational overhead remains unchanged since albert and bert have the same network structure. knowledge distillation is another type of model compression technique, which distills the knowledge from a large teacher model or an ensemble of models to a light-weighted student model @xcite . the student model is trained to intimate the class probabilities produced by the large teacher model. for instance, distil-bert @xcite applies knowledge distillation to bert, and achieves 1.67 × model size reduction and 1.63 × inference speedup, while retaining 97% accuracy on the dev sets on the glue benchmark, compared to bert. patient knowledge distillation @xcite is used to learn from multiple intermediate layers of the teacher model for incremental knowledge extraction. efficient deep learning methods can reduce the model size and accelerate the computation. it is well known that, in practice, the weight representation in deep learning models is redundant. after removing several redundant weights with appropriate model compression algorithms, the deep learning model can have minor accuracy degradation. prior work focused on heuristic and iterative non-structured magnitude weight pruning (a.k.a, irregular pruning) @xcite . it causes overhead in both weight storage and computation in computer systems. on weight storage, it results in irregular, sparse weight matrices (as arbitrary weights can be pruned), and relies on indices to be stored in a compressed format such as coordinate (coo) format. the introduced indices cause extra memory footprint, i.e., at least one index per nonzero value, further degrading the compression rate. on computation, it is difficult to be accelerated on current gpu architectures as reported in (han et al. @xcite @xcite . on the other hand, structured pruning considers regularity in weight pruning focusing on generating regular but smaller and dense matrix with no index. however, it suffers notable accuracy loss due to the poor solution quality, and therefore not suitable for pruning sensitive syntax and semantics information in transformer. ## ablation studies in this section, we perform ablation experiments over several parameters when pruning bert and distilbert to better understand their relative importance and the procedure. we change the selection of following parameters: the numbers of blocks for reweighted training and block structured pruning, retraining epochs, and penalty factors. we also evaluate the knowledge distillation friendly. ## conclusion in this work, we propose an hardware-friendly block structured pruning pruning framework for transformer-based large-scale language representation. we incorporate the reweighted group lasso into for optimization and relax the hard constraints in block structured pruning. we significantly reduce weight storage and computational requirement. experimental results on different models (bert, roberta, and distilbert) on the glue benchmark tasks show that we achieve significant compression rates with zero or minor accuracy degradation on certain benchmarks. our proposed method is orthogonal to existing compact pretrained language models such as distilbert using knowledge distillation. it is suitable to deploy the final compressed model on resource-constrained edge devices.