Rethinking Network Pruning – under the Pre-train and Fine-tune Paradigm (2021)

Abstract: Transformer-based pre-trained language models have significantly improved the performance of various natural language processing (NLP) tasks in the recent years. While effective and prevalent, these models are usually prohibitively large for resource-limited deployment scenarios. A thread of research has thus been working on applying network pruning techniques under the pretrain-then-finetune paradigm widely adopted in NLP. However, the existing pruning results on benchmark transformers, such as BERT, are not as remarkable as the pruning results in the literature of convolutional neural networks (CNNs). In particular, common wisdom in pruning CNN states that sparse pruning technique compresses a model more than that obtained by reducing number of channels and layers, while existing works on sparse pruning of BERT yields inferior results than its small-dense counterparts such as TinyBERT. In this work, we aim to fill this gap by studying how knowledge are transferred and lost during the pre-train, fine-tune, and pruning process, and proposing a knowledge-aware sparse pruning process that achieves significantly superior results than existing literature. We show for the first time that sparse pruning compresses a BERT model significantly more than reducing its number of channels and layers. Experiments on multiple data sets of GLUE benchmark show that our method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy.

## introduction pre-trained language models, such as bert @xcite , become the standard and effective methods for improving the performance of a variety of natural language processing (nlp) tasks. these models are pre-trained in a self-supervised fashion and then fine-tuned for supervised downstream tasks. however, these models suffer from the heavy model size, making them impractical for resourcelimited deployment scenarios and incurring cost concerns @xcite . in parallel, an emerging subfield has studied the redundancy in deep neural network models @xcite and proposed to prune networks without sacrificing performance, such as the lottery ticket hypothesis @xcite . common wisdom in cnn literature shows that sparse pruning leads to more compression rate than structural pruning. for example, for the same number of parameters (0.46m), the sparse mobilenets improve by 11.2% accuracy over the dense ones @xcite . however, similar conclusions are not observed for pre-trained language models. the main question this paper attempts to answer is: how to perform sparse pruning under the pre-train and fine-tune paradigm? answering this question correctly is challenging. first, these models adopt pre-training and fine-tuning procedures, during which the general-purpose language knowledge and the task-specific knowledge are learned respectively. thus, it is desirable and challenging to keep the weights that are important to both knowledge during pruning. second, unlike cnns, pre-trained language models have a complex architecture consisting of embedding, self-attention, and feed-forward layers. to address these challenges, we propose sparse-bert, a knowledge-aware sparse pruning method for pre-trained language models, with a special focus on the widely used bert model. sparsebert is executed in the fine-tuning stage. it preserves both general-purpose and task-specific language knowledge while pruning. to preserve the generalpurpose knowledge learned during pre-training, g is an encoder. g l and g l d are the encoders well-trained on the pre-training and fine-tuning datasets respectively. l and d are the general-purpose language knowledge and the task-specific knowledge respectively. there is a domain error between pre-training and testing, and a generalization error between fine-tuning and testing. (b) and (c) are two basic pruning strategies (section 3.2.1). both l d and l pr are subsets of knowledge l. l d is related to the downstream task. ## related work a lot of efforts have been made on studying network redundancy and pruning networks without accuracy loss @xcite . for example, the work on lottery ticket hypothesis @xcite showed that there exist sparse smaller subnetworks capable of training to full accuracy in cnns. common wisdom in cnn literature shows that spare pruning leads to much more compression rate than structural pruning @xcite . for example, for the same number of parameters (0.46m), the sparse mobilenets achieve 61.8% accuracy while the dense ones achieve 50.6% @xcite . however, similar observations are not observed in existing approaches for pretrained language models @xcite @xcite @xcite . our method aims to fill the gap and summarize these pruning strategies. there are other compression approaches for pre-trained language models, such as quantization @xcite and weight factorization @xcite , which are out of the scope of this work. ## sparsebert we first formalize the knowledge transfer involved in fine-tuning pre-trained language models. then, we introduce our sparsebert. ## conclusion we introduce sparsebert, a knowledge-aware sparse pruning method for pre-trained language models, with a focus on bert. we summarize different types of bert pruning approaches and compare sparsebert with leading competitors. experimental results on glue and squad benchmarks demonstrate the superiority of sparsebert.