Pruning Pre-trained Language Models Without Fine-Tuning (2023)

Abstract: To overcome the overparameterized problem in Pre-trained Language Models (PLMs), pruning is widely used as a simple and straightforward compression method by directly removing unimportant weights. Previous first-order methods successfully compress PLMs to extremely high sparsity with little performance drop. These methods, such as movement pruning, use first-order information to prune PLMs while fine-tuning the remaining weights. In this work, we argue fine-tuning is redundant for first-order pruning, since first-order pruning is sufficient to converge PLMs to downstream tasks without fine-tuning. Under this motivation, we propose Static Model Pruning (SMP), which only uses first-order pruning to adapt PLMs to downstream tasks while achieving the target sparsity level. In addition, we also design a new masking function and training objective to further improve SMP. Extensive experiments at various sparsity levels show SMP has significant improvements over first-order and zero-order methods. Unlike previous first-order methods, SMP is also applicable to low sparsity and outperforms zero-order methods. Meanwhile, SMP is more parameter efficient than other methods due to it does not require fine-tuning.

## introduction pre-trained language models (plms) like bert @xcite have shown powerful performance in natural language processing by transferring the knowledge from large-scale corpus to downstream tasks. these models also require large-scale parameters to cope with the large-scale corpus in pretraining. however, these large-scale parameters are overwhelming for most downstream tasks @xcite , which † corresponding author. results in significant overhead for transferring and storing them. to compress plm, pruning is widely used by removing unimportant weights and setting them to zeros. by using sparse subnetworks instead of the original complete network, existing pruning methods can maintain the original accuracy by removing most weights. magnitude pruning @xcite as a common method uses zeroth-order information to make pruning decisions based on the absolute value of weights. however, in the process of adapting to downstream tasks, the weight values in plms are already predetermined from the original values. to overcome this shortcoming, movement pruning @xcite ) uses firstorder information to select weights based on how they change in training rather than their absolute value. to adapt plms for downstream tasks, most methods like movement pruning perform pruning and fine-tuning together by gradually increasing the sparsity during training. with the development of the lottery ticket hypothesis (lth) @xcite in plms, some methods @xcite find certain subnetworks from the plm by pruning, and then fine-tune these subnetworks from pre-trained weights. moreover, if the fine-tuned subnetwok can match the performance of the full plm, this subnetwork is called winning ticket @xcite . in this work, we propose a simple but efficient first-order method. contrary to the previous pruning method, our method adapts plms by only pruning, without fine-tuning. it makes pruning decisions based on the movement trend of weights, rather than actual movement in movement pruning. to improve the performance of our method, we propose a new masking function to better align the remaining weights according to the architecture of plms. we also avoid fine-tuning weights in the task-specific head by using our head initialization method. by keeping the plm frozen, we can save half of the trainable parameters compared to other first-order methods, and only introduce a binary mask as the new parameter for each downstream task at various sparsity levels. extensive experiments on a wide variety of sparsity demonstrate our methods strongly outperform state-of-the-art pruning methods. contrary to previous first-order methods @xcite , which show poor performance at low sparsity, our method is also applied to low sparsity and achieves better performances than zero-order methods. ## related work compressing plms for transfer learning is a popular area of research. many compression methods are proposed to solve overparameterized problem in plms, such as model pruning @xcite @xcite , knowledge distillation @xcite , quantization @xcite , and matrix decomposition @xcite . among them, pruning methods have been widely studied as the most intuitive approach. pruning methods focus on identifying and removing unimportant weights from the model. zeroorder methods and first-order methods are widely used to prune plms. for zero-order methods, magnitude pruning @xcite simply prunes based on absolute value of their weights. for first-order methods, which are based on first-order taylor expansion to make pruning decision, l 0 regularization @xcite adds the l 0 norm regularization to decrease remaining weights by sampling them with hard-concrete distribution. movement pruning @xcite uses straightthrough estimator @xcite to calculate first-order informantion. based on pruning methods, @xcite proposes lottery ticket hypothesis (lth). lth clarifies the existence of sparse subnetworks (i.e., winning tickets) that can achieve almost the same performance as the full model when trained individually. with the development of lth, lots of works that focus on the plms have emerged. @xcite find that bert contains winning tickets with a sparsity of 40% to 90%, and the winning ticket in the mask language modeling task can be transferred to other downstream tasks. recent works also try to leverage lth to improve the performance and efficiency of plm. @xcite find generalization performance of the winning tickets first improves and then deteriorates after a certain threshold. by leveraging this phenomenon, they show lth can successfully improve the performance of downstream tasks. ## background let @xmath0, where @xmath1, @xmath2. the pruning can be represented by @xmath3, where @xmath4, 1} n×n is the binary mask. we first review two common pruning methods in plms: magnitude pruning @xcite and movement pruning @xcite . magnitude pruning relies on the zeroth-order information to decide m by keeping the top v percent of weights according to their absolute value @xmath5. the importance scores @xmath6: for movement pruning, it relies on the first-order information by learning the importance scores s with gradient. the gradient of s is approximated with the staight-through estimator @xcite , which directly uses the gradient from m. according to @xcite , the importance scores s is: to achieve target sparsity, one common method is automated gradual pruning (michael h. zhu, 2018). the sparsity level v is gradually increased with a cubic sparsity scheduler starting from the training step t 0 : during training, these methods update both w and s to perform pruning and fine-tuning simultaneously. since fine-tuned weights stay close to their pre-trained values @xcite , the importance scores of magnitude pruning is influenced by pre-trained values, which limits its performance at high sparsity. however, magnitude pruning still outperforms movement pruning at low sparsity. ## static model pruning in this work, we propose a simple first-order pruning method called static model pruning (smp). it freezes w to make pruning on plms more efficient and transferable. based on movement pruning @xcite , our importance scores s is: ## conclusion in this paper, we propose a simple but effective task-specific pruning method called static model pruning (smp). considering previous methods, which perform both pruning and fine-tuning to adapt plms to downstream tasks, we find finetuning can be redundant since first-order pruning already converges plms. based on this, our method focuses on using first-order pruning to replace finetuning. without fine-tuning, our method strongly outperforms other first-order methods. extensive experiments also show that our method achieves state-of-the-art performances at various sparsity. for the lottery ticket hypothesis in bert, we find it contains sparsity subnetworks that achieve original performance without training them, and these subnetworks at 80% remaining weights even outperform fine-tuned bert on glue.