DSEE : Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models (2023)

Abstract: Gigantic pre-trained models have become central to natural language processing (NLP), serving as the starting point for fine-tuning towards a range of downstream tasks. However, two pain points persist for this paradigm: (a) as the pre-trained models grow bigger (e.g., 175B parameters for GPT-3), even the fine-tuning process can be time-consuming and computationally expensive; (b) the fine-tuned model has the same size as its starting point by default, which is neither sensible due to its more specialized functionality, nor practical since many fine-tuned models will be deployed in resource-constrained environments. To address these pain points, we propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning - by enforcing sparsity-aware low-rank updates on top of the pre-trained weights; and (ii) resource-efficient inference - by encouraging a sparse weight structure towards the final fine-tuned model. We leverage sparsity in these two directions by exploiting both unstructured and structured sparse patterns in pre-trained language models viaa unified approach. Extensive experiments and in-depth investigations, with diverse network backbones (i.e., BERT, RoBERTa, and GPT-2) on dozens of datasets, consistently demonstrate impressive parameter-/inference-efficiency, while maintaining competitive downstream performance. For instance, DSEE saves about 25% inference FLOPs while achieving comparable performance, with 0.5% trainable parameters on BERT. Codes are available at https://github.com/VITA-Group/DSEE .

## introduction most recent nlp applications have been following the pre-train then fine-tune paradigm, starting from a gigantic pre-trained model and fine-tuning it towards downstream tasks. conventional fine-tuning works through updating all of the parameters in the pre-trained model. however, as the size of pre-trained models grows, updating all parameters becomes less feasible in most practical scenarios, due to the expensive memory and computational requirements. for example, bert base @xcite has 110m trainable parameters, while gpt-2 @xcite has up to 1.5b and the largest version of gpt-3 @xcite has an astonishing 175b trainable parameters. as such, conventional fine-tuning of the larger models could require hundreds of gpu hours. another downside of this paradigm is that it requires storing as many parameters as in the large-scale pretrained models for each downstream task, which poses impediments to the deployment in real-world resource-constrained environments. one solution to address the extensive resource requirement of conventional fine-tuning is model pruning @xcite @xcite @xcite , where unnecessary weights are eliminated to shrink the model size. for example, @xcite leverages ℓ 1 regularization to remove insignificant attention heads and gains 35 ∼ 45% training time with comparable performance; @xcite ; @xcite leverage sparse matrices with fixed structures to reduce pretrained models' sizes. all these studies indicate the rise of sparsity naturally during fine-tuning a general-purpose pre-trained model, to some specialized downstream functionality. one potential interpretation, of why sparsity arises, is that different subsets of the parameters may be responsible for different downstream tasks and data domains @xcite . however, identifying appropriate sparse masks can be burdensome: fine-tuning a large pre-trained language model like gpt-3 for just one step consumes at least 1.2tb of vram and requires 96 pieces of nvidia tesla @xcite , and these methods either require access to pre-trained weights or introduce additional learnable coefficients (such as importance scores of attention heads). a parallel alternative is to design parameterefficient fine-tuning algorithms, which aim at optimizing a small portion of weights while fixing most of them when fine-tuning on downstream tasks. pioneering works along this line, which utilize adapters @xcite , learnable embeddings @xcite , low-rank decomposition @xcite or their combination @xcite , can significantly reduce the number of trainable parameters while preserving good fine-tuning performance. although these methods can substantially improve the storage and deployment efficiency of models, there are two major hurdles: (i) they does not yield any inference efficiency gains since the full pretrained weights are still required to calculate outputs; and (ii) current methods assume the updates on pretrained weights to be either sparse @xcite or low-rank @xcite , yet those assumptions might be oversimplified @xcite and overly restricted to allow for effective updates. these observations have inspired us to explore better parameter-efficiency methods. to improve both resource-and parameterefficiency during model fine-tuning, we explicitly draw on the prior of sparsity for both weight updates and the final weights, and establish a dually sparsity-embedding efficient tuning (dsee) framework. starting from a pre-trained model, dsee first adopts a sparsity-aware low-rank weight update to achieve parameter-efficiency of the finetuning process; and then enforces a sparse weight structure directly from weight updates by masking to achieve resource-efficiency of the fine-tuned model at inference time. our contributions can be summarized as follows: pruning and sparsification pruning is a classical model compression technique that can reduce the number of parameters, which can bring training and inference efficiency. researchers have proposed several pruning methods for pre-trained language models: mccarley et al. (2019); chen et al. (2021b) pruned attention heads that had less contribution during finetuning; sanh et al. (2020) proposed a pruning criterion targeting the weight change after training, which suits the transfer learning better; wang et al. (2020) incorporated lowrank factorization and ℓ 0 regularization for pruning. recently, there is a series of sparsification works that utilize sparse masks with specific structures, called butterflies, and achieve high efficiency in pretraining models @xcite or finetuning on downstream tasks @xcite . however, these methods do not allow for parameterefficient updates. low-rank decomposition low-rank approximation @xcite has broad applications in the machine learning community and is vastly studied. one classical scenario is the robust principal component analysis @xcite , which decomposes a matrix into a low-rank plus a sparse component. existing literature shows that in deep learning, the learned over-parameterized models often naturally bear approximate low-rank weight structures @xcite . some @xcite @xcite @xcite have explicitly imposed the low-rank constraint during training. wang et al. (2020); hu et al. (2021) utilized low-rank decomposition to shrink the model size and trim down the trainable parameters during fine-tuning. however, to our best knowledge, integrating sparsity and low-rank structures has never been studied before for efficient fine-tuning of pre-trained language models. parameter-efficient adaptation. parameterefficient fine-tuning aims to reduce the number of trainable parameters when fine-tuning the models across different downstream domains. unlike pruning, it aims at adapting models with fewer parameters instead of building sparse models. various approaches are proposed to achieve this goal: rebuffi et al. (2017); houlsby et al. (2019) inserted and only trained adapters between existing layers, whose parameters are much less compared to the pretrained models. guo et al. (2020) leveraged ℓ 0 regularization to limit the number of non-zero elements in the update vectors. lester et al. (2021); li and liang (2021); liu et al. (2021) introduced efficient prompt tuning which optimizes only a small continuous task-specific vector. zaken et al. (2021) fine-tunes only the bias terms inside models. hu et al. (2021) proposed a low-rank decomposition-based method, and he et al. (2021) combined low-rank and adapter-based methods for efficient finetuning. however, fine-tuned models yielded by these methods have the same amount of weights as the pre-trained models; hence they contribute no resource efficiency. ## methodology in this section, we describe our notations and definitions of sparsity generation and parameter-efficient fine-tuning in section 3.1, and then introduce the dually sparsity-embedded efficient fine-tuning algorithms in sections 3.2 and 3.3. ## experiment results datasets and models. we use three classical pre-trained language models in our experiments: bert base @xcite , roberta large @xcite and gpt-2 @xcite , which have 12/24/24 layers with hidden size of 768/1024/1024 and 110/380/354m trainable parameters, respectively. for bert and roberta, we evaluate on the glue benchmarks @xcite , and for gpt-2 we use e2e @xcite , webnlg @xcite and dart @xcite . training and evaluation details. for bert and roberta, we follow the default settings in @xcite ; @xcite . we use the adamw @xcite optimizer for downstream fine-tuning, and a batch size of 32 for bert and roberta, and a batch size of 2 for gpt-2. the rest hyper-parameters for training are reported in ## conclusion this paper draws on the prior of sparsity and establishes the dsee framework. it is the first attempt toward jointly optimizing both parameterefficiency of the fine-tuning process, and the resource-efficiency of the fine-tuned model. on