S parse A dapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters (2022)

Abstract: Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, becomes an appealing efficient alternative to the full model fine-tuning. Although computationally efficient, the recent Adapters often increase parameters (e.g. bottleneck dimension) for matching the performance of full model fine-tuning, which we argue goes against their original intention. In this work, we re-examine the parameter-efficiency of Adapter through the lens of network pruning (we name such plug-in concept as SparseAdapter) and find that SparseAdapter can achieve comparable or better performance than standard Adapters when the sparse ratio reaches up to 80%. Based on our findings, we introduce an easy but effective setting “Large-Sparse” to improve the model capacity of Adapters under the same parameter budget. Experiments on five competitive Adapters upon three advanced PLMs show that with proper sparse method (e.g. SNIP) and ratio (e.g. 40%) SparseAdapter can consistently outperform their corresponding counterpart. Encouragingly, with the Large-Sparse setting, we can obtain further appealing gains, even outperforming the full fine-tuning by a large margin.

## introduction the "pretrain-finetune" paradigm has become the de facto standard for the community of natural language processing (nlp) @xcite . given a pretrained language model (plm), the conventional fine-tuning manner is tuning the entire parameters, i.e., full finetuning, for each downstream task @xcite . considering the ever-increasing size of plms @xcite , full fine-tuning has figure 1 : performance of different parameter-efficient tuning methods on tasks from glue benchmark with roberta-base encoder. we report the performance of houlsby adapters, pfeiffer adapters, lora as well as that used in our plug-in method sparseadapter, where we denoted the normal sparse (in despite the progress, existing adapters match the performance of full fine-tuning by increasing the bottleneck dimension @xcite . this increases the overall parameters and flops, violating the original intention of adapters. in this work, we turn to investigate the parameter-efficiency property (the nature of adapters) to answer the following questions: 1 whether the current adapters can be further efficient? 2 how can we increase the representation capacity of adapters within the original parameter budget? to this end, we examine the parameter-efficiency of adapters through the lens of network prun-ing @xcite , which reduces the model size of neural networks by pruning redundant parameters and training the rest ones, therefore, improving the network efficiency. we call such pruned adapters sparseadapter. specifically, we systematically investigate five representative pruning methods in §2.2 to check at what sparse ratio can the adapters maintain the effectiveness. note that to maintain the efficient nature of adapters, we prune all adapters at initialization such that there are no extra computational costs. we find that 1 sparseadapter can achieve comparable (or even better) performance than standard adapters when the sparse ratio reaches up to 80%. such encouraging performance could hold even using the random pruning method (see figure 2 ) on glue benchmark @xcite . based on these insights, we introduce a frustratingly easy setting, namely large-sparse, for sparseadapter. we find that 2 scaling up the bottleneck dimension of sparseadapter with a correspondingly larger sparse ratio (to ensure the same parameter budget, for example, 2× dimension scaling with 50% sparse ratio) can effectively yield significant improvement by augmenting the model capacity. we validate the concept of our proposed sparseadapter upon five advanced adapters, i.e., houlsby @xcite , @xcite , lora @xcite , mam adapter @xcite and adapterfusion @xcite , spanning both natural language understanding (glue and squad) and generation (xsum) benchmarks. we show that with proper sparsity, e.g. 40%, sparseadapter could consistently outperform their correspondingly counterpart baselines. and with our large-sparse setting, sparseadapter could even beat the full fine-tuning method significantly, e.g. 79.6 vs. 79.0 in figure 1 . to check whether augmenting adapters by increasing the parameters is an optimal choice, we decide to revisit the nature of adapters, i.e., parameter efficiency, by pruning the redundant parameters. as shown in figure 2 , randomly pruned adapters can achieve comparable or even better performance than standard adapters, which indicates the existence of redundant parameters. the comparable performance could even be held under 80% sparsity. such preliminary study urges us to investigate the research questions 1 and 2 . we decide to approach them by systematically investigating the effects of different pruning methods. 6sduvlw\ 3huirupdqfh %(57/dujh 6sduvlw\ 3huirupdqfh 5r%(57d/dujh &r/$ 053& 676% 57( figure 3: schematic comparison of (a) standard adapter and (b) our proposed sparseadapter. fine-tune adapter fine-tuned adapter (a) standard adapter tuning. prune initialization fine-tune adapter sparseadapter fine-tuned sparseadapter (b) sparseadapter tuning. ## experiments setup. experiments were conducted on three widely-used benchmarks, spanning understanding and generation tasks: (1) glue @xcite , containing understanding tasks like natural language inference, sentiment analysis, and sentence similarity evaluation; (2) xsum @xcite , a summarization dataset where the models are required to generate a short summary for a given article; (3) squad v1.1 @xcite , a pair-wise dataset for questions and wikipedia paragraphs where models select the answer span to the question from the paragraph. we use adam (kingma and ba, 2014) as the optimizer with β 1 , β 2 = 0.9, 0.98. for regularization, we set the weight decay as 0.1 and grid-search the learning rate from {1e-5, 2e-5, 5e-5, 1e-4, 2e-4}, where we warm up the learning rate in the first 10% steps (of the total training steps). for different data scales, we grid-search the training epoch and batch size from {5, 10, 15, 20}, and {8, 16, 32, 64}, respectively. the maximum length is 512 for glue and 384 for squad. for xsum, we set the max length of source articles to be 512 and the max length of the target summary to be 128. for the glue benchmark, we follow previous works @xcite @xcite to fine-tune the pretrained language models, e.g. bert @xcite and roberta @xcite , on the downstream training set and report results on the dev set using the last checkpoint. for the other tasks, we report the test results. ## conclusion in this work, we systematically reexamine the parameter efficiency property of adapter tuning through the lens of network pruning. based on our findings, we propose a plug-in strategy, i.e., sparseadapter, for existing adapters. our study empirically indicates the potential to make sparseadapter (especially with the large-sparse setting) a golden standard efficient transfer learning strategy for the nlp community. the future work includes applying our proposed sparseadapter to more tasks (e.g. multilingual plm based machine translation @xcite ) and benchmarks, and investigating the parameter efficiency of of other neural network models, especially for scenarios where high efficiency is required, e.g. prompt @xcite .