On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021)

Abstract: Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

## introduction large scale pretrained language models (prlms) @xcite @xcite have achieved state-ofthe-art results on most natural language processing (nlp) tasks, where fine-tuning has become a dominant approach to utilize prlms. a standard finetuning process copies weights from a prlm and tunes them on a downstream task, which requires a new set of weights for each task. adapter-based tuning @xcite has been proposed as a more parameter-efficient alternative. for nlp, adapters are usually light-weight modules inserted between transformer layers @xcite . during model tuning on a downstream task, only the parameters of adapters are updated while the weights of the original prlm are frozen. hence, adapter-based tuning adds only a small amount of parameters for each task, allowing a high degree of parameter-sharing. though using much less trainable parameters, adapter-based tuning has demonstrated comparable performance with full prlm fine-tuning @xcite @xcite . existing work mostly focuses on the parameterefficient aspect of adapters and attempt to derive useful applications from that, which is still the case in most recent works: @xcite explore methods to further improve the parameter and computation efficiency of adapters; @xcite combine knowledge from multiple adapters to improve the performance on downstream tasks; @xcite and @xcite leverage the modular architecture of adapters for parameter-efficient transfer to new languages or tasks, and @xcite utilize the same property for knowledge injection. besides parameter-efficiency, the unique characteristic of adapter-based tuning, with alternating frozen and learnable layers, might be directly useful for improving model performances. @xcite has not yet been discussed in the prior work. in this paper, we first empirically demonstrate that adapter-based tuning better regularizes training than fine-tuning by mitigating the issue of forgetting. we show that it yields representations with less deviation from those generated by the original prlm. next, to see what this property of adapters will help when adapting prlms, we compare the performance of fine-tuning and adapter-based tuning on a wide range of datasets ## monolingual adaptation in this section, we first experiment with eight datasets as used in @xcite including both high-and low-resource tasks ( §3.1). we refer this set of tasks as task adaptation evaluation (tae). we observe that adapter-based tuning consistently outperforms fine-tuning on lowresource tasks, while they perform similarly on high-resource tasks. we further confirm the effectiveness of adapters in low-resource settings on the glue benchmark @xcite ( §3.2). ## cross-lingual adaptation in this section, we further compare fine-tuning and adapter-based tuning in the zero-shot cross-lingual transfer setting. all experiments in this section are based on xlm-r-large @xcite , a recent sota multilingual prlm covering 100 languages. we conduct evaluations on a set of multilingual tasks from xtreme @xcite , including universal dependencies v2.5 tree banks (ud-pos) @xcite , wikiann ner @xcite , and cross-lingual natural language inference (xnli) @xcite . ud-pos contains 34 languages, wikiann ner contains 40 languages, and xnli contains 15 languages. we refer the reader to @xcite for additional details about the datasets. experimental setup on each task, we perform hyperparameter tuning on the english development set. for both fine-tuning and adapter-based tuning, we use batch size 32, and tune the learning rates in {1e-5, 2e-5, 3e-5, 4e-5, 5e-5}. for adapterbased tuning, we further tune the hidden sizes in {64, 128, 256} and find size 256 often performs the best. we train and select models with the english training and development sets and then evaluate the tuned models on test sets of all languages. see appendix a.2 for hyperparameter and training details. results ## analysis adapter hidden size the hidden size m 4 is the only adapter-specific hyperparameter. as indicated in @xcite , the hidden size provides a simple means to trade off performance with parameter efficiency. ## related work fine-tuning pretrained large scale language models has proven its effectiveness on a wide range of nlp tasks @xcite @xcite . however, fine-tuning requires a new set of weights for each task, which is parameter inefficient. adapterbased tuning is proposed to deal with this problem @xcite . most previous work has demonstrated that it achieves comparable performance to fine-tuning @xcite @xcite @xcite . however, existing work mostly focuses on the parameter-efficient aspect while overlooks the effectiveness. fine-tuning prlms in a low-resource setting has been studied for a while @xcite @xcite @xcite . previous work points out that with large-scale parameters, fine-tuning on a few samples can lead to overfitting and bad generalization, which causes the results unstable. @xcite find that pretraining on an intermediate task can improve fine-tuning outcomes. @xcite improve the robustness of fine-tuning by controlling the model complexity and preventing aggressive updating. on the other hand, catastrophic forgetting can appear when transferring a pretrained neural networks @xcite @xcite , where the learned knowledge from pretraining is lost when adapting to downstream tasks. this phenomenon often appears in nlp tasks @xcite . to relieve this problem of adapting pretrained language models, howard and ruder (2018) gradually unfreeze the layers starting from the last layer and @xcite find assigning lower learning rate to the bottom layers can improve the performance. @xcite regularize learning by encouraging the weights of the updated model to stay close to the initial weights. @xcite regularize fine-tuning by introducing noise to the input which is similar to adversarial training for fine-tuning studied in @xcite . @xcite point out that the instability of fine-tuning lies in the optimizer and propose to revise the adam optimizer by replacing it with a de-bias version. @xcite propose a mechanism to recall the knowledge from pretraining tasks. ## conclusion prior work often focuses on the parameter-efficient aspect while overlooks the effectiveness of adapterbased tuning. we empirically demonstrate that adapter-based tuning can better regularize the learning process. we conduct extensive experiments to verify its effectiveness and conclude that 1) it tends to outperform fine-tuning on both low-resource and cross-lingual tasks; 2) it demonstrates higher stability under different learning rates compared to fine-tuning. we hope our study will inspire more future work on prlm adaptation based on adapters and other methods that only tune part of the prlm parameters.