A dapter B ias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks (2022)

Abstract: Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed. AdapterBias adds a token-dependent shift to the hidden output of transformer layers to adapt to downstream tasks with only a vector and a linear layer. Extensive experiments are conducted to demonstrate the effectiveness of AdapterBias. The experiments show that our proposed method can dramatically reduce the trainable parameters compared to the previous works with a minimal decrease in task performances compared with fine-tuned pre-trained models. We further find that AdapterBias automatically learns to assign more significant representation shifts to the tokens related to the task in consideration.

## introduction while large pre-trained language models (plms) reached state-of-the-art results on natural language processing (nlp) tasks, plms require updating all parameters and storing the fully fine-tuned model for each downstream task. these requirements have led to difficulties in real-world applications. moreover, fine-tuning plms on lowresource datasets is subject to instabilities. to tackle these shortcomings, adapters @xcite , a more parameter-efficient alternative training strategy for the transformer architecture @xcite have been proposed. instead of full fine-tuning the whole model, adapters introduce extra tunable weights and freeze the original parameters of plm. adapters demonstrated comparable performance with fully fine-tuning the figure 1 : overview of the main concept of our work compared to @xcite . left: bit-fit tends to add the same representation shift to different tokens. right: our work applies different representation shifts to tokens considering their importance to the downstream task and their characteristics. the shifts of the input words that are more task-related is more significant than that of other tokens. for example, in sst-2 @xcite , which is a semantic task, the representation shifts of the semantic words, such as "kind" and "worse", are larger than that of other words. to develop practical and memory-efficient methods of utilizing plms, diff pruning @xcite enables parameter-efficient transfer learning that scales well with new tasks. the approach learns a task-specific "diff" vector that extends the original pre-trained parameters and encourages the sparsity of the vector through l 0 -norm regularization. another approach is bitfit @xcite , which shows that with small-to-medium training data, fine-tuning only a subset of the bias terms of pre-trained bert models @xcite is competitive with fine-tuning the entire model. the central concept of these approaches is to add task-specific shifts to each output representation of the plm layers so as to adapt to different tasks. in the previous works, ben @xcite ; @xcite both add the same shifts to the output representation regardless of which token is more relevant to the task. however, considering some specific tokens might be more critical to a particular task, the representation can better adapt to the downstream task under a limited amount of parameters if these shifts are based on the input tokens. based on this concept, in this study, we add token-dependent biases to the shifts by proposing adapterbias, which consists of a vector and a linear layer (l α ). the vector represents the task-specific shift, and l α produces the weights for input tokens. thus, with the vector and the weights, adapterbias can add a token-dependent shift to the transformer layer. since the concept of bitfit @xcite is similar to adapterbias by adding a shift to the representation, we demonstrate the difference between bitfit and adapterbias in figure 1 . bit-fit assigns identical shifts to all the tokens, while adapterbias adds more significant shifts to the representations that are related to the task. with fewer trainable parameters required, adapterbias achieves comparable performance on the glue benchmark with @xcite ; @xcite ; @xcite ; @xcite . we further decrease the parameters of adapterbias in different ways, including partial weight-sharing in adapterbias and adding l 0 -norm regularization. finally, adapter-bias has better interpretability due to its simplicity. we use different tools, including word cloud and pca @xcite , to visualize what adapter-bias has learned, and we found that the proposed approach automatically learns to assign larger representation shifts to the task-related tokens. ## related work for nlp tasks, adapters are introduced for the transformer architecture. a set of adapter parameters was added at each transformer layer, which is mostly bottleneck architectures @xcite . by keeping the output dimension identical, they cause no change to the structure or parameters of the original model. adapters quickly gained popularity in nlp with various applications. for multi-task learning @xcite @xcite , a projected self-attention layer is proposed by stickland and murray (2019 besides the applications of adapters, researchers are also dedicated to improving their performance. based on the architecture introduced by @xcite , adapterfusion @xcite leveraged knowledge from multiple tasks with a new two-stage learning algorithm. despite the recent popularity of these methods, they still train a relatively large number of training parameters. recently, studies start to focus on improving the parameter-efficiency of adaptation to a new task. diff-pruning @xcite achieves parameter efficiency by adding a sparse, task-specific difference-vector to the fixed original parameters. the vector is adaptively pruned during training with a differentiable approximation to the l 0 -norm penalty to encourage sparsity. @xcite introduced adapterdrop, which has been recently integrated into adapterhub @xcite . it removes adapters from lower transformer layers during training and inference, which can dynamically reduce the computational cost. @xcite proposed compacter, which improved the trade-off between performance and trainable parameters per task with low-rank optimization. on the other hand, without modifying the architecture of the plm, bitfit @xcite shows that fine-tuning only the bias terms of a large plm is also competitive with fine-tuning the entire model. fine-tuning only the bias terms can be considered as adding a task-specific shift to the token representation. bitfit is most similar to our work. while in bitfit, the shifts added to all the representations are exactly the same for all input tokens, in our work, the shifts are token-dependent. ## method in this section, we present adapterbias, an efficient way to adapt large-scale plms. in order to better adapt to different downstream tasks, the adapter module should be token-specific. adapterbias produces a suitable weight for the bias based on the input token. problem formulation we consider the general problem of fine-tuning plms, where the training data @xmath0, y i ) n @xmath1. assume that given a plm with parameters θ and adapterbias with parameters θ . during the training stage, we freeze θ and tune θ only. ## experiments in this section, we evaluate the effectiveness of our proposed adapter module in nlp training tasks, and provide the analysis of what adapterbias has learned in different tasks. ## conclusion in this study, we present adapterbias. by adding token-dependent representation shifts to the plm, adapterbias shows competitive results even though it uses far fewer parameters than the existing methods. through extensive experiments, not only does adapterbias reach competitive results on the glue benchmark, but also obtain good performance on small-to-medium datasets. in addition, we demonstrate the robustness of adapterbias to different plms. finally, we provide analysis on what adapterbias learns by comparing α, the weights of representation shift for different tokens, finding adapterbias has the ability to identify taskspecific information. our study is different from the previous architectures of adapters by proposing a simple adapter that can produce suitable representation shifts for different tokens. 2620 figure d: word cloud of cola in layer 0 to layer 6.