Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022)

Abstract: Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks by adding and tuning a small number of new parameters. Previously proposed adapter architectures are all feed-forward neural networks. In this paper, we investigate the effectiveness of using tiny-attention—i.e., attention with extremely small per-head dimensionality—as adapters. Our tiny-attention adapter learns to modify the hidden states at each position directly conditioned on the hidden states at all the other positions, which is missed by the previously proposed adapters. Moreover, we view its multiple attention heads as a mixture of experts and propose to average their weights during deployment, which further reduces its inference computation cost. On the GLUE benchmark, our tiny-attention adapter outperforms the other parameter-efficient transfer learning methods as well as full fine-tuning while only updating 0.05% of the parameters. On the FewGLUE benchmark, its performance is comparable to that of GPT-3 and PET.

## introduction transferring a large pretrained language model (plm) is a de facto paradigm of performing downstream tasks in natural language processing (nlp). a general approach is adapter-tuning, which means inserting adapters-i.e., neural networks with small numbers of parameters-into each pretrained layer and only updating the adapter parameters. adapter-tuning is parameter-efficient and enjoys low computation cost since it keeps the plm frozen. but it underperforms full fine-tuning which updates all the parameters of the plm. in this paper, we propose a new adapter architecture which outperforms full fine-tuning yet uses even fewer parameters than the previously proposed adapters as well as other parameter-efficient transfer learning methods; see figure 1 . finetuning 88.9 adapter h 87.8 figure 1 : average performance of different parameterefficient transfer learning methods on the glue benchmark with roberta-large as the plm. our method is tiny-attn (with marker): "1h" and "4h" mean "one attention head" and "four attention heads (with the parameteraveraging trick in section 2.2)" respectively. our adapter is a multi-head attention module and its per-head dimensionality is extremely small; thus we call it tiny-attention. the architecture design is inspired by the following intuitions: therefore, we propose to use the attentative structure that allows the embedding modifications of each token to capture more contextual information by directly looking at the embeddings of all the tokens. the dimensionality of each attention head need not to be large. in nlp tasks, contextual information is often demonstrated more important than model sizes: for example, in language modeling, a smaller model with a larger context window usually outperforms a larger model with a smaller context window @xcite @xcite . additionally, we view the multiple attention heads of our tiny-attention adapter as a mixture of experts and then propose to average their weights during inference. this technique further reduces the inference cost. we evaluated our tiny-attention adapter on the glue @xcite and fewglue @xcite benchmarks. on glue, it updates only 0.05% of the parameters-an order of magnitude smaller than all the other methods-yet still outperforms full finetuning and nearly all the other parameter-efficient tuning methods. on fewglue, it is comparable to several strong competing methods including pet @xcite and gpt-3 @xcite . we also conducted ablation studies to investigate its effectiveness with varying placements and plms. ## the method figure 2a illustrates how a language model performs a downstream task. the language model has l layers. given an input sequence @xmath0. . . x t where x 0 is a special classification (cls) token, each layer ℓ reads the embeddings given by the previous layer ℓ -1 and produces the layer-ℓ embeddings h t . then a task-specific decoderreads the top-layer embedding h (l) 0 of the cls token and predicts a task-specific output ŷ for that sequence. transferring the language model involves updating its trainable parameters to minimize a taskspecific loss(ŷ, y) where y is the ground-truth label for the given x. for adapter-tuning, the trainable parameters only include those of the task-specific decoder and those of the adapters. figure 2b shows a language model layer with our tiny-attention adapter placed between its attention module and feed-forward net: during training, only the decoder and adapter parameters (blue) are updated while the pretrained parameters (green) are all kept frozen. ## related work there are three major paradigms of parameterefficient transfer learning. the first is to only finetune a small subset of the existing parameters in a pretrained language model @xcite @xcite . the second is adapter-tuning @xcite : inserting adapters (i.e., small neural nets) into the language model and only tuning their parameters. the third is prefix-tuning @xcite @xcite adamix @xcite proposes a stochastic routing strategy to mix an ensemble of adapters and it is orthogonal to all the adapter methods including ours. akin to our parameter-averaging trick, they propose to average the adapter parameters for low-cost storage and inference. similar tricks are used in @xcite ; @xcite ; @xcite . ## experiments we evaluated our proposed tiny-attention adapter on a range of natural language understanding tasks including glue and fewglue. our method is implemented in pytorch @xcite and heavily relies on huggingface @xcite . our code is submitted for review and it will be publicly released after the paper is published. in all of our experiments, we set the dimensionality of each tiny-attention head (i.e., the dimension of query, key, and value vectors) to be one. other experiment details (e.g., hyperparameters) can be found in appendix b. ## conclusion in this paper, we presented the tiny-attention adapter. while previous adapter-tuning only processes in-place embeddings, our method considers the context from other positions. thanks to this contextual modeling, the size of our tiny-attention adapter can be extremely light (e.g., only 1 attention head and 1 head dimension). to further enable trade-off between performance and training cost, we proposed the weight-averaging technique. on glue benchmark, our tiny-attention adapter achieved better results than full fine-tuning while only updating 0.05% of parameters. our model also achieved competitive results under the fewshot setting. lastly, we compared our methods with the alternatives (e.g., parallel instead of sequential, without parameter averaging) and also showed the generalization to smaller plms.