Parameter-Efficient Tuning with Special Token Adaptation (2023)

Abstract: Parameter-efficient tuning aims at updating only a small subset of parameters when adapting a pretrained model to downstream tasks. In this work, we introduce PASTA, in which we only modify the special token representations (e.g., [SEP] and [CLS] in BERT) before the self-attention module at each layer in Transformer-based models. PASTA achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with up to only 0.029% of total parameters trained. Our work not only provides a simple yet effective way of parameter-efficient tuning, which has a wide range of practical applications when deploying finetuned models for multiple tasks, but also demonstrates the pivotal role of special tokens in pretrained language models.

## introduction built upon a pretrained language model (plm; @xcite @xcite , many of the recent nlp systems are developed based on task-specific finetuning. in this way, the plm effectively leverages the task-agnostic knowledge captured during selfsupervised pretraining and adapts itself to downstream tasks. however, full finetuning poses a challenge to model deployment under multi-task, memory-limited scenarios, where we need to train and store a separate full-sized model for each substantially distinct task. as an alternative, parameterefficient tuning @xcite aims at only updating a small number of parameters when adapting plms to downstream tasks while making most of the model parameters fixed and shared among tasks, thus reducing memory usage. in this paper, we propose parameter-efficient tuning with special token adaptation (pasta), * work done when visiting usc. 1 our code is publicly available at: https://github. com/luka-group/pasta/ l5-h3 l5-h5 l5-h6 l5-h7 l20-h3 l20-h5 l20-h8 l20-h15 figure 1 : examples of vertical attention heads in the 5th and 20-th layer of bert-large with a random sample from cola @xcite as input. heads in the first row and second row assign most of maximal attention weights to [cls] and [sep] respectively. see appx. §c for the full attention map. ## related work a recent survey @xcite categorizes three types of parameter-efficient tuning methods. addition methods @xcite @xcite introduce a small number of additional trainable parameters while keeping those in the plm unchanged. specification methods @xcite @xcite update a portion of parameters in the plm while keeping others frozen. reparameterization methods @xcite @xcite modify plms' structures to parameter-efficient forms. our method belongs to the addition-based methods and follows the basic settings of p-tuning v2 @xcite , where newly initialized hidden representations of tokens are inserted into each transformer layer. different from most prompt tuning methods that introduce new tokens, we add the introduced vectors to the hidden states of special tokens and keep the sequence length unchanged. previous works use probing tasks @xcite and pruning methods @xcite to study the roles of different modules inside bert. it has been shown that functional specialization exists in bert self-attention heads @xcite , and vertical attention heads 3 take up a large portion @xcite . @xcite al. (2019 ) find that vertical attention heads are almost exclusively associated with attention to [sep] or [cls] tokens, and clark et al. (2019) conclude that heads in early layers often attend to [cls] while in middle layers attend to [sep]. in this work, we demonstrate that adapting hidden representations of special tokens is sufficient to bring the multi-head attention h([cls]) sentence1 frozen trainable e(𝑣 𝟏 𝒍 ) h([sep]) sentence2 h([sep]) e(𝑣 𝟐 𝑙 ) e(𝑣 𝟑 𝑙 ) add & norm feed-forward network add & norm performance of plms to the level of full finetuning. ## pasta given a large plm, our goal is to develop a parameter-efficient tuning method that only updates a small set of parameters when adapting to a downstream task. to this end, we propose a simple yet effective method called pasta, in which we train a hidden vector for every special token at each transformer layer, along with a task-specific classifier, while freezing the parameters of the plm. ## experiments and results we hereby study the downstream performance of pasta and analyze the properties of introduced hidden vectors. ## conclusion we present pasta, a parameter-efficient tuning method that only modifies special token representations at each transformer layer when adapting to downstream tasks. our approach is motivated by the observation that plms have a large amount of vertical attention heads that heavily attend to special tokens, and these heads disseminate value updates from special tokens to all of the other tokens. experiments show that pasta achieves strong per-formance comparable to full finetuning on sentential and lexical tasks with high parameter efficiency. our work not only provides an effective solution for parameter-efficient tuning, but also demonstrates the pivotal role of special tokens in plms.