Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts (2023)

Abstract: Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models (PLMs). However, it is still unsettled how to generate more proper prompts for any individual examples and how to extend prompt tuning to multi-task learning scenarios by leveraging cross-task features. To address these challenges, we propose a token-wise prompt tuning (TPT), in which a bank of finer-grained soft prompt tokens is built for multi-task learning by memory network. The tokens are retrieved from the bank against an input example and assembled to an instance-dependent prompt. Extensive experimental results on 14 datasets demonstrated that the models enhanced by our TPT performed far better than full parameter fine-tuned models and achieved state-of-the-art by tuning only 0.035% parameters.

## introduction the architecture of transformers @xcite has yielded impressive performances on various natural language processing (nlp) tasks and has been widely established as the building block for plms. the dominant paradigm is to pre-train on large-scale unlabeled datasets and then fine-tune on task-related datasets @xcite @xcite . however, performing full parameter fine-tuning for each task would be prohibitively expensive with the growing model scale. thus, there has been growing interest in developing parameter-efficient fine-tuning (peft) methods @xcite @xcite that strive to achieve results comparable to full parameter finetuning with a small number of trainable parameters. prompt learning @xcite , as a new effective method of peft, can boost the model's an input text of length l, {x 1 , x 2 , • • • , x l }, is sent to a pooling module that produces a feature vector x for the input. a retrieval module retrieves the most similar prompt tokens, {t 3 , t 5 , • • • , t n }, from a prompt bank based on the similarity scores estimated between x and n soft prompt tokens {t 1 , t 2 , • • • , t n } stored in the bank. the retrieved prompt tokens are assembled to an instance-dependent prompt and concatenated with the original input. the concatenated result is sent to a frozen lm for both inference and training, in which only the retrieved prompt tokens are tuned through error back-propagation. to overcome the first limitation, we propose a novel approach to automatically generate a more suitable prompt for each input example. @xcite proposed to retrieve non-trainable tokens, also referred to as hard prompt, from the embedding layer of the language model, which can enhance both the training and inference processes of the model. compared to directly copying the fixed embedding layer as a source of extra information for each example, training an additional embedding layer on the target task can provide more appropri-ate information. therefore, we go further on this line by decomposing the trainable soft prompt into finer-grained soft prompt tokens, these tokens constitute the token-wise prompt bank, which can be viewed as a trainable embedding layer. memory network @xcite is also used to store, tune and combine these tokens, as illustrated in figure 1 . in contrast to previous methodologies that consider the soft prompt as a single whole, our approach dissects it into fine-grained prompt tokens. this refined breakdown of the prompt widens the scope of search space, facilitating a more exhaustive combination of soft tokens, and ultimately leading to the generation of superior prompts. to address the second issue, we extend the process of constructing a token-wise prompt bank (i.e., the memory network of token-wise soft prompts) to multi-task learning scenarios. there are many features that can be shared between different tasks, and these features can be learned through multitask learning @xcite . additionally, prior work by @xcite demonstrated that performing prompt tuning on intermediate tasks before doing it on the target task can yield even better results. following their recipe, we first pre-train the token-wise prompt bank across multiple source tasks, then utilize this resulting bank as initialization to train the token-wise prompt bank specifically for the target task. finally, we extend our approach, which is called token-wise prompt tuning (tpt), by combining token-wise prompt bank with task-specific prompt tuning, as illustrated in figure 2 . in our approach, all examples within a given task share a task-level prompt, which is generated by way of task-specific prompt tuning. additionally, for each individual example, an instance-level prompt is retrieved based on the similarity between the input example and tokens in the token-wise prompt bank. these two prompts are concatenated together, incorporating both instance-level and task-level features, as part of the input to facilitate model inference. moreover, extensive experimental results on 14 datasets demonstrated the effectiveness of our methods. the contribution of this study can be summarized as follows: task-dependent prompt. this line of research focuses on enhancing the generation of more effective prompts for specific target tasks. specifically, brown et al. (2020) introduced the utilization of a small set of manually crafted sentences as prompt, typically consisting of task descriptions and relevant examples. the prompt is fed to the frozen model as part of the input, offering the potential to enable the pre-trained model to achieve comparable performance to fine-tuned models, particularly when well-designed. auto-prompt (shin et al., 2020), lm-bff @xcite , and efl @xcite extend this direction by automating the process of generating discrete prompts. however, optimizing prompts within discrete spaces presents challenges and is likely to be sub-optimal. prompt tuning @xcite , prefix tuning @xcite , and p-tuning @xcite adopt an alternative strategy by introducing continuous vectors, known as soft prompt, in front of the input sequence. only these continuous vectors need to be adjusted during training, so the optimization problem in discrete spaces is converted to a continuous optimization task, which can be handled through simple gradient descent. moreover, later work has started to consider prompt tuning in the transfer learning scenario. @xcite and spot @xcite explore the transferability of prompts learned from different tasks, and address the sensitivity of prompt tuning to initialization through transfer learning. @xcite and panda @xcite proposed to learn a transfer prompt on the source tasks through knowledge distillation. instance-dependent prompt. this line of research takes into account the individual characteristics of different examples and generates distinct prompts tailored to each specific example. in particular, @xcite and idpg @xcite generate instance-wise prompts via multi-layer perceptions (mlps) based on the input encoded by the language model. @xcite and @xcite maintain a prompt pool to store the prompts learned over source tasks, where each prompt is classified into specific categories and assigned key vectors. the encoded input serves as the query vector, and the target prompts are obtained by weighing the prompts in the prompt pool according to the results of the query and key vector calculations. in addition, attempt @xcite calculates weights simply based on the similarity between the input and the prompts learned from the source tasks, without the need for pre-computed clusters. unlike these approaches that weighted the soft prompt as a whole, we instead utilize the finergrained prompt tokens for combination, and only the tokens retrieved according to the input receive the gradient during training. therefore, a more appropriate prompt can be generated for each example. @xcite proposed to use retrieved non-trainable hard prompt as a prefix to guide the training of the prompt. in contrast, our method is to retrieve the trainable soft prompt and can be extended to scenarios of multi-task learning to incorporate cross-task features. unlike ft, prompt tuning freezes the pre-trained language model and only needs to train a very small number of parameters. specifically, it prepends m randomly initialized vectors, also known as soft prompt @xmath0, p 2 , • • • , p m }, where p @xmath1, before the input x i , the optimization goal of prompt tuning as follows: prompt transfer. transfer learning methods attempt to learn a new target task given a collection of source tasks t @xmath2, t 2 , • • • , t t }, which have been a long-standing way to improve the effectiveness and efficiency of nlp systems @xcite . recent studies such as @xcite have demonstrated the applicability of transfer learning in the context of prompt tuning, also referred to as prompt transfer. instead of training the prompt from scratch over target task, these approaches employ the source prompts p @xmath3, p 2 , • • • , p t }, which are trained according to equation ( 2 ) over source tasks. these source prompts p source can then serve as either initialization vectors or weighted vectors for training the target prompt. ## method our proposed method tpt (as illustrated in figure 2 ) consists of two stages: pre-training token-wise prompt bank (section 4.1) and jointly prompt tuning (section 4.2). tpt pre-trains a token-wise prompt bank that integrates the cross-task features on various source tasks t @xmath4, t 2 , • • • , t t } and then utilizes this resulting bank as the initial token-wise prompt bank of the next stage to generate instance-level retrieved prompt for each example. in addition, all examples of the target task t target also share the same task-level soft prompt. these two kinds of prompts are concatenated with the input as the final input of the frozen lm and provide both additional instance-level and task-level features for that input, which enhance the model's training and inference processes, leading to improved performance. ## experiments following the previous prompt-based methods @xcite , we perform our experiments on 14 different datasets with fulldataset and few-shot settings, and the experimental results show the effectiveness of our tpt in various scenarios. ## conclusions in this study, we have introduced tpt, a novel parameter-efficient fine-tuning method designed to address the challenges of generating more suitable prompts for individual examples and extending prompt tuning to multi-task learning scenarios for capturing cross-task features. tpt harnesses the power of a memory network to construct a finer-