SP o T : Better Frozen Model Adaptation through Soft Prompt Transfer (2022)

Abstract: There has been growing interest in parameter-efficient methods to apply pre-trained language models to downstream tasks. Building on the Prompt Tuning approach of Lester et al. (2021), which learns task-specific soft prompts to condition a frozen pre-trained model to perform different tasks, we propose a novel prompt-based transfer learning approach called SPoT: Soft Prompt Transfer. SPoT first learns a prompt on one or more source tasks and then uses it to initialize the prompt for a target task. We show that SPoT significantly boosts the performance of Prompt Tuning across many tasks. More remarkably, across all model sizes, SPoT matches or outperforms standard Model Tuning (which fine-tunes all model parameters) on the SuperGLUE benchmark, while using up to 27,000× fewer task-specific parameters. To understand where SPoT is most effective, we conduct a large-scale study on task transferability with 26 NLP tasks in 160 combinations, and demonstrate that many tasks can benefit each other via prompt transfer. Finally, we propose an efficient retrieval approach that interprets task prompts as task embeddings to identify similar tasks and predict the most transferable source tasks for a novel target task.

## introduction the past few years have seen the rapid development of ever larger pre-trained language models, where it has repeatedly been shown that scaling up the model size is a key ingredient for achieving the best performance @xcite @xcite . while this trend has continued to push the boundaries of possibility across various nlp benchmarks, the sheer size of these models presents a challenge for their practical application. for 100b+ parameter models, fine-tuning and deploying a separate instance work done during an internship at google research. superglue score p d s (gpt-3) m t p t m -s m t sp t (ours) figure 1: our spot approach-which transfers a prompt learned from a mixture of source tasks (here, glue) onto target tasks-outperforms vanilla promt-tuning @xcite and gpt-3 @xcite on superglue by a large margin, matching or outperforming modeltuning across all model sizes. at the xxl model size, spot even outperforms multi-taskmodeltuning, which fine-tunes the entire model on the glue mixture before fine-tuning it on individual superglue tasks. see appendix a for full results. more recent work explores methods for learning soft prompts @xcite @xcite , which can be seen as additional learnable parameters injected into the language model. @xcite propose prompttuning, a simple method we learn a single generic source prompt on one or more source tasks, which is then used to initialize the prompt for each target task. right: we learn separate prompts for various source tasks, saving early checkpoints as task embeddings and best checkpoints as source prompts. these form the keys and values of our prompt library. given a novel target task, a user: (i) computes a task embedding, (ii) retrieves an optimal source prompt, and (iii) trains a target prompt, initialized from the source prompt (see §3 for details). in this paper, we propose spot: soft prompt transfer, a novel transfer learning approach in the context of prompt tuning. spot first trains a prompt on one or more source tasks, and then uses the resulting prompt to initialize the prompt for a target (downstream) task. our experiments show that spot offers significant improvements over prompt-tuning across tasks and model sizes. for instance, on the superglue benchmark @xcite , we obtain +10.1 and +2.4 point average accuracy improvements using the t5 base (220m parameter) and t5 xxl (11b parameter) models @xcite , respectively. more importantly, spot is competitive with or outperforms modeltuning across all model sizes (see figure 1 ). motivated by these results, we investigate transferability between tasks, through the lens of soft task prompts. our goal is to answer two questions: (a) for a given target task, when does initializing the prompt from a source task boost performance? (b) can we use task prompts to efficiently predict which source tasks will transfer well onto a novel target task? to answer (a), we conduct a systematic study of the t5 model using 26 nlp tasks in 160 combinations of source and target tasks. our results indicate that many tasks can benefit each other via prompt transfer. to address (b), we interpret the learned task prompts as task embeddings to construct a semantic space of tasks and formalize the similarity between tasks. we design an efficient retrieval algorithm that measures task embedding similarity, allowing practitioners to identify source tasks that will likely yield positive transfer. to summarize, our main contributions are: (1) we propose spot, a novel prompt-based transfer learning approach, and show that scale is not necessary for prompttuning to match the performance of modeltuning; on superglue, spot matches or beats modeltuning across all model sizes. (2) we conduct a large-scale and systematic study on task transferability, demonstrating conditions under which tasks can benefit each other via prompt transfer. (3) we propose an efficient retrieval method that interprets task prompts as task embeddings to construct a semantic space of tasks, and measures task embedding similarity to identify which tasks could benefit each other. (4) to facilitate future work on prompt-based learning, we will release our library of task prompts and pretrained models, and provide practical recommendations for adapting our library to nlp practitioners at https://github.com/google-research/ prompt-tuning/tree/main/prompt_tuning/ spot . ## improving prompttuning with spot to improve performance of prompttuning on a target task, spot introduces source prompt tuning, an intermediate training stage between language model pre-training and target prompt tuning (figure 2 , left), to learn a prompt on one or more source tasks (while still keeping the base model frozen), which is then used to initialize the prompt for the target task. 1 our approach retains all the computational benefits of prompttuning: for each target task, it only requires storing a small task-specific prompt, enabling the reuse of a single frozen pretrained model across all tasks. in this section, we present a generic spot approach where a single transferred prompt is reused for all target tasks. in §3, we explore a targeted approach that retrieves different source prompts for different target tasks. ## predicting task transferability so far, we have seen that soft prompt transfer can significantly boost the performance of prompt tuning, but it is critical to pick the right source tasks for transfer. for instance, through an extensive search, we found that glue and mnli provide excellent source tasks for transferring to individual glue and superglue tasks. but what about a resourceconstrained scenario where a user is not able to exhaustively search over a set of source tasks? can we predict which tasks will best transfer onto a novel target task without testing them one by one? to investigate this, we conduct a large-scale empirical study with 26 nlp tasks. we first measure transferability across all task combinations ( §3.1). next, we show that by interpreting task prompts as task embeddings, we can construct a semantic space of tasks, wherein similar tasks cluster together ( §3.2). based on this observation, we propose a retrieval algorithm ( §3.3) that leverages task embedding similarity to choose which source tasks to use for a given novel target task (figure 2 , right). our proposed approach can eliminate 69% of the source task search space while keeping 90% of the best-case quality gain. ## related work parameter-efficient transfer learning: largescale pre-trained language models have been shown 16 see appendix g for full results. to exhibit remarkable performance on many nlp tasks @xcite @xcite @xcite @xcite . to improve practical applicability of these models, early work introduces compression techniques @xcite @xcite to obtain lightweight models. other work explores updating only small parts of the model @xcite or task-specific modules, such as adapters @xcite or low-rank structures @xcite , while keeping the rest of the model fixed. recently, @xcite demonstrate impressive few-shot performance with promptdesign, where their model is conditioned on a manual text prompt at inference time to perform different tasks. several efforts have since focused on developing prompt-based learning approaches with carefully handcrafted prompts @xcite , prompt mining and paraphrasing @xcite , gradient-based search for improved prompts @xcite , and automatic prompt generation @xcite . the use of hard prompts, however, was found to be sub-optimal and sensitive to the choice of the prompt @xcite . as such, more recent work has shifted toward learning soft prompts @xcite @xcite , which can be seen as learnable parameters injected into the model. we refer readers to @xcite for a recent survey on prompt-based learning research. in concurrent work, @xcite also explore the effectiveness of prompt transfer. their method uses hand-crafted pre-training tasks tailored to specific types of downstream tasks, being less extensible to novel downstream tasks. in contrast, we use existing tasks as source tasks and show that prompt transfer can confer benefits even when there are mismatches (e.g., in task type or input/output format) between the source and target. task transferability we also build on existing work on task transferability @xcite @xcite @xcite . prior work shows effective transfer from data-rich source tasks @xcite , those that require complex reasoning and inference @xcite , or those that are similar to the target task @xcite . there have also been efforts to predict task transferability @xcite @xcite . @xcite use task embeddings derived from either the input text or the diagonal fisher information matrix of the model, while @xcite explore adapterbased alternatives. here, our use of the same model (without task-specific components) with a unifying text-to-text format allows us to more easily model the space of tasks. additionally, prompt-based task embeddings are comparatively cheaper to obtain. ## limitations & future work as other parameter-efficient adaptation methods (see §4) may outperform prompttuning in specific situations, it would be interesting to test whether an approach similar to spot could extend successfully to these methods. at the same time, we believe that prompttuning has its own merits. as pre-trained language models become larger and larger, some advantages of prompttuning over other methods are: (1) among current methods with learnable parameters, prompttuning is the most parameter efficient, requiring less than 0.01% task-specific parameters for most model sizes. (2) prompttuning is simpler than other methods, as it does not modify the internal model architecture (cf. the prefix-tuning method of li and liang (2021), which adds a prefix to each layer of both the transformer encoder and decoder); as such, prompttuning allows mixed-task inference and facilitates transfer learning between tasks. (3) as model capacity increases, prompttuning becomes more competitive with modeltuning; to the best of our knowledge, this has not been shown for other methods. (4) soft prompts could possibly be interpreted as natural language instructions. additionally, since our prompt-based task embedding approach does not capture all of the factors that influence task transferability, we leave further exploration of other task embedding methods to future work. ## conclusion in this paper, we study transfer learning in the context of prompt tuning. we show that scale is not necessary for prompttuning to match the performance of modeltuning. on superglue, our spot approach matches or even exceeds the performance of modeltuning by a large margin across model sizes while being more parameter-efficient. our large-scale study on task transferability indicates that tasks can benefit each other via prompt transfer in various scenarios. finally, we demonstrate that task prompts can be interpreted as task embeddings to formalize the similarity between tasks. we propose a simple yet efficient retrieval approach that measures task similarity to identify which source tasks could confer benefits to a novel target task. taken as a whole, we hope that our work will spur more research into prompt-based transfer learning. appendices a full results for figure 1