Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering (2024)

Abstract: Prompt tuning, which freezes all parameters of a pre-trained model and only trains a soft prompt, has emerged as a parameter-efficient approach. For the reason that the prompt initialization becomes sensitive when the model size is small, the prompt transfer that uses the trained prompt as an initialization for the target task has recently been introduced. Since previous works have compared tasks in large categories (e.g., summarization, sentiment analysis), the factors that influence prompt transfer have not been sufficiently explored. In this paper, we characterize the question answering task based on features such as answer format and empirically investigate the transferability of soft prompts for the first time. We analyze the impact of initialization during prompt transfer and find that the train dataset size of source and target tasks have the influence significantly. Furthermore, we propose a novel approach for measuring catastrophic forgetting and investigate how it occurs in terms of the amount of evidence. Our findings can help deeply understand transfer learning in prompt tuning.

## introduction advances in large language models (llms) @xcite @xcite have continued to be made since the advent of the transformer @xcite . as llms grow larger and larger, prompt tuning @xcite is introduced to reduce the computational costs. this approach, which freezes all parameters of a pre-trained model and only trains a soft prompt, requires updating fewer parameters than fine-tuning while achieving comparable performance in many natural language processing (nlp) systems. however, especially in model sizes below 11b parameters, prompt initialization causes performance differences. recently, prompt transfer (pot) was proposed in @xcite as a way to better initialize prompts, in which a prompt embedding trained for a source task is used for initialization before training the target prompt. tpt task @xcite claims that the performance is effective when initialized with the best zero-shot prompt. several studies modified the prompt learning process to improve performance @xcite @xcite @xcite . the effectiveness of these approaches achieves better or comparable performance with prompt tuning and fine-tuning. nevertheless, previous studies unexplored the factors influencing transferability and only focused on large categories of tasks. therefore, our goal is not only to refine the categorization of question answering (qa) tasks but also to investigate the impact on prompt transferability. our study is the first to examine pot across qa datasets, and we report four important findings: ## results and analysis to study the transferability of soft prompts, we used 16 qa datasets as the source and target tasks. the main terms referred to in this section are as follows: (1) vanilla prompt tuning (vanilla pt), the result of training the prompt in equation 1 after random initializing; (2) zero-shot performance, the result of solving the target task using the source prompt without additional training; and (3) prompt transfer (pot), the result of initializing the target prompt with the selected source prompt and training it as shown in equation 1. for our experiments, we used the t5 base foot_0 as our base lm. further experimental details are in appendix b. ## conclusion in this paper, we study pot in the qa task. in particular, we empirically investigate prompt initialization, demonstrating that the difference of train dataset size between source and target tasks is affecting the transferability. we also define a novel method to measure catastrophic forgetting and show that there is a relationship between the amount of evidence in qa datasets and the tendency of catastrophic forgetting. finally, our finegrained analyses provide meaningful insights to help improve the performance of pot.