How Does In-Context Learning Help Prompt Tuning? (2024)

Abstract: Fine-tuning large language models is becoming ever more impractical due to their rapidly-growing scale. This motivates the use of parameter-efficient adaptation methods such as prompt tuning (PT), which adds a small number of tunable embeddings to an otherwise frozen model, and in-context learning (ICL), in which demonstrations of the task are provided to the model in natural language without any additional training. Recently, (CITATION) propose “instruction prompt tuning” (IPT), which combines PT with ICL by concatenating a natural language demonstration with learned prompt embeddings. While all of these methods have proven effective on different tasks, how they interact with each other remains unexplored. In this paper, we empirically study when and how in-context examples improve prompt tuning by measuring the effectiveness of ICL, PT, and IPT on five text generation tasks with multiple base language models. We observe that (1) IPT does not always outperform PT, and in fact requires the in-context demonstration to be semantically similar to the test input to yield improvements; (2) PT is unstable and exhibits high variance, but combining PT and ICL (into IPT) consistently reduces variance across all five tasks; and(3) prompts learned for a specific source task via PT exhibit positive transfer when paired with in-context examples of a different target task. Our results offer actionable insights on choosing a suitable parameter-efficient adaptation method for a given task.

## introduction as large language models (llms) continue to grow in scale @xcite , it is quickly becoming infeasible to fine-tune all of their parameters to solve a new task. as such, developing methods that efficiently adapt llms to downstream tasks is critical. in this paper, we study the relationship between three such methods: input: how many artists... output: select count(*) ... input: how many doctors... output: select count(*) ... ## background parameter-efficient fine-tuning methods @xcite @xcite specialize llms to a target task while keeping most of their parameters frozen and adjusting just a small number of task-specific parameters. since full-model fine-tuning is prohibitively expensive on consumer-grade hardware for most llms, such methods increase the accessibility of llm research and deployment. here, we give a more formal specification of the parameterefficient tuning methods that we experiment with in this paper. in-context learning: brown et al. (2020) show that their 175b-parameter gpt-3 model is capable of solving unseen tasks by leveraging information from in-context instructions (zero-shot) and/or demonstrations (few-shot). inserting k in-context input-output pairs [x icl ; y icl ] before the test input significantly improves the performance of solving a target task: prompt tuning: in-context learning struggles on out-of-domain tasks, which motivates alternate approaches that tune a small fraction of the llm's parameters @xcite . in this paper, we focus on prompt tuning (pt) @xcite , which prepends soft tunable prompt embeddings to the input tokens x test . pt is easy to implement and, unlike adapter-based @xcite and lora (hu et al., 2022) approaches, does not change the internal model structure. formally, let @xmath0, . . . , e k } be a sequence of new tunable prompt embeddings optimized over a training set, while @xmath1, . . . , x m } denote the token embeddings of the input of an example. then, the input to pt at inference time can be expressed as instruction prompt tuning. more recently, @xcite proposes instruction prompt tuning (ipt), which concatenates the soft prompts with hard in-context demonstrations. using the notation from above, the input of ipt is: note that in our experiments, the prompt embeddings e are task-specific, whereas @xcite share them across multiple tasks in the medical domain. the hybrid of soft and hard prompt tokens has been previously employed by @xcite and @xcite . ipt resembles metaicl @xcite and in-context tuning @xcite in that in-context demonstrations are part of the input during training; however, ipt tunes just the prompt embeddings. ## experimental setup how can a soft prompt benefit from the added information provided by a retrieved in-context demonstration? we run experiments comparing the performance of icl, pt, and ipt across a variety of tasks, configurations, and base language models. dataset: while past research into prompt tuning has mostly focused on natural language understanding tasks @xcite , we focus on language generation tasks, with a specific focus on tasks where either the input or output is (relatively) out-of-domain, which challenges the limits of methods for adapting llms. the tasks we explore are: dart @xcite , totto @xcite , spider (yu et al., 2018), mtop @xcite , and logic-to-text task @xcite . more details about each task are included in appendix a. ## analysis in-context learning underperforms prompt tuning: in line with experiments from prior work @xcite , we observe that icl performs consistently worse than pt and ipt, even when using retrieved demonstrations instead of random demonstrations. this result shows the value of training a small number of new parameters to specialize a language model to the target task, especially for out-of-distribution generation. the lone exception is totto, for which icl is competitive with pt. no clear winner between pt and ipt: despite receiving additional signal from the retrieved incontext demonstration, ipt does not consistently outperform pt. our results in ipt helps when the in-context demonstration is similar to the test input: to understand the effect of in-context demonstrations in ipt, we evaluate the examples grouped by the semantic similarity 2 between the input of in-context example and the input of test example. figure 2 demonstrates that pt and ipt perform worse when in-context examples are less similar to test inputs. ipt with highly-similar examples outperforms pt, but degrades when the examples become less similar. pt outperforms ipt on ood examples (right-most bin). this suggests that low-quality in-context examples can confuse the base lm, which motivates future work on dynamic methods to selectively include examples based on a similarity threshold. overlap in totto inflates ipt performance as shown in ## conclusion in this paper, we empirically analyze the effect of in-context demonstrations on prompt tuning for five language generation tasks. our experiments reveal that while instruction prompt tuning and prompt tuning perform competitively with each other, ipt is more stable, yielding lower variance when varying hyperparameters. ipt also significantly improves over pt when the in-context demonstration closely resembles the test input, which is frequently the case in the totto dataset. finally, soft prompts learned for a source task can exhibit positive transfer to new target tasks when paired with in-context demonstrations.