The Power of Scale for Parameter-Efficient Prompt Tuning (2021)

Abstract: In this work, we explore “prompt tuning,” a simple yet effective mechanism for learning “soft prompts” to condition frozen language models to perform specific downstream tasks. Unlike the discrete text prompts used by GPT-3, soft prompts are learned through backpropagation and can be tuned to incorporate signals from any number of labeled examples. Our end-to-end learned approach outperforms GPT-3’s few-shot learning by a large margin. More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method “closes the gap” and matches the strong performance of model tuning (where all model weights are tuned). This finding is especially relevant because large models are costly to share and serve and the ability to reuse one frozen model for multiple downstream tasks can ease this burden. Our method can be seen as a simplification of the recently proposed “prefix tuning” of Li and Liang (2021) and we provide a comparison to this and other similar approaches. Finally, we show that conditioning a frozen model with soft prompts confers benefits in robustness to domain transfer and enables efficient “prompt ensembling.” We release code and model checkpoints to reproduce our experiments.

## introduction with the wide success of pre-trained large language models, a range of techniques has arisen to adapt these general-purpose models to downstream tasks. elmo @xcite proposed freezing the pre-trained model and learning a task-specific weighting of its per-layer representations. however, since gpt @xcite performance, but requires storing separate copies of the model for each end task. our prompt tuning of t5 matches the quality of model tuning as size increases, while enabling the reuse of a single frozen model for all tasks. our approach significantly outperforms fewshot prompt design using gpt-3. we show mean and standard deviation across 3 runs for tuning methods. @xcite , the dominant adaptation technique has been model tuning (or "fine-tuning"), where all model parameters are tuned during adaptation, as proposed by @xcite . more recently, @xcite showed that prompt design (or "priming") is surprisingly effective at modulating a frozen gpt-3 model's behavior through text prompts. prompts are typically composed of a task description and/or several canonical examples. this return to "freezing" pre-trained models is appealing, especially as model size continues to increase. rather than requiring a separate copy of the model for each downstream task, a single generalist model can simultaneously serve many different tasks. unfortunately, prompt-based adaptation has several key drawbacks. task description is error-prone and requires human involvement, and the effectiveness of a prompt is limited by how much condition- pre-trained model (11b params) task a model (11b params) task b model (11b params) task c model (11b params) a1 a2 b1 c1 c2 task a batch task b batch task c batch pre-trained model (11b params) model tuning prompt tuning a b c mixed-task batch (20k params each) a1 c1 b1 a2 c2 a c b a c task prompts figure 2 : model tuning requires making a taskspecific copy of the entire pre-trained model for each downstream task and inference must be performed in separate batches. prompt tuning only requires storing a small task-specific prompt for each task, and enables mixed-task inference using the original pretrained model. with a t5 "xxl" model, each copy of the tuned model requires 11 billion parameters. by contrast, our tuned prompts would only require 20,480 parameters per task-a reduction of over five orders of magnitude-assuming a prompt length of 5 tokens. in this paper, we propose prompt tuning as a further simplification for adapting language models. we freeze the entire pre-trained model and only allow an additional k tunable tokens per downstream task to be prepended to the input text. this "soft prompt" is trained end-to-end and can condense the signal from a full labeled dataset, allowing our method to outperform few-shot prompts and close the quality gap with model tuning (figure 1 ). at the same time, since a single pre-trained model is recycled for all downstream tasks, we retain the efficient serving benefits of frozen models (figure 2 ). while we developed our method concurrently with li and liang (2021) and @xcite , we are the first to show that prompt tuning alone (with no intermediate-layer prefixes or task-specific output layers) is sufficient to be competitive with model tuning. through detailed experiments in sections 2-3, we demonstrate that language model capacity is a key ingredient for these approaches to succeed. as figure 1 shows, prompt tuning becomes more competitive with scale. we compare with similar approaches in section 4. explicitly separating task-specific parameters from the "generalist" parameters needed for general language-understanding has a range of additional benefits. we show in section 5 that by capturing the task definition in the prompt while keeping the generalist parameters fixed, we are able to achieve better resilience to domain shifts. in section 6, we show that "prompt ensembling", learning multiple prompts for the same task, can boost quality and is more efficient than classic model ensembling. finally, in section 7, we investigate the interpretability of our learned soft prompts. in sum, our key contributions are: proposing prompt tuning and showing its competitiveness with model tuning in the regime of large language models. 2. ablating many design choices, and showing quality and robustness improve with scale. 3. showing prompt tuning outperforms model tuning on domain shift problems. 4. proposing "prompt ensembling" and showing its effectiveness. ## prompt tuning following the "text-to-text" approach of t5 @xcite , we cast all tasks as text generation. instead of modeling classification as the probability of an output class given some input, pr(y|x), where x is a series of tokens and y is a single class label, we now model it as conditional generation, where y is a sequence of tokens that represent a class label. t5 models classification as pr θ (y |x), parameterized by the weights, θ, of the transformers @xcite that make up its encoder and decoder. prompting is the approach of adding extra information for the model to condition on during its generation of y . normally, prompting is done by prepending a series of tokens, p , to the input x, such that the model maximizes the likelihood of the correct y , pr θ (y |[p ; x]), while keeping the model parameters, θ, fixed. in gpt-3, the representations of the prompt tokens, @xmath0, p 2 , . . . , p n }, are part of the model's embedding table, parameterized by the frozen θ. finding an optimal prompt thus requires the selection of prompt tokens, through either manual search or non-differentiable search methods @xcite . prompt tuning removes the restriction that the prompt p be parameterized by θ; instead the prompt has its own dedicated parameters, θ p , that can be updated. while prompt design involves selecting prompt tokens from a fixed vocabulary of frozen embeddings, prompt tuning can be thought of as using a fixed prompt of special tokens, where only the embeddings of these prompt tokens can be updated. our new conditional generation is now pr θ;θ p (y |[p ; x]) and can be trained by maximizing the likelihood of y via backpropagation, while only applying gradient updates to θ p . given a series of n tokens, {x 1 , x 2 , . . . , x n }, the first thing t5 does is embed the tokens, forming a matrix x @xmath1. our soft-prompts are represented as a parameter p @xmath2, where p is the length of the prompt. our prompt is then concatenated to the embedded input forming a single matrix [p e ; x e ] ∈ r (p+n)×e which then flows though the encoder-decoder as normal. our models are trained to maximize the probability of y , but only the prompt parameters p e are updated. ## results our frozen models are built on top of pre-trained t5 checkpoints of all sizes (small, base, large, xl, xxl). we leverage the public t5 1.1 checkpoints, which include improvements over the original t5. 2 our "default" configuration, plotted with a green '×' ( ) throughout, uses an lm-adapted version of t5 trained for an additional 100k steps, initializes using class labels (see section 3.2), and uses a prompt length of 100 tokens. while this is longer than the default 10-token prefix used by li and liang (2021), our method still uses fewer task-specific parameters, as we only tune the input layer, as opposed to overwriting activations in all network layers. see figure 4 for a detailed comparison. we will also see shortly that even much shorter prompts are viable as model size increases. we measure performance on the superglue benchmark @xcite , a collection of eight challenging english language understanding tasks. 3 we report metrics on the development set associated with each dataset. each of our prompts train on a single super-glue task; there was no multi-task setup or mixing of training data across tasks. we translate each superglue dataset into a text-to-text format following @xcite , except that we omit the task names prepended to inputs indicating which superglue task an example belongs to. we train our prompts for 30,000 steps using t5's standard cross-entropy loss, with a constant learning rate of 0.3 and a batch size of 32. checkpoints are selected via early stopping on the development set, where the stopping metric is the default metric for the dataset, or the average of metrics for datasets evaluated with multiple metrics. all experiments were run in jax @xcite using the adafactor optimizer @xcite with weight decay 1e-5, β 2 decay 0.8, and parameter scaling off. the models were implemented in flax @xcite . more details are available in appendix a. ## comparison to similar approaches in this section, we review recent work on learning continuous prompts, and draw comparisons with our method. one important axis of comparison is the number of task-specific parameters each method requires, as shown in figure 4 . among methods with learnable parameters, prompt tuning is the most parameter efficient, requiring less than 0.01% task-specific parameters for models over a billion parameters. 9 li and liang (2021) propose "prefix tuning": learning a sequence of prefixes that are prepended at every transformer layer. this is akin to learning transformer activations that are fixed across exam-ples at every network layer. in contrast, prompt tuning uses a single prompt representation that is prepended to the embedded input. beyond requiring fewer parameters, our approach allows the transformer to update the intermediate-layer task representations, as contextualized by an input example. their work builds on gpt-2 @xcite and bart @xcite , while ours focuses on t5 and examines changes in performance and robustness to design choices as model size increases. when using bart, prefix tuning includes prefixes on both the encoder and decoder network, while prompt tuning only requires prompts on the encoder. li and liang (2021) also rely on a reparameterization of the prefix to stabilize learning, which adds a large number of parameters during training, whereas our configuration does not require this reparameterization and is robust across superglue tasks and model sizes. @xcite propose "warp", where prompt parameters are added to the input layer. this method works with masked language models, relying on a [mask] token and a learnable output layer to project the mask to class logits. this formulation restricts the model to producing a single output, limiting it to classification. prompt tuning does not require any changes to the input or a task-specific head. the performance of prompt tuning is also considerably closer to the strong performance of model tuning. @xcite propose "p-tuning" where learnable continuous prompts are interleaved throughout the embedded input, using patterns based on human design. our approach removes this complication by simply prepending the prompt to the input. to achieve strong superglue results, p-tuning has to be used in conjunction with model tuning, that is, models jointly update both the prompt and the main model parameters, whereas our approach keeps the original language model frozen. 10 qin and eisner (2021) use "soft words" to learn prompts to extract knowledge from pre-trained lms. prompts are positioned in relation to the input based on hand-designed prompt prototypes, and a learned ∆ ℓ i parameter is included for each layer, so parameter cost scales with model depth. @xcite use a learnable prepended token to adapt transformer models to var10 as another difference, p-tuning requires the addition of "anchor" tokens in the input (e.g. a question mark following the hypothesis in the rte task) to achieve strong performance, while prompt tuning leaves inputs untouched. dataset domain model prompt ∆ squad wiki 94.9 ±0.2 94.8 ±0.1 -0.1 textbookqa book 54.3 ±3.7 66.8 ±2.9 +12.5 bioasq bio 77.9 ±0.4 79.1 ±0.3 +1.2 race exam 59.8 ±0.6 60.7 ±0.5 +0.9 re wiki 88.4 ±0.1 88.8 ±0.2 +0.4 duorc movie 68.9 ±0.7 67.7 ±1.1 -1.2 drop wiki 68.9 ±1.7 67.1 ±1.9 -1.8 their base models are small transformers trained from scratch jointly with the task representations, whereas we keep the base model frozen and investigate scaling laws using larger transformers. more generally, work on task prompts is closely aligned with work on "adapters" @xcite , small bottleneck layers inserted between frozen pre-trained network layers. adapters offer another means of reducing task-specific parameters, with @xcite achieving glue performance close to full model tuning when freezing bert-large and only adding 2-4% additional parameters. @xcite use multiple adapters in a multilingual context to explicitly separate language understanding from task specification, similar to our approach. a core difference between adapters and prompt tuning is how the approaches change model behavior. adapters modify the actual function that acts on the input representation, parameterized by the neural network, by allowing the rewriting of activations at any given layer. prompt tuning modifies behavior by leaving the function fixed and adding new input representations that can affect how subsequent input is processed. ## resilience to domain shift by freezing the core language model parameters, prompt tuning prevents the model from modifying its general understanding of language. instead, prompt representations indirectly modulate the representation of the input. this reduces the model's ability to overfit to a dataset by memorizing specific lexical cues and spurious correlations. this restriction suggests that prompt tuning may improve robustness to domain shifts, where the distribution of inputs differs between training and evaluation. we investigate zero-shot domain transfer on two tasks: question answering (qa) and paraphrase detection. for question answering, we use the mrqa 2019 shared task on generalization @xcite . this task collects extractive qa datasets in a unified format and tests how models trained on "in-domain" datasets perform when evaluated on "out-of-domain" datasets. for our experiments, we train on squad @xcite and evaluate on each of the out-of-domain datasets. 11 as a second test of robustness to domain shift, we explore transfer between two paraphrase detection tasks from glue @xcite . the first task is qqp @xcite , which asks if two questions from the community q&a site quora are "duplicates". the second task is mrpc @xcite , which asks if two sentences drawn from news articles are paraphrases. we test transfer in both directions (qqp ⇔ mrpc). as before, we train on the "in-domain" task, select checkpoints using in-domain validation, and evaluate zero-shot on the "out-of-domain" task. ## prompt ensembling ensembles of neural models trained from different initializations on the same data are widely observed to improve task performance @xcite and are useful for estimating model uncertainty @xcite . however, as model size increases, ensembling can become impractical. beyond the space required to store n models (e.g. 42 gib for each copy of t5-xxl), there is a substantial inference cost to running n distinct models, whether in parallel or in series. prompt tuning provides a more efficient way to ensemble multiple adaptations of a pre-trained language model. by training n prompts on the same task, we create n separate "models" for a task, while still sharing the core language modeling parameters throughout. beyond drastically reducing storage costs, the prompt ensemble makes inference more efficient. to process one example, rather than computing forward passes of n different models, we can execute a single forward pass with a batch size of n , replicating the example across the batch and varying the prompt. these savings mirror those seen for multi-tasking in figure 2 . to demonstrate the viability of prompt ensembling, we train five prompts for each superglue task, using a single frozen t5-xxl model with our default hyperparameters. we use simple majority voting to compute predictions from the ensemble . ## interpretability an ideally interpretable prompt would consist of natural language that clearly describes the task at hand, explicitly asks the model for some result or action, and makes it easy to understand why the prompt elicited such behavior from the model. as prompt tuning works in the continuous embedding space rather than the discrete token space, interpreting prompts becomes more difficult. to test the interpretability of our learned soft prompts, we compute the nearest neighbors to each prompt token from the frozen model's vocabulary. we use cosine distance between the vocabulary embedding vector and the prompt token representation as the similarity metric. we observe that for a given learned prompt token, the top-5 nearest neighbors form tight semantic clusters. for example, we see lexically similar clusters such as { technology / technology / technologies / technological / technologies }, as well as more diverse but still strongly related clusters such as { entirely / completely / totally / altogether / 100% }. the nature of these clusters suggests that the prompts are in fact learning "word-like" representations. we found that random vectors drawn from the embedding space do not show this sort of semantic clustering. when initializing the prompts using the "classlabel" strategy, we often find that the class labels persist through training. specifically, if a prompt token is initialized to a given label, that label is often among the learned token's nearest neighbors after tuning. when initializing with the "random uniform" or "sampled vocab" methods, the class labels can also be found in the nearest neighbors of the prompts; however they tend to appear as neighbors to multiple prompt tokens. this suggests that the model is learning to store the expected output classes in the prompts as reference, and initializing the prompt to outputs classes makes this easier and more centralized. when examining longer prompts (e.g. size 100), we often find several prompt tokens with the same nearest neighbors. this suggests there is either excess capacity in the prompt, or that the lack of sequential structure in the prompt representation makes it difficult for the model to localize information to a specific position. while the learned prompts taken as sequences show little interpretability, we do observe a high frequency of words like science, technology and engineering as the nearest neighbors for prompts trained on the boolq dataset and approximately 20% of the questions are in the "nature/science" category. while more investigation is needed, this suggests that one role of the prompt may be to prime the model to interpret inputs in a specific domain or context (e.g. "scientific"). ## conclusion in this paper, we showed that prompt tuning is a competitive technique for adapting frozen pretrained language models to downstream tasks. on the popular superglue benchmark, its task performance rivals that of traditional model tuning, with the gap vanishing as model size increases. on zeroshot domain transfer, we found that prompt tuning leads to improved generalization. this plausibly indicates that freezing general-purpose language understanding parameters and restricting downstream learning to a lightweight parameter footprint can help to avoid overfitting to a specific domain. beyond task quality metrics, we discussed the appeal of moving to frozen pre-trained models in terms of storage and serving costs. this move enables both efficient multi-task serving, as well as efficient high-performing prompt ensembling. looking forward, we believe that factoring out task-defining parameters as distinct from general language-modeling parameters is an exciting step that opens up many avenues for new research.