Prefix-Tuning: Optimizing Continuous Prompts for Generation (2021)

Abstract: Fine-tuning is the de facto way of leveraging large pretrained language models for downstream tasks. However, fine-tuning modifies all the language model parameters and therefore necessitates storing a full copy for each task. In this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks, which keeps language model parameters frozen and instead optimizes a sequence of continuous task-specific vectors, which we call the prefix. Prefix-tuning draws inspiration from prompting for language models, allowing subsequent tokens to attend to this prefix as if it were “virtual tokens”. We apply prefix-tuning to GPT-2 for table-to-text generation and to BART for summarization. We show that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting, outperforms fine-tuning in low-data settings, and extrapolates better to examples with topics that are unseen during training.

## introduction fine-tuning is the prevalent paradigm for using large pretrained language models (lms) @xcite to perform downstream tasks (e.g., summarization), but it requires updating and storing all the parameters of the lm. consequently, to build and deploy nlp systems that rely on large pretrained lms, one currently needs to store a modified copy of all the lm parameters for each task. this can be prohibitively expensive given the size of current lms; for example, gpt-2 has 774m parameters @xcite and gpt-3 has 175b parameters @xcite . a natural approach to this problem is lightweight fine-tuning, which freezes most of the pretrained parameters and only tunes a smaller set of parameters. for example, adapter-tuning (rebuffi et al., 2017; @xcite inserts additional taskspecific layers between the layers of pretrained language models. adapter-tuning has promising performance on natural language understanding and generation benchmarks, attaining comparable performance with fine-tuning while adding only around 2-4% task-specific parameters @xcite . at the limit, @xcite ) can be deployed using in-context learning, which is a form of prompting, without modifying any lm parameters. in in-context learning, @xcite prepend a natural language task instruction (e.g., tl;dr for summarization) and a few examples to the task input, and then generate the task output from the lm. however, since transformers can only condition on a bounded-length context (e.g., 2048 tokens for gpt-3), in-context learning is restricted to very small training sets. in this paper, we propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation (nlg) tasks, inspired by prompting. consider the task of generating a textual description of a data table, as shown in figure 1 , where the task input is a linearized table (e.g., "name: starbucks | type: coffee shop") and the output is a textual description (e.g., "starbucks serves coffee."). prefix-tuning prepends a sequence of continuous task-specific vectors to the input, which we call a prefix, depicted by red blocks in figure 1 (bottom). to generate each token, the lm can attend to the prefix as if it were a sequence of "virtual tokens", but unlike prompting, the prefix consists entirely of free parameters which do not correspond to real tokens. in contrast to fine-tuning in figure 1 (top), which updates all lm parameters and thus requires storing a tuned copy of the model for each task, prefix-tuning only optimizes the prefix. consequently, we only need to store one copy of the large lm and a learned task-specific prefix, yielding a very small overhead for each additional task (e.g., 250k parameters for table-to-text). in contrast to full fine-tuning, prefix-tuning is also modular: we train an upstream prefix which steers an unmodified lm, and therefore, a single lm can support many tasks at once. in the context of personalization where the tasks correspond to users @xcite , we would have a separate prefix for each user trained only on that user's data, thereby avoiding data cross-contamination. moreover, the prefix-based architecture enables us to even process examples from multiple users/tasks in a single batch, something that is not possible with other lightweight fine-tuning approaches like adaptertuning. we evaluate prefix-tuning on table-to-text generation using gpt-2 and abstractive summarization using bart. in terms of storage, prefix-tuning stores 1000x fewer parameters than full fine-tuning. in terms of performance when trained on full datasets, prefix-tuning and fine-tuning are comparable for table-to-text ( §6.1), while prefix-tuning suffers a small degradation for summarization ( §6.2). in low-data settings, prefix-tuning outperforms finetuning on both tasks ( §6.3). prefix-tuning also extrapolates better to tables (for table-to-text) and articles (for summarization) with unseen topics ( §6.4). ## related work fine-tuning for natural language generation. current state-of-the-art systems for natural language generation (nlg) are based on fine-tuning pretrained lms. for table-to-text generation, kale (2020) fine-tunes a sequence-to-sequence model (t5; @xcite . for extractive and abstractive summarization, researchers fine-tune masked language models (e.g., bert; @xcite and encode-decoder models (e.g., bart; @xcite , respectively @xcite @xcite . for other conditional nlg tasks such as machine translation and dialogue generation, fine-tuning is also the prevalent paradigm @xcite @xcite . in this paper, we focus on table-to-text using gpt-2 and summarization using bart, but prefix-tuning in principle can be applied to other generation tasks and pretrained models, such as masked lms. lightweight fine-tuning. prefix-tuning falls under the broad class of lightweight fine-tuning methods, which freeze most of the pretrained parameters and only tune a smaller set of parameters. the key question is how to augment the lm architecture and decide which subset of pretrained parameters to tune. one line of research learns a task-specific parameter mask @xcite . another line of research inserts new modules with trainable parameters. for example, @xcite trains a "side" network that is fused with the pretrained model via summation; adapter-tuning inserts task-specific layers (adapters) between each layer of the pretrained lm @xcite @xcite . compared to this line of work, which tunes around 3.6% of the lm parameters, our method obtains a further 30x reduction in task-specific parameters, tuning only 0.1% while maintaining comparable performance on table-to-text tasks. prompting. prompting is a way of leveraging a pretrained lm by prepending instructions and a few examples to the task input and generating the task output from the lm. for autoregressive lms, the most successful form of prompting is gpt-3's in-context learning @xcite , which uses manually designed prompts to adapt its generation for different tasks in few-shot settings. for masked lms like bert and roberta @xcite , prompt engineering has been explored for natural language understanding tasks @xcite . for example, autoprompt @xcite searches for a sequence of discrete trigger words and concatenates it with each input to elicit sentiment or factual knowledge from bert and roberta. in contrast with autoprompt, our method optimizes continuous prefixes, which are more expressive ( §7.2); moreover, we focus on language generation tasks. continuous vectors have been used to steer lms; for example, @xcite showed that a pretrained lstm language model can reconstruct arbitrary sentences by optimizing a continuous vector for each sentence, making the vector inputspecific. in contrast, prefix-tuning optimizes a taskspecific prefix that applies to all instances of that task. as a result, unlike the previous work whose application is limited to sentence reconstruction, prefix-tuning can be applied to nlg tasks. controllable generation. controllable generation aims to steer a pretrained language model to match a sentence-level attribute (e.g., positive sentiment or sports). such control can happen at training time: @xcite pretrains the language model (ctrl) to condition on metadata such as keywords or urls. the control can also happen at decoding time, by weighted decoding (gedi, @xcite or iteratively updating the past activations (pplm, @xcite . however, there is no straightforward way to apply these controllable generation techniques to enforce fine-grained control over generated contents, as demanded by tasks like table-to-text and summarization. p*-tuning. prefix tuning is an instance of a new class of methods that has emerged, which we call p*-tuning (since the other prominent instances, ptuning and prompt-tuning, also start with p), all based on the idea of optimizing a continuous prefix or prompt. concurrent with our work, qin and eisner (2021) learn mixtures of soft fill-in-the-blank prompts to elicit knowledge from lms such as bert and bart. @xcite learns task-specific embeddings that adapts bert for sentiment classification. both works show that tuning soft prompts outperforms previous work, which optimizes over discrete prompts. p-tuning @xcite shows that jointly updating the prompt embeddings and lm parameters improves gpt-2's performance on natural language understanding tasks, in both few-shot and full data settings. in a followup work, prompt-tuning @xcite simplifies our approach and applies it to t5 @xcite , demonstrating that the performance gap between fine-tuning and p*tuning vanishes as the model size grows. ## problem statement consider a conditional generation task where the input x is a context and the output y is a sequence of tokens. we focus on two tasks, shown in figure 2 (right): in table-to-text, x corresponds to a linearized data table and y is a textual description; in summarization, x is an article and y is a summary. ## prefix-tuning we propose prefix-tuning as an alternative to full fine-tuning for conditional generation tasks. we first provide intuition in §4.1 before defining our method formally in §4.2. ## intrinsic evaluation we compare different variants of prefix-tuning to study the impact of various design decisions. §7.1 studies the impact of the prefix length. §7.2 studies tuning only the embedding layer, which is more akin to tuning a discrete prompt. §7.3 compares prefixing and infixing, which inserts trainable activations between x and y. §7.4 studies the impact of various prefix initialization strategies. §7.5 further studies the data efficiency of prefix-tuning.