Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts (2022)

Abstract: Prompt tuning is a parameter-efficient tuning (PETuning) method for utilizing pre-trained models (PTMs) that simply prepends a soft prompt to the input and only optimizes the prompt to adapt PTMs to downstream tasks. Although it is parameter- and deployment-efficient, its performance still lags behind other state-of-the-art PETuning methods. Besides, the training cost of prompt tuning is not significantly reduced due to the back-propagation through the entire model. Through empirical analyses, we shed some light on the lagging performance of prompt tuning and recognize a trade-off between the propagation distance from label signals to the inserted prompt and the influence of the prompt on model outputs. Further, we present Late Prompt Tuning (LPT) that inserts a late prompt into an intermediate layer of the PTM instead of the input layer or all layers. The late prompt is obtained by a neural prompt generator conditioned on the hidden states before the prompt insertion layer and therefore is instance-dependent. Through extensive experimental results across various tasks and PTMs, we show that LPT can achieve competitive performance to full model tuning and other PETuning methods under both full-data and few-shot scenarios while possessing faster training speed and lower memory cost.

## introduction pre-trained models @xcite @xcite @xcite @xcite have pushed most nlp tasks to state-of-the-art. model tuning (or fine-tuning) is a popular method for utilizing ptms on downstream tasks that needs to tune all parameters of ptms for every task. despite the welcome outcome, it leads to prohibitive adaptation costs, especially for supersized ptms @xcite the details can be found in section 5. @xcite . parameter-efficient tuning (petuning) is a new tuning paradigm that can adapt ptms to downstream tasks by only tuning a very small number of internal or additional parameters. prompt tuning @xcite is a simple and popular petuning method that prepends a sequence of soft prompt tokens to the input and only optimizes the prompt to adapt ptms to downstream tasks. it has an absolute advantage in parameter efficiency and facilitates mixed-task inference, which makes the deployment of ptms convenient. however, compared with other advanced petuning methods, e.g., adapter @xcite , lora @xcite , and bitfit @xcite , prompt tuning suffers from lower performance and convergence rate. compared with full model tuning, although the number of trainable parameters in prompt tuning reduces by ∼17,000× (from 355m to 21k on roberta large ), the training speed only increases by ∼1.5×, and the memory cost only reduces by 29.8%. 1 p-tuning v2 @xcite improves the performance of prompt tuning by inserting soft prompts into every hidden layer of ptms, but it is difficult to optimize and needs more training steps to attain competitive performance. in this paper, we explore why prompt tuning performs poorly and find there is a trade-off between the propagation distance from label signals to the inserted prompt and the influence of the prompt on model outputs. the key to prompt tuning is to make the soft prompt carry task-related information through downstream training. the trained prompt can interact with text inputs during the model forward pass to obtain text representations with taskrelated information. since the prompt is inserted into the input in prompt tuning, it has a strong ability to influence the outputs of ptm through sufficient interactions with text inputs. however, there is a long propagation path from label signals to the prompt. it leads us to ask the question: does this long propagation path cause a lot of task-related information to be lost during propagation and thus affect performance? to verify the impact of the propagation distance on performance, we conduct pilot experiments by shortening it in section 4 and find that the performance first increases then decreases with the shortening of the length. this finding inspires us to present the late prompt (i.e., inserting the prompt into an intermediate hidden layer of ptm). the late prompt not only receives more task-related information at each update due to the shorter propagation path of task-related information but also maintains the adequate ability to influence the outputs of ptm. despite the higher performance and faster convergence rate of late prompt than prompt tuning, the hidden states produced by ptm before the prompt insertion layer are underutilized. to further improve performance and take full advantage of these contextual hidden representations, we introduce a prompt generator to generate the soft prompt (termed as instanceaware prompt) for each instance using the corresponding hidden states. based on the late and instance-aware prompt, we present late prompt tuning (lpt) to improve prompt tuning. since the soft prompt is inserted into an intermediate layer of ptm, we have no need to compute gradients for model parameters be-1 refer to section 6.5 for details. low the prompt insertion layer, and therefore speed up the training process and reduce memory costs. extensive experimental results show that lpt outperforms most prompt-based tuning methods and can be comparable with adapter-based tuning methods and even full model tuning. especially in the few-shot scenario with only 100 training samples, lpt outperforms prompt tuning by 12.4 points and model tuning by 5.0 points in the average performance of ten text classification tasks. besides, it is 2.0× faster and reduces by 56.6% than model tuning in terms of training speed and memory cost on roberta large , respectively. figure 1 shows an overall comparison between lpt and its counterparts. to sum up, the key contributions of this paper are: ## related work adapter-based tuning. one research line of petuning is adapter-based tuning @xcite that inserts some adapter modules between model layers and optimizes these adapters in downstream training for model adaptation. adapter @xcite inserts adapter modules with bottleneck architecture between every consecutive transformer @xcite sublayers. adapterdrop @xcite investigates the efficiency through removing adapters from lower layers. compacter @xcite uses low-rank optimization and parameterized hypercomplex multiplication @xcite to compress adapters. adapter-based tuning methods have comparable results with model tuning when training data is sufficient but don't work well in the few-shot scenario @xcite . prompt-based tuning. another main research line of petuning is prompt-based tuning that inserts some additional soft prompts into the hidden states instead of injecting new neural modules to ptms. prompt tuning @xcite and ptuning @xcite insert a soft prompt to word embeddings only, and can achieve competitive results when applied to supersized ptms. prefixtuning (li and liang, 2021) and p-tuning v2 @xcite insert prompts to every hidden layer of ptm. bbt @xcite optimizes the inserted prompt with derivative-free optimization. some prompt-based tuning methods, like prompt tuning and bbt, formulate downstream tasks as pre-training tasks (e.g., masked language modeling task) to close the gap between pre-training and downstream training @xcite . there are also some prompt-based methods with instanceaware prompt. idpg @xcite uses the prompt generator with parameterized hypercomplex multiplication @xcite to generate a soft prompt for every instance. contexttuning @xcite uses bert model @xcite as the prompt generator and focuses on nlg tasks. ipl @xcite first calculates relevance scores between prompt tokens and inputs, then uses the scores to re-weight the original prompt tokens. but it tunes all parameters of ptm. all the above methods with instanceaware prompt have the same weakness because they need to encode the inputs using an extra encoder, which slows down the training and increases inference latency. there are also some other popular petuning methods, such as bitfit @xcite which only tunes the bias terms, lora @xcite which optimizes low-rank decomposition matrices of the weights within self-attention layers. ## problem formulation given a ptm m, in the setting of model tuning, we first reformulate the inputs with single sentence as e([cls] ⟨s 1 ⟩ @xcite ) and the inputs with sentence pair as e([cls] ⟨s 1 ⟩ [sep] ⟨s 2 ⟩ [sep]), where e is the embedding layer of m. the final hidden state of [cls] token will be used to predict label. in the setting of prompt tuning, we insert a randomly initialized soft prompt p into word embeddings, and also modify the original inputs using different manual templates with a [mask] token for different tasks. for example, the inputs with single sentence from a sentiment analysis task will be transform into concat(p, e([cls] ⟨s 1 ⟩ it was @xcite . [sep])). then, we map the original labels y to some words in the vocabulary v of m, which formulates downstream tasks as a language modeling task to close the gap between pre-training and downstream training. the final hidden state of [mask] token will be used to predict label. in the setting of our proposed method lpt, we use a prompt generator (pg) to generate an independent prompt p for every input. in addition, the layer that the prompt inserts into is an intermediate layer of ptm instead of word embeddings, and we refer to the layer as the prompt layer (pl). ## why prompt tuning performs poorly? the workflow of prompt tuning is to make the inserted soft prompt carry task-related information through downstream training. in the inference phase, this prompt can interact with test inputs during layer-upon-layer propagation so that the hidden representations of these inputs also contain taskrelated information. there are strong interactions between the prompt and text inputs because prompt tuning inserts prompt into word embeddings. however, there is a long propagation path from label signals to the prompt. therefore, we speculate that the poor performance of prompt tuning is due to the long propagation path of task-related information, which causes a lot of task-related information to be lost during propagation in the frozen model and thus affect performance. to verify this conjecture, we conduct some pilot experiments on trec @xcite and rte @xcite datasets using roberta large @xcite . ## lpt: late prompt tuning from the experiment results in section 4, we observe that using late prompt can greatly improve the performance of prompt tuning. moreover, late prompt can bring two other advantages: (1) no gradient calculation for model parameters below the prompt layer; (2) the hidden states produced by the model before the prompt layer can be used to generate a great independent prompt for each instance. based on these advantages, we propose an efficient prompt-based tuning method lpt which combines late and instance-aware prompts. an illustration of lpt is shown in figure 3 . in this section, we will introduce two different prompt generators used in lpt and how to determine the prompt layer. ## conclusion in this paper, we explore why prompt tuning performs poorly and find there is a trade-off between the propagation distance from label signals to the inserted prompt and the influence of the prompt on model outputs. with this discovery, we present a more efficient and effective prompt tuning method lpt with late and instance-aware prompts. experiment results in full-data and few-shot scenarios demonstrate lpt can achieve comparable or even better performance than state-of-the-art petuning methods and full model tuning while having higher training speed and lower memory cost.