SPT : Learning to Selectively Insert Prompts for Better Prompt Tuning (2023)

Abstract: Prompt tuning prepends a soft prompt to the input embeddings or hidden states and only optimizes the prompt to adapt pretrained models (PTMs) to downstream tasks. The previous work manually selects prompt layers which are far from optimal and failed to exploit the potential of prompt tuning. In this work, we propose a novel framework, Selective Prompt Tuning (SPT), that learns to select the proper prompt layers by inserting a prompt controlled by a learnable probabilistic gate at each intermediate layer. We further propose a novel bi-level optimization framework, SPT-DARTS, that can better optimize the learnable gates and improve the final prompt tuning performances of the learned prompt layer settings. We conduct extensive experiments with ten benchmark datasets under the full-data and few-shot scenarios. The results demonstrate that our SPT framework can perform better than the previous state-of-the-art PETuning baselines with comparable or fewer tunable parameters.

## introduction increasingly large pre-trained models (ptms) @xcite @xcite @xcite @xcite @xcite have achieved the state-of-the-art (sota) performances on most nlp tasks. full-model fine-tuning is one of the most widely used method for utilizing ptms. however, fine-tuning @xcite @xcite @xcite needs to tune all parameters of ptms for each task, resulting in large gpu memory and storage costs, especially for supersized ptms @xcite . parameter-efficient tuning (petuning) is a new fine-tuning paradigm that can reduce the adaptation costs of ptms by only tuning a very small * corresponding author: michaelwzhu91@gmail.com number of internal or additional parameters @xcite @xcite . prompt tuning @xcite is a simple and popular petuning method that prepends a sequence of soft prompt tokens to the input sequence and only tunes the prompts to adapt the ptm backbones to downstream tasks. despite its advantages in parameter efficiency and convenience in ptm deployment, prompt tuning suffers from lower performance and convergence rate than other petuning methods like adapters @xcite @xcite , and bitfit @xcite . recently, there has been a branch of literature investigating the advanced techniques for improving the performances of prompt tuning. p-tuning v2 @xcite improves the performance of prompt tuning by inserting soft prompts into every hidden layer of ptms. however, it is difficult to optimize and needs more training steps to attain competitive performance. bbt @xcite optimizes the inserted prompts with derivative-free optimization. idpg @xcite employs a prompt generator with parameterized hyper-complex multiplication @xcite to generate instance-aware soft prompts. lpt @xcite inserts an instance-aware late prompt generated by the neural prompt into an intermediate layer of the ptm instead of the input layer or all layers. @xcite can achieve competitive performance under both full-data and few-shot scenarios. note that the above methods adopt heuristic strategies to determine where to insert prompts on the ptms. in this paper, we first conduct a pilot experiment to show that simple modifications to the prompt inserting strategies in @xcite can result in better performances than these baselines with comparable tunable parameters. the pilot experiments demonstrate that there is a dire need for the optimal strategy of setting prompt layers into ptms. predictably, such an optimal strategy may vary across tasks or ptm backbones and is difficult to construct manually. therefore, we propose the selective prompt tuning (spt) framework (figure 2 ), which automatically searches for the optimal strategy of inserting prompts into the ptms. our spt framework considers a simple search space of whether to insert the generated instanceaware prompts into an intermediate layer of the ptm. as depicted in figure 2 , we initialize a prompt hyper-network where each intermediate layer of ptms inserts a prompt controlled by a learnable probabilistic gate α m . we follow the bilevel optimization strategy of @xcite to optimize the learnable probabilistic gates. after optimization, we keep the top prompt layers that receive the highest probabilities to meet the tunable parameter budgets. to better optimize the learnable gates and obtain better prompt layer settings, we propose spt-darts, which consists of two novel techniques to improve the optimization process of @xcite . our spt framework can automatically determine the suitable promptinserting strategy that achieves a high-quality tradeoff between parameter efficiency and model performance. extensive experiments are conducted on six benchmark datasets from the glue benchmark and four widely studied text classification benchmarks. the results show that spt performs comparable to or outperforms the previous sota petuning methods. especially in the few-shot scenario with 100 training samples, spt outperforms the petuning baselines by a clear margin. figure 1 depicts the overall comparison between our spt method and baselines. to summarize, our contributions are: ## problem formulation for ptm full fine-tuning, the input samples are usually reformulated as [cls] ⟨s 1 ⟩ [sep] if the inputs are single sentences, and as [cls] ⟨s 1 ⟩ [sep] ⟨s 2 ⟩ [sep] if the inputs are sentence pairs. after the ptm backbone encodes the inputs, the final hidden states of the [cls] token will be used to predict classification labels with a linear classification head. in the settings of prompt tuning, the downstream tasks are reformulated as masked language model tasks to close the gap between pre-training and finetuning. specifically, we insert randomly initialized soft prompt p on the word embeddings, and also modify the original inputs using different manually designed templates with a [mask] token for task adaptations. for example, in single-sentence tasks, the input will be transformed into a template like in the setting of our proposed spt framework (depicted in figure 2 ), we investigate the problem of whether to insert newly generated instanceaware prompts at the word embeddings or certain intermediate layers of ptm. for convenience, we will refer to the word embedding layer as the 0th layer of the ptm. we refer to the layers at which new prompts are inserted as the prompt layers (pls). at a certain prompt layer i, we will use a prompt generator pg i to generate a prompt p i from a given input's hidden states at layer i. ## spt: selective prompt tuning in this section, we will elaborate on our selective prompt tuning (spt) framework, which is depicted in figure 2 . we first discuss our motivations. then we will elaborate on our method for determine the prompt layers. ## conclusion in this work, we propose the spt framework to automatically determine the optimal settings for prompt layers under the given ptm backbone and downstream task. we initialize a prompt hypernetwork in which each layer has a prompt generator controlled by a learnable probabilistic gate. to better optimize the prompt hyper-network, we propose a novel spt-darts method containing two novel modifications to the original darts' bi-level optimization process. experiment results in full-data and few-shot scenarios demonstrate that spt can achieve comparable or better performance than state-of-the-art petuning methods while maintaining parameter and inference efficiency.