XP rompt: Exploring the Extreme of Prompt Tuning (2022)

Abstract: Prompt tuning learns soft prompts to condition the frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner. While prompt tuning has gradually reached the performance level of fine-tuning as the model scale increases, there is still a large performance gap between prompt tuning and fine-tuning for models of moderate and small scales (typically less than 11B parameters). In this paper, we empirically show that the trained prompt tokens can have a negative impact on a downstream task and thus degrade its performance. To bridge the gap, we propose a novel Prompt tuning model with an eXtremely small scale (XPrompt) under the regime of lottery tickets hypothesis. Specifically, XPrompt eliminates the negative prompt tokens at different granularity levels through a hierarchical structured pruning, yielding a more parameter-efficient prompt yet with a competitive performance. Comprehensive experiments are carried out on the SuperGLUE tasks, and the results indicate that XPrompt is able to close the performance gap at smaller model scales.

## introduction pre-trained language models (plms) have been widely applied and achieved a remarkable success in various nlp tasks @xcite @xcite under the pretrain-then-finetune paradigm @xcite . despite of its compelling performance, fine-tuning is parameter-inefficient for large scale plms due to the fact that the memory footprint is proportional to the number of trainable parameters whose gradients and optimizer states need to be stored @xcite . dawei song and jingang wang are the corresponding authors. 1 the code is available at https://github.com/bd-mf/ xprompt . ## preliminary built upon the text-to-text approach of t5 @xcite , prompt tuning formulates all tasks as text generation by prepending additional l tunable soft prompt tokens to the input and only updating the parameters of the inserted soft prompt tokens. specifically, given a series of n input tokens @xmath0, x 2 , ..., x n }, t5 first generates the token embeddings x @xmath1, where e is the dimension of the embedding space. it also generates soft prompt embeddings p @xmath2, p 2 , ..., p m } ∈ r m×e , where m is the length of the soft prompt. then the soft prompts are prepended to the input sequence as [p e ; x e ] ∈ r (m+n)×e . the goal of prompt tuning is to maximize the likelihood of the labels y by only optimizing over p e : prompt tuning becomes more effective as the model scale increases. however, there is still a significant performance gap between prompt tuning and fine-tuning especially for models of small and moderate scales. our hypothesis is that not all soft prompt tokens contribute equally to the performance after training on the target task. there exist certain soft prompt tokens that may have negative impacts on the task. therefore, combining the idea of the lottery ticket hypothesis, we propose xprompt with hierarchical structured pruning to identify the optimal soft prompts and bridge the performance gap. ## xprompt the overall process of xprompt is illustrated in figure 3 , which consists of three main stages: prompt-tuning, hierarchical structured pruning and rewinding. specifically, the prompt tuning learns an initial set of values for all soft prompt tokens on the target task. during the hierarchical structured pruning, token-level and piece-level ## analysis and discussion to better understand the effectiveness of the xprompt and explore the impact of various factors in xprompt, we further conduct a series of ablation studies and analysis.