P -Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks (2022)

Abstract: Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning (CITATION) optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.

## introduction pretrained language models @xcite @xcite improve performance on a wide range of natural language understanding (nlu) tasks. a widely-used method, fine-tuning, updates the entire set of model parameters for a target task. while fine-tuning obtains good performance, it is memory-consuming during training because gradients and optimizer states for all parameters must be stored. moreover, keeping a copy of model parameters for each task during inference is inconvenient since pre-trained models are usually large. prompting, on the other hand, freezes all parameters of a pre-trained model and uses a natural language prompt to query a language model (brown † corresponding to: zhilin yang (zhiliny@tsinghua.edu.cn) and jie tang (jietang@tsinghua.edu.cn) * indicates equal contribution. 1 our code and data are released at https://github. com/thudm/p-tuning-v2 . our main contribution in this paper is a novel empirical finding that properly optimized prompt tuning can be comparable to fine-tuning universally across various model scales and nlu tasks. in contrast to observations in prior work, our discovery reveals the universality and potential of prompt tuning for nlu. technically, our approach p-tuning v2 is not conceptually novel. it can be viewed as an optimized and adapted implementation of deep prompt tuning @xcite designed for generation and knowledge probing. the most significant improvement originates from appling continuous prompts for every layer of the pretrained model, instead of the mere input layer. deep prompt tuning increases the capacity of continuous prompts and closes the gap to fine-tuning across various settings, especially for small models and hard tasks. moreover, we present a series of critical details of optimization and implementation to ensure finetuning-comparable performance. experimental results show that p-tuning v2 matches the performance of fine-tuning at different model scales ranging from 300m to 10b parameters and on various hard sequence tagging tasks such as extractive question answering and named entity recognition. p-tuning v2 has 0.1% to 3% trainable parameters per task compared to fine-tuning, which substantially reduces training time memory cost and per-task storage cost. prompt tuning. let v be the vocabulary of a language model m and let e be the embedding layer of m. in the case of discrete prompting @xcite , prompt tokens {"it", "is", "[mask]"} ⊂ v can be used to classify a movie review. for exam-ple, given the input text @xmath0, the input embedding sequence is formulated as [e(x), e("it"), e("is"), e("[mask]")]. @xcite and @xcite introduce trainable continuous prompts as a substitution to natural language prompts for nlu with the parameters of pretrained language models frozen. given the trainable continuous embeddings [h 0 , ..., h i ], the input embedding sequence is written as [e(x), h 0 , ..., h i , e("[mask]")], as illustrated in figure 2 . prompt tuning has been proved to be comparable to fine-tuning on 10billion-parameter models on simple classification tasks @xcite @xcite . ## p-tuning v2 lack of universality across scales. @xcite shows that prompt tuning can be comparable to fine-tuning when the model scales to over 10 billion parameters. however, for medium-sized models (from 100m to 1b) that are widely used, prompt tuning performs much worse than fine-tuning. lack of universality across tasks. though @xcite ; @xcite have shown superiority on some of the nlu benchmarks, the effectiveness of prompt tuning on hard sequence tagging tasks is not verified. sequence tagging predicts a sequence of labels for each input token, which can be harder and incompatible with verbalizers @xcite . in our experiments (cf. section 4.2 and considering these challenges, we propose ptuning v2, which adapts deep prompt tuning @xcite as a universal solution across scales and nlu tasks. ## experiments we conduct extensive experiments over different commonly-used pre-trained models and nlu tasks to verify the effectiveness of p-tuning v2. in this work, all methods except for fine-tuning are conducted with frozen language model backbones, which accords with @xcite 's setting but differs from @xcite 's tuned setting. ratios of task-specific parameters (e.g., 0.1%) are derived from comparing continuous prompts' parameters with transformers' parameters. another thing to notice is that our experiments are all conducted in the fully-supervised setting rather than few-shot setting. nlu tasks. first, we include datasets from su-perglue @xcite to test p-tuning v2's general nlu ability. additionally, we introduce a suite of sequence labeling tasks, including named entity recognition @xcite @xcite , extractive question answering @xcite , and semantic role labeling @xcite ). pre-trained models. we include bert-large @xcite , roberta-large @xcite , deberta-xlarge @xcite , glmxlarge/xxlarge @xcite for evaluation. they are all bidirectional models designed for nlu tasks, covering a wide range of sizes from about 300m to 10b. multitask learning. for the multi-task setting, we combine the training sets of the datasets in each task type (e.g., combing all training sets of semantic role labeling). we use separate linear classifiers for each dataset while sharing the continuous prompts (cf. appendix a). ## conclusions we present p-tuning v2, a prompt tuning method. despite its relatively limited technical novelty, it contributes to a novel finding that prompt tuning can be comparable to fine-tuning universally across scales (from 330m to 10b parameters) and tasks. with high accuracy and parameter efficiency, p-tuning v2 can be a potential alternative for finetuning and a strong baseline for future work.