How many data points is a prompt worth? (2021)

Abstract: When fine-tuning pretrained models for classification, researchers either use a generic model head or a task-specific prompt for prediction. Proponents of prompting have argued that prompts provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. We aim to quantify this benefit through rigorous testing of prompts in a fair setting: comparing prompted and head-based fine-tuning in equal conditions across many tasks and data sizes. By controlling for many sources of advantage, we find that prompting does indeed provide a benefit, and that this benefit can be quantified per task. Results show that prompting is often worth 100s of data points on average across classification tasks.

## introduction the main paradigm for adapting pretrained models for classification @xcite @xcite is fine-tuning via an explicit classifier head. however, an alternative approach has arisen: adapting the pretrained language model directly as a predictor through autoregressive text generation @xcite or completion of a cloze task @xcite . this method is notably used in t5 fine-tuning @xcite leading to state-of-the-art results on the superglue benchmark @xcite . one argument made for classification by direct language generation is that it allows us to pick custom prompts for each task @xcite . while this approach can be used for zero-shot classification @xcite or priming @xcite , it can also be used in fine-tuning to provide extra task information to the classifier, especially in the low-data regime @xcite . code available at https://github.com/tevenlescao/pet if this argument is indeed true, it is natural to ask how it impacts the sample efficiency of the model, or more directly, how many data points is a prompt worth? as with many low-data and pretrainingbased problems, this question is complicated by the fine-tuning setup, training procedure, and prompts themselves. we attempt to isolate these variables through diverse prompts, multiple runs, and best practices in low-training data fine-tuning. we introduce a metric, the average data advantage, for quantifying the impact of a prompt in practice. our experiments find that the impact of tasktargeted prompting can nicely be quantified in terms of direct training data, and that it varies over the nature of different tasks. on mnli @xcite , we find that using a prompt contributes approximately 3500 data points. on su-perglue, it adds approximately 280 data points on rte @xcite and up to 750 on boolq @xcite . in low-to medium-data settings, this advantage can be a real contribution to training a model. ## related work prompting has been used both for zero-shot and fine-tuning based methods. zero-shot approaches attempt to answer a task with a prompt without finetuning through generation @xcite . @xcite extends this approach to a supervised priming method by taking in training data as priming at inference time, so it can attend to them while answering. t5 @xcite and other sequence-to-sequence pretrained models use standard word-based fine-tuning with a marker prompt to answer classification tasks with strong empirical success. our setting differs in that we are interested in using task-based prompts and finetuning, in-between the t5 and gpt2 setting. our setting most closely resembles pet @xcite , which claims that taskspecific prompting helps transfer learning, espe-cially in the low-data regime. however, in order to reach the best possible results on superglue, pet introduces several other extensions: semisupervision via additional pseudo-labeled data, ensembling models trained with several different prompts, and finally distilling the ensemble into a linear classifier rather than a language model. our aim is to isolate the specific contributions of prompting within supervised fine-tuning. finally, recent papers have experimented with discovering prompts through automated processes tailored to the language model @xcite . we limit ourselves to humanwritten prompts, as we are interested into whether prompting itself specifically adds information to the supervised task. it is an interesting question as to whether automatic prompts can have this same impact (relative to the training data they require). ## comparison: heads vs prompts consider two transfer learning settings for text classification: head-based, where a generic head layer takes in pretrained representations to predict an output class; prompt-based, where a task-specific pattern string is designed to coax the model into producing a textual output corresponding to a given class. both can be utilized for fine-tuning with supervised training data, but prompts further allow the user to customize patterns to help the model. for the prompt model we follow the notation from pet @xcite and decompose a prompt into a pattern and a verbalizer. the pattern turns the input text into a cloze task, i.e. a sequence with a masked token or tokens that need to be filled. let us use as example an excerpt from superglue task boolq @xcite , in which the model must answer yes-or-no binary questions. in order to let a language model answer the question in italics, our pattern is in bold (schick and schütze, 2020b): "posthumous marriage -posthumous marriage (or necrogamy) is a marriage in which one of the participating members is deceased. it is legal in france and similar forms are practiced in sudan and china. since world war i, france has had hundreds of requests each year, of which many have been accepted. based on the previous passage, can u marry a dead person in france ? <@xmath0 the masked word prediction is mapped to a verbalizer which produces a class. (here "yes": true. "no": false foot_0 ). several pattern-verbalizer pairs (pvps) could be used for a single task, differing either through the pattern, the verbalizer, or both. fine-tuning is done by training the model to produce the correct verbalization. the loss is the cross-entropy loss between the correct answer and the distribution of probabilities amongst the tokens in the verbalizer. we re-use pattern choices from @xcite ; examples are available in appendix a. ## experimental setting we run all experiments with the same pretrained checkpoint, roberta-large (355m parameters) from roberta @xcite , which we load from the transformers @xcite library. 2 in line with previous observations @xcite @xcite , head-based fine-tuning performance varies considerably. we follow recommendations of @xcite and @xcite to train at a low learning rate (10 -5 ) for a large number of steps (always at least 250, possibly for over 100 epochs). we perform our evaluation on superglue and mnli @xcite . these datasets comprise a variety of tasks, all in english, including entailment (mnli, rte @xcite , cb (de marneffe et al., 2019)), multiple choice question answering (boolq @xcite , multirc @xcite ), and commonsense reasoning (wsc @xcite , copa @xcite , wic (pilehvar and camacho-collados, 2018)). we do not include record @xcite in our comparisons as there is no head model to compare with, since it is already a cloze task. data sizes range from 250 data points for cb to 392, 702 for mnli. as test data is not publicly available for superglue tasks, we set aside part of training (from 50 for cb, copa and multirc to 500 for boolq) to use for development, and evaluate on their original validation sets. for mnli, we use the available matched validation and test sets. we compare models across a scale of available data, starting with 10 data points and increasing exponentially (as high-data performance tends to saturate) to the full dataset. for example, for mul-tirc, which has 969 data points initially, we start by reserving 50 data points for development. this leaves us with 919 training points, and we train models with 10, 15, 20, 32, 50, 70, 100, 150, 200, 320, 500, 750, and 919 training points. we run every experiment 4 times in order to reduce variance, for a total of 1892 training runs across all tasks. at every point, we report the best performance that has been achieved at that amount of data or lower. full graphs are presented in appendix b. ## results figure 1 shows the main results comparing head-and prompt-based fine-tuning with the bestperforming pattern on that task. prompting enjoys a substantial advantage on every task, except for wic as is reported in previous results @xcite . both approaches improve with more training data, but prompting remains better by a varying amount. many tasks in superglue have relatively few data points, but we also see an advantage in large datasets like boolq and mnli. to quantify how many data points the prompt is worth, we first isolate the y-axis band of the lowestand highest-accuracy where the two curves match in accuracy. 3 the horizontal line at these points represents the advantage of prompting. we then take the integral in this region, i.e. area between the linearly-interpolated curves foot_3 , divided by the height of the band. the area has the dimension of a quantity of data points times the metric unit, so dividing by the performance range yields a # of data points advantage. as low data training is sensitive to noise, in addition to following best training practices we run several different experiments for each x-point. we use a bootstrapping approach to estimate confidence over these runs. specifically, we hold out one of the 4 head runs and 4 prompt runs (16 combinations total), and compute the standard deviation of those outcomes. we report these quantities for every task in ## analysis impact of pattern vs verbalizer the intuition of prompts is that they introduce a task description in natural language, even with few training points. to better understand the zero-shot versus adaptive nature of prompts, we consider a null verbalizer, a control with a verbalizer that cannot yield semantic information without training. for every task that requires filling in one word (which excludes the more free-form copa and wsc), we replace the verbalizers, for example, "yes", "no", "maybe", "right" or "wrong", with random first names. ## conclusion we investigate prompting through a systematic study of its data advantage. across tasks, prompting consistently yields a varying improvement throughout the training process. analysis shows that prompting is mostly robust to pattern choice, and can even learn without an informative verbalizer. on large datasets, prompting is similarly helpful in terms of data points, although they are less beneficial in performance. in future work, we hope to study the mechanism and training dynamics of the prompting benefits.