True Few-Shot Learning with P rompts— A Real-World Perspective (2022)

Abstract: Prompt-based approaches excel at few-shot learning. However, Perez et al. (2021) recently cast doubt on their performance as they had difficulty getting good results in a “true” few-shot setting in which prompts and hyperparameters cannot be tuned on a dev set. In view of this, we conduct an extensive study of Pet, a method that combines textual instructions with example-based finetuning. We show that, if correctly configured, Pet performs strongly in true few-shot settings without a dev set. Crucial for this strong performance is a number of design choices, including Pet’s ability to intelligently handle multiple prompts. We put our findings to a real-world test by running Pet on RAFT, a benchmark of tasks taken from realistic NLP applications for which no labeled dev or test sets are available. Pet achieves a new state of the art on RAFT and performs close to non-expert humans for 7 out of 11 tasks. These results demonstrate that prompt-based learners can successfully be applied in true few-shot settings and underpin our belief that learning from instructions will play an important role on the path towards human-like few-shot learning capabilities.

## introduction with pretrained language models (lms) getting ever larger @xcite @xcite , instruction-based learning is a powerful method for few-shot text classification (e.g., @xcite @xcite @xcite . the key idea is to give an lm access to descriptive names for all possible outputs and to short prompts explaining the task to be solved. in settings where at most a few dozen examples are available, this simple idea leads to substantial improvements over other approaches @xcite @xcite . however, recent work has questioned the strong few-shot performance of instruction-based approaches, arguing that they are evaluated in scenarios that are not true few-shot settings @xcite , mainly for two reasons. first, some approaches (e.g., @xcite @xcite ) make use of large development sets to optimize hyperparameters. second, it is argued that manually designed instructions require manual tuning on development sets to achieve strong performance @xcite . indeed, performance can vary greatly-and in mostly unpredictable ways-across different instructions @xcite ; this issue even persists after finetuning on hundreds of instructions @xcite . more generally, the need for human involvement is seen as a serious drawback of manually designed instructions @xcite . thus, several recent studies have abandoned them in favor of automatically generated prompts @xcite @xcite @xcite . contrary to this trend, we argue that when correctly configured, prompt-based approaches achieve strong performance even in true few-shot settings and that there is no problem with using manually designed instructions. quite the opposite: such instructions are often easy to specify if one is familiar with the task to be solved, they provide an intuitive interface to convey taskspecific knowledge, and, if properly used, they can considerably improve model performance in fewshot settings. to provide empirical support for these claims, we revisit pet @xcite , a method for combining instructions with examplebased finetuning, and thoroughly examine its performance with human-made instructions in true few-shot settings. we simulate a real-world scenario by proceeding in two steps: first, we conduct figure 1 : pet achieves near-human performance for 7 out of 11 tasks of the raft benchmark @xcite , for which labeled dev and test sets are not available. this demonstrates the potential of prompt-based learners for few-shot learning in ''true'' real-world settings, i.e., without any tuning of instructions or hyperparameters on a dev set. an extensive study of pet using three academic datasets to analyze its ability to perform true few-shot learning in a controlled environment and derive best practices for the choice of instructions and hyperparameters. we then put our findings to the test and evaluate pet on a large variety of real-world tasks from the raft benchmark @xcite , for which no labeled dev or test sets are available, enforcing a true few-shot setting @xcite . on average, pet clearly outperforms all baselines on this dataset and comes surprisingly close to non-expert human performance (see figure 1 ). this demonstrates that instruction-based learning can successfully be applied to real-world tasks in true few-shot settings. our main contributions are as follows: as a precursor to instruction-based learning, some studies have investigated ways of informing classifiers about the meaning of different output classes both for text @xcite @xcite and image classification @xcite ; providing instructions in the form of short prompts was first proposed by @xcite . this idea has since been applied to solve a wide range of nlp tasks without any task-specific training data @xcite @xcite @xcite @xcite . while most approaches rephrase tasks as a language modeling problem, some use prompts to reformulate them as different tasks for which large amounts of training data are available @xcite @xcite @xcite . instruction-based learning has also been used in few-shot settings; popular variants include in-context learning, where the model's parameters are fixed and examples are provided as additional context @xcite @xcite @xcite , finetuning the entire model @xcite @xcite , and prompt tuning, where only the instruction itself is optimized @xcite @xcite . several works investigating the limitations of instruction-based few-shot approaches find that current lms are mostly unable to understand complex instructions that go beyond short prompts or simple questions @xcite @xcite and that they are highly sensitive to the exact wording of the instructions provided @xcite @xcite . in a similar vein, @xcite and @xcite argue that prior work overestimates few-shot performance as manual prompt tuning is required to achieve good performance. accordingly, some studies attempt to obtain either prompts @xcite @xcite or meaningful names for output classes @xcite without human involvement. figure 2 : different choices of patterns and corresponding verbalizers for classifying movie reviews as positive (+) or negative (-). the input is first converted into a cloze question using the pattern; classification is done by computing the output whose verbalization is, according to the mlm, the most likely substitute for the mask. finally, many benchmarks have been proposed for comparing few-shot approaches in a standardized way (e.g., @xcite @xcite @xcite . as our focus is on the real-world applicability of few-shot methods, we evaluate pet on the raft benchmark @xcite , which measures performance in applied settings. ## pattern-exploiting training we briefly review pattern-exploiting training (pet) @xcite , the method we use for instruction-based text classification. at its core, pet combines textual instructions with regular finetuning using labeled examples. to that end, users must specify one or more patterns that convert an input example x into a cloze question so that it can readily be processed by a masked language model (mlm) @xcite . 1 these patterns can take on very different forms; some examples are shown in figure 2 . in addition, users must inform the model about the meaning of all output classes; this is done with a verbalizer that assigns a natural language expression to each output y (see figure 2 , right). we refer to the combination of a pattern and verbalizer as a pattern-verbalizer pair (pvp). given a single pvp, let p(y | x) be the probability that an mlm assigns to y's verbalization in the cloze question obtained by applying the pattern to x, normalized over all y. the mlm is finetuned on labeled examples (x, y) by minimizing the cross-entropy loss between p(y | x) and a distribution that assigns a probability of 1.0 to y. if a user specifies multiple pvps, individual models are trained for each pair. similar to knowledge distillation @xcite , they are 1 we use the term prompt to refer to a short sequence of tokens that typically contains some form of instruction; pattern is used to denote the function that adds a prompt to an input. ## true few-shot learning with pet after describing our experimental setup, we conduct experiments on academic datasets to answer 6 important research questions (q1-q6) on the extent to which true few-shot learning is possible with pet. the purpose of our experiments is also to establish best practices for real-world scenarios and experiments on raft @xcite . tasks and datasets while they were heavily used in prior work (e.g., @xcite @xcite , we decide against tasks and datasets from glue @xcite and superglue @xcite as they are different from what we expect to see in real-world applications. instead, we experiment with ag's news, yelp reviews full star, and yahoo questions @xcite as these datasets represent classification tasks in three different domains that resemble real-world settings. we create a broad variety of instructions for each task to be able to experiment with a large number of different patterns. we consider settings with @xmath0. for each n, we generate five different training sets per task by randomly sampling examples from the original training set while ensuring that the number of examples is about the same for each possible output class. in addition, for both @xmath1, we sample 1,000 unlabeled examples from the original training set. we repeat all of our experiments for all five training sets and, by default, report average performance. pvps we manually write a total of 23 patterns per task, all of which can be categorized into one of the following groups: foot_0 • null: following logan iv et al. (2021), these patterns simply insert a mask token. for all patterns, we use a single verbalizer, adopted from @xcite . there is often a single natural choice for the verbalizer (e.g., the category names for ag's news / yahoo questions), so finding many good verbalizers is challenging. hyperparameters we consider a setting similar to that of @xcite and @xcite and, unless otherwise specified, use the default settings of the pet library. 3 as our experiments require training hundreds of models, we make a few changes to reduce environmental impact @xcite and computational cost: we use the base variant of roberta @xcite as underlying lm both for individual models and the final classifier, we train only one model per pvp, and we reduce the training steps for all individual models and the final classifier to 100 and 1,000, respectively. monitoring finetuning lms on small datasets is unstable @xcite and sometimes results in poor performance. we aim to detect failed finetuning without a labeled test set using the following two checks: whenever one of these two events occurs, we restart training using a different seed. q1: how can we find the best pattern-or do we even need to? slightly different patterns can have very different performance @xcite @xcite @xcite and popular model selection criteria cannot reliably identify the best-performing patterns in few-shot settings @xcite . we thus investigate to what extent pet can eliminate the need to find the best instruction even in extreme settings where there are dozens of candidates to choose from. setup using our default setup, we train individual models for each pvp and a final pet model; we also train models with ipet, an iterative variant of pet introduced by schick and schütze (2020), using 3 iterations. results performance of individual models for each pattern and of the distilled models obtained using pet and ipet is shown in figure 3 . interestingly, sorting all pattern groups by their average performance gives the exact same order for each task and training set size: null patterns clearly perform worst, followed by punc and prompt; q&a gives the best average results. contrary to findings of @xcite , this shows that lms can benefit considerably from manually written instructions even if combined with finetuning. crucially, pet's performance is much higher than average performance of individual patterns; further, it consistently outperforms even the best pattern, verifying that pet indeed removes the need to find the best pattern. while ipet gives clear improvements for @xmath2, it performs worse than pet for @xmath3. the reason for this may be that we use a much smaller set of unlabeled examples than prior work @xcite . ## pet for real-world tasks we use our insights from §4 to apply pet to the raft benchmark, a collection of 11 diverse real-world tasks whose automated solution has inherent value to someone @xcite . these tasks are challenging for few-shot approaches: they require domain expertise, understanding of detailed instructions, processing of long inputs, and handling a large number of output classes. ## discussion our experimental results in §4 and §5 show that strong performance in few-shot settings is clearly possible without manual prompt tuning or hyperparameter optimization on large dev sets; in other words, pet can successfully be applied in true few-shot settings. while we believe that it should be an important goal of future work to make lms more robust to different instructions, even with current models it is relatively easy to successfully apply pet when following a few simple principles-such as rephrasing the task in a q&a format, using simple vocabulary and single-token verbalizers where possible, and specifying at least a handful of different patterns. in light of these findings, we also hope that future work will not view human involvement in prompt design as a drawback of instruction-based approaches, but rather as an exciting possibility to communicate with models in ways other than exclusively through examples. our study has limitations. first, a major obstacle to using pet in real-world applications is that we do not know a priori how well it performs for a given task; we therefore believe an important next step is to investigate methods for estimating performance without access to large test sets-for example, through model calibration @xcite @xcite -in real-world settings. in addition, we did not fully explore the capabilities of pet; for example, we did not investigate domain-adaptive pretraining @xcite and auxiliary language modeling @xcite , both of which were shown to be helpful by @xcite . we also did not quantify the impact of our decisions regarding b77 and the effectiveness of monitoring ( §4) and only considered english models and datasets. finally, we did not examine pet's performance beyond aggregate scores. while this is not feasible on raft due to its nature, performing such analysis either with other datasets or with methods such as the ones proposed by @xcite would be relevant future work to understand real-world capabilities of instruction-based approaches more comprehensively. ## conclusion in light of recent work casting doubt on the performance of prompt-based approaches in true few-shot settings @xcite , we have conducted an extensive study of pet. in a controlled environment, we found that manually designed instructions outperform null prompts, with q&astyle prompts performing best (q1, q2). across different tasks, models and training set sizes, pet consistently outperforms even the best individual prompt (q1, q2). we have also shown that pet is robust to uninformative prompts and to different choices of hyperparameters (q3, q5), that as little as four prompts are sufficient to reach good performance (q4), and that synthetic examples can be used to replace unlabeled data (q6). based on these insights, we applied pet to a benchmark of real-world tasks, where it achieves near-human performance for 7 out of 11 tasks without any tuning on a dev set, demonstrating the potential of instruction-based approaches in true few-shot settings.