Making Pre-trained Language Models Better Few-shot Learners (2021)

Abstract: The recent GPT-3 model (Brown et al., 2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context. Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient. We present LM-BFF—better few-shot fine-tuning of language models—a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context. Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.

## introduction the gpt-3 model @xcite has made waves in the nlp community by demonstrating astounding few-shot capabilities on myriad language understanding tasks. given only a natural language prompt and a few demonstrations of the task, gpt-3 is able to make accurate predictions without updating any of the weights of its underlying lan-guage model. however, while remarkable, gpt-3 consists of 175b parameters, which makes it challenging to use in most real-wold applications. in this work, we study a more practical scenario in which we only assume access to a moderatelysized language model such as bert @xcite or roberta @xcite , and a small number of examples (i.e., a few-shot setting), which we can use to fine-tune the weights of the language model. this setting is appealing as (1) such models can be trained on typical research hardware; first, we follow the route of prompt-based prediction, first developed by the gpt series @xcite @xcite for zero-shot prediction and recently studied by pet @xcite for fine-tuning. prompt-based prediction treats the downstream task as a (masked) language modeling problem, where the model directly generates a textual response (referred to as a label word) to a given prompt defined by a taskspecific template (see figure 1(c) ). finding the right prompts, however, is an art-requiring both domain expertise and an understanding of the language model's inner workings. even if significant effort is invested, manual prompts are likely to be suboptimal. we address this issue by introducing automatic prompt generation, including a pruned brute-force search to identify the best working label words, and a novel decoding objective to automatically generate templates using the generative t5 model @xcite )-all of which only require the few-shot training data. this allows us to cheaply obtain effective prompts that match or outperform our manually chosen ones. second, we adopt the idea of incorporating demonstrations as additional context. gpt-3's naive "in-context learning" paradigm picks up to 32 randomly sampled examples, and concatenates them with the input. this method is not guaranteed to prioritize the most informative demonstrations, and mixing random examples from different classes together creates long contexts which can be hard to learn from. additionally, the number of usable demonstrations is bounded by the model's maximum input length. we develop a more refined strategy, where, for each input, we randomly sample a single example at a time from each class to create multiple, minimal demonstration sets. we also devise a novel sampling strategy that pairs inputs with similar examples, thereby providing the model with more discriminative comparisons. we present a systematic evaluation for analyzing few-shot performance on 8 single-sentence and 7 sentence-pair nlp tasks. we observe that given a small number of training examples, (1) promptbased fine-tuning largely outperforms standard finetuning; (2) our automatic prompt search method matches or outperforms manual prompts; and (3) incorporating demonstrations is effective for finetuning, and boosts few-shot performance. together, these simple-yet-effective methods contribute towards a dramatic improvement across the tasks we evaluate on, and we obtain gains up to 30% absolute improvement (11% on average) compared to standard fine-tuning. for instance, we find that a roberta-large model achieves around 90% accuracy on most binary sentence classification tasks, while only relying on 32 training examples. we refer to our approach as lm-bff, better few-shot fine-tuning of language models: a strong, taskagnostic method for few-shot learning. automatic prompt search. @xcite and @xcite explore ways of identifying label words automatically, however, none of these results lead to better performance compared to hand-picked ones. in contrast, our method searches over both templates and label words, and is able to match or outperform our manual prompts. several other attempts have been made in addition-yet these approaches either op-erate in limited domains, such as finding patterns to express specific relations @xcite , or require a large number of examples for gradientguided search @xcite . our approach aims to develop general-purpose search methods that rely only on a few annotations. fine-tuning of language models. a number of recent studies have focused on better methods for fine-tuning language models @xcite @xcite . these works mainly focus on optimization and regularization techniques to stabilize fine-tuning. here we use standard optimization techniques, and instead mainly focus our efforts on better prompt-based fine-tuning in a more extreme few-shot setting. we anticipate that results of these studies are largely complementary to ours. few-shot learning. broadly speaking, our setting is also connected to other few-shot learning paradigms in nlp, including (1) semi-supervised learning @xcite @xcite , where a set of unlabeled examples are given; (2) meta-learning @xcite @xcite , where a set of auxiliary tasks are given; and (3) intermediate training @xcite , where a related, intermediate task is given. we deviate from these settings by making minimal assumptions about available resources: we only assume a few annotated examples and a pre-trained language model. our focus is on understanding how far we can push without any other advantages. ## problem setup task formulation. in this work, we assume access to a pre-trained language model l that we wish to fine-tune on a task d with a label space y. for the task, we only assume k training examples per class foot_2 for the task's training set d train , such that the total number of examples is k @xmath0, and our goal is then to develop task-agnostic learning strategies that generalize well to an unseen test set (x test in , y test ) ∼ d test . for model selection and hyper-parameter tuning, we assume a development set d dev , of the same size as the few-shot training set, i.e., |d dev | = |d train |. this distinction is important: using a larger development set confers a significant advantage (see our experiments in appendix a), and subverts our initial goal of learning from limited data. 4 for all of the following experiments (unless specified otherwise), we take @xmath1. evaluation datasets. we conduct a systematic study across 8 single-sentence and 7 sentence-pair english tasks, including 8 tasks from the glue benchmark @xcite , snli @xcite , and 6 other popular sentence classification tasks (sst-5, mr, cr, mpqa, subj, trec). all of the dataset details are provided in appendix b. for single-sentence tasks, the goal is to make a prediction based on an input sentence x @xmath2, such as whether a movie review is positive or not. for sentence-pair tasks, the goal is to take a pair of input sentences x @xmath3, x 2 ) and predict the relationship between them. we also interchangeably refer to the inputs @xmath4, <s 2 >). note that we mainly use sst-2 and snli for pilot experiments and model development, making it close to a true few-shot setting, at least for all the other datasets we evaluate on. evaluation protocol. systematically evaluating few-shot performance can be tricky. it is wellknown that fine-tuning on small datasets can suffer from instability @xcite , and results may change dramatically given a new split of data. to account for this, we measure average performance across 5 different randomly sampled d train and d dev splits. this issue has also been discussed in schick and schütze (2021b)they suggest using a fixed set of training examples. we argue that sampling multiple splits gives a more robust measure of performance, and a better estimate of the variance. we also observe that hyperparameters can make a significant difference, thus we sweep multiple hyper-parameters for each data sample, and take the best setting as measured on the d dev of that sample (see appendix c.1). ## prompt-based fine-tuning given a masked language model l, we first convert input x in to a token sequence x, and the language model l then maps x to a sequence of hidden vectors {h @xmath5. during standard finetuning, we usually take @xmath6, we train a task-specific head, softmax(w o h [cls] ), by maximizing the log-probability of the correct label, where h [cls] is the hidden vector of [cls], and w @xmath7. similarly, for a regression task, we can introduce w @xmath8. in either case, the number of new parameters can be substantialfor example, a simple binary classification task will introduce 2,048 new parameters for a robertalarge model-making it challenging to learn from a small amount of annotated data (e.g., 32 examples). an alternative approach to solving this problem is prompt-based fine-tuning, in which l is directly tasked with "auto-completing" natural language prompts. for instance, we can formulate a binary sentiment classification task using a prompt with input x 1 (e.g., "no reason to watch it .") as: and let l decide whether it is more appropriate to fill in "great" (positive) or "terrible" (negative) for @xcite . we now formalize this approach for classification and regression ( §4.1 and §4.2), and discuss the importance of prompt selection ( §4.3). ## automatic prompt generation we now explore principled ways of automating the search process for label words ( §5.1) and templates ( §5.2). our goals are to reduce the human involvement required to design prompts, and to find more optimal settings than those that we manually choose. here, we assume a classification task, but the process for regression is analogous. ## fine-tuning with demonstrations in this section, we study whether we can leverage demonstrations when fine-tuning medium-sized lms, and find better ways to exploit them. ## experiments we present our main results, and address several research questions pertaining to our lm-bff approach. implementation details are in appendix c. ## discussion reformulating nlp tasks as mlm has exciting implications for few-shot learning, but also has limitations. first, while lm-bff greatly outperforms standard fine-tuning, with respect to automatic prompt generation, despite its effectiveness, we still find it practically challenging to expand the search space, or generalize well based on only approximately 32 examples. this is partly due to our lingering reliance on some manual design-either manual templates (for label word search) or manual label words (for template search), which allows us to get our search off the ground, but does also bias it towards areas of the search space that we might have already imagined. finally, it is important to clarify that lm-bff favors certain tasks which (1) can be naturally posed as a "fill-in-the-blank" problem; (2) have relatively short input sequences; and (3) do not contain many output classes. issues (2) and ( 3 ) might be ameliorated with longer-context language models (e.g., @xcite . for tasks that are not straightforward to formulate in prompting, such as structured prediction, issue (1) is more fundamental. we leave it as an open question for future work. ## conclusion in this paper we presented lm-bff, a set of simple but effective techniques for fine-tuning language models using only a few examples. our approach proposes to (1) use prompt-based finetuning with automatically searched prompts; and