It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2021)

Abstract: When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much “greener” in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.

## introduction pretraining ever-larger language models (lms) on massive corpora has led to large improvements in nlp @xcite @xcite . a standard approach is to replace the pretrained model's output layer with a task-specific head and finetune the entire model on a set of labeled training data. however, language modeling is not only a powerful pretraining objective, but many tasks can be reformulated as cloze questions (e.g., by appending phrases such as "the correct answer is __"), allowing pretrained lms to solve them without any or with only very few labeled examples @xcite . recently, @xcite introduced gpt-3, a pretrained lm with an enormous 175 billion parameters, and showed that it has amazing few-shot abilities: by reformulating tasks as lm problems, 1 our implementation is publicly available at https:// github.com/timoschick/pet . gpt-3 achieves near state-of-the-art results for some superglue @xcite tasks given just 32 labeled examples. this is achieved through priming: gpt-3 is given a few demonstrations of inputs and corresponding outputs as context for its predictions, but no gradient updates are performed. while being straightforward to use, this method has two major drawbacks: in this work, we adapt pet for tasks that require predicting multiple tokens. we then show that in combination with albert @xcite , pet and its iterative variant (ipet) both outperform gpt-3 on superglue with 32 training examples, while requiring only 0.1% of its parameters (figure 1 ). moreover, training with pet can be performed in several hours on a single gpu without requiring expensive hyperparameter optimization. finally, we show that similar performance can also be achieved without unlabeled data and provide a detailed analysis of the factors contributing to pet's strong performance: its ability to combine multiple task formulations, its resilience to wordings that are hard to understand, its usage of labeled data, and characteristics of the underlying lm. given pet's "green" properties, we see our work as an important contribution to an environmentally sound nlp. ## related work enabling lms to perform zero-shot learning by providing task descriptions was proposed by @xcite and has been applied to text classification @xcite , commonsense knowledge mining @xcite and argumentative relation classification @xcite . it is also commonly used for probing the knowledge contained within lms @xcite @xcite @xcite . as finding ways to reformulate tasks as cloze questions that are understood well by lms is difficult @xcite , @xcite propose pet, a method that uses knowledge distillation @xcite and self-training (e.g., @xcite @xcite to easily combine several reformulations. our modified version of pet uses masked language models @xcite to assign probabilities to sequences of text; this is similar to using them in a generative fashion @xcite and has previously been investigated by @xcite and @xcite . in contrast to pet, which uses gradient-based optimization, @xcite ## pattern-exploiting training let m be a masked language model (mlm), t its vocabulary and __ ∈ t the mask token; we denote the set of all token sequences as t * . for some @xmath0, we denote with q k m (t | z) the probability that m assigns to t at the kth masked position in z; the model's logits before applying softmax are denoted with s k m (t | z). we consider the task of mapping inputs @xmath1, for which pet requires a set of pattern-verbalizer pairs (pvps). each pvp @xmath2, v) consists of • a pattern p : @xmath3; as illustrated in figure 2 , the core idea of pet is to derive the probability of y being the correct output for x from the probability of v(y) being the "correct" token at the masked position in p (x). based on this intuition, a conditional probability distribution q p of y given x is defined as for a given task, identifying pvps that perform well is challenging in the absence of a large development set. therefore, pet enables a combination of multiple pvps @xmath4, . . . , p n } as follows: as steps (2) and (3) above closely resemble knowledge distillation @xcite , we also refer to them simply as distillation. importantly, this process does not require holding the entire ensemble of mlms in memory at the same time as each model's predictions can be computed sequentially; therefore, it is not more memory expensive than using a single model. to give mlms trained on different patterns further opportunity to learn from one another, schick and schütze (2021) also propose ipet, an iterative variant of pet in which several generations of models are trained on datasets of increasing size that are labeled by previous generations. this is achieved as follows: first, an ensemble of mlms is trained as in regular pet. for each model m i , a random subset of other models is used to generate ## experiments we compare pet and gpt-3 on superglue @xcite , a natural language understanding benchmark consisting of eight challenging tasks. we cannot evaluate pet using the exact same training data as gpt-3 because for most tasks, gpt-3 uses a different set of training examples for each test example and for the other tasks, training sets were not available upon request; however, the exact choice of examples has little impact on gpt-3's performance. 4 we thus create new training sets by randomly selecting 32 examples for each task using a fixed random seed. we additionally create sets of up to 20,000 unlabeled examples for each task; this is done by removing all labels from the original training sets. we refer to the resulting sets of training examples and unlabeled examples as fewglue. foot_3 4.1 tasks below, we describe each of the superglue tasks and our corresponding pvps. we use a vertical bar (|) to mark boundaries between text segments. of the eight tasks considered, only copa, wsc and record require the use of pet with multiple masks as introduced in section 3.1. boolq @xcite is a qa task where each example consists of a passage p and a yes/no question q. we use the following patterns: cb @xcite and rte @xcite are textual entailment tasks like mnli, so we use pvps similar to schick and schütze (2021). for a premise p and hypothesis h, we use and a verbalizer that maps entailment to yes, disagreement to no and neutral to maybe. given a premise p, the task in copa @xcite is to determine the cause or effect of the premise given two options c 1 and c 2 . for determining the effect, we use the following patterns: for determining the cause, we use the same patterns but replace so with because. the verbalizer for c 1 and c 2 is the identity function. for wic (pilehvar and camacho-collados, 2019), given a word w and two sentences s 1 and s 2 in which it occurs, the task is to decide if w is used with the same sense in both sentences. we use: for the first two patterns, we use yes as verbalization for words used in the same sense and no for other words; for the third pattern, we use b and 2. for wsc @xcite , each example consists of a sentence s with a marked pronoun p and noun n, and the task is to determine whether p refers to n. we follow @xcite and treat wsc as a generative task. we highlight p in s by putting it in asterisks and use the following patterns: we use the identity function as verbalizer for n. note that wsc is different from other tasks in that it requires free-form completion. this in turn requires some modifications during training and inference that are discussed in appendix a. multirc @xcite is a qa task. given a passage p, a question q and an answer candidate a, the task is to decide whether a is a correct answer for q. we use the same verbalizer as for boolq and similar patterns: for record @xcite , given a passage p and a cloze question q, the task is to decide which of a given set of answer candidates is the correct replacement for the placeholder in the cloze question. as this task is already presented in the form of a cloze question, there is little room for designing pvps, so we only use a trivial one: the concatenation of p and q as pattern and the identity function as verbalizer. with only one pvp, there is no need to perform knowledge distillation so we directly use the resulting model as our final classifier. ## analysis we investigate the importance of several factors for few-shot performance: the choice of patterns and verbalizers, the usage of both unlabeled and labeled data, and properties of the underlying language model. we also look into our proposed modification for pet to work with multiple masks and compare it to various baselines. finally, we measure how choosing different sets of training examples affects performance. our analysis focuses on pet as gpt-3 is not publicly available. foot_4 ## conclusion we have proposed a simple yet effective modification of pet, enabling us to use it for tasks that require predicting multiple tokens. in extensive experiments, we have identified several factors responsible for the strong performance of pet combined with albert: the possibility to concurrently use multiple patterns for transforming examples into cloze questions, the ability to compensate for patterns that are difficult to understand, the usage of labeled data to perform parameter updates, and the underlying lm itself. we have shown that using pet, it is possible to achieve few-shot text classification performance similar to gpt-3 on superglue with lms that have three orders of magnitude fewer parameters. this not only lowers financial cost, but above all reduces environmental impact immensely and leads to a much smaller carbon footprint. we see this as an important contribution to achieving the goal of an environmentally more friendly nlp. to enable comparisons with our work, we make our code, models and datasets publicly available. for future work, it would be interesting to see whether pet also works for generative tasks when combined with generative lms and whether further improvements are possible in multi-task settings. alec radford, jeff wu, rewon child, david luan, dario amodei, and ilya sutskever. 2019. language models are unsupervised multitask learners. technical report. colin raffel, noam shazeer, adam roberts, katherine lee, sharan narang, michael matena, yanqi zhou, wei li, and peter j. liu. 2020. exploring the limits of transfer learning with a unified text-totext transformer. journal of machine learning research, 21(140):1-67. julian salazar, davis liang, toan q. nguyen, and katrin kirchhoff. 2020. masked language model scoring. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 2699-2712, online. association for computational linguistics. victor sanh, lysandre debut, julien chaumond, and thomas wolf. 2019. distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. in proceedings of the 5th workshop on energy efficient machine learning and cognitive computing, neurips 2019. victor sanh, thomas wolf, and alexander rush. 2020. movement pruning: adaptive sparsity by finetuning. in advances in neural information processing systems, volume 33, pages 20378-20389. curran associates, inc. timo schick and hinrich schütze. 2020. rare words: a major problem for contextualized embeddings and how to fix it by attentive mimicking. in proceedings of the thirty-fourth aaai conference on artificial intelligence. timo schick and hinrich schütze. 2021. exploiting cloze questions for few shot text classification and natural language inference. in proceedings of the 16th conference of the european chapter of the association for computational linguistics, kyiv, ukraine (online). international committee on computational linguistics.