Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2021)

Abstract: Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with “task descriptions” in natural language (e.g., Radford et al., 2019). While this approach underperforms its supervised counterpart, we show in this work that the two ideas can be combined: We introduce Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task. These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set. For several tasks and languages, PET outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.

## introduction learning from examples is the predominant approach for many nlp tasks: a model is trained on a set of labeled examples from which it then generalizes to unseen data. due to the vast number of languages, domains and tasks and the cost of annotating data, it is common in real-world uses of nlp to have only a small number of labeled examples, making few-shot learning a highly important research area. unfortunately, applying standard supervised learning to small training sets often performs poorly; many problems are difficult to grasp from just looking at a few examples. for instance, assume we are given the following pieces of text: with the rise of pretrained language models (plms) such as gpt @xcite , bert @xcite and roberta @xcite , the idea of providing task descriptions has become feasible for neural architectures: we can simply append such descriptions in natural language to an input and let the plm predict continuations that solve the task @xcite . so far, this idea has mostly been considered in zero-shot scenarios where no training data is available at all. in this work, we show that providing task descriptions can successfully be combined with standard supervised learning in few-shot settings: we introduce pattern-exploiting training (pet), a semi-supervised training procedure that uses natural language patterns to reformulate input examples into cloze-style phrases. as illustrated in figure 1 , pet works in three steps: first, for each pattern a separate plm is finetuned on a small training set t . the ensemble of all models is then used to annotate a large unlabeled dataset d with soft labels. finally, a standard classifier is trained on the soft-labeled dataset. we also devise ipet, an iterative variant of pet in which this process is repeated with increasing training set sizes. on a diverse set of tasks in multiple languages, we show that given a small to medium number of labeled examples, pet and ipet substantially outperform unsupervised approaches, supervised training and strong semi-supervised baselines. other approaches for few-shot learning in nlp include exploiting examples from related tasks @xcite @xcite @xcite and using data augmentation @xcite ; the latter commonly relies on back-translation @xcite , requiring large amounts of parallel data. approaches using textual class descriptors typically assume that abundant examples are available for a subset of classes (e.g., @xcite @xcite . in contrast, our approach requires no additional labeled data and provides an intuitive interface to leverage task-specific human knowledge. the idea behind ipet -training multiple generations of models on data labeled by previous generations -bears resemblance to self-training and bootstrapping approaches for word sense disambiguation @xcite , relation extraction @xcite @xcite , parsing @xcite @xcite , machine translation @xcite , and sequence generation @xcite . ## pattern-exploiting training let m be a masked language model with vocabulary v and mask @xmath0, and let l be a set of labels for our target classification task a. we write an input for task a as a sequence of phrases @xmath1, . . . , s k ) with s @xmath2; for example, @xmath3. we define a pattern to be a function p that takes x as input and outputs a phrase or sentence p (x) ∈ v * that contains exactly one mask token, i.e., its output can be viewed as a cloze question. furthermore, we define a verbalizer as an injective function v : @xmath4. we refer to (p, v) as a pattern-verbalizer pair (pvp). using a pvp (p, v) enables us to solve task a as follows: given an input x, we apply p to obtain an input representation p (x), which is then processed by m to determine the label @xmath5. for example, consider the task of identifying whether two sentences a and b contradict each other (label y 0 ) or agree with each other (y 1 ). for this task, we may choose the pattern p (a, b) = a? ## experiments we evaluate pet on four english datasets: yelp reviews, ag's news, yahoo questions @xcite and mnli @xcite . additionally, we use x-stance @xcite to investigate how well pet works for other languages. for all experiments on english, we use roberta large @xcite as language model; for x-stance, we use xlm-r @xcite . we investigate the performance of pet and all baselines for different training set sizes; each model is trained three times using different seeds and average results are reported. as we consider a few-shot setting, we assume no access to a large development set on which hyperparameters could be optimized. our choice of hyperparameters is thus based on choices made in previous work and practical considerations. we use a learning rate of 1 • 10 -5 , a batch size of 16 and a maximum sequence length of 256. unless otherwise specified, we always use the weighted variant of pet with auxiliary language modeling. for ipet, we set λ = 0.25 and @xmath6; that is, we select 25% of all models to label examples for the next generation and quintuple the number of training examples in each iteration. we train new generations until each model was trained on at least 1000 examples, i.e., we set @xmath7. as we always repeat training three times, the ensemble m (or m 0 ) for n pvps contains 3n models. further hyperparameters and detailed explanations for all our choices are given in appendix b. ## analysis combining pvps we first investigate whether pet is able to cope with situations were some pvps perform much worse than others. for |t | = 10, ## conclusion we have shown that providing task descriptions to pretrained language models can be combined with standard supervised training. our proposed method, pet, consists of defining pairs of cloze question patterns and verbalizers that help leverage the knowledge contained within pretrained language models for downstream tasks. we finetune models for all pattern-verbalizer pairs and use them to create large annotated datasets on which standard classifiers can be trained. when the initial amount of training data is limited, pet gives large improvements over standard supervised training and strong semi-supervised approaches.