Improving and Simplifying Pattern Exploiting Training (2021)

Abstract: Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few-shot learning without any unlabeled data and introduce ADAPET, which modifies PET’s objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data.

## introduction pre-trained language models (lms) have shown significant gains across a wide variety of natural language processing (nlp) tasks in recent years @xcite @xcite . most of these gains are obtained by fine-tuning language models on labeled data for a particular task. however, performance can suffer when there is very limited labeled data available for a downstream task @xcite . recently, gpt-3 @xcite demonstrated how language models, when scaled to hundreds of billions of parameters, can learn well when primed with only a few labeled examples. however, the scale of gpt-3 (175b parameters) makes it impractical to study. there is, therefore, a need to develop smaller language models that can work equally well with limited labeled data. pattern-exploiting training (pet; @xcite reformulates natural language understanding tasks as cloze-style questions and performs gradient-based fine-tuning. in doing so, pet outperforms gpt-3 with few labeled examples using albert @xcite . however, pet uses additional task-specific unlabeled data. we propose adapet (a densely-supervised approach to pattern exploiting training) that uses more supervision by decoupling the losses for the label tokens and a label-conditioned masked language modeling (mlm) objective over the full original input. on superglue @xcite with 32 labeled examples per task, adapet outperforms ipet without any unlabeled data. ## background cloze-style questions and mlm. a cloze task is a problem where certain parts of a text are removed, and the goal is to replace the missing portion based on the context @xcite . here, the text that has some parts removed is considered a cloze-style question. inspired by cloze tasks, bert introduces the mlm objective that tries to predict the original word at the masked out here, the blue boxes refer to the inputs from a task (entailment, in this case). figure 2a shows the decoupling label objective. the model has to predict the correct and incorrect labels at the masked out position, using a bce loss over all labels. for the label conditioning objective in figure 2b , the input text either includes the correct or incorrect label. at a randomly masked out position, the model should predict the original token when the input text has the correct label, and should not predict the original token when the input text has an incorrect label. notation. let g represent a language model, x represent the input example converted into a cloze-style question, and y represent the label at the masked location m. we are interested in the quantity [[g m (x)]] z which represents the logit value for a specific token z at the mask location m. ## adapet our proposed approach, called adapet, modifies the objective from pet so that it can provide more supervision and learn without task-specific unlabeled data. ## results and analyses we run experiments on superglue, and follow the same data split as @xcite , which consists of 32 labeled examples for each task. our code is implemented in pytorch @xcite using huggingface @xcite . we use the same pre-trained model and hyperparameters as pet, except we increased the number of training batches to 1k and choose the best checkpoint on the dev set, since it has been shown that training longer can help even with few samples @xcite . for all ablation experiments, we only use the first pattern 3 and train for 250 2 this assumes the context only makes sense with the correct label. empirically though, we find this to be reasonable. 3 the first pattern for each task can be found in app. a.1 batches. we refer the reader to appendix b for more details. since we do not assume access to unlabeled data (see section 2.1), we do not apply the three-step training procedure of pet and ipet to adapet. we still assume access to the full development set to choose the best masking ratio and checkpoint model, since pet presumably used the full development set to choose their hyperparameters which we copy. ## conclusion in this paper, we propose adapet, a new method for few-shot natural language understanding. crucially, our work does not use unlabeled data and instead leverages more supervision to train the model. assuming the same data budget, our model outperforms gpt-3 on superglue using just 0.1% as many parameters. however, our method has limitations; for example, we use a naive random masking strategy, which might not make sense for label conditioning. future work could look into better masking strategies for labeled conditioned mlm, such as masking important tokens based on the the gradients of the logits for an example, as has been done for interpreting models @xcite . verbalizer: c 1 /c 2