MEAL : Stable and Active Learning for Few-Shot Prompting (2023)

Abstract: Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high variance both across different sets of few shots (*data selection*) and across different finetuning runs (*run variability*). This is problematic not only because it impedes the fair comparison of different approaches, but especially because it makes few-shot learning too unreliable for many real-world applications. To alleviate these issues, we make two contributions for more stable and effective few-shot learning: First, we propose novel ensembling methods and show that they substantially reduce *run variability*. Second, we introduce a new active learning (AL) criterion for *data selection* and present the first AL-based approach specifically tailored towards prompt-based learning. In our experiments, we show that our combined method, MEAL (**M**ultiprompt finetuning and prediction **E**nsembling with **A**ctive **L**earning), improves overall performance of prompt-based finetuning by 2.3 points on five diverse tasks. We publicly share our code and data splits in https://github.com/akoksal/MEAL.

## introduction pretrained language models (plms) are effective few-shot learners when conditioned with a few examples in the input @xcite or finetuned with a masked language modeling objective on samples converted into cloze-style phrases @xcite . prompt-based finetuning is especially promising as it enables researchers to train relatively small models as few-shot classifiers that can make accurate predictions with a minimal investment of time and effort. however, prompt-based finetuning suffers from high variance. we observe two causes in our experiments: run variability (different seeds) and data selection (different training sets). figure 1 illustrates this for five equal-size training sets and 20 runs for rte @xcite and mrpc @xcite . both sources of variance are of particular concern in few-shot learning. we may get lucky and select a "good" training set. but because no dev set is available there is also a high risk of selecting a "bad" training set, resulting in much lower performance than possible for the available annotation budget. in addition, run variability is a great methodological problem because it means that the exact same experimental setup (except for different random seeds, causing variance in the order of training examples and dropout layers) will give different results. this makes fair comparison of different algorithms and architectures difficult. we propose new approaches to few-shot learning that address both sources of variance. we first focus on run variability and show based on loss/accuracy surface visualizations @xcite that run variability in few-shot learning is different from fullysupervised settings: solutions proposed for finetuning plms @xcite do not work for few-shot prompt-based finetuning. thus, we propose ensemble techniques to stabilize finetuning for different runs. after mitigating the effects of run variability via a more stable finetuning mechanism, we are able to address training data selection. we modify existing active learning (al) algorithms and propose a novel approach for selecting training examples that outperforms prior algorithmsnot just in terms of final accuracy, but also regarding the diversity and representativeness of selected examples. in general, we are, to the best of our knowledge, the first to develop al algorithms tailored to prompt-based finetuning. we combine our contributions -decrease run variance and better training sets for improved performance and stability of few-shot classification -in meal (multiprompt finetuning and prediction ensembling with active learning). meal improves performance of prompt-based finetuning by 2.3 points on five tasks. contributions: few-shot classification with language model prompting. gpt-3 @xcite prepends examples as conditioning to the input during inference, without parameter updates. pet @xcite follows a similar approach with finetuning and achieves comparable results, with fewer parameters. lm-bff @xcite and adapet @xcite extend pet. instability. there are two sources of instability in finetuning plms for few-shot classification: run variability and data selection. run variability comes from finetuning plms with random seeds. mosbach et al. (2021) and dodge et al. (2020a) show that finetuning plms is an unstable process for fully supervised training. recently, zheng et al. (2022) demonstrated that few-shot finetuning also exhibits run variability but their experiments have different training sets in a cross-validation scenario. they do not address how much instability comes from finetuning vs. data selection. our findings suggest that the instability issue exists in few-shot training for the same training set, and existing solutions for fully supervised settings (e.g., dodge et al. (2020a), mosbach et al. (2021)) do not stabilize finetuning for few-shot classification. we propose the run ensemble method to improve the stability of few-shot classification. the second type of instability is training data selection; we target this issue with al. active learning. as collecting labeled data is time-consuming and costly, al has been a crucial part of supervised learning @xcite @xcite . apart from efficiency in data labeling, we show that some training sets have significantly worse performance than others for few-shot classification. @xcite also show that the selection of the few shots matters a lot. hence, we follow an al setup to select informative and diverse training sets for few-shot classification. @xcite recently showed that uncertainty al achieves significant improvements for fully supervised settings in plms. following this, we modify a variety of al algorithms including prior works such as cal @xcite and badge @xcite for few-shot prompt-based finetuning, and propose a novel al algorithm, ipusd. ## multiprompt finetuning let m be a masked plm, t its vocabulary, and @xmath0. we use pattern-exploiting training (pet) @xcite for prompt-based finetuning experiments on few-shot classification without knowledge distillation and unlabeled data. patterns (p) transform an input x into a cloze-style phrase x p with a single mask. verbalizers (v ) convert each label @xmath1, representing the taskspecific meaning of the output label. our prediction for a label is its probability, according to the plm, as a substitution for the mask: ## run variability in few-shot classification, finetuning plms such as albert @xcite with an mlm objective on samples converted into cloze-style phrases @xcite performs comparably to much larger gpt-3 @xcite . just as prompting methods are sensitive to data order @xcite and label distributions @xcite , finetuning plms also exhibits sensitivity and instability as shown by @xcite for a fully supervised setting. we show that the instability of finetuning plms also exists in few-shot prompt-based finetuning. even though prompt-based finetuning does not introduce new parameters like classifier heads as in fully supervised classification, there is variance from dropout and training data order. we conduct experiments with multiprompt finetuning with default pet @xcite settings without knowledge distillation. figure 1 shows that runs with different random seeds for the same training set can vary by as much as 23.5 points. @xcite suggest that longer training with a low learning rate and warm-up reduces run variability of plms. their main motivation is to avoid models ending up in suboptimal training loss regions. however, this is not valid in few-shot prompt tuning as the number of training examples is low, and finetuning achieves almost zero training loss quickly. our initial experiments show that longer training does reduce the standard deviation between different runs, but that it also causes lower mean accuracy for most tasks, of up to 7.3 points. in figure 2 , we analyze run variability, by creating a training loss and validation accuracy surface visualization of two rte runs with the same training set and multiprompt finetuning. the failed model θ f (red) achieves 58.5% validation accuracy while the successful model θ s (green) achieves 71.5%. the two models only differ in finetuning random seed. the figure illustrates the training loss and validation accuracy surfaces for combinations of the model weights of the pretrained model (θ p ), the failed model (θ f ), and the successful model (θ s ). we create a two-dimensional space based on f (a, b) = f (θ p + aδ f + bδ s ), where δ @xmath2, δ @xmath3, and f is loss (left) or accuracy (right). we use 16 values for a and b to plot loss and accuracy surface forms. figure 2 shows that there is a large region @xmath4, dark blue) that includes θ f and θ s . however, most of this region is suboptimal in terms of validation accuracy (right graph). this indicates that our instability problem differs from fully supervised finetuning where large learning rates often result in suboptimal training loss; in contrast, we observe ≈0 training loss for each run, including failed ones. therefore, longer training with a low learning rate and warm-up only leads to finetuned models ending up in a similar re-gion with lower variance, but it causes suboptimal validation accuracy scores; see §6 for more details. to overcome run variability, we propose two ensemble models: we ensemble the logits over runs in ensemble pred and take the average of parameters over runs in ensemble para . we will show that, for five tasks, these (i) reduce the effect of failed runs and run variability and (ii) achieve higher accuracy than accuracy averaged over runs. the prediction of ensemble pred for x is: following work on averaging deep networks (izmailov et al., 2018), we average each parameter of the finetuned plms across runs, resulting in a single model. the prediction of ensemble para for x is the prediction of this single model. ## data selection another important source of variance for few-shot classification is training data selection. figure 1 shows this effect: validation accuracy greatly varies, with a difference of up to 13.7. figure 3 shows how we modify al algorithms for data selection in few-shot prompt-based finetuning. first, we use a plm to get contextual embeddings, logits, and probabilities for each unlabeled example in a zero-shot setting. we exploit here that, due to the cloze-style format, plms can make predictions before any finetuning. second, we apply modified al algorithms for prompts. we select all examples at once to simplify the selection process. for each task, we select 16 * l training examples, where l is the number of labels. ## experiments and results setup we use a diverse set of five classification tasks to compare single to multiprompt finetuning, analyze run variability, and evaluate al algorithms: rte @xcite , sst-2, sst-5 @xcite , trec @xcite , and mrpc @xcite . we use four prompts for each, described in §a.3. we report results on the validation set as we conducted all experiments without hyperparameter tuning by assuming a realistic few-shot scenario in which no dev set is available for tuning. 1 for single prompt and multiprompt finetuning, we use pet's defaults @xcite . for longer training, we compare pet's defaults to @xcite 's adapted settings. pet's defaults: 1e-5 learning rate, 10 epochs, no warm-up. adapted settings: 1e-6 learning rate, 50 epochs, linear scheduled warm-up with a 0.1 ratio. ## analysis we now perform an in-depth analysis of al algorithms to understand their relative performance better and to understand failure cases like sst-5. we believe that these insights will lead to improved al strategies in future work. balancing desiderata in al we investigate three desiderata in al: diversity, representativeness and label entropy. diversity @xcite measures the redundancy/similarity of training examples by calculating the reciprocal of the average distance between unlabeled examples and their nearest training example. representativeness @xcite captures the well-known issue of selecting outlier examples in al; it is calculated as the reciprocal of the average distance between the selected training examples and their k (@xmath5. label entropy @xcite is the kl divergence between the class distribution of the unlabeled data and that of the selected training examples. our algorithm ipusd outperforms prior-work al because it achieves a good balance between the three desiderata of a good few-shot training set: diversity, uncertainty and representativeness. ## conclusion we demonstrate two stability problems of few-shot classification with prompt-based finetuning: instability due to run variability and training data selection. we show that existing solutions for instability fail. we first propose finetuning a single model with multiple prompts. this results in better performance and less model space complexity than finetuning several models with single prompts. we then propose run ensemble techniques that improve stability and overall performance. our setup with less run variability allows us to explore training data selection for prompt-based finetuning in a sufficiently stable experimental setup. we compare a set of modified al algorithms to reduce training data selection instability and improve overall performance. our novel al algorithm, inter-prompt uncertainty sampling with diversity (ipusd), outperforms prior al algorithms (and random selection) for both albert and roberta large . apart from our algorithmic innovations for fewshot prompt-based learning, we hope that our study will support fairer comparison of algorithms and thereby help better track progress in nlp. we publicly share our code and data splits in https: //github.com/akoksal/meal .