Zero-Shot Text Classification with Self-Training (2022)

Abstract: Recent advances in large pretrained language models have increased attention to zero-shot text classification. In particular, models finetuned on natural language inference datasets have been widely adopted as zero-shot classifiers due to their promising results and off-the-shelf availability. However, the fact that such models are unfamiliar with the target task can lead to instability and performance issues. We propose a plug-and-play method to bridge this gap using a simple self-training approach, requiring only the class names along with an unlabeled dataset, and without the need for domain expertise or trial and error. We show that fine-tuning the zero-shot classifier on its most confident predictions leads to significant performance gains across a wide range of text classification tasks, presumably since self-training adapts the zero-shot model to the task at hand.

## introduction large language models have revolutionized the field of natural language processing, leading to great leaps in performance across the nlp task landscape @xcite @xcite . the pretrain-finetune paradigm has led to a significant reduction in the amount of labeled data required for obtaining high performance on downstream tasks. however, the need to collect labeled examples for each target task remains an obstacle, limiting the usage of language models in practice, at scale. thus, the more ambitious vision of a generalpurpose zero-shot model -one that can tackle many different tasks without requiring labeled data -has become an enticing goal for the community. this notion is increasingly gaining attention, with recent works suggesting new paradigms that aim to utilize the language understanding capabilities of large models for the zero-shot scenario. in their pioneering work on more generalpurpose zero-shot models, @xcite propose to formulate text classification tasks as a textual entailment problem @xcite . this mapping enables using a model trained on natural language inference (nli) as a zero-shot text classifier for a wide variety of unseen downstream tasks. the underlying idea is fairly intuitive. to determine if a particular text should be assigned to, e.g., the "sports" class or the "politics" class, one constructs sentences such as "this text is about sports" and "this text is about politics", respectively; the model prediction as to which one is most entailed by the original text can then be used to determine the predicted class label. similarly, some recent works have tried to map even more varied types of nlp tasks into a unified cross-task format @xcite @xcite . such unified task formats enable "meta-tuning" a model using existing labeled data from different tasks. by teaching the model to solve the broader "meta-task", it is then able to cope with a wide variety of unseen tasks at inference time. while zero-shot models hold great promise by eliminating the burden of collecting task-specific labeled data, they often still come at the cost of providing mediocre performance compared to models trained in the conventional supervised learning paradigm. thus, improving the prediction performance of zero-shot models is of great practical importance. one of the simplest and most effective approaches for improving performance of classifiers is self-training @xcite . in this paradigm, a model's own predictions on unlabelled data are leveraged for creating pseudo-labels, which are then used for further training the model. in the original setting of self-training, some labeled data is available for training an initial classifier, and the predictions of the classifier on unlabeled data are used for data augmentation (van en- premise (t) hypothesis entailment scores template class (c) i was thrilled when my paper was accepted to emnlp this example is guilt 0.3 this example is anger 0.1 this example is joy 0.9 figure 1: entailment-based zero-shot classification. a text t serves as the premise while the hypothesis is created by placing the class name c within a template. the class with the highest entailment score, joy in this example, is taken as the label for t. here, we aim to study self-training as a method for improving general-purpose zero-shot models, by adapting them to the task at hand. given the distinct properties of such models, applying selftraining in this scenario is not trivial and poses unique challenges. our approach can be viewed as a further extension of self-training -from unsupervised domain-adaptation to unsupervised taskadaptation, where only unlabeled data is available for the target task. as prominent representatives of general-purpose zero-shot models, in this work we focus on nlibased models @xcite , which are increasingly being utilized for zero-shot classification @xcite @xcite . to the best of our knowledge, this is the first work that explores self-training in the context of general-purpose zero-shot models. we release our code foot_0 , including access to all datasets, and an associated automatic evaluation framework, aiming to facilitate further research along the lines explored here. ## self-training of zero-shot text classifiers in self-training, a model m is applied on a collection of unlabeled examples u . the instances on which m provides the most confident predictions are taken as a pseudo-labeled training set. this set is used to re-train m , giving rise to a new model, m ′ . this procedure can be repeated to obtain m ′′ , and so on, in an iterative manner. next, we describe the motivation of applying self-training to zero-shot text classifiers, and the details of our approach. ## experimental results we set n, the number of training examples per class, to be 1% of the unlabeled set u (i.e., @xmath0. for each dataset and zero-shot model, we perform two iterations of self-training. 4 we test 4 settings of adding contradict examples as described in section 2.2.2. classification accuracy before and after selftraining for all models using the contrast-random setting is shown in fig. 3 compares the 4 settings of selecting negative examples. as can be seen, among the four settings, contrast-furthest yields the poorest re-sults. a possible explanation is that in this setting the negative examples are too trivial for the model. contrast-closest yields better results, probably because the negative examples in this setting are more difficult and thus more informative for the model. however, for this same reason, these pseudo labeled examples are expected to suffer from the largest noise. the best performing settings are contrast-all and contrast-random, which represent a better balance between the informativeness of the negative examples and their noise level. taking the computational costs into account, contrast-random emerges as the preferred setting. ## discussion in this paper we look at the applicability of selftraining for adapting a general-purpose zero-shot model, focusing on the scenario of entailmentbased models. we opted for this specific setting due to the high accessibility of these off-the-shelf models. in other words, given that these models are readily available for use, we ask whether selftraining provides a straightforward way for practitioners to adapt the general model for their downstream task, using only a modest collection of unlabeled data. we show that in this setting self-training does indeed provide value, delivering significant performance gains for text classification. the notion of using self-training as a tool to adapt a general-purpose zero-shot model is not specific to entailment models, nor is it limited to classification tasks. thus, a major avenue for future work would be to explore this combination on models that rely on different kinds of mapping functions or "meta-tasks" for formulating downstream tasks within a generic cross-task framework @xcite @xcite . zero-shot text classification is recently drawing much attention, with prominent recent works showing promising results using different kinds of iterative approaches @xcite . such approaches build their zero-shot classi-fiers from scratch -and therefore typically require larger unlabeled datasets to perform well -whereas we aim to utilize and build on the knowledge contained in general-purpose zero-shot classifiers. exploring ways to combine these differing approaches is left for future work. importantly, while our method does assume the existence of a collection of unlabeled examples, our results show that an order of 10k examples is sufficient to benefit from self-training. moreover, the cross-task effects in section 5.2 demonstrate that even unlabeled examples from a similar domain and/or task may be useful in adapting the generalpurpose model for the downstream task. determining the exact conditions under which self-training is useful for adaptation across tasks is a matter for future study. moreover, it would be interesting to explore the effects of self-training on multiple datasets, akin to works on supervised multi-task fine-tuning (e.g., @xcite . in our work, we select examples for pseudolabeling based solely on model confidence (see §2.2). some self-training works opt for more balanced approaches for example selection, aiming for a more diverse and/or more informative set of examples (e.g., @xcite . it would be interesting to explore such questions in the zero-shot scenario. in addition, in section 2.2 we describe our method to select confident examples, namely by looking at the maximal delta between the highest and second highest prediction scores. other alternatives for choosing confident examples, e.g., by looking at the entropy across classes, could be tested as well. to conclude, the method we proposed in this paper can boost the performance of entailmentbased zero-shot text classifiers, with little effort and a modest amount of domain data. this can prove useful to the many practitioners who benefit from the practicality and accessibility of these models.