Zero-Shot Text Classification via Self-Supervised Tuning (2023)

Abstract: Existing solutions to zero-shot text classification either conduct prompting with pre-trained language models, which is sensitive to the choices of templates, or rely on large-scale annotated data of relevant tasks for meta-tuning. In this work, we propose a new paradigm based on self-supervised learning to solve zero-shot text classification tasks by tuning the language models with unlabeled data, called self-supervised tuning. By exploring the inherent structure of free texts, we propose a new learning objective called first sentence prediction to bridge the gap between unlabeled data and text classification tasks. After tuning the model to learn to predict the first sentence in a paragraph based on the rest, the model is able to conduct zero-shot inference on unseen tasks such as topic classification and sentiment analysis. Experimental results show that our model outperforms the state-of-the-art baselines on 7 out of 10 tasks. Moreover, the analysis reveals that our model is less sensitive to the prompt design. Our code and pre-trained models are publicly available at https://github.com/DAMO-NLP-SG/SSTuning .

## introduction recent advances in pre-trained language models (plms) have brought enormous performance improvements in a large variety of nlp tasks @xcite . these paradigm shifts towards leveraging generic features learnt by plms are driven by the high data cost required for learning each new nlp task afresh. one promising learning method that echoes this paradigm shift is zero-shot text classification, which predicts text labels on unseen tasks. zeroshot text classification has attracted considerable research attention in recent years @xcite ; @xcite , as labeled data is no longer a necessity for relearning new feature representations for untrained specific tasks. existing studies on zero-shot text classification can be briefly classified into two types, as shown in figure 1 . the first type is prompting, which uses plms to predict labels with designed templates and verbalizers (figure 1 (a) ). this can be achieved by leveraging the generation capability of large language models @xcite , or reformulating text classification task as a mask-filling task @xcite . likewise, generation-based methods @xcite and mining-based methods (van de @xcite also rely on prompting to generate or filter noisy labeled samples, which are used for further fine-tuning. the second type is meta-tuning which fine-tunes a plm on a collection of labeled data of related tasks before conducting inference on unseen tasks (figure 1 (b )). by reformulating the annotated data into instruction templates @xcite , question-answer pairs @xcite , multiple-choice questions @xcite or entailment pairs @xcite @xcite , and fine-tuning on them, plms perform well on unseen tasks. despite the achieved performance, existing methods have several limitations. prompting has shown to be sensitive to the choice of patterns and verbalizers (van de @xcite . this makes it difficult to design different templates specifically for each task. in addition, generation-based and mining-based methods require fine-tuning plms for each downstream task, which is inefficient for deployment. on the other hand, meta-tuning relies on labeled data of relevant tasks or in specific formats to facilitate the learning of desired patterns. the requirement for such large-scale annotated data narrows its application scope. to address the above issues, we propose to leverage self-supervised learning (ssl) for zero-shot text classification tasks. ssl has been widely used during the pre-training stage of plms to alleviate the need for large-scale human annotations @xcite ) by exploiting the intrinsic structure of free texts. therefore, with a suitable ssl objective, the model is able to capture certain patterns with the auto-constructed training data and can be applied to a wide range of downstream tasks in a zero-shot manner without specific designs. to our best knowledge, this is the first work to exploit ssl at the tuning stage for zero-shot classification, which we refer to as selfsupervised tuning (sstuning). the biggest challenge of applying sstuning to zero-shot text classification tasks is to design a proper learning objective that can effectively construct large-scale training samples without manual annotations. intuitively, the core of the text classification task can be treated as associating the most suitable label to the text, given all possible options. motivated by this observation, we propose a new learning objective named first sentence prediction (fsp) for the sstuning framework to capture such patterns. in general, the first sentence tends to summarize the main idea of a paragraph. therefore, predicting the first sentence with the rest of the paragraph encourages the model to learn the matching relation between a text and its main idea ("label"). to generate training samples, we use the first sentence in the paragraph as the positive option and the rest as text. the first sentences in other paragraphs are used as negative options. specifically, if negative options are from the same article as the positive option, they are regarded as hard negatives since the sentences in the same article normally have some similarities, such as describing the same topic. hard negatives may force the model to learn the semantics of the text instead of simply matching the keywords to complete the task. in the inference phase, we convert all possible labels of a sample into options, which can be done in two simple ways: 1) use original label names; 2) convert labels using the templates (like "this text is about [label name]"). then the text and options are combined to create the final input. the tuned model can thus retrieve the most relevant option as the predicted label of the text. since the tuned model has seen a large number of samples and various first sentences as options, which has a higher chance to consist of similar options to the ones at the inference phase, its performance is less sensitive to verbalizer design. in this way, our sstuning enables efficient deployment of plm for classifying texts of unseen classes on-the-fly without requiring further tuning with labeled data or unlabeled in-domain data. our main contributions are: our analysis further demonstrates that our model is less sensitive to prompt design. ## proposed method in this section, we discuss our proposed framework, sstuning, and provide details for our dataset preparation process using the idea of first sentence prediction (fsp), the tuning phase, and the zero-shot inference phase. ## conclusions in this work, we propose a new learning paradigm called sstuning for zero-shot text classification tasks. by forcing the model to predict the first sentence of a paragraph given the rest, the model learns to associate the text with its label for text classification tasks. experimental results show that our proposed method outperforms state-of-the-art baselines on 7 out of 10 tasks and the performance is more stable with different verbalizer designs. our work proves that applying self-supervised learning is a promising direction for zero-shot learning. in the future, we plan to apply sstuing to other tasks by designing proper learning objectives.