The Benefits of Label-Description Training for Zero-Shot Text Classification (2023)

Abstract: Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data in order to classify among specific label sets in downstream tasks. We propose a simple way to further improve zero-shot accuracies with minimal effort. We curate small finetuning datasets intended to describe the labels for a task. Unlike typical finetuning data, which has texts annotated with labels, our data simply describes the labels in language, e.g., using a few related terms, dictionary/encyclopedia entries, and short templates. Across a range of topic and sentiment datasets, our method is more accurate than zero-shot by 17-19% absolute. It is also more robust to choices required for zero-shot classification, such as patterns for prompting the model to classify and mappings from labels to tokens in the model’s vocabulary. Furthermore, since our data merely describes the labels but does not use input texts, finetuning on it yields a model that performs strongly on multiple text domains for a given label set, even improving over few-shot out-of-domain classification in multiple settings.

## introduction pretrained language models (plms) @xcite @xcite @xcite have produced strong results in zero-shot text classification for a range of topic and sentiment tasks, often using a pattern-verbalizer approach @xcite . with this approach, to classify the restaurant review "overpriced, salty and overrated!", a pattern like "the restaurant is @xcite " is appended to the review and verbalizers are chosen for each label (e.g., "good" for positive sentiment and "bad" for negative). the text is classified by the pretrained masked language modeling (mlm) head to choose † co-senior authors. label input business business finance business is the activity of making one's living or making money by producing or buying and selling products. . . sports sports racing an athletic activity requiring skill or physical prowess and often of a competitive nature, as racing, baseball. . . (a) topic classification label input very negative awful it was terrible. a horrendous experience. very positive great just fantastic. overall, it was outstanding. (b) sentiment classification the most probable verbalizer for the [mask] position. 1 although effective, the approach is sensitive to the choice of specific pattern/verbalizer pairs, with subtle changes in the pattern, the verbalizer, or both, often having a large impact on performance (van de @xcite . to alleviate these issues, we propose a simple alternative approach of training on small curated datasets intended to describe the labels for a task. unlike typical training datasets, which consist of input texts annotated by hand with labels, our data contains only the descriptions of the labels. we refer to this data as labeldesc data and show a few examples for topic and sentiment classification in next, we finetune pretrained models using the pattern-verbalizer approach on labeldesc data and evaluate them for text classification. for topic classification, we use patterns and verbalizers from @xcite to train on our la-beldesc examples by finetuning the model as well as the mlm head (see section 3 for details). we refer to training on labeldesc data as la-beldesctraining. in experiments, we show that labeldesctraining consistently improves accuracy (average improvement of 17-19%) over zero-shot classification across multiple topic and sentiment datasets ( we then conduct additional experiments to reveal the value of labeldesctraining under various circumstances. to study the impact of verbalizer choice, we experiment with uninformative (randomly initialized) and adversarial (intentionally mismatched) verbalizers (section 4.2.1). while accuracy drops slightly, both settings are still much more accurate than zero-shot classification with its original verbalizers. that is, labeldesctrain-ing is able to compensate for knowledge-free or even adversarial verbalizer choice. we also compare to finetuning a randomly initialized classifier head without any patterns or verbalizers, again finding accuracy to be higher than zero-shot (section 4.2.2). collectively, our results demonstrate that labeldesctraining leads to strong performance that is less sensitive than zero-shot classification in terms of pattern/verbalizer choice, while also not requiring a pretrained mlm head. since labeldesc data focuses entirely on the labels without seeking to capture the input text distribution, we would hope that it would exhibit stable performance across datasets with the same labels. so, we compare labeldesctrain-ing to the approach of training on a small super-vised training set from one domain and testing on another (section 4.2.4). in multiple cases, la-beldesctraining actually attains higher accuracy than few-shot supervised learning tested on out-of-domain test sets, even when hundreds of manually labeled training examples are used (albeit from a different input domain). in summary, this paper shows several benefits of labeldesctraining. first, once a practitioner identifies a label set of interest for zero-shot classification, it only requires a few minutes to collect the kind of labeldesc data shown in we evaluate on two types of tasks: topic classification on agnews, yahoo answers, and dbpedia @xcite and sentiment classification on the stanford sentiment treebank (sst) @xcite , yelp reviews @xcite , imdb @xcite , and amazon reviews polarity @xcite . we consider both binary and 5-way classification for sst and yelp datasets (denoted as sst-2, sst-5, yelp-2, and yelp-5 henceforth) and only binary for imdb and amazon (denoted as imdb and amz-2 henceforth). 3 below we describe how we construct la-beldesc data for each label set. dataset statistics as well as all labeldesc data are in section a.5 in the appendix. ## experimental settings the following settings are used in our experiments. unless stated otherwise, we use the pretrained roberta-base (b) and roberta-large (l) models @xcite for all experiments since roberta is the predominant choice in related zeroshot and dataless research @xcite @xcite . additionally, for every dataset, we use the entire available test sets for evaluation. zero-shot classification baseline. we use the standard "pattern-verbalizer" approach for topic and sentiment classification. the set of verbalizers used can be found in each pattern comprises a prompt including a [mask] symbol placed before or after the text input, and we aim to predict the masked token. for example, a prompt is added after the input x to frame classification as a question answering task, e.g., "x question: what is the topic of this newsgroup? answer: @xcite ." we use roberta- labels 1. world 2. sports 3. business 4. sci/tech verbalizers 1. world 2. sports 3. business 4. tech label: sports 1. label term: sports 2. related term: racing 3. wikipedia: sport pertains to any form of competitive physical activity or ... 4. dictionary: an athletic activity requiring skill or physical prowess and often of a competitive nature, as racing, baseball… … select verbalizers construct labeldesc data create input model text input (label desc. data + pattern ) 1) "sports question: what is the topic of this article? answer: [mask]." 2) "racing question: what is the topic of this article? answer: [mask]." 3) "sport pertains to any form of competitive physical activity or ... question: what is the topic of this article? answer: [mask]." … finetune model to predict the correct verbalizer at [mask] "need for carbon sink technologies climate scientists tell a conference that greater efforts should be made to pull co2 from the atmosphere." "need for carbon sink technologies climate scientists tell a conference that greater efforts should be made to pull co2 from the atmosphere. question: what is the topic of this article? answer: [mask]." finetuning inference prediction: sci/tech test data from agnews test data + pattern we present text inputs labeled as "sports" from the topic classification task, and use one of our patterns (see labeldesctraining. we use the same settings as the zero-shot baseline except that we finetune the models on labeldesc data. we do not use any target task data for tuning or early stopping. instead, we fix hyperparameter values, including number of training steps, by tuning on 20ng following the process described below. we used labeldesc data for the four selected 20ng labels as our training data and the original 20ng data (training and test sets) as our dev set, restricted to the four selected labels shown in section 2. we preprocessed the data by removing headers, quotes, and footers. we used a batch size of 1 and tuned over a set of five learning rates ({5e-7, 1e-6, 5e-6, 1e-5, 5e-5}). models were trained for 3500 training steps, evaluating on the dev set after each epoch, i.e., every 24 training steps since it's the size of labeldesc dataset for 20ng. based on tuning accuracies, we chose learning rate 5e-7 and number of training steps 2160 for roberta-base and 1920 for roberta-large. additionally, we explored variations of parameter freezing, such as freezing certain layers of roberta. the best setting on 20ng was to freeze the lower half of the layers (excluding the embedding layer) during finetuning, so we used this for experiments reported below. 4 ## results and analysis in this section we first present the results that are obtained via labeldesctraining and then analyze the benefits of labeldesc data with a range of additional experiments and analysis. ## related work one common approach in zero-shot text classification is to transfer knowledge from seen labels @xcite , which requires observed labels and a notion of label similarity. some sources of semantic knowledge used for this purpose include multiple modalities @xcite , label relationships in knowledge graphs @xcite , and word representations @xcite . there are several other approaches to zero-shot classification. to classify documents, @xcite ## conclusions we presented labeldesctraining, a method for improving the accuracy of zero-shot classification by using small, curated datasets that simply describe the labels for a task in natural language. our method is 17-19% more accurate than zero-shot on average across a range of datasets. labelde-sctraining is also more robust to the choices required for zero-shot classification, such as patterns and verbalizers. furthermore, labeldesc data is domain agnostic and therefore can used for any text classification task as long as it contains the same set of labels. labeldesctraining can even outperform a supervised approach that uses training data from a different domain. one future direction would be to apply the idea to structured prediction, nli, and natural language generation tasks. another would be to investigate ways to reduce the dependence of pretrained models on patterns and verbalizers, such as directly calibrating the marginal probabilities of verbalizers with the goal of minimizing biases of pretrained models.