Label Agnostic Pre-training for Zero-shot Text Classification (2023)

Abstract: Conventional approaches to text classification typically assume the existence of a fixed set of predefined labels to which a given text can be classified. However, in real-world applications, there exists an infinite label space for describing a given text. In addition, depending on the aspect (sentiment, topic, etc.) and domain of the text (finance, legal, etc.), the interpretation of the label can vary greatly. This makes the task of text classification, particularly in the zero-shot scenario, extremely challenging. In this paper, we investigate the task of zero-shot text classification with the aim of improving the ability of pre-trained language models (PLMs) to generalize to both seen and unseen data across varying aspects and domains. To solve this we introduce two new simple yet effective pre-training strategies, Implicit and Explicit pre-training. These methods inject aspect-level understanding into the model at train time with the goal of conditioning the model to build task-level understanding. To evaluate this, we construct and release UTCD, a new benchmark dataset for evaluating text classification in zero-shot settings. Experimental results on UTCD show that our approach achieves improved zero-shot generalization on a suite of challenging datasets across an array of zero-shot formalizations.

## introduction text classification is the process of categorizing text into sets of organized groups where each set consists of similar content in a well-defined manner @xcite . supervised approaches have achieved great success in recent years due to the availability of rich training data and the advent of large pre-trained language models such as bert @xcite . these conventional approaches typically assume the presence of a pre-defined set of labels to which a given text can be classified. however, in real-world applications, several challenges emerge: 1) the label space is constantly evolving. over time, new labels are constantly emerging and the definition of the label space is constantly being refined. for example, intent classification systems such as those used in chatbots and dialogue systems are constantly introducing new intents as their range of supported features increases. social networks such as twitter encounter new and emerging topics on a daily basis from massive amounts of content that need to be classified. figure 1 shows an example of this emerging label space. zero-shot learning (zsl) is aimed at addressing these constraints. zero-shot learners are models capable of predicting unseen classes. when applied to text classification, these models aim to associate a piece of text with a given label without the need for having been trained on that label. however, despite recent advancements in the capabilities of plms, zero-shot models still vastly underperform their supervised counterparts @xcite @xcite . as such, this remains an open research problem. in this paper, we investigate the challenge of reducing the aforementioned performance gap present in these zero-shot models compared to their supervised counterparts on unseen data. we theorize that the poor generalization of these zero-shot models is due to their lack of aspect-level understanding during their training process. to alleviate this we introduce two new simple yet effective pre-training strategies, implicit and explicit pre-training which specifically inject aspect-level understanding into the model. in order to evaluate these strategies, we canvas the range of zero-shot formalizations for enabling zero-shot text classification on plms and apply our techniques. additionally, we introduce the universal text classification dataset (utcd), a largescale text classification dataset for evaluating zeroshot text classification. utcd is a compilation of 18 classification datasets spanning 3 main aspects of sentiment, intent/dialogue, and topic classification. our results on utcd show that by employing both our implicit and explicit pre-training strategies we can achieve improved zero-shot performance on a suite of challenging datasets for which the model was not trained on. specifically, this paper makes the following contributions: in this section, we introduce the task of zero-shot text classification and describe a set of formalizations for facilitating the classification of text in a zero-shot manner, i.e. being able to predict unseen labels. conventional text classification text classification approaches using plms assume the existence of a pre-defined set of labels ty i u 1 n where for a given input sequence x, the model outputs a representation of that sequence as a sequence of hidden states th i u 1 l . hidden states in the final layer are pooled to a single vector. in the case of bert @xcite , the rclss token is taken, and a linear softmax layer is added to predict the probability distribution of the label set: for the zero-shot scenario, this approach breaks since the output class set ty i u 1 n is fixed. this prevents the classification of text to new labels unless the model is re-trained with the new label set or a mapping of existing labels to unseen labels is built, both of which are impractical and cumbersome for real-world scenarios. ## utcd: universal text classification dataset in order to test the zero-shot generalization of these nlp models we introduce utcd. utcd is a compilation of 18 classification datasets spanning 3 main aspects of sentiment, intent/dialogue, and topic classification. a breakdown of each dataset is provided in appendix a. utcd focuses on the task of zero-shot text classification where the candidate labels are descriptive of the text being classified. to make nlp models more broadly useful, zeroshot techniques need to be capable of label, domain & aspect transfer. as such, in the construction of utcd we enforce the following principles: textual labels in utcd, we mandate the use of textual labels. while numerical label values are often used in classification tasks, descriptive textual labels such as those present in the datasets across utcd enable the development of techniques that can leverage the class name which is instrumental in providing zero-shot support. as such, for each of the compiled datasets, labels are standardized such that the labels are descriptive of the text in natural language. ## experimental setup model architectures for binary classification, we use bert base with sentence pair classification as in @xcite . for dual encoding classification, we use sentence-bert @xcite encoder, mean pooling, and cosine similarity as the distance metric. for generative classification, we use the 345m gpt-2 @xcite as the language model and the input representation described in @xcite . these models are denoted binary bert, bi-encoder, and gpt-2 respectively. training we train all models with adamw @xcite and weight decay of 0.01 on all in-domain data for 3 epochs, for both pre-training and fine-tuning stages. for explicit pre-training, we use a learning rate of 2e-5, batch size of 16, and linear learning rate warmup over the first 10% steps with a cosine schedule. for binary and dual encoding we use a learning rate of 2e-5, batch size of 16, with 10% warmup and a linear schedule. for generative classification fine-tuning, we use a learning rate of 4e-5, batch size of 128, with 1% warmup and a cosine schedule as reported in @xcite . we pre-process data and train all models with different random seeds over multiple runs. ## results & discussion in this section we present and analyze the results of our experiments, detailing our insights and discussing the implications of each of our techniques. evaluation task we report accuracy on the test set of all in-domain and out-of-domain datasets. in multi-label cases where there is more than one valid label, the prediction is considered correct if the model predicts any one of the correct labels. for generative classification, we observe instances in which gpt-2 may not generate one of the label options, a known problem for plm generation @xcite . in such cases, we consider the label option most similar to the generated answer as prediction, by mapping the generated output and the valid classes to an embedding space. for this encoding, we use the pre-trained model mpnet @xcite with mean pooling encoder from sentence-bert @xcite for mapping the labels and cosine similarity as the distance metric. this ensures the consistency of gpt-2's output with the other zero-shot formalizations. ## related work zero-shot text classification is the task of classifying text into novel categories unseen during training. early zero-shot classification studies frame the problem as binary classification on whether a given label describes the text @xcite . with the advancement of plms, subsequent works @xcite rely on transformer architectures to learn representations from descriptive labels passed in. in particular, @xcite fine-tune an autoregressive language model to generate titles based on a prompt template containing tweet articles and a list of title options. though the model is trained on a great variety of title options, the approach limits the learning to topic classification only, as the authors only analyze performance on topic datasets, unlike our approach which considers a wide array of aspects, each requiring focus on different sections of a given text. @xcite similarly categorize zero-shot text classification by aspects and implicitly introduce aspects during training with a dedicated template for each aspect. they further propose the classification of a text, label pair as a logic entailment problem. however, the authors analyze a less challenging zero-shot case where a model is trained on a subset of text, label pairs, and evaluated on the remaining text with unseen labels in the same domain. additionally, the authors introduce wordnet definition of the labels as the labels are all single words. this process requires manual intervention and is not applicable for multiple-word label sequences common in intent classification, such as "check balance". our work evaluates a more diverse set of datasets for each aspect and a more comprehensive set of zero-shot architectures. ## conclusion in this paper, we investigate the task of zero-shot text classification with the aim of improving the ability of plms to generalize both seen and unseen data across domains without the need for additional training. we introduce two new simple yet effective pre-training strategies, implicit training & explicit pre-training which specifically inject aspect-level understanding into the model at train time. to evaluate this, we release utcd, a new benchmark dataset for evaluating text classification in zeroshot settings. experimental results on utcd show that our approach achieves improved zero-shot generalization on a suite of challenging datasets in utcd and across many zero-shot formalizations.