Z ero AE : Pre-trained Language Model based Autoencoder for Transductive Zero-shot Text Classification (2023)

Abstract: Many text classification tasks require handling unseen domains with plenty of unlabeled data, thus giving rise to the self-adaption or the so-called transductive zero-shot learning (TZSL) problem. However, current methods based solely on encoders or decoders overlook the possibility that these two modules may promote each other. As a first effort to bridge this gap, we propose an autoencoder named ZeroAE. Specifically, the text is encoded with two separate BERT-based encoders into two disentangled spaces, i.e., label-relevant (for classification) and label-irrelevant respectively. The two latent spaces are then decoded by prompting GPT-2 to recover the text as well as to further generate text with labels in the unseen domains to train the encoder in turn. To better exploit the unlabeled data, a novel indirect uncertainty-aware sampling (IUAS) approach is proposed to train ZeroAE. Extensive experiments show that ZeroAE largely surpasses the SOTA methods by 15.93% and 8.70% on average respectively in the label-partially-unseen and label-fully-unseen scenario. Notably, the label-fully-unseen ZeroAE even possesses superior performance to the label-partially-unseen SOTA methods.

## introduction collecting human-labeled data often comes at a high cost for many nlp tasks, since it typically requires domain expertise and massive labeling efforts @xcite . it is therefore desirable and beneficial to consider the challenging (generalized) zero-shot learning (zsl) that aims to adapt a learner to unseen domains or even unseen tasks without any annotated data @xcite . in particular for the text classification problem, @xcite defines zsl under two scenarios: label-partially-unseen and label-fully-unseen. the former demands domain adaptation that generalizes the model to classify text of unseen classes whose labeled data are unavailable during training. as a further step, the latter requires generalpurpose zsl models for new task adaption without requiring labeled data at all. the key to zsl lies in improving the generalization performance by utilizing external knowledge @xcite . since pretrained language models (plms) memorize rich sources of external knowledge when being pretrained on a large text corpus, they can serve as an extremely powerful hammer for zsl. existing plm-based zsl approaches concentrate on either encoder-based (i.e., discriminative) or decoder-based (i.e., generative) models. 2 specifically, encoder-based methods @xcite @xcite typically treat text classification as a text entailment (te) or a qa task, and fine-tune bert or roberta @xcite as the embedding function to match the texts and labels in the seen classes. these methods then generalize to the unseen classes using the same embedding function and select labels that can best match the text semantically. as pointed out in @xcite , the bert-based models could suffer from the issue of large uncertainty for unseen class generalization. hence, labeled data are still required to stabilize the performance. on the other hand, decoder-based methods @xcite attack the zsl problem from the aspect of data augmentation. they employ gpt-2 to generate training data for the unseen classes given the labels, and next train a classifier based on the augmented data. unfortunately, the data yielded by gpt-2 (i.e., the decoder-based methods) may contain a large portion of low-quality samples that are detrimental to the training of the classifier. worse still, gpt-2 in these approaches cannot be fine-tuned in an end-to-end manner to soften this issue. although gpt-3 can alleviate the data quality problem to some extent, the dauntingly huge size precludes its widespread use @xcite . one promising solution to the above issues is to combine the encoder and the decoder-based models: the former may help to filter out the data of low quality given by the latter, while the latter can generate data with labels to boost the performance of the former. apart from plms, another source of external knowledge is unlabeled data. in practice, we often have access to abundant unlabeled data, and such data can assist in familiarizing the domain (or task) agnostic plms with the target domain (or task) @xcite . the resulting zero-shot learning with unlabeled data is called transductive (generalized) zero-shot learning (tzsl). one appealing approach for tzsl is to involve the unlabeled data in a self-training loop @xcite @xcite by iterating between 1) estimating pseudo labels for all unlabeled data given an encoder-based model (e.g., a bert-based te model) and 2) refining the encoder-based model using the pseudo labels with high confidence. however, these selftraining methods may lead to the problem of error accumulation @xcite , that is, the mistakenly pseudo-labeled data in one iteration can severely affect subsequent iterations and the final predictions. in this paper, we propose an autoencoder framework for tzsl. it harnesses the strength of both plms and unlabeled data, while at the same time bringing the best from the encoder-based and the decoder-based methods together. we name the resulting model zeroae and it is to our knowledge the first approach that aims to solve the tzsl problem in nlp from the perspective of autoencoders. particularly, we specify the encoder and the decoder to be fine-tuned bert and gpt-2 respectively. to enable the two plms to promote each other and to further self-adapt to the task at hand, we design two main types of data flows in zeroae: text reconstruction flow and label reconstruction flow. the first one aims to recover the text data after inputting it to the encoder and subsequently the decoder, while the second tries to recover the label after first generating text given the label via the decoder and then predicting the label given the generated text via the encoder. furthermore, we assume that the latent space can be split into two parts: label-relevant and label-irrelevant. these two parts are discrete (vector-quantized), disentangled, and are given by two different encoders (i.e., fine-tuned bert). only the label-relevant part is used for classification to remove the interference from the label-irrelevant part, while both parts are required for text reconstruction and generation. additionally, to better handle the unlabeled data, we also adopt contrastive learning, and further propose a simple yet effective method named indirect uncertainty-aware sampling (iuas) to train ze-roae, allowing the model to pay more attention to those unlabeled data with high uncertainty as the training process proceeds and lowering down the uncertainty with the assistance of gpt-2. in summary, our key contributions are: ## zeroae we begin this section by defining the transductive (generalized) zero-shot learning (tzsl) problem. let d @xmath0, y s i ) and d @xmath1, y u i ) denote the data for seen and unseen classes respectively, where x i is the text, y i is the label corresponding to x i , and i is the sample index. note that y i can only take values from {y 1 , • • • , y c }, where y j denotes the class name for class j and c is the number of classes. for label-partiallyunseen tzsl, labeled data from d s and unlabeled data from d u are utilized to train the model. we then test the model under the assumption that the labels are from both seen and unseen classes. the objective of label-partially-unseen tzsl is to conduct domain adaption automatically from seen to unseen classes. on the other hand, for label-fullyunseen tzsl (a.k.a. extremely weakly supervised learning @xcite ), we take one step further, and only use unlabeled data from both d s and d u to train the model. this challenging task requires the model to self-adapt to the new task at hand given no labeling information for the task. in order to solve the tzsl problem, we propose zeroae in this paper, whose overall architecture is shown in figure 1 . as mentioned in the introduction, there are two major types of data flows in zeroae, namely, the text reconstruction flow (denoted by the blue arrows) and the label reconstruction flow (denoted by the orange arrows). to provide an illuminating overview of zeroae, we mainly follow the text reconstruction flow and describe an example of how zeroae processes one sentence x i for the purpose of topic categorization. suppose that the text x i is "could stress or an unhealthy diet trigger lung cancer?". we first encode the text into two disentangled latent spaces z r and z i with two separate encoders (see 1 and 2 ). the first latent space z r characterizes the label-relevant information, such as "stress", "unhealthy diet", and "lung cancer", while the second z i represents the label-irrelevant information, such as the syntax "could ... ?". the label-relevant information is further inputted into a classifier (see 3 ) in order to find the correct topic, i.e., "health". a discriminator (see 4 ) is introduced to guarantee that the two latent spaces are disentangled so the label-irrelevant information cannot adversely affect the classifier. finally, the two latent embeddings are fed into the decoder (see 5 ) so as to recover the original text x i . next, we elaborate on each module of zeroae. for ease of exposition, all notations in this paper are summarized in ## conclusion in this paper, we propose a plm-based autoencoder named zeroae for zero-shot text classification. the autoencoder framework enables the pretrained encoder and decoder to further complement and promote each other. furthermore, the proposed iuas training algorithm helps zeroae to deal with unlabeled data. experiments on realworld datasets demonstrates that zeroae provides a much better solution to the domain adaptation (i.e., label-partially-unseen) and task adaptation (i.e., label-fully-unseen) problems in comparison with the sota methods.