Learn to Adapt for Generalized Zero-Shot Text Classification (2022)

Abstract: Generalized zero-shot text classification aims to classify textual instances from both previously seen classes and incrementally emerging unseen classes. Most existing methods generalize poorly since the learned parameters are only optimal for seen classes rather than for both classes, and the parameters keep stationary in predicting procedures. To address these challenges, we propose a novel Learn to Adapt (LTA) network using a variant meta-learning framework. Specifically, LTA trains an adaptive classifier by using both seen and virtual unseen classes to simulate a generalized zero-shot learning (GZSL) scenario in accordance with the test time, and simultaneously learns to calibrate the class prototypes and sample representations to make the learned parameters adaptive to incoming unseen classes. We claim that the proposed model is capable of representing all prototypes and samples from both classes to a more consistent distribution in a global space. Extensive experiments on five text classification datasets show that our model outperforms several competitive previous approaches by large margins. The code and the whole datasets are available at https://github.com/Quareia/LTA .

## introduction text classification plays an important role in many natural language processing (nlp) applications, such as question classification, news categorization, user intent classification and so on @xcite . although a wide variety of methods have been proved successful in supervised text classification, they often break down when applied to make predictions for incrementally emerging classes without labeled training data @xcite . unlike zero-shot learning (zsl) that aims to classify unseen class instances at test time @xcite , generalized zero-shot learning (gzsl), which we focus on in this work, aims to classify text samples from both previous seen and emerging novel classes. since there is a strong bias towards seen classes @xcite , gzsl is a more challenging yet critical problem. previously methods mainly focus on transductive approaches for generalized zero-shot text classification. @xcite use a graph convolution network to enhance the unseen class label embeddings. @xcite and @xcite generate illusion feature embeddings for unseen classes based on side information, i.e., class-level attributes or text description. more recently, @xcite use reinforced self-training methods to leverage unlabeled data during training. with the assumption that no knowledge about unseen categories is available during the model learning phase, researchers resort to inductive approaches to handle generalized zero-shot text classification. recapsnet @xcite uses a dimensional attention-based intent capsule network and constructs zero-shot class prototypes by similarity matrix transformation. seg @xcite exploits an outlier detection approach that can be directly applied on recapsnet, which discriminates the domain first, then outputs the final class label. however, the existing methods still have two key limitations. firstly, while the goal of these methods is to transfer beneficial knowledge for unseen classes, these models merely learn optimal parameters by minimizing the loss of instances from seen classes, regardless of explicitly calibrating the predictions on unseen classes. therefore, domain bias problem @xcite towards seen classes is not fairly resolved. secondly, although some of them take into account the inter-class relationship when constructing prototypes for unseen classes @xcite , the models keep static no matter what different new classes emerge in future applications. as a result, these models show a large quality gap between instances from seen classes and from emerging unseen classes. to address these problems, motivated by the success of meta-learning in the few-shot learning task @xcite @xcite , we present a novel learn to adapt (lta) network for generalized zero-shot text classification. concretely, the proposed lta learns over multiple learning episodes that mimic gzsl setting explicitly during training, making the learning setting consistent with the test environment and thereby improving generalization. the model notably extends its ability from two views: prototype adaptation and sample adaptation. in each episode, the lta adjusts the representative prototypes of both seen classes and "fake" unseen classes, with the assumption that unseen classes will help in calibrating representation of seen ones and thereby enable the model to learn the classsensitive representations. the updating for all prototypes is then used to generate a set of calibration parameters, called semantic components, to guide the adaptation of sample embeddings, which is designed to compensate for the shrinking features @xcite that are ignored during training if they are not discriminating for seen classes, but could be critical for recognizing unseen classes. the refined sample embeddings are then classified based on similarity scores with all adapted class prototypes. the same setting can be directly applied in testing, where the lta executes class prediction and adapts the learned model rationally in an on-the-fly manner. in summary, our contributions include: (i) we propose a novel learn to adapt (lta) network for generalized zero-shot text classification which is capable of adapting incrementally between seen classes and emerging unseen classes at test time. (ii) we propose a methodology for calibrating both prototypes and sample embeddings to deduce a global representation space, efficiently avoiding over-fitting on seen classes. (iii) experimental results on five generalized zero-shot text classification datasets show that our method outperforms previous methods with a large margin. there are two key issues that gzsl has to address: (1) how to incrementally learn beneficial knowledge for unseen classes from seen ones, and (2) how to tackle the domain bias caused by the extremely imbalanced data of seen and unseen domains. to alleviate the first issue, some of the earliest works on zsl attempt to learn a matching model between instance embedding and class prototype embeddings represented by extra information including class-level attribute, text description, or their combinations @xcite @xcite . in a similar vein, other methods @xcite @xcite also investigate the semantic relationship between the side information for obtaining better prototype representation. nevertheless, these models are trained using data from seen classes and fail to incrementally adapt to emerging new classes. the key problem of the second issue is that the model is trained with data from the seen classes and the parameters are actually optimized on the seen domain, thus they are not aware of unseen classes. assuming the extra information about unseen classes is available, another prominent approach attempts to use generative models to generate virtual samples or features for unseen domains @xcite @xcite . by using synthesized samples, the generative approaches can convert gzsl problem to the conventional supervised learning problem where biases towards seen classes are largely alleviated. additionally, studies also extend to exploit the unlabeled data for unseen classes @xcite @xcite . however, these models assume that they have access to the extra information about the unseen classes, which is not very realistic since often neither the test data nor their label descriptions is available at the training phase (as supposed in this work). in contrast, our model can involve all classes (seen and unseen) jointly during inference, essentially it is trained towards continuous generalization for new classes, hence it is capable to adapt to incoming new classes dynamically. episode-based training in gzsl our approach is primarily based on the episodic training paradigm that has been widely used in few-shot learning (fsl) @xcite @xcite . the primitive goal of episodic training is to quickly learn a meta-task from sampled classes. a particular advantage of episodic training is that, by constructing meta-tasks, the setting of training is consistent with that of testing, which is essential for classification problems. studies extend to exploit episodic training in the "generalized" settings. verma et al. (2020) constructs model-agnostic meta-tasks to train generative models on gzsl. in addition, @xcite utilizes weight generators to update unseen prototypes in generalized fsl(gfsl). subsequently, to update both seen and unseen prototypes, @xcite exploits attention mechanism while @xcite takes advantage of graph neural networks in gfsl. @xcite use a generative network to generate unseen prototypes in gzsl. these methods only consider the prototype adaptation while the sample embeddings are still static whatever the unseen classes are. additionally, @xcite uses distributional signatures to update sample embeddings in gfsl. considering that distributional signatures can be equal for two different tasks, our method uses a novel semantic update extractor to update samples following the prototype adaptation rather than statistical information. a compelling property of our method is that it tackles knowledge transferring and domain bias simultaneously in an episodic training framework by adapting both prototypes and sample embeddings, and draws a fast adaptation to the novel classes without the cost of dramatic damage in discriminating the seen classes. ## conclusion this paper proposed a novel adaptive meta-learning network for generalized zero-shot text classification. the model was trained under a consistent setting with testing. in particular, it efficiently alleviated the bias towards seen classes by utilizing both prototype adaptation and sample adaptation. experiments on five text classification datasets validated that our model achieved compelling results on both seen classes and unseen classes meanwhile was capable of fast adapting to new classes.