Sentence-aware Adversarial Meta-Learning for Few-Shot Text Classification (2022)

Abstract: Meta-learning has emerged as an effective approach for few-shot text classification. However, current studies fail to realize the importance of the semantic interaction between sentence features and neglect to enhance the generalization ability of the model to new tasks. In this paper, we integrate an adversarial network architecture into the meta-learning system and leverage cost-effective modules to build a novel few-shot classification framework named SaAML. Significantly, our approach can exploit the temporal convolutional network to encourage more discriminative representation learning and explore the attention mechanism to promote more comprehensive feature expression, thus resulting in better adaptation for new classes. Through a series of experiments on four benchmark datasets, we demonstrate that our new framework acquires considerable superiority over state-of-the-art methods in all datasets, increasing the performance of 1-shot classification and 5-shot classification by 7.15% and 2.89%, respectively.

## introduction deep learning usually relies on a vast amount of labeled examples to accomplish tasks. however, this requirement is problematic for few-shot text classification when only a few examples are available in novel classes, which leads to poor model generalization for new tasks. motivated by the fact that humans can quickly recognize new knowledge after learning a few examples, few-shot learning is becoming a hot research topic. early studies exploit data augmentation and regularization procedures @xcite to deal with the overfitting due to data sparseness. more recent research mainly falls into two approaches: (1) transfer-learning based methods @xcite , which transfer and propagate knowledge attained from the source domain to classify unseen examples in the target domain. (2) meta-learning based methods @xcite , which learn knowledge by repeating lots of meta-tasks in a training episode manner and leverage knowledge extracted to briskly predict new samples during meta-testing. specifically, bao et al. @xcite incorporated distributional signatures of words into the meta-learning framework to make impressive performance. han et al. @xcite first proposed to introduce an adversarial domain adaptation network to strengthen the generalization ability of the meta-learning system. despite the remarkable progress of few-shot classification approaches @xcite @xcite , most existing metalearning models still suffer tough challenges: example diversity. even examples in the same class have various representations, bringing about the difficulty of extracting generic features based on a few training examples and the urgent demand for strong adaptability of the model to new tasks. in this paper, we propose a straightforward but remarkably powerful framework to cope with the above challenges. we deploy an adversarial network architecture to train the meta-learning system and create a novel framework sentence-aware adversarial meta-learner (saaml). specifically, we employ the temporal convolutional network (tcn) @xcite and the multilayer perceptron network (mlp) to build the generator and discriminator, respectively. the model can grasp sentences' inherent semantic information through the adversarial training of the generator and the discriminator. then we artfully fuse the word embeddings and the features of the generator to construct high-quality discriminative sentence features. moreover, we further build a feature enhancer (fe) to leverage the multi-head attention mechanism to fine-tune the features of support examples and query examples, aiming to create more compatible feature representations. our research methodically develops how to better exploit the adversarial network architecture and sentence-aware interaction knowledge to boost few-shot text classification performance. the main innovations of our research are as follows: to evaluate the effectiveness and robustness of the proposed model, we conduct massive comparison experiments and ablation studies on four datasets. the results indicate that our new framework saaml outperforms stateof-the-art approaches with nearly 5.02% performance improvement, and the proposed enhancement modules are more beneficial and flexible. ## related work few-shot text classification is a crucial application scenario of few-shot learning in natural language processing (nlp), which has obtained increasing attention and research. transfer learning @xcite , as a feasible approach to tackle fewshot text classification, intends to make the knowledge learned from the source domain more compatible with the target domain and reduce the shift between different domains. for example, tzeng et al. @xcite leverages the adversarial domain adaptation framework to bridge the domain gaps without example constraints. moreover, with the rapid development of pre-trained language models, it also exhibits excellent performance by fine-tuning the representations of training examples, such as bert @xcite and gpt-3 @xcite . as the dominant program in few-shot text classification, meta-learning @xcite systems develop rapidly and achieve great success. meta-learning approaches are mainly divided into two categories: (1) optimization-based methods, which resort to "learning to fine-tune" strategy to train the model. for example, @xcite @xcite develop an optimization procedure of model parameter initialization to quickly obtain outstanding performance after a small amount of gradient update steps based on few-shot training examples.(2) metricbased methods, which complete the classification task through a specific distance metric, .e.g., matching network @xcite is calculated by the cosine similarity, while the euclidean distance is the measure standard of prototypical network @xcite . relational network @xcite and gnn network @xcite utilize convolutional neural networks and graph neural networks to learn metric functions dynamically. recently, bao et al. @xcite argues that statistical information of sentences plays a vital role in classification tasks. sun et al. @xcite subtly combines data augmentation with meta-learning to generate more diverse samples, preventing model overfitting. han et al. @xcite is the first to explore the adversarial domain adaptation network for the performance improvement of meta-learning framework. ## preliminary the few-shot text classification is usually viewed as a n -way k-shot task. firstly, the input of the model is a set with multiple labeled examples, including support set c train and query set c test , where c train are disjoint from c test . then the meta-learner conducts the episode-based strategy @xcite to train a classifier on c train . finally, the meta-learner accomplishes the goal to predict new examples of c test with the classifier. for the episode-based strategy, we randomly select n classes (n -way) from c train , and then separately sample k examples (k-shot) as the support set s and p examples as the query set q from each of these chosen classes, which can be denoted as: ## our approach our proposed saaml is broadly built to pursue the excellent performance of few-shot text classification. the overall architecture of saaml is shown in figure 1 . first, we leverage the representation generator with tcn as the core block to capture comprehensive semantic information of sentences and exploit the meta discriminator to strengthen the learning ability of the model. then, we contrive a feature enhancer to further fine-tune the example vectors with sentence-aware knowledge for more consistent representations. finally, we jointly train all modules of saaml with different schemes in an end-to-end manner. ## discussion our work follows the research idea proposed by @xcite to exploit the adversarial network to catch feature representations. however, the implementation structure of mlada @xcite is too simple to extract intrinsic sentence feature information and strengthen the generalization ability of the model well. in contrast, we upgrade the whole meta-learning framework with more cost-effective modules. the experimental results in specifically, we build the generator with the temporal convolutional network (tcn) rather than bidirectional lstm (bilstm) to excavate the inherent semantic knowledge of sentences. we also exploit the feature enhancer to grasp the rich feature interaction between different sentence representations through the multi-head attention mechanism. moreover, we adopt the prototype network (pn) based on the difference loss as the classifier, improving the model's discriminative ability. overall, our model saaml can construct more comprehensive transferable representations and accomplish excellent performance. in addition, we further discuss the adaptation potential of the saaml and mlada. we adopt rrml @xcite and pn (snell ## conclusion in this paper, we propose a novel meta-learning framework sentence-aware adversarial meta-learner (saaml) to address few-shot text classification task. exactly, under the architecture of the adversarial network, we explore the representation generator with tcn as the core module to encourage more discriminative representation learning and exploit the feature enhancer to facilitate more consistent and comprehensive feature expression, which exceedingly strengthens the adaptability and generalization of saaml for new classes. we develop comprehensive experiments on four benchmark datasets to demonstrate that the proposed model gains substantial improvements over existing state-of-the-art meta-learning approaches.