TART : Improved Few-shot Text Classification Using Task-Adaptive Reference Transformation (2023)

Abstract: Meta-learning has emerged as a trending technique to tackle few-shot text classification and achieve state-of-the-art performance. However, the performance of existing approaches heavily depends on the inter-class variance of the support set. As a result, it can perform well on tasks when the semantics of sampled classes are distinct while failing to differentiate classes with similar semantics. In this paper, we propose a novel Task-Adaptive Reference Transformation (TART) network, aiming to enhance the generalization by transforming the class prototypes to per-class fixed reference points in task-adaptive metric spaces. To further maximize divergence between transformed prototypes in task-adaptive metric spaces, TART introduces a discriminative reference regularization among transformed prototypes. Extensive experiments are conducted on four benchmark datasets and our method demonstrates clear superiority over the state-of-the-art models in all the datasets. In particular, our model surpasses the state-of-the-art method by 7.4% and 5.4% in 1-shot and 5-shot classification on the 20 Newsgroups dataset, respectively.

## introduction deep learning has achieved great success in many fields but a deficiency of supervised data is often experienced in real-world nlp applications. fewshot text classification aims to perform classification with a limited number of training instances, which is crucial for many applications but remains to be a challenging task. existing approaches for few-shot text classification mainly fall into two categories: i) promptbased learning @xcite @xcite , which utilizes pre-trained language models (plms) to generate a textual answer in response to a given prompt. although producing promising results, these methods suffer from mlada @xcite performs well on the task with high inter-class variance (e.g., task 2: environment, science, world news, taste), while it fails to distinguish the samples from a task with low inter-class variance (e.g., task 1: environment, science, world news, tech). despite the extraordinary effectiveness, we notice that current meta-learning-based approaches may have several limitations. for those methods that learn to represent each class independently in one feature space @xcite , 2019; @xcite , their performance is heavily dependent on the inter-class variance of the support set. specifically, they address the overfitting issue in few-shot learning by directly adopting the hidden features of support samples as a classifier. thus, they can perform well on tasks when the sampled classes are distinct while failing to differentiate classes with similar semantics. as illustrated in figure 1 , mlada @xcite , which leads to state-of-the-art performance, misclassifies the testing samples of science, world news and tech during the testing stage, mostly because science and tech are similar and all three samples contain technology companies, which are difficult to distinguish. if we substitute the support class science with taste, which has clearly different semantic from tech, it can recognize all testing samples except the third one. this example indicates that ignoring task-specific features and treating all tasks identically is inadequate. it is essential to consider the inter-class variance of support sets, particularly when annotated data is scarce. recently, @xcite leveraged distributional signatures to estimate class-specific word importance. however, it requires extracting relevant statistics of each word from the source pool and the support set for each task, which is time-consuming. a natural question arises: how can we design an efficient method capable of capturing both cross-task transferable knowledge and task-specific features to enhance the model's generalization ability? to tackle these issues, we resort to constructing a task-adaptive metric space via the meta-learning framework. figure 2 presents the key idea of the proposed method. intuitively, for comparable classes that cannot be distinguished in the original feature space, if we can project their class prototypes to per-class fixed points, referred to as reference points, in another small space, it is helpful to enhance the divergence between class prototypes in the transformed space. consequently, we pro-pose a novel task-adaptive reference transfer module that uses a linear transformation matrix to project embedding features into a task-specific metric space. in addition, we design a discriminative reference regularization to maximize the distance between transformed prototypes in the task-adaptive metric space for each task. we find the proposed method promotes the model to learn discriminative reference vectors and construct a stable metric space. our key contributions can be summarized as follows. 1) we propose a task-adaptive reference transformation (tart) network for few-shot text classification. the model enhances the generalization by transforming the class prototypes to perclass fixed reference points in task-adaptive metric spaces. 2) we propose a novel discriminative reference regularization to maximize divergence between transformed prototypes in task-adaptive metric spaces to further improve the performance. 3) we evaluate the proposed model on four popular datasets for few-shot text classification. comprehensive experiments demonstrate that our tart consistently outperforms all the baselines for both 1-shot and 5-shot classification tasks. for instance, our model outperforms mlada @xcite model by 7.4% and 5.4% in 1-shot and 5-shot classification on the 20 newsgroups dataset, respectively. transfer learning and prompt learning for plms few-shot text classification relates closely to transfer learning @xcite ) that aims to leverage knowledge from source domains to target domains. fine-tuning pre-trained language models (plms) @xcite @xcite can also be viewed as a type of transfer learning. recently, @xcite proposed a prompt-based approach to fine-tune plms in a few-shot learning setting for similar tasks, which adapts plms to producing specific tokens corresponding to each class, instead of learning the prediction head. metalearning deviates from these settings by learning to quickly adapt the model to different tasks with little training data available @xcite , typically formulated as a n -way k-shot problem. few-shot text classification few-shot text classification has gained increasing attention in recent years. @xcite used an adaptive metric learning approach to select an optimal distance metric for different tasks. induction network @xcite aims to learn an appropriate distance metric to compare validation points with training points and make predictions through matching training points. dmin @xcite utilizes dynamic routing to provide more flexibility to memory-based few-shot learning in order to adapt the support sets better. @xcite leveraged distributional signatures (e.g. word frequency and information entropy) to train a model within a metalearning framework. another group of methods is to improve performance with the help of additional knowledge, including pre-trained text paraphrasing model @xcite and class-label semantic information @xcite . recently, hong and jang (2022) constructed a meta-level attention aspects dictionary and determined the top-k most relevant attention aspects to utilize pre-trained models in few-shot learning. mlada @xcite is an adversarial network, which improves the domain adaptation ability of meta-learning. however, none of these methods consider task-specific features, which is a key factor for few-shot text classification. ## model in this section, we initially discuss the problem setting of few-shot text classification. then, the overview of the proposed tart is presented in section 3.2. the technical details for the task-adaptive reference transfer module and discriminative reference regularization are described in sections 3.3 and 3.4, respectively. ## conclusion in this work, we propose a novel tart for fewshot text classification, which can enhance the generalization by transforming the class prototypes to per-class fixed reference points in task-adaptive