LEA : Meta Knowledge-Driven Self-Attentive Document Embedding for Few-Shot Text Classification (2022)

Abstract: Text classification has achieved great success with the prosperity of deep learning and pre-trained language models. However, we often encounter labeled data deficiency problems in real-world text-classification tasks. To overcome such challenging scenarios, interest in few-shot learning has increased, whereas most few-shot text classification studies suffer from a difficulty of utilizing pre-trained language models. In the study, we propose a novel learning method for learning how to attend, called LEA, through which meta-level attention aspects are derived based on our meta-learning strategy. This enables the generation of task-specific document embedding with leveraging pre-trained language models even though a few labeled data instances are given. We evaluate our proposed learning method on five benchmark datasets. The results show that the novel method robustly provides the competitive performance compared to recent few-shot learning methods for all the datasets.

## introduction a deficiency of supervised data is often experienced in real-world nlp applications. few-shot learning aims to yield an ai-driven nlp model capable of recognizing unseen tasks using a few labeled data. meanwhile, fine-tuning pre-trained models (ptms) @xcite @xcite has been the most successful approach in recent years of nlp. unfortunately, it is still challenging to utilize ptms @xcite in few-shot learning. to address this subtle problem @xcite , we propose a meta-knowledge driven self-attentive embedding transfer method, called lea (learningto-attend), based on a novel meta-learning framework, through which meta-level attention aspects are derived by encoding how to attend for given tasks. lea is an efficient and practical method that facilitates the utilization of large-sized ptms in few-shot learning. there are the two common transfer learning paradigms in nlp: feature-based transfer @xcite and fine-tuning @xcite . our approach belongs to the feature-based transfer. lea includes two key ideas: (1) construction of a meta-level attention aspects dictionary and (2) inference of the task-specific attention aspects upon the arrival of a new task. the former is a process by which useful meta-level attention aspects across tasks are derived based on a particular ptm via our meta-learning framework. the latter refers to as a task-adaption process, where a subset of taskspecific attention aspects is inferred by determining the top-k most relevant attention aspects from the meta-level attention aspects dictionary. while lea can be applied to a wide variety of downstream tasks, we demonstrate lea on few-shot text classification problems in the paper. ## related work few-shot text classification: in @xcite , induction is proposed to build classwise embedding to represent each class using a particular dynamic routing algorithm coalesced with meta-learning. in @xcite , ds is introduced to keep track of underlying word distributions across all available classes and to specify important lexical features for new classes. meta-learning: as a metric learning-based method, @xcite suggested a deep neural network, called a prototype network (proto), through which class representations are composed using a learning similarity metric for members of the same class. in @xcite , similar to proto, a deep neural network, called a relation network, is proposed to learn a non-linear distance metric rather than the euclidean distance. in addition, leo @xcite learns a lowdimensional latent embedding of the model parameters such that the classifiers are generated from the latent space into which the tasks are mapped. frog-gnn @xcite focuses on all querysupport pairs and proposes a multi-perspective aggregation based graph neural network to explicitly reflect intra-class similarity and inter-class dissimilarity. ## proposed method the overall architecture of lea is shown in figure for number of tasks in batch do 6: sample task instance τ i ∼ s tr 7: decide top-k weights g τ i using c τ i 8: ## experimental results we evaluated lea on five text datasets -20 newsgroup @xcite , huffpost headline @xcite , reuters-21578 @xcite , rcv-1 @xcite , and amazon product reviews @xcite -and compared it with current state-of-art methods. we conducted two different experiments: 5-way 1-shot and 5-way 5shot all over datasets. the details of the datasets are introduced in appendix a.1. ## conclusion we hypothesized that a type of task-specific selfattentive mechanism might improve few-shot learning performance, especially when it is prohibitive to fine-tune a large-sized ptm. we have attempted to design a novel embedding transfer method for deriving a meta-level attention aspects dictionary to enable a new task to simply borrow the most relevant attention aspects from the dictionary. as a result, we proposed a novel meta-learning framework for the learning-to-attend and showed that lea is an effective method that facilitates the utilization of large-sized ptms in few-shot learning.