M eta SLRCL : A Self-Adaptive Learning Rate and Curriculum Learning Based Framework for Few-Shot Text Classification (2022)

Abstract: Due to the lack of labeled data in many realistic scenarios, a number of few-shot learning methods for text classification have been proposed, among which the meta learning based ones have recently attracted much attention. Such methods usually consist of a learner as the classifier and a meta learner for specializing the learner to different tasks. For the learner, learning rate is crucial to its performance. However, existing methods treat it as a hyper parameter and adjust it manually, which is time-consuming and laborious. Intuitively, for different tasks and neural network layers, the learning rates should be different and self-adaptive. For the meta learner, it requires a good generalization ability so as to quickly adapt to new tasks. Motivated by these issues, we propose a novel meta learning framework, called MetaSLRCL, for few-shot text classification. Specifically, we present a novel meta learning mechanism to obtain different learning rates for different tasks and neural network layers so as to enable the learner to quickly adapt to new training data. Moreover, we propose a task-oriented curriculum learning mechanism to help the meta learner achieve a better generalization ability by learning from different tasks with increasing difficulties. Extensive experiments on three benchmark datasets demonstrate the effectiveness of MetaSLRCL.

## introduction text classification is one of the most concerned tasks in natural language processing (nlp). at present, most text classification methods are based on supervised learning with a large amount of labeled data. but there is not so much labeled data, even source data, in many scenarios (e.g., news classification in specific domains). some distant supervision methods @xcite have thus been proposed to handle this problem. however, * *corresponding author. the concept of few-shot learning was formally put forward by @xcite . they presented a method for learning from classes with few data, by incorporating generic knowledge which may be obtained from previously learned models of unrelated classes. the existing few-shot learning methods can be divided into three categories @xcite , namely, model fine-tuning based (e.g., @xcite ), metric learning based (e.g., @xcite ), and meta learning based methods (e.g., @xcite ). in recent years, meta learning based methods have attracted lots of interests. however, they still suffer from some challenges. a meta learning method is composed of a learner and a meta learner. for the learner, learning rate is crucial to its performance. nevertheless, in existing methods, it is treated as a hyper parameter and needs to be adjusted manually, which is timeconsuming and laborious. intuitively, for different tasks and different neural network layers, their learning rates should be different. on the other hand, the present meta learning methods cannot be quickly generalized to new tasks @xcite and a good generalization ability to new tasks is necessary for the meta learner. and curriculum learning can help models obtain better generalization performance by guiding the training process towards better regions in the parameter space, i.e., into local minima of the descent procedure associated with good generalization @xcite . for the above reasons, we propose a novel meta learning framework, called metaslrcl, for fewshot text classification, which contains two main mechanisms, i.e., self-adaptive learning rates for the learner and a task-oriented curriculum learning mechanism for the meta learner. our general contributions are three-fold. 1) we present a novel meta learning mechanism with self-adaptive learning rates, which enables different tasks and neural network layers to obtain different learning rates; 2) we introduce curriculum learning for the first time, to the best of our knowledge, into fewshot learning. unlike traditional instance-oriented curriculum learning, the proposed task-oriented curriculum learning mechanism gradually learns from different tasks with increasing difficulties; 3) metaslrcl is evaluated with three typical types of text classification, i.e., relation classification, news classification and topic classification, on three benchmark datasets, namely, fewrel80, 20newsgroup and dbpedia ontology, respectively. experimental results demonstrate its superior performance on all datasets. ## notations in meta learning based few-shot text classification, two datasets are given: d train and d test , which have disjoint label sets. t tasks are sampled from d train and the t-th task (@xmath0, t ]), t ask t , consists of a support set s t and a query set q t . following the setting @xcite , we adopt c-way k-shot (hereinafter denoted as cwks) for few-shot text classification, meaning s t contains c classes and each class has k labeled instances. thus, s t can be formulated as s @xmath1 the diagram of the metaslrcl framework. entity o i t in relation classification. moreover, the query set q t contains u t unlabeled instances for each class in s t , where the i-th instance denotes q i t . q t can thus be formulated as q @xmath2. ## the metaslrcl framework metaslrcl is a generic framework, where fewshot learning models of different categories (i.e., model fine-tuning based, metric learning based, and meta learning based) can be adopted as the learner. as shown in figure 1 , metaslrcl consists of three modules coupled with a task-oriented curriculum learning mechanism. the encoder module. this module maps the instances into the semantic space as embeddings via the encoder network. the task-level learning rate module. this module calculates the task-level learning rate via the number of training classes and the distance between different instances in the support set. the layer-level learning rate module. in this module, the layer-level learning rate is selfadaptively obtained based on the meta learning mechanism. this module contains two main parts: the learner as the classifier and the meta learner above the learner, which allocates learning rates for different network layers of the learner. the task-oriented curriculum learning mechanism. it enables metaslrcl to gradually learn from tasks with more and more classes, thus with increasing difficulties, to make the meta learner achieve a better generalization ability. ## conclusion and future work in this paper, we proposed a novel meta learning framework, called metaslrcl, for few-shot text classification. metaslrcl can self-adaptively obtain different learning rates for different tasks and different network layers. moreover, a task-oriented curriculum learning mechanism is introduced into few-shot learning to achieve a better generalization ability for the meta learner. metaslrcl is evaluated with three typical types of text classification, relation classification, news classification and topic classification, on three benchmark datasets, namely, fewrel80, 20newsgroup and dbpedia ontology, respectively. experimental results demonstrate superior performance of metaslrcl on all datasets. in the future, we will explore few-shot learning under the unbalance learning scenarios because they are ubiquitous in the real world.