Adaptive Meta-learner via Gradient Similarity for Few-shot Text Classification (2022)

Abstract: Few-shot text classification aims to classify the text under the few-shot scenario. Most of the previous methods adopt optimization-based meta learning to obtain task distribution. However, due to the neglect of matching between the few amount of samples and complicated models, as well as the distinction between useful and useless task features, these methods suffer from the overfitting issue. To address this issue, we propose a novel Adaptive Meta-learner via Gradient Similarity (AMGS) method to improve the model generalization ability to a new task. Specifically, the proposed AMGS alleviates the overfitting based on two aspects: (i) acquiring the potential semantic representation of samples and improving model generalization through the self-supervised auxiliary task in the inner loop, (ii) leveraging the adaptive meta-learner via gradient similarity to add constraints on the gradient obtained by base-learner in the outer loop. Moreover, we make a systematic analysis of the influence of regularization on the entire framework. Experimental results on several benchmarks demonstrate that the proposed AMGS consistently improves few-shot text classification performance compared with the state-of-the-art optimization-based meta-learning approaches. The code is available at: https://github.com/Tianyi-Lei .

## introduction as a fundamental task of few-shot learning @xcite in natural language processing theme, few-shot text classification @xcite requires a model to predict categories that are not seen in training. meta learning @xcite , which plays a crucial role in general few-shot learning, aims to improve generalization ability and fast adaptation ability of the learner through modelling the distribution of tasks. to adapt few-shot tasks, typical supervised meta-learning methods @xcite model task distributions from a few support tasks over meta-training episodes. subsequently, numerous methods based on meta-learning @xcite @xcite are proposed to solve few-shot text classification problem. within the meta-learning frameworks, @xcite trains an attention-based model to enhance the text representation of distributional signature, @xcite leverages label-semantic augmentation to help bert compensate for the ambiguity of the class definition caused by the limited data, and @xcite strengthens the generalization of a model using an adversarial domain adaptation network. however, these methods are similar to the traditional meta-learning methods, neglecting the overfitting problem caused by utilizing the few number of data in the complicated models under the meta-learning frameworks. to address the above problem in few-shot text classification, several methods are proposed based on a principle i.e., obtaining more task-distribution can ameliorate the risk of over-fitting to the training task distribution. @xcite alleviates overfitting through joint training of self-supervised tasks and classification tasks in pre-trained models. we also follow this method and use a selfsupervised mask token prediction (mtp) task in meta training phase. unfortunately, the increased task distribution generated by this joint training is not always positive for meta-training. in order to further overcome the overfitting challenge in meta-training, we propose the adaptive meta-learner via gradient similarity based on another principle i.e., distinguishing positive and negative features by feature selection of deep model can enhance generalization by alleviating overfitting. in optimization-based meta-learning framework, the gradient contains all the information transmitted from the inner-learner to the outerand (d) are different schemes of our proposed adaptive meta-learner, which distinguish the positive gradient cosine similarity (scheme 1) and negative gradient cosine similarity (scheme 2 ). if the gradient direction obtained on the query set is similar to the gradient direction of the sum of the t updates (black dashed line), ψ 0 is updated in the direction of the sum of all gradients. while if their gradient directions are opposite, we remove the gradient obtained from the query set. learner, including "features" mentioned in above principle. thus, the gradient obtained by baselearner can be regarded as the "features". to compare with other training strategies for meta-learner, we plot figure 1 . other strategies often adopt all the gradient obtained by the base-learner without distinction. they also may consume enormous computing resources for calculating the hessian matrix, sacrifice the stability and accuracy in order to adopt the first-order algorithm, or discard the query set in the training batch in order to simplify the calculation. by contrast, our method only needs to distinguish the gradient similarity between the gradient of the loss on the query set and the current gradient of the base-learner during the meta-training process. subsequently, we utilize the corresponding gradient of its loss to help metalearner quickly adapt to the optimization space. such method selects the more useful gradients for meta-learner in current training batch. in addition, it neither increases the computational complexity nor causes waste of text information in the same training episode. according to the above principles, we propose a novel adaptive meta-learner via gradient similarity (amgs) algorithm based on optimization-based meta learning scheme. we firstly construct the self-supervised task called mask token prediction (mtp) for the base-learner in the inner loop. such approach can generate the extension of the task distribution from unlabeled text and constraint the gradient updating of primary classification task to increase the robustness of the model. moreover, in the outer loop, we utilize the adaptive meta-learner to improve the utilization of the task features from the inner loop. as figure (1) shows, our strategy can more efficiently leverage query set samples in a training episode, which optimizes the scope of gradient optimization. therefore, the adaptive meta-learner directly accomplish additional amelioration of overfitting. the contributions of this paper are summarized as follows: (1) we construct an optimization-based meta-learning framework named amgs and elaborately design a meta-training algorithm to effectively tackle the overfitting issue in few-shot text classification based on two different principles. (2) we propose an adaptive meta-learner that selects the positive gradients and removes the negative gradients to improve the generalization ability of the model on the few-shot task (3) experimental results demonstrate that the proposed amgs outperforms the state-of-the-art optimization-based meta-learning models. auxiliary learning in general, auxiliary learning can assist the main task to learn more accurately and quickly in deep learning @xcite , especially in the multi-task learning field. ssl-reg @xcite builds a regularizer of the loss of self-supervised learning tasks to improve performance on text classification. besides constructing a task, external auxiliary data can also be introduced into the model to obtain more latent information @xcite . similarly, auxiliary tasks are valuable to adapt the meta-learning scheme. maxl @xcite adopts a self-supervised learning scheme to generate auxiliary labels, improving the generalization ability of the primary task in gradient update. furthermore, self-supervised auxiliary tasks can promote fast adaptation during the testing phase @xcite . hybrid smlmt @xcite creates a specific self-supervised auxiliary task for multi-task learning. similar to these auxiliary tasks, our auxiliary task mtp is self-supervised to generate richer task distribution during meta-training. ## methods in this section, we first introduce the preliminaries for few-shot classification @xcite . bert support set query set classifier predictor next, we describe adaptive meta-learner via gradient similarity (amgs) method in detail. ## conclusion in this paper, we present an adaptive meta-learner via gradient similarity (amgs) framework for few-shot text classification. to be specific, we first leverage the self-supervised mask token prediction (mtp) task to enrich the task distribution with the unlabeled text. such approach can reduce the impact of overfitting caused by the mismatching between the few samples and the deep model. secondly, we construct an adaptive meta-learner via gradient similarity for the outer loop to distinguish the positive and negative gradient. thus, the meta-learner alleviates overfitting by preventing the influence of negative features. experimental results validate that our model achieves significant improvement on the few-shot text classification tasks by effectively alleviating the overfitting issue.