Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks (2020)

Abstract: Self-supervised pre-training of transformer models has revolutionized NLP applications. Such pre-training with language modeling objectives provides a useful initial point for parameters that generalize well to new tasks with fine-tuning. However, fine-tuning is still data inefficient — when there are few labeled examples, accuracy can be low. Data efficiency can be improved by optimizing pre-training directly for future fine-tuning with few examples; this can be treated as a meta-learning problem. However, standard meta-learning techniques require many training tasks in order to generalize; unfortunately, finding a diverse set of such supervised tasks is usually difficult. This paper proposes a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text. This is achieved using a cloze-style objective, but creating separate multi-class classification tasks by gathering tokens-to-be blanked from among only a handful of vocabulary terms. This yields as many unique meta-training tasks as the number of subsets of vocabulary terms. We meta-train a transformer model on this distribution of tasks using a recent meta-learning framework. On 17 NLP tasks, we show that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning. Furthermore, we show how the self-supervised tasks can be combined with supervised tasks for meta-learning, providing substantial accuracy gains over previous supervised meta-learning.

## introduction self-supervised learning has emerged as an important training paradigm for learning model parameters which are more generalizable and yield better representations for many down-stream tasks. this typically involves learning through labels that come * correspondence: tbansal@cs.umass.edu naturally with data, for example words in natural language. self-supervised tasks typically pose a supervised learning problem that can benefit from lots of naturally available data and enable pre-training of model parameters that act as a useful prior for supervised fine-tuning @xcite . masked language modeling @xcite , and other related approaches @xcite @xcite , is an example of such a self-supervised task that is behind the success of transformer models like bert. while self-supervised pre-training is beneficial, it has been recently noted that it is not data-efficient and typically requires large amounts of fine-tuning data for good performance on a target task @xcite . this can be evaluated as a few-shot learning problem, where a model is given only few examples of a new task and is expected to perform well on that task. this paper focuses on this problem of few-shot learning and develops models which demonstrate better few-shot generalization to new tasks. large scale pre-training suffers from a train-test mismatch as the model is not optimized to learn an initial point that yields good performance when fine-tuned with few examples. moreover, finetuning of a pre-trained model typically introduces new random parameters, such as softmax layers, and important hyper-parameters such as learning rate, which are hard to estimate robustly from the few examples. thus, we propose to remove this train-test mismatch, and treat learning an initial point and hyper-parameters jointly from unlabelled data, which allows data-efficient fine-tuning, as a meta-learning problem. meta-learning, or learning to learn @xcite , treats learning a parameterized algorithm, such as a neural net optimized with sgd, that generalizes to new tasks as a learning problem. this typically assumes access to a distribution over tasks in order to enable learning. creating tasks which enable meta-learning is one of the main challenges for meta-learning @xcite @xcite , and typical supervised meta-learning approaches create task distributions from a fixed task dataset with large number of labels by subsampling from the set of labels @xcite . while this enables generalization to new labels, it limits generalization to unseen tasks due to over-fitting to the training task distribution @xcite . moreover, large supervised datasets with a large label set are not always available for meta-learning, as is often the case in many nlp applications. to overcome these challenges of supervised meta-learning, we propose a self-supervised approach and create the task-distribution from unlabelled sentences. taking inspiration from the cloze task @xcite , we create separate multiclass classification tasks by gathering tokens-to-be blanked from a subset of vocabulary words, allowing for as many unique meta-training tasks as the number of subsets of words in the language. the proposed approach, which we call subset masked language modeling tasks (smlmt), enables training of meta-learning methods for nlp at a much larger scale than was previously feasible while also ameliorating the risk of over-fitting to the training task distribution. this opens up new possibilities for applications of meta-learning in nlp, such as few-shot learning, continual learning, architecture search and more. this work focuses on few-shot learning and makes the following contributions: (1) we introduce a self-supervised approach to create tasks for meta-learning in nlp, subset masked language modeling tasks (smlmt), which enables application of meta-learning algorithms for goals like fewshot learning; (2) utilizing smlmt as the training task distribution, we train a state-of-the-art transformer architecture, bert @xcite , using a recent optimization-based meta-learning method which was developed for diverse classification tasks @xcite ; (3) we show that the self-supervised smlmt can also be combined with supervised task data to enable better feature learning, while still allowing for better generalization by avoiding meta-overfitting to the supervised tasks through the use of smlmt; (4) we rigorously evaluate the proposed approach on few-shot gener-alization to unseen tasks as well as new domains of tasks seen during training and show that the proposed approach demonstrates better generalization than self-supervised pre-training or self-supervised pre-training followed by multi-task training; (5) we also study the effect of number of parameters for few-shot learning and find that while bigger pretrained or meta-trained models generalize better than smaller models, meta-learning leads to substantial gains even for the smaller models. ## preliminaries in supervised meta-learning, we typically assume access to a task distribution p(t ). practically, this translates to a fixed set of training tasks {t 1 , . . . , t m }, which are referred to as metatraining tasks. for supervised classification, each task t i is an n i -way classification task. while many meta-learning algorithms assume a fixed nway classification, we follow the more practical approach of @xcite and allow for a diverse set of classification tasks with potentially different number of classes. the goal of a meta-learning algorithm is to utilize the meta-training tasks to learn a learning procedure that generalizes to held-out tasks t ∼ p(t ). model-agnostic meta-learning (maml) @xcite is an example of such a metalearning algorithm. maml learns an initial point θ for a classifier f θ : @xmath0, that can be optimized via gradient descent on the supervised loss l i defined for the task t i , using its support set d tr ∼ t i : ## meta-learning model we now discuss the meta-learning model for learning new nlp tasks. text encoder: the input to the model is natural language sentences. this is encoded using a transformer @xcite text encoder. we follow the bert @xcite model and use the same underlying neural architecture for the transformer as their base model. given an input sentence, the transformer model yields contextualized token representations for each token in the input after multiple layers of self-attention. following bert, we add a special cls token to the start of the input that is used as a sentence representation for classification tasks. given an input sentence x, let f π (x) be the cls representation of the final layer of the transformer with parameters π. meta-learning across diverse classes: our motivation is to meta-learn an initial point that can generalize to novel nlp tasks, thus we consider methods that apply to diverse number of classes. note that many meta-learning models only apply to a fixed number of classes @xcite and require training different models for different number of classes. we follow the approach of @xcite that learns to generate softmax classification parameters conditioned on a task support set to enable training meta-learning models that can adapt to tasks with diverse classes. this combines benefits of metric-based methods @xcite and optimization-based methods for meta-learning. the key idea is to train a deep set encoder g ψ (•), with parameters ψ, which takes as input the set of examples of a class n and generates a (d+1) dimensional embedding that serves as the linear weight and bias for class n in the softmax classification layer. let {x 1n , . . . , x kn } be the k examples for class n in the support set of a task t: using the above model to generate predictions, the parameters are meta-trained using the maml algorithm @xcite . concretely, set θ := {π, φ, w t , b t } for the task-specific inner loop gradient updates in (1) and set θ := {π, ψ, α} for the outer-loop updates in (2). note that we do multiple steps of gradient descent in the inner loop. bansal et al. (2019) performed extensive ablations over parameter-efficient versions of the model and found that adapting all parameters with learned perlayer learning rates performs best for new tasks. we follow this approach. full training algorithm can be found in the appendix. fast adaptation: @xcite proposed an approach which mitigates slow adaption often observed in maml by learning to warp the task loss surface to enable rapid descent to the loss minima. this is done by interleaving a neural network's layers with non-linear layers, called warp layers, which are not adapted for each task but are still optimized across tasks in the outer-loop updates in (2). since introducing additional layers will make computation more expensive, we use existing transformer layers as warp layers. we designate the feed-forward layers in between selfattention layers of bert, which project from dimension 768 to 3072 to 768, as warp-layers. note that these parameters also constitute a large fraction of total parameters (∼ 51%). thus in addition to the benefit from warping, not adapting these layers per task means significantly faster training and smaller number of per-task parameters during fine-tuning. the warp layers are still updated in the outer loop during meta-training. ## related work language model pre-training has recently emerged as a prominent approach to learning general purpose representations @xcite @xcite @xcite . refer to @xcite for a review of self-supervised learning. pre-training is usually a two step process and fine-tuning introduces random parameters making it inefficient when target tasks have few examples @xcite . multi-task learning of pre-trained models has shown improved results on many tasks @xcite . more recently, and parallel to this work, @xcite show that extremely large language models can act as few-shot learners. they propose a query-based approach where few-shot task data is used as context for the language model. in contrast, we employ a fine-tuning based meta-learning approach that enjoys nice properties like consistency which are important for good out-of-distribution generalization @xcite . moreover, we show that self-supervised meta-learning can also improve few-shot performance for smaller models. meta-learning methods can be categorized as: optimization-based @xcite @xcite , model-based @xcite @xcite , and metric-based @xcite . refer to @xcite for an exhaustive review. unsupervised meta-learning has been explored in vision. @xcite cluster images using pre-trained embeddings to create tasks for meta-learning. @xcite meta-learn a biologically-motivated update rule from unsupervised data in a semi-supervised framework. compared to these, we directly utilize text data to automatically create unsupervised tasks without relying on pre-trained embeddings or access to target tasks. in nlp, meta-learning approaches have followed the recipe of using supervised task data and learning models for specific tasks. such approaches @xcite @xcite @xcite train to generalize to new labels of a specific task like relation classification and don't generalize to novel tasks. @xcite proposed an approach that applies to diverse tasks to enable practical meta-learning models and evaluate on generalization to new tasks. however, they rely on supervised task data from multiple tasks and suffer from meta-overfitting as we show in our empirical results. @xcite studied related approaches for the task of wordsense disambiguation. to the best of our knowledge, the method proposed here is the first selfsupervised approach to meta-learning in nlp. ## experiments we evaluate the models on few-shot generalization to new tasks and new domains of train tasks. eval-uation consist of a diverse set of nlp classification tasks from multiple domains: entity typing, sentiment classification, natural language inference and other text classification tasks. our results foot_0 show that self-supervised meta-learning using smlmt improves performance over self-supervised pretraining. moreover, combining smlmt with supervised tasks achieves the best generalization, improving over multi-task learning by up to 21%. ## conclusion we introduced an approach to leverage unlabeled data to crate meta-learning tasks for nlp. this enables better representation learning, learning key hyper-parameters like learning rates, demonstrates data-efficient fine-tuning, and ameliorates metaoverfitting when combined with supervised tasks. through extensive experiments, we evaluated the proposed approach on few-shot generalization to novel tasks and domains and found that leveraging unlabelled data has significant benefits to enabling data-efficient generalization. this opens up the possibility of exploring large-scale meta-learning in nlp for various meta problems, including neural architecture search, continual learning, hyperparameter learning, and more.