Learning to Few-Shot Learn Across Diverse Natural Language Classification Tasks (2020)

Abstract: Pre-trained transformer models have shown enormous success in improving performance on several downstream tasks. However, fine-tuning on a new task still requires large amounts of task-specific labeled data to achieve good performance. We consider this problem of learning to generalize to new tasks, with a few examples, as a meta-learning problem. While meta-learning has shown tremendous progress in recent years, its application is still limited to simulated problems or problems with limited diversity across tasks. We develop a novel method, LEOPARD, which enables optimization-based meta-learning across tasks with a different number of classes, and evaluate different methods on generalization to diverse NLP classification tasks. LEOPARD is trained with the state-of-the-art transformer architecture and shows better generalization to tasks not seen at all during training, with as few as 4 examples per label. Across 17 NLP tasks, including diverse domains of entity typing, natural language inference, sentiment analysis, and several other text classification tasks, we show that LEOPARD learns better initial parameters for few-shot learning than self-supervised pre-training or multi-task training, outperforming many strong baselines, for example, yielding 14.6% average relative gain in accuracy on unseen tasks with only 4 examples per label.

## introduction learning to learn @xcite @xcite from limited supervision is an important problem with widespread application in areas where obtaining labeled data for training large models can be difficult or expensive. we consider this problem of learning in k-shots for natural language processing (nlp) tasks, that is, given k labeled examples of a new nlp task learn to efficiently solve the new task. recently, self-supervised pre-training of transformer models using language modeling objectives @xcite @xcite has achieved tremendous success in learning general-purpose parameters which are useful for a variety of downstream nlp tasks. while pre-training is beneficial, it is not optimized for fine-tuning with limited supervision and such models still require large amounts of task-specific data for fine-tuning, in order to achieve good performance @xcite . on the other hand, meta-learning methods have been proposed as effective solutions for few-shot learning. existing applications of such meta-learning methods have shown improved performance in few-shot learning for vision tasks such as learning to classify new image classes within a similar dataset. however, these applications are often limited to simulated datasets where each classification label is considered a task. moreover, their application in nlp has followed a similar trend @xcite @xcite @xcite . since the input space of natural language is shared across all nlp tasks, it is possible that a meta-learning approach generalizes to unseen tasks. we thus move beyond simulated tasks to investigate meta-learning performance on generalization outside the training tasks, and focus on a diverse task-set with different number of labels across tasks. model agnostic meta-learning (maml) @xcite is an optimization-based approach to metalearning which is agnostic to the model architecture and task specification. hence, it is an ideal candidate for learning to learn from diverse tasks. however, it requires sharing model parameters, including softmax classification layers across tasks and learns a single initialization point across tasks. this poses a barrier for learning across diverse tasks, where different tasks can have potentially disjoint label spaces. contrary to this, multi-task learning @xcite naturally handles disjoint label sets, while still benefiting from sharing statistical strength across tasks. however, to solve a new task, multi-task learning would require training a new classification layer for the task. on the other hand, metric-based approaches, such as prototypical networks @xcite , being non-parametric in nature can handle varied number of classes. however, as the number of labeled examples increase, these methods do not adapt to leverage larger data and their performance can lag behind optimization-based methods. we address these concerns and make the following contributions: (1) we introduce a maml-based meta-learning method, leopard foot_0 , which is coupled with a parameter generator that learns to generate task-dependent initial softmax classification parameters for any given task and enables meta-learning across tasks with disjoint label spaces; (2) we train leopard with a transformer model, bert @xcite , as the underlying neural architecture, and show that it is possible to learn better initialization parameters for few-shot learning than that obtained from just self-supervised pre-training or pre-training followed by multi-task learning; (3) we evaluate on generalization, with a few-examples, to nlp tasks not seen during training or to new domains of seen tasks, including entity typing, natural language inference, sentiment classification, and various other text classification tasks; (4) we study how metalearning, multi-task learning and fine-tuning perform for few-shot learning of completely new tasks, analyze merits/demerits of parameter efficient meta-training, and study how various train tasks affect performance on target tasks. to the best of our knowledge, this is the first application of meta-learning in nlp which evaluates on test tasks which are significantly different than training tasks and goes beyond simulated classification tasks or domain-adaptation tasks (where train and test tasks are similar but from different domains). ## background in meta-learning, we consider a meta goal of learning across multiple tasks and assume a distribution over tasks t i ∼ p (t ). we follow the episodic learning framework of @xcite which minimizes train-test mismatch for few-shot learning. we are given a set of m training tasks {t 1 , . . . , t m }, where each task instance potentially has a large amount of training data. in order to simulate k-shot learning during training, in each episode (i.e. a training step) a task t i is sampled with a training set d tr i ∼ t i , consisting of only k examples (per label) of the task and a validation set d val i ∼ t i , containing several other examples of the same task. the model f is trained on d tr i using the task loss l i , and then evaluated on d val i . the loss on d val i is then used to adjust the model parameters. here the validation error of the tasks serves as the training error for the meta-learning process. at the end of training, the model is evaluated on a new task t m +1 ∼ p (t ), where again the train set of t m +1 contains only k examples per label, and the model can use its learning procedure to adapt to the task t m +1 using the train set. we next discuss model-agnostic meta-learning which is pertinent to our work. model-agnostic meta-learning (maml) @xcite is an approach to optimization-based meta-learning where the goal is to find a good initial point for model parameters θ, which through few steps of gradient descent, can be adapted to yield good performance on a new task. learning in maml consists of an inner loop, which applies gradient-based learning on the task-specific objective, and an outer-loop which refines the initial point across tasks in order to enable fast learning. given a task t i with training datasets d tr i sampled during an episode, maml's inner loop adapts the parameters θ as: typically, more than one step of gradient update are applied sequentially. the learning-rate α can also be meta-learned in the outer-loop @xcite . the parameters θ are then trained by back-propagating through the inner-loop adaptation, with the meta-objective of minimizing the error across respective task validation sets note that even though maml is trained to generate a good initialization point for few-shot adaptation, since the inner-loop employs gradient-based learning, its performance can approach supervised learning in the limit of large data. ## experiments our experiments evaluate how different methods generalize to new nlp tasks with limited supervision. we focus on sentence-level classification tasks, including natural language inference (nli) tasks which require classifying pairs of sentences as well as tasks like entity typing which require classifying a phrase in a sentence. we consider 17 target tasks foot_1 . main results are in sec. 4.3. ## related work meta-learning approaches can be broadly classified as: optimization-based @xcite @xcite , model-based @xcite @xcite , and metric-learning based @xcite