Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP (2021)

Abstract: Meta-learning considers the problem of learning an efficient learning process that can leverage its past experience to accurately solve new tasks. However, the efficacy of meta-learning crucially depends on the distribution of tasks available for training, and this is often assumed to be known a priori or constructed from limited supervised datasets. In this work, we aim to provide task distributions for meta-learning by considering self-supervised tasks automatically proposed from unlabeled text, to enable large-scale meta-learning in NLP. We design multiple distributions of self-supervised tasks by considering important aspects of task diversity, difficulty, type, domain, and curriculum, and investigate how they affect meta-learning performance. Our analysis shows that all these factors meaningfully alter the task distribution, some inducing significant improvements in downstream few-shot accuracy of the meta-learned models. Empirically, results on 20 downstream tasks show significant improvements in few-shot learning – adding up to +4.2% absolute accuracy (on average) to the previous unsupervised meta-learning method, and perform comparably to supervised methods on the FewRel 2.0 benchmark.

## introduction humans show a remarkable capability to accurately solve a wide range of problems efficiently -utilizing a limited amount of computation and experience. deep learning models, by stark contrast, can be trained to be highly accurate on a narrow task while being highly inefficient in terms of the amount of compute and data required to reach that accuracy. within natural language processing (nlp), recent breakthroughs in unsupervised pretraining have enabled reusable models that can be applied to many nlp tasks, however, learning of new tasks is still inefficient @xcite @xcite . @xcite @xcite treats the learning process itself as a learning problem from data, with the goal of learning systems that can generalize to new tasks efficiently. this has the potential to produce fewshot learners that can accurately solve a wide range of new tasks. however, meta-learning requires a distribution over tasks with relevant labeled data that can be difficult to obtain, severely limiting the practical utility of meta-learning methods. in the supervised setting, in particular, metalearning task distribution is often defined by subsampling from the classes in a classification problem over a fixed dataset @xcite . this not only limits the applicability of meta-learning to the underlying classification problem, but also requires a diverse set of supervised datasets with a large number of classes to enable learning. selfsupervised meta-learning, on the other hand, seeks to propose tasks from unlabelled data @xcite , and has great potential to enable numerous important applications @xcite such as neural architecture search, continual learning, hyper-parameter optimization, learning in low-resource settings, etc. existing work in meta-learning for nlp, however, defaults to task distributions that tend to be overly simplistic, e.g. using existing supervised datasets @xcite @xcite or unsupervised cloze-style tasks with uniform selection of words from the vocabulary @xcite . given the lack of exploration on this critical component, we propose to devise and evaluate various task distributions in the context of unsupervised meta-learning for nlp. specifically, we explore a diverse set of approaches to create task distributions that are inductive to better meta-training efficacy. we provide empirical evidence that existing definitions of task distributions are prone to producing tasks that might not be challenging enough for the underlying model to learn useful representations, which in turn translates into poor downstream task performance. we therefore propose several new approaches that instead consider important features of the task distribution including task diversity, difficulty, resemblance to the downstream tasks, and the curriculum or the order in which tasks are presented during training. when evaluated on a suite of 20 nlp classification tasks, our best unsupervised metalearning method leads to an absolute increase of up to +4.2% in average few-shot accuracy over unsupervised baseline results; and it even outperforms supervised meta-learning methods on fewrel 2.0 benchmark @xcite on 5-shot evaluation. the paper is organized as follows. we start by providing some relevant background (2) on metalearning and the unsupervised task generation approach in smlmt. next, we introduce (3) new approaches to improve the task distribution. we then analyze (4.2) the different unsupervised distributions and how they relate to each other. finally, we evaluate (4.3, 4.4) the different unsupervised methods on a wide range of nlp tasks including sentiment classification, entity typing, text classification, sentence-pair classification and relation classification. ## exploring unsupervised task distribution for meta-learning sampling tasks in smlmt depends on sampling of words, which serve as labels, and sampling of sentences containing that word. the original formulation used uniform sampling for both steps. this can lead to several limitations on the quality of the resulting task distribution including task diversity and difficulty. the single-sentence classification tasks also lack cross-sentence reasoning capacities, leading to a severe train-test mismatch for downstream tasks involving sentence pairs. to remedy these problems, we consider alternative distributions that are inductive to more diverse and challenging tasks for meta-training. we also describe an automatic curriculum over tasks that seeks to continuously find challenging tasks for the model during training. ## experiments we evaluate various self-supervised task distributions for their utility in meta-learning for few-shot classification. we first describe the experimental setting, then we perform evaluations to understand how the different self-supervised tasks relate to each other, and finally show performance on a large set of 20 real classification datasets. these datasets cover a wide range of tasks: sentiment classification, entity typing, text classification, sentence pair classification and relation classification. our proposed approach shows significant improvements over previous few-shot classification results @xcite . ## related work meta-learning applications in nlp have yielded improvements on specific tasks @xcite @xcite @xcite . unsupervised meta-learning has been explored in computer vision @xcite and reinforcement learning @xcite . @xcite cluster images using pre-trained embeddings to create tasks. @xcite meta-learn an unsupervised update rule in a semisupervised framework. @xcite developed the smlmt approach to unsupervised meta-learning in nlp. contemporary work @xcite explored the use of clustering, though focused only on natural language inference tasks. curriculum learning @xcite in the context of meta-learning has been unexplored in nlp, prior to this work. @xcite found unsupervised curriculum to be beneficial for metareinforcement learning. we refer to @xcite for a comprehensive review of metalearning. self-supervised learning has emerged as an efficient approach to representation learning in nlp @xcite @xcite @xcite . multi-task learning of pre-trained models has shown improved results on many tasks @xcite , including few-shot setting. @xcite leveraged entailment tasks for few-shot learning. @xcite developed self-training methods for semi-supervised few-shot learning. recently, extremely large language models have been shown to have few-shot capacities @xcite , while schick and schütze (2020) demonstrated few-shot capacities for small models in the semi-supervised setting. meanwhile, @xcite showed meta-learning to be effective at improving few-shot performance in multi-task and unsupervised settings, as well as improving performance for small models. ## conclusion we explored several approaches to self-supervised task distribution for meta-learning. our results demonstrate improvements in few-shot performance over a wide-range of classification tasks. this demonstrates the utility of meta-learning from unlabeled data, opening up the possibility of largescale meta-learning for pertinent applications in nlp such as continual learning, architecture search, learning for low-resource languages, and more.