DR e C a: A General Task Augmentation Strategy for Few-Shot Natural Language Inference (2021)

Abstract: Meta-learning promises few-shot learners that can adapt to new distributions by repurposing knowledge acquired from previous training. However, we believe meta-learning has not yet succeeded in NLP due to the lack of a well-defined task distribution, leading to attempts that treat datasets as tasks. Such an ad hoc task distribution causes problems of quantity and quality. Since there’s only a handful of datasets for any NLP problem, meta-learners tend to overfit their adaptation mechanism and, since NLP datasets are highly heterogeneous, many learning episodes have poor transfer between their support and query sets, which discourages the meta-learner from adapting. To alleviate these issues, we propose DReCA (Decomposing datasets into Reasoning Categories), a simple method for discovering and using latent reasoning categories in a dataset, to form additional high quality tasks. DReCA works by splitting examples into label groups, embedding them with a finetuned BERT model and then clustering each group into reasoning categories. Across four few-shot NLI problems, we demonstrate that using DReCA improves the accuracy of meta-learners by 1.5-4%

## introduction a key desideratum for human-like understanding is few-shot adaptation. adaptation is central to many nlp applications since new concepts and words appear often, leading to distribution shifts. people can effortlessly deal with these distribution shifts by learning these new concepts quickly and we would like our models to have similar capabilities. while finetuning large pre-trained transformers is one way to facilitate this adaptation, this procedure requires thousands of samples where humans might require only a few. can these pre-trained transformers be made to achieve few-shot adaptation? one promising direction is meta-learning @xcite ; ben-figure 1 : overview of our approach. we embed all examples with bert, and then cluster within each label group separately (red and green correspond to entailment and not_entailment respectively). then, we group clusters from distinct label groups to form tasks. @xcite . meta-learning promises few-shot classifiers that can adapt to new tasks by repurposing skills acquired from training tasks. an important prerequisite for successful application of meta-learning is a task-distribution from which a large number of tasks can be sampled to train the meta-learner. while meta-learning is very appealing, applications in nlp have thus far proven challenging due to the absence of a well-defined set of tasks that correspond to re-usable skills. this has led to less effective ad hoc alternatives, like treating entire datasets as tasks. treating entire datasets as tasks has two major issues. the first issue is learner overfitting @xcite , where a meta-learner overfits its adaptation mechanism to the small number of training tasks, since there's only a small number of supervised datasets available for any nlp problem. second, the heterogeneity of nlp datasets can lead to learning episodes that encourage memorization overfitting @xcite , a phenomenon where a meta-learner ignores the support set, and doesn't learn to adapt. to improve the quality and quantity of tasks, we propose the simple approach of decomposing datasets into reasoning categories or dreca. dreca is a meta data augmentation strategy that takes as input a set of tasks (entire datasets), and then decomposes them to approximately recover some of the latent reasoning categories underlying these datasets, such as various syntactic constructs within a dataset, or semantic categories such as quantifiers and negation. these reasoning categories are then used to construct additional fewshot classification tasks, augmenting the original task distribution. we illustrate these steps in fig. 1 . dreca first embeds the examples using a bert model finetuned over all the datasets. we then run k-means clustering over these representations to produce a refinement of the original tasks. experiments demonstrate the effectiveness of our simple approach. as a proof of concept, we adapt the classic sine-wave regression problem from @xcite to mimic the challenges of the nlp setting, and observe that standard metalearning procedures fail to adapt. however, a model that meta-learns over the underlying reasoning types shows a substantial improvement. then, we consider the problem of natural language inference (nli). we show that meta-learners augmented with dreca improve over baselines by 1.5-4 accuracy points across four separate nli few-shot problems without requiring domain-specific engineering or additional unlabeled data. ## related work few-shot learning in nlp. the goal of learning from few examples has been studied for various nlp applications. common settings include few-shot adaptation to new relations @xcite , words @xcite , domains @xcite @xcite , and language pairs @xcite . since these applications come with well-defined task distributions, they do not have the same overfitting challenges. on the other hand, many works deal with few-shot adaptation in settings with no clear task distribution @xcite but do not address meta-overfitting, and thus are complementary to our work. overfitting and task augmentation. the memorization problem in meta-learning is studied in @xcite who propose a meta-regularizer to mitigate memorization overfitting, but don't study learner overfitting. task augmentation for mitigating overfitting in meta-learners is first studied in @xcite in the context of few-shot label adaptation. @xcite propose cactus, a clustering-based approach for unsupervised meta-learning in the context of few-shot label adaptation for images. while also based on clustering, cactus creates meta-learning tasks where the goal is to predict cluster membership of images, whereas our work is focused on using clusters to subdivide pre-existing tasks for mitigating meta-overfitting in nlp. most closely related to our work is the smlmt method from @xcite . smlmt creates new self-supervised tasks that improve meta-overfitting but this does not directly address the dataset-as-tasks problem we identify. in contrast, we focus on using clustering as a way to subdivide and fix tasks that already exist. this approach allows us to mitigate meta-overfitting without additional unlabeled data. in section 6, we compare our model against smlmt, and demonstrate comparable or better performance. ## an illustration of overfitting in meta-learning we illustrate meta overfitting challenges by modifying the classic sine-wave toy example for metalearning from @xcite . dataset. consider the sine-wave regression problem from @xcite where each task corresponds to learning a sine wave mapping with a fixed amplitude and phase offset. as shown in fig. 2(a) , each support and query set consists of points drawn from the same sine wave mapping. the key observation here is that since support and query examples are drawn from the same mapping, we might expect a meta-learner to use the support set for adaptation. in the nlp case, since tasks are heterogeneous, support and query examples may belong to different reasoning categories. we instantiate this by letting support and query points come from different sine waves (fig. 2(b) ). more formally, our construction consists of multiple datasets. each dataset is defined as a unit (a) 1d sine wave regression @xcite . each task is a sine-wave with a fixed amplitude and phase offset. experiments. we use similar hyperparameters as @xcite elaborated in appendix a.1. we start by considering maml-base, a metalearner that is trained directly over a dataset-based task distribution. concretely, we define each training task as a dataset and randomly sample episodes to train the meta-learner. note that since episodes are drawn uniformly at random from an entire dataset, we expect support and query sets to often contain points from disjoint reasoning categories (fig. 2(b) ), making adaptation infeasible. thus, we expect pre and post adaptation losses to be similar, which is indeed reflected in the learning curves in fig. 3(a) . we observe that the orange and blue lines, corresponding to pre and post adaptation losses respectively, almost overlap. in other words, the meta-learner ignores the support set entirely. this is what we mean by memorization overfitting. next we consider maml-oracle, a metalearner that is trained on tasks based on the underlying reasoning categories-distinct sine waves. consequently, support and query sets are both drawn from the same sine wave, similar to these experiments confirm our hypothesis about the challenges of meta-learning with heterogeneous task distributions. since nli datasets require a wide range of skills, we might expect similar challenges on few-shot nli as well. ## dreca in this section, we introduce our approach for extracting reasoning categories for nli. the key observation here is that high quality sentence pair representations, such as those obtained from a finetuned bert model, can bring out the microstructure of nli datasets. indeed, the fact that pretrained transformers can be used to create meaningful clusters has been shown in other recent works (c.f. aharoni and goldberg (2020); joshi et al. ( 2020 )). at a high level, the goal of dreca is to take a heterogeneous task (such as a dataset) and produce a decomposed set of tasks. in doing so, we hope to obtain a large number of relatively homogeneous tasks that can prevent meta overfitting. given a training task t tr i , we first group examples by their labels, and then embed examples within each group with an embedding function embed(.). concretely, for each n -way classification task t tr i we form groups g i @xmath0, y p i ) | y p @xmath1. then, we proceed to refine each label group into k clusters via kmeans clustering to break down t tr i into groups {c j (g i l )} k @xmath2, 2, . . . , n . these cluster groups can be used to produce k n potential dreca tasks. 2 each task is obtained by choosing one of k clusters for each of the n label groups, and taking their union. at meta-training time, learning episodes are sampled uniformly at random from dreca tasks with a probability λ and from one of the original tasks with probability 1 -λ. since our clustering procedure is based on finetuned bert vectors, we expect the resulting clusters to roughly correspond to distinct reasoning categories. indeed, when the true reasoning categories are known, we show in section 7.2 that dreca yields clusters that recover these reasoning categories almost exactly. ## discussion comparing with cactus. our work is most similar to cactus from @xcite . apart from differences in the modality considered (text vs images), we differ in the following ways. conceptually, @xcite consider a fully unsupervised meta-learning setting where no labels are provided and use cluster ids to induce labels, while our goal is to produce additional tasks in a supervised meta-learning setting. second, cactus tasks are constructed by directly applying k-means on the entire training dataset while we apply k-means separately on each label group and construct tasks by choosing a cluster from each label group, leading to tasks with uniform label distribution. finally, while cactus uses constructed tasks directly, our work using them to augment the original task distribution. number of examples in support set. all evaluation in this work considers small support sets where number of examples per label range from 1-4. this setting is somewhat restrictive since in practice, one might be able to get a few hundred examples for the target domain. these moderately sized support sets could themselves be heterogeneous where adapting a single learner might be hard. in such cases, we can use a similar clustering approach to separate out the support set into homogeneous tasks and adapt a separate learner for each task. these learners could then be plugged into a mixture of experts framework for making predictions. using k-means to produce task refinements. while we are able to get sufficiently homogeneous clusters with k-means, we note one shortcoming with this approach. any input has multiple attributes / factors of variations and it may be possible to create a clustering for each factor. the current k-means based approach doesn't model this since we only produce a single clustering of the data. for instance, x 1 = the man was walking in the @xmath3. at the same time, it could also be clustered with x 3 = the man was walking in the @xmath4. a promising direction for future work is to explore these multi-clusterings based on the various factors of variation present in the training data. non-meta learning based few-shot adaptation. in this work, we use tools from meta-learning to directly optimize for few-shot behavior. while not directly comparable to us, there have been many recent approaches to few-shot adaptation for nlp that do not use meta-learning. @xcite show impressive few-shot adaptation in large language models through "in-context learning" which is presumably acquired only through its language modeling objective,. schick and schütze (2020) train multiple models on lexical variations of a small support set and use these to label additional unlabeled examples from the target domain. these "self-labeled" examples are used to train a second model which can then make predictions on query examples. finally, @xcite explore incontext learning of smaller language models for few-shot adaptation. in particular, they introduce a pipeline to identify useful prompts for the target domain, along with informative labeled examples to prepend as context for the lm. ## conclusion many papers point out fundamental challenges in creating systems that achieve human-like understanding of tasks like nli. here, we studied conditions under which systems can learn from extremely few samples. we believe that such systems would complement and enhance further study into more sophisticated challenges such as model extrapolation. one of the main ingredients for successful application of meta-learning is a large number of high quality training tasks to sample learning episodes for the meta-learner. we observe that such a task distribution is usually not available for important nlp problems, leading to less desirable ad hoc attempts that treat entire datasets as tasks. in response, we propose dreca as a simple and general purpose task-augmentation strategy. our approach creates a refinement of the original set of tasks (entire datasets) that roughly correspond to linguistic phenomena present in the dataset. we show that training on a task distribution augmented with dreca leads to consistent improvements on 4 nli few-shot classification problems, matching other approaches that require additional unlabeled data and well as oracles that have access to the true task distribution.