DReCa: A General Task Augmentation Strategy for Few-Shot Natural Language Inference
Abstract
Meta-learning promises “few-shot" learners that can adapt to new distributions by repurposing knowledge acquired from previous training.
However, meta-learning has thus far failed to achieve this in NLP due to the lack of a well-deﬁned task distribution, leading to alternatives that treat datasets as tasks.
Such an ad hoc task distribution has two negative consequences.
The ﬁrst one is due to a lack of quantity—since there’s only a handful of datasets, meta-learners tend to overﬁt their adaptation mechanism.
The second one is due to a lack of quality—since NLP datasets are highly heterogenous, many learning episodes have poor transfer between their support and query sets, which dis-incentivizes the metalearner from adapting.
To alleviate these issues, we propose DReCa ( D ecomposing datasets into Re asoning Ca tegories), a simple method for discovering and using latent reasoning categories in a dataset, to form additional high quality tasks.
DReCa works by splitting examples into label groups, embedding them with a ﬁne-tuned BERT model and then clustering each group into reasoning categories.
Across 4 NLI fewshot problems, we demonstrate that using DReCa improves the performance of meta-learners by 1.5–4 accuracy points.
Introduction
Over the last few years, we have seen tremendous progress on fundamental natural language understanding problems.
At the same time, there is increasing evidence that these models learn superﬁcial correlations that fail to generalize beyond the training distribution (Jia and Liang, 2017; Gururangan et al., 2018; McCoy et al., 2019).
How can we move from doing well on datasets toward more human-like understanding of tasks?
A key desideratum for human-like understanding is few-shot adaptation.
From a practical perspec- tive, adaptation is central to many NLP applications since new words and concepts appear every month, leading to distribution shifts.
People can effortlessly deal with these distribution shifts by learning these new concepts quickly and we would like our models to have similar capabilities.
Recently, pretrained transformers have led to impressive results on many NLP problems, but they still require 1000s of samples where humans might require only a few.
Could these pre-trained transformers also be made to achieve few-shot adaptation?
One promising direction is meta-learning.
promises “few-shot" classiﬁers that can adapt to new tasks by repurposing skills acquired from training tasks.
An important prerequisite for successful application of meta-learning is a task-distribution from which a large number of tasks could be sampled to train the meta-learner.
While meta-learning is very appealing, applications in NLP have thus far proven challenging due to the absence of a welldeﬁned set of tasks that correspond to re-usable skills.
This has led to less effective ad hoc alternatives, like treating entire datasets as tasks.
Treating entire datasets as tasks has two major is- sues.
First, because there’s only a small number of supervised datasets available for any NLP problem, we run into learner overﬁtting (Rajendran et al., 2020): due to the small number of training tasks, the meta-learner overﬁts its adaptation mechanism, and doesn’t generalize to new tasks.
Second, the heterogeneity of NLP datasets can lead to learning episodes that encourage memorization overﬁtting (Yin et al., 2020; Rajendran et al., 2020), a phenomenon where a meta-learner ignores the support set, and doesn’t learn to adapt.
To improve the quality as well as quantity of tasks, we propose D ecomposing datasets into
Re asoning Ca tegories or DReCa.
DReCa is a meta data augmentation strategy that takes as input the original set of tasks (entire datasets), and then decomposes them to approximately recover the latent reasoning categories underlying these datasets.
This allows us to approximately recover reasoning categories, e.g., various syntactic constructs within a dataset, linguistic categories such as quantiﬁers and booleans.
These reasoning categories are then used to construct additional few-shot classiﬁcation tasks, augmenting the original task distribution.
We illustrate these steps in illustrated in Fig.
DReCa
ﬁrst embeds the examples using a BERT model ﬁne-tuned over all the datasets.
We then run kmeans clustering over these representations to produce a reﬁnement of the original tasks.
Experiments demonstrate the effectiveness of our simple approach.
First, we adapt the classic sine-wave regression problem from Finn et al.
(2017) to reﬂect the challenges of our setting, and observe that standard meta-learning procedures fail to adapt.
However, a model that meta-learns over the underlying reasoning types shows a substantial improvement.
Next we consider the problem of natural language inference (NLI).
We show that meta-learners augmented with DReCa improve over baselines by 1.5–4 accuracy points across four separate NLI few-shot problems without requiring domain-speciﬁc engineering, or additional unlabeled data.
Related Work
Few Shot Classiﬁcation in NLP.
The goal of learning from few examples has been studied for various NLP applications.
Common settings include few shot adapting to new relations (Han et al., 2018),
words (Holla et al., 2020) domains (Bao et al., 2020; Yu et al., 2018;
Geng et al., 2019) and language pairs (Gu et al., 2018).
In these works, since task distributions are well deﬁned, they do not have the same overﬁtting challenges.
On the other hand, many works deal with fewshot adaptation in settings with no clear task distribution such as Dou et al.
; Bansal et al.
but do not address meta-overﬁtting.
Overﬁtting and
Task
Augmentation.
The memorization problem in meta-learning is studied in Yin et al.
(2020) who propose a meta-regularizer to encourage the meta-learner to adapt, but this not directly applicable to NLP.
Task Augmentation for mitigating overﬁtting in meta-learners is ﬁrst studied in Rajendran et al.
(2020) in the context of few-shot label adaptation.
Hsu et al.
(2019) propose CACTUs, a clustering based approach for unsupervised meta-learning in the context of few-shot label adaptation for images, but do not study meta-overﬁtting.
Most closely related to our work is the recent work by Bansal et al.
They propose SMLMT, a task augmentation strategy that require a large text corpus to construct augmented tasks.
On the other hand, DReCa creates task augmentations based solely on the provided training data.
In Section-6, we compare our model against SMLMT, and demonstrate comparable performance.
Setting 3. NLI
We consider the problem of Natural Language Inference or NLI (MacCartney and Manning, 2008;
Bowman et al., 2015), also known as Recognising Textual Entailment (RTE) (Dagan et al., 2005).
Given a sentence pair x = ( p, h ) where p is referred to as the premise sentence, and h is the hypothesis sentence, the goal is to output a binary label ˆ y ∈ { 0 , 1 } indicating whether the hypothesis h is entailed by the premise p or not.
For instance, the sentence pair (The dog barked, The animal barked) is classiﬁed as entailed, whereas the sentence pair (The dog barked, The labrador barked) would be classiﬁed as not entailed.
As shown in Table.
NLI datasets typically encompass a broad range of linguistic phenomenon.
Apart from the reasoning types shown in Table.
1, examples may also vary in terms of their genre, syntax, annotator writing style etc.
leading to extensive linguistic variability.
Taken together, these factors of variation make NLI datasets highly heterogenous.
[Table]
Meta learning
The goal of meta-learning is to output a black box meta-learner f : ( S i , x iq )
(cid:55)→ ˆ y that takes as input a support set S i of labeled examples and a query point x iq and returns a prediction ˆ y .
In the usual meta-learning setting, these support and query sets are deﬁned as samples from a task T i , which is a collection of labeled examples { ( x i , y i ) } .
In Nway k -shot adaptation, each T i is an N -way classiﬁcation problem, and f is given k examples per label to adapt.
A simple baseline for meta-learning is to train a supervised model on labeled data from training tasks, and then ﬁne-tune it at test time on the support set.
This can be powerful, but ineffective for very small support sets.
A better alternative is episodic meta-learning, which explicitly trains models to adapt using training tasks
Episodic Training.
In the standard setup for training episodic meta-learners, we are given a collection of training tasks.
We assume that both train and test tasks are i.i.d.
draws from a task distribution ρ ( T ) .
For each training task T tr i ∼ ρ ( T ) , we create learning episodes which are used to train the meta-learner.
Each learning episode consists of a support set and a query set Q i = { ( x iq , y iq ) } .
To make predictions on a query x iq , the metalearner f uses S i to adapt.
The goal of episodic meta-learning is to ensure that the meta-learning loss L ( f ( S i , x iq ) , y iq ) is small on training tasks
T tr i .
Since train tasks are i.i.d.
with test tasks, this results in meta-learners that achieve low loss at test time.
Several algorithms have been proposed for metalearning that follow this general setup, such as
Matching Networks (Vinyals et al., 2016), MANN (Santoro et al., 2016), Prototypical Networks (Snell et al., 2017) and MAML (Finn et al., 2017).
In this work, we use MAML as our meta-learner.
MAML.
In MAML, the meta-learner f takes the form of gradient descent on a model h θ : x (cid:55)→ y using the support set,
[Equation]
where θ (cid:48)
i denotes task speciﬁc parameters obtained after gradient descent.
The goal of MAML is to produce an initialization θ , such that after performing gradient descent on h θ using S i , the updated model h θ (cid:48)
i can make accurate predictions on
Q i .
MAML consists of an inner loop and an outer loop .
In the inner loop, the support set S i is used to update model parameters θ , to obtain task-speciﬁc parameters θ (cid:48)
i ,
[Equation]
These task speciﬁc parameters are then used to make predictions on Q i .
The outer loop takes gradient steps over θ such that task-speciﬁc parameters θ (cid:48)
i perform well on Q i .
Since θ (cid:48)
i is itself a differentiable function of θ , we can perform this outer optimization using gradient descent, θ ← Opt
[Equation]
Here, Opt is an optimization algorithm typically chosen to be Adam.
The outer loop gradient is typically computed in a mini-batch fashion by sampling a batch of episodes from distinct training tasks.
The gradient ∇ θ L ( h θ (cid:48) i ( x iq ) , y iq ) involves back-propagation through the adaptation step which requires computing higher order gradients.
This can be computationally expensive so a ﬁrst order approximation (FoMAML),
[Equation]
is often used instead (Finn et al., 2017).
Overﬁtting in Meta Learning
As mentioned earlier training tasks in NLP are often entire datasets, due to the lack of a well-formed task distribution.
This results in a small number of heterogeneous training tasks, which can lead to learner and memorization overﬁtting.
Learner overﬁtting occurs when the meta-learner is exposed to a very small number of tasks at meta-training time causing it to not generalize to test tasks.
Memorization overﬁtting when the meta-learner ignores its support set and doesn’t learn to adapt at all.
We illustrate memorization overﬁtting challenges through a simple few-shot generalization problem based on 2D sine wave regression.
Dataset.
We extend a standard meta-learning toy regression problem from Finn et al.
(2017) to our setting.
The key hypothesis here is that metalearning on a small number of heterogenous tasks leads to poor performance.
To reﬂect these challenges, we construct a meta-learning problem with a dataset-based task distribution where each dataset consists of multiple reasoning categories (Fig. 2).
Much like the original sine wave problem, the key challenge in adapting to a new reasoning category involves estimating the phase angle of the sine wave mapping from a small number of support set examples.
Our construction consists of multiple datasets.
Each dataset is deﬁned as a unit square sampled from a 10 × 10 grid over x 1 =
[ − 5 , 5] and x 2 =
[ − 5 , 5] .
Within each dataset, we construct multiple reasoning categories by deﬁning each reasoning category to be a sine wave with a distinct phase.
This is illustrated in Fig.
2 where each 1 × 1 represents a dataset, and sine waves along distinct rays correspond to reasoning categories.
The target label y for the regression task is deﬁned for each category by a randomly sampled phase φ ∈ [0 . 1 , 2 π ] and y = sin( (cid:107) x − (cid:98) x (cid:99)(cid:107) 2 − φ ) .
At meta-training time, we sample a subset of these 100 squares as our training datasets, and then evaluate few shot adaptation to reasoning categories from held out datasets at meta-test time.
[Figure]
Experiments.
We use similar hyperparameters as Finn et al.
(2017) elaborated in Appendix A.1.
We start by considering MAML-Base, a metalearner that is trained directly over a dataset-based task distribution.
Concretely, we deﬁne each training task as a dataset and randomly sample episodes to train the meta-learner.
Note that since episodes are drawn uniformly at random from an entire dataset, we expect support and query sets to often contain points from disjoint reasoning categories.
In such scenarios, adaptation is not possible since the model cannot estimate the phase angle for query examples based on support examples.
Thus, we expect pre and post adaptation losses to be similar.
This is indeed reﬂected in the learning curves in
Fig. 3(a).
We observe that the orange and blue lines, corresponding to pre and post adaptation losses respectively, almost overlap.
In other words, the model ignores the support set entirely.
This is what we mean by memorization overﬁtting .
Next we consider MAML-Oracle, a metalearner that is trained on tasks based on the underlying reasoning categories.
In this setting, sup- port and query sets are both drawn from the same sine wave, thus the model should be able to estimate phase angle for query examples from the support.
This suggests that we should expect post adaptation loss to be lower than the pre adaptation loss.
Empirically, from Fig. 3(b), we observe large gaps between pre and post adaptation losses which indicates that memorization overﬁtting has been mitigated.
This leads us to the main question:
Could we discover these reasoning categories?
Can we discover reasoning categories?
In an attempt to discover these latent reasoning categories, we train a feedforward neural net (paramaterized similarly as h θ ) on the union of all the datasets, and use the ﬁnal layer representation to cluster examples.
We then use these clusters instead of the true reasoning categories to augment the original task distribution.
We now show learning curves on held out test tasks in Fig. 4.
As expected MAML-Base fails to adapt to new reasoning categories, indicating that it was unable to acquire the required skill from its training tasks.
On the other hand, MAML-Oracle is able to adapt very well, which conﬁrms our hypothesis that a large number of high quality tasks helps.
Finally, we see that using MAML trained on the augmented task distribution is able to match the performance of the oracle.
[Figure]
Our Approach
Experiments on the 2D sine wave regression problem conﬁrm our hypothesis about the challenges of meta-learning with heterogenous task distributions.
Since NLI datasets require a wide range of skills, we expect similar challenges on few-shot NLI as well.
Motivated by the success of the clustering approach from Section 4, we now demonstrate that a similar procedure can extract reasoning categories for NLI.
The key hypothesis here is that high quality sentence pair representations, such as those obtained from a ﬁne-tuned BERT model, can bring out the micro-structure of NLI datasets.
We start by studying an analogue to our clustering approach for HANS (McCoy et al., 2019), a diagnostic NLI dataset.
HANS consists ofmanually deﬁned syntactic templates which can be grouped into 15 reasoning categories.
We ﬁnetune BERT (Devlin et al., 2019) for 5000 randomly chosen examples from HANS.
To obtain a vector representation for each example x = ( p, h ) , we concatenate the vector at the [CLS] token, along with a mean pooled representation of the premise and hypothesis.
We then use t-SNE (Maaten and
Hinton, 2008) to project these representations onto 2 dimensions.
Each point in Fig.
5 is colored with its corresponding reasoning category, and we can observe a clear clustering of examples according to their reasoning category.
Indeed, the fact that pretrained transformers can be used to create meaningful clusters has been shown in other recent works (c.f. Aharoni and Goldberg (2020); Joshi et al.
The ability of ﬁnetuned BERT representations to discover reasoning categories suggests our more general approach, which we describe below.
DReCa:
The goal of DReCa is to take a heterogeneous task (such as a dataset) and produce a decomposed set of tasks.
In doing so, we hope to obtain a large number of relatively homogeneous tasks that can be used to avoid meta overﬁtting.
Given a training task T tr i , we ﬁrst group examples by their labels, and then embed exam-
[Table]
[Table]
ples within each group with an embedding function EMBED ( . ) .
Concretely, for each N -way classiﬁcation task T tr i we form groups g il =
{ ( EMBED ( x pi ) , y pi ) | y pi = l } .
Then, we proceed to reﬁne each label group into K clusters via kmeans clustering to break down T tr
i into groups { C j ( g il ) }
Kj =1 for l = 1 , 2 . . .
N .
These cluster groups can be used to produce K N
DReCa tasks.
Each task is obtained by choosing one of K clusters for each of the N label groups, and taking their union.
At meta-training time, learning episodes are sampled uniformly at random from
DReCa tasks with a probability λ and from one of the original tasks with probability 1 − λ .
To produce learning episodes from DReCa tasks, we simply sample support and query sets from these augmented tasks.
Since our clustering procedure is based on ﬁnetuned BERT vectors, we expect the resulting clusters to roughly correspond to distinct reasoning categories.
Indeed, when the true reasoning categories are known we show in Section 6.3 that
DReCa yields clusters that recover these reasoning categories almost exactly.
NLI Experiments 6. Datasets
We evaluate DReCa on 4 NLI few-shot learning problems which we describe below.
HANS-fewshot is a few-shot classiﬁcation problem over HANS (McCoy et al., 2019), a synthetic dataset for NLI.
Each example in HANS comes from a hand-designed syntactic template which is associated with a ﬁxed label ( entailment or not_entailment ).
The entire dataset consists ofsuch templates which we use to deﬁne 15 reasoning categories.
We then hold out 5 of these for evaluation, and train on the remaining 10.
While this is a simple setting, it allows us to compare DReCa against an “oracle" with access to the underlying reasoning categories.
DNC-fewshot
uses a subset of DNC (Poliak et al., 2018), a collection of multiple datasets recast as NLI.
We manually write a collection of patterns representing a reasoning category, and match each example against these patterns.
For each pattern, all examples that match this pattern form a task.
This results in 25 distinct reasoning categories, out of which we hold out 8 tasks for evaluation.
CombinedNLI consists of a combination of
NLI datasets: MultiNLI (Williams et al., 2018),
DNC and Semantic Fragments (Richardson et al., 2020) for training and RTE for evaluation.
We convert both MultiNLI and Semantic Fragments to a 2-way classiﬁcation by collapsing contradiction and neutral labels into a not_entailment label.
GLUE-SciTail where we train on all the NLI datasets from the GLUE benchmark (Wang et al., 2019) and evaluate on SciTail (Khot et al., 2018).
This setting is comparable to Bansal et al.
with the difference that we only meta-train on the
NLI subset of GLUE, whereas Bansal et al.
(2019) meta-train on all GLUE tasks.
For GLUE-SciTail, we follow Bansal et al.
and report 2-way 4-shot accuracy on SciTail.
6. Baselines
We compare our approach against several alternatives.
Multitask is a non-episodic baseline that
[Table]
trains h θ on the union of all examples from each
T tr i , and then additionally ﬁne-tunes the trained model separately on the support set of each test task.
MAML-Base is a MAML model where every task corresponds to a dataset.
When the true reasoning categories are known, we also compare with an oracle model MAML-Oracle which is trained over a mixture of dataset-based tasks as well as oracle reasoning categories.
Finally, MAML-DReCa is our model which trains MAML over a mixture of the original dataset-based tasks as well as the augmented tasks from DReCa.
Evaluation.
To control for variations across different support sets, we sample 5–10 random support sets for each test task.
We ﬁne-tune each of our models on these support sets and report means and 95% conﬁdence intervals assuming the accuracies follow a Gaussian.
Training Details.
For computational efﬁciency, we use ﬁrst order MAML (FoMAML).
We use
BERT-base as the parameterization for h θ .
The inner loop optimization involves 10 gradient steps with Adam, with a support set of 2 examples (2way 1-shot) for all except GLUE-SciTail where the support set size is 8 (2-way 4-shot).
For
DReCa, we use the ﬁne-tuned BERT model to deﬁne EMBED (; ) , similar to Section 5.
The mixing weight λ is set to 0.5 for all our experiments.
Results.
We ﬁnd that DReCa improves model performance across all 4 datasets: MAML-DReCa improves over MAML-Base by +4.3 points on
HANS-fewshot, +2.2 points on DNC-fewshot, +2.
points on CombinedNLI and +1.6 points on GLUESciTail (Table 2).
Moreover, we observe that
MAML-DReCa is able to obtain comparable performance as MAML-Oracle (as conﬁdence intervals overlap) on both HANS-fewshot and DNCfewshot.
On GLUE-SciTail, we also compare against the SMLMT model from (Bansal et al., 2020).
We ﬁnd that MAML-DReCa improves over this model by 1.5 accuracy points.
However, we note that the conﬁdence intervals of these approaches overlap, and also that (Bansal et al., 2019) consider the entire GLUE data to train the metalearner whereas we only consider NLI datasets within GLUE.
6. Quantitative evaluation of clusters
To understand whether reasoning categories can be accurately recovered with our approach, we measure the purity of DReCa clusters when true reasoning categories are known.
This is evaluated by ﬁrst computing the number of examples belonging to the majority reasoning type for each cluster and then dividing by the total number of examples.
Since this requires knowing oracle reasoning categories, we compute cluster purity for HANSfewshot and DNC-fewshot.
Results are in Table 3.
We observe a very high cluster purity which provides some evidence that DReCa is able to recover true reasoning categories.
6. Model Ablations
We investigate the effects of number of clusters as well as the choice to use a ﬁne-tuned BERT for clustering via ablation experiments.
Our hypothesis is that ﬁne-tuning representations is essential to bring out the micro-structure speciﬁc to our datasets, resulting in better augmented tasks.
Moreover, a large number of augmented tasks would lead to improved performance due to less metaoverﬁtting.
Results are in Table 4.
We observe that
MAML-Oracle (No ﬁne-tuning) suffers a performance drop of 5.3 accuracy points suggesting that ﬁne-tuning BERT is essential.
Next, we see that decreasing the number of augmented tasks down to 25 ( K = 5 ) from 400 also incurs a drop of 4.
points compared to our full model.
Discussion
In this work, we take a closer look at using metalearning tools for few-shot classiﬁcation problems.
One of the main ingredients for successful application of meta-learning is a large number of high quality training tasks to sample learning episodes for the meta-learner.
We observe that such a task distribution is usually not available for important
NLP problems, leading to less desirable ad hoc alternatives that often treat entire datasets as tasks.
In response, we propose DReCa as a simple and general purpose task augmentation strategy.
From results on 4 NLI few-shot classiﬁcation benchmarks, we conclude the effectiveness of our approach.
Many works suggest that there are fundamental challenges in creating systems that achieve humanlike understanding of tasks like NLI.
In this work, we studied conditions under which systems can learn with extremely few samples, and we believe that such systems would complement and enhance further study into more sophisticated challenges such as model extrapolation.
A Appendix
A. 2D Sine Wave Regression: Training Details
We use a two layer neural network with 40 dimensional hidden representations and ReLU nonlinearity as the parameterization of f .
Following
Finn et al.
(2017), we take a single gradient step on the support set at meta-training time, and take 10 gradient steps at meta-test time.
The MAML weights are optimized with Adam and the inner loop adaptation is done with SGD with a learning rate of 1e-2.
For each outer loop update, we sample 5 tasks, and each episode consists of a support set of size 5 i.e. we consider 5 shot adaptation.