Adversarial Task Up-sampling for Meta-learning
Anonymous Author(s)
Abstract
The success of meta-learning on existing benchmarks is predicated on the assumption that the distribution of meta-training tasks covers meta-testing tasks.
Frequentviolation of the assumption in applications with either insufficient tasks or a verynarrow meta-training task distribution leads to memorization or learner overfitting.
Recent solutions have pursued augmentation of meta-training tasks, while it is stillan open question to generate both correct and sufficiently imaginary tasks.
In thispaper, we seek an approach that up-samples meta-training tasks from the task representation via a task up-sampling network.
Besides, the resulting approach named
Adversarial Task Up-sampling (ATU) suffices to generate tasks that can maximallycontribute to the latest meta-learner by maximizing an adversarial loss.
On fewshot sine regression and image classification datasets, we empirically validate themarked improvement of ATU over state-of-the-art task augmentation strategies inthe meta-testing performance and also the quality of up-sampled tasks.
Introduction
The past few years have seen the burgeoning development of meta-learning, a.k.a. learning to learn,which draws upon the meta-knowledge learned from previous tasks (i e., meta-training tasks ) toexpedite the learning of novel tasks (i e., meta-testing tasks ) with a few examples.
A sufficient numberand diversity of meta-training tasks are pivotal for the generalization capability of the meta-knowledge,so that (1) they cover the true task distribution (i e., environment [3]) from which meta-testing tasks aresampled, discouraging learner overfitting [18] and (2) the meta-knowledge empowers fast adaptationvia the support set for each task, avoiding memorization overfitting [33].
Notwithstanding upto millions of meta-training tasks in benchmark datasets [19, 24], real-world applications such as drugdiscovery [29] and medical image diagnosis [10] usually have access to only thousands or hundredsof tasks, which puts the meta-knowledge at high risk of learner and memorization overfitting.
While early attempts towards improving the generalization capability of the meta-knowledge revolvearound regularization methods that limit the capacity of the meta-knowledge [7, 33], recent works onaugmentation of meta-training tasks have shown a marked improvement [15, 29, 32].
The objectiveof task augmentation is to draw the empirical task distribution which is formed by assembling Diracdelta functions located in each meta-training task closer to the true task distribution.
Consequently,achieving this objective requires a qualified task augmentation approach to simultaneously possessthe following three properties: (1) task-aware : the augmented tasks comply with the true taskdistribution, being not erroneous to lead the meta-knowledge astray (tasks A and B in Figure 1a); (2)task-imaginary : the augmented tasks cover a substantial portion of the true distribution, embracingtask diversity which task-awareness is nonetheless inadequate to guarantee (tasks C, D, and E in
Figure 1b); (3) model-adaptive : the augmented tasks are timely in improving the current metaknowledge, to which the meta-knowledge before augmentation struggles to generalize (task F in
Figure 1c).
[Figure]
Unfortunately, developing such a qualified task augmentation approach remains challenging.
First,the task-aware methods sacrifice task diversity for task-awareness – they establish task-awarenessby injection of the same random noise to labels of the support and query set [18], rotation of bothsupport and query images [15], or mix-up of support and query examples within each task [29], all ofwhich result in augmented tasks that are within the immediate vicinity of sampled meta-training tasksas shown in Figure 1a.
Second, the task-imaginary method [32] that mixes up both support examplesof two distinct tasks and their query examples in the feature space compromises on task-awareness –the resulting examples are even multi-modal in Figure 1e and constitute an erroneous task that failsto comply with the task distribution.
Third, how to adaptively augment tasks that maximally improvethe meta-knowledge and thereby the performance on meta-testing tasks remains unexplored.
To this end, we propose the Adversarial Task Up-sampling (ATU) framework to augment tasks thatare aware of the task distribution, imaginary, and adaptive to the current meta-knowledge.
Groundedon gradient-based meta-learning algorithms that are generally applicable to either regression orclassification problems, ATU takes the initialization of the base learner as the meta-knowledge.
Concretely, ATU consists of a task up-sampling network whose input is a task itself and outputsare augmented tasks.
To ensure that the augmented tasks are imaginary and meanwhile faithful tothe underlying task distribution, we train the up-sampling network to minimize the Earth Mover
Distance between augmented tasks and the local task distribution characterized by a set of sampledtasks.
Besides, we enforce the up-sampling network to produce challenging tasks that complementthe current initialization, by maximizing the loss of the model adapted from the initialization on theirquery examples and minimizing the similarity between the gradient of the initialization with respectto their support examples and that of their query examples.
In summary, our main contributions are three-fold: (1) we present the first task-level augmentationnetwork that learns to generate tasks that simultaneously meet the qualifications of being taskaware, task-imaginary, and model-adaptive; (2) we provide a theoretical analysis to justify thatthe proposed ATU framework indeed promotes task-awareness; (3) we conduct comprehensiveexperiments covering both regression and classification problems and a total of five datasets, wherethe proposed ATU improves the generalization ability of gradient-based meta-learning algorithms byup to 3.81%.
Related Work
[Table]
As a paradigm that effectively adaptsthe meta-knowledge learned frompast tasks to accelerate the learning of new ones, meta-learning hassparked considerable interest, especially for few-shot learning.
It fallsinto four major strands based on whatthe meta-knowledge is, i.e., optimizer-based methods [2, 27], feed-forward methods [19, 6, 28],metric-based methods [21, 23, 26] and gradient-based methods [5, 11, 31, 8], where the inner op- timizer, the mapping function from the support set to the task-specific model, the distance metricmeasuring the similarity between samples, and the parameter initialization are formulated as themeta-knowledge that enables quick adaptation to a task within a small number of steps.
Our methodis primarily evaluated on gradient-based methods which enjoy wide adoption and applicability ineither classification or regression problems.
Within-task Overfitting.
Few-shot learning puts meta-learning, especially gradient-based methodswhich require optimization of high-dimensional parameters within each task, at risk of within-taskoverfitting.
Some works tackle the problem with various ways of reducing the number of parametersto adapt in the inner loop: only updating the head [17] or the feature extractor [16], learning datadependent latent generative embeddings of parameters [20] or context parameters [37] to adapt,imposing gradient dropout [25], and generating stochastic input-dependent perturbations [9].
Theother bunch of works alleviates the problem through data augmentation within each task.
Ni et al.[15] applies standard augmentation like Random Crop and CutMix onto support samples.
Sun etal.
[22] and Zhang et al.
[36] proposed to generate more data within a class via a ball generator andgenerative adversarial networks, respectively.
These techniques designed for within-task overfitting,however, have been proved to lend little support to meta-overfitting which we focus on.
Meta-overfitting.
Distinguished from traditional overfitting within a task, two types of metaoverfitting including memorization and learner overfitting have been pinpointed in [33, 18].
Despitemeta-regularization techniques [7, 33] that limit the capacity of the meta-learner, task augmentationstrategies [18, 14, 15, 12, 32, 29] have emerged as more effective solutions to meta-overfitting.
Table 1presents a summary of these strategies, except the strategies of large rotation [12] being part of MetaMaxup [15] and DReCa [14] applicable to natural language inference tasks only.
MetaAug [18]augments a task by adding a random noise on labels of both support and query sets, and MetaMix [29]mixes support and query examples within a task.
Such within-task augmentation guarantees thevalidity of augmented tasks, i.e., being task-aware, though it almost does not alter the mappingfrom the support to the query set, i.e., generating limited imaginary tasks beyond meta-trainingtasks.
Meta-Maxup [15] and MLTI [32] approach this problem via the cross-task mixup method,unfavorably at the expense of erroneous tasks.
Our work seeks a novel task augmentation frameworkcapable of generating tasks that not only meet the task-awareness and task-imagination needs butalso adapt to maximally benefit the up-to-the-minute meta-learner.
Preliminaries
3.1 Meta-Learning Problem and Gradient-Based Meta-Learning
Meta-Learning model f are trained and evaluated on episodes of few-shot learning tasks.
Assume thetask distribution is p ( T ) .
A few-shot learning task T i
i.i.d.
sampled from p ( T ) consists of a supportset D si =
( X si , Y si )
= { ( x si,j , y si,j ) } K s j =1 and a query set D qi =
( X qi , Y qi ) =
{ ( x qi,j , y qi,j ) } K q j =1 , where
X si and Y si , ( X qi and Y qi ) are the collection of inputs and labels in support (query) set, and K s ( K q )is the size of support (query) set.
The most representative gradient-based meta-learning algorithm is MAML [5].
MAML aims to learnan initialization parameter θ 0 of the model f
that can be adapted to any new task after a few stepsof gradient update.
Concretely, given a specific task D si , D qi and a parametric model f θ , MAMLinitializes the model parameter θ by θ 0 and updates θ by performing gradient descent on the supportset D si .
It then optimizes the initialization parameter θ 0 by minimizing the loss L estimated on thequery set D qi .
The objective of MAML can be formulated as
[Equation]
3.2 Earth Mover’s Distance
To estimate the distance between two tasks, we use Earth-Mover Distance (EMD).
Earth-Mover
Distance, a.k.a.
Wasserstein metric, is a distance measure of two probability distributions or two setsof points, and is widely used in image retrieval and point cloud up-sampling works [35, 34].
Giventwo sets S 1 and S 2 with the same size, EMD calculates their distances as:
[Equation]
𝑇𝑇𝑇𝑇𝑇𝑇 3 Task Patch 𝓣𝓣 𝒑𝒑
Latent task manifold
Set Encoder
MetaLearner
Coarse Tasks 𝓣𝓣 𝒄𝒄
Ground-Truth Task 𝓣𝓣 𝒈𝒈 𝒉𝒉 𝒔𝒔 = 𝒈𝒈 𝒔𝒔 𝓣𝓣 𝒑𝒑 Patch Feature
Decoder 𝒈𝒈 𝒅𝒅 ( ⋅ )
Perturbations 𝓩𝓩 CoarseGenerator 𝒅𝒅 𝑬𝑬𝑬𝑬𝑬𝑬 ( 𝓣𝓣 𝒖𝒖𝒑𝒑 , 𝓣𝓣 𝒈𝒈 )
(EMD Loss) 𝟏𝟏 𝒯𝒯 𝒖𝒖𝒑𝒑 ∑ 𝑻𝑻 𝒊𝒊 ∈𝒯𝒯 𝒖𝒖𝒑𝒑 ℒ 𝒂𝒂𝒅𝒅𝒂𝒂 ( 𝑻𝑻 𝒊𝒊 ) (Adversarial Loss)
Up-sampled Task 𝓣𝓣 𝒖𝒖𝒑𝒑 ⋅⋅⋅ ⋅⋅⋅ 𝒈𝒈 𝒔𝒔 ( ⋅ ) 𝒈𝒈 𝒄𝒄 ( ⋅ )
Stage-I
Stage-II where ϕ is a bijective projection mapping S 1 to S 2 .
The value of EMD in (2) can be obtained bysolving the linear programming problem w.r.t.
ϕ .
Adversarial Task Up-sampling
In practice, the task distribution p ( T ) is unknown and we optimize the meta parameter θ 0 with anempirical estimation of Eq. (1) over of meta-training tasks { T i } N T i =1 as [Equation]
[Equation]
Given a finite set of meta-training tasks, the empirical task distribution may deviate from the truetask distribution.
The meta-model trained on such a finite set of tasks will cause memorization orlearner overfitting [33, 18], which hurts the generalization to new tasks.
To alleviate this problem,we propose a new task up-sampling network to generate a sufficient number of diverse tasks suchthat the empirical task distribution formed by the original meta-training tasks and the augmentedtasks together is closer to the true task distribution.
To achieve this, the tasks generated by the taskup-sampling network should match the true task distribution and cover a large fraction of it.
However,since the true task distribution and its underlying manifold are unknown, we cannot provide thetask up-sampling network with explicit information about it.
Instead, we generate new tasks byperforming Task Up-sampling (TU) from a set of training tasks that implicitly comprise the latenttask manifold information.
The idea of Task Up-sampling is inspired by point cloud up-samplingmethods [35, 34], which generate up-sampled points lying on the latent distribution (i.e., the shape)of the given local point patch.
Similar to the point cloud up-sampling algorithm, our augmentationnetwork receives a task patch consisting of a set of tasks T p
=
{ T i } N p
i
=1 where N p is the set size, andgenerates up-sampled tasks T up = { ˆ D s , ˆ D q } that are uniformly distributed over the same underlyingtask distribution as the task patch.
Due to the high complexity of task generation, it is infeasible to directly generate up-sampling taskswithout sacrificing the quality of the tasks.
Inspired by [35], we propose a two-stage generationstrategy to generate the up-sampled tasks.
In the first stage, we produce a sparse set of tasks, aimingat recovering the global task distribution of the task patch.
The tasks obtained in the first stage arecalled coarse tasks.
In the second stage, we generate multiple tasks for each coarse task, aiming tocharacterize the local task distribution around each coarse task.
To guide the second generation stage,we use the patch features of the input task set as input to provide global task information and alsomultiple random noise vectors as input to provide directional perturbations to generate diverse tasksaround the coarse task.
The generation process is summarized in Fig.
Our proposed task up-sampling network consists of 3 components, namely, a coarse task generatorg c ( · ) , a set encoder g s ( · ) , and a decoder g d ( · ) .
The coarse task generator is similar to a set autoencoder.
It first encodes the information of the whole input task set and then decodes it to generater c N p coarse tasks T c =
{ T ci } .
The set encoder, denoted by g s ( · ) , extracts the set information of theinput task patch as a patch feature h s to provide global information in the second-stage generation.
For each task T ci in T c , the decoder g d ( · ) generates r d tasks located around T ci in the task manifold by taking as input r d random perturbations { z i } ri =1 that are i.i.d.
noise sampled from a uniformdistribution and the set feature
h s as input.
In general, we use the same perturbations for eachcoarse task T ci in T c .
Finally, we obtain the up-sampled task set T up consisting of rN p tasks as
T up = g d ( T c , Z , h s ) , where r = r c × r d is the up-sampling ratio.
We denote the task augmentationnetwork by G θ g ( T p , Z ) where θ g is the trainable parameters of the task augmentation network.
In each iteration of the training phase, we construct rN p tasks to form the ground truth tasks set T g(e.g., randomly select from the meta training task set).
Then we sample N p tasks from T g to formthe task patch T p and randomly sample r perturbation noise vectors to form the perturbation set
Z = { z i } ri =1 .
By feeding T p and Z to the task augmentation network, we obtain the up-sampledtask set T up =
G θ g ( T p , Z ) .
To train the task augmentation network, we apply EMD loss betweenthe up-sampled task set T up and the ground-truth task set T g to encourage the generated task setto have the same distribution as the true task distribution.
However, it may still be insufficient tomake the up-sampled tasks cover a significant fraction of the true task distribution.
In this case, theup-sampling tasks provide limited additional information compared with the original meta-trainingtasks, and thus the meta-learner has restricted benefit from the generated tasks.
To generate moreinformative tasks for the meta-learner, we want the generated tasks to be difficult for the current metamodel θ 0 .
Following [30], we measure the difficulty of the a task for θ 0 by the loss estimated on queryset w.r.t.
to ϕ i , i.e. L ( ϕ i , ˆ D qi ) , and the gradient similarity between the support and query sets w.r.t.
θ 0 ,i.e. ⟨∇ θ
L ( θ 0 , ˆ D si ) , ∇ θ 0 L ( θ 0 , ˆ D qi ) ⟩ .
The large loss and small gradient similarity indicate a difficulttask.
Therefore, we want to maximize the following objective function to generate informative tasks:
[Equation]
where η 1 and η 2 are two hyperparameters that control the strength of the two terms in L adv .
We callthis loss adversarial loss because it aims to increase the difficulty of the up-sampling tasks for themeta-learner while the meta-learner is trained to minimize the loss on the generated difficult tasks.
And we named the proposed algorithm as Adversarial Task Up-sampling (ATU).
Together with the EMD loss, we obtain the objective to train the task up-sampling network:
[Equation]
Note that the gradient of Eq.
(5) will not be backpropagated to the meta model θ 0 and the meta modelwill be updated by minimizing the meta loss in Eq.
(3) on the up-samples tasks T u .
We summarizethe proposed ATU in Algorithm 1 in Appendix A.
4. ATU on Regression and Classification Problem
Regression Tasks.
We consider a simple regression problem: sinusoidal regression, which is widelyused to evaluate the effectiveness of the meta-learning methods.
In sinusoidal regression problem,before feeding a task T i =
( D si , D qi ) to the task up-sampling network, we represent the task bya connection vector as [ x s 1 , y s 1 , x s 2 , y s 2 , ..., x sK s , y sK s , x q 1 , y q 1 , x q 2 , y q 2 , ..., x qK q , y qK q ] .
In particular, wesort the support set and query set by the input respectively, such that x s 1 ≤ x s 2 ≤ ... ≤ x sK s andx q 1 ≤ x q 2 ≤ ... ≤ x qK q .
After sorting, the task input is invariant to the permutation of data insupport and query sets, and the feature extracted for each task is a permutation-invariant feature.
Thissimplifies the design of the task up-sampling network.
To generate the coarse tasks, we first use theset encoder g s to extract the set feature h s and directly generate the coarse tasks from the set featureh s .
For the regression problem, we add an extract EMD loss on the support and query set for eachgenerated task ˆ T i ∈ T up to encourage the points in the generated support set and query set to followthe same sinusoidal distribution, and the objective becomes
[Equation]
After generating the up-sampled tasks, we transform the task representation feature, which is aconcatenating vector, back to a support set and a query set.
Classification Tasks.
Dissimilar to regression tasks, classification tasks’ labels of each class arerandomly given under the mutually-exclusive setting [33].
Therefore, we could neglect the labels and mainly focus on image generation during the classification up-sampling process.
Concretely, foran N -way K s -shot classification problem with K q query samples for each class, we split the taskinto ( K s + K q ) N -way 1 -shot classification tasks without query examples.
We represent the task byconcatenating the input from N classes in a fixed order (based on classes).
We treat the ( K s + K q )tasks as a task patch and feed them to the task up-sampling network.
Since the task distributionof the image classification problem is extremely complex, it is impractical to generate the coarsetasks from a set feature.
Instead, we use the original tasks as the coarse tasks and generate theup-sampled tasks by a more informative perturbance around the original tasks.
To achieve this, wegenerate the perturbation by randomly sampling extra K M images from N different classes in thebase set.
Then for the image x i in a class of a task in the coarse tasks T c , we subtract it from the K Mimages to obtain K M residual images and their corresponding set features (i.e., obtained from g s ( · ) ),concatenate the set features with a noise vector, and use an attention network to obtain a residualimages feature x resi for the image x i given K M residual images.
Finally, we generate the image asx ui = x i + x resi for the augmented task.
We repeat this process for all images in a coarse task to get anaugmented task and apply the r noise vector to get r up-sampled tasks.
The whole training objectivefunction is Eq.
More details of network structures and training details are shown in Appendix B.
Theoretical Analysis
We will introduce the formal definition of an up-sampled task that conforms to task-awareness, basedon which we present the essential property of our proposed ATU framework in maximizing thetask-awareness, compared to previous task augmentation approaches.
Definition 1 (Task-aware Up-sampling) .
Suppose that we are given a set of N p tasks { X i , Y i } N p
i =
1from which we up-sample a new task T u .
For each i -th task, its ground-truth parameter that map theinput X i to the output Y i is θ i , i.e., Y i =
f θ i
( X i ) .
The up-sampled task T u = { X u , Y u } is definedto be task-aware, if and only if θ u
= g ( θ 1 , · · · , θ N u ) and Y u
= f θ u ( X u ) where g is the up-samplingfunction and θ u is the up-sampled parameter.
This definition states two prerequisites a task-aware up-sampling has to meet: (1) the up-sampling isperformed in the functional space, which is to relate N u parameters via g ; (2) the mapping betweenthe input and the output of an up-sampled task satisfies f θ u .
Property 1 (Task-awareness Maximization) .
Consider N u = 2 , g ( θ 1 , θ 2 )
=
(1 − λ )
θ 1 + λθ 2 ,f θ 1 ( · ) =
W 1 , and f θ 2 ( · )
=
W 2 .
The proposed ATU algorithm that pursues an up-sampled task
T up = { X u , Y u } via minimizing the EMD loss between T 1 and T 2 maximizes the task-awareness,i.e., minimizing the distance between Y u and f θ u ( X u ) .
Proof.
According to the definition of EMD (Eq.
, it solves: ϕ ∗ =
arg min
ϕ ∈ Φ (cid:80) j
∥ x 1 ,j − x 2 ,ϕ ( j ) ∥ 2 ,where Φ = {{ 1 , · · · , n 1 }(cid:55)→{ 1 , · · · ,n 2 }} denotes the set containing all possible bijective assignments,each of which gives one-to-one correspondence between T 1 and T 2 .
Based on the optimal assignmentsϕ ∗ , the EMD is known to be defined as d EMD = 1 min { n 1 ,n 2 } (cid:80) j ∥ x 1 ,j − x 2 ,ϕ ∗ ( j ) ∥ 2 .
In light ofthe difficulty in mathematically formulating a possible up-sampled task ˜ T u that lies in the localmanifold of { T 1 , T 2 } , we reasonably assume a simplified way of characterizing an up-sampledtask ˜ T u to be ˜ y u,j = α T 1 ,j Y 1 + α T 2 ,j Y 2 , ˜ x u,j = α T 1 ,j X 1 + α T 2 ,j X 2 , ∀ j, where each sample is a convexcombination of samples from both T 1 and T 2 . The combination coefficients α 1 ,j , α 2 ,j ∈ R ( K s + K q ) × 1 , 245 (cid:80) K s + K q k α 1 ,jk =1 , (cid:80) k
α 2 ,jk =1 , α 1 ,jk , α 2 ,jk ≥ 0 , ∀ k . Different combination coefficients lead to a setof up-sampled task candidates { ˜ T u } .
We evaluate the task-awareness property of each candidate˜ T u , i.e., the distance between ˜ Y u and f θ u ( ˜ X u ) , to be ∥ ˜ Y u − f θ u ( ˜ X u ) ∥ 2 = (cid:80) j ∥ ˜ y u,j − f θ u (˜ x u,j ) ∥ 2 = 248 (cid:80) j ∥ ( W 1 − W 2 ) [ λα T 1 ,j X 1 − (1 − λ )
α T 2 ,j X 2 ] ∥ 2 = LHS .
(See Appendix E.)
Note that LHS ≤ (cid:80) j ∥ W 1 − W 2 ∥ 2 ( λ ∥ ˜ x u,j − x 2 ,ϕ 2 ( j ) ∥ 2 + ∥ X 2 ∥ 2 ) and
LHS ≤ (cid:80) j ∥ W 1 −
W 2 ∥ 2 ((1 − λ )
∥ x 1 ,ϕ 1 ( j ) − ˜ x u,j ∥ 2 + ∥ X 1 ∥ 2 ) . (See Appendix E.) By combining the two inequalities above, we have
[Equation]
−λ ) ∥ λ x 1 ,ϕ 1 ( j ) − ˜ x u,j ∥ 2 + ∥ X 1 ∥ 2 } .
In practice, it is easy to normalize all the tasks in thefeature space, which leads to ∥ X 1 ∥ 2 = ∥
X 2 ∥ 2 .
Therefore, by minimizing the EMDloss d EMD = min { min ϕ 2 (cid:80) j ∥ ˜ x u,j − x 2 ,ϕ 2 ( j ) ∥ 2 , min ϕ 1 (cid:80) j ∥ ˜ x u,j − x 1 ,ϕ 1 ( j ) ∥ 2 } , the proposed task upsampling network identifies from the candidate set { ˜ T u } the task T u that has the minimal distancebetween Y u and f θ u ( X u ) ; in other words, the task-awareness is maximized.
Previous task augmentation approaches directly mix up two tasks without minimizing the EMD loss,i.e., y u,j = (1 − λ )
y 1 ,j + λ y 2 ,j , x u,j =
(1 − λ ) x 1 ,j + λ x 2 ,j .
In this case, the task-awareness is unwarranted as we have illustrated in Section 1, provided that ∥ Y u − f θ u ( X u ) ∥ 2 = (cid:80) j ∥ (1 − λ ) y 1 ,j +λ y 2 ,j − [(1 − λ ) W 1 + λ W 2 ] [(1 − λ ) x 1 ,j + λ x 2 ,j ] ∥ 2 = (cid:80) j λ 2 (1 − λ ) 2 ∥ ( W 1 − W 2 )( x 1 ,j − x 2 ,j ) ∥ 2 .
Experiments
To evaluate the effectiveness of ATU, we conduct extensive experiments to answer the followingquestions: Q1:
How does ATU perform compared to state-of-the-art task-augmentation-basedand regularization meta-learning methods?
Q2:
Whether can the proposed ATU consistently improve performance for different meta-learning methods?
Q3:
What does up-sampled task by ATUlooks like?
Q4:
What is the influence of increasing the task number within meta-training data onthe performance improvement of ATU?
Benchmarks.
We compared ATU with state-of-the-arttask augmentation strategies for meta-learning, including MetaAug [18], MetaMix [29], MetaMaxup [15], MLTI [32], and regularization methods, including MetaDropout [9], TAML [7], and
Meta-Reg [33]for both regression and classification problems.
We also consider a variant of ATUwhich removes the adversarial loss L adv and trains the task augmentation network only through the
EMD loss.
We denote this variant by TU.
To validate the consistent effect of ATU in improvingdifferent meta-learners, we apply ATU and AU on MAML [11], MetaSGD [11] and ANIL [17].
We also consider cross-domain settings where the meta-testing tasks are from different domains. [Table] [Table] Learner: MetaSGD
[Table] [Table]
MLTI
[Table]
[Table]
TU
[Table]
ATU
[Table]
4    6XSSRUW4XHU\ ( [Figure] MLTI [Figure]
[Figure]
[Figure]
0L[HG4XHU\6XSSRUW4XHU\
[Figure]
4  6XSSRUW4XHU\ (
[Figure]
TU    4  6XSSRUW4XHU\ (
[Figure]
) ATU
Experimental Setup.
Following [11], we construct the K-shot regression task by sampling fromthe target sine curve y ( x ) =
Asin ( ωx + b ) , where the amplitude A ∈ [0 . 1 , 5 . 0] , the frequencyω ∈ [0 . 8 , 1 . 2] , the phase b ∈ [0 , π ] and x is sampled from [ − 5 . 0 , 5 . 0] .
In the meta-training phase, eachtask contains K support and K target (K=10) examples.
We adopt mean squared error (MSE) as theloss function.
For the base model f θ , we adopt a small neural network, which consists of an inputlayer of size 1, 2 hidden layers of size 40 with ReLU and an output layer of size 1.
We use onegradient update with a fixed step size α =0 .
01 in inner loop, and use Adam as the outer-loop optimizerfollowing [5, 11].
Moreover, the meta-learner is trained on 240,000 tasks with meta batch-size being4.
In meta-testing stage , we randomly sample 100 sine curves as meta-test tasks, each task containing
K support samples and 100 query examples.
The data points x in query set are evenly distributed on[ − 5 . 0 , 5 . 0] .
The averaged MSE with 95 % confidence intervals upon these 100 sine curves with K=10,20, 30 are reported in Table 2.
We also perform cross-domain experiments by sampling 100 sinecurves which have different frequencies, amplitudes or phases from the tasks in meta-training set andreport the results in Table 3.
More settings about the up-sampling networks are listed in Appendix C.
Performance.
The results in Table 2 and Table 3 show that ATU consistently outperforms the baselinemethods MAML, MetaMix, and MLTI in different K-shot (K ∈ { 10 , 20 , 30 } ) settings and various domain settings.
These results validate that the tasks generated by ATU can better approximate thetrue task distribution and provide more information to the meta-learner than MetaMix and MLTI, thusenabling better generalization of the model.
We further verify the superiority of the proposed methodsby visualizing the augmented tasks generated by the proposed methods and the baseline methods.
The visualization results in Fig.
3 show that the points in the tasks generated by TU and ATU fit thesine curve well, while the points in the tasks generated by MLTI and MetaMix deviate from the sinecurve.
This indicates that augmented tasks generated by TU and ATU match the true task distribution.
It is noteworthy that the support set and query set generated by ATU differ significantly from thosegenerated by TU, which indicates that the task generated by ATU is more difficult .
This, togetherwith the results that ATU outperforms TU in most experiments, demonstrates the effectiveness of theadversarial losses in generating informative tasks to improve generalization of the meta-learner.
In Fig.
4, we also visualize the adaptation of meta-learner trained by different task augmentation methods for a 10-shot meta-test regression task.
Compared to the MAML trained on original meta-training tasks, the MAML trained on tasks generated by TU fits the ground-truth sinusoid after only one update.
And ATU performs even better than TU.
This again validates that the augmented tasks generated by TU and ATU are more informative for the meta-learner to learn the meta knowledge from the true task distribution.
[Figure]
6.2 Classification
[Figure]
miniImagenet
[Table]
ISIC
DermNet
[Table]
Experimental setup.
We follow MLTI to evaluate the performance of task augmentation algorithmsfor few-shot classification problem with limited number of base classes in meta-training set undernon-label-sharing settings.
We consider four datasets (base classes number): miniImagenet-S (12),
ISIC [13] (4), Dermnet-S (30)
[Table]
and Tabular Murris [4] (57) covering classification tasks on generalnatural images, medical images
[Table]
and gene data.
Note that the miniImagenet-S and Dermnet-S areconstructed by limiting the base classes of miniImangenet [26] and Dermnet [1], respectively.
Weconstruct N -way K -shot tasks, setting N = 5 for miniImagenet-S, Derment-S, Tabular Murris and
N = 2 for ISIC dataset due to its limited number of base classes and setting K = 1 or K = 5 .
Recallthat TAU relies on extra K M images to generate augmented tasks in image classification problem.
To make the training process more efficient, we set K M to 3 and the upsampling rate r be 2.
Moredetails of these datasets and the settings of the task augmentation networks are listed in Appendix D.
Performance.
We show the performance on the four datasets in Table 4.
On all four datasets, theproposed ATU consistently outperforms the baseline methods, including the augmentation-basedmethods (i.e. MetaMix, Meta-Maxup and MLTI) and regularization-based methods (Meta-Reg,
TAML and Meta-Dropout).
And TU achieves the second best performance on all experiments.
We also observe that our method achieves a large improvement on the ISIC dataset which consists of only 4 base classes, indicating the effectiveness of our method in limited tasks scenarios.
We further evaluate the effectiveness of the proposed ATU on improving the generalizationfor different backbone meta-learner by conducting experiments under 1-shot setting to comparethe performance of MLTI and ATU in improving the performance of the backbone meta-learner
MetaSGD and ANIL.
The results are presented in Table 5.
ATU again consistently outperforms
MLTI.
All these results validate the superiority of the proposed ATU and TU over the existingbaselines in generating informative tasks to improve the performance of different backbone metalearners.
We also evaluate the performance of ATU in cross-domain adaptaion settings.
In Table 6,we present the results of the experiment that apply the meta-model trained on miniImageNet-Sto Dermnet-S, and vice versa.
ATU improves the generalization performance of MAML (thebackbone meta-learner in this experiment) by a large margin.
This indicates ATU can consistently improve the backbone meta-model’s generalization ability under the challenging cross-domain settings
[Table] [Table] [Table]
[Figure] [Figure]
Effect of the number of meta-training tasks.
We analyze the change of the performance improvement of ATU’s over MAML with the number of meta-training tasks in 1-shot and 5-shot settings.
The results presented in Fig.
5 shows that ATU significantly improve the performance of MAML byabout 4.5%-5% when the number of base classes is 12, while the improvement decreases with thenumber of base classes increasing.
When the number of base classes increases, the number of trainingtasks increases rapidly and the empirical task distribution constructed from the meta-training tasksbecomes closer and closer to the true task distribution.
Therefore, the extra information providedby tasks generated by ATU becomes less.
However, even if all available base classes are used in themeta-training (i.e., 64 meta-training classes), ATU still helps to improve the performance of MAML.
Visualization of the generated tasks.
We visualize the up-sampled tasks by t-SNE to evaluate theirquality for MAML on miniImagenet
-S. We up-sample 100 tasks for 5 original tasks via ATU byusing different perturbations for each task.
Each task is represented by the concatenated vector of thesupport and query sets.
The results presented in Fig.
6 shows that the up-sampled tasks stays near theoriginal tasks, matching the true task distribution.
This indicates the generated tasks are task-aware.
Besides, we can observe that the augmented tasks are diverse and cover a substantial portion aroundthe original tasks.
This demonstrates the task imaginary property of the augmented tasks.
These twoobservations suggest that the proposed ATU is a qualified task augmentation algorithm.
Conclusion and Limitation
In this paper, we propose the first task-level up-sampling network that learns to generate tasks thatsimultaneously meet the qualifications of being task-aware, task-imaginary, and model-adaptive.
Theproposed Adversarial Task Up-sampling (ATU) takes a set of tasks as input and learn to up-sampletasks complying with the true task distribution while being informative to improve the generalizationof the meta-learner.
We theoretically justify that ATU promotes task-awareness and empiricallyverify that ATU improves the generalization of various backbone meta-learner for both regressionand classification tasks on five datasets.
Limitations.
Our theoretical results are obtained undersome strong assumptions, but all the experiments and visualization outcomes validate our method’seffectiveness in real settings.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ??
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] .
(b) Did you describe the limitations of your work?
[Yes] .
See Section (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] .
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] .
See Section
(b) Did you include complete proofs of all theoretical results?
[Yes] .
See Section 53.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] .
We exploit the
open-source datasets and will release the code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?[Yes] .
See Section 6.2 and Appendix C/D.(c)
Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?[Yes] .
See Section 6.2.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] .
See Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?[Yes] .
See Section 6.2.
(b) Did you mention the license of the assets?[N/A] .
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] .
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] .
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A] .
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] .
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A] .