M ETA -L EARNING WITH F EWER T ASKS THROUGH T ASK I NTERPOLATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Meta-learning enables algorithms to quickly learn a newly encountered task with just a few labeled examples by transferring previously learned knowledge.
However, the bottleneck of current meta-learning algorithms is the requirement of a large number of meta-training tasks, which may not be accessible in real-world scenarios.
To address the challenge that available tasks may not densely sample the space of tasks, we propose to augment the task set through interpolation.
By meta-learning with task interpolation (MLTI), our approach effectively generates additional tasks by randomly sampling a pair of tasks and interpolating the corresponding features and labels.
Under both gradient-based and metric-based meta-learning settings, our theoretical analysis shows MLTI corresponds to a data-adaptive meta-regularization and further improves the generalization.
Empirically, in our experiments on eight datasets from diverse domains including image recognition, pose prediction, molecule property prediction, and medical image classification, we find that the proposed general MLTI framework is compatible with representative meta-learning algorithms and consistently outperforms other state-of-the-art strategies.
I NTRODUCTION
Meta-learning has powered machine learning systems to learn new tasks with only a few examples, by learning how to learn across a set of meta-training tasks.
While existing algorithms are remarkably efficient at adapting to new tasks at meta-test time, the meta-training process itself is not efficient.
Analogous to the training process in supervised learning, the meta-training process treats tasks as data samples and the superior performance of these meta-learning algorithms relies on having a large number of diverse meta-training tasks.
However, sufficient meta-training tasks may not always be available in real-world.
Take medical image classification as an example: due to concerns of privacy, it is impractical to collect large amounts of data from various diseases and construct the meta-training tasks.
Under the task-insufficient scenario, the meta-learner can easily memorize these meta-training tasks, limiting its generalization ability on the meta-testing tasks.
To address this limitation, we aim to develop a strategy to regularize meta-learning algorithms and improve their generalization when the meta-training tasks are limited and only sparsely cover the space of relevant tasks.
Recently, a variety of regularization methods for meta-learning have been proposed, including techniques that impose explicit regularization to the meta-learning model (Jamal and Qi, 2019;
Yin et al., 2020) and methods that augment tasks by making modifications to individual training tasks through noise (Lee et al., 2020) or mixup (Ni et al., 2021; Yao et al., 2021).
However, these methods are largely designed to either tackle only the memorization problem (Yin et al., 2020) or to improve performance of meta-learning (Yao et al., 2021) when plenty of meta-training tasks are provided.
Instead, we aim to target the task distribution directly, leading to an approach that is particularly well-suited to settings with limited meta-training tasks.
Concretely, as illustrated in Figure 1, we aim to densify the task distribution by providing interpolated tasks across meta-training tasks, resulting in a new task interpolation algorithm named MLTI ( M etaL earning with T ask I nterpolation).
The key idea behind MLTI is to generate new tasks by interpolating between pairs of randomly sampled meta-training tasks.
This interpolation can be instantiated in a variety of ways, and we present two variants that we find to be particularly effective.
The first labelsharing (LS) scenario includes tasks that share the same set of classes (e.g., RainbowMNIST (Finn
𝒯𝒯𝒯෡𝒯෡𝒯෡𝒯෠𝒯෠𝒯෠𝒯
[Figure]
෠𝒯 1.1 ෠𝒯 1.𝑛 … ෠𝒯 2.1 ෠𝒯 2.𝑚 …
[Figure]
et al., 2019)).
For each LS task pair randomly drawn from the meta-training tasks, MLTI linearly interpolates their features and accordingly applies the same interpolation strategy on the corresponding labels.
The second non-label-sharing (NLS) scenario includes classification tasks with different sets of classes (e.g., miniImagenet).
For each additional NLS task, we first randomly select two original meta-training tasks and then generate new classes by linearly interpolating the features of the sampled classes, which draw one class in each original task without replacement.
Since MLTI is essentially changing only the tasks, it can be readily used with any meta-learning approach and can be combined with prior regularization techniques that target the model.
In summary, our primary contributions are: (1) We propose a new task augmentation method (MLTI) that densifies the task distribution by introducing additional tasks; (2) Theoretically, we prove that
MLTI regularizes meta-learning algorithms and improves the generalization ability.
(3) Empirically, in eight real-world datasets from various domains, MLTI consistently outperforms six prior metalearning regularization methods and is compatible with six representative meta-learning algorithms.
P RELIMINARIES
Problem statement.
In meta-learning, we assume each task T i is i.i.d.
sampled from a task distribution p ( T ) associated with a dataset D i , from which we i.i.d.
sample a support set D si = ( X si , Y si ) = { ( x si,k , y si,k ) }
N s k =1 and a query set D qi =
( X qi , Y qi ) =
{ ( x qi,k , y qi,k ) }
N q k
=1 .
Given a predictive model f (a.k.a., the base model) with parameter θ , meta-learning algorithms first train the base model on meta-training tasks.
Then, during the meta-testing stage, the well-trained base model f is applied to the new task T t with the help of its support set D st and finally evaluate the performance on the query set D qt .
In the rest of this section, we will introduce both gradient-based and metric-based meta-learning algorithms.
For simplicity, we omit the subscript of the meta-training task index i in the rest of this section.
Gradient-based meta-learning.
In gradient-based meta-learning, we use model-agnostic metalearning (MAML)
(Finn and Levine, 2018) as an example and denote the corresponding base model as f MAML .
Here, the goal of MAML is to learn initial parameters θ ∗ such that one or a few gradient steps on D s leads to a model that performs well on task T .
During the meta-training stage, the performance of the adapted model f ϕ is evaluated on the corresponding query set D q and is used to optimize the model parameter θ .
Formally, the bi-level optimization process with expected risk is formulated as:
[Equation]
where η denotes the inner-loop learning rate and L is defined as the loss, which is formulated as crossentropy loss (i.e., E T ∼ p ( T ) [ − (cid:80) k log p ( y qk | x qk , f ϕ )] ) and mean square error (i.e., E T ∼ p ( T )
[ (cid:80) k
∥ f ϕ ( x qk ) − y qk ∥ ] ) for classification and regression problems, respectively.
During the meta-testing stage, for task
T t , the adapted parameter ϕ t is achieved by fine-tuning θ t on the support set D st .
Metric-based meta-learning.
The aim of metric-based meta-learning is to perform a non-parametric learner on the top of meta-learned embedding space.
Taking prototypical network (ProtoNet) with base model f PN as an example (Snell et al., 2017), for each task T , we first compute class prototype representation { c r } Rr =1 as the representation vector of the support samples belonging to class k as:
[Equation]
where D sr represents the subset of support samples labeled as class r and the number of this subset is
N r .
Then, given a query data sample x qk in the query set, the probability of assigning it to the r -th class is measured by the distance d between its representation f PNθ ( x qk ) and prototype representation c r , and the cross-entropy loss of ProtoNet is formulated as:
[Equation]
At the meta-testing stage, the predicted label of each query samples is assigned to the class with maximal probability (i.e., ˆ y qk = arg max r p ( y qk = r | x qk ) ).
The estimation of the expected loss in Eqn.
(1) or (3) is challenging since the distribution p ( T ) is unknown in practical situations.
A common way of estimation is to approximate the expected risk in
Eqn.
(1) by a set of meta-training tasks {T i } | I | i =1 (use MAML as an example):
[Equation]
D
[Equation]
) , where ϕ i = θ − α ∇
[Equation]
L
[Equation]
D
[Equation] [Equation]
However, this approximation method still faces the challenge: optimizing Eqn.
(4), as suggested in (Rajendran et al., 2020; Yin et al., 2020), can result in memorization of the meta-training tasks, thus limiting the generalization of the meta-learning model to new tasks, especially in domains with limited meta-training tasks.
M ETA - LEARNING WITH T ASK I NTERPOLATION
To address the memorization issue described in the last section, we aim to develop a framework that allows meta-learning methods to generalize well to new few-shot learning tasks, even when the provided meta-training tasks are only sparsely sampled from the task distribution.
To accomplish this, we introduce meta-learning with task interpolation (MLTI).
The key idea behind MLTI is to densify the task distribution by generating new tasks that interpolate between provided meta-training tasks.
This approach requires no additional task data or supervision, and can be combined with any base meta-learning algorithm, including MAML and ProtoNet.
Before detailing the proposed strategy, we first discuss two scenarios of meta-training task distributions, label-sharing and non-label-sharing tasks, which have distinct implications for task interpolation.
Formally, we define these two scenarios as:
Definition 1 (label-sharing tasks)
If the labels of all tasks share the same label space, we refer it as the label-sharing (LS) scenario.
Take Pascal3D pose prediction (Yin et al., 2020) as an example, each task is to predict the current orientation of the object relative to its canonical orientation, and the range of canonical orientation is shared across all tasks.
Definition 2 (non-label-sharing tasks)
The non-label-sharing (NLS) scenario assumes that different semantic meanings of labels across tasks.
For example, the piano class in the miniImagenet dataset may correspond to a class label of 0 for one task and 1 for another task.
MLTI for label-sharing tasks .
First, we will discuss MLTI under the label-sharing scenario, where it applies the same interpolation strategy on both features/hidden representations and label spaces.
Concretely, let’s say that a model f consists of L layers and the hidden representation of samples
X at the l -th layer is denoted as H l = f θ l ( X ) ( 0 ≤ l ≤ L s ), where H
= X
and L s represents the number of layers shared across all tasks.
In gradient-based methods, as suggested in (Yin et al., 2020), only part of the layers are shared (i.e., L s < L ).
In metric-based methods, all layers are shared (i.e.,
L s = L ).
Given a pair of tasks with their sampled support and query sets (i.e., T i =
{D si , D qi } and
T j = {D sj , D qj } ) under the same label space, MLTI first randomly selects one layer l and then applies the task interpolation separately on the hidden representations ( H s ( q ) ,l i , H s ( q ) ,l j ) and corresponding labels ( Y s ( q ) i , Y s ( q ) j ) of the support (query) sets as:
[Equation]
where λ ∈ [0 , 1] is sampled from a Beta distribution Beta( α, β ) and the subscript "cr" represents "cross".
Notice that both the support and query sets will be replaced by the interpolated ones in
MLTI, while only the query set is replaced in approaches like Yao et al. (2021).
Besides, Manifold
Mixup (Verma et al., 2019) in Eqn.
(5) can be replaced by different task interpolation methods (e.g.,
Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019)).
MLTI for non-label-sharing tasks.
Under non-label-sharing scenarios, tasks have different label spaces, making it infeasible to directly interpolate the labels.
Instead, we generate the new task by performing the feature-level interpolation and re-assign a new label to the interpolated class.
Specifically, given samples from class r in task T i and class r ′ in task T j , we denote the interpolated features as Intrpl( r , r ′ ), which are formally defined as:
[Equation]
The interpolated samples are regarded as a new class in the interpolated task.
After randomly selecting N class pairs, we can construct an N -way interpolated task.
Take a 3-way classification as an example, assume task T i has classes ( i 1 , i 2 , i 3 ) and task T j has classes ( j 1 , j 2 , j 3 ).
One potential interpolated task could be a 3-way task with classes ( e 1 , e 2 , e 3 ), where the labels are associated with interpolated features (Intrpl( i 1 , j 2 ), Intrpl( i 2 , j 3 ), Intrpl( i 3 , j 1 )).
Note that, for ProtoNet and its variants, we apply the interpolation strategies of Eqn.
(6) on both LS and NLS scenarios since it is intractable to calculate prototypes with mixed labels.
Finally, we note that MLTI supports both inter-task and intra-task interpolation, as we allow the case when i = j .
As we will find in Sec. 6, intra-task interpolation can be complementary to cross-task interpolation and further improve the generalization.
Under this case, the intra-task interpolation can also be replaced by any existing intra-task augmentation strategies (e.g., MetaMix (Yao et al., 2021)).
After generating the interpolated support set D si,cr =
( ˜ H s,li,cr ,
˜ Y si,cr ) and query set D qi,cr =
( ˜ H q,li,cr , ˜ Y qi,cr ) , we replace the original support and query sets with the interpolated ones.
With MAML as an example, we reformulate the optimization process in Eqn.
(4) as:
[Equation]
where the superscript L − l represents the rest of layers after the selected layer l .
Detailed pseudocode of MAML and ProtoNet is shown in Alg.
1 and Alg. 2 in Appendix A, respectively.
T HEORETICAL A NALYSIS
We now theoretically investigate how MLTI improves the generalization performance with both gradient-based and the metric-based meta-learning methods.
Specifically, we theoretically prove that
MLTI essentially induces a data-dependent regularizer on both categories of meta-learning methods and controls the Rademacher complexity, leading to greater generalization.
Here, we only discuss the non-label-sharing (NLS) scenario (see detailed proof in Appendix B.1) and leave the analysis of the label-sharing scenario in Appendix B.2.
4. G RADIENT -B ASED M ETA -L EARNING WITH MLTI
In gradient-based meta-learning, we analyze the generalization ability by considering the two-layer neural network with binary classification.
For the simplicity of presentation, we assume the sample size of different task are the same and equal to N .
Suppose there are | I | tasks.
For each task T i , we consider the logistic loss ℓ ( f MAML ( x ) , y )
= log(1 + exp( f MAML ( x )))
− yf MAML ( x ) with f MAML modeled by f MAMLϕ i ( x i,k ) =
ϕ
⊤ i σ ( Wx i,k ) :
=
ϕ
⊤ i h
i,k , where
h
i,k represents the hidden representation on the first layer of sample x
i,k .
Under the NLS setting, the interpolated task is constructed by Eqn.
We assume the interpolation performs on the hidden layer (following Eqn.
(6) with l = 1 ) and denote the interpolated query set as D qi,cr =
( ˜ H q, 1 i,cr , ˜ Y qi,cr ) .
For simplicity, in this subsection, we omit the superscript q and define the empirical training loss as L t ( {D i,cr } | I | i =1 ) =
| I | − 1 (cid:80) | I | i =
1 L ( D i,cr ) =
( N | I | ) − 1 (cid:80) | I | i =1 (cid:80)
Nk
=1 L ( f ϕ i ( x i,k,cr ) , y i,k,cr ) .
We first present a lemma showing that the loss L t ( {D i,cr } | I | i =1 ) induced by MLTI has a regularization effect.
Lemma 1.
Consider the MLTI with λ ∼ Beta( α, β ) .
Let ψ ( u ) =
e u / (1 + e u )
and N i,r denotes the number of samples from the class r in task T i .
There exists a constant c > 0 , such that the second-order approximation of L t ( {D i,cr } | I | i =1 ) is given by
[Equation]
⊤ i
[Equation]
k
[Equation]
i )
[Equation]
ϕ ⊤ i ( 1 |
[Equation]
| | I |
[Equation]
i =   2 (cid:88) r =
N i,r | I | (cid:88) i =
N i,r (cid:88) k
= h 1 i,k ; r h 1 ⊤ i,k ; r ) ϕ
i , (8) where ¯ λ = E D λ
[ λ ] , with D λ ∼ αα + β Beta( α + 1 , β ) + βα + β Beta( β + 1 , α ) .
This lemma suggests that MLTI induces an (implicit) regularization term on ϕ
i ’s through task interpolation and therefore will lead to a better generalization bound, as we will show in Section 6.
with extensive numerical experiments.
To study the improved generalization more explicitly, we consider the population version of the regularization term in Eqn.
(8) by considering the function class F γ =
{ H 1 ⊤ ϕ : E [ ψ ( H 1 ⊤ ϕ )]
ϕ ⊤ Σ ϕ ≤ γ } , where Σ = E T ∼ p ( T ) E T
[ H 1 H 1 ⊤ ] .
We also define µ T =
E T [ H 1 ] and assume the following condition of the individual task distribution T as: for all
T ∼ p ( T ) , T satisfies
[Equation]
where Σ † denotes the generalized inverse of Σ .
Further, we assume that the distribution of His ρ -retentive for some ρ ∈ (0 , 1 / 2] , that is, if for any non-zero vector v ∈ R d , (cid:2) E [ ψ ( v ⊤ H 1 )] (cid:3) 2 ≥ ρ · min { 1 , E ( v ⊤ H 1 ) 2 } .
Such an assumption has been similarly assumed in (Arora et al., 2020; Zhang et al., 2021) and is satisfied when the weights has bounded ℓ 2 norm.
We also regard L t ( {D i } | I | i =1 ) of tasks {T i } | I | i =1 as the empirical (training) loss R ( {D i } |
I | i =1 ) and its corresponding population loss (on the test data) is defined as R =
[Equation]
, Y i
[Equation]
We then have the following theorem showing the improved generalization gap brought by MLTI.
Theorem 1.
Suppose X i ’s , Y ′
i s and ϕ are bounded in spectral norm and assumption (9) holds.
There exist constants A 1 , A 2 , A 3 > 0 , such that for all f T ∈ F γ , δ ∈ (0 , 1) , with probability at least 1 − δ (over randomness of training sample), we have the following generalization bound
[Equation]
Based on Lemma 1 and Theorem 1, MLTI regularizes on ϕ ⊤ Σ ϕ (implying a small γ ) and therefore achieves a tighter generalization bound than the vanilla gradient-based method (where γ is unconstrained).
Compared with the individual task augmentation (see Figure 1(b)), the regularization effect in Eqn.
(8) induced by MLTI is larger (i.e., smaller γ ) since the total variance is generally larger than the within-group variance (see more details in the Appendix B.3).
Therefore, MLTI reduces the generalization error, which we also empirically validate in the experiments.
4. M ETRIC -B ASED M ETA -L EARNING WITH MLTI
In the metric-based meta-learning, we consider the ProtoNet with linear representation in the binary classification, which has been commonly considered in other theoretical analysis of meta-learning, see, e.g., (Du et al., 2020; Tripuraneni et al., 2020).
Specifically, we assume f PNθ ( x ) =
θ ⊤ x and d ( · , · ) represents the squared Euclidean distance, then the loss of ProtoNet can be simplified as
[Equation]
where c 1 and c 2 are defined as the prototypes of class 1 and 2, respectively.
Under this setting, the interpolation performs on the feature (i.e., l = 0 in Eqn.
We now present the following lemma showing that MLTI induces a regularization on the parameter θ .
Lemma 2.
Considering the interpolated tasks {D i,cr } | I | i =1 with λ ∼ Beta( α, β ) , we define L t ( {D i } | I | i =1 ) =
( N | I | ) − 1 (cid:80) i,k (1 + exp( ⟨ ( x i,k − ( c 1 + c 2 ) / 2 , θ ⟩ ))
− 1 and L t ( {D i,cr } | I | i =1 )
=
( N | I | ) − 1 (cid:80) i,k (1 + exp( ⟨ ( x i,k,cr − ( c 1 ,cr + c 2 ,cr ) / 2 , θ ⟩ ))
− 1 .
Recall ψ ( u ) =
e u / (1 + e u ) 2 .
The secondorder approximation of L t ( {D i,cr } | I | i =1 ) is given by, for some constant c > 0 ,
L
[Equation]
|
[Equation]
=   2 (cid:88) r =
N r | I | (cid:88) i =
N r (cid:88) k
= x
i
[Equation]
r x ⊤ i
[Equation]
k ; r ) θ.
Similar to the last section, we assume that the distribution of x is ρ -retentive for some ρ ∈ (0 , 1 / 2] , and investigate the following function class: let Σ X =
E
[ xx ⊤ ] ,
[Equation] [Equation]
→
[Equation]
x , such that θ
[Equation]
We then have the following theorem on the explicit generalization bound of ProtoNet.
Theorem 2.
Suppose X i ’s, Y i ’s and θ are both bounded in spectral norm, and the distribution of x is ρ -retentive and mean zero.
Let r Σ = rank (Σ X ) , then there exist constants B 1 , B 2 , B 3 > 0 , for any f ∈ W γ , δ ∈ (0 , 1) , with probability at least 1 − δ (over the training sample), such that
[Equation]
By Theorem 2, adding MLTI into ProtoNet would induce a small value of γ and thus improve the generalization compared to the vanilla ProtoNet.
Similarly, MLTI achieves tighter generalization bound than the individual task augmentation with a larger regularization term (i.e., smaller γ ).
R ELATED W ORK
The goal of meta-learning is to enable few-shot generalization of machine learning algorithms by transferring the knowledge acquired from related tasks.
One approach is gradient-based metalearning (Finn and Levine, 2018; Finn et al., 2017; 2018;
Grant et al., 2018; Flennerhag et al., 2020; Lee and Choi, 2018; Li et al., 2017;
Oh et al., 2021; Nichol and Schulman, 2018; Rajeswaran et al., 2019; Rusu et al., 2018), where the meta-knowledge is formulated to be optimization-related parameters (e.g., model initial parameters, learning rate, pre-conditioning matrix).
During the metatraining stage, the model is first adapted to each task via a truncated optimization and then the optimization-related parameters are optimized by maximizing the generalization performance of the model.
Another line of research is metric-based meta-learning (Cao et al., 2021; Garcia and
Bruna, 2018; Liu et al., 2019; Mishra et al., 2018; Snell et al., 2017; Vinyals et al., 2016; Sung et al., 2018; Yoon et al., 2019), which meta-learns an embedding space and uses a non-parametric learner to classify samples.
Unlike prior works that propose new meta-learning algorithms, this work aims to improve the task-level generalization of these algorithms and reduce the negative effect of memorization, especially when the number of meta-training tasks is limited.
To mitigate the influence of memorization and improve the generalization, one line of research focuses on directly imposing regularization on meta-learning algorithms (Guiroy et al., 2019; Jamal and Qi, 2019; Tseng et al., 2020;
Yin et al., 2020).
Another line of research reduces the number of adapted parameters for gradient-based meta-learning (Raghu et al., 2020; Zintgraf et al., 2019).
Instead of imposing regularization strategies (i.e., objectives, dropout, less adapted parameters), our approach focuses on augmenting the set of tasks for meta-training.
Prior works have proposed domain-specific techniques to generate more data by augmenting images (Chen et al., 2019) or by reconstructing tasks with latent reasoning categories for NLP-related tasks (Murty et al., 2021).
Recent domain-agnostic techniques have augmented tasks by imposing label noise (Rajendran et al., 2020) or applying Mixup (Zhang et al., 2018) and its variants (e.g., Manifold Mixup (Verma et al., 2019)) to each task (Ni et al., 2021; Yao et al., 2021).
Unlike these domain-agnostic augmentation strategies that applying data augmentation on each task individually (Figure 1(b)), we directly densify the task distribution by generating additional tasks from pairs of existing tasks (Figure 1(c)).
More discussions with individual task augmentation are provided in Appendix C. Empirically, we find that
MLTI outperforms all of these above strategies in Section 6.
E XPERIMENTS
In this section, we conduct experiments to test and understand the effectiveness of MLTI.
Specifically, we aim to answer the following research questions under both label-sharing and non-label-sharing settings: Q1 :
Compared with prior methods for regularizing meta-learning, how does the MLTI perform?
Q2 : Is MLTI compatible with different backbone meta-learning algorithms and does it improve their performance?
Q3 :
How does MLTI perform compared with only applying intra- or cross-task interpolation?
Q4 : How does the number of tasks affect the performance of MLTI?
[Table] [Table]
We compare MLTI with the following two representative domain-agnostic strategies: (1) directly imposing regularization into the meta-learning framework, including Meta-Reg (Yin et al., 2020),
TAML (Jamal and Qi, 2019)
[Table]
and Meta-dropout (Lee et al., 2020); and (2) individual task augmentation methods, including Meta-Augmentation (Rajendran et al., 2020), MetaMix (Yao et al., 2021), and Meta-Maxup (Ni et al., 2021).
We select MAML and ProtoNet as backbone methods and apply the corresponding meta-learning strategies to them according to their applicable scopes.
Note that we also extend MetaMix and Meta-Maxup to ProtoNet, even though the methods only focus on gradient-based meta-learning in the original papers.
To further test the compatibility of MLTI, we additionally apply MLTI to other meta-learning backbone algorithms, including MetaSGD (Li et al., 2017), ANIL (Raghu et al., 2020), Meta-Curvature (MC) (Park and Oliva, 2019), and MatchingNet (Vinyals et al., 2016).
To provide a fair comparison, all methods use the same architecture of the base model as MLTI and all interpolation-based methods use the same interpolation strategies (see Appendix D.1 and E.1 for details).
6. L ABEL - SHARING S CENARIO
Datasets and experimental setup.
Under the label-sharing scenario, we perform experiments on four datasets to evaluate the performance of MLTI:
(1) PASCAL3D Pose regression (Pose) (Yin et al., 2020): it aims to predict the object pose of a grey-scale image relative to the canonical orientation.
Following Yin et al.
, we select 50 objects for meta-training and 15 objects for meta-testing; (2)
RainbowMNIST (RMNIST) (Finn et al., 2019): it is a 10-way classification dataset wherein each task is constructed by applying a combination of image transformation operators on the original MNIST dataset (e.g., scaling, coloring, rotation).
We here use 14 and 10 combinations for meta-training and meta-testing, respectively.
(3)&(4) NCI (NCI, 2018) and TDC Metabolism (Metabolism) (Huang et al., 2021):
both are 2-way chemical classification datasets, which aim to predict the property of a set of chemical compounds.
We use six data sources for meta-training, and the remaining three sources for meta-testing.
The number of shots for the above four datasets are set as 15, 1, 5, and 5,
[Table]
More details on the datasets and set-up are provided in Appendix D.1.
We adopt MSE to measure the performance for the Pose regression dataset and accuracy for the classification datasets.
Results.
Under the label-sharing scenario, we report the overall performance and analyze the compatibility of MLTI in Table 1 and Appendix D.2, respectively.
According to Table 1, we observe that MLTI outperforms other regularization strategies across the board, including passively adding regularization (i.e., Meta-Reg, TAML, Meta-Dropout) and augmenting tasks individually (i.e., Meta-Aug, MetaMix, Meta-Maxup).
These results indicate that MLTI consistently improves generalization through interpolation on the task distribution.
The claim is further be strengthened by the compatibility analysis (Appendix D.2), where MLTI boosts the performance of a variety of meta-learning algorithms.
We also investigate the effect of the number of meta-training tasks and report the performance in Appendix D.3.
We observe that the improvements from MLTI are robust under different settings but that the greatest improvements come when the number of tasks is limited.
Ablation study.
In Table 2, we conduct an ablation study under the label-sharing scenario.
Here, we investigate how MLTI performs compared with only applying intra-task interpolation (i.e., T i = T j ) and cross-task interpolation (i.e., T i ̸
= T j ), which are denoted as Intra-Intrpl and Cross-Intrpl, respectively.
We observe that both Intra-Intrpl and Cross-Intrpl outperform the vanilla approach without task augmentation and that MLTI achieves the best performance, indicating that the strategies are complementary to some degree.
In addition, cross-interpolation outperforms the intra-interpolation in most datasets.
The results corroborate the effectiveness of cross-task interpolation when tasks are sparsely sampled from the data distribution.
6. N ON - LABEL - SHARING S CENARIO
Datasets and experimental setup.
Under the non-label-sharing scenario, we conduct experiments on four datasets: (1) general image classification on miniImagenet (Vinyals et al., 2016); (2)&(3) medical image classification on ISIC (Milton, 2019) and DermNet (Der, 2016); and (4) cell type classification across organs on Tabular Murris (Cao et al., 2021).
Since a task in meta-learning is defined to correspond to a particular data-generating distribution (Finn et al., 2017; Rajeswaran et al., 2019), the number of distinct meta-training tasks in N -way classification is actually the number of ways to choose N from all base classes.
Thus, for miniImagenet and Dermnet, we reduce the number tasks by limiting the number of meta-training classes (a.k.a., base classes) and obtain the miniImagenet-S , ISIC , DermNet-S , Tabular Murris benchmarks, whose base classes are 12, 4, 30, 57, respectively (see the experiments on full-size miniImagenet and DermNet in Appendix E.2).
The experiments are performed under the N -way K -shot setting (Finn and Levine, 2018), where
N = 2 for ISIC and N = 5 for the rest datasets.
Note that, Meta-Aug (Rajendran et al., 2020) under the non-label-sharing scenario is exactly the same as the label shuffling, which is already adopted in vanilla MAML and ProtoNet.
Due to space limitations, we report only the accuracy for the non-label-sharing scenario here and provide the full table with 95% confidence intervals in
Appendix E.9.
More details about the datasets and set-up are in Appendix E.1.
[Table]
Results.
Table 3 gives the results of MLTI and prior methods.
MLTI consistently outperforms other strategies.
The performance gains suggest that MLTI can improve the generalization ability of metalearning amidst sparsely sampled task distributions.
We also analyze of compatibility of MLTI under the non-label-sharing scenario in Table 9 of Appendix E.3.
The results validate that MLTI can robustly boost performance with different backbone methods.
For the ablation study, we repeat the experiments on the non-label-sharing scenario and report the results in Table 10 of Appendix E.4.
We additionally provide base model and hyperparameter analysis in Appendix E.5.
MLTI achieves the best performance across various settings, further strengthen its effectiveness.
Cross-domain adaptation.
To further evaluate performance of MLTI, we conduct a comparison under the cross-domain adaptation setting where we meta-train the model on one source domain and evaluate it on another target domain.
We perform cross-domain adaptation across miniImagenet-S and Dermnet-S and report the performance under MAML and ProtoNet in Table 4.
The results validate that MLTI can improve generalization even in this more challenging setting.
[Figure]
Effect of the number of meta-training tasks.
We analyze the effect of the number of tasks under 5-shot setting (with a ProtoNet backbone) in Figures 2a and 2b (see more results in Appendix E.6).
We have two key observations: (1)
MLTI consistently improves the performance for all numbers of tasks, showing its effectiveness and robustness; (2) The improvement gap between MLTI and the vanilla model decreases as the the number of tasks increases on miniImagenet, and keeps consistent on DermNet.
We expect this is because the meta-training tasks may be more related to meta-testing tasks in miniImagenet, than in DermNet.
Besides, we conduct an additional experiments in Appendix E.7 to show the promise of MLTI when we only have extremely limited tasks.
[Figure]
Analysis of Interpolated Tasks.
Building upon ProtoNet, we show the t-SNE (Maaten and Hinton, 2008) visualization of both original tasks and interpolated tasks in Figure 3.
Specifically, we randomly select 3 original tasks and 300 interpolated tasks under the 1-shot miniImagenet-S setting, where the color of each interpolated task indicates its proximity to the corresponding original tasks.
Each task is represented by the averaged representation over its corresponding prototypes, where we combine both support and query sets to calculate the prototypes.
The figure suggests that the interpolated tasks generated by MLTI indeed densify the task distribution and bridge the gap between different tasks.
C ONCLUSION
In this paper, we investigate the problem of meta-learning with fewer tasks and propose a new task interpolation strategy MLTI.
The proposed MLTI targets the task distribution directly to generate more meta-training tasks via task interpolation for both label-sharing and non-label-sharing scenarios.
The consistent performance gains across eight datasets demonstrate that MLTI improves the generalization of meta-learning algorithms especially when the number of available meta-training tasks is small, which is further supported by the theoretical analysis.
R EPRODUCIBILITY S TATEMENT
For our theoretical results, a complete proof of all claims and the discussion of assumptions are provided in Appendix B.
For our empirical results, we discuss the details of datasets and list all hyperparameters under the label-sharing scenario and non-label-sharing scenario in Appendix D. and E.1, respectively.
We will open-source the code upon publication.
A P SEUDOCODES
In this section, we show the pseudocodes for MLTI with MAML (meta-training process: Alg. 1, meta-testing process: Alg. 2) and ProtoNet (meta-training process: Alg. 3, meta-testing process:
Alg. 4).
Require: p ( T ) : task distribution; η , γ : inner- and outer-loop learning rate; L s : the number of shared layers; Beta distribution 1: Randomly initialize the model initial parameters θ 2: while not converge do 3:
Randomly sample a batch of tasks {T i } | I | i
=1 with dataset 4: for each task T i do 5:
Sample a support set D si = ( X si , Y si ) and a query set D qi =
( X qi , Y qi ) from D i 6:
Sample another task T j (allow i = j ) from {T i } | I | i =1 with corresponding support set D sj =
( X sj , Y sj ) and query set D qj =
( X qj , Y qj ) 7:
Random sample one layer l from the shared layers 8:
Obtain the hidden representations H s,l i , H q,l i , H s,l j , H q,l j of the support/query sets of task T i and T j 9:
Apply task interpolation between task T i and T j via Eqn.
(5) (label-sharing tasks) or Eqn.
(6) (non-label-sharing tasks), and obtain the interpolated support set D si,cr =
( ˜ H s,li,cr ,
˜ Y si,cr ) and query set D qi,cr =
( ˜ H q,li,cr , ˜ Y qi,cr ) 10:
Calculate the task-specific parameters ϕ
L − l i,cr by the inner-loop adaptation, i.e., ϕ L − l i,cr = θ L − l − η ∇ θ L − l L ( f MAMLθ L − l ; D si,cr ) 11: end for 12:
Optimize the model initial parameters as θ ← θ − γ 1 | I | (cid:80) | I | i =1 L ( f MAMLϕ L − l i,cr ; D qi,cr ) 13: end while
Require: p ( T ) : task distribution; η : inner-loop learning rate; θ ∗ : learned model initial parameters 1: Randomly initialize the model initial parameters θ 2: for each task T t with support set D st and query set D qt do 3:
Calculate the task-specific parameters ϕ
i by the inner-loop adaptation, i.e., ϕ i
= θ ∗ − η ∇ θ ∗ L ( f MAMLθ ∗ ;
D si ) 4:
Obtain the predicted labels of the query set by f MAMLϕ i
( D qi ) and evaluate the performance 5: end for
B A DDITIONAL T HEORETICAL A NALYSIS B. P ROOFS OF N ON - LABEL - SHARING S CENARIO B.1. P ROOF OF L EMMA
Proof.
Recall that the interpolated dataset is D qi,cr =
( ˜ H q, 1 i,cr , ˜ Y qi,cr ) :
= { ( h 1 i,k,cr , y i,k,cr ) }
Nk =1 , where h 1 i,k,cr ; r = λ
h 1 i,k ; r + (1 − λ ) h 1 j,k ′ ; r ′ , y i,k,cr = Lb ( r, r ′ ) .
Here, r = y i,k , λ ∼ Beta( α, β ) , j ∼ U ([ | I | ]) , r ∼ U ([ R i ]) , where R i represents the number of classes in task T i , and Lb ( r, r ′ ) denotes the label uniquely determined by the pair ( r, r ′ ) .
The superscript q is also omitted in the whole section.
Since for a give set of r ′ , r and ( r, r ′ ) has a one-to-one correspondence, without loss of generality, we assume r = ( r, r ′ ) in this classification setting.
Recall that L t ( {D i,cr } | I | i =1 ) =
| I | (cid:80) | I | i =
1 L ( D i,cr ) =
| I | (cid:80) | I | i =1 1 N (cid:80)
Nk
=1 L ( f ϕ i ( x i,k,cr ) , y i,k,cr ) =
| I | (cid:80) ni
=1 1 N (cid:80) Nk
=1 L ( h 1 i,k,cr , y i,k,cr ) .
Then let us compute the second-order Taylor expansion on
L t ( {D i,cr } | I | i =1 ) = 1 | I | (cid:80) | I | i =1 1 N (cid:80) Nk
=1 L ( h 1 i,k,cr , y i,k,cr ) with respect to the first argument around
Require: distribution 1: Randomly initialize the initial parameters θ 2: while not converge do 3:
Randomly sample a batch of tasks {T i } | I | i
=1 with dataset 4: for each task T i do 5:
Sample a support set D si = ( X si , Y si ) and a query set D qi =
( X qi , Y qi ) from D i 6:
Sample another task T j (allow i = j ) from {T i } | I | i =1 with corresponding support set D sj =
( X sj , Y sj ) and query set D qj =
( X qj , Y qj ) 7:
Random sample one the shared layers 8:
Obtain the hidden representations H s,li , H q,li , H s,lj , H q,lj of the support/query sets of task T i and T j 9:
Apply task interpolation between task T i and T j , and obtain the interpolated support set
D si,cr =
( ˜ H s,li,cr ,
˜ Y si,cr ) and query set D qi,cr =
( ˜ H q,li,cr , ˜ Y qi,cr ) 10:
Calculate the prototypes { c r } Rr =1 ( N r represents the number of samples in class r ) by c r = 1 N r (cid:80) (
[Equation]
si
[Equation]
cr
[Equation]
( h si,
[Equation]
,cr ; r ) 11:
Calculate the loss of task T i as L i = − (cid:80)
k log
[Equation]
[Equation] [Equation]
r ′ exp(
[Equation] [Equation]
l (
[Equation]
qi,k,cr )
[Equation]
r ′ ))
12: end for 13:
Update θ ← θ − γ
[Table]
| I | (cid:80) | I | i =
1 L i 14: end while
Require: p ( T ) : task distribution; θ ∗ : learned parameter of the base model 1: for each task T t with support set D st and query set D qt do 2:
Calculate the prototypes { c r } Rr =1 ( N r represents the number of samples in class r ) by c r =
N r (cid:80) ( h si,k ; r , y si,k ; r ) ∈D si ; r f PNθ ( h si,k
[Equation]
r ) 3:
Calculate the probability of each sample being assigned to class r as p ( y qi
[Equation]
k = r | x qi
[Equation]
k
[Equation]
=
[Equation]
r
[Equation]
Obtain the predicted class as ˆ y qi,k = arg max r p ( y qi,k = r | x qi,k ) and evaluate the performance 5: end for 1¯ λ E [ h 1 i,k,cr |
h 1 i,k ] =
h 1 i,k,cr , we have the Taylor expansion of L t ( {D i,cr } | I | i =1 ) up to the second-order equals to | I | | I | (cid:88)
[Equation]
N
[Equation]
(13) =|
I | |
[Equation]
|
[Equation]
|
[Equation]
| (cid:88)
i
[Equation]
N
N (cid:88) k
=
[Equation] [Equation]
k
[Equation]
r
[Equation] [Equation]
[Equation]
| | I | (cid:88) i =
2 (cid:88) r =
N i,r
N i,r (cid:88) k
= h
i,k ; r h 1 ⊤ i,k ; r ) ϕ
i (14) =
L t (¯ λ {D i } | I | i =1 ) + c 1 N | I | | I | (cid:88) i =
N (cid:88) k
= ψ ( h 1 ⊤ i,k ; r ϕ i ) · ϕ ⊤ i ( 1 | I | | I | (cid:88) i =   2 (cid:88) r =
N i,r | I | (cid:88) i =
N i,r (cid:88) k
= h 1 i,k ; r h 1 ⊤ i,k ; r ) ϕ
i , where c = E
[ (1 − λ ) 2 λ 2 ] and the second equality (14) uses the fact that the data is pre-processed so that
[Equation]
B.1. P ROOF OF T HEOREM
We first state a standard uniform deviation bound based on Rademacher complexity (c.f. (Bartlett and
Mendelson, 2002)).
Lemma 3.
Assume { z 1 , ..., z N } are drawn i.i.d.
from a distribution P over Z , and G denotes function class on Z with members mapping from Z to [ a, b ] .
With probability at least 1 − δ over the draw of the sample and δ > 0 , we have the following bound:
[Equation]
where R ( G ; z 1 , ..., z N ) represents the Rademacher complexity of the function class G .
Proof.
We now formulate R ( {D i } | I | i =1 ) − R as
[Equation]
Recall that we consider the function f MAMLϕ
i
[Equation]
i H 1 i and the function class
[Equation]
For each T i , let us consider f ϕ i ( · ) ∈ F γ .
Combining Theorem 3.4 and Theorem A.1 in Zhang et al.
(2021), we have the following result for the Rademacher complexity:
[Equation]
Then, we bound the first term (i) in Eqn.
(15) can be as below.
[Equation]
where C 1 and C 2 are constants, and the additional log( | I | ) term in the last inequality above is caused by taking the union bound on | I | tasks.
Denote function g : T → R such that g ( T ) =
E ( X , Y ) ∼D
( L ( f ϕ ( X ) , Y )) .
Denote
G = { g
[Equation] [Equation]
) ∼D ( L ( f ϕ ( X ) , Y )) , f ϕ ∈ F γ } .
Let A ( x ) = 1 / (1 + e x ) .
The second term (ii) in Eqn.
(15) requires computing the Rademacher complexity for the function class over distributions
[Equation]
Then we have the following bound on (ii):
[Equation]
Combining the pieces, we obtain the desired result.
With probability at least 1 − δ ,
[Equation]
B.1. P ROOF OF L EMMA
Recall that we apply MLTI in the feature space for theoretical analysis, the interpolated dataset is then denoted as D qi,cr =
( ˜ X qi,cr , ˜ Y qi,cr ) :
= { ( x i,k,cr , y i,k,cr ) }
Nk =1 , where x i,k,cr ; r = λ x i,k ; r + (1 − λ )
x j,k ′ ; r ′ , y i,k,cr =
Lb ( r, r ′ ) .
where r = y i,k , λ ∼ Beta( α, β ) , j ∼ U ([ | I | ]) , r ∼ U ([2]) , and Lb ( r, r ′ ) denotes the label uniquely determined by the pair ( r, r ′ ) .
Since for a give set of r ′ , r and ( r, r ′ ) has a one-to-one correspondence, without loss of generality, we assume r = ( r, r ′ ) in this classification setting.
Proof.
To prove Lemma 2, first, we would like to note that since the overall sample mean 1 | I | (cid:80) | I | i
=1 12 (cid:80) 2 r =1 1 N i,r (cid:80) N i,r k =1 x i,k ; r = 0 , we then have
E [ x i,k,cr ; r | x i,k ; r ] = x i,k ; r .
Then let us compute the second-order
Taylor expansion on
L t ( {D i,cr } | I | i =1 ) =|
I | (cid:80) | I | i =1 1 N (cid:80) Nk
[Equation] [Equation]
[Equation] [Equation]
cr
[Equation]
i
[Equation]
cr )
[Equation]
−
[Equation]
with respect to the first argument around 1¯ λ E [ x i,k,cr | x i,k ] = x i,k,cr , we have that the Taylor expansion of L t ( {D i,cr } | I | i =1 ) up to the second-order equals to
[Equation]
i =
[Equation]
r =
N r
N r (cid:88)
[Equation] [Equation]
r
[Equation] [Equation]
r
[Equation]
θ where c = E [ (1 − λ ) 2 λ 2 ] .
B.1. P ROOF OF T HEOREM
Similar to the proof of Theorem 1, we use Lemma 3 in the proof of Theorem 2.
Proof.
We first write R ( {D i } | I | i =1 ) − R as
[Equation]
} | I | i
[Equation]
R
[Equation]
i
[Equation]
Recall that we consider the function f θ ( x ) =
θ ⊤ x and the function class
W γ :=
{ x → θ ⊤ x , such that θ satisfying E x [ ψ ( ⟨ x − ( c 1 + c 2 ) / 2 , θ ⟩ )] · θ ⊤ Σ X θ ≤ γ } , (18)
For each T i , let us consider f θ ( · ) ∈ W γ .
Combining Theorem 3.4 and Theorem A.1 in Zhang et al.
(2021), we have the following result for the Rademacher complexity:
[Equation]
Then the first term (i) in Eqn.
(17) can be bounded as below.
[Equation]
where C 1 and C 2 are constants, and the additional log( | I | ) term in the last inequality above since we take union bound on | I | tasks.
Denote function g : T → R such that g ( T ) =
E ( X , Y ) ∼D
( L ( f θ ( X ) , Y )) .
Denote
[Equation]
Recall that A ( x ) = 1 / (1+ e x ) .
The second term (ii) in Eqn.
(17) requires computing the Rademacher complexity for the function class over distributions
[Equation]
Then we have the following bound on (ii) in Eqn.
[Equation]
Combining the above pieces, we obtain the desired result.
With probability at least 1 − δ ,
[Equation]
T HEORETICAL R ESULTS UNDER THE L ABEL - SHARING S CENARIO
As discussed in Line 131-133 of the main paper, for protonet, it is impractical to calculate the prototypes with mixed labels.
Thus, under the label-sharing scenario, we only analyze the generalization ability of gradient-based meta-learning.
Follow the assumptions under the non-label-sharing scenario, we first present the counterpart of Lemma 1 of the main paper.
Lemma 4.
Consider the MLTI with λ ∼ Beta( α, β ) .
Let ψ ( u ) =
e u / (1 + e u )
and N i,r denote the number of samples from the class r in task T i .
There exists a constant c > 0 , such that the second-order approximation of L t ( {D i,cr } | I | i =1 ) is given by
[Equation]
Proof.
Under the label-sharing scenario, the interpolated dataset D qi,cr =
( ˜
[Equation]
1 i,cr , ˜ Y
qi,cr ) :
= { ( h 1 i,k,cr , y i,k,cr ) }
Nk =1 is constructed as
[Equation]
where λ ∼ Beta( α, β ) , j ∼ U ([ | I | ]) .
By Lemma 3.1 in Zhang et al.
(2021) (with proof on page 13), the data augmentation equals in distribution with the following augmentation
[Equation]
with λ ∼ αα + β Beta( α + 1 , β
[Equation]
+ αα + β Beta
[Equation]
α + 1 , β
[Equation]
, j ∼ U ([ | I | ]) .
Then we apply the same proof technique as the proof of Lemma 1 and obtain that the Taylor expansion of L t ( {D i,cr } | I | i =1 ) up to the second-order equals to
[Equation]
c 1 N | I | | I | (cid:88) i =
N (cid:88) k
= ψ ( h 1 ⊤ i,k ϕ i ) · ϕ ⊤ i ( 1 | I | | I | (cid:88) i =
N | I | | I | (cid:88) i =
N (cid:88) k
= h 1 i,k h 1 ⊤ i,k ) ϕ
i , where c = E D λ
[ (1 − λ ) 2 λ 2 ] and D λ = αα + β Beta
( α + 1 , β ) + αα + β Beta( α + 1 , β ) .
Given Lemma 4, the population version of the regularization term can be defined in the same form of
Eq.
and therefore the generalization theorem and its corresponding conclusions are the same as
Theorem 1 and conclusions in the main paper.
Besides, in this work, the regression setting is only well-defined under the label-sharing scenario.
The theoretical analysis under the label-sharing scenario (i.e., Lemma 4) in Section B.2 are not specific to the classification setting and still hold in the regression setting.
B. D ISCUSSION ABOUT THE VARIANCE OF MLTI
From the above analysis, we can see that the second order of regularization depends on Cov ( h 1 i,k,cr |
h 1 i,k ) in Eqn.
(1) (gradient-based meta-learning) or Cov ( x i,k,cr | x i,k ) in Eqn.
(14) (metric-based meta-learning).
Let G denote the random variable which takes a uniform distribution on the indices of the tasks.
By using the law of total variance, we have Cov ( h 1 i,k,cr |
h 1 i,k ) =
E [ Cov ( h 1 i,k,cr |
G, h 1 i,k )]
+ Cov ( E [ h 1 i,k,cr | G, h 1 i,k ])
≥ E [ Cov ( h 1 i,k,cr | G, h
1 i,k )] , where the later is the covariance matrix induced by the individual task interpolation, i.e., i = j in the interpolation process.
C A DDITIONAL D ISCUSSIONS BETWEEN MLTI AND I NDIVIDUAL T ASK A UGMENTATION
As shown in Figure 1, MLTI directly densifies task distributions by generating more tasks rather than apply augmentation strategies to each individual tasks.
Compared with individual task augmentation (e.g., (Yao et al., 2021; Ni et al., 2021)), the reasons why MLTI leads to more dense task distributions are summarized under both label-sharing and non-label-sharing settings.
• Label-sharing Setting.
MLTI densifies the task distribution by enabling cross-task interpolation.
For example, in Pose prediction, we not only interpolate samples within each object, but cross-task interpolation significantly increases the number of tasks.
Assume we have two objects (O1 and
O2), individual task interpolation approaches (e.g., Meta-Maxup) only generate more samples in
O1 or O2, where only one object information is covered.
However, MLTI further allows generating tasks with both O1 and O2 information by interpolating
data samples from O1 and O2.
• Non-label-sharing Setting.
MLTI also leads to more dense task distribution under the non-labelsharing setting.
For example, in 2-way classification with 3 training classes (C0, C1, C2), there are three original tasks,
i
e., three classification pairs (C0, C1), (C0, C2), (C1, C2).
Individual task interpolation increases the number of samples for each classification pair by enabling data from mix(C0, C1), mix(C0, C2), mix(C1, C2).
However, it does not distinguish pairs like (mix(C0, C1), mix(C0, C2)), whereas MLTI does by allowing cross-tasks interpolation.
D A DDITIONAL E XPERIMENTAL S ETUP AND R ESULTS UNDER L ABEL - SHARING S CENARIO D. D ETAILED D ESCRIPTIONS OF D ATASETS AND E XPERIMENTAL S ETUP
Under the label-sharing scenario, We detail the four datasets as well as their corresponding base models.
All hyperparameters are listed in Table 5, which are selected by the cross-validation.
Notice that all baselines use the same base models and interpolation-based methods (i.e., MetaMix,
Meta-Maxup, MLTI) use the same interpolation strategies.
RainbowMNIST (RMNIST).
Follow Finn et al.
(2019), we create the RainbowMNIST dataset by changing the size (full/half), color (red/orange/yellow/green/blue/indigo/violet) and angle ( 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) of the original MNIST dataset.
Specifically, we combine training and test set of original MNIST data and randomly select 5,600 samples for each class.
We then split the combined dataset and create a series of subdatasets, where each subdataset corresponds to one combination of image transformations and has 1,000 samples, where each class has 100 samples.
Each task in RainbowMNIST is randomly sampled from one subdataset.
We use 16/6/10 subdatasets for meta-training/validation/testing and list their corresponding combinations of image transformations as follows:
Meta-training combinations: (red, full, 90 ◦ ), (indigo, full, 0 ◦ ), (blue, full, 270 ◦ ), (orange, half, 270 ◦ ), (green, full, 90 ◦ ), (green, full, 270 ◦ ), (orange, full, 180 ◦ ), (red, full, 180 ◦ ), (green, full, 0 ◦ ), (orange, full, 0 ◦ ), (violet, full, 270 ◦ ), (orange, half, 90 ◦ ), (violet, half, 180 ◦ ), (orange, full, 90 ◦ ), (violet, full, 180 ◦ ), (blue, full, 90 ◦ )
Meta-validation combinations:
(indigo, half, 270 ◦ ), (blue, full, 0 ◦ ), (yellow, half, 180 ◦ ), (yellow, half, 0 ◦ ), (yellow, half, 90 ◦ ), (violet, half, 0 ◦ )
Meta-testing combinations: (yellow, full, 270 ◦ ), (red, full, 0 ◦ ), (blue, half, 270 ◦ ), (blue, half, 0 ◦ ), (blue, half, 180 ◦ ), (red, half, 270 ◦ ), (violet, full, 90 ◦ ), (blue, half, 90 ◦ ), (green, half, 270 ◦ ), (red, half, 90 ◦ )
To analyze the effect of task number, we sequentially add more combinations, which are listed as follows: (indigo, half, 180 ◦ ), (indigo, full, 180 ◦ ), (violet, half, 90 ◦ ), (green, full, 180 ◦ ), (indigo, half, 0 ◦ ), (yellow, full, 90 ◦ ), (indigo, 0, 90 ◦ ), (indigo, full, 270 ◦ ), (yellow, full, 0 ◦ ), (red, half, 180 ◦ ), (green, half, 0 ◦ ), (violet, half, 270 ◦ ), (yellow, half, 270 ◦ ), (red, full, 270 ◦ ), (orange, half, 180 ◦ ), (orange, half, 0 ◦ ), (green, half, 180 ◦ ), (indigo, half, 90 ◦ ), (blue, full, 180 ◦ ), (violet, full, 0 ◦ ), (yellow, full, 180 ◦ ), (orange, full, 270 ◦ ), (red, half, 0 ◦ ), (green, half, 90 ◦ )
In RainbowMNIST, we apply the standard convolutional neural network with four convolutional blocks as the base learner, where each block contains 32 output channels.
For MAML, we apply the task adaptation process on both the last convolutional block and the classifier.
We further use
CutMix (Yun et al., 2019) for task interpolation.
Pose prediction.
Follow Yin et al.
(2020), pose prediction aims to to predict the pose of each object relative to its canonical orientation.
We use the released dataset from Yin et al.
(2020) to evaluate the performance of MLTI, where 50 and 15 objects are used for meta-training and meta-testing.
Each category includes 100 gray-scale images, and the size of each image is 128 × 128 .
As for the base model, we follow Yin et al.
and define the base model with three fixed blocks and four adaptive blocks, where MAML only performs task-specific adaptation on the adapted blocks.
Each fixed block contains one convolutional layer and one batch normalization layer, where the number of the output channels in the three convolutional layers are set as 32, 48, 64, respectively.
After the second fixed block, we add one max pooling layer, where both the kernel size and stride are set as 2.
The output of the fixed blocks is fed into a fixed Linear layer and reshaped to 14 × 14 × 1 , which is further treated as the input of adapted blocks.
Each adapted block includes one convolutional layer and one batch normalization layer, where the number of output channels of all convolutional layer is set as 64.
ReLU function is used as the activation layer for all blocks in this experiment.
Manifold Mixup (Verma et al., 2019) is used for feature interpolation.
All baselines are rerun under the same environment.
NCI.
We use the "NCI balanced" dataset released in (NCI, 2018), where 9 subdatasets are included (i.e., NCI 1, 33, 41, 47, 81, 83, 109, 123, 145).
Each NCI subdataset is a complete bioassay for an binary anticancer activity classification (i.e., positive/negative), where each assay contains a set of chemical compounds.
We randomly sample 1000 data samples for each subdataset.
In our experiments, we represent each drug compound through the 1024 bit fingerprint features extracted by
RDKit (Landrum, 2016), where each fingerprint bit corresponds to a fragment of the molecule.
We select NCI 41, 47, 81, 83, 109, 145 for meta-training and NCI 1, 33, 123 for meta-testing, where each task is sampled from one subdataset.
The extracted 1024 bit fingerprint features are fed into an neural network with two fully connected blocks and one linear regressor.
Each fully connected block contains one linear layer, one batch normalization layer and one Leakyrelu function (negative slope: 0.01) as activation layer, where the number of output neurons of each fully connected block is set as 500.
In our experiments, for MAML, the parameters in the first fully connected block is globally shared across all tasks, and the rest layers are set as adapted layers.
We adopt Manifold Mixup (Verma et al., 2019) as the interpolation strategy.
TDC Metabolism
Similar to NCI dataset, we create another bio-related dataset – TDC
Metabolism.
In TDC Metabolism, we select 8 subdatasets related to drug metabolism from the whole TDC dataset (Huang et al., 2021), including CYP P450 2C19/2D6/3A4/1A2/2C9 Inhibition,
CYP2C9/CYP2D6/
CYP3A4 Substrate.
The aim of each dataset is to predict whether each drug compound has the corresponding property.
We use P450 1A2/3A4/2D6 and CYP2C9/CYP2D  substrate for meta-training, and CYP2C19/2C9 and CYP3A4 substrate for meta-testing.
We balance each subdataset by randomly selecting at most 1000 data samples and each task is randomly sampled from one subdataset.
Analogy to the NCI dataset, we use the same neural network architecture and features (i
[Table]
e., 1024 bit fingerprint) for TDC Metabolism.
Manifold Mixup (Verma et al., 2019) is used as the interpolation strategy.
[Table]
D. C OMPATIBILITY A NALYSIS UNDER L ABEL - SHARING S CENARIO
In Table
[Table]
we show the additional compatibility analysis under the label-sharing scenario.
We observe that MLTI achieves the best performance under different backbone meta-learning algorithms, indicating the compatibility and effectiveness of MLTI in improving the generalization ability
. D. E FFECT OF THE N UMBER OF M ETA - TRAINING T ASKS UNDER L ABEL - SHARING S CENARIO
For RainbowMNIST, we analyze the effect of the number of meta-training combinations with respect to the performance, where the number of meta-training combinations directly reflects the number of meta-training tasks.
Figure 4a and 4b illustrate the results of MAML and ProtoNet, respectively.
The results indicate that MLTI consistently improves the performance, especially when the number of combinations are limited (e.g., Figure 4b).
[Figure]
[Table]
E A DDITIONAL E XPERIMENTAL S ETUP AND R ESULTS UNDER N ON - LABEL - SHARING S CENARIO E. D ETAILED D ATASET D ESCRIPTIONS OF E XPERIMENTAL S ETUP
In this section, we detail the dataset description and the model architecture under the non-labelsharing scenario.
The hyperparameters are selected by cross-validation and listed in Table 7.
For fair comparison, all baselines adopt the same base models.
Additionally, all interpolation-based methods (i.e., MetaMix, Meta-Maxup, MLTI) adopt the same interpolation strategies.
miniImagenet-S.
In miniImagenet-S, we reduce the number of tasks by controlling the number of meta-training classes.
Specifically, in miniImagenet-S, the following classes are used for metatraining:
n03017168, n07697537, n02108915, n02113712, n02120079, n04509417, n02089867, n03888605, n04258138, n03347037, n02606052, n
To analyze the effect of task number, we incrementally add more classes by the following sequence: n03476684, n02966193, n13133613, n03337140, n03220513, n03908618, n01532829, n04067472, n02074367, n03400231, n02108089, n01910747, n02747177, n02795169, n04389033, n04435653, n02111277, n02108551, n04443257, n02101006, n02823428, n03047690, n04275548, n04604644, n02091831, n01843383, n02165456, n03676483, n04243546, n03527444, n01770081, n02687172, n09246464, n03998194, n02105505, n01749939, n04251144, n07584110, n07747607, n04612504, n01558993, n03062245, n04296562, n04596742, n03838899, n02457408, n13054560, n03924679, n03854065, n01704323, n04515003, n
We apply the same base learner as Finn et al.
(2017) in our experiments, which contains four convolutional blocks and a classifier layer.
Each convolutional block includes a convolutional layer, a batch normalization layer and a ReLU activation layer.
For MAML, we apply the task-specific adaptation on the last convolutional block and the classifier layer, which yields the best empirical performance.
ISIC.
In ISIC dataset, we select task 3 in “ISIC 2018:
Skin Lesion Analysis Towards Melanoma
Detection" challenge (Milton, 2019), where 10,015 medical images are labeled by seven lesion categories: Nevus, Dermatofibroma, Melanoma, Pigmented Bowen’s, Benign Keratoses, Basal Cell
Carcinoma, Vascular.
Follow Li et al.
, we use four categories with the largest number of categories as meta-training classes, including Nevus, Melanoma, Benign Keratoses, Basal Cell
Carcinoma.
The rest three categories are treated as meta-testing classes.
We apply N-way, K-shot settings in ISIC and set N = 2 in our experiments.
Thus, there are only six class combinations for the meta-training process.
Each medical image in ISIC are re-scaled to the size of 84 × 84 × 3 and the base model as well as other settings are the same as miniImagenet-S.
DermNet-S. We construct the Dermnet-S dataset from the public Dermnet Skin Disease Atlas (Der, 2016), which includes more than 22,000 across 625 fine-grained classes after removing duplicated images/classes.
Similar to (Prabhu et al., 2018), we focus on the classes with no less than 30 images, resulting in 203 selected classes.
The base model and other settings are the same as miniImagenet-S and ISIC.
The selected classes has a long-tail and we use the top-30 classes for meta-training and the bottom-53 classes for meta-testing.
The detailed meta-training and meta-testing classes are listed as follows.
Meta-training classes:
Seborrheic Keratoses Ruff, Herpes Zoster, Atopic Dermatitis Adult
Phase, Psoriasis Chronic Plaque, Eczema Hand, Seborrheic Dermatitis,
Keratoacanthoma, Lichen Planus, Epidermal Cyst, Eczema Nummular, Tinea (Ringworm) Versicolor, Tinea (Ringworm) Body, Lichen Simplex Chronicus,
Scabies, Psoriasis Palms Soles, Malignant Melanoma, Candidiasis large
Skin Folds, Pityriasis Rosea, Granuloma Annulare, Erythema Multiforme,
Seborrheic Keratosis Irritated, Stasis Dermatitis and Ulcers, Distal
Subungual Onychomycosis, Allergic Contact Dermatitis, Psoriasis,
Molluscum Contagiosum, Acne Cystic, Perioral Dermatitis, Vasculitis,
Eczema Fingertips
Meta-testing classes:
Warts, Ichthyosis Sex Linked, Atypical Nevi, Venous Lake, Erythema
Nodosum, Granulation Tissue, Basal Cell Carcinoma Face, Acne Closed
Comedo, Scleroderma, Crest Syndrome, Ichthyosis Other Forms, Psoriasis
Inversus, Kaposi Sarcoma, Trauma, Polymorphous Light Eruption,
Dermagraphism, Lichen Sclerosis Vulva, Pseudomonas, Cutaneous Larva
Migrans, Psoriasis Nails, Corns, Lichen Sclerosus Penis, Staphylococcal
Folliculitis, Chilblains Perniosis, Psoriasis Erythrodermic, Squamous
Cell Carcinoma Ear, Basal Cell Carcinoma Ear, Ichthyosis Dominant,
Erythema Infectiosum, Actinic Keratosis Hand, Basal Cell Carcinoma Lid,
Amyloidosis,
Spiders
, Erosio Interdigitalis Blastomycetica,
Scarlet
Fever
, Pompholyx, Melasma, Eczema Trunk Generalized, Metastasis, Warts
Cryotherapy
, Nevus Spilus, Basal Cell Carcinoma Lip, Enterovirus,
Pseudomonas Cellulitis
, Benign Familial Chronic
Pemphigus
Pressure
Urticaria
, Halo Nevus,
Pityriasis Alba
Pemphigus Foliaceous
Cherry
Angioma, Chapped Fissured Feet, Herpes Buttocks, Ridging Beading
To further analyze the effect of task number, similar to miniImagenet, we incrementally add more classes for meta-training by the following sequence:
Lupus Chronic Cutaneous, Rosacea, Genital Warts, Dermatofibroma,
Seborrheic Keratoses Smooth, Basal Cell Carcinoma Lesion, Sun Damaged
Skin, Tinea (Ringworm) Groin, Lichen Sclerosus Skin, Atopic Dermatitis
Childhood Phase, Psoriasis Guttate, Warts Common, Warts Plantar,
Herpes Cutaneous, Eczema Subacute, Psoriasis Scalp, Bullous Pemphigoid,
Sebaceous Hyperplasia, Pyogenic Granuloma, Phototoxic Reactions,
Urticaria Acute, CTCL Cutaneous T-Cell Lymphoma, Drug Eruptions,
Mucous Cyst, Alopecia Areata, Hidradenitis Suppurativa, Herpes Type 1 Recurrent, Viral Exanthems, Skin Tags Polyps, Melanocytic Nevi,
Dermatitis Herpetiformis, Eczema Foot, Morphea, Intertrigo, Atopic
Dermatitis Infant phase, Bowen Disease, Necrobiosis Lipoidica, Lentigo
Adults, Xanthomas, Rhus Dermatitis, Keratosis Pilaris, Schamberg Disease,
Rosacea Nose, Chondrodermatitis Nodularis, Keloids, Tinea (Ringworm) Foot
Webs, Tinea (Ringworm) Laboratory, Porokeratosis, Impetigo, Basal Cell
Carcinoma Pigmented, Porphyrias, Epidermal Nevus, Fixed Drug Eruption,
Venous Malformations, Acne Open Comedo, Perlèche, Acne Pustular, Herpes
Type 1 Primary, Tinea (Ringworm) Scalp, Neurofibromatosis, Warts Flat,
Pityriasis Rubra Pilaris, Hemangioma, Herpes Type 2 Primary, Tinea (Ringworm) Hand Dorsum, Neurotic Excoriations, Tinea (Ringworm)
Primary
Lesion, Basal Cell Carcinoma Nose, Dariers disease, Tinea (Ringworm) Foot
Dorsum, Tinea (Ringworm) Face, Tinea (Ringworm) Incognito, Acanthosis
Nigricans, Onycholysis, Warts Digitate, Psoriasis Pustular Generalized,
Varicella, Basal Cell Carcinoma Superficial, Herpes Simplex, Nevus
Sebaceous
Actinic Keratosis
Acne Keloidalis
Hemangioma Infancy
Candida
Tuberous Sclerosis
Stucco Keratoses
Eczema Herpeticum
Epidermolysis Bullosa
Actinic Cheilitis Squamous Cell
Ticks
Actinic Keratosis
Chronic Paronychia
Biting Insects
Grovers Disease
Atypical
Patch
Telangiectasias
Pityriasis Lichenoides
Psoriasis Hand
Actinic
Keratosis Lesion
Lichen Planus
Tinea
Ringworm
Foot Plantar
Chronic
Herpes Type
Recurrent
Lupus
Eczema Asteatotic
Pilar
Pemphigus
Vitiligo
Exfoliativa
Acquired
Syringoma
Habit Tic Deformity
We use CutMix (Yun et al., 2019) to interpolate samples in the above three image classification datasets.
Besides, the interpolation strategy is applied on the query set when i = j , which empirically achieves better performance.
Tabular Murris.
Follow (Cao et al., 2021), the Tabular Murris dataset is collected from 23 organs, which contains 105,960 cells of 124 cell types.
We aim to classify the cell type of each cell, which is represented by 2,866 genes (i.e, the dimension of features is 2,866).
We use the code of Cao et al.
(2021) to construct tasks, where 15/4/4 organs are selected for meta-training/validation/testing.
The selected organs are detailed as follows:
Meta-training organs: BAT, MAT, Limb Muscle, Trachea, Heart, Spleen, GAT, SCAT, Mammary Gland, Liver, Kidney, Bladder, Brain Myeloid, Brain Non
-Myeloid, Diaphragm
. Meta-validation organs: Skin, Lung, Thymus, Aorta Meta-testing organs: Large Intestine, Marrow, Pancreas, Tongue
In Tabular Murris
[Table]
the base model contains two fully connected blocks and a linear regressor, where each fully connected block contains a linear layer, a batch normalization layer, a ReLU activation layer
[Table]
and a dropout layer.
Follow Cao et al.
[Table]
the default dropout ratio and the output channels of the linear layer are set as 0.2, 64, respectively.
We apply Mainfold Mixup (Verma et al., 2019) as the interpolation strategy.
It also worthwhile to mention that the performance of gradient-based methods (e
[Table]
g., MAML) significantly outperforms the reported results in Cao et al.
(2021) since they only apply 1-step inner-loop gradient descent in their released code.
In addition, during the whole meta-testing process, we change the mode from training to evaluation, resulting in the better performance of metric-based methods (e
[Table]
g., Protonet)
[Table] [Table]
[Table] [Table]
[Table]
E. R ESULTS ON FULL - SIZE FEW - SHOT IMAGE CLASSIFICATION DATASETS
In this subsection
[Table]
we provide the results of MLTI and other strategies on full-size miniImagenet and DermNet in Table 8, where 64 and 150 training classes are used in the meta-training process, respectively.
Under the full-size miniImagenet and DermNet settings, the original meta-training tasks are sufficient to obtain satisfying performance.
Nevertheless, applying MLTI also outperforms other strategies, demonstrating its effectiveness in improving generalization ability in meta-learning.
E. C OMPATIBILITY A NALYSIS UNDER N ON - LABEL - SHARING S CENARIO
In Table
[Table]
we report the results of additional compatibility analysis under the non-label-sharing scenario.
The results validate the effectiveness and compatibility of the proposed MLTI.
[Table]
[Table] [Table]
E. R ESULTS OF A BLATION S TUDY UNDER N ON - LABEL - SHARING S CENARIO
In Table
[Table]
we report the ablation study under the non-label-sharing scenario.
The results indicate that MLTI outperforms all other ablation strategies and achieves better generalization ability
. E. A DDITIONAL A NALYSIS ABOUT M ODEL C APACITY AND H YPERPARAMETERS E.5. M ODEL C APACITY A NALYSIS
Here, we investigate the performance of MLTI with a heavier backbone model.
To increase the model capacity, we use ResNet-12 as the base model.
The results on miniImagenet-S and Dermnet-S are reported in Table 11.
Here, the results of Meta-Maxup and MetaMix are also reported for comparison.
According to the results, MLTI outperforms vanilla MAML/ProtoNet, Meta-Maxup and MetaMix, verifying its effectiveness even with a larger base model.
E.5. A NALYSIS OF THE I NTERPOLATION L AYERS
We further conduct experiments on Metabolism and Tabular Murris to analyze the performance with different interpolation layers when Manifold Mixup (i.e., interpolating features) is used for data interpolation.
Here, ProtoNet is used as backbone.
We report the results in Table 12.
The results indicate (1) fixing the interpolation layer can also boost the performance; (2) randomly selecting
[Table]
the interpolation layer achieves the best performance; (3) interpolating at the lower layer performs similarly as interpolating at the higher layer, indicating the robustness of MLTI with different selected layers.
E. A DDITIONAL R ESULTS OF A NALYSIS ABOUT THE N UMBER OF T ASKS
Besides the results in the main paper, we further provide the 1-shot results for miniImagenet and
Dermnet in Figure 5a and 5b, respectively.
The results corroborate our findings in the main paper that
MLTI consistently improves the performance, especially when the number of tasks is limited.
[Figure]
E. MLTI WITH E XTREMELY L IMITED T ASKS
In this section, we investigate how MLTI performs when we only have extremely limited tasks.
Here, we decrease the number of distinct meta-training tasks of miniImagenet and DermNet toby reducing the number of base classes to 8 since (cid:0) 85 (cid:1) = 56 .
Under this setting, two additional baselines with supervised training process (SL) (Dhillon et al., 2020) and multi-task training process (MTL) (Wang et al., 2021) are also used for comparison.
We also report the results of the best baseline – MetaMix.
All results are listed in Table 13 and corroborate the effectiveness of MLTI even with extremely limited meta-training tasks.
E. R ESULTS ON A DDITIONAL D ATASETS
We further provided two additional datasets under the non-label-sharing setting to show the effectiveness of MLTI – tieredImageNet-S and Huffpost.
Both datasets are non-label-sharing datasets.
We detail the data descriptions and hyperparameters in the following
• tieredImageNet-S .
tieredImageNet (Ren et al., 2018) is a few-shot image classification dataset, which consists of 351/97/160 images for meta-training/validation/testing.
Following miniImagenetS and DermNet-S, we use 35 original meta-training classes in tieredImageNet.
All hyperparameters, base model and interpolation strategies are set as the same as miniImagenet-S.
[Table]
• Huffpost .
Huffpost (Misra, 2018) aims to classify the category for each sentence.
We follow Bao et al.
(2020) to preprocess Huffpost data, where 25/6/10 classes are used for metatraining/validation/testing.
In our experiments, to construct the base model, we use ALBERT (Lan et al., 2019) as the encoder and use two fully connected layers as the classifier.
The query set size for training and testing is set as 5.
Due to the memory limitation, we set the batch size as 1 and
the learning rate (outer-loop learning rate) for MAML as 2e-5.
The inner-loop learning rate for
MAML is set as 0.01.
We set α = β = 2 . 0 in Beta( α, β ) .
The number of inner loop updates in
MAML is set as 5 and the maximum training iteration is set as 10,000.
Manifold Mixup is used for data interpolation.
We report the results in Table 14.
In these two additional datasets, MLTI also outperforms other methods, showing its promise in improving generalization in meta-learning.
[Table]
ABLES WITH C ONFIDENCE I
Table 15, 16 report the full results (accuracy ± 95% confidence interval) of Table 3, 4 in the paper.
[Table] [Table]