A Meta Understanding of Meta-Learning
Abstract
Recent years have witnessed an abundance of new publications and approaches on meta-learning.
This community-wide enthusiasm has sparked great insights but has also created a plethora of seemingly different frameworks, which can be hard to compare and evaluate.
In this paper, we aim to provide a single principled, unifying framework that draws a close connection between meta-learning and traditional supervised learning.
By treating pairs of task-speciﬁc data sets and trained models as (feature, label) samples, we can reduce many meta-learning algorithms to instances of supervised learning.
This view not only uniﬁes meta-learning into an intuitive and practical framework but also allows us to transfer insights from supervised learning directly to improve meta-learning.
For example, we obtain a better understanding of generalization properties, and we can readily transfer well-understood techniques, such as model ensemble, pre-training, joint training, data augmentation, and even nearest neighbor based methods.
We provide an intuitive analogy of these methods in the context of metalearning and show that they give rise to signiﬁcant improvements in model performance.
1. Introduction
Meta-learning, or learning to learn, is the sub-ﬁeld of machine learning occupied with the search for the best learning strategy as the number of tasks and learning experiences increases (Vilalta & Drissi, 2002) and has drawn signiﬁcant attention recently (Finn et al., 2017a; Andrychowicz et al., 2016;
Vinyals et al., 2016).
Meta-learning has been developed in various areas to advanced algorithm design, including few-shot learning (Ravi & Larochelle, 2017;
Snell et al., 2017), optimization (Li & Malik, 2017; Wichrowska et
, 2017), active learning (Bachman et al., 2017), transfer learning (Balaji et al., 2018;
Ying et al., 2018), unsupervised learning (Metz et al., 2018), etc.
Speciﬁcally, meta-learning has demonstrated the capability to generalize learned knowledge to novel tasks, which greatly reduces the need for training data and time to optimize.
Given its wide applicability and many diverse approaches, there is an increasing need for a principled, unifying metalearning framework to facilitate future studies and development.
Finn (2018); Metz et al.
(2018) have made a notable step to provide a broad introduction and compare representative algorithms.
In this paper, we aim to push the direction forward by providing a unifying view of meta-learning that draws a close connection with supervised learning.
Generally, meta-learning can be viewed as learning a mapping that given a training set D tr returns a model h .
For instance, in unsupervised learning D tr can be an unlabeled data set, and h is a feature extractor.
In one-shot learning,
D tr corresponds to a set of C labeled data samples, and h corresponds to a C -way classiﬁer.
This mapping constitutes an interesting connection between meta-learning and supervised learning which, as we argue in this paper, holds across a wide range of meta-learning approaches and tasks.
Similar to supervised learning, we can train a meta-learning model on a set of meta labeled examples (tasks) — ( D tr , h ∗ ) pairs — and evaluate it on a test task D testtr .
We refer to D tr as a meta input (often a training set) and h ∗ as the target model (often obtained with ample training data).
See Fig.
This unifying view of meta-learning as supervised learning allows transferring ideas, experiences, and design principles of supervised learning to meta-learning, which will greatly facilitate its theoretical analysis, algorithm design, and applicability.
In the following we list three notable examples.
First, the unifying view offers an explanation why the learned meta mapping may or may not generalize to novel tasks: here we rediscover well understood pitfalls from supervised learning, such as overﬁtting due to insufﬁcient (meta-) training data and distribution drift.
Second, our unifying view clearly identiﬁes the essential components to design a meta-learning algorithm, providing a principled way to apply it to various areas.
Third, our unifying view broadens the scope of algorithm design for meta-learning by extending concepts from supervised learning.
We conduct extensive experiments on few-shot learning, a representative area where meta-learning is applied.
We empirically show that well-known supervised learning techniques such as data augmentation, bagging (Breiman, 1996), joint training (Argyriou et al., 2007), pre-training
(Yosinski et al., 2014), and non-parametric approaches (Zhang et al., 2006; Weinberger & Saul, 2009) can be applied at the task level to signiﬁcantly facilitate meta-learning.
2. A Unifying View of Meta-Learning 2.1. Background: supervised learning
In supervised learning we collect a training set D tr = { ( x n ∈ X , y n ∈ Y ) }
Nn =1 , composed of N i.i.d.
samples from an unknown distribution D on X × Y .
We call x an input, y an output (or label), and ( x, y ) a labeled example.
We call h : X (cid:55)→ Y a model, which outputs a label for an input.
For instance, in image classiﬁcation, x is an image, y is a class name (e.g., “dog”), D is the distribution of real images, and h is an image classiﬁer.
Supervised learning searches for a model h given D tr , so that h will work well on ( x, y ) sampled from D .
That is, h should have a small generalization error L D ( h ) according to a loss l : Y × Y (cid:55)→ R
[Equation]
To this end, we construct a hypothesis set H = { h } of candidate models and design an algorithm A H to search ˆ h from H by learning from D tr .We denote ˆ h =
A H ( D tr ) .
One example of H is a neural network with a ﬁxed architecture but undecided weights.
A popular framework to design supervised learning algorithms is empirical risk minimization (ERM)
[Equation]
where L S ( h ) is the training loss.
A particular A H is characterized by how it performs ERM (e.g., optimizers).
In practice, we usually design a set of ( H , A H ) pairs, denoted as G , and select the best pair using a held-out validation set D val = { [Equation] x m , y m
[Equation]
Mm =1 sampled i.i.d.
from D
[Equation]
where ˆ h =
A H ( D tr ) .
This is called model selection and L V is the validation error.
2.2. Meta-learning as supervised learning
We provide a framework of meta-learning by drawing analogy to supervised learning.
We use “meta (labeled) example” and “task” interchangeably.
To prevent confusion, we call models in supervised learning “base” models when needed.
Deﬁnition.
In meta-learning we collect a meta-training set D meta-tr = { ( D tr j ∈
I , h ∗ j ∈ O ) } N meta j =1 , composed of N meta i.i.d.
samples from an unknown meta distribution D meta on I × O .
We call D tr a training set (meta input), h ∗ a “target” base model (meta output), and ( D tr , h ∗ ) a meta labeled example (task).
We call g : I (cid:55)→ O a meta model (mapping), which outputs a base model for a training set.
For instance, in one-shot C -way learning for classiﬁcation, D tr = { ( x n , y n ) } Nn =1 contains N labeled images, one for each of the C classes (i.e., N = C ).
h ∗ is a strong classiﬁer trained using a large amount of labeled images.
Meta-learning searches for a meta model g given D meta-tr , so that g will work well on ( D tr , h ∗ ) sampled from D meta .
That is, g should have a small meta generalization error L D meta ( g ) according to a meta loss l meta : [Equation]
×
[Equation]
R
[Equation]
In one-shot learning, we can view g as a predictor of (strong) classiﬁers given small training sets.
Algorithm. To this end, we can follow supervised learning to construct a meta hypothesis set G = { g } of candidate meta models and design an algorithm B G to search ˆ g from
G by learning from D meta-tr We denote ˆ g = B G ( D meta-tr ) .
We can apply ERM
[Equation]
where L S meta is the meta training error.
In practice, we design a set of ( G , B G ) pairs and select the best pair using a held-out meta-validation set D meta-val =
{ ( D tr m , h ∗ m ) } M meta m = sampled i.i.d.
from D meta .
This is called meta model selection (meta-validation).
Discussion. The meta model g and the supervised learning algorithm
A H (cf.
Eq.
(2)) have the same forms of inputs and outputs.
Therefore, g can be seen as a (supervised) learning algorithm which may involve an optimization process.
Indeed, Metz et al.
(2018) point out that many meta-learning algorithms consist of two levels of learning, in which g is applied at the inner loop while our B G is applied at the outer loop.
Nevertheless, the generalization ability of g and A H are fundamentally different.
A H is designed or selected by model selection speciﬁcally for D , while g is learned from D meta for the goal to generalize to tasks sampled from D meta .
Moreover, viewing ( D tr , h ∗ ) as a meta labeled example enables h ∗ to be disentangled from D tr ; i.e., h ∗ can be ﬂexibly deﬁned to provide supervision for various applications, and is not necessarily a base model that performs on data sampled exactly from where D tr is sampled.
This notion broadens the applicability of meta-learning — e.g., to domain generalization where h ∗ is for a different domain, or to unsupervised learning where D tr contains only unlabeled examples { x n } Nn =1 (see section 2.4).
2.3. Generalization ability of learned meta models
Our unifying view enables transferring experiences of learning a base model to learning a meta model.
For example, increasing the size of meta-training set or minimizing the domain shift between D meta and where the novel tasks will be sampled (Ben-David et al., 2010; Gong et al., 2012) should improve the generalizability of the learned meta model ˆ g .
We show that these experiences are applicable in section 4. and E.5.
We can further apply theoretical analysis of supervised learning.
Suppose l meta is bounded and |G| is ﬁnite, the Chernoff bound implies that G is agnostic PAC learnable using ERM (Shalev-Shwartz & Ben-David, 2014).
This is indeed exploited in (Garg & Kalai, 2018) to derive a metalearning bound, but only for meta unsupervised learning.
2.4. A principled way to apply meta-learning
Our unifying view clearly deﬁnes the components of metalearning: (a) meta labeled example ( D tr , h ∗ ) and meta loss l meta ; (b) meta hypothesis set G = { g } and meta algorithm B G .
Applying meta-learning to an area thus requires deﬁning or designing them accordingly.
We discuss ﬁve examples in section
B. We further present a detailed case study on how to apply our meta-learning framework to few-shot learning in section C and demonstrate many representative algorithms follow our framework.
2.5. Supervised learning techniques
Our unifying view enables transferring techniques of supervised learning to meta-learning with minimum adjustment.
In our experiment, we adapt widely-used techniques for improving generalization abilities or facilitating opti- [Figure] mization, including data augmentation, ensemble methods (Breiman, 1996; Zhou, 2012; Dietterich, 2000), joint training (e.g., multi-task learning (Argyriou et al., 2007)), and pre-training (Yosinski et al., 2014) to meta-learning.
Besides, it is well-known that non-parametric models (Weinberger & Saul, 2009) are able to capture local and heterogeneous structures in data.
Contrast to the fact that most existing meta models are parametric, we present a procedure meta-KNN, inspired by SVM-KNN (Zhang et al., 2006), to build a non-parametric meta model in section A.
3. Related Work (See section D) 4. Experiments
We validate the advantages of our unifying view by investigating three aspects of few-shot learning: (a) whether the number of meta-examples inﬂuences the generalization; (b) whether supervised learning techniques are applicable; (c) whether our unifying view can facilitate new applications.
Datasets.
The Mini ImageNet dataset (Vinyals et al., 2016) is a subset of ImageNet (Russakovsky et al., 2015) and is widely-used in few-shot learning.
There are 100 classes and 600 examples per class.
We split the datasets following (Ravi & Larochelle, 2017): there are 64, 16, 20 classes for meta-train-pool, meta-val-pool, and meta-test-pool.
See section C.3 for deﬁnitions and section E.1 for other datasets.
Meta examples (tasks) and evaluation protocols.
We focus on 1-shot 5-way tasks.
We follow (Rusu et al., 2019) to draw 10,000 tasks ( D tr , D val ) from meta-test-pool and there are 15 validation images per class in a task.
Baseline methods.
We investigate MAML (Finn et al., 2017a), ProtoNet (Snell et al., 2017), and MatchNet (Vinyals et al., 2016).
See section E.2 for more details.
4.1. Generalization analysis
Following our supervised view of meta-learning, the generalization ability of the learned meta model should be affected by the “effective” number of meta training examples; i.e., the number of different tasks.
We validate the inﬂuence with various meta-train-pool conﬁgurations on Mini ImageNet.
We keep all the 64 classes in meta-train-pool to keep the meta distribution intact, and change the number of instances in each class from 600 to 50 to construct different meta- [Table] [Table]
train-pools. When there are limited instances per class in the meta-train-pool, the number of unique tasks is constrained as well. We also evaluate the case of fewer classes in section E.3.
Fig.
2 shows the change of few-shot accuracy with different meta-train-pools.
The trend follows the supervised view: the more meta-training examples (few-shot tasks) are provided, the better generalization is achieved.
Speciﬁcally, learning with 50 instances per class signiﬁcantly over-ﬁts.
4.2. Supervised learning techniques for meta-learning
We investigate ﬁve techniques: joint training, pre-training, bagging, data augmentation, and non-parametric methods.
Joint training.
In addition to the original objective (i.e., meta-training loss), learning jointly with related objectives has shown promising results in supervised learning: it serves as a data-dependent regularization to improve generalization.
Here we add another objective: a 64-way classiﬁcation with cross-entropy loss over all classes in meta-train-pool.
This classiﬁer shares weights with the meta model, except for the last layer.
In other words, the shared sub-network should jointly master two tasks: predicting good C -way classiﬁers and extracting discriminative features for 64-way classiﬁcation.
Oreshkin et al.
(2018) explored this idea, yet we provide more insights in Table 1.
Pre-training.
It is well-known that a good model initialization signiﬁcantly facilitates the following model optimization for down-stream task (Erhan et al., 2010; Bengio et al., 2007).
Speciﬁcally, supervised pre-training (Yosinski et al., 2014) on a large labeled dataset has been prevalent in many applications.
While the standard setup in few-shot learning trains the meta model from scratch on the sampled meta labeled examples, we investigate training a 64-way classiﬁer with cross-entropy loss at ﬁrst, and use it to initialize the meta model.
Such a strategy has been explored in (Chen et al., 2019; Qiao et al., 2018; Rusu et al., 2019; Li et al., 2018b), yet we provide more insights as follows.
The results with the two techniques are shown in Table 1.
We list the meta-training and meta-test accuracy. Both techniques improve the test accuracy for all the algorithms.
However, their underlying inﬂuences are different.
Pre-training achieves the highest meta-training accuracy, justifying its effectiveness to facilitate optimization.
Joint training does not increase the meta-training accuracy much but improves the meta-test accuracy, verifying its ability to improve generalization.
In sum, we show that well-known supervised learning techniques are applicable to meta-learning in the same manner they beneﬁt supervised learning.
Ensemble methods.
Ensemble methods leverage the diversity among a set of basic models to construct a robust summarized model.
It has been comprehensively veriﬁed in supervised learning.
We apply a simple ensemble method bagging (Breiman, 1996) to meta-learning, which reduces the model variances by majority voting over many classiﬁers.
We generate diverse meta models by learning from different meta-training sets.
Concretely, we sample 10 different sub meta-train-pools from the original one: each sub-pool contain 48 classes.
Then we train 10 basic ProtoNets over those sub meta-train-pools and ensemble their results.
The results are in Table 2.
“Single” is the model trained on the original meta-train-pool, the same as in Table 1.
The average performance of all 10 basic models are in the “Average” column.
(Note that this is not ensemble yet.)
We investigate “Bagging” by combining the prediction probabilities of the 10 models.
The average performance of basic ProtoNets is mostly worse than that of the ProtoNet trained on the full 64-class meta-train-pool.
But after we ensemble the basic models the performance improves notably, validating the effectiveness of ensemble for meta-learning.
Moreover, we see that pre-training and bagging are compatible and lead to impressive 0.526/0.701 accuracy for 1/5-shot learning, which is on par with state-of-the-arts (Qiao et al., 2018) but with a cleaner meta-training procedure.
Additional Experiments.
See section E.4 and E.5.
5. Conclusion
We cast meta-learning as supervised learning, which explains the generalization ability among tasks and provides principles to apply meta-learning.
Moreover, this framework allows transferring supervised learning techniques to meta-learning.
We conduct extensive empirical studies to demonstrate these advantages, suggesting supervised learning techniques as a toolbox to advance meta-learning.
Supplementary Material A. Meta-KNN
We present a procedure inspired by SVM-KNN (Zhang et al., 2006; Chao et al., 2013) to build a non-parametric meta model.
We call it meta-KNN, as summarized in Algorithm 1.
We empirically show its superior performance on meta labeled examples from heterogeneous domains (cf.
section E.4).
Algorithm 1 meta-KNN : a non-parametric meta-learning algorithm (cf.
Eq.
Required D meta-tr : a meta-training set Required α , β , K : hyper-parameters Required B G : an iterative meta-learning algorithm, with ˆ g = B G ( D meta-tr ) trained till converge Meta input A novel task D tr 1: Search K NN( D tr ): K NN tasks of D tr in D meta-tr 2.
Copy ˜ g from ˆ g .
Fine-tune ˜ g on K NN( D tr ), by applying
B G for β epochs with a step size α
Meta output ˜ g
( D tr )
B. A Principled Way to Apply Meta-Learning
We discuss ﬁve examples as follows.
Fig.
3 gives an illustration.
Unsupervised learning. Let us consider learning a feature extractor from unlabeled data so as to beneﬁt downstream applications (Boney & Ilin, 2018;
Metz et al., 2018;
Yu et al., 2018).
Here, D tr = { x n } Nn =1 , h ∗ is the target feature extractor, and l meta measures the performance gap on the downstream application.
G is a set of objective functions for unsupervised learning, while B G can be ERM by stochastic gradient descent (SGD).
Active learning.
The goal is to minimize the data labeling effort by querying N (cid:48) informative examples.
For classiﬁcation, D tr = { x n } Nn =1 is an unlabeled set, h ∗ is the model learned with the best N (cid:48) queried examples from D tr , and l meta measures the performance gap on classiﬁcation.
g is a learning process constrained to query labels for N (cid:48) examples by investigating x and the learning progress (Ravi & Larochelle, 2018; Sharma et al., 2018; Bachman et al., 2017).
B G can be ERM by SGD or by reinforcement learning algorithms (Ravi & Larochelle, 2018; Sharma et al., 2018).
Domain generalization.
The goal is to train a base model on source domains (SD) and apply it to target domains (TD) without ﬁne-tuning.
Here, D tr = { ( x n , y n ) }
Nn =1 is the labeled data from SD, while h ∗ is the target base model that works well on TD.
l is the performance gap on TD.
g can be a model predictor or a learning process aware of the domain shifts (Li et al., 2018a; Balaji et al., 2018).
B G can be ERM by SGD.
Optimization. One focus is to search a better updating rule in iterative optimization.
In this case, D tr is an objective function L S ( θ ) parameterized by θ ; e.g., the training loss.
h ∗ is the optimal solution of L S or of a related objective L V ; e.g., the validation error.
l meta is the gap of objective values.
g is an iterative optimization algorithm with a (meta) learnable updating rule.
B G can be EMR by
SGD (Andrychowicz et al., 2016; Wichrowska et al., 2017) or reinforcement learning algorithms (Li & Malik, 2017;
Bello et al., 2017).
Imitation learning and reinforcement learning
Imitation learning (Stadie et al., 2018; Frans et al., 2018;
Wang et al., 2016; Duan et al., 2016; 2017; Finn et al., 2017b;
Yu et al., 2018) easily ﬁts into our meta-learning framework, given its similarity to the supervised learning. For few-shot reinforcement learning (Duan et al., 2017; Wang et al., 2016), the D tr is a trial of a few episodes from an MDP and h ∗ is the target policy, which can be realized by maximizing the reward. C. Case Study: Few-Shot Learning
We present a case study on how to apply our meta-learning framework to few-shot learning.
The goal of few-shot learning is to quickly build a model for a novel task; i.e., with minimum training time and training data.
Speciﬁcally, we focus on one-shot C -way learning for image classiﬁcation.
C.1. Meta labeled examples and meta losses
The training set D tr contains one labeled image for each of the C classes; i.e., D tr = { ( x n , y n ) } Nn =1 ( C = N in this case).
The target model h ∗ is a C -way classiﬁer and the loss l meta ( h, h ∗ ) measures how different h compared to h ∗ .
We present two examples of l meta ( h, h ∗ ) as follows.
Meta losses in the model space. h ∗ is the target classiﬁer trained on a larger training set.
Let h be parameterized by θ , l meta ( h, h (cid:48) )
= (cid:107) θ − θ (cid:48) (cid:107)
(Wang & Hebert, 2016; Wang et al., 2017).
Meta losses from example losses. Given a validation set
D val = { ( x m , y m ) }
Mm =1 sampled in the same way as D tr , a reasonable choice of l meta ( h, h ∗ ) is | L V ( h ) − L V ( h ∗ ) | , where L V is the validation error deﬁned in Eq.
Suppose h ∗ minimizes L V , then l meta ( h, h ∗ ) is equivalent to L V ( h ) .
In other words, we replace h ∗ and l meta by D val and l .
The meta-training set D meta-tr thus becomes { ( D tr j , D val j ) } N meta j [-=
-]= and ERM in Eq.
(5) can be re-written accordingly as (with constants ignored)
[Equation]
L V j is the validation loss on D val
j
= { ( x jm , y jm ) }
Mm =1 .
Eq.
(6) has been applied in many few-shot learning algorithms (Hariharan & Girshick, 2017; Ravi & Larochelle, 2017; Gidaris & Komodakis, 2018; Qiao et al., 2018;
Snell et al., 2017; Sung et al., 2018; Bertinetto et al., 2016; Vinyals et al., 2016) and we will focus on it in the next subsections.
C.2. Meta models and meta algorithms
We discuss two exemplar designs of g : one views g as a supervised learning algorithm
(Finn et al., 2017a) and the other views g as a feed-forward model predictor (Snell et al., 2017).
Both algorithms are used in our experiments.
Meta models as learning algorithms. Since g outputs a classiﬁer h given D tr , it can be seen as a learning algorithm.
Suppose h is parameterized by θ , let us apply ERM
[Equation]
Since D tr is small, ERM may suffer over-ﬁtting.
One solution is to apply an iterative optimizer (e.g., gradient descent (GD)) with early stopping and a carefully chose initialization ψ
(Finn et al., 2017a): early stopping limits the hypothesis set while ψ prevents under-ﬁtting.
Suppose L S ( θ ) is differentiable w.r.t.
θ , g with one-step GD of a step size α , initialized at ψ , is
[Equation]
The meta hypothesis set G thus becomes { g ψ } with different initializations.
To search g ˆ ψ , we again apply ERM but on the meta-training set
D meta-tr (cf.
Eq.
[Equation]
If we apply SGD for optimization then this is the one-step
MAML (Finn et al., 2017a).
In other words, MAML and its variants (Lee & Choi, 2018; Finn & Levine, 2018; Li et al., 2017; Rusu et al., 2019) follow our framework.
Meta models as model predictors.
Alternatively, we can view g ( D tr ) as a model predictor.
For example, in Prototypical Network (ProtoNet) (Snell et al., 2017)
[Equation]
where x c is the image of class c , and ψ is a feature extractor.
Snell et al.
(2017) learns ψ via ERM (cf.
Eq.
(6)), essentially following our framework.
Similar approaches are (Qiao et al., 2018; Sung et al., 2018;
Wang & Hebert, 2016; Vinyals et al., 2016; Wang et al., 2017).
C.3. Collecting meta-training sets
After identifying the form of meta labeled example, which is ( D tr , D val ) for few-shot learning, the next step is to collect the meta-training set.
While collecting ( D tr , D val ) oneby-one seems standard, it might be inefﬁcient.
For image classiﬁcation it is easier to collect images class-by-class to ﬁrst create a pool (named meta-train-pool), which contains many classes and each has many examples.
We then synthesize ( D tr , D val ) from the pool.
This is the general setting for few-shot image classiﬁcation.
There will be three pools of disjoint classes: meta-train pool, meta-val-pool, meta-test-pool.
D. Related Work
There are multiple excellent overviews and surveys (Thrun & Pratt, 1998; Vilalta & Drissi, 2002; Lemke et al., 2015;
Vanschoren, 2018; Finn, 2018; Metz et al., 2018).
Ours is different by drawing a clear connection to supervised learning.
There are also theoretical analysis on metalearning (Baxter, 2000; Maurer et al., 2016; Maurer, 2005). However, the ﬂexibility to design h ∗ (e.g., by validation data) is not considered; the analysis thus may not be readily applicable to areas such as unsupervised learning, domain generalization, etc.
Franceschi et al.
(2017) connects hyperparameter tuning and meta-learning, which aligns with the comparison of A H and g in section 2.2.
Garg & Kalai (2018) also relates meta-learning to supervised learning but only for meta-unsupervised learning.
Meta-learning has also been applied to reinforcement learning and imitation learning (Stadie et al., 2018;
Frans et al., 2018;
Wang et al., 2016; Duan et al., 2016; 2017; Finn et al., 2017b; Yu et al., 2018), optimization (Andrychowicz et al., 2016;
Wichrowska et al., 2017; Li & Malik, 2017; Bello et al., 2017), recommendation systems (Vartak et al., 2017), data augmentation (Ratner et al., 2017), natural language processing (Huang et al., 2018), architecture search (Elsken et al., 2018), continual learning (Riemer et al., 2019; Kaiser et al., 2017; Al-Shedivat et al., 2018; Clavera et al., 2019), transfer and multi-task learning (Ying et al., 2018; Zhang et al., 2018), active learning (Ravi & Larochelle, 2018; Sharma et al., 2018; Bachman et al., 2017; Ravi & Larochelle, 2018;
Sharma et al., 2018) , and teaching (Fan et al., 2018).
Some algorithms aim for versatile purposes (Mishra et al., 2018; Munkhdalai & Yu, 2017; Ritter et al., 2018; Finn et al., 2017a;
Santoro et al., 2016; Nichol et al., 2018).
There are meta-learning algorithms that do not explicitly deﬁne h ∗ or split data into D tr and D val ; e.g., Reptile (Nichol et al., 2018).
We argue that Reptile can indeed be viewed as using D tr as D val .
Section E.5 analyzes the limitation of this kind of algorithms.
E. Additional Experiments and Details E.1. Experimental setups
Datasets.
Beside splitting the 100 Mini ImageNet classes into 60, 16, 24 for meta-train-pool, meta-val-pool, and metatest-pool, we also consider a challenging case where the splits are 30, 30, 40 classes, in which the diversity of metatraining examples is limited.
We call the former standard split (SS) and the later challenging split (CS).
We further consider two set of datasets for more challenging settings.
To enlarge the heterogeneity among tasks, we synthesize a “Heterogeneous” dataset from ﬁve different ﬁnegrained classiﬁcation datasets, namely AirCraft (Maji et al., 2013), Car-196 (Krause et al., 2013), Caltech-UCSD Birds (CUB) 200-
(Wah et al., 2011), Stanford Dog (Khosla et al., 2011), and Indoor (Quattoni & Torralba, 2009).
We randomly sampled 60 classes with 50 images each from each of the 5 datasets, and equally split classes into metatrain-pool, meta-val-pool, and meta-test-pool.
That is, there are 100 classes in meta-train-pool (same for the others), which includes 20 classes from each ﬁne-grained dataset.
To investigate the applicability of meta-learning, we use the Ofﬁce-Home datasets (Venkateswara et al., 2017) in a domain generalization problem.
There are 65 classes and [Figure] domains of images per class.
We test two domains, “Clipart” and “Product”, which have the largest number of images.
We follow (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017a) to resize all images to 84 × 84 .
Meta examples (tasks) and evaluation protocols.
We follow (Rusu et al., 2019; Ye et al., 2018) to evaluate the few-shot classiﬁcation by drawing 10,000 tasks from metatest-pool and there are 15 validation images per class in a task.
That is, a meta labeled example ( D tr , D val ) hasimages per class in D val . We report the mean accuracy. We found the 95% conﬁdence interval to be consistently within [0 . 001 , 0 . 004] and thus omit it for brevity. E.2. Implementation details
We investigate three popular baselines, namely Model Agnostic Meta-Learning (MAML) (Finn et al., 2017a), Prototypical Network (ProtoNet) (Snell et al., 2017), and Matching Network (MatchNet) (Vinyals et al., 2016). MAML implements an inner optimizer to update the meta-learned classiﬁer initializer, and ProtoNet/MatchNet learn discriminative embedding for few-shot classiﬁcation. ProtoNet uses the distance-based nearest class mean rule for prediction, while MatchNet utilizes the similarity-based nearest neighbor rule.
We apply the ﬁrst-order MAML and tune the number of updates in Eq.
(7) by meta-validation.
We re-implement all three algorithms and use the same Cway setting in meta-training and meta-test for consistency.
That is, we disregard the trick (Snell et al., 2017) that trains with 30 -way tasks but tests with 5 -way tasks.
Our baseline results are close to those reported in the recent overview of few-shot learning (Chen et al., 2019).
We apply the standard 4-layer ConvNets as the backbone (Vinyals et al., 2016;
Snell et al., 2017).
In each layer, convolution, batch normalization, ReLU, and max-pooling are concatenated sequentially.
During meta-training SGD with Adam (Kingma & Ba, 2014) is employed, with a initial learning rate 2e-3.
[Figure] [Table] E.3. Generalization analysis
Following our supervised view of meta-learning, the generalization ability of the learned meta model should be affected by the “effective” number of meta training examples. Here we investigate such effective number from another perspective, i.e., the number of classes in the in the metatrain-pool. In detail, we keep all 600 instances per class in Mini ImageNet (SS), but change the number of available classes for meta-training in the meta-train-pool, from 8 to 64. When there are limited types of classes in the meta-trainpool, the number of diversiﬁed tasks to be sampled is constrained.
Fig.
4 shows the change of few-shot accuracy with different meta-train-pools. The trend follows the supervised learning intuition as well: where there are more available meta-training classes (types of few-shot tasks), the better generalization results are achieved. In particular, when there are 8 classes, all models, namely MAML, ProtoNet, and
MatchNet overﬁt. The 1-shot classiﬁcation performance becomes stable when the number of meta-training classes is larger than 48. E.4. Supervised learning techniques for meta-learning Given that in section 4.2 the supervised learning techniques appear to be meta-model-agnostic by improving all three meta-learning models consistently, we focus on ProtoNet in the following experiments.
Data Augmentation.
Increasing the number of training examples by data augmentation is a popular technique in supervised learning to improve the generalization ability, especially when the training examples are not diverse enough.
Here we adapt the idea to meta-learning.
We consider a challenging yet more realistic case on Mini ImageNet (CS).
Speciﬁcally, we re-split the dataset into 30, 30, 40 classes for meta-train-pool, meta-val-pool, and meta-test-pool, and focus on 1 -shot 30 -way classiﬁcation.
Note that, this setting might break the applicability of meta-learning to few-shot learning: every time we will sample the same 30 classes from meta-train-pool, greatly limiting the diversity of meta labeled examples.
However, this setting is indeed more realistic in practice according to the well-know long-tailed distribution (Sudderth & Jordan, 2008; Salakhutdinov et al., 2011; Zhu et al., 2014; Wang et al., 2017): there are more few-shot classes than many-shot classes.
We present a data augmentation strategy at the meta example (task) level, inspired by (Hsu et al., 2018).
Speciﬁcally, we perform K-means within each class to split a class into
K subcategories, resulting in 30 × K augmented classes in meta-train-pool (and meta-val-pool).
To ensure the quality of K-means, we pre-train a 30-way classiﬁer on the whole meta-train-pool and use the features for K-means.
We run 30 trials of K-means to compensate its randomness.
We construct a meta-training example (task) by ﬁrst picking a trial and sampling 30 classes from the corresponding augmented meta-train-pool.
Fig.
5 gives an illustration.
The results with data augmentation are in Table 3.
We cluster each class in the meta-train-pool and meta-val-pool with K = 1 , 2 , 4 , 8 subcategories; 1 means no K-means.
We observe a clear trend: the more classes we augment to increase task diversity, the higher accuracy we can achieve.
Non-Parametric Methods.
Non-parametric supervised learning approaches are well-known to capture local and heterogeneous structures in data.
Here we extend the methods to meta-learning by applying meta-KNN (cf.
Algorithm 1) to ProtoNet, named ProtoNet-KNN.
In meta-learning a neighbor is a meta labeled example (i.e., a few-shot task).
We characterize each task by its average feature and use the
Euclidean distance to measure task similarity.
We investigate ProtoNet-KNN on the Heterogeneous dataset described in section E.1.
We focus on ﬁne-grained classiﬁcation: a test task contains 5 classes from one dataset.
In meta-training, we sample a ﬁne-grained task randomly from a dataset to train a single ProtoNet.
We expect the learned ProtoNet to be equipped with dataset-agnostic ﬁnegrained discriminative knowledge.
In meta-test, we then apply ProtoNet-KNN to adapt the ProtoNet to the K nearest neighbors of a test task.
To facilitate neighbor searching, we pre-sample 2,000 training tasks and set K = 100 , β = 1 , and α = 0 .
(cf.
Algorithm 1).
Table 4 shows the meta-test results (2,000 tasks).
Non-parametric approaches greatly improve the performance on heterogeneous tasks.
We expect a larger gap by a more sophisticated task distance.
E.5. Domain generalization and meta domain shifts
Existing few-shot learning settings assume that the training and validation examples (e.g., images) within a task are from the same domain.
In practice it is sometime desirable to remove the assumption.
For instance, we might want a [Table] [Table] robot to learn in a constraint environment, where examples are not diverse and are essentially few-shot, and to be able to apply the knowledge in the wild.
Recall that our supervised view of meta-learning allows users to deﬁne the training set and the ideal model in a task according to the problem at hand.
By deﬁning the ideal model to be the one that will work well in a different domain, we can remove the samedomain assumption in a systematic way.
We investigate this idea and apply the trick in section C to replace the ideal model by a validation set collected in a different domain.
We experiment on the Off-Home dataset (see section E.1), which is designed for domain adaptation.
We focus on two domains: Clipart (C) and Product (P).
There are 65 classes and we split them into 25 for meta-train-pool, 15 for metaval-pool, and 25 for meta-test-pool.
We work on few-shot 5-way tasks in which a task contains a training set D tr from one (source) domain and a validation set D val from the other (target) domain.
Table 5 summarizes the results.
We clearly see the difﬁculty when the source and target domains within a task are different by comparing Case I-1 and I-2 (similarly, II-1 and
II-2).
We note that in each of these cases the distribution of the meta-training and meta-test examples are the same.
Therefore, there is no meta-domain shift and the gap simply indicates the difﬁculty of tasks.
We further investigate meta-domain shifts by constructing meta-training and meta-test examples in a different way (Case I-3, I-4 and II-3, II-4).
We note that some metalearning algorithms can only handle same domains by default (Nichol et al., 2018).
These additional experiments are meant to simulate the results by those algorithms.
The performance in these cases are outperformed by Case Iand II-2 respectively where each group considers the same meta-test examples.
We argue that this gap indeed results from meta-domain shifts (cf. section 2.3) and might be resolved via domain adaptation by casting meta-learning as supervised learning.