M ETA -D ATASET : A D ATASET OF D ATASETS FOR L EARNING TO L EARN FROM F EW E XAMPLES
eleni@cs.toronto.edu
A BSTRACT
Few-shot classiﬁcation refers to learning a classiﬁer for new classes given only a few examples.
While a plethora of models have emerged to tackle it, we ﬁnd the procedure and datasets that are used to assess their progress lacking.
To address this limitation, we propose M ETA -D ATASET : a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks.
We experiment with popular baselines and meta-learners on M ETA -D ATASET , along with a competitive method that we propose.
We analyze performance as a function of various characteristics of test tasks and examine the models’ ability to leverage diverse training sources for improving their generalization.
We also propose a new set of baselines for quantifying the beneﬁt of meta-learning in M ETA -D ATASET .
Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.
I NTRODUCTION
Few-shot learning refers to learning new concepts from few examples, an ability that humans naturally possess, but machines still lack.
Improving on this aspect would lead to more efﬁcient algorithms that can ﬂexibly expand their knowledge without requiring large labeled datasets.
We focus on few-shot classiﬁcation: classifying unseen examples into one of N new ‘test’ classes, given only a few reference examples of each.
Recent progress in this direction has been made by considering a meta-problem: though we are not interested in learning about any training class in particular, we can exploit the training classes for the purpose of learning to learn new classes from few examples , thus acquiring a learning procedure that can be directly applied to new few-shot learning problems too.
This intuition has inspired numerous models of increasing complexity (see Related Work for some examples).
However, we believe that the commonly-used setup for measuring success in this direction is lacking.
Speciﬁcally, two datasets have emerged as de facto benchmarks for few-shot learning:
Omniglot (Lake et al., 2015), and mini -ImageNet (Vinyals et al., 2016), and we believe that both of them are approaching their limit in terms of allowing one to discriminate between the merits of different approaches.
Omniglot is a dataset of 1623 handwritten characters from 50 different alphabets and contains 20 examples per class (character).
Most recent methods obtain very high accuracy on Omniglot, rendering the comparisons between them mostly uninformative.
mini -ImageNet is formed out of 100 ImageNet (Russakovsky et al., 2015) classes (64/16/20 for train/validation/test) and contains 600 examples per class.
Albeit harder than Omniglot, it has the same property that most recent methods trained on it present similar accuracy when controlling for model capacity.
We advocate that a more challenging and realistic benchmark is required for further progress in this area.
More speciﬁcally, current benchmarks: 1) Consider homogeneous learning tasks.
In contrast, real-life learning experiences are heterogeneous: they vary in terms of the number of classes and examples per class, and are unbalanced.
2) Measure only within-dataset generalization.
However, we are eventually after models that can generalize to entirely new distributions (e.g., datasets).
3) Ignore the relationships between classes when forming episodes.
Speciﬁcally, the coarse-grained classiﬁcation of dogs and chairs may present different difﬁculties than the ﬁne-grained classiﬁcation of dog breeds, and current benchmarks do not establish a distinction between the two.
M ETA -D ATASET aims to improve upon previous benchmarks in the above directions: it is signiﬁcantly larger-scale and is comprised of multiple datasets of diverse data distributions; its task creation is informed by class structure for ImageNet and Omniglot; it introduces realistic class imbalance; and it varies the number of classes in each task and the size of the training set, thus testing the robustness of models across the spectrum from very-low-shot learning onwards.
The main contributions of this work are: 1) A more realistic, large-scale and diverse environment for training and testing few-shot learners.
Experimental evaluation of popular models, and a new set of baselines combining inference algorithms of meta-learners with non-episodic training.
Analyses of whether different models beneﬁt from more data, heterogeneous training sources, pre-trained weights, and meta-training.
A novel meta-learner that performs strongly on M ETA -D ATASET .
F EW - SHOT C LASSIFICATION : T ASK F ORMULATION AND A PPROACHES
Task Formulation
The end-goal of few-shot classiﬁcation is to produce a model which, given a new learning episode with N classes and a few labeled examples ( k c per class, c ∈ 1 , . . .
, N ), is able to generalize to unseen examples for that episode.
In other words, the model learns from a training (support) set S
=
{ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
, ( x K , y K ) } (with K = (cid:80) c k c ) and is evaluated on a held-out test (query) set Q = { ( x ∗ 1 , y ∗ 1 ) , ( x ∗ 2 , y ∗ 2 ) , . .
, ( x ∗ T , y ∗ T ) } .
Each example ( x , y ) is formed of an input vector x ∈ R D and a class label y ∈ { 1 , . . .
, N } .
Episodes with balanced training sets (i.e., k c = k, ∀ c ) are usually described as ‘ N -way, k -shot’ episodes.
Evaluation episodes are constructed by sampling their N classes from a larger set C test of classes and sampling the desired number of examples per class.
A disjoint set C train of classes is available to train the model; note that this notion of training is distinct from the training that occurs within a few-shot learning episode.
Few-shot learning does not prescribe a speciﬁc procedure for exploiting C train , but a common approach matches the conditions in which the model is trained and evaluated (Vinyals et al., 2016).
In other words, training often (but not always) proceeds in an episodic fashion.
Some authors use training and testing to refer to what happens within any given episode, and meta-training and meta-testing to refer to using C train to turn the model into a learner capable of fast adaptation and C test for evaluating its success to learn using few shots, respectively.
This nomenclature highlights the meta-learning perspective alluded to earlier, but to avoid confusion we will adopt another common nomenclature and refer to the training and test sets of an episode as the support and query sets and to the process of learning from C train simply as training.
We use the term ‘meta-learner’ to describe a model that is trained episodically, i.e., learns to learn across multiple tasks that are sampled from the training set C train .
Non-episodic Approaches to Few-shot Classiﬁcation A natural non-episodic approach simply trains a classiﬁer over all of the training classes C train at once, which can be parameterized by a neural network with a linear layer on top with one output unit per class.
After training, this neural network is used as an embedding function g that maps images into a meaningful representation space.
The hope of using this model for few-shot learning is that this representation space is useful even for examples of classes that were not included in training.
It would then remain to deﬁne an algorithm for performing few-shot classiﬁcation on top of these representations of the images of a task.
We consider two choices for this algorithm, yielding the ‘ k -NN’ and ‘Finetune’ variants of this baseline.
Given a test episode, the ‘ k -NN’ baseline classiﬁes each query example as the class that its ‘closest’ support example belongs to.
Closeness is measured by either Euclidean or cosine distance in the learned embedding space; a choice that we treat as a hyperparameter.
On the other hand, the ‘Finetune’ baseline uses the support set of the given test episode to train a new ‘output layer’ on top of the embeddings g , and optionally ﬁnetune those embedding too (another hyperparameter), for the purpose of classifying between the N new classes of the associated task.
A variant of the ‘Finetune’ baseline has recently become popular: Baseline++ (Chen et al., 2019), originally inspired by Gidaris & Komodakis (2018);
Qi et al.
It uses a ‘cosine classiﬁer’ as the ﬁnal layer ( (cid:96) 2 -normalizing embeddings and weights before taking the dot product), both during the non-episodic training phase, and for evaluation on test episodes.
We incorporate this idea in our codebase by adding a hyperparameter that optionally enables using a cosine classiﬁer for the ‘ k -NN’ (training only) and ‘Finetune’ (both phases) baselines.
Meta-Learners for Few-shot Classiﬁcation In the episodic setting, models are trained end-to-end for the purpose of learning to build classiﬁers from a few examples.
We choose to experiment with Matching Networks (Vinyals et al., 2016), Relation Networks (Sung et al., 2018), Prototypical
Networks (Snell et al., 2017) and Model Agnostic Meta-Learning (MAML, Finn et al., 2017) since they cover a diverse set of approaches to few-shot learning.
We also introduce a novel meta-learner which is inspired by the last two models.
In each training episode, episodic models compute for each query example x ∗ ∈ Q , the distribution for its label p ( y ∗ | x ∗ , S ) conditioned on the support set S and allow to train this differentiablyparameterized conditional distribution end-to-end via gradient descent.
The different models are distinguished by the manner in which this conditioning on the support set is realized.
In all cases, the performance on the query set drives the update of the meta-learner’s weights, which include (and sometimes consist only of) the embedding weights.
We brieﬂy describe each method below.
Prototypical Networks Prototypical Networks construct a prototype for each class and then classify each query example as the class whose prototype is ‘nearest’ to it under Euclidean distance.
More concretely, the probability that a query example x ∗ belongs to class k is deﬁned as:
[Equation]
where c k is the ‘prototype’ for class k : the average of the embeddings of class k ’s support examples.
Matching Networks Matching Networks (in their simplest form) label each query example as a (cosine) distance-weighted linear combination of the support labels:
[Equation]
where 1 A is the indicator function and α ( x ∗ , x i ) is the cosine similarity between g ( x ∗ ) and g ( x i ) , softmax-normalized over all support examples x i , where 1 ≤ i ≤ |S| .
Relation Networks Relation Networks are comprised of an embedding function g as usual, and a ‘relation module’ parameterized by some additional neural network layers.
They ﬁrst embed each support and query using g and create a prototype p c for each class c by averaging its support embeddings.
Each prototype p c is concatenated with each embedded query and fed through the relation module which outputs a number in [0 , 1] representing the predicted probability that that query belongs to class c .
The query loss is then deﬁned as the mean square error of that prediction compared to the (binary) ground truth.
Both g and the relation module are trained to minimize this loss.
MAML MAML uses a linear layer parametrized by W and b on top of the embedding function g ( · ; θ ) and classiﬁes a query example as
[Equation]
where the output layer parameters W (cid:48) and b (cid:48) and the embedding function parameters θ (cid:48) are obtained by performing a small number of within-episode training steps on the support set S , starting from initial parameter values ( b , W , θ ) .
The model is trained by backpropagating the query set loss through the within-episode gradient descent procedure and into ( b , W , θ ) .
This normally requires computing second-order gradients, which can be expensive to obtain (both in terms of time and memory).
For this reason, an approximation is often used whereby gradients of the within-episode descent steps are ignored.
This variant is referred to as ﬁrst-order MAML (fo-MAML) and was used in our experiments.
We did attempt to use the full-order version, but found it to be impractically expensive (e.g., it caused frequent out-of-memory problems).
Moreover, since in our setting the number of ways varies between episodes, b , W are set to zero and are not trained (i.e., b (cid:48) , W (cid:48) are the result of within-episode gradient descent initialized at 0), leaving only θ to be trained.
In other words, MAML focuses on learning the within-episode initialization θ of the embedding network so that it can be rapidly adapted for a new task.
Introducing Proto-MAML We introduce a novel meta-learner that combines the complementary strengths of Prototypical Networks and MAML: the former’s simple inductive bias that is evidently effective for very-few-shot learning, and the latter’s ﬂexible adaptation mechanism.
As explained by Snell et al.
(2017), Prototypical Networks can be re-interpreted as a linear classiﬁer applied to a learned representation g ( x ) .
The use of a squared Euclidean distance means that output logits are expressed as
[Equation]
where constant is a class-independent scalar which can be ignored, as it leaves output probabilities unchanged.
The k -th unit of the equivalent linear layer therefore has weights W k, · = 2 c k and biases b k
= −|| c k || 2 , which are both differentiable with respect to θ as they are a function of g ( · ; θ ) .
We refer to (fo-)Proto-MAML as the (fo-)MAML model where the task-speciﬁc linear layer of each episode is initialized from the Prototypical Network-equivalent weights and bias deﬁned above and subsequently optimized as usual on the given support set.
When computing the update for θ , we allow gradients to ﬂow through the Prototypical Network-equivalent linear layer initialization.
We show that this simple modiﬁcation signiﬁcantly helps the optimization of this model and outperforms vanilla fo-MAML by a large margin on M ETA -D ATASET .
M ETA -D ATASET : A N EW F EW -S HOT C LASSIFICATION B ENCHMARK
M ETA -D ATASET aims to offer an environment for measuring progress in realistic few-shot classiﬁcation tasks.
Our approach is twofold: 1) changing the data and 2) changing the formulation of the task (i.e., how episodes are generated).
The following sections describe these modiﬁcations in detail.
The code is open source and publicly available 1 .
M ETA -D ATASET ’s data is much larger in size than any previous benchmark, and is comprised of multiple existing datasets .
This invites research into how diverse sources of data can be exploited by a meta-learner, and allows us to evaluate a more challenging generalization problem, to new datasets altogether.
Speciﬁcally, M ETA -D ATASET leverages data from the following 10 datasets:
ILSVRC-2012 (ImageNet, Russakovsky et al., 2015), Omniglot (Lake et al., 2015), Aircraft (Maji et al., 2013), CUB-200-2011 (Birds, Wah et al., 2011), Describable Textures (Cimpoi et al., 2014),
Quick Draw (Jongejan et al., 2016), Fungi (Schroeder & Cui, 2018), VGG Flower (Nilsback &
Zisserman, 2008), Trafﬁc Signs (Houben et al., 2013) and MSCOCO (Lin et al., 2014).
These datasets were chosen because they are free and easy to obtain, span a variety of visual concepts (natural and human-made) and vary in how ﬁne-grained the class deﬁnition is.
More information about each of these datasets is provided in the Appendix.
To ensure that episodes correspond to realistic classiﬁcation problems, each episode generated in
M ETA -D ATASET uses classes from a single dataset.
Moreover, two of these datasets, Trafﬁc Signs and MSCOCO, are fully reserved for evaluation, meaning that no classes from them participate in the training set.
The remaining ones contribute some classes to each of the training, validation and test splits of classes, roughly with 70% / 15% / 15% proportions.
Two of these datasets, ImageNet and Omniglot, possess a class hierarchy that we exploit in M ETA -D ATASET .
For each dataset, the composition of splits is available online 2 .
ImageNet ImageNet is comprised of 82,115 ‘synsets’, i.e., concepts of the WordNet ontology, and it provides ‘is-a’ relationships for its synsets, thus deﬁning a DAG over them.
M ETA -D ATASET uses the 1K synsets that were chosen for the ILSVRC 2012 classiﬁcation challenge and deﬁnes a new class split for it and a novel procedure for sampling classes from it for episode creation, both informed by its class hierarchy.
Speciﬁcally, we construct a sub-graph of the overall DAG whose leaves are the 1K classes of ILSVRC2012.
We then ‘cut’ this sub-graph into three pieces, for the training, validation, and test splits, such that there is no overlap between the leaves of any of these pieces.
For this, we selected the synsets ‘carnivore’ and ‘device’ as the roots of the validation and test sub-graphs, respectively.
The leaves that are reachable from ‘carnivore’ and ‘device’ form the sets of the validation and test classes, respectively.
All of the remaining leaves constitute the training classes.
This method of splitting ensures that the training classes are semantically different from the test classes.
We end up withtraining, 158 validation and 130 test classes, roughly adhering to the standard 70 / 15 / 15 (%) splits.
Omniglot This dataset is one of the established benchmarks for few-shot classiﬁcation as mentioned earlier.
However, contrary to the common setup that ﬂattens and ignores its two-level hierarchy of alphabets and characters, we allow it to inﬂuence the episode class selection in M ETA -D ATASET , yielding ﬁner-grained tasks.
We also use the original splits proposed in Lake et al.
(2015): (all characters of) the ‘background’ and ‘evaluation’ alphabets are used for training and testing, respectively.
However, we reserve the 5 smallest alphabets from the ‘background’ set for validation.
3. E PISODE S AMPLING
In this section we outline M ETA -D ATASET ’s algorithm for sampling episodes, featuring hierarchicallyaware procedures for sampling classes of ImageNet and Omniglot, and an algorithm that yields realistically imbalanced episodes of variable shots and ways.
The steps for sampling an episode for a given split are: Step 0) uniformly sample a dataset D , Step 1) sample a set of classes C from the classes of D assigned to the requested split, and Step 2) sample support and query examples from C .
Step 1:
Sampling the episode’s class set This procedure differs depending on which dataset is chosen.
For datasets without a known class organization, we sample the ‘way’ uniformly from the range [5 , MAX-CLASSES ] , where MAX-CLASSES is either 50 or as many as there are available.
Then we sample ‘way’ many classes uniformly at random from the requested class split of the given dataset.
ImageNet and Omniglot use class-structure-aware procedures outlined below.
ImageNet class sampling We adopt a hierarchy-aware sampling procedure: First, we sample an internal (non-leaf) node uniformly from the DAG of the given split.
The chosen set of classes is then the set of leaves spanned by that node (or a random subset of it, if more than 50).
We prevent nodes that are too close to the root to be selected as the internal node, as explained in more detail in the
Appendix.
This procedure enables the creation of tasks of varying degrees of ﬁne-grainedness: the larger the height of the internal node, the more coarse-grained the resulting episode.
Omniglot class sampling We sample classes from Omniglot by ﬁrst sampling an alphabet uniformly at random from the chosen split of alphabets (train, validation or test).
Then, the ‘way’ of the episode is sampled uniformly at random using the same restrictions as for the rest of the datasets, but taking care not to sample a larger number than the number of characters that belong to the chosen alphabet.
Finally, the prescribed number of characters of that alphabet are randomly sampled.
This ensures that each episode presents a within-alphabet ﬁne-grained classiﬁcation.
Step 2: Sampling the episode’s examples Having already selected a set of classes, the choice of the examples from them that will populate an episode can be broken down into three steps.
We provide a high-level description here and elaborate in the Appendix with the accompanying formulas.
Step 2a: Compute the query set size The query set is class-balanced, reﬂecting the fact that we care equally to perform well on all classes of an episode.
The number of query images per class is set to a number such that all chosen classes have enough images to contribute that number and still remain with roughly half on their images to possibly add to the support set (in a later step).
This number is capped to 10 images per class.
Step 2b:
Compute the support set size We allow each chosen class to contribute to the support set at most 100 of its remaining examples (i.e., excluding the ones added to the query set).
We multiply this remaining number by a scalar sampled uniformly from the interval (0 , 1] to enable the potential generation of ‘few-shot’ episodes even when multiple images are available, as we are also interested in studying that end of the spectrum.
We do enforce, however, that each chosen class has a budget for at least one image in the support set, and we cap the total support set size to 500 examples.
Step 2c: Compute the shot of each class We now discuss how to distribute the total support set size chosen above across the participating classes.
The un-normalized proportion of the support set that will be occupied by a given chosen class is a noisy version of the total number of images of that class in the dataset.
This design choice is made in the hopes of obtaining realistic class ratios, under the hypothesis that the dataset class statistics are a reasonable approximation of the real-world statistics of appearances of the corresponding classes.
We ensure that each class has at least one image in the support set and distribute the rest according to the above rule.
After these steps, we complete the episode creation process by choosing the prescribed number of examples of each chosen class uniformly at random to populate the support and query sets.
R ELATED W ORK
In this work we evaluate four meta-learners on M ETA -D ATASET that we believe capture a good diversity of well-established models.
Evaluating other few-shot classiﬁers on M ETA -D ATASET is beyond the scope of this paper, but we discuss some additional related models below.
Similarly to MAML, some train a meta-learner for quick adaptation to new tasks (Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Rusu et al., 2019; Yoon et al., 2018).
Others relate to Prototypical
Networks by learning a representation on which differentiable training can be performed on some form of classiﬁer (Bertinetto et al., 2019; Gidaris & Komodakis, 2018; Oreshkin et al., 2018).
Others relate to Matching Networks in that they perform comparisons between pairs of support and query examples, using either a graph neural network (Satorras & Estrach, 2018) or an attention mechanism (Mishra et al., 2018).
Finally, some make use of memory-augmented recurrent networks (Santoro et al., 2016), some learn to perform data augmentation (Hariharan & Girshick, 2017; Wang et al., 2018) in a low-shot learning setting, and some learn to predict the parameters of a large-shot classiﬁer from the parameters learned in a few-shot setting (Wang & Hebert, 2016;
Wang et al., 2017).
Of relevance to Proto-MAML is MAML++ (Antoniou et al., 2019), which consists of a collection of adjustments to MAML, such as multiple meta-trained inner loop learning rates and derivative-order annealing.
Proto-MAML instead modiﬁes the output weight initialization scheme and could be combined with those adjustments.
Finally, M ETA -D ATASET relates to other recent image classiﬁcation benchmarks.
The CVPR 2017 Visual Domain Decathlon Challenge trains a model on 10 different datasets, many of which are included in our benchmark, and measures its ability to generalize to held-out examples for those same datasets but does not measure generalization to new classes (or datasets).
Hariharan & Girshick (2017) propose a benchmark where a model is given abundant data from certain base
ImageNet classes and is tested on few-shot learning novel ImageNet classes in a way that doesn’t compromise its knowledge of the base classes.
Wang et al.
(2018) build upon that benchmark and propose a new evaluation protocol for it.
Chen et al.
(2019) investigate ﬁne-grained few-shot classiﬁcation using the CUB dataset (Wah et al., 2011, also featured in our benchmark) and crossdomain transfer between mini -ImageNet and CUB.
Larger-scale few-shot classiﬁcation benchmarks were also proposed using CIFAR-100 (Krizhevsky et al., 2009; Bertinetto et al., 2019; Oreshkin et al., 2018), tiered -ImageNet
(Ren et al., 2018), and ImageNet-21k (Dhillon et al., 2019).
Compared to these, M ETA -D ATASET contains the largest set of diverse datasets in the context of few-shot learning and is additionally accompanied by an algorithm for creating learning scenarios from that data that we advocate are more realistic than the previous ones.
E XPERIMENTS Training procedure
M ETA -D ATASET does not prescribe a procedure for learning from the training data.
In these experiments, keeping with the spirit of matching training and testing conditions, we trained the meta-learners via training episodes sampled using the same algorithm as we used for
M ETA -D ATASET ’s evaluation episodes, described above.
The choice of the dataset from which to sample the next episode was random uniform.
The non-episodic baselines are trained to solve the large classiﬁcation problem that results from ‘concatenating’ the training classes of all datasets.
Validation Another design choice was to perform validation on (the validation split of)
ImageNet only, ignoring the validation sets of the other datasets.
The rationale behind this choice is that the [Table] performance on ImageNet has been known to be a good proxy for the performance on different datasets.
We used this validation performance to select our hyperparameters, including backbone architectures, image resolutions and model-speciﬁc ones.
We describe these further in the Appendix.
Pre-training We gave each meta-learner the opportunity to initialize its embedding function from the embedding weights to which the k -NN Baseline model trained on ImageNet converged to.
We treated the choice of starting from scratch or starting from this initialization as a hyperparameter.
For a fair comparison with the baselines, we allowed the non-episodic models to start from this initialization too.
This is especially important for the baselines in the case of training on all datasets since it offers the opportunity to start from ImageNet-pretrained weights.
Main results Table 1 displays the accuracy of each model on the test set of each dataset, after they were trained on ImageNet-only or all datasets.
Trafﬁc Signs and MSCOCO are not used for training in either case, as they are reserved for evaluation.
We propose to use the average (over the datasets) rank of each method as our metric for comparison, where smaller is better.
A method receives rankif it has the highest accuracy, rank 2 if it has the second highest, and so on.
If two models share the best accuracy, they both get rank 1.5, and so on.
We ﬁnd that fo-Proto-MAML is the top-performer according to this metric, Prototypical Networks also perform strongly, and the Finetune Baseline notably presents a worthy opponent 3 .
We include more detailed versions of these tables displaying conﬁdence intervals and per-dataset ranks in the Appendix.
Effect of training on all datasets instead of ImageNet only It’s interesting to examine whether training on (the training splits of) all datasets leads to improved generalizaton compared to training on (the training split of) ImageNet only.
Speciﬁcally, while we might expect that training on more data helps improve generalization, it is an empirical question whether that still holds for heterogeneous data.
We can examine this by comparing the performance of each model between the top and bottom sets of results of Table 1, corresponding to the two training sources (ImageNet only and all datasets, respectively).
For convenience, Figure 1 visualizes this difference in a barplot.
Notably, for Omniglot,
Quick Draw and Aircraft we observe a substantial increase across the board from training on all We [Figure] sources.
This is reasonable for datasets whose images are signiﬁcantly different from ImageNet’s: we indeed expect to gain a large beneﬁt from training on some images from (the training classes of) these datasets.
Interestingly though, on the remainder of the test sources, we don’t observe a gain from all-dataset training.
This result invites research into methods for exploiting heterogeneous data for generalization to unseen classes of diverse sources.
Our experiments show that learning ‘naively’ across the training datasets (e
[Figure]
g
[Figure]
, by picking the next dataset to use uniformly at random) does not automatically lead to that desired beneﬁt in most cases
[Figure]
Ways and shots analysis We further study the accuracy as a function of ‘ways’ (Figure 2a) and the class precision as a function of ‘shots’ (Figure 2b).
As expected, we found that the difﬁculty increases as the way increases, and performance degrades.
More examples per class, on the other hand, indeed make it easier to correctly classify that class.
Interestingly, though, not all models beneﬁt at the same rate from more data: Prototypical Networks and fo-Proto-MAML outshine other models in very-low-shot settings but saturate faster, whereas the Finetune baseline, Matching Networks, and fo-MAML improve at a higher rate when the shot increases.
We draw the same conclusions when performing this analysis on all datasets, and include those plots in the Appendix. As discussed in the
Appendix, we recommend including this analysis when reporting results on Meta-Dataset, aside from the main table.
The rationale is that we’re not only interested in performing well on average, but also in performing well under different speciﬁcations of test tasks.
Effect of pre-training In Figures 3a and 3b, we quantify how beneﬁcial it is to initialize the embedding network of meta-learners using the weights of the k -NN baseline pre-trained on ImageNet, as opposed to starting their episodic training from scratch.
We ﬁnd this procedure to often be beneﬁcial, both for ImageNet-only training and for training on all datasets. It seems that this ImageNet-inﬂuenced initialization drives the meta-learner towards a solution which yields increased performance on natural image test datasets, especially ILSVRC, Birds, Fungi, Flowers and MSCOCO. [Figure]
Perhaps unsusprisingly, though, it underperforms on signiﬁcantly different datasets such as Omniglot and Quick Draw. These ﬁndings show that, aside from the choice of the training data source(s) (e
[Figure] g., ImageNet only or all datasets, as discussed above), the choice of the initialization scheme can also inﬂuence to an important degree the ﬁnal solution and consequently the aptness of applying the resulting meta-learner to different data sources at test time.
Finally, an interesting observation is that MAML seems to beneﬁt the most from the pre-trained initialization, which may speak to the difﬁculty of optimization associated with that model.
Effect of meta-training We propose to disentangle the inference algorithm of each meta-learner from the fact that it is meta-learned, to assess the beneﬁt of meta-learning on M ETA -D ATASET .
To this end, we propose a new set of baselines: ‘Prototypical Networks Inference’, ‘Matching
Networks Inference’, and ‘fo-Proto-MAML Inference’, that are trained non-episodically but evaluated episodically (for validation and testing) using the inference algorithm of the respective meta-learner.
This is possible for these meta-learners as they don’t have any additional parameters aside from the embedding function that explicitly need to be learned episodically (as opposed to the relation module of Relation Networks, for example).
We compare each Inference-only method to its corresponding meta-learner in Figures 3c and 3d.
We ﬁnd that these baselines are strong: when training on ImageNet only, we can usually observe a small beneﬁt from meta-learning the embedding weights but this beneﬁt often disappears when training on all datasets, in which case meta-learning sometimes actually hurts.
We ﬁnd this result very interesting
and we believe it emphasizes the need for research on how to meta-learn across multiple diverse sources, an important challenge that M ETA -D ATASET puts forth.
Fine-grainedness analysis We use ILVRC-2012 to investigate the hypothesis that ﬁner-grained tasks are harder than coarse-grained ones.
Our ﬁndings suggest that while the test sub-graph is not rich enough to exhibit any trend, the performance on the train sub-graph does seem to agree with this hypothesis.
We include the experimental setup and results for this analysis in the Appendix.
C ONCLUSION
We have introduced a new large-scale, diverse, and realistic environment for few-shot classiﬁcation.
We believe that our exploration of various models on M ETA -D ATASET has uncovered interesting directions for future work pertaining to meta-learning across heterogeneous data: it remains unclear what is the best strategy for creating training episodes, the most appropriate validation creation and the most appropriate initialization.
Current models don’t always improve when trained on multiple sources and meta-learning is not always beneﬁcial across datasets.
Current models are also not robust to the amount of data in test episodes, each excelling in a different part of the spectrum.
We believe that addressing these shortcomings consitutes an important research goal moving forward.
A UTHOR C ONTRIBUTIONS
Eleni, Hugo, and Kevin came up with the benchmark idea and requirements. Eleni developed the core of the project, and worked on the experiment design and management with Tyler and Kevin, as well as experiment analysis. Carles, Ross, Kelvin, Pascal, Vincent, and Tyler helped extend the benchmark by adding datasets.
Eleni, Vincent, and Utku contributed the Prototypical Networks, Matching Networks, and Relation Networks implementations, respectively.
Tyler implemented baselines, MAML (with
Kevin) and Proto-MAML models, and updated the backbones to support them.
Writing was mostly led by Eleni, with contributions by Hugo, Vincent, and Kevin and help from Tyler and Pascal for visualizations.
Pascal and Pierre-Antoine worked on code organization, efﬁciency, and open-sourcing,
Pascal and Vincent optimized the efﬁciency of the data input pipeline.
Pierre-Antoine supervised the code development process and reviewed most of the changes, Hugo and Kevin supervised the overall direction of the research.
A CKNOWLEDGMENTS
We would like to thank Chelsea Finn for fruitful discussions and advice on tuning fo-MAML and ensuring the correctness of implementation, as well as Zack Nado and Dan Moldovan for the initial dataset code that was adapted.
Finally, we’d like to thank John Bronskill for suggesting that we experiment with a larger inner-loop learning rate for MAML which indeed signiﬁcantly improved our fo-MAML results on M ETA -D ATASET .
A PPENDIX . R ECOMMENDATION FOR REPORTING RESULTS ON M ETA -D ATASET
We recommend that future work on M ETA -D ATASET reports two sets of results:
The main tables storing the average (over 600 test episodes) accuracy of each method on each dataset, after it has been trained on ImageNet only and on All datasets, where the evaluation metric is the average rank.
This corresponds to Table 1 in our case (or the more complete version in Table 2 in the Appendix).
The plots that measure robustness in variations of shots and ways.
In our case these are Figures 2b and 2a in the main text for ImageNet-only training, and Figures 5b and 5a in the Appendix for the case of training on all datasets.
We propose to use both of these aspects to evaluate performance on M ETA -D ATASET : it is not only desirable to perform well on average, but also to perform well under different speciﬁcations of test tasks, as it is not realistic in general to assume that we will know in advance what setup (number of ways and shots) will be encountered at test time.
Our ﬁnal source code will include scripts for generating these plots and for automatically computing ranks given a table to help standardize the procedure for reporting results.
. D ETAILS OF M ETA -D ATASET ’ S S AMPLING A LGORITHM
We now provide a complete description of certain steps that were explained on a higher level in the main paper.
S TEP 1: S AMPLING THE EPISODE ’ S CLASS SET
ImageNet class sampling The procedure we use for sampling classes for an ImageNet episode is the following.
First, we sample a node uniformly at random from the set of ‘eligible’ nodes of the DAG structure corresponding to the speciﬁed split (train, validation or test).
An internal node is ‘eligible’ for this selection if it spans at least 5 leaves, but no more than 392 leaves.
The numberwas chosen because it is the smallest number so that, collectively, all eligible internal nodes span all leaves in the DAG.
Once an eligible node is selected, some of the leaves that it spans will constitute the classes of the episode.
Speciﬁcally, if the number of those leaves is no greater than 50, we use all of them.
Otherwise, we randomly choose 50 of them.
This procedure enables the creation of tasks of varying degrees of ﬁne-grainedness.
For instance, if the sampled internal node has a small height, the leaf classes that it spans will represent semanticallyrelated concepts, thus posing a ﬁne-grained classiﬁcation task.
As the height of the sampled node increases, we ‘zoom out’ to consider a broader scope from which we sample classes and the resulting episodes are more coarse-grained.
S TEP 2: S AMPLING THE EPISODE ’ S EXAMPLES
a) Computing the query set size The query set is class-balanced, reﬂecting the fact that we care equally to perform well on all classes of an episode.
The number of query images per class is computed as:
[Equation]
where C is the set of selected classes and Im ( c ) denotes the set of images belonging to class c .
The min over classes ensures that each class has at least q images to add to the query set, thus allowing it to be class-balanced.
The 0 .
5 multiplier ensures that enough images of each class will be available to add to the support set, and the minimum with 10 prevents the query set from being too large.
b) Computing the support set size We compute the total support set size as:
[Equation]
where β is a scalar sampled uniformly from interval (0 , 1] .
Intuitively, each class on average contributes either all its remaining examples (after placing q of them in the query set) if there are less than 100 or 100 otherwise, to avoid having too large support sets.
The multiplication with β enables the potential generation of smaller support sets even when multiple images are available, since we are also interested in examining the very-low-shot end of the spectrum.
The ‘ceiling’ operation ensures that each selected class will have at least one image in the support set.
Finally, we cap the total support set size to 500 .
c)
Computing the shot of each class We are now ready to compute the ‘shot’ of each class.
Speciﬁcally, the proportion of the support set that will be devoted to class c is computed as:
[Equation]
where α c is sampled uniformly from the interval [log(0 . 5) , log(2)) .
Intuitively, the un-normalized proportion of the support set that will be occupied by class c is a noisy version of the total number of images of that class in the dataset Im ( c ) .
This design choice is made in the hopes of obtaining realistic class ratios, under the hypothesis that the dataset class statistics are a reasonable approximation of the real-world statistics of appearances of the corresponding classes.
The shot of a class c is then set to:
[Equation]
which ensures that at least one example is selected for each class, with additional examples selected proportionally to R c , if enough are available.
. D ATASETS
M ETA -D ATASET is formed of data originating from 10 different image datasets.
A complete list of the datasets we use is the following.
(a) ImageNet (b) Omniglot (c) Aircraft (d) Birds (e) DTD
[Figure]
ILSVRC-2012 (ImageNet, Russakovsky et al., 2015)
A dataset of natural images fromcategories (Figure 4a).
We removed some images that were duplicates of images in another dataset in
M ETA -D ATASET (43 images that were also part of Birds) or other standard datasets of interest ( from Caltech-101 and 286 from Caltech-256).
The complete list of duplicates is part of the source code release.
Omniglot (Lake et al., 2015)
A dataset of images of 1623 handwritten characters from 50 different alphabets, with 20 examples per class (Figure 4b).
While recently Vinyals et al.
(2016) proposed a new split for this dataset, we instead make use of the original intended split Lake et al.
(2015) which is more challenging since the split is on the level of alphabets (30 training alphabets and 20 evaluation alphabets), not characters from those alphabets, therefore posing a more challenging generalization problem.
Out of the 30 training alphabets, we hold out the 5 smallest ones (i.e., with the least number of character classes) to form our validation set, and use the remaining 25 for training.
Aircraft (Maji et al., 2013)
A dataset of images of aircrafts spanning 102 model variants, with 100 images per class (Figure 4c).
The images are cropped according to the providing bounding boxes, in order not to include other aircrafts, or the copyright text at the bottom of images.
CUB-200-2011 (Birds, Wah et al., 2011)
A dataset for ﬁne-grained classiﬁcation of 200 different bird species (Figure 4d).
We did not use the provided bounding boxes to crop the images, instead the full images are used, which provides a harder challenge.
Describable Textures (DTD, Cimpoi et al., 2014)
A texture database, consisting of 5640 images, organized according to a list of 47 terms (categories) inspired from human perception (Figure 4e).
Quick Draw (Jongejan et al., 2016)
A dataset of 50 million black-and-white drawings acrosscategories, contributed by players of the game Quick, Draw!
(Figure 4f).
Fungi (Schroeder & Cui, 2018)
A large dataset of approximately 100K images of nearly 1, wild mushrooms species (Figure 4g).
VGG Flower (Nilsback & Zisserman, 2008)
A dataset of natural images of 102 ﬂower categories.
The ﬂowers chosen to be ones commonly occurring in the United Kingdom.
Each class consists of between 40 and 258 images (Figure 4h).
Trafﬁc Signs (Houben et al., 2013)
A dataset of 50,000 images of German road signs in 43 classes (Figure 4i).
MSCOCO Lin et al.
A dataset of images collected from Flickr with 1.5 million object instances belonging to 80 classes labelled and localized using bounding boxes.
We choose the train2017 split and create images crops from original images using each object instance’s groundtruth bounding box (Figure 4j).
. H YPERPARAMETERS
We used three architectures: a commonly-used four-layer convolutional network, an 18-layer residual network and a wide residual network.
While some of the baseline models performed best with the latter, we noticed that the meta-learners preferred the resnet-18 backbone and rarely the four-layerconvnet.
For Relation Networks only, we also allow the option to use another architecture, aside from the aforementioned three, inspired by the four-layer-convnet used in the Relation Networks paper (Sung et al., 2018).
The main difference is that they used the usual max-pooling operation only in the ﬁrst two layers, omitting it in the last two, yielding activations of larger spatial dimensions.
In our case, we found that these increased spatial dimensions did not ﬁt in memory, so as a compromise we used max-pooling on the ﬁrst 3 out of the 4 layer of the convnet.
For fo-MAML and fo-Proto-MAML, we tuned the inner-loop learning rate, the number of inner loop steps, and the number of additional such steps to be performed in evaluation (i.e., validation or test) episodes.
For the baselines, we tuned whether the cosine classiﬁer of Baseline++ will be used, as opposed to a standard forward pass through a linear classiﬁcation layer.
Also, since Chen et al.
(2019) added weight normalization (Salimans & Kingma, 2016) to their implementation of the cosine classiﬁer layer, we also implemented this and created a hyperparameter choice for whether or not it is enabled.
This hyperparameter is independent from the one that decides if the cosine classiﬁer is used.
Both are applicable to the k -NN Basline (for its all-way training classiﬁcation task) and to the
Finetune Baseline (both for its all-way training classiﬁcation and for its within-episode classiﬁcation at validation and test times).
For the Finetune Baseline, we tuned a binary hyperparameter deciding if gradient descent or ADAM is used for the within-task optimization.
We also tuned the decision of whether all embedding layers are ﬁnetuned or, alternatively, the embedding is held ﬁxed and only the ﬁnal classiﬁer on top of it is optimized.
Finally, we tuned the number of ﬁnetuning steps that will be carried out.
We also tried two different image resolutions: the commonly-used 84x84 and 126x126.
Finally, we tuned the learning rate schedule and weight decay and we used ADAM to train all of our models.
All other details, dataset splits and the complete set of best hyperparameters discovered for each model are included in the source code.
. C OMPLETE M AIN R ESULTS AND R ANK C OMPUTATION
Rank computation We rank models by decreasing order of accuracy and handle ties by assigning tied models the average of their ranks.
A tie between two models occurs when a 95% conﬁdence interval statistical test on the difference between their mean accuracies is inconclusive in rejecting the null hypothesis that this difference is 0.
We ended up not observing any such ties in the set of accuracies of our experiments, but our recommendation is that this test is ran to determine this.
As mentioned earlier, our source code will include this computation.
Complete main tables For completeness, Table 2 presents a more detailed version of Table 1 that also displays conﬁdence intervals and per-dataset ranks computed using the above procedure.
. A NALYSIS OF P ERFORMANCE A CROSS S HOTS AND W AYS
For completeness, in Figure 5 we show the results of the analysis of the robustness to different ways and shots for the variants of the models that were trained on all datasets.
We observe the same trends as discussed in our Experiments section for the variants of the models that were trained on ImageNet.
[Figure]
. E FFECT OF TRAINING ON ALL DATASETS OVER TRAINING ON ILSVRC-2012 ONLY
For more clearly observing whether training on all datasets leads to improved generalization over training on ImageNet only, Figure 6 shows side-to-side the performance of each model trained on
ILSVRC only vs. all datasets.
The difference between the performance of the all-dataset trained models versus the ImageNet-only trained ones is also visualized in Figure 1 in the main paper.
As discussed in the main paper, we notice that we do not always observe a clear generalization advantage in training from a wider collection of image datasets.
While some of the datasets that were added to the meta-training phase did see an improvement across all models, in particular for Omniglot and Quick Draw, this was not true across the board. In fact, in certain cases the performance is [Table] s
[Table]
± 1 .
[Table] [Table]
±
[Table] [Table] [Table]
±
[Table] [Table] [Table]
±
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
T e
[Table]
u
[Table]
[Table]
±
[Table]
[Table]
[Table]
±
[Table]
[Table]
[Table]
±
[Table]
[Table]
[Table]
c
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
ung
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
± 1 .
[Table]
[Table]
±
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
VGG
F l
[Table]
w e r
[Table]
±
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
± [Table] [Table] [Table]
T r a f ﬁ c S
i gn s 44 .± 1 .
19 ( 6 ) 66 .± 1 .
31 ( 1 ) 47 .± 1 .
14 ( 4 ) 47 .± 1 .
10 ( 5 ) 50 .± 1 .
[Table]
± 1 . [Table] ± [Table]
[Table] M S C
O C
O 30 .± 0 .
99 ( 6 ) 34 .± 0 . 97 ( 5 ) 34 .± 1 .
00 ( 4 ) 41 .± 1 .
10 ( 2 ) 35 .± 1 .
[Table] ± 1 . [Table]
± [Table] [Table]
A vg .
r a nk 5 .2 .4 .2 .3 .6 .1 .
( b )
M od e l s t r a
i n e d on a ll d a t a s e t s .
T e s t
S ou r ce
M e t hod :
A cc u r ac y ( % ) ± c on ﬁ d e n ce ( % ) (r a nk ) kNN
F i n e t un e
M a t c h i ng
N e t
P r o t o
N e t f oM A
M L
R e l a ti on
N e t
P r o t o - M
[Table]
M
[Table]
I L S
V
RC 38 .± 0 .
94 ( 4 ) 43 .± 1 .
08 ( 3 ) 36 .± 1 .
00 ( 6 ) 44 .± 1 .
05 ( 2 ) 37 .± 1 .
01 ( 5 ) 30 .± 0 .
[Table] [Table] ± 1 .
O m n i g l o t 74 .± 1 .
08 ( 6 ) 71 .± 1 .
37 ( 7 ) 78 .± 1 .
01 ( 5 ) 79 .± 1 .
12 ( 4 ) 83 .± 0 .
95 ( 2 ) 86 .± 0 .
79 ( 1 ) 82 .± 0 .
[Table]
A i r c r a f t 64 .± 0 .
82 ( 7 ) 72 .± 1 .
07 ( 3 ) 69 .± 0 .
96 ( 6 ) 71 .± 0 .
86 ( 4 ) 76 .± 0 .
69 ( 1 ) 69 .± 0 .
[Table] [Table]
± 0 .
[Table]
B i r d s 66 .± 0 .
92 ( 3 ) 59 .± 1 .
15 ( 5 ) 56 .± 1 .
00 ( 6 ) 67 .± 1 .
02 ( 2 ) 62 .± 1 .
08 ( 4 ) 54 .± 0 .
99 ( 7 ) 69 .± 1 .
T e x t u r e s 63 .± 0 .
79 ( 5 ) 69 .± 0 .
85 ( 1 ) 61 .± 0 .
74 ( 6 ) 65 .± 0 .
84 ( 3 ) 64 .± 0 . 83 ( 4 ) 56 .± 0 .
[Table] ± 0 .
[Table]
Q u i c k D r a w 44 .± 1 .
05 ( 7 ) 47 .± 1 .
16 ( 6 ) 60 .± 1 .
03 ( 4 ) 64 .± 0 .
89 ( 2 ) 59 .± 1 .
10 ( 5 ) 61 .± 0 .
97 ( 3 ) 66 .± 0 .
F ung i 37 .± 1 .
06 ( 4 ) 38 .± 1 .
04 ( 3 ) 33 .± 1 . 04 ( 5 ) 40 .± 1 .
13 ( 2 ) 33 .± 1 .
11 ( 6 ) 32 .± 1 .
08 ( 7 ) 41 .± 1 .
VGG
F l o w e r 83 .± 0 .
61 ( 4 ) 85 .± 0 .
69 ( 3 ) 81 .± 0 . 72 ( 5 ) 86 .± 0 .
71 ( 2 ) 79 .± 0 . 84 ( 6 ) 76 .± 0 .
76 ( 7 ) 88 .± 0 .
[Table]
T r a f ﬁ c S
i gn s 40 .±
10 ( 6 ) 66 .± 1 .
23 ( 1 ) 55 .± 1 .
08 ( 2 ) 46 .± 1 . 00 ( 4 ) 42 .± 1 .
31 ( 5 ) 37 .± 0 .
93 ( 7 ) 52 .± 1 .
M S C
O C
O 29 .± 0 . 96 ( 4 ) 35 .± 1 . 08 ( 3 ) 28 .± 0 . 96 ( 6 ) 39 .± 1 .
06 ( 2 ) 29 .± 1 . 08 ( 5 ) 27 .± 0 . 89 ( 7 ) 41 .± 1 .
[Table]
A vg . r a nk 3 .5 .2 .4 .5 .1 . ILSVRC Omniglot
Aircraft
CU Birds Textures Model k-NN baseline Finetune baseline
MatchingNet
ProtoNet fo-MAML RelationNet fo-Proto-MAML
Quick Draw
Fungi VGG Flower
Traffic Sign
MSCOCO
Train Source
ILSVRCAll datasets slightly worse.
We believe that more successfully leveraging diverse sources of data is an interesting open research problem.
. E FFECT OF PRE - TRAINING VERSUS TRAINING FROM SCRATCH
For each meta-learner, we selected the best model (based on validation on ImageNet’s validation split) out of the ones that used the pre-trained initialization, and the best out of the ones that trained from scratch.
We then ran the evaluation of each on (the test split of) all datasets in order to quantify how beneﬁcial this pre-trained initialization is.
We performed this experiment twice: for the models that are trained on ImageNet only and for the models that are trained on (the training splits of) all datasets.
The results of this investigation were reported in the main paper in Figure 3a and Figure 3b, for
ImageNet-only training and all dataset training, respectively.
We show the same results in Figure 7, printed larger to facilitate viewing of error bars.
For easier comparison, we also plot the difference in performance of the models that were pre-trained over the ones that weren’t, in Figures 8a and 8b.
These ﬁgures make it easier to spot that while using the pre-trained solution usually helps for datasets that are visually not too different from ImageNet, it may hurt for datasets that are signiﬁcantly different from it, such as Omniglot, Quickdraw (and surprisingly Aircraft).
Note that these three datasets are the same three that we found beneﬁt from training on All datasets instead of ImageNet-only.
It appears that using the pre-trained solution biases the ﬁnal solution to specialize on ImageNet-like datasets.
. E FFECT OF META - LEARNING VERSUS INFERENCE - ONLY
Figure 9 shows the same plots as in Figures 3c and 3d but printed larger to facilitate viewing of error bars.
Furthermore, as we have done for visualizing the observed gain of pre-training, we also present in Figures 10a and 10b the gain observed from meta-learning as opposed to training the corresponding inference-only baseline, as explained in the Experiments section of the main paper.
This visulization makes it clear that while meta-training usually helps on ImageNet (or doesn’t hurt too much), it sometimes hurts when it is performed on all datasets, emphasizing the need for further research into best practices of meta-learning across heterogeneous sources.
. F INEGRAINEDNESS A NALYSIS
We investigate the hypothesis that ﬁner-grained tasks are more challenging than coarse-grained ones by creating binary ImageNet episodes with the two classes chosen uniformly at random from the
DAG’s set of leaves.
We then deﬁne the degree of coarse-grainedness of a task as the height of the lowest common ancestor of the two chosen leaves, where the height is deﬁned as the length of the longest path from the lowest common ancestor to one of the selected leaves. Larger heights
[Figure]
[Figure] [Figure] [Figure]
[Figure] [Figure] [Table]
[Table]
then correspond to coarser-grained tasks.
We present these results in Figure 11.
We do not detect a signiﬁcant trend when performing this analysis on the test DAG.
The results on the training DAG, though, do seem to indicate that our hypothesis holds to some extent.
We conjecture that this may be due to the richer structure of the training DAG, but we encourage further investigation.
. T HE IMPORTANCE OF MAML’ S INNER - LOOP LEARNING RATE HYPERPARAMETER .
The camera-ready version includes updated results for MAML and Proto-MAML following an external suggestion to experiment with larger values for the inner-loop learning rate α of MAML. We found that re-doing our hyperparameter search with a revised range that includes larger α values signiﬁcantly improved fo-MAML’s performance on M ETA -D ATASET . For consistency, we applied the same change to fo-Proto-MAML and re-ran those experiments too.
We found that the value of this α that performs best for fo-MAML both for training on ImageNet only and training on all datasets is approximately 0.1, which is an order of magnitude larger than our previous best value.
Interestingly, fo-Proto-MAML does not choose such a large α value, with best α being 0.0054 when training on ImageNet only and 0.02 when training on all datasets.
Plausibly this difference can be attributed to the better initialization of Proto-MAML which requires a less aggressive optimization for the adaptation to each new task.
This hypothesis is also supported by the fact that fo-Proto-MAML chooses to take fewer adaptation steps than fo-MAML does.
The complete set of best discovered hyperparameters is available in our public code.
To emphasize the importance of properly tuning this hyperparameter, Table 3 displays the previous best and the new best fo-MAML results side-by-side, showcasing the large performance gap when using the appropriate value for α .
. T HE CHOICE OF A META - VALIDATION PROCEDURE FOR M ETA -D ATASET
The design choice we made, as discussed in the main paper, is to use (the meta-validation set of)
ImageNet only for model selection in all of our experiments.
In the absence of previous results on the topic, this is a reasonable strategy since ImageNet has been known to consitute a useful proxy for performance on other datasets.
However, it is likely that this is not the optimal choice: there might be certain hyperparameters for a given model that work best for held-out ImageNet episodes, but not for held-out episodes of other datasets.
An alternative meta-validation scheme would be to use the average (across datasets) validation accuracy as an indicator for early stopping and model selection.
We did not choose this method due to concerns about the reliability of this average performance.
Notably, taking a simple average would over-emphasize larger datasets, or might over-emphasize datasets with natural images (as opposed to Omniglot and Quickdraw).
Nevertheless, whether this strategy is beneﬁcial is an interesting empirical question.
. A DDITIONAL P ER - DATASET A NALYSIS OF S HOTS AND W AYS
In our previous analysis of performance across different shots and ways (Figures 2b, 2a, and 5), the performance is averaged over all evaluation datasets.
In this section we further break down those plots by presenting the results separately for each dataset.
Figures 12 and 13 show the analysis of performance as a function of ways and shots (respectively) for each evaluation dataset, for models that were trained on ImageNet only.
For completeness, Figures 14 and 15 show the same for the models trained on (the training splits of) all datasets. [Figure] [Figure] c i
[Figure]
i o n
[Figure]
r
[Figure]
i
[Figure]
i o
[Figure]
[Figure]
cc u r a c y
[Figure]
[Figure]
[Figure]
C l a ss P r e c i s i o n (a
[Figure]
ILSVRC
[Figure]
C l a ss P r e c i s i o n (b
Omniglot
[Figure]
[Table]
C l a
[Table]
P
[Table]
i
[Table]
i o n (
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
C l a ss P r e c i s i o n (
Birds
[Figure]
C l a ss P r e c i s i o n
Textures
[Figure]
C l a ss P r e c i s i o n (f 100 Shot 0. C l a ss P r e c i s i o n (g) Fungi
[Figure]
C l a ss P r e c i s i o n (
[Figure]
VGG Flower
[Figure] [Figure]
C l a ss P r e c i s i o n
[Figure]