H OW TO T RAIN Y OUR MAML TO E XCEL IN F EW -S HOT C LASSIFICATION
A BSTRACT
Model-agnostic meta-learning (MAML) is arguably one of the most popular metalearning algorithms nowadays.
Nevertheless, its performance on few-shot classiﬁcation is far behind many recent algorithms dedicated to the problem.
In this paper, we point out several key facets of how to train MAML to excel in few-shot classiﬁcation.
First, we ﬁnd that MAML needs a large number of gradient steps in its inner loop update, which contradicts its common usage in few-shot classiﬁcation.
Second, we ﬁnd that MAML is sensitive to the class label assignments during meta-testing.
Concretely, MAML meta-trains the initialization of an N -way classiﬁer.
These N ways, during meta-testing, then have “ N ! ”
different permutations to be paired with a few-shot task of N novel classes.
We ﬁnd that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classiﬁcation.
Third, we investigate several approaches to make MAML permutation-invariant, among which meta-training a single vector to initialize all the N weight vectors in the classiﬁcation head performs the best.
On benchmark datasets like Mini ImageNet and Tiered ImageNet, our approach, which we name UNICORN -MAML, performs on a par with or even outperforms many recent few-shot classiﬁcation algorithms, without sacriﬁcing MAML’s simplicity.
I NTRODUCTION
Meta-learning is a sub-ﬁeld of machine learning which attempts to search for the best learning strategy as the learning experiences increases (Thrun & Pratt, 2012;
Lemke et al., 2015).
Recent years have witnessed an abundance of new approaches on meta-learning (Vanschoren, 2018; Hospedales et al., 2020), among which model-agnostic meta-learning (MAML)
(Finn et al., 2017; Finn, 2018) is one of the most popular algorithms, owing to its “model-agnostic” nature to incorporate different model architectures and its principled formulation to be applied to different problems.
Concretely, MAML aims to learn a good model initialization (through the outer loop optimization), which can then be quickly adapted to novel tasks given few examples (through the inner loop optimization).
However, in few-shot classiﬁcation (Vinyals et al., 2016; Snell et al., 2017) which many meta-learning algorithms are dedicated to, MAML’s performance has been shown to fall far behind (Wang et al., 2019;
Chen et al., 2019; Triantaﬁllou et al., 2020).
In this paper, we take a closer look at MAML on few-shot classiﬁcation.
The standard setup involves two phases, meta-training and meta-testing, in which MAML learns the model initialization during meta-training and applies it during meta-testing.
In both phases, MAML receives multiple N -way Kshot tasks.
Each task is an N -class classiﬁcation problem provided with K labeled support examples per class.
After the (temporary) inner loop optimization using the support examples, the updated model from the initialization is then evaluated on the query examples of the same N classes.
The loss calculated on the query examples during mete-training is used to optimize the meta-parameters ( i.e. , the model initialization) through the outer loop.
For consistency, we mainly study the scenario where meta-training and meta-testing use the same number of gradient steps in the inner loop optimization.
More speciﬁcally, what MAML learns for few-shot classiﬁcation is the initialization of an N -class classiﬁer.
Without loss of generality, we denote a classiﬁer by ˆ y = arg max c ∈ [ N ] w (cid:62) c f φ ( x ) , where f φ is the feature extractor on an example x ; { w c } Nc =1 are the weight vectors in the linear classiﬁcation head.
We use θ to represent the collection of meta-parameters { φ , w 1 , · · · , w N } .
[Figure]
Our ﬁrst observation is that MAML needs a large number of gradient steps in the inner loop .
For example, on Mini ImageNet (Vinyals et al., 2016) and Tiered ImageNet (Ren et al., 2018a), MAML’s accuracy improves along with the increased number of gradient steps and achieves the highest around 15 ∼ 20 steps, which are much larger than the conventional usage of MAML (Antoniou et al., 2019).
We attribute this to the behavior of the model initialization learned from mutually-exclusive tasks (Yin et al., 2020), which, without any further inner loop optimization, performs at the chance level ( i.e. ,
N % accuracy) on query examples, not only for the meta-testing tasks but also for the meta-training tasks.
In other words, the initialized model needs many gradient steps to attain a high accuracy.
Our second observation is that MAML is sensitive to the permutations of class label assignments during meta-testing .
Concretely, when an N -way K -shot task arrives, MAML pairs the learned initialization of w c with the corresponding class label c ∈ [ N ] of that task.
The issue resides in the “meaning” of c in a task.
In the standard setup, each N -way task is created by drawing N classes from a bigger pool of semantic classes ( e.g. , “dog”, “cat”, “bird”, etc.), followed by a random label re-assignment into c ∈ { 1 , · · · , N } .
In other words, the same set of N semantic classes can be labeled totally differently into { 1 , · · · , N } and thus be paired with { w c } Nc =1 differently.
Taking a ﬁve-way task for example, there are 5!
= 120 permutations to pair the same set of ﬁve semantic classes to the linear classiﬁcation head.
In some of them, a class “dog” may be assigned to c = 1 ; in some others, it may be assigned to c (cid:54) = 1 .
While this randomness has been shown crucial in meta-training to help MAML prevent over-ﬁtting (Rajendran et al., 2020;
Yao et al., 2021;
Yin et al., 2020), we ﬁnd that it makes the meta-testing phase unstable.
Speciﬁcally, different permutations can lead to drastically different meta-testing accuracy — on average, the best permutation for each ﬁve-way one-shot task has ∼ 15% higher accuracy than the worst permutation, on both datasets.
Building upon this observation, we investigate multiple approaches to make MAML permutationinvariant , either in the meta-testing phase alone or in both phases.
We ﬁnd that a simple solution — meta-training only a single vector w and using it to initialize the N linear classiﬁers { w c } Nc = — performs the best.
We name this approach UNICORN -MAML, as illustrated in Figure 1 (b).
Concretely, at the beginning of the inner loop, UNICORN -MAML duplicates w into w c , ∀ c ∈ [ N ] .
After the inner loop optimization, the meta-gradient with respect to { w c } Nc =1 are aggregated to update w in the outer loop (during meta-training).
This design not only makes UNICORN -MAML permutation-invariant, but also ensures that no single model can solve all tasks at once without inner loop optimization, to prevent memorization over-ﬁtting (Yin et al., 2020).
On Mini ImageNet (Vinyals et al., 2016), Tiered ImageNet (Ren et al., 2018a), and CUB datasets (Wah et al., 2011), UNICORNMAML performs on a par with or even outperforms many recent few-shot classiﬁcation algorithms, while preserving the simplicity of MAML without adding any extra network modules or learning strategies. Our code is available at https://github.com/Han-Jia/UNICORN-MAML .
MAML FOR F EW -S HOT C LASSIFICATION 2. P ROBLEM DEFINITION
The goal of few-shot classiﬁcation is to construct a classiﬁer using limited labeled examples.
The challenge is the potential over-ﬁtting or poor generalization problem.
Following (Vinyals et al., 2016), we deﬁne a few-shot classiﬁcation problem as an N -way K -shot task, which has N classes and each class has K labeled support examples.
We denote the labeled support set by S =
{ ( x i , y i ) }
N × K i =1 ; each ( x i , y i ) is a pair of an input ( e.g. , image) and a class label, where y i ∈ { 1 , 2 , · · · , N } =
[ N ] .
The value of K is small, e.g. , K = 1 or K = 5 .
To evaluate the quality of the resulting classiﬁer, each task is associated with a query set Q , which is composed of examples of the same N classes.
The core idea of meta-learning for few-shot classiﬁcation is to sample few-shot tasks T =
( S , Q ) from a set of “base” classes, of which we have ample examples per class.
Meta-learning then learns the ability of “how to build a classiﬁer using limited data” from these tasks.
After this meta-training phase, we then proceed to the meta-testing phase to tackle the true few-shot tasks that are composed of examples from “novel” classes.
By default, the “novel” and “base” classes are disjoint.
It is worth noting that the total number of “base” (and “novel”) classes is usually larger than N (see subsection 2.3).
Thus, to construct an N -way K -shot task in each phase, one usually ﬁrst samples N classes from the corresponding set of classes, and randomly re-labels each sampled class by an index c ∈ [ N ] .
This randomness results in the so-called mutually-exclusive tasks (Yin et al., 2020).
In this paper, we will use base (novel) and meta-training (meta-testing) classes interchangeably.
2. M ODEL - AGNOSTIC META - LEARNING (MAML)
As introduced in section 1, MAML aims to learn the initialization of an N -way classiﬁer, such that when provided with the support set S of an N -way K -shot task, the classiﬁer can be quickly and robustly updated to perform well on the task ( i.e. , classify the query set Q well).
Let us denote a classiﬁer by ˆ y =
h θ ( x ) =
arg max c ∈ [ N ] w (cid:62) c
f φ ( x ) , where f φ is the feature extractor, { w c } Nc =1 are the weight vectors of the classiﬁcation head, and θ = { φ , w 1 , · · · , w N } collects the parameters of both.
MAML evaluates h θ on S and uses the gradient to update θ into θ (cid:48) , so that h θ (cid:48) can be applied to Q .
This procedure is called the inner loop optimization , which usually takes M gradient steps.
[Equation]
for m ∈ [ M ] do (1)
[Equation]
Here, L ( S , θ (cid:48) )
= (cid:80) ( x ,y ) ∈S (cid:96) ( h θ (cid:48)
( x ) , y ) is the loss computed on examples of S and α is the learning rate (or step size).
The cross-entropy loss is commonly used for (cid:96) .
As suggested in the original MAML paper (Finn et al., 2017) and (Antoniou et al., 2019), M is usually set to a small integer ( e.g. , ≤ 5 ).
For ease of notation, let us denote the output θ (cid:48) after M gradient steps by θ (cid:48)
=
InLoop ( S , θ , M ) .
To learn the initialization θ , MAML leverages the few-shot tasks sampled from the base classes.
Let us denote by p ( T ) the distribution of tasks from the base classes, where each task is a pair of support and query sets ( S , Q ) . MAML aims to minimize the following meta-training objective w.r.t.
θ :
[Equation]
Namely, MAML aims to ﬁnd a shared θ among tasks, which, after inner loop updates using S , can lead to a small loss on the query set Q .
(We add the subscript S to θ (cid:48) to show that θ (cid:48)S depends on S .)
To optimize Equation 2, MAML applies stochastic gradient descent (SGD) but at the task level.
That is, at every iteration, MAML samples a task T
=
( S , Q ) and computes the meta-gradient w.r.t.
θ :
[Equation]
In practice, one may sample a mini-batch of tasks and compute the mini-batch meta-gradient w.r.t.
θ to optimize θ .
This SGD for θ is known as the outer loop optimization for MAML.
It is worth noting that calculating the gradient in Equation 3 can impose considerable computational and memory burdens because it involves a gradient through a gradient (along the inner loop but in a backward order)
(Finn et al., 2017).
Thus in practice, it is common to apply the ﬁrst-order approximation (Finn et al., 2017; Nichol et al., 2018), i.e. , ∇ θ L ( Q , θ (cid:48)S ) ≈ ∇ θ (cid:48)S L ( Q , θ (cid:48)S ) .
For additional related work on meta-learning and few-shot learning, please see Appendix A.
2. E XPERIMENTAL SETUP
As our paper is heavily driven by empirical observations, we ﬁrst introduce the three main datasets we experiment on, the neural network architectures we use, and the implementation details.
Dataset.
We work on Mini ImageNet (Vinyals et al., 2016),
Tiered ImageNet (Ren et al., 2018a), and
CUB datasets (Wah et al., 2011).
Mini ImageNet contains 100 semantic classes; each has 600 images.
Following (Ravi & Larochelle, 2017), the 100 classes are split into meta-training/validation/testing sets with 64/16/20 (non-overlapped) classes, respectively.
That is, there are 64 base classes and 20 novel classes; the other 16 classes are used for hyper-parameter tuning.
Tiered ImageNet (Ren et al., 2018a) has 608 semantic classes, which are split into the three sets with 351/97/160 classes, respectively.
On average, each class has ∼ 1 , 300 images.
CUB (Wah et al., 2011) has 200 classes, which are split into the three sets with 200/50/50 classes following (Ye et al., 2020a).
All images are resized to 84 × 84 , following (Lee et al., 2019;
Ye et al., 2020a).
Training and evaluation.
During meta-training, meta-validation, and meta-testing, we sample Nway K -shot tasks from the corresponding classes and images.
We follow the literature (Snell et al., 2017; Vinyals et al., 2016) to study the ﬁve-way one-shot and ﬁve-way ﬁve-shot tasks.
As mentioned in subsection 2.1, every time we sample ﬁve distinct classes, we randomly assign each of them an index c ∈
[ N ] .
During meta-testing, we follow the evaluation protocol in (Zhang et al., 2020; Rusu et al., 2019;
Ye et al., 2020a) to sample 10 , 000 tasks.
In each task, the query set contains 15 images per class.
We report the mean accuracy (in %) and the 95% conﬁdence interval.
Model architecture.
We follow (Lee et al., 2019) to use a ResNet-12 (He et al., 2016) architecture for f φ (cf. subsection 2.2), which has wider widths and Dropblock modules (Ghiasi et al., 2018).
We note that many recent few-shot learning algorithms use this backbone.
We also follow the original
MAML (Finn et al., 2017) to use a 4-layer convolutional network (ConvNet) (Vinyals et al., 2016).
Implementation details.
Throughout the paper, for simplicity and consistency, we use
• the ﬁrst-order approximation for calculating the meta-gradient in the outer loop; • the same number of gradient steps in the inner loop during meta-training and meta-testing; • the weights pre-trained on the entire
meta training set to initialize φ , following the recent practice (Ye et al., 2020a; Rusu et al., 2019; Qiao et al., 2018).
We note that in meta-training we still optimize this pre-trained φ in the “outer” loop to search for a better initialization for MAML.
MAML has several hyper-parameters and we select them on the meta-validation set.
Speciﬁcally, for the outer loop, we learn with at most 10 , 000 tasks: we group every 100 tasks into an epoch.
We apply SGD with momentum 0 .
9 and weight decay 0 . 0005 .
We start with an outer loop learning rate 0 .
002 for ConvNet and 0 .
001 for ResNet-12, which are decayed by 0 .
5 and 0 .
1 after every 20 epochs for ConvNet and ResNet-12, respectively.
For the inner loop, we have to set the number of gradient steps M and the learning rate α (cf.
Equation 1).
We provide more details in the next section.
MAML N EEDS A L ARGE N UMBER OF I NNER L OOP G RADIENT S TEPS
We ﬁnd that for MAML’s inner loop, the number of gradient updates M (cf.
Equation 1) is usually selected within a small range close to 1 , e.g. , M ∈ [1 , 5] (Antoniou et al., 2019).
At ﬁrst glance, this makes sense according to the motivation of MAML (Finn et al., 2017) — with a small number of gradient steps, the resulting model will have a good generalization performance.
In our experiment, we however observe that it is crucial to explore a larger M 1 .
Speciﬁcally, we consider M ∈ [1 , 20] along with α ∈ [10 − 4 , 10 0 ] .
We plot the meta-testing accuracy of ﬁve-way one-shot tasks on the three datasets in Figure 2 2 , using both ResNet and ConvNet backbones.
We ﬁnd that MAML achieves higher and much more stable accuracy (w.r.t.
the learning rate) when M is larger, e.g. , larger than 10 .
Speciﬁcally, for Mini ImageNet with ResNet, the highest accuracy 64 .
42% is obtained with M = 15 , higher than 62 .
90% with M = 5 ; for Tiered ImageNet with ResNet, the highest accuracy 65 . 72% is obtained with M = 15 , higher than 59 .
08% with M = 5 .
As will be seen in section 6, these results with a larger M are already close to the state-of-the-art performance.
[Figure]
Mini [Figure] ResNet 65.
[Figure]
ImageNet 65.
1E-4 5E-
1E-3 5E-3 1ETiered ImageNet, ResNet
[Figure]
1E-3 5E-3 1E-2 5E-2 1ECUB
[Figure] [Figure]
[Figure]
[Figure] [Figure]
2 5E-2 1E4547
[Figure]
1E
[Figure]
1E-
[Figure]
CUB
[Figure] [Figure]
To analyze why MAML needs a large M
[Figure]
we plot the change of classiﬁcation accuracy along with the inner loop updates in Figure 3.
Speciﬁcally, we ﬁrst perform meta-training using the M value selected by meta-validation, for each pair of dataset and backbone.
We then analyze the learned model initialization on few-shot tasks sampled from the meta-training and meta-testing classes, by performing 0 ∼ 30 inner loop updates using the support set and reporting the accuracy on the query set.
We conduct the same experiments for ﬁve-way ﬁve-shot tasks in Appendix D
[Figure]
We have two observations.
First, in general, the more inner loop updates we perform for a few-shot task, the higher the accuracy is, no matter if it is a meta-training or meta-testing task.
This trend aligns with the few-shot regression study in (Finn et al., 2017).
Second, before any inner loop update, the learned initialization θ has a ∼ 20% accuracy on average, i.e. , the accuracy by random classiﬁcation.
Interestingly, this is the case not only for meta-testing tasks but also for meta-training tasks, even though the learned initialization does contain the classiﬁcation head { w c } Nc =1 .
This explains why a larger number of inner loop gradient steps is needed: the learned initialized model has to be updated from performing random predictions to achieving a much higher classiﬁcation accuracy .
We attribute the second observation to the random class label assignments in creating few-shot tasks (cf. subsection 2.1 and subsection 2.3), which make the created tasks mutually-exclusive (Yin et al., 2020) — i.e. , a single model cannot solve them all at once before inner loop optimization.
Concretely, for a few-shot task of a speciﬁc set of N semantic classes ( e.g. , { “dog”, “cat”, · · · ,“bird” } ), such a randomness can turn it into different tasks from MAML’s perspective.
For instance, the class “dog” may be assigned to c = 1 and paired with w 1 at the current task, but to c = 2 and paired with wwhen it is sampled again.
For a ﬁve-way task, the same set of ﬁve semantic classes can be assigned to { 1 , · · · , 5 } via 120 ( i.e. , 5! ) different permutations.
As a result, if we directly apply the learned initialization of MAML without inner loop updates, the accuracy on few-shot tasks of the same set of semantic classes (but in different permutations) can cancels each other out.
(Please see subsection C. for details.)
Besides, since the randomness occurs also in meta-training, each w c will be discouraged to learn speciﬁc knowledge towards any semantic class (Rajendran et al., 2020;
Yao et al., 2021;
Yin et al., 2020), hence producing an accuracy at the chance level even on meta-training tasks.
[Figure]
ImageNet
[Figure]
{+Shot    88   (b) Mini ImageNet, 5-Shot 57 60 63 66 69 
+}
(c) Tiered ImageNet, 1-Shot    88   (d) Tiered ImageNet, 5-Shot
Related work.
Several following-up works of MAML, e g. , (Hsu et al., 2019), use different numbers of inner loop steps in meta-training ( e.g. , M = 1 ∼ 5 ) and meta-testing ( e.g. , M = 20 ∼ 50 ).
We make M equal in the two phases for consistency and provide a detailed analysis of why MAML needs a large M value.
On large-scale tasks beyond few-shot learning, Shin et al.
(2021) also found the necessity of a larger number of inner loop steps, but from a different perspective than ours.
MAML IS S ENSITIVE TO T HE L ABEL P ERMUTATIONS IN M ETA -T ESTING
The randomness in class label assignments raises an interesting question: do different permutations result in different meta-testing accuracy after inner loop updates?
More speciﬁcally, if { w c } Nc =1 are paired with the N classes differently, will the updated model after the inner loop perform differently?
To answer this question, we conduct a detailed experiment: Algorithm 1 summarizes the procedure.
We focus on ﬁve-way one/ﬁve-shot tasks on Mini ImageNet and Tiered ImageNet, using the ResNet backbone.
For each task type and dataset combination, we ﬁrst meta-train the model initialization using MAML, and then evaluate the learned initialized on 2 , 000 meta-testing tasks.
For each task, there are 120 permutations; each permutation, after the inner loop, would likely lead to a different model and query set accuracy.
We sort the 120 accuracy for each task, and take average for each position in the sorted list over 2 , 000 tasks.
This results in 120 averaged accuracy, each for a speciﬁc position in the sorted list.
Speciﬁcally, the highest accuracy corresponds to the case that each task cherry-picks its best permutation according to the query set accuracy after inner loop optimization.
We show the histogram of the 120 average meta-testing accuracy in Figure 4.
There exists a huge variance.
Speciﬁcally, the best permutation can be 15% / 8% higher than the worst in one/ﬁve-shot tasks.
The best permutation is also much higher than vanilla MAML’s results (from section 3), which are 64.42%/83.44%/65.72%/84.37%, corresponding to the four sub-ﬁgures from left to right in Figure 4.
What’s more, the best permutation can easily achieve state-of-the-art accuracy (see section 6).
Of course, so far we ﬁnd the best permutation via cherry-picking — by looking at the meta-testing accuracy — so it is like an upper bound.
However, if we can ﬁnd the best permutation without looking at the query sets’ labels or make MAML permutation-invariant, we may practically improve MAML.
Related work.
We note that we investigate the effect of the permutations of class label assignments differently from (Rajendran et al., 2020;
Yao et al., 2021;
Yin et al., 2020).
There, the permutations lead to mutually-exclusive meta-training tasks, which help prevent MAML from over-ﬁtting.
Here, we look at the meta-testing tasks, and permutations result in a huge variance of meta-testing accuracy.
[Table] [Table]
[Table]
M AKING MAML P ERMUTATION -I NVARIANT DURING M ETA -T ESTING
We study approaches that can make MAML permutation-invariant during meta-testing.
That is, we take the same learned initialization θ as in section 4 without changing the meta-training phase.
We ﬁrst investigate searching for the best permutation for each task .
As we cannot access query sets’ labels, we use the support sets’ data as a proxy.
We choose the best permutation according to which permutation, either before or after inner loop updates (less practical), leads to the highest accuracy or smallest loss on the support set.
Table 1 summarizes the results: none of them leads to consistent gains.
We hypothesize two reasons.
First, due to mutually-exclusive tasks in meta-training, the learned θ by MAML would produce chance-level predictions before updates (see Appendix C).
Second, after updates, the support set accuracy quickly goes to 100% and is thus not informative.
In stead of choosing one from many, we further explore taking ensemble (Breiman, 1996; Zhou, 2012; Dietterich, 2000) over the predictions made by updated models of different permutations.
We note that this makes MAML permutation-invariant but inevitably needs more computations.
To make the ensemble process clear, we permute the weight vectors in { w c } Nc =1 rather than the class label assignments in a task: the two methods are equivalent but the former is easier for aggregating predictions.
We study two variants: (a) full permutations ( i.e. , 120 of them in ﬁve-way tasks), which is intractable for larger N ; (b) rotated permutations, which rotates the index c in { w c } Nc =1 3 , leading to N permutations.
Table 2 shows the results — ensemble consistently improves MAML.
Even with the rotated version that has much fewer permutations than the full version, the gains are comparable.
We emphasize that our focus here is to make MAML permutation-invariant in meta-testing, not to explore and compare all potential ways of performing ensemble on MAML.
We further explore an efﬁcient approach to make MAML permutation-invariant, which is to manipulate the learned initialization of { w c } Cc =1 .
Concretely, MAML is sensitive to the permutations in class assignments because w c (cid:54) =
w c (cid:48) for c (cid:54) =
c (cid:48) .
One method to overcome this is to make w c = w c (cid:48) during meta-testing.
Here, we investigate re-initializing each w c by their average in meta-testing: w c ← 1 N (cid:80) Nc (cid:48)
=1 w c (cid:48) .
By doing so, no matter which permutation we perform, the model after inner loop optimization will be the same and lead to the same query set accuracy.
Table 3 summarizes the results; this approach improves vanilla MAML (see Table 2) in three of the four cases.
At ﬁrst glance, this approach may not make sense since the resulting { w c } Cc =1 , before inner loop updates, are identical and simply make random predictions.
However, please note that even the original { w c } Cc =1 have an averaged chance accuracy (cf.
Figure 3).
In Appendix E, we provide an explanation of this approach by drawing an analogy to dropout (Srivastava et al., 2014).
In short, in meta-training, we receive a task with an arbitrary permutation, which can be seen as drawing a permutation at random for the task.
In meta-testing, we then take expectation over permutations, which essentially lead to the averaged w c .
[Table]
UNICORN -MAML: L EARNING A S INGLE W EIGHT V ECTOR
The experimental results in section 5 are promising: by making MAML permutation-invariant in meta-testing, we can potentially improve vanilla MAML.
While ensemble inevitably increases the computational burdens, the method by manipulating the learned initialization of { w c } Nc =1 keeps the same run time as vanilla MAML.
In this section, we investigate the latter approach further.
We ask:
If we directly
learn a single weight vector w to initialize { w c } Nc =1 in meta-training, making the inner loop optimization in meta-training and meta-testing consistent and both permutation-invariant
, can we further improve the accuracy?
Concretely, we redeﬁne the learnable meta-parameters θ of MAML, which become θ = { φ , w } , where φ is for the feature extractor.
We name this method UNICORN -MAML , as we meta-train only a single weight vector w in the classiﬁcation head.
The inner loop optimization and outer loop optimization of UNICORN -MAML very much follow MAML, with some slight changes.
• Inner loop optimization: At the beginning, w is duplicated into { w c = w } Nc =1 .
That is, we use w to initialize every w c , ∀ c ∈ [ N ] .
These { w c = w } Nc =1 then undergo the same inner loop optimization process as vanilla MAML (cf. Equation 1).
• Outer loop optimization: Let us denote the updated model by θ (cid:48)
= { φ (cid:48) , w (cid:48) 1 , · · · , w (cid:48) N } .
To perform the outer loop optimization for w in meta-training,
we collect the gradients derived from the query set Q .
Let us denote by ∇ w c L ( Q , θ (cid:48) )
the gradient w.r.t.
the initial w c (cf. subsection 2.2).
Since w c is duplicated from w , we obtain the gradient w.r.t.
w by (cid:80) c ∈ [ N ] ∇ w c L ( Q , θ (cid:48) ) .
Table 4 summarizes the results of UNICORN -MAML, MAML, and many existing few-shot learning algorithms.
UNICORN -MAML consistently improves MAML: on Mini ImageNet, UNICORN -MAML has a 0 .
7% gain on one-shot tasks and a 0 . 8% gain on ﬁve-shot tasks; on Tiered ImageNet, UNICORNMAML has signiﬁcant improvements (a 3 . 5% gain on one-shot tasks and a 1 . 6% gain on ﬁve-shot tasks).
More importantly, UNICORN -MAML performs on a par with many recent algorithms on one-shot tasks, and achieves the highest accuracy on ﬁve-shot tasks.
Speciﬁcally, compared to ProtoMAML and MetaOptNet, which are both permutation-invariant variants of MAML (see the related work paragraph at the end of this section), UNICORN -MAML notably outperforms them.
Other results.
We evaluate UNICORN -MAML on CUB (Wah et al., 2011) and use the ConvNet backbone on Mini ImageNet in the appendix.
UNICORN -MAML achieves promising improvements.
Why does UNICORN -MAML work?
The design of UNICORN -MAML ensures that, without using the support set to update the model, the model simply performs at the chance level on the query set. In other words, its formulation inherently helps prevent memorization over-ﬁtting (Yin et al., 2020).
Embedding adaptation is needed.
We analyze UNICORN -MAML in terms of its inner loop updates during meta-testing, similar to Figure 3.
This time, we also investigate updating or freezing the
[Figure]
[Table] [Table]
feature extractor φ .
Figure 5 shows the results on ﬁve-way one- and ﬁve-shot tasks on both datasets.
UNICORN -MAML’s accuracy again begins with 20% but rapidly increases along with the inner loop updates.
In three out of four cases, adapting the feature extractor φ is necessary for claiming a higher accuracy, even if the backbone has been well pre-trained, which aligns with the recent claim by Arnold & Sha (2021): “Embedding adaptation is still needed for few-shot learning.”
Experiments on larger shots and transferability.
MAML or similar algorithms that involve a bi-level optimization problem ( i.e. , inner loop and outer loop optimization) are often considered more complicated and computationally expensive than algorithms without bi-level optimization, such as ProtoNet (Snell et al., 2017) or SimpleShot (Wang et al., 2019).
Nevertheless, the inner loop optimization does strengthen MAML’s adaptability during meta-testing, especially (a) when the meta-testing tasks are substantially different from the meta-training tasks ( e.g. , meta-training using
Mini ImageNet but meta-testing on CUB) or (b) when the number of shots increases ( e.g. , from 1 ∼to 10 ∼ 50 ).
In Table 5 and Table 6, we conduct further experiments to justify these aspects.
Related work.
We note that some variants of MAML are permutation-invariant, even though they are not designed for the purpose.
For example, LEO (Rusu et al., 2019) computes class prototypes ( i.e. , averaged features per class) to encode each class and uses them to produce task-speciﬁc initialization.
However, it introduces additional sub-networks.
MetaOptNet (Lee et al., 2019) performs inner loop optimization only on { w c } Nc =1 (till convergence), making it a convex problem which is not sensitive to the initialization and hence the permutations.
This method, however, has a high computational burden and needs careful hyper-parameter tuning for the additionally introduced regularizers.
Proto-MAML (Triantaﬁllou et al., 2020) initializes the linear classiﬁers { w c } Nc =1 with the prototypes, which could be permutation-invariant but cannot achieve accuracy as high as our UNICORN -MAML.
D ISCUSSION AND C ONCLUSION
There have been an abundance of “novel” algorithms proposed for few-shot classiﬁcation (Hospedales et al., 2020; Wang et al., 2020).
In these papers, MAML (Finn et al., 2017) is frequently considered as a baseline, but shows inferior results.
This raises our interests.
Is it because MAML is not suitable for few-shot classiﬁcation, or is it because MAML has not been applied appropriately to the problem?
We thus conduct a series of analyses on MAML for few-shot classiﬁcation, including hyper-parameter tuning and the sensitivity to the permutations of class label assignments in few-shot tasks.
We ﬁnd that by using a large number of inner loop gradient steps (in both meta-training and meta-testing), MAML can achieve comparable results to many existing algorithms.
By further making MAML permutationinvariant to the class label assignments, we present UNICORN -MAML, which outperforms many existing algorithms on ﬁve-shot tasks, without the need to add extra sub-networks.
We hope that UNICORN -MAML could serve as a strong baseline for future work in few-shot classiﬁcation. A CKNOWLEDGMENTS
This research is supported by National Key R&D Program of China (2020AAA0109401), CCF-Baidu
Open Fund (2021PP15002000), NSFC (61773198, 61921006, 62006112, 62176117), Collaborative Innovation Center of Novel Software Technology and Industrialization, NSF of Jiangsu Province (BK20200313), NSF IIS-2107077, NSF OAC-2118240, NSF OAC- and the OSU GI Development funds. We are thankful for the generous support of computational resources by Ohio
Supercomputer Center and AWS Cloud Credits for Research.
We thank Sébastien M.R. Arnold (USC) for helpful discussions
R EPRODUCIBILITY S TATEMENT
The details of datasets, model architectures, hyper-parameters and evaluation metrics are described in subsection 2.3 and
Appendix B. Our code is available at https://github.com/Han-Jia/
UNICORN-MAML , including the initialization weights pre-trained on the meta-training set and the checkpoints
Appendix
We provide contents that we omit in the main text.
• Appendix A: Additional background and related work.
• Appendix B: details of experimental setups (cf. section 2). • Appendix C: details and analyses of permutations of class label assignments (cf. section 4).
• Appendix D: Additional experimental results.
• Appendix E: Additional explanation of our studied methods.
A B ACKGROUNDS ON M ETA -L EARNING AND F EW -S HOT L EARNING
Training a model under data budgets is important in machine learning, computer vision, and many other application ﬁelds, since the costs of collecting data and labeling them are by no means negligible.
This is especially the case for deep learning models in visual recognition (He et al., 2016;
Dosovitskiy et al., 2021; Simonyan & Zisserman, 2015; Szegedy et al., 2015; Krizhevsky et al., 2012; Huang et al., 2017), which usually need thousands of, millions of, or even more images to train (Russakovsky et al., 2015; Deng et al., 2009; Guo et al., 2016; Thomee et al., 2016; Mahajan et al., 2018; Joulin et al., 2016) in a conventional supervised manner.
Different from training a model to predict at the instance level, meta-learning attempts to learn the inductive bias across training tasks (Baxter, 2000; Vilalta & Drissi, 2002).
More speciﬁcally, meta-learning aims to train a “meta-model” to summarize the common characteristics of tasks and generalize them to those novel but related tasks (Maurer, 2009;
Maurer et al., 2016; Denevi et al., 2018; Ye et al., 2021).
Meta-learning has been applied in various ﬁelds, including few-shot learning (Ravi & Larochelle, 2017;
Snell et al., 2017; Wang & Hebert, 2016;
Ye et al., 2020a;b; Sung et al., 2018; Zhang et al., 2018a; Wang et al., 2018b; Tseng et al., 2020;
Fei et al., 2021)
In few-shot learning (FSL), meta-learning is applied to learn the ability of “how to build a classiﬁer using limited data” that can be generalized across tasks.
Such an inductive bias is ﬁrst learned over few-shot tasks composed of “base” classes, and then evaluated on tasks composed of “novel” classes.
For example, few-shot classiﬁcation can be implemented in a non-parametric way with soft nearest neighbor (Vinyals et al., 2016) or nearest center classiﬁers (Snell et al., 2017), so that the feature extractor is learned and acts at the task level.
The learned features pull similar instances together and push dissimilar ones far away, such that a test instance can be classiﬁed even with a few labeled training examples (Koch et al., 2015).
Considering the complexity of a hypothesis class, the model training conﬁgurations ( i.e. , hyper-parameters) also serve as a type of inductive biases.
Andrychowicz et al.
; Ravi & Larochelle (2017) meta-learn the optimization strategy for each task, including the learning rate and update directions.
Other kinds of inductive biases are also explored.
Hariharan & Girshick (2017); Wang et al.
(2018b) learn a data generation prior to augment examples given few images; Dai et al.
(2017) extract logical derivations from related tasks; Wang et al.
(2017b)
; Shyam et al.
(2017) learn the prior to attend images.
Model-agnostic meta-learning (MAML)
(Finn et al., 2017) proposes another inductive bias, i.e. , the model initialization.
After the model initialization that is shared among tasks has been meta-trained, the classiﬁer of a new few-shot task can be ﬁne-tuned with several steps of gradient descent from that initial point.
The universality of this MAML-type updates is proved in (Finn & Levine, 2018).
MAML has been applied in various scenarios, such as uncertainty estimation (Finn et al., 2018), robotics control (Yu et al., 2018; Clavera et al., 2019), neural translation (Gu et al., 2018), language generation (Huang et al., 2018), etc.
Despite the success, there are still problems with MAML.
For example, Nichol et al.
(2018) handle the computational burden by presenting a family of approaches using ﬁrst-order approximations.
Rajeswaran et al.
(2019) propose to leverage implicit differentiation, making the calculation of the meta-gradients much efﬁcient and accurate.
Antoniou et al.
(2019) provide a bunch of tricks to train and stabilize the MAML framework.
Bernacchia (2021) points out that negative rates of gradient updates help in some scenarios.
Rajendran et al.
; Yao et al.
; Yin et al.
(2020) argue that the learned initialization by MAML may be at high risk of (a) memorization over-ﬁtting, where it solves meta-training tasks without the need of inner loop optimization, or (b) learner over-ﬁtting, where it over-ﬁts to the meta-training tasks and fails to generalize to the meta-testing tasks.
They thus propose to improve MAML by imposing a regularizer or performing data augmentation.
Since MAML applies a uniform initialization to all the tasks ( i.e. , the same set of { w c } Nc =1 and φ ), recent methods explore ways to better incorporate task characteristics.
Lee et al.
; Bertinetto et al.
; Raghu et al. (2020) optimize the linear classiﬁers { w c } Nc =1 (not the feature f φ ) till convergence in the inner loop; Triantaﬁllou et al.
; Ye et al.
(2020b) initialize the linear classiﬁers using class prototypes ( i.e. , aggregated features per class) so they are task-aware even before the inner loop optimization.
Another direction is to enable task-speciﬁc initialization for the entire model (Requeima et al., 2019; Vuorio et al., 2019; Yao et al., 2019; Rusu et al., 2019), which often needs additional sub-networks.
Our work is complementary to the above improvements of MAML: we ﬁnd an inherent permutation issue of MAML in meta-testing and conduct a detailed analysis.
We then build upon it to improve
MAML.
We note that some of the above methods can be permutation-invariant even though they are not designed for the purpose.
For example, LEO (Rusu et al., 2019) computes class prototypes ( i.e. , averaged features per class) to represent each semantic class 4 .
However, it introduces additional sub-networks.
MetaOptNet (Lee et al., 2019) performs inner loop optimization only on { w c } Nc = (till convergence), making it a convex problem which is not sensitive to the initialization and hence the permutations.
This method, however, has a high computational burden and needs careful hyperparameter tuning for the additionally introduced regularizers.
Proto-MAML (Triantaﬁllou et al., 2020) initializes the linear classiﬁers { w c } Nc =1 with the prototypes, which could be permutation-invariant but cannot achieve accuracy as high as our UNICORN -MAML.
We note that one fundamental difference between our work and (Dhillon et al., 2020), LEO (Rusu et al., 2019), and Proto-MAML (Triantaﬁllou et al., 2020) is the motivation: they aim to provide the initialization of the classiﬁer head with better semantic meanings, while our goal is to resolve the sensitivity of MAML to the permutations.
B D ETAILS OF E XPERIMENTAL S ETUPS
We follow the pre-training procedure in (Ye et al., 2020a).
During pre-training we append the feature extractor backbone with a fully-connected layer for classiﬁcation, and train it to classify all classes in the base class set ( e.g. , 64 classes in the Mini ImageNet) with the cross-entropy loss.
In this stage, we apply standard ImageNet image augmentations ( e.g. , random crop and random ﬂip).
The best pre-trained feature extractor ( i.e. , epoch) is selected based on the one-shot classiﬁcation accuracy on the meta-validation set.
Speciﬁcally, we sample one-shot tasks from the meta-validation set, and apply the nearest neighbor classiﬁer on top of the extracted features to evaluate the quality of the backbone.
Finally, the best pre-trained backbone is used to initialize the feature extractor for MAML.
C P ERMUTATIONS OF C LASS L ABEL A SSIGNMENTS
We provide more analyses and discussions on the permutation issue in the class label assignment.
As illustrated in Figure 1 (a), few-shot tasks of the same set of N semantic classes ( e.g. , “unicorn”, “bee”, etc.) can be associated with different label assignments ( i.e. , c ∈ [ N ] ) and are paired with the learned initialization { w c } Nc =1 of MAML differently.
For ﬁve-way tasks, there are 120 permutations. [Figure] pp o r t S e t
A ccc u r a c y ( % ) [Figure] In section 4, we study how the permutations affect the meta-testing accuracy (after the inner loop optimization of φ and { w c } Nc =1 ) and see a high variance among the permutations.
We note that, the inner loop optimization updates not only the linear classiﬁers { w c } Nc =1 , but also φ of the feature extractor.
Different permutations therefore can lead to different feature extractors.
Here, we further sample a ﬁve-way one-shot meta-testing task, and study the change of accuracy along with the inner loop updates (using a MAML trained with a ﬁxed number of inner loop updates).
Speciﬁcally, we plot both the support set and query set accuracy for each permutation.
As shown in Figure F, there exists a high variance of query set accuracy among permutations after the inner loop optimization.
This is, however, not the case for the support set.
(The reason that only three curves appear for the support set is because there are only ﬁve examples, and all the permutations reach 100% support set accuracy within ﬁve inner loop steps.)
Interestingly, for all the permutations, their initialized accuracy ( i.e. , before inner loop optimization) is all 20% .
After an investigation, we ﬁnd that the meta-learned { w c } Nc =1 (initialization) is dominated by one of them; i.e. , all the support or query examples are classiﬁed into one class.
While this may not always be the case for other few-shot tasks or if we re-train MAML, for the task we sampled, it explains why we obtain 20% for all permutations.
We note that, even with an initial accuracy of 20% , the learned initialization can be updated to attain high classiﬁcation accuracy.
We further compare the change of support and query set accuracy along with the inner loop optimization in Figure F. We ﬁnd that, while both accuracy increases, since the support set accuracy converges quickly and has a smaller variance among permutations, it is difﬁcult to use its information to determine which permutation leads to the highest query set accuracy.
This makes sense since the support set is few-shot: its accuracy thus cannot robustly reﬂect the query set accuracy.
This explains why the methods studied in Table 1 cannot determine the best permutation for the query set.
C. M ATHEMATICAL EXPLANATION
We provide a simple mathematical explanation for why, on average, the query set accuracy is at the chance level if we directly apply the learned initialized model.
Suppose we have a ﬁve-way task with ﬁve semantic classes { “dog”, “cat”, “bird”, “car”, “person” } .
Without loss of generality, let us assume that the query set has only ﬁve examples, one from each class.
Let us also assume that the best permutation — i.e. , the best assignment of { w 1 , · · · , w 5 } to these classes — gives a 100% query set accuracy using the initialized model.
Since there are in total 120 possible permutations, there will be 10 of them with 60% accuracy ( i.e. , by switching two-class indices), 20 of them with 40% accuracy ( i.e. , by shufﬂing the indices of three classes such that they do not take their original indices), 45 of them with 20% accuracy, and 54 of them with 0% accuracy.
Taking an average over these permutations gives a 20% accuracy.
In other words, even if one of the permutations performs well, on average the accuracy will be close to random. [Figure] 1E-3 5E-3 1E-2 5E-2 1E- [Figure]
1E-3 5E-3 1E-2 5E-2 1ETiered ImageNet, ResNet [Figure] 1E-3 5E-3 1E-2 5E-2 1ECUB, ResNet [Figure]
[Figure]
[Figure]
[Figure]
1E-
[Figure]
5E-2 1E-
[Figure]
1E-3 5E-3 1E-2 5E-2 1E-
[Figure]
ImageNet,
[Figure]
f
[Figure]
1E-3 5E-3 1E-2 5E-2 1E-
[Figure]
[Figure]
D A DDITIONAL E XPERIMENTAL R ESULTS
Similar to Figure [Figure] we plot the meta-testing accuracy of ﬁve-way ﬁve-shot tasks on the three datasets over ResNet and ConvNet backbones in Figure 2.
We get a similar trend with Figure 2 where MAML achieves higher and much more stable accuracy (w.r.t.
the learning rate) when M is larger than 15 .
Speciﬁcally, for Mini ImageNet with ResNet, the highest accuracy 83 .
44% is obtained with M = 20 , higher than 80 .
81% with M = 5 .
We also plot the change of ﬁve-way ﬁve-shot classiﬁcation accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta-training (red) or meta-testing classes (blue), along with the process of inner loop updates, using the best model initialization learned by MAML in Figure H for [Table] [Table] each pair of dataset and backbone.
There indicates a similar phenomenon as Figure [Table] where MAML needs a large M .
We further evaluate UNICORN -MAML on CUB dataset with ResNet-12 backbone and on Mini ImageNet with the four-layer ConvNet backbone.
The results are listed in Table G and Table H
[Table]
respectively.
By comparing UNICORN -MAML with others
[Table]
we ﬁnd the carefully tuned
MAML shows promising results and UNICORN -MAML outperforms the existing methods.
E A DDITIONAL E XPLANATIONS OF O UR S TUDIED M ETHODS
We provide some more explanations on the ensemble and forced permutation-invariant methods introduced in section 5.
For the ensemble method, given a few-shot task, we can permute { w c } Nc = to pair them differently with such a task.
We can then perform different inner loop optimizations to obtain a set of ﬁve-way classiﬁers that we can perform ensemble upon.
In the main text, we average the posterior probabilities of these ﬁve-way classiﬁers to make the ﬁnal predictions.
Since the permutation affects the meta-training phase as well, we can interpret the meta-training phase as follows.
Ever time we sample a few-shot task T
=
( S , Q ) , we also sample a permutation π : [ N ] (cid:55)→ [ N ] to re-label the classes.
(We note that, this is implicitly done when few-shot tasks are sampled.)
We then take T π =
( S π , Q π ) to optimize θ in the inner loop.
That is, in meta-training, the objective function in Equation 2 can indeed be re-written as
[Equation]
where p ( π ) is a uniform distribution over all possible permutations.
Equation D can be equivalently re-written as
[Equation]
where θ π means that the initialization of the linear classiﬁers { w c } Nc =1 are permuted; θ (cid:48)
π is the corresponding updated model.
This additional sampling process of π is reminiscent of dropout (Srivastava et al., 2014), which randomly masks out a neural network’s neurons or edges to prevent an over-parameterized neural network from over-ﬁtting.
During testing, dropout takes expectation over the masks.
We also investigate a similar idea during meta-testing, by taking expectation ( i.e. , average) on the learned initiation of the linear classiﬁers over different permutations.
This results in a new initialization during the meta-testing phase: w c ← 1 N [Equation]
w [Equation]