A B A S E L I N E F O R F E W - S H O T
I M A G E C L A S S
I F I C
AT
I
O N
Anonymous authors
Paper under double-blind review
A B S T R AC T
Fine-tuning a deep network trained with the standard cross-entropy loss is a strong baseline for few-shot learning.
When ﬁne-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-Imagenet, TieredImagenet, CIFAR-FS and FC-100 with the same hyper-parameters.
The simplicity of this approach enables us to demonstrate the ﬁrst few-shot learning results on the
Imagenet-21k dataset.
We ﬁnd that using a large number of meta-training classes results in high few-shot accuracies even for a large number of few-shot classes.
We do not advocate our approach as the solution for few-shot learning, but simply use the results to highlight limitations of current benchmarks and few-shot protocols.
We perform extensive studies on benchmark datasets to propose a metric that quantiﬁes the “hardness” of a few-shot episode.
This metric can be used to report the performance of few-shot algorithms in a more systematic way.
[Figure]
As image classiﬁcation systems begin to tackle more and more classes, the cost of annotating a massive number of images and the difﬁculty of procuring images of rare categories increases.
This has fueled interest in few-shot learning, where only few labeled samples per class are available for training.
Fig.
1 displays a snapshot of the state-of-the-art.
We estimated this plot by using published numbers for the estimate of the mean accuracy, the 95% conﬁdence interval of this estimate and the number of few-shot episodes.
For MAML (Finn et al., 2017) and MetaOpt SVM (Lee et al., 2019), we use the number of episodes in the author’s Github implementation.
The ﬁeld appears to be progressing steadily albeit slowly based on Fig.
However, the variance of the estimate of the mean accuracy is not the same as the variance of the accuracy.
The former can be zero (e.g., asymptotically for an unbiased estimator), yet the latter could be arbitrarily large.
The variance of the accuracies is extremely large in Fig.
This suggests that progress in the past few years may be less signiﬁcant than it seems.
To compound the problem, many algorithms report results using different models for different number of ways (classes) and shots (number of labeled samples per class), with aggressive hyper-parameter optimization.
1 Our goal is to develop a simple baseline for few-shot learning , one that does not require specialized training depending on the number of ways or shots, nor hyper-parameter tuning for different protocols.
The simplest baseline we can think of is to pre-train a model on the meta-training dataset using the standard cross-entropy loss, and then ﬁne-tune on the few-shot dataset.
Although this approach is basic and has been considered before (Vinyals et al., 2016; Chen et al., 2018), it has gone unnoticed that it outperforms many sophisticated few-shot algorithms.
Indeed, with a small twist of performing ﬁne-tuning transductively, this baseline outperforms all state-of-the-art algorithms on all standard benchmarks and few-shot protocols (cf.
Table 1).
Our contribution is to develop a transductive ﬁne-tuning baseline for few-shot learning, our approach works even for a single labeled example and a single test datum per class.
Our baseline outperforms the state-of-the-art on a variety of benchmark datasets such as Mini-ImageNet (Vinyals et al., 2016),
Tiered-ImageNet (Ren et al., 2018), CIFAR-FS (Bertinetto et al., 2018) and FC-100 (Oreshkin et al., 2018), all with the same hyper-parameters.
Current approaches to few-shot learning are hard to scale to large datasets.
We report the ﬁrst few-shot learning results on the Imagenet-21k dataset (Deng et al., 2009) which contains 14.2 million images across 21,814 classes.
The rare classes in Imagenet-21k form a natural benchmark for few-shot learning.
The empirical performance of this baseline, should not be understood as us suggesting that this is the right way of performing few-shot learning.
We believe that sophisticated meta-training, understanding taxonomies and meronomies, transfer learning, and domain adaptation are necessary for effective few-shot learning.
The performance of the simple baseline however indicates that we need to interpret existing results 2 with a grain of salt, and be wary of methods that tailor to the benchmark.
To facilitate that, we propose a metric to quantify the hardness of few-shot episodes and a way to systematically report performance for different few-shot protocols.
P RO B L E M D E F I N I T I O N A N D R E L AT E D W O R K
We ﬁrst introduce some notation and formalize the few-shot image classiﬁcation problem.
Let ( x, y ) denote an image and its ground-truth label respectively.
The training and test datasets are D s =
{ ( x i , y i ) }
N s i =1 and D q = { ( x i , y i ) }
N q i =1 respectively, where y i ∈
C t for some set of classes C t .
In the few-shot learning literature, training and test datasets are referred to as support and query datasets respectively, and are collectively called a few-shot episode.
The number of ways , or classes, is | C t | .
The set { x i |
y
i
=
k, ( x i , y i )
∈
D s } is the support of class k and its cardinality is (non-zero)
s support shots (more generally referred to as shots ).
s is small in the few-shot setting.
The set { x i |
y
i
=
k, ( x i , y i )
∈
D q } is the query of class k and its cardinality is q query shots .
The goal is to learn a function F to exploit the training set D s to predict the label of a test datum x ,
where ( x, y ) ∈
D q , by
[Equation]
Typical approaches for supervised learning replace D s above with a statistic, θ ∗ =
θ ∗ ( D s ) that is, ideally, sufﬁcient to classify D s , as measured by, say, the cross-entropy loss
[Equation]
where p θ ( ·| x ) is the probability distribution on C t as predicted by the model in response to input x .
When presented with a test datum, the classiﬁcation rule is typically chosen to be of the form
[Equation]
where D s is represented by θ ∗ .
This form of the classiﬁer entails a loss of generality unless θ ∗ is a sufﬁcient statistic, p θ ∗ ( y | x )
=
p ( y | x ) , which is of course never the case, especially given few labeled data in D s .
However, it conveniently separates training and inference phases, never having to revisit the training set.
This might be desirable in ordinary image classiﬁcation, but not in few-shot learning.
We therefore adopt the more general form of F in (1).
If we call the test datum x = x N s+1 , then we can obtain the general form of the classiﬁer by
[Equation]
In addition to the training set, one typically also has a meta-training set, D m = { ( x i , y i ) } N m i =1 , where y i ∈ C m , with set of classes C m disjoint from C t .
The goal of meta-training is to use D m to infer the parameters of the few-shot learning model:
ˆ θ ( D m ; ( D s , D q ))
=
arg min θ 1 N m (cid:80) ( x,y ) ∈D m (cid:96) ( y, F θ ( x ; ( D s , D q ))) , where meta-training loss (cid:96) depends on the method.
2. R ELATED WORK
Learning to learn: The meta-training loss is designed to make few-shot training efﬁcient (Utgoff, 1986; Schmidhuber, 1987; Baxter, 1995; Thrun, 1998).
This approach partitions the problem into a base-level that performs standard supervised learning and a meta-level that accrues information from the base-level.
Two main approaches have emerged to do so.
Gradient-based approaches
: These approaches treat the updates of the base-level as a learnable mapping (Bengio et al., 1992).
This mapping can be learnt using temporal models (Hochreiter et al., 2001;
Ravi & Larochelle, 2016), or one can back-propagate the gradients across the base-level updates (Maclaurin et al., 2015; Finn et al., 2017).
It is challenging to perform this dual or bi-level optimization, respectively.
These approaches have not been shown to be competitive on large datasets.
Recent approaches learn the base-level in closed-form using SVMs (Bertinetto et al., 2018; Lee et al., 2019) which restricts the capacity of the base-level although it alleviates the optimization problem.
Metric-based approaches: A majority of the state-of-the-art algorithms are metric-based approaches.
These approaches learn an embedding that can be used to compare (Bromley et al., 1994; Chopra et al., 2005) or cluster (Vinyals et al., 2016;
Snell et al., 2017) query samples.
Recent approaches build upon this idea with increasing levels of sophistication in learning the embedding (Vinyals et al., 2016;
Gidaris & Komodakis, 2018; Oreshkin et al., 2018), creating exemplars from the support set and picking a metric for the embedding (Gidaris & Komodakis, 2018; Allen et al., 2018; Ravichandran et al., 2019).
There are numerous hyper-parameters involved in implementing these approaches which makes it hard to evaluate them systematically (Chen et al., 2018).
Transductive learning: This approach is more efﬁcient at using few labeled data than supervised learning (Joachims, 1999; Zhou et al., 2004; Vapnik, 2013).
The idea is to use information from the test datum x to restrict the hypothesis space while searching for the classiﬁer F ( x, D s ) at test time .
Our approach is closest to this line of work.
We train a model on the meta-training set D m and initialize a classiﬁer using the support set D s .
The parameters are then ﬁne-tuned to adapt to the new test datum x .
There are recent papers in few-shot learning such as Nichol et al.
; Liu et al.
(2018a) that are motivated from transductive learning and exploit the unlabeled query samples.
The former updates batch-normalization parameters using query samples while the latter uses label propagation to estimate labels of all query samples at once.
Semi-supervised learning: We penalize the Shannon Entropy of the predictions on the query samples at test time.
This is a simple technique in the semi-supervised learning literature, closest to
Grandvalet & Bengio (2005).
Modern augmentation techniques such as Miyato et al.
; Sajjadi et al.
; Dai et al.
(2017) or graph-based approaches (Kipf & Welling, 2016) can also be used with our approach; we used the entropic penalty for the sake of simplicity.
Semi-supervised few-shot learning is typically formulated as having access to extra unlabeled data during meta-training or few-shot training (Garcia & Bruna, 2017; Ren et al., 2018).
This is different from our approach which uses the unlabeled query samples for transductive learning.
Initialization for ﬁne-tuning: We use recent ideas from the deep metric learning literature (Hu et al., 2015;
Movshovitz-Attias et al., 2017; Qi et al., 2018; Chen et al., 2018; Gidaris & Komodakis, 2018) to initialize the meta-trained model for ﬁne-tuning.
These works connect the softmax cross-entropy loss with cosine distance and are discussed further in Section 3.1.
A P P ROAC H
The simplest form of meta-training is pre-training with the cross-entropy loss, which yields
[Equation]
where the second term denotes a regularizer, say weight decay R ( θ )
= (cid:107) θ (cid:107) 2 / 2 .
The model predicts logits z k ( x ; θ ) for k ∈ C m and the distribution p θ ( ·| x ) is computed from these logits using the softmax operator.
This loss is typically minimized by stochastic gradient descent-based algorithms.
If few-shot training is performed according to the general form in (4), then the optimization is identical to that above and amounts to ﬁne-tuning the pre-trained model.
However, the model needs to be modiﬁed to account for the new classes.
Careful initialization can make this process efﬁcient.
3. S UPPORT - BASED INITIALIZATION
Given the pre-trained model (called the “backbone”), p θ (dropping the hat from ˆ θ ), we append a new fully-connected “classiﬁer” layer that takes the logits of the backbone as input and predicts the labels in C t .
For a support sample ( x, y ) , denote the logits of the backbone by z ( x ; θ ) ∈ R | C m | ; the weights and biases of the classiﬁer by w ∈ R | C t |×| C m | and b ∈ R | C t | respectively; and the k th row of w and b by w k and b k respectively.
The ReLU non-linearity is denoted by ( · ) + .
If the classiﬁer’s logits are z (cid:48)
=
wz ( x ; θ ) + + b , the ﬁrst term in the cross-entropy loss: − log p Θ ( y | x ) = − w y z
( x ; θ ) + − b y + log
(cid:80) k
e w k z ( x ; θ ) + + b k would be the cosine distance between w y and z ( x ; θ ) + if both were normalized to unit (cid:96) 2 norm and bias b y = 0 .
This suggests
[Equation]
as a candidate for initializing the classiﬁer, along with normalizing z ( x ; θ ) + to unit (cid:96) 2 norm.
It is easy to see that this maximizes the cosine similarity between features z ( x ; θ ) + and weights w y .
For multiple support samples per class, we take the Euclidean average of features z ( x ; θ ) + for each class in C t , before (cid:96) 2 normalization in (6).
The logits of the classiﬁer are thus given by
[Equation]
where Θ = { θ, w, b } , the combined parameters of the backbone and the classiﬁer.
Note that we have added a ReLU non-linearity between the backbone and the classiﬁer, before the (cid:96) 2 normalization.
All the parameters Θ are trainable in the ﬁne-tuning phase.
Remark 1 (Relation to weight imprinting).
The support-based initialization is motivated from previous papers (Hu et al., 2015; Movshovitz-Attias et al., 2017; Chen et al., 2018; Gidaris & Komodakis, 2018).
In particular, Qi et al.
(2018) use a similar technique, with minor differences, to expand the size of the ﬁnal fully-connected layer (classiﬁer) for low-shot continual learning.
The authors call their technique “weight imprinting” because w k can be thought of as a template for class k .
In our case, we are only interested in performing well on the few-shot classes.
Remark 2 (Using logits of the backbone instead of features as input to the classiﬁer).
A natural way to adapt the backbone to predict new classes is to re-initialize its ﬁnal fully-connected layer (classiﬁer).
We instead append a new classiﬁer after the logits of the backbone.
This is motivated from
Frosst et al.
(2019) who show that for a trained backbone, outputs of all layers are entangled, without class-speciﬁc clusters; but the logits are peaked on the correct class, and are therefore well-clustered.
The logits are thus better inputs to the classiﬁer as compared to the features.
We explore this choice via an experiment in Appendix C.6.
3. T RANSDUCTIVE FINE - TUNING
In (4), we assumed that there is a single query sample.
However, we can also process multiple query samples together, and perform the minimization over all unknown query labels.
However, we introduce a regularizer as we seek outputs with a peaked posterior, or low Shannon Entropy H .
So the transductive ﬁne-tuning phase solves for
[Equation]
Note that the data ﬁtting term uses the labeled support samples whereas the regularizer uses the unlabeled query samples.
The two terms can be highly imbalanced (due to the varying range of values for the two quantities, or due to the variance in their estimates which depend on N s and N q ).
To allow ﬁner control on this imbalance, one can use a coefﬁcient for the entropic term and/or a temperature in the softmax distribution of the query samples.
Tuning these hyper-parameters per dataset and few-shot protocol leads to uniform improvements in the results in Section 4 by 1-2%.
However, we wish to keep in line with our goal of developing a simple baseline and refrain from optimizing these hyper-parameters, and set them equal to 1 for all experiments on benchmark datasets.
E X P E R I M E N TA L R E S U LT S
We show results of transductive ﬁne-tuning on benchmark datasets in few-shot learning, namely
Mini-ImageNet (Vinyals et al., 2016), Tiered-ImageNet (Ren et al., 2018), CIFAR-FS (Bertinetto et al., 2018) and FC-100 (Oreshkin et al., 2018), in Section 4.1.
We also show large-scale experiments on the Imagenet-21k dataset (Deng et al., 2009) in Section 4.2.
Along with the analysis in Section 4.3, these help us design a metric that measures the hardness of an episode in Section 4.4.
We sketch key points of the experimental setup here; see Appendix A for details.
Pre-training: We use the WRN-28-10 (Zagoruyko & Komodakis, 2016) model as the backbone.
We pre-train using standard data augmentation, cross-entropy loss with label smoothing (Szegedy et al., 2016) of (cid:15) = 0.1, mixup regularization (Zhang et al., 2017) of α = 0.25, SGD with batch-size of 256,
Nesterov’s momentum of 0.9, weight-decay of 10 − 4 and no dropout.
We use batch-normalization (Ioffe & Szegedy, 2015) but exclude its parameters from weight decay (Jia et al., 2018).
We use cyclic learning rates (Smith, 2017) and half-precision distributed training on 8 GPUs (Howard et al., 2018) to reduce training time.
Each dataset has a training, validation and test set consisting of disjoint sets of classes.
Some algorithms use only the training set as the meta-training set (Snell et al., 2017; Oreshkin et al., 2018), while others use both training and validation sets (Rusu et al., 2018).
For completeness we report results using both methodologies; the former is denoted as (train) while the latter is denoted as (train + val).
All experiments in Sections 4.3 and 4.4 use the (train + val) setting.
Fine-tuning: We perform ﬁne-tuning on one GPU in full-precision for 25 epochs and a ﬁxed learning rate of 5 × 10 − 5 with Adam (Kingma & Ba, 2014) without any regularization.
We make two weight updates in each epoch: one for the cross-entropy term using support samples and one for the Shannon
Entropy term using query samples (cf.
Hyper-parameters: We used images from Imagenet-1k belonging to the training classes of MiniImageNet as the validation set for pre-training the backbone for Mini-ImageNet.
We used the validation set of Mini-ImageNet to choose hyper-parameters for ﬁne-tuning.
All hyper-parameters are kept constant for experiments on benchmark datasets .
Evaluation: Few-shot episodes contain classes sampled uniformly from classes in the test sets of the respective datasets; support and query samples are further sampled uniformly for each class; the query shot is ﬁxed to 15 for all experiments unless noted otherwise.
All networks are evaluated over 1, few-shot episodes unless noted otherwise.
To enable easy comparison with existing literature, we report an estimate of the mean accuracy and the 95% conﬁdence interval of this estimate.
However, we encourage reporting the standard deviation in light of Section 1 and Fig. 1.
[Table]
4. R ESULTS ON BENCHMARK DATASETS
Table 1 shows the results of transductive ﬁne-tuning on benchmark datasets for standard few-shot protocols.
We see that this simple baseline is uniformly better than state-of-the-art algorithms.
We include results for support-based initialization, which does no ﬁne-tuning; and for ﬁne-tuning, which involves optimizing only the cross-entropy term in (8) using the labeled support samples.
The support-based initialization is sometimes better than or comparable to state-of-the-art algorithms (marked † ).
The few-shot literature has gravitated towards larger backbones (Rusu et al., 2018).
Our results indicate that for large backbones even standard cross-entropy pre-training and support-based initialization work well, similar to observation made by Chen et al.
For the 1-shot 5-way setting, ﬁne-tuning using only the labeled support examples leads to minor improvement over the initialization, and sometimes marginal degradation.
However, for the 5-shot 5-way setting non-transductive ﬁne-tuning is better than the state-of-the-art .
In both (train) and (train + val) settings, transductive ﬁne-tuning leads to 2-7% improvement for 1-shot 5-way setting over the state-of-the-art for all datasets.
It results in an increase of 1.5-4% for the 5-shot 5-way setting except for the Mini-ImageNet dataset, where the performance is matched.
This suggests that the use of the unlabeled query samples is vital for the few-shot setting .
For the Mini-ImageNet, CIFAR-FS and FC-100 datasets, using additional data from the validation set to pre-train the backbone results in 2-8% improvements; the improvement is smaller for TieredImageNet.
This suggests that having more pre-training classes leads to improved few-shot performance as a consequence of a better embedding.
See Appendix C.5 for more experiments.
4. L ARGE - SCALE FEW - SHOT LEARNING
The Imagenet-21k dataset (Deng et al., 2009) with 14.2M images across 21,814 classes is an ideal large-scale few-shot learning benchmark due to the high class imbalance.
The simplicity of our approach allows us to present the ﬁrst few-shot learning results on this large dataset.
We use the
7,491 classes having more than 1,000 images each as the meta-training set and the next 13,007 classes with at least 10 images each for constructing few-shot episodes.
See Appendix B for details.
[Table]
Table 2 shows the mean accuracy of transductive ﬁne-tuning evaluated over 80 few-shot episodes on
Imagenet-21k.
The accuracy is extremely high as compared to corresponding results in Table 1 even for large way.
E.g., the 1-shot 5-way accuracy on Tiered-ImageNet is 72.87 ± 0.71% while it is 89 ± 1.86% here.
This corroborates the results in Section 4.1 and indicates that pre-training with a large number of classes may be an effective strategy to build large-scale few-shot learning systems .
The improvements of transductive ﬁne-tuning are minor for Imagenet-21k because the support-based initialization accuracies are extremely high.
We noticed a slight degradation of accuracies due to transductive ﬁne-tuning at high ways because the entropic term in (8) is much larger than the the cross-entropy loss.
The experiments for Imagenet-21k therefore scale down the entropic term by log | C t | and forego the ReLU in (6) and (7).
This reduces the difference in accuracies at high ways.
4. A NALYSIS
This section presents a comprehensive analysis of transductive ﬁne-tuning on the Mini-ImageNet,
Tiered-ImageNet and Imagenet-21k datasets.
[Figure]
Fig.
2a shows that the mean accuracy improves with query shot if the support shot is low; this effect is mi- nor for Tiered-ImageNet.
The mean accuracy for query shot of 1 is high because transductive ﬁne-tuning can specialize to those queries.
Fig.
2b shows that the mean accuracy degrades logarithmically with way for ﬁxed support shot and query shot (15).
Fig.
2c suggests that the mean accuracy improves logarithmically with the support shot for ﬁxed way and query shot (15).
These trends suggest thumb rules for building few-shot systems.
Robustness of transductive ﬁne-tuning to query shot: Fig.
2a shows the effect of changing the query shot on the mean accuracy.
For the 1-shot 5-way setting, the entropic penalty in (8) helps as the query shot increases.
This effect is minor in the 5-shot 5-way setting as more labeled data is available.
Query shot of 1 achieves a relatively high mean accuracy because transductive ﬁne-tuning can adapt to those few queries.
One query shot is enough to beneﬁt from transductive ﬁne-tuning : for
Mini-ImageNet, the 1-shot 5-way accuracy with query shot of 1 is 66.94 ± 1.55% which is better than non-transductive ﬁne-tuning (59.62 ± 0.66% in Table 1) and higher than other approaches.
Performance for different way and support shot: A few-shot system should be able to robustly handle different few-shot scenarios.
Figs. 2b and 2c, show the performance of transductive ﬁne-tuning with changing way and support shot.
The mean accuracy changes logarithmically with the way and support shot which provides thumb rules for building few-shot systems.
Different backbone architectures:
We include experiments using conv (64) × 4 (Vinyals et al., 2016) and ResNet-12 (He et al., 2016a; Oreshkin et al., 2018) in Table 3, in order to facilitate comparisons for different backbone architectures.
The results for transductive ﬁne-tuning are comparable or better than state-of-the-art for a given backbone architecture, except for those in Liu et al.
(2018b) who use a more sophisticated transductive algorithm using graph propagation, with conv (64) × 4 .
In line with our goal for simplicity, we kept the hyper-parameters for pre-training and ﬁne-tuning the same as the ones used for WRN-28-10 (cf.
Sections 3 and 4).
These results show that transductive ﬁne-tuning is a sound baseline for a variety of backbone architectures .
Computational complexity: There is no free lunch and our advocated baseline has its limitations.
It performs gradient updates during the ﬁne-tuning phase which makes it slow at inference time.
Speciﬁcally, transductive ﬁne-tuning is about 300 × slower (20.8 vs. 0.07 seconds) for a 1-shot 5-way episode with 15 query shot as compared to Snell et al.
(2017) with the same backbone architecture (prototypical networks (Snell et al., 2017) do not update model parameters at inference time).
The latency factor reduces with higher support shot.
Interestingly, for a single query shot, the former takes 4 seconds vs. 0.07 seconds.
This is a more reasonable factor of 50 × , especially considering that the mean accuracy of the former is 66.2% compared to about 58% of the latter in our implementation.
Experiments in Appendix C.3 suggest that using a smaller backbone architecture partially compensates for the latency with some degradation of accuracy.
A number of approaches such as
Ravi & Larochelle (2016);
Finn et al.
(2017); Rusu et al.
; Lee et al.
(2019) also perform additional processing at inference time and are expected to be slow, along with other transductive approaches (Nichol et al., 2018;
Liu et al., 2018b).
Additionally, support-based initialization has the same inference time as Snell et al.
4. A PROPOSAL FOR REPORTING FEW - SHOT CLASSIFICATION PERFORMANCE
As discussed in Section 1, we need better metrics to report the performance of few-shot algorithms.
There are two main issues: (i) standard deviation of the few-shot accuracy across different sampled episodes for a given algorithm, dataset and few-shot protocol is very high (cf.
Fig. 1), and (ii) different models and hyper-parameters for different few-shot protocols makes evaluating algorithmic contributions difﬁcult (cf.
Table 1).
This section takes a step towards resolving these issues.
Hardness of an episode: Classiﬁcation performance on a few-shot episode is determined by the relative location of the features corresponding to labeled and unlabeled samples.
If the unlabeled features are close to the labeled features from the same class, the classiﬁer can distinguish between the classes easily to obtain a high accuracy.
Otherwise, the accuracy would be low.
The following deﬁnition characterizes this intuition.
For training (support) set D s and test (query) set D q , we will deﬁne the hardness Ω ϕ as the average log-odds of a test datum being classiﬁed incorrectly.
More precisely,
[Equation]
where p ( ·| x ) is a softmax distribution with logits z y
= wϕ ( x ) .
w is the weight matrix constructed using (6) and D s ; and ϕ is the (cid:96) 2 normalized logits computed using a rich-enough feature generator, say a deep network trained for standard image classiﬁcation.
This is a clustering loss where the labeled support samples form class-speciﬁc cluster centers.
The cluster afﬁnities are calculated using cosine-similarities, followed by the softmax operator to get the probability distribution p ( ·| x ) .
Note that Ω ϕ does not depend on the few-shot learner and gives a measure of how difﬁcult the classiﬁcation problem is for any few-shot episode, using a generic feature extractor.
Fig.
3 demonstrates how to use the hardness metric.
Few-shot accuracy degrades linearly with hardness .
Performance for all hardness can thus be estimated by testing for two different ways.
We advocate selecting hyper-parameters using the area under the ﬁtted curve as a metric instead of tuning them speciﬁcally for each few-shot protocol.
The advantage of such a test methodology is that it predicts the performance of the model across multiple few-shot protocols systematically.
[Figure]
Different algorithms can be compared directly , e g., transductive ﬁne-tuning (solid lines) and support-based initialization (dotted lines).
For instance, the former leads to large improvements on easy episodes, the performance is similar for hard episodes, especially for Tiered-ImageNet and
Imagenet-21k.
The high standard deviation of accuracy of few-shot learning algorithms in Fig.
1 can be seen as the spread of the cluster corresponding to each few-shot protocol, e.g., the ellipse in Fig.
3 denotes the 5-shot 10-way protocol for Imagenet-21k.
It is the nature of few-shot learning that episodes have varying hardness even if the way and shot are ﬁxed.
However, episodes within the ellipse lie on a different line (with a large negative slope) which indicates that given a few-shot protocol, hardness is a good indicator of accuracy.
Fig.
3 also shows that due to fewer test classes, CIFAR-FS, FC-100 and Mini-ImageNet have less diversity in the hardness of episodes while Tiered-ImageNet and Imagenet-21k allow sampling of both very hard and very easy diverse episodes.
For a given few-shot protocol, the hardness of episodes in the former three is almost the same as that of the latter two datasets.
This indicates that CIFAR-FS,
FC-100 and Mini-ImageNet may be good benchmarks for applications with few classes.
The hardness metric in (9) naturally builds upon existing ideas in deep metric learning (Qi et al., 2018).
We propose it as a means to evaluate few-shot learning algorithms uniformly across different few-shot protocols for different datasets; ascertaining its efﬁcacy and comparisons to other metrics will be part of future work.
D I S C U S S I O N
Our aim is to provide grounding to the practice of few-shot learning.
The current literature is in the spirit of increasingly sophisticated approaches for modest improvements in mean accuracy using an inadequate evaluation methodology.
This is why we set out to establish a baseline, namely transductive ﬁne-tuning, and a systematic evaluation methodology, namely the hardness metric.
We would like to emphasize that our advocated baseline, namely transductive ﬁne-tuning, is not novel and yet performs better than existing algorithms on all standard benchmarks.
This is indeed surprising and indicates that we need to take a step back and re-evaluate the status quo in few-shot learning.
We hope to use the results in this paper as guidelines for the development of new algorithms.
A S E T U P A. D ATASETS
We use the following datasets for our benchmarking experiments.
• The Mini-ImageNet dataset (Vinyals et al., 2016) which is a subset of Imagenet-1k (Deng et al., 2009) and consists of 84 × 84 sized images with 600 images per class.
There aretraining, 16 validation and 20 test classes.
There are multiple versions of this dataset in the literature; we obtained the dataset from the authors of Gidaris & Komodakis (2018) 3 .
• The Tiered-ImageNet dataset (Ren et al., 2018) is a larger subset of Imagenet-1k withclasses split as 351 training, 97 validation and 160 testing classes, each with aboutimages of size 84 × 84.
This dataset ensures that training, validation and test classes do not have a semantic overlap and is a potentially harder few-shot learning dataset.
• We also consider two smaller CIFAR-100 (Krizhevsky & Hinton, 2009) derivatives, both with 32 × 32 sized images and 600 images per class.
The ﬁrst is the CIFAR-FS dataset (Bertinetto et al., 2018) which splits classes randomly into 64 training, 16 validation andtest.
The second is the FC-100 dataset (Oreshkin et al., 2018) which splits CIFAR-100 into 60 training, 20 validation and 20 test classes with minimal semantic overlap.
Each dataset has a training, validation and test set.
The set of classes for each of these sets are disjoint from each other.
For meta-training, we ran two sets of experiments: the ﬁrst, where we only use the training set as the meta-training dataset, denoted by (train); the second, where we use both the training and validation sets as the meta-training dataset, denoted by (train + val).
We use the test set to construct few-shot episodes.
A. P RE - TRAINING
We use a wide residual network (Zagoruyko & Komodakis, 2016; Qiao et al., 2018; Rusu et al., 2018) with a widening factor of 10 and a depth of 28 which we denote as WRN-28-10.
The smaller networks: conv (64) × 4 (Vinyals et al., 2016;
Snell et al., 2017), ResNet-12 (He et al., 2016a; Oreshkin et al., 2018; Lee et al., 2019) and WRN-16-4 (Zagoruyko & Komodakis, 2016), are used for analysis in Appendix C. All networks are trained using SGD with a batch-size of 256, Nesterov’s momentum set to 0.9, no dropout, weight decay of 10 − 4 .
We use batch-normalization (Ioffe & Szegedy, 2015).
We use two-cycles of learning rate annealing (Smith, 2017), these are 40 and 80 epochs each for all datasets except Imagenet-21k, which uses cycles of 8 and 16 epochs each.
The learning rate is set to 0.1 at the beginning of each cycle and decreased to 10 − 6 with a cosine schedule (Loshchilov &
Hutter, 2016).
We use data parallelism across 8 Nvidia V100 GPUs and half-precision training using techniques from Micikevicius et al.
; Howard et al.
We use the following regularization techniques that have been discovered in the non-few-shot, standard image classiﬁcation literature (Xie et al., 2018) for pre-training the backbone.
• Mixup (Zhang et al., 2017):
This augments data by a linear interpolation between input images and their one-hot labels.
If ( x 1 , y 1 ) , ( x 2 , y 2 ) ∈ D are two samples, mixup creates a new sample (˜ x, ˜ y ) where ˜ x = λx 1 +(1 − λ ) x 2 and its label ˜
y = λe y
+(1 − λ )
e y 2 ; here e k is the one-hot vector with a non-zero k th entry and λ ∈ [0 , 1] is sampled from Beta( α, α ) for a hyper-parameter α .
• Label smoothing (Szegedy et al., 2016):
When using a softmax operator, the logits can increase or decrease in an unbounded manner causing numerical instabilities while training.
Label smoothing sets p θ ( k | x ) = 1 − (cid:15) if k = y and (cid:15)/ ( K − 1) otherwise, for a small constant (cid:15) > 0 and number of classes K .
The ratio between the largest and smallest output neuron is thus ﬁxed which helps large-scale training.
• We exclude the batch-normalization parameters from weight-decay (Jia et al., 2018).
We set (cid:15) = 0.1 for label smoothing cross-entroy loss and α = 0.25 for mixup regularization for all our experiments.
A. F INE - TUNING HYPER - PARAMETERS
We used 1-shot 5-way episodes on the validation set of Mini-ImageNet to manually tune hyperparameters.
Fine-tuning is done for 25 epochs with a ﬁxed learning rate of 5 × 10 − 5 with Adam (Kingma & Ba, 2014).
Adam is used here as it is more robust to large changes in the magnitude of the loss and gradients which occurs if the number of classes in the few-shot episode (ways) is large.
We do not use any regularization (weight-decay, mixup, dropout, or label smoothing) in the ﬁne-tuning phase.
These hyper-parameters are kept constant on all benchmark datasets , namely
Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100.
All ﬁne-tuning and evaluation is performed on a single GPU in full-precision.
We update the parameters sequentially by computing the gradient of the two terms in (8) independently.
This updates both the weights of the model and the batch-normalization parameters.
A. D ATA AUGMENTATION
Input images are normalized using the mean and standard-deviation computed on Imagenet-1k.
Our
Data augmentation consists of left-right ﬂips with probability of 0.5, padding the image with 4px and adding brightness and contrast changes of ± 40%.
The augmentation is kept the same for both pre-training and ﬁne-tuning.
We explored augmentation using afﬁne transforms of the images but found that adding this has minor effect with no particular trend on the numerical results.
A. E VALUATION PROCEDURE
The few-shot episode contains classes that are uniformly sampled from the test classes of corresponding datasets.
Support and query samples are further uniformly sampled for each class.
The query shot is ﬁxed to 15 for all experiments unless noted otherwise.
We evaluate all networks over 1, episodes unless noted otherwise.
For ease of comparison, we report the mean accuracy and the 95% conﬁdence interval of the estimate of the mean accuracy.
B S E T U P F O R I M AG E N E T - 2 1 K
[Figure]
The Imagenet-21k dataset (Deng et al., 2009) has 14.2M images across 21,814 classes.
The blue region in Fig.
4 denotes our meta-training set with 7,491 classes, each with more than 1,000 images.
The green region shows 13,007 classes with at least 10 images each, the set used to construct few-shot episodes.
We do not use the red region consisting of 1,343 classes with less than 10 images each.
We train the same backbone (WRN-28-10) with the same procedure as that in Appendix A on 84 × 84 resized images, albeit for only 24 epochs.
Since we use the same hyper-parameters as the other benchmark datasets, we did not create validation sets for pre-training or the ﬁne-tuning phases.
The few-shot episodes are constructed in the same way as Appendix A. We evaluate using fewer few-shot episodes (80) on this dataset because we would like to demonstrate the performance across a large number of different ways.
C A D D I T I O NA L A NA LY S I S
This section contains additional experiments and analysis, complementing Section 4.3.
All experiments use the (train + val) setting, pre-training on both the training and validation sets of the corresponding datasets, unless noted otherwise.
C. T RANSDUCTIVE FINE - TUNING CHANGES THE EMBEDDING DRAMATICALLY
Fig.
5 demonstrates this effect.
The logits for query samples are far from those of their respective support samples and metric-based loss functions, e.g., those for prototypical networks (Snell et al., 2017) would have a high loss on this episode; indeed the accuracy after the support-based initialization is 64%.
Logits for the query samples change dramatically during transductive ﬁne-tuning and majority of the query samples cluster around their respective supports.
The post transductive ﬁne-tuning accuracy of this episode is 73.3%.
This suggests that modifying the embedding using the query samples is crucial to obtaining good performance on new classes.
This example also demonstrates that the support-based initialization is efﬁcient, logits of the support samples are relatively unchanged during the transductive ﬁne-tuning phase.
C. L ARGE VS . SMALL BACKBONES
The expressive power of the backbone plays an important role in the efﬁcacy of ﬁne-tuning.
We observed that a WRN-16-4 architecture (2.7M parameters) performs worse than WRN-28-10 (36M parameters).
The former obtains 63.28 ± 0.68% and 77.39 ± 0.5% accuracy on Mini-ImageNet and 69.04 ± 0.69% and 83.55 ± 0.51% accuracy on Tiered-ImageNet on 1-shot 5-way and 5-shot 5-way protocols respectively.
While these numbers are comparable to those of state-of-the-art algorithms, they are lower than their counterparts for WRN-28-10 in Table 1.
This suggests that a larger network is effective in learning richer features from the meta-training classes, and ﬁne-tuning is effective in taking advantage of this to further improve performance on samples belonging to few-shot classes.
C. L ATENCY WITH A SMALLER BACKBONES
The WRN-16-4 architecture (2.7M parameters) is much smaller than WRN-28-10 (36M parameters) and transductive ﬁne-tuning on the former is much faster.
As compared to our implementation of
Snell et al.
(2017) with the same backbone, WRN-16-4 is 20-70 × slower (0.87 vs. 0.04 seconds for a query shot of 1, and 2.85 vs. 0.04 seconds for a query shot of 15) for the 1-shot 5-way scenario.
Compare this to the computational complexity experiment in Section 4.3.
As discussed in Appendix C.2, the accuracy of WRN-16-4 is 63.28 ± 0.68% and 77.39 ± 0.5% for 1-shot 5-way and 5-shot 5-way on Mini-ImageNet respectively.
As compared to this, our implementation of (Snell et al., 2017) using a WRN-16-4 backbone obtains 57.29 ± 0.40% and 75.34 ± 0.32% accuracies for the same settings respectively; the former number in particular is signiﬁcantly worse than its transductive ﬁne-tuning counterpart.
C. C OMPARISONS AGAINST BACKBONES IN THE CURRENT LITERATURE
We include experiments using conv (64) × 4 and ResNet-12 in Table 3, in addition to WRN-28-10 in
Section 4, in order to facilitate comparisons of the proposed baseline for different backbone architectures.
Our results are comparable or better than existing results for a given backbone architecture, except for those in Liu et al.
(2018b) who use a graph-based transduction algorithm, for conv (64) ×on Mini-ImageNet.
In line with our goal for simplicity, we kept the hyper-parameters for pre-training and ﬁne-tuning the same as the ones used for WRN-28-10 (cf.
Sections 3 and 4).
These results suggest that transductive ﬁne-tuning is a sound baseline for a variety of backbone architectures.
C. U SING MORE META - TRAINING CLASSES
In Section 4.1 we observed that having more pre-training classes improves few-shot performance.
But since we append a classiﬁer on top of a pre-trained backbone and use the logits of the backbone as inputs to the classiﬁer, a backbone pre-trained on more classes would also have more parameters as compared to one pre-trained on fewer classes.
However, this difference is not large: WRN-28for Mini-ImageNet has 0.03% more parameters for (train + val) as compared to (train).
However, in order to facilitate a fair comparison, we ran an experiment where we use the features of the backbone, instead of the logits, as inputs to the classiﬁer.
By doing so, the number of parameters in the pre-trained backbone that are used for few-shot classiﬁcation remain the same for both the (train) and (train + val) settings.
For Mini-ImageNet, (train + val) obtains 64.20 ± 0.65% and 81.
± 0.45%, and (train) obtains 62.55 ± 0.65% and 78.89 ± 0.46%, for 1-shot 5-way and 5-shot 5-way respectively.
These results corroborate the original statement that more pre-training classes improves few-shot performance.
C. U SING FEATURES OF THE BACKBONE AS INPUT TO THE CLASSIFIER
Instead of re-initializing the ﬁnal fully-connected layer of the backbone to classify new classes, we simply append the classiﬁer on top of it.
We implemented the former, more common, approach and found that it achieves an accuracy of 64.20 ± 0.65% and 81.26 ± 0.45% for 1-shot 5-way and 5-shot 5-way respectively on Mini-ImageNet, while the accuracy on Tiered-ImageNet is 67.14 ± 0.74% and 86.67 ± 0.46% for 1-shot 5-way and 5-shot 5-way respectively.
These numbers are signiﬁcantly lower for the 1-shot 5-way protocol on both datasets compared to their counterparts in Table 1.
However, the 5-shot 5-way accuracy is marginally higher in this experiment than that in Table 1.
As noted in Remark 2, logits of the backbone are well-clustered and that is why they work better for few-shot scenarios.
[Table]
C. F REEZING THE BACKBONE RESTRICTS PERFORMANCE
The previous observation suggests that the network changes a lot in the ﬁne-tuning phase.
Freezing the backbone severely restricts the changes in the network to only changes to the classiﬁer.
As a consequence, the accuracy of freezing the backbone is 58.38 ± 0.66 % and 75.46 ± 0.52% on MiniImageNet and 67.06 ± 0.69% and 83.20 ± 0.51% on Tiered-ImageNet for 1-shot 5-way and 5-shot 5-way respectively.
While the 1-shot 5-way accuracies are much lower than their counterparts in
Table 1, the gap in the 5-shot 5-way scenario is smaller.
C. U SING MIXUP DURING PRE - TRAINING
Mixup improves the few-shot accuracy by about 1%; the accuracy for WRN-28-10 trained without mixup is 67.06 ± 0.71% and 79.29 ± 0.51% on Mini-ImageNet for 1-shot 5-way and 5-shot 5-way respectively. C. M ORE FEW - SHOT EPISODES Fig.
1 suggests that the standard deviation of the accuracies achieved by few-shot algorithms is high.
Considering this randomness, evaluations were done over 10,000 few-shot episodes as well.
The accuracies on Mini-ImageNet are 67.77 ± 0.21 % and 80.24 ± 0.16 % and on Tiered-ImageNet are 72.36 ± 0.23 % and 85.70 ± 0.16 % for 1-shot 5-way and 5-shot 5-way respectively.
The numbers are consistent with the ones for 1,000 few-shot episodes in Table 1, though the conﬁdence intervals decreased as the number of episodes sampled increased. C. E VALUATION ON M ETA -D ATASET We ran experiments on Meta-Dataset (Triantaﬁllou et al., 2019), and compared the performance of transductive ﬁne-tuning for meta-training done on Imagenet-1k (ILSVRC) in Table 4. Transductive ﬁne-tuning is better, most times signiﬁcantly, than state-of-the-art on 6 out of 8 tasks in Meta-Dataset; the average rank across all tasks is 1.4375.
The Fungi and Quick Draw datasets were not included because of issues with getting the data; the link to access the dataset for the former does not seem to work and the latter requires certain legal conditions which we are working on obtaining.
The few-shot episode sampling was done the same way as in Triantaﬁllou et al.
except for the few-shot class sampling for Imagenet-1k and Omniglot, which was done uniformly over all few-shot classes (Triantaﬁllou et al.
(2019) uses a hierarchical sampling technique to sample classes that are far from each other in the hierarchy, and hence easier to distinguish between).
The hyper-parameters used for meta-training and few-shot ﬁne-tuning are kept the same as the ones in Section 4.
[Table]
[Equation]
Why has it not been noticed yet that this simple approach works so well?
Non-transductive ﬁne-tuning as a baseline has been considered before (Vinyals et al., 2016;
Chen et al., 2018).
The fact that this is comparable to state-of-the-art has probably gone unnoticed because of the following reasons:
• Given that there are only a few labeled support samples provided in the few-shot setting, initializing the classiﬁer becomes important.
The support-based initialization (cf. Section 3.1) motivated from the deep metric learning literature (Hu et al., 2015; Movshovitz-Attias et al. , 2017; Qi et al., 2018; Gidaris & Komodakis, 2018) classiﬁes support samples correctly (for a support shot of 1, this may not be true for higher support shots).
This initialization, as opposed to initializing the weights of the classiﬁer randomly, was critical to performance in our experiments.
• In our experience, existing meta-training methods, both gradient-based ones and metricbased ones, are difﬁcult to tune for larger architectures.
We speculate that this is the reason a large part of the existing literature focuses on smaller backbone architectures.
The few-shot learning literature has only recently started to move towards bigger backbone architectures (Oreshkin et al., 2018; Rusu et al., 2018).
From Table 3 we see that non-tranductive ﬁnetuning gets better with a deeper backbone architecture.
A similar observation was made by (Chen et al., 2018).
The observation that we can use “simple” well-understood training techniques from standard supervised learning that scale up to large backbone architectures for few-shot classiﬁcation is a key contribution of our paper.
Transductive methods have recently started to become popular in the few-shot learning literature (Nichol et al., 2018;
Liu et al., 2018a).
Because of the scarcity of labeled support samples, it is crucial to make use of the unlabeled query samples in the few-shot regime.
Our advocated baseline makes use of both a good initialization and transduction, relatively new in the few-shot learning literature, which makes this simplistic approach go unrecognized till now
Transductive ﬁne-tuning works better than existing algorithms because of a big backbone architecture.
One should compare on the same backbone architectures as the existing algorithms for a fair comparison.
The current literature is in the spirit of increasingly sophisticated approaches for modest performance gains, often with different architectures (cf.
Table 1).
This is why we set out to establish a baseline.
Our simple baseline is comparable or better than existing approaches.
The backbone we have used is common in the recent few-shot learning literature (Rusu et al., 2018;
Qiao et al., 2018)
(cf.
Table 1).
Additionally, we have included results on smaller common backbone architectures, namely conv (64) × 4 and ResNet-12 in Appendix C.4, and some additional experiments in
Appendix C.2.
These experiments suggest that transductive ﬁne-tuning is a sound baseline for a variety of different backbone architectures.
This indicates that we should take results on existing benchmarks with a grain of salt.
Also see the response to question 1 above.
There are missing entries in Tables 1 and
Is it still a fair comparison?
Tables 1 and 3 show all relevant published results by the original authors.
Re-implementing existing algorithms to ﬁll missing entries without access to original code is impractical and often yields results inferior to those published, which may be judged as unfair.
The purpose of a benchmark is to enable others to test their method easily.
This does not exist today due to myriad performance-critical design choices often not detailed in the papers.
In fact, missing entries in the table indicate the inadequate state of the current literature.
Our work enables benchmarking relative to a simple, systematic baseline.
Fine-tuning for few-shot learning is not novel.
We do not claim novelty in this paper.
Transductive ﬁne-tuning is our advocated baseline for few-shot classiﬁcation.
It is a combination of different techniques that are not novel.
Yet, it performs better than existing algorithms on all few-shot protocols with ﬁxed hyper-parameters.
We emphasize that this indicates the need to re-interpret existing results on benchmarks and re-evaluate the status quo in the literature.
Transductive ﬁne-tuning has a very high latency at inference time, this is not practical.
Our goal is to establish a systematic baseline for accuracy, which might help judge the accuracy of few-shot learning algorithms in the future.
The question of test-time latency is indeed important but we have not focused on it in this paper.
Appendix C.3 provides results using a smaller backbone where we see that the WRN-16-4 network is about 20-70x slower than metric-based approaches employing the same backbone while having signiﬁcantly better accuracy.
The latencies with
WRN-28-10 are larger (see the computational complexity section in Section 4.3) but with a bigger advantage in terms of accuracy.
There are other transductive methods used for few-shot classiﬁcation (Nichol et al., 2018; Liu et al., 2018a), that are expected to be slow as well.
Transductive ﬁne-tuning does not make sense in the online setting when query samples are shown in a sequence.
Transductive ﬁne-tuning can be performed even with a single test datum.
Indeed, the network can specialize itself completely to classify this one datum.
We explore a similar scenario in Section 4. and Fig.
2a, which discuss the performance of transductive ﬁne-tuning with a query shot of 1 (this means 5 query samples one from each class for 5-way evaluation).
Note that the loss function in (8) leverages multiple query samples when available.
It does not require that the query samples be balanced in terms of their ground-truth classes.
In particular, the loss function in (8) is well-deﬁned even for a single test datum.
For concerns about latency, see the question 5 above.
Having transductive approaches will incentivize hacking the query set
There are already published methods that use transductive methods (Nichol et al., 2018; Liu et al., 2018a), and it is a fundamental property of the transductive paradigm to be dependent on the query set, in addition to the support set.
In order to prevent query set hacking, we will make the test episodes public which will enable consistent benchmarking, even for transductive methods. Why is having the same
hyper- parameters for different few-shot protocols so important
A practical few-shot learning algorithm should be able to handle any few-shot protocol.
Having one model for each different scenario is unreasonable in the real-world, as the number of different scenarios is, in principle, inﬁnite.
Current algorithms do not handle this well.
A single model which can handle any few-shot scenario is thus desirable.
Is this over-ﬁtting to the test datum?
No , label of the test datum is not used in the loss function
Can you give some intuition about the hardness metric?
How did you come up with the formula?
The hardness metric is the clustering loss where the labeled support samples form the centers of the class-speciﬁc clusters.
The special form, namely, E ( x,y ) ∈D q log 1 − p ( y | x ) p ( y | x ) (cf.
(9)) allows an interpretation of log-odds.
We used this form because it is sensitive to the number of fewshot classes (cf. Fig. 3).
Similar metrics, e.g., E ( x,y ) ∈D q [ − log p ( y | x )] can also be used but they come with a few caveats.
Note that it is easier for p ( y | x ) to be large for small way because the normalization constant in softmax has fewer terms.
For large way, p ( y | x ) could be smaller.
This effect is better captured by our metric.
11. How does Fig. 3 look for algorithm X, Y, Z?
We compared two algorithms in Fig.
3, namely transductive ﬁne-tuning and support-based initialization.
Section 4.4 and the caption of Fig.
3 explains how the former algorithm is better.
We will consider adding comparisons to other algorithms to this plot in the future.