A C LOSER L OOK AT F EW - SHOT C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot classiﬁcation aims to learn a classiﬁer to recognize unseen classes during training with limited labeled examples.
While signiﬁcant progress has been made, the growing complexity of network designs, meta-learning algorithms, and differences in implementation details make a fair comparison difﬁcult.
In this paper, we present 1) a consistent comparative analysis of several representative few-shot classiﬁcation algorithms, with results showing that deeper backbones signiﬁcantly reduce the gap across methods when domain differences are limited, 2) a slightly modiﬁed baseline method that surprisingly achieves competitive performance when compared with the state-of-the-art on both the mini -ImageNet and the CUB datasets, and 3) a new experimental setting for evaluating the crossdomain generalization ability for few-shot classiﬁcation algorithms.
Our results reveal that reducing intra-class variation is an important factor when the feature backbone is shallow, but not as critical when using deeper backbones.
In a realistic, cross-domain evaluation setting, we show that a baseline method with a standard ﬁne-tuning practice compares favorably against other state-of-the-art few-shot learning algorithms.
I NTRODUCTION
Deep learning models have achieved state-of-the-art performance on visual recognition tasks such as image classiﬁcation.
The strong performance, however, heavily relies on training a network with abundant labeled instances with diverse visual variations (e.g., thousands of examples for each new class even with pre-training on large-scale dataset with base classes).
The human annotation cost as well as the scarcity of data in some classes (e.g., rare species) signiﬁcantly limit the applicability of current vision systems to learn new visual concepts efﬁciently.
In contrast, the human visual systems can recognize new classes with extremely few labeled examples.
It is thus of great interest to learn to generalize to new classes with a limited amount of labeled examples for each novel class.
The problem of learning to generalize to unseen classes during training, known as few-shot classiﬁcation , has attracted considerable attention Vinyals et al.
; Snell et al.
Finn et al.
(2017); Ravi & Larochelle (2017); Sung et al.
(2018); Garcia & Bruna (2018);
Qi et al.
One promising direction to few-shot classiﬁcation is the meta-learning paradigm where transferable knowledge is extracted and propagated from a collection of tasks to prevent overﬁtting and improve generalization.
Examples include model initialization based methods Ravi & Larochelle (2017);
Finn et al.
(2017), metric learning methods Vinyals et al.
; Snell et al.
; Sung et al.
(2018), and hallucination based methods Antoniou et al.
(2018); Hariharan & Girshick (2017);
Wang et al.
Another line of work Gidaris & Komodakis (2018); Qi et al.
(2018) also demonstrates promising results by directly predicting the weights of the classiﬁers for novel classes.
Limitations.
While many few-shot classiﬁcation algorithms have reported improved performance over the state-of-the-art, there are two main challenges that prevent us from making a fair comparison and measuring the actual progress.
First, the discrepancy of the implementation details among multiple few-shot learning algorithms obscures the relative performance gain.
The performance of baseline approaches can also be signiﬁcantly under-estimated (e.g., training without data augmentation).
Second, while the current evaluation focuses on recognizing novel class with limited training examples, these novel classes are sampled from the same dataset.
The lack of domain shift between base and novel classes makes the evaluation scenarios unrealistic.
Our work.
In this paper, we present a detailed empirical study to shed new light on the few-shot classiﬁcation problem.
First, we conduct consistent comparative experiments to compare several representative few-shot classiﬁcation methods on common ground.
Our results show that using a deep backbone shrinks the performance gap between different methods in a setting with limited domain differences.
Second, by replacing the linear classiﬁer with a distance-based classiﬁer as used in Gidaris & Komodakis (2018)
; Qi et al.
, the baseline method is surprisingly competitive to current state-of-art meta-learning algorithms.
Third, we introduce a realistic evaluation setting where there exists domain shift between base and novel classes (e.g., sampling base classes from generic object categories and novel classes from ﬁne-grained categories).
Our results show that sophisticated few-shot learning algorithms do not provide performance improvement over the baseline under this setting.
Through making the source code and model implementations with a consistent evaluation setting publicly available, we hope to foster future progress in the ﬁeld.
Our contributions.
We provide a uniﬁed testbed for several different few-shot classiﬁcation algorithms for a fair comparison.
Our empirical evaluation results reveal that the use of a shallow backbone commonly used in existing work leads to favorable results for methods that explicitly reduce intra-class variation.
Increasing the model capacity of the feature backbone reduces the performance gap between different methods when domain differences are limited.
We show that a baseline method with a distance-based classiﬁer can be surprisingly competitive with the state-of-the-art meta-learning methods on both mini -ImageNet and CUB datasets.
We investigate a realistic evaluation setting where base and novel classes are sampled from different domains.
We show that current few-shot classiﬁcation algorithms fail to address such domain shifts and are inferior to the baseline method, highlighting the importance of learning to adapt to domain differences in few-shot learning.
R ELATED W ORK
Given abundant training examples for the base classes, few-shot learning algorithms aim to learn to recognizing novel classes with a limited amount of labeled examples.
Much efforts have been devoted to overcome the data efﬁciency issue.
In the following, we discuss representative few-shot learning algorithms organized into three main categories: initialization based, metric learning based, and hallucination based methods.
Initialization based methods tackle the few-shot learning problem by “learning to ﬁne-tune”.
One approach aims to learn good model initialization (i.e., the parameters of a network) so that the classiﬁers for novel classes can be learned with a limited number of labeled examples and small number of gradient steps Finn et al.
Another line of work focuses on learning an optimizer .
Examples include the LSTM-based meta-learner for replacing the stochastic gradient decent optimizer Ravi & Larochelle (2017) and the weight-update mechanism with an external memory Munkhdalai & Yu (2017).
While these initialization based methods are capable of achieving rapid adaption with a limited number of training examples for novel classes, our experiments show that these methods have difﬁculty in handling domain shifts between base and novel classes.
Hallucination based methods directly deal with data deﬁciency by “learning to augment”.
This class of methods learns a generator from data in the base classes and use the learned generator to hallucinate new novel class data for data augmentation.
One type of generator aims at transferring appearance variations exhibited in the base classes.
These generators either transfer variance in base class data to novel classes Hariharan & Girshick (2017), or use GAN models Antoniou et al.
(2018) to transfer the style.
Another type of generators does not explicitly specify what to transfer, but directly integrate the generator into a meta-learning algorithm for improving the classiﬁcation accuracy Wang et al.
Since hallucination based methods often work with other few-shot methods together (e.g. use hallucination based and metric learning based methods together) and lead to complicated comparison, we do not include these methods in our comparative study and leave it for future work.
Distance metric learning based methods address the few-shot classiﬁcation problem by “learning to compare”.
The intuition is that if a model can determine the similarity of two images, it can [Figure] classify an unseen input image with the labeled instances Koch et al.
To learn a sophisticated comparison models, meta-learning based methods make their prediction conditioned on distance or metric to few labeled instances during the training process.
Examples of distance metrics include cosine similarity Vinyals et al.
(2016), Euclidean distance to class-mean representation Snell et al.
(2017), CNN-based relation module Sung et al.
(2018), and graph neural network Garcia & Bruna (2018).
In this paper, we compare the performance of three distance metric learning methods.
Our results show that a simple baseline method with a distance-based classiﬁer (without training over a collection of tasks/episodes as in meta-learning) achieves competitive performance with respect to other sophisticated algorithms.
Besides meta-learning methods, both Gidaris & Komodakis (2018) and Qi et al.
(2018) develop a similar method to our Baseline++ (described later in Section 3.2).
The method in Gidaris &
Komodakis (2018) learns a weight generator to predict the novel class classiﬁer using an attentionbased mechanism (cosine similarity), and the Qi et al.
(2018) directly use novel class features as their weights.
Our Baseline++ can be viewed as a simpliﬁed architecture of these methods because our focus is to show that simply reducing intra-class variation in a baseline method using the base class data leads to competitive performance.
Domain adaptation techniques aim to reduce the domain shifts between source and target domain
Pan et al. (2010); Ganin & Lempitsky (2015), as well as novel tasks in a different domain Hsu et al.
Similar to domain adaptation, we also investigate the impact of domain difference on few-shot classiﬁcation algorithms in Section 4.5.
In contrast to most domain adaptation problems where a large amount of data is available in the target domain (either labeled or unlabeled), our problem setting differs because we only have very few examples in the new domain.
Very recently, Dong et al.
Dong & Xing (2018) address the one-shot novel category domain adaptation problem, where both the domain and the category to classify are changed in the testing stage.
Our work instead focuses on the few-shot classiﬁcation performance under domain differences.
O VERVIEW OF F EW - SHOT CLASSIFICATION A LGORITHMS
In this section, we ﬁrst outline the details of the baseline model (Section 3.1) and its variant (Section 3.2), followed by describing representative meta-learning algorithms (Section 3.3) studied in our experiments.
Given abundant base class labeled data X b and a small amount of novel class labeled data X n , the goal of few-shot classiﬁcation algorithms is to train classiﬁers for novel classes (unseen during training) with few labeled examples.
3. B ASELINE
Our baseline model follows the standard transfer learning procedure of network pre-training and ﬁne-tuning.
Figure 1 illustrates the overall procedure.
Training stage.
We train the feature extractor f θ (parametrized by the network parameters θ ) and the classiﬁer C ( ·| W b ) (parametrized by the weight matrix W b ∈ R d × c ) from scratch by minimizing a standard cross-entropy classiﬁcation loss
L pred using the training examples in the base classes x i ∈ X b .
Here, we denote the dimension of the encoded feature as d and the number of output classes as c .
The classiﬁer C ( .
| W b ) consists of a linear layer W (cid:62) b
f θ ( x i ) followed by a softmax function σ .
Fine-tuning stage.
To adapt the model to recognize novel classes in the ﬁne-tuning stage, we ﬁx the pre-trained network parameter θ in our feature extractor f θ and train a new classiﬁer C ( .
| W n ) (parametrized by the weight matrix W n ) by minimizing L pred using few labeled of examples (i.e., the support set) in the novel classes X n .
3. B ASELINE ++
In addition to the baseline model, we also implement a variant of the baseline model, denoted as
Baseline++, which explicitly reduces intra-class variation among features during training.
The importance of reducing intra-class variations of features has been highlighted in deep metric learning Hu et al.
and few-shot classiﬁcation methods Gidaris & Komodakis (2018).
The training procedure of Baseline++ is the same as the original Baseline model except for the classiﬁer design.
As shown in Figure 1, we still have a weight matrix W b ∈ R d × c of the classiﬁer in the training stage and a W n in the ﬁne-tuning stage in Baseline++.
The classiﬁer design, however, is different from the linear classiﬁer used in the Baseline.
Take the weight matrix W b as an example.
We can write the weight matrix W b as [ w 1 , w 2 ,...
w c ] , where each class has a d -dimensional weight vector.
In the training stage, for an input feature f θ ( x i ) where x i ∈ X b , we compute its cosine similarity to each weight vector [ w 1 , ··· , w c ] and get the similarity scores [ cos ( f θ ( x i ) , w 1 ) , ··· , cos ( f θ ( x i ) , w c )] for all classes.
We can then obtain the prediction probability for each class by normalizing these similarity scores with a softmax function.
Here, the classiﬁer makes a prediction based on the cosine distance between the input feature and the learned weight vectors representing each class.
Consequently, training the model with this distance-based classiﬁer explicitly reduce intra-class variations.
Intuitively, the learned weight vectors [ w 1 , ··· , w c ] can be interpreted as prototypes (similar to Snell et al.
(2017); Vinyals et al.
for each class
and the classiﬁcation is based on the distance of the input feature to these learned prototypes.
The softmax function prevents the learned weight vectors collapsing to zeros.
We clarify that the network design in Baseline++ is not our contribution.
The concept of distancebased classiﬁcation has been extensively studied in Mensink et al.
and recently has been revisited in the few-shot classiﬁcation setting Gidaris & Komodakis (2018);
Qi et al.
3. M ETA - LEARNING ALGORITHMS
Here we describe the formulations of meta-learning methods used in our study.
We consider three distance metric learning based methods (MatchingNet Vinyals et al.
ProtoNet Snell et al.
(2017), and
RelationNet Sung et al.
and one initialization based method (MAML Finn et al.
While meta-learning is not a clearly deﬁned, Vinyals et al.
(2016) considers a few-shot classiﬁcation method as meta-learning if the prediction is conditioned on a small support set S , because it makes the training procedure explicitly learn to learn from a given small support set.
As shown in Figure 2, meta-learning algorithms consist of a meta-training and a meta-testing stage.
In the meta-training stage, it would ﬁrst randomly select N classes, and sample small base support set S b and a base query set Q b from data samples within these classes.
The objective is to train a classiﬁcation model M that minimizes N -way prediction loss L N − way of the samples in the query set Q b .
The classiﬁer M is conditioned on support set S b .
That is, given a different support set, the prediction would be different (similar to the nearest neighbor classiﬁer).
By making prediction conditioned on the given support set, a meta-learning method can learn how to learn from limited data through training from a collection of tasks (episodes).
In the meta-testing stage, all novel class
[Figure]
data X n are considered as the support set for novel classes S n , and the classiﬁcation model M can be adapted to predict novel classes with the new support set S n .
Different meta-learning methods differ in their strategies to make prediction conditioned on support set (see Figure 2).
For both MatchingNet Vinyals et al.
and ProtoNet Snell et al.
(2017), the prediction of the examples in a query set Q is based on comparing the distance between the query feature and the support feature from each class.
MatchingNet compares cosine distance between the query feature and each support feature, and computes average cosine distance for each class, while
ProtoNet compares the Euclidean distance between query features and the class mean of support features.
RelationNet Sung et al.
(2018) shares a similar idea, but it replaces distance with a learnable relation module.
The MAML method Finn et al.
(2017) is an initialization based meta-learning algorithm, where each support set is used to adapt the initial model parameters using few gradient updates.
As different support sets have different gradient updates, the adapted model is conditioned on the support set.
Note that when the query set instances are predicted by the adapted model in the meta-training stage, the loss of the query set is used to update the initial model, not the adapted model.
E XPERIMENTAL R ESULTS 4. E XPERIMENTAL SETUP
Datasets and scenarios.
We address the few-shot classiﬁcation problem under three scenarios: 1) generic object recognition, 2) ﬁne-grained image classiﬁcation, and 3) cross-domain adaptation.
For object recognition, we use the mini -ImageNet dataset commonly used in evaluating few-shot classiﬁcation algorithms.
The mini -ImageNet dataset consists of a subset of 100 classes from the
ImageNet dataset Deng et al.
and contains 600 images for each class.
It was ﬁrst proposed by Vinyals et al.
, but recent works use the follow-up setting provided by Ravi & Larochelle (2017), which is composed of randomly selected 64 base, 16 validation, and 20 novel classes.
For ﬁne-grained classiﬁcation, we use CUB-200-2011 Wah et al.
(2011) (referred to as the CUB hereafter) dataset.
The CUB dataset contains 200 classes and 11,788 images in total.
Following the evaluation protocol of Hilliard et al.
(2018), we randomly split the dataset into 100 base,validation, and 50 novel classes. [Table]
For the cross-domain scenario ( mini -ImageNet → CUB) [Table] we use mini -ImageNet as our base class and the 50 validation and 50 novel class from CUB.
Evaluating the cross-domain scenario allows us to understand the effects of domain shifts to existing few-shot classiﬁcation approaches.
Implementation details.
In the training stage for the Baseline and the Baseline++ methods, we train 400 epochs with a batch size of 16.
In the meta-training stage for meta-learning methods, we train 60,000 iterations for 1-shot and 40,000 iterations for 5-shot tasks, and use the validation set to select the iterations with the best accuracy.
In each iteration, we sample N classes to form N -way classiﬁcation ( N is 5 in both meta-training and meta-testing stages unless otherwise mentioned).
For each class, we pick k labeled instances as our support set and 16 instances for the query set for a k -shot task.
In the ﬁne-tuning or meta-testing stage for all methods, we average the results over 600 experiments.
In each experiment, we randomly sample 5 classes from novel classes, and in each class, we also pick k instances for the support set and 16 for the query set.
For Baseline and Baseline++, we use the entire support set to train a new classiﬁer for 100 iterations with a batch size of 4.
For meta-learning methods, we obtain the classiﬁcation model conditioned on the support set as in Section 3.3.
All methods are trained from scratch and use the Adam optimizer with initial learning rate 10 − 3 .
We apply standard data augmentation including random crop, left-right ﬂip, and color jitter in both the training or meta-training stage.
Some implementation details have been adjusted individually for each method.
For MatchingNet, we use an FCE classiﬁcation layer without ﬁne-tuning in all experiments and also multiply cosine similarity by a constant scalar.
For RelationNet, we replace the L2 norm with a softmax layer to expedite training.
For MAML, we use a ﬁrst-order approximation in the gradient for efﬁciency.
The approximation has been shown in the original paper to have almost the same performance as the full version.
We choose the ﬁrst-order approximation for its efﬁciency.
4. S TANDARD S ETTING R EPRODUCTION
We now conduct experiments on the most common setting in few-shot classiﬁcation, 1-shot and 5-shot classiﬁcation, i.e., 1 or 5 labeled instances are available from each novel class.
We use a four-layer convolution backbone (Conv-4) as in Snell et al.
and perform 5-way classiﬁcation for only novel classes during the ﬁne-tuning or meta-testing stage.
To validate the correctness of our implementation, we ﬁrst compare our results to reported numbers for the mini -ImageNet dataset in Table 1.
Note that we have a ProtoNet # , as we use 5-way classiﬁcation in the meta-training and meta-testing stages for all meta-learning methods as mentioned a
[Table]
in Section 4.1; however, the ofﬁcial reported results from ProtoNet uses 30-way for one shot and 20-way for ﬁve shot in the meta-training stage in spite of using 5-way in the meta-testing stage.
We report this result for completeness
[Table]
From Table
[Table]
we can observe that all of our re-implementation for meta-learning methods do not fall more than 2% behind reported performance.
These minor differences can be attributed to our modiﬁcations of some implementation details to ensure a fair comparison among all methods, such as using the same optimizer for all methods.
Moreover, our implementation of existing work also improves the performance of some of the methods.
For example, our results show that the 5-shot Baseline can be improved by a large margin since previous implementations of the Baseline do not include data augmentation in their training stage, which leads to over-ﬁtting.
While our Baseline ∗ is not as good as reported in 1-shot, our Baseline with augmentation still improves on it, and could be even higher if our reproduced Baseline ∗ matches the reported statistics.
In either case, the performance of the Baseline method is severely underestimated .
We also improve the results of MatchingNet by adjusting the input score to the softmax layer to a more appropriate range as stated in Section 4.1.
On the other hand, while ProtoNet # is not as good as ProtoNet, as mentioned in the original paper a more challenging setting in the metatraining stage leads to better accuracy.
We choose to use a consistent 5-way classiﬁcation setting in subsequent experiments to have a fair comparison to other methods.
This issue can be resolved by using a deeper backbone as shown in Section 4.3.
After validating our reimplementation, we now report the accuracy in Table 2.
Besides additionally reporting results on the CUB dataset, we also compare Baseline++ to other methods.
Here, we ﬁnd that Baseline++ improves the Baseline by a large margin and becomes competitive even when compared with other meta-learning methods.
The results demonstrate that reducing intra-class variation is an important factor in the current few-shot classiﬁcation problem setting.
However, note that our current setting only uses a 4-layer backbone, while a deeper backbone could inherently reduce intra-class variation.
Thus, we conduct experiments to investigate the effects of backbone depth in the next section.
4. E FFECT OF INCREASING THE NETWORK DEPTH
Now we change the backbone depth to reduce intra-class variation for all methods.
See appendix for statistics on how network depth correlates with intra-class variation.
Starting from Conv-4, we gradually increase the feature backbone to Conv-6, ResNet-10, 18 and 34, where Conv-6 have two additional convolution blocks without pooling after Conv-4.
ResNet-18 and 34 are the same as described in He et al.
(2016), while ResNet-10 is a simpliﬁed version of ResNet-18 where only one residual building block is used in each layer.
The statistics of this experiment would also be helpful to other works to make a fair comparison under different backbones.
Results of the CUB dataset shows a clearer tendency in Figure 3.
As the backbone gets deeper, the gap among different methods reduces drastically.
Another observation is how ProtoNet improves rapidly as the backbone gets deeper.
While using a consistent 5-way classiﬁcation as discussed in
[Figure] Section 4.2 degrades the accuracy of ProtoNet with Conv-4, it works well with a deeper backbone.
Thus, the two observations above demonstrate that in the CUB dataset, the gap among existing methods would be reduced if their intra-class variation are all reduced by a deeper backbone.
However, the result of mini -ImageNet in Figure 3 is much more complicated.
In the 5-shot setting, both Baseline and Baseline++ achieve good performance with a deeper backbone, but some metalearning methods become worse relative to them.
Thus, other than intra-class variation, we can assume that the dataset is also important in few-shot classiﬁcation.
One difference between CUB and mini -ImageNet is their domain difference in base and novel classes since classes in mini -ImageNet have a larger divergence than CUB in a word-net hierarchy Miller (1995).
Thus, in the next section, we discuss how domain differences between base and novel classes impact few-shot results.
4. E FFECT OF DOMAIN DIFFERENCES BETWEEN BASE AND NOVEL CLASSES
To further dig into the issue of domain difference, we require scenarios that provide such variations.
Thus, besides the ﬁne-grained classiﬁcation and object recognition scenarios, we propose a new cross-domain scenario: mini -ImageNet → CUB as mentioned in Section 4.1.
We argue that this is quite a realistic scenario, since collecting images from a general class may be easy (e.g. due to increased availability) but collecting images from ﬁne-grained classes might be more difﬁcult.
Here we only discuss results with ResNet-18 since the importance of deeper backbone has been discussed before.
As shown in Table 3, the Baseline becomes better than meta-learning methods under this scenario.
While meta-learning methods learn to learn from the support set during the meta-training stage, they could not learn to adapt to novel classes that are too different since all of the base support sets are within the same dataset.
A similar concept is also mentioned in Vinyals et al.
In contrast, the Baseline simply replaces and trains a new classiﬁer based on the few given novel class data, which allows it to quickly adapt to a novel class and is less affected by domain difference.
The Baseline also beats Baseline++, possibly because additionally reducing intra-class variation compromises adaptability.
In Figure 4, we can further observe how Baseline accuracy becomes relatively higher as the domain difference gets larger.
That is, as the domain difference grows larger, the adaptation based on a few novel class instances becomes more important.
However, this is not to say that only Baseline can properly adapt.
In the next section, we discuss how to adapt further meta-learning methods based on given few examples.
4. E FFECT OF FURTHER ADAPTATION
To further adapt meta-learning methods as in Baseline, an intuitive way is to ﬁx the features and train a new softmax classiﬁer.
We apply this adaptation to MatchingNet and ProtoNet.
For MAML, it is not feasible to ﬁx the feature as it is merely an initialization.
In contrast, since it updates the model with the support set for only a few iterations, we can adapt further by updating for as many iterations as is required to train a new classiﬁcation layer, which is 100 updates as mentioned in Section 4.1.
For RelationNet, its features are convolution maps rather than the feature vectors, so we are not able to replace it with a softmax.
As an alternative, we randomly split the few training data in novel class into 3 support and 2 query data to ﬁnetune the relation module for 100 epochs.
[Table] [Figure] [Figure] The results of further adaptation are shown in Figure 5; we can observe that the performance of
MatchingNet and MAML improves signiﬁcantly after further adaptation, particularly in the miniImageNet → CUB scenario.
The results demonstrate that lack of adaptation is the reason they fall behind the Baseline.
However, changing the setting in the meta-testing stage could lead to inconsistency with the meta-training stage.
The ProtoNet result shows that performance can degrade in scenarios with less domain difference.
Thus, learning how to adapt in the meta-training stage is important.
In summary, as domain differences are likely to exist in many real-world applications, we consider that learning to learn adaptation in the meta-training stage would be an important direction for future meta-learning works in few-shot classiﬁcation.
C ONCLUSIONS
In this paper, we have investigated the limits of the standard evaluation setting for few-shot classiﬁcation.
Through comparing methods on a common ground, our results show that the Baseline++ model is competitive to state of art under standard conditions, and the Baseline model achieves competitive performance with recent state-of-the-art meta-learning algorithms on both CUB and mini -ImageNet benchmark datasets when using a deeper feature backbone.
Surprisingly, the Baseline compares favorably against all the evaluated meta-learning algorithms under a realistic scenario where there exists domain shift between the base and novel classes.
By making our source code publicly available, we believe that community can beneﬁt from the consistent comparative experiments and move forward to tackle the challenge of potential domain shifts in the context of few-shot learning.
A PPENDIX
Different meta-learning works use different terminology in their work.
We mark their difference as shown in appendix Table A1 to reduce confusion.
[Table]
Beside, some works (Vinyals et al
[Table]
(2016); Ravi & Larochelle (2017)) use a Baseline with 1-NN classiﬁer in test stage.
We include our result as in Table A2.
The result shows that using 1-NN classiﬁer has better performance than that of using the softmax classiﬁer in 1-shot setting
[Table]
but softmax classiﬁer is better in 5-shot setting instead.
We note that the number presented here are not directly comparable to results in Vinyals et al.
(2016) because we use a different mini -ImageNet separation as in Ravi & Larochelle (2017)
[Table]
[Table]
Also, as mentioned in Section 4.3, here we demonstrate that intra-class variation would be reduced when network depth gets deeper as in Figure A1.
Here we use Davies-Bouldin index Davies &
Bouldin (1979) to represent intra-class variation, which is a metric to evaluate the tightness in a cluster (or class, in our case).
We could observe both intra-class variation in base and novel class feature would be reduced in a deeper backbone.
[Figure]
Baseline
Baseline++ MatchingNet ProtoNet Base class feature Novel class feature
D a v i e s - B ou l d i n i nd e
x
(I n t r a - c l a ss v a r i a ti on ) By the way [Table] we show a larger version of Figure 3 in Figure A2 and show detailed statistics in
Table A3 for easier comparison.
[Table]
[Figure]
Baseline
Baseline++
MatchingNet
ProtoNet
MAML
RelationNet
CUB mini -ImageNet
[Figure]
[Table] [Table] Finally, we show a supplementary experiment here for a practical scenario that handles different testing scenarios. More speciﬁcally, we conduct the experiments of 5-way meta-training and N-way meta-testing (where N = 5, 10, 20) to examine the effect of testing scenarios that are different from training.
As in Table A4, we compare the methods Baseline, Baseline++, MatchingNet, ProtoNet, and RelationNet. Note that we are unable to apply the MAML method as MAML learns the initialization for the classiﬁer and can thus only be updated to classify the same number of classes. Our results show that for classiﬁcation with a larger N-way in the meta-testing stage, the proposed Baseline++ compares favorably against other methods in both shallow or deeper backbone settings. We attribute the results to two reasons.
First, to perform well in a larger N-way classiﬁcation setting, one needs to further reduce the intra-class variation to avoid misclassiﬁcation. Thus, Baseline++ has better performance than Baseline in both backbone settings. Second, as meta-learning algorithms were trained to perform 5-way classiﬁcation in the meta-training stage, the performance of these algorithms may drop signiﬁcantly when increasing the N-way in the meta-testing stage because the tasks of 10-way or 20-way classiﬁcation are harder than that of 5-way one. One may address this issue by performing a larger N-way classiﬁcation in the meta-training stage (as suggested in Snell et al.
However, it may encounter the issue of memory constraint. For example, to perform a 20-way classiﬁcation with 5 support images and 15 query images in each class, we need to ﬁt a batch size of 400 (20 x (5 + 15)) that must ﬁt into the GPUs. Without special hardware parallelization, the large batch size may prevent us from training models with deeper backbones such as ResNet.