F AST T ASK A DAPTATION FOR F EW -S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot classiﬁcation is a challenging task due to the scarcity of training examples for each class.
The key lies in generalization of prior knowledge learned from large-scale base classes and fast adaptation of the classiﬁer to novel classes.
In this paper, we introduce a two-stage framework.
In the ﬁrst stage, we attempt to learn task-agnostic feature on base data with a novel Metric-Softmax loss.
The
Metric-Softmax loss is trained against the whole label set and learns more discriminative feature than episodic training.
Besides, the Metric-Softmax classiﬁer can be applied to base and novel classes in a consistent manner, which is critical for the generalizability of the learned feature.
In the second stage, we design a task-adaptive transformation which adapts the classiﬁer to each few-shot setting very fast within a few tuning epochs.
Compared with existing ﬁne-tuning scheme, the scarce examples of novel classes are exploited more effectively.
Experiments show that our approach outperforms current state-of-the-arts by a large margin on the commonly used mini-ImageNet and CUB-200-2011 benchmarks.
I NTRODUCTION
In recent years, deep learning models have achieved great success in many visual tasks such as image classiﬁcation (Krizhevsky et al., 2012), object detection (Girshick, 2015) and semantic segmentation (Long et al., 2015).
In general, training a deep network needs massively labeled instances which leads to expensive manual annotation cost.
In contrast, humans can learn novel concepts from only one or a few examples.
Analogously, few-shot learning aims to recognize unseen instances by accessing just a small number of labeled images in each class.
Unfortunately, naive methods such as re-training or ﬁne-tuning the model on novel data would severely suffer from overﬁtting and provide poor result (Finn et al., 2017).
The major difﬁculty lies in the exceedingly limited data, which can hardly represent the class distribution.
In the task of few-shot classiﬁcation, we are given three datasets, namely training set, support set and query set.
The training set (also known as base dataset) normally contains large-scale labeled data for learning of prior knowledge.
The support set and query set contain a small number of examples which are drawn from unseen classes disjoint with the training set.
We are expected to classify the query images, leveraging the prior knowledge from the base training set and the limited cue from the support images.
If the support set contains k annotated images for each of m unique classes, then this task is called m -way k -shot classiﬁcation.
Previous works on few-shot learning can be roughly divided into two categories, namely metalearning based and metric learning based.
The former typically learns a meta-learner model on base data and generalizes it to novel unseen data.
Usually, a recurrent neural network (Santoro et al., 2016) or long short-term memory network (Hochreiter & Schmidhuber, 1997) is utilized to learn a memory network to store knowledge.
The latter attempts to learn a feature embedding where samples of the same class are closer to each other than samples of different classes.
To make training on base data and inference on novel data consistent, a so-called episodic training strategy is widely employed (Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018).
There are two limitations in metric learning based approaches.
One is that episodic training only considers local sample similarity in current data batch.
As shown in Snell et al.
(2017), the limited number of classes pertraining episode is harmful for discriminative feature learning, while simply increasing it leads to better performance.
However, in a sampled episode, a large number of classes is infeasible due to limited GPU memory.
The other limitation is that most existing methods only utilize the labeled
[Figure]
𝒇
[Figure]
𝒈 𝑾 ∈ ℝ 𝒅×𝒎 𝑾 :,𝒊 = 𝟏 𝒌෍ 𝒋 𝒌 𝒈 𝒉 (𝒊,𝒋) 𝒔 probabilityscore 𝒚 loss 𝓛 label 𝒇 𝜽 backbonenetwork
Metric-Softmaxclassifier 𝑾 ∈ ℝ 𝒅×𝒄 𝒔 probability score
[Figure]
𝓛 𝒚 label
[Figure]
support images in novel classes as templates for feature similarity comparison.
We argue that the support images could be better exploited to improve the performance.
In this work, we introduce a two-stage framework named Fast Task Adaption (FTA).
As illustrated in Figure 1, the few-shot classiﬁcation task is decomposed into two steps, i.e. task-agnostic feature learning on base data and adaptation of the classiﬁer to each few-shot learning task.
To learn more general and discriminative feature on base data, we propose a novel Metric-Softmax loss, which is an improvement of the Softmax loss.
Speciﬁcally, we replace the score computing in softmax with a
Gaussian kernel-based radial basis function.
The Metric-Softmax loss is trained against the full base classes, overcoming the limitation of small number of classes in episodic training.
Moreover, the
Metric-Softmax classiﬁer can be applied to base and novel classes in the same way.
The consistency between training and inference is well preserved.
After task-agnostic feature is learned, we design a novel task-adaptive transformation to adapt the classiﬁer to current few-shot learning task.
It is essentially an afﬁne transformation applied to features of novel images.
Notably, compared with ﬁne-tuning re-initialized weights, it exhibits the desirable characteristics of fast convergence and superior performance.
In this way, the aforementioned weaknesses of existing metric learning based methods are well addressed, leading to signiﬁcant accuracy improvement.
Experimental results on the mini-ImageNet and CUB-200-2011 datasets show that our approach consistently outperforms current state-of-the-art methods with respect to various backbone networks.
The main contributions of this paper can be summarized as follows.
• We propose the Metric-Softmax loss, which learns highly discriminative feature and ensures the consistency between training and inference as well.
• We design a novel task-adaptive transformation to adapt the classiﬁer to novel classes efﬁciently and effectively.
• We improve few-shot classiﬁcation signiﬁcantly and achieve state-of-the-art performance on two common few-shot learning benchmarks.
R ELATED W ORK
In this section, we brieﬂy review the previous approaches relevant to ours and current state-of-the-art methods we compare with in experiments.
Meta-learning based methods aim to learn task-agnostic knowledge from a sequence of similar tasks which have enough training data and apply it to new tasks for fast adaption.
Ravi & Larochelle
(2017) utilize an LSTM-based network to control the parameter optimization process, which results in better generalization than SGD when training on a few labeled samples.
Qiao et al.
(2018) propose to adapt a pre-trained network to novel classes by directly predicting the parameters from the activations.
MAML (Finn et al., 2017) learns sensitive and general initial parameters, which can fast adapt to a novel task with only one or a few steps of gradient-descent update.
MTL (Sun et al., 2019) learns to adapt a deep neural network for few-shot learning tasks by learning scaling and shifting functions of weights for each task.
Cai et al.
(2018) use a contextual learner to predict the parameters of an embedding network for unlabeled data by using memory slots.
Mishra et al.
(2018) propose a meta-learner architecture that combines temporal convolutions and soft attention.
Metric learning based methods aim to learn a feature embedding that preserves the class neighborhood structure.
Speciﬁcally, samples of the same class are closer than samples of different classes in the learned embedding space.
Matching networks (Vinyals et al., 2016) combine attention and memory together by learning a network that maps a small labeled support set and an unlabeled example to its label.
They also ﬁrst introduce the episodic training strategy, where the training process mimics the test scenario based on support-query metric learning.
Prototypical network (Snell et al., 2017) uses the mean feature of each class as its corresponding prototype representation to learn a similarity metric space.
Relation network (Sung et al., 2018) learns a deep non-linear distance metric by considering the relation between query images and support images.
Satorras & Estrach (2018) propose a graph neural network to learn the feature similarity between query and support set.
Our approach belongs to this category.
Compared with existing methods, our major improvements are the Metric-Softmax loss for generic feature learning and the task-adaptive transformation for fast task adaptation.
Other methods.
Zhang et al.
(2018) propose a GAN-based approach to help few-shot classiﬁers to learn sharper decision boundary.
Si et al.
(2019) propose a progressive cluster puriﬁcation method for transductive few-shot learning.
Li et al.
(2019) propose a deep nearest neighbor neural network for few-shot learning.
Chen et al.
(2019) give a consistent comparative analysis of several representative few-shot classiﬁcation algorithms and show that the depth of backbone network matters.
Besides, they attempt to ﬁne-tune a Softmax classiﬁer on the support set.
On the contrary, we ﬁrst apply an afﬁne transformation to features of the support images, and compute the class centroids of the transformed features as the classﬁer’s weights.
Concretely, we learn the afﬁne transforming matrix rather than directly ﬁne-tune the randomly initialized weights.
Optimizing the transformation is computationally more stable even when one example per-class is available.
M ETHOD
As shown in Figure 1, the proposed FTA framework is composed of two stages, namely task-agnostic feature learning stage and fast task adaptation stage.
In the ﬁrst stage, we train the network on the base data with the proposed Metric-Softmax loss.
After that, we ﬁx the parameters of the network backbone and use it as a feature extractor.
In the second stage, we learn a task-adaptive classiﬁer, which is also a Metric-Softmax classiﬁer with its weights replaced with afﬁne transformed features of the support images.
The afﬁne transformation g is named Task-Adaptive Transformation (TAT), which is learned on the scarce support set.
With proper initialization, training of g converges very fast within a few epochs.
After training ﬁnishes, g is applied to both support and query images.
In this section, we ﬁrst elaborately describe the proposed Metric-Softmax loss.
Since it is an improvement of the Softmax loss, we ﬁrst give a brief review of the Softmax loss and reveal the issue of discrepancy between training and inference.
Then we explain how to address this issue with the
Metric-Softmax loss.
Finally, we introduce the task-adaptive transformation for fast task adaptation.
3. S OFTMAX C LASSIFIER
Feature learning with the Softmax loss has achieved great success in many feature ranking tasks like person re-identiﬁcation (Zhang et al., 2017) and face veriﬁcation (Wen et al., 2016).
Given an image X of base classes, a backbone network f is applied to project X into feature space h
= f ( X ) , h ∈ R d .
Then, the logits z are computed by feeding h into an afﬁne transformation:
[Equation]
where W ∈ R d × c and b ∈ R c are the transforming weights.
c is the number of classes in base data.
The Softmax classiﬁer predicts the probability score that the image belongs to class i by
[Equation]
The Softmax loss is an assembly of the Softmax classiﬁer and cross-entropy loss.
Suppose the one-hot category label of the image is y ∈ { 0 , 1 } c , the training cross-entropy loss is deﬁned as
[Equation]
During inference, if a test image belongs to one of the training classes, its label can be predicted by
[Equation]
In this case, the scores in both training and inference are computed by an afﬁne transformation and softmax function applied to the feature vector h .
However, in the context of few-shot learning, the query image for testing belongs to a novel class unseen in training.
A common way to classify the novel query image is to compare its similarities to the few labeled support images, which share the same label space with the query image.
Concretely, for m -way k -shot classiﬁcation, we can extract features of the support images, and compute the mean feature for each class by
[Equation]
Here, X ij denotes the j -th image of class i in the support set.
The label of a query image X can be predicted by
[Equation]
Rather than an inner-product between h and column vectors of the weight matrix ( z i = W T : ,i h
+ b i ) followed by softmax function in the previous case, the prediction is based on the euclidean distance between the query and support features.
We argue that this discrepancy hurts the transferability of features pre-trained on base data to novel data.
3. M ETRIC -S OFTMAX C LASSIFIER
To eliminate this discrepancy, we improve the Softmax classiﬁer by redeﬁning the probability score calculating function.
Speciﬁcally, we replace the exp( z i ) term in Equation 2 with a Gaussian kernel-based radial basis function, whose center is the column vector of a learnable weight matrix
W : ,i , W ∈ R d × c .
Then the score can be computed by
[Equation]
where α ∈ R , α > 0 is a hyper-parameter for scaling and h is an L2-normalized feature vector.
In this way, it can be easily derived that arg max i s i is equal to arg min
i || h − W : ,i || .
Here W : ,i can be interpreted as the centroid of class i in the learned embedding space.
That means the training procedure is essentially optimizing the euclidean distance-based similarity between images to their corresponding class centroids.
During training, the cross-entropy loss is adopted as in the Softmax loss.
During inference, we replace W : ,i with the mean feature (see Equation 5) of class i in the support set.
Then classiﬁcation of a query image is conducted according to Equation 4.
Like episodic training, the consistency between training and inference is well preserved.
And we easily enjoy the beneﬁt of more general and discriminative feature learned with the cross-entropy loss.
It is worth noting that the number of classes during inference on novel data is independent from the base dataset, which makes it ﬂexible enough to adapt to different few-shot classiﬁcation tasks.
3. F AST T ASK A DAPTATION
Although the feature learned on the large-scale base data is highly general and discriminative, it is unlikely to perfectly ﬁt arbitrary novel classes.
A proper adaptation to each few-shot task is beneﬁcial.
To leverage the valuable cue provided by the support images, a straightforward choice is to ﬁne-tune a speciﬁc Softmax classiﬁer as in Chen et al.
However, even with the feature extractor being ﬁxed, it is still difﬁcult to train the classiﬁer due to easy overﬁtting on small dataset.
In contrast, we design a parameterized transformation function g to adapt the general feature to current few-shot learning task,
[Equation]
Concretely, g is simply a zero-offset afﬁne transformation: g ( h ) =
M T h , M ∈ R d × d .
For mway k -shot classiﬁcation, we reconstruct an m -class
Metric-Softmax classiﬁer whose weight matrix ¯
W ∈ R d × m is derived from the transformed features of the support images
[Equation]
Here h ( i,j ) denotes the feature of j -th image in class i .
It is worth mentioning that both the transformed feature vector g ( h ) and each column of the weight matrix ¯ W should be L2-normalized.
And now, the computation of probability score is deﬁned as
[Equation]
The transformation g is trained on all labeled images in the support set with the cross-entropy loss.
To ease optimization, the transforming matrix M is initialized with an identity matrix.
After training of g ﬁnishes, it is applied to both support and query images.
And the ﬁnal prediction of a query image
X is computed by
[Equation]
E XPERIMENTS 4. D ATASETS
We conduct few-shot classiﬁcation experiments on two common benchmarks, namely miniImageNet (Vinyals et al., 2016) and CUB-200-2011 (Wah et al., 2011).
mini-ImageNet is ﬁrst proposed by Vinyals et al.
(2016) for few-shot classiﬁcation evaluation.
It consists of 100 classes and 600 images per class, which is a subset sampled from the ImageNet dataset (Deng et al., 2009).
In our experiments, we follow the dataset splits proposed in Ravi &
Larochelle (2017), which takes 64, 16 and 20 classes for training, validation and testing respectively.
CUB-200-2011 is initially proposed for ﬁne-grained classiﬁcation of birds.
It contains 200 categories and 11,788 images in total.
We follow the evaluation protocol used in Hilliard et al.
(2018), which randomly splits the dataset into 100, 50 and 50 classes for training, validation and testing.
4. I MPLEMENTATION D ETAILS
To make a fair comparison with existing works, we evaluate our method for three commonly used backbone networks, namely Conv-4 (Vinyals et al., 2016),
ResNet-10 (Chen et al., 2019) and
ResNet-12 (Oreshkin et al., 2018).
Besides difference in depth and architecture, Conv-4 and ResNet12 expect an input size of 84 × 84 , while ResNet-10 takes 224 × 224 images as input, following the setting in existing works.
In our experiments, we mainly focus on the 5-way 1-shot and 5-way 5-shot classiﬁcation settings which are the most commonly used in existing works.
The test episode contains 5 classes and each class contains 1 (or 5) support image(s) and 15 query images.
For all experiments, we report the mean accuracy over 1200 randomly sampled episodes and the 95% conﬁdence interval.
[Table]
Our implementation is based on PyTorch (Paszke et al., 2017).
We apply the same data augmentation as Chen et al.
(2019), including random cropping, horizontal ﬂipping and color jittering in both feature learning and fast task adaptation stages.
The Adam (Kingma & Ba, 2015) optimizer with (cid:15) =
10 − 3 , β 1 = 0 .
9 and β 2 = 0 .
999 is used.
In the feature learning stage, the backbone network is trained from scratch for 300 epochs in total with a batch size of 32.
We decay the learning rate by a factor of 0 .
1 every 75 epochs.
The scaling factor α used in Metric-Softmax is set to 15 for mini-ImageNet and 1 for CUB-200-2011.
In the task adaptation stage, we use the same optimizer with different learning rate.
We set the learning rate to 0.005 for the 1-shot setting and 0.05 for the 5-shot setting.
α is set to 0.25 and 2 for the 1-shot and 5-shot settings respectively.
We ﬁne-tune the weights of the transformation g for 20 epochs.
Note that all the hyper-parameters are determined by the performance on the validation set of each dataset.
4. C OMPARISONS WITH THE S TATE - OF - THE - ARTS
Results on mini-ImageNet.
Table 1 shows a comparison of our method to current state-of-the-arts on the mini-ImageNet dataset.
Classiﬁcation accuracies are reported for two backbone networks, i.e.
Conv-4 and ResNet-12.
For Conv-4 backbone, we achieve the best accuracy in both 5-way 1-shot and 5-way 5-shot settings.
Our method boosts the performance for the 5-shot setting, surpassing the second best by 9.33% and 2.10% for the Conv-4 and ResNet-12 backbones respectively.
A closer analysis on the larger improvement for the 5-shot setting than 1-shot setting is given in the discussion section.
Results on CUB-200-2011.
On this dataset, we ﬁnd that there is no work reporting the few-shot performance for the ResNet-12 backbone.
For a fair comparison, we report the accuracies of the
Conv-4 and ResNet-10 backbones.
As shown in Table 2, similar to mini-ImageNet, our method outperforms the second best method by a large margin (87.92% vs. 81.90% for Conv-4 and 92.89% vs. 87.45% for ResNet-10) in the 5-way 5-shot setting.
In the 1-shot setting, we achieve the best performance for ResNet-10 and comparable accuracy to PABN (65.11% vs. 66.71%) for Conv-4.
It is worth mentioning that in PABN, the max-pooling layers of the backbone network are replaced with bilinear pooling, which is specially optimized for ﬁne-grained classiﬁcation tasks.
We expect further accuracy improvement by incorporating such techniques in our method.
Results under Domain Shift.
To evaluate the generalizability of the task-agnostic feature learned with the proposed Metric-Softmax loss, we perform the cross domain experiments.
In the cross
[Table] [Table]
domain scenario, we swap the training sets of the two datasets. Speciﬁcally, in the miniImageNet → CUB-200-2011 setting, the training set is comprised of the 64 base classes from miniImageNet, while the validation and test data come from CUB-200-2011.
Similarly, in the CUB200-2011 → mini-ImageNet setting, we use the 100-class base data of CUB-200-2011 for general feature learning and evaluate its performance on the mini-ImageNet dataset.
We report the results under the same setting as Devos & Grossglauser (2019) in Table 3.
Compared with existing methods, our approach achieves the highest accuracy in all settings.
Notably, we improve SubspaceNet by 22.67% in the mini-ImageNet → CUB-200-2011 setting.
The strong cross domain performance is mainly attributed to the task-adaptive transformation g of our FTA.
If it is disabled, the accuracy drops drastically from 85.38% to 67.22%.
Another observation is that the performance degrades more in CUB-200-2011 → mini-ImageNet than mini-ImageNet → CUB-200-2011.
This is because mini-ImageNet contains more abundant classes than CUB-200-2011 and the network can learn more general and discriminative feature.
D ISCUSSION
Ablation Study of FTA.
As shown in Table 4, compared with the Softmax classiﬁer, the proposed
Metric-Softmax classiﬁer improves the accuracy by 4.77% and 3.41% in the 5-way 1-shot and 5way 5-shot settings respectively.
This improvement is attributed to elimination of the discrepancy between training and inference brought by our Metric-Softmax classiﬁer.
By incorporating the taskadaptive transformation, the performance is further boosted, especially in the 5-shot setting.
TAT vs. Fine-tuning.
To leverage the cue from the supported set, an alternative method is to ﬁne-tune a re-initialized Metric-Softmax classiﬁer similar to Chen et al.
Figure 2 shows a comparison of the proposed task-adaptive transformation to the alternative ﬁne-tuning scheme.
In terms of convergence speed, TAT takes only 5 epochs to get an appreciable performance.
However, [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
direct ﬁne-tuning takes over 40 epochs to converge for the 5-way 5-shot setting.
For the 5-way 1shot setting, it barely improves the accuracy.
In terms of ﬁnal performance, TAT is far more superior over direct ﬁne-tuning.
Why does FTA improve the 5-shot setting more than 1-shot setting?
As shown in Section 4.3, our FTA outperforms existing methods more signiﬁcantly in the 5-shot setting than 1-shot setting.
It can be explained by the mechanism of the task-adaptive transformation.
In the 1-shot setting, only one image is available in each class.
Then Equation 9 degenerates to ¯ W : ,i = g ( h ( i, 1) ) .
For an image with label y, y ∈ { 1 , 2 , 3 , 4 , 5 } , the numerator of Equation 10 becomes a constant: exp( − α || g ( h ( y, 1) ) − g ( h ( y, 1) )
|| )
= 1 .
Now minimizing the cross-entropy means minimizing the denominator of Equation 10, which is equivalent to maximizing the total distance of this feature to others: (cid:80)
j || g ( h ( y, 1) )
− g ( h ( j, 1) ) || .
As illustrated in Figure 3(a), task-adaptive transformation only enlarges the distance among different classes in the 1-shot setting.
However, for the 5-shot setting (see Figure 3(b)), the transformation not only enlarges inter-class distance but also shrinks the intra-class distance, which helps to improve the classiﬁer.
C ONCLUSIONS
In this paper we propose the Metric-Softmax loss which can adequately explore the feature similarities in the whole base dataset with the premise of keeping the classiﬁer consistent between training and inference.
In addition, we take full advantage of the few labeled novel data by introducing a parameterized transformation to adapt the learned general feature to each few-shot classiﬁcation task.
Our approach is particularly helpful for the 5-shot setting.
We leave further improvement for the 1-shot setting as our future work.