C ROSS -D OMAIN F EW -S HOT C LASSIFICATION VIA L EARNED F EATURE -W ISE T RANSFORMATION
A BSTRACT
Few-shot classiﬁcation aims to recognize novel categories with only few labeled images in each class.
Existing metric-based few-shot classiﬁcation algorithms predict categories by comparing the feature embeddings of query images with those from a few labeled images (support examples) using a learned metric function.
While promising performance has been demonstrated, these methods often fail to generalize to unseen domains due to large discrepancy of the feature distribution across domains.
In this work, we address the problem of few-shot classiﬁcation under domain shifts for metric-based methods.
Our core idea is to use feature-wise transformation layers for augmenting the image features using afﬁne transforms to simulate various feature distributions under different domains in the training stage.
To capture variations of the feature distributions under different domains, we further apply a learning-to-learn approach to search for the hyper-parameters of the feature-wise transformation layers.
We conduct extensive experiments and ablation studies under the domain generalization setting using ﬁve few-shot classiﬁcation datasets: mini-ImageNet, CUB, Cars, Places, and Plantae.
Experimental results demonstrate that the proposed feature-wise transformation layer is applicable to various metric-based models, and provides consistent improvements on the few-shot classiﬁcation performance under domain shift.
I NTRODUCTION
Few-shot classiﬁcation (Lake et al., 2015) aims to recognize instances from novel categories (query instances) with only few labeled examples in each class (support examples).
Among various recent approaches for addressing the few-shot classiﬁcation problem, metric-based meta-learning methods (Garcia & Bruna, 2018; Sung et al., 2018; Vinyals et al., 2016;
Snell et al., 2017; Oreshkin et al., 2018) have received considerable attention due to their simplicity and effectiveness.
In general, metric-based few-shot classiﬁcation methods make the prediction based on the similarity between the query image and support examples.
As illustrated in Figure 1, metric-based approaches consist of 1) a feature encoder and 2) a metric function.
Given an input task consisting of few labeled images (the support set) and unlabeled images (the query set) from novel classes, the encoder ﬁrst extracts the image features.
The metric function then takes the features of both the labeled and unlabeled images as input and predicts the category of the query images.
Despite the success of recognizing novel classes sampled from the same domain as in the training stage (e.g., , both training and testing are on mini-ImageNet classes), Chen et al.
(Chen et al., 2019a) recently raise the issue that existing metric-based approaches often do not generalize well to categories from different domains.
The generalization ability to unseen domains, however, is of critical importance due to the difﬁculty to construct large training datasets for rare classes (e.g., , recognizing rare bird species
CU B ? ? ?
(c) Testing on unseen domains …
C a r s ? ? ? ? ? ? ? ? ?
m i n i - I m ag e N e t ? ? ?
m i n i - I m ag e N e t ! "
#$ % ? ? ?
(a) Training
Support set & = {) % ,$ % }
Query set , = {) - ,$ - } ? ? ? ? ? ? .
(Eqn. 2) ! "
#$ -
Support set &
Query set , (b) Testing on seen domain
Support set &
Query set ,
Support set &
Query set ,
Feature encoder
Metric function " #$ - " #$ - ! !
in a ﬁne-grained classiﬁcation setting).
As a result, understanding and addressing the domain shift problem for few-shot classiﬁcation is of great interest.
To alleviate the domain shift issue, numerous unsupervised domain adaptation techniques have been proposed (Pan & Yang, 2010;
Chen et al., 2018; Tzeng et al., 2017).
These methods focus on adapting the classiﬁer of the same category from the source to the target domain.
Building upon the domain adaptation formulation, Dong and Xing (Dong & Xing, 2018) relax the constraint and transfer knowledge across domains for recognizing novel category in the one-shot setting.
However, unsupervised domain adaptation approaches assume that numerous unlabeled images are available in the target domain during training.
In many cases, this assumption may not be realistic.
For example, the cost and efforts of collecting numerous images of rare bird species can be prohibitively high.
On the other hand, domain generalization methods have been developed (Blanchard et al., 2011; Li et al., 2019) to learn classiﬁers that generalize well to multiple unseen domains without requiring the access to data from those domains.
Yet, existing domain generalization approaches aim at recognizing instance from the same category in the training stage.
In this paper, we tackle the domain generalization problem for recognizing novel category in the few-shot classiﬁcation setting.
As shown in Figure 1(c), our key observation is that the distributions of the image features extracted from the tasks in different domains are signiﬁcantly different.
As a result, during the training stage, the metric function may overﬁt to the feature distributions encoded only from the seen domains and thus fail to generalize to unseen domains.
To address the issue, we propose to integrate feature-wise transformation layer to modulate the feature activations with afﬁne transformations into the feature encoder.
The use of these feature-wise transformation layers allows us to simulate various distributions of image features during the training stage, and thus improve the generalization ability of the metric function in the testing phase.
Nevertheless, the hyper-parameters of the feature-wise transformation layers may require meticulous hand-tuning due to the difﬁculty to model the complex variation of the image feature distributions across various domains.
In light of this, we develop a learning-to-learn algorithm to optimize the proposed feature-wise transformation layers.
The core idea is to optimize the feature-wise transformation layers so that the model can work well on the unseen domains after training the model using the seen domains.
We make the source code and datasets public available to simulate future research in this ﬁeld.
We make the following three contributions in this work:
• We propose to use feature-wise transformation layers to simulate various image feature distributions extracted from the tasks in different domains.
Our feature-wise transformation
layers are method-agnostic and can be applied to various metric-based few-shot classiﬁcation approaches for improving their generalization to unseen domains.
• We develop a learning-to-learn method to optimize the hyper-parameters of the featurewise transformation layers.
In contrast to the exhaustive parameter hand-tuning process, the proposed learning-to-learn algorithm is capable of ﬁnding the hyper-parameters for the feature-wise transformation layers to capture the variation of image feature distribution across various domains.
• We evaluate the performance of three metric-based few-shot classiﬁcation models (including MatchingNet (Vinyals et al., 2016), RelationNet (Sung et al., 2018), and Graph Neural Networks (Garcia & Bruna, 2018)) with extensive experiments under the domain generalization setting.
We show that the proposed feature-wise transformation layers can effectively improve the generalization ability of metric-based models to unseen domains.
We also demonstrate further performance improvement with our learning-to-learn scheme for learning the feature-wise transformation layers.
R ELATED W ORK
Few-shot classiﬁcation.
Few-shot classiﬁcation aims to learn to recognize novel categories with a limited number of labeled examples in each class.
Signiﬁcant progress has been made using the meta-learning based formulation.
There are three main classes of meta-learning approaches for addressing the few-shot classiﬁcation problem.
First, recurrent-based frameworks (Rezende et al., 2016; Santoro et al., 2016) sequentially process and encode the few labeled images of novel categories.
Second, optimization-based schemes (Finn et al., 2017; Rusu et al., 2019; Tseng et al., 2019; Vuorio et al., 2019) learn to ﬁne-tune the model with the few example images by integrating the ﬁne-tuning process in the meta-training stage.
Third, metric-based methods (Koch et al., 2015;
Vinyals et al., 2016; Snell et al., 2017; Oreshkin et al., 2018; Sung et al., 2018; Lifchitz et al., 2019) classify the query images by computing the similarity between the query image and few labeled images of novel categories.
Among these three classes, metric-based methods have attracted considerable attention due to their simplicity and effectiveness.
Metric-based few-shot classiﬁcation approaches consist of 1) a feature encoder to extract features from both the labeled and unlabeled images and 2) a metric function that takes image features as input and predict the category of unlabeled images.
For example,
MatchingNet (Vinyals et al., 2016) applies cosine similarity along with a recurrent network, ProtoNet (Snell et al., 2017) utilizes euclidean distance, RelationNet (Sung et al., 2018) uses CNN modules, GNN (Garcia & Bruna, 2018) employs graph convolution modules as the metric functions.
However, these metric functions may fail to generalize to unseen domains since the distributions of the image features extracted from the task in various domains can be drastically different.
Chen et al.
(Chen et al., 2019a) recently show that the performance of existing few-shot classiﬁcation methods degrades signiﬁcantly under domain shifts.
Our work focuses on improving the generalization ability of metric-based few-shot classiﬁcation models to unseen domains.
Very recently,
Triantaﬁllou et al.
(Triantaﬁllou et al., 2020) also target on the cross-domain few-shot classiﬁcation problem.
We encourage the readers to review for a more complete picture.
Domain adaptation.
Domain adaptation methods (Pan & Yang, 2010) aim to reduce the domain shift between the source and target domains.
Since the emergence of domain adversarial neural networks (DANN) (Ganin et al., 2016), numerous frameworks have been proposed to apply adversarial training to align the source and target distributions on the feature-level (Tzeng et al., 2017; Chen et al., 2018;
Hsu et al., 2020) or on the pixel-level (Tsai et al., 2018; Hoffman et al., 2018; Bousmalis et al., 2017; Chen et al., 2019b; Lee et al., 2018).
Most domain frameworks, however, target at adapting knowledge of the same category learned from the source to target domain and thus are less effective to handle novel category as in the few-shot classiﬁcation scenarios.
One exception is the work by Dong and Xing (Dong & Xing, 2018) that address the domain shift issue in the one-shot learning setting.
Nevertheless, these domain adaptation methods require access to the unlabeled images in the target domain during the training.
Such an assumption may not be feasible in many applications due to the difﬁculty of collecting abundant examples of rare categories (e.g., rare bird species).
Domain generalization.
In contrast to the domain adaptation frameworks, domain generalization (Blanchard et al., 2011) methods aim at generalizing from a set of seen domains to the unseen domain without accessing instances from the unseen domain during the training stage.
Before the emerging of learning-to-learn (i.e., meta-learning) (Ravi & Larochelle, 2017; Finn et al., 2017) approaches, several methods have been proposed for tackling the domain generalization problem.
Examples include extracting domain-invariant features from various seen domains (Blanchard et al., 2011; Li et al., 2018b; Muandet et al., 2013), improving the classiﬁers by fusing classiﬁers learned from seen domains (Niu et al., 2015a;b), and decomposing the classiﬁers into domain-speciﬁc and domain-invariant components (Khosla et al., 2012; Li et al., 2017a).
Another stream of work learns to augment the input data with adversarial learning (Shankar et al., 2018;
Volpi et al., 2018).
Most recently, a number of methods apply the learning-to-learn strategy to simulate the generalization process in the training stage (Balaji et al., 2018; Li et al., 2018a; 2019).
Our method adopts a similar approach to train the proposed feature-wise transformation layers.
The application context, however, differs from prior work as we focus on recognizing novel category from unseen domains in few-shot classiﬁcation.
The goal of this work is to make few-shot classiﬁcation algorithms robust to domain shifts.
Learning-based data augmentation.
Data augmentation methods are designed to increase the diversity of data for the training process.
Unlike the hand-crafted approaches such as horizontal ﬂipping and random cropping, several recent approaches have been proposed to learn the data augmentation (Cubuk et al., 2019; DeVries & Taylor, 2017a;
Lemley et al., 2017; Perez & Wang, 2017; Sixt et al., 2018; Tran et al., 2017).
For instance, the SmartAugmentation (Lemley et al., 2017) scheme trains a network that combines multiple images from the same category.
The Bayesian DA (Tran et al., 2017) method augments the data according to the distribution learned from the training set, and the RenderGAN (Sixt et al., 2018) model simulates realistic images using generative adversarial networks.
In addition, the AutoAugment (Cubuk et al., 2019) algorithm learns the augmentation via reinforcement learning.
Two recent frameworks (Shankar et al., 2018; Volpi et al., 2018) target at augmenting the data by modeling to the variation across different domains with adversarial learning.
Similar to these approaches for capturing the variations across multiple domains, we develop a learning-to-learn process to optimize the proposed feature-wise transformation layers for simulating various distributions of image features encoded from different domains.
Conditional normalization.
Conditional normalization aims to modulate the activation via a learned afﬁne transformation conditioned on external data (e.g., an image of an artwork for capturing a speciﬁc style).
Conditional normalization methods, including Conditional Batch Normalization (Dumoulin et al., 2017), Adaptive Instance Normalization (Huang & Belongie, 2017), and
SPADE (Park et al., 2019), are widely used in the style transfer and image synthesis tasks (Karras et al., 2019; Lee et al., 2020; AlBahar & Huang, 2019).
In addition to image stylization and generation, conditional normalization has also been applied to align different data distributions for domain adaptation (Cariucci et al., 2017; Li et al., 2017b).
In particular, the TADAM method (Oreshkin et al., 2018) applies conditional batch normalization to metric-based models for the few-shot classiﬁcation task.
The TADAM method aims to model the training task distribution under the same domain.
In contrast, we focus on simulating various features distributions from different domains.
Regularization for neural networks.
Adding some form of randomness in the training stage is an effective way to improve generalization (Srivastava et al., 2014; Wan et al., 2013;
Larsson et al., 2017; DeVries & Taylor, 2017b;
Zhang et al., 2018; Ghiasi et al., 2018).
The proposed featurewise transformation layer for modulating the feature activations of intermediate layers (by applying random afﬁne transformations) can also be viewed as a way to regularize network training.
M ETHODOLOGY 3. P RELIMINARIES
Few-shot classiﬁcation and metric-based method.
The few-shot classiﬁcation problem is typically characterized as N w way (number of categories) and N s shot (number of labeled examples for each category).
Figure 1 shows an example of how the metric-based frameworks operate in the 3 -way 3 -shot few shot classiﬁcation task.
A metric-based algorithm generally contains a feature encoder E and a metric function M .
For each iteration during the training stage, the algorithm randomly samples N w categories and constructs a task T .
We denote the collection of input images as
X = { x 1 , x 2 , · · · , x n } and the corresponding categorical labels as and Y = { y 1 , y 2 , · · · , y n } .
A task T consists of a support set S
= {
( X s , Y s ) } and a query set Q = { ( X q , Y q ) } .
The support set S and query set Q are respectively formed by randomly selecting N s and N q samples for each of the
N w categories.
The feature encoder E ﬁrst extracts the features from both the support and query images.
The metric function M then predicts the categories of the query images X q according to the label of support images Y s , the encoded query image E ( x q ) , and the encoded support images E ( X s ) .
The process can be formulated as
[Equation]
Finally, the training objective of a metric-based framework is the classiﬁcation loss of the images in the query set,
[Equation]
The main difference between various metric-based algorithms lies in the design choice for the metric function M .
For instance, the MatchingNet (Vinyals et al., 2016) method utilizes long-short-term memories (LSTM), the RelationNet (Sung et al., 2018) model applies convolutional neural networks (CNN), and the GNN (Garcia & Bruna, 2018) scheme uses graph convolutional networks.
Problem setting.
In this work, we address the few-shot classiﬁcation problem under the domain generalization setting.
We denote a domain consisting of a collection of few-shot classiﬁcation tasks as T =
{ T 1 , T 2 , · · · , T n } .
We assume N seen domains {T seen1 , T seen2 , · · · , T seen N } available in the training phase.
The goal is to learn a metric-based few-show classiﬁcation model using the seen domains, such that the model can generalize well to an unseen domain T unseen .
For example, one can train the model with the mini-ImageNet (Ravi & Larochelle, 2017) dataset as well as some public available ﬁne-grained few-shot classiﬁcation domains, e.g., CUB (Welinder et al., 2010), and then evaluate the generalization ability of the model on an unseen plants domain.
Note that our problem formulation does not access images in the unseen domain at the training stage.
3. F EATURE -W ISE T RANSFORMATION L AYER
Our focus in this work is to improve the generalization ability of metric-based few-shot classiﬁcation models to arbitrary unseen domains.
As shown in Figure 1, due to the discrepancy between the feature distributions extracted from the task in the seen and unseen domains, the metric function M may overﬁt to the seen domains and fail to generalize to the unseen domains.
To address the problem, we propose to integrate a feature-wise transformation to augment the intermediate feature activations with afﬁne transformations into the feature encoder E .
Intuitively, the feature encoder E integrated with the feature-wise transformation layers can produce more diverse feature distributions which improve the generalization ability of the metric function M .
As shown in Figure 2(b), we insert the feature-wise transformation layer after the batch normalization layer in the feature encoder E .
The hyper-parameters θ γ ∈ R C × 1 × 1 and θ β ∈ R C × 1 × 1 indicate the standard deviations of the
Gaussian distributions for sampling the afﬁne transformation parameters.
Given an intermediate feature activation map z in the feature encoder with the dimension of C × H × W , we ﬁrst sample the scaling term γ and bias term β from Gaussian distributions,
[Equation]
We then compute the modulated activation ˆ z as
[Equation]
where ˆ z c,h,w ∈
ˆ z and z c,h,w ∈ z .
In practice, we insert the feature-wise transformation layers to the feature encoder E at multiple levels.
3. L EARNING THE F EATURE -W ISE T RANSFORMATION L AYERS
While we can empirically determine hyper-parameters θ f
= { θ γ , θ β } of the feature-wise transformation layer, it remains challenging to hand-tune a generic set of parameters which are effective
#$ ? ? ?
% " &$'( ! "
#$'( ? ? ? )
*+ , -./0 , , 2./ ) * , 4(6).
, ∀
Pseudo-seen (ps) mini-ImageNet
Pseudo-unseen (pu)
CUB % " &$ : ∈ < =
×0× ?
∈ < =×0×
Feature-wise transformation layer @
A(B)
= {@
E ,@
F }
H (6) ∈ < =
×I×J
KH (6) (Eqn. 5) (Eqn. 7) :
~M(1,softplus(, V )) ?
~M(0,softplus(, X )) , 4(6)./0 , ∀
[Figure]
on different settings (i.e., different metric-based frameworks and different seen domains).
To address this problem, we design a learning-to-learn algorithm to optimize the hyper-parameters of the feature-wise transformation layer.
The core idea is that training the metric-based model integrated with the proposed layers on the seen domains should improve the performance of the model on the unseen domains.
We illustrate the process in Figure 2(b) and Algorithm 1.
In each training iteration t , we sample a pseudo-seen domain T ps and a pseudo-unseen domain T pu from a set of seen domains {T seen1 , T seen2 , · · · , T seen N } .
Given a metric-based model with feature encoder E θ te and metric function M θ tm , we ﬁrst integrate the proposed layers with hyper-parameters θ tf = { θ tγ , θ tβ } into the feature encoder (i.e., E θ te ,θ tf ).
We then use the loss in equation 2 to update the parameters in the metric-based model with the pseudo-seen task T ps = { ( X ps s , Y ps s ) , ( X ps q , Y ps q ) } ∈ T ps , namely
[Equation]
where α is the learning rate.
We then measure the generalization ability of the updated metric-based model by 1) removing the feature-wise transformation layers from the model and 2) computing the classiﬁcation loss of the updated model on the pseudo-unseen task T pu
[Equation]
{ ( X pu s , Y pu s ) , ( X pu q , Y pu q ) } ∈ T pu , namely
[Equation]
Finally, as the loss L pu reﬂects the effectiveness of the feature-wise transformation layers, we optimize the hyper-parameters θ f by
[Equation]
Note that the metric-based model and feature-wise transformation layers are jointly optimized in the training stage.
n learning rate α do
Sample a pesudo
T pu ∈ T pu //
based pseudo task :
Obtain θ t
1 e , θ t +1 m using equation // Update feature-wise transformation with pseudo-unseen task :
Obtain θ t +1 f using equation 6 and equation10 end
E XPERIMENTAL R ESULTS 4. E XPERIMENTAL S ETUPS
We validate the efﬁcacy of the proposed feature-wise transformation layer with three existing metricbased algorithms (Vinyals et al., 2016; Sung et al., 2018; Garcia & Bruna, 2018) under two experimental settings.
First, we empirically determine the hyper-parameters θ f
= { θ γ , θ β } of the featurewise transformation layers and analyze the impact of the feature-wise transformation layers.
We train the few-shot classiﬁcation model on the mini-ImageNet (Bousmalis et al., 2017) domain and evaluate the trained model on four different domains: CUB (Welinder et al., 2010), Cars (Krause et al., 2013), Places (Zhou et al., 2017), and Plantae (Van Horn et al., 2018).
Second, we demonstrate the importance of the proposed learning-to-learn scheme for optimizing the hyper-parameters of feature-wise transformation layers.
We adopt the leave-one-out setting by selecting an unseen domain from CUB, Cars, Places, and Plantae domains.
The mini-ImageNet (Bousmalis et al., 2017) and the remaining domains then serve as the seen domains for training both the metric-based model and feature-wise transformation layers using Algorithm 1.
After the training, we evaluate the trained model on the selected unseen domain.
Datasets.
We conduct experiments using ﬁve datasets: mini-ImageNet (Ravi & Larochelle, 2017),
CUB (Welinder et al., 2010), Cars (Krause et al., 2013), Places (Zhou et al., 2017), and Plantae (Van Horn et al., 2018).
Since the mini-ImageNet dataset serves as the seen domain for all experiments, we select the training iterations with the best accuracy on the validation set of the miniImageNet dataset for evaluation.
More details of dataset processing are presented in Appendix A.1.
Implementation details.
We apply the feature-wise transformation layers to three metric-based frameworks: MatchingNet (Vinyals et al., 2016), RelationNet (Sung et al., 2018), and GNN (Garcia & Bruna, 2018).
We use the public implementation from Chen et al.
(Chen et al., 2019a) to train both the MatchingNet and RelationNet model.
For the GNN approach, we integrate the ofﬁcial implementation for graph convolutional network into Chen’s implementation.
In all experiments, we adopt the ResNet-
(He et al., 2016) model as the backbone network for our feature encoder E .
We present the average results over 1 , 000 trials for all the experiments.
In each trial, we randomly sample N w categories (e.g., 5 classes for 5-way classiﬁcation).
For each category, we randomly select N s images (e.g., 1-shot or 5-shot) for the support set X s and 16 images for the query set X q .
We discuss the implementation details in Appendix A.2.
Pre-trained feature encoder.
Prior to the few-shot classiﬁcation training stage, we ﬁrst pre-train the feature encoder E by minimizing the standard cross-entropy classiﬁcation loss on the 64 training categories in the mini-ImageNet dataset.
This strategy can signiﬁcantly improve the performance of
[Table]
metric-based models and is widely adopted in several recent frameworks (Rusu et al., 2019; Gidaris & Komodakis, 2018; Lifchitz et al., 2019).
4. F EATURE -W ISE T RANSFORMATION WITH M ANUAL P ARAMETER T UNING
We train the model using the mini-ImageNet dataset and evaluate the trained model with four other unseen domains: CUB, Cars, Places, and Plantae.
We add the proposed feature-wise transformation layers after the last batch normalization layer of all the residual blocks in the feature encoder E during the training stage.
We empirically set θ γ and θ β in all feature-wise transformation layers to be 0 . 3 and 0 . 5 , respectively.
Table 1 shows the metric-based model trained with the featurewise transformation layers performs favorably against the individual baselines.
We attribute the improvement of generalization to the use of the proposed layers for making the feature encoder E produce more diverse feature distributions in the training stage.
As a by-product, we also observe the improvement on the seen domain (i.e., mini-ImageNet) since there is still a slight discrepancy between the feature distributions extracted from the training and testing sets of the same domain.
It is noteworthy that we also compare the proposed method with several recent approaches (e.g., Lee et al.
in Table 8 and Table 9.
With the proposed feature-wise transformation layers, the
GNN (Garcia & Bruna, 2018) model performs favorably against the state-of-the-art frameworks on both the seen domain (i.e., mini-ImageNet) and unseen domains.
4. G ENERALIZATION FROM M ULTIPLE D OMAINS
Here we validate the effectiveness of the proposed learning-to-learn algorithm for optimizing the hyper-parameters of the feature-wise transformation layers.
We compare the metric-model trained with the proposed learning procedure to the model trained with the pre-determined feature-wise transformation layers.
The leave-one-out setting is used to select one domain from the CUB, Cars,
Places, and Plantae as the unseen domain for the evaluation.
The mini-ImageNet and the remaining domains serve as the seen domains for training the model.
Since we select the training iteration according to the validation performance on the mini-ImageNet domain for evaluation, we do not consider the mini-ImageNet as the unseen domain.
We present the results in Table 2.
We denote FT and LFT as applying pre-determined feature-wise transformation layers and those layers optimized with the proposed learning-to-learn algorithm, respectively.
The models optimized with proposed learning scheme outperforms those trained with the pre-determined feature-wise transformation layers since the optimized feature-wise transformation layers can better capture the variation of feature distributions across different domains.
Table 1 and Table 2 show that the proposed feature-wise transformation layers together with the learning-to-learn algorithm effectively mitigate the domain shift problem for metric-based frameworks.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Note since the proposed learning-to-learn approach optimizes the hyper-parameters via stochastic gradient descent, it may not ﬁnd the global minimum that achieves the best performance.
It is certainly possible to manually ﬁnd another set of hyper-parameter setting that achieves better performance.
However, this requires meticulous and computationally expensive hyper-parameter tuning.
Speciﬁcally, the dimension of the hyper-parameters θ γ and θ β is c i × 1 × 1 for the i -th feature-wise transformation layer in the feature encoder, where c i is the number of feature channels.
As there are n feature-wise transformation layers in the feature encoder E , we need to perform the hyperparameter search in a ( c 1 + c 2 · · · + c n ) × 2 -dimensional space.
In practice, the dimension of the the search space is 1920 .
Visualizing feature-wise transformed features.
To demonstrate that the proposed feature-wise transformation layers can simulate various feature distributions extracted from the task in different domains, we show the t-SNE visualizations of the image features extracted by the feature encoder in the RelationNet (Sung et al., 2018) model in Figure 3.
The model is trained with 5-way 5-shot classiﬁcation setting on the mini-ImageNet, Cars, Places, and Plantae domains (i.e., corresponding s
o f t p l u s ( ) * ) s o f t p l u s ( ) , ) # - th residual block #
[Figure]
MatchingNet
RelationNet
GNN to the ﬁfth block of the second column in Table 2).
We observe that the distance between features extracted from different domains becomes smaller with the help of feature-wise transformation layers.
Furthermore, the proposed learning-to-learn scheme can further help the feature-wise transformation layers capture the variation of feature distributions from various domains, thus close the domain gap and improve the generalization ability of metric-based models.
Visualizing feature-wise transformation layers.
To better understand how the learned featurewise transformation layers operate, we show the values of the softplus( θ γ ) and softplus( θ γ ) in the feature-wise transformation layer.
Figure 4 presents the visualization.
The values of scaling terms softplus( θ γ ) tend to become smaller in the deeper layers, particularly for those in the last residual block.
On the other hand, the depth of the layer does not seem to have an apparent impact on the distributions of the bias terms softplus( θ β ) .
The distributions are also different across different metric-based classiﬁcation methods.
These results suggest the importance of the proposed learningto-learn algorithm because there does not exist a set of optimal hyper-parameters of the feature-wise transformation layers which work well with all metric-based approaches.
C ONCLUSIONS
We propose a method to effectively enhance metric-based few-shot classiﬁcation frameworks under domain shifts.
The core idea of our method lies in using the feature-wise transformation layer to simulate various feature distributions extracted from the tasks in different domains.
We develop a learning-to-learn approach for optimizing the hyper-parameters of the feature-wise transformation layers by simulating the generalization process using multiple seen domains.
From extensive experiments, we demonstrate that our technique is applicable to different metric-based few-shot classiﬁcation algorithms and show consistent improvement over the baselines
A CKNOWLEDGEMENTS
This work is supported in part by the NSF CAREER Grant #1149783, the NSF Grant #1755785, and gifts from Google.
A A PPENDIX A. D ATASET C OLLECTION
We use ﬁve few-shot classiﬁcation datasets in all of our experiments: mini-ImageNet, CUB, Cars,
Places, and Plantae.
We follow the setting in Ravi & Larochelle (2017) and Hilliard et al.
(2018) to process mini-ImageNet and CUB datasets.
As for the other datasets, we manually process the dataset by random splitting the classes.
The number of training, validation, testing categories for each dataset are summarized in Table 3.
[Table]
A. A DDITIONAL IMPLEMENTATION DETAILS
We use the implementation and adopt the setting of hyper-parameters from Chen et al.
(Chen et al., 2019a).
4 We train the metric-based model and feature-wise transformation layers with a learning rate of 0 .
001 and 40 , 000 iterations.
For feature-wise transformation layers, we apply L2 regularization with a weight of 10 − 8 .
The number of inner iterations adopted in the learning-to-learn scheme is set to be 1 .
Matching Network.
We cannot utilize the MatchingNet implementation from Chen et al.
(Chen et al., 2019a) since they applied the Pytorch built-in LSTM module, which does not support secondorder backpropagation.
Without the second-order backpropagation, we are unable to optimize the feature augmentation layers using the proposed learning-to-learn algorithm.
As a result, we reimplement the LSTM module for the MatchingNet model.
To verify the correctness of our implementation, we evaluate the 5-way 5-shot performance with the ResNet-
(He et al., 2016) backbone network on the mini-ImageNet dataset (Ravi & Larochelle, 2017).
Our implementation reports 68 .
88 ± 0 .
69% accuracy, which is similar to the result (i.e., 68 .
82 ± 0 .
65% ) reported by
Chen et al.
(Chen et al., 2019a).
We make the source code and datasets public available to foster future progress in this ﬁeld.
A. A DDITIONAL EXPERIMENTAL RESULTS
Ablation study on pre-trained metric encoder.
As described in Section 4.1, we pre-trained the metric encoder E by minimizing the cross-entropy classiﬁcation loss using the 64 training categories from the mini-ImageNet dataset.
To understand the impact of the pre-training, we conduct an ablation study using the leave-one-out experiment illustrated in Section 4.3.
As shown in Table 4, pre-training the metric encoder E substantially improve the few-shot classiﬁcation performance of metric-based frameworks.
Note that such a pre-training process is also adopted by several recent frameworks (Rusu et al., 2019; Gidaris & Komodakis, 2018; Lifchitz et al., 2019) to boost the fewshot classiﬁcation performance.
Number of ways in testing stage.
In this experiment, we consider a practical scenario that the number of ways N w in the testing phase is different from that in the training stage.
Since the
GNN (Garcia & Bruna, 2018) framework requires the numbers of ways to be consistent in the training and testing, we evaluate the MatchingNet (Vinyals et al., 2016) and RelationNet (Sung et al., 2018) model with this setting.
Table 5 reports the performances of the models trained on the miniImageNet, Cars, Places, and Plantae domains under the 5-way 5-shot setting (i.e
[Table] [Table]
[Table]
to the fourth and ﬁfth block of the second column in Table 2).
Our proposed learning-to-learned feature-wise transformation layers are capable of improving the generalization of metric-based models to the unseen domain under various numbers of ways in the testing stage.
Pre-determined hyper-parameters of feature-wise transformation layers.
We demonstrate the difﬁculty to hand-tune the hyper-parameters of the proposed feature-wise transformation layers in this experiment.
Different from the setting described in Section 4.2, we set the hyper-parameters θ γ and θ β in all feature-wise transformation layers to be 1 .
The model is trained under 5 -way setting using the mini-ImageNet domain, and evaluate it on the other domains.
We report the 1shot and 5 -shot performance in Table 6.
We denote applying feature-wise transformation layers with { θ γ , θ β } = { 0 . 3 , 0 . 5 } as FT and those with { θ γ , θ β } = { 1 , 1 } as FT*.
We observe that the metric-based models applied with FT perform favorably against to those applied with FT*.
In several cases, applying FT* even yields inferior results compared to the original training without the feature-wise transformation layers.
This suggests the difﬁculty of hand-tuning the hyper-parameters and the importance of the proposed learning-to-learn scheme for optimizing the hyper-parameters of the feature-wise transformation layers.
Hyper-parameter initialization for learning-to-learn.
For all the experiments, we initialize the parameters θ γ and θ β to 0 .
3 and 0 . 5 , which we empirically determine in Section 4.2, to train the feature-wise transformation layers.
In practice, we ﬁnd that the cross-domain performance is not sensitive as long as the initialized values are within the same order (e.g., 0 .
and 0 . 3 ).
Here we report the results of training the RelationNet with the initialization { θ γ , θ β } = { 0 . 1 , 0 . 3 } .
In this experiment, we use the CUB dataset as the unseen domain for evaluation and conduct the training
[Table] [Table]
described in Algorithm 1.
The 5 -way 5 -shot classiﬁcation accuracy on the CUB dataset is 64 .
79 ± 0 .
55% , which is similar to the one we report in Table 2 ( i
[Table]
e., 64 . 99 ± 0 .
Learning-to-learn using a single domain.
The proposed learning-to-learn method requires multiple domains for training.
Here we apply the learning-to-learn method based on one single domain.
More speciﬁcally, we randomly sample two different tasks from the mini-ImageNet dataset in each iteration of the training process described in Algorithm 1.
One task serves as the pseudo-seen task, while the other one serves as the pseudo-unseen task.
We train the RelationNet model with the above-mentioned setting on 5-way 5-shot classiﬁcation using the mini-ImageNet dataset only.
As shown in Table 7, the performance improvement of the RelationNet model is not signiﬁcant compared to the model trained with pre-determined hyper-parameters { θ γ , θ β } = { 0 . 3 , 0 . 5 } .
This suggests that utilizing one single domain for learning the feature-wise transformation is not as effective as that using multiple domains (demonstrated in Table 2).
The reason is that during the training phase, the discrepancy of the feature distributions extracted from the psudo-seen and pseudo-unseen tasks is not as signiﬁcant since these two tasks are sampled from the same domain.
As a result, the hyper-parameters { θ γ , θ β } cannot be effectively optimized to capture the variation of feature distributions sampled from various domains.
Comparison with the state-of-the-art few-shot classiﬁcation on the mini-ImageNet.
We compare the metric-based frameworks applied with the proposed feature-wise transformation layers to the state-of-the-art few-shot classiﬁcation methods in Table 8.
In this experiment, we train the model with the pre-determined hyper-parameters of feature-wise transformation layers on the training set of the mini-ImageNet (Ravi & Larochelle, 2017) dataset.
Note that we do not use the learned version of the feature-wise transformation layers in the training to ensure fair comparison.
Combining Table 4 and Table 8, we observe that the metric-based frameworks train with 1) pre-trained feature encoder, and 2) feature-wise transformation layers with carefully hand-tuned hyper-parameters can demonstrate competitive performance.
[Table] [Table]
Comparison to the state-of-the-art few-shot classiﬁcation under domain shift.
We evaluate the metric-based frameworks with the proposed feature-wise transformation layers and the state-ofthe-art MetaOptNet approach Lee et al.
We use the model trained on the mini-ImageNet dataset provided by the authors for the evaluation on the other datasets.
6 As shown in Table 9, while the MetaOptNet method achieves state-of-the-art performance on the mini-ImageNet dataset, this approach also suffers from the domain shifts in the cross-domain setting.
Training the GNN framework with the pre-trained feature encoder and the proposed feature-wise transformation layers performs favorably against the MetaOptNet method under the cross-domain setting.