Shift and Scale is Detrimental To Few-Shot Transfer
Abstract
Batch normalization is a common component in computer vision models, including ones typically used for few-shot learning.
Batch normalization applied in convolutional networks consists of a normalization step, followed by the application of per-channel trainable afﬁne parameters which shift and scale the normalized features.
The use of these afﬁne parameters can speed up model convergence on a source task.
However, we demonstrate in this work that, on common few-shot learning benchmarks, training a model on a source task using these afﬁne parameters is detrimental to downstream transfer performance.
We study this effect for several methods on well-known benchmarks such as cross-domain few-shot learning (CD-FSL) benchmark and few-shot image classiﬁcation on miniImageNet.
We ﬁnd consistent performance gains, particularly in settings with more distant transfer tasks.
Improvements from applying this low-cost and easy-to-implement modiﬁcations are shown to rival gains obtained by more sophisticated and costly methods.
Introduction
Advances in few shot learning (FSL) have allowed deep learning models to adapt data representations with just a few labelled samples from novel classes [21, 19, 3].
Throughout the literature, most
FSL methods make use of Batch-Norm (BN) [8] layers when training on source datasets to speed up model convergence and overcome the problem of internal covariate shift.
Adding BN layers to deep neural networks stabilizes the distribution of layer inputs by controlling the mean and variance of these distributions, leading to improved performance across a plethora of computer vision tasks.
Despite the ubiquity of BN layers, specially in deep convolutional neural networks (CNN), the source of their effectiveness is still poorly understood.
Furthermore, recent studies have argued that this effectiveness may not be due to reducing the covariate shift, as originally believed [18].
Computationally, BN layers [8] consist of two steps.
First, each input is normalized according to the mean and standard deviation across the spatial dimensions of a mini-batch for each of its channels.
In this paper, we refer to this ﬁrst process as “Feature-Normalization”, or FN.
These normalized inputs are then scaled and shifted by the trainable coefﬁcient γ and bias β (the afﬁne parameters).
From the perspective of domain adaptation, Li et al.
[12] state that “the label related knowledge is stored in the weight matrix of each layer, whereas domain related knowledge is represented by the statistics of the
Batch-Normalization”.
In other words, afﬁne parameters are responsible for adjusting the statistics of Batch-Norm (domain knowledge) according to the underlying correlation among features (label knowledge).
This deﬁnition gives rise to a reasonable question: Are the afﬁne parameters still useful when facing a novel target domain?
From ﬁg. 1, in the absence of the afﬁne parameters, the preceding weight layers could substitute their role, resulting in similar output distributions of BN layers.
The auxiliary beneﬁts of these afﬁne parameters to weight layers has been studied in a recent work [4].
However, the negative effect, when facing novel labels of a new target domain, has not been explored.
In this work, we investigate the effect of replacing the BN layers with Feature-Normalization on the generalizability of deep CNNs within the setting of cross domain few-shot transfer.
In subsequent sections, we provide a formal deﬁnition of Feature-Normalization and then evaluate the effect of this replacement on some well-known cross domain few-shot methods and datasets.
Methods
Let S be a batch of labelled examples { ( x si , y si ) }
Ni =1 of size N from a source domain D s and Θ be a deep convolutional neural network consisting of L layers with weight matrices θ l where l represents the layer index.
If h represents the intermediate features of Θ for layer l , the Feature-Normalization layer at layer l is computed for each channel and can be deﬁned as 2 :
[Equation]
Here, c is the channel, µ c and σ c are the ﬁrst and second moments of h c respectively deﬁned as:
[Equation]
where H and W are the spatial dimensions of h
[Figure]
Related work
Few shot learning Research on FSL has typically been made on the basis that there is little domain shift between the source and novel classes [3, 15, 20, 9, 5, 2, 21].
Recently, research has been carried out to tackle FSL settings where the domain gap between the source and target data is large [14].
Work in semi-supervised FSL [16, 10, 17, 22] utilizes unlabelled data to train the softmax classiﬁer in the evaluation stages.
Advances in self-training and self-supervised learning have led to promising solutions for FSL tasks with large domain gaps.
A notable state-ofthe-art approach in the setting of distant tasks is STARTUP [14], which employs a combination of self-training and self-supervised learning components for cross domain fewshot learning (CDFSL).
Batch-normalization BN layers enable the training of deeper networks while speeding up model convergence [8] and producing a slight regularization effect [13].
The authors of Batch-Norm [8] claim that it reduces internal covariate shift under the assumption that the normalization of features alleviates the dramatic changes in distribution to a weight layer’s input.
This assumption was cast into uncertainty in [18] where the authors manually induced internal covariate shift post batch-normalization to ﬁnd minimal changes in training time.
Further studies show that BN layers smoothen the optimization process [18] and help prevent the exploding gradients problem [1].
On domain adaptation, AdaBN [11] utilizes BatchNorm to transform data features from different domains into representations with similar distributions.
In image classiﬁcation tasks where the source and target data are from the same distribution, Frankle et al.
[4] highlight the expressive powers of the BN afﬁne parameters.
They conduct experiments which show that these afﬁne parameters play a positive role towards improving model performance.
However, this work does not take into consideration settings where there is a distributional gap between the training and target labels.
In this paper, we explore the role of afﬁne parameters towards the generizability of few shot learners in the presence of a distributional shift between the source training and target data.
Experiments
[Table] The experiments are conducted on the publicly available CDFSL benchmark [6], STARTUP [14] codes and provided datasets.
For fairness and simplicity, in this work, we follow the same evaluation setting used for experiments on the CDFSL benchmark [6] and STARTUP [14].
The optimizer settings and learning rate used in this work follows STARTUP.
The Baseline is standard transfer learning trained for 1000 epochs on miniImageNet with a batch size of 128.
AdaBN utilizes the Baseline model and adapts to the novel domain using 20% of unlabelled samples from the target dataset, trained for an additional 10 epochs.
STARTUP’s teacher model is trained for 400 epochs on miniImageNet and its student model is trained for 1000 epochs on unlabelled samples from 20% of each target dataset, both using a batch size of 256.
The remaining 80% of target datasets are utilized for ﬁne-tuning, as described in section 4.1.
All methods make use of the ResNet-10 architecture [7].
Results reported in tableare evaluated on samples from 900 unseen classes of ImageNet.
In all experiments, the weights of the feature extractor are frozen after the representation learning phase.
A linear classiﬁer is then trained on the support set of the target dataset and the model is evaluated on the query set.
All the experiments were carried out using the Tesla V100 SXM2 16 GB GPU.
4.1 Benchmark
We perform experiments on the challenging CDFSL benchmark [6].
MiniImageNet [21] is used as the base representation learning dataset.
The target dataset is comprised of 4 datasets, each with samples from very different distributions: EuroSAT (satellite imagery to determine land usage), CropDiseases (plant images to identify botanical diseases), ChestX (chest X-rays to determine diagnosis) and ISIC2018 (images of skin abrasions to detect melanoma).
Similar to [6], we perform experiments in an FSL linear classiﬁcation setting where the support set is composed of 5 classes with k samples per class (5-way k-shot), where k ∈ {1, 5, 20, 50} and the overall scores are an average of accuracies over all target datasets for k ∈ {5, 20, 50}.
Evaluation is carried out over 600 episodes, 95% conﬁdence intervals with reported mean accuracy.
For the experiments on AdaBN and STARTUP, we randomly sample 20% of unlabelled images from novel classes in the target dataset, following a similar setup to that of STARTUP [14].
The remaining samples are used during inference.
4.2 Results
Table 2 reports the results of our experiments.
Across all datasets, and the 5, 20 and 50 levels of shot (as is in the CDFSL benchmark), the average performance of the models conﬁgured with FN exceeds that of BN models across all four datasets.
FN outperforms BN models, notably surpassing [Table] the BN baseline by nearly 5 points for 1-shot classiﬁcation on CropDisease.
It can be observed that simply conﬁguring the Baseline model with FN obtains results that rival (within error bars) the more complex and computationally expensive STARTUP, which employs a large amount of unlabelled data to bridge the domain gap.
Table 1 reports the performance of the FN and BN conﬁgurations on the experiment with the unseen classes of ImageNet.
Although the accuracies of both models on validation data are almost the same (ﬁg. 2), in FSL settings where there is a relatively smaller domain shift, FN consistently surpasses
BN models by an average of 1.56%.
However, from ﬁg. 2, it can be inferred that the FN does not improve performance on non-Few-Shot settings, and the afﬁne parameters indeed aid convergence on in-distribution validation data.
Conclusion
Feature-Normalization layer improves the few-shot generalization accuracy on shifted domains by leveraging a lower amount of the model’s parameters.
By stabilizing the output distribution of convolutional layers, Feature-Normalization improves robustness against distributional shifts.
It captures and normalizes the statistical distribution of data features while preventing the afﬁne from overﬁtting to the training source labels.
Feature-Normalization is completely consistent with well known Batch-Norm implementations.
It could be easily integrated in existing CNN architectures and used by SOTA methods.
It is observed that the proposed normalization technique only helps in few-shot transfer and the effect is more pronounced as the data distribution shifts.
This is an initial study and we will consider further experiments on the effect of afﬁne parameters in other few shot benchmarks and methods.
Acknowledgements
The authors are grateful to CIFAR for funding and Compute Canada for computing resources.