SMILE: Sample-to-feature MIxup for Efficient Transfer LEarning
Abstract
To improve the performance of deep learning, mixup has been proposed to force the neural networks favoring simple linear behaviors in-between training samples.
Performing mixup for transfer learning with pre-trained models however is not that simple, a high capacity pre-trained model with a large fully-connected (FC) layer could easily overfit to the target dataset even with samples-to-labels mixed up.
In this work, we propose SMILE — Sample-to-feature Mixup for EffIcient Transfer
LEarning.
With mixed images as inputs, SMILE regularizes the outputs of CNN feature extractors to learn from the mixed feature vectors of inputs, in addition to the mixed labels.
SMILE incorporates a mean teacher to provide the surrogate "ground truth" for mixed feature vectors.
Extensive experiments have been done to verify the performance improvement made by SMILE , in comparisons with a wide spectrum of transfer learning algorithms, including fine-tuning, L2-SP, DELTA,
BSS, RIFLE, Co-Tuning and RegSL, even with mixup strategies combined.
Introduction
Mixup [1] is an effective strategy for improve generalization performance of DNN , where the objective is to have DNNs in the learning procedure favor the linear behaviors in-between training samples.
To achieve the goal, the mixup strategy picks up multiple images from the training set, mixes the samples and labels proportionally to generate a new pair of sample and label for data augmentation.
The regularization effects brought by mixup could help control the complexity of
DNN models [2, 3] while largely improving the robustness and generalization performance [4].
As a kind of data augmentation, mixup is naturally expected to be more essential when training data are insufficient.
However, we surprisingly find the contrary in the scenario of deep transfer learning: when fine-tuning a pre-trained model with a few target examples, mixup improves the performance with reduced margins or even downgrades the performance when the target dataset is small.
See
Figure 1 for detailed results.
Considering the strong capacity of pre-trained models and the limited training dataset, our research yields the concern that, can fine-tuning with pre-trained models overfit to the mixed-up samples and labels? [Figure]
[Figure] Our Observations We find fine-tuning with high capacity pre-trained models CAN overfit to the mixup samples/labels.
From mixup, we simply derive a linear interpolation ( IL ) loss to measure the error of linear interpolation between a pair of samples ( x 1 , y 1 ) and ( x 2 , y 2 ) for the model f ( · ) ,
[Equation]
(1) where a lower linear interpolation loss indicates stronger linear behaviors in-between the samples and usually better generalization performance [4].
Our experiments however find that fine-tuning with mixup could obtain a low interpolation loss in the training set while suffering a high interpolation loss in the test set ( ≥ 25% higher interpolation loss on the testing set than the one on training set, please see also in Section 4 ).
This observation indicates that the linear behaviors gained by vanilla mixup could not well generalize to the testing dataset and overfit to the mixup samples/labels from the training dataset.
These inaccurately interpolated deep features may help less or even harm the learning of the target task.
Thus, our research intends to study a way to make mixup strategies generalizable in deep transfer learning settings while significantly improving the performance of DNNs.
Our Work To achieve the above goal, in this work, we propose SMILE —Sample-to-feature Mixup strategies for Efficient Transfer Learning.
One of the major challenges lies in that, unlike the vanilla mixup, there are actually no ground-truth labels for mixed features.
To tackle this problem, we introduce two kinds of pseudo feature labels , by deeply exploiting the generalizability of the pretrained model.
Specifically, given two samples drawn from the target domain as the input, SMILE first linearly combines two samples proportionally and sends the mixed-up sample to the target model.
Then, the following two regularizers are employed on the mixed output.
• Mixup on target deep features.
It constrains the Euclidean distance between the output of target model’s CNN feature extractor and a mixed-up feature vector (i
e., linear combination of a mean teacher model’s outputs for the two samples).
• Mixup on adapted source labels.
An additional FC classifier for the target network to adapt the target dataset but in the source label domain.
It is regularized to learn from the linear combination of classification results given by the teacher classifier.
We carry out extensive experiments using a wide range of source and target datasets.
Results show that SMILE can outperform a number of state-of-the-art baseline algorithms including L 2 -SP [5],
DELTA [6], BSS [7], RIFLE [8], Co-Tuning [9] and RegSL [10] with/without vanilla mixup strategies.
Related work
The related works to this study include [11, 12] for mixup strategies and [6–9] for transfer learning through regularizing feature space.
We here particularly focus on the discussion on the manifold [Figure] mixup strategy [11] and CutMix [12].
Compared to the feature-to-label mixup used in [11], SMILE proposed to a sample-to-feature strategy.
Though MixCut [12] also uses sample-to-feature mixup to spatially fuse multiple images into one to form new feature maps accordingly, SMILE regularizes the CNN feature extractor to learn from a surrogate of “fine-tuned” feature vectors of mixed-up images even when the CNN has not yet been well-tuned in the target domain.
Our work is also highly relevant with a concurrent work MixSKD [13] which also imposes sample-to-feature mixup, while our paper clearly figures out the over-fitting risk of vanilla mixup in few-sample fine-tuning scenarios.
Experiments
We first present the experiment results based on Image Classification tasks on three popular object recognition datasets: CUB-200-2011 [16], Stanford Cars [17] and FGVC-Aircraft [18], which are intensively used in state-of-the-art transfer learning literatures [7–9].
Each of these datasets contains about 6k - 8k training samples.
We use ImageNet [19] pre-trained ResNet-50 [20] as the source model.
For each dataset, we create four random subsets with different number of categories and training examples.
To further confirm the performance improvement by SMILE is independent with the choice of pretrained datasets and model architectures, we conduct additional experiments comparing our method with competitive baselines.
Specifically, we use the Places365 [21] pre-trained ResNet-50 to perform fine-tuning on MIT-Indoors-67 [22], which is a scene classification task.
We also evaluate our method on a more powerful model EfficientNet-B4 [23] designed by NAS over a large scale dataset
Food-101 [24].
The descriptions about these benchmarks are summarized in Table 1.
SMILE is compared against multiple state-of-the-art fine-tuning algorithms including L 2 [14], L 2SP [5], DELTA [6], BSS [7], RIFLE [8], Co-Tuning [9], RegSL [10] and Mixup [1].
As observed in Table 2 and 3, our proposed SMILE achieves remarkable improvements to vanilla fine-tuning on three standard benchmarks, and outperforms all state-of-the-art methods.
As the number of training examples becomes smaller, our method yields more significant benefits, e [Table] g. SMILE outperforms vanilla fine-tuning by more than 8% on FGVC-Aircraft when 15% training samples are used [Table]
[Table]
[Table]
[Table]
[Table]
Linear Interpolation Effects and Generalization
We use Eq 1 to measure Label-IL using the classifier outputs and Feature-IL using the last hidden layer of ResNet-50 for different transfer learning methods, with CUB-200-2011 (with 30% sampling rates) as the training set for all methods.
Several arguments can be deduced from results in Table 4.
(1) More Data, Better Generalization, and Lower Label-IL and Feature-IL.
There is no doubt to assume that, in practice, a model trained with more data should enjoy better generalization performance.
In addition to improve the testing accuracy , we find that, when we involve additional training samples, both Label-IL and Feature-IL would be lower on the testing sets, compared to vanilla fine-tuning. (2 Fine-tuning with vanilla mixup is NOT generalizable even in the label space, due to the lack of linear interpolation in feature spaces.
As shown in Table 4, although Label-IL of the vanilla mixup is significantly lower on the training set than other methods, its Label-IL is high on the testing set (not generalizable).
Furthermore, compared to other methods on both training/testing sets (even
Fine-tuning on the testing set), Feature-IL of the vanilla mixup is high, i.e., poor linear interpolation in feature spaces. (3) Sample-to-Feature Mixup could ensure the generalizability of mixup effects in the label space, as
SMILE is with low Feature-IL and Label-IL on both training and testing sets.
While SMILE achieves the lowest Feature-IL on both training and testing datasets, it also achieves the lowest testing Label-IL.
The comparisons with vanilla mixup suggest that doing mixup in the label space is just not enough for fine-tuning.
These arguments solidify our motivation of sample-to-feature mixup for fine-tuning.
Conclusion
In this work, we figure out the difficulty of applying mixup in transfer learning, and introduce SMILE —
Sample-to-feature Mixup strategies for Efficient Transfer Learning.
Beyond a direct combination of fine-tuning and mixup, SMILE pursues generalizable linear behaviors through both features of the target domain and the label space of the source domain.
We conduct extensive experiments using a wide spectrum of target datasets.
Results show that SMILE can significantly promote the effectiveness of fine-tuning and outperform various competitive fine-tuning algorithms.
Ablation studies and empirical discussions further backup our design intuition and purposes.