Smooth Transfer Learning for Source-to-Target Generalization
Abstract
Transfer learning for deep models has shown great success for various recognition tasks.
Typically, a backbone network is pre-trained on a source dataset, then ﬁne-tuned on a target dataset.
We considered that when both datasets are at hand, learning them simultaneously at least for some period of iterations would yield higher test performance rather than the step-wise optimization.
We propose
Smooth Transfer Learning, which uses a learnable scheduler function for the loss coefﬁcients so that degrees of contributions from two datasets can be smoothly changed along training time for optimal target performance.
The scheduler function is designed so that it can express either pre-training-then-ﬁne-tuning or multi-task learning with ﬁxed weights as special cases.
Our method consistently outperforms these special cases in object classiﬁcation with CIFAR-10 and CIFAR-100, and in digit classiﬁcation with SVHN and MNIST.
Introduction
It is widely known that domain gap between the training and test deteriorates generalizability of a machine-learning model.
If, however, one has even a small dataset from the target domain at training time, there is a higher chance to greatly improve the model performance.
Transfer learning is one of the popular methods [1, 2, 3, 4] for such cases.
The popularity stems from the fact that the model ﬁne-tuning on a target dataset likely improves performance compared to direct training on a target dataset from random initialization.
The performance gain is reasoned by the generalizability of feature extractors obtained by the pre-training [3], especially when a massive source dataset is used.
However, the performance gain becomes negligible [5] or even negative [6] under certain conditions.
He et al.
have shown evidences [5] that direct training on a target detection dataset with sufﬁciently many iterations yields no clear performance drop compared to ﬁne-tuning after pre-training on a large dataset such as ImageNet [7].
Fine-tuning can even deteriorate the performance when source and target domains signiﬁcantly differ [6].
This type of conﬂicting settings is disregarded in this work.
Is it best to adopt the step-wise optimization; i.e., ﬁne-tuning on a target dataset after pre-training on a source dataset (“pre-training-then-ﬁne-tuning")?
We hypothesized that when both datasets are at hand, learning both datasets simultaneously for an entire period, or at least for some period ∗ Equal contribution. of iterations would yield higher test performance rather than the step-wise optimization.
Switching datasets in the middle of training almost surely deteriorates performance for the source domain towards the end; therefore, it may limit the generalizability of the feature extractors.
When data from both datasets are used for update, the task required by the source dataset likely works as regularizer , which is somewhat similar to auxiliary training [8].
A non-triviality about the above idea of simultaneous learning is the asymmetry between the training and test objectives.
The training objective now consists of two sub-objectives from the source and target tasks, while the test objective is solely from the target task.
Naive minimization of weighted loss functions with either constant weights [9] or with adaptive weights [10, 11] as seen in multi-task learning may not be the best strategy for the test objective.
In this work, we employ hyperparameter optmization scheme [12], as it can handle an arbitrary metric for the validation dataset, to optimize a specially-designed, learnable scheduler function for the source and target loss weights.
Contributions of this work are summarized as follows:
• We propose Smooth Transfer Learning that generalizes the popular practice of pre-training-thenﬁne-tuning by introducing a specially-designed, learnable scheduler function for source and target loss weights.
The proposed method can express pre-training-then-ﬁne-tuning and multi-task learning with ﬁxed weights as special cases. • We demonstrate that the proposed method consistently outperforms pre-training-then-ﬁne-tuning and multi-task learning with ﬁxed weights for object classiﬁcation task using CIFAR-10 and CIFAR-100 datasets [13] and digit classiﬁcation task using SVHN [14] and MNIST [15] datasets.
• We report intriguing tendencies found by experiments such that the semi-optimized scheduler function approaches pre-training-then-ﬁne-tuning behavior as the size of the target dataset becomes small, and approaches the behavior of multi-task learning with ﬁxed weights as the size of the target dataset becomes large.
In the intermediate region, the semi-optimized scheduler function prefers a transition period that gradually switch from the source to the target dataset
Smooth Transfer Learning
We assume that a source dataset D S and a target dataset for classiﬁcation are given.
The target dataset is splitted into a training set D Ttr and a validation set D Tva .
The objective is to generate a network that yields high test performance on the target domain.
To simplify notations, we assume the dimension of an input x and that of a target variable y , which is a one-hot vector indicating the true class label, are the same between the target and source datasets, while one can easily extend different dimensionalities.
We propose Smooth Transfer Learning, which uses a specially-designed, learnable scheduler function for the loss weights such that degrees of contributions from two datasets can be smoothly changed along training time.
The form of the scheduler function is designed with only a few learnable parameters so that it can express either naive pre-training-then-ﬁne-tuning or multi-task learning with ﬁxed weights; therefore, the proposed method is a generalization of these extreme cases.
The difﬁculty on optimizing such a scheduler lies in the asymmetry between the training objective for the network (source and target) and the test objective (target).
To address this asymmetry, we employ a hyperparameter search to optimize the parameters of the scheduler function for the target domain.
The network architecture considered in this work has a domain-generic backbone g θ ( · ) with a parameter set θ and two parallel domain-speciﬁc heads h T φ T ( · ) , h S φ S ( · ) with target-speciﬁc parameter set φ T and source-speciﬁc parameter set φ S .
The sample-wise loss functions are deﬁned as
[Equation]
where s ( · ) denotes the softmax function and L ( · ) denotes the cross-entropy loss function.
The mini-batch loss function is then deﬁned as
[Equation]
[Equation]
T/
T t 0.
( t )
T/
T t 0.
( t )
T/
T t 0.
( t )
T/
T t 0.
( t )
[Figure]
Here, a mini-batch consists of a sub-batch B T from the target dataset D Ttr and a sub-batch B S from the source dataset D S .
The set Θ consists of θ, φ T and φ S .
The degrees of contributions from two sub-batches are controlled by the loss weight λ 0 ∈ [0 , 1] , which we will extend to a function of training time t (in the units of iterations) as discussed below.
We deﬁne the scheduler function for the loss weight as the sigmoid function of the form
[Equation]
where T is the total number of iterations and ω = { t 0 , τ, α } denotes a set of tunable scalars.
Now, we illustrate that this scheduler function can realize the following extreme cases.
Pre-training-then-ﬁne-tuning:
In the limit of τ → 0 with a ﬁnite α ,
[Equation]
[Equation] > [Equation] In this extreme case, λ ω ( t ) essentially behaves as a step function that changes the value at T/ 2 + t 0 .
In other words, the training process boils down to the conventional pre-training-then-ﬁne-tuning, where the transfer timing is controlled by t 0 , as illustrated in Fig.
1 (a) and (b).
Multi-task learning with ﬁxed weights: In the limit of τ → ∞ with a ﬁnite t 0 ,
[Equation]
which is just a constant function, whose value is controlled by α .
This tells that multi-task learning with ﬁxed weights can be expressed in this limit, as illustrated in Fig.
1 (c) and (d).
It is worth mentioning about the parametrization of λ ω ( t ) .
A sigmoid function is typically expressed with two parameters of gain and offset.
The reason why we introduced the third one is to characterize each parameter to carry a speciﬁc meaning.
Namely, the gain τ is responsible for the maximum slope, the gain-independent offset t 0 is responsible for the transfer timing, and the gain-dependent offset α is responsible for the ﬁxed weight.
We could eliminate, say α , but it requires a very complex parameter tuning for both τ and t 0 such that t
= α (cid:48) τ → ∞ to express a constant function.
We use a hyperparameter search to optimize the scheduler parameters ω for the validation dataset of the target domain.
With a trial ω , we apply the following update rule or its variant to the parameter Θ
For t = 1 , · · · , T :
[Equation]
(7) where B T t and B S t are uniformly sampled sub-batches from D Ttr and D S , respectively.
Temporal model generation is repeated many times with different trials of ω to search for the best ω and Θ in terms of validation accuracy in the target domain.
We employ Tree-structured Parzen Estimator (TPE) [16] for the search.
The gain τ is searched in log domain, and other two offsets are searched in linear domain so that parameter setting close to the two extreme cases can be sufﬁciently searched.
Evaluation
The experimental purpose is to compare the test performance of models trained by Smooth Transfer
Learning and simple baselines including pre-training-then-ﬁne-tuning and multi-task learning with
[Table]
( t ) 74k
148k t 0.
( t ) 74k
148k t 0.
( t )
N = 65 ,
N = 20 ,
N = 6 ,
N = 2 ,
N =
N = 78k
157k t 0.
( t ) 78k 157k t 0. ( t ) 78k 157k t 0.
( t ) 78k 157k t 0. ( t ) 78k 157k t 0.
( t ) 78k 157k t 0.
( t )
Roughly speaking, the scheduler approaches pre-training-then-ﬁne-tuning (multi-task learning with ﬁxed weights) when N is small (large). ﬁxed weights.
We are also interested in how a semi-optimal scheduler behaves for | D Ttr | (cid:28) | D S | and | D Ttr | (cid:39) | D S | .
In the former case where the target dataset is much smaller than the source dataset, we expect that it approaches pre-training-then-ﬁne-tuning behavior, since ﬁne-tuning has been found to be beneﬁcial for this case. In the latter case, the maximum slope would be much more moderate. We conducted two sets of experiments. One is for object classiﬁcation using CIFAR-10 [13] as the target dataset and CIFAR-100 [13] as the source dataset.
The other is for digit classiﬁcation using
SVHN [14] as the target and MNIST [15] as the source.
Training details are given in Appendix.
Quantitative results for generalization abilities are summarized in Table 1.
Recalling that the proposed method is a generalization of the baselines and the hyperparameter search yields semi-optimal hyperparameters for the validation set, it is no surprise that ours outperforms the baselines.
One can observe that ‘Fine-tuning’ (‘Multi-task’) tends to perform better than ‘Multi-task’ (‘Fine-tune’) when the target dataset size is much smaller than (similar to) the source dataset size.
When the target dataset size is relatively small, having many iterations with the target samples would easily cause overﬁtting.
Thus, switching the datasets in the middle of the training would be a better strategy than to keep using the target samples from the beginning with a ﬁxed loss weight.
In contrast, when the target dataset is as large as the source dataset, target samples can be trained longer without experiencing overﬁtting.
Dataset switching would not be ideal in this case because forgetting of the source dataset likely happens.
Obtained scheduler functions are shown in Fig.
Although the number of trials by TPE is not large enough in these preliminary experiments 2 to yield truly the best validation performance, we still observe following tendencies.
Roughly speaking, the semi-optimal scheduler approaches pre-trainingthen-ﬁne-tuning (multi-task learning with ﬁxed weights) when the target dataset size is much smaller than (similar to) the source dataset size.
In the object classiﬁcation task with non-extreme N (say
N = 1 , 420 ), the transition is quite smooth; λ ω ( t ) changes from a value (cid:39) 0 to a value (cid:39) 1 slowly over the entire period of training.
Conclusion
We proposed Smooth Transfer Learning, which uses a learnable scheduler function for the loss weight.
The scheduler function is designed to generalize the pre-training-then-ﬁne-tuning and multi-task learning with ﬁxed weights.
We conﬁrmed that our method outperforms these special cases in image classiﬁcation tasks.
Obtained scheduler functions nearly behave as the pre-training-then-ﬁne-tuning (multi-task learning with ﬁxed weights) when the target dataset size is much smaller than (similar to) the source dataset size
Acknowledgements. This work is an outcome of a research project, Development of Quality Foundation for Machine-Learning Applications, supported by DENSO IT LAB Recognition and Learning Algorithm Collaborative Research Chair (Tokyo Tech.)
A Training details A.1 Training details speciﬁc to the object classiﬁcation task
We use ResNet20 [17] architecture in all experiments of the object classiﬁcation task.
Among the target CIFAR-10 samples, we randomly split 50,000 default training samples into 45,000 training samples and 5,000 validation samples.
From the 45,000 training samples, we further construct partial datasets of different sizes N by random sampling to examine how performance differs for different training dataset sizes.
We use the same sub-batch size 128 for the source and target in the multi-task learning and Smooth Transfer Learning.
Other than those, we use the mini-batch size of 128, except for the single dataset training with
N = 140 and the ﬁne-tuning with N = 140 , where the mini-batch size 16 is used.
We run the single dataset training and the source dataset training in the conventional transfer learning for 200 epochs.
Duration of the multi-task learning and that of Smooth Transfer Learning is 380 epochs with respect to the source dataset.
The learning rate is scheduled by the cosine annealing rule.
The peak and ﬁnal learning rates are 0.1 and 1e-3, respectively.
Weight decay rate is 1e-4.
We do not use dropout in the ResNet20.
We used batch normalization in all experiments of the object classiﬁcation task.
In the multi-task learning and Smooth Transfer Learning, batch statistics are shared between the source and target datasets.
TPE setting is as follows.
We use 11 workers that run in parallel.
Each worker is responsible for training one model at a time.
We repeat the parallel training 9 times, and at each time a worker generates a trial parameter set ω which stems from the one that performs best so far on the validation dataset.
A.2 Training details speciﬁc to the digit classiﬁcation task
We use WideResNet-10-2 [18] architecture in all experiments of the digit classiﬁcation task.
Among the target SVHN samples, we split 73,257 default training samples into 65,931 training samples (90%) and 7,326 validation samples (10%).
From the 65,931 training samples, we further construct partial datasets of different sizes N by random sampling to examine how performance differs for different training dataset sizes.
We use the same sub-batch size 128 for the source and target in the multi-task learning and Smooth Transfer Learning.
Other than those, we use the mini batch size of 128.
We run the single dataset training and the source dataset training in the conventional transfer learning for 160 epochs. Duration of the multi task
learning
and that of Smooth Transfer Learning is 336 epochs with respect to the source dataset
The learning rate is scheduled by the cosine annealing rule.
The peak and ﬁnal learning rates are 1e-2 and 1e-4, respectively
Weight decay rate is 5e-4.
Dropout ratio used in the WideResNet-10-2 is 0.
We used batch normalization in all experiments of the digit classiﬁcation task.
In the multi-task learning and Smooth Transfer Learning, dataset-speciﬁc batch statistics are used; i.e. , the source and target datasets have different sets of batch statistics.
We checked that use of shared batch statistics brings noticeable drops in test accuracies.
As for TPE, we used 9 workers which sequentially train models 11 times.
A.3 Training details common in all experiments
We adopt cosine annealing for learning rate scheduling.
Linear warmup is adopted in the beginning part of training for a period of 2.5% of total duration.
We used momentum SGD with momentum rate of 0.9.
Early stopping is applied on each experiment.
Model parameters are saved as certain intervals during total training epochs mentioned above and a model that yields best validation accuracy is tested.