Improved Fine-Tuning by Better Leveraging Pre-Training Data
Abstract
As a dominant paradigm, fine-tuning a pre-trained model on the target data is widely used in many deep learning applications, especially for small data sets.
However, recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy once the number of training samples is increased in some vision tasks.
In this work, we revisit this phenomenon from the perspective of generalization analysis by using excess risk bound which is popular in learning theory.
The result reveals that the excess risk bound may have a weak dependency on the pre-trained model.
The observation inspires us to leverage pre-training data for fine-tuning, since this data is also available for fine-tuning.
The generalization result of using pre-training data shows that the excess risk bound on a target task can be improved when the appropriate pre-training data is included in fine-tuning.
With the theoretical motivation, we propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
Extensive experimental results for image classification tasks on 8 benchmark data sets verify the effectiveness of the proposed data selection based fine-tuning pipeline.
Our code is available at https://github.com/ziquanliu/NeurIPS2022_UOT_fine_tuning .
Introduction
After the success on ImageNet [1], deep learning attracts much attention and improves the performance of various computer vision tasks significantly, e.g., object detection [2], semantic segmentation[3].
However, as a result of expensive labeling, it is unlikely that we have sufficient labels for every application.
Fortunately, given a model pre-trained on a large-scale data set like
ImageNet, an effective model for the target data set, which may only have hundreds of examples, can be learned by fine-tuning the pre-trained model.
It is because many vision tasks are related [4] and a model learned from ImageNet that consists of more than one million examples can contain diverse semantic information and provides a better initialization than random initialization.
Despite the prevalence of fine-tuning pre-trained models, its theoretical understanding is unclear.
On the one hand, when sufficient training data is available, some research [5] has shown that training
[Figure]
from scratch can achieve the same accuracy as initialization with ImageNet pre-trained models after additional training.
As a warm-up of this work, we aim to understand this phenomenon from the theoretical side of generalization analysis [6].
Our analysis reveals that the final prediction precision may have a weak dependency on the pre-trained model when the target training set is large enough.
On the other hand, it is not realistic to have abundant labeled data for every target task.
So in many vision applications, training from scratch generally cannot match the performance of fine-tuning a pre-trained model.
However, our theoretical result tells us that when the pre-training data are too far from the target data, the domain gap will hurt the accuracy of target tasks.
These two observations lead to the following question: can we develop a new strategy of fine-tuning that achieves better generalization performance than the standard framework by reusing pre-training data in fine-tuning to reduce the domain gap?
This work will address this question with an affirmative answer.
Inspired by the theoretical observation, we propose to leverage the pre-training data, which is also available for fine-tuning, for target tasks.
Concretely, we propose to reuse pre-training data and optimize its task loss (e.g., cross-entropy loss for classification task) along with the target data when fine-tuning.
The generalization analysis confirms that the performance on the target data can be improved when an appropriate portion of pre-training data is selected, as illustrated in Figure 1.
Since target data can be from different domains, we study the reusing strategy of pre-training data for different cases.
First, when the target data is closely related to the pre-training data, one can randomly sample a number of pre-training data for fine-tuning, which is referred to as random selection .
Second, if the label information of pre-training data is available and the classes overlapped with target data are identifiable, one can directly use those data with overlapped classes in fine-tuning.
For example, given the data set of CUB [7], which consists of birds images, 59 bird classes [8] in
ImageNet can be reused in fine-tuning.
This scheme is referred to as label-based selection .
Finally, when the labels between pre-training and target domains cannot match exactly or pre-training data has no labels as in self-supervised pre-training, the similarity measured by representations extracted from the corresponding pre-trained model will be adopted for selection.
The last setting is prevalent in real-world applications and referred to as similarity-based selection .
Given the large scale of pre-training data, the representations from the pre-trained model can capture semantic similarity [9].
Based on this observation, we propose a novel selection algorithm to obtain a subset from pre-training data closest to the target data by solving an unbalanced optimal transport (UOT) problem.
Interestingly, the proposed method performs consistently well on other scenarios, e.g., labels are overlapped, which reduces the effort of identifying overlapped pre-training classes.
The main contributions of this work are summarized as follows.
• From the perspective of generalization analysis, this work explains the phenomenon that training from scratch has a similar final performance as fine-tuning the pre-trained model in some computer vision tasks, when the training data in the target domain is sufficient.
• We develop the generalization analysis when pre-training and target data are used in finetuning simultaneously, under some mild assumptions.
It demonstrates that the performance on target data will likely be improved when the pre-training data is similar to the target data.
• With the insight of generalization analysis, we propose to select a subset of pre-training data with better similarity to the target data to further boost the final performance.
A novel UOT-based algorithm is developed to handle target data from different scenarios.
• The performance of the proposed fine-tuning process is evaluated on 8 benchmark data sets for image classification tasks.
When a self-supervised pre-trained model is used, our method, UOT fine-tuning, surpasses the conventional fine-tuning pipeline by a large margin of 2 .
93% averaged over all tasks, verifying the effectiveness of reusing pre-training data.
Related Work
Fine-tuning as a special case of transfer learning [10, 11, 12, 13] aims to improve the performance on the target data by transferring the knowledge from a large-scale pre-training data to a target domain.
For example, supervised pre-trained models on ImageNet have been extensively used in image classification [9], object detection [2, 14] and semantic segmentation [3, 15].
However, the empirical study in [5] shows that the advantage of a supervised pre-trained model over random initialization cannot be observed when the gap between pre-training and target task is large, or the target task has sufficient training data and is trained for sufficient time.
Later, [4] demonstrates that self-supervised pre-training improves upon training from scratch in object detection and other vision tasks with strong data augmentation, indicating that self-supervised pre-training learns more general visual representations.
Our work considers a general pre-training paradigm including both supervised and self-supervised approaches, and explains why pre-trained models fail to significantly outperform random initialization from the view of generalization theory.
Different from existing work that regularizes the fine-tuning optimization explicitly [16, 17], we propose to reuse pre-training data in target training based on the theoretical findings.
There are several existing papers that explore the source data selection [18, 19, 20].
[18] improves fine-tuning by borrowing data from a source domain which is similar to the target domain.
The difference between [18] and our work are two-fold: 1) our work proposes a novel pre-training and fine-tuning pipeline while [18] is a joint training framework without pre-training ; 2) our proposed data selection is a global search method using deep features from the pre-training model while [18] uses low-level features and retrieves similar images with local search.
Our paper demonstrates the weakness of local search compared to global search in our experiment.
[19, 20] proposed similar schemes that pre-train the model on the selected subset from the pre-training data according to a domain similarity measure, but they do not use the selected data along with target data in the finetuning but re-pre-train on the selected data for every target task, which is a fundamental difference between their works and ours.
It is not surprising that such a re-pre-training framework brings benefit to a target task but it costs more computational time and resources than our fine-tuning framework.
Another type of works uses the relationship between source and target to improve fine-tuning [21, 22].
[21] exploits the relationship between source and target labels, and [22] trains a policy network to control the gradient mask for backbone’s blocks.
Our paper explicitly adds a selected portion of pre-training data in fine-tuning and the selection strategy can handle self-supervised pre-training since it does not need source labels.
[23] proposes to optimize a weight for the target data samples instead of selecting pre-training data for improving fine-tuning.
Such target-weighting methods are compatible with ours, which selects source data, and future work will consider their combination.
The similarity-based data selection scheme in this work is the one based on a variant of optimal transport (OT) optimization.
General OT is often used in computer vision to estimate or/and minimize the distance between two probability measures, such as prediction probabilities in classification [24], density maps in crowd counting [25] and the reconstruction loss in generative models [26, 27].
[28] measures the distance between two data sets by using OT and label information.
Our paper solves an unbalanced optimal transport (UOT) problem between pre-training and target data to obtain a similarity vector for pre-training data to select a portion of data close to the target task.
Main Results 3.1 Theoretical Understanding
Preliminary.
The target problem of interest that we aim to optimize can be formulated as
[Equation]
1) where θ is the model parameter to be learned; ( x, y ) is the input-label pair that follows a unknown distribution P ; E ( x,y ) ∼ P [ · ] is the expectation that takes over a random variable ( x, y ) while we use
E[ · ] for the sake of simplicity when the randomness is obvious; f ( · ; x, y ) is a loss function.
Suppose we have a set of training data { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } drawn from P are given, where n is the sample size.
In practice, we want to solve the following empirical version of problem (1):
[Equation]
Stochastic gradient descent (SGD) [29] is a very popular algorithm for solving problem (2) in many computer vision tasks, whose updating is given by θ t +
=
θ t − η ∇ θ f ( θ t ; x i t , y i t ) , where t = 0 , 1 , . . .
, η > 0 is the learning rate, ∇ θ f ( θ ; x, y ) is the gradient of function f
( θ ; x, y ) with respect to variable θ .
When the variable to be taken a gradient is obvious, we use ∇ f ( θ ; x, y ) for simplicity.
We use the excess risk (ER) as the performance measurement for a solution (cid:98) θ : F ( (cid:98) θ ) − F ( θ ∗ ) , where θ ∗ ∈ arg min θ ∈ R d F ( θ ) is the optimal solution of (1) and (cid:98) θ is the output of SGD.
In order to describe the pre-trained model, we denote by G ( θ ) :
=
E ( x ′ ,y ′ ) ∼ Q
[ g ( θ ; x ′ , y ′ )]
the objective function that the pre-trained model aims to optimize.
We also suppose that we have a set of training data { ( x ′ 1 , y ′ 1 ) , . . .
, ( x ′ m , y ′ m ) } drawn from Q .
Usually, the sample size of pre-training data is larger than that of target data, i.e., m ≫ n .
For the sake of analysis, we let m be large enough and both the pre-trained model and the target learning task share the same set of parameters.
In order to ensure that the model learned by optimizing G ( θ ) will be valuable to the optimization of F ( θ ) , we assume that the difference of their gradients is bounded.
That is, there exists a constant ∆ > 0 such that ∥∇ F ( θ ) − ∇ G ( θ ) ∥ ≤ ∆ , ∀ θ ∈ R d .
Value of Pre-trained Model.
To see the value of a pre-trained model, we present the excess risk bounds of the pre-trained model θ p and the final model θ f after fine-tuning θ p for the target task in the following lemma.
Lemma 1 (Informal) .
We have the following performance guarantees for target task F ( θ ) in expectation.
(1) The pre-trained model θ p provides F ( θ p ) − F ( θ ∗ )
≤ O (∆ 2 ) .
The final model θ f after fine-tuning θ p against a set of n training examples provides F ( θ f ) − F ( θ ∗ ) ≤ O (cid:16) log( n ∆ 2 ) n (cid:17) .
First, the performance gap between pre-trained model θ p and the optimal model θ ∗ is bounded by
O (∆ 2 ) , where ∆ describes the approximation accuracy when replacing ∇ F ( θ ) with ∇ G ( θ ) .
Second, note that ∆ only appears in the logarithmic term, implying that the excess risk bound has a weak dependency on the pre-trained model when n is large.
That is to say, when n is lager, the pre-trained model has a small effect on the final performance, which is consistent with the empirical results found in [5].
Value of Pre-training Data.
In reality, a target task often does not have enough data to fine-tune the model for a long time, so the pre-trained models are still better than random initialization in many vision tasks.
Nevertheless, Lemma 1 reveals that even n is not large, the dependency of generalization on a pre-trained model is potentially weak.
This leads to a natural question: is it possible to design a better fine-tuning process that can overcome the limitation of the existing one?
To this end, we develop a new approach for fine-tuning that aims to leverage the data used for the pre-trained model during the phase of fine-tuning.
In this way, we aim to solve the following problem during the fine-tuning process:
[Equation]
where α ∈ (0 , 1] is a constant, F n ( θ ) is defined in (2), and H m ( θ ) :
= 1 m (cid:80) mj
=1 h ( θ ; x ′ j , y ′ j ) where h is a loss function related to target task and ξ j := ( x ′ j , y ′ j ) is drawn from Q .
Note that the loss function h can be same as the loss function of the target task.
The solution is updated by SGD: θ t +1 =
θ t − η ∇ (cid:101) f ( θ t ) where ∇ (cid:101) f
( θ t ) is the stochastic gradient related to (3).
The theorem below provides a performance guarantee for fine-tuning via (3).
Theorem 1 (Informal) .
We have the following performance guarantee for target task F ( θ ) in expectation,
[Equation]
where δ 2 := max θ t ,ξ it { E[ ∥∇ F ( θ t ) − ∇ h
( θ t ; ξ i t )
∥ 2 ] } and θ f ∗ is the final model for the target task.
When δ 2 is small, by choosing appropriate α ∈ (0 , 1] , we may be able to further reduce the error from F ( θ f ∗ ) .
When δ 2 is large, that is, when the second term of bound in (4) dominates the total
[Figure]
error, then it would be worse than the result of standard fine-tuning a pre-trained model
Lemma 1.
Therefore, our goal is to select appropriate ξ i t , the training examples from pre-training data, such that ∇ h ( θ t ; ξ i t ) can better approximate ∇ F ( θ t ) .
These theoretical observations inspire us to design a selection strategy for pre-training data, that is, to select images similar to those of target data from pre-training data and use these selected images during fine-tuning.
3.2 The Proposed Data Selection Strategy
Theorem 1 shows that the benefit of a pre-trained model can be enhanced when pre-training data are used during the fine-tuning process.
This inspires us to propose data selection strategies and to choose an appropriate portion of pre-training data.
In experiments, we follow the standard pre-training practice in computer vision to use a deep neural network pre-trained on ImageNet, and then select data from ImageNet to help fine-tuning on target classification tasks.
We summarize the proposed pre-training data reuse strategies as follows.
Label-based Selection When the label information of pre-training data is available, and the overlapped classes with target data are recognizable
, one can simply select the overlapped classes and use them during the fine-tuning.
For instance, the bird images from ImageNet are all selected when fine-tuning CUB.
Unfortunately, this scheme heavily depends on the label match between pre-training and target data, which may worsen the performance in some real-world applications without perfectly matched classes.
Random Selection
The second data selection scheme is to choose classes with uniform sampling, referred to as random selection.
This strategy can improve the performance of target tasks if the domain gap δ 2 between pre-training and target data is small, and keep the weights close to initialization if the selected data are sufficiently large.
The drawback of uniform selection is that the domain gap δ 2 is not considered in the data reusing process, so the performance heavily depends on the inherent property of data sets.
Similarity-based Selection To reduce the domain gap
, we propose the third data selection scheme, an UOT-based method, to choose data classes from the pre-training set whose distributional distance to the target data set is small.
The UOT-selection method is able to handle pre-training data with and without labels.
When a supervised pre-trained model is used for fine-tuning, there are class labels for pre-training images so the selection unit of UOT is class.
When a self-supervised pre-trained model is used and there are no labels for pre-training images, we index the pre-training data by clustering in the feature space of the corresponding pre-trained backbone.
Therefore, for unlabeled pre-training data, the selection unit is cluster.
With the labels or cluster indices, each class/cluster is represented as the mean of deep features from the pre-trained model, e.g., 512-dim features from the penultimate layer of a pre-trained ResNet model.
Since the training set often has balanced classes, all classes or clusters are assigned with unit weights for both pre-training and target set.
So we have two density measures for the target set and pre-training set, i.e. { ( a i , w ( f ) i = 1) }
K f i =1 and { ( b j , w ( g ) j = 1) }
K g j =1 , respectively.
Denote the features of target and pre-training data as v ( f ) i and v ( g ) j , a i = (cid:80) y s =
i v ( f ) s /n
( f ) i and b j = (cid:80)
y t
=
j v ( g ) t /m
( g ) j , where n ( f ) i is the number of images in i -th class of target data and m ( g ) j is defined similarly for pre-training data.
In the general case where K f ̸ = K g , the two measures have different total masses so we propose to compute the unbalanced OT distance between the two by a generalized Sinkhorn iteration [30].
Specifically, the optimization objective is formulated as a UOT problem,
[Equation]
where C i,j is the distance between a i and b j ; P is the transportation matrix solved by the generalized
Sinkhorn iteration; τ 1 and τ 2 determine the constraint on the reconstruction loss of pre-training and target density measures; KL ( · , · ) and h ( · ) are Kullback-Leibler divergence and entropy function.
Note that as a result of unbalanced total masses, we cannot perfectly reconstruct pre-training and target measures at the same time.
Using this property, we can create a similarity ranking effect in the P1 vector by using a large value for τ 2 but a small value for τ 1 .
P1 is the density measure of pre-training data and P T 1 is the measure for target data.
Since we want all classes of the target data to be covered, a large τ 2 is needed; while we need to select a subset of classes, τ 1 should be small to relax the constraint.
Thus, a large [ P1 ] j indicates a high similarity of class- j of pre-training data to the target data.
Finally by ranking the elements in P1 and selecting top-K classes, we obtain the selected classes for a target data set.
Fig.
2 visualizes the UOT selection and the similarity vector given by UOT on the CUB data set.
3.3 Gradient Computation
We demonstrate how the gradient combination (3) is computed in the experiment of this work.
In the case where pre-training data has labels, we add two classification heads on top of the network backbone.
One classification head has K f -dim output to predict the target data and the other has K g -dim output to predict the pre-training data.
The optimization objective for the labeled case is
[Equation]
where { x t i , y t i } are the target data, { x ′
s i ,
y ′ s i } are the pre-training data and ˜ m, ˜ n are batch size.
λ is the weight for pre-training classification loss, which controls the weight α in (3).
Although the classification heads are different for pre-training and target data, we assume the optimization variables θ in f and h are consistent since the output layers only have a small amount of parameters compared to the backbone.
In the case where pre-training data has no labels, the unlabeled data is used in a semi-supervised way,
[Equation]
where the unlabeled pre-training data is processed with weak and strong data augmentation [31] respectively, and the probability prediction of weakly augmented data p ( y t i | x
′ s i ) is taken as the soft pseudo-label for the strongly augmented data ˜ x
′ s i .
Note that there is only one classification head in (7) and temperature or threshold is not used in the weak-strong training.
Experiments
This section presents the empirical analysis of reusing pre-training data in image classification tasks.
The experiment uses both supervised and self-supervised pre-trained models to fine-tune a variety of image classification data sets.
First, data reusing fine-tuning schemes consistently improves the performance of vanilla fine-tuning, which corroborates our theoretical result.
Second, the comparison between different data selection strategies demonstrates that the UOT selection is advantageous over random and greedy selection.
Third, we simulate the situations where the training data are scarce by sub-sampling the given training data and show that as the training data get insufficient, the performance gain of the pre-training data reusing method will increase.
Finally, some ablation studies on experimental settings are given.
4.1 Experiment Setup
The empirical study is done on both supervised and self-supervised pre-trained models.
For the supervised training, we use the official ResNet18 [32] pre-trained on ImageNet.
For the selfsupervised training, we use the official MoCo-v2 [33] ResNet50 pre-trained with 800 epochs.
In similarity-based selection, images are represented in the supervised pre-trained ResNet18 by 512dim features from the penultimate layer while in MoCo-v2 by 128-dim features from the final
[Table]
FC layer.
The pre-trained model is tested on 8 target image classification data sets, i.e. Stanford dogs (Dogs) [34], Stanford cars (Cars) [35], Caltech-UCSD birds (CUB) [7], Oxford-IIIT Pet (Pets) [36], SUN [37], FGVC-Aircraft (Aircraft) [38], Describable Textures data set (DTD) [39] and Caltech101 (Caltech) [40].
During the fine-tuning process, both the backbone and randomly initialized classification heads are updated using SGD with Nesterov Momentum.
The training epochs are fixed to be 100 in our experiment for sufficient training and the learning rate is divided by 10 at 60 and 80 epoch.
Other hyperparameters like initial learning rate, weight decay and λ are determined by grid search for all selection methods in the comparison (details in the Appendix).
When there are no labels in the pre-training data, K-means clustering [41] is used to estimate the cluster assignment with 128-dim features as input and cosine similarity as distance.
The cluster number is set asand
we give an ablation study in Table 2 on the cluster number.
We test 3 data selection methods: random selection, greedy selection and UOT selection, and set the number of selected classes to be 100 unless mentioned otherwise.
Specifically, we use the OT-based greedy algorithm [19] for comparison.
The batch size for fine-tuning data is 256, and if pre-training data are reused, the batch size keeps the same as target data which makes a total batch size of 512.
In random selection, we use the uniform selection over classes or clusters to be consistent with the other data selection methods.
In Greedy-OT, we use the same setting as in the original paper where C ij is the l 2 distance.
In UOT, we set ϵ = 1 . 0 , τ 1 = 1 . 0 and τ 2 = 100 . 0 .
The distance cost is based on the cosine similarity C ij = − cos( a i , b j )
+1 ϵ c with ϵ c = 0 .
For the supervised pre-trained model, we also compare with Co-Tuning [21], a strong transfer learning baseline.
The experiment setting follows the original paper and we search the initial learning rate from {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} on a validation set and report the test accuracy trained on the original training or train+val set.
Note that Co-Tuning relies on the labels and the classification head in pre-training so it is non-trivial to use Co-Tuning for a self-supervised pre-trained model.
We only compare with [21] since it is the most recent baseline which surpasses existing baselines on those datasets we evaluate.
[22] needs to train another policy network to help the adaptive fine-tuning, which is more complex than ours.
[19, 20] are pre-training methods that needs pre-training for every new task.
[18] does not consider the pre-training and fine-tuning pipeline and needs low-level features to do data selection.
4.2 Comparison of Data Selection Strategies
Tab. 1 shows the comparison between the standard fine-tuning, Co-Tuning and 3 data reuse methods on 8 image classification data sets, with supervised and self-supervised pre-trained model.
In the
[Figure]
cc u r ac y
[Figure]
u r
[Figure]
G a p
[Figure]
cc u
[Figure]
[Table]
[Figure]
Cars
[Table]
labeled data, 100 classes of ImageNet data are reused.
In the unlabeled data, 200 clusters are used to keep the number of selected images about the same as that in labeled data selection.
The first observation is that, since the pre-training data are large enough to have similar images to target ones, even random selection achieves better performance than the standard fine-tuning in most data sets.
Secondly, the benefit of data reuse is amplified by the similarity-based data selections, as predicted by Theorem 1.
Thirdly, UOT is better than Co-Tuning in 6 out of 8 data sets and its average accuracy has a clear benefit over Co-Tuning, indicating that the effectiveness of Co-Tuning is not robust to task variation.
In addition, the proposed data selection is more versatile since it performs well on the self-supervised model while Co-Tuning cannot handle the self-supervised model.
Finally, the comparison between Greedy-OT and UOT data selections demonstrates the advantage of the global UOT in terms of the similarity measure.
Fig.
3.a1 shows the performance of label-based selection on CUB (blue line), since the birds classes happen to exist in ImageNet.
It turns out the accuracy of UOT selection method (77.21%) is comparable to the label-based selection’s (76.89%).
In addition, we also test the performance of label-based selection on Dogs (selecting 118 dog classes of ImageNet), the performance of which (85.05%) is again comparable to UOT’s (84.67%).
This comparison demonstrates that the proposed UOT selection is generic yet effective.
The advantage of UOT is more evident in the self-supervised pre-training case than in the supervised pre-training one.
The most considerable improvement is achieved in Dogs, Birds and Pets data sets because the animal-related classes are dominant in ImageNet (398 classes of birds, dogs, animals and mammals) and self-supervised training learns good visual features without label information.
Once the label information is added to the fine-tuning process by data reuse, the model is taught to recognize those familiar features and achieves giant improvements.
Note that the only data on which data reuse does not help is the Cars, indicating that the gap between ImageNet and Cars data is large when measured by the self-supervised model.
When image labels in self-supervised pre-training are not available, the proposed data selection framework still outperforms the vanilla fine-tuning baseline on most data sets.
The benefit of UOT over greedy-OT is large on Dogs and CUB data sets, indicating that UOT is still better at selecting similar images from ImageNet even when label information is completely unknown.
[Figure]
A cc u r ac y
Similarity Similarity
UO T S c o r e ! !
R eca ll R a t e
4.3 Simulation of Low-Data Regime and Long-Tail Label Distribution
To study the effect of data reusing in the scarce data scenario, we simulate low-data target tasks in this experiment by sub-sampling CUB and Caltech training data.
We select these two data sets because they represent the fine-grained and general classification task, respectively.
For each class of training data we randomly sample 20% , 40% , 60% and 80% of images to get class-balanced training data.
Figure 3b shows that accuracy and performance gap between vanilla and data-reusing fine-tuning when different amounts of training data are available.
On both data sets, the performance gap is increased as the training data size decreases, indicating that the UOT-selection data reusing scheme helps more when the target data is insufficient.
This experiment demonstrates that the proposed data reusing paradigm is particularly effective when the target task does not have enough data, which could be a typical case in real-world applications.
[Table]
The long-tail class distribution is also a challenge in real-world applications.
We simulate long-tail data sets with CUB, Aircraft and Caltech by sampling images from classes with a Pareto distribution [42] as in [43], where the number of images in the largest class is 10 times that in the smallest class.
The results of fine-tuning a ResNet18 are shown in the Table 3.
The proposed method has a more significant improvement when the target data is imbalanced compared with the original balanced class data sets.
The reason is that the selected pre-training data can be controlled to have a balanced label distribution and the gradients of pre-training data have a regularization effect, so the model does not easily overfit images in minor classes.
4.4 Ablation Study
Number of selected pre-training classes.
We investigate the effect of selected class numbers on the target classification accuracy.
Figure 3a shows the performance of target tasks (CUB and
Caltech) when the number of selected classes ranges from 50 to 300 in UOT selection.
The increased pre-training data added in fine-tuning do not improve the performance of CUB, since there areclasses of birds in the ImageNet and more reused images enlarge the gap δ 2 .
Surprisingly, we observe that only using the bird images (blue line) is not the best strategy on CUB.
It is because that there can be a certain number of related classes in ImageNet, which will help the classification of bird images.
The result shows that even when labels of pre-training and target data are given and overlapped,
UOT selection can achieve better performance by including extra relevant classes from pre-training data.
On the general classification data set (Caltech), more reused pre-training data help gain the performance improvement because the diverse data set needs a large number of images to have a small domain gap.
On both data sets, the UOT selection performs better than the random selection as the number of selected classes changes.
Cluster number in K-means.
The cluster assignments are crucial to the performance of similaritybased selection, so we change the cluster number in the experiment and show the fine-tuning results of 8 data sets in Table 2.
On most data sets, K=2000 achieves better generalization than K=1000, indicating that fine-grained clustering helps the following data selection step.
However, if we further increase K, the clustering result will be quite noisy and extremely small or large clusters will be found, which makes the fine-tuning result worse than K=2000.
[Table]
Distance function and ϵ c .
To investigate the influence of different factors in the UOT selection, we define a recall rate as a metric to make the comparison.
For a target data set whose classes happen to exist in ImageNet, the similarity-based data selection is expected to choose those matched classes, e.g., selecting all 59 birds classes from ImageNet when fine-tuning on CUB.
Thus, the recall rate on
CUB is defined as the ratio between the number of bird classes in the top-100 similar vector or EMD similarity and 59.
With the performance metric, we first compare UOT with Greedy-OT using l 2 and cosine distance in Table 4.
With a supervised pre-trained model, Greedy-OT is only slightly worse than UOT, while with a self-supervised model the weakness of Greedy-OT is amplified.
It means that Greedy-OT heavily relies on the label information in supervised training but UOT only needs generic visual features to have a good similarity measure.
In addition, the cosine distance is better than the l 2 distance, especially in the self-supervised model.
The importance of cosine distance is due to the cosine similarity loss used in MoCo training [33].
Finally, Fig. 4a shows the recall rate when using different ϵ c .
The supervised model is not sensitive to the choice of ϵ c but a small ϵ c is crucial to the good performance of OT-selection in the self-supervised model.
Note that the recall rate of
Greedy-OT does not depend on ϵ c so the performance is worse than UOT no matter what ϵ c is used.
The impact of the domain gap.
We split ImageNet into 10 subsets based on UOT score (higher means higher similarity) to CUB data and reuse the 10 subsets in fine-tuning to see the results, which are presented in Fig. 4.b1-2.
With both supervised and self-supervised model, when the UOT similarity score rapidly decreases, the performance of UOT-fine-tuning drops and is lower than the fine-tuning baseline.
This experiment indicates that the dissimilar data hurt the performance when added to fine-tuning and highlights the importance of similarity-based data selection.
Subset of Pre-Training Data.
In the case where the pre-training data is larger than the available storage capacity, a subset of the pre-training data could be used.
We subsample 50% images from each class of ImageNet and use the 50% ImageNet data in the supervised ResNet18 fine-tuning.
On
CUB and Caltech, the UOT fine-tuning achieves 77.03% and 91.00% accuracy respectively, which only drop 0.18% and 0.11% compared with using the full ImageNet.
The experiment indicates that even with half of ImageNet, fine-tuning with UOT data selection is still quite effective.
Conclusion
This paper provides the generalization analysis of pre-trained models when fine-tuned on target tasks by using excess risk bound, which suggests that the pre-trained model can have little positive influence on learning from target data under certain conditions.
It also shows that the performance on the target data can be improved when similar data is selected from the pre-training data for fine-tuning.
Inspired by this result, a novel similarity-based selection algorithm is developed, which is evaluated on 8 data sets and shown to be effective.
Our future work will further explore the data reusing strategy in other computer vision tasks.
Broadly speaking, our research underpins the importance of pre-training data in downstream applications, which is neglected by current research focusing on pre-trained models, and thus has an impact in advocating the data-centric Artificial Intelligence (AI) [44].
One limitation of the current work is that sometimes the pre-training data is private (e.g., JFT-3B [45]) and not accessible.
Given the power of pre-training data as unveiled by our work, we advocate that more pre-training data should be open for the better use of pre-trained models.
Acknowledgement
This work was supported by Alibaba Group through Alibaba Research Intern Program, the Fundamental Research Funds for the Central University of China (DUT No. 82232031) and a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No.
CityU 11215820).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We study the general property of fine-tuning in computer vision.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the supplemental.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the supplemental.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See the supplemental.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We only report the best result among 3 trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See the supplemental.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] See the creators’ papers or reports.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[No] See the creators’ papers or reports.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] We use public datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Additional Experiment Details
Our experiment is run on 4 Nvidia V100 GPUs using PyTorch.
The hyperparameters in the fine-tuning are selected by a grid search on validation sets and the hyperparameter setting with the best accuracy is reported in Table 4 and 5.
The learning rate is searched over [0 . 1 , 0 . 03 , 0 . 01 , 0 . 003 , 0 .
001] , λ is over [1 . 0 , 0 . 3 , 0 . 1] , weight decay is over [10 − 4 , 10 − 5 , 0 . 0] .
The head and backbone learning rate ratio is searched over [10 . 0 , 1 . 0 , 0 . 1] and the ratio is fixed for one data set’s experiment.
In Table 4 and 5, for fine-tuning, the hyperparameter vector denotes [ learning rate , weight decay ] ; for data reusing, the hyperparameter vector denotes [ learning rate , λ, weight decay ] .
The hyperparameters are searched on a validation set and with the searched hyperparameters, we combine the training and validation set as the training set and report the test accuracy in Table 1.
Note that we choose the best performance for both baselines and our method, so the comparison is fair.
We also run 5 trials for UOT fine-tuning on CUB and Caltech using the supervised pre-trained ResNet18, the result is 76.99%±0.31% and 91.08%±0.24%, which are also clearly higher than the maximum performance of standard fine-tuning.
We next introduce individual data set.
Stanford Dogs.
The data set contains 120 dog classes and 20580 images.
We use the official train/test split in our experiment, where the training set has 12000 images with 100 images per class and the test set has 8580 dog images.
The performance metric for this data set is the top-1 accuracy.
Stanford Cars.
There are 196 car classes and 16185 car images.
We use the official train/test split where the training set has 8144 car images and 8041 car images.
The metric for the fine-tuning performance is the top-1 accuracy.
CUB.
The data set has 11788 bird images of 200 bird classes and is split into 5994 training images and 5794 for test images.
For the
classification performance, we use top-1 accuracy as the measurement.
Pets.
There are 37 pet categories with about 200 images for each class.
The data set contains a training set of 3680 images and a test set of 3669 images.
We use the (MPA) mean per-class accuracy as the performance metric for this data set following [46] for a consistent comparison.
SUN.
The scene understanding data set has 397 scene categories.
We use the first train/test split from the official data set.
There are 19850 training images and 19850 test images.
We report the topclassification accuracy.
Aircraft.
There are 100 aircraft classes in the data set, where the images are split into a training set with 3334 images, a validation set with 3333 images and a test set with 3333 images.
We report the
MPA for this data set.
DTD.
The texture classification set has 47 classes.
We use the first train/val/test split in the official data set, where the training, validation and test set has the same number of images, i.e., 1880.
Note that the accuracy reported in this paper is the result trained on train+val set, following [46].
We measure the classification performance using the top-1 accuracy.
Caltech.
Different from the previous data sets, this data set represents a general image classification task with 101 categories.
There is no official train/test splits, so we followed [46] to randomly sample 30 training images per class with the rest images used for testing.
We use MPA as the classification performance metric for this data set.
In the training data sub-sampling experiment, the UOT data selection is done on the sub-sampled data.
To keep the ratio between target data and pre-training data the same in different training data sizes, we sub-sample the images in each selected pre-training class with the same sub-sampling ratio as in target data.
The pre-training data sub-sampling is done for both UOT and random selection.
To show the importance of using pre-trained models in the UOT fine-tuning, we start the “fine-tuning” without pre-trained model initialization with only UOT selected pre-training data.
To make a fair comparison, we increase the training time from 100 epoch to 300 epoch and decay the learning rate at 180 and 240 epoch.
On CUB and Caltech, the accuracy without the supervised pre-trained ResNet is accuracy is 71.75% and 61.96%, which is far below the standard fine-tuning accuracy.
The reason is that the target data size is often much smaller compared with the pre-training data.
Such imbalance is an obstacle to learning the target task.
Starting from a pre-trained model, the imbalance issue is not serious, since the pre-trained model already learns a good initialization for optimization and the model has a lower loss on the pre-training data, which will not affect learning in the target task.
[Table]
[Table]
B Detailed Generalization Analysis
To establish the generalization bound, we first present some assumptions for problem (1) that will be used in the analysis.
Specifically, we make the following two assumptions, which are widely used in the literature [47, 48, 49].
Assumption 1.
There exists a constant ∆ > 0 such that ∥∇ F ( θ ) − ∇ G ( θ ) ∥ ≤ ∆ , ∀ θ ∈ R d .
Assumption 2 (Unbiased and Variance Bounded) .
The stochastic gradient of F ( θ ) is unbiased, i.e.,
E ( x,y ) [ ∇ f
( θ ; x, y )]
= ∇ F ( θ ) , and the variance of stochastic gradient is bounded, i.e., there exists a constant σ 2 > 0 , such that E ( x,y ) (cid:104) ∥∇ f
( θ ; x, y ))
− ∇ F ( θ ) ∥ 2 (cid:105) ≤ σ 2 .
Assumption 3 (Smoothness) .
F ( θ ) is smooth with an L -Lipchitz continuous gradient, i.e., it is differentiable and there exists a constant L >
such that ∥∇ F ( θ 1 ) − ∇ F ( θ 2 ) ∥
≤ L ∥ θ 1 − θ 2 ∥ , ∀ θ 1 , θ 2 ∈ R d .
Assumption 3 says the objective function F ( θ ) is smooth with module parameter L > 0 .
This assumption has an equivalent expression according to [50]: F ( θ 1 ) − F ( θ 2 ) ≤ ⟨∇ F ( θ 2 ) , θ 1 − θ 2 ⟩ + L 2 ∥
θ 1 − θ 2 ∥ 2 , ∀ θ 1 , θ 2 ∈ R d .
We further assume the objective function F ( θ ) satisfies a
Polyak-Łojasiewicz (PL) condition [51] with parameter µ > 0 .
Assumption 4 (PL condition) .
There exists a constant µ > 0 such that 2 µ ( F ( θ ) − F ( θ ∗ )) ≤ ∥∇ F ( θ ) ∥ 2 , ∀ θ ∈ R d , where θ ∗ ∈ arg min θ ∈ R d F ( θ ) is a optimal solution.
This PL condition is widely used in the literature (e.g., [48, 52, 53, 54]), and it has been observed in training deep neural networks both theoretically [55] and empirically [56].
Next, we describe an estimator of gradient ∇ F ( θ t ) for (3).
At each iteration t of fine tuning, the gradient estimator is given as
[Equation]
where α ∈ (0 , 1] , ζ i t :
= ( x i t , y i t ) is a training example from target data, ξ i t :=
{ ( x i t , y i t ) , i t = 1 , . . .
, (cid:101) m } is a set of training examples from pre-training data, and h is a loss function (e.g., crossentropy loss) that is related to target task.
Since the sample size of pre-training data is large enough, we use mini-batching stochastic gradient ∇ h ( θ t ; ξ i t ) :
= 1 (cid:101) m (cid:80) (cid:101) mi t =1 ∇ h
( θ t ; x i t , y i t ) where (cid:101) m is the batch size.
We then state the theoretical results as follows.
Lemma 2 (Formal Version of Lemma 1, Value of Pre-trained Model) .
We have following results for the pre-trained model.
(1) Under Assumptions 1, 3, 4, suppose the function g satisfies the condition of unbiased and bounded stochastic gradient as described in Assumption 2, by setting the learning rate η = min (cid:0) 1 , ∆ 2 / (2 σ 2 ) (cid:1) 1 /L in pre-training, then the pre-trained model, denoted by θ p , provides the following performance guarantee for the target task F ( θ ) , E[ F ( θ p ) − F ( θ ∗ )]
≤ ∆ 2 µ .
(2) Under Assumptions 2, 3, 4, suppose the learning rate η = 2 nµ log (cid:16) nµ ∆ 2
Lσ 2 (cid:17) ≤ 1 L in fine-tuning, then the final model after fine tuning θ p against a set of n training examples, denoted by θ f , provides the following performance guarantee for the target task F (
[Equation]
Theorem 2 (Formal Version of Theorem 1, Value of Pre-training Data) .
Under Assumptions 1, 2, 3, 4, suppose the function h satisfies the condition of unbiased and bounded stochastic gradient as described in Assumption 2 and the learning rate η = 2 nµ log (cid:16) nµ ∆ 2 2 αLσ 2 (cid:17) ≤ 1 L , by using the gradient estimator in (8) for updating solutions, we have
[Equation]
where δ 2 := max θ t ,ξ it { E[ ∥∇ F ( θ t ) − ∇ h
( θ t ; ξ i t )
∥ 2 ] } and θ f ∗ is the final model for the target task.
B.1 Proofs B.1.1 Proof of Lemma 2 (1)
Proof.
For the sake of simplicity, let the training examples ξ t :=
( x i t , y i t ) , i t = 1 , . . .
, m are sampled from Q .
By the smoothness of function F from Assumption 3, we have
[Equation]
where the last equality uses E[ ∇ g ( θ t ; ξ t )]
= ∇ G ( θ t ) .
Due to Assumptions 1, the condition of unbiased and bounded stochastic gradient for pre-trained objective function G ( θ ) and η ≤ 1 /L we have
[Equation]
Since F ( · ) is a µ -PL function under Assumption
[Equation]
we have
[Equation] [Equation]
and thus
[Equation]
13) where θ ∗ ∈ arg min θ ∈ R d F ( θ ) .
By selecting that η is small such that η ≤ ∆ 2 2 σ 2 L and selecting that T is sufficiently large, i.e. exp (cid:16) − ηµT 2 (cid:17) ( F ( θ 1 ) − F ( θ ∗ )) ≤ ∆ 2 4 µ , we have the following guarantee for the pre-trained model θ p
[Equation]
B.1.2 Proof of Lemma 2 (2)
Proof.
For the sake of simplicity, let the training examples ζ t :=
( x i t , y i t ) , i t = 1 , . . .
, n are sampled from P .
By the smoothness of function F from Assumption 3, we have
[Equation]
where the second equality is due to E[ ∇ f
( θ ; ζ )]
= ∇ F ( θ ) in Assumption 2 and the last inequality uses the fact that η ≤ 1 /L and Assumption 2.
Following the similar analysis as that for Lemma 2 (1) with θ 1 = θ p , for t , by using the condition that F ( · ) is a µ -PL function in Assumption 4 we have
[Equation]
and therefore
[Equation]
We complete the proof by plugging the bound for F ( θ p ) from Lemma 2 (1), i.e.,
[Equation]
By setting η = 2 nµ log (cid:16) nµ ∆ 2
Lσ 2 (cid:17)
[Equation]
we will have the following bound
[Equation]
B.1.3 Proof of Theorem
Proof.
Let ∇ H ( θ ) :
= E ξ [ ∇ h ( θ ; ξ )] .
By the smoothness of function F from Assumption 3, following the standard analysis, we have
E
[Equation]
m where δ 2 := max θ t
[Equation]
ξ it { E[ ∥∇ F ( θ t ) − ∇ h ( θ t ; ξ i t ) ∥ 2 ] } ; the last inequality is due to η ≤ 1 /L .
As a result
[Equation]
we have
[Equation]
By setting η = 2 nµ log (cid:16) nµ ∆
[Equation]
2 (cid:17) and θ f ∗ = θ n +1 , the inequality (17) will lead to the following bound
[Equation]
where the last inequality is due to (cid:101) m is large enough such that (cid:101) m ≥ 2 σ 2 δ .