Improved Fine-Tuning by Better Leveraging Pre-Training Data
Anonymous Author(s)
AffiliationAddress email
Abstract
As a dominant paradigm, fine-tuning a pre-trained model on the target data iswidely used in many deep learning applications, especially for small data sets.
However, recent studies have empirically shown that training from scratch has thefinal performance that is no worse than this pre-training strategy once the numberof training samples is increased in some vision tasks.
In this work, we revisit thisphenomenon from the perspective of generalization analysis by using excess riskbound which is popular in learning theory.
The result reveals that the excess riskbound may have a weak dependency on the pre-trained model.
The observationinspires us to leverage pre-training data for fine-tuning, since this data is alsoavailable for fine-tuning.
The generalization result of using pre-training data showsthat the excess risk bound on a target task can be improved when the appropriatepre-training data is included in fine-tuning.
With the theoretical motivation, wepropose a novel selection strategy to select a subset from pre-training data to helpimprove the generalization on the target task.
Extensive experimental results forimage classification tasks on 8 benchmark data sets verify the effectiveness of theproposed data selection based fine-tuning pipeline.
Introduction
After the success on ImageNet [1], deep learning attracts much attention and improves the performance of various computer vision tasks significantly, e g., object detection [2], semanticsegmentation[3].
However, as a result of expensive labeling, it is unlikely that we have sufficientlabels for every application.
Fortunately, given a model pre-trained on a large-scale data set like
ImageNet, an effective model for the target data set, which may only have hundreds of examples, canbe learned by fine-tuning the pre-trained model.
It is because many vision tasks are related [4] and amodel learned from ImageNet that consists of more than one million examples can contain diversesemantic information and provides a better initialization than random initialization.
Despite the prevalence of fine-tuning pre-trained models, its theoretical understanding is unclear.
Onthe one hand, when sufficient training data is available, some research [5] has shown that trainingfrom scratch can achieve the same accuracy as initialization with ImageNet pre-trained models afteradditional training.
As a warm-up of this work, we aim to understand this phenomenon from thetheoretical side of generalization analysis [6].
Our analysis reveals that the final prediction precisionmay have a weak dependency on the pre-trained model when the target training set is large enough.
On the other hand, it is not realistic to have abundant labeled data for every target task.
So in manyvision applications, training from scratch generally cannot match the performance of fine-tuning apre-trained model.
However, our theoretical result tells us that when the pre-training data are too farfrom the target data, the domain gap will hurt the accuracy of target tasks.
These two observations lead to the following question: can we develop a new strategy of fine-tuningthat achieves better generalization performance than the standard framework by reusing pre-training
[Figure]
generalization pre training method vanilla data in fine-tuning to reduce the domain gap?
This work will address this question with an affirmativeanswer.
Inspired by the theoretical observation, we propose to leverage the pre-training data, whichis also available for fine-tuning, for target tasks.
Concretely, we propose to reuse pre-training dataand optimize its task loss (e.g., cross-entropy loss for classification task) along with the target datawhen fine-tuning.
The generalization analysis confirms that the performance on the target data can beimproved when an appropriate portion of pre-training data is selected, as illustrated in Figure 1.
Since target data can be from different domains, we study the reusing strategy of pre-training datafor different cases.
First, when the target data is closely related to the pre-training data, one canrandomly sample a number of pre-training data for fine-tuning, which is referred to as randomselection .
Second, if the label information of pre-training data is available and the classes overlappedwith target data are identifiable, one can directly use those data with overlapped classes in fine-tuning.
For example, given the data set of CUB [7], which is a data set consisting of birds images, 59 birdclasses [8] in ImageNet can be included in fine-tuning.
This scheme is referred to as label-basedselection .
Finally, when the labels between pre-training and target domains cannot match exactlyor pre-training data has no labels as in self-supervised pre-training, the similarity measured byrepresentations extracted from the corresponding pre-trained model will be adopted for selection.
The last setting is prevalent in real-world applications and referred to as similarity-based selection .
Given the large scale of pre-training data, the representations from the pre-trained model can capturesemantic similarity [9].
Based on this observation, we propose a novel selection algorithm to obtain asubset from pre-training data closest to the target data by solving an unbalanced optimal transport(UOT) problem.
Interestingly, the proposed method performs consistently well on other scenarios,e.g., labels are overlapped, which reduces the effort of identifying overlapped pre-training classes.
The main contributions of this work are summarized as follows.
• From the perspective of generalization analysis, this work explains the phenomenon that
training from scratch has a similar final performance as fine-tuning the pre-trained model insome computer vision tasks, when the training data in the target domain is sufficient.
• We develop the generalization analysis when pre-training and target data are used in finetuning simultaneously, under some mild assumptions.
It demonstrates that the performance
on target data will likely be improved when the pre-training data is similar to the target data.• With the insight of generalization analysis, we propose to select a subset of pre-training
data with better similarity to the target data to further boost the final performance.
A novel
UOT-based algorithm is developed to handle target data from different scenarios.
• The performance of the proposed fine-tuning process is evaluated on 8 benchmark data sets for image classification tasks.
When a self-supervised pre-trained model is used, our method surpasses the conventional fine-tuning pipeline by a large margin of 2 .
93% averaged overall tasks, verifying the effectiveness of reusing pre-training data.
Related Work
Fine-tuning as a special case of transfer learning [10, 11, 12, 13] aims to improve the performanceon the target data by transferring the knowledge from a large-scale pre-training data.
For example,supervised pre-trained models on ImageNet have been extensively used in image classification [9],object detection [2, 14] and semantic segmentation [3, 15].
However, the empirical study in [5] showsthat the advantage of a supervised pre-trained model over random initialization cannot be observedwhen the gap between pre-training and target task is large, or the target task has sufficient training dataand is trained for sufficient time.
Later, [4] demonstrates that self-supervised pre-training improvesupon training from scratch in object detection and other vision tasks with strong data augmentation, indicating that self-supervised pre-training learns more general visual representations.
Our workconsiders a general pre-training paradigm including both supervised and self-supervised approaches,and explains why pre-trained models fail to significantly outperform random initialization fromthe view of generalization theory.
Different from existing work that regularizes the fine-tuningoptimization explicitly [16, 17], we propose to reuse pre-training data in target training based on thetheoretical findings.
There are several existing papers that explore the source data selection [18, 19, 20].
[18] improves finetuning by borrowing data from a source domain which is similar to the target domain.
The differencebetween [18] and our work are two-fold: 1) our work proposes a novel pre-training and fine-tuningpipeline while [18] is a joint training framework without pre-training ; 2) our proposed data selectionis a global search method using deep features from the pre-training model while [18] uses low-levelfeatures and retrieves similar images with local search.
This paper demonstrates the weakness oflocal search compared to global search in our experiment.
[19, 20] proposed similar schemes thatpre-train the model on the selected subset from the pre-training data according to a domain similaritymeasure, but they do not use the selected data along with target data in the fine-tuning but re-pre-trainon the selected data for every target task, which is a fundamental difference between their worksand this paper.
It is not surprising that such a re-pre-training framework brings benefit to a targettask but it costs more computational time and resources than our fine-tuning framework.
Anothertype of works uses the relationship between source and target to improve fine-tuning [21, 22].
[21]exploits the relationship between source and target labels, and [22] trains a policy network to controlthe gradient mask for backbone’s blocks.
This paper explicitly adds a selected portion of pre-trainingdata in fine-tuning and the selection strategy can handle self-supervised pre-training since it does notneed source labels.
[23] proposes to optimize a weight for the target data samples instead of selectingpre-training data for improving fine-tuning.
Such target-weighting methods are compatible with ours,which selects source data, and future work will consider their combination.
The similarity-based data selection scheme in this work is the one based on a variant of optimaltransport (OT) optimization.
General OT is often used in computer vision to estimate or/and minimizethe distance between two probability measures, such as prediction probabilities in classification [24],density maps in crowd counting [25] and the reconstruction loss in generative models [26, 27].
[28]measures the distance between two data sets by using OT and label information.
This paper solvesan unbalanced optimal transport (UOT) problem between pre-training and target data to obtain asimilarity vector for pre-training data to select a portion of data close to the target task.
Main Results
3.1 Theoretical Understanding
Preliminary.
The target problem of interest that we aim to optimize can be formulated as
[Equation]
where θ is the model parameter to be learned; ( x, y ) is the input-label pair that follows a unknowndistribution P ; E ( x,y ) ∼ P [ · ] is the expectation that takes over a random variable ( x, y ) while we use
E[ · ] for the sake of simplicity when the randomness is obvious; f ( · ; x, y ) is a loss function.
Supposewe have a set of training data { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } drawn from P are given, where n is the samplesize.
In practice, we want to solve the following empirical version of problem (1):
[Equation]
Stochastic gradient descent (SGD) [29] is a very popular algorithm for solving problem (2) in manycomputer vision tasks, whose updating is given by θ t {++
+}
= θ t − η ∇ θ f ( θ t ; x i t , y i t ) , where t = 0 , 1 , . . .
,η > 0 is the learning rate, ∇ θ f ( θ ; x, y ) is the gradient of function f
( θ ; x, y ) with respect to variableθ .
When the variable to be taken a gradient is obvious, we use ∇ f ( θ ; x, y ) for simplicity.
Weuse the excess risk (ER) as the performance measurement for a solution (cid:98) θ : F ( (cid:98) θ ) − F ( θ ∗ ) , whereθ ∗ ∈ arg min θ ∈ R d F ( θ ) is the optimal solution of (1) and (cid:98) θ is the output of SGD.
In order to describe the pre-trained model, we denote by G ( θ ) :
=
E ( x ′ ,y ′ ) ∼ Q
[ g ( θ ; x ′ , y ′ )]
the objectivefunction that the pre-trained model aims to optimize.
We also suppose that we have a set of trainingdata { ( x ′ 1 , y ′ 1 ) , . . .
, ( x ′ m , y ′ m ) } drawn from Q .
Usually, the sample size of pre-training data is largerthan that of target data, i.e., m ≫ n .
For the sake of analysis, we let m be large enough and both thepre-trained model and the target learning task share the same set of parameters.
In order to ensure that the model learned by optimizing G ( θ ) will be valuable to the optimization of F ( θ ) , we assumethat the difference of their gradients is bounded.
That is, there exists a constant ∆ > 0 such that∥∇ F ( θ ) − ∇ G ( θ ) ∥ ≤ ∆ , ∀ θ ∈ R d .
Value of Pre-trained Model.
To see the value of a pre-trained model, we present the excess riskbounds of the pre-trained model θ p and the final model θ f after fine-tuning θ p for the target task inthe following lemma.
Lemma 1 (Informal) .
We have the following performance guarantees for target task F ( θ ) inexpectation.
(1) The pre-trained model θ p provides F ( θ p ) − F ( θ ∗ )
≤ O (∆ 2 ) .
The final model θ fafter fine-tuning θ p against a set of n training examples provides F ( θ f ) − F ( θ ∗ ) ≤ O (cid:16) log( n ∆ 2 ) n (cid:17) .
First, the performance gap between pre-trained model θ p and the optimal model θ ∗ is bounded by
O (∆ 2 ) , where ∆ describes the approximation accuracy when replacing ∇ F ( θ ) with ∇ G ( θ ) .
Second,note that ∆ only appears in the logarithmic term, implying that the excess risk bound has a weakdependency on the pre-trained model when n is large.
That is to say, when n is lager, the pre-trainedmodel has a small effect on the final performance, which is consistent with the empirical resultsfound in [5].
Value of Pre-training Data.
In reality, a target task often does not have enough data to fine-tune themodel for a long time, so the pre-trained models are still better than random initialization in manyvision tasks.
Nevertheless, Lemma 1 reveals that even n is not large, the dependency of generalizationon a pre-trained model is potentially weak.
This leads to a natural question: is it possible to designa better fine-tuning process that can overcome the limitation of the existing one?
To this end, wedevelop a new approach for fine-tuning that aims to leverage the data used for the pre-trained modelduring the phase of fine-tuning.
In this way, we aim to solve the following problem during thefine-tuning process:
[Equation]
where α ∈ (0 , 1] is a constant, F n ( θ ) is defined in (2), and H m ( θ ) :
= 1 m (cid:80) mj
=1 h ( θ ; x ′ j , y ′ j ) whereh is a loss function related to target task and ξ j := ( x ′ j , y ′ j ) is drawn from Q .
Note that the lossfunction h can be same as the loss function of the target task.
The solution is updated by SGD:
[Equation] f ( θ t ) where ∇ (cid:101) f
( θ t ) is the stochastic gradient related to (3).
The theorem belowprovides a performance guarantee for fine-tuning via (3).
Theorem 1 (Informal) .
We have the following performance guarantee for target task F ( θ ) inexpectation,
[Equation]
where δ 2 := max θ t ,ξ it { E[ ∥∇ F ( θ t ) − ∇ h
( θ t ; ξ i t )
∥ 2 ] } and θ f ∗ is the final model for the target task.
When δ 2 is small, by choosing appropriate α ∈ (0 , 1] , we may be able to further reduce the errorfrom F ( θ f ∗ ) .
When δ 2 is large, that is, when the second term of bound in (4) dominates the totalerror, then it would be worse than the result of standard fine-tuning a pre-trained model in Lemma 1.
Therefore, our goal is to select appropriate ξ i t , the training examples from pre-training data, suchthat ∇ h ( θ t ; ξ i t ) can better approximate ∇ F ( θ t ) .
These theoretical observations inspire us to designa selection strategy for pre-training data, that is, to select images similar to those of target data frompre-training data and use these selected images during fine-tuning.
3.2 The Proposed Data Selection Strategy
Theorem 1 shows that the benefit of a pre-trained model can be enhanced when pre-training data areused during the fine-tuning process.
This inspires us to propose data selection strategies and to choosean appropriate portion of pre-training data.
In experiments, we follow the standard pre-trainingpractice in computer vision to use a deep neural network pre-trained on ImageNet, and then selectdata from ImageNet to help fine-tuning on target classification tasks.
We summarize the proposedpre-training data reuse strategies as follows.
Label-based Selection When the label information of pre-training data is available, and theoverlapping classes with target data are recognizable, one can simply select the overlapping classesand use them during the fine-tuning.
For instance, the bird images from ImageNet are all selectedwhen fine-tuning CUB.
Unfortunately, this scheme heavily depends on the label match betweenpre-training and target data, which may worsen the performance in some real-world applicationswithout perfectly matched classes.
[Figure]
Random Selection
The second data selection scheme is to choose classes with uniform sampling,referred to as random selection.
This strategy can improve the performance of target tasks ifthe domain gap δ 2 between pre-training and target data is small, and keep the weights close toinitialization if the selected data are sufficiently large.
The drawback of uniform selection is that thedomain gap δ 2 is not considered in the data reusing process, so the performance heavily depends onthe inherent property of data sets.
Similarity-based Selection To reduce the domain gap
, we propose the third data selection scheme,an UOT-based method, to choose data classes from the pre-training set whose distributional distanceto the target data set is small.
The UOT-selection method is able to handle pre-training data with andwithout labels.
When a supervised pre-trained model is used for fine-tuning, there are class labels forpre-training images so the selection unit of UOT is class.
When a self-supervised pre-trained modelis used and there are no labels for pre-training images, we index the pre-training data by clustering inthe feature space of the corresponding pre-trained backbone.
Therefore, for unlabeled pre-trainingdata, the selection unit is cluster.
With the labels or cluster indices, each class/cluster is represented as the mean of deep features from 199 the pre-trained model, e.g., 512-dim features from the penultimate layer of a pre-trained ResNet18model.
Since the training set often has balanced classes, all classes or clusters are assigned with unit 201 weights for both pre-training and target set.
So we have two density measures for the target set andpre-training set, i.e. { ( a i , w ( f ) i = 1) }
K f i =1 and { ( b j , w ( g ) j = 1) }
K g j =1 , respectively.
Denote the featuresof target and pre-training data as v ( f ) i and v ( g ) j , a i = (cid:80)
y s =
i v ( f ) s /n
( f ) i and b j = (cid:80)
y t
=
j v ( g ) t /m
( g ) j ,where n ( f ) i is the number of images in i -th class of target data and m ( g ) j is defined similarly forpre-training data.
In the general case where K f ̸ = K g , the two measures have different total masses
so we propose to compute the unbalanced OT distance between the two by a generalized Sinkhorniteration [30].
Specifically, the optimization objective is formulated as a UOT problem,min
[Equation]
where C i,j is the distance between a i and b j ; P is the transportation matrix solved by the generalized
Sinkhorn iteration; τ 1 and τ 2 determine the constraint on the reconstruction loss of pre-training andtarget density measures; KL ( · , · ) and h ( · ) are Kullback-Leibler divergence and entropy function.
Note that as a result of unbalanced total masses, we cannot perfectly reconstruct pre-training andtarget measures at the same time.
Using this property, we can create a similarity ranking effect inthe P1 vector by using a large value for τ 2 but a small value for τ 1 .
P1 is the density measure ofpre-training data and P T 1 is the measure for target data.
Since we want all classes of the target datato be covered, a large τ 2 is needed; while we need to select a subset of classes, τ 1 should be smallto relax the constraint.
Thus, a large [ P1 ] j indicates a high similarity of class- j of pre-training datato the target data.
Finally by ranking the elements in P1 and selecting top-K classes, we obtain theselected classes for a target data set.
Fig.
2 visualizes the UOT selection and the similarity vectorgiven by UOT on the CUB data set.
3.3 Gradient Computation
We demonstrate how the gradient combination (3) is computed in the experiment of this work.
In 222 the case where pre-training data has labels, we add two classification heads on top of the networkbackbone.
One classification head has K f -dim output to predict the target data and the other has 224 K g -dim output to predict the pre-training data.
The optimization objective for the labeled case is
[Equation]
where { x t i , y t i } are the target data, { x ′
s i ,
y ′ s i } are the pre-training data and ˜ m, ˜ n are batch size.
λis the weight for pre-training classification loss, which controls the weight α in (3).
Although the 227 classification heads are different for pre-training and target data, we assume the optimization variables [Table] θ in f and h are consistent since the output layers only have a small amount of parameters compared 229 to the backbone.
In the case where pre-training data has no labels, the unlabeled data is used in asemi-supervised way,
[Equation]
where the unlabeled pre-training data is processed with weak and strong data augmentation [31]respectively, and the probability prediction of weakly augmented data p ( y t i | x
′ s i ) is taken as the softpseudo-label for the strongly augmented data ˜ x
′ s i .
Note that there is only one classification head in(7) and temperature or threshold is not used in the weak-strong training.
Experiments
This section presents the empirical analysis of reusing pre-training data in image classification tasks.
The experiment uses both supervised and self-supervised pre-trained models to fine-tune a varietyof image classification data sets.
First, data reusing fine-tuning schemes consistently improves theperformance of vanilla fine-tuning, which corroborates our theoretical result.
Second, the comparisonbetween different data selection strategies demonstrates that the UOT selection is advantageousover random and greedy selection.
Third, we simulate the situations where the training data arescarce by sub-sampling the given training data and show that as the training data get insufficient,the performance gain of the pre-training data reusing method will increase.
Finally, some ablationstudies on experimental settings are given.
4.1 Experiment Setup
The empirical study is done on both supervised and self-supervised pre-trained models.
For thesupervised training, we use the official ResNet18 [32] pre-trained on ImageNet.
For the selfsupervised training, we use the official MoCo-v2 [33] ResNet50 pre-trained with 800 epochs.
Insimilarity-based selection, images are represented in the supervised pre-trained ResNet18 by 512dim features from the penultimate layer while in MoCo-v2 by 128-dim features from the final
FC layer.
The pre-trained model is tested on 8 target image classification data sets, i.e. Stanforddogs (Dogs) [34], Stanford cars (Cars) [35], Caltech-UCSD birds (CUB) [7], Oxford-IIIT Pet(Pets)
[36], SUN [37], FGVC-Aircraft (Aircraft)
[38], Describable Textures data set (DTD)
[39]and Caltech101 (Caltech) [40].
During the fine-tuning process, both the backbone and randomlyinitialized classification heads are updated using SGD with Nesterov Momentum.
The training epochsare fixed to be 100 in our experiment for sufficient training and the learning rate is divided by 10 at60 and 80 epoch.
Other hyperparameters like initial learning rate, weight decay and λ are determinedby grid search for all selection methods in the comparison (details in the supp.).
When there are nolabels in the pre-training data, K-means clustering [41] is used to estimate the cluster assignmentwith 128-dim features as input and cosine similarity as distance.
The cluster number is set as 2000and we give an ablation study in Table 2 on the cluster number. [Table]
We test 3 data selection methods: random selection, greedy selection and UOT selection, and set thenumber of selected classes to be 100 unless mentioned otherwise.
Specifically, we use the OT-basedgreedy algorithm [19] for comparison.
The batch size for fine-tuning data is 256, and if pre-trainingdata are reused, the batch size keeps the same as target data which makes a total batch size of 512.
Inrandom selection, we use the uniform selection over classes or clusters to be consistent with the otherdata selection methods.
In Greedy-OT, we use the same setting as in the original paper where C ij isthe l 2 distance.
In UOT, we set ϵ = 1 . 0 , τ 1 = 1 . 0 and τ 2 = 100 . 0 .
The distance cost is based on thecosine similarity C ij = − cos( a i , b j )
+1 ϵ c with ϵ c = 0 .
For the supervised pre-trained model, we also compare with Co-Tuning [21], a strong transfer learningbaseline.
The experiment setting follows the original paper and we search the initial learning ratefrom {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} on a validation set and report the test accuracy trained on theoriginal training or train+val set.
Note that Co-Tuning relies on the labels and the classification headin pre-training so it is non-trivial to use Co-Tuning for a self-supervised pre-trained model.
We onlycompare with [21] since it is the most recent baseline which surpasses existing baselines on thosedatasets we evaluate.
[22] needs to train another policy network to help the adaptive fine-tuning,which is more complex than ours.
[19, 20] are pre-training methods that needs pre-training for everynew task.
[18] does not consider the pre-training and fine-tuning pipeline and needs low-level featuresto do data selection.
4.2 Comparison of Data Selection Strategies
Tab. 1 shows the comparison between the standard fine-tuning, Co-Tuning and 3 data reuse methodson 8 image classification data sets, with supervised and self-supervised pre-trained model.
In thelabeled data, 100 classes of ImageNet data are reused.
In the unlabeled data, 200 clusters are used tokeep the number of selected images about the same as that in labeled data selection.
The first observation is that, since the pre-training data are large enough to have similar images totarget ones, even random selection achieves better performance than the standard fine-tuning in mostdata sets.
Secondly, the benefit of data reuse is amplified by the similarity-based data selections, aspredicted by Theorem 1.
Thirdly, UOT is better than Co-Tuning in 6 out of 8 data sets and its averageaccuracy has a clear benefit over Co-Tuning, indicating that the effectiveness of Co-Tuning is notrobust to task variation.
In addition, the proposed data selection is more versatile since it performswell on the self-supervised model while Co-Tuning cannot handle the self-supervised model.
Finally,the comparison between Greedy-OT and UOT data selections demonstrates the advantage of theglobal UOT in terms of the similarity measure.
Fig.
3.b1 shows the performance of label-basedselection on CUB (blue line), since the birds classes happen to exist in ImageNet.
It turns out theaccuracy of UOT selection method (77.21%) is comparable to the label-based selection’s (76.89%).
In addition, we also test the performance of label-based selection on Dogs (selecting 118 dog classesof ImageNet), the performance of which (85.05%) is again comparable to UOT’s (84.67%).
Thiscomparison demonstrates that the proposed UOT selection is generic yet effective.
The advantage of UOT is more evident in the self-supervised pre-training case than in the supervisedpre-training one.
The most considerable improvement is achieved in Dogs, Birds and Pets data setsbecause the animal-related classes are dominant in ImageNet (398 classes of birds, dogs, animalsand mammals) and self-supervised training learns good visual features without label information.
Once the label information is added to the fine-tuning process by data reuse, the model is taught torecognize those familiar features and achieves giant improvements.
Note that the only data on whichdata reuse does not help is the Cars, indicating that the gap between ImageNet and Cars data is largewhen measured by the self-supervised model.
When image labels in self-supervised pre-training arenot available, the proposed data selection framework still outperforms the vanilla fine-tuning baselineon most data sets.
The benefit of UOT over greedy-OT is large on Dogs and CUB data sets, indicatingthat UOT is still better at selecting similar images from ImageNet even when label information iscompletely unknown.
(b1) Sub-Sampling CUB (b2) Sub-Sampling Caltech
A cc u r ac y A cc u r ac y
G a p
A cc u r ac y
G a p
A cc u r ac y
Percentage of Training Data
Percentage of Training Data (a1) CUB (a2) Caltech
A cc u r ac y A cc u r ac y
G a p
A cc u r ac y
[Figure]
To study the effect of data reusing in the scarce data scenario, we simulate low-data target tasks in thisexperiment by sub-sampling CUB and Caltech training data. We select these two data sets becausethey represent the fine-grained and general classification task, respectively.
For each class of trainingdata we randomly sample 20% , 40% , 60% and 80% of images to get class-balanced training data.
Figure 3a shows that accuracy and performance gap between vanilla and data-reusing fine-tuningwhen different amounts of training data are available.
On both data sets, the performance gap isincreased as the training data size decreases, indicating that the UOT-selection data reusing schemehelps more when the target data is insufficient.
This experiment demonstrates that the proposed datareusing paradigm is particularly effective when the target task does not have enough data, whichcould be a typical case in real-world applications.
[Table]
The long-tail class distribution is also a challenge inreal-world applications.
We simulate long-tail datasets with CUB, Aircraft and Caltech by samplingimages from classes with a Pareto distribution [42]as in [43].
The number of images in the largestclass is 10 times that in the smallest class.
Theresults of fine-tuning a ResNet18 are shown in the
Table 3.
The proposed method has a more significant improvement when the target data is imbalancedcompared with the original balanced class data sets.
The reason is that the selected pre-trainingdata is controlled to have a balanced label distribution and the gradients of pre-training data have aregularization effect, so the model does not easily overfit images in minor classes.
4.4 Ablation Study
Number of selected pre-training classes.
We investigate the effect of selected class numberson the target classification accuracy.
Figure 3 b shows the performance of target tasks (CUB and
Caltech) when the number of selected classes ranges from 50 to 300 in UOT selection.
The increasedpre-training data added in fine-tuning do not improve the performance of CUB, since there are 59classes of birds in the ImageNet and more reused images enlarge the gap δ 2 .
Surprisingly, we observethat only using the bird images (blue line) is not the best strategy on CUB.
It is because that there canbe a certain number of related classes in ImageNet, which will help the classification of bird images.
The result shows that even when labels of pre-training and target data are given and overlapped,
UOT selection can achieve better performance by including extra relevant classes from pre-trainingdata.
On the general classification data set (Caltech), more reused pre-training data help gain theperformance improvement because the diverse data set needs a large number of images to have asmall domain gap.
On both data sets, the UOT selection performs better than the random selection asthe number of selected classes changes.
Cluster number in K-means.
The cluster assignments are crucial to the performance of similaritybased selection, so we change the cluster number in the experiment and show the fine-tuning resultsof 8 data sets in Table 2.
On most data sets, K=2000 achieves better generalization than K=1000,indicating that fine-grained clustering helps the following data selection step. However, if we further A cc u r ac y
Similarity Similarity UO T S c o r e ! ! R eca ll R a t e increase K, the clustering result will be quite noisy and small clusters befound,
Method Metric [Table]
[Table]
Distance function and ϵ c .
To investigate the influence of different factors in the UOT selection, wedefine a recall rate as a metric to make the comparison.
For a target data set whose classes happen toexist in ImageNet, the similarity-based data selection is expected to choose those matched classes,e.g., selecting all 59 birds classes from ImageNetwhen fine-tuning on CUB.
Thus, the recall rate on
CUB is defined as the ratio between the number ofbird classes in the top-100 similar vector or EMDsimilarity and 59.
With the performance metric, we first compare UOT with Greedy-OT using l 2 andcosine distance in Table 4.
With a supervised pre-trained model, Greedy-OT is only slightly worsethan UOT, while with a self-supervised model the weakness of Greedy-OT is amplified.
It meansthat Greedy-OT heavily relies on the label information in supervised training but UOT only needsgeneric visual features to have a good similarity measure.
In addition, the cosine distance is betterthan the l 2 distance, especially in the self-supervised model.
The importance of cosine distance isdue to the cosine similarity loss used in MoCo training.
Finally, Fig.
4a shows the recall rate whenusing different ϵ c .
The supervised model is not sensitive to the choice of ϵ c but a small ϵ c is crucialto the good performance of OT-selection in the self-supervised model.
Note that the recall rate of
Greedy-OT does not depend on ϵ c so the performance is worse than UOT no matter what ϵ c is used.
The impact of the domain gap.
We split ImageNet into 10 subsets based on UOT score (highermeans higher similarity) to CUB data and reuse the 10 subsets in fine-tuning to see the results,which are presented in Fig.
4b1-2.
With both supervised and self-supervised model, when the UOTsimilarity score rapidly decreases, the performance of UOT-fine-tuning drops and is lower than thefine-tuning baseline.
This experiment indicates that the dissimilar data hurt the performance whenadded to fine-tuning and highlights the importance of similarity-based data selection.
Subset of Pre-Training Data.
In the case where the pre-training data is larger than the availablestorage capacity, a subset of the pre-training data could be used.
We subsample 50% images fromeach class of ImageNet and use the 50% ImageNet data in the supervised ResNet18 fine-tuning.
On CUB and Caltech, the UOT fine-tuning achieves 77.03% and 91.00% accuracy respectively, whichonly drop 0.18% and 0.11% compared with using the full ImageNet. The experiment indicates thateven with half of ImageNet, fine-tuning with UOT data selection is still quite effective.  Conclusion
This paper provides the generalization analysis of pre-trained models when fine-tuning on targettasks by using excess risk bound, which suggests that the pre-trained model can have little positiveinfluence on learning from target data under certain conditions.
It also shows that the performance onthe target data can be improved when similar data is selected from the pre-training data for fine-tuning.
Inspired by this result, a novel similarity-based selection algorithm is developed, which is evaluatedon 8 data sets and shown to be effective.
Our future work will further explore the data reusing strategyin other computer vision tasks.
One limitation of the current work is that sometimes the pre-trainingdata is private (e.g., JFT-3B [44]) and not accessible. Given the power of pre-training data as unveiledby our work, we advocate that more pre-training data should be open for the better use of pre-trainedmodels.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and
scope?
[Yes]
(b) Did you describe the limitations of your work
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We study
the general property of fine-tuning in computer vision.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See the supplemental.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] See the
supplemental.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See the supplemental.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[No] We only report the best result among 3 trials.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See the supplemental.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] See the creators’ papers or reports.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] See the creators’ papers or reports.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] We use public datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]