Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance
Abstract
Fine-tuning of large pre-trained image and language models on small customized datasets has become increasingly popular for improved prediction and efﬁcient use of limited resources.
Fine-tuning requires identiﬁcation of best models to transfer-learn from and quantifying transferability prevents expensive re-training on all of the candidate models/tasks pairs.
In this paper, we show that the statistical problems with covariance estimation drive the poor performance of H-score [1] — a common baseline for newer metrics — and propose shrinkage-based estimator.
This results in up to 80% absolute gain in H-score correlation performance, making it competitive with the state-of-the-art LogME measure by [26].
Our shrinkagebased H-score is 3 − 55 times faster than LogME.
Additionally, we look into a less common setting of target (as opposed to source) task selection.
We highlight previously overlooked problems in such settings with different number of labels, class-imbalance ratios etc.
for some recent metrics e.g., NCE [24], LEEP [18] that misrepresented them as leading measures.
We propose a correction and recommend measuring correlation performance against relative accuracy in such settings.
We support our ﬁndings with ∼ 65,000 (ﬁne-tuning trials) experiments.
Introduction
Transfer learning (TL) is a set of techniques of using abundant somewhat related source data p ( X ( s ) , Y ( s ) ) to ensure that a model can generalize well to the target domain, deﬁned as either little amount of labelled data p ( X, Y ) (supervised), and/or a lot of unlabelled data p ( X ) (unsupervised
TL).
TL is most commonly achieved via ﬁne-tuning.
Fine-tuning (FT) is a process of adapting a model trained on source data by using target data to do several optimization steps (for example, SGD) that update the model parameters.
FT is becoming increasing popular because large models like
ImageNet [12], Bert [5] etc. are released by companies and are easily modiﬁable.
Training such large models from scratch is often prohibitively expensive for the end user.
In this paper, we are primarily interested in effectively measuring transferability before training of the ﬁnal model begins.
Given a source data/model, a transferability measure (TM) quantiﬁes how much knowledge of source domain/model is transferable to the target model.
If a measure is capable of efﬁciently and accurately measuring transferability across arbitrary tasks, the problem of transfer learning is greatly simpliﬁed by using the measure to search over candidate sources and targets.
Related Work Negative Conditional Entropy (NCE) [24] uses pre-trained source model and evaluates conditional entropy between target pseudo labels (source models’ assigned labels) and real target labels.
Log Expected Empirical Predictor (LEEP)
[18] modiﬁes NCE by using soft predictions from the source model.
[3] propose representing each output class by the mean of all images from that class and computing Earth Mover’s distance between the centroids of the source classes and target classes. [15] proposed N LEEP that ﬁts a Gaussian mixture model on the target feature embeddings and computes the LEEP score between the probabalistic assignment of target features to different clusters and the target labels.
[10] proposed TransRate — a computationally-friendly surrogate of mutual information (using coding rate) between the target feature embeddings and the target labels.
[1] introduced H-score that takes into account inter-class feature variance and feature redundancy.
[26] proposed LogME that considers an optimization problem rooted in Bayesian statistics to maximize the marginal likelihood.
[4] introduced LFC to measure in-class similarity of target feature embeddings.
Transferability setup We consider two FT scenarios based on existing literature.
(i) Source Model
Selection (SMS) :
For a particular target data/task, this regime aims to select the “optimal" source model (or data) to transfer-learn from, from a collection of candidate models/data.
(ii) Target Task
Selection (TTS) :
For a particular (source) model, this regime aims to ﬁnd the most related target data/task.
In addition, we consider two FT strategies: (a) Linear FT : All layers except for the penultimate layer are frozen.
Only the weights of the head classiﬁer are re-trained while ﬁne-tuning.
(b) Nonlinear FT : Any arbitrary layer can be designated as a feature extractor, up to which all the layers are frozen; all subsequent (nonlinear) layers along with head are re-trained.
Contributions Our contributions are three-fold: (i)
We show that H-score, a baseline for newer measures, suffers from instability due to poor covariance estimation.
We propose shrinkage-based estimation of H-score with regularized covariance estimation techniques.
We show 80% absolute increase over the original H-score and show superior performance in 9/15 cases against all newer
TMs across various FT scenarios.
(ii) We present a fast implementation of our estimator that is 3 −times faster than state-of-the-art LogME measure.
Unlike LogME, our estimator is tractable even for really high-dimensional feature embeddings ∼ 10 5 .
(iii)
We identify problems with 3 other measures (NCE, LEEP and N LEEP) in target task selection when either the number of classes or the class imbalance varies across candidate target tasks.
We propose measuring correlation against relative target accuracy (instead of vanilla accuracy) in such scenarios.
65,000 FT experiments with ImageNet models and different regimes constructed from CIFAR-100/CIFAR-10 validates our proposals.
This paper is organized as follows.
Section 2 addresses shortcomings of the pioneer TM (H-Score) that arise due to limited target data.
Section 3 identiﬁes problems with recent NCE, LEEP and
N LEEP metrics and proposes corrections.
Finally, Section 4 presents a meta study of all metrics.
Improved estimation of H-score for limited target data
H-score [1] is the pioneer measure, often used as a baseline for newer measures that often demonstrate the improved performance.
It characterizes discriminatory strength of feature embedding for classiﬁcation: H( f ) = tr ( Σ f − 1 Σ z ) , where d is the embedding dimension, f i = h ( x i
) ∈ R
d is the target feature embeddings when the feature extractor h ( . ) from the source model is applied to the target sample x
i , F ∈ R n t × d denotes the target feature matrix, Y ∈ { 1 , · · · , C } are the target labels, Σ f ∈ R d × d denotes the sample feature covariance matrix of f , z = E [ f | Y ] ∈ R d and Z ∈ R n t × d denotes the target conditioned feature matrix, Σ z ∈ R d × d denotes the sample covariance of z .
We hypothesize that the sub-optimal performance of H-Score (compared to that of more recent metrics) for measuring transferability in many of the evaluation cases, e.g., in [18], is due to lack of robust estimation of H-Score — see Fig.
1 for a synthetic example showing the non-reliability of empirical H-score over various sample sizes when compared with its population version.
Given that many of the deep learning models in the context of TL have high-dimensional feature embedding space — typically larger than the number of target samples — the estimation of the two covariance matrices in H-score becomes challenging: the sample covariance matrix of the feature embedding has a large condition number 1 in small data regimes.
In many cases, it cannot even be inverted.
[1] used a pseudo-inverse of the covariance matrix Σ f .
However, this method of estimating a precision matrix can be sub-optimal as inversion can amplify estimation error [13].
We propose to use well-conditioned shrinkage estimators from rich literature in statistics on the estimation of high-dimensional covariance matrices [20].
Such shrinkage estimators can offer signiﬁcant gain in the performance of H-score in predicting transferability, making it a leading TM.
Proposed Transferability Measure We propose the following shrinkage based H-score:
[Equation]
Estimating Σ fα : While there are several possibilities to obtain a regularized covariance matrix [20], we present an approach that considers a linear operation on the eigenvalues of the sample version of the feature embedding covariance matrix.
Similar ideas of using well-conditioned plug-in covariance matrices are used in the context of discriminant analysis [8].
In particular, we improve the conditioning of the covariance matrix by considering its weighted convex combination with a scalar multiple of the identity matrix: Σ fα = (1 − α ) Σ f + ασ I d , where α ∈ [0 , 1] is the shrinkage parameter and σ is the average variance computed as tr ( Σ f ) /d .
Note that the inverse of Σ fα can be computed for every α , by using the eigen-decomposition of Σ f .
The shrinkage parameter controls the bias and variance trade-off; the optimal α needs to be selected.
This distribution-free estimator is well-suited for our application as the explicit convex linear combination is easy to compute and makes the covariance estimates well-conditioned and more accurate [13, 2, 21]. [Figure]
shrinkage based H to samples is ∼ 75 times larger with a size of 10 6 ) contrast
Score is more reliable Understanding (1 − α ) Σ z : The scaling factor (1 − α ) can be understood in terms of ridge-regularized covariance estimation:
[-|| 
 + λ || ˆ Σ-][Equation]
where λ ≥ 0 is the ridge penalty.
Choosing λ
= α/ (1 − α ) , it becomes clear that (1 − α )
Σ z is the regularized covariance.
Choice of α [13] proposed a covariance matrix estimator that minimizes mean squared error loss between the shrinkage based covariance estimator and the true covariance matrix.
The optimization with respect to α considers the following objective:
[Equation]
{+where (cid:107) A (cid:107) 
+}
= tr ( AA T ) /d . This optimization problem permits a closed-form solution for the optimal shrinkage parameter, which is given by: [Equation]
where (5) deﬁnes a valid estimator (not dependent on true covariance matrix) for practical use.
For proof, we refer the readers to Section 2.1 and 3.3 in [13].
Additional Discussion
The covariance Σ z can not be shrunk independently of Σ f in the estimation of H α ( f ) — the two covariances are coupled by the law of total covariance:
[Equation]
where f Y denotes the feature embedding of target samples that belong to class Y ∈ Y and Σ f Y =
Cov( f | Y ) denotes the class-conditioned covariances.
We can write
[Equation]
Comparing (7) with (6), we see that the same shrinkage parameter α should be used when using shrinkage estimators, to preserve law of total covariance.
The ﬁrst two terms on the right side in (7) can be understood as shrinkage of class-conditioned covariances to the average (global) variance.
The third term in (7) (e.g. (1 − α ) Σ z ) can then be understood as ridge shrinkage as in (2).
We ﬁrst provide validation of shrinkage-based estimation of H-Score on synthetic classiﬁcation data.
We generated 1 million 1000-dimensional features with 10 classes using Sklearn multi-class dataset generation function [19].
Number of informative features is set to 500 with rest ﬁlled with random noise.
We visualize the original and the population version of the H-score and the shrinkage-based
H-Score for different sample sizes in Fig.
We observe that the original H-Score becomes highly unreliable as the number of samples decreases.
In contrast, the shrunken estimation of H-Score is highly stable and has a small error when compared with the population H-Score.
We provide an efﬁcient implementation for small target data ( C ≤ n t < d ) in Supplement Section S2.1 that leads to 3 −
55 times faster computation than the state-of-the-art LogME measure (see Table S2 in Supplement Section S2.1.2).
A closer look at NCE, LEEP and N LEEP measures
[Figure]
We pursue a deeper investigation of some of the newer metrics (reported to be superior to
H-Score) and bring to light what appear to be some overlooked issues with these metrics in Target Task Selection (TTS) scenario.
TTS has received less attention than Source Model
Selection (SMS).
We are the ﬁrst to highlight these problematic aspects with NCE, LEEP and N LEEP, which can potentially lead to the misuse of these metrics in measuring transferability.
These measures are sensitive to the number of target classes ( C ) and tend to be smaller when C is larger (see Fig. 2[Left]).
Therefore, use of these measures for target tasks with different C will most likely select the task with a smaller C .
However, in practice, transferring to a task with a smaller C is not always easier; for example, reframing a multiclass classiﬁcation into a set of binary tasks can create more difﬁcult to learn boundaries [6].
Furthermore, the measures are also problematic if two candidate target tasks have different imbalances in their classes even if C is the same.
The measures would predict higher transferability for imbalanced data regimes over balanced settings (see Fig.
2[Right]).
However, imbalanced datasets are typically harder to learn.
If these measures are correlated against vanilla accuracy, which tends to be higher as the imbalance increases e.g. for binary classiﬁcation, the measures would falsely suggest they are good indicators of performance.
Earlier work erroneously showed good correlation of these metrics against vanilla accuracy to show dominance of such metrics in TTS with different C [18, 23] and imbalance [23].
Here, we propose a method to ameliorate the shortcomings of NCE, LEEP and N LEEP to prevent misuse of these measures, so that they lead to more reliable conclusions.
We propose to standardize the metrics by the entropy of the target label priors, leading to the following deﬁnitions:
[Equation]
We provide additional motivation (and proof) for these deﬁnitions in Supplement Section S3.
For scenarios where candidate target tasks have different C , we propose an alternative evaluation criteria ( relative accuracy) instead of vanilla accuracy — see Section 4 for more details.
We provide empirical validation of the proposed normalization to these measures in Table 2 in Section 4.
We also show that our proposed H α ( f ) is the leading metric even in these scenarios.
Empirical Evaluation & Conclusion
We evaluate existing TMs and our proposed modiﬁcations in various FT regimes and data settings.
We are inspired by [18] who consider TTS and SMS.
The regimes highlight important aspects of
TMs, e.g., dataset size, number of target classes, and feature dimension etc.
Some of these aspects have been overlooked when evaluating TMs, leading to improper conclusions.
Evaluation criteria TMs are often evaluated by how well they correlate with the test accuracy after
FT the model on target data.
Following [24, 18, 10], we used Pearson correlation.
We include additional results with rank correlations (e.g., Spearman) in Supplement Section S7.
We argue that considering correlation with vanilla accuracy is ﬂawed in some scenarios.
In particular, for TTS, it is wrong to compare target tasks based on accuracy when C is different e.g 5 vs 10 classes.
In such a case, task with 10 classes will more likely have lower test accuracy compared to that for task with 5 classes.
It is more appropriate to consider the gain in accuracy achieved by the model over it’s random baseline.
Hence we propose relative accuracy (for balanced classes): Accuracy − 1 /C 1 /C .
This measure is more effective in capturing the performance gain achieved by the same model in transferring to two domains with different C .
This also highlights the limitation of NCE, LEEP and
N LEEP which are sensitive to C and tend to have smaller values with higher C ; these measures do not provide useful information about how hard these different tasks when evaluated on the original accuracy scale.
Correlations marked with asterisks (*) in Tables 1, 2, 3 are not statistically signiﬁcant ( p -value > 0 . 05 ).
Hyphen (-) indicates the computation ran out of memory or was really slow.
Target Task Selection We consider Small balanced (S-B) target data, small imbalanced (S-IB) and
Large Balance (L-B).
See additional details in Supplement Section S6.
We compare our proposed
H α ( f ) against the original H-Score by [1].
Table 1 demonstrates 80% absolute gains in correlation performance of H α ( f ) over H( f ) , making it a leading metric in many cases in small target data regimes.
Table 2 shows how various supervised TMs perform in TTS when the number of target classes varies .
H α ( f ) dominates the performance in both cases, surpassing all supervised TMs.
[Table] [Table]
Source Model Selection We select 9 small to large ImageNet models.
We evaluate SMS for FT under small sample setting.
We sample 50 images per class from all classes available in the original train split of CIFAR-100/CIFAR-10.
We designate 10 samples per class for hyperparameter tuning.
[Table]
For SMS, the feature dimensions vary signiﬁcantly across source models (see Table S3 in Supplement
Section S5).
In such scenarios, dimensionality reduction is useful for H-score for more meaningful comparison across different models.
We apply random projection (RP) on target feature embeddings to project to 128 -dimensional space.
This provides the gains of proposed H α ( f ) in SMS as well for small samples as given in Table 3, making it again a leading metric in SMS.
Conclusion To summarize, regularized covariance estimation in H-Score (pioneer TM) makes it a leading metric (9/15 cases) against newer measures across various FT scenarios and data settings.
With fast implementation, our estimator can be 3 − 55 times quicker than state-of-the-art LogME measure.
Standardization of NCE, LEEP and N LEEP and correlation against relative accuracy in target selection can provide for more conclusive evaluation about performance of such measures.
Acknowledgments
[-where (cid:107) A (cid:107) 
-]
The research was conducted while Shibal Ibrahim was at Google, and part of the writing was done during his time at MIT.
At MIT, Shibal Ibrahim and Rahul Mazumder acknowledge research funding from the Ofﬁce of Naval Research [Grant ONR-N000141812298]
S Acronyms
[Table]
S Supervised Transferability metrics S2.1 Efﬁcient Computation for small target data
For small target data ( C ≤ n t < d ), the naive implementation of H α ( f ) can be very slow.
We propose an optimized implementation for our shrinkage-based H-Score that exploits diagonal plus low-rank structure of Σ fα for efﬁcient matrix inversion and the low-rank structure of Σ z for faster matrix-matrix multiplications.
We assume F (and correspondingly Z ) are centered.
The optimized computation of H α ( f ) is given by:
[Equation]
(S1) where R = (cid:2) √ n 1 ¯ f Y =1 , · · · , √ n C ¯ f Y = C (cid:3) ∈ R d × C , G = F R ∈ R n t × C , W = n t ασ I n + F F T ∈
R n t × n t .
The derivation is provided in the Supplement Section S2.1.1.
We make a timing comparison of our optimized implementation of H α ( f ) against the computational times of the state-of-the-art
LogME measure and demonstrate 3 − 55 times faster computation (see Table S2 in Supplement
Section S2.1.2).
S2.1.1 Derivation of optimized implementation for H α ( f )
We derive an optimized computation for our proposed shrinkage-based H-score H α ( f ) for small target data ( C ≤ n t < d ) as follows:
[Equation]
where R = (cid:2) √ n 1 ¯ f Y =1 , · · · , √ n C ¯ f Y = C (cid:3) ∈ R d × C , G = F R ∈ R n t × C , W = n t ασ I n + F F T ∈
R n t × n t , (S2) follows by Woodbury matrix identity ( ( I + UV ) −
=
I − U
( I + V U ) − 1 V ) [17] and (S3) and (S4) follow by trace properties.
S2.1.2 Timing comparison between LogME and H α ( f )
We empirically investigate the computational times of H α ( f ) when computed via our optimized implementation in (S1).
For this exercise, we generate synthetic multi-class classiﬁcation data using
Sklearn [19] multi-class dataset generation function that is adapted from [7].
We investigate different values for number of samples ( n t ), feature dimension ( d ) and number of classes ( C ).
For data generation, we set number of informative features to be 100 with the rest of the features ﬁlled with random noise.
Table S2 demonstrates a signiﬁcant computational advantage of H α ( f ) over LogME.
We observe 3 − 55 times faster computational times.
LogME seems intractable both with respect to memory and time for d ∼ 10 5 as exposed by the nonlinear settings in Table 1 and 3.
[Table]
S Normalization of NCE, LEEP and N LEEP
Our proposed deﬁnition for normalized NCE in Section 3 ensures the normalized NCE is bounded between [0 , 1] .
For proof, see Section S3.1.
n-NCE is in fact equivalent to normalized mutual information and has been extensively used to measure correlation between two different labelings/clustering of samples [25].
Given the similar behavior of LEEP and N LEEP different C and class imbalance as that for NCE (shown in Fig. 2), we suggest similar normalization.
However, this normalization does not ensure boundedness of n-LEEP score (and by extension n- N LEEP) in the range [0 , 1] as in the case of n-NCE.
S3.1 Normalization of NCE
NCE [24] evaluates conditional entropy between target pseudo labels Z ( t ) (source model’s assigned labels) and actual target labels, as given by:
[Equation]
The conditional entropy of the target labels conditioned on the dummy labels (source model’ labels on target data) is:
[Equation]
It holds that 0 ≤ H ( Y | Z ) ≤ H ( Y ) (see appendix S3.1.1).
Negative Conditional Entropy (NCE) is given by NCE = − H
( Y | Z ) and it holds that 0 ≤ − NCE ≤ H ( Y ) .
We normalize the NCE as follows:
[Equation]
where in (S7) we use the fact that entropy is greater than 0 for any practical classiﬁcation task.
S
[Equation]
where (S12) holds because H ( Y | Z = z ) ≥ 0 and holds with equality if and only if Y is a deterministic function of Z .
[Equation]
where (S16) holds by Jensen inequality.
S Fine-tuning with hyperparameter optimization
Current practices for FT typically involve a selection of values for hyperparameters when retraining the model on target data.
Given that the target datasets are typically small in the transfer learning scenarios, the typical strategy is to adopt the default hyperparameters for training large models while using smaller initial learning rate and fewer epochs for FT.
It has been believed that adhering to the original hyperparameters for FT with small learning rate prevents catastrophic forgetting of the originally learned knowledge or features.
Many studies have used ﬁxed hyperparameters (e.g. learning rate, momentum and weight decay, number of epochs) for FT.
However, the choice of hyperparameters is not necessarily optimal for FT on target tasks.
Earlier work has reported that the performance is sensitive to the default hyperparameter selection, in particular learning rate, momentum (for stochastic gradient descent), weight decay and number of epochs [16, 11, 14].
The optimal choice of these parameters is not only target data dependent but also sensitive to the domain similarity between the source and target datasets [14].
Therefore, in order to ensure the target task accuracy (against which the correlation of transferability metrics is measured) is optimal, we repeat the FT exercise for 100 trials of hyperparameter settings.
We employ Adam for FT experiments and optimize over batch size, learning rate, number of epochs and weight decay (L2 regularization on the classiﬁer head).
We select the space of these hyperparameters based on existing literature on FT, e.g. the learning rate is varied in the range [1 e − 1 , 1 e − 5] , the number of epochs between { 25 , 50 , 75 , 100 , 125 , 150 , 175 , 200 } , the batch size between { 32 , 64 , 128 } and the weight decay in the range [1 e − 6 , 1 e − 2] .
[Table]
S Feature Embedding layers for linear and nonlinear Finetuning S Target Task Selection Experiments setup
This evaluation regime is motivated by task transfer policy learning in robotics/reinforcement learning.
Under this regime, transferability measures can be used to greedily optimize a task transfer policy given a collection of tasks.
For instance, a robot has to automatically select which new object to pick up.
Given that the robot has learned to pick up a few objects before [Table] it would be beneﬁcial for the robot to optimally select the most transferable source/task object pair and improve it’s maneuvering ability throughout the process in a highly efﬁcient manner.
TMs can also shed light on the relatedness of different tasks in reinforcement learning setups for better understanding
[Table] We currently evaluate target task selection regime on visual classiﬁcation tasks with both VGG [22] and ResNet50 [9] models on subsets of CIFAR-100/CIFAR-10 data under three different dataset regimes following [18].
In all 3 cases outlined below, 20% of the samples from the randomly generated subsets is designated as validation set for hyperparameter tuning to ﬁnd the model with with optimal validation accuracy.
We use all examples in the original test set for evaluating out-ofsample accuracy performance on the target data.
Both training and validation samples in the subsets are used for computation of transferability metrics and we report the correlation of these measures against the (relative) test accuracies for the randomly generated subsets.
• Small-Balanced Target Data: We make a random selection of 5 classes from CIFAR-100/CIFAR10 and sample 50 samples per class from the original train split, out of which we designatesamples per class for validation.
We repeat this exercise 50 times (with a different selection ofclasses), ﬁne-tune the model for each selection (100 hyperparameter tuning trials per selection to ﬁnd ﬁnetuned model with optimal validation accuracy) and evaluate performance of those optimal models in terms of test accuracy.
We then evaluate rank correlations of TMs across theexperiments with random selection of 5 sub-classes.
• Small-Imbalanced Target Data: We make 50 random selections of 2 classes from CIFAR100/CIFAR-10, sample between 30 − 60 samples from the ﬁrst class and sample 5 ×
the number of samples from the second class.
This makes for a binary imbalanced classiﬁcation task.
We again measure performance of transferability measures against optimal target test accuracy.
• Large-Balanced Target Data with different number of classes: We randomly select 2-100 classes from CIFAR-100 and include all samples from the chosen classes (500 samples per class).
This constructs a range of large balanced dataset target task selection cases.
We evaluate correlation of TMs with relative target test accuracy across the variable number of target classes.
S Spearman Rank Correlation Performance of supervised Transferability Measures
We present rank correlation performance of all supervised TMs across various FT scenarios (target task selection and source model selection), FT strategies (linear and nonlinear) in various data regimes.
Table S4 combines setups in Tables 1, and 3 and presents Spearman correlation performance of
H α ( f
[Table]
With respect to Spearman correlations in the table above, our shrinkage-based H-score H α ( f ) leads in 7/15 cases and LogME leads in 8/15 cases.
In terms of Pearson correlations, H α ( f ) leads in 9/ cases (Tables 1, 2, 3) and LogME leads in 4/15 cases.
Additionally, LogME seems to be intractable with respect to memory and computational speed for nonlinear settings where feature dimension is large ( d ∼ 10 5 ).
Our efﬁcient implementation for H α ( f ) provides a 3 − 55 times computational advantage over LogME.
S Experimental code and type of resources
We use Tensorﬂow Keras for our implementation.
Imagenet checkpoints (Resnet and VGG) come from Keras https://keras.io/api/applications/. For experiments, we use 2 P100 GPUs per model, 15GB
RAM per GPU