Transferability Between Regression Tasks
Abstract
Transfer learning has been a widely used technique to adapt a deep learning model trained for one task to another when there is a data distribution shift between these tasks.
To improve the effectiveness of transfer learning and to understand relationships between tasks, we consider the problem of transferability estimation between regression tasks and propose two novel transferability estimators that are simple, computationally efficient, yet effective and theoretically grounded.
We test our proposed methods extensively in various challenging, practical scenarios and show they significantly outperform existing state-of-the-art regression task transferability estimators in both accuracy and efficiency
Introduction
Transfer learning [15, 21] has been a popular technique to adapt a deep learning model trained for one task to another when there is a data distribution shift between these tasks.
To improve transfer learning and to understand task relationships, transferability estimation [3, 18, 24] has recently been studied.
The aim of transferability estimation is to derive a computationally efficient metric (a score) that can predict the effectiveness of transferring a deep learning model from source to target tasks.
This problem has recently gained attention as a means for model and task selection [3, 6, 18, 24, 29] that can potentially improve the performance and reduce the cost of transfer learning.
Transferability estimators were also developed and used for checkpoint ranking [12, 14] and few-shot learning [23].
Nearly all existing methods consider only the transferability of classification tasks [3, 8, 13, 14, 17, 18, 22, 24] with very few designed for regression tasks [13, 29].
Furthermore, those few methods proposed for estimating regression model transferability have so far been either complex and inefficient [29] or ineffective [13], as we will show in our experiments.
Compared to these previous work, we offer efficient and theoretically grounded transferability estimation for regression problems.
Specifically, we propose two families of simple transferability estimators that utilize the penalized mean squared error (MSE) of a linear regression model computed from the source and target training sets.
The first family, Linear MSE , uses features extracted from a model trained on the source task (the source model ), while the second family, Label MSE , uses the dummy labels obtained from the source model.
In special cases where the source and target data share the inputs, the Label MSE estimators can be computed efficiently from true labels without a source model.
Besides simplicity, our methods also have theoretical properties relating to the expected risk of the target model.
We conduct experiments on two real-world keypoint detection datasets, CUB-200-2011 [26] and OpenMonkey [28], to show the advantages of our methods.
The results clearly demonstrate that the proposed methods outperform recently published, state-of-the-art (SotA) transferability estimators for regression problems, such as LogME [29] and TransRate [13], in both effectiveness and efficiency.
• Other Related Work.
Besides transferability estimation, our paper is also related to previous work that developed theoretical bounds for transfer learning [2, 4, 5, 16, 25, 27]. These bounds are usually hard to compute
in practice and thus unsuitable for transferability estimation.
Some previous
transferability estimators
have
theoretical bounds on
the
empirical loss [ 24], but they were shown for classification and did not bound the expected
loss Our work, on the other hand focuses on regression and proves generalization bounds for the expected loss.
Transferability Between Regression Tasks
• Transfer learning settings.
Consider a source training set D s
= { ( x si , y si ) } n s i
=1 and a target training set D t = { ( x ti , y ti ) } n t
i
=1 consisting of n s and n t examples respectively, where x si , x ti ∈ R d are d -dimensional input vectors, y si ∈ R
d s is a d s -dimensional source label vector, and y ti ∈ R d t is a d t -dimensional target label vector.
Note that we allow multi-output regression tasks where d s , d t > 1 .
From the source dataset D s , we train a deep learning model ( w ∗ , h ∗ ) consisting of a feature extractor w ∗ and a regression head h ∗ that minimizes the empirical MSE loss.
That is, w ∗ , h ∗ = argmin w,h L ( w, h ; D s ) , where w : R d → R d r is a feature extractor network, h : R d r → R d s is a source regression head network, and L ( w, h ; D s )
= 1 n s (cid:80) n s i =1 ∥
y si − h ( w ( x si ))
∥ 2 is the empirical MSE loss of the whole model ( w, h ) on D s , with ∥ · ∥ being the ℓ 2 norm.
In practice, we usually consider the source model as a whole (e.g., a ResNet model [11]) and use its first l layers from the input (for some chosen number l ) as the feature extractor w .
The regression head h is the remaining part of the model from the l -th layer to the output layer, and the prediction for any input x is h ( w ( x )) .
After training the optimal source model ( w ∗ , h ∗ ) we perform transfer learning to the target task by freezing the optimal feature extractor w ∗ and re- training a new regression head k ∗ using the target dataset D t , also by minimizing the empirical MSE loss: k ∗ = argmin k L ( w ∗ , k ; D t ) =
argmin k (cid:8) 1 n t (cid:80) n t i =1 ∥ y ti − k
( w ∗ ( x ti ))
∥ 2 (cid:9) , where k : R d r → R d t is a target regression head network that may have a different architecture than that of h .
In general the regression heads h and k may contain multiple layers and are not necessarily linear.
This transfer learning setting is usually called head re-training [9, 19, 20] and will be used in our theoretical analysis.
In our experiments, we also consider another transfer learning setting called fine-tuning , where we fine-tune the trained feature extractor w ∗ on the target set in addition to re-training k ∗ [1, 7, 10].
• Transferability Estimators.
A transferability estimator measures the effectiveness of transfer learning given a pair of source and target tasks.
Intuitively we can use the MSE of the transferred target model L ( w ∗ , k ∗ ; D t ) as an estimator.
However, that means we need to run the actual transfer learning algorithm, which could be expensive if the network architecture of the target regression heads (e.g., k and k ∗ ) is complex.
A simple way to reduce this computational cost is to approximate L ( w ∗ , k ∗ ; D t ) using an ℓ 2 regularized linear regression (or Ridge regression) head; that is, we consider the family of Linear MSE estimators below.
In our definitions, ∥ · ∥ F is the Frobenius norm.
Definition 1.
A Linear MSE transferability estimator between a source dataset D s and a target dataset D t , with a regularization parameter λ ≥ 0 , is defined as: T lin λ
( D s , D t ) :
= min A,B (cid:8) 1 n t (cid:80) n t i =1 ∥
y ti −
Aw ∗ ( x ti ) − B ∥ 2 + λ ∥
A ∥ 2 F (cid:9) , where A ∈ R d r × d t and B ∈ R d t .
We use a regularizer in the estimator to avoid overfitting when D t is small.
Previous work such as
LogME [29] prevents overfitting by taking a more complicated and expensive Bayesian approach.
Here we argue that a simple regularization can tackle this issue effectively and more efficiently.
Given a pre-trained feature extractor w ∗ , we can compute T lin λ
( D s , D t ) efficiently by solving a Ridge regression problem.
If the target head is restricted to only linear regression model, the non-regularized
Linear MSE T lin0 ( D s , D t ) is equal to the MSE of the transferred target model ( w ∗ , k ∗ ) on D t .
If the target head has more than one layer with a non-linear activation function, T lin λ
( D s , D t ) can be regarded as using a regularized linear model to approximate this non-linear head.
Although Linear MSE scores can be computed efficiently, they may still be expensive if the feature vectors w ∗ ( x ti ) are high-dimensional.
Furthermore, in many cases, we need to compute and compare the scores for several pairs of datasets, resulting in high computational costs.
To further reduce the costs, we propose the Label MSE transferability estimators below, which replaces w ∗ ( x t i ) by the “dummy” source label z
i = h ∗ ( w ∗ ( x t i )) .
Using dummy labels from the trained source model ( w ∗ , h ∗ ) is a technique previously used in the LEEP transferability scores for classification tasks [18].
Definition 2.
A Label MSE transferability estimator between a source dataset D s and a target dataset
D t , with a regularization parameter λ ≥ 0 , is defined as: T lab λ
( D s , D t ) :
= min A,B (cid:8) 1 n t (cid:80) n t i =1 ∥
y ti −
Az i − B ∥ 2 + λ ∥
A ∥ 2 F (cid:9) , where A ∈ R d s × d t , B ∈ R d t , and z i = h ∗ ( w ∗ ( x ti )) , ∀ i .
[Table]
From our definitions, the lower the Linear MSE or Label MSE, the better the transferability.
Since the size of z i is usually much smaller than that of w ∗ ( x ti ) (i.e., d s ≪ d r ), computing the Label MSE is usually cheaper than computing the Linear MSE.
We prove novel properties of Label MSE below.
• Theoretical Results.
We prove some theoretical properties (in the form of generalization bounds) for the Label MSE with ReLU feed-forward neural networks.
We first make the standard assumption that the source data and the target data are drawn iid from the unknown distributions P ( X s , Y s ) and P ( X t , Y t ) respectively.
Given any model ( w, k ) for the target task, the expected risk of ( w, k ) is defined as R ( w, k ) =
E ( x t ,y t ) ∼ P ( X t ,Y t ) (cid:8) ∥
y t − k ( w ( x t ))
∥ 2 (cid:9) .
Our main theoretical result is Theorem 1 below.
In this theorem, L is the number of layers of the ReLU feed-forward neural network ( w, k ) , and we assume the number of hidden nodes and parameters in each layer are upper bounded by H and M ≥ 1 respectively.
Theorem 1.
For any λ ≥ 0 and δ > 0 , with probability at least 1 − δ , we have: R ( w ∗ , k ∗ ) ≤
T lab λ
( D s , D t )
+ C ( d, d t , M, H, L, δ ) / √ n t .
This theorem shows that the expected risk is upper bounded by the Label MSE plus a complexity term C ( d, d t , M, H, L, δ ) / √ n t that depends on the architecture of the target network and the target dataset (i.e., the input/output dimensions and the dataset size).
When the complexity term is small (e.g., when n t is large), the Label MSE will be a tighter bound for the expected risk.
• Setting with Shared Inputs.
When the source and target data have the same inputs, i e., x si = x ti = x i and n s = n t = n , we can compute the Label MSE directly from the labels without training the source model ( w ∗ , h ∗ ) or computing the dummy labels.
Formally, we can use the following new definition for the Label MSE:
[Equation]
where we train a Ridge regression model directly from the corresponding label pairs ( y si , y ti ) .
This new transferability estimator is less expensive since we do not need to train the source model.
We can also prove similar generalization bounds with (cid:98) T lab λ
( D s , D t ) .
Experiments
We evaluate our transferability estimators on the keypoint regression tasks using two large-scale public datasets: (1) CUB-200-2011 [26]: This dataset contains 11,788 bird images with 15 labeled keypoints indicating 15 different parts of a bird body, and (2) OpenMonkey [28]: This dataset contains over 100,000 monkey images in natural contexts, annotated with 17 body keypoints.
We use ResNet34 [11] as the model backbone due to its good performance.
Following [18, 24], we investigate how well our transferability estimators correlate
(using Pearson correlation) with the test MSE of the target model obtained from actual transfer learning.
We consider three transfer learning algorithms: (1) head re-training : We fix all layers of the source model up until the penultimate layer and re-train the last fully-connected (FC) layer; (2) half fine-tuning : We fine-tune the last convolutional block and all the FC layers of the source model (around half of the parameters in the network), while keeping all other layers fixed; and (3) full fine-tuning : We fine-tune the whole source model using the target training set.
For our methods, we consider λ =[-
-]
LinMSE0 and LabMSE0 ) and λ = LinMSE1 and
LabMSE
We compare with two recent SotA baselines for regression: LogME and Tran- sRate [13]
Besides their usual versions, we also consider the label-based versions where these estimators are computed from dummy labels and target labels ( LabLogME and
LabTransRate As in [13], we divide the target label values into 5 equal-sized bins to compute TransRate and LabTransRate.
• General Transfer Between Two Different Domains.
We first consider the general case where source models are trained on OpenMonkey and then transferred to CUB-200-2011.
Specifically, we
[Table]
train a source model for each of the 17 OpenMonkey keypoints and transfer them to each of the[-
-]
CUB-200-2011 keypoints, resulting in a total of 255 final models.
Since each keypoint consists of x and y positions, all source and target tasks in this experiment have two dimensional labels.
The actual MSEs of these models are computed on the respective test sets and then used to calculate the
Pearson correlation coefficients with the transferability estimators.
In this experiment,
the label-based estimators are computed from the dummy labels and
the
actual target labels
Table shows the results for this experiment.
1 From the results, TransRate and LabTransRate perform poorly in this regression setting, while our methods are equal or better than LogME and LabLogME in all cases, especially when using λ = 1 (LinMSE1) or dummy labels LabMSE0 and LabMSE1).
We observe that adding a regularizer in LabMSE1 does not significantly change its correlation coefficient compared to LabMSE0.
This is because we scaled the labels to [0 , 1] as a pre-processing step, so a regularizer with λ = 1 has negligible effect on the scores.
It is also surprising that LabMSE
and LabMSE1 are better than LinMSE0 and LinMSE1.
One possible explanation for this phenomenon is that the dummy labels (i.e., body parts of monkeys) give more information about the target labels (i.e., body parts of birds) than the extracted features.
• Transfer Between Tasks with Shared Inputs.
In this experiment, we consider the setting where the source and target tasks have the same inputs.
Since each image in our datasets contains multiple keypoints, we can use any two different keypoints on the same dataset as source and target tasks.
In total, we can construct 210 source-target pairs for CUB-200-2011 and 272 pairs for OpenMonkey.
The labels for all source and target tasks are also 2-dimensional real values.
We repeat the previous experiment with these source-target pairs for each dataset.
The main difference in this experiment is that we use the true source labels (instead of dummy labels) when computing LabLogME, LabTransRate, LabMSE0, and LabMSE1.
Thus, these estimators can be computed without any source models in this setting, and hence incurring very low computational costs.
Results for this experiment are in Table 2.
Here both versions of TransRate perform poorly on CUB-200-2011, while TransRate is slightly better than LogME on OpenMonkey.
In the firstsettings of Table 2, LabMSE0 and LabMSE1 both outperform LabLogME and LabTransRate, while LinMSE0 and LinMSE1 both outperform LogME and TransRate.
In these settings, the LinMSE methods achieve the best correlations overall.
In the last setting (full fine-tuning on CUB-200-2011), all methods perform poorly.
• Efficiency of Our Methods.
In the last row of Table 2, we compare the average running time of different methods in the shared inputs setting on CUB-200-2011.
We can observe that LabMSE0, LabMSE1, LinMSE0, and LinMSE1 are all faster than the corresponding label-based or feature-based baselines.
Overall, LabMSE1 and LinMSE1 achieve the best running time among the label-based and feature-based methods respectively.
Conclusion
We proposed Linear MSE and Label MSE, two families of simple but effective transferability estimation methods for regression, and proved novel theoretical bounds for the expected risk of the target models using these estimators.
[-Thus, applying the Contraction Lemma (Lemma 26.9 in [25]), weobtain (cid:98) R D t ( F j ) ≤ 4 dM L +1 H L (cid:98) R D xt (Φ j ) ≤ 8 dd t M 2 L +
-]
[-12 × 10 − 3 (LogME), 6 . 
-]
Our experiments showed that the proposed methods are superior to recent, relevant SotA methods in terms of both effectiveness and efficiency.