The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift s
AfﬁliationAddress
Abstract
We study linear regression under covariate shift, where the marginal distributionover the input covariates differs in the source and the target domains, while theconditional distribution of the output given the input covariates is similar acrossthe two domains.
We investigate a transfer learning approach with pretraining onthe source data and ﬁnetuning based on the target data (both conducted by SGD)for this problem.
We establish sharp instance-dependent excess risk upper andlower bounds for this approach.
Our bounds suggest that for a large class of linearregression instances, transfer learning with O ( N 2 ) source data (and scarce or notarget data) is as effective as supervised learning with N target data.
In addition, weshow that ﬁnetuning, even with only a small amount of target data, could drasticallyreduce the amount of source data required by pretraining.
Our theory sheds light onthe effectiveness and limitation of pretraining as well as the beneﬁts of ﬁnetuningfor tackling covariate shift problems.
Introduction
In transfer learning [Pan and Yang, 2009, Sugiyama and Kawanabe, 2012], an algorithm is providedwith abundant data from a source domain and scarce or no data from a target domain and aims totrain a model that generalizes well on the target domain.
A simple yet effective approach is to pretraina model with the rich source data and then ﬁnetune the model with the available target data via, e g.,stochastic gradient descent (SGD) (see, e.g., Yosinski et al.
Despite its wide applicability inpractice, the power and limitation of the pretraining-ﬁnetuning based transfer learning framework isnot fully understood in theory.
The focus of this work is to consider this issue in a speciﬁc transferlearning setup known as covariate shift [Pan and Yang, 2009, Sugiyama and Kawanabe, 2012] wherethe source and target distributions differ in their marginal distributions over the input, but coincide intheir conditional distribution of the output given the input.
Regarding the theory of learning with covariate shift, there exists a rich set of results [Ben-Davidet al., 2010, Germain et al., 2013, Mansour et al., 2009, Mohri and Muñoz Medina, 2012, Cortes and
Mohri, 2014, Cortes et al., 2019, Kpotufe and Martinet, 2018, Hanneke and Kpotufe, 2019, Ma et al.
,2022] for the (regularized) empirical risk minimizer, which minimizes the empirical loss over thesource data and target data (if available) with potential regularization terms (e.g., ` 2 -regularization).
However, in most of these works [Ben-David et al., 2010, Germain et al., 2013, Mansour et al., 2009,
Mohri and Muñoz Medina, 2012, Cortes and Mohri, 2014, Cortes et al., 2019], the generalizationerror on the target domain is bounded by the sum of a vanishing term (e.g., the training error) and adivergence between the two domains (see, e.g., discussions in Kpotufe and Martinet [2018], with afew notable exceptions which we will discuss later).
Such bounds are very pessimistic because theadditive error contributed by the source-target divergence only captures the worst case performance gap caused by distribution mismatch [David et al., 2010] and is too crude to describe the intriguingproperties of pretraining-ﬁnetuning across different domains.
In this paper, we take a different approach to directly study the generalization performance of thepretraining-ﬁnetuning method.
In particular, we consider linear regression under covariate shift, anda SGD estimator which is ﬁrstly trained with the source data and then ﬁnetuned with the target data.
We develop a target domain risk bound that is stated as a function of (i) the spectrum of the sourceand target population data covariance matrices, (ii) the amount of source and target data, and (iii)the (initial) stepsizes for pretraining and ﬁnetuning (see Theorem 3.1 for more details).
Moreover, anearly matching lower bound is provided to justify the tightness of our upper bound.
The derivedbounds comprehensively characterize the effects of pretraining and ﬁnetuning for each covariateshift problem and each algorithm conﬁguration, based on which we make the following importantobservations:
• We compare the generalization (i
e., target domain excess risk) between pretraining (with sourcedata) vs. supervised learning (with target data).
We show that, for a large class of problems,O ( N 2 ) source data is sufﬁcient for pretraining to match the generalization of supervisedlearning with N target data.
• We next show the beneﬁts of ﬁnetuning with scarce target data.
In particular, for the problemclass considered in above, ﬁnetuning can reduce by at least constant factors the amount of sourcedata required by pretraining.
Moreover, there exist problem instances where the pretraining-
ﬁnetuning approach costs polynomially less amount of total data than pretraining (with source
data) or supervised learning (with target data).
• Finally, our bounds can also be applied to the supervised learning setting, i
e., linear regression
with last iterate SGD.
In this case, our upper bound sharpens that of Wu et al.
[2021] by a
logarithmic factor, and as a consequence we close the gap between the upper and lower boundsfor last iterate SGD when the signal-to-noise ratio is bounded.
Notation.
For two positive-value functions f ( x ) and g ( x ) we write f ( x ) .
g ( x ) or f ( x ) & g ( x )if f ( x )  cg ( x ) or f ( x )  cg ( x ) for some absolute constant c > 0 respectively, and we writef ( x ) h g ( x ) if f ( x ) . g ( x ) . f ( x ) .
For two vectors u and v in a Hilbert space, their innerproduct is denoted by h u , v i or equivalently, u > v .
For a matrix A , its spectral norm is denotedby k A k 2 .
For two matrices A and B of appropriate dimension, their inner product is deﬁned ash A , B i :=
tr( A > B ) .
For a positive semi-deﬁnite (PSD) matrix A and a vector v of appropriatedimension, we write k v k 2 A := v > Av .
For a symmetric matrix A and a PSD matrix B , we writek A k 2 B :=
k B  12 AB  12 k 22 .
The Kronecker/tensor product is denoted by ⌦ .
For a set S , we use | S | to 68 denote its cardinality.
1.1 Additional Related Work
We review some additional works that are mostly related to ours.
Learning under Covariate Shift.
Kpotufe and Martinet [2018], Pathak et al.
[2022] proposed newsimilarity measures to the source and target domains, and proved covariate shift bounds that do notcontain an additive error of the divergence between the source and target distribution.
Compared toour results, theirs can be applied to nonlinear regression/classiﬁcations as well; however in the caseof linear regression, our bounds are more ﬁne-grained and are tight upto constant factors for a broadclass of problems (see Theorem 3.2), beyond being only optimal in the worst case.
It is worth noting that Hanneke and Kpotufe [2019] studied the value of target data in addressingcovariate shift problems.
Their discussion is based on the minimax risk bounds afforded by agiven number of source and target data.
In contrast, our discussion on the beneﬁts of ﬁnetuningwith target data is based on a completely different perspective, which is by comparing the sampleinﬂation [Bahadur, 1967, 1971, Zou et al., 2021a] between pretraining-ﬁnetuning vs. pretraining vs.supervised learning, i.e., for each covariate shift problem instance, how much source (and target) dataare necessary for pretraning (and ﬁnetuning) to match the performance of supervised learning withcertain amount of target data.
More recently, Ma et al.
[2022] studied covariate shift problem in the nonparameteric kernel regressionsetting, with the assumption that the density ratio (or second moment ratio) between the target and source distribution is bounded.
Their results are similar to ours in that their bounds reﬂect the effectof the spectrum of the source population data covariance.
Since our results are dimension-free, ourbounds can also be applied in the nonparameteric kernel regression setting.
There are two notabledifferences: ﬁrstly, their estimator is (weighted) ridge regression and ours is given by SGD; moreover,our results do not rely on the bounded density ratio or bounded second moment condition.
In addition, there is a vast literature on constructing more sample-efﬁcient transfer learning algorithms,e.g., importance weighting methods [Shimodaira, 2000, Cortes et al., 2010] and learning invariantrepresentations [Arjovsky et al., 2019, Wu et al., 2019], to mention a few.
Along this line, Lei et al.[2021] proposed nearly minimax optimal estimator for linear regression under distribution shift, buttheir method relies on the knowledge of target population covariance matrix.
Developing new transferlearning algorithms is beyond the agenda in this paper.
SGD.
The pretraining and ﬁnetuning discussed in this work are both conducted by online SGD,therefore our results are closely related to the generalization analysis of online SGD for linearregression in the supervised learning context [Bach and Moulines, 2013, Dieuleveut et al., 2017, Jainet al., 2017a,b, Ge et al., 2019, Zou et al., 2021b, Varre et al., 2021, Wu et al., 2021].
From a technicalpoint of view, our theoretical results can be viewed as an extension of the SGD analysis from thesupervised learning setting to the covariate shift setting.
Problem Setup
Transfer Learning.
We use x to denote a covariate in a Hilbert space (that can be d -dimensional orcountably inﬁnite dimensional), and y 2 R to denote its response.
Consider a source and a targetdata distribution, denoted by D source and D target respectively.
In the problem of transfer learning ,we are given with M data sampled independently from the source distribution, and N data sampledindependently from the target distribution (where N ⌧ M or even N = 0 ), denoted by( x i , y i )
M + N i =1 , where ( x i , y i )
⇠
[Equation]
The goal of transfer learning is to learn a model based on the M + N data that can generalize onthe target domain.
We are particularly interested in the covariate shift problem in transfer learning,where the source and target distributions satisfy: D source ( y | x )
=
D target ( y | x ) but D source ( x ) 6 =
D target ( x ) .
Linear Regression under Covariate Shift.
A covariate shift problem is formally deﬁned in thecontext of linear regression by Deﬁnitions 1 and 2.
Deﬁnition 1 (Covariances conditions) .
Assume that each entry and the trace of the source and targetdata covariance matrices are ﬁnite.
Denote the source and target data covariance matrices by
[Equation]
respectively, and denote their eigenvalues by ( µ i ) i  1 and (  i )
i  1 , respectively.
For convenienceassume that both G and H are strictly positive deﬁnite.
Deﬁnition 2 (Model conditions) .
For a parameter w , deﬁne its source and target risks by
[Equation]
respectively.
Assume that there is a parameter w ⇤ that simultaneously minimizes both sourceand target risks, i.e., w ⇤ 2 arg min w Risk source ( w ) \ arg min w Risk target ( w ) .
For convenienceassume that w ⇤ is unique.
We remark that the strict positive deﬁniteness of G and H in Deﬁnition 1 and the uniquenessof w ⇤ in Deﬁnition 2 are only made for the ease of presentation.
Otherwise one can set w ⇤ tobe the minimum-norm solution,
i.e., w ⇤ = arg min { k w k 2 : w 2 arg min w Risk source ( w ) \ 127 arg min w Risk target ( w ) } , and our results still hold.
This argument also holds in a reproducingkernel Hilbert space
[Schölkopf et al., 2002].
Excess Risk.
For linear regression under covariate shift, the performance of a parameter w ismeasured by its target domain excess risk , i.e., ExcessRisk
[Equation]
SGD.
The transfer learning algorithm of our interests is pretraining-ﬁnetuning via stochastic gradientdescent with geometrically decaying stepsizes 1 (SGD).
Without lose of generality, we assume the SGD iterates are initialized from w 0 = 0 .
Then the SGD iterates are sequentially updated as follows:
[Equation]
where
[Equation]
=
[Equation] and the output is the last iterate, i.e., w M + N .
Here  0 and  M are two hyperparameters thatcorrespond to the initial stepsizes for pretraining and ﬁnetuning, respectively.
In both pretraining andﬁnetuning phases, the stepsize scheduler in ( SGD ) is epoch-wisely a constant and decays geometricallyevery certain number of epochs, which is widely used in deep learning
[He et al., 2015].
We notethat such ( SGD ) for linear regression has been analyzed by Ge et al.
[2019], Wu et al.
[2021] in thecontext of supervised learning.
Our goal in this work is to understand the generalization of ( SGD ) inthe covariate shift problems.
Assumptions.
The following assumptions [Zou et al., 2021b, Wu et al., 2021] are crucial in ouranalysis.
Assumption 1 (Fourth moment conditions) .
Assume that for both source and target distribution thefourth moment of the covariates is ﬁnite.
Moreover:
A There is a constant ↵ > 0 such that for every PSD matrix A it holds that
[Equation]
Clearly, it must hold that
↵  1 .
B There is a constant  > 0 such that for every PSD matrix A it holds that
[Equation]
Assumption 1 holds with ↵ = 3 and  = 1 given that D source ( x ) =
N ( 0 , G ) and D target ( x ) =
N ( 0 , H ) .
Moreover, Assumption 1A holds if both H  12 · D source ( x ) and G  12 · D target ( x ) have 151 sub-Gaussian tails [Zou et al., 2021b].
For more exemplar distributions that satisfy Assumption 1, werefer the reader to Wu et al.
Assumption 2 (Noise condition) .
Assume that there is a constant  2 > 0 such that
[Equation]
Assumption 2 puts mild requirements on the conditional distribution of the response given inputcovariates for both source and target distribution.
In particular, Assumption 2 is directly implied bythe following Assumption 2’ for a well-speciﬁed linear regression model under covariate shift.
Assumption 2’ (Well-speciﬁed noise) .
Assume that for both source and target distributions, theresponse (conditional on input covariates) is given byy = x > w ⇤ + ✏ , where ✏ ⇠ N (0 ,  2 ) and ✏ is independent with x .
Additional Notation.
Let N + := { 1 , 2 , . . . } .
For an index set K ⇢ N + , its complement is deﬁnedby K c : = N +  K .
Then for an index set K ⇢ N + and a scalar a  0 , we deﬁne
[Equation]
K
[Equation]
where (  i ) i  1 and ( v i )
i  1 are corresponding eigenvalues and eigenvectors of H .
One can verifythat H  1 K is equivalent to the (pseudo) inverse of H K .
Similarly, we deﬁne G J , G  1 J and a I J + G J caccording to the eigenvalues and eigenvectors of G .
Main Results
An Upper Bound.
We begin with presenting an upper bound for the target domain excess riskachieved by the pretraining-ﬁnetuning method.
Theorem 3.1 (An upper bound) .
Suppose that Assumptions 1A and 2 hold.
Let w M + N bethe output of ( SGD ) .
Let M eff := M/ log( M ) , N eff :
= N/ log( N ) .
Suppose that  0 ,  M <min { 1 / (4 ↵ tr( G )) , 1 / (4 ↵ tr( H )) } .
Then it holds that
[Equation] BiasError [Equation] VarError .
Moreover, for any two index sets J , K ⇢ N + , it holds that
[Equation]
where
[Equation]
In particular, the upper bounds are optimized when
[Equation]
The upper bound in Theorem 3.1 contains a bias error stemming from the incorrect initializationw [-
 ̸-]0 6 = w ⇤ , and a variance error caused by the additive label noise y  x > w ⇤ 6 = 0 .
In particular, M effand N eff are the effective number of source and target data , respectively, due to the effect of thegeometrically decaying stepsizes in ( SGD ).
Moreover, D eff can be regarded as the effective dimensionof supervised learning [Wu et al., 2021] and D finetuneeff can be regared as the effective dimension ofpretraining-ﬁnetuning .
Note that D finetuneeff is determined jointly by the spectrum of the source andtarget population covariance matrices as well as the stepsizes for pretraining and ﬁnetuning.
To better illustrate the spirit of Theorem 3.1, let us consider an example where k w 0  w ⇤ k 22 ,  2 . 1 , 181  0 h 1 , and tr( G )
h tr( H ) h 1 (so that the spectrum of G and H must decay fast), then the boundin Theorem 3.1 vanishes provided that
[Equation]
For the ﬁrst condition in (3) to happen one needs
[Equation]
which can be satisﬁed when (i) the number of target data N is large and the ﬁnetuning stepsize M h 1 , or when (ii) N is small and  M is also small (which can depend on N ).
The secondcondition in (3) can happen under various situations, e.g., when (i) N is large and  M h 1 , or when(ii) N ,  M are small but the amount of source data M is large and that
[Equation]
, which will hold when G aligns well with H (as a sanity check these hold automatically
when G = Hand M is large).
To summarize, in case (i) the amount of target data is plentiful so that ﬁnetuningwith large stepsize leads to generalization (which is essentially supervised learning); and in case (ii) ,even though the target data is scarce, pretraining with abundant source data can still generalize giventhat the source and target population covariance matrices are well aligned.
A Lower Bound.
The following theorem provides a nearly matching lower bound.
Theorem 3.2 (A lower bound) .
Suppose that Assumptions 1B and 2’ hold.
Let w M + N be theoutput of ( SGD ) .
Let M eff := M/ log( M ) , N eff :
= N/ log( N ) , and suppose that M eff , N eff  10 .
Suppose that  0 < 1 / k G k 2 ,  M < 1 / k H k 2 .
Then it holds that
ExcessRisk ( w M + N ) = BiasError + VarError .
Moreover, for the index sets K and J deﬁned in (2) , it holds that
[Equation]
where D eff and D finetuneeff are as deﬁned in (1) .
The lower bound in Theorem 3.1 suggests that the upper bound in Theorem 3.1 is tight upto constantfactor in terms of variance error, and is also tight in terms of bias error except for the followingadditional parts in the respective places:
[Equation]
In particular, the upper and lower bounds match ignoring constant factors provided that
[Equation]
which hold in a statistically interesting regime where the signal-to-noise ratios, k w 0  w ⇤ k 2 G /  2 ,k w 0  w ⇤ k 2 H /  2 , are bounded and G commutes with H .
Implication for Pretraining.
If target data is unavailable, Theorems 3.1 and 3.2 imply the followingcorollary for pretraining.
Corollary 3.3 (Learning with only source data) .
Suppose that Assumptions 1A and 2 hold.
Letw M +0 be the output of ( SGD ) with M > 100 source data and 0 target data.
Suppose that  :=
0 <1 / (4 ↵ tr( H )) .
Then it holds that
ExcessRisk ( w M +0 ) .
[Equation]
where D pretraineff := tr( HG  1 J ) + M 2 eff  2 · tr( HG J c ) and J ⇢ N + can be any index set.
If inaddition Assumptions 1B and 2’ hold, then for the index set J deﬁned in (2) , it holds that
ExcessRisk
[Equation]
Corollary 3.3 sharply characterizes the generalization of pretraining method, and is tight upto constantfactors provided with a bounded signal-to-noise ratio, i.e., k w 0  w ⇤ k 2 G .  2 .
Corollary 3.3 canbe interpreted in a similar way as Theorem 3.1.
Moreover, these sharp bounds for pretraining andpretraining-ﬁnetuning enable us to study the effects of pretraining and ﬁnetuning thoroughly, whichwe will do in Section 4.
Implication for Supervised Learning.
As a bonus, we can also apply Theorems 3.1 and 3.2 in thesetting of supervised learning.
Corollary 3.4 (Learning with only target data) .
Suppose that Assumptions 1A and 2 hold.
Let w 0+ Nbe the output of ( SGD ) with 0 source data and N > 100 target data.
Suppose that  :=
M <1 / (4 ↵ (tr( G ))) .
Then it holds that
ExcessRisk ( w 0+ N ) .
[Equation]
where D eff is as deﬁned in (1) and K ⇢ N + can be any index set.
If in addition Assumptions 1B and2’ hold, then for the index set K deﬁned in (2) , it holds that
[Equation]
We remark that the upper bound in Corollary 3.4 improves the related bound in Wu et al.
[2021] by alogarithmic factor, and matches the lower bound upto constant factors given that k w 0  w ⇤ k 2 H .  2 .
Discussions
With the established bounds, we are ready to discuss the power and limitation of pretraining andﬁnetuning by comparing them to supervised learning.
The Power of Pretraining.
We ﬁrst discuss the effect of pretraining by comparing Corollary 3.3with Corollary 3.4.
Theorem 4.1 (Pretraining vs. supervised learning) .
Suppose that Assumptions 1 and 2’ hold.
Letw
+ N sl be the output of ( SGD ) with optimally tuned initial stepsize, 0 source data and N sl > 100target data.
Let w M +0 be the output of ( SGD ) with optimally tuned initial stepsize, M source dataand 0 target data.
Let M eff := M/ log( M ) , N sleff :
=
N sl / log( N sl ) .
Suppose all SGD methodsare initialized from 0 .
Then for every covariate shift problem instance ( w ⇤ , H , G ) such that H , Gcommute, it holds that
[Equation]
provided that
[Equation]
where
[Equation]
and  sl < 1 / k H k 2 refers to the optimal initial stepsize for supervised learning.
We now explain the implication of Theorem 4.1.
First of all, it is of statistical interest to considera bounded signal-to-noise ratio, i.e., k w ⇤ k 2 G /  2 . 1 .
Note that D sleff  1 when N sl is large.
Moreover, recall that | K ⇤ |  D sleff
= o
( N sleff ) when supervised learning can achieve a vanishingexcess risk.
Finally, k H K ⇤ k G :=
k G  12 H K ⇤ G  12 k 2 . 1 can be satisﬁed if the top | K ⇤ |
= o ( N sleff ) 244 eigenvalues subspace of H mostly falls into the top eigenvalues subspace of G .
Under these remarks, Theorem 4.1 suggests that: in the bounded signal-to-noise cases, pretraining with O ( N 2 ) sourcedata is no worse than supervised learning with N target data (ignoring constant factors), for everycovariate shift problem such that the top eigenvalues subspace of the target covariance matrix alignswell with that of the source covariance matrix.
The Power of Pretraining-Finetuning.
We next discuss the effect of pretraining-ﬁnetuning bycomparing Theorem 3.1 with Corollary 3.4.
Theorem 4.2 (Pretraining-ﬁnetuning vs. supervised learning) .
Suppose that Assumptions 1 and 2’hold.
Let w
+ N sl be the output of ( SGD ) with optimally tuned initial stepsize, 0 source data and
N sl > 100 target data.
Let w M
+
N be the output of ( SGD ) with optimally tuned initial stepsize,
M source data and N target data.
Let M eff := M/ log( M ) , N eff :
= N/ log( N ) , N sleff :=
N sl / log( N sl ) .
Suppose all SGD methods are initialized from 0 .
Then for every covariate shiftproblem instance ( w ⇤ , H , G ) such that H , G commute, it holds that
[Equation]
provided that
[Equation]
where
[Equation]
and K ⇤ , D sleff ,  sl are as deﬁned in Theorem 4.1.
Theorem 4.2 can be interpreted in a similar way as Theorem 4.1.
The only difference is that
Theorem 4.2 puts a milder condition regrading the alignment of H and G than Theorem 4.1.
Inparticular, Theorem 4.2 only requires the “middle” eigenvalues subspace of H mostly falls into thetop eigenvalues subspace of G .
Moreover, the index set K † shrinks (hence k H K † k G decreases) as 264 the number of target data for ﬁnetuning increases.
This indicates that ﬁnetuning can help save theamount of source data for pretraining.
The Limitation of Pretraining vs. the Power of Finetuning.
The following example furtherdemonstrates the limitation of pretraining and the power of ﬁnetuning.
Example 4.3 (Pretraining-ﬁnetuning vs. pretraining vs. supervised learning) .
Let ✏ > 0 be asufﬁciently small constant.
Consider a covariate shift problem instance given by
[Equation]
One may verify that tr( H ) h tr( G ) h 1 and that k w ⇤ k 2 H h k w ⇤ k G h  2 h 1 .
The following holdsfor the ( SGD ) output:
• supervise learning: for ExcessRisk ( w 0+ N ) < ✏ , it is necessary to have that N & ✏  1 . 5 ;
• pretraining: for ExcessRisk ( w M +0 ) < ✏ , it is necessary to have that M & ✏  2 ;• pretrain-ﬁnetuning: for ExcessRisk ( w M + N ) < ✏ , it sufﬁces to set  0 h 1 ,  M h ✏ , andM h ✏  1 log( ✏  1 ) , N h ✏  1 log 2 ( ✏  1 ) .
It is clear that whenever target data are available, the optimally tuned pretraining-ﬁnetuning methodis always no worse than the optiamlly tuned pretraining method, as one can simply set the ﬁnetuningstepsize to be small (or zero) so that the former reduces to the latter.
Moreover, Example 4.3shows a covariate shift problem instance such that pretraining-ﬁnetuning can save polynomiallyamount of data compared to pretraining (or supervised learning).
This example demonstrates thelimitation of pretraining and the beneﬁts of ﬁnetuning.
As a ﬁnal remark for Example 4.3, directcomputation implies that K ⇤ = { 1 , 2 , . . .
, 2 ✏  0 .
5 + 1 } and K † = { 2 , 3 , . .
, 2 ✏  0 .
5 + 1 } , thereforek H K
⇤ k G = ✏  2  k H K † k G = ✏ 0 .
5 , so the implication of Example 4.3 is consistent with Theorems 284 4.1 and 4.2.
Numerical Simulations.
We perform experiments on synthetic data to verify our theory.
Recallthat the effectiveness of pretraining and ﬁnetuning depends on the alignment between source andtarget covariance matrices, therefore we design experiments where the source and target covariancematrices are aligned at different levels.
In particular, we consider commutable matrices G and Hwith eigenvalues { µ i } i  1 =
{ i  2 } i  1 and (  i ) i {+ 
+} =
( i  1 . 5 )
i  1 , respectively.
To simulate different 290 alignments between G and H , we ﬁrst sort them so that both of their eigenvalues are in descendingorder, and then reverse the top- k eigenvalues of G .
In mathematics, for a given k > 0 , the probleminstance P ( k ) :
= ( w ⇤ , H , G ,  2 ) is designed as follows: [Equation]
k copies
[Equation]
One can verify that tr( H ) h tr( G ) h 1 and that k w ⇤ k 2 H h k w ⇤ k 2 G h  2 h 1 .
Clearly, a larger 294 k implies a worse alignment between G and H .
We then test three problem instances P ( k = 5) ,
P ( k = 10) , and P ( k = 20) , and compare the excess risk achieved by pretraining, pretrainingﬁnetuning, and supervise learning.
The results are presented in Figure 1, which lead to the followinginformative observations:
• For problem P ( k = 5) where H and G are aligned very well, pretraining (without ﬁnetuning!)
can already match the generalization performance of supervised learning.
This veriﬁes the power ofpretraining for tackling transfer learning with mildly shifted covariate.• For problem P ( k = 10) where H and G are moderately aligned, there is a signiﬁcant gap betweenthe risk of pretraining and that of supervised learning.
Yet, the gap is closed when the pretrained [Figure] model is ﬁnetuned with scarce target data.
This demonstrates the limitation of pretraining and thepower of ﬁnetuning for tackling transfer learning with moderate shifted covariate.
• For problem P ( k = 20) where H and G are poorly aligned, the risk of pretraining can hardlycompete with that of supervised learning.
Moreover, for ﬁnetuning to match the performance ofsupervised learning, it requires nearly the same amount of target data as that used by supervised
learning.
This reveals the limitation of pretraining and ﬁnetuning for tackling transfer learning withseverely shifted covariate.
Concluding Remarks
We consider linear regression under covariate shift, and a SGD estimator that is ﬁrstly trained withsource domain data and then ﬁnetuned with target domain data.
We derive sharp upper and lowerbounds for the estimator’s target domain excess risk.
Based on the derived bounds, we show thatfor a large class of covariate shift problems, pretraining with O ( N 2 ) source data can match theperformance of supervised learning with N target data.
Moreover, we show that ﬁnetuning withscarce target data can signiﬁcantly reduce the amount of source data required by pretraining.
Finally,when applied to supervised linear regression, our results improve the upper bound in [Wu et al., 2021]by a logarithmic factor, and close its gap with the lower bound (ignoring constant factors) when thesignal-to-noise ratio is bounded.
Several future directions are worth discussing.
Model Shift.
An immediate follow-up problem is to extend our results from the covariate shiftsetting to more general transfer learning settings, e.g., with both covariate shift and model shift , wherethe true parameter w ⇤ could also be different for source and target distribution.
Under model shift,the power of pretraining with source data is limited, and we expect that ﬁnetuning with target databecomes even more important.
Ridge Regression.
For inﬁnite-dimensional least-squares in the supervised learning context, instancewisely tight bounds for both ridge regression and SGD have been established by Bartlett et al.
Tsigler and Bartlett [2020], Zou et al.
[2021b], Wu et al.
For inﬁnite-dimensional leastsquares under covariate shift, this paper presents nearly instance-wisely tight bounds for SGD.
Asridge regression is popular in covariate shift literature (see Ma et al.
[2022] and references herein),an interesting future direction is studying the instance-wisely tight bounds for ridge regression inthe setting of inﬁnite-dimensional least-squares under covariate shift — a tight bias analysis is ofparticular interest.
Unlabeled Data.
In this work we assume the provided source and target data are both labeled.
However in many practical scenarios, additional unlabeled source and target data are also available.
In this case our results cannot be directly applied as it remains unclear how to utilize unlabeled datawith SGD.
An important future direction is to extend our framework to incorporate with unlabeledsource and target data.
[-For t = 0 , it is clear that C 0 = 
-]
[-Theorem C.
-]
[-M eff γ 
-]
[-M eff γ 
-]
[-= ( I − γ t G )( w 0 − w ∗ ) (cid:13)(cid:13)(cid:13) 
-]
[-{ j : µ j ≥ 1 / ( M eff γ 0 ) } so that( ♣ ) ≤ α ⟨ H , γ 
-]
[-I ⟩ ≤ αγ 
-]
[-Finally, we choose γ 
-]