Under review as a conference paper at ICLR
N EAR -O PTIMAL L INEAR R EGRESSION UNDER D ISTRIBUTION S HIFT
Anonymous authors
Paper under double-blind review
A BSTRACT
Transfer learning is an essential technique when sufﬁcient data comes from the source domain, while no or scarce data is from the target domain.
We develop estimators that achieve minimax linear risk for linear regression problems under the distribution shift.
Our algorithms cover different kinds of settings with covariate shift or model shift.
We also consider when data are generating from either linear or general nonlinear models.
We show that afﬁne minimax rules are within an absolute constant of the minimax risk even among nonlinear rules for various source/target distributions.
I NTRODUCTION
The success of machine learning crucially relies on the availability of labeled data.
The data labeling process usually requires much human labor and can be very expensive and time-consuming, especially for large datasets like ImageNet (Deng et al., 2009).
On the other hand, models trained on one dataset, despite performing well on test data from the same distribution they are trained on, are often sensitive to distribution shifts , i.e., they do not adapt well to related but different distributions.
Even small distributional shift can result in substantial performance degradation (Recht et al., 2018;
Lu et al., 2020).
Transfer learning has been an essential paradigm to tackle the challenges associated with insufﬁcient labeled data (Pan & Yang, 2009; Weiss et al., 2016; Long et al., 2017).
The main idea is to make use of a source domain with a lot of labeled data (e.g. ImageNet), and to try to learn a model that performs well on our target domain (e.g. medical images) where few or no labels are available.
Despite the lack of labeled data, we may still use unlabeled data from the target domain, which are usually much easier to obtain and can provide helpful information about the target domain.
Although this approach has been integral to many applications, many fundamental questions are left open even in very basic settings.
In this work, we focus on the setting of linear regression under distribution shift and ask the fundamental question of how to optimally learn a linear model for a target domain, using labeled data from a source domain and unlabeled data (and possibly some labeled data) from the target domain.
For various settings, including covariate shift (i.e., when p ( x ) changes) and model shift (i.e., when p ( y | x ) changes), we develop estimators that achieve near minimax risk (up to universal constant factors) among all linear estimation rules.
Here linear estimators refer to all estimators that depend linearly on the label vector; these include almost all popular estimators known in linear regression, such as ridge regression and its variants.
When the input covariances in source and target domains commute, we prove that our estimators achieve near minimax risk among all possible estimators.
Under review as a conference paper at ICLR
A key insight from our results is that, when covariate shift is present, we need to apply datadependent regularization that adapts to changes in the input distribution.
For linear regression, this can be given by the input covariances of source and target tasks, which can be estimated using unlabeled data.
Our experiments verify that our estimator has signiﬁcant improvement over ridge regression and similar heuristics.
1. R ELATED WORK
Different types of distribution shift are introduced in (Storkey, 2009; Quionero-Candela et al., 2009).
Speciﬁcally, covariate shift occurs when the marginal distribution on P ( X ) changes from source to target domain (Shimodaira, 2000; Huang et al., 2007).
Wang et al.
; Wang &
Schneider (2015) tackle model shift ( P ( Y | X ) ) provided the change is smooth as a function of X .
Sun et al.
(2011) design a two-stage reweighing method based on both covariate shift and model shift.
Other methods like the change of representation, adaptation through prior, and instance pruning are proposed in (Jiang & Zhai, 2007).
In this work, we focus on the above two kinds of distribution shift.
For modeling target shift ( P ( Y ) ) and conditional shift ( P ( X | Y ) ), Zhang et al.
(2013) exploits the beneﬁt of multi-layer adaptation by some location-scale transformation on X .
Transfer learning/domain adaptation are sub-ﬁelds within machine learning to cope with distribution shift.
A variety of prior work roughly falls into the following categories.
1) Importancereweighting is mostly used in the covariate shift.
(Shimodaira, 2000; Huang et al., 2007; Cortes et al., 2010); 2) One fruitful line of work focuses on exploring robust/causal features or domaininvariant representations through invariant risk minimization (Arjovsky et al., 2019), distributional robust minimization (Sagawa et al., 2019), human annotation (Srivastava et al., 2020), adversarial training (Long et al., 2017; Ganin et al., 2016), or by minimizing domain discrepancy measured by some distance metric (Pan et al., 2010; Long et al., 2013; Baktashmotlagh et al., 2013;
Gong et al., 2013; Zhang et al., 2013; Wang & Schneider, 2014) ; 3) Several approaches seek gradual domain adaptation (Gopalan et al., 2011;
Gong et al., 2012; Glorot et al., 2011; Kumar et al., 2020) through self-training or a gradual change in the training distribution.
Near minimax estimations are introduced in Donoho (1994) for linear regression problems with
Gaussian noise.
For a more general setting, Juditsky et al.
(2009) estimate the linear functional using convex programming.
Blaker (2000) compares ridge regression with a minimax linear estimator under weighted squared error.
Kalan et al.
(2020) considers a setting similar to this work of minimax estimator under distribution shift, but focuses on computing the lower bound for linear and one-hidden-layer neural network under distribution shift.
A few more interesting results are derived on the generalization lower bound for distribution shift under various settings (David et al., 2010; Hanneke & Kpotufe, 2019; Ben-David et al., 2010; Zhao et al., 2019).
P RELIMINARY
We formalize the setting considered in this paper for transfer learning under the distribution shift.
Notation and setup.
Let p S ( x ) and p T ( x ) be the marginal distribution for x in source and target domain.
The associated covariance matrices are Σ S , and Σ T .
We assume to have sufﬁcient unlabeled data to estimate Σ T accurately.
We observe n S , n T labeled samples from source and target domain.
Data is scarce in target domain: n S (cid:29) n T and n T can be 0 .
Speciﬁcally, X S =
[ x (cid:62) 1 | x (cid:62) 2 | · · · | x (cid:62) n S ] (cid:62) ∈ R n × d , with x i , i ∈ [ n S ] drawn from p S , noise z =
[ z 1 , z 2 , · · · z n S ] (cid:62) ,
z i ∼
N (0 , σ 2 ) .
y S =
[ y 1 , y 2 , · · · , y n S ] (cid:62) ∈ R n S , with each y i
= f ∗ ( x i )
+ z i ( X T ∈ R n T × d and
Under review as a conference paper at ICLRy T ∈ R n T are similarly deﬁned).
Denote by ˆΣ S = X (cid:62) S X S /n
S the empirical covariance matrix.
The positive part of a number is denoted by ( x ) + .
We consider both linear ( f ∗ ( x ) =
x (cid:62) β ∗ ) and general nonlinear ground truth models.
When the optimal linear model changes from source to domain we add a subscript for distinction, i.e. , β ∗ S and β ∗ T .
We use bold ( x ) symbols for vectors, lower case letter ( x ) for scalars and capital letter ( A ) for matrices.
Minimax (linear) risk.
In this work, we focus on designing linear estimators ˆ β =
A y S 1 for parameter β ∗ ∈ B .
Our estimator is evaluated by the excess risk on target domain, with the worst case β ∗ in some set B : L B ( ˆ β ) = max β ∗ ∈B E y S (cid:107) Σ 1 / 2 T ( ˆ β ( y S ) − β ∗ ) (cid:107) 2 .
Minimax linear risk and minimax risk among all estimators are respectively deﬁned as:
[Equation]
The subscript “N" or “L" is a mnemonic for “non-linear" or “linear" estimators.
R N is the optimal risk with no restriction placed on the class of estimators.
R L only considers the linear function class for ˆ β .
Minimax linear estimator and minimax estimator are the estimators that respectively attain R L and R N within universal multiplicative constants.
Normally we only consider B = { β |(cid:107) β (cid:107) 2 ≤ r } .
When there is no ambiguity, we simplify ˆ β ( y S ) by ˆ β .
Our meta-algorithm.
Our paper considers different settings with distribution shift.
Our methods are uniﬁed under the following meta-algorithm:
Step 1: Find an unbiased sufﬁcient statistic ˆ β SS2 for the unknown parameter
Step 2: Find ˆ β MM , a linear operator applied to ˆ β SS that minimizes L B ( ˆ β MM ) .
For each setting, we will show that ˆ β MM achieves linear minimax risk R L (asymptotically or in ﬁxed design).
Furthermore, under some conditions, the minimax risk R N is uniformly lower bounded by a universal constant times L B ( ˆ β MM ) .
Outline.
In the sections below, we tackle the problem in different settings.
In Section 3 we design algorithms with only covariate shift: 1) n T = 0 and f ∗ ( x ) is linear (Section 3.1); 2) n T =and f ∗ ( x ) is a general nonlinear function (Section 3.2); 3) n T >
and f ∗ is linear (Section 3.3).
Finally, we cope with the model shift for linear models ( β ∗ S (cid:54) =
β ∗ T ) in Section 4.
M INIMAX ESTIMATOR WITH COVARIATE SHIFT
In this section, we consider the setting with only covariate shift.
That is, only Σ S (marginal distribution p S ( x ) ) changes to Σ T ( p T ( x ) ), but f ∗ =
E [ y | x ] (conditional distribution p ( y | x ) ) is shared.
We ﬁrst consider the case when f ∗ is a linear map: x → x (cid:62) β ∗ and then consider the problem with approximation power.
3. C OVARIATE SHIFT WITH LINEAR MODELS
We observe n S samples from source domain: y S =
X S β ∗ + z , z ∼ N (0 , σ 2 I ) and no labeled samples from the target domain.
Our goal is to ﬁnd the minimax linear estimator ˆ β MM ( y S ) =
A y S with some linear mapping A that attains R L ( B ) .
Following our meta-algorithm, let ˆ β SS = 1 n S ˆΣ − 1 S X (cid:62) S y S 3 be an unbiased sufﬁcient statistic for β ∗ :
[Equation]
Here we consider X S as ﬁxed values, and randomness only comes from noise z .
We prove that the minimax linear estimator is of the form ˆ β MM =
C ˆ β SS and then design algorithms that calculate the optimal C .
Claim 3.1.
The minimax linear estimator is of the form ˆ β MM =
C ˆ β SS for some C ∈ R d × d .
Warm-up: commutative covariance matrices.
In order to derive the minimax linear estimator, we ﬁrst consider the simple case when Σ T and ˆΣ S are simultaneously diagonalizable.
We apply
Pinsker’s Theorem (Johnstone, 2011) and get:
Theorem 3.2 (Linear Minimax Risk with Covariate Shift) .
Suppose the observations follow sequence model y S
=
X S β ∗ + z , z ∼ N (0 , σ 2 I n ) .
If Σ T = U diag( t ) U (cid:62) and ˆΣ S ≡ X (cid:62) S X S /n S =
U diag( s ) U (cid:62) , then the minimax linear risk
[Equation]
where B = { β |(cid:107) β (cid:107) ≤ r } , and λ = λ ( r ) is determined by σ 2 n S (cid:80) di
=1 1 s i ( √ t i /λ − 1) + = r 2 .
The linear minimax estimator is given by:
[Equation]
Since r is unknown in practice, we could simply view either r or directly λ as the tuning parameter.
We compare the functionality of λ with that of ridge regression: ˆ β λ RR = arg min ˆ β E 12 n (cid:107) X S
ˆ β − y S (cid:107)
+ λ 2 (cid:107) ˆ β (cid:107) 2 =
(ˆΣ S + λI ) − 1 X (cid:62) S y S /n S .
For both algorithms, λ is to balance the bias and variance: λ = 0 gives an unbiased estimator, and a big λ gives a (near) zero estimator with no variance.
The difference is, our estimator shrinks some signal directions based on the value of t i .
The estimator tends to sacriﬁce the directions of signal where t i is smaller.
Ridge regression, however, respects the value of s i .
A natural counterpart is for ridge to also regularize based on t : let ˆ β λ RR ,T = arg min 1 n (cid:107) Σ 1 / 2 T ( β − ˆΣ − 1 S X (cid:62) S y S ) (cid:107)
+ λ (cid:107) β (cid:107) 2 =
(Σ T + λI ) − 1 Σ T ˆ β SS .
We will compare their performances in the experimental section.
Non-commutative covariance matrices.
For non-commutative covariate shift, we follow the same procedure.
Our estimator is achieved by optimizing over C :
ˆ β MM =
C ˆ β SS :
[Equation]
(Claim 3.8)
Under paper
[Equation]
Unlike the commutative case, this problem doesn’t have a closed form solution, but is still solvable:
Proposition 3.3.
Problem (3) is a convex program and thus solvable.
We achieve near-optimal minimax risk among all estimators under some conditions:
Theorem 3.4 (Near minimaxity of linear estimators) .
When Σ S , Σ T commute, or Σ T is rank 1, the best linear estimator from (2) or (3) achieves near-optimal minimax risk: L B ( ˆ β MM ) =
R L ( B ) ≤ 1 .
25 R N ( B ) .
Note that R N ≤ R L .
Therefore 1) our estimator ˆ β MM is near-optimal, and 2) our lower bound for
R N is tight.
Lower bounds (without matching upper bounds) for general non-commutative problem is presented in (Kalan et al., 2020) and we improve their result for the commutative case and provide a matching algorithm.
Their lower bound scales with dn S min i t i s i for large r , while ours becomes 1 n S (cid:80) i t
i s i .
Our lower bound is always larger and thus tighter, and potentially arbitrarily larger when max i t i s i and min i t
i s i are very different.
We defer our proof to the appendix.
Remark 3.1 (Beneﬁt of minimax linear estimator) .
Consider estimators from ridge regression: ˆ β λ RR = arg min ˆ β E 12 n (cid:107) X S
ˆ β − y S (cid:107)
+ λ 2 (cid:107) ˆ β (cid:107) 2 .
There is an example that R L ( B ) ≤
O ( d − 1 / 4 L B ( ˆ β λ RR )) even with the optimal hyperparameter λ .
3. L INEAR MINIMAX ESTIMATOR WITH APPROXIMATION ERROR
Now we consider observations coming from nonlinear models: y S = f ∗ ( X S )
+ z .
Let β ∗ S =
arg min
β E x ∼ p S ,z ∼N (0 ,σ 2 ) [( f ∗ ( x ) + z − β (cid:62) x ) 2 ] , and similarly for β ∗ T .
Notice now even with f ∗ unchanged across domains, the input distribution affects the best linear model.
Approximation error is a S ( x ) =
f ∗ ( x ) − x (cid:62) β ∗ S and vice versa for a T .
Deﬁne the reweighing vector w ∈ R n as w i = p T ( x i ) /p
S ( x i ) .
We form unbiased estimator via
[Equation]
Claim 3.5.
ˆ β LS is asymptotically unbiased and normally distributed:
[Equation]
Denote by m ( x ) = a T ( x ) + z .
We want to minimize the worst case risk:
[Equation]
Therefore our estimator is ˆ β MM ← ˆ C ˆ β LS , where ˆ C ﬁnds
[Equation]
Claim 3.6.
Let B = { β |(cid:107) β (cid:107) ≤ r } , and f ∗ ∈ F is some compact symmetric function class: f ∈ F ⇔ − f ∈ F .
Then linear minimax estimator is of the form C ˆ β LS for some C .
When ˆ C solves Eqn.
(4) , L B ( ˆ β MM ) asymptotically matches R L ( B ) , the linear minimax risk.
By reducing from y S to ˆ β LS we eliminate n − d dimensions, and this claim says that X (cid:62) S y S is sufﬁcient to predict β ∗ T .
We note that f ∗ is more general than a linear function and therefore the lower bound could only be larger than R N ( B ) deﬁned in the previous section.
3. U TILIZE SOURCE AND TARGET LABELED DATA JOINTLY
In some scenarios we have moderate amount of labeled data from target domain as well.
Then it is important to utilize the source and target labeled data jointly.
Let y S
=
X S β ∗ + z S , y T =
X T β ∗ + z T .
We consider X S , X T as deterministic variables, ˆΣ − 1 S X (cid:62) S y S /n S ∼ N ( β ∗ , σ 2 n S ˆΣ − 1 S ) and ˆΣ − 1 T X (cid:62) T y T /n
T ∼ N ( β ∗ , σ 2 n T ˆΣ − 1 T ) .
Therefore conditioned on the observations y S , y T , a sufﬁcient statistic for β ∗ is ˆ β SS :=
( n S ˆΣ S + n T ˆΣ T ) − 1 ( X (cid:62) S y S + X (cid:62) T y T ) .
Claim 3.7.
ˆ β SS is an unbiased sufﬁcient statistic of β ∗ with samples y S , y T .
ˆ β SS ∼
N ( β ∗ , σ 2 ( n S ˆΣ S + n T ˆΣ T ) − 1 ) .
Algorithm: First consider the estimator ˆ β SS =
( n S ˆΣ S + n T ˆΣ T ) − 1 ( X (cid:62) S y S + X (cid:62) T y T ) .
Next ﬁnd the best linear function of ˆ β SS :
[Equation]
Proposition 3.8.
The minimax estimator ˆ β MM is of the form C ˆ β SS for some C .
When choosing
C with our proposed algorithm and when ˆΣ S commutes with ˆΣ T , we achieve the minimax risk
R L ( B ) ≤ 1 .
25 R N ( B ) .
N EAR MINIMAX ESTIMATOR WITH MODEL SHIFT
The general setting of transfer learning in linear regression involves both model shift and covariate shift.
Namely, the generative model of the labels might be different: y S =
X S β ∗ S + z S , and y T
=
X T β ∗ T + z T .
Denote by δ := β ∗ S − β ∗ T as the model shift.
We are interested in the minimax linear estimator when (cid:107) δ (cid:107) ≤ γ and (cid:107)
β ∗ T (cid:107) ≤ r .
Thus our problem becomes to ﬁnd minimax estimator for β ∗ T ∈ B = { β |(cid:107) β (cid:107) ≤ r } from y S , y T .
Algorithm:
First consider a sufﬁcient statistic ( ¯ β S , ¯ β T ) for ( β ∗ T , δ ) .
Here ¯ β S = ˆΣ − 1 S X (cid:62) S y S /n
S ∼ N ( β ∗ T + δ , σ 2 n S ˆΣ − 1 S ) , and ¯ β T =
ˆΣ − 1 T X (cid:62) T y T ∼ N ( β ∗ T , σ 2 n T ˆΣ − 1 T ) .
Then consider the best linear estimator on top of it: ˆ β
=
A 1 ¯ β S + A 2 ¯ β T .
Write ∆ = { δ |(cid:107) δ (cid:107) ≤ γ } and
L B , ∆ ( ˆ β ) :
= max β ∗ T ∈B , δ ∈ ∆ (cid:107) Σ 1 / 2 T ( ˆ β − β ∗ T ) (cid:107) 2 .
[Equation]
Therefore we optimize over this upper bound and reformulate the problem as a convex program:
[Equation]
Our estimator is given by:
[Equation]
=
[Equation]
A
[Equation]
β S + ˆ A 2 ¯ β T .
Since ˆ β MM is a relaxation of the linear minimax estimator, it is important to understand how well ˆ β MM performs on the original objective:
[Equation]
[Equation]
Finally we show with the relaxation we still achieve a near-optimal estimator even among all nonlinear rules.
Theorem 4.2.
When Σ T commutes with Σ S , it satisﬁes:
[Equation]
Here R N
[Equation]
is the minimax risk.
Proof sketch of Theorem 4.2.
For the ease of understanding, we provide a simple proof sketch when Σ S = Σ T are diagonal.
We ﬁrst deﬁne the hardest hyperrectangular subproblem.
Let
B ( τ ) =
{ b : | β i | ≤ τ i } be a subset of B and similarly for ∆( ζ ) .
We show that R L ( B , ∆) =
max τ ∈B , ζ ∈ ∆ R L ( B ( τ ) , ∆( ζ )) , and clearly R N ( B , ∆) ≥ max τ ∈B , ζ ∈ ∆ R N ( B ( τ ) , ∆( ζ )) .
Meanwhile we show when the sets are hyperrectangles the minimax (linear) risk could be decomposed to 1-d problems: R L ( B ( τ ) , ∆( ζ ))
= (cid:80) i R L ( τ i , ζ i ) .
Each R L ( τ i , ζ i ) is the linear minimax risk to estimate β i from x ∼ N ( β i
+ δ i , 1) and y ∼ N ( β i , 1) where | β i | ≤ τ i and | δ i | ≤ ζ i .
This 1-d problem for linear risk has a closed form solution, and the minimax risk can be lower bounded using Le Cam’s two point lemma.
We show R L ( τ i , ζ i ) ≤ 13 .
5 R N ( τ i , ζ i ) and therefore:
[Equation]
[Figure]
E XPERIMENTS
Our estimators are provably near optimal for the worst case β ∗ .
However, it remains unknown whether on average they outperform other baselines.
With synthetic data we explore the performances with random β ∗ .
We are also interested to investigate the conditions when we win more.
Setup.
We set n S = 2000 , d = 50 , σ = 1 , r = √ d .
For each setting, we sample β ∗ T from standard normal distribution and rescale it to be norm r .
We assume to know Σ T .
We compare our estimator with ridge regression (S-ridge) and a variant of ridge regression transformed to target domain (T-
[Equation]
Covariate shift.
In order to understand the effect of covariate shift on our algorithm, we consider three types of settings, each with a unique varying factor that inﬂuences the performance: 1) covariate eigenvalue shift with shared eigenspace; 2) covariate eigenspace shift with ﬁxed eigenvalues 5 ; 3) signal strength change.
We also have an additional 200 labeled data from target domain as validation set only for hyper-parameter tuning.
Model shift.
Next we consider the problem with model shift.
We sample a random δ with norm γ varying from 0 to r = √ d and observe data generated by y S =
X S ( β ∗ T + δ ) + z S ∈ R 2000 , z S ∼
N (0 , I ) and y T =
X T β ∗ T + z T ∈ R 500 , z T ∼ N (0 , I ) .
We compare our estimator with two baselines: "ridge-source" ridge using only source data and "ridge-
Figure 1 demonstrates the better performance of our estimator in all circumstances.
From (a) we see that with more discrepancy between Σ S and Σ T , our estimator tends to perform better.
(b) shows our estimator is better when the signal is relatively stronger.
From (c) we can see that with the increasing model shift measured by γ/r , ridge-source becomes worse and is outperformed by ridge-target that remains unchanged.
Our estimator becomes slightly worse as well due to the less utility from source data, but remains the best among others.
When γ/r ≈ 0 . 2 , our method has the most improvement in percentage compared to the best result among ridge-source and ridge-target.
A O MITTED PROOF FOR MINIMAX ESTIMATOR WITH COVARIATE SHIFT A.
P INSKER ’ S T HEOREM
AND COVARIATE SHIFT WITH LINEAR MODEL
Theorem A.1 (Pinsker’s Theorem) .
Suppose the obervations follow sequence model
y i
=
θ ∗ i + (cid:15)
i z i , (cid:15) i > 0 , i ∈
[ d ] , and Θ is an ellipsoid in R d : Θ = Θ( a, C ) =
{ θ : (cid:80) i
a 2 i θ 2 i ≤ C 2 } .
Then the minimax linear risk
[Equation]
where µ = µ ( C ) is determined by
[Equation]
The linear minimax estimator is given by
[Equation]
and is Bayes for a Gaussian prior π C having independent components θ i ∼ N (0 , τ 2 i ) with τ ∗ i = (cid:15)
i ( µ/a i − 1) + .
Our theorem 3.2 is to connect our parameter β ∗ to the θ ∗ in pinsker’s theorem.
First we show that reformulating the problem from a linear map of n dimensional observations y S to a linear map on the d -dimensional statistic ˆ β SS is sufﬁcient, i.e., Claim 3.1:
Proof of Claim 3.1.
This is to show that if ˆ β ( y S ) :
=
A y S is a minimax estimator, each row vector of A ∈ R d × n is in the column span of X S .
Write A
=
A 1 X (cid:62) S + A 2 W (cid:62) where W ∈ R n × ( n − d ) , columns of which forms the orthonormal complement for the column space of X S .
Equivalently we want to show A 2 = 0 .
We have
[Equation]
Since W (cid:62) X S = 0 )
[Equation]
(Other cross terms vanish since E [ z ] = 0 ) [Equation] where the last equation is because [Equation]
[Equation] Clearly, at min-max point, without loss of generality we can take A 2 = 0 .
Formally the proof for Theorem 3.2 is presented here:
Proof of Theorem 3.2.
To use Pinsker’s theorem to prove Theorem 3.2, we simply need to transform the problem match its setting.
Let y T
=
Σ 1 / 2 T ˆΣ − 1 S X (cid:62)
S y S
θ ∗ T + z T , where θ ∗ T =
U (cid:62) Σ 1 / 2 T β ∗ and z T ∼ N (0 , σ 2 diag([ t i /s i ] di =1 ) /n S ) .
The set for θ ∗ T is Θ = { θ |(cid:107)
Σ − 1 / 2 S
[Equation]
[Equation] Θ = { θ | (cid:80) i
θ
i /t i ≤ r 2 } .
Now with Pinsker’s theorem, ˆ θ ( y T ) i =
(1 − 1 / ( µ √ t i ))
+ ( y T )
i is the best linear estimator for θ ∗ T , where µ = µ ( r ) solves
[Equation]
Connecting to the original problem, we get that the best estimator for Σ 1 / 2 T β ∗ is U ( I − 1 µ diag([1 / √ t i ] di =1 ))
y T = U ( I − 1 µ diag([1 / √ t i ] di =1 )) U (cid:62) Σ 1 / 2 T Σ − 1 S X (cid:62) S y S /n S .
A. O MITTED
Convex program.
Our estimator for β ∗ can be achieved through convex programming:
Proof of Proposition 3.3.
First note the objective function is quadratic in C and linear in τ , therefore we only need to prove the constraint S
= { ( C, τ ) | ( C − I ) (cid:62) Σ T ( C − I ) (cid:22) τI } is a convex set.
Notice for ( C 1 , τ 1 ) , ( C 2 , τ 2 ) ∈ S , i.e., ( C i − I ) (cid:62) Σ T ( C i − I ) (cid:22)
τ i
I, i ∈ { 1 , 2 } .
We simply need to prove for C α :=
αC 1 + (1 − α ) C 2 , τ α :=
τ 1 α + τ 2 (1 − α ) , ( C α − I ) (cid:62) Σ T ( C α − I ) (cid:22) τ α
I for any α ∈ [0 , 1] .
First, notice ( C 1 − C 2 ) (cid:62) Σ T ( C 1 − C 2 ) (cid:23) 0 . Next, ( C
[Equation] I ) (cid:62) Σ [Equation] ( C α − I ) =
[Equation]
+
[Equation]
(cid:62)
[Equation]
( C
[Equation]
) −
[Equation]
(cid:22) α ( C 1 − I ) (cid:62) Σ T ( C 1 − I )
+ (1 − α )
( C 2 − I ) (cid:62) Σ
[Equation]
( C 2 − I ) (cid:22) τ α I.
Beneﬁt of our estimator.
Compared to ridge regression, our estimator could possibly achieve much better ( d − 1 / 4 ) improvements:
Proof of Remark 3.1.
We consider diagonal covariance matrices ˆΣ S = diag( s ) , Σ T = diag( t ) , σ = 1 .
First we calculate the expected risk obtained with ridge regression: ˆ β λ RR =
( X (cid:62) S X S /n +
[Equation]
Compared to our risk:
[Equation]
where 1 n (cid:80) di =1 √ t i s i ( µ − 1 √ t i )
+ =
r 2 .
Let r 2 = √ dn S , s i
= 1 , ∀ i, t i
= 1 , ∀ i ∈
[ d 0 ] , t i =
d − 1 / 2 , d 0 < i ≤ d, where d 0 = √
d d 1 / 4 − 1 ≈ d 1 / 4 .
Then µ = 1 , and R L ( B ) =
d 1 / 4 n .
In this case,
[Equation]
Therefore min λ L B ( ˆ β λ RR ) ≥ d 1 / 4 R L ( B ) / 2 .
Near minimax risk.
Even among all nonlinear estimators, our estimator is within 1.25 of the minimax risk:
Proof of Theorem 3.4.
First we note that for both linear and nonlinear estimators, it is sufﬁcient to use ˆ β SS instead of the original observations y S .
See Lemma A.2 and its corollary.
Therefore it sufﬁces to do the following reformulations of the problem.
When Σ S and Σ T commute, we formulate the problem as the following Gaussian sequence model.
Recall ˆ Σ S =
U diag( s ) U (cid:62) , Σ T =
U diag( t ) U (cid:62) .
Let θ ∗ =
U (cid:62) Σ 1 / 2 T β ∗ , and y
=
U (cid:62) Σ 1 / 2 T ˆ β SS ∼
N ( θ ∗ , σ 2 n S diag( t / s )) .
Our objective of minimizing (cid:107) Σ 1 / 2 T ( ˆ β ( y S ) − ˆ β ∗ ) (cid:107) from linear estimator is equivalent to minimizing (cid:107) U
(ˆ θ ( y ) − ˆ θ ∗ ) (cid:107) =
(cid:107) ˆ θ ( y )
− ˆ θ ∗ (cid:107) from linear estimator.
The set for the parameter that satisﬁes θ ∗ =
U (cid:62) Σ 1 / 2 T β ∗ , (cid:107) β ∗ (cid:107) ≤ r is equivalent to (cid:107) Σ − 1 / 2 T
U θ ∗ (cid:107) ≤ r ⇔ (cid:107) θ ∗ i / √ t i (cid:107) ≤ r is an axis-aligned ellipsoid.
Then we could directly derive our result from
Corollary 4.26 from Johnstone (2011).
Note that this result is a special case of Theorem 4.2 and we have provided a detailed proof in Section
B. Therefore here we save further descriptions.
For the case when Σ T = aa (cid:62) is rank-1, the objective function becomes:
[Equation]
Under review as a conference paper at ICLR
Then the result could be derived from Corollary 1 of Donoho (1994), which reformulate the problem to the hardest one-dimensional problem which becomes tractable.
In the proof above, we equate the best nonlinear estimator on y S as the best nonlinear estimator on ˆ β SS .
The reasoning is as follows:
Lemma A.2 (Sufﬁcient statistic is enough to achieve a best estimator) .
Consider the statistical problem of estimating β ∗ ∈ B from observations y ∈ Y .
B (cid:96) 2 -compact.
If S ( y ) is a sufﬁcient statistic of β ∗ , then the best estimator that achieves min ˆ β max B (cid:96) ( ˆ β , β ∗ ) is of the form ˆ β = f
( S ( y )) with some function f , for any loss (cid:96) :
Y → [0 , ∞ ) .
This Lemma is restated from Proposition 3.13 from Johnstone (2011).
Corollary A.3 (Corollary of Lemma A.2) .
Under the same setting of Lemma A.2, R N ( B ) is achieved with the form ˆ β = f ( S ( y )) .
A. O MITTED PROOF WITH APPROXIMATION ERROR
Unbiased estimator for ˆ β ∗ T .
Proof of Claim 3.5.
[Equation]
Notice E x ∼ p S
[ x a T ( x ) p T ( x ) p S ( x ) ] = E x ∼ p T [ x a T ( x )] = 0 .
E x i ∼ p S
[ X (cid:62) S diag( w
[Equation]
X S ] = E x i ∼ p S (cid:80) ni
=1 p T ( x i ) p S ( x i ) x
{+i x (cid:62) i = E x j ∼ p T (cid:80) nj =
+}
[ x j x (cid:62) j ] = n S Σ T .
Therefore
[Equation]
Proof of Claim 3.6.
Recall X S =
[ x (cid:62) 1 | x (cid:62) 2 | · · · | x (cid:62) n ] (cid:62) ∈ R n × d , with x i , ∀ i ∈ [ n ] drawn from p S , and a T = [ a T ( x 1 ) , a T ( x 2 ) , · · · a T ( x n )] (cid:62) ∈ R n , y =
[ y ( x 1 ) , y ( x 2 ) , · · · , y ( x n )] (cid:62) ∈ R n , noise z = y − f ∗ ( X ) .
w =
[ p T ( x i ) /p S ( x i )] (cid:62) .
To prove the, we only need to show the minimax linear estimator A y is achieved of the form
A 1 X (cid:62) diag( w ) , i.e., the row span of A is in the row span of X (cid:62) diag( w ) .
[Equation]
[-2 + E (cid:107) Σ 1 / 2 T ( A a T − E [ A a T ]) (cid:107) 2 + E (cid:107) Σ 1 / 
-],a T
[Equation] [Equation]
Write A =
A 1 X (cid:62) diag( w )+ A 2 W (cid:62) , where X ∈ R n × d and W ∈ R n × ( n − d ) forms the orthogonal complement for the column span of diag( w ) X .
Therefore X (cid:62) diag( w ) W = 0 , and W (cid:62) W =
I n − d .
Also, notice E x i ∼ p S
[ X (cid:62) diag( w ) a T ] = n E x ∼ p T
[ x a T ( x )] = 0 .
Therefore plugging it in
R L ( B ) , we have:
[Equation]
We could view E [ W (cid:62) a T ] and W (cid:62) a T − E
[ W (cid:62) a T ] separately.
First notice at min-max point, if E [ W (cid:62) a T ] = 0 , the minimizer A 2 should be 0 since it only appears in the third and last nonnegative terms.
If E [ W (cid:62) a T ] (cid:54) = 0 , the cross term of the bias should be non-negative, or otherwise since both f ∗ and − f ∗ are in the set, a T , β ∗ T could be replaced by − a T , − β ∗ T and the loss increases.
Clearly in this case A 2 should also be 0 at min-max point.
A. O MITTED PROOF FOR UTILIZING SOURCE AND TARGET DATA JOINTLY
Sufﬁcient statistic
Proof of Claim 3.7.
Denote by ¯ β S :
= ˆΣ −
[Equation]
S X (cid:62) S y S /n S ∼ N ( β ∗ , σ 2 n S ˆΣ − 1 S ) and ¯ β T := ˆΣ − 1 T X (cid:62) T y T /n
T ∼ N ( β ∗ , σ 2 n T ˆΣ − 1 T ) .
We use the Fisher–Neyman factorization theorem to derive the sufﬁcient statistics.
The likelihood of observing ¯ β S , ¯ β T from parameter β ∗ is:
[Equation]
where g ( β ∗ , T ( β ∗ ))
=
e − ( β ∗ − ˆ β SS ) (cid:62) ( nSσ 2 ˆΣ S + nTσ 2 ˆΣ T ) − 1 ( β ∗ − ˆ β SS ) , and c is some constant.
Therefore it’s easy to see that T ( β ∗ )
=
ˆ β SS is the sufﬁcient statistic for β ∗ .
Proof of Claim 3.8.
With similar procedure as before, and notice z S and z T are independent, we could ﬁrst conclude that the optimal estimator is of the form ˆ β
=
A ˆΣ − 1 S X (cid:62) S y S /n
S
+
B ˆΣ − 1 T X (cid:62) T y T /n
T ∼ N (( A + B ) β ∗ , σ 2 n S
A ˆΣ − 1 S A (cid:62) + σ 2 n T B ˆΣ − 1 T B (cid:62) ) .
[Equation]
[Equation]
Take gradient w.r.t A and B respectively we have:
[Equation]
Notice the ﬁrst terms are equivalent.
Therefore 1 n S A ˆΣ − 1 S = 1 n T B ˆΣ − 1 T thus the optimal ˆ β is of the form C ( X (cid:62) S y S + X
(cid:62) T y T ) for some matrix C , thus ﬁnishing the proof.
B O MITTED P ROOF WITH M ODEL S HIFT
Deﬁnition B.1 (Orthosymmetry) .
A set Θ is said to be solid and orthosymmetric if θ ∈ Θ and | ζ i | ≤ | θ i | for all i implies that ζ ∈ Θ .
If a solid, orthosymmetric Θ contains a point τ , then it contains the entire hyperrectangle that τ deﬁnes: Θ( τ ) ≡ { θ ||
θ
i | ≤ τ i , ∀ i } ⊂ Θ .
Proof of Claim 4.1.
First notice for any estimator ˆ β , it all satisﬁes
[Equation]
The ﬁrst inequality is straightforward with the same reasoning of AM-GM as the derivation of (5).
As for the second inequality, we take a closer look at (5).
Notice that when max β ∗ T ∈B , δ ∈ ∆ is achieved, the cross term has to be non-negative, or otherwise one could ﬂip the sign of β ∗ T to make the value larger.
Therefore at maximum (cid:107) Σ 1 / 2 T (( A 1 + A 2 − I ) β ∗ T (cid:107) 2 + (cid:107) Σ 1 / 2 T A 1 δ (cid:107) 2 ≤ (cid:107) Σ 1 / 2 T ((
A 1 + A 2 − I )
β ∗ T + Σ 1 / 2 T A 1 δ (cid:107) 2 , and notice the remaining parts are all non-negative.
Therefore r B , ∆ ( ˆ β ) ≤ 2 L B , ∆ ( ˆ β ) .
Now let ˆ β ∗ =
arg min ˆ β
=
A 1 ¯ y S + A 2 ¯ y S L B , ∆ ( ˆ β ) .
We have:
[Equation]
The inequality (a) is by deﬁnition of ˆ β ∗ while (b) is from the deﬁnition of ˆ β MM .
B. L OWER B OUND WITH M ODEL S HIFT
In order to derive the lower bound, we abstract the problem to the following more general one:
Problem 1.
For arbitrary diagonal matrix D ∈ R d × d , two (cid:96) 2 -compact, solid, orthosymmetric, and quadratically convex sets Θ , ∆ ⊂ R d , let P Θ , ∆ ,D = (cid:26) N (cid:18)(cid:20) D θ + δ θ (cid:21) , (cid:20)
I
I (cid:21)(cid:19)(cid:12)(cid:12)(cid:12)
[Equation]
Under Let R L (Θ , ∆ , D ) and R N (Θ , ∆ , D ) be the minimax linear risk and minimax risk respectively for estimating θ within the distribution class P Θ , ∆ ,D :
[Equation]
Here r P (ˆ θ ) :=
E x ∼ P (cid:107) ˆ θ ( x ) − θ ( P ) (cid:107) 22 .
We want to derive a uniform lower bound for R N with
R L , i.e., R N ≥ µ ∗ R L , where µ ∗ is universal and doesn’t depend on the choices of D , Θ or ∆ .
Before proving the lower bound, we establish its connection to our considered problem:
Remark B.1.
Suppose Σ S =
U diag( s ) U (cid:62) and Σ T =
U diag( t ) U (cid:62) share the same eigenspace.
Recall our samples a ∼ N (Σ 1 / 2 S ( β ∗ T + δ ) , σ 2 I ) , b ∼ N (Σ 1 / 2 T β ∗ T , σ 2 I ) .
Our goal to uniformly lower bound R N ( r, γ ) by R L ( r, γ ) is essentially Problem 1, where
[Equation]
Proof of Remark B.1.
Our target considers samples drawn from distributions x ∼ N (Σ 1 / 2 S ( β ∗ T + δ ) , σ 2 I ) , y ∼ N (Σ 1 / 2 T β ∗ T , σ 2 I ) .
(cid:20) ab (cid:21) ∼ N (cid:18)(cid:20)
U diag( s 1 / 2 ) U (cid:62) ( β ∗ T + δ )
{+U diag( t 1 / 2 ) U (cid:62) β ∗ T (cid:21) , (cid:20) σ 2 I  σ 
+}
I
(cid:21)(cid:19) , θ ∈ Θ , δ ∈ ∆ ⇐⇒ (cid:20) U (cid:62) a /σ U (cid:62) b /σ (cid:21) ∼ N (cid:18)(cid:20) diag( s 1 / 2 ) U (cid:62) ( β ∗ T + δ ) diag( t 1 / 2 ) U (cid:62) β ∗ T (cid:21) , (cid:20)
I
I (cid:21)(cid:19) , (cid:107) β ∗ T (cid:107) ≤ r, (cid:107) δ (cid:107) ∈ γ
Let ¯ a = U (cid:62) a /σ,
¯ b = U (cid:62) b /σ, Θ = { θ |(cid:107) diag( t − 1 / 2 ) θ (cid:107) ≤ r } , ∆ = {(cid:107) diag( s − 1 / 2 ) δ (cid:107) ≤ γ } .
¯ θ =
U (cid:62) Σ 1 / 2 T β ∗ T , δ = U (cid:62) Σ 1 / 2 S δ , and D = diag( s 1 / 2 t − 1 / 2 ) .
We get: (cid:20) U
(cid:62) a /σ
U (cid:62) b /σ (cid:21) ∼ N (cid:18)(cid:20) diag( s 1 / 2 ) U (cid:62) ( β ∗ T + δ ) diag( t 1 / 2 ) U (cid:62) β ∗ T (cid:21) , (cid:20)
I
I (cid:21)(cid:19) , (cid:107) β T (cid:107) ≤ r, (cid:107) δ (cid:107) ∈
γ ⇐⇒ (cid:20) ¯ a ¯ b (cid:21) ∼ P θ , δ ,D :=
N (cid:18)(cid:20)
D ¯ θ + ¯ δ ¯ θ (cid:21) , (cid:20)
I
I (cid:21)(cid:19) , ¯ θ ∈ Θ , ¯ δ ∈ ∆ .
Let P Θ , ∆ ,D :=
(cid:8) P ¯ θ , ¯ δ ,D (cid:12)(cid:12) ¯ θ ∈ Θ , ¯ δ ∈ ∆ (cid:9) .
Since U is an invertible matrices, observing
U (cid:62) a /σ, U (cid:62) b /σ instead of a , b has no affect on the performance of the best estimator.
Also Θ , ∆ are axis-aligned ellipsoid and thus satisfy orthosymmetry.
Therefore our problem is essentially reduced to Problem 1.
Lemma B.2.
Let Θ( τ ) =
{ θ | θ i ≤ τ i , ∀ i, θ ∈ Θ } and similarly for ∆( ζ ) =
{ δ | δ i ≤ ζ i , δ ∈ ∆ } ,
D is some diagonal matrix.
[Equation]
Write samples drawn from some P θ , δ ,D ∈ P Θ , ∆ ,D as ( x , y ) :
x ∼ N ( D θ + δ , I ) , y ∼ N ( θ , I ) .
Lemma B.3.
The minimax linear estimator ˆ θ : ( x , y )
→ A x + B y has the form ˆ θ a , b ( x , y )
=
(cid:80) i a
i x i + (cid:80) i b i
y i for some a , b ∈ R d .
Namely,
[Equation]
Proof.
According to the proof of Proposition B.4.a, by discarding off-diagonal terms, the maximum risk of any linear estimator ˆ θ A,B over any hyperrectangles Θ( τ ) , ∆( ζ ) is reduced.
[Equation]
Further we have:
[Equation]
Therefore all four terms have to be equal, thus ﬁnishing the proof.
Notice Θ( τ ) and ∆( ζ ) are hyperrectangles in R d .
Therefore we could decompose the problem to some 2-d problems:
Proposition B.4.
Under the same setting as Problem 1,
[Equation]
If ˆ θ A,B ( x , y )
=
A x + B y is minimax linear estimator over P Θ( τ ) , ∆( ζ ) ,D , then necessarily A, B must be diagonal.
[Equation]
Proof of Proposition B.4.a .
First review our notation:
[Equation]
Our objective is
[Equation]
We will show that restricting A, B to be diagonal will not include the RHS value.
For any ¯ τ ∈ Θ( τ ) , ¯ ζ ∈ ∆( ζ ) , let set V (¯ τ , ¯ ζ )
= { ( θ , δ ) | ( θ i , δ i ) ∈ { (¯ τ i , ¯ ζ i ) , ( − ¯ τ i , − ¯ ζ i ) }} be the subset of vertices of Θ(¯ τ ) × ∆(¯ ζ ) .
Let π (¯ τ , ¯ ζ ) be uniform distribution on this ﬁnite set.
Due to the symmetry of this distribution, we have
[Equation]
We utilize the distribution to ﬁnd the explicit value of the maximum (in fact the maximum will only be obtained inside the vertices set V (¯ τ , ¯ ζ ) ):
[Equation]
) = Tr(( AD + B − I ) (cid:62) ( AD + B − I )diag(¯ τ 2 )) + Tr( A (cid:62) A diag(¯ ζ 2 ))
+ Tr(( AD + B − I ) (cid:62) A diag(¯ τ ¯ ζ )) + Tr( AA (cid:62) ) + Tr( BB (cid:62) ) = (cid:88)
i (cid:107) ( AD + B − I ) : ,i ¯ τ i + A : ,i ¯ ζ i (cid:107) 2 + Tr( AA (cid:62) ) + Tr( BB (cid:62) ) ≥ (cid:88) i (( A ii D ii + B ii {+− 1)¯ τ i + A ii ¯ ζ i ) 
+}
+ A 2 ii + B 2 ii = (cid:107) ( diag( A ) D + diag( B ) − I ) θ + diag( A ) δ (cid:107) {+
+}
+ Tr(diag( A ) 2 ) + Tr(diag( B ) 2 ) , ( ∀ ( θ , δ ) ∈ V (¯ τ , ¯ ζ ) )
=
max V (¯ τ , ¯ ζ ) (cid:107) (diag( A ) D + diag( B ) − I ) θ + diag( A ) δ (cid:107)
+ Tr(diag( A ) 2 )
+ Tr(diag( B ) 2 )
Therefore we have:
[Equation]
max
[Equation]
max
[Equation] = [Equation] Next, since the optimal solution on the minimizer is always obtained by diagonal A, B , it becomes straightforward that each axis could be viewed in separation, thus ﬁnishing the proof for part a.
The nonlinear part is a straightforward extension of Proposition 4.16 from Johnstone (2011).
Theorem B.5 (Restated Le Cam Two Point Theorem Wainwright (2019)) .
Let P be a family of distribution, and θ : P → Θ is some associated parameter.
Let ρ : Θ × Θ → R + be some metric deﬁned on Θ and Φ : R + → R + is a monotone non-decreasing function with Φ(0)
= 0 .
For any α ∈ (0 , 1) ,
[Equation]
Lemma B.6.
Consider a class of distribution P τ,ζ,s = { P θ,δ,s | P
θ,δ,s
[Equation]
= N ([ sθ + δ, θ ]
[Equation]
, I 2 ) , | θ | ≤ τ, | δ | ≤ ζ } .
Deﬁne
[Equation]
We have
R
[Equation]
Proof of Lemma B.6.
We ﬁrst calculate an upper bound of R L and connect it to a lower bound of
R N .
[Equation]
By some detailed calculations, we get the RHS is equal to:
[Equation]
For simplify this form, we could see that
Next, we use Le cam two point theorem to lower bound R N ( τ, ζ, s ) where the metric ρ is Euclidean distance and Φ is squared function.
Therefore
[Equation]
Since the total variation distance is related to Kullback-Leibler divergence by Pinsker’s inequality: (cid:107)· , [Equation]
[Equation]
) , it’s sufﬁcient to replace the constraint as:
[Equation]
[Equation]
Recall R L ≤ min { 1 , 2 τ 2 , 1+4 ζ s 2 +1 } .
We ﬁrst note that c 2 ≤ 4 τ 2 and setting α =
we have R N ≥ τ 2 / 2 ≥ 1 / 4 R L .
For In the following we look at other cases when the bound for c 2 is smaller.
When 2 ζ ≥ sc, will set d = − sc and c 2 = 2 α 2 .
Let α = 2 / 3 for large τ we get : c 2 (1 − α ) / 8 = 2 / 27 ≥ 2 / 27 R L .
When 2 ζ ≤ sc we set d = − 2 ζ and require ( sc − 2 ζ )
+ c 2 ≤ 2 α 2 .
We have ( sc − 2 ζ ) 2 + c 2 = s 2 c 2 + 4 ζ 2 − 4 ζsc + c 2 ≤ s 2 c 2 + 4 ζ 2 − 8 ζ
+
c
=
( s 2 + 1) c 2 − 4 ζ 2 .
Therefore as we set c 2 =
α 2 +4 ζ 2 s 2 +1 , the original inequality is satisﬁed.
Again by setting α = 2 / 3 we have c 2 ≥ 8 / 9 1+4 ζ 2 s 2 +1 ≥ 8 / 9 R L .
Therefore in this case R N ≥ 227 R L .
C M ORE EMPIRICAL RESULTS
We include some more empirical studies.
In the main text our results have small noise.
Here we show some more results with larger noise, and also the case with varied eigenspace.
For the following results, we use σ = 10 and r = 0 .
2 √ d .
Other meta data remains the same as presented in the main text.
Figure 2 (a)(b) show similar phenomenon as the small noise setting presented in (a) covariate eigen-spectrum
[Figure]
the main text.
From Figure 2 (c) we see no particular relationship between the performance of each algorithm with eigenspace shift.