Under review as a conference paper at ICLR
N EAR -O PTIMAL L INEAR R EGRESSION UNDER D ISTRIBUTION S HIFT
Anonymous authors
Paper under double-blind review
A BSTRACT
Transfer learning is an essential technique when sufﬁcient data comes from the source domain, while no or scarce data is from the target domain.
We develop estimators that achieve minimax linear risk for linear regression problems under the distribution shift.
Our algorithms cover different kinds of settings with covariate shift or model shift.
We also consider when data are generating from either linear or general nonlinear models.
We show that afﬁne minimax rules are within an absolute constant of the minimax risk even among nonlinear rules for various source/target distributions.
I NTRODUCTION
The success of machine learning crucially relies on the availability of labeled data.
The data labeling process usually requires much human labor and can be very expensive and time-consuming, especially for large datasets like ImageNet (Deng et al., 2009).
On the other hand, models trained on one dataset, despite performing well on test data from the same distribution they are trained on, are often sensitive to distribution shifts , i.e., they do not adapt well to related but different distributions.
Even small distributional shift can result in substantial performance degradation (Recht et al., 2018;
Lu et al., 2020).
Transfer learning has been an essential paradigm to tackle the challenges associated with insufﬁcient labeled data (Pan & Yang, 2009; Weiss et al., 2016; Long et al., 2017).
The main idea is to make use of a source domain with a lot of labeled data (e.g. ImageNet), and to try to learn a model that performs well on our target domain (e.g. medical images) where few or no labels are available.
Despite the lack of labeled data, we may still use unlabeled data from the target domain, which are usually much easier to obtain and can provide helpful information about the target domain.
Although this approach has been integral to many applications, many fundamental questions are left open even in very basic settings.
In this work, we focus on the setting of linear regression under distribution shift and ask the fundamental question of how to optimally learn a linear model for a target domain, using labeled data from a source domain and unlabeled data (and possibly some labeled data) from the target domain.
For various settings, including covariate shift (i.e., when p ( x ) changes) and model shift (i.e., when p ( y | x ) changes), we develop estimators that achieve near minimax risk (up to universal constant factors) among all linear estimation rules.
Here linear estimators refer to all estimators that depend linearly on the label vector; these include almost all popular estimators known in linear regression, such as ridge regression and its variants.
When the input covariances in source and target domains commute, we prove that our estimators achieve near minimax risk among all possible estimators.
Under review as a conference paper at ICLR
A key insight from our results is that, when covariate shift is present, we need to apply datadependent regularization that adapts to changes in the input distribution.
For linear regression, this can be given by the input covariances of source and target tasks, which can be estimated using unlabeled data.
Our experiments verify that our estimator has signiﬁcant improvement over ridge regression and similar heuristics.
1. R ELATED WORK
Different types of distribution shift are introduced in (Storkey, 2009; Quionero-Candela et al., 2009).
Speciﬁcally, covariate shift occurs when the marginal distribution on P ( X ) changes from source to target domain (Shimodaira, 2000; Huang et al., 2007).
Wang et al.
; Wang &
Schneider (2015) tackle model shift ( P ( Y | X ) ) provided the change is smooth as a function of X .
Sun et al.
(2011) design a two-stage reweighting method based on both covariate shift and model shift.
Other methods like the change of representation, adaptation through prior, and instance pruning are proposed in (Jiang & Zhai, 2007).
In this work, we focus on the above two kinds of distribution shift.
For modeling target shift ( P ( Y ) ) and conditional shift ( P ( X | Y ) ), Zhang et al.
(2013) exploits the beneﬁt of multi-layer adaptation by some location-scale transformation on X .
Transfer learning/domain adaptation are sub-ﬁelds within machine learning to cope with distribution shift.
A variety of prior work roughly falls into the following categories.
1) Importancereweighting is mostly used in the covariate shift.
(Shimodaira, 2000; Huang et al., 2007; Cortes et al., 2010); 2) One fruitful line of work focuses on exploring robust/causal features or domaininvariant representations through invariant risk minimization (Arjovsky et al., 2019), distributional robust minimization (Sagawa et al., 2019), human annotation (Srivastava et al., 2020), adversarial training (Long et al., 2017; Ganin et al., 2016), or by minimizing domain discrepancy measured by some distance metric (Pan et al., 2010; Long et al., 2013; Baktashmotlagh et al., 2013;
Gong et al., 2013; Zhang et al., 2013; Wang & Schneider, 2014) ; 3) Several approaches seek gradual domain adaptation (Gopalan et al., 2011;
Gong et al., 2012; Glorot et al., 2011; Kumar et al., 2020) through self-training or a gradual change in the training distribution.
Near minimax estimations are introduced in Donoho (1994) for linear regression problems with
Gaussian noise.
For a more general setting, Juditsky et al.
(2009) estimate the linear functional using convex programming.
Blaker (2000) compares ridge regression with a minimax linear estimator under weighted squared error.
Kalan et al.
(2020) considers a setting similar to this work of minimax estimator under distribution shift, but focuses on computing the lower bound for linear and one-hidden-layer neural network under distribution shift.
A few more interesting results are derived on the generalization lower bound for distribution shift under various settings (David et al., 2010; Hanneke & Kpotufe, 2019; Ben-David et al., 2010; Zhao et al., 2019).
P RELIMINARY
We formalize the setting considered in this paper for transfer learning under the distribution shift.
Notation and setup.
Let p S ( x ) and p T ( x ) be the marginal distribution for x in source and target domain.
The associated covariance matrices are Σ S , and Σ T .
We assume to have sufﬁcient unlabeled data to estimate Σ T accurately.
We observe n S , n T labeled samples from source and target domain.
Data is scarce in target domain: n S (cid:29) n T and n T can be 0 .
Speciﬁcally, X S =
[ x (cid:62) 1 | x (cid:62) 2 | · · · | x (cid:62) n S ] (cid:62) ∈ R n × d , with x i , i ∈ [ n S ] drawn from p S , noise z =
[ z 1 , z 2 , · · · z n S ] (cid:62) ,
z i ∼
N (0 , σ 2 ) .
y S =
[ y 1 , y 2 , · · · , y n S ] (cid:62) ∈ R n S , with each y i
= f ∗ ( x i )
+ z i ( X T ∈ R n T × d and
Under review as a conference paper at ICLRy T ∈ R n T are similarly deﬁned).
Denote by ˆΣ S = X (cid:62) S X S /n
S the empirical covariance matrix (Throughout the paper we assume data is centered: E p S [ x ] =
E p T [ x ] = 0 ).
The positive part of a number is denoted by ( x ) + .
We consider both linear ( f ∗ ( x ) =
x (cid:62) β ∗ ) and general nonlinear ground truth models.
When the optimal linear model changes from source to domain we add a subscript for distinction, i.e. , β ∗ S and β ∗ T .
We use bold ( x ) symbols for vectors, lower case letter ( x ) for scalars and capital letter ( A ) for matrices.
Minimax (linear) risk.
In this work, we focus on designing linear estimators ˆ β =
A y S 1 for parameter β ∗ ∈ B .
Our estimator is evaluated by the excess risk on target domain, with the worst case β ∗ in some set B : L B ( ˆ β ) = max β ∗ ∈B E y S (cid:107) Σ 1 / 2 T ( ˆ β ( y S ) − β ∗ ) (cid:107) 2 .
Minimax linear risk and minimax risk among all estimators are respectively deﬁned as:
[Equation]
The subscript “N" or “L" is a mnemonic for “non-linear" or “linear" estimators.
R N is the optimal risk with no restriction placed on the class of estimators.
R L only considers the linear function class for ˆ β .
Minimax linear estimator and minimax estimator are the estimators that respectively attain R L and R N within universal multiplicative constants.
Normally we only consider B = { β |(cid:107) β (cid:107) 2 ≤ r } .
When there is no ambiguity, we simplify ˆ β ( y S ) by ˆ β .
Our meta-algorithm.
Our paper considers different settings with distribution shift.
Our methods are uniﬁed under the following meta-algorithm:
Step 1: Find an unbiased sufﬁcient statistic ˆ β SS2 for the unknown parameter.
Step 2: Find ˆ β MM , a linear operator applied to ˆ β SS that minimizes L B ( ˆ β MM ) .
For each setting, we will show that ˆ β MM achieves linear minimax risk R L (asymptotically or in ﬁxed design).
Furthermore, under some conditions, the minimax risk R N is uniformly lower bounded by a universal constant times L B ( ˆ β MM ) .
Outline.
In the sections below, we tackle the problem in different settings.
In Section 3 we design algorithms with only covariate shift: 1) n T = 0 and f ∗ ( x ) is linear (Section 3.1); 2) n T =and f ∗ ( x ) is a general nonlinear function (Section 3.2); 3) n T >
and f ∗ is linear (Section 3.3).
Finally, we cope with the model shift for linear models ( β ∗ S (cid:54) =
β ∗ T ) in Section 4.
M INIMAX ESTIMATOR WITH COVARIATE SHIFT
In this section, we consider the setting with only covariate shift.
That is, only Σ S (marginal distribution p S ( x ) ) changes to Σ T ( p T ( x ) ), but f ∗ =
E [ y | x ] (conditional distribution p ( y | x ) ) is shared.
We ﬁrst consider the case when f ∗ is a linear map: x → x (cid:62) β ∗ and then consider the problem with approximation power.
Under review as a conference paper at ICLR
3. C OVARIATE SHIFT WITH LINEAR MODELS
We observe n S samples from source domain: y S =
X S β ∗ + z , z ∼ N (0 , σ 2 I ) and no labeled samples from the target domain.
Our goal is to ﬁnd the minimax linear estimator ˆ β MM ( y S ) =
A y S with some linear mapping A that attains R L ( B ) .
Following our meta-algorithm, let ˆ β SS = 1 n S ˆΣ − 1 S X (cid:62) S y S 3 be an unbiased sufﬁcient statistic for β ∗ :
[Equation]
The fact that ˆ β S S ( y S ) is a sufﬁcient statistic is proven in Claim 3.7 for a more general case, using the Fisher-Neyman factorization theorem.
Here we consider X S as ﬁxed values, and randomness only comes from noise z .
We prove that the minimax linear estimator is of the form ˆ β MM =
C ˆ β SS and then design algorithms that calculate the optimal C .
Claim 3.1.
The minimax linear estimator is of the form ˆ β MM =
C ˆ β SS for some C ∈ R d × d .
Warm-up: commutative covariance matrices.
In order to derive the minimax linear estimator, we ﬁrst consider the simple case when Σ T and ˆΣ S are simultaneously diagonalizable.
We apply
Pinsker’s Theorem (Johnstone, 2011) and get:
Theorem 3.2 (Linear Minimax Risk with Covariate Shift) .
Suppose the observations follow sequence model y S
=
X S β ∗ + z , z ∼ N (0 , σ 2 I n ) .
If Σ T = U diag( t ) U (cid:62) and ˆΣ S ≡ X (cid:62) S X S /n S =
U diag( s ) U (cid:62) , then the minimax linear risk
[Equation]
where B = { β |(cid:107) β (cid:107) ≤ r } , and λ = λ ( r ) is determined by σ 2 n S (cid:80) di
=1 1 s i ( √ t i /λ − 1) + = r 2 .
The linear minimax estimator is given by:
[Equation]
Since r is unknown in practice, we could simply view either r or directly λ as the tuning parameter.
We compare the functionality of λ with that of ridge regression: ˆ β λ RR = arg min ˆ β E 12 n (cid:107) X S
ˆ β − y S (cid:107)
+ λ 2 (cid:107) ˆ β (cid:107) 2 =
(ˆΣ S + λI ) − 1 X (cid:62) S y S /n S .
For both algorithms, λ is to balance the bias and variance: λ = 0 gives an unbiased estimator, and a big λ gives a (near) zero estimator with no variance.
The difference is, our estimator shrinks some signal directions based on the value of t i .
The estimator tends to sacriﬁce the directions of signal where t i is smaller.
Ridge regression, however, respects the value of s i .
A natural counterpart is for ridge to also regularize based on t : let ˆ β λ RR ,T = arg min 1 n (cid:107) Σ 1 / 2 T ( β − ˆΣ − 1 S X (cid:62) S y S ) (cid:107)
+ λ (cid:107) β (cid:107) 2 =
(Σ T + λI ) − 1 Σ T ˆ β SS .
We will compare their performances in the experimental section.
Under review as a conference paper at ICLR
Non-commutative covariance matrices.
For non-commutative covariate shift, we follow the same procedure.
Our estimator is achieved by optimizing over C :
ˆ β MM =
C ˆ β SS :
[Equation]
Unlike the commutative case, this problem doesn’t have a closed form solution, but is still solvable:
Proposition 3.3.
Problem (3) is a convex program and thus solvable.
We achieve near-optimal minimax risk among all estimators under some conditions:
Theorem 3.4 (Near minimaxity of linear estimators) .
When Σ S , Σ T commute, or Σ T is rank 1, the best linear estimator from (2) or (3) achieves near-optimal minimax risk: L B ( ˆ β MM ) =
R L ( B ) ≤ 1 .
25 R N ( B ) .
Note that R N ≤ R L by deﬁnition.
Therefore 1) our estimator ˆ β MM is near-optimal, and 2) our lower bound for R N is tight.
Lower bounds (without matching upper bounds) for general noncommutative problem is presented in (Kalan et al., 2020) and we improve their result for the commutative case and provide a matching algorithm.
Their lower bound scales with dn S min i t i s i for large r , while ours becomes 1 n S (cid:80) i t
i s i .
Our lower bound is always larger and thus tighter, and potentially arbitrarily larger when max i t i s i and min i t
i s i are very different.
We defer our proof to the appendix.
Remark 3.1 (Beneﬁt of minimax linear estimator) .
Consider estimators from ridge regression: ˆ β λ RR = arg min ˆ β E 12 n (cid:107) X S
ˆ β − y S (cid:107)
+ λ 2 (cid:107) ˆ β (cid:107) 2 .
There is an example that R L ( B ) ≤
O ( d − 1 / 4 L B ( ˆ β λ RR )) even with the optimal hyperparameter λ .
Remark 3.2 (Incorporating the randomness of source and target features) .
For clean presentation purposes, in the main text we assume to have access to Σ T .
In practice, we will need to estimate Σ T by ﬁnite unlabeled samples from target domain.
In Appendix C.1 we show that our estimator remains near-optimal if we have (cid:29) d unlabeled target samples under some standard light-tail assumptions.
Theorem3.4is comparing our estimator with the optimal nonlinear estimator using the same data
X S from the source domain.
In appendix C we compare our estimator with a stronger notion of linear estimator with inﬁnite access to p S and show that our estimator is still within multiplicative factor of it.
3. L INEAR MINIMAX ESTIMATOR WITH APPROXIMATION ERROR
Now we consider observations coming from nonlinear models: y S = f ∗ ( X S )
+ z .
Let β ∗ S =
arg min
β E x ∼ p S ,z ∼N (0 ,σ 2 ) [( f ∗ ( x ) + z − β (cid:62) x ) 2 ] , and similarly for β ∗ T .
Notice now even with f ∗ unchanged across domains, the input distribution affects the best linear model.
Approximation error is a S ( x ) =
f ∗ ( x )
− x (cid:62) β ∗ S and vice versa for a T .
Under
Deﬁne the reweighting vector w ∈ R n as w i = p T ( x i ) /p
S ( x i ) .
We form unbiased estimator via
[Equation]
Claim 3.5.
ˆ β LS is asymptotically unbiased and normally distributed:
[Equation]
Denote by m ( x ) = a T ( x ) + z .
We want to minimize the worst case risk:
[Equation]
Therefore our estimator is ˆ β MM ← ˆ C ˆ β LS , where ˆ C ﬁnds ˆ C ← arg min τ,C
[Equation]
Claim 3.6.
Let B = { β |(cid:107) β (cid:107) ≤ r } , and f ∗ ∈ F is some compact symmetric function class: f ∈ F ⇔ − f ∈ F .
Then linear minimax estimator is of the form C ˆ β LS for some C .
When ˆ C solves Eqn.
(4) , L B ( ˆ β MM ) asymptotically matches R L ( B ) , the linear minimax risk.
By reducing from y S to ˆ β LS we eliminate n − d dimensions, and this claim says that X (cid:62) S y S is sufﬁcient to predict β ∗ T .
We note that f ∗ is more general than a linear function and therefore the lower bound could only be larger than R N ( B ) deﬁned in the previous section.
3. U TILIZE SOURCE AND TARGET LABELED DATA JOINTLY
In some scenarios we have moderate amount of labeled data from target domain as well.
Then it is important to utilize the source and target labeled data jointly.
Let y S
=
X S β ∗ + z S , y T =
X T β ∗ + z T .
We consider X S , X T as deterministic variables, ˆΣ − 1 S X (cid:62) S y S /n S ∼ N ( β ∗ , σ 2 n S ˆΣ − 1 S ) and ˆΣ − 1 T X (cid:62) T y T /n
T ∼ N ( β ∗ , σ 2 n T ˆΣ − 1 T ) .
Therefore conditioned on the observations y S , y T , a sufﬁcient statistic for β ∗ is ˆ β SS :=
( n S ˆ Σ S + n T ˆ Σ T ) − 1 ( X (cid:62) S y S + X
(cid:62) T y T ) .
Claim 3.7.
ˆ β SS is an unbiased sufﬁcient statistic of β ∗ with samples y S , y T .
ˆ β SS ∼
N ( β ∗ , σ 2 ( n S ˆΣ S + n T ˆΣ T ) − 1 ) .
Algorithm: First consider the estimator ˆ β SS =
( n S ˆΣ S + n T ˆΣ T ) − 1 ( X (cid:62) S y S + X (cid:62) T y T ) .
Next ﬁnd the best linear function of ˆ β SS :
[Equation]
Proposition 3.8.
The minimax estimator ˆ β MM is of the form C ˆ β SS for some C .
When choosing C with our proposed algorithm and when ˆΣ S commutes with ˆΣ T and Σ T , we achieve the minimax risk R L ( B ) ≤ 1 . 25 R N ( B ) .
Under review as a conference paper at ICLR
N EAR MINIMAX ESTIMATOR WITH MODEL SHIFT
The general setting of transfer learning in linear regression involves both model shift and covariate shift.
Namely, the generative model of the labels might be different: y S =
X S β ∗ S + z S , and y T
=
X T β ∗ T + z T .
Denote by δ := β ∗ S − β ∗ T as the model shift.
We are interested in the minimax linear estimator when (cid:107) δ (cid:107) ≤ γ and (cid:107)
β ∗ T (cid:107) ≤ r .
Thus our problem becomes to ﬁnd minimax estimator for β ∗ T ∈ B = { β |(cid:107) β (cid:107) ≤ r } from y S , y T .
Algorithm:
First consider a sufﬁcient statistic ( ¯ β S , ¯ β T ) for ( β ∗ T , δ ) .
Here ¯ β S = ˆΣ − 1 S X (cid:62) S y S /n
S ∼ N ( β ∗ T + δ , σ 2 n S ˆΣ − 1 S ) , and ¯ β T =
ˆΣ − 1 T X (cid:62) T y T /n
T ∼ N ( β ∗ T , σ 2 n T ˆΣ − 1 T ) .
Then consider the best linear estimator on top of it: ˆ β
=
A 1 ¯ β S + A 2 ¯ β T .
Write ∆ = { δ |(cid:107) δ (cid:107) ≤ γ } and
L
[Equation]
Therefore we optimize over this upper bound and reformulate the problem as a convex program:
[Equation]
Our estimator is given by: ˆ β MM =
ˆ A 1 ¯ β S
[Equation]
ˆ A 2 ¯ β T .
Since ˆ β MM is a relaxation of the linear minimax estimator
[Equation]
it is important to understand how well ˆ β MM performs on the original objective:
Claim
[Equation]
Finally we show with the relaxation we still achieve a near-optimal estimator even among all nonlinear rules.
Theorem 4.2.
When Σ T commutes with ˆΣ S , it satisﬁes:
[Equation]
Here R N ( B , ∆) :
= min ˆ β ( y S , y T ) max β ∗ T ∈B , δ ∈ ∆ (cid:107) Σ 1 / 2 T ( ˆ β − β ∗ T ) (cid:107) is the minimax risk.
Proof sketch of Theorem 4.2.
For the ease of understanding, we provide a simple proof sketch when Σ S = Σ T are diagonal.
We ﬁrst deﬁne the hardest hyperrectangular subproblem.
Let
B ( τ ) =
{ b : | β i | ≤ τ i } be a subset of B and similarly for ∆( ζ ) .
We show that R L ( B , ∆) =
Under a conference paper at
max τ ∈B , ζ ∈ ∆ R L ( B ( τ ) , ∆( ζ )) , and clearly R N ( B , ∆) ≥ max τ ∈B , ζ ∈ ∆ R N ( B ( τ ) , ∆( ζ )) .
Meanwhile we show when the sets are hyperrectangles the minimax (linear) risk could be decomposed to 1-d problems: R L ( B ( τ ) , ∆( ζ ))
= (cid:80) i R L ( τ i , ζ i ) .
Each R L ( τ i , ζ i ) is the linear minimax risk to estimate β i from x ∼ N ( β i
+ δ i , 1) and y ∼ N ( β i , 1) where | β i | ≤ τ i and | δ i | ≤ ζ i .
This 1-d problem for linear risk has a closed form solution, and the minimax risk can be lower bounded using Le Cam’s two point lemma.
We show R L ( τ i , ζ i ) ≤ 13 .
5 R N ( τ i , ζ i ) and therefore:
[Figure]
is the model shift measured by γ/r .
Performance with standard error bar is from 40 runs.
E XPERIMENTS
Our estimators are provably near optimal for the worst case β ∗ .
However, it remains unknown whether on average they outperform other baselines.
With synthetic data we explore the performances with random β ∗ .
We are also interested to investigate the conditions when we win more.
Setup.
We set n S = 2000 , d = 50 , σ = 1 , r = √ d .
For each setting, we sample β ∗ T from standard normal distribution and rescale it to be norm r .
We assume to know Σ T .
We compare our estimator with ridge regression (S-ridge) and a variant of ridge regression transformed to target domain
[Equation]
Covariate shift.
In order to understand the effect of covariate shift on our algorithm, we consider three types of settings, each with a unique varying factor that inﬂuences the performance: 1) covariate eigenvalue shift with shared eigenspace; 2) covariate eigenspace shift with ﬁxed eigenvalues 5 ;
Under review as a conference paper at ICLR3) signal strength change.
We also have an additional 200 labeled data from target domain as validation set only for hyper-parameter tuning.
Model shift.
Next we consider the problem with model shift.
We sample a random δ with norm γ varying from 0 to r = √ d and observe data generated by y S =
X S ( β ∗ T + δ ) + z S ∈ R 2000 , z S ∼
N (0 , I ) and y T =
X T β ∗ T + z T ∈ R 500 , z T ∼ N (0 , I ) .
We compare our estimator with two baselines: "ridge-source" denotes ridge regression using only source data, and "ridge-target" is from ridge regression with target data.
Figure 1 demonstrates the better performance of our estimator in all circumstances.
From (a) we see that with more discrepancy between Σ S and Σ T , our estimator tends to perform better.
(b) shows our estimator is better when the signal is relatively stronger.
From (c) we can see that with the increasing model shift measured by γ/r , ridge-source becomes worse and is outperformed by ridge-target that remains unchanged.
Our estimator becomes slightly worse as well due to the less utility from source data, but remains the best among others.
When γ/r ≈ 0 . 2 , our method has the most improvement in percentage compared to the best result among ridge-source and ridge-target.
A O MITTED PROOF FOR MINIMAX ESTIMATOR WITH COVARIATE SHIFT A.
P INSKER
’ S T HEOREM AND COVARIATE SHIFT WITH LINEAR MODEL
Theorem A.1 (Pinsker’s Theorem) .
Suppose the obervations follow sequence model
y i
=
θ ∗ i + (cid:15)
i z i , (cid:15) i > 0 , i ∈
[ d ] , and Θ is an ellipsoid in R d : Θ = Θ( a, C ) =
{ θ : (cid:80) i
a 2 i θ 2 i ≤ C 2 } .
Then the minimax linear risk
[Equation]
where µ = µ ( C ) is determined by
[Equation]
The linear minimax estimator is given by
[Equation]
and is Bayes for a Gaussian prior π C having independent components θ i ∼ N (0 , τ 2 i ) with τ ∗ i = (cid:15)
i ( µ/a i − 1) + .
Our theorem 3.2 is to connect our parameter β ∗ to the θ ∗ in pinsker’s theorem.
First we show that reformulating the problem from a linear map of n dimensional observations y S to a linear map on the d -dimensional statistic ˆ β SS is sufﬁcient, i.e., Claim 3.1:
Proof of Claim 3.1.
This is to show that if ˆ β ( y S ) :
=
A y S is a minimax linear estimator, each row vector of A ∈ R d × n is in the column span of X S .
Write A
=
A 1 X (cid:62) S + A 2 W (cid:62) where
W ∈ R n × ( n − d ) , columns of which forms the orthonormal complement for the column space of
X S .
Equivalently we want to show A 2 = 0 .
We have
[Equation]
Since W
[Equation]
(Other cross terms vanish since E [ z ] = 0 )
[Equation]
where the last equation is because
[Equation] [Equation]
Clearly, at min-max point, without loss of generality we can take A 2 = 0 .
Formally the proof for Theorem 3.2 is presented here:
Proof of Theorem 3.2.
To use Pinsker’s theorem to prove Theorem 3.2, we simply need to transform the problem match its setting.
Let y T
=
Σ 1 / 2 T ˆΣ − 1 S X (cid:62) S y S /n S
= θ ∗ T + z T , where θ ∗ T =
U (cid:62) Σ 1 / 2 T β ∗ and z T ∼ N (0 , σ 2 diag([ t i /s i ] di =1 ) /n S ) .
The set for θ ∗ T is Θ = { θ |(cid:107)
Σ − 1 / 2 T
U θ (cid:107) ≤ r } , i.e., Θ = { θ | (cid:80) i θ
i /t i ≤ r 2 } .
Now with Pinsker’s theorem, ˆ θ ( y T ) i =
(1 − 1 / ( µ √ t i ))
+ ( y T )
i is the best linear estimator for θ ∗ T , where µ = µ ( r ) solves
[Equation]
Connecting to the original problem, we get that the best estimator for Σ 1 / 2 T β ∗ is U ( I − 1 µ diag([1 / √ t i ] di =1 ))
y T =
U ( I − 1 µ diag([1 / √ t i ] di =1 ))
U (cid:62) Σ 1 / 2 T Σ − 1 S X (cid:62) S y S /n S .
A
[Equation]
O MITTED PROOF FOR NONCOMMUTE COVARIANCE MATRICES Convex program.
Our estimator for β ∗ can be achieved through convex programming:
Proof of Proposition 3.3.
First note the objective function is quadratic in C and linear in τ , therefore we only need to prove the constraint S
= { ( C, τ ) | ( C − I ) (cid:62) Σ T ( C − I ) (cid:22) τI } is a convex set.
Notice for ( C 1 , τ 1 ) , ( C 2 , τ 2 ) ∈ S , i.e., ( C i − I ) (cid:62) Σ T ( C i − I ) (cid:22)
τ i
I, i ∈ { 1 , 2 } .
We simply need to prove for C α :=
αC 1 + (1 − α ) C 2 , τ α :=
τ 1 α + τ 2 (1 − α ) , ( C α − I ) (cid:62) Σ T ( C α − I ) (cid:22) τ α
I for any α ∈ [0 , 1] .
First, notice ( C 1 − C 2 ) (cid:62) Σ T ( C 1 − C 2 ) (cid:23) 0 .
Next, ( C α − I ) (cid:62) Σ T ( C α − I )
=
α ( C 1 − I ) (cid:62) Σ T ( C 1 − I )
+ (1 − α )
( C 2 − I ) (cid:62) Σ T ( C 2 − I ) − α (1 − α )
( C 1 − C 2 ) (cid:62) Σ T ( C 1 − C 2 ) (cid:22) α ( C 1 − I ) (cid:62) Σ T ( C 1 − I )
+ (1 − α )
( C 2 − I ) (cid:62) Σ T ( C 2 − I ) (cid:22) τ α I.
Beneﬁt of our estimator.
Compared to ridge regression, our estimator could possibly achieve much better ( d − 1 / 4 ) improvements:
Proof of Remark 3.1.
We consider diagonal covariance matrices ˆΣ S = diag( s ) , Σ T = diag( t ) , σ = 1 .
First we calculate the expected risk obtained with ridge regression: ˆ β λ RR =
( X (cid:62) S X S /n +
[Equation]
Compared to our risk:
[Equation]
where 1 n (cid:80) di =1 √ t i s i ( µ − 1 √ t i )
+ =
r 2 .
Let r 2 = √ dn S , s i
= 1 , ∀ i, t i
= 1 , ∀ i ∈
[ d 0 ] , t i =
d − 1 / 2 , d 0 < i ≤ d, where d 0 = √
d d 1 / 4 − 1 ≈ d 1 / 4 .
Then µ = 1 , and R L ( B ) =
d 1 / 4 n .
In this case,
[Equation]
Therefore min λ L B ( ˆ β λ RR ) ≥ d 1 / 4 R L ( B ) / 2 .
Near minimax risk.
Even among all nonlinear estimators, our estimator is within 1.25 of the minimax risk:
Proof of Theorem 3.4.
First we note that for both linear and nonlinear estimators, it is sufﬁcient to use ˆ β SS instead of the original observations y S .
See Lemma A.2 and its corollary.
Therefore it sufﬁces to do the following reformulations of the problem.
When Σ S and Σ T commute, we formulate the problem as the following Gaussian sequence model.
Recall ˆ Σ S =
U diag( s ) U (cid:62) , Σ T =
U diag( t ) U (cid:62) .
Let θ ∗ =
U (cid:62) Σ 1 / 2 T β ∗ , and y
=
U (cid:62) Σ 1 / 2 T ˆ β SS ∼
N ( θ ∗ , σ 2 n S diag( t / s )) .
Our objective of minimizing (cid:107) Σ 1 / 2 T ( ˆ β ( y S ) − ˆ β ∗ ) (cid:107) from linear estimator is equivalent to minimizing (cid:107) U
(ˆ θ ( y ) − ˆ θ ∗ ) (cid:107) =
(cid:107) ˆ θ ( y )
− ˆ θ ∗ (cid:107) from linear estimator.
The set for the parameter that satisﬁes θ ∗ =
U (cid:62) Σ 1 / 2 T β ∗ , (cid:107) β ∗ (cid:107) ≤ r is equivalent to (cid:107) Σ − 1 / 2 T
U θ ∗ (cid:107) ≤ r ⇔ (cid:107) θ ∗ i / √ t i (cid:107) ≤ r is an axis-aligned ellipsoid.
Then we could directly derive our result from
Corollary 4.26 from Johnstone (2011).
Note that this result is a special case of Theorem 4.2 and we have provided a detailed proof in Section
B. Therefore here we save further descriptions.
For the case when Σ T = aa (cid:62) is rank-1, the objective function becomes:
[Equation]
Under review as a conference paper at ICLR
Then the result could be derived from Corollary 1 of Donoho (1994), which reformulate the problem to the hardest one-dimensional problem which becomes tractable.
In the proof above, we equate the best nonlinear estimator on y S as the best nonlinear estimator on ˆ β SS .
The reasoning is as follows:
Lemma A.2 (Sufﬁcient statistic is enough to achieve a best estimator) .
Consider the statistical problem of estimating β ∗ ∈ B from observations y ∈ Y .
B (cid:96) 2 -compact.
If S ( y ) is a sufﬁcient statistic of β ∗ , then the best estimator that achieves min ˆ β max B (cid:96) ( ˆ β , β ∗ ) is of the form ˆ β = f
( S ( y )) with some function f , for any loss (cid:96) :
Y → [0 , ∞ ) .
This Lemma is restated from Proposition 3.13 from Johnstone (2011).
Corollary A.3 (Corollary of Lemma A.2) .
Under the same setting of Lemma A.2, R N ( B ) is achieved with the form ˆ β = f ( S ( y )) .
A. O MITTED PROOF WITH APPROXIMATION ERROR
Unbiased estimator for ˆ β ∗ T .
Proof of Claim 3.5.
[Equation]
=
[Equation]
z ))
Notice E x ∼ p S
[ x a T ( x ) p T ( x ) p S ( x ) ]
=
E x ∼ p T [ x a T ( x )] = 0 .
This is due to the KKT condition for the minimizer of l ( β ) :
= E x ∼ p T (cid:107) f ∗ ( x ) − β (cid:62) x (cid:107) 2 at β ∗ T : ∇ β f ( β ∗ ) =
→ E x ∼ p T [ x ( f ∗ − x (cid:62) β ∗ T )]
= 0 , i.e., E x ∼ p T
[ x a T ( x )] = 0 .
Next we have: E x i ∼ p S
[ X (cid:62) S diag( w ) X S ] =
E x i ∼ p S (cid:80) ni
=1 p T ( x i ) p S ( x i ) x
i x (cid:62) i =
E x j ∼ p T (cid:80) nj =
[ x j x (cid:62) j ] = n S Σ T .
Therefore
[Equation]
Proof of Claim 3.6.
Recall X S =
[ x (cid:62) 1 | x (cid:62) 2 | · · · | x (cid:62) n ] (cid:62) ∈ R n × d , with x i , ∀ i ∈ [ n ] drawn from p S , and a T = [ a T ( x 1 ) , a T ( x 2 ) , · · · a T ( x n )] (cid:62) ∈ R n , y =
[ y ( x 1 ) , y ( x 2 ) , · · · , y ( x n )] (cid:62) ∈ R n , noise z = y − f ∗ ( X ) .
w =
[ p T ( x i ) /p S ( x i )] (cid:62) .
To prove the, we only need to show the minimax linear estimator A y is achieved of the form
A 1 X (cid:62) diag( w ) , i.e., the row span of A is in the row span of X (cid:62) diag( w ) .
R L ( B
[Equation] [Equation]
Write A =
A 1 X (cid:62) diag( w )+ A 2 W (cid:62) , where X ∈ R n × d and W ∈ R n × ( n − d ) forms the orthogonal complement for the column span of diag( w ) X .
Therefore X (cid:62) diag( w ) W = 0 , and W (cid:62) W =
I n − d .
Also, notice E x i ∼ p S
[ X (cid:62) diag( w ) a T ] = n E x ∼ p T
[ x a T ( x )] = 0 .
Therefore plugging it in
R L ( B ) , we have:
[Equation]
We could view E [ W (cid:62) a T ] and W (cid:62) a T − E
[ W (cid:62) a T ] separately.
First notice at min-max point, if E [ W (cid:62) a T ] = 0 , the minimizer A 2 should be 0 since it only appears in the third and last nonnegative terms.
If E [ W (cid:62) a T ] (cid:54) = 0 , the cross term of the bias should be non-negative, or otherwise since both f ∗ and − f ∗ are in the set, a T , β ∗ T could be replaced by − a T , − β ∗ T and the loss increases.
Clearly in this case A 2 should also be 0 at min-max point.
A.
O MITTED PROOF FOR UTILIZING SOURCE AND TARGET DATA JOINTLY
Sufﬁcient statistic.
Proof of Claim 3.7.
Denote by ¯ β S
[Equation]
and ¯ β T := ˆΣ − 1 T X (cid:62) T y T /n
T ∼ N ( β ∗ , σ 2 n T ˆΣ − 1 T ) .
We use the Fisher–Neyman factorization theorem to derive the sufﬁcient statistics.
The likelihood of observing ¯ β S , ¯ β T from parameter β ∗ is:
[Equation]
where g ( β ∗ , T ( β ∗ ))
=
e − ( β ∗ − ˆ β SS )
[Equation]
( nSσ 2 ˆΣ S + nTσ 2 ˆΣ T ) − 1 ( β ∗ − ˆ β SS ) , and c is some constant.
Therefore it’s easy to see that T ( β ∗ )
=
ˆ β SS is the sufﬁcient statistic for β ∗ .
Proof of Claim 3.8.
With similar procedure as before, and notice z S and z T are independent, we could ﬁrst conclude that the optimal estimator is of the form ˆ β
=
A ˆΣ − 1 S X (cid:62) S y S /n
S
+
B
[Equation]
+
[Equation]
B ˆΣ
[Equation]
B
[Equation] [Equation]
[Equation]
Take gradient w.r.t A and B respectively we have:
[Equation]
Notice the ﬁrst terms are equivalent.
Therefore 1 n S A ˆΣ − 1 S = 1 n T B ˆΣ − 1 T thus the optimal ˆ β is of the form C ( X (cid:62) S y S + X
(cid:62) T y T ) for some matrix C , thus ﬁnishing the proof.
B O MITTED P ROOF WITH M ODEL S HIFT
Deﬁnition B.1 (Orthosymmetry) .
A set Θ is said to be solid and orthosymmetric if θ ∈ Θ and | ζ i | ≤ | θ i | for all i implies that ζ ∈ Θ .
If a solid, orthosymmetric Θ contains a point τ , then it contains the entire hyperrectangle that τ deﬁnes: Θ( τ ) ≡ { θ ||
θ
i | ≤ τ i , ∀ i } ⊂ Θ .
Proof of Claim 4.1.
First notice for any estimator ˆ β , it all satisﬁes
[Equation]
The ﬁrst inequality is straightforward with the same reasoning of AM-GM as the derivation of (5).
As for the second inequality, we take a closer look at (5).
Notice that when max β ∗ T ∈B , δ ∈ ∆ is achieved, the cross term has to be non-negative, or otherwise one could ﬂip the sign of β ∗ T to make the value larger.
Therefore at maximum (cid:107) Σ 1 / 2 T (( A 1 + A 2 − I ) β ∗ T (cid:107) 2 + (cid:107) Σ 1 / 2 T A 1 δ (cid:107) 2 ≤ (cid:107) Σ 1 / 2 T ((
A 1 + A 2 − I )
β ∗ T + Σ 1 / 2 T A 1 δ (cid:107) 2 , and notice the remaining parts are all non-negative.
Therefore r B , ∆ ( ˆ β ) ≤ 2 L B , ∆ ( ˆ β ) .
Now let ˆ β ∗ =
arg min ˆ β
=
A 1 ¯ y S + A 2 ¯ y S L B , ∆ ( ˆ β ) .
We have:
[Equation]
The inequality (a) is by deﬁnition of ˆ β ∗ while (b) is from the deﬁnition of ˆ β MM .
B. L OWER B OUND WITH M ODEL S HIFT
In order to derive the lower bound, we abstract the problem to the following more general one:
Under
Problem 1.
For arbitrary diagonal matrix D ∈ R d × d , two (cid:96) 2 -compact, solid, orthosymmetric, and quadratically convex sets Θ , ∆ ⊂ R d , let
[Equation]
Let R L (Θ , ∆ , D ) and R N (Θ , ∆ , D ) be the minimax linear risk and minimax risk respectively for estimating θ within the distribution class P Θ ,
[Equation]
[Equation]
Here r P ( ˆ θ ) :=
E x ∼ P (cid:107) ˆ θ ( x ) − θ ( P ) (cid:107) 2 2 .
We want to derive a uniform lower bound for R N with
R L , i.e., R N ≥ µ ∗ R L , where µ ∗ is universal and doesn’t depend on the choices of D , Θ or ∆ .
Before proving the lower bound, we establish its connection to our considered problem:
Remark B.1.
Suppose Σ S =
U diag( s ) U (cid:62) and Σ T =
U diag( t ) U (cid:62) share the same eigenspace.
Recall our samples a ∼ N (Σ 1 / 2 S ( β ∗ T + δ ) , σ 2 I ) , b ∼ N (Σ 1 / 2 T β ∗ T , σ 2 I ) .
Our goal to uniformly lower bound R N ( r, γ ) by R L ( r, γ ) is essentially Problem 1, where
[Equation]
Proof of Remark B.1.
Our target considers samples drawn from distributions x ∼ N (Σ 1 / 2 S ( β ∗ T + δ ) , σ 2 I ) , y ∼ N (Σ 1 / 2 T β ∗ T , σ 2 I ) .
(cid:20) ab (cid:21) ∼ N (cid:18)(cid:20)
U diag( s 1 / 2 ) U (cid:62) ( β ∗ T + δ )
U diag( t 1 / 2 ) U (cid:62) β ∗ T (cid:21) , (cid:20) σ 2 I  σ
I
(cid:21)(cid:19) , θ ∈ Θ ,
δ ∈ ∆ ⇐⇒ (cid:20) U (cid:62) a /σ
U (cid:62) b /σ (cid:21) ∼ N (cid:18)(cid:20) diag( s 1 / 2 ) U (cid:62) ( β ∗ T + δ ) diag( t 1 / 2 ) U (cid:62) β ∗ T (cid:21) , (cid:20)
I
I (cid:21)(cid:19) , (cid:107) β ∗ T (cid:107) ≤ r, (cid:107) δ (cid:107) ∈ γ
Let ¯ a = U (cid:62) a /σ,
¯ b = U (cid:62) b /σ, Θ = { θ |(cid:107) diag( t − 1 / 2 ) θ (cid:107) ≤ r } , ∆ = {(cid:107) diag( s − 1 / 2 ) δ (cid:107) ≤ γ } .
¯ θ =
U (cid:62) Σ 1 / 2 T β ∗ T , δ = U (cid:62) Σ 1 / 2 S δ , and D = diag( s 1 / 2 t − 1 / 2 ) .
We get: (cid:20) U
(cid:62) a /σ
U (cid:62) b /σ (cid:21) ∼ N (cid:18)(cid:20) diag( s 1 / 2 ) U (cid:62) ( β ∗ T + δ ) diag( t 1 / 2 ) U (cid:62) β ∗ T (cid:21) , (cid:20)
I
I (cid:21)(cid:19) , (cid:107) β T (cid:107) ≤ r, (cid:107) δ (cid:107) ∈ γ ⇐⇒
[Equation]
(cid:18)(cid:20)
[Equation]
(cid:20)
I
I (cid:21)(cid:19) , ¯ θ ∈ Θ , ¯ δ ∈ ∆ .
Let P Θ , ∆ ,D :=
(cid:8) P ¯ θ , ¯ δ ,D (cid:12)(cid:12) ¯ θ ∈ Θ , ¯ δ ∈ ∆ (cid:9) .
Since U is an invertible matrices, observing
U (cid:62) a /σ, U (cid:62) b /σ instead of a , b has no affect on the performance of the best estimator.
Also Θ , ∆ are axis-aligned ellipsoid and thus satisfy orthosymmetry.
Therefore our problem is essentially reduced to Problem 1.
Lemma B.2.
Let Θ( τ ) =
{ θ | θ i ≤ τ i , ∀ i, θ ∈ Θ } and similarly for ∆( ζ ) =
{ δ | δ i ≤ ζ i , δ ∈ ∆ } ,
D is some diagonal matrix.
[Equation]
Write samples drawn from some P θ , δ ,D ∈ P Θ , ∆ ,D as ( x , y ) :
x ∼ N ( D θ + δ , I ) , y ∼ N ( θ , I ) .
Lemma B.3.
The minimax linear estimator ˆ θ : ( x , y )
→ A x + B y has the form ˆ θ a , b ( x , y )
=
(cid:80) i a
i x i + (cid:80) i b i
y i for some a , b ∈ R d .
Namely,
[Equation]
Proof.
According to the proof of Proposition B.4.a, by discarding off-diagonal terms, the maximum risk of any linear estimator ˆ θ A,B over any hyperrectangles Θ( τ ) , ∆( ζ ) is reduced.
[Equation]
Further we have:
[Equation]
Therefore all four terms have to be equal, thus ﬁnishing the proof.
Notice Θ( τ ) and ∆( ζ ) are hyperrectangles in R d .
Therefore we could decompose the problem to some 2-d problems:
Proposition B.4.
Under the same setting as Problem 1,
[Equation]
If ˆ θ A,B ( x , y )
=
A x + B y is minimax linear estimator over P Θ( τ ) , ∆( ζ ) ,D , then necessarily A, B must be diagonal.
[Equation]
Proof of Proposition B.4.a .
First review our notation:
[Equation]
Our objective is
[Equation]
We will show that restricting A, B to be diagonal will not include the RHS value.
For any ¯ τ ∈ Θ( τ ) , ¯ ζ ∈ ∆( ζ ) , let set V (¯ τ , ¯ ζ )
= { ( θ , δ ) | ( θ i , δ i ) ∈ { (¯ τ i , ¯ ζ i ) , ( − ¯ τ i , − ¯ ζ i ) }} be the subset of vertices of Θ(¯ τ ) × ∆(¯ ζ ) .
Let π (¯ τ , ¯ ζ ) be uniform distribution on this ﬁnite set.
Due to the symmetry of this distribution, we have
[Equation]
We utilize the distribution to ﬁnd the explicit value of the maximum (in fact the maximum will only be obtained inside the vertices set V (¯ τ , ¯ ζ ) ):
[Equation]
(cid:62) )
+ Tr( A E [ δδ (cid:62) ]
A (cid:62) )
+ 2Tr(( AD + B − I ) E
[ θδ (cid:62) ]
A (cid:62) )
+
Tr( AA (cid:62) ) + Tr( BB (cid:62) ) =
Tr(( AD + B − I ) (cid:62) ( AD + B − I )diag(¯ τ 2 ))
+
Tr( A (cid:62) A diag(¯ ζ 2 ))
+
Tr(( AD + B − I ) (cid:62) A diag(¯ τ ¯ ζ )) +
Tr( AA (cid:62) ) + Tr( BB (cid:62) )
= (cid:88)
i (cid:107) ( AD + B − I ) : ,i ¯ τ
i + A : ,i ¯ ζ i (cid:107) 2 + Tr( AA (cid:62) )
+ Tr( BB (cid:62) ) ≥ (cid:88) i (( A ii D ii + B ii − 1)¯ τ i + A ii
¯ ζ i )
+ A 2 ii + B 2 ii = (cid:107) (
diag( A ) D + diag( B ) − I ) θ + diag( A ) δ (cid:107)
+ Tr(diag( A ) 2 ) + Tr(diag( B ) 2 ) , ( ∀ ( θ , δ ) ∈ V (¯ τ , ¯ ζ ) )
=
max V (¯ τ , ¯ ζ ) (cid:107) (diag( A ) D + diag( B ) − I ) θ + diag( A ) δ (cid:107)
+ Tr(diag( A ) 2 )
+ Tr(diag( B ) 2 )
Therefore we have:
[Equation]
max
[Equation]
Under review as a conference paper at
Next, since the optimal solution on the minimizer is always obtained by diagonal A, B , it becomes straightforward that each axis could be viewed in separation, thus ﬁnishing the proof for part a.
The nonlinear part is a straightforward extension of Proposition 4.16 from Johnstone (2011).
Theorem B.5 (Restated Le Cam Two Point Theorem Wainwright (2019)) .
Let P be a family of distribution, and θ : P → Θ is some associated parameter.
Let ρ : Θ × Θ → R + be some metric deﬁned on Θ and Φ : R + → R + is a monotone non-decreasing function with Φ(0)
= 0 .
For any α ∈ (0 , 1) ,
[Equation]
Lemma B.6.
Consider a class of distribution P τ,ζ,s =
[Equation]
P
[Equation]
τ, |
[Equation]
Deﬁne
[Equation]
and R N ( τ, ζ, s ) =
min ˆ θ
[Equation]
We have
[Equation]
Proof of Lemma B.6.
We ﬁrst calculate an upper bound of R L and connect it to a lower bound of
R N .
[Equation]
By some detailed calculations
[Equation]
we get the RHS is equal to:
[Equation]
For simplify this form
[Equation]
we could see that
Next, we use Le cam two point theorem to lower bound R N ( τ, ζ, s ) where the metric ρ is Euclidean distance and Φ is squared function.
Therefore
[Equation]
Under
Since the total variation distance is related to Kullback-Leibler divergence by Pinsker’s inequality: (cid:107)· , ·(cid:107) TV ≤ (cid:113) 12 D KL ( ·(cid:107)· ) , it’s sufﬁcient to replace the constraint as:
[Equation]
Recall R L ≤ min { 1 , 2 τ 2 , 1+4 ζ s 2 +1 } .
We ﬁrst note that c 2 ≤ 4 τ 2 and setting α =
we have R N ≥ τ 2 / 2 ≥ 1 / 4 R L .
For In the following we look at other cases when the bound for c 2 is smaller.
When 2 ζ ≥ sc, will set d = − sc and c 2 = 2 α 2 .
Let α = 2 / 3 for large τ we get : c 2 (1 − α ) / 8 = 2 / 27 ≥ 2 / 27 R L .
When 2 ζ ≤ sc we set d = − 2 ζ and require ( sc − 2 ζ )
+ c 2 ≤ 2 α 2 .
We have ( sc − 2 ζ ) 2 + c 2 = s 2 c 2 + 4 ζ 2 − 4 ζsc + c 2 ≤ s 2 c 2 + 4 ζ 2 − 8 ζ
+
c
=
( s 2 + 1) c 2 − 4 ζ 2 .
Therefore as we set c 2 =
α 2 +4 ζ 2 s 2 +1 , the original inequality is satisﬁed.
Again by setting α = 2 / 3 we have c 2 ≥ 8 / 9 1+4 ζ 2 s 2 +1 ≥ 8 / 9 R L .
Therefore in this case R N ≥ 227 R L .
C D ISCUSSIONS ON R ANDOM D ESIGN UNDER C OVARIATE S HIFT .
In the main text, we present the results where we consider X S as ﬁxed and Σ T to be known.
In this section, we view both source and target input data as random, and generalize the results of Section 3 while training is on ﬁnite observations and testing is on the (worst case) population loss, under some light-tail properties of the input data samples.
C. R ANDOM DESIGN ON TARGET COVARIANCE MATRIX
In Section 3, we consider the case when Σ T is known exactly.
This could be viewed as the ﬁxed design setting where training and testing are on the same set of data.
In this section, our analysis will include the estimation error on observing ﬁnite unlabeled samples of target domain.
Let
X T =
[ x 1 , · · · x n U ] (cid:62) ∈ R n U ×
d be n U (Here U stands for unlabeled data and is used to distinguish from n T labeled target samples) data samples where x i ∼ p T , and we will use the unlabeled target samples to conduct estimation.
We let ˆΣ T =
X
(cid:62) T X T /n
U .
Let ˆ L B to denote the worst case excess risk measured on the observed target samples: ˆ L B ( ˆ β ) = max β ∗ ∈B E y S 1 n U (cid:107) X T ( ˆ β ( y S ) − β ∗ ) (cid:107) 2 .
To ﬁnd the best linear estimator that minimizes ˆ L B , our proposed algorithm becomes:
[Equation]
And set ˆ β = ˆ C ˆΣ − 1 S X (cid:62) S y S /n S .
We want to show that in spite of the existence of estimation error due to the replacement of Σ T with ˆΣ T , our generated ˆ β performs well on the worst-case population risk L B ( ˆ β ) :=
[Equation]
S
[Equation]
x ∼ p T (cid:107) x (cid:62) (ˆ β ( y S ) − β ∗ ) (cid:107) 2 and achieves minimax linear risk (up to constant multiplicative error).
In this section we assume that the data samples is light tail:
Deﬁnition C.1 ( ρ 2 -subgaussian distribution) .
We call a distribution p to be ρ 2 -subgaussian when there exists ρ > 0 such that the random vector ¯ x ∼ ¯ p is ρ 2 -subgaussian.
¯ p is the whitening of p such that ¯ x ∼ ¯ p is equivalent to x = Σ 1 /
¯ x ∼ p , where Σ is the covariance matrix of p .
Notice that here the subgaussian parameter is deﬁned on the whitening of the data, and ρ doesn’t depend on how large (cid:107) Σ (cid:107) op is.
Theorem C.2.
Fix a failure probability δ ∈ (0 , 1) .
Suppose target distribution p T is ρ 2subgaussian, and the sample size in target domain satisﬁes n U (cid:29) ρ 4 ( d + log 1 δ ) .
Let ˆ β : y S
→ ˆ C ˆΣ − 1 S X (cid:62) S y S where ˆ C is deﬁned from Eqn.
Then with probability at least 1 − δ over the unlabeled samples from target domain, and for each ﬁxed X S from source domain, our learned estimator ˆ β ( y S ) satisﬁes:
[Equation]
Speciﬁcally, when Σ T commutes with ˆΣ S or is rank 1, we have:
[Equation]
Similarly all other results in the paper could be extended to random design with ﬁnite samples X T .
Proof of Theorem C.2.
The proof relies on the two technical claims C.3, C.4.
Let ˆ β R be the optimal linear estimator on L B , i.e., L B ( ˆ β R ) =
min β linear in y S L B ( β ) =
R L ( B ) .
L B (
[Equation]
β ) ≤ (
[Equation]
+ O ( (cid:114) ρ 4 ( d + log(
[Equation]
δ ))
n )
[Equation]
ˆ L B ( ˆ β
[Equation]
(Claim C.4) ≤ (1 + O ( (cid:114)
ρ 4 ( d + log(1 /δ )) n ))
ˆ L B ( ˆ β R ) (from deﬁnition of ˆ β ) ≤ (1 + O ( (cid:114) ρ 4 ( d + log(
[Equation]
δ
[Equation]
n
[Equation] [Equation]
) (Claim C.4) ≤ (1 + O ( (cid:114) ρ 4 ( d + log(
[Equation]
δ
[Equation]
n
[Equation]
B
[Equation]
ˆ β )
=
[Equation] [Equation]
O
[Equation]
(cid:114) ρ 4 ( d + log(1 /δ ))
n ))
R L ( B ) .
(from ρ 4 ( d +log(1 /δ ))
n (cid:28) 1 , and deﬁnition of ˆ β R )
From Theorem 3.4 we know R L ( B ) ≤ 1 . 25 R N ( B ) when Σ T is rank-1 matrix or commute with ˆΣ S which further ﬁnishes the whole proof.
Claim C.3 (Restated Claim A.6 from Du et al.
Fix a failure probability δ ∈ (0 , 1) , and assume n (cid:29) ρ 4 ( d + log(1 /δ ))
Then with probability at least 1 − δ 10 over the inputs x 1 , . .
, x n , if x i ∼ p and p is a ρ 2 -subgaussian distribution, we have
[Equation]
where Σ = E x ∼ p
[ xx (cid:62) ] .
With the help of Claim C.3 we directly get:
Claim C.4.
Fix a failure probability δ ∈ (0 , 1) , and assume n U (cid:29) ρ 4 ( d + log(1 /δ )) , X T =
[ x 1 , · · · , x n U ] (cid:62) ∈ R n U × d satisﬁes x i ∼ p T where p T is ρ 2 -subgaussian.
We have for any estimator β :
[Equation]
with high probability 1 − δ/ 10 over the random samples X T .
Proof of Claim C.4.
Recall
[Equation]
Therefore for any estimator ˆ β , it satisﬁes
[Equation]
which ﬁnishes the proof.
Under review as a conference paper at ICLR
C. R ANDOM DESIGN ON SOURCE DOMAIN .
In the main text or the subsection above, the worst case excess risk is upper bounded by 1 . 25 R N , which is achieved by best estimator that is using the same set of training data ( X S , y S ).
Here we would like to take into consideration the randomness of X S and compare the worst case excess risk using our estimator with a stronger notion of linear estimator.
For this purpose, we consider estimators that are linear functionals of y R :
= Σ 1 / 2 S β ∗ + z ∈
R d , z ∼ N (0 , σ 2 /n S I d ) (this σ 2 /n S is the correct scaling since X (cid:62) S X S /n S is comparable to Σ S ).
We consider the minimax linear estimator with y R and with access to Σ S , and we compare our estimator against this oracle linear estimator.
This estimator is not computable in practice since Σ S must be estimated, but we will show that our estimator is within an absolute multiplicative constant in minimax risk of the oracle linear estimator.
To recap the notations and setup, let
[Equation]
Our target is to ﬁnd the best linear estimator using ˆ L B ( ˆ β ) (trained with X T ) and prove its performance on the population (worst-case) excess risk L B ( ˆ β ) is no much worse compared to the minimax linear risk trained on y R and Σ S .
Theorem C.5.
Fix a failure probability δ ∈ (0 , 1) .
Suppose both target and source distributions p S and p T are ρ 2 -subgaussian, and the sample sizes in source domain and target domain satisﬁes n S , n U (cid:29) ρ 4 ( d + log 1 δ ) .
Let ˆ C be the solution for Eqn.
(10) , and set ˆ β ( y S )
← ˆ C ˆΣ − 1 S X (cid:62) S y S .
Then with probability at least 1 − δ over all the unlabeled samples from target domain and all the labeled samples X S from source domain, our estimator ˆ β ( y R ) yields the worst case expected excess risk that satisﬁes:
[Equation]
Proof of Theorem C.5.
For each matrix C ∈ R d × d , we ﬁrst conduct bias-variance decomposition and rewrite each worst-case risk with linear estimator in terms of a matrix C .
When ˆ β ( y S )
=
C ˆΣ − 1 S X (cid:62) S y S , we have:
[Equation]
Similarly, when ˆ β R = C Σ − 1 / 2 S y R , we have:
[Equation]
Under review paper at
Claim C.6.
Fix a failure probability δ ∈ (0 , 1) , and assume n U , n S (cid:29) ρ 4 ( d + log(1 /δ )) , X S ∈ R n S × d , X
T ∈ R n U ×
d are respectively from p S p T which are both ρ 2 -subgaussian.
We have for any matrix C ∈ R d × d :
[Equation]
with high probability 1 − δ/ 10 over the random samples X T .
[Equation]
with high probability 1 − δ/ 10 over the random samples X S .
Proof of Claim C.6.
We omit the proof of the ﬁrst inequality since it’s exactly the same as proof of
Claim C.4.
For the second line, we have:
[Equation]
Therefore we prove the RHS of the second inequality.
The LHS follows with the same proof techniques.
Now let ˆ C be the minimizer for ˆ l ( C ) , and C R be the minimizer for l R ( C ) .
[Equation]
O ( (cid:115) ρ 4 ( d + log(1 /δ )) n U ))
ˆ l ( ˆ C ) (w.p. 1 − δ/ 10 ; due to Claim C.6) ≤ (1 + O ( (cid:115)
ρ 4 ( d + log(1 /δ )) n U ))
ˆ l ( C R )
(Due to the deﬁnition of ˆ C ) ≤ (1 + O ( (cid:115) ρ 4 ( d + log(1 /δ )) n U ))
2 l ( C R ) (w.p. 1 − δ/ 5 ; due to Claim C.6) =(1 + O ( (cid:115) ρ 4 ( d + log(1 /δ ))
n U ))
l ( C R ) (since n U is large enough) ≤ (1 + O ( (cid:115)
ρ 4 ( d + log(1 /δ )) n U ))
(1 + O ( (cid:115)
ρ 4 ( d + log(1 /δ )) n T ))
l R ( C R ) (w.p. 1 − 3 δ/ 10 ; due to Claim C.6)
Under review as a conference paper at ICLR
[Equation]
This ﬁnishes the proof.
D M ORE EMPIRICAL RESULTS
We include some more empirical studies.
In the main text our results have small noise.
Here we show some more results with larger noise, and also the case with varied eigenspace.
For the following results, we use σ = 10 and r = 0 .
2 √ d .
Other meta data remains the same as presented in the main text.
Figure 2 (a)(b) show similar phenomenon as the small noise setting presented in (a) covariate eigen-spectrum
[Figure]
the main text.
From Figure 2 (c) we see no particular relationship between the performance of each algorithm with eigenspace shift
D. E XPERIMENTS WITH APPROXIMATION ERROR
Finally, we conduct empirical studies with nonlinear models.
Setup.
We choose n S = 2000 , d = 50 .
Let
X S ∈ R 2000 × 50 be generated randomly under
Gaussian distribution N (0 , Σ S ) .
We also generate a small validation dataset from target domain:
X CV ∈ R 500 × 50 , sampled from N (0 , Σ T ) , y CV = f ∗ ( X CV ) + z CV , with z CV ∼ N (0 , σ 2 I ) .
We choose λ
i (Σ S )
∝ i, λ i (Σ T )
∝ 1 /i , and the eigenspace for both Σ S and Σ T are random orthonormal matrices.
( (cid:107) Σ S (cid:107) 2 F = (cid:107) Σ T (cid:107) 2 F = d. )
The ground truth model is a one-hidden-layer ReLU network: f ∗ ( x ) = 1 /d a (cid:62) ( W x ) + , where W and a are randomly generated from standard Gaussian distribution.
We observe noisy labels: y S =
f ∗ ( x ) + z , where z i ∼ N (0 , σ 2 ) .
Under review as a conference paper at ICLR
Estimating weights p T ( x ) /p S ( x ) .
Since the generated data samples are Gaussian, the absolute weights for p T ( x ) /p S ( x )
= (cid:113) | Σ S | | Σ T | exp( 12 x (cid:62) (Σ − 1 S − Σ − 1 T ) x ) .
However, this absolute value has an exponential factor and can amplify the noise level.
Meanwhile, when one multiplies both
X S , y S by 10, the ground truth β doesn’t change but the absolute value for p T ( x ) /p S ( x ) will change drastically.
This discrepancy highlights the importance of relative magnitudes (among samples) instead of the absolute value Kanamori et al.
To obtain a relative score, we ﬁrst estimate the absolute value of p T ( x ) /p S ( x ) by l ( x ) :
= x (cid:62) (ˆΣ − 1 S − ˆΣ − 1 T ) x .
We then uniformly assign the weight for each sample by 10 discrete values 1 , 2 , 3 · · · 10 based on their scoring l ( x ) and then rescale the reweighting vector properly.
We implement our method (Eqn. 4) using the estimated weights as above.
Refer to Figurefor the results.
The baselines we choose are ordinary least square ("OLS" in Figure (3)), ridge regression (Legend is "Ridge") and classic weighted least square Kanamori et al.
(Legend is "Reweighting"; ˆ β LS in our main text).
For both ridge regression and our methods, we tune hyperparameters through cross-validation.
All results are presented from 40 runs where the randomness comes from f ∗ and the eigenspaces of Σ S , Σ T .