Learning Invariant Representations under General Interventions on the Response
Abstract
It has become increasingly common nowadays to collect observations of feature and response pairs from different environments.
As a consequence, one has to apply learned predictors to data with a different distribution due to distribution shifts.
One principled approach is to adopt the structural causal models to describe training and test models, following the invariance principle which says that the conditional distribution of the response given its predictors remains the same across environments.
However, this principle might be violated in practical settings when the response is intervened.
A natural question is whether it is still possible to identify other forms of invariance to facilitate prediction in unseen environments.
To shed light on this challenging scenario, we introduce invariant matching property (IMP) which is an explicit relation to capture interventions through an additional feature.
This leads to an alternative form of invariance that enables a unified treatment of general interventions on the response.
We analyze the asymptotic generalization errors of our method under both the discrete and continuous environment settings, where the continuous case is handled by relating it to the semiparametric varying coefficient models.
We present algorithms that show competitive performance compared to existing methods over various experimental settings.
( The long version of this paper can be found in https: // arxiv.
org/ abs/ 2208.
Introduction
How to make reliable prediction in unseen environments that are different from training environments is a challenging problem, which is fundamentally different from the classical machine learning settings [39, 49, 11].
Modeling these distribution shifts in a principled way is of great importance in many fields including robotics, medical imaging, and environmental science.
Apparently, this problem is ill-posed without any constraints on the relationship between training and test distributions, as the test distribution may be arbitrary.
Consider the problem of predicting the response Y given its predictors X =
( X 1 , ..., X d ) ⊤ in unseen environments.
To model distribution changes across different environments (or training and test distributions), we follow the approach of using structural causal models (SCMs) [33, 36] to model different data-generating mechanisms.
The common assumption is that the assignment for Y does not change across environments (or Y is not intervened), which allow for natural formulations of the invariant conditional distribution of Y given a subset of X [44, 50, 35, 36, 41, 20, 8].
The underlying principle is known as invariance, autonomy or modularity [19, 1, 33, 24].
We provide a brief review of the related works that follow this principle in
Appendix A.
In practical settings, however, the structural assignment of Y might change across environments, namely, Y might be intervened.
How to relax this assumption in a principled way is one of the main motivations in our work.
We propose to explore alternative forms of invariance, and make an attempt in this direction by focusing on linear SCMs.
Concretely, the assignment for Y allows general interventions
[Equation]
where Y can be intervened through coefficient β e and/or the noise ε eY , to capture the dependence of structural assignment across different environments.
Under linear SCMs, anchor regression [43] considers interventions on Y through a shift added to ε eY , which is special case of the general interventions on Y studied in this work.
Another formulation for interventions on Y is through hidden parents of Y that are intervened (see [43, 32, 10]), but we focus on the rarely studied setting of direct interventions on Y and leave the settings with hidden variables for future work.
We consider a multi-environment regression setting for domain adaption: There are multiple training data ( X e , Y e ) for e ∈ E train that are generated from a training model and one test data (indexed by e test ) from a test model; we assume the training model and test model follow SCMs with the same graph structure, but we allow β e and the mean and variance of ε eY to be arbitrarily different under the two models.
To avoid the setting to be ill-posed, a key necessary condition is that Y e needs to have at least one child in the SCMs, as prediction is not possible otherwise given that Y e may change arbitrarily over environments.
The main challenge lies in whether it is still possible to identify other forms of invariance to facilitate prediction in the test environment.
We propose an alternative form of invariance P e
( Y | ϕ e ( X ))
=
P h
( Y | ϕ h ( X ))
that is enabled by a family of conditional invariant transforms Φ ∋ ϕ e , ϕ h .
Under general interventions on Y , we provide explicit constructions of such transforms by developing invariant matching property (IMP) , a deterministic relation between an estimator of Y and X along with an additional predictor constructed from X .
Background and Problem Formulation
Consider a linear acyclic SCM M over ( X, Y ) (see Appendix B.1), the coefficients and noise distributions may change when ( X, Y ) is observed in different environments (e.g., different experiment settings for data collection).
In the following, we use interventions on the SCM M to model such changes.
Let E all denote the set of all possible environments 1 , which consists of multiple training environments E train and one test environment { e test } such that E all =
E train ∪ { e test } .
For each e ∈ E all , an acyclic linear SCM over ( X e , Y e ) is given by
[Equation]
A variable from { X 1 , . . .
, X d , Y } is intervened if the parameters or noise distribution in its assignment changes over different e ∈ E all .
This formulation of M e is fairly general, and we discuss several special cases in Appendix C.
We observe n e i.i.d.
samples { ( x 1 , y 1 ) , ..., ( x n e , y n e ) } from each training environment distribution
P e for e ∈ E train , but in the test environment e test we only observe m i.i.d.
samples { x 1 , ..., x m } from
P test .
The goal is to learn a function f : X → ˆ Y that works well on e test in the sense that it minimizes the test population loss
[Equation]
where l is the square loss function l ( y, ˆ y )
= ( y − ˆ y ) 2 .
To make this problem tractable, we assume that ( X, Y ) under P test and P e are generated according to the SCM M e
but we allow for general types of interventions .
It is well-known that if Y is not intervened, a general form of invariance principle applies, assuming the existence of some subset S ⊆ { 1 , ..., d } such that P e ( Y | X S )
=
P h
( Y | X S ) holds for any e, h ∈ E all .
The main challenge in our setting comes from the general interventions on Y , making the traditional invariance principle not applicable.
In this work, we propose to exploit an alternative form of invariance to tackle this problem.
Definition 1.
A function ϕ : ( E all , R d ) → R q is called a conditional invariant transform if the following invariance property holds for any e, h ∈ E all
[Equation]
Under general intervention settings, we denote this class of conditional invariant transforms as Φ , and we provide explicit characterizations of it via the invariant matching property (IMP) (see Definition 2).
For each ϕ ∈ Φ , the invariance property (4) enables us to compute
[Equation]
for any e ∈ E all , where the function g : R q → ˆ Y is invariant across environments and is nonlinear in general.
Equivalently, this solves a relaxed version of (3) by minimizing L test ( f ϕ ) over { ϕ ∈ Φ } .
A Motivating Example
Example 1.
Consider ( Y e , X e ) , e ∈ E toy = { 1 , 2 } , with X e :=
( X e 1 , X e 2 , X e 3 ) ⊤ satisfying the following linear acyclic SCM (illustrated in Figure.
[Equation]
where X e 1 , X e 2 , N e 3 , and N eY are independent and N (0 , 1) -distributed for every e ∈ E toy .
Since ( Y e , X e ) is multivariate Gaussian, the MMSE estimator of Y e given X e is
[Equation]
Similarly, one can compute E P e [ X 3 | X [Equation]
[Equation]
= [Equation] + X e 2 .
As a result, there exists a deterministic linear relation, which we refer to as matching ,
[Equation]
with coefficients λ = 1 / 2 and η = ( − 1 , 0 , 1 / 2) ⊤ that are invariant with respect to the environment.
Moreover, one can verify that P e ( Y | X 1 , X 3 , E [ X 3 | X 1 , X 2 ]) is invariant since the corresponding conditional mean and variance are invariant.
A prediction model in (6) with invariant coefficients is often not unique when it exists.
One can show that E P [Equation]
However, such invariant relations do not hold for E P e
[ X 1 | X 2 , X 3 ] .
To further illustrate the invariant relations, we provide simulations in Appendix D. Moreover, in Appendix E, we extend Example 1 to allow for interventions on X 1 , X 2 , and Y through the means and/or variances of the noise variables.
Invariant Matching Property
In this section, we generalize the invariant relations observed in Example 1 to a class of such relations for M e , e ∈ E all .
To handle non-Gaussian cases (beyond Example 1), we choose to adopt the linear MMSE (or LMMSE) estimators for constructing linear invariant relations. For a target variable
Y ∈ R given a vector of predictors X ∈ R p , the LMMSE estimator is define as
[Equation]
where θ ols :
= Cov( X, X ) − 1 Cov ( X, Y ) is called the population ordinary least squares (OLS) estimator. With a slight abuse of notation, we write E l, P e [ Y | X ] to denote the LMMSE of Y given
X with respect to ( X, Y ) ∼ P e .
To simplify presentation, we focus on ( X e , Y e ) with zero means for each e ∈ E all , while the non-zero mean settings can be handled by introducing the constant one as an additional predictor
Definition 2.
For k ∈ { 1 , . . .
, d } , R ⊆ { 1 , . . .
, d } \ k , and S ⊆ { 1 , . . .
, d } , we say that the tuple ( k, R, S ) satisfies the invariant matching property (IMP) if, for every e ∈ E all , E l, P e [ Y | X S ] = λ E l, [Equation]
[Equation]
(7) for some λ ∈ R and η ∈ R d that do not depend on e .
We denote I M :
= { ( k, R, S ) : (7) holds } for model M , and we call ( η ⊤ , λ )
⊤ the matching parameters.
Observe that E l, P e [ Y | X S ] is not directly applicable to the test environment due to its components depending on e , but those components are fully captured by E l,
P e [ X k | X R ] .
We formally define this class of additional features as follows.
Definition 3.
For any k ∈ { 1 , . . .
, d } and R ⊆ { 1 , . . .
, d } \ k , we call E l, P e
[ X k | X R ] a prediction module .
If a prediction module satisfies an IMP for some S ⊆ { 1 , . . .
, d } , we call it a matched prediction module for S .
Now we discuss the relationship between the IMP and the invariance property P e
( Y | ϕ e ( X ))
=
P h
( Y | ϕ h ( X )) in (4).
Define ϕ
( k,R,S )
e ( X e ) :
=
( X eS ′ , E l, P e [ X k | X R ]) ⊤ , where X eS ′ is a row vector for some S ′ ⊆ { 1 , . . .
, d } .
In general, the invariance of the matching parameters { λ, η } does not imply that the invariance property (4) holds for some ϕ ( k,R,S ) e ( X e ) .
In Section 6, we will characterize a class of IMPs that each satisfies (4).
It is crucial to note that we only use the IMP to identify the transform to satisfy the invariance property.
When the invariance property is in place, one can apply the general conditional expectation f ϕ e ( x ) =
E P e
[ Y | ϕ e ( x )] as in (5), since the linear estimator from the IMP is in general sub-optimal for the non-Gaussian cases.
It is noteworthy that since E l, P e [ X k | X R ] is a linear function of X eR , the matching parameters are not unique given a single environment e ∈ E train .
We show that two training environments are sufficient for the identification of the matching parameters in Appendix F.1.
In our toy examples, recall that the IMPs are derived by first computing E P e [ Y | X S ] and E P e [ X k | X R ] separately and then fitting a linear relation from ( E P e [ X k | X R ] , X S ) to E P e
[ Y | X S ] . These two steps reveal a natural decomposition of the IMP, which we term as the first and second matching properties below. Definition 4. We say that S ⊆ { 1 , . . .
, d } satisfies the first matching property if, for every e ∈ E all , [Equation] ) for some λ Y ∈ R and η Y ∈ R d that do not depend on e . First, observe that the first matching property holds for S = PA ( Y ) since E l, P e
[ Y | X PA ( Y ) ]
=
E P e
[ Y | X PA ( Y ) ]
= ( β e ) ⊤ X e .
The first matching property concerns the set S such that the components in E l, P e [ Y | X S ] that depends on e are fully captured by the causal function E P e
[ Y | X PA ( Y ) ] .
However, this invariant relation is not directly useful for the prediction of Y e test , since the causal function can change arbitrarily with e . To this end, we identify another invariant relation from M e which is called the second matching property. Definition 5. For k ∈ { 1 , . . .
, d } and R ⊆ { 1 , . . .
, d } \ k , we say that a tuple ( k, R ) satisfies the second matching property if, for every e ∈ E all ,
[Equation]
for some λ X ∈ R and η X ∈ R d that do not depend on e .
It is straightforward to see that, if λ X ̸ = 0 in the second matching property, the first and second matching properties imply the IMP as follows,
[Equation]
For prediction tasks under SCMs, the causal function often plays a central role.
Our first and second matching properties show how the LMMSE estimator E l, P e [ Y | X S ] and the matched prediction module E l,
P e
[ X k | X R ] are connected with the causal function, respectively.
Together, the two individual connections make up the IMP (illustrated in Fig.
4 from Appendix F.2).
Characterization of Invariant Matching Properties
First, we consider model M e with interventions only on Y through the coefficients β e 2 (denoted as M e, 1 ).
To distinguish the parents of Y with varying and invariant coefficients, we decompose β e in M e into two parts α e and β .
Without loss of generality, we assume that α ej ̸
= 0 if and only if α ej is a non-constant function of e , and we define PE = { j ∈ { 1 , . . .
, d } :
α ej ̸ = 0 } .
Recall that prediction modules do not rely on the response Y but the relations between the predictors for each environment.
When Y is unobserved (or equivalently, substituting Y in (2) into (1)), the relations between the predictors are as follows,
[Equation]
(10) where γε Y + ε X a vector of dependent random variables when γ is not a zero vector.
If α e vanishes from (10), the distribution of X e becomes invariant with respect to environments, but the distribution of Y e changes arbitrarily due to the change of α e , which makes the prediction problem ill-posed.
Observe that α e is non-vanishing in (10) only if γ is not a zero vector, which brings up the following key assumption.
Assumption 1.
When Y is intervened, we assume that Y has at least one child.
The first and second matching properties enable us to characterize the tuples ( k, R, S ) ’s that satisfy IMPs through the characterizations of S (for the first matching property) and ( k, R ) (for the second matching property) separately.
In the following theorem, we show that a class of IMPs implied by the first and second invariant matching properties satisfy the invariance property (4).
Theorem 1.
For model M e, 1 , the first and second matching properties hold in the following cases.
On the first MP: For each S ⊆ { 1 , . . .
, d } such that PE ⊆ S , the first matching property holds.
On the second MP: For each k ∈ { 1 , . . .
, d } \ PE and R ⊆ { 1 , . . .
, d } \ k such that PE ⊆ R , the second matching property holds.
For any tuple ( k, R, S ) above such that R ⊆ S , if λ X ̸ = 0 in the second matching property, then ϕ e ( X e ) = ( X eS , E l, P e [ X k | X R ])
⊤ satisfies (4) .
Furthermore, L test ( f ϕ ) is minimized by any ϕ with
S = { 1 , ..., d } .
It is noteworthy that Assumption 1 is a necessary condition for λ X ̸ = 0 , and we provide a sufficient condition for λ X ̸ = 0 in a concrete setting with S
= { 1 , . . .
, d } in Proposition 2 in Appendix G.
More details regarding the characterization of IMPs are provided in Appendix G, along with the characterization of IMP with interventions on both the predictors and response.
Due to space limit, we provide our algorithms developed for both discrete and continuous environment setting in Appendix I.
To handle the continuous environment setting, we bridge our framework with the profile likelihood estimators developed in the semiparametric literature.
A note on the profile likelihood estimation for semiparameteric varying coefficient models can be found in Appendix K.
Our algorithms are examined over various synthetic data sets under different intervention settings in
Appendix J. Finally, we analyze the asymptotic generalization errors of the proposed estimators for both discrete and continuous environment settings in Appendix L. C Types of Interventions on Linear SCMs D Simulations on Example E Another Toy Example
F Properties of the IMP
F.
Identification of the Matching Parameters . . . . . . . . . . . . . . . . . . . . . . F. An Illustration for the First, Second, and Invariant Matching Properties . . . . . . . G More on the Characterization of IMPs
G.1 Interventions on the Response . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G.2 Interventions on both Predictors and Response . . . . . . . . . . . . . . . . . . . .
H Proofs for the Theoretical Results in Section 6 and Section G
H.1 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.2 Proof of Proposition 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.3 Proof of Proposition 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.4 Proof of Corollary 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
H.5 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I
Algorithms
I
Discrete
Environments
I. Continuous Environments
J Experiments
J.
Discrete environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
J.
Interventions on both X and Y (continuous) . . . . . . . . . . . . . . . . . . . . .
J.
Robustness . .
K Note on Semi-parametric Varying Coefficient Models and Profile Least-Squares Estimation
L Asymptotic Generalization Error
M Proofs for the Theoretical Results in Section L[-
-]
M.1 Technical Lemmas for the Proof of Theorem 3 . . .
M.2 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
M.3 Proof of Corollary 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
M.4 Proof of Corollary 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Related Works
The invariance-based causal prediction initiated in [35] (also see [31] and [8] and references therein) assumes that the conditional distribution of Y given a set of predictors X S ⊆ { X 1 , ..., X d } is invariant in all environments, i.e., P e ( Y | X S )
=
P h
( Y | X S ) for environments e and h , where ( X, Y ) is generated according to the joint distribution P e := P X,Ye .
Focusing on linear SCMs, it assumes the existence of a linear model that is invariant across environments, with an unknown noise distribution and arbitrary dependence among predictors (see extensions to nonlinear [22] and time series [37] settings).
Following this framework, theoretical guarantees for domain adaption have been developed in [41, 30].
More recently, a multi-environment regression method for domain adaption called the stabilized regression [38] explicitly enforces stability (based on a weaker version of invariance
E P e
[ Y | X S
= x s ] =
E P h
[ Y | X S
= x s ] ) by introducing the stable blanket , which is a refined version of the Markov blanket to promote generalization.
The tradeoff between predictive performance on training and test data has been studied via regularization under shift interventions [43].
Motivated by [35], the invariant risk minimization (IRM) [2] imposes P e ( Y | ϕ
( X ))
=
P h
( Y | ϕ ( X )) , where ϕ is invariant across environments, leading to a bi-leveled optimization problem that is not practical.
Several relaxed versions of IRM have been proposed in [2], but they behave very differently from the original IRM (see, e.g., [42, 25]).
For a framework of the out-of-distribution setting from a causal perspective with a focus on minimizing the worst-case risk, see [10] and references therein.
In this line of invariance-based work, the fundamental assumption is that interventions on the target variable
Y is not allowed .
In [9], the authors have provided a systematic treatment of domain adaption using the SCMs to enable analysis and comparisons of domain adaption methods, which leads to the conditional invariant residual matching (CIRM) method.
The CIRM and its variants combine the domain invariant projection (DIP)-type methods (see [5, 14] and the generalized label shift to handle target label perturbation [27, 48]) with the idea of conditional invariance penalty (appeared in [17, 21] under slightly different settings) that assumes the existence of conditional invariant components (CICs) in the anticausal setting where Y causes X .
Theoretical guarantees have been provided for the prediction performance under shift interventions on Y , while numerical studies are provided for interventions on the noise variance of Y [9].
It has also been pointed out that the general mixedcausal-anticausal domain adaptation problem remain open .
We aim to shed light on this challenging setting by constructing explicit conditional invariant transforms.
The role of causality in facilitating domain adaptation problem is first articulated in [44], focusing on causal and anticausal predictions.
Reweighting methods have been extensively studied for covariate shift [39, 46, 47], which assumes that only the feature distribution changes over environments while the conditionals remain the same.
The label shift, which aligned with the anticausal setting, has attracted much attention recently [28, 3, 16].
Many other interesting domain adaptation methods have been developed but they are less related to this work.
The performance bounds using VapnikChervonenkis (VC) theory has been initiated in [7].
There are fundamental works from the robust statistics perspective including distributional robust learning [4, 23, 45, 15, 26, 12] and adversarial machine learning [18, 40].
B Definitions B.1 Linear SCM
Consider a response Y ∈ R and a vector of predictors X
=
( X 1 , . . .
, X d )
⊤ ∈ X ⊆ R d , a linear
SCM over ( X, Y ) is defined by
[Equation]
where β, γ ∈ R d , B ∈ R d × d , ε X = ( ε X 1 , . .
. , ε X d ) ⊤ , and the noise variables { ε X 1 , . .
ε X d } and ε Y are jointly independent.
We use G ( M ) to denote the directed acyclic graph induced by M , with edges determined by the non-zero coefficients in M .
C Types of Interventions on Linear SCMs
This formulation of the interventions on M as in M e is fairly general.
From the structural perspective, it consists of causal, anticausal, and mixed-causal-anticausal settings [44].
It should be noted that we only adopt the linear SCM rather than the fully specified SCMs as in [34], since learning the functional forms can be more complicated than the prediction problem we aim to solve.
Regarding the interventions types, we discuss several special cases to put it into perspective.
Shift interventions on X or Y :
A variable X j is intervened through a shift if the mean of the noise variable ε eX,j changes with e ∈ E all .
For the shift intervention on Y , the mean of ε eY changes .
Interventions on the coefficients of X or Y : A variable X j is intervened through coefficients if the coefficients { γ ej , B ej · } change with e ∈ E all .
For Y , the change is on the coefficient vector β e . 3.
Interventions on the noise variance of X or Y : Similar to shift interventions, a variable X j or Y is intervened if its noise variance changes.
D Simulations on Example
Let X 4 :=
E P e [ X 3 | X 1 , X 2 ] , X 5 :=
E P e [ X 2 |
X 1 , X 3 ] , and X 6 :
= E P e [ X 1 | X 2 , X 3 ] .
We illustrate the linear invariant relations that correspond to X 4 and X 5 as provided in Example 1.
Recall that such invariant relations do not hold for X 6 . for ( [Table] [ Y | [Table]
[Table] Regarding the estimation of the MMSE estimators and the coefficients in the linear relations, we take the estimation procedure for Figure 2.(a) as an example in the following.
For e ∈ { 1 , 2 } , X e 4 and
E P e
[ Y | X ] are estimated using OLS.
Then, η (denoted as η a ⃝ in Figure 2.(a)) is estimated using OLS by regressing E P e [ Y | X ] on ( X e 4 , X e ) using the pooled data.
E Another Toy Example
We extend Example 1 to allow for interventions on X 1 , X 2 , and Y through the means and/or variances of the noise variables (see Fig. 3).
Example 2.
Consider training model M e toy in Example 1 with additional shift interventions and/or interventions on the noise variances.
The results are summarized in the following.
Under shift interventions on Y and X 1 and an intervention on the variance of X 1 , the two invariant relations in Example 1 hold with additional intercept terms.
When Y is intervened through the variance of N Y , the relations in Example 1 will not hold.
In this case, new relations can be established if E P e [ Y | X ] in Example 1 is replaced by E P e [ Y | X PA ( Y ) ] =
E P e
[ Y | X 1 , X 2 ] .
When X 2 is intervened through either the mean or variance, a relation as in Example 1 that is based on E P e [ X 2 | X 1 , X 3 ] will not hold.
Combining the interventions above, there will be one invariant relation left,
[Equation]
for some intercept b ∈ R .
It is noteworthy that (11) will fail to hold if X 3 is intervened.
However, due to the intervention on the noise variance of Y , P e ( Y | X S , E P e [ X 3 | X 1 , X 2 ]) is not invariant for any S ⊆ { 1 , 2 , 3 } , since Var P e ( Y | X S , E P e [ X 3 | X 1 , X 2 ]) changes with e .
F Properties of the IMP F.1 Identification of the Matching Parameters
We rewrite (7) in a compact form as
[Equation]
(12) where (cid:101)
[Equation]
(13) and θ = ( η ⊤ , λ ) ⊤ denotes the matching parameter.
Proposition 1.
For a tuple ( k, R, S ) that satisfies an IMP, the matching parameter θ can be uniquely identified in E train if |E train | ≥ 2 and E [Equation] , P e [
[Equation]
for some e, h ∈ E train and x ∈ X R .
Proof.
Let E train =
{ e 1 , . . .
, e n } .
For a tuple ( k, S, R ) that satisfies the IMP, let ˆ Y =
( E [ Y e 1 |
X e 1 S ] , . . .
, E [ Y e n | X
e
n S ]) ⊤ and
[Equation]
where the rows of ˜ X are independent.
According to (12), we have ˆ Y = ˜
X θ .
Then, if E [ ˜ X ⊤ ˜ X ] is invertible, we have
[Equation]
Now, we prove the invertibility.
Observe that
[Equation]
where E [ X ⊤ X ] = (cid:80) ni =1 E [ X e i
( X e i ) ⊤ ] is inveritble since it is a sum of positive-definite matrices.
Then, E [ ˜ X ⊤ ˜ X ] is invertible if and only if
[Equation]
This is equivalent to
E [( v − X β ) ⊤ ( v − X β )]
̸ = 0 , where β :=
E − 1 [ X ⊤ X ] E [ X ⊤ v ] .
This is true since there is no b ∈ R d such that v = X
b almost surely by our assumption in (14).
Therefore, θ is uniquely determined by (15).
F.2 An Illustration for the First, Second, and Invariant Matching Properties
G More on the Characterization of IMPs G.1 Interventions on the Response
Corollary 1.
For model M e, 1 , the first and second matching properties hold in the following cases.
On the first MP: The first matching property holds for S
= { 1 , . . .
, d } .
On the second MP: For each k ∈ { j ∈ MB ( Y ) : α ej = 0 } and R = − k := { 1 , . . .
, d } \ k , the second matching property holds.
Proposition 2.
Under Assumption 1, for each ( k, R ) in Corollary 1, we have λ X ̸ = 0 in the second matching property if B − k,k is not in the following hyperplane,
[Equation]
) where w ∈ R ( d − 1) and b ∈ R are determined by the parameters in M e, 1 other than B − k,k .
The explicit expressions of w and b using the parameters in M e, 1 are provided in the proof of
Proposition 2.
For generic choices of the parameters, the second matching property holds with λ X ̸ = 0 since B k, − k is not necessarily on the hyperplane described in (16).
When Y is additionally intervened through the noise variance, the proof of Theorem 1 will break down in general (see Remark 2 in Appendix H.1).
However, recall that the first matching property holds for S = PA ( Y ) by definition.
In this case, we provide an example for the second matching property in the following corollary.
Corollary 2.
Under Assumption 1, if Y is intervened through the noise variance in model M e, 1 , the second matching property holds for k ∈ CH ( Y ) such that k ̸∈ DE ( i ) for any i ∈ CH ( Y ) \ k , and
R = { 1 , . . .
, d } \ DE ( Y ) .
The resulting IMPs no longer satisfy the invariance property (4), but we can use the IMP directly for the prediction of Y e test .
Remark 1.
To sum up, the class of IMPs constructed under interventions on Y only through the coefficients and shifts will in general imply the invariance property (4) , but it is not the case under interventions on Y through the noise variance.
G.2 Interventions on both Predictors and Response
To generalize the setting when only Y is intervened to the general setting when X and Y are both intervened, an idea is to the merge the setting when only Y is intervened with the one when only X is intervened.
The later setting has been studied in the stabilized regression framework [38].
The following set of predictors is identified (see Definition 3.4 therein),
[Equation]
such that j [Equation] which contains the intervened children of Y (denoted by CH
I ( Y ) ) and the descendants of such children.
This useful notion can be defined for each X j ∈ { X 1 , . . .
, X d } , denoted by X int ( X j ) for each X j .
When only X is intervened, the invariance principle P e ( Y | X S ) =
P h
( Y | X S ) holds for S ∗
= { 1 , . . .
, d } \ X int ( Y ) ; the Markov blanket of Y defined with respect to X S ∗ is called the stable blanket of Y in [38].
In other words, by excluding the predictors in X int ( Y ) , the target Y is blocked from the interventions on X when conditioning on X S ∗ .
This holds when Y is additionally intervened, as if only Y is intervened given X S ∗ .
In order for S ∗ to include as least one child of Y as in Assumption 1, we need the following assumption.
Assumption 2.
When Y is intervened, we assume that Y has at least one child that is not intervened and that child is not a descendent of some intervened child of Y .
Based on the observation above, we identify an important class of IMPs for the general setting in the following Theorem.
Theorem 2.
For the training model M e without the intervention on the noise variance of Y , the first and second matching properties hold in the following cases.
On the first MP: For S = { 1 , . . .
, d } \ X int ( Y ) , the first matching property holds.
On the second MP: For each k ∈ { 1 , . . .
, d } \ { PE ∪ X int ( Y ) } , and R = { 1 , . . .
, d } \ { k, X int ( X k ) ∪ X int ( Y ) } , the second matching property holds.
Furthermore, if λ X ̸ = 0 in the second matching property, then ϕ e ( X e ) = ( X eS , E l, P e
[ X k | X R ]) ⊤ satisfies (4) .
Similar to the argument in the proof of Theorem 1, the class of ϕ ’s from Theorem 2 will lead to the same test population loss, as they depend on the same S that is fixed in this setting.
Assumptionis necessary for λ X ̸ = 0 , while sufficient conditions for λ X ̸ = 0 can be found similarly as in
Proposition 2.
When Y is additionally intervened through the noise variance, the second matching property in Theorem 2 still holds (see Remark 3 in Appendix H.4).
H Proofs for the Theoretical Results in Section 6 and Section G
By introducing an environmental random variable E ∈ E all , we define a mixture of M e ’s, e ∈ E all , as follows,
[Equation]
where E in a root node in G ( M ) and the noise variables are jointly independent given E .
We do not specify the distribution of E but assume that E does not have a degenerate distribution.
Under this formulation, the invariance property (4) is equivalent to
[Equation]
(17) and the invariant, first, and second matching properties can be equivalently written as
[Equation]
[Equation]
[Equation]
(20) for e ∈ E all .
As a special case of M , we define a mixture of M e, 1 ’s as
[Equation]
[Equation]
where only Y is intervened through the coefficients.
The proofs of Theorems 1, 2 and Corollaries 1, 2 will be presented under this formulation.
H.1 Proof of Theorem
For the first part, let Z ( E ) =
α ⊤ ( E ) X denote an additional node in the acyclic graph G ( M 1 ) , then the assignment of Y in (22) becomes
[Equation]
23) where E is no longer a parent of Y .
Note that Z ( E ) =
E [ Y | X PA ( Y ) , E ] − β ⊤ X as we assume both
X and Y have zero means.
Since E is a root node, observe that E and Y can be d-connected through only two types of paths as follows, E →
[Equation]
E → Z ( E ) ← X
i →
[Equation]
· → X l ← [Equation] ← Y , where i ∈ PE , and X i → · · · → X l ← · · · ← Y is a V-structure for some l ∈ DE ( i ) ∩ DE ( Y ) .
Note that the second type of path does not exist if Assumption 1 is not satisfied or CH ( i ) \ Y is empty.
We start by showing that the d-separation Y ⊥⊥ G E | { Z ( E ) , X S } holds given PE ⊆ S .
First, the first path is immediately blocked by Z ( E ) .
Second, for any s ∈ CH ( i ) \ Y , the path E →
Z ( E ) ← X
i
→
X s is blocked by { Z ( E ) , X i } .
Thus, the second path is blocked given Z ( E ) and
X S .
According to the Markov property of SCMs [33], the d-separation Y ⊥⊥ G E | { Z ( E ) , X S } implies [Equation]
[Equation]
Now, we prove that the above conditional independence implies the first matching property (19).
By definition, the LMMSE estimators only rely on the (finite) first two moments of the variables.
Thus we start with the case when ( X, Y ) |
E = e is jointly Gaussian, for each e ∈ E all .
First we have
[Equation]
where ( a ) follows from the Gaussian assumption on ( X, Y )
| E = e and ( b ) from the fact that Z ( e ) is a function of { X S , E = e } given our assumption that PE ⊆ S .
This implies that
[Equation]
where ( a ) follows from the conditional independence relation (24).
Using the Gaussian assumption again, we have E [ Y | Z ( e ) , X S ] =
E l [ Y | Z ( e ) , X S ] .
Thus putting all the pieces together, we obtain
[Equation]
where a ∈ R and b ∈ R | S | that are not functions of E .
When ( X, Y ) | E = e is non-Gaussian, one can replace it with Gaussian random variables with the matching first and second moments.
Then the same argument leading to (26) still holds.
Then, the first matching property follows from the fact that
[Equation]
Similarly, for the second part, we first show the following d-separation X k ⊥⊥ G E | { Z ( E ) , X R } .
Observe that X k and E can be d-connected through two types of paths as follows,
E → Z ( E ) → Y · · · X k , 2.
E → Z ( E ) ← X i · · · X k ,
with i ∈ { j ∈ { 1 , . . .
, d } : α j ( E ) ̸ = 0 } , where Y · · · X k denotes any directed path between Y and
X k , and similarly for X i · · · X k .
The two types of paths are immediately blocked by { Z ( E ) , X R } under our assumption that PE ⊆ R .
We thus have the following when X | E = e is jointly Gaussian,
[Equation]
[Equation]
for some c ∈ R and d ∈ R | R | do not depend on E .
The non-Gaussian cases can be handled in the same way as before, and thus the second property follows again from Z ( E ) =
E [ Y | X PA ( Y ) , E ] − β ⊤ X .
Observe that we have λ X
=
c in the second matching property and Assumption 1 is a necessary condition for λ X ̸ = 0 .
Finally, given R ⊆ S , we have that (27) with c ̸ =
(i.e., λ X ̸ = 0 ) provides a one-to-one mapping between { Z ( E ) , X S } and { E l [ X k | X R , E ] , X S } .
Therefore, the conditional independence (24) is equivalent to
[Equation]
This implies that, using our previous notation, ϕ e ( X e ) =
( X eS , E l, P e [ X k | X R ])
⊤ satisfies the invariance property (4), which implies 3 E P e
[ Y | X S ] = E P e [ Y | ϕ e ( X )] .
Effectively, E P e [ Y | ϕ e ( X )] serves as a representation of E P e [ Y | X S ] that is invariant.
Note that Φ is not empty when λ X ̸ = 0 holds with
R ⊆ S .
This implies that any ϕ ∈ Φ with S
= { 1 , . . .
, d } minimizes L test ( f ϕ ) , since the optimality of ϕ ∈ Φ only relies on the corresponding S .
Remark 2.
When ε Y is replace by ε Y ( E ) , we consider the following two cases.
The mean of ε
eY is a function of e , and its variance is a constant.
The variance of ε
eY is a function of E , and its mean can be either a function of e or a constant.
For the first case, we can introduce X d +
= 1 that is a parent of Y with α d +1 :=
E [ ε Y ( E | E ] .
Additionally, X d +1 is a parent of every X j such that ε X,j has a non-zero mean.
Specifically, the coefficient of X d +1 in the assignment of X j will be E [ ε X,j ] .
Thus, the problem reduces to the setting when ε eX and ε eY have zero means, which has been proved in Theorem 1.
For the second case, however, the varying variance of ε Y ( E ) cannot be separated as the mean, thus E is always a parent of Y and the path E
→ Y cannot be blocked by Z or any X S ⊆
{ X 1 , . . .
, X d } , i.e., the proof of
Theorem 1 breaks down.
H.2 Proof of Proposition
The following lemma is a slight extension of Lemma 3.6 from [38], where we consider linear models with dependent noise variables rather than linear SCMs considered in [38].
Lemma 1.
Consider V ∈ R and X =
( X 1 , . . .
, X p ) ⊤ ∈ R p satisfying a linear model,
[Equation]
where e V ∈ R , g, h, e X ∈ R p , and A ∈ R p × p .
Assume that I − ˜
A is invertible, the population OLS estimator when regressing V on X is given by
[Equation]
where c := (cid:16) [Equation] + [Equation]
2 g ⊤ [Equation] and Cov( e, e ) := (cid:20) σv ⊤ v Σ (cid:21) , with σ 2 :=
Var( e V ) , Σ :=
Cov( e X , e X ) , and v :=
Cov( e X , e V ) .
Proof.
Let P =
[0 1 × d , I d × d ] , we continue from Equation A.7 in [38] as follows,
[Equation]
and it has been shown that
[Equation]
] , for M =
( I − A − gh ⊤ ) − 1 and w =
Mg .
Then,
[Equation]
where ˜Σ := Σ + vg ⊤ + gv ⊤ and it was computed in [38] using the Sherman-Morrison formula [6] that
[Equation]
with c := (cid:16) 1 + σ 2 g ⊤ ˜Σ − 1 g (cid:17) − 1 .
Then, some simple algebra leads to
[Equation]
H.3 Proof of Proposition
For each tuple ( k, R, S ) , k ∈ { j ∈ MB ( Y ) : α ej = 0 } .
R = − k := { 1 , . . .
, d } \ k , S = { 1 , . . .
, d } , we prove that λ X ̸ = 0 .
Equivalently, we prove that E l [ X k | X − k , E = e ] is a non-constant function of e .
First, recall that, when the target variables Y is unobserved, the relations between the predictors in
M e, 1 , are described by
[Equation]
Now, we rewrite the above equation in the same form as the linear model (29) as follows,
[Equation]
where the top-left element of the coefficient matrix is zero, i.e., γ k ( β k
+ α ek ) + B k,k = [-
 sinceα-]0 since α ek = 0 (by assumption), B k,k =
(due to acyclicity), and γ k β k =
(since X k cannot be both a child and a parent of Y ).
Now, by Lemma 1, the population OLS estimator when regressing X k on
X − k given E = e is
[Equation]
where h = γ k ( β + α e ) − k + B ⊤ k, − k , v = γ k σ 2 Y γ − k , g = β k γ − k + B − k,k , A = γ − k ( β + α e )
⊤− k + B − k − k Σ =
Cov( N X − k N X − k )
= Σ + vg ⊤ + gv = (cid:16) σ 2 g − (cid:17) −
[Equation]
k ∈ CH ( Y ) :
First, observe that β k = 0 implies g = B − k,k .
By plugging h and A into (30), we obtain
[Equation]
where α e − k in the first term is non-vanishing only if
[Equation]
where we use (31).
k ∈ PA ( Y ) : Observe that v =
( d − 1) and γ k = 0 , then
[Equation]
where the first term is not vanishing only if
[Equation]
k ∈ { j : ∃ i ∈ CH ( X j )
such that i ∈ CH ( Y ) } :
Again, we have v =
( d − [Equation] × [Equation]
and γ k = 0 . Additionally, we have β k = 0 , then α e − k in θ ols ,k ( e ) is non-vanishing only if
[Equation]
H.4 Proof of Corollary
First
[Equation] we extract the assignments of ( X k , X R ) ’s from (10) with E = e , where k ∈ CH ( Y ) and k ̸∈ DE ( X i ) for any i ∈ CH ( Y ) \ k and R = { 1 , . . .
, d } \ DE ( Y ) as follows, (cid:20) X k X R (cid:21)
[Equation]
where the zeros in the coefficient matrix are due to the fact that all the descendants of X k are excluded from X R since DE ( X k )
⊆
DE ( Y ) and R = { 1 , . . .
, d } \ DE ( Y ) .
Note that X k is a child of Y that is not a descendent of any other child of Y , thus any removed node j ∈ { 1 , . . .
, d } \ { k, R } can not be the parent of any remaining node i ∈ { k, R } .
Now, using Lemma 1, the population OLS estimator when regressing X k on X R given E = e is
[Equation]
This implies
[Equation]
where we use the fact that β j = α ej
= B k,j = 0 for j ̸∈ R (recall that { 1 , . . .
, d }\ R does not contain either the parents of X k or parents of Y ).
Therefore,
[Equation]
which is the second matching property.
Remark 3.
Observe that θ ols ( e ) does not depend on B R,R or the covariance of ε X,R , thus the second matching property holds even if every X j ∈ X R is intervened.
H.5 Proof of Theorem
For the first part, we only need to prove that Y and E are d-connected only through the arrow E → Y when conditioning on X S , S = { 1 , . . .
, d } \ X int ( Y ) .
The rest is simply the proof of the first part of
Theorem 1.
Since E is a root node, Y and E can only be d-connected through the two types of paths,
E → · · · → Y , 2.
Y → · · · → X j ← · · · ← E ,
where j ∈ DE ( Y ) ∩ DE ( E ) , and E → · · · → Y denotes any directed path from E to Y .
For the first type of paths, since PA ( Y ) ⊆ S , the only unblocked path when conditioning on X S is
E → Y .
For the second type of paths that are V-structures, we have j ̸∈ S since X int ( Y ) contains all the intervened children of Y and the descendants of such children, implying that the V-structures are always blocked when conditioning on X S .
Thus, the only unblocked path between E and Y is
E → Y when conditioning on X S .
For the second part, similarly, for k ∈ { 1 , . . .
, d } \ { PE ∪ X int ( Y ) } and R = { 1 , . . .
, d } \ { k, X int ( X k ) ∪ X int ( Y ) } , we prove that X k is d-connected with E only through paths that contain the subpath E
→ Y when conditioning on X R (i.e., the first type of path below).
Following the same idea as in the first part, Y and E are d-connected only through the arrow E → Y when conditioning on X R since X int ( Y ) ∩ R = ∅ and PA ( Y ) ⊆
R .
Then, since X k is not intervened (i.e., E ̸∈ PA ( X k ) ), the variables E and X k can only be d-connected through two types of paths as follows,
E → Y → · · · → X k 2.
E → · · · → X i ← · · · ← X k ,
where i ∈ R
and we use the fact that k ̸∈ X int ( Y ) (i.e., the node k is not an intervened child of Y or a descendant of an intervened child of Y ).
To handle the second type of paths, we will need two technical results.
(I)
If i ∈ R , then j ∈
R for any j ∈ PA ( i ) such that j ̸ = k .
This can be proved by contradiction.
If j ̸∈ R (i.e., j ∈ X int ( Y ) ∪ X int ( X k ) ), then we have i ∈
X int ( Y )
∪ X int ( X k ) by the definition of X int ( Y ) and
X int ( Y ) , i.e., i ̸∈ R .
(II)
For i ∈ R , observe that X i can only be a child of X k , otherwise the path
X i ← X j ← · · · ← X k is blocked by X j ∈ PA ( X i ) when conditioning on X R , since i ∈ R implies j ∈ R .
Now, we proved that the second type of paths are always blocked when conditioning on X R , by focusing on E → · · · → X i .
For any X i ∈ CH ( X k ) , the subpath E → · · · → X i cannot be E → X i , since i ∈ R implies that X i is not an intervened child of X k .
Finally, the path
E → · · · → X l → X i is blocked by X l ∈ PA ( X i ) when conditioning on X R , since i ∈ R implies that l ∈ R .
I Algorithms
For each e ∈ E train , we are given the i.i.d.
training data X e ∈ R n e × d , Y e ∈ R n e , and we observe the i.i.d.
test data X τ ∈ R m × d and aim to predict Y τ ∈ R m .
Let X ∈ R n × d with n := (cid:80) |E train | i = 1 n e denote the pooled data of X e ’s. In this section, we present the implementation of our method starting with the case when e is sampled from a discrete distribution with a finite support.
In this setting, we expect to have n e ≫ 1 for every e ∈ E train in general, thus it is possible to do estimation based on the data from each single environment.
The challenging setting of continuous environments will be handled afterwards.
I.1 Discrete Environments
To implement our method, the main task is to identify the set of IMPs I M from the training data.
For each tuple ( k, R, S ) in Algorithm 1, we test the following null hypothesis
[Equation]
We propose two test procedures.
Test of the Deterministic Relation: Since the IMPs are linear and deterministic (i.e., noiseless), we test whether the residual vector R ∈ R n of fitting an IMP on ( k, R, S ) is a zero vector or not using the test statistics,
[Equation]
where R is a pooled data vector of R e ’s defined below.
To fit an IMP, we first estimate the two
LMMSE estimators in (7) using OLS for each environment,
[Equation]
Let ˆ L 1 ∈ R n and ˆ L 2 ∈ R n denote the pooled data of ˆ L e, 1 ’s
and ˆ L e, 2 ’s, respectively.
It is noteworthy that ˆ L 2 only depends on X , thus ˆ L τ 2 for the test data can be computed similarly using
X τ .
Next we estimate the matching parameter using OLS on the pooled data (recall that the matching parameter cannot be identified using the data from a single environment).
The OLS estimator of the matching parameter is
[Equation]
For each e ∈ E train , we obtain the residual vector of fitting an IMP
[Equation]
Approximate Test of Invariant Residual Distributions :
According to the invariance property (5), we test whether the residual when regressing Y on [ X S , L 2 ] has constant mean and variance.
Specifically, we use the t-test and F-test with corrections for multiple hypothesis testing from [35] (see Section 2.1 Method II).
The test yields a p-value.
The test statistic from the first procedure and the p-value from the second procedure quantify how likely an IMP holds (i.e., the smaller the more likely), and thus we will refer to either one of them as an IMP score denoted by s IMP .
Let ˆ I = { ( k, R, S ) : s IMP ( k, R, S ) <
c IMP } denote the set of IMPs identified from the training data, where c IMP is some cutoff parameter.
Then, since IMPs are not equally predictive in general, we focus on the most predictive ones by introducing the mean squared prediction error as a prediction score s pred , and we select the set of IMPs that are more predictive (cid:100)
I pred =
{ ( k, R, S ) ∈
ˆ I : s pred ( k, R, S ) <
c pred } with some cutoff parameter c pred .
For the second
IMP score that is a p-value, the cutoff parameter c
IMP is simply a significance level that is fixed to 0 .
05 in this work.
For choosing the rest cutoff parameters, we follow a bootstrap procedure from [37] with one subtle difference: We sample the same amount of bootstrap samples from each environment rather than sampling over the pool data as in [37], since our procedure involves estimations using the data from each environment.
Algorithm 1 Invariant Prediction using the IMP (discrete) procedure I DENTIFY IMP S FROM THE T RAINING DATA for k ∈ { 1 , . . .
, d } , S ⊆ { 1 , . . .
, d } , R ⊆
S \ k do
Compute the IMP score s imp and the prediction score s pred for i =
( k, R, S )
Regress Y on [ X S , ˆ L 2 ] to obtain f i
Identify ˆ I and (cid:100)
I pred procedure P REDICTION ON THE T ESTING DATA
ˆ
[Equation]
In practice, there can be spurious IMPs that have extremely small IMP scores but have large prediction scores, e.g., when Y is independent of X S and X k is independent of X R .
To this end, we will preselect ( k, R, S ) ’s with prediction scores smaller than the median of the all the computed prediction scores before identifying ˆ I .
If the regression function f i in Algorithm 1 is chosen to be linear, one can use the IMP directly, i.e.,
[Equation]
which we call the discrete IMP estimator denoted by IMP d .
To make use of all the IMPs selected in (cid:100) I pred , we use an averaging step for the prediction of Y τ in Algorithm 1.
I.2 Continuous Environments
To model continuous environments, we introduce an environmental variable U that is a continuous random variable with support U .
Apparently, this is a much more challenging setting compared with the discrete environment case, as we only have one training data sample for each u ∈ U , making the OLS a poor estimate of E l, P u
[ Y | X S ] .
Fortunately, it turns out that we can leverage the semi-parametric varying coefficient (SVC) models [13] (see Appendix ?? )
to remedy this issue.
In particular, we estimate E l, P u [ Y | X S ] by fitting,
[Equation]
where N is independent of U and the two vectors of predictors W ∈ R p (for the varying coefficient) and Z ∈ R q (for the invariant coefficients) with p + q = | S | .
Since we assume N ⊥⊥ U , we focus on the settings when Y is not intervened through the noise variance.
Remark 4.
Our estimation procedure for the discrete environments can also be formulated under the SVC model with a discrete random variable U , where we treat all the coefficients as varying coefficients (i.e., β = 0 ), and IMP d becomes an estimate of M .
An SVC model over ( Y τ , W τ , Z τ , N τ ) for the test data can be defined similarly, where σ 2 =
E [( N τ ) 2 ] is the population generalization error of the IMP estimator.
Observe that the linear SCM
M u in ( ?? ) can be viewed as a collection of SVC models parameterized by U = u .
Thus the estimation tasks for the linear SCMs from continuous environments can greatly benefit from the existing theories developed for SVC models.
More precisely, we employ the following estimate
[Equation]
where the profile least-squares estimation of β and M proposed in [13] can be found in Appendix ?? .
Similarly, E l, P u [ X k | X R ] can be estimated by fitting another semi-parametric varying coefficient model
[Equation]
where V denotes any X k , and X R is divided into Z V ∈ R r and W .
It is noteworthy that the two SVC models share the same set of predictors with varying coefficients, which we explain below.
A challenge for fitting such models is that the vector of predictors with varying coefficients, namely W , needs to be known.
For continuous environments, we focus on discovering IMPs that can be decomposed into the first and second matching properties.
Thus, since the causal function captures the predictors with varying coefficients, the first and second matching properties imply that the vector W is simply X PE , i.e., the parents of Y with varying coefficients in
M u , for both models.
Based on this observation and Theorem 1, we replace the exhaustive search over ( k, R, S ) in
Algorithm 1 by a search over ( P, k, R, S )
according to the conditions in Theorem 1 with PE = P .
That is, we choose ( P, k, R, S ) from
[Equation]
such that W = X P , Z = X S \ P , V = X k , and Z V =
X R \ P .
Unlike IMP d in (32), we make use of the fact that β is invariant and propose the continuous IMP estimator denoted by IMP c as follows
[Equation]
35) with the matching parameter w ∈ R p +1 estimated by
[Equation]
The data matrices for the two models (e.g., W ∈ R n × p ) can be defined accordingly and we provide the details in Appendix M.1.
In this case, the residual vector for the first IMP score is given by
[Equation]
Note the the second IMP score is not applicable for continuous environments due to the small sample size in each environment, thus we focus the first IMP score.
Remark 5.
The IMP d and IMP c are similar in spirit, the IMP c relies on the first and second matching properties for identifying W (so we can reuse Z in (35) ), whereas IMP d directly tests the IMP since the estimation process treats all the coefficients as varying coefficients (also see the proof of
Corollary 4).
J Experiments
The prediction performance is measured by the mean residual sum of squares (RSS) on the test environments.
We compare our method with several baseline methods: Ordinary Least Squares (OLS), stabilized regression (SR) [38], anchor regression (AR) [43].
We have also compared with domain invariant projection (DIP) [5], conditional invariance penalty (CIP) [21], conditional invariant residual matching (CIRM) [9], and invariant risk minimization (IRM) [2]; it turns out that the empirical performance of these methods are not as competitive as the other baselines in our experimental settings, thus we do not report them below.
The two IMP scores lead to two versions of our algorithm, and we refer to the first one IMP and the second one IMP inv (as it tests the invariance of the noise mean and variance).
We focus on linear functions
f i ’s in Algorithm 1, namely, we use the IMP estimators.
For the profile likelihood estimation, we adopt the Epanechnikov kernel k ( u ) = 0 .
75 max(1 − u 2 , 0) with the bandwidth fixed to be 0 .
We test DIP, CIP, CIRM, and their variants provided in [9] with the default parameters.
For the anchor regression, we use a 5 -fold cross-validation procedure to select the hyper-parameter γ from { 0 , 0 . 05 , 0 . 1 , . . .
The significance levels are fixed to be 0 .
for all methods.
We randomly simulate 500 data sets for each experiment, if not mentioned otherwise.
J.1 Discrete environments
First, we generate linear SCMs M e ’s without interventions.
For each e ∈ E train = { 1 , . . .
, 5 } or e ∈ E test = { 6 , . .
. , 10 } , we randomly generate a linear SCM with 11 variables as follows.
The graph G ( M e ) is specified by a lower triangular matrix of i.i.d.
Bernoulli(1 / 2) random variables.
The response Y is randomly selected from the 11 variables and we require that Y has a least one parent and one child in G ( M e ) .
For each linear SCM, the non-zero coefficients are sampled from
Unif[ − 1 . 5 , − 0 .
5] ∪ [0 . 5 , 1 . 5] and the noise variables are standard normal.
For each training or test environment, we simulate i.i.d.
data of sample size 300 .
J.1.1 Interventions on X
Since the baseline methods have been examined extensively under shift interventions, we focus on shift interventions on X for the comparison.
The general interventions on X will be considered in
Section J.3.
Specifically, for each training environment, we randomly selected 4 predictors to be intervened through shifts sampled from Unif[ − 2 , 2] .
For each test environment, the shifts are sampled from Unif[ − 10 , 10] .
In Fig.
5, IMP inv performs similarly to SR since they share a similar idea when only X is intervened.
But our IMP method can potentially improve upon these two methods, since there are IMPs beyond the ones that imply invariance (see an example in ( ?? )), which is left for future work.
Due to the averaging steps of IMP, IMP inv , and SR, they have smaller variances compared with
OLS and AR (a similar result has been reported in [37]). I
[Equation]
R
O [Equation]
A R       m ean
R
SS
J.1.2 Interventions on Y
We consider the response Y to be intervened through both the coefficients and shifts.
We randomly select n p ∼ Unif { 1 , . . .
, | PA ( Y ) |} of parents of Y to have varying coefficients.
For each training environment, we add perturbation terms sampled from Unif[ − 2 , 2] to the original coefficients.
For each test environment, the perturbations are sampled from Unif[ − 10 , 10] .
The shift intervention on Y is the same as the shift interventions on X in Section J.1.1.
In this setting, since none of the baseline methods allow interventions on Y through the coefficients, they cannot even improve upon
OLS.
In Fig.
6, IMP performs slightly better than IMP inv , which may due to the fact that the IMP method aims to find all possible IMPs, but the IMP inv only looks for IMPs that imply invariance.
[Figure]
M P I M P _ i n v
S R
O L S
A R       m ea n R
SS
J.1.3 Interventions on both X and Y
The setting of interventions on both X and Y is simply a combination of the two setting above.
We require that Y has at least one child that is not intervened when selecting the predictors to be intervened.
In this challenging setting, our method outperforms the baselines by a large margin.
[Figure]
S R
O L S
A R
[Figure]
J.2 Interventions on both X and Y (continuous)
In this more challenging setting, we only compare with OLS, since AR only considers shift interventions while the other baselines are not developed for continuous environments, and also recall that IMP inv is proposed for discrete environments.
First, we define { U 1 , . . .
, U 800 } sampled from
Unif[0 , 1] and { U τ 1 , . .
, U τ 800 } sampled from Unif[1 , 2] as the training and test environments, respectively.
Similar to the setting of discrete environments, we randomly generate the linear SCMs without interventions first and then add interventions to the model.
Due to the high computational complexity, we focus on graphs with 5 nodes where 2 predictors are intervened.
The interventions on the coefficients and shift interventions are defined by adding a perturbation term a sin(2 πwU i ) , where w is sampled from Unif[0 .
The parameter a is fixed to be 2 for the training environments (i.e., the same range as for the discrete case) and 5 for the test environments.
Since the parameter space is much smaller than the previous experiments, we only generated 100 data sets.
Overall, the performance of our IMP algorithm is similar to that in the discrete setting.
[Figure]
J.3 Robustness
In the previous experiments, we only consider shift interventions on X and interventions on Y other than the noise variance.
In this experiment, we consider an extreme case when only a child of Y with the highest causal ordering is not intervened (i.e., Assumption 2 is satisfied), all other variables are intervened through every parameter .
Specifically, a shift intervention or an intervention on the coefficient is defined by adding a perturbation term to the original parameter.
The perturbation term is sampled from Unif[ − 2 , 2] for the training data, and from Unif[ − 5 , 5] for the test data.
The intervened noise variances are sampled from Unif[0 .
75 , 1 . 25] for each training environment, and from Unif[0 .
5 , 1 . 5] for each test environment.
To test how sensitive our method is with respect to
Assumption 2 in this challenging setting, we gradually add interventions to the child of Y that is not intervened, where the shifts and coefficient interventions are sampled from Unif[ − 2 λ, 2 λ ] and
Unif[ − 5 λ, 5 λ ] for the training and test environments, respectively.
The intervened noise variances are sample from Unif[1 − 0 .
25 λ, 1 + 0 .
25 λ ] for training, and from Unif[1 − 0 .
5 λ, 1 + 0 .
5 λ ] for testing.
The parameter λ ∈ [0 , 1] controls the intervention strength.
Due to the results in Section J.1.3, it would not be informative to compare with the baseline methods, so we focus on our IMP method.
Note that our IMP inv is also not included since IMPs will not imply invariance in this case (see
Remark 1).
Overall, as shown in Fig.
9, the median of the mean RSS is not too sensitive with respect to mild interventions on the child that is not intervened, but the variance increases rapidly.
K Note on Semi-parametric Varying Coefficient Models and Profile Least-Squares Estimation
First, we introduce the semi-parametric varying coefficient model following most of the notation in [13].
Consider Y ∈ R , U ∈ U , and two vectors of predictors W =
( W 1 , . . .
, W p ) ⊤ and
Z =
( Z 1 , . . .
, Z q ) ⊤ such that Z j ’s have invariant coefficients, a semi-parametric varying coefficient
m ean
R
SS model over ( U, Y, W, Z ) is defined by
[Equation]
where N is independent of ( U, W, Z ) .
We briefly introduce the profile least-squares estimator of β proposed in [13].
Denote n i.i.d.
samples of ( U, Y, Z, N ) as U =
( U 1 , . . .
, U n ) ⊤ , Y =
( Y 1 , . . .
, Y n ) ⊤ , W =
( W 1 , . . .
, W n ) ⊤ , W i = ( W i 1 , . . .
, W ip ) ⊤ , Z =
( Z 1 , . . .
, Z n ) ⊤ , Z i =
( Z i 1 , . . .
, Z iq ) ⊤ , and N = ( N 1 , . .
. , N n ) ⊤ .
We thus have M =
( M 1 , . .
, M n ) ⊤ with M i = α ⊤
U i ) W
i .
Let K u
=
diag
( K h ( U 1 − u ) , . . .
, K h ( U n − u )) for some kernel function K h ( · ) =
K ( · /h ) /h with bandwidth h , and
[Equation]
For u in a neighborhood of u 0 ∈ U , assume that each function α i ( u ) in (37) can be approximated locally by the first-order Taylor expansion
α i ( u )
≈
α
i + b i ( u − u 0 ) .
Then the varying coefficients α ( U 1 ) , . .
, α ( U n ) can be estimated by solving the weighted local least squares problem
[Equation]
which has the solution
[Equation]
for u ∈ { U 1 , . . .
, U n } .
Then, each variable M i can be estimated by ˆ M i
= (cid:80) pj
=
ˆ a j ( u ) X ij [Equation] and thus the vector M can be estimated by
[Equation]
which depends on the unknown parameter β that will be estimated below.
Substituting ˆ M ( β ) into the vector form of (37), we obtain ( I − A )
Y =
( I − A )
Z β + N .
The profile least-squares estimator of β is given by
[Equation]
Finally, by replacing β in (39) with ˆ β , the final form of the estimator for M is given by
[Equation]
L Asymptotic Generalization Error
In this section, we provide the asymptotic generalization errors (as n, m → ∞ ) of the IMP c and
IMP d estimators for ( k, R, S ) ’s that satisfy IMPs, i.e., ( k, R, S ) ∈ I M .
Recall that σ 2 = E [( N τ ) 2 ] is the population generalization error of the IMP estimators.
Due to the estimations on both training and test data, the asymptotic generalization error can be decompose to the error terms depend on the training data size n and the test data size m as follows.
Let c n = (cid:110) log(1 /h )
nh (cid:111) 1 / 2 + h 2 , where h is the kernel bandwidth (see Appendix K for details).
Theorem 3.
For any ( k, R, S ) ∈ I M , under the technical assumptions in Appendix M.1, the asymptotic generalization error of the IMP c estimator is given by
[Equation]
The following corollary considers the setting when the amount of unlabeled training and test data grows in a higher order than the amount of labels in the training data.
The generalization error due to the estimation on the test data disappears.
Corollary 3.
Given i.i.d.
training data of size n with 0 < l n < n labels and test data of size m , if max( l n n , l n m ) → 0 as min( m, n ) → ∞ , under the technical assumptions in Appendix M.1, the asymptotic generalization error of the IMP c estimator is given by
[Equation]
The setting of discrete environments can be viewed as a special case of continuous environments, where the error term c n ∨ n − 1 / 2 due to the kernel estimation procedure is replaced by an error term from multiple OLS estimations.
Corollary 4.
For any ( k, R, S ) ∈ I M , under the technical assumptions in Appendix M.1, the asymptotic generalization error of the IMP d estimator is given by
[Equation]
where a n = (min e ∈E train n e ) − 1 / 2 and a m = m − 1 / 2 .
This asymptotic generalization error heavily depends on the environment with the smallest sample size, which also supports the fact that IMP d should not be employed for continuous environment settings.
M Proofs for the Theoretical Results in Section L M.1 Technical Lemmas for the Proof of Theorem
First, we present some technical assumptions and two technical lemmas from [13].
Let c ′ n = (cid:110) log(1 /h )
nh (cid:111) 1 / 2 and c n = c ′ n + h 2 .
U has a bounded support U and has density function f ( · ) that is Lipschitz continuous and bounded away from 0 .
For each U i , the matrix E [ W ⊤ W | U i ] is non-singular, and the matrices E
[ W ⊤ W
| U i ] , (cid:0) E
[ W ⊤ W
| U i ] (cid:1) − 1 , and E [ WZ ⊤
| U i ] are Lipschitz continuous.
α 1 ( u ) , . . . , α p ( u ) have continuous second derivatives.
K ( · ) is a symmetric density function.
Lemma 2 ([29]) .
Let ( U 1 , Y 1 ) , . . .
, ( U n , Y n ) be i.i.d.
random vectors in R 2 .
Assume that E [ | Y | s ] < ∞ and sup x (cid:82) |
y | s f ( u, y ) dy < ∞ , where f ( u, y ) is the density of ( U, Y ) .
Let K be a bounded positive function with a bounded support that satisfies a Lipschitz condition.
Given that n 2 ε − 1 h → ∞ for some ε < 1 − s − 1 , then
[Equation]
Lemma 3 ([13]) .
Under suitable assumptions, √ n (ˆ β − β ) d −→ N (0 , Σ) as n → ∞ , where Σ =
Var( N ) · C − 1 with
[Equation]
In the following lemma, we provide the rates of several quantities needed for the proof of Theorem 3.
Lemma 4.
Under the same assumptions as in Lemma 3,
[Equation]
Proof.
First, (41) and the vector form of (37) give
[Equation]
Observe that R 2 ∼ R 4 can be defined through
[Equation]
where P can be replaced by W , Z , M , or N (note that the rows of P are i.i.d.).
We also have
R 1 = (cid:80)
i
=
I i , where
[Equation]
It can be shown that I 1 , I
= O p ( c n ) (we use this as a shorthand for I 1 =
O p ( c n ) , I
=
O p ( c n ) , as I
and I 3 are not equal), I 2 =
O p ( n − 1 / 2 ) , I 5 =
O p ( n − 1 ) , I 4 , I 6 =
O p ( c 2 n ) , I 8 =
O p ( c n n − 1 / 2 ) ,
I 7 , I 9 =
O p ( c 2 n ∨ c n n − 1 / 2 ) , which implies R 1 = O p ( c 2 n ∨ n − 1 ) , R 2 , R 3 =
O p ( c n ∨ n − 1 / 2 ) and
R
=
O p ( c n ) .
The techniques for proving the rates of I 1 ∼ I 9 are similar; observe that all the components in I 4 ∼ I 9 are already computed to obtain I 1 ∼ I 3 , thus we only provide the proof for
I 1 ∼ I 3 for simplicity of presentation.
Using Lemma 2, it has been shown in [13] that ˜ W ⊤ u
i K u i ˜
W u i can be equivalently expressed as
[Equation] [Equation]
where µ 2 = (cid:82) U u 2 K ( u ) du , and the four block matrices are B 1 = S
( U i ) , B 2 = B 3 = S 1 ( U i ) , and
B 4 = S 2 ( U i ) , with respect to
[Equation]
Since the techniques for proving (42), omitted in [13], will be used repeatedly in the rest of this paper, we provide the proof for completeness.
Applying Lemma 2, it holds uniformly in u ∈ U that S k ( u ) can be expressed as
[Equation]
where (43) is due to the change of variable V
=
( U − u ) /h , (44) uses the Lipschitz continuity assumptions on E [ XX T | U ] and f ( · ) , and (45) is by the symmetry of the kernel function K ( · ) .
Similarly, we obtain
[Equation]
Recall the expression of A in (39), we have
[Equation]
Using (42) and (46) [Equation] we obtain
[Equation]
where P i denotes the i th row of P and the last equality is due to the law of large numbers.
For I 2 , similarly as above, we compute
[Equation]
Since β − ˆ β
=
O p ( n − 1 / 2 ) by Lemma [Equation] we obtain
[Equation]
where, again, the last equality is due to the law of large numbers.
Finally, for
[Equation]
the same argument for (45) leads to
[Equation]
where E [ WN ⊤ | U i ] = 0 since N is independent of W and U i , and N has a zero mean.
Thus, by the law of large numbers,
[Equation]
Proofs of Theorem 3 and Corollaries 3 and
To reuse model (37) for the prediction of Y τ , the main challenge comes from M that changes with U (while β remains invariant).
First, we introduce some notation for the proofs.
We define ( Y , W , W i ) and ( Z , Z i , M , N ) in the same way as in Appendix K. Similarly, we define Z V , M V , N V , and all the corresponding data matrices for the test data (e.g., Y τ ), and let σ 2 = E
[( N τ ) 2 ] .
With this notation, the IMP (7) implies that
[Equation]
for some ζ ∈ R p .
Let ˆ W V :=
[ W , ˆ M V ] and ˆ W τ V :=
[ W τ , ˆ M τ V ] .
Then, the OLS estimator of w according to the above equation is given by
[Equation]
We predict Y τ using the continuous IMP estimator
[Equation]
where ˆ β , ˆ M , and ˆ M V are provided in Appendix K.
Lemma 5.
Under assumptions 1) ∼ 4), it holds that
[Equation]
Proof.
Using the fact that
[Equation]
we obtain
[Equation]
where we use R 1 and R 2 from Lemma 4 and the law of large numbers.
Similarly,
[Equation]
Note that the rate is not directly implied by Lemma 4, but it can be proved using the same techniques demonstrated in the proof Lemma 4.
Therefore, using (47),
[Equation]
M.2 Proof of Theorem
Using (49) and Y τ
= M τ + Z τ β + N τ , we derive ˆ Y [Equation] Y
[Equation]
Then, the generalization error is given by
[Equation]
where
[Equation]
We can show the following rates of these terms through simple applications of Lemmas 3, 4, 5, along with the law of large number and the central limit theorem.
J 2 , J 3 =
O p ( c 2 n ∨ n − 1 ) , J 4 =
O p ( n − [Equation]
[Equation]
p
[Equation] [Equation]
and J 1 , J 6 ∼ J 9 have higher orders compared with either O p ( c n ∨ n − 1 / 2 ) or O p ( c m ∨ m − 1 / 2 ) in
J 10 .
This completes the proof of Theorem 3.
M.3 Proof of Corollary
We use l as the shorthand notation for l n in the proof.
Since the estimation of β, M , and w are based on the labeled data, we have ˆ β − β
= O p ( l − 1 / 2 ) in Lemma 3 and R 1 , R 2 , R 3 =
O p ( c l ∨ l − 1 / 2 ) and R 4 =
O p ( c l ) in Lemma 4.
In the proof of Lemma 5, the rate of ˆ W ⊤ V
ˆ W V remains the same since the estimation of M V only uses the unlabeled data but the rate of ˆ W ⊤ V
ˆ M now depends on l .
This observation implies ˆ w − w =
O p ( c l ∨ l − 1 / 2 ) .
Now, observe that J 2 =
O p ( c 2 l ∨ l − 1 ) , J 3 =
O p ( c m ∨ m − 1 / 2 ) , J 4 = O p ( l − 1 ) , J 5 = σ 2 + O p ( m − 1 / 2 ) , and J 10 =
O p ( c l ∨ l − 1 / 2 ) + O p ( m − 1 / 2 ) .
Since max( ln , lm ) → 0 as min( n, m ) → ∞ , we get (cid:80) 10 i =
1 J i
= O p ( c l ∨ l − 1 / 2 ) .
M.4 Proof of Corollary
According to the definition of the discrete IMP estimator, all the coefficients are treated as varying coefficients (i.e., β = 0 and β V = 0 ), and M is estimated by performing the OLS for each environment and then putting the estimates into one vector, thus R 1 = O p ( a 2 n ) and R 2 , R 4 =
O p ( a n ) in Lemma 4, with a n = (min e ∈E train n e ) − 1 / 2 by the asymptotic normality of the OLS estimators ( note that X and Z are assumed to have finite fourth moments in Lemma 3).
Accordingly, we have ˆ w − ˆ w =
O p ( a n ) in Lemma 5 using the law of large numbers.
Setting Z τ = 0 in J 1 ∼ J 10 , we obtain J 2 , J 3 =
O p ( a 2 n ) , J 4 = 0 , J 5 =
σ 2 + O p ( m − 1 / 2 ) , J 10 = O p ( a n ) + O p ( a m ) , and the other terms have higher orders compared with O p ( a n ) or O p ( a m ) .
Similar to Theorem 3, the asymptotic generalization error is dominated by J 10 .