Original: L ATENT C AUSAL I NVARIANT M ODEL
Revised: L ATENT C AUSAL I NVARIANT M ODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Current supervised learning can learn spurious correlation during the data-ﬁtting process, imposing issues regarding interpretability, out-of-distribution (OOD) generalization, and robustness.
Revised: Current supervised learning can learn spurious correlation during the data-ﬁtting process, imposing issues regarding interpretability, out-of-distribution (OOD) generalization, and robustness.

Original: To avoid spurious correlation, we propose a La tent
Revised: To avoid spurious correlation, we propose a La tent

Original: C ausal I nvariance M odel (LaCIM) which pursues causal prediction .
Revised: C ausal I nvariance M odel (LaCIM) which pursues causal prediction .

Original: Speciﬁcally, we introduce latent variables that are separated into (a) output-causative factors and (b) others that are spuriously correlated to the output via confounders, to model the underlying causal factors.
Revised: Speciﬁcally, we introduce latent variables that are separated into (a) output-causative factors and (b) others that are spuriously correlated to the output via confounders, to model the underlying causal factors.

Original: We further assume the generating mechanisms from latent space to observed data to be causally invariant .
Revised: We further assume the generating mechanisms from latent space to observed data to be causally invariant .

Original: We give the identiﬁable claim of such invariance, particularly the disentanglement of output-causative factors from others, as a theoretical guarantee for precise inference and avoiding spurious correlation.
Revised: We give the identiﬁable claim of such invariance, particularly the disentanglement of output-causative factors from others, as a theoretical guarantee for precise inference and avoiding spurious correlation.

Original: We propose a Variational-Bayesian-based method for estimation and to optimize over the latent space for prediction.
Revised: We propose a Variational-Bayesian-based method for estimation and to optimize over the latent space for prediction.

Original: The utility of our approach is veriﬁed by improved interpretability, prediction power on various OOD scenarios (including healthcare) and robustness on security.
Revised: The utility of our approach is veriﬁed by improved interpretability, prediction power on various OOD scenarios (including healthcare) and robustness on security.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Current data-driven deep learning models, revolutionary in various tasks though, heavily rely on i.i.d data to exploit all types of correlations to ﬁt data well.
Revised: Current data-driven deep learning models, revolutionary in various tasks though, heavily rely on i.i.d data to exploit all types of correlations to ﬁt data well.

Original: Among such correlations, there can be spurious ones corresponding to biases ( e.g. , selection or confounding bias due to coincidence of the presence of the third factor) inherited from the data provided.
Revised: Among such correlations, there can be spurious ones corresponding to biases ( e.g. , selection or confounding bias due to coincidence of the presence of the third factor) inherited from the data provided.

Original: Such data-dependent spurious correlations can erode the (i) interpretability of decision-making, (ii) ability of out-of-distribution (OOD) generalization, i.e. , extrapolation from observed to new environments, which is crucial especially in safety-critical tasks such as healthcare, and (iii) robustness to small perturbation (Goodfellow et al., 2014).
Revised: Such data-dependent spurious correlations can erode the (i) interpretability of decision-making, (ii) ability of out-of-distribution (OOD) generalization, i.e. , extrapolation from observed to new environments, which is crucial especially in safety-critical tasks such as healthcare, and (iii) robustness to small perturbation (Goodfellow et al., 2014).

Original: Recently, there is a Renaissance of causality in machine learning, expected to pursue causal prediction (Sch ¨ olkopf, 2019).
Revised: Recently, there is a Renaissance of causality in machine learning, expected to pursue causal prediction (Sch ¨ olkopf, 2019).

Original: The so-called “causality” is pioneered by Judea Pearl (Pearl, 2009), as a mathematical formulation of this metaphysical concept grasped in the human mind.
Revised: The so-called “causality” is pioneered by Judea Pearl (Pearl, 2009), as a mathematical formulation of this metaphysical concept grasped in the human mind.

Original: The incorporation of a priori about cause and effect endows the model with the ability to identify the causal structure which entails not only the data but also the underlying process of how they are generated.
Revised: The incorporation of a priori about cause and effect endows the model with the ability to identify the causal structure which entails not only the data but also the underlying process of how they are generated.

Original: For causal prediction, the old-school methods (Peters et al., 2016; B ¨ uhlmann, 2018) causally related the output label Y to the observed input X , which however is NOT conceptually reasonable in scenarios with sensory-level observed data ( e.g. modeling pixels as causal factors of Y does not make much sense ).
Revised: For causal prediction, the old-school methods (Peters et al., 2016; B ¨ uhlmann, 2018) causally related the output label Y to the observed input X , which however is NOT conceptually reasonable in scenarios with sensory-level observed data ( e.g. modeling pixels as causal factors of Y does not make much sense ).

Original: For such applications, we rather adopt the manner in Bengio et al.
Revised: For such applications, we rather adopt the manner in Bengio et al.

Original: ; Biederman (1987) to relate the causal factors of Y to unobserved abstractions denoted by S , i.e. , Y ← f y ( S, ε y ) via mechanism f y .
Revised: ; Biederman (1987) to relate the causal factors of Y to unobserved abstractions denoted by S , i.e. , Y ← f y ( S, ε y ) via mechanism f y .

Original: We further assume existence of additional latent components denoted as Z , that together with
Revised: We further assume existence of additional latent components denoted as Z , that together with

Original: S generates the input X via mechanism f x as X ← f x ( S, Z, ε x ) .
Revised: S generates the input X via mechanism f x as X ← f x ( S, Z, ε x ) .

Original: Taking image classiﬁcation as an example, the S and Z respectively refer to object-related abstractions ( e.g. , contour, texture, color) and contextual information ( e.g. , light, view).
Revised: Taking image classiﬁcation as an example, the S and Z respectively refer to object-related abstractions ( e.g. , contour, texture, color) and contextual information ( e.g. , light, view).

Original: Such an assumption is similarly adopted in the literature of nonlinear Independent Components Analysis (ICA)
Revised: Such an assumption is similarly adopted in the literature of nonlinear Independent Components Analysis (ICA)

Original: (Hyvarinen and Morioka, 2016; Hyv¨arinen et al., 2019; Khemakhem, Kingma and Hyv¨arinen, 2020;
Revised: (Hyvarinen and Morioka, 2016; Hyv¨arinen et al., 2019; Khemakhem, Kingma and Hyv¨arinen, 2020;

Original: Teshima et al., 2020) and latent generative models (Suter et al., 2019), which are however without separation of output ( y )-causative factors ( a.k.a , S ) and other correlating factors ( a.k.a , Z ) that can both be learned in data-ﬁtting process.
Revised: Teshima et al., 2020) and latent generative models (Suter et al., 2019), which are however without separation of output ( y )-causative factors ( a.k.a , S ) and other correlating factors ( a.k.a , Z ) that can both be learned in data-ﬁtting process.

Original: We encapsulate these assumptions into a novel causal model, namely La tent C ausal I nvariance M odel (LaCIM) as illustrated in Fig.
Revised: We encapsulate these assumptions into a novel causal model, namely La tent C ausal I nvariance M odel (LaCIM) as illustrated in Fig.

Original: 1, in which we assume the structural equations f x (associated with
Revised: 1, in which we assume the structural equations f x (associated with

Original: S, Z → X ), f y (associated with S → Y ) to be the C ausal I nvariant Me chanisms (CIMe) that hold under any circumstances with P ( S, Z ) allowed to be varied across domains.
Revised: S, Z → X ), f y (associated with S → Y ) to be the C ausal I nvariant Me chanisms (CIMe) that hold under any circumstances with P ( S, Z ) allowed to be varied across domains.

Original: The incorporation of these priories can explain the spurious correlation embedded in the back-door path from Z to Y (contextual information to the class label in image classiﬁcation).
Revised: The incorporation of these priories can explain the spurious correlation embedded in the back-door path from Z to Y (contextual information to the class label in image classiﬁcation).

Original: To avoid learning spurious correlations, our goal is to identify the intrinsic CIMe f x , f y .
Revised: To avoid learning spurious correlations, our goal is to identify the intrinsic CIMe f x , f y .

Original: Speciﬁcally, we ﬁrst prove the identiﬁability ( i.e. , the possibility to be precisely inferred up to an equivalence relation) of the CIMe.
Revised: Speciﬁcally, we ﬁrst prove the identiﬁability ( i.e. , the possibility to be precisely inferred up to an equivalence relation) of the CIMe.

Original: Notably, far beyond the scope in existing literature (Khemakhem, Kingma and Hyv¨arinen, 2020), our results can implicitly, and are the ﬁrst to disentangle the output-causative factors ( a.k.a , S ) from others ( a.k.a , Z ) for prediction, to ensure the isolation of undesired spurious correlation.
Revised: Notably, far beyond the scope in existing literature (Khemakhem, Kingma and Hyv¨arinen, 2020), our results can implicitly, and are the ﬁrst to disentangle the output-causative factors ( a.k.a , S ) from others ( a.k.a , Z ) for prediction, to ensure the isolation of undesired spurious correlation.

Original: Guaranteed by such, we propose to estimate the CIMe by extending the variational auto-encoder (VAE) (Kingma and Welling, 2014) to the supervised scenario.
Revised: Guaranteed by such, we propose to estimate the CIMe by extending the variational auto-encoder (VAE) (Kingma and Welling, 2014) to the supervised scenario.

Original: For OOD prediction, we propose to optimize over latent space under the identiﬁed CIMe.
Revised: For OOD prediction, we propose to optimize over latent space under the identiﬁed CIMe.

Original: To verify the correctness of our identiﬁability claim, we conduct a simulation experiment.
Revised: To verify the correctness of our identiﬁability claim, we conduct a simulation experiment.

Original: We further demonstrate the utility of our LaCIM via high explainable learned semantic features, improved prediction power on various OOD scenarios (including tasks with confounding and selection bias, healthcare), and robustness on security.
Revised: We further demonstrate the utility of our LaCIM via high explainable learned semantic features, improved prediction power on various OOD scenarios (including tasks with confounding and selection bias, healthcare), and robustness on security.

Original: We summarize our contribution as follows: (i) Methodologically , we propose in section 4.1 a latent causal model in which only a subset of latent components are causally related to the output, to avoid spurious correlation and beneﬁt OOD generalization; (ii) Theoretically , we prove the identiﬁability (in theorem 4.3) of CIMe f x , f y from latent variables to observed data, which disentangles outputcausative factors from others; (iii) Algorithmically , guided by the identiﬁability, we in section 4.
Revised: We summarize our contribution as follows: (i) Methodologically , we propose in section 4.1 a latent causal model in which only a subset of latent components are causally related to the output, to avoid spurious correlation and beneﬁt OOD generalization; (ii) Theoretically , we prove the identiﬁability (in theorem 4.3) of CIMe f x , f y from latent variables to observed data, which disentangles outputcausative factors from others; (iii) Algorithmically , guided by the identiﬁability, we in section 4.

Original: reformulate Variational Bayesian method to estimate CIMe during training and optimize over latent space during the test; (iv) Experimentally , LaCIM outperforms others in terms of prediction power on OOD tasks and interpretability in section 5.2, and robustness to tiny perturbation in section 5.3.
Revised: reformulate Variational Bayesian method to estimate CIMe during training and optimize over latent space during the test; (iv) Experimentally , LaCIM outperforms others in terms of prediction power on OOD tasks and interpretability in section 5.2, and robustness to tiny perturbation in section 5.3.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The invariance/causal learning proposes to learn the assumed invariance for transferring.
Revised: The invariance/causal learning proposes to learn the assumed invariance for transferring.

Original: For the invariance learning methods in Krueger et al.
Revised: For the invariance learning methods in Krueger et al.

Original: and Sch ¨ olkopf (2019), the “invariance” can refer to stable correlation rather than causation, which lacks the interpretability and impedes its generalization to a broader set of domains.
Revised: and Sch ¨ olkopf (2019), the “invariance” can refer to stable correlation rather than causation, which lacks the interpretability and impedes its generalization to a broader set of domains.

Original: For causal learning, Peters et al.
Revised: For causal learning, Peters et al.

Original: ; B ¨ uhlmann (2018); Kuang et al.
Revised: ; B ¨ uhlmann (2018); Kuang et al.

Original: (2018); Heinze-Deml and Meinshausen (2017) assume causal factors as observed input, which is inappropriate for sensory-level observational data.
Revised: (2018); Heinze-Deml and Meinshausen (2017) assume causal factors as observed input, which is inappropriate for sensory-level observational data.

Original: In contrast, our LaCIM introduces latent components as causal factors of the input; more importantly , we explicitly separate them into the output-causative features and others, to avoid spurious correlation.
Revised: In contrast, our LaCIM introduces latent components as causal factors of the input; more importantly , we explicitly separate them into the output-causative features and others, to avoid spurious correlation.

Original: Further, we provide the identiﬁability claim of causal invariant mechanisms.
Revised: Further, we provide the identiﬁability claim of causal invariant mechanisms.

Original: In independent and concurrent works, Teshima et al.
Revised: In independent and concurrent works, Teshima et al.

Original: (2020) and Ilse et al.
Revised: (2020) and Ilse et al.

Original: (2020) also explore latent variables in causal relation.
Revised: (2020) also explore latent variables in causal relation.

Original: As comparisons,
Revised: As comparisons,

Original: Teshima et al.
Revised: Teshima et al.

Original: (2020) did not differentiate S from Z ; and Ilse et al.
Revised: (2020) did not differentiate S from Z ; and Ilse et al.

Original: (2020) proposed to augment intervened data, which can be intractable in real cases.
Revised: (2020) proposed to augment intervened data, which can be intractable in real cases.

Original: Other works which are conceptually related to us, as a non-exhaustive review, include (i) transfer learning which also leverages invariance in the context of domain adaptation (Sch ¨ olkopf et al., 2011;
Revised: Other works which are conceptually related to us, as a non-exhaustive review, include (i) transfer learning which also leverages invariance in the context of domain adaptation (Sch ¨ olkopf et al., 2011;

Original: Zhang et al., 2013;
Revised: Zhang et al., 2013;

Original: Gong et al., 2016) or domain generalization (Li et al., 2018; Shankar et al., 2018); and (ii) causal inference (Pearl, 2009; Peters et al., 2017) which proposes a structural causal model to incorporate intervention via “do-calculus” for cause-effect reasoning and counterfactual learning; (iii) latent generative model which also assumes generation from latent space to observed data (Kingma and Welling, 2014; Suter et al., 2019) but aims at learning generator in the unsupervised scenario.
Revised: Gong et al., 2016) or domain generalization (Li et al., 2018; Shankar et al., 2018); and (ii) causal inference (Pearl, 2009; Peters et al., 2017) which proposes a structural causal model to incorporate intervention via “do-calculus” for cause-effect reasoning and counterfactual learning; (iii) latent generative model which also assumes generation from latent space to observed data (Kingma and Welling, 2014; Suter et al., 2019) but aims at learning generator in the unsupervised scenario.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem Setup & Notation Let X, Y respectively denote the input and output variables.
Revised: Problem Setup & Notation Let X, Y respectively denote the input and output variables.

Original: The training data {D e } e ∈E train are collected from the set of multiple environments E train , where each e is with distribution P e
Revised: The training data {D e } e ∈E train are collected from the set of multiple environments E train , where each e is with distribution P e

Original: ( X, Y ) over X × Y and D e = { x ei , y ei , d e } i ∈
Revised: ( X, Y ) over X × Y and D e = { x ei , y ei , d e } i ∈

Original: [ n e ] i.i.d ∼ P e with [ k ] := { 1 , ..., k } for any k ∈ Z + .
Revised: [ n e ] i.i.d ∼ P e with [ k ] := { 1 , ..., k } for any k ∈ Z + .

Original: The d e ∈ { 0 , 1 } m denotes the one-hot encoded domain index for e , where 1 ≤ m :=
Revised: The d e ∈ { 0 , 1 } m denotes the one-hot encoded domain index for e , where 1 ≤ m :=

Original: (cid:12) (cid:12) E train (cid:12) (cid:12) ≤ n := (cid:80) e ∈E train n e .
Revised: (cid:12) (cid:12) E train (cid:12) (cid:12) ≤ n := (cid:80) e ∈E train n e .

Original: Our goal is to learn a model f : X (cid:55)→ Y that learns output ( y )
Revised: Our goal is to learn a model f : X (cid:55)→ Y that learns output ( y )

Original: -causative factors for prediction and performs well on the set of all environments E ⊃ E train , which is aligned with existing OOD generalization works (Arjovsky et al., 2019; Krueger et al., 2020).
Revised: -causative factors for prediction and performs well on the set of all environments E ⊃ E train , which is aligned with existing OOD generalization works (Arjovsky et al., 2019; Krueger et al., 2020).

Original: We use respectively upper, lower case letter and Cursive letter to denote the random variable, the instance and the space, e.g. , a is an instance in the space A of random variable A .
Revised: We use respectively upper, lower case letter and Cursive letter to denote the random variable, the instance and the space, e.g. , a is an instance in the space A of random variable A .

Original: The [ f ] A denotes the f restricted on dimensions of A .
Revised: The [ f ] A denotes the f restricted on dimensions of A .

Original: The Sobolev space W k,p ( A ) contains all f such that
Revised: The Sobolev space W k,p ( A ) contains all f such that

Original: Structural Causal Model.
Revised: Structural Causal Model.

Original: The structural causal model (SCM) is deﬁned as the causal graph assigned with structural equations.
Revised: The structural causal model (SCM) is deﬁned as the causal graph assigned with structural equations.

Original: The causal graph encodes the assumptions in missing arrows in a directed acylic graph (DAG): G =
Revised: The causal graph encodes the assumptions in missing arrows in a directed acylic graph (DAG): G =

Original: ( V, E ) with V, E respectively denoting the node set and the edge set.
Revised: ( V, E ) with V, E respectively denoting the node set and the edge set.

Original: The
Revised: The

Original: Pa ( k ) denotes the set of parent nodes of V k for each V k ∈ V and the X → Y ∈ E indicates the causal effect of X on Y .
Revised: Pa ( k ) denotes the set of parent nodes of V k for each V k ∈ V and the X → Y ∈ E indicates the causal effect of X on Y .

Original: The structural equations { V k ← f k ( Pa ( k ) , ε k ) } V k ∈ V , quantify the causal effects shown in the causal graph G .
Revised: The structural equations { V k ← f k ( Pa ( k ) , ε k ) } V k ∈ V , quantify the causal effects shown in the causal graph G .

Original: By assuming independence among exogenous variables { ε k } k , the Causal Markov Condition states that P ( { V k
Revised: By assuming independence among exogenous variables { ε k } k , the Causal Markov Condition states that P ( { V k

Original: = v k } V k ∈ V ) =
Revised: = v k } V k ∈ V ) =

Original: Π k P ( V k
Revised: Π k P ( V k

Original: =
Revised: =

Original: v
Revised: v

Original: k
Revised: k

Original: | Pa ( k ) =
Revised: | Pa ( k ) =

Original: pa ( k )) .
Revised: pa ( k )) .

Original: A back-door path from V a to V b is deﬁned as a path that ends with an arrow pointing to V a (Pearl, 2009).
Revised: A back-door path from V a to V b is deﬁned as a path that ends with an arrow pointing to V a (Pearl, 2009).

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: We build our causal model associated with C ausal I nvariant Me chanism (CIMe, i.e. , f x , f y ) and a priori about the generating process in section 4.1, followed by our identiﬁability result for CIMe in section 4.2, and ﬁnished by learning method to estimate CIMe in section 4.3.
Revised: We build our causal model associated with C ausal I nvariant Me chanism (CIMe, i.e. , f x , f y ) and a priori about the generating process in section 4.1, followed by our identiﬁability result for CIMe in section 4.2, and ﬁnished by learning method to estimate CIMe in section 4.3.

Original: 4. L A TENT C AUSAL I NVARIANCE M ODEL
Revised: 4. L A TENT C AUSAL I NVARIANCE M ODEL

Original: The variables marked by white (gray) color represent the unobserved (observed) variables.
Revised: The variables marked by white (gray) color represent the unobserved (observed) variables.

Original: Each arrow represents the causal effect from the variable it points from on the one it points to.
Revised: Each arrow represents the causal effect from the variable it points from on the one it points to.

Original: The C denotes the confounder of
Revised: The C denotes the confounder of

Original: S, Z , which are the causal factors of
Revised: S, Z , which are the causal factors of

Original: X, Y .
Revised: X, Y .

Original: The D denotes domain index, which varies across domains and characterizes the distribution of C .
Revised: The D denotes domain index, which varies across domains and characterizes the distribution of C .

Original: We introduce latent variables to model the abstractions/concepts that play as causal factors that generate the observed variables ( X, Y ) , which is more reasonable than assuming the X as the direct cause of Y in scenarios with sensory-level data.
Revised: We introduce latent variables to model the abstractions/concepts that play as causal factors that generate the observed variables ( X, Y ) , which is more reasonable than assuming the X as the direct cause of Y in scenarios with sensory-level data.

Original: We explicitly separate the latent variables into two parts: the S and Z that respectively denote the y (output)-causative and y -non-causative factors, as shown by the arrow S → Y in Fig.
Revised: We explicitly separate the latent variables into two parts: the S and Z that respectively denote the y (output)-causative and y -non-causative factors, as shown by the arrow S → Y in Fig.

Original: Besides, the X and Y are respectively generated by S, Z and S , via structural equations (with noise) f x , f y , which are denoted as C ausal I nvariant
Revised: Besides, the X and Y are respectively generated by S, Z and S , via structural equations (with noise) f x , f y , which are denoted as C ausal I nvariant

Original: Me chanisms (CIMe) that hold across all domains.
Revised: Me chanisms (CIMe) that hold across all domains.

Original: The output Y denotes the label generated by human knowledge, e.g. , the semantic shape, the contour to discern the object, etc.
Revised: The output Y denotes the label generated by human knowledge, e.g. , the semantic shape, the contour to discern the object, etc.

Original: Hence, we assume the Y as the outcome/effect of these high-level abstractions (Biederman, 1987) rather than the cause (detailed comparison with Y → S is left in supplementary 7.7.1).
Revised: Hence, we assume the Y as the outcome/effect of these high-level abstractions (Biederman, 1987) rather than the cause (detailed comparison with Y → S is left in supplementary 7.7.1).

Original: We call the model associated with the causal graph in Fig.
Revised: We call the model associated with the causal graph in Fig.

Original: 1 as La tent C ausal
Revised: 1 as La tent C ausal

Original: I nvariance M odel (LaCIM), with formal deﬁnition given in Def. 4.1.
Revised: I nvariance M odel (LaCIM), with formal deﬁnition given in Def. 4.1.

Original: As an illustration, we consider the image classiﬁcation in which X, Y denote the image and  class label.
Revised: As an illustration, we consider the image classiﬁcation in which X, Y denote the image and the class label.

Original: Instead of X , i.e. , the pixels, it is more reasonable to assume the causal factors (of X, Y ) as latent concepts ( S, Z ) that can denote light, angle, the shape of the object to generate X following the physical mechanisms.
Revised: Instead of X , i.e. , the pixels, it is more reasonable to assume the causal factors (of X, Y ) as latent concepts ( S, Z ) that can denote light, angle, the shape of the object to generate X following the physical mechanisms.

Original: Among these concepts, only the ones that are related to the object, i.e. ,
Revised: Among these concepts, only the ones that are related to the object, i.e. ,

Original: S ( e.g. , shape) are causal factors of the object label, i.e. , Y .
Revised: S ( e.g. , shape) are causal factors of the object label, i.e. , Y .

Original: Following the physical or natural law, the mechanisms S, Z → X, S
Revised: Following the physical or natural law, the mechanisms S, Z → X, S

Original: → Y invariantly hold across domains.
Revised: → Y invariantly hold across domains.

Original: The S :=
Revised: The S :=

Original: R q s , Z :=
Revised: R q s , Z :=

Original: R q z denote the space of S, Z , with P e ( S, Z ) varies across E ( e.g. , the object is more associated with some speciﬁc types of scenes, as illustrated by the example in the subsequent paragraph).
Revised: R q z denote the space of S, Z , with P e ( S, Z ) varies across E ( e.g. , the object is more associated with some speciﬁc types of scenes, as illustrated by the example in the subsequent paragraph).

Original: We assume that the y -non-causative factor ( i.e. , Z ) is associated with (but not causally related to) S , Y through the confounder C , which is allowed to take a speciﬁc value for each sample unit.
Revised: We assume that the y -non-causative factor ( i.e. , Z ) is associated with (but not causally related to) S , Y through the confounder C , which is allowed to take a speciﬁc value for each sample unit.

Original: Therefore, the back-door path Z ← C → S → Y induces the correlation between Z and Y in each single domain.
Revised: Therefore, the back-door path Z ← C → S → Y induces the correlation between Z and Y in each single domain.

Original: Rather than invariant causation, this correlation is data-dependent and can vary across domains, which is known as “spurious correlation” .
Revised: Rather than invariant causation, this correlation is data-dependent and can vary across domains, which is known as “spurious correlation” .

Original: In real applications, this spurious correlation corresponds to the bias inherited from data, e.g. the contextual information in object classiﬁcation.
Revised: In real applications, this spurious correlation corresponds to the bias inherited from data, e.g. the contextual information in object classiﬁcation.

Original: The change of this S - Z correlation, can be explained by an additional source variable D , which takes a speciﬁc value for each domain and functions the prior of distribution of  C , as illustrated in Fig.
Revised: This domain-speciﬁc S - Z correlation, can be explained by the source variable D , which takes a speciﬁc value for each domain and functions the prior of distribution of the confounder C , as illustrated in Fig.

Original: This source variable D can refer to attributes/parameters that characterize the distribution of S, Z in each domain.
Revised: This source variable D can refer to attributes/parameters that characterize the distribution of S, Z in each domain.

Original: When such attributes cannot be observed, wecan use the domain index as a substitute.
Revised: When such attributes are unobserved, we use the domain index as a substitute.

Original: Consider the cat/dog classiﬁcation task as an illustration, the animal in each image is either associated with the snow or grass.
Revised: Consider the cat/dog classiﬁcation task as an illustration, the animal in each image is either associated with the snow or grass.

Original: The S, Z respectively denote the concepts of animals and scenes.
Revised: The S, Z respectively denote the concepts of animals and scenes.

Original: The D denotes the sampler, which can be characterized as the ratio of all types of (scenes,animals).
Revised: The D denotes the sampler, which can be characterized as the ratio of all types of (scenes,animals).

Original: The D generates the C that denotes the (time, weather) to go outside and collect samples.
Revised: The D generates the C that denotes the (time, weather) to go outside and collect samples.

Original: Since each sampler may have a ﬁxed pattern ( e.g. gets used to going out in the sunny morning (or in the snowy evening)), the data he/she collects, may have sample selection bias ( e.g. with dogs (cats) more associated with grass (snow) in the sunny morning (or snowy evening) ).
Revised: Since each sampler may have a ﬁxed pattern ( e.g. gets used to going out in the sunny morning (or in the snowy evening)), the data he/she collects, may have sample selection bias ( e.g. with dogs (cats) more associated with grass (snow) in the sunny morning (or snowy evening) ).

Original: In this regard, the scene concepts Z can be correlated with the animal concepts S , and also the label Y .
Revised: In this regard, the scene concepts Z can be correlated with the animal concepts S , and also the label Y .

Original: Deﬁnition 4.1 (LaCIM) .
Revised: Deﬁnition 4.1 (LaCIM) .

Original: The La tent C ausal I nvariance M odel (LaCIM) for e ∈ E is deﬁned as a
Revised: The La tent C ausal I nvariance M odel (LaCIM) for e ∈ E is deﬁned as a

Original: SCM characterized by (i) the causal graph , i.e., the G =
Revised: SCM characterized by (i) the causal graph , i.e., the G =

Original: ( V, E ) with V =
Revised: ( V, E ) with V =

Original: { C, S, Z, X, Y } and
Revised: { C, S, Z, X, Y } and

Original: E =
Revised: E =

Original: { C → S, C
Revised: { C → S, C

Original: → Z, Z → X, S → X, S → Y } ; and (ii) structural equations with causal mechanisms { f c , f z , f s , f x , f y } embodying the quantitative causal information: c ← f c ( d e , ε c ) ,
Revised: → Z, Z → X, S → X, S → Y } ; and (ii) structural equations with causal mechanisms { f c , f z , f s , f x , f y } embodying the quantitative causal information: c ← f c ( d e , ε c ) ,

Original: z ← f z ( c, ε z ) , s ← f s ( c, ε s );
Revised: z ← f z ( c, ε z ) , s ← f s ( c, ε s );

Original: x ← f x ( s, z, ε x ); y ← f y ( s, ε y ) , in which { ε c , ε z , ε s , ε x , ε y } are independent exogenous variables that induce p f c ( c | d e ) , p f z ( z | c ) , p f s ( s | c ) , p f x ( x | s, z ) , p f y
Revised: x ← f x ( s, z, ε x ); y ← f y ( s, ε y ) , in which { ε c , ε z , ε s , ε x , ε y } are independent exogenous variables that induce p f c ( c | d e ) , p f z ( z | c ) , p f s ( s | c ) , p f x ( x | s, z ) , p f y

Original: ( y | s ) .
Revised: ( y | s ) .

Original: The CIMe f x , f y are assumed to be invariant across E .
Revised: The CIMe f x , f y are assumed to be invariant across E .

Original: We call the environment-dependent parts: P e ( S, Z ) and
Revised: We call the environment-dependent parts: P e ( S, Z ) and

Original: P e ( S, Z | X ) as S, Z -prior and S, Z -inference in the following.
Revised: P e ( S, Z | X ) as S, Z -prior and S, Z -inference in the following.

Original: Remark 1.
Revised: Remark 1.

Original: We denote LaCIM- d s and LaCIM- d as two versions of LaCIM, with the source variable
Revised: We denote LaCIM- d s and LaCIM- d as two versions of LaCIM, with the source variable

Original: d s with practical meaning ( e.g. attributes or parameters of P ( S, Z ) ) observed or not.
Revised: d s with practical meaning ( e.g. attributes or parameters of P ( S, Z ) ) observed or not.

Original: The observation of d s can be possible in some applications (e.g., age, gender that characterizes population in medical diagnosis).
Revised: The observation of d s can be possible in some applications (e.g., age, gender that characterizes population in medical diagnosis).

Original: As for the LaCIM- d with d s unobserved, we use domain index D as a substitute.
Revised: As for the LaCIM- d with d s unobserved, we use domain index D as a substitute.

Original: Denote C as the space of C .
Revised: Denote C as the space of C .

Original: We assume that the C is ﬁnite union of disjoint sets {C r } R r =1 , i.e.
Revised: We assume that the C is ﬁnite union of disjoint sets {C r } R r =1 , i.e.

Original: C :=
Revised: C :=

Original: ∪ Rr =1 C r , such that for each c r,i , c r,j ∈ C r , it holds that p ( s, z | c r,i ) =
Revised: ∪ Rr =1 C r , such that for any c r,i (cid:54) = c r,j ∈ C r , it holds that p ( s, z | c r,i ) =

Original: p ( s, z | c r,j ) for any ( s, z ) .
Revised: p ( s, z | c r,j ) for any ( s, z ) .

Original: Returning to the cat/dog classiﬁcation example, the C denotes the range of time to collect samples, i.e. , 00 : 00 - 24 : 00 .
Revised: Returning to the cat/dog classiﬁcation example, the C denotes the range of time to collect samples, i.e. , 00 : 00 - 24 : 00 .

Original: The C can be divided into several time periods C 1 , ..., C R , such that the proportion of (animal,scene) given any c in the same period is unchanged, e.g. , the dog often comes up on the grass in the morning.
Revised: The C can be divided into several time periods C 1 , ..., C R , such that the proportion of (animal,scene) given any c in the same period is unchanged, e.g. , the dog often comes up on the grass in the morning.

Original: Further, since p ( x | s, z ) and p ( y | s ) are invariant across domains, we have for each C r that p ( x, y | c r,i ) =
Revised: Further, since p ( x | s, z ) and p ( y | s ) are invariant across domains, we have for each C r that p ( x, y | c r,i ) =

Original: (cid:82) p ( x | s, z ) p ( y | s ) p ( s, z | c r,i ) dsdz
Revised: (cid:82) p ( x [Equation] s, z ) [Equation]

Original: = (cid:82) p ( x | s, z ) p ( y | s ) p ( s, z | c r,j ) dsdz
Revised: [Equation] [Equation] [Equation] [Equation]

Original:  = p ( x, y | c r,j ) for any ( x, y ) .
Revised: [Equation] [Equation] = p [Equation]

Original:  That is, the { p ( x, y | c r } c r ∈C r for each ( x, y ) collapse to a single point, namely p ( x, y | c r ) .
Revised: x, y [Equation] ) for any ( x, y ) . That is, the { p ( x, y | c r } c r ∈C r for each ( x, y ) collapse to a single point, namely p ( x, y | c r ) .

Original: In this regard, we have p e ( x, y ) :
Revised: In this regard, we have p e ( x, y ) :

Original: = p ( x, y | d e ) = (cid:80)
Revised: = p ( x, y | d e ) = (cid:80)

Original: Rr =1 p ( x, y | c r ) p ( c r | d e ) .
Revised: Rr =1 p ( x, y | c r ) p ( c r | d e ) .

Original: Besides, we assume the Additive Noise Model (ANM) for f x ( s, z, ε x ) , i.e. , f x ( s, z, ε x ) =
Revised: Besides, we assume the Additive Noise Model (ANM) for f x ( s, z, ε x ) , i.e. , f x ( s, z, ε x ) =

Original: ˆ f x ( s, z ) + ε x (we replace ˆ f x with f x without loss of generality), which has been widely adopted to identify the causal factors (Janzing et al., 2009; Peters et al., 2014; Khemakhem, Kingma and Hyv ¨ arinen, 2020).
Revised: ˆ f x ( s, z ) + ε x (we replace ˆ f x with f x without loss of generality), which has been widely adopted to identify the causal factors (Janzing et al., 2009; Peters et al., 2014; Khemakhem, Kingma and Hyv ¨ arinen, 2020).

Original: We need to identify the CIMe ( i.e. , f x , f y ), guaranteed by the identiﬁability that ensures the learning method to distinguish S from Z to avoid spurious correlation, as presented in section 4.2.
Revised: We need to identify the CIMe ( i.e. , f x , f y ), guaranteed by the identiﬁability that ensures the learning method to distinguish S from Z to avoid spurious correlation, as presented in section 4.2.

Original: Traditionally speaking, the identiﬁability means the parameter giving rise to the observational distribution p θ (cid:63) ( x, y | d e ) can be uniquely determined, i.e. , p θ ( x, y | d e ) =
Revised: Traditionally speaking, the identiﬁability means the parameter giving rise to the observational distribution p θ (cid:63) ( x, y | d e ) can be uniquely determined, i.e. , p θ ( x, y | d e ) =

Original: p ˜ θ
Revised: p ˜ θ

Original: ( x, y | d e )
Revised: ( x, y | d e )

Original: = ⇒ θ =
Revised: = ⇒ θ =

Original: ˜ θ .
Revised: ˜ θ .

Original: Instead of strict uniqueness, we rather identify an equivalent class of θ (cid:63) (in Def. 4.2) that sufﬁces to disentangle the y -causative features S from Z to avoid learning spurious correlation.
Revised: Instead of strict uniqueness, we rather identify an equivalent class of θ (cid:63) (in Def. 4.2) that sufﬁces to disentangle the y -causative features S from Z to avoid learning spurious correlation.

Original: To achieve this goal, we ﬁrst narrow our interest in case when p ( s, z | c ) is exponential family in Eq.
Revised: To achieve this goal, we ﬁrst narrow our interest in case when p ( s, z | c ) is exponential family in Eq.

Original: (1), in which we can respectively identify the S, Z up to linear and point-wise transformations given by theorem 4.3; then we generalize to any p ( s, z | c ) as long as it belongs to Sobolev space, as explained in theorem 4.4.
Revised: (1), in which we can respectively identify the S, Z up to linear and point-wise transformations given by theorem 4.3; then we generalize to any p ( s, z | c ) as long as it belongs to Sobolev space, as explained in theorem 4.4.

Original: A reformulated VAE is proposed to learn the CIMe practically.
Revised: A reformulated VAE is proposed to learn the CIMe practically.

Original: For generalization, note that the gap between two environments in terms of prediction given x , i.e. , (cid:12) (cid:12) E p
Revised: For generalization, note that the gap between two environments in terms of prediction given x , i.e. , (cid:12) (cid:12) E p

Original: e 

Revised: e

Original: [ Y | X
Revised: [ Y | X

Original: = x ] − E p
Revised: = x ] − E p

Original: e 

Revised: e

Original: [ Y | X
Revised: [ Y | X

Original: = x ] (cid:12)(cid:12) =
Revised: = x ] (cid:12)(cid:12) =

Original: (cid:82) S (cid:12)(cid:12) p
Revised: (cid:82) S (cid:12)(cid:12) p

Original: e 2 ( s | x ) − p e 1 ( s | x ) (cid:12)(cid:12)
Revised: e 2 ( s | x ) − p e 1 ( s | x ) (cid:12)(cid:12)

Original: p f y
Revised: p f y

Original: ( y | s ) ds , is mainly due to the inconsistency of S, Z -inference, i.e. , p e ( s, z | x ) (cid:54) =
Revised: ( y | s ) ds , is mainly due to the inconsistency of S, Z -inference, i.e. , p e ( s, z | x ) (cid:54) =

Original: p e (cid:48) ( s, z | x ) for e (cid:48) (cid:54) =
Revised: p e (cid:48) ( s, z | x ) for e (cid:48) (cid:54) =

Original: e (for details please refer to theorem 7.1 in supplement 7.1).
Revised: e (for details please refer to theorem 7.1 in supplement 7.1).

Original: Therefore, one cannot directly apply the trained { p e ( s, z | x ) , p e ( y | x ) } e ∈E train to the inference model of new environment, i.e. p e (cid:48)
Revised: Therefore, one cannot directly apply the trained { p e ( s, z | x ) , p e ( y | x ) } e ∈E train to the inference model of new environment, i.e. p e (cid:48)

Original: ( s, z | x ) , p e (cid:48)
Revised: ( s, z | x ) , p e (cid:48)

Original: ( y | x ) for e (cid:48) / ∈ E train .
Revised: ( y | x ) for e (cid:48) / ∈ E train .

Original: To solve this problem and generalize to new environment, we note that since p f x ( x | s, z ) and p f y ( y | s ) are shared among all environments, we propose to inference s, z that give rise to the test sample x via maximizing the identiﬁed p
Revised: To solve this problem and generalize to new environment, we note that since p f x ( x | s, z ) and p f y ( y | s ) are shared among all environments, we propose to inference s, z that give rise to the test sample x via maximizing the identiﬁed p

Original: f x ( x | s, z ) , as a pseudo-likelihood of x given ( s, z ) , rather than using S, Z -inference model which is inconsistent among environments.
Revised: f x ( x | s, z ) , as a pseudo-likelihood of x given ( s, z ) , rather than using S, Z -inference model which is inconsistent among environments.

Original: Then, we feed estimated s into invariant predictor p f y ( y | s ) for prediction.
Revised: Then, we feed estimated s into invariant predictor p f y ( y | s ) for prediction.

Original: 4. I DENTIFIABILITY OF C AUSAL I NVARIANT M ECHANISMS
Revised: 4. I DENTIFIABILITY OF C AUSAL I NVARIANT M ECHANISMS

Original: We present the identiﬁability claim about the CIMe f x , f y , which implicitly distinguishes the ycausative factors ( a.k.a , S ) from others ( a.k.a , Z ) for prediction, to provide a theoretical guarantee for avoiding spurious correlations.
Revised: We present the identiﬁability claim about the CIMe f x , f y , which implicitly distinguishes the ycausative factors ( a.k.a , S ) from others ( a.k.a , Z ) for prediction, to provide a theoretical guarantee for avoiding spurious correlations.

Original: Notably, the S and Z play “asymmetric roles” in terms of generating process, as reﬂected in additional generating ﬂow from S to Y .
Revised: Notably, the S and Z play “asymmetric roles” in terms of generating process, as reﬂected in additional generating ﬂow from S to Y .

Original: This “information intersection” property of S , i.e. , f − 1 y (¯ y ) =
Revised: This “information intersection” property of S , i.e. , f − 1 y (¯ y ) =

Original: [ f − 1 x ] S (¯ x ) for any (¯ x, ¯ y ) ∈
Revised: [ f − 1 x ] S (¯ x ) for any (¯ x, ¯ y ) ∈

Original: f x ( S , Z ) × f y ( S )
Revised: f x ( S , Z ) × f y ( S )

Original: if y = f y ( s ) + ε y , is exploited to disentangle S from Z .
Revised: if y = f y ( s ) + ε y , is exploited to disentangle S from Z .

Original: Such a disentanglement analysis, is crucial to causal prediction but lacked in existing literature about identiﬁability, such as those identifying the discrete latent confounders (Janzing, Sgouritsa, Stegle, Peters and Sch¨olkopf, 2012; Sgouritsa et al., 2013); or those relying on ANM assumption (Janzing, Peters, Mooij and Sch¨olkopf, 2012); linear ICA (Eriksson and Koivunen, 2003); (Khemakhem, Kingma and Hyv¨arinen, 2020; Khemakhem, Monti, Kingma and Hyv¨arinen, 2020;
Revised: Such a disentanglement analysis, is crucial to causal prediction but lacked in existing literature about identiﬁability, such as those identifying the discrete latent confounders (Janzing, Sgouritsa, Stegle, Peters and Sch¨olkopf, 2012; Sgouritsa et al., 2013); or those relying on ANM assumption (Janzing, Peters, Mooij and Sch¨olkopf, 2012); linear ICA (Eriksson and Koivunen, 2003); (Khemakhem, Kingma and Hyv¨arinen, 2020; Khemakhem, Monti, Kingma and Hyv¨arinen, 2020;

Original: Teshima et al., 2020) (Please refer to supplement 7.6 for more broad reviews).
Revised: Teshima et al., 2020) (Please refer to supplement 7.6 for more broad reviews).

Original: Besides, our analysis extends the scope of Khemakhem, Kingma and Hyv¨arinen (2020) to categorical
Revised: Besides, our analysis extends the scope of Khemakhem, Kingma and Hyv¨arinen (2020) to categorical

Original: Y and general forms of P ( S, Z ) that belongs to Sobolev space, in theorem 4.4.
Revised: Y and general forms of P ( S, Z ) that belongs to Sobolev space, in theorem 4.4.

Original: Note that our analysis does NOT require observing the original source variable d s .
Revised: Note that our analysis does NOT require observing the original source variable d s .

Original: We ﬁrst narrow our interest to a family class of LaCIM denoted as P exp such that any p ∈ P exp satisﬁes that (i) the S,
Revised: We ﬁrst narrow our interest to a family class of LaCIM denoted as P exp such that any p ∈ P exp satisﬁes that (i) the S,

Original: Z belong to the exponential family; and that (ii) the Y is generated from the
Revised: Z belong to the exponential family; and that (ii) the Y is generated from the

Original: ANM.
Revised: ANM.

Original: We will show later that P exp can approximate any P ( S, Z ) ∈ W r, 2 ( S × Z ) for some r ≥ 2 :
Revised: We will show later that P exp can approximate any P ( S, Z ) ∈ W r, 2 ( S × Z ) for some r ≥ 2 :

Original: P exp = (cid:8) LaCIM with y = f
Revised: P exp = (cid:8) LaCIM with y = f

Original: y ( s ) + ε y , p ( s, z | c ) :
Revised: y ( s ) + ε y , p ( s, z | c ) :

Original: = p T z , Γ zc ( z | c ) p T s , Γ sc
Revised: = p T z , Γ zc ( z | c ) p T s , Γ sc

Original: ( s | c ) (cid:9) with p T t , Γ tc ( t ) :
Revised: ( s | c ) (cid:9) with p T t , Γ tc ( t ) :

Original: =
Revised: =

Original: q t (cid:89) i = exp (cid:16)
Revised: q t (cid:89) i = exp (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: j =
Revised: j =

Original: T ti,j ( t i )Γ tc,i,j + B i ( t i )
Revised: T ti,j ( t i )Γ tc,i,j + B i ( t i )

Original: − A tc,i (cid:17) for t =
Revised: − A tc,i (cid:17) for t =

Original: s, z, and e ∈ E ,
Revised: s, z, and e ∈ E ,

Original: [Equation]
Revised: [Equation]

Original: 1) where { T t i,j ( t i ) } , { Γ t c,i,j } denote the sufﬁcient statistics and natural parameters, { B i } , { A t c,i } denote the base measures and normalizing constants to ensure the integral of distribution equals to 1.
Revised: 1) where { T t i,j ( t i ) } , { Γ t c,i,j } denote the sufﬁcient statistics and natural parameters, { B i } , { A t c,i } denote the base measures and normalizing constants to ensure the integral of distribution equals to 1.

Original: Let T t (
Revised: Let T t (

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the ∼ p -identiﬁability for θ :=
Revised: We deﬁne the ∼ p -identiﬁability for θ :=

Original: { f x , f y , T s , T z } as:
Revised: { f x , f y , T s , T z } as:

Original: Deﬁnition 4.2 ( ∼ p -identiﬁability) .
Revised: Deﬁnition 4.2 ( ∼ p -identiﬁability) .

Original: We deﬁne a binary relation on the parameter space of X × Y : θ ∼ p ˜
Revised: We deﬁne a binary relation on the parameter space of X × Y : θ ∼ p ˜

Original: θ if there exist two sets of permutation matrices and vectors, ( M s , a s ) and ( M z , a z ) for s and z respectively, such that for any ( x, y ) ∈ X × Y ,
Revised: θ if there exist two sets of permutation matrices and vectors, ( M s , a s ) and ( M z , a z ) for s and z respectively, such that for any ( x, y ) ∈ X × Y ,

Original: [Equation]
Revised: [Equation]

Original: We say that θ is ∼ p -identiﬁable, if for any ˜ θ , p eθ ( x, y ) =
Revised: We say that θ is ∼ p -identiﬁable, if for any ˜ θ , p eθ ( x, y ) =

Original: p e ˜ θ ( x, y ) ∀ e ∈ E train , implies θ ∼ p ˜ θ .
Revised: p e ˜ θ ( x, y ) ∀ e ∈ E train , implies θ ∼ p ˜ θ .

Original: It can be shown that ∼ p satisﬁes the reﬂective property ( θ ∼ p θ ), the symmetric property ( if θ ∼ p ˜ θ then ˜ θ ∼ p θ ), and the transitive property ( if θ 1 ∼ p θ 2 and θ 2 ∼ p θ 3 , then θ 1 ∼ p θ 3 ), and hence is an equivalence relation (details in supplement 7.2).
Revised: It can be shown that ∼ p satisﬁes the reﬂective property ( θ ∼ p θ ), the symmetric property ( if θ ∼ p ˜ θ then ˜ θ ∼ p θ ), and the transitive property ( if θ 1 ∼ p θ 2 and θ 2 ∼ p θ 3 , then θ 1 ∼ p θ 3 ), and hence is an equivalence relation (details in supplement 7.2).

Original: This deﬁnition states that the S, Z can be identiﬁed up to permutation and point-wise transformation, which is sufﬁcient for disentanglement of S and identifying the predicting mechanism p f y
Revised: This deﬁnition states that the S, Z can be identiﬁed up to permutation and point-wise transformation, which is sufﬁcient for disentanglement of S and identifying the predicting mechanism p f y

Original: ( y | [ f − 1 x ] S ( x )) .
Revised: ( y | [ f − 1 x ] S ( x )) .

Original: Speciﬁcally, the deﬁnition regarding f x implies the separation of S and Z unless the extreme case when S can be represented by Z , i.e. , there exists a function h : S → Z such that [ f − 1 x ] S ( x ) =
Revised: Speciﬁcally, the deﬁnition regarding f x implies the separation of S and Z unless the extreme case when S can be represented by Z , i.e. , there exists a function h : S → Z such that [ f − 1 x ] S ( x ) =

Original: h
Revised: h

Original: ([ f − 1 x ] Z ( x )) .
Revised: ([ f − 1 x ] Z ( x )) .

Original: This deﬁnition is inspired by but beyond the scope of unsupervised scenario considered in nonlinear ICA
Revised: This deﬁnition is inspired by but beyond the scope of unsupervised scenario considered in nonlinear ICA

Original: (Hyv ¨ arinen et al., 2019; Khemakhem, Kingma and Hyv ¨ arinen, 2020) to further distinguish of S from Z .
Revised: (Hyv ¨ arinen et al., 2019; Khemakhem, Kingma and Hyv ¨ arinen, 2020) to further distinguish of S from Z .

Original: Besides, the p f y ( y | [ f − 1 x ] S ( x )) =
Revised: Besides, the p f y ( y | [ f − 1 x ] S ( x )) =

Original: p ˜ f
Revised: p ˜ f

Original: y ( y |
Revised: y ( y |

Original: [ ˜ f − 1 x ] S ( x )) further guarantees the identiﬁability of prediction: predict using f y ( s ) with s obtained from f x .
Revised: [ ˜ f − 1 x ] S ( x )) further guarantees the identiﬁability of prediction: predict using f y ( s ) with s obtained from f x .

Original: The following theorem presents the ∼ p -identiﬁability for P exp :
Revised: The following theorem presents the ∼ p -identiﬁability for P exp :

Original: Theorem 4.3 ( ∼ p -identiﬁability) .
Revised: Theorem 4.3 ( ∼ p -identiﬁability) .

Original: For θ in the LaCIM p eθ ( x, y ) ∈ P exp for any e ∈ E train , we assume that i) CIMe satisﬁes that f x , f (cid:48) x and f (cid:48)(cid:48) x are continuous and that f x , f y are bijective; ii) the T ti,j are twice differentiable for any t =
Revised: For θ in the LaCIM p eθ ( x, y ) ∈ P exp for any e ∈ E train , we assume that i) CIMe satisﬁes that f x , f (cid:48) x and f (cid:48)(cid:48) x are continuous and that f x , f y are bijective; ii) the T ti,j are twice differentiable for any t =

Original: s, z, i ∈
Revised: s, z, i ∈

Original: [ q t ] , j ∈ [ k t ] ; iii) the exogenous variables satisfy that the characteristic functions of ε x , ε y are almost everywhere nonzero.
Revised: [ q t ] , j ∈ [ k t ] ; iii) the exogenous variables satisfy that the characteristic functions of ε x , ε y are almost everywhere nonzero.

Original: Under the diversity condition on A :
Revised: Under the diversity condition on A :

Original: =
Revised: =

Original: [ P (cid:62) d e 1 , ..., P (cid:62) d em ] (cid:62) ∈
Revised: [ P (cid:62) d e 1 , ..., P (cid:62) d em ] (cid:62) ∈

Original: R m × R with P d e :=
Revised: R m × R with P d e :=

Original: [ p ( c 1 | d e ) , ..., p ( c R | d e ] that the A and (cid:2)
Revised: [ p ( c 1 | d e ) , ..., p ( c R | d e ] that the A and (cid:2)

Original: [ Γ t = s,z c− Γ t =
Revised: [ Γ t = s,z c− Γ t =

Original: s,z c] T , ..., [ Γ t =
Revised: s,z c] T , ..., [ Γ t =

Original: s,z c R − Γ t =
Revised: s,z c R − Γ t =

Original: s,z c] T (cid:3) T have full column rank for both t
Revised: s,z c] T (cid:3) T have full column rank for both t

Original: = s and t
Revised: = s and t

Original: =
Revised: =

Original: z , we have that the θ :=
Revised: z , we have that the θ :=

Original: { f x , f y , T s , T z } are ∼ p identiﬁable.
Revised: { f x , f y , T s , T z } are ∼ p identiﬁable.

Original: The bijectivity of f x and f y has been widely assumed in Janzing et al.
Revised: The bijectivity of f x and f y has been widely assumed in Janzing et al.

Original: ; Peters et al.
Revised: ; Peters et al.

Original: (2014; 2017); Khemakhem, Kingma and Hyv¨arinen (2020);
Revised: (2014; 2017); Khemakhem, Kingma and Hyv¨arinen (2020);

Original: Teshima et al.
Revised: Teshima et al.

Original: as a basic condition for identiﬁability.
Revised: as a basic condition for identiﬁability.

Original: It naturally holds for f x to be bijective since the latent components S, Z , as high-level abstractions which can be viewed as embeddings in auto-encoder (Kramer, 1991), lies in lower-dimensional space compared with input X which is supposed to have more variations, i.e. , ( q s + q z < q x ).
Revised: It naturally holds for f x to be bijective since the latent components S, Z , as high-level abstractions which can be viewed as embeddings in auto-encoder (Kramer, 1991), lies in lower-dimensional space compared with input X which is supposed to have more variations, i.e. , ( q s + q z < q x ).

Original: For categorical Y , the f y which generates the classiﬁcation result, i.e. , p ( y = k | s ) =
Revised: For categorical Y , the f y which generates the classiﬁcation result, i.e. , p ( y = k | s ) =

Original: [ f y ] k ( s ) / ( (cid:80) k
Revised: [ f y ] k ( s ) / ( (cid:80) k

Original: [ f y ] k ( s )) , will be shown later to be identiﬁable.
Revised: [ f y ] k ( s )) , will be shown later to be identiﬁable.

Original: The diversity condition implies that i) m ≥ R ≥ max( k z ∗ q z , k s ∗ q s ) + 1 ; and that ii) different environments are variant enough in terms of S - Z correlation (which is also assumed in Arjovsky et al.
Revised: The diversity condition implies that i) m ≥ R ≥ max( k z ∗ q z , k s ∗ q s ) + 1 ; and that ii) different environments are variant enough in terms of S - Z correlation (which is also assumed in Arjovsky et al.

Original: (2019)), as a necessary for the invariant one to be identiﬁed.
Revised: (2019)), as a necessary for the invariant one to be identiﬁed.

Original: As noted in the formulation, a larger m would be easier to satisfy the condition, which agrees with the intuition that more environments can provide more complementary information for the identiﬁcation of the invariant mechanisms.
Revised: As noted in the formulation, a larger m would be easier to satisfy the condition, which agrees with the intuition that more environments can provide more complementary information for the identiﬁcation of the invariant mechanisms.

Original: Remark 2.
Revised: Remark 2.

Original: The dimensions of the ground-truth S, Z are unknown, making the check about whether m is large enough impossible.
Revised: The dimensions of the ground-truth S, Z are unknown, making the check about whether m is large enough impossible.

Original: Besides, in some real applications, the training environments are passively observed and may not satisfy the condition.
Revised: Besides, in some real applications, the training environments are passively observed and may not satisfy the condition.

Original: However, we empirically ﬁnd the improvement of LaCIM in terms of both OOD prediction and interpretability, if the multiple environments provided are diverse enough.
Revised: However, we empirically ﬁnd the improvement of LaCIM in terms of both OOD prediction and interpretability, if the multiple environments provided are diverse enough.

Original: Besides, a training environment can be the mixture of many sub-environments, which motivates to splitting the data according to their source ID or clustering results (Teney et al., 2020) to obtain more environments, making the condition easier to satisfy.
Revised: Besides, a training environment can be the mixture of many sub-environments, which motivates to splitting the data according to their source ID or clustering results (Teney et al., 2020) to obtain more environments, making the condition easier to satisfy.

Original: Extension to the general form of LaCIM.
Revised: Extension to the general form of LaCIM.

Original: We generalize the identiﬁable result in theorem 4.3 to any LaCIM as long as its P ( S, Z ) ∈ W r, 2 ( S × Z ) (for some r ≥ 2 ) and categorical Y , in the following theorem.
Revised: We generalize the identiﬁable result in theorem 4.3 to any LaCIM as long as its P ( S, Z ) ∈ W r, 2 ( S × Z ) (for some r ≥ 2 ) and categorical Y , in the following theorem.

Original: This is accomplished by showing that any such LaCIM can be approximated by a sequence of distributions in P exp , motivated by the facts in Barron and Sheu (1991) that the exponential family is dense in the set of distributions with bounded support, and in Maddison et al.
Revised: This is accomplished by showing that any such LaCIM can be approximated by a sequence of distributions in P exp , motivated by the facts in Barron and Sheu (1991) that the exponential family is dense in the set of distributions with bounded support, and in Maddison et al.

Original: (2016) that the continuous variable with multinomial logit model can be approximated by a series of distributions with i.i.d Gumbel noise as the temperature converges to inﬁnity.
Revised: (2016) that the continuous variable with multinomial logit model can be approximated by a series of distributions with i.i.d Gumbel noise as the temperature converges to inﬁnity.

Original: Theorem 4.4 (Asymptotic ∼ p -identiﬁability) .
Revised: Theorem 4.4 (Asymptotic ∼ p -identiﬁability) .

Original: Consider a LaCIM P satisfying that p f x ( x | s, z ) and p f y
Revised: Consider a LaCIM P satisfying that p f x ( x | s, z ) and p f y

Original: ( y | s ) are smooth w.r.t s, z and s respectively.
Revised: ( y | s ) are smooth w.r.t s, z and s respectively.

Original: Suppose P e ( S, Z
Revised: Suppose P e ( S, Z

Original: ) ∈ W r, 2 ( S × Z ) for some r ≥ 2 , we have that P is asymptotically ∼ p -identiﬁable deﬁned as: ∀ (cid:15) > 0 , ∃ ∼ p -identiﬁable ˜ P θ ∈ P exp , s.t. d Pok ( p e ( x, y ) ,
Revised: ) ∈ W r, 2 ( S × Z ) for some r ≥ 2 , we have that P is asymptotically ∼ p -identiﬁable deﬁned as: ∀ (cid:15) > 0 , ∃ ∼ p -identiﬁable ˜ P θ ∈ P exp , s.t. d Pok ( p e ( x, y ) ,

Original: ˜ p eθ ( x, y )) < (cid:15), ∀ e ∈ E train , ( x, y ) ∈
Revised: ˜ p eθ ( x, y )) < (cid:15), ∀ e ∈ E train , ( x, y ) ∈

Original: X × Y 1 .
Revised: X × Y 1 .

Original: 4. C AUSAL S UPERVISED V ARIATIONAL A UTO -E NCODER
Revised: 4. C AUSAL S UPERVISED V ARIATIONAL A UTO -E NCODER

Original: Guided by identiﬁability of CIMe f x , f y , we ﬁrst provide the training method by reformulating VAE in a supervised scenario, followed by optimization over latent space for inference and test.
Revised: Guided by identiﬁability of CIMe f x , f y , we ﬁrst provide the training method by reformulating VAE in a supervised scenario, followed by optimization over latent space for inference and test.

Original: Training.
Revised: Training.

Original: To learn the CIMe and p f x ( x | s, z ) , p f y ( y | s ) for invariant prediction, we implement the generative model to learn { p e ( x, y ) }
Revised: To learn the CIMe and p f x ( x | s, z ) , p f y ( y | s ) for invariant prediction, we implement the generative model to learn { p e ( x, y ) }

Original: e ∈E train , which has been guaranteed by identiﬁability result in theorem 4.3, 4.4 to identify the predicting mechanism.
Revised: e ∈E train , which has been guaranteed by identiﬁability result in theorem 4.3, 4.4 to identify the predicting mechanism.

Original: Speciﬁcally, we reformulate the objective of VAE, as a generative model proposed in (Kingma and Welling, 2014), in supervised scenario.
Revised: Speciﬁcally, we reformulate the objective of VAE, as a generative model proposed in (Kingma and Welling, 2014), in supervised scenario.

Original: For unsupervised learning, the VAE introduces the variational distribution q ψ parameterized by ψ to approximate the intractable posterior by maximizing the following E vidence L ower B ound (ELBO): −L φ,ψ = E p ( x ) (cid:2)
Revised: For unsupervised learning, the VAE introduces the variational distribution q ψ parameterized by ψ to approximate the intractable posterior by maximizing the following E vidence L ower B ound (ELBO): −L φ,ψ = E p ( x ) (cid:2)

Original: E q ψ ( z | x ) log p φ ( x,z ) q ψ ( z | x ) (cid:3) , as a tractable surrogate of maximum likelihood E p ( x ) log p φ ( x ) .
Revised: E q ψ ( z | x ) log p φ ( x,z ) q ψ ( z | x ) (cid:3) , as a tractable surrogate of maximum likelihood E p ( x ) log p φ ( x ) .

Original: Speciﬁcally, the ELBO is less than and equal to E p ( x ) (cid:2) log p
Revised: Speciﬁcally, the ELBO is less than and equal to E p ( x ) (cid:2) log p

Original: φ ( x ) (cid:3) and the equality can only be achieved when q ψ ( z | x )=
Revised: φ ( x ) (cid:3) and the equality can only be achieved when q ψ ( z | x )=

Original: p φ ( z | x ) .
Revised: p φ ( z | x ) .

Original: Therefore, maximizing the ELBO over p φ and q ψ will drive (i) q ψ ( z | x ) to learn p φ ( z | x ) ; (ii) p φ to learn the ground-truth model p (including p φ ( x | z ) to learn p ( x | z ) ).
Revised: Therefore, maximizing the ELBO over p φ and q ψ will drive (i) q ψ ( z | x ) to learn p φ ( z | x ) ; (ii) p φ to learn the ground-truth model p (including p φ ( x | z ) to learn p ( x | z ) ).

Original: In our supervised scenario, we introduce the variational distribution q eψ ( s, z | x, y ) and the corresponding ELBO for any e is −L eφ,ψ = E p e ( x,y ) (cid:2)
Revised: In our supervised scenario, we introduce the variational distribution q eψ ( s, z | x, y ) and the corresponding ELBO for any e is −L eφ,ψ = E p e ( x,y ) (cid:2)

Original: E q eψ ( s,z | x,y ) log p eφ ( x,y,s,z ) q eψ ( s,z | x,y ) (cid:3) .
Revised: E q eψ ( s,z | x,y ) log p eφ ( x,y,s,z ) q eψ ( s,z | x,y ) (cid:3) .

Original: Similarly, minimizing L eφ,ψ can drive p φ ( x | s, z ) , p φ ( y | s ) to learn the CIMe ( i.e. p f x ( x | s, z ) , p f y ( y | s ) ), and also q eψ ( s, z | x, y ) to learn p eφ ( s, z | x, y ) .
Revised: Similarly, minimizing L eφ,ψ can drive p φ ( x | s, z ) , p φ ( y | s ) to learn the CIMe ( i.e. p f x ( x | s, z ) , p f y ( y | s ) ), and also q eψ ( s, z | x, y ) to learn p eφ ( s, z | x, y ) .

Original: In other words, the q ψ can inherit the properties of p φ .
Revised: In other words, the q ψ can inherit the properties of p φ .

Original: As p eφ ( s, z | x, y )
Revised: As p eφ ( s, z | x, y )

Original: =
Revised: =

Original: p eφ ( s,z | x ) p φ ( y | s ) p eφ
Revised: p eφ ( s,z | x ) p φ ( y | s ) p eφ

Original: ( y | x ) for our DAG in Fig.
Revised: ( y | x ) for our DAG in Fig.

Original: 1, we can similarly reparameterize q eψ ( s, z | x, y ) as q eψ ( s,z | x ) q ψ ( y | s ) q eψ ( y | x ) .
Revised: 1, we can similarly reparameterize q eψ ( s, z | x, y ) as q eψ ( s,z | x ) q ψ ( y | s ) q eψ ( y | x ) .

Original: According to Causal Markov Condition, we have that p eφ ( x, y, s, z ) =
Revised: According to Causal Markov Condition, we have that p eφ ( x, y, s, z ) =

Original: p φ ( x | s, z ) p eφ ( s, z ) p φ ( y | s ) .
Revised: p φ ( x | s, z ) p eφ ( s, z ) p φ ( y | s ) .

Original: Substituting the above reparameterizations into the ELBO with q ψ ( y | s ) replaced by p φ ( y | s ) , the L eφ,ψ can be rewritten as:
Revised: Substituting the above reparameterizations into the ELBO with q ψ ( y | s ) replaced by p φ ( y | s ) , the L eφ,ψ can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where q eψ ( y | x ) = (cid:82) S q eψ ( s | x ) p φ ( y | s ) ds .
Revised: where q eψ ( y | x ) = (cid:82) S q eψ ( s | x ) p φ ( y | s ) ds .

Original: The overall loss function is: L φ,ψ = (cid:80) e ∈E train L eφ,ψ .
Revised: The overall loss function is: L φ,ψ = (cid:80) e ∈E train L eφ,ψ .

Original: The training datasets {D e } e ∈E train are applied to optimize prior model p eφ ( s, z ) , inference
Revised: The training datasets {D e } e ∈E train are applied to optimize prior model p eφ ( s, z ) , inference

Original: [Table]
Revised: [Table]

Original: shared
Revised: shared

Original: [Table] [Table]
Revised: [Table] [Table]

Original: respec
Revised: respec

Original: [Table]
Revised: [Table]

Original: tively
Revised: tively

Original: [Table]
Revised: [Table]

Original: Inference & Test.
Revised: Inference & Test.

Original: When d e (cid:48) s can be acquired during test for e (cid:48)
Revised: When d e (cid:48) s can be acquired during test for e (cid:48)

Original: ∈
Revised: ∈

Original: E test , we can predict y as arg max
Revised: E test , we can predict y as arg max

Original: y p φ ( y | x, d e (cid:48) s )
Revised: y p φ ( y | x, d e (cid:48) s )

Original: =
Revised: =

Original: (cid:82) q ψ ( s | x, d e (cid:48) s )
Revised: (cid:82) q ψ ( s | x, d e (cid:48) s )

Original: p φ ( y | s ) ds .
Revised: p φ ( y | s ) ds .

Original: Otherwise, for LaCIM- d with d s unobserved, we ﬁrst optimize s, z via ( s (cid:63) , z (cid:63) ) :
Revised: Otherwise, for LaCIM- d with d s unobserved, we ﬁrst optimize s, z via ( s (cid:63) , z (cid:63) ) :

Original: = arg max s,z log p φ ( x | s, z ) and predict y as arg max y q ψ ( y | s (cid:63) ) .
Revised: = arg max s,z log p φ ( x | s, z ) and predict y as arg max y q ψ ( y | s (cid:63) ) .

Original: Speciﬁcally, we adopt the strategy for optimization in Schott et al.
Revised: Speciﬁcally, we adopt the strategy for optimization in Schott et al.

Original: (2018) that we ﬁrst sample initial points and select the one with the maximum log p φ ( x | s, z ) , then we optimize for 50 iterations using
Revised: (2018) that we ﬁrst sample initial points and select the one with the maximum log p φ ( x | s, z ) , then we optimize for 50 iterations using

Original: Adam.
Revised: Adam.

Original: The implementation details and optimization effect are shown in supplement 7.9.
Revised: The implementation details and optimization effect are shown in supplement 7.9.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate LaCIM on (I) synthetic data to verify the identiﬁability in theorem 4.3; (II) OOD challenges: object classiﬁcation with sample selection bias (Non-I.I.D. Image dataset with Contexts (NICO)); Hand-Writing Recognition with confounding bias (Colored MNIST (CMNIST)); prediction of Alzheimer’s Disease (Alzheimer’s Disease Neuroimaging Initiative (ADNI www.loni.ucla.
Revised: We evaluate LaCIM on (I) synthetic data to verify the identiﬁability in theorem 4.3; (II) OOD challenges: object classiﬁcation with sample selection bias (Non-I.I.D. Image dataset with Contexts (NICO)); Hand-Writing Recognition with confounding bias (Colored MNIST (CMNIST)); prediction of Alzheimer’s Disease (Alzheimer’s Disease Neuroimaging Initiative (ADNI www.loni.ucla.

Original: edu/ADNI )
Revised: edu/ADNI )

Original: ; (III) Robustness on detecting images with small perturbation (FaceForensics++).
Revised: ; (III) Robustness on detecting images with small perturbation (FaceForensics++).

Original: 5. S IMULATION
Revised: 5. S IMULATION

Original: To verify the identiﬁability claim and effectiveness of our learning method, we implement LaCIM on synthetic data.
Revised: To verify the identiﬁability claim and effectiveness of our learning method, we implement LaCIM on synthetic data.

Original: The data generating process is provided in Supplement 7.8.
Revised: The data generating process is provided in Supplement 7.8.

Original: The domain index
Revised: The domain index

Original: D ∈ R m is denoted as a one-hot encoded vector with m = 5 .
Revised: D ∈ R m is denoted as a one-hot encoded vector with m = 5 .

Original: As veriﬁcation of the utility of training on multiple domains, we also conduct LaCIM by pooling data from all m domains together, namely pool-LaCIM for comparison.
Revised: As veriﬁcation of the utility of training on multiple domains, we also conduct LaCIM by pooling data from all m domains together, namely pool-LaCIM for comparison.

Original: We randomly generate 5 datasets and run 20 times for each.
Revised: We randomly generate 5 datasets and run 20 times for each.

Original: We compute the metric mean correlation coefﬁcient (MCC) adopted in Khemakhem, Kingma and
Revised: We compute the metric mean correlation coefﬁcient (MCC) adopted in Khemakhem, Kingma and

Original: Hyv ¨ arinen (2020) to measure the goodness of identiﬁability under permutation by introducing cost optimization to assign each learned component to the source component.
Revised: Hyv ¨ arinen (2020) to measure the goodness of identiﬁability under permutation by introducing cost optimization to assign each learned component to the source component.

Original: This measurement is aligned with the goal of ∼ p -identiﬁability, which allows us to distinguish S from Z .
Revised: This measurement is aligned with the goal of ∼ p -identiﬁability, which allows us to distinguish S from Z .

Original: Table 5.1 shows that
Revised: Table 5.1 shows that

Original: our
Revised: our

Original: LaCIM- d , LaCIM- d s perform much better than pool-LaCIM in terms of the CIMe relating to S, Z under permutation, by means of multiple diverse experiments.
Revised: LaCIM- d , LaCIM- d s perform much better than pool-LaCIM in terms of the CIMe relating to S, Z under permutation, by means of multiple diverse experiments.

Original: Besides, we consider LaCIM- d on m = 3 , 5 , 7 with the same number of samples.
Revised: Besides, we consider LaCIM- d on m = 3 , 5 , 7 with the same number of samples.

Original: It yields that more environments can achieve better results, although m = 3 still performs much better than pool-LaCIM.
Revised: It yields that more environments can achieve better results, although m = 3 still performs much better than pool-LaCIM.

Original: As an illustration of learning effect, we visualize the learned Z in Fig.
Revised: As an illustration of learning effect, we visualize the learned Z in Fig.

Original: 7.8, with S left in supplement 7.8 due to space limit.
Revised: 7.8, with S left in supplement 7.8 due to space limit.

Original: [Figure]
Revised: [Figure]

Original: 5. R EAL - WORLD OOD C HALLENGE
Revised: 5. R EAL - WORLD OOD C HALLENGE

Original: We present our LaCIM’s results on three OOD tasks, with different environments associated with different values of d s .
Revised: We present our LaCIM’s results on three OOD tasks, with different environments associated with different values of d s .

Original: We implement both versions of LaCIM, i.e. , LaCIM- d s and LaCIM- d , with task-dependent deﬁnition of d s .
Revised: We implement both versions of LaCIM, i.e. , LaCIM- d s and LaCIM- d , with task-dependent deﬁnition of d s .

Original: In CMNIST, the d s (digit color) is a fully observed confounder, and
Revised: In CMNIST, the d s (digit color) is a fully observed confounder, and

Original: LaCIM- d s in this case is the ceiling of LaCIM- d under the same implementation.
Revised: LaCIM- d s in this case is the ceiling of LaCIM- d under the same implementation.

Original: In NICO and ADNI, the LaCIM- d even outperform LaCIM- d s , when the source variables are only partially observed.
Revised: In NICO and ADNI, the LaCIM- d even outperform LaCIM- d s , when the source variables are only partially observed.

Original: Dataset.
Revised: Dataset.

Original: We describe the datasets as follows (the X denote image; the Y denote label):
Revised: We describe the datasets as follows (the X denote image; the Y denote label):

Original: NICO : we evaluate the cat/dog classiﬁcation in “Animal” dataset in NICO, a benchmark for non-i.i.d problem in He et al.
Revised: NICO : we evaluate the cat/dog classiﬁcation in “Animal” dataset in NICO, a benchmark for non-i.i.d problem in He et al.

Original: Each animal is associated with “grass”,“snow” contexts with different proportions, denoted as d s ∈ R 4 (cat,dog in grass,snow).
Revised: Each animal is associated with “grass”,“snow” contexts with different proportions, denoted as d s ∈ R 4 (cat,dog in grass,snow).

Original: We set m = 8 and m = 14 .
Revised: We set m = 8 and m = 14 .

Original: The C, Z, S respectively denote the (time,whether) of sampling, the context and semantic shape of cat/dog.
Revised: The C, Z, S respectively denote the (time,whether) of sampling, the context and semantic shape of cat/dog.

Original: CMNIST : We relabel the digits 0-4 and 5-9 as y = 

Revised: CMNIST : We relabel the digits 0-4 and 5-9 as y =

Original: and y
Revised: and y

Original: = 1 , based on MNIST.
Revised: = 1 , based on MNIST.

Original: Then we color p e ( 1 − p e ) of images with y = 

Revised: Then we color p e ( 1 − p e ) of images with y =

Original: ( y = 1 ) as green and color others as red.
Revised: ( y = 1 ) as green and color others as red.

Original: We set m = 2 with p e 1 = 0 .
Revised: We set m = 2 with p e 1 = 0 .

Original: 9 , p e 2 = 0 .
Revised: 9 , p e 2 = 0 .

Original: The d es is p e to describe the intensity of spursiou correlation caused by color.
Revised: The d es is p e to describe the intensity of spursiou correlation caused by color.

Original: We do not ﬂip y with 25% like Arjovsky et al.
Revised: We do not ﬂip y with 25% like Arjovsky et al.

Original: (2019) 2 , since doing so will cause the digit correlated rather than causally related to the label, which is beyond our scope.
Revised: (2019) 2 , since doing so will cause the digit correlated rather than causally related to the label, which is beyond our scope.

Original: The Z, S respectively represent the color and number.
Revised: The Z, S respectively represent the color and number.

Original: The C can also denote (time,whether) for which the painter draws the number and color, e.g. , the painter tends to draw red 0 more often than green 1 in the sunny morning.
Revised: The C can also denote (time,whether) for which the painter draws the number and color, e.g. , the painter tends to draw red 0 more often than green 1 in the sunny morning.

Original: ADNI.
Revised: ADNI.

Original: The data are obtained from the ADNI databaset, the Y :
Revised: The data are obtained from the ADNI databaset, the Y :

Original: = { 0 , 1 , 2 } with 0,1,2 respectively denoting AD, Mild Cognitive Impairment (MCI) and Normal Control (NC).
Revised: = { 0 , 1 , 2 } with 0,1,2 respectively denoting AD, Mild Cognitive Impairment (MCI) and Normal Control (NC).

Original: The X is Magnetic resonance imaging (sMRI).
Revised: The X is Magnetic resonance imaging (sMRI).

Original: We set m = 2 .
Revised: We set m = 2 .

Original: We consider two types of d s : Age and TAU (a biomarker
Revised: We consider two types of d s : Age and TAU (a biomarker

Original: Humpel and Hochstrasser (2011)).
Revised: Humpel and Hochstrasser (2011)).

Original: The S ( Z ) denote the disease-related (-unrelated) brain regions.
Revised: The S ( Z ) denote the disease-related (-unrelated) brain regions.

Original: The C denotes the hormone level that can affect the brain structure development.
Revised: The C denotes the hormone level that can affect the brain structure development.

Original: Compared Baselines.
Revised: Compared Baselines.

Original: We compare with (i) Cross-Entropy (CE) from X → Y (CE X → Y ), (ii) domain-adversarial neural network (DANN) for domain adaptation Ganin et al.
Revised: We compare with (i) Cross-Entropy (CE) from X → Y (CE X → Y ), (ii) domain-adversarial neural network (DANN) for domain adaptation Ganin et al.

Original: , (iii) Maximum
Revised: , (iii) Maximum

Original: Mean Discrepancy with Adversarial Auto-Encoder (MMD-AAE) for domain generalization Li et al.
Revised: Mean Discrepancy with Adversarial Auto-Encoder (MMD-AAE) for domain generalization Li et al.

Original: (2018), (iv) Domain Invariant variational autoencoders (DIVA)
Revised: (2018), (iv) Domain Invariant variational autoencoders (DIVA)

Original: Ilse et al.
Revised: Ilse et al.

Original: (2019) (v) Selecting Data
Revised: (2019) (v) Selecting Data

Original: Augmentation (SDA) Ilse et al.
Revised: Augmentation (SDA) Ilse et al.

Original: (2020), (vi) Invariant Risk Mnimization (IRM) Arjovsky et al.
Revised: (2020), (vi) Invariant Risk Mnimization (IRM) Arjovsky et al.

Original: , (vii) CE ( X, d s ) →
Revised: , (vii) CE ( X, d s ) →

Original: Y , (viii) VAE with causal graph C → V → { X, Y } with V mixing S, Z
Revised: Y , (viii) VAE with causal graph C → V → { X, Y } with V mixing S, Z

Original: and we call it sVAE for simplicity.
Revised: and we call it sVAE for simplicity.

Original: We only implement SDA on CMNIST, since the intervened-data generation of SDA requires knowing the S, Z , which cannot be explicitly extracted in ADNI and
Revised: We only implement SDA on CMNIST, since the intervened-data generation of SDA requires knowing the S, Z , which cannot be explicitly extracted in ADNI and

Original: NICO.
Revised: NICO.

Original: For fair comparison, we keep the model capacity (numer of parameters) in the same level.
Revised: For fair comparison, we keep the model capacity (numer of parameters) in the same level.

Original: Implementation Details.
Revised: Implementation Details.

Original: For each domain e , we implement the reparameterization with ρ s , ρ z : s (cid:48) , z (cid:48)
Revised: For each domain e , we implement the reparameterization with ρ s , ρ z : s (cid:48) , z (cid:48)

Original: =
Revised: =

Original: ρ es ( s ) , ρ ez ( z ) , to transform the p e ( s, z ) into isotropic Gaussian; then the generative models are correspondingly modiﬁed as { p φ ( x | ( ρ es ) − 1 ( s ) , ( ρ ez ) − 1 ( z )) , p φ ( y | ( ρ es ) − 1 ( s )) } according to rule of change of variables.
Revised: ρ es ( s ) , ρ ez ( z ) , to transform the p e ( s, z ) into isotropic Gaussian; then the generative models are correspondingly modiﬁed as { p φ ( x | ( ρ es ) − 1 ( s ) , ( ρ ez ) − 1 ( z )) , p φ ( y | ( ρ es ) − 1 ( s )) } according to rule of change of variables.

Original: The optimized parameters are {{ q eψ ( s, z | x ) } e , p φ ( x | s, z ) , p φ ( y | s ) , { ρ et = s,z } e } , with the encoder q e ψ ( s, z | x ) being sequentially composed of: i) the sequential of Conv-BN-ReLUMaxPool blocks that shared among E train , followed by ii) the sequential of ReLU-FC for the mean and log-variance of S, Z that are speciﬁc to e .
Revised: The optimized parameters are {{ q eψ ( s, z | x ) } e , p φ ( x | s, z ) , p φ ( y | s ) , { ρ et = s,z } e } , with the encoder q e ψ ( s, z | x ) being sequentially composed of: i) the sequential of Conv-BN-ReLUMaxPool blocks that shared among E train , followed by ii) the sequential of ReLU-FC for the mean and log-variance of S, Z that are speciﬁc to e .

Original: The structure of ρ et = s,z is FC-ReLU-FC.
Revised: The structure of ρ et = s,z is FC-ReLU-FC.

Original: The decoder p φ ( x | s, z ) is the sequential of upsampling, several TConv-BN-ReLU blocks and Sigmoid.
Revised: The decoder p φ ( x | s, z ) is the sequential of upsampling, several TConv-BN-ReLU blocks and Sigmoid.

Original: The predictor p φ ( y | s ) is sequential of FC → BN → ReLU blocks, followed by Softmax (or Sigmoid) for classiﬁcation.
Revised: The predictor p φ ( y | s ) is sequential of FC → BN → ReLU blocks, followed by Softmax (or Sigmoid) for classiﬁcation.

Original: The network structure and the output channel size for CMNIST, NICO and ADNI are introduced in supplement 7.11, 7.12, 7.13, Tab. 14, 13.
Revised: The network structure and the output channel size for CMNIST, NICO and ADNI are introduced in supplement 7.11, 7.12, 7.13, Tab. 14, 13.

Original: We implement SGD as optimizer: with learning rate (lr) 0.5 and weight decay (wd) 1 e - 5 for CMNIST; lr 0.01 with decaying 0.2 × every 60 epochs, wd 5 e - 5 for NICO and ADNI (wd is 2 e - 4 ).
Revised: We implement SGD as optimizer: with learning rate (lr) 0.5 and weight decay (wd) 1 e - 5 for CMNIST; lr 0.01 with decaying 0.2 × every 60 epochs, wd 5 e - 5 for NICO and ADNI (wd is 2 e - 4 ).

Original: The batch size are set to 80,200 and 200 for
Revised: The batch size are set to 80,200 and 200 for

Original: CMNIST, NICO, ADNI.
Revised: CMNIST, NICO, ADNI.

Original: The “FC”, ”BN” stand for Fully-Connected, Batch-Normalization.
Revised: The “FC”, ”BN” stand for Fully-Connected, Batch-Normalization.

Original: Results.
Revised: Results.

Original: We report accuracy over three runs for each method.
Revised: We report accuracy over three runs for each method.

Original: As shown in Tab. 2 3 our LaCIM- d performs comparable and better than others on all applications, except the 99 . 3 achieved by SDA on
Revised: As shown in Tab. 2 3 our LaCIM- d performs comparable and better than others on all applications, except the 99 . 3 achieved by SDA on

Original: CMNIST, which is comparable to the result on the original MNIST.
Revised: CMNIST, which is comparable to the result on the original MNIST.

Original: This is because during training, the SDA implemented data augmentation with random colors, which decorrelate the color-label.
Revised: This is because during training, the SDA implemented data augmentation with random colors, which decorrelate the color-label.

Original: When S cannot be explicitly extracted in general case, the SDA is not tractable.
Revised: When S cannot be explicitly extracted in general case, the SDA is not tractable.

Original: [Table]
Revised: [Table]

Original: Discussions.
Revised: Discussions.

Original: The advantage over invariant learning method (IRM) and CE ( X, d s ) → Y which also takes d s into prediction can be contributed to the identiﬁcation of true causal mechanisms.
Revised: The advantage over invariant learning method (IRM) and CE ( X, d s ) → Y which also takes d s into prediction can be contributed to the identiﬁcation of true causal mechanisms.

Original: Further, the improvement over sVAE is beneﬁted from our separation of ycausative factors ( a.k.a , S ) from others to avoid spurious correlation.
Revised: Further, the improvement over sVAE is beneﬁted from our separation of ycausative factors ( a.k.a , S ) from others to avoid spurious correlation.

Original: Besides, as shown from results on
Revised: Besides, as shown from results on

Original: NICO, a larger m (with the total number of samples n ﬁxed) can bring further beneﬁt, which may due to the easier satisfaction of the diversity condition in theorem 4.3.
Revised: NICO, a larger m (with the total number of samples n ﬁxed) can bring further beneﬁt, which may due to the easier satisfaction of the diversity condition in theorem 4.3.

Original: One thing worth particular mention is that on NICO and ADNI (when d s denotes TAU), our LaCIM- d performs comparable and even better than LaCIM- d s , due to the existence of unobserved partial variables.
Revised: One thing worth particular mention is that on NICO and ADNI (when d s denotes TAU), our LaCIM- d performs comparable and even better than LaCIM- d s , due to the existence of unobserved partial variables.

Original: For example, each d s only contains one attribute each time in ADNI.
Revised: For example, each d s only contains one attribute each time in ADNI.

Original: For completeness, we conduct experiments with fully observed confounders in supplement 7.13.
Revised: For completeness, we conduct experiments with fully observed confounders in supplement 7.13.

Original: Besides, we apply our method on intervened data, the result of which can validate more robustness of LaCIM, as shown in supplement 7.12.
Revised: Besides, we apply our method on intervened data, the result of which can validate more robustness of LaCIM, as shown in supplement 7.12.

Original: Interpretability.
Revised: Interpretability.

Original: We visualize learned S as side proof of interpretability.
Revised: We visualize learned S as side proof of interpretability.

Original: Speciﬁcally, we select the s ∗ that has the highest correlation with y among all dimension of S , and visualize the derivatives of s ∗ with respect to the image.
Revised: Speciﬁcally, we select the s ∗ that has the highest correlation with y among all dimension of S , and visualize the derivatives of s ∗ with respect to the image.

Original: For CE x → y and CE ( x, d s ) → y , we visualize the derivatives of predicted class scores with respect to the image.
Revised: For CE x → y and CE ( x, d s ) → y , we visualize the derivatives of predicted class scores with respect to the image.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5.2, LaCIM (the 4th column) can identify more explainable semantic features, which veriﬁes the identiﬁability and effectiveness of the learning method.
Revised: 5.2, LaCIM (the 4th column) can identify more explainable semantic features, which veriﬁes the identiﬁability and effectiveness of the learning method.

Original: Supplement 7.12 provides more results.
Revised: Supplement 7.12 provides more results.

Original: [Figure]
Revised: [Figure]

Original: 5. R OBUSTNESS ON S ECURITY
Revised: 5. R OBUSTNESS ON S ECURITY

Original: [Table]
Revised: [Table]

Original: Average over three runs.
Revised: Average over three runs.

Original: [Table]
Revised: [Table]

Original: We consider the DeepFake-related security problem, which targets on detecting small perturbed fake images that can spread fake news.
Revised: We consider the DeepFake-related security problem, which targets on detecting small perturbed fake images that can spread fake news.

Original: The Rossler et al.
Revised: The Rossler et al.

Original: (2019) provides FaceForensics++ dataset from
Revised: (2019) provides FaceForensics++ dataset from

Original: Youtube videos for training and 1,000 benchmark images from other sources (OOD) for testing.
Revised: Youtube videos for training and 1,000 benchmark images from other sources (OOD) for testing.

Original: We split the train data into m = 2 environments according to video ID.
Revised: We split the train data into m = 2 environments according to video ID.

Original: The considerable result in Tab.
Revised: The considerable result in Tab.

Original: 5.3 veriﬁes potential value on security.
Revised: 5.3 veriﬁes potential value on security.

Original:  C ONCLUSIONS & D ISCUSSIONS
Revised:  C ONCLUSIONS & D ISCUSSIONS

Original: We incorporate the causal structure as prior knowledge in proposed LaCIM, by introducing: (i) latent variables and explicitly separate them into y -causative factors ( a.k.a , S ) and others ( a.k.a , Z ) which are spuriously correlated with the output; (ii) the source variable d s that explains the distributional inconsistency among domains.
Revised: We incorporate the causal structure as prior knowledge in proposed LaCIM, by introducing: (i) latent variables and explicitly separate them into y -causative factors ( a.k.a , S ) and others ( a.k.a , Z ) which are spuriously correlated with the output; (ii) the source variable d s that explains the distributional inconsistency among domains.

Original: When the environments are diverse and much enough, we can successfully identify the causal invariant mechanisms, and also y -causative factors for prediction without a mix of others.
Revised: When the environments are diverse and much enough, we can successfully identify the causal invariant mechanisms, and also y -causative factors for prediction without a mix of others.

Original: Our LaCIM shows potential value regarding robustness to OOD tasks with confounding bias, selection bias and others such as healthcare and security.
Revised: Our LaCIM shows potential value regarding robustness to OOD tasks with confounding bias, selection bias and others such as healthcare and security.

Original: A possible drawback of our model lies in our requirement of the number of environments (which may be not satisﬁed in some scenarios) for identiﬁability, and the relaxation of which is left in the future work.
Revised: A possible drawback of our model lies in our requirement of the number of environments (which may be not satisﬁed in some scenarios) for identiﬁability, and the relaxation of which is left in the future work.

Original:  S UPPLEMENTARY M ATERIALS 7. O.O.D G ENERALIZATION ERROR B OUND
Revised:  S UPPLEMENTARY M ATERIALS 7. O.O.D G ENERALIZATION ERROR B OUND

Original: Denote E p
Revised: Denote E p

Original: [ y | x ] := (cid:82) Y yp
Revised: [ y | x ] := (cid:82) Y yp

Original: ( y | x ) dy for any x, y ∈ X ×Y .
Revised: ( y | x ) dy for any x, y ∈ X ×Y .

Original: We have E p e [ y | s ] = (cid:82) Y yp
Revised: We have E p e [ y | s ] = (cid:82) Y yp

Original: ( y | s ) dy according to that p ( y | s ) is invariant across E , we can omit p e in E p e
Revised: ( y | s ) dy according to that p ( y | s ) is invariant across E , we can omit p e in E p e

Original: [ y | s ] and denote g ( S ) :
Revised: [ y | s ] and denote g ( S ) :

Original: =
Revised: =

Original: E [ Y | S ] .
Revised: E [ Y | S ] .

Original: Then, the OOD bound (cid:12)(cid:12)
Revised: Then, the OOD bound (cid:12)(cid:12)

Original: E p e 1 ( y | x ) − E p e 2 ( y | x ) (cid:12)(cid:12) , ∀ ( x, y ) is bounded as follows:
Revised: E p e 1 ( y | x ) − E p e 2 ( y | x ) (cid:12)(cid:12) , ∀ ( x, y ) is bounded as follows:

Original: Theorem 7.1 (OOD genearlization error) .
Revised: Theorem 7.1 (OOD genearlization error) .

Original: Consider two LaCIM P e 1 and P e 2 , suppose that their densities , i.e., p e 1 ( s | x ) and p e 2 ( s | x ) are absolutely continuous having support ( −∞ , ∞ ) .
Revised: Consider two LaCIM P e 1 and P e 2 , suppose that their densities , i.e., p e 1 ( s | x ) and p e 2 ( s | x ) are absolutely continuous having support ( −∞ , ∞ ) .

Original: For any ( x, y ) ∈ X × Y , assume that
Revised: For any ( x, y ) ∈ X × Y , assume that

Original: • g ( S ) is a Lipschitz-continuous function; • π x ( s ) :
Revised: • g ( S ) is a Lipschitz-continuous function; • π x ( s ) :

Original: = p e 2 ( s | x ) p e 1 ( s | x ) is differentiable and E p e 1 (cid:2)
Revised: = p e 2 ( s | x ) p e 1 ( s | x ) is differentiable and E p e 1 (cid:2)

Original: π x ( S ) (cid:12)(cid:12) g ( S ) − µ 1 (cid:12)(cid:12)(cid:3) <
Revised: π x ( S ) (cid:12)(cid:12) g ( S ) − µ 1 (cid:12)(cid:12)(cid:3) <

Original: ∞ with µ 1 :=
Revised: ∞ with µ 1 :=

Original: E p e 1 [ g ( S ) | X = x ] = (cid:82) S g ( s )
Revised: E p e 1 [ g ( S ) | X = x ] = (cid:82) S g ( s )

Original: p e 1 ( s | x ) ds ;
Revised: p e 1 ( s | x ) ds ;

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: When e 1 ∈ E train and e 2 ∈ E test , the theorem 7.1 describes the error during generalization on efor the strategy that trained on e 1 .
Revised: When e 1 ∈ E train and e 2 ∈ E test , the theorem 7.1 describes the error during generalization on efor the strategy that trained on e 1 .

Original: The bound is mainly affected by: (i) the Lipschitz constant of g , i.e. , (cid:107) g (cid:107) ∞ ; (ii) (cid:107) π (cid:48) x (cid:107) ∞ which measures the difference between p e 1 ( s, z ) and p e 2 ( s, z ) ; and (iii)
Revised: The bound is mainly affected by: (i) the Lipschitz constant of g , i.e. , (cid:107) g (cid:107) ∞ ; (ii) (cid:107) π (cid:48) x (cid:107) ∞ which measures the difference between p e 1 ( s, z ) and p e 2 ( s, z ) ; and (iii)

Original: the
Revised: the

Original: Var p e 1 ( S | x ) that measures the intensity of x → ( s, z ) .
Revised: Var p e 1 ( S | x ) that measures the intensity of x → ( s, z ) .

Original: These terms can be roughly categorized into two classes: (i),(iii) which are related to the property of CIMe and gave few space for improvement; and the (ii) that describes the distributional change between two environments.
Revised: These terms can be roughly categorized into two classes: (i),(iii) which are related to the property of CIMe and gave few space for improvement; and the (ii) that describes the distributional change between two environments.

Original: Speciﬁcally for the ﬁrst class, the (i) measures the smoothness of E ( y | s ) with respect to s .
Revised: Speciﬁcally for the ﬁrst class, the (i) measures the smoothness of E ( y | s ) with respect to s .

Original: The smaller value of (cid:107) g (cid:48) (cid:107) ∞ implies that the ﬂatter regions give rise to the same prediction result, hence easier transfer from e 1 to e 2 and vice versa.
Revised: The smaller value of (cid:107) g (cid:48) (cid:107) ∞ implies that the ﬂatter regions give rise to the same prediction result, hence easier transfer from e 1 to e 2 and vice versa.

Original: For the term (iii), consider the deterministic setting that ε x = 0 (leads to
Revised: For the term (iii), consider the deterministic setting that ε x = 0 (leads to

Original: Var p e 1 ( S | x ) = 0 ), then s can be determined from x for generalization if the f is bijective function.
Revised: Var p e 1 ( S | x ) = 0 ), then s can be determined from x for generalization if the f is bijective function.

Original: The term (ii) measures the distributional change between posterior distributions p e 1 ( s | x ) and p e 2 ( s | x ) , which contributes to the difference during prediction: (cid:12)(cid:12)
Revised: The term (ii) measures the distributional change between posterior distributions p e 1 ( s | x ) and p e 2 ( s | x ) , which contributes to the difference during prediction: (cid:12)(cid:12)

Original: E p e 1 ( y | x ) − E p e 2 ( y | x ) (cid:12)(cid:12) =
Revised: E p e 1 ( y | x ) − E p e 2 ( y | x ) (cid:12)(cid:12) =

Original: (cid:82) S ( p e 1 ( s | x ) − p e 1 ( s | x ))
Revised: (cid:82) S ( p e 1 ( s | x ) − p e 1 ( s | x ))

Original: p f y
Revised: p f y

Original: ( y | s ) ds .
Revised: ( y | s ) ds .

Original: Such a change is due to the inconsistency between priors p e 1 ( s, z ) and p e 2 ( s, z ) , which is caused by different value of the confounder d s .
Revised: Such a change is due to the inconsistency between priors p e 1 ( s, z ) and p e 2 ( s, z ) , which is caused by different value of the confounder d s .

Original: Proof.
Revised: Proof.

Original: In the following, we will derive the upper bound
Revised: In the following, we will derive the upper bound

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: where π x ( s ) =:
Revised: where π x ( s ) =:

Original: p e 2 ( s | x ) p e 1 ( s | x ) and g ( s ) is assumed to be Lipschitz-continuous.
Revised: p e 2 ( s | x ) p e 1 ( s | x ) and g ( s ) is assumed to be Lipschitz-continuous.

Original: To begin with, note that
Revised: To begin with, note that

Original: [Equation]
Revised: [Equation]

Original: Let p 1 ( s | x ) =
Revised: Let p 1 ( s | x ) =

Original: p e 1 ( s | x ) , p 2 ( s | x ) =
Revised: p e 1 ( s | x ) , p 2 ( s | x ) =

Original: p e 2 ( s | x ) .
Revised: p e 2 ( s | x ) .

Original: For ease of notations, we use P 1 and P 2 denote the distributions with densities p 1 ( s | x ) and p 2 ( s | x ) and suppose S 1 ∼ P 1 and S 2 ∼ P 2 , where x is omitted as the following analysis is conditional on a ﬁxed X = x .
Revised: For ease of notations, we use P 1 and P 2 denote the distributions with densities p 1 ( s | x ) and p 2 ( s | x ) and suppose S 1 ∼ P 1 and S 2 ∼ P 2 , where x is omitted as the following analysis is conditional on a ﬁxed X = x .

Original: Then we may rewrite the difference of conditional expectations as
Revised: Then we may rewrite the difference of conditional expectations as

Original: [Equation]
Revised: [Equation]

Original: where E [ g ( S j ))]
Revised: where E [ g ( S j ))]

Original: = (cid:82) g
Revised: = (cid:82) g

Original: ( s ) p
Revised: ( s ) p

Original: j ( s | x ) ds denotes the expectation over P j .
Revised: j ( s | x ) ds denotes the expectation over P j .

Original: Let µ 1 :=
Revised: Let µ 1 :=

Original: E p
Revised: E p

Original: [Equation]
Revised: [Equation]

Original: 1 [ g ( S ) | X
Revised: 1 [ g ( S ) | X

Original: =
Revised: =

Original: x
Revised: x

Original: =
Revised: =

Original: E [ g ( S 1 )]
Revised: E [ g ( S 1 )]

Original: = (cid:82) g ( s ) p 1 ( s | x ) ds .
Revised: = (cid:82) g ( s ) p 1 ( s | x ) ds .

Original: Then
Revised: Then

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Further, we have the following transformation
Revised: Further, we have the following transformation

Original: [Equation]
Revised: [Equation]

Original: In the following, we will use the results of the Stein kernel function.
Revised: In the following, we will use the results of the Stein kernel function.

Original: Please refer to Deﬁnition 7.2 for a general deﬁnition.
Revised: Please refer to Deﬁnition 7.2 for a general deﬁnition.

Original: Particularly, for the distribution P 1 ∼ p 1 ( s | x ) , the Stein kernel τ 1 ( s ) is
Revised: Particularly, for the distribution P 1 ∼ p 1 ( s | x ) , the Stein kernel τ 1 ( s ) is

Original: [Equation]
Revised: [Equation]

Original: where E ( S 1 ) =
Revised: where E ( S 1 ) =

Original: (cid:82) s · p 1 ( s | x ) ds .
Revised: (cid:82) s · p 1 ( s | x ) ds .

Original: Further, we deﬁne ( τ 1 ◦ g )
Revised: Further, we deﬁne ( τ 1 ◦ g )

Original: ( s ) as
Revised: ( s ) as

Original: [Equation]
Revised: [Equation]

Original: Under the second condition listed in Theorem 7.
Revised: Under the second condition listed in Theorem 7.

Original: [Equation]
Revised: [Equation]

Original: we may apply the result of Lemma
Revised: we may apply the result of Lemma

Original: [Equation]
Revised: [Equation]

Original: Speciﬁcally, by the equation (8), we have
Revised: Speciﬁcally, by the equation (8), we have

Original: [Equation]
Revised: [Equation]

Original: Then under the ﬁrst condition in Theorem 7.
Revised: Then under the ﬁrst condition in Theorem 7.

Original: [Equation]
Revised: [Equation]

Original: we can obtain the following inequality by Lemma 7.4,
Revised: we can obtain the following inequality by Lemma 7.4,

Original: [Equation]
Revised: [Equation]

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we show that the Stein kernel is non-negative, which enables E [ | τ 1 ( S 1 ) | ] =
Revised: we show that the Stein kernel is non-negative, which enables E [ | τ 1 ( S 1 ) | ] =

Original: E [ τ 1 ( S 1 )] .
Revised: E [ τ 1 ( S 1 )] .

Original: According to the deﬁnition, τ 1 ( s ) =
Revised: According to the deﬁnition, τ 1 ( s ) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s |
Revised: s |

Original: [Equation]
Revised: [Equation]

Original: (cid:82) s
Revised: (cid:82) s

Original: [Equation]
Revised: [Equation]

Original: ( E ( S 1 ) − t ) p 1 ( t | x ) dt , where E ( S 1 )=
Revised: ( E ( S 1 ) − t ) p 1 ( t | x ) dt , where E ( S 1 )=

Original: (cid:82) ∞
Revised: (cid:82) ∞

Original: [Equation]
Revised: [Equation]

Original: t · p 1 ( t | x ) dt .
Revised: t · p 1 ( t | x ) dt .

Original: Let F 1 ( s ) =
Revised: Let F 1 ( s ) =

Original: (cid:82) s
Revised: (cid:82) s

Original: −∞ p 1 ( t | x ) dt be the distribution function for P 1 .
Revised: −∞ p 1 ( t | x ) dt be the distribution function for P 1 .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: The last inequality is based on E( S 1 | S 1 ≤ s ) − E
Revised: The last inequality is based on E( S 1 | S 1 ≤ s ) − E

Original: ( S 1 ) ≤ 0 that can be proved as the following
Revised: ( S 1 ) ≤ 0 that can be proved as the following

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: Therefore, τ 1 ( s ) ≥ 0 and hence E [ | τ 1 ( S 1 ) | ] =
Revised: Therefore, τ 1 ( s ) ≥ 0 and hence E [ | τ 1 ( S 1 ) | ] =

Original: E [ τ 1 ( S 1 )] in (6).
Revised: E [ τ 1 ( S 1 )] in (6).

Original: Besides, by equation (9)
Revised: Besides, by equation (9)

Original: [Equation]
Revised: [Equation]

Original: the special case of Lemma 7.3, we have
Revised: the special case of Lemma 7.3, we have

Original: [Equation]
Revised: [Equation]

Original: To sum up,
Revised: To sum up,

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 7.2 ( the Stein Kernel τ P of distribution P ).
Revised: Deﬁnition 7.2 ( the Stein Kernel τ P of distribution P ).

Original: Suppose X ∼ P with density p .
Revised: Suppose X ∼ P with density p .

Original: The Stein kernel of P is the function x
Revised: The Stein kernel of P is the function x

Original: (cid:55)→ τ P ( x ) deﬁned by
Revised: (cid:55)→ τ P ( x ) deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where Id is the identity function for Id( x )
Revised: where Id is the identity function for Id( x )

Original: =
Revised: =

Original: x .
Revised: x .

Original: More generally, for a function h satisfying E [ | h ( X ) | ] < ∞ , deﬁne ( τ P ◦ h )( x ) as
Revised: More generally, for a function h satisfying E [ | h ( X ) | ] < ∞ , deﬁne ( τ P ◦ h )( x ) as

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: For a differentiable function ϕ such that E
Revised: For a differentiable function ϕ such that E

Original: [ | ( τ P ◦ h )( x ) ϕ (cid:48)
Revised: [ | ( τ P ◦ h )( x ) ϕ (cid:48)

Original: ( X ) | ] < ∞ , we have
Revised: ( X ) | ] < ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let µ h =: E ( h ( X ))
Revised: Let µ h =: E ( h ( X ))

Original: [Equation]
Revised: [Equation]

Original: As E ( h ( X ) − µ h )
Revised: As E ( h ( X ) − µ h )

Original: = 0 ,
Revised: = 0 ,

Original: [Equation]
Revised: [Equation]

Original: Particularly, taking h ( X ) =
Revised: Particularly, taking h ( X ) =

Original: X and ϕ ( X ) =
Revised: X and ϕ ( X ) =

Original: X − E ( X ) , we immediately have
Revised: X − E ( X ) , we immediately have

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.4.
Revised: Lemma 7.4.

Original: Assume that E ( | X | ) <
Revised: Assume that E ( | X | ) <

Original: ∞ and the density p is locally absolutely continuous on ( −∞ , ∞ ) and h is a Lipschitz continuous function.
Revised: ∞ and the density p is locally absolutely continuous on ( −∞ , ∞ ) and h is a Lipschitz continuous function.

Original: Then we have | f h | ≤ (cid:107) h (cid:48) (cid:107) ∞ for
Revised: Then we have | f h | ≤ (cid:107) h (cid:48) (cid:107) ∞ for

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This is a special case of Corollary 3.15 in D ¨ obler et al.
Revised: This is a special case of Corollary 3.15 in D ¨ obler et al.

Original: (2015), taking the constant c = 1 .
Revised: (2015), taking the constant c = 1 .

Original: 7. P ROOF OF THE E QUIVALENCE OF D EFINITION 4. 
Revised: 7. P ROOF OF THE E QUIVALENCE OF D EFINITION 4.

Original: Proposition 7.5.
Revised: Proposition 7.5.

Original: The binary relation ∼ p deﬁned in Def.
Revised: The binary relation ∼ p deﬁned in Def.

Original: 4.2 is an equivalence relation.
Revised: 4.2 is an equivalence relation.

Original: Proof.
Revised: Proof.

Original: The equivalence relation should satisfy three properties as follows:
Revised: The equivalence relation should satisfy three properties as follows:

Original: • Reﬂexive property: The θ ∼ p θ with M z , M s being identity matrix and a s , a z being 0.
Revised: • Reﬂexive property: The θ ∼ p θ with M z , M s being identity matrix and a s , a z being 0.

Original: • Symmtric property: If θ ∼ p ˜ θ , then there exists block permutation matrices M z and M s such that
Revised: • Symmtric property: If θ ∼ p ˜ θ , then there exists block permutation matrices M z and M s such that

Original: [Equation]
Revised: [Equation]

Original: The we have M − 1 s and M − 1 z are also block permutation matrices and such that:
Revised: The we have M − 1 s and M − 1 z are also block permutation matrices and such that:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have ˜ θ ∼ p θ .
Revised: Therefore, we have ˜ θ ∼ p θ .

Original: • Transitive property: if θ
Revised: • Transitive property: if θ

Original: ∼
Revised: ∼

Original: and
Revised: and

Original: ∼
Revised: ∼

Original: p
Revised: p

Original: 3 with
Revised: 3 with

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: for block permutation matrices M 1 s ,
Revised: for block permutation matrices M 1 s ,

Original: [Equation]
Revised: [Equation]

Original: 1 z , M 2 s , M 2 z and vectors
Revised: 1 z , M 2 s , M 2 z and vectors

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 z ,
Revised: 1 z ,

Original: [Equation]
Revised: [Equation]

Original: 2 z .
Revised: 2 z .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Besides
Revised: Besides

Original: [Equation]
Revised: [Equation]

Original: it is apparent that
Revised: it is apparent that

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have θ 1 ∼ p θ 3 since M 2 s M 1 s and M 

Revised: Therefore, we have θ 1 ∼ p θ 3 since M 2 s M 1 s and M

Original: z M 1 z are also permutation matrices.
Revised: z M 1 z are also permutation matrices.

Original: With above three properties satisﬁed, we have that ∼ p is a equivalence relation.
Revised: With above three properties satisﬁed, we have that ∼ p is a equivalence relation.

Original: 7. P ROOF OF T HEOREM 4. 
Revised: 7. P ROOF OF T HEOREM 4.

Original: In the following, we write p e ( x, y ) as p ( x, y | d e ) and also Γ t = s,z c :=
Revised: In the following, we write p e ( x, y ) as p ( x, y | d e ) and also Γ t = s,z c :=

Original: Γ t = s,z ( d e ) , S c,i =
Revised: Γ t = s,z ( d e ) , S c,i =

Original: S i ( d e ) , Z c,i =
Revised: S i ( d e ) , Z c,i =

Original: Z i ( d e ) .
Revised: Z i ( d e ) .

Original: To prove the theorem 4.3, we ﬁrst prove the theorem 7.6 for the simplest case when c | d e = d e , then we generalize to the case when C :=
Revised: To prove the theorem 4.3, we ﬁrst prove the theorem 7.6 for the simplest case when c | d e = d e , then we generalize to the case when C :=

Original: ∪ r C r .
Revised: ∪ r C r .

Original: The overall roadmap is as follows: we ﬁrst prove the ∼ A -identiﬁability in theorem 7.9, and the combination of which with lemma 7.12, 7.11 give theorem 7.6 in the simplest case when c | d e = d e .
Revised: The overall roadmap is as follows: we ﬁrst prove the ∼ A -identiﬁability in theorem 7.9, and the combination of which with lemma 7.12, 7.11 give theorem 7.6 in the simplest case when c | d e = d e .

Original: Then we generalize the case considered in theorem 7.6 to the more general case when C :=
Revised: Then we generalize the case considered in theorem 7.6 to the more general case when C :=

Original: ∪ r C r .
Revised: ∪ r C r .

Original: Theorem 7.6 ( ∼ p -identiﬁability) .
Revised: Theorem 7.6 ( ∼ p -identiﬁability) .

Original: For θ in the LaCIM p eθ ( x, y ) ∈ P exp for any e ∈ E train , we assume that (1) the CIMe satisﬁes that f x , f (cid:48) x and f (cid:48)(cid:48) x are continuous and that f x , f y are bijective; (2) that the T ti,j are twice differentiable for any t =
Revised: For θ in the LaCIM p eθ ( x, y ) ∈ P exp for any e ∈ E train , we assume that (1) the CIMe satisﬁes that f x , f (cid:48) x and f (cid:48)(cid:48) x are continuous and that f x , f y are bijective; (2) that the T ti,j are twice differentiable for any t =

Original: s, z, i ∈
Revised: s, z, i ∈

Original: [ q t ] , j ∈ [ k t ] ; (3) the exogenous variables satisfy that the characteristic functions of ε x , ε y are almost everywhere nonzero; (4) the number of environments, i.e., m ≥ max( q s ∗ k s , q z ∗ k z ) + 1 and (cid:2) Γ t = s,z d e− Γ t
Revised: [ q t ] , j ∈ [ k t ] ; (3) the exogenous variables satisfy that the characteristic functions of ε x , ε y are almost everywhere nonzero; (4) the number of environments, i.e., m ≥ max( q s ∗ k s , q z ∗ k z ) + 1 and (cid:2) Γ t = s,z d e− Γ t

Original: = s,z d e 1 , ..., Γ t = s,z d em − Γ t =
Revised: = s,z d e 1 , ..., Γ t = s,z d em − Γ t =

Original: s,z d e 1 (cid:3) have full column rank for both t = s and t
Revised: s,z d e 1 (cid:3) have full column rank for both t = s and t

Original: =
Revised: =

Original: z , we have that the parameters θ :=
Revised: z , we have that the parameters θ :=

Original: { f x , f y , T s , T z } are ∼ p identiﬁable.
Revised: { f x , f y , T s , T z } are ∼ p identiﬁable.

Original: To prove theorem 7.6, We ﬁrst prove the ∼ A -identiﬁability that is deﬁned as follows:
Revised: To prove theorem 7.6, We ﬁrst prove the ∼ A -identiﬁability that is deﬁned as follows:

Original: Deﬁnition 7.7 ( ∼ A -identiﬁability) .
Revised: Deﬁnition 7.7 ( ∼ A -identiﬁability) .

Original: The deﬁnition is the same with the one deﬁned in 4.2, with
Revised: The deﬁnition is the same with the one deﬁned in 4.2, with

Original: M s , M z being invertible matrices which are not necessarily to be the permutation matrices in
Revised: M s , M z being invertible matrices which are not necessarily to be the permutation matrices in

Original: Def. 4.2.
Revised: Def. 4.2.

Original: Proposition 7.8.
Revised: Proposition 7.8.

Original: The binary relation ∼ A deﬁned in Def.
Revised: The binary relation ∼ A deﬁned in Def.

Original: 7.7 is an equivalence relation.
Revised: 7.7 is an equivalence relation.

Original: Proof.
Revised: Proof.

Original: The proof is similar to that of proposition 7.5.
Revised: The proof is similar to that of proposition 7.5.

Original: The following theorem states that any LaCIM that belongs to P exp is ∼ A -identiﬁable.
Revised: The following theorem states that any LaCIM that belongs to P exp is ∼ A -identiﬁable.

Original: Theorem 7.9 ( ∼ A -identiﬁability) .
Revised: Theorem 7.9 ( ∼ A -identiﬁability) .

Original: For θ in the LaCIM p eθ ( x, y ) ∈ P exp for any e ∈ E train , we assume (1) the CIMe satisﬁes that f x , f y are bijective; (2) the T ti,j are twice differentiable for any t =
Revised: For θ in the LaCIM p eθ ( x, y ) ∈ P exp for any e ∈ E train , we assume (1) the CIMe satisﬁes that f x , f y are bijective; (2) the T ti,j are twice differentiable for any t =

Original: s, z, i ∈
Revised: s, z, i ∈

Original: [ q t ] , j ∈ [ k t ] ; (3) the exogenous variables satisfy that the characteristic functions of ε x , ε y are almost everywhere nonzero; (4) the number of environments, i.e., m ≥ max( q s ∗ k s , q z ∗ k z ) +and (cid:2)
Revised: [ q t ] , j ∈ [ k t ] ; (3) the exogenous variables satisfy that the characteristic functions of ε x , ε y are almost everywhere nonzero; (4) the number of environments, i.e., m ≥ max( q s ∗ k s , q z ∗ k z ) +and (cid:2)

Original: [ Γ td e 2 − Γ td e 1 ] T , ..., [ Γ td em − Γ td e 1 ] T (cid:3) T have full column rank for t =
Revised: [ Γ td e 2 − Γ td e 1 ] T , ..., [ Γ td em − Γ td e 1 ] T (cid:3) T have full column rank for t =

Original: s,
Revised: s,

Original: z , we have that the parameters { f x , f y , T s , T z } are ∼ p identiﬁable.
Revised: z , we have that the parameters { f x , f y , T s , T z } are ∼ p identiﬁable.

Original: Proof.
Revised: Proof.

Original: Suppose that θ = { f x , f y , T s , T z } and ˜ θ =
Revised: Suppose that θ = { f x , f y , T s , T z } and ˜ θ =

Original: { ˜ f x , ˜ g y , ˜ T s , ˜ T z } share the same observational distribution for each environment e ∈ E train , i.e. ,
Revised: { ˜ f x , ˜ g y , ˜ T s , ˜ T z } share the same observational distribution for each environment e ∈ E train , i.e. ,

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where vol J f ( X ) :
Revised: where vol J f ( X ) :

Original: =
Revised: =

Original: det( J f ( X )) for any square matrix X and function f with “ J ” standing for the
Revised: det( J f ( X )) for any square matrix X and function f with “ J ” standing for the

Original: Jacobian.
Revised: Jacobian.

Original: The ˜ p T s , Γ s , T z , Γ z ,f x ( x ) in Eq.
Revised: The ˜ p T s , Γ s , T z , Γ z ,f x ( x ) in Eq.

Original: (16) is denoted as p T s , Γ s , T z , Γ z ( f − 1 x ( x | d e )vol J f − 1 ( x ) .
Revised: (16) is denoted as p T s , Γ s , T z , Γ z ( f − 1 x ( x | d e )vol J f − 1 ( x ) .

Original: The ’*’ in Eq.
Revised: The ’*’ in Eq.

Original: (17) denotes the convolution operator.
Revised: (17) denotes the convolution operator.

Original: The F [ · ] in Eq.
Revised: The F [ · ] in Eq.

Original: (18) denotes the Fourier transform, where φ ε x ( ω ) =
Revised: (18) denotes the Fourier transform, where φ ε x ( ω ) =

Original: F [ p ε x ]( ω ) .
Revised: F [ p ε x ]( ω ) .

Original: Since we assume that the ϕ ε x ( ω ) is non-zero almost everywhere, we can drop it to get Eq.
Revised: Since we assume that the ϕ ε x ( ω ) is non-zero almost everywhere, we can drop it to get Eq.

Original: Similarly, we have that:
Revised: Similarly, we have that:

Original: [Equation]
Revised: [Equation]

Original: and that
Revised: and that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where v :=
Revised: where v :=

Original: [ x (cid:62) , y
Revised: [ x (cid:62) , y

Original: [Equation]
Revised: [Equation]

Original: ] (cid:62) , ε :=
Revised: ] (cid:62) , ε :=

Original: [ ε (cid:62) x , ε (cid:62) y ] (cid:62) , h ( v ) =
Revised: [ ε (cid:62) x , ε (cid:62) y ] (cid:62) , h ( v ) =

Original: [[ f x ] − 1 Z ( x ) (cid:62) , f − 1 y
Revised: [[ f x ] − 1 Z ( x ) (cid:62) , f − 1 y

Original: ( y ) (cid:62) ] (cid:62) .
Revised: ( y ) (cid:62) ] (cid:62) .

Original: According to Eq.
Revised: According to Eq.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Suppose that the assumption (4) holds, then we have
Revised: Suppose that the assumption (4) holds, then we have

Original: [Equation]
Revised: [Equation]

Original: for all k ∈ [ m ] , where ¯Γ ( d ) =
Revised: for all k ∈ [ m ] , where ¯Γ ( d ) =

Original: Γ ( d ) − Γ ( d e 1 ) .
Revised: Γ ( d ) − Γ ( d e 1 ) .

Original: Denote ˜ b s ( k ) =
Revised: Denote ˜ b s ( k ) =

Original: (cid:80) i ˜
Revised: (cid:80) i ˜

Original: A i ( d e 1 )
Revised: A i ( d e 1 )

Original: A i ( d ek ) ˜
Revised: A i ( d ek ) ˜

Original: A i ( d ek ) A i ( d e 1 ) for k ∈ [ m ] , then we have
Revised: A i ( d ek ) A i ( d e 1 ) for k ∈ [ m ] , then we have

Original: [Equation]
Revised: [Equation]

Original: Similarly, from Eq.
Revised: Similarly, from Eq.

Original: (20) and Eq.
Revised: (20) and Eq.

Original: (38), there exists ˜ b z ,
Revised: (38), there exists ˜ b z ,

Original: ˜ b
Revised: ˜ b

Original: s such that
Revised: s such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ˜ b z ( k ) =
Revised: where ˜ b z ( k ) =

Original: (cid:80) i ˜
Revised: (cid:80) i ˜

Original: Z
Revised: Z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: d e 1 )
Revised: d e 1 )

Original: Z i
Revised: Z i

Original: [Equation]
Revised: [Equation]

Original: d ek ) ˜
Revised: d ek ) ˜

Original: Z
Revised: Z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: d ek ) Z
Revised: d ek ) Z

Original: i
Revised: i

Original: d e 1 ) for k ∈ [ m ] ; and that,
Revised: d e 1 ) for k ∈ [ m ] ; and that,

Original: [Equation]
Revised: [Equation]

Original: Substituting Eq.
Revised: Substituting Eq.

Original: (41) to Eq.
Revised: (41) to Eq.

Original: (42) and Eq.
Revised: (42) and Eq.

Original: (43), we have that
Revised: (43), we have that

Original: [Equation]
Revised: [Equation]

Original: According to assumption (4)
Revised: According to assumption (4)

Original: [Equation]
Revised: [Equation]

Original: the Γ s, (cid:62) and Γ z, (cid:62) have full column rank.
Revised: the Γ s, (cid:62) and Γ z, (cid:62) have full column rank.

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Applying the (Khemakhem, Kingma and Hyv¨arinen, 2020, Lemma 3) we have that there exists k s points ¯ x 1 , ..., ¯ x k s , ˜¯ x 1 , ..., ˜¯ x k z such that (cid:16) ( T s ) (cid:48)
Revised: Applying the (Khemakhem, Kingma and Hyv¨arinen, 2020, Lemma 3) we have that there exists k s points ¯ x 1 , ..., ¯ x k s , ˜¯ x 1 , ..., ˜¯ x k z such that (cid:16) ( T s ) (cid:48)

Original: i
Revised: i

Original: [ f − 1 x ] S i ( x 1 i )) , ..., ( T s ) (cid:48) i ([ f − 1 x ]
Revised: [ f − 1 x ] S i ( x 1 i )) , ..., ( T s ) (cid:48) i ([ f − 1 x ]

Original: S i ( x k s i ))
Revised: S i ( x k s i ))

Original: (cid:17) for each i ∈ [ q s ] and (cid:16) ( T z ) (cid:48) i
Revised: (cid:17) for each i ∈ [ q s ] and (cid:16) ( T z ) (cid:48) i

Original: ([ f − 1 x ] Z i (˜ x 1 i )) , ..., ( T z ) (cid:48) i ([ f − 1 x ] S i (˜ x k z i )) (cid:17) for each i ∈ [ q t ] are linearly independent.
Revised: ([ f − 1 x ] Z i (˜ x 1 i )) , ..., ( T z ) (cid:48) i ([ f − 1 x ] S i (˜ x k z i )) (cid:17) for each i ∈ [ q t ] are linearly independent.

Original: By differentiating Eq.
Revised: By differentiating Eq.

Original: (45) and Eq.
Revised: (45) and Eq.

Original: (46) for each ¯ x i with i ∈
Revised: (46) for each ¯ x i with i ∈

Original: [ q s ] and ˜¯ x i with i ∈
Revised: [ q s ] and ˜¯ x i with i ∈

Original: [ q z ] respectively, we have that
Revised: [ q z ] respectively, we have that

Original: [Equation]
Revised: [Equation]

Original: The linearly independence of
Revised: The linearly independence of

Original: [Equation]
Revised: [Equation]

Original: and (cid:16)
Revised: and (cid:16)

Original: [Equation]
Revised: [Equation]

Original: T z )
Revised: T z )

Original: [Equation]
Revised: [Equation]

Original: i ([
Revised: i ([

Original: [Equation]
Revised: [Equation]

Original: 1 x ] Z i (˜
Revised: 1 x ] Z i (˜

Original: [Equation]
Revised: [Equation]

Original: ) , ..., ( T
Revised: ) , ..., ( T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i ([ f − 1 x ] S i (˜ x
Revised: i ([ f − 1 x ] S i (˜ x

Original: [Equation]
Revised: [Equation]

Original: z i )) (cid:17) imply that the (cid:0) J T s (¯ x 1 ) , ..., J T s (¯ x k s ) (cid:1) and (cid:0) J T z (˜¯ x 1 ) , ..., J T z (˜¯ x k z ) (cid:1) are invertible, which implies the invertibility of matrix M s and M z .
Revised: z i )) (cid:17) imply that the (cid:0) J T s (¯ x 1 ) , ..., J T s (¯ x k s ) (cid:1) and (cid:0) J T z (˜¯ x 1 ) , ..., J T z (˜¯ x k z ) (cid:1) are invertible, which implies the invertibility of matrix M s and M z .

Original: The rest is to prove
Revised: The rest is to prove

Original: [Equation]
Revised: [Equation]

Original: ( x )) .
Revised: ( x )) .

Original: This can be shown by applying Eq.
Revised: This can be shown by applying Eq.

Original: (31) again.
Revised: (31) again.

Original: Speciﬁcally, according to Eq.
Revised: Speciﬁcally, according to Eq.

Original: (31), we have that
Revised: (31), we have that

Original: [Equation]
Revised: [Equation]

Original: Denote l
Revised: Denote l

Original: [Equation]
Revised: [Equation]

Original: ¯ x ) , we have
Revised: ¯ x ) , we have

Original: [Equation]
Revised: [Equation]

Original: Taking the log transformation on both sides of Eq.
Revised: Taking the log transformation on both sides of Eq.

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: Subtracting Eq.
Revised: Subtracting Eq.

Original: (57) with y 2 from Eq.
Revised: (57) with y 2 from Eq.

Original: (57) with y 1 , we have
Revised: (57) with y 1 , we have

Original: [Equation]
Revised: [Equation]

Original: for any y 1 ∈ Y .
Revised: for any y 1 ∈ Y .

Original: This completes the proof.
Revised: This completes the proof.

Original: Understanding the assumption (4) in Theorem 7.9 and 7.6.
Revised: Understanding the assumption (4) in Theorem 7.9 and 7.6.

Original: Recall that we assume the confounder d s in LaCIM is the source variable for generating data in corresponding domain.
Revised: Recall that we assume the confounder d s in LaCIM is the source variable for generating data in corresponding domain.

Original: Here we also use the C to denote the space of d s (since d s := c ), then we have the following theoretical conclusion that the as long as the image set of C is not included in any sets with Lebesgue measure 0, the assumption (4) holds.
Revised: Here we also use the C to denote the space of d s (since d s := c ), then we have the following theoretical conclusion that the as long as the image set of C is not included in any sets with Lebesgue measure 0, the assumption (4) holds.

Original: This conclusion means that the assumption (4) holds generically.
Revised: This conclusion means that the assumption (4) holds generically.

Original: Theorem 7.10.
Revised: Theorem 7.10.

Original: Denote h t =
Revised: Denote h t =

Original: s,z ( d ) :=
Revised: s,z ( d ) :=

Original: (cid:16) Γ t 1 , 1 ( d ) − Γ t 1 , 1 ( d e 1 ) , ..., Γ tq t ,k t ( d ) − Γ t 1 , 1 ( d e 1 ) (cid:17) (cid:62) , h ( C ) :
Revised: (cid:16) Γ t 1 , 1 ( d ) − Γ t 1 , 1 ( d e 1 ) , ..., Γ tq t ,k t ( d ) − Γ t 1 , 1 ( d e 1 ) (cid:17) (cid:62) , h ( C ) :

Original: = h s ( S ) ⊕ h z
Revised: = h s ( S ) ⊕ h z

Original: ( Z ) ⊂
Revised: ( Z ) ⊂

Original: R q z ∗ k z ⊕ R q s ∗ k s , then assumption (4) holds if h ( C ) is not included in any zero-measure set of R q z ∗
Revised: R q z ∗ k z ⊕ R q s ∗ k s , then assumption (4) holds if h ( C ) is not included in any zero-measure set of R q z ∗

Original: k z ⊕ R q s ∗ k s .
Revised: k z ⊕ R q s ∗ k s .

Original: Denote r s :=
Revised: Denote r s :=

Original: q s ∗ k s and r z :=
Revised: q s ∗ k s and r z :=

Original: q z ∗ k z .
Revised: q z ∗ k z .

Original: Proof.
Revised: Proof.

Original: With loss of generality, we assume that r s ≤ r z .
Revised: With loss of generality, we assume that r s ≤ r z .

Original: Denote Q as the set of integers q such that there exists d e 2 , ..., d q +1 that the rank([ h z ( d e 2 ) , ..., h z
Revised: Denote Q as the set of integers q such that there exists d e 2 , ..., d q +1 that the rank([ h z ( d e 2 ) , ..., h z

Original: ( d e q +1 )])
Revised: ( d e q +1 )])

Original: =
Revised: =

Original: min( q, r z ) and rank([ h s ( d e 2 ) , ..., h s ( d e q +1 )])
Revised: min( q, r z ) and rank([ h s ( d e 2 ) , ..., h s ( d e q +1 )])

Original: =
Revised: =

Original: min( q, r s ) .
Revised: min( q, r s ) .

Original: Denote u := max( Q ) .
Revised: Denote u := max( Q ) .

Original: We discuss two possible cases for u , respectively:
Revised: We discuss two possible cases for u , respectively:

Original: • Case 1.
Revised: • Case 1.

Original: u < r s ≤ r z .
Revised: u < r s ≤ r z .

Original: Then there exists d e 2 , ..., d e u +1 s.t.
Revised: Then there exists d e 2 , ..., d e u +1 s.t.

Original: h z ( d e 2 ) , ..., h z ( d e u +1 ) and h s ( d e 2 ) , ..., h s ( d e u +1 ) are linearly independent.
Revised: h z ( d e 2 ) , ..., h z ( d e u +1 ) and h s ( d e 2 ) , ..., h s ( d e u +1 ) are linearly independent.

Original: Then ∀ c , we have h z ( d ) ∈ L ( h z ( d e 2 ) , ...,
Revised: Then ∀ c , we have h z ( d ) ∈ L ( h z ( d e 2 ) , ...,

Original: h z ( d e u +1 )) or h s ( d ) ∈ L ( h s ( d e 2 ) , ..., h s ( d e u +1 )) .
Revised: h z ( d e u +1 )) or h s ( d ) ∈ L ( h s ( d e 2 ) , ..., h s ( d e u +1 )) .

Original: Therefore, so we have h z ( d ) ⊕ h s ( d ) ∈
Revised: Therefore, so we have h z ( d ) ⊕ h s ( d ) ∈

Original: [ L ( h z ( d e 2 ) , ..., h z ( d e u +1 ))
Revised: [ L ( h z ( d e 2 ) , ..., h z ( d e u +1 ))

Original: ⊕ R r s ] ∪ [ R r z ⊕ L
Revised: ⊕ R r s ] ∪ [ R r z ⊕ L

Original: ( h s ( d e 2 ) , ..., h s ( d e u +1 ))] , which has measure 0 in R r z ⊕ R r s
Revised: ( h s ( d e 2 ) , ..., h s ( d e u +1 ))] , which has measure 0 in R r z ⊕ R r s

Original: • Case 2.
Revised: • Case 2.

Original: r s ≤ u < r z .
Revised: r s ≤ u < r z .

Original: Then there exists d e 2 , ..., d e u +1 s.t.
Revised: Then there exists d e 2 , ..., d e u +1 s.t.

Original: h z ( d e 2 ) , ..., h z ( d e u +1 ) are linearly independent and rank ([ h s ( d e 1 ) , ..., h s ( d e u )])
Revised: h z ( d e 2 ) , ..., h z ( d e u +1 ) are linearly independent and rank ([ h s ( d e 1 ) , ..., h s ( d e u )])

Original: =
Revised: =

Original: r s .
Revised: r s .

Original: Then ∀ c , we have h z ( d ) ∈ L ( h z ( d e 1 ) , ...,
Revised: Then ∀ c , we have h z ( d ) ∈ L ( h z ( d e 1 ) , ...,

Original: h z ( d e u +1 )) , which means that h z ( d ) ⊕ h s ( d ) ∈ L ( h z ( d e 1 ) , ...,
Revised: h z ( d e u +1 )) , which means that h z ( d ) ⊕ h s ( d ) ∈ L ( h z ( d e 1 ) , ...,

Original: h z
Revised: h z

Original: ( d e u +1 ))
Revised: ( d e u +1 ))

Original: ⊕ R r s , which has measure 0 in R r z ⊕ R r s .
Revised: ⊕ R r s , which has measure 0 in R r z ⊕ R r s .

Original: The above two cases are contradict to the assumption that h ( C ) is not included in any zero-measure set of R r z ⊕ R r s .
Revised: The above two cases are contradict to the assumption that h ( C ) is not included in any zero-measure set of R r z ⊕ R r s .

Original: Lemma 7.11.
Revised: Lemma 7.11.

Original: Consider the cases when k s ≥ 2 .
Revised: Consider the cases when k s ≥ 2 .

Original: Then suppose the assumptions in theorem 7.9 are satisﬁed.
Revised: Then suppose the assumptions in theorem 7.9 are satisﬁed.

Original: Further assumed that
Revised: Further assumed that

Original: • The sufﬁcient statistics T si,j are twice differentiable for each i ∈ [ q s ] and j ∈ [ k s ] .
Revised: • The sufﬁcient statistics T si,j are twice differentiable for each i ∈ [ q s ] and j ∈ [ k s ] .

Original: • f y is twice differentiable.
Revised: • f y is twice differentiable.

Original: Then we have M s in theorem 7.9 is block permutation matrix.
Revised: Then we have M s in theorem 7.9 is block permutation matrix.

Original: Proof.
Revised: Proof.

Original: Directly applying (Khemakhem, Kingma and Hyv ¨ arinen, 2020, Theorem 2) with f x , A, b, T , x replaced by f y , M s , a s , T s , y .
Revised: Directly applying (Khemakhem, Kingma and Hyv ¨ arinen, 2020, Theorem 2) with f x , A, b, T , x replaced by f y , M s , a s , T s , y .

Original: Lemma 7.12. Consider the cases when k s = 1 .
Revised: Lemma 7.12. Consider the cases when k s = 1 .

Original: Then suppose the assumptions in theorem 7.9 are satisﬁed.
Revised: Then suppose the assumptions in theorem 7.9 are satisﬁed.

Original: Further assumed that
Revised: Further assumed that

Original: • The sufﬁcient statistics T si are not monotonic for i ∈ [ q s ] .
Revised: • The sufﬁcient statistics T si are not monotonic for i ∈ [ q s ] .

Original: • g is smooth.
Revised: • g is smooth.

Original: Then we have M s in theorem 7.9 is block permutation matrix.
Revised: Then we have M s in theorem 7.9 is block permutation matrix.

Original: Proof.
Revised: Proof.

Original: Directly applying (Khemakhem, Kingma and Hyv ¨ arinen, 2020, Theorem 3) with f x , A, b, T , x replaced by f y , M s , a s , T s , y .
Revised: Directly applying (Khemakhem, Kingma and Hyv ¨ arinen, 2020, Theorem 3) with f x , A, b, T , x replaced by f y , M s , a s , T s , y .

Original: Proof of Theorem 7.6.
Revised: Proof of Theorem 7.6.

Original: According to theorem 7.9, there exist invertible matrices M s and M z such that
Revised: According to theorem 7.9, there exist invertible matrices M s and M z such that

Original: [Equation]
Revised: [Equation]

Original: s , where T =
Revised: s , where T =

Original: [ T s, (cid:62) , T z, (cid:62) ] (cid:62) , and
Revised: [ T s, (cid:62) , T z, (cid:62) ] (cid:62) , and

Original: [Equation]
Revised: [Equation]

Original: By further assuming that the sufﬁcient statistics T si,j are twice differentiable for each i ∈ [ q s ] and j ∈ [ k s ] for k s ≥ 2 and not monotonic for k s = 1 .
Revised: By further assuming that the sufﬁcient statistics T si,j are twice differentiable for each i ∈ [ q s ] and j ∈ [ k s ] for k s ≥ 2 and not monotonic for k s = 1 .

Original: Then we have that M s is block permutation matrix.
Revised: Then we have that M s is block permutation matrix.

Original: By further assuming that T zi,j are twice differentiable for each i ∈ [ n z ] and j ∈ [ k z ] for k z ≥ 2 and not monotonic for k z
Revised: By further assuming that T zi,j are twice differentiable for each i ∈ [ n z ] and j ∈ [ k z ] for k z ≥ 2 and not monotonic for k z

Original: = 1 and applying the lemma 7.11 and 7.12 respectively, we have that A is block permutation matrix.
Revised: = 1 and applying the lemma 7.11 and 7.12 respectively, we have that A is block permutation matrix.

Original: Therefore, M z is also a block permutation matrix.
Revised: Therefore, M z is also a block permutation matrix.

Original: Proof of Theorem 4.3.
Revised: Proof of Theorem 4.3.

Original: We consider the general case when C := ∪ Rr =1 C r , in which each C r can be simpliﬁed as a representative point c r .
Revised: We consider the general case when C := ∪ Rr =1 C r , in which each C r can be simpliﬁed as a representative point c r .

Original: For environment d e , let P d e =
Revised: For environment d e , let P d e =

Original: [P( C = c 1 | d e ) , · · · , P( C =
Revised: [P( C = c 1 | d e ) , · · · , P( C =

Original: c R | d e )] be the vector of probability mass of C in the environment d e .
Revised: c R | d e )] be the vector of probability mass of C in the environment d e .

Original: And E train has m environments with indexes d e 1 , · · · , d e m .
Revised: And E train has m environments with indexes d e 1 , · · · , d e m .

Original: The latent factors ( S, Z ) belongs to the exponential family distribution p ( s, z | c ) =
Revised: The latent factors ( S, Z ) belongs to the exponential family distribution p ( s, z | c ) =

Original: p T z , Γ z ( d ) ( z ) p T s , Γ s ( d ) ( s ) .
Revised: p T z , Γ z ( d ) ( z ) p T s , Γ s ( d ) ( s ) .

Original: Suppose that θ = { f x , f y , T s , T z } and ˜ θ =
Revised: Suppose that θ = { f x , f y , T s , T z } and ˜ θ =

Original: { ˜ f x , ˜ g y , ˜ T s , ˜ T z } share the same observational distribution for each environment, i.e. , p θ ( x, y | d e ) =
Revised: { ˜ f x , ˜ g y , ˜ T s , ˜ T z } share the same observational distribution for each environment, i.e. , p θ ( x, y | d e ) =

Original: p ˜ θ
Revised: p ˜ θ

Original: ( x, y | d e ) , then we have that
Revised: ( x, y | d e ) , then we have that

Original: [Equation]
Revised: [Equation]

Original: Let ∆ x,y =
Revised: Let ∆ x,y =

Original: [ p θ ( x, y | c 1 ) − p ˜
Revised: [ p θ ( x, y | c 1 ) − p ˜

Original: θ ( x, y | c 1 ) , · · · , p θ ( x, y | c m ) − p ˜
Revised: θ ( x, y | c 1 ) , · · · , p θ ( x, y | c m ) − p ˜

Original: θ ( x, y | c m )]
Revised: θ ( x, y | c m )]

Original: T , then Eq.
Revised: T , then Eq.

Original: (62) can be written as A ∆ x,y = 0 .
Revised: (62) can be written as A ∆ x,y = 0 .

Original: Denote A := P (cid:62) d e 1 ∈ R m × R .
Revised: Denote A := P (cid:62) d e 1 ∈ R m × R .

Original: According the diversity condition , we have that A and the [[ Γ t ( c 2 ) − Γ t ( c 1 )]
Revised: According the diversity condition , we have that A and the [[ Γ t ( c 2 ) − Γ t ( c 1 )]

Original: T , ..., [ Γ t ( c m ) − Γ t ( c 1 )]
Revised: T , ..., [ Γ t ( c m ) − Γ t ( c 1 )]

Original: T ] T have full column rank, therefore we have that ∆ x,y = 0 , i.e. p θ ( x, y | c r ) =
Revised: T ] T have full column rank, therefore we have that ∆ x,y = 0 , i.e. p θ ( x, y | c r ) =

Original: p
Revised: p

Original: ˜ θ
Revised: ˜ θ

Original: ( x, y | c r ) for each r ∈
Revised: ( x, y | c r ) for each r ∈

Original: [ R ] .
Revised: [ R ] .

Original: The left proof is the same with the one in theorem 7.6.
Revised: The left proof is the same with the one in theorem 7.6.

Original: 7. P ROOF OF T HEOREM 4. 
Revised: 7. P ROOF OF T HEOREM 4.

Original: Proof of Theorem 4.4.
Revised: Proof of Theorem 4.4.

Original: Due to Eq.
Revised: Due to Eq.

Original: (62), it is sufﬁces to prove the conclusion for every c r ∈ { c r } r ∈
Revised: (62), it is sufﬁces to prove the conclusion for every c r ∈ { c r } r ∈

Original: [ R ] .
Revised: [ R ] .

Original: Motivated by Barron and Sheu (1991, Theorem 2) that the distribution p e ( s, z ) deﬁned on bounded set can be approximated by a sequence of exponential family with sufﬁcient statistics denoted as polynomial terms, therefore the T t =
Revised: Motivated by Barron and Sheu (1991, Theorem 2) that the distribution p e ( s, z ) deﬁned on bounded set can be approximated by a sequence of exponential family with sufﬁcient statistics denoted as polynomial terms, therefore the T t =

Original: s,z are twice differentiable hence satisﬁes the assumption (2) in theorem 4.3 and assumption (1) in lemma 7.11.
Revised: s,z are twice differentiable hence satisﬁes the assumption (2) in theorem 4.3 and assumption (1) in lemma 7.11.

Original: Besides, the lemma 4 in Barron and Sheu (1991) informs us that the KL divergence between p θ 0 ( s, z | c r ) ( θ 0 :
Revised: Besides, the lemma 4 in Barron and Sheu (1991) informs us that the KL divergence between p θ 0 ( s, z | c r ) ( θ 0 :

Original: =
Revised: =

Original: ( f x , f y , T z , T s , Γ z 0 , Γ s 0 ) and p θ 1 ( s, z |
Revised: ( f x , f y , T z , T s , Γ z 0 , Γ s 0 ) and p θ 1 ( s, z |

Original: c r ) ( θ 1 :=
Revised: c r ) ( θ 1 :=

Original: ( f x , f y , T z , T s , Γ z 1 , Γ s 1 ) (the p θ 0 ( s, z | c r ) , p θ 1 ( s, z | c r ) belong to exponential family with polynomial sufﬁcient statistics terms) can be bounded by the (cid:96) 2 norm of [( Γ s ( c r ) − Γ s 1 ( c r )) (cid:62) , ( Γ z 0 ( c r ) − Γ z 1 ( c r )) (cid:62) ] (cid:62) .
Revised: ( f x , f y , T z , T s , Γ z 1 , Γ s 1 ) (the p θ 0 ( s, z | c r ) , p θ 1 ( s, z | c r ) belong to exponential family with polynomial sufﬁcient statistics terms) can be bounded by the (cid:96) 2 norm of [( Γ s ( c r ) − Γ s 1 ( c r )) (cid:62) , ( Γ z 0 ( c r ) − Γ z 1 ( c r )) (cid:62) ] (cid:62) .

Original: Therefore, ∀ (cid:15) > 0 , there exists a open set of Γ( c r ) such that the D KL ( p ( s, z | c r ) , p θ ( s, z | c r )) < (cid:15) .
Revised: Therefore, ∀ (cid:15) > 0 , there exists a open set of Γ( c r ) such that the D KL ( p ( s, z | c r ) , p θ ( s, z | c r )) < (cid:15) .

Original: Such an open set is with non-zero Lebesgue measurement therefore can satisfy the assumption (4) in theorem 4.3, according to result in theorem 7.10.
Revised: Such an open set is with non-zero Lebesgue measurement therefore can satisfy the assumption (4) in theorem 4.3, according to result in theorem 7.10.

Original: The left is to prove that for any p deﬁned by a LaCIM following Def. 4.1, there is a sequence of { p m } n ∈ P exp such that the d Pok ( p, p n ) → 0 that is equivalent to p n d → p .
Revised: The left is to prove that for any p deﬁned by a LaCIM following Def. 4.1, there is a sequence of { p m } n ∈ P exp such that the d Pok ( p, p n ) → 0 that is equivalent to p n d → p .

Original: For any A, B , we consider to prove that
Revised: For any A, B , we consider to prove that

Original: [Equation]
Revised: [Equation]

Original: where p n ( x ∈ A, y n ∈ B | c r ) = (cid:82) S (cid:82) Z p ( x ∈
Revised: where p n ( x ∈ A, y n ∈ B | c r ) = (cid:82) S (cid:82) Z p ( x ∈

Original: A | s, z ) p ( y n ∈ B | s ) p n ( s, z | c r ) dsdz with
Revised: A | s, z ) p ( y n ∈ B | s ) p n ( s, z | c r ) dsdz with

Original: [Equation]
Revised: [Equation]

Original: 64) for y ∈ R k denoting the k -dimensional one-hot vector for categorical variable and ε y, 1 ,..., k are
Revised: 64) for y ∈ R k denoting the k -dimensional one-hot vector for categorical variable and ε y, 1 ,..., k are

Original: Gumbel i.i.d.
Revised: Gumbel i.i.d.

Original: According to (Maddison et al., 2016, Proposition 1) that the y n ( i ) d
Revised: According to (Maddison et al., 2016, Proposition 1) that the y n ( i ) d

Original: → y ( i ) with p ( y ( i
Revised: → y ( i ) with p ( y ( i

Original: [Equation]
Revised: [Equation]

Original: As long as f y is smooth, we have that the p ( y n | s ) is continuous.
Revised: As long as f y is smooth, we have that the p ( y n | s ) is continuous.

Original: We have that
Revised: We have that

Original: I
Revised: I

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For I n, 1 , if y is itself additive model with y = f y ( s ) + ε y , then we just set y n d = y , then we have that I n, 1 = 0 .
Revised: For I n, 1 , if y is itself additive model with y = f y ( s ) + ε y , then we just set y n d = y , then we have that I n, 1 = 0 .

Original: Therefore, we only consider the case when y denotes the categorical variable with softmax distribution, i.e. , Eq.
Revised: Therefore, we only consider the case when y denotes the categorical variable with softmax distribution, i.e. , Eq.

Original: ∀ c r ∈ C := { c 1 , ..., c R } and ∀ (cid:15) > 0 , there exists M c r s and M c r z such that p ( s, z ∈ M c r s × M c r z | c r ) ≤ (cid:15) ;
Revised: ∀ c r ∈ C := { c 1 , ..., c R } and ∀ (cid:15) > 0 , there exists M c r s and M c r z such that p ( s, z ∈ M c r s × M c r z | c r ) ≤ (cid:15) ;

Original: Denote M s ∆ = ∪ mk =1 M c r s and M z ∆ = ∪ mk =1 M c r z , we have that p ( s, z ∈ M s
Revised: Denote M s ∆ = ∪ mk =1 M c r s and M z ∆ = ∪ mk =1 M c r z , we have that p ( s, z ∈ M s

Original: ×
Revised: ×

Original: M z | c ) ≤ 2 (cid:15) for all c r ∈ C .
Revised: M z | c ) ≤ 2 (cid:15) for all c r ∈ C .

Original: Since ∀ s 1 ∈ M s , ∃ N s 1 such that ∀ n ≥ N s 1 , we have that (cid:12)(cid:12)(cid:12) p ( y ∈ B | s 1 ) − p ( y ∈ B | s 1 ) | ≤ (cid:15) from that y n d → y .
Revised: Since ∀ s 1 ∈ M s , ∃ N s 1 such that ∀ n ≥ N s 1 , we have that (cid:12)(cid:12)(cid:12) p ( y ∈ B | s 1 ) − p ( y ∈ B | s 1 ) | ≤ (cid:15) from that y n d → y .

Original: Besides, there exists open set
Revised: Besides, there exists open set

Original: O s 1 such that ∀ s ∈
Revised: O s 1 such that ∀ s ∈

Original: O s 1 and
Revised: O s 1 and

Original: [Equation]
Revised: [Equation]

Original: Again, according to Heine–Borel theorem, there exists ﬁnite s , namely s 1 , ..., s l such that M s ⊂ ∪ li =1 O
Revised: Again, according to Heine–Borel theorem, there exists ﬁnite s , namely s 1 , ..., s l such that M s ⊂ ∪ li =1 O

Original: ( s i ) .
Revised: ( s i ) .

Original: Then there exists N ∆ = max { N s 1 , ..., N s l } such that ∀ n ≥ N , we have that
Revised: Then there exists N ∆ = max { N s 1 , ..., N s l } such that ∀ n ≥ N , we have that

Original: [Equation]
Revised: [Equation]

Original: Therefore, I n, 1 ≤ (cid:82) M s × M z 3 (cid:15)p ( x ∈ A | s, z ) p ( s, z | c ) dsdz ≤ 3 (cid:15) .
Revised: Therefore, I n, 1 ≤ (cid:82) M s × M z 3 (cid:15)p ( x ∈ A | s, z ) p ( s, z | c ) dsdz ≤ 3 (cid:15) .

Original: Hence, I n, 1 → 0 as n → ∞ .
Revised: Hence, I n, 1 → 0 as n → ∞ .

Original: Besides, we have that I n, 2 ≤ (cid:82) M s × M z 2 (cid:15)p ( s, z | c r ) dsdz ≤ 2 (cid:15) .
Revised: Besides, we have that I n, 2 ≤ (cid:82) M s × M z 2 (cid:15)p ( s, z | c r ) dsdz ≤ 2 (cid:15) .

Original: Therefore, we have that (cid:12)(cid:12) (cid:82) S×Z p ( x ∈
Revised: Therefore, we have that (cid:12)(cid:12) (cid:82) S×Z p ( x ∈

Original: A | s, z ) ( p ( y ∈ B | s ) − p ( y n ∈ B | s ))
Revised: A | s, z ) ( p ( y ∈ B | s ) − p ( y n ∈ B | s ))

Original: p ( s, z | c r ) dsdz (cid:12)(cid:12) → 0 as n → ∞ .
Revised: p ( s, z | c r ) dsdz (cid:12)(cid:12) → 0 as n → ∞ .

Original: For I n, 3 , we have that
Revised: For I n, 3 , we have that

Original: I n
Revised: I n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The I n, 3 , 1 ≤ (cid:15) 1 − (cid:15) .
Revised: The I n, 3 , 1 ≤ (cid:15) 1 − (cid:15) .

Original: Denote ˜ p
Revised: Denote ˜ p

Original: ( s, z |
Revised: ( s, z |

Original: c r ) :
Revised: c r ) :

Original: = p ( s,z | c r ) 1 ( s,z
Revised: = p ( s,z | c r ) 1 ( s,z

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: z ) p ( s,z
Revised: z ) p ( s,z

Original: [Equation]
Revised: [Equation]

Original: s × M z | c r ) , according to (Barron and Sheu, 1991, Theorem 2), there exists a sequence of p n ( s, z | c ) deﬁned on a compact support M s × M z such that ∀ c r ∈ C , we have that
Revised: s × M z | c r ) , according to (Barron and Sheu, 1991, Theorem 2), there exists a sequence of p n ( s, z | c ) deﬁned on a compact support M s × M z such that ∀ c r ∈ C , we have that

Original: [Equation]
Revised: [Equation]

Original: Applying again the Heine–Borel theorem, we have that ∀ (cid:15) , ∃ N such that ∀ n ≥ N , we have
Revised: Applying again the Heine–Borel theorem, we have that ∀ (cid:15) , ∃ N such that ∀ n ≥ N , we have

Original: [Equation]
Revised: [Equation]

Original: which implies that I n, 3 , 2 → 0 as n → ∞ combining with the fact that p ( x, y | s, z ) is continuous with respect to s, z .
Revised: which implies that I n, 3 , 2 → 0 as n → ∞ combining with the fact that p ( x, y | s, z ) is continuous with respect to s, z .

Original: For I n, 4 , we have that
Revised: For I n, 4 , we have that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality is from that the p n ( s, z | c r ) is deﬁned on M s × M z .
Revised: where the ﬁrst equality is from that the p n ( s, z | c r ) is deﬁned on M s × M z .

Original: Then we have that
Revised: Then we have that

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: 7. R EPARAMETERIZATION FOR L A CIM- d
Revised: 7. R EPARAMETERIZATION FOR L A CIM- d

Original: We provide an alternative training method to avoid parameterization of prior p ( s, z | d e ) to increase the diversity of generative models in different environments.
Revised: We provide an alternative training method to avoid parameterization of prior p ( s, z | d e ) to increase the diversity of generative models in different environments.

Original: Speciﬁcally, motivated by Hyv¨arinen and Pajunen (1999) that any distribution can be transformed to isotropic Gaussian with the density denoted by p Gau , we have that for any e ∈ E train , we have
Revised: Speciﬁcally, motivated by Hyv¨arinen and Pajunen (1999) that any distribution can be transformed to isotropic Gaussian with the density denoted by p Gau , we have that for any e ∈ E train , we have

Original: [Equation]
Revised: [Equation]

Original: with s (cid:48) , z (cid:48) :
Revised: with s (cid:48) , z (cid:48) :

Original: = ρ es ( s )
Revised: = ρ es ( s )

Original: [Equation]
Revised: [Equation]

Original: ρ ez ( z ) ∼ N (0 , I ) .
Revised: ρ ez ( z ) ∼ N (0 , I ) .

Original: We can then rewrite ELBO for LaCIM- d for environment e as:
Revised: We can then rewrite ELBO for LaCIM- d for environment e as:

Original: [Equation]
Revised: [Equation]

Original: 7. I DENTIFIABILITY
Revised: 7. I DENTIFIABILITY

Original: Earlier works that identify the latent confounders rely on strong assumptions regarding the causal structure, such as the linear model from latent to observed variable or ICA in which the latent component are independent Silva et al.
Revised: Earlier works that identify the latent confounders rely on strong assumptions regarding the causal structure, such as the linear model from latent to observed variable or ICA in which the latent component are independent Silva et al.

Original: (2006), or noise-free model
Revised: (2006), or noise-free model

Original: Shimizu et al.
Revised: Shimizu et al.

Original: ; Davies (2004).
Revised: ; Davies (2004).

Original: The Hoyer et al.
Revised: The Hoyer et al.

Original: (2008); Janzing, Peters, Mooij and Sch ¨ olkopf (2012) extend to the additive noise model (ANM) and other causal discovery assumptions.
Revised: (2008); Janzing, Peters, Mooij and Sch ¨ olkopf (2012) extend to the additive noise model (ANM) and other causal discovery assumptions.

Original: Although the Lee et al.
Revised: Although the Lee et al.

Original: (2019) relaxed the constraints put on the causal structure, it required the latent noise is with small strength, which does not match with many realistic scenarios, such as the structural MRI of Alzheimer’s Disease considered in our experiment.
Revised: (2019) relaxed the constraints put on the causal structure, it required the latent noise is with small strength, which does not match with many realistic scenarios, such as the structural MRI of Alzheimer’s Disease considered in our experiment.

Original: The works which also based on the independent component analysis (ICA), i.e. , the latent variables are (conditionally) independent, include Davies (2004); Eriksson and Koivunen (2003); recently, a series of works extend the above results to deep nonlinear ICA (Hyvarinen and Morioka, 2016;
Revised: The works which also based on the independent component analysis (ICA), i.e. , the latent variables are (conditionally) independent, include Davies (2004); Eriksson and Koivunen (2003); recently, a series of works extend the above results to deep nonlinear ICA (Hyvarinen and Morioka, 2016;

Original: Hyv¨arinen et al., 2019; Khemakhem, Kingma and Hyv¨arinen, 2020;
Revised: Hyv¨arinen et al., 2019; Khemakhem, Kingma and Hyv¨arinen, 2020;

Original: Khemakhem, Monti, Kingma and Hyv¨arinen, 2020; Teshima et al., 2020).
Revised: Khemakhem, Monti, Kingma and Hyv¨arinen, 2020; Teshima et al., 2020).

Original: However, these works require that the value of confounder of these latent variables is ﬁxed, which cannot explain the spurious correlation in a single dataset.
Revised: However, these works require that the value of confounder of these latent variables is ﬁxed, which cannot explain the spurious correlation in a single dataset.

Original: In contrast, our result can incorporate these scenarios by assuming that each sample has a speciﬁc value of the confounder.
Revised: In contrast, our result can incorporate these scenarios by assuming that each sample has a speciﬁc value of the confounder.

Original: Other works assume discrete distribution for latent variables, such as Janzing, Sgouritsa, Stegle, Peters and Sch¨olkopf (2012); Kocaoglu et al.
Revised: Other works assume discrete distribution for latent variables, such as Janzing, Sgouritsa, Stegle, Peters and Sch¨olkopf (2012); Kocaoglu et al.

Original: ; Sgouritsa et al.
Revised: ; Sgouritsa et al.

Original: However, in the literature, no existing works can disentangle the prediction-causative features from others, in the scenario of avoiding spurious correlation in order for
Revised: However, in the literature, no existing works can disentangle the prediction-causative features from others, in the scenario of avoiding spurious correlation in order for

Original: OOD generalization.
Revised: OOD generalization.

Original: 7. C OMPARISON WITH EXISTING WORKS 7.7. Y → S OR S → Y ?
Revised: 7. C OMPARISON WITH EXISTING WORKS 7.7. Y → S OR S → Y ?

Original: Many existing works Rojas-Carulla et al.
Revised: Many existing works Rojas-Carulla et al.

Original: ; Khemakhem, Monti, Kingma and Hyv¨arinen (2020);
Revised: ; Khemakhem, Monti, Kingma and Hyv¨arinen (2020);

Original: Ilse et al.
Revised: Ilse et al.

Original: (2020; 2019) assumed Y → S ( X ) as the causal direction.
Revised: (2020; 2019) assumed Y → S ( X ) as the causal direction.

Original: Such an difference from ours can mainly be contributed to the generating process of Y .
Revised: Such an difference from ours can mainly be contributed to the generating process of Y .

Original: Different understanding leads to different causal graph.
Revised: Different understanding leads to different causal graph.

Original: The example of digital hand-writing in Peters et al.
Revised: The example of digital hand-writing in Peters et al.

Original: (2017) provides a good explanation.
Revised: (2017) provides a good explanation.

Original: Consider the case that the writer is provided with a label ﬁrst (such as ”2”) before writing the digit (denoted as X ), then it should be Y → X .
Revised: Consider the case that the writer is provided with a label ﬁrst (such as ”2”) before writing the digit (denoted as X ), then it should be Y → X .

Original: Consider another case, when the writing is based on the incentive (denoted as S ) of which digit to write, then the writer record the label Y and the digit X concurrently, in which case it should be X ← S
Revised: Consider another case, when the writing is based on the incentive (denoted as S ) of which digit to write, then the writer record the label Y and the digit X concurrently, in which case it should be X ← S

Original: → Y .
Revised: → Y .

Original: For Y → S , the Y is thought to be the source variable that generates the latent components and is observed before X .
Revised: For Y → S , the Y is thought to be the source variable that generates the latent components and is observed before X .

Original: In contrast, we deﬁne Y as ground-truth labels given by humans.
Revised: In contrast, we deﬁne Y as ground-truth labels given by humans.

Original: Taking image classiﬁcation as an example, it is the human that give the classiﬁcation of all things such as animals.
Revised: Taking image classiﬁcation as an example, it is the human that give the classiﬁcation of all things such as animals.

Original: In this case, it can be assumed that the label given by humans are ground-truth labels.
Revised: In this case, it can be assumed that the label given by humans are ground-truth labels.

Original: This assumption can be based by the work Biederman (1987) in the ﬁeld of psychology that humans can factorize the image X by many components due to the powerful perception learning ability of human beings.
Revised: This assumption can be based by the work Biederman (1987) in the ﬁeld of psychology that humans can factorize the image X by many components due to the powerful perception learning ability of human beings.

Original: These components which denoted as
Revised: These components which denoted as

Original: S , can be accurately detected by humans, therefore we can approximately assume that it is the S generating the label Y .
Revised: S , can be accurately detected by humans, therefore we can approximately assume that it is the S generating the label Y .

Original: Consider the task of early prediction in Alzheimer’s Disease, the disease label is given based on the pathological analysis and observed after the MRI X .
Revised: Consider the task of early prediction in Alzheimer’s Disease, the disease label is given based on the pathological analysis and observed after the MRI X .

Original: Such a labelling outcome can be regarded as the ground-truth which itself is deﬁned by medical science.
Revised: Such a labelling outcome can be regarded as the ground-truth which itself is deﬁned by medical science.

Original: The corresponding pathology features, as the evidences for labelling, can also thought as the generators of X .
Revised: The corresponding pathology features, as the evidences for labelling, can also thought as the generators of X .

Original: In these cases, it is more appropriate to assume the Y as the outcome than the cause.
Revised: In these cases, it is more appropriate to assume the Y as the outcome than the cause.

Original: For example, the Peters et al.
Revised: For example, the Peters et al.

Original: ; Kuang et al.
Revised: ; Kuang et al.

Original: (2018) assumed X S → Y .
Revised: (2018) assumed X S → Y .

Original: As an adaptation to sensory-level data such as image, we assume S → Y with S are latent variables to model high-level explanatory factors, which coincides with existing literature Teshima et al.
Revised: As an adaptation to sensory-level data such as image, we assume S → Y with S are latent variables to model high-level explanatory factors, which coincides with existing literature Teshima et al.

Original: Another difference lies in the deﬁnition of Y .
Revised: Another difference lies in the deﬁnition of Y .

Original: The Invariant Risk Minimization (we will give a detailed comparison later) Arjovsky et al.
Revised: The Invariant Risk Minimization (we will give a detailed comparison later) Arjovsky et al.

Original: (2019) assumes that X → ˜ S
Revised: (2019) assumes that X → ˜ S

Original: → Y by deﬁning the Y as the label with noise.
Revised: → Y by deﬁning the Y as the label with noise.

Original: The ˜ S denoted as the extracted hidden components by observer.
Revised: The ˜ S denoted as the extracted hidden components by observer.

Original: 7.7. C OMPARISONS WITH DATA AUGMENTATION & ARCHITECTURE DESIGN
Revised: 7.7. C OMPARISONS WITH DATA AUGMENTATION & ARCHITECTURE DESIGN

Original: The goal of data augmentation Shorten and Khoshgoftaar (2019) is increase the variety of the data distribution, such as geometrical transformation Kang et al.
Revised: The goal of data augmentation Shorten and Khoshgoftaar (2019) is increase the variety of the data distribution, such as geometrical transformation Kang et al.

Original: Taylor and Nitschke (2017), ﬂipping, style transfer Gatys et al.
Revised: Taylor and Nitschke (2017), ﬂipping, style transfer Gatys et al.

Original: adversarial robustness Madry et al.
Revised: adversarial robustness Madry et al.

Original: On the other way round, an alternative kind of approaches is to integrate into the model corresponding modules that improve the robustness to some types of variations, such as Worrall et al.
Revised: On the other way round, an alternative kind of approaches is to integrate into the model corresponding modules that improve the robustness to some types of variations, such as Worrall et al.

Original: ; Marcos et al.
Revised: ; Marcos et al.

Original: However, these techniques can only make effect because they are included in the training data for neural network to memorize Zhang et al.
Revised: However, these techniques can only make effect because they are included in the training data for neural network to memorize Zhang et al.

Original: (2016); besides, the improvement is only limited to some speciﬁc types of variation considered.
Revised: (2016); besides, the improvement is only limited to some speciﬁc types of variation considered.

Original: As analyzed in Xie et al.
Revised: As analyzed in Xie et al.

Original: Krueger et al.
Revised: Krueger et al.

Original: (2020), the data augmentation trained with empirical risk minimization or robust optimization Ben-Tal et al.
Revised: (2020), the data augmentation trained with empirical risk minimization or robust optimization Ben-Tal et al.

Original: (2009) such as adversarial training Madry et al.
Revised: (2009) such as adversarial training Madry et al.

Original: ; Sagawa et al.
Revised: ; Sagawa et al.

Original: (2019) can only achieve robustness on interpolation (convex hull) rather than extrapolation of training environments.
Revised: (2019) can only achieve robustness on interpolation (convex hull) rather than extrapolation of training environments.

Original: 7.7. C OMPARISONS WITH EXISTING WORKS IN DOMAIN ADAPTATION
Revised: 7.7. C OMPARISONS WITH EXISTING WORKS IN DOMAIN ADAPTATION

Original: Apparently, the main difference lies in the problem setting that (i) the domain adaptation (DA) can access the input data of the target domain while ours cannot; and (ii) our methods need multiple training data while the DA only needs one source domain.
Revised: Apparently, the main difference lies in the problem setting that (i) the domain adaptation (DA) can access the input data of the target domain while ours cannot; and (ii) our methods need multiple training data while the DA only needs one source domain.

Original: For methodology, our LaCIM shares insights but different with DA.
Revised: For methodology, our LaCIM shares insights but different with DA.

Original: Speciﬁcally, both methods assume some types of invariance that relates the training domains to the target domain.
Revised: Speciﬁcally, both methods assume some types of invariance that relates the training domains to the target domain.

Original: For DA, one stream is to assume the same conditional distribution shared between the source and the target domain, such as covariate shift Huang et al.
Revised: For DA, one stream is to assume the same conditional distribution shared between the source and the target domain, such as covariate shift Huang et al.

Original: (2007); Ben-David et al.
Revised: (2007); Ben-David et al.

Original: ; Johansson et al.
Revised: ; Johansson et al.

Original: ; Sugiyama et al.
Revised: ; Sugiyama et al.

Original: (2008) in which P ( Y | X ) are assumed to be the same across domains,
Revised: (2008) in which P ( Y | X ) are assumed to be the same across domains,

Original: concept shift Zhang et al.
Revised: concept shift Zhang et al.

Original: (2013) in which the P ( X | Y ) is assumed to be invariant.
Revised: (2013) in which the P ( X | Y ) is assumed to be invariant.

Original: Such an invariance is related to representation, such as Φ( X ) in Zhao et al.
Revised: Such an invariance is related to representation, such as Φ( X ) in Zhao et al.

Original: and P ( Y | Φ( X )) in Pan et al. (2010); Ganin et al. (2016); Magliacane et al.
Revised: and P ( Y | Φ( X )) in Pan et al. (2010); Ganin et al. (2016); Magliacane et al.

Original: However, these assumptions are only distribution-level rather than the underlying causation which takes the data-generating process into account.
Revised: However, these assumptions are only distribution-level rather than the underlying causation which takes the data-generating process into account.

Original: Taking the image classiﬁcation again as an example, our method ﬁrst propose a causal graph in which the latent factors are introduced as the explanatory/causal factors of the observed variables.
Revised: Taking the image classiﬁcation again as an example, our method ﬁrst propose a causal graph in which the latent factors are introduced as the explanatory/causal factors of the observed variables.

Original: These are supported by the framework of generative model Khemakhem, Kingma and Hyv¨arinen (2020); Khemakhem, Monti, Kingma and Hyv¨arinen (2020); Kingma and Welling (2014); Suter et al.
Revised: These are supported by the framework of generative model Khemakhem, Kingma and Hyv¨arinen (2020); Khemakhem, Monti, Kingma and Hyv¨arinen (2020); Kingma and Welling (2014); Suter et al.

Original: (2019) which has natural connection with the causal graph Sch¨olkopf (2019) that the edge in the causal graph reﬂects both the causal effect and also the generating process.
Revised: (2019) which has natural connection with the causal graph Sch¨olkopf (2019) that the edge in the causal graph reﬂects both the causal effect and also the generating process.

Original: Until now, perhaps the most similar work to us are Romeijn and Williamson (2018) and Teshima et al.
Revised: Until now, perhaps the most similar work to us are Romeijn and Williamson (2018) and Teshima et al.

Original: (2020) which also need multiple training domains and get access to a few samples in the target domain.
Revised: (2020) which also need multiple training domains and get access to a few samples in the target domain.

Original: Both work assumes the similar causal graph with us but unlike our
Revised: Both work assumes the similar causal graph with us but unlike our

Original: LaCIM, they do not separate the latent factors which can not explain the spurious correlation learned by supervised learning Ilse et al.
Revised: LaCIM, they do not separate the latent factors which can not explain the spurious correlation learned by supervised learning Ilse et al.

Original: Besides, the multiple training datasets in Romeijn and
Revised: Besides, the multiple training datasets in Romeijn and

Original: Williamson (2018) refer to intervened data which may hard to obtain in some applications.
Revised: Williamson (2018) refer to intervened data which may hard to obtain in some applications.

Original: We have veriﬁed in our experiments that explicitly disentangle the latent variables into two parts can result in better OOD prediction power than mixing them together.
Revised: We have veriﬁed in our experiments that explicitly disentangle the latent variables into two parts can result in better OOD prediction power than mixing them together.

Original: 7.7. C OMPARISONS WITH DOMAIN GENERALIZATION
Revised: 7.7. C OMPARISONS WITH DOMAIN GENERALIZATION

Original: For domain generalization (DG), similar to the invariance assumption in DA, a series of work proposed to align the representation Φ( X ) that assumed to be invariant across domains Li et al.
Revised: For domain generalization (DG), similar to the invariance assumption in DA, a series of work proposed to align the representation Φ( X ) that assumed to be invariant across domains Li et al.

Original: ; Muandet et al.
Revised: ; Muandet et al.

Original: As discussed above, these methods lack the deep delving of the underlying causal structure and precludes the variations of unseen domains.
Revised: As discussed above, these methods lack the deep delving of the underlying causal structure and precludes the variations of unseen domains.

Original: Recently, a series of works leverage causal invariance to enable OOD generalization on unseen domains, such as Ilse et al.
Revised: Recently, a series of works leverage causal invariance to enable OOD generalization on unseen domains, such as Ilse et al.

Original: (2019) which learns the representation that is domain-invariant.
Revised: (2019) which learns the representation that is domain-invariant.

Original: Notably, the Invariant Causal Prediction Peters et al.
Revised: Notably, the Invariant Causal Prediction Peters et al.

Original: (2016) formulates the assumption in the deﬁnition of
Revised: (2016) formulates the assumption in the deﬁnition of

Original: Structural Causal Model and assumes that Y = X S β (cid:63) S + ε Y where ε Y satisﬁes Gaussian distribution and S denotes the subset of covariates of X .
Revised: Structural Causal Model and assumes that Y = X S β (cid:63) S + ε Y where ε Y satisﬁes Gaussian distribution and S denotes the subset of covariates of X .

Original: The Rojas-Carulla et al.
Revised: The Rojas-Carulla et al.

Original: ; B ¨ uhlmann (2018) relaxes such an assumption by assuming the invariance of f y and noise distribution ε y in Y ← f y ( X S , ε y ) which induces P ( Y | X S ) .
Revised: ; B ¨ uhlmann (2018) relaxes such an assumption by assuming the invariance of f y and noise distribution ε y in Y ← f y ( X S , ε y ) which induces P ( Y | X S ) .

Original: The similar assumption is also adopted in Kuang et al.
Revised: The similar assumption is also adopted in Kuang et al.

Original: However, these works causally related the output to the observed input, which may not hold in many real applications in which the observed data is sensory-level, such as audio waves and pixels.
Revised: However, these works causally related the output to the observed input, which may not hold in many real applications in which the observed data is sensory-level, such as audio waves and pixels.

Original: It has been discussed in Bengio et al.
Revised: It has been discussed in Bengio et al.

Original: ; Bengio (2017) that the causal factors should be high-level abstractions/concepts.
Revised: ; Bengio (2017) that the causal factors should be high-level abstractions/concepts.

Original: The Heinze-Deml and Meinshausen (2017) considers the style transfer setting in which each image is linear combination of shape-related variable and contextual-related variable, which respectively correspond to S and Z in our LaCIM in which the nonlinear mechanism (rather than linear combination in Heinze-Deml and Meinshausen (2017)) is allowed.
Revised: The Heinze-Deml and Meinshausen (2017) considers the style transfer setting in which each image is linear combination of shape-related variable and contextual-related variable, which respectively correspond to S and Z in our LaCIM in which the nonlinear mechanism (rather than linear combination in Heinze-Deml and Meinshausen (2017)) is allowed.

Original: Besides, during testing, our method can generalize to the OOD sample with intervention such as adversarial noise and contextual intervention.
Revised: Besides, during testing, our method can generalize to the OOD sample with intervention such as adversarial noise and contextual intervention.

Original: Recently, the most notable work is Invariant Risk Minimization Arjovsky et al.
Revised: Recently, the most notable work is Invariant Risk Minimization Arjovsky et al.

Original: (2019), which will be discussed in detail in the subsequent section.
Revised: (2019), which will be discussed in detail in the subsequent section.

Original: 7.7. C OMPARISONS WITH I NVARIANT R ISK M INIMIZATION A RJOVSKY ET AL . (2019) AND REFERENCES THERE IN
Revised: 7.7. C OMPARISONS WITH I NVARIANT R ISK M INIMIZATION A RJOVSKY ET AL . (2019) AND REFERENCES THERE IN

Original: The Invariant Risk Minimization (IRM) Arjovsky et al.
Revised: The Invariant Risk Minimization (IRM) Arjovsky et al.

Original: (2019) assumes the existence of invariant representation Φ( X ) that induces the optimal classiﬁer for all domains,
Revised: (2019) assumes the existence of invariant representation Φ( X ) that induces the optimal classiﬁer for all domains,

Original: i.e. , the E
Revised: i.e. , the E

Original: [ Y | Pa ( Y )] is domain-independent in the formulation of SCM.
Revised: [ Y | Pa ( Y )] is domain-independent in the formulation of SCM.

Original: Similar to our LaCIM, the Pa ( Y ) can refer to latent variables.
Revised: Similar to our LaCIM, the Pa ( Y ) can refer to latent variables.

Original: Besides, to identify the invariance and the optimal classiﬁer, the training environments also need to be diverse enough.
Revised: Besides, to identify the invariance and the optimal classiﬁer, the training environments also need to be diverse enough.

Original: As aforementioned, this assumption is almost necessary to differentiate the invariance mechanism from the variant ones.
Revised: As aforementioned, this assumption is almost necessary to differentiate the invariance mechanism from the variant ones.

Original: To learn such an invariance, a regularization function is proposed.
Revised: To learn such an invariance, a regularization function is proposed.

Original: The difference of our LaCIM with IRM lies in two aspects: the direction of causal relation and the methodology.
Revised: The difference of our LaCIM with IRM lies in two aspects: the direction of causal relation and the methodology.

Original: For the direction, as aforementioned in section 7.7.1, the IRM assumes X → S rather than the S, Z → X in our LaCIM.
Revised: For the direction, as aforementioned in section 7.7.1, the IRM assumes X → S rather than the S, Z → X in our LaCIM.

Original: This is because the IRM deﬁnes Y as label with noise while ours deﬁnie the Y as the ground-truth label hence should be generated by the ground-truth hidden components that generating S .
Revised: This is because the IRM deﬁnes Y as label with noise while ours deﬁnie the Y as the ground-truth label hence should be generated by the ground-truth hidden components that generating S .

Original: Such an inconsistency can be reﬂected by experiment regarding to the CMNIST in which the number is the causal factors of the label Y , rather than only invariant correlation.
Revised: Such an inconsistency can be reﬂected by experiment regarding to the CMNIST in which the number is the causal factors of the label Y , rather than only invariant correlation.

Original: Besides, in terms of methodology, the theoretical claim of IRM only holds in linear case; in contrast, the CIMe f x , f y are allowed to be nonlinear.
Revised: Besides, in terms of methodology, the theoretical claim of IRM only holds in linear case; in contrast, the CIMe f x , f y are allowed to be nonlinear.

Original: Some other works share the similar spirit with or based on IRM.
Revised: Some other works share the similar spirit with or based on IRM.

Original: The Risk-Extrapolation (REx)
Revised: The Risk-Extrapolation (REx)

Original: Krueger et al.
Revised: Krueger et al.

Original: (2020) proposed to enforce the similar behavior of m classiﬁers with variance of which proposed as the regularization function.
Revised: (2020) proposed to enforce the similar behavior of m classiﬁers with variance of which proposed as the regularization function.

Original: The work in Xie et al.
Revised: The work in Xie et al.

Original: (2020) proposed a Quasidistribution framework that can incorporate empirical risk minimization, robust optimization and
Revised: (2020) proposed a Quasidistribution framework that can incorporate empirical risk minimization, robust optimization and

Original: REx.
Revised: REx.

Original: It can be concluded that the robust optimization only generalizes the convex hull of training environments (deﬁned as interpolation) and the REx can generalize extrapolated combinations of training environments.
Revised: It can be concluded that the robust optimization only generalizes the convex hull of training environments (deﬁned as interpolation) and the REx can generalize extrapolated combinations of training environments.

Original: This work lacks model of underlying causal structure, although it performs similarly to IRM experimentally.
Revised: This work lacks model of underlying causal structure, although it performs similarly to IRM experimentally.

Original: Besides, the Teney et al.
Revised: Besides, the Teney et al.

Original: (2020) proposed to unpool the training data into several domains with different environment and leverages Arjovsky et al.
Revised: (2020) proposed to unpool the training data into several domains with different environment and leverages Arjovsky et al.

Original: (2019) to learn invariant information for classiﬁer.
Revised: (2019) to learn invariant information for classiﬁer.

Original: Recently, the Bellot and van der Schaar (2020) also assumes the invariance to be generating mechanisms and can generalize the capability of IRM when unobserved confounder exist.
Revised: Recently, the Bellot and van der Schaar (2020) also assumes the invariance to be generating mechanisms and can generalize the capability of IRM when unobserved confounder exist.

Original: However, this work also lacks the analysis of identiﬁability result.
Revised: However, this work also lacks the analysis of identiﬁability result.

Original: We ﬁnish this section with the following summary of methods in section 7.7.4 and the IRM, in terms of causal factor, invariance type, direction of causal relation, theoretical judgement and the ability to generalize to intervened data
Revised: We ﬁnish this section with the following summary of methods in section 7.7.4 and the IRM, in terms of causal factor, invariance type, direction of causal relation, theoretical judgement and the ability to generalize to intervened data

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 7. I MPLEMENTATION D ETAILS AND M ORE R ESULTS FOR S IMULATION
Revised: 7. I MPLEMENTATION D ETAILS AND M ORE R ESULTS FOR S IMULATION

Original: Data Generation We set m = 5 , n e = 1000 for each e .
Revised: Data Generation We set m = 5 , n e = 1000 for each e .

Original: The generating process of d s ∈ R q ds ,
Revised: The generating process of d s ∈ R q ds ,

Original: Z ∈ R q z , S ∈ R q s ,
Revised: Z ∈ R q z , S ∈ R q s ,

Original: X ∈ R q x and Y ∈ R q y is introduced in the supplement 7.8.
Revised: X ∈ R q x and Y ∈ R q y is introduced in the supplement 7.8.

Original: We set q d s = q s = q z = q y = 2 and q x = 4 .
Revised: We set q d s = q s = q z = q y = 2 and q x = 4 .

Original: For each environment e ∈ [ m ] with m = 5 , we generate 1000 samples
Revised: For each environment e ∈ [ m ] with m = 5 , we generate 1000 samples

Original: D e = { x i , y i } i.i.d ∼ (cid:82) p f x ( x | s, z ) p f y ( y | s )
Revised: D e = { x i , y i } i.i.d ∼ (cid:82) p f x ( x | s, z ) p f y ( y | s )

Original: p e ( s, z | d es ) dsdz .
Revised: p e ( s, z | d es ) dsdz .

Original: The d es = (cid:0) N (0 , I q ds × q ds ) + 5 ∗ e (cid:1) ∗ 2 ; the s, z | d es ∼ N (cid:16) µ φ (cid:63)s,z ( s, z | d es ) , σ 2 φ (cid:63)s,z ( s, z | d es ) (cid:17) with µ φ (cid:63)s,z =
Revised: The d es = (cid:0) N (0 , I q ds × q ds ) + 5 ∗ e (cid:1) ∗ 2 ; the s, z | d es ∼ N (cid:16) µ φ (cid:63)s,z ( s, z | d es ) , σ 2 φ (cid:63)s,z ( s, z | d es ) (cid:17) with µ φ (cid:63)s,z =

Original: A µs,z ∗ d es and log σ φ (cid:63)s,z
Revised: A µs,z ∗ d es and log σ φ (cid:63)s,z

Original: = A σs,z ∗ d es ( A µs,z , A σs,z are random matrices); the x | s, z ∼ N (cid:16) µ φ (cid:63)x ( x | s, z ) , σ 2 φ (cid:63)x ( x | s, z ) (cid:17) with µ φ (cid:63)s,z = h
Revised: = A σs,z ∗ d es ( A µs,z , A σs,z are random matrices); the x | s, z ∼ N (cid:16) µ φ (cid:63)x ( x | s, z ) , σ 2 φ (cid:63)x ( x | s, z ) (cid:17) with µ φ (cid:63)s,z = h

Original: ( A µ, 3 x ∗ h ( A µ, 2 x ∗ h ( A µ, 2 x ∗ [ s (cid:62) , z (cid:62) ] (cid:62) ])))
Revised: ( A µ, 3 x ∗ h ( A µ, 2 x ∗ h ( A µ, 2 x ∗ [ s (cid:62) , z (cid:62) ] (cid:62) ])))

Original: and log σ φ (cid:63)s,z
Revised: and log σ φ (cid:63)s,z

Original: = h ( A σ, 3 x ∗ h (
Revised: = h ( A σ, 3 x ∗ h (

Original: A σ, 2 x ∗ h ( A σ, 2 x ∗ [ s (cid:62) , z (cid:62) ] (cid:62) ])))
Revised: A σ, 2 x ∗ h ( A σ, 2 x ∗ [ s (cid:62) , z (cid:62) ] (cid:62) ])))

Original: ( h is
Revised: ( h is

Original: LeakyReLU activation function with slope = 0 .
Revised: LeakyReLU activation function with slope = 0 .

Original: 5 and A µ,i =1 , 2 , 3 x , A σ,i =1 , 2 , 3 x are random matrices); the y | s is similarly to x | s, z with A µ,i =1 , 2 , 3 x , A σ,i =1 , 2 , 3 x respectively replaced by A µ,i =1 , 2 , 3 y , A σ,i =1 , 2 , 3 y .
Revised: 5 and A µ,i =1 , 2 , 3 x , A σ,i =1 , 2 , 3 x are random matrices); the y | s is similarly to x | s, z with A µ,i =1 , 2 , 3 x , A σ,i =1 , 2 , 3 x respectively replaced by A µ,i =1 , 2 , 3 y , A σ,i =1 , 2 , 3 y .

Original: Implementation Details We parameterize p θ ( s, z | d ) , q φ ( s, z | x, y, d ) , p θ ( x | s, z ) and p θ ( y | s ) as 3layer MLP with the LeakyReLU activation function.
Revised: Implementation Details We parameterize p θ ( s, z | d ) , q φ ( s, z | x, y, d ) , p θ ( x | s, z ) and p θ ( y | s ) as 3layer MLP with the LeakyReLU activation function.

Original: The Adam with learning rate 5 × 10 − 4 is implemented for optimization.
Revised: The Adam with learning rate 5 × 10 − 4 is implemented for optimization.

Original: We set the batch size as 512 and run for 2,000 iterations in each trial.
Revised: We set the batch size as 512 and run for 2,000 iterations in each trial.

Original: Visualization.
Revised: Visualization.

Original: As shown from the visualization of S is shown in Fig.
Revised: As shown from the visualization of S is shown in Fig.

Original: 7.8, our LaCIM can identify the causal factor S .
Revised: 7.8, our LaCIM can identify the causal factor S .

Original: [Figure]
Revised: [Figure]

Original: The setting when C can take a value in a sample-level.
Revised: The setting when C can take a value in a sample-level.

Original: We consider the generation process of D e as D e = { x i , y i } i.i.d ∼ (cid:82) p f x ( x | s, z ) p f y
Revised: We consider the generation process of D e as D e = { x i , y i } i.i.d ∼ (cid:82) p f x ( x | s, z ) p f y

Original: ( y | s ) p ( s, z | c ) p ( c | d e ) dsdzdc , with q c := 2 .
Revised: ( y | s ) p ( s, z | c ) p ( c | d e ) dsdzdc , with q c := 2 .

Original: The generation is the same except that the after obtaining d s , we additionally generate c with c :=
Revised: The generation is the same except that the after obtaining d s , we additionally generate c with c :=

Original: N ( d s , I ) .
Revised: N ( d s , I ) .

Original: The results are summarized in Tab. 7.8.
Revised: The results are summarized in Tab. 7.8.

Original: [Table]
Revised: [Table]

Original: I MPLEMENTATION D ETAILS FOR O PTIMIZATION OVER S, Z
Revised: I MPLEMENTATION D ETAILS FOR O PTIMIZATION OVER S, Z

Original: Recall that we ﬁrst optimize s ∗ , z ∗ according to
Revised: Recall that we ﬁrst optimize s ∗ , z ∗ according to

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst sample some initial points from each posterior distribution q eψ ( s | x ) and then optimize foriterations.
Revised: We ﬁrst sample some initial points from each posterior distribution q eψ ( s | x ) and then optimize foriterations.

Original: We using Adam as optimizer, with learning rate as 0.002 and weight decay 0.0002.
Revised: We using Adam as optimizer, with learning rate as 0.002 and weight decay 0.0002.

Original: The
Revised: The

Original: Fig.
Revised: Fig.

Original: 7.9 shows the optimization effect of one run in CMNIST.
Revised: 7.9 shows the optimization effect of one run in CMNIST.

Original: As shown, the test accuracy keeps growing as iterates.
Revised: As shown, the test accuracy keeps growing as iterates.

Original: For time saving, we chose to optimize for 50 iterations.
Revised: For time saving, we chose to optimize for 50 iterations.

Original: As shown, the test accuracy increases as iterates.
Revised: As shown, the test accuracy increases as iterates.

Original: 7. I MPLEMENTATIONS F OR B ASELINE
Revised: 7. I MPLEMENTATIONS F OR B ASELINE

Original: For the CE X → Y and the CE X, d s → Y , they both composed of two parts: (i) feature extractor, followed by (ii) classiﬁer.
Revised: For the CE X → Y and the CE X, d s → Y , they both composed of two parts: (i) feature extractor, followed by (ii) classiﬁer.

Original: The network structure of the feature extractor for CE X → Y is the same with that of our encoder; while the extracted features for CE X, d
Revised: The network structure of the feature extractor for CE X → Y is the same with that of our encoder; while the extracted features for CE X, d

Original: → Y is the concatenation of the features encoded from X → S, Z via the network with the same network structure of our encoder; and the network with the same structure of our prior network for LaCIM- d .
Revised: → Y is the concatenation of the features encoded from X → S, Z via the network with the same network structure of our encoder; and the network with the same structure of our prior network for LaCIM- d .

Original: The network structures of the classiﬁer for both methods are the same to that of our p φ ( y | s ) .
Revised: The network structures of the classiﬁer for both methods are the same to that of our p φ ( y | s ) .

Original: The IRM and SDA adopt the same structure as CE X → Y .
Revised: The IRM and SDA adopt the same structure as CE X → Y .

Original: DANN adopt the same structure of CE X → Y and a additional domain classiﬁer which is the same as that of p φ ( y | s ) .
Revised: DANN adopt the same structure of CE X → Y and a additional domain classiﬁer which is the same as that of p φ ( y | s ) .

Original: sVAE adopt the same structure as LaCIM- d s with the exception that the p φ ( y | s ) is replaced by p φ ( y | z, s ) .
Revised: sVAE adopt the same structure as LaCIM- d s with the exception that the p φ ( y | s ) is replaced by p φ ( y | z, s ) .

Original: MMD-AAE adopt the same structure of encoder, decoder and classiﬁer as LaCIM- d and a additional 2-layer MLP with channel 256-256- dim z is used to extract latent z .
Revised: MMD-AAE adopt the same structure of encoder, decoder and classiﬁer as LaCIM- d and a additional 2-layer MLP with channel 256-256- dim z is used to extract latent z .

Original: 7. S UPPLEMENTARY FOR C OLORED MNIST Implementation details
Revised: 7. S UPPLEMENTARY FOR C OLORED MNIST Implementation details

Original: The network structure for inference model is composed of two parts, with the ﬁrst part shared among all environments and multiple branches corresponding to each environment for the second part.
Revised: The network structure for inference model is composed of two parts, with the ﬁrst part shared among all environments and multiple branches corresponding to each environment for the second part.

Original: The network structure of the ﬁrst-part encoder is composed of four blocks, each block is the sequential of Convolutional Layer (Conv), Batch Normalization (BN), ReLU and maxpooling with stride 2.
Revised: The network structure of the ﬁrst-part encoder is composed of four blocks, each block is the sequential of Convolutional Layer (Conv), Batch Normalization (BN), ReLU and maxpooling with stride 2.

Original: The output number of feature map is accordingly 32, 64, 128, 256.
Revised: The output number of feature map is accordingly 32, 64, 128, 256.

Original: The second part network structure that output the mean and log-variance of S, Z is Conv-bn-ReLU(256) →
Revised: The second part network structure that output the mean and log-variance of S, Z is Conv-bn-ReLU(256) →

Original: Adaptive (1) → FC(256, 256) → ReLU → FC(256, q t = s,z ) with FC stands for fully-connected layer.
Revised: Adaptive (1) → FC(256, 256) → ReLU → FC(256, q t = s,z ) with FC stands for fully-connected layer.

Original: The structure of ρ t = s,z in Eq.
Revised: The structure of ρ t = s,z in Eq.

Original: (72) is FC( q t , 256)
Revised: (72) is FC( q t , 256)

Original: → ReLU → FC(256, q t ).
Revised: → ReLU → FC(256, q t ).

Original: The network structure for generative model p φ ( x | s, z ) is the sequential of three modules: (i) Upsampling with stride 2; (ii) four blocks of Transpose-Convolution (TConv), BN and ReLU with respective output dimension being 128, 64, 32, 16; (iii) Conv-BN-ReLU-Sigmoid with number of channels in the output as 3, followed by cropping step in order to make the image with the same size as input dimension, i.e. , 3 × 28 × 28 .
Revised: The network structure for generative model p φ ( x | s, z ) is the sequential of three modules: (i) Upsampling with stride 2; (ii) four blocks of Transpose-Convolution (TConv), BN and ReLU with respective output dimension being 128, 64, 32, 16; (iii) Conv-BN-ReLU-Sigmoid with number of channels in the output as 3, followed by cropping step in order to make the image with the same size as input dimension, i.e. , 3 × 28 × 28 .

Original: The network structure for generative model p φ ( y | s ) is commposed of FC (512) → BN → ReLU →
Revised: The network structure for generative model p φ ( y | s ) is commposed of FC (512) → BN → ReLU →

Original: FC (256) → BN → ReLU → FC ( |Y| ).
Revised: FC (256) → BN → ReLU → FC ( |Y| ).

Original: The q t = s,z is set to 32.
Revised: The q t = s,z is set to 32.

Original: We implement SGD as optimizer with learning rate 0.5, weight decay 1 e − 5 and we set batch size as 256.
Revised: We implement SGD as optimizer with learning rate 0.5, weight decay 1 e − 5 and we set batch size as 256.

Original: The total training epoch is 80.
Revised: The total training epoch is 80.

Original: We ﬁrst explain why we do not ﬂip y with 25% in the manuscript, and then provide further exploration of our method for the setting with ﬂipping y .
Revised: We ﬁrst explain why we do not ﬂip y with 25% in the manuscript, and then provide further exploration of our method for the setting with ﬂipping y .

Original: Invariant Causation v.s. Invariant Correlation by Flipping y in Arjovsky et al.
Revised: Invariant Causation v.s. Invariant Correlation by Flipping y in Arjovsky et al.

Original: The y is further ﬂipped with 25% to obtain the ﬁnal label in IRM setting and this step is omitted in ours.
Revised: The y is further ﬂipped with 25% to obtain the ﬁnal label in IRM setting and this step is omitted in ours.

Original: The difference lies in the deﬁnition of invariance.
Revised: The difference lies in the deﬁnition of invariance.

Original: Our LaCIM deﬁnes invariance as the causal relation between S and the label Y , while the one in IRM can be correlation.
Revised: Our LaCIM deﬁnes invariance as the causal relation between S and the label Y , while the one in IRM can be correlation.

Original: As illustrated in Handwritting
Revised: As illustrated in Handwritting

Original: Sample Form in Fig.
Revised: Sample Form in Fig.

Original: 7.11 in Grother (1995), the generting direction should be Y → X .
Revised: 7.11 in Grother (1995), the generting direction should be Y → X .

Original: If we denote the variable by ﬂipping Y as ˜ Y ( a.k.a , the ﬁnal label in IRM), then the causal graph should be
Revised: If we denote the variable by ﬂipping Y as ˜ Y ( a.k.a , the ﬁnal label in IRM), then the causal graph should be

Original: X ← Y
Revised: X ← Y

Original: → ˜ Y .
Revised: → ˜ Y .

Original: In this case, the ˜ Y is correlated rather than causally related to the digit X .
Revised: In this case, the ˜ Y is correlated rather than causally related to the digit X .

Original: For our
Revised: For our

Original: LaCIM, we deﬁne the label as interpretable human label (which can approximate to y for any image x ) and represented by Y in our experiments.
Revised: LaCIM, we deﬁne the label as interpretable human label (which can approximate to y for any image x ) and represented by Y in our experiments.

Original: The reason why we do not deﬁne the Y as ground-truth label is that (i) the prediction is only based on the extracted components of image which may be determined not only by the ground-truth label; (ii) the learning of ground-truth is interpretable that relevant to human.
Revised: The reason why we do not deﬁne the Y as ground-truth label is that (i) the prediction is only based on the extracted components of image which may be determined not only by the ground-truth label; (ii) the learning of ground-truth is interpretable that relevant to human.

Original: For example, if a writer is provided with digit “ 2 ” but he wrote it mistakenly as “ 4 ”, then it is more interpretable that we can predict the digit as “ 4 ” rather than “ 2 ”.
Revised: For example, if a writer is provided with digit “ 2 ” but he wrote it mistakenly as “ 4 ”, then it is more interpretable that we can predict the digit as “ 4 ” rather than “ 2 ”.

Original: For the digit with ambiguous label from the perspective of image, even if we predict it mistakenly, it is also interpretable in terms of prediction given the information of only digit.
Revised: For the digit with ambiguous label from the perspective of image, even if we predict it mistakenly, it is also interpretable in terms of prediction given the information of only digit.

Original: Returning back to the IRM setting, the label is ﬂipping without reference to the semantic shape of digit.
Revised: Returning back to the IRM setting, the label is ﬂipping without reference to the semantic shape of digit.

Original: Therefore, the ﬂipping may happen to noiseless digits rather than noisy and unsure ones, making the shape of number less semantically related to the label.
Revised: Therefore, the ﬂipping may happen to noiseless digits rather than noisy and unsure ones, making the shape of number less semantically related to the label.

Original: Experiment with IRM setting We further conduct the experiment on IRM setting, with the ﬁnal label y deﬁned by ﬂipping original label with 25% , and further color p e proportions of digits with corresponding color-label mapping.
Revised: Experiment with IRM setting We further conduct the experiment on IRM setting, with the ﬁnal label y deﬁned by ﬂipping original label with 25% , and further color p e proportions of digits with corresponding color-label mapping.

Original: If we assume the original ground-truth label to be the effect of the digit number of S , then the anti-causal relation with Z and Y can make the identiﬁability of S difﬁcult in this ﬂipping scenario.
Revised: If we assume the original ground-truth label to be the effect of the digit number of S , then the anti-causal relation with Z and Y can make the identiﬁability of S difﬁcult in this ﬂipping scenario.

Original: Note that the causal effect between S and Y is invariant across domains, therefore we adopt to regularize the branch of inferring S to be shared among inference models for multiple environments.
Revised: Note that the causal effect between S and Y is invariant across domains, therefore we adopt to regularize the branch of inferring S to be shared among inference models for multiple environments.

Original: Besides, we regularize the causal effect between S and Z to be shared among different environments via pairwise regularization.
Revised: Besides, we regularize the causal effect between S and Z to be shared among different environments via pairwise regularization.

Original: The combined loss is formulated as:
Revised: The combined loss is formulated as:

Original: [Equation]
Revised: [Equation]

Original: with q eψ ( s, z | x ) in Eq.
Revised: with q eψ ( s, z | x ) in Eq.

Original: (72) factorized as q ψ ez ( z ) q ψ s ( s ) and ρ s shared among m environments.
Revised: (72) factorized as q ψ ez ( z ) q ψ s ( s ) and ρ s shared among m environments.

Original: The appended loss is coincide with recent study Risk-Extropolation (REx) in Krueger et al.
Revised: The appended loss is coincide with recent study Risk-Extropolation (REx) in Krueger et al.

Original: (2020), with the difference of separating y -causative factors S from others.
Revised: (2020), with the difference of separating y -causative factors S from others.

Original: We name such a training method as
Revised: We name such a training method as

Original: LaCIM-REx.
Revised: LaCIM-REx.

Original: For implementation details, in addition to shared encoder regarding S , we set learning rate as 0.1, weight decay as 0.0002, batch size as 256.
Revised: For implementation details, in addition to shared encoder regarding S , we set learning rate as 0.1, weight decay as 0.0002, batch size as 256.

Original: we have that p ( y | x ) =
Revised: we have that p ( y | x ) =

Original: (cid:82) S q ψ s ( s | x ) p φ ( y |
Revised: (cid:82) S q ψ s ( s | x ) p φ ( y |

Original: ρ s ( s ))
Revised: ρ s ( s ))

Original: for any x .
Revised: for any x .

Original: We consider two settings: setting#1 with m 2 and p e 1 = 0 .
Revised: We consider two settings: setting#1 with m 2 and p e 1 = 0 .

Original: 9 , p e 2 = 0 . 8 ; and setting#2 with m = 4 with p e 1 = 0 .
Revised: 9 , p e 2 = 0 . 8 ; and setting#2 with m = 4 with p e 1 = 0 .

Original: 9 , p e 2 = 0 .
Revised: 9 , p e 2 = 0 .

Original: 8 , p e 3 = 0 .
Revised: 8 , p e 3 = 0 .

Original: 7 , p e 4 = 0 .
Revised: 7 , p e 4 = 0 .

Original: We only report the number of IRM since the cross entropy performs poorly in both settings.
Revised: We only report the number of IRM since the cross entropy performs poorly in both settings.

Original: As shown, our model performs comparably with
Revised: As shown, our model performs comparably with

Original: LaCIM- d s and better than IRM Arjovsky et al.
Revised: LaCIM- d s and better than IRM Arjovsky et al.

Original: due to separation of S znd Z .
Revised: due to separation of S znd Z .

Original: [Table]
Revised: [Table]

Original: 7. S UPPLEMENTARY FOR NICO
Revised: 7. S UPPLEMENTARY FOR NICO

Original: Implementation Details Due to size difference among images, we resize each image into 256 × 256.
Revised: Implementation Details Due to size difference among images, we resize each image into 256 × 256.

Original: The network structure of p θ ( z, s | d s ) , q φ ( z, s | x, d s ) , p θ ( x | z, s ) , p θ ( y | s ) for cat/dog classiﬁcation is the same with the one implemented in early prediction of Alzheimer’s Disease with exception of 3D convolution/Deconvolution replaced by 2D ones.
Revised: The network structure of p θ ( z, s | d s ) , q φ ( z, s | x, d s ) , p θ ( x | z, s ) , p θ ( y | s ) for cat/dog classiﬁcation is the same with the one implemented in early prediction of Alzheimer’s Disease with exception of 3D convolution/Deconvolution replaced by 2D ones.

Original: For each model, we train for 200 epochs using sgd, with learning rate (lr) set to 0.01, and after every 60 epochs the learning rate is multiplied by lr decay parameter that is set to 0.2.
Revised: For each model, we train for 200 epochs using sgd, with learning rate (lr) set to 0.01, and after every 60 epochs the learning rate is multiplied by lr decay parameter that is set to 0.2.

Original: The weight decay coefﬁcients parameter is set to 5 × 10 − 4 .
Revised: The weight decay coefﬁcients parameter is set to 5 × 10 − 4 .

Original: The
Revised: The

Original: [Table] [Table]
Revised: [Table] [Table]

Original: batch size is set to 20.
Revised: batch size is set to 20.

Original: The training environments which is characterized by c can be referenced in
Revised: The training environments which is characterized by c can be referenced in

Original: Table 7.12.
Revised: Table 7.12.

Original: For visualization, we implemented the gradient-based method Simonyan et al.
Revised: For visualization, we implemented the gradient-based method Simonyan et al.

Original: (2013) to visualize the neuron (in fully connected layer for both CE x
Revised: (2013) to visualize the neuron (in fully connected layer for both CE x

Original: → y and CE ( x, d s ) → y ; in s layer for
Revised: → y and CE ( x, d s ) → y ; in s layer for

Original: LaCIM- d s ) that is most correlated to label y .
Revised: LaCIM- d s ) that is most correlated to label y .

Original: The d s for m environments We summarize the d s of m = 8 and m = 14 environments in Table 7.12.
Revised: The d s for m environments We summarize the d s of m = 8 and m = 14 environments in Table 7.12.

Original: As shown, the value of d s in the test domain is the extrapolation of the training environments, i.e. , the d test s is not included in the convex hull of { d e i } 14 i =1 .
Revised: As shown, the value of d s in the test domain is the extrapolation of the training environments, i.e. , the d test s is not included in the convex hull of { d e i } 14 i =1 .

Original: More Visualization Results Fig.
Revised: More Visualization Results Fig.

Original: 7 shows more visualization results.
Revised: 7 shows more visualization results.

Original: Results on Intervened Data.
Revised: Results on Intervened Data.

Original: We test our model and the baseline on intervened data, in which each image is generated by intervention on Z , i.e., taking a speciﬁc value of Z .
Revised: We test our model and the baseline on intervened data, in which each image is generated by intervention on Z , i.e., taking a speciﬁc value of Z .

Original: This intervention breaks the correlation between S and Z , thus the distribution of which can be regarded as a speciﬁc type of
Revised: This intervention breaks the correlation between S and Z , thus the distribution of which can be regarded as a speciﬁc type of

Original: OOD.
Revised: OOD.

Original: Speciﬁcally, we replace the scene of an image with the scene from the another image, as shown in Fig.
Revised: Speciﬁcally, we replace the scene of an image with the scene from the another image, as shown in Fig.

Original: We generate 120 images, including 30 images of types: cat on grass, dog on grass, cat on snow, and dog on grass.
Revised: We generate 120 images, including 30 images of types: cat on grass, dog on grass, cat on snow, and dog on grass.

Original: We evaluate LaCIM- d , CE X → Y , IRM, DANN, NCBB, MMD-AAE, and
Revised: We evaluate LaCIM- d , CE X → Y , IRM, DANN, NCBB, MMD-AAE, and

Original: DIVA methods on this intervened dataset.
Revised: DIVA methods on this intervened dataset.

Original: As shown in Tab 9, our LaCIM- d can performs the best among all methods, which validate the robustness of our LaCIM.
Revised: As shown in Tab 9, our LaCIM- d can performs the best among all methods, which validate the robustness of our LaCIM.

Original: 7. D ISEASE P REDICTION OF A LZHEIMER ’ S D ISEASE
Revised: 7. D ISEASE P REDICTION OF A LZHEIMER ’ S D ISEASE

Original: Dataset Description.
Revised: Dataset Description.

Original: The dataset contains in total 317 samples with 48 AD, 75 NC, and 194 MCI.
Revised: The dataset contains in total 317 samples with 48 AD, 75 NC, and 194 MCI.

Original: Denotation of Attributes d s .
Revised: Denotation of Attributes d s .

Original: The C ∈ R 9 includes personal attributes ( e.g. , age Guerreiro and Bras (2015), gender Vina and Lloret (2010) and education years Mortimer (1997) that play as potential
Revised: The C ∈ R 9 includes personal attributes ( e.g. , age Guerreiro and Bras (2015), gender Vina and Lloret (2010) and education years Mortimer (1997) that play as potential

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: risks of AD), gene ( ε 4 allele), and biomarkers ( e.g. , changes of CSF, TAU, PTAU, amyloid β , cortical amyloid deposition (AV45) Humpel and Hochstrasser (2011)).
Revised: risks of AD), gene ( ε 4 allele), and biomarkers ( e.g. , changes of CSF, TAU, PTAU, amyloid β , cortical amyloid deposition (AV45) Humpel and Hochstrasser (2011)).

Original: Implementation Details For LaCIM- d s , we parameterize inference model q ψ ( s, z
Revised: Implementation Details For LaCIM- d s , we parameterize inference model q ψ ( s, z

Original: | x, d s ) , p φ ( s, z | d s ) , p φ ( x | z, s ) and p φ ( y | s ) and S, Z ∈ R 64 .
Revised: | x, d s ) , p φ ( s, z | d s ) , p φ ( x | z, s ) and p φ ( y | s ) and S, Z ∈ R 64 .