Original: E MPIRICAL OR I NVARIANT R ISK M INIMIZATION ? A S AMPLE C OMPLEXITY P ERSPECTIVE
Revised: E MPIRICAL OR I NVARIANT R ISK M INIMIZATION ? A S AMPLE C OMPLEXITY P ERSPECTIVE

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, invariant risk minimization (IRM) was proposed as a promising solution to address out-of-distribution (OOD) generalization.
Revised: Recently, invariant risk minimization (IRM) was proposed as a promising solution to address out-of-distribution (OOD) generalization.

Original: However, it is unclear when IRM should be preferred over the widely-employed empirical risk minimization (ERM) framework.
Revised: However, it is unclear when IRM should be preferred over the widely-employed empirical risk minimization (ERM) framework.

Original: In this work, we analyze both these frameworks from the perspective of sample complexity, thus taking a ﬁrm step towards answering this important question.
Revised: In this work, we analyze both these frameworks from the perspective of sample complexity, thus taking a ﬁrm step towards answering this important question.

Original: We ﬁnd that depending on the type of data generation mechanism, the two approaches might have very different ﬁnite sample and asymptotic behavior.
Revised: We ﬁnd that depending on the type of data generation mechanism, the two approaches might have very different ﬁnite sample and asymptotic behavior.

Original: For example, in the covariate shift setting we see that the two approaches not only arrive at the same asymptotic solution, but also have similar ﬁnite sample behavior with no clear winner.
Revised: For example, in the covariate shift setting we see that the two approaches not only arrive at the same asymptotic solution, but also have similar ﬁnite sample behavior with no clear winner.

Original: For other distribution shifts such as those involving confounders or anti-causal variables, however, the two approaches arrive at different asymptotic solutions where IRM is guaranteed to be close to the desired OOD solutions in the ﬁnite sample regime for polynomial generative models, while ERM is biased even asymptotically.
Revised: For other distribution shifts such as those involving confounders or anti-causal variables, however, the two approaches arrive at different asymptotic solutions where IRM is guaranteed to be close to the desired OOD solutions in the ﬁnite sample regime for polynomial generative models, while ERM is biased even asymptotically.

Original: We further investigate how different factors — the number of environments, complexity of the model, and IRM penalty weight — impact the sample complexity of IRM in relation to its distance from the OOD solutions.
Revised: We further investigate how different factors — the number of environments, complexity of the model, and IRM penalty weight — impact the sample complexity of IRM in relation to its distance from the OOD solutions.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A recent study shows that models trained to detect COVID-19 from chest radiographs rely on spurious factors such as the source of the data rather than the lung pathology (DeGrave et al., 2020).
Revised: A recent study shows that models trained to detect COVID-19 from chest radiographs rely on spurious factors such as the source of the data rather than the lung pathology (DeGrave et al., 2020).

Original: This is just one of many alarming examples of spurious correlations failing to hold outside a speciﬁc training distribution.
Revised: This is just one of many alarming examples of spurious correlations failing to hold outside a speciﬁc training distribution.

Original: In one commonly cited example, Beery et al.
Revised: In one commonly cited example, Beery et al.

Original: (2018) trained a convolutional neural network (CNN) to classify camels from cows.
Revised: (2018) trained a convolutional neural network (CNN) to classify camels from cows.

Original: In the training data, most pictures of the cows had green pastures, while most pictures of camels were in the desert.
Revised: In the training data, most pictures of the cows had green pastures, while most pictures of camels were in the desert.

Original: The CNN picked up the spurious correlation and associated green pastures with cows thus failing to classify cows on beaches.
Revised: The CNN picked up the spurious correlation and associated green pastures with cows thus failing to classify cows on beaches.

Original: Recently, Arjovsky et al.
Revised: Recently, Arjovsky et al.

Original: (2019) proposed a framework called invariant risk minimization (IRM) to address the problem of models inheriting spurious correlations.
Revised: (2019) proposed a framework called invariant risk minimization (IRM) to address the problem of models inheriting spurious correlations.

Original: They showed that when data is gathered from multiple environments, one can learn to exploit invariant causal relationships, rather than relying on varying spurious relationships, thus learning robust predictors.
Revised: They showed that when data is gathered from multiple environments, one can learn to exploit invariant causal relationships, rather than relying on varying spurious relationships, thus learning robust predictors.

Original: More recent work suggests that empirical risk minimization (ERM) is still state-of-the-art on many problems requiring
Revised: More recent work suggests that empirical risk minimization (ERM) is still state-of-the-art on many problems requiring

Original: OOD generalization (Gulrajani & Lopez-Paz, 2020).
Revised: OOD generalization (Gulrajani & Lopez-Paz, 2020).

Original: This gives rise to a fundamental question: when is IRM better than ERM (and vice versa)?
Revised: This gives rise to a fundamental question: when is IRM better than ERM (and vice versa)?

Original: In this work, we seek to answer this question through a systematic comparison of the sample complexity of the two approaches under different types of train and test distributional mismatches.
Revised: In this work, we seek to answer this question through a systematic comparison of the sample complexity of the two approaches under different types of train and test distributional mismatches.

Original: The distribution shifts (cid:0) P train ( X, Y ) (cid:54) =
Revised: The distribution shifts (cid:0) P train ( X, Y ) (cid:54) =

Original: P test ( X, Y ) (cid:1) that we consider informally stated satisfy an invariance condition – there exists a representation Φ ∗ of the covariates such that P train ( Y | Φ ∗ ( X ))
Revised: P test ( X, Y ) (cid:1) that we consider informally stated satisfy an invariance condition – there exists a representation Φ ∗ of the covariates such that P train ( Y | Φ ∗ ( X ))

Original: =
Revised: =

Original: P test ( Y | Φ ∗ ( X ))
Revised: P test ( Y | Φ ∗ ( X ))

Original: =
Revised: =

Original: P ( Y | Φ ∗ ( X )) .
Revised: P ( Y | Φ ∗ ( X )) .

Original: A special case of this occurs when Φ ∗ is identity – P train ( X ) (cid:54) =
Revised: A special case of this occurs when Φ ∗ is identity – P train ( X ) (cid:54) =

Original: P test ( X ) but P train ( Y | X ) =
Revised: P test ( X ) but P train ( Y | X ) =

Original: P test ( Y | X ) – such a shift is known as a covariate-shift (Gretton et al., 2009).
Revised: P test ( Y | X ) – such a shift is known as a covariate-shift (Gretton et al., 2009).

Original: In many other settings Φ ∗ may not be identity (denoted as I ), examples include settings with confounders or anti-causal variables (Pearl, 2009) where covariates appear spuriously correlated with the label and P train ( Y | X ) (cid:54) =
Revised: In many other settings Φ ∗ may not be identity (denoted as I ), examples include settings with confounders or anti-causal variables (Pearl, 2009) where covariates appear spuriously correlated with the label and P train ( Y | X ) (cid:54) =

Original: P test ( Y | X ) .
Revised: P test ( Y | X ) .

Original: We use causal Bayesian networks to illustrate these shifts in Figure 1.
Revised: We use causal Bayesian networks to illustrate these shifts in Figure 1.

Original: Suppose X e =
Revised: Suppose X e =

Original: [ X e 1 , X e 2 ] represents the image, where X e 1 is the shape of the animal and X e 2 is the background color, Y e is the label of the animal, and e is the index of the environment/domain.
Revised: [ X e 1 , X e 2 ] represents the image, where X e 1 is the shape of the animal and X e 2 is the background color, Y e is the label of the animal, and e is the index of the environment/domain.

Original: In Figure 1a)
Revised: In Figure 1a)

Original: X e 2 is independent of ( Y e , X e 1 ) , it represents the covariate shift case ( Φ ∗ = I ).
Revised: X e 2 is independent of ( Y e , X e 1 ) , it represents the covariate shift case ( Φ ∗ = I ).

Original: In Figure 1b)
Revised: In Figure 1b)

Original: X e 2 is spuriously correlated with Y e through the confounder ε e .
Revised: X e 2 is spuriously correlated with Y e through the confounder ε e .

Original: In
Revised: In

Original: Figure 1c)
Revised: Figure 1c)

Original: X e 2 is spuriously correlated with Y e as it is anti-causally related to Y e .
Revised: X e 2 is spuriously correlated with Y e as it is anti-causally related to Y e .

Original: In both Figure 1b) and c) Φ ∗ (cid:54) =
Revised: In both Figure 1b) and c) Φ ∗ (cid:54) =

Original: I ; Φ ∗ is a block diagonal matrix that selects X e 1 .
Revised: I ; Φ ∗ is a block diagonal matrix that selects X e 1 .

Original: Our setup assumes we are given data from multiple training environments satisfying the invariance condition, i.e., P ( Y | Φ ∗ ( X )) is the same across all of them.
Revised: Our setup assumes we are given data from multiple training environments satisfying the invariance condition, i.e., P ( Y | Φ ∗ ( X )) is the same across all of them.

Original: Ideally, we want to learn and predict using E [ Y | Φ ∗ ( X )] ; this predictor has a desirable OOD behavior as we show later where we prove min-max optimality with respect to (w.r.t.)
Revised: Ideally, we want to learn and predict using E [ Y | Φ ∗ ( X )] ; this predictor has a desirable OOD behavior as we show later where we prove min-max optimality with respect to (w.r.t.)

Original: unseen test distributions satisfying the invariance condition.
Revised: unseen test distributions satisfying the invariance condition.

Original: Our goal is to analyze and compare ERM and IRM’s ability to learn E [ Y | Φ ∗ ( X )] from ﬁnite training data acquired from a ﬁxed number of training environments.
Revised: Our goal is to analyze and compare ERM and IRM’s ability to learn E [ Y | Φ ∗ ( X )] from ﬁnite training data acquired from a ﬁxed number of training environments.

Original: Our analysis has two parts.
Revised: Our analysis has two parts.

Original: 1) Covariate shift case ( Φ ∗ =
Revised: 1) Covariate shift case ( Φ ∗ =

Original: I ):
Revised: I ):

Original: ERM and IRM achieve the same asymptotic solution E
Revised: ERM and IRM achieve the same asymptotic solution E

Original: [ Y | X ] .
Revised: [ Y | X ] .

Original: We prove (Proposition 4) that the sample complexity for both the methods is similar thus there is no clear winner between the two in the ﬁnite sample regime.
Revised: We prove (Proposition 4) that the sample complexity for both the methods is similar thus there is no clear winner between the two in the ﬁnite sample regime.

Original: For the setup in
Revised: For the setup in

Original: Figure 1a), both ERM and IRM learn a model that only uses X e 1 .
Revised: Figure 1a), both ERM and IRM learn a model that only uses X e 1 .

Original: 2) Confounder/Anti-causal variable case ( Φ ∗ (cid:54) =
Revised: 2) Confounder/Anti-causal variable case ( Φ ∗ (cid:54) =

Original: I ):
Revised: I ):

Original: We consider a family of structural equation models (linear and polynomial) that may contain confounders and/or anti-causal variables.
Revised: We consider a family of structural equation models (linear and polynomial) that may contain confounders and/or anti-causal variables.

Original: For the class of models we consider, the asymptotic solution of ERM is biased and not equal to the desired E [ Y | Φ ∗ ( X )] .
Revised: For the class of models we consider, the asymptotic solution of ERM is biased and not equal to the desired E [ Y | Φ ∗ ( X )] .

Original: We prove that IRM can learn a solution that is within O ( √ (cid:15) ) distance from E [ Y | Φ ∗ ( X )] with a sample complexity that increases as O ( 1 (cid:15) 2 ) and increases polynomially in the complexity of the model class (Proposition 5, 6); (cid:15) (deﬁned later) is the slack in IRM constraints.
Revised: We prove that IRM can learn a solution that is within O ( √ (cid:15) ) distance from E [ Y | Φ ∗ ( X )] with a sample complexity that increases as O ( 1 (cid:15) 2 ) and increases polynomially in the complexity of the model class (Proposition 5, 6); (cid:15) (deﬁned later) is the slack in IRM constraints.

Original: For the setup in Figure 1b) and c), IRM gets close to only using X e 1 , while ERM even with inﬁnite data (Proposition 17 in the supplement) continues to use X e 2 .
Revised: For the setup in Figure 1b) and c), IRM gets close to only using X e 1 , while ERM even with inﬁnite data (Proposition 17 in the supplement) continues to use X e 2 .

Original: We summarize the results in Table 1.
Revised: We summarize the results in Table 1.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2019) proposed the colored MNIST (CMNIST) dataset; comparisons on it showed how ERM-based models exploit spurious factors (background color).
Revised: (2019) proposed the colored MNIST (CMNIST) dataset; comparisons on it showed how ERM-based models exploit spurious factors (background color).

Original: The CMNIST dataset relied on anti-causal variables.
Revised: The CMNIST dataset relied on anti-causal variables.

Original: Many supervised learning datasets may not contain anti-causal variables (e.g. human labeled images).
Revised: Many supervised learning datasets may not contain anti-causal variables (e.g. human labeled images).

Original: Therefore, we propose and analyze three new variants of CMNIST in addition to the original one that map to different real-world settings: i) covariate shift based CMNIST (CS-CMNIST): relies on selection bias to induce spurious correlations, ii) confounded CMNIST (CF-CMNIST): relies on confounders to induce spurious correlations, iii) anti-causal CMNIST (ACCMNIST): this is the original CMNIST proposed by Arjovsky et al.
Revised: Therefore, we propose and analyze three new variants of CMNIST in addition to the original one that map to different real-world settings: i) covariate shift based CMNIST (CS-CMNIST): relies on selection bias to induce spurious correlations, ii) confounded CMNIST (CF-CMNIST): relies on confounders to induce spurious correlations, iii) anti-causal CMNIST (ACCMNIST): this is the original CMNIST proposed by Arjovsky et al.

Original: (2019), and iv) anti-causal and confounded (hybrid) CMNIST (HB-CMNIST): relies on confounders and anti-causal variables to induce spurious correlations.
Revised: (2019), and iv) anti-causal and confounded (hybrid) CMNIST (HB-CMNIST): relies on confounders and anti-causal variables to induce spurious correlations.

Original: On the latter three datasets, which belong to the Φ ∗ (cid:54) =
Revised: On the latter three datasets, which belong to the Φ ∗ (cid:54) =

Original: I class described above, IRM shows a much better OOD behavior than ERM, which performs poorly regardless of the data size.
Revised: I class described above, IRM has a much better OOD behavior than ERM, which performs poorly regardless of the data size.

Original: However,we ﬁnd IRM and ERM have a similar performance on CS-CMNIST( Φ ∗ = I ) with no clear winner.
Revised: However, IRM and ERM have a similar performance on CS-CMNIST with no clear winner.

Original: These results are consistent with our theory and are further validatedwith experiments inthe regression setting.
Revised: These results are consistent with our theory and are also validated in regression experiments.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: IRM based works.
Revised: IRM based works.

Original: Following the original work IRM from Arjovsky et al.
Revised: Following the original work IRM from Arjovsky et al.

Original: (2019), there have been several interesting works — (Teney et al., 2020; Krueger et al., 2020; Ahuja et al., 2020; Chang et al., 2020; Mahajan et al., 2020) is an incomplete representative list — that build new methods inpired from IRM to address the OOD generalization problem.
Revised: (2019), there have been several interesting works — (Teney et al., 2020; Krueger et al., 2020; Ahuja et al., 2020; Chang et al., 2020; Mahajan et al., 2020) is an incomplete representative list — that build new methods inpired from IRM to address the OOD generalization problem.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2019) prove OOD guarantees for linear models with access to inﬁnite data from ﬁnite environments.
Revised: (2019) prove OOD guarantees for linear models with access to inﬁnite data from ﬁnite environments.

Original: We generalize these results in several ways.
Revised: We generalize these results in several ways.

Original: We provide a ﬁrst ﬁnite sample analysis of IRM.
Revised: We provide a ﬁrst ﬁnite sample analysis of IRM.

Original: We characterize the impact of hypothesis class complexity, number of environments, weight of IRM penalty on the sample complexity and its distance from the OOD solution for linear and polynomial models.
Revised: We characterize the impact of hypothesis class complexity, number of environments, weight of IRM penalty on the sample complexity and its distance from the OOD solution for linear and polynomial models.

Original: Theory of domain generalization and domain adaption.
Revised: Theory of domain generalization and domain adaption.

Original: Following the seminal works (BenDavid et al., 2007; 2010), there have been many interesting works — (Muandet et al., 2013;
Revised: Following the seminal works (BenDavid et al., 2007; 2010), there have been many interesting works — (Muandet et al., 2013;

Original: Ajakan et al., 2014; Zhao et al., 2019; Albuquerque et al., 2019;
Revised: Ajakan et al., 2014; Zhao et al., 2019; Albuquerque et al., 2019;

Original: Li et al., 2017; Piratla et al., 2020; Matsuura & Harada, 2020; Deng et al., 2020; David et al., 2010; Pagnoni et al., 2018) is an incomplete representative list (see Redko et al.
Revised: Li et al., 2017; Piratla et al., 2020; Matsuura & Harada, 2020; Deng et al., 2020; David et al., 2010; Pagnoni et al., 2018) is an incomplete representative list (see Redko et al.

Original: for further references) — that build the theory of domain adaptation and generalization and construct new methods based on it.
Revised: for further references) — that build the theory of domain adaptation and generalization and construct new methods based on it.

Original: While many of these works develop bounds on loss over the target domain using train data and unlabeled target data,
Revised: While many of these works develop bounds on loss over the target domain using train data and unlabeled target data,

Original: [Table]
Revised: [Table]

Original: case: E e [ Y e | Φ ∗ ( X e )] is invariant,
Revised: case: E e [ Y e | Φ ∗ ( X e )] is invariant,

Original: Linear, Polynomial models, |E tr | = O ( n p ) (Proposition 5, 6,17)
Revised: Linear, Polynomial models, |E tr | = O ( n p ) (Proposition 5, 6,17)

Original: [Table]
Revised: [Table]

Original: some (Ben-David & Urner, 2012; David et al., 2010; Pagnoni et al., 2018) analyze the ﬁnite sample (PAC) guarantees for domain adaptation under covariate shifts.
Revised: some (Ben-David & Urner, 2012; David et al., 2010; Pagnoni et al., 2018) analyze the ﬁnite sample (PAC) guarantees for domain adaptation under covariate shifts.

Original: These works (Ben-David & Urner, 2012; David et al., 2010; Pagnoni et al., 2018) access unlabeled data from a target domain, which we do not.
Revised: These works (Ben-David & Urner, 2012; David et al., 2010; Pagnoni et al., 2018) access unlabeled data from a target domain, which we do not.

Original: Instead, we have data from multiple training domains (as in domain generalization).
Revised: Instead, we have data from multiple training domains (as in domain generalization).

Original: In these works
Revised: In these works

Original: [Table]
Revised: [Table]

Original: the guarantees are w.r.t.
Revised: the guarantees are w.r.t.

Original: a speciﬁc target domain, while we provide (for linear and polynomial models) worst-case guarantees w.r.t.
Revised: a speciﬁc target domain, while we provide (for linear and polynomial models) worst-case guarantees w.r.t.

Original: all the unseen domains satisfying the invariance condition.
Revised: all the unseen domains satisfying the invariance condition.

Original: Also, we consider a larger family of distribution shifts including covariate shifts.
Revised: Also, we consider a larger family of distribution shifts including covariate shifts.

Original: The above two categories are not exhaustive – e.g., there are some recent works that characterize how some inductive biases favor extrapolation Xu et al.
Revised: The above two categories are not exhaustive – e.g., there are some recent works that characterize how some inductive biases favor extrapolation Xu et al.

Original: and can be better for OOD generalization.
Revised: and can be better for OOD generalization.

Original:  S AMPLE C OMPLEXITY OF I NVARIANT R ISK M INIMIZATION 3. I NVARIANT R ISK M INIMIZATION
Revised:  S AMPLE C OMPLEXITY OF I NVARIANT R ISK M INIMIZATION 3. I NVARIANT R ISK M INIMIZATION

Original: We start with some background on IRM (Arjovsky et al., 2019).
Revised: We start with some background on IRM (Arjovsky et al., 2019).

Original: Consider a dataset D
Revised: Consider a dataset D

Original: = { D e } e ∈E tr , which is a collection of datasets D e
Revised: = { D e } e ∈E tr , which is a collection of datasets D e

Original: =
Revised: =

Original: ( x ei , y ei , e ) } n e i =1 } obtained from a set of training environments E tr , where e is the index of the environment, i is the index of the data point in the environment, n e is the number of points from environment, x ei ∈ X ⊆ R n is the feature value and
Revised: ( x ei , y ei , e ) } n e i =1 } obtained from a set of training environments E tr , where e is the index of the environment, i is the index of the data point in the environment, n e is the number of points from environment, x ei ∈ X ⊆ R n is the feature value and

Original: y ei ∈ Y ⊆
Revised: y ei ∈ Y ⊆

Original: R is the corresponding label.
Revised: R is the corresponding label.

Original: Deﬁne a probability distribution { π e } e ∈E tr , π e is the probability that a training data point is from environment e .
Revised: Deﬁne a probability distribution { π e } e ∈E tr , π e is the probability that a training data point is from environment e .

Original: Deﬁne a probability distribution of points conditional on environment e as P e , ( X e , Y e ) ∼ P e .
Revised: Deﬁne a probability distribution of points conditional on environment e as P e , ( X e , Y e ) ∼ P e .

Original: Deﬁne the joint distribution ¯ P , ( X e , Y e , e ) ∼ ¯ P , d ¯ P ( X e , Y e , e ) =
Revised: Deﬁne the joint distribution ¯ P , ( X e , Y e , e ) ∼ ¯ P , d ¯ P ( X e , Y e , e ) =

Original: π e d P e
Revised: π e d P e

Original: ( X e , Y e ) .
Revised: ( X e , Y e ) .

Original: D is a collection of i.i.d.
Revised: D is a collection of i.i.d.

Original: samples from ¯ P .
Revised: samples from ¯ P .

Original: Deﬁne a predictor f : X → R and the space F of all the possible maps from X → R .
Revised: Deﬁne a predictor f : X → R and the space F of all the possible maps from X → R .

Original: Deﬁne the risk achieved by f in environment e as R e ( f ) =
Revised: Deﬁne the risk achieved by f in environment e as R e ( f ) =

Original: E e (cid:2) (cid:96) (cid:0) f
Revised: E e (cid:2) (cid:96) (cid:0) f

Original: ( X e ) , Y e (cid:1)(cid:3) , where (cid:96) is the loss, f ( X e ) is the predicted value, Y e is the corresponding label and E e is the expectation conditional on environment e .
Revised: ( X e ) , Y e (cid:1)(cid:3) , where (cid:96) is the loss, f ( X e ) is the predicted value, Y e is the corresponding label and E e is the expectation conditional on environment e .

Original: The overall expected risk across the training environments is R ( f )
Revised: The overall expected risk across the training environments is R ( f )

Original: = (cid:80) e ∈E tr π e R e ( f ) .
Revised: = (cid:80) e ∈E tr π e R e ( f ) .

Original: We are interested in two settings: regression (square loss) and binary-classiﬁcation (cross-entropy loss).
Revised: We are interested in two settings: regression (square loss) and binary-classiﬁcation (cross-entropy loss).

Original: In the main body, our focus is regression (square loss) and we mention wherever the results extend to binary-classiﬁcation (cross-entropy).
Revised: In the main body, our focus is regression (square loss) and we mention wherever the results extend to binary-classiﬁcation (cross-entropy).

Original: We discuss these extensions in the supplement.
Revised: We discuss these extensions in the supplement.

Original: OOD generalization problem.
Revised: OOD generalization problem.

Original: We want to construct a predictor f that performs well across many unseen environments E all , where E all ⊇ E tr .
Revised: We want to construct a predictor f that performs well across many unseen environments E all , where E all ⊇ E tr .

Original: For o ∈ E all \E tr , the distribution P o can be very different from the train environments.
Revised: For o ∈ E all \E tr , the distribution P o can be very different from the train environments.

Original: Next we state the OOD problem.
Revised: Next we state the OOD problem.

Original: [Equation]
Revised: [Equation]

Original: The above problem is very challenging to solve since we only have access to data from training environments E tr but are required to ﬁnd the robust solution over all environments E all .
Revised: The above problem is very challenging to solve since we only have access to data from training environments E tr but are required to ﬁnd the robust solution over all environments E all .

Original: Next, we make assumptions on E all and characterize the optimal solution to equation 1.
Revised: Next, we make assumptions on E all and characterize the optimal solution to equation 1.

Original: Assumption 1.
Revised: Assumption 1.

Original: Invariance condition.
Revised: Invariance condition.

Original: There exists a representation Φ ∗ that transforms X e to
Revised: There exists a representation Φ ∗ that transforms X e to

Original: Z e =
Revised: Z e =

Original: Φ ∗ ( X e ) and ∀ e, o ∈ E all , ∀ z ∈ Φ ∗ ( X ) satisﬁes E e
Revised: Φ ∗ ( X e ) and ∀ e, o ∈ E all , ∀ z ∈ Φ ∗ ( X ) satisﬁes E e

Original: [ Y e | Z e = z ] =
Revised: [ Y e | Z e = z ] =

Original: E o
Revised: E o

Original: [ Y o | Z
Revised: [ Y o | Z

Original: o =
Revised: o =

Original: z ] .
Revised: z ] .

Original: Also, ∀ e ∈ E all , ∀ z ∈ Φ ∗ ( X ) , Var e
Revised: Also, ∀ e ∈ E all , ∀ z ∈ Φ ∗ ( X ) , Var e

Original: [ Y e | Z e = z ] = ξ 2 , where Var e is the conditional variance.
Revised: [ Y e | Z e = z ] = ξ 2 , where Var e is the conditional variance.

Original: The above assumption is inspired from causality (Pearl, 2009).
Revised: The above assumption is inspired from causality (Pearl, 2009).

Original: Φ ∗ acts as the causal feature extractor and from the deﬁnition of causal features, it follows that E e [ Y e | Z e = z ] does not vary across environments.
Revised: Φ ∗ acts as the causal feature extractor and from the deﬁnition of causal features, it follows that E e [ Y e | Z e = z ] does not vary across environments.

Original: When a human labels a cow she uses Φ ∗ to extract causal features from the pixels to identify cow while ignoring the background.
Revised: When a human labels a cow she uses Φ ∗ to extract causal features from the pixels to identify cow while ignoring the background.

Original: The ﬁrst part of the above assumption encompasses a large class of distribution shifts including standard covariate shifts (Gretton et al., 2009).
Revised: The ﬁrst part of the above assumption encompasses a large class of distribution shifts including standard covariate shifts (Gretton et al., 2009).

Original: Covariate shift assumes ∀ e, o ∈ E all , ∀ x ∈ X , P
Revised: Covariate shift assumes ∀ e, o ∈ E all , ∀ x ∈ X , P

Original: ( Y e | X e = x ) and P ( Y o |
Revised: ( Y e | X e = x ) and P ( Y o |

Original: X o = x ) are equal thus implying
Revised: X o = x ) are equal thus implying

Original: E e
Revised: E e

Original: [ Y e | X e = x ] = E
Revised: [ Y e | X e = x ] = E

Original: o
Revised: o

Original: [ Y o |
Revised: [ Y o |

Original: X
Revised: X

Original: o = x ] .
Revised: o = x ] .

Original: Therefore, for covariate shifts, Φ ∗ is identity in Assumption 1.
Revised: Therefore, for covariate shifts, Φ ∗ is identity in Assumption 1.

Original: A simple instance illustrating Assumption 1 with Φ ∗
Revised: A simple instance illustrating Assumption 1 with Φ ∗

Original: = I is when Y e ← g ( X e )+ ε e , where E e [ ε e ] = 0 , E e
Revised: = I is when Y e ← g ( X e )+ ε e , where E e [ ε e ] = 0 , E e

Original: [( ε e ) 2 ] = σ 2 , ε e ⊥ X e .
Revised: [( ε e ) 2 ] = σ 2 , ε e ⊥ X e .

Original: Using Assumption 1, we deﬁne the invariant map m : Φ ∗ ( X ) → R as follows
Revised: Using Assumption 1, we deﬁne the invariant map m : Φ ∗ ( X ) → R as follows

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Z e = Φ
Revised: Z e = Φ

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 2.

Original: Existence of an environment where the invariant representation is sufﬁcient.
Revised: Existence of an environment where the invariant representation is sufﬁcient.

Original: ∃ an environment e ∈ E all such that Y e ⊥ X e | Z e
Revised: ∃ an environment e ∈ E all such that Y e ⊥ X e | Z e

Original: Assumption 2 states there exists an environment where the information that X e has about Y e is also contained in Z e .
Revised: Assumption 2 states there exists an environment where the information that X e has about Y e is also contained in Z e .

Original: Deﬁne a composition m ◦ Φ ∗ , ∀ x ∈ X , m ◦ Φ ∗ ( x ) =
Revised: Deﬁne a composition m ◦ Φ ∗ , ∀ x ∈ X , m ◦ Φ ∗ ( x ) =

Original: E e
Revised: E e

Original: [ Y e | Z e = Φ ∗ ( x )] .
Revised: [ Y e | Z e = Φ ∗ ( x )] .

Original: Proposition 1.
Revised: Proposition 1.

Original: If (cid:96) is the square loss, and Assumptions 1 and 2 hold, then m ◦ Φ ∗ solves the OOD problem (equation 1).
Revised: If (cid:96) is the square loss, and Assumptions 1 and 2 hold, then m ◦ Φ ∗ solves the OOD problem (equation 1).

Original: The proofs of all the propositions are in the supplement.
Revised: The proofs of all the propositions are in the supplement.

Original: A similar result holds for the cross-entropy loss (discussion in supplement).
Revised: A similar result holds for the cross-entropy loss (discussion in supplement).

Original: For the rest of the paper, we focus on learning m ◦ Φ ∗ as it solves the OOD problem.
Revised: For the rest of the paper, we focus on learning m ◦ Φ ∗ as it solves the OOD problem.

Original: For covariate shifts Φ ∗ =
Revised: For covariate shifts Φ ∗ =

Original: I , m ( x ) =
Revised: I , m ( x ) =

Original: E e
Revised: E e

Original: [ Y e | X e = x ] is the OOD solution.
Revised: [ Y e | X e = x ] is the OOD solution.

Original: In
Revised: In

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2019), a proof connecting m ◦ Φ ∗ and OOD was not stated.
Revised: (2019), a proof connecting m ◦ Φ ∗ and OOD was not stated.

Original: Recently, in Koyama & Yamaguchi (2020), a result similar to Proposition 1 was shown but with a few differences.
Revised: Recently, in Koyama & Yamaguchi (2020), a result similar to Proposition 1 was shown but with a few differences.

Original: The authors assume conditional probabilities are invariant unlike our assumption that only requires conditional expectations and variances to be invariant.
Revised: The authors assume conditional probabilities are invariant unlike our assumption that only requires conditional expectations and variances to be invariant.

Original: However, their result applies to more losses.
Revised: However, their result applies to more losses.

Original: m ◦ Φ ∗ is the target we want to learn.
Revised: m ◦ Φ ∗ is the target we want to learn.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2019) proposed IRM since standard min-max optimization over the training environments E tr
Revised: (2019) proposed IRM since standard min-max optimization over the training environments E tr

Original: and ERM fail to learn m ◦ Φ ∗ in many cases.
Revised: and ERM fail to learn m ◦ Φ ∗ in many cases.

Original: The authors in Arjovsky et al.
Revised: The authors in Arjovsky et al.

Original: (2019) identify a crucial property of m ◦ Φ ∗ and use it to deﬁne an object called invariant predictor that we deﬁne next.
Revised: (2019) identify a crucial property of m ◦ Φ ∗ and use it to deﬁne an object called invariant predictor that we deﬁne next.

Original: Invariant predictor and IRM optimization.
Revised: Invariant predictor and IRM optimization.

Original: Deﬁne a representation map Φ : X → Z from feature space to representation space Z ⊆ R q .
Revised: Deﬁne a representation map Φ : X → Z from feature space to representation space Z ⊆ R q .

Original: Deﬁne a classiﬁer map, w : Z → R from representation space to real values.
Revised: Deﬁne a classiﬁer map, w : Z → R from representation space to real values.

Original: Deﬁne H Φ and H w as the spaces of representations and classiﬁers respectively.
Revised: Deﬁne H Φ and H w as the spaces of representations and classiﬁers respectively.

Original: A data representation Φ elicits an invariant predictor w ◦ Φ across environments e ∈ E tr if there is a classiﬁer w that achieves the minimum risk simultaneously for all the environments, i.e., ∀ e ∈
Revised: A data representation Φ elicits an invariant predictor w ◦ Φ across environments e ∈ E tr if there is a classiﬁer w that achieves the minimum risk simultaneously for all the environments, i.e., ∀ e ∈

Original: E tr , w ∈ arg min ¯ w ∈H w R e ( ¯ w ◦ Φ) .
Revised: E tr , w ∈ arg min ¯ w ∈H w R e ( ¯ w ◦ Φ) .

Original: Observe that if we we transform the data with representation Φ ∗ then m will achieve the minimum risk simultaneously in all the environments.
Revised: Observe that if we we transform the data with representation Φ ∗ then m will achieve the minimum risk simultaneously in all the environments.

Original: If Φ ∗ ∈ H Φ and m ∈ H w , then m ◦ Φ ∗ is an invariant predictor.
Revised: If Φ ∗ ∈ H Φ and m ∈ H w , then m ◦ Φ ∗ is an invariant predictor.

Original: IRM selects the invariant predictor with least sum risk across environments (results presented later can be adapted if invariant predictor was selected based on the worst-case risk over the environments as well) as follows:
Revised: IRM selects the invariant predictor with least sum risk across environments (results presented later can be adapted if invariant predictor was selected based on the worst-case risk over the environments as well) as follows:

Original: [Equation]
Revised: [Equation]

Original: From the above discussion we know m ◦ Φ ∗ is a feasible solution to equation 3.
Revised: From the above discussion we know m ◦ Φ ∗ is a feasible solution to equation 3.

Original: It is also the ideal solution we want IRM to ﬁnd since it solves equation 1.
Revised: It is also the ideal solution we want IRM to ﬁnd since it solves equation 1.

Original: Later in Propositions 4, 5, and 6, we show that IRM actually solves equation equation 1.
Revised: Later in Propositions 4, 5, and 6, we show that IRM actually solves equation equation 1.

Original: For the setups in Proposition 5, and 6, conventional
Revised: For the setups in Proposition 5, and 6, conventional

Original: ERM based approaches fail thus justifying the need for above formulation.
Revised: ERM based approaches fail thus justifying the need for above formulation.

Original: 3. S AMPLE C OMPLEXITY OF G RADIENT C ONSTRAINT F ORMULATION OF IRM
Revised: 3. S AMPLE C OMPLEXITY OF G RADIENT C ONSTRAINT F ORMULATION OF IRM

Original: In Arjovsky et al.
Revised: In Arjovsky et al.

Original: , a gradient constrained alternate (derived below in equation 4) to equationwas proposed, which focuses on linear and scalar classiﬁers ( Z = R , Φ : X → R , H w = R ).
Revised: , a gradient constrained alternate (derived below in equation 4) to equationwas proposed, which focuses on linear and scalar classiﬁers ( Z = R , Φ : X → R , H w = R ).

Original: In this case, the composite predictor w ◦ Φ is a multiplication of w and Φ written as w · Φ .
Revised: In this case, the composite predictor w ◦ Φ is a multiplication of w and Φ written as w · Φ .

Original: (For binaryclassiﬁcation predictor’s output w · Φ( x ) represents logits.)
Revised: (For binaryclassiﬁcation predictor’s output w · Φ( x ) represents logits.)

Original: From the deﬁnition of invariant predictors and H w
Revised: From the deﬁnition of invariant predictors and H w

Original: =
Revised: =

Original: R it follows that if ∀ ¯ w ∈ R , R e (1 · Φ) ≤ R e ( ¯ w · Φ) , then Φ is an invariant predictor.
Revised: R it follows that if ∀ ¯ w ∈ R , R e (1 · Φ) ≤ R e ( ¯ w · Φ) , then Φ is an invariant predictor.

Original: For square and cross-entropy losses, R e ( w · Φ) is convex in w .
Revised: For square and cross-entropy losses, R e ( w · Φ) is convex in w .

Original: Therefore, a gradient constraint (cid:107)∇ w | w =1 . 

Revised: Therefore, a gradient constraint (cid:107)∇ w | w =1 .

Original: R e ( w · Φ) (cid:107) = 0 is equivalent to the condition that ∀ ¯ w ∈ R , R e (1 · Φ) ≤ R e ( ¯ w · Φ) , which implies Φ is an invariant predictor.
Revised: R e ( w · Φ) (cid:107) = 0 is equivalent to the condition that ∀ ¯ w ∈ R , R e (1 · Φ) ≤ R e ( ¯ w · Φ) , which implies Φ is an invariant predictor.

Original: Recall that IRM aims is to search among invariant predictors and ﬁnd one that minimizes the risk.
Revised: Recall that IRM aims is to search among invariant predictors and ﬁnd one that minimizes the risk.

Original: We state this as a gradient constrained optimization as follows
Revised: We state this as a gradient constrained optimization as follows

Original: [Equation]
Revised: [Equation]

Original: We propose an (cid:15) approximation of the above with (cid:15) slack in the constraint.
Revised: We propose an (cid:15) approximation of the above with (cid:15) slack in the constraint.

Original: Deﬁne R (cid:48) (Φ) =
Revised: Deﬁne R (cid:48) (Φ) =

Original: (cid:80) e ∈E tr π e (cid:107)∇ w | w =1 . 

Revised: (cid:80) e ∈E tr π e (cid:107)∇ w | w =1 .

Original: R e ( w · Φ) (cid:107) 2 and a set S IV ( (cid:15) ) =
Revised: R e ( w · Φ) (cid:107) 2 and a set S IV ( (cid:15) ) =

Original: { Φ | R (cid:48) (Φ) ≤ (cid:15), Φ ∈ H Φ } .
Revised: { Φ | R (cid:48) (Φ) ≤ (cid:15), Φ ∈ H Φ } .

Original: Note that R (cid:48) is very similar to the penalty deﬁned in Arjovsky et al.
Revised: Note that R (cid:48) is very similar to the penalty deﬁned in Arjovsky et al.

Original: The (cid:15) approximation of equation 4 is
Revised: The (cid:15) approximation of equation 4 is

Original: [Equation]
Revised: [Equation]

Original: If (cid:15) = 0 , then equation 4 and equation 5 are equivalent.
Revised: If (cid:15) = 0 , then equation 4 and equation 5 are equivalent.

Original: In all the optimizations so far, the expectations are computed w.r.t.
Revised: In all the optimizations so far, the expectations are computed w.r.t.

Original: the distributions P e , which are unknown.
Revised: the distributions P e , which are unknown.

Original: Therefore, we develop an empirical version of equation 5 below (in equation 6) and call it empirical IRM (EIRM).
Revised: Therefore, we develop an empirical version of equation 5 below (in equation 6) and call it empirical IRM (EIRM).

Original: We replace
Revised: We replace

Original: R and R (cid:48) with empirical estimators ˆ R and ˆ R (cid:48) respectively.
Revised: R and R (cid:48) with empirical estimators ˆ R and ˆ R (cid:48) respectively.

Original: For R we use a simple plugin estimator (sample mean of loss across all the samples in D ).
Revised: For R we use a simple plugin estimator (sample mean of loss across all the samples in D ).

Original: For R (cid:48) we construct a new estimator that enables the use of standard concentration inequalities.
Revised: For R (cid:48) we construct a new estimator that enables the use of standard concentration inequalities.

Original: Deﬁne a set ˆ S IV ( (cid:15) )
Revised: Deﬁne a set ˆ S IV ( (cid:15) )

Original: =
Revised: =

Original: { Φ | ˆ R (cid:48) (Φ) ≤ (cid:15), Φ ∈ H Φ } .
Revised: { Φ | ˆ R (cid:48) (Φ) ≤ (cid:15), Φ ∈ H Φ } .

Original: [Equation]
Revised: [Equation]

Original: If we replaced ˆ S IV ( (cid:15) ) with H Φ in equation 6, then we get the standard ERM.
Revised: If we replaced ˆ S IV ( (cid:15) ) with H Φ in equation 6, then we get the standard ERM.

Original: ERM aims to solve min Φ ∈H Φ ˆ R (Φ) .
Revised: ERM aims to solve min Φ ∈H Φ ˆ R (Φ) .

Original: The sample complexity analysis of ERM aims to understand the distance between the empirical solutions and the expected solutions as a function of the number of samples.
Revised: The sample complexity analysis of ERM aims to understand the distance between the empirical solutions and the expected solutions as a function of the number of samples.

Original: Similarly, we seek to understand the relationship between solutions of equation 6 and equation 5.
Revised: Similarly, we seek to understand the relationship between solutions of equation 6 and equation 5.

Original: Assumption 3.
Revised: Assumption 3.

Original: Bounded loss and bounded gradient of the loss.
Revised: Bounded loss and bounded gradient of the loss.

Original: ∃ L < ∞ , L (cid:48) < ∞ such that ∀ Φ
Revised: ∃ L < ∞ , L (cid:48) < ∞ such that ∀ Φ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: If every Φ in the hypothesis class H Φ is bounded by M and the label space Y is bounded, then for both square and cross-entropy loss, (cid:96) (Φ( · ) , · ) and ∂(cid:96) ( w · Φ( · ) , · ) ∂w | w =1 . 

Revised: If every Φ in the hypothesis class H Φ is bounded by M and the label space Y is bounded, then for both square and cross-entropy loss, (cid:96) (Φ( · ) , · ) and ∂(cid:96) ( w · Φ( · ) , · ) ∂w | w =1 .

Original: are bounded.
Revised: are bounded.

Original: Deﬁne κ = min
Revised: Deﬁne κ = min

Original: Φ ∈H Φ | R (cid:48) (Φ) − (cid:15) | ; κ measures how close any penalty can get to the boundary (cid:15) .
Revised: Φ ∈H Φ | R (cid:48) (Φ) − (cid:15) | ; κ measures how close any penalty can get to the boundary (cid:15) .

Original: κ quantiﬁes how good the ﬁnite sample approximation ˆ R (cid:48) need to be in order to get ˆ S IV ( (cid:15) )
Revised: κ quantiﬁes how good the ﬁnite sample approximation ˆ R (cid:48) need to be in order to get ˆ S IV ( (cid:15) )

Original: =
Revised: =

Original: S IV ( (cid:15) ) .
Revised: S IV ( (cid:15) ) .

Original: Deﬁne ν to quantify the approximation w.r.t.
Revised: Deﬁne ν to quantify the approximation w.r.t.

Original: optimal risk.
Revised: optimal risk.

Original: Proposition 2.
Revised: Proposition 2.

Original: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, Assumption 3 holds, κ > 0 , and if the number of samples | D | is greater than max (cid:8) 16 L (cid:48) 4 κ 2 , 8 L 2 ν 2 (cid:9) log (cid:0) 4 |H
Revised: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, Assumption 3 holds, κ > 0 , and if the number of samples | D | is greater than max (cid:8) 16 L (cid:48) 4 κ 2 , 8 L 2 ν 2 (cid:9) log (cid:0) 4 |H

Original: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) is a ν approximation of IRM, i.e. ˆΦ ∈ S IV ( (cid:15) ) , R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )
Revised: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) is a ν approximation of IRM, i.e. ˆΦ ∈ S IV ( (cid:15) ) , R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )

Original: + ν , where Φ ∗ is a solution to IRM (equation 5).
Revised: + ν , where Φ ∗ is a solution to IRM (equation 5).

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The standard analysis in learning theory on ERM or regularized/constrained ERM typically relies on linearly separable loss functions.
Revised: The standard analysis in learning theory on ERM or regularized/constrained ERM typically relies on linearly separable loss functions.

Original: In such cases, we can use standard plugin estimators and analyze their behavior using concentration inequalities.
Revised: In such cases, we can use standard plugin estimators and analyze their behavior using concentration inequalities.

Original: In our setting, R (cid:48) is a weighted sum of squares of expectation and thus it is not linearly separable.
Revised: In our setting, R (cid:48) is a weighted sum of squares of expectation and thus it is not linearly separable.

Original: We develop a new way of expressing R (cid:48) that allows us to make it linearly separable.
Revised: We develop a new way of expressing R (cid:48) that allows us to make it linearly separable.

Original: Next, in order to ensure
Revised: Next, in order to ensure

Original: R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )
Revised: R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )

Original: + ν
Revised: + ν

Original: we ﬁrst need to guarantee that the set of invariant predictors is exactly recovered
Revised: we ﬁrst need to guarantee that the set of invariant predictors is exactly recovered

Original: , i.e., ˆ S IV ( (cid:15) )
Revised: , i.e., ˆ S IV ( (cid:15) )

Original: =
Revised: =

Original: S IV ( (cid:15) )
Revised: S IV ( (cid:15) )

Original: (exact recovery is typically not required in existing constrained analysis such as Woodworth et al.
Revised: (exact recovery is typically not required in existing constrained analysis such as Woodworth et al.

Original: Agarwal et al.
Revised: Agarwal et al.

Original: We show that if the number of samples grow as 1 κ 2 even the closest points on either side of the boundary of the set S IV ( (cid:15) ) are correctly discriminated, which guarantees exact recovery of S IV ( (cid:15) ) .
Revised: We show that if the number of samples grow as 1 κ 2 even the closest points on either side of the boundary of the set S IV ( (cid:15) ) are correctly discriminated, which guarantees exact recovery of S IV ( (cid:15) ) .

Original: Once the exact set is recovered, beyond this we use standard learning theory tools to ensure R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )
Revised: Once the exact set is recovered, beyond this we use standard learning theory tools to ensure R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )

Original: + ν .
Revised: + ν .

Original: The above result holds for both square and cross-entropy loss.
Revised: The above result holds for both square and cross-entropy loss.

Original: For ease of exposition, we use the standard setting of ﬁnite hypothesis class and extend all the results to inﬁnite hypothesis classes in the supplement (summary of insights from the extension are in Section 3.3.2).
Revised: For ease of exposition, we use the standard setting of ﬁnite hypothesis class and extend all the results to inﬁnite hypothesis classes in the supplement (summary of insights from the extension are in Section 3.3.2).

Original: Next, we state a standard result on ERM’s sample complexity.
Revised: Next, we state a standard result on ERM’s sample complexity.

Original: Deﬁne a Φ + such that Φ + ∈ arg min Φ ∈H Φ R (Φ)
Revised: Deﬁne a Φ + such that Φ + ∈ arg min Φ ∈H Φ R (Φ)

Original: Proposition 3.
Revised: Proposition 3.

Original: (Shalev-Shwartz & Ben-David, 2014)
Revised: (Shalev-Shwartz & Ben-David, 2014)

Original: For every ν > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, Assumption 3 holds, and if the number of samples | D | is greater than
Revised: For every ν > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, Assumption 3 holds, and if the number of samples | D | is greater than

Original: 8 L 2 ν 2 log (cid:0) 2 |H
Revised: 8 L 2 ν 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1)
Revised: Φ | δ (cid:1)

Original: , then with a probability at least 1 − δ , every solution Φ † to ERM is an ν approximation of expected risk minimization, i.e., R (Φ + ) ≤ R (Φ † ) ≤ R (Φ + ) + ν .
Revised: , then with a probability at least 1 − δ , every solution Φ † to ERM is an ν approximation of expected risk minimization, i.e., R (Φ + ) ≤ R (Φ † ) ≤ R (Φ + ) + ν .

Original: Proposition 2 vs. 3 Since κ ≤ (cid:15) , the sample complexity of EIRM grows at least as O (max { 1 (cid:15) 2 , 1 ν 2 } ) .
Revised: Proposition 2 vs. 3 Since κ ≤ (cid:15) , the sample complexity of EIRM grows at least as O (max { 1 (cid:15) 2 , 1 ν 2 } ) .

Original: Let us look at the two terms inside max - i) 1 ν 2 growth term is similar to ERM, it ensures ν approximate optimality in the overall risk R , ii) 1 (cid:15) 2 growth ensures the IRM penalty R (cid:48) is less than (cid:15) .
Revised: Let us look at the two terms inside max - i) 1 ν 2 growth term is similar to ERM, it ensures ν approximate optimality in the overall risk R , ii) 1 (cid:15) 2 growth ensures the IRM penalty R (cid:48) is less than (cid:15) .

Original: A direct comparison of sample complexities in Propositions 2 and 3 suggests that the sample complexity of EIRM is higher than ERM, which is not the complete picture.
Revised: A direct comparison of sample complexities in Propositions 2 and 3 suggests that the sample complexity of EIRM is higher than ERM, which is not the complete picture.

Original: The two approaches may not converge to the same solutions and IRM may converge to a solution with better OOD behavior than one achieved by ERM.
Revised: The two approaches may not converge to the same solutions and IRM may converge to a solution with better OOD behavior than one achieved by ERM.

Original: Therefore, a fair comparison is only possible when we also study the OOD properties of the solutions achieved by the two approaches, which is the subject of the next section.
Revised: Therefore, a fair comparison is only possible when we also study the OOD properties of the solutions achieved by the two approaches, which is the subject of the next section.

Original: 3. OOD P ERFORMANCE : ERM VS . IRM
Revised: 3. OOD P ERFORMANCE : ERM VS . IRM

Original: We divide the comparisons based on distributional shift assumptions that decide whether ERM and
Revised: We divide the comparisons based on distributional shift assumptions that decide whether ERM and

Original: IRM arrive at the same asymptotic solutions or not.
Revised: IRM arrive at the same asymptotic solutions or not.

Original: 3.3. C OVARIATE SHIFT
Revised: 3.3. C OVARIATE SHIFT

Original: Assumption 4.
Revised: Assumption 4.

Original: Invariance w.r.t.
Revised: Invariance w.r.t.

Original: all the features.
Revised: all the features.

Original: ∀ e, o ∈ E all and ∀ x ∈ X , E [ Y e |
Revised: ∀ e, o ∈ E all and ∀ x ∈ X , E [ Y e |

Original: X e = x ] =
Revised: X e = x ] =

Original: E
Revised: E

Original: [ Y o | X o = x ] .
Revised: [ Y o | X o = x ] .

Original: ∀ e ∈ E all , X e ∼ P eX e and the support of P eX e is equal to X .
Revised: ∀ e ∈ E all , X e ∼ P eX e and the support of P eX e is equal to X .

Original: As stated earlier, the ﬁrst part of the above assumption follows from standard covariate shift assumptions (Gretton et al., 2009) and is a special case of the ﬁrst part of the Assumption 1 with Φ ∗ set to I .
Revised: As stated earlier, the ﬁrst part of the above assumption follows from standard covariate shift assumptions (Gretton et al., 2009) and is a special case of the ﬁrst part of the Assumption 1 with Φ ∗ set to I .

Original: If Φ ∗ =
Revised: If Φ ∗ =

Original: I , then m (equation 2), which simpliﬁes to m ( x ) =
Revised: I , then m (equation 2), which simpliﬁes to m ( x ) =

Original: E [ Y e | X e = x ] , solves the OOD problem equation 1.
Revised: E [ Y e | X e = x ] , solves the OOD problem equation 1.

Original: A generative model that satisﬁes the above Assumption 4 is given as
Revised: A generative model that satisﬁes the above Assumption 4 is given as

Original: [Equation]
Revised: [Equation]

Original: In the above model X e is the cause, Y e is the effect, and g a general non-linear function (it satisﬁes
Revised: In the above model X e is the cause, Y e is the effect, and g a general non-linear function (it satisﬁes

Original: Assumption 4 with m = g ).
Revised: Assumption 4 with m = g ).

Original: Next, we compare ERM and IRM’s ability to learn m under covariate shifts.
Revised: Next, we compare ERM and IRM’s ability to learn m under covariate shifts.

Original: In Figure 1 a), we show Deﬁne ˜ κ = min
Revised: In Figure 1 a), we show Deﬁne ˜ κ = min

Original: Φ 1 , Φ 2 ∈H Φ , Φ 1 (cid:54) =
Revised: Φ 1 , Φ 2 ∈H Φ , Φ 1 (cid:54) =

Original: Φ 2 | R (Φ 1 ) − R (Φ 2 ) | , which measures the minimum separation between the risks of any two distinct hypothesis in H Φ .
Revised: Φ 2 | R (Φ 1 ) − R (Φ 2 ) | , which measures the minimum separation between the risks of any two distinct hypothesis in H Φ .

Original: Proposition 4.
Revised: Proposition 4.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, m ∈ H Φ , Assumptions 3, 4 hold, and
Revised: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, m ∈ H Φ , Assumptions 3, 4 hold, and

Original: • if the number of samples | D | is greater than max (cid:8) 8 L 2 ν 2 log( 4 |H
Revised: • if the number of samples | D | is greater than max (cid:8) 8 L 2 ν 2 log( 4 |H

Original: Φ | δ ) , 16 L (cid:48) 4 (cid:15) 2 log( 2 δ ) (cid:9) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes R ( m ) ≤ R (ˆΦ) ≤ R ( m )+ ν .
Revised: Φ | δ ) , 16 L (cid:48) 4 (cid:15) 2 log( 2 δ ) (cid:9) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes R ( m ) ≤ R (ˆΦ) ≤ R ( m )+ ν .

Original: If also ν < ˜ κ , then ˆΦ = m .
Revised: If also ν < ˜ κ , then ˆΦ = m .

Original: • if the number of samples | D | is greater than 8 L 2 ν 2 log( 2 |H
Revised: • if the number of samples | D | is greater than 8 L 2 ν 2 log( 2 |H

Original: Φ | δ ) , then with a probability at least 1 − δ , every solution Φ † to ERM satisﬁes R ( m ) ≤ R (Φ † ) ≤ R ( m ) + ν .
Revised: Φ | δ ) , then with a probability at least 1 − δ , every solution Φ † to ERM satisﬁes R ( m ) ≤ R (Φ † ) ≤ R ( m ) + ν .

Original: If also ν < ˜ κ , then Φ † = m .
Revised: If also ν < ˜ κ , then Φ † = m .

Original: Implications of Proposition 4.
Revised: Implications of Proposition 4.

Original: ERM and EIRM both asymptotically achieve the ideal OOD solution; the above proposition helps compare them in a ﬁnite sample regime.
Revised: ERM and EIRM both asymptotically achieve the ideal OOD solution; the above proposition helps compare them in a ﬁnite sample regime.

Original: The second term inside the max for EIRM, 16 L (cid:48) 4 (cid:15) 2 log( 2 δ ) , does not depend on the size of the hypothesis class.
Revised: The second term inside the max for EIRM, 16 L (cid:48) 4 (cid:15) 2 log( 2 δ ) , does not depend on the size of the hypothesis class.

Original: Hence, for large hypothesis classes, the sample complexity of EIRM equals 8 L 2 ν 2 log( 4 |H
Revised: Hence, for large hypothesis classes, the sample complexity of EIRM equals 8 L 2 ν 2 log( 4 |H

Original: Φ | δ ) .
Revised: Φ | δ ) .

Original: Consequently, the sample complexity of EIRM and ERM differs by a constant 8 L 2 ν 2 log(2) .
Revised: Consequently, the sample complexity of EIRM and ERM differs by a constant 8 L 2 ν 2 log(2) .

Original: Thus we conclude, for large hypothesis classes, both ERM and EIRM have similar sample complexity.
Revised: Thus we conclude, for large hypothesis classes, both ERM and EIRM have similar sample complexity.

Original: Next, we contrast the sample complexity of EIRM in Proposition 4, O ( 1 ν 2 ) , to Proposition 2, O (max { 1 (cid:15) 2 , 1 ν 2 } ) ; the additional covariate shift assumption in Proposition 4 helps get to a lower sample complexity of O ( 1 ν 2 ) .
Revised: Next, we contrast the sample complexity of EIRM in Proposition 4, O ( 1 ν 2 ) , to Proposition 2, O (max { 1 (cid:15) 2 , 1 ν 2 } ) ; the additional covariate shift assumption in Proposition 4 helps get to a lower sample complexity of O ( 1 ν 2 ) .

Original: In Proposition 4, we assumed square loss, but a similar result extends to cross-entropy loss as well.
Revised: In Proposition 4, we assumed square loss, but a similar result extends to cross-entropy loss as well.

Original: 3.3. D ISTRIBUTIONAL S HIFT WITH C ONFOUNDERS AND ( OR ) A NTI -C AUSAL V ARIABLES
Revised: 3.3. D ISTRIBUTIONAL S HIFT WITH C ONFOUNDERS AND ( OR ) A NTI -C AUSAL V ARIABLES

Original: In this section, we consider more general models than equation 7, which only contained cause X e and effect Y e .
Revised: In this section, we consider more general models than equation 7, which only contained cause X e and effect Y e .

Original: We also allow confounders and anti-causal variables.
Revised: We also allow confounders and anti-causal variables.

Original: However, we restrict g to polynomials.
Revised: However, we restrict g to polynomials.

Original: We start with linear models from Arjovsky et al.
Revised: We start with linear models from Arjovsky et al.

Original: Assumption 5.
Revised: Assumption 5.

Original: [Equation]
Revised: [Equation]

Original: We assume that Z 1 component of S is invertible,
Revised: We assume that Z 1 component of S is invertible,

Original: i.e. ∃ ˜ S such that ˜ S
Revised: i.e. ∃ ˜ S such that ˜ S

Original: ( S ( Z 1 , Z 2 ))
Revised: ( S ( Z 1 , Z 2 ))

Original: =
Revised: =

Original: Z 1 , and γ (cid:54) = 0 .
Revised: Z 1 , and γ (cid:54) = 0 .

Original: ∀ e ∈ E tr , π e ≥ π min |E tr | > 0 .
Revised: ∀ e ∈ E tr , π e ≥ π min |E tr | > 0 .

Original: Deﬁne Σ
Revised: Deﬁne Σ

Original: e = E
Revised: e = E

Original: [ X e X e, T ] .
Revised: [ X e X e, T ] .

Original: ∀ e ∈ E tr , Σ e is positive deﬁnite.
Revised: ∀ e ∈ E tr , Σ e is positive deﬁnite.

Original: The support of distribution of Z e =
Revised: The support of distribution of Z e =

Original: ( Z e 1 , Z e 2 ) , P eZ e , is bounded and the norm of S , (cid:107) S (cid:107) =
Revised: ( Z e 1 , Z e 2 ) , P eZ e , is bounded and the norm of S , (cid:107) S (cid:107) =

Original: σ max ( S ) (maximum singular value of S ), is also bounded.
Revised: σ max ( S ) (maximum singular value of S ), is also bounded.

Original: In the above model, Z e 1 is the cause of X e and Y e
Revised: In the above model, Z e 1 is the cause of X e and Y e

Original: but may not be directly observed.
Revised: but may not be directly observed.

Original: Z e 2 may be arbitrarily correlated with Z e 1 and (cid:15) e .
Revised: Z e 2 may be arbitrarily correlated with Z e 1 and (cid:15) e .

Original: We observe a scrambled transformation of ( Z e 1 , Z e 2 ) in
Revised: We observe a scrambled transformation of ( Z e 1 , Z e 2 ) in

Original: X e .
Revised: X e .

Original: If Z e 2 is an effect of Y e ( Z e 2 ← Y e + N e ), then Z e 2 is an anti-causal variable.
Revised: If Z e 2 is an effect of Y e ( Z e 2 ← Y e + N e ), then Z e 2 is an anti-causal variable.

Original: If H e causes both ε e ( ε e ← H e + ¯ N e ) and Z e 2 ( Z e 2 ← H e + ˜ N e ), then H e is a confounder.
Revised: If H e causes both ε e ( ε e ← H e + ¯ N e ) and Z e 2 ( Z e 2 ← H e + ˜ N e ), then H e is a confounder.

Original: In both these cases, Z e 2 is spuriously correlated with the label Y e .
Revised: In both these cases, Z e 2 is spuriously correlated with the label Y e .

Original: Consequently, the standard ERM based models estimate Z e 2 from X e and end up being biased w.r.t the desired OOD model, which does not use
Revised: Consequently, the standard ERM based models estimate Z e 2 from X e and end up being biased w.r.t the desired OOD model, which does not use

Original: Z e 2 .
Revised: Z e 2 .

Original: If Assumptions 5, 2 hold, then a linear model ˜ S
Revised: If Assumptions 5, 2 hold, then a linear model ˜ S

Original: T γ ( X e predict −−−−→ γ T ˜
Revised: T γ ( X e predict −−−−→ γ T ˜

Original: SX e =
Revised: SX e =

Original: γ T Z e 1 ) solves the
Revised: γ T Z e 1 ) solves the

Original: OOD problem in equation 1 ( Φ ∗ =
Revised: OOD problem in equation 1 ( Φ ∗ =

Original: ˜ S , m = γ T in Proposition 1) and it relies only on Z e 1 .
Revised: ˜ S , m = γ T in Proposition 1) and it relies only on Z e 1 .

Original: In the supplement (Proposition 17), we prove that ERM based models do not recover ˜ S T γ .
Revised: In the supplement (Proposition 17), we prove that ERM based models do not recover ˜ S T γ .

Original: Assumption 6.
Revised: Assumption 6.

Original: Linear general position.
Revised: Linear general position.

Original: A set of training environments E tr is said to lie in a linear general position of degree r for some r ∈ N if |E tr | > n − r + n/r and for all non-zero x ∈ R n
Revised: A set of training environments E tr is said to lie in a linear general position of degree r for some r ∈ N if |E tr | > n − r + n/r and for all non-zero x ∈ R n

Original: [Equation]
Revised: [Equation]

Original: where span is the linear span, dim is the dimension, and recall n is dimension of X e .
Revised: where span is the linear span, dim is the dimension, and recall n is dimension of X e .

Original: This assumption checks for diversity in the environments and holds almost everywhere (Arjovsky et al., 2019).
Revised: This assumption checks for diversity in the environments and holds almost everywhere (Arjovsky et al., 2019).

Original: Assumption 7.
Revised: Assumption 7.

Original: Inductive bias.
Revised: Inductive bias.

Original: H Φ is a ﬁnite set of linear models parametrized by Φ ∈ R n (output
Revised: H Φ is a ﬁnite set of linear models parametrized by Φ ∈ R n (output

Original: [Equation]
Revised: [Equation]

Original: Informally stated, the above assumption requires the OOD optimal predictor ˜ S T γ to lie in the interior of the search space and not on the boundary.
Revised: Informally stated, the above assumption requires the OOD optimal predictor ˜ S T γ to lie in the interior of the search space and not on the boundary.

Original: If Assumptions 5, 7 hold, then Assumption 3 holds.
Revised: If Assumptions 5, 7 hold, then Assumption 3 holds.

Original: Hence, we can use the bounds L and L (cid:48) on (cid:96) and ∂(cid:96)
Revised: Hence, we can use the bounds L and L (cid:48) on (cid:96) and ∂(cid:96)

Original: ( w · Φ( · ) , · ) ∂w | w =1 . 0 respectively in our next result.
Revised: ( w · Φ( · ) , · ) ∂w | w =1 . 0 respectively in our next result.

Original: Deﬁne the minimum eigenvalue across all Σ e as λ min =
Revised: Deﬁne the minimum eigenvalue across all Σ e as λ min =

Original: min e ∈E tr λ min (Σ e ) ,
Revised: min e ∈E tr λ min (Σ e ) ,

Original: [Equation]
Revised: [Equation]

Original: and τ
Revised: and τ

Original: [Equation]
Revised: [Equation]

Original: (cid:113)
Revised: (cid:113)

Original: [Equation]
Revised: [Equation]

Original: Next, we analyze how EIRM learns ˜ S T γ .
Revised: Next, we analyze how EIRM learns ˜ S T γ .

Original: Proposition 5.
Revised: Proposition 5.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: For every (cid:15) ∈ (0 , (cid:15) th ) and δ ∈ (0 , 1) , if Assumptions 5,(with r = 1 ), 7 hold and if the number of data points | D | is greater than 16 L (cid:48) 4 (cid:15) 2 log (cid:0) 2 |H
Revised: For every (cid:15) ∈ (0 , (cid:15) th ) and δ ∈ (0 , 1) , if Assumptions 5,(with r = 1 ), 7 hold and if the number of data points | D | is greater than 16 L (cid:48) 4 (cid:15) 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ( ˜ S T γ (cid:1) α , where α ∈ [
Revised: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ( ˜ S T γ (cid:1) α , where α ∈ [

Original: [Equation]
Revised: [Equation]

Original: Proof Sketch.
Revised: Proof Sketch.

Original: In learning theory it is common to analyze the concentration of empirical risks around the expected risks.
Revised: In learning theory it is common to analyze the concentration of empirical risks around the expected risks.

Original: In our case, we have a target ideal solution to equation 1 ( ˜ S T γ ) and we want our empirical solutions to concentrate around that.
Revised: In our case, we have a target ideal solution to equation 1 ( ˜ S T γ ) and we want our empirical solutions to concentrate around that.

Original: A direct ﬁnite sample approximation of equation 4 is hard to analyze.
Revised: A direct ﬁnite sample approximation of equation 4 is hard to analyze.

Original: Therefore, we introduce an intermediate problem in equationand then develop a ﬁnite sample approximation of it in equation 6.
Revised: Therefore, we introduce an intermediate problem in equationand then develop a ﬁnite sample approximation of it in equation 6.

Original: We ﬁrst show that solving equation 5 leads to solutions in the neighborhood of the target.
Revised: We ﬁrst show that solving equation 5 leads to solutions in the neighborhood of the target.

Original: To show this we use the linear general position assumption.
Revised: To show this we use the linear general position assumption.

Original: Next, we connect equation 6 and equation 5 using our new estimator for R (cid:48) and Hoeffding’s inequality.
Revised: Next, we connect equation 6 and equation 5 using our new estimator for R (cid:48) and Hoeffding’s inequality.

Original: Implications of Proposition 5.
Revised: Implications of Proposition 5.

Original: Convergence rate of ERM vs. EIRM: Recall that (cid:15) is the slack on IRM penalty R (cid:48) .
Revised: Convergence rate of ERM vs. EIRM: Recall that (cid:15) is the slack on IRM penalty R (cid:48) .

Original: If (cid:15) is sufﬁciently small and the data grows as O ( 1 (cid:15) 2 ) , every solution ˆΦ to EIRM (equation 6) is in √ (cid:15) radius of the OOD solution, i.e., (cid:107) ˆΦ − ˜ S T γ (cid:107)
Revised: If (cid:15) is sufﬁciently small and the data grows as O ( 1 (cid:15) 2 ) , every solution ˆΦ to EIRM (equation 6) is in √ (cid:15) radius of the OOD solution, i.e., (cid:107) ˆΦ − ˜ S T γ (cid:107)

Original: =
Revised: =

Original: O ( √ (cid:15) ) .
Revised: O ( √ (cid:15) ) .

Original: We contrast these rates to ones in the covariate shift setting (Section 3.3.1).
Revised: We contrast these rates to ones in the covariate shift setting (Section 3.3.1).

Original: Let E [ Y e |
Revised: Let E [ Y e |

Original: X e = x ] =
Revised: X e = x ] =

Original: Ψ T x .
Revised: Ψ T x .

Original: If the data grows as
Revised: If the data grows as

Original: O ( 1 ν 2 ) , then both ERM and EIRM solution converge to Ψ as (cid:107) ˆΦ − Ψ (cid:107) =
Revised: O ( 1 ν 2 ) , then both ERM and EIRM solution converge to Ψ as (cid:107) ˆΦ − Ψ (cid:107) =

Original: O ( √ ν ) (from Proposition 4).
Revised: O ( √ ν ) (from Proposition 4).

Original: This shows that EIRM works in more settings (Proposition 4, 5) than ERM while matching the convergence rate of ERM.
Revised: This shows that EIRM works in more settings (Proposition 4, 5) than ERM while matching the convergence rate of ERM.

Original: Sample complexity grows polynomially
Revised: Sample complexity grows polynomially

Original: in data dimension to ensure OOD generalization:
Revised: in data dimension to ensure OOD generalization:

Original: Next, we set (cid:15) = µ(cid:15) th with µ ∈ [0 , 1) , and |E tr | = 2 n (satisﬁes Assumption 6 for r = 1 ).
Revised: Next, we set (cid:15) = µ(cid:15) th with µ ∈ [0 , 1) , and |E tr | = 2 n (satisﬁes Assumption 6 for r = 1 ).

Original: A simple manipulation of terms in Proposition 5 shows that sample complexity with quadratic growth in data dimension n , O (cid:16) n 2 µ 2 log (cid:0) 2 |H
Revised: A simple manipulation of terms in Proposition 5 shows that sample complexity with quadratic growth in data dimension n , O (cid:16) n 2 µ 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1)(cid:17) , ensures ˆ Φ =
Revised: Φ | δ (cid:1)(cid:17) , ensures ˆ Φ =

Original: ( ˜ S T γ (cid:1) α with α ∈ [ 1 1+ √ µ ( √ 2 − 1) , 1 1 −√ µ ( √ 2 − 1) ] .
Revised: ( ˜ S T γ (cid:1) α with α ∈ [ 1 1+ √ µ ( √ 2 − 1) , 1 1 −√ µ ( √ 2 − 1) ] .

Original: Comparison with Proposition 2: Lastly, we contrast sample complexity of EIRM in Proposition 5, O ( 1 (cid:15) 2 ) , to Proposition 2, O
Revised: Comparison with Proposition 2: Lastly, we contrast sample complexity of EIRM in Proposition 5, O ( 1 (cid:15) 2 ) , to Proposition 2, O

Original: (max { 1 (cid:15) 2 , 1 ν 2 } ) ; the additional distributional assumptions in
Revised: (max { 1 (cid:15) 2 , 1 ν 2 } ) ; the additional distributional assumptions in

Original: Proposition 5 help arrive at a lower sample complexity of O ( 1 (cid:15) 2 ) .
Revised: Proposition 5 help arrive at a lower sample complexity of O ( 1 (cid:15) 2 ) .

Original: The bound in Proposition 2,
Revised: The bound in Proposition 2,

Original: O (max { 1 (cid:15) 2 , 1 ν 2 } ) , is larger than the one in Proposition 4, O ( 1 ν 2 ) , and Proposition 5, O ( 1 (cid:15) 2 ) , but is more general as it is agnostic to the distributional assumptions.
Revised: O (max { 1 (cid:15) 2 , 1 ν 2 } ) , is larger than the one in Proposition 4, O ( 1 ν 2 ) , and Proposition 5, O ( 1 (cid:15) 2 ) , but is more general as it is agnostic to the distributional assumptions.

Original: A simple illustration summarizing Propositions 4, 5: Set S to identity in Assumption 5.
Revised: A simple illustration summarizing Propositions 4, 5: Set S to identity in Assumption 5.

Original: Recall
Revised: Recall

Original: Z e 1 and Z e 2 from Assumption 5.
Revised: Z e 1 and Z e 2 from Assumption 5.

Original: Since S is identity X e can be written as [ X e 1 , X e 2 ] , where X e 1 =
Revised: Since S is identity X e can be written as [ X e 1 , X e 2 ] , where X e 1 =

Original: Z eand X
Revised: Z eand X

Original: e 

Revised: e

Original: =
Revised: =

Original: Z e 2 .
Revised: Z e 2 .

Original: If X e 2 ⊥ ε e , then E [ Y e | X e ] is invariant and Assumption 4 holds.
Revised: If X e 2 ⊥ ε e , then E [ Y e | X e ] is invariant and Assumption 4 holds.

Original: This corresponds to the setup in Figure 1a).
Revised: This corresponds to the setup in Figure 1a).

Original: We can now use Proposition 4 and deduce that ERM and IRM have same sample complexity and end up learning the ideal model that only uses the causal features X e 1 .
Revised: We can now use Proposition 4 and deduce that ERM and IRM have same sample complexity and end up learning the ideal model that only uses the causal features X e 1 .

Original: If
Revised: If

Original: X e 2 ← ε e + N e , then this corresponds to the setup Figure 1b), X e 1 is the cause and X e 2 is spuriously correlated with label Y e through the confounder ε e .
Revised: X e 2 ← ε e + N e , then this corresponds to the setup Figure 1b), X e 1 is the cause and X e 2 is spuriously correlated with label Y e through the confounder ε e .

Original: If X e 2 ← Y e + N e , then this corresponds to the setup in Figure 1c), X e 1 is the cause and X e 2 is anti-causally related to the label Y e .
Revised: If X e 2 ← Y e + N e , then this corresponds to the setup in Figure 1c), X e 1 is the cause and X e 2 is anti-causally related to the label Y e .

Original: In both these cases, the ideal OOD solution that solves equation 1 will only exploit X e 1 to make predictions.
Revised: In both these cases, the ideal OOD solution that solves equation 1 will only exploit X e 1 to make predictions.

Original: From
Revised: From

Original: Propositions 5, it follows that IRM when fed with O ( 1 (cid:15) 2 ) samples, it is in √ (cid:15) radius of the target
Revised: Propositions 5, it follows that IRM when fed with O ( 1 (cid:15) 2 ) samples, it is in √ (cid:15) radius of the target

Original: OOD solution, while ERM is asymptotically biased and exploits X e 2 (Proposition 17).
Revised: OOD solution, while ERM is asymptotically biased and exploits X e 2 (Proposition 17).

Original: We deﬁne a polynomial version of the model in Assumption 5.
Revised: We deﬁne a polynomial version of the model in Assumption 5.

Original: We only need to change Y e ← γ T ( Z e 1 ) + ε e to Y e ← γ t ζ p ( Z e 1 ) + ε e .
Revised: We only need to change Y e ← γ T ( Z e 1 ) + ε e to Y e ← γ t ζ p ( Z e 1 ) + ε e .

Original: ζ p is a polynomial feature map of degree p deﬁned as ζ p : R c
Revised: ζ p is a polynomial feature map of degree p deﬁned as ζ p : R c

Original: → R c (cid:48) , where c is the dimension of the input Z e 1 , ζ p ( W )
Revised: → R c (cid:48) , where c is the dimension of the input Z e 1 , ζ p ( W )

Original: = (cid:0) W, W ⊗ W, . . .
Revised: = (cid:0) W, W ⊗ W, . . .

Original: , ( W ⊗
Revised: , ( W ⊗

Original: W... p times ⊗ W ) (cid:1) =
Revised: W... p times ⊗ W ) (cid:1) =

Original: (cid:0) ( W ⊗ i ) pi =1 (cid:1) and ⊗ is the Kronecker product.
Revised: (cid:0) ( W ⊗ i ) pi =1 (cid:1) and ⊗ is the Kronecker product.

Original: Also, c (cid:48) =
Revised: Also, c (cid:48) =

Original: (cid:80) pi
Revised: (cid:80) pi

Original: =1 c i .
Revised: =1 c i .

Original: Can we directly use the analysis from the linear case by transforming X e appropriately?
Revised: Can we directly use the analysis from the linear case by transforming X e appropriately?

Original: No, we ﬁrst need to ﬁnd an appropriate transformation for the scrambling matrix S that satisﬁes the conditions (invertibiltiy) in Assumption 5 while maintaining a linear relationship between transformations of
Revised: No, we ﬁrst need to ﬁnd an appropriate transformation for the scrambling matrix S that satisﬁes the conditions (invertibiltiy) in Assumption 5 while maintaining a linear relationship between transformations of

Original: X e and Z e .
Revised: X e and Z e .

Original: We present the main result informally below (details are in the supplement).
Revised: We present the main result informally below (details are in the supplement).

Original: Proposition 6.
Revised: Proposition 6.

Original: (Informal statement) For sufﬁciently small (cid:15) and δ ∈ (0 , 1) , if Assumptions similar to
Revised: (Informal statement) For sufﬁciently small (cid:15) and δ ∈ (0 , 1) , if Assumptions similar to

Original: Proposition 5 hold and | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 2 |H
Revised: Proposition 5 hold and | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 2 |H

Original: Φ | δ ) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ¯˜ S T γ ( α ) , where ¯˜ S T γ is the OOD optimal model (deﬁned in the supplement), α ∈ [
Revised: Φ | δ ) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ¯˜ S T γ ( α ) , where ¯˜ S T γ is the OOD optimal model (deﬁned in the supplement), α ∈ [

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Insights from the polynomial case and inﬁnite hypothesis case.
Revised: Insights from the polynomial case and inﬁnite hypothesis case.

Original: In the polynomial case, we adapt the linear general position Assumption 6, the number of environments |E tr | are now required to grow as O ( n p ) .
Revised: In the polynomial case, we adapt the linear general position Assumption 6, the number of environments |E tr | are now required to grow as O ( n p ) .

Original: As a result, in the sample complexity analysis we discussed, we replace n with n p to obtain that a sample complexity of O (cid:16) n 2 p µ 2 log (cid:0) 2 |H
Revised: As a result, in the sample complexity analysis we discussed, we replace n with n p to obtain that a sample complexity of O (cid:16) n 2 p µ 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1)(cid:17) ensures ˆΦ = ¯˜ S T γ ( α ) with α ∈ [ 1 1+ √ µ ( √ 2 − 1) , 1 1 −√ µ ( √ 2 − 1) ] .
Revised: Φ | δ (cid:1)(cid:17) ensures ˆΦ = ¯˜ S T γ ( α ) with α ∈ [ 1 1+ √ µ ( √ 2 − 1) , 1 1 −√ µ ( √ 2 − 1) ] .

Original: In the inﬁnite hypothesis case, the main change in the results is that we replace |H
Revised: In the inﬁnite hypothesis case, the main change in the results is that we replace |H

Original: Φ | with an appropriate model complexity metric (Shalev-Shwartz & Ben-David, 2014).
Revised: Φ | with an appropriate model complexity metric (Shalev-Shwartz & Ben-David, 2014).

Original: Consider Proposition 5, a sample complexity of O (cid:0) n 3 µ 2 log (cid:0) nµ (cid:1)(cid:1) ensures ˆΦ =
Revised: Consider Proposition 5, a sample complexity of O (cid:0) n 3 µ 2 log (cid:0) nµ (cid:1)(cid:1) ensures ˆΦ =

Original: ( ˜ S T γ (cid:1) α with α ∈ [ 1 1+ √ µ ( √ 2 − 1) , 1 1 − √ µ ( √ 2 − 1) ] in contrast to O (cid:16) n 2 µ 2 log (cid:0) 2 |H
Revised: ( ˜ S T γ (cid:1) α with α ∈ [ 1 1+ √ µ ( √ 2 − 1) , 1 1 − √ µ ( √ 2 − 1) ] in contrast to O (cid:16) n 2 µ 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1) (cid:17) in the ﬁnite hypothesis case.
Revised: Φ | δ (cid:1) (cid:17) in the ﬁnite hypothesis case.

Original: We showed the beneﬁts of IRM for polynomial models and other extensions (non-linear S ) are future work.
Revised: We showed the beneﬁts of IRM for polynomial models and other extensions (non-linear S ) are future work.

Original: In the supplement, we provide a dialogue explaining how our work ﬁts in the big picture.
Revised: In the supplement, we provide a dialogue explaining how our work ﬁts in the big picture.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we discuss classiﬁcation experiments (regression experiments with similar qualitative ﬁndings are in the supplement).
Revised: In this section, we discuss classiﬁcation experiments (regression experiments with similar qualitative ﬁndings are in the supplement).

Original: We introduce three new variants of the colored MNIST (CMNIST) dataset in (Arjovsky et al., 2019).
Revised: We introduce three new variants of the colored MNIST (CMNIST) dataset in (Arjovsky et al., 2019).

Original: We divide the training data in MNIST digits into two environments ( e = 1 , 2 ) equally and the testing data in MNIST digits is assigned to another environment ( e = 3 ).
Revised: We divide the training data in MNIST digits into two environments ( e = 1 , 2 ) equally and the testing data in MNIST digits is assigned to another environment ( e = 3 ).

Original: X eg : gray scale image of the digit, Y eg : label of the gray scale digit (digits ≥ 5 have Y eg = 1 and digits < 5 have Y eg = 0 ).
Revised: X eg : gray scale image of the digit, Y eg : label of the gray scale digit (digits ≥ 5 have Y eg = 1 and digits < 5 have Y eg = 0 ).

Original: X e : ﬁnal colored image and Y e : ﬁnal label are generated as follows.
Revised: X e : ﬁnal colored image and Y e : ﬁnal label are generated as follows.

Original: Deﬁne
Revised: Deﬁne

Original: Bernoulli variables G , N , N e that take a value 1 with probability θ , β and β e and 0 otherwise.
Revised: Bernoulli variables G , N , N e that take a value 1 with probability θ , β and β e and 0 otherwise.

Original: Deﬁne a color variable C e , where C e = 0 is red and C e
Revised: Deﬁne a color variable C e , where C e = 0 is red and C e

Original: = 1 is green.
Revised: = 1 is green.

Original: Let ⊕ denotes xor operation.
Revised: Let ⊕ denotes xor operation.

Original: Y eg ← L ( X eg ) , L : Human labeling ,
Revised: Y eg ← L ( X eg ) , L : Human labeling ,

Original: Y e ← L ( X eg ) ⊕ N, Corrupt the original labels with noise
Revised: Y e ← L ( X eg ) ⊕ N, Corrupt the original labels with noise

Original: C e ← G ( Y e ⊕ N e ) + (1 − G )( N ⊕ N e ) , Use G to select b/w anti-causal or confounded
Revised: C e ← G ( Y e ⊕ N e ) + (1 − G )( N ⊕ N e ) , Use G to select b/w anti-causal or confounded

Original: X e ← T ( X g , C e ) , T : transformation to color the image (10)
Revised: X e ← T ( X g , C e ) , T : transformation to color the image (10)

Original: If the probability θ = 1 , then G = 1 and that gives us back the original CMNIST in (Arjovsky et al., 2019), which we call anti-causal CMNIST (AC-CMNIST).
Revised: If the probability θ = 1 , then G = 1 and that gives us back the original CMNIST in (Arjovsky et al., 2019), which we call anti-causal CMNIST (AC-CMNIST).

Original: If θ = 0 , then we get confounded colored MNIST (CF-CMNIST).
Revised: If θ = 0 , then we get confounded colored MNIST (CF-CMNIST).

Original: If 0 < θ < 1 , we get a hybrid dataset (HB-CMNIST).
Revised: If 0 < θ < 1 , we get a hybrid dataset (HB-CMNIST).

Original: The above model in equation 10 has features of the model in Assumption 5, where X eg , C e , L , T take the role of Z e 1 , Z e 2 , γ , S .
Revised: The above model in equation 10 has features of the model in Assumption 5, where X eg , C e , L , T take the role of Z e 1 , Z e 2 , γ , S .

Original: We set the noise N parameter β = 0 .
Revised: We set the noise N parameter β = 0 .

Original: 25 , and the parameter for N e in the three environments [ β 1 , β 2 , β 3 ] =
Revised: 25 , and the parameter for N e in the three environments [ β 1 , β 2 , β 3 ] =

Original: Color is spuriously correlated with the label; P ( C e = 1 | Y e = 1) varies drastically in the three environments ( [0 . 9 , 0 . 8 , 0 . 1] for AC-CMNIST).
Revised: Color is spuriously correlated with the label; P ( C e = 1 | Y e = 1) varies drastically in the three environments ( [0 . 9 , 0 . 8 , 0 . 1] for AC-CMNIST).

Original: In the variants of CMNIST we discussed, P ( Y e | X e ) varies across the environments.
Revised: In the variants of CMNIST we discussed, P ( Y e | X e ) varies across the environments.

Original: We now deﬁne a covariate shift based CMNIST (CS-CMNIST) ( P ( Y e | X e ) is invariant).
Revised: We now deﬁne a covariate shift based CMNIST (CS-CMNIST) ( P ( Y e | X e ) is invariant).

Original: We use selection bias to induce spurious correlations.
Revised: We use selection bias to induce spurious correlations.

Original: Generate a color C e uniformly at random.
Revised: Generate a color C e uniformly at random.

Original: Select the pair ( X eg , C e ) with probability 1 − ψ e if the label Y eg and C e are the same, else select them with a probability ψ e .
Revised: Select the pair ( X eg , C e ) with probability 1 − ψ e if the label Y eg and C e are the same, else select them with a probability ψ e .

Original: If the pair is selected, then color the image X e ← T ( X g , C e ) and Y e ← Y eg .
Revised: If the pair is selected, then color the image X e ← T ( X g , C e ) and Y e ← Y eg .

Original: Selection probability ψ e for the three environments are [ ψ 1 , ψ 2 , ψ 3 ] =
Revised: Selection probability ψ e for the three environments are [ ψ 1 , ψ 2 , ψ 3 ] =

Original: Due to the selection bias, color is spuriously correlated, P ( C e = 1 |
Revised: Due to the selection bias, color is spuriously correlated, P ( C e = 1 |

Original: Y e = 1) varies drastically [0 . 9 , 0 . 8 , 0 . 1] .
Revised: Y e = 1) varies drastically [0 . 9 , 0 . 8 , 0 . 1] .

Original: We provide the graphical models for the CMNIST variants and computations of P ( C e | Y e ) , P ( Y e | X e ) in the supplement.
Revised: We provide the graphical models for the CMNIST variants and computations of P ( C e | Y e ) , P ( Y e | X e ) in the supplement.

Original: [Figure]
Revised: [Figure]

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: We use the ﬁrst two environments ( e = 1 , 2 ) to train and third environment ( e = 3 ) to test.
Revised: We use the ﬁrst two environments ( e = 1 , 2 ) to train and third environment ( e = 3 ) to test.

Original: Other details of the training (models, hyperparameters, etc.) are in the supplement.
Revised: Other details of the training (models, hyperparameters, etc.) are in the supplement.

Original: For each of the above datasets, we run the experiments for different amounts of training data from 1000 to up tosamples (10 trials for each data size).
Revised: For each of the above datasets, we run the experiments for different amounts of training data from 1000 to up tosamples (10 trials for each data size).

Original: In Figure 2, we compare the models trained using IRM and
Revised: In Figure 2, we compare the models trained using IRM and

Original: ERM in terms of the classiﬁcation error on the test environment e = 3 (a poor performance indicates model exploits the color) for varying number of train samples.
Revised: ERM in terms of the classiﬁcation error on the test environment e = 3 (a poor performance indicates model exploits the color) for varying number of train samples.

Original: We also provide the performance of the ideal hypothetical optimal invariant model.
Revised: We also provide the performance of the ideal hypothetical optimal invariant model.

Original: Observe that except for in the covariate shift setting where IRM and ERM are similar as seen in Figure 2a (as predicted from Proposition 4), IRM outperforms ERM in the remaining three datasets (as predicted from Proposition 5) as seen in Figure
Revised: Observe that except for in the covariate shift setting where IRM and ERM are similar as seen in Figure 2a (as predicted from Proposition 4), IRM outperforms ERM in the remaining three datasets (as predicted from Proposition 5) as seen in Figure

Original: 2b-d. We further validate this claim through the regression experiments provided in the supplement.
Revised: 2b-d. We further validate this claim through the regression experiments provided in the supplement.

Original: In CF-CMNIST, IRM achievs an error of 0 . 45 , which is much better than error of ERM (0 . 7) but is marginally better than a random guess.
Revised: In CF-CMNIST, IRM achievs an error of 0 . 45 , which is much better than error of ERM (0 . 7) but is marginally better than a random guess.

Original: This suggests that confounder induced spurious correlations are harder to mitigate and may need more samples than needed in anti-causal case (AC-CMNIST).
Revised: This suggests that confounder induced spurious correlations are harder to mitigate and may need more samples than needed in anti-causal case (AC-CMNIST).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a sample complexity analysis of IRM to answer the question: when is IRM better than
Revised: We presented a sample complexity analysis of IRM to answer the question: when is IRM better than

Original: ERM (and vice-versa)?
Revised: ERM (and vice-versa)?

Original: For distribution shifts such as the covariate shifts, we proved that both IRM and ERM have similar sample complexity and arrive at the desired OOD solution asymptotically.
Revised: For distribution shifts such as the covariate shifts, we proved that both IRM and ERM have similar sample complexity and arrive at the desired OOD solution asymptotically.

Original: For distribution shifts involving confounders and (or) anti-causal variables and polynomial generative models, we proved that IRM is guaranteed to achieve the desired OOD solution while ERM can be asymptotically biased.
Revised: For distribution shifts involving confounders and (or) anti-causal variables and polynomial generative models, we proved that IRM is guaranteed to achieve the desired OOD solution while ERM can be asymptotically biased.

Original: We proposed new variants of original colored MNIST dataset from Arjovsky et al.
Revised: We proposed new variants of original colored MNIST dataset from Arjovsky et al.

Original: (2019), which are more comprehensive and better capture how spurious correlations occur in reality.
Revised: (2019), which are more comprehensive and better capture how spurious correlations occur in reality.

Original: To the best of our knowledge, we believe this to be the ﬁrst work that provides a rigorous characterization of impact of factors such as model complexity, number of environments on the sample complexity and its distance from the OOD solution in distribution shifts that go beyond covariate shifts.
Revised: To the best of our knowledge, we believe this to be the ﬁrst work that provides a rigorous characterization of impact of factors such as model complexity, number of environments on the sample complexity and its distance from the OOD solution in distribution shifts that go beyond covariate shifts.

Original:  A CKNOWLEDGEMENTS
Revised:  A CKNOWLEDGEMENTS

Original: This work was supported in part by the Rensselaer-IBM AI Research Collaboration (part of the IBM
Revised: This work was supported in part by the Rensselaer-IBM AI Research Collaboration (part of the IBM

Original: AI Horizons Network).
Revised: AI Horizons Network).

Original:  SUPPLEMENT 7. D IALOGUE ON IRM C ONTINUED
Revised:  SUPPLEMENT 7. D IALOGUE ON IRM C ONTINUED

Original: [ In the original IRM paper by Arjovsky et al.
Revised: [ In the original IRM paper by Arjovsky et al.

Original: (2019), we left two graduate students E RIC
Revised: (2019), we left two graduate students E RIC

Original: and I RMA strolling along the Parisian streets on a warm summer evening.
Revised: and I RMA strolling along the Parisian streets on a warm summer evening.

Original: A lot has transpired since then.
Revised: A lot has transpired since then.

Original: E RIC is now holed up inside his room at Cit´e Universitaire and I RMA has returned to her parents’ home in Provence.
Revised: E RIC is now holed up inside his room at Cit´e Universitaire and I RMA has returned to her parents’ home in Provence.

Original: The two are talking through a Zoom call. ]
Revised: The two are talking through a Zoom call. ]

Original: • E RIC :
Revised: • E RIC :

Original: After reading Arjovsky et al.
Revised: After reading Arjovsky et al.

Original: and Gulrajani & Lopez-Paz (2020), I was not sure of how to understand the settings where IRM is beneﬁcial over ERM and vice-versa?
Revised: and Gulrajani & Lopez-Paz (2020), I was not sure of how to understand the settings where IRM is beneﬁcial over ERM and vice-versa?

Original: From
Revised: From

Original: Gulrajani & Lopez-Paz (
Revised: Gulrajani & Lopez-Paz (

Original: ), I understood that ERM
Revised: ), I understood that ERM

Original: continues to
Revised: continues to

Original: be
Revised: be

Original: the state-ofthe-art if
Revised: the state-ofthe-art if

Original: sufﬁcient care is taken in performing
Revised: sufﬁcient care is taken in performing

Original: model selection
Revised: model selection

Original: • I RMA :
Revised: • I RMA :

Original: But after reading this manuscript I understand that there are settings where no matter how one selects the model, ERM is bound to be at a disadvantage.
Revised: But after reading this manuscript I understand that there are settings where no matter how one selects the model, ERM is bound to be at a disadvantage.

Original: • E RIC : What you say seems to contradict my understanding of Gulrajani & Lopez-Paz (2020).
Revised: • E RIC : What you say seems to contradict my understanding of Gulrajani & Lopez-Paz (2020).

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Actually, they ...
Revised: Actually, they ...

Original: [I RMA ’s audio and video freeze for a minute, leaving E
Revised: [I RMA ’s audio and video freeze for a minute, leaving E

Original: RIC
Revised: RIC

Original: to ponder this conundrum for a little bit on his own.
Revised: to ponder this conundrum for a little bit on his own.

Original: I
Revised: I

Original: RMA
Revised: RMA

Original: stops
Revised: stops

Original: her
Revised: her

Original: video and continues only with audio and the conversation is able to resume. ]
Revised: video and continues only with audio and the conversation is able to resume. ]

Original: • I RMA :
Revised: • I RMA :

Original: Sorry, how much did you hear?
Revised: Sorry, how much did you hear?

Original: • E RIC : You were just starting to explain why ERM can be at a disadvantage, but I didn’t hear anything after that.
Revised: • E RIC : You were just starting to explain why ERM can be at a disadvantage, but I didn’t hear anything after that.

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Okay, let me start my explanation again.
Revised: Okay, let me start my explanation again.

Original: There is no contradiction.
Revised: There is no contradiction.

Original: The current manuscript says that in covariate shift settings, there maybe no clear winner.
Revised: The current manuscript says that in covariate shift settings, there maybe no clear winner.

Original: Many of the datasets considered in Gulrajani & Lopez-Paz (2020) are perhaps similar to covariate shift settings.
Revised: Many of the datasets considered in Gulrajani & Lopez-Paz (2020) are perhaps similar to covariate shift settings.

Original: Also, there could be another reason why IRM did not outperform ERM in Gulrajani & Lopez-Paz (2020) and I will come to it in a bit.
Revised: Also, there could be another reason why IRM did not outperform ERM in Gulrajani & Lopez-Paz (2020) and I will come to it in a bit.

Original: • E RIC : Yes, you are right!
Revised: • E RIC : Yes, you are right!

Original: The datasets are human labeled images if I recall correctly.
Revised: The datasets are human labeled images if I recall correctly.

Original: In these datasets it’s safe to assume P ( Y e | X e ) is close to invariant across domains (as long as the cohort of humans in the different domains are not very different).
Revised: In these datasets it’s safe to assume P ( Y e | X e ) is close to invariant across domains (as long as the cohort of humans in the different domains are not very different).

Original: What happens when P ( Y e | X e ) varies a lot?
Revised: What happens when P ( Y e | X e ) varies a lot?

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Yes, when P ( Y e | X e ) varies a lot, IRM can be at an advantage over ERM.
Revised: Yes, when P ( Y e | X e ) varies a lot, IRM can be at an advantage over ERM.

Original: In this manuscript, I learned that when data generation involves confounders or anti-causal variables, it is possible to show that IRM methods converge to desirable OOD solutions with good convergence rates.
Revised: In this manuscript, I learned that when data generation involves confounders or anti-causal variables, it is possible to show that IRM methods converge to desirable OOD solutions with good convergence rates.

Original: • E RIC :
Revised: • E RIC :

Original: Interesting!
Revised: Interesting!

Original: I should read that.
Revised: I should read that.

Original: The experiment on colored MNIST by Arjovsky et al.
Revised: The experiment on colored MNIST by Arjovsky et al.

Original: (2019) contained anti-causal variables right?
Revised: (2019) contained anti-causal variables right?

Original: Is that the reason why IRM performed better than ERM
Revised: Is that the reason why IRM performed better than ERM

Original: ? • I
Revised: ? • I

Original: RMA :
Revised: RMA :

Original: Yes, you are right.
Revised: Yes, you are right.

Original: In the current manuscript, the authors also develop other variants of colored MNIST (covariate shift based, confounder based, anti-causal based, and hybrid of confounder and anti-causal).
Revised: In the current manuscript, the authors also develop other variants of colored MNIST (covariate shift based, confounder based, anti-causal based, and hybrid of confounder and anti-causal).

Original: IRM retains its advantage on all but the ﬁrst dataset.
Revised: IRM retains its advantage on all but the ﬁrst dataset.

Original: I liked that IRM has advantage over ERM in confounded datasets as I believe many real datasets can have confounders generating spurious correlations.
Revised: I liked that IRM has advantage over ERM in confounded datasets as I believe many real datasets can have confounders generating spurious correlations.

Original: • E RIC :
Revised: • E RIC :

Original: Wow!
Revised: Wow!

Original: These different variants of CMNIST sound exciting.
Revised: These different variants of CMNIST sound exciting.

Original: Going back to the covariate shift case, do you think that we can perhaps come up with a ﬁner criterion to say when IRM is better than ERM and vice-versa?
Revised: Going back to the covariate shift case, do you think that we can perhaps come up with a ﬁner criterion to say when IRM is better than ERM and vice-versa?

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Yes, actually I have been thinking about this problem and would be happy to share my initial thoughts.
Revised: Yes, actually I have been thinking about this problem and would be happy to share my initial thoughts.

Original: Identifying a representation Φ ∗ that leads to invariant conditional distributions is crucial to the success of IRM.
Revised: Identifying a representation Φ ∗ that leads to invariant conditional distributions is crucial to the success of IRM.

Original: A subtle factor that is implicitly assumed is representations obtained from multiple domains should overlap.
Revised: A subtle factor that is implicitly assumed is representations obtained from multiple domains should overlap.

Original: • E RIC : Can you clarify what you mean by overlap?
Revised: • E RIC : Can you clarify what you mean by overlap?

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Sure!
Revised: Sure!

Original: Consider the dataset from domain 1, say photographs of birds, and domain 2, say sketches of birds.
Revised: Consider the dataset from domain 1, say photographs of birds, and domain 2, say sketches of birds.

Original: Imagine I have access to the oracle representation Φ ∗ .
Revised: Imagine I have access to the oracle representation Φ ∗ .

Original: Now I pass the data from domain 1 and 2 through Φ ∗ to get the representations.
Revised: Now I pass the data from domain 1 and 2 through Φ ∗ to get the representations.

Original: If these representations from the two domains live in very different parts of the representation space, then we cannot hope that IRM will offer any advantage.
Revised: If these representations from the two domains live in very different parts of the representation space, then we cannot hope that IRM will offer any advantage.

Original: This is the other explanation, which I was going to come to, why IRM did not outperform ERM in Gulrajani & Lopez-Paz (2020).
Revised: This is the other explanation, which I was going to come to, why IRM did not outperform ERM in Gulrajani & Lopez-Paz (2020).

Original: • E RIC :
Revised: • E RIC :

Original: How about when there is a complete overlap?
Revised: How about when there is a complete overlap?

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Yes, if there is a strong or complete overlap in the representations from the two domains, then it is possible that IRM can help.
Revised: Yes, if there is a strong or complete overlap in the representations from the two domains, then it is possible that IRM can help.

Original: In fact I believe in such settings, if the ERM models trained on the two domains disagree a lot, then that can be a strong indication towards the models heavily exploiting spurious correlations.
Revised: In fact I believe in such settings, if the ERM models trained on the two domains disagree a lot, then that can be a strong indication towards the models heavily exploiting spurious correlations.

Original: In such cases, I believe IRM can again offer some advantage.
Revised: In such cases, I believe IRM can again offer some advantage.

Original: • E RIC : I will try to put these ideas down on paper and meet you again for a discussion.
Revised: • E RIC : I will try to put these ideas down on paper and meet you again for a discussion.

Original: • I
Revised: • I

Original: RMA :
Revised: RMA :

Original: Great!
Revised: Great!

Original: I truly hope that it can be in person at the caf´e in Palais-Royal where we ﬁrst started this conversation.
Revised: I truly hope that it can be in person at the caf´e in Palais-Royal where we ﬁrst started this conversation.

Original: So long for now!
Revised: So long for now!

Original: [ The students end their call. ]
Revised: [ The students end their call. ]

Original: 7. S UPPLEMENTARY M ATERIALS FOR E XPERIMENTS
Revised: 7. S UPPLEMENTARY M ATERIALS FOR E XPERIMENTS

Original: In this section, we cover the supplementary materials for the experiments.
Revised: In this section, we cover the supplementary materials for the experiments.

Original: The code to reproduce the results presented in this work can be found at https://github.com/IBM/OoD .
Revised: The code to reproduce the results presented in this work can be found at https://github.com/IBM/OoD .

Original:  C LASSIFICATION
Revised:  C LASSIFICATION

Original: We ﬁrst describe the model and other training details.
Revised: We ﬁrst describe the model and other training details.

Original: Choice of H Φ and other training details We use the same architecture for both ERM and IRM.
Revised: Choice of H Φ and other training details We use the same architecture for both ERM and IRM.

Original: We choose the architecture that was used in Arjovsky et al.
Revised: We choose the architecture that was used in Arjovsky et al.

Original: (2019): 2 layer MLP.
Revised: (2019): 2 layer MLP.

Original: The ﬁrst two layer consists of 390 hidden nodes, and the output layer has two nodes (for the two classes).
Revised: The ﬁrst two layer consists of 390 hidden nodes, and the output layer has two nodes (for the two classes).

Original: We use
Revised: We use

Original: ReLU activation in each layer, a regularization weight of 0.0011 is used for each layer.
Revised: ReLU activation in each layer, a regularization weight of 0.0011 is used for each layer.

Original: We use a learning rate of 4.9e-4, batch size of 512 for both ERM and IRM.
Revised: We use a learning rate of 4.9e-4, batch size of 512 for both ERM and IRM.

Original: We use 1000 gradient steps for
Revised: We use 1000 gradient steps for

Original: IRM.
Revised: IRM.

Original: As was done in the original IRM work (Arjovsky et al., 2019), we use a threshold on steps (190) after which a large penalty is imposed for violating the IRM constraint.
Revised: As was done in the original IRM work (Arjovsky et al., 2019), we use a threshold on steps (190) after which a large penalty is imposed for violating the IRM constraint.

Original: We use the train domain validation set procedure described in Gulrajani & Lopez-Paz (2020) to select the penalty value from the set { 1 e 4 , 3 . 3 e 4 , 6 . 6 e 4 , 1 e 5 } (with 4:1 train-validation split).
Revised: We use the train domain validation set procedure described in Gulrajani & Lopez-Paz (2020) to select the penalty value from the set { 1 e 4 , 3 . 3 e 4 , 6 . 6 e 4 , 1 e 5 } (with 4:1 train-validation split).

Original: With the same learning rate, we observed ERM was slower at learning than IRM.
Revised: With the same learning rate, we observed ERM was slower at learning than IRM.

Original: To ensure ERM always converges we set the number of epochs to a very high value 100 ( 118 k steps).
Revised: To ensure ERM always converges we set the number of epochs to a very high value 100 ( 118 k steps).

Original: A. Covariate shift based for CMNIST
Revised: A. Covariate shift based for CMNIST

Original: We provide the generative model for CS-CMNIST below.
Revised: We provide the generative model for CS-CMNIST below.

Original: [Equation]
Revised: [Equation]

Original: A.1.
Revised: A.1.

Original: Compute P ( C e | Y e ) and P ( Y e | X e ) .
Revised: Compute P ( C e | Y e ) and P ( Y e | X e ) .

Original: We compute P ( C e | Y e ) and P ( Y e | X e ) for the covariate shift based CMNIST described by equation 11.
Revised: We compute P ( C e | Y e ) and P ( Y e | X e ) for the covariate shift based CMNIST described by equation 11.

Original: P ( C e | Y e ) helps us understand how the spurious correlations vary across the environments.
Revised: P ( C e | Y e ) helps us understand how the spurious correlations vary across the environments.

Original: P ( Y e | X e ) helps us understand if the covariate shift condition is satisﬁed or not.
Revised: P ( Y e | X e ) helps us understand if the covariate shift condition is satisﬁed or not.

Original: Compute the probability P
Revised: Compute the probability P

Original: ( C e | Y e = 1) =
Revised: ( C e | Y e = 1) =

Original: P ( C e | Y eg
Revised: P ( C e | Y eg

Original: = 1 , U e = 1) as follows.
Revised: = 1 , U e = 1) as follows.

Original: [Equation]
Revised: [Equation]

Original: From the above simpliﬁcation we gather that P ( C e = 1 | Y e = 1) is 0 . 9 , 0 . 8 and 0 .
Revised: From the above simpliﬁcation we gather that P ( C e = 1 | Y e = 1) is 0 . 9 , 0 . 8 and 0 .

Original: 1 in the three environments.
Revised: 1 in the three environments.

Original: Deﬁne p l = P ( Y eg = 1 | X eg ) .
Revised: Deﬁne p l = P ( Y eg = 1 | X eg ) .

Original: In MNIST data it is reasonable to assume deterministic labeling, i.e. p l = 1 or p l = 0 .
Revised: In MNIST data it is reasonable to assume deterministic labeling, i.e. p l = 1 or p l = 0 .

Original: From the way the data is constructed we can assume that T is invertible, i.e. from colored image
Revised: From the way the data is constructed we can assume that T is invertible, i.e. from colored image

Original: X e we can get back the grayscale image and the color X eg , C e .
Revised: X e we can get back the grayscale image and the color X eg , C e .

Original: We now move to computing P ( Y e | X e ) .
Revised: We now move to computing P ( Y e | X e ) .

Original: We assume Y eg
Revised: We assume Y eg

Original: = 1 in the simplifcation below.
Revised: = 1 in the simplifcation below.

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Observe that under standard assumption of deterministic labeling p l = 1 , P ( Y e = 1 | X e ) remains invariant and for high values of p l it is relatively stable across the environments.
Revised: Observe that under standard assumption of deterministic labeling p l = 1 , P ( Y e = 1 | X e ) remains invariant and for high values of p l it is relatively stable across the environments.

Original: A.2 Graphical model for covariate shift based CMNIST.
Revised: A.2 Graphical model for covariate shift based CMNIST.

Original: In Figure 3, we provide the graphical model for covariate shift based CMNIST described in equation 11.
Revised: In Figure 3, we provide the graphical model for covariate shift based CMNIST described in equation 11.

Original: A
Revised: A

Original: . Results with numerical values and
Revised: . Results with numerical values and

Original: standard errors.
Revised: standard errors.

Original: In Table 2, we provide the numerical values for the results showed in the Figure 2 a) along with the standard errors.
Revised: In Table 2, we provide the numerical values for the results showed in the Figure 2 a) along with the standard errors.

Original: B. Colored MNIST with anti-causal variables (AC-CMNIST)
Revised: B. Colored MNIST with anti-causal variables (AC-CMNIST)

Original: B.1.
Revised: B.1.

Original: Compute P ( C e | Y e ) and P ( Y e | X e ) .
Revised: Compute P ( C e | Y e ) and P ( Y e | X e ) .

Original: We ﬁrst compute P ( C e | Y e ) .
Revised: We ﬁrst compute P ( C e | Y e ) .

Original: P ( C e = 1 | Y e = 1) =
Revised: P ( C e = 1 | Y e = 1) =

Original: P ( Y e ⊕ N e = 1 | Y e = 1) = P ( N e = 

Revised: P ( Y e ⊕ N e = 1 | Y e = 1) = P ( N e =

Original: | Y e = 1)
Revised: | Y e = 1)

Original: = β e .
Revised: = β e .

Original: Therefore, P ( C e = 1 | Y e = 1) is 0 .
Revised: Therefore, P ( C e = 1 | Y e = 1) is 0 .

Original: 9 , 0 .and 0 .
Revised: 9 , 0 .and 0 .

Original: 1 in environments 1 , 2 , and 3 respectively.
Revised: 1 in environments 1 , 2 , and 3 respectively.

Original: Next, we compute P ( Y e | X e ) .
Revised: Next, we compute P ( Y e | X e ) .

Original: We assume Y eg
Revised: We assume Y eg

Original: =
Revised: =

Original: in the simplﬁcation below.
Revised: in the simplﬁcation below.

Original: We also assume deteministic labeling.
Revised: We also assume deteministic labeling.

Original: In the simplﬁcation that follows we use β = 0 . 25 .
Revised: In the simplﬁcation that follows we use β = 0 . 25 .

Original: [Table]
Revised: [Table]

Original: P ( Y e = 1 , C e = 0 | X eg ) = P ( Y e = 1 , C e = 0 | X eg ) =
Revised: P ( Y e = 1 , C e = 0 | X eg ) = P ( Y e = 1 , C e = 0 | X eg ) =

Original: P ( Y e = 1 | Y eg = 1) P ( C e = 

Revised: P ( Y e = 1 | Y eg = 1) P ( C e =

Original: | Y e = 1) = 0 .
Revised: | Y e = 1) = 0 .

Original: 75 β e
Revised: 75 β e

Original: P ( Y e = 0 , C e = 0 | X eg ) =
Revised: P ( Y e = 0 , C e = 0 | X eg ) =

Original: P ( Y e = 1 , C e = 0 | X eg ) =
Revised: P ( Y e = 1 , C e = 0 | X eg ) =

Original: P ( Y e = 

Revised: P ( Y e =

Original: | Y eg = 1) P ( C e = 

Revised: | Y eg = 1) P ( C e =

Original: | Y e = 0)
Revised: | Y e = 0)

Original: = 0 .
Revised: = 0 .

Original: 25(1 − β e )
Revised: 25(1 − β e )

Original: P ( Y e = 1 | X e ) =
Revised: P ( Y e = 1 | X e ) =

Original: P ( Y e = 1 |
Revised: P ( Y e = 1 |

Original: X eg , C e = 0) = 3 β e 2 β e +
Revised: X eg , C e = 0) = 3 β e 2 β e +

Original: P ( Y e = 1 | X e ) = 0 .
Revised: P ( Y e = 1 | X e ) = 0 .

Original: 25 (For environment 1)
Revised: 25 (For environment 1)

Original: P ( Y e = 1 | X e ) = 0 .
Revised: P ( Y e = 1 | X e ) = 0 .

Original: 428 (For environment 2)
Revised: 428 (For environment 2)

Original: P ( Y e = 1 | X e ) = 0 .
Revised: P ( Y e = 1 | X e ) = 0 .

Original: 964 (For environment 2) (14)
Revised: 964 (For environment 2) (14)

Original: B.2 Graphical model for anti-causal CMNIST In Figure 4, we provide the graphical model for
Revised: B.2 Graphical model for anti-causal CMNIST In Figure 4, we provide the graphical model for

Original: AC-CMNIST described in equation 10 (for G = 1 ).
Revised: AC-CMNIST described in equation 10 (for G = 1 ).

Original: B.3. Results with
Revised: B.3. Results with

Original: numerical values and standard errors.
Revised: numerical values and standard errors.

Original: In Table 3, we provide the numerical values for the results showed in the Figure 2 c) along with the standard errors.
Revised: In Table 3, we provide the numerical values for the results showed in the Figure 2 c) along with the standard errors.

Original: C. Colored MNIST with confounded variables (CF-CMNIST) C. 
Revised: C. Colored MNIST with confounded variables (CF-CMNIST) C.

Original: Compute P ( C e | Y e ) and P ( Y e | X e ) .
Revised: Compute P ( C e | Y e ) and P ( Y e | X e ) .

Original: We start by computing P ( C e | Y e ) .
Revised: We start by computing P ( C e | Y e ) .

Original: Recall that C e
Revised: Recall that C e

Original: =
Revised: =

Original: ( N ⊕ N e ) .
Revised: ( N ⊕ N e ) .

Original: In the simplﬁcation that follows we use β = 0 . 25 .
Revised: In the simplﬁcation that follows we use β = 0 . 25 .

Original: [Equation]
Revised: [Equation]

Original: We use the above to compute P ( C e = 

Revised: We use the above to compute P ( C e =

Original: | Y e = 1)
Revised: | Y e = 1)

Original: =
Revised: =

Original: P ( N = 0 , N e = 0 |
Revised: P ( N = 0 , N e = 0 |

Original: Y e = 1)
Revised: Y e = 1)

Original: + P ( N = 1 , N e = 1 |
Revised: + P ( N = 1 , N e = 1 |

Original: Y e = 1)
Revised: Y e = 1)

Original: = (1 − β e )0 .
Revised: = (1 − β e )0 .

Original: 75 + 0 .
Revised: 75 + 0 .

Original: 25 β e = 0 .
Revised: 25 β e = 0 .

Original: 75 − 0 .
Revised: 75 − 0 .

Original: 5 β e .
Revised: 5 β e .

Original: For environment 1, 2, 3 the above probability P ( C e = 

Revised: For environment 1, 2, 3 the above probability P ( C e =

Original: | Y e = 1) is 0 .
Revised: | Y e = 1) is 0 .

Original: 65 and 0 .
Revised: 65 and 0 .

Original: 30 respectively.
Revised: 30 respectively.

Original: Next, we compute the probability P
Revised: Next, we compute the probability P

Original: ( Y e | X e ) .
Revised: ( Y e | X e ) .

Original: Suppose Y e g
Revised: Suppose Y e g

Original: = 1 for the calculation below.
Revised: = 1 for the calculation below.

Original: We also assume deterministic labeling.
Revised: We also assume deterministic labeling.

Original: [Table]
Revised: [Table]

Original: P ( Y e = 1 , C e = 1 | X eg ) = P ( Y e = 1 , C e = 1 | X eg ) =
Revised: P ( Y e = 1 , C e = 1 | X eg ) = P ( Y e = 1 , C e = 1 | X eg ) =

Original: P ( Y e = 1 | Y eg = 1) P ( C e = 1 | N = 0)
Revised: P ( Y e = 1 | Y eg = 1) P ( C e = 1 | N = 0)

Original: = 0 .
Revised: = 0 .

Original: 75 β e
Revised: 75 β e

Original: P ( Y e = 0 , C e = 1 | X eg ) =
Revised: P ( Y e = 0 , C e = 1 | X eg ) =

Original: P ( Y e = 1 , C e = 1 | X eg ) =
Revised: P ( Y e = 1 , C e = 1 | X eg ) =

Original: P ( Y e = 

Revised: P ( Y e =

Original: | Y eg = 1) P ( C e = 1 | N = 1)
Revised: | Y eg = 1) P ( C e = 1 | N = 1)

Original: = 0 .
Revised: = 0 .

Original: 25(1 − β e )
Revised: 25(1 − β e )

Original: P ( Y e = 1 | X e ) =
Revised: P ( Y e = 1 | X e ) =

Original: P ( Y e = 1 |
Revised: P ( Y e = 1 |

Original: X eg , C e = 1)
Revised: X eg , C e = 1)

Original: = 3 β e 2 β e +
Revised: = 3 β e 2 β e +

Original: P ( Y e = 1 | X e ) = 0 .
Revised: P ( Y e = 1 | X e ) = 0 .

Original: 25 (For environment 1)
Revised: 25 (For environment 1)

Original: P ( Y e = 1 | X e ) = 0 .
Revised: P ( Y e = 1 | X e ) = 0 .

Original: 428 (For environment 2)
Revised: 428 (For environment 2)

Original: P ( Y e = 1 | X e ) = 0 .
Revised: P ( Y e = 1 | X e ) = 0 .

Original: 964 (For environment 2) (16)
Revised: 964 (For environment 2) (16)

Original: C.2 Graphical model for confounded CMNIST.
Revised: C.2 Graphical model for confounded CMNIST.

Original: In Figure 4, we provide the graphical model for confounded CMNIST described in equation 10 (for G = 0 ).
Revised: In Figure 4, we provide the graphical model for confounded CMNIST described in equation 10 (for G = 0 ).

Original: C.3.
Revised: C.3.

Original: Results with numerical values and standard errors.
Revised: Results with numerical values and standard errors.

Original: In Table 4, we provide the numerical values for the results showed in the Figure 2 b) along with the standard errors.
Revised: In Table 4, we provide the numerical values for the results showed in the Figure 2 b) along with the standard errors.

Original: D. Colored MNIST with
Revised: D. Colored MNIST with

Original: anti-causal variables
Revised: anti-causal variables

Original: and
Revised: and

Original: confounded variables
Revised: confounded variables

Original: HB-CMNIST)
Revised: HB-CMNIST)

Original: D.1.
Revised: D.1.

Original: Compute P ( C e | Y e ) and P ( Y e | X e ) .
Revised: Compute P ( C e | Y e ) and P ( Y e | X e ) .

Original: We start by computing P ( C e | Y e ) .
Revised: We start by computing P ( C e | Y e ) .

Original: Recall that C e =
Revised: Recall that C e =

Original: G ( Y e ⊕ N e )
Revised: G ( Y e ⊕ N e )

Original: + (1 − G )
Revised: + (1 − G )

Original: ( N ⊕ N e ) .
Revised: ( N ⊕ N e ) .

Original: G = 1 with probability θ
Revised: G = 1 with probability θ

Original: and 0 otherwise.
Revised: and 0 otherwise.

Original: [Equation]
Revised: [Equation]

Original: We used θ = 0 . 8 in the experiments.
Revised: We used θ = 0 . 8 in the experiments.

Original: P ( Y e | X e ) can be computed on the same lines as was shown for anti-causal and confounded model and it varies signiﬁcantly across the environments.
Revised: P ( Y e | X e ) can be computed on the same lines as was shown for anti-causal and confounded model and it varies signiﬁcantly across the environments.

Original: D.2 Graphical model for confounded CMNIST.
Revised: D.2 Graphical model for confounded CMNIST.

Original: In Figure 4, we provide the graphical model for confounded CMNIST described in equation 10 (for 0 < θ < 1 ).
Revised: In Figure 4, we provide the graphical model for confounded CMNIST described in equation 10 (for 0 < θ < 1 ).

Original: D.3.
Revised: D.3.

Original: Results with numerical values and standard errors.
Revised: Results with numerical values and standard errors.

Original: In Table 5, we provide the numerical values for the results showed in the Figure 2 d) along with the standard errors.
Revised: In Table 5, we provide the numerical values for the results showed in the Figure 2 d) along with the standard errors.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: R EGRESSION
Revised: R EGRESSION

Original: We use the same structure for the generative model as described by Arjovsky et al.
Revised: We use the same structure for the generative model as described by Arjovsky et al.

Original: We work with the four different variants on the same lines as CMNIST (covariate shift based, confounded, anti-causal, hybrid).
Revised: We work with the four different variants on the same lines as CMNIST (covariate shift based, confounded, anti-causal, hybrid).

Original: The comparisons in Arjovsky et al.
Revised: The comparisons in Arjovsky et al.

Original: (2019) were for anti-causal and hybrid models.
Revised: (2019) were for anti-causal and hybrid models.

Original: The general model is written as
Revised: The general model is written as

Original: [Equation]
Revised: [Equation]

Original: H e is the hidden confounder, X e =
Revised: H e is the hidden confounder, X e =

Original: [ X e 1 , X e 2 ] is the observed covariate vector, Y e is the label.
Revised: [ X e 1 , X e 2 ] is the observed covariate vector, Y e is the label.

Original: Different W ’s correspond to the weight vectors that multiply with the covariates and the confounders.
Revised: Different W ’s correspond to the weight vectors that multiply with the covariates and the confounders.

Original: The four datasets differ in the weight W vectors and we describe them below.
Revised: The four datasets differ in the weight W vectors and we describe them below.

Original: σ e is environment
Revised: σ e is environment

Original: 𝑌 𝑌 𝑌 𝑌 𝑌 𝑌
Revised: 𝑌 𝑌 𝑌 𝑌 𝑌 𝑌

Original: [Figure]
Revised: [Figure]

Original: dependent standard deviation, we use two training environments with σ 1 = 0 .
Revised: dependent standard deviation, we use two training environments with σ 1 = 0 .

Original: 2 and σ 2 = 2 .respectively
Revised: 2 and σ 2 = 2 .respectively

Original: and both environments
Revised: and both environments

Original: • Covariate shift case (CS-regresion):
Revised: • Covariate shift case (CS-regresion):

Original: In this case, we ﬁx W h → 2 , W h → y , and W y → 2 to zero and we draw each entry of W 1 → y from 1 s N (0 , 1) and set W h → 1 to identity.
Revised: In this case, we ﬁx W h → 2 , W h → y , and W y → 2 to zero and we draw each entry of W 1 → y from 1 s N (0 , 1) and set W h → 1 to identity.

Original: • Confounded variable case (CF-regression): Set W y
Revised: • Confounded variable case (CF-regression): Set W y

Original: → 2 to zero and we draw each entry in W 1 → y , W h → 1 , W h → 2 , W h
Revised: → 2 to zero and we draw each entry in W 1 → y , W h → 1 , W h → 2 , W h

Original: →
Revised: →

Original: y from 1 s N (0 , 1) .
Revised: y from 1 s N (0 , 1) .

Original: • Anti-causal variable case (AC-regression): Set W h
Revised: • Anti-causal variable case (AC-regression): Set W h

Original: → y , W h → 1 and W h
Revised: → y , W h → 1 and W h

Original: → 2 to zero and draw each entry of W 1 → y and W y
Revised: → 2 to zero and draw each entry of W 1 → y and W y

Original: →
Revised: →

Original: 2 from 1 s N (0 , 1) .
Revised: 2 from 1 s N (0 , 1) .

Original: • Hybrid confounded and anti-causal variable case (HB-regression):
Revised: • Hybrid confounded and anti-causal variable case (HB-regression):

Original: Draw each W h → y , W h → 1 , W h → 2 , W 1 → y and W y
Revised: Draw each W h → y , W h → 1 , W h → 2 , W 1 → y and W y

Original: → 2 from 1 s N (0 , 1) .
Revised: → 2 from 1 s N (0 , 1) .

Original: We also present the graphical models for the four types of models used in Figure 5.
Revised: We also present the graphical models for the four types of models used in Figure 5.

Original: 7.2. C HOICE OF H Φ AND OTHER TRAINING DETAILS
Revised: 7.2. C HOICE OF H Φ AND OTHER TRAINING DETAILS

Original: We use a linear model that takes as input X e .
Revised: We use a linear model that takes as input X e .

Original: For ERM we use standard linear regression from sklearn.
Revised: For ERM we use standard linear regression from sklearn.

Original: For IRM, we use 50k gradient steps with learning rate 1e-3, batch size is equal to the size of the training data.
Revised: For IRM, we use 50k gradient steps with learning rate 1e-3, batch size is equal to the size of the training data.

Original: We use the train domain validation set procedure described by Gulrajani &
Revised: We use the train domain validation set procedure described by Gulrajani &

Original: Lopez-Paz (2020) to select the penalty value from the set { 0 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 } (with 4:1 train-validation split).
Revised: Lopez-Paz (2020) to select the penalty value from the set { 0 , 1 e − 5 , 1 e − 4 , 1 e − 3 , 1 e − 2 , 1 e − 1 } (with 4:1 train-validation split).

Original: We average the results over 25 trials.
Revised: We average the results over 25 trials.

Original: 7.2. R ESULTS
Revised: 7.2. R ESULTS

Original: We discuss results for the case when the length of the covariate vector X e is 10.
Revised: We discuss results for the case when the length of the covariate vector X e is 10.

Original: The desired optimal invariant predictor is W ∗ =
Revised: The desired optimal invariant predictor is W ∗ =

Original: [ W 1 → y , 0] .
Revised: [ W 1 → y , 0] .

Original: We will compare ERM and IRM in terms of the model estimation error, i.e., the distance between the model estimated by the method ˆ W and the true model given as (cid:107) ˆ W − W ∗ (cid:107) 2 .
Revised: We will compare ERM and IRM in terms of the model estimation error, i.e., the distance between the model estimated by the method ˆ W and the true model given as (cid:107) ˆ W − W ∗ (cid:107) 2 .

Original: In Figures 6, 7, 8, 9, we compare the model estimation error vs. the number of samples when the model.
Revised: In Figures 6, 7, 8, 9, we compare the model estimation error vs. the number of samples when the model.

Original: In these comparisons, we see that consistent with the classiﬁcation experiments and predictions from Proposition 4 in the covariate shift case (See Figure 6) there is no clear winner between the two approaches.
Revised: In these comparisons, we see that consistent with the classiﬁcation experiments and predictions from Proposition 4 in the covariate shift case (See Figure 6) there is no clear winner between the two approaches.

Original: There are gains from using IRM in the other cases (Figures 7, 8, 9).
Revised: There are gains from using IRM in the other cases (Figures 7, 8, 9).

Original: However, in the confounder case in Figure 7, the gains from IRM appear in the low sample regime but are not there in the high sample regime.
Revised: However, in the confounder case in Figure 7, the gains from IRM appear in the low sample regime but are not there in the high sample regime.

Original: This is because for this setup the asymptotic bias of ERM is also very small.
Revised: This is because for this setup the asymptotic bias of ERM is also very small.

Original: In addition to the Figures 6, 7, 8, 9, we provide the tables (see Table 6, 7, 8, 9) with the numerical values for the mean model estimation (and the standard error) error shown in the ﬁgures.
Revised: In addition to the Figures 6, 7, 8, 9, we provide the tables (see Table 6, 7, 8, 9) with the numerical values for the mean model estimation (and the standard error) error shown in the ﬁgures.

Original: [Figure]
Revised: [Figure]

Original: IRM
Revised: IRM

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: P ROOFS FOR THE P ROPOSITIONS
Revised: P ROOFS FOR THE P ROPOSITIONS

Original: In the results to follow, we will rely on Hoeffding’s inequality.
Revised: In the results to follow, we will rely on Hoeffding’s inequality.

Original: We restate the inequality below for convenience.
Revised: We restate the inequality below for convenience.

Original: Lemma 1.
Revised: Lemma 1.

Original: (Hoeffding’s inequality).
Revised: (Hoeffding’s inequality).

Original: Let θ 1 , . . .
Revised: Let θ 1 , . . .

Original: θ m be a sequence of i.i.d.
Revised: θ m be a sequence of i.i.d.

Original: random variables and assume that for all i , E [ θ i ]
Revised: random variables and assume that for all i , E [ θ i ]

Original: = µ and P [ a ≤ θ i ≤ b ] = 1 .
Revised: = µ and P [ a ≤ θ i ≤ b ] = 1 .

Original: Then, for any (cid:15) >
Revised: Then, for any (cid:15) >

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: We restate the propositions from the main body.
Revised: We restate the propositions from the main body.

Original: Next, we prove Proposition 1 from the main body of the manuscript.
Revised: Next, we prove Proposition 1 from the main body of the manuscript.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Proposition 7.
Revised: Proposition 7.

Original: If (cid:96) is square loss
Revised: If (cid:96) is square loss

Original: [Table]
Revised: [Table]

Original: and Assumptions 1,2 hold, then m ◦ Φ ∗ solves the OOD problem (equation 1)
Revised: and Assumptions 1,2 hold, then m ◦ Φ ∗ solves the OOD problem (equation 1)

Original: Proof.
Revised: Proof.

Original: Deﬁne a predictor w ◦ Φ ∗ , where Φ ∗ is deﬁned in Assumption 1.
Revised: Deﬁne a predictor w ◦ Φ ∗ , where Φ ∗ is deﬁned in Assumption 1.

Original: Let us simplify the expression for the risk for this predictor R e ( w ◦ Φ ∗ ) using square loss for (cid:96) .
Revised: Let us simplify the expression for the risk for this predictor R e ( w ◦ Φ ∗ ) using square loss for (cid:96) .

Original: Recall Z e =
Revised: Recall Z e =

Original: Φ ∗ ( X e )
Revised: Φ ∗ ( X e )

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In the above simpliﬁcation in equation
Revised: In the above simpliﬁcation in equation

Original: [Equation]
Revised: [Equation]

Original: we use the following equation 22 and equation 23, which rely on the law of total expectation.
Revised: we use the following equation 22 and equation 23, which rely on the law of total expectation.

Original: [Equation]
Revised: [Equation]

Original: In the last equality in equation
Revised: In the last equality in equation

Original: [Equation]
Revised: [Equation]

Original: we use the Assumption 1 and obtain ξ 2 .
Revised: we use the Assumption 1 and obtain ξ 2 .

Original: Therefore, substituting w = m in equation 21 achieves a risk of ξ 2 for all the environments.
Revised: Therefore, substituting w = m in equation 21 achieves a risk of ξ 2 for all the environments.

Original: [Equation]
Revised: [Equation]

Original: Consider the environment q that satisﬁes Assumption 2.
Revised: Consider the environment q that satisﬁes Assumption 2.

Original: Let us simplify the expression for the risk achieved by every predictor f ∈ F in the environment q following the steps similar to equation 21.
Revised: Let us simplify the expression for the risk achieved by every predictor f ∈ F in the environment q following the steps similar to equation 21.

Original: [Equation]
Revised: [Equation]

Original: In the above simpliﬁcation in equation
Revised: In the above simpliﬁcation in equation

Original: [Equation]
Revised: [Equation]

Original: we use the following equation
Revised: we use the following equation

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Therefore, for environment q satisfying Assumption 2 from equation 25, it follows that for all f ∈ F ,
Revised: Therefore, for environment q satisfying Assumption 2 from equation 25, it follows that for all f ∈ F ,

Original: R q ( f ) ≥ ξ 2 .
Revised: R q ( f ) ≥ ξ 2 .

Original: Therefore, we can write that
Revised: Therefore, we can write that

Original: [Equation]
Revised: [Equation]

Original: Therefore, from equation 27 it directly follows that
Revised: Therefore, from equation 27 it directly follows that

Original: [Equation]
Revised: [Equation]

Original: We showed in equation 24, R e ( m ◦ Φ ∗ )
Revised: We showed in equation 24, R e ( m ◦ Φ ∗ )

Original: =
Revised: =

Original: ξ 2 for all the environments.
Revised: ξ 2 for all the environments.

Original: Hence, f = m ◦ Φ ∗ achieves the RHS of equation 28.
Revised: Hence, f = m ◦ Φ ∗ achieves the RHS of equation 28.

Original: This completes the proof.
Revised: This completes the proof.

Original: Some of the proofs that we describe next take a few intermediate steps to build.
Revised: Some of the proofs that we describe next take a few intermediate steps to build.

Original: Here we give a brief preview of the key ingredients that we developed to build these propositions.
Revised: Here we give a brief preview of the key ingredients that we developed to build these propositions.

Original: • In Proposition 2, our goal is to carry out a sample complexity analysis of EIRM in the same spirit as ERM.
Revised: • In Proposition 2, our goal is to carry out a sample complexity analysis of EIRM in the same spirit as ERM.

Original: However, there are two key challenges that we are faced with - i) the IRM penalty R (cid:48) is not separable (as it is composed of terms involving squares of expectations) and ii) unlike ERM, IRM is a constrained optimization problem.
Revised: However, there are two key challenges that we are faced with - i) the IRM penalty R (cid:48) is not separable (as it is composed of terms involving squares of expectations) and ii) unlike ERM, IRM is a constrained optimization problem.

Original: To deal with i), we develop an estimator in the next section that allows us to re-express IRM penalty in a separable fashion.
Revised: To deal with i), we develop an estimator in the next section that allows us to re-express IRM penalty in a separable fashion.

Original: To deal with ii), we deﬁne a parameter κ that measures the minimum separation between IRM penalty and (cid:15) .
Revised: To deal with ii), we deﬁne a parameter κ that measures the minimum separation between IRM penalty and (cid:15) .

Original: We show that as long as this separation for all the predictors in the hypothesis class is positive, then we can rely on κ -representative property from ShalevShwartz & Ben-David (2014) applied to the new estimator that we build to show that the set of empirical invariant predictors ˆ S IV ( (cid:15) ) are the same as exact invariant predictors S IV ( (cid:15) )
Revised: We show that as long as this separation for all the predictors in the hypothesis class is positive, then we can rely on κ -representative property from ShalevShwartz & Ben-David (2014) applied to the new estimator that we build to show that the set of empirical invariant predictors ˆ S IV ( (cid:15) ) are the same as exact invariant predictors S IV ( (cid:15) )

Original: • In Proposition 5, our goal is to show that approximate OOD can be achieved by IRM in the ﬁnite sample regime.
Revised: • In Proposition 5, our goal is to show that approximate OOD can be achieved by IRM in the ﬁnite sample regime.

Original: This result builds on the inﬁnite sample result from Arjovsky et al.
Revised: This result builds on the inﬁnite sample result from Arjovsky et al.

Original: In Theorem 9 in Arjovsky et al.
Revised: In Theorem 9 in Arjovsky et al.

Original: (2019), it was shown that for linear models (deﬁned in Assumption 5) obeying linear general position 6, if the gradient constraints in the exact gradient constraint IRM equation 4 are satisﬁed, then the OOD solution is achieved.
Revised: (2019), it was shown that for linear models (deﬁned in Assumption 5) obeying linear general position 6, if the gradient constraints in the exact gradient constraint IRM equation 4 are satisﬁed, then the OOD solution is achieved.

Original: We extend this result to show that if the constraints in the approximate (cid:15) penalty based IRM in equation 5 are satisﬁed, then we are guaranteed to be in the √ (cid:15) neighborhood of the OOD solution.
Revised: We extend this result to show that if the constraints in the approximate (cid:15) penalty based IRM in equation 5 are satisﬁed, then we are guaranteed to be in the √ (cid:15) neighborhood of the OOD solution.

Original: Note that this result is again in the inﬁnite sample regime as it proves the approximation for solutions of the problem equation 5, which involves expectations w.r.t true distributions.
Revised: Note that this result is again in the inﬁnite sample regime as it proves the approximation for solutions of the problem equation 5, which involves expectations w.r.t true distributions.

Original: Next, we exploit similar tools that we introduced to prove Propositionto also prove the ﬁnite sample extension.
Revised: Next, we exploit similar tools that we introduced to prove Propositionto also prove the ﬁnite sample extension.

Original: • In later sections, we show the generalizations to inﬁnite hypothesis classes.
Revised: • In later sections, we show the generalizations to inﬁnite hypothesis classes.

Original: In particular, we focus on parametric model families that are Lipschitz continuous.
Revised: In particular, we focus on parametric model families that are Lipschitz continuous.

Original: The extension to inﬁnite hypothesis classes is based on carefully exploiting the covering number based techniques Shalev-Shwartz & Ben-David (2014) for the IRM penalty estimator that we introduced.
Revised: The extension to inﬁnite hypothesis classes is based on carefully exploiting the covering number based techniques Shalev-Shwartz & Ben-David (2014) for the IRM penalty estimator that we introduced.

Original: We also provide generalizations of the results for linear models to polynomial models.
Revised: We also provide generalizations of the results for linear models to polynomial models.

Original: To arrive at these results, we exploit some standard properties of tensor products.
Revised: To arrive at these results, we exploit some standard properties of tensor products.

Original: 7.3. E MPIRICAL E STIMATOR OF R (cid:48)
Revised: 7.3. E MPIRICAL E STIMATOR OF R (cid:48)

Original: Next, we deﬁne an estimator for R (cid:48) .
Revised: Next, we deﬁne an estimator for R (cid:48) .

Original: We ﬁrst simplify R (cid:48) as follows.
Revised: We ﬁrst simplify R (cid:48) as follows.

Original: Observe that
Revised: Observe that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In the above simpliﬁcation, we used Leibniz integral rule and take the derivative inside the expectation.
Revised: In the above simpliﬁcation, we used Leibniz integral rule and take the derivative inside the expectation.

Original: Also we can write E [ X ] 2 = E [ AB ] , where A and B are independent and identical random variables with same distribution as X .
Revised: Also we can write E [ X ] 2 = E [ AB ] , where A and B are independent and identical random variables with same distribution as X .

Original: Therefore, we consider two independent data points ( X e , Y e ) ∼ P e
Revised: Therefore, we consider two independent data points ( X e , Y e ) ∼ P e

Original: and ( ˜ X e , ˜
Revised: and ( ˜ X e , ˜

Original: Y e ) ∼ P e .
Revised: Y e ) ∼ P e .

Original: [Equation]
Revised: [Equation]

Original: In the above the expectation E e is taken over the joint distribution over pairs of distributions of pairs ( X e , Y e ) , ( ˜ X e , ˜ Y e ) from the same environment e .
Revised: In the above the expectation E e is taken over the joint distribution over pairs of distributions of pairs ( X e , Y e ) , ( ˜ X e , ˜ Y e ) from the same environment e .

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: In the above simpliﬁcation, we used equation 30.
Revised: In the above simpliﬁcation, we used equation 30.

Original: Deﬁne a joint distribution ˜
Revised: Deﬁne a joint distribution ˜

Original: P over the tuple ( e, ( X e , Y e ) , ( ˜ X e , ˜
Revised: P over the tuple ( e, ( X e , Y e ) , ( ˜ X e , ˜

Original: Y e )) , where e ∼ { π o } o ∈E tr , ( X e , Y e ) ∼ P e and ( ˜ X e , ˜
Revised: Y e )) , where e ∼ { π o } o ∈E tr , ( X e , Y e ) ∼ P e and ( ˜ X e , ˜

Original: Y e ) ∼ P e .
Revised: Y e ) ∼ P e .

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Original: We rewrite the above expression equation 31 in terms of an expectation w.r.t ˜ P , which we represent as ˜ E follows
Revised: We rewrite the above expression equation 31 in terms of an expectation w.r.t ˜ P , which we represent as ˜ E follows

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Substitute equation 34 in equation 33 to obtain
Revised: Substitute equation 34 in equation 33 to obtain

Original: [Equation]
Revised: [Equation]

Original: We construct a simple estimator ˆ R (cid:48) (Φ) by pairing the data points in each environment.
Revised: We construct a simple estimator ˆ R (cid:48) (Φ) by pairing the data points in each environment.

Original: For simplicity assume that each environment has even number of points.
Revised: For simplicity assume that each environment has even number of points.

Original: In environment e , which has n e points we construct n e 2 pairs.
Revised: In environment e , which has n e points we construct n e 2 pairs.

Original: Deﬁne a set of such pairs as
Revised: Deﬁne a set of such pairs as

Original: [Equation]
Revised: [Equation]

Original: There can be other estimators of R (cid:48) (Φ) where we separately estimate each term (cid:107)∇ w | w =1 . 

Revised: There can be other estimators of R (cid:48) (Φ) where we separately estimate each term (cid:107)∇ w | w =1 .

Original: R e ( w. Φ) (cid:107) 2 and π e in the summation.
Revised: R e ( w. Φ) (cid:107) 2 and π e in the summation.

Original: We rely on the above estimator equation 37 as its separability allows us to use standard concentration inequalities, e.g., Hoeffding’s inequality.
Revised: We rely on the above estimator equation 37 as its separability allows us to use standard concentration inequalities, e.g., Hoeffding’s inequality.

Original: 7.3. (cid:15) - REPRESENTATIVE TRAINING SET FOR R AND R (cid:48)
Revised: 7.3. (cid:15) - REPRESENTATIVE TRAINING SET FOR R AND R (cid:48)

Original: We use the deﬁnition of (cid:15) -representative sample from Shalev-Shwartz & Ben-David (2014) and state it appropriately for both R and R (cid:48) .
Revised: We use the deﬁnition of (cid:15) -representative sample from Shalev-Shwartz & Ben-David (2014) and state it appropriately for both R and R (cid:48) .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A training set S is called (cid:15) -representative
Revised: A training set S is called (cid:15) -representative

Original: (w.r.t.
Revised: (w.r.t.

Original: domain Z , hypothesis H , loss (cid:96) and distribution D )
Revised: domain Z , hypothesis H , loss (cid:96) and distribution D )

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: 38) where R ( h ) =
Revised: 38) where R ( h ) =

Original: E D [ (cid:96) ( h ( X ) , Y ] and ( X, Y ) ∼ D .
Revised: E D [ (cid:96) ( h ( X ) , Y ] and ( X, Y ) ∼ D .

Original: Following the above deﬁnition, we apply it to the set of points ˜ D deﬁned above in equation 36.
Revised: Following the above deﬁnition, we apply it to the set of points ˜ D deﬁned above in equation 36.

Original: ˜ D is called (cid:15) -representative w.r.t.
Revised: ˜ D is called (cid:15) -representative w.r.t.

Original: domain X , hypothesis H Φ , loss (cid:96) (cid:48) (equation 33) and distribution ˜ P (equation 32) if
Revised: domain X , hypothesis H Φ , loss (cid:96) (cid:48) (equation 33) and distribution ˜ P (equation 32) if

Original: [Equation]
Revised: [Equation]

Original: (39) where
Revised: (39) where

Original: R
Revised: R

Original: [Equation]
Revised: [Equation]

Original: (from equation 34) and ( e, ( X e , Y e ) , ( ˜ X e , ˜ Y e )) ∼ ˜ P .
Revised: (from equation 34) and ( e, ( X e , Y e ) , ( ˜ X e , ˜ Y e )) ∼ ˜ P .

Original: Recall the deﬁnition of κ , κ = min Φ ∈H Φ | R (cid:48) (Φ) − (cid:15) | .
Revised: Recall the deﬁnition of κ , κ = min Φ ∈H Φ | R (cid:48) (Φ) − (cid:15) | .

Original: Next, we show that if ˜ D is κ 2 -representative w.r.t X , H Φ , loss (cid:96) (cid:48) , distribution ˜ P then the set of invariant predictors in equation 6 ( ˆ S IV ( (cid:15) ) ) and the set of invariant predictors in equation 5 ( S IV ( (cid:15) ) ) are equal.
Revised: Next, we show that if ˜ D is κ 2 -representative w.r.t X , H Φ , loss (cid:96) (cid:48) , distribution ˜ P then the set of invariant predictors in equation 6 ( ˆ S IV ( (cid:15) ) ) and the set of invariant predictors in equation 5 ( S IV ( (cid:15) ) ) are equal.

Original: Lemma 2.
Revised: Lemma 2.

Original: If κ > 0 and ˜ D is κ 2 -representative w.r.t X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P , then ˆ S IV ( (cid:15) )
Revised: If κ > 0 and ˜ D is κ 2 -representative w.r.t X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P , then ˆ S IV ( (cid:15) )

Original: =
Revised: =

Original: S IV ( (cid:15) ) .
Revised: S IV ( (cid:15) ) .

Original: Proof.
Revised: Proof.

Original: First we show S IV ( (cid:15) ) ⊆
Revised: First we show S IV ( (cid:15) ) ⊆

Original: ˆ S IV ( (cid:15) ) .
Revised: ˆ S IV ( (cid:15) ) .

Original: From the deﬁnition of κ , κ = min Φ ∈H Φ | R (cid:48) (Φ) − (cid:15) | it follows that ∀ Φ ∈ H Φ
Revised: From the deﬁnition of κ , κ = min Φ ∈H Φ | R (cid:48) (Φ) − (cid:15) | it follows that ∀ Φ ∈ H Φ

Original: [Equation]
Revised: [Equation]

Original: Consider any Φ in S IV ( (cid:15) ) .
Revised: Consider any Φ in S IV ( (cid:15) ) .

Original: [Equation]
Revised: [Equation]

Original: Given the deﬁnition of κ and equation 40, we obtain
Revised: Given the deﬁnition of κ and equation 40, we obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore, S IV ( (cid:15) ) ⊆
Revised: Therefore, S IV ( (cid:15) ) ⊆

Original: S IV ( (cid:15) − κ ) .
Revised: S IV ( (cid:15) − κ ) .

Original: Also, it follows from the deﬁnition of the set S IV ( (cid:15) )
Revised: Also, it follows from the deﬁnition of the set S IV ( (cid:15) )

Original: that S IV ( (cid:15) − κ )
Revised: that S IV ( (cid:15) − κ )

Original: ⊆
Revised: ⊆

Original: S IV ( (cid:15) ) .
Revised: S IV ( (cid:15) ) .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Consider any Φ in S IV ( (cid:15) )
Revised: Consider any Φ in S IV ( (cid:15) )

Original: [Equation]
Revised: [Equation]

Original: From equation 43)
Revised: From equation 43)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From the deﬁnition of κ 2 -representativeness it follows that | R (cid:48) (Φ) − ˆ R (cid:48) (Φ) | ≤ κ 2 and substituting this in equation 44 we get
Revised: From the deﬁnition of κ 2 -representativeness it follows that | R (cid:48) (Φ) − ˆ R (cid:48) (Φ) | ≤ κ 2 and substituting this in equation 44 we get

Original: [Equation]
Revised: [Equation]

Original: Next we show ˆ S IV ( (cid:15) )
Revised: Next we show ˆ S IV ( (cid:15) )

Original: ⊆
Revised: ⊆

Original: S IV ( (cid:15) ) .
Revised: S IV ( (cid:15) ) .

Original: Consider Φ ∈ ˆ S IV ( (cid:15) )
Revised: Consider Φ ∈ ˆ S IV ( (cid:15) )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From the deﬁnition of κ 2 -representativeness it follows that | R (cid:48)
Revised: From the deﬁnition of κ 2 -representativeness it follows that | R (cid:48)

Original: [Equation]
Revised: [Equation]

Original: Φ
Revised: Φ

Original: [Equation]
Revised: [Equation]

Original: − ˆ R
Revised: − ˆ R

Original: [Equation]
Revised: [Equation]

Original: (Φ) | ≤ κ 2 and substituting this in equation 46 we get
Revised: (Φ) | ≤ κ 2 and substituting this in equation 46 we get

Original: [Equation]
Revised: [Equation]

Original: From equation 40, it follows that R (cid:48) (Φ) ≤ (cid:15) + κ 2 = ⇒ R (cid:48)
Revised: From equation 40, it follows that R (cid:48) (Φ) ≤ (cid:15) + κ 2 = ⇒ R (cid:48)

Original: (Φ) ≤ (cid:15) .
Revised: (Φ) ≤ (cid:15) .

Original: Therefore, Φ ∈ S IV ( (cid:15) ) .
Revised: Therefore, Φ ∈ S IV ( (cid:15) ) .

Original: This proves the second part ˆ S IV ( (cid:15) ) ⊆
Revised: This proves the second part ˆ S IV ( (cid:15) ) ⊆

Original: S IV ( (cid:15) ) and completes the proof.
Revised: S IV ( (cid:15) ) and completes the proof.

Original: Lemma 3.
Revised: Lemma 3.

Original: If κ > 0 , D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution ¯ P (joint distribution over ( e, X e , Y e ) deﬁned in Section 3.1) and and ˜ D is κ 2 -representative w.r.t X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P , then every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ is in S IV ( (cid:15) ) and R (Φ ∗ ) ≤
Revised: If κ > 0 , D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution ¯ P (joint distribution over ( e, X e , Y e ) deﬁned in Section 3.1) and and ˜ D is κ 2 -representative w.r.t X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P , then every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ is in S IV ( (cid:15) ) and R (Φ ∗ ) ≤

Original: R (ˆΦ) ≤ R (Φ ∗ ) + (cid:15) , where Φ ∗ is the solution of IRM in equation 5.
Revised: R (ˆΦ) ≤ R (Φ ∗ ) + (cid:15) , where Φ ∗ is the solution of IRM in equation 5.

Original: Proof.
Revised: Proof.

Original: Given the condition in the above lemma, we are able to use the previous Lemma 2 to deduce that ˆ S IV ( (cid:15) ) =
Revised: Given the condition in the above lemma, we are able to use the previous Lemma 2 to deduce that ˆ S IV ( (cid:15) ) =

Original: S IV ( (cid:15) ) .
Revised: S IV ( (cid:15) ) .

Original: This makes the set of predictors satisfying the constraints in EIRM equationand IRM equation 5 the same.
Revised: This makes the set of predictors satisfying the constraints in EIRM equationand IRM equation 5 the same.

Original: Φ ∗ solves equation 5 and ˆΦ solves equation 6.
Revised: Φ ∗ solves equation 5 and ˆΦ solves equation 6.

Original: From ν 2 -representativeness we know that R (ˆΦ) − ν 2 ≤ ˆ R (ˆΦ) .
Revised: From ν 2 -representativeness we know that R (ˆΦ) − ν 2 ≤ ˆ R (ˆΦ) .

Original: From the optimality of ˆΦ we know that ˆ R (ˆΦ) ≤ ˆ R (Φ ∗ ) ( Φ ∗ ∈
Revised: From the optimality of ˆΦ we know that ˆ R (ˆΦ) ≤ ˆ R (Φ ∗ ) ( Φ ∗ ∈

Original: ˆ S IV ( (cid:15) )
Revised: ˆ S IV ( (cid:15) )

Original: =
Revised: =

Original: S IV ( (cid:15) ) ).
Revised: S IV ( (cid:15) ) ).

Original: Moreover, from ν 2 -representativeness we know that
Revised: Moreover, from ν 2 -representativeness we know that

Original: ˆ R (Φ ∗ ) ≤ R (Φ ∗ ) + ν 2 .
Revised: ˆ R (Φ ∗ ) ≤ R (Φ ∗ ) + ν 2 .

Original: We combine these conditions as follows.
Revised: We combine these conditions as follows.

Original: [Equation]
Revised: [Equation]

Original: Comparing the ﬁrst and third inequality in the above equations we get R (ˆΦ) ≤ R (Φ ∗ )
Revised: Comparing the ﬁrst and third inequality in the above equations we get R (ˆΦ) ≤ R (Φ ∗ )

Original: + ν .
Revised: + ν .

Original: From the optimality of Φ ∗ over the set S IV ( (cid:15) ) and since ˆΦ ∈ S IV ( (cid:15) )
Revised: From the optimality of Φ ∗ over the set S IV ( (cid:15) ) and since ˆΦ ∈ S IV ( (cid:15) )

Original: it follows that R (Φ ∗ ) ≤ R (ˆΦ) .
Revised: it follows that R (Φ ∗ ) ≤ R (ˆΦ) .

Original: Hence,
Revised: Hence,

Original: R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ ) + ν .
Revised: R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ ) + ν .

Original: This completes the proof.
Revised: This completes the proof.

Original: Next, we prove Proposition 2 from the main body of the manuscript.
Revised: Next, we prove Proposition 2 from the main body of the manuscript.

Original: Proposition 8.
Revised: Proposition 8.

Original: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, Assumption 3 holds, κ > 0 , and if the number of samples | D | is greater than max (cid:8) 16 L (cid:48) 4 κ 2 , 8 L 2 ν 2 (cid:9) log (cid:0) 4 |H
Revised: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, Assumption 3 holds, κ > 0 , and if the number of samples | D | is greater than max (cid:8) 16 L (cid:48) 4 κ 2 , 8 L 2 ν 2 (cid:9) log (cid:0) 4 |H

Original: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ of EIRM (equation 6) is a ν approximation of IRM, i.e. ˆΦ ∈ S IV ( (cid:15) ) , R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )
Revised: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ of EIRM (equation 6) is a ν approximation of IRM, i.e. ˆΦ ∈ S IV ( (cid:15) ) , R (Φ ∗ ) ≤ R (ˆΦ) ≤ R (Φ ∗ )

Original: + ν , where Φ ∗ is a solution to IRM (equation 5).
Revised: + ν , where Φ ∗ is a solution to IRM (equation 5).

Original: Proof.
Revised: Proof.

Original: From Lemma 3, we know that if D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution
Revised: From Lemma 3, we know that if D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution

Original: P and if ˜ D is κ 2 -representative X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P , then the the claim in the above
Revised: P and if ˜ D is κ 2 -representative X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P , then the the claim in the above

Original: Proposition is true.
Revised: Proposition is true.

Original: Deﬁne an event
Revised: Deﬁne an event

Original: A : D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution ¯ P .
Revised: A : D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution ¯ P .

Original: Deﬁne an event B : ˜ D is κ 2 -representative X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P .
Revised: Deﬁne an event B : ˜ D is κ 2 -representative X , H Φ , loss (cid:96) (cid:48) and distribution ˜ P .

Original: Deﬁne success as A ∩ B .
Revised: Deﬁne success as A ∩ B .

Original: Next, we show that if | D | is greater than max (cid:8) 16 L (cid:48) 4 κ 2 , 8 L 2 ν 2 (cid:9) log (cid:0) 4 |H
Revised: Next, we show that if | D | is greater than max (cid:8) 16 L (cid:48) 4 κ 2 , 8 L 2 ν 2 (cid:9) log (cid:0) 4 |H

Original: Φ | δ (cid:1) , then P ( A ∩ B ) occurs with a probability at least 1 − δ .
Revised: Φ | δ (cid:1) , then P ( A ∩ B ) occurs with a probability at least 1 − δ .

Original: P ( A ∩ B ) = 1 − P ( A c ∪ B c ) ≥ 1 − P
Revised: P ( A ∩ B ) = 1 − P ( A c ∪ B c ) ≥ 1 − P

Original: ( A c ) −
Revised: ( A c ) −

Original: P ( B c ) .
Revised: P ( B c ) .

Original: If we bound P ( A c ) ≤ δ 2 and P ( B c ) ≤ δ 2 , then we know the probability of success is at least 1 − δ
Revised: If we bound P ( A c ) ≤ δ 2 and P ( B c ) ≤ δ 2 , then we know the probability of success is at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: The loss function is bounded | (cid:96) (Φ( · ) , · ) | ≤ L .
Revised: The loss function is bounded | (cid:96) (Φ( · ) , · ) | ≤ L .

Original: From Hoeffding’s inequality in Lemma 1 it follows that
Revised: From Hoeffding’s inequality in Lemma 1 it follows that

Original: [Equation]
Revised: [Equation]

Original: Using this expression equation
Revised: Using this expression equation

Original: [Equation]
Revised: [Equation]

Original: in equation
Revised: in equation

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The gradient of loss function is bounded | ∂(cid:96)
Revised: The gradient of loss function is bounded | ∂(cid:96)

Original: ( h ( · ) , · ) ∂w | w =1 . 

Revised: ( h ( · ) , · ) ∂w | w =1 .

Original: | ≤ L (cid:48) .
Revised: | ≤ L (cid:48) .

Original: From the deﬁnition of (cid:96) (cid:48) ( h ( · ) , · ) in equation 34, we can infer that | (cid:96) (cid:48)
Revised: From the deﬁnition of (cid:96) (cid:48) ( h ( · ) , · ) in equation 34, we can infer that | (cid:96) (cid:48)

Original: ( h ( · ) , · ) | ≤ L (cid:48) 2 .
Revised: ( h ( · ) , · ) | ≤ L (cid:48) 2 .

Original: Recall that R (cid:48)
Revised: Recall that R (cid:48)

Original: ( h ) =
Revised: ( h ) =

Original: ˜ E (cid:104) (cid:96)
Revised: ˜ E (cid:104) (cid:96)

Original: [Equation]
Revised: [Equation]

Original: (cid:16) h, (cid:0) ( X e , Y e ) , ( ˜ X e , ˜
Revised: (cid:16) h, (cid:0) ( X e , Y e ) , ( ˜ X e , ˜

Original: Y e ) (cid:1)(cid:17)(cid:105)
Revised: Y e ) (cid:1)(cid:17)(cid:105)

Original: From Hoeffding’s inequality in Lemma 1 it follows that
Revised: From Hoeffding’s inequality in Lemma 1 it follows that

Original: [Equation]
Revised: [Equation]

Original: Using the above equation 53 in equation 52 we get
Revised: Using the above equation 53 in equation 52 we get

Original: [Equation]
Revised: [Equation]

Original: Combining the two conditions in equation 51 and equation 51 we get that if
Revised: Combining the two conditions in equation 51 and equation 51 we get that if

Original: [Equation]
Revised: [Equation]

Original: then with probability at least 1 − δ event A ∩ B occurs.
Revised: then with probability at least 1 − δ event A ∩ B occurs.

Original: 7.3. P ROPERTY OF LEAST SQUARES OPTIMAL SOLUTIONS
Revised: 7.3. P ROPERTY OF LEAST SQUARES OPTIMAL SOLUTIONS

Original: We ﬁrst remind ourselves of a simple property of least squares minimization.
Revised: We ﬁrst remind ourselves of a simple property of least squares minimization.

Original: Consider the least squares minimization setting, where R ( h )
Revised: Consider the least squares minimization setting, where R ( h )

Original: =
Revised: =

Original: E [( Y − h ( X )) 2 ] .
Revised: E [( Y − h ( X )) 2 ] .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the above simpliﬁcation
Revised: In the above simpliﬁcation

Original: [Equation]
Revised: [Equation]

Original: we use the law of total expectation.
Revised: we use the law of total expectation.

Original: Both the terms in the above equations are always greater than or equal to zero.
Revised: Both the terms in the above equations are always greater than or equal to zero.

Original: The ﬁrst term does not depend on h , which implies the minimization can focus on second term only.
Revised: The ﬁrst term does not depend on h , which implies the minimization can focus on second term only.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Assume that P has full support over X .
Revised: Assume that P has full support over X .

Original: Deﬁne ∀ x ∈ X , h ∗ ( x ) =
Revised: Deﬁne ∀ x ∈ X , h ∗ ( x ) =

Original: E [ Y | X = x ] .
Revised: E [ Y | X = x ] .

Original: ∀ h, R ( h ) ≥
Revised: ∀ h, R ( h ) ≥

Original: E X (cid:104) Var (cid:2) Y (cid:12)(cid:12) X (cid:3) (cid:105) .
Revised: E X (cid:104) Var (cid:2) Y (cid:12)(cid:12) X (cid:3) (cid:105) .

Original: Since R ( h ∗ ) =
Revised: Since R ( h ∗ ) =

Original: E X (cid:104) Var (cid:2) Y (cid:12)(cid:12) X
Revised: E X (cid:104) Var (cid:2) Y (cid:12)(cid:12) X

Original: (cid:3)(cid:105) .
Revised: (cid:3)(cid:105) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Moreover, we conclude that h ∗ is the unique minimizer.
Revised: Moreover, we conclude that h ∗ is the unique minimizer.

Original: Observe that E X (cid:104)(cid:0) E
Revised: Observe that E X (cid:104)(cid:0) E

Original: [Equation]
Revised: [Equation]

Original: Y (cid:12)(cid:12) X (cid:3) − h
Revised: Y (cid:12)(cid:12) X (cid:3) − h

Original: ( X ) (cid:1) 2 (cid:105) is zero for h
Revised: ( X ) (cid:1) 2 (cid:105) is zero for h

Original: = h ∗ .
Revised: = h ∗ .

Original: From Theorem 1.6.6 in (Ash et al., 2000), it follows that any other minimizer is same as h ∗ except over a set of measure zero.
Revised: From Theorem 1.6.6 in (Ash et al., 2000), it follows that any other minimizer is same as h ∗ except over a set of measure zero.

Original: Recall the deﬁnition of m ( x ) =
Revised: Recall the deﬁnition of m ( x ) =

Original: E [ Y e | X e = x ]
Revised: E [ Y e | X e = x ]

Original: Lemma 4.
Revised: Lemma 4.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: If Assumption 4 holds and m ∈ H Φ , then m uniquely solves expected risk minimization m ∈ arg min Φ ∈H Φ R (Φ) and also uniquely solves IRM (equation 5).
Revised: If Assumption 4 holds and m ∈ H Φ , then m uniquely solves expected risk minimization m ∈ arg min Φ ∈H Φ R (Φ) and also uniquely solves IRM (equation 5).

Original: Proof.
Revised: Proof.

Original: R (Φ) =
Revised: R (Φ) =

Original: (cid:80) e ∈E tr π e R e (Φ) .
Revised: (cid:80) e ∈E tr π e R e (Φ) .

Original: From Assumption 4 and the observation in equation 57, it follows that the unique optimal solution to expected risk minimization for each R e is m .
Revised: From Assumption 4 and the observation in equation 57, it follows that the unique optimal solution to expected risk minimization for each R e is m .

Original: Therefore, m also minimizes the weighted combination R .
Revised: Therefore, m also minimizes the weighted combination R .

Original: To show the latter part of the Lemma, if we can show that m ∈ S IV ( (cid:15) ) , then the rest of the proof follows from the previous part as we already showed m is a minimizer among all the functions in
Revised: To show the latter part of the Lemma, if we can show that m ∈ S IV ( (cid:15) ) , then the rest of the proof follows from the previous part as we already showed m is a minimizer among all the functions in

Original: H Φ and S IV ( (cid:15) ) ⊆
Revised: H Φ and S IV ( (cid:15) ) ⊆

Original: H Φ .
Revised: H Φ .

Original: Suppose m (cid:54)∈ S IV ( (cid:15) ) .
Revised: Suppose m (cid:54)∈ S IV ( (cid:15) ) .

Original: This implies there exists at least one environment for which (cid:107)∇ w | w =1 . 

Revised: This implies there exists at least one environment for which (cid:107)∇ w | w =1 .

Original: R e ( w · m ) (cid:107) 2 > 0 =
Revised: R e ( w · m ) (cid:107) 2 > 0 =

Original: ⇒ ∇ w | w =1 . 

Revised: ⇒ ∇ w | w =1 .

Original: R e ( w · m ) (cid:54) = 0 .
Revised: R e ( w · m ) (cid:54) = 0 .

Original: As a result ∃ w in the neighborhood of w = 1 . 

Revised: As a result ∃ w in the neighborhood of w = 1 .

Original: where
Revised: where

Original: R e ( w · m ) <
Revised: R e ( w · m ) <

Original: R e ( m ) (if such a point does not exist and all the points in the neigbohood of w = 1 .are greater than or equal to R e ( m ) that would make ∇ w | w =1 . 

Revised: R e ( m ) (if such a point does not exist and all the points in the neigbohood of w = 1 .are greater than or equal to R e ( m ) that would make ∇ w | w =1 .

Original: R e ( w · m )
Revised: R e ( w · m )

Original: = 0 , which would be a contradiction).
Revised: = 0 , which would be a contradiction).

Original: Therefore, R e ( w · m ) <
Revised: Therefore, R e ( w · m ) <

Original: R e ( m ) .
Revised: R e ( m ) .

Original: However, this is a contradiction as we know that m is the unique optimizer for each environment.
Revised: However, this is a contradiction as we know that m is the unique optimizer for each environment.

Original: Hence, m (cid:54)∈ S IV ( (cid:15) ) cannot be true and thus m ∈ S IV ( (cid:15) ) .
Revised: Hence, m (cid:54)∈ S IV ( (cid:15) ) cannot be true and thus m ∈ S IV ( (cid:15) ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: Next, we prove Proposition 4 from the main body of the manuscript.
Revised: Next, we prove Proposition 4 from the main body of the manuscript.

Original: Proposition 9.
Revised: Proposition 9.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, m ∈ H Φ , Assumptions 3, 4 hold, and
Revised: For every ν > 0 , (cid:15) > 0 and δ ∈ (0 , 1) , if H Φ is a ﬁnite hypothesis class, m ∈ H Φ , Assumptions 3, 4 hold, and

Original: • if the number of samples | D | is greater than max (cid:8) 8 L 2 ν 2 log( 4 |H
Revised: • if the number of samples | D | is greater than max (cid:8) 8 L 2 ν 2 log( 4 |H

Original: Φ | δ ) , 16 L (cid:48) 4 (cid:15) 2 log( 2 δ ) (cid:9) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes R ( m ) ≤ R (ˆΦ) ≤ R ( m )+ ν .
Revised: Φ | δ ) , 16 L (cid:48) 4 (cid:15) 2 log( 2 δ ) (cid:9) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes R ( m ) ≤ R (ˆΦ) ≤ R ( m )+ ν .

Original: If also ν < ˜ κ , then Φ † = m .
Revised: If also ν < ˜ κ , then Φ † = m .

Original: • if the number of samples | D | is greater than 8 L 2 ν 2 log( 2 |H
Revised: • if the number of samples | D | is greater than 8 L 2 ν 2 log( 2 |H

Original: Φ |
Revised: Φ |

Original: δ ) , then with a probability at least 1 − δ , every solution Φ † to ERM satisﬁes R ( m ) ≤ R (Φ † ) ≤ R ( m ) + ν .
Revised: δ ) , then with a probability at least 1 − δ , every solution Φ † to ERM satisﬁes R ( m ) ≤ R (Φ † ) ≤ R ( m ) + ν .

Original: If also ν < ˜ κ , then Φ † = m .
Revised: If also ν < ˜ κ , then Φ † = m .

Original: Proof.
Revised: Proof.

Original: We ﬁrst cover the second part of the Proposition.
Revised: We ﬁrst cover the second part of the Proposition.

Original: From Proposition 3, we know that the output of ERM will satisfy R (Φ + ) ≤ R (Φ † ) ≤ R (Φ + ) + ν .
Revised: From Proposition 3, we know that the output of ERM will satisfy R (Φ + ) ≤ R (Φ † ) ≤ R (Φ + ) + ν .

Original: In this case from Lemma 4, it follows that Φ + = m .
Revised: In this case from Lemma 4, it follows that Φ + = m .

Original: From the deﬁnition of ˜ κ and the fact that ν < ˜ κ implies that Φ † = m .
Revised: From the deﬁnition of ˜ κ and the fact that ν < ˜ κ implies that Φ † = m .

Original: We now move to the ﬁrst part of the Proposition.
Revised: We now move to the ﬁrst part of the Proposition.

Original: For EIRM we will derive a tighter bound on sample complexity than the one in Proposition 2 since we can now use the Assumption 4.
Revised: For EIRM we will derive a tighter bound on sample complexity than the one in Proposition 2 since we can now use the Assumption 4.

Original: Observe that ∀ e ∈ E tr , ∇ w | w =1 . 

Revised: Observe that ∀ e ∈ E tr , ∇ w | w =1 .

Original: R e ( w · m ) = 

Revised: R e ( w · m ) =

Original: (see the proof of Lemma 4).
Revised: (see the proof of Lemma 4).

Original: Therefore, R (cid:48) ( m ) = 0 .
Revised: Therefore, R (cid:48) ( m ) = 0 .

Original: Deﬁne an event
Revised: Deﬁne an event

Original: A : D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution ¯ P .
Revised: A : D is ν 2 -representative w.r.t X , H Φ , loss (cid:96) and distribution ¯ P .

Original: Deﬁne an event B : ˜ D is such that | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 .
Revised: Deﬁne an event B : ˜ D is such that | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 .

Original: Since R (cid:48) ( m ) = 0 , | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15)
Revised: Since R (cid:48) ( m ) = 0 , | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15)

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:15) 2 .
Revised: ≤ (cid:15) 2 .

Original: Therefore, m ∈ ˆ S IV ( (cid:15) ) .
Revised: Therefore, m ∈ ˆ S IV ( (cid:15) ) .

Original: If A ∩ B occurs, then R ( m ) ≤ R (ˆΦ) ≤ R ( m ) + ν ; we justify claim next.
Revised: If A ∩ B occurs, then R ( m ) ≤ R (ˆΦ) ≤ R ( m ) + ν ; we justify claim next.

Original: Suppose ˆΦ solves equation 6.
Revised: Suppose ˆΦ solves equation 6.

Original: If event A occurs, then from ν 2 -representative condition we know that R (ˆΦ) − ν 2 ≤ ˆ R (ˆΦ) .
Revised: If event A occurs, then from ν 2 -representative condition we know that R (ˆΦ) − ν 2 ≤ ˆ R (ˆΦ) .

Original: From optimality of ˆΦ it follows that ˆ R (ˆΦ) ≤ ˆ R ( m ) (event B = ⇒ m
Revised: From optimality of ˆΦ it follows that ˆ R (ˆΦ) ≤ ˆ R ( m ) (event B = ⇒ m

Original: ∈ ˆ S IV ( (cid:15) ) ).
Revised: ∈ ˆ S IV ( (cid:15) ) ).

Original: Moreover, from ν 2 -representative property, we conclude that
Revised: Moreover, from ν 2 -representative property, we conclude that

Original: ˆ R ( m ) ≤ R ( m ) + ν 2 .
Revised: ˆ R ( m ) ≤ R ( m ) + ν 2 .

Original: We combine these conditions as follows.
Revised: We combine these conditions as follows.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From the above we have R ( m ) ≤ R (ˆΦ) ≤ R ( m ) + ν .
Revised: From the above we have R ( m ) ≤ R (ˆΦ) ≤ R ( m ) + ν .

Original: Recall the deﬁnition of ˜ κ and since ν < ˜ κ = ⇒ ˆΦ = m .
Revised: Recall the deﬁnition of ˜ κ and since ν < ˜ κ = ⇒ ˆΦ = m .

Original: Next, we bound the probability of success.
Revised: Next, we bound the probability of success.

Original: P ( A ∩ B ) = 1 − P ( A c ∪ B c ) ≥ 1 − P ( A c ) − P ( B c ) .
Revised: P ( A ∩ B ) = 1 − P ( A c ∪ B c ) ≥ 1 − P ( A c ) − P ( B c ) .

Original: If we can bound P ( A c ) ≤ δ 2 and P ( B c ) ≤ δ 2 , then we know the probability of success is at least 1 − δ .
Revised: If we can bound P ( A c ) ≤ δ 2 and P ( B c ) ≤ δ 2 , then we know the probability of success is at least 1 − δ .

Original: We write
Revised: We write

Original: [Equation]
Revised: [Equation]

Original: From equation 51 if the condition
Revised: From equation 51 if the condition

Original: [Equation]
Revised: [Equation]

Original: is true, then is true, then event A c occurs with probability at most δ 2 .
Revised: is true, then is true, then event A c occurs with probability at most δ 2 .

Original: We write P ( B ) =
Revised: We write P ( B ) =

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: > (cid:15)
Revised: > (cid:15)

Original: [Equation]
Revised: [Equation]

Original: (cid:111)(cid:17) .
Revised: (cid:111)(cid:17) .

Original: The gradient of loss function is bounded | ∂(cid:96)
Revised: The gradient of loss function is bounded | ∂(cid:96)

Original: (Φ( · ) , · ) ∂w |
Revised: (Φ( · ) , · ) ∂w |

Original: [Equation]
Revised: [Equation]

Original: | ≤ L (cid:48) .
Revised: | ≤ L (cid:48) .

Original: From Hoeffding’s inequality in
Revised: From Hoeffding’s inequality in

Original: Lemma 1 it follows that
Revised: Lemma 1 it follows that

Original: [Equation]
Revised: [Equation]

Original: We bound the above equation 60 by δ 2 to get
Revised: We bound the above equation 60 by δ 2 to get

Original: [Equation]
Revised: [Equation]

Original: Combining the two conditions equation 59 and equation 61,
Revised: Combining the two conditions equation 59 and equation 61,

Original: [Equation]
Revised: [Equation]

Original: This ensures P ( A ∩ B ) ≥ 1 − δ .
Revised: This ensures P ( A ∩ B ) ≥ 1 − δ .

Original: This completes the proof.
Revised: This completes the proof.

Original: Before stating the proof of Proposition 5, we will prove an intermediate proposition.
Revised: Before stating the proof of Proposition 5, we will prove an intermediate proposition.

Original: For clarity, we will restate the result (Theorem 9 from Arjovsky et al.
Revised: For clarity, we will restate the result (Theorem 9 from Arjovsky et al.

Original: next.
Revised: next.

Original: Proposition 10.
Revised: Proposition 10.

Original: (Theorem 9 Arjovsky et al.
Revised: (Theorem 9 Arjovsky et al.

Original: If Assumptions 5 and 6 (with r = 1 ) hold and let Φ ∈ R n × 1 ( Φ (cid:54) = 0 ), then
Revised: If Assumptions 5 and 6 (with r = 1 ) hold and let Φ ∈ R n × 1 ( Φ (cid:54) = 0 ), then

Original: [Equation]
Revised: [Equation]

Original: holds for all e ∈ E tr iff Φ = ˜ S T γ .
Revised: holds for all e ∈ E tr iff Φ = ˜ S T γ .

Original: Next, we propose an (cid:15) -approximation of Proposition 10.
Revised: Next, we propose an (cid:15) -approximation of Proposition 10.

Original: [Equation]
Revised: [Equation]

Original: Proposition 11.
Revised: Proposition 11.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: If Assumptions 5, 6 (with r = 1 ) and 7 hold , then for all 0 < (cid:15) < (cid:15) 0 , the solution Φ
Revised: If Assumptions 5, 6 (with r = 1 ) and 7 hold , then for all 0 < (cid:15) < (cid:15) 0 , the solution Φ

Original: [Equation]
Revised: [Equation]

Original: satisﬁes Φ =
Revised: satisﬁes Φ =

Original: ˜ S
Revised: ˜ S

Original: T γ ( α ) , where α ∈ [
Revised: T γ ( α ) , where α ∈ [

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us start by simplifying ∇ w | w =
Revised: Let us start by simplifying ∇ w | w =

Original: [Equation]
Revised: [Equation]

Original: R e ( w · Φ) , using square loss for (cid:96) and linear representation Φ ∈ R n × 1 .
Revised: R e ( w · Φ) , using square loss for (cid:96) and linear representation Φ ∈ R n × 1 .

Original: [Equation]
Revised: [Equation]

Original: Plug the above equation 64 in the condition R (cid:48) (Φ) ≤ (cid:15) to get
Revised: Plug the above equation 64 in the condition R (cid:48) (Φ) ≤ (cid:15) to get

Original: R
Revised: R

Original: [Equation]
Revised: [Equation]

Original: From the bound on π e in Assumption 5 it follows that for each e ∈ E tr
Revised: From the bound on π e in Assumption 5 it follows that for each e ∈ E tr

Original: [Equation]
Revised: [Equation]

Original: Note that if the condition above equation 66 is not true, then the preceeding condition in equationcannot be true as contribution from one term in the summation itself will exceed (cid:15) .
Revised: Note that if the condition above equation 66 is not true, then the preceeding condition in equationcannot be true as contribution from one term in the summation itself will exceed (cid:15) .

Original: In the above equation 66, we are using the positive square root since RHS has to be greater than or equal to zero.
Revised: In the above equation 66, we are using the positive square root since RHS has to be greater than or equal to zero.

Original: We compute the second derivative of loss w.r.t w
Revised: We compute the second derivative of loss w.r.t w

Original: [Equation]
Revised: [Equation]

Original: Since Σ e is symmetric, we can use the eigenvalue decomposition of Σ e =
Revised: Since Σ e is symmetric, we can use the eigenvalue decomposition of Σ e =

Original: U Λ U T in equationto get Φ T Σ e Φ =
Revised: U Λ U T in equationto get Φ T Σ e Φ =

Original: Φ T U Λ U T Φ .
Revised: Φ T U Λ U T Φ .

Original: Substitute ˜Φ
Revised: Substitute ˜Φ

Original: = U T Φ to get Φ T Σ e Φ = ˜Φ T Λ˜Φ ≥ λ min (Σ e ) (cid:107) ˜Φ (cid:107) 

Revised: = U T Φ to get Φ T Σ e Φ = ˜Φ T Λ˜Φ ≥ λ min (Σ e ) (cid:107) ˜Φ (cid:107)

Original: =
Revised: =

Original: λ min (Σ e )Φ T UU T Φ = λ min (Σ e ) (cid:107) Φ (cid:107) 2 .
Revised: λ min (Σ e )Φ T UU T Φ = λ min (Σ e ) (cid:107) Φ (cid:107) 2 .

Original: From Assumption 5, we have λ min (Σ e ) ≥ λ min and from
Revised: From Assumption 5, we have λ min (Σ e ) ≥ λ min and from

Original: Assumption 7 we have (cid:107) Φ (cid:107) 2 ≥ ω .
Revised: Assumption 7 we have (cid:107) Φ (cid:107) 2 ≥ ω .

Original: Therefore, we can deduce that 2Φ T Σ e Φ ≥ 2 λ min ω .
Revised: Therefore, we can deduce that 2Φ T Σ e Φ ≥ 2 λ min ω .

Original: Therefore, the second derivative deﬁned in equation 67 is always greater than or equal to 2 λ min ω .
Revised: Therefore, the second derivative deﬁned in equation 67 is always greater than or equal to 2 λ min ω .

Original: [Equation]
Revised: [Equation]

Original: Let (cid:15) (cid:48)
Revised: Let (cid:15) (cid:48)

Original: = 2 (cid:113) (cid:15) |E tr | 4 π min = (cid:113) (cid:15) |E tr | π min .
Revised: = 2 (cid:113) (cid:15) |E tr | 4 π min = (cid:113) (cid:15) |E tr | π min .

Original: We rewrite equation 66 in terms of ∇ w | w =1 . 

Revised: We rewrite equation 66 in terms of ∇ w | w =1 .

Original: R e ( w · Φ) and (cid:15) (cid:48) to get
Revised: R e ( w · Φ) and (cid:15) (cid:48) to get

Original: [Equation]
Revised: [Equation]

Original: Since the second derivative (equation 68) is strictly positive and larger than or equal to λ min ω =
Revised: Since the second derivative (equation 68) is strictly positive and larger than or equal to λ min ω =

Original: ⇒ ∃
Revised: ⇒ ∃

Original: w e in the neighborhood of w = 1 . 0 at which ∇ w | w = w e R e ( w · Φ)
Revised: w e in the neighborhood of w = 1 . 0 at which ∇ w | w = w e R e ( w · Φ)

Original: = 0 .
Revised: = 0 .

Original: This holds for all the environments in E tr .
Revised: This holds for all the environments in E tr .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Also, deﬁne c (cid:48) ( w ) =
Revised: Also, deﬁne c (cid:48) ( w ) =

Original: ∂c ( w ) ∂w .
Revised: ∂c ( w ) ∂w .

Original: Also, c (cid:48) ( w ) = ∇ 2 w R e ( w · Φ) .
Revised: Also, c (cid:48) ( w ) = ∇ 2 w R e ( w · Φ) .

Original: Suppose ∇ w | w =1 . 

Revised: Suppose ∇ w | w =1 .

Original: R e ( w · Φ) =
Revised: R e ( w · Φ) =

Original: c (1) < 0 .
Revised: c (1) < 0 .

Original: Since for all w , c (cid:48) ( w ) > 0 (from equation 68), ∃ w e > 1 , where c ( w e ) = 0 .
Revised: Since for all w , c (cid:48) ( w ) > 0 (from equation 68), ∃ w e > 1 , where c ( w e ) = 0 .

Original: Using fundamental theorem of calculus, we write
Revised: Using fundamental theorem of calculus, we write

Original: [Equation]
Revised: [Equation]

Original: Substituting w = w e in the above
Revised: Substituting w = w e in the above

Original: [Equation]
Revised: [Equation]

Original: Suppose ∇ w,w =1 . 

Revised: Suppose ∇ w,w =1 .

Original: R e ( w. Φ) =
Revised: R e ( w. Φ) =

Original: c (1) > 0 .
Revised: c (1) > 0 .

Original: Using fundamental theorem of calculus we can write
Revised: Using fundamental theorem of calculus we can write

Original: [Equation]
Revised: [Equation]

Original: Substituting w = w e in the above
Revised: Substituting w = w e in the above

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Combining equation 70 and equation 71 and using the deﬁnition of η 1 and η 2 , we can conclude that w e ∈ [ 1 1+ η 1 , 1 1+ η 2 ] .
Revised: Combining equation 70 and equation 71 and using the deﬁnition of η 1 and η 2 , we can conclude that w e ∈ [ 1 1+ η 1 , 1 1+ η 2 ] .

Original: If we reparamterize w e = 1 1+ η e , then η e ∈ [ η 2 , η 1 ] .
Revised: If we reparamterize w e = 1 1+ η e , then η e ∈ [ η 2 , η 1 ] .

Original: We expand this condition ∇ w | w = w e R e ( w · Φ)
Revised: We expand this condition ∇ w | w = w e R e ( w · Φ)

Original: = 0 .
Revised: = 0 .

Original: [Equation]
Revised: [Equation]

Original: Assume that Φ 1 1+ η e (cid:54) =
Revised: Assume that Φ 1 1+ η e (cid:54) =

Original: ˜ S T γ , for all e ∈ E tr .
Revised: ˜ S T γ , for all e ∈ E tr .

Original: If this assumption is not true and Φ 1 1+ η e = ˜ S T γ for some e ∈ E tr , then it already establishes the claim we set out to prove in this Proposition (since η e ∈ [ η 2 , η 1 ] ).
Revised: If this assumption is not true and Φ 1 1+ η e = ˜ S T γ for some e ∈ E tr , then it already establishes the claim we set out to prove in this Proposition (since η e ∈ [ η 2 , η 1 ] ).

Original: Deﬁne q e = (cid:104) E e (cid:2)
Revised: Deﬁne q e = (cid:104) E e (cid:2)

Original: X
Revised: X

Original: e X e, T (cid:3)(cid:16)
Revised: e X e, T (cid:3)(cid:16)

Original: Φ 1 1+ η e − ˜ S T γ (cid:17) − E e (cid:2)
Revised: Φ 1 1+ η e − ˜ S T γ (cid:17) − E e (cid:2)

Original: X e ε e (cid:3)(cid:105) .
Revised: X e ε e (cid:3)(cid:105) .

Original: From the Assumption 6 and since Φ 1 1+ η e (cid:54) =
Revised: From the Assumption 6 and since Φ 1 1+ η e (cid:54) =

Original: ˜ S T γ , we know dim (cid:16) span (cid:110)
Revised: ˜ S T γ , we know dim (cid:16) span (cid:110)

Original: q e (cid:111)(cid:17) >
Revised: q e (cid:111)(cid:17) >

Original: n − 1 .
Revised: n − 1 .

Original: From rank-nullity theorem we know dimension of kernel space of Φ (rank of Φ is 1 ) is n − 1 .
Revised: From rank-nullity theorem we know dimension of kernel space of Φ (rank of Φ is 1 ) is n − 1 .

Original: From equation 72 it follows that q e is in kernel space of Φ .
Revised: From equation 72 it follows that q e is in kernel space of Φ .

Original: Therefore, dim ( Ker (Φ)) =
Revised: Therefore, dim ( Ker (Φ)) =

Original: n − 1 = ⇒ dim (cid:16) span (cid:110) q e (cid:111)(cid:17) ≤ n − 1 which leads to a contradiction.
Revised: n − 1 = ⇒ dim (cid:16) span (cid:110) q e (cid:111)(cid:17) ≤ n − 1 which leads to a contradiction.

Original: Therefore, Φ 1 1+ η e = ˜ S T γ at least for one environment.
Revised: Therefore, Φ 1 1+ η e = ˜ S T γ at least for one environment.

Original: If Φ = ˜ S T γ (1 + η e ) , then (cid:107) Φ (cid:107) 

Revised: If Φ = ˜ S T γ (1 + η e ) , then (cid:107) Φ (cid:107)

Original: = (cid:107) ˜
Revised: = (cid:107) ˜

Original: S T γ (cid:107) 2 (1 + η e ) 2 ≥ (cid:107) ˜ S T γ (cid:107) 2 (1 + η 2 ) 2 ≥ (cid:107) ˜
Revised: S T γ (cid:107) 2 (1 + η e ) 2 ≥ (cid:107) ˜ S T γ (cid:107) 2 (1 + η 2 ) 2 ≥ (cid:107) ˜

Original: S T γ (cid:107) 2 12 ≥ ω .
Revised: S T γ (cid:107) 2 12 ≥ ω .

Original: In this simpliﬁcation, we use (1 + η 2 ) 2 ≥ 1 (cid:16)
Revised: In this simpliﬁcation, we use (1 + η 2 ) 2 ≥ 1 (cid:16)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:17)≥ 1 (cid:16)
Revised: (cid:17)≥ 1 (cid:16)

Original: [Equation]
Revised: [Equation]

Original: min (cid:17) 2 = 1 (cid:16) 1+ √ 3 − 2 √ 2 (cid:17) 2 = 12 and (cid:107) ˜
Revised: min (cid:17) 2 = 1 (cid:16) 1+ √ 3 − 2 √ 2 (cid:17) 2 = 12 and (cid:107) ˜

Original: S T γ (cid:107) 2 ≥ 2 ω (from Assumption 7).
Revised: S T γ (cid:107) 2 ≥ 2 ω (from Assumption 7).

Original: This ensures that for any solution of the form Φ 1 1+ η e the assumption (cid:107) Φ (cid:107) 2 ≥ ω is automatically satisﬁed.
Revised: This ensures that for any solution of the form Φ 1 1+ η e the assumption (cid:107) Φ (cid:107) 2 ≥ ω is automatically satisﬁed.

Original: If Φ = ˜ S T γ (1+ η e ) , then (cid:107) Φ (cid:107) 

Revised: If Φ = ˜ S T γ (1+ η e ) , then (cid:107) Φ (cid:107)

Original: = (cid:107) ˜
Revised: = (cid:107) ˜

Original: S T γ (cid:107) 2 (1+ η e ) 2 ≤ (cid:107) ˜ S T γ (cid:107) 2 (1+ η 1 ) 2 ≤ (cid:107) ˜ S
Revised: S T γ (cid:107) 2 (1+ η e ) 2 ≤ (cid:107) ˜ S T γ (cid:107) 2 (1+ η 1 ) 2 ≤ (cid:107) ˜ S

Original: [Equation]
Revised: [Equation]

Original: γ (cid:107)
Revised: γ (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:16) 3+
Revised: (cid:16) 3+

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: (cid:17) ≤ Ω .
Revised: (cid:17) ≤ Ω .

Original: In the above simpliﬁcation, we use (1 + η 1 ) 2 ≤
Revised: In the above simpliﬁcation, we use (1 + η 1 ) 2 ≤

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:16) 1 − √
Revised: 1 (cid:16) 1 − √

Original: [Equation]
Revised: [Equation]

Original: √ 2 (cid:17)
Revised: √ 2 (cid:17)

Original: 2 = 3+2 √ 2and (cid:107) ˜
Revised: 2 = 3+2 √ 2and (cid:107) ˜

Original: S T γ (cid:107) 2 ≤ 2 3+2 √ 2 Ω (from Assumption 7).
Revised: S T γ (cid:107) 2 ≤ 2 3+2 √ 2 Ω (from Assumption 7).

Original: This ensures that for any solution of the form Φ 1 1+ η e the assumption (cid:107) Φ (cid:107) 2 ≤ Ω is automatically satisﬁed.
Revised: This ensures that for any solution of the form Φ 1 1+ η e the assumption (cid:107) Φ (cid:107) 2 ≤ Ω is automatically satisﬁed.

Original: The entire proof so far has characterized the property of Φ that satisﬁes R (cid:48) (Φ) ≤ (cid:15) .
Revised: The entire proof so far has characterized the property of Φ that satisﬁes R (cid:48) (Φ) ≤ (cid:15) .

Original: But how do we know such a Φ exists.
Revised: But how do we know such a Φ exists.

Original: Recall ˜ S
Revised: Recall ˜ S

Original: t γ ∈ H Φ .
Revised: t γ ∈ H Φ .

Original: For each environment e ∈ E tr
Revised: For each environment e ∈ E tr

Original: [Equation]
Revised: [Equation]

Original: We use Assumption 5 in the simpliﬁcation above in equation 

Revised: We use Assumption 5 in the simpliﬁcation above in equation

Original: Therefore, R (cid:48) ( ˜ S t γ ) = 

Revised: Therefore, R (cid:48) ( ˜ S t γ ) =

Original: and as a result the existence of a Φ that satisﬁes R (cid:48) (Φ) ≤ (cid:15) is guaranteed.
Revised: and as a result the existence of a Φ that satisﬁes R (cid:48) (Φ) ≤ (cid:15) is guaranteed.

Original: This completes the proof.
Revised: This completes the proof.

Original: Before proving Proposition 5, we ﬁrst establish that Assumptions 5 and 7 are sufﬁcient to ensure that Assumption 3 holds and we can thus use bounds L and L (cid:48) deﬁned in Assumption 3.
Revised: Before proving Proposition 5, we ﬁrst establish that Assumptions 5 and 7 are sufﬁcient to ensure that Assumption 3 holds and we can thus use bounds L and L (cid:48) deﬁned in Assumption 3.

Original: Proving Assumption 3 for square loss (cid:96) from Assumptions 5, 7.
Revised: Proving Assumption 3 for square loss (cid:96) from Assumptions 5, 7.

Original: From Assumption 5, we have
Revised: From Assumption 5, we have

Original: [Equation]
Revised: [Equation]

Original: From Assumption 5, (cid:107) S (cid:107) is bounded and (cid:107) Z e (cid:107) is bounded =
Revised: From Assumption 5, (cid:107) S (cid:107) is bounded and (cid:107) Z e (cid:107) is bounded =

Original: ⇒ (cid:107)
Revised: ⇒ (cid:107)

Original: X e (cid:107) is bounded as well (from equation 74).
Revised: X e (cid:107) is bounded as well (from equation 74).

Original: Therefore, there exists X sup < ∞ , s.t. (cid:107) X e (cid:107) ≤ X sup .
Revised: Therefore, there exists X sup < ∞ , s.t. (cid:107) X e (cid:107) ≤ X sup .

Original: [Equation]
Revised: [Equation]

Original: In the last step of equation 75, we use (cid:107) ˜
Revised: In the last step of equation 75, we use (cid:107) ˜

Original: S T γ (cid:107) 2 ≤ 2 3+2 √ 2 Ω
Revised: S T γ (cid:107) 2 ≤ 2 3+2 √ 2 Ω

Original: (Assumption 7), (cid:107) X e (cid:107) ≤ X sup derived above (equation 74), and | ε e | ≤ ε sup (Assumption 5).
Revised: (Assumption 7), (cid:107) X e (cid:107) ≤ X sup derived above (equation 74), and | ε e | ≤ ε sup (Assumption 5).

Original: Therefore, Y e is bounded and there exists a
Revised: Therefore, Y e is bounded and there exists a

Original: K such that | Y e | ≤ K ≤ √ Ω X sup + ε sup .
Revised: K such that | Y e | ≤ K ≤ √ Ω X sup + ε sup .

Original: Therefore, ∀ Φ ∈ H Φ and for all X e , Y e sampled from the model in Assumption 5 we have
Revised: Therefore, ∀ Φ ∈ H Φ and for all X e , Y e sampled from the model in Assumption 5 we have

Original: [Equation]
Revised: [Equation]

Original: From equation 76, we conclude that (cid:96) (Φ( · ) , · ) is bounded and there exists an L such that | (cid:96) (Φ( · ) , · ) | ≤ L ≤ ( K + √ Ω X sup ) 2 .
Revised: From equation 76, we conclude that (cid:96) (Φ( · ) , · ) is bounded and there exists an L such that | (cid:96) (Φ( · ) , · ) | ≤ L ≤ ( K + √ Ω X sup ) 2 .

Original: From equation 77, we conclude that ∂(cid:96) ( w. Φ T X
Revised: From equation 77, we conclude that ∂(cid:96) ( w. Φ T X

Original: [Equation]
Revised: [Equation]

Original: ∂w
Revised: ∂w

Original: [Equation]
Revised: [Equation]

Original: is bounded and there exists an L (cid:48)
Revised: is bounded and there exists an L (cid:48)

Original: such that (cid:12)(cid:12)(cid:12)
Revised: such that (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Deﬁne (cid:15) th
Revised: Deﬁne (cid:15) th

Original: = 24 − 16 √ 2π min |E tr | ( ωλ min ) 2 and τ = 1 2 ωλ min (cid:113) 3 |E tr | 2 π min
Revised: = 24 − 16 √ 2π min |E tr | ( ωλ min ) 2 and τ = 1 2 ωλ min (cid:113) 3 |E tr | 2 π min

Original: Next, we prove Proposition 5 from the main body of the manuscript.
Revised: Next, we prove Proposition 5 from the main body of the manuscript.

Original: Proposition 12.
Revised: Proposition 12.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: For every (cid:15) ∈ (0 , (cid:15) th ) and δ ∈ (0 , 1) , if Assumptions 5,(with r = 1 ), 7 hold and if the number of data points | D | is greater than 16 L (cid:48) 4 (cid:15) 2 log (cid:0) 2 |H
Revised: For every (cid:15) ∈ (0 , (cid:15) th ) and δ ∈ (0 , 1) , if Assumptions 5,(with r = 1 ), 7 hold and if the number of data points | D | is greater than 16 L (cid:48) 4 (cid:15) 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ( ˜ S T γ (cid:1) α , where α ∈ [
Revised: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ( ˜ S T γ (cid:1) α , where α ∈ [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne an event
Revised: Deﬁne an event

Original: A : { ˜ D : ∀ Φ ∈ H Φ , | ˆ R
Revised: A : { ˜ D : ∀ Φ ∈ H Φ , | ˆ R

Original: [Equation]
Revised: [Equation]

Original: (Φ) − R
Revised: (Φ) − R

Original: [Equation]
Revised: [Equation]

Original: (Φ) | ≤ (cid:15) 2 } .
Revised: (Φ) | ≤ (cid:15) 2 } .

Original: If event A happens, then
Revised: If event A happens, then

Original: [Equation]
Revised: [Equation]

Original: If event A happens, then every solution ˆΦ to EIRM (equation 6) satisﬁes equation 78.
Revised: If event A happens, then every solution ˆΦ to EIRM (equation 6) satisﬁes equation 78.

Original: From equation 78, we see that we should substitute (cid:15) with 3 (cid:15) 2 and use Proposition 11.
Revised: From equation 78, we see that we should substitute (cid:15) with 3 (cid:15) 2 and use Proposition 11.

Original: If the Assumptions 5, 6 (with r = 1 ), 7 hold and event A happens, then for all 0 < 32 (cid:15) < (cid:15) 

Revised: If the Assumptions 5, 6 (with r = 1 ), 7 hold and event A happens, then for all 0 < 32 (cid:15) < (cid:15)

Original: the output of EIRM equation 6 ˆΦ = ˜ S t ( α ) , where α ∈ [
Revised: the output of EIRM equation 6 ˆΦ = ˜ S t ( α ) , where α ∈ [

Original: [Equation]
Revised: [Equation]

Original: Note that 3 2 (cid:15) < (cid:15) 

Revised: Note that 3 2 (cid:15) < (cid:15)

Original: = ⇒ (cid:15) < (cid:15) th .
Revised: = ⇒ (cid:15) < (cid:15) th .

Original: Hence, all that remains to be shown is that event
Revised: Hence, all that remains to be shown is that event

Original: A occurs with a probability at least 1 − δ .
Revised: A occurs with a probability at least 1 − δ .

Original: Next, we show that if | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 2 |H
Revised: Next, we show that if | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 2 |H

Original: Φ | δ ) , then with probability 1 − δ event
Revised: Φ | δ ) , then with probability 1 − δ event

Original: A happens.
Revised: A happens.

Original: Now we need to bound the probability P ( A ) .
Revised: Now we need to bound the probability P ( A ) .

Original: We will ﬁnd an upper bound on the failure probability using Hoeffding’s inequality (Lemma 1) and the bound L (cid:48) (derived in equation 77) as follows.
Revised: We will ﬁnd an upper bound on the failure probability using Hoeffding’s inequality (Lemma 1) and the bound L (cid:48) (derived in equation 77) as follows.

Original: We redo the same analysis as was done in equation 54 for reader’s convenience.
Revised: We redo the same analysis as was done in equation 54 for reader’s convenience.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Hence, we know that if | D | ≥ 16 L
Revised: Hence, we know that if | D | ≥ 16 L

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 2 log (cid:16) 2 |H
Revised: 2 log (cid:16) 2 |H

Original: Φ | δ (cid:17) , then with probability 1 − δ event
Revised: Φ | δ (cid:17) , then with probability 1 − δ event

Original: A happens.
Revised: A happens.

Original: The proof characterized the property of ˆΦ that satisﬁes ˆ R (cid:48) (ˆΦ) ≤ (cid:15) .
Revised: The proof characterized the property of ˆΦ that satisﬁes ˆ R (cid:48) (ˆΦ) ≤ (cid:15) .

Original: But how do we know such a ˆΦ exists; we show that a ˆΦ always exists.
Revised: But how do we know such a ˆΦ exists; we show that a ˆΦ always exists.

Original: Consider a Φ that satisﬁess the following.
Revised: Consider a Φ that satisﬁess the following.

Original: [Equation]
Revised: [Equation]

Original: ˆ R
Revised: ˆ R

Original: [Equation]
Revised: [Equation]

Original: ( follows from event A )
Revised: ( follows from event A )

Original: [Equation]
Revised: [Equation]

Original: From equation 73 in the proof of Proposition 11, we know that R (cid:48)
Revised: From equation 73 in the proof of Proposition 11, we know that R (cid:48)

Original: ( ˜ S T γ ) = 0 .
Revised: ( ˜ S T γ ) = 0 .

Original: From equation 80, ˜ S T γ ∈ H Φ satisﬁes ˆ R (cid:48) (Φ) ≤ (cid:15) .
Revised: From equation 80, ˜ S T γ ∈ H Φ satisﬁes ˆ R (cid:48) (Φ) ≤ (cid:15) .

Original: Since equation 6 is a constrained optimization, a penalty based version (IRMv1) was proposed by
Revised: Since equation 6 is a constrained optimization, a penalty based version (IRMv1) was proposed by

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2019) that minimizes ˆ R (Φ)
Revised: (2019) that minimizes ˆ R (Φ)

Original: + λ ˆ R (cid:48) (Φ) .
Revised: + λ ˆ R (cid:48) (Φ) .

Original: Both Propositions 4 and Proposition 5 can be extended to IRMv1.
Revised: Both Propositions 4 and Proposition 5 can be extended to IRMv1.

Original: Below we show the sample complexity analysis for IRMv1 applied to the setting assumed in Proposition 5.
Revised: Below we show the sample complexity analysis for IRMv1 applied to the setting assumed in Proposition 5.

Original: Sample complexity analysis of IRMv1 below shows that distance to the OOD solution decays as O ( (cid:112) 1 /λ ) .
Revised: Sample complexity analysis of IRMv1 below shows that distance to the OOD solution decays as O ( (cid:112) 1 /λ ) .

Original: Deﬁne λ th = max { 5 σ 2 / 3 (cid:15) th , 1 }
Revised: Deﬁne λ th = max { 5 σ 2 / 3 (cid:15) th , 1 }

Original: Corollary 1.
Revised: Corollary 1.

Original: For every δ ∈ (0 , 1) , λ > λ th , if Assumptions 5, 6 (with r = 1 ), 7 hold, and | D | ≥ max (cid:8) 64 L (cid:48)
Revised: For every δ ∈ (0 , 1) , λ > λ th , if Assumptions 5, 6 (with r = 1 ), 7 hold, and | D | ≥ max (cid:8) 64 L (cid:48)

Original: 4 λ 2 25 σ 4 log (cid:0) 4 |H
Revised: 4 λ 2 25 σ 4 log (cid:0) 4 |H

Original: Φ | δ (cid:1) , 32 L 2 λ 2 25 σ 4 log( 4 δ ) (cid:9) , then with a probability at least 1 − δ every solution ˆΦ of IRMv1 satisﬁes ˆΦ = (cid:0)
Revised: Φ | δ (cid:1) , 32 L 2 λ 2 25 σ 4 log( 4 δ ) (cid:9) , then with a probability at least 1 − δ every solution ˆΦ of IRMv1 satisﬁes ˆΦ = (cid:0)

Original: ˜ S T γ (cid:1) α , where α ∈ [
Revised: ˜ S T γ (cid:1) α , where α ∈ [

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The empirical version of IRMv1 minimizes
Revised: The empirical version of IRMv1 minimizes

Original: [Equation]
Revised: [Equation]

Original: Let us compute the risk achieved by the ideal invariant predictor ˜ S T γ .
Revised: Let us compute the risk achieved by the ideal invariant predictor ˜ S T γ .

Original: [Equation]
Revised: [Equation]

Original: Deﬁne event A : (cid:110)
Revised: Deﬁne event A : (cid:110)

Original: [Equation]
Revised: [Equation]

Original: If event A holds, then
Revised: If event A holds, then

Original: [Equation]
Revised: [Equation]

Original: Deﬁne event B : (cid:110)
Revised: Deﬁne event B : (cid:110)

Original: [Equation]
Revised: [Equation]

Original: If event B holds, then | ˆ R (cid:48) (cid:0)
Revised: If event B holds, then | ˆ R (cid:48) (cid:0)

Original: ˜ S
Revised: ˜ S

Original: T γ (cid:1) − R (cid:48) (cid:0)
Revised: T γ (cid:1) − R (cid:48) (cid:0)

Original: ˜ S
Revised: ˜ S

Original: T γ (cid:1) | ≤ (cid:15) 2 and if we plug in R (cid:48) (cid:0) ˜ S T γ (cid:1)
Revised: T γ (cid:1) | ≤ (cid:15) 2 and if we plug in R (cid:48) (cid:0) ˜ S T γ (cid:1)

Original: = 

Revised: =

Original: (from equation 73), then
Revised: (from equation 73), then

Original: [Equation]
Revised: [Equation]

Original: Deﬁne success as A ∩ B .
Revised: Deﬁne success as A ∩ B .

Original: If event A ∩ B occurs, then from equation 82 and equation 83 the following is true for any solution Φ of IRMv
Revised: If event A ∩ B occurs, then from equation 82 and equation 83 the following is true for any solution Φ of IRMv

Original: [Equation]
Revised: [Equation]

Original: Since ˆ R (Φ) ≥ 

Revised: Since ˆ R (Φ) ≥

Original: it follows that
Revised: it follows that

Original: [Equation]
Revised: [Equation]

Original: In the last implication in the above equation 85, we use the condition that event B occurs.
Revised: In the last implication in the above equation 85, we use the condition that event B occurs.

Original: Let (cid:15) =
Revised: Let (cid:15) =

Original: σ 2 λ and substitute in the above equation 85 to get
Revised: σ 2 λ and substitute in the above equation 85 to get

Original: [Equation]
Revised: [Equation]

Original: Recall Proposition 11 and see 5 σ 2 2 λ takes the role of (cid:15) .
Revised: Recall Proposition 11 and see 5 σ 2 2 λ takes the role of (cid:15) .

Original: If 5 σ 2 2 λ ≤ (cid:15)= ⇒ λ ≥
Revised: If 5 σ 2 2 λ ≤ (cid:15)= ⇒ λ ≥

Original: [Equation]
Revised: [Equation]

Original: 2 3 (cid:15) th , then the condition in the Proposition 11 is true, then every solution of IRMv1 is ˜ S T γ ( α ) , where α ∈ [
Revised: 2 3 (cid:15) th , then the condition in the Proposition 11 is true, then every solution of IRMv1 is ˜ S T γ ( α ) , where α ∈ [

Original: [Equation]
Revised: [Equation]

Original: We arrived at the above result assuming A ∩ B occurs.
Revised: We arrived at the above result assuming A ∩ B occurs.

Original: We will now show that if | D | ≥ max { 16 L (cid:48)
Revised: We will now show that if | D | ≥ max { 16 L (cid:48)

Original: [Equation]
Revised: [Equation]

Original: 2 σlog( 4 |H Φ | δ ) , 8 L 2 λ 2 σ 4 log( 4 δ ) } , then with a probability at least 1 − δ , A ∩ B occurs.
Revised: 2 σlog( 4 |H Φ | δ ) , 8 L 2 λ 2 σ 4 log( 4 δ ) } , then with a probability at least 1 − δ , A ∩ B occurs.

Original: We write P ( A ) =
Revised: We write P ( A ) =

Original: P (cid:16)(cid:110) D : | ˆ R (cid:0) ˜ S
Revised: P (cid:16)(cid:110) D : | ˆ R (cid:0) ˜ S

Original: T γ (cid:1) − R (cid:0)
Revised: T γ (cid:1) − R (cid:0)

Original: ˜ S T γ (cid:1) | ≤ (cid:15) 2 (cid:111)(cid:17) = 1 − P (cid:16)(cid:110) D : | ˆ R ( ˜
Revised: ˜ S T γ (cid:1) | ≤ (cid:15) 2 (cid:111)(cid:17) = 1 − P (cid:16)(cid:110) D : | ˆ R ( ˜

Original: [Equation]
Revised: [Equation]

Original: T γ )
Revised: T γ )

Original: [Equation]
Revised: [Equation]

Original: ) | >
Revised: ) | >

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:111)(cid:17) .
Revised: 2 (cid:111)(cid:17) .

Original: From Hoeffding’s inequality in Lemma 1 and 3 it follows that
Revised: From Hoeffding’s inequality in Lemma 1 and 3 it follows that

Original: [Equation]
Revised: [Equation]

Original: Next, we will show next that if | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 4 |H
Revised: Next, we will show next that if | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 4 |H

Original: Φ | δ ) , then with probability at least 1 − δ 2 event B happens.
Revised: Φ | δ ) , then with probability at least 1 − δ 2 event B happens.

Original: Now we need to bound the probability P ( B ) .
Revised: Now we need to bound the probability P ( B ) .

Original: We will ﬁnd an upper bound on the failure probability using Hoeffding’s inequality and 3 as follows
Revised: We will ﬁnd an upper bound on the failure probability using Hoeffding’s inequality and 3 as follows

Original: [Equation]
Revised: [Equation]

Original: Hence, we know that if | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 4 |H
Revised: Hence, we know that if | D | ≥ 16 L (cid:48) 4 (cid:15) 2 log( 4 |H

Original: Φ | δ ) , then with a probability at least 1 − δ 2 event B happens.
Revised: Φ | δ ) , then with a probability at least 1 − δ 2 event B happens.

Original: Combining equation 87 and equation 88 we get if
Revised: Combining equation 87 and equation 88 we get if

Original: [Equation]
Revised: [Equation]

Original: then A ∩ B occurs with at least 1 − δ probability.
Revised: then A ∩ B occurs with at least 1 − δ probability.

Original: Substitute (cid:15) =
Revised: Substitute (cid:15) =

Original: 5 σ 2 2 λ
Revised: 5 σ 2 2 λ

Original: and this completes the proof.
Revised: and this completes the proof.

Original: 7. E XTENSIONS : P OLYNOMIAL MODELS , INFINITE HYPOTHESIS CLASSES , BINARY - CLASSIFICATION 7.4. P OLYNOMIAL MODEL
Revised: 7. E XTENSIONS : P OLYNOMIAL MODELS , INFINITE HYPOTHESIS CLASSES , BINARY - CLASSIFICATION 7.4. P OLYNOMIAL MODEL

Original: In the main body of the work, we did not cover the polynomial model in detail due to space limitations.
Revised: In the main body of the work, we did not cover the polynomial model in detail due to space limitations.

Original: In this section, we study the polynomial model and show how the results for the linear model can be generalized to this case.
Revised: In this section, we study the polynomial model and show how the results for the linear model can be generalized to this case.

Original: We ﬁrst begin by stating the model itself.
Revised: We ﬁrst begin by stating the model itself.

Original: Assumption 8.
Revised: Assumption 8.

Original: [Equation]
Revised: [Equation]