Original: Assume that Z e 1 component of S is invertible, i.e. ∃ ˜ S such that ˜ S ( S ( Z e 1 , Z e 2 ))
Revised: Assume that Z e 1 component of S is invertible, i.e. ∃ ˜ S such that ˜ S ( S ( Z e 1 , Z e 2 ))

Original: =
Revised: =

Original: Z e 1 and also ˜ S t γ (cid:54) = 0 .
Revised: Z e 1 and also ˜ S t γ (cid:54) = 0 .

Original: In the above ζ ap is a polynomial feature map of degree p deﬁned as ζ ap : R a → R a (cid:48) , where a denotes the dimension of the input to the map ζ ap , ζ ap ( W ) =
Revised: In the above ζ ap is a polynomial feature map of degree p deﬁned as ζ ap : R a → R a (cid:48) , where a denotes the dimension of the input to the map ζ ap , ζ ap ( W ) =

Original: [ W, W ⊗ W, . .
Revised: [ W, W ⊗ W, . .

Original: , ( W ⊗ W...
Revised: , ( W ⊗ W...

Original: p times ⊗
Revised: p times ⊗

Original: W )]
Revised: W )]

Original: =
Revised: =

Original: [( W ⊗ i ) pi =1 ] and ⊗ is the Kronecker product.
Revised: [( W ⊗ i ) pi =1 ] and ⊗ is the Kronecker product.

Original: Also, a (cid:48) =
Revised: Also, a (cid:48) =

Original: (cid:80) pi
Revised: (cid:80) pi

Original: =1 a i .
Revised: =1 a i .

Original: ∀ e ∈ E tr , π e ≥ π min |E tr | .
Revised: ∀ e ∈ E tr , π e ≥ π min |E tr | .

Original: The support of distribution of Z e =
Revised: The support of distribution of Z e =

Original: [ Z e 1 , Z e 2 ] , P eZ e , is bounded and the operator norm of S , (cid:107) S (cid:107) =
Revised: [ Z e 1 , Z e 2 ] , P eZ e , is bounded and the operator norm of S , (cid:107) S (cid:107) =

Original: σ max ( S ) ( σ max ( S ) is maximum singular value of S ), is also bounded.
Revised: σ max ( S ) ( σ max ( S ) is maximum singular value of S ), is also bounded.

Original: Deﬁne Z e =
Revised: Deﬁne Z e =

Original: ( Z e 1 , Z e 2 ) and say Z e 1 ∈ R c and Z e 2 ∈ R d .
Revised: ( Z e 1 , Z e 2 ) and say Z e 1 ∈ R c and Z e 2 ∈ R d .

Original: Can we directly use the analysis from the linear case?
Revised: Can we directly use the analysis from the linear case?

Original: We cannot directly use the polynomial map for the features as we also need to ﬁnd an appropriate transformation of the matrix S , which preserves the linear relationship between the transformed features and the transformed variables Z .
Revised: We cannot directly use the polynomial map for the features as we also need to ﬁnd an appropriate transformation of the matrix S , which preserves the linear relationship between the transformed features and the transformed variables Z .

Original: We carry out this exercise below.
Revised: We carry out this exercise below.

Original: Deﬁne ¯ S = diag
Revised: Deﬁne ¯ S = diag

Original: (cid:104)
Revised: (cid:104)

Original: ( S ⊗ i ) pi =1 (cid:105) , where ¯ S is a block diagonal matrix with diagonal matrices deﬁning it given as (cid:110) S, S ⊗ 2 . . .
Revised: ( S ⊗ i ) pi =1 (cid:105) , where ¯ S is a block diagonal matrix with diagonal matrices deﬁning it given as (cid:110) S, S ⊗ 2 . . .

Original: , S ⊗ p (cid:111) .
Revised: , S ⊗ p (cid:111) .

Original: Deﬁne ¯ X e =
Revised: Deﬁne ¯ X e =

Original: ζ np (cid:0) X e (cid:1) , where ζ np ( X e ) is the polynomial feature map of degree p of n dimensional input X e .
Revised: ζ np (cid:0) X e (cid:1) , where ζ np ( X e ) is the polynomial feature map of degree p of n dimensional input X e .

Original: Similarly, deﬁne ¯ Z e 1 = ζ cp ( Z e 1 ) , where ζ cp ( Z e 1 ) is the polynomial feature map of degree p of c dimensional input Z e 1 and deﬁne ¯ Z e = ζ c + d p ( Z e ) , where ζ c + d p ( Z e ) is the polynomial feature map of degree p of c + d dimensional input Z e
Revised: Similarly, deﬁne ¯ Z e 1 = ζ cp ( Z e 1 ) , where ζ cp ( Z e 1 ) is the polynomial feature map of degree p of c dimensional input Z e 1 and deﬁne ¯ Z e = ζ c + d p ( Z e ) , where ζ c + d p ( Z e ) is the polynomial feature map of degree p of c + d dimensional input Z e

Original: =
Revised: =

Original: ( Z e 1 , Z e 2 ) .
Revised: ( Z e 1 , Z e 2 ) .

Original: Observe that each component of ¯ Z e 1 is also in ¯ Z .
Revised: Observe that each component of ¯ Z e 1 is also in ¯ Z .

Original: From the model we know that X e = SZ e .
Revised: From the model we know that X e = SZ e .

Original: We would like to remind the reader of the mixed product property of tensors.
Revised: We would like to remind the reader of the mixed product property of tensors.

Original: Consider matrices A ∈ R i × j , B ∈ R k × l C ∈ R
Revised: Consider matrices A ∈ R i × j , B ∈ R k × l C ∈ R

Original: j × p , D ∈
Revised: j × p , D ∈

Original: [Equation]
Revised: [Equation]

Original: In the expressions that follow
Revised: In the expressions that follow

Original: [Equation]
Revised: [Equation]

Original: we exploit the mixed-product property of Kronecker product stated above.
Revised: we exploit the mixed-product property of Kronecker product stated above.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The dimensionality of ¯ X e is n
Revised: The dimensionality of ¯ X e is n

Original: [Equation]
Revised: [Equation]

Original: Assumption 9.
Revised: Assumption 9.

Original: Inductve bias.
Revised: Inductve bias.

Original: H Φ is a ﬁnite set of linear models (bounded) parametrized by Φ ∈
Revised: H Φ is a ﬁnite set of linear models (bounded) parametrized by Φ ∈

Original: R n (cid:48) .
Revised: R n (cid:48) .

Original: ¯˜ S
Revised: ¯˜ S

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: We next compute the norm of ¯ S in terms of the norm of S .
Revised: We next compute the norm of ¯ S in terms of the norm of S .

Original: Recall we are using operator norm deﬁned as (cid:107) S (cid:107) =
Revised: Recall we are using operator norm deﬁned as (cid:107) S (cid:107) =

Original: σ max ( S ) .
Revised: σ max ( S ) .

Original: (cid:107) ¯ S (cid:107) =
Revised: (cid:107) ¯ S (cid:107) =

Original: (cid:13)(cid:13)(cid:13) diag (cid:104) ( S ⊗ i ) pi =1 (cid:105)(cid:13)(cid:13)(cid:13) .
Revised: (cid:13)(cid:13)(cid:13) diag (cid:104) ( S ⊗ i ) pi =1 (cid:105)(cid:13)(cid:13)(cid:13) .

Original: Since ¯ S is a diagonal matrix (cid:107) ¯ S (cid:107) = max i =1 ,..,p {(cid:107) S ⊗ i (cid:107)} .
Revised: Since ¯ S is a diagonal matrix (cid:107) ¯ S (cid:107) = max i =1 ,..,p {(cid:107) S ⊗ i (cid:107)} .

Original: Also, note that (cid:107) S ⊗ i (cid:107)
Revised: Also, note that (cid:107) S ⊗ i (cid:107)

Original: =
Revised: =

Original: (cid:107) S (cid:107) i
Revised: (cid:107) S (cid:107) i

Original: (Laub, 2005).
Revised: (Laub, 2005).

Original: Therefore, (cid:107) ¯ S (cid:107) = max i =1 ....,p {(cid:107) S (cid:107) i } .
Revised: Therefore, (cid:107) ¯ S (cid:107) = max i =1 ....,p {(cid:107) S (cid:107) i } .

Original: Hence, if (cid:107) S (cid:107) is bounded, (cid:107) ¯ S (cid:107) is also bounded.
Revised: Hence, if (cid:107) S (cid:107) is bounded, (cid:107) ¯ S (cid:107) is also bounded.

Original: Also, (cid:107) ¯ Z e (cid:107) 

Revised: Also, (cid:107) ¯ Z e (cid:107)

Original: =
Revised: =

Original: (cid:80) i (cid:107) Z e, ⊗ i (cid:107) 2 .
Revised: (cid:80) i (cid:107) Z e, ⊗ i (cid:107) 2 .

Original: Observe that (cid:107) Z e, ⊗ i (cid:107)
Revised: Observe that (cid:107) Z e, ⊗ i (cid:107)

Original: =
Revised: =

Original: (cid:107) Z e (cid:107) i .
Revised: (cid:107) Z e (cid:107) i .

Original: Hence, if (cid:107) Z e (cid:107) is bounded, (cid:107) ¯ Z e (cid:107) is also bounded.
Revised: Hence, if (cid:107) Z e (cid:107) is bounded, (cid:107) ¯ Z e (cid:107) is also bounded.

Original: Since ¯ X e = ¯ S ¯ Z e .
Revised: Since ¯ X e = ¯ S ¯ Z e .

Original: We can conclude that (cid:107) ¯ X e (cid:107) is also bounded.
Revised: We can conclude that (cid:107) ¯ X e (cid:107) is also bounded.

Original: We can now follow the same line of reasoning as in equation 74, equation 75,equation 76, equation 77 to conclude that the loss and the gradient of the loss are bounded.
Revised: We can now follow the same line of reasoning as in equation 74, equation 75,equation 76, equation 77 to conclude that the loss and the gradient of the loss are bounded.

Original: We rewrite the above model in Assumption 8 as a linear model in terms of the transformed features.
Revised: We rewrite the above model in Assumption 8 as a linear model in terms of the transformed features.

Original: [Equation]
Revised: [Equation]

Original: We showed above in equation 92 that ¯ Z e 1 deﬁned in equation 91 component of ¯ Z e is invertible, ¯˜ S ¯ S ¯ Z e = ¯ Z e 1 .
Revised: We showed above in equation 92 that ¯ Z e 1 deﬁned in equation 91 component of ¯ Z e is invertible, ¯˜ S ¯ S ¯ Z e = ¯ Z e 1 .

Original: We have also shown above that support of ¯ Z e is bounded and the norm of ¯ S is bounded and as a result (cid:107)
Revised: We have also shown above that support of ¯ Z e is bounded and the norm of ¯ S is bounded and as a result (cid:107)

Original: ¯ X e (cid:107) and the loss and the gradient of the loss (conditions in Assumptionare satisﬁed) are bounded.
Revised: ¯ X e (cid:107) and the loss and the gradient of the loss (conditions in Assumptionare satisﬁed) are bounded.

Original: We adapt the linear general position assumption (Assumption 6) for the polynomial case below.
Revised: We adapt the linear general position assumption (Assumption 6) for the polynomial case below.

Original: Assumption 10.
Revised: Assumption 10.

Original: Linear general position of training environments.
Revised: Linear general position of training environments.

Original: A set of training environments
Revised: A set of training environments

Original: E tr is said to lie in a linear general position of degree r for some r ∈ N if |E tr | > n (cid:48) − r + n (cid:48) /r and for non-zero x ∈ R n (cid:48) and
Revised: E tr is said to lie in a linear general position of degree r for some r ∈ N if |E tr | > n (cid:48) − r + n (cid:48) /r and for non-zero x ∈ R n (cid:48) and

Original: [Equation]
Revised: [Equation]

Original: We denote E [ ¯ X e ¯ X e, T ] =
Revised: We denote E [ ¯ X e ¯ X e, T ] =

Original: ¯Σ e
Revised: ¯Σ e

Original: Assumption 11.
Revised: Assumption 11.

Original: For all the environments e ∈ E tr , ¯Σ e is positive deﬁnite.
Revised: For all the environments e ∈ E tr , ¯Σ e is positive deﬁnite.

Original: Deﬁne the minimum eigenvalue over all the matrices ¯Σ e as ¯ λ min = min
Revised: Deﬁne the minimum eigenvalue over all the matrices ¯Σ e as ¯ λ min = min

Original: [Equation]
Revised: [Equation]

Original: ∈E
Revised: ∈E

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: From the analysis in this section, we see that we have been able to construct a linear model identical to 5, where the role of X e , Z e , S , ˜ S , Σ e , λ min , (cid:15) th is taken by ¯ X e , ¯ Z e , ¯ S , ¯˜ S , ¯Σ e , ¯ λ min , ¯ (cid:15) th .
Revised: From the analysis in this section, we see that we have been able to construct a linear model identical to 5, where the role of X e , Z e , S , ˜ S , Σ e , λ min , (cid:15) th is taken by ¯ X e , ¯ Z e , ¯ S , ¯˜ S , ¯Σ e , ¯ λ min , ¯ (cid:15) th .

Original: Now we are ready to use the result already proven for linear model and state the next Proposition in terms of the parameters for the polynomial model.
Revised: Now we are ready to use the result already proven for linear model and state the next Proposition in terms of the parameters for the polynomial model.

Original: Proposition 13.
Revised: Proposition 13.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: For every (cid:15) ∈ (0 , ¯ (cid:15) th ) , δ ∈ (0 , 1) , if Assumptions 8, 9, 10 (with r = 1 ), 11 hold and if the number of data points | D | is greater than 16 L (cid:48) 4 (cid:15) 2 log (cid:0) 2 |H
Revised: For every (cid:15) ∈ (0 , ¯ (cid:15) th ) , δ ∈ (0 , 1) , if Assumptions 8, 9, 10 (with r = 1 ), 11 hold and if the number of data points | D | is greater than 16 L (cid:48) 4 (cid:15) 2 log (cid:0) 2 |H

Original: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ( ¯˜ S T γ (cid:1) α , where α ∈ [ 1 1+ τ √ (cid:15) , 1 1 − τ √ (cid:15) ] .
Revised: Φ | δ (cid:1) , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ( ¯˜ S T γ (cid:1) α , where α ∈ [ 1 1+ τ √ (cid:15) , 1 1 − τ √ (cid:15) ] .

Original: 7.4. I NFINITE H YPOTHESIS C LASSES
Revised: 7.4. I NFINITE H YPOTHESIS C LASSES

Original: In the work so far we have assumed that the hypothesis class H
Revised: In the work so far we have assumed that the hypothesis class H

Original: Φ is ﬁnite.
Revised: Φ is ﬁnite.

Original: In this section, we discuss inﬁnite hypothesis class extensions.
Revised: In this section, we discuss inﬁnite hypothesis class extensions.

Original: Before we do that we state an important result on covering numbers that we will use soon.
Revised: Before we do that we state an important result on covering numbers that we will use soon.

Original: Lemma 5.
Revised: Lemma 5.

Original: (Shalev-Shwartz & Ben-David, 2014) Deﬁne a set
Revised: (Shalev-Shwartz & Ben-David, 2014) Deﬁne a set

Original: A = { a ∈ R k , (cid:107) a (cid:107) 2 ≤
Revised: A = { a ∈ R k , (cid:107) a (cid:107) 2 ≤

Original: A sup } .
Revised: A sup } .

Original: Covering number for η -cover of A given as N η ( A ) is bounded as
Revised: Covering number for η -cover of A given as N η ( A ) is bounded as

Original: [Equation]
Revised: [Equation]

Original: A. Inﬁnite Hypothesis Class: Confounders and Anti-causal variables
Revised: A. Inﬁnite Hypothesis Class: Confounders and Anti-causal variables

Original: In this section, we seek to extend Proposition 5 to inﬁnite hypothesis classes.
Revised: In this section, we seek to extend Proposition 5 to inﬁnite hypothesis classes.

Original: We restate the Assumption 7 for linear models.
Revised: We restate the Assumption 7 for linear models.

Original: Assumption 12.
Revised: Assumption 12.

Original: Inductve bias.
Revised: Inductve bias.

Original: H Φ is a set of linear models (bounded) parametrized by Φ ∈ R n
Revised: H Φ is a set of linear models (bounded) parametrized by Φ ∈ R n

Original: [Equation]
Revised: [Equation]

Original: Note that the only difference between Assumption 7 and Assumption 12 is that the hypothesis class is not required to be ﬁnite anymore.
Revised: Note that the only difference between Assumption 7 and Assumption 12 is that the hypothesis class is not required to be ﬁnite anymore.

Original: We already established in equation 75, equation 76, equation 77 that from Assumptions 5 and 7, we can show that the conditions in Assumption 3 hold, i.e., loss and the gradient of the loss are bounded, and also X e , Y e are bounded .
Revised: We already established in equation 75, equation 76, equation 77 that from Assumptions 5 and 7, we can show that the conditions in Assumption 3 hold, i.e., loss and the gradient of the loss are bounded, and also X e , Y e are bounded .

Original: The same conclusion follows from Assumptions 5 and Assumption 12.
Revised: The same conclusion follows from Assumptions 5 and Assumption 12.

Original: Hence, for the rest of this section, we can state that (cid:107) X e (cid:107) ≤ X sup , | Y e | ≤ K , the square loss (cid:96) (Φ( · ) , · ) is bounded by L and ∂(cid:96) ( w · (Φ( · ) , · ) ∂w
Revised: Hence, for the rest of this section, we can state that (cid:107) X e (cid:107) ≤ X sup , | Y e | ≤ K , the square loss (cid:96) (Φ( · ) , · ) is bounded by L and ∂(cid:96) ( w · (Φ( · ) , · ) ∂w

Original: [Equation]
Revised: [Equation]

Original: w =1 . 0 is bounded by L (cid:48) .
Revised: w =1 . 0 is bounded by L (cid:48) .

Original: In the next lemma, we aim to show that if Assumption 5 and 12 hold, then R (cid:48) (Φ) is Lipschitz continuous.
Revised: In the next lemma, we aim to show that if Assumption 5 and 12 hold, then R (cid:48) (Φ) is Lipschitz continuous.

Original: Lemma 6.
Revised: Lemma 6.

Original: If Assumption 5 and 12 hold, then R (cid:48) (Φ) is Lipschitz continuous in Φ .
Revised: If Assumption 5 and 12 hold, then R (cid:48) (Φ) is Lipschitz continuous in Φ .

Original: Proof.
Revised: Proof.

Original: The output of the model | Φ T X | ≤ (cid:107) Φ (cid:107)(cid:107) X (cid:107) ≤ √ Ω X sup (From Cauchy-Schwarz).
Revised: The output of the model | Φ T X | ≤ (cid:107) Φ (cid:107)(cid:107) X (cid:107) ≤ √ Ω X sup (From Cauchy-Schwarz).

Original: [Equation]
Revised: [Equation]

Original: We bound each term in the summation in the equation 96 above
Revised: We bound each term in the summation in the equation 96 above

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] 
Revised: [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We bound each term in the last line in equation
Revised: We bound each term in the last line in equation

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Substituting equation 98 and equation 99 in equation 97 to get
Revised: Substituting equation 98 and equation 99 in equation 97 to get

Original: [Equation]
Revised: [Equation]

Original: Therefore, R (cid:48) is Lipschitz with a constant C (cid:48) ≤ 2 √ Ω
Revised: Therefore, R (cid:48) is Lipschitz with a constant C (cid:48) ≤ 2 √ Ω

Original: ( X sup ) 2 ( √ Ω X sup + K )(2 √ Ω X sup + K ) .
Revised: ( X sup ) 2 ( √ Ω X sup + K )(2 √ Ω X sup + K ) .

Original: We just showed that R (cid:48) is Lipschitz continuous and we set its Lipschitz constant as C (cid:48) .
Revised: We just showed that R (cid:48) is Lipschitz continuous and we set its Lipschitz constant as C (cid:48) .

Original: Proposition 14.
Revised: Proposition 14.

Original: Let (cid:96) be the square loss.
Revised: Let (cid:96) be the square loss.

Original: For every (cid:15) ∈ (0 , (cid:15) th ) and δ ∈ (0 , 1) , if Assumptions 5,(with r = 1 ), 12 hold and if the number of samples | D | is greater than 32 L
Revised: For every (cid:15) ∈ (0 , (cid:15) th ) and δ ∈ (0 , 1) , if Assumptions 5,(with r = 1 ), 12 hold and if the number of samples | D | is greater than 32 L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: + log (cid:16) 2 δ (cid:17)(cid:105) , then with a probability at least 1 − δ every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ˜ S T γ ( α ) , where α ∈ [ 1 1+ τ √ (cid:15) , 1 1 − τ √ (cid:15) ] .
Revised: + log (cid:16) 2 δ (cid:17)(cid:105) , then with a probability at least 1 − δ every solution ˆΦ to EIRM (equation 6) satisﬁes ˆΦ = ˜ S T γ ( α ) , where α ∈ [ 1 1+ τ √ (cid:15) , 1 1 − τ √ (cid:15) ] .

Original: Proof.
Revised: Proof.

Original: Following the proof of Proposition 5, our goal is to compute the probability of event A : {∀ Φ ∈ H Φ , | ˆ R (cid:48) (Φ) − R (cid:48) (Φ) | ≤ (cid:15) 2 } .
Revised: Following the proof of Proposition 5, our goal is to compute the probability of event A : {∀ Φ ∈ H Φ , | ˆ R (cid:48) (Φ) − R (cid:48) (Φ) | ≤ (cid:15) 2 } .

Original: We construct a minimum cover of size N η ( H Φ )
Revised: We construct a minimum cover of size N η ( H Φ )

Original: (See Lemma 5) with points C = { Φ j } bj =1 .
Revised: (See Lemma 5) with points C = { Φ j } bj =1 .

Original: Compute the probability of failure at one point Φ j in the cover
Revised: Compute the probability of failure at one point Φ j in the cover

Original: [Equation]
Revised: [Equation]

Original: We use union bound to bound the probability of failure over the cover C as follows
Revised: We use union bound to bound the probability of failure over the cover C as follows

Original: [Equation]
Revised: [Equation]

Original: Now consider any Φ ∈ H Φ and suppose Φ j is nearest point to it in the cover.
Revised: Now consider any Φ ∈ H Φ and suppose Φ j is nearest point to it in the cover.

Original: [Equation]
Revised: [Equation]

Original: In the above simpliﬁcation, we exploited the Lipschitz continuity of R (cid:48) .
Revised: In the above simpliﬁcation, we exploited the Lipschitz continuity of R (cid:48) .

Original: Therefore, for each Φ ∈ H Φ
Revised: Therefore, for each Φ ∈ H Φ

Original: [Equation]
Revised: [Equation]

Original: Set η = (cid:15) 8 C (cid:48) in equation 104 and from equation 102 with probability at least 1 − N η
Revised: Set η = (cid:15) 8 C (cid:48) in equation 104 and from equation 102 with probability at least 1 − N η

Original: ( H Φ )2 e − (cid:15) 2 | D | 32 L (cid:48)max
Revised: ( H Φ )2 e − (cid:15) 2 | D | 32 L (cid:48)max

Original: [Equation]
Revised: [Equation]

Original: (since max Φ j
Revised: (since max Φ j

Original: [Equation]
Revised: [Equation]

Original: | R (cid:48)
Revised: | R (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: We bound N η
Revised: We bound N η

Original: ( H Φ )2 e − (cid:15) 2 | D | 32 L (cid:48) 4 ≤ δ and solve for bound on | D | to get
Revised: ( H Φ )2 e − (cid:15) 2 | D | 32 L (cid:48) 4 ≤ δ and solve for bound on | D | to get

Original: [Equation]
Revised: [Equation]

Original: Use Lemma
Revised: Use Lemma

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, if condition in equation 106 holds, then event A occurs and following the same argument as in the proof of Proposition 5 the proof is complete.
Revised: Therefore, if condition in equation 106 holds, then event A occurs and following the same argument as in the proof of Proposition 5 the proof is complete.

Original: B. Inﬁnite hypothesis class: Lipschitz continuous functions
Revised: B. Inﬁnite hypothesis class: Lipschitz continuous functions

Original: In this section, we seek to extend Proposition 2 and 4 to inﬁnite hypothesis class of Lipschitz continuous functions that we formally deﬁne next.
Revised: In this section, we seek to extend Proposition 2 and 4 to inﬁnite hypothesis class of Lipschitz continuous functions that we formally deﬁne next.

Original: Deﬁne a map Φ : P × X → R from the parameter space P and the feature space X to reals.
Revised: Deﬁne a map Φ : P × X → R from the parameter space P and the feature space X to reals.

Original: Each p ∈ P is a possible choice for the representation Φ( p, · ) .
Revised: Each p ∈ P is a possible choice for the representation Φ( p, · ) .

Original: Consider neural networks as an example, P represents the set of the values the weights of the network can take.
Revised: Consider neural networks as an example, P represents the set of the values the weights of the network can take.

Original: Assumption 13.
Revised: Assumption 13.

Original: Φ : P × X
Revised: Φ : P × X

Original: → R is a a Lipschitz continuous function (with Lipschitz constant say
Revised: → R is a a Lipschitz continuous function (with Lipschitz constant say

Original: Q ).
Revised: Q ).

Original: P ⊂ R k is closed and bounded, thus there exists a P < ∞ such that ∀ p ∈ P , (cid:107) p (cid:107) 2 ≤ P .
Revised: P ⊂ R k is closed and bounded, thus there exists a P < ∞ such that ∀ p ∈ P , (cid:107) p (cid:107) 2 ≤ P .

Original: X ⊂ R n is closed and bounded, thus there exists a X sup < ∞ such that ∀ x ∈ X , (cid:107) x (cid:107) ≤ X sup .
Revised: X ⊂ R n is closed and bounded, thus there exists a X sup < ∞ such that ∀ x ∈ X , (cid:107) x (cid:107) ≤ X sup .

Original: Y ⊂ R is closed and bounded, thus there exists a K < ∞ such that ∀ y ∈ Y , | y | ≤ K
Revised: Y ⊂ R is closed and bounded, thus there exists a K < ∞ such that ∀ y ∈ Y , | y | ≤ K

Original: Assumption 14.
Revised: Assumption 14.

Original: Lipschitz loss and gradient of loss.
Revised: Lipschitz loss and gradient of loss.

Original: R (Φ) is Lipschitz with a constant C , R (cid:48) (Φ) is
Revised: R (Φ) is Lipschitz with a constant C , R (cid:48) (Φ) is

Original: Lipchitz with a constant C (cid:48) .
Revised: Lipchitz with a constant C (cid:48) .

Original: From Assumption 13 derive the conditions in Assumption 3 and Assumption 14 Φ : P×X → R is a continuous function deﬁned over closed and bounded domain P × X (domain is compact) and as a result Φ is bounded say by M .
Revised: From Assumption 13 derive the conditions in Assumption 3 and Assumption 14 Φ : P×X → R is a continuous function deﬁned over closed and bounded domain P × X (domain is compact) and as a result Φ is bounded say by M .

Original: Consider square loss (cid:96) (Φ( p, X ) , Y ) =
Revised: Consider square loss (cid:96) (Φ( p, X ) , Y ) =

Original: ( Y − Φ( p, X )) 2 ≤ ( M + K ) 2 .
Revised: ( Y − Φ( p, X )) 2 ≤ ( M + K ) 2 .

Original: Hence, there exists an L such that | (cid:96) (Φ( · ) , · ) | ≤ L ≤ ( M + K ) 2 .
Revised: Hence, there exists an L such that | (cid:96) (Φ( · ) , · ) | ≤ L ≤ ( M + K ) 2 .

Original: [Equation]
Revised: [Equation]

Original: is
Revised: is

Original: [Equation]
Revised: [Equation]

Original: Hence, there exists an L (cid:48) such that | ∂(cid:96) ( w. Φ(
Revised: Hence, there exists an L (cid:48) such that | ∂(cid:96) ( w. Φ(

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) ∂w
Revised: ) ∂w

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.
Revised: Lemma 7.

Original: If Assumption 13 holds, then R (Φ( p, · )) and R (cid:48)
Revised: If Assumption 13 holds, then R (Φ( p, · )) and R (cid:48)

Original: (Φ( p, · )) are Lipschitz continuous in p .
Revised: (Φ( p, · )) are Lipschitz continuous in p .

Original: R is Lipschitz:
Revised: R is Lipschitz:

Original: [Equation]
Revised: [Equation]

Original: Therefore, R is Lipschitz with a constant C ≤ 2( M + K ) Q
Revised: Therefore, R is Lipschitz with a constant C ≤ 2( M + K ) Q

Original: R (cid:48) is Lipschitz:
Revised: R (cid:48) is Lipschitz:

Original: [Equation]
Revised: [Equation]

Original: We bound each term in the summation in the equation 96 above
Revised: We bound each term in the summation in the equation 96 above

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Substituting equation 110 and equation 111 in equation 109 to get
Revised: Substituting equation 110 and equation 111 in equation 109 to get

Original: [Equation]
Revised: [Equation]

Original: Therefore, R (cid:48) is Lipschitz with a constant C (cid:48)
Revised: Therefore, R (cid:48) is Lipschitz with a constant C (cid:48)

Original: ≤ 2 M ( M + K )(2 M + K ) Q .
Revised: ≤ 2 M ( M + K )(2 M + K ) Q .

Original: C. EIRM: Sample complexity with no distributional assumptions
Revised: C. EIRM: Sample complexity with no distributional assumptions

Original: In this section, we discuss the extension of Proposition 2 to the inﬁnite hypothesis class case.
Revised: In this section, we discuss the extension of Proposition 2 to the inﬁnite hypothesis class case.

Original: Consider the problem in equation 6 and replace the (cid:15) with (cid:15) + κ .
Revised: Consider the problem in equation 6 and replace the (cid:15) with (cid:15) + κ .

Original: Deﬁne distance between the two sets
Revised: Deﬁne distance between the two sets

Original: S IV ( (cid:15) ) and its approximation S IV ( (cid:15) + κ ) as follows
Revised: S IV ( (cid:15) ) and its approximation S IV ( (cid:15) + κ ) as follows

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where d ( g, h ) is some metric that measures the distance between functions g and h .
Revised: where d ( g, h ) is some metric that measures the distance between functions g and h .

Original: Observe that if a ≤ b , dis ( a ) ≤ dis ( b ) .
Revised: Observe that if a ≤ b , dis ( a ) ≤ dis ( b ) .

Original: Assumption 15.
Revised: Assumption 15.

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Proposition 15.
Revised: Proposition 15.

Original: For every ν > 0 and δ ∈ (0 , 1) , if Assumption 13, 15 hold, then ∃ κ > 0 such that if the number of samples | D | is greater than D ∗ , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (replace (cid:15) with (cid:15) + κ inequation 6) in S IV ( (cid:15) + 2 κ ) and | R (ˆΦ) − R
Revised: For every ν > 0 and δ ∈ (0 , 1) , if Assumption 13, 15 hold, then ∃ κ > 0 such that if the number of samples | D | is greater than D ∗ , then with a probability at least 1 − δ , every solution ˆΦ to EIRM (replace (cid:15) with (cid:15) + κ inequation 6) in S IV ( (cid:15) + 2 κ ) and | R (ˆΦ) − R

Original: Φ ∗ ) | ≤ ν , where Φ ∗ is a solution of IRM in equation 5.
Revised: Φ ∗ ) | ≤ ν , where Φ ∗ is a solution of IRM in equation 5.

Original: Proof.
Revised: Proof.

Original: We divide the proof in two parts.
Revised: We divide the proof in two parts.

Original: Deﬁne an event
Revised: Deﬁne an event

Original: A : { ˜ D : ∀ p ∈ P , | R (cid:48) (Φ( p, · ))
Revised: A : { ˜ D : ∀ p ∈ P , | R (cid:48) (Φ( p, · ))

Original: − ˆ R (cid:48) (Φ( p, · ))
Revised: − ˆ R (cid:48) (Φ( p, · ))

Original: | ≤ κ } .
Revised: | ≤ κ } .

Original: In the ﬁrst half, we will show that if event A occurs, then S IV ( (cid:15) ) ⊆
Revised: In the ﬁrst half, we will show that if event A occurs, then S IV ( (cid:15) ) ⊆

Original: ˆ S IV ( (cid:15) + κ ) ⊆
Revised: ˆ S IV ( (cid:15) + κ ) ⊆

Original: S IV ( (cid:15) + 2 κ ) and then bound the probability of A not occuring.
Revised: S IV ( (cid:15) + 2 κ ) and then bound the probability of A not occuring.

Original: [Equation]
Revised: [Equation]

Original: Therefore, S IV ( (cid:15) ) ⊆
Revised: Therefore, S IV ( (cid:15) ) ⊆

Original: ˆ S IV ( (cid:15) + κ ) ˆ R (cid:48)
Revised: ˆ S IV ( (cid:15) + κ ) ˆ R (cid:48)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: = ⇒
Revised: = ⇒

Original: [Equation]
Revised: [Equation]

Original: R (cid:48) (Φ( p
Revised: R (cid:48) (Φ( p

Original: [Equation]
Revised: [Equation]

Original: (cid:48) (Φ(
Revised: (cid:48) (Φ(

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:48) (Φ( p
Revised: (cid:48) (Φ( p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, ˆ S IV ( (cid:15) + κ ) ⊆
Revised: Therefore, ˆ S IV ( (cid:15) + κ ) ⊆

Original: S IV ( (cid:15)
Revised: S IV ( (cid:15)

Original: [Equation]
Revised: [Equation]

Original: We bound the probability of event A not occurring.
Revised: We bound the probability of event A not occurring.

Original: Using the covering number (from Lemma 5) we construct a minimum cover of size b = N η ( P ) with points C 1 = { p j } bj =1 .
Revised: Using the covering number (from Lemma 5) we construct a minimum cover of size b = N η ( P ) with points C 1 = { p j } bj =1 .

Original: Compute the probability of failure at one point p j in the cover
Revised: Compute the probability of failure at one point p j in the cover

Original: [Equation]
Revised: [Equation]

Original: We use union bound to bound the probability of the failure over the entire cover C 1 as
Revised: We use union bound to bound the probability of the failure over the entire cover C 1 as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now consider any p ∈ P and suppose p j is nearest point to it in the cover.
Revised: Now consider any p ∈ P and suppose p j is nearest point to it in the cover.

Original: [Equation]
Revised: [Equation]

Original: In the above simplﬁciation, we exploit the Lipschitz continuity of R (cid:48) .
Revised: In the above simplﬁciation, we exploit the Lipschitz continuity of R (cid:48) .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Set η = κ 4 C (cid:48) in equation 119 and from equation 117 with probability at least 1 − N η ( P )2 e − κ 2 | D | 8 L (cid:48)max p ∈P | R (cid:48) (Φ( p, · )) − ˆ R (cid:48) (Φ( p, · ))
Revised: Set η = κ 4 C (cid:48) in equation 119 and from equation 117 with probability at least 1 − N η ( P )2 e − κ 2 | D | 8 L (cid:48)max p ∈P | R (cid:48) (Φ( p, · )) − ˆ R (cid:48) (Φ( p, · ))

Original: | ≤ κ (since max p j ∈C 1 | R (cid:48) (Φ( p j , · )) − ˆ R (cid:48)
Revised: | ≤ κ (since max p j ∈C 1 | R (cid:48) (Φ( p j , · )) − ˆ R (cid:48)

Original: (Φ( p j , · )) | ≤ κ/ 2 ) (120)
Revised: (Φ( p j , · )) | ≤ κ/ 2 ) (120)

Original: We bound N η ( P )2 e − κ 2 | D | 8 L (cid:48) 4 ≤ δ and solve for bound on | D | to get
Revised: We bound N η ( P )2 e − κ 2 | D | 8 L (cid:48) 4 ≤ δ and solve for bound on | D | to get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, if condition in equation 121 holds, then event A occurs.
Revised: Therefore, if condition in equation 121 holds, then event A occurs.

Original: If event A occurs, then S IV ( (cid:15) ) ⊆
Revised: If event A occurs, then S IV ( (cid:15) ) ⊆

Original: ˆ S IV ( (cid:15) + κ ) ⊆
Revised: ˆ S IV ( (cid:15) + κ ) ⊆

Original: S IV ( (cid:15) + 2 κ ) .
Revised: S IV ( (cid:15) + 2 κ ) .

Original: Φ( p ∗ , · ) is a solution to IRM equation 

Revised: Φ( p ∗ , · ) is a solution to IRM equation

Original: and it satisﬁes ∀ Φ( p, · ) ∈ S IV ( (cid:15) )
Revised: and it satisﬁes ∀ Φ( p, · ) ∈ S IV ( (cid:15) )

Original: R (Φ( p ∗ )) ≤ R (Φ( p, · ) .
Revised: R (Φ( p ∗ )) ≤ R (Φ( p, · ) .

Original: Deﬁne an event B : { D : ∀ p ∈ P , | R (Φ( p, · )) − ˆ R (Φ( p, · )) | ≤ ν 2 } .
Revised: Deﬁne an event B : { D : ∀ p ∈ P , | R (Φ( p, · )) − ˆ R (Φ( p, · )) | ≤ ν 2 } .

Original: If event B occurs, then for a solution Φ(ˆ p, · ) of equation 6 (where (cid:15) is replaced with (cid:15) + κ ) satisﬁes
Revised: If event B occurs, then for a solution Φ(ˆ p, · ) of equation 6 (where (cid:15) is replaced with (cid:15) + κ ) satisﬁes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the covering number (from Lemma 5) we construct a minimum cover of size b = N η ( P ) with points C 1 = { p j } bj =1 .
Revised: Using the covering number (from Lemma 5) we construct a minimum cover of size b = N η ( P ) with points C 1 = { p j } bj =1 .

Original: Let us bound the probability of failure at one point p j in the cover
Revised: Let us bound the probability of failure at one point p j in the cover

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We use union bound to bound the probability deﬁned as
Revised: We use union bound to bound the probability deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Now consider any p ∈ P and suppose p j is nearest point to it in the cover.
Revised: Now consider any p ∈ P and suppose p j is nearest point to it in the cover.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, for each p ∈ P
Revised: Therefore, for each p ∈ P

Original: [Equation]
Revised: [Equation]

Original: Set η = ν 8 C in equation 124 and from equation 126 with probability at least 1 − N η ( P )2 e − ν 2 | D | 32 L (cid:48)
Revised: Set η = ν 8 C in equation 124 and from equation 126 with probability at least 1 − N η ( P )2 e − ν 2 | D | 32 L (cid:48)

Original: [Equation]
Revised: [Equation]

Original: since
Revised: since

Original: [Equation]
Revised: [Equation]

Original: We bound N η ( P )2 e − ν 2 | D | 32 L 2 ≤ δ and solve for bound on | D | to get
Revised: We bound N η ( P )2 e − ν 2 | D | 32 L 2 ≤ δ and solve for bound on | D | to get

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: if condition in equation 128 holds
Revised: if condition in equation 128 holds

Original: [Equation]
Revised: [Equation]

Original: then with probability at least 1 − δ 2 event A occurs.
Revised: then with probability at least 1 − δ 2 event A occurs.

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: then the event A ∩ B occurs with at least 1 − δ probability.
Revised: then the event A ∩ B occurs with at least 1 − δ probability.

Original: Project Φ(ˆ p, · ) ∈ ˆ S IV ( (cid:15) + κ ) on S IV ( (cid:15) ) , i.e., ﬁnd the closest function in terms of the metric dis , to obtain Φ(˜ˆ p, · ) .
Revised: Project Φ(ˆ p, · ) ∈ ˆ S IV ( (cid:15) + κ ) on S IV ( (cid:15) ) , i.e., ﬁnd the closest function in terms of the metric dis , to obtain Φ(˜ˆ p, · ) .

Original: If event A occurs, then ˆ p ∈ S IV ( (cid:15) + 2 κ ) .
Revised: If event A occurs, then ˆ p ∈ S IV ( (cid:15) + 2 κ ) .

Original: The distance (cid:107) ˆ p −
Revised: The distance (cid:107) ˆ p −

Original: ˜ˆ p (cid:107) ≤ dis (2 κ ) .
Revised: ˜ˆ p (cid:107) ≤ dis (2 κ ) .

Original: [Equation]
Revised: [Equation]

Original: We choose κ 0 such that κ < κ 0 (use Assumption 15) such that C dis (2 κ ) ≤ ν .
Revised: We choose κ 0 such that κ < κ 0 (use Assumption 15) such that C dis (2 κ ) ≤ ν .

Original: [Equation]
Revised: [Equation]

Original: Therefore, by combining equation 122 amd equation 131, we can conclude that if event A ∩ B occurs, then Φ(ˆ p, · ) ∈ S IV ( (cid:15) + 2 κ ) and | R (Φ( p ∗ , · )) − R (Φ(ˆ p, · )) | ≤ ν .
Revised: Therefore, by combining equation 122 amd equation 131, we can conclude that if event A ∩ B occurs, then Φ(ˆ p, · ) ∈ S IV ( (cid:15) + 2 κ ) and | R (Φ( p ∗ , · )) − R (Φ(ˆ p, · )) | ≤ ν .

Original: From the conditions on | D | , we know that A ∩ B occurs with probability 1 − δ .
Revised: From the conditions on | D | , we know that A ∩ B occurs with probability 1 − δ .

Original: We substitute Φ( p ∗ , · ) as Φ ∗ and Φ(ˆ p, · ) as ˆΦ and this completes the proof.
Revised: We substitute Φ( p ∗ , · ) as Φ ∗ and Φ(ˆ p, · ) as ˆΦ and this completes the proof.

Original: D. OOD Performance: Covariate shift case
Revised: D. OOD Performance: Covariate shift case

Original: In this section, we discuss the extension of Proposition 4 to the inﬁnite hypothesis class case.
Revised: In this section, we discuss the extension of Proposition 4 to the inﬁnite hypothesis class case.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Deﬁne D
Revised: Deﬁne D

Original: [Equation]
Revised: [Equation]

Original: Proposition 16.
Revised: Proposition 16.

Original: If Assumptions 4, 13 hold, m ∈ H Φ and if the number of samples | D | is greater than D ∗ 1 , then with a probability at least 1 − δ , every solution ˆΦ to EIRM satisﬁes R ( m ) ≤ R (ˆΦ) ≤
Revised: If Assumptions 4, 13 hold, m ∈ H Φ and if the number of samples | D | is greater than D ∗ 1 , then with a probability at least 1 − δ , every solution ˆΦ to EIRM satisﬁes R ( m ) ≤ R (ˆΦ) ≤

Original: R ( m ) + ν .
Revised: R ( m ) + ν .

Original: If Assumptions 4, 13 hold, m ∈ H Φ and if the number of samples | D | is greater than D ∗ 2 , then with a probability at least 1 − δ every solution Φ † of ERM satisﬁes R ( m ) ≤ R (Φ † ) ≤ R ( m ) + ν .
Revised: If Assumptions 4, 13 hold, m ∈ H Φ and if the number of samples | D | is greater than D ∗ 2 , then with a probability at least 1 − δ every solution Φ † of ERM satisﬁes R ( m ) ≤ R (Φ † ) ≤ R ( m ) + ν .

Original: Proof.
Revised: Proof.

Original: We begin with the ﬁrst part.
Revised: We begin with the ﬁrst part.

Original: Following the proof of Proposition 5, our goal is to compute the probability of event A : {∀ p ∈ P , | ˆ R (Φ( p, · )) − R (Φ( p, · )) | ≤ ν 2 } .
Revised: Following the proof of Proposition 5, our goal is to compute the probability of event A : {∀ p ∈ P , | ˆ R (Φ( p, · )) − R (Φ( p, · )) | ≤ ν 2 } .

Original: Using the covering number (from Lemma 5) we construct a minimum cover of size b = N η ( P ) with points C = { p j } bj =1 .
Revised: Using the covering number (from Lemma 5) we construct a minimum cover of size b = N η ( P ) with points C = { p j } bj =1 .

Original: Compute the probability of failure at one point p j in the cover
Revised: Compute the probability of failure at one point p j in the cover

Original: [Equation]
Revised: [Equation]

Original: We use union bound to bound the probability of failure over the cover C
Revised: We use union bound to bound the probability of failure over the cover C

Original: [Equation]
Revised: [Equation]

Original: Now consider any p ∈ P and suppose p j is nearest point to it in the cover.
Revised: Now consider any p ∈ P and suppose p j is nearest point to it in the cover.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the above simpliﬁcation
Revised: In the above simpliﬁcation

Original: [Equation]
Revised: [Equation]

Original: we used the Lipschitz continuity of R .
Revised: we used the Lipschitz continuity of R .

Original: Therefore
Revised: Therefore

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Set η = ν 8 C in equation 135 and from equation 133 with probability at least 1 − N η
Revised: Set η = ν 8 C in equation 135 and from equation 133 with probability at least 1 − N η

Original: ( P
Revised: ( P

Original: [Equation]
Revised: [Equation]

Original: 2 e
Revised: 2 e

Original: [Equation]
Revised: [Equation]

Original: We bound N η ( P )2 e − ν 2 | D | 32 L 2 ≤ δ and solve for bound on | D | to get
Revised: We bound N η ( P )2 e − ν 2 | D | 32 L 2 ≤ δ and solve for bound on | D | to get

Original: [Equation]
Revised: [Equation]

Original: Therefore, if condition in equation 137 holds, then event A occurs.
Revised: Therefore, if condition in equation 137 holds, then event A occurs.

Original: Observe that the optimal solution p ∗ for expected risk minimization p ∗ ∈ arg min p ∈P
Revised: Observe that the optimal solution p ∗ for expected risk minimization p ∗ ∈ arg min p ∈P

Original: R
Revised: R

Original: (Φ( p, · )) satisﬁes Φ( p ∗ , · )
Revised: (Φ( p, · )) satisﬁes Φ( p ∗ , · )

Original: = m ( · ) (From
Revised: = m ( · ) (From

Original: Lemma 4 and m ∈ H Φ ).
Revised: Lemma 4 and m ∈ H Φ ).

Original: If event A occurs, then from same argument used in equation 48, a solution p + ∈ P of ERM
Revised: If event A occurs, then from same argument used in equation 48, a solution p + ∈ P of ERM

Original: satisﬁes R (Φ( p ∗ , · )) ≤ R (Φ( p + , · ))
Revised: satisﬁes R (Φ( p ∗ , · )) ≤ R (Φ( p + , · ))

Original: ≤ R (Φ( p ∗ , · ))
Revised: ≤ R (Φ( p ∗ , · ))

Original: + ν is true.
Revised: + ν is true.

Original: We now move to the second part.
Revised: We now move to the second part.

Original: From the ﬁrst part of the proof, we conclude that when
Revised: From the ﬁrst part of the proof, we conclude that when

Original: [Equation]
Revised: [Equation]

Original: with a probability at least 1 − δ 2 event
Revised: with a probability at least 1 − δ 2 event

Original: A occurs.
Revised: A occurs.

Original: Deﬁne an event B : ˜ D is such that | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 .
Revised: Deﬁne an event B : ˜ D is such that | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 .

Original: Since R (cid:48) ( m ) = 0 , | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 = ⇒ | ˆ R (cid:48) ( m ) | ≤ (cid:15) 2 = ⇒ ˆ R (cid:48) ( m ) ≤ (cid:15) 2 .
Revised: Since R (cid:48) ( m ) = 0 , | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 = ⇒ | ˆ R (cid:48) ( m ) | ≤ (cid:15) 2 = ⇒ ˆ R (cid:48) ( m ) ≤ (cid:15) 2 .

Original: Therefore, m ∈ ˆ S IV ( (cid:15) ) .
Revised: Therefore, m ∈ ˆ S IV ( (cid:15) ) .

Original: We write P ( B ) =
Revised: We write P ( B ) =

Original: P ( { ˜ D : | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 } ) =
Revised: P ( { ˜ D : | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | ≤ (cid:15) 2 } ) =

Original: 1 − P ( { ˜ D : | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | > (cid:15) 2 } ) .
Revised: 1 − P ( { ˜ D : | ˆ R (cid:48) ( m ) − R (cid:48) ( m ) | > (cid:15) 2 } ) .

Original: The gradient of loss function is bounded | ∂(cid:96)
Revised: The gradient of loss function is bounded | ∂(cid:96)

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | ≤ L (cid:48) .
Revised: | ≤ L (cid:48) .

Original: From Hoeffding’s inequality in Lemma 1 it follows that
Revised: From Hoeffding’s inequality in Lemma 1 it follows that

Original: [Equation]
Revised: [Equation]

Original: Combining the two conditions equation 138 and equation 139,
Revised: Combining the two conditions equation 138 and equation 139,

Original: [Equation]
Revised: [Equation]

Original: log (cid:16)δ (cid:17)(cid:111)
Revised: log (cid:16)δ (cid:17)(cid:111)

Original: This ensures P ( A ∩ B ) ≥ 1 − δ .
Revised: This ensures P ( A ∩ B ) ≥ 1 − δ .

Original: If event A ∩ B occurs, then we follow the same justiﬁcation as in the proof of Proposition 4 to claim that a solution ˆ p ∈ P to EIRM equation 6 satisﬁes R (Φ( p ∗ , · )) ≤
Revised: If event A ∩ B occurs, then we follow the same justiﬁcation as in the proof of Proposition 4 to claim that a solution ˆ p ∈ P to EIRM equation 6 satisﬁes R (Φ( p ∗ , · )) ≤

Original: R (Φ(ˆ p, · )) ≤ R (Φ( p ∗ , · ))
Revised: R (Φ(ˆ p, · )) ≤ R (Φ( p ∗ , · ))

Original: + ν
Revised: + ν

Original: This completes the proof.
Revised: This completes the proof.

Original: 7.4. E XTENSIONS TO BINARY CLASSIFICATION ( CROSS - ENTROPY )
Revised: 7.4. E XTENSIONS TO BINARY CLASSIFICATION ( CROSS - ENTROPY )

Original: In the main body of the manuscript, we focused on regression (square-loss).
Revised: In the main body of the manuscript, we focused on regression (square-loss).

Original: In this section, we discuss the results that can be extended to binary classiﬁcation (cross-entropy) loss.
Revised: In this section, we discuss the results that can be extended to binary classiﬁcation (cross-entropy) loss.

Original: We will not go in the order in which the results were introduced in the manuscript but in an order that makes for easier exposition for the classiﬁcation case.
Revised: We will not go in the order in which the results were introduced in the manuscript but in an order that makes for easier exposition for the classiﬁcation case.

Original: We begin by showing how to extend Proposition 4 to binary classiﬁcation (cross-entropy).
Revised: We begin by showing how to extend Proposition 4 to binary classiﬁcation (cross-entropy).

Original: Recall that the entropy of a distribution P X is H ( P ) = − E P (cid:2) log( d P ) (cid:3) .
Revised: Recall that the entropy of a distribution P X is H ( P ) = − E P (cid:2) log( d P ) (cid:3) .

Original: Recall that the cross entropy of
Revised: Recall that the cross entropy of

Original: Q relative to P is H ( P , Q ) =
Revised: Q relative to P is H ( P , Q ) =

Original: − E P (cid:2) log( d Q ) (cid:3) =
Revised: − E P (cid:2) log( d Q ) (cid:3) =

Original: H ( P ) + KL ( d P (cid:107) d Q ) .
Revised: H ( P ) + KL ( d P (cid:107) d Q ) .

Original: The cross entropy loss (cid:96) for binary classiﬁcation when using a predictor f : X → [0 , 1] ( f ( X e ) is the probability of label 1 conditional on X e ) is given as (cid:96) (cid:0)
Revised: The cross entropy loss (cid:96) for binary classiﬁcation when using a predictor f : X → [0 , 1] ( f ( X e ) is the probability of label 1 conditional on X e ) is given as (cid:96) (cid:0)

Original: f ( X e ) , Y e (cid:1) =
Revised: f ( X e ) , Y e (cid:1) =

Original: Y e log (cid:0) f
Revised: Y e log (cid:0) f

Original: ( X e ) (cid:1) + (1 − Y e ) log (cid:0) 1 − f ( X e ) (cid:1) .
Revised: ( X e ) (cid:1) + (1 − Y e ) log (cid:0) 1 − f ( X e ) (cid:1) .

Original: For the discussion below Q ( Y e | X e ) is deﬁned in terms of f as follows Q ( Y e = 1 | X e ) = f
Revised: For the discussion below Q ( Y e | X e ) is deﬁned in terms of f as follows Q ( Y e = 1 | X e ) = f

Original: ( X e ) (cid:0)
Revised: ( X e ) (cid:0)

Original: Q ( Y
Revised: Q ( Y

Original: [Equation]
Revised: [Equation]

Original: = 

Revised: =

Original: | X
Revised: | X

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: f ( X
Revised: f ( X

Original: [Equation]
Revised: [Equation]

Original: ) (cid:1) .
Revised: ) (cid:1) .

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: From the above it is clear that Q ( Y e | X e (cid:1)
Revised: From the above it is clear that Q ( Y e | X e (cid:1)

Original: =
Revised: =

Original: P ( Y e | X e (cid:1) minimizes the risk in an individual environment.
Revised: P ( Y e | X e (cid:1) minimizes the risk in an individual environment.

Original: Assumption 16.
Revised: Assumption 16.

Original: Invariance w.r.t all the features.
Revised: Invariance w.r.t all the features.

Original: For all e, o ∈ E all and for all x ∈ X , E [ Y e | X e = x ] =
Revised: For all e, o ∈ E all and for all x ∈ X , E [ Y e | X e = x ] =

Original: E
Revised: E

Original: [ Y o | X o = x ] .
Revised: [ Y o | X o = x ] .

Original: X e ∼ P eX e and ∀ e ∈ E all support of P eX e is equal to X .
Revised: X e ∼ P eX e and ∀ e ∈ E all support of P eX e is equal to X .

Original: Observe that in the binary-classiﬁcation setting the above assumption amounts to equating the conditional probabilities P ( Y e | X e ) and P ( Y o | X o ) .
Revised: Observe that in the binary-classiﬁcation setting the above assumption amounts to equating the conditional probabilities P ( Y e | X e ) and P ( Y o | X o ) .

Original: Recall that map m (from equation 2) simpliﬁes to ∀ x ∈ X
Revised: Recall that map m (from equation 2) simpliﬁes to ∀ x ∈ X

Original: [Equation]
Revised: [Equation]

Original: If Assumption 16 holds, then from cross-entropy decomposition in equation 140 it is clear that m solves the OOD problem (as it is optimal w.r.t each environment).
Revised: If Assumption 16 holds, then from cross-entropy decomposition in equation 140 it is clear that m solves the OOD problem (as it is optimal w.r.t each environment).

Original: It is also the unique minimizer.
Revised: It is also the unique minimizer.

Original: We can justify it based on the same argument presented in equation 56.
Revised: We can justify it based on the same argument presented in equation 56.

Original: Suppose there was another optimizer which was different from m over a set with a non-zero measure.
Revised: Suppose there was another optimizer which was different from m over a set with a non-zero measure.

Original: Over such a set the the
Revised: Over such a set the the

Original: KL divergence term inside equation 140 will be greater than zero, thus making the second term in equation 140 positive thus contradicting the optimality.
Revised: KL divergence term inside equation 140 will be greater than zero, thus making the second term in equation 140 positive thus contradicting the optimality.

Original: This shows m is the unique optimizer.
Revised: This shows m is the unique optimizer.

Original: The rest of the arguments presented in the proof of Proposition 4 carry over to this case.
Revised: The rest of the arguments presented in the proof of Proposition 4 carry over to this case.

Original: Therefore,
Revised: Therefore,

Original: Proposition 4 extends to the cross-entropy loss.
Revised: Proposition 4 extends to the cross-entropy loss.

Original: Note that Proposition 2’s proof was agnostic to loss type and only used boundedness, which holds for both cross-entropy as long as the probability output are in the strict interior of [0 , 1] deﬁned by [ p min , p max ] ⊂ [0 , 1] .
Revised: Note that Proposition 2’s proof was agnostic to loss type and only used boundedness, which holds for both cross-entropy as long as the probability output are in the strict interior of [0 , 1] deﬁned by [ p min , p max ] ⊂ [0 , 1] .

Original: We could not generalize Proposition 5 to cross-entropy loss and that is left as future work.
Revised: We could not generalize Proposition 5 to cross-entropy loss and that is left as future work.

Original: Next, we move to showing how Proposition 1 can be generalized to binary classiﬁcation.
Revised: Next, we move to showing how Proposition 1 can be generalized to binary classiﬁcation.

Original: Assumption 17.
Revised: Assumption 17.

Original: Existence of an invariant representation.
Revised: Existence of an invariant representation.

Original: ∃
Revised: ∃

Original: Φ ∗ : X → Z such that ∀ e, o ∈ E all and ∀ x ∈ X , E [ Y e | Φ ∗ ( x )] =
Revised: Φ ∗ : X → Z such that ∀ e, o ∈ E all and ∀ x ∈ X , E [ Y e | Φ ∗ ( x )] =

Original: E [ Y o | Φ ∗ ( x )] .
Revised: E [ Y o | Φ ∗ ( x )] .

Original: Recall m deﬁned in equation 2, ∀ z ∈ Φ ∗ ( X )
Revised: Recall m deﬁned in equation 2, ∀ z ∈ Φ ∗ ( X )

Original: [Equation]
Revised: [Equation]

Original: Deﬁne a composite predictor w ◦ Φ ∗ .
Revised: Deﬁne a composite predictor w ◦ Φ ∗ .

Original: Substituting f = w ◦ Φ ∗ in equation 140 we get the following.
Revised: Substituting f = w ◦ Φ ∗ in equation 140 we get the following.

Original: For the discussion below, a distribution R ( Y e | X e ) is deﬁned in terms of w ◦ Φ ∗ as follows R ( Y e = 1 | X e ) =
Revised: For the discussion below, a distribution R ( Y e | X e ) is deﬁned in terms of w ◦ Φ ∗ as follows R ( Y e = 1 | X e ) =

Original: w ◦ Φ ∗ ( X e ) ( R ( Y e = 0 | X e ) = 1 − w ◦ Φ ∗ ( X e ) ).
Revised: w ◦ Φ ∗ ( X e ) ( R ( Y e = 0 | X e ) = 1 − w ◦ Φ ∗ ( X e ) ).

Original: R e ( w ◦ Φ
Revised: R e ( w ◦ Φ

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: E e (cid:104)
Revised: E e (cid:104)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If all the data is transformed by Φ ∗ , then from the above decomposition equation 143 it is clear that R ( Y e | Z e ) =
Revised: If all the data is transformed by Φ ∗ , then from the above decomposition equation 143 it is clear that R ( Y e | Z e ) =

Original: P ( Y e | Z e ) is the optimal predictor for each environment.
Revised: P ( Y e | Z e ) is the optimal predictor for each environment.

Original: Hence, w ∗ ( Z e ) =
Revised: Hence, w ∗ ( Z e ) =

Original: P ( Y e = 1 | Z e ) is the best choice for w .
Revised: P ( Y e = 1 | Z e ) is the best choice for w .

Original: Assumption 18.
Revised: Assumption 18.

Original: Existence of an environment where the invariant representation is sufﬁcient.
Revised: Existence of an environment where the invariant representation is sufﬁcient.

Original: ∃ an environment e ∈ E all such that Y e ⊥ X e | Z e , where Z e =
Revised: ∃ an environment e ∈ E all such that Y e ⊥ X e | Z e , where Z e =

Original: Φ ∗ ( X e ) .
Revised: Φ ∗ ( X e ) .

Original: We derive a relationship as follows for the environment q satisfying Assumption 18.
Revised: We derive a relationship as follows for the environment q satisfying Assumption 18.

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: follows from conditional independence in Assumption 18)
Revised: follows from conditional independence in Assumption 18)

Original: [Equation]
Revised: [Equation]

Original: Also, note that since Z q = Φ ∗ ( X q ) we have
Revised: Also, note that since Z q = Φ ∗ ( X q ) we have

Original: [Equation]
Revised: [Equation]

Original: From equation 144 and equation 145 we have
Revised: From equation 144 and equation 145 we have

Original: [Equation]
Revised: [Equation]

Original: We use equation 146 in the cross entropy decomposition from equation
Revised: We use equation 146 in the cross entropy decomposition from equation

Original: [Equation]
Revised: [Equation]

Original: Recall Q ( Y q = 1 | X q ) =
Revised: Recall Q ( Y q = 1 | X q ) =

Original: f ( X q ) ( Q ( Y q = 

Revised: f ( X q ) ( Q ( Y q =

Original: | X q )
Revised: | X q )

Original: = 1 − f
Revised: = 1 − f

Original: ( X q ) ).
Revised: ( X q ) ).

Original: Also, recall w ∗ ( Z q ) =
Revised: Also, recall w ∗ ( Z q ) =

Original: P ( Y q = 1 | Z q ) .
Revised: P ( Y q = 1 | Z q ) .

Original: From the above it is clear that f = w ∗ ◦ Φ ∗ is the optimal predictor for environment q .
Revised: From the above it is clear that f = w ∗ ◦ Φ ∗ is the optimal predictor for environment q .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The expected conditional entropy for environment e is deﬁned as ¯ H e =
Revised: The expected conditional entropy for environment e is deﬁned as ¯ H e =

Original: E e (cid:104) H (cid:0) P ( Y e | Z e ) (cid:1)(cid:105) is the risk achieved by w ∗ ◦ Φ ∗ .
Revised: E e (cid:104) H (cid:0) P ( Y e | Z e ) (cid:1)(cid:105) is the risk achieved by w ∗ ◦ Φ ∗ .

Original: Also, ¯ H e measures the amount of noise in the environment.
Revised: Also, ¯ H e measures the amount of noise in the environment.

Original: This is much like the variance that remains in the least squares minimization.
Revised: This is much like the variance that remains in the least squares minimization.

Original: In the next assumption, we state that the noise in all the environments is bounded above.
Revised: In the next assumption, we state that the noise in all the environments is bounded above.

Original: We also assume that one of the environments which achieves the maximum noise level is environment q , which satisﬁes Assumption 18.
Revised: We also assume that one of the environments which achieves the maximum noise level is environment q , which satisﬁes Assumption 18.

Original: Assumption 19.
Revised: Assumption 19.

Original: ∀
Revised: ∀

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: From equation 143 for all the environments
Revised: From equation 143 for all the environments

Original: [Equation]
Revised: [Equation]

Original: Observe that max e ∈E all R e ( w ∗ ◦ Φ ∗ )
Revised: Observe that max e ∈E all R e ( w ∗ ◦ Φ ∗ )

Original: =
Revised: =

Original: ¯ H sup .
Revised: ¯ H sup .

Original: From the above assumption it is clear that for all predictors f : X → [0 , 1] .
Revised: From the above assumption it is clear that for all predictors f : X → [0 , 1] .

Original: [Equation]
Revised: [Equation]

Original: Since max e ∈E all R e ( w ∗ ◦ Φ ∗ ) =
Revised: Since max e ∈E all R e ( w ∗ ◦ Φ ∗ ) =

Original: ¯ H sup , we conclude that w ∗ ◦ Φ ∗ is the predictor that solves the
Revised: ¯ H sup , we conclude that w ∗ ◦ Φ ∗ is the predictor that solves the

Original: OOD problem in equation 1.
Revised: OOD problem in equation 1.

Original: This completes the extension of Proposition 1 to cross-entropy.
Revised: This completes the extension of Proposition 1 to cross-entropy.

Original: O N THE B IASEDNESS OF ERM
Revised: O N THE B IASEDNESS OF ERM

Original: Consider the model in Assumption 5.
Revised: Consider the model in Assumption 5.

Original: For each environment e ∈ E tr , deﬁne a vector ρ e = E e (cid:2)
Revised: For each environment e ∈ E tr , deﬁne a vector ρ e = E e (cid:2)

Original: ε e X e (cid:3) .
Revised: ε e X e (cid:3) .

Original: Deﬁne a matrix ¯ ρ with ρ e as column vectors ¯ ρ = [ ρ 1 , . .
Revised: Deﬁne a matrix ¯ ρ with ρ e as column vectors ¯ ρ = [ ρ 1 , . .

Original: , ρ |E tr | ] .
Revised: , ρ |E tr | ] .

Original: Deﬁne a vector ¯ π =
Revised: Deﬁne a vector ¯ π =

Original: [ π 1 , . . .
Revised: [ π 1 , . . .

Original: , π |E tr | ] , where recall from Assumption 5 π o is probability a point comes from environment o .
Revised: , π |E tr | ] , where recall from Assumption 5 π o is probability a point comes from environment o .

Original: Proposition 17.
Revised: Proposition 17.

Original: If Assumption 5 holds, H Φ is a linear hypothesis class with parameter Φ and if the rank of ¯
Revised: If Assumption 5 holds, H Φ is a linear hypothesis class with parameter Φ and if the rank of ¯

Original: ρ is at least one, then ERM is asymptotically biased, i.e., even with inﬁnite data ERM will not achieve the desired solution ˜ S T γ , except over a set of measure zero of probability distributions ¯ π .
Revised: ρ is at least one, then ERM is asymptotically biased, i.e., even with inﬁnite data ERM will not achieve the desired solution ˜ S T γ , except over a set of measure zero of probability distributions ¯ π .

Original: Proof.
Revised: Proof.

Original: Consider the case when ERM has access to inﬁnite data, i.e., we are solving the expected risk minimization problem stated as min Φ ∈H Φ R (Φ) .
Revised: Consider the case when ERM has access to inﬁnite data, i.e., we are solving the expected risk minimization problem stated as min Φ ∈H Φ R (Φ) .

Original: We will consider the linear model in Assumption 5 and assume H Φ linear hypothesis class parametrized by Φ ∈ R n .
Revised: We will consider the linear model in Assumption 5 and assume H Φ linear hypothesis class parametrized by Φ ∈ R n .

Original: We simplify the ∇ Φ R (Φ) for the square loss below
Revised: We simplify the ∇ Φ R (Φ) for the square loss below

Original: [Equation]
Revised: [Equation]

Original: We compute the gradient for Φ = ˜ S T γ as
Revised: We compute the gradient for Φ = ˜ S T γ as

Original: [Equation]
Revised: [Equation]

Original: Recall ρ e = E e (cid:2)
Revised: Recall ρ e = E e (cid:2)

Original: ε e X e (cid:3) and ¯ ρ =
Revised: ε e X e (cid:3) and ¯ ρ =

Original: [ ρ 1 , . . .
Revised: [ ρ 1 , . . .

Original: , ρ |E tr | ] .
Revised: , ρ |E tr | ] .

Original: Recall ¯ π =
Revised: Recall ¯ π =

Original: [ π 1 , . . .
Revised: [ π 1 , . . .

Original: , π |E tr | ] .
Revised: , π |E tr | ] .

Original: Setting the gradient deﬁned in equation 153 to zero and using the above matrix notation we get
Revised: Setting the gradient deﬁned in equation 153 to zero and using the above matrix notation we get

Original: [Equation]
Revised: [Equation]

Original: If ¯ π satisﬁes equation 154, then ERM is unbiased, else it is not.
Revised: If ¯ π satisﬁes equation 154, then ERM is unbiased, else it is not.

Original: Consider the set of vectors in the probability simplex { ¯ π | 1 T ¯ π = 1 , ¯ π ≥ 0 } and deﬁne a uniform probability distribution over it.
Revised: Consider the set of vectors in the probability simplex { ¯ π | 1 T ¯ π = 1 , ¯ π ≥ 0 } and deﬁne a uniform probability distribution over it.

Original: Since rank of ¯ ρ > 0 at least one of the columns of ¯ ρ is non-zero.
Revised: Since rank of ¯ ρ > 0 at least one of the columns of ¯ ρ is non-zero.

Original: As a result a uniform random draw from this set of probablity distributions would have zero probability of satisfying ¯ ρ ¯ π = 0 .
Revised: As a result a uniform random draw from this set of probablity distributions would have zero probability of satisfying ¯ ρ ¯ π = 0 .

Original: Therefore, ˜ S T γ is not the optimal solution to ERM and thus the solution of ERM would be biased away from ˜ S T γ .
Revised: Therefore, ˜ S T γ is not the optimal solution to ERM and thus the solution of ERM would be biased away from ˜ S T γ .

Original: In Proposition 5, we had assumed Assumptions 5, 6, 7 hold.
Revised: In Proposition 5, we had assumed Assumptions 5, 6, 7 hold.

Original: If we also assume that rank of ¯ ρ is at least one, the Proposition 5 continues to hold.
Revised: If we also assume that rank of ¯ ρ is at least one, the Proposition 5 continues to hold.

Original: If for at least one e ∈ E tr , E e (cid:2) ε e X e (cid:3) is non-zero, then the rank of ¯ ρ is at least one.
Revised: If for at least one e ∈ E tr , E e (cid:2) ε e X e (cid:3) is non-zero, then the rank of ¯ ρ is at least one.

Original: From proof of Theorem 10 in Arjovsky et al.
Revised: From proof of Theorem 10 in Arjovsky et al.

Original: (2019), linear general position continues to hold except over a set of covariance matrices with measure zero even when one of the E e (cid:2)
Revised: (2019), linear general position continues to hold except over a set of covariance matrices with measure zero even when one of the E e (cid:2)

Original: ε e X e (cid:3) is non-zero.
Revised: ε e X e (cid:3) is non-zero.

Original: Also, in the above Proposition 17, we only required that rank of ¯ ρ is at least one.
Revised: Also, in the above Proposition 17, we only required that rank of ¯ ρ is at least one.

Original: However, if we make the additional assumptions 5, 6, 7, the result of the above Proposition continues to hold.
Revised: However, if we make the additional assumptions 5, 6, 7, the result of the above Proposition continues to hold.

Original: Therefore, if Assumption 5, 6, 7 hold and rank of ¯ ρ is at least one, then ERM is asymptotically biased and IRM can be within √ (cid:15) neighborhood of the ideal solution with the sample complexity shown in Proposition 5.
Revised: Therefore, if Assumption 5, 6, 7 hold and rank of ¯ ρ is at least one, then ERM is asymptotically biased and IRM can be within √ (cid:15) neighborhood of the ideal solution with the sample complexity shown in Proposition 5.