Original: Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization
Revised: Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization

Original: Abstract
Revised: Abstract

Original: Real-world data collected from multiple domains can have multiple, distinct distribution shifts over multiple attributes.
Revised: Real-world data collected from multiple domains can have multiple, distinct distribution shifts over multiple attributes.

Original: However, state-of-the art advances in domain generalization (DG) algorithms focus only on specific shifts over a single attribute.
Revised: However, state-of-the art advances in domain generalization (DG) algorithms focus only on specific shifts over a single attribute.

Original: We introduce datasets with multi -attribute distribution shifts and find that existing DG algorithms fail to generalize.
Revised: We introduce datasets with multi -attribute distribution shifts and find that existing DG algorithms fail to generalize.

Original: Using causal graphs to characterize the different types of shifts, we show that each multi-attribute causal graph entails different constraints over observed variables, and therefore any algorithm based on a single, fixed independence constraint cannot work well across all shifts.
Revised: Using causal graphs to characterize the different types of shifts, we show that each multi-attribute causal graph entails different constraints over observed variables, and therefore any algorithm based on a single, fixed independence constraint cannot work well across all shifts.

Original: We present Causally Adaptive Constraint
Revised: We present Causally Adaptive Constraint

Original: Minimization (CACM) , an algorithm for identifying the correct independence constraints for regularization.
Revised: Minimization (CACM) , an algorithm for identifying the correct independence constraints for regularization.

Original: Experiments confirm our theoretical claim: correct independence constraints lead to the highest accuracy on unseen domains.
Revised: Experiments confirm our theoretical claim: correct independence constraints lead to the highest accuracy on unseen domains.

Original: Our results demonstrate the importance of modeling the causal relationships inherent in a data-generating process, without which it can be impossible to know the correct regularization constraints for a dataset.
Revised: Our results demonstrate the importance of modeling the causal relationships inherent in a data-generating process, without which it can be impossible to know the correct regularization constraints for a dataset.

Original: 1. Introduction
Revised: 1. Introduction

Original: To perform reliably in real world settings, machine learning models must be robust to distribution shifts – where the training distribution differs from the test distribution.
Revised: To perform reliably in real world settings, machine learning models must be robust to distribution shifts – where the training distribution differs from the test distribution.

Original: The domain generalization (DG) task (Wang et al., 2021; Zhou et al., 2021) encapsulates this challenge by evaluating accuracy on an unseen domain given data from multiple domains that share a common optimal predictor.
Revised: The domain generalization (DG) task (Wang et al., 2021; Zhou et al., 2021) encapsulates this challenge by evaluating accuracy on an unseen domain given data from multiple domains that share a common optimal predictor.

Original: Recent state-of-theart advances in representation learning for DG (Li et al., 2018a;
Revised: Recent state-of-theart advances in representation learning for DG (Li et al., 2018a;

Original: Arjovsky et al., 2019; Krueger et al., 2021; Mahajan
Revised: Arjovsky et al., 2019; Krueger et al., 2021; Mahajan

Original: [Table]
Revised: [Table]

Original: Test
Revised: Test

Original: [Table]
Revised: [Table]

Original: Y=
Revised: Y=

Original: Y= 15° 60° 90°
Revised: Y= 15° 60° 90°

Original: Y=
Revised: Y=

Original: Y=
Revised: Y=

Original: Multi-attribute shift (0.9,15°) (0.8,60°) (0.1,90°)
Revised: Multi-attribute shift (0.9,15°) (0.8,60°) (0.1,90°)

Original: Y=
Revised: Y=

Original: Y=
Revised: Y=

Original: C o l o r
Revised: C o l o r

Original: R o t
Revised: R o t

Original: C o l + R o t (a) (b)
Revised: C o l + R o t (a) (b)

Original: ) to distinct shifts
Revised: ) to distinct shifts

Original: Color and Rotation
Revised: Color and Rotation

Original: The causal graph representing the data generating process for (a) – Color has a correlation with Y which changes across environments while Rotation varies independently.
Revised: The causal graph representing the data generating process for (a) – Color has a correlation with Y which changes across environments while Rotation varies independently.

Original: et al., 2021; Sun & Saenko, 2016) focus on a limited setting where the domains exhibit a single kind of distribution shift over one attribute (where an attribute refers to a spurious high-level variable).
Revised: et al., 2021; Sun & Saenko, 2016) focus on a limited setting where the domains exhibit a single kind of distribution shift over one attribute (where an attribute refers to a spurious high-level variable).

Original: Using MNIST as an example, domains are created either by adding new values of a spurious attribute like rotation (e.g., Rotated-MNIST dataset (Ghifary et al., 2015;
Revised: Using MNIST as an example, domains are created either by adding new values of a spurious attribute like rotation (e.g., Rotated-MNIST dataset (Ghifary et al., 2015;

Original: Piratla et al., 2020)) or by changing the correlation between the class label and a spurious attribute like color (e.g., Colored-MNIST (Arjovsky et al., 2019)), but not both simultaneously.
Revised: Piratla et al., 2020)) or by changing the correlation between the class label and a spurious attribute like color (e.g., Colored-MNIST (Arjovsky et al., 2019)), but not both simultaneously.

Original: Recent work (Wiles et al., 2022; Ye et al., 2022) shows that the accuracy of state-of-the-art DG algorithms are not consistent over these different datasets, indicating the importance of the kind of shift in a dataset.
Revised: Recent work (Wiles et al., 2022; Ye et al., 2022) shows that the accuracy of state-of-the-art DG algorithms are not consistent over these different datasets, indicating the importance of the kind of shift in a dataset.

Original: In real-world data, however, different sources of distribution shift can co-exist .
Revised: In real-world data, however, different sources of distribution shift can co-exist .

Original: Differences across domains may involve multiple attributes with different kinds of shifts.
Revised: Differences across domains may involve multiple attributes with different kinds of shifts.

Original: For example, in our Col+Rot-MNIST dataset (see Figure 1), the color and rotation angle of digits can shift independently across data distributions.
Revised: For example, in our Col+Rot-MNIST dataset (see Figure 1), the color and rotation angle of digits can shift independently across data distributions.

Original: In satellite imagery (Koh et al., 2021), the appearance of land cover such as vegetation (trees and grasses) changes seasonally and independently of regional variations in vegetation.
Revised: In satellite imagery (Koh et al., 2021), the appearance of land cover such as vegetation (trees and grasses) changes seasonally and independently of regional variations in vegetation.

Original: To capture such data, we provide a
Revised: To capture such data, we provide a

Original: [Figure]
Revised: [Figure]

Original: characterization of multi-attribute distribution shifts based on the relationship between each attribute and the class label.
Revised: characterization of multi-attribute distribution shifts based on the relationship between each attribute and the class label.

Original: Using causal graphs and the principle of d-separation, we show that each type of shift leads to a different set of independence constraints on the observed variables.
Revised: Using causal graphs and the principle of d-separation, we show that each type of shift leads to a different set of independence constraints on the observed variables.

Original: As a consequence, for datasets like Col+Rot-MNIST and a multiattribute dataset derived from small NORB (LeCun et al., 2004), we find that existing DG algorithms that are often targeted for a specific shift fail to generalize.
Revised: As a consequence, for datasets like Col+Rot-MNIST and a multiattribute dataset derived from small NORB (LeCun et al., 2004), we find that existing DG algorithms that are often targeted for a specific shift fail to generalize.

Original: Specifically, applying an incorrect constraint leads to substantially lower accuracy than the correct constraint, highlighting the importance of identifying a suitable constraint for each dataset.
Revised: Specifically, applying an incorrect constraint leads to substantially lower accuracy than the correct constraint, highlighting the importance of identifying a suitable constraint for each dataset.

Original: Beyond existing DG algorithms, our theoretical analysis shows that any representation learning algorithm based on a single, fixed independence constraint will fail to generalize under multi-attribute shifts.
Revised: Beyond existing DG algorithms, our theoretical analysis shows that any representation learning algorithm based on a single, fixed independence constraint will fail to generalize under multi-attribute shifts.

Original: Therefore, we propose to leverage the information provided by multiple independent shifts across attributes, assuming structural knowledge of the shifts.
Revised: Therefore, we propose to leverage the information provided by multiple independent shifts across attributes, assuming structural knowledge of the shifts.

Original: Then, given a dataset and the canonical causal graph for multi-attribute shifts (Figure 2), our proposed algorithm, Causally Adaptive Constraint Minimization (CACM) , identifies the correct constraints and applies them as a regularizer in the learning algorithm’s loss function.
Revised: Then, given a dataset and the canonical causal graph for multi-attribute shifts (Figure 2), our proposed algorithm, Causally Adaptive Constraint Minimization (CACM) , identifies the correct constraints and applies them as a regularizer in the learning algorithm’s loss function.

Original: 2. Generalization under multi-attribute shifts
Revised: 2. Generalization under multi-attribute shifts

Original: We focus on representation learning-based (Wang et al., 2021) DG algorithms, typically characterized by a regularization term that constrains an ERM loss such as crossentropy (see Table 5 in Suppl.).
Revised: We focus on representation learning-based (Wang et al., 2021) DG algorithms, typically characterized by a regularization term that constrains an ERM loss such as crossentropy (see Table 5 in Suppl.).

Original: Which regularization constraint is the correct one?
Revised: Which regularization constraint is the correct one?

Original: This question has attracted much discussion (Johansson et al., 2019; Mahajan et al., 2021;
Revised: This question has attracted much discussion (Johansson et al., 2019; Mahajan et al., 2021;

Original: Wiles et al., 2022; Ye et al., 2022;
Revised: Wiles et al., 2022; Ye et al., 2022;

Original: Zhao et al., 2019) without resolution.
Revised: Zhao et al., 2019) without resolution.

Original: Various works have discussed failure modes of unconditional independence (Akuzawa et al., 2019;
Revised: Various works have discussed failure modes of unconditional independence (Akuzawa et al., 2019;

Original: Johansson et al., 2019; Zhao et al., 2019), conditional indepen- dence (Mahajan et al., 2021) and Invariant Risk Minimization (Rosenfeld et al., 2021).
Revised: Johansson et al., 2019; Zhao et al., 2019), conditional indepen- dence (Mahajan et al., 2021) and Invariant Risk Minimization (Rosenfeld et al., 2021).

Original: Moreover, recent empirical work (Wiles et al., 2022; Ye et al., 2022) shows that different algorithms perform better under different shifts, but none performs across all shifts.
Revised: Moreover, recent empirical work (Wiles et al., 2022; Ye et al., 2022) shows that different algorithms perform better under different shifts, but none performs across all shifts.

Original: As a result, (Wiles et al., 2022) suggest that instead of a universal algorithm for any shift, adaptable algorithms that use auxiliary attribute information can be more useful.
Revised: As a result, (Wiles et al., 2022) suggest that instead of a universal algorithm for any shift, adaptable algorithms that use auxiliary attribute information can be more useful.

Original: To explore this question beyond domains with a single distribution shift, we consider the generalization problem over a more realistic setup where each domain can have multiple shifts over different attributes.
Revised: To explore this question beyond domains with a single distribution shift, we consider the generalization problem over a more realistic setup where each domain can have multiple shifts over different attributes.

Original: 2.1. Risk-invariant predictor over a set of distributions
Revised: 2.1. Risk-invariant predictor over a set of distributions

Original: We consider the supervised learning setup from Wiles et al.
Revised: We consider the supervised learning setup from Wiles et al.

Original: (2022) where each row of train data ( x i , a i , y i )
Revised: (2022) where each row of train data ( x i , a i , y i )

Original: ni
Revised: ni

Original: =1 contains input features x
Revised: =1 contains input features x

Original: i (e.g., X-ray pixels), a set of nuisance or spurious attributes a i (e.g., hospital) and class label
Revised: i (e.g., X-ray pixels), a set of nuisance or spurious attributes a i (e.g., hospital) and class label

Original: y i (e.g., disease diagnosis).
Revised: y i (e.g., disease diagnosis).

Original: The attributes represent variables that are often recorded during data collection or can be inferred.
Revised: The attributes represent variables that are often recorded during data collection or can be inferred.

Original: Since the nuisance attribute’s distribution or its correlation with the label can change, we obtain different data distributions.
Revised: Since the nuisance attribute’s distribution or its correlation with the label can change, we obtain different data distributions.

Original: Given a set of domains sampled from P , the train data is sampled from domains, P E tr = { P E 1 , P E 2 , · · · } ⊂ P while the test data is assumed to be sampled from a single unseen domain, P E te = { P Ete } ⊂ P .
Revised: Given a set of domains sampled from P , the train data is sampled from domains, P E tr = { P E 1 , P E 2 , · · · } ⊂ P while the test data is assumed to be sampled from a single unseen domain, P E te = { P Ete } ⊂ P .

Original: The goal is to learn a classifier g ( x ) using train domains such that it generalizes and achieves a similar, small risk on test data from P Ete as it achieves on P E tr .
Revised: The goal is to learn a classifier g ( x ) using train domains such that it generalizes and achieves a similar, small risk on test data from P Ete as it achieves on P E tr .

Original: Formally, given a set of distributions P , we define a risk-invariant predictor (Makar et al., 2022) as,
Revised: Formally, given a set of distributions P , we define a risk-invariant predictor (Makar et al., 2022) as,

Original: Definition 2.1.
Revised: Definition 2.1.

Original: Optimal Risk Invariant Predictor for P (from (Makar et al., 2022))
Revised: Optimal Risk Invariant Predictor for P (from (Makar et al., 2022))

Original: Define the risk of predictor g on distribution P ∈ P as R P ( g ) =
Revised: Define the risk of predictor g on distribution P ∈ P as R P ( g ) =

Original: E x ,y ∼ P ℓ ( g ( x ) , y ) where ℓ is cross-entropy or another classification loss.
Revised: E x ,y ∼ P ℓ ( g ( x ) , y ) where ℓ is cross-entropy or another classification loss.

Original: Then, the set of risk-invariant predictors obtain the same risk across all distributions P ∈ P , and set of the optimal risk-invariant predictors is defined as the risk-invariant predictors that obtain minimum risk on all distributions.
Revised: Then, the set of risk-invariant predictors obtain the same risk across all distributions P ∈ P , and set of the optimal risk-invariant predictors is defined as the risk-invariant predictors that obtain minimum risk on all distributions.

Original: 2.2. Using causal graphs for multi-attribute shifts
Revised: 2.2. Using causal graphs for multi-attribute shifts

Original: To specify the set of distributions P to generalize over, using causal graphs, we characterize the different data-generating processes that can lead to a multi-attribute shift dataset.
Revised: To specify the set of distributions P to generalize over, using causal graphs, we characterize the different data-generating processes that can lead to a multi-attribute shift dataset.

Original: Figure 2 shows the corresponding causal directed acylic graph (DAG).
Revised: Figure 2 shows the corresponding causal directed acylic graph (DAG).

Original: Shaded nodes represent observed variables X , Y ; and the sets of attributes A ind , A ind , and E such that
Revised: Shaded nodes represent observed variables X , Y ; and the sets of attributes A ind , A ind , and E such that

Original: A ind ∪ A ind ∪ { E } =
Revised: A ind ∪ A ind ∪ { E } =

Original: A .
Revised: A .

Original: A ind represents the attributes correlated with label, A ind the attributes that are independent of label, while E is a special attribute for the domain.
Revised: A ind represents the attributes correlated with label, A ind the attributes that are independent of label, while E is a special attribute for the domain.

Original: All attributes, along with the stable/causal features X c , determine the observed features X .
Revised: All attributes, along with the stable/causal features X c , determine the observed features X .

Original: And the stable features,
Revised: And the stable features,

Original: X c are the only features that cause Y .
Revised: X c are the only features that cause Y .

Original: In the simplest case, we assume no label shift across environments i.e. marginal distribution of Y is constant across train domains and test,
Revised: In the simplest case, we assume no label shift across environments i.e. marginal distribution of Y is constant across train domains and test,

Original: P Etr ( y ) =
Revised: P Etr ( y ) =

Original: P Ete ( y ) (see Figure 2(a)).
Revised: P Ete ( y ) (see Figure 2(a)).

Original: More generally, different domains may have different distribution of objects and hence there may be a correlation between E and Obj , as represented by the right subfigure (Figure 2(b)).
Revised: More generally, different domains may have different distribution of objects and hence there may be a correlation between E and Obj , as represented by the right subfigure (Figure 2(b)).

Original: We characterize different kinds of shifts based on the relationship between nuisance attributes A and the classification label Y .
Revised: We characterize different kinds of shifts based on the relationship between nuisance attributes A and the classification label Y .

Original: Specifically, A ind has varying distribution across environments but is Independent of the class label.
Revised: Specifically, A ind has varying distribution across environments but is Independent of the class label.

Original: The dashed bidirectional arrow represents the correlation between A ind and Y .
Revised: The dashed bidirectional arrow represents the correlation between A ind and Y .

Original: There are different mechanisms which can introduce the dashed-line relationship (Figure 2(c)) – direct-causal relationship ( Y causing A ind ), confounding between Y and A ind due to a common cause, or selection during the data-generating process.
Revised: There are different mechanisms which can introduce the dashed-line relationship (Figure 2(c)) – direct-causal relationship ( Y causing A ind ), confounding between Y and A ind due to a common cause, or selection during the data-generating process.

Original: Thus, we define four kinds of shifts based on the causal graph: Independent ,
Revised: Thus, we define four kinds of shifts based on the causal graph: Independent ,

Original: Causal , Confounded , and Selected .
Revised: Causal , Confounded , and Selected .

Original: As we shall see, these shifts correspond to different independence constraints between observed variables.
Revised: As we shall see, these shifts correspond to different independence constraints between observed variables.

Original: Thus, in addition to the dataset, to fully specify the problem of multi-attribute shift generalization for a learning algorithm, we require knowledge of the kind of shift for each observed attribute.
Revised: Thus, in addition to the dataset, to fully specify the problem of multi-attribute shift generalization for a learning algorithm, we require knowledge of the kind of shift for each observed attribute.

Original: Definition 2.2.
Revised: Definition 2.2.

Original: Generalization under Multi-attribute shifts.
Revised: Generalization under Multi-attribute shifts.

Original: Given training data ( x i , a i , y i ) ni =1 and the type of causal relationship of each attribute A with the label Y , construct a realized causal graph G based on the canonical graph in Figure 2 and define P G as the set of all distributions obtained by changing the relationship between Y and each attribute while keeping the same graph (type of shift).
Revised: Given training data ( x i , a i , y i ) ni =1 and the type of causal relationship of each attribute A with the label Y , construct a realized causal graph G based on the canonical graph in Figure 2 and define P G as the set of all distributions obtained by changing the relationship between Y and each attribute while keeping the same graph (type of shift).

Original: The generalization goal is to learn an optimal risk-invariant predictor over P G .
Revised: The generalization goal is to learn an optimal risk-invariant predictor over P G .

Original: Availability of multiple attributes.
Revised: Availability of multiple attributes.

Original: Unlike the full causal graph, type of relationship between label and an attribute is often known.
Revised: Unlike the full causal graph, type of relationship between label and an attribute is often known.

Original: Suppl.
Revised: Suppl.

Original: A contains real-world examples where these relationships as well as attribute values are known.
Revised: A contains real-world examples where these relationships as well as attribute values are known.

Original: 3. Correct regularizer for multi-attribute shifts 3.1. Deriving conditional independence constraints for a risk-invariant representation
Revised: 3. Correct regularizer for multi-attribute shifts 3.1. Deriving conditional independence constraints for a risk-invariant representation

Original: We assume the predictor can be represented as g ( x ) =
Revised: We assume the predictor can be represented as g ( x ) =

Original: g 1 ( ϕ ( x )) where ϕ is the learnt representation.
Revised: g 1 ( ϕ ( x )) where ϕ is the learnt representation.

Original: To derive the constraints that should be satisfied by a risk-invariant g 1 ( ϕ ) , we utilize a strategy from past work (Mahajan et al., 2021;
Revised: To derive the constraints that should be satisfied by a risk-invariant g 1 ( ϕ ) , we utilize a strategy from past work (Mahajan et al., 2021;

Original: Veitch et al., 2021).
Revised: Veitch et al., 2021).

Original: We identify the conditional independence constraints satisfied by X c in the causal graph and enforce that ϕ should follow the same constraint.
Revised: We identify the conditional independence constraints satisfied by X c in the causal graph and enforce that ϕ should follow the same constraint.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Given a dataset ( x i , a i , y i ) ni =1 and a causal DAG G over ⟨ X c , X , A , Y ⟩ such that X c is the only variable (or set of variables) that causes Y and is not independent of X , then the conditional independence constraints satisfied by X c are necessary for a risk-invariant predictor over P G .
Revised: Given a dataset ( x i , a i , y i ) ni =1 and a causal DAG G over ⟨ X c , X , A , Y ⟩ such that X c is the only variable (or set of variables) that causes Y and is not independent of X , then the conditional independence constraints satisfied by X c are necessary for a risk-invariant predictor over P G .

Original: That is, if a predictor does not satisfy any of these constraints, then there exists a data distribution
Revised: That is, if a predictor does not satisfy any of these constraints, then there exists a data distribution

Original: P ′ ∈ P G such that predictor’s risk will be higher than its risk in other distributions.
Revised: P ′ ∈ P G such that predictor’s risk will be higher than its risk in other distributions.

Original: We examine two common constraints on independence between ϕ and a nuisance attribute: either unconditional (Albuquerque et al., 2020;
Revised: We examine two common constraints on independence between ϕ and a nuisance attribute: either unconditional (Albuquerque et al., 2020;

Original: Ganin et al., 2016b; Muandet et al., 2013) or conditional on the label Y
Revised: Ganin et al., 2016b; Muandet et al., 2013) or conditional on the label Y

Original: (Ghifary et al., 2016;
Revised: (Ghifary et al., 2016;

Original: Hu et al., 2019; Li et al., 2018c;d) (see Suppl.
Revised: Hu et al., 2019; Li et al., 2018c;d) (see Suppl.

Original: for details on these baseline methods).
Revised: for details on these baseline methods).

Original: Under the canonical graph from
Revised: Under the canonical graph from

Original: Figure 2(b), none of these constraints are valid because there could be a correlation path between X c and E (under the X-ray example, this can be because more women visit one hospital compared to the other).
Revised: Figure 2(b), none of these constraints are valid because there could be a correlation path between X c and E (under the X-ray example, this can be because more women visit one hospital compared to the other).

Original: When we simplify the graph by removing the correlation between object and
Revised: When we simplify the graph by removing the correlation between object and

Original: E (Figure 2(a))
Revised: E (Figure 2(a))

Original: , the unconditional constraint is true when
Revised: , the unconditional constraint is true when

Original: A ⊥⊥ Y ( A ∈ A ind ) but not always for A ind .
Revised: A ⊥⊥ Y ( A ∈ A ind ) but not always for A ind .

Original: For any attribute A ∈ A ind , if the relationship between Y and A is
Revised: For any attribute A ∈ A ind , if the relationship between Y and A is

Original: Confounded , then the unconditional constraint is correct; if it is Causal or Selected , then the conditional constraint is correct.
Revised: Confounded , then the unconditional constraint is correct; if it is Causal or Selected , then the conditional constraint is correct.

Original: Below we provide the set of valid constraints.
Revised: Below we provide the set of valid constraints.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Given a causal DAG with the structure as shown in Figure 2(a), the correct constraint depends on the relationship of label Y with the nuisance attributes A .
Revised: Given a causal DAG with the structure as shown in Figure 2(a), the correct constraint depends on the relationship of label Y with the nuisance attributes A .

Original: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =
Revised: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =

Original: A cause ∪ A conf ∪ A sel ).
Revised: A cause ∪ A conf ∪ A sel ).

Original: Then, the (conditional) independence constraints that X c should satisfy are,
Revised: Then, the (conditional) independence constraints that X c should satisfy are,

Original: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.
Revised: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.

Original: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.
Revised: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.

Original: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 
Revised: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E Selected X ⊥⊥ A sel

Revised: Y

Original:  Corollary 3.1.1.
Revised: X ⊥⊥ A sel Y, E Corollary 3.1.1.

Original: All the above derived constraints are valid for Graph 2(a).
Revised: All the above derived constraints are valid for Graph 2(a).

Original: However, in the presence of a correlation between E and Obj (Graph 2(b)), only the constraints conditioned on E hold true.
Revised: However, in the presence of a correlation between E and Obj (Graph 2(b)), only the constraints conditioned on E hold true.

Original: Hence, if information on Obj - E correlation is not available, it is advisable to use E - conditioned constraints.
Revised: Hence, if information on Obj - E correlation is not available, it is advisable to use E - conditioned constraints.

Original: While we list all constraints, if one of the attributes is unobserved (say
Revised: While we list all constraints, if one of the attributes is unobserved (say

Original: A ind or A ind is not available), then we use the subset of constraints derived for the observed features.
Revised: A ind or A ind is not available), then we use the subset of constraints derived for the observed features.

Original: 3.2. An algorithm for generalizing under multi-attribution shifts
Revised: 3.2. An algorithm for generalizing under multi-attribution shifts

Original: We now describe the proposed CACM algorithm.
Revised: We now describe the proposed CACM algorithm.

Original: CACM algorithm. Given a causal graph, the CACM algorithm first utilizes the steps highlighted above to identify the correct independence constraints.
Revised: Given a causal graph, the CACM algorithm first utilizes the steps highlighted above to identify the correct independence constraints.

Original: Then it applies those constraints as a regularizer to the standard
Revised: Then it applies those constraints as a regularizer to the standard

Original: ERM loss, g 1 , ϕ = arg min g 1 ,ϕ ;
Revised: ERM loss, g 1 , ϕ = arg min g 1 ,ϕ ;

Original: [Equation]
Revised: [Equation]

Original: + λ ∗ ( RegPenalty ) , where λ is a hyperparameter and ℓ is cross-entropy loss.
Revised: λ ∗ ( RegPenalty ) , where λ is a hyperparameter and ℓ is cross-entropy loss.

Original: We design the regularizer such that it optimizes for valid constraints over all observed variables
Revised: We design the regularizer such that it optimizes for valid constraints over all observed variables

Original: V ∈ A .
Revised: V ∈ A .

Original: If there is a choice between multiple constraints, we choose the constraint that will be valid over both Figure 2(a) and 2(b).
Revised: If there is a choice between multiple constraints, we choose the constraint that will be valid over both Figure 2(a) and 2(b).

Original: We describe implementation details, full algorithms and the regularization penalty for individual shifts in Section D.3.
Revised: We describe implementation details, full algorithm and the regularization penalty for individual shifts in Section D.3.

Original: 3.3. A fixed conditional independence constraint cannot work for all datasets
Revised: 3.3. A fixed conditional independence constraint cannot work for all datasets

Original: Since the observed data distribution can be identical for all three types of relationship between Y and A ind , the type of relationship cannot be learned from observed data.
Revised: Since the observed data distribution can be identical for all three types of relationship between Y and A ind , the type of relationship cannot be learned from observed data.

Original: Since the constraints are different for different relationship types, it implies that any algorithm relying on a single (conditional) independence constraint (Gretton et al., 2012; Arjovsky et al., 2019; Li et al., 2018b; Sun & Saenko, 2016) cannot work for all datasets.
Revised: Since the constraints are different for different relationship types, it implies that any algorithm relying on a single (conditional) independence constraint (Gretton et al., 2012; Arjovsky et al., 2019; Li et al., 2018b; Sun & Saenko, 2016) cannot work for all datasets.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Under the canonical causal graph in Figure 2, there exists no (conditional) independence constraint such that it is valid for all realizations of the graph as the type of multi-attribute shifts vary.
Revised: Under the canonical causal graph in Figure 2, there exists no (conditional) independence constraint such that it is valid for all realizations of the graph as the type of multi-attribute shifts vary.

Original: Thus, for any predictor algorithm for Y that uses a single type of (conditional) independence constraint, there exists a realized graph G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor across distributions in P G .
Revised: Thus, for any predictor algorithm for Y that uses a single type of (conditional) independence constraint, there exists a realized graph G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor across distributions in P G .

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4. Empirical Evaluation
Revised: 4. Empirical Evaluation

Original: We perform experiments on semi-synthetic (MNIST) and natural (small NORB) datasets to demonstrate our main claims: CACM with the correct graph-based constraints significantly outperforms these algorithms
Revised: We perform experiments on semi-synthetic (MNIST) and natural (small NORB) datasets to demonstrate our main claims: CACM with the correct graph-based constraints significantly outperforms these algorithms

Original: [Table]
Revised: [Table]

Original: and incorrect constraints cannot match the above accuracy.
Revised: and incorrect constraints cannot match the above accuracy.

Original: We compare to baseline algorithms: IRM (Arjovsky et al., 2019),
Revised: We compare to baseline algorithms: IRM (Arjovsky et al., 2019),

Original: VREx (Krueger et al., 2021), MMD (Li et al., 2018b),
Revised: VREx (Krueger et al., 2021), MMD (Li et al., 2018b),

Original: CORAL (Sun & Saenko, 2016), DANN (Gretton et al., 2012), Conditional-MMD (C-MMD) (Li et al., 2018b), and conditional-DANN (CDANN) (Li et al., 2018d).
Revised: CORAL (Sun & Saenko, 2016), DANN (Gretton et al., 2012), Conditional-MMD (C-MMD) (Li et al., 2018b) [Table] and conditional-DANN (CDANN) (Li et al., 2018d).

Original: Refer to
Revised: Refer to

Original: Suppl.
Revised: Suppl.

Original: D for further experimental details.
Revised: D for further experimental details.

Original: MNIST.
Revised: MNIST.

Original: Colored (Arjovsky et al., 2019) and Rotated (Ghifary et al., 2015)
Revised: Colored (Arjovsky et al., 2019) and Rotated (Ghifary et al., 2015)

Original: MNIST present A cause and A ind distribution shifts, respectively.
Revised: MNIST present A cause and A ind distribution shifts, respectively.

Original: We combine these to obtain a multi-attribute dataset with A cause = { color } and
Revised: We combine these to obtain a multi-attribute dataset with A cause = { color } and

Original: A ind = { rotation } .
Revised: A ind = { rotation } .

Original: small NORB.
Revised: small NORB.

Original: We use small NORB (LeCun et al., 2004;
Revised: We use small NORB (LeCun et al., 2004;

Original: Wiles et al., 2022), an object recognition dataset, to create a challenging task with multi-valued classes and attributes over realistic 3D objects with varying lighting ( l ) and azimuths ( azi ).
Revised: Wiles et al., 2022), an object recognition dataset, to create a challenging task with multi-valued classes and attributes over realistic 3D objects with varying lighting ( l ) and azimuths ( azi ).

Original: We create multi-attribution shifts,
Revised: We create multi-attribution shifts,

Original: [Table]
Revised: [Table]

Original: wherein there is a correlation between lighting condition
Revised: wherein there is a correlation between lighting condition

Original: A cause = { l } and object category y ; and A ind = { azi } that varies independently across domains.
Revised: A cause = { l } and object category y ; and A ind = { azi } that varies independently across domains.

Original: Further dataset details are in Suppl.
Revised: Further dataset details are in Suppl.

Original: D.2.
Revised: D.2.

Original: To compare the effect of shifts over two attributes, for each dataset, we also create single-attribute shift datasets involving a change in only one of the attributes.
Revised: To compare the effect of shifts over two attributes, for each dataset, we also create single-attribute shift datasets involving a change in only one of the attributes.

Original: Thus, we have three evaluation setups for each dataset: A cause , A ind and A cause ∪ A ind .
Revised: Thus, we have three evaluation setups for each dataset: A cause , A ind and A cause ∪ A ind .

Original: 4.1. Results
Revised: 4.1. Results

Original: Correct constraint derived from the causal graph matters.
Revised: Correct constraint derived from the causal graph matters.

Original: Table 1 shows the accuracy on MNIST dataset.
Revised: Table 1 shows the accuracy on MNIST dataset.

Original: Comparing the three prediction tasks, for all algorithms, accuracy is lowest under two-attribute shift ( A ind ∪ A cause ), reflecting the difficulty of a distribution shift over multiple attributes.
Revised: Comparing the three prediction tasks, for all algorithms, accuracy is lowest under two-attribute shift ( A ind ∪ A cause ), reflecting the difficulty of a distribution shift over multiple attributes.

Original: On the two-attribute shift task, all DG algorithms obtain less than 40% accuracy whereas CACM obtains a 14.5% absolute improvement.
Revised: On the two-attribute shift task, all DG algorithms obtain less than 40% accuracy whereas CACM obtains a 14.5% absolute improvement.

Original: Results on the small NORB dataset (Table 2) are similar – CACM obtains 69.6% accuracy on the two-attribute task while the nearest baseline is
Revised: Results on the small NORB dataset (Table 2) are similar – CACM obtains 69.6% accuracy on the two-attribute task while the nearest baseline is

Original: ERM at 64%.
Revised: ERM at 64%.

Original: CACM also obtains highest accuracy on the
Revised: CACM also obtains highest accuracy on the

Original: A cause task for both datasets.
Revised: A cause task for both datasets.

Original: On MNIST, we find that
Revised: On MNIST, we find that

Original: CACM achieves a substantially higher accuracy (70%) than
Revised: CACM achieves a substantially higher accuracy (70%) than

Original: IRM and VREx, just 5 units lower than the optimal 75%.
Revised: IRM and VREx, just 5 units lower than the optimal 75%.

Original: While the A ind task is relatively easier, algorithms optimizing for the correct constraint achieve highest accuracy.
Revised: While the A ind task is relatively easier, algorithms optimizing for the correct constraint achieve highest accuracy.

Original: Note that MMD, CORAL, DANN, and CACM are based on the same independence constraint (see Table 5 in Suppl.).
Revised: Note that MMD, CORAL, DANN, and CACM are based on the same independence constraint (see Table 5 in Suppl.).

Original: These results indicate the importance of regularization based on data-specific correct constraints for generalization.
Revised: These results indicate the importance of regularization based on data-specific correct constraints for generalization.

Original: Incorrect constraints hurt generalization.
Revised: Incorrect constraints hurt generalization.

Original: We now directly compare the effect of using correct versus incorrect (but commonly used) constraints for a dataset.
Revised: We now directly compare the effect of using correct versus incorrect (but commonly used) constraints for a dataset.

Original: To isolate the effect of a single constraint, we consider the single-attribute shift on A cause .
Revised: To isolate the effect of a single constraint, we consider the single-attribute shift on A cause .

Original: Comparing small NORB and MNIST (Table 3) reveals the importance of making the right structural assumptions.
Revised: Comparing small NORB and MNIST (Table 3) reveals the importance of making the right structural assumptions.

Original: Typically, DG algorithms assume that distribution of causal features X c does not change across domains.
Revised: Typically, DG algorithms assume that distribution of causal features X c does not change across domains.

Original: Then, both X c ⊥⊥ A cause | Y, E and X c ⊥⊥ A cause | Y should be correct constraints.
Revised: Then, both X c ⊥⊥ A cause | Y, E and X c ⊥⊥ A cause | Y should be correct constraints.

Original: However, conditioning on both Y and E provides a 5% point gain over conditioning on Y in NORB while the accuracy is comparable for
Revised: However, conditioning on both Y and E provides a 5% point gain over conditioning on Y in NORB while the accuracy is comparable for

Original: MNIST.
Revised: MNIST.

Original: Auxiliary information about the data-generating process explains the result: Different domains in MNIST include samples from the same distribution whereas small
Revised: Auxiliary information about the data-generating process explains the result: Different domains in MNIST include samples from the same distribution whereas small

Original: NORB domains are sampled from a different set of toy objects, thus creating a correlation between Obj and E (Figure 2(b)).
Revised: NORB domains are sampled from a different set of toy objects, thus creating a correlation between Obj and E (Figure 2(b)).

Original: Without such auxiliary information, such gains will be difficult.
Revised: Without such auxiliary information, such gains will be difficult.

Original: More ablations are in Suppl.
Revised: More ablations are in Suppl.

Original: E.
Revised: E.

Original: To investigate the differences in shifts over Independent , Causal and Confounded , results of evaluation on synthetic data are in Suppl.
Revised: To investigate the differences in shifts over Independent , Causal and Confounded , results of evaluation on synthetic data are in Suppl.

Original: E.2.
Revised: E.2.

Original: 5. Discussion
Revised: 5. Discussion

Original: We introduced CACM , an adaptive OoD generalization algorithm to characterize multi-attribute shifts and apply the correct independence constraints.
Revised: We introduced CACM , an adaptive OoD generalization algorithm to characterize multi-attribute shifts and apply the correct independence constraints.

Original: Through empirical experiments and theoretical analysis, we show the importance of modeling the causal relationships in the data-generating process.
Revised: Through empirical experiments and theoretical analysis, we show the importance of modeling the causal relationships in the data-generating process.

Original: The main limitation is that CACM does not address data sparsity – applying the constraints might be statistically inefficient if an attribute value is undersampled compared to others.
Revised: The main limitation is that CACM does not address data sparsity – applying the constraints might be statistically inefficient if an attribute value is undersampled compared to others.

Original: Future work includes statistical improvements in the regularization penalty (e.g., multiple regularization coefficients λ ).
Revised: Future work includes statistical improvements in the regularization penalty (e.g., multiple regularization coefficients λ ).

Original: A. Presence of auxiliary attribute information in datasets
Revised: A. Presence of auxiliary attribute information in datasets

Original: Unlike the full causal graph, attribute values as well as the relationships between class labels and attributes is often known.
Revised: Unlike the full causal graph, attribute values as well as the relationships between class labels and attributes is often known.

Original: CACM assumes access to attribute labels A only during training time, which are collected as part of the data collection process (e.g., as metadata with training data (Makar et al., 2022)).
Revised: CACM assumes access to attribute labels A only during training time, which are collected as part of the data collection process (e.g., as metadata with training data (Makar et al., 2022)).

Original: We start by discussing the availability of attributes in WILDS (Koh et al., 2021), a set of real-world datasets adapted for the domain generalization setting.
Revised: We start by discussing the availability of attributes in WILDS (Koh et al., 2021), a set of real-world datasets adapted for the domain generalization setting.

Original: Attribute labels available in the datasets include, the time (year) and region associated with satellite images in FMoW dataset (Christie et al. 2018) for predicting land use category, hospital from where the tissue patch was collected for tumor detection in
Revised: Attribute labels available in the datasets include, the time (year) and region associated with satellite images in FMoW dataset (Christie et al. 2018) for predicting land use category, hospital from where the tissue patch was collected for tumor detection in

Original: Camelyon17 dataset (Bandi et al., 2018) and the demographic information for CivilComments dataset (Borkan et al., 2019).
Revised: Camelyon17 dataset (Bandi et al., 2018) and the demographic information for CivilComments dataset (Borkan et al., 2019).

Original: (Koh et al., 2021) create different domains in WILDS using this metadata, consistent with our definition of E ∈ A as a special domain attribute.
Revised: (Koh et al., 2021) create different domains in WILDS using this metadata, consistent with our definition of E ∈ A as a special domain attribute.

Original: In addition, CACM requires the type of relationship between label Y and attributes.
Revised: In addition, CACM requires the type of relationship between label Y and attributes.

Original: This is often known, either based on how the dataset was collected or inferred based on domain knowledge or observation.
Revised: This is often known, either based on how the dataset was collected or inferred based on domain knowledge or observation.

Original: While the distinction between
Revised: While the distinction between

Original: A ind and A ind can be established using a statistical test of independence on a given dataset, the distinction between
Revised: A ind and A ind can be established using a statistical test of independence on a given dataset, the distinction between

Original: A cause , A sel and A conf within A ind must be provided by the user.
Revised: A cause , A sel and A conf within A ind must be provided by the user.

Original: In the above datasets, for FMoW, time can be considered an Independent attribute ( A ind ) since it reflects the time at which images are captured which is not correlated with Y ; whereas region is a Confounded attribute since certain regions associated with certain Y labels are over-represented due to ease of data collection.
Revised: In the above datasets, for FMoW, time can be considered an Independent attribute ( A ind ) since it reflects the time at which images are captured which is not correlated with Y ; whereas region is a Confounded attribute since certain regions associated with certain Y labels are over-represented due to ease of data collection.

Original: Note that region cannot lead to Causal shift since the decision to take images in a region was not determined by the final label nor Selected for the same reason that the decision was not taken based on values of Y .
Revised: Note that region cannot lead to Causal shift since the decision to take images in a region was not determined by the final label nor Selected for the same reason that the decision was not taken based on values of Y .

Original: Similarly, for the Camelyon17 dataset, it is known that differences in slide staining or image acquisition leads to variation in tissue slides across hospitals , thus implying that hospital is an Independent attribute ( A ind ) (Koh et al., 2021; Komura &
Revised: Similarly, for the Camelyon17 dataset, it is known that differences in slide staining or image acquisition leads to variation in tissue slides across hospitals , thus implying that hospital is an Independent attribute ( A ind ) (Koh et al., 2021; Komura &

Original: Ishikawa, 2018; Tellez et al., 2019); As another example from healthcare, a study in MIT Technology Review 2 discusses biased data where a person’s position ( A conf ) was spuriously correlated with disease prediction as patients lying down were more likely to be ill.
Revised: Ishikawa, 2018; Tellez et al., 2019); As another example from healthcare, a study in MIT Technology Review 2 discusses biased data where a person’s position ( A conf ) was spuriously correlated with disease prediction as patients lying down were more likely to be ill.

Original: As another example, (Sagawa* et al., 2020) adapt MultiNLI dataset for OoD generalization due to the presence of spurious correlation between negation words (attribute) and the contradiction label between “premise” and “hypothesis” inputs.
Revised: As another example, (Sagawa* et al., 2020) adapt MultiNLI dataset for OoD generalization due to the presence of spurious correlation between negation words (attribute) and the contradiction label between “premise” and “hypothesis” inputs.

Original: Here, negation words are a result of the contradiction label ( Causal shift), however this relationship between negation words and label may not always hold.
Revised: Here, negation words are a result of the contradiction label ( Causal shift), however this relationship between negation words and label may not always hold.

Original: Finally, for the CivilComments dataset, we expect the demographic features to be Confounded attributes as there could be biases which result in spurious correlation between comment toxicity and demographic information.
Revised: Finally, for the CivilComments dataset, we expect the demographic features to be Confounded attributes as there could be biases which result in spurious correlation between comment toxicity and demographic information.

Original: To provide examples showing the availability of attributes and their type of relationship with the label, Table 4 lists some popular datasets used for DG and the associated auxiliary information present as metadata.
Revised: To provide examples showing the availability of attributes and their type of relationship with the label, Table 4 lists some popular datasets used for DG and the associated auxiliary information present as metadata.

Original: In addition to above discussed datasets, we include the popularly used Waterbirds dataset (Sagawa* et al., 2020) where the type of background (land/water) is assigned to bird images based on bird label; hence, being a Causal attribute.
Revised: In addition to above discussed datasets, we include the popularly used Waterbirds dataset (Sagawa* et al., 2020) where the type of background (land/water) is assigned to bird images based on bird label; hence, being a Causal attribute.

Original: [Table]
Revised: [Table]

Original: Koh et al
Revised: Koh et al

Original: [Table] [Table]
Revised: [Table] [Table]

Original: et
Revised: et

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Koh et al., 2021)
Revised: Koh et al., 2021)

Original: [Table]
Revised: [Table]

Original: Datasets cited in this section
Revised: Datasets cited in this section

Original: G. Christie, N. Fendley, J. Wilson, and R. Mukherjee.
Revised: G. Christie, N. Fendley, J. Wilson, and R. Mukherjee.

Original: Functional map of the world.
Revised: Functional map of the world.

Original: In Computer Vision and
Revised: In Computer Vision and

Original: Pattern Recognition (CVPR), 2018.
Revised: Pattern Recognition (CVPR), 2018.

Original: P. Bandi, O. Geessink, Q. Manson, M. V. Dijk
Revised: P. Bandi, O. Geessink, Q. Manson, M. V. Dijk

Original: B. Related Work
Revised: B. Related Work

Original: Improving the robustness of models in the face of distribution shifts is a key challenge.
Revised: Improving the robustness of models in the face of distribution shifts is a key challenge.

Original: Several works have attempted to tackle the domain generalization problem (Wang et al., 2021;
Revised: Several works have attempted to tackle the domain generalization problem (Wang et al., 2021;

Original: Zhou et al., 2021) using different approaches – data augmentation (Cubuk et al., 2020; He et al., 2016; Zhu et al., 2017), and representation learning (Arjovsky et al., 2019; Deng et al., 2009; Higgins et al., 2017) being popular ones.
Revised: Zhou et al., 2021) using different approaches – data augmentation (Cubuk et al., 2020; He et al., 2016; Zhu et al., 2017), and representation learning (Arjovsky et al., 2019; Deng et al., 2009; Higgins et al., 2017) being popular ones.

Original: Trying to gauge the progress made by these approaches, Gulrajani and
Revised: Trying to gauge the progress made by these approaches, Gulrajani and

Original: Lopez-Paz (Gulrajani & Lopez-Paz, 2021) find that existing state-of-the-art DG algorithms do not improve over ERM.
Revised: Lopez-Paz (Gulrajani & Lopez-Paz, 2021) find that existing state-of-the-art DG algorithms do not improve over ERM.

Original: More recent work (Wiles et al., 2022;
Revised: More recent work (Wiles et al., 2022;

Original: Ye et al., 2022) empirically shows that different algorithms perform well over different distribution shifts, but no single algorithm performs consistently across all.
Revised: Ye et al., 2022) empirically shows that different algorithms perform well over different distribution shifts, but no single algorithm performs consistently across all.

Original: While they evaluate on single-attribute shift datasets, (Wiles et al., 2022) discuss the importance of having auxiliary knowledge of and evaluating methods under different underlying shifts.
Revised: While they evaluate on single-attribute shift datasets, (Wiles et al., 2022) discuss the importance of having auxiliary knowledge of and evaluating methods under different underlying shifts.

Original: To this end, we provide (1) multi-attribute shift benchmark datasets; (2) a causal interpretation of different kinds of shifts; and (3) an adaptive algorithm to identify the correct regularizer.
Revised: To this end, we provide (1) multi-attribute shift benchmark datasets; (2) a causal interpretation of different kinds of shifts; and (3) an adaptive algorithm to identify the correct regularizer.

Original: Causally-motivated learning.
Revised: Causally-motivated learning.

Original: There has been recent work focused on causal representation learning (Arjovsky et al., 2019; Krueger et al., 2021; Locatello et al., 2020;
Revised: There has been recent work focused on causal representation learning (Arjovsky et al., 2019; Krueger et al., 2021; Locatello et al., 2020;

Original: Sch ¨ olkopf et al., 2021) for OoD generalization.
Revised: Sch ¨ olkopf et al., 2021) for OoD generalization.

Original: While these works attempt to learn the constraints for causal features from input features, we show that it is necessary to model the data-generating process and have access to auxiliary attributes to obtain a risk-invariant predictor, especially in multi-attribute distribution shift setups.
Revised: While these works attempt to learn the constraints for causal features from input features, we show that it is necessary to model the data-generating process and have access to auxiliary attributes to obtain a risk-invariant predictor, especially in multi-attribute distribution shift setups.

Original: Recent research has shown how causal graphs can be used to characterize and analyze the different kinds of distribution shifts that occur in real-world settings (Makar et al., 2022; Veitch et al., 2021).
Revised: Recent research has shown how causal graphs can be used to characterize and analyze the different kinds of distribution shifts that occur in real-world settings (Makar et al., 2022; Veitch et al., 2021).

Original: Our approach is similar in motivation but we extend from single-domain, single-attribute setups in past work to formally introduce multi-attribute distribution shifts in more complex and real-world settings.
Revised: Our approach is similar in motivation but we extend from single-domain, single-attribute setups in past work to formally introduce multi-attribute distribution shifts in more complex and real-world settings.

Original: Additionally, we do not restrict ourselves to binary-valued classes and attributes.
Revised: Additionally, we do not restrict ourselves to binary-valued classes and attributes.

Original: C. Proofs C.1. Proof of Proposition 3. 
Revised: C. Proofs C.1. Proof of Proposition 3.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Given a dataset ( x i , a i , y i ) ni =1 and a causal DAG G over ⟨ X c , X , A , Y ⟩ such that X c is the only variable (or set of variables) that causes Y and is not independent of X , then the conditional independence constraints satisfied by
Revised: Given a dataset ( x i , a i , y i ) ni =1 and a causal DAG G over ⟨ X c , X , A , Y ⟩ such that X c is the only variable (or set of variables) that causes Y and is not independent of X , then the conditional independence constraints satisfied by

Original: X c are necessary for a risk-invariant predictor over P G .
Revised: X c are necessary for a risk-invariant predictor over P G .

Original: That is, if a predictor does not satisfy any of these constraints, then there exists
Revised: That is, if a predictor does not satisfy any of these constraints, then there exists

Original: a data distribution P ′ ∈
Revised: a data distribution P ′ ∈

Original: P G such that predictor’s risk will be higher than its risk in other distributions.
Revised: P G such that predictor’s risk will be higher than its risk in other distributions.

Original: Proof.
Revised: Proof.

Original: Let X , Y, X c be random variables where X c causes Y .
Revised: Let X , Y, X c be random variables where X c causes Y .

Original: X c also causes the observed features X
Revised: X c also causes the observed features X

Original: but X may be additionally affected by the attributes A .
Revised: but X may be additionally affected by the attributes A .

Original: Let ˆ y
Revised: Let ˆ y

Original: =
Revised: =

Original: g
Revised: g

Original: ( x ) be a candidate predictor.
Revised: ( x ) be a candidate predictor.

Original: Then g ( X ) represents a random vector based on a deterministic function g of X .
Revised: Then g ( X ) represents a random vector based on a deterministic function g of X .

Original: Suppose there is an independence constraint ψ that is satisfied by X c but not g ( X ) .
Revised: Suppose there is an independence constraint ψ that is satisfied by X c but not g ( X ) .

Original: Since A refers to the set of all other variables (attributes) that also cause X , A cannot be empty otherwise X is only caused by X c and hence would satisfy all independence constraints that X c satisfies.
Revised: Since A refers to the set of all other variables (attributes) that also cause X , A cannot be empty otherwise X is only caused by X c and hence would satisfy all independence constraints that X c satisfies.

Original: Below we show that such a predictor g is not risk-invariant: there exist two data distributions generated according to Definition 2.2 such that the risk of g is different for them.
Revised: Below we show that such a predictor g is not risk-invariant: there exist two data distributions generated according to Definition 2.2 such that the risk of g is different for them.

Original: Without loss of generality, we can write g ( x ) as,
Revised: Without loss of generality, we can write g ( x ) as,

Original: [Equation]
Revised: [Equation]

Original: where h is an arbitrary, non-zero, deterministic function of the random variable X c .
Revised: where h is an arbitrary, non-zero, deterministic function of the random variable X c .

Original: Since X c satisfies the (conditional) independence constraint ψ and h is a deterministic function, h
Revised: Since X c satisfies the (conditional) independence constraint ψ and h is a deterministic function, h

Original: ( X c ) also satisfies ψ .
Revised: ( X c ) also satisfies ψ .

Original: Also since the predictor g ( X ) does not satisfy the constraint ψ , it implies that the random vector g ′ ( X , X c ) cannot satisfy the constraint ψ .
Revised: Also since the predictor g ( X ) does not satisfy the constraint ψ , it implies that the random vector g ′ ( X , X c ) cannot satisfy the constraint ψ .

Original: Thus, g ′ ( X , X c ) cannot be a function of X c only.
Revised: Thus, g ′ ( X , X c ) cannot be a function of X c only.

Original: Since X has two parents, X c and A , this implies that g ′ ( X , X c ) and A are not independent.
Revised: Since X has two parents, X c and A , this implies that g ′ ( X , X c ) and A are not independent.

Original: Now, let us construct two data distributions P 1 and P 2 such that P ( X c , Y ) stays invariant, i.e., P 1 ( X c , Y )
Revised: Now, let us construct two data distributions P 1 and P 2 such that P ( X c , Y ) stays invariant, i.e., P 1 ( X c , Y )

Original: =
Revised: =

Original: P 2 ( X c , Y ) .
Revised: P 2 ( X c , Y ) .

Original: But P ( A ) can change or P ( A | Y ) can change.
Revised: But P ( A ) can change or P ( A | Y ) can change.

Original: Since A causes X , the conditional distribution P ( Y | X ) will also change.
Revised: Since A causes X , the conditional distribution P ( Y | X ) will also change.

Original: Further, since g ′ ( X , X c ) and A are not independent, P ( Y | g ′
Revised: Further, since g ′ ( X , X c ) and A are not independent, P ( Y | g ′

Original: ( X , X c )) will change,
Revised: ( X , X c )) will change,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., P 1 ( Y | g ′ ( X , X c )) ̸ =
Revised: e., P 1 ( Y | g ′ ( X , X c )) ̸ =

Original: P 2 ( Y | g ′ ( X , X c ))
Revised: P 2 ( Y | g ′ ( X , X c ))

Original: [Equation]
Revised: [Equation]

Original: The risk over any distribution P can be written as (using the cross-entropy loss),
Revised: The risk over any distribution P can be written as (using the cross-entropy loss),

Original: [Equation]
Revised: [Equation]

Original: The risk difference is,
Revised: The risk difference is,

Original: [Equation]
Revised: [Equation]

Original: where the second equality is because P 1 ( X c , Y )
Revised: where the second equality is because P 1 ( X c , Y )

Original: =
Revised: =

Original: P 2 ( X c , Y ) .
Revised: P 2 ( X c , Y ) .

Original: The risk of h ( X c )) would be the same across P 1 and Pbut not for g ′ since g ′ ( X , X c ) changes across the two distributions.
Revised: The risk of h ( X c )) would be the same across P 1 and Pbut not for g ′ since g ′ ( X , X c ) changes across the two distributions.

Original: Thus the absolute risk difference is non-zero,
Revised: Thus the absolute risk difference is non-zero,

Original: [Equation]
Revised: [Equation]

Original: and g is not a risk-invariant predictor.
Revised: and g is not a risk-invariant predictor.

Original: Hence, satisfying conditional independencies that X c satisfies is necessary for a risk-invariant predictor.
Revised: Hence, satisfying conditional independencies that X c satisfies is necessary for a risk-invariant predictor.

Original: C.2. Proof of Theorem 3. 
Revised: C.2. Proof of Theorem 3.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Given a causal DAG with the structure as shown in Figure 2(a), the correct constraint depends on the relationship of label Y with the nuisance attributes A .
Revised: Given a causal DAG with the structure as shown in Figure 2(a), the correct constraint depends on the relationship of label Y with the nuisance attributes A .

Original: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =
Revised: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =

Original: A cause ∪ A conf ∪ A sel ).
Revised: A cause ∪ A conf ∪ A sel ).

Original: Then, the (conditional) independence constraints that X c should satisfy are,
Revised: Then, the (conditional) independence constraints that X c should satisfy are,

Original: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.
Revised: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.

Original: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.
Revised: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.

Original: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 4.
Revised: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 4.

Original: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E
Revised: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E

Original: Proof.
Revised: Proof.

Original: The proof follows from d-separation (Pearl, 2009) on the causal graphs realized from Figure 2(a).
Revised: The proof follows from d-separation (Pearl, 2009) on the causal graphs realized from Figure 2(a).

Original: For each condition,
Revised: For each condition,

Original: Independent , Causal , Confounded and Selected , we provide the realized causal graphs below and derive the constraints.
Revised: Independent , Causal , Confounded and Selected , we provide the realized causal graphs below and derive the constraints.

Original: Independent: As we can see in Figure 3(a), we have a collider X on the path from X c to A ind and X c to E .
Revised: Independent: As we can see in Figure 3(a), we have a collider X on the path from X c to A ind and X c to E .

Original: Since there is a single path here, we obtain the independence constraints X c ⊥⊥ A ind and X c ⊥⊥ E .
Revised: Since there is a single path here, we obtain the independence constraints X c ⊥⊥ A ind and X c ⊥⊥ E .

Original: Additionally, we see that conditioning on Y or E would not block the path from X c to A ind , which results in the remaining constraints:
Revised: Additionally, we see that conditioning on Y or E would not block the path from X c to A ind , which results in the remaining constraints:

Original: [Equation]
Revised: [Equation]

Original: ind | Y ; X c ⊥⊥ A ind | E and X c ⊥⊥ A ind | Y, E .
Revised: ind | Y ; X c ⊥⊥ A ind | E and X c ⊥⊥ A ind | Y, E .

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: Causal: From Figure 3(b), we see that while the path X c
Revised: Causal: From Figure 3(b), we see that while the path X c

Original: → X → A cause from X c to A cause contains a collider X ,
Revised: → X → A cause from X c to A cause contains a collider X ,

Original: X c ̸⊥⊥ A cause due to the presence of node Y as a chain.
Revised: X c ̸⊥⊥ A cause due to the presence of node Y as a chain.

Original: By the d-separation criteria, X c and A cause are conditionally independent given Y =
Revised: By the d-separation criteria, X c and A cause are conditionally independent given Y =

Original: ⇒
Revised: ⇒

Original: X c ⊥⊥
Revised: X c ⊥⊥

Original: A cause | Y .
Revised: A cause | Y .

Original: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between X c and A cause =
Revised: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between X c and A cause =

Original: ⇒ X c ⊥⊥
Revised: ⇒ X c ⊥⊥

Original: A cause | Y, E .
Revised: A cause | Y, E .

Original: We get the constraint X c ⊥⊥ E since all paths connecting
Revised: We get the constraint X c ⊥⊥ E since all paths connecting

Original: X c to E contain a collider (collider X in X c → X → A cause → E , collider A cause in X c → Y → A cause → E ).
Revised: X c to E contain a collider (collider X in X c → X → A cause → E , collider A cause in X c → Y → A cause → E ).

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Confounded: From Figure 3(c), we see that all paths connecting X c and A conf contain a collider (collider X in
Revised: Confounded: From Figure 3(c), we see that all paths connecting X c and A conf contain a collider (collider X in

Original: X c → X → A conf , collider Y in X c → Y → C → A conf ).
Revised: X c → X → A conf , collider Y in X c → Y → C → A conf ).

Original: Hence, X c ⊥⊥ A conf .
Revised: Hence, X c ⊥⊥ A conf .

Original: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between X c and A conf =
Revised: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between X c and A conf =

Original: ⇒ X c ⊥⊥ A conf | E .
Revised: ⇒ X c ⊥⊥ A conf | E .

Original: We get the constraint X c ⊥⊥ E since all paths connecting X c and E also contain a collider (collider X in X c → X → A conf → E , collider Y in X c → Y → C → A conf → E ).
Revised: We get the constraint X c ⊥⊥ E since all paths connecting X c and E also contain a collider (collider X in X c → X → A conf → E , collider Y in X c → Y → C → A conf → E ).

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: Selected:
Revised: Selected:

Original: For the observed data, the selection variable is always conditioned on, with S = 1 indicating inclusion of sample in data.
Revised: For the observed data, the selection variable is always conditioned on, with S = 1 indicating inclusion of sample in data.

Original: The selection variable S is a collider in Figure 3(d) and we condition on it.
Revised: The selection variable S is a collider in Figure 3(d) and we condition on it.

Original: Hence, X c ̸⊥⊥ A sel .
Revised: Hence, X c ̸⊥⊥ A sel .

Original: Conditioning on Y breaks the edge X c → Y , and hence all paths between X c and A sel now contain a collider (collider X in X c → X → A sel ) =
Revised: Conditioning on Y breaks the edge X c → Y , and hence all paths between X c and A sel now contain a collider (collider X in X c → X → A sel ) =

Original: ⇒
Revised: ⇒

Original: X c ⊥⊥
Revised: X c ⊥⊥

Original: A sel | Y .
Revised: A sel | Y .

Original: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between
Revised: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between

Original: X c and A sel =
Revised: X c and A sel =

Original: ⇒ X c ⊥⊥
Revised: ⇒ X c ⊥⊥

Original: A sel | Y, E .
Revised: A sel | Y, E .

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: C.2.1. P ROOF OF C OROLLARY 3.1. 
Revised: C.2.1. P ROOF OF C OROLLARY 3.1.

Original: Corollary 3.1.1.
Revised: Corollary 3.1.1.

Original: All the above derived constraints are valid for Graph 2(a).
Revised: All the above derived constraints are valid for Graph 2(a).

Original: However, in the presence of a correlation between E and Obj (Graph 2(b)), only the constraints conditioned on E hold true.
Revised: However, in the presence of a correlation between E and Obj (Graph 2(b)), only the constraints conditioned on E hold true.

Original: If there is a correlation between Obj and E , X c ̸⊥⊥ E .
Revised: If there is a correlation between Obj and E , X c ̸⊥⊥ E .

Original: We can see from Figure 3 that in the presence of Obj − E correlation,
Revised: We can see from Figure 3 that in the presence of Obj − E correlation,

Original: X c ̸⊥⊥ A ind ; X c ̸⊥⊥ A ind | Y ( 3(a)), X c ̸⊥⊥ A cause | Y ( 3(b)), X c ̸⊥⊥ A conf ( 3(c)) and X c ̸⊥⊥ A sel | Y ( 3(d)).
Revised: X c ̸⊥⊥ A ind ; X c ̸⊥⊥ A ind | Y ( 3(a)), X c ̸⊥⊥ A cause | Y ( 3(b)), X c ̸⊥⊥ A conf ( 3(c)) and X c ̸⊥⊥ A sel | Y ( 3(d)).

Original: Hence, conditioning on environment E is required for the valid independence constraints.
Revised: Hence, conditioning on environment E is required for the valid independence constraints.

Original: C.3. Proof of Theorem 3. 
Revised: C.3. Proof of Theorem 3.

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Under the canonical causal graph in Figure 2, there exists no (conditional) independence constraint such that it is valid for all realizations of the graph as the type of multi-attribute shifts vary.
Revised: Under the canonical causal graph in Figure 2, there exists no (conditional) independence constraint such that it is valid for all realizations of the graph as the type of multi-attribute shifts vary.

Original: Thus, for any predictor algorithm for
Revised: Thus, for any predictor algorithm for

Original: Y that uses a single type of (conditional) independence constraint, there exists a realized graph G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor across distributions in P G .
Revised: Y that uses a single type of (conditional) independence constraint, there exists a realized graph G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor across distributions in P G .

Original: Proof.
Revised: Proof.

Original: The proof follows from an application of Theorem 3.1 and Proposition 3.1.
Revised: The proof follows from an application of Theorem 3.1 and Proposition 3.1.

Original: Under the canonical graph from
Revised: Under the canonical graph from

Original: Figure 2(a or b), the four types of attribute shifts possible are Independent , Causal , Confounded and Selected .
Revised: Figure 2(a or b), the four types of attribute shifts possible are Independent , Causal , Confounded and Selected .

Original: From the constraints provided for these four types of attribute shifts in Theorem 3.1, it is easy to observe that there is no single constraint that is satisfied across all four shifts.
Revised: From the constraints provided for these four types of attribute shifts in Theorem 3.1, it is easy to observe that there is no single constraint that is satisfied across all four shifts.

Original: Thus, given a data distribution (and hence, dataset) with specific types of multi-attribute shifts such that X c satisfies certain (conditional) independence constraints, it is always possible to change the type of at least one of the those shifts to create a new data distribution (dataset) where the same constraints will not hold.
Revised: Thus, given a data distribution (and hence, dataset) with specific types of multi-attribute shifts such that X c satisfies certain (conditional) independence constraints, it is always possible to change the type of at least one of the those shifts to create a new data distribution (dataset) where the same constraints will not hold.

Original: To prove the second claim, suppose that there exists a predictor for Y based on a single type of conditional independence constraint.
Revised: To prove the second claim, suppose that there exists a predictor for Y based on a single type of conditional independence constraint.

Original: Since the same constraint is not valid across all attribute shifts, we can always construct a data distribution (corresponding to a realized graph G ) where X c would not satisfy the same constraint, by changing the type of at least one attribute shift.
Revised: Since the same constraint is not valid across all attribute shifts, we can always construct a data distribution (corresponding to a realized graph G ) where X c would not satisfy the same constraint, by changing the type of at least one attribute shift.

Original: From Proposition 3.1, all conditional independence constraints satisfied by X c under G are necessary to be satisfied for a risk-invariant predictor.
Revised: From Proposition 3.1, all conditional independence constraints satisfied by X c under G are necessary to be satisfied for a risk-invariant predictor.

Original: Hence, for the class of distributions P G , a single constraint-based predictor cannot be a risk-invariant predictor.
Revised: Hence, for the class of distributions P G , a single constraint-based predictor cannot be a risk-invariant predictor.

Original: D. Experimental Details
Revised: D. Experimental Details

Original: All experiments are performed in PyTorch 1.10 with NVIDIA Tesla P40 and P100 GPUs.
Revised: All experiments are performed in PyTorch 1.10 with NVIDIA Tesla P40 and P100 GPUs.

Original: We build upon the code from
Revised: We build upon the code from

Original: DomainBed (Gulrajani & Lopez-Paz, 2021) and OoD-Bench (Ye et al., 2022).
Revised: DomainBed (Gulrajani & Lopez-Paz, 2021) and OoD-Bench (Ye et al., 2022).

Original: Regularizing on g 1 ( ϕ ( x )) provided better accuracy than ϕ ( x ) ; hence we adopt it for all our experiments.
Revised: Regularizing on g 1 ( ϕ ( x )) provided better accuracy than ϕ ( x ) ; hence we adopt it for all our experiments.

Original: D.1. Additional details about baseline methods
Revised: D.1. Additional details about baseline methods

Original: Table 5 lists the baseline methods we compare to, the independence constraints imposed and the statistics matched/optimized by each method across environments E .
Revised: Table 5 lists the baseline methods we compare to, the independence constraints imposed and the statistics matched/optimized by each method across environments E .

Original: [Table]
Revised: [Table]

Original: D.2. Datasets
Revised: D.2. Datasets

Original: MNIST.
Revised: MNIST.

Original: Rotated (Ghifary et al., 2015) and Colored MNIST (Arjovsky et al., 2019) present distinct distribution shifts.
Revised: Rotated (Ghifary et al., 2015) and Colored MNIST (Arjovsky et al., 2019) present distinct distribution shifts.

Original: While Rotated MNIST only has A ind wrt.
Revised: While Rotated MNIST only has A ind wrt.

Original: rotation attribute ( R ), Colored MNIST only has A cause wrt.
Revised: rotation attribute ( R ), Colored MNIST only has A cause wrt.

Original: color attribute ( C ).
Revised: color attribute ( C ).

Original: We combine these datasets to obtain a multi-attribute dataset with A cause =
Revised: We combine these datasets to obtain a multi-attribute dataset with A cause =

Original: { C } and A ind = { R } .
Revised: { C } and A ind = { R } .

Original: Each domain
Revised: Each domain

Original: E i has a specific rotation angle r i and a specific correlation corr i between color C and label Y .
Revised: E i has a specific rotation angle r i and a specific correlation corr i between color C and label Y .

Original: Our setup consists ofdomains: E 1 , E 2 ∈ E tr (training), E 3 ∈ E te (test).
Revised: Our setup consists ofdomains: E 1 , E 2 ∈ E tr (training), E 3 ∈ E te (test).

Original: We define corr i
Revised: We define corr i

Original: = P
Revised: = P

Original: Y =
Revised: Y =

Original: Y =(a)
Revised: Y =(a)

Original: Y =
Revised: Y =

Original: Y =(b)
Revised: Y =(b)

Original: Y =
Revised: Y =

Original: Y =(c) small NORB.
Revised: Y =(c) small NORB.

Original: Moving beyond simple binary classification, we use small NORB (LeCun et al., 2004), an object recognition dataset, to create a challenging setup with multi-valued classes and attributes over realistic 3D objects.
Revised: Moving beyond simple binary classification, we use small NORB (LeCun et al., 2004), an object recognition dataset, to create a challenging setup with multi-valued classes and attributes over realistic 3D objects.

Original: It consists of images of toys of five categories with varying lighting ( l ), elevation ( ele ) and azimuths ( azi ).
Revised: It consists of images of toys of five categories with varying lighting ( l ), elevation ( ele ) and azimuths ( azi ).

Original: The objective is to classify unseen samples of the five categories.
Revised: The objective is to classify unseen samples of the five categories.

Original: (Wiles et al., 2022) introduced single-attribute shifts for this dataset.
Revised: (Wiles et al., 2022) introduced single-attribute shifts for this dataset.

Original: We combine them to yield A cause =
Revised: We combine them to yield A cause =

Original: { l } wherein there is a correlation between lighting condition l and toy category y ; and A ind = { azi } that varies independently across domains.
Revised: { l } wherein there is a correlation between lighting condition l and toy category y ; and A ind = { azi } that varies independently across domains.

Original: Training domains have 0.9 and 0.95 spurious correlation with l whereas there is no correlation in test domain.
Revised: Training domains have 0.9 and 0.95 spurious correlation with l whereas there is no correlation in test domain.

Original: We add 5% label noise in all environments.
Revised: We add 5% label noise in all environments.

Original: We use ResNet-18 (pre-trained on ImageNet) for all settings and fine tune for our task.
Revised: We use ResNet-18 (pre-trained on ImageNet) for all settings and fine tune for our task.

Original: Y =
Revised: Y =

Original: Y = 1 Y =
Revised: Y = 1 Y =

Original: Y = 

Revised: Y =

Original: Y =(a)
Revised: Y =(a)

Original: Y =
Revised: Y =

Original: Y = 1 Y =
Revised: Y = 1 Y =

Original: Y = 4 Y =(b)
Revised: Y = 4 Y =(b)

Original: Y =
Revised: Y =

Original: Y = 1 Y =
Revised: Y = 1 Y =

Original: Y = 

Revised: Y =

Original: Y =(c)
Revised: Y =(c)

Original: D.3. Implementation details
Revised: D.3. Implementation details

Original: All methods are trained using Adam optimizer.
Revised: All methods are trained using Adam optimizer.

Original: MNIST dataset is trained for 5000 steps (default in DomainBed (Gulrajani &
Revised: MNIST dataset is trained for 5000 steps (default in DomainBed (Gulrajani &

Original: Lopez-Paz, 2021)) while small NORB is trained for 2000 steps.
Revised: Lopez-Paz, 2021)) while small NORB is trained for 2000 steps.

Original: Consistent with the default value in DomainBed, we use a batch size 64 for MNIST and 128 for small NORB.
Revised: Consistent with the default value in DomainBed, we use a batch size 64 for MNIST and 128 for small NORB.

Original: Regularization Penalty.
Revised: Regularization Penalty.

Original: Since A includes multiple attributes, the regularizer penalty depends on the type of distribution shift for each attribute.
Revised: Since A includes multiple attributes, the regularizer penalty depends on the type of distribution shift for each attribute.

Original: For instance, for A ∈ A ind ( Independent ), to enforce ϕ ( x ) ⊥⊥ A , we aim to minimize the distributional discrepancy between P ( g ( x ) |
Revised: For instance, for A ∈ A ind ( Independent ), to enforce ϕ ( x ) ⊥⊥ A , we aim to minimize the distributional discrepancy between P ( g ( x ) |

Original: A = a i ) and P ( g ( x ) | A = a j ) , for all i, j values of A .
Revised: A = a i ) and P ( g ( x ) | A = a j ) , for all i, j values of A .

Original: However, the same constraint is applicable on E .
Revised: However, the same constraint is applicable on E .

Original: So if domain variable E is available, it is statistically efficient to apply the constraint on
Revised: So if domain variable E is available, it is statistically efficient to apply the constraint on

Original: E since there would typically be multiple closely related values of A in a domain (e.g., slide stains collected from one hospital may be spread over similar colors, but not exactly the same).
Revised: E since there would typically be multiple closely related values of A in a domain (e.g., slide stains collected from one hospital may be spread over similar colors, but not exactly the same).

Original: Hence, we apply the constraint on distributions
Revised: Hence, we apply the constraint on distributions

Original: P ( g 1 ( ϕ ( x ))
Revised: P ( g 1 ( ϕ ( x ))

Original: | E = E i ) and P ( g 1 ( ϕ ( x ))
Revised: | E = E i ) and P ( g 1 ( ϕ ( x ))

Original: | E = E j ) if E is observed (and A may/may not be unobserved), otherwise we apply the constraint over A .
Revised: | E = E j ) if E is observed (and A may/may not be unobserved), otherwise we apply the constraint over A .

Original: [Equation]
Revised: [Equation]

Original: For A ∈ A cause ( Causal ), following Theorem 3.1, we consider distributions P ( g 1 ( ϕ ( x ))
Revised: For A ∈ A cause ( Causal ), following Theorem 3.1, we consider distributions P ( g 1 ( ϕ ( x ))

Original: |
Revised: |

Original: A = a i , Y = y ) and
Revised: A = a i , Y = y ) and

Original: P ( g 1 ( phi ( x ))
Revised: P ( g 1 ( phi ( x ))

Original: | A = a j , Y = y ) .
Revised: | A = a j , Y = y ) .

Original: We additionally condition on domain E as there may be a correlation between E and Obj (Figure 2(b)), which renders other constraints incorrect (Corollary 3.1.1).
Revised: We additionally condition on domain E as there may be a correlation between E and Obj (Figure 2(b)), which renders other constraints incorrect (Corollary 3.1.1).

Original: RegPenalty
Revised: RegPenalty

Original: [Equation]
Revised: [Equation]

Original: The final RegPenalty is a sum of penalties over all attributes, RegPenalty = (cid:80)
Revised: The final RegPenalty is a sum of penalties over all attributes, RegPenalty = (cid:80)

Original: A ∈
Revised: A ∈

Original: A Penalty A .
Revised: A Penalty A .

Original: We choose the Maximum
Revised: We choose the Maximum

Original: Mean Discrepancy ( MMD ) (Gretton et al., 2012) metric to implement our penalty (although, in principle, any estimable metric for enforcing conditional independence would work).
Revised: Mean Discrepancy ( MMD ) (Gretton et al., 2012) metric to implement our penalty (although, in principle, any estimable metric for enforcing conditional independence would work).

Original: Unlike prior work (Makar et al., 2022; Veitch et al., 2021), we do not restrict ourselves to binary-valued attributes and classes.
Revised: Unlike prior work (Makar et al., 2022; Veitch et al., 2021), we do not restrict ourselves to binary-valued attributes and classes.

Original: We provide the CACM algorithm for a general graph G below.
Revised: We provide the CACM algorithm for a general graph G below.

Original: Algorithm 1 CACM
Revised: Algorithm 1 CACM

Original: Input: Dataset ( x i , a i , y i ) ni =1 , causal DAG G
Revised: Input: Dataset ( x i , a i , y i ) ni =1 , causal DAG G

Original: Output: Function g ( x ) =
Revised: Output: Function g ( x ) =

Original: g 1 ( ϕ ( x )) :
Revised: g 1 ( ϕ ( x )) :

Original: X → Y
Revised: X → Y

Original: A ← set of observed variables in G except Y, E (special domain attribute)
Revised: A ← set of observed variables in G except Y, E (special domain attribute)

Original: C ← {} { mapping of A to A s }
Revised: C ← {} { mapping of A to A s }

Original: Phase I: Derive correct independence constraints for A ∈ A do if ( X c , A ) are d-separated then
Revised: Phase I: Derive correct independence constraints for A ∈ A do if ( X c , A ) are d-separated then

Original: X c ⊥⊥ A is a valid independence constraint else if ( X c , A ) are d-separated conditioned on any subset A s of the remaining observed variables in A \ { A } then
Revised: X c ⊥⊥ A is a valid independence constraint else if ( X c , A ) are d-separated conditioned on any subset A s of the remaining observed variables in A \ { A } then

Original: X c ⊥⊥ A | A s is a valid independence constraint
Revised: X c ⊥⊥ A | A s is a valid independence constraint

Original: C
Revised: C

Original: [ A ] =
Revised: [ A ] =

Original: A s end if end for
Revised: A s end if end for

Original: Phase II: Apply regularization penalty using constraints derived for
Revised: Phase II: Apply regularization penalty using constraints derived for

Original: A ∈
Revised: A ∈

Original: A do if X c ⊥⊥ A then
Revised: A do if X c ⊥⊥ A then

Original: RegPenalty A = (cid:80)
Revised: RegPenalty A = (cid:80)

Original: | E | (cid:80) | A | i =1 (cid:80)
Revised: | E | (cid:80) | A | i =1 (cid:80)

Original: j>i MMD( P
Revised: j>i MMD( P

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: , P
Revised: , P

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: ϕ
Revised: ϕ

Original: [Equation]
Revised: [Equation]

Original: x ))
Revised: x ))

Original: | A j ))
Revised: | A j ))

Original: else if A is in C then
Revised: else if A is in C then

Original: [Equation]
Revised: [Equation]

Original: end if end for
Revised: end if end for

Original: RegPenalty =
Revised: RegPenalty =

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: A g
Revised: A g

Original: [Equation]
Revised: [Equation]

Original: min
Revised: min

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Remark.
Revised: Remark.

Original: If E is observed, we always condition on E because of Corollary 3.1.1.
Revised: If E is observed, we always condition on E because of Corollary 3.1.1.

Original: We provide the regularization penalty ( RegPenalty ) for Independent , Causal , Confounded and Selected shifts for our causal graph in Figure 2.
Revised: We provide the regularization penalty ( RegPenalty ) for Independent , Causal , Confounded and Selected shifts for our causal graph in Figure 2.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We want to emphasize that the constraints from the CACM algorithm are necessary but not sufficient.
Revised: We want to emphasize that the constraints from the CACM algorithm are necessary but not sufficient.

Original: While regularizers like CACM restrict the set of possible solutions to a smaller subset that contains X c (Mahajan et al., 2021), they are not guaranteed to return X c .
Revised: While regularizers like CACM restrict the set of possible solutions to a smaller subset that contains X c (Mahajan et al., 2021), they are not guaranteed to return X c .

Original: Formally, X c is not identified under the current graph.
Revised: Formally, X c is not identified under the current graph.

Original: Model Selection.
Revised: Model Selection.

Original: We create 90% and 10% splits from each domain to be used for training and model selection (as needed) respectively.
Revised: We create 90% and 10% splits from each domain to be used for training and model selection (as needed) respectively.

Original: For our main results, we use a validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al., 2019;
Revised: For our main results, we use a validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al., 2019;

Original: Ye et al., 2022; Wiles et al., 2022).
Revised: Ye et al., 2022; Wiles et al., 2022).

Original: Specifically, we adopt the test-domain validation from DomainBed where early stopping is not allowed and all models are trained for the same fixed number of steps to limit test domain access.
Revised: Specifically, we adopt the test-domain validation from DomainBed where early stopping is not allowed and all models are trained for the same fixed number of steps to limit test domain access.

Original: We additionally report results using test-domain validation with early stopping as well as train-domain validation in Suppl.
Revised: We additionally report results using test-domain validation with early stopping as well as train-domain validation in Suppl.

Original: E. Train-domain validation uses a validation set that follows the distribution of the training domains.
Revised: E. Train-domain validation uses a validation set that follows the distribution of the training domains.

Original: D.4. Hyperparameter search
Revised: D.4. Hyperparameter search

Original: Following DomainBed (Gulrajani & Lopez-Paz, 2021), we perform a random search 20 times over the hyperparameter distribution and this process is repeated for total 3 seeds.
Revised: Following DomainBed (Gulrajani & Lopez-Paz, 2021), we perform a random search 20 times over the hyperparameter distribution and this process is repeated for total 3 seeds.

Original: The best models are obtained across the three seeds over which we compute the mean and standard error.
Revised: The best models are obtained across the three seeds over which we compute the mean and standard error.

Original: The hyperparameter search space for all datasets and algorithms is given in Table 16.
Revised: The hyperparameter search space for all datasets and algorithms is given in Table 16.

Original: E. Results
Revised: E. Results

Original: [Table]
Revised: [Table]

Original: E.1. Incorrect constraints hurt generalization.
Revised: E.1. Incorrect constraints hurt generalization.

Original: Here, we present additional experiments to support the claim in Section 4.1 that incorrect constraints hurt model generalization.
Revised: Here, we present additional experiments to support the claim in Section 4.1 that incorrect constraints hurt model generalization.

Original: Theorem 3.1 provides the correct constraint for A cause : X c ⊥⊥ A cause | Y, E .
Revised: Theorem 3.1 provides the correct constraint for A cause : X c ⊥⊥ A cause | Y, E .

Original: In addition, using d-separation on Figure 2, we see the following invalid constraints, X c ⊥⊥ A cause | E , X c ⊥⊥ A cause .
Revised: In addition, using d-separation on Figure 2, we see the following invalid constraints, X c ⊥⊥ A cause | E , X c ⊥⊥ A cause .

Original: Without knowing that the shift corresponds to a Causal shift, one may apply these constraints that do not condition on the class.
Revised: Without knowing that the shift corresponds to a Causal shift, one may apply these constraints that do not condition on the class.

Original: Results on small NORB (Table 6) show that using the incorrect constraint has an adverse effect on model performance.
Revised: Results on small NORB (Table 6) show that using the incorrect constraint has an adverse effect on model performance.

Original: The correct constraint yields 85% accuracy while the best incorrect constraint achieves 79.7%.
Revised: The correct constraint yields 85% accuracy while the best incorrect constraint achieves 79.7%.

Original: Moreover, application of the incorrect constraint is sensitive to the λ (regularization weight)
Revised: Moreover, application of the incorrect constraint is sensitive to the λ (regularization weight)

Original: parameter (Figure 6): as λ increases, accuracy drops to less than 40%.
Revised: parameter (Figure 6): as λ increases, accuracy drops to less than 40%.

Original: However, accuracy with the correct constraint stays invariant across different values of λ .
Revised: However, accuracy with the correct constraint stays invariant across different values of λ .

Original: Finally, we consider the multi-attribute shift setting for small NORB to demonstrate the significance of applying the correct constraints from the causal graph.
Revised: Finally, we consider the multi-attribute shift setting for small NORB to demonstrate the significance of applying the correct constraints from the causal graph.

Original: In addition to applying the correct CACM constraints, we consider a case where we interchange the variables before inputting to CACM algorithm
Revised: In addition to applying the correct CACM constraints, we consider a case where we interchange the variables before inputting to CACM algorithm

Original: ( A ind gets used as A cause and vice-versa) and then apply the resultant (incorrect) constraints.
Revised: ( A ind gets used as A cause and vice-versa) and then apply the resultant (incorrect) constraints.

Original: Accuracy with interchanged variables ( 65 . 1 ± 1 . 6 ) is lower than that of correct CACM ( 69 . 6 ± 1 . 6 ).
Revised: Accuracy with interchanged variables ( 65 . 1 ± 1 . 6 ) is lower than that of correct CACM ( 69 . 6 ± 1 . 6 ).

Original: E.2. Synthetic Dataset
Revised: E.2. Synthetic Dataset

Original: We create a synthetic dataset to investigate the differences in constraints over Independent , Causal and Confounded shifts.
Revised: We create a synthetic dataset to investigate the differences in constraints over Independent , Causal and Confounded shifts.

Original: We also study the existing DG algorithms in a fairer synthetic setting and show their inability to close the performance gap with CACM even in such setting (Section E.3).
Revised: We also study the existing DG algorithms in a fairer synthetic setting and show their inability to close the performance gap with CACM even in such setting (Section E.3).

Original: Dataset description.
Revised: Dataset description.

Original: Our synthetic dataset is constructed based on the data-generating processes of the slab dataset (Mahajan et al., 2021; Shah et al., 2020).
Revised: Our synthetic dataset is constructed based on the data-generating processes of the slab dataset (Mahajan et al., 2021; Shah et al., 2020).

Original: The original slab dataset was introduced by (Shah et al., 2020) to demonstrate the simiplicity bias in neural networks as they learn the linear feature which is easier to learn in comparison to the slab feature.
Revised: The original slab dataset was introduced by (Shah et al., 2020) to demonstrate the simiplicity bias in neural networks as they learn the linear feature which is easier to learn in comparison to the slab feature.

Original: Our extended slab dataset, adds to the setting from (Mahajan et al., 2021) by using non-binary attributes and class labels to create a more challenging task and allows us to study DG algorithms in the presence of linear spurious features.
Revised: Our extended slab dataset, adds to the setting from (Mahajan et al., 2021) by using non-binary attributes and class labels to create a more challenging task and allows us to study DG algorithms in the presence of linear spurious features.

Original: Our dataset consists of label Y ( | Y | = 5) and 3-dimensional input X consisting of features X c , A ind and A ind .
Revised: Our dataset consists of label Y ( | Y | = 5) and 3-dimensional input X consisting of features X c , A ind and A ind .

Original: This is consistent with the graph in Figure 2 where attributes and causal features together determine observed features X ; we concatenate X c , A ind and A ind to generate X in our synthetic setup.
Revised: This is consistent with the graph in Figure 2 where attributes and causal features together determine observed features X ; we concatenate X c , A ind and A ind to generate X in our synthetic setup.

Original: Causal feature X c has a non-linear “slab” relationship with Y while A ind has a linear, Causal relationship with Y .
Revised: Causal feature X c has a non-linear “slab” relationship with Y while A ind has a linear, Causal relationship with Y .

Original: A ind is independent of Y and has varying uniform distribution p ind across environments.
Revised: A ind is independent of Y and has varying uniform distribution p ind across environments.

Original: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).
Revised: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).

Original: X c has a uniform distribution Uniform[0 , 1] across all environments.
Revised: X c has a uniform distribution Uniform[0 , 1] across all environments.

Original: [Equation]
Revised: [Equation]

Original: Hence, we have a five-way classification setup with multi-valued attributes and multi-attribute distribution shifts.
Revised: Hence, we have a five-way classification setup with multi-valued attributes and multi-attribute distribution shifts.

Original: Following (Mahajan et al., 2021), the two training domains have p as 0.9 and 1.0, and the test domain has p = 0.0.
Revised: Following (Mahajan et al., 2021), the two training domains have p as 0.9 and 1.0, and the test domain has p = 0.0.

Original: We add 10% noise to Y in all environments.
Revised: We add 10% noise to Y in all environments.

Original: We use the default 3-layer MLP architecture from DomainBed and use mean difference (L2) instead of MMD as the regularization penalty given the simplicity of the data.
Revised: We use the default 3-layer MLP architecture from DomainBed and use mean difference (L2) instead of MMD as the regularization penalty given the simplicity of the data.

Original: Experiments.
Revised: Experiments.

Original: We run all baselines and CACM on the synthetic dataset similar to experiments in Section 4 (Table 7).
Revised: We run all baselines and CACM on the synthetic dataset similar to experiments in Section 4 (Table 7).

Original: We can see that CACM significantly outperforms all algorithms for Causal and multi-attribute shifts.
Revised: We can see that CACM significantly outperforms all algorithms for Causal and multi-attribute shifts.

Original: As discussed previously,
Revised: As discussed previously,

Original: A ind is a relatively easier task; however, algorithms optimizing for the correct constraint achieve highest accuracy.
Revised: A ind is a relatively easier task; however, algorithms optimizing for the correct constraint achieve highest accuracy.

Original: Note that MMD, CORAL, DANN, and CACM are based on the same independence constraint in the presence of A ind (see
Revised: Note that MMD, CORAL, DANN, and CACM are based on the same independence constraint in the presence of A ind (see

Original: Table 5 in Suppl., Theorem 3.1).
Revised: Table 5 in Suppl., Theorem 3.1).

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E.3. Providing attribute information to DG algorithms for a fairer comparison
Revised: E.3. Providing attribute information to DG algorithms for a fairer comparison

Original: CACM leverages attribute labels to apply the correct independence constraints derived from the causal graph.
Revised: CACM leverages attribute labels to apply the correct independence constraints derived from the causal graph.

Original: However, existing DG algorithms only use the input features X and the domain attribute.
Revised: However, existing DG algorithms only use the input features X and the domain attribute.

Original: Here we provide this attribute information to existing DG algorithms to create a more favorable setting for their application.
Revised: Here we provide this attribute information to existing DG algorithms to create a more favorable setting for their application.

Original: We show that even in a relatively fairer setup, these algorithms are not able to close the performance gap with CACM , showing the importance of the causal information through graphs.
Revised: We show that even in a relatively fairer setup, these algorithms are not able to close the performance gap with CACM , showing the importance of the causal information through graphs.

Original: We consider our Synthetic dataset with Causal distribution shift where our observed features X =
Revised: We consider our Synthetic dataset with Causal distribution shift where our observed features X =

Original: ( X c , A cause ) .
Revised: ( X c , A cause ) .

Original: Note that by construction of X , since one of our input dimensions already consists of A cause , we explicitly make A cause available to all DG algorithms for applying their respective constraints.
Revised: Note that by construction of X , since one of our input dimensions already consists of A cause , we explicitly make A cause available to all DG algorithms for applying their respective constraints.

Original: Thus, in the synthetic setup, all baselines do receive information about A cause in addition to the domain attribute E .
Revised: Thus, in the synthetic setup, all baselines do receive information about A cause in addition to the domain attribute E .

Original: As a more informative way of providing the attribute information ( A cause ) for existing DG algorithms, we run a separate experiment where the attribute is provided as the domain indicator.
Revised: As a more informative way of providing the attribute information ( A cause ) for existing DG algorithms, we run a separate experiment where the attribute is provided as the domain indicator.

Original: Using the same underlying data distribution, we group the data (i.e., create environments/domains) based on A cause i.e, each environment E has samples with same value of
Revised: Using the same underlying data distribution, we group the data (i.e., create environments/domains) based on A cause i.e, each environment E has samples with same value of

Original: A cause .
Revised: A cause .

Original: In this setup (Table 8, third column), we see MMD, CORAL, DANN and CDANN show significant improvement in accuracy but the best performance is still 36% lower than CACM while showing high estimate variance.
Revised: In this setup (Table 8, third column), we see MMD, CORAL, DANN and CDANN show significant improvement in accuracy but the best performance is still 36% lower than CACM while showing high estimate variance.

Original: This reinforces our motivation to use the causal graph of the data-generating process to derive the constraint, as the attribute values alone are not sufficient.
Revised: This reinforces our motivation to use the causal graph of the data-generating process to derive the constraint, as the attribute values alone are not sufficient.

Original: We also see IRM and VREx perform much worse than earlier, highlighting the sensitivity of DG algorithms to domain definition.
Revised: We also see IRM and VREx perform much worse than earlier, highlighting the sensitivity of DG algorithms to domain definition.

Original: In contrast, CACM uses the causal graph to study the structural relationships and derive the regularization penalty, which remains the same in this new dataset too.
Revised: In contrast, CACM uses the causal graph to study the structural relationships and derive the regularization penalty, which remains the same in this new dataset too.

Original: E.4. Comparing constraints for Confounded vs Causal shift
Revised: E.4. Comparing constraints for Confounded vs Causal shift

Original: Here, we extend our experiments from the main paper to consider a Confounded shift setting.
Revised: Here, we extend our experiments from the main paper to consider a Confounded shift setting.

Original: In Theorem 3.1, we see that for the Causal shift, X c ⊥⊥ A cause | Y ; X c ̸⊥⊥ A cause (also, X c ⊥⊥ A cause | Y, E ; X c ̸⊥⊥ A cause | E ) whereas for the
Revised: In Theorem 3.1, we see that for the Causal shift, X c ⊥⊥ A cause | Y ; X c ̸⊥⊥ A cause (also, X c ⊥⊥ A cause | Y, E ; X c ̸⊥⊥ A cause | E ) whereas for the

Original: Confounded shift, X c ⊥⊥ A conf
Revised: Confounded shift, X c ⊥⊥ A conf

Original: ; X c ̸⊥⊥ A conf | Y (also, X c ⊥⊥ A conf | E ; X c ̸⊥⊥ A conf | Y, E ).
Revised: ; X c ̸⊥⊥ A conf | Y (also, X c ⊥⊥ A conf | E ; X c ̸⊥⊥ A conf | Y, E ).

Original: We construct a synthetic setup with Confounded shift to demonstrate the importance of using the valid independence constraints for different kinds of shifts.
Revised: We construct a synthetic setup with Confounded shift to demonstrate the importance of using the valid independence constraints for different kinds of shifts.

Original: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).
Revised: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).

Original: X c has a uniform distribution Uniform[0 , 1] across all environments.
Revised: X c has a uniform distribution Uniform[0 , 1] across all environments.

Original: Our confounding variable c has different functional relationships with Y and A conf which vary across environments.
Revised: Our confounding variable c has different functional relationships with Y and A conf which vary across environments.

Original: Our observed input X is 2-dimensional and formed by concatenating X c and A conf .
Revised: Our observed input X is 2-dimensional and formed by concatenating X c and A conf .

Original: c E
Revised: c E

Original: [Equation]
Revised: [Equation]

Original: E 2 = (cid:26) 1 with prob.
Revised: E 2 = (cid:26) 1 with prob.

Original: = 0 .0 with prob.
Revised: = 0 .0 with prob.

Original: = 0 .c
Revised: = 0 .c

Original: E 3 = (cid:26) 1 with prob.
Revised: E 3 = (cid:26) 1 with prob.

Original: = 0 .0 with prob.
Revised: = 0 .0 with prob.

Original: = 0 .y
Revised: = 0 .y

Original: true =    0 if X c ∈ [0 , 0 . 25) 

Revised: true =    0 if X c ∈ [0 , 0 . 25)

Original: if X c ∈ [0 . 25 , 0 . 5) 2 if X c ∈ [0 . 5 , 0 .
Revised: if X c ∈ [0 . 25 , 0 . 5) 2 if X c ∈ [0 . 5 , 0 .

Original: if X c ∈ [0 . 75 , 1 . 0]
Revised: if X c ∈ [0 . 75 , 1 . 0]

Original: y
Revised: y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2 = (cid:26) y true
Revised: 2 = (cid:26) y true

Original: + c with prob.
Revised: + c with prob.

Original: = 0 .y true with prob.
Revised: = 0 .y true with prob.

Original: = 0 .y
Revised: = 0 .y

Original: E 3 =
Revised: E 3 =

Original: y true
Revised: y true

Original: [Equation]
Revised: [Equation]

Original: conf
Revised: conf

Original: [Equation]
Revised: [Equation]

Original: with prob
Revised: with prob

Original: [Equation]
Revised: [Equation]

Original: 0 with prob
Revised: 0 with prob

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Table 9 compares the performance of these constraints in synthetic Confounded and Causal setups (Section E.2).
Revised: Table 9 compares the performance of these constraints in synthetic Confounded and Causal setups (Section E.2).

Original: We can see that the valid constraints according to the graph significantly outperform the incorrect constraints in both shifts.
Revised: We can see that the valid constraints according to the graph significantly outperform the incorrect constraints in both shifts.

Original: Hence, the information on the specific relationship between Y and A is necessary for obtaining an optimal predictor.
Revised: Hence, the information on the specific relationship between Y and A is necessary for obtaining an optimal predictor.

Original: E.5. Complete results
Revised: E.5. Complete results

Original: We provide complete results here for experiments in Section 4.
Revised: We provide complete results here for experiments in Section 4.

Original: Tables 10, 11 and 12 show results on Causal ( A cause ), Independent ( A ind ) and multi-attribute ( A cause ∪ A ind ) shifts respectively for MNIST.
Revised: Tables 10, 11 and 12 show results on Causal ( A cause ), Independent ( A ind ) and multi-attribute ( A cause ∪ A ind ) shifts respectively for MNIST.

Original: Tables 13, 14 and 15 show results on Causal ( A cause ), Independent ( A ind ) and multi-attribute
Revised: Tables 13, 14 and 15 show results on Causal ( A cause ), Independent ( A ind ) and multi-attribute

Original: ( A cause ∪ A ind ) shifts respectively for small NORB.
Revised: ( A cause ∪ A ind ) shifts respectively for small NORB.

Original: While we report results using test-domain validation without early stopping in Section 4.1, we present additional results here using early stopping.
Revised: While we report results using test-domain validation without early stopping in Section 4.1, we present additional results here using early stopping.

Original: Overall, early stopping improves accuracy across datasets and shifts for all methods.
Revised: Overall, early stopping improves accuracy across datasets and shifts for all methods.

Original: CACM outperforms all methods using model selection with as well as without early stopping, with the exception of Table 11.
Revised: CACM outperforms all methods using model selection with as well as without early stopping, with the exception of Table 11.

Original: Table 11 shows results for the Independent shift which is a relatively easier task and hence all methods perform similarly.
Revised: Table 11 shows results for the Independent shift which is a relatively easier task and hence all methods perform similarly.

Original: For Independent shift in MNIST (Table 11), CORAL achieves the highest accuracy.
Revised: For Independent shift in MNIST (Table 11), CORAL achieves the highest accuracy.

Original: It is important to note that CORAL uses the same valid independence constraint derived by CACM for Independent shift (Theorem 3.1).
Revised: It is important to note that CORAL uses the same valid independence constraint derived by CACM for Independent shift (Theorem 3.1).

Original: For completeness, we also include results using train-domain validation .
Revised: For completeness, we also include results using train-domain validation .

Original: However, as noted by previous work (Ye et al., 2022), using a validation set based on training domain distribution may not be suitable in the presence of spurious correlations as achieving high accuracy in training domains often leads to low accuracy in sufficiently different, novel test domains.
Revised: However, as noted by previous work (Ye et al., 2022), using a validation set based on training domain distribution may not be suitable in the presence of spurious correlations as achieving high accuracy in training domains often leads to low accuracy in sufficiently different, novel test domains.

Original: F. Anti-Causal Graph
Revised: F. Anti-Causal Graph

Original: Figure 7 shows causal graphs used for specifying multi-attribute distribution shifts in an anti-causal setting.
Revised: Figure 7 shows causal graphs used for specifying multi-attribute distribution shifts in an anti-causal setting.

Original: These graphs are identical to Figure 2, with the exception of change in direction of causal arrow from X c −→ Y to Y −→ X c .
Revised: These graphs are identical to Figure 2, with the exception of change in direction of causal arrow from X c −→ Y to Y −→ X c .

Original: We derive the (conditional) independence constraints for the anti-causal DAG for Independent , Causal , Confounded and
Revised: We derive the (conditional) independence constraints for the anti-causal DAG for Independent , Causal , Confounded and

Original: Selected shifts.
Revised: Selected shifts.

Original: Theorem F.1.
Revised: Theorem F.1.

Original: Given a causal DAG with the structure as shown in Figure 7(a), the correct constraint depends on the relationship of label Y with the nuisance attributes A .
Revised: Given a causal DAG with the structure as shown in Figure 7(a), the correct constraint depends on the relationship of label Y with the nuisance attributes A .

Original: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =
Revised: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =

Original: A cause ∪ A conf ∪ A sel ).
Revised: A cause ∪ A conf ∪ A sel ).

Original: Then, the (conditional) independence constraints that X c should satisfy are,
Revised: Then, the (conditional) independence constraints that X c should satisfy are,

Original: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.
Revised: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.

Original: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.
Revised: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.

Original: Confounded: X c ⊥⊥ A conf | Y ; X c ⊥⊥ E ; X c ⊥⊥ A conf | Y, E 4.
Revised: Confounded: X c ⊥⊥ A conf | Y ; X c ⊥⊥ E ; X c ⊥⊥ A conf | Y, E 4.

Original: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E
Revised: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E

Original: Proof.
Revised: Proof.

Original: The proof follows from d-separation using the same logic as earlier proof in Section C.2.
Revised: The proof follows from d-separation using the same logic as earlier proof in Section C.2.

Original: We observe that for all attributes A ∈ A ind ( A cause , A conf , A sel ), it is required to condition on Y to obtain valid constraints as Y node appears as a chain or fork in the causal graph but never as a collider due to the Y −→ X c causal arrow.
Revised: We observe that for all attributes A ∈ A ind ( A cause , A conf , A sel ), it is required to condition on Y to obtain valid constraints as Y node appears as a chain or fork in the causal graph but never as a collider due to the Y −→ X c causal arrow.

Original: Corollary F.1.1.
Revised: Corollary F.1.1.

Original: All the above derived constraints are valid for Graph 7(a).
Revised: All the above derived constraints are valid for Graph 7(a).

Original: However, in the presence of a correlation between E and Obj (Graph 7(b)), only the constraints conditioned on E hold true.
Revised: However, in the presence of a correlation between E and Obj (Graph 7(b)), only the constraints conditioned on E hold true.

Original: [Figure]
Revised: [Figure]

Original: G. Broader impact and ethical considerations
Revised: G. Broader impact and ethical considerations

Original: Our work on modeling the data-generating process for improved out-of-distribution generalization is an important advance in building robust predictors for practical settings.
Revised: Our work on modeling the data-generating process for improved out-of-distribution generalization is an important advance in building robust predictors for practical settings.

Original: Such prediction algorithms, including methods building on representation learning, are increasingly a key element of decision-support and decision-making systems.
Revised: Such prediction algorithms, including methods building on representation learning, are increasingly a key element of decision-support and decision-making systems.

Original: We expect our approach to creating a robust predictor to be particularly valuable in real world setups where spurious attributes and real-world multi-attribute settings lead to biases in data.
Revised: We expect our approach to creating a robust predictor to be particularly valuable in real world setups where spurious attributes and real-world multi-attribute settings lead to biases in data.

Original: While not the focus of this paper, CACM may be applied to mitigate social biases (e.g., in language and vision datasets) whose structures can be approximated by the graphs in Figure 2.
Revised: While not the focus of this paper, CACM may be applied to mitigate social biases (e.g., in language and vision datasets) whose structures can be approximated by the graphs in Figure 2.

Original: Risks of using methods such as CACM , include excessive reliance or a false sense of confidence.
Revised: Risks of using methods such as CACM , include excessive reliance or a false sense of confidence.

Original: While methods such as CACM ease the process of building robust models, there remain many ways that an application may still fail (e.g., incorrect structural assumptions).
Revised: While methods such as CACM ease the process of building robust models, there remain many ways that an application may still fail (e.g., incorrect structural assumptions).

Original: AI applications must still be designed appropriately with support of all stakeholders and potentially affected parties, tested in a variety of settings, etc.
Revised: AI applications must still be designed appropriately with support of all stakeholders and potentially affected parties, tested in a variety of settings, etc.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]