Original: M ODELING THE D ATA -G ENERATING P ROCESS IS N ECESSARY FOR O UT - OF -D ISTRIBUTION G ENERALIZATION
Revised: M ODELING THE D ATA -G ENERATING P ROCESS IS N ECESSARY FOR O UT - OF -D ISTRIBUTION G ENERALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent empirical studies on domain generalization
Revised: Recent empirical studies on domain generalization

Original: DG) have shown that DG algorithms that perform well on some distribution shifts fail on others
Revised: DG) have shown that DG algorithms that perform well on some distribution shifts fail on others

Original: , and no state of-the-art DG algorithm performs consistently well on all shifts
Revised: , and no state of-the-art DG algorithm performs consistently well on all shifts

Original: Moreover, real- world data often has multiple distribution shifts over different attributes; hence we introduce multi -attribute distribution shift datasets and find that the accuracy of existing DG algorithms falls even further.
Revised: Moreover, real- world data often has multiple distribution shifts over different attributes; hence we introduce multi -attribute distribution shift datasets and find that the accuracy of existing DG algorithms falls even further.

Original: To explain these results, we provide a formal characterization of generalization under multi-attribute shifts using a canonical causal graph.
Revised: To explain these results, we provide a formal characterization of generalization under multi-attribute shifts using a canonical causal graph.

Original: Based on the relationship between spurious attributes and the classification label, we obtain realizations of the canonical causal graph that characterize common distribution shifts and show that each shift entails different independence constraints over observed variables.
Revised: Based on the relationship between spurious attributes and the classification label, we obtain realizations of the canonical causal graph that characterize common distribution shifts and show that each shift entails different independence constraints over observed variables.

Original: As a result, we prove that any algorithm based on a single, fixed constraint cannot work well across all shifts, providing theoretical evidence for mixed empirical results on DG algorithms.
Revised: As a result, we prove that any algorithm based on a single, fixed constraint cannot work well across all shifts, providing theoretical evidence for mixed empirical results on DG algorithms.

Original: Based on this insight, we develop Causally Adaptive Constraint Minimization (CACM) , an algorithm that uses knowledge about the data-generating process to adaptively identify and apply the correct independence constraints for regularization.
Revised: Based on this insight, we develop Causally Adaptive Constraint Minimization (CACM) , an algorithm that uses knowledge about the data-generating process to adaptively identify and apply the correct independence constraints for regularization.

Original: Results on fully synthetic,
Revised: Results on fully synthetic,

Original: MNIST, small NORB and Waterbirds datasets, covering binary and multi-valued attributes and labels, show that adaptive dataset-dependent constraints lead to the highest accuracy on unseen domains whereas incorrect constraints fail to do so.
Revised: MNIST, small NORB and Waterbirds datasets, covering binary and multi-valued attributes and labels, show that adaptive dataset-dependent constraints lead to the highest accuracy on unseen domains whereas incorrect constraints fail to do so.

Original: Our results demonstrate the importance of modeling the causal relationships inherent in the data-generating process.
Revised: Our results demonstrate the importance of modeling the causal relationships inherent in the data-generating process.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: To perform reliably in real world settings, machine learning models must be robust to distribution shifts – where the training distribution differs from the test distribution.
Revised: To perform reliably in real world settings, machine learning models must be robust to distribution shifts – where the training distribution differs from the test distribution.

Original: Given data from multiple domains that share a common optimal predictor, the domain generalization (DG) task (Wang et al., 2021; Zhou et al., 2021) encapsulates this challenge by evaluating accuracy on an unseen domain.
Revised: Given data from multiple domains that share a common optimal predictor, the domain generalization (DG) task (Wang et al., 2021; Zhou et al., 2021) encapsulates this challenge by evaluating accuracy on an unseen domain.

Original: Recent empirical studies of DG algorithms (Wiles et al., 2022; Ye et al., 2022) have characterized different kinds of distribution shifts across domains.
Revised: Recent empirical studies of DG algorithms (Wiles et al., 2022; Ye et al., 2022) have characterized different kinds of distribution shifts across domains.

Original: Using MNIST as an example, a diversity shift is when domains are created either by adding new values of a spurious attribute like rotation (e.g.,
Revised: Using MNIST as an example, a diversity shift is when domains are created either by adding new values of a spurious attribute like rotation (e.g.,

Original: Rotated-MNIST dataset (Ghifary et al., 2015;
Revised: Rotated-MNIST dataset (Ghifary et al., 2015;

Original: Piratla et al., 2020)) whereas a correlation shift is when domains exhibit different values of correlation between the class label and a spurious attribute like color (e.g., Colored-MNIST (Arjovsky et al., 2019)).
Revised: Piratla et al., 2020)) whereas a correlation shift is when domains exhibit different values of correlation between the class label and a spurious attribute like color (e.g., Colored-MNIST (Arjovsky et al., 2019)).

Original: Partly because advances in representation learning for DG (Ahuja et al., 2021; Krueger et al., 2021; Mahajan et al., 2021; Arjovsky et al., 2019;
Revised: Partly because advances in representation learning for DG (Ahuja et al., 2021; Krueger et al., 2021; Mahajan et al., 2021; Arjovsky et al., 2019;

Original: Li et al., 2018a; Sun & Saenko, 2016) have focused on either one of the shifts, these studies find that performance of state-of-the-art DG algorithms are not consistent across different shifts: algorithms performing well on datasets with one kind of shift fail on datasets with another kind of shift.
Revised: Li et al., 2018a; Sun & Saenko, 2016) have focused on either one of the shifts, these studies find that performance of state-of-the-art DG algorithms are not consistent across different shifts: algorithms performing well on datasets with one kind of shift fail on datasets with another kind of shift.

Original: In this paper, we pose a harder, more realistic question: What if a dataset exhibits two or more kinds of shifts simultaneously ?
Revised: In this paper, we pose a harder, more realistic question: What if a dataset exhibits two or more kinds of shifts simultaneously ?

Original: Such shifts over multiple attributes (where an attribute refers to a spurious high-level variable like rotation) are often observed in real data.
Revised: Such shifts over multiple attributes (where an attribute refers to a spurious high-level variable like rotation) are often observed in real data.

Original: For example, satellite imagery data demonstrates distribution shifts over time as well as the region captured (Koh et al., 2021).
Revised: For example, satellite imagery data demonstrates distribution shifts over time as well as the region captured (Koh et al., 2021).

Original: To study this question, we introduce multi- attribute distribution shift datasets.
Revised: To study this question, we introduce multi- attribute distribution shift datasets.

Original: For instance, in our Col+Rot-MNIST dataset (see Figure 1), both the color and rotation angle of digits can shift 
Revised: For instance, in our Col+Rot-MNIST dataset (see Figure 1), both the color and rotation angle of digits can shift [Figure]

Revised: Y

Original: [Figure]
Revised: [Figure]

Original: C o l o r
Revised: C o l o r

Original: R o t
Revised: R o t

Original: C o l +
Revised: C o l +

Original: R o t
Revised: R o t

Original: [Figure]
Revised: [Figure]

Original: across data distributions.
Revised: across data distributions.

Original: We find that existing DG algorithms that are often targeted for a specific shift fail to generalize in such settings: best accuracy falls from 50-62% for individual shift MNIST datasets to < 50% (lower than a random guess) for the multi-attribute shift dataset.
Revised: We find that existing DG algorithms that are often targeted for a specific shift fail to generalize in such settings: best accuracy falls from 50-62% for individual shift MNIST datasets to < 50% (lower than a random guess) for the multi-attribute shift dataset.

Original: To explain such failures, we propose a causal framework for generalization under multi-attribute distribution shifts.
Revised: To explain such failures, we propose a causal framework for generalization under multi-attribute distribution shifts.

Original: We use a canonical causal graph to model commonly observed distribution shifts.
Revised: We use a canonical causal graph to model commonly observed distribution shifts.

Original: Under this graph, we characterize a distribution shift by the type of relationship between spurious attributes and the classification label, leading to different realized causal DAGs.
Revised: Under this graph, we characterize a distribution shift by the type of relationship between spurious attributes and the classification label, leading to different realized causal DAGs.

Original: Using d -separation on the realized DAGs, we show that each shift entails distinct constraints over observed variables and prove that no conditional independence constraint is valid across all shifts.
Revised: Using d -separation on the realized DAGs, we show that each shift entails distinct constraints over observed variables and prove that no conditional independence constraint is valid across all shifts.

Original: As a special case of multi -attribute, when datasets exhibit a single -attribute shift across domains, this result provides an explanation for the inconsistent performance of DG algorithms reported by Wiles et al.
Revised: As a special case of multi -attribute, when datasets exhibit a single -attribute shift across domains, this result provides an explanation for the inconsistent performance of DG algorithms reported by Wiles et al.

Original: ; Ye et al.
Revised: ; Ye et al.

Original: It implies that any algorithm based on a single, fixed independence constraint cannot work well across all shifts: there will be a dataset on which it will fail (Section 3.3).
Revised: It implies that any algorithm based on a single, fixed independence constraint cannot work well across all shifts: there will be a dataset on which it will fail (Section 3.3).

Original: We go on to ask if we can develop an algorithm that generalizes to different kinds of individual shifts as well as simultaneous multi- attribute shifts.
Revised: We go on to ask if we can develop an algorithm that generalizes to different kinds of individual shifts as well as simultaneous multi- attribute shifts.

Original: For the common shifts modeled by the canonical graph, we show that identification of the correct regularization constraints requires knowing only the type of relationship between attributes and the label, not the full graph.
Revised: For the common shifts modeled by the canonical graph, we show that identification of the correct regularization constraints requires knowing only the type of relationship between attributes and the label, not the full graph.

Original: As we discuss in Section 3.1, the type of shift for an attribute is often available or can be inferred for real-world datasets.
Revised: As we discuss in Section 3.1, the type of shift for an attribute is often available or can be inferred for real-world datasets.

Original: Based on this, we propose Causally Adaptive Constraint Minimization (CACM) , an algorithm that leverages knowledge about the data-generating process (DGP) to identify and apply the correct independence constraints for regularization.
Revised: Based on this, we propose Causally Adaptive Constraint Minimization (CACM) , an algorithm that leverages knowledge about the data-generating process (DGP) to identify and apply the correct independence constraints for regularization.

Original: Given a dataset with auxiliary attributes and their relationship with the target label,
Revised: Given a dataset with auxiliary attributes and their relationship with the target label,

Original: CACM constrains the model’s representation to obey the conditional independence constraints satisfied by causal features of the label, generalizing past work on causality-based regularization (Mahajan et al., 2021; Veitch et al., 2021; Makar et al., 2022) to multi-attribute shifts.
Revised: CACM constrains the model’s representation to obey the conditional independence constraints satisfied by causal features of the label, generalizing past work on causality-based regularization (Mahajan et al., 2021; Veitch et al., 2021; Makar et al., 2022) to multi-attribute shifts.

Original: We evaluate CACM on novel multi-attribute shift datasets based on MNIST, small NORB, and
Revised: We evaluate CACM on novel multi-attribute shift datasets based on MNIST, small NORB, and

Original: Waterbirds images.
Revised: Waterbirds images.

Original: Across all datasets, applying the incorrect constraint, often through an existing
Revised: Across all datasets, applying the incorrect constraint, often through an existing

Original: DG algorithm, leads to significantly lower accuracy than the correct constraint.
Revised: DG algorithm, leads to significantly lower accuracy than the correct constraint.

Original: Further, CACM achieves substantially better accuracy than existing algorithms on datasets with multi-attribute shifts as well as individual shifts.
Revised: Further, CACM achieves substantially better accuracy than existing algorithms on datasets with multi-attribute shifts as well as individual shifts.

Original: Our contributions include:
Revised: Our contributions include:

Original: • Theoretical result that an algorithm using a fixed independence constraint cannot yield an optimal classifier on all datasets.
Revised: • Theoretical result that an algorithm using a fixed independence constraint cannot yield an optimal classifier on all datasets.

Original: • An algorithm, Causally Adaptive Constraint Minimization (CACM) , to adaptively derive the correct regularization constraint(s) based on the causal graph that outperforms existing DG algorithms.
Revised: • An algorithm, Causally Adaptive Constraint Minimization (CACM) , to adaptively derive the correct regularization constraint(s) based on the causal graph that outperforms existing DG algorithms.

Original: • Multi-attribute shifts-based benchmarks for domain generalization where existing algorithms fail.
Revised: • Multi-attribute shifts-based benchmarks for domain generalization where existing algorithms fail.

Original:  G ENERALIZATION UNDER M ULTI -A TTRIBUTE S HIFTS
Revised:  G ENERALIZATION UNDER M ULTI -A TTRIBUTE S HIFTS

Original: We consider the supervised learning setup from (Wiles et al., 2022) where each row of train data ( x i , a i , y i ) ni
Revised: We consider the supervised learning setup from (Wiles et al., 2022) where each row of train data ( x i , a i , y i ) ni

Original: =1 contains input features x
Revised: =1 contains input features x

Original: i (e.g., X-ray pixels), a set of nuisance or spurious attributes a i (e.g., vertical shift, hospital) and class label
Revised: i (e.g., X-ray pixels), a set of nuisance or spurious attributes a i (e.g., vertical shift, hospital) and class label

Original: y
Revised: y

Original: i (e.g., disease diagnosis).
Revised: i (e.g., disease diagnosis).

Original: The attributes represent variables that are often recorded or implicit in data collection procedures.
Revised: The attributes represent variables that are often recorded or implicit in data collection procedures.

Original: Some attributes represent a property of the input (e.g., vertical shift) while others represent the domain from which the input was collected (e.g., hospital).
Revised: Some attributes represent a property of the input (e.g., vertical shift) while others represent the domain from which the input was collected (e.g., hospital).

Original: The attributes affect the observed input features x i but do not cause the target label, and hence are spurious attributes.
Revised: The attributes affect the observed input features x i but do not cause the target label, and hence are spurious attributes.

Original: The final classifier g ( x ) is expected to use only the input features.
Revised: The final classifier g ( x ) is expected to use only the input features.

Original: However, as new values of attributes are introduced or as the correlation of attributes with the label changes, we obtain different conditional distributions P ( Y | X ) .
Revised: However, as new values of attributes are introduced or as the correlation of attributes with the label changes, we obtain different conditional distributions P ( Y | X ) .

Original: Given a set of data distributions
Revised: Given a set of data distributions

Original: P , we assume that the train data is sampled from distributions,
Revised: P , we assume that the train data is sampled from distributions,

Original: P E tr = { P E 1 , P E 2 , · · · } ⊂ P while the test data is assumed to be sampled from a single unseen distribution, P E te = { P Ete } ⊂ P .
Revised: P E tr = { P E 1 , P E 2 , · · · } ⊂ P while the test data is assumed to be sampled from a single unseen distribution, P E te = { P Ete } ⊂ P .

Original: Attributes and class labels are assumed to be discrete.
Revised: Attributes and class labels are assumed to be discrete.

Original: 2. R ISK INVARIANT PREDICTOR FOR GENERALIZATION UNDER SHIFTS
Revised: 2. R ISK INVARIANT PREDICTOR FOR GENERALIZATION UNDER SHIFTS

Original: The goal is to learn a classifier g ( x ) using train domains such that it generalizes and achieves a similar, small risk on test data from unseen P Ete as it achieves on the train data.
Revised: The goal is to learn a classifier g ( x ) using train domains such that it generalizes and achieves a similar, small risk on test data from unseen P Ete as it achieves on the train data.

Original: Formally, given a set of distributions P , we define a risk-invariant predictor (Makar et al., 2022) as,
Revised: Formally, given a set of distributions P , we define a risk-invariant predictor (Makar et al., 2022) as,

Original: Definition 2.1.
Revised: Definition 2.1.

Original: Optimal Risk Invariant Predictor for P (from (Makar et al., 2022))
Revised: Optimal Risk Invariant Predictor for P (from (Makar et al., 2022))

Original: Define the risk of predictor g on distribution P ∈ P as R P ( g ) =
Revised: Define the risk of predictor g on distribution P ∈ P as R P ( g ) =

Original: E x ,y ∼ P ℓ ( g ( x ) , y ) where ℓ is cross-entropy or another classification loss.
Revised: E x ,y ∼ P ℓ ( g ( x ) , y ) where ℓ is cross-entropy or another classification loss.

Original: Then, the set of risk-invariant predictors obtain the same risk across all distributions P ∈ P , and set of the optimal risk-invariant predictors is defined as the risk-invariant predictors that obtain minimum risk on all distributions.
Revised: Then, the set of risk-invariant predictors obtain the same risk across all distributions P ∈ P , and set of the optimal risk-invariant predictors is defined as the risk-invariant predictors that obtain minimum risk on all distributions.

Original: [Equation]
Revised: [Equation]

Original: An intuitive way to obtain a risk-invariant predictor is to consider only the parts of the input features ( X ) that cause the label Y and ignore any variation due to the spurious attributes.
Revised: An intuitive way to obtain a risk-invariant predictor is to consider only the parts of the input features ( X ) that cause the label Y and ignore any variation due to the spurious attributes.

Original: Let such latent, unobserved causal features be X c .
Revised: Let such latent, unobserved causal features be X c .

Original: Due to independence and stability of causal mechanisms (Peters et al., 2017), we can assume that P ( Y | X c ) remains invariant across different distributions.
Revised: Due to independence and stability of causal mechanisms (Peters et al., 2017), we can assume that P ( Y | X c ) remains invariant across different distributions.

Original: Using the notion of risk invariance, we can now define the multi-attribute generalization problem as,
Revised: Using the notion of risk invariance, we can now define the multi-attribute generalization problem as,

Original: Definition 2.2.
Revised: Definition 2.2.

Original: Generalization under Multi-attribute shifts.
Revised: Generalization under Multi-attribute shifts.

Original: Given a target label Y , input features
Revised: Given a target label Y , input features

Original: X , attributes A , and latent causal features X c , consider a set of distributions P such that P ( Y | X c ) remains invariant while P ( A | Y ) changes across individual distributions.
Revised: X , attributes A , and latent causal features X c , consider a set of distributions P such that P ( Y | X c ) remains invariant while P ( A | Y ) changes across individual distributions.

Original: Using a training dataset ( x i , a i , y i ) ni
Revised: Using a training dataset ( x i , a i , y i ) ni

Original: =1 sampled from a subset of distributions P E tr ⊂ P , the generalization goal is to learn an optimal risk-invariant predictor over P .
Revised: =1 sampled from a subset of distributions P E tr ⊂ P , the generalization goal is to learn an optimal risk-invariant predictor over P .

Original: Special case of single-attribute shift.
Revised: Special case of single-attribute shift.

Original: When | A | = 1 , we obtain the single-attribute shift problem that is widely studied (Wiles et al., 2022; Ye et al., 2022; Gulrajani & Lopez-Paz, 2021).
Revised: When | A | = 1 , we obtain the single-attribute shift problem that is widely studied (Wiles et al., 2022; Ye et al., 2022; Gulrajani & Lopez-Paz, 2021).

Original: 2. A GENERAL PRINCIPLE FOR NECESSARY CONDITIONAL INDEPENDENCE CONSTRAINTS
Revised: 2. A GENERAL PRINCIPLE FOR NECESSARY CONDITIONAL INDEPENDENCE CONSTRAINTS

Original: match matches the statistic across E .
Revised: match matches the statistic across E .

Original: h is domain classifier (loss ℓ d ) using shared representation ϕ .
Revised: h is domain classifier (loss ℓ d ) using shared representation ϕ .

Original: [Table]
Revised: [Table]

Original: In practice, the causal features X c are unobserved and a key challenge is to learn X c using the observed ( X , Y, A ) .
Revised: In practice, the causal features X c are unobserved and a key challenge is to learn X c using the observed ( X , Y, A ) .

Original: We focus on representation learning-based (Wang et al., 2021) DG algorithms, typically characterized by a regularization constraint that is added to a standard
Revised: We focus on representation learning-based (Wang et al., 2021) DG algorithms, typically characterized by a regularization constraint that is added to a standard

Original: ERM loss such as cross-entropy.
Revised: ERM loss such as cross-entropy.

Original: Table 1 shows three independence constraints that form the basis of many popular DG algorithms, assuming environment/domain E as the attribute and ℓ as the main classifier loss.
Revised: Table 1 shows three independence constraints that form the basis of many popular DG algorithms, assuming environment/domain E as the attribute and ℓ as the main classifier loss.

Original: We now provide a general principle for deciding which constraints to choose for learning a risk-invariant predictor for a dataset.
Revised: We now provide a general principle for deciding which constraints to choose for learning a risk-invariant predictor for a dataset.

Original: We utilize a strategy from past work (Mahajan et al., 2021; Veitch et al., 2021) to use graph structure of the underlying data-generating process (DGP).
Revised: We utilize a strategy from past work (Mahajan et al., 2021; Veitch et al., 2021) to use graph structure of the underlying data-generating process (DGP).

Original: We assume that the predictor can be represented as g ( x )
Revised: We assume that the predictor can be represented as g ( x )

Original: =
Revised: =

Original: g 1 ( ϕ ( x )) where ϕ is the representation.
Revised: g 1 ( ϕ ( x )) where ϕ is the representation.

Original: To learn a risk-invariant ϕ , we identify the conditional independence constraints satisfied by causal features X c in the causal graph and enforce that learnt representation ϕ should follow the same constraints.
Revised: To learn a risk-invariant ϕ , we identify the conditional independence constraints satisfied by causal features X c in the causal graph and enforce that learnt representation ϕ should follow the same constraints.

Original: If ϕ satisfies the constraints, then any function g 1 ( ϕ ) will also satisfy them.
Revised: If ϕ satisfies the constraints, then any function g 1 ( ϕ ) will also satisfy them.

Original: Below we show that the constraints are necessary under simple assumptions on the causal DAG representing the DGP for a dataset.
Revised: Below we show that the constraints are necessary under simple assumptions on the causal DAG representing the DGP for a dataset.

Original: All proofs are in Suppl.
Revised: All proofs are in Suppl.

Original: B.
Revised: B.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Consider a causal DAG G over ⟨ X c , X , A , Y ⟩ and a corresponding generated dataset ( x i , a i , y i ) ni
Revised: Consider a causal DAG G over ⟨ X c , X , A , Y ⟩ and a corresponding generated dataset ( x i , a i , y i ) ni

Original: =1 , where X c is unobserved.
Revised: =1 , where X c is unobserved.

Original: Assume that graph G has the following property: X c is defined as the set of all parents of Y ( X c → Y ); and X c , A together cause X ( X c → X , and
Revised: Assume that graph G has the following property: X c is defined as the set of all parents of Y ( X c → Y ); and X c , A together cause X ( X c → X , and

Original: A → X ).
Revised: A → X ).

Original: The graph may have any other edges (see, e.g., DAG in Figure 1(b)).
Revised: The graph may have any other edges (see, e.g., DAG in Figure 1(b)).

Original: Let P G be the set of distributions consistent with graph G , obtained by changing P ( A | Y )  .
Revised: Let P G be the set of distributions consistent with graph G , obtained by changing P ( A | Y ) but not P ( Y | X c ) .

Original: Then the conditional independence constraints satisfied by X c are necessary for a (cross-entropy) risk-invariant predictor over P G .
Revised: Then the conditional independence constraints satisfied by X c are necessary for a (cross-entropy) riskinvariant predictor over P G .

Original: That is, if a predictor for Y does not satisfy any of these constraints, then there exists
Revised: That is, if a predictor for Y does not satisfy any of these constraints, then there exists

Original: a data distribution P ′ ∈
Revised: a data distribution P ′ ∈

Original: P G such that predictor’s risk will be higher than its risk in other distributions.
Revised: P G such that predictor’s risk will be higher than its risk in other distributions.

Original: Thus, given a causal DAG, using d -separation on X c and observed variables, we can derive the correct regularization constraints to be applied on ϕ .
Revised: Thus, given a causal DAG, using d -separation on X c and observed variables, we can derive the correct regularization constraints to be applied on ϕ .

Original: This yields a general principle to learn a risk-invariant predictor.
Revised: This yields a general principle to learn a risk-invariant predictor.

Original: We use it to theoretically explain the inconsistent results of existing DG algorithms (Sec. 3.3) and to propose an Out-of-Distribution generalization algorithm CACM (Sec. 4).
Revised: We use it to theoretically explain the inconsistent results of existing DG algorithms (Sec. 3.3) and to propose an Out-of-Distribution generalization algorithm CACM (Sec. 4).

Revised: Note

Revised: that constraints from CACM are necessary but not sufficient as X c is not identifiable.

Revised: NEW (LyJp)

Original:  S TUDYING DISTRIBUTION SHIFTS THROUGH A CANONICAL CAUSAL GRAPH 3. C ANONICAL CAUSAL GRAPH FOR COMMON DISTRIBUTION SHIFTS
Revised:  S TUDYING DISTRIBUTION SHIFTS THROUGH A CANONICAL CAUSAL GRAPH 3. C ANONICAL CAUSAL GRAPH FOR COMMON DISTRIBUTION SHIFTS

Original: We consider a canonical causal graph (Figure 2) to specify the common data-generating processes that can lead to a multi-attribute shift dataset.
Revised: We consider a canonical causal graph (Figure 2) to specify the common data-generating processes that can lead to a multi-attribute shift dataset.

Original: Shaded nodes represent observed variables X , Y ; and the sets of attributes A ind , A ind , and E such that A ind ∪ A ind ∪ { E }
Revised: Shaded nodes represent observed variables X , Y ; and the sets of attributes A ind , A ind , and E such that A ind ∪ A ind ∪ { E }

Original: =
Revised: =

Original: A .
Revised: A .

Original: A ind represents the attributes correlated with label, A ind the attributes that are independent of label, while E is a special attribute for the domain/environment from which a data point was collected.
Revised: A ind represents the attributes correlated with label, A ind the attributes that are independent of label, while E is a special attribute for the domain/environment from which a data point was collected.

Original: Not all attributes need to be observed.
Revised: Not all attributes need to be observed.

Original: For example, in some cases, only E and a subset of A ind , A ind may be observed.
Revised: For example, in some cases, only E and a subset of A ind , A ind may be observed.

Original: In other cases, only A ind and A ind may be observed while E is not available.
Revised: In other cases, only A ind and A ind may be observed while E is not available.

Original: Regardless, we assume that all attributes, along with the causal features X c , determine the observed features X .
Revised: Regardless, we assume that all attributes, along with the causal features X c , determine the observed features X .

Original: And the features X c are the only features that cause Y .
Revised: And the features X c are the only features that cause Y .

Original: In the simplest case, we assume no label shift across environments i.e. marginal distribution of Y is constant across train domains and test, P Etr ( y ) =
Revised: In the simplest case, we assume no label shift across environments i.e. marginal distribution of Y is constant across train domains and test, P Etr ( y ) =

Original: P Ete ( y ) (see Figure 2a).
Revised: P Ete ( y ) (see Figure 2a).

Original: More generally, different domains may have different distribution of causal features (in the X-ray example, more women visit one hospital compared to the other), as shown bythe dotted arrow between E and X c (Figure 2b).
Revised: More generally, different domains may have different distribution of causal features (in the X-ray example, more women visit one hospital) as shown by E < – > X c (Figure 2b).

Original: Under the canonical graph, we characterize different kinds of shifts based on the relationship between spurious attributes A and the classification label Y .
Revised: Under the canonical graph, we characterize different kinds of shifts based on the relationship between spurious attributes A and the classification label Y .

Original: Specifically, A ind is independent of the class label and a change in P ( A ind ) leads to an Independent distribution shift.
Revised: Specifically, A ind is independent of the class label and a change in P ( A ind ) leads to an Independent distribution shift.

Original: For A ind , there are
Revised: For A ind , there are

Original: [Figure]
Revised: [Figure]

Original: Different mechanisms for Y - A ind relationship that lead to Causal , Confounded and Selected shifts.

Original: three mechanisms which can introduce the dashed-line correlation between A ind and Y (Figure 2c) – direct-causal ( Y causing A ind ), confounding between Y and A ind due to a common cause, or selection during the data-generating process.
Revised: three mechanisms which can introduce the dashed-line correlation between A ind and Y (Figure 2c) – direct-causal ( Y causing A ind ), confounding between Y and A ind due to a common cause, or selection during the data-generating process.

Original: Overall, we define four kinds of shifts based on the causal graph: Independent , Causal , Confounded , and Selected  .
Revised: Overall, we define four kinds of shifts based on the causal graph: Independent , Causal , Confounded , and Selected 1 .

Original: While the canonical graph in
Revised: While the canonical graph in

Original: Figure 2a is general, resolving each dashed edge into a specific type of shift (causal mechanism) leads to a realized causal DAG for a particular dataset.
Revised: Figure 2a is general, resolving each dashed edge into a specific type of shift (causal mechanism) leads to a realized causal DAG for a particular dataset.

Original: As we shall see, knowledge of these shift types is sufficient to determine the correct independence constraints between observed variables.
Revised: As we shall see, knowledge of these shift types is sufficient to determine the correct independence constraints between observed variables.

Original: Our canonical multi-attribute graph generalizes the DG graph from Mahajan et al.
Revised: Our canonical multi-attribute graph generalizes the DG graph from Mahajan et al.

Original: (2021) that considered an Independent domain/environment as the only attribute.
Revised: (2021) that considered an Independent domain/environment as the only attribute.

Original: Under the special case of a single attribute ( | A | = 1 ), the canonical graph helps interpret the validity of popular DG methods for a dataset by considering the type of the attribute-label relationship in the data-generating process.
Revised: Under the special case of a single attribute ( | A | = 1 ), the canonical graph helps interpret the validity of popular DG methods for a dataset by considering the type of the attribute-label relationship in the data-generating process.

Original: For example, let us consider two common constraints in prior work on independence between ϕ and a spurious attribute: unconditional ( ϕ ( x ) ⊥⊥ A )
Revised: For example, let us consider two common constraints in prior work on independence between ϕ and a spurious attribute: unconditional ( ϕ ( x ) ⊥⊥ A )

Original: (Veitch et al., 2021;
Revised: (Veitch et al., 2021;

Original: Albuquerque et al., 2020; Ganin et al., 2016) or conditional on the label ( ϕ ( x ) ⊥⊥
Revised: Albuquerque et al., 2020; Ganin et al., 2016) or conditional on the label ( ϕ ( x ) ⊥⊥

Original: A | Y )
Revised: A | Y )

Original: (Ghifary et al., 2016;
Revised: (Ghifary et al., 2016;

Original: Hu et al., 2019; Li et al., 2018c;d).
Revised: Hu et al., 2019; Li et al., 2018c;d).

Original: Under the canonical graph in Figure 2a, the unconditional constraint is true when
Revised: Under the canonical graph in Figure 2a, the unconditional constraint is true when

Original: A ⊥⊥ Y ( A ∈ A ind ) but not always for A ind (true only under Confounded shift).
Revised: A ⊥⊥ Y ( A ∈ A ind ) but not always for A ind (true only under Confounded shift).

Original: If the relationship is Causal or Selected , then the conditional constraint is correct.
Revised: If the relationship is Causal or Selected , then the conditional constraint is correct.

Original: Critically, as Veitch et al.
Revised: Critically, as Veitch et al.

Original: (2021) show for a single-attribute graph, the conditional constraint is not always better; it is an incorrect constraint (not satisfied by X c ) under Confounded setting.
Revised: (2021) show for a single-attribute graph, the conditional constraint is not always better; it is an incorrect constraint (not satisfied by X c ) under Confounded setting.

Original: Further, under the canonical graph [with
Revised: Further, under the canonical graph [with

Original: E - X c edge] from Figure 2b, none of these constraints are valid due to a correlation path between X c and E .
Revised: E - X c edge] from Figure 2b, none of these constraints are valid due to a correlation path between X c and E .

Original: This shows the importance of considering the generating process for a dataset.
Revised: This shows the importance of considering the generating process for a dataset.

Revised: Inferring

Original: [Table]
Revised: [Table]

Original:  type.
Revised: relationship type.

Original: Whether an attribute belongs to A ind or A ind can be learned from data (since A ind ⊥⊥ Y ).
Revised: Whether an attribute belongs to A ind or A ind can be learned from data (since A ind ⊥⊥ Y ).

Original: Under some special conditions with the graph in Figure 2a— assuming all attributes are observed and all attributes in A ind are of the same type—we can also identify the type of A ind shift: Y ⊥⊥ E | A ind implies Selected ; if not, then X ⊥⊥ E | A ind , A ind , Y implies Causal , otherwise it is Confounded .
Revised: Under some special conditions with the graph in Figure 2a— assuming all attributes are observed and all attributes in A ind are of the same type—we can also identify the type of A ind shift: Y ⊥⊥ E | A ind implies Selected ; if not, then X ⊥⊥ E | A ind , A ind , Y implies Causal , otherwise it is Confounded .

Original: In the general case of Figure 2b, however, it is not possible to differentiate between A cause , A conf and A sel using observed data and needs manual input.
Revised: In the general case of Figure 2b, however, it is not possible to differentiate between A cause , A conf and A sel using observed data and needs manual input.

Original: Fortunately, unlike the full causal graph, the type of relationship between label and an attribute is easier to obtain.
Revised: Fortunately, unlike the full causal graph, the type of relationship between label and an attribute is easier to obtain.

Original: For example, in text toxicity classification, toxicity labels are found to be spuriously correlated with certain demographics ( A ind ) (Dixon et al., 2018;
Revised: For example, in text toxicity classification, toxicity labels are found to be spuriously correlated with certain demographics ( A ind ) (Dixon et al., 2018;

Original: Koh et al., 2021;
Revised: Koh et al., 2021;

Original: Park et al., 2018); while in medical applications where data is collected from small number of hospitals, shifts arise due to different methods of slide staining and image acquisition ( A ind ) (Koh et al., 2021;
Revised: Park et al., 2018); while in medical applications where data is collected from small number of hospitals, shifts arise due to different methods of slide staining and image acquisition ( A ind ) (Koh et al., 2021;

Original: Komura &
Revised: Komura &

Original: Ishikawa, 2018; Tellez et al., 2019).
Revised: Ishikawa, 2018; Tellez et al., 2019).

Original: Suppl.
Revised: Suppl.

Original: A contains additional real-world examples with attributes.
Revised: A contains additional real-world examples with attributes.

Original: 3. I NDEPENDENCE CONSTRAINTS DEPEND ON ATTRIBUTE ↔ LABEL RELATIONSHIP
Revised: 3. I NDEPENDENCE CONSTRAINTS DEPEND ON ATTRIBUTE ↔ LABEL RELATIONSHIP

Original: We list the independence constraints between ⟨ X c , A , Y ⟩ under the canonical graphs from Figure 2, which can be used to derive the correct regularization constraints to be applied on ϕ (Theorem 2.1).
Revised: We list the independence constraints between ⟨ X c , A , Y ⟩ under the canonical graphs from Figure 2, which can be used to derive the correct regularization constraints to be applied on ϕ (Theorem 2.1).

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Given a causal DAG realized by specifying the target-attributes relationship in
Revised: Given a causal DAG realized by specifying the target-attributes relationship in

Original: Figure 2a, the correct constraint depends on the relationship of label Y with the attributes A .
Revised: Figure 2a, the correct constraint depends on the relationship of label Y with the attributes A .

Original: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =
Revised: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =

Original: A cause ∪ A conf ∪ A sel ).
Revised: A cause ∪ A conf ∪ A sel ).

Original: Then, the (conditional) independence constraints X c should satisfy are,
Revised: Then, the (conditional) independence constraints X c should satisfy are,

Original: [Equation] c ⊥⊥ A ind [Equation] E 
Revised: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.

Original: [Equation]

Original: E [Equation] ⊥⊥ A ind | Y, E 2. Causal: X c ⊥⊥ A caus | Y ; X c ⊥⊥ E ; X c ⊥⊥ A caus | Y, E 3.
Revised: Causal: X c ⊥⊥ A caus | Y ; X c ⊥⊥ E ; X c ⊥⊥ A caus | Y, E 3.

Original: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 4.
Revised: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 4.

Original: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E
Revised: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: All the above derived constraints are valid for Graph 2a.
Revised: All the above derived constraints are valid for Graph 2a.

Original: However, in the presence of a correlation between E and X c (Graph 2b), only the constraints conditioned on E hold true.
Revised: However, in the presence of a correlation between E and X c (Graph 2b), only the constraints conditioned on E hold true.

Original: Corollary 3.1 implies that if we are not sure about E - X c correlation, E - conditioned constraints should be used.
Revised: Corollary 3.1 implies that if we are not sure about E - X c correlation, E - conditioned constraints should be used.

Original: By considering independence constraints over attributes that may represent any observed variable, our graph-based characterization unites the single-domain (group-wise) (Sagawa et al., 2020) and multi-domain generalization tasks.
Revised: By considering independence constraints over attributes that may represent any observed variable, our graph-based characterization unites the single-domain (group-wise) (Sagawa et al., 2020) and multi-domain generalization tasks.

Original: Whether attributes represent auxiliary attributes, group indicators, or data sources, Proposition 3.1 provides the correct regularization constraint.
Revised: Whether attributes represent auxiliary attributes, group indicators, or data sources, Proposition 3.1 provides the correct regularization constraint.

Original: 3. A FIXED CONDITIONAL INDEPENDENCE CONSTRAINT CANNOT WORK FOR ALL SHIFTS
Revised: 3. A FIXED CONDITIONAL INDEPENDENCE CONSTRAINT CANNOT WORK FOR ALL SHIFTS

Original: Combining with Theorem 2.1, Proposition 3.1 shows that the necessary constraints for a risk-invariant predictor’s representation ϕ ( X ) are different for different types of attributes.
Revised: Combining with Theorem 2.1, Proposition 3.1 shows that the necessary constraints for a risk-invariant predictor’s representation ϕ ( X ) are different for different types of attributes.

Original: This leads us to our key result: under multi-attribute shifts, a single (conditional) independence constraint cannot be valid for all kinds of shifts.
Revised: This leads us to our key result: under multi-attribute shifts, a single (conditional) independence constraint cannot be valid for all kinds of shifts.

Original: Remarkably, this result is true even for single-attribute shifts: any algorithm with a fixed conditional independence constraint (e.g., as listed in Table 1 (Gretton et al., 2012; Arjovsky et al., 2019; Li et al., 2018b; Sun & Saenko, 2016)) cannot work for all datasets.
Revised: Remarkably, this result is true even for single-attribute shifts: any algorithm with a fixed conditional independence constraint (e.g., as listed in Table 1 (Gretton et al., 2012; Arjovsky et al., 2019; Li et al., 2018b; Sun & Saenko, 2016)) cannot work for all datasets.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Under the canonical causal graph in Figure 2(a,b), there exists no (conditional) independence constraint over ⟨ X c , A , Y ⟩ that is valid for all realized DAGs as the type of multiattribute shifts vary.
Revised: Under the canonical causal graph in Figure 2(a,b), there exists no (conditional) independence constraint over ⟨ X c , A , Y ⟩ that is valid for all realized DAGs as the type of multiattribute shifts vary.

Original: Hence, for any predictor algorithm for Y that uses a single (conditional) independence constraint over its representation ϕ
Revised: Hence, for any predictor algorithm for Y that uses a single (conditional) independence constraint over its representation ϕ

Original: ( X ) , A and Y , there exists a realized DAG G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor for distributions in P G , where P G is the set of distributions obtained by changing P ( A | Y ) .
Revised: ( X ) , A and Y , there exists a realized DAG G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor for distributions in P G , where P G is the set of distributions obtained by changing P ( A | Y ) .

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: Even when | A | = 1 , an algorithm using a single independence constraint over ⟨ ϕ ( X ) , A, Y ⟩ cannot yield a risk-invariant predictor for all kinds of single-attribute shift datasets.
Revised: Even when | A | = 1 , an algorithm using a single independence constraint over ⟨ ϕ ( X ) , A, Y ⟩ cannot yield a risk-invariant predictor for all kinds of single-attribute shift datasets.

Original: Corollary 3.2 adds theoretical evidence for past empirical demonstrations of inconsistent performance of DG algorithms (Wiles et al., 2022;
Revised: Corollary 3.2 adds theoretical evidence for past empirical demonstrations of inconsistent performance of DG algorithms (Wiles et al., 2022;

Original: Ye et al., 2022).To demonstrate its significance, we provide
Revised: Ye et al., 2022).To demonstrate its significance, we provide

Original: OoD generalization results on a simple “slab” setup (Shah et al., 2020) with three datasets ( Causal ,
Revised: OoD generalization results on a simple “slab” setup (Shah et al., 2020) with three datasets ( Causal ,

Original: Confounded , and Selected shifts) in Suppl.
Revised: Confounded , and Selected shifts) in Suppl.

Original: E.2.
Revised: E.2.

Original: We evaluate two constraints motivated by DG literature Mahajan et al.
Revised: We evaluate two constraints motivated by DG literature Mahajan et al.

Original: (2021): unconditional X c ⊥⊥ A | E , and conditional on label
Revised: (2021): unconditional X c ⊥⊥ A | E , and conditional on label

Original: X c ⊥⊥ A | Y, E .
Revised: X c ⊥⊥ A | Y, E .

Original: As predicted by Corollary 3.2, neither constraint obtains best accuracy on all three datasets (Table 6).
Revised: As predicted by Corollary 3.2, neither constraint obtains best accuracy on all three datasets (Table 6).

Original:  C AUSALLY A DAPTIVE C ONSTRAINT M INIMIZATION ( CACM )
Revised:  C AUSALLY A DAPTIVE C ONSTRAINT M INIMIZATION ( CACM )

Original: Motivated by Sec. 3, we present CACM , an algorithm that adaptively chooses regularizing constraints for multi-attribute shift datasets (full algorithm for any general DAG in Suppl.
Revised: Motivated by Sec. 3, we present CACM , an algorithm that adaptively chooses regularizing constraints for multi-attribute shift datasets (full algorithm for any general DAG in Suppl.

Original: C).
Revised: C).

Original: It has two phases.
Revised: It has two phases.

Original: Phase I. Derive correct independence constraints.
Revised: Phase I. Derive correct independence constraints.

Original: If a dataset’s DGP satisfies the canonical graph,
Revised: If a dataset’s DGP satisfies the canonical graph,

Original: CACM requires a user to specify the relationship type for each attribute and uses the constraints from
Revised: CACM requires a user to specify the relationship type for each attribute and uses the constraints from

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: For other datasets, CACM requires a causal graph describing the dataset’s DGP and uses the following steps to derive the independence constraints.
Revised: For other datasets, CACM requires a causal graph describing the dataset’s DGP and uses the following steps to derive the independence constraints.

Original: Let V be the set of observed variables in the graph except Y , and C be the list of constraints.
Revised: Let V be the set of observed variables in the graph except Y , and C be the list of constraints.

Original: For each observed variable V ∈ V , check whether ( X c , V ) are d -separated.
Revised: For each observed variable V ∈ V , check whether ( X c , V ) are d -separated.

Original: Add X c ⊥⊥ V to C . 2.
Revised: Add X c ⊥⊥ V to C . 2.

Original: If not, check if ( X c , V ) are d -separated conditioned on any subset Z of the remaining observed variables in Z = { Y } ∪ V \ { V } .
Revised: If not, check if ( X c , V ) are d -separated conditioned on any subset Z of the remaining observed variables in Z = { Y } ∪ V \ { V } .

Original: For each subset Z with d -separation, add X c ⊥⊥ V | Z to C .
Revised: For each subset Z with d -separation, add X c ⊥⊥ V | Z to C .

Original: Phase II.
Revised: Phase II.

Original: Apply regularization penalty using derived constraints.
Revised: Apply regularization penalty using derived constraints.

Original: In Phase II CACM applies those constraints as a regularizer 
Revised: In Phase II, CACM applies those constraints as a regularizer to the standard ERM loss, g 1 , ϕ = arg min g 1 ,ϕ ;

Original: to the standard
Revised: [Equation]

Original: ERM loss, g 1 , ϕ = arg
Revised: [Equation]

Original: min g ϕ 
Revised: ( ϕ ( x ))

Original: [Equation]
Revised: [Equation]

Revised: y )

Revised: +

Original: RegPenalty , where ℓ is cross-entropy loss.
Revised: RegPenalty , where ℓ is cross-entropy loss.

Original: The regularizer optimizes for valid constraints over all observed variables V ∈ V .
Revised: The regularizer optimizes for valid constraints over all observed variables V ∈ V .

Original: Below we provide the regularizer term for datasets following the canonical graphs from Figure 2 ( V = A ).
Revised: Below we provide the regularizer term for datasets following the canonical graphs from Figure 2 ( V = A ).

Original: We choose Maximum Mean Discrepancy ( MMD ) (Gretton et al., 2012) to apply our penalty (in principle, any metric for conditional independence would work).
Revised: We choose Maximum Mean Discrepancy ( MMD ) (Gretton et al., 2012) to apply our penalty (in principle, any metric for conditional independence would work).

Original: Since A includes multiple attributes, the regularizer penalty depends on the type of distribution shift for each attribute.
Revised: Since A includes multiple attributes, the regularizer penalty depends on the type of distribution shift for each attribute.

Original: For instance, for A ∈ A ind ( Independent ), to enforce ϕ ( x ) ⊥⊥ A , we aim to minimize the distributional discrepancy between P ( ϕ ( x ) |
Revised: For instance, for A ∈ A ind ( Independent ), to enforce ϕ ( x ) ⊥⊥ A , we aim to minimize the distributional discrepancy between P ( ϕ ( x ) |

Original: A = a i ) and P ( ϕ ( x ) | A = a j ) , for all i, j values of A .
Revised: A = a i ) and P ( ϕ ( x ) | A = a j ) , for all i, j values of A .

Original: However, since the same constraint is applicable on E , it is statistically efficient to apply the constraint on E (if available) as there may be multiple closely related values of A in a domain (e.g., slide stains collected from one hospital may be spread over similar colors, but not exactly the same).
Revised: However, since the same constraint is applicable on E , it is statistically efficient to apply the constraint on E (if available) as there may be multiple closely related values of A in a domain (e.g., slide stains collected from one hospital may be spread over similar colors, but not exactly the same).

Original: Hence, we apply the constraint on distributions P ( ϕ ( x ) | E
Revised: Hence, we apply the constraint on distributions P ( ϕ ( x ) | E

Original: = E i ) and P ( ϕ ( x ) | E = E j ) if E is observed (and A may/may not be unobserved), otherwise we apply the constraint over A .
Revised: = E i ) and P ( ϕ ( x ) | E = E j ) if E is observed (and A may/may not be unobserved), otherwise we apply the constraint over A .

Original: [Equation]
Revised: [Equation]

Original: For A ∈ A cause ( Causal ), following Proposition 3.1, we consider distributions P ( ϕ ( x ) |
Revised: For A ∈ A cause ( Causal ), following Proposition 3.1, we consider distributions P ( ϕ ( x ) |

Original: A = a
Revised: A = a

Original: i , Y = y ) and P ( ϕ ( x ) | A = a j , Y = y ) .
Revised: i , Y = y ) and P ( ϕ ( x ) | A = a j , Y = y ) .

Original: We additionally condition on E as there may be a correlation between E and X c (Figure 2b), which renders other constraints incorrect (Corollary 3.1).
Revised: We additionally condition on E as there may be a correlation between E and X c (Figure 2b), which renders other constraints incorrect (Corollary 3.1).

Original: We similarly obtain regularization terms for Confounded and Selected (Suppl. C).
Revised: We similarly obtain regularization terms for Confounded and Selected (Suppl. C).

Original: RegPenalty A cause = (cid:88)
Revised: RegPenalty A cause = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The final RegPenalty is a sum of penalties over all attributes, RegPenalty = (cid:80)
Revised: The final RegPenalty is a sum of penalties over all attributes, RegPenalty = (cid:80)

Original: A ∈
Revised: A ∈

Original: A λ A Penalty A , where λ A are the hyperparameters.
Revised: A λ A Penalty A , where λ A are the hyperparameters.

Original: Unlike prior work (Makar et al., 2022; Veitch et al., 2021), we do not restrict ourselves to binary-valued attributes and classes.
Revised: Unlike prior work (Makar et al., 2022; Veitch et al., 2021), we do not restrict ourselves to binary-valued attributes and classes.

Original: CACM ’s relationship with existing DG algorithms.
Revised: CACM ’s relationship with existing DG algorithms.

Original: Table 1 shows common constraints used by popular DG algorithms.
Revised: Table 1 shows common constraints used by popular DG algorithms.

Original: CACM ’s strength lies in adaptively selecting constraints based on the causal relationships in the DGP.
Revised: CACM ’s strength lies in adaptively selecting constraints based on the causal relationships in the DGP.

Original: Thus, depending on the dataset, applying CACM for a single-attribute shift may involve applying the same constraint as in MMD or C-MMD algorithms.
Revised: Thus, depending on the dataset, applying CACM for a single-attribute shift may involve applying the same constraint as in MMD or C-MMD algorithms.

Original: For example, in Rotated-MNIST dataset with E =
Revised: For example, in Rotated-MNIST dataset with E =

Original: A ind = rotation , the effective constraint for MMD, DANN,
Revised: A ind = rotation , the effective constraint for MMD, DANN,

Original: CORAL algorithms ( ϕ ⊥⊥ E ) is the same as CACM ’s constraint ϕ ⊥⊥ A ind for Independent shift.
Revised: CORAL algorithms ( ϕ ⊥⊥ E ) is the same as CACM ’s constraint ϕ ⊥⊥ A ind for Independent shift.

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: We perform experiments on MNIST, small NORB, and Waterbirds datasets to demonstrate our main claims: existing DG algorithms perform worse on multi-attribute shifts; CACM with the correct graph-based constraints significantly outperforms these algorithms; and incorrect constraints cannot match the above accuracy.
Revised: We perform experiments on MNIST, small NORB, and Waterbirds datasets to demonstrate our main claims: existing DG algorithms perform worse on multi-attribute shifts; CACM with the correct graph-based constraints significantly outperforms these algorithms; and incorrect constraints cannot match the above accuracy.

Original: While we provide constraints for all shifts in Proposition 3.1, our empirical experiments with datasets focus on commonly occurring Causal and Independent shifts.
Revised: While we provide constraints for all shifts in Proposition 3.1, our empirical experiments with datasets focus on commonly occurring Causal and Independent shifts.

Original: All experiments are performed in PyTorch 1.10 with NVIDIA Tesla P40 and P100 GPUs, and building on DomainBed (Gulrajani & Lopez-Paz, 2021) and OoD-Bench (Ye et al., 2022).
Revised: All experiments are performed in PyTorch 1.10 with NVIDIA Tesla P40 and P100 GPUs, and building on DomainBed (Gulrajani & Lopez-Paz, 2021) and OoD-Bench (Ye et al., 2022).

Original: Regularizing on model’s logit scores provides better accuracy than ϕ ( x ) ; hence we adopt it for all our experiments.
Revised: Regularizing on model’s logit scores provides better accuracy than ϕ ( x ) ; hence we adopt it for all our experiments.

Original: 5. D ATASETS & B ASELINE DG ALGORITHMS
Revised: 5. D ATASETS & B ASELINE DG ALGORITHMS

Original: We introduce three new datasets for the multi-attribute shift problem.
Revised: We introduce three new datasets for the multi-attribute shift problem.

Original: For all datasets, details of environments, architectures, visualizations, and setup generation are in Suppl.
Revised: For all datasets, details of environments, architectures, visualizations, and setup generation are in Suppl.

Original: D.1.
Revised: D.1.

Original: MNIST.
Revised: MNIST.

Original: Colored (Arjovsky et al., 2019) and Rotated MNIST (Ghifary et al., 2015) present Causal
Revised: Colored (Arjovsky et al., 2019) and Rotated MNIST (Ghifary et al., 2015) present Causal

Original: ( A cause = color ) and Independent ( A ind = rotation ) distribution shifts, respectively.
Revised: ( A cause = color ) and Independent ( A ind = rotation ) distribution shifts, respectively.

Original: We combine these to obtain a multi-attribute dataset with A cause and A ind ( col + rot ).
Revised: We combine these to obtain a multi-attribute dataset with A cause and A ind ( col + rot ).

Original: For comparison, we also evaluate on single-attribute A cause (Colored) and A ind (Rotated) MNIST datasets.
Revised: For comparison, we also evaluate on single-attribute A cause (Colored) and A ind (Rotated) MNIST datasets.

Original: small NORB (LeCun et al., 2004).
Revised: small NORB (LeCun et al., 2004).

Original: This dataset was used by Wiles et al.
Revised: This dataset was used by Wiles et al.

Original: (2022) to create a challenging DG task with single-attribute shifts, having multi-valued classes and attributes over realistic 3D objects.
Revised: (2022) to create a challenging DG task with single-attribute shifts, having multi-valued classes and attributes over realistic 3D objects.

Original: We create a multi-attribute shift dataset ( light + azi ), consisting of a causal connection, A cause = lighting , between lighting and object category Y ; and A ind = azimuth that varies independently across domains.
Revised: We create a multi-attribute shift dataset ( light + azi ), consisting of a causal connection, A cause = lighting , between lighting and object category Y ; and A ind = azimuth that varies independently across domains.

Original: We also evaluate on single-attribute A cause ( lighting ) and
Revised: We also evaluate on single-attribute A cause ( lighting ) and

Original: A ind ( azimuth ) datasets.
Revised: A ind ( azimuth ) datasets.

Original: Waterbirds.
Revised: Waterbirds.

Original: We use the original dataset (Sagawa et al., 2020) where bird type (water or land) ( Y ) is spuriously correlated with background ( A cause ).
Revised: We use the original dataset (Sagawa et al., 2020) where bird type (water or land) ( Y ) is spuriously correlated with background ( A cause ).

Original: To create a multi-attribute setup, we add different weather effects ( A ind ) to train and test data with probability p = 0.5 and 1.0 respectively.
Revised: To create a multi-attribute setup, we add different weather effects ( A ind ) to train and test data with probability p = 0.5 and 1.0 respectively.

Original: Baseline DG algorithms & implementation.
Revised: Baseline DG algorithms & implementation.

Original: We consider baseline algorithms optimizing for different constraints and statistics to compare to causal adaptive regularization: IRM (Arjovsky et al., 2019), IB-ERM and IB-IRM (Ahuja et al., 2021), VREx (Krueger et al., 2021), MMD (Li et al., 2018b), CORAL (Sun & Saenko, 2016), DANN (Gretton et al., 2012), Conditional-MMD (C-MMD)
Revised: We consider baseline algorithms optimizing for different constraints and statistics to compare to causal adaptive regularization: IRM (Arjovsky et al., 2019), IB-ERM and IB-IRM (Ahuja et al., 2021), VREx (Krueger et al., 2021), MMD (Li et al., 2018b), CORAL (Sun & Saenko, 2016), DANN (Gretton et al., 2012), Conditional-MMD (C-MMD)

Original: (Li et al., 2018b), Conditional-DANN (CDANN) (Li et al., 2018d), GroupDRO (Sagawa et al., 2020), Mixup (Yan et al., 2020), MLDG (Li et al., 2018a), SagNet (Nam et al., 2021), and
Revised: (Li et al., 2018b), Conditional-DANN (CDANN) (Li et al., 2018d), GroupDRO (Sagawa et al., 2020), Mixup (Yan et al., 2020), MLDG (Li et al., 2018a), SagNet (Nam et al., 2021), and

Original: RSC (Huang et al., 2020).
Revised: RSC (Huang et al., 2020).

Original: Following DomainBed (Gulrajani & Lopez-Paz, 2021), a random search is performed 20 times over the hyperparameter distribution for 3 seeds.
Revised: Following DomainBed (Gulrajani & Lopez-Paz, 2021), a random search is performed 20 times over the hyperparameter distribution for 3 seeds.

Original: The best models obtained across the three seeds are used to compute the mean and standard error.
Revised: The best models obtained across the three seeds are used to compute the mean and standard error.

Original: We use a validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al., 2019; Sagawa et al., 2020; Wiles et al., 2022;
Revised: We use a validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al., 2019; Sagawa et al., 2020; Wiles et al., 2022;

Original: Ye et al., 2022).
Revised: Ye et al., 2022).

Original: Further details are in Suppl.
Revised: Further details are in Suppl.

Original: D.
Revised: D.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Correct constraint derived from the causal graph matters.
Revised: Correct constraint derived from the causal graph matters.

Original: Table 2 shows the accuracy on test domain for all datasets.
Revised: Table 2 shows the accuracy on test domain for all datasets.

Original: Comparing the three prediction tasks for MNIST and small NORB, for all algorithms, accuracy on unseen test domain is highest under A ind shift and lowest under twoattribute shift ( A ind ∪ A cause ), reflecting the difficulty of a multi-attribute distribution shift.
Revised: Comparing the three prediction tasks for MNIST and small NORB, for all algorithms, accuracy on unseen test domain is highest under A ind shift and lowest under twoattribute shift ( A ind ∪ A cause ), reflecting the difficulty of a multi-attribute distribution shift.

Original: On the two-attribute shift task in MNIST, all DG algorithms obtain less than 50% accuracy whereas CACM obtains a 5% absolute improvement.
Revised: On the two-attribute shift task in MNIST, all DG algorithms obtain less than 50% accuracy whereas CACM obtains a 5% absolute improvement.

Original: Results on small NORB dataset are similar: CACM obtains 69.6% accuracy on the two-attribute task while the nearest baseline is MLDG at 64.2%.
Revised: Results on small NORB dataset are similar: CACM obtains 69.6% accuracy on the two-attribute task while the nearest baseline is MLDG at 64.2%.

Original: On both MNIST and small NORB, CACM also obtains highest accuracy on the A cause task.
Revised: On both MNIST and small NORB, CACM also obtains highest accuracy on the A cause task.

Original: On
Revised: On

Original: MNIST, even though IRM and VREx have been originally evaluated for the Color-only ( A cause ) task, under an extensive hyperparameter sweep as recommended in past work (Gulrajani & Lopez-Paz, 2021; Krueger et al., 2021; Ye et al., 2022), we find that CACM achieves a substantially higher accuracy (70%) than these methods, just 5 units lower than the optimal 75%.
Revised: MNIST, even though IRM and VREx have been originally evaluated for the Color-only ( A cause ) task, under an extensive hyperparameter sweep as recommended in past work (Gulrajani & Lopez-Paz, 2021; Krueger et al., 2021; Ye et al., 2022), we find that CACM achieves a substantially higher accuracy (70%) than these methods, just 5 units lower than the optimal 75%.

Original: While the A ind task is relatively easier, algorithms optimizing for the correct constraint achieve highest accuracy.
Revised: While the A ind task is relatively easier, algorithms optimizing for the correct constraint achieve highest accuracy.

Original: Note that
Revised: Note that

Original: MMD, CORAL, DANN, and CACM are based on the same independence constraint (see Table 1).
Revised: MMD, CORAL, DANN, and CACM are based on the same independence constraint (see Table 1).

Original: As mentioned in Section 4, we use the the domain attribute E for CACM ’s regularization constraint for A ind task, for full comparability with other algorithms that also use E .
Revised: As mentioned in Section 4, we use the the domain attribute E for CACM ’s regularization constraint for A ind task, for full comparability with other algorithms that also use E .

Original: The results indicate the importance of adaptive regularization for generalization.
Revised: The results indicate the importance of adaptive regularization for generalization.

Original: Table 2 also shows the OoD accuracy of algorithms on the original Waterbirds dataset (Sagawa et al., 2020) and its multi-attribute shift variant.
Revised: Table 2 also shows the OoD accuracy of algorithms on the original Waterbirds dataset (Sagawa et al., 2020) and its multi-attribute shift variant.

Original: Here we evaluate using worst-group accuracy consistent with past work (Sagawa et al., 2020; Yao et al., 2022).
Revised: Here we evaluate using worst-group accuracy consistent with past work (Sagawa et al., 2020; Yao et al., 2022).

Original: We observe that on single-attribute ( A cause ) as well as multi-attribute shift, CACM significantly outperforms baselines ( ∼ 6% absolute improvement).
Revised: We observe that on single-attribute ( A cause ) as well as multi-attribute shift, CACM significantly outperforms baselines ( ∼ 6% absolute improvement).

Original: Incorrect constraints hurt generalization.
Revised: Incorrect constraints hurt generalization.

Original: We now directly compare the effect of using correct versus incorrect (but commonly used) constraints for a dataset.
Revised: We now directly compare the effect of using correct versus incorrect (but commonly used) constraints for a dataset.

Original: To isolate the effect of a single constraint, we first consider the single-attribute shift on A cause and compare the application of different regularizer constraints.
Revised: To isolate the effect of a single constraint, we first consider the single-attribute shift on A cause and compare the application of different regularizer constraints.

Original: Proposition 3.1 provides the correct constraint for A cause : X c ⊥⊥
Revised: Proposition 3.1 provides the correct constraint for A cause : X c ⊥⊥

Original: A cause | Y, E .
Revised: A cause | Y, E .

Original: In addition, using d -separation on Causal -realized DAG from Figure 2, we see the following invalid constraints, X c ⊥⊥ A cause | E , X c ⊥⊥ A cause .
Revised: In addition, using d -separation on Causal -realized DAG from Figure 2, we see the following invalid constraints, X c ⊥⊥ A cause | E , X c ⊥⊥ A cause .

Original: Without knowing that the DGP corresponds to a Causal shift, one may apply these constraints that do not condition on the class.
Revised: Without knowing that the DGP corresponds to a Causal shift, one may apply these constraints that do not condition on the class.

Original: Results on small NORB (Table 3) show that using the incorrect constraint has an adverse effect: the correct constraint yields 85% accuracy while the best incorrect constraint achieves 79.7%.
Revised: Results on small NORB (Table 3) show that using the incorrect constraint has an adverse effect: the correct constraint yields 85% accuracy while the best incorrect constraint achieves 79.7%.

Original: Moreover, unlike the correct constraint, application of the incorrect constraint is sensitive to the λ (regularization weight) parameter: as λ increases, accuracy drops to less than 40% (Suppl.
Revised: Moreover, unlike the correct constraint, application of the incorrect constraint is sensitive to the λ (regularization weight) parameter: as λ increases, accuracy drops to less than 40% (Suppl.

Original: E.3, Figure 7).
Revised: E.3, Figure 7).

Original: Comparing these constraints on small NORB and MNIST (Table 4) reveals the importance of making the right structural assumptions.
Revised: Comparing these constraints on small NORB and MNIST (Table 4) reveals the importance of making the right structural assumptions.

Original: Typically, DG algorithms assume that distribution of causal features
Revised: Typically, DG algorithms assume that distribution of causal features

Original: X c does not change across domains (as in the graph in Fig. 2a).
Revised: X c does not change across domains (as in the graph in Fig. 2a).

Original: Then, both X c ⊥⊥ A cause | Y, E and X c ⊥⊥ A cause | Y should be correct constraints.
Revised: Then, both X c ⊥⊥ A cause | Y, E and X c ⊥⊥ A cause | Y should be correct constraints.

Original: However, conditioning on both Y and E provides a 5% point gain over conditioning on Y in NORB while the accuracy is comparable for
Revised: However, conditioning on both Y and E provides a 5% point gain over conditioning on Y in NORB while the accuracy is comparable for

Original: MNIST.
Revised: MNIST.

Original: Information about the data-generating process explains the result: Different domains in
Revised: Information about the data-generating process explains the result: Different domains in

Original: MNIST include samples from the same distribution whereas small NORB domains are sampled from a different set of toy objects, thus creating a correlation between X c and E , corresponding to the graph in Fig. 2b.
Revised: MNIST include samples from the same distribution whereas small NORB domains are sampled from a different set of toy objects, thus creating a correlation between X c and E , corresponding to the graph in Fig. 2b.

Original: Without information on the correct DGP, such gains will be difficult.
Revised: Without information on the correct DGP, such gains will be difficult.

Original: Finally, we replicate the above experiment for the multi-attribute shift setting for small NORB.
Revised: Finally, we replicate the above experiment for the multi-attribute shift setting for small NORB.

Original: To construct an incorrect constraint, we interchange the variables before inputting to CACM algorithm
Revised: To construct an incorrect constraint, we interchange the variables before inputting to CACM algorithm

Original: ( A ind gets used as A cause and vice-versa).
Revised: ( A ind gets used as A cause and vice-versa).

Original: Accuracy with interchanged variables ( 65 . 1 ± 1 . 6 ) is lower than that of correct CACM ( 69 . 6 ± 1 . 6 ).
Revised: Accuracy with interchanged variables ( 65 . 1 ± 1 . 6 ) is lower than that of correct CACM ( 69 . 6 ± 1 . 6 ).

Original: More ablations where baseline DG algorithms are provided CACM -like attributes as environment are in Suppl.
Revised: More ablations where baseline DG algorithms are provided CACM -like attributes as environment are in Suppl.

Original: E.4.
Revised: E.4.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Improving the robustness of models in the face of distribution shifts is a key challenge.
Revised: Improving the robustness of models in the face of distribution shifts is a key challenge.

Original: Several works have attempted to tackle the domain generalization problem (Wang et al., 2021;
Revised: Several works have attempted to tackle the domain generalization problem (Wang et al., 2021;

Original: Zhou et al., 2021) using different approaches – data augmentation (Cubuk et al., 2020; He et al., 2016; Zhu et al., 2017), and representation learning (Arjovsky et al., 2019; Deng et al., 2009; Higgins et al., 2017) being popular ones.
Revised: Zhou et al., 2021) using different approaches – data augmentation (Cubuk et al., 2020; He et al., 2016; Zhu et al., 2017), and representation learning (Arjovsky et al., 2019; Deng et al., 2009; Higgins et al., 2017) being popular ones.

Original: Trying to gauge the progress made by these approaches, Gulrajani and
Revised: Trying to gauge the progress made by these approaches, Gulrajani and

Original: Lopez-Paz (Gulrajani & Lopez-Paz, 2021) find that existing state-of-the-art DG algorithms do not improve over ERM.
Revised: Lopez-Paz (Gulrajani & Lopez-Paz, 2021) find that existing state-of-the-art DG algorithms do not improve over ERM.

Original: More recent work (Wiles et al., 2022;
Revised: More recent work (Wiles et al., 2022;

Original: Ye et al., 2022) uses datasets with different single-attribute shifts and empirically shows that different algorithms perform well over different distribution shifts, but no single algorithm performs consistently across all.
Revised: Ye et al., 2022) uses datasets with different single-attribute shifts and empirically shows that different algorithms perform well over different distribution shifts, but no single algorithm performs consistently across all.

Original: We provide (1) multiattribute shift benchmark datasets; (2) a causal interpretation of different kinds of shifts; and (3) an adaptive algorithm to identify the correct regularizer.
Revised: We provide (1) multiattribute shift benchmark datasets; (2) a causal interpretation of different kinds of shifts; and (3) an adaptive algorithm to identify the correct regularizer.

Revised: While we focus on images, OoD generalization

Revised: NEW (QsVS) on graph data is also challenged by multiple types of distribution shifts (Chen et al., 2022).

Original: Causally-motivated learning.
Revised: Causally-motivated learning.

Original: There has been recent work focused on causal representation learning (Arjovsky et al., 2019; Krueger et al., 2021; Locatello et al., 2020; Sch¨olkopf et al., 2021) for
Revised: There has been recent work focused on causal representation learning (Arjovsky et al., 2019; Krueger et al., 2021; Locatello et al., 2020; Sch¨olkopf et al., 2021) for

Original: OoD generalization.
Revised: OoD generalization.

Original: While these works attempt to learn the constraints for causal features from input features, we show that it is necessary to model the data-generating process and have access to auxiliary attributes to obtain a risk-invariant predictor, especially in multi-attribute distribution shift setups.
Revised: While these works attempt to learn the constraints for causal features from input features, we show that it is necessary to model the data-generating process and have access to auxiliary attributes to obtain a risk-invariant predictor, especially in multi-attribute distribution shift setups.

Original: Recent research has shown how causal graphs can be used to characterize and analyze the different kinds of distribution shifts that occur in real-world settings (Makar et al., 2022; Veitch et al., 2021).
Revised: Recent research has shown how causal graphs can be used to characterize and analyze the different kinds of distribution shifts that occur in real-world settings (Makar et al., 2022; Veitch et al., 2021).

Original: Our approach is similar in motivation but we extend from single-domain, single-attribute setups in past work to formally introduce multi-attribute distribution shifts in more complex and real-world settings.
Revised: Our approach is similar in motivation but we extend from single-domain, single-attribute setups in past work to formally introduce multi-attribute distribution shifts in more complex and real-world settings.

Original: Additionally, we do not restrict ourselves to binary-valued classes and attributes.
Revised: Additionally, we do not restrict ourselves to binary-valued classes and attributes.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We introduced CACM , an adaptive OoD generalization algorithm to characterize multi-attribute shifts and apply the correct independence constraints.
Revised: We introduced CACM , an adaptive OoD generalization algorithm to characterize multi-attribute shifts and apply the correct independence constraints.

Original: Through empirical experiments and theoretical analysis, we show the importance of modeling the causal relationships in the data-generating process.
Revised: Through empirical experiments and theoretical analysis, we show the importance of modeling the causal relationships in the data-generating process.

Original: That said, our work has limitations: the constraints from CACM are necessary but not sufficient for a risk-invariant predictor (e.g., unable to remove influence of unobserved spurious attributes).
Revised: That said, our work has limitations: the constraints from CACM are necessary but not sufficient for a risk-invariant predictor (e.g., unable to remove influence of unobserved spurious attributes).

Original:  E THICS AND B ROADER I MPACT S TATEMENT
Revised:  E THICS AND B ROADER I MPACT S TATEMENT

Original: Our work on modeling the data-generating process for improved out-of-distribution generalization is an important advance in building robust predictors for practical settings.
Revised: Our work on modeling the data-generating process for improved out-of-distribution generalization is an important advance in building robust predictors for practical settings.

Original: Such prediction algorithms, including methods building on representation learning, are increasingly a key element of decisionsupport and decision-making systems.
Revised: Such prediction algorithms, including methods building on representation learning, are increasingly a key element of decisionsupport and decision-making systems.

Original: We expect our approach to creating a robust predictor to be particularly valuable in real world setups where spurious attributes and real-world multi-attribute settings lead to biases in data.
Revised: We expect our approach to creating a robust predictor to be particularly valuable in real world setups where spurious attributes and real-world multi-attribute settings lead to biases in data.

Original: While not the focus of this paper, CACM may be applied to mitigate social biases (e.g., in language and vision datasets) whose structures can be approximated by the graphs in Figure 2.
Revised: While not the focus of this paper, CACM may be applied to mitigate social biases (e.g., in language and vision datasets) whose structures can be approximated by the graphs in Figure 2.

Original: Risks of using methods such as CACM , include excessive reliance or a false sense of confidence.
Revised: Risks of using methods such as CACM , include excessive reliance or a false sense of confidence.

Original: While methods such as CACM ease the process of building robust models, there remain many ways that an application may still fail (e.g., incorrect structural assumptions).
Revised: While methods such as CACM ease the process of building robust models, there remain many ways that an application may still fail (e.g., incorrect structural assumptions).

Original: AI applications must still be designed appropriately with support of all stakeholders and potentially affected parties, tested in a variety of settings, etc.
Revised: AI applications must still be designed appropriately with support of all stakeholders and potentially affected parties, tested in a variety of settings, etc.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide all required experimental details in Suppl.
Revised: We provide all required experimental details in Suppl.

Original: D including dataset details, training details and hyperparameter search sweeps.
Revised: D including dataset details, training details and hyperparameter search sweeps.

Original: We additionally submit our code as part of supplementary material which can be used to reproduce the experiments.
Revised: We additionally submit our code as part of supplementary material which can be used to reproduce the experiments.

Original: We provide proofs for all our theoretical results in
Revised: We provide proofs for all our theoretical results in

Original: Suppl.
Revised: Suppl.

Original: B.
Revised: B.

Original: A P RESENCE OF AUXILIARY ATTRIBUTE INFORMATION IN DATASETS
Revised: A P RESENCE OF AUXILIARY ATTRIBUTE INFORMATION IN DATASETS

Original: Unlike the full causal graph, attribute values as well as the relationships between class labels and attributes is often known.
Revised: Unlike the full causal graph, attribute values as well as the relationships between class labels and attributes is often known.

Original: CACM assumes access to attribute labels A only during training time, which are collected as part of the data collection process (e.g., as metadata with training data (Makar et al., 2022)).
Revised: CACM assumes access to attribute labels A only during training time, which are collected as part of the data collection process (e.g., as metadata with training data (Makar et al., 2022)).

Original: We start by discussing the availability of attributes in WILDS (Koh et al., 2021), a set of real-world datasets adapted for the domain generalization setting.
Revised: We start by discussing the availability of attributes in WILDS (Koh et al., 2021), a set of real-world datasets adapted for the domain generalization setting.

Original: Attribute labels available in the datasets include, the time (year) and region associated with satellite images in FMoW dataset (Christie et al., 2018) for predicting land use category, hospital from where the tissue patch was collected for tumor detection in Camelyon17 dataset (Bandi et al., 2018) and the demographic information for
Revised: Attribute labels available in the datasets include, the time (year) and region associated with satellite images in FMoW dataset (Christie et al., 2018) for predicting land use category, hospital from where the tissue patch was collected for tumor detection in Camelyon17 dataset (Bandi et al., 2018) and the demographic information for

Original: CivilComments dataset (Borkan et al., 2019).
Revised: CivilComments dataset (Borkan et al., 2019).

Original: (Koh et al., 2021) create different domains in WILDS using this metadata, consistent with our definition of E ∈ A as a special domain attribute.
Revised: (Koh et al., 2021) create different domains in WILDS using this metadata, consistent with our definition of E ∈ A as a special domain attribute.

Original: In addition, CACM requires the type of relationship between label Y and attributes.
Revised: In addition, CACM requires the type of relationship between label Y and attributes.

Original: This is often known, either based on how the dataset was collected or inferred based on domain knowledge or observation.
Revised: This is often known, either based on how the dataset was collected or inferred based on domain knowledge or observation.

Original: While the distinction between A ind and A ind can be established using a statistical test of independence on a given dataset, in general, the distinction between A cause , A sel and A conf within A ind needs to be provided by the user.
Revised: While the distinction between A ind and A ind can be established using a statistical test of independence on a given dataset, in general, the distinction between A cause , A sel and A conf within A ind needs to be provided by the user.

Original: As we show for the above datasets, the type of relationship can be inferred based on common knowledge or information on how the dataset was collected.
Revised: As we show for the above datasets, the type of relationship can be inferred based on common knowledge or information on how the dataset was collected.

Original: For FMoW dataset, time can be considered an Independent attribute ( A ind ) since it reflects the time at which images are captured which is not correlated with Y ; whereas region is a Confounded attribute since certain regions associated with certain Y labels are over-represented due to ease of data collection.
Revised: For FMoW dataset, time can be considered an Independent attribute ( A ind ) since it reflects the time at which images are captured which is not correlated with Y ; whereas region is a Confounded attribute since certain regions associated with certain Y labels are over-represented due to ease of data collection.

Original: Note that region cannot lead to Causal shift since the decision to take images in a region was not determined by the final label nor Selected for the same reason that the decision was not taken based on values of Y .
Revised: Note that region cannot lead to Causal shift since the decision to take images in a region was not determined by the final label nor Selected for the same reason that the decision was not taken based on values of Y .

Original: Similarly, for the Camelyon17 dataset, it is known that differences in slide staining or image acquisition leads to variation in tissue slides across hospitals , thus implying that hospital is an Independent attribute ( A ind ) (Koh et al., 2021; Komura & Ishikawa, 2018; Tellez et al., 2019); As another example from healthcare, a study in MIT Technology Review 1 discusses biased data where a person’s position ( A conf ) was spuriously correlated with disease prediction as patients lying down were more likely to be ill.
Revised: Similarly, for the Camelyon17 dataset, it is known that differences in slide staining or image acquisition leads to variation in tissue slides across hospitals , thus implying that hospital is an Independent attribute ( A ind ) (Koh et al., 2021; Komura & Ishikawa, 2018; Tellez et al., 2019); As another example from healthcare, a study in MIT Technology Review 2 discusses biased data where a person’s position ( A conf ) was spuriously correlated with disease prediction as patients lying down were more likely to be ill.

Original: As another example, (Sagawa et al., 2020) adapt MultiNLI dataset for OoD generalization due to the presence of spurious correlation between negation words (attribute) and the contradiction label between “premise” and “hypothesis” inputs.
Revised: As another example, (Sagawa et al., 2020) adapt MultiNLI dataset for OoD generalization due to the presence of spurious correlation between negation words (attribute) and the contradiction label between “premise” and “hypothesis” inputs.

Original: Here, negation words are a result of the contradiction label ( Causal shift), however this relationship between negation words and label may not always hold.
Revised: Here, negation words are a result of the contradiction label ( Causal shift), however this relationship between negation words and label may not always hold.

Original: Finally, for the CivilComments dataset, we expect the demographic features to be Confounded attributes as there could be biases which result in spurious correlation between comment toxicity and demographic information.
Revised: Finally, for the CivilComments dataset, we expect the demographic features to be Confounded attributes as there could be biases which result in spurious correlation between comment toxicity and demographic information.

Original: To provide examples showing the availability of attributes and their type of relationship with the label, Table 5 lists some popular datasets used for DG and the associated auxiliary information present as metadata.
Revised: To provide examples showing the availability of attributes and their type of relationship with the label, Table 5 lists some popular datasets used for DG and the associated auxiliary information present as metadata.

Original: In addition to above discussed datasets, weinclude the popularly used Waterbirds dataset (Sagawa et al., 2020) where the type of background (land/water) is assigned to bird images based on bird label; hence, being a Causal attribute (results on Waterbirds dataset are in Table 2).
Revised: In addition to above discussed datasets, weinclude the popularly used Waterbirds dataset (Sagawa et al., 2020) where the type of background (land/water) is assigned to bird images based on bird label; hence, being a Causal attribute (results on Waterbirds dataset are in Table 2).

Original: [Table]
Revised: [Table]

Original: machine-learning-a i-failed-covid-hospital-diagnosis-pandemic/
Revised: machine-learning-a i-failed-covid-hospital-diagnosis-pandemic/

Original: B P ROOFS B. P ROOF OF T HEOREM 2. 
Revised: B P ROOFS B. P ROOF OF T HEOREM 2.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Consider a causal DAG G over ⟨ X c , X , A , Y ⟩ and a corresponding generated dataset ( x i , a i , y i ) ni
Revised: Consider a causal DAG G over ⟨ X c , X , A , Y ⟩ and a corresponding generated dataset ( x i , a i , y i ) ni

Original: =1 , where X c is unobserved.
Revised: =1 , where X c is unobserved.

Original: Assume that graph G has the following property: X c is defined as the set of all parents of Y ( X c → Y ); and X c , A together cause X ( X c → X , and
Revised: Assume that graph G has the following property: X c is defined as the set of all parents of Y ( X c → Y ); and X c , A together cause X ( X c → X , and

Original: A → X ).
Revised: A → X ).

Original: The graph may have any other edges (see, e.g., DAG in Figure 1(b)).
Revised: The graph may have any other edges (see, e.g., DAG in Figure 1(b)).

Original: Let P G be the set of distributions consistent with graph G , obtained by changing P ( A | Y )  .
Revised: Let P G be the set of distributions consistent with graph G , obtained by changing P ( A | Y ) but not P ( Y | X c ) .

Original: Then the conditional independence constraints satisfied by X c are necessary for a (cross-entropy) risk-invariant predictor over P G .
Revised: Then the conditional independence constraints satisfied by X c are necessary for a (cross-entropy) riskinvariant predictor over P G .

Original: That is, if a predictor for Y does not satisfy any of these constraints, then there exists
Revised: That is, if a predictor for Y does not satisfy any of these constraints, then there exists

Original: a data distribution P ′ ∈
Revised: a data distribution P ′ ∈

Original: P G such that predictor’s risk will be higher than its risk in other distributions.
Revised: P G such that predictor’s risk will be higher than its risk in other distributions.

Original: Proof.
Revised: Proof.

Original: We consider X , Y, X c ,
Revised: We consider X , Y, X c ,

Original: A as random variables that are generated according to the datagenerating process corresponding to causal graph G .
Revised: A as random variables that are generated according to the datagenerating process corresponding to causal graph G .

Original: We assume that X c represents all the parents of Y .
Revised: We assume that X c represents all the parents of Y .

Original: X c also causes the observed features X
Revised: X c also causes the observed features X

Original: but X may be additionally affected by the attributes
Revised: but X may be additionally affected by the attributes

Original: A .
Revised: A .

Original: Let ˆ y
Revised: Let ˆ y

Original: =
Revised: =

Original: g
Revised: g

Original: ( x ) be a candidate predictor.
Revised: ( x ) be a candidate predictor.

Original: Then g ( X ) represents a random vector based on a deterministic function g of X .
Revised: Then g ( X ) represents a random vector based on a deterministic function g of X .

Original: Suppose there is an independence constraint ψ that is satisfied by X c but not g ( X ) .
Revised: Suppose there is an independence constraint ψ that is satisfied by X c but not g ( X ) .

Original: 2 Below we show that such a predictor g is not risk-invariant: there exist two data distributions with different
Revised: 3 Below we show that such a predictor g is not risk-invariant: there exist two data distributions with different

Original: P ( A | Y ) such that the risk of g is different for them.
Revised: P ( A | Y ) such that the risk of g is different for them.

Original: Without loss of generality, we can write g ( x ) as,
Revised: Without loss of generality, we can write g ( x ) as,

Original: [Equation]
Revised: [Equation]

Original: where h is an arbitrary, non-zero, deterministic function of the random variable X c .
Revised: where h is an arbitrary, non-zero, deterministic function of the random variable X c .

Original: Since X c satisfies the (conditional) independence constraint ψ and h is a deterministic function, h
Revised: Since X c satisfies the (conditional) independence constraint ψ and h is a deterministic function, h

Original: ( X c ) also satisfies ψ .
Revised: ( X c ) also satisfies ψ .

Original: Also since the predictor g ( X ) does not satisfy the constraint ψ , it implies that the random vector g ′ ( X , X c ) cannot satisfy the constraint ψ .
Revised: Also since the predictor g ( X ) does not satisfy the constraint ψ , it implies that the random vector g ′ ( X , X c ) cannot satisfy the constraint ψ .

Original: Thus, g ′ ( X , X c ) cannot be a function of X c only; it needs to depend on X too.
Revised: Thus, g ′ ( X , X c ) cannot be a function of X c only; it needs to depend on X too.

Original: Since X has two parents in the causal graph, X c and A , this implies that g ′ ( X , X c ) must depend on A too, and hence g ′
Revised: Since X has two parents in the causal graph, X c and A , this implies that g ′ ( X , X c ) must depend on A too, and hence g ′

Original: ( X , X c ) and A are not independent.
Revised: ( X , X c ) and A are not independent.

Original: Now, let us construct two data distributions P 1 and P 2 with the same marginal distributions of
Revised: Now, let us construct two data distributions P 1 and P 2 with the same marginal distributions of

Original: P ( Y ) , P ( A ) and P ( X c ) , such that P ( A | Y ) changes across them.
Revised: P ( Y ) , P ( A ) and P ( X c ) , such that P ( A | Y ) changes across them.

Original: Note that P ( Y | X c ) stays invariant because of the independent and stable causal mechanism property, i.e., P 1 ( Y | X c ) =
Revised: Note that P ( Y | X c ) stays invariant because of the independent and stable causal mechanism property, i.e., P 1 ( Y | X c ) =

Original: P 2 ( Y | X c ) .
Revised: P 2 ( Y | X c ) .

Original: For these two data distributions, change in P ( A | Y ) implies a change in P ( Y | A ) , i.e.,
Revised: For these two data distributions, change in P ( A | Y ) implies a change in P ( Y | A ) , i.e.,

Original: P 1 ( Y | A ) ̸
Revised: P 1 ( Y | A ) ̸

Original: = P 2 ( Y | A ) , since P ( Y | A ) =
Revised: = P 2 ( Y | A ) , since P ( Y | A ) =

Original: P ( A | Y ) P ( Y ) /P
Revised: P ( A | Y ) P ( Y ) /P

Original: ( A ) .
Revised: ( A ) .

Original: Also, since g ′ ( X , X c ) and A are not independent, P ( Y | g ′
Revised: Also, since g ′ ( X , X c ) and A are not independent, P ( Y | g ′

Original: ( X , X c )) will change, i.e., P 1 ( Y | g ′
Revised: ( X , X c )) will change, i.e., P 1 ( Y | g ′

Original: ( X , X c ))
Revised: ( X , X c ))

Original: ̸
Revised: ̸

Original: = P 2 ( Y | g ′
Revised: = P 2 ( Y | g ′

Original: ( X , X c )) .
Revised: ( X , X c )) .

Original: The risk over any distribution P can be written as (using the cross-entropy loss),
Revised: The risk over any distribution P can be written as (using the cross-entropy loss),

Original: [Equation]
Revised: [Equation]

Original: The risk difference is,
Revised: The risk difference is,

Original: [Equation]
Revised: [Equation]

Original: where the third and fourth terms cancel out because P 1 ( X c , Y )
Revised: where the third and fourth terms cancel out because P 1 ( X c , Y )

Original: =
Revised: =

Original: P 2 ( X c , Y ) and thus the risk of h ( X c ) is the same across P 1 and P 2 .
Revised: P 2 ( X c , Y ) and thus the risk of h ( X c ) is the same across P 1 and P 2 .

Original: However, the risk for g ′ ( X , X c ) is not the same since
Revised: However, the risk for g ′ ( X , X c ) is not the same since

Original: P 1 ( Y | g ′ ( X , X c ))
Revised: P 1 ( Y | g ′ ( X , X c ))

Original: ̸
Revised: ̸

Original: = P 2 ( Y | g ′
Revised: = P 2 ( Y | g ′

Original: ( X , X c )) .
Revised: ( X , X c )) .

Original: Thus the absolute risk difference is non-zero,
Revised: Thus the absolute risk difference is non-zero,

Original: [Equation]
Revised: [Equation]

Original: ) and g is not a risk-invariant predictor.
Revised: ) and g is not a risk-invariant predictor.

Original: Hence, satisfying conditional independencies that X c satisfies is necessary for a risk-invariant predictor.
Revised: Hence, satisfying conditional independencies that X c satisfies is necessary for a risk-invariant predictor.

Original: Remark.
Revised: Remark.

Original: In the above Theorem, we considered the case where P ( A | Y ) changes across distributions.
Revised: In the above Theorem, we considered the case where P ( A | Y ) changes across distributions.

Original: In the case where A and Y are independent, P ( A | Y ) =
Revised: In the case where A and Y are independent, P ( A | Y ) =

Original: P ( A ) and thus P ( A ) would change across distributions while P ( Y | A )
Revised: P ( A ) and thus P ( A ) would change across distributions while P ( Y | A )

Original: = P ( Y ) remained constant.
Revised: = P ( Y ) remained constant.

Original: Since g ′ ( X , X c ) depends on A and X c , we obtain P 1 ( Y
Revised: Since g ′ ( X , X c ) depends on A and X c , we obtain P 1 ( Y

Original: | g ′
Revised: | g ′

Original: ( X , X c ))
Revised: ( X , X c ))

Original: =
Revised: =

Original: P 2 ( Y | g ′ ( X , X c )) .
Revised: P 2 ( Y | g ′ ( X , X c )) .

Original: However, the risk difference can still be non-zero since P 1 ( A ) ̸
Revised: However, the risk difference can still be non-zero since P 1 ( A ) ̸

Original: = P 2 ( A ) and the risk expectation E P [ (cid:80)
Revised: = P 2 ( A ) and the risk expectation E P [ (cid:80)

Original: y y log g ′
Revised: y y log g ′

Original: ( X , X c )] is over
Revised: ( X , X c )] is over

Original: P ( Y, X c , A ) .
Revised: P ( Y, X c , A ) .

Original: B. P ROOF OF P ROPOSITION 3. 
Revised: B. P ROOF OF P ROPOSITION 3.

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: Given a causal DAG realized by specifying the target-attributes relationship in
Revised: Given a causal DAG realized by specifying the target-attributes relationship in

Original: Figure 2a, the correct constraint depends on the relationship of label Y with the attributes A .
Revised: Figure 2a, the correct constraint depends on the relationship of label Y with the attributes A .

Original: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =
Revised: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =

Original: A cause ∪ A conf ∪ A sel ).
Revised: A cause ∪ A conf ∪ A sel ).

Original: Then, the (conditional) independence constraints X c should satisfy are,
Revised: Then, the (conditional) independence constraints X c should satisfy are,

Original: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.
Revised: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.

Original: Causal: X c ⊥⊥ A caus | Y ; X c ⊥⊥ E ; X c ⊥⊥ A caus | Y, E 3.
Revised: Causal: X c ⊥⊥ A caus | Y ; X c ⊥⊥ E ; X c ⊥⊥ A caus | Y, E 3.

Original: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 4.
Revised: Confounded: X c ⊥⊥ A conf ; X c ⊥⊥ E ; X c ⊥⊥ A conf | E 4.

Original: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E
Revised: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E

Original: Proof.
Revised: Proof.

Original: The proof follows from d-separation (Pearl, 2009) on the causal DAGs realized from Figure 2a.
Revised: The proof follows from d-separation (Pearl, 2009) on the causal DAGs realized from Figure 2a.

Original: For each condition, Independent , Causal , Confounded and Selected , we provide the realized causal graphs below and derive the constraints.
Revised: For each condition, Independent , Causal , Confounded and Selected , we provide the realized causal graphs below and derive the constraints.

Original: [Figure]
Revised: [Figure]

Original: Independent: As we can see in Figure 3a, we have a collider X on the path from X c to A ind and
Revised: Independent: As we can see in Figure 3a, we have a collider X on the path from X c to A ind and

Original: X c to E .
Revised: X c to E .

Original: Since there is a single path here, we obtain the independence constraints X c ⊥⊥ A ind and X c ⊥⊥ E .
Revised: Since there is a single path here, we obtain the independence constraints X c ⊥⊥ A ind and X c ⊥⊥ E .

Original: Additionally, we see that conditioning on Y or E would not block the path from
Revised: Additionally, we see that conditioning on Y or E would not block the path from

Original: X c to A ind , which results in the remaining constraints: X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E and
Revised: X c to A ind , which results in the remaining constraints: X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E and

Original: X c ⊥⊥ A ind | Y, E .
Revised: X c ⊥⊥ A ind | Y, E .

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: Causal: From Figure 3b, we see that while the path X c
Revised: Causal: From Figure 3b, we see that while the path X c

Original: → X → A cause from X c to A cause contains a collider X , X c ̸⊥⊥ A cause due to the presence of node Y as a chain.
Revised: → X → A cause from X c to A cause contains a collider X , X c ̸⊥⊥ A cause due to the presence of node Y as a chain.

Original: By the d-separation criteria, X c and A cause are conditionally independent given Y =
Revised: By the d-separation criteria, X c and A cause are conditionally independent given Y =

Original: ⇒
Revised: ⇒

Original: X c ⊥⊥
Revised: X c ⊥⊥

Original: A cause | Y .
Revised: A cause | Y .

Original: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between X c and
Revised: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between X c and

Original: A cause =
Revised: A cause =

Original: ⇒ X c ⊥⊥
Revised: ⇒ X c ⊥⊥

Original: A cause | Y, E .
Revised: A cause | Y, E .

Original: We get the constraint X c ⊥⊥ E since all paths connecting
Revised: We get the constraint X c ⊥⊥ E since all paths connecting

Original: X c to E contain a collider (collider X in X c → X → A cause → E , collider A cause in
Revised: X c to E contain a collider (collider X in X c → X → A cause → E , collider A cause in

Original: X c → Y → A cause → E ).
Revised: X c → Y → A cause → E ).

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: Confounded: From Figure 3c, we see that all paths connecting X c and A conf contain a collider (collider X in X c → X → A conf , collider Y in X c → Y → C
Revised: Confounded: From Figure 3c, we see that all paths connecting X c and A conf contain a collider (collider X in X c → X → A conf , collider Y in X c → Y → C

Original: → A conf ).
Revised: → A conf ).

Original: Hence, X c ⊥⊥ A conf .
Revised: Hence, X c ⊥⊥ A conf .

Original: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between
Revised: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between

Original: X c and A conf =
Revised: X c and A conf =

Original: ⇒ X c ⊥⊥ A conf | E .
Revised: ⇒ X c ⊥⊥ A conf | E .

Original: We get the constraint X c ⊥⊥ E since all paths connecting
Revised: We get the constraint X c ⊥⊥ E since all paths connecting

Original: X c and E also contain a collider (collider X in X c → X → A conf → E , collider Y in
Revised: X c and E also contain a collider (collider X in X c → X → A conf → E , collider Y in

Original: X c → Y → C → A conf → E ).
Revised: X c → Y → C → A conf → E ).

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: Selected:
Revised: Selected:

Original: For the observed data, the selection variable is always conditioned on, with S = 1 indicating inclusion of sample in data.
Revised: For the observed data, the selection variable is always conditioned on, with S = 1 indicating inclusion of sample in data.

Original: The selection variable S is a collider in Figure 3d and we condition on it.
Revised: The selection variable S is a collider in Figure 3d and we condition on it.

Original: Hence, X c ̸⊥⊥ A sel .
Revised: Hence, X c ̸⊥⊥ A sel .

Original: Conditioning on Y breaks the edge X c → Y , and hence all paths between
Revised: Conditioning on Y breaks the edge X c → Y , and hence all paths between

Original: X c and A sel now contain a collider (collider X in X c → X → A sel ) =
Revised: X c and A sel now contain a collider (collider X in X c → X → A sel ) =

Original: ⇒
Revised: ⇒

Original: X c ⊥⊥
Revised: X c ⊥⊥

Original: A sel | Y .
Revised: A sel | Y .

Original: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between
Revised: Additionally, conditioning on E is valid since E does not appear as a collider on any paths between

Original: X c and A sel =
Revised: X c and A sel =

Original: ⇒ X c ⊥⊥
Revised: ⇒ X c ⊥⊥

Original: A sel | Y, E .
Revised: A sel | Y, E .

Original: Hence, we obtain,
Revised: Hence, we obtain,

Original: [Equation]
Revised: [Equation]

Original: B.2. P ROOF OF C OROLLARY 3.0. 
Revised: B.2. P ROOF OF C OROLLARY 3.0.

Original: Corollary 3.1.
Revised: Corollary 3.1.

Original: All the above derived constraints are valid for Graph 2a.
Revised: All the above derived constraints are valid for Graph 2a.

Original: However, in the presence of a correlation between E and X c (Graph 2b), only the constraints conditioned on E hold true.
Revised: However, in the presence of a correlation between E and X c (Graph 2b), only the constraints conditioned on E hold true.

Original: If there is a correlation between X c and E , X c ̸⊥⊥ E .
Revised: If there is a correlation between X c and E , X c ̸⊥⊥ E .

Original: We can see from Figure 3 that in the presence of X c − E correlation, X c ̸⊥⊥ A ind ; X c ̸⊥⊥ A ind | Y ( 3a), X c ̸⊥⊥ A cause | Y ( 3b), X c ̸⊥⊥ A conf ( 3c) and X c ̸⊥⊥ A sel | Y ( 3d).
Revised: We can see from Figure 3 that in the presence of X c − E correlation, X c ̸⊥⊥ A ind ; X c ̸⊥⊥ A ind | Y ( 3a), X c ̸⊥⊥ A cause | Y ( 3b), X c ̸⊥⊥ A conf ( 3c) and X c ̸⊥⊥ A sel | Y ( 3d).

Original: Hence, conditioning on environment E is required for the valid independence constraints.
Revised: Hence, conditioning on environment E is required for the valid independence constraints.

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF T HEOREM 3.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Under the canonical causal graph in Figure 2(a,b), there exists no (conditional) independence constraint over ⟨ X c , A , Y ⟩ that is valid for all realized DAGs as the type of multiattribute shifts vary.
Revised: Under the canonical causal graph in Figure 2(a,b), there exists no (conditional) independence constraint over ⟨ X c , A , Y ⟩ that is valid for all realized DAGs as the type of multiattribute shifts vary.

Original: Hence, for any predictor algorithm for Y that uses a single (conditional) independence constraint over its representation ϕ
Revised: Hence, for any predictor algorithm for Y that uses a single (conditional) independence constraint over its representation ϕ

Original: ( X ) , A and Y , there exists a realized DAG G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor for distributions in P G , where P G is the set of distributions obtained by changing P ( A | Y ) .
Revised: ( X ) , A and Y , there exists a realized DAG G and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor for distributions in P G , where P G is the set of distributions obtained by changing P ( A | Y ) .

Original: Proof.
Revised: Proof.

Original: The proof follows from an application of Proposition 3.1 and Theorem 2.1.
Revised: The proof follows from an application of Proposition 3.1 and Theorem 2.1.

Original: First claim.
Revised: First claim.

Original: Under the canonical graph from Figure 2(a or b), the four types of attribute shifts possible are Independent , Causal , Confounded and Selected .
Revised: Under the canonical graph from Figure 2(a or b), the four types of attribute shifts possible are Independent , Causal , Confounded and Selected .

Original: From the constraints provided for these four types of attribute shifts in Proposition 3.1, it is easy to observe that there is no single constraint that is satisfied across all four shifts.
Revised: From the constraints provided for these four types of attribute shifts in Proposition 3.1, it is easy to observe that there is no single constraint that is satisfied across all four shifts.

Original: Thus, given a data distribution (and hence, dataset) with specific types of multi-attribute shifts such that X c satisfies a (conditional) independence constraint w.r.t.
Revised: Thus, given a data distribution (and hence, dataset) with specific types of multi-attribute shifts such that X c satisfies a (conditional) independence constraint w.r.t.

Original: a subset of attributes A s ⊆
Revised: a subset of attributes A s ⊆

Original: A , it is always possible to change the type of at least one of the those attributes’ shifts to create a new data distribution (dataset) where the same constraint will not hold.
Revised: A , it is always possible to change the type of at least one of the those attributes’ shifts to create a new data distribution (dataset) where the same constraint will not hold.

Original: Second Claim.
Revised: Second Claim.

Original: To prove the second claim, suppose that there exists a predictor for Y based on a single conditional independence constraint over its representation, ψ ( ϕ ( X ) , A s , Y ) where A s ⊆ A .
Revised: To prove the second claim, suppose that there exists a predictor for Y based on a single conditional independence constraint over its representation, ψ ( ϕ ( X ) , A s , Y ) where A s ⊆ A .

Original: Since the same constraint is not valid across all attribute shifts, we can always construct a realized graph G (and a corresponding data distribution) by changing the type of at least one attribute shift
Revised: Since the same constraint is not valid across all attribute shifts, we can always construct a realized graph G (and a corresponding data distribution) by changing the type of at least one attribute shift

Original: A ∈ A s , such that X c would not satisfy the same constraint as ϕ ( X ) .
Revised: A ∈ A s , such that X c would not satisfy the same constraint as ϕ ( X ) .

Original: Further, under this G ,
Revised: Further, under this G ,

Original: X c would satisfy a different constraint on the same attributes.
Revised: X c would satisfy a different constraint on the same attributes.

Original: From Theorem 2.1, all conditional independence constraints satisfied by X c under G are necessary to be satisfied for a risk-invariant predictor.
Revised: From Theorem 2.1, all conditional independence constraints satisfied by X c under G are necessary to be satisfied for a risk-invariant predictor.

Original: Hence, for the class of distributions P G , a single constraint-based predictor cannot be a risk-invariant predictor.
Revised: Hence, for the class of distributions P G , a single constraint-based predictor cannot be a risk-invariant predictor.

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: Even when | A | = 1 , an algorithm using a single independence constraint over ⟨ ϕ ( X ) , A, Y ⟩ cannot yield a risk-invariant predictor for all kinds of single-attribute shift datasets.
Revised: Even when | A | = 1 , an algorithm using a single independence constraint over ⟨ ϕ ( X ) , A, Y ⟩ cannot yield a risk-invariant predictor for all kinds of single-attribute shift datasets.

Original: Proof.
Revised: Proof.

Original: Given a fixed (conditional) independence constraint over a predictor’s representation, ψ ( ϕ ( X ) , A s , Y ) , the proof of Theorem 3.1 relied on changing the target relationship type (and hence distribution shift type) for the attributes involved in the constraint.
Revised: Given a fixed (conditional) independence constraint over a predictor’s representation, ψ ( ϕ ( X ) , A s , Y ) , the proof of Theorem 3.1 relied on changing the target relationship type (and hence distribution shift type) for the attributes involved in the constraint.

Original: When | A | = 1 , the constraint is on a single attribute A , ψ ( ϕ ( X ) , A, Y ) and the same proof logic follows.
Revised: When | A | = 1 , the constraint is on a single attribute A , ψ ( ϕ ( X ) , A, Y ) and the same proof logic follows.

Original: From Proposition 3.1, given a fixed constraint, we can always choose a single-attribute shift type (and realized
Revised: From Proposition 3.1, given a fixed constraint, we can always choose a single-attribute shift type (and realized

Original: DAG G )
Revised: DAG G )

Original: such that the constraint is not valid for X c .
Revised: such that the constraint is not valid for X c .

Original: Moreover, under G , X c would satisfy a different conditional independence constraint wrt the same attribute.
Revised: Moreover, under G , X c would satisfy a different conditional independence constraint wrt the same attribute.

Original: From Theorem 2.1, since the predictor does not satisfy a conditional independence constraint satisfied by X c , it cannot be a risk-invariant predictor for datasets sampled from P G .
Revised: From Theorem 2.1, since the predictor does not satisfy a conditional independence constraint satisfied by X c , it cannot be a risk-invariant predictor for datasets sampled from P G .

Original: C CACM A LGORITHM
Revised: C CACM A LGORITHM

Original: We provide the CACM algorithm for a general graph G below (Algorithm 1).
Revised: We provide the CACM algorithm for a general graph G below (Algorithm 1).

Original: Algorithm 1 CACM
Revised: Algorithm 1 CACM

Original: Input: Dataset ( x i , a i , y i ) ni =1 , causal DAG G
Revised: Input: Dataset ( x i , a i , y i ) ni =1 , causal DAG G

Original: Output: Function g ( x ) =
Revised: Output: Function g ( x ) =

Original: g 1 ( ϕ ( x )) :
Revised: g 1 ( ϕ ( x )) :

Original: X → Y
Revised: X → Y

Original: A ← set of observed variables in G except Y, E (special domain attribute)
Revised: A ← set of observed variables in G except Y, E (special domain attribute)

Original: C ← {} ▷ mapping of A to A s
Revised: C ← {} ▷ mapping of A to A s

Original: Phase I: Derive correct independence constraints for A ∈ A do if ( X c , A ) are d-separated then
Revised: Phase I: Derive correct independence constraints for A ∈ A do if ( X c , A ) are d-separated then

Original: X c ⊥⊥ A is a valid independence constraint else if ( X c , A ) are d-separated conditioned on any subset A s of the remaining observed variables in A \ { A } ∪ { Y } then
Revised: X c ⊥⊥ A is a valid independence constraint else if ( X c , A ) are d-separated conditioned on any subset A s of the remaining observed variables in A \ { A } ∪ { Y } then

Original: X c ⊥⊥ A | A s is a valid independence constraint
Revised: X c ⊥⊥ A | A s is a valid independence constraint

Original: C
Revised: C

Original: [ A ] =
Revised: [ A ] =

Original: A s end if end for
Revised: A s end if end for

Original: Phase II: Apply regularization penalty using constraints derived for
Revised: Phase II: Apply regularization penalty using constraints derived for

Original: A ∈
Revised: A ∈

Original: A do if X c ⊥⊥ A then
Revised: A do if X c ⊥⊥ A then

Original: RegPenalty A = (cid:80)
Revised: RegPenalty A = (cid:80)

Original: | E | (cid:80) |
Revised: | E | (cid:80) |

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: MMD( P ( ϕ ( x ) | A i ) , P ( ϕ ( x ) | A j ))
Revised: MMD( P ( ϕ ( x ) | A i ) , P ( ϕ ( x ) | A j ))

Original: else if A is in C then
Revised: else if A is in C then

Original: A s = C
Revised: A s = C

Original: [ A ]
Revised: [ A ]

Original: RegPenalty A = (cid:80)
Revised: RegPenalty A = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: s (cid:80)
Revised: s (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ( P ( ϕ ( x ) | A
Revised: ( P ( ϕ ( x ) | A

Original: i , a ) , P ( ϕ ( x ) |
Revised: i , a ) , P ( ϕ ( x ) |

Original: A j , a )) end if end for
Revised: A j , a )) end if end for

Original: RegPenalty =
Revised: RegPenalty =

Original: (cid:80) A ∈A λ
Revised: (cid:80) A ∈A λ

Original: A RegPenalty A g 1 , ϕ = arg min g 1 ,ϕ ; ℓ ( g 1 ( ϕ ( x )) , y )
Revised: A RegPenalty A g 1 , ϕ = arg min g 1 ,ϕ ; ℓ ( g 1 ( ϕ ( x )) , y )

Original: + RegPenalty
Revised: + RegPenalty

Original: Remark.
Revised: Remark.

Original: If E is observed, we always condition on E because of Corollary 3.1.
Revised: If E is observed, we always condition on E because of Corollary 3.1.

Original: For the special case of Figure 2, CACM uses the following regularization penalty ( RegPenalty ) for
Revised: For the special case of Figure 2, CACM uses the following regularization penalty ( RegPenalty ) for

Original: Independent , Causal , Confounded and Selected shifts,
Revised: Independent , Causal , Confounded and Selected shifts,

Original: [Equation]
Revised: [Equation]

Original: MMD( P ( ϕ ( x ) | a i,ind ) , P ( ϕ ( x ) | a j,ind ))
Revised: MMD( P ( ϕ ( x ) | a i,ind ) , P ( ϕ ( x ) | a j,ind ))

Original: RegPenalty A cause = (cid:88) | E | (cid:88) y ∈ Y
Revised: RegPenalty A cause = (cid:88) | E | (cid:88) y ∈ Y

Original: | A cause | (cid:88) i = (cid:88)
Revised: | A cause | (cid:88) i = (cid:88)

Original: j>i
Revised: j>i

Original: MMD( P ( ϕ ( x ) | a i,cause , y ) , P ( ϕ ( x ) | a j,cause , y ))
Revised: MMD( P ( ϕ ( x ) | a i,cause , y ) , P ( ϕ ( x ) | a j,cause , y ))

Original: RegPenalty A conf = (cid:88) | E | | A conf | (cid:88) i = (cid:88)
Revised: RegPenalty A conf = (cid:88) | E | | A conf | (cid:88) i = (cid:88)

Original: j>i
Revised: j>i

Original: MMD( P ( ϕ ( x ) | a i,conf ) , P ( ϕ ( x ) | a j,conf ))
Revised: MMD( P ( ϕ ( x ) | a i,conf ) , P ( ϕ ( x ) | a j,conf ))

Original: RegPenalty A sel = (cid:88) | E | (cid:88) y ∈ Y
Revised: RegPenalty A sel = (cid:88) | E | (cid:88) y ∈ Y

Original: | A sel | (cid:88) i = (cid:88)
Revised: | A sel | (cid:88) i = (cid:88)

Original: j>i
Revised: j>i

Original: MMD( P ( ϕ ( x ) | a i,sel , y ) , P ( ϕ ( x ) | a j,sel , y ))
Revised: MMD( P ( ϕ ( x ) | a i,sel , y ) , P ( ϕ ( x ) | a j,sel , y ))

Original: D E XPERIMENTAL D ETAILS D. D ATASETS
Revised: D E XPERIMENTAL D ETAILS D. D ATASETS

Original: MNIST.
Revised: MNIST.

Original: Rotated (Ghifary et al., 2015) and Colored MNIST (Arjovsky et al., 2019) present distinct distribution shifts.
Revised: Rotated (Ghifary et al., 2015) and Colored MNIST (Arjovsky et al., 2019) present distinct distribution shifts.

Original: While Rotated MNIST only has A ind wrt.
Revised: While Rotated MNIST only has A ind wrt.

Original: rotation attribute ( R ), Colored MNIST only has A cause wrt.
Revised: rotation attribute ( R ), Colored MNIST only has A cause wrt.

Original: color attribute ( C ).
Revised: color attribute ( C ).

Original: We combine these datasets to obtain a multi-attribute dataset with A cause =
Revised: We combine these datasets to obtain a multi-attribute dataset with A cause =

Original: { C } and A ind = { R } .
Revised: { C } and A ind = { R } .

Original: Each domain E i has a specific rotation angle r i and a specific correlation corr i between color C and label Y .
Revised: Each domain E i has a specific rotation angle r i and a specific correlation corr i between color C and label Y .

Original: Our setup consists of 3 domains:
Revised: Our setup consists of 3 domains:

Original: E 1 , E 2 ∈ E tr (training), E 3 ∈ E te (test).
Revised: E 1 , E 2 ∈ E tr (training), E 3 ∈ E te (test).

Original: We define corr i
Revised: We define corr i

Original: = P ( Y = 

Revised: = P ( Y =

Original: | C = 1) =
Revised: | C = 1) =

Original: P ( Y = 0 | C = 0) in E i .
Revised: P ( Y = 0 | C = 0) in E i .

Original: In our setup, r 1 = 15 ◦ , r 2 = 60 ◦ , r 3 = 90 ◦
Revised: In our setup, r 1 = 15 ◦ , r 2 = 60 ◦ , r 3 = 90 ◦

Original: and corr 1 = 0 .
Revised: and corr 1 = 0 .

Original: 9 , corr 2 = 0 .
Revised: 9 , corr 2 = 0 .

Original: 8 , corr 3 = 0 .
Revised: 8 , corr 3 = 0 .

Original: All environments have 25% label noise, as in (Arjovsky et al., 2019).
Revised: All environments have 25% label noise, as in (Arjovsky et al., 2019).

Original: For all experiments on MNIST, we use a two-layer perceptron consistent with previous works (Arjovsky et al., 2019; Krueger et al., 2021).
Revised: For all experiments on MNIST, we use a two-layer perceptron consistent with previous works (Arjovsky et al., 2019; Krueger et al., 2021).

Original: Y =
Revised: Y =

Original: Y =(a)
Revised: Y =(a)

Original: Y =
Revised: Y =

Original: Y =(b)
Revised: Y =(b)

Original: Y =
Revised: Y =

Original: Y =(c) small NORB.
Revised: Y =(c) small NORB.

Original: Moving beyond simple binary classification, we use small NORB (LeCun et al., 2004), an object recognition dataset, to create a challenging setup with multi-valued classes and attributes over realistic 3D objects.
Revised: Moving beyond simple binary classification, we use small NORB (LeCun et al., 2004), an object recognition dataset, to create a challenging setup with multi-valued classes and attributes over realistic 3D objects.

Original: It consists of images of toys of five categories with varying lighting, elevation, and azimuths.
Revised: It consists of images of toys of five categories with varying lighting, elevation, and azimuths.

Original: The objective is to classify unseen samples of the five categories.
Revised: The objective is to classify unseen samples of the five categories.

Original: (Wiles et al., 2022) introduced single-attribute shifts for this dataset.
Revised: (Wiles et al., 2022) introduced single-attribute shifts for this dataset.

Original: We combine the Causal shift,
Revised: We combine the Causal shift,

Original: A cause = lighting wherein there is a correlation between lighting condition lighting i and toy category
Revised: A cause = lighting wherein there is a correlation between lighting condition lighting i and toy category

Original: y
Revised: y

Original: i ; and Independent shift, A ind = azimuth that varies independently across domains, to generate our multi-attribute dataset light + azi .
Revised: i ; and Independent shift, A ind = azimuth that varies independently across domains, to generate our multi-attribute dataset light + azi .

Original: Training domains have 0.9 and 0.95 spurious correlation with lighting whereas there is no correlation in test domain.
Revised: Training domains have 0.9 and 0.95 spurious correlation with lighting whereas there is no correlation in test domain.

Original: We add 5% label noise in all environments.
Revised: We add 5% label noise in all environments.

Original: We use ResNet-18 (pre-trained on ImageNet) for all settings and fine tune for our task.
Revised: We use ResNet-18 (pre-trained on ImageNet) for all settings and fine tune for our task.

Original: Y =
Revised: Y =

Original: Y = 1 Y =
Revised: Y = 1 Y =

Original: Y = 

Revised: Y =

Original: Y =(a)
Revised: Y =(a)

Original: Y =
Revised: Y =

Original: Y = 1 Y =
Revised: Y = 1 Y =

Original: Y = 4 Y =(b)
Revised: Y = 4 Y =(b)

Original: Y =
Revised: Y =

Original: Y = 1 Y =
Revised: Y = 1 Y =

Original: Y = 

Revised: Y =

Original: Y =(c)
Revised: Y =(c)

Original: Waterbirds.
Revised: Waterbirds.

Original: We use the Waterbirds dataset from Sagawa et al.
Revised: We use the Waterbirds dataset from Sagawa et al.

Original: This dataset classifies birds as “waterbird” or “landbird”, where bird type ( Y ) is spuriously correlated with background ( bgd ) – “waterbird” images are spuriously correlated with “water” backgrounds ( ocean, natural lake ) and “landbird” images with “land” backgrounds ( bamboo forest, broadleaf forest ).
Revised: This dataset classifies birds as “waterbird” or “landbird”, where bird type ( Y ) is spuriously correlated with background ( bgd ) – “waterbird” images are spuriously correlated with “water” backgrounds ( ocean, natural lake ) and “landbird” images with “land” backgrounds ( bamboo forest, broadleaf forest ).

Original: Since background is selected based on Y , A cause = background .
Revised: Since background is selected based on Y , A cause = background .

Original: The dataset is created by pasting bird images from CUB dataset (Wah et al., 2011) onto backgrounds from the Places dataset (Zhou et al., 2018).
Revised: The dataset is created by pasting bird images from CUB dataset (Wah et al., 2011) onto backgrounds from the Places dataset (Zhou et al., 2018).

Original: There is 0.95 correlation between the bird type and background during training i.e., 95% of all waterbirds are placed against a water background, while 95% of all landbirds are placed against a land background.
Revised: There is 0.95 correlation between the bird type and background during training i.e., 95% of all waterbirds are placed against a water background, while 95% of all landbirds are placed against a land background.

Original: We create training domains based on background ( |E tr | = | A cause | = 2) as in Yao et al.
Revised: We create training domains based on background ( |E tr | = | A cause | = 2) as in Yao et al.

Original: We evaluate using worst-group error consistent with past work, where a group is defined as ( background, y ).
Revised: We evaluate using worst-group error consistent with past work, where a group is defined as ( background, y ).

Original: We generate the dataset using the official code from Sagawa et al.
Revised: We generate the dataset using the official code from Sagawa et al.

Original: and use the same train-validation-test splits.
Revised: and use the same train-validation-test splits.

Original: To create the multi-attribute shift variant of Waterbirds, we add weather effects ( A ind ) using the
Revised: To create the multi-attribute shift variant of Waterbirds, we add weather effects ( A ind ) using the

Original: Automold library 3 .
Revised: Automold library 4 .

Original: We add darkness effect (darkness coefficient = 0.7) during training with 0.
Revised: We add darkness effect (darkness coefficient = 0.7) during training with 0.

Original: probability and rain effect (rain type = ‘drizzle’, slant = 20) with 1.0 probability during test.
Revised: probability and rain effect (rain type = ‘drizzle’, slant = 20) with 1.0 probability during test.

Original: Hence, | A ind | =3 ( { no effect, darkness, rain } ).
Revised: Hence, | A ind | =3 ( { no effect, darkness, rain } ).

Original: Weather effect is applied independent of class label Y .
Revised: Weather effect is applied independent of class label Y .

Original: Our training domains are based on background and we perform worst-group evaluation, same as the setup described above.
Revised: Our training domains are based on background and we perform worst-group evaluation, same as the setup described above.

Original: Examples from train and test domains for multi-attribute shift dataset are provided in Figure 6.
Revised: Examples from train and test domains for multi-attribute shift dataset are provided in Figure 6.

Original: We use ResNet-50 (pre-trained on ImageNet) for all settings consistent with past work (Sagawa et al., 2020; Yao et al., 2022).
Revised: We use ResNet-50 (pre-trained on ImageNet) for all settings consistent with past work (Sagawa et al., 2020; Yao et al., 2022).

Original: All models were evaluated at the best early stopping epoch (as measured by the validation set), again consistent with Sagawa et al.
Revised: All models were evaluated at the best early stopping epoch (as measured by the validation set), again consistent with Sagawa et al.

Original: [Figure]
Revised: [Figure]

Original: Y =

Original: [Figure]
Revised: [Figure]

Revised: =

Original: [Figure]
Revised: [Figure]

Original: Y = Y =
Revised: Y = Y =(

Original: [Figure]
Revised: [Figure]

Revised: Y =

Original: [Figure]
Revised: Y =(c)

Original: D. I MPLEMENTATION DETAILS
Revised: D. I MPLEMENTATION DETAILS

Original: All methods are trained using Adam optimizer.
Revised: All methods are trained using Adam optimizer.

Original: MNIST dataset is trained for 5000 steps (default in
Revised: MNIST dataset is trained for 5000 steps (default in

Original: DomainBed (Gulrajani & Lopez-Paz, 2021)) while Waterbirds and small NORB are trained forsteps.
Revised: DomainBed (Gulrajani & Lopez-Paz, 2021)) while Waterbirds and small NORB are trained forsteps.

Original: Consistent with the default value in DomainBed, we use a batch size of 64 per domain for
Revised: Consistent with the default value in DomainBed, we use a batch size of 64 per domain for

Original: MNIST.
Revised: MNIST.

Original: For small NORB, we use a batch size of 128 per domain and a batch size of 16 per domain for Waterbirds.
Revised: For small NORB, we use a batch size of 128 per domain and a batch size of 16 per domain for Waterbirds.

Original: Model Selection.
Revised: Model Selection.

Original: We create 90% and 10% splits from each domain to be used for training and model selection (as needed) respectively.
Revised: We create 90% and 10% splits from each domain to be used for training/evaluation and model selection (as needed) respectively.

Original: We use a validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al., 2019;
Revised: We use a validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al., 2019;

Original: Ye et al., 2022; Wiles et al., 2022; Yao et al., 2022).
Revised: Ye et al., 2022; Wiles et al., 2022; Yao et al., 2022).

Original: Specifically, we adopt the test-domain validation from DomainBed for
Revised: Specifically, we adopt the test-domain validation from DomainBed for

Original: Synthetic, MNIST, and small NORB datasets where early stopping is not allowed and all models are trained for the same fixed number of steps to limit test domain access.
Revised: Synthetic, MNIST, and small NORB datasets where early stopping is not allowed and all models are trained for the same fixed number of steps to limit test domain access.

Original: For Waterbirds, we perform early stopping using the validation set consistent with past work (Sagawa et al., 2020; Yao et al., 2022).
Revised: For Waterbirds, we perform early stopping using the validation set consistent with past work (Sagawa et al., 2020; Yao et al., 2022).

Original: CACM and baselines implementation details. 
Revised: NEW (kwon) MMD implementation details. We use the radial basis function (RBF) kernel to compute the MMD penalty. Our implementation is adopted from DomainBed (Gulrajani & Lopez-Paz, 2021).

Revised: The kernel bandwidth is a hyperparameter and we perform a sweep over the hyperparamaeter search space to select the best RBF kernel bandwidth.

Revised: The search space for hyperparameter sweeps is provided in

Revised: Table 9, where γ corresponds to 1/bandwidth.

Revised: CACM and baselines implementation details.

Original: We provide the regularization constraints for different shifts used by CACM in Section C. For statistical efficiency, we use a single λ value as hyperparameter for MNIST and small NORB datasets.
Revised: We provide the regularization constraints for different shifts used by CACM in Section C. For statistical efficiency, we use a single λ value as hyperparameter for MNIST and small NORB datasets.

Original: The search space for hyperparameters is given in Table 9.
Revised: The search space for hyperparameters is given in Table 9.

Original: In MNIST and NORB, we input images and domains ( E = A ind ) to all baseline methods; CACM receives additional input A cause .
Revised: In MNIST and NORB, we input images and domains ( E = A ind ) to all baseline methods; CACM receives additional input A cause .

Original: Hence, in the Independent single-attribute shift, CACM and all baselines have access to exactly the same information.
Revised: Hence, in the Independent single-attribute shift, CACM and all baselines have access to exactly the same information.

Original: In Waterbirds, since E is not defined in the original dataset, we follow the setup from Yao et al.
Revised: In Waterbirds, since E is not defined in the original dataset, we follow the setup from Yao et al.

Original: (2022) to create domains based on backgrounds.
Revised: (2022) to create domains based on backgrounds.

Original: Here, we provide images and domains ( E = background ) as input to all baselines except GroupDRO; to ensure fair comparison with GroupDRO, we follow Sagawa et al.
Revised: Here, we provide images and domains ( E = background ) as input to all baselines except GroupDRO; to ensure fair comparison with GroupDRO, we follow Sagawa et al.

Original: and provide 4 groups as input based on ( background, y ) , along with images.
Revised: and provide 4 groups as input based on ( background, y ) , along with images.

Original: For CACM , we do not use background domains but provide the attribute A cause = background for the single-attribute dataset, and both
Revised: For CACM , we do not use background domains but provide the attribute A cause = background for the single-attribute dataset, and both

Original: A cause = background and A ind = weather for the multi-attribute shift dataset.
Revised: A cause = background and A ind = weather for the multi-attribute shift dataset.

Original: Hence, for the single-shift Waterbirds dataset, all baselines receive the same information as CACM .
Revised: Hence, for the single-shift Waterbirds dataset, all baselines receive the same information as CACM .

Original: D. H YPERPARAMETER SEARCH
Revised: D. H YPERPARAMETER SEARCH

Original: Following DomainBed (Gulrajani & Lopez-Paz, 2021), we perform a random search 20 times over the hyperparameter distribution and this process is repeated for total 3 seeds.
Revised: Following DomainBed (Gulrajani & Lopez-Paz, 2021), we perform a random search 20 times over the hyperparameter distribution and this process is repeated for total 3 seeds.

Original: The best models are obtained across the three seeds over which we compute the mean and standard error.
Revised: The best models are obtained across the three seeds over which we compute the mean and standard error.

Original: The hyperparameter search space for all datasets and algorithms is given in Table 9.
Revised: The hyperparameter search space for all datasets and algorithms is given in Table 9.

Original: E R ESULTS E. S YNTHETIC DATASET
Revised: E R ESULTS E. S YNTHETIC DATASET

Original: Our synthetic dataset is constructed based on the data-generating processes of the slab dataset (Mahajan et al., 2021; Shah et al., 2020).
Revised: Our synthetic dataset is constructed based on the data-generating processes of the slab dataset (Mahajan et al., 2021; Shah et al., 2020).

Original: The original slab dataset was introduced by (Shah et al., 2020) to demonstrate the simiplicity bias in neural networks as they learn the linear feature which is easier to learn in comparison to the slab feature.
Revised: The original slab dataset was introduced by (Shah et al., 2020) to demonstrate the simiplicity bias in neural networks as they learn the linear feature which is easier to learn in comparison to the slab feature.

Original: Our extended slab dataset, adds to the setting from (Mahajan et al., 2021) by using non-binary attributes and class labels to create a more challenging task and allows us to study DG algorithms in the presence of linear spurious features.
Revised: Our extended slab dataset, adds to the setting from (Mahajan et al., 2021) by using non-binary attributes and class labels to create a more challenging task and allows us to study DG algorithms in the presence of linear spurious features.

Original: Our dataset consists of 2-dimensional input X consisting of features X c and A ind .
Revised: Our dataset consists of 2-dimensional input X consisting of features X c and A ind .

Original: This is consistent with the graph in Figure 2 where attributes and causal features together determine observed features
Revised: This is consistent with the graph in Figure 2 where attributes and causal features together determine observed features

Original: X ; we concatenate X c and A ind to generate X in our synthetic setup.
Revised: X ; we concatenate X c and A ind to generate X in our synthetic setup.

Original: Causal feature X c has a non-linear “slab” relationship with Y while A ind has a linear relationship with Y .
Revised: Causal feature X c has a non-linear “slab” relationship with Y while A ind has a linear relationship with Y .

Original: We create three different datasets with Causal (E.1.1), Confounded (E.1.2) and Selected (E.1.3) A ind − Y relationship respectively.
Revised: We create three different datasets with Causal (E.1.1), Confounded (E.1.2) and Selected (E.1.3) A ind − Y relationship respectively.

Original: Implementation details.
Revised: Implementation details.

Original: In all setups, X c is a single-dimensional variable and has a uniform distribution Uniform[0 , 1] across all environments.
Revised: In all setups, X c is a single-dimensional variable and has a uniform distribution Uniform[0 , 1] across all environments.

Original: We use the default 3-layer MLP architecture from
Revised: We use the default 3-layer MLP architecture from

Original: DomainBed and use mean difference (L2) instead of MMD as the regularization penalty given the simplicity of the data.
Revised: DomainBed and use mean difference (L2) instead of MMD as the regularization penalty given the simplicity of the data.

Original: We use a batch size of 128 for all datasets.
Revised: We use a batch size of 128 for all datasets.

Original: E.1. Causal SHIFT
Revised: E.1. Causal SHIFT

Original: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).
Revised: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).

Original: X c has a uniform distribution Uniform[0 , 1] across all environments.
Revised: X c has a uniform distribution Uniform[0 , 1] across all environments.

Original: [Equation]
Revised: [Equation]

Original: Hence, we have a five-way classification setup ( | Y | = 5) with multi-valued attributes.
Revised: Hence, we have a five-way classification setup ( | Y | = 5) with multi-valued attributes.

Original: Following (Mahajan et al., 2021), the two training domains have p as 0.9 and 1.0, and the test domain has p = 0.0.
Revised: Following (Mahajan et al., 2021), the two training domains have p as 0.9 and 1.0, and the test domain has p = 0.0.

Original: We add 10% noise to Y in all environments.
Revised: We add 10% noise to Y in all environments.

Original: E.1. Confounded SHIFT
Revised: E.1. Confounded SHIFT

Original: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).
Revised: We have three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).

Original: X c has a uniform distribution Uniform[0 , 1] across all environments.
Revised: X c has a uniform distribution Uniform[0 , 1] across all environments.

Original: Our confounding variable c has different functional relationships with Y and A conf which vary across environments.
Revised: Our confounding variable c has different functional relationships with Y and A conf which vary across environments.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The true function for Y is given by,
Revised: The true function for Y is given by,

Original: [Equation]
Revised: [Equation]

Original: Observed Y and A conf are functions of confounding variable c and their distribution changes across environments as described below:
Revised: Observed Y and A conf are functions of confounding variable c and their distribution changes across environments as described below:

Original: [Equation]
Revised: [Equation]

Original: E.1. Selected SHIFT
Revised: E.1. Selected SHIFT

Original: Selected shifts arise due to selection effect in the data generating process and induce an association between Y and A sel .
Revised: Selected shifts arise due to selection effect in the data generating process and induce an association between Y and A sel .

Original: A data point is included in the sample only if selection variable S = 1 holds;
Revised: A data point is included in the sample only if selection variable S = 1 holds;

Original: S is a function of Y and A sel .
Revised: S is a function of Y and A sel .

Original: The selection criterion may differ between domains (Veitch et al., 2021).
Revised: The selection criterion may differ between domains (Veitch et al., 2021).

Original: We construct three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).
Revised: We construct three environments, E 1 , E 2 ∈ E tr (training) and E 3 ∈ E te (test).

Original: X c has a uniform distribution Uniform[0 , 1] across all environments.
Revised: X c has a uniform distribution Uniform[0 , 1] across all environments.

Original: Our selection variable S is a function of Y and
Revised: Our selection variable S is a function of Y and

Original: A sel .
Revised: A sel .

Original: We add 10% noise to Y in all environments
Revised: We add 10% noise to Y in all environments.

Original: [Equation]
Revised: [Equation]

Original: The true function for Y is given by,
Revised: The true function for Y is given by,

Original: [Equation]
Revised: [Equation]

Original: The function used to decide the selection variable S (and hence the selection shift) varies across environments through the parameter p .
Revised: The function used to decide the selection variable S (and hence the selection shift) varies across environments through the parameter p .

Original: [Equation]
Revised: [Equation]

Original: E. A F IXED CONDITIONAL INDEPENDENCE CONSTRAINT CANNOT WORK ACROSS ALL SHIFTS
Revised: F IXED CONDITIONAL INDEPENDENCE CONSTRAINT CANNOT WORK ACROSS ALL SHIFTS

Original: Here, we compare the performance of two popular independence constraints in the literature Mahajan et al.
Revised: Here, we compare the performance of two popular independence constraints in the literature Mahajan et al.

Original: (2021): unconditional X c ⊥⊥ A | E , and conditional on label
Revised: (2021): unconditional X c ⊥⊥ A | E , and conditional on label

Original: X c ⊥⊥ A | Y, E (we condition on
Revised: X c ⊥⊥ A | Y, E (we condition on

Original: E for fully generality) on Synthetic Causal , Confounded and Selected shift datasets (Table 6).
Revised: E for fully generality) on Synthetic Causal , Confounded and Selected shift datasets (Table 6).

Original: We train a model using ERM (cross-entropy) where the representation is regularized using either of the constraints.
Revised: We train a model using ERM (cross-entropy) where the representation is regularized using either of the constraints.

Original: As predicted by Theorem 3.1, neither constraint obtains best accuracy on all three datasets.
Revised: As predicted by Theorem 3.1, neither constraint obtains best accuracy on all three datasets.

Original: The conditional constraint is better on A cause and A sel datasets, whereas the unconditional constraint is better on A conf , consistent with Proposition 3.1.
Revised: The conditional constraint is better on A cause and A sel datasets, whereas the unconditional constraint is better on A conf , consistent with Proposition 3.1.

Original: Predictors with the correct constraint are also more risk-invariant, having lower gap between train and test accuracy.
Revised: Predictors with the correct constraint are also more risk-invariant, having lower gap between train and test accuracy.

Original: [Table]
Revised: [Table]

Original: E FFECT OF VARYING R EGULARIZATION PENALTY COEFFICIENT
Revised: E FFECT OF VARYING R EGULARIZATION PENALTY COEFFICIENT

Original: To understand how incorrect constraints affect model generalization capabilities we study the
Revised: To understand how incorrect constraints affect model generalization capabilities we study the

Original: Causal shift setup in small NORB.
Revised: Causal shift setup in small NORB.

Original: From Theorem 3.
Revised: From Theorem 3.

Original: we know the correct constraint for A cause :
Revised: we know the correct constraint for A cause :

Original: X c ⊥⊥
Revised: X c ⊥⊥

Original: A cause | Y, E .
Revised: A cause | Y, E .

Original: In addition, we see the following invalid constraint, X c ⊥⊥ A cause | E .
Revised: In addition, we see the following invalid constraint, X c ⊥⊥ A cause | E .

Original: We compare the performance of these two conditional independence constraints while varying the regularization penalty coefficient ( λ ) (Figure 7).
Revised: We compare the performance of these two conditional independence constraints while varying the regularization penalty coefficient ( λ ) (Figure 7).

Original: We perform our evaluation across three setups with different spurious correlation values of training environments and have consistent findings.
Revised: We perform our evaluation across three setups with different spurious correlation values of training environments and have consistent findings.

Original: We observe that application of the incorrect constraint is sensitive to λ (regularization weight) parameter : as λ increases, accuracy drops to less than 40%.
Revised: We observe that application of the incorrect constraint is sensitive to λ (regularization weight) parameter : as λ increases, accuracy drops to less than 40%.

Original: However, accuracy with the correct constraint stays invariant across different values of λ .
Revised: However, accuracy with the correct constraint stays invariant across different values of λ .

Original: [Figure]
Revised: [Figure]

Original: E. P ROVIDING ATTRIBUTE INFORMATION TO DG ALGORITHMS FOR A FAIRER COMPARISON
Revised: E. P ROVIDING ATTRIBUTE INFORMATION TO DG ALGORITHMS FOR A FAIRER COMPARISON

Original: CACM leverages attribute labels to apply the correct independence constraints derived from the causal graph.
Revised: CACM leverages attribute labels to apply the correct independence constraints derived from the causal graph.

Original: However, existing DG algorithms only use the input features X and the domain attribute.
Revised: However, existing DG algorithms only use the input features X and the domain attribute.

Original: Here we provide this attribute information to existing DG algorithms to create a more favorable setting for their application.
Revised: Here we provide this attribute information to existing DG algorithms to create a more favorable setting for their application.

Original: We show that even in this setup, these algorithms are not able to close the performance gap with CACM , showing the importance of the causal information through graphs.
Revised: We show that even in this setup, these algorithms are not able to close the performance gap with CACM , showing the importance of the causal information through graphs.

Original: E.4. S YNTHETIC DATASET
Revised: E.4. S YNTHETIC DATASET

Original: We consider our Synthetic dataset with Causal distribution shift where our observed features X =
Revised: We consider our Synthetic dataset with Causal distribution shift where our observed features X =

Original: ( X c , A cause ) .
Revised: ( X c , A cause ) .

Original: Note that by construction of X , one of our input dimensions already consists of
Revised: Note that by construction of X , one of our input dimensions already consists of

Original: A cause .
Revised: A cause .

Original: Hence, all baselines do receive information about A cause in addition to the domain attribute
Revised: Hence, all baselines do receive information about A cause in addition to the domain attribute

Original: E .
Revised: E .

Original: However, to provide a fairer comparison with CACM , we now additionally explicitly make A cause available to all DG algorithms for applying their respective constraints by creating domains based on
Revised: However, to provide a fairer comparison with CACM , we now additionally explicitly make A cause available to all DG algorithms for applying their respective constraints by creating domains based on

Original: A cause in our new setup.
Revised: A cause in our new setup.

Original: Using the same underlying data distribution, we group the data (i.e., create environments/domains) based on A cause i.e, each environment E has samples with same value of
Revised: Using the same underlying data distribution, we group the data (i.e., create environments/domains) based on A cause i.e, each environment E has samples with same value of

Original: A cause .
Revised: A cause .

Original: [Table]
Revised: [Table]

Original: In this setup (Table 7, third column)
Revised: In this setup (Table 7, third column)

Original: [Table]
Revised: [Table]

Original: we see IB-IRM, DANN
Revised: we see IB-IRM, DANN

Original: [Table]
Revised: [Table]

Original: and Mixup show significant improvement in accuracy but the best performance is still 14% lower than CACM .
Revised: and Mixup show significant improvement in accuracy but the best performance is still 14% lower than CACM .

Original: We additionally observe baselines to show higher estimate variance in this setup.
Revised: We additionally observe baselines to show higher estimate variance in this setup.

Original: This reinforces our motivation to use the causal graph of the data-generating process to derive the constraint, as the attribute values alone are not sufficient.
Revised: This reinforces our motivation to use the causal graph of the data-generating process to derive the constraint, as the attribute values alone are not sufficient.

Original: We also see MMD, CORAL, GroupDRO, and MLDG perform much worse than earlier, highlighting the sensitivity of DG algorithms to domain definition.
Revised: We also see MMD, CORAL, GroupDRO, and MLDG perform much worse than earlier, highlighting the sensitivity of DG algorithms to domain definition.

Original: In contrast, CACM uses the causal graph to study the structural relationships and derive the regularization penalty, which remains the same in this new dataset too.
Revised: In contrast, CACM uses the causal graph to study the structural relationships and derive the regularization penalty, which remains the same in this new dataset too.

Original: E.4. W ATERBIRDS
Revised: E.4. W ATERBIRDS

Original: We perform a similar analysis on the Waterbirds multi-attribute shift dataset.
Revised: We perform a similar analysis on the Waterbirds multi-attribute shift dataset.

Original: In order to provide the same information to other DG algorithms as CACM , we create domains based on A cause x
Revised: In order to provide the same information to other DG algorithms as CACM , we create domains based on A cause x

Original: A ind in this setup (Table 8).
Revised: A ind in this setup (Table 8).

Original: We observe mixed results – while some algorithms show significant improvement (ERM, IRM, VREx, MMD, MLDG, RSC), there is a performance drop for some others (IB-ERM, IB-IRM, CORAL, C-MMD, GroupDRO, Mixup).
Revised: We observe mixed results – while some algorithms show significant improvement (ERM, IRM, VREx, MMD, MLDG, RSC), there is a performance drop for some others (IB-ERM, IB-IRM, CORAL, C-MMD, GroupDRO, Mixup).

Original: CACM uses the knowledge of the causal relationships between attributes and the label and hence the evaluation remains the same.
Revised: CACM uses the knowledge of the causal relationships between attributes and the label and hence the evaluation remains the same.

Original: Hence, we empirically demonstrate the importance of using information of the causal graph in addition to the attributes.
Revised: Hence, we empirically demonstrate the importance of using information of the causal graph in addition to the attributes.

Original: F E − X c RELATIONSHIP
Revised: F E − X c RELATIONSHIP

Original: The E - X c edge shown in Figure 2b represents correlation of E with X c , which can change across environments.
Revised: The E - X c edge shown in Figure 2b represents correlation of E with X c , which can change across environments.

Original: As we saw for the Y - A ind edge, this correlation can be due to causal relationship (Figure 8a), confounding with a common cause (Figure 8b), or selection (Figure 8c); all our results (Proposition 3.1, Corollary 3.1) hold for any of these relationships.
Revised: As we saw for the Y - A ind edge, this correlation can be due to causal relationship (Figure 8a), confounding with a common cause (Figure 8b), or selection (Figure 8c); all our results (Proposition 3.1, Corollary 3.1) hold for any of these relationships.

Original: To see why, note that there is no collider introduced on X c or E in any of the above cases.
Revised: To see why, note that there is no collider introduced on X c or E in any of the above cases.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: Figure 9 shows causal graphs used for specifying multi-attribute distribution shifts in an anti-causal setting.
Revised: Figure 9 shows causal graphs used for specifying multi-attribute distribution shifts in an anti-causal setting.

Original: These graphs are identical to Figure 2, with the exception of change in direction of causal arrow from X c −→ Y to Y −→ X c .
Revised: These graphs are identical to Figure 2, with the exception of change in direction of causal arrow from X c −→ Y to Y −→ X c .

Original: We derive the (conditional) independence constraints for the anti-causal DAG for Independent ,
Revised: We derive the (conditional) independence constraints for the anti-causal DAG for Independent ,

Original: Causal , Confounded and Selected shifts.
Revised: Causal , Confounded and Selected shifts.

Original: Proposition G.1.
Revised: Proposition G.1.

Original: Given a causal DAG realized from the canonical graph in Figure 9a, the correct constraint depends on the relationship of label Y with the nuisance attributes A .
Revised: Given a causal DAG realized from the canonical graph in Figure 9a, the correct constraint depends on the relationship of label Y with the nuisance attributes A .

Original: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =
Revised: As shown, A can be split into A ind , A ind and E , where A ind can be further split into subsets that have a causal ( A cause ), confounded ( A conf ), selected ( A sel ) relationship with Y ( A ind =

Original: A cause ∪ A conf ∪ A sel ).
Revised: A cause ∪ A conf ∪ A sel ).

Original: Then, the (conditional) independence constraints that X c should satisfy are,
Revised: Then, the (conditional) independence constraints that X c should satisfy are,

Original: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.
Revised: Independent: X c ⊥⊥ A ind ; X c ⊥⊥ E ; X c ⊥⊥ A ind | Y ; X c ⊥⊥ A ind | E ; X c ⊥⊥ A ind | Y, E 2.

Original: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.
Revised: Causal: X c ⊥⊥ A cause | Y ; X c ⊥⊥ E ; X c ⊥⊥ A cause | Y, E 3.

Original: Confounded: X c ⊥⊥ A conf | Y ; X c ⊥⊥ E ; X c ⊥⊥ A conf | Y, E 4.
Revised: Confounded: X c ⊥⊥ A conf | Y ; X c ⊥⊥ E ; X c ⊥⊥ A conf | Y, E 4.

Original: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E
Revised: Selected: X c ⊥⊥ A sel | Y ; X c ⊥⊥ A sel | Y, E

Original: Proof.
Revised: Proof.

Original: The proof follows from d -separation using the same logic as earlier proof in Section B.2.
Revised: The proof follows from d -separation using the same logic as earlier proof in Section B.2.

Original: We observe that for all attributes A ∈ A ind ( A cause , A conf , A sel ), it is required to condition on Y to obtain valid constraints as Y node appears as a chain or fork in the causal graph but never as a collider due to the Y −→ X c causal arrow.
Revised: We observe that for all attributes A ∈ A ind ( A cause , A conf , A sel ), it is required to condition on Y to obtain valid constraints as Y node appears as a chain or fork in the causal graph but never as a collider due to the Y −→ X c causal arrow.

Original: Corollary G.1.
Revised: Corollary G.1.

Original: All the above derived constraints are valid for Graph 9a.
Revised: All the above derived constraints are valid for Graph 9a.

Original: However, in the presence of a correlation between E and X c (Graph 9b), only the constraints conditioned on E hold true.
Revised: However, in the presence of a correlation between E and X c (Graph 9b), only the constraints conditioned on E hold true.

Original: [Table]
Revised: [Table]