Original: CAGE: P ROBING C AUSAL R ELATIONSHIPS IN D EEP G ENERATIVE M ODELS
Revised: CAGE: P ROBING C AUSAL R ELATIONSHIPS IN D EEP G ENERATIVE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep generative models excel at generating complex, high-dimensional data, often exhibiting impressive generalization beyond the training distribution.
Revised: Deep generative models excel at generating complex, high-dimensional data, often exhibiting impressive generalization beyond the training distribution.

Original: The learning principle for these models is however purely based on statistical objectives and it is unclear to what extent such models have internalized the causal relationships present in the training data, if at all.
Revised: The learning principle for these models is however purely based on statistical objectives and it is unclear to what extent such models have internalized the causal relationships present in the training data, if at all.

Original: With increasing real-world deployments, such a causal understanding of generative models is essential for interpreting and controlling their use in high-stake applications that require synthetic data generation.
Revised: With increasing real-world deployments, such a causal understanding of generative models is essential for interpreting and controlling their use in high-stake applications that require synthetic data generation.

Original: We propose CAGE, a framework for inferring the cause-effect relationships governing deep generative models.
Revised: We propose CAGE, a framework for inferring the cause-effect relationships governing deep generative models.

Original: CAGE employs careful geometrical manipulations within the latent space of a generative model for generating counterfactuals and estimating unit-level generative causal effects.
Revised: CAGE employs careful geometrical manipulations within the latent space of a generative model for generating counterfactuals and estimating unit-level generative causal effects.

Original: CAGE does not require any modiﬁcations to the training procedure and can be used with any existing pretrained latent variable model.
Revised: CAGE does not require any modiﬁcations to the training procedure and can be used with any existing pretrained latent variable model.

Original: Moreover, the pretraining can be completely unsupervised and does not require any treatment or outcome labels.
Revised: Moreover, the pretraining can be completely unsupervised and does not require any treatment or outcome labels.

Original: Empirically, we demonstrate the use of CAGE for: (a) inferring cause-effect relationships within a deep generative model trained on both synthetic and high resolution images, and (b) guiding data augmentations for robust classiﬁcation where CAGE achieves improvements over current default approaches on image datasets.
Revised: Empirically, we demonstrate the use of CAGE for: (a) inferring cause-effect relationships within a deep generative model trained on both synthetic and high resolution images, and (b) guiding data augmentations for robust classiﬁcation where CAGE achieves improvements over current default approaches on image datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Causal models play a fundamental role in analyzing real-world datasets, as many of the questions underlying scientiﬁc research require reasoning beyond simply associations (Pearl, 2009; Rubin, 2005).
Revised: Causal models play a fundamental role in analyzing real-world datasets, as many of the questions underlying scientiﬁc research require reasoning beyond simply associations (Pearl, 2009; Rubin, 2005).

Original: Fundamentally, a causal model characterizes the generative process of the observed data via establishing cause-effect relationships between attributes; for example, a disease causes a symptom effect (and not vice versa).
Revised: Fundamentally, a causal model characterizes the generative process of the observed data via establishing cause-effect relationships between attributes; for example, a disease causes a symptom effect (and not vice versa).

Original: Consequently, a causal analysis is concerned with data generated and collected in the real world via natural processes and biological agents (Glymour et al., 2019).
Revised: Consequently, a causal analysis is concerned with data generated and collected in the real world via natural processes and biological agents (Glymour et al., 2019).

Original: In contrast, a major series of advancements in machine learning in the last decade concerns deep generative models that are learned via statistical objectives to artiﬁcially generate high-dimensional data.
Revised: In contrast, a major series of advancements in machine learning in the last decade concerns deep generative models that are learned via statistical objectives to artiﬁcially generate high-dimensional data.

Original: The widespread use of these models across the artiﬁcial intelligence spectrum, from vision (Karras et al., 2020) and language (Brown et al., 2020) to scientiﬁc discovery (SanchezLengeling & Aspuru-Guzik, 2018) and decision making (Chen et al., 2021), raises an important question: What are the causal mechanisms governing a deep generative model?
Revised: The widespread use of these models across the artiﬁcial intelligence spectrum, from vision (Karras et al., 2020) and language (Brown et al., 2020) to scientiﬁc discovery (SanchezLengeling & Aspuru-Guzik, 2018) and decision making (Chen et al., 2021), raises an important question: What are the causal mechanisms governing a deep generative model?

Original: With a statistical training objective, it is unclear to what extent (if any) the causal relationships from the observed dataset are retained within a generative model and consequently, the outputs expected from the model under interventions and counterfactuals.
Revised: With a statistical training objective, it is unclear to what extent (if any) the causal relationships from the observed dataset are retained within a generative model and consequently, the outputs expected from the model under interventions and counterfactuals.

Original: In addition to controlling and interpreting these black-box models, such a ﬁne-grained understanding can be paramount for formally characterizing their biases and generalization capabilities prior to high-stake deployments.
Revised: In addition to controlling and interpreting these black-box models, such a ﬁne-grained understanding can be paramount for formally characterizing their biases and generalization capabilities prior to high-stake deployments.

Original: Present Work .
Revised: Present Work .

Original: We propose CAGE, a framework for inferring implicit cause-effect relationships in the latent space of deep generative models.
Revised: We propose CAGE, a framework for inferring implicit cause-effect relationships in the latent space of deep generative models.

Original: Our framework builds off the potential outcomes framework of Neyman-Rubin causal model (Neyman, 1923; Rubin, 1974) and deﬁnes a new notion of generative average treatment effects (GATE).
Revised: Our framework builds off the potential outcomes framework of Neyman-Rubin causal model (Neyman, 1923; Rubin, 1974) and deﬁnes a new notion of generative average treatment effects (GATE).

Original: A fundamental problem of causal inference is that, by construction, we cannot observe the potential outcomes under all treatments (Holland, 1986) i.e., at any given time, any individual can be assigned only one treatment (a.k.a.
Revised: A fundamental problem of causal inference is that, by construction, we cannot observe the potential outcomes under all treatments (Holland, 1986) i.e., at any given time, any individual can be assigned only one treatment (a.k.a.

Original: the factual) but not both.
Revised: the factual) but not both.

Original: However, when studying treatment effects for deep generative models, CAGE exploits the generative nature of such models to overcome this challenge and explicitly generate the counterfactual.
Revised: However, when studying treatment effects for deep generative models, CAGE exploits the generative nature of such models to overcome this challenge and explicitly generate the counterfactual.

Original: In particular, our approach utilizes probabilistic classiﬁers for characterizing the axis of variations for relevant attributes in the latent space of a deep generative model.
Revised: In particular, our approach utilizes probabilistic classiﬁers for characterizing the axis of variations for relevant attributes in the latent space of a deep generative model.

Original: Given the axis for the treatment attribute classiﬁer, we develop a geometric manipulation scheme for simulating the assignment of speciﬁc treatments (in particular, the counterfactual) to an individual.
Revised: Given the axis for the treatment attribute classiﬁer, we develop a geometric manipulation scheme for simulating the assignment of speciﬁc treatments (in particular, the counterfactual) to an individual.

Original: Thereafter, we use an outcome attribute classiﬁer to quantify the difference in outcomes for the factual and counterfactual generations and thus, estimate GATE.
Revised: Thereafter, we use an outcome attribute classiﬁer to quantify the difference in outcomes for the factual and counterfactual generations and thus, estimate GATE.

Original: We refer to our overall framework for causal probing of deep generative models as CAGE.
Revised: We refer to our overall framework for causal probing of deep generative models as CAGE.

Original: Figure 1 shows illustrative samples generated from our approach.
Revised: Figure 1 shows illustrative samples generated from our approach.

Original: While there is a body of existing work on enforcing a causal structure on deep generative models e.g., (Kocaoglu et al., 2017; Yang et al., 2021; Suter et al., 2019) or using generative models as tools to aid causal inference e.g., (Louizos et al., 2017; Yoon et al., 2018; Sauer & Geiger, 2021), our work stands out in extracting the causal structure inherent in existing generative models.
Revised: While there is a body of existing work on enforcing a causal structure on deep generative models e.g., (Kocaoglu et al., 2017; Yang et al., 2021; Suter et al., 2019) or using generative models as tools to aid causal inference e.g., (Louizos et al., 2017; Yoon et al., 2018; Sauer & Geiger, 2021), our work stands out in extracting the causal structure inherent in existing generative models.

Original: As such,
Revised: As such,

Original: CAGE applies readily to any pretrained latent variable model, without any changes to their training procedure that can introduce any undesirable trade-offs in learning (Locatello et al., 2019).
Revised: CAGE applies readily to any pretrained latent variable model, without any changes to their training procedure that can introduce any undesirable trade-offs in learning (Locatello et al., 2019).

Original: We further leverage the treatment effects estimated via CAGE to deﬁne a natural measure of causal direction over a pair of variables associated with a deep generative model.
Revised: We further leverage the treatment effects estimated via CAGE to deﬁne a natural measure of causal direction over a pair of variables associated with a deep generative model.

Original: Our score is simply deﬁned as the difference in magnitudes of GATE when considering each variable as a treatment .
Revised: Our score is simply deﬁned as the difference in magnitudes of GATE when considering each variable as a treatment .

Original: The intuition behind such a score is that the ATE when considering the effect as a treatment should be approximately zero.
Revised: The intuition behind such a score is that the ATE when considering the effect as a treatment should be approximately zero.

Original: In contrast, we expect the magnitude of the ATE when considering the cause as a treatment to be large.
Revised: In contrast, we expect the magnitude of the ATE when considering the cause as a treatment to be large.

Original: We study the consistency of the inferred causal directions for two highdimensional image datasets, MorphoMNIST (Pawlowski et al., 2020) and CelebaHQ (Karras et al., 2017), with known or biologically guessed prior cause-effect relationships.
Revised: We study the consistency of the inferred causal directions for two highdimensional image datasets, MorphoMNIST (Pawlowski et al., 2020) and CelebaHQ (Karras et al., 2017), with known or biologically guessed prior cause-effect relationships.

Original: Finally, we use these inferred causal directions to guide counterfactual generation for data augmentations.
Revised: Finally, we use these inferred causal directions to guide counterfactual generation for data augmentations.

Original: Data augmentations from generative models are immensely useful for dealing with class and subgroup imbalance (Ramaswamy et al., 2021;
Revised: Data augmentations from generative models are immensely useful for dealing with class and subgroup imbalance (Ramaswamy et al., 2021;

Original: Xu et al., 2018; Sattigeri et al., 2018).
Revised: Xu et al., 2018; Sattigeri et al., 2018).

Original: We show that augmentations generated by respecting the causal directions inferred via CAGE improve classiﬁcation accuracy by over 2.7% over the baseline and 11 .
Revised: We show that augmentations generated by respecting the causal directions inferred via CAGE improve classiﬁcation accuracy by over 2.7% over the baseline and 11 .

Original: 7% over the anti causal direction on the
Revised: 7% over the anti causal direction on the

Original: CelebaHQ dataset for the worst subgroup in the challenging group DRO task.
Revised: CelebaHQ dataset for the worst subgroup in the challenging group DRO task.

Original: Our main contributions are summarized as follows
Revised: Our main contributions are summarized as follows

Original: We introduce the notion of generative average treatment effects (GATE) for probing causeeffect relationships in the latent space of deep generative models.
Revised: We introduce the notion of generative average treatment effects (GATE) for probing causeeffect relationships in the latent space of deep generative models.

Original: We propose CAGE, a GATE estimation framework for deep generative models based on latent counterfactual manipulations.
Revised: We propose CAGE, a GATE estimation framework for deep generative models based on latent counterfactual manipulations.

Original: Finally, as a practical application, we use the ATEs to infer causal direction over pairs of attributes within deep generative models and employ CAGE for generating counterfactual data augmentations with an application to distributionally robust optimization.
Revised: Finally, as a practical application, we use the ATEs to infer causal direction over pairs of attributes within deep generative models and employ CAGE for generating counterfactual data augmentations with an application to distributionally robust optimization.

Original:  B ACKGROUND AND P RELIMINARIES 2. T HE POTENTIAL OUTCOMES FRAMEWORK
Revised:  B ACKGROUND AND P RELIMINARIES 2. T HE POTENTIAL OUTCOMES FRAMEWORK

Original: We brieﬂy overview the potential outcomes framework of Neyman (1923) and Rubin (1974), upon which we base our proposed method in the next section.
Revised: We brieﬂy overview the potential outcomes framework of Neyman (1923) and Rubin (1974), upon which we base our proposed method in the next section.

Original: In its simplest form, this framework considers the causal effect of assigning a binary treatment, T ∈ { 0 , 1 } .
Revised: In its simplest form, this framework considers the causal effect of assigning a binary treatment, T ∈ { 0 , 1 } .

Original: Such a framework posits the existence of potential outcomes for the i th individual both whilst receiving treatment, Y T =1 ( i ) , and when treatment is withheld, Y T =

Revised: Such a framework posits the existence of potential outcomes for the i th individual both whilst receiving treatment, Y T =1 ( i ) , and when treatment is withheld, Y T =

Original: ( i ) .
Revised: ( i ) .

Original: The causal effect for a given individual, i , is deﬁned as the difference in these potential outcomes.
Revised: The causal effect for a given individual, i , is deﬁned as the difference in these potential outcomes.

Original: The “fundamental problem of causal inference” is that whilst we may posit the existence of both potential outcomes, Y T =

Revised: The “fundamental problem of causal inference” is that whilst we may posit the existence of both potential outcomes, Y T =

Original: ( i ) and Y T
Revised: ( i ) and Y T

Original: =

Revised: =

Original: ( i ) , we only ever observe the outcome under one treatment (Holland, 1986).
Revised: ( i ) , we only ever observe the outcome under one treatment (Holland, 1986).

Original: For this reason, causal inference is often performed at the population level, by considering quantities such as the average treatment effect (ATE):
Revised: For this reason, causal inference is often performed at the population level, by considering quantities such as the average treatment effect (ATE):

Original: [Equation]
Revised: [Equation]

Original: Equation (1) is the expected difference in potential outcomes of individuals receiving treatment
Revised: Equation (1) is the expected difference in potential outcomes of individuals receiving treatment

Original: T = 1 and T = 0 .
Revised: T = 1 and T = 0 .

Original: Under standard assumptions of unconfoundedness, positivity, and stable unit treatment value assumption, the ATE is identiﬁable and can be estimated by a statistical estimate of the associational difference in expectations (Rubin, 1978).
Revised: Under standard assumptions of unconfoundedness, positivity, and stable unit treatment value assumption, the ATE is identiﬁable and can be estimated by a statistical estimate of the associational difference in expectations (Rubin, 1978).

Original: 2. D EEP LATENT VARIABLE GENERATIVE MODELS
Revised: 2. D EEP LATENT VARIABLE GENERATIVE MODELS

Original: The goal of generative modeling is to learn an underlying data distribution given a training dataset.
Revised: The goal of generative modeling is to learn an underlying data distribution given a training dataset.

Original: For high-dimensional data, this can be done effectively via deep neural networks by learning a mapping G from unobserved factors of variation Z (i.e., latent variables) to observed variables
Revised: For high-dimensional data, this can be done effectively via deep neural networks by learning a mapping G from unobserved factors of variation Z (i.e., latent variables) to observed variables

Original: X (e.g., raw pixels).
Revised: X (e.g., raw pixels).

Original: The mapping G is parameterized using deep neural networks, and can be learned via a variety of training objectives, such as adversarial training (e.g., generative adversarial networks (GAN; Goodfellow et al.
Revised: The mapping G is parameterized using deep neural networks, and can be learned via a variety of training objectives, such as adversarial training (e.g., generative adversarial networks (GAN; Goodfellow et al.

Original: (2014)), exact maximum likelihood estimation (e.g., Normalizing
Revised: (2014)), exact maximum likelihood estimation (e.g., Normalizing

Original: Flows (Dinh et al., 2017)), and variational Bayes (e.g., Variational Autoencoders (VAE; Kingma &
Revised: Flows (Dinh et al., 2017)), and variational Bayes (e.g., Variational Autoencoders (VAE; Kingma &

Original: Welling (2013); Rezende & Mohamed (2015)).
Revised: Welling (2013); Rezende & Mohamed (2015)).

Original: For our current work, we can use any such latent variable generative model.
Revised: For our current work, we can use any such latent variable generative model.

Original: Our experiments will exhibit this generality by considering state-of-theart ﬂows (Papamakarios et al., 2017; Song et al., 2019) and GANs (Karras et al., 2020).
Revised: Our experiments will exhibit this generality by considering state-of-theart ﬂows (Papamakarios et al., 2017; Song et al., 2019) and GANs (Karras et al., 2020).

Original: In addition to G (i.e., the decoder), we will assume the existence of an encoder E NC : X → Z that can map any observed datapoint to its latent vector representation.
Revised: In addition to G (i.e., the decoder), we will assume the existence of an encoder E NC : X → Z that can map any observed datapoint to its latent vector representation.

Original: For instance, for ﬂows, the encoder is given by the inverse of the generator mapping, E NC =
Revised: For instance, for ﬂows, the encoder is given by the inverse of the generator mapping, E NC =

Original: G − 1 .
Revised: G − 1 .

Original: For VAEs, the encoder is the variational posterior distribution trained alongside G .
Revised: For VAEs, the encoder is the variational posterior distribution trained alongside G .

Original: For GANs, either the encoder can be trained during training (e.g., Donahue et al.
Revised: For GANs, either the encoder can be trained during training (e.g., Donahue et al.

Original: ; Dumoulin et al.
Revised: ; Dumoulin et al.

Original: (2016)), after training (e.g., Richardson et al.
Revised: (2016)), after training (e.g., Richardson et al.

Original: (2021)), or implicitly learned by backpropogating through G to ﬁnd a latent vector z ∈ Z that best represents a target input x , as measured via a reconstruction loss (e.g., Lipton & Tripathi (2017);
Revised: (2021)), or implicitly learned by backpropogating through G to ﬁnd a latent vector z ∈ Z that best represents a target input x , as measured via a reconstruction loss (e.g., Lipton & Tripathi (2017);

Original: Bora et al.
Revised: Bora et al.

Original: (2017)) or a pretrained domain classiﬁer (e.g., (Karras et al., 2020)).
Revised: (2017)) or a pretrained domain classiﬁer (e.g., (Karras et al., 2020)).

Original:  E STIMATING C AUSAL E FFECTS IN D EEP G ENERATIVE M ODELS
Revised:  E STIMATING C AUSAL E FFECTS IN D EEP G ENERATIVE M ODELS

Original: Our goal is to interrogate a deep generative model with a view to better understanding which—if any—causal associations have been implicitly encoded.
Revised: Our goal is to interrogate a deep generative model with a view to better understanding which—if any—causal associations have been implicitly encoded.

Original: We therefore look to answer the question:
Revised: We therefore look to answer the question:

Original: In the context of a generator, G , does considering the counterfactual over a given “causal” attribute (e.g., gender) lead to signiﬁcant changes in a “effect” attribute (e.g., hair color)?
Revised: In the context of a generator, G , does considering the counterfactual over a given “causal” attribute (e.g., gender) lead to signiﬁcant changes in a “effect” attribute (e.g., hair color)?

Original: Formally, we assume whitebox access to a pretrained latent variable deep generative model
Revised: Formally, we assume whitebox access to a pretrained latent variable deep generative model

Original: G : Z → X .
Revised: G : Z → X .

Original: In addition, we further assume a ﬁnite dataset of n annotated observations,
Revised: In addition, we further assume a ﬁnite dataset of n annotated observations,

Original: D =
Revised: D =

Original: (cid:83) n i =1 { x D ( i ) , m 1 ( i ) , m 2 ( i ) } .
Revised: (cid:83) n i =1 { x D ( i ) , m 1 ( i ) , m 2 ( i ) } .

Original: Here, the examples { x D (1) . . .
Revised: Here, the examples { x D (1) . . .

Original: , x D ( N ) } are drawn from the model’s training distribution and we annotate each example
Revised: , x D ( N ) } are drawn from the model’s training distribution and we annotate each example

Original: x D ( i ) with binary metadata for two variables m 1 ( i ) ∈ { 0 , 1 } and m 2 ( i ) ∈ { 0 , 1 } .
Revised: x D ( i ) with binary metadata for two variables m 1 ( i ) ∈ { 0 , 1 } and m 2 ( i ) ∈ { 0 , 1 } .

Original: These two annotated variables can be drawn from a larger unobserved (w.r.t.
Revised: These two annotated variables can be drawn from a larger unobserved (w.r.t.

Original: G ) causal process.
Revised: G ) causal process.

Original: For example, x D ( i ) could correspond to a highdimensional image and m 1 ( i ) and m 2 ( i ) could be any two attributes of the image, such as gender expression and hair color in Figure 1.
Revised: For example, x D ( i ) could correspond to a highdimensional image and m 1 ( i ) and m 2 ( i ) could be any two attributes of the image, such as gender expression and hair color in Figure 1.

Original: Apriori, we do not know which of the two variables (if any) is cause or effect but through D and G , we seek to identify the nature of their relationship within G .
Revised: Apriori, we do not know which of the two variables (if any) is cause or effect but through D and G , we seek to identify the nature of their relationship within G .

Original: We note that any causal conclusions obtained under our proposed framework only reﬂect implicit properties of a generator G , as opposed to properties of the data on which the generator was trained.
Revised: We note that any causal conclusions obtained under our proposed framework only reﬂect implicit properties of a generator G , as opposed to properties of the data on which the generator was trained.

Original: That is to say, it is perfectly permitted, for G to have learned a causal relationship that is inconsistent with the observational data used for training.
Revised: That is to say, it is perfectly permitted, for G to have learned a causal relationship that is inconsistent with the observational data used for training.

Original: As we shall see in §4, shedding light on the implicit causal relationship learned by G is abundantly useful for pinpointing shortcomings in the model and informing best practices for counterfactual generations.
Revised: As we shall see in §4, shedding light on the implicit causal relationship learned by G is abundantly useful for pinpointing shortcomings in the model and informing best practices for counterfactual generations.

Original: As a result, this work—by design— is in sharp contrast with the majority of causal inference literature that focuses on identifying the generative mechanisms of a ﬁxed dataset as opposed to our focus on the generative model itself.
Revised: As a result, this work—by design— is in sharp contrast with the majority of causal inference literature that focuses on identifying the generative mechanisms of a ﬁxed dataset as opposed to our focus on the generative model itself.

Original: 3. O VERVIEW OF THE CAGE FRAMEWORK
Revised: 3. O VERVIEW OF THE CAGE FRAMEWORK

Original: For answering the above question, we propose to extend the potential outcomes framework described in §2.1 to accommodate latent variable deep generative models.
Revised: For answering the above question, we propose to extend the potential outcomes framework described in §2.1 to accommodate latent variable deep generative models.

Original: Our approach is premised on computing a generative average treatment effect for a candidate causal attribute, m c , on a given effect attribute, m e , for a given generator, G .
Revised: Our approach is premised on computing a generative average treatment effect for a candidate causal attribute, m c , on a given effect attribute, m e , for a given generator, G .

Original: Unlike traditional approaches to treatment effect estimation, the key observation in CAGE is that we can simulate the counterfactuals under a given treatment.
Revised: Unlike traditional approaches to treatment effect estimation, the key observation in CAGE is that we can simulate the counterfactuals under a given treatment.

Original: We refer to our overall framework for ca usal probing of generative models as CAGE.
Revised: We refer to our overall framework for ca usal probing of generative models as CAGE.

Original: For example, in order to investigate the cause-effect relationship between gender and hair color, let us initially arbitrarily assign gender to be the treatment m 

Revised: For example, in order to investigate the cause-effect relationship between gender and hair color, let us initially arbitrarily assign gender to be the treatment m

Original: = m c (cause) and hair color to be the outcome m 

Revised: = m c (cause) and hair color to be the outcome m

Original: = m e (effect).
Revised: = m e (effect).

Original: We note this choice is arbitrary and the overall algorithm will also consider the reverse assignment (i.e., m 

Revised: We note this choice is arbitrary and the overall algorithm will also consider the reverse assignment (i.e., m

Original: = m e and m 

Revised: = m e and m

Original: = m c ) and if needed, also reveal independence between the variables.
Revised: = m c ) and if needed, also reveal independence between the variables.

Original: Then we may simulate the effects of treatment assignments by manipulating the gender attribute in the latent space and measuring the corresponding change in outcomes.
Revised: Then we may simulate the effects of treatment assignments by manipulating the gender attribute in the latent space and measuring the corresponding change in outcomes.

Original: We next discuss our strategies for counterfactual manipulation and outcome quantiﬁcation.
Revised: We next discuss our strategies for counterfactual manipulation and outcome quantiﬁcation.

Original: 3. S IMULATING T REATMENTS VIA L ATENT C OUNTERFACTUAL M ANIPULATIONS
Revised: 3. S IMULATING T REATMENTS VIA L ATENT C OUNTERFACTUAL M ANIPULATIONS

Original: [Equation]
Revised: [Equation]

Original: TreatmentHyperplane
Revised: TreatmentHyperplane

Original: Outcome Hyperplane
Revised: Outcome Hyperplane

Original: In order to simulate the assignment of counterfactuals, we follow a three-step procedure.
Revised: In order to simulate the assignment of counterfactuals, we follow a three-step procedure.

Original: Step 1: Our ﬁrst step is to train a latent space classiﬁer for the attributes using D .
Revised: Step 1: Our ﬁrst step is to train a latent space classiﬁer for the attributes using D .

Original: Here, we use the encoder to ﬁrst project every input x D ( i ) in D to its latent encoding z D ( i ) =
Revised: Here, we use the encoder to ﬁrst project every input x D ( i ) in D to its latent encoding z D ( i ) =

Original: E NC ( x D ( i )) .
Revised: E NC ( x D ( i )) .

Original: Given the latent encodings z D ( i ) and their annotations for m c and m e attributes, we train a (probabilistic) linear classiﬁer, φ c : Z → [0 , 1] , to discriminate between binary causal attribute values given latent representations.
Revised: Given the latent encodings z D ( i ) and their annotations for m c and m e attributes, we train a (probabilistic) linear classiﬁer, φ c : Z → [0 , 1] , to discriminate between binary causal attribute values given latent representations.

Original: Concretely, by restricting ourselves to linear classiﬁers we obtain a unit hyperplane that encodes the classiﬁcation boundary.
Revised: Concretely, by restricting ourselves to linear classiﬁers we obtain a unit hyperplane that encodes the classiﬁcation boundary.

Original: We use h c to denote the normal vector to the hyperplane for attribute m c .
Revised: We use h c to denote the normal vector to the hyperplane for attribute m c .

Original: Step 2: Since we are interested in casual discovery for the generative model, in the second step, we use G to create a dataset of annotated examples by sampling k latent vectors
Revised: Step 2: Since we are interested in casual discovery for the generative model, in the second step, we use G to create a dataset of annotated examples by sampling k latent vectors

Original: Z = (cid:83) ki
Revised: Z = (cid:83) ki

Original: =1 { z ( i ) } from the prior of the generative model.
Revised: =1 { z ( i ) } from the prior of the generative model.

Original: Let X =
Revised: Let X =

Original: (cid:83) ki =1 { x ( i ) =
Revised: (cid:83) ki =1 { x ( i ) =

Original: G ( z ( i )) } denote the corresponding generations.
Revised: G ( z ( i )) } denote the corresponding generations.

Original: Since sampling latents from the prior generative model are typically inexpensive, our dataset size can be fairly large.
Revised: Since sampling latents from the prior generative model are typically inexpensive, our dataset size can be fairly large.

Original: For each generated example x ( i ) ∈ X , we obtain its factual treatment by simply using the latent space classiﬁer as m c ( i ) =
Revised: For each generated example x ( i ) ∈ X , we obtain its factual treatment by simply using the latent space classiﬁer as m c ( i ) =

Original: [ φ c ( z ( i )) > 0] .
Revised: [ φ c ( z ( i )) > 0] .

Original: Step 3:
Revised: Step 3:

Original: For a given generated sample, x ( i ) ∈ X , assume without loss of generality, that m c
Revised: For a given generated sample, x ( i ) ∈ X , assume without loss of generality, that m c

Original: = 0 is the factual treatment obtained via Step 2.
Revised: = 0 is the factual treatment obtained via Step 2.

Original: We can derive analogous expressions for factual treatments m c = 1 , but we skip those for brevity.
Revised: We can derive analogous expressions for factual treatments m c = 1 , but we skip those for brevity.

Original: Finally, we deﬁne the counterfactual latent with respect to setting the treatment attribute m c = 1 as using the do operator as:
Revised: Finally, we deﬁne the counterfactual latent with respect to setting the treatment attribute m c = 1 as using the do operator as:

Original: [Equation]
Revised: [Equation]

Original: where α ∈ R + is a positive scalar that controls the extent to which we manipulate.
Revised: where α ∈ R + is a positive scalar that controls the extent to which we manipulate.

Original: Equation (3.2) corresponds to moving linearly along the hyperplane normal, h c , which encodes attribute m c for simulating the counterfactual.
Revised: Equation (3.2) corresponds to moving linearly along the hyperplane normal, h c , which encodes attribute m c for simulating the counterfactual.

Original: Such an approach is premised on the assumption that Z is linearly separable with respect to a semantically meaningful latent attribute, which was ﬁrst observed and empirically validated for GANs (Denton et al., 2019).
Revised: Such an approach is premised on the assumption that Z is linearly separable with respect to a semantically meaningful latent attribute, which was ﬁrst observed and empirically validated for GANs (Denton et al., 2019).

Original: A counterfactual sample can subsequently be obtained by simply pushing forward the counterfactual latent through G :
Revised: A counterfactual sample can subsequently be obtained by simply pushing forward the counterfactual latent through G :

Original: [Equation]
Revised: [Equation]

Original: Figure 2 shows an illustration of our counterfactual manipulation strategy.
Revised: Figure 2 shows an illustration of our counterfactual manipulation strategy.

Original: 3. Q UANTIFYING T REATMENT E FFECTS VIA P ROXY C LASSIFIERS
Revised: 3. Q UANTIFYING T REATMENT E FFECTS VIA P ROXY C LASSIFIERS

Original: Equations (3.2-2) deﬁne the geometric manipulations required to obtain interventional samples x
Revised: Equations (3.2-2) deﬁne the geometric manipulations required to obtain interventional samples x

Original: ( i ; do ( m c = 1)) from a generative model, G .
Revised: ( i ; do ( m c = 1)) from a generative model, G .

Original: However, in order to estimate treatment effects, we require an estimate of the presence or absence of the effect attribute, m e , for the counterfactuals.
Revised: However, in order to estimate treatment effects, we require an estimate of the presence or absence of the effect attribute, m e , for the counterfactuals.

Original: To this end, we propose the use of binary classiﬁers trained to detect the presence of an effect variable.
Revised: To this end, we propose the use of binary classiﬁers trained to detect the presence of an effect variable.

Original: These classiﬁers can be trained directly in the latent space and serve to measure the effect of intervening on the treatment attribute m c for an individual x ( i ) be quantiﬁed as:
Revised: These classiﬁers can be trained directly in the latent space and serve to measure the effect of intervening on the treatment attribute m c for an individual x ( i ) be quantiﬁed as:

Original: [Equation]
Revised: [Equation]

Original: Alternatively, we can train another probabilistic classiﬁer ψ e : X → [0 , 1] that classiﬁes the counterfactual generations in Equation (2).
Revised: Alternatively, we can train another probabilistic classiﬁer ψ e : X → [0 , 1] that classiﬁes the counterfactual generations in Equation (2).

Original: In this manner, we can then quantify the effect of intervening on the treatment attribute m c for an individual x ( i ) as:
Revised: In this manner, we can then quantify the effect of intervening on the treatment attribute m c for an individual x ( i ) as:

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we use latent space classiﬁers for quantifying treatment effects, as in Equation (3).
Revised: For simplicity, we use latent space classiﬁers for quantifying treatment effects, as in Equation (3).

Original: We can deﬁne the generative individual treatment effect (GITE) estimate τ GITE as:
Revised: We can deﬁne the generative individual treatment effect (GITE) estimate τ GITE as:

Original: [Equation]
Revised: [Equation]

Original: Taking an expectation over the generated examples, we get an estimate for the generative average treatment effect (GATE) of m c on m e as:
Revised: Taking an expectation over the generated examples, we get an estimate for the generative average treatment effect (GATE) of m c on m e as:

Original: [Equation]
Revised: [Equation]

Original: As we can see from Equation (6), the correctness of our estimate relies on a number of assumptions.
Revised: As we can see from Equation (6), the correctness of our estimate relies on a number of assumptions.

Original: In addition to standard assumptions from causality (unconfoundedness, positivity, SUTVA), the quality of the generative model and classiﬁers play an important role.
Revised: In addition to standard assumptions from causality (unconfoundedness, positivity, SUTVA), the quality of the generative model and classiﬁers play an important role.

Original: For the generative model, we designed our counterfactual manipulation scheme assuming that the latent space is linearly separable in the attributes of interest.
Revised: For the generative model, we designed our counterfactual manipulation scheme assuming that the latent space is linearly separable in the attributes of interest.

Original: Further, implicit in our framework is the assumption that the generative model can generalize outside the training set to include the support of the counterfactual distribution.
Revised: Further, implicit in our framework is the assumption that the generative model can generalize outside the training set to include the support of the counterfactual distribution.

Original: On the use of classiﬁers, we need accurate and calibrated classiﬁers (ideally, Bayes optimal) for both manipulating latent vectors and quantifying treatment effects.
Revised: On the use of classiﬁers, we need accurate and calibrated classiﬁers (ideally, Bayes optimal) for both manipulating latent vectors and quantifying treatment effects.

Original: While it is impossible to test in—full veracity— the above assumptions on real-world distributions, there is signiﬁcant empirical evidence in the last few years that suggest that modern deep generative models and classiﬁers can indeed satisfy the above requirements for many practical usecases (Denton et al., 2019; Brown et al., 2020).
Revised: While it is impossible to test in—full veracity— the above assumptions on real-world distributions, there is signiﬁcant empirical evidence in the last few years that suggest that modern deep generative models and classiﬁers can indeed satisfy the above requirements for many practical usecases (Denton et al., 2019; Brown et al., 2020).

Original: The theoretical properties and these necessary assumptions of CAGE are further discussed in §A.
Revised: The theoretical properties and these necessary assumptions of CAGE are further discussed in §A.

Original: ∆ τ as a measure of causal direction.
Revised: ∆ τ as a measure of causal direction.

Original: Finally, we note that the preceding sections have focused on the challenge of quantifying if a variable, m c , has a causal effect on second variable, m e .
Revised: Finally, we note that the preceding sections have focused on the challenge of quantifying if a variable, m c , has a causal effect on second variable, m e .

Original: This assumes knowledge of the causal ordering over variables and maybe considered a special case of the more general causal discovery problem.
Revised: This assumes knowledge of the causal ordering over variables and maybe considered a special case of the more general causal discovery problem.

Original: As such, we can extend our approach to deﬁne a measure of causal direction over any pair of variables by considering the difference in absolute generative
Revised: As such, we can extend our approach to deﬁne a measure of causal direction over any pair of variables by considering the difference in absolute generative

Original: ATE scores when either variable is considered as the treatment:
Revised: ATE scores when either variable is considered as the treatment:

Original: [Equation]
Revised: [Equation]

Original: The ∆ τ score determines the magnitude of the difference in outcomes when either attribute is prescribed as the treatment.
Revised: The ∆ τ score determines the magnitude of the difference in outcomes when either attribute is prescribed as the treatment.

Original: Intuitively, when using the true causal attribute, according to G , as the
Revised: Intuitively, when using the true causal attribute, according to G , as the

Original: [Figure]
Revised: [Figure]

Original: treatment we expect a signature of the interventions to be large in magnitude.
Revised: treatment we expect a signature of the interventions to be large in magnitude.

Original: Conversely, the magnitude of disturbance measured when manipulating the true effect variable should be markedly smaller.
Revised: Conversely, the magnitude of disturbance measured when manipulating the true effect variable should be markedly smaller.

Original: Thus ∆ τ > 0 corroborates that the chosen causal ordering is indeed the correct one as supported by G , while a ∆ τ < 0 implies that the chosen causal ordering may in fact be reversed.
Revised: Thus ∆ τ > 0 corroborates that the chosen causal ordering is indeed the correct one as supported by G , while a ∆ τ < 0 implies that the chosen causal ordering may in fact be reversed.

Original: A null distribution for treatment effects.
Revised: A null distribution for treatment effects.

Original: A further important consideration relates to how we might determine whether an estimated GATE is statistically signiﬁcant.
Revised: A further important consideration relates to how we might determine whether an estimated GATE is statistically signiﬁcant.

Original: To address this, we can obtain an empirical sample of GATE scores under the null hypothesis where the attribute m c has no causal association with m e .
Revised: To address this, we can obtain an empirical sample of GATE scores under the null hypothesis where the attribute m c has no causal association with m e .

Original: A simple manner through which this can be obtained is via randomization and permutation testing.
Revised: A simple manner through which this can be obtained is via randomization and permutation testing.

Original: In particular, we can randomly shufﬂe the values of a causal attribute, m c , thereby removing any potential causal association.
Revised: In particular, we can randomly shufﬂe the values of a causal attribute, m c , thereby removing any potential causal association.

Original: This corresponds to performing interventions over the latent space of G which are effectively random projections.
Revised: This corresponds to performing interventions over the latent space of G which are effectively random projections.

Original: This process is repeated many times to obtain an empirical distribution for a GATE under the null hypothesis.
Revised: This process is repeated many times to obtain an empirical distribution for a GATE under the null hypothesis.

Original: Given an empirical distribution of GATEs under the null, we can obtain a p -value for our observed GATE.
Revised: Given an empirical distribution of GATEs under the null, we can obtain a p -value for our observed GATE.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the ability of CAGE towards inferring causal relationships in pretrained deep generative models on both synthetic and high-dimensional datasets such as high-resolution images of faces.
Revised: We evaluate the ability of CAGE towards inferring causal relationships in pretrained deep generative models on both synthetic and high-dimensional datasets such as high-resolution images of faces.

Original: 4. S YNTHETIC M IXTURE OF G AUSSIANS
Revised: 4. S YNTHETIC M IXTURE OF G AUSSIANS

Original: We consider the following toy setup: data is generated according to two distinct mixture distributions as shown in the left panel of Figure 3 where each color denotes a mixture.
Revised: We consider the following toy setup: data is generated according to two distinct mixture distributions as shown in the left panel of Figure 3 where each color denotes a mixture.

Original: We deﬁne the causal attribute, m c , to be which mixture each sample is drawn from (i.e., from the mixture of 8 Gaussians, in blue, or the mixture of 3 Gaussians, in orange).
Revised: We deﬁne the causal attribute, m c , to be which mixture each sample is drawn from (i.e., from the mixture of 8 Gaussians, in blue, or the mixture of 3 Gaussians, in orange).

Original: We further deﬁne an effect variable, m e , to be deﬁned as one, if the mean of the ( x, y ) -coordinates is less than 2 . 5 and zero otherwise.
Revised: We further deﬁne an effect variable, m e , to be deﬁned as one, if the mean of the ( x, y ) -coordinates is less than 2 . 5 and zero otherwise.

Original: In this manner, we can see that m c has a causal inﬂuence over m e by determining which mixture each sample is drawn from.
Revised: In this manner, we can see that m c has a causal inﬂuence over m e by determining which mixture each sample is drawn from.

Original: We employ a Masked Autoregressive Flow (Papamakarios et al., 2017) as the deep generative model.
Revised: We employ a Masked Autoregressive Flow (Papamakarios et al., 2017) as the deep generative model.

Original: The middle panel of Figure 3 visualises the distribution of τ GATE under the null as well as the estimated GATE, τ GATE ( m c → m e ) , which is signiﬁcantly larger in magnitude.
Revised: The middle panel of Figure 3 visualises the distribution of τ GATE under the null as well as the estimated GATE, τ GATE ( m c → m e ) , which is signiﬁcantly larger in magnitude.

Original: Finally, the right panel plots ∆ τ as a function of α , positive values corrobrate that m c → m e .
Revised: Finally, the right panel plots ∆ τ as a function of α , positive values corrobrate that m c → m e .

Original: 4. I MAGE D ATASETS
Revised: 4. I MAGE D ATASETS

Original: Morphomnist .
Revised: Morphomnist .

Original: We evaluate the causal relationships learned by powerful deep normalizing ﬂow models on a synthetic dataset based on MNIST dubbed MorphoMNIST as introduced by Pawlowski
Revised: We evaluate the causal relationships learned by powerful deep normalizing ﬂow models on a synthetic dataset based on MNIST dubbed MorphoMNIST as introduced by Pawlowski

Original: [Table]
Revised: [Table]

Original: et al.
Revised: et al.

Original: Here, the original MNIST digits are modiﬁed to respect a causal structure whereby the stroke thickness of the digits is a cause to the brightness (i.e. T −→ I ).
Revised: Here, the original MNIST digits are modiﬁed to respect a causal structure whereby the stroke thickness of the digits is a cause to the brightness (i.e. T −→ I ).

Original: Speciﬁcally, thicker digits are brighter while thinner digits are dimmer under the prescribed causal graph.
Revised: Speciﬁcally, thicker digits are brighter while thinner digits are dimmer under the prescribed causal graph.

Original: We train a powerful normalizing ﬂow in Mintnet (Song et al., 2019) and interrogate the direction of causality—if any— between thickness and intensity by ﬁrst projecting all test set samples to their corresponding latent vectors.
Revised: We train a powerful normalizing ﬂow in Mintnet (Song et al., 2019) and interrogate the direction of causality—if any— between thickness and intensity by ﬁrst projecting all test set samples to their corresponding latent vectors.

Original: Fig 4 shows qualitative examples of counterfactuals as well a ∆ τ curve as a function of α .
Revised: Fig 4 shows qualitative examples of counterfactuals as well a ∆ τ curve as a function of α .

Original: Here the generated counterfactuals reduce the thickness while maintaining the average intensity.
Revised: Here the generated counterfactuals reduce the thickness while maintaining the average intensity.

Original: As observed, the model generates digits that contain holes but to compensate for the drop in intensity
Revised: As observed, the model generates digits that contain holes but to compensate for the drop in intensity

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: the model instead increases the intensity of the background pixels.
Revised: the model instead increases the intensity of the background pixels.

Original: Such generated samples are in line with expectations as we notice that the ∆ τ score indicates that the model has learned that thickness causes intensity and as such manipulating the causal variable propagates inﬂuence to the effect variable which manifests itself as an increase in background intensity.
Revised: Such generated samples are in line with expectations as we notice that the ∆ τ score indicates that the model has learned that thickness causes intensity and as such manipulating the causal variable propagates inﬂuence to the effect variable which manifests itself as an increase in background intensity.

Original: CelebaHQ .
Revised: CelebaHQ .

Original: We perform causal discovery over a pair of binary attributes in photo-realistic HD images as found in the CelebaHQ dataset (Karras et al., 2017).
Revised: We perform causal discovery over a pair of binary attributes in photo-realistic HD images as found in the CelebaHQ dataset (Karras et al., 2017).

Original: For our G we employ a powerful generative model in StyleGAN2 (Karras et al., 2020) which is pretrained on the FlickrFacesHQ (Karras et al., 2019).
Revised: For our G we employ a powerful generative model in StyleGAN2 (Karras et al., 2020) which is pretrained on the FlickrFacesHQ (Karras et al., 2019).

Original: To align the latent spaces between our desired causal discovery and FlickrFaces we ﬁnetune the pretrained model using 10% of CelebaHQ.
Revised: To align the latent spaces between our desired causal discovery and FlickrFaces we ﬁnetune the pretrained model using 10% of CelebaHQ.

Original: In table 1, we report the direction of causality found by each of the baselines and our approach using ∆ τ .
Revised: In table 1, we report the direction of causality found by each of the baselines and our approach using ∆ τ .

Original: For each DAG considered we also generate counterfactual samples by manipulating one attribute and qualitatively observing downstream effects on the other.
Revised: For each DAG considered we also generate counterfactual samples by manipulating one attribute and qualitatively observing downstream effects on the other.

Original: To diagnose independence between attributes, we also plot a histogram of ∆ τ scores under 100 random latent projections which allow us to reject the null hypothesis if the true ∆ τ (black line) is signiﬁcant.
Revised: To diagnose independence between attributes, we also plot a histogram of ∆ τ scores under 100 random latent projections which allow us to reject the null hypothesis if the true ∆ τ (black line) is signiﬁcant.

Original: In the absence of ground truth labels for causal associations, we posit a plausible causal relationship between each pair of attributes (e.g. Gender → Mustache) which we accept as a non-controversial biological prior.
Revised: In the absence of ground truth labels for causal associations, we posit a plausible causal relationship between each pair of attributes (e.g. Gender → Mustache) which we accept as a non-controversial biological prior.

Original: We observe that ∆ τ in all but one case—unlike each baseline —is consistently capable of diagnosing the causal relationship implicit to G .
Revised: We observe that ∆ τ in all but one case—unlike each baseline —is consistently capable of diagnosing the causal relationship implicit to G .

Original: For example, for the causal relationship: Gender → Mustache, we observe counterfactual samples where gender (top row) is manipulated—i.e. from male to female–the mustache attribute also changes.
Revised: For example, for the causal relationship: Gender → Mustache, we observe counterfactual samples where gender (top row) is manipulated—i.e. from male to female–the mustache attribute also changes.

Original: In contrast, when the effect variable (bot row) is manipulated there is a reduced impact on the perceived gender.
Revised: In contrast, when the effect variable (bot row) is manipulated there is a reduced impact on the perceived gender.

Original: We also consider three causal discovery baselines: 1. LiNGAM
Revised: We also consider three causal discovery baselines: 1. LiNGAM

Original: (Shimizu et al., 2006) 2.
Revised: (Shimizu et al., 2006) 2.

Original: Additive Noise Model (Hoyer et al., 2008), and 3.
Revised: Additive Noise Model (Hoyer et al., 2008), and 3.

Original: DAGs with No Tears (Zhang et al., 2018).
Revised: DAGs with No Tears (Zhang et al., 2018).

Original: For a fair comparison, and to be able to make causal statements about a generator as opposed to observed data, the causal discovery baselines operate on the class probabilities learned by the proxy classiﬁers described in §3.3 (see §B for further information on implementation details).
Revised: For a fair comparison, and to be able to make causal statements about a generator as opposed to observed data, the causal discovery baselines operate on the class probabilities learned by the proxy classiﬁers described in §3.3 (see §B for further information on implementation details).

Original: 4. C AUSAL D ATA A UGMENTATION
Revised: 4. C AUSAL D ATA A UGMENTATION

Original: We now turn to a practical downstream application of generative modeling in the data augmentation of rare groups within a supervised learning task.
Revised: We now turn to a practical downstream application of generative modeling in the data augmentation of rare groups within a supervised learning task.

Original: Speciﬁcally, we consider the experimental setup in Sagawa et al.
Revised: Speciﬁcally, we consider the experimental setup in Sagawa et al.

Original: and perform group Distributionally Robust Optimization (DRO) over rare groups in CelebA. For a fair comparison with Sagawa et al.
Revised: and perform group Distributionally Robust Optimization (DRO) over rare groups in CelebA. For a fair comparison with Sagawa et al.

Original: we choose to classify a person’s hair color ( H ) which maybe is causally related with gender ( G ) (Hysi et al., 2018).
Revised: we choose to classify a person’s hair color ( H ) which maybe is causally related with gender ( G ) (Hysi et al., 2018).

Original: In this case, the smallest group are blonde males that constitute only 1387162770 = 0 .
Revised: In this case, the smallest group are blonde males that constitute only 1387162770 = 0 .

Original: 85% of training examples.
Revised: 85% of training examples.

Original: We use a ﬁnetuned StyleGAN2 to augment the total number of blonde males by a factor of 10 in two separate ways.
Revised: We use a ﬁnetuned StyleGAN2 to augment the total number of blonde males by a factor of 10 in two separate ways.

Original: First, we manipulate the hair color of all non-blonde males which we hypothesize corresponds to changing only the effect variable.
Revised: First, we manipulate the hair color of all non-blonde males which we hypothesize corresponds to changing only the effect variable.

Original: The second approach seeks to generate additional samples of blond males by manipulating the gender of blond females in CelebA. For both approaches, we generate images of dimensions 1024 × 1024 which are then downsampled via bilinear interpolation to a resolution of 224 × 224 which is then consumed by a Resnet50 classiﬁer.
Revised: The second approach seeks to generate additional samples of blond males by manipulating the gender of blond females in CelebA. For both approaches, we generate images of dimensions 1024 × 1024 which are then downsampled via bilinear interpolation to a resolution of 224 × 224 which is then consumed by a Resnet50 classiﬁer.

Original: In table 2, we report train, validation, and test accuracies on average across all subgroups and the worst-group (Blonde Males, i.e., G =1, H =1), while in ﬁgure 5 we report all subgroup accuracies for both the baseline approach and with data augmentation with α determined using the validation set.
Revised: In table 2, we report train, validation, and test accuracies on average across all subgroups and the worst-group (Blonde Males, i.e., G =1, H =1), while in ﬁgure 5 we report all subgroup accuracies for both the baseline approach and with data augmentation with α determined using the validation set.

Original: Additionally, we perform early stopping using the worst subgroup validation accuracy for all approaches.
Revised: Additionally, we perform early stopping using the worst subgroup validation accuracy for all approaches.

Original: Fig 1, illustrates samples when gender for blonde females are manipulated to males (the causal variable) versus manipulating the hair color of non-blonde males to blonde.
Revised: Fig 1, illustrates samples when gender for blonde females are manipulated to males (the causal variable) versus manipulating the hair color of non-blonde males to blonde.

Original: Qualitatively, it is clear that if the end goal is to generate more blonde males the generative model must manipulate the effect variable.
Revised: Qualitatively, it is clear that if the end goal is to generate more blonde males the generative model must manipulate the effect variable.

Original: Quantitatively, we compute a bootstrapped FID score over 10 trials for each setup’s generated samples, using their respective against the true distribution of blonde males.
Revised: Quantitatively, we compute a bootstrapped FID score over 10 trials for each setup’s generated samples, using their respective against the true distribution of blonde males.

Original: We ﬁnd that the FIDs for the counterfactuals in the inferred causal direction ( G → H ) improve over the anticausal direction by 19 .
Revised: We ﬁnd that the FIDs for the counterfactuals in the inferred causal direction ( G → H ) improve over the anticausal direction by 19 .

Original: 5% revealing that using G ’s learned causal relationship yields semantically better samples.
Revised: 5% revealing that using G ’s learned causal relationship yields semantically better samples.

Original: Finally, we ﬁnd that augmentation by manipulating the effect, hair color, in this case, leads to a 2 .
Revised: Finally, we ﬁnd that augmentation by manipulating the effect, hair color, in this case, leads to a 2 .

Original: 7% gain in test accuracy in comparison to a − 1 .
Revised: 7% gain in test accuracy in comparison to a − 1 .

Original: 6% decrease when manipulating gender.
Revised: 6% decrease when manipulating gender.

Original: Moreover, we observe the largest drop in test worst group accuracy when we generate synthetic samples without respecting the implicit causal associations in G , with classiﬁcation accuracy dropping to 75 % —serving as an important example for the need for a causal understanding of G .
Revised: Moreover, we observe the largest drop in test worst group accuracy when we generate synthetic samples without respecting the implicit causal associations in G , with classiﬁcation accuracy dropping to 75 % —serving as an important example for the need for a causal understanding of G .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: CAGE is conceptually related to several methods, both within the generative modeling literature as well as the causal inference literature in general.
Revised: CAGE is conceptually related to several methods, both within the generative modeling literature as well as the causal inference literature in general.

Original: With regards to the generative models, there is a rich line of work exploring the use of causal models to learn disentangled representations as advocated by Sch ¨ olkopf et al.
Revised: With regards to the generative models, there is a rich line of work exploring the use of causal models to learn disentangled representations as advocated by Sch ¨ olkopf et al.

Original: (2021) and Gresele et al.
Revised: (2021) and Gresele et al.

Original: Both Sauer & Geiger (2021) and Besserve et al.
Revised: Both Sauer & Geiger (2021) and Besserve et al.

Original: (2018) leverage to notion of independent mechanisms , encoded as structural equation models, to obtain disentangled generative models which are subsequently employed to generate counterfactual samples.
Revised: (2018) leverage to notion of independent mechanisms , encoded as structural equation models, to obtain disentangled generative models which are subsequently employed to generate counterfactual samples.

Original: Pawlowski et al.
Revised: Pawlowski et al.

Original: (2020) propose a framework which incorporates deep probabilistic models as components within a structural equation model.
Revised: (2020) propose a framework which incorporates deep probabilistic models as components within a structural equation model.

Original: In contrast with this work, CAGE instead focuses on quantifying when such causal biases have been internalized by a generator.
Revised: In contrast with this work, CAGE instead focuses on quantifying when such causal biases have been internalized by a generator.

Original: In this sense, CAGE can be seen as solving the reverse problem to CausalGAN (Kocaoglu et al., 2017), which trains generative models such that they are consistent with a given causal graph.
Revised: In this sense, CAGE can be seen as solving the reverse problem to CausalGAN (Kocaoglu et al., 2017), which trains generative models such that they are consistent with a given causal graph.

Original: Concurrently, there is a body of related work which leverages deep generative models to solve causal inference tasks.
Revised: Concurrently, there is a body of related work which leverages deep generative models to solve causal inference tasks.

Original: Louizos et al.
Revised: Louizos et al.

Original: (2017) and Yoon et al.
Revised: (2017) and Yoon et al.

Original: (2018) employ generative models to estimate individual treatment effects by generating counterfactuals under alternative treatment regimes.
Revised: (2018) employ generative models to estimate individual treatment effects by generating counterfactuals under alternative treatment regimes.

Original: Tran & Blei (2017) and Wang & Blei (2019) synthesize ideas from causality and generative modeling to address challenges associated with unobserved confounding.
Revised: Tran & Blei (2017) and Wang & Blei (2019) synthesize ideas from causality and generative modeling to address challenges associated with unobserved confounding.

Original: Finally, deep generative models have also been used for causal discovery tasks, examples include Kalainathan et al.
Revised: Finally, deep generative models have also been used for causal discovery tasks, examples include Kalainathan et al.

Original: (2018); Lopez-Paz & Oquab (2016) and Khemakhem et al.
Revised: (2018); Lopez-Paz & Oquab (2016) and Khemakhem et al.

Original: (2021), which is in sharp contrast with this work which focuses on uncovering causal structure within G , as opposed to the training data.
Revised: (2021), which is in sharp contrast with this work which focuses on uncovering causal structure within G , as opposed to the training data.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed CAGE, a framework for inferring cause-effect relationships within the latent space of deep generative model via geometric manipulations.
Revised: We proposed CAGE, a framework for inferring cause-effect relationships within the latent space of deep generative model via geometric manipulations.

Original: CAGE is well suited to a wide family of modern deep generative models trained on complex high-dimensional data and does not require any altering of the original training objective nor hard to obtain counterfactual data.
Revised: CAGE is well suited to a wide family of modern deep generative models trained on complex high-dimensional data and does not require any altering of the original training objective nor hard to obtain counterfactual data.

Original: Empirically, we ﬁnd CAGE reliably extracts correct cause-effect relationships in controlled settings like MorphoMNIST.
Revised: Empirically, we ﬁnd CAGE reliably extracts correct cause-effect relationships in controlled settings like MorphoMNIST.

Original: On high-resolution image data such as CelebAHQ, CAGE reveals—occasionally contrary to a biological prior—cause-effect relationships that are best supported by generated counterfactual samples.
Revised: On high-resolution image data such as CelebAHQ, CAGE reveals—occasionally contrary to a biological prior—cause-effect relationships that are best supported by generated counterfactual samples.

Original: Finally, using causal insights supplied by CAGE we also perform guided data augmentations and power a supervised distributionally robust optimization task, obtaining a 2 .
Revised: Finally, using causal insights supplied by CAGE we also perform guided data augmentations and power a supervised distributionally robust optimization task, obtaining a 2 .

Original: 7% improvement over the state-of-the-art in rare subgroup classiﬁcation.
Revised: 7% improvement over the state-of-the-art in rare subgroup classiﬁcation.

Original: Limitations .
Revised: Limitations .

Original: Despite CAGE’s broad applicability to deep generative models, one important limitation is the requirement of a latent space.
Revised: Despite CAGE’s broad applicability to deep generative models, one important limitation is the requirement of a latent space.

Original: Fully observed generative models (e.g., autoregressive modes) do not have an explicit latent space and we will need to adapt their internal representations for probing via CAGE.
Revised: Fully observed generative models (e.g., autoregressive modes) do not have an explicit latent space and we will need to adapt their internal representations for probing via CAGE.

Original: In addition, the notion of GATE in CAGE, while being novel and tested rigorously empirically, does not entirely follow the same analysis as ATE.
Revised: In addition, the notion of GATE in CAGE, while being novel and tested rigorously empirically, does not entirely follow the same analysis as ATE.

Original: We implicitly borrow some key assumptions regarding unconfoundedness and introduce new ones such as linear separability in semantic attributes of the latent space in designing our estimator.
Revised: We implicitly borrow some key assumptions regarding unconfoundedness and introduce new ones such as linear separability in semantic attributes of the latent space in designing our estimator.

Original: These assumptions may not always hold in practice and future work can investigate both theoretically and empirically if and when mitigation strategies are needed.
Revised: These assumptions may not always hold in practice and future work can investigate both theoretically and empirically if and when mitigation strategies are needed.

Original: Finally, as we consider only binary attributes in this work, the extension of our framework to richer non-binary attributes and more than two variables (as in structural causal models) is an interesting direction for future work.
Revised: Finally, as we consider only binary attributes in this work, the extension of our framework to richer non-binary attributes and more than two variables (as in structural causal models) is an interesting direction for future work.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: As deep generative models grow in their practicality and deployment in high-stake applications, it is important to take a critical view of the inherent biases implicitly encoded  (Sheng et al., 2019; Choi et al., 2020; Bender et al., 2021).
Revised: As deep generative models grow in their practicality and deployment in high-stake applications, it is important to take a critical view of the inherent biases implicitly encoded in these models (Sheng et al., 2019; Choi et al., 2020; Bender et al., 2021).

Original: Our framework allows for a principled study of such biases through a causal analysis, potentially paving the way to mitigate the undesirable harms of deep generative models.
Revised: Our framework allows for a principled study of such biases through a causal analysis, potentially paving the way to mitigate the undesirable harms of deep generative models.

Original: For example, our approach highlights important failings of the generative model itself—e.g. when cause-effect relationships are inconsistent with biological priors—which may consequently inform practitioners on deriving better data acquisition strategies for ﬁnetuning.
Revised: For example, our approach highlights important failings of the generative model itself—e.g. when cause-effect relationships are inconsistent with biological priors for human subjects or with physical relationships for scientiﬁc applications.

Original: Like many other efforts in generative modeling, our endeavors for better understanding these models  canpotentially be exploited by malicious actors for generating fake content, commonly known as 
Revised: In such cases, the use of these models in any high-stakes downstream application can be very costly without any intervention. Examples range from the current use of generative models for accelerating scientiﬁc discovery as well as societal applications in training fair classiﬁers.

Original:  Deepfakes (Korshunov & Marcel, 2018). 
Revised: Ironically, in all such applications, the use of generative models can worsen the problem at hand by introducing new blindspots through the use of generated data that does not conform to the desired cause-effect relationship. Explicitly inferring these failings provides researchers and practitioners with an opportunity to develop algorithms and evaluation metrics that go beyond the current focus on visual ﬁdelity or likelihoods as well as derive better data acquisition strategies for ﬁnetuning generative models. On the other hand, we acknowledge that like many other efforts in generative modeling, our endeavors for better understanding these models can potentially be exploited by malicious actors for generating fake content, commonly known as Deepfakes (Korshunov & Marcel, 2018). Finally, causality has proven to be a highly powerful tool for formalizing important concerns of machine learning models with regards to fairness, robustness, and interpretability.

Original: Finally, as highlighted in our limitations, our framework  itself implicitly encodes some assumptions that may not hold in practice, and hence, researchers and practitioners should exercise due caution and appropriate judgment in interpreting the ﬁndings.
Revised: However, as highlighted in our limitations, any causal framework including ours itself implicitly encodes some assumptions that may not hold in practice. Hence, researchers and practitioners should exercise due caution and appropriate judgment in interpreting the ﬁndings.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Throughout the paper, we tried to provide as many details as possible in order for the results of the main paper to be reproducible.
Revised: Throughout the paper, we tried to provide as many details as possible in order for the results of the main paper to be reproducible.

Original: In particular, we provide implementation details in Appendix B for all the main models used.
Revised: In particular, we provide implementation details in Appendix B for all the main models used.

Original: To ensure, the estimated ∆ τ values are signiﬁcant we compute p -values under random projections in the latent space for CAGE.
Revised: To ensure, the estimated ∆ τ values are signiﬁcant we compute p -values under random projections in the latent space for CAGE.

Original: All the baseline approaches are additionally run over multiple trials.
Revised: All the baseline approaches are additionally run over multiple trials.

Original: Furthermore, we provide additional generated samples in Appendix D to supplant all counterfactual samples provided in the main text.
Revised: Furthermore, we provide additional generated samples in Appendix D to supplant all counterfactual samples provided in the main text.

Original: Finally, code to reproduce all experiments will be made public after the review process.
Revised: Finally, code to reproduce all experiments will be made public after the review process.

Original: A E XTENDING A VERAGE T REATMENT E FFECTS TO G ENERATIVE M ODELS
Revised: A E XTENDING A VERAGE T REATMENT E FFECTS TO G ENERATIVE M ODELS

Original: For average treatment effects, we are interested in computing
Revised: For average treatment effects, we are interested in computing

Original: [Equation]
Revised: [Equation]

Original: There are two important distinctions between the use of GATE in CAGE and a traditional ATE computation:
Revised: There are two important distinctions between the use of GATE in CAGE and a traditional ATE computation:

Original: • First, as we are interrogating a generative model, we can observe both Y m c =1 ( i ) and Y m c =

Revised: • First, as we are interrogating a generative model, we can observe both Y m c =1 ( i ) and Y m c =

Original: ( i ) .
Revised: ( i ) .

Original: This is a signiﬁcant difference to standard ATE calculations, which can only observe one outcome.
Revised: This is a signiﬁcant difference to standard ATE calculations, which can only observe one outcome.

Original: Second, we are interested in understanding implicit causal structure in the generator, G , the expectation over samples from the prior of G as opposed to over samples from an observational dataset.
Revised: Second, we are interested in understanding implicit causal structure in the generator, G , the expectation over samples from the prior of G as opposed to over samples from an observational dataset.

Original: This is an important distinction as it disambiguates the causal structure present in an observation dataset, which is the focus of traditional causal discovery methodology, with any causal structure implicit within the generator, which is our primary concern.
Revised: This is an important distinction as it disambiguates the causal structure present in an observation dataset, which is the focus of traditional causal discovery methodology, with any causal structure implicit within the generator, which is our primary concern.

Original: For unbiased estimation of ATE (a causal quantity) via statistical quantities, it is standard to assume unconfoundedness, positivity, and SUTVA.
Revised: For unbiased estimation of ATE (a causal quantity) via statistical quantities, it is standard to assume unconfoundedness, positivity, and SUTVA.

Original: For the use of GATE in CAGE, we implicitly make further assumptions such as:
Revised: For the use of GATE in CAGE, we implicitly make further assumptions such as:

Original: Counterfactual parameterization :
Revised: Counterfactual parameterization :

Original: The latent space of a generator, G , must be linearly separable with respect to the attributes of interest.
Revised: The latent space of a generator, G , must be linearly separable with respect to the attributes of interest.

Original: This assumption is required as CAGE parameterizes the assignment of counterfactual treatments as linear manipulations over the latent space of a generator.
Revised: This assumption is required as CAGE parameterizes the assignment of counterfactual treatments as linear manipulations over the latent space of a generator.

Original: We note that such an assumption has been shown to hold in practice for deep generative models (Denton et al., 2019; Ramaswamy et al., 2021).
Revised: We note that such an assumption has been shown to hold in practice for deep generative models (Denton et al., 2019; Ramaswamy et al., 2021).

Original: Complete support of the generator : The generator, G , can generate high quality samples over the entire support of its latent space, Z .
Revised: Complete support of the generator : The generator, G , can generate high quality samples over the entire support of its latent space, Z .

Original: This assumption is required to guarantee that linear manipulations over the latent space of a generator continue to produce high quality images, such that our resulting conclusions are not the result of e.g., image artefacts.
Revised: This assumption is required to guarantee that linear manipulations over the latent space of a generator continue to produce high quality images, such that our resulting conclusions are not the result of e.g., image artefacts.

Original: Sufﬁciently expressive and accurate classiﬁers : CAGE is premised on the use of proba-
Revised: Sufﬁciently expressive and accurate classiﬁers : CAGE is premised on the use of proba-

Original: bilistic classiﬁers in order to quantify the presence/absence of an effect variable on counterfactual examples and thus compute treatment effects as speciﬁed in equations (5-6).
Revised: bilistic classiﬁers in order to quantify the presence/absence of an effect variable on counterfactual examples and thus compute treatment effects as speciﬁed in equations (5-6).

Original: We require accurate and calibrated classiﬁers (ideally, Bayes optimal) for computing GATE.
Revised: We require accurate and calibrated classiﬁers (ideally, Bayes optimal) for computing GATE.

Original: B M ODEL D ETAILS
Revised: B M ODEL D ETAILS

Original: For our experiments, whenever possible, we used the default settings found in the original papers of all chosen models.
Revised: For our experiments, whenever possible, we used the default settings found in the original papers of all chosen models.

Original: In particular, we used the default settings for both Mintnet (Song et al., 2019) and
Revised: In particular, we used the default settings for both Mintnet (Song et al., 2019) and

Original: StyleGAN2 (Karras et al., 2020) which were pretrained on MNIST and FlickrFacesHQ respectively.
Revised: StyleGAN2 (Karras et al., 2020) which were pretrained on MNIST and FlickrFacesHQ respectively.

Original: For Mintnet we ﬁnetuned on a MorphoMnist dataset for 250 epochs using the Adam optimizer with default settings.
Revised: For Mintnet we ﬁnetuned on a MorphoMnist dataset for 250 epochs using the Adam optimizer with default settings.

Original: Similarly, we also ﬁnetuned StyleGAN2 on CelebAHQ for 2000 iterations using 10% of CelebAHQ.
Revised: Similarly, we also ﬁnetuned StyleGAN2 on CelebAHQ for 2000 iterations using 10% of CelebAHQ.

Original: Our synthetic experiments on the other required us to train a Masked
Revised: Our synthetic experiments on the other required us to train a Masked

Original: AutoRegressive Flow (Papamakarios et al., 2017) that consisted of 10 layers with 4 blocks per layer.
Revised: AutoRegressive Flow (Papamakarios et al., 2017) that consisted of 10 layers with 4 blocks per layer.

Original: The Masked AutoRegressive Flow was trained for 5000 iterations using 5000 data samples.
Revised: The Masked AutoRegressive Flow was trained for 5000 iterations using 5000 data samples.

Original: For data-augmentation we train a ResNet50 classiﬁer from scratch using weight decay of 1 e − 3 as well as the default hyperparams for group DRO suggested in Sagawa et al.
Revised: For data-augmentation we train a ResNet50 classiﬁer from scratch using weight decay of 1 e − 3 as well as the default hyperparams for group DRO suggested in Sagawa et al.

Original: We also search over α in the range of 1 − 20 while perorming early stopping using the worst subgroup validation accuracy.
Revised: We also search over α in the range of 1 − 20 while perorming early stopping using the worst subgroup validation accuracy.

Original: Note that the worst subgroup need not be blonde males throughout the entire optimization procedure.
Revised: Note that the worst subgroup need not be blonde males throughout the entire optimization procedure.

Original: A NOTE ON BASELINE METHODS FOR C ELEB HQ
Revised: A NOTE ON BASELINE METHODS FOR C ELEB HQ

Original: We compared CAGE to well-established causal discovery algorithms when looking to infer structure over a deep generative model, G .
Revised: We compared CAGE to well-established causal discovery algorithms when looking to infer structure over a deep generative model, G .

Original: As noted in §4.2, the causal discovery baselines considered operate over the class probabilities output by probabilistic classiﬁers trained as §3.3.
Revised: As noted in §4.2, the causal discovery baselines considered operate over the class probabilities output by probabilistic classiﬁers trained as §3.3.

Original: We note that while it would be possible to learn causal structure using baseline methods, such as LiNGAM, over the metadata (e.g., hair color and gender), this would not necesarly provide any insights into the causal structure implicit within the generator, G .
Revised: We note that while it would be possible to learn causal structure using baseline methods, such as LiNGAM, over the metadata (e.g., hair color and gender), this would not necesarly provide any insights into the causal structure implicit within the generator, G .

Original: For this reason, we instead focus on applying causal discovery methods over the output of proxy classiﬁers.
Revised: For this reason, we instead focus on applying causal discovery methods over the output of proxy classiﬁers.

Original: C A DDITIONAL B ASELINE E XPERIMENTS
Revised: C A DDITIONAL B ASELINE E XPERIMENTS

Original: In this section we perform a series of ablation studies that stress test our framework CAGE under various experimental settings.
Revised: In this section we perform a series of ablation studies that stress test our framework CAGE under various experimental settings.

Original: Hard vs. Soft .
Revised: Hard vs. Soft .

Original: As we have access to ground truth labels it is tempting to consider whether using hard labels as opposed to the classiﬁers probability is better suited to computing our ∆ τ metric.
Revised: As we have access to ground truth labels it is tempting to consider whether using hard labels as opposed to the classiﬁers probability is better suited to computing our ∆ τ metric.

Original: In the table below we repeat our causal discovery experiment over CelebAHQ.
Revised: In the table below we repeat our causal discovery experiment over CelebAHQ.

Original: As observed, all baselines provide unreliable estimates to the causal relationship between variables when compared to the main table which is computed using soft labels.
Revised: As observed, all baselines provide unreliable estimates to the causal relationship between variables when compared to the main table which is computed using soft labels.

Original: Finally, we found it useful to assign soft labels provided by the classiﬁers used in the ATE computations for all augmented images during group DRO training.
Revised: Finally, we found it useful to assign soft labels provided by the classiﬁers used in the ATE computations for all augmented images during group DRO training.

Original: [Table]
Revised: [Table]

Original: Baselines without using CounterFactuals .
Revised: Baselines without using CounterFactuals .

Original: We now turn to the use of Counterfactuals when computing our baseline scores.
Revised: We now turn to the use of Counterfactuals when computing our baseline scores.

Original: Speciﬁcally, we attempt causal discovery purely using observation data and labels with both hard and soft labels, which is in contrast to the main paper which considered baselines which had access to classiﬁer probabilities on counterfactual data.
Revised: Speciﬁcally, we attempt causal discovery purely using observation data and labels with both hard and soft labels, which is in contrast to the main paper which considered baselines which had access to classiﬁer probabilities on counterfactual data.

Original: The table below shows this result of this ablation.
Revised: The table below shows this result of this ablation.

Original: [Table]
Revised: [Table]

Original: −→ −→ −→
Revised: −→ −→ −→

Original: D A DDITIONAL G ENERATED SAMPLES
Revised: D A DDITIONAL G ENERATED SAMPLES

Original: We now provide additional qualitiative examples of the generated counterfactuals on the same set of DAG’s as considered in the main text.
Revised: We now provide additional qualitiative examples of the generated counterfactuals on the same set of DAG’s as considered in the main text.

Original: In the ﬁrst row of all settings we generate counterfactuals by manipulating the ﬁrst attribute variable (e.g. Gender) while the second row corresponds to counterfactuals where the second attribute is manipulated.
Revised: In the ﬁrst row of all settings we generate counterfactuals by manipulating the ﬁrst attribute variable (e.g. Gender) while the second row corresponds to counterfactuals where the second attribute is manipulated.

Original: The third row can either be the ﬁrst or second attribute depending on context.
Revised: The third row can either be the ﬁrst or second attribute depending on context.

Original: We note that the generated samples may expose unintended biases that may be present in CelebAHQ and can be learned by the generator.
Revised: We note that the generated samples may expose unintended biases that may be present in CelebAHQ and can be learned by the generator.

Original: While CAGE does not introduce any new sources of biases it should be noted that generating counterfactuals in this manner can further exacerbate these initially hidden biases.
Revised: While CAGE does not introduce any new sources of biases it should be noted that generating counterfactuals in this manner can further exacerbate these initially hidden biases.

Original: As a result, CAGE can serve as an important aid in empirically illuminating important ethical concerns that should be taken into account for any purposeful uses of generative models.
Revised: As a result, CAGE can serve as an important aid in empirically illuminating important ethical concerns that should be taken into account for any purposeful uses of generative models.

Original: Gender, Mustache .
Revised: Gender, Mustache .

Revised: We consider generating samples by either manipulating the mustache attribute (top two rows) and then manipulating gender.

Revised: As we can see, when the Gender is male we are able to semantically preserve the gender as we increase the presence of mustache (top row).

Revised: Similarly, we ﬁnd that when the gender is female we are still able to semantically preserve the (subjective) gender as we manipulate the mustache attribute (middle row).

Revised: Finally, in the third row we change the gender of from female to male and notice that although the mustache attribute was not explicitly manipulated it appears as we manipulate gender more aggressively to male (bottom row).

Revised: These results suggests that the generative models learned causal relationship is that gender causes mustache .

Original: [Table]
Revised: We consider generating samples by either manipulating the goatee attribute (top two rows) and manipulating gender. In the ﬁrst row we see that manipulating the goatee attribute leads to a visual and subjective change the samples perceived gender (top row). Similarly, we ﬁnd that when the gender is already male increasing the presence of a goatee does not reverse the gender to female as expected (middle row). Finally, in the third row we change the gender of from female to male and notice that there is no increase in goatee in the generated samples.

Revised: These results suggests that the generative models learned causal relationship is that gender is independent of goatee .

Original: [Table]
Revised: Gender, Smiling . We consider generating samples by either manipulating the smiling attribute (top two rows) and manipulating gender. In the ﬁrst two rows we see that increasing the smiling attribute does not cause a (subjective) change the samples perceived gender. Conversely, in the last row manipulating a samples gender from female to male preserves the smiling attribute (or lack thereof) in the generated samples.

Revised: These results suggests that the generative models learned causal relationship is that gender is independent of smiling .

Original: [Table]
Revised: Age, Bald . We consider generating samples by either manipulating the baldness attribute (top two rows) and then manipulating age (last row). In the ﬁrst two row we see that increasing the baldness attribute does considerably change the age of the generated sample. On the other hand, in the last row we increase the age of the candidate and observe that while the baldness attribute increases it is not the extent as samples found in the ﬁrst two rows.

Revised: These results suggests that the generative models learned causal relationship, in contrast to our biological prior, is that baldness is the cause of age .

Original: [Table]
Revised: We consider generating samples by either manipulating the pointy nose attribute (top two rows) and manipulating brown hair. In the ﬁrst two rows we see that increasing the pointy nose attribute does cause a change in the samples hair color from dark brown to more blonde. Conversely, in the last row manipulating a samples hair color to be more brown does not appear to signiﬁcantly change the pointy nose attribute in the generated samples.

Revised: These results suggests that the generative models learned causal relationship is that pointy nose causes brown hair .

Revised: Gender, Rosy Cheeks .

Revised: We consider generating samples by either manipulating the rosy cheek attribute (top two rows) and manipulating gender.

Revised: In the ﬁrst two rows we see that increasing the rosy cheek attribute does not cause a change in the samples (subjective) gender—i.e. both females and males retain their gender as we increase their rosy cheeks.

Revised: On the other hand, in the last row manipulating a samples gender from female to male appears to slighltly decrease the rosy cheek attribute in the generated samples.

Revised: These results suggests that the generative models learned causal relationship is that gender causes rosy cheeks .

Original: [Table]
Revised: We consider generating samples by either manipulating the blonde hair attribute (top two rows) and manipulating gender. As observed in the ﬁrst two rows increasing the blondeness of a samples hair does not change the perceived gender from female to male. In sharp contrast, the last row we see that manipulating the gender from male to female leads to the samples hair color changing towards being more blonde.

Revised: These results suggests that the generative models learned causal relationship is that gender causes blonde hair .

Revised: E MMD DISTANCE BETWEEN CONDITIONAL AND COUNTERFACTUAL DISTRIBUTIONS

Original: [Table]
Revised: In this appendix we plot the Maximum Mean Discrepancy distance between the conditional distribution on the latent space of an attribute and the counterfactual distribution realized through our CAGE framework. Speciﬁcally, we plot the MMD distance as a function of the interpolant strength α and we observe that the two distributions grow further apart the further we interpolate past the linear classiﬁers hyperplane as expected.