Original: L EARNING L ATENT S TRUCTURAL C AUSAL M ODELS
Revised: L EARNING L ATENT S TRUCTURAL C AUSAL M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Causal learning has long concerned itself with the accurate recovery of underlying causal mechanisms.
Revised: Causal learning has long concerned itself with the accurate recovery of underlying causal mechanisms.

Original: Such causal modelling enables better explanations of out-ofdistribution data.
Revised: Such causal modelling enables better explanations of out-ofdistribution data.

Original: Prior works on causal learning assume that the high-level causal variables are given.
Revised: Prior works on causal learning assume that the high-level causal variables are given.

Original: However, in machine learning tasks, one often operates on low-level data like image pixels or high-dimensional vectors.
Revised: However, in machine learning tasks, one often operates on low-level data like image pixels or high-dimensional vectors.

Original: In such settings, the entire Structural Causal Model (SCM) – structure, parameters, and high-level causal variables – is unobserved and needs to be learnt from low-level data.
Revised: In such settings, the entire Structural Causal Model (SCM) – structure, parameters, and high-level causal variables – is unobserved and needs to be learnt from low-level data.

Original: We treat this problem as Bayesian inference of the latent SCM, given low-level data.
Revised: We treat this problem as Bayesian inference of the latent SCM, given low-level data.

Original: For linear Gaussian additive noise SCMs, we present a tractable approximate inference method which performs joint inference over the causal variables, structure and parameters of the latent SCM from random, known interventions.
Revised: For linear Gaussian additive noise SCMs, we present a tractable approximate inference method which performs joint inference over the causal variables, structure and parameters of the latent SCM from random, known interventions.

Original: Experiments are performed on synthetic datasets and a causally generated image dataset to demonstrate the efficacy of our approach.
Revised: Experiments are performed on synthetic datasets and a causally generated image dataset to demonstrate the efficacy of our approach.

Original: We also perform image generation from unseen interventions, thereby verifying out of distribution generalization for the proposed causal model.
Revised: We also perform image generation from unseen interventions, thereby verifying out of distribution generalization for the proposed causal model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning variables of interest and uncovering causal dependencies is crucial for intelligent systems to reason and predict in scenarios that differ from thedistribution of the training data.
Revised: Learning variables of interest and uncovering causal dependencies is crucial for intelligent systems to reason and predict in scenarios that differ from the training distribution.

Original: In the causality literature, causal variables and mechanisms are often assumed to be known.
Revised: In the causality literature, causal variables and mechanisms are often assumed to be known.

Original: This knowledge enables reasoning and prediction under unseen interventions.
Revised: This knowledge enables reasoning and prediction under unseen interventions.

Original: In machine learning however, one does not have direct access to the underlying variables of interest nor the causal structure and mechanisms corresponding to these variables.
Revised: In machine learning, however, one does not have direct access to the underlying variables of interest nor the causal structure and mechanisms corresponding to them.

Original: Rather, these have to be learned from observed low-level data like pixels of an image which are usually high-dimensional.
Revised: Rather, these have to be learned from observed low-level data like pixels of an image which are usually high-dimensional.

Original: Having a learned causal model can then be useful for generalizing to out-of-distribution data (Scherrer et al., 2022; Ke et al., 2021), estimating the effect of interventions (Pearl, 2009; Sch¨olkopf et al., 2021), disentangling underlying factors of variation (Bengio et al., 2012; Wang & Jordan, 2021), and transfer learning (Schoelkopf et al., 2012; Bengio et al., 2019).
Revised: Having a learned causal model can then be useful for generalizing to out-of-distribution data (Scherrer et al., 2022; Ke et al., 2021), estimating the effect of interventions (Pearl, 2009; Sch ¨ olkopf et al., 2021), disentangling underlying factors of variation (Bengio et al., 2012; Wang and Jordan, 2021), and transfer learning (Schoelkopf et al., 2012; Bengio et al., 2019).

Original: In structure learning (Spirtes et al., 1993; Zheng et al., 2018), one learns the structure and parameters of the Structural Causal Model (SCM) (Pearl, 2009) that best explains some observed high-level causal variables.
Revised: Structure learning (Spirtes et al., 2000; Zheng et al., 2018) learns the structure and parameters of the Structural Causal Model (SCM) (Pearl, 2009) that best explains some observed high-level causal variables.

Original: However, in causal machine learning and representation learning,  these causal variables might not be observable anymore.
Revised: In causal machine learning and representation learning, however, these causal variables may no longer be observable.

Original: This serves as the motivation for our work.
Revised: This serves as the motivation for our work.

Original: We address the problem of learning the entire SCM – which consists of causal variables, structure and parameters – which is latent, by learning to generate observed low-level data.
Revised: We address the problem of learning the entire SCM – consisting its causal variables, structure and parameters – which is latent, by learning to generate observed low-level data.

Original: Since one often operates in low-data regimes or non-identifiable settings, we adopt a Bayesian formulation so as to quantify epistemic uncertainty over the learned latent SCM.
Revised: Since one often operates in low-data regimes or non-identifiable settings, we adopt a Bayesian formulation so as to quantify epistemic uncertainty over the learned latent SCM.

Original: Given a dataset, we use variational inference to learn a joint posterior over the causal variables, structure and parameters of the latent SCM.
Revised: Given a dataset, we use variational inference to learn a joint posterior over the causal variables, structure and parameters of the latent SCM.

Original: To the best of our knowledge, ours is the first work to address the problem of  causal discovery in linear 
Revised: To the best of our knowledge, ours is the first work to address the problem of Bayesian causal discovery in linear Gaussian latent

Original: Gaussian latent SCMs from low-level data, where causal variables are unobserved.
Revised: SCMs from low-level data, where causal variables are unobserved.

Original: We make the following contributions:
Revised: Our contributions are as follows:

Original: • We propose a general algorithm for Bayesian causal discovery in the latent space of a generative model , learning a distribution over causal variables, structure and parameters in linear Gaussian latent SCMs with random, known interventions.
Revised: • We propose a general algorithm for Bayesian causal discovery in the latent space of a generative model , learning a distribution over causal variables, structure and parameters in linear Gaussian latent SCMs with random, known interventions.

Original: Figure 1 illustrates an overview of the proposed method.
Revised: Figure 1 illustrates an overview of the proposed method.

Original: • By learning the structure and parameters of a latent SCM, we implicitly induce a joint distribution over the causal variables.
Revised: • By learning the structure and parameters of a latent SCM, we implicitly induce a joint distribution over the causal variables.

Original: Hence, sampling from this distribution is equivalent 
Revised: Hence, sampling from this distribution is equivalent to ancestral sampling through the latent SCM. As such, we address a challenging, simultane-

Original: Posterior overnoise variables ~
Revised: Posterior overnoise variables ~

Original: Po s t e r i
Revised: Po s t e r i

Original: o r o v e r  e
Revised: o r o v e r  e

Original: d g e w e
Revised: d g e w e

Original: i g h t s ~
Revised: i g h t s ~

Original: Po s t e r i
Revised: Po s t e r i

Original: o r o v e r  n o d e o r d
Revised: o r o v e r  n o d e o r d

Original: e
Revised: e

Original: r
Revised: r

Original: i n g s ~  

Revised: i n g s ~

Original: + +
Revised: + +

Original: MLP
Revised: MLP

Original: Ancestral sampling
Revised: Ancestral sampling

Original: MLP
Revised: MLP

Original: Ancestral sampling 

Revised: Ancestral sampling

Original: Latent SCM 

Revised: Latent SCM

Original: Intervened SCM
Revised: Intervened SCM

Original: Generated low-level data to ancestral sampling through the latent SCM. As such, we address a hard, simultaneous optimization problem that is often encountered during causal discovery in latent space: one cannot find the right graph without the right causal variables, and vice versa.
Revised: Generated low-level data ous optimization problem that is often encountered during causal discovery in latent space: one cannot find the right graph without the right causal variables, and vice versa.

Original: • On a synthetically generated dataset and an image dataset used to benchmark causal model performance (Ke et al. ), we evaluate our method along three axes – uncovering causal variables, structure and parameters – consistently outperforming baselines.
Revised: • On a synthetically generated dataset and an image dataset used to benchmark causal model performance (Ke et al., 2021), we evaluate our method along three axes – uncovering causal variables, structure, and parameters – consistently outperforming baselines.

Original: We demonstrate our model’s ability to perform image generation from unseen interventional distributions.
Revised: We demonstrate its ability to perform image generation from unseen interventional distributions.

Original:  P RELIMINARIES 2. S TRUCTURAL C AUSAL M ODELS
Revised:  P RELIMINARIES 2. S TRUCTURAL C AUSAL M ODELS

Original: z
Revised: z

Original: G Θ
Revised: G Θ

Original: N
Revised: N

Original: A Structural Causal Model (SCM) is defined by a set of equations which represent the mechanisms by which each endogenous variable z i depends on its direct causes z G pa ( i ) and a corresponding exogenous noise variable
Revised: A Structural Causal Model (SCM) is defined by a set of equations which represent the mechanisms by which each endogenous variable Z i depends on its direct causes Z G pa ( i ) and a corresponding exogenous noise variable

Original: ϵ i .
Revised: ϵ i .

Original: The direct causes are subsets of other endogenous variables.
Revised: The direct causes are subsets of other endogenous variables.

Original: If the causal parent assignment is assumed to be acyclic, then an SCM is associated with a Directed Acyclic Graph (DAG)
Revised: If the causal parent assignment is assumed to be acyclic, then an SCM is associated with a Directed Acyclic Graph (DAG)

Original: G =
Revised: G =

Original: ( V, E ) , where V corresponds to the endogenous variables and E encodes direct cause-effect relationships.
Revised: ( V, E ) , where V corresponds to the endogenous variables and E encodes direct cause-effect relationships.

Original: The exact value  taken on by a causal variable z i , is given by local causal mechanisms f i conditional on  z G pa ( i ) , the parameters Θ i , and the node’s noise variable
Revised: The exact value z i taken on by a causal variable Z i , is given by local causal mechanisms f i conditional on the values of its parents z G pa ( i ) , the parameters Θ i , and the node’s noise variable

Original: ϵ i , as given in equation 1.
Revised: ϵ i , as given in equation 1.

Original: For linear Gaussian additive noise SCMs with equal noise variance, which is the setting that we focus on in this work, all f i ’s are linear functions, and Θ denotes the weighted adjacency matrix W , where each W ji is the edge weight from j → i .
Revised: For linear Gaussian additive noise SCMs with equal noise variance, i.e., the setting that we focus on in this work, all f i ’s are linear functions, and Θ denotes the weighted adjacency matrix W , where each W ji is the edge weight from j → i .

Original: The linear Gaussian additive noise SCM thus reduces to equation 2.
Revised: The linear Gaussian additive noise SCM thus reduces to equation 2,

Original: [Equation]
Revised: [Equation]

Original: 2. C AUSAL D ISCOVERY
Revised: 2. C AUSAL D ISCOVERY

Original: Structure learning in prior work refers to learning a DAG according to some optimization criterion with or without the notion of causality (e.g., He et al.
Revised: Structure learning in prior work refers to learning a DAG according to some optimization criterion with or without the notion of causality (e.g., He et al.

Original: The task of causal discovery on the other hand, is more specific in that it refers to learning the structure (also parameters, in some cases) of SCMs, and subscribes to causality and interventions like that of Pearl (2009).
Revised: The task of causal discovery on the other hand, is more specific in that it refers to learning the structure (also parameters, in some cases) of SCMs, and subscribes to causality and interventions like that of Pearl (2009).

Original: That is, the methods aim to estimate ( G , Θ) .
Revised: That is, the methods aim to estimate ( G , Θ) .

Original: These approaches often resort to modular likelihood scores over causal variables – like the BGe score (Geiger & Heckerman, 1994; Kuipers et al., 2022) and BDe score (Heckerman et al., 1995) – to learn the right structure.
Revised: These approaches often resort to modular likelihood scores over causal variables – like the BGe score (Geiger and Heckerman, 1994; Kuipers et al., 2022) and BDe score (Heckerman et al., 1995) – to learn the right structure.

Original: However,all these methods  assumethat a dataset of  causal variables are observed.
Revised: However, these methods all assume a dataset of observed causal variables.

Original: These approaches either obtain a maximum likelihood estimate:
Revised: These approaches either obtain a maximum likelihood estimate,

Original: [Equation]
Revised: [Equation]

Original: or in the case of Bayesian causal discovery (Heckerman et al., 1997), variational inference is typically used to approximate a joint posterior distribution q ϕ ( G , Θ) to the true posterior p ( G , Θ | Z ) by minimizing the KL divergence between the two: [Equation]
Revised: or in the case of Bayesian causal discovery (Heckerman et al., 1997), variational inference is typically used to approximate a joint posterior distribution q ϕ ( G , Θ) to the true posterior p ( G , Θ | Z ) by minimizing the KL divergence between the two,

Original: [Equation]
Revised: [Equation]

Original: ) where p ( G , Θ) is a prior over the structure and parameters of the SCM – possibly encoding DAGness, sparse connections, or low-magnitude edge weights.
Revised: where p ( G , Θ) is a prior over the structure and parameters of the SCM – possibly encoding DAGness.

Original: Figure 2 shows the Bayesian Network (BN) over which inference is performed for causal discovery tasks.
Revised: Figure 2 shows the Bayesian Network (BN) over which inference is performed for causal discovery tasks.

Original: 2. L ATENT CAUSAL DISCOVERY
Revised: 2. L ATENT CAUSAL DISCOVERY

Original: x z
Revised: x z

Original: G Θ
Revised: G Θ

Original: N
Revised: N

Original: Ina more realistic scenario, the learner does not directly observe causal variables and they have to be learned from lowlevel data.
Revised: In more realistic scenarios, the learner does not directly observe causal variables and they must be learned from lowlevel data.

Original: The causal variables, structure, and parameters are part of a latent SCM.
Revised: The causal variables, structure, and parameters are part of a latent SCM.

Original: The goal of causal representation learning models is tobe able to perform inference of, and generation from, the true latent SCM.
Revised: The goal of causal representation learning models is to perform inference of, and generation from, the true latent SCM.

Original: Yang et al.
Revised: Yang et al.

Original: (2021) proposes a Causal VAE but is in a supervised setup where one has labels on causal variables and the focus is on disentanglement.
Revised: (2021) proposes a Causal VAE but is in a supervised setup where one has labels on causal variables and the focus is on disentanglement

Original: Kocaoglu et al.

Original: (2017) present causal generative models trained in an adversarial manner but assumes observations of causal variables.
Revised: (2017) present causal generative models trained in an adversarial manner but assumes observations of causal variables.

Original: Given the right causal structure as a prior, the work focuses on generation from conditional and interventional distributions.
Revised: Given the right causal structure as a prior, the work focuses on generation from conditional and interventional distributions.

Original: In both the causal representation learning and causal generative model scenarios mentioned above, the Ground Truth (GT) causal graph and parameters of the latent SCM are arbitrarily defined on real datasets and the setting is supervised.
Revised: In both the causal representation learning and causal generative model scenarios mentioned above, the Ground Truth (GT) causal graph and parameters of the latent SCM are arbitrarily defined on real datasets and the setting is supervised.

Original: Contrary to this, our setting is unsupervised and we are interested in recovering the GT underlying SCM and causal variables that generate the low-level observed data – we define this as the problem of latent causal discovery , and the BN over which we want to perform inference on is given in figure 3.
Revised: Contrary to this, our setting is unsupervised and we are interested in recovering the GT underlying SCM and causal variables that generate the low-level observed data – we define this as the problem of latent causal discovery , and the BN over which we want to perform inference on is given in figure 3.

Original: In the upcoming sections, we discuss related work, formulate our problem setup and propose an algorithm for Bayesian latent causal discovery, evaluate with experiments on causally created vector data and image data, and perform sampling from unseen interventional image distributions to showcase generalization of learned latent SCMs.
Revised: In the upcoming sections, we discuss related work, formulate our problem setup and propose an algorithm for Bayesian latent causal discovery, evaluate with experiments on causally created vector data and image data, and perform sampling from unseen interventional image distributions to showcase generalization of learned latent SCMs.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior work can be classified into Bayesian (Koivisto & Sood, 2004; Heckerman et al., 2006; Friedman & Koller, 2013) or maximum likelihood (Brouillard et al., 2020; Wei et al., 2020; Ng et al., 2022) methods, that learn the structure and parameters of SCMs using either score-based (Kass &
Revised: Prior work can be classified into Bayesian (Koivisto and Sood, 2004; Heckerman et al., 2006; Friedman and Koller, 2013) or maximum likelihood (Brouillard et al., 2020; Wei et al., 2020; Ng et al., 2022) methods, that learn the structure and parameters of SCMs using either score-based (Kass and

Original: Raftery, 1995; Barron et al., 1998; Heckerman et al., 1995) or constraint-based (Cheng et al., 2002;
Revised: Raftery, 1995; Barron et al., 1998; Heckerman et al., 1995) or constraint-based (Cheng et al., 2002;

Original: Lehmann & Romano, 2005) approaches.
Revised: Lehmann and Romano, 2005) approaches.

Original: Causal discovery and structure learning: Work in this category assume causal variables are observed and do not operate on low-level data (Spirtes et al., 1993; Viinikka et al., 2020;Yu et al.,
Revised: Causal discovery: Work in this category assume causal variables are observed and do not operate on low-level data (Spirtes et al., 2000; Viinikka et al., 2020;

Original: ; Zhang et al., 2022).
Revised: Yu et al., 2021; Zhang et al., 2022).

Original: Peters & B¨uhlmann (2014) prove identifiability of linear Gaussian SCMs with equal noise variances.
Revised: Peters and B ¨ uhlmann (2014) prove identifiability of linear Gaussian SCMs with equal noise variances.

Original: Bengio et al.
Revised: Bengio et al.

Original: (2019) use the speed of adaptation as a signal to learn the causal direction.
Revised: (2019) use the speed of adaptation as a signal to learn the causal direction.

Original: Ke et al.
Revised: Ke et al.

Original: (2019) explore learning causal models from unknown, while Scherrer et al.
Revised: (2019) explore learning causal models from unknown, while Scherrer et al.

Original: ; Tigas et al.
Revised: ; Tigas et al.

Original: ; Agrawal et al.
Revised: ; Agrawal et al.

Original: ; Toth et al.
Revised: ; Toth et al.

Original: (2022) focus on active learning and experimental design setups on how to perform interventions to efficiently learn causal models.
Revised: (2022) focus on active learning and experimental design setups on how to perform interventions to efficiently learn causal models.

Original: Transformer (Vaswani et al., 2017) based approach learns structure from synthetic datasets and generalize to naturalistic graphs (Ke et al., 2022).
Revised: Transformer (Vaswani et al., 2017) based approach learns structure from synthetic datasets and generalize to naturalistic graphs (Ke et al., 2022).

Original: Zheng et al.
Revised: Zheng et al.

Original: (2018) introduce an acyclicity constraint that penalizes cyclic graphs, thereby restricting search close to the DAG space.
Revised: (2018) introduce an acyclicity constraint that penalizes cyclic graphs, thereby restricting search close to the DAG space.

Original: Lachapelle et al.
Revised: Lachapelle et al.

Original: (2019) leverages this constraint to learn DAGs in nonlinear SCMs.
Revised: (2019) leverages this constraint to learn DAGs in nonlinear SCMs.

Original: Pamfil et al.
Revised: Pamfil et al.

Original: ; Lippe et al.
Revised: ; Lippe et al.

Original: (2022) perform structure learning on temporal data.
Revised: (2022) perform structure learning on temporal data.

Original: Latent variable models with learned structure : GraphVAE
Revised: Structure learning with latent variables : Markham and Grosse-Wentrup (2020) introduces the concept of Measurement Dependence Inducing Latent Causal Models (MCM). The proposed algorithm finds a minimal-MCM that induces the dependencies between observed variables. However, similar to VAEs, the method assumes no causal links between latent variables. Kivva et al.

Revised: (2021) provides the conditions under which the number of latent variables and structure can be uniquely identified for discrete latent variables, given the adjacency matrix between the hidden and measurement variables has linearly independent columns.

Revised: Elidan et al.

Original: (He et al., 2019) learns the edges between latent variables without incorporating notions of causality.
Revised: (2000) detects the signature of hidden variables using semi-cliques and then performs structure learning using the structural-EM algorithm (Friedman, 1998) for discrete random variables. Anandkumar et al. (2012) and Silva et al. (2006) considers the identifiability of linear Bayesian Networks when some variables are unobserved.

Revised: In the former work, the identifiability results hold only for particular classes of DAGs which follow certain structural constraints.

Revised: Assuming non-Gaussian noise and that certain sets of latents have a lower bound on the number of pure measurement child variables, Xie et al.

Revised: (2020) proposes the GIN condition to identify the structure between latent confounders.

Original:  Brehmer et al.
Revised: The formulation in the above works involves SCMs where some variables are observed and the others are unobserved. In contrast, the entire SCM is latent in our setup. Finally, GraphVAE (He et al., 2019) learns a structure between latent variables but does not incorporate notions of causality. Supervised causal representation learning : Brehmer et al.

Original: (2022) present identifiability theory for learning causal representations and propose a practical algorithm for this under the assumption of having access to pairs of observational and interventional data. 
Revised: (2022) present identifiability theory for learning causal representations and propose an algorithm assuming access to pairs of observational and interventional data. Ahuja et al. (2022a) studies identifiability in a similar setup with the use of sparse perturbations.

Original: Supervised causal representation learning : Kocaoglu et al.
Revised: Ahuja et al. (2022c) discusses identifiability for causal representation learning when one has access to interventional data. Kocaoglu et al.

Original: ; Shen et al.
Revised: ; Shen et al.

Original: ; Moraffah et al.
Revised: ; Moraffah et al.

Original: (2020) introduce generative models that use an SCM-based prior in latent space.
Revised: (2020) introduce generative models that use an SCM-based prior in latent space.

Original: In Shen et al.
Revised: In Shen et al.

Original: (2020), the goal is to learn causally disentangled variables.
Revised: (2020), the goal is to learn causally disentangled variables.

Original: Yang et al.
Revised: Yang et al.

Original: (2021) learn a DAGon CelebA and a causally generated pendulum image dataset but assume complete access to the causal variables.
Revised: (2021) learn a DAG but assumes complete access to the causal variables.

Original: Lopez-Paz et al.
Revised: Lopez-Paz et al.

Original: (2016) establishes observable causal footprints in images.
Revised: (2016) establishes observable causal footprints in images.

Revised: We refer the reader to section A.9 for a discussion on identifiability.

Original:  L EARNING LATENT SCM S FROM LOW - LEVEL DATA 4. 
Revised:  L EARNING LATENT SCM S FROM LOW - LEVEL DATA 4. P ROBLEM S CENARIO

Original: P [Equation] S

Original: [Equation]

Original: We are given a dataset D
Revised: We are given a dataset D

Original: = { x (1) , ..., x ( N ) } , where each x ( i ) is a high-dimensional observed data – for simplicity, we assume x ( i ) is a vector in R D but the setup extends to other inputs as well (like images, as we will see in the next section).
Revised: = { x (1) , ..., x ( N ) } , where each x ( i ) is a high-dimensional observed data – for simplicity, we assume x ( i ) is a vector in R D but the setup extends to other inputs as well (like images, as we will see in the next section).

Original: We assume that there exist latent variables
Revised: We assume that there exist latent variables

Original: Z = { z ( i
Revised: Z = { z ( i

Original: ) ∈ R d } Ni =1 with d ≤ D , that explain the data, and these latent variables have an SCM with structure
Revised: ) ∈ R d } Ni =1 with d ≤ D , that explain the data, and these latent variables have an SCM with structure

Original: G GT and parameters Θ GT associated with them.
Revised: G GT and parameters Θ GT associated with them.

Original: We wish to invert the data generation process g : ( Z , G , Θ) → D where the causal variables Z are in the latent space.
Revised: We wish to invert the data generation process g : ( Z , G , Θ) → D where the causal variables Z are in the latent space.

Original: In the setting, we also have access to the intervention targets I
Revised: In the setting, we also have access to the intervention targets I

Original: = {I ( i ) } Ni =1 where each I ( i ) ∈ { 0 , 1 } d .
Revised: = {I ( i ) } Ni =1 where each I ( i ) ∈ { 0 , 1 } d .

Original: The j th dimension of
Revised: The j th dimension of

Original: I ( i ) takes a value of 1 if node j was intervened on in data sample i , and 0 otherwise.
Revised: I ( i ) takes a value of 1 if node j was intervened on in data sample i , and 0 otherwise.

Original: 4. G ENERAL METHOD
Revised: 4. G ENERAL METHOD

Original: We aim to obtain a posterior estimate over the entire latent SCM, p ( Z , G , Θ | D ) .
Revised: We aim to obtain a posterior estimate over the entire latent SCM, p ( Z , G , Θ | D ) .

Original: Computing the true posterior analytically requires calculating the marginal likelihood p ( D ) which gets quickly intractable due to the number of possible DAGs growing super-exponentially with respect to the number of nodes.
Revised: Computing the true posterior analytically requires calculating the marginal likelihood p ( D ) which gets quickly intractable due to the number of possible DAGs growing super-exponentially with respect to the number of nodes.

Original: Thus, we resort to variational inference (Blei et al., 2017) that provides a tractable way to learn an approximate posterior q ϕ ( Z , G , Θ) with variational parameters ϕ , close to the true posterior p ( Z , G , Θ | D ) by maximizing the Evidence Lower Bound (ELBO):
Revised: Thus, we resort to variational inference (Blei et al., 2017) that provides a tractable way to learn an approximate posterior q ϕ ( Z , G , Θ) with variational parameters ϕ , close to the true posterior p ( Z , G , Θ | D ) by maximizing the Evidence Lower Bound (ELBO),

Original: [Equation]
Revised: [Equation]

Original: (5) where p ( Z , G , Θ) is the prior, p ψ ( D | Z , G , Θ) is the likelihood model with parameters ψ , the likelihood model maps the latent variables Z to high-dimensional vectors X .
Revised: where p ( Z , G , Θ) is the prior, p ψ ( D | Z , G , Θ) is the likelihood model with parameters ψ , the likelihood model maps the latent variables Z to high-dimensional vectors X .

Original: An approach to learn this posterior could be to factorize it as:
Revised: An approach to learn this posterior could be to factorize it as

Original: [Equation]
Revised: [Equation]

Original: Given a way to obtain q ϕ ( Z ) , the conditional q ϕ ( G , Θ | Z ) can be obtained using existing Bayesian structure learning methods.
Revised: Given a way to obtain q ϕ ( Z ) , the conditional q ϕ ( G , Θ | Z ) can be obtained using existing Bayesian structure learning methods.

Original: Otherwise, one has to perform a hard simultaneous optimization which would require alternating optimizations on Z and on ( G , Θ) given an estimate of Z .
Revised: Otherwise, one has to perform a hard simultaneous optimization which would require alternating optimizations on Z and on ( G , Θ) given an estimate of Z .

Original: Difficulty of such an alternate optimization is discussed in Brehmer et al.
Revised: Difficulty of such an alternate optimization is discussed in Brehmer et al.

Original: Alternate factorization of the posterior : Rather than factorizing as in equation 6, we propose to factorize according to the BN in figure 3. This is given by q ϕ ( Z , G , Θ)=
Revised: Alternate factorization of the posterior : Rather than factorizing as in equation 6, we propose to only introduce a variational distribution q ϕ ( G , Θ) over structures and parameters, so that the approximation is given by q ϕ ( Z , G , Θ)

Original: q ϕ ( Z | G , Θ) · q ϕ ( G , Θ) .
Revised: = p ( Z | G , Θ) · q ϕ ( G , Θ) .

Original: The advantage of this factorization is that the distribution  over Z is completely determined from the SCM given ( G , Θ) and exogenous noise variables (assumed to be Gaussian).Thus, the prior p ( Z | G , Θ) and the posterior p ( Z | G , Θ , D ) = q ϕ ( Z | G , Θ) are identical.
Revised: The advantage of this factorization is that the distribution p ( Z | G , Θ) over Z is completely determined from the SCM given ( G , Θ) and exogenous noise variables (assumed to be Gaussian).

Original: This conveniently avoids the hard simultaneous optimization problem mentioned above since optimizing for q ϕ ( Z ) is not necessary.
Revised: This conveniently avoids the hard simultaneous optimization problem mentioned above since optimizing for q ϕ ( Z ) is not necessary.

Original: Equation 5 then can be simplified as:
Revised: Hence, equation 5 simplifies to:

Original: [Equation]
Revised: [Equation]

Original: Such a posterior can be used to obtain an SCM by sampling G and Θ from the approximated posterior.
Revised: Such a posterior can be used to obtain an SCM by sampling G and Θ from the approximated posterior.

Original: As long as the samples G are always acyclic, one can perform ancestral sampling through the SCM to obtain samples corresponding to the causal variables ˆ z ( i ) .
Revised: As long as the samples G are always acyclic, one can perform ancestral sampling through the SCM to obtain samples corresponding to the causal variables ˆ z ( i ) .

Original: For additive noise models like in equation 2, these samples are already reparameterized and differentiable with respect to their parameters.
Revised: For additive noise models like in equation 2, these samples are already reparameterized and differentiable with respect to their parameters.

Original: The samples of causal variables are then fed to the likelihood model to predict  ˆ x ( i ) to reconstruct the observed data x ( i ) .
Revised: The samples of causal variables are then fed to the likelihood model to predict samples ˆ x ( i ) that reconstruct the observed data x ( i ) .

Original: 4. P OSTERIOR PARAMETERIZATIONS AND PRIORS
Revised: 4. P OSTERIOR PARAMETERIZATIONS AND PRIORS

Original: For linear Gaussian latent SCMs, which is the focus of this work, learning a posterior over ( G , Θ) is equivalent to learning q ϕ ( W, Σ) – a posterior over weighted adjacency matrices W and noise covariances Σ .
Revised: For linear Gaussian latent SCMs, which is the focus of this work, learning a posterior over ( G , Θ) is equivalent to learning q ϕ ( W, Σ) – a posterior over weighted adjacency matrices W and noise covariances Σ .

Original: We follow an approach similar to (Cundy et al., 2021).
Revised: We follow an approach similar to (Cundy et al., 2021).

Original: We express W via a permutation matrix P 1 and a lower triangular edge weight matrix L , according to W = P T L T P .
Revised: We express W via a permutation matrix P 1 and a lower triangular edge weight matrix L , according to W = P T L T P .

Original: Here, L is defined in the space of all weighted adjacency matrices with a fixed node ordering where node j can be a parent of node i only if j > i .
Revised: Here, L is defined in the space of all weighted adjacency matrices with a fixed node ordering where node j can be a parent of node i only if j > i .

Original: Search over permutations corresponds to search over different node orderings and thus, W and Σ parameterize the space of SCMs.
Revised: Search over permutations corresponds to search over different node orderings and thus, W and Σ parameterize the space of SCMs.

Original: Further, we factorize the approximate posterior q ϕ ( P, L, Σ) as:
Revised: Further, we factorize the approximate posterior q ϕ ( P, L, Σ) as

Original: [Equation]
Revised: [Equation]

Original: Combining equation 7 and 8 leads to the following ELBO which has to be maximized (derived in
Revised: Combining equation 7 and 8 leads to the following ELBO which has to be maximized (derived in

Original: A.1), and the overall algorithm for Bayesian latent causal discovery is summarized in algorithm 1.
Revised: A.1), and the overall algorithm for Bayesian latent causal discovery is summarized in algorithm 1,

Original: [Equation]
Revised: [Equation]

Original: Distribution over ( L, Σ) :
Revised: Distribution over ( L, Σ) :

Original: The posterior distribution q ϕ ( L, Σ) has ( d ( d − 1)+1) elements to be learnt in the equal noise variance setting.
Revised: The posterior distribution q ϕ ( L, Σ) has ( d ( d − 1)+1) elements to be learnt in the equal noise variance setting.

Original: This is parameterized as a diagonal covariance normal distribution.
Revised: This is parameterized as a diagonal covariance normal distribution.

Original: For the prior p ( L ) over the edge weights, we promote sparse DAGs by using a horseshoe prior (Carvalho et al., 2009), similar to Cundy et al.
Revised: For the prior p ( L ) over the edge weights, we promote sparse DAGs by using a horseshoe prior (Carvalho et al., 2009), similar to Cundy et al.

Original: A Gaussian prior is defined over log Σ .
Revised: A Gaussian prior is defined over log Σ .

Revised: Distribution over P : Since the values of P are discrete, performing a discrete optimization is combinatorial and becomes quickly intractable with increasing d .

Revised: This can be handled by relaxing the discrete permutation learning problem to a continuous optimization problem.

Revised: This is commonly done by introducing a Gumbel-Sinkhorn (Mena et al., 2018) distribution and where one has to calculate S (( T + γ ) /τ ) , where T is the parameter of the Gumbel-Sinkhorn, γ is a matrix of standard

Revised: Gumbel noise, and τ is a fixed temperature parameter.

Revised: The logits T are predicted by passing the

Original: Input: D , I
Revised: Input: D , I

Original: Output: G , Θ , Z 1: Initialize q ϕ ( L, Σ) , MLP ϕ
Revised: Output: G , Θ , Z 1: Initialize q ϕ ( L, Σ) , MLP ϕ

Original: ( T ) , p ψ ( X | Z ) ,
Revised: ( T ) , p ψ ( X | Z ) ,

Original: τ and set learning rate α 2: for num epochs do 3: ( (cid:98) L, (cid:98) Σ)
Revised: τ and set learning rate α 2: for num epochs do 3: ( (cid:98) L, (cid:98) Σ)

Original: ∼ q ϕ
Revised: ∼ q ϕ

Original: ( L, Σ) 4:
Revised: ( L, Σ) 4:

Original: T ← MLP ϕ ( T ) ( (cid:98) L, (cid:98) Σ)
Revised: T ← MLP ϕ ( T ) ( (cid:98) L, (cid:98) Σ)

Original: ▷ Compute logits for sampling from q ϕ ( P | L, Σ) 5: γ ∈ R d × d ∼ standard Gumbel 6: (cid:98) P soft ← Sinkhorn (( T + γ ) /τ )
Revised: ▷ Compute logits for sampling from q ϕ ( P | L, Σ) 5: γ ∈ R d × d ∼ standard Gumbel 6: (cid:98) P soft ← Sinkhorn (( T + γ ) /τ )

Original: 7: (cid:98) P hard ← Hungarian ( (cid:98) P soft  ) 8: (cid:99) W ← (cid:98) P T (cid:98)
Revised: 7: (cid:98) P hard ← Hungarian ( (cid:98) P soft ; τ → 0 ) 8: (cid:99) W ← (cid:98) P T (cid:98)

Original: L T (cid:98) P 9: for i ← 1 to N do 10:
Revised: L T (cid:98) P 9: for i ← 1 to N do 10:

Original: C ( i ) ← argwhere ( I ( i ) = 1) 11: (cid:102) W = copy ( (cid:99) W ) 12: (cid:102) W
Revised: C ( i ) ← argwhere ( I ( i ) = 1) 11: (cid:102) W = copy ( (cid:99) W ) 12: (cid:102) W

Original: [: , C ( i ) ]
Revised: [: , C ( i ) ]

Original: ←▷ Mutated weighted adjacency matrix according to I ( i ) 13: (cid:99)
Revised: ←▷ Mutated weighted adjacency matrix according to I ( i ) 13: (cid:99)

Original: W I ( i )
Revised: W I ( i )

Original: ← (cid:102) W 14:
Revised: ← (cid:102) W 14:

Original: ˆ z ( i ) ← AncestralSample ( (cid:99) W
Revised: ˆ z ( i ) ← AncestralSample ( (cid:99) W

Original: I ( i ) , (cid:98) Σ) 15: end for 16: ˆ Z ← { ˆ z ( i ) } Ni = 17: ˆ D ∼ p ψ ( X | Z ) 18: ψ ← ψ + α · ∇ ψ ( L ( ψ, ϕ ))
Revised: I ( i ) , (cid:98) Σ) 15: end for 16: ˆ Z ← { ˆ z ( i ) } Ni = 17: ˆ D ∼ p ψ ( X | Z ) 18: ψ ← ψ + α · ∇ ψ ( L ( ψ, ϕ ))

Original: ▷ Update network parameters 19: ϕ ← ϕ + α · ∇ ϕ ( L ( ψ, ϕ )) 20: end for 21: return binary ( (cid:99) W ) , ( (cid:99) W, (cid:98) Σ) , ˆZ 
Revised: ▷ Update network parameters 19: ϕ ← ϕ + α · ∇ ϕ ( L ( ψ, ϕ )) 20: end for 21: return binary ( (cid:99) W ) , ( (cid:99) W, (cid:98) Σ) , ˆZ predicted ( L, Σ) through an MLP.

Original: Distribution over P : Since the values of P are discrete, performing a discrete optimization is combinatorial and becomes quickly intractable with increasing d .

Original: This can be handled by relaxing the discrete permutation learning problem to a continuous optimization problem.

Original: This is commonly done by introducing a Gumbel-Sinkhorn (Mena et al., 2018) distribution and where one has to calculate S (( T + γ ) /τ ) , where T is the parameter of the Gumbel-Sinkhorn, γ is a matrix of standard

Original: Gumbel noise, and τ is a temperature parameter.

Original: The logits T are predicted by passing the predicted ( L, Σ) through an MLP.

Original: In the limit of infinite iterations and as τ → 0 , sampling from the distribution returns a doubly stochastic matrix.
Revised: In the limit of infinite iterations and as τ → 0 , sampling from the distribution returns a doubly stochastic matrix.

Original: During the forward pass, a hard permutation P is obtained by using the Hungarian algorithm (Kuhn, 1955) which allows τ → 0 .
Revised: During the forward pass, a hard permutation P is obtained by using the Hungarian algorithm (Kuhn, 1955) which allows τ → 0 .

Original: During the backward pass, a soft permutation is used to calculate gradients similar to (Cundy et al., 2021; Charpentier et al., 2022).
Revised: During the backward pass, a soft permutation is used to calculate gradients similar to (Cundy et al., 2021; Charpentier et al., 2022).

Original: We use a uniform prior p ( P ) over permutations.
Revised: We use a uniform prior p ( P ) over permutations.

Original:  E XPERIMENTS AND EVALUATION
Revised:  E XPERIMENTS AND EVALUATION

Original: We perform experiments to evaluate the learned ( Z , G , Θ) of the true linear Gaussian latent SCM from high-dimensional data.
Revised: We perform experiments to evaluate the learned ( Z , G , Θ) of the true linear Gaussian latent SCM from high-dimensional data.

Original: We aim to highlight the performance of our proposed method on latent causal discovery.
Revised: We aim to highlight the performance of our proposed method on latent causal discovery.

Original: As proper evaluation in such a setting would require access to the GT causal graph that generated the high-dimensional observations, we test our method against baselines on synthetically generated vector data and in the realistic case of learning the SCM from pixels in the chemistry environment dataset of (Ke et al., 2021), both of which have a GT causal structure to be compared with.
Revised: As proper evaluation in such a setting would require access to the GT causal graph that generated the high-dimensional observations, we test our method against baselines on synthetically generated vector data and in the realistic case of learning the SCM from pixels in the chemistry environment dataset of (Ke et al., 2021), both of which have a GT causal structure to be compared with.

Original: Further, we evaluate the ability of our model to sample images from unseen interventional distributions.
Revised: Further, we evaluate the ability of our model to sample images from unseen interventional distributions.

Original: Baselines : Since we are, to the best of our knowledge, the first to study this setting of  learning  latent SCMs from low level observations, we are not aware of baseline methods that solve this task.
Revised: Baselines : Since we are, to the best of our knowledge, the first to study this setting of Bayesian learning of latent SCMs from low level observations, we are not aware of baseline methods that solve this task.

Original: However, we compare our approach against two baselines: (i) Against VAE that has a marginal independence assumption between latent variables and thus have a predefined structure in the latent space, and (ii) against GraphVAE (He et al., 2019) that learns a structure between latent variables.
Revised: However, we compare our approach against two baselines: (i) Against VAE that has a marginal independence assumption between latent variables and thus have a predefined structure in the latent space, and (ii) against GraphVAE (He et al., 2019) that learns a structure between latent variables.

Original: For all baselines, we treat the learned latent variables as causal variables and compare the recovered 
Revised: For all baselines, we treat the learned latent variables as causal variables and compare the recovered structure, parameters, and causal variables recovered.

Revised: Since GraphVAE does not learn the parameters, we fix the edge weight over all predicted edges to be 1.

Revised: Evaluation metrics : To evaluate the learned structure , we use two metrics commonly used in the literature – the expected Structural Hamming Distance ( E -SHD , lower is better) obtains the SHD (number of edge flips, removals, or additions) between the predicted and GT graph and then takes an expectation over SHDs of posterior DAG samples, and the Area Under the Receiver Operating

Original: SHD 0 . 00 .
Revised: SHD 0 . 00 .

Original: AUROC 2571 

Revised: AUROC 2571

Original: MCC( z , z ) 0 .0 .1 .1 .
Revised: MCC( z , z ) 0 .0 .1 .1 .

Original: MSE( L , L ) d
Revised: MSE( L , L ) d

Original: = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R112d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 .
Revised: = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R112d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 .

Original: 0d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 42571 0d
Revised: 0d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 42571 0d

Original: = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E ROurs
Revised: = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E ROurs

Original: GraphVAE
Revised: GraphVAE

Original: VAEstructure, parameters, and causal variables recovered. Since GraphVAE does not learn the parameters, we fix the edge weight over all predicted edges to be 1. Evaluation metrics : To evaluate the learned structure , we use two metrics commonly used in the literature – the expected Structural Hamming Distance ( E -SHD , lower is better) obtains the SHD (number of edge flips, removals, or additions) between the predicted and GT graph and then takes an expectation over SHDs of posterior DAG samples, and the Area Under the Receiver Operating
Revised: VAE

Original: Characteristic curve ( AUROC , higher is better) where a score of 0.5 corresponds to a random DAG baseline.
Revised: Characteristic curve ( AUROC , higher is better) where a score of 0.5 corresponds to a random DAG baseline.

Original: To evaluate the learned parameters of the linear Gaussian latent SCM, we use the Mean
Revised: To evaluate the learned parameters of the linear Gaussian latent SCM, we use the Mean

Original: Squared Error ( MSE , lower is better) between the true and predicted edge weights.
Revised: Squared Error ( MSE , lower is better) between the true and predicted edge weights.

Original: To evaluate the learned causal variables , we use the Mean Correlation Coefficient ( MCC , higher is better) following Hyvarinen & Morioka (2017); Zimmermann et al.
Revised: To evaluate the learned causal variables , we use the Mean Correlation Coefficient ( MCC , higher is better) following Hyvarinen and Morioka (2017); Zimmermann et al.

Original: (2021) and Ahuja et al.
Revised: (2021) and Ahuja et al.

Original: (2022) which calculates a score between the true and predicted causal variables.
Revised: (2022b) which calculates a score between the true and predicted causal variables.

Original: See appendix A.3 and A.7 for training curves and more extensive evaluations of the experiments along other metrics.
Revised: See appendix A.3 and A.8 for training curves and more extensive evaluations of the experiments along other metrics.

Original: Our results are presented over 20 random DAGs with a learning rate of 0 . 0008 . All our implementations are in JAX (Bradbury et al., 2018).
Revised: All our implementations are in JAX (Bradbury et al., 2018) and results are presented over 20 random DAGs.

Original: 5. E XPERIMENTS ON S YNTHETIC D ATA
Revised: 5. E XPERIMENTS ON S YNTHETIC D ATA

Original: We evaluate our proposed method with the baselines on synthetically generated dataset, where we have complete control over the data generation procedure.
Revised: We evaluate our proposed method with the baselines on synthetically generated dataset, where we have complete control over the data generation procedure.

Original: 5.1. S YNTHETIC V ECTOR D ATA G ENERATION
Revised: 5.1. S YNTHETIC V ECTOR D ATA G ENERATION

Original: To generate high-dimensional vector data with a known causal structure, we first generate a random
Revised: To generate high-dimensional vector data with a known causal structure, we first generate a random

Original: DAG and linear SCM parameters, and generate true causal variables by ancestral sampling.
Revised: DAG and linear SCM parameters, and generate true causal variables by ancestral sampling.

Original: This is then used to generate corresponding high-dimensional dataset with a random projection function.
Revised: This is then used to generate corresponding high-dimensional dataset with a random projection function.

Original: Generating the DAG and causal variables : Following many works in the literature, we sample random Erd˝os–R´enyi (ER) DAGs (Erdos et al., 1960) with degrees in { 1 , 2 , 4 } to generate the DAG.
Revised: Generating the DAG and causal variables : Following many works in the literature, we sample random Erd˝os–R´enyi (ER) DAGs (Erdos et al., 1960) with degrees in { 1 , 2 , 4 } to generate the DAG.

Original: For every edge in this DAG, we sample the magnitude of edge weights uniformly as | L |∼ U (0 . 5 , 2 . 0) and randomly sample the permutation matrix.
Revised: For every edge in this DAG, we sample the magnitude of edge weights uniformly as | L |∼ U (0 . 5 , 2 . 0) and randomly sample the permutation matrix.

Original: We perform ancestral sampling through this random
Revised: We perform ancestral sampling through this random

Original: DAG with intervention targets
Revised: DAG with intervention targets

Original: I , to obtain Z and then project it to D dimensions to obtain { x ( i ) } Ni =1 .
Revised: I , to obtain Z and then project it to D dimensions to obtain { x ( i ) } Ni =1 .

Original: Generating high-dimensional vectors from causal variables : We consider two different cases of generating the high-dimensional data from the causal variables obtained in the previous step: (i) x
Revised: Generating high-dimensional vectors from causal variables : We consider two different cases of generating the high-dimensional data from the causal variables obtained in the previous step: (i) x

Original: ( i ) is a random linear projection of causal variables, z ( i ) , from R d to R D , according to x = z ˜P , where ˜P ∈ R d × D is a random projection matrix.
Revised: ( i ) is a random linear projection of causal variables, z ( i ) , from R d to R D , according to x = z ˜P , where ˜P ∈ R d × D is a random projection matrix.

Original: (ii) x
Revised: (ii) x

Original: ( i ) is a nonlinear projection [Figure] [Figure] SHD d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R
Revised: ( i ) is a nonlinear projection of causal variables, z ( i ) , modeled by a 3-layer MLP.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: AUROC d

Original: [Figure]

Original: E R

Original: [Figure]

Original: d

Original: [Figure]

Original: 5.1. R ESULTS ON S YNTHETIC V ECTOR D ATA
Revised: 5.1. R ESULTS ON S YNTHETIC V ECTOR D ATA

Original: Results on linear projection of causal variables : We present results on the learned causal variables, structure, and parameters in two scenarios: (i) when the true node ordering or permutation is given (e.g., as in He et al.
Revised: Results on linear projection of causal variables : We present results on the learned causal variables, structure, and parameters in two scenarios: (i) when the true node ordering or permutation is given (e.g., as in He et al.

Original: and (ii) when the node ordering is not given and one has to additionally also infer the permutation P .
Revised: and (ii) when the node ordering is not given and one has to additionally also infer the permutation P .

Original: For d = 5 , 10 , 20 nodes projected to D = 100 dimensions, we evaluate our algorithm on synthetic ER-1, ER-2, and ER-4 DAGs.
Revised: For d = 5 , 10 , 20 nodes projected to D = 100 dimensions, we evaluate our algorithm on synthetic ER-1, ER-2, and ER-4 DAGs.

Original: The model was trained forepochs{+d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R
 SHD d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 . AUROC d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 42571 
 MCC( z , z ) d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 .0 .1 .1 .+}
Revised: The model was trained {+d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R
 SHD d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 . AUROC d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 42571 
 MCC( z , z ) d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 .0 .1 .1 .+}

Original:  so as to reach convergence.
Revised: MSE( L , L ) d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R246d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R0 .0 .0 .1 .d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R - 42571 0d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E ROurs GraphVAE VAE for 5000 epochs so as to reach convergence.

Original: Figure 4 summarizes the results for d = 5 nodes, for which we use 500 observational data points and 2000 interventional data points.
Revised: Figure 20 summarizes the results for d = 5 nodes, for which we use 500 observational data points and 2000 interventional data points.

Original: Of the 2000 interventional data, we generate 100 random interventional data points per set over 20 intervention sets 2 .
Revised: Of theinterventional data, we generate 100 random interventional data points per set over 20 intervention sets 2 .

Original: It can be seen that when permutation is given, the proposed method can recover the causal graph correctly in all the cases, achieving E -SHD of 0 and AUROC of 1.
Revised: It can be seen that when permutation is given, the proposed method can recover the causal graph correctly in all the cases, achieving E -SHD of 0 and AUROC of 1.

Original: When the permutation is learned, the proposed method still recovers the true causal structure very well.
Revised: When the permutation is learned, the proposed method still recovers the true causal structure very well.

Original: However, this is not the case with baseline methods of VAE and GraphVAE, which perform significantly worse on most of the metrics.
Revised: However, this is not the case with baseline methods of VAE and GraphVAE, which perform significantly worse on most of the metrics.

Original: Figure 11 and 12 (in Appendix) show the results for d = 10 and d = 20 nodes.
Revised: Figure 11 and 12 (in Appendix) show the results for d = 10 and d = 20 nodes.

Original: Results on nonlinear projection of causal variables : For d = 5 , 10 , 20 nodes projected to D =dimensions, we evaluate our algorithm on synthetic ER-1, ER-2, and ER-4 DAGs, given the permutation P .
Revised: Results on nonlinear projection of causal variables : For d = 5 , 10 , 20 nodes projected to D = 100 dimensions, we evaluate our algorithm on synthetic ER-1, ER-2, and ER-4 DAGs, given the permutation P .

Original: Figure 5 summarizes the results for 5 and 10 nodes.
Revised: Figure 5 summarizes the results for 5 and 10 nodes.

Original: As in the linear case, the proposed method recovers the true causal structure and the true causal variables, and is significantly better than the VAE and GraphVAE baselines on all the metrics considered.
Revised: As in the linear case, the proposed method recovers the true causal structure and the true causal variables, and is significantly better than the VAE and GraphVAE baselines on all the metrics considered.

Original: For experiments in this setting, we noticed empirically that learning the permutation is hard, and performs not so different from a null graph baseline (Figure 13).
Revised: For experiments in this setting, we noticed empirically that learning the permutation is hard, and performs not so different from a null graph baseline (Figure 13).

Original: This observation works that theoretically show the identifiability result that recovery of latent variables is possible only upto a permutation in latent causal models (Brehmer et al., 2022;
Revised: This observation complements the identifiability result that recovery of latent variables is possible only upto a permutation in latent causal models (Brehmer et al., 2022;

Original: Liu et al., 2022) for general nonlinear mappings between causal variables and low-level data.
Revised: Liu et al., 2022) for general nonlinear mappings between causal variables and low-level data.

Original: This supports our observation of not being able to learn the permutation in nonlinear projection settings – but once the permutation is given to the model, it can quickly recover the SCM (figures 5,13).
Revised: This supports our observation of not being able to learn the permutation in nonlinear projection settings – but once the permutation is given to the model, it can quickly recover the SCM (figures 5,13).

Original: Refer figure 14 (in Appendix) for results on d = 20 nodes.
Revised: Refer figure 14 (in Appendix) for results on d = 20 nodes.

Original: 5. R ESULTS ON LEARNING LATENT SCM S FROM PIXEL DATA
Revised: 5. R ESULTS ON LEARNING LATENT SCM S FROM PIXEL DATA

Original: Dataset and Setup : A major challenge with evaluating latent causal discovery models on images is that it is hard to obtain images with corresponding GT graph and parameters.
Revised: Dataset and Setup : A major challenge with evaluating latent causal discovery models on images is that it is hard to obtain images with corresponding GT graph and parameters.

Original: Other works (Kocaoglu et al., 2017; Yang et al., 2021; Shen et al., 2020) handle this by assuming the dataset is generated from certain causal variables (assumed to be attributes like gender, baldness, etc.) and a causal structure that is heuristically set by experts, usually in the CelebA dataset (Liu et al., 2015).
Revised: Other works (Kocaoglu et al., 2017; Yang et al., 2021; Shen et al., 2020) handle this by assuming the dataset is generated from certain causal variables (assumed to be attributes like gender, baldness, etc.) and a causal structure that is heuristically set by experts, usually in the CelebA dataset (Liu et al., 2015).

Original: This makes evaluation particularly noisy.
Revised: This makes evaluation particularly noisy.

Original: Given these limitations, we verify if our model can perform latent causal discovery by evaluating on images from the chemistry dataset proposed in Ke et al.
Revised: Given these limitations, we verify if our model can perform latent causal discovery by evaluating on images from the chemistry dataset proposed in Ke et al.

Revised: (2021) – a scenario where all GT factors are known.

Revised: We use the environment to generate blocks of different intensities according to a linear Gaussian latent SCM where the parent block colors affect the child block colors then obtain the corresponding images of blocks.

Revised: The dataset allows generating pixel data from random DAGs and linear SCMs.

Revised: For this step, we use the same technique to generate causal variables as in the synthetic dataset section.

Revised: Similar to experiments on nonlinear vector data, we are given the node ordering in this setting.

Original: d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 .2 .5 .7 .
Revised: d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 .2 .5 .7 .

Original: SHD d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R0 .0 .1 .
Revised: SHD d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R0 .0 .1 .

Original: AUROC d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R

Revised: AUROC d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R

Original: MCC( z , z ) d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 .
Revised: MCC( z , z ) d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 .

Original: MSE( L , L ) d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R0 1234d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R0 .0 .1 .d
Revised: MSE( L , L ) d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R0 1234d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R0 .0 .1 .d

Original: = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R2468d =
Revised: = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R2468d =

Original: 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R - 4 0 .0 .1 .
Revised: 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R - 4 0 .0 .1 .

Original: Ours
Revised: Ours

Original: GraphVAE
Revised: GraphVAE

Original: VAE
Revised: VAE

Original: (2021) – a scenario where all GT factors are known.

Original: We use the environment to generate blocks of different intensities according to a linear Gaussian latent SCM where the parent block colors affect the child block colors then obtain the corresponding images of blocks.

Original: The dataset allows generating pixel data from random DAGs and linear SCMs.

Original: For this step, we use the same technique to generate causal variables as in the synthetic dataset section.

Original: Results : We perform experiments to evaluate latent causal discovery from pixels and known interventions.
Revised: Results : We perform experiments to evaluate latent causal discovery from pixels and known interventions.

Original: The results are summarized in figure 6.
Revised: The results are summarized in figure 6.

Original: It can be seen that the proposed approach can recover the SCM significantly better than the baseline approaches in all the metrics even in the realistic dataset.
Revised: It can be seen that the proposed approach can recover the SCM significantly better than the baseline approaches in all the metrics even in the realistic dataset.

Original: In figure 7, we also assess the ability of the model to sample images from unseen interventions in the chemistry dataset by examining the generated images with GT interventional samples.
Revised: In figure 7, we also assess the ability of the model to sample images from unseen interventions in the chemistry dataset by examining the generated images with GT interventional samples.

Original: The matching intensity of each block corresponds to matching causal variables, which demonstrates model generalization.
Revised: The matching intensity of each block corresponds to matching causal variables, which demonstrates model generalization.

Original:  C ONCLUSION
Revised:  L IMITATIONS

Revised: Our approach makes a number of assumptions.

Revised: First, we assume the latent SCM is linear Gaussian.

Revised: However, removing this restriction might be crucial to make the approach more general.

Revised: Second, we assume access to the number of latent causal variables d .

Revised: Extensions must consider how to infer the number of latent variables.

Revised: We also make the assumption of known intervention targets whereas this might be restrictive for real-world applications.

Revised: Future work could overcome this limitation by inferring interventions as in H¨agele et al.

Revised: Finally, we have assumed feasibility of interventions and known causal orderings for some of our experiments.

Revised: However, in reality, some interventions could be infeasible and the causal ordering of latent variables might not be known.

Revised:  C ONCLUSION

Original: We presented a tractable approximate inference technique to perform Bayesian latent causal discovery that jointly infers the causal variables, structure and parameters of linear Gaussian latent SCMs under random, known interventions from low-level data.
Revised: We presented a tractable approximate inference technique to perform Bayesian latent causal discovery that jointly infers the causal variables, structure and parameters of linear Gaussian latent SCMs under random, known interventions from low-level data.

Original: The learned causal model is also shown to generalize to unseen interventions.
Revised: The learned causal model is also shown to generalize to unseen interventions.

Original: Our Bayesian formulation allows uncertainty quantification and mutual information estimation which is well-suited for extensions to active causal discovery.
Revised: Our Bayesian formulation allows uncertainty quantification and mutual information estimation which is well-suited for extensions to active causal discovery.

Original: Extensions of the proposed method to learn nonlinear, non-Gaussian latent SCMs from unknown interventions would also open doors to general algorithms that can learn causal representations.
Revised: Extensions of the proposed method to learn nonlinear, non-Gaussian latent SCMs from unknown interventions would also open doors to general algorithms that can learn causal representations.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: The details regarding synthetic data generation of the DAGs, causal variables, and the high dimensional data is mentioned in section 5.
Revised: The details regarding synthetic data generation of the DAGs, causal variables, and the high dimensional data is mentioned in section 5.

Original: The code for data generation, the models used, as well as for all the experiments is available at anonymous.4open.science/r/anon-biols-86E7.
Revised: The code for data generation, the models used, as well as for all the experiments is available at anonymous.4open.science/r/anon-biols-86E7.

Original: The average runtime of all experiments is documented in table 1.
Revised: The average runtime of all experiments is documented in table 3.

Original: Appendix A.7 further evaluates experiments along additional metrics.
Revised: Appendix A.8 further evaluates experiments along additional metrics.

Original: A A PPENDIX A. D ERIVATION OF THE ELBO 
Revised: A A PPENDIX A. D ERIVATION OF THE ELBO From

Original: From equation  we want to minimize the KL divergence between the true and approximate posteriors: 
Revised: equation 7, we want to minimize the KL divergence between the true and approximate posteriors: [Equation] factorization

Original: [Equation]
Revised: [Equation]

Original: Since the log evidence log p ( D ) is a constant
Revised: Since the log evidence log p ( D ) is a constant

Original: [Equation]
Revised: [Equation]

Original: minimizing the KL divergence corresponds to maximizing the following ELBO:
Revised: minimizing the KL divergence corresponds to maximizing the following ELBO:

Original: [Equation]
Revised: [Equation]

Original: log q ϕ
Revised: For all our experiments, we use the AdaBelief (Zhuang et al., 2020) optimizer with ϵ = 10 −and a learning rate of 0 . 0008 . τ is set to 0.2. Our experiments are fairly robust with respect to hyperparameters and we did not perform hyperparameter tuning for any of our experiments.

Original:  ( L, Σ) p ( L ) p (Σ) (cid:35) A. I MPLEMENTATION DETAILS A. T RAINING CURVES
Revised: Table 1 and 2 summarizes the network details for MLP ϕ ( T ) and the decoder p ( X | Z ) .

Original: [Figure]
Revised: [Table]

Revised: Linear

Original: A. L INEAR PROJECTION EXPERIMENTS
Revised: GeLU Linear GeLU Linear d

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: Details forfigure 11, d = 10 nodes: The dataset consists of 500 observational points andinterventional points.
Revised: Figures 8, 9, and 10 summarize the training curves for d = 6 , 20 , 50 nodes on ER-DAGs of degree 1, 2, and 4. Each plot contains 3 lines that shows training with observational, single-interventional, and multi-interventional data. [Figure]

Original: To sample the 20000 interventional points, we randomly choose 200 intervention sets, and for each intervention set we sample 100 data points. The model was trained forepochs to reach convergence.
Revised: projected causal variables for random ER-1 DAGs with d = 6 , 20 , 50 nodes. The model was trained for 5000 iterations over 3500 data samples out of which 500 were observational points for the single and multi node intervention runs.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [-SHD 0 . AUROC 9991 
 MCC( z , z ) MSE(-] L , L ) d
Revised: [-SHD 0 . AUROC 9991 
 MCC( z , z ) MSE(-]A. L INEAR PROJECTION EXPERIMENTS

Original: [Figure] d [Figure] [Figure]
Revised: Details for figure 11, d = 10 nodes: The dataset consists of 500 observational points andinterventional points.

Original: R - [Figure] d [Figure]
Revised: To sample the 20000 interventional points, we randomly choose 200 intervention sets, and for each intervention set we sample 100 data points.

Original: [-[Figure] E R
-]
Revised: [-[Figure] E R
-]The model was trained forepochs to reach convergence

Original: [Figure] Details for figure 12, d = 20 nodes: The dataset consists of 500 observational points andinterventional points.
Revised: Details for figure 12, d = 20 nodes: The dataset consists of 500 observational points andinterventional points.

Original: To sample the 20000 interventional points, we randomly choose 100 intervention sets, and for each intervention set we sample 200 data points.
Revised: To sample the 20000 interventional points, we randomly choose 100 intervention sets, and for each intervention set we sample 200 data points.

Original: The model was trained forepochs to reach convergence.
Revised: The model was trained forepochs to reach convergence.

Original: [Figure]

Original: A. N ONLINEAR PROJECTION EXPERIMENTS
Revised: A. N ONLINEAR PROJECTION EXPERIMENTS

Original: Figure 13 contains the results for the latent causal discovery problem with and without learning a permutation.
Revised: Figure 13 contains the results for the latent causal discovery problem with and without learning a permutation.

Original: Figure 14 shows the results for 500 observational samples and 10000 interventional samples with 100 intervention sets and 100 samples per set on d = 20 nodes and D = 100 dimensions. {+SHD 0 . AUROC 9991 
+}
Revised: Figure 14 shows the results for 500 observational samples and 10000 interventional samples with 100 intervention sets and 100 samples per set on d = 20 nodes and D = 100 dimensions. {+SHD 0 . AUROC 9991 
+}

Original: [Figure]
Revised: MCC( z , z ) MSE( L , L ) d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R123d = 1 0 , E R - 1 d = 1 0 , E R - 2 d = 1 0 , E R0 .0 .1 .d

Original: [Figure]
Revised: [Figure]

Original: SHD 0 .
Revised: 0 , E

Original:  AUROC 7891 

Revised: [Figure] [Figure] SHD 0 . AUROC 7891

Original: MCC( z , z )  {+
+}
Revised: MCC( z , z )  {+
+}

Original: MSE( L , L ) d = 6 , E R - 1 d = 6 , E R - 2 d = 6 , E R11d = 6 , E R - 1 d = 6 , E R - 2 d = 6 , E R0 .0 .0 .d
Revised: MSE( L , L ) d = 2 0 , E R - 1 d = 2 0 , E R - 2 d = 2 0 , E R51 0d = 2 0 , E R - 1 d = 2 0 , E R - 2 d = 2 0 , E R -

Original: = 6 , E R - 1 d = 6 , E R - 2 d = 6 , E R - 4 45678d = 6 , E R - 1 d = 6 , E R - 2 d = 6 , E R0 .1 .1 .2 .
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: 0 .0 .0 .0 .d

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: d = 2 0 , E R - 1 d = 2 0 , E R - 2 d [-= 2 0 , E R2571 
-]
Revised: R [Figure] d [-= 2 0 , E R2571 
-][Figure]

Original: SHD d = 2 0 , E R - 1 d = 2 0 , E R - 2 d = 2 0 , E R - 4 0 .5 0 .
Revised: [Figure] R [Figure] 68d

Original: AUROC d = 2 0 , E R - 1 d[-= 2 0 , E R - 2 d = 2 0 , E R4681 
-]
Revised: [Figure] 2 0 , E R [Figure] 1 d[-= 2 0 , E R - 2 d = 2 0 , E R4681 
-]

Original: MCC( z , z ) d = 2 0 , E R - 1 d = 2 0 , E R - 2 d= 2 0 , E RMSE( L , L )
Revised: [Figure] 2 0 , E R [Figure] 2 d

Original: Ours Null graph
Revised: [Figure] [Figure]

Original: A. A BLATION STUDY : EFFECT OF PERFORMANCE WITH RESPECT TO NUMBER OF INTERVENTION TYPES
Revised: A. A BLATION STUDY : EFFECT OF PERFORMANCE WITH RESPECT TO NUMBER OF INTERVENTION TYPES

Original: Here, we study how the performance and recovery of the latent SCM is affected with respect to the number of intervention types in the dataset.
Revised: Here, we study how the performance and recovery of the latent SCM is affected with respect to the number of intervention types in the dataset.

Original: The number of intervention types refers to different combinations of nodes we perform interventions on.
Revised: The number of intervention types refers to different combinations of nodes we perform interventions on.

Original: For all experiments in this subsection, we use 100 interventional samples per type of intervention.
Revised: For all experiments in this subsection, we use 100 interventional samples per type of intervention.

Original: Figure 15 and 16 show results on vector data where the high-dimensional vector is a linear projection of the causal variables.
Revised: Figure 15 and 16 show results on vector data where the high-dimensional vector is a linear projection of the causal variables.

Original: Figure 17 andsummarize results on vector data where the high-dimensional vector is a nonlinear projection of the causal variables.
Revised: Figure 17 andsummarize results on vector data where the high-dimensional vector is a nonlinear projection of the causal variables.

Original: SHD 0 .0 .0 .1 .
Revised: A. A DDITIONAL RELATED WORK

Original: [-AUROC 891 
-]
Revised: [-AUROC 891 
-]Latent variable models with predefined structure : Examples include VAE (Kingma and Welling, 2013; Rezende et al., 2014) which has an independence assumption between latent variables. To overcome this, Sønderby et al. and Zhao et al.

Original: MCC( z , z ) MSE( L , L ) [Figure] [Figure]
Revised: (2017) define latent variables with a chain

Original: SHD 0 .
Revised: SHD 0 .

Original: AUROC 9 7 .9 8 .9 8 .9 9 .9 9 .1 0 0 .{+7891 
+}
Revised: AUROC {+7891 
+}

Original: MCC( z , z )0 .
Revised: MCC( z , z )

Original: [Figure] [Figure]
Revised: MSE( L , L ) d = 6 , E R - 1 d = 6 , E R - 2 d = 6 , E R11d = 6 , E R - 1 d = 6 , E R - 2 d = 6 , E R -

Original: [Figure]
Revised: [Figure]

Original: ( L , L )

Original: [Figure]
Revised: [Figure]

Original: [-SHD 0 . AUROC 788991 
 MCC( z , z ) 0 .0 .0 .0 .0 .1 .-]
Revised: [-SHD 0 . AUROC 788991 
 MCC( z , z ) 0 .0 .0 .0 .0 .1 .-]d =

Original: MSE( L , L )
Revised: [Figure] , E R

Original: [Figure]
Revised: [Figure]

Original: [-SHD 0 . AUROC 999991 
 MCC( z , z ) 0 .0 .0 .0 .0 . MSE( L , L )-]
Revised: [-SHD 0 . AUROC 999991 
 MCC( z , z ) 0 .0 .0 .0 .0 . MSE( L , L )-]d

Original: [Figure]
Revised: [Figure]

Revised: , E R

Original: [Figure]
Revised: [Figure]

Original: Projection Steps Avg. runtime (min)
Revised: R

Original: [Figure]
Revised: [Figure]

Original: Image data
Revised: [Figure] given a node ordering for nonlinear projection of causal variables for d = 20 nodes, D = 100 dimensions. structure in VAEs. Kingma et al.

Original: Nonlinear
Revised: (2016) uses inverse autoregressive flows to improve upon the diagonal covariance of latent variables in VAEs. Causal discovery : Annadani et al. (2021) casts the Bayesian structure learning problem as an autoregressive one by sequentially predicting edges, in hopes of capturing the potentially multi-modal posterior.

Original: Image data
Revised: Deleu et al.

Original: Nonlinear
Revised: (2022) uses Generative Flow Networks, or GFlowNets (Bengio et al., 2021), a new class of probabilistic methods that lies at the intersection of reinforcement learning and variational inference. The work uses the transitive closure property ensuring that the action space is constrained to actions that do not introduce cycles. Chickering (2002) proposes a greedy search algorithm, but does not scale to large number of nodes. Wang et al.

Original: Image data
Revised: leverages sum product networks to perform exact Bayesian structure learning. H¨agele et al. (2022) extends the framework of Lorch et al.

Original: Nonlinear
Revised: (2021) to perform Bayesian causal discovery in a setting where interventions are unknown. Xie et al. (2022) is in a setting where the edges exist not just between latent causal variables but with high-dimensional variables in the dataset as well. Other efforts include (Shimizu et al., 2011; Lopez-Paz and Oquab, 2016; Yu et al., 2019; Ghoshal and Honorio, 2018; Ng et al., 2020; Li et al., 2022).

Original: A. C OMPLETE EVALUATION
Revised: A. C OMPLETE EVALUATION

Original: For the experiments already presented in the main text, this section contains additional a more comprehensive evaluation on the following metrics:
Revised: For the experiments already presented in the main text, this section contains additional a more comprehensive evaluation on the following metrics:

Original: • SHD C :
Revised: • SHD C :

Original: The expected CPDAG SHD between the GT and predicted DAG’s skeletons. {+SHD 0 .0 .0 .1 . AUROC 891 
 MCC( z , z )+}
Revised: The expected CPDAG SHD between the GT and predicted DAG’s skeletons. {+SHD 0 .0 .0 .1 . AUROC 891 
 MCC( z , z )+}

Revised: MSE( L , L )

Original:  SHD : The expected SHD between the GT and predicted DAG.
Revised: [Figure] [Figure] 15 types 20 types SHD 0 .

Revised: AUROC 9 7 .9 8 .9 8 .9 9 .9 9 .1 0 0 .

Original: {+MCC( z , z ) 0 . 00 . 00 . 00 . 00 . 00 . 
+}
Revised: {+MCC( z , z ) 0 . 00 . 00 . 00 . 00 . 00 . 
+}

Revised: MSE( L , L )

Revised: [Figure]

Revised: • SHD : The expected SHD between the GT and predicted DAG.

Original: • MCC :
Revised: • MCC :

Original: The expected Mean Correlation Coefficient between the predicted and true causal variables.
Revised: The expected Mean Correlation Coefficient between the predicted and true causal variables.

Original: • AUROC between predicted and true graph structure.
Revised: • AUROC between predicted and true graph structure.

Original: • FPR : False positive rate • FN : False negative[Figure] • TP : True positive • AUPRC G : Area under precision-recall curves • Precision of the graph structure prediction • TN :
Revised: • FPR : False positive rate • FN : False negative • TP : True positive • AUPRC G : Area under precision-recall curves • Precision of the graph structure prediction • TN :

Original: True negatives • FP : False positives • TPR :
Revised: True negatives • FP : False positives • TPR :

Original: True positive rate • Recall of the graph structure prediction • AUPRC W : • F1 Score
Revised: True positive rate • Recall of the graph structure prediction • AUPRC W : • F1 Score

Original: = 2 ∗ Precision ∗ Recall/ ( Precision + Recall ) • L MSE : Mean squared error between predicted and true edge weights.
Revised: = 2 ∗ Precision ∗ Recall/ ( Precision + Recall ) • L MSE : Mean squared error between predicted and true edge weights.

Original: • X MSE :
Revised: • X MSE :

Original: Mean squared reconstruction error over the high dimensional (low-level) data • true obs KL term Z : The KL divergence between the predicted and GT observational joint distributions.
Revised: Mean squared reconstruction error over the high dimensional (low-level) data • true obs KL term Z : The KL divergence between the predicted and GT observational joint distributions.

Original: Figure 19 shows the complete evaluation on 18 different metrics for d = 5 nodes.
Revised: Figure 19 shows the complete evaluation on 18 different metrics for d = 5 nodes.

Original: The dataset consists of 500 observational points and 2000 interventional points.
Revised: The dataset consists of 500 observational points and 2000 interventional points.

Original: To sample the 2000 interventional points, we randomly choose 20 intervention sets, and for each intervention set we sample 100 data points with random intervention values. {+SHD 0 . AUROC 788991 
+}
Revised: To sample the 2000 interventional points, we randomly choose 20 intervention sets, and for each intervention set we sample 100 data points with random intervention values. {+SHD 0 . AUROC 788991 
+}

Original: A. A DDITIONAL RELATED WORK
Revised: MCC( z , z ) 0 .0 .0 .0 .0 .1 . MSE( L , L )

Original: Causal discovery and structure learning : Annadani et al.{+[Figure] [Figure] AUROC 999991 
 MCC( z , z ) 0 .0 .0 .0 .0 .+}
Revised: {+[Figure] [Figure] AUROC 999991 
 MCC( z , z ) 0 .0 .0 .0 .0 .+}

Original: (2021) casts the Bayesian structure learning problem as an autoregressive one by sequentially predicting edges, in hopes of capturing the potentially multi-modal posterior.
Revised: MSE( L , L ) [Figure] A. D ISCUSSION ON IDENTIFIABILITY Identifiability is an important topic of discussion when making conclusions about the recovering the structure of a causal model.

Original: Deleu et al.
Revised: However in our work, we approximate a full posterior distribution over the latent SCMs, instead of returning just a single graph. In this setting, questions of identifiability become less critical, as we can assign probabilities for many possible candidate graphs (and parameters) to express our level of confidence that a particular SCM yields the correct causal conclusions. This is a softer guarantee than what identifiability would provide.

Original: (2022) uses Generative Flow Networks, or GFlowNets (Bengio et al., 2021), a new class of probabilistic methods that lies at the intersection of reinforcement learning and variational inference.
Revised: Nevertheless, we refer the reader to recent works in causal representation learning (Brehmer et al., 2022; Ahuja et al., 2022a;c), that have given some identifiability guarantees in conditions similar to ours.

Original: The work uses the transitive closure property ensuring that the action space is constrained to actions that do not introduce cycles. Chickering (2002) proposes a greedy search algorithm, but does not scale to large number of nodes.
Revised: Particularly, we rely on the identifiability results presented in Brehmer et al. But identifiability does not imply learnability . Thus, in this work, we are concerned with the problem of how one can devise a principled practical algorithm to learn a distribution over latent SCMs.

Original: Wang et al.
Revised: [Figure]

Original: leverages sum product networks to perform exact Bayesian structure learning.
Revised: Nodes ( d ) Dataset size Experiment Projection

Original: H¨agele et al.
Revised: Steps

Original: (2022) extends the framework of Lorch et al.
Revised: Avg. runtime (min) Vector data Linear (fixed ordering)

Original: (2021) to perform Bayesian causal discovery in a setting where interventions are unknown.
Revised: Vector data Linear (fixed ordering) Vector data Linear (learned ordering)

Original: Xie et al.
Revised: Vector data Linear (fixed ordering) Vector data Linear (learned ordering)

Original: (2022) is in a setting where the edges exist not just between latent causal variables but with high-dimensional variables in the dataset as well.
Revised: Vector data Linear (learned ordering) Vector data Nonlinear

Original: Other efforts include (Shimizu et al., 2011; Lopez-Paz & Oquab, 2016; Yu et al., 2019; Ghoshal &
Revised: Vector data

Original: Honorio, 2018; Ng et al., 2020; Li et al., 2022).
Revised: Nonlinear Vector data Nonlinear Image data

Original: Latent variable models with predefined structure : Examples include VAE (Kingma & Welling, 2013; Rezende et al., 2014) which has an independence assumption between latent variables.
Revised: Nonlinear [Figure] Image data Nonlinear

Original: To overcome this, Sønderby et al.{+Image data Nonlinear Image data Nonlinear    
+}
Revised: {+Image data Nonlinear Image data Nonlinear    
+}

Original: and Zhao et al.{+SHD 0 . 00 . AUROC 
+}
Revised: {+SHD 0 . 00 . AUROC 
+}

Original: (2017) define latent variables with a chain structure in VAEs. Kingma et al.
Revised: MCC( z , z ) 0 .0 .1 .1 . MSE( L , L ) d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R112d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 4 0 . 00 .

Original: (2016) uses inverse autoregressive flows to improve upon the diagonal covariance of latent variables in VAEs.
Revised: 0d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E R - 42468d = 5 , E R - 1 d = 5 , E R - 2 d = 5 , E ROurs GraphVAE VAE