Paper: Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand (ICLR-2024)

Reviewer comment (Experiments):
While using a causal graph to model the relationships between statistical variables, deriving identifiability conditions is crucial. However, when extending this to high-dimensional data, the theory is elegant, but the practical applications are quite tricky. For instance, as seen in the experiments within the paper, they are entirely simulated cases. I find this to be not very practical.

Author response:
We thank the reviewer for their comment. We humbly point the reviewer to section 5.2 of our paper where we shared our
algorithm performance of
an experiment on the real-world COVID
X-RAY dataset (https://www.kaggle.com/datasets/andyczhao/covidx-cxr2/data). However, if
the reviewer could kindly suggest any challenging applications or datasets suitable for causal inference,
we would be very happy to evaluate our algorithm on those.

For the reviewer's convenience, here we share our real-world experiment in detail.
The COVID X-RAY contains 30k X-ray images and COVID and pneumonia labels.
With this dataset, we associated the front door causal graph:
$$\mathrm{Covid Symptoms} \rightarrow \mathrm{Xray Images} \rightarrow \mathrm{Pneumonia Detection};
\mathrm{Covid Symptoms} \leftarrow U \rightarrow \mathrm{Pneumonia Detection}$$
where $U$ is an unobserved variable (patient/hospital location).
Our aim was to generate high-quality samples from $P(\mathrm{Xray images}| do(\mathrm{Covid Symptoms}))$ and estimate
$P(\mathrm{Pneumonia Detection}|do(\mathrm{Covid Symptoms}))$, i.e., how likely a patient will be diagnosed with
Pneumonia if we intervene on the COVID-19 symptoms, assessing the causal effect.

For the reviewer's convenience, here we share part of the section 5.2 experiment results. Let N= Pneumonia detection and
C = Covid Symptoms.

### Result 1:

We evaluated the image quality of the diffusion model approximating $P(X|C)$. Below we provide the Frechet Inception
Distance (FID)
scores of generated image samples compared to the true image samples. Here, `Generated C=c`, means we sample from
the diffusion model conditioned on the value $C=c$. `Real (C=c)` refers to the corresponding image samples consistent
with $C=c$. Low values on the diagonal and high values on the off-diagonal imply we are sampling correctly from conditional
distributions.

Class-conditional FID scores for generated Covid XRAY images (lower is better).

| FID scores of <br/>Generated vs Real image | Real: C = 0 | Real: C = 1 |
|--------------------------------------------|-------------|-------------|
| Generated: C = 0                           | 15.77       | 61.29       |
| Generated C = 1                            | 101.76      | 23.34       |

### Result 2:

We evaluate the query of interest $P_C(N)$. Since we do not have access to the ground truth of an interventional
distribution
for real data, we consider i) our evaluated
$P_C(N)$ versus ii) an ablated version where we replace the diffusion sampling mechanism with $\hat P(X|C)$, where we
randomly select an X-ray image from the validation set. We also consider the query $P_C(N)$ if there were iii) no
latent confounders in the graph, in which case, the interventional query $P_C(N)$ is equal to $P(N|C)$.

| $P_{C}(N=1)$     | c=0   | c=1   |
|------------------|-------|-------|
| i) Diffusion     | 0.622 | 0.834 |
| ii) No Diffusion | 0.623 | 0.860 |
| iii) No Latent   | 0.406 | 0.951 |

Reviewer comment (Experiments):
If the authors could provide some simulation data in the experimental section, it would better support the conclusions in the paper. This is because, when relying solely on image data, evaluation metrics can sometimes be less accurate.

Author response:
We agree with the reviewer that it will be useful to include experiments with simulated data where variables are low
dimensional discrete.
Then we would be able to estimate $P_{X}(Y)$ and compare with the ground truth and evaluate them with different measures
such as
total variation distance (TVD) or KL divergence. Since our main target was to sample from the high-dimensional
interventional
distribution, we only considered image data during submission. Based on the reviewer's suggestion, we will add these new
experiments with discrete data in the camera-ready version of our paper. If the reviewer has
any specific suggestions about simulated datasets, we would be happy to include them.

Reviewer comment (Experiments):
Could the authors provide some failure cases, where the situations that would occur if the assumptions in the paper are not met?

Author response:
Here we provide some failure cases that may occur if our assumptions are not satisfied:
1. If the causal graph is incorrect and does not reflect the data-generating process properly, then we
might not get correct samples from the target interventional distribution, although please note that this is a common
assumption in
causality literature.
2. If the conditional generative models are not correctly trained then the generated samples will not represent the
target interventional
distributions. Thus, we assume model convergence similar to many papers that apply deep-learning approaches.

### DAG vs ADMG

For the reviewer's convenience, we provide a short description of ADMG here. In causal inference literature, ADMG refers
to acyclic directed mixed graphs. It is equivalent to a directed acyclic graph (DAG) where some variables
are unobserved (latent confounders cause two observed variables). For example, for our COVID experiment, $C \leftarrow
Xray \rightarrow N; C\leftarrow Patient Location \rightarrow N$
is a DAG. Now, suppose we can not observe patient location in the data, then we have an ADMG which we represent as
$C \leftarrow Xray \rightarrow N; C\leftrightarrow N$.

Reviewer comment (Novelty):
While estimating high-dimensional, arbitrary conditional distributions to arbitrary precision is difficult, doing so in a more general sense is not impossible, and there exist different methods (Bayes' rule, explicit modelling, approximations). The introduction describes all Pearl-related causal work precisely but cites almost nothing from the generative (probabilistic modelling) side apart from a few deep models. There is relevant work on probabilistic circuits by Darwiche, Poon, Domingos, Peharz, Vergari, Van den Broeck, etc., some with causality results. Examples include NeurIPS 2021 "Interventional Sum-Product Networks" and AISTATS 2023 "Compositional Probabilistic and Causal Inference using Tractable Circuit Models". Authors are encouraged to review this literature extensively.

Author response:
We agree with the reviewer that causal reasoning with sum-product networks is an important research direction that can
deal with the tractability issues associated with Bayesian networks. However, there is one major difference between the
goal of these works and the goal of our paper. We are interested in enabling high-dimensional causal inference,
including graphs that contain multiple image nodes, using finite data. Thus, our objective is to develop a method that
has low **sample-complexity**. On the other hand, probabilistic circuits literature is interested in reducing **computational
complexity** of the (causal) inference problem. This important difference separates us from them, in the sense that it is crucial for us to show that we can sample from any interventional distribution using only feedforward models. The results in the said literature do not subsume our results. For example, in the paper "Interventional Sum-Product Networks: Causal Inference with Tractable Probabilistic Models", the authors say:

"The causal effect $p(y|do(x))$ in the napkin graph: $$W \rightarrow Z \rightarrow X \rightarrow Y; X \leftrightarrow W
\leftrightarrow Y $$ is identified as $$p(y| do(x)) = \frac{\sum_{w} p(y,x|w,z) p(w)}{\sum_{w} p(x|w,z) p(w)}$$ using
the do-calculus where each of the r.h.s. components can be modeled by (i)SPN respectively."

Thus, the proposal is to model each probabilistic term in the expression with (i)SPNs. But this does not give us a way
how to connect those consistently to samples from the overall distribution. Note that, if sampling each of the variables
is not done in a consistent way, there might be cyclic dependency during the sampling process (as we showed above). The
necessity of consistent sampling was not clear until our work. We show that the ID algorithm can be taken as a guide,
but it has to be modified otherwise there will be cyclic dependencies during the sampling process.

Although the above work mentions that they can quantify $p(y|do(x))$ as proven in Proposition 1 of their paper, they do
not suggest how they can do that with only observational data which is known as identifiability in causality literature.
Proposition 1 and all other proofs require access to both observational **and interventional** data. On the other hand, our
the algorithm only requires observational data for training the conditional model, thus offering causal identifiability capability.
Our result does not contradict the probabilistic circuits literature, but if anything, enables it by showing that we now
have an algorithm that only needs feed-forward models. Again, this result was not known before, although it seems to
have been vaguely claimed in the above paper.

We will also add a discussion about the 2nd paper mentioned by the reviewer: "Compositional Probabilistic and Causal Inference using Tractable Circuit Models". This paper proposes an interesting method that can estimate causal effects efficiently by utilizing tractable probabilistic modeling. However, it is unclear if this method can utilize generative models such as diffusion models (similar to ours) to sample from high-dimensional interventional distributions (such as images).

We will add these discussions in the main paper as well as cite all the papers in the probabilistic circuits community on
causal inference. Again, we greatly appreciate the reviewer for pointing out this line of work to us. This allowed us to
more clearly express the contribution of our work in regard to the related works.

Reviewer comment (Writing):
\( X \) is not defined in Lemma 3.3.

Author response:
Definitions: We redefined $X$ and $P_{X}(Y)$ in Lemma 3.3 and $\pi$ in Algorithm 1. We would like to kindly point out
  that our algorithm is for acyclic directed mixed graph (ADMG), we used the term DAG to refer to the set of trained
  neural networks which have a different structure, edges, and even nodes compared to the original causal graph. To avoid
  confusion, we replaced the "DAG"
  term with a "network" of trained models.

Reviewer comment (Writing):
\( P_x(y) \) is not defined when first introduced in Lemma 3.3.

Author response:
Definitions: We redefined $X$ and $P_{X}(Y)$ in Lemma 3.3 and $\pi$ in Algorithm 1. We would like to kindly point out
  that our algorithm is for acyclic directed mixed graph (ADMG), we used the term DAG to refer to the set of trained
  neural networks which have a different structure, edges, and even nodes compared to the original causal graph. To avoid
  confusion, we replaced the "DAG"
  term with a "network" of trained models.

Reviewer comment (Presentation):
Figure presentation should be improved with respect to: (i) proportions — Figures 2, 3, 4, 5 have small font sizes; (ii) descriptions — missing legends and inadequate captions that don’t communicate key takeaways; and (iii) consistency — for example, Figure 3 uses an uncompiled \sum.

Author response:
Figures: We i) improved the objects in Figures 2,3,4 and 5 and mentioned the significance of those, and ii) increased the
  readability of some texts. We iii) added more descriptive captions to assist the readers. We iv) fixed the uncompiled
  equations in the figures.

Reviewer comment (Writing):
DAGs are never defined. The paper begins with ADMGs and then replaces them with DAGs without notice. This further gives a false sense of generality of the results, since ADMGs generalize DAGs.

Author response:
Definitions: We redefined $X$ and $P_{X}(Y)$ in Lemma 3.3 and $\pi$ in Algorithm 1. We would like to kindly point out
  that our algorithm is for acyclic directed mixed graph (ADMG), we used the term DAG to refer to the set of trained
  neural networks which have a different structure, edges, and even nodes compared to the original causal graph. To avoid
  confusion, we replaced the "DAG"
  term with a "network" of trained models.

Reviewer comment (Writing):
\(\pi\) in Algorithm 1 is not defined.

Author response:
Definitions: We redefined $X$ and $P_{X}(Y)$ in Lemma 3.3 and $\pi$ in Algorithm 1. We would like to kindly point out
  that our algorithm is for acyclic directed mixed graph (ADMG), we used the term DAG to refer to the set of trained
  neural networks which have a different structure, edges, and even nodes compared to the original causal graph. To avoid
  confusion, we replaced the "DAG"
  term with a "network" of trained models.

Reviewer comment (Reproducibility):
Key parts of the algorithm, such as ConstructDAG or the modification in Step 7, are not discussed in the main paper.

Author response:
Discussion: We added a new section (Appendix D) in our updated submission, where we described the ConstructDAG and the
  step 7 of our algorithm in more detail.

Reviewer comment (Theory):
Theorem 4.1 is neither covered nor proven in the main paper. The style appears to rely on "exploiting" the appendix for extra space, which conflicts with ICLR guidelines. While the reviewer checked the appendix, they note that this should not be required, as it undermines the 9-page main paper’s meaning.

Author response:
We would like to humbly disagree with the reviewer's concern about us exploiting the appendix. The reviewer is right
that the proofs are provided in Appendix B and C. However, to our knowledge, it is quite common in the ML community to
postpone theorem proofs to the appendix. We also believe that the statement of Theorem 4.1 is self-explanatory as it
only states soundness and completeness, again a well-understood notion in the community. As evidence that this practice
does not violate ICLR guidelines, we refer to two ICLR-2023 accepted papers (top 5\%:
openreview.net/pdf?id=7YfHla7IxBJ; openreview.net/pdf?id=l6CpxixmUg) where the authors not only put all their proofs in
the appendix but also explained their theorems very little, if any. Therefore, we would kindly request the reviewer to
reconsider his claim that we have "exploited" the appendix. We believe this unfortunate choice of words was unjustified.
To consider the reviewer's feedback for improving our paper, we will add a proof sketch and elaborate on what soundness
and completeness means in the main paper for Theorem 4.1.

Reviewer comment (Writing):
Another proofreading pass, possibly by a non-author, is needed to fix writing mistakes not caught by common software. Examples include "international" instead of "interventional" in Related Work, and missing punctuation such as a missing period between two sentences at the start of Section 4.

Author response:
Another pass: We made another pass through the paper and fixed the remaining minor issues mentioned by the reviewer. 
  Since ICLR policy allows to update the submitted paper with minor changes during rebuttal, we fixed all the writing
  issues mentioned by the reviewer and will upload the new version as soon as possible. We will be very happy if the
  reviewer does not finalize his decision about our paper based on these.

Reviewer comment (Novelty):
The idea of sampling according to the topological order is not new. Most of the causal effects estimators (explicitly or implicitly) utilise this technique to draw samples from the interventional distribution using conditional distributions. For example, [1][2][3]. Could the author please highlight the technical novelties of the proposed method?

Author response:
We thank the reviewer for their feedback and questions and appreciate them acknowledging the utility of our results in
different domains and our practical relevance.
Below we address all of the reviewer's concerns.

## Sampling according to the topological order is not sufficient:

> The idea of sampling according to the topological order is not new. Most of the causal effects estimators (explicitly
> or implicitly)
> Utilize this technique to draw samples from the interventional distribution using conditional distributions.

The reviewer raised an interesting concern that the idea of sampling according to the topological order is not new and
we agree with them. However, this approach is **only feasible when there exists no unobserved confounders in the causal
graph**.
For graphs with confounders, sampling according to topological order has failure cases.
Consider this causal graph: 
$$X \rightarrow W_1 \rightarrow W_2 \rightarrow Y; X \leftrightarrow W_2; W_1 \leftrightarrow
Y.$$
Here, we can not sample variables one-by-one following the order {$X,W_1,W_2,Y$} since {$X,W_2$} and {$W_1,
Y$} are confounded and we have to sample them together, i.e., sample {$X, W_2$} or sample {$W_1, Y$}. But which one
to sample first? If we consider sampling {$X, W_2$} first, we need $W_1$ as input to the model to generate $W_2$ since
$W_1$ is $W_2$'s parent. If we consider sampling {$W_1, Y$} first, we need both $X_1$ and $W_2$ as input to the model
for the same reason. This creates a cycle in the sampling network.

Our algorithm takes several steps to avoid this issue for any causal graph and identifiable interventional query. For
example, we train models for {$X, W_2$} considering all possible values of $W_1$ as input. Similarly, we train models for the 2nd c-component considering
all possible values of its input variables. Finally, we connect the trained models from two c-components to build a
single sampling network.
In this sampling network, we do not have to sample any variables together with another one, and we can follow the topological order to
sample all variables.
Therefore, we would state that sampling according to the topological ordering is one single part of our whole solution.
There are more subtleties such as:
*  Building a sampling network: we carefully follow the steps of the ID algorithm to recursively divide the problem into
c-components, train corresponding models, and build a consistent sampling network with the trained models.
*  New interventional data generation: We generate new interventional datasets by training conditional models on
observational data and utilize them in later steps.
* Maintaining sampling consistency: We avoid the cyclic dependency issue mentioned above by strategically updating the
causal graph and the intervention set.

We describe all these steps in more detail in the next paragraph.

Reviewer comment (Experiments):
The experiments lack comparisons with baseline methods.

Author response:
## New experiment with Baseline comparison

> Is it possible to compare with some baselines?

According to the reviewer's suggestion, we compared our performance with two baselines and now provide the results below:
We updated our submission and provided the details in Appendix E.1.5.

For the reviewer's convenience, we provide a short description of the ground truth first.

### Ground truth:

The causal graph:
$$W_1 \rightarrow W_2 \rightarrow X \rightarrow Y;
X \leftrightarrow W_1 \leftrightarrow Y $$
Dataset:
$W_1, X $ and $Y$ are digit images and $W_2$ are discrete properties (thickness and color) of $W_1$.
Each digit image can take a value from [0,9] and a color from [Red, Green, Blue, Yellow, Magenta, Cyan].

### Baselines:

We compared our algorithm with two baselines:
Baseline 1: A classifier-free diffusion model that samples from the conditional distribution: $Y\sim P(Y|X)$.
Baseline 2: The NCM algorithm (Xia et al 2021) that samples from the interventional distribution $P(Y|do(X))$.

### Algorithm execution:

To easily convey the distinction between different methods, we chose two images i) of digit 3 and ii) of digit 5, both
colored red
and use them as the intervention value for $X$. Then we used these images to sample from the corresponding distributions $P(Y|X)$
and $P(Y|do(X))$ using different methods. Although the implementation of these methods is different,
we considered the performance of each method after running them for 300 epochs. The NCM algorithm took around 50 hours
to complete while our algorithm took approximately 16 hours to complete.

### Results:

We would kindly request the reviewer to look at Figure 8 in Appendix E.1.5 for the results. For the reviewer's
convenience,
we describe the results in detail here as well.

**Image quality**: First, we discuss the image quality of the generated samples from each approach.
The conditional model and our algorithm both generate high-quality images of digit 3 and digit 5 with a specific color from
the six possible colors. While the colors in generated images from the NCM algorithm can not remain at any specific
value and get blended (such as green+yellow etc).

Below we provide the Frechet Inception Distance (FID) of each method (the lower the better).

|           | Conditional | NCM    | Our algorithm |
|-----------|-------------|--------|---------------|
| FID Score | 67.012      | 71.646 | 43.507        |

We observe that our algorithm has the lowest FID score, i.e., our algorithm generates the most high-quality images from
interventional distribution.

**Correctness of the sampling distribution**
We use a classifier to detect the color frequency from the generated images of the conditional model and our algorithm.
We observe the following probabilities.

| Predicted  color probabilities       | Red    | Green  | Blue   | Yellow | Magenta | Cyan   |
|--------------------------------------|--------|--------|--------|--------|---------|--------|
| Conditional model: P(Y &#124; X=red) | 0.1889 | 0.4448 | 0.1612 | 0.1021 | 0.0232  | 0.0798 |
| Ours: P(Y &#124; do(X=red))          | 0.1278 | 0.2288 | 0.2097 | 0.1445 | 0.1177  | 0.1715 |

In the generated samples from the conditional model,
the color of digit image $X$ and digit image $Y$ are correlated due to confounding through backdoor paths.
For example, for a digit image $X$ with color as red, $Y$
takes a value from [Red, Green, Blue] with high probability. Thus, the conditional model does not have the ability to
generate interventional samples.
On the other hand, our algorithm generates samples from the interventional distribution $P(Y|do(X))$ and the generated
samples chooses different colors:[Red, Green, Blue, Yellow, Magenta, Cyan] with almost the same probability, which is
the expected result given the ground-truth SCM.

Therefore, we show our performance with a close baseline NCM and illustrate our capability of sampling from
high-dimensional
interventional distribution.

[Xia et al 2021.]  Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection:
Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems, 34:10823–10836,

We hope to have addressed all of the reviewer's concerns. We are looking forward to having more discussions with the
reviewer. If the reviewer has further experiment suggestions and baselines we would be happy to add them to
camera-ready.

Reviewer comment (Experiments):
Is it possible to compare with some baselines?

Author response:
## New experiment with Baseline comparison

> Is it possible to compare with some baselines?

According to the reviewer's suggestion, we compared our performance with two baselines and now provide the results below:
We updated our submission and provided the details in Appendix E.1.5.

For the reviewer's convenience, we provide a short description of the ground truth first.

### Ground truth:

The causal graph:
$$W_1 \rightarrow W_2 \rightarrow X \rightarrow Y;
X \leftrightarrow W_1 \leftrightarrow Y $$
Dataset:
$W_1, X $ and $Y$ are digit images and $W_2$ are discrete properties (thickness and color) of $W_1$.
Each digit image can take a value from [0,9] and a color from [Red, Green, Blue, Yellow, Magenta, Cyan].

### Baselines:

We compared our algorithm with two baselines:
Baseline 1: A classifier-free diffusion model that samples from the conditional distribution: $Y\sim P(Y|X)$.
Baseline 2: The NCM algorithm (Xia et al 2021) that samples from the interventional distribution $P(Y|do(X))$.

### Algorithm execution:

To easily convey the distinction between different methods, we chose two images i) of digit 3 and ii) of digit 5, both
colored red
and use them as the intervention value for $X$. Then we used these images to sample from the corresponding distributions $P(Y|X)$
and $P(Y|do(X))$ using different methods. Although the implementation of these methods is different,
we considered the performance of each method after running them for 300 epochs. The NCM algorithm took around 50 hours
to complete while our algorithm took approximately 16 hours to complete.

### Results:

We would kindly request the reviewer to look at Figure 8 in Appendix E.1.5 for the results. For the reviewer's
convenience,
we describe the results in detail here as well.

**Image quality**: First, we discuss the image quality of the generated samples from each approach.
The conditional model and our algorithm both generate high-quality images of digit 3 and digit 5 with a specific color from
the six possible colors. While the colors in generated images from the NCM algorithm can not remain at any specific
value and get blended (such as green+yellow etc).

Below we provide the Frechet Inception Distance (FID) of each method (the lower the better).

|           | Conditional | NCM    | Our algorithm |
|-----------|-------------|--------|---------------|
| FID Score | 67.012      | 71.646 | 43.507        |

We observe that our algorithm has the lowest FID score, i.e., our algorithm generates the most high-quality images from
interventional distribution.

**Correctness of the sampling distribution**
We use a classifier to detect the color frequency from the generated images of the conditional model and our algorithm.
We observe the following probabilities.

| Predicted  color probabilities       | Red    | Green  | Blue   | Yellow | Magenta | Cyan   |
|--------------------------------------|--------|--------|--------|--------|---------|--------|
| Conditional model: P(Y &#124; X=red) | 0.1889 | 0.4448 | 0.1612 | 0.1021 | 0.0232  | 0.0798 |
| Ours: P(Y &#124; do(X=red))          | 0.1278 | 0.2288 | 0.2097 | 0.1445 | 0.1177  | 0.1715 |

In the generated samples from the conditional model,
the color of digit image $X$ and digit image $Y$ are correlated due to confounding through backdoor paths.
For example, for a digit image $X$ with color as red, $Y$
takes a value from [Red, Green, Blue] with high probability. Thus, the conditional model does not have the ability to
generate interventional samples.
On the other hand, our algorithm generates samples from the interventional distribution $P(Y|do(X))$ and the generated
samples chooses different colors:[Red, Green, Blue, Yellow, Magenta, Cyan] with almost the same probability, which is
the expected result given the ground-truth SCM.

Therefore, we show our performance with a close baseline NCM and illustrate our capability of sampling from
high-dimensional
interventional distribution.

[Xia et al 2021.]  Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection:
Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems, 34:10823–10836,

We hope to have addressed all of the reviewer's concerns. We are looking forward to having more discussions with the
reviewer. If the reviewer has further experiment suggestions and baselines we would be happy to add them to
camera-ready.

Reviewer comment (Novelty):
What are technical novelties of the proposed method?

Author response:
## Technical novelties of our paper

> What are the technical novelties of the proposed method?

We appreciate the reviewer for allowing us to highlight the strengths of our paper. Our main contribution is to show, by
construction, that one can sample from any identifiable interventional distribution only using observational data and
feed-forward, i.e., conditional generative models. Note that this was not known before our paper. Our construction uses
the ID algorithm to propose a sampling algorithm so that the new algorithm can deal with high-dimensional variables.
Here we describe the major technical differences of our algorithm:

1. **Building the sampling network**: At step 4 of our algorithm we split the variable set into multiple c-components and obtain corresponding models for each c-component recursively. We merge these trained models to build
   a single sampling network. While traversing the recursive steps of our algorithm, we stay in the DAG realm, i.e., no
   cycles in the sampling network, train the required models, and merge them. After finishing all the recursive steps, we sample using the final merged sampling network, which we show to be acyclic. This makes sure that variable values are consistent in each sample and respect the interventional distribution.

2. **New interventional data generation**: Step 7 of our algorithm requires us to generate a new interventional
   dataset from specific models that are trained on observational data. This newly generated interventional dataset is
   passed as the dataset argument for the next recursive step of the algorithm. This new dataset will be utilized to
   train conditional models corresponding to the deeper level of the recursion.

3. **New arguments to maintain sampling consistency**:
   Suppose we have two c-components $S_1$ and $S_2$ and $V$ is a set of all variables.
   While estimating $P_{V\setminus S_i}(S_i)$, the ID algorithm drops some of the intervened variables $V\setminus S_i$
   from the causal graph after they are applied to estimate the causal effect. Unlike the ID algorithm, we keep the
   intervened variables connected in
   the causal graph and utilize them as input conditions while training all conditional models at the deeper level of the
   recursion. This allows for sampling from multiple c-components in a consistent manner respecting the
   distribution. For example, models in both $S_1$ and $S_2$ will train considering each other's values such that during
   the sampling step, they can generate consistent samples.

We precisely described these contributions in Appendix D in the updated submission.

## Difference from the related work reviewer mentioned

[1], [2] and [3] propose interesting and novel approaches to solve the causal effect estimation
problem using variational autoencoders. However, their proposed solution and theoretical guarantees for causal
estimation are tailored for specific causal graphs containing treatment, effect, and covariates (or observed proxy
variables) where they apply the backdoor adjustment formula.

On the other hand, we consider any arbitrary causal graphs (such as the popular napkin graph in Figure 3: $W_1 \rightarrow W_2 \rightarrow X \rightarrow Y;
X \leftrightarrow W_1 \leftrightarrow Y $)
 possibly containing confounders that require a more complicated identifiability expression (such as fractions: $p(y| do(x)) = \frac{\sum_{w_1} p(y,x|w_1,w_2) p(w_1)}{\sum_{w_1} p(x|w_1,w_2) p(w_1)}$) compared to the
adjustment formula. Our proposed solution can sample from any identifiable query by training diffusion models regardless
of how complicated the identifiability expression is.

[1] Louizos, C., Shalit, U., Mooij, J. M., Sontag, D., Zemel, R., & Welling, M. (2017). Causal effect inference with
deep
latent-variable models. Advances in neural information processing systems, 30.\
[2] Zhang, W., Liu, L., & Li, J. (2021, May). Treatment effect estimation with disentangled latent factors. In
Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 10923-10930).\
[3] Vo, T. V., Bhattacharyya, A., Lee, Y., & Leong, T. Y. (2022). An Adaptive Kernel Approach to Federated Learning of
Heterogeneous Causal Effects. Advances in Neural Information Processing Systems, 35, 24459-24473.

Reviewer comment (Writing):
The paper's presentation could be a little clearer with more precise definitions of the notation (e.g. \(p(...\mid do(x=..))\) vs \(p_x(...)\)).

Author response:
We thank the reviewer for pointing it out. Both notations are meant to represent the same notion. We will make it more
clear in our background section.

Reviewer comment (Experiments):
Even though the paper cites [1] it simply states that they do not handle high-dimensional variables. It would be interesting to see a comparison of both methods to get a better understanding of the performance of ID-DAG.

Author response:
We appreciate the reviewer for suggesting NCM (Xia et al 2021) as a baseline.
According to the reviewer's suggestion, we compared our performance with NCM and another baseline and provided the
results
below:
We updated our submission and provided the details in Appendix E.1.5.

For the reviewer's convenience, we provide a short description of the ground truth first.

### Ground truth:

The causal graph:
$$W_1 \rightarrow W_2 \rightarrow X \rightarrow Y;
X \leftrightarrow W_1 \leftrightarrow Y $$
Dataset:
$W_1, X $ and $Y$ are digit images and $W_2$ are discrete properties (thickness and color) of $W_1$.
Each digit image can take a value from [0,9] and color from [Red, Green, Blue, Yellow, Magenta, Cyan].

### Baselines

We compared our algorithm with two baselines:
Baseline 1: A classifier-free diffusion model that samples from the conditional distribution: $Y\sim P(Y|X)$.
Baseline 2: The NCM algorithm (Xia et al 2021) that samples from the interventional distribution $P(Y|do(X))$.

### Algorithm execution

To easily understand the distinction among different methods, we chose two images i) of digit 3 and ii) of digit 5, both
colored red
as the intervention value for $X$. Then we used these images to sample from the corresponding distribution $P(Y|X)$
and $P(Y|do(X))$ distributions using different methods. Although the implementation of these methods is different,
we considered the performance of each method after running them for 300 epochs. The NCM algorithm took around 50 hours
to complete while our algorithm took approximately 16 hours to complete.

### Results

We would request the reviewer to look at Figure 8 in Appendix E.1.5 to get visual observation. For the reviewer's
convenience,
we describe the results in detail here as well.

**Image quality**: First, we discuss the image quality of the generated samples from each approach.
The conditional model and our algorithm both generate high-quality images of digit 3 and digit 5 with a specific color
from
the six possible colors. While the colors in generated images from the NCM algorithm can not remain at any specific
value and
get blended (such as green+yellow etc).

Below we provide the Frechet Inception Distance (FID) of each method (the lower the better).

|           | Conditional | NCM    | Our algorithm |
|-----------|-------------|--------|---------------|
| FID Score | 67.012      | 71.646 | 43.507        |

We observe that our algorithm has the lowest FID score, i.e., our algorithm generates the most high-quality images from
interventional distribution.

**Correctness of the sampling distribution**
We use a classifier to detect the color frequency from the generated images of the conditional model and our algorithm.
We observe the following probabilities.

| Predicted  color probabilities       | Red    | Green  | Blue   | Yellow | Magenta | Cyan   |
|--------------------------------------|--------|--------|--------|--------|---------|--------|
| Conditional model: P(Y &#124; X=red) | 0.1889 | 0.4448 | 0.1612 | 0.1021 | 0.0232  | 0.0798 |
| Ours: P(Y &#124; do(X=red))          | 0.1278 | 0.2288 | 0.2097 | 0.1445 | 0.1177  | 0.1715 |

In the generated samples from the conditional model,
the color of digit image $X$ and digit image $Y$ are correlated due to confounding through backdoor paths.
For example, for a digit image $X$ with color as red, $Y$
takes a value from [Red, Green, Blue] with high probability. Thus, the conditional model does not have the ability to
generate interventional samples.
On the other hand, our algorithm generates samples from the interventional distribution $P(Y|do(X))$ and the generated
samples chooses different colors:[Red, Green, Blue, Yellow, Magenta, Cyan] with almost the same probability.

Therefore, we show our performance with a close baseline NCM and illustrate our capability of sampling from
high-dimensional
interventional distribution.

[Xia et al 2021.]  Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection:
Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems, 34:10823–10836,

Reviewer comment (Experiments):
The paper mentions that the groundtruth interventional distribution in the case of MNIST is not accessible even though the data is synthetically generated.

Author response:
This ground truth image dataset $D$ we generated corresponds to the observational distribution $P(W_1, X, Y)$.
Thus, for specific image $X$ we know how image $Y\sim P(Y|X)$ will look like since we
can check that in the observational dataset $D$. However, the ground truth dataset $D$ does not let us know,
how image $Y\sim P(Y|do(X))$ would look like for a specific image $X$ since those are interventional samples.
This is what we meant when we said that we do not have access to the ground truth interventional image data.
However, we still evaluate our interventional samples. Please check below.

Since we generated the images from some discrete properties such as digit$\sim[0,9]$, thickness$\sim[0,1]$, and color$\sim[0,5]$ (
using MorphMNIST, etc), we have access to these discrete values. Therefore, we can use the original ID algorithm
to estimate true $P(Y.color|do(X.color))$.

After generating interventional image samples $Y\sim P(Y|do(X))$ with our models, we use a classifier to obtain the color of
$X$ and the generated $Y$, i.e, $X.color$ and $Y.color$. Thus, we can estimate $P'(Y.color|do(X.color))$ from our
generated interventional samples and compare against the true $P(Y.color|do(X.color))$. We showed this comparison in Table 1 in our paper.

Reviewer comment (Experiments):
It is recommended to look at an evaluation with MorphoMNIST [2] (and cite the package).

Author response:
We thank the reviewer for suggesting the MorphoMNIST [2] package
and would like to happily share that we are already using this package from
https://github.com/dccastro/Morpho-MNIST/ for data generation of our MNIST image experiment.
In the napkin graph, three out of four variables are images: $W_1, X$, and $Y$. To generate these images we picked a random digit from the original MNIST dataset and thickened or thinned it using the MorphoMNIST package.

Nonetheless, evaluating our algorithm performance with this package would be certainly an interesting step. 
For example: after training our models on the colored MNIST dataset, it would be interesting to observe how
our models perform if we intervene with a digit image perturbed with swelling or fracture. How does using 
a set of diverse and unseen interventions affect the interventional samples generated from the algorithm? We would
include these evaluations in the camera-ready version of the paper.
We mentioned the URL in our paper, we will cite it now.

Reviewer comment (Writing):
It would be nice to spell out the differences of this algorithm in case of no unobserved confounding.

Author response:
When there exists no unobserved confounder in the causal graph, each variable is an individual c-component of size 1.
Thus, our algorithm will first visit step 4 and then directly go
to step 6: the base case. At step 6, due to no confounding, we simply train one conditional model for each variable
$V_i$ to match $P(V_i|V_{\pi}^{(i-1)}))$, i.e., the probability of $V_i$ given all variables appear before it in the topological
order. We will make it more clear in the paper.

When there exist confounders in the causal graph, we would have larger c-components, and we will recursively solve the
problem for each c-component.

Reviewer comment (Theory):
It seems that most equations are written assuming discrete variables. Why is that?

Author response:
We followed the notation of the ID algorithm (Shpitser et al 2008), which is practical when variables are discrete,
which is a major drawback. We used those equations to indicate which steps of
the ID algorithm we are following and how we are replacing them. Our algorithm is not limited to discrete variables and can adapt to
continuous high-dimensional probability distributions. We will clarify that the p(.) could refer to any probability measure, continuous or discrete.

* Shpitser et al (2008) "Complete identification methods for the causal hierarchy."

Reviewer comment (Writing):
Fig. 2 uses \(Y_2\) while the text mentions \(Y\) – is this a typo?

Author response:
We thank the reviewer for pointing out the typo. We apologize for this and already fixed it.

Reviewer comment (Writing):
What does it mean for a digit to be thin or thick?

Author response:
By thin or thick, we referred to the thickness of the image of the digit. A thick
digit image implies that the lines in that image will be comparatively thicker. Thin refers to the opposite. Please
check Figure 4.
The MorphoMNIST package (the one the reviewer suggested) allows one to perturb an image according to some properties
such as
thickness (high, low).

Reviewer comment (Theory):
For the covid graph, in the real world wouldn't we assume an edge from C→N as well?

Author response:
We apologize for being imprecise.
For the reviewer's convenience, we re-state the graph here: $C \rightarrow X \rightarrow N; C \leftrightarrow N$.
We considered C as the COVID symptoms and N as the pneumonia diagnosis by a healthcare specialist. We assumed that
pneumonia
diagnosis is feasible
through the inspection of an X-ray image ($X \rightarrow N$) and an Xray image is affected by the presence of Covid
symptoms ($C\rightarrow X$).

Since COVID symptoms affect pneumonia diagnosis only through the inspection of the X-ray image by some specialists,
we assume no direct edge from COVID-19 to Pneumonia detection.
We will make it precise in our paper.