Paper: Advancing Counterfactual Inference through Quantile Regression (ICLR-2024)

Reviewer comment (Writing):
In the Introduction section, notations X, Y, Z, and E are used before being defined. Figure 1 only shows the dose-response curve of X on Y, which is a typical continuous treatment setting. Additionally, the green nodes are not defined, and the τ-value is also not reflected in the figure.

Author response:
Thanks for your suggestions. We have added the definitions of $X, Z, E$ in the Section 1. The green nodes denote the sample that has the same $x,z$ but different noise $e$ and they lead to different CDF $P(Y\leq y|x,z)$. We have added such an explanation in the updated draft. Thanks!

Reviewer comment (Theory):
Wrong Critical Statement: In causal inference, most works assume that the covariates Z are collected before the treatment X is administered, and the outcomes Y are observed after the treatment is implemented. This is fairly common in real-world applications. For example, doctors provide treatment based on the patient's health condition, and these variables also affect the patient's final treatment outcome. The reviewer believes that such causal models and assumptions are reasonable and valid, and therefore disagrees with the author's criticism of the causal models proposed by Johansson et al., 2016; Yoon et al., 2018; Bica et al., 2020; Liuyi Yao, 2018, claiming that they are often unknown and difficult to identify with limited samples.

Author response:
We apologize for potential confusion about the sentence in Section 1.  We did not mean to criticize the baseline methods as "causal models and assumptions are not reasonable and valid".  We also totall agree that in many cases "covariates Z are collected before the treatment X is administered, and the outcomes Y are observed after the treatment is implemented".   Our main point is that some previous methods need to estimate a regression model or structural causal model, which may be hard to identify in practice. For instance, it can be very challenging to identify the true function $f$ and the noise $E$ where $Y=f(X,Z,E)$, given only observations $X,Z,Y$.  while our proposed method does not need to identify the causal model but uses quantiles alone.

Reviewer comment (Theory):
Potential Contradictory Statement: There seems to be some inconsistency in the paper. On one hand, the paper states that causal models are often unknown and hard to identify in practical applications with finite samples. On the other hand, the paper relies on the same predefined causal model, requiring covariates Z as a set of common causes for X and Y. This seems to contradict the previous statement.

Author response:
Thanks for your comments. Note that we only assume the qualitative causal structure $X\rightarrow Y \leftarrow Z$ following most baseline methods, but not the quantitative causal model. When we say causal model, we refer to the functional causal model, i.e., $Y=f(X,Z,E)$. It is challenging to recover the true $f$ from the samples without any conditions. We have revised the draft to make it clearer.

Reviewer comment (Theory):
Question: Can the proposed method address unmeasured confounding bias, specifically eliminating the indirect effect from unmeasured confounders, i.e., \( E \not\perp (X; Z; Y) \)?

Author response:
If there exist confounders between $X, Z$, our method can still recover the counterfactual outcome as our assumption in theorem 1 is not violated. However, if the confounder causes $X,Y$ or $Z,Y$, the counterfactual outcome may not be identifiable since $P(Y|X,Z)$ cannot represent the property of  the noise $E$ again due to the existence of confounder. We have provided the results of three confounding cases in appendix F of our updated draft. The empirical results aligns with our analysis that we are still able to perform accurate counterfactual inference even in the presence of confounder between $X,Z$.

Reviewer comment (Evaluation):
Large part of the evaluations focus more on the "interventional" performance rather than the "counterfactual" performance. The average treatment effect evaluation actually assesses E[Y|do(T=0)] - E[Y|do(T=1)], which are quantities derived from rung 2 interventional causal queries that do not require counterfactuals for the estimation. While I see the point of formulating it as the difference over the counterfactual outcomes of individual observations, it actually reduces to the difference between the means of the interventional distributions. Here, additional comparison with point-wise counterfactuals would be more insightful, as done in the related deep-learning literature you are referencing.

Author response:
Thank you for your valuable suggestions. In response, we have presented the results of individual-level counterfactual inference in Table 1 and plan to conduct additional experiments along the same lines, as per your suggestion. The rationale behind comparing our methods with those in Individual Treatment Effect (ITE) is that our proposed approaches are versatile and can be effectively applied to ITE scenarios as well. In addition, the metric $\epsilon ATE$ measure the population-level as you mentioned, but a lower $\sqrt{\epsilon PEHE} $ also needs the invidual predictions to be close to the ground truth counterfactual outcomes.

Reviewer comment (Writing):
In the three-step procedure’s prediction step, it can be clearer specified that estimated noise values are utilized, instead of just mentioning "U".

Author response:
We have made it clearer in the updated version. We also have unified the notations for noise terms. Thanks!

Reviewer comment (Experiments):
Many deep learning models cited in related works are not used for comparison, despite being considered state-of-the-art for counterfactual inference. Why not?

Author response:
We want to respectively emphasize that all methods we compare are most recent deep-learning-based works. Specifically, BGM uses the neural spline flow models. CFQP-U uses the Unet architecture, which is also adopted in the large text2image diffusion model, such as stable diffusion and DALLE-2.  As for CQFP-T, it leverages the powerful transformer architecture, which is the key to recent large language models.  There are mainly two reasons that we did not add comparisons with some baselines in the related work: 1) They do not have published code, e.g., CTRL. 2) Although they are related to our work, they are mainly developed for different tasks; e.g., "Diffusion Causal Models for Counterfactual Estimation" is developed for image generation.

Reviewer comment (Theory):
The paragraph after Theorem 1 needs more clarification: In what scenarios is it not identifiable when using the mean, while it is using quantiles?

Author response:
We appreciate your good question. When using the mean, there is no guarantee that the counterfactual outcome is identifiable because it just considers the average of sample points and ignores the individual property, which makes the counterfactual outcome impossible. In comparison, we estimate the quantile which represents the specific property of the individual sample noise (first step of Pearls' counterfactual framework), and then we can perform counterfactual inference and the outcome is identifiable as long as our monotonic assumption holds.

Reviewer comment (Theory):
While you only refer to "discrete data", it should be noted that the functional causal models listed (e.g., additive noise model) are intended for non-categorical data. In categorical cases, point estimates for counterfactuals are not possible.

Author response:
Thanks for your reminder. The definition of a structural causal model, as given in [1], does not have a limitation on data types; it applies to both continuous and discrete data. Our theory holds as long as the monotonic assumption is satisfied. However, as you probably intended to say, if the output $Y$ is truncated or manually categorized into discrete, the point estimate is not possible.

Reviewer comment (Theory):
In the modeling part of your method, where do assumptions such as monotonicity come into play, and how they are encoded in the model? In the list of models you provide at the end of Section 3, this is for instance encoded by restricting the model classes to ensure invertibility with respect to E. Or in case of an autoregressive flow model, this is given by the invertibility property, etc.

Author response:
Thanks for pointing this out. We assume that the true model class satisfies our monotonic assumption. Moreover, during the estimation, we make use of quantile properties, which do not need to estimate the causal model explicitly.

Reviewer comment (Theory):
I believe that the Introduction should provide a more detailed explanation of the main idea of the paper. Currently, I am unable to understand how the main idea of the paper is introduced in Section 1. Specifically, I am confused about why $P(Y \leq Y_{X=x'} | x',Z=0.5) = P(E \leq -0.5)$. If we are given $X=x'$, then $Y = Y_{X = x'}$. Therefore, it seems to me that $P(Y \leq Y_{X=x'} | x',Z=0.5) = P(E \leq -0.5) = 1$ since $Y = Y_{X = x'}$ given $X=x'$.

Author response:
Thank you for your careful reading. Below, we give a more detailed explanation and we replaced replaced $Y_{X=x^\prime}$ with the $y^\prime$ to indicate that it is a real value in our updated version.. The causal model is defined as $Y=f(X,Z,E)=\sin(4\pi X)+Z+E$. When we conditional on $X=x^\prime, Z=0.5$, we have $Y=\sin(4\pi x^\prime)+0.5+E$ is a random variable and the true counterfactual outcome for the particular sample $\langle x,z,y \rangle$ is defined as a real value $y^\prime=\sin(4\pi x^\prime)+0.5+e=\sin(4\pi x^\prime)+0.5+(-0.5)=\sin(4\pi x^\prime)$. Therefore, $P(Y\leq y_{X=x^\prime}|X=x^\prime, Z=0.5)=P(\sin(4\pi x^\prime)+0.5+E \leq \sin(4\pi x^\prime)) = P(0.5+E\leq 0)  = P(E\leq -0.5)=\Phi(-0.5)$. At the same time, we have $P(Y\leq y|X=x, Z=z)=P(\sin(4\pi x)+z+E\leq y)=P(\sin(4\pi x)+z+E\leq \sin(4\pi x)+z+e)=P(E\leq e)=P(E\leq -0.5)=\Phi(-0.5)$. So, we have $P(Y\leq y_{X=x^\prime}|X=x^\prime, Z=z)=P(Y\leq y|X=x, Z=z)$. The equation allows us to perform counterfactual inference by estimating the quantile of the conditional distribution instead of identifying the true functional causal model $f$.  We have updated the caption in figure1 and provide the detailed explanation in the appendix A due to space limitation.

Reviewer comment (Theory):
This paper lacks mathematical rigor, resulting in logical gaps in the proof of Theorem 1. These gaps undermine the credibility and validity of the results. For instance, in the proof of Theorem 1, what is meant by "$Y_{X=x'} \vert X=x, Y=y, Z=z$ can be calculated as $f^i(X=x',Z=z,E=e^i)$"? If this implies that they are equal, please explain the reasoning behind their equality. It appears to me that $f^i(X=x',Z=z,E=e^i)$ represents $Y_{X=x'}(e^i)$ when $Z=z$. Even if $e^i$ is evidence of $(X=x,Y=y,Z=z)$, it does not imply that $Y_{X=x'}(e^i)$ is equal to $Y_{X=x}' \vert X=x, Y=y, Z=z$. The former is a real value, while the latter is a random variable.

Author response:
Thanks for your comment, and we would like to make the following clarifications for "$Y_{X=x'}|X=x,Y=y,Z=z$ can be calculated as $f^i(X=x',Z=z,E=e^i)$".  

Given the observed evidence $X=x,Y=y,Z=z$, we can determine the causal model $f^i$ and the associated noise value $e^i$. (Notice that we use the index $i$ to emphasize that the causal model and noise value may not be uniquely identifiable.) As $f^i$ and $e^i$ have been determined, then based on the causal model, we can calculate the value of the counterfactual outcome $Y_{X=x'}|X=x,Y=y,Z=z$, which is $f^i(X=x',Z=z,E=e^i)$. It is crucial to emphasize that $Y_{X=x^\prime}(X=x, Z=z, E=e)$ is a real value, as well as $f^i(X=x',Z=z,E=e^i)$. Our approach aligns with Judea Pearl's definition of counterfactual outcomes for each individual, which is deterministic. This stands in contrast to the potential outcome framework, which defines outcomes for populations in a probabilistic manner.

Reviewer comment (Theory):
In the abstract, “Traditional counterfactual inference usually assumes the availability of a structural causal model” is a false statement. No existing causal inference frameworks assume the availability of structural causal models. Instead, they assume a proxy of the causal model, such as a graph and a distribution (or data that follows the distribution). As an alternative to the graph, the conditional independence embedded by the structural causal model can be assumed (e.g., ignorability). Overall, the availability of the causal model is not assumed. However, some restrictions on the distribution in the form of a graph or independence are assumed.

Author response:
Thanks for the point. In this paper, we focus on the counterfactual under Judea Pearl's framework, which is about the counterfactual outcome for each individual. For example, we want to answer the question: what would George cough had he had yellow fingers, given that he does not have yellow fingers and coughs?  

It is different from the potential outcome framework, which is about the expectation of the population-level counterfactual estimation, under the same assumption as you mentioned in your comments.  

Specifically, in Judea Pearl's counterfactual framework, for most previous methods, the causal model needs to be given or estimated from data as you can see from Pearl's three steps for counterfactual inference, i.e., Abduction, Action, and Prediction, provided in Section 2. A more detailed example is given in Model 4.1 on page 95 of  "CAUSAL INFERENCE IN STATISTICS" by Judea Pearl. It assumes a linear causal model and uses the evidence to get the noise of the interested sample, then feed the noise into the pre-assumed SCM to perform counterfactual inference.  
We have updated our abstract to make it more rigorous. Thanks!

Reviewer comment (Theory):
More explanation is needed for the equation $P(Y \leq Y_{X=x'} | x',Z=0.5) = P(E \leq -0.5)$.

Author response:
Thank you for your careful reading. Below, we give a more detailed explanation and we replaced replaced $Y_{X=x^\prime}$ with the $y^\prime$ to indicate that it is a real value in our updated version.. The causal model is defined as $Y=f(X,Z,E)=\sin(4\pi X)+Z+E$. When we conditional on $X=x^\prime, Z=0.5$, we have $Y=\sin(4\pi x^\prime)+0.5+E$ is a random variable and the true counterfactual outcome for the particular sample $\langle x,z,y \rangle$ is defined as a real value $y^\prime=\sin(4\pi x^\prime)+0.5+e=\sin(4\pi x^\prime)+0.5+(-0.5)=\sin(4\pi x^\prime)$. Therefore, $P(Y\leq y_{X=x^\prime}|X=x^\prime, Z=0.5)=P(\sin(4\pi x^\prime)+0.5+E \leq \sin(4\pi x^\prime)) = P(0.5+E\leq 0)  = P(E\leq -0.5)=\Phi(-0.5)$. At the same time, we have $P(Y\leq y|X=x, Z=z)=P(\sin(4\pi x)+z+E\leq y)=P(\sin(4\pi x)+z+E\leq \sin(4\pi x)+z+e)=P(E\leq e)=P(E\leq -0.5)=\Phi(-0.5)$. So, we have $P(Y\leq y_{X=x^\prime}|X=x^\prime, Z=z)=P(Y\leq y|X=x, Z=z)$. The equation allows us to perform counterfactual inference by estimating the quantile of the conditional distribution instead of identifying the true functional causal model $f$.  We have updated the caption in figure1 and provide the detailed explanation in the appendix A due to space limitation.

Reviewer comment (Writing):
What does $E \perp (X;Z)$ mean? Does it mean that $E \perp \{X,Z\}$ or $E \perp X \vert Z$?

Author response:
We appreciate your question, we mean the former one, i.e., that the noise $E$ is independent from $X$ and $Z$. That is, $P(E,X,Z)=P(E)P(X,Z)$. We also have updated it to avoid future confusion in our manuscript. Thanks!