Paper: Causal Inference with Conditional Front-Door Adjustment and Identifiable Variational Autoencoder (ICLR-2024)

Reviewer comment (Theory):
There seems to be a gap in the theoretical part. The model identifiability analysis in Section 4.3 shows that the front-door adjustment variable Z obtained with VAE is identifiable up to a transformation (more or less the result from Khemakhem, 2020). But Theorem 3 is based on the assumption that the front-door adjustment variable Z is observed. It is not clear if Theorem 3 is still applicable if it is used for a transformed Z.

Author response:
Regarding the first weakness, we think there might be some misunderstanding by the reviewer. We do not assume Z_CFD is observed. In fact, we consider Z_CFD to be unobserved and it is why CFDiVAE learns the representation of Z_CFD that satisfies the CFD criterion such that we can use the learned representation for CFD adjustment by following Theorem 3 (specifically Eqn. (3)). In other words, Theorem 3 states how to do CFD adjustment when a CFD adjustment variable is given, but this does not mean that our work assumes a CFD adjustment variable is known.

Reviewer comment (Writing):
Some small inconsistencies in the notation, at least: Equation (4): LHS has T and W but RHS does not. Equation (6): formulas for the mean and variance have j on the LHS but not on the RHS.

Author response:
Following your suggestions, we addressed the inconsistencies in the notation and added two comparison methods based on the standard front door adjustment. 

Please find the latest version in the Rebuttal Revision.

Reviewer comment (Theory):
It is very hard to recover the adjustment variable from the proxy if the proxy noise is not very small. The proxy setting is challenging in itself, and the only rigorous identification result under this setting known to the reviewer is from Miao et al. (2018), which depends on two types of proxies and has several assumptions about the proxies. In the current theory, the results are meaningful only under small proxy noise (ε in eq10). As seen in eq25, only \( f^{-1}(X) \) is identified (up to deterministic transformation) but not the latent variable. Furthermore, \( f^{-1}(X) = f^{-1}(f(Z)+\epsilon) \neq Z \) if \( \epsilon \neq 0 \). If \( \epsilon = 0 \), then X and Z are related by an injective function, making it less meaningful to call X a proxy of Z, since the proxy setting presumes unmeasured information of Z not captured by X. This weakness does not need to be fixed in the rebuttal, but should be discussed and made explicit in the revised version.

Author response:
Regarding your first weakness about the difficulty of recovering the adjustment variable from its proxies, we have expanded the discussion in the latest version of the paper and included references [3,4] as you suggested.

Reviewer comment (Theory):
The connection between the front-door setting and the proposed VAE is weakened because the outcome is not modeled by the VAE, and the outcome estimation is done in a standalone step (Sec 5). It is unclear how this two-step process implements adjustment eq3, and how exactly the ATE is computed. Additionally, the adjustment and outcome variables follow linear models, which may not be a fundamental limitation but should be mentioned in the Conclusion. For an important component of the approach, citing a web article is concerning; an original reference from a research paper should be cited.

Author response:
Outcome estimation: As you recommended, we have now provided more detailed explanations to bridge the gap between Equation (3) and the ATE estimation method. In the latest version, we use the discrete data case as an example to demonstrate that Algorithm 1 in Appendix C aligns with Equation (3). Although the solution for continuous data with the linear model does not mirror Equation (3), it is still consistent with the (conditional) front-door adjustment shown in Theorem 3. As we know, the front-door adjustment consists of two back-door adjustments. In our case with continuous variables, these two back-door adjustments are implemented by regression adjustments, as shown in Equation (30) and Equation (31) in Appendix C. Specifically, for the first backdoor adjustment, we use regression adjustment to get the causal effect of $T$ on ${Z_{\text{\rm CFD}}}$ (i.e., $\beta_{T,{Z_{\text{\rm CFD}}}}$) adjusting on $W$, as shown in Equation (30) and Step 1 of the process in Appendix C; for the second backdoor adjustment, we also use regression adjustment as shown in Equation (31) in Appendix C to obtain the causal effect of ${Z_{\text{\rm CFD}}} $ on $Y$ adjusting on $W$ and $U$. However, for this regression adjustment, since $U$ is unobserved, we follow the approach [1-2] to firstly obtain the error $e_{Z_{\text{\rm CFD}}}$, which is independent of $U$, and regress $e_{Z_{\text{\rm CFD}}}$ on $Y$ instead. In this way, we can eliminate the confounding bias caused by $U$. This is shown in Step 2 of the process in Appendix C.

Reviewer comment (Experiments):
The experiments are insufficient or unclear. There are no comparisons to methods designed for the front-door setting, making the results in Fig 4 less meaningful. The reviewer questions the exclusion of FINDFDSET and LISTFDSETS since they require a known DAG, which is available in synthetic datasets. The claim that there are no other methods for the FDA is doubtful. If this cannot be addressed, it should be mentioned in the Conclusion.

Author response:
Following your recommendations, we have incorporated FindFDSet and ListFDsets into our experimental comparison and have stated the limitations of our model, particularly its diminished performance with small sample sizes.

Reviewer comment (Experiments):
As shown in Table 1, the proposed method requires a very large amount of data to perform well. In prior work, sample sizes under 1k were common, but the proposed method performs worse than others when using fewer than 1k samples. This should be examined seriously, as it could be a major weakness if it is a general observation.

Author response:
Following your recommendations, we have incorporated FindFDSet and ListFDsets into our experimental comparison and have stated the limitations of our model, particularly its diminished performance with small sample sizes.

Reviewer comment (Experiments):
The experiment on the Adult dataset is confusing. First, the “proxy” X also affects Y (as shown in the Appendix), raising the question of whether it can still be termed a “proxy” and whether the Sec 5 process still applies. Second, the discussion of direct and indirect causal effects cannot be understood without referring to the Appendix; it should be stated that X (e.g., marital status) also affects Y and is the “mediator” here. Third, statements about “significant indirect discrimination … via marital status” and “significant discrimination against sex through the stereotype” seem contradictory.

Author response:
Concerning the Adult dataset, we concur that there is no causal path from W to T, as illustrated in Figure 8. We retain the causal path W to Y based on the comprehensive DAG. In our experimental setup, we categorize all mediating factors under the umbrella of stereotypes, which are regarded as proxies for stereotypes. Thus, the findings on indirect discrimination and discrimination through stereotypes are in alignment. Owing to the scarcity of real-world examples, we can only substantiate the applicability of CFDiVAE in this manner. Following your advice, we will include a discussion on direct and indirect causal effects in the main body of the text.

Reviewer comment (Theory):
The reviewer believes only the link W → Z is necessary, not W → T and W → Y. The Adult dataset does not have W → T.

Author response:
Concerning the Adult dataset, we concur that there is no causal path from W to T, as illustrated in Figure 8. We retain the causal path W to Y based on the comprehensive DAG. In our experimental setup, we categorize all mediating factors under the umbrella of stereotypes, which are regarded as proxies for stereotypes. Thus, the findings on indirect discrimination and discrimination through stereotypes are in alignment. Owing to the scarcity of real-world examples, we can only substantiate the applicability of CFDiVAE in this manner. Following your advice, we will include a discussion on direct and indirect causal effects in the main body of the text.

Reviewer comment (Novelty):
The Contributions and Conclusion should more clearly state that it is the FDA—a well-established result—that addresses unobserved confounding in this approach. The VAE learns the adjustment variable but does not perform the adjustment using eq3. The CFD is a slight extension of the original FDA.

Author response:
Following your recommendations, we have incorporated FindFDSet and ListFDsets into our experimental comparison and have stated the limitations of our model, particularly its diminished performance with small sample sizes. We have also revised any unsupported or misleading statements, ensuring that our contributions and conclusions are articulated more clearly.

Reviewer comment (Theory):
The statement “there exist unobserved confounding variables and the standard front-door adjustment is no longer applicable” can be misleading, as it suggests FDA does not handle unobserved confounding, but the proposed VAE does.

Author response:
We have also revised any unsupported or misleading statements, ensuring that our contributions and conclusions are articulated more clearly.

Reviewer comment (Theory):
The statement “the latent variable (i.e., ZCFD) refers to the variable that is not measured, but its information is captured by its proxy” is problematic. As noted, a proxy does not capture all the information of a latent variable; it might capture enough for causal effect estimation, but this requires challenging analysis and specific assumptions as in Miao (2018).

Author response:
Regarding your first weakness about the difficulty of recovering the adjustment variable from its proxies, we have expanded the discussion in the latest version of the paper and included references [3,4] as you suggested.

Reviewer comment (Experiments):
Additional results on real-world datasets and analysis could strengthen the paper.

Author response:
Thanks for your review and insights. We revised the typo in the latest version. Following your comments, we updated more explanations for the real-world application, added two comparison methods based on the standard front door adjustment, and conducted additional experiments on dimensionality analysis. Please find the latest version in the Rebuttal Revision.

Reviewer comment (Presentation):
The section references in the supplemental file are confusing. E.g., the title of section C.4 refers to ‘Section 4.5’ in the main text. However, section 4.5 in the main text does not exist. This causes trouble for the readers.

Author response:
Thanks for your review and insights. We revised the typo in the latest version. Following your comments, we updated more explanations for the real-world application, added two comparison methods based on the standard front door adjustment, and conducted additional experiments on dimensionality analysis. Please find the latest version in the Rebuttal Revision.

Reviewer comment (Experiments):
The authors could apply the method to additional real-world datasets, conduct a comparison with existing methods, and provide the analysis to strengthen the paper.

Author response:
Thanks for your review and insights. We revised the typo in the latest version. Following your comments, we updated more explanations for the real-world application, added two comparison methods based on the standard front door adjustment, and conducted additional experiments on dimensionality analysis. Please find the latest version in the Rebuttal Revision.

Reviewer comment (Presentation):
The author needs to address the errors in the section reference.

Author response:
Thanks for your review and insights. We revised the typo in the latest version. Following your comments, we updated more explanations for the real-world application, added two comparison methods based on the standard front door adjustment, and conducted additional experiments on dimensionality analysis. Please find the latest version in the Rebuttal Revision.