Paper: Neuro-Causal Factor Analysis (ICLR-2024)

Reviewer comment (Experiments):
How sensitive the results are to the hyperparameters? And in practice, how should users choose them?

Author response:
> how sensitive the results are to the hyperparameters? and in practice, how should users choose them?

The main hyperparameter, lambda, is similar to the choice of latent dimension in any generative model: If lambda is too small, then we would run into the same kinds of problems training any generative model with insufficient capacity. It does not have to be tuned carefully in practice.

More precisely, lambda should be greater than the number of causal latents, but how much greater is not as important. In practice (and our experiments on real data), VAEs are likely to have high enough latent DoF that experimentally exploring this further is beyond the scope of our presentation of the NCFA framework. Also see our response to the third question of Reviewer WRCe for discussion of how this parameter should be chosen by users.

Reviewer comment (Experiments):
Is NCFA scalable for larger datasets and denser causal connections? What's the running speed of this method?

Author response:
> is NCFA scalable for larger datasets and denser casual connections? what's the running speed of this method?

Our real data application in Section 5 (especially Table 1, and further discussion in appendices F and C) demonstrates that NCFA scales to large data sets like MNIST, even with dense (fully connected) structures and up to a 1/4 of a million latent degrees of freedom in the VAE. The computational bottleneck is the NP-hard problem of finding an exact minimum edge clique cover, however our applications in Section 5 demonstrate that using polynomial-time ECC heuristics (based on the 1-pure-child assumption) bring the runtime to the order of standard VAE methods without sacrificing much of the accuracy of the exact method.

Reviewer comment (Experiments):
How does NCFA compare to other causal inference methods for deep generative models?

Author response:
> how does NCFA compare to other causal inference methods for deep generative models?

This is discussed in Section 2.3. Our results are complementary to existing results, and make different, sometimes weaker, assumptions. We are happy to clarify any more specific questions you have on this.

Reviewer comment (Theory):
The first part of NCFA depends on the quality of UDG and minimum MCM graphs. UDG appears to be a Markov Network for statistical independence representation. By the nature of Markov Networks, UDG can be guaranteed to be an I-map but not be a perfect map for the data generation distribution. The UDG and induced minimum MCM graphs therefore may not always capture the underlying causal structure faithfully. For example, when we have a M_1 to M_5 forming a Markov chain. Based on the described algorithm, the UDG may just have a complete graph with 5-node clique, which does not really provide any additional causal understanding or constraints in NCFA.

Author response:
Such graphs are not within the model class we are concerned with. Namely, your example is Markov to a DAG over the measurement variables, while we are explicitly concerned with causal factor models, which we motivate beginning in the second paragraph of Section 1.

Reviewer comment (Theory):
The authors may want to clearly define what they meant by NCFA being 'non-parametric' and 'identifiable'. For example, the overall FA framework is a parametric setting and VAEs also typically are considered parametric for the source-measurement mapping. If the authors meant that there is no need to specify mapping functional families as being 'non-parametric', it may need to be explained clearly. It may also not be fair to claim the advantages on identifiability of minimum MCM graphs in NCFA over the existing other causal structure discovery methods due to the assumed specific settings.

Author response:
Our _statistical model_ is nonparametric (infinite-dimensional without functional restrictions), whereas the _estimator_ is parametric (finite-dimensional). The latter is of course always necessary if it is to be implemented on a computer, e.g. as with a neural network.  A similar phenomenon arises in density estimation: The model (usually H"older, Sobolev, etc.) is infinite-dimensional, but the estimator is finite-dimensional (e.g. kernel estimates). In other words, VAEs are parametric estimators of nonparametric models.  

To the best of our knowledge, our identifiability results are the first of their kind for nonparametric causal factor models. Other methods make parametric assumptions and use this added power to relax their model class at the expense of stronger assumptions. Our approach has the advantage of not needing the stronger parametric assumptions, but of course if one is willing to make such assumptions for a given application, other methods are more appropriate.

Reviewer comment (Writing):
What does MCM stand for? Medil Causal Models? If so, please provide the full name in the main text too.

Author response:
Yes, MCM stands for MeDIL causal model. We have added this to the main text.

Reviewer comment (Reproducibility):
What are the details of model structure and training, especially the VAE model constructed following the induced minimum MCM graphs? These do not appear to be included either in the main text or appendix.

Author response:
These details are given in Appendix C.

Reviewer comment (Theory):
Fig 2: it seems that all graphs can be aligned as the hierarchical structure. Is this an assumption or based on some theoretical results?

Author response:
It is a well-known fact (due to Pearl) that any latent variable model can be represented minimally in this way. So this is not an assumption, and follows from known theoretical results.

Reviewer comment (Writing):
Definition 3.5: why this is called "observationally equivalent"? How about calling “equivalent wrt d-separation”?

Author response:
The phrase "observationally equivalent" is standard in the causal graphical models literature. For example, see Judea Pearl's seminal book "Causality", which uses the phrase in Theorem 1.2.8 (on page 19, in the printing from 2000).

Reviewer comment (Writing):
Clarity is one of the main issues of this work. Many terminology descriptions are missing or without explanation. For example, what is the full name of MCM graph and ECC model?

Author response:
MCM abbreviates "MeDIL causal model", and ECC abbreviates "edge clique cover". The MCM graph is rigorously defined in Definition 3.2 and ECC-observational equivalence is rigorously defined in Definition 3.5.

Reviewer comment (Theory):
The notations are confusing and the theoretical results in this paper are problematic. Theorem 3.6 states that the DAG G is identifiable with the conditions that $M_{i}\perp M_{j}\iff i-j\notin E^{\mathcal{U}}$, which is problematic. Since a DAG identifiable means that every direction for every edge in the DAG will be identified, which is impossible without any assumptions under the existence of a latent variable. In fact, this theorem relies on the theory of Markham & Grosse-Wentrup, 2020 which has several underlying assumptions and constrained which is missing in the statement of the theorem.

Author response:
The underlying assumptions and theory required for Theorem 3.6 are clearly stated in the first sentence of the theorem, in the phrase "the data-generating distribution is Markov to a minimum MCM graph". Being Markov to a minimum MCM graph means that each clique in the ECC of U implies the existence of a latent variable in G (and this is clearly stated in the proof, which explicitly references all necessary supporting theory). This latent variable (by the assumption that the MCM graph is minimum), necessarily has exactly the edges directed to the measurement variables in its corresponding clique. Hence, the clearly stated assumption renders the latent DAG (and every direction of every edge in it) identifiable.

Reviewer comment (Experiments):
The algorithm's results depend on the latent degree of freedom lambda.

Author response:
The experiments show that this parameter does _not_ need to be carefully tuned, and indeed this is a key contribution. Simply put, we need lambda to be larger than the true number of causal latents, and small enough to train in practice. In other words, lambda could be in the thousands and it does not matter very much. (If lambda is too small, then we would run into the same kinds of problems training any generative model with insufficient capacity.)

Reviewer comment (Theory):
How can we ensure the identifiability of loading functions (f) and residual measurement errors?

Author response:
Our focus is on identifying the causal structure. We have left identifiability of the parameters to future work (and would note that there is already extensive work on this problem). Our approach is to emphasize causal interpretability and accurate generative modeling as opposed to factor loading identification.

Reviewer comment (Theory):
Could you provide an intuitive proof framework to demonstrate the correctness of Theorem 3.6? Are there any graphical conditions when UDGs have a unique minimum edge clique cover?

Author response:
In Corollary 3.7 (and its proof in Appendix B), we prove that a UDG has a unique minimum ECC when it has equal intersection and independence numbers. We also show that this condition on UDGs is equivalent to the 1-pure-child condition on latent DAG models known in the literature.

Reviewer comment (Experiments):
In practice, how do we set the latent degree of freedom lambda?

Author response:
The experiments show that this parameter does _not_ need to be carefully tuned, and indeed this is a key contribution. Simply put, we need lambda to be larger than the true number of causal latents, and small enough to train in practice. In other words, lambda could be in the thousands and it does not matter very much. (If lambda is too small, then we would run into the same kinds of problems training any generative model with insufficient capacity.)

Reviewer comment (Theory):
Section 2.2 on Page 4: which relax the Causal Markov Assumption? It should be the Causal Sufficiency Assumption, right?

Author response:
The causal Markov assumption (CMA) is basically (as mentioned in the second sentence of Section 2.2) Reichenbach's common cause principle plus the causal sufficiency assumption (CSA), so relaxing the CSA is one (but not the only) way of relaxing the CMA.

Reviewer comment (Presentation):
Figure 2: double l_2,2?

Author response:
Yes, thanks for pointing out the typo! It should be l_2,1 and l_2,2.