Paper: Causal Structure Recovery with Latent Variables under Milder Distributional and Graphical Assumptions (ICLR-2024)

Reviewer comment (Theory):
Linear relations: authors make a linearity assumption between latent-latent, latent-observed, and observed-observed features. The linearity assumption between observed-observed features seems very strong.

Author response:
**A1**:We have added some explanations about this point after the definition of the linear latent variable model (Definition 1) in Section 2.

First, most existing works about latent causal structure learning focused on the linear case. Although few works can avoid this assumption, they typically have other significant limitations. For instance, [1] allows non-linearity, but it requires that all latent variables are discrete; a contemporaneous work [2] allows non-linearity, but it forbids causal edges between any two variables that share at least one common parent.

Second, some theoretical results in our paper can actually generalize to two specific non-linear cases. We provided a detailed discussion in Appendix C.1. Roughly speaking, thanks to the Extended Trek Separation Theorem [3], all results in Section 3.1 and Section 3.2 are still valid as long as all causal relations involving generalized pure children (pure children and variables in a pseudo-pure pair) are still linear. Besides, if the underlying causal model follows a specific post-nonlinear model [4], we find that observed variables follow a nonparanormal distribution [5] and all theoretical results in Section 3.1 and Section 3.2 can still hold.

Finally, we agree that nonlinear is common in the real world and we list the problem of causal discovery for nonlinear cases as our future work.

Reviewer comment (Theory):
Generalised pure pair seems to be a very strong requirement; how do you ensure this is followed in practice? For that matter, how do you test for this?

Author response:
**A2**: Considering that many previous works mentioned in our paper assume that each variable has at least two or three pure children, we politely disagree that our requirement for a generalized pure pair is very strong since two pure children are exactly a special case of a generalized pure pair. In other words, **our requirement for a generalized pure pair is much weaker compared to previous requirement for pure children**. Pure children are believed to exist because sparsity of causal edges holds in many fields such as biology and physics [6].

In fact, testing assumptions about the structure of the underlying causal model is a quite difficult problem, especially those involving latent variables. For instance, even the casually sufficiency assumption that there exists no latent variable cannot be tested in most cases, let alone more complicated assumptions about pure children or generalized pure pairs. It may be possible to test this assumption with some auxiliary information (such as interventional data), but this problem is out of our current scope, we will leave it to future work.

By the way, there really exists some works that can recover the latent causal structure without pure children or generalized pure pairs, but they require other potentially stronger assumptions. An example is [7], whose assumption implies that latent variables have a sufficient number of observed children, the difference between latent variables in the number of observed children is small enough, any two latent variables do not share too many common observed variables, etc.

Reviewer comment (Reproducibility):
Algorithmic description is missing; please provide a detailed description of the algorithm along with the intuition for the steps.

Author response:
**A4**: Thanks for pointing this out. We have provided detailed versions of our algorithms in Appendix D and illustrative examples to show how each step proceeds.

Reviewer comment (Experiments):
I understand the limited space constraint, but it would be nice to have some details on the experimental setup and the results in the main paper and discuss their implications.

Author response:
**A3**: We have moved the most representative experimental results to the main text as Section 6, which indicates that our algorithms outperform others when none of the non-Gaussianity, purity, and two-pure-children holds. Generally, SID and SHD are used in the case where there exists no latent variable [8,9]. In Section 6, since only our algorithms can recover the whole causal structure involving both latent and observed variables while others can only recover the latent causal structure, we only report the SHD of our algorithms, the average of which is 3.9 and 5.5 on $\mathcal{G}_1$ and $\mathcal{G}_2$ with sample size $N=2000$. In fact, the evaluation metrics used in our paper can sufficiently demonstrate the superiority of our algorithms. Our algorithms achieve 0% and 20% error rate on $\mathcal{G}_1$ and $\mathcal{G}_2$ while others always achieve 100% error rate. This means that in an ideal case where PC-MIMBuild makes no mistake, our algorithms can recover the correct latent causal structure in all cases for $\mathcal{G}_1$ and in 80% cases for $\mathcal{G}_2$ while other algorithms always return wrong results. In Appendix A.1, we have provided extensive experimental results on many other causal graphs. In particular, we also consider the scenario where there exists no latent variable. We are not sure what "synthetic datasets with known latent nodes" means, could you please provide more explanations? Last but not least, please note that the main contributions of our paper lie in our exhaustive theoretical analyses rather than empirical results.

Reviewer comment (Theory):
The proposed method for finding all the generalised pairs is strongly dependent on relations between latent and observational factors being linear, which is not that convincing. It would be nice to see some discussion on relaxing this behaviour or ways to test this behaviour of real datasets.

Author response:
Dear reviewer 7d8K,

Thanks very much for your kind reply. We are glad that our rebuttal has addressed most of your concerns, and we are also happy to give more discussion on nonlinear relations between observed variables.

We agree that many causal discovery approaches (with sufficiency assumptions) don't rely on the linearity assumption. However, please note that recovering the whole causal structure involving both latent and observed variables is much more challenging than causal discovery without latent variables. Besides, most previous works about latent causal structure learning [2,3,4,5,6] directly employed the purity assumption that there is no causal relation between observed variables, and some studies avoiding the purity assumption [1,7] also assumed linear causal relations between observed variables.

Nonetheless, we still delve deeper into non-linearity and provide more discussions in Appendix C.1. Here we present two important conclusions.

1. If only causal relations between $O_1$ and $O_2$ are nonlinear where {$O_1,O_2$} $\subset \mathbf{O}$ is not a pseudo-pure pair, then all theoretical results in Section 3.1, 3.2, and 3.3 still hold. 

2. If all causal relations are nonlinear but follow a special post-nonlinear model defined as Equation (28), then all theoretical results in Section 3.1 and 3.2 still hold.

We hope the above results can address your concerns, at least partially. Finally, please note that the main focus of our work still lies in the "linear latent variable models". After all, even the linear problems with latent variables have not been completely solved. Therefore, non-linearity is somewhat out of our scope, and we list it as our future work. Anyway, thanks again for your constructive suggestions and kind feedback!


Reference

[1] Silva, Ricardo, et al. "Learning the Structure of Linear Latent Variable Models." JMLR 2006.

[2] Cai, Ruichu, et al. "Triad constraints for learning causal structure of latent variables." NeurIPS 2019.

[3] Xie, Feng, et al. "Generalized independent noise condition for estimating latent variable causal graphs." NeurIPS 2020.

[4] Xie, Feng, et al. "Identification of linear non-gaussian latent hierarchical structure." ICML 2022.

[5] Huang, Biwei, et al. "Latent hierarchical causal structure discovery with rank constraints." NeurIPS 2022.

[6] Chen, Zhengming, et al. "Some General Identification Results for Linear Latent Hierarchical Causal Structure." IJCAI 2023.

[7] Xie, Feng, et al. "Causal discovery of 1-factor measurement models in linear latent variable models with arbitrary noise distributions." Neurocomputing 2023.

Reviewer comment (Presentation):
Presentation concerns: please introduce all the assumptions before referencing them for easier readability.

Author response:
**A5**: Thanks for this constructive suggestion. We have introduced all previously used assumptions (including the three-pure-children, two-pure-children, non-Gaussianity, and purity assumption) in the second paragraph of the Introduction. Besides, our proposed assumptions are formulated as Assumption 1, 2, 3 respectively.

Reviewer comment (Presentation):
The paper is not self-contained; please consider including a background section briefly describing previous methods which are directly involved in your framework.

Author response:
**A6**: We have added a paragraph at the end of Section 5 (Relation to Existing Work) to briefly describe previous methods that are involved in our framework, including the Tetrad Representation Theorem [10] (It builds a connection between the structure of underlying causal structure and the covariance of variables that can be calculated from samples), Darmois-Skitovich Theorem [11] (As long as two variables share any non-Gaussian, independent component, they cannot be statistically independent), and PC-MIMBuild [12] (an overview is shown as Algorithm 4). All of them were proposed before 2010 and serve as the cornerstones of many other works. However, other more recent results they derived cannot be directly adopted in our framework where the required assumptions are not satisfied.

Reviewer comment (Writing):
Careful proofreading would improve the paper further.

Author response:
**Q1**: Careful proofreading would improve the paper further

**A1**: We have tried our best to avoid typos in the proof and also invited some colleagues to check the proof. We will continually improve the proof before submitting the camera ready.

Reviewer comment (Theory):
While there are several different results in the paper, it would be great if a common theme could be established, e.g., a more systematic understanding of the right identifiability conditions.

Author response:
**Q2**: Lack of a more systematic understanding of the right identifiability conditions.

**A2**: Thanks for this constructive suggestion. We have provided some explanations in the first paragraph of Section 3.4. Specifically, we formulate two cases where none of the non-Gaussianity, purity, and two-pure-children assumption holds. The two cases both allow causal edges between observed variables but make different trade-offs between graphical and distributional assumptions. On the one hand, Case I which allows completely arbitrary distribution is more general than Case II requires partial non-Gaussianity. On the other hand, Case II which requires no pure child is more general than Case I which requires one pure child per latent variable.

Reviewer comment (Experiments):
This might be a bit much to ask, but can the conditions be tested?

Author response:
**Q3**: Can the conditions be tested?

**A3**: We have proposed an expedient in the second paragraph of Section 3.4. Roughly speaking, Assumption 2 allows arbitrary distribution while Assumption 3 requires partial non-Gaussianity, we test whether some specific variables are all non-Gaussian. If they are, we choose Algorithm 1+3, otherwise, we choose Algorithm 1+2. Although this expedient is not a perfect method, it is better than most previous works that directly circumvented this problem. In fact, testing assumptions is a quite hard problem with only observation data, especially when there exist latent variables. On the other hand, this is really an important problem and we list it as our future work in Section 7.

Reviewer comment (Theory):
The graphical conditions are still quite restrictive (but probably unavoidable). An interesting direction might be to consider when we can answer some 'causal queries' without full identifiability under milder assumptions.

Author response:
**Q4**: The graphical conditions are still quite restrictive (but probably unavoidable) ……

**A4**: We agree that some causal questions can be answered without full identification of the causal graph. This is really an interesting research problem and we list it as our future work.

Reviewer comment (Presentation):
First, part of Section 3 was not very helpful for me because the results are not established at this point.

Author response:
**Q5**: Part of Section 3 was not very helpful for me because the results are not established at this point.

**A5**: To solve this problem, we have moved some subsidiary theoretical results to Appendix.

Reviewer comment (Writing):
The definition of the set $\mathbb{S}$ seems to be missing. (Only in Algorithm it becomes clear what it is).

Author response:
**Q6**: The definition of the set \mathbb{S} seems to be missing.

**A6**: Thanks for pointing this out, we define it after Definition 5 (definition of generalized pure pair) in the revised manuscript.

Reviewer comment (Theory):
I could not follow Assumption 2. Maybe this can be clarified. Does it mean $Nei(L)\geq 4$ and, moreover, in the case where $Nei(L)=4$ such that there is one latent and three observed neighbours then there is a pure pair...?

Author response:
**Q7**: I could not follow Assumption 2. Maybe this can be clarified.

**A7**: We are sorry that this causes confusion. We have clarified it in the new manuscript. Assumption 2 means that if a latent variable L has exactly one latent parent L’, three observed children $O_1$, $O_2$, $O_3$, and two of which (without loss of generality, $O_1$ and $O_2$) are its pure children, then the neighbors of $O_3$ are required not to be exactly $L$ and $L’$.

Reviewer comment (Experiments):
The main weakness would be the empirical evaluation. Currently, the results are only for 10 models, and the standard deviation of the accuracy is not reported. A lot of the methods discussed are not compared against. For example: Huang et al. (2022), Xie et al. (2023b) just to name a few.

Author response:
We have reported the standard derivation and added more experimental results in Appendix A. We draw 10 sample sets for each graph following previous works [1,2,3]. We haven’t included some methods in our experiments because it can be expected that these methods cannot yield correct results when none of the non-Gaussianity, purity, and two-pure-children assumption holds theoretically, and their codes have not been publicly open-sourced. Also, our main contributions are theoretical rather than empirical.

Reviewer comment (Presentation):
The fact that the experiments had to be deferred to the Appendix is an indication that this paper is too long already.

Author response:
we have moved some subsidiary theoretical results to Appendix and moved the most representative experimental results from Appendix to the main text.

Reviewer comment (Presentation):
It would be useful to readers to have graphical depictions for what causal graphs your algorithm does and does not allow. To further delineate from previous works, it may be useful to have depictions of graphs that previous works allow as well.

Author response:
Thanks for this constructive suggestion. We have already shown $\mathcal{G}_1$ and $\mathcal{G}_2$ in Figure 1 which our algorithms can handle and previous works cannot handle. Furthermore, we have also displayed $\mathcal{G}_3$~$\mathcal{G}_8$ in Figure 3. Specifically, $\mathcal{G}_3$ satisfies the three-pure-children assumption (BPC and FOFC can also yield correct results theoretically), $\mathcal{G}_4$ violates Assumption 2(a), $\mathcal{G}_5$ violates Assumption 2(b), $\mathcal{G}_6$ satisfies the non-Gaussianity, purity, two-pure-children assumptions (GIN can also yield correct results theoretically), $\mathcal{G}_7$ violates Assumption 3(a), and $\mathcal{G}_8$ violates Assumption 3(b).

Reviewer comment (Theory):
Given that the assumptions required are mostly on latent variables, is it possible at all to get an indication of when these assumptions might hold given a dataset?

Author response:
It is well-known that testing assumptions with only observational data is quite difficult, especially when there exist latent variables. In Section 3.4, we only provide an expedient to guide the choice between Algorithm 2 and Algorithm 3, based on the difference between Assumption 2 and Assumption 3 in the requirement for non-Gaussianity. Although this is not a perfect method, it is better than most previous works that directly circumvented the problem of testing assumptions. This is really an important problem and we list it as our future work in Section 7.