Paper: A Versatile Causal Discovery Framework to Allow Causally-Related Hidden Variables (ICLR-2024)

Reviewer comment (Theory):
One minor weakness/limitation of the paper is that its method is confined to linear SCM. If the variables are mixtures of discrete and continuous random variables, or if they follow a nonlinear SCM, are there any opportunities for the proposed method to contribute?

Author response:
Q1: Are there any opportunities for the proposed method to be extended to nonlinear cases?

A1: Thank you for your insightful question. Intuitively speaking, the rank of the covariance matrix serves as a certain kind of "information bottleneck", indicating the presence and dimensionality of mediators or common causes between two sets of variables. From this perspective, the proposed method has the potential to be extended to non-linear scenarios. Such an extension would require a non-linear version of 'rank' information, possibly through the utilization of kernel representations, and we will leave it for future exploration.

Reviewer comment (Presentation):
It’s a minor comment. I think the definition of $\Sigma_{\mathbf{A},\mathbf{B}}$ should be within Theorem 3.

Author response:
Q3: Definition of $\Sigma_{\mathbf{A},\mathbf{B}}$ within Theorem 3 and runtime analysis compared to methods such as FCI and LiNGAM.

A3: We appreciate the reviewer's suggestion on writing and have made the corresponding revisions. Regarding the runtime analysis, it can be found in Appendix B.16. Specifically, the proposed method requires approximately 3 hours to complete all experiments (across three random seeds and three different sample sizes). FCI takes approximately 10 minutes, whereas RCD and a LiNGAM-based method each take around two days. In comparison to FCI, as we explore a much larger hypothesis space aiming to uncover the entire underlying causal structure (while FCI only indicates the existence of latent variables), RLCD is slower than FCI, but considerably faster than LiNGAM-based methods.

Reviewer comment (Theory):
If $\mathbf{A},\mathbf{B}$ are not t-separable, what does Theorem 3 imply?

Author response:
Q2: If $\mathbf{A},\mathbf{B}$  are not t-separable, what does Theorem 3 imply?

A2: By the definition of t-separation, a pair of sets of variables $\mathbf{A},\mathbf{B}$ are always t-separable: at least $\mathbf{A},\mathbf{B}$ can be t-separated by $(\mathbf{A},\emptyset)$ or $(\emptyset,\mathbf{B})$. Combining this with Theorem 3, we will arrive at $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})\leq \min(|\mathbf{A}|,|\mathbf{B}|)$ which is consistent with the rank of a general matrix. We thank the reviewer for pointing out the confusion and will add this discussion to the revision.

Reviewer comment (Experiments):
I am curious about the benefits of the proposed algorithm in terms of running speed and time complexity compared to other existing algorithms, such as FCI or LiNGAM.

Author response:
Q3: Definition of $\Sigma_{\mathbf{A},\mathbf{B}}$ within Theorem 3 and runtime analysis compared to methods such as FCI and LiNGAM.

A3: We appreciate the reviewer's suggestion on writing and have made the corresponding revisions. Regarding the runtime analysis, it can be found in Appendix B.16. Specifically, the proposed method requires approximately 3 hours to complete all experiments (across three random seeds and three different sample sizes). FCI takes approximately 10 minutes, whereas RCD and a LiNGAM-based method each take around two days. In comparison to FCI, as we explore a much larger hypothesis space aiming to uncover the entire underlying causal structure (while FCI only indicates the existence of latent variables), RLCD is slower than FCI, but considerably faster than LiNGAM-based methods.

Reviewer comment (Reproducibility):
The paper is rigorously written for the most part, but some parts are missing important details. For example, the steps in Algorithm 1 after Phases 1-3 are not described in detail, specifically how the cluster information is aggregated after the first three phases. It is also not very clear how the additional information is added into the CI skeleton.

Author response:
A1: We thank the reviewer for the valuable suggestion. Accordingly, we have added a more detailed description of these steps in Appendix D.5.

Reviewer comment (Evaluation):
A bigger focus on the evaluation in the main paper would be warranted, since the experimental section seems a bit sparse. It is not quite clear why and where RCLD performs so much better than the competitors, just by looking at the F1 score. It would have been helpful to see the output of these different algorithms on a running example.

Author response:
A2: We appreciate your helpful suggestion. We have added illustrative examples to Appendix D.4 for comparing the outputs of various methods, together with a discussion about why the proposed method performs better. Please see the resubmission for details, and below is a summary:  
RLCD correctly recovers the structure asymptotically, while FCI cannot recover edges involving latent variables, and Hier. rank and GIN cannot recover edges between two observed variables and the edges from observed variables to latent variables.

Reviewer comment (Presentation):
Some references are duplicated (Judea Pearl - *Probabilistic reasoning in intelligent systems*, Shohei Shimizu et al. - *A linear non-gaussian acyclic model for causal discovery*).

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 1, first paragraph in introduction: "ICA-based techniques ... that further **leverage**".

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 2: repetition in "our main contributions are mainly three-fold".

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 3: instead of the unusual construction ", - we basically", a semicolon should be employed, or simply start a new sentence.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 3, Section 3.2: A sentence should not start with "E.g", but instead say 'For instance,'.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Presentation):
Page 5, Algorithm 1 is introduced too early in the paper. It should be moved to Section 4.2, where it is first explained.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 5, before Theorem 7: "uesful" is a typo.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Presentation):
Page 6, Figure 3 is introduced too early in the paper. It should be moved to Section 4.2, where it is first used to explain Algorithm 1.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 6, Figure 3(c) caption: "**Take** variables from...".

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 6, Condition 1: "triangle structure" should be defined.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 7, Section 4.2, second paragraph: "Condition (i)-(iv)" should be **conditions (i)-(iv)**.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 7, last paragraph before Section 4.4: "We further determine the **neighbour** set...".

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 7, last paragraph: "we **increase** *k* by 1".

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Presentation):
Page 8, Table 2 appears too early in the paper, since the experiment section is on the next page.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
Page 9, Section 7: "causal discovery approach that allows causally-related variables" seems to be an unfinished thought.

Author response:
A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

Reviewer comment (Writing):
The meaning of "latent ones" is unclear. If it refers to latent variables, it would be better to say latent variables.

Author response:
A6: Yes. We thank the reviewer for pointing out this potential confusion and we have revised accordingly.

Reviewer comment (Writing):
In Theorem 7, the distinction between $\mathbf{X}$ and $\mathbf{X}_\mathcal{G}$ is unclear.

Author response:
A5: As in Definition 1, $\mathbf{X_{\mathcal{G}}}$ refers to the set of all observed variables in the underlying graph $\mathcal{G}$, while $\mathbf{X}$ is a set of observed variables such that $\mathbf{X}\subseteq \mathbf{X_{\mathcal{G}}}$.

Reviewer comment (Experiments):
The importance of having a unified causal discovery framework, in the sense that rank constraints are used for finding the CI skeleton in Phase 1 of the procedure, is questioned. The reviewer wonders if it is more important to find an accurate CI skeleton, whether rank constraints provide more accurate *d*-separation statements than conditional independence tests, and whether mixing different types of tests could be better.

Author response:
A7: We appreciate your insightful question. Theoretically, using rank constraints in specific ways following Lemma 10 and using CI tests would arrive at the same CI skeleton (for linear causal models). Also, empirically, we found that using CI tests and rank tests for Phase 1 would arrive at almost the same results (in terms of precision and recall of edges). We further looked into the type-2 errors of these two tests and found that the type-2 errors are also almost the same. For instance, in 1000 random trials each time using 1000 random samples, where H0 assumes independence and it is not true, when alpha=0.005 both the rank test and CI test have 70 type-2 errors. Given the substantial similarity in outcomes, we opted for the use of rank constraints to enhance the overall unity and coherence of the framework.

Reviewer comment (Evaluation):
The zero F1 score of FCI for all variables in Table 2 is questioned. The reviewer asks if this means that FCI did not get any edge right at all, even between observed variables, as Table 3 also suggests.

Author response:
A4: An F1 score of zero indicates that none of the edges are correctly identified. While, in theory, FCI can identify edges between observed variables up to a certain type of equivalence class, our experimental results with a finite sample size reveal that it does not perform well, as you noticed. There might be two main reasons: (i) the graphs we tested have relatively few direct edges between observed variables, and (ii) FCI relies on conditional independence (CI) tests that require conditioning on multiple variables. However, statistical tests for independence conditioned on a large number of variables have been reported to have low power [1], especially with relatively small sample sizes.  

[1] Spirtes, Peter. "An anytime algorithm for causal inference." International Workshop on Artificial Intelligence and Statistics. PMLR, 2001.

Reviewer comment (Writing):
It is unclear what happens after Phase 3 in Algorithm 1. How is the information from $\mathcal{G''}$ transferred to the Markov equivalence class? How are the rest of the orientations performed? Also, the fact that $\mathcal{G'}$ is supposed to be the skeleton on the observed variables (output of Algorithm 2 from PC), yet toward the end it becomes the MEC over both observed and latent variables (output of Algorithm 1), causes confusion.

Author response:
A1: We thank the reviewer for the valuable suggestion. Accordingly, we have added a more detailed description of these steps in Appendix D.5.

Reviewer comment (Theory):
The idea of learning part of the latent structure explicitly has important ramifications. The reviewer asks what extra information can be ascertained relative to methods like FCI, for which the latent structure is implicit. They also ask whether the difference lies solely in the fact that some latent variables can be identified, and whether RLCD will always provide more structural information than FCI or other CI-based causal discovery algorithms.

Author response:
A8: We thank the reviewer for the insightful question. It does not seem straightforward for extra information to be ascertained relative to methods like FCI, because CI relations among the observed variables are just a particular feature of the join distribution. On the other hand, rank constraints, if used in the way advocated in the paper, naturally capture more information about the distribution relevant to the underlying causal structure. Specifically, based on Lemma 10, we can see that all d-separations can be captured by rank constraints as special cases, that is, rank constraints are strictly more informative than CI information regarding the underlying structure.  

As for the comparison between algorithms, CI-based methods such as FCI can only infer edges between observed variables up to a partial ancestral graph, which may contain many uncertainties.  In contrast, the proposed RLCD offers more comprehensive information of the underlying structure: RLCD can identify all latent variables given the assumed conditions; at the same time, it can discover all causal edges among all observed or latent variables, without introducing extra edges (please note that some of the edges with double circles by FCI may not be causal edges, but just represent the existence of possible latent confounders).

Reviewer comment (Theory):
Corollary 1 says it degenerates to PC when there is no latent variable. I suppose that is under the assumption that the causal model is linear as assumed throughout in this paper. In other words, if the truth is not linear, the PC is asymptotically correct but the proposed algorithm may not -- is it right?

Author response:
Yes, you are totally correct, and we have made this statement explicit in the updated paper. 

Let us add that while we assume linear causal models throughout this paper, the framework has the potential to handle nonlinear relations. Intuitively, the rank of the covariance matrix serves as a certain kind of "information bottleneck", indicating the presence and dimensionality of mediators or common causes between two sets of variables. From this perspective, the proposed method has the potential to be extended to non-linear scenarios. This extension might involve incorporating a non-linear version of 'rank' information, possibly through the utilization of kernel representations. We thank the reviewer for the insightful comment and we will leave it for future exploration.

Reviewer comment (Evaluation):
SID and SHD are more natural metrics for graphical evaluations.

Author response:
We thank the reviewer for the valuable comment. We have added new experimental results using SHD  in Appendix D.1, and we find that the proposed RLCD continues to outperform all baselines across all tested settings. For example, given 10K samples generated from latent measurement models, the average SHD of RLCD is 2.9, while the runner-up is 7.4. As for SID we are still running experiments and will include the results once they are available.

Reviewer comment (Theory):
Assumptions are not justified. This might be for the field in general.

Author response:
We appreciate your insightful comment. In practical applications, certain strategies can be employed to gain a preliminary understanding of whether the assumptions hold. One effective approach is to utilize domain knowledge of the data. For instance, if we know in advance that we have enough measurements of a latent variable that serves as pure children, then the atomic cover condition is satisfied. Nevertheless, we agree that a rigorous verification of graphical assumptions before the procedure serves as a  general challenge to the field and will be a focus for future research. At the same time, as illustrated below, 
in some cases one can verify the conditions after the procedure.

Instead, traditional wisdom often focuses on another critical question: what implications arise if conditions are violated? We thank the reviewer for the valuable question and have provided a detailed discussion in Appendix D.2 with illustrative examples, which is summarized as follows.
If a latent variable does not have enough pure children and neighbors, then Condition 1 is not satisfied. In this case, the result is still consistent with ground truth in the sense that they are rank-equivalent to each other, and at times, it can still provide valuable insights into the underlying structure. Moreover, if Condition 2 is not satisfied, then we will arrive at a self-contradictory situation, and thus we can infer that the condition is violated in hindsight.

Reviewer comment (Novelty):
Relation to previous work does not contain enough information, making it hard to judge the exact contribution of this work. Some detail is given in the introduction.

Author response:
A detailed discussion about how is our condition weaker than previous methods can be found in Appendix B.1 with illustrations in Figure 11.  

Compared to the most related work Hier. Rank (Huang et al. 2022), our graphical conditions are not only strictly weaker but also significantly weaker. Specifically, our conditions are strictly weaker in the sense that Hier. Rank can be taken as a restrictive, special case of the proposed method by prohibiting direct edges between observed variables, which is formally stated by our Corollary 1, and prohibiting direct causal influences from measured variables to latent ones. This is illustrated in Figure 11 (b) compared to Figure 11 (d).

In practice, some measured variables may directly influence each other and they (e.g., environmental factors) may cause latent causal variables (which can cause other latent or measured variables). So our conditions are much weaker and hopefully open the gate to a much wider range of applications.  

Intuitively, our ability to identify latent structures that Hier. Rank cannot lies in the more flexible and comprehensive use of rank constraints. Specifically, Hier. Rank utilizes only a portion of the rank information, denoted as $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$, where $\mathbf{A}\cap\mathbf{B}=\emptyset.$ In contrast, our proposed method utilizes $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$ with arbitrary $\mathbf{A}$ and $\mathbf{B}$, which may share some variables in common, enabling the inference of more types of t-separations. This additional graphical information empowers us, e.g., to leverage Lemma 10 to identify edges between observed variables and Thm 8 to identify partially hidden and partially observed atomic covers.

We thank the reviewer for the insightful question and have added the above discussion to the paper, at the end of Related Work.

Reviewer comment (Novelty):
Related work has been moved to the Appendix but there is not enough information about what the differences to similar works are. It would be useful if the most related works (e.g. Huang et al. 2022) are described in more detail.

Author response:
A detailed discussion about how is our condition weaker than previous methods can be found in Appendix B.1 with illustrations in Figure 11.  

Compared to the most related work Hier. Rank (Huang et al. 2022), our graphical conditions are not only strictly weaker but also significantly weaker. Specifically, our conditions are strictly weaker in the sense that Hier. Rank can be taken as a restrictive, special case of the proposed method by prohibiting direct edges between observed variables, which is formally stated by our Corollary 1, and prohibiting direct causal influences from measured variables to latent ones. This is illustrated in Figure 11 (b) compared to Figure 11 (d).

In practice, some measured variables may directly influence each other and they (e.g., environmental factors) may cause latent causal variables (which can cause other latent or measured variables). So our conditions are much weaker and hopefully open the gate to a much wider range of applications.  

Intuitively, our ability to identify latent structures that Hier. Rank cannot lies in the more flexible and comprehensive use of rank constraints. Specifically, Hier. Rank utilizes only a portion of the rank information, denoted as $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$, where $\mathbf{A}\cap\mathbf{B}=\emptyset.$ In contrast, our proposed method utilizes $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$ with arbitrary $\mathbf{A}$ and $\mathbf{B}$, which may share some variables in common, enabling the inference of more types of t-separations. This additional graphical information empowers us, e.g., to leverage Lemma 10 to identify edges between observed variables and Thm 8 to identify partially hidden and partially observed atomic covers.

We thank the reviewer for the insightful question and have added the above discussion to the paper, at the end of Related Work.

Reviewer comment (Writing):
Hier. rank, which uses rank to discover hierarchical structures, is not described in sufficient detail. This leaves the question in the reader's mind: what specifically allows for the identification of children of latents and mediator latents etc., as opposed to this work.

Author response:
A detailed discussion about how is our condition weaker than previous methods can be found in Appendix B.1 with illustrations in Figure 11.  

Compared to the most related work Hier. Rank (Huang et al. 2022), our graphical conditions are not only strictly weaker but also significantly weaker. Specifically, our conditions are strictly weaker in the sense that Hier. Rank can be taken as a restrictive, special case of the proposed method by prohibiting direct edges between observed variables, which is formally stated by our Corollary 1, and prohibiting direct causal influences from measured variables to latent ones. This is illustrated in Figure 11 (b) compared to Figure 11 (d).

In practice, some measured variables may directly influence each other and they (e.g., environmental factors) may cause latent causal variables (which can cause other latent or measured variables). So our conditions are much weaker and hopefully open the gate to a much wider range of applications.  

Intuitively, our ability to identify latent structures that Hier. Rank cannot lies in the more flexible and comprehensive use of rank constraints. Specifically, Hier. Rank utilizes only a portion of the rank information, denoted as $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$, where $\mathbf{A}\cap\mathbf{B}=\emptyset.$ In contrast, our proposed method utilizes $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$ with arbitrary $\mathbf{A}$ and $\mathbf{B}$, which may share some variables in common, enabling the inference of more types of t-separations. This additional graphical information empowers us, e.g., to leverage Lemma 10 to identify edges between observed variables and Thm 8 to identify partially hidden and partially observed atomic covers.

We thank the reviewer for the insightful question and have added the above discussion to the paper, at the end of Related Work.

Reviewer comment (Theory):
The assumptions in condition 1 are on unmeasured latent variables; how would you verify this before carrying out the graph search procedure?

Author response:
We appreciate your insightful comment. In practical applications, certain strategies can be employed to gain a preliminary understanding of whether the assumptions hold. One effective approach is to utilize domain knowledge of the data. For instance, if we know in advance that we have enough measurements of a latent variable that serves as pure children, then the atomic cover condition is satisfied. Nevertheless, we agree that a rigorous verification of graphical assumptions before the procedure serves as a  general challenge to the field and will be a focus for future research. At the same time, as illustrated below, 
in some cases one can verify the conditions after the procedure.

Instead, traditional wisdom often focuses on another critical question: what implications arise if conditions are violated? We thank the reviewer for the valuable question and have provided a detailed discussion in Appendix D.2 with illustrative examples, which is summarized as follows.
If a latent variable does not have enough pure children and neighbors, then Condition 1 is not satisfied. In this case, the result is still consistent with ground truth in the sense that they are rank-equivalent to each other, and at times, it can still provide valuable insights into the underlying structure. Moreover, if Condition 2 is not satisfied, then we will arrive at a self-contradictory situation, and thus we can infer that the condition is violated in hindsight.

Reviewer comment (Theory):
How can you verify assumptions in condition 2? This seems particularly strong to me.

Author response:
We appreciate your insightful comment. In practical applications, certain strategies can be employed to gain a preliminary understanding of whether the assumptions hold. One effective approach is to utilize domain knowledge of the data. For instance, if we know in advance that we have enough measurements of a latent variable that serves as pure children, then the atomic cover condition is satisfied. Nevertheless, we agree that a rigorous verification of graphical assumptions before the procedure serves as a  general challenge to the field and will be a focus for future research. At the same time, as illustrated below, 
in some cases one can verify the conditions after the procedure.

Instead, traditional wisdom often focuses on another critical question: what implications arise if conditions are violated? We thank the reviewer for the valuable question and have provided a detailed discussion in Appendix D.2 with illustrative examples, which is summarized as follows.
If a latent variable does not have enough pure children and neighbors, then Condition 1 is not satisfied. In this case, the result is still consistent with ground truth in the sense that they are rank-equivalent to each other, and at times, it can still provide valuable insights into the underlying structure. Moreover, if Condition 2 is not satisfied, then we will arrive at a self-contradictory situation, and thus we can infer that the condition is violated in hindsight.

Reviewer comment (Novelty):
Is Lemma 10 a contribution or has this been stated by previous work? If so, a reference is missing.

Author response:
Yes, to be best of our knowledge, both of them are novel. Specifically, Theorem 4 provides a sufficient condition to determine nonadjacency, which generalizes PC’s condition to scenarios where latent variables may exist and Lemma 10 captures how partial correlation tests (as specific conditional independence tests) can be taken as a special case of rank constraints. As illustrated in the paper, they enable the proposed procedure to discover the underlying causal graph under much weaker assumptions.

Reviewer comment (Novelty):
Is Theorem 4 a contribution?

Author response:
Yes, to be best of our knowledge, both of them are novel. Specifically, Theorem 4 provides a sufficient condition to determine nonadjacency, which generalizes PC’s condition to scenarios where latent variables may exist and Lemma 10 captures how partial correlation tests (as specific conditional independence tests) can be taken as a special case of rank constraints. As illustrated in the paper, they enable the proposed procedure to discover the underlying causal graph under much weaker assumptions.