Latent Hierarchical Causal Structure Discovery with Rank Constraints
Abstract
Most causal discovery procedures assume that there are no latent confounders in the system, which is often violated in real-world problems.
In this paper, we consider a challenging scenario for causal structure identiﬁcation, where some variables are latent and they form a hierarchical graph structure to generate the measured variables; the children of latent variables may still be latent and only leaf nodes are measured, and moreover, there can be multiple paths between every pair of variables (i.e., it is beyond tree structure).
We propose an estimation procedure that can e  ciently locate latent variables, determine their cardinalities, and identify the latent hierarchical structure, by leveraging rank deﬁciency constraints over the measured variables.
We show that the proposed algorithm can ﬁnd the correct Markov equivalence class of the whole graph asymptotically under proper restrictions on the graph structure.
Introduction
In many cases, the common assumption in causal discovery algorithms–no latent confounders–may not hold.
For example, in complex systems, it is usually hard to enumerate and measure all taskrelated variables, so there may exist latent variables that inﬂuence multiple measured variables, the ignorance of which may introduce spurious correlations among measured variables.
Much e ↵ ort has been made to handle the confounding problem in causal structure learning.
One research line considers the causal structure over measured variables, including FCI and its variants [Spirtes et al., 2000, Pearl, 2000, Colombo et al., 2012, Akbari et al., 2021], matrix decomposition-based approaches [Chandrasekaran et al., 2011, 2012, Frot et al., 2019] and over-complete ICA-based ones [Hoyer et al., 2008, Salehkaleybar et al., 2020].
Another line focuses on identifying the causal structure among latent variables, including Tetrad condition-based approaches [Silva et al., 2006, Kummerfeld and Ramsey, 2016], high-order momentsbased ones [Shimizu et al., 2009, Cai et al., 2019, Xie et al., 2020, Adams et al., 2021,
Chen et al., 2022], matrix decomposition-based approach Anandkumar et al.
[2013], copula model-based approach [Cui et al., 2018], mixture oracles-based approach [Kivva et al., 2021], and multiple domains-based approach [Zeng et al., 2021].
Moreover, regarding the scenario of latent hierarchical structures, previous work along this line assumes a tree structure [Pearl, 1988, Zhang, 2004, Choi et al., 2011, Drton et al., 2017], where there is one and only one undirected path between every pair of variables.
This assumption is rather restrictive and the structure in real-world problems could be more complex–beyond a tree.
In this paper, we consider a more challenging scenario where latent causal variables form a hierarchical graph structure to generate measured variables—the children of latent variables may still be latent
L
L 2 L
L 4 L
L 6 L 7 L
L 9 L 10 X 12 X 13 X 14 X 15 X
X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 X 10 X(a)
X
X
X X
L
L L
L L
X X
L
X 0 X
L
X X
X X
X
[Figure]
b)
[Figure]
Parents
[Figure]
and only the leaf nodes are measured, and moreover, there can be multiple paths between every pair of variables (see the example hierarchical graphs in Figure 1).
We aim to ﬁnd out identiﬁability conditions of the hierarchical structure that are as mild as possible, and meanwhile, develop an e  cient algorithm with theoretical guarantees to answer the following questions.
(1) How can we locate latent parents for both measured and latent variables, as well as determining the cardinality of the latent parents, by only providing the leaf nodes?
(2) How can we identify the causal relationships among latent variables and those from latent variables to measured variables?
Interestingly, we can answer these questions by properly making use of rank deﬁciency constraints; ﬁnding and leveraging rank properties in speciﬁc ways enable us to identify the Markov equivalence class of the whole graph, under appropriate conditions.
Our contribution is mainly two-fold:
• We propose a structure identiﬁcation algorithm that can e  ciently locate latent variables (including their cardinalities) and identify the latent hierarchical structure, by leveraging the rank deﬁciency.
• We show that the proposed algorithm can ﬁnd the correct graph asymptotically under mild restrictions of the graph structure.
Roughly speaking, we show that it is su  cient to have k +pure children (which can be latent), as well as another k + 1 neighbors
, to identify the latent variable set with size k (see the detailed conditions in Deﬁnition 4 and Condition 1).
It is worth mentioning that rank constraints have been used in previous methods [Silva et al., 2006,
Kummerfeld and Ramsey, 2016], but they assume that each latent variable has three measured ones as children and each measured variable has only one latent parent.
There are also other methods for latent structure learning; for instance, Anandkumar et al.
[2013], which uses matrix decomposition, needs 3 k measured children, and the GIN-based method [Xie et al., 2020], which makes use of high-order statistics, needs 2 k measured children.
However, all those developments require that every latent variable should have measured variables as children.
Very recently, Xie et al.
[2022] proposes an approach for latent hierarchical structure by leveraging the GIN condition under linear non-Gaussian models, but it assumes that each variable has only one parent, where both ﬁgures in
Figure 1 do not satisfy.
This paper is organized as follows.
In Section 2, we give formal deﬁnitions of the latent hierarchical causal model under investigation and give conditions that are essential to the identiﬁability of the graph structure.
In Section 3, we propose an e  cient algorithm that makes use of rank-deﬁciency constraints to identify the latent hierarchical structure.
Moreover, we show theoretically in Section 4 that the proposed algorithm outputs the correct Markov equivalence class of the whole graph asymptotically.
In Section 5, we empirically validate the proposed approach on synthetic data.
Notations for graphical representations that are used in the paper are provided in Table 1.
Latent Hierarchical Causal Model
In this paper, we focus on latent hierarchical causal model with graph structure G , where both measured variables X G =
{ X 1 , · · · , X m } and latent variables L G =
{ L 1 , · · · , L n } are generated by their latent parents in a directed acyclic graph (DAG) with linear relationships:
[Equation]
where b ij and c jk are the causal strength from L j to X i and from L k to L j , respectively, and " X i and " L j are noise terms that are independent of each other.
Without loss of generality, we assume that all variables in X G and L G have zero mean.
Below, we ﬁrst give the general deﬁnition of a linear latent hierarchical graphical model in Deﬁnition 1.
Then we give more detailed conditions on the graph structure (Deﬁnitions 2-4) that are essential to formalize the identiﬁability condition of the latent hierarchical structure.
Deﬁnition 1 (Linear Latent Hierarchical (L 2 H) Model) .
A graphical model, with its graph G =
( V G , E G ) , is a linear latent hierarchical model if:
V G = X G S L G , where X G is the set of measured variables and L G is the set of latent variables, 2.
there is at least one undirected path between every pair of variables, and 3.
each variable in X G and L G are generated by the structural equation models in Eq.
Generally, without further constraints, the causal structure of the L 2 H model is hard to be identiﬁed
It has been shown that if the underlying graph structure satisﬁes a tree [Pearl, 1988], then the structure is identiﬁable.
However, this structural constraint may be too strong to hold in many real-world problems.
In this paper, we give su  cient conditions that are much milder than previous ones, as well as an e  cient search algorithm, for the identiﬁability of the causal structure.
We now give the corresponding deﬁnitions, including pure children , pure descendants , and e ↵ ective cardinality , that will be used in the identiﬁability condition, together with illustrative examples.
Deﬁnition 2 (Pure Children) .
Variables V are pure children of a set of latent variables L in a graph
G , if Pa G ( V ) =
L and V \ L = ; .
That is, V have no other parents than L .
We denote the pure children of L by PCh G ( L ) .
Accordingly, Pure Descendants of a set of latent variables L are deﬁned as all recursive pure children of L (including PCh G ( L ) , PCh G ( PCh G ( L )), etc.), denoted by PDe G ( L ).
Furthermore, measured variables that are pure descendent of L are called Measured Pure Descendants , denoted by M G ( L ).
Example 1.
In Figure 1(a), the pure children of { L 2 , L 3 } are { L 6 , · · · , L 10 } , its pure descendants are { L 6 , · · · , L 10 , X 1 , · · · , X 11 } , and its measured pure descendants are { X 1 , · · · , X 11 } .
Deﬁnition 3 (E ↵ ective Cardinality) .
For a set of latent variables L , denote by C the largest subset of
PCh G ( L ) such that there is no subset C 0 ✓ C satisfying | C 0 | > | Pa G ( C 0 ) | and | Pa G ( C 0 )
| < | L | .
Then, the e ↵ ective cardinality of L ’s pure children is | C | .
In the case when L and its pure children are fully connected, the e ↵ ective cardinality is just the cardinality of the pure children of L .
However, for Figure 1(a), the e ↵ ective cardinality of the pure children of { L 7 , L 8 } is 3, because here the largest subset that satisﬁes the condition is { X 5 , X 6 , X 7 } .
We further deﬁne latent atomic cover that constrains the number of pure children and neighbours for latent variables, which are essential for structural identiﬁability.
Deﬁnition 4 (Latent Atomic Cover) .
Let L =
{ L 1 , ..., L k } be a set of latent variables in graph G , with | L | =
k. We say that L is a latent atomic cover if the following conditions are met:
there exists a subset of pure children C 0 ✓ PCh G ( L ) with e ↵ ective cardinality  k + 1 ; 2.
there exists a neighbour set B to L s.t.
B
[Equation]
there does not exist a partition of L =
L 1 [ L 2 , so that both L 1 , L 2 satisfy conditions 1 and 2 and { PCh G ( L 1 ) [ PCh G ( L 2 ) }\ L = PCh G ( L ) .
Example 2.
In Figure 1(a), L = { L 2 , L 3 } is a latent atomic cover with k = 2 , because (1) there exists a subset of pure children C {+
+}
= { L 6 , L 7 , L 8 } with e ↵ ective cardinality 3 = k + 1 , (2) there exists a neighbor set B =
{ L 1 , L 9 , L 10 } , s.t. B \ C {+
+}
= ; and | B | = 3 = k + 1 , and (3) neither { L 2 } or { L 3 } satisﬁes the above two conditions.
We now give the conditions for structural identiﬁability from measured variables X G alone, including those on the structural constraints (Condition 1) and rank faithfulness assumption (Condition 2).
Condition 1 (Irreducible Linear Latent Hierarchical (IL 2 H) Graph) .
An L 2 H graph G is an IL 2 H graph if
every latent variable L 2 L in G belongs to at least one latent atomic cover, 2.
for any pair of latent atomic covers ( L A , L B ) , if PDe G ( L A ) T PDe G ( L B ) , ; , then either (a) L A ⇢ L B or (b) L A ⇢ PDe G ( L B ) or (c) L B ⇢ L A or (d) L B ⇢ PDe G ( L A ) , and 3.
for any three latent atomic covers L A , L B , L C , if the causal structure satisﬁes L A ! L B ! L C or L A L B ! L C , and | L B
| =
k, then L B has 2 k neighbors, except for L A , L C and the parents in the v structure where L B is a collider.
Next, we give the faithfulness assumption, which holds for generic covariance matrices consistent with G [Spirtes, 2013].
Condition 2 (Rank Faithfulness) .
A probability distribution P is rank faithful to a DAG G if every rank constraint on a sub-covariance matrix that holds in P is entailed by every linear structural model with respect to G .
We will show in Section 4 that if the underlying latent hierarchical graph satisﬁes an IL 2 H graph and the rank faithfulness holds, then the location and cardinality of latent variables, and the causal structure among latent atomic covers and that from latent atomic covers to measured variables, are identiﬁable with appropriate search procedures.
Below, let us ﬁrst present the identiﬁcation procedure.
Structure Identiﬁcation with Rank-Deﬁciency Constraints
We propose an identiﬁcation algorithm (Algorithm 1) to identify the structure of IL 2 H graphs, by leveraging rank-deﬁciency constraints of measured variables.
In particular, the algorithm includes three phases: ﬁnding causal clusters and assigning latent atomic covers in a greedy manner (“ ﬁndCausalClusters "), reﬁning incorrect clusters and covers due to the greedy search (“ reﬁneClusters "), and reﬁning edges and ﬁnding v structures (“ reﬁneEdges ").
Input : Date from a set of measured variables
X G Output: Markov equivalence class G 0 1 G 0 =
ﬁndCausalClusters ( X G ) ; // find clusters and assign latent covers greedily 2 G 0 =
reﬁneClusters ( G 0 ) ; // refine incorrect clusters and covers from greedy search 3 G 0 = reﬁneEdges ( G 0 ) ; // refine some edges and find v structures
Before describing the identiﬁcation algorithm, we ﬁrst give a theorem that relates the graphical structure of an IL 2 H graph to the rank constraints over the covariance matrix of measured variables.
Theorem 1 (Graphical Implication of Rank Constraints in IL 2 H Graphs) .
Suppose G satisﬁes an IL 2 H graph.
Under the rank faithfulness assumption, the cross-covariance matrix ⌃ X A , X B over measured variables X A and X B in G (with | X A | , | X B | > r) has rank r, if and only if there exists a subset of latent variables L with | L | = r such that L d-separates X A from X B , and there is no L 0 with | L 0 | < | L | that d-separates X A from X B .
That is,
[Equation]
For instance, in Figure 1(a), rank ( ⌃ { X 1 , X 2 } , { X 3 , X 4 } )
= 1, because L 6 d-separates { X 1 , X 2 } from { X 3 , X 4 } with | L 6 | = 1.
Rank Test.
We test rank deﬁciency by leveraging canonical correlations [Anderson, 1984].
Specifically, the number of non-zero canonical correlations between two random vectors is equal to the rank.
Denote by ↵ i the i -th canonical correlation coe  cient between X A and X B .
Then under the null hypothesis that rank ( ⌃ X A , X B )  r , the statistics  ( N  ( p + q + 3) / 2) [Equation] log( [Equation]
↵
[Equation] ) is approximately  2 distributed with ( p  r )( q  r ) degrees of freedom, where p = |
X A | , q = | X B | , and N is the sample size.
We further show that for any subset of latent variables in an IL 2 H graph, we can use the measured variables as surrogates to estimate the rank, as indicated in the following theorem.
Theorem 2 (Measurement as a surrogate) .
Suppose G is an IL 2 H graph.
Denote by A , B ✓ V G two subsets of variables in G , with A \ B = ; .
Furthermore, denote by X A the set of measured variables that are d-separated by A from all other measures, and by X B the set of measured variables that are d-separated by B from all other measures.
Then rank ( ⌃ A , B ) = rank ( ⌃ X A , X B ) .
A special case when Theorem 2 holds is that X A and X B are the measured pure descendants of A and B , respectively.
Note that Theorem 1 is a special case of Theorem 2.8 in Sullivant et al.
[2010] when applied to IL 2 H graphs.
Di ↵ erent from the setting in Sullivant et al.
[2010] where access to the full covariance matrix ⌃ V G , V G is assumed, we only have access to the covariance matrix ⌃ X G , X
G over the measured variables X G , which we will use to infer the causal structure over the entire graph G .
For this reason, although we can infer the number of latent variables that d-separate any two sets of measured variables X A , X B , we cannot directly know the exact location of these variables in the graph.
Fortunately, the structure constraints of the IL 2 H graph and Theorem 2 will allow us to reconstruct the graph with certain search procedures, as shown below.
3.1 Phase I: Finding Causal Clusters
We start to discover clusters in a recursive and greedy manner, by performing rank-deﬁciency tests over measured variables.
We denote by S a set of active variables that is under investigation; S is set to X G initially and will be updated to include latent atomic covers when rank deﬁciency is discovered.
Below, we ﬁrst give the deﬁnition about set size and the deﬁnition of atomic rank-deﬁciency set , which will be used in Rule 1 and Algorithm 2.
Deﬁnition 5 (Set Size) .
Suppose S is a set.
We denote by |S| the cardinality of S ; that is, |S| is the number of elements S i in S , for S i 2 S .
We denote by k S k the number of variables in S , where k S k
= | S i S i | , for S i 2 S .
For example, suppose S =
{ L 1 } , { L 6 } , { L 7 , L 8 } , { L 9 , L 10 } .
Then |S| = 4 and k S k = 6.
Deﬁnition 6 (Atomic Rank-Deﬁciency Set) .
Given a graph G 0 .
Denote by S ✓ V G 0 an active set of variables that is under investigation.
Let A ⇢ S and B = S\ A .
Denote by X A = M G 0 ( A ) and X
B = M G 0 ( B ) the measured pure descendants of A and B , respectively.
If (1) ⌃ A , B is rank deﬁcient, i.e., rank ( ⌃ A , B ) < min { k A k , k B k } ), and (2) no proper subset of A is rank deﬁcient, then A is called an atomic rank-deﬁcient set, and ⌃ A , B can be estimated by ⌃ X A , X B .
For example, in Figure 1(a), suppose now we have the measured variables X G under investigation, so the active variable set S =
X G .
A =
{ X 1 , X 2 } ⇢ S forms an atomic rank-deﬁciency set because the cross-covariance matrix of X A against all other measures has rank 1 (i.e., rank deﬁcient).
According to Theorem 1, the rank deﬁciency occurs because the latent parent L 6 of { X 1 , X 2 } d-separates them from all other measures.
This naturally leads to the following rule that assigns a latent atomic cover over the rank-deﬁciency set:
Rule 1: If A is a rank-deﬁciency set with rank ( ⌃ X A , X B ) =
k , then assign a latent atomic cover L of size k as the parent of every variable A i 2 A .
Later in Phase II, we will show that Rule 1 may not correctly identify the latent atomic cover in some cases, and, accordingly, we will further provide an e  cient revision procedure in Phase
II.
At the current phase, we use Rule 1, together with certain search procedures, to identify the clusters and latent atomic covers.
The detailed search procedure of Phase I is given in Algorithm 2 ( ﬁndCausalClusters ), which tests for rank deﬁciency recursively to discover clusters of variables and their latent atomic clusters.
The set of active variables S is set to X G initially (line 1) and will be updated as the search goes on (line 14).
We start to identify the latent atomic cover with size k =(line 1).
We consider any subset of the latent atomic covers in S and replace them with their pure children, resulting in ˜ S (line 4).
Then we draw a subset of variables A ⇢ ˜ S with cardinality at least k + [-
-]1 and conduct a rank deﬁciency test of A against S\ A by estimating the rank of rank ( ⌃ X A , X B ) (lines 6-7).
Note that here X A and X B are the measured pure descendants of A and B , respectively, in the currently learned graph.
This step is repeated until all subsets are tested (lines 5-9).
If rank deﬁciency is found, we merge the overlapping groups into a cluster and add latent covers over the cluster (lines 10-14).
We further reset k = 1 and resume the search (line 15).
Otherwise, if no latent cover is found, we increment k
= k + 1 (line 17).
This procedure is repeated until no more clusters are found.
Finally, we connect the elements in S into a chain structure (line 19).
Figure 2 illustrates an example procedure of ﬁnding new clusters by applying ﬁndCausalClusters to the measured variables
X G generated from the structure in Figure 1(a).
3.2 Phase II: Reﬁning Clusters
As we have mentioned above, the naive assignment of causal clusters in Algorithm 2 ( ﬁndCausalClusters ) may not be correct in some cases.
In this section, we provide a precise characterization of the cases where ﬁndCausalClusters incorrectly clusters variables and, accordingly, propose an e  cient algorithm to correct such cases.
Take Figure 3(a) as an example to illustrate the issue.
By applying ﬁndCausalClusters to the measured variables X G , at k = 2, we discovered that { X 9 , X 10 , X 11 } form a cluster and then set a 2-latent atomic cover { L 0 1 , L 0 2 } over it (Figure 3(b)).
This is not correct, because X 9 , X 10 , X 11 actually belong to three di ↵ erent 1-latent atomic covers, L 1 , L 2 , L 3 , respectively.
The incorrect clustering and covering are because when discovering the rank-deﬁciency set { X 9 , X 10 , X 11 } , the latent covers { L 4 , L 5 } , { L 6 , L 7 } have not been identiﬁed yet.
If { L 4 , L 5 } had already been discovered, we would correctly ﬁnd that { X 9 , L 4 , L 5 } form a 1-latent atomic cover, and the algorithm would proceed correctly thereafter.
More generally, ﬁndCausalClusters may set incorrect cluster for the rank-deﬁciency set { X 9 , X 10 , X 11 } , in
(d)
[Equation]
{ L 6 } , { L 9 , L 10 } .
L L
L
L(e) Connect { L 2 , L 3 } and { L 7 , L the case when their parents L 1 , L 2 , L 3 split the graph into two or more disconnected graphs.
This result is formally stated in the following deﬁnition and theorem.
Deﬁnition 7 (Bond Set) .
Consider a set of measured variables X ✓ X G , and a minimal set of latent variables L ✓ L G that d-separate X from all other measures X 0 B X G \ X .
We say that X is a bond set if L also d-separates some partition
X A [Equation] from one another.
From the previous example, we have seen that with the existence of bond sets, ﬁndCausalClusters may end up with incorrect clusters and covers.
The following theorem shows that the presence of bond sets is the only reason for incorrect clusters and covers with ﬁndCausalClusters .
Theorem 3 (Correct Cluster Condition) .
Suppose G is an IL 2 H graph with measured variables X G .
Consider the output G 0 from applying ﬁndCausalClusters over X G .
If none of the clusters in G 0 is the bond set in G , then all latent atomic covers have been correctly identiﬁed.
However, without access to the true graph G , we are not able to identify which clusters formed from ﬁndCausalClusters are bond sets.
Fortunately, the following properties of the clusters formed over bond sets provide a way to remove bond sets in our discovered graph even without such knowledge, so that the clusters and latent covers can be correctly identiﬁed.
Theorem 4 (Correcting Clusters) .
Denote by G 0 the output from ﬁndCausalClusters and by G the true graph.
For a latent atomic cover L 0 in G 0 , if the measured pure descendants of L 0 is a bond set in the true graph G , then there exist a set of siblings S of L 0 in G 0 , a set of children C of L 0 , and a set of grandparents P of L 0 , such that M G 0 ( S [ C [ P ) forms a cluster that is not a bond set in G .
The above theorem shows that whenever forming a bond set, the siblings and grandparents of the latent cover are the key to correcting the incorrect clusters and covers.
Speciﬁcally, we will remove such a cover and use its children to form new clusters with the siblings and grandparents (see details in Algorithm 3), and Theorem 4 guarantees that these new clusters will not contain bond sets, and thus, providing correct clusters.
Example 3.
In Figure 3(b), the formed cluster { X 9 , X 10 , X 11 } is the bond set in the true graph in
Figure 3(a), and the covers { L 0 1 , L 0 2 } , { L 0 7 } , and { L 0 8 } are not correct.
Illustration of the algorithm for this example, including reﬁning the incorrect clusters and covers, is given in Appendix A5.7.
[Figure]
Furthermore, with Theorem 4, we give the following rule for correcting the clusters and covers.
Rule 2: For each discovered latent atomic cover L , let V =
Gp G 0 ( L )
S [Equation]
L ) and apply ﬁndCausalClusters to V to reﬁne the clusters.
Before introducing the detailed reﬁning algorithm reﬁneClusters based on Rule 2, we ﬁrst introduce an operator makeRoot that will be used in the algorithm, which will not change the rank deﬁciency constraints (Lemma 5).
Deﬁnition 8 ( makeRoot ) .
Give a graph G 0 and a latent atomic cover L in G 0 .
A makeRoot operator of L , denoted by makeRoot ( L ) , reorients all outgoing edges of L to L , such that L is a root variable.
Lemma 5 (Rank Invariance) .
Denote by G 0 the output from ﬁndCausalClusters and by L a latent atomic cover in G 0 .
Then all rank constraints, that are possible to be executed by ﬁndCausalClusters prescribed by G 0 , before and after the operator makeRoot
( L ) are identical.
Algorithm 3 introduces the procedure of reﬁning clusters based on Rule 2.
Starting with the output
G 0 from ﬁndCausalClusters , we proceed in a breadth-ﬁrst search from the root variable (lines 1-3, 5).
Because the root is trivially not over a bond set, Theorem 4 guarantees that for each child of the root, we will form new clusters that are not over bond sets.
For each cover L , if it has only one child, we will recursively consider the children of its child instead, since if L is over a bond set then its single child will also be.
We add the ﬁrst set of children into the search set V (line 4).
We construct the search set (line 6) and remove the covers we are reﬁning (lines 7-8).
Finally, we make L the root (line 8) and conduct the search to form new clusters (line 9).
The algorithm ends after reﬁning every latent cover in G 0 .
With this reﬁning procedure, we will derive the correct latent covers for the graph in Figure 3(a).
3.3 Phase III: Reﬁning Edges
With ﬁndCausalClusters and reﬁneClusters , the output G 0 correctly identiﬁes the latent variables.
Moreover, G 0 correctly identiﬁes the following d-separation: for every L 2 L G 0 , its parents Pa G 0 ( L ) d-separates L and its descendants from the ancestors of Pa
G 0 ( L ), and thus, there cannot be any edges from each L to any of its ancestors beyond its own parents.
However, G 0 may still have incorrect edges locally .
Speciﬁcally, each time we create an atomic cover L in ﬁndCausalClusters , the implicit assumption is that each of its children is conditionally independent of the other children given the parents L .
However, this assumption is not necessarily true, as (i) the children of L may be directly connected to one another, and (ii) L may only be directly connected to a subset of its children.
In other words, previous steps did not consider condition independence relationships across clusters.
Hence, we need a further step to correct edges over each L and its children.
For example, consider the true graph in Figure 4(a), which ends up with the graph in Figure 4(b) with the ﬁrst two phases.
However, note that the ﬁrst two phases did not consider the d-separation between
L 0 2 and L 0 3 , and thus the edges among L 0 1 , L 0 2 , L 0 3 , L 0 4 may not be correct, including the v structure.
In particular, in the true graph, L 1 d-separates L 2 from L 3 , while { L 1 , L 4 } does not d-separate L 2 from
L 3 , but these d-separations are not reﬂected in the discovered G 0 in the ﬁrst two phases.
Thus, we need to reﬁne the edges.
To this end, we ﬁrst set L 0 1 , L 0 2 , L 0 3 and L 0 2 , L 0 3 , L 0 4 to be fully connected, and consider testing A
= { L 0 2 , X 1 } against B = { L 0 3 , X 2 } , where we partition the children of L 0 1 into two sets and put them in A and B , respectively.
By doing so, we force L 0 1 in G 0 to be in the separating set.
Since rank ( ⌃ A , B ) = 1, it implies that no other variable is in the separating set, and therefore we can conclude that L 0 1 d-separates L 0 2 from L 0 3 .
This principle is characterized in the following lemma.
Lemma 6 (Cross-Cover Test) .
Given a set of variables S , consider two latent atomic covers L A , L B
S , and a potential separating set C =
{ L C i } ✓ S\{ L A , L B } .
For each L C i , consider C Ai , C Bi ✓
PCh ( L C i ) with C Ai , C Bi , ; and C Ai \ C Bi = ; , and denote the cardinality k Ai B min ( | L C i | , | C Ai | ) , k Bi B min ( | L C
i | , | C Bi | ) , respectively.
Then there is no edge between L A and L B if and only if there exists a separating set C such that rank ( ⌃ A , B ) < min ( | L A | +
P i k Ai , | L B | + P i k Bi ) , where A =
{ L A , C A 1 ,
C A 2 , ... } and B = { L B , C B 1 , C B 2 , ... } .
In this case, we say that C satisﬁes the cross-cover test of L A against L B .
Note that in order to ﬁnd rank deﬁciency when performing the cross-cover test, L C i needs to satisfy the third condition in Condition 1.
Based on Lemma 6, we use the following rule to reﬁne the edges.
Rule 3: For a pair of latent covers ( L A , L B ), let A { L A , C A 1 , C A 2 , ... } and B { L B , C B 1 , C B 2 , ... } .
If there exists such
A , B such that rank G 0 ( ⌃ A , B ) is rank deﬁcient, then remove all edges between L
A , L B in G 0 .
Furthermore, with Rule 3, Algorithm 4a ( CrossCoverTest ) in Appendix A gives the procedure of reﬁning the edges over a set of latent variables S to correct the causal skeleton.
Furthermore, we are going to identify the v structures among latent atomic covers.
In the output
G 0 from phase II, for any L 0 with a child C {+
+}
i and parent P 0 , it is not possible to have a collider
C 0 i !
L P 0 in the ground-truth graph, because in this case the cluster would not have been rank deﬁcient.
Therefore, the only v structures in the graph are amongst the variables in L 0 [ PCh G 0 ( L 0 ), and similar to crossCoverTest , we only need to test for v structures locally.
Continue to consider the example in Figure 4.
The edge between L 0 2 and L 0 3 is missing because the rank over { L 0 2 , X 1 } and { L 0 3 , X 2 } is 1, implying that L 0 1 d-separates L 0 2 from L 0 3 , as is done in crossCoverTest .
Now, since L 0 2  L 0 4  L 0 3 forms an unshielded triplet, we want to test if a collider exists at L 0 4 .
We ﬁnd that the rank over A =
{ L 0 2 , L 0 4 , X 1 } and B = { L 0 3 , X 2 } is 2 > 1, and the rank over A
=
{ L 0 2 , X 1 } and B = { L 0 3 , L 0 4 , X 2 } is 2 > 1, so L2 ! L
L 0 3 .
For general cases, the rule for ﬁnding v structures are formulated in the following lemma and Algorithm 4b ( ﬁndColliders ) in Appendix A.
Lemma 7 (V-Structure Test) .
For any unshielded triangle L A  L C  L B , let A , B be the set of variables in Lemma [-
-]6 such that ⌃
A , B was rank deﬁcient.
Let k = rank ( ⌃ A , B ) , k 1 = rank ( ⌃ A [ L C , B ) , and k 2 = rank ( ⌃ A , B [ L C ) .
Then, L A ! L B L C if and only if k < min( k 1 , k 2 ) .
As mentioned above, we only need to perform cross-cover test and v-structure test locally in the estimated graph G 0 .
Algorithm 4 (Phase III: reﬁneEdges ) combines the search procedure of the two components together with the output G 0 from reﬁneClusters as the input.
Speciﬁcally, for each latent cover L 0 in G 0 , we only need to consider testing for edges amongst L 0 and its children C =
Ch G 0 ( L 0 ).
Thus, we perform a depth-ﬁrst traversal of the output graph G 0 starting from Root ( G 0 ) (lines 1-3), and apply the cross-cover test at each L to reﬁne the skeleton of G 0 (lines 4-7, 11) and the collider test to identify the v structures (lines 10, 11).
After determining v-structures, we can ﬁnd more directions by applying Meek’s rule (line 12), analogous to that in the PC algorithm
[Spirtes et al., 2000].
G refineEdges
): 1 foreach latent atomic 0 doif not latent children then return ;foreach latent child C
i 0 do refineEdges
;if C B PCh G is a latent then ;else S L 0 [ GmakeRoot G 0 ( and remove all edges amongst S in G 00 ; // temp graph
Edgeset E crossCoverTest
( S
G 00 );if no conditional independencies found then return G
;else
Collider set C findColliders ( S , E
G 00 );in G 0 , remove all edges amongst S and use E , C to connect variables in S ;apply Meek’s rule to G 0 ; 13 convert G 0 to its Markov equivalence class; 14 return
G
L
L
L L X 1 X [Table]
X
[Table]
X
X
[Table]
X
[Table]
X
[Table]
[Table]
[Table]
Theoretical Results
In this section, we show the correctness of the algorithms proposed in Section 3.
In particular, by making use of the rank constraints of only the measured variables, the proposed algorithms output the correct Markov equivalence class of the IL 2 H graph asymptotically, under the minimal-graph operator and skeleton operator , with their deﬁnitions given below.
Deﬁnition 9 (Markov Equivalence Class of IL 2 H graphs) .
Two IL 2 H graphs G 1 and G 2 are in the same Markov equivalence class, denoted by G [-
 ≈-]1 ⇡ G 2 , if and only (1) they have the same set of variables (both measured and latent variables), (2) have the same causal skeleton, and (3) have the same V-structures L i !
L k L j , where L i , L j , L k represent latent atomic covers.
Deﬁnition 10 (Minimal-Graph Operator) .
Suppose G is an IL 2 H graph.
For every latent atomic cover L in G , merge L to its parents P if the following conditions hold: (1) L is the pure children of
P , (2) | L | = | P | , and (3) the pure children of L form one latent atomic cover, or the siblings of L form one latent atomic cover.
We call such operator the minimal-graph operator and denote it by O min ( G ) .
Deﬁnition 11 (Skeleton Operator) .
Suppose G is an IL 2 H graph.
A skeleton operator of G , denoted by
O s ( G ) is deﬁned as follows: for any latent atomic cover L , draw an edge from l j 2 L to c k 2 PCh G ( L i ) , if l j and c k are not directly connected in G .
Note that the minimal-graph operator and the skeleton operator will not change the rank constraints, or in other words, graphs before and after applying the operators are indistinguishable with rank constraints.
This result is shown in the following lemma.
Lemma 8.
Suppose G is an IL 2 H graph.
The rank constraints are invariant with the minimal-graph operator and the skeleton operator; that is, G and O skeleton
( O min ( G )) are rank equivalent.
These two operators have already been achieved in Algorithm 1, so the output of the algorithm is the rank-equivalent graph with the minimal number of latent atomic covers.
We next proceed to show that phases I-III will output a graph G 0 such that G 0 will be in the same
Markov equivalence class as O min ( O s ( G )), denoted by G 0 ⇡ G .
We have already shown earlier in
Theorem 3 that ﬁndCausalClusters gives correct latent covers when there is no bond set.
Furthermore,
Theorem 4 shows that even in the presence of bond sets, Phase II reﬁneClusters is able to reﬁne clusters into those without bond sets.
Therefore, Phase I-II can correctly identify the latent atomic covers of O min ( G ), which is given in the following theorem.
Theorem 9 (Identiﬁability of Latent Variables) .
Suppose G is an IL 2 H graph with measured variables
X G .
Phases I-II in Algorithm 1 over X G can asymptotically identify the latent atomic covers of O min ( G ) , with only the ﬁrst two conditions in Condition 1.
Moreover, Phase III reﬁneEdges further guarantees correct skeletons and v structures.
Therefore, the following theorem shows that Algorithm 1, which includes Phases I-III, can asymptotically identify the Markov equivalence class, up to the skeleton operator and the minimal-graph operator.
Theorem 10 (Identiﬁability of Causal Graph) .
Suppose G is an IL 2 H graph with measured variables
X G .
Algorithm 1, including Phases I-III, over X G can asymptotically identify the Markov equivalence class of O min ( O s ( G )) .
Experimental Results
We applied the proposed algorithm to synthetic data to learn the latent hierarchical causal graph.
Speciﬁcally, we considered di ↵ erent types of latent graphs and di ↵ erent sample sizes.
The causal strength was generated uniformly from [  5 ,  0 .
[ [0 . 5 , 5], and the noise term was randomly chosen from a normal distribution with noise variance uniformly sampled from [1 , 5].
To the best of our knowledge, this is the ﬁrst algorithm that can identify such general latent hierarchical structures, so to fairly compare with other methods, besides general IL 2 H graphs, we also considered tree structures and measurement models.
We compared the proposed method with tree-based method–Chow-Liu
Recursive Grouping (CLRG)
[Choi et al., 2011], as well as measurement-model-based methods, including FOFC [Kummerfeld and Ramsey, 2016] and GIN [Xie et al., 2020].
Table 1 gives the estimation results evaluated on three types of latent graphs: IL 2 H graphs, tree structures, and measurement models.
The performance is measured by the percentage of correctly identiﬁed causal clusters over only measured variables.
Our method gives the best results on all types of graphs, indicating that it can handle not only the tree-based and measurement-based structures, but also the latent hierarchical structure.
The CLRG algorithm does not perform well on tree-based structure because the metric is rather strict–even a single misclustered variable outputs an error.
Complete experimental settings and more results, including performance measured by other metrics and the case when the noise terms are uniformly distributed, are given in Appendix B.
[Table]
Conclusions and Future Work
In this paper [Table] we formulated a speciﬁc type of latent hierarchical causal model and proposed a method to identify its graph by making use of rank deﬁciency constraints.
Theoretically, we show that the proposed algorithm can ﬁnd the correct Markov equivalence class of the whole graph asymptotically under mild restrictions of the graph structure.
For more general graphs, only using the second-order statistics may result in a rank equivalence class that contains multiple DAGs, so how to further leverage high-order statistics to distinguish between causal graphs within the equivalence class will be our future work.
Other future research directions include allowing nonlinear causal relationships and allowing measured variables to cause latent variables (existing techniques, e.g., Adams et al.
[2021], Squires et al.
[2022], may help to mitigate this issue).
Acknowledgement
BH would like to acknowledge the support of Apple Scholarship.
Kun Zhang was partially supported by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence
Accelerator Track-D award #2134901, by a grant from Apple Inc., and by a grant from KDDI
Research Inc. FX acknowledges the support by National Natural Science Foundation of China (NSFC 62006051).
A Algorithms on crossCoverTest and ﬁndColliders
Algorithm 4a: ( CrossCoverTest ) gives the procedure of reﬁning the edges over a set of latent variables
S .
It ﬁrst fully connects the latent covers in S (line 2).
Then for every pair of latent covers, it performs the cross-cover test (lines 3-17).
If rank deﬁciency is found, then remove the corresponding edges (lines 10-12).
Algorithm 4b: ( ﬁndColliders ) gives the procedure of ﬁnding v structures.
For every unshielded triangle L 1  L 3  L 2 , it performs v structure tests and compares with the rank which does not involve
L 3 (lines 22-23).
If the rank that involves L 3 is larger, then the unshielded triangle forms a v structure (lines 24-25).
B Complete Experimental Results
We applied the proposed algorithm to synthetic data to learn the latent hierarchical causal graph.
Speciﬁcally, we considered di ↵ erent types of latent graphs and di ↵ erent sample sizes (with N = 2 k , 5 k , 10 k ).
The causal strength was generated uniformly from [  5 ,  0 .
[ [0 . 5 , 5], and the noise term either follows a Gaussian distribution (with noise variance uniformly sampled from [1 , 5]) or a uniform distribution U (  2 , 2).
findColliders ( S
E , G 0 ): 19 Collider C ; ; 20 foreach unshielded triangle L 1  L L 2 in S based on E do let A , B be the set of variables in Algorithm 4a: such that ⌃ A , B was rank deﬁcient with rank k ; krank G 0 ( ⌃ A
[ L 3 , B );krank G 0 ( ⌃ A , B [ L 3 );if k < min( k 1 , k 2 ) then add collider L 1 ! L L 2 to C ; return Collider set C
To the best of our knowledge, this is the ﬁrst algorithm that can identify such general latent hierarchical structures, so to fairly compare with other methods, besides general IL 2 H graphs (see Figure 7), we also considered tree structures (see Figure 5) and measurement models (see Figure 6).
We compared the proposed method with the tree-based method–Chow-Liu Recursive Grouping (CLRG)
[Choi et al., 2011], as well as measurement-model-based methods, including FOFC [Kummerfeld and Ramsey, 2016] and GIN Xie et al.
We used the following metrics to evaluate the performance:
• Causal cluster recovery rate over measured variables (metric 1) : measured by the percentage of correctly identiﬁed causal clusters over measured variables, with m 1 = correctly found # clusters over measured variables total # clusters over measured variables .
• Causal cluster recovery rate over all variables (metric 2) , measured by the percentage of correctly identiﬁed causal clusters over all variables, with m 2 = correctly found # clusters over all variables total # clusters over all variables .
• Percentage di ↵ erences between estimated and true adjacency matrices (metric 3) , with
[Equation]
where Adj denotes the adjacency matrix, i and j denote the i -th and j -th entry, respectively, and n X and n L are the number of measured variables and latent variables, respectively.
Note that the indices of the latent variables in the estimated graph may not be aligned to those in the true graph.
To remove this ambiguity, we tried all permutations of the latent indices in the estimated graph and used the one which has the smallest di ↵ erence from the true graph.
Moreover, if
the estimated
number
of latent variables is smaller than the true number of latent variables add extra latent variables to G
that do not have edges with others.
If the estimated number of latent variables is larger than the true number of latent variables, then ﬁnd a subset of the latent variables in G 0 that best aligns the true ones.
It is worth mentioning that how to measure the performance of the estimated latent hierarchical graph is a nontrivial problem and will be further investigated.
The experimental results were reported in Tables 4 and 5, where the noise terms are Gaussian distributed and uniformly distributed, respectively.
Our method gives the best results on all types of graphs, indicating that it can handle not only the tree-based and measurement-based structures, but also the latent hierarchical structure.
The CLRG algorithm does not perform well on tree-based structure because the ﬁrst two metrics are rather strict–even a single mis-clustered variable outputs an error.
[Table]
Note that X i , j , k means X i , X j , X k .
L
[Table]
C Related Work
Identiﬁcation of causal relationships from observational data, known as causal discovery , is attractive for the reason that traditional randomized control trials may be hard or even impossible to do.
Most state-of-the-art approaches in causal discovery assume that the measured variables are the underlying causal variables and that no latent confounders inﬂuence the measured variables [Spirtes et al., 2000,
Chickering, 2002, Shimizu et al., 2006, Hoyer et al., 2009, Zhang and Hyvärinen, 2009].
However, in many real-world problems, this assumption may not hold.
For example, in complex systems, it is usually hard to enumerate and measure all task-related variables, so there may exist latent variables that inﬂuence multiple observed variables, the ignorance of which may introduce spurious correlations between measured variables.
A more complex scenario is that the variables form a hierarchical structure, where the latent variables may generate latent variables in a hierarchical way, while only the leaf nodes are measured, which is common in real-world scenarios.
For instance, in fMRI data analysis, hundreds of thousands of voxels are recorded, where these micro-variables may not be necessary to have clear semantic meaning.
Therefore, from the measured voxels, we aim to automatically identify conceptually meaningful functional brain regions of di ↵ erent levels, where the lower level represents simpler functional regions and the higher level represents more abstract and complex functional regions, which thus form a hierarchical structure.
[Table]
We may also see similar structures in image representation learning–image pixels are dependent [Table] and it seems sensible to consider them as observations generated by multiple-layer hidden concepts.
Previous causal discovery approaches that can handle latent confounders are mainly based on the following criteria.
• Conditional independence constraints.
The FCI algorithm [Spirtes et al., 2000], as well as its variants [Colombo et al., 2012, Pearl, 2000, Akbari et al., 2021], makes use of conditional independence tests over observed variables to identify the causal structure over observed variables up to a maximal ancestral graph.
This type of methods can handle both linear and nonlinear causal relationships, but the limitation is that there are large indeterminacies in the resulting graph about the existence of an edge, as well as the existence of confounders.
In practice, it is often the case that the resulting graph contains many undetermined edges, denoted by ◆ , where the circle can be either tail or arrow.
Moreover, they do not consider the causal relationships among latent variables.
• Tetrad condition.
With the Tetrad condition, i.e., the rank constraints of every 2 ⇥ 2 o ↵ -diagonal sub-covariance matrix, one is able to locate latent variables and identify the causal skeleton among them in linear-Gaussian models [Silva et al., 2006, Kummerfeld and Ramsey, 2016, Wang, 2020].
These methods assume that each observed variable is inﬂuenced by only one latent parent, and
[Table]
each latent variable has at least three pure measured children.
Moreover, the Tetrad condition can also be used to identify a latent tree structure [Pearl, 1988].
• Matrix decomposition.
It has been shown that, under certain conditions, the precision matrix can be decomposed into a low-rank matrix and a sparse matrix, where the low-rank matrix represents the causal structure from latent variables to observed variables and the sparse matrix gives the structural relationships over observed variable.
To achieve such decomposition, however, certain assumptions are imposed on the structure [Chandrasekaran et al., 2011, 2012].
A related work [Anandkumar et al., 2013] decomposed the covariance matrix into a low-rank matrix and a diagonal matrix, which requires three times more measured variables than latent variables.
[Anandkumar et al., 2013] can also handle multi-level DAGs that some latent variables do not have measured variables as children, but it requires that the underlying graph can be partitioned into multiple levels such that all the edges are between nodes in adjacent layers; the graphs in Figure 1(b) in the main text, and Figures 5-6 and Figures 7(a, b, d) in Appendix are not satisﬁed.
• Over-complete independent component analysis (ICA)-based methods.
Several methods [Shimizu et al., 2009] make use of over-complete ICA to learn the causal structure with latent variables, since it allows more source signals than observed variables.
These methods do not consider the causal structure among latent variables and the size of the equivalence class of the identiﬁed structure could be large.
In addition, in practice, the estimation of over-complete ICA models is easy to
L
L L
L L
L
L 7 L
L 9 L 10 X 12 X 13 X 14 X 15 X
X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 X 10 X(a)
X
X
X X
L
L L
L L
X X
L
X 0 X
L
X X
X X
X(b)
L L L L
L
L L L L
[Table] [Table] X [Table]
(c)
L
[Table]
L
[Table]
L
[Table]
L
L
L
L
[Table]
X
[Table]
X
X
[Table]
X
[Table]
X [Table] (d) get stuck in local optima, unless the underlying sources are very sparse [Entner and Hoyer, 2010,
Tashiro et al., 2014].
• Generalized independent noise (GIN) condition.
The GIN condition is an extension of the independent noise condition in the existence of latent confounders
It assumes the noise terms are non-Gaussian and
leverages
higher-order statistics to identify latent structures.
In particular, Xie et al.
[2020] proposes a GIN-based approach
that allows
multiple latent parents
behind
every
pair of observed variables and can identify causal directions among latent variables, but it still requires that each latent variable set should have
at
least twice more measured variables as children.
• Moreover, Huang* et al.
[2020] considered a special type of confounders in heterogeneous data, where the confounder can be represented as a function of domain index or a smooth function of time, so one may use the known domain index or time index as a surrogate variable to remove the inﬂuence from those confounders and thus identify causal structure over observed variables.
• Mixture oracles-based method.
Recently, Kivva et al.
[2021] proposed a mixture oracles-based method to identify the latent variable graph that allows nonlinear causal relationships.
It is based on assumptions that the latent variables are discrete and each latent variable has measured variables as children.
Thanks to the discreteness assumption, it can handle more general DAGs over latent variables.
D.1 Proof of Theorem
Theorem 1 (Graphical Implication of Rank Constraints in IL 2 H Graphs) .
Suppose G satisﬁes an IL 2 H graph.
Under the rank faithfulness assumption, the cross-covariance matrix ⌃ X A , X B over measured variables X A and X B in G (with | X A | , | X B | > r) has rank r, if and only if there exists a subset of latent variables L with | L | = r such that L d-separates X A from X B , and there is no L 0 with | L 0 | < | L | that d-separates X A from X B .
That is,
[Equation]
Proof.
Theorem 1 is a special case of Theorem 2.8 in Sullivant et al.
[2010] when applied to IL 2 H graphs.
Di ↵ erent from the setting in Sullivant et al.
[2010] which access to the full covariance matrix ⌃ V G , V G is assumed, we only have access to the covariance matrix ⌃ X G , X
G over the measured variables
X G .
It is enough to show that for IL 2 H graphs, ( C A , C B ) t-separating X A from X B is equivalent to L d-separating X A from X B , where C A , C B ⇢ L G .
Since L d-separates X A from X B and since any X 2 X G cannot be the choke point, we can choose
C A = L and C B = ; , so that ( C A , C B ) t-separates X A from X B in IL 2 H graphs.
Therefore, combining Theorem 2.8 in Sullivant et al.
[2010] and the above equivalence, the theorem is proved.
⇤
D.2 Proof of Theorem
Theorem 2 (Measurement as a surrogate) .
Suppose G is an IL 2 H graph.
Denote by A , B ✓ V G two subsets of variables in G , with A \ B = ; .
Furthermore, denote by X A the set of measured variables that are d-separated by A from all other measures, and by X B the set of measured variables that are d-separated by B from all other measures.
Then rank ( ⌃ A , B ) = rank ( ⌃ X A , X B ) .
Proof.
According to Theorem 1, the rank of ⌃ X A , X B is the minimal number of latent variables L that d-separate X A and X B ; that is, L block all paths between X A and X B with the smallest cardinality.
Furthermore, since A d-separates X A from all other measures, X A are the measured pure descendants of A .
Similarly, X B are the measured pure descendants of B .
Moreover, given the structure of IL 2 H graphs, L also block all paths between A and B with the smallest cardinality.
Therefore, the rank of ⌃ A , B is also | L | , equivalent to that of ⌃ X A , X B .
⇤
D.3 Proof of Theorem
Theorem 3 (Correct Cluster Condition) .
Suppose G is an IL 2 H graph with measured variables X G .
Consider the output G 0 from applying ﬁndCausalClusters over X G .
If none of the clusters in G 0 is the bond set in G , then all latent atomic covers have been correctly identiﬁed.
Before the proof of Theorem 3, we ﬁrst give the following lemma which shows that if X ✓ X G is not a correct cluster in the true graph G , then X forms a bond set.
In other words, If a set of measured variables does not form a bond set of G , then it must form a correct cluster.
Lemma 1 (Fake-Cluster ) Bond Set) .
A set of measured variables X ✓ X G that is mistakely tested as a rank-deﬁcient set by ﬁndCausalClusters but does not form a cluster in the true graph G (that is, a fake cluster), only if X forms a bond set of G .
Below, we ﬁrst give the proof of Lemma 1.
Proof.
Suppose we found a rank deﬁcient set of variables
X
=
{ X 1 , X 2 , · · · } which is not a cluster in
G .
This implies that (i) there exists at least two disjoint latent atomic covers, L 1 , L 2 , which d-separate variables in X from all other measures, and (ii) X does not contain all the pure children of either
L 1 nor L 2 , because otherwise, due to the IL 2 H requirement that each L i has > | L i | pure children, we would have discovered that a subset of X was rank deﬁcient and clustered together earlier by ﬁndCausalClusters .
This implies that L 1 , L 2 will d-separate the remaining pure child of L 1 from that of L 2 , implying that X is a bond set.
⇤
Now we are ready to prove Theorem 3.
Proof.
We will prove that all latent atomic covers can be correctly identiﬁed from bottom to top , if there is no bond set.
First, from Lemma 1 we know that if a set of measured variables is not a bond set, then they form a correct causal cluster, and thus the corresponding identiﬁed latent cover is correct.
Denote the latent atomic covers identiﬁed at this step by L 0 1 , and denote by G 0 1 the currently estimated graph.
After identifying the latent atomic covers at the downmost level, we next continue to form causal clusters from root variables in G 0 1 , including L 0 1 and the remaining measured variables that did not form clusters in the previous step.
If any of the latent covers in L 0 1 have latent children L i in the true graph G that have not been identiﬁed in the current step, then reverse these edges such that the children become parents.
Such an operation will not a ↵ ect the discovery of latent covers, and L i will be found in later steps.
So by further leveraging Lemma 1 and by treating L 0 1 and the remaining measured variables that did not form clusters in the previous step as “measured variables", this step results in correctly identiﬁed latent covers L 0 2 and estimated graph G 0 2 .
We can now iteratively repeat the previous step to discover new latent atomic covers from the root variables in the estimated graph in the previous step, until no more rank deﬁcient sets can be found. ⇤
D.4 Proof of Theorem
Theorem 4 (Correcting Clusters) .
Denote by G 0 the output from ﬁndCausalClusters and by G the true graph.
For a latent atomic cover L 0 in G 0 , if the measured pure descendants of L 0 is a bond set in the true graph G , then there exist a set of siblings S of L 0 in G 0 , a set of children C of L 0 , and a set of grandparents P of L 0 , such that M G 0 ( S [ C [ P ) forms a cluster that is not a bond set in G .
Proof.
Suppose the measured pure descendants of L 0 , X B M G 0 ( L 0 ), is a bond set in the true graph
G .
Denote by L S ✓ L G the minimal set of latent variables in G that d-separates X from all other measures
X 0 B X G \ X , and since X is a bond set, L S also d-separates some disjoint partition of measures X i ⇢ X 0 from X j ⇢ X 0 , and accordingly, denote by G i the subgraph that contains measures
X i and by G j the subgraph that contains measures X j .
Moreover, denote by L i ⇢ L S the set of latent variables that d-separates X i from other measures.
The proof contains three steps.
In step 1, we show that for each L 0 ’s sibling, its measured pure descendants are the same set of measured variables as that in the subgraph G i for some i .
To this end, we ﬁrst show that variables in each subgraph G i only formed clusters with variables in the same G i .
Without loss of generality, suppose there are only two subgraphs.
Suppose for contradiction that we discovered a cluster of variables V such that M G 0 ( V ) comprises measures from G 1 , G 2 .
Then V can be separated into variable sets V 1 , V 2 belonging to G 1 , G 2 respectively.
However, the minimal d-separating set for V 1 , V 2 must not overlap, since they are in di ↵ erent subgraphs.
This implies that either V 1 , V 2 forms a rank deﬁcient set by itself, which should have been discovered earlier.
Hence, we reach a contradiction.
We next show that for each L 0 ’s sibling, its measured pure descendants are not a proper subset of the measured variables as that in G i .
If it was the case, the variables in G i would continue to form variables with one another until L i is in the minimal d-separating set, where M G 0 ( A 0 )
=
X i , hence showing the claim.
In step 2, we show that the cardinality of each sibling of L 0 is equal to | L i | for some i .
We ﬁrst show that for a variable set V i such that M ( V i ) ✓ M ( G i ), it is not possible that | V i | <
| L i | .
This is because if this was the case, it implies that there exists a latent set of smaller cardinality that separates the measured variables in G i from the rest of the graph, which contradicts the fact that L i is minimal.
So it is always the case that | V i |  | L i | .
Next, we show that as long as any such variable set | V i | > | L i | , it will be able to form a cover with cardinality <
| V i | .
This is because if no rank deﬁcient sets exist for cardinality k <
| L i | ,
{ V i : the measured variables in V i = the measured variables in G i } will form a cluster of cardinality | L i | , since | L | >
| L i | < | V i | .
Thus, by combining step 1 and step 2, we have shown that each sibling of L 0 corresponding to L i for some i .
Finally, in step 3, we show that, for L 0 , there exist a set of siblings S , a set of children C , and a set of grandparent P , so that their union S [ C [ P will not form a bond cover.
Note that the measured pure descendants of siblings or grandparents are X i , which is the reason why we need to consider the siblings and grandparents.
Moreover, note that the reason L 0 is formed as bond cover is that when it is formed, its slibings have not been found yet.
Intuitively, now its siblings have been found, so we can ﬁnd the correct clusters.
Suppose for contradiction that reﬁning clusters will discover a new bond cover L bond , and without loss of generality, suppose the minimal d-separating set involves some distinct covers L A , L B .
Each of
L A , L B must respectively d-separate some partition of variables V A , V B ⇢ V from all other variables remaining in V .
We also know that in order to ﬁnd rank deﬁciency, ||
L A ||
+ || L B
|| <
|| V
A
||
+ || V B || , implying that
|| L A
|| <
|| V
A || or || L B
|| <
|| V B || .
However, since L A , L B d-separates V A , V B from all other variables respectively, testing either V A , V B must have been rank deﬁcient.
Since either of them are over a smaller latent cardinality ( || L A || ,
|| L B || <
||
L A ||
+ || L B
|| )
, one of them must have been discovered as a cluster earlier.
Hence, we reach a contradiction.
⇤
D.5 Proof of Lemma
Lemma 5 (Rank Invariance) .
Denote by G 0 the output from ﬁndCausalClusters and by L a latent atomic cover in G 0 .
Then the rank constraints over X G prescribed by G 0 before and after the operation
makeRoot ( L ) are identical.
Proof.
For an IL 2 H graph G 0 and a latent atomic cover L in G 0 , after applying the makeRoot operator to L , which results in G 00 , G 0 and G 00 are in the same Markov equivalence class.
Therefore, G 0 and
G 00 have the same rank constraints. ⇤
D.6 Proof of Lemma
Lemma 6 (Cross-Cover Test) .
Given a set of variables S , consider two latent atomic covers L A , L B
S , and a potential separating set C =
{ L C i } ✓ S\{ L A , L B } .
For each L C i , consider C Ai , C Bi ✓
PCh ( L C i ) with C Ai , C Bi , ; and C Ai \ C Bi = ; , and denote the cardinality k Ai B min ( | L C i | , | C Ai | ) , k Bi B min ( | L C
i | , | C Bi | ) , respectively.
Then there is no edge between L A and L B if and only if there exists a separating set C such that rank ( ⌃ A , B ) < min ( | L A | +
P i k Ai , | L B | + P i k Bi ) , where A =
{ L A , C A 1 ,
C A 2 , ... } and B = { L B , C B 1 , C B 2 , ... } .
In this case, we say that C satisﬁes the cross-cover test of L A against L B .
Proof.
We ﬁrst show that if there is no edge between L A and L B , then rank ( ⌃ A , B ) < min ( | L A | + P i k A i , | L B | + P i k B i ).
Since there is no edge between L A and L B , there exists a set C
=
{ L C i } , so that given C , L A and L B are d-separated.
Since C A i and C B i are the children of L C i , C d-separates A from B as well.
Then according to Theorem 7.1, rank ( ⌃ A , B ) = | C | .
Moreover, since | C | < min ( | L A | + P i k Ai , | L B | + P i k Bi ), we have rank ( ⌃ A , B ) < min ( | L A | +
P i k Ai , | L B | + P i k Bi ).
Next we show that rank ( ⌃ A , B ) < min ( | L A | +
P i k Ai , | L B | + P i k Bi ), then there is no edge between L A and L B .
Now suppose that there are edges between L A and L B .
Then ⌃ A , B is not rank deﬁcient; that is rank ( ⌃
A , B ) = min ( | L A | + P i k Ai , | L B | + P i k Bi ).
Therefore, if rank ( ⌃ A , B ) < min ( | L A | + P i k
Ai , | L B | + P i k Bi ), then there is no edge between L A and L B . ⇤
D.7 Proof of Lemma
Lemma 7 (V-Structure Test) .
For any unshielded triangle L A  L C  L B , let A , B be the set of variables in Lemma 6 such that ⌃
A , B was rank deﬁcient.
Let k = rank ( ⌃ A , B ) , k 1 = rank ( ⌃ A [ L C , B ) , and k 2 = rank ( ⌃ A , B [ L C ) .
Then, L A ! L B L C if and only if k < min( k 1 , k 2 ) .
Proof.
We ﬁrst show that if L A ! L C L B , then k < min( k 1 , k 2 ).
Since L A ! L C L B , L C cannot be in the separation set of L A and L B ; that is, given L C , L A and
L B are d-connected.
Hence, k < k 1 and k < k 2 , and thus k < min( k 1 , k 2 ).
Next we show that if k < min( k 1 , k 2 ), then L A ! L C L B .
Suppose L A , L C , L B do not form a v-structure; that is L A ! L C ! L B or L A L C
L B . Then k = min( k 1 , k 2 ), since L C has been considered before in order to achieve rank deﬁciency of ⌃ A , B .
Therefore, if k < min( k 1 , k 2 ), then L A ! L C L B .
⇤
D.8 Proof of Lemma
Lemma 8.
Suppose G is an IL 2 H graph.
The rank constraints are invariant with the minimal-graph operator and the skeleton operator; that is, G and O skeleton
( O min ( G )) are rank equivalent.
Proof.
We ﬁrst show that the minimal-graph operator will not change rank deﬁciency constraints.
Denote by G 1 and G 2 the graph before and after applying the minimal-graph operator, respectively.
For every latent atomic cover L in G 1 , since those three conditions hold, for any C ✓ PCh G 0 ( L 0 ) and for any S ✓ S ib G 0 ( L 0 ) with C , S , ; , rank ( ⌃ A , B ) =
| P | , where A = C [ S and B = X G 0 \M G 0 ( A ).
So, after merging L to its parents P , the cardinality of the d-separation set between any two sets of variables does not change.
Thus, according to Theorem 1, the rank constraints will not change after merging L to its parents P .
Moreover, it is trivial to show that the skeleton operator will not change rank deﬁciency constraints, because the d-separation set between any two sets of variables will not change. ⇤
D.9 Proof of Theorem
Theorem 9.
Suppose G is an IL 2 H graph with measured variables X G .
Phases I-II in Algorithmover X G can asymptotically identify the latent atomic covers of O min ( G ) , with the ﬁrst two conditions in Condition 1.
Proof.
Theorem 3 has shown that ﬁndCausalClusters gives correct latent covers when there is no bond set.
Furthermore, Theorem 4 shows that even in the presence of bond sets, reﬁning the set over
Ch ( L 0 ) [ S ib ( L 0 ) [ Gp ( L 0 ), where M G 0 ( L 0 ) forms a bond set, can correct the clusters.
Phase II reﬁneClusters reﬁnes clusters over Ch ( L 0 ) [ S ib ( L 0 )
[ Gp ( L 0 ) in a breadth-ﬁrst search from the root variable, and it ends after reﬁning every latent cover in G 0 .
Therefore, with this reﬁning procedure, we will derive correct latent covers. ⇤
D.10 Proof of Theorem
Theorem 10.
Suppose G is an IL 2 H graph with measured variables X G .
Algorithm 1, including
Phases I-III, over X G can asymptotically identify the Markov equivalence class of O min ( O s ( G )) .
Proof.
Theorem 5 has shown that Phases I-II can ﬁnd the correct clusters and latent atomic covers of
O min ( G ).
Moreover, Lemma 2 and Lemma 3 have shown that by performing Cross-Cover Test and
V-Structure Test, the skeleton and v structure among every triple of latent variables can be correctly identiﬁed.
Phase III reﬁneEdges reﬁnes the edges over L 0 [ PCh ( L 0 ) [ PCh
PCh ( L 0 )  by performing CrossCover Test and V-Structure Test in a depth-ﬁrst search from the root variable, and it ends until Ldoes not have latent children.
Therefore, with this reﬁning procedure, we will derive correct skeletons and v structures.
⇤ Theorems E.1 More Explanations on Deﬁnition 3 ( E ↵ ective Cardinality )
The e ↵ ective cardinality, deﬁned in Deﬁnition 3, can be estimated with the following procedure.
[Equation]
while j < | L | do Find the largest subset of variables C✓ C such that | C 0 | > | Pa G ( C 0 ) | =
j ; Introduce a set of latents L 0 with | L 0 | = | Pa G ( C 0 ) | ; add L 0 as new children of Pa G ( C [Equation]
[Equation] 0 ; j += 1; end while return | C |
For example, for Figure 1(a), the e ↵ ective cardinality of the pure children of { L 7 , L 8 } is 3, because |{ X 4 , X 5 }| > |{ L 7 }|
and we replace { X 4 , X 5 } with a single latent variable L 0 , so the cardinality of the resulting children set is [Equation]
E.2 More Explanations on Deﬁnition 4 ( Latent Atomic Cover )
The ﬁrst two conditions in Deﬁnition 4 ensure that there are enough variables in the current active variable set to ﬁnd the rank deﬁciency, so that we can determine the latent atomic cover with size k .
However, note that they may not be the necessary conditions.
For example, for the graphs in Figure 8, some of the latent atomic covers only have k neighbors (except for the k + 1 pure children), but they are still identiﬁable.
The ﬁrst half of the third condition, “there does not exist a partition of L =
L 1 [ L 2 , so that both
L 1 , L 2 satisfy conditions 1 and 2", ensure that the latent atomic cover L is atomic.
The second half of the third condition, “there does not exist a partition of L =
L 1 [ L 2 , so that { PCh [Equation] L [Equation]
, is needed in the overlapping case.
Consider the following graph: L 1 ! X 1 , X 2 , X 3 } , L 2 ! { X 3 , X 4 , X 5 } .
Here, L 1 = { L 1 } , L 2 = { L 2 } , and L = { L 1 , L 2 } are latent atomic covers; however, { PCh G ( L 1 )
= { X 1 , X 2 } , { PCh G ( L 2 ) =
{ X 4 , X 5 } , and { PCh G ( L ) =
{ X 1 , X 2 , X 3 , X 4 , X 5 } , so { PCh G ( L 1 ) [ PCh G ( L 2 ) }\ L , PCh G ( L ).
Therefore, although both L 1 and L 2 satisfy the ﬁrst two conditions, L satisﬁes Deﬁnition 4.
E.3 More Explanations on Condition 1 ( IL 2 H graph )
The ﬁrst two conditions in Condition 1 guarantee to identify the latent atomic covers, while the last one is used to identify the edges among latent atomic covers when performing Cross-Cover Test and
V-Structure Test.
Speciﬁcally, the second condition says that two latent atomic covers that are partly overlapped is not allowed, except for the case that one latent atomic cover is contained in another one; otherwise, the cover creation rule can be non-trivial.
For example, for the graph in Figure 9(a), although
PDe G ( { L 3 , L 4 } ) T PDe G ( { L 4 } ) =
{ X 3 , X 4 } , it satisﬁes the second condition in Condition 1 because { L 3 } ⇢ { L 2 , L 3 } .
In contrast, for the graph in Figure 9(b), PDe G ( { L 3 , L 4 } ) T PDe G ( { L 4 , L 5 } ) =
{ X 3 } , but { L 3 , L 4 } , { L 4 , L 5 } are not subsets or descendants of one another.
Hence, it does not satisfy the second condition of an IL 2 H graph.
E.4 More Explanations on Algorithm 2 ( ﬁndCausalClusters )
The search procedure in Algorithm 2 ( ﬁndCausalClusters ) contains the following two key updates.
• Latent-atomic-cover size update.
We start to identify latent atomic covers with size k = 1.
If a rank-deﬁciency set is not found, then increment k
= k + 1; otherwise, reset k = 1.
• Active variable set update.
The set of active variables S is set to X G initially.
We consider any subset of the latent atomic covers in S and replace them with their pure children, resulting in ˜ S .
At
L
L 2 L
L 4 L
L 6 L 7 L
L 9 L 10 X 9 X 10 X 11 X
X 1 X 2 X 3 X 4 X 5 X 6 X 7 X(a)
X
X X
L
L L
L L
X X
L
X 0 X
L
X X
X X
X(b)
L
L 2 L
L L
L L
X X
X X
X X
X X
X 9 X 10 X(c)
L
L L
L L
L L
L
L 9 L
L 11 L 12 L 13 L 14 L
X 1 X 2 X 3 X 4 X 5 X 6 X 7 X 8 X 9 X 10 X 11 X 12 X 13 X 14 X 15 X 16 X 17 X(d)
L
L L
L L
L L
L
X 1 X 2 X 3 X 4 X 5 X
X
X
X 9 X X 11 X
[Figure]
[Figure]
[Figure]
the same time, we search for the rank-deﬁciency set A with rank k in ˜ S ; if it is found, then assign a latent atomic cover L of size k as the parent A , and accordingly, the active variable set is updated as ( S\ A ) [ L .
Note that in Algorithm 2 ﬁndCausalClusters , we search over ˜ S , instead of S , to avoid adding a certain type of redundant latent variables when some latent atomic covers have overlapping variables, including the v structure; see Figure 10 for an illustration.
[Figure]
Moreover, in Algorithm 2 ﬁndCausalClusters , if there are conﬂicts when the search goes on, then we just ignore it, and such conﬂicts will be handled in Algorithm 3 reﬁneClusters .
Also, note that except for the v structure where the measured variable is a collider, other “v structures" in the intermediate output in Algorithms 2 and 3 are not true v structures.
For the illustration of Algorithm 2 given in Figure 2, here we give more detailed explanations.
Speciﬁcally, we ﬁrst set k = 1 and the active set is S =  X 1 , · · · , X 16 and ˜ S = S , and we can ﬁnd the clusters in (a), and no further cluster can be found with k = 1.
Then we increase k towith the active set S =
{  L 6 } , { L 7 } , X 6 , · · · , X 16 and ˜ S = S , and then we can ﬁnd the clusters in (b).
Then, the active set is S =  { L 4 , L 5 } , { L 6 } , { L 7 , L 8 } , { L 9 , L 10 } and we set back k = 1, and when ˜
S =
{{ L 4 , L 5 } , X 1 , · · · , X 11 } we ﬁnd the cluster in (c).
Note that when testing the rank over { L 4 , L 5 } against other variables, we use their measured pure descendants in the currently estimated graph instead.
The above procedure is repeated to further ﬁnd the cluster in (d).
Finally, when there are no enough variables for testing, we connect the elements in the active variable set: connecting { L 2 , L 3 } to { L 7 , L 8 } in (e).
E.5 Explanation of Minimal-Graph Operator and Skeleton Operator
The minimal-graph operator and the skeleton operator will not change the rank constraints, or in other words, graphs before and after applying the operators are indistinguishable with rank constraints.
Give an IL 2 H graph in Figure 11(a), after applying the minimal-graph operator, the latent atomic cover L 4 will be merged to its parent L 5 , resulting in the graph in Figure 11(b), while the rank constraints will not change.
Furthermore, after applying the skeleton operator to the graph in Figure 11(b), L 1 has an edge to X 7 and L 3 has an edge to X 1 , resulting in the graph in Figure 11(c), which also does not change the rank constaints.
E.6 Complexity of Algorithm
The time complexity of the algorithm is upper bounded by O ( r
P l + 1 k =⇣ mk ⌘ ), and this bound is further upper bounded by O ( r (1 + m ) l + 1 ), where m is the number of measured variables, l is the cardinality of the
[Figure]
E.7 Illustrative Examples of the Entire Algorithm
Figure 12 and Figure 13 give two illustrative examples of the entire algorithm, showing how each step proceeds.
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
f) Next perform ﬁndCausalClusters over {
[Table]
} we can ﬁnd a
[Table]
[Table]
[Table]
(g) Next perform ﬁndCausalClusters over { L 0 11 , L 0 4 , L 0 5 , X 9 } we can ﬁnd a latent cover L 0 12 .
[Table]
(i) Reﬁne { L 0 3 } by ﬁrst removing { L 0 3 } and its par- ents { L 0 10 } and performing ﬁndCausalClusters over { L 0 6 , L 0 7 , L 0 11 , X 11 } , and then we can ﬁnd a latent cover { L 0 13 } .
[Table]
[Table]
[Table]
{ L
[Table]
[Figure]