Causal Discovery in Linear Latent Variable Models Subject to Measurement Error
Abstract
We focus on causal discovery in the presence of measurement error in linear systems where the mixing matrix, i.e., the matrix indicating the independent exogenous noise terms pertaining to the observed variables, is identiﬁed up to permutation and scaling of the columns.
We demonstrate a somewhat surprising connection between this problem and causal discovery in the presence of unobserved parentless causes, in the sense that there is a mapping, given by the mixing matrix, between the underlying models to be inferred in these problems.
Consequently, any identiﬁability result based on the mixing matrix for one model translates to an identiﬁability result for the other model.
We characterize to what extent the causal models can be identiﬁed under a two-part faithfulness assumption.
Under only the ﬁrst part of the assumption (corresponding to the conventional deﬁnition of faithfulness), the structure can be learned up to the causal ordering among an ordered grouping of the variables but not all the edges across the groups can be identiﬁed.
We further show that if both parts of the faithfulness assumption are imposed, the structure can be learned up to a more reﬁned ordered grouping.
As a result of this reﬁnement, for the latent variable model with unobserved parentless causes, the structure can be identiﬁed.
Based on our theoretical results, we propose causal structure learning methods for both models, and evaluate their performance on synthetic data.
Introduction
Learning causal structure among the variables of a system from observational data has received considerable attention in the literature, since subject matter knowledge on causal relationships is often incomplete or impossible to obtain in many applications [24, 16].
In many real-life systems, not all variables of interest or all direct common causes of those variables can be observed.
This necessitates approaches for structure learning capable of dealing with latent variables.
For the case that no a-priori restrictions on the functional form of causal mechanisms are imposed, constraint-based algorithms such as the Fast Causal Inference (FCI) algorithm have been proposed [24].
Such approaches are often unable to identify the direction of the majority of causal connections in the system, which motivates placing additional, often parametric, restrictions on the generating model.
One of the most commonly used restrictions is to assume that the model is linear with non-Gaussian exogenous noise terms [21, 22, 12, 10].
This generating model leads to identiﬁcation of the existence and orientation of all edges in a causal model, assuming that no latent variables that are common causes of two or more variables exist [21, 22].
Moreover, in such models even when latent variable common causes exist, it is still possible to orient additional edges compared to algorithms such as FCI [12, 19, 1].
In certain applications, although some of the variables of interest (called underlying variables) are latent, we may have access to noisy measurements of them [17, 13, 29, 18, 11].
In this case, conditional independence patterns among observed variables (that are noisy measurements of underlying variables) are different from those among the underlying (measurement-error-free) variables.
In general, constraint-based approaches to causal discovery are not able to correct for the difference in observed independence due to the presence of measurement error, and thus will produce erroneous edge adjacencies and orientations.
Similarly, naive application of methods based on non-Gaussian exogenous noise assumption may also fail in recovering the correct causal relations in the presence of measurement error.
This is due to the fact that the extra measurement noise terms break the asymmetry utilized by those methods.
In this paper, we bring additional insight into the problem of causal discovery under measurement error by showing a surprising connection between linear structural equation models (SEMs) where variables are measured with error, and linear SEMs with hidden variables.
In particular, we consider two linear SEMs.
In the ﬁrst model, the unobserved variables are in fact of interest and can appear in any part of the causal structure of the system, but we assume that noisy measurements of these unobserved variables are available.
We refer to this model as the linear SEM with measurement error (SEM-ME).
The second model does not require measurements of the latent variables, as they are not of primary interest, yet we assume that all latent variables are root variables, meaning that they are not causally affected by any other latent or observed variables in the system.
We refer to this model as the linear SEM with unobserved roots (SEM-UR).
It is noteworthy that assuming latent variables are limited to be root variables does not affect the estimated total causal effects among the observed variables (see Remark 1 in Section 2.2), and provides a representation of a latent variable model that is consistent with these causal effects.
For this reason, this restriction is common in the literature of causal inference [8, 9, 12, 5].
We study the identiﬁcation of linear SEM-MEs and SEM-URs in a setup where the independent exogenous noise terms that causally (directly or indirectly) affect each observed variable can be identiﬁed.
That is, the mixing matrix of the linear system that transforms exogenous noise terms to observed variables is identiﬁed up to permutation and scaling of the columns.
This can be satisﬁed, for example, if all independent exogenous noise terms are non-Gaussian.
Our ﬁrst main contribution is presenting a mapping between the linear SEM-MEs and SEM-URs, which demonstrates a correspondence between a weighted causal diagram 2 generated by SEM-UR and a set of weighted causal diagrams from the SEM-ME (Theorem 1).
The models in this correspondence all possess the same mixing matrix.
Consequently, any identiﬁability result based on the mixing matrix for one model is applicable to the other model.
This allows us to study the problem of causal discovery in these two models together.
Additionally, we study the identiﬁability of linear SEM-MEs and SEM-URs, and show how it beneﬁts from a two-part faithfulness assumption.
The ﬁrst part prevents existence of zero total causal effects of a variable on its descendants and the second part prevents cancellation or proportionality among speciﬁc edges.
Our second main contribution is to characterize the extent of identiﬁability of the causal model under our faithfulness assumption.
We demonstrate that if only the ﬁrst part of the faithfulness assumption is imposed, the model can be learned up to an equivalence class characterized by an ordered grouping of the variables which we call ancestral ordered grouping (AOG).
In this grouping, the induced graph on the variables of each group is a star graph (see Section 4.3).
Although the AOG is identiﬁed, not all edges across the groups or the group centers (or their exogenous noise terms) are identiﬁable (Theorem 2).
The AOG characterization is a reﬁned version of the ordered grouping proposed in [29].
We further show that if both parts of the faithfulness assumption are imposed, the model can be learned up to an equivalence class characterized by a more reﬁned ordered grouping which we call direct ordered grouping (DOG).
Edges across groups can be identiﬁed under this characterization, yet the the group centers (or their exogenous noise terms) remain unidentiﬁed (Theorem 3).
This characterization further implies that the ground-truth structure of a SEM-UR is uniquely identiﬁable (Corollary 3).
Lastly, we show that models in the DOG equivalence class are strictly sparser than other models in the AOG equivalence class (Proposition 4).
We propose causal structure learning methods for both SEM-ME and SEM-UR based on this property, and evaluate their performance on synthetic data.
Model description
In this section, we ﬁrst introduce the two models considered in this paper, namely linear SEM-ME and SEM-UR.
We then discuss a model assumption needed for our identiﬁability results.
2.1 Linear structural equation model with measurement error (SEM-ME)
Deﬁnition 1 (Linear SEM-ME)
A linear SEM-ME consists of a set of “underlying” variables which can be partitioned into unobserved underlying variables Z
= { Z 1 , ..., Z p z } and observed underlying variables Y =
{ Y 1 , ...,
Y p − p z } .
In addition, we have another set of observed variables
U =
{ U 1 , · · · , U p z } corresponding to the noisy measurements of Z .
The underlying variables in
Z ∪ Y can be arranged in a total (causal) order (such that no later variable in the order can cause any earlier variable), and variables V i ∈
Z ∪ Y and U i ∈
U are generated as follows:
[Equation]
where [ n ] := { 1 , 2 , · · · , n } , Pa ( V i ) ⊆
Z ∪ Y denotes direct causes of V i .
N V i (resp.
N U i ) is the exogenous noise term (resp.
the measurement error) corresponding to V i (resp.
U i ).
We deﬁne the weighted causal diagram of the linear SEM-ME as a weighted directed graph where the nodes are the variables in Z ∪ Y .
There is a directed edge (causal connection) from V i to V j with weight c ji if and only if c ji (cid:54) = 0 , for V
i , V j ∈ Z ∪ Y .
Because of our causal order assumption, the causal diagram will be acyclic.
We use the terms node and variable interchangeably.
The linear
SEM-ME in this work is a generalization of the linear causal model with measurement error proposed in [29], in the sense that some of the underlying variables can be observed.
We deﬁne an unobserved leaf node ( u-leaf node) as an unobserved underlying variable in Z that does not have any children in Z ∪ Y .
The rest of the underlying variables are referred to as non u-leaf nodes ( nu-leaf nodes): These include unobserved non-leaf underlying variables, which have children in Z ∪ Y , and observed underlying variables in Y .
Similar to the argument in [29], given observed data generated by the linear SEM-ME in Equation (1), for a u-leaf node Z i , its exogenous noise term N Z
i is not distinguishable from its measurement error N U i .
This follows because Z i is not observed, and N Z i only inﬂuences one observed variable which is the noisy measurement
U i .
Therefore, we restrict our focus to the following canonical form.
Deﬁnition 2 (Canonical form of a linear SEM-ME)
The canonical form of a linear SEM-ME is the one in which all u-leaf nodes have no exogenous noise terms.
Speciﬁcally,
[Equation]
where Z NL , Z L , Y are the vectors of unobserved underlying non-leaf variables, u-leaf nodes, and observed underlying variables, respectively.
N Z NL , N Z L , and N Y are the corresponding noise vectors.
Z NU =
[ Z NL ; Y ] is the vector of nu-leaf nodes.
U is the vector of noisy measurements and N U NL and N U L denote the noise vectors corresponding to the measurements of Z NL and Z L , respectively.
C represents the causal connections among nu-leaf nodes, and can be partitioned into [ C Z ; C Y ] corresponding to [ Z NL ; Y ] , respectively.
D represents the causal connections from nu-leaf nodes to u-leaf nodes.
Note that C can be arranged into a strictly lower triangular matrix via simultaneous column and row permutations (due to the acyclicity assumption).
2.2 Linear structural equation model with unobserved roots (SEM-UR)
Deﬁnition 3 (Linear SEM-UR)
A linear SEM-UR consists of a set of observed variables X
= { X 1 , · · · , X q } and another set of latent variables H
= { H 1 , · · · , H l } .
All latent variables are assumed to be root variables (have no direct causes).
The observed variables are arranged in a causal order, and each observed variable is directly inﬂuenced by a linear combination of other observed and latent variables, plus an independent exogenous noise term.
[Equation]
We deﬁne the weighted causal diagram of the linear SEM-UR as a directed graph where the nodes are the observed and latent variables in X ∪ H .
There is a causal connection from X i (resp.
H i ) to X j with weight a ji (resp.
b ji ) if and only if a ji (resp.
b ji ) (cid:54) = 0 .
Because of our causal order assumption, the causal diagram will be acyclic.
Note that unlike in [12] and [19], we consider a broader model which allows latent variables to have a single observed child.
However, unlike the works considering relations among latent variables [1, 26, 3, 27], we restrict all latent variables to roots; see Remark 1.
A linear SEM-UR can be written in the following matrix form:
[Equation]
where X =
[ X 1 · · · X q ] (cid:62) and H =
[ H 1 · · · H l ] (cid:62) are the vectors of observed and latent variables, respectively.
N X
=
[ N X 1 · · · N X q ] (cid:62) and N H =
[ N H 1 · · · N H l ] (cid:62) represent the vectors of independent noise terms associated with X and H , respectively.
B represents the causal connections from latent to observed variables, and A represents the causal connections among observed variables, which can be assumed to be a strictly lower-triangular matrix (due to the acyclicity assumption).
Remark 1 There exist works aiming to ﬁnd the causal relations among the latent variables (as well as their relations with the measured variables) [23, 26, 3, 1, 27].
Our deﬁnition of a linear SEM-UR requires all the latent variables to be parentless.
This assumption is common in the literature of causal inference, as it does not restrict the feasible total causal effects (i.e., sum of products of path coefﬁcients) among the observed variables [8, 9, 5].
Speciﬁcally, let us start from a general linear latent variable model in which some latent variables have parents.
Hoyer et al.
[12] proposed an algorithm which maps such general model to one in which latent variables are parentless.
They showed that the resulting model is observationally and causally equivalent to the original model, i.e., the joint distributions of observed variables are identical, and all causal effects of observed variables on other observed variables are identical in the models before and after the mapping.
2.3 Separability assumption
For both models in Sections 2.1 and 2.2, we assume that the noise components can be separated from observations (i.e., the mixing matrices are recoverable).
We ﬁrst describe the mixing process for each of the two models, and then state our separability assumption.
From Equation (2), we can write all underlying variables in the linear SEM-ME in terms of the independent noise terms, as follows.
[Equation]
where W ME =  
[Equation]
− C
[Equation]
Y 
[Equation]
I is the identity matrix, and ( I − C )
− 1 Z
and ( I − C ) − 1 Y represent the rows of ( I − C ) − 1 corresponding to Z NL and Y , respectively.
Using the relation between U and Z in Equation (2), the two types of observed variables, i.e., measurements U and observed underlying variables Y , can be written as
[Equation]
We refer to W as the overall mixing matrix of the system.
Given any column permuted and rescaled version of W , we can recover matrix W ME by removing the submatrix [ I 0 ] (cid:62) of size p × p z corresponding to U .
Recall that p z is the cardinality of Z (which is the same as the cardinality of U ).
Similarly, according to Equation (4), observed variables in the linear SEM-UR can be written in terms of the independent noise terms as
[Equation]
where
[Equation]
Assumption 1 (Separability)
The linear SEM-ME (resp.
linear SEM-UR) is separable, that is, the mixing matrix W ME in Equation (5) (resp.
W UR in Equation (7) ) can be recovered from observations of [ U ; Y ] (resp.
X ) up to permutation and scaling of its columns.
Separability assumption states that for every observed mixture, the independent exogenous noise terms pertaining to this mixture can be separated, i.e., the mixing matrix can be recovered up to permutation and scaling of its columns.
An example of a setting where this assumption holds is when all exogenous noises are non-Gaussian.
In this case, if the model satisﬁes the requirement in [7, Theorem 1] (SEM-ME always does), overcomplete Independent Component Analysis (ICA) can be used to recover the mixing matrix up to permutation and scaling of its columns.
Another example where separability assumption is satisﬁed is the setup in which the noise terms are piecewise constant functionals satisfying a set of mild conditions [2].
On the other hand, an example where this assumption is violated is when all exogenous noise terms have Gaussian distributions.
In this case, the mixing matrix can only be recovered up to an orthogonal transformation.
Mapping between the weighted graphs of the models for identiﬁability
A key observation in this work is that there exists a mapping between the weighted causal diagrams of the introduced SEM-ME and SEM-UR, which leads to the corresponding mixing matrices W ME and W UR being transposes of one another.
Deﬁne the mapping ϕ from the set of weighted causal diagrams of linear SEM-URs, denoted by
M UR , to the set of weighted causal diagrams of canonical linear SEM-MEs, denoted by M ME , where for each diagram M ∈ M UR , ϕ ( M ) ⊂
M ME is constructed as follows: (a) Replace each latent variable in M with a Z variable.
(b) Replace each observed root variable in M with a Y variable.
(c) Replace the rest of the observed variables in M with either Z or Y variables.
(d) Reverse all the edges in M .
Note that ϕ ( M ) encompasses a set of models, corresponding to the possible choices in step (c).
Similarly deﬁne the mapping φ from the set M ME to the set M UR , where for each diagram
M (cid:48) ∈
M ME , φ ( M (cid:48) )
∈
M UR is constructed as follows.
(a) Replace each u-leaf variable in M (cid:48) with a latent variable.
(b) Replace the rest of the variables in M (cid:48) with observed variables.
(c) Reverse all the edges in M (cid:48) .
Note that from the deﬁnition of the two mappings, for any given M ∈ M UR ,
M = φ ( ϕ ( M )) .
For any given M (cid:48)
∈
M ME , M (cid:48)
∈ ϕ ( φ ( M (cid:48) )) .
Theorem 1 Let M ∈ M UR with mixing matrix W UR ( M ) and M (cid:48) be a corresponding element in
M ME with mixing matrix W ME ( M (cid:48) ) ,
where M (cid:48) ∈ ϕ
( M ) and M = φ ( M (cid:48) ) .
Then there exist a permutation of the columns of W UR ( M ) , denoted by ˜ W UR ( M ) , and a permutation of the columns of W ME ( M (cid:48) ) , denoted by ˜ W ME ( M (cid:48) ) , which satisfy ˜ W ME ( M (cid:48) )
=
( ˜ W UR ( M )) (cid:62) .
The proofs of all the results are provided in the Appendix.
Note that as it can be seen from the proof of Theorem 1, whether the nu-leaf variables, that are non-leaf in the SEM-ME, are observed or not does not change the corresponding mixing matrix.
Remark
The mapping introduced here is between a weighted diagram in SEM-UR and a set of weighted diagrams in SEM-ME.
The elements of the set in fact correspond to different settings of the non-leaf underlying variables being observed or not.
Note that in most applications, it is clear to the researcher whether an underlying variable is observed or measured with error.
Therefore, only one element of the set is relevant to the application and the set reduces to a single element.
Consequently, in our identiﬁability results, without loss of generality, we assume the set is a singleton.
X X
X
H a c b d (a) Linear SEM-UR    0
d a 1 0 bd + c ab b  
W UR
Z L
Y
Z 3 /Y
Z 2 /Ya c b d (b) Linear SEM-ME   d bd + c b 1 1 a ab 

W ME
Theorem 1 implies that under separability assumption, any identiﬁability result based on the mixing matrix for one model is applicable to the other model, by reversing all edges.
This allows us to study the problem of causal discovery of these two models simultaneously.
Example 1 Consider the linear SEM-UR in Figure 1(a) comprised of three observed and one latent variable (denoted by a single circle).
The corresponding linear
SEM-ME under the mapping ϕ is shown in Figure 1(b).
Speciﬁcally, (i) the latent variable H is mapped to the u-leaf variable Z L (denoted by a double circle); (ii) the observed root X 1 is mapped to the observed underlying variable Y 1 ; and (iii) observed variables X 2 and Xcould be mapped to either observed or unobserved underlying variables; hence the UR graph will be mapped to a set of 4 graphs in the ME model.
The mixing matrices for both models are shown in Figure 1.
It can be readily seen that there exists a permutation of the columns of W UR , which is equal to the transpose of W ME .
Identiﬁability of models
In this section, we study the extent of identiﬁability of linear SEM-ME and SEM-UR under the separability assumption.
We consider both a weaker notion of structure identiﬁcation, where only the graph structure of the model is obtained, and a stronger notion of model identiﬁcation where, in addition to the graph structure, model parameters are recovered as well.
In Section 4.1, we propose
SEM-ME and SEM-UR faithfulness assumptions, which consist of two parts, the ﬁrst of which is identical to the conventional faithfulness in linear causal models.
We ﬁrst discuss model identiﬁability under conventional faithfulness (i.e., the ﬁrst part of our assumptions), which can be characterized by ancestral ordered grouping (AOG) of the variables, explained in Section 4.2.
Next, under both parts of our proposed faithfulness assumptions, the extent of identiﬁability can be characterized by direct ordered grouping (DOG) of the variables, explained in Section 4.3.
This leads to a subsequent result on structure identiﬁability for SEM-UR.
In Section 4.4, we show that the DOG characterization possesses a sparsity property, which can be used to construct recovery algorithms for both models.
4.1 Faithfulness assumption
Assumption 2 (SEM-ME faithfulness) (a) The total causal effect of any underlying variable on its descendant is not equal to zero.
(b) For each variable V in a SEM-ME, the causal effect of parents of
V on V cannot be replicated by fewer or equal variables due to ﬁne tuning of the parameters.
(See
Appendix B.1 for the formal statement.)
Assumption 3 (SEM-UR faithfulness) (a) The total causal effect of any observed or latent variable on its descendant is not equal to zero.
(b) For each variable V in a SEM-UR, the causal effect of V on children of V cannot be replicated by fewer or equal variables due to ﬁne tuning of the parameters.
(See Appendix B.1 for the formal statement.)
Assumptions 2 (a) and 3 (a) are identical to conventional faithfulness in linear causal models.
They require that when multiple causal paths exist from any (observed or unobserved) variable to its descendants, their combined effect (i.e., sum of products of path coefﬁcients) is not equal to zero, see [24].
Assumptions 2 (b) and 3 (b) prevent certain path cancellation or parameter proportionality.
Importantly, Assumptions 2 and 3 are violated with probability zero if all model coefﬁcients are drawn randomly and independently from continuous distributions; see Appendix B.3 for the proof.
As an example for violation of Assumption 2 (b) , consider the linear SEM-ME generated by the graph in Figure 2(a), which satisﬁes Assumption 2 (a) .
However, we can write Z 4 as ( b + c ) Z 2 + N Z 3 , which is caused by the path cancellation of the triangle ( Z 1 , Z 3 , Z 4 ) .
This means that that the causal effect of Z 1 and Z 2 on Z 4 can be summarized by Z 2 alone.
Therefore, the model violates Assumption 2 (b) .
Lastly, Assumption 3 is strictly weaker than bottleneck faithfulness in [1]; see Appendix B. for the proof.
Please refer to Appendix B for more discussion about our faithfulness assumption and
Appendix D for a detailed comparison between our identiﬁability results and the results in [1].
4.2 Ancestral ordered grouping
[Equation]
We ﬁrst study the extent of identiﬁability of the models only under conventional faithfulness, i.e., Assumptions 2 (a) and 3 (a) .
Deﬁnition 4 (Ancestral ordered grouping (AOG))
The AOG of a SEM-ME (resp.
SEM-UR) is a partition of Z ∪ Y (resp.
H ∪ X ) into distinct sets.
This partition is described as follows: (1) Assign each nu-leaf node (resp.
observed node) to a distinct group.
(2) SEM-ME:
For each u-leaf node Z j ∈ Z , if there exists one parent V i such that Z j has no other parents, or all other parents of Z j are also ancestors of V i , assign Z j to the same group as V i .
Otherwise, assign Z j to a separate group (with no nu-leaf node).
(2) SEM-UR:
For each latent variable H j , if there exists one child X i such that H j has no other children, or all other children of H j are also descendants of X i , assign H j to the same group as X i .
Otherwise, assign
H j to a separate group (with no observed variable).
Deﬁnition 5 (AOG equivalence class)
The AOG equivalence class of a linear SEM-ME (resp. SEMUR) is a set of models where the elements of this set all have the same mixing matrix (up to permutation and scaling) and same ancestral ordered groups.
For example, the two models represented by the graphs in Figure 2 have the same mixing matrix and ancestral ordered groups.
The elements of an AOG equivalence class possess the following property.
Proposition 1 Models in the same AOG equivalence class have the same causal order among the groups, but not necessarily all the same edges across the groups.
The AOG characterization is a reﬁned version of the original ordered grouping proposed in [29], where no variables in later groups cause variables in earlier groups.
Yet, the grouping introduced in that work is order dependent and hence in general not unique.
Moreover, it does not always characterize the extent of identiﬁability.
Our AOG characterization also uses a similar idea as the learning approach in [19]; however, that work allows for latent variables which have parents and in general does not recover the full structure.
Note that the AOG can be recovered based on the support of the mixing matrix under Assumption 2 (a) (or 3 (a) ).
Please refer to Appendix A for more details.
According to Deﬁnition 4, there is at most one nu-leaf node in each ancestral ordered group of a
SEM-ME.
Furthermore, each u-leaf node Z j is assigned to the ancestral ordered group of at most one of its parents:
Following the true causal order among nu-leaf nodes, only the last parent of Z j may have the same ancestor set.
Hence, if a group has more than one node, then there must be exactly one nu-leaf node, and the rest of the nodes are u-leaf nodes which are children of this node.
This concludes that the induced structure on each ancestral ordered group is a star graph.
Similar property holds for SEM-UR.
Deﬁne the center of the ancestral ordered group as the nu-leaf node (resp.
the observed node), or the u-leaf node (resp.
latent node) if the group is of size one.
The following result illustrates that ﬁxing the center of the ancestral ordered groups for SEM-ME, and ﬁxing the exogenous noise term of the center of the ancestral ordered groups as well as the choice of scaling and permutation of the columns of B for SEM-UR, leads to unique identiﬁcation of the models.
Proposition 2 (i) Models in an AOG equivalence class of a SEM-ME can be identiﬁed by the choice of the centers of the groups.
That is, for a given choice of the centers, there is only one corresponding model.
(ii) Models in an AOG equivalence class of a SEM-UR can be identiﬁed (up to the permutation and scaling of the columns of B in (4) ) by the choice of the exogenous noise terms associated to the centers of the groups.
That is, for a given assignment of the exogenous noise terms, all corresponding models have the same A in (4) , and have the same B up to permutation and scaling of the columns.
Equipped with Proposition 2, we are now ready to state our identiﬁability results for SEM-ME and
SEM-UR under Assumptions 2 (a) and 3 (a) ), respectively.
Theorem 2 Under Assumptions 1 and 2(a) (resp. 3(a)), the linear SEM-ME (resp.
SEM-UR) can be identiﬁed up to its AOG equivalence class.
Corollary 1 (i)
Denote the AOG of a linear SEM-ME (resp.
SEM-UR) as {G ( g ) } g ∈ [ g k ] , where g k is the number of ancestral ordered groups.
The size of the AOG equivalence class, described in
Theorem 2, is equal to (cid:81)
g ∈
[ g k ] |G ( g ) | .
(ii) Under Assumptions 1 and 2(a), a linear SEM-ME can be uniquely identiﬁed
if and only if no u-leaf node has precisely the same ancestors as any nu-leaf node.
(iii) Under Assumptions 1 and 3(a), a linear SEM-UR can be identiﬁed up to the permutation and scaling of the columns of B if and only if no latent variable has precisely the same descendants as any observed variable.
As stated in Theorem 2, in general, the edges across the groups cannot all be discovered and the centers of the groups (or their corresponding exogenous noise terms) are not identiﬁable.
This is due to the fact that when only Assumption 2 (a) holds, certain path cancellation or parameter proportionality in the model can occur.
This is illustrated in the following example.
Example 2 Consider the SEM-ME generated by causal graph shown in Figure 2(a).
The AOG is {{ Z 1 } , { Z 2 } , { Z 3 , Z 4 }} .
Because either Z 3 or Z 4 can be the center of the last group, the AOG equivalence class includes the ground-truth and the model represented by Figure 2(b).
Since both models have the same mixing matrix, an identiﬁcation algorithm merely based on the mixing matrix and Assumption 2(a) cannot distinguish these two models.
Note that not only can we not learn the direction of the edge in group { Z 3 , Z 4 } , but also the existence of some of the edge across groups, such as Z 1 → Z 4 , cannot be established in the ground-truth because Assumption 2(b) is violated.
4.3 Identiﬁability of the model up to equivalence classes
Under Assumptions 1 and 2 (resp. 3), the extent of identiﬁability of a SEM-ME (resp.
SEM-UR) can be characterized by the direct ordered grouping deﬁned in Deﬁnition 6 below.
We ﬁrst present two conditions that are used to implement the DOG.
We then give the formal deﬁnition of DOG.
In the following, we use Ch ( V ) to denote the set of children of variable V in the causal diagram.
Condition 1 (SEM-ME edge identiﬁability) For a given edge from an nu-leaf node V i to a u-leaf node Z l , at least one of the following two conditions is satisﬁed: (a) Pa ( Z l ) \ { V i } is not a subset of Pa ( V i ) .
That is, there exists another parent V j of Z l , which is not a parent of V i .
(b)
Pa ( Z l ) is not a subset of
∩ V k ∈ Ch
( V i ) \{ Z l } Pa ( V k ) .
That is, there exists a child V k of V
i and a parent V j of Z l such that V j is not a parent of V k .
Condition 2 (SEM-UR edge identiﬁability)
For a given edge from a latent variable H l to an observed variable X i , at least one of the following two conditions is satisﬁed: (a) Ch ( H l ) \ { X i } is not a subset of Ch ( X i ) .
That is, there exists another observed child X j of H l , which is not a child of
X i .
(b) Ch ( H l ) is not a subset of ∩ X k ∈ Pa ( X i ) \{ H l } Ch
( X k ) .
That is, there exists an observed (or latent) parent X k (or H k ) of X i and a child
X j of H l such that X j is not a child of X k (or H k ).
Figure 3(b) demonstrates an example of a graph structure which satisﬁes Assumption 2 while containing an edge which violates Condition 1.
Condition 1 is an equivalent formulation of the conditions for unique identiﬁability derived in [29].
In that work, Zhang et al. proved that a linear
SEM-ME can be uniquely recovered if Condition 1 is satisﬁed for every edge from an nu-leaf node to a u-leaf node.
They also conjectured that this condition is necessary for identiﬁability.
We prove their conjecture here under a slightly different faithfulness assumption.
Furthermore, we demonstrate that the conditions can be used to characterize an equivalence class of the models which characterizes the extent of identiﬁability under Assumptions 1 and 2; the same assertion holds for linear SEM-UR.
This characterization is done based on the notion of direct ordered grouping, deﬁned as follows.
Deﬁnition 6 (Direct ordered grouping (DOG))
The DOG of a linear SEM-ME (resp.
SEM-UR) is a partition of Z ∪ Y (resp.
H ∪ X ) into distinct sets.
This partition is described as follows:
(1) Assign each nu-leaf node (resp.
observed node) to a distinct ordered group.
(2) SEM-ME:
For each u-leaf node Z j ∈ Z , if there exists one parent V i such that the edge from V i to Z j violates Condition 1, assign Z j to the
same ordered group as V i .
Otherwise, assign Z j to a separate ordered group (with no nu-leaf node).
(2) SEM-UR:
For each latent variable H j , if there exists one child X i such that the edge from H j to X i violates Condition 2, assign H j to the same ordered group as X i .
Otherwise, assign H j to a separate ordered group (with no observed variable).
Deﬁnition 7 (DOG equivalence class)
The DOG equivalence class of a linear SEM-ME (resp. SEMUR) is a set of models where the elements of this set all have the same mixing matrix (up to permutation and scaling) and same direct ordered groups.
For example, the two models represented by the graphs in Figure 2 have the same mixing matrix, but different direct ordered groups.
We have the following property regarding elements of an DOG equivalence class.
Proposition 3 Models in the same DOG equivalence class have the same edges across the groups.
The counterpart of Proposition 2 holds for the case of direct ordered groups as well and we avoid repeating it.
Based on that counterpart, we have the following result regarding identiﬁability of linear
SEM-ME and SEM-UR under Assumptions 2 and 3, respectively.
Theorem 3 Under Assumptions 1 and 2 (resp. 3), the linear SEM-ME (resp.
SEM-UR) can be identiﬁed up to its DOG equivalence class.
Theorem 3 states that we can learn the structure among the groups, but the center of the group (or the noise term corresponding to it) will remain unidentiﬁed.
Unlike the case of the AOG equivalence class, all the edges across the groups will be identiﬁed.
That is, the choice of center does not change the edges across groups in DOG equivalence class, while it may change the edges in AOG equivalence class.
Further, the sizes of the direct ordered groups are smaller than the sizes of the ancestral ordered groups.
As an example, for the model shown in Figure 2(b), all direct ordered groups are of size one.
Corollary 2 (i)
The counterpart of Corollary 1(i) is true for the case of DOG as well and we do not repeat it here.
(ii) Under Assumptions 1 and 2, a SEM-ME can be uniquely identiﬁed
if and only if for every edge from an nu-leaf node to a u-leaf node, Condition 1 is satisﬁed.
(iii) Under
Assumptions 1 and 3, a SEM-UR can be identiﬁed up to the permutation and scaling of the columns of
B if and only if for every edge from a latent variable to an observed variable, Condition 2 is satisﬁed.
Identiﬁability of the structure of the SEM-UR.
As shown in Theorem 3, for a SEM-UR, the only undetermined part in the DOG equivalence class pertains to the assignment of the exogenous noises and coefﬁcients, but the structure is the same.
Consequently, if only the identiﬁcation of the structure without weights is of interest, Assumptions 1 and 3 are sufﬁcient.
See Figure 3(a) for an example of two DOG equivalent SEM-URs.
This identiﬁability result is summarized in the following corollary.
Corollary 3 Under Assumptions 1 and 3, the structure of a SEM-UR can be uniquely recovered.
4.4 Recovery algorithm
X X
H
N X N X
N H ba (a) Equivalence class of a SEM-UR
X X
H
N H
N X
N X− ba
+ b
N Z N Z
Z 2 Z
Z b a (b) Equivalence class of a SEM-ME
N Z
N Z 2 Z
Z
Z − b a + b
We note that our faithfulness assumption implies that the ground-truth model has strictly fewer edges than any model that has the same mixing matrix and satisﬁes conventional faithfulness (i.e., in the AOG equivalence class) but does not belong to the DOG equivalence class.
This property can be leveraged to recover the groundtruth model or a member of its DOG equivalence class.
Proposition 4 Suppose a SEM-ME (resp.
SEM-UR) satisﬁes Assumptions 1 and 2 (resp. 3).
Any model that belongs to the same AOG equivalence class but does not belong to the same DOG equivalence class has strictly more edges than any member in the DOG equivalence class.
The steps of our recovery algorithm are as follows: (1)
Recover the mixing matrix W ME (resp.
W UR ) from observational data.
(2) Return the AOG of the true model by comparing the support across different rows/columns in W ME (resp.
W UR ).
See
Appendix A.2 for details and pseudo-code for AOG recovery.
For all possible choices of the center (resp.
noise term associated to the center) of each ancestral ordered group, ﬁnd a choice that leads to the graph with fewest number of edges in the recovery output (see the proof of Lemma 1).
Simulations
We evaluated the performance of our recovery algorithm on randomly generated linear SEM-MEs and
SEM-URs with different number of variables.
4 We considered two cases: (1) when a noisy version of the mixing matrix is given, where the noise is Gaussian with different choices of variance denoted by d 2 , and (2) when synthetic data comes from a linear generating model with non-Gaussian noises with different sample sizes, and the mixing matrix needs to be estimated.
We used the Reconstruction
ICA algorithm [14] as our overcomplete ICA method.
For Case (1), we compared our recovery method (which we refer to as DOG) with an approach based on AOG equivalence class (similar to the learning method in [19]).
In addition to the AOG method, for Case (2), we compared our method with ICA-LiNGAM [21].
Note that the ICA-LiNGAM method is not designed for systems with latent variables, but beneﬁts from strong performance for recovering the mixing matrix in causally sufﬁcient settings.
The goal of this comparison was to demonstrate the necessity of using methods designed speciﬁcally to handle latent variables.
We compared the recovery of the adjacency matrix to the ground-truth, where we used normalized structural Hamming distance (SHD/Edge) and F1 score as our performance metrics.
Our results for
Cases (1) and (2) are shown in top and bottom two rows in Figure 4, respectively.
As seen in these ﬁgures, our recovery algorithm outperforms recovery algorithm based on AOG.
In particular, our method recovers the structure in SEM-UR model, and can recover part of the structure in SEM-ME that is shared among DOG equivalence classes.
Moreover, both methods outperform ICA-LiNGAM.
Please refer to Appendix E for additional results and analysis.
Conclusion and Discussion
In real-world applications, we do not observe the exact value of all relevant variables; the measurements of some variables are prone to errors, or some other variables cannot be observed altogether.
For example, in neuroscience and genomics, measured brain signals obtained by functional magnetic resonance imaging (fMRI) or the measured gene expression using RNA sequencing usually contain errors through the measurement process [18, 29].
Other examples include responses to psychometric questionnaires where the questions represent noisy views of various traits [1, 20], and returns in stock market, where they may be confounded by several unmeasured economic and political factors [1].
In this work, we considered the problem of causal discovery in setups with such challenges, particularly under two settings: (1) in the presence of measurement error, and (2) with unobserved parentless causes.
Unlike previous work, our proposed SEM-ME allows for applications containing a mixture of variables measured exactly (without error) or measured with error.
We demonstrated a mapping between these two models that preserves their mixing matrix.
Based on this mapping, we derived identiﬁability results for both models under different faithfulness assumptions, and proposed structure learning algorithms.
Our results further implied conditions for unique identiﬁability of the structure in SEM-UR.
These conditions do not pose any restrictions on the graphical model, and hence signiﬁcantly relax the existing graphical conditions in the literature [19, 1, 25, 4].
Our results have several implications in the literature of causal discovery including the following:
The mapping proposed in our work between linear SEM-ME and SEM-UR allows us to translate identiﬁability results for one model to the other.
We note that linear SEM-UR has been widely studied in the literature, while only a few works considered linear SEM-ME.
Therefore, an important implication of our work is that the introduced mapping can be utilized to ﬁll the gaps for the less studied model.
Another important implication of our result is that it can be used for evaluating new algorithms: We showed that under two different faithfulness assumptions, the model can only be identiﬁed up to AOG and DOG equivalence classes.
This result provides the extent of identiﬁability, and can serve as a basis for characterizing theoretical guarantees such as consistency of new algorithms.
Finally, we showed in Proposition 4 that under our faithfulness assumptions, the true generating model is always sparser than any other model in the same AOG equivalence class (of the ground-truth) but does not belong to the DOG equivalence class.
This serves as a ground for positing sparsity assumptions that appear frequently in the literature without rigorous justiﬁcation.
Similar to other causal discovery methods which are based on the mixing matrix, the performance of our proposed recovery algorithms relies on the accuracy of the utilized mixing matrix estimation approach.
Hence, the recovered structure should be interpreted with caution if the mixing matrix estimation approach is unreliable.
Devising more accurate approaches for estimating the mixing matrix, as well as extending the method proposed in this work to non-linear models are important directions of future research.
Acknowledgments and Disclosure of Funding
Negar Kiyavash’s research was in part supported by the Swiss National Science Foundation under
NCCR Automation, grant agreement 51NF40_180545 and Swiss SNF project 200021_204355 /1.
Kun Zhang was partially supported by NIH under Contract R01HL159805, by the NSF-Convergence
Accelerator Track-D award #2134901, by a grant from Apple Inc., and by a grant from KDDI
Research Inc.
Ilya Shpitser was supported by grants ONR N00014-21-1-2820, NSF 2040804, NSF
CAREER 1942239 and NIH R01 AI127271-01A1.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Please refer to Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[No] We believe that causal structure learning from observational data avoids potential negative societal impacts in causal discovery that might be associated with conducting randomized experiments.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Assumptions 1 and 2 (resp. 3) are our identiﬁability assumptions for linear SEM-ME (resp.
SEM-UR).
(b) Did you include complete proofs of all theoretical results?
[Yes] Please refer to Appendix C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please refer to Appendix E for instructions.
Our code is available at: https://github.com/Yuqin-Yang/SEM-ME-UR .
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] This is explained in detail in Appendix E. (c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Please refer to Appendix E.4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Simulations are conducted using the CPU of a MacBook laptop.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We cited the original paper of the baseline algorithm.
(b) Did you mention the license of the assets?
[N/A] Our simulations are conducted on synthetic data only.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Causal Discovery in Linear Latent Variable Models
Subject to
Measurement
Error
Supplementary Material
A Further discussion on ancestral ordered grouping
First, we compare our ancestral ordered grouping (AOG) in Deﬁnition 4 with the ordered grouping proposed in [29].
Next, we compare our AOG to the equivalence classes for identiﬁability of a linear SEM with latent variables under conventional faithfulness, proposed in [19].
Finally, we present the details of our AOG recovery algorithm.
A.1 Comparison of AOG to previous work
In the following, we present an equivalent deﬁnition of the ordered grouping proposed in [29], which is closer in spirit to our AOG deﬁnition.
As we mentioned earlier, the model proposed in [29] assumes that all the underlying variables are measured with error (unobserved), i.e., Y in Equation (1) is an empty set.
Subsequently, a u-leaf node is an unobserved underlying variable with no children in Z and an nu-leaf node is an unobserved underlying variable with at least one child in Z .
Deﬁnition 8 (Ordered group decomposition (OGD) [29])
The ordered group decomposition of a linear SEM- ME is a partition of Z into distinct sets.
This partition is described as follows:
(1) Select a causal order k among nu-leaf nodes in Z that is consistent with the generating model.
Assign each nu-leaf node to a distinct ordered group.
For each u-leaf node Z j ∈ Z , assign Z j to the same group as its parent with the largest index in k .
The ordered grouping by [29] depends on the selected causal order and hence might not be unique, since there can be more than one causal order consistent with the generating model.
Subsequently, this ordered grouping does not permit characterizing the extent of identiﬁability under conventional faithfulness alone, i.e., the equivalence classes of the generating model in Theorem 2.
In contrast, our AOG is a more reﬁned grouping than Deﬁnition 8 and returns a unique partition that does not depend on the selection of the causal order.
Further, our AOG characterizes the extent of identiﬁability under conventional faithfulness (cf.
Theorem 2).
The following example illustrates the difference between the ordered grouping in [29] and our AOG.
Example 3 Consider a collider structure over three underlying variables in a linear SEM-ME: Z 1 → Z 3 ← Z 2 , where Z 3 is a u-leaf node.
There are two possible causal orders that correspond to this structure, namely (1,2,3) and (1,3,2).
Hence, based on the ordered grouping deﬁnition of [29], there are two possible corresponding partitions, namely, {{ Z 1 } , { Z 2 , Z 3 }} and {{ Z 2 } , { Z 1 , Z 3 }} .
However, following our deﬁnition of AOG, i.e., Deﬁnition 4, the AOG is {{ Z 1 } , { Z 2 } , { Z 3 }} , which is a reﬁned partition of either of the two partitions by [29].
Further, the AOG of the model can be recovered based on (the support of) the mixing matrix W ME : Z 3 includes two noise terms, while Z 1 and Z 2 each includes one.
Next, we show the connection between our AOG and the observed descendant sets in [19] proposed for a linear SEM with latent variables.
The “observed descendant set” of a (latent or observed) variable V i is deﬁned as the set of all observed descendants of V i , including V i itself if it is observed.
The following proposition states the equivalence of the two notions in a linear SEM-UR.
Proposition
In a linear SEM-UR, two variables are in the same ancestral ordered group if and only if they have the same observed descendant set.
The authors of [19] showed that using “observed descendant sets”, the causal order among the observed variables can be identiﬁed under conventional faithfulness assumption (i.e., Assumption 3 (a) ).
Further, the total causal effects among observed variables can be estimated.
However, the model in [19] allows for latent variables which have parents (i.e., not roots) and in general, the full structure cannot be identiﬁed.
In contrast, our AOG equivalence class characterization (Theorem 2) extends the result in [19].
Speciﬁcally, we can recover the causal order among the ancestral ordered groups, where each ancestral ordered group has at most one observed variable.
Moreover, we can estimate the exact structure and the coefﬁcients of every model in the AOG equivalence class.
A.2 AOG recovery algorithm
Based on Assumption 1, we can recover W ME (resp.
W UR ) from the observed data.
Algorithm 1 shows how to recover the AOG in Deﬁnition 4 from W ME .
The Algorithm ﬁrst randomly chooses a row in W ME with one
Input: Recovered mixing matrix W ME .
1 Calculate the number of non-zero entries in each row of W ME .
Denote the vector of these numbers as n , where each entry in n corresponds to a row in W ME .
2 Initialize ˜ W = W ME .
3 while ˜ W is not empty
do
Find a row in ˜ W that contains only one non-zero entry, and has the smallest corresponding value in n .
If there are multiple such rows, randomly select one.
Denote the selected row as w , and its corresponding value in n as n 0 .
Consider the rows in ˜ W with the same support as w (including w itself).
Denote the set of variables that correspond to these rows as Z W .
Assign all variables in Z W , with the same corresponding value as n 0 in n , to a single ordered group.
Assign each of the remaining variables in Z W to a separate ordered group.
Remove from ˜ W the rows corresponding to the variables in Z W , and the column containing the corresponding non-zero entries in these rows.
Output: AOG of the SEM-ME.
non-zero entry and ﬁnds all other rows with the same support, and puts all the corresponding variables in a single ancestral ordered group.
Next, it removes the selected rows and the column that contains the corresponding non-zero entry.
In the remaining matrix, say ˜ W , the Algorithm again chooses a row w with one non-zero entry, but has the smallest number of non-zero entries in W ME .
Then, it selects all other rows in ˜ W with the same support and the same number of non-zero entries in W ME , as w .
The rows in ˜ W with exactly one non-zero entry and having the same (fewest) number of non-zero entries in W ME correspond to an nu-leaf node and its direct children (with the same support in W ME ), and hence they go together into the same group.
The remaining rows with more non-zero entries in W ME are u-leaf nodes that go to separate distinct groups.
Finally, this procedure is repeated until all variables are assigned to ancestral ordered groups.
Since Algorithm 1 is based on the mixing matrix, it follows from the constructed mapping in Theorem 1 that the same Algorithm can be applied to recover the AOG of a linear SEM-UR, by replacing “rows” with “columns” and vice versa, in the Algorithm.
B SEM-ME and SEM-UR faithfulness assumptions
We ﬁrst present the formal statement of SEM-ME and SEM-UR faithfulness assumptions in Appendix B.1.
We also provide examples of violations of faithfulness.
In Appendix B.2, we present an equivalent representation of
SEM-ME and SEM-UR faithfulness, using the mixing matrix.
Next, in Appendix B.3, we show that violation of faithfulness assumptions is a measure-zero event.
Finally, we present the relation between bottleneck faithfulness [1] and SEM-UR faithfulness assumptions in Appendix B.4.
B.1 Formal statement of faithfulness assumption
For a variable V i , we deﬁne the ancestor set, An ( V i ) (resp.
the descendant set, De ( V i ) )
as the sets of variables that have directed paths to V i (resp.
from V i ); both excluding V i itself.
Note that in the literature, some works use the convention V
i ∈
An ( V i ) and V i ∈
De ( V i ) .
We do not use this convention here.
For SEM-ME, deﬁne the possible parent set of a variable V
i ∈
Z ∪ Y , as the union of the ancestor set of V i , An ( V i ) , and the set of u-leaf nodes whose parents are subsets of An ( V i ) .
For any set V ⊆
Z ∪ Y , deﬁne TE ( V , V i ) as the vector whose entries represent the total causal effects (i.e., sum of products of path coefﬁcients) of the elements of V on V i .
We deﬁne the total causal effect of a variable on itself to be 1.
Similarly, for
SEM-UR, deﬁne the possible children set of V i ∈ H ∪ X as the union of the descendant set of V i , De ( V i ) , and the set of latent variables whose children sets are subsets of De ( V i ) .
For any set V ⊆ H ∪ X , deﬁne TC
( V i , V ) as the vector whose entries represent the total causal effects of V i on the elements of V .
Assumption 2 (SEM-ME faithfulness)
(a) The total causal effect of an underlying variable
V
i ∈
Z ∪ Y on its descendant V j ∈ Z ∪ Y is not equal to zero.
(b) For each variable V
i ∈
Z ∪ Y ,
b1) TE
V i ) , V i ) is linearly independent of any k
V i
| vectors in { TE
V
a possible parent
when k
and
node V l that corresponds to one
k vectors, Pa
V
) ⊆
Pa
V
b2)
a
node, then for each
Z Z
Z Z
Z
Z 6 b d  a c (a)
Z Z
Z Z
Z
Z 6 a +c +b d  − a − c (b)  
Z 1 Z
Z 3 Z
Z 5 Z  =           a + 1 c +  b + d b d   (cid:34) N Z 1 N
Z 2 N Z(cid:35)
W ME (c)
V j of V
i
TE ( An ( V
) is linearly independent of any k ≤ | Pa
∪
| − 1 vectors
Pa and for each latent variable H l that corresponds to one of these k vectors, Ch ( H l ) ⊆
Ch ( V i ) ; (b2) If V i is a latent variable, then for each child X j of V i , TC
( V i , De ( V i ) \{ X j } ) is linearly independent of any k ≤ | Ch
( X j ) ∪
Ch
( V i ) |− 1 vectors in { TC ( V, De
( V i ) \{ X j } ) : V is a possible child of X j } , except for the case when k = | Ch
( X j ) ∪
Ch
( V i )
|−and for each latent variable H l that corresponds to one of these k vectors, Ch ( H l ) ⊆
Ch ( X j ) ∪ Ch
( V i ) .
We explain the aforementioned faithfulness assumptions using the following examples, which investigate various cases of violation of SEM-ME faithfulness, and show that our identiﬁability result in Theorem 3 may no longer hold under such violation.
In Appendix B.3, we show that violation of SEM-ME faithfulness is a measure-zero event.
Example 4 Consider the linear SEM-ME generated by the graph in Figure 2(a).
The mixing matrix is given in Figure 2.
We have TE ( { Z 1 , Z 2 } , Z 4 ) =
[ a ( b + c ); b + c ] =
( b + c ) TE ( { Z 1 , Z 2 } , Z 2 ) .
This implies that TE ( An ( Z 4 ) \ { Z 3 } , Z 4 ) is linearly dependent on one vector TE ( An ( Z 4 ) \ { Z 3 } , Z 2 ) where Z 2 is a possible parent of Z 3 .
Hence the model violates Assumption 2(b2).
According to Deﬁnition 6, the DOG of the ground truth model (Figure 2(a)) is {{ Z 1 } , { Z 2 } , { Z 3 , Z 4 }} .
Because either Z 3 or Z 4 can be the center of the last group, the DOG equivalence class should include both the ground- truth and the model represented by Figure 2(b).
However, due to violation of Assumption 2(b2), the model represented by Figure 2(b), does not belong to the same DOG equivalence class as the ground-truth, and hence can be distinguished from the ground truth.
Therefore, our identiﬁability result in Theorem 3 does not hold because of violation of faithfulness.
Example 5 Consider the linear SEM-ME generated by the graph in Figure 5(a).
The mixing matrix is given in Figure 5(c).
We have TE ( An ( Z 6 ) , Z 6 ) =
[ b + d ; b ; d ] , TE ( An ( Z 6 ) , Z 4 ) =
[ a +1; 1; 0] , and TE ( An ( Z 6 ) , Z 5 ) =
[ c + 1; 0; 1] .
If ab + cd = 0 , then TE ( An ( Z 6 ) , Z 6 ) can be written as b TE ( An ( Z 6 ) , Z 4 ) + d TE ( An ( Z 6 ) , Z 5 ) .
That is, TE ( An ( Z 6 ) , Z 6 ) can be written as the linear combination of two vectors; however, the variables corresponding to these two vectors, i.e., Z 4 , Z 5 , are both u-leaf nodes and have a parent Z 1 that is not a parent of Z 6 .
Therefore the model violates Assumption 2(b1).
Note that ab + cd = 0 is a measure-zero event if all parameters are randomly and independently drawn from continuous distributions.
-ME
Figure 5( is
.e result in Theorem
B.2 Equivalent matrix form for faithfulness assumptions
In this section we present an equivalent matrix representation of Assumptions 2 (b) and 3 (b) .
For each variable V i in SEM-ME, deﬁne Supp ( V i ) as the support of the row in W ME corresponding to V i :
Supp ( V i ) =
{ N V j | j : [ W ME ] ij (cid:54) = 0 } .
For each variable V i in SEM-UR, deﬁne Supp ( N V i ) as the support of the column in W UR corresponding to N V i : Supp ( N V i )
= { V j | j : [ W UR ] ji (cid:54) = 0 } .
Under this formulation, a variable V j in a SEM-ME is a possible parent of V i
if and only if Supp ( V j ) is a subset of Supp ( V i ) \ { N V i } , i.e., Supp ( V i ) excluding the exogenous noise term N V i
if V i is an nu-leaf node [28].
A variable V j in a SEM-UR is a possible child of V i
if and only if Supp ( N V j ) is a subset of Supp ( N V
i ) \ { V i } , i.e., Supp ( V i ) excluding V
i itself if V i is observed.
Proposition 6 Assumptions 2(b) and 3(b) have the following equivalent matrix form:
Assumption 2(b) :
For a variable V i ∈
Z ∪ Y , (b1) denote W MEi as the submatrix of W ME with rows corresponding to the possible parents of V i , and columns corresponding to Supp ( V i ) \ { N V i } .
Then
W ME [ V i , Supp ( V i ) \ { N V i } ] cannot be written as any linear combination of k ≤ | Pa ( V i ) | rows in W MEi , except for the case when k
= | Pa ( V i ) | and for each u-leaf node V l that corresponds to one of these k rows,
Pa ( V l ) ⊆
Pa ( V i ) .
(b2)
If V i is a u-leaf node, then for each parent V j of V i , denote W MEij as the submatrix of W ME with rows corresponding to the possible parents of V j , and columns corresponding to Supp ( V i ) \ { N V j } .
Then
W ME [ V i , Supp ( V i ) \ { N V j } ] cannot be written as any linear combination of k ≤ | Pa ( V i ) ∪ Pa ( V j ) | −rows in W MEij , except for the case when k
= | Pa ( V i ) ∪ Pa ( V j ) | − 1 and for each u-leaf node V l that corresponds to one of these k rows, Pa ( V l ) ⊆
Pa ( V i ) ∪ Pa ( V j ) .
Assumption 3(b) :
For a variable V i ∈
H ∪ X , (b1) denote W URi as the submatrix of W UR with columns corresponding to the exogenous noise terms of the possible children of V i , and rows corresponding to
Supp ( N V i ) \ { V i } .
Then W UR
[ Supp ( N V i ) \ { V i } , N V i ] cannot be written as any linear combination of k ≤ | Ch
( V i ) | columns in W URi , except for the case when k = |
Ch
( V i ) | and for each latent variable H l whose exogenous noise term corresponds to one of these k columns, Ch ( H l ) ⊆
Ch ( V i ) .
(b2)
If V i is a latent variable, then for each child X j of V i , denote W URji as the submatrix of W UR with columns corresponding to the exogenous noise terms of the possible children of X j , and rows corresponding to Supp ( N V
i ) \
{ X j } .
Then W UR
[ Supp ( N V i ) \ { X j } , N V i ] cannot be written as any linear combination of k ≤ | Ch
( V i ) |− 1 columns in W URji , except for the case when k = | Ch ( X j ) ∪
Ch
( V i ) |− 1 and for each latent variable H l whose exogenous noise term corresponds to one of these k columns, Ch ( H l ) ⊆
Ch ( X j ) ∪ Ch
( V i ) .
B.3 Violation of our faithfulness assumptions is a measure-zero event
In the following, we show that Assumption 2 is violated with probability zero if all model coefﬁcients are randomly and independently drawn from continuous distributions.
Same argument holds for Assumption 3, and hence the proof is omitted.
Note that Assumption 2 (a) is identical to conventional faithfulness in linear causal models, the violation of which is known to be a measure-zero event [15, 24].
Thus, it sufﬁces to show that violation of Assumption 2 (b) is a measure-zero event.
We do so for both parts of Assumption 2 (b) .
Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) Assumption 2 (b1) : Let T i denote the set of vectors { TE ( An ( V i ) , V ) : V is a possible parent of V i } .
Recall that the total effects of An ( V i ) on V i , i.e., TE ( An ( V i ) , V i ) , can be written as a linear combination of the total effects of An ( V i ) on parents of V i , i.e., P i = { TE ( An ( V
[Equation]
[Equation]
Further, A i = { TE ( An ( V i ) , V ) : V ∈
An ( V i ) } constitutes a basis for span( T i ) :
Vectors in A i are linearly independent (because each ancestor has its exogenous noise term) and every vector in T i (which corresponds to a possible parent of V i ) can be written as a linear combination of vectors in A i .
Note that P i ⊆
A i .
Therefore, if Assumption 2 (b1) is violated, then TE ( An ( V i ) , V i ) can either be (i) linearly dependent on k < | Pa
( V i ) | vectors in T i , or (ii) linearly dependent on | Pa ( V i )
| vectors in T i and at least one of these vectors does not belong to span( P i ) .
Both of these events are of measure zero, when all { a ij | j : V j ∈ Pa ( V i ) } in Equation (8) are randomly and independently selected from continuous distributions.
This follows because if we represent
TE ( An ( V i ) , V i ) and all vectors in T i as linear combinations of the vectors in A i , then case (i) corresponds to a solution of a linear system with k variables and at least | Pa ( V i ) | equations (constraints), and case (ii) corresponds to a solution of a linear system with | Pa ( V i ) | variables and at least | Pa ( V i ) | + 1 equations.
Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) Assumption 2 (b2) :
Similarly, we ﬁrst show that for any parent V j of V i , TE ( An ( V i ) \ { V j } , V i ) can be written as the linear combination of | Pa ( V i ) ∪ Pa ( V j ) | − 1 linearly independent vectors in T i with probability one.
To show this, we ﬁrst note that Equation (8) holds for any subvector of TE ( An ( V i ) , V i ) .
Therefore,
[Equation]
Further, similar to Equation (8), TE ( An ( V j ) , V j ) can also be written as a linear combination of the vectors in { TE ( An ( V j ) , V k ) :
V k ∈
Pa ( V j ) } .
Since the total causal effect from any variable in An ( V i ) \ ( An ( V j ) ∪{ V j } ) to V j or any parent of V j is zero, we have
[Equation]
Combining these two equations, we have
[Equation]
Note that the right hand side of Equation (9) includes | Pa ( V i ) ∪ Pa ( V j ) | − 1 vectors, and for each vector that is included in both summations, the coefﬁcients cancel out each other with probability zero.
Let T ji , A ji , P ji denote the set of vectors { TE ( An ( V i ) \ { V j } , V ) : V is a possible parent of V j } , { TE ( An ( V i ) \ { V j } , V ) :
V ∈
An ( V j ) } , { TE ( An ( V i ) \ { V j } , V ) : V ∈ Pa ( V j ) ∪ Pa ( V i ) \ { V j }} , re- spectively.
Similarly, A ji constitutes a basis of span T ji .
If Assumption 2 (b2) is violated, then all parents of V i are ancestors of V j , otherwise TE ( An ( V i ) \ { V j } , V i ) cannot belong to span( T ji ) .
This means that all vectors on the right hand side of Equation (9), i.e., P ji , belong to A ji .
Further, TE ( An ( V i ) \ { V j } , V i ) can either be (i) linearly dependent on k < | Pa
( V i ) ∪ Pa ( V j )
| − 1 =
|P ji | vectors in T ji , or (ii) linearly dependent on | Pa ( V i ) ∪ Pa ( V j ) | − 1 vectors in T ji and at least one of these vectors does not belong to span( P ji ) .
Both of these events are measure zero, when all { a ij (cid:48) | j (cid:48) :
V j (cid:48)
∈
Pa ( V i ) , j (cid:48) (cid:54) =
V j } ∪ { a jk |
k : V k ∈ Pa ( V j ) } in
Equation (9) are randomly and independently selected from continuous distributions.
B.4 Bottleneck faithfulness implies SEM-UR faithfulness
We show that our SEM-UR faithfulness is weaker than Bottleneck faithfulness in [1], in the sense that more models satisfy SEM-UR faithfulness.
Bottleneck faithfulness is deﬁned as follows:
Condition 3 (Bottleneck faithfulness) Deﬁne a bottleneck B from J to K as a set of variables where any directed path from j ∈ J to k ∈ K includes some variables in B , and a minimal bottleneck B from J to K as the bottleneck with the smallest size.
Then for every J ⊆ H ∪ X , K ⊆ X , if B is a minimal bottleneck from J to K , then Rank( W JK ) =
| B | , where W JK is the submatrix of W UR representing the total causal effect from variables in J to variables in K .
[Table] [Table]
has
[Table]
in [1] that Condition 3 implies As- sumption 3 (a) .
For Assumption 3 (b) , it sufﬁces to show that if either Assumption 3 (b1) or (b2) is violated, then
Condition 3 is violated.
Assumption 3(b1)
: Suppose V i violates Assumption (b1) .
Then TC ( V i , De ( V i )) is either: (i) linearly dependent on k < | Ch
( V i )
| vectors in the set T i
=
TC ( V, De ( V i ))
: V is a possible child of V i } , or (ii) linearly dependent on k = |
Ch ( V i ) | vectors in T i and the children set of at least one latent variable H j corresponding to these vectors is not a subset of | Ch ( V i ) | .
Without loss of generality, suppose k is minimal, i.e,
TC
( V i , De ( V i )) is linearly independent of any l vectors in T
i for any l < k .
Denote the set of possible children of V i corresponding to these vectors as P .
Consider the set J = P ∪ { V i } , and K =
Des ( V i ) .
6 Then Rank( W JK ) =
|P| =
k , because (i) W V i K can be written as a linear combination of the columns in W PK , and (ii) columns in W PK are linearly independent (otherwise k is not minimal).
In the following we show that any bottleneck from J to K is at least of size k + 1 .
First, any bottleneck from P to K is at least of size |P| =
k .
This is because if there exists a bottleneck of size k − 1 , then according to
Proposition 1 of [1], Rank( W JK ) ≤ k − 1 .
This contradicts the minimality of k .
Next, for any bottleneck B from P to K with size k : (i)
If k < | Ch ( V i ) | , then B will cover at most k children of V i .
(ii)
If k = | Ch ( V i ) | .
Note that there exists at least one child of H j , denoted by X k , that is not a child of V i .
Therefore, to block the edge H j → X k , B must include at least one variable between H j and X k .
This implies that B includes at most | Ch
( V i ) | − 1 children of V i .
In either case, there is at least one directed edge from V i to one of its children that cannot be blocked by B .
Therefore, the minimal bottleneck from J to K is at least of size k + 1 , which violates bottleneck faithfulness condition.
Assumption 3(b2) :
Similarly, suppose V i and its child X j violates Assumption (b2) .
Then TC ( V i , De ( V i ) \{ X j } ) is either: (i) linearly dependent on k < | Ch
( V i ) ∪ Ch ( X j ) |− 1 vectors in the set T ji
= { TC ( V, De
( V i ) \{ X j } ) :
V is a possible child of X j } , or (ii) linearly dependent on k = |
Ch ( V i ) ∪ Ch ( X j ) | − 1 vectors in T ji , and the children set of at least one latent variable H k corresponding to these vectors is not a subset of Ch ( V i ) ∪ Ch ( X j ) .
Suppose k is minimal, and denote the set of possible children corresponding to these k vectors as P .
Consider the set J = P ∪ { V i } and K =
Des ( V i ) \ { X j } .
Then Rank( W JK ) =
|P| =
k .
However, any bottleneck from J to K is at least of size k + 1 .
This is because: (i) Any bottleneck from P to K is at least of size |P| =
k ; (ii) For any bottleneck B from P to K with size k , there are | Ch
( V i ) ∪ Ch ( X j ) \ { X j
}| variables in K that is a child of either V i or X j .
Therefore there is at least one directed edge from V i or X j to one of their children (excluding X j ) that cannot be blocked by B .
(If | B | = | Ch ( V i ) ∪ Ch ( X j ) | − 1 , then there is at least one variable in B that blocks the edge from H k to its child that does not belong to Ch ( V i ) ∪ Ch ( X j ) .)
Therefore bottleneck faithfulness condition is violated.
“SEM-UR faithfulness does not imply bottleneck faithfulness SEM-UR faithfulness does not imply bottleneck faithfulness
[Table]
SEM
[Table]
does not imply bottleneck [Table]
: In the following we will provide an example which satisﬁes Assumption 3 but violates Condition 3.
This example is introduced in [1].
Consider the following linear SEM-UR involving three latent variables H
=
[ H 1 , H 2 , H 3 ] and four observed variables X
=
[ X 1 , X 2 , X 3 , X 4 ] :
H = N H ,
X = [Equation] 
[Equation]
It has been shown in [1] that the minimal bottleneck from H =
{ H 1 , H 2 , H 3 } to X = { X 1 , X 2 , X 3 , X 4 } is H with |H| =
3 , while Rank( W HX )
= 2 .
Therefore the model violates Condition 3.
On the contrary, the model satisﬁes Assumption 3.
This is because the possible children set of each observed variable is empty set (meaning that (b2) is satisﬁed), and the possible children set of each latent variable only includes its children (meaning that (b1) is satisﬁed).
C Proofs of the main results C.1 Proof of Theorem
We ﬁrst observe that in a SEM-ME, whether a non-leaf underlying variable V i is observed (i.e., V i ∈ Y ) or measured with error (i.e., V i ∈ Z ) does not change the relevant mixing matrix W ME .
Speciﬁcally, in the overall mixing matrix W , cf.
Equation (6), the exogenous noise terms associated with non-leaf underlying variables correspond to columns with at least two non-zero entries.
This follows because a non-leaf variable inﬂuences at least two other variables (its own measurement and its child(ren) in Z ∪ Y ).
On the other hand, the measurement error terms associated with the non-leaf underlying variables correspond to columns with only one non-zero entry, since the measurement error term only inﬂuences one variable (i.e., the corresponding measurement in U ).
Thus, by removing the identity matrix part of W , the remaining part, i.e., W ME does not change if a non-leaf underlying variable is observed or measured with error.
For a linear SEM-ME, M (cid:48) , without loss of generality, suppose that all non-leaf underlying variables are observed.
That is, all nu-leaf nodes belong to Y , and u-leaf nodes belong to Z .
Denote the dimension of Y (i.e., the number of the nu-leaf nodes) as p n .
Let us ﬁrst permute the variables in Y in the reversed causal order, and denote the result of this permutation by Y ∗ .
Thus,
[Equation]
where P is a p n × p n permutation matrix, and N Y ∗
=
P N Y is the permuted noise vector corresponding to Y ∗ .
Further, C ∗ =
PCP (cid:62) is a row and column permuted version of C , such that C ∗ is strictly upper triangular.
Therefore, the mixing matrix W ME corresponding to M (cid:48) can be written as
[Equation]
P , where W ME ∗ = (cid:20)
[Equation]
Next, we derive the mixing matrix of the corresponding SEM-UR under the mapping φ .
Suppose the correspond- ing SEM-UR can be written as
[Equation]
Note that under mapping φ , all u-leaf nodes in M (cid:48) are mapped to latent variables in H φ , and all nu-leaf nodes in M (cid:48) are mapped to observed variables in X φ .
Further, all edges are reversed.
This means that there exists a permutation of H φ , denoted by H ∗ φ , such that Z i is mapped to H ∗ φ i and Y ∗ i is mapped to X i .
Note that the X variables are arranged according to a certain causal order, which is not necessarily the case for the Y variables.
Further, if there is an edge from Y ∗ i to Y ∗ j in M (cid:48) with weight w , then there exists an edge from X i to
X j in φ ( M (cid:48) ) with weight w .
Therefore, A φ = ( C ∗ ) (cid:62) .
Similarly, B φ P H = D (cid:62) , where P H represents the re-permutation of the latent variables corresponding to the order of the u-leaf nodes.
The mixing matrix of φ ( M (cid:48) ) can be written as
[Equation]
Comparing Equation (10) to Equation (11), we have
[Equation]
Therefore, there exist column permutations of W ME ( M (cid:48) ) and W UR ( M ) , namely, ˜ W ME ( M (cid:48) )
=
W ME ( M (cid:48) ) P (cid:62) ,
[-W UR ( M ) (cid:20) P H 
-][Equation]
such that ˜ W ME ( M (cid:48) ) =
( ˜ W UR ( M )) (cid:62) .
C.2 Proof of Theorem
In order to prove Theorem 2, we need to prove Propositions 1 and 2, and that the AOG equivalence class is the extent level of identiﬁability under Assumption 2 (a) .
We begin by proving Proposition 2 (and its counterpart for direct ordered grouping).
We formulate these in the following lemma.
Lemma
Given the mixing matrix W ME (resp.
W UR ) and the AOG/DOG of the model, all the coefﬁcients are identiﬁed (up to the permutation and scaling of the columns of B in SEM-UR)
if and only if the centers (resp.
exogenous noise assignments) of the ancestral/direct ordered groups are identiﬁed.
Proof.
“only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction only-if direction”: If all the coefﬁcients are identiﬁed, i.e., matrices C , D in Equation (2) (and matrices A , B in Equation (4)) are identiﬁed, then the structure of the model is identiﬁed.
Therefore, we can identify ancestral/direct ordered grouping of the model and ﬁnd the center of each ancestral/direct ordered group.
“if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction if direction”: For SEM-ME, given the identiﬁed centers in each ordered group, we compute the coefﬁcients of the model from the mixing matrix W ME as follows.
According to the deﬁnition of AOG/DOG, only the ﬁrst p n ordered groups can include more than one element; recall that p n denotes the number of nu-leaf nodes.
Therefore, for any possible selection of the centers in each group, consider the submatrix of W ME where the rows correspond to the ﬁrst p n selected centers.
This submatrix can be permuted to a lower-triangular matrix with non-zero diagonal, and maps to ( I − C ) − 1 in Equation (5); recall that C represents the edges among the ﬁrst p n centers (i.e., the nu-leaf nodes in the model).
That is, C can be identiﬁed by ﬁrst normalizing the columns of W ME , and then taking the inverse of the submatrix that corresponds to the identiﬁed centers.
Note that the lower-triangular matrix (after permutation) must have diagonal one.
If the diagonal entry has value not equal to one, then this number represents the noise coefﬁcient, and can be removed by column normalization.
Matrix D , which represents the edges from nu-leaf nodes to u-leaf nodes can be deduced from remaining submatrix of W ME that maps to D ( I − C ) − 1 .
Therefore, all coefﬁcients can be identiﬁed.
For SEM-UR, given the exogenous noise terms of the identiﬁed centers in each ordered group, we use a similar method to compute the coefﬁcients of the model from the mixing matrix W UR .
Speciﬁcally, A in Equation (4) can be identiﬁed using the submatrix of W UR where the columns correspond to the last q selected centers ( q denotes the number of observed variables).
Matrix B can be deduced from the remaining submatrix of W UR .
Note that permuting and normalizing the columns of the submatrix that maps to ( I − A ) − 1 does not affect the the submatrix that maps to B ( I − A ) − 1 .
This means that the true permutation and scaling of B cannot be identiﬁed.
In the remainder of the proof of Theorem 2, we focus on linear SEM-ME; the proof for linear SEM-UR follows similarly.
The proof of the remaining assertions consists of two parts.
In the ﬁrst part, we show that models in the same AOG equivalence class satisfy Proposition 1 and Assumption 2 (a) .
That is, given the mixing matrix W ME , under different choices of the center in each ancestral ordered group, the alternative model (1) satisﬁes
Assumption 2 (a) , and (2) preserves the causal order among the ancestral ordered groups
We use Pa M ( V i ) and An M ( V i ) to denote the parent set and ancestor set of variable V i in model M throughout the remaining of the proof.
Proof of the ﬁrst part :
Proof of (1) : Note that Assumption 2 (a) (i.e., conventional faithfulness assumption) is equivalent to the following:
If V i is an ancestor of V j , then the support of the row of W ME corresponding to V
i is a subset of the support of the row of W ME corresponding to V j .
Further, the converse is also true given that V i is an nu-leaf node.
Let M denote the ground-truth, and let M (cid:48) denote an alternative model that belongs to the same AOG equivalence class as M .
We ﬁrst show that if some center nodes in M are swapped with other nodes in their respective ancestral ordered groups, then an added edge(s) to any node V in M (cid:48) can only originate from variables in
An M ( V ) .
To show this, suppose that V
i 1 , · · · , V i m represent some centers of the ancestral ordered groups in M , where m ≤ p n , and { i 1 , · · · , i m } is consistent with the causal order among the nu-leaf nodes in M .
Now, the centers in M (cid:48) are changed from V i 1 , · · · , V i m to V j 1 , · · · , V j m , where V i k and V j k are in the same ancestral ordered group, ∀ k ∈ [ m ] .
Consider the structural equations of V i 1 (ﬁrst changed center in M ) and V j 1 (the swapped center in M (cid:48) ), and V k 1 which is any child of V i 1 in M (other than V k 1 ).
Since V i 1 and V j 1 belong to the same ancestral ordered group, if follows by the deﬁnition of AOG that parents of V j 1 , except for V i 1 , are also ancestors of V i 1 in M .
Therefore the structural equations are:
[Equation]
Now, we choose V j 1 to be the center of the corresponding ordered group in M (cid:48)
(replacing the center V i 1 in M ).
Now, in M (cid:48) , V i 1 is a child of V j 1 and is a u-leaf node.
Also, the children of V i 1 in M are now children of V j 1 in M (cid:48) .
Therefore, the structural equations of V i 1 , V j 1 and V k 1 in M (cid:48) are
[Equation]
Comparing both sets of structural equations, we see that the additional edges to V j
and V k 1 in M (cid:48) originate from variables in An
M ( V i 1 ) .
Since V i 1 is a parent of both, this means that variables in An M ( V i 1 ) are also ancestors of V j 1 and V k 1 .
This further implies that the ancestor sets of all variables remain the same after this change, i.e., if V i is an ancestor of V j in M , then V i (or the node that replaces V i ) is an ancestor of V j (or the node that replaces V j ) in M (cid:48) .
By repeating the same procedure, i.e., swapping V i k with V j k for k = 2 , 3 , · · · , m , the resulting model M (cid:48) will only add edges from variables in An M ( V ) to V , ∀ V ∈ Y ∪ Z .
Next, we show, by contradiction, that if M satisﬁes Assumption 2 (a) , then M (cid:48) also satisﬁes Assumption 2 (a) .
Suppose M (cid:48) violates Assumption 2 (a) .
This means that there exist V 0 and V such that V 0 is an ancestor of V in M (cid:48) but the support of V 0 (i.e., its corresponding row in W ME ) is not a subset of the support of V .
Note that since V 0 is an nu-leaf node in M (cid:48) , hence it is a center node of some ancestral ordered group in M (cid:48) .
Let V (cid:48) 0 denote the center of the ancestral ordered group of V 0 in M (which can be V 0 itself).
Note that V 0 has the same support as V (cid:48) 0 since they belong to the same ancestral ordered group.
Therefore, the support of V (cid:48) 0 is not a subset of the support of V .
Since M satisﬁes Assumption 2 (a) , this means that V (cid:48) 0 is not an ancestor of V in
M . However, in obtaining M (cid:48) from M (i.e., by switching some centers in M ), the edges added to V in M (cid:48) only come from ancestors of V in M .
This means that V 0 cannot be an ancestor of V in M (cid:48) , which leads to a contradiction.
Therefore, M (cid:48) satisﬁes Assumption 2 (a) .
Proof of (2) :
Given that the alternative model M (cid:48) satisﬁes Assumption 2 (a) , we show that M (cid:48) preserves the causal order among the ancestral ordered groups. We show this by contradiction. Suppose that a variable V 0 in a later group causes another variable V in an earlier group in M (cid:48) ; according to the causal order among the groups in M .
Note that V 0 is an nu-leaf node in M (cid:48) since it causes V , and hence it is the center of some ancestral ordered group in M (cid:48) . Let V (cid:48) 0 be the center of the corresponding group in M , with an associated exogenous noise term denoted by N V (cid:48) 0 . Since V 0 and V (cid:48) 0 belong to the same ancestral ordered group, N V (cid:48) 0 must belong to the support of V 0 . However, since V is in an earlier group than V 0 in M (cid:48) , N V (cid:48) 0 is not in the support of V .
Therefore the support of V 0 is not a subset of the support of V , and hence, M (cid:48) violates Assumption 2 (a) .
Proof of the second part :
If the alternative model M (cid:48) does not belong to the AOG equivalence class of the ground-truth M , then at least one of the ﬁrst p n ancestral ordered groups in M does not include any of the nu-leaf nodes in M (cid:48) .
Denote the center of this ordered group as V i , and let N V
i denote its exogenous noise.
Since V i is a u-leaf node in M (cid:48) , there is at least one parent of V i in M (cid:48) that also includes N V i (i.e., is affected by N V i directly or indirectly).
Denote this variable as V j .
Since V j includes N V i , it must be a descendant of V i in M .
As M satisﬁes Assumption 2 (a) , this means that the support of V j (in W ME ) is a superset of the support of V i .
Further, since the ancestral ordered group of V i does not include any nu-leaf nodes in M (cid:48) , V j does not belong to the ancestral ordered group of V i . This means that the support of V j (in W ME ) is not the same as the support of V i . Therefore, the support of V j (in W ME ) is a strict superset of the support of V i . Since V j is a parent of V i in M (cid:48) , this violates Assumption 2 (a) on the alternative model M (cid:48) .
Conclusion : We showed that by under different choices of the centers in the ancestral ordered groups, we can always ﬁnd a model M (cid:48)
that: (1) has the same mixing matrix; (2) satisﬁes Assumption 2 (a) ; (3) preserves the causal order among ancestral ordered groups.
Thus, we cannot distinguish M (cid:48) from the ground-truth under
Assumption 2 (a) alone.
Further, we show that any model that have the same mixing matrix but does not belong to the AOG equivalence class does violate Assumption 2 (a) and can be distinguished from the ground-truth M .
Therefore the extent level of identiﬁability can be characterized by AOG equivalence class.
C.3 Proof of Theorem
To prove Theorem 3, we need to prove Propositions 3 and 4, and that the DOG equivalence class is the extent level of identiﬁability under Assumption 2.
The proof can be divided into two parts.
We ﬁrst show the necessity part, that is, models in the same DOG equivalence class as the ground-truth satisfy Assumption 2 and Proposition 3, hence cannot be distinguished from the ground-truth.
We then show the sufﬁciency part, i.e., any model that has the same mixing matrix but does not belong to the DOG equivalence class of the ground-truth violates Assumption 2.
C.3.1 Proof of necessity
We have shown in Lemma 1 that models in the same DOG equivalence class can be identiﬁed by the choice of centers of the direct ordered groups.
In the following, given the ground-truth model M and a certain choice of the centers that corresponds to an alternative model M (cid:48) , where M (cid:48) belongs to the same DOG equivalence class as M , we show that in the alternative model: (1) No edges are added; (2) No edges are removed; (3) Faithfulness is preserved.
Therefore we cannot distinguish the alternative model from the ground truth.
Suppose that V i 1 , · · · , V i m represent some centers of the direct ordered groups in the ground-truth M , where m ≤ p n , and { i 1 , · · · , i m } is consistent with the causal order among the nu-leaf nodes in M .
Now, the centers in M (cid:48) are changed from V i 1 , · · · , V i m to V j 1 , · · · , V j m , where V i k and V j k are in the same direct ordered group, ∀ k ∈ [ m ] .
It follows by the deﬁnition of DOG that the edge V i k
→ V j k violates Condition 1, ∀ k ∈ [ m ] .
Note that according to Condition 1, we have the following two properties: (1) For each nu-leaf node V and each u-leaf node V (cid:48) that belongs to the same direct ordered group as V , Pa ( V (cid:48) )
\ { V } ⊆
Pa ( V ) .
(2) All u-leaf nodes that belong to the same direct ordered group have the same parent set; otherwise, Condition 1 will be satisﬁed and the u-leaf nodes cannot belong to the same direct ordered group.
Proof of (1) : Consider a set of models M 0 , M 1 , · · · , M m , where M 0 = M , and M k changes the centers V i 1 · · · , V i k in M 0 to V j 1 · · · , V j k ; k ∈ [ m ] .
In the following, we show that for all k ∈ [ m ] , obtaining M k from M 0 as described above does not add extra edges in M k .
We start with k = 1 .
The structural equations of V i 1 and V j 1 in M 0 are:
[Equation]
Further, the structural equation of any child V k 1 of V i 1 in M 0 is:
[Equation]
Note that N V k 1 = [-
-]0 if and only if V k 1 is a u-leaf node.
In M 1 , since we only swap V i 1 and V j 1 , the structural equations of all variables in Z ∪ Y \ { V i 1 , V j 1 } ∪ Ch[-M 
-]
M 0 ( V i 1 ) remain the same as those in M 0 .
From (12) and (13), the structural equation of V j 1 in M 1 :
V j 1 = (cid:88)
[Equation]
Since the edge V i 1 {+→ V j 1 violates Condition 1(a), Pa M 
+}
( V j 1 ) \
{ V i 1 } ⊆
Pa M 0 ( V i 1 ) .
By deﬁning a (cid:48)
j 1 l (cid:44) a [Equation] a
[Equation] 1 [ V l ∈ Pa M 0 ( V j 1 )] for all V l ∈ Pa M {+
+} ( V i 1 ) , we can re-write V j 1 as
V j 1 = (cid:88)
[Equation]
Next, from (13), the structural equation of V i 1 in M 1 is:
[Equation]
(cid:88)
[Equation]
Lastly, by substituting V i 1 in (14) by (16), the structural equation of V k 1 in M[-
 ( V i-] 1 is:
V k 1 = (cid:88)
[Equation]
Since V i 1 → V j 1 violates Condition 1(b), Pa M 0 ( V j 1 ) ⊆
Pa M 0 ( V k 1 ) .
Therefore, by deﬁning a (cid:48) k 1 l k (cid:44) a k 1 l k −
[Equation] [Equation]
[Equation] 1 lk
[Equation]
1 [ V l k ∈ Pa M 0 ( V j 1 )] , we can re-write V k 1 as V k 1 = (cid:88)
[Equation]
Recall that in M 1 , the structural equations of all underlying variables in Z ∪ Y \ { V [-j 1 }∪ Ch M 
-][Equation] [Equation] ∪
[Equation] M [Equation] V i 1 ) are the same as M 0 .
Further, the structural equations of V i 1 , V j 1 and any V k 1 ∈
Ch M
( V i 1 ) in (12)-(14) are replaced by (15)-(17).
Thus, we have Pa M 1 ( V j 1 ) =
Pa M
( V i 1 ) , Pa M 1 ( V i 1 )
= { V j 1 }∪ Pa M 0 ( V j 1 ) \{ V i 1 } , and Pa M 1 ( V k 1 ) =
{ V j 1 } ∪ Pa M
( V k 1 ) \ { V i 1 } .
Therefore no edge will be added in M 1 , and V i g
→ V j g still violates Condition 1 in M 1 for all g =
[2 : m ] .
Note that Pa M 1 ( V j 1 ) (resp.
Pa M 1 ( V k 1 ) ) may be a subset of Pa M {+
+}
( V i 1 ) (resp.
V j 1 } ∪ Pa M
( V k 1 ) \ { V i 1 } ) due to certain parameter cancellation in a (cid:48)
j 1 l (resp.
a (cid:48) k 1 l k ).
However, it does not affect our conclusion here as we show that no edges are added in M 1 .
We will show in Part (2) that if this parameter cancellation happens (at any point of the procedure of swapping centers), then the model violates faithfulness assumption.
Next, consider k = 2 .
Repeating the same procedure of swapping centers, the structural equations of M 2 can be obtained from M 1 .
Similarly, we have Pa M 2 ( V j 2 ) =
Pa M 1 ( V i 2 ) , Pa M 2 ( V i 2 ) =
{ V j 2 }∪ Pa M 1 ( V j 2 ) \{ V i 2 } , and Pa
M 2 ( V k 2 )
= { V j 2 } ∪ Pa M 1 ( V k 2 ) \ { V i 2 } .
Therefore, no edge will be added when obtaining M 2 from M 0 .
By repeating the same procedure for k = 3 , · · · , m , we conclude that obtaining the ﬁnal model M m from M 0 does not add additional edges in M m compared to M 0 .
Proof of (2) :
Following the procedure of swapping centers described in Part (1), we show that if any edge from M 0 is removed in M m , then the ground-truth model M 0 violates Assumption 2.
Suppose the edge from V 0 to V in M 0 is removed in M m .
Here “edge removal” means that there is an edge from V 0 to V in M 0 , but there is no edge from V 0 (or the node that replaces V 0 ) to V (or the node that replaces V ) in M m . Note that V 0 is an nu-leaf node in M 0 , which is not necessarily the case for V .
Hence we consider each of the following cases: (i
) V is not swapped with any other nodes, i.e., V (cid:54)∈ { V i 1 , · · · , V i m , V j 1 · · · , V j m } .
This means that V 0 is a parent of V in M 0 , but V 0 (or the node that replaces V 0 ) is not a parent of V in M m .
(ii) V =
V i
k for some k ∈ [ m ] .
This means that V 0 is a parent of a center node
V
i k in M 0 , but V 0 (or the node that replaces V 0 ) is not a parent of the center node V j k in M m .
(iii) V =
V j k for some k ∈ [ m ] .
This means that V 0 is a parent of a non-center node V j k in M 0 , but V 0 (or the node that replaces V 0 ) is not a parent of the non-center node V i k in M m .
Case (i).
Suppose V is not swapped with any other nodes.
Since the edge from V 0 to V is removed in M m , V has at least one less parent in M m than in M 0 . This means that the total effects of An M 0 ( V ) on V , i.e.,
TE ( An M 0 ( V ) , V ) , can be written as a linear combination of the total effects of An M m ( V ) on parents of V in M m , i.e., { TE ( An M m ( V ) , V (cid:48) ) : V (cid:48) ∈
Pa M m ( V ) } .
7 Further, following the procedure of swapping centers, all parents of V in M m are possible parents of V in M 0 .
Therefore, V violates Assumption 2 (b1) in M 0 .
Case (ii).
Suppose V = V i k for some k ∈ [ m ] .
Since the edge from V 0 to V j k is removed in M m , V j k has at least one less parent in M m than V
i k has in M 0 .
This means that TE ( An M 0 ( V j k ) \ { V i k } , V j k ) can be written as the linear combination of at most | Pa M {+
+}
( V i k ) |− 1 vectors in { TE ( An M 0 ( V j k ) \{ V i k } , V (cid:48) ) : V (cid:48)
∈
Pa M m ( V j k ) } .
Note that | Pa[-M 
-]
M 0 ( V i k ) ∪ Pa M 0 ( V j k ) |− 1 = | Pa M 0 ( V i k ) | since Pa M 0 ( V j k ) \{ V i k } is a subset of Pa {+M 
+}
( V i k ) .
Further, all parents of V j k in M m are possible parents of V i k in M 0 .
Therefore, V j k violates Assumption 2 (b2) in M 0 .
Case (iii).
Suppose V = V j k for some k ∈ [ m ] .
Note that following the procedure of swapping centers, for any g > k , replacing V i g by V j g does not change the structural equations of V i k and V j k .
Therefore, if V 0 (or the node that replaces V 0 ) is not a parent of V
i k in M m , then it is not a parent of V
i k in M k .
Further, according to Equations (13) and (16), when replacing V j k by V i k , the parent set of V i k in M k , excluding V j k , is exactly the same as the parent set of V j k before the swapping (i.e., in M k − 1 ) excluding V
i k .
Therefore, V 0 (or the node that replaces V 0 ) is not a parent of V j k in M k − 1 , while V 0 is a parent of a V j k in M 0 .
Recall that V j k is not swapped with any other nodes from M 0 to M k − 1 .
Therefore this reduces to the Case (i) above, which indicates that V j k violates Assumption 2 (b1) in M 0 .
In conclusion, if the ground-truth satisﬁes Assumption 2, then the procedure of swapping centers described in Part (1) does not remove any edge from M 0 .
Proof of (3) :
From Parts (1) and (2), we conclude that the structure of the new model M m is isomorphic to M 0 .
Speciﬁcally, we can construct a mapping σ from the nodes in M 0 to the nodes in M m , where
[Equation]
Under this mapping, there is an edge from node V (cid:48) to V (cid:48)(cid:48) in M 0 if and only if this is an edge from σ ( V (cid:48) ) to σ ( V (cid:48)(cid:48) ) in M m , for all V (cid:48) , V (cid:48)(cid:48)
∈
Z ∪ Y .
Note that this mapping only switches pairs of nodes that belong to the same direct ordered group.
This means that M m has the same edges across groups, which completes the proof of Proposition 3.
In the following, we show that if M 0 satisﬁes Assumption 2, then M m satisﬁes Assumption 2.
Note that since the DOG is a more reﬁned partition of AOG, according to the proof of Theorem 2, Assumption 2 (a) is satisﬁed.
Therefore we only need to verify Assumption 2 (b) .
For each variable V , we consider each of the three following cases:
(i) V (cid:54)∈ { V i 1 , · · · , V i m , V j 1 · · · , V j m } , i
e., V is not changed from
M to M m
(ii) V =
V i
k for some k ∈ [ m ] , i
e., V is a center node in M 0 and is changed to a non-center node in M m .
(iii) V =
V j k for some k ∈ [ m ] , i.e., V is a non-center node in M 0 and is changed to a center node in M m .
Case (i).
Suppose V (cid:54)∈ { V i 1 , · · · , V i m , V j 1 · · · , V j m } .
Note that any possible parent of V in M 0 is also a possible parent of V in M m , and for all V (cid:48) ∈ { Possible parents of V in M m } ∪ { V } , TE ( An M 0 ( V ) , V (cid:48) )
=
TE ( An M m ( V ) , V (cid:48) ) .
Further, if the parent set of any u-leaf node V (cid:48) in M 0 is a subset of Pa M 0 ( V ) , then the parent set of any u-leaf node in this direct ordered group in M m is a subset of Pa M m ( V ) .
Therefore Assumption 2 (b1) holds for V in M m .
Similarly, for Assumption 2 (b2) , for each parent V 0 of u-leaf node V in M m , denote V (cid:48)
as the center of the direct ordered group of V 0 in M m .
Then any possible parent of V (cid:48) 0 in M 0 is also a possible parent of V 0 in M m , and TE ( An M 0 ( V ) \ { V (cid:48) 0 } , V (cid:48) )
=
TE ( An M m ( V ) \ { V 0 } , V (cid:48) ) for all V (cid:48) ∈ { Possible parents of V 0 in M m } ∪ { V } .
Therefore Assumption 2 (b2) holds for V in M m .
Case (ii).
Suppose V = V i k for some k ∈ [ m ] .
Since V i k is a nu-leaf node in M 0 , TE ( An M 0 ( V i k ) , V i k ) =
TE ( An
M m ( V i k ) \ { V j k } , V j k ) .
Further, any possible parent of V i k in M 0 is a possible parent of V j k in M m , and | Pa
M 0 ( V i k ) | = | Pa M m ( V j k ) | = | Pa M m ( V j k ) ∪ Pa M m ( V i k ) \ { V j k }| .
Besides, if the parent set of any u-leaf node V (cid:48) in M 0 is a subset of Pa M
( V i k ) , then the parent set of any u-leaf node in this direct ordered group in M m is also a subset of Pa M m ( V j k ) .
Therefore, since Assumption 2 (b1) holds for V
i k in M 0 ,
Assumption 2 (b2) holds for V
i k in M m .
As for Assumption 2 (b1) , note that the set of possible parents of V i k in M m includes all other variable in the same ancestral ordered group of V i k in M 0 and is not the same as the set of possible parents of V i k in M 0 .
Therefore, if TE ( An M m ( V i k ) , V i k ) can be written as a linear combination of k ≤ | Pa M m ( V i k ) | vectors from the set { TE ( An M m ( V i k ) , V (cid:48) ) :
V (cid:48) is a possible parent of V
i k in M 0 } , denote the set of the possible parents that corresponds to these vectors as K .
Then at least one of the variables in K belongs to the ancestral ordered group of V i k in M 0 .
Denote this variable as V .
This means that TE ( An M m ( V i k ) , V ) can be written as a linear combination of k vectors which includes TE ( An M m ( V i k ) , V i k ) and { TE ( An M m ( V i k ) , V (cid:48) ) :
V (cid:48)
∈
K \ { V }} .
Since V is a u-leaf node in M 0 , any variable in V (cid:48) is a possible parent of V in M 0 , and TE ( An M m ( V i k ) , V (cid:48) )
=
TE ( An M 0 ( V ) , V (cid:48) ) for all V (cid:48)
∈ K ∪ { V i k } .
This means that that TE ( An M 0 ( V ) , V ) can be written as a linear combination of k vectors which includes TE ( An M 0 ( V ) , V i k ) and { TE ( An M 0 ( V ) , V (cid:48) ) :
V (cid:48)
∈
K \ { V }} .
Besides, according to our condition, | Pa M m ( V i k ) | ≤ | Pa M 0 ( V ) | (otherwise V i k does belong to the same direct ordered group as V j k ).
Therefore, since Assumption 2 (b1) holds for V on M 0 , | Pa M 0 ( V ) | = |
Pa M m ( V i k ) | =
k , and the parent set of the variables in K that are u-leaf nodes in M 0 must be a subset of Pa M 0 ( V ) . This implies that parent set of the variables in K that are u-leaf nodes in M m must be a subset of Pa M m ( V i k ) .
Therefore, Assumption 2 (b1) holds for V
i k in M m .
Case (iii).
Suppose V = V j k for some k ∈ [ m ] .
Since V j k is a u-leaf node in M 0 , TE ( An M m ( V j k ) , V j k ) =
TE ( An
M 0 ( V j k ) \ { V i k } , V j k ) .
Further, any possible parent of V i k in M 0 is a possible parent of V j k in M m , and | Pa
M m ( V j k ) | = | Pa M 0 ( V i k ) | .
Besides, if the parent set of any u-leaf node V (cid:48) in M 0 is a subset of Pa
M 0 ( V i k ) , then the parent set of any u-leaf node in this direct ordered group in M m is also a subset of Pa
M m ( V j k ) .
Therefore, since Assumption 2 (b2) holds for V j k in M 0 , Assumption 2 (b1) holds for V j k in M m .
To sum up, Assumption 2 (b) holds for every variable in M m .
Hence M m satisﬁes Assumption 2.
Conclusion : We showed that by replacing some centers in the ground-truth M 0 by some other nodes in the same direct ordered group, we can get an alternative model M m that (1) has the same mixing matrix, (2) has an isomorphic graph structure to M 0 , and (3) satisﬁes Assumption 2.
Therefore we cannot distinguish M 0 from any other model in the DOG equivalence class.
C.3.2 Proof of sufﬁciency
To prove sufﬁciency, we show that any model that has the same mixing matrix as the ground-truth but does not belong to the same DOG equivalence class violates Assumption 2.
Recall that Theorem 2 implies that any model with the same mixing matrix as the ground-truth, but does not belong to its AOG equivalence class, violates Assumption 2 (a) .
Consequently, it remains to prove that any model that belongs to the AOG equivalence class of the ground-truth, but does not belong to its DOG equivalence class, violates Assumption 2 (b) .
Recall from Proposition 2 that models in the same AOG equivalence class can be identiﬁed by the choice of the centers in each ancestral ordered group.
Therefore, we show that, given the ground-truth and the choice of the centers, where at least one selected center does not belong to the direct ordered group of the u-leaf node in the ground-truth: (1) at least one edge is added to the ground-truth; (2) no edge in the ground-truth is removed, and at least one added edge is not removed; (3) the alternative model violates Assumption 2 (b) .
Therefore Proposition 4 holds.
In the following, we refer to the centers as the centers of the ancestral ordered groups, unless otherwise stated.
Suppose that V i 1 , · · · , V i m represent some centers of the ancestral ordered groups in the ground-truth M , where m ≤ p n , and { i 1 , · · · , i m } is consistent with the causal order among the nu-leaf nodes in M . Now, the centers in M (cid:48) are changed from V i 1 , · · · , V i m to V j 1 , · · · , V j m , where V i k and V j k belong to the same ancestral ordered group, ∀ k ∈ [ m ] .
Further, at least one pair of ( V i k , V j k ) , k ∈ [ m ] does not belong to the same direct ordered group.
Proof of (1) : Consider the same set of models M 0 , M 1 , · · · , M m as in the necessity proof, where M 0 = M , and M k changes the centers V i 1 · · · , V i k in M 0 to V j 1 · · · , V j k ; k ∈ [ m ] .
In the following, we show that if at least one pair of ( V i g , V j g ) , g ∈ [ k ] does not belong to the direct ordered group, then obtaining M k from M 0 as described above adds at least one extra edge.
We start from k = 1 .
We have shown that if V j 1 and V i 1 are in the same direct ordered group, then obtaining M 1 from M 0 does not add any edge to the model.
Therefore we only focus on the case when V j 1 and V i 1 are in different direct ordered groups.
The true structural equations of V i 1 and {+V j 
+}
and V k 1 ∈
Ch M
( V i 1 ) in M 0 are the same as Equations (12)-(14)
so we will not repeat them here.
The structural equation of V j 1 in M 1 is:
[Equation]
Since the edge V i 1 → V j 1 satisﬁes Condition 1, Pa M 0 ( V j 1 ) \ { V i 1 } may not be a subset of Pa M
( V i 1 ) .
Therefore, by deﬁning a (cid:48)
j 1 l (cid:44) a j 1 i 1 a
i 1 l + a j 1 l 1 [ V l ∈
Pa M 0 ( V j 1 )] for all V l ∈ Pa M [-
-]0 ( V i 1 ) , and S j {+
+} =
Pa M 0 ( V j 1 ) \ ( { V i 1 } ∪ Pa M
( V i 1 )) , the above equation can be written as
[Equation]
The new structural equation of i 1 is the same as Equation (16), since no parameter calculation is required:
[Equation]
[Equation] As for V k 1 , using the same analysis as in Equation (18), deﬁne a (cid:48) [Equation]
a
[Equation] k −
[Equation] ] , and S k 1 =
Pa M 0 ( V j 1 ) \ Pa M
( V k 1 ) .
Then structural equation of V k 1 is: [Equation]
[-Pa M 
 ( V k 1 ) \{ V i 1 } a (cid:48) k 1 l k V l k + (cid:88) l j : V lj ∈S ka j 1 l j a j 1 i 1 V l j + a k 1 i 1 a j 1 i 1 V j 1 + N V k 1 .-][Equation]
Recall that in M 1 , the structural equations of all underlying variables in Z ∪Y \{ V i 1 , V j 1 }∪ Ch M
( V i 1 ) are the same as M 0 , and the structural equations of V
i 1 , V i 1 and any V k 1 ∈
Ch M
( V i 1 ) in (12)-(14) are replaced by (18)- (20).
Since the edge V i 1 → V j 1 satisﬁes Condition 1, at least one of the sets in {S j 1 }∪{S k 1 : k ∈ Ch M {+
+} ( V i 1 ) } must be non-empty.
Therefore, compared to M 0 , there is at least one edge added in M 1 .
Next consider k = 2 .
Similarly, if V i 2 and V j 2 are in the same direct ordered group, then no edges are added to M 2 .
If V i 2 and V j 2 belong to different direct ordered groups, then there is at least one edge added to M 2 (compared with M 0 ).
Note that this edge may be the same as the edges added to M 1 .
However, we will show in Part (2) that at least one of the the added edges after swapping both centers does not cancel out each other.
Therefore it is still an added edge compared with M 0 .
By repeating the same procedure for k = 3 , · · · , m , we conclude that, since at least one of the new centers does not belong the same direct ordered group,at least one more edge is added in M m compared to M 0 .
Proof of (2) :
We ﬁrst show that if M m removes any edge from M 0 , then the ground-truth model M 0 violates Assumption 2.
The proof is the same as the proof of Part (2) in the necessity proof: We only used the fact that all variables in the same direct ordered group have the same support, which also holds for ancestral ordered group.
Therefore if any edge in M 0 is removed, then M 0 violates Assumption 2 (b) .
Next, we show that there is at least one added edge that cannot be canceled out.
We prove by contradiction, i.e, if all added edges cancel out each other in M m , then M 0 violates Assumption 2 (b) .
Suppose the edge from Vto V is is ﬁrstly added in M k 0 for some k 0 ∈ [ m ] and is removed in M m .
Note that the position of V 0 might be changed from M 0 to M k 0 − 1 .
Denote V (cid:48)
as the center of the ancestral ordered group of V 0 in M 0 (which might be V 0 itself).
Recall that if an edge from V 0 to V is added, then there must be a causal path from V (cid:48) 0 to V in M 0 .
We further assume that no edges are added from any ancestor of V (cid:48)
in M 0 to other nodes, and for any variable ˜ V on the causal path from V (cid:48) 0 to V , no edges are added from V (cid:48) 0 to ˜ V .
This can be satisﬁed by selecting V (cid:48)with the smallest index among all added edges following the causal order, and selecting V that belongs to the ancestral ordered group with the smallest index among all added edges from V 0 .
We consider each of the three following cases:
(i) V does not change its position after swapping all centers, i e., V (cid:54)∈ { V i 1 , · · · , V i m , V j 1 · · · , V j m } .
This means that the edge from V 0 to V is added in M k 0 , but V 0 is not a parent of V in M m .
(ii) V =
V i
k for some k ≥ k 0 .
This means that the edge from V 0 to V i k is added in M k 0 , but V 0 is not a parent of V j k in M m .
(iii) V =
V j k for some k ≥ k 0 .
This means that the edge from V 0 to V j k is added in M k 0 , but V 0 is not a parent of V i k in M m .
Case (i).
Suppose V does not change its position.
This means that there is a causal path V (cid:48)
→ V i k
→ V in
M 0 , while V (cid:48) 0 is not connected to V .
Besides, since the edge from V 0 to V is added by replacing V i k 0 with
V j k 0 in M k 0 , V (cid:48) 0 is a parent of V j k 0 in M 0 .
If all added edges are cancelled out, then V has at most | Pa M 0 ( V ) | parents in M m , i.e., TE ( An M 0 ( V ) , V ) can be written as the linear combination of at most | Pa M 0 ( V ) | vectors in { TE ( An
M 0 ( V ) , V (cid:48) ) : V (cid:48)
∈
Pa M m ( V ) } .
Further, if no edges in M 0 are canceled out in M m , then V j k 0 is a parent of V in M m .
However, V j k 0 is a u-leaf node in M 0 with parent set not being a subset of Pa M 0 ( V ) (because V (cid:48)
∈
Pa M 0 ( V j k 0 ) and V (cid:48)
(cid:54)∈ Pa M 0 ( V ) ).
Therefore M 0 violates Assumption 2 (b1) .
Case (ii).
Suppose V = V i k for some k ∈ [ m ] .
This means that there is a causal path V (cid:48)
→ V i k 
 → V i k
→ V j k and a connection from V (cid:48)
to V j k 0 in M 0 .
Further, there is a connection from V 0 to V j k in M k which may or may not be an added edge.
If all added edges are cancelled out, then V j k has at most | Pa M 0 ( V i k ) | parents in M m , i.e., TE ( An M 0 ( V j k ) \ { V i k } , V j k ) can be written as the linear combination of at most | Pa M
( V i k ) | vectors in { TE ( An
M 0 ( V j k ) \{ V i k } , V (cid:48) ) : V (cid:48)
∈
Pa M m ( V j k ) } .
Note that | Pa M
( V i k ) | ≤ | Pa M 0 ( V i k ) ∪ Pa M
( V j k ) ⊆
{ V i k }| , and the equality holds only if Pa M 0 ( V j k ) ⊆
{ V i k } ⊆
Pa M 0 ( V i k ) , i.e., parents of V j k in M 0 are all parents of V i k , except for V
i k itself.
In this case, V j k 0 is a parent of V j k in M m , but it is a u-leaf node in Mwith the parent set not being a subset of V
i k (because of V (cid:48) 0 ).
Therefore M 0 violates Assumption 2 (b2) .
Case (iii).
Suppose V = V j k for some k ∈ [ m ] .
Note that the proof of Case (iii) in Part (2) of the necessity proof also applies here: If V 0 is not a parent of V i k in M m , then V 0 is not a parent of V i k in M k .
According to Equations (13) and (19), this means that V 0 is not a parent of V j k in M k − 1 , which falls into Case (i) and implies that M 0 violates Assumption 2 (b1) .
In conclusion, if the ground-truth satisﬁes Assumption 2, then M m has at least one more added edges than M 0 , which completes the proof of Proposition 4.
Proof of (3) :
From Parts (1) and (2), we conclude that no edges are removed from M 0 in M m , and there is at least one edge added in M m compared to to M 0 .
In the following we show that because of this additional edge, M m violates Assumption 2.
Suppose the edge from V 0 to V is added in M m .
Similar as Part (2), the position of V 0 might be changed.
Denote V (cid:48)
as the center of the ancestral ordered group of V 0 in M 0 (which may be V 0 itself).
We consider each of the three following cases:
(i) V does not change its position between M 0 and M m ,
i .e.
, V (cid:54)∈ { V i 1 , · · · , V i m , V j 1 · · · , V j
m This means that V (cid:48) 0 is not a parent of V in M k 0 , but V 0 is a parent of V in M m
(ii) V =
V i
k for some k ∈ [ m ] .
This means that V (cid:48) 0 is not a parent of V
i k in M k 0 , but V 0 is a parent of V j k in M m .
(iii) V =
V j k for some k ∈ [ m ] .
This means that V (cid:48) 0 is not a parent of V j k in M k 0 , but V 0 is a parent of V i k in M m .
Case (i).
Suppose V does not change its position.
If the edge from V (cid:48) 0 to V is added in M m , then V has at least one more parent in M m than in M 0 , i.e., | Pa M m ( V ) | > | Pa M
( V ) | .
Recall that in the ground-truth, TE ( An M 0 ( V ) , V ) can be written as the linear combination of | Pa M
( V ) | vectors in { TE ( An
M 0 ( V ) , V (cid:48) ) : V (cid:48)
∈
Pa M 0 ( V ) } , where parents of V in M 0 are all possible parents of V in M m .
Further, TE ( An M 0 ( V ) , V (cid:48) )
=
TE ( An M m ( V ) , V (cid:48) ) for all V (cid:48)
∈
Pa M 0 ( V ) ∪ { V } .
Therefore, TE ( An
M m ( V ) , V ) can be written as the linear combination of strictly less than | Pa M m ( V ) | vectors in { TE ( An
M m ( V ) , V (cid:48) ) : V (cid:48) is a possible parent of ( V ) in M m } , which means that M m violates Assumption 2 (b1) .
Case (ii).
Suppose V = V i k for some k ∈ [ m ] .
In this case, V j k has at least one more parents in M m than V
i k has in M 0 , i.e., | Pa M m ( V j k ) | > | Pa M
( V i k ) | .
Further, TE ( An M m ( V i k ) \ { V j k } , V i k ) =
TE ( An M
( V i k ) , V i k ) , and parents of V
i k in M 0 are all possible parents of V j k in M m .
Therefore, TE ( An M m ( V i k ) \ { V j k } , V i k ) can be written as the linear combination of | Pa M {+
+}
( V i k )
| < | Pa M m ( V j k ) ∪ Pa M m ( V i k ) \ { V j k }|
vectors in { TE ( An M m ( V ) , V (cid:48) ) :
V (cid:48) is a possible parent of V j k in M m } , which means that M m violates Assumption 2 (b2) .
Case (iii).
Suppose V = V j k for some k ∈ [ m ] .
In this case, | Pa M m ( V i k ) | > | Pa M 0 ( V j k ) | .
Recall that TE ( An
M 0 ( V j k ) , V j k ) can be written as the linear combination of | Pa M 0 ( V j k ) | vectors in { TE ( An M 0 ( V j k ) , V (cid:48) ) :
V (cid:48)
∈
Pa M 0 ( V j k ) } .
Since V j k is a u-leaf node in M 0 , TE ( An M 0 ( V j k ) , V i k ) can be written as the linear combination of | Pa M 0 ( V j k ) | vectors in { TE ( An M 0 ( V j k ) , V (cid:48) ) :
V (cid:48)
∈
Pa M 0 ( V j k ) ∪ { V j k } \ { V i k }} .
Notice that TE ( An M 0 ( V j k ) , V (cid:48) ) =
TE ( An M m ( V i k ) , V (cid:48) ) for all V (cid:48)
∈
Pa M 0 ( V j k ) ∪ { V j k } , and parents of V j k in M 0 other than V
i k are all possible parents of V
i k in M m .
Therefore, TE ( An M m ( V i k ) , V i k ) can be written as the linear combination of | Pa M 0 ( V j k ) ∪ { V j k } \ { V i k }| = | Pa M 0 ( V j k ) | < |
Pa M m ( V i k ) | vectors in { TE ( An M m ( V i k ) , V (cid:48) ) :
V (cid:48) is a possible parent of ( V i k ) in M m } , which means that M m violates Assumption 2 (b1) .
To summarize, if any edge is added to the model, then this edge must cause certain violation of Assumption 2.
Conclusion : We showed that by replacing some u-leaf nodes in the ground-truth M 0 by some other nodes in ancestral ordered group of each u-leaf node, where at least one of them is not in the same different direct ordered group of the nu-leaf node, the alternative model M m (1) has the same mixing matrix, (2) has at least one more edge than the graph structure of M 0 , and (3) violates Assumptions 2.
Therefore we can distinguish
M 0 from any other model outside the DOG equivalence class.
Combining the necessity part and the sufﬁciency part together, we conclude that DOG equivalence class is the extent level of identiﬁability under Assumption 1 and 2.
D Comparison with partially observed linear causal models
The linear SEM-UR deﬁned in Deﬁnition 3 is a subclass of the linear SEM with latent variables proposed in [1].
Speciﬁcally, [1] does not require the latent variables to be parent-less.
Further, the linear SEM-ME in Deﬁnition 2 can also be viewed as a subclass of the model in [1], where the latent variables correspond to unobserved underlying variables, Z , and the observed variables correspond to observed underlying variables Y and the measurements of Z (i.e., U ).
X
X
X
H a c b (a)
X X
X
H a − c b + c − ac(b)
X
X X
H a − c b + c (c)
The authors of [1] derived necessary conditions for unique identiﬁability of the model with the fewest number of edges, which are also sufﬁcient together with “bottleneck faithfulness assumption” (cf.
Condition 3).
The derived conditions are two-fold: (1) The bottleneck condition, which requires the children set of any (observed or latent) variable V i to be the smallest set to block all directed paths from V i to its observed descendants and (
Recall that from Proposition 4, our identiﬁability condition also guarantees that if the ground-truth is uniquely identiﬁable, then any other model that has the same mixing matrix as the ground truth and satisﬁes conventional faithfulness has strictly more edges.
We have shown in Appendix B.4 that our SEM-UR assumption is implied from bottleneck faithfulness assumption.
Further, our condition for unique identiﬁability of a SEM-UR is also implied from from the identiﬁability conditions in [1].
Speciﬁcally, for the linear SEM-UR, the bottleneck condition is automatically satisﬁed, and the strong non-redundancy condition implies Condition 2(a). In contrast, our unique identiﬁability condition for the linear SEM-UR does not imply the conditions in [1]: A linear SEM-UR satisfying Condition 2 may not be uniquely identiﬁable in [1]. This is because the model considered in [1] is a larger class than the SEM-UR.
An example is shown in Figure 6, where the linear SEM-UR generated according to Figure 6(a) can be uniquely identiﬁed (as a SEM-UR).
The only other model in its AOG equivalence class is shown in Figure 6(b) (by changing the exogenous noise term associated with X 2 from N X 2 to N H ), which violates Assumption 3 and also has more edges than (a).
However, the ground-truth is not uniquely identiﬁable in [1], where an alternative model is shown in Figure 6(c).
Note that Figure 6(c) and Figure 6(a) have the same number of edges, and hence [1] cannot distinguish it from the ground-truth.
On the contrary, Figure 6(c) is not a SEM-UR since H has a parent, and can distinguished from the ground-truth.
Note that Figure 6(a) and 6(c) have exactly the same total causal effects among all pairs of observed variables.
Finally, we note that if we were to apply the identiﬁability result in [1] to the SEM-ME, this leads to a trivial special case.
Speciﬁcally, if the nun-redundancy condition is satisﬁed, then all leaf nodes must be observed (i.e., there is no u-leaf node in the model).
The reasoning behind this is as follows.
If there is an unobserved leaf node Z i with its measurement U i , we can remove Z i , and draw edges from the parents of Z i to U i .
The resulting model has the same mixing matrix and one less edge ( Z i → U i removed).
However, this corresponds to a trivial special case of SEM-ME:
If there is no u-leaf node, then the SEM-ME can always be uniquely identiﬁed, since each nu-leaf node is in a separate ancestral ordered group.
Consequently, our identiﬁability results for the linear SEM-ME do not follow from the identiﬁability results in [1].
E Numerical experiments
We evaluated the performance of our recovery algorithm on randomly generated linear SEM-MEs and SEM-URs with different number of variables.
We considered two cases: (1) when a noisy version of the mixing matrix is given, where the noise is Gaussian with different choices of variance denoted by d 2 , and (2) when synthetic data comes from a linear generating model with non-Gaussian noises with different sample sizes.
E.1 Simulation settings
Model generation.
We randomly generate synthetic linear SEM-MEs and SEM-URs as follows.
For SEM-ME, we ﬁrst generate a linear SEM with p underlying variables, where each pair of observed variables is connected with probability p e .
Then, we randomly select some of the p underlying variables to be unobserved (measured with error), where the measurement error is added after the coefﬁcients are drawn; note that we only add measurement error to the unobserved variables corresponding to the non-leaf nodes.
The corresponding mixing matrix is calculated (i.e., ( I − A ) − 1 ), where A is the adjacency matrix among underlying variables in Z ∪ Y .
For SEM-UR, we generate a SEM with q observed variables and l latent variables.
Each pair of observed variables, and each pair of latent and observed variables, is connected with probability p e .
We ensure that each latent variable has at least two children.
The strength (coefﬁcients) of the edges in both models are independently drawn from the uniform distribution [0 . 5 , 1] .
After the mixing matrix is obtained, we randomly permute its rows and columns to hide the true causal order.
Baseline methods.
We compare the performance of our recovery algorithm (which we denote by DOG) with the following methods: (1) AOG-based method (AOG): After recovering the AOG from the mixing matrix, we randomly choose the centers in each ancestral ordered group.
We then recover the full model based on the selection of the centers.
This approach is similar to the recovery method in [19].
(2) ICA-LiNGAM [21].
Note that ICA-LiNGAM method is not designed for systems with latent variables, but beneﬁts from strong performance for recovering the mixing matrix in causally sufﬁcient settings.
The goal of this comparison is to demonstrate the necessity of using methods designed speciﬁcally to handle latent variables.
We compare with AOG both in Case (1) and Case (2), and we compare with ICA-LiNGAM only in Case (2).
Metrics.
We compare the recovered matrices with the ground-truth matrix based on the following metrics:
(1) SHD / Edge: The normalized structural Hamming distance divided by the number of true edges in the graph; (2) Precision: The fraction of edges in the recovered model that is also in the ground-truth; (3) Recall: The fraction of edges in the ground-truth that can be recovered; (4) F1 Score:
The harmonic mean of Precision and Recall.
In linear SEM-ME, since we can only recover the ground-truth model up to its DOG equivalence class, we also compared the recovered matrices with the matrix that has the smallest l 2 distance (Frobenius norm) among all models in the DOG equivalence class of the ground-truth.
Furthermore, In linear SEM-UR, DOG and AOG methods return both the adjacency matrix A among observed variables and the adjacency matrix B from latent to observed variables.
On the contrary, ICA-LiNGAM only returns the matrix A .
Therefore, we compare the performance of all three methods in recovering A , and we compare the performance of DOG and AOG methods in recovering B .
Note that B can only be learned up to permutation and scaling of the columns.
Pre-processing of the mixing matrix.
In linear SEM-ME, the mixing matrix used in recovery ( W ME ) is a submatrix of the overall mixing matrix W .
Given W and the indices corresponding to the unobserved variables, we extract W ME from W as follows.
We ﬁrst normalize each column by the entry with the largest absolute value.
Next, for each index of an unobserved variable i , we ﬁnd the column vector in W that has the smallest l 2 -distance to the one-hot vector e i (i.e., only the i -th entry is 1), and remove that column from W .
We repeat until all one-hot vectors of the measurement errors are removed.
We note that the recovery of AOG from the mixing matrix W ME / W UR (cf.
Algorithm 1) requires that we can always ﬁnd a row/column with only one non-zero entry in each iteration of the algorithm.
To ensure that the recovered mixing matrix satisﬁes this property, we sort all non-zero entries in the mixing matrix with respect to their absolute values (from smaller to larger), and iteratively remove the entries with the smallest absolute values until a valid AOG is returned.
E.2 Noisy mixing matrix
We test the performance of our method for Case (1), i.e., when a noisy version of the mixing matrix is given, .
Given the true mixing matrix, the noisy matrix is constructed by ﬁrst adding an independent Gaussian noise with variance d 2 to each non-zero entry of W , and then adding independent Gaussian noise with variance d 2 to each entry of W with probability 0 .
d is the parameter controlling the noise variance, which is selected as [0.01, 0.05, 0.15].
For linear SEM-ME, We select p =
[5 , 10 , 15 , 20 , 25] , and we randomly select 80% of the variables to be measured with error.
We set p e = 0 . 4 , and repeat the simulation for 50 times.
The average performance is shown in Figure 7.
For linear SEM-UR, We select q = [4 , 8 , 12 , 16 , 20] , and we select l = [1 , 2 , 3 , 4 , 5] , respectively.
We set p e = 0 . 4 , and repeat the simulation for 50 times.
The average performance is shown in Figure 8.
We observe that our DOG method performs better than AOG method for the recovery of both models.
Besides, the performance of both algorithms are better if the variance d is small.
Further, Recall from Corollary 3 that all
SEM-URs in the same DOG have the same structure.
This can be seen in Figure 8, where with d = 0 .
01 , our DOG algorithm has a high accuracy in recovering the graph structure of the ground-truth.
As for the recovery of
SEM-ME in Figure 7, we observe that even though theoretically the structure of the ground-truth model can only be identiﬁed up to the DOG equivalence class, our DOG algorithm can recover part of the ground-truth structure that is shared among the DOG equivalence class. [Figure] [Figure]
E.3 Synthetic data with non-Gaussian noises
We test the performance of our recovery algorithm for Case (2), where the synthetic data is generated from the linear causal model with non-Gaussian exogenous noise terms, and the sample size is proportional to the number of observed variables.
That is, the sample size can be expressed as N p (resp.
N q ), where p , q are the number of observed variables in both models, and N is selected to be { 200 , 500 } .
In this case, the mixing matrix needs to be estimated using ICA methods.
Note that the overall mixing matrices in both problems have more columns than rows, therefore we need to use an overcomplete ICA method.
The overcomplete ICA method that we use is Reconstruction ICA [14].
We assume that the number of sources is known for overcomplete ICA.
In linear SEM-ME, we select p = 3 , 4 , 5 , 6 , 7 , 8 .
We assume that when p = 3 , 4 , 5 , there is only one unobserved non-leaf node, and each leaf node is observed with probability 0.5.
Under all three settings, the overall mixing matrix is of dimension p × ( p + 1) .
Similarly, when p = 6 , 7 , 8 , there are two unobserved non-leaf nodes, and each leaf node is observed with probability 0.5.
In linear SEM-UR, we select q = 3 , 4 , 5 , 6 , 7 , 8 .
Similar to SEM-ME, we assume that when q = 3 , 4 , 5 , there is only one latent confounder, and the overall mixing matrix is of dimension q × ( q + 1) .
When p = 6 , 7 , 8 , there are two latent confounders in the system.
We set p e = min(0 .
5 / ( p − 1))
(resp.
5 / ( q − 1) ) in both models, and the exogenous noises are independently drawn from uniform distributions between [ − 0 . 5 , 0 . 5] .
We use bootstrapping method [6] to improve the stability of the recovery of overcomplete ICA with 50 iterations.
In each iteration, we randomly select 80% of the sampled data, and get an estimate of the mixing matrix using
Reconstruction ICA.
After each iteration, we repermute and rescale the columns of the recovered mixing matrix [Figure] [Figure] such that they are aligned with the mixing matrix recovered in the ﬁrst iteration.
Lastly, we use hypothesis testing to prune the recovered mixing matrix and remove the entries whose absolute value is below a threshold of 0.1 in SEM-UR and 0.2 in SEM-ME, with 95% conﬁdence level.
We compared our DOG algorithm with AOG and ICA-LiNGAM for both models (SEM-ME and SEM-UR), and we record the average of the performance metrics after 50 graphs are generated for each model.
The performance of recoveries for linear SEM-ME and SEM-UR are shown in Figure 9 and Figure 10, respectively.
We observe that the performance of ICA-LiNGAM depends on the number of unobserved variables/latent confounders:
There is a sharp decrease in the performance of ICA-LiNGAM when we increase the number of unobserved variables/latent confounders (from 1 unobserved/latent variable for p = 3 , 4 , 5 to 2 unobserved/latent variables for p = 6 , 7 , 8 ).
On the contrary, DOG and AOG methods outperform ICA-LiNGAM for both models.
This implies that developing methods for latent variable models is necessary.
Besides, our DOG method outperforms AOG methods for both models, which demonstrates the effectiveness of our proposed algorithm.
E.4 Plots of the error bars
Figure 4 in the main text includes three algorithms and multiple experimental settings.
We chose not to include error bars in that ﬁgure so that the presentation does not become cluttered.
Instead, here, we report the error bars for the simulation results in Figure 4, for all three algorithms.