Independence Testing-Based Approach to Causal Discovery under Measurement Error and Linear Non-Gaussian Models
Abstract
Causal discovery aims to recover causal structures generating the observational data.
Despite its success in certain problems, in many real-world scenarios the observed variables are not the target variables of interest, but the imperfect measures of the target variables.
Causal discovery under measurement error aims to recover the causal graph among unobserved target variables from observations made with measurement error.
We consider a specific formulation of the problem, where the unobserved target variables follow a linear non-Gaussian acyclic model, and the measurement process follows the random measurement error model.
Existing methods on this formulation rely on non-scalable over-complete independent component analysis (OICA).
In this work, we propose the Transformed Independent
Noise ( TIN ) condition, which checks for independence between a specific linear transformation of some measured variables and certain other measured variables.
By leveraging the non-Gaussianity and higher-order statistics of data, TIN is informative about the graph structure among the unobserved target variables.
By utilizing TIN , the ordered group decomposition of the causal model is identifiable.
In other words, we could achieve what once required OICA to achieve by only conducting independence tests.
Experimental results on both synthetic and realworld data demonstrate the effectiveness and reliability of our method.
Introduction
Discovery of causal relations is a fundamental goal of science.
To identify causal relations from observational data, known as causal discovery, has thus drawn much attention in various scientific fields, including economics, biology, and social science [45, 27, 11].
Methods for causal discovery can be roughly categorized into constraint-based ones (e.g.,
PC [43]), score-based ones (e.g., Greedy Equivalence Search (GES) [8]), and ones based on structural equation models (SEM) [38, 17, 51].
Almost all these methods assume that the recorded values are values of the variables of interest, which however, is usually not the case in real-world scenarios.
Some variables may be impossible to observe or quantify, so recorded values are actually a proxy of them (e.g., measure one’s mental status by survey questionnaire), and some variables, though quantifiable, may subject to error introduced by instruments (e.g., measure brain signals using functional magnetic resonance (fMRI)).
The difference between quantities of interest and their measured value is termed as measurement error [12].
Measurement error adversely impairs causal discovery [28, 24, 36].
The measuring process can be viewed as directed edges from underlying variables of interest (unobservable) to measured values (observable), and the d-separation patterns on underlying variables typically do no hold on measured ones.
Consider the causal effects from factory emissions to air quality then to residents’ lung health , as shown in Figure 1, while we only have corresponding measured quantities: chimney statistics , PM 2 . 5 , and hospital reported cases .
Though and are independent given , and are however dependent given PM 2 . 5 .
If measurement error is severe, and even tend to be marginally independent [53, 36], which makes PM 2 .
5 look like a collider (common child).
One might thus incorrectly infer that, lung cancer causes air pollution.
In fact, such measurement error is always a serious threat in environmental epidemiologic studies [30, 10].
Denote by ˜ X
= { ˜ X i } ni =1 the latent measurement-error-free variables and X = { X i } ni =1 observed ones.
While there are different models for measuring process [12, 6, 49], in this paper, we consider the random measurement error model [36], where the observed variables are generated from the latent measurement-error-free variables ˜
X i with additive random measurement errors E = {
[Equation] i
[Equation] [Equation]
Measurement errors E are assumed to be mutually independent and independent of ˜ X .
We assume causal sufficiency relative to ˜ X (i.e., no confounder of ˜ X who does not have a respective measurement), and focus on the case where ˜ X is generated by a linear, non-Gaussian, acyclic model (LiNGAM [38], see §3.1).
Note that here w.l.o.g., the linear weights of { ˜ X i → X i } ni =1 are assumed to be one (since we do not care about scaling).
Generally, if observations are measured by X i =
c i ˜
X
i + E i with weights { c i } ni =1 not necessarily being one, all results in this paper still hold.
The objective of causal discovery under measurement error is to recover causal structure among latent variables ˜ X , denoted as ˜ G , a directed acyclic graph (DAG), from contaminated observations
X .
As illustrated by Figure 1, causal discovery methods that utilize (conditional) independence produce biased estimation (see Proposition 1 for details).
SEM-based methods also typically fail to find correct directions, since the SEM for ˜
X usually do not hold on X .
Unobserved ˜ X are actually confounders of X , and there exists approaches to deal with confounders, such as Fast Causal Inference (FCI [44]).
However, they focus on structure among observed variables instead of the unobserved ones, which is what we aim to recover here.
With the interest for the latter, another line of research called causal discovery with latent variables is developed, which this paper is also categorized to.
However, existing methods [41, 42, 47, 46, 1, 23, 50, 35] cannot be adopted either, since they typically require at least two measurements (indicators) for each latent variable, while we only have one for each here (and is thus a more difficult task).
Specifically on the measurement error problem, [16] proposes anchored causal inference in the binary setting.
In the linear Gaussian setting, [53] presents identifiability conditions by factor analysis.
A main difficulty here is the unknown variances of the measurement errors E , otherwise the covariance matrix of ˜ X can be obtained and readily used.
To this end, [2] provides an upper-bound of E and [34] develops a consistent partial correlations estimator.
In linear non-Gaussian settings (i.e., the setting of this paper), [54] shows that the ordered group decomposition of ˜ G , which contains major causal information, is identifiable.
However, the corresponding method relies on over-complete independent component analysis (OICA [19]), which is notorious for suffering from local optimal and high computational complexity [39, 18].
Hence, the identifiability results in [54], despite the theoretical correctness, is far from practical achievability.
The main contributions of this paper are as follows: 1) We define the T ransformed I ndependent N oise ( TIN ) condition, which finds and checks for independence between a specific linear transformation (combination) of some variables and others.
The existing Independent Noise ( IN [40]) and Generalized Independent Noise ( GIN [50]) conditions are special cases of TIN .
We provide graphical criteria of TIN , which might further improve identifiability of causal discovery with latent variables.
3) We exploit TIN on a specific task, causal discovery under measurement error and LiNGAM, and identify the ordered group decomposition .
This identifiability result once required computationally and statistically ineffective OICA to achieve, while we achieve it merely by conducting independence tests.
Evaluation on both synthetic and real-world data demonstrate the effectiveness of our method.
Motivation: Independence Condition and Structural Information
The example in Figure 1 illustrates how the (conditional) (in)dependence relations differ between observed X and latent ˜ X , and thus lead to biased discovery results.
To put it generally, we have,
Proposition 1 ( rare d-separation) .
Suppose variables follow random measurement error model defined in Equation (1) .
For disjoint sets of observed variables Z , Y , S and their respective latent ones ˜ Z , ˜ Y , ˜ S , d-separation Z ⊥⊥ d Y | S holds, only when marginally ˜ Z ⊥⊥ d ˜ Y , and ˜ Z ⊥⊥ d ˜ Y | ˜
S hold.
By ‘rare’ we mean that the d-separation patterns among ˜ X usually do not hold among X (except for rare marginal ones), since the observed variables are not causes of any other (though the latent variables they intend to measure might be).
For example, consider the underlying ˜ G to be chain structure (Figure 2a) and fully connected DAG (Figure 2c).
There exists no (conditional) independence on either, and PC algorithm will output just a fully connected skeleton on both cases.
Then, without (conditional) independence (which is non-parametric) to be directly used, can we create independence , by leveraging the parametric assumption (LiNGAM) and benefit from non-Gaussianity?
Naturally we recall the Independent Noise (IN) condition proposed in Direct-LiNGAM [40]:
Definition 1 ( IN condition) .
Let Y i be a single variable and Z be a set of variables.
Suppose variables follow LiNGAM.
We say ( Z , Y i ) satisfies IN condition, denoted by IN( Z , Y i ) , if and only if the residual of regressing Y i on Z is statistically independent to Z .
Mathematically, let ˜ ω be the vector of regression coefficients, i.e., ˜ ω :
= cov( Y i , Z ) cov( Z , Z ) − 1 ; IN( Z , Y i ) holds iff Y i − ˜ ω
⊺ Z ⊥⊥ Z .
Here “ cov ” denotes the variance-covariance matrix.
IN identifies exogenous (root) variables, based on which the causal ordering of variables can be determined (Lemma 1 in [40]).
However, IN cannot be applied to measurement error model.
With hidden confounders ( ˜ X ) behind observed X , independence between regressor and residual typically does not exist on any regression among X .
In fact, X follows errors-in-variables models [15, 7], for which the identifiability w.r.t.
˜ G is not clear.
However, we might still benefit from this idea to leverage non-Gaussianity of exogenous noises.
Consider the Figure 1 example and abstract it to ˜ X 1 → a ˜ X 2 → b ˜ X 3 with { ˜ X
i → 1 X i }
i =1 .
Although
IN does not hold on any of X , interestingly, there exists a linear transformation of observations bX 2 − X 3 , which contains only { ˜ E 3 , E 2 , E 3 } ( ˜ E 1 and ˜ E 2 are cancelled out) and shares no common non-Gaussian noise term with X 1 .
Hence, by the Darmois–Skitovich theorem [21], bX 2 − X 3 is independent of X 1 .
This finding is echoed in Generalized Independent Noise ( GIN [50]) condition:
Definition 2 ( GIN condition) .
Let Z and Y be two sets of random variables that follow LiNGAM.
We say ( Z , Y ) satisfies the GIN condition, denoted by GIN( Z , Y ) , if and only if the following two conditions are satisfied: 1) There exists nonzero solution vectors ω ∈ R | Y | to equation cov( Z , Y ) ω
= 0 , and 2)
Any such solution ω makes the linear transformation ω ⊺ Y independent of Z .
Here | Y | denotes the dimensionality of Y .
The intuition of GIN is that, despite no independent residual by normal regression, it is possible to realize independent “pseudo-residuals” [5] by regressing with “reference variables” .
[50] shows IN as a special case of GIN (Proposition 2), and further gives graphical criteria (Theorem 2), based on which a recursive learning algorithm is developed to solve the latent-variable problem.
Each latent variable is required to have at least two observations.
Interestingly we find that, in measurement error models, if each latent variable ˜ X
i has two measurements
X i 1 , X i 2 , then the GIN condition can be readily used to fully identify the structure of ˜ G , which is already a breakthrough over existing methods [42, 47, 46, 23].
See Appendix B for the whole procedure.
With regard to our more challenging task where each ˜ X
i has only one measurement X i , a natural question is that, can GIN also help?
Given the example in Figure 1 illustrated above, the answer seems to be affirmative:
GIN( { X i } , X \{ X i } ) only holds for i = 1 , so the root can be identified.
More generally: ˜ X˜ X˜ X˜ X˜
Xa b c d (a) ˜ X˜ X˜ X˜ X˜
Xa c
b d e (b) ˜ X˜
X˜ X 3 ˜
Xa b c d e f (c) ˜ X˜ X˜ X˜ X˜ X(d)
Example 1 ( GIN on chain structure) .
Consider cases where the underlying graph ˜ G is a chain structure with n ( n ≥ 3 ) vertices and directed edges { ˜ X i
→ ˜ X i +1 } n − 1 i =1 .
Figure 2a is an example with n = 5 .
We find that GIN( Z = X 1 , Y = X 2 , 3 , 4 , 5 ) holds (where X 2 , 3 , 4 , 5 denotes { X 2 , X 3 , X 4 , X 5 } ; same below), with solution ω
=
[ − bx − bcy − bcdz, x, y, z ] ⊺ , x, y, z ∈ R .
ω ⊺ Y cancels noise components in Y that are also shared by Z , and
thus ω ⊺ Y ⊥⊥ Z .
However, GIN( X i , X \{ X i } ) is violated for any other i ̸
= 1 (see Example 12 for a detailed derivation).
With this asymmetry, the latent root ˜
X 1 can be identified.
Furthermore, GIN( X i , X i +1 , ··· ,n ) holds for any i = 1 , ··· , n − 2 .
Example 1 might give us an intuition that by recursively testing
GIN (over the newly found subroot and the remaining vari- ables), we could identify the causal ordering of first n − 2 variables for any DAG.
However, this is over-optimistic thanks to the sparsity of chain structure.
Consider a denser structure:
Example 2 ( GIN on fully connected DAG) .
Consider cases where ˜ G is a fully connected DAG with n ( n ≥ 3 ) vertices and directed edges { ˜ X i
→ ˜ X j } for every i < j .
Figure 2c is an example with n = 4 .
We first find that GIN( X 1 , X 2 , 3 , 4 ) holds.
However, in contrast to the chain structure,
GIN( X 2 , X 3 , 4 ) does not hold - there is no way to cancel both ˜ E 1 , 2 from X 3 , 4 .
More generally we have: GIN( X 1 , X 2 , ··· ,n ) , GIN( X 1 , 2 , X 3 , ··· ,n ) , ··· , GIN( X 1 , ··· ,k , X k +1 , ··· ,n ) , k
= ⌊ ( n − 1) / 2 ⌋ .
Since a fully connected DAG is the densest extreme, Example 2 might give us an intuition that GIN could identify at least the causal ordering of the first half of variables.
Unfortunately, this is still over-optimistic, since we could not know beforehand the structure type of ˜ G .
Example 3 ( GIN on chain structure with triangular head ) .
Figure 2b shows a variation of chain structure, with edges ˜ X 1 → ˜ X 2 , ˜ X 1 → ˜ X 3 , ˜ X
→ ˜ X 2 , and { ˜ X i
→ ˜ X i +1 } n − 1 i =3 .
We name it “chain structure with triangular head ”.
Interestingly, GIN( X 2 , X 3 , ··· ,n ) , which is satisfied on Figure 2a, is also satisfied here.
E.g., in Figure 2b ( n = 5 ), the solution ω
=
[ − dx − dey, x, y ] ⊺ , x, y ∈ R .
ω ⊺ Y cancels ˜
E 1 , 3 from Y , and thus ω ⊺ Y ⊥⊥ Z .
Actually, the chain structure with triangular head ˜ G ▽ C is unidentifiable with chain structure ˜ G C w.r.t.
GIN conditions, i.e., for any two sets of observed variables Z , Y ⊆ X , GIN( Z , Y ) holds on ˜ G ▽ C if and only if GIN( Z , Y ) holds on ˜ G C .
Consequently, if directly using any recursive algorithm by GIN as in Example 1, the output causal ordering would still be ˜ X 1 , ˜ X 2 , ˜ X 3 , ··· , which is incorrect due to ˜ X 3 → ˜ X 2 in the triangular head.
The above examples show that it is not as simple as it seems to use GIN for one-measurement model: only part of the causal ordering can be identified, and worse yet, rather complicated error correction is needed to deal with possible incorrect orderings.
However, after a closer look at the unidentifiable examples above, we find that actually more information can be uncovered beyond the GIN condition:
Example 4 (Asymmetry beyond GIN ) .
1) Consider Example 2 where only the root variable ˜
X 1 can be identified by GIN and ˜ X 2 , 3 , 4 are unidentifiable , i.e., permutation of the labeling of X 2 , 3 , 4 will still preserve the GIN condition over any two subsets Z , Y .
However, there actually exists an asymmetry between X 2 and X 3 , 4 : we could construct linear transformation of X 1 , 3 , 4 : cd − bed
X 1 + df + e d X 3 −
X 4 s.t.
it is independent of X 2 , while there exists no such linear transformation of X 1 , 2 , 4 to be independent of X 3 , and also for X 1 , 2 , 3 to X 4 . 2) Consider Examples 1 and 3 where ˜ G C
and
˜ G ▽ C are unidentifiable w.r.t.
GIN conditions.
Let Z :=
X 4 and Y :
= X 1 , 2 , 3 , GIN( Z , Y ) is violated on both graphs.
However, an asymmetry actually exists: on ˜ G ▽ C , we could construct aX 1 − X 2 + bX(which cancels ˜
E 1 , 3 ) to be independent to X 4 , while this is impossible on ˜ G C .
To put simply, the motivation of independent “pseudo-residual” behind GIN actually limits the power of non-Gaussianity, with the coefficients vector ω only characterized from variance-covariance matrix (2nd-order).
There are actually two cases for GIN( Z , Y ) to be violated: 1) though not all solution ω makes ω ⊺ Y ⊥⊥ Z , there exists non-zero ω s.t. ω ⊺ Y ⊥⊥ Z , and 2) there naturally exists no non-zero ω s.t. ω ⊺
Y ⊥⊥ Z .
The original GIN cannot distinguish between these two cases.
Hence in this paper, we first aim to distinguish between the two, generalizing GIN condition to TIN condition.
With Transformed Independent Noise Condition
In the above discussion, one can see that the presence of measurement error affects the conditional independence relations among the variables and the independent noise condition.
However, we will show in Theorem 3 (§4) that a specific type of independence conditions are shared between the underlying error-free variables and the measured variables with error.
In this section, we will formulate such independence conditions and investigate their graphical implications for error-free variables (i.e., variables generated by the LiNGAM without measurement error).
In §4, we will then extend the results to the measured variables.
Please note that in contrast to other sections, the notation used in this section, including X , Y , and S , denotes error-free variables generated by the LiNGAM.
3.1 Notations
Let G be a directed acyclic graph with the vertex set V ( G ) =
[ n ] := { 1 , 2 , · · · , n } and edge set
E ( G ) .
A directed path P =
( i 0 , i 1 , · · · , i k ) in G is a sequence of vertices of G where there is a directed edge from i j to i j +1 for any 0 ≤ j ≤ k − 1 .
We use notation i ⇝ j to show that there exists a directed path from vertex i to j .
Note that a single vertex is also a directed path, i.e., i ⇝ i holds.
Let Z ⊆
[ n ] be a subset of vertices.
Define ancestors Anc( Z ) :
=
{ j |∃ i ∈ Z , j ⇝ i } .
Note that Z ⊆
Anc( Z ) .
Further let S be a subset of vertices.
We use notation i ⇝ (cid:26)(cid:26)
[ S ] j to show that there exists a directed path from vertex i to j without passing through S , i.e., there exists a directed path
P =
( i, m 0 , · · · , m k , j ) in G s.t.
i, j ̸∈ S and m l ̸∈ S for any 0 ≤ l ≤ k .
Define ancestors outside S accordingly: for two vertex sets Y , S , denote ancestors of Y that have directed paths into Y without passing through S as Anc out( S ) ( Y ) :
= { j |∃ i ∈ Y , j ⇝ (cid:26)(cid:26)
[ S ] i } .
Note that the graphical definitions here can also be translated to trek [47] language (see Appendix C for details).
Assume random variables X :=
{ X i } i ∈ [ n ] are generated by LiNGAM w.r.t.
graph G , i.e.,
[Equation]
where E = { E i } i ∈ [ n ] are corresponding mutually independent exogenous noises.
A is the adjacency matrix where entry
A j,i is the linear weight of direct causal effect of variable X i on X j .
A j,i ̸
=
if and only if there exists edge i → j .
X can also be written directly as a mixture of exogenous noises
X = BE .
If the entry of mixing matrix B j,i ̸ = 0 , then i ∈ Anc( { j } ) .
Note that here and in what follows, we use boldface letters A , B to denote matrices, and use boldface letters S , W , X , Y , Z with notation abuse: it can denote vertices set, respective random variables set, or random vector.
When we say “two variables sets Z , Y ”, if not otherwise specified, Z , Y need not be disjoint.
3.2 Independent Linear Transformation Subspace and its Characterization
We first give the definition and characterization of the independent linear transformation subspace .
Definition 3 (Independent linear transformation subspace) .
Let Z and Y be two subsets of random variables.
Suppose the variables follow the linear non-Gaussian acyclic causal model.
Denote:
[Equation]
By the property of independence, Ω Z ; Y is closed under scalar multiplication and addition, and thus is a subspace in R | Y | .
In fact, Ω Z ; Y can be characterized as a nullspace as follows:
Theorem 1 (Characterization of Ω Z ; Y ) .
For two variables subsets Z and Y , Ω Z ; Y satisfies:
[Equation]
where null( · ) denotes nullspace.
B Y , nzcol( B Z , : ) denotes the submatrix of mixing matrix B , with rows indexed by Y and columns indexed by nzcol( B Z , : ) .
nzcol( B Z , : ) denotes the column indices where the submatrix B Z , : has non-zero entries.
nzcol( B Z , : ) actually corresponds to the exogenous noises that constitute Z .
Particularly, if assuming “if i ⇝ j then B j,i ̸
= 0 ”, then, nzcol( B Z , : ) =
Anc( Z ) .
Proof of Theorem 1 is straight-forward by the Darmois–Skitovich theorem [21]: linear transformation ω ⊺ Y ⊥⊥ Z if and only
if ω ⊺ Y shares no common non-Gaussian exogenous noise components with Z .
Example 5 (Revisiting examples in §2 from Ω Z ; Y perspective) .
For illustration, now we revisit the examples in §2 from the perspective of independent linear transformation subspace.
a    ab b   abc bc c  abcd bcd cd d
   ,   0a + bcb 0c  0cd
d 1cde 0 de e   
 ,    a   ad + b d  a ( df + e ) + bf + c df + e f     (5)
Equation (5) shows the corresponding mixing matrix B for graph ˜ G in Figures 2a to 2c, respectively.
Suppose we have access to underlying variables ˜
X i and only focus on ˜ G .
Colored blocks denote submatrices of B . 1)
For the fully connected DAG (Figure 2c, the right matrix), to identify the root ˜ X 1 , GIN( ˜ X 1 , ˜ X 2 , 3 , 4 ) is satisfied, corresponding to Ω Z ; Y = null( ⊺ ) .
For ( Z , Y ) :
= ( ˜ X 2 , ˜ X 3 , 4 ) or ( ˜ X 4 , ˜ X 1 , 2 , 3 ) , there exists no non-zero ω s.t. ω ⊺ Y ⊥⊥ Z , because the lower part and are full row rank.
However, if we set ( Z , Y ) :=
( ˜ X 2 , ˜ X 1 , 3 , 4 ) , we actually have ˜
X 2 ⊥⊥ cd − bed ˜
X
1 + df + e d ˜ X 3 − ˜ X 4 , because the stacked two parts of has rank 2 < 3 - though
GIN is still violated because cov( Z , Y ) has rank 1 < 2 . 2)
For the chain structure
˜ G C
(Figure 2a, the left matrix) and chain with triangular head ˜ G ▽ C (Figure 2b, the middle matrix), GIN is satisfied on ( ˜ X 1 , ˜ X 2 , 3 , 4 , 5 ) , ˜ X 2 , ˜ X 3 , 4 , 5 ) , ( ˜ X 3 , ˜ X 4 , 5 ) , with the B Y , Anc( Z ) submatrices being , , respectively.
Note that though the shape of submatrices are different between ˜ G C and ˜ G ▽ C , their ranks are always equal, and is thus unidentifiable by GIN .
However, let ( Z , Y ) :
= ( ˜ X 4 , ˜ X 1 , 2 , 3 ) , an asymmetry actually exists: in ˜ G ▽ C , ˜ X 4 ⊥⊥ a ˜ X 1 − ˜ X
2 + b ˜ X 3 , because in the right matrix has rank 2 < 3 , while in the left matrix is full row rank, so there is no such independence in ˜ G C .
3.3 Graphical Criteria of Independent Linear Transformation Subspace
§3.2 characterizes Ω Z ; Y by submatrix of B , which entails information on graph structure and edge weights.
The following sections go one step further, explicitly exhibit the graphical criteria of Ω Z ; Y , and investigate how Ω Z ; Y could help to identify the causal structure.
We first give the assumption:
Assumption 1 (Rank faithfulness) .
Denote by B ( G ) the parameter space of mixing matrix B consistent with the DAG G .
For any two subsets of variables Z , Y ⊆ X , we assume that
[Equation]
Roughly speaking, we assume there are no edge parameter couplings to produce coincidental low rank.
This is slightly stronger than “ nzcol( B Z , : ) =
Anc( Z ) ”.
See Appendix F.1 for elaboration.
In other words, the graphical criteria holds on a dense open subset of the edge parameter space.
Note that Assumption 1 is the only other parametric assumption we make besides LiNGAM throughout the paper, where violation of Assumption 1 is of Lebesgue measure 0, and LiNGAM is testable.
Graphically, we first define vertex cut , and then give the graphical criteria based on it:
Definition 4 (Vertex cut) .
Let W , Y , S be three vertices subsets of V ( G ) which need not be disjoint.
We say that S is a vertex cut from W to Y , if and only if there exists no directed paths in G from W to Y without passing through S .
With basic notations in §3.1, the following statements are equivalent:
1) S is a vertex cut from W to Y ; 2) ∀ i ∈ W , j ∈ Y , i ⇝ (cid:26)(cid:26)
[ S ] j does not hold; 3) Anc out( S ) ( Y ) ∩ W = ∅ ; 4) S ’s removal from G
ensures
there is
no
directed paths from W \ S to Y \ S .
More details on vertex cut are in Appendix C. Note that trivially W itself and Y itself are vertex cuts.
Theorem 2 (Graphical criteria of Ω Z ; Y ) .
Let Z , Y be two subsets of variables (vertices), we have:
[Equation]
a
[Equation]
cut from Anc
[Equation]
Z
[Equation]
to Y } .
[Equation]
where dim(Ω Z ; Y ) denotes the dimension of the subspace Ω Z ; Y , i.e., the degree of freedom of ω .
By Theorem 1, | Y | − dim(Ω Z ; Y ) is exactly the rank of B Y , Anc( Z ) .
Theorem 2 can then be proved by a combinatorial interpretation of mixing matrices’ determinants.
See Appendix A for details.
From the graphical view, a vertex cut S from Anc( Z ) to Y means that the causal effect from Z and the common causes of Z and Y , if there is any, must affect Y through S - there is no any bypass.
To interpret in trek-separation [47] language, it is equivalent to “ ( ∅ , S ) t-separates ( Z , Y ) ” 2 .
Z
[Equation]
Y
N o s u c h b
[Equation]
p
[Equation]
s
[Equation]
✗
[Equation]
( Z ) to Y .
From the noise view, the noise components that constitute variables Z are exactly the exogenous noises corresponding to vertices Anc( Z ) .
All these noises must contribute to Y (if any) via S , and thus Y can be written as
Y =
L S + E ′ Y , where L denotes a linear transformation, and
E ′ Y ⊥⊥ Z .
Denote by S ∗ Z ; Y the critical vertex cut from Z to Y 3 , the noise components of ω ⊺ Y is exactly the exogenous noises corresponding to Anc out( S ∗ Z ; Y ) ( Y ) .
3.4 Formal Definition of TIN
With the graphical criteria given in §3.3, we could use it for structure inference as long as we have the (dimension of) independent linear transformation subspace Ω Z ; Y .
In §5 we will comprehensively discuss methods to estimate Ω Z ; Y , while for now, we could just safely suppose we have Ω Z ; Y : since independence is testable, theoretically one could get Ω Z ; Y at least by traversing over R | Y | .
[Table]
Definition 5 ( TIN function) .
Let Z and Y be two subsets of observed random variables.
Suppose variables follow LiNGAM.
We define a function TIN as follows:
[Equation]
TIN is a function that takes as input two random vectors Z , Y and returns an integer in range [0 , | Y | ] .
Note that this is different to IN or GIN , which returns only a bool value (satisfied or not).
GIN can be viewed as a special case of TIN , i.e., GIN( Z , Y ) is satisfied if and only if TIN( Z , Y ) =
rank(cov
( Z , Y ))
<
| Y | (where rank(cov( · )) can be characterized by [47]).
IN can also be viewed as a special case of TIN , i.e., IN( Z , Y i ) is satisfied if and only if TIN( Z , Z ∪ { Y i } ) =
| Z | .
Example 6 (Review TIN graphical criteria on graphs) .
For better understanding of TIN , we demonstrate some representative examples of TIN over different graph structures.
Results are shown in Table 1.
We use the four graph structures in Figure 2.
For illustration, assume we have access to latent variables to directly conduct TIN over ˜ X .
Graphical criteria correspond to ˜ G .
For readability, we omit all ˜ notation.
We assume there are 5 vertices in the fully connected DAG (Figure 2c): consider for example, GIN( X 3 , X 1 , 2 , 4 , 5 ) does not hold, since rank(cov( Z , Y )) is only one (restricted by Z size).
However, there exists ω with degree of freedom 1 s.t. ω
⊺ Y ⊥⊥ Z - this corresponds to a 3-variables vertex cut X 1 , 2 , 3 ( Anc( Z ) itself).
Consider on Figure 2d, TIN( X 1 , 2 , X 4 , 5 ) = 1 , corresponding to the 1-variable vertex cut X 4 .
This example shows that vertex cut does not necessarily yield a d-separation pattern (but blocking, instead), since here X 5 ̸⊥⊥ X 1 | X 4 .
The graphical criteria for TIN over any two sets of variables Z , Y are given as above.
Interestingly, we find that a special type of TIN is particularly simple in form and informative for structure inference:
Lemma 1 (Graphical criteria of TIN on one-and-others) .
Assume we have access to all variables X on a DAG G .
For each singleton variable X i ∈ X , let Z :=
{ X i } and Y :=
X \{ X i } , we have,
[Equation]
Due to a page limit, here we only give the main criteria of TIN .
See Appendix D for more properties.
TIN Condition-Based Method for Measurement Error Models
In §3 we propose TIN condition over general LiNGAM model and give its graphical criteria.
In this section, we aim to exploit TIN on our specific task of interest: measurement error models.
[Figure]
4.1 Identifiability of Ordered Group Decomposition
Under our problem setting where ˜ X follows LiNGAM, identifiability results can greatly benefit from the non-Gaussianity of data.
[54] shows that the ordered group decomposition of ˜ G is identifiable.
First review definitions:
Definition 6 (Pure leaf child) .
On a DAG G , a vertex j is said to be a “pure leaf child” of another vertex i , iff j is a leaf node with only one parent, i .
Particularly, if a variable ˜ X j is a pure leaf child of ˜ X i in ˜ G , then ˜
X
j and ˜ X j are naturally unidentifiable (e.g., ˜ X 2 and ˜ X 7 in Figure 4).
The reason is that the exogenous noise ˜ E j of leaf ˜
X j does not contribute to any other variables, just like its measurement noise E j .
Consequently, X i and X j can be viewed as two equally positioned measurements of ˜ X i , without any asymmetry.
Definition 7 (Ordered group decomposition) .
Consider the underlying causal model ˜ G .
The ordered group decomposition can be defined by the following procedure: at each step, remove root vertices and their pure leaf children nodes (if any) from graph, and append the removed ones as a new group.
Repeat this procedure to remove root vertices from the remaining graph, until the graph is empty.
This procedure is equivalent to Definition 2 in [54] 4 .
See Figure 4: roots are removed from graph in order (color dark to light).
The ordered group decomposition is ˜ X 1 → ˜ X 2 , 7 → ˜ X 3 , 6 → ˜ X 4 , 5 .
4.2 Exploit TIN to Identify Ordered Group Decomposition
Our task is to recover ˜ G over ˜ X by testing TIN over X .
In notations above, ˜ X and X are strictly distinguished to denote unobserved and observed variables.
However, actually this can be escaped:
Theorem 3 (Equivalence of TIN over latent and observed variables) .
For two disjoint observed variables subsets Z , Y , and their respective underlying latent variables subsets ˜ Z , ˜ Y
[Equation]
Theorem 3 can either be proved by graphical criteria, or by showing how the rank of submatrices of
B is preserved among latent and observed variables.
Interestingly, recall Proposition 1, we show the inequivalence of d-separation and IN condition held between ˜ X and X ( raw independence ).
However,
TIN condition, which essentially finds transformed independence , holds equivalently on latent and observed variables.
With this equivalence, we can conduct TIN over observed variables X just as if we have access to the latent ones ˜ X .
Thus the problem can be restated without measurement error: assuming causal sufficiency, by only using TIN over disjoint Z , Y , to what extent is G identifiable?
Under this equivalent problem, Lemma 1 can be used directly to identify the ordered group decomposition of ˜ G : for each singleton observed variable X i , test TIN and assign an order ord ( X i ) :
=
TIN( { X i } , X \{ X i } ) .
Group variables with same ord , and then sort the groups by their orders.
Obviously, the ordered groups obtained by this procedure is consistent with Definition 7.
Example 7 ( TIN on Figure 4) .
ord ( X i ) 7 i =1 are respectively 1 , 2 , 3 , 4 , 4 , 3 , 2 (can verify by characterization or graphically), so the group ordering is identified as ˜ X
→ ˜ X 2 , 7 → ˜ X 3 , 6 → ˜ X 4 , 5 .
Estimating Linear Independent Transformation Subspace Ω Z ; Y
In the above sections we safely assume that we can always get Ω Z ; Y , since theoretically independence is testable and ω can be exhaustively traversed.
In this section, we give practical methods to estimate Ω Z ; Y .
Due to page limit we only give a summary for each.
Please see Appendix E for details.
5.1 Tackling Down to Subsets of Y
Theorem 4 ( TIN over Y subsets) .
For two variables sets Z , Y , TIN( Z , Y ) =
k (assume k > 0 ), iff the following two conditions hold: 1) ∀ Y ′ ⊆
Y with | Y ′ |
= k
+ 1 (if any), there exists non-zero ω
s.t. ω ⊺ Y ′ ⊥⊥ Z ; and 2) ∃ Y ′ ⊆
Y with | Y ′ |
= k , there exists no non-zero ω
s.t. ω ⊺ Y ′ ⊥⊥ Z .
This transforms the task of estimating the dimension of Ω Z ; Y to a simpler one: counting size of the subsets Y ′ .
Instead of all independence, here we only need to check existence of independence.
5.2 Constrained Independent Subspace Analysis (ISA)
Conduct Independent Subspace Analysis (ISA [48]) over variables Z and Y in the following form:
[Equation]
where the de-mixing matrix is masked to only update the lower-right | Y | × | Y | block W YY , with upper-left | Z |×| Z | block fixed as the identity and elsewhere fixed as zero.
The independence between
W YY Y as a group to Z is maximized.
Since W YY is invertible, its rows span the whole R | Y | , so the maximum number of rows that achieves W YY ⊺ i, : Y ⊥⊥ Z is exactly the dimension of Ω Z ; Y .
5.3 Stacked Cumulants: Ranks Stopped Increasing
Definition 8 (Stacked 2D slices of cumulants) .
For two variables sets Z , Y and order k ≥ 2 , define:
[Equation]
[Equation]
C
[Equation]
⊺ , where C ( k ) is matrix with C ( k )
i
[Equation]
Ψ ( k ) Z ; Y is a ( k − 1) | Z | × | Y | matrix that vertically stacks 2D cumulants slices between Z , Y with orders from 2 to k .
When k = 2 it is cov
( Z , Y ) .
Ranks of Ψ ( k ) Z ; Y in the sequence k
= 2 , 3 , ... satisfies:
F u ll y c onn ec t e d
DAG 3 4 5 6 7 8 9 10 # Variables 0.
K - t d i s t a n c e ( l o w e r b e tt e r )
NSR=0.
[Figure]
[Figure]
[Figure]
NSR
[Figure]
[Figure]
[Figure]
NSR
[Figure]
NSR
[Figure]
ICA
[Figure]
PC
GES
NOTEARS
SCORE
C h a i n s t r u c t u r e
[Figure]
K - t
[Figure]
i s t a n c e ( l o w e r b e tt e r )
[Figure]
Theorem 5 ( rank(Ψ ( k ) Z ; Y ) stopped increasing) .
For two variables sets Z , Y , there exists a finite order k ≥ 2 s.t.
rank(Ψ ( k +1)
Z ; Y )
=
rank(Ψ ( k ) Z ; Y ) .
Moreover, TIN( Z , Y ) equals rank(Ψ ( k ) Z ; Y ) with this k .
We will show detailed characterization and graphical criteria of Ψ ( k ) Z ; Y in Appendix E.1.
5.4 TIN in Two Steps: Solve Equations, and then Test for Independence
Motivation of §5.3 is that independence yields zero cumulants (not only 2nd-order uncorrelatedness), i.e., Ω Z ; Y ⊆ null(Ψ
( k ) Z ; Y ) , for any k ≥ 2 .
Hence, we could solve equations introduced at each order k and then check whether all solution ω ∈ null(Ψ ( k ) Z ; Y ) makes ω ⊺ Y ⊥⊥ Z (similar to GIN procedure).
More generally, Ω Z ; Y ⊆ null(cov( f ( Z ) , Y )) for any real-valued function f ( · ) .
E.g., solve equations system { cov(log( Z 2 ) , Y ) ω = 0 ; cov(sin( Z ) , Y ) ω = 0 ; ···} and test whether ω ⊺ Y ⊥⊥ Z holds.
Experimental Results
In this section, we evaluate the performance of TIN in recovering the ordered group decomposition of ˜ G under measurement error.
Specifically, four implementations of TIN are evaluated:
TIN-subsets (§5.1), TIN-ISA (§5.2), TIN-rank (§5.3), and TIN-2steps (§5.4).
We compare our method with PC [43], GES [8], ICA-LiNGAM [38], Direct-LiNGAM [40], CAMME-OICA [54],
NOTEARS [55], and SCORE [32].
Experiments are conducted on both synthetic and real-world data.
6.1 Synthetic Data
While our proposed method outputs correct ordered group decomposition for any ˜ G (without assumptions on graph structure), in the following simulation we consider specifically two cases: fully connected DAG (Figure 2c) and chain structure (Figure 2a), of which the ordered group decomposition are both ˜ X 1 (cid:41) ··· (cid:41) ˜
X n − 2 , ˜ X
n − 1 ,n .
We consider ˜ G with the number of vertices n = 3 , ··· , 10 .
Edges weights (i.e., the nonzero entries of matrix A ) are drawn uniformly from [ − 0 .
9 , − 0 .
5] ∪ [0 . 5 , 0 . 9] .
Exogenous noises ˜ E are sampled from uniform ∪ [0 , 1] to the power of c , c ∼ ∪ [5 , 7] , and measurement errors are sampled from Gaussian N (0 , 1) to the power of c , c ∼ ∪ [2 , 4] .
Sample size is 5 , 000 .
Observations are generated by X i =
˜
X
i + E i .
To show the effect of measurement error, we simulate with noise-to-signal ratio NSR :
= var( E i ) / var( ˜ X i ) in { 0 . 5 , 1 , 2 , 3 , 4 } .
To evaluate the output group ordering, we use Kendall tau distance [22] to the ground-truth (in range [0 , 1] , the lower the better).
For algorithms returning DAG/PDAG, its ordering is first extracted according to Definition 7.
Figureshows the results.
Each column subplot indicates an NSR setting.
The error bar is from 50 random generated instances.
We can see that TIN-subsets and TIN-2steps are two best methods (ranks first on 68 / 80 cases), which steadily stick near x-axis.
They both find/check independence by solving equations cov( f ( Z ) , Y ) ω
= 0 .
TIN-ISA is slightly inferior (though still ranks 3rd), maybe due to the bias to independence (of ω vectors) by ISA.
TIN-rank fluctuates and performs worst (among TIN s), especially on the fully connected dense case.
This might be explained by higher order cumulants’ sensitivity to outliers, and unreliable numerical rank tests (we simply use SVD and thresholding).
Among the competitors, ICA-LiNGAM and SCORE are the strongest two, which remain relatively stable with NSR growing larger (while e.g., Direct-LiNGAM deteriorates rapidly).
Interestingly, they both perform much better on fully connected DAGs than on chain structures, and ICA-LiNGAM even performs better on larger graphs than on smaller ones.
We will investigate the reason.
Consider
CAMME-OICA, it relies heavily on mixing matrix parameter initialization, and thus performs generally weak.
Weak results by PC, GES might be because of the unfair setting for them: they output
CPDAG and in both cases (fully connected and chain) here, naturally no ordering (directions) can be determined.
More experimental configuration and results discussions are in Appendix G.2.
6.2 Real-World Data
Sachs’s [33] is a real dataset that measures the expression levels of proteins in human cells under various phospholipids.
The ground-truth graph structure [33] contains 17 edges on 11 variables (cell types), of which the ordered group decomposition is {plc,pkc} (cid:41) {pip2,pip3,pka,p38,jnk} (cid:41) {raf} (cid:41) {mek} (cid:41) {erk,akt}.
Result given by TIN-subsets achieves the best distance score 0 .
33 : {plc,pkc,p38} (cid:41) {raf,mek,pip2,pka,jnk} (cid:41) {pip3} (cid:41) {erk,akt}.
This score on PC, GES, ICA-LiNGAM are 0 .
69 , 0 .respectively.
E.g., PC outputs {raf,pka} (cid:41) {mek} (cid:41) {plc,pip2,pip3,pkc,p38,jnk} (cid:41) {erk,akt}.
Experiments on another dataset, Teacher Burnout [4], also show TIN ’s good performance.
See Appendix H.5.
Conclusion and Discussions
In this work we define the T ransformed
I ndependent N oise ( TIN ) condition based on LiNGAM causal model, which finds and checks for independence between a specific linear transformation of some variables and others.
We provide graphical criteria of TIN , which might further improve identifiability of the latent-variable problem.
Specifically on causal discovery under measurement error, we exploit TIN to achieve identifiability of ordered group decomposition.
We summarize the future work as three fold: 1) For the measurement error model, in addition to the special type (one-over-others), TIN over general Z , Y pairs can further improve identifiability.
See Appendix F.2; 2) TIN now only considers the dimension of Ω Z ; Y , while parameters might also help to recover edges weights.
See Appendix F.3; and 3) Reliable estimation of Ω Z ; Y can be formulated as an orthogonal research problem.
We believe there exists more solutions.
See Appendix F.4.
Acknowledgments and Disclosure of Funding
The authors would like to thank Georges Darmois, Viktor Skitovich, Bernt Lindström, Ira Gessel,
Gérard Viennot, and Seth Sullivant for initializing the beautiful theorems that this work is built upon.
Thank Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, and Zhifeng Hao for the GIN condition.
Thanks to the anonymous reviewers and Joseph Ramsey, Zeyu Tang, Yujia Zheng, Ignavier
Ng, Justin Ding, Mengyao Lu, Haoqin Tu, Qiyu Wu, Muyang Li, Jinkun Cao, and Jinhao Zhu for helpful feedback, proofreading, and discussions.
The work was partially supported by the NSF under
Project Number A221500S001, by the NSF-Convergence Accelerator Track-D award #2134901, by NIH-NHLB1 9R01HL159805-05A1, by a grant from Apple Inc., and by a grant from KDDI Research
Inc.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See §7.
First, our proposed method specifically work when data generation model follows LiNGAM
(though this is testable).
Second, in this work we only use the proposed TIN condition in an one-and-others type to identify the ordered group decomposition , while by using over more general Z , Y pairs, identifiability results might be further improved.
We will work on this in the near future.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] Details of proofs are in Ap- pendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We have included the main functionalities of TIN , synthetic data generator for reproducibility.
Details are in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A]
Our problem setting requires no training process.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We carefully design experiments, by running experiments multiple times, including error bars in report to ensure reproducibility.
For instance, Figure 5 contains error bar information by 50 random runs on each setting.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Details included in Appendix G. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We share new assets, include main functionalities of our algorithm, and code for data simulation.
A private URL is provided.
See details in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
us- ing/curating?
[Yes] Due to page limit, the detailed discussion is available in the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
[N/A] We double-checked that the data we are using/curating contains no personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
A Proofs of Main Results A.1 Proof of Proposition
Proposition 1 ( rare d-separation) .
Suppose variables follow random measurement error model defined in Equation (1) .
For disjoint sets of observed variables Z , Y , S and their respective latent ones ˜ Z , ˜ Y , ˜ S , d-separation Z ⊥⊥ d Y | S holds, only when marginally ˜ Z ⊥⊥ d ˜ Y , and ˜ Z ⊥⊥ d ˜ Y | ˜
S hold.
Proof of Proposition 1.
The whole graph we consider is the graph ˜ G among latent variables ˜ X and measurement edges ˜
X
i
→ X i .
Consider observed variables
X i , X j and subset S .
Denote by “ des ” the descendants of some vertices on graph.
By definition of d-separation, if X i ⊥⊥ d X j | S , then for every undirected path p linking X i and X j (if there is any), p is blocked by S , i.e., either 1) there exists a collider W , s.t. W ̸∈ Z and des( W ) ∩ S = ∅ , or 2) there exists a non-collider W s.t. W ∈ S .
Since observed variables are all leaf nodes of their respective latent nodes, for every undirected path p linking X i and X j , p must be in form of X i − ˜ X
i − · · · − ˜ X
j − X j .
Hence, W must be in latent nodes, and only case 1) is possible, which means that there exists a collider on every p linking X i and X j , and thus X i and X j is also d-separated by ∅ (conclusion 1).
Specifically on case 1) , W ̸∈ S is obvious (since W ∈ ˜
X and S ∈ X ).
And, by des( W ) ∩ S = ∅ , we have W ̸∈ ˜ S and des( W )
∩ ˜ S
= ∅ (easy to show since S ⊆ des( ˜ S ) ), and thus among latent nodes, there is ˜ X i ⊥⊥ d ˜ X
j | ˜ S (conclusion 2).
Combining conclusion 1 and 2, let S := ∅ , we further have marginally ˜
X i ⊥⊥ d ˜ X j holds.
Remark 1.
Roughly speaking, the d-separation patterns among ˜ X usually do not hold among X (except for rare marginal ones), since the observed variables are all leaf nodes, which are not causes of any other (though the latent variables they intend to measure might be).
By a similar proof procedure, we shall have a full version of Proposition 1: for observed variables
X i , X j and subset S ,
If X i ⊥⊥ d X j | S , then among latent variables, marginally ˜ X
i ⊥⊥ d ˜ X j , and ˜ X i ⊥⊥ d ˜ X
j | ˜ S holds.
If X i ̸⊥⊥ d X j | S , which means that there exists a path p from X i to X j unblocked by Z .
And, a) If on p there is a non-collider W ∈ ˜ S , then
˜ X
i ⊥⊥ d ˜ X
j | ˜ S ; b) Otherwise (on p there is no non-collider W ∈ ˜ S ),
˜ X i ̸⊥⊥ d ˜ X j | ˜ S .
A.2 Proof of Theorem
Theorem 1 (Characterization of Ω Z ; Y ) .
For two variables subsets Z and Y , Ω Z ; Y satisfies:
[Equation]
where null( · ) denotes nullspace.
B Y , nzcol( B Z , : ) denotes the submatrix of mixing matrix B , with rows indexed by Y and columns indexed by nzcol( B Z , : ) .
nzcol( B Z , : ) denotes the column indices where the submatrix B Z , : has non-zero entries.
nzcol( B Z , : ) actually corresponds to the exogenous noises that constitute Z .
Particularly, if assuming “if i ⇝ j then B j,i ̸
= 0 ”, then, nzcol( B Z , : ) =
Anc( Z ) .
Proof of Theorem 1.
We write variables in terms of linear combination of exogenous noises,
X = BE .
For variables set Z = B Z , : E = B Z , nzcol( B Z , : ) E nzcol( B Z , : ) , where nzcol( B Z , : ) denotes the column indices where the submatrix B
Z , : has non-zero entries, i.e., Z contains and only contains noise terms E nzcol( B Z , : ) .
For a vector ω ∈ R | Y | , ω ⊺ Y = ω ⊺ B Y , : E = ω ⊺ B Y , nzcol( B Z , : ) E nzcol( B Z , : ) + ω ⊺ B Y , ∼ nzcol( B Z , : )
E ∼ nzcol( B Z , : ) (“ ∼ ” denotes complement set).
By the Darmois–Skitovich theorem [21], ω ⊺ Y ⊥⊥ Z if and only if ω ⊺ Y shares no common non-Gaussian noise terms with Z , i.e., ω ⊺ B Y , nzcol( B Z , : )
= 0 .
Moreover, if assuming “if i ⇝ j then B j,i ̸
= 0 ” (a weaker faithfulness assumption, see Appendix F.1), then, nzcol( B Z , : ) =
Anc( Z ) , i.e., variables set
Z contains and only contains exogenous noises w.r.t.
its ancestors set.
A.3 Proof of Theorem
Theorem 2 (Graphical criteria of Ω Z ; Y ) .
Let Z , Y be two subsets of variables (vertices), we have:
[Equation]
S is a vertex cut from Anc( Z ) to Y } .
[Equation]
where dim(Ω Z ; Y ) denotes the dimension of the subspace Ω Z ; Y , i.e., the degree of freedom of ω .
To prove Theorem 2 we mainly use the Lindström-Gessel-Viennot theorem [25, 13] in algebraic combinatorics, which gives a combinatorial interpretation of the determinants of certain matrices:
Theorem 6 (Lindström-Gessel-Viennot theorem [25, 13]) .
Let G be a directed acyclic graph with vertex set [ n ] .
Each directed edge i → j is assigned with a weight e ( i, j ) .
For each directed path
P from vertices i to j , let wt( P ) :
= Π m → l ∈ P e ( m, l ) , the product of the weights of the edges of the path.
For any two vertices i, j , denote P ( i, j ) the set of all directed paths from i to j . Write an n × n matrix M , with entries defined as M i,j = (cid:80) P ∈P
( i,j ) wt( P ) , the sum of path weights over all paths from i to j .
For two subsets S, T ⊆ [ n ] with | S | = | T | =
k (letters “ S ” means source and “ T ” means sink), we have:
[Equation]
where the sum is taken over all k -tuples P
=
( P 1 , . . .
, P k ) of non-intersecting paths from S to T , and σ ( P ) is the sign of the corresponding permutation of elements in P .
“non-intersecting” means that for any two paths P i , P j ∈ P with i ̸
= j , P i and P j have no two vertices in common (not even endpoints).
In particular, det( M S,T ) =
if and only if there exists no such k -tuple non-intersecting paths, i.e., for every system of k paths from S to T , there exists two paths that share a vertex.
Based on Theorem 6, we can readily give proof to Theorem 2.
Note that in our setting where
X =
AX + E = BE with B =
( I − A ) − 1 , we know that A j,i is the e ( i, j ) above, and B is exactly
M ⊺ , with B j,i =
(cid:80) P ∈P ( i,j ) (cid:81) k
→ l ∈ P
A l,k , the total causal effect from i to j .
Proof of Theorem 2.
From Theorem 1 and Assumption 1 and the rank-nullity theorem, | Y | − dim(Ω Z ; Y ) is equal to rank( B Y , Anc( Z ) ) .
By the max-flow min-cut theorem (vertex version, known as Menger’s theorem) [9, 3, 26], the maximum amount of non-intersecting paths from source to sink is equal to the size of the minimum vertex cut from source to sink.
Hence, if the minimum vertex cut from Anc( Z ) to Y is of size k , then there exists a k -tuples of non-intersecting paths from some subset of Anc( Z ) to some subset of Y , and this is the largest possible non-intersecting paths system from Anc( Z ) to Y .
By Theorem 6 and Assumption 1 (no parameter coupling to make coincidental low rank), this means that all ( k + 1) × ( k + 1) minors of B Y , Anc( Z ) is zero and at least one k × k minor of B Y , Anc( Z ) is non-zero.
Hence rank( B Y , Anc( Z ) )
=
k .
Interestingly, we find that our defined vertex cut has connection with trek-separation [47], i.e. “ S is a vertex cut from Anc( Z ) to Y ” is equivalent to “ ( ∅ , S ) t-separates ( Z , Y ) ”
(see Appendix C).
Trek-separation theorem states that:
Theorem 7 (Trek-separation for directed graphical models, Theorem 2.8 in [47]) .
For two vertices sets W , Y , the variance-covariance matrix cov( W , Y ) has rank less than or or equal to k for all covariance matrices consistent with the graph G if and only if there exists subsets S W , S Y
⊆ V ( G ) with | S W | + | S Y | ≤ k such that ( S W , S Y ) t-separates ( W , Y ) .
Consequently, rank
[Equation] [Equation]
and equality holds for generic covariance matrices (i.e., no coincidental low rank in variancecovariance matrix) consistent with G .
We now show that Theorem 2 can also be proved by trek-separation theorem:
Proof of Theorem 2 (another version).
From Theorem 1 and Assumption 1 and the rank-nullity theorem, | Y | − dim(Ω Z ; Y ) is equal to rank( B Y , Anc( Z ) ) .
Then, what is rank( B Y , Anc( Z ) ) ?
Let us consider two variables sets Anc( Z ) and Y and their respective variance-covariance matrix.
We write
Anc( Z ) as mixed noise components Anc( Z ) =
B Anc( Z ) , Anc( Z ) E Anc( Z ) , where B Anc( Z ) , Anc( Z ) is a square matrix which can be simultaneously permuted to lower triangular with diagonals one, and thus is full rank.
Then write Y = B Y , : E = B Y ,
Anc( Z ) E Anc( Z ) + B Y , ∼ Anc( Z ) E ∼ Anc
( Z ) , where the second part are Y ’s noise components that is not shared in Anc( Z ) , so is independent to Anc( Z ) and can be dropped in calculating covariance.
From above, cov(Anc( Z ) , Y ) =
B Anc( Z ) , Anc( Z ) Φ( E Anc( Z ) )
B ⊺ Y , Anc( Z ) , where Φ( E Anc( Z ) ) is a diagonal matrix with diagonal entries being variance of exogenous noise terms in E Anc( Z ) .
Since both B Anc( Z ) , Anc( Z ) and Φ( E Anc( Z ) ) are full rank square matrices, rank( B Y , Anc( Z ) ) is equal to rank(cov [Table]
[Table] According to Theorem 7 and Assumption 1, rank(cov( W , Y )) is equal to min {| S W | + | S Y
||
( S W , S Y ) t-separates ( W , Y ) } .
Further we obtain a lemma: if W = Anc( W ) , i.e., ancestors are self-contained in W , then rank(cov( W , Y ))
=
min {| S | | ( ∅ , S ) t-separates ( W , Y ) } .
This can be proved by that, for self-contained W , for any ( S W , S Y ) that t-separates ( W , Y ) , ( ∅ , S W ∪ S Y ) also t-separates ( W , Y ) .
Another lemma is that, ( ∅ , S ) t-separates (Anc( Z ) , Y ) if and only if ( ∅ , S ) t-separates ( Z , Y ) (see Appendix C).
With lemmas above, we immediately have that rank( B Y , Anc( Z ) ) is equal to the size of the minimum vertices set S s.t.
( ∅ , S ) t-separates ( Z , Y ) , i.e., the size of the minimum vertex cut from Anc( Z )
to
Y .
A.4 Proof of Theorem
Theorem 3 (Equivalence of TIN over latent and observed variables) .
For two disjoint observed variables subsets Z , Y , and their respective underlying latent variables subsets ˜ Z , ˜ Y ,
[Equation]
Proof of Theorem 3.
Theorem 3 can either be proved by the graphical criteria (where observed variables are all leaf nodes), or by mathematically showing how rank of submatrices of B preserves among latent and observed variables.
Consider the latent ˜ X in LiNGAM:
[Equation]
when X i =
˜
X
i + E i , write latent and observed variables together
, we have:
[Equation]
(cid:34) ˜
X
X (cid:35) + (cid:34) ˜
E
E (cid:35) , where A ′ =
(cid:34) ˜
A
I (cid:35) , (cid:34) ˜
X
X (cid:35) =
B ′ · (cid:34) ˜
E
E (cid:35) , where B ′ = (cid:34) ˜ B ˜ B
I (cid:35) , Σ ′ =
cov (cid:32)(cid:34) ˜
X
X (cid:35)(cid:33) =
(cid:34) ˜
Σ ˜ Σ
˜ Σ ˜ Σ
+ Φ E (cid:35) , where Φ E = diag(var( E )) .
(A.4)
More generally, when observations are measured with X i =
c i ˜
X
i + E i , let
C = diag
[ c 1 , · · · , c n ] ⊺ ) :
A ′ = (cid:34) ˜
A
C (cid:35) , B ′ = (cid:34) ˜
B
C ˜ B
I (cid:35) , Σ ′ = (cid:34) ˜ Σ ˜ ΣC ⊺
C ˜ Σ
C ˜ ΣC ⊺ + Φ E (cid:35) (A.5)
For two disjoint sets Z , Y , consider B Y , Anc( Z ) , a submatrix in [ C ˜ B | I ] .
Anc( Z ) = Anc(˜ Z ) ∪ Z , where for the second Z parts, its indexed columns in B Y , : must be all zero (since Y and Z are disjoint indices in I ), and thus can be dropped.
For the first Anc(˜ Z ) part, B Y , Anc( ˜ Z ) is B ˜ Y , Anc( ˜ Z ) with rows scaled by C , and thus the rank holds.
Consequently, for two disjoint vertices sets Z , Y ,
TIN( Z , Y ) = TIN(˜
Z , ˜ Y ) .
A.5 Proofs of Other Results
For other lemmas and theorems in this paper: GIN , IN as special cases of TIN and Lemma 1 follows directly from the graphical criteria in Theorem 2.
Theorem 4 can be proved in a similar way as the proof to Theorem 2, where the subsets and subdeterminants are considered.
For ranks stopped increasing in Theorem 5, please refer to Appendix E.1.
B Using GIN Condition-Based Algorithm Under 2-Measurements Model
As is illustrated in §2, when each latent variable ˜
X i has two pure measurements
X i 1 , X i 2 (by “pure” it means that each of X i 1 , X i 2 has only one latent parent ˜ X i and no observed parents), graph structure ˜
G over latent variables is fully identifiable by GIN (a simpler case).
This is already a breakthrough comparing to existed methods [42, 46, 23], which only identify a partial graph.
Here is an illustrating example: consider a simple 2-variables example, ˜ X
→ ˜ Y , with their respective measurements X 1 , X 2 and Y 1 , Y 2 .
One may check the entailed vanishing correlations: ρ
[Equation] [Equation]
and
[Equation]
where ρ denotes correlation coefficient.
These (in)equations exhibit no asymmetry between ˜ X and ˜ Y .
Indeed, for the inverse direction ˜ X ← ˜ Y , all the Tetrad constraints among X 1 , X 2 , Y 1 , Y 2 hold the same.
Therefore, the direction between ˜ X and ˜ Y is unidentifiable.
However, the GIN condition can identify an asymmetry: GIN ( X 1 , Y 1 , 2 ) holds, while GIN ( Y 1 , X 1 , 2 ) is violated, and thus the direction ˜
X
→ ˜ Y is identified.
One can see this from the definition of GIN (Definition 2).
Below we give the general algorithm of using GIN to fully identify ˜ G .
Note that here by “ GIN condition”, it is actually a bit different from the original paper [50]: it takes into account one more thing than the original definition: the degeneration of ω (see Appendix D for details).
Here is the procedure:
Given 2 n measured variables (where n is the number of vertices in ˜ G ), let two variables be Y and the rest 2 n − 2 variables be Z , GIN( Z , Y ) if and only if these two variables are the two measurements of a same latent variable.
Following this, the 2 n measured variables can first be pairwise clustered, and labeled as { X i 1 , X i 2 } i
=1 , ··· ,n .
One may also obtain this pairwise labeling by prior knowledge (e.g., in survey questions design, one already knows which two questions indicate a same latent factor).
Then, find the graph structure ˜ G over n latent variables:
Output: of ˜
Algorithm 1 follows a similar procedure as Direct-LiNGAM [40]: Lines 3-5 sorts the vertices by causal ordering, where there is no edge from later ones to earlier ones.
Then according to degeneration graphical criteria in Appendix D, Line 10 identifies the direct parents set of each vertex from its causally earlier vertices set.
Further consider the coefficients.
Denote the linear coefficients of latent variable
˜ X
i to two measurements
X i 1 , 2 as α i 1 , 2 respectively.
The ratio α i 1 /α
i 2 is accessible when testing GIN for pairwise clusters.
Then, in Line 10, to identify parents set for each vertex j , we find from A , the vertices earlier than j in ordered list K .
We write a scaling vector s :
= { α j 2 /α
i 2 | i ∈
A } , and denote the coefficients vector from A to j as c (zero if no direct edge).
Note that here ω must only have a free degree of one (according to Theorem 2 and Appendix D.1, critical vertex cut is A ).
So set the value of ω on X j 2 as − 1 , then the value of ω on other X i 2 s is exactly the point-wise multiplication of s and c .
If we further assume that linear coefficients from latent variables to measurements are all same (e.g., one, X i 1 , 2 = ˜
X
i + E i 1 , 2 ), or equivalently, the measurement errors are uni-variance, then the coefficients among ˜ G is also fully identifiable.
C Elaboration on Vertex Cut and Graph Definitions
We first give more detailed definitions to the concepts in §3.
Definition 9 (Directed paths) .
A directed path P =
( i 0 , i 1 , · · · , i k ) in G is a sequence of vertices of
G where there is a directed edge from i j to i j +1 for any 0 ≤ j ≤ k − 1 .
We use notation i ⇝ j to show that there exists a directed path from vertex i to j .
Remark 2.
Note that a single vertex is also a directed path, i.e., i ⇝ i holds true.
Definition 10 (Directed paths without passing through S ) .
Let S be a subset of vertices.
We use notation i ⇝ (cid:26)(cid:26)
[ S ] j to show that there exists a directed path from vertex i to j without passing through
S , i.e., there exists a directed path P
=
( i, m 0 , · · · , m k , j ) in G s.t.
i, j ̸∈ S and m l ̸∈ S for any 0 ≤ l ≤ k .
Remark 3.
Note that when S is empty, i ⇝ j is equivalent to i ⇝ (cid:26)(cid:26)
[ S ] j .
Definition 11 (Ancestors) .
Let W be a subset of vertices.
Ancestors Anc( W ) :
=
{ j |∃ i ∈ W , j ⇝ i } .
Remark 4.
Note that W ⊆
Anc( W ) .
Under faithfulness assumption (no parameter coupling),
Anc( W ) means all noise components that W carries, i.e., writing the corresponding variables set { X i | i ∈ W } as linear combination of noises, it contains and only contains exogenous noises from { E i |
i ∈
Anc( W ) } .
Definition 12 (Ancestors outside S ) .
Let W , S be two subsets of vertices.
We denote ancestors of W that has directed paths into W without passing through S as Anc out( S ) ( W ) :
= { j |∃ i ∈ W , j ⇝ (cid:26)(cid:26)
[ S ] i } .
Remark 5.
According to definitions above,
Anc out( ∅ ) ( W ) =
Anc ( W ) .
Anc out( W ) ( W ) = ∅ .
S ∩ Anc out( S ) ( W ) = ∅ .
W \ S ⊆
Anc out( S ) ( W ) .
For overlapped S , W , Anc out( S ) ( W ) =
Anc out( S ) ( W \ S ) .
Roughly speaking, Anc out( S ) ( W ) means all noise components that can contribute to W without passing S .
With slight notation abuse, we can write variables W as W =
A S + E W , where A S is a linear transformation to S , and E W is a linear transformation to exogenous noises set that contains and only contains { E i | i ∈ Anc out( S ) ( W ) } .
Definition 13 (Existence of causal effect from W 1 to W 2 ) .
Let W 1 , W 2 be two subsets of vertices.
We say there exists causal effect from W 1 to W 2 if and only if there exists a directed path i ⇝ j with i ∈ W 1 and j ∈ W 2 .
Remark 6.
According to definitions above,
Note that if W 1 and W 2 are not disjoint, then there must exist causal effect from W 1 to W 2 .
An equivalent definition is that, Anc( W 2 ) ∩ W 1 ̸ = ∅ .
Definition 14 (Existence of causal effect from W 1 to W 2 without passing through S ) .
Let
W 1 , W 2 , S be three subsets of vertices.
We say there exists causal effect from W 1 to W 2 without passing through S
if
and only if there exists a directed no-passing path i ⇝ (cid:26)(cid:26)
[ S ] j with i ∈ W 1 and j ∈ W 2 .
Remark 7.
According to definitions above,
An equivalent definition is that, Anc out( S ) ( W 2 ) ∩ W 1 ̸ = ∅ .
There exists no causal effect from S to W 1 without passing S , i.e., S ∩ Anc out( S ) ( W ) = ∅ .
This definition shows whether S chokes all directed paths from W 1 to W 2 . 4.
By Definition 4, the following statements are equivalent: 1) there exists no causal effect from W 1 to W 2 without passing through S ; 2) S is a vertex cut from W 1 to W 2 ; 3) ∀ i ∈ W 1 , j ∈ W 2 , i ⇝ (cid:26)(cid:26)
[ S ] j does not hold; 4) Anc out( S ) ( W 2 ) ∩ W 1 = ∅ ; 5) S ’s removal from G ensures there is no directed paths from W 1 \ S to W 2 \ S .
Now we have complete our graphical definitions.
Let us also review trek-separation [47].
Definition 15 (Trek) .
A trek in G from i to j is an ordered pair of directed paths ( P 1 , P 2 ) where Phas sink i , P 2 has sink j , and both P 1 and P 2 have the same source k .
Note that one or both of Pand P 2 may consist of a single vertex, e.g., (( i ) , ( i )) is a trek from vertex i to i .
Definition 16 (t-separation) .
Let W , Y , S W , S Y be four subsets of V ( G ) which need not be disjoint.
We say that the pair ( S W , S Y ) trek separates (or t-separates ) W from Y if for every trek ( P 1 , P 2 ) from a vertex in W to a vertex in Y , either P 1 contains a vertex in S W or P 2 contains a vertex in
S Y .
The above two definitions are directly from [47].
By the “ancestors” related definitions introduced above and in §3, we can immediately get an equivalent restatement of t-separation as:
Theorem 8 (Restatement of t-separation) .
Let W , Y , S W , S Y be four subsets of V ( G ) which need not be disjoint.
The pair ( S W , S Y ) t-separates W from Y , if and only if there exists no causal effect from Anc out( S W ) ( W ) to Y without passing passing S Y (see Definition 14).
Note that the above graph condition also has an equivalent restatement:
· · · if and only if there exists no causal effect from Anc out( S Y ) ( Y ) to W without passing passing S W .
Both mean that Anc out( S W ) ( W ) ∩
Anc out( S Y ) ( Y ) = ∅ , i.e., if some noise components can flow into W without passing S W , then it cannot also flow into Y without passing S Y , or vice versa.
Remark 8.
Further, by definitions above and the rank constraints (in trek-separation theorem Theorem 7), we have the followings:
1. rank(cov
( W , Y )) ≥ | W ∩ Y | , since we must have W ∩ Y ⊆ S W ∪ S Y if ( S W , S Y ) t-separates W from Y , otherwise some unblocked vertex is in Anc out( S W ) ( W ) ∩
Anc out( S Y ) ( Y ) .
rank(cov( W , Y )) ≤ min( | W | , | Y | ) , since ( W , ∅ ) and ( ∅ , Y ) always t-separates W from Y , i.e., Anc out( W ) ( W ) = ∅ or Anc out( Y ) ( Y ) = ∅ .
3. rank(cov( W , Y )) ≤ | Anc( W ) ∩ Anc( Y ) | , since (Anc( W ) ∩ Anc( Y ) , ∅ ) and ( ∅ , Anc( W ) ∩ Anc( Y )) always t-separates W from Y .
The pair ( S W , S Y ) t-separates W from Y , if and only if the pair ( S Y , S W ) t-separates Y from W .
From above we have seen the interpretation of t-separation from the “ancestors” language set.
Then combining Definition 4 and Theorem 8, we know that the following statements are equivalent: 1) S is a vertex cut from Anc( Z ) to Y ; 2) ( ∅ , S ) t-separates ( Z , Y ) ; 3)
There exists no causal effect from
Anc( Z ) to Y without passing through S ; 4) There exists no causal effect from Anc out( S ) ( Y ) to Z .
D More Properties of TIN Condition D.1 Critical Vertex Cut
From the above §3 and Appendix C graphical criteria, we know that TIN( Z , Y ) is equal to the size of the minimum vertex cut from Anc( Z ) to Y .
Remark 9.
Following Definition 4, we first elaborate more on vertex cut:
For any Z , Y , any superset of Y (including Y ) is a vertex cut from Anc( Z ) to Y . For any Z , Y , any superset of Anc ( Z ) (including Anc ( Z ) ) is a vertex cut from Anc( Z ) to Y .
For any vertex cut from Anc( Z ) to Y , Anc( Z ) ∩ Y ⊆ S (to choke single vertex paths).
Following point 3, for overlapped Z , Y in testing TIN condition, any vertex cut S must contains (at least) Z ∩ Y (the observed/testable intersection) as its subset.
Following point 4, if Y ⊆ Z , then there exists no non-zero ω s.t. ω ⊺
Y ⊥⊥ Z . 6.
Note that though expressed as “ S is a vertex
cut from Anc( Z ) to Y ”, it never implicitly implies a causal ordering of Z → S → Y .
E.g., in graph D ← A → C ← B , consider
TIN( Z = { A } , Y = { B, C } )
= 1 where the minimum vertex cut is S
= { C } , not causally earlier than Y ; TIN( Z = { B, C } , Y =
{ A, D } )
= 1 with the minimum vertex cut
S =
{ A } , but Z is neither causally earlier than Y nor than S .
Following point 6, roughly speaking, TIN tells size of the minimum vertex cut, but not exactly the causal ordering.
For the existence of non-zero ω s.t. ω ⊺ Y ⊥⊥ Z , there can be some vertices in Y that are in or causally earlier than Z , i.e. Anc( Z ) ∩ Y ̸ = ∅ - as long as there are not “too many” (less than the cardinality of possible S ).
Note that the minimum vertex cut may not be unique.
E.g., 1) Consider example in point 6,
both S =
{ C } and S =
{ A } are minimum vertex cuts in TIN( Z = { A } , Y =
{ B, C } )
= 1 . 2) Consider a chain structure with TIN( Z = { X 1 } , Y = { X 2 , · · · , X n } )
= 1 , both S =
{ X 1 } and S =
{ X 2 } are minimum vertex cuts.
Following point 6 of Remark 9, since the minimum vertex cut from Anc( Z ) to Y may not be unique in a TIN( Z , Y ) , to better use the graphical criteria, now we define the critical vertex cut :
Definition 17 (Critical vertex cut) .
Denote S ( Z , Y ) the collection of all sets S ⊆ V ( G ) s.t.
S is a minimum vertex cut from Anc( Z ) to Y (“minimum” means that | S | =
TIN( Z , Y ) ).
For a vertex cut S ∈ S ( Z , Y ) , we say S is critical if and only if there exists no causal effect from all (other) minimum vertex cuts to Y without passing through S , i.e. Anc out( S ) ( Y ) ∩
Anc( (cid:83) S
( Z , Y ))
= ∅ .
Remark 10.
Roughly speaking, when there are multiple minimum vertex cuts, i.e., these multiple sets can all cut from Anc( Z ) to Y , then a critical one means a “last” one (furthest from Z , deepest to
Y ): it not only cuts Anc( Z ) to Y , but also cuts all other vertex cuts to Y .
E.g., consider examples in point 8 of Remark 9, 1) { C } is critical while { A } is not, because { C } can cut { A } to { B, C } , but { A } cannot cut { C } to { B, C } . 2) {
X 2 } is critical while { X 1 } is not.
Theorem 9 (Uniqueness of critical gin-separation set) .
For two vertices sets Z and Y and their respective TIN( Z , Y ) , there exists one and only one corresponding critical vertex cut, denoted as
S ∗ Z , Y .
D.2 Noise Components of Linear Transformation ω ⊺ Y
From above Appendix D.1 we defined the critical vertex cut S ∗ Z , Y behind a TIN( Z , Y ) , with special property on it.
Now we analyze the linear transformation ω ⊺ Y :
Theorem 10 (Noise components of linear transformation ω ⊺ Y ) .
For two vertices sets Z and Y and their respective TIN( Z , Y ) , for generic choice of ω (i.e., no coincidental noise cancelling by ω ), the corresponding linear transformation ω ⊺ Y contains and only contains exogenous noises introduced by vertices that has directed paths to Y without passing through the critical vertex cut
S ∗ Z , Y , i.e.,
E ( ω ⊺ Y ) =
{ E i | i ∈ Anc out( S ∗ Z , Y ) ( Y ) } , where E ( · ) denotes the exogenous noises components set that a variable · is constituted of, and E i is the exogenous noise from vertex i .
Remark 11.
A vertex cut S from Anc( Z ) to Y yields that all noise components that Z carries (i.e.,
Anc( Z ) ) cannot flow into (causal affects / contribute to) Y without passing through S , then Y can be written as Y =
L S + E ′ Y , where L denotes a linear transformation, and E ′ Y denotes noise components that can contribute to Y without passing through S (i.e., Anc out( S ) ( Y ) ) - so E ′ Y ⊥⊥ Z , but not necessarily E ′
Y ⊥⊥ S .
Also, we define Ω Z ; Y as { ω | ω ⊺ Y ⊥⊥ Z } , while actually for such ω , ω ⊺ Y is independent to more variables:
Theorem 11 (Full version of ω ⊺ Y independence) .
For two vertices sets Z and Y and their respective critical vertex cut S ∗ Z , Y , for any variable X i ∈
X (i.e., respective vertex i ∈
V ( G ) ), ω ⊺ Y ⊥⊥ X i
if and only if there exists no causal effect from Anc out( S ∗ Z , Y ) ( Y ) to { i } , i.e., Anc out( S ∗ Z , Y ) ( Y ) ∩
Anc( { i } )
= ∅ .
Remark 12.
With Definition 13 and Theorem 11, we can immediately get the following:
1. ω ⊺ Y ⊥⊥ Z - it can be derived from Theorem 11, Definition 11 and Definition 17.
ω ⊺ Y ⊥⊥ Anc( Z ) - it can be derived from Theorem 11 and Definition 17. 3.
Theorem 11 is straightforward by seeing ω ⊺ Y as a linear transformation of its noise sources { E i | i ∈
Anc out( S ∗ Z , Y ) ( Y ) } .
Then any variable is independent to ω ⊺ Y if and only if it does not carry noise from these sources (i
e., vertex has no ancestors in Anc out( S ∗ Z , Y ) ( Y ) ), by the Darmois–Skitovich theorem.
With Theorem 11, after testing on TIN( Z , Y ) , one can do more independence test over other variables (as long as they are observed/testable), and may get more information about the whole graph structure and the location of critical vertex cut.
Further, we notice that in the independent linear transformation subspace Ω Z ; Y , some indices of ω may be degenerated (i.e., fixed to zero).
Consider following examples for an intuition: 1) On a chain structure Figure 2a with TIN( { X 2 } , { X 1 , X 3 , X 4 , · · · , X n } )
= 2 , ω index on X 1 must be zero (not include X 1 in linear transformation) to make ω ⊺ Y ⊥⊥ Z , while in a fully connected
DAG Figure 2c with also TIN( { X 2 } , { X 1 , X 3 , X 4 , · · · , X n } )
= 2 , ω is not degenerated on any indices.
On a chain structure Figure 2a or a chain structure with triangular head Figure 2b,
TIN( { X 1 , X 3 } , { X 2 , X 4 , X 5 , · · · , X n } ) = 2 holds, while ω index on X 2 must be zero.
3) in Figure 2d, TIN( { X 1 } , { X 2 , X 5 } )
= 1 , while actually ω is degenerated on X 5 index, which means that the linear transformation actually does not include X 5 and is just trivially X
2 independent of X(here S ∗
Z , Y is just X 5 ).
Now, we would like to first give mathematical characterization for such ω indices degeneration:
Theorem 12.
Since Ω Z ; Y = null( B ⊺ Y , Anc( Z ) ) , Ω Z ; Y degenerates on an index y ∈ Y
if and only if: remove the corresponding y -th column in B ⊺ Y , Anc( Z ) to get submatrix B ⊺ Y \{ y } , Anc( Z ) , the rank of submatrix is one less than the rank of full matrix B ⊺ Y , Anc( Z ) .
Then we give the equivalent graphical criteria for such ω indices degeneration:
We already know that the vertex cut S ∗ Z , Y cuts Anc( Z ) to Y .
Moreover, each part of S ∗ Z , Y has its “own indispensable work” in cutting, so we first define:
Definition 18 (Local cut scope) .
For each vertex s ∈ S ∗ Z , Y , define its local choke scope as LC( s ) :
= { y ∈ Y | there exists causal effect from Anc( Z ) to { y } without passing through S ∗ Z , Y \{ s }} .
Furthermore, for each subset S ⊆ S ∗ Z , Y , define LC( S ) :
= { y ∈ Y | there exists causal effect from
Anc( Z ) to
{ y } without passing through S ∗ Z , Y \ S } .
Remark 13.
With Definition 18 we have the following:
S ∗ Z , Y = ∅
if and only if Z , Y are marginally independent, i.e., B Y , Anc( Z ) are all zero (no shared noise components).
LC( s ) means the part of Y that would not be cut/choked, had there been no s .
In other word, the part of Y that s has its own indispensable work.
LC( S ) = ∪ s ∈ S LC ( s ) .
LC( S 1 ∪ S 2 ) =
LC( S 1 ) ∪ LC( S 2 ) .
S ⊆ Anc(LC( S )) .
For any subset S , | LC( S ) | ≥ | S | (so | LC( s ) | ≥ 1 for any vertex s ).
For any two different vertices s 1 , s 2 , it does not necessarily yield that LC( s 1 )
∩ LC
( s 2 )
= ∅ - they may work together to cut/choke a part and either is indispensable for this part.
LC( S ∗ Z , Y ) may not be the whole Y , but a proper subset.
The rest Y \ LC( S ∗ Z , Y ) is exactly part of Y that is marginally independent to Z (i
e., no directed paths from Anc( Z ) to that part).
Theorem 13 (Graphical criteria for degeneration) .
Ω Z ; Y degenerates on on the indexes subset
Y ⊂ Y
if and only if: there exists a subset S ⊂ S ∗
Z , Y such that its local choke scope LC( S )
=
Y , and | Y | = | S | = | LC( S ) | .
Remark 14.
We already have Theorem 12 for math condition.
And for graphical criteria Theorem 13:
A rough interpretation: in general we would expect a smaller S to choke a larger Y .
However, if for an S , its local choke scope Y is of the same size as S , then removing S will only affect the same size Y (a feeling that this S is “wasted”).
Then this part Y will be degenerated.
From Y side, it means that this Y requires a same size of separation set S to choke (a feeling that Y is “too expensive”).
TIN( Z , Y \ Y ) =
TIN( Z , Y ) − | Y | , with critical vertex cut being S ∗ Z , Y \ S , and with no degeneration.
Note that degeneration does not yield independence, i.e., if TIN( Z , Y ) with Ω Z ; Y degenerated on Y k , it does not necessarily yield that ω ⊺ Y ⊥⊥ Y k .
Because ω is applied to variables, not noise components.
For example, the v-structure { A, B } → C , TIN( A, BC ) = 1 with C degenerated.
But ω ⊺ BC , which is simply B , is not independent to variable C . 5.
The inverse direction of 4. is also not sufficient: if for TIN( Z , Y ) and some Y k ∈ Y , there is also ω ⊺
Y ⊥⊥ Y k ( ω ⊺ Y independent to not only to Z but also some part in Y ), it still does not necessarily yield that Y k is degenerated.
E.g., in chain structure Figure 2a, TIN( { ˜ X 1 } , { ˜ X 2 , · · · , ˜ X n } ) = 1 with ω ⊺ Y also independent to ˜ X 2 , but ˜ X 2 index is not degenerated in Ω Z ; Y . 6.
Note that while as a special case of point 5, in measurement error case, independence in Y yields degeneration.
Because each variable Y k ∈ Y is associated with measurement
noise E k which is only in Y k , not in any other variables (so cannot be cancelled).
Then to make ω ⊺ Y ⊥⊥ Y k , at least E k must be removed,
i
[Table]
e., Y k degenerated.
E.g., in chain structure Figure 2a, TIN( { X 1 } , { X 2 , · · · , X n } ) = 1 with no degeneration.
So ω ⊺ Y is only independent of X 1 , not any other in observed variables Y =
{ X 2 , · · · , X n } - specifically, ω ⊺ Y ⊥⊥ { ˜ X 1 , ˜ X 2 , X 1 } .
[Table]
, 2 , 3 X 1 , 2 X 1 , 2 X 1 , 2 , 3 X 1 , 3 X 1 , 2 ,
X
X 1 , 2 , 3 , 4 X 1 , 3 , 4 X 1 , 2 , 3 , 4 X 1 , 2 , 3 , 4 S ∗ Z , Y
X
X
X 1 , 2 X
X
X
X 4 , 5 X 4 X 1 , 2 , 4 X 1 , 3 X 1 , 2 ,
X
X 3 , X 3 ,
X 3 , 4 ,
X 3 , 4 A o( S ∗ ) ( Y )
X 4 , 5 X 4 , 5 X 3 , 4 , 5 X 3 ,
X
X∅ X 3 ,
X
X 2 , 4 , 5 X 4 , 5 X 1 , 2 , 4 ,
X
X∅
X 5 E ( ω ⊺ Y )
E 4 , 5 E 4 , 5 E 3 , 4 , 5 E 3 ,
E
E∅ E 3 ,
E
E 2 , 4 , 5 E 4 , 5 E 1 , 2 , 4 ,
E
E∅
E 5 ω ⊺ Y ⊥⊥ to
X 1 , 2 , 3 X 1 , 2 , 3 X 1 , 2 X 1 , 2 X 1 , 2 , 3 , 4 X 1 , 2 , 3 , 4 const X 1 , 2 X 1 , 2 , 3 , 4 X 1 , 3 X 1 , 2 ,
X
X 1 , 2 , 3 , 4 X 1 , 2 , 3 , 4 const X 1 , 2 , 3 , 4 ω degenerate \ \ \ \ \ \ ω 4 ,\ ω 1 ,\ \ \ ωωω 3 , 4 ,\
Table 2 is a full version of Table 1, where we could use examples to better understand the above properties about TIN condition: e.g., different cases for GIN( Z , Y ) to be violated (see rank of B Y , Anc( Z ) and rank of cov( Z , Y ) ); noise components of ω ⊺ Y is exactly corresponding to Anc out( S ∗ Z , Y ) ( Y ) ; the graphical criteria for some ω indices degeneration, etc.
D.3 Subsets Implications of the TIN Condition
In §5 we give Theorem 4 for estimation of Ω Z ; Y , by tackling down Y to subsets:
Theorem 4 ( TIN over Y subsets) .
For two variables sets Z , Y , TIN( Z , Y ) =
k (assume k > 0 ), iff the following two conditions hold: 1) ∀ Y ′ ⊆
Y with | Y ′ |
= k
+ 1 (if any), there exists non-zero ω
s.t. ω ⊺ Y ′ ⊥⊥ Z ; and 2) ∃ Y ′ ⊆
Y with | Y ′ |
= k , there exists no non-zero ω
s.t. ω ⊺ Y ′ ⊥⊥ Z .
Remark 15.
About how to use this “big to small” property, here are some notes:
Condition 1) can also be “ | Y ′
| ≥ k + 1 ” (a weaker/stronger version).
This can be shown by that if a set S is a vertex cut from Anc( Z ) to Y , then S is also a vertex cut from any subset of Anc( Z ) to any subset of Y .
It does not yield that all these TIN conditions on subsets Y ′ has a same rank k , and even with a same rank, not necessarily a same critical vertex cut S ∗
Z , Y .
E.g., consider a 3-vstructure { A, B, C } → D , TIN( A, BCD )
=
1 and S ∗ Z , Y = D , while TIN( A, BC ) =(
Y ′ = BC ) and S ∗ Z , Y = ∅ .
TIN( A, ABD ) =
1 with S ∗ Z , Y =
A , while TIN( A, BD ) =with S ∗ Z , Y = D (though A is still a minimum vertex cut, it is not critical).
Any transformation vector ω ∈ Ω Z ; Y ′ is also in Ω Z ; Y , with the other Y \ Y ′ indices set to zero.
It does not yield that for Y ′ with | Y ′
| ≤ k there exists no non-zero vector ω to make ω ⊺ Y ′ ⊥⊥ Z (so in condition 2) it is “ ∃ Y ′ ⊆ Y ”).
Theorem 4 can help the estimation of Ω Z ; Y (existence is easier to check than dimension of all), and can also help the prunning process when we need to test over Y with size from big to small (to find latent clusters).
Then, with a same Y but different Z , we also have the following properties:
Lemma 2 (Subset of whole independence set) .
For two variables sets Z and Y and their respective TIN( Z , Y ) , denote Ind Z , Y :=
{ i | ω ⊺ Y ⊥⊥ X i } .
From Theorem 11 we have Ind Z , Y =
{ i | Anc out( S ∗ Z , Y ) ( Y )
∩
Anc( { i } ) = ∅ } .
Then, ∀ Z ′ ⊆
Ind Z , Y , TIN( Z ′ , Y )
≤ TIN( Z , Y ) .
Specifically, if Z ⊆ Z ′ , then TIN( Z ′ , Y ) =
TIN( Z , Y ) , and moreover, the independent linear transformation subspace is the same: Ω Z ; Y = Ω Z ′ ;
Y , and the critical vertex cut over all such Z ′ is also the same as S ∗ Z , Y .
More properties about subset implications (e.g., combination and expansion of Z and more independent variables) can be derived from e.g., Theorem 11.
Another interesting question is, except for pruning in practical algorithms or for easier estimation, how to use these subset implication relationships to help identify the graph structure?
E Methods Details for Estimating Ω Z ; Y E.1 For TIN-rank : Stacked Cumulants
To estimate the subspace Ω Z ; Y , we give a method named “ranks stopped increasing” in §5.3 based on cumulants among variables.
Now we give more details on this method.
Definition 19 (Cumulants [31]) .
Define cumulant among k variables
X
i 1 , . . .
X i k as: cum (
X i 1 , . . .
, X i k ) =
(cid:88)
( A 1 ,...,A L ) ( − 1)
L − 1 ( L − 1)!
E   (cid:89)
j ∈
A
X j   E   (cid:89)
j ∈
A
X j   · · · E   (cid:89)
j ∈
A L
X j   , (E.1) where the sum is taken over all partitions ( A 1 , . . .
, A L ) of the set { i 1 , . .
, i k } .
Remark 16.
About cumulant defined in Definition 19:
Suppose variables are zero-meaned, then sum is taken over all partitions where each A i has size at least 2.
For example, in the following: 2.
cum( X i ) = 0 .
cum( X i 1 , X i 2 ) =
E [ X i 1 X i 2 ] =
cov
( X i 1 , X i 2 ) .
cum( X i 1 , X i 2 , X i 3 ) =
E [ X i 1 X i 2 X i 3 ] =
3rd order moment of ( X i 1 , X i 2 , X i 3 ) .
cum( X i 1 , X i 2 , X i 3 , X i 4 ) =
E [ X i 1 X i 2 X i 3 X i 4 ] − E [ X i 1 X i 2 ] E [ X i 3 X i 4 ]
[Equation]
X i 1 X
i
[Equation] [Equation]
[Equation] [Equation]
X
[Equation]
2 X i 3 ] .
As is shown above,
the 4-th order cumulant is not equal to the 4-th order momentum.
In general, cumulant ̸ = momentum when order k ≥ 4 .
We use cumulant, for reason in point 7: 7.
If variables X i 1 , . .
, X i k are mutually independent, then cum ( X i 1 , . . .
, X i k )
= 0 .
Note that it is zero cumulant, not zero momentum.
Definition 20 (Cross cumulant tensor) .
For a random vector X
=
[ X 1 , · · · , X m ] ⊺ , denote its cross cumulant tensor at order k as T ( k ) X , an m × · · · × m (cid:124)
[Equation]
tensor, where each entry
[Equation]
Now suppose these random variables follow a linear acyclic SEM model, with X =
AX + E .
Because of acyclicity, we could also write X = BE , where B =
( I − A ) − 1 .
Then we have the following:
Theorem 14 (Cross cumulant tensor in linear acyclic SEM) .
k -th order cross cumulant tensor equals
[Equation]
where T ( k ) E is the k -th order cross cumulant tensor of E , and ‘ • ’ denotes the tensor dot, i.e.,
[Equation]
Since exogenous noises E are mutually independent, T ( k ) E is a diagonal tensor.
In this case, the above equation needs not to be summed over all Cartesian product [ m ] k , but just over each j ∈ [ m ] .
Remark 17.
About cross cumulant tensor in Linear acyclic SEM in Theorem 14:
For example, in 2nd order case, T (2) X is the cross covariance matrix Σ :=
cov( X , X ) .
We have Σ = B Φ B ⊺ , where Φ is a diagonal matrix with entries Φ i,i =
var( E i ) .
Proof to 1: for every two variables
X i , X j , cov
( X i , X j ) =
(cid:80) k B ik B jk var( E k ) .
Point 2 means that the covariance between X i , X j is contributed by all noise that is contained in both X i and X j .
By ‘common noise’, we mean ‘confounders’, ‘common ancestors’, or the ‘top-node’ of each trek between ( X i , X j ) - and this is the start of the proof to trek-separation.
In general, any order of the cumulant cum( X i 1 , · · · , X i k ) is contributed by the ‘common noise’ that X i 1 , · · · , X i k all share, i.e., (cid:84) l ∈ [ k ] Anc( X i l ) , the common ancestors.
Since we only care the pairwise relationship between any two subsets Z , Y , we can take a 2D matrix slice out from each order of cross cumulant tensors:
Definition 21 (2D slice of cross cumulant tensor) .
For a random vector X with k -th order cross cumulant tensor T ( k ) X , denote its 2D matrix slice of k -th order cross cumulant tensor as C ( k ) , where
[Equation]
Remark 18.
About 2D slice of cross cumulant tensor defined in Definition 21: 1.
For simplicity, here we omit the subscript X in C ( k ) X and just write as C ( k ) .
In particular, when k = 2 , C (2) is the variance covariance matrix Σ X .
C ( k ) is n × n matrix, and is not necessarily symmetric when k > 2 .
Then similar to Theorem 14, we formulate 2D slice of cross cumulant tensor in linear acyclic
SEM:
Theorem 15 (2D slice of cross cumulant tensor in linear acyclic SEM) .
C ( k ) equals
[Equation]
where B k − 1 is the element-wise power (i.e., B k − 1 =
B ◦ · · · ◦ B (cid:124)
[Equation]
(cid:125)
k − 1 times , ‘ ◦ ’ denotes element-wise product (Hadamard product), and Φ ( k ) E is a diagonal matrix with entries Φ ( k ) E i,i =
cum
( E i , · · · E i (cid:124) (cid:123)(cid:122) (cid:125) k times ) .
Moreover, for two vertices sets Z , Y , similar to Theorem 1, we have
[Equation]
where e.g., C ( k ) Z , Y denotes the submatrix of C ( k ) with rows indexed by Z and columns indexed by Y .
Proof to Theorem 15 is straightforward by plugging Definition 21 into tensor dot of Theorem 14.
Since independence yields zero cumulant, we have that for two vertices sets Z , Y and ω ∈ R | Y | , if ω ⊺ Y ⊥⊥ Z , then C ( k ) Z , Y ω = 0 .
In other words,
[Equation]
, for any k ≥
[Equation] [Equation]
[Equation]
This can be shown by two ways: one is that cum( Z , · · · , Z , ω ⊺ Y ) =
cum( Z , · · · , Z , Y ) ω , another is to use Equation (4) we build in Theorem 1: ω ⊺ Y ⊥⊥ Z ⇔ B ⊺ Y , Anc( Z ) ω
= 0 .
We shall also recap the original GIN condition: first solve equation by cov( Z , Y ) , then check whether any solution ω satisfies ω ⊺
Y ⊥⊥ Z (i.e., whether null(cov( Z , Y )) =
Ω Z ; Y ).
However, when GIN is not satisfied (i.e., Ω Z ; Y ⊊ null(cov( Z , Y )) ), it is not necessarily that Ω Z ; Y = R 0 - e.g., the rank may just be limited by the size of Z .
This is exactly the motivation why we need to further generalize
GIN to TIN : can we escape from the ‘unwanted restriction on rank’ (e.g., size of Z ) and find exactly the Ω Z ; Y ?
Fortunately, by above implication from independence to zero cumulant, we could solve equation not only by 2-nd order cov( Z , Y ) , but more (on any order) C ( k ) Z , Y .
Definition 22 (Stacked 2D slices of cumulants) .
For two vertices sets Z , Y and order k ≥ 2 , define: Ψ
[Equation]
Ψ ( k ) Z ; Y is a ( k − 1) | Z | × | Y | matrix that vertically stacks 2D cumulants slices between Z , Y with order from 2 to k .
Since independence yields zero cumulant, similarly we have Ω Z
[Equation]
Y ⊂ null(Ψ ( k ) Z [Equation] Y
[Equation]
, for any k ≥
[Equation] [Equation]
E
[Equation]
For example, a fully connected DAG with 4 variables { X 1 , X 2 , X 3 , X 4 } ⊺ , the edges parameters are:
[Equation]
     a   b d  c e f   ; B =      a   ad + b d  a ( df + e ) + bf + c df + e f 
 (E.11)
Denote cumulants of exogenous noises φ ( k )
i
[Equation]
1) Let Z :=
{ X 1 } , Y :=
{ X 2 , X 3 , X 4 } , we have:
[Equation]
The independence subspace Ω Z ; Y = null( B ⊺ Y ,Anc ( Z ) )
=
null ([ a ad + b a ( df + e ) + bf + c ]) , dimension=2 .
(E.13)
Observe that null(Ψ (2) Z ; Y ) =
Ω Z ; Y , (and also = null(Ψ
(3) Z ; Y ) = · · · ).
Let
Z :=
{ X 2 } , Y :=
{ X 1 , X 3 , X 4 }
, we
have: Ψ
[Equation]
The independence subspace Ω Z ;
[Equation]
=
[Equation]
= null
[Equation]
, dimension=1 .
(E.15)
Clearly null(Ψ (2) Z ; Y ) ̸
= Ω Z ; Y , since the rank of Ψ (2) Z ; Y is only 1.
However, as long as there is no parameter coupling in cumulants, or specifically, aφ
[Equation]
[Equation] [Equation]
= φ
[Equation]
φ (2)2 , then null(Ψ (3) Z ; Y ) =
Ω Z ; Y (with the rank increasing to 2).
We could verify the solution:
[Equation]
= contains only { E C , E D } , and
thus ω ⊺ Y ⊥⊥ Z .
[Equation]
According to original GIN definition, there is only GIN( { X 1 } , { X 2 , X 3 , X 4 } ) , and X 2 , X 3 , Xcannot be distinguished.
However here by using TIN , we could also identify X 2 .
3) Let Z :=
{ X 3 } or { X 4 } , Y :=
X \ Z , there is no non-zero ω s.t., ω ⊺
Y ⊥⊥ Z .
Observe that:
[Equation]
Above example gives us a motivation to use a sequence of stacked 2D cumulants { Ψ ( i ) Z ; Y } i =2 , 3 , ··· .
Remark 19.
About this sequence of stacked 2D cumulants, we have:
Ψ ( i +1)
Z ; Y contains Ψ ( i )
Z
; Y as some-rows-indexed submatrix, so: 2.
Rank does not drop,
i
e., rank(Ψ ( i +1)
Z ; Y ) ≥ rank(Ψ ( i )
Z ; Y ) .
Nullspaces null(Ψ ( i +1)
Z ; Y ) ⊆
null(Ψ ( i ) Z ; Y ) .
Independent subspace Ω Z ; Y ⊆ null(Ψ
( i ) Z ; Y ) , for any k ≥ 2 .
rank(Ψ ( i ) Z ; Y ) ≤ | Y | − dim(Ω Z ; Y ) , for any k ≥ 2 .
Note that in above statements, no assumptions on edge parameters and noise components’ cumulants are made, and they are purely by definition.
Then, does there exist a finite integer K ∈ N + where the shrinking nullspaces stop hereafter at Ω Z ; Y , i.e., Ω Z ; Y = null(Ψ ( i ) Z ; Y ) , for any i ≥ K ?
The answer is yes, under the generic assumptions on edge parameters and noise components’ cumulants:
Assumption 2 (Generic edge parameters and noise components’ cumulants) .
On a LiNGAM instance
L = G ( G, B , E ) defined by graph structure G , edge parameters B and noise components E , assume that for any two variables sets Z , Y and order k ≥ 2 ,
[Equation]
where B ′ , E ′ are traversed over the whole edge parameters and noise components space.
This is to assume that there is no coincidental low rank parameterized by the LiNGAM instance L .
Note that Assumption 2 is stronger than Assumption 1 in §3.
Here Assumption 2 assumes not only generic edge parameters, but also noise parameters.
Under Assumption 2 we have the following graphical criteria over stacked 2D cumulants:
Theorem 16.
For two vertices sets Z , Y and order k ≥ 2 , we define a new DAG associated with G , denoted as ˆ G ( k ) , which has kn vertices { 1 , 2 , · · · , n } ∪ { 1 (2) , 2 (2) , · · · , n (2) } ∪ · · · ∪ { 1 ( k ) , 2 ( k ) , · · · , n ( k ) } with edges i →
j if i → j is in G , { j ( l ) → i ( l ) } l =2 , ··· ,k if i → j is in G , and { i ( l ) → i } l =2 , ··· ,k for i ∈ [ n ] .
Define a new vertices set Z ′ :
= ∪{ i (2) , · · · , i ( k ) }
i ∈
Z , then we have:
[Equation]
Note that the trek-separation theorem can be viewed as a special case of Theorem 16 with k = 2 , where “ ( S W , S Y ) t-separates ( W , Y ) ” is equivalent to “ S ′ W ∪ S Y vertex cuts W ′ to Y ”.
The proof to Theorem 16 also basically follow the proof to Theorem 2.8 in [47]: using the LindströmGessel-Viennot theorem [25, 13], the max-flow min-cut theorem (vertex version, known as Menger’s theorem) [9, 3, 26], and applying the Cauchy–Binet determinant expansion formula and Schur properties repeatedly on the Hadamard products in Equation (E.6).
With the graphical criteria stated in Theorem 16 and under generic Assumption 2, we could have a method to implement TIN by ranks of stacked cumulants in sequence:
Theorem 17 (Use ranks’ stopped increasing to implement TIN ) .
For two variables sets Z , Y , there must exists a finite order k ≥ 2 s.t.
[Equation]
Moreover, this one-step-stop yields an infinite-steps-stop, i.e.,
[Equation]
) , for any l > k.
(E.20) and, this stopped-increasing rank equals exactly to TIN( Z , Y ) , i.e., s.t.
[Equation]
(E.21)
The original GIN condition using only covariance matrix can be viewed as a special case, which could be implemented as “
[Equation]
Note that independence test is not used in this method.
We could also use independence tests to test whether null(Ψ ( k ) Z ; Y ) is equal to Ω Z ; Y , just like the 2-steps method in §5.4.
Independence yields zero cumulants, and also yields independence among functions of variables.
Hence in term of solving equations system, null(Ψ ( k ) Z ; Y ) and cov( f ( Z ) , Y )
ω
= 0 are both correct.
However, the latter does not have additional graphical criteria as Theorem 16.
Empirically, the latter performs better, since higher order cumulants yield higher order exponential, which is sensitive to outliers.
E.2 For TIN-2steps , TIN-subsets , and TIN-ISA
Implementation details for these three methods can be referred in Appendix G.1.
Specifically,
TIN-ISA directly follows the Independent Subspace Analysis (ISA) from the original paper [48].
F Discussions F.1 Details on Assumptions
In this paper, except for the LiNGAM assumption for the causal model, we also give Assumptionin §3:
Assumption 1 (Rank faithfulness) .
Denote by B ( G ) the parameter space of mixing matrix B consistent with the DAG G .
For any two subsets of variables Z , Y ⊆ X , we assume that
[Equation]
Roughly speaking, Assumption 1 assumes there are no edge parameter couplings to produce coincidental low rank.
Note that violation of Assumption 1 is of Lebesgue measure 0, and LiNGAM is testable.
Here we discuss more details on Assumption 1 by two examples of violation:
[Equation]
Violation example 1 : Consider the graph in Equation (F.1), if coincidentally the edge weights c = − ab , then the noise components E 1 will be cancelled from X 3 , and marginally X 1 ⊥⊥ X 3 .
In this violation, graphically Anc( X 3 ) = { X 1 , X 2 , X 3 } , but the column indices of B X 3 , : with non-zero entries is just {
X 2 , X 3 } .
X
X
X
Xa b c d e ; A =    0  0c a 0d b e 
 ; B =    0  0c a 1ce + d ae + b e 
 (F.2)
Violation example 2 : Consider the graph in Equation (F.2).
Let Z :=
{ X 1 , X 2 } and Y :=
{ X 3 , X 4 } , by the graphical criteria we have TIN( Z , Y )
= 2 , with the critical vertex cut S ∗ Z ; Y =
{ X 3 , X 4 } .
Mathematically, B Y ;nzcol( B Z , : )
= (cid:20) c
a ce + d ae + b (cid:21) , which has rank 2 under generic parameters choice.
However, if bc = ad , then coincidentally the rank will drop to 1 , and thus Assumption 1 is violated.
Note that in this violation example, there is no noise cancelling (like violation example 1), i.e., here nzcol( B Z , : is exactly Anc( Z ) , but there is still coincidental low rank by parameter coupling.
Now we further discuss an example where Assumption 1 is satisfied (and thus is a valid case in this paper), but is not a valid case in the trek-separation paper [47] or the GIN paper [50]:
[Equation] [Equation]
0  1

[Equation]
B
[Equation]
   0  01 − 1 11 − 1 1
 (F.3)
Satisfaction example 3 : Consider the graph in Equation (F.2).
For every pair of Z , Y , there is no coincidental low rank in B Y , Anc( Z ) .
Hence, Assumption 1 is satisfied.
E.g., let Z :=
{ X 2 } , Y :=
{ X 3 , X 4 } , by the graphical criteria TIN( Z , Y )
=
1 (with S ∗ Z ; Y = { X 3 } ), and B Y , Anc( Z ) is also of rank 1 .
However, if we carefully choose noise components’ parameters so that the variance of exogenous noise E
and E 2 are equal ( var( E 1 )
=
var( E 2 ) ), then the variance-covariance matrix cov( { X 2 } , { X 3 , X 4 } ) would be [0 0] (coincidentally dropped to rank 1 ).
This coincidental low rank is due to noise parameters, and will not affect our proposed method in this paper, because we directly find Ω Z ; Y .
However, e.g., in GIN where ω is characterized by 2nd-order variance-covariance matrix, by solving equation here, any w ∈ R 2 is a solution.
Then, not every linear combination of
X 3 and X 4 is independent to X 2 , so GIN will output ‘ GIN( Z , Y ) violated’ in this case, though according to the graphical criteria, GIN( Z , Y ) is satisfied here.
F.2 More than Ordered Group Decomposition can be Identified
In this paper, we use the TIN condition to identify the ordered group decomposition of ˜ G in the measurement error model.
Specifically, we only use a special type of TIN , one-and-others (Lemma 1).
However, actually by using the TIN condition over more general pairs of Z , Y , more information of ˜
G can be identified.
For example, in the chain structure (Figure 2a) and the fully connected DAG (Figure 2c), the ordered group decomposition are both { ˜ X 1 } → { ˜ X 2 } → · · · → { ˜ X n − 2 } → { ˜ X n − 1 , ˜ X n } .
However, the two can actually be distinguished:
In the fully connected DAG, TIN( { X 2 } , { X 3 , · · · , X n } ) = 2 , while in the chain structure, TIN( { X 2 } , { X 3 , · · · , X n } )
= 1 .
Even under a same pair of Z , Y , the ω degeneration may be different.
E.g., TIN( { X 2 } , { X 1 , X 3 , X 4 , · · · } ) = 2 in both graphs.
However, in the chain structure, ω is degenerated on the index X 1 (i.e., the linear combination of Y cannot include X 1 .
If ω 1 X 1 + ω 3 X
+
ω
X
+ · · · is independent to X 2 , then ω 1 must be zero), while there is no degeneration of ω in the fully connected DAG.
Generally speaking, our final objective is to identify an “equivalence class” of ˜ G w.r.t.
the TIN condition.
We have talked about the concept of “unidentifiable” in §2.
Here, two graphs (either non-isomorphic or isomorphic but with labelling permutation) are unidentifiable w.r.t.
the TIN condition, if and only if for any two pairs Z , Y , TIN( Z , Y ) are same (with same degeneration).
About “equivalence class”, we already knew some features that an equivalence class should possess, e.g., a variable is naturally unidentifiable with its pure leaf child in ˜ G
(see Definition 6).
Apparently, there are more such features to be discovered.
Here are some of the examples:
Example 8 (Equivalence class for the chain structure) .
Consider a chain structure with 5 nodes ˜
X 1 → · · · → ˜ X 5 , and the following graphs with 5 nodes:
1. 5 edges: ˜ X 1 → · · · → ˜ X 5 , with an additional ˜ X 3 → ˜ X 5 .
5 edges: ˜ X 1 → · · · → ˜ X 5 , with an additional ˜ X 2 → ˜ X 4 .
5 edges: ˜ X 1 → · · · → ˜ X 5 , with an additional ˜ X 1 → ˜ X 3 . 4.
6 edges: ˜ X 1 → · · · → ˜ X 5 , with additional ˜ X 1 → ˜ X 3 and ˜ X 3 → ˜ X 5 .
For these five non-isomorphic graphs, with two equivalent permutations of each (swap the labeling of ˜
X 4 and ˜ X 5 ) - these 10 graphs form an equivalence class.
One might be curious: what if a graph with one more edge, i.e.,
7 edges: edges: ˜ X 1 → · · · → ˜ X 5 , with additional ˜ X 1 → ˜ X 3 , ˜ X 2 → ˜ X 4 and ˜ X 3 → ˜ X 5 .
However, this graph is no longer in the equivalence class.
For example, TIN( X 3 , X 4 , 5 ) = 1 for the chain structure (and its equivalence class), while TIN( X 3 , X 4 , 5 ) = 2 for this graph.
Example 9 (An equivalence class with one unique graph) .
Consider a ˜ G with 5 nodes and 7 edges: ˜
X 1 → { ˜ X 2 , ˜ X 3 , ˜ X 4 } , ˜ X 2 → { ˜ X 3 , ˜ X 5 } , and ˜ X 3 → { ˜ X 4 , ˜ X 5 } : surprisingly, its equivalence class contains only one graph, itself.
I.e., by TIN conditions this structure should be uniquely recovered.
With the equivalence class, the identifiability result could be improved, and constrained O-ICA may be further applied to identify a final graph.
It would be an interesting future work to characterize the “equivalence class” w.r.t.
TIN , and then design an algorithm to identify it
. F.3 More than Dimension of Ω Z ; Y : Parameters
Currently we only care about the dimension of the independent subspace Ω Z ; Y , but not the exact parameters .
If we have obtained exactly the Ω Z ; Y , we could write its basis matrix M Ω Z ; Y in shape | Y | × dim(Ω Z ; Y ) , with each column vector being a basis.
Then, the subspace spanned by row vectors of B Y , Anc( Z ) , which reflects edge parameters, is exactly the left nullspace of M Ω Z ; Y .
The degeneration of ω we discussed in Appendix F.3 and Theorem 13 is actually a special case of recovering information from Ω Z ; Y parameters.
For edge parameters, it means that rank of B Y , Anc( Z ) will drop one if deleting the respective degenerated columns in Y .
More general exploitation of Ω Z ; Y parameters is an interesting future work.
F.4 More Possible Solutions for Estimation of Ω Z ; Y
In §5 we propose four methods to estimate Ω Z ; Y : tackling down to subsets of Y (§5.1), constrained independent subspace analysis (ISA) (§5.2), stacked cumulants’ ranks stopped increasing (§5.3), and TIN in two steps: solve equations, and then test for independence (§5.4).
Generally, reliable estimation of Ω Z ; Y can be formulated as an orthogonal research problem, and we believe that there exists more solutions.
For example, if we only care about the dimension of Ω Z ; Y , the following heuristic method might help.
The intuition is that, uniformly sample infinite many random points on the surface of a unit sphere (centered on origin point) at R n , denote d ( k ) the average distance from these points to a subspace in
R n with dimension k ( 0 ≤ k ≤ n ).
Then this average distance is monotonic over k : d ( k 1 ) <
d ( k 2 ) if and only if k 1 > k 2 .
For example, on an 2D circle, d (0) = 1 (to center; radius), d (1) =
2 /π (to diameter), and d (2) =
(already on 2D)
; on a 3D sphere surface, d (0) = 1 (to center; radius), d (1) = π/ 4 (to diameter), d (2) = 1 / 2 (to diameter plane), and d (3) =
(already on 3D).
If we assume the independence tests return a bool value (independent or not), then this method will not help, because generally, the measure of Ω Z ; Y relative to R | Y | is always zero.
However, if we assume that, for a unit vector ω ∈ R | Y | , there exists a monotonic relationship between the independence strength of Ind( ω ⊺ Y ; Z ) (e.g., mutual information) and the distance to the subspace dist( ω ; Ω Z ; Y ) , then we could have a non-parametric method to recover ˜ G : for each variable X
i , uniformly sample many { ω l } l =1 , ··· from R ( n − 1) and calculate the average independence avg l Ind( ω ⊺ l [ X \ X i ]; X i ) , then sort X i by their respective average independence (i.e., dimensions of there respective Ω Z ; Y ) to get an estimation of the group ordering.
F.5 What if Causal Sufficiency is Not Satisfied in ˜ G ?
In this paper we assumed causal sufficiency relative to ˜ X .
Though it is reasonable to assume causal sufficiency in this context (which, to the best of our knowledge, is indeed a common assumption in the current literature of causal discovery with measurement error), this assumption itself, is a strong one and is not testable.
Thus, it would be interesting to investigate the case where causal sufficiency is violated (in a sense of “latents of latents”): Will TIN -based method still output a correct ordering?
If not, by which correction rules or algorithm relaxations can the identifiability be still partially preserved?
We leave this as an interesting future work.
For now, we try to provide some hints from examples (where we still use the Lemma 1-based method in this paper):
Example 10 (A still (partially) identifiable case) .
Consider a chain structure ˜ X 1 → ˜ X 2 → · · · → ˜ X n (or similarly, a fully connected DAG) with a common hidden confounder ˜
L pointing to them all: ˜
L → { ˜ X i } ni
=1 .
If ˜ L is not measured and only measurements X = { X 1 , · · · , X n } are available, we have now: ord ( X 1 ) =
TIN( X 1 , X \ X 1 ) = 2 , ord ( X 2 )
= 3 , · · · , ord ( X n − 3 )
=
n − 2 , and ord ( X n − 2 )
= ord ( X n − 1 )
= ord ( X n ) = n − 1 .
We shall see that: 1) The causal ordering of all but the last 3 variables is identifiable.
While without ˜ L (our previous result), this identifiability result is all
but the last 2 variables (see Example 6), and 2) the existence of hidden (root) confounder(s) will also be reported, since there is no root (with ord = 1 ) found across measurements X .
Example 11 (A no-longer identifiable case) .
Consider a simple fork ˜
X 2 ← ˜ X 1 → ˜ X 3 , with a hidden confounder ˜
L : ˜ L → ˜ X 1 and ˜
L → ˜ X 2 .
Then, ord ( X 1 ) =
TIN( X 1 , X \ X 1 ) = 2 , ord ( X 2 )
= 1 , and ord ( X 3 )
= 2 .
Sorting by ord , we have the group decomposition as { ˜ X 2 } → { ˜ X 1 , ˜ X 3 } , while this is incorrect: there exists directed edge(s) from later groups to earlier groups, ˜ X 1 → ˜ X 2 .
F.6 What if some Measurements are Caused by Multiple Latent Variables?
In this paper, we consider the measurement error model, where each measurement is caused by only one latent variable.
For GIN , it can generally handle the cases where measurements are caused by multiple latent variables, as long as each latent variable has enough pure indicators.
Interestingly however, we find that this may also be relaxed for our case (where there are not enough pure indicators), and our TIN -based method may still work (in identifying the correct group ordering).
See below for some simple examples:
Consider a 3 -nodes chain structure ˜
A → ˜ B → ˜ C , and their respective measurements A, B, C .
We have the ordered group decomposition { ˜ A } → { ˜ B, ˜ C } , with ord being 1 and 2 .
Then, what if we add an edge from a latent variable to another measured variable?
There are 3 × 2 = 6 ways of adding an edge.
Surprisingly, among these 6 ways, there are 5 which preserves exactly the same TIN results over A, B, C .
The only one difference is by adding ˜ C
→ A , where TIN( A, BC ) = 2 , instead of 1 .
It would be interesting to generalize this observation: What if more nodes?
What if more edges?
G Implementation and Evaluation G.1 Implementation Details
In this section we provide the information required to reproduce our results reported in the main text.
We also commit to making our implementations of TIN public.
Simulation setup In simulation we consider specifically two cases: fully connected DAG (Figure 2c) and chain structure (Figure 2a), of which the ordered group decomposition are both ˜
X 1 (cid:41) ··· (cid:41) ˜
X n − 2 , ˜ X
n − 1 ,n .
We consider ˜ G with the number of vertices n = 3 , ··· , 10 .
Edges weights (i.e., the nonzero entries of matrix A ) are drawn uniformly from [ − 0 .
9 , − 0 .
5] ∪ [0 . 5 , 0 . 9] .
Exogenous noises ˜ E are sampled from uniform ∪ [0 , 1] to the power of c , c ∼ ∪ [5 , 7] , and measurement errors are sampled from Gaussian N (0 , 1) to the power of c , c ∼ ∪ [2 , 4] .
Sample size is 5 , 000 .
Observations are generated by X i =
˜
X
i + E i .
To show the effect of measurement error, we simulate with noise-to-signal ratio NSR :
= var( E i ) / var( ˜ X i ) in { 0 . 5 , 1 , 2 , 3 , 4 } .
On each configuration (under a graph type, measurement noise scaling, and the number of vertices), 50 random graphs are generated for repeated experiments.
PC We use the implementation from the causal-learn package 5 .
Kernel-based conditional independence test [52] is used.
For speed consideration, datasets are downsampled to 1 , 000 on PC runs.
The significance level alpha is set to 0 .
to Definition 7.
GES
We use the implementation from the causal-learn package 6 .
The score used is local-BIC-score [37].
Direct-LiNGAM and ICA-LiNGAM
We use the implementation from the lingam package 7 .
Note that for Direct-LiNGAM, actually the method used is based on pairwise likelihood ratios [20].
CAMME-OICA
We use the implementation from LFOICA 8 (Likelihood-Free Overcomplete ICA).
It estimates the mixing matrix by first transforming random noise into components, and then mimic the mixing procedure from components to noise with MMD score as a metric.
Below we give details on implementations of TIN .
Specifically,
Independence test We use the HSIC (Hilbert-Schmidt independence criterion) test [14] with the implementation from lingam package 9 .
The kernel width is set to 0 .
1 times the standard deviation of the data samples.
The significance level alpha of p-value is set to 0 .
Note that when the noise-to-signal ratio is large (e.g. > 3 ), usually observed variables are already ‘independent enough’ (i.e., with p-value given by HSIC test on raw data samples already > 0 . 05 ).
In this case, we use the difference of 1000 ∗ severity sample size between Z ; Y and Z ; ω ⊺ Y to show how much independence is ‘gained’ by linear transformation.
The threshold for this criterion is set to 0 .
TIN-ISA
We implement the constrained ISA where the de-mixing matrix is masked to only update the lower-right | Y | × | Y | block W YY , with upper-left | Z | × | Z | block fixed as the identity and elsewhere fixed as zero.
We follow [29] for the estimation of conditional score function.
Independence between Z and ω ⊺ Y for each row of W YY is then tested by HSIC test, as is described aboce.
TIN-rank Numerical rank of a 2D matrix is calculated by SVD (singular value decomposition), with tolerance ϵ set to 0 .
Singular values below threshold T are considered zero, where
T =
ϵ ∗ max( S ) ∗ max( M, N ) .
S is all singular values, and M, N are shape of the 2D matrix.
According to Theorem 5 , we use the rank where stacked 2D slices of cumulants stops increasing rank as the output of TIN .
TIN-2steps To solve euqations system { cov( f ( Z ) , Y ) ω
= 0 } , functions f contain: Z , Z 2 , Z 3 , | Z | , e Z , log( | Z | ) , sin( Z ) , cos( Z ) , sigmoid( Z ) , tanh( Z ) .
Nullspace is calculated by SVD, while we do not set a hard threshold of singular value to determine its space (like TIN-rank).
Instead, we test
HSIC between Z and ω ⊺ Y for each ω in the | Y | × | Y | unitary matrix V , and count the number of independence achieved.
TIN-subsets
The core to find the existence of transformed independence is similar to TIN-2steps.
Then, for the part of traversing over Y ’s subsets, “all Y ′ ...”
and “exists a Y ′ ...”
are characterized by 90% and 10% percentile of the independence statistics (e.g., p-value of HSIC test) respectively.
Noise synthesis Edges weights (i.e., the nonzero entries of matrix A ) are drawn uniformly from [ − 0 .
9 , − 0 .
5] ∪ [0 . 5 , 0 . 9] .
Exogenous noises ˜ E of the latent variables are sampled from uniform ∪ [0 , 1] to the power of c , c ∼ ∪ [5 , 7] , and measurement errors are sampled from Gaussian N (0 , 1) to the power of c , c ∼ ∪ [2 , 4] .
Sample size is 5 , 000 .
Below we show a synthetic dataset with ˜ G being a fully connected DAG with n = 7 , and the noise-to-signal ratio being 3 (Figures 6 and 7):
[Figure] [Figure]
[Figure]
G.2 Evaluation Details
To evaluate the output group ordering, we use Kendall tau distance [22] to the ground-truth (in range [0 , 1] , the lower the better).
Kendall tau distance counts the number of pairwise disagreements between two orderings.
Specifically, for two variables i, j and a grouped ordering τ , we define:
[Equation] [Equation]
[Equation]
  in τ, i is in an earlier group than j in τ, i is in a same group with j −in τ, i is in a later group than j (G.1)
Then, for the n variables [ n ] and two ordered group decompositions τ 1 , τ 2 on them, the Kendall tau distance is defined as:
[Equation]
For algorithms returning DAG/PDAG, its ordering is first extracted according to Definition 7.
Specifically, for PDAG, a vertex’s ancestors is defined as all vertices that has mixed paths (no directed edges backward) to it.
For intuition, here we give some typical examples: if the true ordering is {{ X 1 } , { X 2 } , { X 3 } , { X 4 , X 5 }} , the following group orderings have the respective distances:
{{ X 5 } , { X 4 } , { X 3 } , { X 2 , X 1 }} : 1 . 0 ; 2. {{ X 5 } , { X 4 , X 3 , X 2 , X 1 }} : 1 . 0 ; 3.
{{ X 1 , X 2 , X 3 , X 4 , X 5 }} : 0 . 9 ; 4.
{{ X 2 , X 4 } , { X 1 , X 3 , X 5 }} : 0 . 8 ; 5.
{{ X 1 , X 4 } , { X 2 , X 3 , X 5 }} : 0 .
{{ X 1 , X 2 , X 4 } , { X 3 , X 5 }} : 0 . 6 ; 7.
{{ X 2 } , { X 1 , X 4 } , { X 3 , X 5 }} : 0 . 5 ; 8. {{ X 1 , X 3 } , { X 2 , X 4 , X 5 }} : 0 . 4 ; 9.
{{ X 1 , X 2 } , { X 3 , X 4 , X 5 }} : 0 . 3 ; 10. {{ X 1 , X 2 } , { X 3 } , { X 4 , X 5 }} : 0 . 1 ;
H Detailed Elaboration on Examples
To explain the TIN condition’s definition, characterization, and graphical criteria, in this section we provide some step-by-step derivation of typical examples.
H.1 Motivation Examples
We first give an example of GIN to show our motivation (§2):
Example 12 ( GIN on chain structure Figure 2a) .
Let Z :=
{ X 1 } , Y :=
{ X 2 , X 3 , X 4 , X 5 } .
Calculate cov( X 1 , X 2 ) =
cov( ˜ X 1 + E 1 , ˜ X
2 + E 2 ) =
cov( ˜ X 1 , ˜ X 2 ) =
cov( ˜ X 1 , a ˜ X 1 + ˜ E 2 )
= a var( ˜ X 1 ) .
Similarly, we get the covariance matrix
[Equation]
(H.
[Equation]
Solve the linear homogeneous equations cov( Z , Y ) ω
= 0 , we have ω =
[ − bx − bcy − bcdz, x, y, z ] ⊺ , x, y, z ∈ R .
Plug ω into ω ⊺ Y : ω ⊺
[Equation]
=
[Equation]
− (cid:26)(cid:26)
[Equation]
bcy − (cid:24)(cid:24)(cid:24)
[Equation]
(H.2)
By above we have GIN( { X 1 } , { X 2 ,
, X n } ) satisfied.
H.2 Examples of Ordered Group Decomposition
In Definition 7 we define the ordered group decomposition of a graph.
Actually there is slight difference between Definition 7 and Definition 2 in [54].
According to Definition 7, when the graph has only one subroots at each step, the two definitions are the same.
However, when there are multiple subroots at some step, Definition 7 takes all such subroots (and their pure leaf children) as a new group, while in [54], each group has one and only one non-leaf node, and thus only one subroot (and its pure leaf children) is taken (and removed from graph) as a new group, which yields multiple ordered group decompositions.
Here we only return one ordered group decomposition mainly for simplicity.
To obtain the multiple ordered group decompositions defined in [54] from our result, we could do the following: for each pair of X i , X j in a new group with multiple variables, test TIN with
Z being X i and Y being X j and all variables in the previous groups, to see whether X i , X j are in a same cluster .
Below we give an example to show this slight difference between two definitions:
Example 13 (Ordered group decomposition with multiple subroots) .
Consider the graph D ←
A → C ← B .
The ordered group decomposition we defined in Definition 7 is { A, B, D } → { C } , while Definition 2 in [54] will give two ordered group decompositions: { A, D } → { B, C } , and { B } → { A, C, D } .
H.3 A Concrete Example of Using TIN on Fully Connected DAG
Consider a fully connected DAG (Figure 2c) with 4 variables { X 1 , X 2 , X 3 , X 4 } ⊺ (suppose we have access directly to the measurement-error-free variables and directly test TIN over them), the edges parameters are:
[Equation]
1) Let Z :=
{ X 1 }
Y :=
{ X 2 , X 3 , X 4 } ,
we have the independent subspace:
Ω Z ; Y
=
null
B
⊺
Y , Anc
[Equation]
Z
[Equation]
) = null
[Equation] [Equation]
Two basis of Ω Z ; Y are:
[Equation]
For any ω = k 1 ω 1 + k 2 ω 2 , k 1 , k 2 ∈ R , ω ⊺
Y does not contain noise E 1 , so ω ⊺ Y ⊥⊥ Z .
We have TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) =
3 − 2 = 1 .
Graphically, the minimum vertex cut from
Anc( { X 1 } ) =
{ X 1 } to { X 2 , X 3 , X 4 } is { X 1 } , with size 1.
And, | Anc( X 1 ) | = 1 .
2) Let Z :=
{ X 2 } , Y :=
{ X 1 , X 3 , X 4 } ,
we have the independence subspace: Ω
Z ; Y = null( B ⊺ Y , Anc( Z
= null
[Equation]
dimension
[Equation] [Equation]
One basis of Ω Z ; Y is:
[Equation]
For any ω = k 1 ω 1 , k 1 ∈ R , ω ⊺ Y does not contain noise E 1 , E 2 , so ω ⊺ Y ⊥⊥ Z .
We have
TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) =
3 − 1 = 2 .
Graphically
, the minimum vertex cut from Anc( { X 2 } )
= { X 1 , X 2 } to { X 1 , X 3 , X 4 } is { X 1 , X 2 } , with size 2.
And, | Anc( X 2 ) | = 2 . 3) Let Z := { X 3 } , Y :=
{ X 1 , X 2 , X 4 } , we have the independence subspace: Ω Z ; Y = null( B ⊺ Y , Anc( Z ) ) =
null
[Equation]
, dimension=0 .
(H.8) B ⊺ Y , Anc( Z ) is full column rank, so that there exists no non-zero ω s.t. ω ⊺ Y ⊥⊥ Z , i.e., Ω Z ; Y contains only origin point R 0 , with dimension 0.
We have TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) = 3 − 0 = 3 .
Graphically, the minimum vertex cut from Anc( { X 3 } )
=
{ X 1 , X 2 , X 3 } to { X 1 , X 2 , X 4 } is { X 1 , X 2 , X 3 } or { X 1 , X 2 , X 4 } , with size 3.
And, | Anc( X 3 ) | = 3 .
4) Let Z :=
{ X 4 } , Y :=
{ X 1 , X 2 , X 3 } , we have the independence subspace: Ω Z ; Y = null( B ⊺ Y , Anc( Z ) ) =
null     1 a ad + b  d      
   , dimension=0 .
(H.9)
B ⊺ Y , Anc( Z ) is full column rank, so that there exists no non-zero ω s.t. ω ⊺ Y ⊥⊥ Z , i.e., Ω Z ; Y contains only origin point R 0 , with dimension 0.
We have TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) = 3 − 0 = 3 .
Graphically, the minimum vertex cut from Anc( { X 4 } )
=
{ X 1 , X 2 , X 3 , X 4 } to { X 1 , X 2 , X 3 } is { X 1 , X 2 , X 3 } , with size 3.
And, | Anc( X 4 ) | = 4 .
Since X 4 is a leaf node, TIN( Z , Y ) = 4 − 1 = 3 .
By above, we obtain the ordered group decomposition {{ X 1 } , { X 2 } , { X 3 , X 4 }} .
H.4 A Concrete Example of Using TIN on the Chain Structure
To align with Appendix H.3, here we consider a chain structure with 4 variables { X 1 , X 2 , X 3 , X 4 } ⊺ .
The edges parameters are:
A =   0 0a 0 0 b 0  c   ; B =    0a 0ab
b 1abc bc c
[Equation]

[Equation]
H
[Equation]
10) 1) Let
Z :=
X
Y :=
X 2 , X 3 , X 4 } ,
we have the independent subspace:
[Equation]
[Equation]
Two basis of Ω Z ; Y are:
[Equation]
We have TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) =
3 − 2 = 1 .
Analysis is similar to that of Appendix H.3.
2) Let Z :=
{ X 2 } , Y :=
{ X 1 , X 3 , X 4 } , we have the independence subspace:
Ω Z ; Y = null( B ⊺ Y , Anc
( Z ) )
= null
(cid:18)(cid:20) 1 ab abc b bc (cid:21)(cid:19) , dimension=1 .
(H.13)
One basis of Ω Z ; Y is:
[Equation]
We have TIN( Z , Y ) =
| Y | − dim
Ω Z ; Y
= 3 − 1 = 2 .
Analysis is similar to that of Appendix H.3.
Let Z := { X 3 } , Y :=
{ X 1 , X 2 , X 4 } , we have the independence subspace:
Ω Z
Y = null
B ⊺ Y , Anc
Z
= null
(cid:32)(cid:34) 1 a abc  bc  c (cid:35)(cid:33) ,
dimension
=0 .
(H.15)
We have TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) = 3 − 0 = 3 .
Analysis is similar to that of Appendix H.3.
Let Z :=
{ X 4 } , Y :=
{ X 1 , X 2 , X 3 } , we have the independence subspace:
Ω Z ; Y = null( B ⊺ Y , Anc
( Z ) )
=
null     1 a ab  b      
   , dimension=0 .
(H.16)
We have TIN( Z , Y ) =
| Y | − dim(Ω Z ; Y ) = 3 − 0 = 3 .
Analysis is similar to that of Appendix H.3.
By above, we get the group ordering {{ X 1 } , { X 2 } , { X 3 , X 4 }} .
Recall the chain structure with triangular head example, we could distinguish it from the chain structure with ordered group decomposition {{ X 1 } , { X 2 , X 3 , X 4 }} .
H.5 Experiments on another Real-world Dataset: Teacher Burnout
Except for Sach’s dataset discussed in §6.2, we also conduct experiments on another real-world dataset,
Teacher Burnout [4].
It is from a sociology survey conducted by Barbara Byrne to investigate the influence on the three facets (emotional exhaustion, depersonalization, and personal accomplishment) of full-time elementary teachers’ burnout from factors including: organizational (role ambiguity, role conflict, work overload, classroom climate, decision making, superior support, peer support) and personality (self-esteem, external locus of control) variables.
Please see chapter six of [4] for more details about the dataset (Page 161), and the structure (Page 191).
While in the raw dataset, each (latent/target) variable has more than one measurements/indicators, in this experiment we pick only one measurement for each to demonstrate the measurement error situation.
Specifically, we pick ten variables (according to the ten latent variables in Figure 6.
of [4]): RA 1 , RC 1 , CC 1 , DM 1 , SS 1 , SE 1 , ELC 1 , EE 1 , DP 1 , and PA 1 .
Though for a thorough study of the dataset, one could try other combinations of measurements, e.g., RA 2 , RC 3 , ..., in this experiment we only study one combination as above for illustration.
According to Figure 6.10 of [4], the ordered group decomposition of the ground-truth underlying causal graph is { RA, RC, CC, DM, SS } (cid:41) { SE, ELC } (cid:41) { EE } (cid:41) { DP, PA } .
Result given by
TIN-subsets is { DM, SE } (cid:41) { CC, SS } (cid:41) { RA, ELC } (cid:41) { RC, EE, DP } (cid:41) { PA } (with the oneover-others TIN being 5 , 6 , 7 , 8 , 9 respectively).
This is similar to Byrne’s conclusion (the true ordering) according to the domain knowledge.
For example, 1) the three facets of burnout (emotional exhaustion, depersonalization, and personal accomplishment) are caused by other factors and are at the end of the ordered groups, 2) decision making, classroom climates and superior support are root causes (in the first two groups), and 3) self-esteem and role conflict influences external locus of control.
Interestingly, some of the ordering inconsistent with the ground-truth might also be reasonable to some extent.
For example, 1) self-esteem is among the first group (though should be in the second), maybe because it is “root-like”: it is only caused by two root causes and causes another four variables, 2) decision making and superior support are in the first and second groups respectively (though should both be in the first, as two root causes for self-esteem), maybe because there exists difference in their impact on others, and 3) role ambiguity is in the third group (though should be in the first), maybe because that though it is a root, it has only one child, personal accomplishment, which is a leaf node in the graph; the same may applies to role conflict: though being a root, it is even among the second to last group, which is also echoed by other methods.
Here is an overview of the distance scores and ordered groups returned by all methods:
TIN-2steps : 0.49, { CC, SE, ELC } (cid:41) { RA, DM, SS } (cid:41) { EE, PA } (cid:41) { RC, DP } . 2.
TIN-subsets : 0.47, { DM, SE } (cid:41) { CC, SS } (cid:41) { RA, ELC } (cid:41) { RC, EE, DP } (cid:41) { PA } .
TIN-ISA : 0.56, { RA, DM, SE } (cid:41) { SS, ELC } (cid:41) { RC, CC, EE, DP, PA } . 4.
TIN-rank : 0.60, { CC, SE } (cid:41) { RA, RC, DM, ELC, DP } (cid:41) { SS, EE, PA } .
ICA-LiNGAM: 0.56, { CC } (cid:41) { SE } (cid:41) { ELC } (cid:41) { RA } (cid:41) { SS } (cid:41) { EE } (cid:41) { RC, PA } (cid:41) { DM, DP } . 6.
Direct-LiNGAM: 0.73, { DP } (cid:41) { SE } (cid:41) { SS } (cid:41) { RA } (cid:41) { RC, PA } (cid:41) { CC } (cid:41) { EE } (cid:41) { DM, ELC } .
CAMME-OICA: 0.78, { CC, SS, EE, DP, PA } (cid:41) { RA, RC, DM, SE, ELC } . 8.
PC: 0.73, { RA, RC, CC, DM, SS, SE, ELC, EE, DP, PA } . 9.
GES: 0.73, { CC, DM, EE, DP } (cid:41) { RA } (cid:41) { SS, SE, PA } (cid:41) { RC, ELC } .
NOTEARS: 0.82, { CC, SE, ELC, PA } (cid:41) { RA, EE, DP } (cid:41) { RC } (cid:41) { DM, SS } .
SCORE: 0.76, { SE } (cid:41) { SS } (cid:41) { RA, CC, DM, ELC, EE, DP, PA } (cid:41) { RC } .