Independence Testing-Based Approach to Causal Discovery under Measurement Error and Linear Non-Gaussian Models
Abstract
Causal discovery aims to recover causal structures generating the observational data.
Despite its success in certain problems, in many real-world scenarios the observedvariables are not the target variables of interest, but the imperfect measures ofthe target variables.
Causal discovery under measurement error aims to recoverthe causal graph among unobserved target variables from observations made withmeasurement error.
We consider a speciﬁc formulation of the problem, wherethe unobserved target variables follow a linear non-Gaussian acyclic model, andthe measurement process follows the random measurement error model.
Existingmethods on this formulation rely on non-scalable over-complete independentcomponent analysis (OICA).
In this work, we propose the Transformed Independent
Noise ( TIN ) condition, which checks for independence between a speciﬁc lineartransformation of some measured variables and certain other measured variables.
By leveraging the non-Gaussianity and higher-order statistics of data, TIN isinformative about the graph structure among the unobserved target variables.
Byutilizing TIN , the ordered group decomposition of the causal model is identiﬁable.
In other words, we could achieve what once required OICA to achieve by onlyconducting independence tests.
Experimental results on both synthetic and realworld data demonstrate the effectiveness and reliability of our method.
Introduction
Discovery of causal relations is a fundamental goal of science.
To identify causal relations fromobservational data, known as causal discovery, has thus drawn much attention in various scientiﬁcﬁelds, including economics, biology and social science [33, 19, 8].
Methods for causal discoverycan be roughly categorized into constraint-based ones (e g., PC [31]), score-based ones (e g., Greedy
Equivalence Search (GES) [6]) and ones based on structural equation models (SEM) [26, 12, 39].
Almost all these methods assume that the recorded values are values of the variables of interest,which however, is usually not the case in real-world scenarios.
Some variables may be impossible toobserve or quantify, so recorded values are actually a proxy of them (e.g., measure one’s mental statusby survey questionnaire), and some variables, though quantiﬁable, may subject to error introduced byinstruments (e.g., measure brain signals using functional magnetic resonance (fMRI)).
The differencebetween quantities of interest and their measured value is termed as measurement error [9].
[Figure]
Measurement error adversely impairs causal discovery [20, 18, 25].
Themeasuring process can be viewed as directed edges from underlyingvariables of interest (unobservable) to measured values (observable), andthe d-separation patterns on underlying variables typically do no hold onmeasured ones.
Consider the causal effects from factory emissions toair quality then to residents’ lung health , as shown in Figure 1, whilewe only have corresponding measured quantities: chimney statistics ,
PM 2 . 5 , and hospital reported cases .
Though and are independentgiven , and are however dependent given PM 2 .
If measurementerror is severe, and even tend to be marginally independent [40, 25],which makes PM 2 .
5 look like a collider.
One might thus incorrectly infer that, lung cancer causes air pollution.
In fact, such measurement error isalways a serious threaten in environmental epidemiologic studies [21, 7].
Denote by ˜ X
= { ˜ X i } ni =1 the latent measurement-error-free variables and X = { X i } ni =1 observed 44 ones.
While there are different models for measuring process [9, 4, 37], in this paper, we consider therandom measurement error model [25], where the observed variables are generated from the latentmeasurement-error-free variables ˜
X i with additive random measurement errors E =
{ E i } ni =1 :
X
[Equation]
= ˜
X [Equation] [Equation] Measurement errors E are assumed to be mutually independent and independent of ˜ X .
We particularlyfocus on the case where the generating process for ˜ X follows a linear, non-Gaussian, acyclic model(LiNGAM [26], see §3.1 for details).
Note that here w.l.o.g., the linear weights of { ˜ X i !
X i } ni =1 areassumed to be one (since we do not care about scaling).
Generally, if observations are measured by X i =
c i ˜
X
i + E i with weights { c i } ni =1 not necessarily being one, all results in this paper still hold.
The objective of causal discovery under measurement error is to recover causal structure amonglatent variables ˜ X , denoted as ˜ G , a directed acyclic graph (DAG), from contaminated observations
X .
As illustrated by Figure 1, causal discovery methods that utilize (conditional) independenceproduce biased estimation (see Proposition 1 for details).
SEM-based methods also typically fail toﬁnd correct directions, since the SEM for ˜
X usually do not hold on X .
Unobserved ˜ X are actuallyconfounders of X , and there exists approaches to deal with confounders, such as Fast Causal Inference(FCI [32]).
However, they focus on structure among observed variables instead of the unobservedones, which is what we aim to recover here.
With the interest for the latter, another line of researchcalled causal discovery with latent variables is developed, which this paper is also categorizedto.
However, existing methods [29, 30, 35, 34, 1, 17, 38, 24] cannot be adopted either, since theytypically require at least two measurements (indicators) for each latent variable, while we only haveone for each here (and is thus a more difﬁcult task).
Speciﬁcally on the measurement error problem,[11] proposes anchored causal inference in the binary setting.
In the linear Gaussian setting, [40]presents identiﬁability conditions by factor analysis.
A main difﬁculty here is the unknown variancesof the measurement errors E , otherwise the covariance matrix of ˜ X can be obtained and readily used.
To this end, [2] provides an upper-bound of E and [23] develops a consistent partial correlationsestimator.
In linear non-Gaussian settings (i.e., the setting of this paper), [41] shows that the orderedgroup decomposition of ˜ G , which contains major causal information, is identiﬁable.
However, thecorresponding method relies on over-complete independent component analysis (OICA [14]), whichis notorious for suffering from local optimal and high computational complexity [27, 13].
Hence, theidentiﬁability results in [41], despite the theoretical correctness, is far from practical achievability.
The main contributions of this paper are as follows: 1) We deﬁne the T ransformed I ndependent N oise( TIN ) condition, which ﬁnds and checks for independence between a speciﬁc linear transformation(combination) of some variables and others.
The existing Independent Noise ( IN [28]) and Generalized Independent Noise ( GIN [38]) conditions are special cases of TIN .
We provide graphicalcriteria of TIN , which might further improve identiﬁability of causal discovery with latent variables.3)
We exploit TIN on a speciﬁc task, causal discovery under measurement error and LiNGAM, andidentify the ordered group decomposition .
This identiﬁability result once required computationallyand statistically ineffective OICA to achieve, while we achieve it merely by conducting independencetests.
Evaluation on both synthetic and real-world data demonstrate the effectiveness of our method.
Motivation: Independence Condition and Structural Information
The example in Figure 1 illustrates how the (conditional) (in)dependence relations differ betweenobserved X and latent ˜ X , and thus lead to biased discovery results.
To put it generally, we have,
Proposition 1 ( rare d-separation) .
Suppose variables follow random measurement error modeldeﬁned in Equation (1) .
For disjoint sets of observed variables Z , Y , S and their respective latentones ˜ Z , ˜ Y , ˜ S , d-separation Z ??
d Y | S holds, only when marginally ˜ Z ??
d ˜ Y , and ˜ Z ??
d ˜ Y | ˜ S hold.
By ‘rare’ we mean that the d-separation patterns among ˜ X usually do not hold among X (exceptfor rare marginal ones), since the observed variables are not causes of any other (though the latentvariables they intend to measure might be).
For example, consider the underlying ˜ G to be chainstructure (Figure 2a) and fully connected DAG (Figure 2c).
There exists no (conditional) independenceon either, and PC algorithm will output just a fully connected skeleton on both cases.
Then, without(conditional) independence (which is non-parametric) to be directly used, can we create independence ,by leveraging the parametric assumption (LiNGAM) and beneﬁt from non-Gaussianity?
Naturally we recall the Independent Noise (IN) condition proposed in Direct-LiNGAM [28]:
Deﬁnition 1 ( IN condition) .
Let Y i be a single variable and Z be a set of variables.
Suppose variablesfollow LiNGAM.
We say ( Z , Y i ) satisﬁes IN condition, denoted by IN( Z , Y i ) , if and only if theresidual of regressing Y i on Z is statistically independent to Z .
Mathematically, let ˜ !
be the vector ofregression coefﬁcients, i.e., ˜ !
=
cov( Y i , Z ) cov( Z , Z )  1 ; IN( Z , Y i ) holds iff Y i  ˜ !
| Z ??
Z .
Here “ cov ” denotes the variance-covariance matrix.
IN identiﬁes exogenous (root) variable, basedon which the causal ordering of variables can be determined (Lemma 1 in [28]).
However, INcannot be applied to measurement error model.
With hidden confounders ( ˜ X ) behind observed X ,independence between regressor and residual typically does not exist on any regression among X .
Infact, X follows errors-in-variables models [10, 5], for which the identiﬁability w.r.t.
˜ G is not clear.
However, we might still beneﬁt from this idea to leverage non-Gaussianity of exogenous noises.
Consider the Figure 1 example and abstract it to ˜ X 1 !
a ˜ X 2 !
b ˜ X 3 with { ˜ X i !
1 X i } 3 i =1 .
Although 107 IN does not hold on any of X , interestingly, there exists a linear transformation of observationsbX 2  X 3 , which contains only { ˜ E 3 , E 2 , E 3 } ( ˜ E 1 and ˜ E 2 are cancelled out) and shares no commonnon-Gaussian noise term with X 1 .
Hence, by the Darmois–Skitovich theorem [15], bX 2  X 3 isindependent of X 1 .
This ﬁnding is echoed in Generalized Independent Noise ( GIN [38]) condition:
Deﬁnition 2 ( GIN condition) .
Let Z and Y be two sets of random variables that follow LiNGAM.
We say ( Z , Y ) satisﬁes the GIN condition, denoted by GIN( Z , Y ) , if and only if the following twoconditions are satisﬁed: 1) There exists nonzero solution vectors !
2 R | Y | to equation cov( Z , Y ) !
=0 , and 2)
Any such solution ! makes the linear transformation !
| Y independent of Z .
Here | Y | denotes the dimensionality of Y .
The intuition of GIN is that, despite no independent residual by normal regression, it is possible to realize independent “pseudo-residuals” [3] by regressingwith “reference variables” .
[38] shows IN as a special case of GIN (Proposition 2), and further givesgraphical criteria (Theorem 2), based on which a recursive learning algorithm is developed to solve thelatent-variable problem.
Each latent variable is required to have at least two observations.
Interestinglywe ﬁnd that, in measurement error models, if each latent variable ˜ X
i has two measurements
X i 1 , X i 2 ,then the GIN condition can be readily used to fully identify the structure of ˜ G , which is already abreakthrough over existing methods [30, 35, 34, 17].
See Appendix B for the whole procedure.
Withregard to our more challenging task where each ˜ X
i has only one measurement X i , a natural questionis that, can GIN also help?
Given the example in Figure 1 illustrated above, the answer seems to beafﬁrmative: GIN( { X i } , X \{ X i } ) only holds for i = 1 , so the root can be identiﬁed.
More generally:˜ X
[Equation]
X˜ X˜ X˜
Xa b c d (a) ˜ X˜ X˜ X˜ X˜
Xa c
b d e (b) ˜ X˜
X˜ X 3 ˜
Xa b c d e f (c) ˜ X˜ X˜ X˜ X˜
[Equation]
(d)
, observed X , measurement edges are omitted.
127 Example 1 ( GIN on chain structure) .
Consider cases where theunderlying graph ˜ G is a chain structure with n ( n  3 ) verticesand directed edges { ˜ X i !
˜
X i +1 } n  1 i
=1 .
Figure 2a is an example 130 with n = 5 .
We ﬁnd that GIN( Z = X 1 , Y =
X 2 , 3 , 4 , 5 )
holds(where X 2 , 3 , 4 , 5 denotes { X 2 , X 3 , X 4 , X 5 } ; same below), with 132 solution !
=
[Equation]
Ycancels noise components in Y that is also shared by Z , andthus !
| Y ??
Z .
However, GIN( X i , X \{ X i } ) is violated for 135 any other i 6 = 1 (see Example 8 for a detailed derivation).
Withthis asymmetry, the latent root ˜
X 1 can be identiﬁed.
Furthermore, GIN( X i , X i +1 , ··· ,n ) holds for any i = 1 , ··· , n  2 .
Example 1 might give us an intuition that by recursively testing
GIN , we could identify the causal ordering of ﬁrst n  2 variables for any DAG.
However, this is over-optimistic thanks tothe sparsity of chain structure.
Consider a denser structure:
Example 2 ( GIN on fully connected DAG) .
Consider cases where ˜ G is a fully connected DAGwith n ( n  3 ) vertices and directed edges { ˜ X i !
˜
X j } for every i < j .
Figure 2c is an examplewith n = 4 .
We ﬁrst ﬁnd that GIN( X 1 , X 2 , 3 , 4 ) holds.
However, in contrast to the chain structure,
GIN( X 2 , X 3 , 4 ) does not hold - there is no way to cancel both ˜ E 1 , 2 from X 3 , 4 .
More generally wehave: GIN( X 1 , X 2 , ··· ,n ) , GIN( X 1 , 2 , X 3 , ··· ,n ) , ··· , GIN( X 1 , ··· ,k , X k +1 , ··· ,n ) , k = b ( n  1) / 2 c .
Since a fully connected DAG is the densest extreme, Example 2 might give us an intuition that GINcould identify at least the causal ordering of the ﬁrst half of variables.
Unfortunately, this is stillover-optimistic, since we could not know beforehand the structure type of ˜ G .
Example 3 ( GIN on chain structure with triangular head ) .
Figure 2b shows a variation of chainstructure, with edges ˜
X 1 !
˜
X 2 , ˜
X 1 !
˜ X 3 , ˜
X 3 !
˜
X 2 , and { ˜ X i !
˜
X i +1 } n  1 i =3 .
We name it “chain structure with triangular head ”.
Interestingly, GIN( X 2 , X 3 , ··· ,n ) , which is satisﬁed on Figure 2a,is also satisﬁed here.
E.g., in Figure 2b ( n = 5 ), the solution !
=
[  dx  dey, x, y ] | , x, y 2 R .!
| Y cancels ˜
E 1 , 3 from Y , and thus !
| Y ??
Z .
Actually, the chain structure with triangularhead ˜ G O C is unidentiﬁable with chain structure ˜ G C w.r.t.
GIN conditions, i.e., for any two sets ofobserved variables Z , Y ✓ X , GIN( Z , Y ) holds on ˜ G O C
if and only if GIN( Z , Y ) holds on ˜ G C .
Consequently, if directly using any recursive algorithm by GIN as in Example 1, the output causalordering would still be ˜ X 1 , ˜ X 2 , ˜ X 3 , ··· , which is incorrect due to ˜ X 3 !
˜
X 2 in the triangular head.
The above examples show that it is not as simple as it seems to use GIN for one-measurement model:only part of the causal ordering can be identiﬁed, and worse yet, rather complicated error correctionis needed to deal with possible incorrect orderings.
However, after a closer look at the unidentiﬁableexamples above, we ﬁnd that actually more information can be uncovered beyond GIN condition:
Example 4 (Asymmetry beyond GIN ) .
1) Consider Example 2 where only the root variable ˜
X
1 canbe identiﬁed by GIN and ˜ X 2 , 3 , 4 are unidentiﬁable , i.e., permutation of the labeling of X 2 , 3 , 4 will stillpreserve the GIN condition over any two subsets
Z , Y .
However, there actually exists an asymmetrybetween X 2 and X 3 , 4 : we could construct linear transformation of X 1 , 3 , 4 : cd  bed
X
+ df + e d X 3
X 4 s.t.
it is independent of X 2 , while there exists no such linear transformation of X 1 , 2 , 4 to beindependent of X 3 , and also for X 1 , 2 , 3 to X 4 . 2) Consider Examples 1 and 3 where ˜ G C
and ˜
G O Care unidentiﬁable w.r.t.
GIN conditions.
Let Z :=
X 4 and Y :
= X 1 , 2 , 3 , GIN( Z , Y ) is violated onboth graphs.
However, an asymmetry actually exists: on ˜ G O C , we could construct aX 1  X 2 + bX 3(which cancels ˜ E 1 , 3 ) to be independent to X 4 , while this is impossible on ˜ G C .
To put simply, the motivation of independent “pseudo-residual” behind GIN actually limits thepower of non-Gaussianity, with the coefﬁcients vector !
only characterized from variance-covariancematrix (2nd-order).
There are actually two cases for GIN( Z , Y ) to be violated: 1) though not allsolution !
makes !
| Y ??
Z , there exists non-zero !
s.t. !
| Y ??
Z , and 2) there naturally exists nonon-zero !
s.t. !
| Y ??
Z .
The original GIN cannot distinguish between these two cases.
Hence inthis paper, we ﬁrst aim to distinguish between the two, generalizing GIN condition to TIN condition.
With Transformed Independent Noise Condition
In the above discussion, one can see that the presence of measurement error affects the conditionalindependence relations among the variables and the independent noise condition.
However, we willshow in Theorem 3 (§4) that a speciﬁc type of independence conditions are shared between theunderlying error-free variables and the measured variables with error.
In this section, we will formulatesuch independence conditions and investigate their graphical implications for error-free variables(i.e.
, variables generated by the LiNGAM without measurement error).
In §4, we will then extendthe results to the measured variables.
Please note that different from other sections, the notation usedin this section, including X , Y , and S , denotes error-free variables generated by the LiNGAM.
3.1 Notations
Let G be a directed acyclic graph with the vertex set V ( G ) =
[ n ] := { 1 , 2 , · · · , n } and edge set 190 E ( G ) .
A directed path P =
( i 0 , i 1 , · · · , i k ) in G is a sequence of vertices of G where there is a 191 directed edge from i j to i j +1 for any 0  j  k  1 .
We use notation i j to show that thereexists a directed path from vertex i to j .
Note that a single vertex is also a directed path, i.e., i iholds.
Let ˜ Z ✓ [ n ] be a subset of vertices.
Deﬁne ancestors Anc( Z ) :
= { j | 9 i 2 Z , j i } .
Notethat Z ✓ Anc( Z ) .
Further let S be a subset of vertices.
We use notation i ⇢⇢ [ S ] j to show that thereexists a directed path from vertex i to j without passing through S , i.e., there exists a directed path
P =
( i, m 0 , · · · , m k , j ) in G s.t.
i, j 62 S and m l 62 S for any 0  l  k .
Deﬁne ancestors outside S 197 accordingly: for two vertices sets Y , S , denote ancestors of Y that have directed paths into Y withoutpassing through S as Anc out( S ) ( Y ) :
= { j | 9 i 2 Y , j ⇢⇢ [ S ] i } .
Note that the graphical deﬁnitions herecan also be translated to trek [35] language set (see Appendix C for details).
Assume random variables X :=
{ X i } i 2 [ n ] are generated by LiNGAM w.r.t.
graph G , i.e.,
X =
AX + E = BE , with B
=
( I  A )  1 .
(2) where E = { E i } i 2 [ n ] are corresponding mutually independent exogenous noises.
A is the adjacency 202 matrix where entry
A j,i is the linear weight of direct causal effect of variable X i on X j .
A j,i 6 =
ifand
only if there exists edge i !
j .
X can also be written directly as a mixture of exogenous noises
X = BE .
If the entry of mixing matrix B j,i 6 = 0 , then i 2 Anc( { j } ) .
Note that here and in whatfollows, we use boldface letters A , B to denote matrices, and use boldface letters S , W , X , Y , Zwith notation abuse: it can denote vertices set, respective random variables set, or random vector.
When we say “two variables sets Z , Y ”, if not otherwise speciﬁed, Z , Y need not be disjoint.
3.2 Independent Linear Transformation Subspace and its Characterization
We ﬁrst give the deﬁnition and characterization of the independent linear transformation subspace .
Deﬁnition 3 (Independent linear transformation subspace) .
Let Z and Y be two subsets of randomvariables.
Suppose the variables follow the linear non-Gaussian acyclic causal model.
Denote:
[Equation]
By the property of independence, ⌦ Z ; Y is closed under scalar multiplication and addition, and thus isa subspace in R | Y | .
In fact, ⌦ Z ; Y can be characterized as a nullspace as follows:
Theorem 1 (Characterization of ⌦ Z ; Y ) .
For two variables subsets Z and Y , ⌦ Z ; Y satisﬁes:
[Equation]
where null( · ) denotes nullspace.
B Y , nzcol( B Z , : ) denotes the submatrix of mixing matrix B , with rowsindexed by Y and columns indexed by nzcol( B Z , : ) .
nzcol( B Z , : ) denotes the column indices wherethe submatrix B Z , : has non-zero entries.
nzcol( B Z , : ) actually corresponds to the exogenous noisesthat constitute Z .
Particularly, if assuming “if i j then B j,i 6 = 0 ”, then, nzcol( B Z , : ) =
Anc( Z ) .
Proof to Theorem 1 is straight-forward by the Darmois–Skitovich theorem [15]: linear transformation!
| Y ??
Z
if and only if !
| Y shares no common non-Gaussian exogenous noise components with Z .
Example 5 (Revisiting examples in §2 from ⌦ Z ; Y perspective) .
For illustration, now we revisit theexamples in §2 from the perspective of independent linear transformation subspace.
a    ab b   abc bc c  abcd bcd cd d   77775 ,   0a + bcb 0c  0cd
d 1cde 0 de e    77775 ,    a   ad + b d  a ( df + e ) + bf + c df + e f    775 (5)
Equation (5) shows the corresponding mixing matrix B for graph ˜ G in Figures 2a to 2c, respectively.
Suppose we have access to underlying variables ˜
X i and only focus on ˜ G .
Coloredblocks denote submatrices of B . 1)
For the fully connected DAG (Figure 2c, the right matrix),to identify the root ˜ X 1 , GIN( ˜ X 1 , ˜ X 2 , 3 , 4 ) is satisﬁed, corresponding to ⌦ Z ; Y = null( | ) .
For( Z , Y ) :
= ( ˜ X 2 , ˜ X 3 , 4 ) or ( ˜ X 4 , ˜ X 1 , 2 , 3 ) , there exists no non-zero !
s.t. !
| Y ??
Z , because the lowerpart and are full row rank.
However, if we set ( Z , Y ) :=
( ˜ X 2 , ˜ X 1 , 3 , 4 ) , we actually have˜ X 2 ??
cd  bed ˜
X
1 + df + e d ˜ X 3  ˜
X 4 , because the stacked two parts of has rank 2 < 3 - though
GIN is still violated because cov( Z , Y ) has rank 1 < 2 . 2)
For the chain structure
˜ G C
(Figure 2a,the left matrix) and chain with triangular head ˜ G O C (Figure 2b, the middle matrix), GIN is satisﬁed on ( ˜ X 1 , ˜ X 2 , 3 , 4 , 5 ) , ˜ X 2 , ˜ X 3 , 4 , 5 ) , ( ˜ X 3 , ˜ X 4 , 5 ) , with the B Y , Anc( Z ) submatrices being , ,respectively.
Note that though the shape of submatrices are different between ˜ G C and ˜ G O C , theirranks are always equal, and is thus unidentiﬁable by GIN .
However, let ( Z , Y ) :
= ( ˜ X 4 , ˜ X 1 , 2 , 3 ) , anasymmetry actually exists: in ˜ G O C , ˜ X
a ˜ X 1  ˜ X
2 + b ˜ X 3 , because in the right matrix hasrank 2 < 3 , while in the left matrix is full row rank, so there is no such independence in ˜ G C .
3.3 Graphical Criteria of Independent Linear Transformation Subspace
§3.2 characterizes ⌦ Z ; Y by submatrix of B , which entails information on graph structure and edgeweights.
Following sections go one step further, explicitly exhibit the graphical criteria of ⌦ Z ; Y , andinvestigate how ⌦ Z ; Y could help to identify the causal structure.
We ﬁrst give the assumption:
Assumption 1 (Rank faithfulness) .
Denote by B ( G ) the parameter space of mixing matrix Bconsistent with the DAG G .
For any two subsets of variables Z , Y ✓ X , we assume that
[Equation]
Roughly speaking, we assume there are no edge parameter couplings to produce coincidental lowrank.
This is slightly stronger than “ nzcol( B Z , : ) =
Anc( Z ) ”.
See Appendix F.1 for elaboration.
Inother words, the graphical criteria holds on a dense open subset of the edge parameter Note that if not otherwise speciﬁed, Assumption 1 is the only one we make besides LiNGAM throughoutthe paper, where violation of Assumption 1 is of Lebesgue measure 0, and LiNGAM is testable.
Graphically, we ﬁrst deﬁne vertex cut , and then give the graphical criteria based on it:
Deﬁnition 4 (Vertex cut) .
Let W , Y , S be three vertices subsets of V ( G ) which need not be disjoint.
We say that S is a vertex cut from W to Y , if and only if there exists no directed paths in G from Wto Y without passing through S .
With basic notations in §3.1, the following statements are equivalent:1) S is a vertex cut from W to Y ; 2)
i 2 W , j 2 Y , i ⇢⇢ [ S ] j does not hold; 3) Anc out( S ) ( Y ) \ W =?
; 4) S ’s removal from G ensures there is no directed paths from W \ S to Y \ S .
More details on vertex cut are in Appendix C. Note that trivially W itself and Y itself are vertex cuts.
Theorem 2 (Graphical criteria of ⌦ Z ; Y ) .
Let Z , Y be two subsets of variables (vertices), we have:
[Equation]
where dim( ⌦ Z ; Y ) denotes the dimension of the subspace ⌦
Z ; Y , i.e., the degree of freedom of ! .
With the characterization in Theorem 1, | Y |  dim( ⌦ Z ; Y ) is exactly the rank of submatrix B Y , Anc( Z ) ,which equals the size of the minimum vertex cut from Anc( Z ) to Y .
See Appendix A for proofs.
From the graphical view, a vertex cut S from Anc( Z ) to Y means that the causal effect from Z andthe common causes of Z and Y , if there is any, must affects Y through S - there is no any bypass.
To interpret in trek-separation [35] language set, it is equivalent to “ ( ? , S ) t-separates ( Z , Y ) ” 1 .
Z
S
Y
N o s u c h b y p
a s s
[Figure] c
[Figure]
265 From the noise view, the noise components that constitute variables
Z areexactly the exogenous noises corresponding to vertices Anc( Z ) .
All thesenoises must contribute to Y (if any) via S , and thus Y can be written as
Y =
L S + E 0 Y , where L denotes a linear transformation, and E 0 Y ??
Z .
Denote by S ⇤ Z ; Y the critical vertex cut from Z to Y 2 , the noise componentsof !
| Y is exactly the exogenous noises corresponding to Anc out( S ⇤ Z ; Y ) ( Y ) .
3.4 Formal Deﬁnition of TIN
With the graphical criteria given in §3.3, we could use it for structure inferenceas long as we have the (dimension of) independent linear transformationsubspace ⌦ Z ; Y .
In §5 we will comprehensively discuss methods to estimate⌦ Z ; Y , while for now, we could just safely suppose we have ⌦ Z ; Y : sinceindependence is testable, theoretically one could get ⌦ Z ; Y at least by traversing over R | Y | .
Deﬁnition 5 ( TIN function) .
Let Z and Y be two subsets of observed random variables.
Supposevariables follow LiNGAM.
We deﬁne a function TIN as follows:
[Equation]
TIN is a function that takes as input two random vectors Z , Y and returns an integer in range [0 , | Y | ] .
Note that this is different to IN or GIN , which returns only a bool value (satisﬁed or not).
GINcan be viewed as a special case of TIN , i.e., GIN( Z , Y ) is satisﬁed if and only if TIN( Z , Y ) =
rank(cov
( Z , Y ))
<
| Y | (where rank(cov( · )) can be characterized by [35]).
IN can also be viewedas a special case of TIN , i.e., IN( Z , Y i ) is satisﬁed if and only if TIN( Z , Z [ { Y i } )
=
| Z | .
Example 6 (Review TIN graphical criteria on graphs) .
For better understanding of TIN , we demonstrate some representative examples of TIN over different graph structures.
Results are shownin Table 1.
We use the four graph structures in Figure 2.
For illustration, assume we have access tolatent variables to directly conduct TIN over ˜ X .
Graphical criteria corresponds to ˜ G .
For readability,we omit all ˜ notation.
We assume there are 5 vertices in fully connected DAG (Figure 2c): considerfor example, GIN( X 3 , X 1 , 2 , 4 , 5 ) does not hold, since rank(cov( Z , Y )) is only one (restricted by
Z size).
However, there exists !
with degree of freedom 1 s.t. !
| Y ??
Z - this corresponds to a3-variables vertex cut X 1 , 2 , 3 ( Anc( Z ) itself).
Consider on Figure 2d, TIN( X 1 , 2 , X 4 , 5 ) = 1 , corresponding to the 1-variable vertex cut X 4 .
This example shows that vertex cut does not necessarilyyield a d-separation pattern (but blocking, instead), since here X 5 6??
X 1 | X 4 .
The graph criteria for TIN over any two sets of variables Z , Y is given as above.
Interestingly, weﬁnd that a special type of TIN is particularly simple in form and informative for structure inference:
[Table]
Lemma 1 (Graph criteria of TIN on one-and-others) .
Assume we have access to all variables
X ona DAG G .
For each singleton variable X i 2 X , let Z :=
{ X i } and Y :=
X \{ X i } , we have,
TIN
[Equation] [Equation]
Due to page limit, we only give above main criterias of TIN .
See Appendix D for more properties.
TIN Condition-Based Method for Measurement Error Models
In §3 we propose TIN condition over general LiNGAM model and give its graphical criteria.
Inthis section, we aim to exploit TIN on our speciﬁc task of interest: measurement error models.
[Figure] [Figure]
4.1 Identiﬁability of Ordered Group Decomposition
[Figure]
Under our problem setting where ˜ X follows LiNGAM, identiﬁability resultscan greatly beneﬁt from the non-Gaussianity of data.
[41] shows that theordered group decomposition of ˜ G is identiﬁable.
First review deﬁnitions:
Deﬁnition 6 (Pure leaf child) .
On a DAG G , a vertex j is said to be a “pureleaf child” of another vertex i , iff j is a leaf node with only one parent, i .
Particularly, if a variable ˜ X j is a pure leaf child of ˜ X i in ˜ G , then ˜ X j and ˜ X jare naturally unidentiﬁable (e.g., ˜ X 2 and ˜ X 7 in Figure 4).
The reason is thatthe exogenous noise ˜
E j of leaf ˜
X j does not contribute to any other variables,just like its measurement noise E j .
Consequently, X i and X j can be viewedas two equally positioned measurements of ˜ X
i , without any asymmetry.
Deﬁnition 7 (Ordered group decomposition) .
Consider the underlying causal model ˜ G .
The orderedgroup decomposition can be deﬁned by the following procedure: at each step, remove root verticesand their pure leaf children nodes (if any) from graph, and append the removed ones as a new group.
Repeat this procedure to remove root vertices from the remaining graph, until the graph is empty.
This procedure is equivalent to Deﬁnition 2 in [41] 3 .
See Figure 4: roots are removed from graph inorder (color dark to light).
The ordered group decomposition is ˜ X 1 !
˜
X 2 , 7 !
˜
X 3 , 6 !
˜
X 4 , 5 .
4.2 Exploit TIN to Identify Ordered Group Decomposition
Our task is to recover ˜ G over ˜ X by testing TIN over X .
In notations above, ˜ X and X are strictlydistinguished to denote unobserved and observed variables.
However, actually this can be escaped:
Theorem 3 (Equivalence of TIN over latent and observed variables) .
For two disjoint observedvariables subsets Z , Y , and their respective underlying latent variables subsets ˜ Z , ˜ Y ,
[Equation]
Theorem 3 can either be proved by graphical criteria, or by showing how rank of submatrices of
B preserves among latent and observed variables.
Interestingly, recall Proposition 1, we show theinequivalence of d-separation and IN condition held between ˜ X and X ( raw independence ).
However,
TIN condition, which essentially ﬁnds transformed independence , holds equivalently on latent andobserved variables.
With this equivalence, we can conduct TIN over observed variables X just as ifwe have access to the latent ones ˜ X .
Thus the problem can be restated without measurement error:assuming causal sufﬁciency, by only using TIN over disjoint Z , Y , to what extent is G identiﬁable?
Under this equivalent problem, Lemma 1 can be used directly to identify the ordered groupdecomposition of ˜ G : for each singleton observed variable X i , test TIN and assign an orderord ( X i ) :
=
TIN( { X i } , X \{ X i } ) .
Group variables with same ord , and then sort the groups by their 335 orders.
Obviously, the ordered groups obtained by this procedure is consistent with Deﬁnition 7.
Example 7 ( TIN on Figure 4) .
ord ( X i ) 7 i =1 are respectively 1 , 2 , 3 , 4 , 4 , 3 , 2 (can verify by characterization or graphically), so the group ordering is identiﬁed as ˜ X 1 !
˜
X 2 , 7 !
˜
X 3 , 6 !
˜
X 4 , 5 .
Estimating Linear Independent Transformation Subspace ⌦ Z ; Y
In the above sections we safely assume that we can always get ⌦ Z ; Y , since theoretically independenceis testable and ! can be exhaustively traversed.
In this section, we give practical methods to estimate⌦ Z ; Y .
Due to page limit we only give a summary for each.
Please see Appendix E for details.
5.1 Tackling Down to Subsets of Y
Theorem 4 ( TIN over Y subsets) .
For two variables sets Z , Y , TIN( Z , Y ) =
k (assume k > 0 ), iffthe following two conditions hold: 1) 8 Y 0 ✓ Y with | Y 0 | = k + 1 (if any), there exists non-zero !
345 s.t. !
| Y 0 ??
Z ; and 2) 9 Y 0 ✓ Y with | Y 0 |
= k , there exists no non-zero !
s.t. !
| Y 0 ??
Z .
This transforms the task of estimating the dimension of ⌦ Z ; Y to a simpler one: counting size of thesubsets Y 0 .
Instead of all independence, here we only need to check existence of independence.
5.2 Constrained Independent Subspace Analysis (ISA)
Conduct Independent Subspace Analysis (ISA [36]) over variables Z and Y in the following form:
[Equation]
(11) 351 where the de-mixing matrix is masked to only update the lower-right | Y | ⇥ | Y | block W YY , with 352 upper-left | Z | ⇥ | Z | block ﬁxed as the identity and elsewhere ﬁxed as zero.
The independence between
W YY Y as a group to Z is maximized.
Since W YY is invertible, its rows span the whole R | Y | , sothe maximum number of rows that achieves W YY | i, : Y ??
Z is exactly the dimension of ⌦ Z ; Y .
5.3 Stacked Cumulants: Ranks Stopped Increasing
Deﬁnition 8 (Stacked 2D slices of cumulants) .
For two variables sets Z , Y and order k  2 , deﬁne:( k ) Z ; Y :=
h C (2) | Z , Y · · · C ( k )
| Z , Y i | , where C ( k ) is matrix with C ( k ) i,j :=
cum( X i , · · · , X i | {z } k  1 times , X j ) .
(12) ( k ) Z ; Y is a ( k  1) | Z | ⇥ | Y | matrix that vertically stacks 2D cumulants slices between Z , Y withorders from 2 to k .
When k = 2 it is cov
( Z , Y ) .
Ranks of ( k ) Z ; Y in the sequence k = 2 , 3 , ...
satisﬁes:360 Theorem 5 ( rank( ( k ) Z ; Y ) stopped increasing) .
For two variables sets Z , Y , there exists a ﬁnite orderk  2 s.t. rank( ( k +1)
Z ; Y )
= rank( ( k ) Z ; Y ) .
Moreover, TIN( Z , Y ) equals to rank( ( k ) Z ; Y ) with this k .
We will show detailed characterization and graphical criteria of ( k ) Z ; Y in Appendix E.1.5.4 TIN in Two Steps: Solve Equations, and then Test for Independence
Motivation of §5.3 is that independence yields zero cumulants (not only 2nd-order uncorrelatedness),i.e.
, ⌦ Z ; Y ✓ null( ( k ) Z ; Y ) , for any k  2 .
Hence, we could solve equations introduced at each orderk
and then check whether all solution !
2 null( ( k ) Z ; Y ) makes !
| Y ??
Z (similar to GIN procedure).
More generally, ⌦ Z ; Y ✓ null(cov( f ( Z ) , Y )) for any real-valued function f ( · ) .
E.g., solve equationssystem { cov(log( Z 2 ) , Y ) !
= 0 ; cov(sin( Z ) , Y ) !
= 0 ; ···} and test whether !
| Y ??
Z holds.
Experimental Results
In this section, we evaluate the performance of our proposed method in recovering the ordered groupdecomposition of ˜ G in the presence of measurement error.
Speciﬁcally, four implementations of
TIN are evaluated: TIN-subsets (§5.1), TIN-ISA (§5.2), TIN-rank (§5.3), and TIN-2steps (§5.4).
We compare our method with PC [31], GES [6], ICA-LiNGAM [26], Direct-LiNGAM [28], and CAMME-OICA
Experiments are conducted on both synthetic and real-world data.
6.1 Synthetic Data
While our proposed method outputs correct ordered group decomposition for any ˜ G (without assumptions on graph structure), in the following simulation we consider speciﬁcally two cases: fullyconnected DAG (Figure 2c) and chain structure (Figure 2a), of which the ordered group decompositionare both ˜ X 1 ) ··· )
˜ X n  2 , ˜ X n  1 ,n .
We consider ˜ G with the number of vertices n = 3 , ··· , 10 .
Edgesweights (i.e., the nonzero entries of matrix A ) are drawn uniformly from [  0 .
9 ,  0 .
F u ll y c onn ec t e d
DAG
C h a i n s t r u c t u r e
Exogenous noises ˜ E are sampled from uniform [ [0 , 1] to the power of c , c
⇠
[ [5 , 7] , and measurement errors are sampled from Gaussian N (0 , 1) to the power of c , c ⇠ [ [2 , 4] .
Sample size is 5 , 000 .
Observations are generated by X i =
˜
X
i + E i .
To show the effect of measurement error, we simulatewith noise-to-signal ratio NSR :
= var( E i ) / var( ˜ X i ) in { 0 . 5 , 1 , 2 , 3 , 4 } .
To evaluate the output groupordering, we use Kendall tau distance [16] to the ground-truth (in range [0 , 1] , the lower the better).
For algorithms returning DAG/PDAG, its ordering is ﬁrst extracted according to Deﬁnition 7.
Figure 5 shows the results.
Each column subplot indicates an NSR setting.
Error bar is from 50random generated instances.
We can see that TIN-subsets and TIN-2steps are two best methods(ranks ﬁrst on 68 / 80 cases), which steadily stick near x-axis.
They both ﬁnd/check independenceby solving equations cov( f ( Z ) , Y ) !
= 0 .
TIN-ISA is slightly inferior (though still ranks 3rd),maybe due to the bias to independence (of ! vectors) by ISA.
TIN-rank ﬂuctuates and performsworst (among TIN s), especially on the fully connected dense case.
This might be explained byhigher order cumulants’ sensitivity to outliers, and unreliable numerical rank tests (we simply use SVD and thresholding).
Among the competitors, ICA-LiNGAM is the strongest one: with NSRgrowing larger, Direct-LiNGAM deteriorates rapidly, while ICA-LiNGAM remains relatively stablew.r.t.
measurement error.
More interestingly, our methods show a deteriorating trend with the graphgrowing larger, which might be because of more complicated equations system, while ICA-LiNGAMon fully connected DAG, shows a trend of getting better.
We will investigate the reason.
Consider
CAMME-OICA, it relies heavily on mixing matrix parameter initialization, and thus performsgenerally weak.
Weak results by PC, GES might be because of the unfair setting for them: theyoutput CPDAG and in both cases (fully connected and chain) here, naturally no ordering (directions)can be determined.
More experimental conﬁguration and results discussions are in Appendix G.2.
6.2 Real-World Data
Sachs’s [22] is a real dataset that measures the expression levels of proteins in human cells undervarious phospholipids.
The ground-truth graph structure [22] contains 17 edges on 11 variables(cell types), of which the ordered group decomposition is {plc,pkc} )
{pip2,pip3,pka,p38,jnk} ) {raf} ){mek} ) {erk,akt}.
Result given by TIN-subsets achieves the best distance score 0 .
33 : {plc,pkc,p38} ){raf,mek,pip2,pka,jnk} ) {pip3} ) {erk,akt}.
This score on PC, GES, ICA-LiNGAM are 0 .
8respectively.
E.g., PC outputs {raf,pka} ) {mek} ) {plc,pip2,pip3,pkc,p38,jnk} ) {erk,akt}.
Conclusion and Discussions
In this work we deﬁne the T ransformed I ndependent N oise ( TIN ) condition based on LiNGAMcausal model, which ﬁnds and checks for independence between a speciﬁc linear transformationof some variables and others.
We provide graphical criteria of TIN , which might further improveidentiﬁability of the latent-variable problem.
Speciﬁcally on causal discovery under measurementerror, we exploit TIN to achieve identiﬁability of ordered group decomposition.
We summarize the future work as three folds: 1) For measurement error model, in addition tothe special type (one-over-others), TIN over general Z , Y pairs can further improve identiﬁability.
See Appendix F.2; 2) TIN now only considers the dimension of ⌦ Z ; Y , while parameters might alsohelp to recover edges weights.
See Appendix F.3; and 3) Reliable estimation of ⌦ Z ; Y can be formulated as an orthogonal research problem.
We believe there exists more solutions.
See Appendix F.4.