Robust Identiﬁability in Linear Structural Equation Models of Causal Inference
Abstract
We consider the problem of robust parameter estimation from observational data in the context of linear structural equation models (LSEMs).
Under various conditions on LSEMs and the model parameters the prior work provides efﬁcient algorithms to recover the parameters.
However, these results are often about generic identiﬁability.
In practice, generic identiﬁability is not sufﬁcient and we need robust identiﬁability: small changes in the observational data should not affect the parameters by a huge amount.
Robust identiﬁability has received far less attention and remains poorly understood.
Sankararaman et al.
(2019) recently provided a set of sufﬁcient conditions on parameters under which robust identiﬁability is feasible.
However, a limitation of their work is that their results only apply to a small sub-class of LSEMs, called “bow-free paths.”
In this work, we show that for any “bow-free model”, in all but 1 poly( n ) -measure of instances robust identiﬁability holds.
Moreover, whenever an instance is robustly identiﬁable, the algorithm proposed in Foygel et al.
, (2012) can be used to recover the parameters in a robust fashion.
In contrast, for generic identiﬁability Foygel et al.
, (2012) proved that with measure 1 , instances are generically identiﬁable.
Thus, we show that robust identiﬁability is a strictly harder problem than generic identiﬁability.
Finally, we validate our results on both simulated and real-world datasets.
INTRODUCTION
Causal inference is a central problem in a variety of ﬁelds in the natural and social sciences.
The goal of causal inference is to design methodologies that infer if a group of events cause a particular phenomenon or not.
A canonical example is the age-old debate on whether smoking causes cancer (Terry [1964]).
The causal inference problem has been extensively studied in statistics, economics, epidemiology, computer science among others ( e.g., Guyon et al.
[2010], Holland et al.
[1985], Pearl [2009], Peters et al.
[2017], Pearl and Mackenzie [2018]) and several schools of thought exist.
One important and popular model is the linear structural equation model ( LSEM ); see, e.g., Bentler and Weeks [1980] and Bollen [1989].
Informally, the experimenter has a model of the world and a dataset (represented as samples from a latent distribution) collected during the experiment.
The goal is to use the samples and the model to infer the strength of dependencies between various quantities of interest.
In LSEM , the experimenter’s model is a
Gaussian linear model which is formally deﬁned as follows.
The model of the causal relationship is given by a mixed graph G =
( V, E, F ) , where the vertex set V of size n corresponds to the set of observable random variables.
Let
X 2 R n ⇥ 1 denote the vector of random variables corresponding to the vertices in V .
The set E of directed edges captures the direction of causality in the model: an edge from vertex u to vertex v implies that X u causes X v .
We will assume that the edges in E form an acyclic directed graph.
The set F of bidirected edges denotes the presence of confounding effects (described shortly).
Let ⌘ 2 R n ⇥denote a vector of noise random variables whose covariance matrix is given by ⌦ 2 R n ⇥ n .
We assume that ⌘ is a zero-mean multivariate Gaussian random variable.
Let
⇤ 2 R n ⇥ n denote the matrix of edge weights on the directed edges; the entries in ⇤ can be interpreted as encoding the strength of causal inﬂuence.
The LSEM model posits that the dependencies between observed variables are linear: the effect on a particular random variable X u is jointly determined by its immediate parents in the directed component of the graph plus a Gaussian noise ( ⌘ u ), which we can represent as
[Equation]
The edge set E puts constraint on the zero pattern of ⇤ : if ( u, v ) / 2 E , then ⇤ ( u,v ) = 0 .
Let us denote the set of such matrices by W ( E ) .
The bidirected edge set F speciﬁes the zero pattern of ⌦ : if u 6 = v and ( u, v ) / 2 F , then ⌦ ( u,v ) = 0 .
Let PD ( F ) denote the set of positive semideﬁnite matrices satisfying this constraint, and let PD be the set of positive semideﬁnite matrices whose dimensions will be clear from the context.
We assume that the dataset is sampled from a distribution that is unknown to the experimenter and has the following properties.
Since the random vector ⌘ is a Gaussian random variable with mean zero, it follows that X is also a Gaussian random variable with mean zero.
Thus, the tuple ( ⇤ , ⌦ ) deﬁnes the distribution of X .
We are interested in this map and its invertibility.
Since X is Gaussian, instead of working with its distribution we can work with its covariance matrix which is a sufﬁcient statistic.
This is what we will do in the sequel.
Let ⌃ denote the covariance matrix of X and let  G : ( ⇤ , ⌦ ) 7!
⌃ be the map of interest.
From the linear relationship in Eq.
(1) we have
[Equation]
Hence, the map  G : W ( E ) ⇥
PD ( F ) !
PD is given by
[Equation]
The (global) identiﬁability question for G , namely are the parameters ( ⇤ , ⌦ ) recoverable from ⌃ for all ⌃ 2 PD , has a positive answer
iff  G is invertible.
The class of mixed graphs G for which  G is invertible has been precisely characterized by Drton et al.
But this turns out to be too strong a restriction and a slightly weaker notion of generic identiﬁability is considered.
A mixed graph
G is said to be generically identiﬁable if for almost all ( ⇤ , ⌦ ) 2 W ( E ) ⇥
PD ( F ) , we can recover these parameters from  G ( ⇤ , ⌦ ) .
Here “almost all” is meant in the measure-theoretic sense for any reasonable measure such as the Lebesgue or Gaussian measure on W ( E ) ⇥ PD ( F ) .
A central question in the study of LSEM s is determining if a mixed graph is generically identiﬁable (GI) and estimating the parameters from the covariance matrix when GI does hold.
While mixed graphs for which GI holds have not been completely characterized, many classes of such graphs have been found, ( e.g., Brito and Pearl [2006], Drton and Weihs [2016], Drton et al.
[2011], Foygel et al.
[2012], McDonald [2002]).
In particular, bow-free graphs (Brito and Pearl [2006]) form one such class and will be studied in this paper.
For this class, we can ﬁrst compute the matrix ⇤ from the covariance matrix ⌃ and then recover ⌦ by computing ( I  ⇤ ) T ⌃ ( I  ⇤ ) .
Since this does not involve matrix inversion, this can be done in a robust manner.
Note that this ⌦ may not satisfy the zero-patterns mandated by the model; however, this can be remedied by solving the convex optimization problem for ﬁnding the closest PSD matrix satisfying the required zero-pattern.
Triangle inequality implies that the optimal solution to the convex optimization problem is a PSD matrix that is also close to the original ⌦ with the same zero-patterns.
Thus, we will be primarily interested in the inverse map
[Equation]
Much of the prior work has focused on designing algorithms with the assumption that the exact joint distribution over the variables is available, which in turn gives exact ⌃ .
However, in practice, the data is noisy and inaccurate and the joint distribution is generated via ﬁnitely many samples from this noisy data.
This leads to the question of (generic) robust identiﬁability (RI): if ⌃ is perturbed slightly, does  1 G ( ⌃ ) change only slightly?
We will formalize this notion in terms of the condition number.
For parameter estimation algorithms to be useful we need robust identiﬁability to hold because of unavoidable inaccuracies in the input in practice.
1 Motivated by this, the key question we consider in this paper is the following.
Are bow-free LSEMs robustly identiﬁable?
Our contributions and discussion.
We answer the question in afﬁrmative, by showing that the space of instances for which the identiﬁability algorithm in Foygel et al.
[2012] is robust is large .
In other words, for a natural measure over the space of parameters ( ⇤ , ⌦ ) for acyclic graphs satisfying bow-free condition, the probability that an instance can be robustly identiﬁed using the algorithm in Foygel et al.
[2012] is at least 1  1 poly( n ) , where n denotes the number of observable variables in the system.
This is in contrast to generic identiﬁability, where the authors in Foygel et al.
[2012] show that the probability is 1 .
To achieve this, we prove a stronger statement, namely, sufﬁcient conditions for robust identiﬁability an arbitrary instance should satisfy when perturbed with adversarial noise (See Fig.
2 where random instances violating it can lead to exponential growth of condition number).
Then we show that when the instances compound are sampled from a natural measure over the set of instances ( i.e., uniform distribution over ⇤ and Wishart distribution over ⌦ ), it satisﬁes the sufﬁcient condition with probability at least 1  1 poly( n ) .
We corroborate our theoretical analysis with simulations on a gene expression dataset used in Drton et al.
[2009] and also on additional simulated datasets.
Our paper has both conceptual and technical novelty compared to Sankararaman et al.
First, Sankararaman et al.
[2019] analyze the error accumulated on every edge; such a strategy fails for anything beyond paths.
Here, we instead analyze the total error accumulated across many edges together.
The key challenge is in ﬁnding the right set of edges to be grouped.
Here we show that we need to analyze the total error in computing the weight parameter of all the incoming edges to a vertex v .
On the technical side, while we use the same high-level idea of induction, we need to work with matrices instead of scalars.
This brings up many new non-trivial challenges requiring matrix-theoretic arguments.
It is occasionally pointed out that the algorithms mentioned above ( e.g., Foygel et al.
[2012]) are designed for the purpose of identiﬁability and not for parameter estimation, and as such should not be used for the latter.
While, a priori, this could be true, for the speciﬁc case of the above algorithms we do not see any reason for not using them for parameter estimation other than the fact that they assume access to the exact covariance matrix.
That the access to the exact covariance matrix is not essential under reasonable conditions on parameters is in fact the main point of our paper.
This shows that the algorithms designed assuming exact access can be used for parameter estimation in realistic situations.
It’s also pertinent to note here that the ﬁeld of robust statistics seeks to deal with similar situations (under various models of perturbations, often adversarial) by designing new algorithms with the explicit goal of robust identiﬁability (see
Diakonikolas and Kane [2019] and references therin for a recent survey).
Our results show that under a reasonable model of perturbation, existing algorithms are already robust.
We are not aware of any work on LSEMs in the robust statistics literature.
A related point is that if one were to not use the above algorithms for parameter estimation then one needs alternative algorithms.
Unfortunately, we are not aware of any algorithms with provable guarantees for parameter estimation other than the ones mentioned above—regardless of the access to the covariance matrix being exact or not.
RICF algorithm
(Drton et al.
[2009]) is designed expressly for parameter estimation using the maximum likelihood principle from ﬁnitely many samples.
Maximum likelihood based algorithms come equipped with conﬁdence intervals which provide an estimate of uncertainty in parameter estimation and could potentially be useful for our problem.
Unfortunately this is not the case: For one, we are not aware of a quantitative analysis using conﬁdence intervals.
Second, we allow adversarial perturbations for which conﬁdence intervals are not applicable.
Third, while practically useful, RICF does not provide any theoretical guarantees on ﬁnding the correct parameters.
It only guarantees that the parameters it ﬁnds achieve a local maximum of the likelihood (there are empirical indications that under some conditions it does ﬁnd the global maximum).
Thus, there is a need for algorithms for parameter estimation with provable guarantees without assuming exact access to the covariance matrix or the distribution.
As already mentioned, in this paper we show that the existing identiﬁability algorithms are in fact such algorithms under reasonable conditions on parameters.
For another discussion of the identiﬁability vs. estimation issue we refer the reader to a recent manuscript (Maclaren and Nicholson [2019]), though they do not provide any positive result like ours.
Related work.
The issue of robust identiﬁability for causal models has started to gain attention only recently.
Schulman and Srivastava [2016], Sankararaman et al.
[2019], Maclaren and Nicholson [2019], Gordon et al.
[2021] are the only papers we are aware of.
Schulman and
Srivastava [2016] showed by means of an example that the recovered parameters can be very sensitive to errors in the data and so robust recovery is not always possible.
They worked in the setting of semi-Markovian models (see, e.g., Shpitser and Pearl [2008]).
Their example is carefully constructed for the purpose of showing that robust recovery is not possible, and it is not clear if such examples are likely to arise in practice.
In other words, their result leaves open the possibility that robust recovery may be possible for a large part of parameter space (according to some reasonable probability measure).
A result in this direction was provided by Sankararaman et al.
[2019] for a subclass of LSEMs.
For bow-free paths they show that if the parameters are chosen from a certain random distribution then the parameters are robustly identiﬁable with high probability.
Our results in the present paper build upon Sankararaman et al.
In particular, our Lemma 1 generalizes Lemma 1 in Sankararaman et al.
Moreover, given this lemma, the proof for the bound on the condition number follows as in prior work.
Finally, Maclaren and Nicholson [2019] provide an abstract framework for studying the robust identiﬁability problems within the context of causal inference.
They also relate it to the extensive literature on similar problems in statistics and inverse problems and provide an entry point to this literature.
Ghoshal and Honorio [2018, 2017] gave an algorithm for parameter estimation and structure learning for linear SEMs from observational data with theoretically good sample and computational complexity and under stochastic noise under certain conditions on the parameters.
However, they make the strong assumption that the noise covariance matrix ⌦ is diagonal (and in the second paper under the stronger assumption that ⌦ is a multiple of the identity) which may be overly restrictive in many settings (Drton et al.
Thus their result is not comparable to ours.
There is also a signiﬁcant body of work on problems such as model misspeciﬁcation.
These are related to but are distinct from the problem studied in the present paper.
We refer to [Sankararaman et al., 2019, Sec. 1.2] for references and commentary on the differences.
A very recent example in the same vein is (Cinelli et al.
Again, while sharing similar general motivation, this work is complementary to ours.
PRELIMINARIES
Notation.
Throughout this paper, we use the notation G =
( V, E, F ) to represent a causal mixed graph structure where
V denotes the set of vertices, E denotes the set of directed (causal) edges and F denotes the set of bidirected (covariance of noise) edges.
For simplicity, we assume that the vertices in the set V are indexed { 1 , 2 , . . .
, | V |} .
Throughout this paper, we assume that the directed edges E induce an acyclic graph.
For a matrix A , we use the notation k
A k := max x 6 =0 k
Ax k 2 k x k 2 to denote the spectral norm of this matrix.
For a vector b , we denote k b k = p b T · b to be the 2 -norm.
We use many standard properties of the spectral norm in the proofs of this paper.
Lemma 7 in the appendix summarizes these for completeness.
We use  1 ( A ) ,  1 ( A ) to denote the largest singular and eigenvalue respectively, of matrix A .
We let ⇤ [ I, J ] , ⌦
[ I, J ] , ⌃ [ I, J ] 2 R | I | ⇥ | J | to denote the sub-matrix of ⇤ , ⌦ , ⌃ respectively, corresponding to vertices in the index set I and J .
For two given vertices u, v 2 V , we use ⇤ u,v , ⌦ u,v , ⌃ u,v to refer to the ( u, v ) -entry of respective matrices.
We use poly( n ) to denote a function which is polynomial in n .
U [ a, b ] denotes the uniform distribution on the interval [ a, b ] with pdf f ( x ) = 1 / ( b  a ) .
We denote layer ( i ) to be the set of vertices such that vlayer ( i ) if and only if the longest directed path ending in v has length i .
Thus, layer (1) denotes the set of vertices with no incoming directed edge.
For any vertex v , we denote pa( v ) to be the set of vertices in V such that there is a directed edge from every vertex in pa( v ) to v .
Additionally, we use the notation spa( v ) :
= pa(pa( v )) .
Since the graph is acyclic, there exists a topological sort order of the vertices
V (Foygel et al.
Throughout this paper, we assume that n is an asymptotic parameter; thus o (1) denotes terms that go to 0 as n goes to inﬁnity.
Deﬁnition 1 ( k -bow-free causal graphs) .
A causal graph
G =
( V, E, F ) is called a k -bow-free causal graph if it has the following properties.
Bow-free.
The graph is bow-free i.e., between any two vertices u and v , there is never both a directed and bidirected edge.
Maximum in-degree or out-degree of k .
For any vertex v 2 V , the total number of directed edges coming into v is at most k .
Likewise, the total number of directed edges leaving v is also k .
Thus, | pa( v ) |  k for every v 2 V .
Figure 1 pictorially denotes an example of k -bow-free causal graph.
Throughout this paper, k should be viewed as a small constant (for instance in our experiments k is either 2 or 7 ).
As in prior work (Sankararaman et al.
[2019], Schulman and
Srivastava [2016]), we use the notion of condition number to measure the robustness of the models.
Before we deﬁne the condition number, we deﬁne the relative distance between two matrices.
Given matrices A , B 2 R n ⇥ m , we deﬁne the relative distance , denoted by Rel( A , B ) as the following:
[Equation]
The ` 1 -condition number is deﬁned as follows.
Deﬁnition 2 (Relative ` 1 -condition number) .
Let ⌃ be a given data covariance matrix and ⇤ be the corresponding parameter matrix.
Let a  -perturbed family of matrices be denoted by F  ( i.e., set of matrices
˜ ⌃  such that
Rel( ⌃ , ˜ ⌃
)   ).
For any ˜ ⌃  2 F  let the corresponding recovered parameter matrix be denoted by ˜ ⇤  .
Then the relative ` 1 -condition number is deﬁned as,
[Equation]
Condition number as the notion of stability is useful since a bound on this quantity translates to an upper-bound on the sample complexity.
More precisely, to get an error of ✏ in the output polynomial in 1 / ✏ , condition number and other parameters of the input number of samples sufﬁce ( e.g., Srivastava et al.
WORK
We give a self-contained background needed from Foygel et al.
[2012] for our paper.
Deﬁnition 3 (Half-trek (Foygel et al.
For any given vertex v 2 V , the set htr ( v ) denotes the set of vertices that can be reached from v via a path of the form,
[Equation]
Deﬁnition 4 (Parameter Recovery Algorithm from Foygel et al.
Consider a vertex v 2 V such that pa ( pa ( v ))
6 =  .
The goal is to compute the vector ⇤ [ pa ( v ) , { v } ] .
Let Y v = { y 1 , y 2 , . . .
, y k } be a given set of vertices corresponding to vertex v .
Let pa ( v ) =
{ p 1 , p 2 , . . .
, p k } denote the set of parents of v .
Let A be a matrix such that A i,j =
[( I  ⇤ ) T · ⌃ ] y i ,p j if y i 2 htr ( v ) and A i,j = [ ⌃ ] y i ,p j otherwise.
Likewise, let b denote a vector such that b i =
[( I  ⇤ ) T · ⌃ ] y
i ,v if y i 2 htv ( v ) and b i =
[ ⌃ ] y i ,v otherwise.
Then we have,
[Equation]
For vertices v 2 V such that pa(pa( v ))
=
we compute ⇤ [ pa ( v ) , { v } ] using the expression,
[Equation]
INVERSE PROBLEM WITH ADVERSARIAL NOISE
In this section, we consider LSEM s with k -bow-free graph and show that under a sufﬁcient condition (formally deﬁned in the assumptions of Model 1), these models can be robustly identiﬁed using the algorithm in Foygel et al.
[2012] in the presence of adversarial noise.
The model we consider is as follows.
Model 1.
We consider the following model of perturbation.
Assume that we are given a data covariance matrix ⌃ .
Let
E 2 R n ⇥ n denote the matrix of perturbations.
Fix a small 0 <  < 1 n 4 .
Thus, the perturbed matrix is ˜ ⌃ :=
⌃ + E .
Additionally, we posit the following property on the perturbation.
For every entry ( i, j ) we have E i,j   p k
⌃ i,j .
WLOG we assume that there exists an entry ( i, j ) such that
E i,j =  p
k
⌃ i,j .
We have the following assumptions for every vertex v 2 V .
(A.1) Input Condition Number.
The condition number of the principal sub-matrix ⌃ [pa( v ) , pa( v )] , deﬁned as  ( ⌃ [pa( v ) , pa( v )] ) :
= k ⌃  1 [pa( v ) , pa( v )] kk ⌃ [pa( v ) , pa( v )]
k   0  12  .
(A.2) Diagonal dominance.
For some 0 < ↵ < 1 , the following hold:
[Equation]

[Equation]
k
[Equation]
k and k
[Equation]
(A.3)
Normalized parameters.
We have k ⇤ [spa( v ) , pa(
v )
] k   < 1 .
Additionally, for every directed edge ( u ! v ) in the causal DAG, we have | ⇤ u,v |  1  > 1 n 2 where ⇤ u,v represents the edge-weight.
Intuition on the assumptions.
Before we state our theorem, we provide some intuition on the assumptions.
An upperbound on the condition number of the input matrix (as in
Assumption (A.1)) is a necessary condition even in the simplest case of robustly solving a system of linear equations.
More speciﬁcally, the relative error in solving a system of linear equations compared to a perturbed instance is upperbounded by the condition number of the constraint matrix (Example 3.4 in Stewart [1998]).
Since LSEM s signiﬁcantly generalize this, it is natural that such a condition should be necessary .
Assumption (A.3) states that k ⇤ k corresponding to all incoming edges for any set of vertices pa( v ) is upperbounded by a constant less than 1 .
Intuitively, it means that the total “information” passed from the vertices appearing earlier in the topological order to those in the later parts does not blow up.
The a priori limiting assumption is Assumption (A.2); this is required for technical reasons to make the analysis go through.
Intuitively, this assumption is a version of the diagonal-dominance in matrices; however, we require a comparison between a principal sub-matrix and a neighboring k -dimensional sub-matrix.
We show in
Section 4 that under an arguably natural generative model for LSEM s, with high probability the generated LSEM satisﬁes Assumption (A.2) suggesting that it is in fact not a strong assumption.
The main result of the paper is the following bound on the ` 1 -condition number of any bow-free LSEM satisfying the assumptions in Model 1.
Theorem 1.
Consider a k -bow-free causal model denoted by the mixed graph G =
( V, E, F ) .
If ↵ 0 < 0 . 99 and
[Equation] [Equation]
then for the model of perturbations described in Model 1 we have that the condition number
[Equation]
O
[Equation]
To prove the main theorem, we ﬁrst show the following lemma which bounds the difference between the true and the recovered parameter.
Lemma 1.
If ↵ 0 < 1 and
[Equation]
k then for every v 2 layer ( j ) and every j  2 we have, k
[Equation]
⌘ ·  , where ⌘ is the following depending on the parameters in Model
[Equation]
Proof outline.
At a high level, our proof strategy is similar in spirit to that of Sankararaman et al.
[2019]; they prove an analogous result for graphs that are paths (for a model similar to Model 1).
However, since we prove such a result for general graphs, our setting faces many additional technical challenges.
Similar to Sankararaman et al.
[2019], we prove the main technical Lemma 1, using induction over the layers.
For any vertex v , we can compute ⇤ [pa( v ) , v ] using equations 5 and 6.
Using the induction hypothesis, we get that ⇤ for the previously considered layers has a sufﬁciently “small” error.
Let A v and b v denote A and b from equation 5 for vertex v when working with the true (unperturbed) ⌃ , and let ˜ A v and ˜ b v denote the corresponding matrices for ˜ ⌃ .
We show that the spectral norm of the matrix ˜
A v
A v and the norm of the vector ˜ b
v  b v is sufﬁciently small.
We use this and bounds on the norms of A v and b v to show that the norm of ˜ A  1 v ˜ b v  A  1 v b v is small.
These steps pose multiple subtle technical challenges in comparison to
Sankararaman et al.
[2019], and require new ideas to handle them.
Proof of Theorem 1.
From Lemma 1 we have that k ⇤ [pa( v ) , v ]  ˜ ⇤ [pa( v ) , v ] k  ⌘ .
From Prop.
(P.6) in
Lemma 7 we have that the absolute value of every entry in the matrix ( ⇤ [pa( v ) , v ]  ˜ ⇤ [pa( v ) , v ] ) is at most k ⇤
[pa( v ) , v ]  ˜ ⇤ [pa( v ) , v ] k  ⌘ .
Combining this with Assumption (A.3) we have Rel( ⇤ , ˜ ⇤ )  ⌘ .
Moreover, from
Model 1 we have that Rel( ⌃ , ˜ ⌃ ) =
p k .
Thus, we get that the condition number is at most  ( ⇤ , ⌃ )  ⌘ p k  .
From Assumption (A.3), we have 1   1 n 2 which implies that  ( ⇤ , ⌃ )  ⌘ p kn 2 .
From the deﬁnition of ⌘ and the premise of Theorem 1 we have that ⌘  O  1 k  .
Thus, we get the stated bound.
RANDOM MODEL PARAMETERS
In this section, we will consider LSEM s that are generated from random model parameters and show that they satisfy the model properties in Model 1.
Thus, we show that on a large set of input parameters the assumptions in Modelhold with high-probability.
Combining this with Theoremimplies that inputs from this parameter space can be robustly identiﬁed using existing algorithms provably .
Model 2 (Generative model) .
Every non-zero entry in ⇤ 2 R n ⇥ n is an i.i.d.
sample from the uniform distribution
U h  1 2 kµ , 1
kµ i \ ⇥  1 n 2 , 1 n⇤ for some ﬁxed µ  10( k + 1) .
The matrix ⌦ 2 R n ⇥ n is generated as follows.
We sample vectors v 1 , v 2 , . .
. , v n 2 R d from a d -dimensional unit sphere such that the following correlation holds.
Each vector v
i is a uniform sample from the sub-space perpendicular to SPAN( { v j }
j 2 V
I  1 ) .
The matrix ⌦ is constructed by letting the ( i, j ) -th entry be h v i , v j i .
Thus, this matrix follows the zero-patterns mandated by the model.
For the Model 2 deﬁned above, we have the following theorem.
Theorem 2.
Let µ  10( k +1) , ↵ = 1 µ + o (1) ,  = 1 µ ,  = n 2 ,  0 = ⇣ 1+ µµ ⌘ 4 + ( µ +1) 2 5 µ 2 ( µ  1) + o (1) .
Then with probability at least 1  1 poly( n ) the following hold simultaneously.
For every v 2 V we have  ( ⌃ [pa( v ) , pa( v )] )   0 .
For every v 2 V , we have that k ⌃ [pa( v ) , v ] k  ↵ k ⌃ [pa( v ) , pa( v )] k , k ⌃ [spa( v ) , pa( v )]
k  ↵ k ⌃ [pa( v ) , pa( v )] k and k ⌃ [spa( v ) , v ] k  ↵ k ⌃ [pa( v ) , pa( v )] k . 3.
For every directed edge ( u ! v ) in the causal DAG, we have that 1 n 2  | ⇤ u,v | .
Moreover, for every v 2 V we have, k ⇤ [spa( v ) , pa( v )] k   .
Proof Outline.
We prove high-probability bounds on the norm of sub-matrices of ⌦ and ⇤ using the concentration properties of the inner-product of the random vectors.
We then use the Taylor series expansion for ( I  ⇤ )  1 to obtain an expression for ⌃ .
Using the various properties of the spectral norm of matrices, and the computed high-probability bounds we obtain the required bounds.
EXPERIMENTS
In this section, we describe the results of our simulation studies.
We ﬁrst perform simulation studies to identify the importance of the various assumptions in Model 1 on random perturbations .
Next we use both real-world and simulated datasets to study the effect of graph sparsity on the condition number.
Thus, using simulations we complement our theoretical understanding of the problem and show evidence of good and bad conditioned instances that go well beyond the sufﬁcient conditions proved.
5.1 SIMULATIONS TO UNDERSTAND THE IMPORTANCE OF EACH OF THE ASSUMPTIONS IN MODEL
To study the effect of various assumptions on the growth of condition number, we perform the following study.
We generate ⇤ and ⌦ randomly using the same generative model in
Section 4.
We generate random perturbations by modifying each non-zero entry of the associated ⌃ using a N (0 , 1 e  2) random variable independently.
We compute the condition number averaged over 20 independent runs of the perturbation.
To violate the various assumptions, we vary the parameter  that is used to determine the range |  i,j | for a directed edge i !
j .
Recall that since  = 1 µ we also affect the values of ↵ and  0 .
We look at the following scenarios and compute the growth of the condition number in each of these cases.
Our biggest take-away is that the assumption that affects the growth of condition number ( i.e., exponential versus polynomial) is Assumption (A.3).
Moreover, the constants proved in theory are worst-case perturbations; for random perturbations, our simulations show that these can potentially be signiﬁcantly improved while maintaining robustness.
All the assumptions in theory are satisﬁed (Figure 3).
All assumptions except Assumption (A.3) are satisﬁed (Figure 4).
Both Assumption (A.3) and Assumption (A.2) are slightly violated (Figure 5).
Assumption ↵ ⇤  ⇤  0 < 1 and Assumption (A.3) are violated (Figure 7).
Assumption
[Equation]
and Assumption (A.3) are violated (Figure 6).
Large edge weights with  > 1 (Figure 8).
As can be seen from the ﬁgures, except in the last scenario (Figure 8), in all other scenarios the condition number does not grow exponentially.
5.2 EFFECT OF GRAPH SPARSITY.
We consider general bow-free graphs and random noise.
Before we describe the experimental procedure, we brieﬂy describe the challenges in running experiments; this explain why experiments in prior works are almost non-existent.
The key issue with experimentation is that the ground-truth model is unknown and the datasets do not come with the true underlying model.
In particular, LSEM is a modelbased approach where designing the right model is part of the hypothesis held by the experimenter.
The dataset only contains the observational data; part of the challenge in inferring causality using LSEM is in devising an appropriate model based on domain knowledge.
Thus, here and in prior works (Drton et al.
[2009], Sankararaman et al.
[2019]) the experimental setup simulates various possible hypotheses in the hypothesis space.
Gene expression dataset.
We use the dataset that corresponds to experiments on gene expression in Arabidopsis thaliana from Wille et al.
We look at the 13 genes which belong to a single pathway.
There are n = 118 microarray experiments.
Thus, the input matrix X 2 R 118 ⇥ 13 .
We have 13 vertices, one corresponding to each of the genes.
First, we choose a random permutation ⇡ to order the vertices.
For any pair of vertices i, j such that ⇡ ( i ) < ⇡ ( j ) we add a directed edge from i to j with probability p .
For every vertex j , we choose a vertex i 6 = pa ( j ) uniformly at random and add a bidirected edge between i and j .
For every other pair of vertices, if there exists no directed edge between them, we add a bidirected edge with probability 0 .
For a given value of p , we generate 30 random graph structures using the above procedure.
To evaluate the condition number, we add independent N (0 , ✏ 2 ) noise to each entry in the matrix X to obtain the perturbed dataset ˜ X ✏ .
We then compute the corresponding covariance matrix ˜ ⌃ ✏ .
We use the algorithm in Foygel et al.
[2012] to recover parameters ⇤ and ˜ ⇤ ✏ corresponding to the matrices ⌃ and ˜ ⌃ ✏ .
For a given realization of the random graph, we generate 20 different datasets ˜ X ✏ .
For each of these 20 datasets, we compute the corresponding covariance matrices and run the parameter recovery algorithm Foygel et al.
[2012] on them.
We then average the condition numbers ( i.e., maximum relative change in ⇤ to the maximum relative change in ⌃ ) across various values realizations of the random graph.
Thus, a single experiment is averaged over the 30 different random graphs multiplied by the 20 different runs for a ﬁxed graph.
We run two kinds of experiments for each p : (1) in which we normalize the dataset ( i.e., every row in the matrix X has a norm of 1 ) (2) in which the dataset is not normalized.
Figure 9 shows the results of our experiments.
We run simulations for p 2 { 0 . 05 , 0 .
As can be seen from the results when the values of p are small (sparse regime), the average condition number tends to be small.
However, as the value of p becomes large (dense models) the condition number increases.
This can be explained by the fact that when errors across many edges accumulate, the total error gets compounded.
Simulated dataset.
We consider two sets of experiments that differ in the number of vertices in any layer: we consider k = 2 and k = 7 .
For each setting of k , we consider p = 0 .(sparse regime) and p = 0 .
8 (dense regime).
When k
=we consider graphs where the total number of vertices is in the set { 20 , 30 , 40 , 50 } while when k = 7 the number of vertices were in the set { 14 , 21 , 35 , 49 } .
For each triple ( k, p, n ) , we generate many random graphs exactly as in the main section of the experiments.
We generate a random ⇤ corresponding to the random graph instance, where every edge is given a weight uniformly at random from [  range , range] .
We use two values of range in the experiments ( range = 1 / 7 and range = 1 ).
For every bidirected edge between ( i, j ) we sample a N (0 , 1) random variable !
and let both ⌦ i,
j = ⌦ j,i = ! .
For every i 2 [ n ] we let ⌦ i,i to be the sum of absolute values in row i added to a  21 -random variable.
The construction implies that ⌦ is a Symmetric Diagonally Dominant matrix and thus, is
Positive Deﬁnite.
We compute the covariance matrix from
Eq.
To compute the condition number, we considersamples from this model and construct the sample covariance matrix ˜ ⌃ which constitutes our perturbed instance.
We then compute the average condition number between the exact computation of ⌃ and the one obtained via ﬁnite samples.
Figure 10, 12, 13 and 11 denotes the results of this experiment.
As can be seen, in the sparse regime the condition number is fairly low, while in the dense regime the condition number is almost a factor of 10 2 .
Thus, these results indicate two things.
First, it veriﬁes the claim in this paper that when the assumptions on range are satisﬁed the instances are well-conditioned.
Second, it also seems to indicate that when range is large, then some of the assumptions in Model 1 are also necessary.
CONCLUSION
In this paper, we consider the problem of robust identiﬁability in bow-free LSEM s. We give a sufﬁcient condition when bow-free LSEM s can be identiﬁed in a robust manner.
As a corollary, this implies that all but a tiny set of instances are robustly identiﬁable.
An important open direction is to provide sufﬁcient conditions for robust identiﬁability in other models of causal inference, particularly the semi-Markovian model (note that Proposition 1.3 in Schulman and Srivastava [2016] is one such sufﬁcient condition).
Another is to combine robust identiﬁability with model misspeciﬁcation ( e.g., Cinelli et al.
[2019]) where all edges in the model are not correctly speciﬁed; Existing works assume access to the exact covariance matrix.  ACKNOWLEDGEMENTS AL was supported in part by SERB
Award
ECR/2017/003296 and a Pratiksha Trust Young Investigator Award.
AL is also grateful to Microsoft Research for supporting this collaboration.