Causal Discovery in Linear Structural Causal Models with Deterministic Relations
Editors: Bernhard Schölkopf, Caroline Uhler and Kun Zhang
Abstract
Linear structural causal models (SCMs)– in which each observed variable is generated by a subset of the other observed variables as well as a subset of the exogenous sources– are pervasive in causal inference and casual discovery.
However, for the task of causal discovery, existing work almost exclusively focus on the submodel where each observed variable is associated with a distinct source with non-zero variance.
This results in the restriction that no observed variable can deterministically depend on other observed variables or latent confounders.
In this paper, we extend the results on structure learning by focusing on a subclass of linear SCMs which do not have this property, i.e., models in which observed variables can be causally affected by any subset of the sources, and are allowed to be a deterministic function of other observed variables or latent confounders.
This allows for a more realistic modeling of inﬂuence or information propagation in systems.
We focus on the task of causal discovery form observational data generated from a member of this subclass.
We derive a set of necessary and sufﬁcient conditions for unique identiﬁability of the causal structure.
To the best of our knowledge, this is the ﬁrst work that gives identiﬁability results for causal discovery under both latent confounding and deterministic relationships.
Further, we propose an algorithm for recovering the underlying causal structure when the aforementioned conditions are satisﬁed.
We validate our theoretical results both on synthetic and real datasets.
1. Introduction
Causal discovery, which refers to the problem of learning causal relationships among the variables of a system, has received extensive attention with applications ranging from social sciences, economics, all the way to biology.
The gold standard of causal discovery is performing interventions (i.e., controlled experiments).
But such experiments could be costly, infeasible, or even at times unethical.
This necessitates developing statistical methods based on purely observational data, where further assumptions on the data generating process are needed. xxxss 2 s [Figure] x L xxxss
[Figure] s [Figure] Linear structural causal models (SCMs) have been extensively considered in the literature perhaps as the most pervasive causal data generating model (Pearl, 2009; Spirtes et al., 2000; Peters et al., 2017).
In this model, the system is comprised of a set of observed (endogenous) variables and a set of source (exogenous) variables.
Each observed variable x is generated as a linear combination of a subset of the other observed variables P a ( x ) (called direct causes of x ) plus a function of a subset of the source variables S ( x ) .
We refer to this model as the linear general SCM (G-SCM).
Literature on linear G-SCM for causal discovery almost exclusively focuses on a subset of G-SCM, in which for each observed variable x , at least one source in S ( x ) is distinct and has non-zero variance (Peters et al., 2017).
Here, a distinct source of observed variable x means a source s ∈ S ( x ) that does not belong to any other subsets S ( x (cid:48) ) , x (cid:48) (cid:54) =
x .[-
-]
Then the non-distinct (i.e., shared) sources can be explained as latent confounders in the system.
We refer to this submodel of G-SCM as distinct source SCM (DS-SCM).
This restriction implies that no observed variable can be a deterministic function of any other observed variables and/or any latent confounders in the system.
However, in a G-SCM, an observed variable is allowed to have no distinct sources (or equivalently can have a distinct source with zero-variance).
Hence, deterministic relations are allowed.
Deterministic relations are problematic in causal discovery as they introduce additional dependencies among variables (Peters et al., 2017).
Speciﬁcally, when deterministic relations exist, d-separation is not complete anymore (although it remains sound).
Such complications are perhaps the reason that deterministic relations are avoided for the most part in the literature.
Furthermore, G-SCMs without distinct sources can have practical signiﬁcance in problems beyond causal modelling.
This subset of G-SCMs can be used to model inﬂuence propagation through networks, where the observed nodes are inﬂuenced directly or indirectly by a set of external factors (sources).
Examples of such factors include sources of news or rumors, or sources of diseases.
Speciﬁcally, G-SCMs without distinct sources can be used to model scenarios with less factors than observed nodes; this situation arises when a few sources inﬂuence a group of observed variables in the system.
For instance, in a setting of spread of news, a large group of individuals may have their news exclusively from a limited set of news sources (Guille et al., 2013; Quinn et al., 2015).
Other examples include spread of epidemics (Nowzari et al., 2016), political campaigns (González-Bailón and Wang, 2016), message passing in human brain networks (Karwowski et al., 2019).
In this paper, we focus on another subclass of linear G-SCMs which relaxes the distinct source assumption of DS-SCM and hence, allows for deterministic relations among the observed variables.
In our proposed model, we consider a set of jointly independent source variables.
Each observed variable is generated as a linear combination of the other observed variables plus a linear combination of the sources.
We relax the distinct source assumption of DS-SCM by assuming instead that each observed variable contains strictly more sources than any of its direct causes.
Figure 1(a) depicts an example which satisﬁes our assumption but violates the assumption of DS-SCM.
As previously mentioned, sharing sources introduces latent confounding among the variables.
Hence our introduced model allows for both latent confounders as well as deterministic relations.
We refer to this model as the linear propagation SCM (P-SCM).
Majority of works in the literature on linear causal models in fact consider an intersection of P-SCM and DS-SCM, which we refer to as DS-P-SCM.
Therefore, our work strictly expands the considered model space compared to those works.
See Figure 1(c) for a schematic representation of the relations among the linear models.
We study the problem of causal discovery when the data generating model is a linear P-SCM, that is, to learn the directions and strengths of the causal inﬂuences among the observed variables, as well as the direct connections from the sources to the observed variables.
We derive a set of necessary and sufﬁcient conditions for unique identiﬁability of the causal structure from observational data under the assumptions of faithfulness and separability of the sources.
To the best of our knowledge, this is the ﬁrst work that gives identiﬁability results for causal discovery under both latent confounding and deterministic relationships.
We further propose an algorithm which uniquely identiﬁes the causal structure when the conditions are satisﬁed.
Also, we derive an equivalent condition when the generating model is a linear DS-P-SCM.
We examine our algorithm on synthetic and real datasets, and compare with existing causal discovery methods developed for linear DS-SCMs with or without latent confounders.
The results show that our proposed algorithm outperforms those methods in recovering the underlying structure.
Related work.
Under causal sufﬁciency, Shimizu et al.
(2006) showed that a linear non-Gaussian additive model (LiNGAM), in which source variables have non-Gaussian distributions, allows for unique identiﬁability of the causal structure from observations.
Causal sufﬁciency is a common assumption in causal discovery which means there exist no latent confounders for any pair of observed variables.
Yet, without considering the effect of latent confounders, one may infer wrong causal relations among observed variables.
Hoyer et al.
(2008) and Salehkaleybar et al.
(2020) considered an extension of LiNGAM in the presence of latent variables (lvLiNGAM).
Our model generalizes lvLiNGAM to the case when observed variables are allowed to deterministically depend on other observed variables or latent confounders.
We further comment on the connection of our model to lvLiNGAM in Appendix C.4.
SCMs with deterministic relations have been considered in a few work.
In (Geiger et al., 1990;
Spirtes et al., 2000), D-separation condition (with capital letter D) is proposed for graphically determining conditional independencies when deterministic relations are allowed.
Yet, it remained unclear whether D-separation condition can capture all conditional independencies induced from the distribution.
Daniušis et al.
(2010) and Janzing et al. (2012) considered a system with only two variables, one variable deterministically causing the other, and showed that the correct causal direction can be learnt if the deterministic function contains no information about the cause variable and vice versa.
Their analysis does not hold for linear relations though.
Scheines et al.
(1996) considered recovering a reduced model in which all deterministic variables are removed.
Luo (2006); Mabrouk et al.
(2014); Lemeire et al.
(2012) adapted conventional causal discovery methods such as PC and greedy-search algorithms (Pearl, 2009; Tsamardinos et al., 2006; Chickering, 2002), where deterministic relations are detected either by additional independence tests, or by calculating conditional entropy among variables.
The aforementioned methods suffer from identiﬁability problem.
In particular, majority of these works do not discuss the capability of identifying the underlying structure by their proposed algorithms (e.g., the equivalence classes of the recovery result).
Besides, these methods do not consider the presence of latent confounders in the system.
Please refer to Appendix A for a detailed background on causal discovery methods.
2. System model
Notation.
We use upper-case letters for vectors and bold upper-case letters for matrices.
We use [ n ] (cid:44) { 1 , 2 , · · · , n } .
If X = [ x 1 , · · · , x p ] , X
[ i : j ] is the sub-vector
[ x i , x i +1 , · · · , x j ] ; 1 ≤ i < j ≤ p .
We use x i (cid:32) x
j to refer to a directed path from node x i to node x j (in the corresponding graph).
2.1. Generating model
As we mentioned in the introduction, linear G-SCM is one of the main data generating models assumed in the literature.
A G-SCM (Pearl, 2009, Deﬁnition 7.1.1) is a 4-tuple (cid:104)S , X , F , P ( S ) (cid:105) . S is a set of source variables, X is a set of observed variables, F = { f x } x ∈X is a set of functions such that for each x ∈ X , x = f x ( P a ( x ) , S ( x )) , P a ( x ) ⊆
X \ { x } and S ( x ) ⊆ S , and P ( S ) is a joint probability distribution over S .
In the case of linear functions, the model is restricted as follows:
Deﬁnition 1 (Linear G-SCM)
A linear G-SCM is the subclass of G-SCM in which each observed variable x is generated as x = f x ( P a ( x ))
+ g x ( S ( x )) , where f x is a linear function and g x is a function indicating the effects from the sources.
We refer to g x ( S ( x )) as the source mixture corresponding to observed variable x .
Previous works on causal discovery mostly focus on a subclass of linear G-SCM which satisﬁes the distinct source assumption deﬁned below.
Assumption 1 (Distinct source assumption)
A linear G-SCM satisﬁes the distinct source assumption if each S ( x ) contains at least one distinct source, that is, a source with non-zero variance that does not appear in any other S ( x (cid:48) ) for x (cid:48) (cid:54) = x .
We refer to the subclass of linear G-SCMs satisfying Assumption 1 as linear DS-SCM.
The nondistinct (i.e., shared) sources represent latent confounders in the system.
Assumption 1 can be interpreted as: The source mixture of one variable cannot be a deterministic function of the components in the source mixtures of the rest of the variables.
1 Equivalently, no observed variable can be a deterministic function of any other observed variables and/or any latent confounders in the system.
In this paper, we introduce another subclass of linear G-SCMs, which we refer to as the linear propagation SCM (P-SCM).
Consider a set of observed variables
X = { x 1 , · · · , x p } arranged in a causal order (no latter variable causes any earlier variable), and another set of jointly independent source variables S
= { s 1 , · · · , s m } .
In the generating model of a linear P-SCM, each observed variable consists of a linear combination of earlier observed variables plus another linear combination of the sources.
Speciﬁcally,
[Equation]
(1) a ij is the strength of the direct causal effect from x j to x i .
˜
s i ,
i ∈
[ p ] represents the mixture of sources exogenously causing x i .
b ij is the strength of the direct exogenous effect from s j to x i .
We refer to { ˜ s 1 , · · · , ˜ s p } as source mixtures or simply mixtures .
Note that since (1) allows observed variables to share sources, our model allows for latent confounding.
However, the latent confounding can only be linear, i.e., it is due to the source mixtures being linear combinations of elements in S .
In our model we require the following assumption on the generating model which is weaker than
Assumption 1 of DS-SCM.
Assumption 2 If x i is a direct cause of x j , then there is at least one source in x j that is not in x i .
Unlike the distinct source assumption in DS-SCMs, Assumption 2 does not necessarily require each observed variable to be associated with a distinct source (see Figure 1(a) for an example).
Therefore, our model allows for deterministic relations.
3 We provide a detailed comparison of our model with
DS-SCM in Section 4.
We deﬁne the causal diagram of the model as a directed graph in which the vertex set consists of the observed variables X and the source variables S .
There is a directed edge from observed variable x i to x j if and only if a ji (cid:54) =
(we refer to such an edge as a causal connection), and a directed edge from source s i to observed variable x j if and only if b ji (cid:54) =
(we refer to such an edge as an exogenous connection).
Due to the assumed causal order, the causal diagram is acyclic, and hence the model is a directed acyclic graph (DAG).
The model in (1) can be written in the matrix form as
[Equation]
where X (cid:44) [ x 1 , · · · , x p ] (cid:62) , ˜ S (cid:44) [˜ s 1 , · · · , ˜ s p ] (cid:62) , S (cid:44) [ s 1 , · · · , s m ] (cid:62) .
A is a p × p strictly lower triangular matrix with the coefﬁcient a ij on the ( i, j ) -th entry.
B is a p × m matrix with the coefﬁcient b ij on the ( i, j ) -th entry.
The strictly lower triangular matrix A attests the existence of a causal order among the observed variables (the generating process is recursive (Bollen, 1989)).
Mixtures ˜ s i , i ∈ [ p ] represent additive exogenous noises .
2.2. Identiﬁability assumptions
In (2), each x i can be written as a linear combination of the sources.
Let W be a mixing matrix s.t.
[Equation]
where
[Equation]
Deﬁne the component set of a variable x i = (cid:80)
j w ij s j (or a mixture ˜ s i
= (cid:80)
j b ij s j ) as Comp( x i )
= {
s j ∈ S | w ij (cid:54) =
( or b ij (cid:54) = 0) } .
We require the following assumptions for identiﬁability.
Assumption 3 (P-SCM faithfulness)
(a) If x i is an ancestor of x j , then all source components in x i must also appear in x j .
(b) Let B p be an arbitrarily permuted version of matrix B (either row or column permutations).
Any submatrix of B p (with non-zero rows or columns) is of full rank.
Assumption 4 (Separability)
We assume the linear P-SCM is separable– its corresponding mixing matrix W , cf.
(3) , can be correctly recovered using observations X , up to scaling and permutation of its columns.
P-SCM faithfulness assumption is extended from the common faithfulness assumption in studying causal models (Spirtes et al., 2000; Pearl, 2009).
This assumption is satisﬁed almost surely if all model coefﬁcients { a ij } and { b ij } are drawn randomly and independently from continuous distributions (Meek, 1995).
Separability, on the other hand, can be achieved using blind source separation (BSS) methods (Comon and Jutten, 2010), given certain assumption on the sources.
For example, when the a deterministic function of x i and shared sources.
This example is however a degenerate case.
In particular, given a deterministic relation between x i and x j , it is not possible to recover their causal direction based on observations.
sources are non-Gaussian random variables, the mixing matrix W can be recovered using Independent
Component Analysis (ICA) or overcomplete ICA methods (Comon, 1994;
Hyvarinen et al., 2002;
Lewicki and Sejnowski, 2000).
Other BSS methods include Statistical Blind Source Separation
Regression (SBSSR) model and Non-negative Matrix Factorization (NMF).
See Appendices B and C for detailed explanation about both assumptions, and how BSS methods are applied to our model.
Note that the result of BSS may suffer from permutation and scalability indeterminacies, due to lack of prior information about the sources.
In Appendix C.4 we show that these indeterminacies do not affect the recovery performance of our algorithm.
Speciﬁcally, we show that, even in the presence of these indeterminacies, our recovered model is identical to the true model.
3. Conditions for unique identiﬁability
In this section, we present the necessary and sufﬁcient conditions for unique identiﬁability of a linear
P-SCM, which consists of two parts, the unique components condition and marriage condition.
Both conditions are imposed on the “possible parent set” of each observed variable which is a superset of its ancestors that can be recovered from the mixing matrix (from observational data).
We ﬁrst deﬁne the possible parent set, and then present the combinatorial version of the conditions.
Equivalent algebraic (matrix) representations of the conditions are provided in Appendix E.1.
The necessary and sufﬁcient condition we derive for unique identiﬁability of a linear P-SCM, as well as our proposed algorithm, require a search over the possible parent set of an observed variable x k which is deﬁned as follows.
Deﬁnition
The possible parent set of an observed variable x k is deﬁned as the set of observed variables in X whose component set is a strict subset of the component set of x k .
That is, P k
= { x i ∈ X | Comp( x i ) (cid:40)
Comp( x k ) } .
We refer to the elements of this set as the “possible parents” of x k since each element is a candidate to be a parent of x k , according to P-SCM faithfulness assumption.
The possible parent set of x k (i) can be deduced from the mixing matrix and (ii) can be strictly smaller than the set of observed variables preceding x k in the causal order; see Appendix D.3 for an example.
In linear DS-SCMs, since each observed variable has a distinct source, P k only includes the ancestors of x k in the generating model (see Section 4.1 for more explanation).
This property allows to identify all the observable ancestors of each variable x k from the mixing matrix.
Subsequently, an algorithm, such as lvLiNGAM in (Salehkaleybar et al., 2020), can recover the directed causal paths among variables.
By contrast, in our linear P-SCM, the possible parent set P k can include observed variables which are not ancestors of x k .
The reason behind this is that there may exist an observed variable which does not have a distinct source, hence this variable can mimic an ancestor of x k by its exogenous connections to the sources.
From (3), the observed variables are given by X =(
I − A ) − 1 B S =(
I − A ) − 1 ˜ B S (cid:48) , where ˜ B is a version of B with the columns arbitrarily permuted and rescaled such that ˜ B S (cid:48)
=
B S .
Note that the ( i, j ) -th entry of ( I − A ) − 1 represents the total causal effect from observed variable x j to observed variable x
i (Spirtes et al., 2000).
Our task is to uniquely recover ( I − A ) − 1 , i.e., the total causal effects among observed variables, and to recover B up to permutation and scaling of its columns, i.e., the exogenous connections from the sources to the observed variables.
Note that the permutation indeterminacy of B is immaterial since there exists no ordering between the exogenous connections, and we are not concerned about the scalability indeterminacy in recovering their effect.
Next, we introduce the necessary and sufﬁcient conditions for recovering ( I − A ) − 1 and ˜ B .
3.1. Unique components condition
When recovering the total causal effect from possible parents of x k (i.e., variables in P k ) to x k , the question is whether each existing component in x k (i.e., s ∈ (cid:83) x i ∈P k Comp( x i ) ) comes from a possible parent ( x i ∈ P k ), or comes from the exogenous connections to x k (i.e., ˜ s k = (cid:80) ml
=1 b kl s l ), or maybe both.
To address this question, we propose the concept of unique components .
For a possible parent x i ∈ P k , if ˜ s i has a component that is not shared by the source mixture of any other possible parent in P k
(i.e., any ˜ s j ; j (cid:54)
=
i , x j ∈ P k ), then we can use this unique component to compute the total causal effect from x i to x k (by dividing the coefﬁcient of this unique component in x k by its coefﬁcient in x i , see Remark 9 in the sufﬁciency proof for more details).
We can recursively use the unique components idea to ﬁnd the total causal effects from all possible parents in P k to x k : After using the unique components to recover total causal effects from the corresponding possible parents to x k , we remove those variables and search for unique components with respect to the remainder subset of P k .
Thus, we extend the concept of unique components to subsets of P k by considering only the possible parents in these subsets.
We deﬁne the unique components over the mixtures ˜ S
k
=
˜ s i : x i ∈ P k } in an iterative manner as follows.
Deﬁnition
Fix an observed variable x k .
For each possible parent of x k , i.e., x i ∈ P k , we deﬁne the unique component set of x i , denoted by U k ( i ) , as follows. (i) Let J (0)
=
{ i : x i ∈ P k } , i.e., the index set of the possible parent set of x k . (ii) For n = 1 , 2 , · · · ( n is the iteration index): - For i ∈ J ( n − 1) , deﬁne ˜ U ( n ) k
( i ) =
Comp (˜ s i ) (cid:15) (cid:83) j ∈ J ( n − 1) ,j (cid:54) = i Comp (˜ s j ) .
- Let J ( n ) = { i ∈ J ( n − 1)
| ˜
U ( n ) k
( i ) = ∅} . - Repeat until | J ( N +1) | = | J ( N ) | for some N ∈ N .
Deﬁne I k = { x l : l ∈ J ( N ) } . For each x i ∈ P k , U k ( i ) =
˜ U ( n ) k ( i ) , i ∈ J ( n − 1) \ J ( n ) , n = 1 , · · · , N ; U k ( i ) = ∅ , i ∈ J ( N ) . In Deﬁnition 3, when n = 1 , ˜ U (1) k ( i ) represents the components in ˜ s i but not in any other ˜ s j , where x j ∈ P k (or equivalently j ∈ J (0) ).
J (1) represents the index set of P (1) k
which is the subset of P k composed of observed variables with no unique components in the ﬁrst iteration.
In the next iteration, ˜ U (2) k ( i ) is deﬁned similarly but only for i ∈ J (1) .
J (2) is the index set of the subset of P (1) k with no unique components.
This procedure is repeated until no more unique components are found;
I k ⊆
P k is the remaining subset of possible parents with no unique components after the procedure terminates.
Appendix D.4 provides an example to demonstrate this iterative deﬁnition.
We use Deﬁnition 3 as follows.
We (i) identify the possible parents in P k with unique components, (ii) compute their total causal effects to x k using these unique components, and (iii) subtract the learnt causal effects from x k .
Next, we consider only the possible parents with no unique components in the ﬁrst iteration ( x i ∈ P (1) k ), and ﬁnd which of these have unique components among variables in
P (1) k .
We then use these unique components to compute total causal effects from the corresponding variables in P (1) k to x k .
Based on Deﬁnition 3, we state the unique components condition as follows.
Condition 1 (Unique components condition)
(i) For each possible parent x i of x k (i.e., x i ∈ P k ) with a non-empty unique component set
(i.e., | U k
( i ) | (cid:54) = 0 ), the unique components of x i are not exogenously connected to x k .
That is, Comp(˜ s k ) ∩ U k
( i ) = ∅ ,
∀ x i ∈ P k s.t.
U k
( i ) (cid:54) =
∅ .
(ii) Further, for any x i ∈ P k with an empty unique component set ( | U k
( i ) |
= 0 ), the exogenous connections to x i and x k are disjoint.
That is, Comp(˜ s k ) ∩ Comp(˜ s i )
= ∅ ,
∀ x i ∈ P k s.t.
U k
( i ) = ∅ .
increasing order permuted Condition 1 restricts certain exogenous connections from sources to observed variables in the groundtruth structure. The ﬁrst part of Condition 1 states that if a source s is a unique component of any possible parent of x k , then s should not have an exogenous connection to x k .
The second part states that any possible parent of x k with no unique components should not overlap in any exogenous connections with x k . {+
+}
In general, x k can only be exogenously connected to (i) source components that are shared only among its possible parents that have unique component(s), (ii) new source components that are not connected to any of its possible parents.
Please refer to Example 7 in Appendix D.5 to see why this condition is necessary for unique identiﬁability.
3.2. The marriage condition
The second part of the necessary and sufﬁcient conditions for unique identiﬁability of a linear P-SCM is identical to the marriage condition of Hall’s marriage theorem (Cameron, 1994).
The combinatorial formulation of this theorem considers a collection of ﬁnite sets and provides a necessary and sufﬁcient condition (the marriage condition) for the feasibility of selecting a distinct element from each set.
Lemma 4 (Hall’s marriage theorem and marriage condition)
Let X be a collection of ﬁnite subsets of a set S .
The members of X can be counted with multiplicity (two or more members are identical).
Deﬁne the mapping f ( x ) :
X (cid:55)→ S s.t.
f selects a distinct element from each set x ∈ X .
The collection X satisﬁes the marriage condition if for every sub-collection C ⊆ X , | C | ≤ | (cid:83) x ∈ C x | , i.e., every C ⊆ X covers at least | C | different elements of S .
Hall’s Marriage Theorem states that, the mapping f exists if and only if X satisﬁes the marriage condition.
In our framework, the collection X corresponds to the family of component sets of possible parents of a variable x k .
Speciﬁcally, members of the collection X are the ﬁnite subsets Comp( x i ) ⊆
S , x i ∈ P k , S is the set of sources.
The marriage condition translates to the following:
Condition 2 (The marriage condition)
For each observed variable x k with possible parent set
P k , every subset of the possible parent set, X C ⊆
P k , satisﬁes | X C | ≤ (cid:12)(cid:12) (cid:83) x
i ∈
X C Comp( x i ) (cid:12)(cid:12) .
That is, every subset X C ⊆
P k includes at least | X C | different source variables.
Condition 2 implies that the possible parent set of every observed variable must have a sufﬁcient number of exogenous connections (from source variables) so that it allows for unique identiﬁability of the causal effects to this observed variable.
Since each variable x k can be written as a linear combination of the source mixtures in its possible parent set ( ˜ S k
= { ˜ s i : x i ∈
P k } )
plus ˜ s k , Condition 2 can be equivalently imposed on the collection ˜ S
k :
For ˜ S C ⊆ ˜ S k , | ˜ S C | ≤ (cid:12)(cid:12)(cid:12)(cid:83) ˜ s
i ∈ ˜ S C Comp(˜ s i ) (cid:12)(cid:12)(cid:12) .
See Appendix F.3 for the proof.
Recall that Condition 1 is also imposed on
˜ S k (the unique components are deﬁned over the source mixtures in P k ).
Please refer to Example 8 in Appendix D.5 to see the necessity of the marriage condition for unique identiﬁability of a linear P-SCM.
3.3. Main result
Our main result states the necessary and sufﬁcient conditions for unique identiﬁability of the total causal effects among the observed variables and the exogenous connections from the sources.
Theorem 5 Let A = { a ij , b ik : i, j ∈ { 1 , · · · , p } , k ∈ { 1 , · · · , m }} .
A fully parameterizes a linear P-SCM.
Let π denote the Lebesgue measure over A .
The causal effects and exogenous connections of a linear P-SCM are uniquely identiﬁable from observations almost surely (w.r.t.
the measure π ) if and only if, for any observed variable x k , both Conditions 1 and 2 are satisﬁed.
The proof of Theorem 5 appears in Appendix E. Necessity of the conditions is proved using similar reasoning as in Examples 7 and 8: If either condition is not satisﬁed, it is impossible to distinguish the true generating model from certain different generating model(s).
For sufﬁciency of the conditions, we propose the P-SCM Recovery algorithm
(Algorithm 1) and prove its correctness, i.e., the algorithm recovers the true causal structure when the conditions are satisﬁed.
For each observed variable x k with possible parent set P k , Algorithm 1 computes the total causal effects from x i ∈ P k to x k using the unique components in U k
( i ) in the same iterative manner as described in
Deﬁnition 3, until no more unique components can be found in the last subset I k .
The algorithm computes the total causal effect from all x i ∈
I k to x k by solving an overdetermined linear system.
After computing total causal effects from all possible parents to x k , the remaining parts in x k are considered as exogenous connections.
4. Comparison with linear DS-SCM
There are two main differences between our proposed linear P-SCM and linear DS-SCM.
The ﬁrst difference regards the type of latent confounders they allow.
From (1), in linear P-SCM, the source mixture associated with each observed variable is comprised of a linear combination of jointly independent sources, which can lead to linear latent confounding.
That is, the dependency between two source mixtures is due to the fact that they are both linear combinations of the same set of sources.
This is not necessarily the case for linear DS-SCM, i.e., the source mixtures can be nonlinear combinations of the sources.
The second difference between P-SCM and DS-SCM regards the restriction of distinct sources.
Linear P-SCM relaxes the distinct source assumption to the requirement in Assumption 2 which assumes that each observed variable contains strictly more sources than any of its direct causes.
This means that unlike DS-SCM, in our model, an observed variable can be a deterministic function of its direct causes and the latent confounders.
Note that all the works in the literature on linear DS-SCM that we are aware of– with the exception of (Wang and Drton, 2020)– in fact consider an intersection of P-SCM and DS-SCM, in which all three assumptions of linear latent confounding, jointly independent sources (not necessarily independent source mixtures), and Assumption 1 hold (Hoyer et al., 2008; Entner and Hoyer, 2010;
Tashiro et al., 2014; Salehkaleybar et al., 2020).
In this case, the subclass of P-SCMs representing this subclass are those with matrix B in (2) of the form B =
[ I B s ] .
We refer to this subclass of linear G-SCM as linear DS-P-SCM.
Therefore, our work strictly expands the considered model space compared to those works.
To demonstrate the practical signiﬁcance of linear P-SCM over linear
DS-P-SCM in causal discovery, we show that there are models that can be correctly explained by a linear P-SCM, while linear DS-P-SCMs either are inapplicable or give misleading results (hence the model cannot be recovered using causal discovery methods for linear DS-P-SCMs).
These examples appear in Appendix D.2.
4.1. Reduction of the conditions for linear DS-P-SCM
To show how our conditions for unique identiﬁability of linear P-SCMs can be reduced for linear
DS-P-SCMs, consider the setting in which each observed variable is associated with a distinct source.
Under this setting, the model can be equivalently written as a linear DS-SCM with linear latent confounding and jointly independent sources (see Appendix F.1 for more details).
This setting reduces Conditions 1 and 2 in Theorem 5 as follows.
First, all possible parents of an observed variable x k have unique components, i.e., U k
( i ) (cid:54) =
∅ for all x i ∈ P k .
This further implies that the marriage condition is automatically satisﬁed.
Second, for each x k , the possible parent set P k is equivalent to the ancestor set of x k .
This follows because, for each x i ∈ P k with a distinct source s j , since s j is not connected to any other observed variables including x k , the s j component in x k must result from x i .
Hence, there must be a causal connection/path from x i to x k .
Theorem 6 below provides the equivalent necessary and sufﬁcient conditions for this setting.
See Appendix F.4 for the proof, as well as a graphical representation of the condition.
Theorem 6 Suppose every observed variable in the linear P-SCM is associated with a distinct source.
The causal effects and exogenous connections of a linear P-SCM are uniquely identiﬁable from observations almost surely (in the sense described in Theorem 5) if and only if the following condition holds:
For a source s j that is exogenously connected to an observed variable x k , either there are no causal paths from any observed variable containing s j to x k , or there are at least two distinct causal paths from the observed variable(s) with s j in their source mixtures to x k .
Remark 7 The condition in Theorem 6 for a linear P-SCM with distinct source implies the condition for unique identiﬁability of a linear DS-P-SCM in (Salehkaleybar et al., 2020, Theorem 16), but the reverse is not true.
Thus, a linear DS-P-SCM can be uniquely identiﬁable in (Salehkaleybar et al., 2020) but not uniquely identiﬁable (as a linear P-SCM) in our setting.
The reason is that the search space in Theorem 6, which is the whole class of linear P-SCMs, is strictly larger than the search space in (Salehkaleybar et al., 2020), which is linear DS-P-SCMs.
See Appendix F.5 for the proof.
4.2. Choice of the representation of linear P-SCM
Note that equivalently we can say P-SCM also requires distinct sources but those sources can have zero variance (i.e., zero).
In this case, as shown in Appendix F.2, instead of the form in (2), our model can be equivalently written as [Figure]
[Equation]
where X l is the vector of latent confounders; S d is the vector of (possibly zero) jointly independent distinct sources; A ol represents the latent confounding.
This representation is perhaps more familiar to the reader (as it is considered in most works on linear DS-P-SCM), yet we claim that the representation in (2) is more suitable for the task of identiﬁcation when zero variance sources exist in the model: First, in order to fully learn the structure of the model in (4), the number of latent confounders must be known in advance (Hoyer et al., 2008; Salehkaleybar et al., 2020), which cannot be deduced from the mixing matrix W when deterministic relations are present.
In this case, we can only learn the sum of the number of latent confounders (i.e., cardinality of X l ) and non-zero distinct sources (i.e., support of S d ).
On the contrary, by considering both latent confounders and non-zero distinct sources as S in (2), we do not need to know the number of latent confounders or the number of non-zero distinct sources in the recovery.
Second, our conditions for identiﬁability of linear
P-SCM depend on the exogenous connection matrix B , which includes both latent confounding and non-deterministic relationships in (4) (i.e., matrix A ol and the non-zero entries in S d ).
As a result, the identiﬁability results derived in this work would be more complicated if translated on the model in (4), and we are not aware of a straightforward way to do this translation.
5. Numerical experiments
We ﬁrst provide estimates for the likelihood of satisﬁability of the conditions in Theorem 5.
Specifically, we randomly generate linear P-SCMs and linear DS-P-SCMs with ﬁxed expected degrees among observed variables and sources, and investigate what portion of them satisfy the conditions.
We refer to the generating model as a linear DS-P-SCM when each observed variable in the generated
P-SCM has a distinct source.
Next, we evaluate the performance of Algorithm 1 for P-SCM recovery on synthetic models under different settings.
Finally, for real data, we consider the closing prices of ﬁve world stock indices, and model the corresponding returns as a linear P-SCM.
Satisﬁability of the conditions.
We test the satisﬁability of our derived conditions on linear
P-SCM and linear DS-P-SCM with different pairs of ( d e , d o ) , where d e ( d o ) is the expected number of causal (exogenous) connections for each observed (source) variable.
Figure 2 (left) shows the average number of generated models (attempts) required to obtain a model that satisﬁes our conditions, plus/minus half the standard deviations w.r.t.
the source to observed node ratio (the number of sources divided by the number of observed variables).
The result shows that our conditions are likely to be satisﬁed on (1) linear P-SCMs when the number of sources is large and the average degrees are small, and (2) linear DS-P-SCMs when both the number of sources and average degrees are small.
We believe that satisﬁability of our conditions depends on (1) the existence and similarity of exogenous connections among observed variables, (2) the ratio of overlap among exogenous connections (how many sources are shared among observed variables) and (3) the structural complexity of the causal connections among observed variables.
See Appendix G.1 for more details.
Performance of recovery.
We evaluate the performance of our recovery algorithm on models generated under each of the following settings: (1) P-SCM with equal number of observed variables and sources ( P-SCM_Equal ); (2) P-SCM with fewer sources than observed variables ( P-SCM_Fewer )
(this indicates that deterministic relation must exist); (3) DS-P-SCM with more sources than observed variables ( DS-P-SCM ); (4) Same as Setting (1) but the model does not satisfy our conditions ( P-SCM_NonUnique ).
We only select the generating models which satisfy
Conditions 1 and 2 in the ﬁrst three settings.
The sources are independently drawn from uniform distributions; we use FastICA (Hyvarinen, 1999) and ReconstructionICA (Le et al., 2011) for BSS.
We compare our proposed algorithm with ICA-LiNGAM (Shimizu et al., 2006), DirectLiNGAM
(Shimizu et al., 2011),
ParceLiNGAM
(Tashiro et al., 2014), Pairwise lvLiNGAM (Entner and Hoyer, 2010), in terms of the differences between recovered and true adjacency matrix A , and recovered and true exogenous connection matrix B (after normalization and repermutation).
6 Evaluating B is only applicable to our algorithm since other methods do not return B .
We report the results for the normalized Structural Hamming Distance over true edges (SHD/Edges); comparisons with additional metrics appear in Appendix G.2.
We compare the performance under (i) all ICA recoveries and (ii) only the successful recoveries ( ˜ W recovered by ICA is highly accurate).
Figure 3 shows the performance of recoveries for all algorithms (whenever applicable), averaged over models randomly generated under Settings (1)-(4).
The number of observed nodes ranges from 5 to 10.
We observe that when the generating model is a P-SCM, our algorithm can learn the model more accurately than existing algorithms, even if the conditions are not satisﬁed.
This is because existing algorithms may misinterpret the shared sources among observed variables as confounding or direct causal connections, which results in adding additional edges to the recovered graph.
When the generating model is a linear DS-P-SCM, the performance of our method is also comparable to existing algorithms.
Besides, our algorithm performs signiﬁcantly better than others when the ICA [Figure] recovery is successful.
Our algorithm assumes that the true mixing matrix W can be recovered up to permutation and scaling of its columns, hence depends heavily on the accuracy of BSS recovery.
Performance on real data.
We test the performance of our recovery algorithm on the daily closing prices of the following ﬁve world stock indices: DJI, N225, N100, HSI, SSEC (Hyvärinen et al., 2010; Salehkaleybar et al., 2020).
We consider the corresponding return (relative daily change) of each index as an observed variable, and model these observed variables to be generated by a linear
P-SCM with ﬁve sources, where the sources are non-Gaussian random variables.
Using P-SCM
Recovery algorithm, the directed graph among these observed variables can be recovered as the right plot in Figure 2.
We observe that DJI is a root node of this DAG.
Further, DJI, N225 N100 all have causal effects on HSI.
Both observations are known to be true either from common belief in economy, or from previous results in (Hyvärinen et al., 2010).
6. Conclusion
We proposed the linear propagation SCM (P-SCM), which establishes a way of allowing observed variables in a linear structural causal model to deterministically depend on other observed variables or latent confounders.
We considered the problem of ﬁnding the necessary and sufﬁcient conditions to uniquely identify a linear P-SCM based on observational data.
All theoretical results are summarized in Figure 4.
We proposed an algorithm to recover the underlying causal model when a linear P-SCM is assumed to be the data generating process.
As mentioned earlier, the performance of our proposed algorithm relies on the accuracy of blind source separation (BSS) recovery.
Future directions include investigating algorithms that are robust to BSS errors.
Further, it is of interest to investigate whether relaxing the unique identiﬁability requirement could result in simpler necessary and sufﬁcient conditions that can be efﬁciently veriﬁed. Acknowledgments
This research was in part supported by the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40_180545 and Swiss SNF project 200021_204355 /1.
Appendix A. Related work: A review of causal discovery methods
Conventional methods for recovering causal directions from observational data often make use of
Bayesian graphical models (Pearl, 2009; Spirtes et al., 2000).
Bayesian networks allow for the use of efﬁcient algorithms, such as constraint-based PC and Fast Causal Inference (FCI) algorithms (Pearl, 2009;
Spirtes et al., 2000), and score-based Greedy Equivalent Search (GES) algorithm (Chickering, 2002).
However, these methods suffer from an identiﬁability problem.
In particular, they only allow for estimating the underlying causal structure from data up to its Markov equivalence class , i.e., the set of structures representing the same conditional independence assertions under causal sufﬁciency.
It turns out the Markov equivalence of the causal structure is a fundamental limit of identiﬁcation for an underlying data generating process that is a linear DS-SCM with additive Gaussian noises.
Further assumptions on the data generating process can narrow down the set of possible models compatible with the observed data.
These assumptions can be made on (i) the functional form relating the observed variables and/or (ii) the distributions of the sources (noises).
For example, Shimizu et al.
(2006) showed that assuming a linear non-Gaussian additive model (LiNGAM), in which the noises have arbitrary non-Gaussian distributions, enables unique identiﬁability of the causal structure under causal sufﬁciency.
Alternative assumptions may also lead to unique identiﬁability: (Peters and
Bühlmann, 2014) showed that if all variances of the noises in a linear Gaussian DS-SCM are equal, then one can uniquely identify the causal structure.
Hoyer et al.
(2009) assumed (i) speciﬁc types of nonlinear relationships among observed variables and (ii) additive exogenous noises, and showed that, for the case of two observed variables, nonlinearity allows for identifying the causal structure.
These developments, including LiNGAM algorithm
(Shimizu et al., 2006) and its variant DirectLiNGAM (Shimizu et al., 2011) assume causal sufﬁciency.
Without considering the effect of latent confounders, one may infer wrong causal relations among observed variables.
Studying causal models with latent confounders might be essential to infer the correct causal structure.
Hoyer et al.
(2008) and Salehkaleybar et al.
(2020) considered an extension of LiNGAM in the presence of latent confounders (lvLiNGAM).
Please refer to Appendix C.1 for detailed discussion about both methods.
LiNGAM algorithm is partly based on independent component analysis (ICA) (Comon, 1994;
Hyvarinen et al., 2002).
The identiﬁability proof and the recovery algorithm both begin with using
ICA to estimate the source mixing matrix from observations.
In lvLiNGAM, the number of sources is larger than the number of observed variables since the latent confounders are associated with additional sources.
This results in an overcomplete basis for the mixing matrix, and hence lvLiNGAM is based on overcomplete ICA (Lewicki and Sejnowski, 2000; Eriksson and Koivunen, 2004).
Our linear P-SCM generalizes lvLiNGAM to the case when the number of sources can be equal or less than the number of observed variables, since each observed variable is not necessarily associated with a distinct source.
Instead, each observed variable is exogenously connected to a subset of the jointly independent set of sources, which allows the observed variable to deterministically depend on other observed variables and latent confounders.
We further assume separability of the sources from observations, which means all sources composing each observed variable can be correctly estimated, up to scaling and permutation of the coefﬁcients.
SCMs with deterministic relations have been previously considered in the literature.
In (Geiger et al., 1990; Spirtes et al., 2000), D-separation 7 condition is proposed for graphically determining conditional independencies when deterministic relations are allowed.
Yet, it remains unclear whether D-separation condition can capture all conditional independencies induced from the distribution.
Further, a few approaches have been proposed for causal discovery.
When the system only consists of two variables with one variable deterministically causing the other, Daniušis et al.
(2010) and Janzing et al. (2012) showed that the correct causal direction can be learnt if there is no correlation between the density of the cause variable and the slope of the deterministic function (w.r.t.
a reference measure).
Their analysis however does not hold for linear relations.
Janzing et al.
(2010) and Chen et al.
(2013) considered the deterministic relation between two high-dimensional observed variables, and showed that the correct causal direction can be recovered using the trace of the covariance matrix of the cause variable and the deterministic function.
The trace method in (Chen et al., 2013) was extended by (Zeng et al., 2021) to the case of multi-variable causal discovery.
However, all these methods assume that there are only deterministic relations among observed variables.
In a system where both deterministic and non-deterministic relations are present, Scheines et al.
(1996) considered recovering the reduced model where all deterministic variables are removed.
Luo (2006) adapted the conventional constraint-based IC algorithm (Pearl, 2009) and added new rules in the independence tests to detect deterministic relations.
Mabrouk et al.
(2014) combined constraintbased methods with greedy search (Tsamardinos et al., 2006), where the deterministic relation is detected by calculating the conditional entropy among variables.
Lemeire et al.
(2012) introduced information equivalence, where two variables are information equivalent (w.r.t.
a third variable) if knowing one variable is equivalent to knowing the other, from the viewpoint of the third variable.
Information equivalence is thus a generalization of deterministic relations.
Lemeire et al.
(2012) adapted the PC algorithm with additional tests to detect information equivalence among variables.
The aforementioned methods all suffer from the same identiﬁability problem as in conventional causal discovery methods, where the underlying causal structure can only be estimated up to certain equivalence classes. In fact, majority of these works do not discuss the capability of identifying the underlying structure by their proposed algorithms (i.e., the equivalence class of the recovery result).
Further, these methods do not consider the presence of latent confounders in the system.
Appendix B. Detailed descriptions of model assumptions B.1. P-SCM faithfulness assumption
For a G-SCM, faithfulness assumption means that all marginal and conditional independencies in the underlying causal graph are captured (required) by the Markov condition. 8 For linear DS-SCMs, it is often assumed that any variable is marginally dependent on all its descendants (Salehkaleybar et al., 2020).
Thus, the faithfulness assumption translates to the linear coefﬁcients not canceling out causal effects among variables.
That is, when multiple causal effects exist from one observed variable to another, their combined effect is not exactly equal to zero.
P-SCM faithfulness assumption extends this idea to linear P-SCMs.
Speciﬁcally, Assumption 3(a) states that when multiple distinct causal paths exist from some observed variables– containing source a s j – to an observed variable x i , their combined effect do not cancel out one another, or cancel out the exogenous connection from s j to x i (i.e., s j component in x i is not equal to zero).
This is equivalent to the faithfulness assumption in (Salehkaleybar et al., 2020) for linear DS-SCMs, and is less restrictive than the conventional faithfulness assumption for G-SCMs.
Assumption 3(b) considers the coefﬁcients of matrix B , and can be explained as: Any source mixture ˜ s i cannot be written as a linear combination of other source mixtures.
Assumption 3(b) is to prevent the case when the coefﬁcients of the exogenous connections from a subset of two or more sources (say ˜ S ) to an observed variable are proportional to the coefﬁcients of the exogenous connections from ˜ S to another observed variable.
For example, suppose x 1 =
b 11 s 1 + b 21 s 2 and x 2 = b 12 s 1 + b 22 s 2 . Assumption 3(b) prevents ( b 11 , b 21 ) from being proportional to ( b 12 , b 22 ) . Note that when all model coefﬁcients { a ij } and { b ij } are independently drawn from continuous distributions, Assumptions (a) and (b) are satisﬁed almost surely (Meek, 1995), hence P-SCM faithfulness is a reasonable assumption.
B.2. Separability assumption
Separability can be achieved using blind source separation (BSS) methods, which is to separate a set of sources from a set of their mixtures given little to no information about the sources and the mixing process.
For example, when the sources are known to be non-Gaussian random variables, the mixing matrix W can be recovered using Independent Component Analysis (ICA) or overcomplete ICA methods, depending on the number of observed variables and sources (Comon, 1994; Hyvarinen et al., 2002; Lewicki and Sejnowski, 2000; Eriksson and Koivunen, 2004).
Separability can also be achieved when the sources are piecewise constant functionals satisfying a set of mild conditions (Behr et al., 2018).
We discuss BSS in more details in Appendix C.
To satisfy the separability property, we assume the availability of a large number of data vectors { X ( i ) : i ∈ [ n ] } , and that each is generated according to the described process, and using the same coefﬁcients { a ij } and { b ij } .
Appendix C. Blind source separation methods
Blind source separation (BSS) is the problem of separating a set of source signals (sources) from a set of mixed signals (mixtures) given little to no information about the sources and the mixing process (Comon and Jutten, 2010).
In the linear setting where the mixtures are linear combinations of the sources, the blindness often refers to not knowing the source realizations nor the mixing weights.
Without any further knowledge about the sources, the problem is infeasible, and hence further assumptions on the sources are needed to facilitate source separation.
An example of BSS is the independent component analysis framework in which the sources are assumed to be drawn from independent non-Gaussian distributions.
Other examples include the statistical blind source separation regression (SBSSR) model in (Behr et al., 2018) and non-negative matrix factorization in (Donoho and Stodden, 2004).
In this section, we provide a brief description of these methods, and present how to utilize them as seeds to our causal structure learning framework.
C.1. Independent component analysis
Independent component analysis (Comon, 1994) is a statistical technique to separate independent, non-Gaussian random variables (sources) from their observed linear mixtures.
Speciﬁcally, consider a vector of p observed variables (mixtures) X
=
[ x 1 , · · · , x p ] (cid:62) which is generated as
[Equation] [Equation] where S =
[ s 1 , · · · , s m ] (cid:62) are m unknown real-valued independent non-degenerate random variables (sources). W is a constant p × m unknown mixing matrix.
( W , S ) is called the representation of X .
If the observed data is invertible mixtures of non-Gaussian components (i.e., the mixing matrix
W is invertible), Comon (1994) showed that the representation of X can be uniquely identiﬁed up to scaling and permutation of the columns of W (given enough data vectors X ).
Further, if W is of full column rank and at most one source in S is Gaussian, the representation of X is still identiﬁable up to scaling and permutation indeterminacies (Eriksson and Koivunen, 2004).
For detailed explanation about ICA, the reader can refer to (Hyvärinen and Oja, 2000; Comon and Jutten, 2010).
When the number of observed variables is less than the number of sources, Eriksson and Koivunen (2004) showed that if all the m sources are non-Gaussian and the representation of X is irreducible, then W can be identiﬁed up to scaling and permutation of its columns.
Irreducibility means that the columns of W are pairwise linearly independent.
If two columns in W are linearly dependent, with corresponding sources s i and s j , then it is impossible to to distinguish between the true representation of X and another representation with m − 1 sources, where the two linearly dependent columns are merged into a single column.
We now describe LiNGAM algorithm, which uniquely recovers the causal structure of the underlying model when the mixing matrix is invertible and the sources are non-Gaussian (Shimizu et al., 2006).
The underlying DS-SCM can be written as
[Equation]
where S consists of m = p jointly independent non-Gaussian sources, and W =
( I − A ) − 1 .
Note that (6) ﬁts into the standard ICA framework in (5).
Thus, W can be identiﬁed up to scaling and permutation of its columns.
Without prior knowledge about the scale and ordering of the sources, the following general ICA model for X holds:
[Equation]
˜ W is the recovered mixing matrix, with scaling and permutation indeterminacies, and is given by
[Equation]
where P is a permutation matrix and Γ is a diagonal scaling matrix.
S (cid:48) contains the corresponding set of sources with reordering and rescaling that correspond to P and Γ .
The corresponding causal model, represented by matrix A , can be uniquely identiﬁed due to the acyclicity assumption.
Speciﬁcally, by some scaling and permutation of its columns and rows, ˜ W −can be converted uniquely to a lower triangular matrix with all ones on its main diagonal.
In the presence of latent variables, Hoyer et al.
(2008) and Salehkaleybar et al.
(2020) utilized overcomplete ICA to extend LiNGAM algorithm.
Given the output of the overcomplete ICA, the recovered mixing matrix ˜ W is unique up to scaling and permutation of its columns, under the assumption of irreducibility.
The number of latent variables in the system is assumed to be known.
The task is to identify the columns of ˜ W which correspond to exogenous noises associated with observed variables, and subsequently apply LiNGAM algorithm after eliminating the columns corresponding to latent variables.
Hoyer et al.
(2008) proposed an algorithm which brute-forces all possible classiﬁcations of the columns of ˜ W (whether each of the columns corresponds to distinct source associated with an observed variable, or to a latent confounder).
The algorithm then identiﬁes the possible selections that are compatible with faithfulness and acyclicity assumptions.
More recently, Salehkaleybar et al.
(2020) proposed a more efﬁcient algorithm that identiﬁes the set of columns corresponding to observed variables.
The idea is to learn the skeleton of the underlying graph via pairwise comparisons of the observed variables (based on their corresponding rows in ˜ W ).
This is possible due to faithfulness and acyclicity assumptions.
Using the learnt skeleton, the algorithm selects only the columns whose non-zero entries are compatible with the observable descendants of each observed variable.
C.2. Statistical blind source separation regression model
Statistical blind source separation regression (SBSSR) model is a ﬁnite alphabet BSS method proposed in (Behr et al., 2018) to separate piecewise constant source functions from their linear mixtures.
Suppose we have a set of m source functions S
= { s 1 , s 2 , · · · , s m } , each of which consists of an array of constant segments (i.e., a step function) on [0 , 1] .
The function values are selected from a known ﬁnite alphabet, yet the jump sizes, numbers, and locations of change points for each function are unknown.
Speciﬁcally, let U = { u 1 , u 2 , · · · , u k } ⊂ R be a known ﬁnite and totally ordered alphabet, i.e., u 1 < u 2 < · · · < u k .
Each source function belongs to the class
S =
[Equation]
1 = 1 , K
[Equation]
N   .
K is the unknown number of changes, which is assumed to be ﬁnite.
θ 1 , · · · , θ K are the function values selected from the alphabet U such that θ j (cid:54)
= θ j +1 for j = 0 , 1 , · · · , K .
τ 1 , · · · , τ K are the change points.
We observe samples of p linear mixtures { x k ( t ) , k = 1 , · · · , p } at n uniformly selected location points t i in [0 , 1] :
[Equation]
W k =
[ w k 1 , · · · , w km ] (cid:62) is the vector of mixing weights for mixture x k with
[Equation]
S ( t i ) =
[ s 1 ( t i ) , · · · , s m ( t i )]
(cid:62) is the vector of values of the source functions at point t i for all i ∈ [ n ] .
(cid:15) ki is an additive Gaussian noise with zero mean and unit variance, and σ > 0 is the known noise variance.
Behr et al.
(2018) showed that, under mild separability conditions among weights and source functions, both the source functions and the mixing weights can be identiﬁed up to permutations, including the number of change points, change point locations, and the function values on each segment for all source functions.
Unlike ICA, the mixing matrix W in SBSSR does not necessarily need to satisfy the irreducibility assumption, due to the prior knowledge about the sources alphabet.
In fact, the recovered mixing matrix ˜ W using SBSSR can include two linearly independent columns, while the method is able to separate the corresponding sources.
However, even with the ability to separate such sources, we require to merge them into one combined source, since the two corresponding generating models are observationally equivalent, as we will shortly explain.
C.3. Non-negative matrix factorization
Non-negative matrix factorization (NMF) is another BSS method to decompose large matrices into the multiplication of low-rank, non-negative smaller matrices (Lee and Seung, 1999).
Suppose each observed variable is an n 1 × n 2 matrix X k with rank r k .
Using NMF, X k can be decomposed into
[Equation]
W k is a non-negative n 1 × r k weight matrix.
F k is a non-negative n 2 × r k feature matrix.
W ki , F ki are the column vectors of W k , F k for all i
= 1 , · · · , r k .
If we consider the feature vectors F ki as sources, then we can write each observed matrix X k into a linear combination of sources (here, both sources and weights are column vectors).
Donoho and Stodden (2004) showed that under certain conditions on W k and F k , the factorization (10) is unique up to permutation and scaling of the feature column vectors, i.e., the sources.
C.4. Application to our model
Under different assumptions on the sources and the mixing matrix, we can use the aforementioned
BSS methods to estimate the correct representation of the observed variables.
In the BSS methods described in Appendices C.1-C.3, the permutation and scalability indeterminacies always exist, except for SBSSR, where we do not have scalability indeterminacy due to the prior information about the sources alphabets and the mixing weights.
In general, more prior information about the sources may eliminate the scalability indeterminacy.
Another example where the scalability indeterminacy relaxes is when the sources in the ICA framework are all known to have unit variance.
In the same spirit, having prior information about the ordering may eliminate the permutation indeterminacy.
The output of BSS is the factorization of X as in (7), where under permutation and scalability indeterminacies, ˜ W is given by (8) , and S (cid:48) is given by
[Equation]
P is the m × m permutation matrix with PP (cid:62)
=
I , and Γ is the diagonal scaling matrix.
Recall that in LiNGAM, the mixing matrix W =
( I − A ) − 1 is lower triangular with all ones on the main diagonal, since A is strictly lower triangular due to the DAG-ness assumption of the underlying graph.
By contrast, in our model, the mixing matrix in (3) does not possess these structural restrictions since
W =
( I − A ) − 1 B , where B can have an arbitrary structure.
As in LiNGAM, our recovery algorithm is robust to permutation and scaling ambiguities.
From (3) and (8), the recovered mixing matrix in our model can be written as
[Equation]
By writing ˜ B as ˜ B
[Equation]
BPΓ [Equation] [Equation] we have
[Equation]
Our recovery algorithm is in fact able to recover ( I − A ) − 1 and ˜ B using ˜ W , where ˜ B is a columnrepermuted and rescaled version of B .
The permutation indeterminacy of B is immaterial because there exists no ordering among the exogenous connections from the sources, and we are not concerned about the scalability indeterminacy in recovering the strengths of these exogenous connections.
Therefore, permutation and scalability indeterminacies do not affect the recovery performance of our algorithm.
In the following, we provide an example for the aforementioned indeterminacies.
Example 1 Consider the generating model in Figure 5.
In the matrix form, the generating model is
[Equation]
[Equation]
[Equation]
The observed variables can be represented as linear combinations of the sources as follows:
[Equation]
where w 31 = a 31 b 11 + a 32 b 21 + b 31 .
Given samples of the observed variables X , we can apply BSS methods to learn the representation of X up to permutation and scaling of the columns of W .
Suppose the recovered mixing matrix is [Equation]
γb [Equation]
αa
[Equation]
γwβa
[Equation]
  . The permutation and scaling matrices are given by
[Equation]
γ
[Equation]
and the corresponding sources are
[Equation]
α, β, γ (cid:54) = 0 are the scaling coefﬁcients.
Given the recovered mixing matrix ˜ W , our P-SCM recovery algorithm uniquely recovers A and ˜ B = BPΓ , as explained in Section 3.3.
Thus the recovered model is
[Equation]
  αbγb  γbβb γb     s (cid:48)s (cid:48)s (cid:48)  .
Note that ˜ B S (cid:48)
=
B S , hence the recovered model in (14) is identical to the generating model in (13) .
However, we do not learn the representation B S uniquely, instead we learn ˜ B S (cid:48) .
The irreducibility assumption in our linear P-SCM is without loss of generality.
Under P-SCM faithfulness assumption, reducibility occurs if and only if two sources s i and s j are exogenously connected to a common observed variable, and are not exogenously connected to any other observed variables.
Speciﬁcally, we can write the corresponding columns in W =
( I − A ) − 1 B (cf.
(3)) as
[Equation]
where B i and B j are the columns in B corresponding to sources s i and s j .
Since ( I − A ) − 1 is an invertible square matrix, W i and W j are linearly dependent if and only if B i and B j are linearly dependent.
Under P-SCM faithfulness assumption (b), this happens if and only if [ B i B j ] have only one non-zero row, which means s i and s j are exogenously connected to a common observed variable, and to this variable only.
Therefore, when W i and W j are linearly dependent, i.e., W i = αW j , α (cid:54) = 0 , the corresponding sources s i , s j appear at each node x k as w kj ( s j + αs i ) , where w kj ∈ R .
Since s i and s j are connected to the same variable, replacing the exogenous connections from s i , s j by a combined source s (cid:48) i =
s j + αs i in the generating model results in a model that has the same distribution as the original generating model (i.e., an observationally equivalent model).
In conclusion, under different problem settings and assumptions, given observations of variables
X , we can use BSS methods to recover the mixing matrix W up to permutation and/or scalability indeterminacies, which have no impact in recovering the true generating model.
We call these models separable, as mentioned in Section 2.2.
Connection to LiNGAM.
According to the above analysis and the comparison between DSSCM and P-SCM in Section 4, both LiNGAM and lvLiNGAM models are included in our linear
P-SCM.
Speciﬁcally, our model reduces to LiNGAM when each observed variable is directly inﬂuenced by a single source, and the sources are jointly independent non-Gaussian random variables with non-zero variances.
Here, separability holds by using ICA.
Similarly, lvLiNGAM corresponds to the case when the sources are non-Gaussian, and each of the source mixtures ˜ s i has at least one distinct source.
As we previously mentioned, for lvLiNGAM, separability hold using overcomplete
ICA.
In this work, we impose a more general assumption on the separability of the source variables from observations.
This framework includes ICA and overcomplete ICA as special cases, but is not restricted to these two methods.
Appendix D. Examples D.1. Generalized distinct source assumption
As mentioned in Section 2.1, Assumption 1 is stronger than what is needed for structure learning methods developed for linear DS-SCMs. Here, we provide a weaker version of the distinct source assumption, under which methods developed for linear DS-SCMs still work. Assumption 5 (Generalized distinct source assumption)
A linear G-SCM satisﬁes the distinct source assumption if for each observed variable x , there exists a source such that the information regarding that source in the source mixture of x cannot be fully described by the source mixtures of the rest of the variables.
The intuition behind Assumption 5 can be explained as follows.
If, for example, source s is shared between two variables x 1 and x 2 , yet x 2 is only a function of a coarsened version of s , then the remaining information of s could be considered as a distinct source for x 1 . In the following we provide a concrete example where depending on the set of functions { g x } x ∈X , the model with the same causal structure may or may not satisfy Assumption 5. Example 2 Consider a linear G-SCM with two observed variables x 1 , x 2 and two independent source variables s 1 , s 2 . Suppose s 2 is discrete with support { 1 , 2 , 3 , 4 } , and the generating models of x 1 and x 2 are
[Equation]
where g x 1 , 1 , g x 1 , 2 and g x 2 are possibly nonlinear functions.
• If all three functions are linear, then the information regarding s 2 in the mixture of x 2 , i.e., g x 2 ( s 2 ) , can be written as a deterministic function of g x 1 , 2 ( s 2 ) , the information regarding sin the mixture of x 1 .
That is, x 2 does not have a distinct source and hence, the model is not a linear DS-SCM.
• If g x 2 ( s
) is a linear function of s 2 , but g x 1 , 2 ( s 2 ) only depends on whether s 2 is an even number (
e.g., g x 1 , 2 ( s 2 ) =
c 1 1 [ s 2 ∈ { 1 , 3 } ]+ c 2 1 [ s 2 ∈ { 2 , 4 } ] for some constants c 1 , c 2 ); then we can write s 2 as the combination of two separate sources s 3 and s 4 , where s 3 , s 4 ∈ { 0 , 1 } , and
[Equation]
Note that s 2 = 2 s 3 + s 4 + 1 , and g x 1 , 2 ( s 2 ) is only a function of s 4 .
We can replace s 2 in the model by the combination of s 3 and s 4 , and rewrite the model as
[Equation]
g
[Equation]
where f ( s 4 ) = c 1 1 [ s 4 = 0]+ c [ s 4 = 1] .
Therefore, because of s 3 , the information regarding s 2 in the mixture of x 2 (i.e., g x 2 (2 s 3 + s 4 + 1) ) cannot be written as a deterministic function xxxxxssssxxxxxxxssss of the information regarding s 2 in the mixture of x 1 (i.e., f ( x 4 ) ). Further, we can consider sas a distinct source of x 2 , and x 4 as a latent confounder between x 1 and x 2 . Therefore, both variables have distinct sources, and the model is a linear DS-SCM. D.2.
Examples showing the distinction between linear P-SCM and linear DS-P-SCM
In this subsection, we demonstrate the practical signiﬁcance of linear P-SCM over a linear DS-PSCM.
Speciﬁcally, we answer the following question: Using the more general linear P-SCM, can we correctly identify models that are not identiﬁable using linear DS-P-SCMs?
In the following, we provide two examples where linear P-SCMs can explain the underlying causal model, while linear
DS-P-SCMs, with or without latent confounders, either are inapplicable or give misleading results.
Therefore, the model cannot be recovered using causal discovery methods for linear DS-P-SCMs, but can be correctly recovered by our P-SCM Recovery algorithm.
Example
In this example, we begin with the ground truth (i.e., the underlying model), and show that linear DS-P-SCMs fail to explain the model, while our linear P-SCM explains the model.
Consider the graph on the left in Figure 6, which represents the true generating model for this example.
We have ﬁve observed variables x 1 , · · · , x 5 and four jointly independent sources s 1 , · · · , s 4 .
Since there are less sources than observed variables, the model cannot be represented by a linear DS-PSCM with or without latent confounders.
This is because each observed variable in linear DS-P-SCMs must be associated with an distinct source that is not shared with other observed variables.
Hence the number of independent sources has to be greater than or equal to the number of observed variables in linear DS-P-SCMs.
On the other hand, it is easy to see that the ground truth in Figure 6 can be modeled by our linear P-SCM, which allows for observed variables not to have exogenous connections, and explains “latent confounding” simply by “shared” exogenous connections from the sources (for instance, shas exogenous connections to both x 1 and x 2 ).
Lastly, it is worth mentioning that the generating model in this example has a hierarchical structure that entails practical signiﬁcance: x 1 , x 2 and x 3 are exogenously connected to different pairs of sources, while x 4 and x 5 are composed of different triplets of sources (through different pairs of causal connections from x 1 , x 2 and x 3 ).
We thus can treat { x 1 , x 2 , x 3 } as one layer, and xxxxssssxxxxxssssxxxxssss { x 4 , x 5 } as another layer below { x 1 , x 2 , x 3 } .
This hierarchical structure may arise naturally in inﬂuence propagation problems through networks of nodes.
For instance, one layer of twitter users tweet news directly from the news sources (NY-times, Washington-Post, etc), then another layer of users retweet from the ﬁrst layer, and so on.
In conclusion, this example show that our linear P-SCM is more suited to model such a setting than a linear DS-P-SCM.
Example 4 We begin with a ground truth graph and show that– as in the previous example– linear
DS-P-SCMs with latent confounders fail to explain the model.
Further, we show that linear DS-PSCMs without latent confounders can explain the model, yet the recovery under this assumption results in misinterpreted edges (low accuracy).
A linear P-SCM, on the other hand, explains the model and results in the recovery of a correct network.
Consider the graph on the left in Figure 7, which represents the true generating model.
We have four observed variables x 1 , · · · , x 4 and four jointly independent sources s 1 , · · · , s 4 .
Notice that sis exogenously connected to two observed variables. To explain the model by a linear DS-P-SCM, a latent confounder x 5 must be added to the model as shown in the middle in Figure 7 (to explain the shared sources), which leaves observed variable x 3 having no distinct sources. Therefore the ground truth cannot be explained by a DS-P-SCM with latent confounders. On the contrary, the ground truth can be explained by a linear P-SCM, which allows each observed variable to be exogenously connected to a subset of the sources.
Note that, since we have four observed variables and four sources, the model can be modeled by a linear DS-P-SCM without latent confounders (i.e., both models have the same joint distribution over the observed variables), where each observed variable is exogenously connected to one distinct source, and to this source only.
This however leads to incorrect recovery as explained in the plot on the right in Figure 7.
Speciﬁcally, in the true generating model, s 3 is exogenously connected to both x 3 and x 4 , while x 4 has a distinct source s 4 .
Thus, assuming a linear DS-P-SCM as the generating model, s 3 will be the distinct source of x 3 , and the exogenous connection from s 3 to xwill be interpreted as a causal connection from x 3 to x 4 .
This will in turn necessitate the erroneous causal connection from x 1 to x 4 to cancel out the causal effect of x 1 on x 4 through x 3 .
Notice here that these additional connections break the marginal independency between x 1 and x 4 , therefore the recovered model is unfaithful to the true generating model.
D.3. Example explaining the possible parent set
Example 5 Suppose we have 3 observed variables x 1 , x 2 , x 3 and 4 source variables s 1 , s 2 , s 3 , s 4 . The generating model is given by x 1 =
b 11 s 1 + b 12 s 2 ; x 2 =
b 21 s 1 + b 23 s 3 ; x 3 = a 31 x 1 + b 34 s 4 .
The observed variables can be written as a linear combination of the sources as follows:
[Equation]
The recovered mixing matrix ˜ W is a column-permuted and rescaled version of W .
Using ˜ W , one possible causal order of the observed variables is ( x 1 , x 2 , x 3 ) , where one may further conclude that x 1 and x 2 have no parents and x 3 has x 1 as a parent.
Note that both x 1 and x 2 precede x 3 in this causal order, hence both need to be considered when recovering the causal structure of x 3 .
However, the possible parent set of x 3 is P 3 = { x 1 } (as deﬁned in Section 3): x 2 cannot be a parent of x 3 since x 3 does not have s 3 as a component.
To conclude, the possible parent set of x 3 , i.e., P 3 , is a strict subset of the observed variable preceding x 3 in the casual order deduced from ˜ W .
D.4. Example explaining the deﬁnition of unique component
Example 6 Consider a generating model with seven observed variables x 1 , · · · , x 7 and eight source variables s 1 , · · · , s 8 .
Let us ﬁx x 7 .
Suppose all the other six variables are possible parents of x 7 , i.e., P 7 = { x 1 , · · · , x 6 } .
The exogenous connections from the source variables to the observed variables in P 7 are
[Equation]
Let us ﬁrst consider the possible parent set of x 7 , i.e., P 7 .
The index set of P 7 is J (0) =
According to the equations, s 1 is connected only to x 1 , and s 2 is connected only to x 2 .
Thus, the unique component set of x 1 is U 7 (1) =
{ s 1 } , and the unique component set of x 2 is U 7 (2)
= { s 2 } .
The other observed variables in P 7 do not have unique components.
Thus, the subset of P 7 with no unique components is P (1) =
{ x 3 , x 4 , x 5 , x 6 } .
The corresponding index set is J (1) = { 3 , 4 , 5 , 6 } .
Next, we consider only the observed variables in P (1) with indices in J (1) .
s 4 is connected only to x 3 , and hence U 7 (3)
= { s 4 } .
Similarly, U 7 (4) = { s 5 } , and J (2) = { 5 , 6 } .
Finally, each of the observed variables with indices in J (2) ( x 5 , x 6 ) is connected to both sources s 7 and s 8 .
Thus, neither x 5 nor x 6 has a unique component, and the procedure terminates.
To conclude, the set of observed variables with no unique components is I 7 = { x 5 , x 6 } .
D.5. Examples showing the necessity of the conditions
In this subsection, we provide examples on how Conditions 1 and 2 are necessary for the unique identiﬁability of a linear P-SCM.
More speciﬁcally, in Example 7 below, we show that, for a given observed variable x k , if any source variable that is a unique component of a possible parent x i ∈ P k has an exogenous connection to x k , then we can not uniquely identify the total causal effect from x i to x k in the corresponding generating model.
In other words, for unique identiﬁability of the causal effects to x k , ˜ s k must not contain a unique component of any x i ∈ P k .
Notice that since we consider a ﬁxed observed variable, x k , and aim to recover/estimate the total causal effects from the observed variables in P k , we do not consider the causal connections among these possible parents.
Example 7 Consider the following generating model: [Equation] [Equation] [Equation]
[Equation]
b
[Equation] [Equation] b [Equation]
b
[Equation]

   sss
[Equation]
The observed variables can be represented as linear combinations of the sources as follows:
[Equation] [Equation]
Given the observations X =
( x 1 , x 2 , x 3 ) (cid:62) , the mixing matrix W in (16) can be recovered up to permutation and scaling indeterminacies as described in Appendix C.4.
For simplicity, we assume that the recovered mixing matrix ˜ W is the same as W .
The intricacies resulting from permutation and scaling indeterminacies is resolved in our proposed P-SCM Recovery algorithm.
We assume that the causal structure of x 1 and x 2 have been correctly recovered as in (15) , and consider observed variable x 3 .
From the blind source separation output, i.e., the recovered mixing matrix W in (16) , we deduce that x 1 and x 2 are possible parents of x 3 .
We can rewrite the generating model of x 3 in (15) as follows:
[Equation]
where
[Equation]
[Equation]
Notice that s 2 is a unique component of x 1 .
The s 2 component in x 3 (cf.
(16) ) consists of two parts: a 31 b 12 s 2 which results from a causal connection from x 1 , and b 32 s 2 which results from an exogenous connection from s 2 .
In order to learn the causal structure of x 3 , we ought to be able to differentiate between these two parts from their mixture in the mixing matrix W , which is not possible in this example.
Speciﬁcally, from the recovered mixing matrix ˜ W = W , we can not distinguish the true generating model in (17) from the two following generating models: Model 1 :
[Equation]
where ˜ s (1) =
[Equation]
b 11 bb 12 s 1 , which corresponds to the s 2 component in x 3 being caused only by a causal connection from x 1 ; and
Model 2 :
[Equation]
where ˜ s (2)
= a 31 b 11 s 1 + ( a 31 b 12 +
[Equation]
s
[Equation]
which corresponds to the s 2 component being caused only by an exogenous connection from s 2 .
Similarly, if the generating model in (17) is such
that ˜ s 3 =
b 33 s 3 , the causal effects cannot be uniquely identiﬁed, since s 3 is a unique component for x 2 .
In conclusion, to uniquely recover the causal structure of x 3 , neither s 2 nor s 3 can be included in ˜ s 3 .
Next, we present an example about the necessity of the marriage condition for unique identiﬁability a in linear P-SCM.
Recall that the marriage condition is: For a given observed variable x k and its possible parent set P k , every subset of the possible parents X C ⊆
P k must include at least | X C | different source variables.
Example 8 (Marriage condition) Consider the following generating model:
[Equation]
The observed variables can be written as follows:
[Equation]
We recover the mixing matrix W using BSS.
As in the previous example, we assume that ˜ W = W .
Consider the observed variable x 6 .
From W , the possible parents of x 6 are P 6 = { x 1 , · · · , x 5 } .
Notice that the marriage condition is not satisﬁed in this example: (cid:12)(cid:12)(cid:83) x
i ∈P 6 Comp( x i ) (cid:12)(cid:12)
= 4 while |P 6 | = 5 , therefore, |P 6 | (cid:2) (cid:12)(cid:12)(cid:83) x
i ∈P 6 Comp( x i ) (cid:12)(cid:12) .
In the following, we show how this renders the unique identiﬁability to be impossible.
Consider X [1:4] =
( x 1 , x 2 , x 3 , x 4 ) (cid:62) .
These observed variables only contain the source variables
S [1:4] =
( s 1 , · · · , s 4 ) (cid:62) , and thus can be written as X [1:4]
= W 0 S [1:4] , where W 0 is the upper left 4 × 4 submatrix of W in (19) .
W 0 is invertible due to P-SCM faithfulness assumption.
Now, xcontains only the sources ( s 3 , s 4 ) , and hence can be written as a linear combination of X [1:4] as follows:
[Equation]
where c i , i = 1 , · · · , 4 can be calculated using the exogenous connections.
The fact that x 5 is linearly dependant on (i.e., can be represented as a linear combination of) x 1 , · · · , x 4 results in impossibility of learning the causal connections to x 6 from observations.
Speciﬁcally, it is impossible to distinguish the true generating model of x 6 in (18) from the following model, in which x 5 in the true generating model is replaced by (20) :
[Equation]
Therefore, the generating model is not uniquely identiﬁable.
As we have seen in this example, unique identiﬁability of causal effects to x 6 fails because, in the representation of X recovered by BSS (cf.
(19) ), x 5 is linearly dependent on the remaining possible parents of x 6 , i.e., X [1:4] .
In Appendix F.3, we show that linear independence among possible parents in P k holds if and only if the marriage condition is satisﬁed.
Appendix E. Proof of main theorem E.1. Matrix representation of the necessary and sufﬁcient conditions
Let us ﬁrst deﬁne the existing component set E k for each observed variable x k as follows.
Deﬁnition
The existing component set of x k , denoted by E k , is the set of all components in the possible parents of x k , i.e., E k
=
(cid:83) x
i ∈P k Comp( x i ) .
Recall that matrix B in (2) consists of all the exogenous connections from sources to observed variables.
Let B k denote the submatrix of B with rows corresponding to the possible parents of x k , i.e., P k , and columns corresponding to the existing component set of x k , i.e., E k .
That is,
[Equation]
We repermute the rows and columns of B k , and label the columns, as follows:
(i)
We ﬁnd the columns in B k with only one non-zero entry and label these columns as type-1.
We repermute the columns of B k such that all type-1 columns are at the leftmost of B k .
(ii)
We repermute the rows of B k such that all nonzero entries in type-1 columns appear in the upper rows.
Note that two type-1 columns can have their non-zero entry in the same row.
(iii)
Among the columns that are not type-1 (have two or more non-zero entries), we ﬁnd those columns with their non-zero entries appearing in the non-zero part of type-1 columns (i.e., the upper non-zero rows).
We label these columns as type-2.
We repermute the columns of B k such that type-2 columns are next to type-1 columns.
The matrix B k can be now written as
[Equation]
[ U 1 ; 0 ] are the type-1 columns, [ X 1 ; 0 ] are the type-2 columns, and [ Y 1 ; Z 1 ] are the remaining columns.
Each column of Z 1 has at least one non-zero entry.
(iv) We repeat steps (i) to (iii) to permute the columns and rows of Z 1 such that
[Equation]
The columns of B k corresponding to the columns of U 2 and X 2 (which were unlabeled in (23)) are now labeled as type-1 and type-2 respectively.
(v) For Z n , n = 2 , 3 , · · · , we repeat steps (i) to (iii) and label the corresponding columns in B k as in (iv); until this factorization does not hold for Z N for some N ∈ N .
We label the columns of B k that correspond to the columns of Z N as type-3 columns.
The column and row permutations in (i)-(v) are analogous to the iterative process in Deﬁnition 3.
Type-1 columns in (i) correspond to the unique components in the ﬁrst iteration, i.e., among possible parents in P k .
The rows selected in (ii) correspond to the possible parents in P k \ P (1) k .
Typecolumns correspond to the sources that are shared only among the possible parents in P k \ P (1) k .
Z 1 in (23) represents the exogenous connections of observed variables in P (1) k .
For the iteration in (i)-(v), Z n , n = 1 , 2 , · · · , N , can be written as
[Equation]
Lastly, the rows in Z N correspond to the possible parents in I k , and the sources shared among these variables correspond to type-3 columns.
Given the matrix permutation and labeling in (i)-(v), the unique components condition translates to: Each observed variable x k can only be exogenously connected to either source variables corresponding to type-2 columns in B k , or new source components that are not in E k (i.e., do not correspond to columns in B k ).
Further, the marriage condition translates to the matrix B k being of full row rank, which is equivalent to Z N being of full row rank.
The equivalence between the marriage condition and the full-row rank of B k is explained in Appendix F.3.
E.2. Proof of sufﬁciency
For any data generating model that satisﬁes Conditions 1 and 2 in Sections 3.1 and 3.2, we show that using the recovered mixing matrix ˜ W , we can construct an algorithm that uniquely recovers both the causal effects among observed variables and the exogenous connections from the sources.
Speciﬁcally, we show that P-SCM Recovery Algorithm (Algorithm 1) recovers the exact matrix of total causal effects, i.e., ( I − A ) − 1 , and the exogenous connection matrix B up to permutation and scaling, i.e., ˜ B in (11).
The row permutation in Step 1 of Algorithm 1 has the property that for each observed variable x k , the possible parents of x k are all preceding x k in the row ordering.
This follows because for every x i ∈ P k , Comp( x i ) (cid:40) Comp( x k ) according to Deﬁnition 2.
Without loss of generality, assume that the ordering among observed variables after the permutation in Step 1 is a natural ordering.
In this case, we have P k ⊆ { x 1 , · · · , x k − 1 } for all x k .
We provide the proof by induction on the index of the observed variables k , where the induction base is k = 1 .
For k = 1 , there are no observed variables in P 1 , hence all the source components of x 1 must result from exogenous connections, which are learnt from the recovered mixing matrix.
For the induction hypothesis, we assume that the total causal effects among { x 1 , · · · , x k − 1 } and their exogenous connections from the sources are given.
We show that we can recover the total causal effects as well as the exogenous connections to x k .
From the induction hypothesis, we know the exogenous connections (source mixtures) to the possible parents of x k , i.e.,
˜ S
k
=
˜ s i : x i ∈
P k } .
Our algorithm follows a similar iterative procedure as in Deﬁnition 3, in order to identify the unique components of each x i ∈ P k , while simultaneously solve for the total causal effect from x i to x k .
First, the algorithm ﬁnds the observed variables with unique components among all possible parents in P k , i.e., x i ∈ P k \P (1) k
(see Section 3.1).
For each x i ∈ P k \P (1) k with a unique component s j , it follows from the unique component condition that s j is not exogenously connected to x k .
Thus, the s j component in x k must result from the causal connection/path from x i .
The strength of the causal effect can be correctly recovered via dividing s j component in x k by s j component in ˜ s i , i.e., ˜ w kj / ˜ b ij .
Subsequently, we correctly recover the total causal effect from all x i ∈ P k \ P (1) k to x k .
Remark 9 Note that s j component in x i only results from an exogenous connection from s j to x i , since the parents of x i are also in P k and s j is a unique component.
Hence the s j component in ˜ s i is the same as the s j component in x i .
This explains our use of “dividing by s j coefﬁcient in x i ”, instead of in ˜ s i , in Section 3.1.
Next, we ﬁnd the total causal effects from x i ∈ P (1) k .
Similarly, consider x i ∈ P (1) k \ P (2) k with a unique component s j .
It follows from the unique components condition that s j is not exogenously connected to x k .
Thus, s j component in x k results from the following effects: (i) causal connection/path from x i and/or (ii) causal connections/paths from observed variables in P k \ P (1) k .
The latter causal effects are recovered in the last step and subtracted from x k , and hence we calculate the total causal effect from x i to x k by the residual of s j component in x k divided by s j component in ˜ s i .
Therefore, we correctly recover the total causal effects from all x i ∈ P (1) k \ P (2) k to x k .
Repeat the last step for x i ∈ P ( n ) k \P ( n +1) k for n = 2 , · · · , N − 1 , i.e., all observed variables with unique components.
Next, we ﬁnd the total causal effects from x i ∈
I k (with no unique components) to x k .
The unique components condition implies that x k cannot be exogenously connected to any source variable that belongs to (cid:83)
i ∈ J ( N )
Comp(˜ s i ) (recall that J ( N ) is the index set of I k ).
That is, the exogenous connections to x k can not overlap with the exogenous connections to any x i ∈
I k .
Then, we recover the total causal effects from x i ∈
I k to x k using the residuals of the set of sources (cid:83) i ∈ J ( N ) Comp(˜ s i ) in x k (after all subtractions in the previous steps).
This results in an overdetermined linear system of the total causal effects:
[Equation]
w is the vector of the residuals of the source components in x k that correspond to (cid:83) i ∈
J ( N ) Comp(˜ s i ) ;
˜ a k is the vector of total causal effects from the observed variables in I k to x k ; Z N represents the exogenous connections to observed variables in I
k
(see Appendix E.1).
According to the marriage condition, Z N is of full row rank, see Appendix E.1.
Thus, the overdetermined linear system in (25) has at most one solution which corresponds to the true total causal effects from each x i ∈
I
k .
Note that ˜ B k has the same rank as B k since permuting the columns does not change the rank.
To sum up, Algorithm 1 correctly recovers the total causal effects from the possible parents of x k ( x i ∈ P k ) to x k .
The remaining source components in x k represent the exogenous connections from the corresponding sources to x k , which can be recovered up to permutation and scalability indeterminacies.
The recovered matrix ˜ B has the same columnpermutation and scaling as the output of BSS ˜ W .
That is, Algorithm 1 is able to recover the exact adjacency matrix A using the unique components and/or matrix inversion, by shifting all the indeterminacies into the residual values to recover ˜ B .
Recall, for matrix B , the column permutation indeterminacy is irrelevant; the scalability indeterminacy is unavoidable in our setting unless we have prior information about the sources.
The column permutation is maintained for ˜ B because the order of the sources, given in ˜ W , is not changed throughout Algorithm 1.
For scalability of the causal effects, suppose the learnt exogenous connections from a source s l to all x i ∈ P k (i.e., the column of ˜ B k corresponding to s l ) share the same scale as the corresponding column in ˜ W .
Algorithm
ﬁrst recovers the correct total causal effect ˜ a ki = ˜
w kj / ˜ b ij (cf.
Step 6) using the unique components for x i ∈ P k \ P (1) k .
The residual of s l in x k after subtracting the effect of x i (Step 7), which is ˜ w kl − ˜ a ki ˜ b il , shares the same scale as ˜ w kl .
Similarly, for all x i with unique components, i.e., x i ∈ P k \ I k , Algorithm 1 recovers the correct total causal effect ˜ a ki , and the residual shares the same scale as ˜ w kl .
For x i ∈ I k , Algorithm 1 recovers the correct total causal effect using (25) (cf.
Step 10), since each column of Z N in (25) shares the same scale as the corresponding entry of w .
Finally, the remaining s l component in x k shares the same scale as ˜ w kl , which is then considered as the exogenous connection ˜ b kl .
Therefore, the learnt exogenous connections from s l to all observed variables x k , which are in the column of ˜ B corresponding to s l , share the same scale as the corresponding column in ˜ W .
In conclusion, given the causal structure and exogenous connections to the possible parents of x k , we are able to recover the exact total causal effects to x k along with its exogenous connections.
This establishes the induction step and completes our sufﬁciency proof.
Remark 10 As we mentioned earlier, for each observed variable x k , the possible parents of x k , i.e.,
P k , are all preceding x k in the row ordering deduced in Step 1 of Algorithm 1.
Since the ancestors of x k are also included in P k , this implies that by sorting the observed variables according to the number of source components in the recovered mixing matrix ˜ W , we can ﬁnd a causal order among the observed variables which is consistent with the correct order.
This method is faster than the pairwise comparison in (Salehkaleybar et al., 2020, Lemma 5).
E.3. Proof of necessity
We show that for any generating model G that violates either the unique component condition or the marriage condition, by using the corresponding ˜ W , it is not possible to distinguish between the true generating model G and another generating model G (cid:48) .
Without loss of generality, assume that the observed variables are ordered such that for any x k , P k ⊆ { x 1 , · · · , x k − 1 } .
This ordering can be achieved by permuting the rows of ˜ W to the corresponding order.
Now, consider a generating model G that does not satisfy either or both of the conditions.
Suppose x k has the smallest index among the observed variables such that the conditions are not satisﬁed.
That is, the submodel { x 1 , · · · , x k − 1 } does satisfy the unique component and marriage conditions.
We can write the generating model G of x k as:
[Equation]
where we redeﬁne ˜ a ki to be the total causal effect from observed variable x i to x k , and b kj is the strength of the exogenous connection from s j to x k .
Non-satisﬁability of the theorem conditions can only happen when either of the following cases hold.
(i) ∃ x i 0 ∈
P k with a non-empty unique component set, such that at least one unique component of x i 0 is exogenously connected to x k .
(ii) ∃ x i
∈
P k with an empty unique component set such that at least one component of ˜ s i 0 is exogenously connected to x k .
(iii)
The collection of component sets of the source mixtures ˜ S
k = { Comp(˜ s i ) : x i ∈ P k } does not satisfy the marriage condition.
Case (i).
Suppose there exists an index i 0 such that U k
( i 0 )
∩ Comp(˜ s k ) (cid:54) = ∅ .
Let s j 0 ∈
U k
( i 0 )
∩ Comp(˜ s k ) , i.e., s j 0 is a unique component of x i 0 and is also exogenously connected to x k .
In the following, we show we cannot determine if the component s j 0 in x k results from a causal connection/path or an exogenous connection to x k , or both.
We ﬁrst take out the terms that include ˜ s i 0 and s j 0 from x k in (26), and rewrite (26) as
[Equation]
k
[Equation]
Next, consider the exogenous connections to x i 0 (i.e., ˜ s i 0 ), which can be expressed as
[Equation]
By substituting (28) in (27), we get x k = (cid:88)
[Equation]
 = (cid:88)
[Equation]
Suppose the total causal effects from all observed variables x i ∈ P k \ { x i 0 } are recovered as in (29).
Now, our task is to recover the total causal effect from x i 0 to x k .
Let us consider the s j 0 component in (29).
It consists of two parts: ˜ a ki 0 b i
j 0 s j 0 which results from the causal connection/path from x i 0 , and b kj 0 s j 0 which results from the exogenous connection from s j 0 .
In order to learn the causal structure, we ought to be able to differentiate between these two parts from their mixture in (29), which is not possible.
More speciﬁcally, we can not distinguish the true model from the following two generating models:
[Equation]
which corresponds to the s j 0 component in x k being caused only by the causal effect from x i 0 .
Model 2:
[Equation]
which corresponds to the s j 0 component being caused only by the exogenous connection from s j 0 .
In contrast, if the unique component s j 0 is not exogenously connected to x k , then we can use this unique component to compute the total causal effect from x i 0 to x k (i.e., ˜ a ki 0 ), which in turn can be used to determine if there exist exogenous connections from the remaining components in ˜ s i 0 to x k .
Notice that an observed variable x i may have multiple unique components in the unique component set
U k
( i ) .
Suppose there exists another source s l 0 which is a unique component of x i 0 and is not exogenously connected to x k , while the unique component s j 0 of x i 0 is exogenously connected to x k .
Since we have no prior information about which of the two unique components is not exogenously connected to x k , there are two candidate models that we cannot distinguish between: In each of these two models, the total causal effect from x i 0 to x k is calculated using one of the two unique components in x i 0 , and there is an extra exogenous connection from the other unique component.
Thus none of the unique components in U k
( i ) can be exogenously connected to x k .
Note that having multiple unique components for a single observed variable does not contradict the irreducibility assumption.
This is because unique components are deﬁned over the source mixtures of the possible parents of x k , i.e., ˜ S k , and irreducibility assumption is deﬁned over the whole mixing matrix.
Case (ii).
Suppose there exists an index i 0 such that Comp(˜ s i 0 ) ∩ Comp(˜ s k ) (cid:54) = ∅ .
Let s j 0 ∈
Comp(˜ s i 0 ) ∩ Comp(˜ s k ) , i.e., s j 0 is exogenously connected to both x i 0 and x k , where x idoes not have unique components.
Note that (27)-(29) hold for this case as well; (27)-(29) do not depend on whether x i 0 has unique components.
When x i 0 has no unique components, and one of its components s j 0 is exogenously connected to x k , this results in a similar indistinguishably problem as in the previous case.
In particular, we are not able to distinguish between the part of s j 0 component in x k that results from the causal connection/path from x i 0 , and the part that results from the exogenous connection from s j 0 .
Thus, given ˜ W , we can not distinguish the true model from the two models in (30), (31).
The only difference from case (i) is that x i 0 does not have unique components.
In this case, we cannot have any of the components in ˜ s i 0 to be exogenously connected to x k .
In particular, all the shared (non-unique) components of ˜ s i 0 are pivotal to recover the total causal effect from x i 0 to x k and including any of them in ˜ s k will always result in indistinguishably of the generating model.
Case (iii).
Suppose the marriage condition is not satisﬁed for the collection ˜ S k .
Then as described in Appendix E.1, the matrix B k is not of full row rank, where B k is deﬁned as B k =
[ B ij ] i,j : x i ∈P k , s j ∈E k .
Given the recovered mixing matrix ˜ W , suppose that the exogenous connections to x k , i.e., ˜ s k , are recovered as in (26).
Then the remaining source components in x k that result from the causal connections/paths from the possible parents in P k can be written as
[Equation]
where { ˜ w kj } j : s j ∈ Comp( x k ) are the entries corresponding to x k in ˜ W (i.e., x k = (cid:80) s
j ∈ Comp( x k ) ˜
w kj s j ).
{ ˜ w (cid:48) kj } are also known, as they result from subtracting the components of ˜ s k (known by assumption) from x k .
Our task is to recover the total causal effect ˜ a ki from each possible parent x i ∈
P k to x k , which translates to ﬁnding the solution of (cid:88)
[Equation]
If we write each source mixture ˜ s i in terms of its source components, then (32) can be written as
[Equation]
[Equation]
(33) must hold for each source component s j ∈ E k , which means (cid:88) i
[Equation]
where ˜ a k = [˜ a ki ] i : x i ∈P k , and ˜ w (cid:48) k
= [ ˜ w (cid:48) kj ]
j : s j ∈E k .
Recall, B (cid:62) k is not of full column rank, hence (34) has inﬁnite number of non-zero solutions.
Denote these solutions as { ˜ a ( l ) k : l = 1 , 2 , · · · } .
For each l , we obtain a different generating model of x k as
[Equation]
[Equation]
We conclude that, given the mixing matrix, there exist inﬁnite number of generating models that we cannot distinguish from one another, i.e., the true generating model cannot be uniquely identiﬁed.
Appendix F. Other proofs F.1. Proof of equivalence between two representations of linear DS-P-SCM
In this subsection, we prove that linear DS-P-SCM can be equivalently written as a linear P-SCM or a linear DS-SCM.
That is, the submodel of linear P-SCM under distinct source assumption is equivalent to the submodel of linear DS-SCM under linear latent confounding and jointly independent sources.
First, we show that a linear DS-SCM with linear latent confounding and jointly independent sources can be written as a linear P-SCM with every observed variable associated with a distinct source.
Denote the vector of observed variables as X .
A linear DS-SCM with linear latent confounding and jointly independent sources can be written as
[Equation]
X l represents the vector of jointly independent latent confounders; S d represents the vector of jointly independent distinct sources; A represents the causal connection among observed variables;
A ol represents the linear latent confounding.
Note that each column in A ol contains at least two non-zero entries because it is a confounder.
Under acyclicity assumption, A can be converted to a strictly lower triangular matrix following the causal order among observed variables.
Hence, (36) is equivalent to linear P-SCM in (2), where the adjacency matrix is A , the exogenous connection matrix is B =
[ A ol I ] , and S =
[ X l ; S d ] .
Since I is a submatrix of B , this means that each observed variable is associated with a distinct source, i.e., a source that is not shared with any other observed variables.
Next, we show by construction that a linear P-SCM with distinct sources can be written as a linear
DS-SCM with linear latent confounding and jointly independent sources.
Consider a linear P-SCM,
X =
A X + B S .
A distinct source that is associated to only one observed variable corresponds to a column in B with only one non-zero entry.
Since each observed variable is associated with a distinct source, the number of these columns is exactly equal to the number of observed variables, and these columns can be permuted and scaled into an identity matrix.
The remaining columns in B must have two or more non-zero entries (same structure as A ol ), and hence each of these columns corresponds to a latent confounder where the confounding is linear.
Thus, the linear P-SCM reduces to (36).
This concludes the proof.
F.2. Proof of equivalence between two representations of linear P-SCM
In this subsection, we prove that linear P-SCM (cf.
(2)) can be equivalently written as (4), which is considered in most works studying linear DS-P-SCM.
That is, instead of allowing an observed variable to have no distinct source, we can require the observed variable to have a distinct source but the source can have zero variance.
Notice that (4) can be equivalently written as
[Equation]
where S o is the vector of distinct sources with non-zero variance.
B oo is composed of one-hot column vectors (i.e., binary vectors with exactly one non-zero entry) representing the correspondence between observed variables in X and their associated non-zero distinct sources.
If deterministic relation are not present (i.e., the distinct sources are all with non-zero variance), then B oo is identity, and (4) is the same as (37).
If some of the distinct sources have zero variance, then there are less sources in S o than observed variables.
In this case, we can always rewrite (4) as (37) by removing the sources in S d with zero variance, and remove the corresponding column vectors in I .
We can also rewrite (37) as (4) by adding a distinct source with zero variance for each observed variable that is not associated with a non-zero distinct source in (37), and remove B oo .
To show that (37) reduces to a linear P-SCM, we have
[Equation]
Assuming the model is acyclic, A can be permuted into a strictly lower triangular matrix, hence (38) reduces to a linear P-SCM.
Next, we show that a linear P-SCM can be written as (37).
Consider the linear P-SCM, X =
A X + B S , where some observed variables are not associated with distinct sources.
Recall from
Appendix F.1 that a distinct source that is associated with only one observed variable corresponds to a column in B with one non-zero entry.
However, the number of these columns is less than the number of observed variables, and hence these columns can not be permuted to identity.
Thus, we can write B as [ A ol B oo ] , where each column of B oo have exactly one non-zero entry, yet B oo have more rows than columns, and hence some of its rows are zero.
Columns of A ol have two or more non-zero entries and still represent linear latent confounding.
After we scale the non-zero entries in B oo to 1 (by changing the scales of the sources in S corresponding to these columns), the linear
P-SCM reduces to (38), which can be further translated to (37).
F.3. Proof of equivalence between two representations of the marriage condition
We ﬁrst show that the marriage condition, when deﬁned over the collection { Comp( x i ) : x i ∈ P k } , is equivalent to linear independence of the possible parents in P k in the mixing matrix W (cf.
Recall that, for each x i ∈ P k , Comp( x i ) includes the source components with indices corresponding to the non-zero entries in x i ’s row of W .
Consider only the rows of W corresponding to possible parents in P k .
The non-zero entries in these rows correspond to the existing source components, i.e., E k =
(cid:83) x
i ∈P k Comp( x i ) (cf.
Deﬁnition 8).
Let X k
=
[ x i ] x i ∈P k and S k
=
[ s j ] s j ∈E k .
Then we have
[Equation]
The following lemma shows the connection between the marriage condition and the rank of the submatrix W k .
Lemma 11 (Edmonds (1967)) Let W be an m × n matrix, where m ≤ n , such that any submatrix (with non-zero rows or columns) of an arbitrarily permuted version of W is of full rank.
Then, W has rank m if and only if for every subset of C rows, the corresponding submatrix of W has at least
C non-zero columns.
According to Lemma 11, the marriage condition on the collection { Comp( x i ) : x i ∈ P k } is equivalent to the matrix W k in (39) being of full row rank.
That is, by applying Lemma 11 to W k , for W k to be of full rank, every subset of C rows of W k (i.e., every C possible parents) must have at least C non-zero columns (i.e., C different source components).
W k is full rank means that each possible parent is linearly independent from the other possible parents in P k .
Note that the recovered mixing matrix ˜ W from BSS is a column-repermuted and rescaled version of W , which preserves the rank of the submatrices of W .
Thus the marriage condition is also equivalent to the linear independency among possible parents in the representation from BSS (as stated in Example 8).
Similarly, for each possible parent x i ∈ P k , Comp(˜ s i ) includes the source components with indices corresponding to the non-zero entries in x i ’s row of B .
Since each x i ∈ P k can be written as a linear combination of the source mixtures ˜ s in its parent set (plus ˜ s i ), all of which are included in ˜ S k
= { ˜ s i : x i ∈ P k } , we have E k =
(cid:83) x
i ∈P k Comp( x i ) =
(cid:83) ˜ s i ∈ ˜ S
k Comp(˜ s i ) .
Therefore, similar to (39), by considering only the rows of B that correspond to the possible parents in P k , the non-zero entries in these rows correspond to E k .
[Equation]
Once again, applying Lemma 11 to matrix B k , the marriage condition on the collection { Comp(˜ s i ) :
˜ s i ∈ ˜
S k } is equivalent to the full row rank of B k .
Note that the parents of x i ∈
P
k are also possible parents of x k (belong to P k ).
Thus, the rows of A that correspond to possible parents of x k have non-zero entries only in the columns of A that also correspond to the possible parents of x k .
Therefore, X k can be written as
[Equation]
where A k =
[ A ij ] i,j : x i ∈P k , x j ∈P k is the submatrix of A with both rows and columns corresponding to possible parents of x k .
Comparing (39) and (41), we have
[Equation]
Since A k can be permuted to a strictly lower triangular matrix, I − A k is of full rank.
Thus W k is of full row rank if and only if B k is of full row rank.
To conclude, the marriage condition deﬁned over the collection of the component sets of possible parents, is equivalent to the condition over the collection of the component sets of their source mixtures.
F.4. Proof of Theorem
We have mentioned in Section 4.1 that if each observed variable in the linear P-SCM is associated with a distinct source, then for each observed variable x k , the possible parent set is exactly the ancestor set of x k , and every possible parent x i ∈ P k has a unique component (i.e., U k
( i ) (cid:54) = ∅ ).
Besides, for each observed variable x i , the component set of its mixtures, i.e. Comp(˜ s i ) , includes a distinct source.
Thus, for any subset X C of the observed variables in X , (cid:12)(cid:12)(cid:12)(cid:83) i : x i ∈
X
C Comp(˜ s i ) (cid:12)(cid:12)(cid:12) ≥ | X C
| .
This implies that the marriage condition (Condition 2) holds for every observed variable x k , where we select the collection X C as any subset of the possible parent set P k .
Therefore, a reduced linear P-SCM could be uniquely identiﬁed from the recovered mixing matrix ˜ W if and only if for each observed variable x k , the unique components condition (Condition 1) is satisﬁed:
For every exogenous connection from s j to x k , it must belong to one of the two following cases:
• s j is a new component that is not exogenously connected to any of the possible parents of x k .
This means that no observed variables containing s j are possible parents of x k , thus there are no causal paths from any observed variable containing s j to x k .
• s j is connected to some possible parents in P k .
Condition 1 implies that s j cannot be a unique component of any x i ∈ P k , and it must be exogenously connected to at least two observed variables x i 1 , x i 2 in P k , where i 1 (cid:54) =
i 2 .
Since P k is the ancestor set of x k , there are at least two distinct causal paths x
i 1 (cid:32) x k and x i 2 (cid:32) x k from the observed variables with s j in their source mixtures (i.e., exogenously connected to s j ) to x k .
Remark
In a graphical representation of the reduced linear P-SCM, the condition in Theorem 6 can be expressed as follows.
Let us deﬁne a causal cycle as an undirected cycle in the graph, where one directed edge on this cycle is from a source variable s j to an observed variable x k , and the remaining edges of the cycle form a directed path from s j to x k .
We call the source s j and the observed variable x k as the source and sink of the causal cycle.
Then the reduced linear P-SCM is uniquely identiﬁable if and only if for each causal cycle in the graph, there exists at least one other causal cycle that shares the same source and sink (including the exogenous connection between them).
A simpler representation of the condition (which is not generally necessary and sufﬁcient) is that:
The generating model is not uniquely identiﬁable if the graphical representation includes a causal cycle, and none of the edges on this cycle is shared with other cycles in the graph.
F.5. Proof of Remark
For the sake of completeness, we state the following theorem.
Lemma 13 (Theorem 16, Salehkaleybar et al.
Let des o ( x i ) be the observed descendant set of variable x
i in a linear DS-P-SCM with latent confounders (including x i itself if it is observed).
Then the linear DS-P-SCM is uniquely identiﬁable from observations (i.e., the total causal effect between any two observed variables can be identiﬁed), if for any observed variable x i and any latent variable x k , des o ( x i ) (cid:54) =
des o ( x k ) .
As we explained in Appendix F.1, a latent confounder in the linear DS-P-SCM corresponds to a shared source that is connected to at least two observed variables in the representation of linear
P-SCM (with distinct source).
In the following, we ﬁrst show that the condition in Theorem 6 implies the condition in Lemma 13.
Note that the observed descendant set of a latent confounder and that of an observed variable must be different if both variables are not causally connected.
Therefore, it sufﬁces to show that if the condition in Theorem 6 is satisﬁed, then for a source s j that is exogenously connected to at least two observed variables including x k , des o ( s j ) (cid:54) =
des o ( x k ) .
According to Theorem 6, since s j is exogenously connected to x k , one of the following situations must hold:
• There are at least two distinct causal paths from observed variables (which are exogenously connected to s j ) to x k .
Suppose x i is an ancestor of x k and is exogenously connected to s j .
Then we have x i ∈ des
o ( s j ) , and x i (cid:54)∈ des o ( x k ) .
Thus des o ( s j ) (cid:54) =
des o ( x k ) .
• There are no causal paths from observed variables containing s j to x k .
Recall that s j is exogenously connected to at least one observed variable other than x k .
Denote x l as the observed variable exogenously connected to s j that has the smallest index in the causal order among observed variables (other than x k ).
That is, all observed variables that are exogenously connected to s j must follow x l in the causal order except for x k .
In this case we have x l ∈ des o ( s j ) .
In the following we show that x l (cid:54)∈ des o ( x k ) , which means that des o ( s j ) (cid:54) =
des o ( x k ) , and completes the proof.
The proof follows by contradiction.
Suppose x l ∈ des o ( x k ) .
This means that there is a causal path from x k to x l .
Now consider x l .
Since s j is connected to both x k and x l , according to Theorem 6, there must exist another causal path from an observed variable (exogenously connected to s j ) to x l .
However, such an observed variable must precede x l in the causal order, which contradicts the fact that x l has the smallest index among all observed variables connected to s j other than x k .
Therefore x l (cid:54)∈ des o ( x k ) .
To show the second part of the remark, i.e., the condition in Lemma 13 does not imply the condition in Theorem 6, we provide an example which satisﬁes the condition in Lemma 13 but does not satisfy the condition in Theorem 6.
Hence, the generating model in this example is uniquely identiﬁable among all linear DS-P-SCMs, but is not uniquely identiﬁable among all linear P-SCMs.
Example 9 Consider the following linear DS-P-SCM with three observed variables x 1 , x 2 , x 3 and a latent confounder x L :
[Equation]
The graphical representation of this DS-P-SCM is shown in the plot on the left in Figure 8.
The observed descendant sets of all four variables are
des o ( x L ) =
{ x 1 , x 2 , x 3 } ; des o ( x 1 ) = { x 1 , x 3 } ; des o ( x 2 ) = { x 2 } ; des o ( x 3 ) = { x 3 } .
x L xxxss L ssxxxss L ssxxxss L ss
Therefore, all four variables have distinct observed descendant sets, which satisﬁes the condition in
Lemma 13.
Hence this linear DS-P-SCM is uniquely identiﬁable.
The linear DS-P-SCM in (43) can be equivalently written into the following linear P-SCM:
[Equation]
The graphical representation of this linear P-SCM is shown in the plot in the middle in Figure 8.
This model violates the condition in Theorem 6: s L is exogenously connected to x 3 , but there is only one other path from an observed variable containing s L to x 3 (i.e., x 1 → x 3 ).
Thus, this linear P-SCM is not uniquely identiﬁable, and an alternative linear P-SCM is shown in the plot on the right in Figure 8.
The generating models of x 1 and x 2 in this alternative P-SCM are the same as in (44) , but the generating model of x 3 is
[Equation]
s 1 + s 3 . Notice that this linear P-SCM cannot be represented as a linear DS-P-SCM, since there is no distinct source associated with x 1 .
Hence it does not affect the unique identiﬁability of the generating model in the class of linear DS-P-SCMs.
Appendix G. Numerical experiments details
We generate synthetic models according to a linear P-SCM with p observed variables and rp source variables, where r is the ratio of source to observed variables.
Each pair of observed variables is connected with probability d e / ( p − 1) , and each pair of source and observed variable is connected with probability d
o /p .
d e (resp.
d o ) is the average number of causal (resp. exogenous) connections for an observed (resp.
a source) variable.
That is, on average, each observed variable is causally connected to d e other observed variables, and each source variable is exogenously connected to d o observed variables.
We refer to r , d e , and d o as the source to observed node ratio, average observed node degree, and average source node degree, respectively.
The strength of each connection is drawn is an increasing order ( ; 3 Repermute derived : p do satisfy Assumption veriﬁcation steps 10, 15, 16 yield true for all k , the model satisﬁes our conditions. independently from a uniform distribution over the interval [ − 1 , − 0 .
5] ∪ [0 . 5 , 1] .
We randomly permute both the observed and source variables to hide the true causal order after model generation.
G.1. Satisﬁability of the conditions
To test the restrictiveness of Conditions 1 and 2, i.e., the unique component and marriage conditions in Sections 3.1 and 3.2, we propose Algorithm 2 which veriﬁes whether the generating model satisﬁes these conditions.
Given matrices A and B of a linear P-SCM, Algorithm 2 ﬁrst computes the true mixing matrix W , and use it to retrieve the possible parent set for each observed variable x k .
Subsequently, the algorithm implements the same iterative procedure as in Deﬁnition 3 to verify that the unique components condition is satisﬁed.
From Appendix E.1, the marriage condition can be equivalently applied to the set of possible parents with no unique components, i.e., I k .
The marriage condition then translates to: For any
X C ⊆
I k , | X C | ≤ | (cid:83) i : x i ∈
X C Comp(˜ s i ) | .
Note that, Algorithm 2 only veriﬁes the satisﬁability of this condition for the subset X C
= I k .
Thus, the condition veriﬁed in Algorithm 2 is weaker than the marriage condition; yet it is much more efﬁcient to verify.
Indeed, we do not observe much difference in the results when we only verify the marriage condition for X C
= I k , as opposed to verifying the condition for all X C ⊆
I k .
Satisﬁability of P-SCMs.
We ﬁrst test the satisﬁability of the conditions with different source to observed node ratios and different average degrees.
We select p = 10 , 15 observed variables, average observed node degree d e = 1 . 5 , 2 , and average source node degree d o = 1 , 1 . 5 .
Under each setting, we set the source to observed node ratio r to be ranging from 0.8 to 2.6, with a step size of 0.2.
We report the number of generated models (attempts) required to obtain a model that satisﬁes our conditions.
We repeat the experiment 100 times, and calculate the average number of attempts (to ﬁnd a satisfying model) and the standard deviations.
The common logarithms (base 10) of average # attempts are shown in Figure 9.
The shaded area around each curve represents the logarithm of average # attempts plus/minus half the standard deviation.
We observe that when there are less or equal number of sources than observed variables, our conditions are unlikely to be satisﬁed.
This is due to the high probability of overlap among the exogenous connections, i.e., the same source is connected to multiple observed variables.
This leads to a high probability of the model violating the unique component condition.
When the ratio r increases, the sources are less likely to be connected to multiple observed variables, hence the probability of satisﬁability increases, converging to a ﬁxed value.
Now, let us compare the # attempts for different average degrees within each subplot ( p =and p = 15 ).
We observe when r is large (greater than 2), increasing the observed node degree, d e , is more likely to result in complex structures (among observed nodes).
Further, increasing the source node degree, d o , implies that more sources are shared among observed variables.
Thus, increasing d e and d o increases the # attempts required to obtain a satisfying model.
When r is small (less than 1), increasing d e again results in a complex structure.
However, since the number of source variables is small, a small d o implies that observed variables are more likely to have either no exogenous connections, or a few that are shared among them.
Subsequently, when r is small, the average # attempts required to generate a satisfying model is large for large d e and small d o .
This is shown by the green line in both subplots.
We conclude that the satisﬁability of our conditions depends on:
The existence and similarity of exogenous connections among observed variables, 2.
The ratio of overlap among exogenous connections (i e., how many sources are shared among observed variables), 3.
The structure of the causal graph among observed variables.
Satisﬁability of DS-P-SCMs.
To further demonstrate our ideas, we test the satisﬁability of the conditions when each observed variable in the generating model has a distinct source that is not shared with any other observed variable.
As discussed in Section 4, in this case, the model is a linear
DS-P-SCM.
We select p = 10 , 15 , 20 observed nodes, d e = 2 , 4 , and d o = 1 . 5 , 2 .
We report the average and the standard deviation of the # attempts to generate 100 satisfying models.
Figureshows the logarithm of the # attempts for the ratio r ranging from 1 to 2.6.
When r = 1 , the model can be exactly represented as a linear DS-P-SCM without latent confounding.
Indeed, all models generated under this setting satisfy our conditions (satisﬁability probability is equal to 1).
Further, we observe that the # attempts needed to obtain satisfying models increases as r increases.
Also, it is more likely for sparser graphs (small d e and d o ) to satisfy our conditions than denser graphs (large d e and/or d o ).
These observations are aligned with our aforementioned conclusions.
G.2. Performance of recovery
Uniquely identiﬁable cases.
We test the performance of our P-SCM recovery algorithm (Algorithm 1) for models generated according to each of the following settings:
(1) P-SCM with equal number of observed variables and sources, and ( d e , d o ) =
We refer to this setting as P-SCM_Equal .
(2) P-SCM with 2 less sources than observed variables, and ( d e , d o ) = (1 . 5 , 1 . 5) .
We refer to this setting as P-SCM_Fewer .
(3) DS-P-SCM with 3 more sources than observed variables, and ( d e , d o ) = (2 , 1 . 5) .
We refer to this setting as DS-P-SCM .
We select only the generated models that satisfy our conditions, and hence are uniquely identiﬁable.
The sources are independent and identically distributed according to a uniform distribution over the interval [ − 0 .
We generate n = 1000 samples of observations.
For P-SCM_Equal and P-SCM_Fewer , we use FastICA (Hyvarinen, 1999) for blind source separation, and for SCM , we use ReconstructionICA (Le et al., 2011).
We prune the mixing matrix using bootstrapping method (Efron and Tibshirani, 1994).
We ﬁrst generate n boot =
50 bootstrap samples by sampling from the observed data.
Then, for each bootstrap sample, we use ICA method to deduce estimates ˜ W
( i ) , i = 1 , · · · , n boot of the mixing matrix.
Recall that each estimate suffers from permutation and scaling indeterminacies.
For each estimate ˜ W ( i ) , we normalize each column by the entry with the largest absolute value on that column.
Further, we permute the columns of each ˜ W ( i ) , i = 2 , 3 , · · · , n boot such that the Frobenius distance between the permuted ˜ W ( i ) and ˜ W (1) is minimized.
Lastly, we compute the average of the bootstrap estimates (samples) of the mixing matrix, and prune this average by applying a Student t-test element-wise in order to reject the entries with small variance around zero.
In particular, we keep each entry with 95% conﬁdence level.
The recovered mixing matrix ˜ W is the resulting matrix after pruning the average of the bootstrap estimates.
Notice that we assume that the number of sources is known for
ICA recovery.
We evaluate the performance of ICA recovery by comparing the recovered mixing matrix ˜ W and the actual mixing matrix W , and call the ICA recovery successful when ˜ W has the same non-zero entries (support) as W (after column permutations).
That is, ICA learns the exact structure of W from observed data.
We evaluate the performance of our P-SCM Recovery algorithm using the recovered mixing matrix ˜ W from ICA, and compare the performance with four other algorithms: ICA-LiNGAM(Shimizu et al., 2006), DirectLiNGAM (Shimizu et al., 2011), Pairwise lvLiNGAM (Entner and
Hoyer, 2010) and ParceLiNGAM
(Tashiro et al., 2014), whenever applicable.
These baseline methods (i) are all based on LiNGAM (or lvLiNGAM) and (ii) can recover the strengths of causal connections between observed variables, i.e., matrix A .
We implement all baseline methods using the codes released by their corresponding authors, and select default hyperparameters throughout the simulation.
A well-known class of methods for causal structure learning in the presence of latent confounders is FCI algorithm (Spirtes et al., 2000) and its variants.
However, we do not compare our approach with these methods since these methods returns a PAG, which is a class of Maximal Ancestral Graphs (MAGs), and our approach mostly focuses on unique identiﬁcation of the ground truth (i.e., DAG).
First of all, since our hypothesis space of possible graphical models (linear P-SCMs) is different from the class considered by FCI (DS-SCMs), the returned PAG may not contain the ground truth even if FCI does not make any mistakes in the estimation, and hence we do not believe that comparison is fair.
Secondly, we note that in a MAG, two variables can be adjacent even if in the ground truth
DAG they are not.
Therefore, it is not straightforward to deﬁne a measure of performance between a
MAG and the ground truth DAG.
Third, we note that the size of the PAG returned by FCI can be exponentially large and hence, even if we have an evaluation method for each MAG, still computation time can make the comparison infeasible.
We replace the step of solving an overdetermined linear system (Step 10 in Algorithm 1) by ﬁnding its least-squares approximation, in order to improve stability.
We prune the recovered matrices A , by removing the edges with causal strength | a ij | < 0 .
1 for all methods including ours.
We compare the differences between the recovered adjacency matrix, and the true adjacency matrix
A , using the following four test metrics:
(1) SHD / Edge; the normalized structural Hamming distance divided by the number of edges in the true model; (2) Frobenius Norm; (3) “Precision”:
The fraction of edges in the recovered model that appear in the true model; (4) “Recall” (true positive rate): The fraction of edges in the true model that are correctly recovered.
Note that for SHD / Edge and Frobenius Norm, lower values mean better performance.
For Precision and Recall, higher values mean better performance.
We also evaluate the performance of our P-SCM Recovery algorithm for recovering the matrix
B , up to permutation and scaling indeterminacies, i.e., ˜ B .
We normalize both B and ˜ B such that the entry with the maximum absolute value on each column is one.
We then permute the columns of ˜ B by minimizing the Frobenius distance:
[Equation]
where P is a permutation matrix.
We compare the difference between ˜ B ∗ and B using the same four metrics, and compare these results with the recovery of A .
We use P-SCMR_A and P-SCMR_B to represent the recovery of matrices A and B by our algorithm.
We repeat the simulation until 50 models can be successfully recovered by ICA, and report the average of the aforementioned four metrics.
Figures 11 - 13 show the performance of recoveries for all algorithms under Settings (1)-(3) for model generation, with the number of observed variables ranging from 5 to 10.
We compare the performance for (i) all ICA recoveries, and (ii) only the successful recoveries by ICA.
We observe that for P-SCM_Equal and P-SCM_Fewer , our algorithm can learn the model more accurately than the existing algorithms.
This is because existing algorithms may misinterpret the shared sources among observed variables as confounding or direct causal connections.
We observe from the plots for SHD/Edge and Precision that this misinterpretation may add additional edges to the recovered graph, which do not exist in the true model.
For the third setting, DS-P-SCM , the performance of our method is comparable to existing algorithms.
Besides, we observe that our algorithm performs signiﬁcantly better than other algorithms
when
ICA recovery is successful, both in learning the structure of matrices A , B , and in learning the causal strengths (the coefﬁcients).
As discussed earlier, our algorithm assumes that the true mixing matrix
W can be correctly recovered up to permutation and scaling of its columns.
Thus, our algorithm depends on the accuracy of BSS recovery.
Non-uniquely identiﬁable cases.
We evaluate the performance of our algorithm when the generating model does not satisfy the necessary and sufﬁcient conditions for unique identiﬁability, i.e., Conditions 1 and 2.
Since the generating model cannot be uniquely recovered, there exist multiple generating models that correspond to the same mixing matrix W .
However, since our conditions are imposed separately on every observed variable x k , our algorithm can recover part of the generating model where the conditions are satisﬁed.
Here, we consider P-SCM_Equal setting (the number of observed variables is equal to the number of sources).
We only select the generating models that do not satisfy our conditions and refer to the modiﬁed setting as P-SCM_NonUniq .
Figure 14 shows the recovery performance for our algorithm, compared with the baseline algorithms.
Not surprisingly, for P-SCM_NonUniq , our algorithm does not perfectly recover the generating model, even when ICA recovery is successful (cf.
Figure 14).
However, our method outperforms the other algorithms in learning the structure and the causal strengths.
G.3. Performance on real data
We test the performance of our recovery algorithm on the daily closing prices of the following ﬁve world stock indices from 01/01/2015 to 06/30/2020: (1) Dow Jones Industrial Average (DJI) in USA, (2) Nikkei 225 (N225) in Japan, (3) Euronext 100 (N100) in Europe, (4) Hang Seng Index (HSI) in
Hong Kong, and (5) the Shanghai Stock Exchange Composite Index (SSEC) in China.
We only select the days where all ﬁve stock markets are active; we have T = 1192 days for our observations.
This data is obtained from Yahoo ﬁnance database 11 .
The same setting, but with different data (different dates) has been considered in (Salehkaleybar et al., 2020, Section 5.2).
Let c i ( t ) be the closing price of the i -th index on day t .
We deﬁne the corresponding return of an index i ∈ { DJI, N225, N100, HSI, SSEC } , at day t as R i ( t ) =
( c i ( t ) − c
i ( t − 1)) /c
i ( t − 1) , for all t = 2 , 3 , · · · , T .
In this setting, we consider the corresponding return of an index i to be an observed variable.
We suppose these observed variables were generated according to a linear P-SCM with ﬁve sources, where the sources are non-Gaussian random variables.
We apply FastICA with bootstrapping for source separation from observations.
The recovered mixing matrix ˜ W is    0 . 9096 0 .
1805 − 0 . 0 .
2048 1 .0 .0 .
  , (45) where the ﬁve rows correspond to DJI, N225, N100, HSI and SSEC respectively.
Using P-SCM
Recovery algorithm, the directed graph among these observed variables can be recovered as in Figure 15.
For a directed acyclic graph (DAG), let the DAG-source node be the node with no parents, and the DAG-sink node be the node with no children.
We observe from both ˜ W and Figure 15 that DJI is a DAG-source, with the fewest number of source components among all other indices.
Further, we observe that DJI, N225 and N100 all have causal effects on HSI.
Both observations are known to be true from common belief in economy, and from previous results in (Hyvärinen et al., 2010; [Figure]
Salehkaleybar et al., 2020).
One main difference in our result from the result in (Hyvärinen et al., 2010) is that our algorithm cannot detect the directed edge from SSEC to HSI.
This is because ˜ W in (45) has the same components for HSI and SSEC, and our algorithm concludes that there are no causal relation between these two observed variables.
This may happen when the exogenous connections for these two variables are fully overlapped, as in this example.
Recall that our linear P-SCM is advantageous when there are more sources than observed variables.
However, in this example, we are assuming a number of sources equal to the number of observed variables.
This is mainly so that we are able to use FastICA, which provides a highly accurate estimate for the mixing matrix (when a ground truth is available).
The existing overcomplete
ICA methods on the other hand are not as accurate.