Multivariable Causal Discovery with General Nonlinear Relationships
Abstract
Today’s methods for uncovering the causal relationship(s) from observational data either constrain the function class (linearity/additive noise) or the data.
We make assumptions on the data to develop a framework for Causal Discovery (CD) that works for general non-linear dependencies.
Similar to previous work, we use nonlinear Independent Component Analysis (ICA) to infer the underlying sources from the observed variables.
Instead of using conditional independence tests to determine the causal directions, we rely on the Jacobian of the inference function; thus, generalizing LiNGAM’s approach to the nonlinear case.
We show that causal models resolve the permutation indeterminacy of ICA and prove that under strong identifiability, the inference function’s Jacobian captures the sparsity structure of the causal graph.
We demonstrate that our method can infer the causal graph on multiple synthetic data sets.
INTRODUCTION
Traditional statistical learning methods model correlations in data.
Though they have achieved super-human performance in multiple fields [53, 12, 49], they have limited value in understanding cause-effect relationships.
A prevalent consequence of this shortcoming is the observed tendency for models to learn shortcuts [6] (e g., classifying objects based on their backgrounds).
Conversely, causal models [40] construct the world according to the Independent Causal Mechanisms (ICM) principle [42], where building blocks (mechanisms) neither influence nor inform each other.
Modeling temperature T and altitude A is a classic example [42]: changing A affects T , but not vice versa.
This independence translates to the Directed Acyclic Graph (DAG)
A → T .
Causal Discovery (CD) describes the process of extracting causal structure from data in the form of a DAG.
Having interventional data–such as in the form of Randomized Controlled Trials (RCTs)–is desirable as it enables answering questions of interventional nature, such as ’What will happen if variable X is changed?’.
However, RCTs can be costly, infeasible [4], or even unethical.
Thus, developing effective CD methods reliant on observational data alone is of significant interest.
In general, inferring the causal direction is provably impossible without additional constraints or assumptions [61]; therefore, existing methods constrain either the model class (i.e., the functions generating the observations) or the data distribution.
On the model side, these constraints include linear [48, 52, 46, 62] or specific nonlinear relationships (e.g., with additive noise) [13, 44, 59, 47, 28, 38].
On the data side, assumptions include non-stationarity [35] or exchangeability [10].
CD aims to infer the ground-truth cause-effect relationships, which connects it to the identifiability literature, where the goal is to learn a model equivalent to the ground truth (up to indeterminacies, such as permutations or element-wise nonlinearities).
An extensively studied method for learning identifiable representations is Independent Component Analysis (ICA)
[2, 18], which requires that the inferred components (sources) are independent.
Recent work has relied on
NonLinear Independent Component Analysis (NLICA) [63, 15, 15, 57, 23, 20, 37, 35, 24, 8, 16, 19, 11, 29] for identifiability.
Our work builds on Monti et al.
[35], which showed that NLICA can be used for CD with general nonlinear functions and observational data.
Instead of using pairwise independence tests, we draw inspiration from the Linear
Non-Gaussian Acyclic Model (LiNGAM) [48], which uses a weight matrix to infer the DAG of a linear causal model.
We extend this approach to the nonlinear case by showing that the Jacobian of the inference function (mapping
X 1 = f 1 ( N 1 )
X 2 = f 2 ( N 1 , N 2 )
X 3 = f 3 ( N 1 , N 2 , N 3 )  
ˆ Nˆ Nˆ N 
J f − 1 =   ∗  ■ ∗  ▲ ∗  
InferenceModel f −
N
N N
X X
X
Equation be
Jacobian of model that identifies N under certain assumptions on N from observations X to noise variables N ) captures the sparsity structure of the DAG, when strong identifiability is fulfilled [24, Def.1].
Relying on the Jacobian improves scalability, since it removes the cost of d 2 independence tests for a DAG with d nodes.
We train our model with
NLICA, and show that the DAG underlying the Data
Generating Process (DGP) provides an inductive bias to account for the permutation indeterminacy of NLICA.
Our contributions can be summarized as follows:
We show that causal models allow us to account for the permutation indeterminacy of ICA; 2.
We prove that we can infer the DAG from the Jacobian of the inference function and also improve scalability by removing the need for independence tests.
We propose a multivariable CD method for general nonlinear functions from observational data; 4.
We experimentally show that our proposed method can infer the DAG across multiple synthetic data sets.
BACKGROUND
Here, we describe causal models and connect their estimation to ICA.
We defer technical details to Appx.
A.
Structural
Equation
Models (SEMs).
Given ddimensional observed X =( X 1 , . . .
, X d ) and noise (independent) variables
N =(
N 1 , . .
. , N d ) , their causal relationship is given by d deterministic functional assignments [42],
[Equation]
where P a i ⊂ X are the parents of X i and f i are the components of the vector-valued function f .
We describe the computation of X for a given N with an iterative process (denoting the iteration step with a superscript), which is a useful concept for justifying our proposal (§ 3).
Initially, N is drawn from its density.
To calculate X for
N , the functional assignment f needs to be applied
d times.
Namely, according to (1), each X i requires that its parents
P a
i are calculated.
After sampling N , only the (empty) parent sets of root nodes are calculated.
Thus, the first application of f yields the X i values for such nodes.
In the second iteration, the children of root nodes can be calculated (since we have all parents from the first iteration), and so on.
This yields an iterative algorithmic formulation of the SEM, describing the computational graph given by the DAG as:
[Equation]
where X 0 is the initial value (w.l.o.g., we assume X
= 0 , since calculating the functional assignments will overwrite every X i ).
As in most previous works [55, Table 1], we assume no confounders (all variables are observed) and faithfulness (loosely speaking, the coefficients/functions will not cancel an edge, cf.
Assum.
A.1).
Causal Discovery (CD).
In CD, the data is assumed to be generated by a causal process, and the aim is to infer the corresponding DAG, which enables reasoning about interventions (without the DAG, the joint distribution p ( N ) only admits observational queries) [42, 41].
Algorithmic approaches include combinatoric search [48, 13, 14, 21, 34, 43, 50, 55], continuous optimization [62, 30, 56, 39, 55], and neural networks [59, 38, 25, 58, 7, 22, 55, 27, 36]—we focus on the latter.
Zhang et al.
[61] proved that identifying the causal direction in a general SEM is impossible without constraints on the function class and/or data distribution.
Functional constraints can include linear [48, 62], additive nonlinear ( X i = f i ( P a i ) + N i ) [13, 38, 28, 44], or affine nonlinear ( X i
=
f i ( P a i ) + h i ( N i ) )
[25, 47] models.
Regarding the data distribution, some models require access to interventions [1, 45, 31]; others assume that N is Gaussian [22, 28] or non-Gaussian [48]; or require non-stationarity [35], exchangeability [10], or discreteness [45] of N .
Our work was inspired by [35], which provides a bivariate CD method for general nonlinear functions and non-stationary data.
The authors leverage recent results in NLICA (cf.
next section for details) to identify the causal direction.
Although they demonstrate applicability to multivariable problems, the use of pairwise independence tests constrains scalability.
In this work, we extend these results with a more scalable, end-toend solution.
For this purpose, we draw inspiration from
LiNGAM [48].
Assuming that the inference model learns to map observations to latents (i.e., it “inverts" the SEM), we illustrate how the weight matrix is used to extract the DAG for a linear SEM in the following example.
Example 1 (Motivating example for linear SEMs) .
Assume a linear causal model with three variables, the DAG
X 1 → X 2 → X 3 , and functional relationships: X 1 = N 1 ;
X 2 = aX 1 + N 2 ; X 3 = bX 2 + N 3 : a, b ∈ R \ { 0 } .
The
DGP generates samples according to the DAG and has the matrix form on the left—we focus on the elements below the main diagonal as for recovering the DAG, only the paths (i.e., series of directed edges) between X i and X j are required and the main diagonal expresses the N i − X i edges.
Inverting the DGP with an inference model (i.e., expressing N i as a function of X j ) yields the matrix on the right with elements below the main diagonal capturing the
DAG’s
X i − X j edges (as shown by color coding):
[Equation] [Equation]

[Equation]
N
[Equation]
N
[Equation]
N
[Equation]

 =     − a   − b   
X X
X 
This is the motivation for LiNGAM to infer the DAG from a weight matrix [48]—we use the same insight in the nonlinear case (cf. § 3) on the Jacobian of the inference model.
As the Jacobian is a local property, we will reason about the absolute value of the maximum Jacobian, where the maximum is taken over the input space.
DAG equivalence.
To justify using the Jacobian of the inference network f − 1 , akin to LiNGAM’s use of a weight matrix, we first connect the DAG and J f − 1 via fundamental concepts from graph theory.
The adjacency matrix A of a graph with d nodes is a binary d × d matrix where each matrix element indicates the presence, or absence, of an edge between a pair of nodes X i , X j (Defn.
A.4).
The connectivity matrix of a graph with d nodes is a binary d × d matrix where each matrix element indicates the presence, or absence, of a path (i.e., series of directed edges) between two nodes X i , X j (Defn. A.5).
For DAGs, both A and C are strictly lower-triangular —this is why we considered only the elements below the main diagonal in Ex. 1.
The inference network (cid:98) f − 1 generally differs from the true inverse of f up to indeterminacies (e.g., scaling, permutation, sign flips, element-wise transformations) [18, 23, 63].
Furthermore, the main diagonal of J f − 1 has non-zero elements (Ex. 1).
Thus, we describe the relationship between
J (cid:98) f − 1 and ( I d − A ) for a DAG via structural equivalence , and investigate its symmetries ( ◦ denotes composition):
Definition 1 ( ∼ DAG ) .
Two matrices S , R are structurally equivalent if ( S ) ij =
⇐⇒ ( R ) ij = 0 : ∀ i, j ; with the properties:
(i) D -invariance: a non-singular diagonal matrix D preserves the sparsity structure; thus, ( D ◦ S ) ∼ DAG S (ii) h 0 -invariance: for zero-preserving transformations h 0 : ( h 0 ( S )) ij =
⇐⇒ ( S ) ij = 0 then h ( S ) ∼ DAG S (iii) π -equivariance: a permutation π affects the positions of zeros; thus, both operands need to be permuted with the same π to maintain ∼ DAG ,
i
e., S ∼ DAG R ⇐⇒ ( π ◦ S ) ∼ DAG ( π ◦ R ) , (iv) Transitivity: S ∼ DAG P
∧ P ∼ DAG R =
⇒ S ∼ DAG R (v) Commutativity: S ∼ DAG R
⇐⇒ R ∼ DAG S .
S ∼ DAG R thus implies the matrices have the same sparsity structure.
Thereby, if S and R are adjacency matrices, they describe the same DAG.
Identifiability and ICA.
Independent Component Analysis (ICA)
[2, 18] models the observed variables X as a mixture of independent variables N via a deterministic function f , and focuses on defining models that are identifiable —i.e., N can be recovered up to indeterminacies (e.g., scaling, permutation, sign flips, element-wise transformations).
Since this is provably impossible in the nonlinear case without further assumptions [3, 17, 32], recent work has focused on incorporating auxiliary variables [20, 8, 23, 5], exploiting temporal structure in the data [16, 15, 11, 37, 35, 19, 26, 63], or restricting the model class [48, 13, 60, 9] .
Several works have related (nonlinear) ICA to SEM estimation [9, 35, 48, 54] by inverting the
DGP—i.e., estimating f − 1 with an inference model .
PROPOSED METHODS
We propose an extension of LiNGAM [48] to general nonlinear relationships.
We require strong identifiability [24,
Def.1] of the inference function f − 1 for extracting the
DAG via the Jacobian J f − 1 from observational data.
First, we observe that by assuming a DAG for the DGP, the permutation indeterminacy of ICA can be accounted for (cf.
Appx.
A.1 for the origin of the two permutations) —we then exploit this in Prop. 1 to prove that strongly identified models fulfil J f − 1 ∼ DAG ( I d − A ) .
Lemma 1 (DAG DGPs with unique π provide additional information for resolving the permutation ambiguity of ICA) .
When f describes a DAG, then the permutation indeterminacy of ICA π ICA can be resolved uniquely, even with unknown but unique causal ordering π .
Proof.
The unknown causal ordering π of N i implies the right-multiplication of J f − 1 with π − 1 , whereas the permutation indeterminacy of ICA implies the left-multiplication with π ICA , yielding the following estimated Jacobian:
[Equation]
where π ICA and π − 1 are not necessarily the same.
As
SEMs have a lower-triangular Jacobian and we assume that π is unique, this inductive bias on J f − 1 provides an unsupervised means to resolve π ICA and π − 1 and recover
J f − 1 from the estimated J (cid:98) f − 1 .
Relying on Lemma 1 and the properties of ∼ DAG , we prove that J f − 1 can be used to extract the DAG for general nonlinear functions (akin to the linear case shown in Ex. 1):
Proposition 1 ( J (cid:98) f − 1 ∼ DAG
( I d − A ) ) .
The inference network Jacobian J (cid:98) f − 1 is structurally equivalent to ( I d − A ) if f − 1 is strongly identified [24, Def.1] up to scalings, sign flips, permutations, and zero-preserving transformations.
Proof.
The proof consists of two steps: 1) leveraging the iterative formulation of the SEM (2), proving that
J f − 1 ∼ DAG ( I d − A ) and 2) relying on the properties of ∼ DAG and Lemma 1, showing J f − 1 ∼ DAG J (cid:98) f − 1 .
We start by formulating J f (recall that X
= X d ) based on the iterative SEM expression (2):
[Equation]
where A describes the X i − X j edges in the DAG (i.e.,
A ∼ DAG A ), B is diagonal (as the X d − 1 values are fixed) and both A , B are independent from t (superscript).
Realizing that (4) gives us a recursive formula, and recalling that X 0 = 0 , we can unroll (4) iteratively for t =
d − 1 , d − 2 , . .
[Equation]
where the last equality expresses the sum of the geometric series with elements
A i (the sum is finite as A is strictly lower triangular).
By invoking the inverse function theorem, we can express J f − 1 :
[Equation]
J f − 1 ∼ DAG ( I d − A ) follows as A ∼ DAG A and B is diagonal (the invariance of ∼ DAG follows from Prop. 1(i)).
For proving that J (cid:98) f − 1 ∼ DAG
( I d − A ) , we need J f − 1 ∼ DAG
J (cid:98) f − 1 (Prop. 1(iv)), which requires us to account for all indeterminacies of strong identifiability: i) Prop. 1(i) accounts for scalings and sign flips; ii) Prop. 1(ii) for zero-preserving transformations; and iii) Prop. 1(iii) for permutations, which can be extracted as shown in Lemma 1.
Prop. 1 implies that we can extract the DAG when f − 1 can be strongly identified [24, Def.1]—i.e.
, we can reason about interventions (cf.
We note that if B = I d , then (9) describes Additive Noise Models (ANMs) [13], whereas when additionally A is constant, we recover LiNGAM [48].
Description of the algorithm for CD and determining π .
We propose a two-step approach for extracting the DAG from observational data (Alg. 1) for general nonlinear f :
Algorithm
Input: dataset D , network parameters θ , Sinkhorn networks S ICA , S π
Initialize θ while L CL not converged do sample batch from D calculate L CL update θ end while extract J (cid:98) f
[Equation]
while L π not converged do
[Equation]
update S ICA , S π end while
we estimate f − 1 with an inference model that ensures (strong) identifiability, 2.
we account for the ordering to resolve the permutation indeterminacy.
Regarding the second step, the training objective for learning the permutations in (3) is inspired by LiNGAM [48] and leverages the observation that in SEMs, the ground-truth
Jacobian J f − 1 is lower-triangular:
[Equation]
where S ICA , S π are doubly-stochastic matrices, ( K ) i ≥ j are the lower-, ( K ) i<
j the strictly upper-triangular elements of K , and α { d,l,u } > 0 .
L π encourages K to be lowertriangular by simultaneously: maximizing i) the sum of the main diagonal; ii) the lower-triangular part; while also iii) minimizing the stricly-upper triangular part of K .
EXPERIMENTS
Experimental setup.
To (strongly) identify the SEM (quantified by Mean Correlation Coefficient (MCC) [15]), we use contrastive NLICA [63] to estimate (cid:98) f − 1 , and satisfy the assumptions on the DGP underlying the proof of identifiability [63, Thm. 6]) accordingly: the latent space is a hyperrectangle in R d , the marginal p ( N ) is uniform, the conditional p ( ˜ N | N ) is Laplace, X is generated by a smooth and bijective mapping; and the contrastive loss uses the same metric as the conditional, which is L 1 for our case (Assum. B.1).
Our architecture for the inference model is the same MultiLayer Perceptron (MLP), as in [63] (Tab. 3).
To account for the permutation indeterminacies, we use two
Sinkhorn networks [33], which are differentiable models for learning doubly-stochastic matrices.
We observed that set-
ting the lowest d ( d − 1) / 2 elements to zero and converting the resulting K matrix to binary often helped the convergence of the Sinkhorn networks.
Moreover, instead using max to aggregate the different Jacobians over the batch, we found using the mean operator more stable in practice.
We experiment with three DGPs: i) linear and ii) nonlinear
SEMs (in the simple form of X = f ( W N ) , as well as iii)
MLPs with triangular weight matrices (as used in [35]).
In all cases, the nonlinear activations are leaky ReLUs (with a slope of 0 .
25 for the SEMs and 0 .
1 for the triangular MLPs).
Additionally, we ensure that the ordering of N i is unique (all cases), and that the DGP weights are ≫ 0 (for the SEM
DGPs) as otherwise we would be unable to distinguish weak connections from small elements in the Jacobian.
That is, the estimate of a weak connection could be the same order of magnitude as the estimate of a zero element due to finite numerical precision—we do not enforce this property for the triangular MLPs to compare to the results of [35], where such modification was not present.
For the SEM DGPs, we sample 6 different orderings and 5 seeds for each problem dimensionality { 3; 5; 8 } .
For the triangular MLP, we use d = 6 to compare to [35, Fig. 2] and vary the number of layers in the mixing.
We measure learning the correct ordering by the ordering accuracy ( Acc π )—i.e., ratio when S π inverts π .
We also report the accuracy ( Acc ) and the Structural
Hamming Distance (SHD) (we use 1e − 3 as the threshold in all scenarios) for inferring the edges of the DAG, as is standard practice in the literature[28, 35, 45, 55].
We use the linear and nonlinear SEM DGPs to showcase that our method can infer the DAG while also learning the correct ordering.
Then, we compare to the methods reported in [35], which unlike our proposal, assume that π is the identity.
Results.
Tab. 1 demonstrates that our method works almost perfectly in the linear case, whereas its performance is slightly worse in the nonlinear case in terms of accuracy,
SHD and MCC.
This means that most edges are inferred correctly and identifiability is achieved.
Nonetheless, accounting for both the ICA permutation indeterminacy and π degrades with increasing d .
Nonetheless, erroneous solutions resulting from optimization issues (the most frequent problem according to our observations) can be simply filtered out: in this case the doubly stochastic matrices usually do not converge to a permutation matrix.
Inspecting their elements or automatically rejecting such solutions based on their entropy is straightforward (permutation matrices have minimal entropy among doubly stochastic matrices, so higher entropy means to a suboptimal solution)
[Table]
Tab. 2 summarizes our results with the triangular MLP of [35].
Despite having small weights in the ground truth Jacobian J f − 1 , our method was able to infer most edges in the
DAG.
Importantly, the resulting accuracies are larger than for NonSENS [35].
Moreover, our method has the advantage of simultaneously inferring all edges based on the structure of J (cid:98) f − 1 —thus, it does not require d 2 pairwise independence
[Table]
test
[Table]
a DAG with d nodes.
[Table]
DISCUSSION
We introduced a two-step process to leverage strong identifiability for inferring the DAG of multivariable causal models with general nonlinear functions.
Our method uses the Jacobian of the inference function (mapping from observables to independent variables) and can be thought as a generalization of LiNGAM to the nonlinear case.
We prove that this Jacobian captures the sparsity structure of the DAG, and show that by working with causal models, we can resolve the permutation indeterminacy of ICA under certain assumptions.
Since we do not use conditional independence tests, but learn the causal ordering with Sinkhorn networks, our method provides an end-to-end solution for CD and avoids the cost of exponentially many independence tests.
We experimentally demonstrate that our proposal can infer the DAG in multiple synthetic data sets.
Limitations.
Our theory requires the guarantees of strong identifiability but not the use of a specific (NLICA) algorithm.
Though our experiments demonstrate that fulfilling strong identifiability is sufficient for CD, we do not vary the
NLICA algorithm.
Our method’s applicability is limited for inferring weak edges, similar to [48, 52, 46, 28].
Author Contributions
Wieland Brendel initiated the project.
Ferenc Huszár and
Patrik Reizinger derived the theory, Patrik Reizinger and
Yash Sharma ran the experiments and all authors wrote the paper.
Acknowledgements
The authors would like to thank Ricardo Pio Monti and Scott
W. Linderman for helpful correspondence.
Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no.
BR 6382/1-1.
Wieland Brendel is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645.
The authors thank the
International Max Planck Research School for Intelligent
Systems (IMPRS-IS) for supporting Yash Sharma and Patrik
Reizinger.
Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent
Systems (ELLIS) PhD program
A SEMS
Definition A.1 (SEM) .
A SEM describes causal relationships via a set of structural assignments [42]:
X i :=
f i ( P a i , N i ) , ∀
[Equation]
= { , . . .
, d } , (12) where X i are the endogenous, N i the exogenous/noise variables, P a i ⊆
X \{ X i } denotes the parent set of X i , I the set of indices and f i the mappings.
Definition A.2 (Reduced form of SEM) .
The reduced form of the SEM expresses all X i as a function of only the N i variables,
[Equation]
with the same notation as in Defn.
A.1, slightly abusing f i and denoting a subset of N by N i ⊆ N .
Definition A.3 (Causal ordering) .
The causal ordering π is a bijective automorphism on the index set I .
Namely, π : I → I so that
∀
X
i ̸
= X j , it holds that if π ( i ) < π ( j ) =
⇒ X
j ̸∈ P a i .
The definition means that only a node with a smaller index in π can be a parent of a node with a larger index.
Note that though X i can be a parent of X j , it is not necessary, but X j cannot be a parent of X i .
Multiple orderings may exist, e.g. if there are multiple X i so that they only have a single parent.
π helps to have a unique description of the edges in the graph.
Namely, if the edges are organized in the adjacency matrix A accoridng to π , then A will be strictly lower triangular.
Definition A.4 (Adjacency matrix) .
The adjacency matrix
A is a binary d × d matrix, where A ij = 1 ⇐⇒ X j ∈
P a i .
The rows of A are ordered by π ; thus, A is strictly lower-triangular.
A only describes the edges of the DAG, which gives the direct cause-effect relationships.
Nodes can be influence each other via paths (i.e., a set of directed edges that can be traversed between the two nodes), which can be described by the connectivity matrix C
Definition A.5 (Connectivity matrix) .
The connectivity matrix C is a binary d × d matrix, where C = 1 ⇐⇒ ∃ p :
X j → · · · → X i .
C
= (cid:80) dk
=1 A k .
The rows of C are ordered by π ; thus, C is strictly lower-triangular.
Assumption A.1 (SEM assumptions) .
We assume that the causal DGP fulfils:
(i) (1) describes a DAG (ii) N i are jointly independent (iii)
There are no hidden confounders (faithfulness/stability), i.e., all (iv) π is unique (v) Each f i is a homomorphism (but they can be general nonlinear functions)
Requiring a unique π is a simplifying assumptions that to avoid ambiguities when presenting results, so it is without loss of generality
Definition A.6 (DGP with known π ) .
The DGP is described by the SEM, when π is known.
I.e., the flow of information is: N SEM −−−→ X .
Definition A.7 (DGP with unknown π ) .
The DGP with unknown π is given by the SEM, and by a permutation matrix π (with a slight abuse of notation) applied to X .
I.e., the flow of information is: N SEM −−−→ X π −→ ˆ X .
Lemma A.1 ( J f ∼ DAG ( I d + C ) ) .
Given Assum.
A.1, the partial derivatives of f i w.r.t.
N j provide information about
C , as
[Table]
We emphasize that the derivatives are also non-zero in the case of indirect paths, i.e., when ∃ X i ∈ p :
i ̸
= k, l. Furthermore, the strictly lower triangular part of J f has the describes the same DAG as C –or equivalently,
J f ∼ DAG ( I d + C ) .
A.1 WHY ARE THERE TWO PERMUTATION INDETERMINACIES IN Lemma 1?
In this section, we elaborate on the need to account for two permutations in Lemma 1: besides the well-understood indeterminacy coming from NLICA [17], the unknown causal order of N i also implies a permutation (Defn.
A.7).
Namely, a SEM with unknown causal ordering can be described as i) applying the SEM equations, ii) followed by a permutation matrix π .
This implies a right-multiplication of J f − 1 with π − 1 to extract the original causal ordering.
Accounting for the causal ordering is, to the best of our knowledge, only found in [48].
Binary CD methods such as [35] alleviate this step as they work on an edge-by-edge basis.
Other non-ICA-base methods can also avoid this step since the DAG is invariant to changes in the causal ordering –meaning that reordering X i in the observation vector
X (cf.
Defn.
A.7) does not affect the edges of the graph.
However, to resolve the permutation indeterminacy of ICA, we need to account for the causal ordering, since only then can the Jacobian be lower-triangular.
Although extracting a lower-triangular Jacobian is easier to interpret and potentially better suited, e.g., as a building block of causal representation learning (since the causal ordering of N i is always the same), our method extracts the DAG even without resolving these indeterminacies.
B EXPERIMENTAL DETAILS
Assumption B.1 (NLICA assumptions) .
We assume the setting of [63], specifically that of Thm. 6, under which, an encoder which minimizes a contrastive loss was proven to estimate the noise variables (often referred to as "sources" in the ICA literature) up to a composition of input independent permutations, sign flips, and rescaling.
For completeness, we restate the assumptions below:
(i) the space of sources/latent/noise variables, is a convex body in R d , i.e. a hyperrectangle/cube.
(ii) p ( N ) , the marginal distribution, is uniform (iii) p ( ˜ N | N ) , the conditional distribution, is a rotationally asymmetric generalized normal distribution [51],
i
e. a Laplace distribution.
(iv) the observations are generated by a smooth, bijective (i.e., invertible) mapping (v) the contrastive objective uses the same metric as p ( ˜ N | N ) , i.e. L 1 for Laplace (cf.
[63, Def. 1]).
[Table]
C NOTATION
ACRONYMS
ANM Additive Noise Model
CD Causal Discovery
CL Contrastive Learning
DAG Directed Acyclic Graph
DGP Data Generating Process
ICA Independent Component Analysis
ICM Independent Causal Mechanisms
LiNGAM
Linear Non-Gaussian Acyclic Model
MCC Mean Correlation Coefficient
MLP MultiLayer Perceptron
NLICA NonLinear Independent Component Analysis
SEM Structural Equation Model
SHD Structural Hamming Distance
NOMENCLATURE α scalar field
D diagonal matrix
I d d -dimensional identity matrix
J Jacobi matrix
L CL contrastive loss function
L π regularizer for learning π
S Sinkhorn network
L loss function ∼ DAG structural equivalence d problem dimensionality
Causality
[Table]
X observation vector
A adjacency matrix of a SEMs C connectivity matrix of a SEMs f structural assignment in SEMs
I index set π causal ordering f a component of f