A Multivariate Causal Discovery based on Post-Nonlinear Model
: Bernhard Sch¨olkopf, Caroline Uhler and Kun Zhang
Abstract
Understanding causal relations of systems is a fundamental problem in science.
The study of causal discovery aims to infer the underlying causal structure from uncontrolled observational samples. One major approach is to assume that causal structures follow structural equation models (SEMs), such as the additive noise model (ANM) and the post-nonlinear (PNL) model, and to identify these causal structures by estimating the SEMs.
Although the PNL model is the most general SEM for causal discovery, its estimation method has not been well-developed except for the bivariate case.
In this paper, we propose a new causal discovery method based on the multivariate PNL model.
We extend the bivariate method to estimate multi-cause PNL models and combine it with the iterative sink search scheme used for the ANM.
We apply the proposed method to synthetic and real-world causal discovery problems and show its effectiveness.
1. Introduction
Understanding causal structures of systems is a fundamental question in science and has been studied in various fields such as biology, economics, and social science (Rhein and Strimmer, 2007;
Londei et al., 2006; Morgan and Winship, 2014; Moneta et al., 2013).
While conducting randomized experiments is the most effective way to identify causal structures, it is often impossible for ethical, technical or cost reasons (Spirtes and Zhang, 2016).
Therefore, it is important to develop causal discovery methods that infer causal structures from uncontrolled observational data.
Using structural equation models (SEMs) is one major approach in causal discovery.
In SEMbased methods, causal structures are assumed to follow parameterized causal models and inferred by estimating the parameters from data.
An important property of causal SEMs is identifiability that is a theoretical guarantee that true causal structures can be uniquely identifiable if the joint distributions of data are known.
Constructing flexible identifiable models and then developing their estimation methods are the main focuses in SEM-based study.
Various identifiable SEMs and their estimation methods have been proposed so far.
The linear non-Gaussian acyclic model (LiNGAM) (Shimizu et al., 2006) is one of the most studied models.
In
LiNGAM, a causal structure is represented as a directed acyclic graph (DAG) and each effect variable is generated as the linear combination of cause variables followed by an additive unobserved noise that is independent of the causes.
There has been proposed an efficient estimation method named DirectLiNGAM
(Shimizu et al., 2011; Hyv¨arinen and Smith, 2013).
While LiNGAM is a simple and well-studied model, it may suffer from degradation in complex real-world applications due to the linear assumption.
The additive noise model (ANM) is proposed as a more flexible model that can handle nonlinear causal relations (Hoyer et al., 2009).
In ANM, cause variables are transformed nonlinearly and then added to an unobserved noise independent from the causes.
ANM is proved identifiable except for some trivial cases and an estimation method based on regressions, named regression with subsequent independence test (RESIT), is proposed (Hoyer et al., 2009; Peters et al., 2014).
As a more realistic model, Zhang and Hyv¨arinen (2010) proposed the post-nonlinear (PNL) model inspired by real-world data observation processes.
In the PNL model, cause variables are transformed and added to a noise variable in the same manner as ANM and then transformed again by a nonlinear invertible function.
The second transformation represents the sensor or measurement distortion, which is frequently encountered in practice.
Although the PNL model is one of the most flexible identifiable causal SEMs, which covers ANM and LiNGAM, its estimation methods are not well-developed except the bivariate case (Zhang and Hyv¨arinen, 2009;
Zhang et al., 2015; Uemura and Shimizu, 2020; Tu et al., 2021).
Uemura and Shimizu (2020) proposed an estimation method of the bivariate PNL model and a causal discovery procedure based on it, named autoencoder-based causal discovery for the PNL model (AbPNL).
AbPNL represents the two nonlinearities with neural networks and estimates them by minimizing two loss terms corresponding to two assumptions of the model: the independence of the cause and the noise and the invertibility of the second nonlinearity.
The losses are defined as Hilbert-Schmidt independence criterion (HSIC)
(Gretton et al., 2005) and a reconstruction loss, respectively, and minimized directly by a stochastic gradient descent method.
Given samples of two variables, AbPNL estimates the models of the two candidate causal directions and infer the true direction as the one that fits the model better.
While AbPNL is easy to use owing to no assumptions on neural network structures and the straightforward optimization procedure, it is limited to the bivariate cases and cannot be applied to multivariate cases unless an exhaustive search is performed, which is usually infeasible in practice.
In this paper, we propose a new method for multivariate nonlinear causal discovery based on the
PNL model.
First, we extend the estimation of the bivariate PNL model of AbPNL to the multi-cause model with multiple cause variables and a single effect variable.
Then, we iterate the multi-cause model estimation to identify the causal order of the variables in the same manner as the ANM case in RESIT.
Finally, we prune redundant edges of the causal DAG constructed from the causal order to obtain the true causal structure.
We also show the theoretical validity of the proposed procedure in the multivariate PNL model.
We apply the proposed method to synthetic and real-world datasets and show its effectiveness.
2. Preliminaries 2.1. Multivariate Causal Discovery
In this paper, we address the following multivariate causal discovery problem.
Let d variables X := { x 1 , x 2 , . . .
, x d } with a density p ( x 1 , x 2 , . . .
, x d ) have causal relations represented as a directed graph with an adjacency matrix A :
= [ a i,j ] d × d ∈ { 0 , 1 } d × d , where x j is a direct cause of x i if and [Figure] only if a i,j = 1 .
Our goal is to estimate A from observations D := { ( x ( s ) 1 , x ( s ) 2 , . . .
, x ( s ) d ) } s =1 ,...,n sampled i.i.d.
from p ( x 1 , x 2 , . . .
, x d ) .
We assume that the causal graph is a directed acyclic graph (DAG), i.e. there is no cycle of directed edges on the graph.
2.2. Post-Nonlinear Causal Model
Zhang and Hyv¨arinen (2010) proposed the post-nonlinear (PNL) causal model taking account of real-world data generating processes.
In the PNL model, variables are generated as follows:
[Equation]
where X pa i is a set of direct cause variables, or parents, of x i and e i is an unobserved noise variable.
The inner function f i : R | X pa i | → R represents nonlinear mixture of the causes and the outer one g i : R → R represents nonlinear distortion of sensors that often observed in real-world situations.
The noises are independent of each other, e i ⊥ e j ( ∀ i ̸ = j ) , and thus, a noise is independent of each of the corresponding causes, e i ⊥ x j ∈ X pa i .
The distortion g i is assumed to be invertible.
The PNL model is identifiable except for some special combinations of the functions and the noise distributions (Zhang and Hyv¨arinen, 2009; Peters et al., 2011, 2014).
The PNL model is known as the most general identifiable causal model and is equivalent to LiNGAM when f is linear and g is the identity and ANM when g is the identity.
2.3. Autoencoder-based causal discovery for PNL model (AbPNL)
Uemura and Shimizu (2020) proposed a bivariate causal discovery method named AbPNL.
AbPNL assumes that a causal relation follows the bivariate PNL model.
Given two variables ( x 1 , x 2 ) ,
AbPNL estimates two models corresponding to two candidate causal directions, x 1 → x 2 and x 1 ← x 2 , and infers that the one with the better model fit is the correct direction based on the identifiability of the model.
In the estimation process of a model, AbPNL uses neural networks to represent the functions and trains them by minimizing a loss consisting of two terms corresponding to two assumptions of the model: the independence between the cause and the noise and the invertibility of the distortion.
Suppose two variables { x, y } have the causal direction x → y , and consider the estimation of the bivariate PNL model,
[Equation]
Figure 1 shows the network structure for the estimation.
By the invertibility of g , the noise in (2) can be written as
[Equation]
AbPNL uses neural networks ˆ f and ˆ h for f and g − 1 in (3), respectively.
The first loss term encodes the independence between x and e , which is measured by the empirical estimator of Hilbert-Schmidt independence criterion (HSIC) (Gretton et al., 2005).
Let k and l be Gaussian kernel functions for x and e , respectively, and { ( x ( s ) , y ( s ) ) } s =1 ,...
m be the given samples, the loss is defined as
[Equation]
where e ( s ) :
= ˆ h
( y ( s ) ; ϕ ) − ˆ f ( x ( s ) ; θ ) , K :=
[ k ( x ( s 1 ) , x ( s 2 ) )] m × m and L :=
[ l ( e ( s 1 ) , e ( s 2 ) )] m × m are Gram matrices, H :
= I − m − 1 11 T , and 1 is an m -dimensional vector of ones.
HSIC takes a non-negative value and zero if and only if two variables are independent.
The second loss term encodes the invertibility of g .
By representing g with a neural network ˆ g as well as g − 1 , the second loss term is defined as the reconstruction error of y ,
[Equation]
where ˜ y ( s ) :
= ˆ g (ˆ h ( y ( s ) ; ϕ ); ψ ) is a reconstructed sample of y ( s ) .
The final loss function is defined as
[Equation]
where λ ∈ (0 , 1) is a balancing weight.
3. Proposed Method
In this section, we propose a new multivariate nonlinear causal discovery method based on the PNL model.
Our basic idea is to extend the estimation method of the bivariate PNL model in AbPNL to the multi-cause model and estimate the multivariate model by following the principle for the multivariate ANM used in RESIT (Peters et al., 2014).
In the following sections, we first develop the estimation method of the multi-cause PNL model that has multiple causes and a single effect by modifying the bivariate one in AbPNL.
Then, we introduce the procedure to identify the causal order of variables by iterating multi-cause estimations and the theoretical validity.
Finally, we describe the pruning method to construct a causal graph from the causal order.
3.1. Estimation of the multi-cause PNL model
We extend the bivariate estimation procedure in AbPNL described in Section 2.3 to the multi-cause
PNL model defined as
[Equation]
input : Samples of d variables X = { x 1 , . . .
, x d } output : Causal order π
R ← { 1 , . . .
, d } , π ← [] repeat for j ∈ R do
Estimate (7) with causes { x i } i ∈ R \{ j } and effect x j .
q j ← degree of model fit end j ∗ ← argmax j q j π ← [ j ∗ , π ] R ← R \ { j ∗ } until | R | = π ← [ R (1) , π ] where X is a set of causes, y is an effect, and e is an unobserved noise independent of ∀ x ∈ X .
The difference from the bivariate model in (2) is the number of causes.
Therefore, we can use most parts of the bivariate algorithm except the function f and the first loss term L 1 .
We deal with the former simply by increasing the input dimension of the network
ˆ f to | X | .
For the latter, the loss should quantify the degree of dependence between the noise and each cause.
To achieve this, we define the multi-independence loss term as
[Equation]
Although other aggregation schemes such as mean and median could be used, we employ the maximum to make all HSIC values decrease evenly.
Note that our modification is the generalization of the original bivariate estimation, which is in the case of | X | = 1 .
3.2. Identification of causal order
To identify the causal order of variables, we follow the iterative procedure proposed by Peters et al.
While the procedure is introduced for the multivariate ANM, it is also valid on the multivariate PNL model by the following theorem.
Theorem 1 Suppose variables
X follow a multivariate PNL model on DAG G .
Then there exists a multi-cause PNL model with an effect x ∈ X and causes X \ { x } if and only if x is a sink on G .
The “if” part is obvious from the definition.
The converse can be easily verified in the same manner as the ANM case (see Peters et al., 2014, A. 15) 1 .
Based on the theorem, we can find a sink variable on the true causal graph with estimations of multi-cause PNL models.
Algorithm 1 shows the pseudocode for identifying the causal order.
Given d variables X , we first estimate d possible multi-cause models with ( d − 1) causes.
Then, we select the effect of the model that satisfies the PNL assumptions the best as a sink variable and exclude it from the variable set.
By iterating these steps, we identify the causal order.
input : Samples of d variables X = { x 1 , . . .
, x d } and causal order π output : Adjacency matrix A
= { a i,j } d × d ∈ { 0 , 1 } d × d
A ← O for k ∈ { 2 , . . .
, d } do i ← π ( k ) , C ← π (1 : k − 1) if | C | ≥ 2 then for j ′ ∈ C do
Estimate (7) with causes { x j } j ∈ C \{ j ′ } and effect x i .
if the model satisfy the assumptions then
C ← C \ { j ′ } /
* Remove the edge x j ′ → x i .
*/ end end else if x i is independent of x C (1) then
C ← ∅ /* Remove the edge x C (1) → x i .
*/ end end for j ∈ C do a
i,j ←end end
We use residual independence loss values L 1 for evaluating the goodness of model fit.
To reduce the effect of the overfitting, we calculate the values from test samples that are not used for the model estimation.
Although the reconstruction loss L 2 should also be considered for fair evaluation, it showed a small enough value in almost all the preliminary experiments.
Therefore, we pay attention to the independence loss, which is more difficult to minimize 2 .
3.3. Pruning of redundant edges
To remove redundant edges on a DAG constructed from the identified causal order, we apply the greedy pruning procedure (Peters et al., 2014). Algorithm 2 shows the pseudocode. The overall flow is the same except that we modify the method of determining to prune to suit the PNL model. Considering the causal DAG in which each variable is a direct cause of variables with later causal order, we greedily remove unnecessary edges on each of its multi-cause substructures.
The greedy pruning procedure is done as follows.
On a multi-cause substructure, we search for a cause x j ′ such that the other causes { x j } j ∈ C \{ j ′ } and the effect x i still follow the PNL model if the cause is removed.
If such a cause exists, we remove the corresponding edge and repeat the procedure until all the causes are checked.
[Figure]
When there are one or more causes, we evaluate the goodness of model fit by checking whether the independence assumption holds, as in Section 3.2.
However, in contrast to the relative evaluation in the sink identification, we use absolute criteria.
It is because if a model y =
g
( f ( x )+ e ) holds, the model with another variable y =
g ′ ( f ′ ( x, z ) + e ′ ) also holds.
It may cause the failure in removing z in practice if we use the relative measure such as a difference of loss values.
Therefore, we use the independence test of HSIC (Gretton et al., 2008) between the estimated noise and the excluded cause.
When there is no cause, we use the test between the effect and the excluded cause.
4. Experiments
In this section, we conduct numerical experiments and show the effectiveness of the proposed method on synthetic and real-world multivariate causal discovery problems.
4.1. Synthetic data
To investigate the empirical performance of the proposed method, we generated synthetic problems with the ground truths and evaluated the performance of the proposed method.
We randomly generated causal structures with linear and nonlinear causal relations.
For each problem, we first [Table] constructed a causal DAG with d nodes by generating each edge with the probability 3 of 2 / ( d − 1) , and allocated the variables for the nodes randomly.
We used d = 4 in this experiment.
Then, for a linear problem, we assigned linear causal relations, x i = (cid:80) x j ∈ X pa i β ij x j + e i with uniformly chosen coefficients β ij ∼ U ( − 1 , 1) .
For a nonlinear problem, we generated the PNL model in (1), where f i and g i were sampled as the weighted sums of Gaussian processes and sigmoid functions, respectively.
We adjusted scales of variables so that signal-to-noise ratios become one to avoid extreme cases.
We generated 100 linear and 100 nonlinear causal structures and, on each structure, 2000 samples from uniform noises with the zero mean and the unit deviation.
Figure 2 shows examples of linear and nonlinear problems with their ground truth DAGs.
We followed the settings used in Uemura and Shimizu (2020) including the hyperparameters and structures of neural networks.
We used a half of samples for the model estimation and the rest for the evaluation of model fit.
In the identification of a sink, we estimated each model for t = 9 times and used the median values of L 2 of test samples over trials that achieve L 1 < 10 − 3 .
Similarly, in the pruning, we remove an edge if L 1 < 10 − 3 achieved and the independence test ( p = 5% ) passed in at least one estimation out of t .
As compared methods, we used DirectLiNGAM (Shimizu et al., 2011) and RESIT (Peters et al., 2014) with linear, GAM and GP regressions.
Table 1 shows the results.
As performance measures, we used the edit distance and the number of reversed edges,
[Equation]
where A :=
[ a i,j ] d × d ∈ { 0 , 1 } d × d and B :=
[ b i,j ] d × d ∈ { 0 , 1 } d × d are an estimated adjacency matrix and its ground truth, respectively.
While d edit represents the performance of the overall estimation, N rev measures that of the causal order identification more directly.
On the linear problems, all methods show good performance with d edit < 1 .
Intuitively, DirectLiNGAM achieved the best and the performance decreases as the assumed models become more complex.
Although the proposed method took the largest edit distance, the difference from the best is 0 . 6 , which we think is still practically good enough.
From the value of N rev , we can find that RESIT and DirectLiNGAM estimated correct causal orders and their misestimation is mainly sim sim d edit N rev d edit N rev Proposed
RESIT(GAM)
DirectLiNGAM    due to the pruning process.
On the other hand, the proposed method failed in estimating the causal order on some problems than other methods.
On the nonlinear problems, the result shows the opposite trend.
DirectLiNGAM and RESIT with the linear regression deteriorated because of the violation of their linear assumption.
By virtue of the nonlinearity, RESITs with GAM and GP regression improved the performance.
The proposed method improved the performance of DirectLiNGAM by 2 . 28 and RESIT by 1 . 05 in terms of the edit distance.
Considering that the expected number of edges in each problem is d = 4 , we conclude that the improvement is significant.
4.2. fMRI simulation data
To evaluate the proposed method in a more realistic scenario, we applied it to simulated fMRI data (Smith et al., 2011).
We used sim1 and sim2 datasets with 5 and 10 variables, respectively.
Each dataset consists of 50 time series and each series has 200 data points, which results in 10000 data points in total.
We used randomly chosen 5000 points and used 1000 as test samples in the proposed method.
Table 2 and Figure 3 show the results.
For RESIT, we only show the result of the GAM regression that outperformed the others.
While all the three methods are comparable in terms of N rev , the proposed method showed the best performance in d edit .
These facts and the estimated DAGs in the figure suggest that the proposed method works correctly not only in the causal order identification but also in the pruning procedure.
4.3. General Social Survey data
Finally, we applied the proposed method to General Social Survey data used in Shimizu et al.
and analyzed the result on a real-world problem.
This sociological data consists of 1380 samples and each sample has 6 variables: Father’s education, Father’s occupation, Number of siblings, Son’s education, Son’s occupation and Son’s income.
It is challenging to estimate their causal relations because they may violate the model assumption such as the independence of noises.
To obtain results with higher reliability, we estimated a DAG 10 times with bootstrapping (Efron and Tibshirani, 1993) and output the edges that appeared in at least half of all estimations.
We set the number of bootstrap samples to 600 .
Figure 4 shows the estimated DAG.
Although the true causal structure is unknowable, the proposed method estimated the direction from father to son, which is the only undeniable true direction.
Moreover, the direction from son’s occupation to son’s income seems to be reasonable.
On the other hand, there are directions contrary to our intuition such as the one from occupation to education. In
[Figure] [Figure] the case where the model assumptions are violated, a more detailed analysis of the proposed method is necessary to ensure robustness, which we think is important future work.
5. Discussion
In this section, we discuss the properties of the proposed method, specifically the computational complexity and the stabilization.
In the proposed method, we use neural networks to represent models and estimate them by optimizing their parameters using stochastic gradient descent methods.
Since each neural network represents a simple scalar function, it is unnecessary to be a complex design.
Therefore, the train- ing time is much shorter than the general deep models such as image recognition and language processing.
On the other hand, the number of estimations grows exponentially as the number of variables increases.
Given n variables, the identification of causal order requires n − 1 iterations.
In each iteration, we compare k models for the remaining k variables.
Therefore, a total of n + ( n − 1) + · · · + 2 = n ( n − 1) / 2 − 1 model estimations are required, which is the same as RESIT and DirectLiNGAM.
Additionally, the proposed method trains one model several times to stabilize results.
Consequently, the total number of training becomes t ( n ( n − 1) /
2 − 1) , where t is the number of training trials per one model.
Similar to the identification of causal order, the pruning procedure also requires an exponential number of estimations.
For the worst case, where all the edges will be pruned, a total of t ( k ( k − 1) / 2 − 1) estimations are required for each multi-cause substructure with k cause variables 4 .
The number of estimations increases as the ground truth DAG becomes sparse.
One effective approach to alleviate the computational time is parallelization.
Since all the estimations in one iteration of the identification of causal order can be performed separately, we can reduce the time by parallelizing them.
If enough computational resources are available, we can identify the causal order with n − 1 serial iterations.
Similarly, the time of the pruning can be reduced by the same manner.
In the experiment in Section 4, we used this parallelization.
We additionally applied the proposed method to synthetic problems with n = 15 and confirmed that it requires roughly one day for one problem on a server capable of (maximum) 40-threading 5 .
Estimating one model multiple times in the proposed method has a high impact on the total computational time.
Therefore, we think improving the stability is an important issue that should be addressed not only for improving the performance of causal discovery but also for reducing the computational time.
One of the typical approaches is to change the optimizing algorithm, while there is a trade-off between stability and speed.
Another possible option is to add a regularization term to the loss.
To use this idea, we need to consider carefully the impact of changing the loss on using its value when comparing models in the sink identification.
In this paper, we proposed a new multivariate nonlinear causal discovery method based on the PNL model.
We extended the bivariate estimation method to the multi-cause one and combined it with the iterative schema for identifying the causal order.
We then applied the greedy pruning method to construct a resulting DAG.
We showed the effectiveness of the proposed method with synthetic and simulated fMRI data and analyzed the result on the real-world sociological data.
We discussed the computational cost of the proposed method and suggested parallelization as one alleviation. A more fundamental improvement of stability is an important issue that needs to be addressed in the future. As suggested in Section 4.3, a comprehensive analysis of the impact of model violation is also necessary to ensure its performance on real-world applications. Appendix A. Proof of Theorem
In the following, we show detailed sketch of the proof of Theorem 1.
For more strict proof, see (Peters et al., 2014). For “if” part, let x be a sink variable and X ′ be the other variables. From the definition of the multivariate PNL model (1), there exists a PNL model such that x = g ( f ( X ′ )
+ e ) , where e ⊥ x ′ ∈ X ′ .
Note that non-parent variables of x are ignored in terms of the function f .
For “only if” part, suppose there exists a PNL model with an effect y that is not a sink on the true DAG G .
Since y is not a sink, y has children on G .
Let z be a sink on the subgraph consisting of all children of y and D be descendants of z .
Since D does not have children of y ,
D ⊥ y | S ∪ { z } holds, where S = X \ { y, z } ∪ D . Therefore, from the assumption, there exists a PNL model y = g ′ ( f ′ ( S, z ) + e ′ ) .
On the other hand, since z is a child of y , there exists a
PNL model z = g ( f ( S, y ) + e ) .
Given the situation in which S was observed, this contradicts the identifiability of the bivariate PNL model.
To investigate the applicability of the proposed method on a larger problem, we applied it to synthetic problems used in Section 4.1 with n = 15 . We used the same parameter settings of both the problem generation and the proposed method, which are designed and tuned based on smaller problems. We conducted the experiments on Intel Xeon E5-2690 v4 2.60GHz servers with a maximum of 40 parallelizations.
The application to one problem was finished in about one day.
Table 3 shows the average results.
While DirectLiNGAM outperforms the others on linear problems, the proposed method shows better performance on nonlinear problems.
Note that since the parameters of the proposed method were tuned on smaller problems, we need additional experiments and analysis for a more comprehensive evaluation.