O PTIMAL TRANSPORT FOR CAUSAL DISCOVERY
A BSTRACT
To determine causal relationships between two variables, approaches based on
Functional Causal Models (FCMs) have been proposed by properly restricting model classes; however, the performance is sensitive to the model assumptions, which makes it difﬁcult to use.
In this paper, we provide a novel dynamical-system view of FCMs and propose a new framework for identifying causal direction in the bivariate case.
We ﬁrst show the connection between FCMs and optimal transport, and then study optimal transport under the constraints of FCMs.
Furthermore, by exploiting the dynamical interpretation of optimal transport under the FCM constraints, we determine the corresponding underlying dynamical process of the static cause-effect pair data.
It provides a new dimension for describing static causal discovery tasks while enjoying more freedom for modeling the quantitative causal inﬂuences.
In particular, we show that Additive Noise Models (ANMs) correspond to volume-preserving pressureless ﬂows.
Consequently, based on their velocity ﬁeld divergence, we introduce a criterion for determining causal direction.
With this criterion, we propose a novel optimal transport-based algorithm for ANMs which is robust to the choice of models and extend it to post-nonlinear models.
Our method demonstrated state-of-the-art results on both synthetic and causal discovery benchmark datasets
I NTRODUCTION
Determining causal relationships between two variables is a fundamental and challenging causal discovery task (Janzing et al., 2012).
Conventional constraint-based and score-based causal discovery methods identify causal structures only up to Markov equivalent classes (Spirtes et al., 2001), in which some causal relationships are undetermined.
To address this challenge, properly constrained functional causal models (FCMs) have been proposed.
FCMs represent the effect as a function of its cause and independent noise and can help identify the causal direction between two variables by imposing substantial structural constraints on model classes, such as additive noise models (ANMs)
(Shimizu et al., 2006; Hoyer et al., 2008) and post-nonlinear models (PNLs) (Zhang and Hyv¨arinen, 2009b).
While some of the models, such as PNLs, are highly ﬂexible, the constraints are still restrictive and difﬁcult to interpret and relax.
Inevitably, the performance of these methods is sensitive to model assumptions and optimization algorithms, especially in real-world applications.
To handle the mentioned issues, we consider FCMs from a dynamical-system view.
By augmenting a time dimension for the static causal discovery task, we interpret FCMs with dynamical causal processes under the least action principle (Arnol’d, 2013).
The new interpretation connects FCMs with a large class of models in dynamical systems.
It then provides more freedom to model causal inﬂuences, possibilities to derive new causal discovery criteria, and a potential direction to generalize causal models with identiﬁable causal direction.
In particular, we exploit the above idea by leveraging the intrinsic connection between FCMs and optimal transport.
Optimal transport is originally introduced by Monge (1781), which has been applied in a large range of applications, not only because it is a natural way to describe moving particles (Ambrosio et al., 2012) but also because of its recent improvement in the computational methods (Cuturi, 2013; Kolouri et al., 2019).
Recently, it has also been largely applied to generative models for measuring the distance of probability distributions (Arjovsky et al., 2017; Kolouri et al., 2018; Genevay et al., 2018).
Among different optimal transport deﬁnitions, the L 2 Wasserstein distance got extensive applications in statistics (Rachev and R¨uschendorf, 1998), functional analysis (Barthe, 1998), et al.
(McCann, 1997; Otto, 1997).
The dynamical formulation of the L 2 Wasserstein distance is introduced by Benamou and Brenier (2000) for relaxing the computational costs.
We ﬁnd that in the context of the dynamical formulation, FCMs can be connected with optimal transport.
Furthermore, with the dynamical interpretation of optimal transport, one can naturally understand
FCMs from a dynamical-system view, which makes it possible to derive new criteria to identify causal direction.
Moreover, it also enables us to develop practical algorithms with optimal transport for static causal discovery tasks without learning a regression model.
Our main contributions are:
Dynamical interpretation of FCMs in the bivariate case.
We provide dynamical interpretations of optimal transport under the constraints of FCMs
Furthermore, we introduce a time variable,
determine the underlying dynamical process under the least action principle (Arnol’d, 2013) for the static bivariate causal discovery task, and characterize properties of the corresponding dynamical systems (Sec. 3.1 and Sec. 3.2).
A criterion for determining causal relationships between two variables.
We study the corresponding dynamical systems of FCMs and prove that ANMs correspond to volume-preserving pressureless ﬂows.
Moreover, based on the divergence of their velocity ﬁelds, we propose a criterion for determining causal relationships and show that under the
identiﬁability conditions of ANMs it is a valid criterion for ANMs, which can be extended to PNLs directly (Sec. 3.2).
An optimal transport-based approach (DIVOT) for distinguishing cause from effect between two variables.
DIVOT inherits the advantages of one-dimensional optimal transport.
It can be computed efﬁciently and does not require independence tests, learning a regression model, or deriving likelihood functions for complicated distributions.
Experimental results show that our method is robust to the choice of models and has a promising performance compared with the state-of-the-art methods on both synthetic and real cause-effect pair datasets (Sec. 4 and Sec. 6).
P RELIMINARIES
Optimal transport: the underdetermined Jacobian problem.
We mainly follow the notations and the deﬁnitions of (Benamou and Brenier, 2000).
Suppose that two (probability) density functions, p 0 ( x ) and p T ( x ) where x ∈ R d , are non-negative and bounded with total mass one.
The transfer of p 0 ( x ) to p T ( x ) is realized with a smooth one-to-one map M : R d → R d .
The Jacobian problem is to ﬁnd M that satisﬁes the Jacobian equation , p 0 ( x 0 ) =
p T ( M ( x 0 ))
| det ( ∇ M ( x 0 ))
| , where x
T =
M ( x 0 ) , ∇ is the gradient in vector calculus, and det ( · ) denotes determinant.
This is an underdetermined problem as many maps can be the solutions.
A natural way is to choose the optimal one, e.g., the one with the lowest cost.
A common cost function is the L p Wasserstein distance.
L p Wasserstein distance and its one-dimensional closed-form solution.
The L p Wasserstein distance between p 0 and p T , denoted by W p ( p 0 , p T ) , is deﬁned by W p ( p 0 , p T )
p = inf M (cid:82)
| M ( x 0 ) − x 0 | p p 0 ( x 0 )
d x 0 , where p ≥ 1 (Kantorovich, 1948).
In this work, we mainly use the square of the L 2 Wasserstein distance, denoted by W 22 .
Moreover, the one-dimensional (1D)
L p Wasserstein distance has a closed-form solution, e.g., the 1D optimal solution of W 22 is
M ∗ =
P − 1 T ◦ P 0 , where P 0 and P T are the cumulative distribution functions for p 0 and p T , and “ ◦ ” represents the function composition.
In practice, the 1D optimal solution can be computed with the average square distance between the sorted samples from p 0 and p T (Kolouri et al., 2019).
Functional causal models.
FCMs represent the effect Y as a function f ( · ) of the direct cause X and independent noise E y , where function f describes the causal inﬂuence of X on Y , and E y is the exogenous variable/noise.
Without any additional assumption on the functional classes, the causal direction is not identiﬁable (Hyv¨arinen and Pajunen, 1999; Zhang et al., 2015a).
Roughly speaking,
[Figure]
because given variable pair ( X, Y ) , one can always construct Y = f
( X, E y ) and another different FCM, X = (cid:101) f
( Y, E x ) , such that both of them have independent “noise” (Hyv ¨ arinen and Pajunen, 1999; Zhang et al., 2015a).
Several works further introduce proper assumptions on model classes, which guarantees that the independence of cause and noise only holds in the causal direction, e.g.,
[Equation]
where g and h are nonlinear functions and h is invertible.
D YNAMICAL I NTERPRETATION OF F UNCTIONAL C AUSAL M ODELS
We ﬁrst show the connection between FCMs and optimal transport in Sec. 3.1.
In Sec. 3.2, we further elaborate the analogy between the optimal transport problem and the causal direction determination problem.
We then study the optimal transport under the constraints of FCMs, show the corresponding dynamical systems of FCMs, and characterize the properties of such systems.
3. A REFORMULATION OF F UNCTIONAL C AUSAL M ODELS
As introduced in Sec. 2, FCMs are used to approximate the true data generation process.
Given the
FCM, Y = f
( X, E y ) , we rewrite it in the vector form,
[Equation]
where x 0 , x T ∈ R 2 , their probability densities p 0 , p T ≥ 0 , and M : R 2 → R 2 .
As an analogy to the mass transfer scenario (Monge, 1781), we consider the samples of independent noise E x and
E y as the particles of materials and regard the map M in Eqn.
(3) as a special transformation of the independent noise samples.
As shown in Fig.
1, one can consider the data points are transferred from the original positions (which are unmeasured) in the plane E x – E y at time 0 to the observed positions in the plane X – Y at time T .
Such transformation considers the transfer as a dynamical process which moves the unmeasured independent noise x 0 =
[ E x , E y ] (cid:48) 1 and consequently leads to the observations x T
=
[ X, Y ] (cid:48) .
From the perspective of FCM-based causal discovery approaches, causal inﬂuences are represented by FCMs which represent the effect as a function of its direct cause and an unmeasured noise satisfying the FCM constraints :
(i) The map constraint: the values of X are determined by the values of its corresponding noise, i.e., X = E x , while the values of the effect depend on cause X and noise E y ; (ii) The independence constraint: the noise terms are independent,
i
e, E x is independent of E y .
Note that the optimal transport M ∗ with the minimal L p Wasserstein distance is not necessary to be the one in Eqn.
(3), because it has no information about the FCM constraints or the true data generation process.
In other words, given two sample sets of x 0 and x T , the couplings given by optimal transport are not necessary to be the ones generated from the ground-truth FCM.
D YNAMICAL INTERPRETATION OF FCM S : O PTIMAL TRANSPORT UNDER THE FCM CONSTRAINTS
In this section, we jointly consider the causality and the optimality of the maps in the Jacobian problem.
It provides both a causal sense of the transformation and a dynamical view of FCMs.
We ﬁrst recap the dynamical formulation of the L 2 Wasserstein distance, study such dynamical systems under the FCM constraints, and then show their properties under the FCM and ANM constraints.
Dynamical L 2 Wasserstein distance.
Benamou and Brenier (2000) formulate the L 2 MongeKantorovich problem as a convex space-time minimization problem in a continuum mechanics framework.
Fixing a time interval [0 , T ] , they introduce the concepts of the smooth time-dependent density ρ ( t, x t ) ≥ 0 and the velocity ﬁeld v ( t, x t ) .
When they are clear from context, we denote them by ρ and v .
Because we are considering the bivariate case, x t ∈ R 2 and v ∈ R 2 .
Then, they give the dynamical formulation of W 22 :
[Equation]
[Equation]
initial and ﬁnal conditions
[Equation]
the continuity equation:
[Equation]
where ∇· denotes the divergence in vector calculus.
They show that minimizing the objective function in the optimization problem (4) is equivalent to ﬁnding the dynamical system with the least action (Arnol’d, 2013) and prove that the solutions of (4) are pressureless potential ﬂows , of which the ﬂuid particles are not subject to any pressure or force and the trajectories are determined given their initial positions and velocities or given their initial and ﬁnal positions.
Suppose that M ∗ is the solution given by W 22 .
The corresponding ﬂows follow the time evolution equation ,
[Equation]
The time evolution equation shows that x t is a convex combination of x 0 and M ∗ ( x 0 ) and that the velocity ﬁelds do not depend on time.
As an analogy between the optimal transport problem and causal direction determination, the density ρ and the velocity v of moving particles can be considered as the probability density and the velocity of changing values of data points.
Moreover, the dynamical interpretation of the L 2 Wasserstein distance introduces a time variable and provides a natural time interpolation ρ ( t, x t ) of ρ 0 and ρ T together with the velocity ﬁeld v ( t, x t ) .
Similarly, we can also have the natural time interpolation p ( t, x t ) between p 0 and p T as well as the velocity ﬁeld v ( t, x t ) under the least action principle (Arnol’d, 2013), which is the dynamical interpretation of FCMs.
Dynamical L 2 Wasserstein distance under the FCMs constraints.
First, we introduce FCM constraints in the context of the dynamical L 2 Wasserstein distance.
According to the time evolution equation (5), we know that the velocity is fully determined by the initial and ﬁnal values of x t .
We ﬁrst consider FCM constraint (i).
For the initial and ﬁnal values of the cause, the value of its observation X is equal to its noise value E x in Eqn.
Consequently, denoting v = [ v x , v y ] (cid:48) , where v x and v y represent the velocities along the E x / X -axis and E y / Y -axis respectively as shown in Fi.g 1, we know that ∀ t ∈ [0 , T ] , x t ∈ R , v x ( t, x t ) = 0 .
As for FCM constraint (ii), it implies that the initial noise of cause and effect, corresponding to E x and E y
at the time 0 , are independent.
Therefore, we have the FCM constraints for the dynamical L 2 Wasserstein distance,
(I)
The map constraints : v ( t, x t ) =
[ v x ( t, x t ) , v y ( t, y t )] (cid:48) , ∀ x t , y t ∈ R , t ∈ [0 , T ] , v x ( t, x t )
= 0 ; (II)
The independence constraint : two random variables (cause and effect) at the initial time have the joint probability density function p 0 ( x 0 ) and they are independent.
Second, we characterize the properties of the dynamical L 2 Wasserstein distance under constraints (I) and (II).
According to Eqn.
(3), the form of M ∗ is determined as
[Equation]
otherwise, the FCM constraints will be violated.
Moreover, the W 22 under the FCM constraints can be computed with the one-dimensional W 22 as shown in Prop. 1 (the derivation is in App. D).
Proposition 1.
Under constraints (I) and (II) , the square of the L 2 Wasserstein distance between pand p T is
[Equation]
Furthermore, we consider the constraints of ANMs and characterize the corresponding dynamical systems for now, which can be directly extended to PNLs as mentioned in Sec. 4.2.
Based on constraints (I) and (II), we further introduce the ANM constraint , (III) the effect is the sum of noise and a nonlinear function of cause as deﬁned in Eqn.
Theorem 1 ( Zero divergence of the velocity ﬁeld ) .
Under constraints (I) and (II) , the dynamical systems given by the L 2 Wasserstein distance are pressureless ﬂows.
Further under ANM constraint (III) , they become volume-preserving pressureless ﬂows, of which the divergence of the velocity ﬁeld, v ( t, x t ) =
[ v x ( t, x t ) , v y ( t, y t )] (cid:48) , satisﬁes
[Equation]
where div is the divergence operator in vector calculus.
Thm. 1 determines the corresponding dynamical systems of FCMs and ANMs by analyzing their densities and velocity ﬁelds (the details of the proof are in App. D) and shows an essential property of the corresponding dynamical systems of ANMs.
The property indicates a potential criterion for causal direction determination, which the divergence of the velocity ﬁeld is zero everywhere in the causal direction, while it may not always hold in the reverse direction.
Next, we will verify the criterion rigorously, propose an algorithm based on it, and show the extension for the PNL cases.
C AUSAL DIRECTION DETERMINATION WITH OPTIMAL TRANSPORT
In this section, we deﬁne a divergence measure as a criterion for determining causal direction between two variables for ANMs.
Based on the criterion, we then provide an algorithm to identify the causal direction, named by the divergence measure with optimal transport (DIVOT).
4. D IVERGENCE MEASURE AS A CAUSAL DISCOVERY CRITERION
We ﬁrst deﬁne the divergence measure and then show that it is a valid criterion for identifying the causal direction in the bivariate case under the identiﬁability conditions of ANMs, as shown in Prop.
2 (the proof is in App. D).
Proposition 2 (Divergence measure as a causal discovery criterion) .
Deﬁne the divergence measure,
[Equation]
(8) where v = M ∗ ( x 0 ) − x 0 .
Suppose that constraints (I) , (II) , (III) , and the identiﬁability conditions of ANMs (Hoyer et al., 2008) are satisﬁed.
The divergence measure of the corresponding dynamical system satisﬁes D ( v ) =
if and only if X is the direct cause of Y .
Nevertheless, there are some challenges to compute the divergence measure.
For example, we need to solve a two-dimensional optimal transport problem and compute the derivative of a velocity ﬁeld for all samples.
Another challenge of computing M ∗ is that we in general have no information about x 0 .
Such issues are all solved with DIVOT.
4. P ROPOSED METHOD : DIVOT
We ﬁrst provide an overview of the algorithm, DIVOT, for determining the causal direction between two variables and then introduce the four steps to compute the divergence measure.
Overview of DIVOT.
Alg. 1 is based on the divergence-measure criterion for causal direction determination and determines causal direction between two variables.
Given data of X and Y , we want to infer whether X causes Y ( X → Y ) or Y causes X ( Y → X ).
We compute the divergence measure in both directions.
In practice, noise can have different variance in different applications, and the larger variance can lead to the larger measure value with ﬁnite samples.
So we normalize the variance-based measure value with the estimated noise variance.
And then the one with the smaller normalized measure value is the causal direction.
In App.
A, we provide the modiﬁed algorithms for including the independent case and the signiﬁcance of the results with bootstrapping.
θ
Output: X → Y or Y → X .
1 Def DIVOT( { ( x i , y i ) } , p ( E ; θ 0 ) ) : data, θ = { ( x i , y i ) } , θ
L X → Y = Div(data, p ( E ; θ ) ) data, θ = { ( y i , x i ) } , θ
L Y → X = Div(data, p ( E ; θ ) )
X
10 Def Div(
, p ( E ; θ ) ) :
( p (
; θ ) )
∗ ))
Noise data generation: the ﬁrst step of computing the divergence measure.
To compute the divergence measure, we need to know the velocity ﬁeld v as deﬁned in the time evolution equation (5).
It requires the couplings of the data of x 0 =
[ E x , E y ] (cid:48) and x T =
[ X, Y ] (cid:48) .
But in the bivariate causal discovery task, only the data of x T are given.
Therefore, as shown in Line 11 of Alg. 1, we ﬁrst deal with the issue due to the lack of the noise data of x 0 , denoted by { ( e ix , e iy ) } .
To obtain the noise data, we may assume a multivariate probability distribution of x 0 with the density p
( E x , E y ) and then sample data from it, represented by ( e ix , e iy ) ∼ p
( E x , E y ) .
Fortunately, due to the FCM constraints, we know that p
( E x , E y )
=
p 0 ( X, E y ) =
p ( X ) p ( E y ) .
So we only need to assume the probability distribution of E y and parameterize it with θ , denoted by p ( E y ; θ ) .
Suppose that the dataset of x T with N samples is given, denoted by { ( x i , y i ) } N .
We ﬁrst sample a data set of
E y with the sample size N , denoted by { e iy } N , e.g., in the experiments of this work, we use the simpliﬁed reparameterization trick,
[Equation]
where f noise θ is a monotonic function, and e source y is sampled from a standard normal distribution or a uniform distribution.
As the monotonic f noise θ can be very ﬂexible, we can represent ﬂexible noise distributions with monotonic neural networks as in (Huang et al., 2018).
Next, we randomly match the data of X , denoted by { x i } N , with { e iy } N , which gives the { ( x i , e iy ) } N as the data of x 0 .
Optimal transport ﬁnds the couplings of the observation and the generated noise.
Computing the divergence measure requires the couplings of the data of x 0 and x T because of v =
M ∗ ( x 0 ) − x 0 ; in other words, given the data of x 0 and x T , we need to solve a two-dimensional optimal transport problem, which gives M ∗ and the couplings as in Line 12 of Alg. 1.
According to
Prop. 1, solving the two-dimensional optimal transport problem under the FCM constraints is equivalent to solving one-dimensional optimal transport problems.
More speciﬁcally, the expectation in
Eqn.
(7) can be computed with the Monte Carlo estimator by using N samples of E x , denoted by { e ix } N , and then the two-dimensional W 22 in Eqn.
(7) is computed with N one-dimensional W 22 , i.e., W 22 ( p 0 , p T )
≈ 1 N (cid:80) i W
( p ( E y ) , p ( Y | E x = e ix )) .
Importantly, solving the one-dimensional optimal transport problem or computing the one-dimensional W 22 can be implemented with a sorting operation as in (Kolouri et al., 2019).
Therefore, the couplings are found by which given each value of E x , we ﬁnd the corresponding values of E y and Y from { ( e ix , e iy ) } N and { ( x i , y i ) } N and then match such sorted values of E y and Y .
Variance-based divergence measure.
Suppose that the previous step has found the couplings, i.e., given any sample ( e ix , e iy ) , we have its corresponding ( x i , y i ) .
Then, we will compute the divergence of a velocity ﬁeld as in Line 13 of Alg. 1.
According to the deﬁnition, | div v | 2 = ( dv y dy ) 2 .
Since all the e iy are matched with y i , we have v iy =
y i − e iy .
A straightforward way to approximate the derivative is using its nearest neighbour pair ( e ixb , e iyb ) and ( xb i , yb i ) and then approximate it with ddy ( v y ) | y
= y i
=
( y i − e iy ) − ( yb i − e iyb )
y i − yb i ; however, it suffers the following issues (especially in the few-sample scenario) :
(a) the denominator is in general a small number, and the distance to the nearest neighbour can be large in the few-sample case, which makes the computation unstable and inaccurate; (b) the deviation on the X -axis makes the approximation a biased estimate especially when the gradient of g ( X ) + E y at X = x is large.
Therefore, we propose the variance-based divergence measure working better in practice.
It is straightforward to see that under constraints (I) and (II), the value of the divergence measure of an ANM is zero if and only if the value of the variance-based divergence measure is zero, which is deﬁned as
[Equation]
where V [ V y | X | X ] represents a conditional variance of the velocity ﬁeld V y (a random variable) at position X at the initial time.
For example, V [ V y | x | X = x ] represents the variance of all the velocities v y at the positions
where X = x at the initial time.
{ ( x i , y i ) } N denotes the cause-effect pair dataset with the sample size N .
{ v iy | x } N x denotes all the velocities v iy at position X
= x at the initial time, where the sample size is N x , and their mean value is v y | x .
Then, V [ V y | x | X = x ] = (cid:80) N x i =1 ( v iy | x − v y | x ) 2 / ( N x − 1) , and
D var
[Equation]
where { x i } N represents the set of all the values of X and its sample size is N ; −→ y x is the vector of the
Y samples where X = x ; −→ e y is the vector of the E y samples where E x = x ; sort ( · ) sorts a vector; ave ( · ) computes the vector mean; and (cid:107)·(cid:107) 22 is the square of a (cid:96) 2 norm.
Minimization w.r.t θ .
Given the data { ( x i , y i ) } N and the density p ( E y ; θ ) , we can compute the divergence measure with the variance-based method.
Note that we only initialize θ with some random value, and p ( E y ; θ ) is not necessary to be the true distribution or even signiﬁcantly different from the true one, which can lead to the wrong result of the divergence measure.
Therefore, as shown in Line 13 of Alg. 1, we minimize the divergence measure w.r.t.
θ .
For the minimization, one can derive the gradient w.r.t θ in a simple parameterization case as (9), while in the complex case one can use auto-differentiation.
According to Prop. 2, the divergence measure in the causal direction is zero if and only if p ( E y ; θ ∗ ) with the optimal parameter θ ∗ is the true noise distribution a.e., implied by the identiﬁability of ANMs.
In this work, we used autograd and RMSProp (gradient descent) of JAX for the minimization.
Extension to PNLs.
The measure (11) can be directly extended to the PNL cases.
Since h in PNL (2) is an invertible function, by considering h − 1 ( Y ) as a new random variable, h − 1 ( Y )
=
g ( X )+ E y is an ANM.
Thus, for PNLs, under the identiﬁabililty conditions of PNLs (Zhang and Hyv¨arinen, 2009b), Prop. 2 still holds; and we only need replace −→ y x in (11) with −→ ˜ y x
= f PNL ω ( −→ y x ) , where f PNL ω is an invertible function, e.g., it can be the simpliﬁed version of (Zhang et al., 2015a),
[Equation]
(12) where ω a , ω b , and ω c are positive scalars.
Moreover, f PNL ω can also be the monotonic neural networks, such as (Huang et al., 2018).
Extension to the multivariate case.
For simplicity and clarity of the paper, we focus on elaborating the connection between FCMs and dynamical systems and developing the theoretical basis and the method in the bivariate case, which are essential for the further development of FCM-based causal discovery methods.
In the case of multiple variables, one can use a constraint-based method to ﬁnd the causal skeleton (the undirected causal graph) and then use the extension of our method for the edge orientation similar as (Zhang and Hyv¨arinen, 2009b; Monti et al., 2020; Khemakhem et al., 2021).
See App.
B for details.
[Figure]
In panel (a), each result represents the percentage of correct results on datasets with a sample size generated from an ANM.
Panel (b shows the results of applying DIVOT using different types of distributions, p ( E y ; θ ) , to the datasets with uniform distribution y ∼
R ELATED WORK
There are mainly two types of causal discovery methods for static causal direction determination between two variables.
The ﬁrst one introduces model assumptions to achieve the identiﬁability of causal direction.
Most of such methods are based on LiNGAM (Shimizu et al., 2006) and ANMs (Hoyer et al., 2008; Mooij et al., 2009).
Some of them are based on the more general models, e.g., PNLs using MLP for representing nonlinear functions (PNL-MLP ) (Zhang and Hyv ¨ arinen, 2009b) and
PNLs using warped Gaussian process and mixture of Gaussian noise (PNL-WGP-MoG) (Zhang et al., 2015a).
Recently, Khemakhem et al.
(2021) propose an autoregressive ﬂow-based model (CAREFL), of which the assumption is more general than ANMs and stricter than PNLs.
They commonly apply (non)linear regression to learn the function g in (1) (or together with h in (2)), and then test the independence between the independent noise E y in (1) (the residual) and the cause X .
And the independence test is commonly Hilbert-Schmidt independence criterion (HSIC)
(Gretton et al., 2005); however, as argued by Yamada and Sugiyama (2010), the kernel width limits its practical use and its common heuristic value limits the ﬂexibility of the function approximation.
Moreover, there are other criteria proposed in the ﬁrst type of methods, such as the l ikelihood ratio (LLR) (Hyv ¨ arinen and Smith, 2013), maximum likelihood-based criterion (MML), regression error-based causal inference (RECI) (
Bl ¨ obaum et al., 2018), and mutual information (Zhang and
Hyv ¨ arinen, 2009a; Yamada and Sugiyama, 2010).
Nevertheless, the common issue of the ﬁrst type of methods is that they restrict the model classes and that their performance is sensitive to model assumptions.
The second type of methods achieves identiﬁability by proposing other principles instead of restricting model classes, such as exogeneity (Zhang et al., 2015b), the randomness of exogeneous variables (Entropic ) (Compton et al., 2021), and independent causal mechanisms, e.g.,
GPI (Stegle et al., 2010) and IGCI (Janzing et al., 2012).
E XPERIMENTS
We demonstrate and evaluate our method on the synthetic and real-world cause-effect pair data (Mooij et al., 2016).
Moreover, we also provide the experiments and the discussion of our method in the presence of unknown confounding in App. C.
The details about experiments are in App.
F.
Synthetic data.
We evaluate DIVOT with Gaussian noise as in (9) on the datasets with different (non)linear functions and samples sizes.
We generate synthetic data with the ANMs: 1) Y = X
+
E y ; 2) Y = 0 .
5 X ) 3 − 0 .
1 X + E y ; 3) Y = sin(4 X ) + E y ; 4) if x < 0 , Y = 0 .
5 X 3 −
X + E y
and if x ≥ 0 , Y = 1 − 0 .
5 X 3 + X + E y , where X ∼ U ( − 1 , 1) and E y ∼ U (0 , 1) are uniform distribution.
For each ANM, we generate datasets with the sample sizes 10 , 25 , 50 , 100 , 200 , and 500 .
For each sample size, 100 different datasets are generated.
Fig.
2a shows that
DIVOT consistently recovers the causal direction for all the cases.
Since DIVOT has no smoothness constraint of functions, it can deal with the case 4) which is a discontinuous function.
We compared
DIVOT with the results of CAREFL (Khemakhem et al., 2021), RECI (Bl¨obaum et al., 2018), and
[Table]
other benchmark methods in Appendix.
As shown in Fig.
7 of App.
F, our method performs better than the others.
Moreover, we also show the robustness of DIVOT to the prior misspeciﬁcation as the experiments in CAREFL (Khemakhem et al., 2021).
The synthetic data are generated with uniform distribution noise E y ∼ U (0 , 1) , where the p ( E y ; θ ) of DIVOT is either uniform distribution U (0 , 1) , beta distribution
B ( a = 0 .
5 , b = 0 . 5) , or standard normal distribution N (0 , 1) .
As shown in Fig.
2b, DIVOT with the misspeciﬁed noise distributions has similar performance with the one using the correct class of distributions.
T ¨ ubingen cause-effect pair dataset.
We apply DIVOT to the T ¨ ubingen cause-effect pair dataset (Mooij et al., 2016).
This is a collection of real-world cause-effect pairs.
We use the variancebased divergence measure (11) and parameterize p ( E y ; θ ) as in (9).
As for the PNL extension, we implement (12) without the positivity constraint (in practice imposing no constraint on ω also performs well in this simple formulation).
Moreover, we found that given a value of X or Y of the datasets, there are often few samples.
Thus, we consider a range of values of X , which may introduce bias of the divergence measure; hence we also use a linear debiasing function for reducing the bias.
Then, we minimize the variance-based divergence measure over parameters with autograd of JAX.
More details about the debiasing function and optimization can be found in App.
F. To compare with the results reported in other works, we use the maximum number of the sample size
N = 500 as (Stegle et al., 2010; Zhang et al., 2015a) and run all the experiments with 3 random seeds as (Stegle et al., 2010), though DIVOT is efﬁcient enough for datasets with larger sample sizes as shown in Appendix.
In addition, we normalize data and select the ones within 2 standard deviation, because DIVOT is sensitive to the outliers as optimal transport.
As shown in Tab. 1, our proposed DIVOT, especially the extension for PNLs, outperforms than the other methods.
The reported results in Tab. 1 are taken from (Stegle et al., 2010) and (Zhang et al., 2015a).
And our results are based on the same datasets as them, i.e., with 68 and 77 cause-effect pairs respectively.
Moreover, the entropic causal inference (Compton et al., 2021) is reported with 64 . 21% accuracy;
CAREFL (Khemakhem et al., 2021) is reported with 73% accuracy on 108 pairs; RECI (Bl ¨ obaum et al., 2018) is reported with 76% weighted accuracy on the 100 pairs.
From Tab. 1, we can also see that the other ANM/PNL-based methods are more sensitive to the choice of noise distributions.
As shown in Tab. 1, the ANM/PNL with Gaussian noise performs worse than the one with a more complex distribution (mixture of Gaussian) or the one combined with a more stable measure, such as HSIC independence test.
In contrast, the PNL extension of DIVOT with the Gaussian noise has a state-of-the-art result.
C ONCLUSION
In this paper, we provide a new dynamical-system perspective of FCMs in the context of identifying causal relationships in the bivariate case.
We ﬁrst demonstrate the connection between FCMs and optimal transport and study the dynamical systems of the optimal transport under constraints of FCMs.
We then show that FCMs correspond to pressureless potential ﬂows and that ANMs as a special case corresponding to the pressureless potential ﬂows with the velocity ﬁeld divergence equal to zero.
Based on such ﬁndings, we propose a divergence measure-based criterion for causal discovery and provide an efﬁcient optimal transport-based algorithm, DIVOT, for identifying causal direction between two variables.
The experimental results on both synthetic and real datasets show that compared with the state-of-the-art methods, DIVOT has state-of-the-art results and promising properties for ﬂexibility, efﬁciency, and prior misspeciﬁcation robustness.
We hope that the connection between FCMs and optimal transport has the potential of helping understand general FCMs from the dynamical perspective and inspiring more generic causal discovery methods.
ACKNOWLEDGEMENTS
KZ would like to acknowledge the support by the National Institutes of Health (NIH) under Contract
R01HL159805, by the NSF-Convergence Accelerator Track-D award (cid:93) and by the United
States Air Force under Contract No.
FA8650-17-C7715.
RT would like to acknowledge the funding support of the Swedish e-Science Research Centre
A PPENDIX
In App.
A, we provide the modiﬁcation of Alg.
1 for including the independent case and the signiﬁcance of the results.
In App.
B, we provide the outlook of the extension to the multivariate case.
In App.
C, we provide the analysis, the discussion, and the experimental results of our method in the presence of unknown confounding.
In App.
D, we provide the proofs of Prop. 1, the extension of Prop. 1, Prop. 2, and Thm. 1.
In App.
E, we include the identiﬁability conditions of ANMs in (Hoyer et al., 2008).
In App.
F, we introduce the details of the DIVOT implementation and the experiments:
• App.
F.1: introduce the concept, position, for computing the divergence measure (11); • App.
F.2: show a potential problem of computing the variance-based divergence measure in the ﬁnite/limited-sample case and introduce batches of data to deal with the problem; • App.
F.3: show that using batches can introduce bias and lead to wrong causal relationships (especially in the few-sample case) and then introduce debiasing functions to reduce the bias, which is used for the experiments on real-world data in Sec. 6; • App.
F.4: introduce the optimization methods used in the experiments on synthetic and real-world data in Sec. 6 and show the convexity of the objective function, the divergence measure (11), under the construction in Sec. 6; • App.
F.5: show the robustness of DIVOT to prior misspeciﬁcation; • App.
F.6: compare DIVOT with other benchmark methods; • App.
F.7: show the efﬁciency of DIVOT with its running time.
A M ODIFICATION OF A LG . 1 FOR INCLUDING THE INDEPENDENT CASE AND THE SIGNIFICANCE OF THE RESULTS
The proposed measure is able to deal with the independent case without relying on other methods or tests after adapting the output conditions of the algorithm.
The values of our proposed measure in the independent case are zero in the two directions, while the measure values of the causal case are a zero value in one direction and a non-zero value in another direction.
The modiﬁed output conditions are shown in Tab. 2.
To determine that the value of our measure is zero or not, one could follow a
[Table]
similar way as (Bl¨obaum et al., 2018)
choosing a threshold for real-world applications practically.
However, different applications may have different thresholds.
For example, in practice, when the noise distributions have different variance in the ﬁnite/few sample scenario, the larger variance can lead to the larger measure value.
Although one can handle the problem well by normalizing the variance-based measure value with the estimated noise variance, deriving a statistical test for the ﬁnite/few sample case is still the ideal way, which is another nontrivial task without assuming the type of noise distributions and will be the future work of our method.
Bootstrapping for the signiﬁcance of the results.
Instead of testing whether a measure value is signiﬁcantly zero or non-zero, we suggest using a bootstrapping method and testing whether the two measure values are signiﬁcantly different (one could also test whether the difference between the two measure values are signiﬁcantly zero or not):
Step 1.
use bootstrapping (resampling with replacement) to get B (e
g., 50) bootstrapping datasets; Step 2. compute two measure values in the two directions for each bootstrapping dataset; Step 3.
apply a two-sample test (T-test) to check whether the mean of one measure value in one direction is signiﬁcantly different from another one in the other direction.
Step 4.
if two measure values are signiﬁcantly different, we conclude that the smaller one is in the causal direction; if they are not signiﬁcantly different, we conclude that it is the independent case.
Moreover, we did experiments for the case where the causal relationship is so weak that the FCM with a causal relationship is similar to the independent case (e.g., when the coefﬁcient of the direct cause is extremely small in the linear case, it is close to the independent case where the measure values are close).
We generate four datasets with 1000 samples for the four FCMs in the synthetic data experiments in Sec.
6 with a weighting factor w : Y = w × f
( X ) + E y , and then use the bootstrapping method and T-test.
Tab. 3 shows the p-values
[Table]
of which small values indicate the signiﬁcantly different measure values.
If the signiﬁcant level is 0.05, our method can tell the difference between two measures in the causal case when w > 0 .
[Table]
[Table]
B E XTENSION TO THE MULTIVARIATE CASE
The problem of the naive solution to the multivariate case.
A direct extension to the multivariate case is as the extensions of other bivariate causal discovery methods, such as (Khemakhem et al., 2021), (Zhang and Hyv ¨ arinen, 2009b), and (Monti et al., 2020).
One can ﬁrst apply constraint- /score-based methods to get a causal skeleton, an undirected causal graph, and then use the extension of our measure for ﬁnding all the causal directions.
A problem of the extension in some of the other bivariate works assuming causal sufﬁciency is that they directly applied their bivariate methods to each edge of the causal skeleton without considering the DAG structure.
This can lead to the wrong results, especially in the case where they disregard the common parent/confounder of an edge.
Extending our method directly to the multivariate case.
We provided the extensions of Prop.
1 and the variance-based measure, which can be used for computing the measure value considering the DAG structure and then orienting the edges based on the causal skeleton in the multivariate case.
Our extension to the multivariate case has the following properties:
Given the causal skeleton,
it distinguishes Markov equivalent classes under the identiﬁability conditions of ANMs (or PNLs); 2.
it still beneﬁts from the closed-form 1D optimal transport solution and doesn’t have the computational issue as in the high-dimensional optimal transport methods because of the FCM constraints; 3.
it provides a score of which the causal structure has the minimum value compared with all DAGs of the causal skeleton; moreover, the measure value of a DAG is the summation of all measure values of the causal modules/conditionals/mechanisms.
First, we show the derivation of the extension of Prop. 1 for ANMs, given which the derivation in the PNL case is straightforward (one can consider what we did for Eqn.
(12) in our paper).
The proposition shows how to efﬁciently compute the L 2 Wasserstein distance between high dimensional distributions under FCM constraints.
Suppose that the general ANM is X i
=
g i ( PA i ) + E i , where i = 1 , ..., m , E
i is the noise term of X i , and PA i denotes the parent variables of X i .
The square of
L 2 Wasserstein distance is
[Equation]
of which the derivation is shown in App. D.2.
Second, as a direct implication of Thm. 1, given the couplings of E i and X i , the corresponding dynamical system has zero divergence of its velocity ﬁeld; in other words, the corresponding dynamical system which moves the samples of x 0 to the samples of x T under ANM constraints has zero divergence on each dimension.
And the variance-based measure is
D var
[Equation]
where −−−−−→ x i | PA i =
k is the data vector, of which the elements are the X i values of the samples with PA i taking the value k ; −→ e i is the data vector, of which the elements are the generated noise samples; N k represents the length of the vector −−−−−→ x i | PA i = k or −→ e i .
Next, as the direct extension, one could enumerate all possible DAGs of the causal skeleton and compute their measure values, of which the minimum value is corresponding to the causal graph.
Because the causal skeleton is given, it must be the case where one of the two variables of an edge is the cause and the other one is the effect.
So the enumerated graphs have two situations: 1) all the edges are correctly oriented; 2) the causal direction of at least one edge is wrong such that the measure value of at least one causal module is signiﬁcantly larger than the correct one (note that considering a child as the direct cause leads to increasing the measure value, while omitting a cause is not necessary to increase the measure value of the causal module).
Therefore, we can simply choose the graph with the minimum measure value as the causal one.
As mentioned in the paper, it is also very important to develop practical algorithms for large-scale real-world problems, and there are some points for future works to further explore:
Testing the signiﬁcance of the results in the multivariate case, i
e., whether the minimum one is signiﬁcantly smaller than the others.
One could apply bootstrapping to the dataset and have a pvalue for the measure value.
One could also apply bootstrapping for each causal module, however, there may exist the problem of the multiple statistical test issue with family-wise errors in this way.
Developing an efﬁcient search algorithm in the multivariate case without relying on constraint- /score-based methods.
This requires to analysing the measure value of a causal module in more situations (e
g., omitting a parent, involving an independent variable/non-child descendant/non-parent ancestor, or a case mixing the mentioned factors) and considering the characteristics to develop an efﬁcient search algorithm similar as the greedy search algorithm (Chickering, 2002).
C U NKNOWN CONFOUNDING
The unknown confounding has different inﬂuences on the results of our method in different situations: 1) independent case: when two variables are independent; 2) causal case: when there is a causal relationship between two variables.
Depending on how the unknown confounding inﬂuences the pair of variables, it can make our method
1. reverse the direction of the result in the causal case; 2.
disregard the causal relationship in the causal case; 3. introduce extraneous causal relationship in the independent case.
To understand the results, we could analyze some toy examples intuitively.
One extreme independent case is that X
=
U and Y =
E y
+ U , where U is the unobserved confounder
; E y is the noise of Y ; X has no noise.
Then our method will show that X is the cause of Y , even though there is no causal relationship.
Similarly, suppose that U and the noise of X and Y have the same distribution with variance equal to 1, and that X
=
E x + 1000 U and Y =
E y
+ U .
Then, the distribution of X can be dominated by U , which leads to the wrong result for the same reason as the extreme case.
In the causal case , it follows the same reason.
For example, X = E x + U and Y = X + E y
+1000 U .
Then the distribution of Y can be dominated by U .
Therefore, the FCM is close to X =
E x + U and Y = 1000 U , which leads to reversing the causal direction in the result.
Moreover, when
X =
E x
+1000 U and Y =
X
+ E
y
+1000 U , then both distributions of X and Y can be dominated by U ; hence, it is close to X = 1000 U and Y = 1000 U , which leads to disregarding the causal relationship in the result.
Nevertheless, in practice, the impact of unknown confounding is more complex and all factors can be mixed together with the impact of ﬁnite samples and the function properties.
Therefore, we provide the experiments based on the synthetic datasets.
From the experimental results, we can also ﬁnd that the strength of the confounding and the difference of the confounding strength on the two variables are two important factors resulting in the wrong results of our method.
Experimental results for the unknown confounder.
We generate a dataset with 1000 samples for each FCM:
[Equation]
where E x and E y are the noise terms; U is the unknown confounder; w x and w y are the coefﬁcients of U representing the confounding strength; we used f ( X ) =
X and f ( X ) = sin (4 X ) in the experiments.
As for the data generation, E x , E y , and U follow the uniform distribution, U (0 , 1) , and we vary the coefﬁcients w x and w y .
Instead of testing whether a result is signiﬁcantly close to zero, we use bootstrapping to resample 50 datasets for each generated dataset and then apply a two-sample test to test whether the measure values in the two different directions are signiﬁcantly different.
Because although the measure value, in theory, is zero in the causal direction when there is no unknown confounder, with ﬁnite samples the measure value will be larger than zero in practice.
And to decide which measure value is close to zero or not, it requires a threshold which can vary in different applications, or deriving a statistical test which is nontrivial for the measure without assuming the type of noise distributions and can be the future work of our method.
Thus, we test whether two measure values are signiﬁcantly different, and if so, we then conclude that the smaller one is in the causal direction; if they are not signiﬁcantly different, we then conclude that it is the independent case.
As for the experimental results, when the p-value is close to zero, it means that the two measure values are signiﬁcantly different.
And in general, one could take the signiﬁcant level at 0.05 to make a conclusion.
We used 0.05 for the experiments.
The experimental result of FCM1 is that the p-value is equal to 1.0, which means that the measure values in the two directions are not signiﬁcantly different.
As for the experimental results of FCM2, we found that in the independent case, when the coefﬁcients w x and w y are the same, we can get the correct results; when the coefﬁcients are different, the method will give the wrong results.
In different applications/scenarios, there are different tolerance ranges for our methods such that when the difference of the confounding coefﬁcients is within the range, even if the confounding coefﬁcients are different, we can still get the correct results.
[Table]
As for the experimental results of FCM3, we found that in the causal case, the confounding strength is a factor inﬂuencing the results more, which is different from the independent case.
Moreover, we found that when the nonlinear function f ( X ) is non-monotonic, such as sin (4 X ) , the number of correct results is larger than the one in the linear case.
Because the non-monotonic function itself can introduce a type of asymmetry which can indicate the causal direction for our method, the non-monotonic functions can be easier than the monotonic functions.
Future work for dealing with the unknown confounders.
Causal discovery in the presence of unknown confounding is still an open problem, but there are some promising results in recent years such as (Janzing and Sch¨olkopf, 2018; Mastakouri et al., 2021).
Especially, (Mastakouri et al.,
[Table] [Table]
2021) shows the identiﬁability results on time-series data in the presence of memoryless unknown confounders where the confounder doesn’t have an autocorrelation effect.
A potential research direction with our framework is that by recovering the corresponding dynamical process of a static causal discovery problem while considering the memoryless confounding, we can determine the causal direction between two variables in the presence of memoryless unknown confounders.
D P ROOFS AND DERIVATION D. D ERIVATION OF P ROP .
Under constraints (I) and (II),
[Equation]
deﬁnition of
[Equation]
Eqn.
[Equation] [Equation]
, and the independence of
[Equation]
and E y
[Equation]
D. P ROOF OF THE EXTENSION OF P ROP .
Suppose that in the multivariate ANM, x 0 =
[ E 1 , ..., E m ] (cid:48) , and x T =
[ X 1 , ..., X m ] (cid:48) , and X i =
g i ( PA i ) + E i , where PA i denotes the parent of X i .
[Equation]
(where M ∗ i represents the i-th element of M ∗ )
[Equation]
(where 1 : m − i represents 1 , ..., i − 1 , i + 1 , ..., m , and the independence of noise implies p ( E i ) p ( E 1: m − i ) ) =
[Equation]
(where applying the change of variable formula)
[Equation]
(where | det ( J ) | − 1 = 1 for ANMs)
[Equation]
D. P ROOF OF T HM .
Part I.
We ﬁrst derive the time interpolation p ( t, x t ) of p 0 and p T under dynamical FCM constraints.
Suppose that the FCMs are Y = f
( X, E y ) .
As for the dynamical formulation of the L 2 Wasserstein distance under constraints (I) and (II), according to the Jacobian equation, p ( t, x t ) =
p 0 ( x 0 ) / | det( J ∂ x t ∂ x 0 ) | , t ∈ (0 , T ) , where x t , x 0 ∈ R 2 and J ∂ x t ∂ x 0 is the Jacobian matrix w.r.t x 0 .
Moreover, ∂ log p ( t, x t ) ∂t = − ∂∂t | det
[Equation]
J
[Equation]
x t
[Equation]
0 ) | | det( J ∂ x t ∂ x 0 ) | , where J ∂ x t ∂ x 0 = (cid:34) 1 + tT , tT ∂f∂X , tT ( ∂f∂E
[Equation]
+(cid:35) .
According to Jacobi’s formula and (Tao, 2013),
[Equation]
where (cid:12) is element wise multiplication and adj ( A ) is adjugate matrix of A and the sign takes the same sign as det( J ∂ x t ∂ x 0 ) .
Furthermore, since we know that under the structural constraints (I) and (II), M ∗ is Eqn.
Thus, we replace A ( t ) with J ∂ x t ∂ x
=
I + tT ( ∇ M ∗ − I ) , and then we have
[Equation]
Therefore,
[Equation]
which takes the same sign as det( J ∂ x t ∂ x
[Equation]
Part II.
We then proof the property of the corresponding dynamical systems of ANMs in Thm. 1.
Because ∂ log p ( t, x t ) ∂t = − ∂∂t | det( J ∂ x t ∂ x 0 ) | | det( J ∂ x t ∂ x 0 ) | , under the conditions in Thm. 1, ∂f∂E y
=
and it is obvious that ddt | det( J ∂ x t ∂ x 0 )
| = 0 and | det( J ∂ x t ∂ x 0 ) | (cid:54) = 0 , ∀ t ∈ [0 , T ] .
Furthermore, according to the theorem of instantaneous change of variables (Chen et al., 2018), which is a variant of Fokker-Plank equation, we know that
[Equation]
Therefore, div v = 0 .
D. P ROOF OF P ROP .
Necessary direction.
We prove the necessary direction by showing that given X is the direct cause of Y in an ANM, D ( v ) = 0 .
In the causal direction, because of the time evolution equation (5), we know that it is sufﬁcient to check the divergence of the velocity ﬁeld at time 0 , and that the divergence taking zero value everywhere at time 0 implies the divergence taking zero value everywhere for t ∈ [0 , T ] .
Because of Thm. 1, we know the div v = 0 at time 0 .
Therefore,
D ( v ) = 0 .
Sufﬁcient direction.
We prove the sufﬁcient direction by showing the contradiction with the identiﬁability of ANMs.
Given that D ( v ) =
under the constraints (I), (II), (III), and the identiﬁability conditions of ANMs, it tells us that there is an ANM in the form of Eqn.
(1) which is consistent with the data distribution and has independent noise.
Because of the identiﬁability of ANMs shown by (Hoyer et al., 2008), there is no ANM with Y → X which is consistent with the data distribution and has independent noise at the same time.
Therefore, it can only be the case the X is the cause of
Y .
Moreover, there is a concern that it may happen that in the direction which is not the causal direction, the model is not an ANM and its divergence measure is equal to zero.
This may be problematic if the divergence measure is applied to applications.
However, we will show that in general, this will not happen by proving that under weak assumptions D ( v ) = 0 implies the model is an ANM.
First, D ( v ) = 0 implies div v = 0 for all x 0 with positive probability densities p ( x 0 ) > 0 .
In the following we say div v = 0 in short.
Second, div v = 0 implies ∂p ( t, x t ) ∂t = 0 a.e.
according to Thm.
Consequently, d
[Equation]
dt =0, which leads to ∂f∂E y = 1 where Y = f
( X, E y ) a.e.
according to the proof of Thm. 1.
Therefore, under the assumptions: 1) p ( E y ) is positive in a continuous range of
E y ; 2) in the range, ∂f∂E y
= 1 a.e. implies that it holds everywhere, we have Y =
E y + C , where C is a quantity which doesn’t change with E y ,
e.g., it can be a function of X or a constant.
Therefore,
D ( v ) = 0 implies Y =
E y + g
( X ) further under assumptions 1) and 2).
E I DENTIFIABILITY CONDITIONS OF ANM S IN (H OYER ET AL ., 2008)
Because the identiﬁability conditions of ANMs in (Hoyer et al., 2008) are important and necessary for our Thm. 1, we include them here:
Let the joint probability density of x and y be given by p ( x, y )
= p n ( y − f ( x ))
p x ( x ) , where p n , p x are probability densities on R .
If there is a backward model of the same form, i.e., p ( x, y ) = p ˜ n ( x − g ( y ))
p y
( y ) , then, denoting ν := log p n and ξ := log p x , the triple ( f, p x , p n ) must satisfy the following differential equation for all x , y with ν (cid:48)(cid:48)
( y − f ( x ))
f (cid:48) ( x ) (cid:54) = 0 :
[Equation]
−
[Equation]
where we have skipped the arguments y − f ( x ) , x , and x for ν , ξ , and f and their derivatives, respectively.
Moreover, if for a ﬁxed pair ( f, ν ) there exists y ∈ R such that ν (cid:48)(cid:48)
( y − f ( x ))
f (cid:48) ( x ) (cid:54)
=
for all but a countable set of points x ∈ R , the set of all p x for which p has a backward model is contained in a 3-dimensional afﬁne space.
F E XPERIMENTS AND DETAILS OF DIVOT
For the synthetic data experiments in Sec. 6, we visualized the generated data with the sample size 500 in Fig.
For determining causal direction, we used the variance-based divergence measure in
[Figure] [Figure]
Eqn.
The experiment setup is the same for experiments with different ANMs.
For each ANM, we run experiments with different sample sizes, 10 , 25 , 50 , 100 , 200 , and 500 .
For each sample size, 100 different datasets are generated.
For all the synthetic data experiments in Sec. 6, we use batches for computing Eqn.
(11) without using debiasing functions.
In the following, we introduce positions (App. F.1), batches (App. F.2), debiasing functions (App. F.3), and optimization details (App. F.4) of DIVOT; as well as show its robustness to prior misspeciﬁcation (App. F.5), more comparison with other benchmark methods (App. F.6), and its efﬁciency (App. F.7).
[Figure]
[Figure] [Figure]
[Figure] [Figure]
F. N UMBER OF POSITIONS FOR COMPUTING THE VARIANCE - BASED DIVERGENCE MEASURE
Suppose a dataset { ( x i , y i ) } N with sample size N .
For computing the divergence measure, we call x ∈ { x i } N as a position and for each x ∈ { x i } N we need to specify the corresponding vector −→ y x , of which the number of elements is N x and generated samples of −→ e y .
In practice, it is not necessary to use all the N positions for computing Eqn.
For the synthetic data experiments in Sec. 6, when N > 50 , we choose 50 positions out of N ; otherwise, we choose all the N positions.
To select a position (when N > 50 ), we ﬁrst ﬁnd the maximal and minimal values of { x i } N , compute the interval length, (cid:0) max( { x i } N ) − min( { x i } N ) (cid:1) / 50 , and then choose a position every such length (or its nearest position available in the data).
We can then compute Eqn.
(11) for causal direction determination.
F. B ATCHES FOR THE FINITE ( OR LIMITED ) SAMPLE SCENARIO
In the ﬁnite sample scenario, it can be the case that there is no other data at a position x .
Therefore, we use the neighbors of x as a batch of data for computing Eqn.
We consider all the data in a batch as having the same value x .
We represent the batch size with the percentage of the total number N of samples.
For the synthetic data experiments in Sec. 6, we use the batch size 0 .
4 for the datasets with sample size 10 ; 0 . 2 for the datasets with sample sizes 25 and 50 ; 0 .
15 for the datasets with sample sizes 100 and 200 ; and 0 .
for the datasets with sample size 500 .
Note that for the synthetic data experiments in Sec. 6, the batch size is not larger than 0 .
2 except the extremely small datasets with 10 samples.
Because using larger batch sizes introduces bias by concatenating data at different positions in a batch.
For the experiments in Fig.
4a, we generate a synthetic dataset with sample size 100 and use all the 100 positions for computing Eqn.
We can see that choosing a proper batch size can increase the accuracy of DIVOT by increasing the number of samples in a position; however, if further increasing the batch size such that it is larger than 0 .
2 , the accuracy is decreased because concatenating data at different positions leads to a different corresponding noise distribution compared with the ground-true p ( E y ) .
Thus, as shown in
Fig. 4b, the estimation of the noise distribution becomes worse and worse with increasing the batch size.
Such bias can be problematic especially in the few-sample case where we have to choose a larger batch size for computing the divergence measure.
Therefore, we use a debiasing function such
[Figure]
[Equation]
that we can use a larger sample size for the few-sample case without sacriﬁcing the performance of
DIVOT
F. D EBIASING FUNCTIONS FOR THE FEW - SAMPLE SCENARIO
Because the mentioned bias is introduced by omitting the position information, we introduce it back with a debiasing function g d ( x ) to reduce the bias by modifying Eqn.
(11) as
[Equation]
− −→
[Equation] [Equation]
where g d ( x ; w ) is a (non)linear function parameterized with w , e.g., in our experiments the linear debiasing function is g d ( x ) =
w × x. As shown in Fig.
4, using the linear debiasing function guarantees not only the accuracy while using a larger batch size than 0 . 2 , but also the correctness of the density estimation.
For a more complicated scenario, e.g., the ANM uses a nonlinear function, one can apply a neural network to the debiasing function.
We found that as a by-product, a sufﬁcient ﬂexible debiasing function can be used for estimating g ( x ) of ANMs in Eqn.
As shown in Fig.
5, we generate synthetic 500 data of an ANM with the nonlinear function Y
= g ( X )+ E y = 0 .
5 X ) 3 − 0 .
1 X
+
E y , where X ∼ U ( − 1 , 1) and E y ∼ U ( − 0 .
5 , 0 . 5) satisfy uniform distribution.
The g d ( x ) is close to the ground-truth g ( x ) .
Because this is not the main focus of the work, we would like to refer the readers to the Jupyter notebook in supplementary materials for the details of the implementation.
Nevertheless, the purpose of using debiasing functions is not to estimate the g ( x ) accurately but to reduce the bias introduced by using a large batch size.
This means that when we use a restrictive class of g d ( x ) , it can lead to a noticeable difference of g d ( x ) and g ( x ) .
But for the purpose of causal direction determination, the divergence measure does not require to specify the nonlinear functional form of ANMs.
Moreover, DIVOT based on optimal transport does not require to estimate g ( x ) either.
Therefore, DIVOT can still determine causal direction correctly in such case.
F. O PTIMIZATION METHODS AND THE CONVEXITY OF THE OBJECTIVE FUNCTIONS OF DIVOT
In this section, we introduce the optimization method of the synthetic experiments in Sec.
6 and the convexity of the objective function.
Next, we introduce the optimization method of DIVOT in the real-world data experiments in Sec. 6, of which the divergence measure uses a linear debiasing function and standard Gaussian noise
[Equation]
As for the synthetic data experiments, we minimize the variance-based divergence measure in Eqn.
(11) as an objective function, of which the parameter is θ .
Because when θ > 0 , the linear function f noise θ ( e source y ) does not change the order of sorting results of −−−−→ e source y , we have
[Equation]
It is obvious that the divergence measure in Eqn.
(14) is convex on θ > 0 .
The convexity is achieved by parameterizing the noise distribution with a linear function f noise , and there are many exited toolboxes for solving the minimization problem.
For the synthetic data experiments, we used gradient descend for ﬁnding the optimal θ ∗ .
The gradient jax.grad ( loss ) is computed with the autograd in JAX (Bradbury et al., 2018).
We update θ by specifying a step size sz and θ := θ − jax.grad ( loss ) × sz .
If after the update θ < 0 (which has never happened), we set the value of θ as a positive number close to zero.
We used sz = 1 for all the synthetic data experiments.
Moreover, because of the convexity, one can also use an one-step update method to directly get the best parameter θ ∗ by ﬁnding the root of the gradient function of
Eqn.
We simply give a range of θ and ﬁnd θ ∗ by a binary search method such that the gradient at θ ∗ is equal to zero.
The experiments of DIVOT for ANMs in the real-world data experiments also used the one-step update for updating θ and the range θ is speciﬁed as [0 , 100] .
In addition, we need to optimize over the parameter w of the linear debiasing function.
We use gradient descent to ﬁnd the best parameter, w ∗ .
We update θ every 10 updates of w .
For the PNL extension of DIVOT, we used the cyclic learning rate (Smith, 2017) with gradient descent.
We update θ every 10 updates of w and ω in (12).
The program of DIVOT is terminated when the divergence measure converges.
If a more complicated scenario requires f noise θ to be a nonlinear function, one may need to use the gradient descend method instead of the one-step update.
We ﬁnd that it is sufﬁcient to use the standard
Gaussian distribution with the linear f noise θ for DIVOT to have promising results in the experiments on the T¨ubingen datasets, which indicates that DIVOT is robust to the choice of models.
We then investigate the robustness of DIVOT to prior misspeciﬁcation.
F. R OBUSTNESS TO PRIOR MISSPECIFICATION
We use different hypothesized noise distributions in DIVOT to test the robustness to the misspeciﬁcation of noise distribution.
The synthetic data are generated as in Sec. 6 with uniform distrbution noise E y ∼ U (0 , 1) .
In DIVOT, we use one of the three hypothesized distributions: uniform distribution U (0 , 1) , beta distribution B ( a = 0 .
5 , b = 0 . 5) , and standard normal distribution N (0 , 1) .
As shown in Fig.
6, DIVOT with the misspeciﬁed noise distributions has similar performance with the one using the correct class of distributions, which shows the robustness of DIVOT.
F.
C OMPARISON WITH RESULTS OF BENCHMARK METHODS
We compare DIVOT with other benchmark methods, such as ANM (Hoyer et al., 2008), CAREFL (Khemakhem et al., 2021), RECI (Bl¨obaum et al., 2018), and LLR (Hyv¨arinen and Smith, 2013).
As in (Khemakhem et al., 2021), the synthetic data are generated with the Laplace distribution as
[Figure] [Figure]
Ilyes Khemakhem ⇤ , Ricardo P. Monti ⇤ , Robert Leech
[Figure]
Aapo Hyvärinen
[Figure] [Figure]
[Table]
in Table 1.
We note that the proposed method performs better than alternative algorithms
Arrow of time on EEG data
Finally, we consider the performance of CAREFL in inferring the arrow of time from open-access electroencephalogram (EEG) time series (Dornhege et al., 2004).
The data consists of 118 EEG channels for one subject.
We only consider the ﬁrst n time points, where n 2{ 150 , 500 } , after which each of the channels is randomly reversed.
More details on the preprocessing can be found in Appendix F.3.
The goal is to correctly infer whether x t !
x t +1 or x t +1 !
x t for each channel.
This is a useful test case for causal methods since the true direction is known to be from the past to the future.
We report in Figure 3 the accuracy as a function of the percentage of channels considered, sorted from highest to lowest conﬁdence ( i.e. by how high the amplitude of the output of each algorithm is).
For average to high conﬁdence, CAREFL is comparable in performance to the baseline methods but performs better in the low conﬁdence regime.
We also note that the performance of CAREFL improves by increasing the sample-size, which is to be expected from a method based deep
Interventions To demonstrate
CAREFL can answer interven- tional queries, we will consider both a synthetic con- trolled example, as well as real fMRI data.20 40 60 80 100 DeFLsLRn rDte 0.0 0.2 0.4 0.6 0.8 1.0 P r R S R r t L R n F R rr e F t n 150 0 20 40 60 80 100 DeFLsLRn rDte 0.0 0.2 0.4 0.6 0.8 1.0 P r R S R r t L R n F R rr e F t n 500 AlgRrLthP CA5EFL CA5EFL-16 LLneDr L5 5ECI A10 Figure 3: Performance on ﬁnding the arrow of time of EEG data, as a function of decision rate (percentage of channels — sorted by decreasing conﬁdence — we have to classify).
the noise distribution.
DIVOT uses batches without debiasing functions.
And the data preprocessing is the same as in the real-world data experiments in Sec. 6.
We run experiments on each dataset for 100 times.
As shown in Fig.
7, for the sample size larger than 100 the accuracy of DIVOT is 100% which shows that DIVOT has promising results and performs better compared the other methods, especially in the linear case.
F. E FFICIENCY OF DIVOT
For common causal discovery tasks in the bivariate case, the sample size 10000 is a large and challengeable one.
Causal discovery methods need to consider the efﬁciency of algorithms especially in the large sample size scenario.
We apply DIVOT with batches and no debiasing functions to the synthetic data, of which the sample size is 10000 generated with Y = 0 .
5 X
) 3 − X )
+ E y and E y ∼ U (0 , 1) .
As shown in Table 7, we test the running time of DIVOT to determine causal direction with different number of positions and different batch sizes.
The experiments are based on
MacBook Pro (15-inch, 2018) with 2.9 GHz 6-Core Intel Core i9.
Our implementation is based on
JAX (Bradbury et al., 2018) which uses Apache License and the running time is measured with the command %timeit in JAX.
[Table]