V ARIABLE C OMPLEXITY IN THE U NIVARIATE AND M ULTIVARIATE S TRUCTURAL C AUSAL M ODEL
Anonymous authors
Paper under double-blind review
A BSTRACT
We provide a critical view of the univariate Structural Causal Model. We show that by comparing the individual complexities of univariate cause and effect, in the Structural Causal Model, one can identify the cause and the effect, without considering their interaction at all.
The entropy of each variable is ineffective in measuring the complexity, and We propose We suggest to capture it the complexity by the reconstruction error of an autoencoder that operates on the list of sorted samples the percentiles of the distribution.
Comparing the reconstruction errors of the two autoencoders, one for each variable, is shown to perform well on the accepted benchmarks of the ﬁeld.
Hence, the decision as to which of the two is the cause and which is the effect may not be based on causality but on complexity.
In the multivariate case, where one can ensure that the complexities of the cause and effect are balanced, we propose a new adversarial training method that mimics the disentangled structure of the causal model.
We extend the results of Zhang & Hyvrinen (2010) to the multidimensional case, showing that such modeling is only likely in the direction of causality.
Furthermore, the learned model is shown theoretically to perform the separation to the causal component and to the residual (noise) component.
Our multidimensional method obtains a signiﬁcantly higher accuracy than the literature methods.
I NTRODUCTION
A long standing debate in the causality literature, is whether causality can be inferred without intervention (Pearl, 2009;
Spirtes et al., 2000).
The Structural Causal Model (SCM) (Spirtes et al., 2000) is a simple causative model for which many results demonstrate the possibility of such inference (Stegle et al., 2010;
Bloebaum et al., 2018; Goudet et al., 2018; Lopez-Paz et al., 2017; 2015).
In this model, the effect ( Y ) is a function of the cause ( X ) and some independent random noise ( E ).
In this work, we take a critical perspective of the univariate SCM. We demonstrate empirically that for the 1D univariate case, which is the dominant case in the existing literature, the SCM model leads to an effect that has a lower complexity than the cause.
Therefore, one can identify the cause and the effect by measuring their individual complexities, with no need to make the inference based on both variables simultaneously.
Thus, the decision as to which of the two is the cause and which is the effect may not be based on causality but on complexity.
Since we are dealing with unordered 1D univariate random variables, the complexity measure has to be based on the probability distribution function.
As we show empirically, comparing the entropies of the distribution two random variables is ineffective for inferring the causal direction.
We, therefore, consider the quantiles, i.e, ﬁxed sized vectors that are obtained as sub-sequences of the sorted sampled values of the variable.
We consider suitable complexity scores for these vectors.
In our analysis, we show that the reconstruction error of an autoencoder of a multivariate random variable is a valid complexity measure.
In addition, we link the reconstruction error based complexity, in the case of variational autoencoders, to the differential entropy of the input random variable. Hence, by computing the reconstruction errors of trained autoencoders on these vectors we estimate the entropies of the quantile vectors of
X and Y . The challenges of measuring causality independently of complexity in the 1D case lead us to consider the multidimensional case, where the complexity can be controlled by, e.g., manipulating the dimension of the noise signal in the SCM model.
Note that unlike Goudet et al.
(2018), we consider pairs of multivariate vectors and not many univariate variables in a graph structure.
We demonstrate that for the multidimensional case, any method that is based on comparing the complexity of the individual random variables X
and Y fails to infer causality of random variables.
Furthermore, we extend the result of Zhang & Hyvrinen (2010) to the multidimensional case and prove that an SCM is unlikely to hold in both directions X
→ Y and Y → X for reasonable conditions.
Based on our observations, we propose a new causality inference method for multidimensional cause and effect.
The algorithm learns three networks in a way that mimics the parts of the SCM model.
The noise part is unknown and is replaced by a function that is constrained to be independent of the cause, as captured by an adversarial loss.
However, we show analytically and empirically that even without the explicit constraint, in several cases, such an independence emerges.
Our empirical results support our analysis and demonstrate that in the 1D univariate case, assigning cause and effect based on complexity is competitive with the state of the art methods.
In the multidimensional case, we show that the proposed method outperforms existing and new extensions of the literature methods to the multidimensional case.
1. P ROBLEM S ETUP
We investigate the problem of causal inference from observational data.
A non-linear structural causal model (SCM for short) is a generative process of the following form:
[Equation]
The functions g : R d f + d
e
→ R n and f : R n → R d f are ﬁxed and unknown.
In general, g and f are non-linear.
Here, X is the input random variable and E is the environment random variable that is independent of X .
We say that X causes Y if they satisfy a generative process such as Eq. 1.
We present methods for inferring whether X causes Y (denoted by X → Y ) or Y causes X , or neither.
The algorithm is provided with i.i.d samples { ( x i , y i ) } mi =1 ∼ P mX,Y (the distribution of m i.i.d samples from the joint distribution P X,Y ) from the generative process of Eq.
In general, by (cf.
Prop 4.8, (Peters et al., 2017)), for any joint distribution P X,Y of two random variables X and Y , there is an SCM, Y = g
( f ( X ) , E ) , where E is a noise variable, such that, X
| =
E and h is some (measurable) function.
Therefore, in general, deciding whether X causes Y or vice versa is ill-posed when only provided with samples from the joint distribution.
However, Zhang & Hyvrinen (2010) showed for the one dimensional case (i.e., X, Y ∈ R ) that under reasonable conditions, a representation Y
= g ( f ( X )+ E ) holds only in one direction.
In Sec.
3.2 we extend this theorem and show that a representation Y =
g ( f ( X ) , E ) holds only in one direction when g and f are assumed to be neural networks and X, Y are multidimensional (we call such SCMs neural SCMs).
Throughout the paper, we denote by P U [ u ] := P [ U ≤ u ] the cumulative distribution function of a uni/multi-variate real valued random variable U and P is a standard Lebesgue measure.
Additionally, we denote by p U ( u ) = dd u P U
[ u ] the probability density function of U (if exists, i.e., P U [ u ] is absolutely continuous).
We denote by E u ∼ U [ f ( u )] or by E p U
[ f ( u )] the expected value of f ( u ) for u that is distributed by P U [ u ] , depending on the context.
The identity matrix of dimension n × n is denoted by I n .
We recall a few notations from information theory (Shannon, 1948; Cover &
Thomas, 2006).
The entropy of a random variable U , is deﬁned as h ( U ) :
= − E p U [log( p U ( u ))] , where U is the domain of U .
The mutual information between two random variables U and V is denoted by I ( U ; V ) :
= E p ( U,V ) (cid:104) log (cid:16) p U,V ( u,v ) p U ( u ) p V ( v ) (cid:17)(cid:105) .
Sometimes, we will denote I p ( U ; V ) , to specify the PDF, p , of the variables ( U, V ) . 1. R ELATED W ORK
In causal inference, the algorithm is provided with a dataset of matched samples ( x, y ) of two random variables X and Y and decides whether X causes Y or vice versa.
The early wisdom in this area asserted that this asymmetry of the data generating process (i.e., that Y is computed from X and not vice versa) is not apparent from looking at P X,Y alone.
That is, in general, provided with samples from the joint distribution P X,Y of two variables X, Y does tell us whether it has been induced by an SCM from X to Y or from Y to X .
In publications such as (Pearl, 2009; Spirtes et al., 2000) it is argued that in order to decide whether
X causes Y or vice versa, one needs to observe the inﬂuence of interventions on the environment parameter. To avoid employing interventions, most publications assume prior knowledge on the generating process and/or independence between the cause and the mechanism. Various methods for causal inference under the SCM model have been suggested. Many of these methods are based on independence testing, where the algorithm models the data as Y =
g
( f ( X ) , E ) (and vice versa) and decides upon the side that provides a better ﬁtting in terms of mapping accuracy and independence between f ( X ) and E = r
( X, Y ) .
The LiNGAM (Shimizu et al., 2006) algorithm assumes that the SCM takes the form Y = βX + E , where X | =
E and β ∈ R and
E is non-Gaussian.
The algorithm learns β , such that, X and Y − βX are independent by applying independent component analysis (ICA for short).
The Direct-LiNGAM (Shimizu et al., 2011) extends this method and replaces the mutual information minimization with a non-parametric kernel based loss (Bach & Jordan, 2003).
However, the computation of this loss is of order Θ( m 2 ) in the the worst case ( m is the number of samples).
The ANM approach (Hoyer et al., 2009) extends LiNGAM’s modeling and assumes that Y =
f ( X ) + E , where X | = E .
In their modeling they employ a Gaussian process for the learned mechanism between the two random variable.
The function f is trained to map between X and Y (and vice versa) and the method then tests whether, X and f ( X ) − Y are independent.
The independence test is based on kernels (Gretton et al., 2005).
A different extension of LiNGAM is the PNL algorithm by Zhang & Hyvrinen (2010).
This algorithm learns a mapping between X and Y (and vice versa) of the form Y =
g
( f ( X ) + E ) , where f ( X ) and E are restricted to be independent.
To do so, PNL trains two neural networks f
and g to minimize the mutual information between f ( X ) and E = g − 1 ( Y ) − f ( X ) .
The main disadvantage of this methods is the reliance on the minimization of the mutual information.
It is often hard to measure and optimize the mutual information directly, especially in higher dimensions.
In many cases, it requires having an explicit modeling of the density functions, because of the computation of expected log-probability within the formulation of the entropy measure.
In our method, we take a similar approach to the above methods.
However, our independence constraint is non-parametric and is applied on the observations rather on an explicit modeling of the density functions.
In addition, we do not assume restrictive structural assumptions and treat the generic case where the effect is of the form Y =
g
( f ( X ) , E ) .
In addition, our method is computationally efﬁcient.
Several other methods take different approaches.
For example, the Information Geometric Causal
Inference algorithm (IGCI) (Daniusis et al., 2012) approach determines the causal relationship in a deterministic setting Y =
f
( X ) under an independence assumption between the cause X and the mechanism f , Cov(log f (cid:48) ( x ) , p X ) = 0 .
The Conditional Distribution Similarity Statistic (CDS) (Fonollosa, 2016) measures the standard deviation of the values of Y (resp.
X ) after binning in the X (resp.
Y ) direction.
The lower the standard deivation, the more likely the pair to be X → Y .
The CURE algorithm (Sgouritsa et al., 2015) compares between X → Y and Y → X directions in the following manner: is given as follows: if we can estimate p X | Y based on samples from p Y more accurately than p Y | X based on samples from p X , then X → Y is inferred.
The BivariateFit method learns a Gaussian process regressor in both directions and decides upon the side that had the lowest error.
The RECI method (Bloebaum et al., 2018) trains a regression model in both directions, and returns the side that produced a lower MSE loss.
The utilized regression models were: a logistic function, polynomial functions, support vector regression and neural networks.
The
CGNN algorithm (Goudet et al., 2018) uses the Maximum Mean Discrepancy (MMD) distance between the distribution produced by modeling Y as an effect of X , ( X, g ( X, E ))
(and vice versa), and the ground truth distribution.
The algorithm compares the two distances and returns the direction that led to a smaller distance.
The Gaussian Process Inference model (GPI) (Stegle et al., 2010) builds two generative models, one for X → Y and one for Y → X .
The distribution of the candidate cause variable is modelled as a Gaussian mixture model, and the mechanism f is a Gaussian process.
The causal direction is determined from the generative model that best ﬁts the data.
Finally, it is worth mentioning that several other methods, such as (Heinze-Deml et al., 2017; Zhang et al., 2011) assume a different type of SCM, where the algorithm is provided with separate datasets that correspond to different environments, i.e., sampled i.i.d from P X,Y | E , where the value of E is ﬁxed for all samples in the dataset.
In these publications, a different independence condition is assumed: Y is independent of E given X .
This assumption typically fails in our setting.
In our paper, we focus on a vanilla SCM, where the algorithm is only provided with observational samples of X and Y =
g
( f ( X ) , E ) (i.i.d samples).
The samples are not divided into subsets that are invariant w.r.t E .
T HE U NIVARIATE C ASE
In this section, we intend to show that the univariate case is a bit simplistic.
For this purpose, we start by describing a method for identifying the cause and the effect, which considers each of the two variables independently without considering the mapping between them.
The method is based on the statistics of the quantiles of each of the two distributions.
In Sec. 2.2, we show that the reconstruction error of an autoencoder is directly linked to the differential entropy of the distribution it models.
2. A M ETHOD FOR I NFERRING D IRECTIONALITY
We hypothesize that due to the information processing inequality, the entropy of the cause h ( X ) in the SCM model is higher than the entropy h ( Y ) of the effect.
However, estimating the entropy of each random variable from its samples does not present a consistent difference between the entropies h ( X ) and h ( Y ) .
Our method, therefore,The proposed method computes an alternative a complexity score for X and, independently, for Y .
It then compares the scores and decides that the cause is the random variable that has the larger score among them.
Our scoring function has a few stages.
As a ﬁrst step, it converts the random variable at hand (say, X ) into a multivariate random variable.
This is done by sorting the samples of the random variable, and then cutting the obtained list into ﬁxed sized vectors of length k .
We discard the largest measurements in the case where the number of samples is not a multiple of k .
We denote the random variable obtained this way by U .
At the second stage, the method trains an autoencoder A : R k
→ R k on these quantile vectors.
Formally, A is trained to minimize the following objective:
[Equation]
where (cid:96) ( a, b ) is some loss function.
In our implementation, we employ the L 2 -loss function, deﬁned as (cid:96) ( a, b ) =
(cid:107) a − b (cid:107) 22 .
Finally, the method uses the value of L recon ( A ) , which we refer to as the AEQ score, as a proxy for the complexity of X (smaller loss means lower complexity).
It decides that X or Y is the cause, based on which side provides a higher AEQ.
2. R ECONSTRUCTION E RROR AS AN E STIMATOR OF THE E NTROPY
Next, we would like to provide better understanding of the AEQ method.
Since we use the reconstruction error as a measure of complexity, we next link it to entropy, which is an acceptable complexity measure.
This will explain that the AEQ method estimates and compares the entropies of the percentiles of X and Y .
This does not imply that the AEQ method compares between the entropies of X and Y .
The analysis This is done in the context of a learned variational autoencoder (VAE) (Kingma & Welling, 2014), with a ﬁxed latent space dimension.
We are given a real-valued vector-random-variable U (not necessarily quantiles) with a PDF p U ( u ) .
We recall the setting of VAEs and the analysis of Zhao et al.
The VAE model employs a d dimensional latent variable z ∼ N (0 , I d ) , and considers the family of the distribution p θ ( u | z ) parameterized by θ ∈ Θ .
The objective of this framework is to select θ that maximizes the likelihood of the samples of U within p θ ( u | z ) · p ( z ) ,
[Equation]
where p ( z ) = exp( −(cid:107) z (cid:107) 22 / 2) · (2 π ) − k/ 2 is the PDF of z and p θ ( u )
= p θ ( u | z ) · p ( z ) .
In typical situations, we select p θ to be the PDF of a normal distribution N ( f θ ( z ) , σ 21 · I n ) , where f θ ( z ) ∈ F is a learned neural network and σ 21 is a variance hyperparameter of the framework.
We denote by
P = { p θ ( u | z ) | z ∈ R d , θ ∈ Θ } the class of density functions over X .
To optimize this objective, it is required to evaluate p − 1 θ ( u ) , which involves the computation of a high dimensional integral.
Even though this integral can be approximated by sampling, this is a costy process that has to be done for every sample u .
To solve this problem, we have a second distribution that is responsible to model the latent codes
z that are most probable given u .
This PDF is denoted by q φ ( z | u ) and corresponds to a normal distribution
N ( g φ ( u ) , σ 22 · I d ) .
Here, g φ ( u ) is a neural network with parameters φ ∈ Φ .
Suppose we are given a ﬁxed inference distribution q φ ( z | u ) , which maps (probabilistically) inputs u to features z .
We denote by q φ ( u, z ) :
= p U ( u ) · q φ ( z | u ) the joint distribution of u and z , the marginal q φ ( z ) :
= (cid:82)
u q φ ( u, z ) d u and the posterior q φ ( u | z ) =
q φ ( z | u ) · p U ( u ) /q
φ ( z ) .
The optimization criterion is redeﬁned as follows, where for each z we use a member of a class of distributions, P , to ﬁt a different q φ ( u | z ) rather than the entire p U .
[Equation]
The following lemma links the reconstruction loss and the entropy (all proofs are in the appendix)
Lemma 1.
In the setting of Sec. 2.2.
Let θ ∗ be the global optimum of Eq.
4 for a sufﬁciently large
F .
Assume that P is sufﬁciently large, such that:
[Equation]
Then,
[Equation]
Therefore, from this lemma, we observe that if g φ is learned, such that, the expected norm of the encodings of u , E u ∼ U (cid:2) (cid:107) g φ ( u ) (cid:107) 22 (cid:3) is bounded, then, the differential entropy h ( U ) is proportional reconstruction error of the learned autoencoder.
In particular, for two random variables U 1 and U 2 , their entropies can be compared, by considering the reconstruction error of their autoencoders with the same hyperparameters (latent space dimension d , σ 21 , σ 2 2 , etc’).
T HE M ULTIVARIATE C ASE
For the univariate case, one can consider the complexity of the X and Y variables of the SCM model and infer directionality.
We propose the AEQ complexity for this case, since more conventional complexities are ill-deﬁned for unordered 1D data or, in the case of entropy, found to be ineffective.
In the multivariate case, one can consider the complexity of the random variables in various ways.
We focus on the family of complexity measures C ( X ) that satisfy the assumption that when X and
Y are independent
[Equation]
Sample complexity measures that satisfy this condition are the Shannon Entropy and the Kolmogorov Complexity.
Lem.
3 in Appendix.
B.2 shows that the complexity that is based on autoencoder modeling is also in this family.
The next result shows that for any complexity measure C in this family, one cannot infer directionality in the multivariate SCM based on C .
Lemma 2.
Let C be a complexity measure of multivariate random variables (i.e, non-negative and satisﬁes Eq. 7).
Then, there are triplets of random variables ( X, E, Y ) and ( ˆ X, E, Y ) , such that,
Y = g
( X, E ) , Y = g (cid:48)
( ˆ X, E ) , C ( X ) <
C ( Y ) and C ( ˆ X ) >
C ( Y ) .
Therefore, C cannot serve as a score for causal inference.
3. A M ETHOD FOR C AUSAL I NFERENCE
The causality inference algorithm trains neural networks G, F, R and D .
The success of ﬁtting these networks serves as the score for the causality test.
The function F models the function f , G models g and R ( Y ) aims to model the environment parameter E .
In general, our method aims at solving the following objective:
[Equation]
where A is either X or Y and B is the other option, and a i = x
i , b i =
y i or a i
= y
i , b i = x i accordingly.
To decide whether X → Y or vice versa, we train a different triplet G, F, R for each direction and see if we can minimize the translation error subject to independence.
We decide upon a speciﬁed direction if the loss can be minimized subject to independence.
In general, searching within the space of functions that satisfy A | =
R ( B ) is an intractable problem.
However, we can replace it with a loss term that is minimized when A | = R ( B ) .
Independence loss
We would like R ( B ) to capture the information encoded in E .
Therefore, restrict R ( B ) and A to be independent in each other.
We propose an adversarial loss for this purpose, which is a modiﬁed version of a loss proposed by Brakel & Bengio (2017) and later analyzed by (Press et al., 2019).
This loss measures the discrepancy between the joint distribution P A,R ( B ) and the product of the marginal distributions P A × P R ( B ) .
Let d F ( d R ) be the dimension of F ’s output ( R ).
To measure the discrepancy, we make use of a discriminator D : R n + d R → [0 , 1] from a class C that maximizes the following term:
[Equation]
where D is a discriminator network, and l ( p, q ) =
− ( q log( p ) + (1 − q ) log(1 − p )) is the binary cross entropy loss for p ∈ [0 , 1] and q ∈ { 0 , 1 } .
In addition, { (ˆ a i , ˆ b i ) } mi =1 are i.i.d samples from
P A × P B .
To create such samples, we can simply generate samples from ˆ a i ∼ P A and ˆ b i ∼ P B independently and then arbitrarily match them into couples (ˆ a i , ˆ b i ) .
To restrict that R ( B ) and A are independent, we train R to fool the discriminator D and to convince it that the two sets of samples are from the same distribution,
[Equation]
Full objective
The full objective of our method is then translated into the following program:
[Equation]
Where λ 1 is some positive constant.
The discriminator D minimizes the loss L D ( D ; R ) concurrently with the other networks.
We denote by c real the percentage of samples ( a i , b i ) that the discriminator classiﬁes as 1 and by c fake the percentage of samples (ˆ a i , ˆ b i ) that are classiﬁed as 0 .
We note that when c real + c fake ≈ 1 , the discriminator is unable to discriminate between the two distributions, i.e., it is wrong in classifying half of the samples.
Our method decides if X causes Y or vice versa by comparing the score L err ( G, F, R ) .
A lower error means a better ﬁt.
The full description of the architecture employed for the encoders, generator and discriminator is given in Appendix.
A.
3. A NALYSIS
In this section, we analyze the proposed method.
In Thm. 1, we show that if X and Y admit a SCM in one direction, then it admits a SCM in the opposite direction only if the involved functions satisfy a speciﬁc partial differential equation.
Theorem 1 (Identiﬁability of neural SCMs) .
Let P X,Y admit a neural SCM from X to Y as in Eq.
1, such that p X , and the activation functions of f and g are three-times differentiable.
Then it admits a neural SCM from Y to X only if p X , f , g satisfy Eq.
34 in the Appendix.
This result generalizes the one-dimensional case presented in (Zhang & Hyvrinen, 2010), where a one-dimensional version of this differential equation is shown to hold in the analog case.
In the following theorem, we show that minimizing the proposed losses is sufﬁcient to recover the different components, i.e., F ( X ) ∝ f ( X ) and R ( Y ) ∝ E , where A ∝ B means that A = f ( B ) for some invertible function f .
Theorem 2 (Uniqueness of Representation) .
Let P X,Y admit a nonlinear model from X to Y as in
Eq.
1, i.e., Y = g
( f ( X ) , E ) for some random variable E | =
X .
Assume that f and g are invertible.
Let G , F and R be functions, such that, L err :
=
E ( x,y ) ∼ ( X,Y )
[ (cid:107) G ( F ( x ) , R ( y )) − y (cid:107) 22 ] = 0 and G and F are invertible functions and X | =
R ( Y ) .
Then, F ( X ) ∝ f ( X ) and R ( Y ) ∝ E .
Here, L err :=
E ( x,y ) ∼ ( X,Y )
[ (cid:107) G ( F ( x ) , R ( y )) − y (cid:107) 22 ] is the mapping error proposed in Eq.
In addition, the assumption X | =
R ( Y ) is sufﬁced by the independence loss.
In Lem.
6 in Appendix.
B. we extend this lemma to the case where the mapping loss is not necessarily zero and get rid of the assumption that G is invertible.
In this case, instead of showing that R ( Y ) ∝ E , we provide an upper bound on the reconstruction of E out of R ( Y ) (and vice versa) that improves as L err decreases.
We note that when the entropy of R ( Y ) is smaller than the entropy of E , then, the independence
X | =
R ( Y ) follows, even without the discriminator.
Typically, when R has a limited capacity (i.e., small number of layers or output dimension), then, the entropy of R ( Y ) is limited as well.
Theorem 3 (Emergence of independent representations) .
Let P X,Y admits a nonlinear model from X to Y as in Eq.
1, i.e., Y = g
( f ( X ) , E ) for some random variable E | =
X .
Assume that X and Y are discrete random variables.
Let G , F and R be functions, such that,
{+L err := E ( x,y ) ∼ ( X,Y ) [ (cid:107) G ( F ( x ) , R ( y )) − y (cid:107) 22 ] = 
+}
and G is an invertible function.
Assume that h ( R ( Y )) ≤ h ( E ) and that g is invertible.
Then, we have: F ( X ) | =
R ( Y ) , F ( X ) ∝ f ( X ) and
R ( Y ) ∝ E .
To conclude this section, by Thm. 1, under reasonable assumptions, if X and Y admit a multivariate
SCM in direction X
→ Y , then, there is no such representation in the other direction.
By Thm. 2, by training our method in both directions, one is able to capture the causal model in the correct direction.
This is something that is impossible to do in the other direction by Thm. 1.
By Thm. 3, we show that in cases where h ( R ( Y ))
≤ h ( E ) , the independence between F ( X ) and R ( Y ) emerges implicitly.
This is the typical case, when R ’s capacity is limited.
E XPERIMENTS
This section is divided into two parts.
The ﬁrst, is devoted to showing that causal inference in the one-dimensional case highly depends on the complexities of the distributions of X and Y .
In the second part of this section, we show that our multivariate causal inference method outperforms existing baselines.
Most of the baseline implementations were taken from the Causality Discovery Toolbox of Kalainathan & Goudet (2019).
The experiments with PNL (Zhang & Hyvrinen, 2010),
LiNGAM (Shimizu et al., 2006) and GPI (Stegle et al., 2010) are based on their original matlab code.
4. O NE -D IMENSIONAL D ATA
We compared the autoencoder method on several well-known one dimensional cause-effect pairs datasets.
Each dataset consists of a list of pairs of real valued random variables ( X, Y ) with their direction 1 or 0 depending on X → Y or Y → X (resp.).
For each pair, we have a dataset of samples { ( x i , y i ) } m i =1 .
Five cause-effect inference datasets, covering a wide range of associations, are used.
CENet (Goudet et al., 2018) contains 300 artiﬁcial cause-effect pairs generated using random distributions as causes, and neural networks as causal mechanisms.
CE-Gauss contains 300 artiﬁcial causeeffect pairs as generated by Mooij et al.
(2016), using random mixtures of Gaussians as causes, and
Gaussian process priors as causal mechanisms.
CE-Multi (Goudet et al., 2018) contains 300 artiﬁcial cause-effect pairs built with random linear and polynomial causal mechanisms.
In this dataset, simulated additive or multiplicative noise is applied before or after the causal mechanism.
The real-world datasets include the diabetes dataset by Frank & Asuncion (2010), where causality is from Insulin → Glucose.
Glucose curves and Insulin dose were analysed for 69 patients, each serves as a separate dataset.
To match the literature protocols, the pairs are taken in an orderless manner, ignoring the time series aspect of the problem.
Finally, the T¨ubingen cause-effect pairs dataset by Mooij et al.
(2016) is employed.
This dataset is a collection of 100 heterogeneous, handcollected, real-world cause-effect samples.
The autoencoder employed in our method is a fully-connected 5-layered neural network with 3 layers for the encoder and 2 layers for the decoder.
The hyperparameters of this algorithm are the sizes of each layer, the activation function and the input dimension, i.e., length of sorted cuts (denoted by k in Sec. 2).
Throughout the experiments, we noticed that the hyperparameter that has the highest inﬂuence is the input dimension.
For all datasets results are stable in the range of 200 ≤ k ≤ 300 , and we therefore use k = 250 throughout the experiments.
For all dataset we employed the ReLU activation function, except the T ¨ ubingen dataset, where the sigmoid activation function produced better results (results are also reasonable with ReLU, but not state of the art).
In addition to our method, we also employ the entropy of each individual variable as a complexity method.
This is done by binning the values of the variables into 50 bins.
Other numbers of bins produce similar results.
Tab. 1 presents the mean AUC for each literature benchmark.
As can be seen, the AEQ complexity measure produces reasonable results in comparison to the state of the art methods, indicating that the 1D SCM model can be overcome by comparing per-variable scores.
On the popular T ¨ ubingen dataset, the one-sided AEQ computation outperforms all literature methods.
Tab. 2 presents accuracy rates for various methods on the T ¨ ubingen dataset, where such results are often reported in the literature.
As can be seen, our simple method outperforms almost all other methods, including methods that employ supervised learning of the cause-effect relation.
4. M ULTIVARIATE D ATA
We compared our method on several synthetic datasets.
Each dataset consists of a list of pairs of real multivariate random variables ( X, Y ) with their direction 1 or 0 , depending on X → Y or Y → X (resp.).
For each pair, we have a dataset of samples { ( x i , y i ) } mi =1 .
We employ ﬁve datasets, covering multiple associations.
Each dataset contains 300 artiﬁcial causeeffect pairs.
The cause random variable is of the form X = h ( z ) , where h is some function and z ∼ N (0 , σ 21 · I n ) .
The effect is of the form Y =
g ( u ( X, E )) , where E ∼ N (0 , σ 22 · I n ) is independent of X , u is a ﬁxed function that combined the cause X and the noise term E and g is the causal mechanism.
For each dataset, the functions h and g are taken from the same family of causal mechanisms H .
Each pair of random variables is speciﬁed by randomly selected functions h and g .
MCE-Poly is generated element-wise polynomials composed on linear transformations as mechanisms and u ( X, E ) =
X + E .
MCE-Net pairs are generated using neural networks as causal mechanisms and u is the concatenation operator.
The mechanism in MCE-SigMix consists of linear transformation followed by element wise application of q a,b,c ( x ) :=
ab (˜ x + c ) / 1 + | b · (˜ x + c ) | , where a, b, c are random real valued numbers, which are sampled for each pair and ˜ x = x + e , where e is the environment random variable.
In this case, u ( X, E ) =
X + E .
In order to create each one of the synthetic datasets we took the standard synthetic data generators of
Kalainathan & Goudet (2019) and extended them to produce multivariate causal pairs.
We noticed that a-priori, the produced datasets are imbalanced in a way that the reconstruction error of a standard autoencoder on each random variable can be employed as a score that predicts the cause variable with a high accuracy.
Therefore, in order to create balanced datasets, we varied the amount of noise [Table] [Table] dimensions and their intensity until the autoencoder reconstruction error of both X and Y became similar.
Note that for these multivariate variables, we do not use quantiles and use the variables themsevles.
As the AutoEncoder reconstruction results in Tab. 3 show, in the MCE-SigMix dataset balancing was only partly successful.
In Tab. 3 we also compare our method to BivariateFit and ANM (Hoyer et al., 2009) algorithms on these datasets and present favorable performance.
The other literature methods are designed for the univariate case and therefore, we were unable to extend them.
The only baseline methods that are extendable to multi dimensions are PNL (Zhang & Hyvrinen, 2010), CGNN (Goudet et al., 2018) and RECI (Bloebaum et al., 2018).
However, their runtime becomes too long.
Speciﬁcally, However, RECI’s runtime is of order O ( n 3 ) , where n is the input dimension. [Table]
In addition to the synthetic dataset, we also employ the MOUS-MEG real world dataset, provided to us by the authors of (Anonymous, 2020).
This dataset is part of Mother Of Uniﬁcation Studies (MOUS) dataset
(Schoffelen et al., 2019).
This dataset contains magneto-encephalography (MEG) recordings of 102 healthy dutch-speaking subjects performing a reading task (9 of them were excluded due to corrupted data).
Each subject was asked to read 120 sentences in Dutch, both in the right order and randomly mixed order, which adds up to a total of over 1000 words. Each word was presented on the computer screen for 351ms on average and was separated from the next word by 3-4 seconds. Each time step consists of 301 MEG readings of the magnetometers, attached to different parts of the head. For more information see (Schoffelen et al., 2019).
For each pair ( X, Y ) ,
X is the interval [ − 1 .
5 s, − 0 .
5 s ] relative to the word onset concatenated with the word embedding (using the spaCy python module with Dutch language model), this presents the subject in his “rest” state (i.e. the cause).
{+Y is the interval [0 , 1 . 
+}
s ] relative to the word onset, which presents the subject in his “active” state (i.e. the effect).
The results, also given in Tab. 3 show a clear advantage over the literature methods.
To validate the soundness of the dataset, we ran made a few experiments on variations of the dataset and report the results in Tab. 4.
As can be seen, a dataset where the cause consists of the word embedding and the effect consists of the subject’s “active” state is highly imbalanced.
This is reasonable since the word embedding and the MEG readings are encoded differently and are of different dimensions.
In addition, when the cause is selected to be the “rest” state and the effect is the “active” state, the various algorithms are unable to infer which side is the cause and which one is the effect since the word is missing.
To check the importance of our adversarial loss in identifying the direction of causality and capturing the implicit independent representation f ( X ) and E , we applied our method without training R against the discriminator.
Therefore, in this case, the discriminator only serves as a test whether X and R ( Y ) are independent or not.
Recall that our analysis shows that the discriminator is not necessary under reasonable conditions.
As can be seen in Tab. 5, the adversarial loss improves the results when there is no implicit emergence of independence, however, in cases where there is emergence of independence, the results are similar.
This is veriﬁed in Tab. 5, which shows similar performance with and without the discriminator.
As mentioned in Sec. 3.1, the distance between c real + c fake to 1 indicates the amount of dependence between X and R ( Y ) .
We denote by Ind C
the mean values of | c real + c fake − 1 | over all pairs of random variables, epochs and samples when training our method in the causal direction.
The same mean score when training in the anti-causal direction is denoted Ind E.
As is evident from
Tab. 5, the independence is similar between the two directions, emphasizing the importance of the reconstruction error in the score.
We noticed that the values of Ind C and Ind E are smaller for the full method.
However, in MCE-Poly and MCE-SigMix they are still very small, and therefore, there is implicit emergence of independence between X and R ( Y ) , even without explicitly training R ( Y ) to be independent of X .
This can explain the fact that the drop in the AUC is signiﬁcant but not drastic. [Table] [Table]
S UMMARY
We identify an inbalance in the complexities of cause and effect in the unidimensional SCM model and suggest a method to exploit it.
Since the method does not consider the interactions between the variables, its success in predicting cause and effect indicates an inherent bias in the unidimensional datasets.
Turning our attention to the multivariate case, where the complexity can be actively balanced [Table] we propose a new method in which the learned networks models the underlying SCM model itself.
Since the noise term E is unknown, we replace it by a function of Y that is enforced to be independent of X .
We also show that under reasonable conditions, the independence emerges even without explicitly enforcing it.
A A RCHITECTURE
The function G , F , R and D in the multivariate method in Sec.
3 are fully connected neural networks and their architectures are as follows: F is a 2-layered network with dimensions 100 → 60 → 50 ,
R is a 3-layered network with dimensions 100 → 50 → 50 → 20 , G is a 2-layers neural network with dimensions 50 + 20 → 80 → 100 (the input has 50 dimensions for F ( X ) and 20 for R ( Y ) ).
The discriminator is a 3-layers network with dimensions 100 + 20 → 60 → 50 → 2 (the input is X and R ( Y ) ).
The activation function in all networks is the sigmoid function except the discriminator that applies the leaky ReLU activation.
For all networks, the activation is not applied at the output layer.
B A NALYSIS B. T ERMINOLOGY AND N OTATIONS
We recall some relevant notations and terminology.
The identity matrix of dimension n is denoted by I n .
For a vector x = ( x 1 , . . .
, x n ) ∈ R n we denote (cid:107) x (cid:107) 2 :=
(cid:112)(cid:80) n i =1 x 2 i the Euclidean norm of x .
For a differentiable function f : R m → R n and x ∈ R m , we denote by
[Equation]
the Jacobian matrix of f in x .
For a twice differentiable function f : R m → R , we denote by
[Equation]
) the Hessian matrix of f in x .
Additionally, for a twice differentiable function f : R m → R n , f ( x ) =
( f 1 ( x ) , . . .
, f n ( x )) , we denote the Hessian of f by H ( f ( x )) :
[Equation]
( H ( f 1 ( x )) , . . .
, H ( f n ( x ))) .
For a scalar function f : R m → R instead of using the Jacobian notation, the gradient notation will be employed, ∇ ( f ( x )) :
=
J ( f ( x )) .
B. A UTOENCODER AS A C OMPLEXITY M EASURE
Let F = {H d } ∞ d =1 be a family of classes of autoencoders
A : R d → R d . Assume that the family
F is closed to ﬁxations, i.e., for any autoencoder
A ∈ H d 1 + d 2 and any ﬁxed vector y ∗ ∈ R d( x ∗ ∈ R d 1 ), we have: A ( x, y ∗ ) 1: d 1 ∈ H d 1 ( A ( x ∗ , y )
d 1 +1: d 2 ∈ H d 2 ).
Here, v i : j = ( v i , . . .
, v j ) .
We consider that this is the typical situation when considering neural networks with biases.
Let X be a random variable.
Let X be a multivariate random variable dimension d .
We deﬁne the autoencoding complexity of X as follows:
[Equation]
Lemma 3.
Let {H d } ∞ d =1 be a family of classes of autoencoders that is closed to ﬁxations.
The function C ( X ; F ) is a complexity measure according to Sec. 3.
Proof.
First, since (cid:96) ( a, b ) ≥ 0 for all a, b ∈ R k , this function is non-negative.
Next, we would like to show that C α ( X, Y ) ≥ max( C ( X ) , C ( Y )) .
Let A ∗ be the minimizer of E x ∼ X
[ (cid:96) ( A ∗ ( x ) , x )] within H d 1 + d 2 .
We consider that there is a vector y ∗ , such that,
[Equation]
y
[Equation]
y ))
[Equation] [Equation]
We note that A ( x, y ∗ ) 1: d 1 ∈ H d 1 .
Therefore,
[Equation]
By similar considerations, C ( X, Y ; F ) .
B. P ROOFS FOR THE R ESULTS
In this section we provide the proofs of the main results in the paper.
Lemma 1.
In the setting of Sec. 2.2.
Let θ ∗ be the global optimum of Eq.
4 for a sufﬁciently large
F .
Assume that P is sufﬁciently large, such that:
[Equation]
Then,
[Equation]
Proof.
This is a corollary of the analysis of Zhao et al.
By their analysis, we have:
[Equation] [Equation]
Since we selected p θ ∗ ( u | z ) to be the PDF of a normal distribution N ( f θ ∗ ( z ) , σ 21 · I n ) , we have:
[Equation]
Therefore, since z distributed by q φ ( z | u ) can be represented as z | u ∼ g φ ( u ) + σ 22 · I d · (cid:15) , where (cid:15) ∼ N (0 , I d ) ,
[Equation]
In addition
[Equation]
we have:
[Equation]
Therefore
[Equation]
we obtained the desired equation
[Equation]
Lemma 2.
Let C be a complexity measure of multivariate random variables (i
[Equation]
e, non-negative and satisﬁes Eq. 7).
Then, there are triplets of random variables ( X, E, Y ) and ( ˆ X, E, Y ) , such that,
Y = g
( X, E ) , Y = g (cid:48)
( ˆ X, E ) , C ( X ) <
C ( Y ) and C ( ˆ X ) >
C ( Y ) .
Therefore, C cannot serve as a score for causal inference.
Proof.
Let X be a random variable and E | =
X , such that, Y = g
( X, E ) .
Assume that C ( X ) <
C ( Y ) .
Then, let X (cid:48) be a random variable independent of X , such that, C ( X (cid:48) )
> C ( Y ) .
Then, we have: C ( X, X (cid:48) )
> C ( Y ) and we have: Y = g (cid:48)
( X, X (cid:48) , E ) , for g (cid:48)
( a, b, c ) =
g ( a, c ) .
The following lemma is an extension of Thm.
1 in (Zhang & Hyvrinen, 2010)
to real valued random variables of dimension > 1 .
Lemma 4.
Assume that ( X, Y ) can be described by both:
[Equation]
and
[Equation]
Assume that g 1 and g 2 are invertible and let:
[Equation]
and
[Equation]
Assume that the involved densities p T 2 , p E 1 and nonlinear functions f 1 , g 1 and f 2 , g 2 are third order differentiable.
We then have the following equations for all ( X, Y ) satisfying:
[Equation]
where η 1 ( t 2 ) := log p T 2 ( t 2 ) and η 2 ( e 1 ) :
= log p
E 1 ( e 1 ) .
Proof.
The proof is an extension of the proof of Thm.
1 in (Zhang & Hyvrinen, 2010).
We deﬁne:
[Equation]
and h
[Equation]
and
[Equation]
Since g 2 is invertible, the independence between X and E 1 is equivalent to the independence between T 2 and E 1 .
Similarly, the independence between Y and E 2 is equivalent to the independence between T 1 and E 2 .
Consider the transformation F :
[Equation]
[Equation]
The Jacobian matrix of this transformation is given by:
[Equation]
Since I commutes with any matrix, by Thm. 3 in (Silvester
[Equation]
1999), we have:
[Equation]
Therefore, we have: p T 2 ( t 2 ) · p E 1 ( e 1 ) =
p T 1 ,E 2 ( t 1 , e 2 ) / | det J | = p T 1 ,E 2 ( t 1 , e 2 ) .
Hence, log( p T 1 ,E
[Equation]
1 , e 2 ))
=
η 1 ( t 2 ) + η 2 ( e 1 ) and we have:
[Equation]
Therefore,
[Equation]
The independence between T 1 and E 2 implies that for every possible ( t 1 , e 2 ) , we have:
[Equation]
= [Equation]
Lemma 5 (Reduction to post-linear models) .
Let f ( x ) =
σ 1 ( W d . . .
σ 1 ( W 1 x )) and g ( u, v ) =
σ 2 ( U k . . .
σ 2 ( U 1 ( u, v ))) be two neural networks.
Then, if Y = g ( f ( X ) , E ) for some E | =
X , we can represent Y
= ˆ g
( ˆ f ( X ) + N ) for some N | =
X .
Proof.
Let f ( x ) =
σ 1 ( W d . . .
σ 1 ( W 1 x )) and g ( u, v ) =
σ 2 ( U k . . .
σ 2 ( U 1 ( u, v ))) be two neural networks.
Here, ( u, v ) is the concatenation of the vectors u and v .
We consider that U k
( f ( X ) , E ) =
U 11 f ( X ) + U 21 E .
We deﬁne a noise variable N :=
U 21 E and have: X | = N .
In addition, let ˆ f ( x ) :=
U 11 f ( x ) and ˆ g
( z ) :
= σ 2 ( U k . . .
σ 2 ( U 2 σ 2 ( z ))) .
We consider that:
Y =
ˆ g
( ˆ f ( X )+ N ) as desired.
Theorem 1 (Identiﬁability of neural SCMs) .
Let P X,Y admit a neural SCM from X to Y as in Eq.
1, such that p X , and the activation functions of f and g are three-times differentiable.
Then it admits a neural SCM from Y to X only if p X , f , g satisfy Eq.
34 in the Appendix.
Proof.
Let f i ( z ) =
σ 1 ( W i,d . . .
σ 1 ( W i, 1 z )) and g i ( u, v ) =
σ 2 ( U i,k . . .
σ 2 ( U i, 1 ( u, v )))
(where i = 1 , 2 ) be pairs of neural networks, such that, σ 1 and σ 2 are three-times differentiable.
Assume that:
[Equation]
and
X =
g ( f ( Y [Equation] , E
[Equation]
31) for some E 1 | =
X and E 2 | =
Y .
By Lem.
5, we can represent
[Equation]
where N 1 = U 21 , 1 E 1 , ˆ f
= U 11 , 1 f 1 ( X ) and ˆ g 1 ( z )
=
σ 2 ( U 1 ,k . . .
σ 2 ( U 1 , 2 σ 2 ( z )))
(32) and also,
X =
[Equation]
, where N 2 = U 22 ,
[Equation]
E
[Equation]
[Equation]
From the proof of Lem.
5, it is evident that the constructed ˆ g 1 , ˆ f 1 and ˆ g 1 , ˆ f 2 are three-times differentiable whenever σ 1 and σ 2 are.
Therefore, by Thm. 1, the following differential equation holds:
[Equation]
34) where
T 1 :
=
ˆ g − 1 1 ( Y ) and h 1 := ˆ f 2 ◦ ˆ g
T 2 :
=
ˆ g − 1 2 ( X ) and h 2 :
= ˆ f 1 ◦ ˆ g(35) and η 1 ( t 2 ) := log p T 2 ( t 2 ) and η 2 ( n 1 ) :
= log p N 1 ( n 1 ) .
Theorem 2 (Uniqueness of Representation) .
Let P X,Y admit a nonlinear model from X to Y as in
Eq.
1, i.e., Y = g
( f ( X ) , E ) for some random variable E | =
X .
Assume that f and g are invertible.
Let G , F and R be functions, such that, L err :
=
E ( x,y ) ∼ ( X,Y )
[ (cid:107) G ( F ( x ) , R ( y )) − y (cid:107) 22 ] = 0 and G and F are invertible functions and X | =
R ( Y ) .
Then, F ( X ) ∝ f ( X ) and R ( Y ) ∝ E .
Proof.
Since F and f are invertible, one can represent: F ( X ) =
F ( f − 1 ( f ( X ))) and f ( X ) =
f ( F − 1 ( F ( X ))) .
Similarly, since G and g are invertible, we also have: ( F ( X ) , R ( Y ))
∝ ( f ( X ) , E ) .
Since ( F ( X ) , R ( Y ))
∝ ( f ( X ) , E ) and F ( X ) ∝ f ( X ) , we have: R ( Y ) =
Q ( F ( X ) , E ) .
However, R ( Y ) | =
F ( X ) and therefore, we can represent R ( Y ) =
P ( E ) and vice versa.
Theorem 3 (Emergence of independent representations) .
Let P X,Y admits a nonlinear model from X to Y as in Eq.
1, i.e., Y = g
( f ( X ) , E ) for some random variable E | =
X .
Assume that X and Y are discrete random variables.
Let G , F and R be functions, such that,
L err := E ( x,y ) ∼ ( X,Y ) [ (cid:107) G
( F ( x ) , R ( y )) − y (cid:107) 22 ] =
and G is an invertible function.
Assume that h ( R ( Y )) ≤ h ( E ) and that g is invertible.
Then, we have: F ( X ) | =
R ( Y ) , F ( X ) ∝ f ( X ) and
R ( Y ) ∝ E .
Proof.
Since g and f are invertible, we have: h ( Y ) =
h ( f ( X ))
+ h ( E ) ≤ h ( X )
+ h ( E ) .
In addition, we have: h ( R ( Y ))
≤ h ( E ) .
Therefore, since G is invertible, h ( Y )
=
h ( F ( X ) , R ( Y ))
≤ h ( F ( X ))
+ h ( R ( Y ))
≤ h ( X ) + h ( E ) =
h ( Y ) .
Hence, F ( X ) and R ( Y ) are independent.
In the following three lemmas we extend Thms.
2 and 3.
Lemma 6.
Let P X,Y admit a nonlinear model from X to Y as in Eq.
1, i.e., Y = g
( f ( X ) , E ) , for some random variable E | =
X .
Assume that f and g are invertible.
Assume that X and Y are discrete and Y is over a set Y , such that, ∀ y 1 (cid:54) =
y 2 ∈ Y , we have, (cid:107) y 1 − y 2 (cid:107) 2 > ∆ > 0 .
Let G ,
F and R be functions, such that, L err :
= E ( x,y ) ∼ ( X,Y ) (cid:2) (cid:107) y − G ( F ( x ) , R ( y ))
(cid:107) 22 (cid:3) <
∆ / 2 and F is invertible.
Then, F ( X ) ∝ f ( X ) and I ( R ( Y ); E ) ≥ (cid:0) 1 − L err ∆ (cid:1) · h ( E ) − (cid:113)
L err ∆ .
Proof.
First, since both F and f are invertible, F ( X ) ∝ f ( X ) .
By the proof of Lem.
10 in (Press et al., 2019), there is a function r , such that: [Equation]
[Equation]
Since g is invertible, there is a function u ( Y ) =
E .
In particular,
[Equation]
Therefore, by Lem. 6 in (Press et al., 2019), we have:
[Equation]
By the analysis in the proof of Lem.
10 in (Press et al., 2019), we derive that:
[Equation]
Finally, since F ( X ) is a function of X which is independent of E , we obtain the desired inequality:
[Equation]
We mention that in Lem.
6, the function R ( Y ) can hold all of the information present in Y .
Therefore, in order to sufﬁce that R ( Y ) holds only the information present in E , one can restrict that h ( R ( Y ))
≤ h ( E ) + (cid:15) as will be shown in the next lemma.
We note that under the conditions of
Lem.
6, h ( R ( Y ))
≤ h ( E ) + (cid:15) 1 is equivalent to I ( F ( X ); R ( Y )) ≤ (cid:15) 2 for (cid:15) 1 , (cid:15) 2 that are functions of each other.
Lemma 7.
In the setting of Lem. 6.
Assume that h ( R ( Y )) ≤ h ( E ) + (cid:15) , for some constant (cid:15) > 0 .
Then, I ( F ( X ); R ( Y )) ≤ L err ∆ · h ( Y ) + (cid:113)
L err ∆ + (cid:15) .
In addition, there are functions r 1 , r 2 , such that,
[Equation]
and
[Equation]
Proof.
By Lem.
10 in Press et al.
(2019), we have:
[Equation]
In addition, by the data processing inequality and the assumption h ( R ( Y )) ≤ h ( E ) + (cid:15) , we have:
[Equation]
Finally
[Equation]
by Feder & Merhav (1994) [Equation] there is a function r 1 , such that,
[Equation]
In addition
[Equation]
there is a function r 2 , such that,
[Equation]
as desired.