Bivariate Causal Discovery via Conditional Divergence
Abstract
Telling apart cause and effect is a fundamental problem across many science disciplines.
However, the randomized controlled trial, which is the golden-standard solution for this, is not always physically feasible or ethical.
Therefore, we can only rely on passively observational data in such cases, making the problem highly challenging.
Inspired by the observation that the conditional distribution of effect given cause, also known as the causal mechanism, is typically invariant in shape, we aim to capture the mechanism through estimating the stability of the conditional distribution.
In particular, based on the inverse of stability – the divergence – we propose Conditional Divergence based Causal Inference ( CDCI ), a novel algorithm for detecting causal direction in purely observational data.
By doing this, we can relax multiple strict assumptions commonly adopted in the causal discovery literature, including functional form and noise model.
The proposed approach is generic and applicable to arbitrary measures of distribution divergence.
The effectiveness of our method is demonstrated on a variety of both synthetic and real data sets, which compares favorably with existing state-of-the-art methods.
1. Introduction
Distinguishing cause from effect is both a fascinating and highly complex problem across many scientiﬁc disciplines.
Knowing the underlying causal mechanism of nature allows us to precisely predict the outcomes of any imaginable interventions.
Ideally, by actively intervening on the variables of interest then analyzing feedback data, the randomized controlled trials (RCT) enable us to discriminate cause from effect in the most reliable way.
However, the imperfect reality disallows us to have such convenient tool available for all interesting problems.
Performing RCT experiments in real life is majorly impossible due to the limitations of resource, expenses, or ethical issues.
Therefore in such cases, passively observational data is the only clue we have to disentangle the hidden mechanism of reality.
In contrary to RCTs, predicting causal relationships using solely observational data is exceedingly challenging, and sometimes impossible.
Recent years have seen countless breakthroughs in the ﬁeld of machine learning (ML), with the spectacular ability to capture non-linear and highly complex relationships even with unstructured data (e.g., image, text, audio, etc.).
However, current ML systems merely learn correlations from observational data, without understanding the real nature.
For those reasons, the scientiﬁc community has been continuously trying to solve the problem and pushing the boundary of human knowledge about causality mechanisms, which has resulted in a variety of well-developed algorithms.
In this work, we exclusively focus on the two-variable instance of the problem.
That is, given a ﬁnite sample of two univariate variables X and Y , determine whether X causes Y ( X → Y ), Y causes X ( Y → X ), or there is no causal relationships between them.
This problem setting has also received enormous attentions from researchers in the last
[Figure]
decades.
The majority of these methods base on the formalism of functional causal models (FCM) (Pearl, 2009), where the effect is assumed to be generated from the cause via a functional mechanism.
Methods following this formalism aims to exploit the inherent asymmetry between cause and effect, and employ an array of different assumptions ranging from realistic to highly restrictive.
Linear non-Gaussian acyclic models (LiNGAM)
(Shimizu et al., 2006), additive noise models (ANM) (Hoyer et al., 2008), and post-nonlinear model (Zhang and Hyv¨arinen, 2009) address the problem by assuming a functional form on the causal mechanism, as well as a noise model.
In information geometric causal inference (IGCI), Daniuˇsis et al.
(2010) pay attention to a more limited scenario, where the causal mechanism is assumed to be invertible with little or no noise allowed.
Other exciting frameworks have also been applied against this challenge, such as minimum description length (MDL) principle (Marx and Vreeken, 2017; Mian et al., 2021), generative neural networks (GNN) (Goudet et al., 2018), and meta learning (Ton et al., 2021).
Present work.
In detecting potential binary causal relations, we follow the assumption that the causal mechanism should be stable for the true causal direction (Janzing and Sch¨olkopf, 2010).
In other words, if X → Y is the true causal relation, for example, then the conditional distribution of the effect given the cause P ( Y | X ) is independent with the marginal distribution of the cause P ( X ) .
This is in accordance with the postulate that the shape of the conditional distribution P
( Y | X = x ) is invariant for different values of x (Fonollosa, 2019).
Particularly, in this paper, we propose to capture the stability of the conditional distribution, by measuring its inverse – the divergence of conditional distributions, with a novel algorithm called conditional divergence based causal infer- ence ( CDCI ).
1 The gist of this idea is that the more the conditional distributions of the assumed effect P ( Y |
X
) diverge, conditioned on the assumed cause ( X ), the less likely X → Y is the true relation between the two variables.
Figure 1 illustrates the motivation of our approach, where a measure of conditional divergence can be jointly combined with an independence test to derive a useful and understandable rule for orienting causal relations.
Several variants of CDCI , different in the divergence measures, are implemented, including
Chi-squared distance (Pearson, 1900), Hellinger distance (Diaconis and Zabell, 1982),
KullbackLeibler divergence (Kullback and Leibler, 1951), Kolmogorov metric (Kolmogorov, 1933), and total variation distance (Gibbs and Su, 2002).
We compare the performance of our methods against state-of-the-art baselines.
The experimental results demonstrate that our CDCI variants are able to consistently outperform traditional causal direction detectors on a variety of simulated and real datasets.
For instance, on a mixed simulated and real dataset, while the best baseline gains an AUC of 0.742, our proposed CDCI methods achieve a much higher AUC, ranging from 0 .
865 to 0 .
Contributions.
The key contributions of our study can be summarized as follows:
We propose an approach to estimate the average divergence of normalized conditional distributions.
This approach is generic, which smoothly accommodates several well-found measures of distribution divergence.
We provide a set of conditional divergence based predictors of causal direction.
We demonstrate the effectiveness of the proposed conditional divergence based methods against state-of-the-arts in predicting causal directions on both simulated and real data sets.
The paper is structured as follows:
In section 2 we present the key concepts of causal discovery and highlights the state-of-the-arts in the ﬁeld.
Next, section 3 explains the proposed CDCI method that solves the bivariate causal discovery problem based on the stability of conditional distributions.
Subsequently, section 5 presents the experimental settings and main results on the effectiveness of our method in predicting causal orientation.
Finally, in section 6 we review the contributions of our study and provide directions for future developments.
2. Background
Here we highlight general approaches that have been tried to address the causal direction prediction problem or similar issues.
2.1. Preliminaries
Following the FCM formalism (Pearl, 2009), we assume that the causal relationship is non-bidirectional and the effect is generated as a function of the cause and a noise:
[Equation]
where (cid:15) is the background noise associated with Y .
In many formalizations, X and (cid:15) are assumed to be independent.
However, in this work we allow for the dependence between them in acknowledgment for cases that the effect is generated by both the cause and a confounder, for example:
[Equation]
where Z is a confounder, i.e., the common cause between X and Y .
That is, our method aims to detect the true causal direction despite the presence of confounders.
Then, the goal of differentiating cause and effect problem is to classify an empirical bivariate joint distribution P ( X, Y ) into one of three main categories: X → Y ( X causes Y ), Y → X ( Y causes X ), and ‘non-causal’ relationship, such as X ↔ Y (neither X or Y causes the other but both are inﬂuenced by another factor) and X ⊥ Y
( X and Y are independent).
Following the
Cause-Effect Pairs Challenge 2013 (Guyon and Statnikov, 2019) and many closely related works (Fonollosa, 2019; Goudet et al., 2018, 2019; Mooij et al., 2016; Ton et al., 2021), three classes
X → Y , Y → X , and ‘non-causal’ are numerically encoded into 1 , − 1 and 0 , respectively.
The prediction is allowed to be any real number, which can be interpreted as the ranking score, where positive values stand for X → Y , negative values suggest Y → X , and zero indicates no causal relationships.
2.2. Related works 2.2.1. M ULTIVARIATE METHODS
From a wider perspective, the causal relation prediction task is not generally restricted to only two variables.
There have been great successes in ﬁnding causal relationships among a causal network of multiple variables (Spirtes et al., 2000; Chickering, 2002;
Tsamardinos et al., 2006; Ogarrio et al., 2016; B¨uhlmann et al., 2014; Mian et al., 2021; Dhir and Lee, 2020).
One of the traditional approaches is constraint-based methods (Spirtes et al., 2000), which exploit the conditional independence relationships among variables to recover the causal structure.
However, the conditional independence tests are not applicable to the bivariate case because they require at least one more variable to condition on, apart from the two we have, and they cannot always orient all edges in a causal network.
Alternatively, score-based methods (Chickering, 2002) search for the causal graph with highest ﬁtting score, which can be computationally intensive and inaccurate.
Hybrid methods (Tsamardinos et al., 2006; Ogarrio et al., 2016), which combine constraint-based and score-based approaches, have been shown to be more accurate in many situations than the original approaches.
In general, to orient causal directions, the majority of multivariate methods rely greatly on the global structure of the causal network, which cannot be exploited in the case of only two variables.
2.2.2. A SYMMETRY BETWEEN CAUSE AND EFFECT
In the bivariate setting, methods inspired by the notion of asymmetry between cause and effect are the most widely recognized.
A foundational approach is proposed by Dodge and Rousson (2000, 2001), who exploit the correlation coefﬁcient in linear models to develop asymmetric indicators (e.g., skewness ratio) that can tell apart the explanatory and the response variables.
Another early and successful attempt to realize this idea is based on the additive noise assumption (Shimizu et al., 2006; Hoyer et al., 2008; Shimizu et al., 2011; Zhang and Hyv¨arinen, 2009;
Mooij et al., 2016).
These methods assume that Y is generated as a function of X plus an independent noise: Y :
= f ( X ) + (cid:15), X ⊥ (cid:15) .
Many approaches have followed this direction and partially exploited the linear (Shimizu et al., 2006, 2011), non-linear (Hoyer et al., 2008), and post-nonlinear relationships (Zhang and Hyv¨arinen, 2009).
Recently, (Bl¨obaum et al., 2019) show that if both variables are equivalently scaled and the functional mechanism is nearly deterministic, then the regression error is smaller in the true causal model, so the causal direction can be inferred by comparing the residuals after ﬁtting a function (e.g., polynomial, logistic function, or neural network) on both possible causal directions.
In addition, under the “equal variance” assumption, where the noise terms have a common variance, identiﬁability can be ensured in several models by exploiting the order of variables induced by their variances (Peters and B¨uhlmann, 2013; Chen et al., 2019; Park and Kim, 2020).
Overall, the aforementioned methods typically impose very strong assumptions on the functional mechanism family and the properties of the noise variables, which are usually not readily available in practice, thus making them very limited for a vast variety of applications.
2.2.3. L EARNING - BASED APPROACHES
A recent class of methods attempts to adapt learning algorithms to classify causal directions.
In randomized causation coefﬁcient (RCC) (Lopez-Paz et al., 2015) and neural causation coefﬁcient (NCC) (Lopez-Paz et al., 2017), causality is posed as a classiﬁcation problem, where the input bivariate distribution is featurized using either kernel mean embeddings or neural networks, which is subsequently used as the input for a binary classiﬁer.
Apart from that, non-linear SEM estimation using non-stationarity (NonSENS) (Monti et al., 2020) and causal mosaic (Wu and Fukumizu, 2020) base their approaches on the non-linear independent component analysis methods, implemented by neural networks.
Causal generative neural networks (CGNN) (Goudet et al., 2018) adopts generative neural networks to model the causal relationships for both directions, then compare them to decide the causal direction.
Meta-CGNN (Ton et al., 2021) extends this idea with meta learning to allow for the employment of similar data sets to predict causal direction of unseen data.
2.2.4. I NDEPENDENCE BETWEEN CAUSE AND MECHANISM
Most similar to our method is a branch of works that rely on the postulate called ‘independence between cause and mechanism’, which states that for the true causal direction X
→ Y , the conditional distribution of effect given cause P ( Y | X ) is independent of the marginal distribution P ( X ) , but not for the opposite direction.
Information geometric causal inference (IGCI)
(Daniuˇsis et al., 2010) assumes an invertible causal mechanism, with low to no noise.
The authors suppose that the effect is less uniform than the cause, which leads them to deﬁne the measure of independence between cause and mechanism using the entropies of X and Y .
Another line of works postulates that if X → Y is the true causal direction then the total
Kolmogorov complexities (Gr¨unwald and Vit´anyi, 2008) of the marginal distribution P ( X ) and the conditional distribution P ( Y | X ) should be less than the other way around: K ( P ( X ))
+
K ( P ( Y | X ))
<
K ( P ( Y ))
+ K ( P ( X | Y ))
(Janzing and Sch¨olkopf, 2010; Stegle et al., 2010;
Marx and Vreeken, 2017), rendering Occam’s razor philosophy: the simpler explanation is more likely the true one.
Since the Kolmogorov is not computable, they upper bound it by either the minimum message length (MML) principle (Stegle et al., 2010) or minimum description length (MDL) principle (Marx and Vreeken, 2017; Mian et al., 2021).
In causal inference via kernel deviance measures (Mitrovic et al., 2018), the authors develop a fully non-parametric method based on the independence of Kolmogorov complexity of the mechanism in the causal direction and the cause, in which they use the framework of reproducing kernel
Hilbert spaces (RKHS) to measure the variability of Kolmogorov complexities of the mechanism.
An alternative assumption is that if X → Y is the underlying causal relation, then the shapes of conditional distributions of effect given cause P ( Y | X = x ) should be similar for different values of x (Fonollosa, 2019), where the standard deviation of conditionals distributions P ( Y | X ) is used as the the direction score.
3. Conditional Divergence based Causal Inference (CDCI) 3.1. Conditional Divergence
Assumption.
We assume that if X → Y is the true causal direction then the cause, i.e., X , is independent with the geometrical shape of the conditional distribution of the effect Y given the cause X , i.e., P ( Y | X ) .
We emphasize that this assumption is a generalization of several existing assumptions, including:
• Independence between cause and noise: for instance, the generating process Y :=
X + (cid:15) with (cid:15) ∼ U (0 , X ) satisﬁes our assumption even though X (cid:54)⊥ (cid:15) .
• Additive Noise Models ( Y :=
f ( X ) + (cid:15), X ⊥ (cid:15) ) and Multiplicative Noise Models ( Y := (cid:15)
× f
( X ) , X ⊥ (cid:15) ) :
If a bivariate joint distribution P ( X, Y ) admits one of these models, then the conditional distributions of Y given X only differ in either location or scale, respectively.
• Functional form: we do not impose any major restriction on the causal mechanism family.
That being said, CDCI is fully non-parametric and involves a richer family of models.
To realize this idea, we propose to measure the invariance of conditional distributions shapes.
We practically achieve this by estimating the divergence of conditional distributions.
For this purpose, we deﬁne a conditional distribution divergence measure called Normalized Conditional Divergence (NCD) , which estimates the average distance of conditional distributions from a central one.
Generally, NCD of Y given X can be mathematically formulated as:
[Equation]
where D is any valid probability distance measure (e.g., Kullback-Leibler divergence), and (cid:98) Y is a random variable generated for the k -th sample (cid:0) x
( k ) , y ( k ) (cid:1) as follows:
[Equation]
This is essentially the standardization of the conditional distribution, which helps eliminate both location and scale variations, allowing NCD to be inﬂuenced by the distribution shape only.
In our implementation, any continuous variable is discretized before computation as follows: ﬁrst, outliers with deviations greater than a speciﬁed max dev standard deviations are grouped, then the resulting range is divided into 4 × max dev + 1 equally spaced bins.
Since NCD adopts the notion of probability distance, the higher values indicate the higher dissimilarity in the shape of conditional distributions, whereas the minimum value of zero is reached when all conditional distributions share an identical shape.
Regarding the choices for the probability distance D , in subsection 3.3 we showcase some options for this as variants of our method and demonstrate that high performance is achievable regardless of the choice.
3.2. Conditional Divergence based Causal Score
Algorithm 1 The C algorithm
Compute
using the probability distance D , as in Equation (1):
[Equation]
[Equation] [Equation]
the
X → Y and
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
To determine the causal direction, only the conditional distribution divergence on one direction is not enough (i.e., Y given X ).
More particularly, it is usually subjective and not trivial to tell whether an arbitrary conditional divergence is too small or too large.
For those reasons, we furthermore devise a conditional divergence based causal score as the direct indicator for the causal direction.
The causal score for direction X → Y is given by:
[Equation]
[Equation]
The conditional divergence based causal score is a powerful causal direction indicator and clearly interpretable: a positive value means that the conditional distributions of Y given X are more similar and hence X
→ Y is more likely to be the true orientation, while a negative value suggests the opposite, and the value of zero is a sign of two variables being independent.
The intuition is as follows: if X is a true cause of Y , then the conditional distributions P ( Y | X ) diverges less
[Table]
in shape, entailing NCD ( Y | X ) is lower than NCD ( X | Y ) , making NCD ( X | Y ) − NCD
( Y | X ) positive.
The prediction causal direction can then be obtained with a decision threshold of approximately zero.
To sum up, Algorithm 1 illustrates the big picture of our framework.
3.3. Variants of CDCI
Here we brieﬂy describe ﬁve variants of CDCI where each one adopts a different probability metric commonly used in statistics and machine learning (see Gibbs and Su (2002) for a wider range of metrics and their interconnections).
Each considered distance measure takes two probability density functions P and Q as input and returns their distance:
• Variant 1: CCS for the Chi-squared distance (Pearson, 1900):
[Equation]
• Variant 2: CHD for the Hellinger distance (Diaconis and Zabell, 1982): D HD
[Equation]
P, Q ) :
= (cid:114)(cid:88) (cid:16)(cid:112) P ( x ) − (cid:112) Q ( x ) (cid:17)• Variant 3: CKL for the Kullback-Leibler divergence (Kullback and Leibler, 1951):
[Equation]
• Variant 4: CKM for the Kolmogorov metric (Kolmogorov, 1933):
[Equation]
• Variant 5: CTV for the Total Variation distance (Gibbs and Su, 2002):
[Equation]
For each of those variants, the placeholder D in equation (1) is substituted with its corresponding probability metric.
Table 1 presents some properties of the CDCI variants and their associated distance measures.
4. Identiﬁability
Using CDCI , the identiﬁability of the causal direction is ensured if exactly one of P ( Y | X ) and
P ( X | Y ) is invariant in shape, which can be determined via the closed-form expressions of the conditional distributions.
For example, if P ( Y | X ) and P ( X | Y ) are Gaussian or Uniform, then regardless of parameters, the causal direction is unidentiﬁable.
In what follows, we investigate the identiﬁability of a family of causal models where the shape invariance is already met in the true direction.
Proposition 1.
Given X ⊥ (cid:15) , g (cid:54)≡ 0 , and P ( (cid:15) ) is symmetrical, then any causal model Y :
= f ( X ) + (cid:15) × g ( X ) has P ( Y | X ) invariant in shape.
Proof.
The conditional normalization of Y is (cid:98) Y | X
= x = Y − E
[ Y | X = x ] √ Var
[ Y | X = x ] = ± (cid:15) − E
[ (cid:15) ] √ Var[ (cid:15) ] , which follows a symmetrical distribution independent of X .
Furthermore, if g ( x ) > 0 ∀ x ∈ supp ( X ) or g ( x ) < 0 ∀ x ∈ supp ( X ) then P ( Y | X ) is shape-invariant regardless of the asymmetry of P ( (cid:15) ) .
While Proposition 1 characterizes a causal model family satisfying the shape invariance criteria for the true direction, it does not guarantee the opposite for the false direction.
To see this, consider g ( X ) ≡ 1 , (cid:15) ∼ N (cid:0) 0 , σ 2 (cid:1) , and f ( X ) is linear, then the model reduces to the well-known linear
Gaussian additive noise model, which is unidentiﬁable since both P ( Y | X ) and P ( X | Y ) are Gaussian with the same bell-curve shape.
Therefore, if the shape of P ( X | Y ) is dependent on Y , then we can distinguish the cause from the effect.
Consequently, given the speciﬁcation of a generative model following the form characterized by Proposition 1, we can determine its identiﬁability by analytically examining the conditional normalization (cid:98) X | Y
=
y
=
X − E
[ X | Y =
y ] √ Var
[ X | Y
=
y ] .
If the closed-form simpliﬁcation of (cid:98) X | Y =
y involves y then we can conclude that the shape of P ( X | Y ) depends on Y , hence identiﬁability is ensured, and vice versa.
Following this procedure, one can show that under simple conﬁgurations where f, g ∈
Constant , Linear } and P ( X ) , P ( (cid:15) )
∈ { Uniform , Gaussian } , only in two cases ( f, g, P ( X ) , P ( (cid:15) ))
∈ { ( Linear , Constant , Gaussian , Gaussian ) , ( Linear , Constant , Uniform , Uniform ) } wherein P ( X | Y ) is Gaussian and Uniform, respectively, the causal direction is not recoverable.
This suggests that in general, with more complex settings, P ( X | Y ) is less likely to be shape-invariant and CDCI can correctly predict the causal direction in most situations.
5. Experiments
To demonstrate the effectiveness of our proposed CDCI approach, we conduct intensive experiments on a broad variety of both synthetic and real data sets.
In this section, we ﬁrst describe the experiment settings, including data sets, baseline methods, and evaluation criterion.
Then, we provide results on the performance of our method in comparison with state-of-the-arts and the stability of our method under different settings.
Finally, we conclude the section with some discussions.
5.1. Synthetic Data
For synthetic data sets, we employ the following ﬁve different data sets, which are also used in closely similar experiment settings from (Goudet et al., 2018; Ton et al., 2021):
CE-Cha (Guyon et al., 2019): 300 pairs of both continuous variables from the Cause-Effect Pairs
Challenge 2013 (Guyon and Statnikov, 2019), with labels restricted to either X →
CE-Multi (Goudet et al., 2018):
The CE-Multi data set includes 300 generated pairs of continuous variables, with post-additive noise ( Y :
= f ( X ) + (cid:15), X ⊥ (cid:15) ), pre-additive noise ( Y :=
f ( X + (cid:15) ) , X ⊥ (cid:15) ), post-multiplicative noise ( Y :=
f ( X ) × (cid:15), X ⊥ (cid:15) ), and pre-multiplicative noise ( Y :=
f ( X × (cid:15) ) , X ⊥ (cid:15) ) models.
CE-Net (Goudet et al., 2018): CE-Net is comprised of 300 numerical pairs generated using neural networks with multiple choices for the noise distribution of the cause, including Exponential,
Gamma, log-normal, and Laplace distributions.
CE-Gauss (Mooij et al., 2016):
This data set consists of 300 numerical pairs artiﬁcially generated as follows:
X := f X ( (cid:15) X ) and Y :=
f Y ( X, (cid:15) Y )
where (cid:15) X and (cid:15) Y are randomly drawn from Gaussian mixture distributions, whereas f X and f Y are randomly generated Gaussian processes.
CE-All: a concatenation of all four data sets above, creating a total of 1200 data pairs of continuous variables.
The target set for all those data sets is { X → Y, Y → X } with roughly balance quantities.
For each pair we have 1500 samples.
All four data sets are publicly available and can be freely downloaded (Goudet et al., 2018).
5.2. Real Data
In addition to synthetic data sets, we furthermore evaluate our methods and baselines on ﬁve real data sets from the Dialogue for Reverse Engineering Assessments and Methods challenge 3 , ninth edition (referred to as DREAM4) in 2009 (Marbach et al., 2009, 2010; Prill et al., 2010), where the ground-truth causal networks are available.
In the challenge, participants are asked to infer the topology of regulatory networks given gene expression data.
Each data set includes a ground truth transcriptional regulatory network of Escherichia coli and Saccharomyces cerevisiae , as well as observations of gene expression measurements.
For causal orientation task, we extract all directed edges and several pairs with no connection from the networks to create data sets containing all three causal classes X
→ Y , Y → X , and ‘non-causal’ (which includes both sub-classes
X ← Z
→ Y and X ⊥ Y ), with equal proportions for each class.
Details for the data sets used are described as follows:
• D4-S1 is from the sub-challenge one of the contest, containing 36 variable pairs withsamples for each pair.
• D4-S2A , D4-S2B , and D4-S2C are from the second sub-challenge, with 528 , 747 , andpairs of variables, respectively, and each pair contains 210 samples.
• D4-All is a concatenation of the above four data sets, creating a total of 1890 pairs.
All the data sets are publicly accessible and can be freely downloaded.
5.3. Baseline Methods and Evaluation Metric
Similarly to several related studies (Goudet et al., 2018; Wu and Fukumizu, 2020; Ton et al., 2021), we compare the predictive power of our methods, including CCS, CHD, CKL, CKM, and CTV variants, against four competing methods which are brieﬂy described below:
Additive noise model (ANM) (Mooij et al., 2016) with Gaussian Process mechanism and HSIC test for independence between cause and residuals.
Conditional distribution similarity (CDS) (Fonollosa, 2019) measures the variability of conditional distributions via standard deviation.
The variability of conditional distributions for both ( X, Y ) and ( Y, X ) are compared to decide the causal direction.
Information geometric causal inference (IGCI) (Daniuˇsis et al., 2010) compares the entropies of two variables and sets the one whose lower entropy as the cause..
Regression error based causal inference (RECI) (Bl¨obaum et al., 2019) ﬁts a monomial of degree three to the normalized variables then compare the goodness of ﬁt for both directions, in terms of mean squared error.
The implements of the Causal Discovery Toolbox (Kalainathan et al., 2020) for these methods are used with default parameters.
For performance metric, we use the bi-directional area under the receiver operating characteristic curve (AUC, the higher the better), which is the average of two AUC scores.
The ﬁrst AUC score measures the performance in classifying whether a pair ( X, Y ) belongs to X → Y class or not, while the second AUC score accounts for the class Y → X .
Regarding parameters for our methods, based on sample size, we use max dev = 3 and max dev = 2 for the discretization step in synthetic and real data, respectively.
5.4. Results 5.4.1. S YNTHETIC D ATA
Figure 2 shows how our methods consistently outperform all other methods across all synthetic data settings (for detailed ﬁgures, see Table 3 in the Appendix).
Our best methods always achieve higher
AUC compared with competing baselines.
Even the one with worst performance of our methods
[Table]
dominates or is as good as the top competing methods in three out of ﬁve cases (CE-Net, CEGauss, and CE-All).
The most signiﬁcant difference can be seen in the highly representative data set CE-All, where CKM, with AUC of 0.865, surpasses the best baseline score of 0.742 (RECI), despite being the worst performer among variants of CDCI in this case.
Additionally, we are able to maintain an adequately high lower bound of 0.7 in all synthetic data sets, while the performance of baseline methods can be as low as 0.16.
5.4.2. R EAL D ATA
Figure 3 demonstrates that even for real data, our methods are still capable of stably outperforming existing baselines (for detailed ﬁgures, see Table 3 in the Appendix).
The results show that variants of CDCI can provide up to 16% of AUC improvement in comparison with the best baseline methods (in D4-S2B).
Not only that, the worst performers of our variants also have higher scores than the best state-of-the-arts in all ﬁve cases, with the most signiﬁcant difference can be observed in D4-S2B, where the AUC scores of our methods are at least 12% higher than that of competing methods.
5.4.3. I NFLUENCE OF H YPERPARAMETERS
Our proposed method is inﬂuenced by two hyper-parameters, namely the probability metric - as indicated by CDCI variants, and the discretization parameter max dev .
If max dev is too small, it can lead to only a few discretized values ( 5 values for max dev = 1 ), which may be insufﬁcient to capture the conditional distributions accurately.
Meanwhile, if max dev is set too large then our methods may suffer from outliers.
To investigate the effect of hyper-parameters on the performance of our methods, in Table 2 we compare the AUC scores of CDCI variants across different values of max dev in both synthetic and real data.
The empirical results show that there is a visible degrade in performance of all CDCI variants for max dev = 1 and synthetic data, which may be in accordance with our expectation.
However, on the remaining scenarios, the AUC score exhibits no signiﬁcant variation across hyperparameters, where the standard deviations over all settings for synthetic and real data are only 1% and 4% of the average scores, respectively.
However, max dev = 4 should be opted for since it results in the highest average scores (over
CDCI variants) in both synthetic and real data, with the numbers being 0 .
87 ± 0 .
01 and 0 .
66 ± 0 . 01 , respectively.
As for the best CDCI variant, CTV has the highest average AUC scores (over different values of max dev ) in both synthetic and real data sets, which are 0 .
85 ± 0 .
07 and 0 .
63 ± 0 .
03 , respectively, thus it can be preferred over other variants in practice.
5.5. Discussions
Theoretically, having a richer function class should lead to more unidentiﬁable cases, where both directions simultaneously satisfy the assumption so the causal direction cannot be decided.
Yet, ﬁrst theoretical analyses and empirical results have suggested that these cases are rare and does not signiﬁcantly affect the effectiveness of our methods.
The advantages of our method are three-fold.
First, it is generic and adaptable with alternative measures of divergence.
The experimental results have conﬁrmed that CDCI is robust regardless of the choice of divergence measures.
Second, it does not require training, thus it is fast and always ready to be used for unseen data.
Finally, it is insensitive to hyper-parameters and does not rely on randomness as in random sampling methods, hence allowing for reproducibility.
6. Conclusion
To realize the idea that a causal mechanism is stable, we base on the inverse of stability – divergence – to build divergence based causal direction predictors namely CDCI (Conditional Divergence based Causal Inference).
We test CDCI with ﬁve variants, different in the measure of divergence, for the task of causal direction prediction.
We benchmark the performance of our proposed methods against state-of-the-art approaches on a variety of synthetic and real data sets.
We ﬁnd that all the variants of CDCI perform really well on the data sets and are able to consistently outperform existing state-of-the-art solutions.
Future work should involve theoretical aspects of CDCI , wherein the identiﬁability results can be more concretely investigated, as well as extending CDCI to multivariate causal discovery, of which the bivariate causal relation inference studied in this work is a special case.
Appendix A. Detailed Experimental Results
[Table]