A Distance Covariance-based Kernel for Nonlinear Causal Clustering in Heterogeneous Populations
Editors: Bernhard Sch ¨ olkopf, Caroline Uhler Kun Zhang
Abstract
We consider the problem of causal structure learning in the setting of heterogeneous populations, i.e., populations in which a single causal structure does not adequately represent all population members, as is common in biological and social sciences.
To this end, we introduce a distance covariance-based kernel designed specifically to measure the similarity between the underlying nonlinear causal structures of different samples.
Indeed, we prove that the corresponding feature map is a statistically consistent estimator of nonlinear independence structure, rendering the kernel itself a statistical test for the hypothesis that sets of samples come from different generating causal structures.
Even stronger, we prove that the kernel space is isometric to the space of causal ancestral graphs, so that distance between samples in the kernel space is guaranteed to correspond to distance between their generating causal structures.
This kernel thus enables us to perform clustering to identify the homogeneous subpopulations, for which we can then learn causal structures using existing methods.
Though we focus on the theoretical aspects of the kernel, we also evaluate its performance on synthetic data and demonstrate its use on a real gene expression data set.
1. Introduction
Learning causal relationships from observational and experimental data is one of the fundamental goals of scientific research, and causal inference methods are thus used in a wide variety of fields.
The resulting variety of applications nevertheless share some common difficulties, such as causal inference from complex time-series data (Eichler, 2012) or the underlying causal structure being obscured by unmeasured confounders (Greenland et al., 1999).
Another common difficulty, especially for applications in the biological and social sciences, is causal inference from heterogeneous populations (Xie, 2013; Brand and Thomas, 2013)—addressing this difficulty is our main motivation.
In general terms, we understand a heterogeneous population to be one whose members are not adequately described by a single model but rather better described by a collection of models.
Within our context of causal structure learning, this means a population is heterogeneous if some samples are generated by different causal structures—we call this structural heterogeneity.
We note that there are other kinds of heterogeneity, such as that in samples generated by different joint distributions over the same causal structure, but these are outside our scope in this work.
A specific example of structural heterogeneity can be found in genetics: causal methods are used to learn the structure of gene regulatory networks (Altay and Emmert-Streib, 2010), and gene expression data from a single recording or experiment may include thousands of genes, many of which are involved in entirely different networks (Liu, 2015); thus, attempting to learn a single causal structure for all of the genes will obscure the fact that different sets of them have different structures.
The bulk of our work in this paper, and our main contribution, is to introduce the dependence contribution kernel , which facilitates a flexible and easily extensible approach to causal clustering: first perform clustering to identify structurally homogeneous subsets of samples, and then proceed with the actual learning task on each cluster.
We prove that our kernel space is isometric to the space of causal ancestral graphs, and hence our kernel can be used to find clusters that minimize structural heterogeneity for causal structure learning tasks.
Furthermore, the kernel is derived from the distance covariance (Sz´ekely et al., 2007), imbuing it with the ability to detect nonlinear dependence.
It can easily be used in a wide array of clustering algorithms, such as k -means, DBSCAN, spectral clustering, or any other method that analogously makes use of a similarity (or distance) measure between samples (Filippone et al., 2008).
The rest of the paper is organized as follows: We finish this section by discussing some of the most relevant related work from the causal inference and statistics literature.
All of Section 2 is devoted to the theory underlying our dependence contribution kernel, including a comparison of the familiar product-moment covariance with the distance covariance (Section 2.1), defining an equivalence class of causal models with a convenient representation in the kernel space (Section 2.2), and the actual definition of our kernel and proofs of its relevant properties (Section 2.3).
Next, in
Section 3, we supplement the preceding theoretical analysis of our kernel by evaluating its use in a combination of clustering, dimensionality reduction, and structure learning tasks on synthetic and real data.
Finally, we conclude in Section 4 mentioning possible future work.
1.1. Related Work
Causal inference in heterogeneous populations sometimes refers to data-fusion (Bareinboim and
Pearl, 2016), i.e., combining known homogeneous subpopulations and performing causal inference on the resulting heterogeneous population, or similarly, it can refer to meta-learning using known subpopulations (Sharma et al., 2019).
Other times, it refers to estimating heterogeneous treatment effects (Xie et al., 2012; Athey and Imbens, 2015).
However, in our case, the subpopulations are not known and we rather consider the problem of learning which samples come from which subpopulation, and these are differentiated according to structure instead of treatment effect.
Previous work on causal clustering has focused more on the causal modeling aspect, using stronger assumptions about the underlying structures to learn more detailed models.
For example,
Kummerfeld et al.
(2014); Kummerfeld and Ramsey (2016) focus on causal clustering in measurement models, with the goal of clustering different features together to study their latent causal structure, based on tetrad constraints within the linear product-moment covariance matrix.
Huang and Zhang (2019) define a class of causal models facilitating mechanism-based clustering, learning causal models both for clusters of samples as well as a shared one for all samples, assuming the underlying structures are linear non-Gaussian.
Saeed et al.
(2020) characterize distributions arising from mixtures of directed acyclic graph (DAG) causal models (i.e., causal models without latent or selection variables), trying to learn both the component DAGs and a representation of how they are mixed.
All of these approaches, like most causal inference methods, make specific (and for some applications, restrictive) assumptions about the underlying distributions or causal structures.
In contrast, our method is not tied to specific distributional assumptions such as linearity or (non)Gaussianity—we assume there are enough samples for statistical inference, as well as the usual causal Markov and faithfulness assumptions.
For the first step, we cluster samples together if they (implicitly, in the kernel space) have similar nonlinear independence structures.
For the second step, causal structure learning, any existing method (along with its corresponding assumptions) can in principle be used.
In our gene expression data application (Section 3.2), the measurement dependence inducing latent (MeDIL) causal model framework (Markham and Grosse-Wentrup, 2020), which assumes the data consists of measurement variables that are causally connected only through latent variables, seems appropriate, however other applications can easily use other methods.
For example, component and mixture DAGs (Saeed et al., 2020) can be better learned when one first knows which samples come from which component—clustering with our kernel ensures samples in different clusters come from different DAGs, and so using their method instead of the MeDIL framework would be a natural choice for applications in which a DAG (without any latents) is more appropriate.
Finally, there is some work from the statistics literature that sounds superficially similar to our distance covariance-based kernel but is conceptually quite different.
Namely, another well-known measure of nonlinear independence, the Hilbert-Schmidt Independence Criterion (Gretton et al., 2005, 2007), is part of a class of reproducing kernel Hilbert space- (RKHS-) based dependence measures that Sejdinovic et al.
(2013) show is equivalent to distance-based measures such as the distance covariance.
Our dependence contribution kernel, unlike these, is not a dependence measure between features—it rather uses the distance covariance to measure the similarity of samples based on patterns in the dependence structure of their features, and is rather more like a whole-graph embedding (Cai et al., 2018; Maddalena et al., 2020).
2. Theory 2.1. Product-moment Covariance, Distance Covariance, and Dependence Contribution
Though there is more to causal relationships than probabilistic dependence, causal inference methods based on graphical models ultimately rely on at least implicitly learning conditional independence (CI) relations.
CI relations can be estimated in many ways, with different dependence measures and tests each having their own theoretical guarantees and being better suited for distributions of various different kinds of data (e.g., categorical, discrete, or continuous) and with various kinds of relationships (e.g., linear, monotonic nonlinear, arbitrary nonlinear) and with different testing assumptions (see Tjøstheim et al., 2018, for a comprehensive overview).
A widely used measure of dependence is the product-moment covariance , often just called covariance, which is defined for two zero-mean random variables X 1 and X 2 as the scalar value cov( X 1 , X 2 ) =
E[ X 1 X 2 ] .
This can be extended from a pair of random variables to every pair of variables in a random vector, thus returning a matrix instead of a scalar.
The covariance matrix for a vector of zero-mean random variables X =
( X 1 , . . .
, X m ) can be estimated from a set S ∈ R n,m of n samples as ˆΣ X = 1 n S ⊤ S , and the j, j ′ -th value of ˆΣ X is thus the estimate ˆcov( X j , X ′ j ) .
Probabilistic independence (denoted ⊥⊥ ) of two random variables implies that their productmoment covariance is zero, i.e., X j ⊥⊥ X j ′ =
⇒ cov ( X j , X j ′ )
=
(importantly, the inverse of this does not hold).
Thus, the estimated product-moment covariance can be used in statistical hypothesis testing for independence (Wasserman, 2013, Ch. 10): X j and X j ′ are taken to be independent if and only if ˆcov( X j , X j ′ ) is sufficiently close to 0.
However, this method has an important flaw: the product-moment covariance as a test statistic is only valid against linear dependence.
Sz´ekely et al.
(2007) introduce the distance covariance to remedy this problem: random variables are probabilistically independent if and only if their distance covariance is zero, i.e., X j ⊥⊥ X j ′ ⇐⇒
dCov( X j , X j ′ )
= 0 , resulting in the estimated distance covariance being a valid test statistic against all types of dependence.
The distance covariance is related to the product-moment covariance by dCov 2 ( X j , X j ′ )
=
cov( | X j − X
j | , | X j ′ − X
′ j ′ | ) − 2cov( | X j − X
j | , | X j ′ − X ′′ j ′ | ) , where ( X ′ j , X ′ j ′ ) and ( X ′′ j , X ′′ j ′ ) are independent and identically distributed (iid) copies of ( X j , X j ′ )
(Sz´ekely and
Rizzo, 2014).
The key intuition here is that the distances (e.g., | X j − X
′ j | ) constitute a nonlinear projection, so that using the linear product-moment covariance in this projected space allows for the detection of nonlinear dependence in the original space.
Note that dCov is typically defined to be a scalar value when taken between two arbitrarydimensional random vectors, but our restricted presentation of it above in terms of random variables is to make it more obviously analogous to the product-moment covariance between random variables.
Thus, corresponding to ˆΣ X for random vectors, we define the following:
Definition 1 Let S ∈ R n,m be a set of n samples from the vector of random variables
X
=
( X 1 , . . .
, X m ) .
For each j ∈ { 1 , . . .
, m } and i, i ′ ∈ { 1 , . . . , n } , define the pairwise distance matrix
D j , with values given by D ji,i ′ :
=
| S i,j − S i ′ ,j | .
Now define the corresponding doubly-centered matrices C ji,i ′ :
= D ji,i ′ − ¯ D ji, · − ¯ D j · ,i ′ + ¯ D j · , · , where putting a bar over the matrix and replacing an index i or i ′ with · denotes taking the mean over that index.
Define the matrix L ∈ R n 2 ,m so that each column is a flattened doubly-centered distance matrix, L :=
(vec( C 1 ) , . .
, vec( C m )) , where vec( C j ) denotes “flattening” matrix C j into a column vector.
Finally, the estimated distance covariance matrix over sample S is defined as:
[Equation]
Analogous to ˆΣ X , the j, j ′ -th entry of ˆ∆ X corresponds to ˆ dCov 2 ( X j , X j ′ ) —indeed it is mathematically equivalent to computing each pairwise distance covariance value and then manually filling in the matrix.
The novelty of our Definition 1 is in finding a matrix of pairwise values instead of a single value for the distance covariance between random vectors, which helps provide an intuition for our next definition:
Definition 2 Let S ∈ R n,m be a set of n samples from the vector of random variables
X
=
( X 1 , . . .
, X m ) ; note that we consistently use indices i, i ′ ∈ { 1 , . . . , n } and j, j ′ ∈ { 1 , . . .
, m } .
Let D ∈ R n,n,m denote the 3-dimensional array of stacked pairwise distance matrices defined by
D i,i ′ ,j := | S i,j − S i ′ ,j | , and use C ∈ R n,n,m to denote these same distance matrices after being doubly-centered, i.e., C i,i ′ ,j :=
D i,i ′ ,j − ¯ D i, · ,j − ¯ D · ,i ′ ,j + ¯ D · , · ,j , where replacing an index i or i ′ with · denotes the entire (lower-dimensional) subarray over that index, and writing a bar, ¯ D , denotes taking the mean over that subarray.
Then standardize the doubly-centered distances to get Z i,i ′ ,j :=
C i,i ′ ,j ¯ D · , · ,j .
Finally, the dependence contribution map , φ : R m → R m,m , is defined as
[Equation]
where T ( α ) ∈ R m,m is a matrix of scaled critical values corresponding to a given significance level α with zeros along the diagonal, i.e., T ( α ) j,j ′
=
(cid:40) 0 , if j = j ′ 1 n χ 21 − α (1) , otherwise, with χ 21 − α (1) being the 1 − α quantile of the chi-square distribution with 1 degree of freedom.
Notice the similarity between Definitions 2 and 1: if we set T ( α ) to be a matrix of 0s and forgo standardization (i.e., use C instead of Z ), then 1 n 2 (cid:80) ni =1 φ ( S i, · ) =
ˆ∆ X .
Now, the differences:
ˆ∆ X is a single matrix computed over an entire set of samples, whereas φ is a map that projects each given sample to the new feature space; each entry of ˆ∆ X is simply a distance covariance value, whereas each entry of the sum of φ ( S i, · ) over i , by using standardization (using Z instead of C ) and subtracting a critical value, corresponds to the result of using a distance covariance value in a statistical hypothesis test for independence—indeed:
Lemma 3 Let S ∈ R n,m be a set of n iid samples from random variables X 1 , . . .
, X m with finite first moments.
For a given significance level α , under the null hypothesis of X j ⊥⊥ X j ′ , rejecting h ∅ if (cid:0) (cid:80) ni =1 φ ( S i, · ) (cid:1) j,j ′ > 0 is a statistically consistent test against all types of dependence.
Proof This follows from (Sz´ekely and Rizzo, 2009, Theorem 5 and Corollary 2) and how φ is defined to correspond to the difference between distance covariance and critical values.
These differences between ˆ∆ X and φ serve two important purposes: first, they ensure φ maps to a reproducing kernel Hilbert space so that our Definition 9 is a corresponding kernel function (Sch¨olkopf et al., 2001); and second, as the name “dependence contribution map” suggests, they ensure φ ( S i, · ) is informative not just about distance covariance but about nonlinear dependence and about how the inclusion of sample S i, · in a set of samples S contributes to the dependence patterns estimated from S — this is the key intuition behind how our kernel function is used to learn structurally homogeneous sample subsets, as explicated in the following sections.
Compared to other general measures of dependence, such as mutual information Shannon (1948); Cover (1999) or ball covariance Pan et al.
(2019), we chose the distance covariance in particular because it has mathematical properties that are convenient (if not necessary) for defining the kernel: namely, (1) the pairwise distance correlation matrix can be rewritten in terms of the per-sample covariance (or dependence, if one subtracts a critical value) contribution matrix, which facilitates comparing two samples and thus defining the kernel—this is for example complicated by the logarithms used in defining mutual information; and (2) the distance covariance under the null hypothesis provably approaches the chi-square distribution in the sample limit, facilitating the direct computation of critical values corresponding to a given significance level ( T ( α ) in Definition 2), whereras estimating critical values in the case of mutual information or ball covariance using permutations would be computationally infeasible.
2.2. Causal Graphs in Kernel Space
In general, a full causal structure can only be learned with sufficient data about the effects of interventions, and thus causal structure learning from purely observational data is usually possible only up to an equivalence class of causal graphs (Spirtes et al., 2000; Pearl, 2009).
For example, the classic PC and IC algorithms, under the assumptions of no selection bias and no confounding by latent variables, do not necessarily return a fully-specified DAG but instead return a mixed graph, containing possibly directed and undirected edges, representing the Markov equivalence class (Spirtes and Glymour, 1991; Pearl and Verma, 1995).
We now define a set of equivalence classes for ancestral graphs (AGs), which—unlike causal
DAGs—do not assume the absence of selection bias and latent confounders (Richardson et al., 2002):
Definition 4 Consider an arbitrary ancestral graph A with the set of vertices V A and edge function
E A , and denote the set of unconditional m -connection statements entailed by their corresponding unique maximal ancestral graph as M A
= { ( j, j ′ ) :
j / ⊥ m j ′ | ∅} ⊆
V A × V A .
For any ancestral graph A ′ such that V A
′ =
V A , define the unconditional equivalence relation denoted by ‘ ∼ U ’ as
A ∼ U A ′
if and only if
M A = M A
Lemma
This lemma has two parts: (i) the relation ∼ U is an equivalence relation over the set of ancestral graphs A ; (ii) for an arbitrary ancestral graph A ∈ A , the bidirected graph
U A =
( V A , E U ) , where E U maps all pairs ( j, j ′ )
∈
M A to the bidirected edge symbol ‘ ↔ ’, is a unique representative of the equivalence class [ A ] .
Proof For (i), recall that an equivalence relation is any relation satisfying reflexivity, symmetry, and transitivity (Devlin, 2003), all of which are satisfied by ∼ U because of its correspondence to the relation ‘ = ’ between sets.
Thus, to prove (ii), it suffices to show that the map s : A / ∼ U →
A , [ A ] (cid:55)→ U A is injective (i.e, that it is a section ) and that [ s ([ A ])] =
[ A ] (Mac Lane, 2013).
The key to the proof is the observation that U A , because it contains only bidirected edges, is maximal and therefore entails exactly the unconditional m -separation statements M A , thus by (i) we have
U A ∼ U A or equivalently
U A ∈ [ A ] or equivalently [ U A ] =
[ A ] .
Let A , A ′ be arbitrary AGs, and assume s ([ A ])
=
s ([ A ′ ]) .
Then by definition of s we have U
A = U A ′ , and by the observation above,
U A ∈ [ A ′ ] and thus [ A ] =
[ A ′ ] , making s injective.
And finally, by the definition of s and also by the observation above, [ s ([ A ])] =
[ U A ] =
[ A ] , completing the proof.
This equivalence relation and its representatives has some important but perhaps subtle properties.
First, it is different from Markov equivalence over AGs (which is characterized by partial ancestral graphs, PAGs) (Zhang, 2007)—it uses only unconditional m -separation while PAGs are learned from conditional m -separation statements.
Second, because all DAGs are AGs, ∼ U is also an equivalence relation over DAGs.
Third, being a representative means that every equivalence class includes exactly one fully bidirected graph (along with other equivalent AGs).
Fourth, because each representative is formed by considering m -connected paths, U A is not equivalent to what would be generated by some “edge-wise” procedure, such as simply replacing every edge in a PAG/AG/DAG/Markov random field/moralized DAG with bidirected edges.
Finally, its most important property is that it facilitates Theorem 8, for which we first need a few more definitions.
Definition
Given arbitrary ancestral graphs A ,
A ′ ∈
A over the same set of vertices, define the Hamming similarity product, denoted ‘ • ’, as • : A × A → A and
A • A ′ (cid:55)→ H , where
H =
( V A , E H ) and the function E H ( j, j ′ )
= ‘ ↔ ’ if and only if E A ( j, j ′ ) =
E A ′
( j, j ′ ) .
In words, the Hamming similarity product between two ancestral graphs returns a fully bidirected graph, with edges only where the two graphs have the same edge type.
Now, shifting from ancestral graphs to real-valued square matrices:
Definition 7 Let ‘ ∼ O ’ denote the orthant equivalence relation (‘orthant’ is the generalization of ‘quadrant’ from R 2 to arbitrarily higher dimensions) in square real matrices, i.e., for matrices
Y, Y ′ ∈ R m,m and with the element-wise function sign( Y ) j,j ′
= (cid:40) 1 , if Y j,j ′ > 0 or j = j ′ − 1 , otherwise ,
Y ∼ O Y ′
if and only if sign( Y ) j,j ′ = sign( Y ′ )
j,j ′ for all j, j ′ .
Theorem 8 Let a be the map from the set of unconditional equivalence classes over ancestral graphs with m vertices, A m / ∼ U
=
U m , to the set of orthant equivalence classes over the image of φ , i.e., m × m symmetric real matrices with positive diagonal entries, φ ( R m ) / ∼ O
= O m , defined by a : U (cid:55)→ O , where O j,j ′ = (cid:40) 1 , if E U ( j, j ′ )
= ‘ ↔ ’ or j = j ′ − 1 , otherwise .
Then a is a group isomorphism between ( U m , • ) and ( O m , ⊙ ) , where ‘ ⊙ ’ denotes the element-wise product.
Proof First, note that ( U m , • ) is indeed a group, satisfying the three group axioms (Artin, 2011): the representative of its identity element is the fully connected bidirected graph over m vertices, U 1 ; each element is its own inverse; and • is associative.
Likewise, ( O m , ⊙ ) is a group with identity element [ 1 m,m ] , each element its own inverse, and the associative element-wise product operator.
Now, to show the two groups are isomorphic, it suffices to show (i) that a is bijective and (ii) that for arbitrary U , U ′ ∈
U m , a ( U ) ⊙ a ( U ′ )
= a ( U • U ′ ) .
For (i) notice that if U ̸ = U ′ , then there must be at least one pair of vertices j, j ′ such that E U ( j, j ′ )
̸
= E U ′ ( j, j ′ ) and thus clearly O j,j ′ ̸
= O ′ j,j ′ , so a in injective.
Furthermore, notice that every distinct O ∈
O m is the image of some graph U , so a is also surjective.
For (ii), for every j, j ′ ∈ { 1 , . . .
, m } , the definitions of a , ⊙ , and • ensure a ( U ) j,j ′ ⊙ a ( U ′ )
j,j ′ =
1 ⇐⇒ E U ( j, j ′ )
=
E U ′
( j, j ′ )
⇐⇒
= a ( U • U ′ ) , completing the proof.
For causal inference, which (often, but not necessarily) amounts to taking several samples in real space and inferring a single corresponding member in the space of ancestral graphs (or, more often, its quotient set by some equivalence relation), Theorem 8 means we can compare the different graphs of different sample sets without having to first move to the ancestral graph space.
Finally, notice the space of real square matrices is not a typical sample space but rather precisely (a superspace of) the space that our dependence contribution map φ (Definition 2) maps samples to—this means that mapping samples with φ allows us to make use of the group isomorphism.
Though this already provides an intuition for why using φ would help with causal clustering, explicitly mapping each sample with it would be unnecessarily computationally expensive, and we are ultimately interested in morphisms between metric spaces (not just groups) of samples and graphs.
To address this, we therefore now move on to defining a kernel for φ .
2.3. The Dependence Contribution Kernel
Definition 9 Let S, Z, T , and φ be as in Definition 2.
We define the dependence contribution kernel using the Frobenius (denoted by the subscript F ) inner product and norm: κ ( S i, · , S i ′ , · ) =
⟨ φ ( S i, · ) ,φ ( S i ′ , · ) ⟩ F ∥ φ ( S i, · )
∥ F ∥ φ ( S i ′ , · ) ∥ F .
A more convenient expression for applying the kernel to a data set is obtained by first defining a helper kernel, γ , using it along with vec from Definition 1:
[Equation]
This allows us to write
[Equation]
Finally, note that κ can be readily implemented on an entire set of samples, returning an entire
Gram (kernel) matrix instead of a scalar value, by replacing the matrix operations above with tensor operations and specifying the correct axes along which summation occurs—an open source Python implementation can be found at https://causal.dev/code/dep_con_kernel.py .
A proper distance metric can also be obtained from this kernel through function composition: arccos ◦ κ .
The key idea behind the kernel is that it is the cosine similarity in the space that φ maps to, meaning for arbitrary sample points x, x ′ it evaluates to cos( θ ) , where θ is the angle between φ ( x ) and φ ( x ′ ) .
In this space, θ represents the dissimilarity of the dependence patterns underlying x and x ′ , without being biased by the possibly different magnitudes of φ ( x ) and φ ( x ′ ) due to differing variances .
Indeed, it can be used as a statistical test of whether samples come from different dependence structures and therefore causal models:
Theorem 10 Let S ∈ R n,m , S ′ ∈ R n ′ ,m be sets of n, n ′ iid samples drawn respectively from the random variables X
=
( X 1 , . . .
, X m ) and X ′ =
( X ′ 1 , . . .
, X ′ m ) with finite first moments.
Then, (cid:80) ni = [Equation] (cid:80) n ′ i ′
=1 κ ( S i, · , S ′ i ′ , · ) < [-
-]0 =
⇒ ∃ j, j ′ ∈ { 1 , . .
, m } such that I ( X j , ∅ , X j ′ )
̸
= I ( X ′ j , ∅ , X ′ j ′ ) .
Proof Through Slutsky’s Theorem (see Takeshi, 1985, Theorem 3.2.7) and the continuous mapping theorem (see Van der Vaart, 2000, Theorem 2.3), the consistency of φ (Lemma 3) guarantees the consistency of κ .
Because the numerator of κ is a Frobenius inner product of φ ,
[Equation]
Thus, in order for (cid:80) i,i ′ κ ( S i, · , S ′ i ′ , · ) < 0 , there must be a j and j ′ for which {+φ ( S i, · ) j,j ′ > 
+}
but φ ( S ′ i ′ , · ) j,j ′ < 0 (or vice versa), and thus the hypothesis test in Lemma 3 would reject the null hypothesis that X j ⊥⊥ X j ′ but fail to reject that X
′ j ⊥⊥ X ′ j ′ .
Corollary 11 Due to the relationship between independence and causal structure, an immediate result of Theorem 10 is that (cid:80) i,i κ ( S i, · , S ′ i ′ , · ) < 0 implies X and X ′ have different causal structures.
Theorem 12 Let d be the distance measure between unconditional equivalence classes of ancestral graphs over m vertices, d ( U , U ′ )
= m 2 − |{ ( j, j ′ ) :
E U •U ′ ( j, j ′ ) =
‘ ↔ ’ }| − m .
For given sample sets S, S ′ (i.e., real n × m matrices), use ¯ φ ( S ) to denote the mean of the sample in kernel space, (cid:80) i φ ( S i,. ) , and say S ∼ K S ′
if and only if ¯ φ ( S ) ∼ O ¯ φ ( S ′ ) ; denote the corresponding quotient set by this equivalence class as R n,m / ∼ K = K n,m and a representative from each equivalence class as Q ∈ [ S ] .
Let δ be the distance between sets of samples in K defined as δ ( Q, Q ′ )
= m 2 − 12 n 2 (cid:80) i,i ′ γ
( Q i, · , Q ′ i, · ) .
Let b : U m → K n,m , b : U (cid:55)→ Ω , where Ω is the unique element in
K such that sign( ¯ φ (Ω))
=
a ( U ) .
Then b is a distance-preserving map (i.e., an isometry) from the metric space ( U m , d ) to ( K n,m , δ ) .
Proof Notice that ( U m , d ) is indeed a metric space (Choudhary, 1993, Ch. 2): d ( U , U ′ )
=
iff
U − 1 • U ′ is the empty graph, which happens iff U =
U ′ ; the symmetry of d follows from the symmetry of • ; and for subadditivity of d , observe that for vertices j, j ′ in arbitrary 2-vertex graphs
U , U ′ , U ” we have either d ( U , U ”)
= 2 , in which case d ( U , U ′ ) + d ( U ′ , U ”)
= 4 , or we have d ( U , U ”)
= 0 , in which case d ( U , U ′ )
+ d ( U ′ , U ”) is either 0 or 4—in both cases d ( U , U ”)
≤ d ( U , U ′ )
+ d ( U ′ , U ”) ; this easily extends to graphs of arbitrary numbers of vertices.
Likewise, ( K n,m , δ ) is a metric space: δ ( Q, Q ′ )
=
⇐⇒ 12 n 2 (cid:80) i,i ′ γ ( Q i, · , Q ′ i, · )
= m 2 ⇐⇒ ¯ φ ( Q ) j,j ′ =
¯ φ ( Q ) j,j ′ , for all j, j ′ , so iff Q = Q ′ ; symmetry and subadditivity of δ follow from the symmetry and subadditivity of γ .
Finally, to show b is an isometry, we must show (i) that it is bijective and (ii) that for all
U , U ′ ∈
U m , d ( U , U ′ )
=
δ ( b ( U ) , b ( U ′ )) .
For (i), observe that by the group isomorphism a and definition of b , we have U ̸
= U ′ =
⇒ a ( U ) ̸ = a ( U ′ )
=
⇒ Q ̸ = Q ′ =
⇒ b ( U ) ̸
= b ( U ′ ) and so b is injective.
Also observe that because K is exactly the set of representatives of orthant equivalence classes of sample sets in kernel space, then for every Q ∈ K , there exists a U such that b ( U )
=
Q , and so b is surjective.
For (ii), isomorphism a and the relation between element-wise product and
Frobenius inner product allow us to write d ( U , U ′ )
= m 2 − (cid:80) j,j ′ ( O ⊙ O ′ )
j,j ′ =
m 2 − ⟨ O, O ′ ⟩ F .
Substituting O, O ′ with their corresponding Ω , Ω ′ , and because the Frobenius inner product is a sesquilinear form, we can write d ( U , U ′ )
= m 2 − 1 n 2 (cid:80)
i,i ′ ⟨ φ (Ω i, · ) , φ (Ω ′ i, · ) ⟩ F , which by Definition 9 finally gives us that d ( U , U ′ )
=
δ (Ω , Ω ′ ) , completing the proof.
In less formal terms, Theorem 12 shows how the space of unconditional equivalence classes of ancestral graphs corresponds to the space of real matrices, which is a common space for samples to lie in.
More specifically, it shows how the structure defined by distances between graphs is the same as the structure defined by distances between sets of samples and how this sample distance is related to our kernel κ .
Note that this is much stronger than Theorem 10: not only can κ tell us that two sets of samples come from different causal models, it gives a measure of just how different the causal models are, in terms of their differing unconditional nonlinear independencies.
To summarize, we began by defining φ (Definition 2), which maps a given data set into a new higher-dimensional feature space.
This feature space corresponds to a space of causal graphical models, such that samples which are similar in the new feature space must come from similar causal models (Theorem 8).
Our main contribution then is to propose the dependence contribution kernel κ (Definition 9).
This kernel κ is guaranteed not only to tell us that two sets of samples come from different causal models (Theorem 10 and Corollary 11) but furthermore exactly how different the causal models are (Theorem 12), all without the computational expense of explicitly projecting samples or learning causal models.
Thus, κ is well-suited for addressing the causal clustering problem and ensures that resulting clusters will be structurally homogeneous so that subsequent causal structure learning will be more informative.
3. Applications 3.1. Synthetic Data
Python code for generating this data and our plots is open source and available at https:// causal.dev/code/depcon-kernel-evaluation.tar.gz .
One data set consists ofsamples, 100 each from six random DAGs over 10 variables.
For the linear case, we generated random parameters for a structural equation model corresponding to each DAG.
We then generated [Figure] 100 of these data sets, and performed clustering on each with kernel k -means using our dependence contribution kernel as well as with three baseline methods: (1) the radial basis function (RBF) kernel, for its universality, (2) the d = 2 polynomial kernel, for its ability to detect patterns in linear correlation between samples, and (3) plain k -means with no kernel, for its simplicity.
For the nonlinear case, we then repeated this process, except that we first generated three random DAGs, gave them random parameters for corresponding linear structural equal models (SEMs), and then added a copy of each of the three SEMs but having replaced independencies (so an edge parameter of 0) with nonlinear dependencies.
The linear data set gives a best-case scenario for the the d = 2 polynomial kernel baseline method (which looks at the linear correlation patterns between samples), while the nonlinear data set gives a worst-case scenario in which nonlinear dependencies have zero linear correlation and thus are harder to detect with the polynomial kernel baseline method.
For each method on each data set, we chose k for k -means by optimizing the Variance Ratio
Criterion (VRC) (Cali´nski and Harabasz, 1974), so that clustering results realistically reflect how the method would perform when the ground truth is not known.
We evaluated the clusterings in two ways (Figure 1): first, by using the adjusted Rand index (Rand, 1971) to compare a given clustering to the ground truth, with 1 being a perfect score and 0 being the score by chance (Figure 1( a )); and second, by looking at the predicted number of clusters k found by optimizing the VRC (Figure 1( b )).
The mean adjusted Rand indices show that our kernel performs as well as the best of the baseline methods in the linear case and better than all baseline methods in the nonlinear case (Figure 1( a )).
The histograms of the predicted k according to the VRC show that our kernel performs better than all baseline methods in the linear case (Figure 1( b ), and the same holds for the unshown nonlinear case).
These results also show that, despite being defined in terms of equivalence classes of causal graphs, our kernel performs reasonably well at clustering samples according to individual DAGs.
This is perhaps explained by the fact that the clusters are detectable not only by looking at their causal structure but also by simply looking at their mean in the sample space (indicated by plain k -means’ performance being better than chance), or perhaps by the fact that different random DAGs tend to belong to different equivalence classes (especially as the number of nodes increases).
[Figure]
Furthermore, we can gain some intuition for how the kernel works, i.e., how the samples look in the kernel space and how this representation relates to its clustering performance, by using kernel principal component analysis (kPCA) (Sch¨olkopf et al., 1998).
Figure 2( a ) shows projection onto the first two principal components (according to (linear) PCA) of one of the linear data sets, while
Figure 2( b ) shows the same thing but using kPCA with our kernel.
Samples from the different causal models (represented by different colors) are much more easily separated when using our kernel.
3.2. Real data
We use kernel k -means with our dependence contribution kernel to cluster a gene expression data set and then use the measurement dependence inducing latent (MeDIL) causal model framework for structure learning within each cluster (Markham and Grosse-Wentrup, 2020).
The goal of causal clustering here is to reason about the different latent transcription factor (TF) networks governing gene expression (see Verny et al., 2017; Hackett et al., 2020, for other latent causal model approaches to learning TF networks).
The original data set comes from Iyer et al.
and can be found at genome-www.stanford.edu/serum/data/fig2clusterdata.txt , with subsequent analysis by Dhillon et al.
Our code is available at https://causal.dev/code/ fibroblast_clustering.py .
The data consists of the measured gene expression levels of 517 different genes, measured atdifferent time points, i.e., there are 517 samples and 11 different features.
In genetics applications, it is not unusual to consider genes to be samples and expression (over time) to be features—indeed the three previous analyses of this data all have this approach—and the intuition is simply that we wish to cluster genes based on patterns in their expression levels over time, in order to identify subsets of genes that are controlled by the same gene regulatory network.
Such data exemplifies the structurally heterogeneous populations discussed in Section 1: different genes can of course be regulated by different TFs, and so we can better represent the data by first clustering it into subpopulations that are more homogeneous and then performing causal structure learning on each subpopulation.
For clustering, we used k = 6 , which we found by looking at both the VRC and the Silhouette
Coefficients (Rousseeuw, 1987), computed with the scikit-learn machine learning toolbox (Pedregosa et al., 2011).
We implemented (unweighted) kernel k -means ourselves, using the pseudocode given [Figure] by Dhillon et al.
We then used the MeDIL (Markham et al., 2020) package to learn the dependence structure and latent causal models for each cluster.
Figure 3 shows an example of our results for three of the six gene clusters: Figure 3( a ) shows their distance covariance heatmaps and estimated nonlinear dependence structure (so the axes are the 11 different features, i.e., the time, in hours, at which gene expression level was measured), while
Figure 3( b ) shows their corresponding causal structures, with measurement variables M 0 – M 10 for each of the features and learned latent variables L for different posited TFs.
The results show a clear difference in causal structure for the different clusters and allow us to reason about the latent TFs regulating genes in different clusters: notice that the latents in cluster K each cause only two or three measurement variables that tend to be close together—e.g., L 1 causes
M 1 and M 2 , indicating the TF corresponding to L 1 is “short-acting”, only affecting gene expression from 30 minutes ( M 1 ) to 1 hour ( M 2 ) after serum exposure; in contrast, the latents in cluster K each cause between two and seven measurement variables that tend to be more spread out—e.g.,
L 1 causes M 1 and M 7 , indicating the corresponding TF is more complicated, “long-acting” but not continuously so, affecting gene expression 30 minutes ( M 1 ) and 12 hours ( M 7 ) after serum exposure, but independently of gene expression in the time between.
Our results are especially noteworthy compared to what happens if one ignores the heterogeneity of the data and learns a single causal structure for the entire data set without first clustering with our kernel into structurally homogeneous subpopulations: in that case, all of the measurement variables are dependent, with a single latent causing them all, and no meaningful conclusions can be drawn about how unmeasured transcription factors regulate measured gene expression, i.e., heterogeneity obscures the underlying causal structures.
In summary, our causal clustering analysis reveals which subpopulations (clusters) of genes have similar latent TF networks as well as how the TF networks differ between clusters—information that is obscured when analyzing the structurally heterogeneous data set as a whole.
Additionally, our kernel’s ability to measure similarity of nonlinear dependence structure makes it more sensitive than previous analyses of this data set using linear correlation (Dhillon et al., 2003, 2004).
4. Discussion
We address the problem of causal clustering—that is, finding the different causal structures underlying a structurally heterogeneous data set.
Our main contribution is to develop the dependence contribution kernel and prove its suitability for the causal clustering task.
This allows us to first use the kernel with existing clustering methods, such as kernel k -means or DBSCAN, to identify homogeneous subpopulations.
Then we use existing causal structure learning methods on each subpopulation.
The kernel guarantees that each subpopulation is more structurally homogeneous and therefore that the learned causal structures better capture the true causal structures within the data than if a single model were learned for the entire heterogeneous population—however, increasingly homogeneous subpopulations comes at the cost of decreasing sample sizes within each subpopulation, so (as is generally the case in clustering) care should be taken when choosing the parameters for whichever clustering method is used.
Furthermore, we prove several interesting theoretical properties of our kernel, including (i) that it can be used as a statistical test for the hypothesis that two sets of samples come from different causal structures, as well as (ii) how it induces a metric space that is isometric to the one defined by Hamming distance between ancestral graphs, i.e., comparing sets of samples with our kernel is equivalent to first estimating the causal graphs of the different sets and then comparing those graphs.
Beyond the practical applications of our kernel, as shown by our application in reasoning about latent transcription factor networks that regulate gene expression, this work also draws from and suggests further fruitful connections between a variety of fields, including causal inference, kernel methods, and algebraic statistics.
Finally, we emphasize that though clustering is our motivating application in this paper, the dependence contribution kernel can be used in the full variety of machine learning tasks solvable with kernel methods, including, e g., data visualization/reduction with kernel principal component analysis and classification tasks using kernel support vector machines. Acknowledgments
We thank Anja Meunier (University of Vienna) and Liam Solus (KTH Royal Institute of Technology) for helpful discussions and comments on a previous draft.
This work is is partially supported by the
Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice
Wallenberg Foundation.