Differentiable Causal Discovery Under Latent Interventions
Editors: Bernhard Sch¨olkopf, Caroline Uhler and Kun Zhang
Abstract
Recent work has shown promising results in causal discovery by leveraging interventional data with gradient-based methods, even if the intervened variables are unknown.
That previous work, however, assumes that the correspondence between samples and interventions is known, which is often unrealistic.
We consider a scenario in which there is an underlying observational distribution that undergoes multiple interventions, but without knowledge of which intervention (if any) corresponds to each sample, and of how the interventions affect the system; i.e. , the interventions are entirely latent.
To address this scenario, we propose a method based on neural networks and variational inference, by framing the problem as that of learning a shared causal graph among an inﬁnite mixture (under a Dirichlet process prior) of intervention structural causal models.
Experiments with synthetic and real data show that our approach and its semi-supervised variant are able to discover causal relations in this challenging scenario.
1. Introduction
Discovering causal relations among variables has countless applications in many scientiﬁc and data analysis ﬁelds (Peters et al., 2017).
However, causal graphs are notoiously hard to learn from data: only strong assumptions can ensure that the correct causal structures can be recovered from observations alone, even with an unlimited supply of data.
Recent work by Brouillard et al.
(2020) has shown very promising results in causal discovery by leveraging interventional data with gradientbased learning, even if which variables are intervened (targeted) in each intervention is unknown.
However, that work assumes that the correspondence between samples and interventions is known, which is often an unrealistic assumption.
Our work addresses this limitation by proposing a method for causal discovery under fully latent interventions, through a neural-based variational approach that infers the correspondences between data samples and interventions.
Our framework falls into the class of continuous constrained optimization methods for ﬁnding a DAG ( directed acyclic graph ); other approaches include constraint-based (Spirtes et al., 1993; Pearl, 2000; Spirtes et al., 2013) and score-based methods (Bouckaert, 1993; Heckerman et al., 1994; Chickering and Heckerman, 1997; Kuipers et al., 2021).
We assume (following Brouillard et al.
that the data is clustered into intervention groups, but we relax their assumption that the correspondence between intervention groups and samples is known, opening the door to more realistic scenarios.
We model these latent interventions as being generated by a Dirichlet process prior (DPP) and formulate the problem as one of end-to-end maximization of an evidence lower bound (ELBO).
Our contributions are summarized as follows:
• We formulate causal discovery under latent interventions as searching for the shared causal graph among an inﬁnite mixture of intervened structural causal models.
• We propose a variational formulation with a DPP to model this inﬁnite mixture.
We use latent intervention embeddings with shared parameters to model an unlimited number of interventions.
• We develop a semi-supervised variant of our method; for when we know the correspondence for a subset of the samples.
Our experiments on both synthetic and real data show that our method is able to discover causal structures, outperforming several baselines and reaching similar performance to methods that have full knowledge of the correspondence between samples and interventions.
Although by no means an exhaustive experimental evaluation, our results are very promising and support the conclusion that the proposed method extends the range of scenarios in which causal discovery can be performed.
2. Background 2.1. Structural causal models
We assume a structured causal model (SCM) M := ( S, p ( E )) over d endogenous variables X
= { x 1 , . . .
, x d } , associated to d independent exogenous variables E
= { ε 1 , . .
. , ε d } , functions F = { ζ 1 , . .
. , ζ d } , and a collection S of d assignments,
[Equation]
where PA G j ⊆ X \{ x j } is the set of parents of x j according to a directed acyclic graph (DAG)
G , i.e. , x i ∈ PA G j if and only if there is a directed edge x i → x j in G .
An SCM M deﬁnes a unique joint distribution for X , usually referred to as the entailed distribution p M ( X ) , which can be factorized as
[Equation]
where p M ( x j | PA G j ) is the conditional distribution of x j , given its parents (Spirtes et al., 1993).
2.2. Interventions
Given an SCM M , we obtain an interventioned SCM ˜ M by replacing one (or more) of the original assignments.
Let I be the set of variables targeted by the intervention; if I = ∅ , ˜ M = M .
For each variable j ∈ I , the intervention consists in one or more of the following actions: replacing the assignment function ζ j by ˜ ζ j ; replacing the parents PA G j by a subset ˜ PA G j ; changing the exogenous (noise) variable from ε j to ˜ ε j .
The SCM ˜ M generally has a different entailed distribution, called the intervention distribution:
[Equation]
If there are K possible interventions, we denote the corresponding sets of target variables as I ( k ) , for k = 1 , . . .
, K, and the corresponding SCMs by ˜ M ( k ) .
We divide the types of interventions into: atomic , if the target variable x
j is set to a constant value ( i.e. , function ˜ ζ j is constant); stochastic , if x j is set to a random variable ˜ ε j ( i.e. , if ˜ ζ j ( ˜ PA G j , ˜ ε j ) = ˜ ε j ); imperfect (or soft ), if the interventioned assignment is changed and it still depends on a nonempty subset of PA G j , i.e. , ˜ PA G j ⊆ PA G j and ˜ PA G j (cid:54) = ∅ .
We do not consider interventions that are able to add new elements to PA G j .
This means that the intervention graph only differs from the observational by the removal of edges.
2.3. Faithfulness and Markov equivalence classes
Given a set F = { ζ 1 , . . . , ζ d } , where each function ζ i is sufﬁciently dependent on all the arguments in PA G i , we obtain an SCM M whose computations strictly follow the structure of G .
In this scenario, G and p M ( X ) are said to be mutually faithful since G encodes all and only the conditional independencies that hold in the entailed distribution.
The set of faithful graphs that could entail a particular joint distribution is called the Markov equivalence class (MEC) (Verma and Pearl, 2013).
If there is access to intervention data (in a set of interventions I ), it is possible to shrink the MEC to the so-called I -MEC (Hauser and B¨uhlmann, 2012): the subset of graphs in the MEC that have the same conditional independencies after applying the interventions in I .
2.4. Continuous constrained optimization for structure learning
Our work builds on a recent line of research that uses continuous constrained optimization to address causal discovery, initiated by Zheng et al.
and extended by Brouillard et al.
(2020) to cases where there is data from intervention distributions.
In general, these approaches adopt the maximum a posteriori (MAP) criterion (a.k.a.
penalized maximum likelihood).
Based on a generative/sampling model p ( D|G , θ ) for data D , given the graph structure G and parameters θ , and on a prior p ( G ) over graphs, the graph estimate is sought by maximizing the score function
[Equation]
If D is a collection of i.i.d.
observations, then p ( D|G , θ ) = (cid:81) ni
=1 p ( x i |G , θ ) .
The prior p ( G ) penalizes graph complexity to avoid over-ﬁtting, with a typical choice being p ( G ) ∝ exp( − ξ |G| ) , for ξ > 0 and where |G| is some graph complexity measure ( e.g. , number of edges).
With ﬁnite data, exact independence seldom occurs, thus graphs maximizing log p ( D|G , θ ) alone would almost always be fully connected.
Central to this class of methods is the weighted adjacency matrix W G ∈ R d × d + , where W G ij >is equivalent to ( i, j ) ∈ G , where G is treated as a parameter itself or as a function of the parameters.
To ensure the estimated graph is a DAG, Zheng et al.
(2018) proposed the constraint
[Equation]
where e A denotes the matrix exponential of matrix A and (cid:12) is the Hadamard product.
Several other methods apply non-linear models such as neural networks (Lachapelle et al., 2019; Zheng et al., 2020) and deﬁne W G differently.
The works by Ng et al.
(2020), Kalainathan et al.
(2020), and Brouillard et al.
(2020) treat the adjacency matrix as a random variable and relax the score in Eq.
(4) to an argument Λ ∈ R d × d ,
[Equation]
where σ (Λ) is the element-wise application of the sigmoid function σ ( u ) = exp( u ) / (1 + exp( u )) , and Bern (cid:0) G ; σ (Λ) (cid:1) is a distribution over graphs, with mutually independent edges, with expected value σ (Λ) .
This score tends asymptotically to S ( G ) as σ (Λ) progressively concentrates its mass on a single DAG G .
3. Differentiable causal discovery under latent interventions
In this section, we present a score for perfect or imperfect fully latent interventions, and show how this score can be approximately maximized by using an efﬁcient variational optimization algorithm.
3.1. Mixture of intervention distributions
We assume that the dataset D is produced by a mixture of SCMs, each resulting from an intervention applied to a base SCM M .
More speciﬁcally, D is partitioned into K + 1 exchangeable groups, with group k containing i.i.d.
samples from the intervention SCM ˜ M ( k ) resulting from applying the k th intervention to the base SCM M ; the index k = 0 indicates the absence of intervention, i.e. , ˜ M (0)
=
M (observational model).
We denote by ˜ M
=
{ ˜ M (0) , ..., ˜ M ( K ) }
the ensemble of SCMs.
The latent variables z ( i ) ∈ { 0 , ..., K } indicate which SCM generated the i -th sample, with z ( i ) =
k meaning that x ( i ) is a sample of the SCM ˜ M ( k ) .
Treating these correspondences z ( i ) as latent is a distinctive aspect of our work; while Brouillard et al.
(2020) also assume unknown ˜ M , they assume that z ( i ) is observed.
We call the scenario where both ˜ M and z ( i ) are unknown as fully latent interventions .
Marginalizing with respect to the latent z ( i ) yields the mixture model
[Equation]
Conditioning on z ( i ) and invoking Eq.
(3) leads to
[Equation]
We also consider that, like the group memberships, the set of targets I ( k ) of each intervention is unknown, except for I (0) = ∅ .
3.2. Distribution over causal graphs
We represent the causal graph G via the adjacency matrix A G ∈ { 0 , 1 } d × d .
Following previous work, our prior models each entry A G ij , corresponding to edge x i → x j , as a Bernoulli variable independent of all the others,
[Equation]
(8) where the ξ ij are hyper-parameters.
In this paper, we set ξ ij = ξ G , for all i, j ; however, in practice, a domain expert using the proposed method can embed prior knowledge in these hyper-parameters (our method can be straightforwardly adapted to that case). This prior over graphs is simplistic since it does not encode that G has to be a DAG.
The adoption of a probabilistic prior p ( G ) should not be seen as expressing that it is in fact a random object, but rather as a subjective prior in the context of epistemic uncertainty about it.
3.3. Intervention embeddings and shared intervention space
We use density estimators, e.g. , neural networks and normalizing ﬂows (Rezende and Mohamed, 2015), to model the conditional densities in both the observational and interventional distributions.
With this goal in mind, we use an appropriate encoding of the changes in the intervened assignments and the intervention targets.
The set of targets in the k -th intervention ˜ M ( k ) is indicated by a ddimensional binary vector r k =
[ r k 1 , . . .
, r kd ] ∈ { 0 , 1 } d , with r kj = [-
 if and only if-]1 ⇔ j ∈ I ( k ) .
Since I (0) has no targets (it corresponds to the observational SCM M ), we have r
=
To encode the type of intervention, we introduce the intervention embedding vector u k ∈ R h , where h is a hyper-parameter.
Each vector u k represents the changes in the affected assignments for intervention ˜ M ( k ) , in a way that will become clear in the next paragraph.
We denote by R =
[ r 0 , r 1 , ..., r K ] ∈ { 0 , 1 } ( K +1) ×
d the matrix of intervention target indicators, and by U
=
[ u 0 , u 1 , ..., u K ] ∈ R ( K +1) × h the matrix of intervention embeddings.
The pair ( R , U ) represents the set of interventions ˜ M .
Putting everything together, when given the graph G , interventions ˜ M , indicator z , and assuming the intervention is imperfect, the conditional log-probability of single data-point x is given by log p ( x | z, ˜ M , G ; θ )
= =
[Equation]
where e z ∈ R K +1 is a one-hot vector representation of z , A G j is the j -th row of the adjacency matrix A G , thus A G j (cid:12) x corresponds to selecting the entries of x in PA G j . The conditional densities g 1 , . .
. , g d are parameterized by θ =
( θ 1 , . .
, θ d ) .
We will consider below several forms for these conditional densities, e.g. , using parametric families and normalizing ﬂows.
Crucially, each g j is a conditional density of x j , and the parameters θ j are shared by all the interventions—only the intervention-speciﬁc embedding vector u k changes depending on the intervention, as explained below.
This enables dealing with an unlimited number of interventions, as we shall see.
3.4. Modeling the conditional densities
A simple nonlinear model for the conditional densities g j can be constructed using neural networks.
We use a neural network NN ([ u k , A G j (cid:12) x ]; θ j ) :
R ( h + d ) → R m , a non-linear mapping parameterized by θ j , that receives the concatenation of the parents of x j and the intervention embedding u k and outputs the m parameters of some distribution f ( x j ; NN ([ u k , A G j (cid:12) x ]; θ j )) for variable x j .
There are many possible choices for the distribution f , depending on the problem at hand and on whether x j is discrete or continuous; for example, Poisson ( m = 1 ), Bernoulli ( m = 1 ), univariate Gaussian
( m = 2 ), categorical ( m is the numner of categories minus one).
In this paper, we focus on three density families in R , with which we experiment in Section 4.
Linear Gaussian: in this case, we use a neural network NN ( u k ; θ j ) to output coefﬁcients ˜ a j ∈
R d , and ˜ σ j , ˜ b j ∈ R .
Then, we use these as parameters of a Gaussian distribution whose mean is an afﬁne transformation of the values of the parents of x j :
[Equation]
Non-Linear Gaussian: this model uses a neural network NN ([ u k , A G j (cid:12) x ]; θ j ) to output coefﬁcients ˜ µ j ∈ R and ˜ σ j ∈ R , given the values of the parents of x j as input.
Then, we use these as parameters of a Gaussian distribution:
[Equation]
Normalizing ﬂows: to model non-linear non-Gaussian conditional densities, we use normalizing ﬂows (Rezende and Mohamed, 2015), which transform a base probability density (Gaussian, in our case) through a sequence of invertible mappings τ ( x j ; ˜ W j ) =
τ l ◦ τ l − 1 · · · ◦ τ 1 ( x j ; ω 1 ) , where ˜ W j = { ω 1 , . . .
, ω l } .
We use a model introduced by Huang et al.
(2018), called deep sigmoidal ﬂows (DSF), where each of the invertible mappings has the form
[Equation]
where ∆ F − 1 is the probability simplex and F is a hyper-parameter.
We use a neural network
NN ([ u k , A G j (cid:12) x ]; θ j ) to output the parameters ˜ W j .
With the former, we obtain a controllable ﬂow τ ( x j ; ˜ W j ) that, given x j , outputs the parameters of a Gaussian distribution ˜ µ , ˜ σ .
Altogether, the joint density has the following form:
[Equation]
3.5. Modeling latent interventions with a Dirichlet process
To obtain a complete statistical description of the data-generating process, we still need a prior distribution for the latent interventions, apart from the sampling model g j .
Namely, we need a prior distribution for the correspondence z , the intervention embeddings U , and the intervention targets R .
Furthermore, while experimentally we can have scenarios where K , the number of latent interventions, is known, in general, given a data set, it may not be clear what the number of latent interventions is.
Therefore, we will formulate the model with a potentially unspeciﬁed K .
We do this by using a Dirichlet process prior (DPP) with a stick-breaking representation (Ferguson, 1973; Sethuraman, 1991) as the prior distribution for the variables associated with the latent interventions.
The generative story underlying the prior is as follows.
We ﬁrst draw the graph G from p ( G ) , as described in Eq.
Then, for k = 0 , 1 , . .
, we sample the variables u k and r k , associated with each intervention ˜
M ( k ) , as well as the probability β k of picking that intervention as a stickbreaking process, with scaling parameter α > 0 (which controls the clustering effect of the DPP)
[Figure]
and hyperparameter γ (which controls the sparsity of the intervention targets), as follows:
[Equation]
Then, to generate the data, we ﬁrst sample the intervention index z ( i ) , and then sample a point x ( i ) conditioning on the corresponding intervention ˜
M ( z ( i ) ) :
[Equation]
Figure 1 contains a graphical model representation of this joint distribution.
3.6. Variational approximation
To obtain the marginal likelihood p ( D|G ; θ ) to be used in Eq. (4), it is necessary to marginalize over the joint distribution of the model present in Section 3.5, i
[Equation]
e. , [Equation] [Equation] However, the exact maximization of this marginal log-likelihood (which involves a product of Gaussians, Beta distributions, and complex conditional distributions generated by neural networks) is intractable.
Therefore, we resort to approximate variational inference (Blei et al., 2017), based on a family of tractable variational distributions q φ ( z ( i ) , ˜ M ) to approximate the true posterior p ( z ( i ) ,
˜ M|
x ( i ) , G ; θ ) .
For the variables associated with latent interventions ˜ M , we propose the fully factorized and ﬁnite variational family Q
[Equation]
where the hyper-parameter K deﬁnes the truncation level of the variational approximation, and the variational marginals associated with v k , u kl , and r kj take the following forms:
[Equation]
where π kj , µ kl , σ kl , ρ k , and w k are free parameters to be optimized, for each k ∈ { 0 , . .
, K } , l ∈ { 1 , . .
, h } , and j ∈ { 1 , . . .
, d } .
For the distribution of interventional assignments z , we propose the following variational posterior:
[Equation]
where NN ( x ; φ Z ) : R d → R h is a neural network.
Appendix D provides details about the derivation and, in the case of the Beta distribution, closed-form approximation of the Kullback–Leibler divergence between the proposals and the prior. The architecture of the neural networks used in the experiments is described in Appendix F.
We use the shorthand φ to denote the vector of all variational parameters, which includes φ Z , µ kl , σ kl , ρ k , w k , π kj for all k ∈ { 0 , . . .
K } , l ∈ { 1 , . . .
, h } , and j ∈ { 1 , . . .
, d } .
The ingredients above yield the following lower bound for the marginal log-likelihood in Eq.
[Equation]
For different choices of φ , we get different lower bound approximations to the marginal log-likelihood.
By maximizing the ELBO w.r.t.
φ we minimize the approximation gap, which equals he KL divergence between the approximate and the true posterior.
3.7. A score for latent interventions
Using the log-likelihood in Eq.
(12), we write a new score function S ( G ) for our model with latent interventions, with an associated relaxation to support a weighted adjacency σ (Λ) , as shown in Eq.
As discussed in Section 3.6, in general, we will not be able to exactly maximize this score.
However, using the variational approximation above, and associated variational family Q , we can approximate the score S ( G ) with a surrogate score S Q as follows:
[Equation]
The gap between the relaxed score S (Λ) and our surrogate S Q (Λ) tends asymptotically to the
KL divergence between the best approximate posterior from Q and the true posterior, as σ (Λ) progressively concentrates its mass on single DAG G ; more concretely,
[Equation]
where θ ∗ and φ ∗ are, respectively, the model and variational parameters that maximize Eq.
3.8. Inference algorithm
The surrogate score, coupled with the acyclicity constraint of Zheng et al.
(2018), enables formulating causal discovery under latent interventions as the following optimization problem: [Equation] to
[Equation]
Following Zheng et al.
(2018), we use the augmented Lagrangian (AL) method (Hestenes, 1969; Powell, 1969; Glowinski and Marroco, 1975) to address this problem via a sequence of unconstrained ones.
Estimating the gradients using the path-wise gradient estimator (Kingma and Welling, 2014; Rezende et al., 2014), each unconstrained optimization subproblem reduces to sampling the graph and the latent variables from the variational posteriors using the reparametrization trick, minimizing the following objective:
[Equation]
where Ω( φ ) denotes the sum of the KL divergences, and µ t and ϕ t are the AL parameters at iteration t .
To estimate the gradients of Λ , q ( z ) , and q ( r kj ) , we use a Gumbel-softmax continuous relaxation (Jang et al., 2017; Maddison et al., 2017), which, for the causal graph’s distribution, was combined with the straight-through gradient estimator (Bengio et al., 2013), to make sure the graph samples actually represent the hard dependencies of the SCM, instead of fractional ones.
Having estimated the gradients, for each sample, we average them and feed them to the ﬁrst order stochastic optimization algorithm Adam (Kingma and Ba, 2015).
The method is implemented in PyTorch (Li et al., 2020) and the code is available at github.com/goncalorafaria/ causaldiscovery-latent-interventions .
3.9. Semi-supervised extensions
The proposed causal discovery algorithm can be extended to cases where we have information about the correspondences z ( i ) and/or the intervention targets I ( k ) .
To achieve this, we only have to ignore the corresponding variational posteriors and use the observed values
z ( i ) and r ( i ) as constants.
We designate the original version of our model as latent , the one with observed z ( i ) as the unknown variant, and the one with observed z ( i ) and r ( i ) as known .
It is also straightforward to extend our method to a semi-supervised setting, in which the correspondences z ( i ) are observed for a fraction of the samples.
In this scenario, we still use the samples from the variational posterior to predict the unobserved z ( i ) and use the observed ones to improve the variational posterior.
In essence, the semisupervised variant interpolates between the latent and the unknown variants.
Following Kingma et al.
(2014), we extend the ELBO objective (in our case, the lower bound on log p ( D|G ; θ ) ).
Letting O be the set of the indices for which the intervention assignment z
( i ) is known, we write the semisupervised score S SSL (Λ) as the surrogate score from Eq. (16), by replacing the ELBO with L SSL ( D , G ) =
[Equation]
In the ﬁrst term, for i ∈ O , rather than sampling from q ( z ( i ) | x ( i ) , ˜ M ; φ z ) , we use the observed value.
Finally, κ ∈ ]0 , 1[ is a hyper-parameter controlling the relative importance of the supervised component of the objective.
4. Experiments
We tested our method on synthetic and real data.
Experiments on synthetic data allow for a systematic, controlled assessment of different methods in different scenarios (graph size and density, intervention and assignment types).
To generate SCMs, intervene on them, and sample from the corresponding distributions, we created a Pytorch package available at github.com/goncalorafaria/ PyCausal .
Appendix E provides a detailed description of how we generate SCMs.
Single-node interventions : We generate SCMs with d = 10 variables with a Erd˝os-R´enyi scheme (Erd˝os and R´enyi, 1959) with expected number of edges per node e ∈ { 1 , 4 } .
[Figure] We generated 10 SCMs for each combination of e , conditional density (linear Gaussian, non-linear Gaussian, and normalizing ﬂow), and intervention type (stochastic and imperfect).
In each of the SCMs, we performed one intervention for every variable.
For each SCM in each conﬁguration, we explored the following hyperparameter range: ξ G ∈ {− . 1 , − . 01 , 0 , .
1 } , and γ ∈ {− . 1 , − . 01 } .
For the remaining hyperparameters, we set α = 9 , h = 248 , and the truncation level K = 11 —the hyperparameter conﬁguration that achieved the best log-likelihood on a validation set.
From the generated SCM, we produced a dataset with n = 10 4 samples, where each intervention has (cid:98) nd
+
(cid:99) elements.
This dataset was split into training (80%) and validation (20%).
The models were trained for 500 epochs for the ﬁrst iteration of the augmented Lagrangian and 50 epochs for the remaining ones, with a full batch ( B = 8000 ) until h (Λ) < 10 − 8 , and learning rate of 10 − 2 . 5 .
The results of these experiment are shown in Table 1.
The metric wed is Hamming distance (HD) between the adjacency matrices of the ground-truth graph and the estimated one.
The columns in Table 1 correspond to the different variants described in Section 3.9.
We additionally include a “na¨ıve” observational model (a baseline which ignores the existence of intervention data, i.e. , that assumes K = 0 as if all data was generated by the observational model).
The experimental results show that our method (the latent variant) consistently outperforms the observational baseline and is worse than the unknown variant, as expected since it uses less information, but only slightly.
This indicates that taking into account latent interventions, when they are present, improves the recovery of the causal graph.
Single-node interventions on cause-effect pairs:
We generated two-variable linear Gaussian SCMs (cause-effect pairs) with edge probability e = 23 (equal probability for each of the three possible graphs).
We sampled 60 SCMs, and generate a dataset of n = 999 samples, where each intervention has 333 elements.
The sampled graph G , with variables A and B is one of 3 possible graphs, A causes B , B causes A , or A and B have no cause-effect relation.
We compare the variants presented in Section 3.9 in addition to the na¨ıve observational baseline.
Figure 2 contains an histogram of Hamming distances for each of the variants.
Edges in the anti-causal direction cost HD = 2 , missing edge or wrong edge is HD = 1 , and correct graph is HD = 0 .
The results show [Table] for this simple problem that the causal graph cannot be identiﬁed from observational data alone, and that our method correctly identiﬁes most of the cause-effect pairs, even without information about the intervention assignments.
Real -world data:
Finally, we tested our method on the ﬂow cytometry dataset of Sachs et al.
Table 2 compares the estimated graph, under different conditional density assumptions and intervention types, to the consensus graph of Sachs et al.
The results on this dataset show that our method outperforms several baselines, including methods that use information about the correspondences between interventions and targets, as well as FCI (Spirtes et al., 1993), which supports latent confounders (but not intervention data).
Reasons that might justify the relatively good results on this standard problem include: (i) the causal sufﬁciency assumption may not hold, (ii) the interventions may not be as speciﬁc as stated, and (iii) the ground truth network is possibly not a DAG, since feedback loops are common in cellular signaling networks as noted by Mooij et al.
; Brouillard et al.
These reasons can potentially be detrimental to the other methods, whereas our method appears to be robust to them.
5. Related Work
The previous work that is closest to ours in that by Brouillard et al.
They assume, as we do, that the data is clustered in intervention groups, but we relax their assumption that the correspondence between intervention groups and samples is known, which is a more realistic assumption.
The so-called “known” and “unknown” variants of our method share the assumptions of Brouillard et al.
(2020); however, in our method, the number of neural networks is independent on the number of interventions, which allows scaling to many interventions.
We achieve this by encoding the assignments change associated with each intervention using a speciﬁc latent variable that we call intervention embedding and conditioning a shared model on it when computing the log-probability.
[Table]
There are many works that do causal discovery with latent confounders (Spirtes et al., 1993; Colombo et al., 2012; Claassen et al., 2013; Ogarrio et al., 2016).
We view those works as complementary to ours, since in many situations the inﬂuence of latent confounding can render unreliable methods that assume causal sufﬁciency, such as ours. The scenario of causal discovery with latent confounders and ours are distinct in terms of underlying assumptions.
We propose a scenario where interventions exists and they are fully latent: i.e. , we do not know what they alter or which samples are affected. While, we could regard these interventions as variables in a expanded causal model, hence latent confounders in that causal model, treating them in that way would disregard the structure of the problem, which is key to the success of our approach (looking for this structure is what enables discovering causal-effect relations). In practice, this leads to inferior results, as the results from Table 2 with the FCI (Spirtes et al., 1993) algorithm show. Our method can be seen as a variational autoencoder (VAE) with a DPP, with a stick-breaking representation.
The work by Nalisnick and Smyth (2017) introduces a VAE where the stick-breaking weights are the latent variables.
Our model differs from theirs in that we use the entire DP (including the atoms).
We only use the stick-breaking weights in the KL divergence of the correspondence variable z .
This KL divergence has a closed-form, so we do not sample the stick-breaking weights as they do.
van den Oord et al.
(2017) proposes a VQ-VAE model with discrete latent variables, each represented as a latent embedding vector (an atom).
Our approach is similar in that we use atoms (the intervention embeddings and intervention targets) to represent discrete latent variables.
However, we do it in a statistically sounder way that more naturally ﬁts our application.
6. Conclusion
We introduced an efﬁcient variational optimization algorithm for causal structure learning under latent interventions.
Our results are competitive with other state-of-the-art algorithms on the ﬂow cytometry data.
On synthetic data, our approach recovers causal relations even in our most challenging scenario, and it consistently outperforms a purely observational baseline.
A limitation of our method is the variational family we use.
The proposal for variational posterior considers that the stick-breaking weights, the intervention embeddings, and intervention targets are independent of each other.
In some cases, this can potentially create a surrogate score whose maximum structure is not an element of the I -Markov equivalence class.
There are many avenues for future work.
Our framework is particularly appealing for problems where performing interventions explicitly is expensive or unethical, but where interventions occur naturally in the data without being explicitly observed.
Experimenting with more ﬂexible variational families also seems appealing, albeit this may come at the cost of the closed-form expressions for the KL divergences.
Another direction of future research is to address the case where the dataset does not contain samples from the source SCM (observation distribution).
In our formulation, we pin the empty set to the ﬁrst intervention, i.e., the ﬁrst intervention SCM is the source SCM, which necessarily assumes that we always have samples from the observation distribution.
We can extend our method for cases where this is not the case.
Under such circumstances, we can deﬁne the observational model as the one corresponding to the union of the parent sets of each particular variable across the different intervention regimes.
Acknowledgments
This work was supported by the European Research Council (ERC StG DeepSPIN 758969) and by the Fundac¸˜ao para a Ciˆencia e Tecnologia through contract UIDB/50008/2020. Appendix A. Perfect Interventions If we assume that the intervention is perfect (stochastic or atomic), the log-probability of single data-point is given by
[Equation]
that is, we make the conditional density of x j be only dependent on u k when intervened.
Appendix B. Hyper-parameter Selection
We searched hyper-parameters for the α , γ , ξ G and learning rate in a validation set based on logevidence.
The remaining hyper-parameters pertaining to the neural networks architecture were selected based on validation datasets (data and graphs) and were held ﬁxed for all of the experiments.
The architecture parameters can be found in Table 3.
[Table]
[Table]
wd
[Table]
[Table]
ϕ 0 , µ 0 , η , δ ) ( 0 . 0 , 10 − 8 , 2 . 0 , 0 . 9 ) parameters ( ϕ 0 , µ 0 , η , δ ) are respectively set to ( 0 . 0 , 10 − 8 , 2 . 0 , 0 . 9 ).
We ﬁxed the number of iterations of each subproblem to 500.
We found this value large enough to reach convergence in the small-scale problems considered in this paper.
Appendix C. Scalability
In our experiments, we mostly considered problems with a small number of variables, particularly 2, 10, and 20, and just a few number of examples ( 10 4 ).
The main computational bottleneck is the trace exponential matrix operation which is O ( d 3 ) , where d is the number of variables.
We measured the required GPU memory of our model, as we increasing the number of variables, using the default hyper-parameters and a batch-size of 128, and obtained the plot present in Figure 3.
As expected, the memory increases quadratically due to the fact that we represent the graph using an adjacency matrix.
Since, we are using mini-batches to estimate the gradients of the loss w.r.t.
the parameters of our model the memory requirements stay constant as we scale the number of examples.
However, we can expect the convergence time to increase.
Appendix D. Kullback–Leibler divergences
We detail below the expressions for the Kullback-Leibler (KL) divergences used in this paper.
[Equation]
Stick-breaking weights
[Equation]
Intervention embeddings
[Equation]
Intervention targets
[Equation]
Correspondence
[Equation]
where
[Equation]
and ψ is the digamma function.
We approximate using the Taylor series expansion, where we use the reparametrization trick on u k to estimate the expectations.
Appendix E. Description of the synthetic datasets
To generate the SCMs, we ﬁrst generated the causal graph.
The graphs were generated going in order from node 1 to node N and adding edges from the already visited nodes to the current node i with probability p obtaining the adjacency matrix A G .
In order to obtain graphs that weren’t topologically ordered we applied a random permutation P to A G .
Then, given the causal graph, we sampled the noise distributions and the assignments as described in Table 4.
For every SCM, we generate d interventions, one for each variable.
How we sample the new assignment for each intervened variables is described in Table 5.
Before ﬁtting the model, the data is always normalized.
We subtract the mean and divide by the standard deviation.
For each setting, we sample n/ ( d +1) examples.
We used n = 10000 in the 10 variable data set.
Before using our method, the data is always normalized.
We subtract the mean and divide by the standard deviation.
For stochastic interventions in variable j , we sampled a new Gaussian variable ε j , as described in Table 5, and assigned it to x j .
Using imperfect interventions, we go to the last layer of the neural network ( in the case of linear model its the weight vectors ) representing the assignment, and sample new weights according to Table 5.
Table 4 contains a description of how we sample assignments.
Particularly, the architecture of the neural network, when it is non-
linear
SCM.
Figure 4 contains scatter plots obtained after sampling the generated two variable SCMs.
[Table] [Table]
Appendix F. Neural network details
The neural network we used is a feed-forward (fully connected) network.
Given a vector x ∈ R d in , we parameterize with a vector of weights θ a non-linear mapping FFN : R d in → R d out .
This mapping is a composition of linear transformation interchanged with a non-linear element-wise operation.
We use many tricks that have been shown to improve the results with neural network modules, particularly dropout, SiLU activation function, layer normalization and residual connections.
We group these techniques together into a sub-layer that form the building block of our relatively more complex neural network models, as shown in Figure 5.
To enable the use of skip connections, we have to make sure that the inputs and outputs of the sub-layer have the same dimension.
For this reason before applying the stack of sub-layers we apply a linear transformation from R d in to R d h and after passing the stack from R d h to R d out , as shown in the diagram in Figure 6.
Appendix G. Semi-supervised learning experiments
We test our semi-supervised method, on 10 variable linear SCMs generated according to Appendix
E. We consider stochastic and imperfect interventions.
We sampled a 10000 samples dataset from each SCM.
For each dataset, we created multiple semi-supervised learning datasets by splitting the original 10000 samples into labelled (with intervention assignments) and unlabelled (without intervention assignments) according to some fraction f .
The fraction f goes from 0 to 1 , sampled every interval of 0 . 2 .
This means that, for each fraction f , we have 10 distinct semi-supervised datasets for both perfect and imperfect interventions.
We picked the default hyper-parameters, particularly, ξ G = − 0 .
1 , and γ = − 0 .
The results are contained in Figure 7.
The results suggest that there is no signiﬁcant improvement from using the correspondences for a fraction of the samples under the tested conditions.