D EPENDENCY S TRUCTURE D ISCOVERY FROM I NTERVENTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Promising results have driven a recent surge of interest in continuous optimization methods for Bayesian network structure learning from observational data.
However, there are theoretical limitations on the identiﬁability of underlying structures obtained from observational data alone.
Interventional data provides much richer information about the underlying data-generating process.
However, the extension and application of methods designed for observational data to include interventions is not straightforward and remains an open problem.
In this paper we provide a general framework based on continuous optimization and neural networks to create models for the combination of observational and interventional data.
The proposed method is applicable even in the challenging and realistic case that the identity of the intervened upon variable is unknown.
We examine the proposed method in the setting of graph recovery both de novo and from a partially-known edge set.
We establish strong benchmark results on several structure learning tasks, including structure recovery of both synthetic graphs as well as standard graphs from the
Bayesian Network Repository.
I NTRODUCTION
Structure learning concerns itself with the recovery of the graph structure of Bayesian networks (BNs) from data samples.
A natural application of Bayesian networks is to describe cause-effect relationships between variables.
In that context, one may speak of causal structure learning .
Causal structure learning is challenging because purely observational data may be satisfactorily explained by multiple Bayesian networks (a Markov equivalence class ), but only one is the most robust to distributional shifts:
The one with the correct graph.
A more powerful tool than BNs is thus needed to model causal relationships.
Structural Causal Models (SCMs) are that tool.
An SCM over a set of random variables is a collection of assignments to these variables and a directed acyclic graph of dependencies between them (Peters et al., 2017, §6.2).
Each assignment is a function of only the direct causes of a variable, plus an independent noise source.
An SCM entails precisely one (observational) data distribution.
Interventions on an SCM’s assignments, such as setting a random variable to a ﬁxed value ( a hard intervention ), entail new interventional data distributions (Peters et al., 2017, §6.3).
SCMs can be used to answer higher-order questions of cause-and-effect, up the ladder of causation (Pearl & Mackenzie, 2018).
Causal structure learning using SCMs has been attempted in several disciplines including biology (Sachs et al., 2005; Hill et al., 2016), weather forecasting (Abramson et al., 1996) and medicine (Lauritzen & Spiegelhalter, 1988; Korb & Nicholson, 2010).
Causal structure is most frequently learned from data drawn from observational distributions.
Structure learning methods generally cannot do more than identify the causal graph up to a Markov equivalence class (Spirtes et al., 2000).
In order to fully identify the true causal graph, a method must either make restrictive assumptions about the underlying data-generating process, such as linear but non-Gaussian data (Shimizu et al., 2006), or must access enough data from outside the observational distribution (i.e., from interventions).
Under certain assumptions about the number, diversity, and nature of the interventions, the true underlying causal graph is always identiﬁable, given that the method knows the intervention performed (Heckerman et al., 1995).
In much of the prior work on causal model induction it is assumed that there is an experimenter and this experimenter performs interventions.
However, in the real world, interventions can also be performed by other agents, which could lead to unknown interventions (interventions with unknown target variables).
A few works have attempted to learn structures from unknown-intervention data (Eaton & Murphy, 2007a; Mooij et al., 2016).
Although there is no theoretical guarantee that the true causal graph can be identiﬁed in that setting, evidence so far points to that still being the case.
Another common setting is when the graph structure is partially provided, but must be completed.
An example is protein structure learning in biology, where we may have deﬁnitive knowledge of some causal edges in the protein-protein interactome, but the remaining causal edges must be discovered.
We will call this setting “partial graph completion”.
This is an easier task compared to learning the entire graph, since it limits the number of edges that have to be learned.
Recently, a ﬂurry of work on structure learning using continuous optimization methods has appeared (Zheng et al., 2018; Yu et al., 2019).
These methods operate on observational data and are competitive with other methods.
Because of the theoretical limitations on identiﬁcation from purely observational data cited above, it would be interesting to extend these methods to interventional data.
However, it is not straightforward to apply continuous optimization methods to structure learning from interventional data.
Our key contributions are to answer the following questions experimentally:
Can the proposed model recover true causal structure?
Yes, see Figure §4. 2.
How does the proposed model compare against state of the art causal methods on real-world datasets?
Favourably; see §
.4 and Table §
Does a proposed model generalize well to unseen interventions?
Yes, see §5.5.
How does the proposed model perform on partial graph recovery?
It scales to ∼ 50 variables while the other baselines can’t.
see §5.7.
P RELIMINARIES
Causal modeling.
A Structural Causal Model (SCM) (Peters et al., 2017) over a ﬁnite number M of random variables
X i is a set of structural assignments
[Equation]
Identiﬁability.
In a purely-observational setting, it is known that causal graphs can be distinguished only up to a Markov equivalence class.
In order to identify the true causal graph structure, interventional data is needed (Eberhardt et al., 2012).
Interventions.
There are several types of common interventions which may be available (Eaton &
Murphy, 2007b).
These are: No intervention: only observational data is obtained from the ground truth model.
Hard/perfect: the value of a single or several variables is ﬁxed and then ancestral sampling is performed on the other variables.
Soft/imperfect: the conditional distribution of the variable on which the intervention is performed is changed.
Uncertain: the learner is not sure of which variable exactly the intervention affected directly.
Here we make use of soft intervention because they include hard intervention as a limiting case and hence are more general.
Structure discovery using continuous optimization.
Structure discovery is a super-exponential search problem that searches though all possible directed acyclic graphs (DAGs).
Previous continuousoptimization structure learning works (Zheng et al., 2018; Yu et al., 2019; Lachapelle et al., 2019) mitigate the problem of searching in the super-exponential set of graph structures by considering the degree to which a hypothesis graph violates “DAG-ness” as an additional penalty to be optimized.
If there are M such variables, the strategy of considering all the possible structural graphs as separate hypotheses is not feasible because it would require maintaining O (2 M 2 ) models of the data.
R ELATED W ORK
The recovery of the underlying structural causal graph from observational and interventional data is a fundamental problem (Pearl, 1995; 2009; Spirtes et al., 2000).
Different approaches have been studied: score-based, constraint-based, asymmetry-based and continuous optimization methods.
Score-based methods search through the space of all possible directed acyclic graphs (DAGs) representing the causal structure based on some form of scoring function for network structures (Heckerman et al., 1995; Chickering, 2002;
Tsamardinos et al., 2006; Hauser & Bühlmann, 2012;
Goudet et al., 2017; Cooper & Yoo, 1999; Zhu & Chen, 2019).
Constraint-based methods (Spirtes et al., 2000; Sun et al., 2007; Zhang et al., 2012;
Monti et al., 2019; Zhu & Chen, 2019) infer the
DAG by analyzing conditional independences in the data.
Eaton & Murphy (2007c) use dynamic programming techniques to accelerate Markov Chain Monte Carlo (MCMC) sampling in a Bayesian approach to structure learning for discrete variable DAGs.
Peters et al.
; Ghassami et al.
(2017); Rojas-Carulla et al.
(2018) exploit invariance across environments to infer causal structure, which faces difﬁculty scaling due to the iteration over the super-exponential set of possible graphs.
Recently, (Zheng et al., 2018; Yu et al., 2019; Lachapelle et al., 2019) framed the structure search as a continuous optimization problem, however, the methods only uses observational data and are non-trivial to extend to interventional data.
In our paper, we present a method that uses continuous optimization methods that works on both observational and interventional data.
For interventional data, it is often assumed that the models have access to full intervention information, which is rare in the real world.
Rothenhäusler et al.
(2015) have investigated the case of additive shift interventions, while Eaton & Murphy (2007b) have examined the situation where the targets of experimental interventions are imperfect or uncertain.
This is different from our setting where the intervention is unknown to start with and is assumed to arise from other agents and the environment.
Learning based methods have been proposed (Guyon, 2013; 2014; Lopez-Paz et al., 2015) and there also exist recent approaches using the generalization ability of neural networks to learn causal signals from purely observational data (Kalainathan et al., 2018; Goudet et al., 2018).
Neural network methods equipped with learned masks, such as (Ivanov et al., 2018;
Li et al., 2019; Yoon et al., 2018;
Douglas et al., 2017), exist in the literature, but only a few (Kalainathan et al., 2018) have been adapted to causal inference.
This last work is, however, tailored for causal inference on continuous variables and from observations only.
Adapting it to a discrete-variable setting is made difﬁcult by its use of a Generative Adversarial Network (GAN) Goodfellow et al.
(2014) framework.
S TRUCTURE D ISCOVERY FROM I NTERVENTIONS M ETHOD
Scope of Applicability and Objective.
The proposed method, like any structure learning algorithm, assumes the availability of a data-generating process based on ancestral sampling of a ground-truth
SCM of M variables, which can be queried for samples.
The SCM supports applying and retracting known or unknown interventions.
The method can support inﬁnite- or ﬁnite-data as well as inﬁniteor ﬁnite-intervention regimes.
The objective is, then, to learn the SCM’s structure from the insights that each intervention gives about cause-effect relationships between variables in the SCM.
4. P ROBLEM S ETTING AND A SSUMPTIONS
In this paper, we restrict the problem setting to speciﬁc, but still broad classes of SCMs and interventions.
In particular, we assume that:
Data is discrete-valued.
The SCM’s random variables are all categorical.
Causal sufﬁciency.
For every data sample, the value of all variables are available; There are no latent confounders.
Interventions are localized.
They affect only a single variable (but which one may not be known).
Interventions are soft.
An intervention does not necessarily pin its target random
variable to a ﬁxed value (though it may, as a special case).
It changes the relationship of a variable with its parents.
Interventions do not stack.
Before a new intervention is made, the previous one is fully retracted.
This stops the SCM from wandering away from its initial, observational conﬁguration after a long series of interventions.
No control over interventions.
The structure learning algorithm has control neither of the target, nor the nature of the next intervention on the SCM.
For a detailed description of the interventions, refer to §A.2.
4. V ARIATIONS AND P RIOR K NOWLEDGE
In the problem setting above, the ground-truth SCM is completely opaque to us.
However, we consider two interesting relaxations of this formulation:
Complete or partial graph recovery.
We may already know the existence of certain cause-effect edges and non-edges within the ground-truth SCM.
If such prior information is available, it turns a complete graph recovery problem into one of partial graph recovery .
Larger SCMs can be tackled if only parts of the graph need to be recovered.
Known or unknown interventions: The interventions can either be known or unknown to the learned model.
We demonstrate that the proposed method can naturally incorporate this prior information to improve its performance.
4. M ETHOD O VERVIEW
The proposed method is a score-based , iterative , continuousoptimization method consisting of three phases that ﬂow into one other (See Figure 2).
During the three-phase procedure, a structural representation of a DAG and a functional representation of a set of independent causal mechanisms are trained jointly until convergence.
Because the structural and functional parameters are not independent and do inﬂuence each other, we train them in alternating phases, a form of block coordinate descent optimization.
4.3. P ARAMETRIZATION
We distinguish two sets of parameters:
The structural parameters γ and the functional parameters θ .
Given a graph of M variables, we parametrize the structure γ as a matrix R M × M such that σ ( γ ij ) is our belief in random variable
X j being a direct cause of X i , where σ ( x ) = 1 / (1 + exp( − x )) is the sigmoid function.
The matrix σ ( γ ) is thus a soft adjacency matrix .
The set of functional parameters θ i parametrizes the conditional probability distribution of X i given its parent set X pa ( i,C ) , with C ∼ Ber( σ ( γ ))
a hypothesized conﬁguration of the SCM’s DAG.
4.3. P HASE 1: G RAPH F ITTING ON O BSERVATIONAL D ATA
During Phase 1, the functional parameters θ are trained to maximize the likelihood of randomly drawn observational data under graphs randomly drawn from our current beliefs about the edge structure.
We draw graph conﬁgurations C ij ∼ Ber( σ ( γ ij )) and batches of observational data from the unintervened ground-truth SCM, then maximize the log-likelihood of the batch under that conﬁguration using SGD.
The use of graph conﬁgurations sampling from Bernoulli distributions is analogous to dropout on the inputs of the functional models (in our implementation, MLPs), giving us an ensemble of neural networks that can model the observational data.
4.3. P HASE 2: G RAPH S CORING ON I NTERVENTIONAL D ATA
During Phase 2, a number of graph conﬁgurations are sampled from the current edge beliefs parametrized by γ , and scored on data samples drawn from the intervention SCM.
Intervention applied:
At the beginning of Phase 2, an intervention is applied to the ground-truth
SCM.
This intervention is not under the control of the method.
In our implementation, and unbeknownst to the model, the target variable is chosen uniformly randomly from all M variables throughout the optimization process.
Intervention predicted: If the target of the intervention is not known, it is predicted using a simple heuristic.
A small number of interventional data samples are drawn from the SCM and more graphs are sampled from our current edge beliefs.
The average log-likelihood of each individual variable
X i across the samples is then computed using the functional model parameters θ ﬁne-tuned on observational data in Phase 1.
The variable X i showing the greatest deterioration in log-likelihood is assumed to be the target because the observational distribution most poorly predicts that variable.
If the target of the intervention is known, then this is taken as ground-truth knowledge for the purpose of subsequent steps, and no prediction needs to be done.
Graphs Sampled and Scored:
A new set of interventional data samples and graph conﬁgurations are now drawn from the intervention SCM and edge beliefs respectively.
The log-likelihood of the data batches under the hypothesized conﬁgurations is computed, with one modiﬁcation: The contribution to the total log-likelihood of a sample X coming from the target (or predicted-target) intervention variable
X
i is masked.
Because X i was intervened upon (in the manner of a Pearl do-operation, soft or hard), the values one gets for that variable should be taken as givens, not as contributors to the total log-likelihood of the sample.
As well, no gradient should be allowed to propagate into the variable’s learned functional parametrization θ i , because it was not actually responsible for the outcome.
Intervention retracted: After Phase 2, the intervention is retracted, per our modelling assumptions.
4.3. P HASE 3: C REDIT A SSIGNMENT TO S TRUCTURAL P ARAMETERS
During Phase 3, the scores of the interventional data batches over various graph conﬁgurations are aggregated into a gradient for the structural parameters γ .
Because a discrete Bernoulli random sampling process was used to sample graph conﬁgurations under which the log-likelihoods were computed, we require a gradient estimator to propagate gradient through to the γ structural parameters.
Several alternatives exist, but we adopt for this purpose the REINFORCE-like gradient estimator g ij proposed by Bengio et al.
[Equation]
where the ( k ) superscript indicates the values obtained for the k -th draw of C under the current edge beliefs parametrized by γ .
Therefore, L ( k ) C,i ( X ) can be read as the log-likelihood of variable X i in the data sample X under the k ’th conﬁguration, C ( k ) , drawn from our edge beliefs.
Using the estimated gradient, we then update γ with SGD, and return to Phase 1 of the continuous optimization process.
The gradient estimator g ij minimizes an implicit empirical risk objective with respect to γ ij .
When the functional and structural parameters θ and γ are “sufﬁciently close” to their minima, the estimator g ij empirically converges quickly towards that minimum γ ∗ as shown in Figure 16 of Appendix A.13.
Acyclic Constraint: We include a regularization term J DAG ( γ ) that penalizes length-2 cycles in the learned adjacency matrix σ ( γ ) , with a tunable strength λ DAG .
The regularization term is J DAG ( γ ) =
(cid:80)
[Equation]
∀ i, j ∈ { 0 , . . .
, M − 1 } and is derived from Zheng et al.
The details of the derivation are in the Appendix.
We explore several different values of λ
DAG and their effects in our experimental setup.
Suppression of longer-length cycles was not found to be worthwhile for the increased computational expense.
E XPERIMENTAL S ETUP AND R ESULTS
We ﬁrst evaluate the proposed method on a synthetic dataset where we have control over the number of variables and causal edges in the ground-truth SCM.
This allows us to analyze the performance of the proposed method under various conditions.
We then evaluate the proposed method on real-world datasets from the BnLearn dataset repository.
We also consider the two variations of §4.2: Recovering only part of the graph (when the rest is known), and exploiting knowledge of the intervention target.
The summary of our ﬁndings is: 1) We show strong results for graph recovery for all synthetic graphs in comparisons with other baselines, measured by Hamming distance.
The proposed method achieves high accuracy on partial graph recovery for large, real-world graphs. 3)
The proposed method’s intervention target prediction heuristic closes the gap between the known- and unknowntarget intervention scenarios.
The proposed method generalizes well to unseen interventions. 5)
The proposed method’s time-to-solution scaling appears to be driven by the number of edges in the groundtruth graph moreso than the number of variables
[Figure]
5. M ODEL D ESCRIPTION
[Figure]
[Table]
Learner model.
Without loss of generality, we let θ i
=
{ W0 i , B0 i , W1 i , B1 i } deﬁne a stack of M one-hidden-layer MLPs, one for each random variable
X i .
A more appropriate model, such as a CNN, can be chosen using domainspeciﬁc knowledge; the primary advantage of using MLPs is that the hypothesized DAG conﬁgurations c ij can be readily used to mask the inputs of MLP i , as shown in Figure 3.
To force the structural equation f i corresponding to X i to rely exclusively on its direct ancestor set pa ( i, C ) under hypothesis adjacency matrix
C (See Eqn. 1), the one-hot input vector X j for variable
X
i ’s MLP is masked by the Boolean element c ij .
An example of the multi-MLP architecture with M =4 categorical variables of
N =3 categories is shown in Figure 3.
For more details, refer to Appendix A.4.
Ground-truth model.
Ground-truth SCM models are parametrized either as CPTs with parameters from BnLearn (in the case of real-world graphs), or as a second stack of MLPs similar to the learner model, with randomly-initialized functional parameters θ GT and the desired adjacency matrix γ GT .
Interventions.
In all experiments, at most one (soft) intervention is concurrently performed.
To simulate a soft intervention on variable X
i , we reinitialize its ground-truth conditional distribution’s
MLP parameters or CPT table randomly, while leaving the other variables untouched.
For more details about the interventions, please refer to Appendix A.2.
5. S YNTHETIC D ATASETS E XPERIMENTS
[Equation]
A r e a U nd e r C u r v e 0 . 0 0 .0 .
G a mm a C r o ss - E n t r o p y ( a v g b i t s ) 0K 10K 20K 30K 40K 50K Steps(#) chain3chain4chain5chain6chain7chain 0 . 0 0 .0 .
A r e a U nd e r C u r v e 0 . 0 0 .0 .
G a mm a C r o ss - E n t r o p y ( a v g b i t s ) 0K 10K 20K 30K 40K 50K Steps(#) jungle3jungle4jungle5jungle6 jungle7 jungle 0 . 0 0 .0 .
A r e a U nd e r C u r v e 0 . 0 0 .0 .
G a mm a C r o ss - E n t r o p y ( a v g b i t s ) 0K 10K 20K 30K 40K 50K Steps(#) full3full4full5full6full7full
We ﬁrst evaluate the model’s performance on several randomlyinitialized SCMs with speciﬁc, representative graph structures.
Since the number of possible DAGs grows super-exponentially withthenumberof variables,for M =4 up to 13 a selection of representative and edge-case DAGs are chosen.
chainM and fullM ( M =3-13) are the minimallyand maximally-connected M -variable DAGs, while treeM and jungleM are tree-like intermediate graphs.
colliderM is the ( M − 1) → 1 collider graph.
The details of the setup is in Appendix A.6.
Results.
The model can recover most synthetic DAGs with high accuracy, as measured by Structural
Hamming Distance (SHD) between learned and ground-truth DAGs.
Table 1 shows our proposed method outperforming all other baseline methods, and learns all graphs perfectly for 3 to 13 variables (excepting full ).
For DAGs ranging from 3 to 8 variables, the AUROCs all eventually reach 1.
(indicating perfect classiﬁcation into edge/non-edge; Refer to Figure 4).
For both large ( M > 10 )
[Table]
and dense DAGs (e.g. full13 ) the model begins encountering difﬁculties, as shown in Table 1 and
Appendix §A.6.
[Table]
Small graphs ( M < 10 ) are less sensitive than larger ones to our hyperparameters, notably the sparsity and acyclic regularization (§4.3.4) terms.
In §A.5, we perform an analysis of these hyperparameters.
5. R EAL -W ORLD D ATASETS : B N L EARN
The Bayesian Network Repository is a collection of commonly-used causal Bayesian networks from the literature, suitable for Bayesian and causal learning benchmarks.
We evaluate the proposed method on the Earthquake (Korb & Nicholson, 2010), Cancer (Korb & Nicholson, 2010), Asia (Lauritzen &
Spiegelhalter, 1988) and Sachs (Sachs et al., 2005) datasets ( M = 5, 5, 8 and 11-variables respectively, maximum in-degree 3) in the BnLearn dataset repository.
Results.
As shown in Table 1, the proposed method perfectly recovers the DAG of Asia, while making a small number of errors (SHD=6) for Sachs (11-variables).
It thus signiﬁcantly outperforms all other baselines models.
Figures 8 & 9 visualize what the model has learned at several stages of learning.
Results for Cancer and Asia can be found in the appendices, Figure 17 and 18.
5. C OMPARISONS WITH OTHER METHODS
As shown in Table 1, we compared the proposed SDI method to ICP ((Peters et al., 2016)), non-linear
ICP ((Heinze-Deml et al., 2018b)), and (Eaton & Murphy, 2007b; Zheng et al., 2018;
Yu et al., 2019) on Asia (Lauritzen & Spiegelhalter, 1988), Sachs (Sachs et al., 2005) and representative synthetic graphs.
Eaton & Murphy (2007b) handles uncertain interventions and Peters et al.
Heinze-Deml et al.
(2018b) handles unknown interventions.
However, neither attempts to predict the intervention.
As shown in Table 1, we signiﬁcantly outperform ICP, non-linear ICP, and the methods in (Yu et al., 2019) and (Zheng et al., 2018).
Furthermore, Eaton & Murphy (2007b) runs out of memory for graphs larger than M = 10 because modelling of uncertain interventions is done using “shadow” random variables (as suggested by the authors), and thus recovering the DAG internally requires solving a d = 2 M -variable problem.
Their method’s extremely poor time- and space-scaling of O ( d 2 d ) makes it unusable beyond d > 20 .
For SDI s, we threshold our edge beliefs at σ ( γ ) = 0 .
5 to derive a graph, but the continued decrease of the cross-entropy loss (Figure 4) hints at SDI ’s convergence onto the correct causal model.
Please refer to Appendix §A.8 for full details and results.
5. G ENERALIZATION TO P REVIOUSLY U NSEEN I NTERVENTIONS
[Table]
It is often argued that machine learning approaches based purely on capturing joint distributions do not necessarily yield models that generalize to unseen experiments, since they do not explicitly model changes through interventions.
By way of contrast, causal models use the concept of interventions to explicitly model changing environments and thus hold the promise of robustness under distributional shifts (Pearl, 2009; Schölkopf et al., 2012; Peters et al., 2017).
To test the robustness of causal modelling to previously unseen interventions (new values for an intervened variable), we evaluate a well-trained causal model against a variant, non-causal model trained with c ij = 1 , i (cid:54) =
j .
An intervention is performed on the ground-truth SCM, fresh interventional data is drawn from it, and the models, with knowledge of the intervention target, are asked to predict the other variables given their parents.
The average log-likelihoods of the data under both models are computed and contrasted.
The intervention variable’s contribution to the loglikelihood is masked.
For all 3-variable graphs ( chain3 , fork3 , collider3 , confounder3 ), the causal model attributes higher log-likelihood to the intervention distribution’s samples than the non-causal variant, thereby demonstrating causal models’ superior generalization ability in transfer tasks.
Table 2 collects these results.
5. V ARIANT : P REDICTING INTERVENTIONS
[Table]
In Phase 2 (§4.3.3), we use a simple heuristic to predict the intervention target variable.
Experiments show that this heuristic functions well in practice, yielding correct predictions far more often than by chance alone (Table 3).
Guessing the intervention variable randomly, or not guessing it at all, leads to a signiﬁcant drop in the model performance, even for 3-variable graphs (Fig.
11 Left).
Training SDI with intervention prediction closely tracks training with leaked knowledge of the ground-truth intervention on larger, 7-variable graphs (Fig.
11 Right).
5. V ARIANT : P ARTIAL G RAPH R ECOVERY
Instead of learning causal structures de novo , we may have partial information about the ground-truth SCM and may only need to ﬁll in missing information (§4.2).
An example is protein structure discovery in biology, where some causal relationships have been deﬁnitely established and others remain open hypotheses.
This is an easier task compared to full graph recovery, since the model only has to search for missing edges.
[Table]
We evaluate the proposed method on Barley (Kristensen & Rasmussen, 2002)
( M = 48 ) and Alarm (Beinlich et al., 1989) ( M = 37 ) from the BnLearn repository.
The model is asked to predict 50 edges from Barley and 40 edges from Alarm.
The model reached ≥ 90% accuracy on both datasets, as shown in Table 4.
5. A BLATION AND ANALYSIS
As shown in Figure 12, larger graphs (such as M > 6 ) and denser graphs (such as full8 ) are progressively more difﬁcult to learn.
For denser graphs, the learned models have higher sample complexity, higher variance and slightly worse results.
Refer to Appendix §A.9 for complete results on all graphs.
Hyperparameters.
Hyperparameters for all experiments were kept identical unless otherwise stated.
We study the effect of DAG and sparsity penalties in the following paragraph.
For more details, please refer to Appendix §A.5 .
Importance of regularization.
Valid conﬁgurations C for a causal model are expected to be a) sparse and b) acyclic.
To promote such solutions, we use DAG and sparsity regularization with tunable hyperparameters.
We set the DAG penalty to 0 .
5 and sparsity penalty to 0 .
We run ablation studies on different values of the regularizers and study their effect.
We ﬁnd that smaller graphs are less sensitive to different values of regularizer than larger graphs.
For details, refer to Appendix §A.12.
Importance of dropout.
To train functional parameter for an observational distribution, sampling adjacency matrices is required.
We "drop out" each edge (with a probability of σ ( γ ) ) in our experiments during functional parameter training of the conditional distributions of the SCM.
Please refer to Appendix §A.14 for a more detailed analysis.
C ONCLUSION
In this work, we introduced an experimentally successful method ( SDI ) for causal structure discovery using continuous optimization, combining information from both observational and interventional data.
We show in experiments that it can recover true causal structure, that it generalizes well to unseen interventions, that it compares very well against start-of-the-art causal discovery methods on real world datasets, and that it scales even better on problems where only part of the graph is known.