Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling
CIFAR Fellow
Abstract
Latent variable models have become a go-to tool for analyzing biological data, especially in the field of single-cell genomics.
One remaining challenge is the identification of individual latent variables related to biological pathways, more generally conceptualized as disentanglement.
Although versions of variational autoencoders that explicitly promote disentanglement were introduced and applied to single-cell genomics data, the theoretical feasibility of disentanglement from independent and identically distributed measurements has been challenged.
Recent methods propose instead to leverage non-stationary data, as well as the sparse mechanism assumption in order to learn disentangled representations, with a causal semantic.
Here, we explore the application of these methodological advances in the analysis of single-cell genomics data with genetic or chemical perturbations.
We benchmark these methods on simulated single cell expression data to evaluate their performance regarding disentanglement, causal target identification and outof-domain generalisation.
Finally, by applying the approaches to a large-scale gene perturbation dataset, we find that the model relying on the sparse mechanism shift hypothesis surpasses contemporary methods on a transfer learning task.
Introduction
Machine learning methods have been key to gaining insights from large, high-dimensional genomic datasets, especially in single cell genomics [1].
Variational Auto-Encoders (VAEs, [2, 3]), a recent approach in inferring complex data generative processes, are often well-suited for these applications, because they allow for flexible model design, while keeping necessary changes to the inference procedure relatively minimal.
Many generative models have been proposed for analyzing diverse biological data modalities, including gene (RNA) expression, chromatin accessibility and quantitative protein measurements [4, 5].
However, VAEs suffer from a key disadvantage due to the lack of interpretability, reflected as the absence of direct correspondence between individual latent variables and biological processes [6].
While disentanglement-promoting VAEs (e.g., as in [7]) can help better relate the two in genomics [8], these methods often compromise the quality of the latent variables for downstream tasks [9].
This is in line with recent theoretical developments, showing that disentanglement, hereafter defined as the recovery of ground truth latent variables, is impossible from independent and identically distributed (i.i.d.)
measurements [10].
Recent efforts in disentanglement instead focus on the assumption of non-stationary data [11], where data must be (i) observed in different regimes, with known pairing between data points and regimes, (ii) generated such that regimes are incurring changes in latent variables, and (iii) latent variables are conditionally independent given the regime.
In this configuration, latent recovery with a conditional VAE [12] is theoretically possible.
Follow-up work [13, 14] also draws connections to causal representation learning [15], in which regimes may be seen as interventions on latent variables with unknown targets.
Recent advances in biotechnology made non-stationary data increasingly available, especially in the context of genetic or chemical perturbation screens with single cell genomic profiles as a readout [16, 17].
A recent Compositional Perturbation Autoencoder (CPA [18]) embedded perturbation profiles in latent space of an auto-encoder, focusing of the prediction the effect of unseen combinations of perturbations.
However, this approach did not exploit the non-stationary assumption in its probabilistic model.
Thus, to the best of our knowledge, there are so far no applications of the principles exposed in [11, 13] to these new data types.
Here, we propose to explore real-world applications of [11, 13].
The promise is that such models may eventually yield representations of perturbations and cells that have the benefits of a causal model, in being more mechanistically interpretable and more efficient for out-of-domain generalization.
Biologically, this means being able to identify biological processes (i.e., gene programs) that are affected upon perturbations [19], to project new perturbation data onto an existing perturbation atlas [20] with transfer learning [21].
After a brief presentation of recent advances in disentanglement (Section 2), we describe our motivation and effort in applying the introduced causal representation framework to gene expression measurements from single-cell RNA-seq (scRNA-seq) experiments with perturbations (Section 3).
Then, we introduce a benchmarking tool for simulation of single-cell data with perturbation and evaluation of algorithms for latent variable recovery (Section 4).
Finally, we present an application of the methods to a large-scale genetic screening experiment (Section 5) in which we show that the model based on sparse mechanism shift assumption outperforms all methods by a significant margin in a transfer learning task.
Such results suggest that causal inference is a promising paradigm for modeling the effects of perturbation in modern screenings data sets from molecular biology.
Background
The goal of representation learning for a given random vector x ∈ R d , is to learn a mapping to a lower dimensional (latent) space
z = ( z 1 , . . .
z p ) ∈ R p , with p ≪ d .
This paper is concerned with recovering the latent variables that initially generated the data, and attributing a causal semantic to the perturbations.
Therefore, we now introduce non-linear Independent Component Analysis (ICA) and its relationship to causal representation learning.
2.1 Non-linear Independent Component Analysis
ICA assumes that x is generated using p independent latent variables, called independent components [22].
More precisely, observations x are generated as x = f ( z ) + ϵ with f a mixing function and ϵ some exogenous noise.
The ICA literature focuses on the identifiable case (e.g. if f is a linear function, then the original z may be recovered).
In the general case of a non-linear mixing function f , however, the model is unidentifiable from i.i.d.
observations of x [23].
Given this negative result, several papers introduced identifiable forms [24, 25, 26, 27] of non-linear
ICA models, based on the assumption that components ( z i ) pi =1 are conditionally independent given some additional auxiliary random variable a ∈ R K :
[Equation]
Examples of auxiliary variables a include the past components in the case of time series analysis or some form of class label [26].
In this setting, latent recovery is possible up to a linear transformation under sufficient conditions [26].
The iVAE framework [11] proposes a VAE approach [2, 3] for learning the parameters θ of a generative model p θ ( x | z, a ) p θ ( z | a ) , as well as ϕ , those of a variational approximation q ϕ ( z | x, a ) to the posterior p θ ( z | x, a ) .
The iVAE specifies p θ ( z | a ) =
N ( µ a , I ) as a Gaussian location family with isotropic variance.
As for the VAE, the parameters ( θ, ϕ ) of the iVAE are learned via maximization of the evidence lower bound (ELBO):
[Equation]
2.2 Causal Inference from Unknown Interventions in Latent Space
Recent theoretical work [13, 14] proposed to investigate the assumption of sparse connections between the individual auxiliary variables ( a l ) Kl =1 and the latent components ( z i ) pi =1 , encoded in the form of a bipartite graph G a = ( { 1 , . . .
K } , { 1 , . . .
p } , E ) , where the edge set E encodes those connections.
In the case where a describes a discrete data regime (i.e., via one-hot encoding), the sparsity of G a corresponds to the sparsity of the mean vectors µ a of p θ ( z | a ) .
Compared to [11], the novel sparsity assumption allows for a new principle to achieve recovery of latent units up to a permutation, a much stronger result than the iVAE.
Perhaps as importantly, it also allows for the interpretation of the graph G a from a causal perspective.
More precisely, [13] considers the case where a ∈ { e 1 , e 2 , . .
, e K } , where each of e l for l ∈ { 1 , . . .
, K } is a one-hot vector encoding the l -th intervention, and each intervention has unknown targets on the set of latent components of z .
In that context, the unknown graph G a describes which latent components are targeted by the intervention, that is G ai,l = [-
-]1 if
and only if the l -th intervention targets z i .
In this context the sparsity assumption corresponds precisely to the sparse mechanism shift hypothesis from [15] i.e. that only a few mechanisms change at a time.
The VAE variant introduced in [13] (sparse VAE; sVAE) has an identical generative model as the one from the iVAE, except for the prior p θ ( z | a ) for which we apply a (stochastic) binary mask ˆ G a to the location parameter via element-wise product.
The estimation procedure follows the one from iVAE with an addition of the regularization term α ∥
ˆ G a ∥ 1 to the ELBO, where α is a hyper parameter.
To allow gradient-based optimization, the binary masks are independently sampled from
Bernoulli distributions with probabilities sigmoid ( γ ai ) .
We optimize for parameters γ a using the
Gumbel-Softmax gradient estimator [28, 29].
A Sparse Mechanism Shift Model for Single Cell Measurements
Single-cell genomics data is of immense interest across diverse research areas of biology such as development [30], autoimmunity [31], and cancer [32].
Due to the high-dimensional nature of the measurements and the high level of noise, the proper modeling and interpretation of such data remains a topic of active research [33].
The gene expression data matrix X =
( X 1 , . . .
X N ) ∈
R N × G consists of i.i.d.
observations of the random vector X , with N the number of instances (cells), and G the number of genes.
Individual entries of this matrix x ng count the number of transcripts aligned to gene g in cell n .
Typically, latent variable models are learned from gene expression measurements [5], and cell-level latent variables z (embeddings) are later used in downstream tasks such as clustering, imputation, or differential expression analysis [34].
One significant advance is the ability to actively intervene and change the properties of a single-cell by some genetic [35] or chemical [36] perturbation.
Each cell may be subjected to one or several (simultaneous) perturbations.
Assays capture this information alongside gene expression levels (often with sequencing of intervention-specific DNA barcodes), and report a summary of this readout as a multiple treatment intervention design matrix I ∈ { 0 , 1 } N × K , where K denotes the total number of treatments.
Our goal is to leverage the perturbation information I in order to disentangle the latent variables z and learn the sparse causal graph G a that describes the (unknown) targets of each intervention.
To do this, we modify the sVAE [13] (as well as the iVAE [11]) to allow for proper modeling of scRNA-seq data, following previous work [34].
Briefly, two important modifications are needed: (i) we choose a negative binomial distribution as the likelihood model for the observations, and (ii) we incorporate a scaling factor to the mean of the negative binomial for normalization purposes.
The implementation is performed within the scvi-tools codebase [37].
An appealing aspect of the sVAE model is the interpretability of G a , indicating which latent variables are affected by which perturbations.
Ideally, this would help considering individual latent variables as pathways and / or gene modules coordinately regulated by the perturbation, consistent with the underlying organization of a cellular molecular circuits [38].
Another interesting benefit from the causal semantic and the stronger disentanglement guarantees is that one can reasonably expect the learned representations to perform better at downstream tasks such as transfer learning [15], a growing use-case of deep generative models in single-cell genomics [21].
The presented modification of the sVAE is to the best of our knowledge the first proposal to explicitly model cellular perturbations as interventions on latent variables for the purpose of understanding causal mechanisms, while incorporating a model of experimental noise from single cell RNA-Seq assays.
A Sandbox for Evaluation of Learned Representations
Simulations We simulate perturbation gene expression data as follows.
We assume we have measurements from N cells.
Each of the cells, for example cell n ∈ [ N ] , has been exposed to a perturbation/intervention a n ∈ [ K ] .
We use K = 100 interventions in our simulations, and sample
N = 100 , 000 cells in total.
The first 80 interventions form the training set, and the last 20 form the test set.
Each intervention a ∈ [ K ] is represented by a sparse perturbation embedding µ a ∈ R K , where d z = 15 is the dimension of the embedding.
For each intervention, we treat the number of affected latent dimensions t a = { 1 , 2 , 3 } uniformly at random.
The indices of affected dimensions are also drawn without replacement from [ p ] , encoded into a binary vector β a,.
∈ { 0 , 1 } p .
Finally, each component µ a,i of µ a = ( µ a, 1 , . . .
, µ a,p ) for i ∈
[ p ] is generated as:
[Equation]
where δ 0 designates the Dirac delta distribution with mass at 0 .
For cell n , exposed to intervention a n , latent variable z n is generated as:
[Equation]
where each individual component of z n encodes the activity of a pathway, shifted by the intervention.
Measurements x ng from a single cell n and a gene g are generated from a count distribution:
[Equation]
where l n is the library size fixed to 10 5 , and the mixing function f is a neural network with three hidden layers of 40 units, Leaky-ReLU activations with a negative slope of 0.2, and a softmax non-linearity on the last layer to convert the outputs to counts [34].
The weight matrices of f are sampled according to an isotropic Gaussian distribution, with orthogonal columns, to make sure f is injective [13].
Metrics Based on the ground truth provided by our simulation framework, we may evaluate for disentanglement, causal structure learning, or transferability.
In particular, we report the MCC, a metric for permutation equivalence proposed by [11] that measures the average Pearson (or Spearman) correlation coefficients between pairs of ground truth and estimated latent variable, for the best possible permutation.
A high MCC means that we successfully identified the true parameters and recovered the true sources.
We also report R 2 metric for assessing linear equivalence.
To evaluate for learned causal relationships, we report the precision, recall and F1 score of the learned adjacency matrix of ˆ G a compared to the ground truth G a .
Finally, to assess transferability, we report the negative ELBO of data points from holdout perturbations after fine-tuning the embeddings, while keeping the generative model fixed [39]. [Table]
Benchmark Methods Our set of benchmark methods consist of the sVAE, the iVAE, as well as the vanilla VAE and the β -VAE [7].
We tune the hyper parameters of each method (number of epochs and sparsity penalty) with UDR [40].
All experiments were run for 5 different random initialisations, for d z ∈ [5 , 10 , 15 , 20] , d a = 100 and 500 cells per treatment.
Results We report the results for disentanglement, causal target identification and transferability in Table 1 for d z = 15 .
All methods have high R 2 , indicating linear disentanglement, but sVAE outperforms all methods in terms of permutation recovery of latent variables, evaluated via MCC.
sVAE also performs favorably compared to other methods when evaluating the inferred causal graph ˆ G a .
We include additional experimental results from our synthetic framework in Appendix A.
In terms of transferability, sVAE performs as well as other methods, with a slight advantage.
Transferability and Interpretability on a Large-scale Genetic Screen
[Table] We finally apply our benchmark models to real-world data, a recent large-scale Perturb-seq experiment [35] where 105 ,cells from an erythrocytic leukemia cell line (K562) were profiled after interventions targeting one or two of 112 genes, including cell cycle regulators, transcription factors, kinases, phosphatases [Figure] and genes of unknown function.
After quality control and data filtering, we retained 96 , 221 cells of undergoing 212 different genetic interventions and 8 , 907 unperturbed (control) cells.
Because of experimental limitations [19] [Figure] we observe signal only for a subset of several thousand genes (here, d = 3 , 000 ).
The goal of the experiment was to understand the mechanisms of genetic interactions and recover gene regulatory logics.
In order to simulate an out-of-domain scenario, we selected the top- 30 interventions with the most significant effect on gene expression, as assessed by the maximum mean discrepency [41] estimated with a linear kernel on a PCA with dimension
[Figure]
and held out the corresponding cells as a test set
[Figure]
We applied each studied method to this dataset.
Without ground truth, we use datapoints from held-out interventions to evaluate the models after transfer learning.
We report the negative ELBO for all models, evaluated on a validation data set (including additional cells with the same perturbations as in the training data), as well on the held-out perturbations (test set) in Table 2.
Both iVAE and sVAE are providing a large improvement in terms of data fit compared to VAE and β -VAE, for both metrics.
While the iVAE provides the best fit in terms of the validation data set, with a thin margin, it fits the test set poorly compared to sVAE.
This suggests that sVAE has stronger transfer capabilities than other methods, and learns a more causal representation of the data.
We also performed a preliminary biological interpretation of the sVAE model.
We first visualized the effect of perturbations in latent space in the form of a weighted adjacency matrix W ij for G a , where the weight encodes the shift in the mean of the corresponding latent component
z i for perturbation j
(Figure 2).
For visual convenience we focused on a subset of perturbations and latent components to retain the most informative data.
First, many perturbations have similar effects on latent variables, as it has been observed previously
I R F
I R F 1 , S E T
C E B P B , M A
P K
E T S 2 , M A P
K 1 M A P K
E T S 2 , M A P 7 D 1 S P
I
C E B P E , E T S
C E B P E , S P
I 1 C E B P
A
C E B P A , C E B
P E
C E B P A , K L F 1 C E B P E
C E B P E , F O
S B
C E B P B , O S
R
F O S B , O S R 2 C E B P B
C E B P B , L Y L
S G K 1 , T B X
T B X 2 , T B X
S G K 1 , T B X
A H R , F E V
F E V , M A P
D 1 F E V
F E V ,
I S L
C B F A 2 T 3 , F
E V
S 1 P R 2 , S G
K
I G D CC 3 , T
G F B R 2 P R T G
P R T G , T G F
B R
D U S P 9 , K L F
D U S P 9 , P R T
G
D U S P 9 , I G D
CC 3 D U S P
D U S P 9 , E T S
D U S P 9 , S N
A I 1 A R
I D
A H O X
A 13 H O X C
F O X
A 1 , H O
X B
F O X F 1 , H O
X B
F O X
A 1 , F O
X L
F O X L 2 , M E
I
S
F O X F 1 , F O X
L
I E R 5 L , L Y L
C B F A 2 T 3 , P
O U 3 F
F O X L 2 , P O
U 3 F
E L M S A N 1 ,
Z B T B
P T P N 12 , Z
B T B
F O X A 3 , H O
X B
F O X
A 1 , F O
X A
F O X A 3 , F O
X F
F O X
A 1 , F O
X F 1 H N F 4 A
F O S B , I K Z F
P R D M
I K Z F 3 , M A
P K
I K Z F 3 , M A
P 2 K
I K Z F 3 , M A
P 2 K
I K Z F
E T S 2 , I K Z F 3 T P
C B L , T G F B
R
A T L 1 C L D N 6 R
H
O X F
R H
O X F 2 , Z B
T B
C NN 1 , U B A S
H 3 B
C B L , U B A S
H 3 B
C NN 1 , U B A S
H 3 A
C B L , C NN
S A M D 1 , Z B
T B
U B A S H 3 B , Z
B T B
C B L , P T P N
P T P N 12 , Z
B T B
P T P N 12 , U
B A S H 3 B
P T P N 12 , U
B A S H 3 A
S A M D 1 , U B
A S H 3 B
P T P N 12 , S
A M D
P T P N 12 , P T
P N
P T P N 9 , U B
A S H 3 B P T P N
C B L , P T P N
B P G M , S A M
D
B P G M , Z B T
B
K L F 1 , S E T
C L D N 6 , K L F
A H R , K L F
F O X
A 1 , K L F
C O L 2 A 1 , K L
F
C O L 2 A 1 C O L 1 A
R H
O X F 2 , S E
T S E T
C E B P E , S E T
[Table]
[Figure]
at the level of individual genes and programs [19].
This can be interpreted as the perturbed genes being a part of the same pathway.
Indeed, perturbations involving the same gene (in different combinations) grouped together by their shared effect on latent factors, as did those involving different genes from related pathways (e.g. FOXO and homeobox TFs affecting cell differentiation), whereas perturbations in genes from different pathways had different effects (e.g. IRF1 vs. CEBP family TFs).
Second, we investigated how statistics of the number of perturbed latent variables and / or of the effect size was changing according to whether one or two genes were targeted in the cell (Figure 1a).
The distribution of both statistics is significantly higher for double vs. single gene perturbations, as overall expected.
Third, we sought to assess whether the learned latent variables are reflective of known patterns in genetic interactions.
Two examples of genetic interactions are pointed out in Figure 1b.
In the first one, we may notice that the latent shift for the perturbation that involved a combination of perturbations in
CEBPA and KLF1 has a pattern mostly similar to the shift of a single gene perturbation CEBPA, as previously reported [35].
This is an example of a dominant interaction, already visible in Figure 2, in other combinations (e.g., DUSP9, ETS2).
In the second example (CLB, CNN1), the sparsity pattern identifies two latent variables (number 39 and 99; black rectangle) with a shift that did not appear in any of the individual perturbations.
We applied Integrated Gradients [42] to each of those two components of the encoder network to obtain a list of 50 most important genes, and used EnrichR [43] to obtain an associated gene signature.
A positive change in latent variable 99 was associated with hemoglobin alpha binding, and hydrogen peroxyde metabolic process, both important in the context of erythocytes (the source cell line).
Latent variable 39 was associated with RNA binding.
Discussion
We propose to explicitly model perturbations in single-cell genomics as interventions on a latent space, with a causal semantic.
This naturally leads to our application of the sVAE [13] and iVAE [11] framework to disentangle the latent space of single cell data by leveraging additional knowledge of perturbations.
We provided a benchmarking framework for assessing the performance of the learned representations in terms of level of disentanglement, causal target identification, as well as transfer learning.
In simulated data, both approaches outperform the β -VAE and the vanilla VAE, with a strong advantage for sVAE, explicitly assuming sparsity in mechanism shifts for each perturbation.
We also applied all methods to a real dataset, and our preliminary analysis suggests that sparsity may help in transfer learning, interpretability, as well as capturing genetic interactions.
Importantly, we see in Figure 2 that multiple latent variables are affected by each intervention, which suggests more informative constraints to the model could be added to further improve its interpretability (e.g., [44]).
The hypotheses from the sVAE model in its current state present, however, two major limitations for biological applications.
First, although it may be reasonable to expect that genetic (and often chemical) interventions directly trigger a sparse subset of a cell’s circuitry (e.g., blocking a single pathway [19]), there are important molecular feedback mechanisms that can induce indirect downstream effects in other pathways, especially as increasing time passes from the initial perturbation [45].
Because many experiments measure gene expression from hours to days after intervention, sparsity may be a limiting assumption without resolving interactions between pathways (e.g., as in more traditional causal discovery learning methods [46, 47, 48, 49]).
This issue could be resolved in the future with the potential availability of time-resolved single-cell perturbation experiments.
Second, perfect disentanglement requires the intervention to at least cover all of the latent variables in the model (see precise conditions in [13, 14]), but because experiments only focus on subsets of perturbations (due to cost and labor limitations), not all latent variables may be impacted.
More comprehensive perturbation atlases, such as recent genome-wide screens [50] or combinatorial screens, may help mitigate this issue, as does the fact that many different interventions in biological systems converge on the same processes. Acknowledgments and Disclosure of Funding
We thank Sébastien Lachapelle and Chandler Squires for insightful conversations about causal latent variable models.
We acknowledge Kelvin Chen, Taka Kudo, Jan-Christian Huetter, Zia Khan, Basak
Eraslan, and Anna Klimovskaia Susmelj for discussions about modeling single-cell perturbation data sets.
{+Last but not least, we acknowledge the reviewers from the NeurIPS 
+}
Workshop on Causal Machine Learning for Real-World Impact who provided very insightful feedback about this manuscript. Disclosures: Romain Lopez, Natasa Tagasovska, Stephen Ra, and Kyunghyun Cho are employees of Genentech.
Stephen Ra and Kyunghyun Cho have equity in Roche.
Jonathan Pritchard acknowledges support from grant R01HG008140 from the National Human Genome Research Institute.
Aviv Regev is a co-founder and equity holder of Celsius Therapeutics and an equity holder in Immunitas.
She was an SAB member of ThermoFisher Scientific, Syros Pharmaceuticals, Neogene Therapeutics, and
Asimov until July 31st, 2020; she has been an employee of Genentech since August 1st, 2020, and has equity in Roche.
Code Availability Statement
We implement our new model and benchmarks using the scvi-tools library, and release it as opensource software at https://github.com/Genentech/sVAE .
A Additional results from synthetic experiments
For quantitative comparison of the considered baseline methods, we leverage the availability of ground truth latent variables in simulated data sets.
In the proposed sandbox in section 4, we have control over: the size of the effect for interventions - e a , the sparsity of the interventions - t a , as well as the dimensionality of the latent and auxiliary variables.
The aim of this paper is to evaluate the extent to which sparse, identifiable methods provide disentangled, or ultimately “generalizable/transferable” representations.
To systematically assess the quality i.e. transferability of the learned representations, we propose the following scenarios:
• transfer to unseen, in-domain interventions (hold-out interventions form a data set with same level of sparsity and effect size); • transfer to unseen out-of domain interventions (hold-out interventions form a data set with different level of sparsity or effect size).
A.1 In-domain interventions
The results included in the main section of the paper correspond to the in-domain scenario.
Namely, we generate a sample data set with sparse interventions, we train each baseline on cells affected by only 80 of those interventions, and we evaluate on the hold out, not seen 20 interventions.
Here, we extend this results to different dimensionality of the latent variable.
In Table 4, we include the results for in-domain interventions with same levels of sparsity in both train and test sets.
That is, we train a model on a data set where a fraction s ′ of all possible edges in G a are included, and we test on a data set with same sparsity.
[Table] d z =0 .
59 ± 0 .0 .
69 ± 0 .0.71 ± 0.
63 ± 0 .d
z =0 .
55 ± 0 .0 .
66 ± 0 .0.72 ± 0.
58 ± 0 .d
z =0 .
56 ± 0 .0 .
68 ± 0 .0.72 ± 0.
58 ± 0 .d z =0 . 47 ± 0 .0 . 59 ±
[Table]
Our experiments for the in-domain interventions provide the following takeaways:
• observing the ELBO scores, the sparser the ground truth latent model is, the more difficult it is to fit it for all baselines • observing the MCC scores, disentanglement is easier to achieve in smaller dimensionality of the latent space and when sparse shifts are present • observing the area under the Precision recall curve (AUC PR), we notice that the identifiable methods iVAE and sVAE are more successful in recovering the causal structure model then their unsupervised counterparts
A.2 Out-of-domain interventions
Here, we use the flexibility of our sandbox to simulate different scenarios to mimic different interventions regarding change in effect size from train to test set, or change in sparsity of the bipartite graph
G a .
In Table 6, we include the results for out-of-domain interventions with different levels of sparsity between train and test sets.
That is, we train a model on a data set where s ′ of all possible edges in
G a are included, and we test on a data set with reduced sparsity, s ′′ , s.t. s ′′ ≥ s ′ .
Similarly, we include results for different effect sizes.
We train all models on a smaller effect interventions e ′ and test on interventions with larger effects e ′′ .
[Table]
[Table]
[Table]
[Table]
Our experiments for the out-of-domain interventions provide the following takeaways:
• observing the ELBO scores, the more sparse the ground truth latent model in the transfer domain, the more difficult it is to fit it for all baselines • observing the MCC scores, disentanglement, or identifiability of the true latent variables is not possible in the ood interventions.
• observing the area under the Precision recall curve (AUC PR), we notice that the identifiable methods iVAE and sVAE are more successful in recovering the causal structure model then their unsupervised counterparts.
• it is possible to leverage representations/models learned on sparser interventions, to recover a causal structures in data sets with denser interventions (PR AUC, last column).