Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxies
Anonymous Author(s)
AffiliationAddress email
Abstract
Estimating the effect of an intervention while accounting for confounding variablesis a key task in causal inference.
Oftentimes, the confounders are unobserved, butwe have access to large amounts of unstructured data (images, text) that containvaluable proxy signal about the missing confounders.
This paper demonstratesthat leveraging unstructured data that is often left unused by existing algorithmsimproves the accuracy of causal effect estimation.
Specifically, we introduce deepmulti-modal structural equations, a generative model in which confounders arelatent variables and unstructured data are proxy variables.
This model supportsmultiple multi-modal proxies (images, text) as well as missing data.
We empirically demonstrate on tasks in genomics and healthcare that our approach correctsfor confounding using unstructured inputs, potentially enabling the use of largeamounts of data that were previously not used in causal inference
Introduction
An important goal of causal inference is to understand from observational data the causal effect ofperforming an intervention—e.g., the effect of a behavioral choice on an individual’s health [33].
As an initial motivating example for this work, consider the problem of determining the effect ofsmoking on an individual’s risk of heart disease.
This problem is complicated by the presence of confounders: e.g., it is possible that individualswho smoke have a higher likelihood to be sedentary, which is a lifestyle choice that also negativelyimpacts their heart disease risk.
If the individual’s lifestyle is available to us as a well-defined feature,we may adjust for this factor while computing treatment effects [33].
However, confounders are oftennot observed and not available in the form of features, making accurate causal inference challenging.
Oftentimes, datasets in domains such as medicine or genomics come with large amounts of unstructured data —e.g., medical images, clinical notes, wearable sensor measurements [7].
Thisdata contains strong proxy signal about unobserved confounding factors—e.g., wearable sensormeasurements can help reveal individuals who are sedentary.
However, existing causal inferencemethods usually do not leverage this “dark data” for causal effect estimation [18].
The goal of this paper is to develop methods in causal inference that improve the estimation of causaleffects in the presence of unobserved confounders by leveraging additional sources of unstructuredmulti-modal data, such as images and text.
For example, given time series from patients’ wearables,our methods may disentangle the effects of being sedentary from the effects of smoking by usingclusters in the sensor measurements (which would correspond to groups of active and sedentaryindividuals) as a proxy for the patients’ lifestyles and without requiring explicit lifestyle features.
Concretely, our paper formalizes the task of estimating causal effects using rich, unstructured multimodal proxy variables (e.g. images, text, time series) and introduces deep multi-modal structural equations, a generative model in which confounders are latent variables.
This model supports multiplemulti-modal proxy variables and can perform causal effect estimation with missing data by leveragingapproximate variational inference and learning algorithms [6] developed for this problem.
We evaluate our methods on an important real-world causal inference problem—estimating the effectsof genetic mutations in genome-wide association studies (GWASs)—as well as on benchmarks derivedfrom popular causal inference datasets.
On the GWAS task, we demonstrate that our algorithmsenable incorporating new types of environmental data (e.g., historical weather time series) whichenables discovering causal genetic factors in plants more accurately than using modern state-of-the-art
GWAS analysis methods.
Our experiments demonstrate the potential of using large amounts of datathat were previously left unused by existing methods to improve causal effect estimation.
Contributions In summary, this paper makes three contributions: (1) we define the task of estimating causal effects using rich, unstructured multi-modal proxy variables; (2) we introduce deepmulti-modal structural equations, a generative model tailored to this problem, and we describe associated variational learning and inference algorithms; (3) we demonstrate on an important real-worldproblem (GWAS) that unstructured data can improve causal effect estimation, enabling the use oflarge amounts of “dark data” that was previously not used in causal inference.
Background
Notation Formally, we are given an observational dataset D = { ( x ( i ) , y ( i ) , t ( i ) ) }
ni =1 consisting ofn individuals, each characterized by features x ( i )
∈ X ⊆ R d , a binary treatment t ( i ) ∈ { 0 , 1 } , and ascalar outcome y ( i ) ∈ R .
We assume binary treatments and scalar outcomes, although our methodnaturally extends beyond this setting.
We also use z ( i ) ∈
R p to model latent confounding factors that influence both the treatment and theoutcome [26].
We assume that ( x ( i ) , y ( i ) , t ( i ) , z ( i ) ) represent i.i.d.
realizations of random variables
X, Y, T, Z ∼ P from a data distribution P that decomposes as P ( X | Z ) P ( Y | Z, T ) P ( T
| Z ) P ( Z ) (thus
Z is indeed a confounder).
We are interested in recovering the true effect of T = t in terms of itsindividual and average treatment effects:
[Equation]
where do ( · ) denotes an intervention [35].
Structural Equations One of the main approaches to causal inference are structural equationmodels of the form
[Equation]
where Z ∼ p
( Z ) is drawn from a prior and the ε i are noise variables drawn independently fromtheir distributions [10].
Structural equations define a generative model of the data.
When this modelencodes the true dependency structure of P , we can estimate the true effect of an intervention byclamping T = t and drawing samples.
Deep Structural Equations Equations 2 can be parameterized with deep neural networks, whichyields deep structural equation models.
Expressive neural networks may better fit the data and recoverthe true distribution, which improves causal inference; they also often scale well to large datasets.
Such models have been used for GWAS analysis [46] and to correct for proxy variables [26].
Causal Effect Estimation With Unstructured Proxy Variables
Oftentimes, datasets in domains such as medicine or genomics come with large amounts of unstructured data (medical images, clinical notes), which contains strong proxy signal about unobservedconfounding factors.
Our paper seeks to develop methods that leverage unstructured data withincausal inference.
We start by formalizing this task as causal effect estimation with unstructured proxyvariables; these proxies may come from multiple diverse modalities (images, text).
3.1 Task Definition
Formally, consider a causal inference dataset D = { ( x ( i ) , y ( i ) , t ( i ) ) } ni =1 in which x ( i ) =(
x ( i ) 1 , x ( i ) 2 , . . .
, x ( i ) m ) is a vector of m distinct input modalities x
( i )
j ∈ X j
(e.g., images, text, timeseries, etc.).
In other words, X = X 1 × . . .
×
X m , where each X j corresponds to a space of images,time series, or other unstructured modalities.
Here, t ( i ) ∈ T (binary or continuous) is the treatmentand y
( i ) ∈
Y is the output.
Some modalities may also be missing at training or inference time.
We assume that the x ( i ) j are i.i.d.
realizations of a random variable X j with X =
( X 1 , . . .
, X m ) .
Weare interested in recovering the true effect of T = t in terms of the individual and average treatmenteffects.
We are specifically interested in estimating the individual treatment effect (ITE) from arbitrarysubsets of modalities M ⊆ { 1 , 2 , ..., m } , indicating that certain inputs may be missing at test time.
Y [ x, t, M ] =
E [ Y | do ( T = t ) , X j = x j for j in M ] ITE ( x, M )
=
Y [ x, t = 1 , M ] − Y [ x, t = 0 , M ] (3)
3.2 Motivating Applications
To help make this setup more concrete, we define two motivating applications.
Healthcare Consider the task of determining the effect of smoking on heart disease from anobservational dataset of patients.
The observational study may contain additional unstructured dataabout individuals, e.g., clinician notes, medical images, wearable sensor data, etc.
This data may holdinformation about hidden confounders: for example, raw wearable sensor data can be clustered touncover sedentary and active indivudals, revealing a latent confounding factor, sedentary lifestyle.
Genomics Consider the problem of estimating the effects of genetic variants via a genome-wideassociation study (GWAS).
Modern GWAS datasets in plants or humans feature large amounts ofunstructured inputs [7]: clinical notes, medical records, meteorological time series.
Historical weatherdata (e.g., precipitation, wind strength, etc.) can reveal distinct climatic regions that affect phenotypesin a plant dataset and whose confounding effects should be corrected in a GWAS [51].
Deep Structural Equations for Causal Effect Estimation
Next, we derive models and inference algorithms for the task of causal effect estimation withunstructured proxy variables.
Our approach uses deep structural equations to extract confoundingsignal from the multi-modal proxies x ( i ) j .
We use neural networks because they naturally handleunstructured modalities via specialized architectures (e.g., convolutions for images) that can learnlearn high-level representations over raw unstructured inputs (e.g., pixels).
Parameterizing structural models with neural networks also presents challenges: they induce complexlatent variable models that require the development of efficient approximate inference algorithms[10, 6].
We present an instantiation of Equations 2 that admits such efficient algorithms.
4.1 Deep Multi-Modal Structural Equations
We start by introducing deep multi-modal structural equations (DMSEs), a generative model forestimating causal effects using unstructured proxy variables in which confounders are latent variablesand unstructured data are proxy variables.
We define a DMSE model as follows:
[Equation]
where P X j , P Y are probability distributions with a tractable density and the µ, σ, π, θ are functionsparameterized by neural networks that output the parameters of their respective probability distributionas a function of ancestor variables in the causal graph (Figure 1).
The P X j , P Y are typically Gaussian conditioned on Z .
Figure 1 shows the dependencies.
These modeling choices will shortly enable usto apply stochastic variational inference algorithms.
Note that the models for µ X j , σ X j are domain-specific—certain modalities will benefit from specialized architectures, e.g., a convolutional model for images and a recurrent one for time series.
See
Appendix B for details on recommended architectures for all the variables µ, σ, π, θ .
4.2 Approximate Inference and Learning Algorithms
In the full multi-modal setting, the X j are conditionally independent given Z (Figure 1), whichenables us to apply efficient algorithms inspired by Wu & Goodman [52].
These algorithms offerthe following improvements: (1) we may perform learning, inference, and causal estimation withmissing modalities X j ; (2) the process for performing causal inference is simpler than in previouswork [26] and does not require training auxiliary inference networks.
The DMSE model induces a tractable joint density p ( X, Y, T, Z ) , which allows us to fit its parametersusing stochastic variational inference by optimizing the evidence lower bound (ELBO):
[Equation]
where p ( y ( i ) , t ( i ) , z )
=
p ( y ( i ) | t ( i ) , z ) p ( t ( i )
| z ) p ( z ) and q ( z | x ( i ) , y ( i ) , t ( i ) ) is the approximate variational posterior.
We assume a total of m modalities.
Structured Multi-Modal Variational Inference
We may use the independence structure of p(Figure 1) to derive a more efficient structured form for q than the one prescribed by classicalvariational inference.
First, observe that because the true posterior factorizes as p ( z | x, t, y ) ∝( p ( z | t, y ) (cid:81) mj
=1 p ( z | x j )) / (cid:81) m − 1 j =1 p ( z ) , the optimal approximate posterior q must also factorize asq ( z | x, t, y )
∝ ( q ( z | t, y ) (cid:81) mj
=1 q ( z | x j )) / (cid:81) m − 1 j =1 p
( z ) .
This decomposition implies that we can maintain the optimal structure of q by training modality-specific inference networks ˜ q ( z | t, y ) and ˜ q ( z | x j )such that q ( z | x j ) =
˜ q
( z | x j ) p ( z ) and q ( z | t, y ) =
˜ q
( z | t, y ) p ( z ) and by defining a joint posterior as
[Equation]
This network can be seen as a product of experts (PoE)
Computing the density q is in general not possible.
However, because p ( z ) , q ( z | t, y ) and q ( z | x j )are Gaussians, we may use the fact that a product of Gaussians with means µ i and covariances V iis µ = ( (cid:80) µ i T i ) /
( (cid:80) T i ) and V =
( (cid:80) i T i ) − 1 , where T i =
1 /V
i .
Thus, computing q ( z | x, t, y ) forany subset of modalities is possible without having to train an inference network for each subset ofmodalities separately.
A Multi-Modal ELBO Training this architecture with ELBO X will not necessarily yield goodsingle-modality inference networks, while training with each modality separately will prevent thenetwork from learning how the modalities are related to each other.
Hence, we train our model witha sub-sampled ELBO objective [52] that is computed on the full set of modalities, each individualmodality, and a few subsets of modalities together.
For this, we randomly pick at each gradient steps non-empty subsets { M k } sk =1 of the set of modalities M k ⊆ { 1 , 2 , ..., m } .
The final objective is
ELBO X + (cid:80) mj
=1 ELBO { j } + (cid:80) sk =1 ELBO M k , where
[Equation]


Our model doesn’t train extra auxiliary inference networks, unlike that of Louizos et al.
4.3 Deep Gaussian Structural Equations
[Figure]
DMSE can be simplified in settings in whichthere is only one type of auxiliary data x (i.e.,m = 1 ).
This yields the same model as Equations (4) except that we have only one equation X ∼ P X
( θ X ( Z )) for the proxy variables.
This simplified model, which we refer to asdeep gaussian structural equations (DGSEs),has a tractable joint density p ( X, Y, T, Z ) =p ( Z ) p ( X | Z ) p ( T | Z ) p ( Y | Z, T ) , where the latent Z is Gaussian.
The DGSE model can also be fit using stochastic variational inference by optimizing the ELBO objective (cid:80) ni
=1 E q (cid:2) log p ( x ( i ) , y ( i ) , t ( i ) , z ) − log q ( z | x ( i ) , y ( i ) , t ( i ) ) (cid:3) , where q ( z | x, y, t ) is an approximate variational posterior.
We optimize the above objective using gradient descent, applying thereparameterization trick to estimate the gradient.
We compute the counterfactual Y [ x, t ] usingauxiliary inference networks as in earlier work [26].
See Appendix B for the full derivation.
4.4 Properties of Deep Multi-Modal Structural Equations
Recovering Causal Effects The DMSE and DGSE models determine the true causal effect whentheir causal graph is correct and they recover the true data distribution.
The following argument isanalogous to that made in most previous methods on causal deep generative models [26, 53, 32].
Theorem 1.
The DMSE and DGSE models recover the true ITE ( x, M ) for any subset M ⊆{ 1 , 2 , ..., m } of observed modalities whenever they represent the true data distribution p ( x, y, t, z ) .
Proof: We establish the theorem for DMSEs; the proof for DGSEs is analogous with m = 1 .
Letx M = {
x j | j ∈ M } be the data from the observed subset of modalities.
We need to show thatp ( y | x M , do ( t = t ′ )) is identifiable for any t ′ .
Observe thatp
[Equation]
where the second equality follows from the rule of do-calculus (applying backdoor adjustment).
Sinceour proof holds for any t ′ and all elements on the right-hand side are identifiable, the claim follows.
Note that in practice our assumption may not hold (e.g., neural network optimization is non-convexand may fail), but there is evidence of both real-world failure modes [42] as well as success storieswhere deep latent variable models provide useful causal estimates [53, 32, 27, 45].
See our Discussionsection for additional details.
Computing Causal Effects Given a subset of modalities M , we can compute the ATE & ITE as 187 E ( ITE ( x, M )) , where ITE ( x, M ) =
Y [ x, t = 1 , M ] − Y [ x, t = 0 , M ] and
[Equation]
where we use our variational posterior formulation from Equation 6 to approximate the true posteriorp ( z | x ) .
Experimental Results
5.1 Synthetic Demonstration Dataset
We start with a demonstration that provides intuition for why proxy variables are important, and howunstructured proxies can serve in place for featurized (structured) ones.
The following small-scalesynthetic setup (Louizos et al.
[26]) involves a data distribution P over binary variables
y, t, z, x :
[Equation]
where 0 < ρ x , ρ t < 1 are parameters.
Crucially, we introduce an unstructured proxy variable X thatrepresents an “image version” of x .
The variable X will be a random MNIST image of a zero or one,depending on whether x = 0 or x = 1 .
Formally, X is distributed as follows:
P ( X | x = 1) is unif.
over MNIST images of ”1”
P ( X | x = 0) is unif.
over MNIST images of ”0”
[Table]
First, this is a setup that requires us to [Table] model proxies (see also Appendix A):treating X as a confounder as using amodel of P ( y | X , t ) recovers the true
[Table]
ATE only when ρ t = ρ x = 0 .
5 (i.e.,when X is perfectly informative of z ),otherwise it fails. [Table]
We also show that structural equationssolve this task.
We fit DGSE mod-
[Table]
els to 10000 training data points of
[Table]
{ x, y, t } (the B INARY setting) as wellas of { X , y, t } (the I MAGE setting).
We note the Average Treatment Effect (ATE) on 600 test datapoints sampled independently from P and we report results in Table 1.
DGSE performs as well asmodels trained on the featurized proxy variables
5.2 Benchmark Datasets for Causal Effect Estimation
embedding (8 attrs + image) shows some increase in ATE error as
(9 attrs) is better than dropping this modality altogether (
[Table]
IHDP The Infant Health and Development
Project (IHDP) is a popular benchmark for causal inference algorithms [13] that contains the outcomesof comprehensive early interventions for premature,low birth weight infants.
We create a benchmarkfor multi-modal causal inference based on IHDP inwhich we can replace certain features with their ”unstructured version”.
We choose 9 of the 25 featuresavailable in IHDP in order to magnify their relativeimportance and accurately measure the effects oftheir removal.
Please refer to Appendix C and D fordetailed setup.
STAR The Student-Teacher Achievement Ratio(STAR) experiment [1] studied the effect of classsize on the performance of students.
We considersmall class size as treatment; the outcome is thesum of the reading and math scores of a student.
We ‘derandomize’ this dataset by removing 80%of the data corresponding to white students in thetreated population.
Similarly to IHDP, we select 8attributes for the multi-modal experiment.
Furtherdetails can be found in the Appendix C and D.
Adding Unstructured Modalities We create abenchmark for multi-modal causal inference derived from IHDP and STAR in which we replacefeatures with unstructured inputs that contain the same information as their featurized versions.
On
IHDP, we replace the attribute ‘baby’s gender’ with the CLIP embedding [22] of an image of a childbetween ages 3 to 8 years, drawn from the UTK dataset [58].
On STAR, we replace the attributescorresponding to the student’s ethnicity and gender by selecting an image of a child with the sameethnicity and gender from the UTK dataset.
We train and evaluate models on datasets where the image ‘replaces’ the attribute.
(e.g., 8 ATTRS+ IMAGE ).
We also consider two other settings for comparison: a) the original attribute is included(e.g., 9 ATTRS ) and b) the attribute is dropped from the reduced set of input features (8 ATTRS ).
Results As seen in Table 2, the degradation in ATE error from replacing the baby’s gender by aphotograph is lower as compared to removing the attribute entirely.
This shows our models leveragesignal found in the unstructured image modality with the help of deep neural networks.
We compare these results with a simple Ordinary Least Squares model (OLS) baseline as describedby Shalit et al.
[44] to predict treatment effect.
OLS shows a similar behavior when replacing baby’sgender with corresponding image, however ATE errors are generally worse as compared to ATE errorproduced by DGSE.
We also compare this with Counterfactual Regression Network (CFRNet)
[16]baseline.
However CFRNet did not show benefits of using image modality unlike our approach.
In Table 6, replacing gender and ethnicity attributes on STAR with the corresponding image improves
ATE errors as compared to dropping these two attributes entirely.
This shows that we can use animage to extract multiple attributes while doing causal inference.
The CFRNet baseline shows asimilar behavior, but the difference between average ATE errors across different setups is small.
5.3 Genome-Wide Association Studies
We evaluate our methods on an important real-world causal inference problem—genome-wideassociation study analysis (GWASs).
A GWAS is a large observational study that seeks to determine the causal effects of genetic markers (or genotypes) on specific traits (known as phenotypes
Standard error of the mean (sem) is computed over 10 seeds.
[Table]
Background and Notation Asmotivation, consider the problemof linking a plant’s genetic variants t ∈ { 0 , 1 } d with nutritionalyield , which we model via a variable y ∈ R .
Our goal is todetermine if each variant t j iscausal for yield, meaning that itinfluences biological mechanismswhich affect this phenotype [8].
We also want to leverage largeamounts of unstructured data x(e.g., health records, physiological data) that are often availablein modern datasets [7, 23].
A key challenge in finding causalvariants is ancestry-based confounding [2, 49].
Suppose thatwe are doing a GWAS of plants from Countries A and B; plants in Country A get more rainfall, andthus grow faster and are more nutritious.
A simple linear model of y and t will find that any variantthat is characteristic of plants in Country A
(e.g., bigger leaves to capture rain) is causal for nutritionalyield.
Methods and Baselines Most existing GWAS analysis methods for estimating the effect of avariant t j rely on latent variable models: (1) they treat all remaining variants x as proxies and obtainz via a linear projection (e.g., PCA [36, 37] or LMM [56, 24]) of x into a lower dimensional spacewhere genomes from Countries A and B tend to form distinct clusters (because plants from the samecountry breed and are similar); (2) we assume a linear model β ⊤ t of y and add z into it, whicheffectively adds the country as a feature ( z reveals the cluster for each country); this allows the modelto regress out the effects of ancestry and assign the correct effect to variants t (one at a time).
Baselines.
Our main baselines are Principal Component Analysis (PCA) and Linear Mixed Model(LMM), as described above and implemented via the popular LIMIX library [25].
We also compareagainst Factor Analysis (FA), a standard linear technique for deriving latent variables, Uniform
Manifold Approximation and Projection (UMAP), a manifold learning technique for dimensionalityreduction [28] and a linear model with no correction for confounding.
DSE Models.
We compare against deep structural equation models that leverage one or more sets ofproxy variables coming from the following unstructured modalities: genomic sequences, weather time series, simulated physiological time series.
We fit DSE models via a stagewise strategy analogous tohow classical GWAS models are fit: (1) we fit the component p ( z k | x k ) for each proxy x k ; (2) we fita linear model of y given t and the z to estimate causal effects.
Thus, our p ( x k | z ) components aredeep, while p ( y | x, z ) is shallow (following standard assumptions on epistasis in GWAS).
[Figure]
5.3.1 Simulated Human GWAS
[Figure]
We used the 1000 Human Genomes [3] dataset to generate a simulated multi-modal GWAS dataset, following the biologically-inspired“Spatial” strategy studied in [46].
In addition to genotypes, we generated random physiological time series by sampling Fourier seriesconditioned on the confounders.
Please see Appendix F for details.
Table 3 shows our results.
Amongst the non-oracle baselines, multimodal DSE has the smallest error in estimating causal effect, as wellas the highest precision and recall at identifying causal SNPs.
Theuni-modal DSE, while worse than LMM, still beats PCA and FA.
Notethat the LMM model is not compatible with multiple unstructuredproxies.
In general, we see that precision starts to deteriorate fasterthan recall, suggesting that false positives are more likely from theweaker linear deconfounding methods such as PCA/FA.
Our resultshere further support that additional sources of unstructured multi-modal data can improve GWAS.
5.3.2 Real-World Plant GWAS
modality is missing
DMSE still produces reasonable ATE errors as it can compute the latent subset modalities better
Dataset
Model
[Table]
We also tested our methods on realworld plant GWAS dataset from the1001 Genomes Project for Arabidopsis
Thaliana plants [51].
The challengehere is that we don’t know true causaleffects; therefore, we define a phenotype y for which the true causal effectis zero—specifically, we use the GDPof the country where each plant grows.
Because the genomes of plants fromthe same country are similar, there exist spurious correlation caused by latent subpopulation groups z .
Our goal is to detect and correct for these confounding effects.
In addition to genomic data, we gather historical weather time series from the location of each plant (see Appendix E) and use both modalities to correct for confounding.
[Table]
We evaluate whether each model learned
[Table]
to correctly account for latent confounding effects by measuring the predictivepower of β ⊤ t for y , where β are thecausal effects and t is the vector of variants.
Specifically, we compare the R 2correlation between β ⊤ t and y —here,lower is better , since a model that haslearned the causal effects will not cor-
[Table]
rectly predict the phenotype.
In Table 5,we see the effect of extracting confound-
[Table]
ing variables using DSE as opposed to
[Table]
using the standard PCA technique.
We
[Table]
can see that the R 2 values produced us-
[Table]
ing DGSE for extracting confounders are
[Table]
closer to 0 as compared to using PCA.
[Table]
This experiment shows that neural network architectures are effective in deal-
[Table]
ing with unstructured genomic and weather data while correcting for confounding.
Please refer to
Appendix E for details.
5.4 Multimodal Experiments With Missing Modalities
We demonstrate the ability of the DMSE model to handle missing data.
Our IHDP and STARbenchmarks involve two modalities: images (e.g., baby’s gender in IHDP) and tabular data (e.g., theremaining features).
We compare DMSE and DGSE models on these datasets when some of themodalities may be missing.
For DMSE, we define two different modalities X 1 and X 2 for the tabular and image modalitiesrespectively.
For DGSE, we concatenate the image embedding to the tabular modality while trainingthe network.
We evaluate ATE while randomly dropping 50% of the images.
DMSE utilizes itsproduct-of-experts inference network to approximate the posterior distribution when modalities aremissing.
DGSE cannot do this, and we resort to feeding it a vector of zeros when an image is missing.
Table 4 shows that DMSE produces improved ATE estimates as compared to the DGSE modelingmodalities.
Related Work
Multi-Modal Causal Inference While previous work analyzed unstructured interventions t consisting of natural language [39–41] (e.g., determining the effect of a polite vs. a rude response) as wellas unstructured y [5, 32] (e.g., MRI images), our work explores unstructured x , not y, t .
Veitch et al.developed models that correct for confounders from a single unstructured confounder x derived fromtext [30, 48] or a graph [47].
These approaches rely on a propensity scoring framework—they train adiscriminative model of p ( t | x ) —hence do not support proxy variables or missing data, and requirepre-trained text embeddings.
Our method (i) works across all modalities (beyond text or graphs), (ii)supports arbitrary numbers of proxies, (iii) supports missing data by virtue of being generative.
Deep Latent Variable Models Representation learning in causal inference has been studied by
Johansson et al.
[16, 15, 17] and Sch¨olkopf et al.
Deep latent variable models find applicationsthroughout causal inference [26, 27, 53, 32, 57, 50, 19].
Pawlowski et al.
[32] study unstructuredoutcomes y (MRI scans), but do not support proxies.
Louizos et al.
[26] use variational auto-encodersto estimate confounders from proxies; we introduce a more structured model that handles multipleproxies that can be missing, and obviates the need for auxiliary modules.
Tran & Blei [46] proposeimplicit deep structural equations for GWAS; ours are explicit and thus easier to train.
Discussion and Conclusion
Identifiability Approaches to causal effect estimation that rely on deep learning [26, 44, 32, 45, 27,11, 57, 55] can never guarantee the recovery of causal effects: neural network is non-convex and hasno guarantees itself.
Other failure modes of deep latent variable models (DLVMs) include not havinga sufficiently expressive model, not having enough data to learn the model and failures of approximateinference.
That said, there is evidence of both real-world failures [42] and successes of DLVMs.[53, 32, 27].The DLVM approach is appealing to us because: (i) it naturally handles proxies asobserved variables conditioned on a latent and (ii) neural networks are effectively necessary to processunstructured data (alternatives are worse in practice).
Rissanen & Marttinen [42] empirically identifymultiple failure modes of DLVMs; however we also identify successes (e.g., in GWAS), and ultimatelythe validation of DLVM methods needs to be empirical in specific domains [45, 55, 47, 11, 19].
Missing Data We make the common assumption that data is missing at random (MAR).
This poseschallenges if, for example, patients missing outcomes are ones that are more likely to be sick.
Whentwo modalities and their missingness are correlated, their x j , x k nodes in Figure 1 could be merged,somewhat addessing the issue.
We leave the full exploration of non-MAR models to future work.
In conclusion, we proposed an approach based on deep structural equations to utilize useful datapresent in unstructured form to perform causal inference.
We showed that deep structural equationscan estimate causal effects with unstructured proxies across a real-world genomics task and onpopular benchmarks, including with missing data.
A Details of the toy experiment.
Data domains.
In this experiment, there are 5 variables: a latent binary variable Z , a binaryvariable X ∗ , an MNIST image X , and two other binary variables y, t , that is:
Z i , X ∗ i ∈ { 0 , 1 } : binary values
X i ∈
R 28 ∗ 28 : an MNIST image of 0 ory i , t i ∈ { 0 , 1 } : binary values i ∈ { 1 , 2 , · · · , N } , N = 3000 : datapoint index
The previously mentioned variables are sampled from the following distribution to form oursynthetic dateset:
P
[Equation] [Equation]
=
[Equation] [Equation]
[Equation]
P
[Equation] [Equation]
[Equation]
X = (cid:26) a binarized image of 0 when X ∗ =a binarized image of when X ∗ y =
t
Z
ATE computation.
As mentioned before, the definition of the Average Treatment Effect (ATE) is as follows:
[Equation]
Now we consider the first term within the outer expectation:
[Equation]
where the second last equation is due to the fact that y = t ⊕ Z .
Similarly, we can computethe second term as:
[Equation]
meanwhile, according to the generation process of our data, we know that
[Equation]
where x is an image of x ∗ .
Plugging int he previous results, we can compute the individualtreatment effect (ITE):
[Equation]
Notice that,
[Equation]
therefore we can plug in the previous equation and get the final result of ATE:
[Equation]
B Neural Architecture of Deep Structural Equations and ApproximateInference Networks
Causality in Machine Learning Pearl [34] describes structural equations to mathematicallyrepresent causal relations between variables, extending the idea from linear models to non-parametricmodels and graphs.
Xia et al.
[54] provide theoretical analysis of inductive biases needed to learncausal effects using neural models.
Neural networks have been used to learn counterfactual modeland propensity score model under no unobserved confounding setup [16, 55, 45].
Our Architecture In this section, we will introduce the details of the DGSE and DMSE architecturewe used.
X i denotes an input datapoint, i.e. the feature vector (possibly containing multiplemodalities), t i is the treatment assignment, y i denotes the corresponding outcome and Z i is thelatent hidden confounder.
Within DGSE and DMSE, the latent variable is modelled to be Gaussiandistributed.
For DGSE, we can model each of the corresponding factors (similar to Louizos et al.[26]) as:
[Equation]
where σ ( · ) is the sigmoid function, Bern is the Bernoulli distribution, D x , D z are the dimensionsof X and Z respectively, p [ X ij |
Z i ] is an appropriate probability distribution for the covariate j .
Ifthe treatment variable is not binary, we can modify the distribution appropriately to work with that.
Within DMSE, it is possible to further factorize the distribution p ( X i | Z i ) into product of distributionsover component modalities owing to the conditional independence.
If the outcome y is discrete, we parameterize its probability distribution as a Bernoulli distribution:
[Equation]
and if it is continuous, we parameterize its distribution as a Gaussian with a fixed variance ˆ v , definedas:
[Equation]
Here each of the NN i ( · ) is a neural network.
The posterior distribution for DGSE is approximated as
[Equation]
where
[Equation]
For DMSE, the posterior distribution is computed differently using Product-of-Experts (PoE)
[52]formulation, due to which it can handle missing modalities during training and inference gracefully.
The objective of DGSE model is the variational lower bound defined as:
[Equation]
The DMSE, on the other hand, requires a sub-sampled training objective to ensure that the modalityspecific posterior networks are trained and the relationships between individual modalities is captured.
For DGSE, we also define the auxiliary encoders and the extra term in the variational lower boundfollowing Louizos et al.
Auxiliary Encoder:
[Equation]
For discrete y i :
[Equation]
For continuous y i :
[Equation]
This introduces the following extra term in the Variational lower bound:
[Equation]
DMSE does not involve these extra terms within its ELBO objective.
Compared with Louizos et al.
[26], DGSE can be extended to be applied over different types ofarchitecture for the posterior distribution
q [ Z i | X i , t i , y i ] .
When X is an image (e.g. medical scans,patient photos), we can use a convolutional neural network, which has been shown to effectivein extracting information in image files [21].
In our experiments with image modality, we usedpretrained CLIP embeddings [22] in the first layer to extract relevant features from the images.
Toavoid the overwhelming difference between the image and two binary variables t, y , we also applydimension reduction techniques such as Principle Component Analysis to the embeddings of theimage before feeding it into the network that is shared with t, y .
When X is time-series data, (e.g.text, recording), we can change the architecture to recurrent neural networks such as Long Short Term
Memory [14].
More generally, we can choose modality specific architectures and make appropriatedesign choices to perform learning and inference over unstructured modalities as inputs.
C Setups used for IHDP and STAR Dataset experimentsC.1 IHDP Experiments
The data corresponding to non-white mothers in the treated set of children is removed so that causaleffect of the intervention cannot be estimated directly.
The column corresponding to mother’s raceis removed so that this confounder cannot be obtained directly from the input.
We consider 100replicates of this dataset, where the output is simulated according to setting ’A’ of NPCI package[9].
The true treatment effect is known as the simulation provides expected output values for bothvalues of binary treatment variable.
We train a DGSE model on each replicate with a 63/27/10 ratioof training, validation and test dataset size.
We set the latent dimension to be 20 units and the numberof hidden layers to be 2.
The hidden layers have size of 20 units.
The IHDP-Full Setting There are 25 input features in this experimental setting.
We report theabsolute error in ATE produced by DGSE and OLS for this setting in Table 7.
The IHDP-Mini Setting Here, we choose 9 features from the 25 input features so that removalof the feature ‘baby’s gender’ produces statistically significant treatment effect.
We used mutualinformation and F-statistics between each of the original 25 features and the target variable y toassess the importance of each feature in the initial 100 replicates of IHDP.
While making sure thatthe absolute ATE errors don’t deviate too much from the corresponding errors produced by IHDPFull setting, we experimented with several combinations of the high ranking features to select thefollowing 9 features in the IHDP-Mini setting.
Feature 6: ‘sex of baby’
Feature 0: ‘birth-weight’
Feature 1: ‘b.head’
Feature 2: ‘preterm’
Feature 3: ‘birth.o’6.
Feature 8: ‘mom married?’7.
Feature 9: ‘mom’s education lower than high school?’
Feature 12: ‘Smoked cig during pregnancy?
Feature 20: ‘harlem’
[Table]
Table 7 shows a comparison of absolute ATE errors between the IHDP-Full setting and IHDP-Minisetting.
Table 2 shows the comparison of our approach with CFRNet [16] and Ordinary Least Squares(OLS) approach.
OLS takes the concatenation of the covariates and treatment variable value as inputto produce output.
We see that the CFRNet baseline does not utilize the image effectively while
OLS shows small difference between the setting where baby’s gender was removed (8 attrs) andthe setting where baby’s gender was retained (9 attrs) setting.
Hence the replacement of image (8attrs+image) produces a small average improvement as compared to the setting where baby’s genderwas dropped (8 attrs).
We also note that we have dropped 6 replicates from the 100 IHDP replicatesunder consideration.
These 6 replicates showed a large degradation in ATE estimates by addingbaby’s gender (9 attrs) as compared to removing it (8 attrs).
C.2 STAR Experiments
We ‘derandomize’ this dataset by removing 80% of the data corresponding to white students in thetreated population.
The dataset [1] has 15 input attributes.
The true treatment effect can be estimateddirectly since the original dataset corresponds to a randomized controlled trial.
Hence, it is possibleto compute ATE error as the absolute difference between true ATE estimate and the ATE as predictedby the model.
Similar to IHDP, we choose the following subset of attributes for the STAR experiment
Feature 2: ‘Student grade’2.
Feature 3: ‘Student class-type’3.
Feature 4: ‘Highest degree obtained by teacher’
Feature 5: ‘Career ladder position of teacher’
[Table]
Feature 6: ‘Number of years of experience of teacher’
[Table]
Feature 7: ‘Teacher’s race’
[Table]
Feature 10: ‘Student’s gender’
[Table]
Feature 11: ‘Student’s ethnicity’
[Table] [Table]
In Table
[Table]
we repeat the experiment 100 times and report average ATE errors along with standard error.
We removed 8 repetitions in the experiment where DGSE or CFRNet showed lack of convergence asevidenced by very high validation loss on any setting of input attributes.
[Table]
D Evaluating Quality of Pre-Trained Embeddings
We demonstrate that our pre-trained embeddings contain useful signal by building a neural networkmodel that predicts the gender, age and ethnicity from the CLIP embedding of the correspondingimage.
We build a simple neural network model that takes the CLIP embedding of an image as input andpredicts the age of the person in that image.
We use 5-dimensional PCA embeddings of 500 randomlychosen images of people aged 10-45 years (corresponding to the age-group in the IHDP experiment).
We have an independent test dataset corresponding to 100 images chosen in a similar way.
Wesee that the R 2 value for the age prediction on test dataset is 0.45.
If we increase the size of PCAembedding to 50, this R 2 value increases to 0.58.
Thus, it is possible to extract the age informationfrom randomly chosen images using a simple neural network.
In the above setting, we also studied the classification accuracy of separate neural networks thatpredict gender and ethnicity from the CLIP embeddings.
We saw that gender was predicted with 94%accuracy and ethnicity was predicted with 58% accuracy using a 5-dimensional PCA embedding.
After increasing the size of PCA embedding to 50 dimensions, the gender prediction accuracyincreased to 95% and ethnicity prediction accuracy increased to 77%.
This further supports ouridea of replacing the attribute corresponding to ’s baby’s gender or student ethnicity/gender with anappropriate image.
E Plant GWAS
Setup We apply our deep structural equations framework for correcting the effects of confounding.
We fit a DGSE model in two stages: (1) first, we only fit the model of p ( T | Z ) using the DGSE
ELBO objective; (2) then we fit p ( Y | Z, T ) with a fixed Z produced by the auxiliary model q ( Z | T ) .
We found this two-step procedure to produce best results.
The subsampled SNPs corresponding toeach genome are taken as input X .
The encoder and decoder use a single hidden layer of 256 unitswhile a 10-dimensional latent variable Z is used.
This network is optimized using ClippedAdamwith learning rate of 0.01, further reduced exponentially over 20 training epochs.
The confoundingvariable for each genome can now be computed as latent representation produced by the VAE.
Tomeasure the success of confounding correction, we compute the R 2 values between the true GDP ofthe region and the GDP output as predicted via our model and the baselines.
If we have corrected forconfounding, then we should get low R 2 values.
Historical
Weather Data We used historical weather data collected from Menne et al.
Weuse per day precipitation data from year 2000 collected by weather station closest in distance to thelatitude/longitude coordinates of the location from which the SNPs of plant were collected.
For thelocations where weather data was missing, we replaced those entries with zeros.
F Simulated GWAS Experiments
We demonstrate that representations trained via standard deep latent variable models (i.e. variationalautoencoders [20]) are good representations of confounding variables in a simulated GWAS setting,with a time-series proxy.
F.
1 Data generating process
To simulate the confounders, SNPs (genotypes), and the outcomes (phenotypes), we follow the“Spatial” simulation setup from Appendix D.1 & D.2 of [46].
Namely, we randomly generate a lowrank factorization of the allele frequency logits F = σ − 1 (Γ S ) , where σ is sigmoid, as is commonin the literature [4, 38] – here, and in “Spatial” of [46], the S matrix is interpreted as geographicspatial positions of the individuals.
For the m -th SNP of the n -th individual, we generate the SNP
X nm ∼ Bin(3 , σ ( F nm )) .
In this simulation, we considered M = 10 SNPs with N = 10000individuals.
Individuals are clustered into K = 3 groups based on their locations, and the individual’s clusteris the unobserved confounder.
Then, the outcomes are calculated from both the SNPs, where onlyc = 2 SNPs have a non-zero causal effect, plus a confounding term that is a function of the cluster,and some i.i.d.
Gaussian noise.
One small deviation from [46] is that our noise is i.i.d.
Gaussian,rather than having the variance depend on the confounder as well.
We further augmented the dataset to include a time-series proxy that can help identify the unobservedspatial position of individuals.
For each cluster, we came up with some Fourier coefficients,
Cluster 0: a (0)0 = 0 . 0 , a (0)1 = − 1 . 0 , a (0)2 = 1 . 0 , b (0)1 = − 1 . 0 , b (0)2 = 1 .Cluster 1: a (1)0 = 1 . 0 , a (1)1 = − 5 . 0 , a (1)2 = 2 . 0 , b (1)1 = − 5 . 0 , b (1)2 = 2 .Cluster 2: a (2)0 = − 1 . 0 , a (2)1 =
[Equation]
For each individual in the k -th cluster, we sampled at N samples =
50 uniformly spaced times,across N periods = 2 periods of length T = 5 .
That is, our time series proxy consists of points
{ x i } i ∈ [ N samples ] ,x i = 1 2 a ( k ) 0 + 2 (cid:88) ℓ = a ( k ) ℓ cos (cid:18) 2 πt i ( ℓ − 1)
T (cid:19) + 2 (cid:88) ℓ = b ( k ) ℓ sin (cid:18) 2 πt i ( ℓ − 1)
T (cid:19) , t i = TN periods
i N samples .
Please see Fig.
3 for a visualization of the time-series generated for each cluster.
F.2 VAE Modeling and evaluation setup
To handle the SNP data, we trained an MLP-based β -variational autoencoder [12] with ReLUactivations, which output parameters of a Gaussian latent distribution.
The MLPs consisted of 3hidden layers, each with 64 units, and our latent dimension z was 2.
To handle time series data (dataof another modality), we trained a β -VAE that instead encoded and decoded with 1D convolutions,as 1D convolutions have been shown to strongly approximate fast Fourier transforms [31].
Therewere 3 hidden convolutional layers, each of which had 32 output channels, with a kernel of length 3and a stride of 1.
Following the convolution was an MLP with 1 hidden layer with 64 units.
ReLUactivations were used throughout the architecture.
In our experiments, we swept over β ∈ { 0 . 1 , 0 . 2 , 0 .
Our general hypothesis wasthat lower β values would make our VAE perform better, as unlike in the standard VAE evaluationscheme, we sample from the posterior to generate latents for individuals, and not the prior, so the KLdivergence term in the ELBO doesn’t matter as much.
When evaluating on a set of SNPs and time series data corresponding to a set of individuals, wegenerate latents by passing the data of each respective type into the appropriate encoder and samplingfrom the resulting distributions.
The latent z for each individual is the concatenation of the SNPlatent z snp and the time series latent z ts .
Once we have the latent confounder z for an individual, we calculate the causal effect of the m th SNPby performing the linear regression,
[Equation]
where ϵ i are i.i.d.
standard normal.
We solved this using the closed form solution for least-squareslinear regression, given by
[Equation]
where A i =
[ z i , 1 , x m ] is the i th row in the matrix A .
Then (cid:98) γ m is our estimated causal effect for them th SNP.
We let (cid:98) γ be a vector of length M to denote our estimates for each SNP’s causal effect.
Comparison to CEVAE [26] [26] proposed the causal effect VAE (CEVAE) to estimate causaleffect with unobserved confounders, but their approach is specialized to the uni-modal, binarytreatment setting.
The CEVAE uses two separate encoders and decoders to infer the outcomes of ygiven both values of the treatment t .
It does this by first learning the treatment distribution p θ ( t | x ) ,followed by the outcome distribution p θ ( y | t, x ) for t ∈ { 0 , 1 } and then the latent distributionp ( z | x, t, y ) given a sampled y from the appropriate treatment distribution.
It is possible to extend
CEVAE to K different treatments, but this would require training K different encoders and decodersto learn p θ ( y | t
= k, x ) , ∀ k ∈ [ K ] , etc.
This linear scaling is not ideal, especially for GWAS, wherefor a SNPs vector of length M , there are 3 M possible SNP combinations; if we assume that each SNPis independent, each SNP can be interpreted as a ternary treatment, but CEVAE would nonethelessrequire fitting O
( M ) encoders, which is prohibitively expensive since M is typically large.
For the unimodal setting, we propose to simply encode the SNPs vector using a vanilla VAE, andterm our approach “SNP VAE”.
The main benefit of a SNP VAE over a causal effect VAE, then, isthat our VAE is scalable to treatments over a variety of different modalities and sizes.
Our proposedmethod does not encode and decode the treatment t for every individual x —it merely encodes fromx , assuming we have all the necessary information about the individual to infer possible confoundersfor them.
For the multi-modal setting, we handle the data of different modalities (image, text, time series) usingappropriate deep learning methods (for images, we will use convolutional neural networks, for text ortime series data, 1D convolutions or recurrent neural networks/transformers).
We note that while thenumber of SNPs can be very large, the number of different data modalities seen in GWAS is generallymuch smaller, allowing our model to scale appropriately without a very expensive computationalburden.
F.3 Results
We tested our VAE model and multi-modal VAE model against several linear latent model baselines.
Two baselines, Principle Components Analysis (PCA) and Factor Analysis (FA), also generate alatent z , which we then use our linear regression approach to calculate (cid:98) γ .
We also ran the LMMimplementation by Limix.
We also plot two oracle baselines.
The first is when (cid:98) γ
= γ ⋆ , labeled“truth”.
The second is when z is the true confounder used when generating the data, according to“Spatial”.
Let γ ⋆ denote the ground truth causal effect vector.
We report ∥ (cid:98) γ − γ ⋆ ∥ for ℓ 1 and ℓ 2 norms.
We also report true/false positive/negatives, which we define as follows.
For an individual, letτ =
min i : γ i ̸ =0 | γ i | / 2 .
Then, SNP m is a
True Positive (tp): if (cid:98)
γ m ≥ τ and γ ⋆m ≥ τ or (cid:98) γ m ≤ − τ and γ ⋆m ≤ − τ .
True Negative (tn): if | (cid:98) γ m | < τ and | γ ⋆m | < τ .
False Negative (fn): if | (cid:98) γ m | < τ but | γ ⋆m | > τ .
False Positive (fp): if none of the above hold.
Concretely
, there are two cases.
First, if
| (cid:98) γ m | > τ but | γ ⋆m | < τ .
Second, if the direction is wrong, i.e. (cid:98) γ m > τ but γ ⋆m < − τ , or
(cid:98) γ m < − τ
but γ ⋆m > τ .
Finally, recall that precision is tp/ ( tp + fp ) and recall is tp/ ( tp + fn ) , and higher is better for both.
In the following table, we show the mean and standard error of the mean (sem) over 10 seeds atevaluation time.
Here, we use a VAE to instantiate DSEs.
The unimodal VAE only encodes the SNPsvector into a 2 dimensional embedding.
The multimodal VAE also encodes the time-series vector intoa 2 dimensional embedding, and we concatenate both 2 -dimensional embeddings into a 4 dimensionalembedding to use in our linear regression.
PCA k denotes PCA with k components, and similarlyfor FA.
Amongst the non-oracle baselines, multi-modal VAE has the smallest errors and highestprecision and recall, and importantly beats all linear baselines including LMM.
Unimodal VAE, whileworse than LMM, still beats PCA and FA.
In general, we see that precision starts to deteriorate fasterthan recall, suggesting that false positives are more likely from the weaker linear deconfoundingtechniques such as PCA/FA.
It’s also interesting that when PCA/FA used more components, wesee worse performance across the board.
This illustrates the importance of picking the right latentdimension, which is an open question.
If the latent dimension is too small then we may not capture all the confounders.
If the latent dimension is too large, the latent encoding may also end up capturingspurious correlations.
[-(b) P ( x 1 ′ = 
-]
[-P ( x 1 ′ = 
-]
[-P ( x 2 = 
-]
[-b) P ( x 1 ′ = 
-]
[-P ( x 1 ′ = 
-]
[-b) P ( x 1 ′ = 
-]
[-P ( x 1 ′ = 
-]
[Table] [Table]
[-X 1 → X 
-]
[-P ( x 2 = 
-]
[-P ( x 3 ′ = 
-]
[-P ( x ′ i = 
-]