Weakly supervised causal representation learning
Abstract
Learning high-level causal representations together with a causal model from unstructured low-level data such as pixels is impossible from observational data alone.
We prove under mild assumptions that this representation is however identifiable in a weakly supervised setting.
This involves a dataset with paired samples before and after random, unknown interventions, but no further labels.
We then introduce implicit latent causal models, variational autoencoders that represent causal variables and causal structure without having to optimize an explicit discrete graph structure.
On simple image data, including a novel dataset of simulated robotic manipulation, we demonstrate that such models can reliably identify the causal structure and disentangle causal variables
Introduction
Interven’on 𝑥 "𝑥 𝑧 ̃𝑧
The dynamics of many systems can be described in terms of some high-level variables and causal relations between them.
Often, these causal variables are not known but only observed in some unstructured, low-level representation, such as the pixels of a camera feed.
Learning the causal representations together with the causal structure between them is a challenging problem and may be important for instance for applications in robotics and autonomous driving [1].
Without prior assumptions on the data-generating process or supervision, it is impossible to uniquely identify the causal variables and their causal structure [2, 3].
In this work, we show that a weak form of supervision is sufficient to identify both the causal representations and the structural causal model between them.
We consider a setting in which we have access to data pairs, representing the system before and after a randomly chosen, unknown intervention while preserving the noise.
This may approximate the generative process of data collected from a video feed of an external agent or demonstrator interacting with a system.
Neither labels on the intervention targets nor active control of the interventions are necessary for our identifiability theorem, making this setting useful for offline learning.
We prove that with this form of weak supervision, and under certain assumptions (including that the interventions are stochastic and perfect and that all interventions occur in the dataset), latent causal models (LCMs)— structural causal models (SCMs) together with a decoder from the causal factors to the data space— are identifiable up to a relabelling and elementwise reparameterizations of the causal variables.
We then discuss two practical methods for LCM inference.
First, we define explicit latent causal models (ELCMs) as a variational autoencoder (VAE) [4] in which the causal variables are the latent variables and the prior is based on an SCM.
While this approach works in simple problems, it can be finicky and is difficult to scale.
We trace this to a major challenge in causal representation learning, namely that it is a chicken-and-egg problem: it can be difficult to learn the causal variables when the causal graph is not yet learned, and it is difficult to learn the graph without knowing the variables.
To overcome this optimization difficulty, we introduce a second model class: implicit latent causal models (ILCMs).
These models can represent causal structure and variables without requiring an explicit, discrete graph representation, which makes gradient-based optimization easier.
Nevertheless, these models still contain the causal structure implicitly, and we discuss two algorithms that can extract it after the model is trained.
Finally, we demonstrate ILCMs on synthetic datasets, including the new
CausalCircuit dataset of a robot arm interacting with a causally connected system of light switches.
We show that these models can robustly learn the true causal variables and the causal structure from pixels.
Related work
Our work builds on the work of Locatello et al.
[5] on disentangled representation learning .
The authors introduce a similar weakly supervised setting where observations are collected before and after unknown interventions.
In contrast to our work, however, they focus on disentangled representations, i. e. (conditionally) independent factors of variation with a trivial causal graph, which our work subsumes as a special case.
Other relevant works on disentangled representation learning and (nonlinear) independent component analysis include Refs.
The problem of causal representation learning has been gaining attention lately, see the recent review by Schölkopf et al.
Lu et al.
[13] learn causal representations by observing similar causal models in different environments.
von Kügelgen et al.
[14] use the weakly supervised setting to study selfsupervised learning, using a known but non-trivial causal graph between content and style factors.
Lippe et al.
[15] learn causal representations from time-series data from labelled interventions, assuming that causal effects are not instantaneous but can be temporally resolved.
Yang et al.
[16] propose to train a VAE with an SCM prior, but require the true causal variables as labels.
Other relevant works include Refs.
To the best of our knowledge, our work is the first to provide identifiability guarantees for arbitrary, unknown causal graphs in this weakly supervised setting.
Identifiability of latent causal models from weak supervision
In this section, we show theoretically that causal variables and causal mechanisms are identifiable from weak supervision.
In Sec.
4 we will then demonstrate how we can learn causal models in practice by training a causally structured VAE.
3.1 Setup
We begin by defining latent causal models and the weakly supervised setting.
Here, we only provide informal definitions and assume familiarity with common concepts from causality as introduced for instance in Ref.
We provide a complete and precise treatment in Appendix A and discuss limitations of our setup and possible generalizations in Appendix B.
We describe the causal structure between latent variables as a Structural Causal Model (SCM).
An SCM C describes the relation between causal variables z 1 , . . .
, z n with domains Z i and noise variables ϵ 1 , . . .
, ϵ n with domains E i along a directed acyclic graph (DAG) G ( C ) .
Causal mechanisms f i : E i × (cid:81) j ∈ pa i Z j → Z i describe how the value of a causal variable is determined from the associated noise variables, as well as the values of its parents in the graph.
Finally, an SCM includes a probability measure for the noise variables.
𝑧 !
𝑧 " 𝑧 # ̃𝑧 !
̃𝑧 " ̃𝑧 # 𝑧 #$ 𝑧 "$ 𝑧 ! $ ̃𝑧 #$ ̃𝑧 "$ ̃𝑧 !
$ 𝜑 𝑥 %𝑥 ∼ 𝑝(𝑥, %𝑥) 𝜑 𝑔 𝑔′ 𝑔 𝑔 $
I n t e r v e n e
I n t e r v e n e
An SCM entails a unique solution s :
E → Z defined by successively applying the causal mechanisms.
We require the causal mechanisms to be pointwise diffeomorphic, that is, for any value of the parents z pa i we have that f i ( · ;
z pa i ) is invertible, differentiable, and its inverse is differentiable.
Then s is also diffeomorphic and thus noise variables can be uniquely inferred from causal variables.
This simplifies the weakly supervised distribution, as the only stochasticity comes from the noise variables and the intervention.
The SCM also entails an observational p C ( z ) (Markov with respect to the graph of the SCM), which is the pushforward of p through the solution.
A perfect, stochastic intervention
( I, ( ˜ f i ) i ∈ I )
modifies an SCM by replacing for a subset of the causal variables, called the intervention target set I ⊂ { 1 , ..., n } , the causal mechanism f
i with a new mechanism ˜ f
i : E i → Z i , which does not depend on the parents.
The intervened SCM has a new solution ˜
s
I : E → Z .
We call interventions atomic if the number of targeted variables is one or zero.
We will reason about generative models in a data space X , in which the causal structure is latent.
Also including a distribution of interventions (as in Ref.
[23]), we define LCMs:
Definition 1 (Latent causal model (LCM)) .
A latent causal model M = ⟨C , X , g, I , p I ⟩ consists of
• an acyclic SCM C , which is faithful (all independencies are encoded in its graph [24]), • an observation space X , • a decoder g : Z → X that is diffeomorphic onto its image, • a set I of interventions on C , and • a probability measure p I over I .
We define two LCMs as equivalent if all of their components are equal up to a permutation of the causal variables and elementwise diffeomorphic reparameterizations of each variable, see Fig. 2.
Definition 2 (LCM isomorphism (informal)) .
Let M = ⟨C , X , g, I , p I ⟩ and M ′ = ⟨C ′ , X , g ′ , I ′ , p ′I ′ ⟩ be two LCMs with identical observation space.
An LCM isomorphism between them is a graph isomorphism ψ : G ( C ) → G ( C ′ ) together with elementwise diffeomorphisms for noise and causal variables that tell us how to reparameterize them, such that the structure functions, noise distributions, decoder, intervention set, and intervention distribution of M ′ are compatible with the corresponding elements of M reparameterized through the graph isomorphism and elementwise diffeomorphisms.
M and M ′ are equivalent, M ∼ M ′ , if and only if there is an LCM isomorphism between them.
Following Locatello et al.
[5], we define a generative process of pre- and post-interventional data:
Definition 3 (Weakly supervised generative process) .
Consider an LCM M where the underlying
SCM has continuous noise spaces E i , independent probabilities p E i and admits a solution s .
We define the weakly supervised generative process of data pairs ( x, ˜ x ) ∼ p XM ( x, ˜ x ) as follows: ϵ ∼ p E , z
= s ( ϵ ) , x = g ( z ) ,
I ∼ p
I , ∀ i ∈ I ,
˜ ϵ i ∼ p ˜
E
i , ∀ i ̸∈ I , ˜ ϵ
i =
ϵ i , ˜ z
= ˜ s I
(˜ ϵ ) ,
˜ x
= g (˜ z ) .
3.2 Identifiability result
The main theoretical result of this paper is that an LCM M can be identified from p ( x, ˜ x ) up to a relabeling and elementwise transformations of the causal variables:
Theorem 1 (Identifiability of R -valued LCMs from weak supervision) .
Let M = ⟨C , X , g, I , p I ⟩ and M ′ = ⟨C ′ , X , g ′ , I ′ , p ′I ′ ⟩ be LCMs with the following properties:
• The LCMs have an identical observation space X .
• The SCMs C and C ′ both consist of n real-valued endogeneous causal variables and corresponding exogenous noise variables, i. e. E i
= Z i =
Z ′ i
=
E ′ i
= R .
• The intervention sets I and I ′
consist of all atomic, perfect interventions, I = {∅ , { z 0 } , . .
, { z n }} and similar for I ′ .
• The intervention distribution p I and p ′I ′ have full support.
Then the following two statements are equivalent:
The LCMs entail equal weakly supervised distributions, p XM ( x, ˜ x ) =
p XM ′
( x, ˜ x ) .
The LCMs are equivalent in the sense of Def. 2, M ∼ M ′ .
Let us summarize the key steps of our proof, which we provide in its entirety in Sec. A in the supplementary material.
The direction 2 ⇒ 1 follows from the definition of equivalence.
The direction 1 ⇒ 2 is proven constructively along the following steps:
We begin by defining a diffeomorphism φ = g ′− 1 ◦ g : Z → Z ′ and note that if z, ˜
z ∼ p ZC ( z , ˜ z ) , the weakly supervised distribution of causal variables of model C , then φ ( z ) , φ (˜ z ) ∼ p Z ′ C ′
( z ′ , ˜ z ′ ) .
The distribution over z, ˜ z is a mixture, where each intervention target I gives a mixture component; each component is supported on a different ( n + 1) - dimensional submanifold.
Therefore, there exists a bijection between the components ψ : [ n ] → [ n ] that maps intervention targets I in M to intervention targets I ′ =
ψ ( I ) in
M ′ .
Furthermore, because the joint distribution z, ˜ z is preserved by φ , first mapping with φ , then intervening, Z φ −→ Z
′ I ′ −→ (cid:101)
Z ′ , equals Z I −→ (cid:101)
Z φ −→ (cid:101) Z ′ .
Because I = { i } is a perfect intervention, for the map Z φ −→ Z
′ I ′ −→ (cid:101) Z ′ ,
˜ z ′ i ′ is independent of z ′ .
Thus, in both maps, ˜ z ′ i ′ is independent of z .
This means that for the path through (cid:101) Z , the intervention sample ˜ z
i is transformed into ˜ z
′ i ′
independently of z .
For R -valued variables, this statistical independence implies that the transformation is constant in z , and thus φ ( z ) i ′ is constant in z j for j ̸
= i .
φ is therefore an elementwise reparametrization.
Using this, we can show that ψ is a causal graph isomorphism and that it is compatible with the causal mechanisms.
This proves LCM equivalence M ∼ M ′ .
Practical latent causal models
Theorem 1 means that it is possible to learn causal structure from pixel-level data in the weakly supervised setting.
Consider a system that is described by an unknown true LCM and assume that we have access to data pairs ( x, ˜ x ) sampled from its probability density.
Then we can train another
LCM with learnable components by maximum likelihood.
Assuming sufficient data and perfect optimization, this model’s density will match that of the ground-truth LCM.
Our identifiability result guarantees that the trained LCM then has the same causal variables and causal structure as the ground truth, up to relabelling.
In the following, we describe two neural LCM implementations that can be trained on data.
[Figure]
4.1 Explicit latent causal models (ELCMs)
To implement LCMs with neural networks, we use the variational autoencoder (VAE) framework [4].
We first consider an approach where the causal variables ( z, ˜ z ) are the latent variables.
Data ( x, ˜ x ) and latents ( z, ˜ z ) are linked by a stochastic encoder q
( z | x ) and decoder p ( x | z ) ; unlike the deterministic decoder from Sec.
this allows us to map high-dimensional data spaces (like images) to low-dimensional causal variables.
The causal structure is encoded in the prior p ( z, ˜ z ) and consists of a learnable causal graph and learnable causal mechanisms f i .
The first contribution to the prior density is the observational probability density p ( z ) , which factorizes according to the causal graph into components p ( z i | z pa i ) , which are given by fixed base densities and the causal mechanisms.
The second contribution is the interventional conditional density p (˜ z | z ) , which is also computable from the graph and causal mechanisms.
The model can be trained on the ELBO loss, a variational bound on − log p ( x, ˜ x ) .
For more details, see Appendix E.
We call this an explicit latent causal model (ELCM), as the model directly parameterizes all components of an LCM.
In particular, ELCMs contain an explicit representation of the causal graph and causal mechanisms.
The graph can be learned by an exhaustive search over all DAGs or through a differentiable DAG parameterization [26–29] and gradient descent.
In our experiments with ELCMs, which we describe in Appendix E, we find that optimally trained
ELCMs indeed correctly identify the causal structure and disentangle causal variables on simple datasets.
However, jointly learning explicit graph and variable representations presents a challenging optimization problem.
In particular, we observe that the loss landscape has local minima corresponding to wrong graph configurations.
4.2 Implicit latent causal models (ILCMs)
To enable causal representation learning in a more robust, scalable way, we propose a second LCM implementation: Implicit Latent Causal Models (ILCMs).
Like ELCMs, ILCMs are also variational autoencoders with a causally structured prior.
The key difference is that ILCMs represent the causal structure through neural solution functions s ( e ) .
Under our assumption of diffeomorphic causal mechanisms, the solution function—which maps the vector of noise variables to the causal variables— contains the same information as the causal graph and causal mechanisms that we parameterize with neural networks in ELCMs (see Appendix C).
However, unlike ELCMs, this parameterization does not require an explicit graph parameterization.
In practice, ILCMs are thus easier to train than ELCMs.
Latents The latent variables in an ILCMs are noise encodings , defined through the inverse solution function as e =
s − 1 ( z ) and ˜
e = s − 1 (˜ z ) .
The pre-intervention noise encoding e is identical to the SCM noise variables.
The post-intervention noise encoding ˜
e corresponds to the value of the
SCM noise variables that would have generated the post-intervention causal variables ˜
z under the unintervened SCM mechanisms.
ILCMs contain a stochastic encoder q
( e | x ) and decoder p ( x | e ) that map data ( x, ˜ x ) to noise encodings ( e, ˜ e ) .
Noise encodings have the convenient property that under an intervention with intervention targets I , precisely the components e I change value: e i ̸ =
˜
e i ⇔ i ∈ I with probability 1.
We prove this property in Appendix A.
This means that from noise encodings e, ˜ e , we can infer interventions easily.
We use a simple heuristic intervention encoder that assigns higher intervention probability q
( i ∈ I | x, ˜ x ) to a component
i the more this component of the noise encoding changes under interventions:
[Equation]
where µ e ( x ) is the mean function of the noise encoder q ( e | x ) and h is a quadratic function with learnable parameters.
Both the equality pattern of e under interventions and this heuristic intervention encoder are similar to the ones used for disentangled representation learning in Ref.
Prior Given encoders for noise encodings and intervention targets, let us now write down the prior p ( e, ˜
e, I ) , which encodes the structure of the weakly supervised setting.
The intervention-target prior p ( I ) and the pre-intervention noise distribution p ( e ) are given by simple base densities, which we choose as uniform categorical and standard Gaussian, respectively.
The post-intervention noise encodings ˜
e follow the conditional probability distribution p
[Equation]
(cid:12) (cid:12) (cid:12)(cid:12) ,
[Equation]
=
[Equation]
i (
[Equation]
i
[Equation]
e \ i ) .
In the second equality we have parameterized the conditional density p (˜ e i | e ) with a conditional normalizing flow consisting of a learnable diffeomorphic transformation ˜
e i (cid:55)→ ¯ z i =
¯ s i (˜ e i ; e \ i ) and a base density ˜ p on ¯ z i , which we choose as standard Gaussian.
How does this prior encode causal structure?
We rely on three key properties of SCMs, shown in
Appendix A: 1) the noise variables e i are independent of each other; 2) upon intervening on variable i , the post-intervention causal variable ˜
z i are independent of all e j ; 3) while for the other variables j ̸
= i , the noise encodings are unchanged ˜
e j = e j .
These three properties are ensured in the ILCM prior in Eq.
We show in Appendix C that therefore each ILCM is equivalent to a unique ELCM.
For each variable i , the ILCM function ¯
s i is equal to the solution function s i of the equivalent ELCM, which maps from noise variables to causal variable z i .
Thus, by learning to transform ˜ e i into ˜ z
i = ¯ s (˜ e i ; e ) in the ILCM, we learn the solution function of the corresponding ELCM.
This implicitly describes both the causal graph and the causal mechanisms f i .
We can thus learn a causal model without ever explicitly modelling a graph.
The final question is how to implement the first terms in Eq.
(3), which encode that those noise encodings that are not part of the intervention targets I should not change value under the intervention.
We enforce this in the encoder by setting the non-intervention components of e and ˜
e to the same value [similar to 5].
In Appendix C this procedure is described in more detail.
We will refer to this projective noise encoder as q ( e, ˜
e | x, ˜ x, I ) .
Learning Putting everything together, an ILCM consists of an intervention encoder q
( I | x, ˜ x ) , a noise encoder q ( e, ˜ e | x, ˜ x, I ) , a noise decoder p ( x | e ) , and transformations / solution functions s i ( · ; e ) , see Fig.
All of these components are implemented with neural networks and learnable, see
Appendix C for details.
The lower bound on the joint log likelihood of pre-intervention and postintervention data is given by
[Equation]
The model is trained by minimizing the corresponding VAE loss, learning to map low-level data to noise variables (with q ) and to map noise variables to causal variables (with s ).
The expectation over
I is computed via summation, but could alternatively be done with sampling.
In practice, we find it beneficial to add a regularization term to the loss that disincentivizes collapse to a lower-dimensional submanifold of the latent space.
For each batch of training data, we compute the batch-aggregate intervention posterior
q
I
I ) =
E x, ˜ x ∈ batch [ q ( I | x, ˜ x )] .
To the beta-VAE loss we then add the negative entropy of this distribution, weighted with a hyperparameter.
Downstream tasks Despite the implicit representation of causal structure, we argue that ILCMs let us solve various tasks:
• Causal representation learning / disentanglement : ILCMs allow us to map low-level data x to causal variables z by applying the encoder q followed by the solution functions s .
• Intervention inference : It is also straightforward to infer intervention targets from an observed pair ( x, ˜ x ) of pre-intervention and post-intervention data, as this just requires evaluating the intervention-target encoder
q I ( x, ˜ x ) .
• Causal discovery / identification : We propose two methods to infer the causal graphs after training an ILCM.
One is to use an off-the-shelf method for causal discovery on the learned representations.
Since the ILCM allows us to infer intervention targets, we can use intervention-based algorithms.
In this paper, we use ENCO [28], a recent differentiable causal discovery method that exploits interventions to obtain acyclic graphs without requiring constrained optimization.
Alternatives to ENCO include DCDI [27] and GIES [30].
Alternatively, we can analyze the causal structure implicitly represented in the learned solution functions s i .
We propose a heuristic algorithm that proceeds in three steps.
First, it infers the topological order by sorting variables such that
s i only depends on e j if z i is after z j in the topological order.
It then iteratively rewrites the solution functions such that they only depend on ancestors in the topological order.
Finally, it determines which causal ancestors are direct parents by testing the functional dependence of the causal mechanisms.
We describe this algorithm in more detail in Appendix C. • Generation of interventions and counterfactuals : The ILCM entails a generative model for pairs of pre- and post-intervention data.
It is straightforward to sample from the joint distribution p ( x, ˜ x, I ) , from the conditional p ( x, ˜ x | I ) , or from the conditional p (˜ x | x, I ) .
Experiments
Finally, we demonstrate latent causal models in practice.
Here we focus on implicit LCMs; explicit
LCMs are demonstrated in similar experiments in Appendix E.
We evaluate the causal graphs learned by the ILCM models either with ENCO (ILCM-E) or with the heuristic algorithm described above (ILCM-H).
Baselines Since we are to the best of our knowledge the first to study causal representation learning in this weakly supervised setting, we are not aware of any baseline methods designed for this task.
We nevertheless compare ILCMs to three other methods.
First, we define a disentanglement VAE that models the weakly supervised process, but assumes independent factors of variation rather than a non-trivial causal structure between the variables.
This baseline is similar to the method proposed by
Ref.
[5], but it differs in some implementation details to be more comparable to our ILCM setup.
We infer the causal graph between the learned representations with ENCO (dVAE-E).
We also compare to an unstructured β -VAE that treats x and ˜ x as i. i. d. and uses a standard Gaussian prior.
Finally, for the pixel-level data, we consider a slot attention model [31], which segments the image unsupervisedly into as many objects as there are causal variables.
The latent representation associated to each object is considered a learned causal variable.
5.1 2D toy experiment
We first demonstrate LCMs in a pedagogical toy experiment with X =
Z = R 2 .
Training data is generated from a nonlinear SCM with the graph z 1 → z 2 and mapped to the data space through a randomly initialized normalizing flow.
An ILCM trained in the weakly supervised setting is able to reconstruct the causal factors accurately up
[Figure] [Figure]
Data
[Figure]
[Figure] Learned factors to elementwise reparameterizations, as shown in Fig.
In Tbl.
1 we quantify the quality of the learned representations with the DCI disentanglement score [32].
We find that our LCM is able to disentangle the causal factors almost perfectly, while the baselines, which assume independent factors of variation, fail as expected.
Both the ILCM and the dVAE baseline infer the intervention targets with high accuracy.
Finally, we test the quality of the learned causal graphs.
We infer the implicit graph with
ENCO and the heuristic algorithm discussed above.
In both cases, the learned causal graph is identical to the correct one, whereas the representations found by the dVAE baseline induce a wrong graph.
5.2 Causal3DIdent
We then turn to pixel-level data and more complex causal graphs.
We test ILCMs on an adaptation of the
Causal3DIdent dataset [14], which contains images of three-dimensional objects under variable positions and lighting conditions.
We consider three causal variables representing object hue, the spotlight hue, and the position of the spotlight.
We construct six versions of this dataset, each with a different causal graph, randomly initialized nonlinear structure functions, and heteroskedastic noise.
These are mapped to images with a resolution of 64 × 64 , see Fig.
5 for examples.
ILCMs are again able to disentangle the causal variables reliably.
The results in Tbl.
1 show that the learned representations are more disentangled than those learned by methods that do not account for causal structure.
The
LCM as well as the dVAE baseline can infer interventions with almost perfect accuracy.
We demonstrate this in Fig.
5 by comparing true and inferred interventions, see Sec. D.3 of the supplementary material for details.
The ILCMs also learn the causal graphs accurately, while the acausal dVAE-E baseline does in most cases not find the correct causal graphs.
5.3 CausalCircuit
While Causal3DIdent provides a good test of the ability to disentangle features that materialize in pixel space in different ways, like through the position of lights and the color of objects, the underlying causal structure we imposed may feel rather ad-hoc.
To explore causal representation learning in a more intuitively causal setting, we introduce a new dataset, which we call CausalCircuit.
[Figure]
The CausalCircuit system consists of a robot arm that can interact with multiple touch-sensitive lights.
The lights are connected with a stochastic circuit: a light is more likely to be on if its button is pressed or if its parent lights are on.
The robot arm itself can be seen as part of the causal system.
Concretely, we consider the causal graph shown in
Fig.
This system is observed from a fixed-position camera, and we generate samples in 512 × 512 × 3 resolution with MuJoCo
[33], see
Sec. D.4 of the supplementary material for more details.
ILCMs are again able to disentangle the causal variables reliably and better than the acausal baselines, see Tbl.
As shown in Appendix D, the slot attention model fails because the lights have no limited spatial extent and thus are not well represented by segments of the image.
Interventions are identified with high accuracy.
ILCMs also correctly learn the causal graph shown in Fig.
[-=
-]
6, both when extracted with ENCO and with our heuristic algorithm.
In Fig.
6 we demonstrate how ILCMs let us infer and manipulate causal factors and reason about interventions.
By studying variations of this dataset, we tested the limitations of our method.
We find that it works reliably only as long as the causal variables are continuous (that is, when we model the lights with a continuous intensity).
As soon as we consider discrete states, the assumptions of our identifiability theorem are violated and the model has difficulty disentangling these variables.
5.4 Scaling with graph size
[Figure]
I
[-ϵ 
-]
[Figure]
i s e n t a n g l e m e n t s c o r e
[Figure]
Finally, we study how LCMs scale with the size of the causal system.
We generate simple synthetic datasets with X
=
Z = R n .
For each dimension n , we generate three datasets, using linear SCMs with random DAGs, in which each edge in a fixed topological order is sampled from a Bernoulli distribution with probability 0.5.
The causal variables are mapped to the data space through a randomly sampled SO ( n ) rotation.
We find that ILCMs are able to reliably disentangle the causal variables in systems with up to approximately 10 causal variables, see Fig. 8.
In this regime, the true causal graphs are also identified with good accuracy, see Sec.
D.5 of the supplementary material.
In larger causal systems, both disentanglement and graph accuracy become worse; more work is required to improve the scaling of our approach to causal representation learning.
Discussion
[Table]
What makes a variable causal?
One school of thought is that it that causal variables are those aspects of a system that can be intervened upon [34].
Following this logic
[Table]
we find it interesting to ask: can we uniquely determine the causal variables underlying a system just by observing the effect of interventions?
In this work we have found a partial answer to this question: we have shown in theory and practice that under certain assumptions, causal variables and their causal structure are identifiable from low-level representations like the pixels of a camera feed if the system is observed before and after random, unlabeled interventions.
Our identifiability theorem extends the results by Locatello et al.
[5] from independent factors of variation (trivial causal graphs) to arbitrary causal graphs.
Latent causal structure can be described in a variational autoencoder setup.
However, a straightforward, explicit parameterization of the causal structure requires simultaneously learning the variables and the causal graph.
We found that leads to challenging optimization problems, especially when scaling to larger systems. As a more robust alternative, we introduced implicit latent causal models (ILCMs), which parameterize causal structure without requiring an explicit graph representation.
We also discussed two algorithms for extracting the learned causal mechanisms and graph after training.
In first experiments, we demonstrated that ILCMs let us reliably disentangle causal factors, identify causal graphs, and infer interventions from unstructured pixel data.
For these experiments, we introduced the new CausalCircuit dataset, which consists of images of a robot arm interacting with connected switches and lights.
The setting we consider is motivated by a potentially useful scenario: learning causal structure from passive observations of an agent (or demonstrator) interacting with a causal system.
However, it is currently far from practical.
Our identifiability result relies on a number of assumptions, including that interventions are stochastic and perfect, that all atomic interventions may be observed, and that the causal variables are real-valued.
In addition, realistic temporal sequence data are not likely to exactly correspond to a causal system before and after an intervention (while preserving the noise variables); whether our causal abstraction provides a useful approximation remains to be tested.
We discuss these requirements and their potential relaxation in Appendix B. Similarly, our practical implementation has so far been restricted to simplified datasets with relatively few, continuous causal variables, and when trying to relax these limitations we saw the model performance decrease quickly.
While more work will be required to make latent causal models applicable to real-world settings, we believe that our results demonstrate that causal representation learning is possible without explicit labels.
Acknowledgments We want to thank Joey Bose, Thomas Kipf, Dominik Neuenfeld, and Frank Rösler for useful discussions and Gabriele Cesa, Yang Yang, and Yunfan Zhang for helping with our experiments.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes] We strive to be transparent about the limitations of our work.
In the theory section we list the assumptions, in the experiment section we discuss observed failures, and in the conclusions and in Appendix B we discuss the requirements of our approach again.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix F. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We provide our list of assumptions in Sec.
5 and Appendix A and further discuss them in Appendix B. (b) Did you include complete proofs of all theoretical results?
[Yes] Our identifiability theorem is proven in Appendix A, 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Not yet, but we aim to publish them as soon as we obtain approval to do so.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We provide these details in Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use the Causal3DIdent dataset from von Kügelgen et al.
(b) Did you mention the license of the assets?
[No] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] We strive to publish our CausalCircuit dataset as soon as possible.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We only consider synthetic data showing simulated objects.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]