Interventional Causal Representation Learning
Abstract
The theory of identifiable representation learning aims to build general-purpose methods that extract high-level latent (causal) factors from low-level sensory data.
Most existing works focus on identifiable representation learning with observational data, relying on distributional assumptions on latent (causal) factors.
However, in practice, we often also have access to interventional data for representation learning.
How can we leverage interventional data to help identify high-level latents?
To this end, we explore the role of interventional data for identifiable representation learning in this work.
We study the identifiability of latent causal factors with and without interventional data, under minimal distributional assumptions on the latents.
We prove that, if the true latent variables map to the observed high-dimensional data via a polynomial function, then representation learning via minimizing the standard reconstruction loss of autoencoders identifies the true latents up to affine transformation.
If we further have access to interventional data generated by hard do interventions on some of the latents, then we can identify these intervened latents up to permutation, shift and scaling.
Introduction
Modern deep learning models like GPT-3 (Brown et al., 2020) and CLIP (Radford et al., 2021) are remarkable representation learners (Bengio et al., 2013).
Despite the successes, these models continue to be far from the human ability to adapt to new situations (distribution shifts) or carry out new tasks (Geirhos et al., 2020; Bommasani et al., 2021).
Humans encapsulate the causal knowledge of the world in a way that is highly reusable and recomposable (Goyal and Bengio, 2020), which helps them adapt to new tasks in an ever-distribution-shifting world.
How to make modern deep learning models extract a similar causal understanding of the world?
This question is central to the emerging field of causal representation learning (Schölkopf et al., 2021).
A core task in causal representation learning is provable representation identification , i e. developing conditions under which representation learning algorithms can provably identify latent objects (or factors) and their causal relationships.
Towards understanding this task, several notable works have shown that provable representation identification for arbitrary data generation process (DGP) is impossible if we only enforce the independence between the latent factors (Hyvärinen and Pajunen, 1999; Locatello et al., 2019).
Yet, real data generation processes often have additional structures we can leverage to achieve provable representation identification.
Such structures include the independence between the latent factors conditional on auxiliary information (Khemakhem et al., 2020a), the sparsity of the causal connections among the latents (Lachapelle et al., 2022), and the sparsity of the mechanisms that govern the variation of the latents (Locatello et al., 2020; Ahuja et al., 2022a; Klindt et al., 2020).
Despite these efforts toward provable representation identification, most existing works focus on representation learning with observational data, relying on distributional assumptions on latent (causal) factors to achieve identification.
However, in practice, we often also have access to interventional data for representation learning.
For example, for representation learning in robotics, we often have access to interventional data from robotic manipulation experimentss (Collins et al., 2019); for genomics and neuroscience, we also often have access to interventional data from genetic perturbation experiments (Dixit et al., 2016) and from electrical stimulation experiments (Nejatbakhsh et al., 2021) respectively.
In this work, we seek to understand how we can leverage such interventional data to identify high-level (causal) factors from low-level data.
The key findings are summarized below.
• Under the assumption that the true latent factors map to the high dimensional observations via a finite degree multivariate polynomial, we first show that it is possible to achieve affine identification with respect to the true latents with minimal assumptions on the support of the true latents and no further distributional assumptions.
• If we also observe data where some latents undergo a hard do intervention (Pearl, 2009), then we can guarantee affine identification up to the block of these hard intervened latents.
As a result, if only one latent variable undergoes a do intervention in some environments, then those latents are identified up to permutation, shift, and scaling.
Representation Identification with Observational and Interventional Data
We begin with setting up the representation learning problem.
We then present a suite of identifiability results that explore the role of interventional data in achieving representation identification.
The data generating process.
We consider a data generating process where the observations x ∈ X ⊆ R n are generated from some underlying latent variables z ∈ Z ⊆ R d , with n ≥ d .
This data generating process follows
[Equation]
where P Z is the distribution from which the latent z is sampled, and x is the observed data point rendered from the underlying latent z using an injective decoder g : R d → R n .
We denote Z as the support of P Z ; as a consequence, the support of the observations x is X
= g ( Z ) .
The identifiable representation learning task.
To perform representation learning, we aim to find an encoder—also known as the representation function—that can help us estimate the underlying true latent variables z .
Specifically, the goal is to find an encoder f : R n → R d and a decoder h : R d → R n such that the encoder and decoder jointly satisfy the following reconstruction identity,[-
-]
[Equation]
Given the learned encoder f , the resulting representation is ˆ z ≜ f ( x ) , which holds the value of the latents that the encoder guesses.
Note that Equation 2 is highly underspecified and cannot in general identify the latents: it can have many solutions such that the resulting representation ˆ z do not coincide with the true latents z .
For example, if we take any solution f , h of the reconstruction identity, then b ◦ f , h ◦ b − 1 is another valid solution where b is an invertible map.
However, in practice, exact identification of the latents is often neither necessary nor reasonable.
For example, we may not care about the labels given to each latent, i.e., about the coordinate permutations of z .
Thus in this work, we study conditions under which the true latents are identifiable up to certain transformations, e.g. affine transformations, coordinate permutations, etc.
Overview of results.
Below we present a suite of identifiability results that explore the role of interventional data for representation identification.
We first show that, when the true decoder g is a polynomial function of z with a known degree and the learned decoder h is also a polynomial with the same degree, then the latent z can be identified from observational data up to affine transformations using the encoder learned from the reconstruction identity (Equation 2).
We then extend this result to settings where we do not know the exact degree of g but only its upper bound.
We also provide approximate affine identification guarantees for the setting when the true decoder g can be ϵ -approximated by a polynomial function.
Next, we study the identifiability of the latent z when additional interventional data is available.
We prove that, if we observe data where some latents undergo a hard do intervention (Pearl, 2009), then we can achieve affine identification up to the block of the latents that underwent the hard do interventions.
As a result, if only one latent variable underwent a hard do intervention in some environments, then that latent variable is identifiable up to shift and scaling.
2.1 Affine representation identification with observational data
We first establish an affine identification result for representation learning from observational data, relying on the decoders being multivariate polynomial functions.
We begin with the few assumptions required.
Assumption 1.
The interior of the support of Z (denoted as Z ) is a non-empty subset of R d .[-
-]
Assumption 2.
The decoder g is a polynomial of degree p whose corresponding coefficient matrix
G (a.k.a.
the weight matrix) has full column rank.
Specifically, the decoder g is determined by the coefficient matrix G as follows,
[Equation]
where ¯ ⊗ represents the Kronecker product with all distinct entries; for example, if z = [ z 1 , z 2 ] , then z ¯ ⊗ z =
[ z 21 , z 1 z 2 , z 22 ] .
The assumption that the matrix G ∈ R n × q has a full column rank of q implies that the decoder g is guaranteed to be injective; see the appendix for a proof of this claim.
This injectivity condition on g is common in identifiable representation learning since otherwise the problem of identification will become ill-defined: multiple latents can give rise to the same observation x .
We note that the full-column-rank condition for G in Assumption 2 imposes an implicit constraint on the dimensionality n of the data; it requires that the dimensionality n is greater than the number of terms in the polynomial of degree p , namely n = O ( d p ) , where d is the dimensionality of z .
Assumption 3.
The learned decoder h is a polynomial of degree p whose corresponding coefficient matrix H has full column rank.
Similar to the decoder g , the encoder h is determined by the coefficient matrix H as follows,
[Equation]
where ¯ ⊗ represents the Kronecker product with all distinct entries.
Under Assumptions 1 to 3, we show that the representation ˆ z resulting from solving the reconstruction identity must identify the true latents z up to affine transformations.
Specifically, by leveraging the relationship x = g ( z ) in Equation 1, we write the representation ˆ z = f ( x ) as ˆ z = f ◦ g
( z ) =
a ( z ) with a ≜ f ◦ g .
We then show that the a function must be an affine transformation.
Theorem 1.
If the data generation follows Equation 1 and Assumptions 1 to 3 hold, then any encoder f and decoder h that solve the reconstruction identity (Equation 2) must achieve affine identification: ∀ z ∈ Z , we must have ˆ z =
Az + c , where ˆ z = f ( x ) is the output of the encoder and z is the true latent.
Moreover, A is an invertible d × d matrix and c ∈ R d .
The proof is in the Appendix.
To understand its intuition, we consider one-dimensional latent z , three-dimensional observation x , and the true decoder g and the learned decoder h each being a degree-two polynomial.
We first solve the reconstruction identity on all x , which gives h (ˆ z ) =
g ( z ) , and equivalently H [1 , ˆ z, ˆ z 2 ] ⊤ =
G [1 , z, z 2 ] ⊤ .
It implies that ˆ z is at most a degree-two polynomial of z because both h and g are degree-two polynomials with full-column-rank coefficient matrices.
It also implies that ˆ z 2 is also a polynomial of at most degree two in z .
We next argue that ˆ z must be a degree-one polynomial of z by contradiction.
If ˆ z is a degree-two polynomial of z , then ˆ z 2 is degree four, which contradicts the fact that ˆ z 2 is at most degree two in z .
Therefore, ˆ z is a degree-one polynomial in z .
[Figure]
Extensions of Theorem 1 to polynomials with an unknown degree p .
Theorem 1 requires that the degree p is known.
We extend it to settings where p is unknown but an upper bound on the value p is known, which we denote as s .
(It implicitly requires that the dimensionality n of the data must be sufficiently large, i.e., n
= O ( d s ) , since the coefficient matrix of H must have full column rank.)
To achieve affine identification in this setting, we perform an iterative procedure.
The learner first tries to solve the reconstruction identity with a polynomial h ( · ) of degree equal to the upper bound s .
If s > p , then we can show that there exists no solution to the reconstruction identity (Equation 2); see Appendix for the detailed justification.
The learner then decreases the degree and searches over all full-rank polynomials h ( · ) of degree s − 1 .
She repeats this procedure until the degree is p , which is when a solution to the reconstruction identity exists.
Extensions of Theorem 1 beyond polynomial decoders.
While Theorem 1 assumes polynomial decoders, we know that, from Stone-Weiresstrass theorem (Rudin et al., 1976), a continuous map on a closed and bounded set can always be approximated with a high dimensional polynomial.
Inspired by this result, we extend our results to a class of maps g ( · ) that are ϵ -approximable by a polynomial of sufficiently high degree in the Appendix.
We show that the map a ( · ) that connects ˆ z to true z must be an approximately linear map, namely the polynomial expansion of the map a ( · ) must have sufficiently small weights (in terms of their norm) on the higher-order (degree greater than equal to two) terms.
2.2 Representation identification with interventional data
In the previous section, we considered representation identification from some observational data where the data is generated from Equation 1. The latent variable Z is drawn from an arbitrary distribution P Z , which does not necessarily come from any structural causal model.
We next study how interventional data could enhance representation identification.
We consider the general case where at least one component of Z , say the i th component, is set to a fixed value, and the remaining components are sampled from a distribution Q Z − i . Note Q Z − i does not have to be equal to the distribution P Z −
i | z i = z ∗ over samples Z − i generated from Equation 1 when z i = z ∗ . The
[Figure]
data-generating process is written as follows
[Equation]
where the variable z i is fixed to be equal to z ∗ ; the remaining d − 1 variables in z (denoted as z − i ) are sampled from Q Z − i ; and the function g is the true decoder that generates observed x from z . The results we present below only require a restriction on the support of Q Z − i and this flexibility allows
Q Z − i to model standard do interventions (Pearl, 2009), i.e., Q Z − i = P Z − i | do ( z i = z ∗ ) as we illustrate below. Our DGP also allows the possibility that Q Z
[Equation] Notation-wise, we use D 0 to denote the observational data generated from Equation 1, and use D i to denote the data generated from Equation 5 when i th latent variable is fixed to z ∗ .
We denote the support of Z for DGP in Equation 5 as Z i . We denote the support of the latents other than z i , i.e., the support of Q Z − i , as ˜ Z i .
We also denote the corresponding support of x as X i , where X i =
g
( Z i ) .
Given the observational data ( D 0 ) and the data from equation (5) ( D i ), we perform representation learning via the reconstruction identity as follows,
[Equation]
We further need to enforce the constraint on the encoder such that, for all the data points x ∈ X
i , the intervened component (say the k th component) must take some fixed value z † due to the intervention:
[Equation]
where f k ( x ) denotes the k th component of f ( x ) and is required to take some fixed value z † for all x ∈ X i .
Illustrating interventions using a structural causal model for Z .
For this example only, we make an additional assumption that Z is drawn from a structural causal model, and consider the setting where we have access to both observational and interventional data.
In the interventional data, we assume exactly one latent undergoes a hard do intervention as in Equation 5.
In performing representation learning with the reconstruction identity, we further enforce the constraint (Equation 7) that exactly one component of the encoder also takes a fixed value.
We illustrate this setting with observational and interventional data using a simple example.
Suppose Z =
[ Z 1 , Z 2 , Z 3 , Z 4 ] is drawn from a structural causal model with the underlying directed acyclic graph (DAG) displayed in Figure 1a.
Figure 1b (resp.
Figure 1c) shows the DAG when Z 3 (resp.
Z 2 ) undergoes a hard intervention and is set to z ∗ 3 (resp.
z ∗ 2 ).
Under Figure 1a, we write down the reconstruction loss based on reconstruction identity.
Under Figure 1b (resp.
Figure 1c), we write down the reconstruction loss based on the reconstruction identity (Equation 6), along with the penalty that enforces the constraint as in Equation 7.
Below we make an assumption on the support of the latents and state the identifiability result given both observational and interventional data.
Assumption 4.
The interior of support of latents other than i , ˜ Z i , is a non-empty subset of R d − 1 .
Theorem 2.
Suppose the observational data is generated from Equation 1 and the interventional data is generated from Equation 5.
If Assumptions 1 to 4 are satisfied, then the intervened latent z i is identified up to shift and scaling, and the other latents z − i are identified up to affine transformations: the solution to Equations 6 and 7 must satisfy ˆ z k
= az
i + b and ˆ z − k
= Ez − i + f , where ˆ z − k denotes the estimate of the latents other than ˆ z k , and z − i denotes the vector of true latents other than z i .
Moreover, a ∈ R , b ∈ R , E ∈ R d − 1 × d − 1 , and f ∈ R d − 1 .
The proof of Theorem 2 is in the Appendix.
We provide some intuition here.
First, given
Assumptions 1 to 3, we can already achieve affine identification due to Theorem 1.
As a consequence, we have ˆ z k
=
a ⊤− i z − i
+ az
i + b , where z − i includes all entries of z other than z i , and a − i is a vector of the corresponding coefficients.
Next, because both ˆ z k and z i are set to a fixed value, we have that a ⊤− i z − i must also take a fixed value for all values of z − i ∈ ˜
Z i .
Finally, we argue a − i = 0 by contradiction.
If a − i ̸ = 0 , then any changes to z − i in the direction of a − i will also reflect as a change ˆ z k , which contradicts the fact that ˆ z k takes a fixed value.
We thus conclude that a − i = 0 .
We note that Theorem 2 does not rely on any distributional assumptions (e.g., parametric assumptions) on Z ; not does it rely on the nature of graphical model for Z (e.g., Z factorizes according to a certain
DAG or a Markov random field).
The only assumption we require is on the support of Z in observational data and interventional data; we require that the support of the variables must have a non-empty interior if they do not undergo any hard do interventions.
We illustrate this support assumption in Figure 2.
More generally, Theorem 2 can be extended to setting with data from multiple environments.
One such setting is where each environment corresponds to a hard do intervention on a distinct latent variable.
Under the same assumptions of Theorem 2, we can identify each of the intervened latents up to permutation (since we do not know the index of the intervened latents), shift, and scaling.
This setting requires that at most one of the latent variables undergoes a hard do intervention in each environment.
A further extension of this setting is where multiple latent variables can undergo hard do interventions in an environment.
In this setting, we can follow the exact proof recipe of Theoremand achieve affine identification with respect to the block of hard intervened latents and block of remaining latents separately.
Conclusion
We studied the role of interventional data (level-two data) in causal representation learning.
We show that, under minimal distributional conditions, latent factors are identifiable up to affine transformations from observational data under a polynomial decoder assumption.
With additional interventional data, the latent variables that undergo do interventions can further be identified up to permutation, shift, and scaling.
Extending these identifiability results beyond polynomials to more general decoder functions g is an interesting venue for future work.
One may also consider introducing appropriate notions of approximate identification to further extend Theorems 1 and 2.
Finally, we focus on interventional data from hard do interventions in this work and crucially used the nature of hard interventions in Theorem 2.
Extending the results to soft interventions for flexible families of DAGs can be another fruitful direction.
A Proofs and Technical Details A.1 Proof of Theorem
We restate the theorems from the main body of the paper for convenience.
Lemma 1.
If the matrix G that defines the polynomial g is full rank, then g is injective.
Proof Suppose this is not the case and g ( z 1 )
=
g ( z 2 ) for some z 1 ̸ =
z 2 .
Thus
[Equation]
Since z 1 ̸ =
z 2 we find a non-zero vector in the null space of G which contradicts the fact that G has full column rank.
Therefore, it cannot be the case that g ( z 1 ) =
g ( z 2 ) for some z 1 ̸ =
z 2 .
Thus g has to be injective.
Theorem 3 (Restatement of Theorem 1) .
If the data generation follows Equation 1 and Assumptionsto 3 hold, then any encoder f and decoder h that solve the reconstruction identity (Equation 2) must achieve affine identification: ∀ z ∈ Z , we must have ˆ z =
Az + c , where ˆ z = f ( x ) is the output of the encoder and z is the true latent.
Moreover, A is an invertible d × d matrix and c ∈ R d .
Proof.
We start by restating the reconstruction identity.
For all x ∈ X
[Equation]
Following the assumptions, h is restricted to be polynomial but f bears no restriction.
If H = G and f = g − 1 , we get the ideal solution ˆ z
= z , thus a solution to the above identity exists.
Since H has full column rank, we can select q rows of H such that ˜ H ∈ R q × q and rank ( ˜ H )
=
q .
Denote the corresponding matrix G that selects the same rows as ˜ G .
We restate the identity in Equation 9 in terms of ˜ H and ˜ G as follows.
[Equation]
 
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
 [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Suppose at least one of ˜
A 2 , · · · , ˜ A p is non-zero.
Among the matrices ˜
A 2 , · · · , ˜ A p which are non-zero, pick the matrix ˜
A k with largest index k .
Suppose row
i of ˜
A k has some non-zero element.
Now consider the element in the row in the LHS of (10) corresponding to ˆ z pi .
Observe that ˆ z pi is a polynomial of z of degree kp , where k ≥ 2 .
In the RHS, we have a polynomial of degree at most p .
The equality between LHS and RHS is true for all z ∈ Z (and correspondingly all x ∈ X ).
The difference of LHS and RHS is an analytic function.
Note that Z has a non-empty interior, which implies Z has a positive Lebesgue measure in R d .
Therefore, from Mityagin (2015) it follows that the LHS is equal to RHS on entire R d .
If two polynomials are equal everywhere, then their respective coefficients have to be the same.
Based on supposition, LHS has non zero coefficient for terms with degree kp while RHS has zero coefficient for terms higher than degree p .
This leads to a contradiction.
As a result, none of ˜
A 2 , · · · , ˜ A p can be non-zero.
Thus ˆ z =
˜
A 1 z + c .
Note that ˜
A 1 is also invertible.
Suppose ˜
A 1 was not invertible, then we take a latent z and perturb it in the direction in the null space of ˜
A 1 .
Note that under this perturbation ˆ z does not change but z changes.
Since z changes x has to change.
However, ˆ z is same so the reconstructed ˆ x has to be the same.
This leads to a violation of the reconstruction identity, which is a contradiction.
Therefore, ˜ Ais invertible.
A.2 Extensions to polynomial g ( · ) with unknown degree
We provide further explanation for the case when we do not know the degree.
The learner starts with solving the reconstruction identity by setting the degree of h ( · ) to be s ; here we assume H has full rank (this implicitly requires that n is greater than the number of terms in the polynomial of degree s ).
[Equation]


[Equation] [Equation]
We can restrict H to rows such that it is a square invertible matrix ˜ H .
Denote the corresponding restriction of G as ˜ G .
The equality is stated as follows.
[Equation]
[Equation]
If s > p , then ˆ z ¯ ⊗ · · · ¯ ⊗ ˆ z (cid:124)
[Equation]
s times is a polynomial of degree at least p + 1 .
Since the RHS contains a polynomial of degree at most p the two sides cannot be equal over a set of values of z with positive
Lebesgue measure in R d .
Thus the reconstruction identity will only be satisfied when s = p .
Thus we can start with the upper bound and reduce the degree of the polynomial on LHS till the identity is satisfied.
A.3 Extensions from polynomials to ϵ -approximate polynomials
We now discuss how to relax the polynomial assumption we discussed above.
Suppose g is a continuous function that can be ϵ -approximated by a polynomial of degree p on entire R d .
If we continue to use h as a polynomial, then satisfying the exact reconstruction is not possible.
Instead, we enforce approximate reconstruction as follows.
For all x ∈ X , we want
[Equation]
where ϵ is the tolerance on reconstruction error.
We assume that h ( · ) is expressive enough that the above identity is satisfied up to ϵ tolerance.
Recall ˆ z
= f ( x ) .
We further simplify it as ˆ z = f ◦ g
( z ) =
a ( z ) .
We also assume that a can be η -approximated on entire R d with a polynomial of sufficiently high degree say q .
We write this as follows.
For all z ∈ R d ,
[Equation]
We want to show that the norm of Θ k for all k ≥ 2 is sufficiently small.
We state some assumptions needed in theorem below.
Assumption 5.
Encoder f does not take values near zero, i.e., f i ( x ) ≥ γη for all x ∈ X and for all i ∈ { 1 , · · · , d } , where γ > 1 .
The absolute value of each element in ˜ H − 1 ˜ G is bounded by a fixed constant.
Consider the absolute value of the singular values of ˜ H ; we assume that the smallest absolute value is strictly positive and bounded below by ζ .
Theorem 4.
Suppose f : R n → R d and g : R d → R n are functions such that g and a = f ◦ g can be approximated by polynomials on entire R d with ϵ 2 and η tolerance respectively.
If Z = [ − z max , z max ] d , where z max is sufficiently large, and Assumptions 1, 3 and 5 hold, then the polynomial approximation of a (recall ˆ z = a ( z ) ) corresponding to solutions of approximate reconstruction identity in Equation 13 is approximately linear, i.e., the norms of the weights on higher order terms is sufficiently small.
Proof sketch.
We start by restating the approximate reconstruction identity.
We use the fact that g can be approximated with a polynomial of say degree p to simplify the identity below. [Equation]
[Equation]
Since H is full rank, we select rows of H such that ˜ H is square and invertible.
The corresponding selection for G is denoted as ˜ G .
We write the identity in terms of these matrices as follows.
[Equation]
where | σ min ( ˜ H ) | is the singular value with smallest absolute value corresponding to the matrix ˜ H .
Now we write that the polynomial that approximates ˆ z i
= a i ( z ) as follows.
[Equation]
From Assumption 5 we know that
ˆ z i ≥ γη
[Equation]
where γ > 2 .
It follows from the above equation that
[Equation]
For ˆ z i ≥ γη , we track how ˆ z pi grows below.
[Equation]
We consider z =
[ z max , · · · , z max ] .
Consider of the terms θ ij z k max inside the polynomial in the RHS above.
We assume all components of θ are positive.
Suppose θ ij ≥ 1 z k − κ − 1 max , where κ ∈ (0 , 1) , then the RHS in Equation 20 grows at least z (1+ κ ) p max (cid:0) γ − 2 γ − 1 (cid:1) p .
From Equation 16, ˆ z pi is very close to degree p polynomial in z .
Under the assumption that the terms in ˜ H − 1 ˜ G are bounded by a constant the polynomial of degree p grows at at most z p max .
The difference in growth rates the Equation 16 is an increasing function of z max for ranges where z max is sufficiently large.
Therefore, the reconstruction identity in Equation 16 cannot be satisfied for points in the neighborhood of z =
[ z max , · · · , z max ] .
Therefore, θ ij <
[Equation]
We can consider other vertices of the hypercube Z and conclude that | θ ij | <
[Equation]
A.4 Proof of Theorem
Theorem 5 (Restatement of Theorem 2) .
Suppose the observational data is generated from Equationand the interventional data is generated from Equation 5.
If Assumptions 1 to 4 are satisfied, then the intervened latent z i is identified up to shift and scaling, and the other latents z − i are identified up to affine transformations: the solution to Equations 6 and 7 must satisfy ˆ z k
= az
i + b and ˆ z − k
= Ez − i + f , where ˆ z − k denotes the estimate of the latents other than ˆ z k , and z − i denotes the vector of true latents other than z i .
Moreover, a ∈ R , b ∈ R , E ∈ R
d − 1 × d − 1 , and f ∈ R d − 1 .
Proof.
First note that since Assumptions 1-3 hold, we can continue to use the result from Theorem 1.
From Theorem 1, it follows that the estimated latents ˆ z are an affine function of the true z .
[Equation]
where a ∈ R d , b ∈ R .
We write z ∈
Z i as [ z ∗ , z − i ] .
We consider a z ∈
Z i such that z − i is in the interior of ˜
Z i .
We can write ˆ z i = a i z ∗ + a ⊤− i z − i + b , where a − i is the vector of the values of coefficients in a other than the coefficient of i th dimension, a i is i th component of a , z − i is the vector of values in z other than z i .
From the constraint in Equation 7 it follows that for all z ∈ Z i , ˆ z i = z † .
We use these expressions to carry out the following simplification.
[Equation]
[Equation]
Consider another data point z ′ ∈
Z i from the same interventional distribution such that z ′ − i =
z − i + θe j is in the interior of ˜
Z i , where e j is vector with one in j th coordinate and zero everywhere else.
From Assumption 4, we know that there exists a small enough θ such that z ′ −
i is in ˜ Z i .
Since the point is from the same interventional distribution z ′
i =
z ∗ .
For z ′ − i we have
[Equation]
We take a difference of the two equations (21) and (22) to get [Equation] [Equation]
[Equation]
From the above, we get that the j th component of a − i is zero.
We can repeat the above argument for all j and get that a − i = 0 .
B Related Work
This work is related to multiple threads of work in identifiable representation learning.
We discuss them in groups based on the type of information they leverage for representation identification.
Time-series datasets.
Several works have leveraged the structure of latent variables in time-series data to achieve identification.
The canonical data generating process in these works follows x t ← g ( z t ) and the latents z t evolve under a structured time-series.
Early works in this area consider non-stationary evolution of latents (i.e. assuming no dependence between time frames) (Hyvarinen and Morioka, 2016) and then came the models that considered stationary Markovian evolution (Hyvarinen and Morioka, 2017).
In recent years, these models have been generalized significantly in works like Lachapelle et al.
; Ahuja et al. (2021); Lippe et al.
Contrastive observation-based datasets.
In another family of works including Zimmermann et al.
; Von Kügelgen et al.
; Brehmer et al.
; Locatello et al.
; Ahuja et al. (2022a), one assumes access to contrastive observation pairs ( x, ˜ x ) .
For instance, an image and its rotated version can serve as contrastive observation pairs ( x, ˜ x ) (Zimmermann et al., 2021).
In works such as Brehmer et al. (2022), the pair ( x, ˜ x ) corresponds to a data point pre and post-intervention on the latents.
The data generation process is similar to time-series datasets in several aspects but there are a few key differences, including (a) the points ( x, ˜ x ) are not necessarily ordered by time, and (b) there may not exist any causal connections between the latents associated with x and ˜ x , unlike those in time-series datasets (e.g., Lachapelle et al.
; Lippe et al.
Auxiliary information datasets.
In the third line of work (Khemakhem et al., 2020a,b; Ahuja et al., 2022b), one assumes access to the high-dimensional observation x
(e.g., an image) and some auxiliary information u .
If u is the label of the image, we obtain standard supervised learning datasets.
If u = ˜ x is the positive pair (e.g., rotation of the image), we obtain contrastive observation-based datasets.
If u is the time stamp and previous time information x t − 1 , then we obtain the time-series datasets.
This line of work (Khemakhem et al., 2020a) often relies on strong assumptions on the interaction between latents and auxiliary information (e.g., latents are independent conditioned on auxiliary information) to guarantee provable identification.
On the role of interventional data in causal representation learning.
Finally, we contrast our work with the existing works discussed above in terms of the type of information we leverage for representation identification: we leverage interventional knowledge (level-two knowledge in Pearl’s ladder of causation (Bareinboim et al., 2022)) while most existing works leverage either observational data (level-one knowledge) (e.g. Khemakhem et al., 2020a) or counterfactual information (level-three knowledge) (e.g. Brehmer et al., 2022).
Specifically in this work, we focus on studying “to what extent can we identify the latent causal variables if the data comprises different interventional distributions?”
This question about causal representation learning shares the same spirit with causal discovery using interventional data, where we seek to understand how different interventional distributions help identify the underlying causal graph (Yang et al., 2018); both tasks rely on interventional data but they target different causal inference goals.
In contrast to our work, most existing works have leveraged other types of information for representation identification.
For example, Brehmer et al.
(2022) assume that the representation learner has access to a pair of pre- and post-intervention observations, and the data generation process therein requires their noise to be set to the same realization across the pair of points at all the nodes except the intervened nodes.
Therefore, they leverage counterfactual information (level-three knowledge) for representation identification.
As another set of examples, Lippe et al. ; Lachapelle et al.
(2022) leverage pre- and post-intervention observations in adjacent time frames to study the causal relationships between the latents.
Other works (e.g. Khemakhem et al. (2020a,b); Ahuja et al.
(2022b)) directly work with observational data, i.e., level-one knowledge, and do not require or take advantage of interventional data.
Meanwhile, these works often achieve identification guarantees by making strong assumptions on the structure of the underlying causal connections between the latents, relying on observations of auxiliary information such as the label, or capitalizing on parametric assumptions on the distribution of the latent.