Original: Interventional Causal Representation Learning
Revised: Interventional Causal Representation Learning

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: The theory of identifiable representation learning aims to build general-purposemethods that extract high-level latent (causal) factors from low-level sensory data.
Revised: The theory of identifiable representation learning aims to build general-purpose methods that extract high-level latent (causal) factors from low-level sensory data.

Original: Most existing works focus on identifiable representation learning with observationaldata, relying on distributional assumptions on latent (causal) factors.
Revised: Most existing works focus on identifiable representation learning with observational data, relying on distributional assumptions on latent (causal) factors.

Original: However,in practice, we often also have access to interventional data for representationlearning, e.g. from robotic manipulation experiments in robotics, from geneticperturbation experiments in genomics, or from electrical stimulation experimentsin neuroscience.
Revised: However, in practice, we often also have access to interventional data for representation learning, e.g. from robotic manipulation experiments in robotics, from genetic perturbation experiments in genomics, or from electrical stimulation experiments in neuroscience.

Original: How can we leverage interventional data to help identifyhigh-level latents?
Revised: How can we leverage interventional data to help identify high-level latents?

Original: To this end, we explore the role of interventional data foridentifiable representation learning in this work.
Revised: To this end, we explore the role of interventional data for identifiable representation learning in this work.

Original: We study the identifiabilityof latent causal factors with and without interventional data, under minimaldistributional assumptions on latents.
Revised: We study the identifiability of latent causal factors with and without interventional data, under minimal distributional assumptions on latents.

Original: We prove that, if the true latent maps to theobserved high-dimensional data via a polynomial function, then representationlearning via minimizing standard reconstruction loss (used in autoencoders) canidentify the true latents up to affine transformation.
Revised: We prove that, if the true latent maps to the observed high-dimensional data via a polynomial function, then representation learning via minimizing standard reconstruction loss (used in autoencoders) can identify the true latents up to affine transformation.

Original: If we further have access tointerventional data generated by hard do interventions on some latents, then wecan identify these intervened latents up to permutation, shift and scaling.
Revised: If we further have access to interventional data generated by hard do interventions on some latents, then we can identify these intervened latents up to permutation, shift and scaling.

Original:  Introduction
Revised:  Introduction

Original: Modern deep learning models like GPT-3 (Brown et al., 2020) and CLIP (Radford et al., 2021) areremarkable representation learners (Bengio et al., 2013).
Revised: Modern deep learning models like GPT-3 (Brown et al., 2020) and CLIP (Radford et al., 2021) are remarkable representation learners (Bengio et al., 2013).

Original: Despite the successes, these models continueto be far from the human ability to adapt to new situations (distribution shifts) or carry out new tasks(Geirhos et al., 2020; Bommasani et al., 2021).
Revised: Despite the successes, these models continue to be far from the human ability to adapt to new situations (distribution shifts) or carry out new tasks (Geirhos et al., 2020; Bommasani et al., 2021).

Original: Humans encapsulate the causal knowledge of theworld in a way that is highly reusable and recomposable (Goyal and Bengio, 2020), which helps themadapt to new tasks in an ever-distribution-shifting world.
Revised: Humans encapsulate the causal knowledge of the world in a way that is highly reusable and recomposable (Goyal and Bengio, 2020), which helps them adapt to new tasks in an ever-distribution-shifting world.

Original: How to make modern deep learning modelsextract a similar causal understanding of the world?
Revised: How to make modern deep learning models extract a similar causal understanding of the world?

Original: This question is central to the emerging field ofcausal representation learning (Schölkopf et al., 2021).
Revised: This question is central to the emerging field of causal representation learning (Schölkopf et al., 2021).

Original: A core task in causal representation learning is provable representation identification , i.e. developingconditions under which representation learning algorithms can provably identify latent objects (orfactors) and their causal relationships.
Revised: A core task in causal representation learning is provable representation identification , i.e. developing conditions under which representation learning algorithms can provably identify latent objects (or factors) and their causal relationships.

Original: Towards understanding this task, several notable works haveshown that provable representation identification for arbitrary data generation process (DGP) isimpossible if we only enforce the independence between the latent factors (Hyvärinen and Pajunen,1999; Locatello et al., 2019).
Revised: Towards understanding this task, several notable works have shown that provable representation identification for arbitrary data generation process (DGP) is impossible if we only enforce the independence between the latent factors (Hyvärinen and Pajunen, 1999; Locatello et al., 2019).

Original: Yet, real data generation processes often have additional structure we canleverage to achieve provable representation identification.
Revised: Yet, real data generation processes often have additional structure we can leverage to achieve provable representation identification.

Original: Such structures include the independencebetween the latent factors conditional on auxiliary information (Khemakhem et al., 2020a), thesparsity of the causal connections among the latents (Lachapelle et al., 2022), and the sparsity of themechanisms that govern the variation of the latents (Locatello et al., 2020; Ahuja et al., 2022a; Klindtet al., 2020). 
Revised: Such structures include the independence between the latent factors conditional on auxiliary information (Khemakhem et al., 2020a), the sparsity of the causal connections among the latents (Lachapelle et al., 2022), and the sparsity of the mechanisms that govern the variation of the latents (Locatello et al., 2020; Ahuja et al., 2022a; Klindt et al., 2020). Processing Systems (NeurIPS 2022)

Original: Despite these efforts toward provable representation identification, most of them have focused onrepresentation learning with observational data, relying on distributional assumptions on latent (causal)factors to achieve identification.
Revised: Despite these efforts toward provable representation identification, most existing works focus on representation learning with observational data, relying on distributional assumptions on latent (causal) factors to achieve identification.

Original: However, in practice, we often also have access to interventionaldata for representation learning, e.g. from robotic manipulation experiments in robotics (Collins et al. ,2019), from genetic perturbation experiments in genomics (Dixit et al., 2016), or from electricalstimulation experiments in neuroscience (Nejatbakhsh et al., 2021).
Revised: However, in practice, we often also have access to interventional data for representation learning, e.g. from robotic manipulation experiments in robotics (Collins et al., 2019), from genetic perturbation experiments in genomics (Dixit et al., 2016), or from electrical stimulation experiments in neuroscience (Nejatbakhsh et al., 2021).

Original: In this work, we seek tounderstand how we can leverage such interventional data to identify high level (causal) factors fromlow level data.
Revised: In this work, we seek to understand how we can leverage such interventional data to identify high level (causal) factors from low level data.

Original: Our key findings are summarized below.
Revised: Our key findings are summarized below.

Original: • Under the assumption that the true latent factors map to the high dimensional observationsvia a finite degree multivariate polynomial, we first show that it is possible to achieve affine identification with respect to the true latents with minimal assumptions on the support of thetrue latents and no further distributional assumptions.• If we also observe data where some latents undergo a hard do intervention (Pearl, 2009),then we can guarantee affine identification up to the block of these hard intervened latents.
Revised: • Under the assumption that the true latent factors map to the high dimensional observations via a finite degree multivariate polynomial, we first show that it is possible to achieve affine identification with respect to the true latents with minimal assumptions on the support of the true latents and no further distributional assumptions. • If we also observe data where some latents undergo a hard do intervention (Pearl, 2009), then we can guarantee affine identification up to the block of these hard intervened latents.

Original: As a result, if in some environments only one latent variable undergoes a do intervention, then those latents are identified up to permutation, shift and scaling.
Revised: As a result, if in some environments only one latent variable undergoes a do intervention, then those latents are identified up to permutation, shift and scaling.

Original: Due to space limitations, we relegate the related works to the Appendix.
Revised: Due to space limitations, we relegate the related works to the Appendix.

Original:  Representation Identification with Observational & Interventional Data
Revised:  Representation Identification with Observational & Interventional Data

Original: The data generating process.
Revised: The data generating process.

Original: We begin with defining the data generation process.
Revised: We begin with defining the data generation process.

Original: We observex ∈ X ⊆ R n
Revised: We observe x ∈ X ⊆ R n

Original: and it is generated from underlying latent variables z ∈ Z ⊆ R d , with n ≥ d .
Revised: and it is generated from underlying latent variables z ∈ Z ⊆ R d , with n ≥ d .

Original: Supposethe data generating process follows
Revised: Suppose the data generating process follows

Original: [Equation]
Revised: [Equation]

Original: (1) where P Z is the distribution from which the latent z is sampled and x is the observed data point thatis rendered using an injective decoder g : R d → R n from the underlying latent.
Revised: where P Z is the distribution from which the latent z is sampled and x is the observed data point that is rendered using an injective decoder g : R d → R n from the underlying latent.

Original: We denote Z as thesupport of P Z , and X =
Revised: We denote Z as the support of P Z , and X =

Original: g ( Z ) .
Revised: g ( Z ) .

Original: Hence, the support of observations x is X .
Revised: Hence, the support of observations x is X .

Original: The identifiable representation learning task.
Revised: The identifiable representation learning task.

Original: To perform representation learning, we learnencoders (also known as the representation function) that aim to estimate the true latents z .
Revised: To perform representation learning, we learn encoders (also known as the representation function) that aim to estimate the true latents z .

Original: Specifically, we aim to find an encoder f : R n → R d and a decoder h : R d → R n such thatthe encoder and decoder jointly satisfy the following reconstruction identity.
Revised: Specifically, we aim to find an encoder f : R n → R d and a decoder h : R d → R n such that the encoder and decoder jointly satisfy the following reconstruction identity.

Original: For all x ∈ X[-
-]
Revised: For all x ∈ X[-
-]

Original: [Equation]
Revised: [Equation]

Original: We denote the learned representation as ˆ z ≜ f ( x ) , where ˆ z is the latent that the encoder guesses.
Revised: We denote the learned representation as ˆ z ≜ f ( x ) , where ˆ z is the latent that the encoder guesses.

Original: Note that equation (2) is highly underspecified and can have many solutions that are not guaranteedto identify the latents
Revised: Note that equation (2) is highly underspecified and can have many solutions that are not guaranteed to identify the latents

Original: z (e.g., take any solution f , h of the reconstruction identity, b ◦ f , h ◦ b − 1 isanother valid solution where b is an invertible map.).
Revised: z (e.g., take any solution f , h of the reconstruction identity, b ◦ f , h ◦ b − 1 is another valid solution where b is an invertible map.).

Original: In many problems, exact identification of latentsis not needed, nor reasonable (for example we do not care about the labels given to each latent, i.e.,about coordinate permutations of z ).
Revised: In many problems, exact identification of latents is not needed, nor reasonable (for example we do not care about the labels given to each latent, i.e., about coordinate permutations of z ).

Original: Therefore, we study conditions under which the true latents areidentifiable up to certain transformations, e.g. affine transformations, coordinate permutations, etc.
Revised: Therefore, we study conditions under which the true latents are identifiable up to certain transformations, e.g. affine transformations, coordinate permutations, etc.

Original: Overview of results.
Revised: Overview of results.

Original: Below we present a suite of identifiability results.
Revised: Below we present a suite of identifiability results.

Original: We first show that, when thetrue decoder g is a polynomial of z with known degree and the learned decoder h is also a polynomialwith the same degree, then latents z can be identified up to affine transfomations using the encoderlearned from the reconstruction identity (Equation (2)).
Revised: We first show that, when the true decoder g is a polynomial of z with known degree and the learned decoder h is also a polynomial with the same degree, then latents z can be identified up to affine transfomations using the encoder learned from the reconstruction identity (Equation (2)).

Original: We relax the result to settings where we donot know the exact degree of g
Revised: We relax the result to settings where we do not know the exact degree of g

Original: but we know an upper bound on it.
Revised: but we know an upper bound on it.

Original: We also provide approximateaffine identification guarantees for the setting when g is ϵ -approximated by a polynomial.
Revised: We also provide approximate affine identification guarantees for the setting when g is ϵ -approximated by a polynomial.

Original: In thesecond half, we investigate the role of interventions.
Revised: In the second half, we investigate the role of interventions.

Original: If we observe data where some latents undergoa hard do intervention (Pearl, 2009), then we can guarantee affine identification up to the block ofthese hard intervened latents.
Revised: If we observe data where some latents undergo a hard do intervention (Pearl, 2009), then we can guarantee affine identification up to the block of these hard intervened latents.

Original: As a result, if in some environments only one latent variable variableundergoes a do intervention, then that latent is identifiable up to shift and scaling
Revised: As a result, if in some environments only one latent variable variable undergoes a do intervention, then that latent is identifiable up to shift and scaling.

Original: 2.1 Affine identifiability with observational data
Revised: 2.1 Affine identifiability with observational data

Original: Assumption 1.
Revised: Assumption 1.

Original: The interior of the support of Z denoted as Z is a non-empty subset of R d .[-
-]
Revised: The interior of the support of Z denoted as Z is a non-empty subset of R d .[-
-]

Original: Assumption 2.
Revised: Assumption 2.

Original: The decoder g is a polynomial of degree p and its corresponding coefficient matrix(a.k.a.
Revised: The decoder g is a polynomial of degree p and its corresponding coefficient matrix (a.k.a.

Original: weight matrix) G has full column rank.
Revised: weight matrix) G has full column rank.

Original: g is determined by G as follows.
Revised: g is determined by G as follows.

Original: For all z ∈ R d
Revised: For all z ∈ R d

Original: [Equation]
Revised: [Equation]

Original: where ¯ ⊗ represents the Kronecker product with all distinct entries, for example
Revised: where ¯ ⊗ represents the Kronecker product with all distinct entries, for example

Original: , z =
Revised: , z =

Original: [ z 1 , z 2 ] , thenz ¯ ⊗ z
Revised: [ z 1 , z 2 ] , then z ¯ ⊗ z

Original: =
Revised: =

Original: [ z 21 , z 1 z 2 , z 22 ] .
Revised: [ z 21 , z 1 z 2 , z 22 ] .

Original: In identifiable representation learning, it is common to assume that g is injective or else the problemof identification can become ill defined as multiple latents can give rise to the same observation x .
Revised: In identifiable representation learning, it is common to assume that g is injective or else the problem of identification can become ill defined as multiple latents can give rise to the same observation x .

Original: Ifthe matrix G ∈ R n × q has full column rank of q , then g is guranteed to be injective; the proof of thisclaim is in the Appendix.
Revised: If the matrix G ∈ R n × q has full column rank of q , then g is guranteed to be injective; the proof of this claim is in the Appendix.

Original: The full-column-rank condition for G in Assumption 2 implicitly requiresthat the dimensionality n of the data is greater than the number of terms in the polynomial of degreep , namely n =
Revised: The full-column-rank condition for G in Assumption 2 implicitly requires that the dimensionality n of the data is greater than the number of terms in the polynomial of degree p , namely n =

Original: O ( d p ) , where d is the dimensionality of z .
Revised: O ( d p ) , where d is the dimensionality of z .

Original: Assumption 3.
Revised: Assumption 3.

Original: The learned decoder h is a polynomial of degree p and its corresponding coefficientmatrix H has a full column rank.
Revised: The learned decoder h is a polynomial of degree p and its corresponding coefficient matrix H has a full column rank.

Original: h is determined by H as follows.
Revised: h is determined by H as follows.

Original: For all z ∈ R d
Revised: For all z ∈ R d

Original: [Equation]
Revised: [Equation]

Original: where ¯ ⊗ represents the Kronecker product with all distinct entries.
Revised: where ¯ ⊗ represents the Kronecker product with all distinct entries.

Original: Recall ˆ z = f ( x ) .
Revised: Recall ˆ z = f ( x ) .

Original: Here we can substitute x = g ( z ) to get ˆ z = f ◦ g
Revised: Here we can substitute x = g ( z ) to get ˆ z = f ◦ g

Original: ( z ) =
Revised: ( z ) =

Original: a ( z ) , where a ≜ f ◦ g .
Revised: a ( z ) , where a ≜ f ◦ g .

Original: Inthe next theorem, we show that a is an affine transform under the assumptions stated above.
Revised: In the next theorem, we show that a is an affine transform under the assumptions stated above.

Original: Theorem 1.
Revised: Theorem 1.

Original: If the data generation follows equation (1) and Assumptions 1-3 hold, then the encoderf and decoder h that solve reconstruction identity in equation (2) achieve affine identification, i.e.,∀ z ∈ Z , ˆ z = Az + c , where ˆ z is the output of the encoder f and z is the true latent and A is aninvertible d × d matrix and c ∈ R d .
Revised: If the data generation follows equation (1) and Assumptions 1-3 hold, then the encoder f and decoder h that solve reconstruction identity in equation (2) achieve affine identification, i.e., ∀ z ∈ Z , ˆ z = Az + c , where ˆ z is the output of the encoder f and z is the true latent and A is an invertible d × d matrix and c ∈ R d .

Original: The proof is provided in the Appendix.
Revised: The proof is provided in the Appendix.

Original: To get some intuition for the proof, we consider z to beone-dimensional, g to be a degree two polynomial, and x to be three-dimensional.
Revised: To get some intuition for the proof, we consider z to be one-dimensional, g to be a degree two polynomial, and x to be three-dimensional.

Original: By solving thereconstruction identity on all x , we obtain that ˆ z is at most a degree-two polynomial of z .
Revised: By solving the reconstruction identity on all x , we obtain that ˆ z is at most a degree-two polynomial of z .

Original: We alsoobtain that ˆ z 2 is also a polynomial of at most degree two in z .
Revised: We also obtain that ˆ z 2 is also a polynomial of at most degree two in z .

Original: If ˆ z is a degree-two polynomial of z ,then ˆ z 2 is degree four, which cannot be the case.
Revised: If ˆ z is a degree-two polynomial of z , then ˆ z 2 is degree four, which cannot be the case.

Original: Therefore, ˆ z is a degree-one polynomial.
Revised: Therefore, ˆ z is a degree-one polynomial.

Original: Extension of Theorem 1 when degree p is unknown.
Revised: Extension of Theorem 1 when degree p is unknown.

Original: In the theorem above, we required that thedegree p is known.
Revised: In the theorem above, we required that the degree p is known.

Original: We can extend the above findings to settings where we know an upper bound onthe value p , denoted as s .
Revised: We can extend the above findings to settings where we know an upper bound on the value p , denoted as s .

Original: Through a simple iterative procedure, which we describe next, we claim thelearner can achieve affine identification.
Revised: Through a simple iterative procedure, which we describe next, we claim the learner can achieve affine identification.

Original: The learner first tries to solve the reconstruction identity withwith a polynomial h ( · ) of degree equal to the upper bound s .
Revised: The learner first tries to solve the reconstruction identity with with a polynomial h ( · ) of degree equal to the upper bound s .

Original: We assume that H is full rank, whichimplicitly requires n to be sufficiently large, i.e., n = O ( d s ) .
Revised: We assume that H is full rank, which implicitly requires n to be sufficiently large, i.e., n = O ( d s ) .

Original: If s > p , then we can show that there isno solution to the reconstruction identity in equation (2) (see Appendix for further justification).
Revised: If s > p , then we can show that there is no solution to the reconstruction identity in equation (2) (see Appendix for further justification).

Original: Inthe next step, the learner searches over full rank polynomials h ( · ) of degree s − 1 and this procedureis repeated until the degree is p , which is when the reconstruction identity is satisfied.
Revised: In the next step, the learner searches over full rank polynomials h ( · ) of degree s − 1 and this procedure is repeated until the degree is p , which is when the reconstruction identity is satisfied.

Original: Extension beyond polynomial decoders.
Revised: Extension beyond polynomial decoders.

Original: From Stone-Weiresstrass theorem (Rudin et al. , 1976),we know that a continuous map on a closed and bounded set can be approximated with a highdimensional polynomial.
Revised: From Stone-Weiresstrass theorem (Rudin et al., 1976), we know that a continuous map on a closed and bounded set can be approximated with a high dimensional polynomial.

Original: Inspired by this, we extend our results to a class of maps g ( · ) that are ϵapproximated by a polynomial of sufficiently high degree.
Revised: Inspired by this, we extend our results to a class of maps g ( · ) that are ϵ approximated by a polynomial of sufficiently high degree.

Original: Essentially, we show that the map a ( · )that connects ˆ z to true z is an approximately linear map, i.e., the norm of the weight on higher orderterms in the polynomial expansion is sufficiently small.
Revised: Essentially, we show that the map a ( · ) that connects ˆ z to true z is an approximately linear map, i.e., the norm of the weight on higher order terms in the polynomial expansion is sufficiently small.

Original: [Figure]
Revised: [Figure]

Original: 2.2 Identifiability with interventional data
Revised: 2.2 Identifiability with interventional data

Original: In the previous section, we considered z to be drawn from an arbitrary distribution P Z . We call thedata generated from equation (1) as the observational data.
Revised: In the previous section, we considered representation identification from some observational data where the data is generated from Equation (1). The latent variable Z is drawn from an arbitrary distribution P Z , which does not necessarily come from any structural causal model.

Original: We now define interventional data withhard do interventions.
Revised: We next study how interventional data could enhance representation identification.

Original: In the interventional data that we consider, we assume that at least one of thecomponents of Z  is set to a fixed value.
Revised: We consider the general case where at least one component of Z , say the i th component, is set to a fixed value, and the remaining components are sampled from a distribution Q Z − i . Note Q Z − i does not have to be equal to the distribution P Z − i | z i =

Revised: z ∗ over samples Z − i generated from Equation (1) when z i =

Revised: z ∗ .

Original: The interventional data generation follows
Revised: The data-generating process is written as follows

Original: [Equation]
Revised: [Equation]

Original: where  variable z iundergoes a hard do intervention and is set to z ∗ , the remaining d − 1 variables inz denoted as z − i  are sampled from distribution P Z − i | z i = z ∗ , and  g is the true decoder that generatesobserved x  . We denote the data x ′ s generated under intervention on  i th latent variable as D i .
Revised: where the variable z i is fixed to be equal to z ∗ ; the remaining d − 1 variables in z (denoted as z − i ) are sampled from Q Z − i ; and the function g is the true decoder that generates observed x from z . The results we present below only require a restriction on the support of Q Z − i and this flexibility allows

Revised: Q

Original: Wedenote the support of latent z under the above intervention as Z  i and the corresponding supportof x ′ s as X i .
Revised: Z − i to model standard do interventions (Pearl, 2009), i.e., Q Z − i = P Z − i | do ( z i = z ∗ ) as we illustrate below.

Original: We also denote the support of latents other than z i as ˜ Z  i.
Revised: Our DGP also allows the possibility that Q Z − i

Original: In contrast, we denotethe observational data generated from equation (1) as D 0 .
Revised: = P Z − i | z i = z ∗ .

Original: We write the reconstruction identity forobservational data ( D 0 ) and interventional data ( D i ) as follows.For all x ∈ X ∪ X
Revised: We denote the support of x under interventional data generated from equation (5) as X i . We write the reconstruction identity as follows.

Original: i

Original: [Equation]
Revised: [Equation]

Original: Further, we also enforce the constraint on the encoder that on all the data points x ∈ X i one of thecomponents (say k ) takes some arbitrary fixed value z † .
Revised: Further, we also enforce the constraint on the encoder that on all the data points x ∈ X i one of the components (say k ) takes some arbitrary fixed value z † .

Original: For all x ∈ X i .
Revised: For all x ∈ X i .

Original: [Equation]
Revised: [Equation]

Original: where f k ( x ) denotes the k th component which is required to take a fixed value z † .
Revised: where f k ( x ) denotes the k th component which is required to take a fixed value z † .

Original: Structural causal model for Z and associated interventions In the work so far, we have notspecifically invoked the condition that Z is generated from a structural causal model.
Revised: Structural causal model for Z and associated interventions In the work so far, we have not specifically invoked the condition that Z is generated from a structural causal model.

Original: Our resultsare more general and apply to Z that are not generated from a structural causal model as well.
Revised: Our results are more general and apply to Z that are not generated from a structural causal model as well.

Original: Now suppose Z is drawn from a structutral causal model.
Revised: Now suppose Z is drawn from a structutral causal model.

Original: Say we observe both observational andinterventional data.
Revised: Say we observe both observational and interventional data.

Original: In the interventional data, in equation (5) exactly one latent undergoes a hard do
Revised: In the interventional data, in equation (5) exactly one latent undergoes a hard do

Original: [Figure]
Revised: [Figure]

Original: intervention.
Revised: intervention.

Original: The constraint in equation (7) enforces that exactly one of the components of encoderalso takes a fixed value.
Revised: The constraint in equation (7) enforces that exactly one of the components of encoder also takes a fixed value.

Original: We illustrate this with a simple example.
Revised: We illustrate this with a simple example.

Original: Suppose Z =
Revised: Suppose Z =

Original: [ Z 1 , Z 2 , Z 3 , Z 4 ]is drawn from a structural causal model with the underlying DAG shown in Figure 1 a. Figure 1 b(c) shows the DAG when Z 3 ( Z 2 ) undergoes a hard intervention and is set to z ∗ 3 ( z ∗ 2 ).
Revised: [ Z 1 , Z 2 , Z 3 , Z 4 ] is drawn from a structural causal model with the underlying DAG shown in Figure 1 a. Figure 1 b (c) shows the DAG when Z 3 ( Z 2 ) undergoes a hard intervention and is set to z ∗ 3 ( z ∗ 2 ).

Original: Under Figure 1a), we show the reconstruction loss based on reconstruction identity.
Revised: Under Figurea), we show the reconstruction loss based on reconstruction identity.

Original: Under Figure 1 b,c, we showthe reconstruction loss based on the reconstruction identity (equation (6)) along with a penalty thatenforces the constraint in equation (7).
Revised: Under Figure 1 b,c, we show the reconstruction loss based on the reconstruction identity (equation (6)) along with a penalty that enforces the constraint in equation (7).

Original: Assumption 4.
Revised: Assumption 4.

Original: The interior of support of latents other than i , ˜ Z i , is a non-empty subset of R d − 1 .
Revised: The interior of support of latents other than i , ˜ Z i , is a non-empty subset of R d − 1 .

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose the observational data is generated from equation (1) and interventional data isgenerated from equation (5) .
Revised: Suppose the observational data is generated from equation (1) and interventional data is generated from equation (5) .

Original: If Assumption 1- 4 are satisfied, then the solution to equation (6) and (7)satisfy ˆ z k
Revised: If Assumption 1- 4 are satisfied, then the solution to equation (6) and (7) satisfy ˆ z k

Original: = az
Revised: = az

Original: i + b and ˆ z − k
Revised: i + b and ˆ z − k

Original: = Ez − i + f , where ˆ z − k denotes the estimate of the latents other thanˆ z k and z − i denotes the vector of true latents other than z i , a ∈ R , b ∈ R ,
Revised: = Ez − i + f , where ˆ z − k denotes the estimate of the latents other than ˆ z k and z − i denotes the vector of true latents other than z i , a ∈ R , b ∈ R ,

Original: E ∈ R
Revised: E ∈ R

Original: d − 1 × d − 1 , f ∈ R d − 1 ,i.e., z i is identified up to shift and scaling, and z − i is affine identified.
Revised: d − 1 × d − 1 , f ∈ R d − 1 , i.e., z i is identified up to shift and scaling, and z − i is affine identified.

Original: The proof of the above theorem is in the Appendix and we provide some intuition next.
Revised: The proof of the above theorem is in the Appendix and we provide some intuition next.

Original: From
Revised: From

Original: Assumptions 1-3 hold, we can continue to use the result from previous Theorem 1 and claim affineidentification.
Revised: Assumptions 1-3 hold, we can continue to use the result from previous Theorem 1 and claim affine identification.

Original: From affine identification it follows that ˆ z k
Revised: From affine identification it follows that ˆ z k

Original: = a t − i z − i
Revised: = a t − i z − i

Original: + az
Revised: + az

Original: i + b , where z − i is vectorof z other than z i and a − i is a vector of corresponding coefficients.
Revised: i + b , where z − i is vector of z other than z i and a − i is a vector of corresponding coefficients.

Original: Note that both ˆ z k
Revised: Note that both ˆ z k

Original: and z i are setto a fixed value.
Revised: and z i are set to a fixed value.

Original: As a result, a t − i z − i takes a fixed value for all values of z − i ∈
Revised: As a result, a t − i z − i takes a fixed value for all values of z − i ∈

Original: ˜
Revised: ˜

Original: Z i .
Revised: Z i .

Original: Suppose a − i ̸
Revised: Suppose a − i ̸

Original: = 0then this implies any changes to z − i in the direction of a − i will also reflect as a change ˆ z k but that isa contradiction, which implies a − i = 0 .
Revised: =then this implies any changes to z − i in the direction of a − i will also reflect as a change ˆ z k but that is a contradiction, which implies a − i = 0 .

Original: In Theorem 2, we did not make distributional assumptions(e.g. , no parametric assumption) on Z or on the nature of graphical model for Z (e.g., Z factorizesaccording to a certain DAG or a Markov random field).
Revised: In Theorem 2, we did not make distributional assumptions (e.g., no parametric assumption) on Z or on the nature of graphical model for Z (e.g., Z factorizes according to a certain DAG or a Markov random field).

Original: We only make assumptions on the support of
Revised: We only make assumptions on the support of

Original: Z in observational data and interventional data.
Revised: Z in observational data and interventional data.

Original: We require that the support of the variables that donot undergo a hard intervention have a non-empty interior.
Revised: We require that the support of the variables that do not undergo a hard intervention have a non-empty interior.

Original: We illustrate this in Figure 2.
Revised: We illustrate this in Figure 2.

Original: Suppose we have data from multiple environments, where each environment corresponds to a harddo intervention on a distinct latent variable.
Revised: Suppose we have data from multiple environments, where each environment corresponds to a hard do intervention on a distinct latent variable.

Original: Under same Assumptions as Theorem 2, we can identifyeach of the interevened latents up to permutation (since we do not know the index of the latent),shift, and scaling.
Revised: Under same Assumptions as Theorem 2, we can identify each of the interevened latents up to permutation (since we do not know the index of the latent), shift, and scaling.

Original: We now consider a natural extension of the above setting.
Revised: We now consider a natural extension of the above setting.

Original: In the above setting, weassumed that in each environment at most one of the latent variables undergoes a hard do intervention.
Revised: In the above setting, we assumed that in each environment at most one of the latent variables undergoes a hard do intervention.

Original: Suppose multiple latent variables undergo a hard do intervention.
Revised: Suppose multiple latent variables undergo a hard do intervention.

Original: We can follow the exact proofrecipe of Theorem 2 and achieve affine identification with respect to the block of hard intervenedlatents and block of remaining latents separately.
Revised: We can follow the exact proof recipe of Theorem 2 and achieve affine identification with respect to the block of hard intervened latents and block of remaining latents separately.

Original:  Conclusion
Revised:  Conclusion

Original: We studied the role of interventional data (level-two data) in causal representation learning.
Revised: We studied the role of interventional data (level-two data) in causal representation learning.

Original: Weshow that latent variables that undergo do interventions can be identified up to permutation, shift,and scaling when the true decoder is a finite degree polynomial.
Revised: We show that latent variables that undergo do interventions can be identified up to permutation, shift, and scaling when the true decoder is a finite degree polynomial.

Original: Extending these results beyondpolynomials and do interventions are important next steps.
Revised: Extending these results beyond polynomials and do interventions are important next steps.

Original: A Appendix
Revised: A Appendix

Original: We restate the theorems from the main body of the paper for convenience.
Revised: We restate the theorems from the main body of the paper for convenience.

Original: Lemma 1.
Revised: Lemma 1.

Original: If the matrix G that defines the polynomial g is full rank, then g is injective.
Revised: If the matrix G that defines the polynomial g is full rank, then g is injective.

Original: Proof Suppose this is not the case and g ( z 1 )
Revised: Proof Suppose this is not the case and g ( z 1 )

Original: =
Revised: =

Original: g ( z 2 ) for some z 1 ̸ =
Revised: g ( z 2 ) for some z 1 ̸ =

Original: z 2 .
Revised: z 2 .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Since z 1 ̸ =
Revised: Since z 1 ̸ =

Original: z 2 we find a non-zero vector in the null space of G which contradicts the fact that G hasfull column rank.
Revised: z 2 we find a non-zero vector in the null space of G which contradicts the fact that G has full column rank.

Original: Therefore, it cannot be the case that g ( z 1 ) =
Revised: Therefore, it cannot be the case that g ( z 1 ) =

Original: g ( z 2 ) for some z 1 ̸ =
Revised: g ( z 2 ) for some z 1 ̸ =

Original: z 2 .
Revised: z 2 .

Original: Thus g hasto be injective.
Revised: Thus g has to be injective.

Original: Theorem 3.
Revised: Theorem 3.

Original: If the data generation follows equation (1) and Assumptions 1-3 hold, then the encoderf and decoder h that solve reconstruction identity in equation (2) achieve affine identification, i.e.,∀ z ∈ Z , ˆ z = Az + c , where ˆ z is the output of the encoder f and z is the true latent and A is aninvertible d × d matrix.
Revised: If the data generation follows equation (1) and Assumptions 1-3 hold, then the encoder f and decoder h that solve reconstruction identity in equation (2) achieve affine identification, i.e., ∀ z ∈ Z , ˆ z = Az + c , where ˆ z is the output of the encoder f and z is the true latent and A is an invertible d × d matrix.

Original: Proof.
Revised: Proof.

Original: We start by restating the reconstruction identity.
Revised: We start by restating the reconstruction identity.

Original: For all x 
Revised: For all x ∈ X

Original: [Equation]
Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Original: Following the assumptions, h is restricted to be polynomial but f bears no restriction.
Revised: Following the assumptions, h is restricted to be polynomial but f bears no restriction.

Original: If H = Gand f = g − 1 , we get the ideal solution ˆ z
Revised: If H = G and f = g − 1 , we get the ideal solution ˆ z

Original: = z , thus a solution to the above identity exists.
Revised: = z , thus a solution to the above identity exists.

Original: Since Hhas full column rank, we can select q rows of H such that ˜ H ∈ R q × q and rank ( ˜ H )
Revised: Since H has full column rank, we can select q rows of H such that ˜ H ∈ R q × q and rank ( ˜ H )

Original: =
Revised: =

Original: q .
Revised: q .

Original: Denote thecorresponding matrix G that select the same rows as ˜ G .
Revised: Denote the corresponding matrix G that select the same rows as ˜ G .

Original: We restate the identity in equation (9) interms of ˜ H and ˜ G as follows.
Revised: We restate the identity in equation (9) in terms of ˜ H and ˜ G as follows.

Original: [Equation]
Revised: [Table]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 
Revised: [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Suppose at least one of ˜
Revised: Suppose at least one of ˜

Original: A 2 , · · · , ˜ A p is non-zero.
Revised: A 2 , · · · , ˜ A p is non-zero.

Original: Among the matrices ˜
Revised: Among the matrices ˜

Original: A 2 , · · · , ˜ A p which are non-zero,pick the matrix ˜
Revised: A 2 , · · · , ˜ A p which are non-zero, pick the matrix ˜

Original: A k with largest index k .
Revised: A k with largest index k .

Original: Suppose row
Revised: Suppose row

Original: i of ˜
Revised: i of ˜

Original: A k has some non-zero element.
Revised: A k has some non-zero element.

Original: Nowconsider the element in the row in the LHS of (10) corresponding to ˆ z pi .
Revised: Now consider the element in the row in the LHS of (10) corresponding to ˆ z pi .

Original: Observe that ˆ z pi is apolynomial of z of degree kp , where k ≥ 2 .
Revised: Observe that ˆ z pi is a polynomial of z of degree kp , where k ≥ 2 .

Original: In the RHS, we have a polynomial of degree at mostp .
Revised: In the RHS, we have a polynomial of degree at most p .

Original: The equality between LHS and RHS is true for all z ∈ Z (and correspondingly all x ∈ X ).
Revised: The equality between LHS and RHS is true for all z ∈ Z (and correspondingly all x ∈ X ).

Original: Thedifference of LHS and RHS is an analytic function.
Revised: The difference of LHS and RHS is an analytic function.

Original: From Mityagin (2015) it follows that the LHS isequal to RHS on entire R d .
Revised: From Mityagin (2015) it follows that the LHS is equal to RHS on entire R d .

Original: If two polynomials are equal everywhere, then their respective coefficientshave to be the same.
Revised: If two polynomials are equal everywhere, then their respective coefficients have to be the same.

Original: Based on supposition, LHS has non zero coefficient for terms with degree kpwhile RHS has zero coefficient for terms higher than degree p .
Revised: Based on supposition, LHS has non zero coefficient for terms with degree kp while RHS has zero coefficient for terms higher than degree p .

Original: This leads to a contradiction.
Revised: This leads to a contradiction.

Original: Asa result, none of ˜
Revised: As a result, none of ˜

Original: A 2 , · · · , ˜ A p can be non-zero.
Revised: A 2 , · · · , ˜ A p can be non-zero.

Original: Thus ˆ z =
Revised: Thus ˆ z =

Original: ˜
Revised: ˜

Original: A 1 z + c .
Revised: A 1 z + c .

Original: Note that ˜
Revised: Note that ˜

Original: A 1 is also invertible.
Revised: A 1 is also invertible.

Original: Suppose ˜
Revised: Suppose ˜

Original: A 1 was not invertible, then we take a latent z and perturb it in the direction in the null spaceof ˜
Revised: A 1 was not invertible, then we take a latent z and perturb it in the direction in the null space of ˜

Original: A 1 .
Revised: A 1 .

Original: Note that under this perturbation ˆ z does not change but z changes.
Revised: Note that under this perturbation ˆ z does not change but z changes.

Original: Since z changes x has tochange.
Revised: Since z changes x has to change.

Original: However, ˆ z is same so the reconstructed ˆ x has to be the same.
Revised: However, ˆ z is same so the reconstructed ˆ x has to be the same.

Original: This leads to a violation of thereconstruction identity.
Revised: This leads to a violation of the reconstruction identity.

Original: A.1 Unknown degree of g ( · )
Revised: A.1 Unknown degree of g ( · )

Original: We provide further explanation for the case when we do not know the degree.
Revised: We provide further explanation for the case when we do not know the degree.

Original: The learner starts withsolving the reconstruction identity by setting the degree of h ( · ) to be s ; here we assume H has fullrank (this implicitly requires that n is greater than the number of terms in polynomial of degree s ).H
Revised: The learner starts with solving the reconstruction identity by setting the degree of h ( · ) to be s ; here we assume H has full rank (this implicitly requires that n is greater than the number of terms in polynomial of degree s ).

Original: [Equation]
Revised: [Equation]

Original:    (11)
Revised: [Equation]

Original: We can restrict H to rows such that it is a square invertible matrix ˜ H .
Revised: We can restrict H to rows such that it is a square invertible matrix ˜ H .

Original: Denote the correspondingrestriction of G as ˜ G .
Revised: Denote the corresponding restriction of G as ˜ G .

Original: The equality is stated as follows.
Revised: The equality is stated as follows.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If s > p , then ˆ z ¯ ⊗ · · · ¯ ⊗ ˆ z (cid:124) (cid:123)(cid:122) (cid:125)
Revised: If s > p , then ˆ z ¯ ⊗ · · · ¯ ⊗ ˆ z (cid:124) (cid:123)(cid:122) (cid:125)

Original: s times is a polynomial of degree at least p + 1 .
Revised: s times is a polynomial of degree at least p + 1 .

Original: Since the RHS contains apolynomial of degree at most p the two sides cannot be equal over a set of values of z with positive
Revised: Since the RHS contains a polynomial of degree at most p the two sides cannot be equal over a set of values of z with positive

Original: Lebesgue measure in R d .
Revised: Lebesgue measure in R d .

Original: Thus the reconstruction identity will only be satisfied when s = p .
Revised: Thus the reconstruction identity will only be satisfied when s = p .

Original: Thuswe can start with the upper bound and reduce the degree of polynomial on LHS till the identity issatisfied.
Revised: Thus we can start with the upper bound and reduce the degree of polynomial on LHS till the identity is satisfied.

Original: A.2 Extension from polynomials to ϵ -approximate polynomials
Revised: A.2 Extension from polynomials to ϵ -approximate polynomials

Original: We now discuss how to relax the polynomial assumption we discussed above.
Revised: We now discuss how to relax the polynomial assumption we discussed above.

Original: Suppose g is acontinuous function that can be ϵ -approximated by a polynomial of degree p on entire R d .
Revised: Suppose g is a continuous function that can be ϵ -approximated by a polynomial of degree p on entire R d .

Original: If wecontinue to use h as a polynomial, then satisfying the exact reconstruction is not possible.
Revised: If we continue to use h as a polynomial, then satisfying the exact reconstruction is not possible.

Original: Instead,we enforce approximate reconstruction as follows.
Revised: Instead, we enforce approximate reconstruction as follows.

Original: For all x ∈ X we want
Revised: For all x ∈ X we want

Original: [Equation]
Revised: [Equation]

Original: (13) where ϵ is the tolerance on reconstruction error.
Revised: 13) where ϵ is the tolerance on reconstruction error.

Original: We assume that h ( · ) is expressive enough thatthe above identity is satisfied up to ϵ tolerance.
Revised: We assume that h ( · ) is expressive enough that the above identity is satisfied up to ϵ tolerance.

Original: Recall ˆ z
Revised: Recall ˆ z

Original: = f ( x ) .
Revised: = f ( x ) .

Original: We further simplify it asˆ z = f ◦ g
Revised: We further simplify it as ˆ z = f ◦ g

Original: ( z ) =
Revised: ( z ) =

Original: a ( z ) .
Revised: a ( z ) .

Original: We also assume that a can be η -approximated on entire R d with a polynomialof sufficiently high degree say p and q respectively.
Revised: We also assume that a can be η -approximated on entire R d with a polynomial of sufficiently high degree say p and q respectively.

Original: We write this as follows.(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: We write this as follows.

Original: [Equation]
Revised: [Equation]

Original: We want to show that the norm of Θ k for all k ≥ 2 is sufficiently small.
Revised: We want to show that the norm of Θ k for all k ≥ 2 is sufficiently small.

Original: We state some assumptionsneeded in theorem below.
Revised: We state some assumptions needed in theorem below.

Original: Assumption 5.
Revised: Assumption 5.

Original: Encoder f does not take values near zero, i.e., f i ( x ) ≥ γη for all x ∈ X and forall i ∈ { 1 , · · · , d } , where γ > 1 .
Revised: Encoder f does not take values near zero, i.e., f i ( x ) ≥ γη for all x ∈ X and for all i ∈ { 1 , · · · , d } , where γ > 1 .

Original: The absolute value of each element in ˜ H − 1 ˜ G is bounded by afixed constant.
Revised: The absolute value of each element in ˜ H − 1 ˜ G is bounded by a fixed constant.

Original: Consider the absolute value of the singular values of ˜ H ; we assume that the smallestabsolute value is strictly positive and bounded below by ζ .
Revised: Consider the absolute value of the singular values of ˜ H ; we assume that the smallest absolute value is strictly positive and bounded below by ζ .

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose f : R n → R d and g : R d → R n are functions such that g and a = f ◦ gcan be approximated by polynomials on entire R d .
Revised: Suppose f : R n → R d and g : R d → R n are functions such that g and a = f ◦ g can be approximated by polynomials on entire R d .

Original: If Z = [ − z max , z max ] d , where z max is sufficientlylarge, and Assumption 1,3, and 5 hold, then the polynomial approximation of a (recall ˆ z = a ( z ) )corresponding to solutions of approximate reconstruction identity in equation (13) is approximatelylinear, i.e., the norms of the weights on higher order terms is sufficiently small.
Revised: If Z = [ − z max , z max ] d , where z max is sufficiently large, and Assumption 1,3, and 5 hold, then the polynomial approximation of a (recall ˆ z = a ( z ) ) corresponding to solutions of approximate reconstruction identity in equation (13) is approximately linear, i.e., the norms of the weights on higher order terms is sufficiently small.

Original: Proof sketch We start by restating the approximate reconstruction identity.
Revised: Proof sketch We start by restating the approximate reconstruction identity.

Original: We use the fact that g canbe approximated with a polynomial of say degree p to simplify the identity below.
Revised: We use the fact that g can be approximated with a polynomial of say degree p to simplify the identity below.

Original: [Equation]
Revised: [Equation]

Original: Since H is full rank
Revised: Since H is full rank

Original: [Equation]
Revised: [Equation]

Original: we select rows of H such that ˜ H is square and invertible.
Revised: we select rows of H such that ˜ H is square and invertible.

Original: The correspondingselection for G is denoted as ˜ G .
Revised: The corresponding selection for G is denoted as ˜ G .

Original: We write the identity in terms of these matrices as follows.
Revised: We write the identity in terms of these matrices as follows.

Original: [Equation]
Revised: [Equation]

Original: where | σ min ( ˜ H ) | is the singular value with smallest absolute value corresponding to the matrix ˜ H .
Revised: where | σ min ( ˜ H ) | is the singular value with smallest absolute value corresponding to the matrix ˜ H .

Original: Now we write that the polynomial that approximates ˆ z i
Revised: Now we write that the polynomial that approximates ˆ z i

Original: = a i ( z ) as follows.
Revised: = a i ( z ) as follows.

Original: [Equation]
Revised: [Equation]

Original: From Assumption 5 we know that
Revised: From Assumption 5 we know that

Original: ˆ z i ≥ γη , where γ > 2 .
Revised: ˆ z i ≥ γη , where γ > 2 .

Original: It follows from the above equation that
Revised: It follows from the above equation that

Original: [Equation]
Revised: [Equation]

Original: For ˆ z i ≥ γη , let us track how ˆ z pi grows.
Revised: For ˆ z i ≥ γη , let us track how ˆ z pi grows.

Original: [Equation]
Revised: [Equation]

Original: We consider z =
Revised: We consider z =

Original: [ z max , · · · , z max ] .
Revised: [ z max , · · · , z max ] .

Original: Consider of the terms θ ij z k max inside the polynomial in the RHSabove.
Revised: Consider of the terms θ ij z k max inside the polynomial in the RHS above.

Original: We assume all components of θ are positive.
Revised: We assume all components of θ are positive.

Original: Suppose θ ij ≥ 1 z k − κ − 1 max , where κ ∈ (0 , 1) , then the RHS grows at least z (1+ κ ) p max .
Revised: Suppose θ ij ≥ 1 z k − κ − 1 max , where κ ∈ (0 , 1) , then the RHS grows at least z (1+ κ ) p max .

Original: From equation (16), ˆ z pi is very close to degree p polynomial in z .
Revised: From equation (16), ˆ z pi is very close to degree p polynomial in z .

Original: Under the assumption that the terms in ˜ H − 1 ˜ G are bounded by a constant the polynomial of degree pgrows at z p max .
Revised: Under the assumption that the terms in ˜ H − 1 ˜ G are bounded by a constant the polynomial of degree p grows at z p max .

Original: The above expression in equation (20) grows at z (1+ κ ) p max (cid:0) γ − 2 γ − 1 (cid:1) p .
Revised: The above expression in equation (20) grows at z (1+ κ ) p max [Equation] p .

Original: The difference ingrowth rates the equation (16) is an increasing function of z max for ranges where z max is sufficientlylarge.
Revised: The difference in growth rates the equation (16) is an increasing function of z max for ranges where z max is sufficiently large.

Original: Therefore, the reconstruction identity in equation (16) cannot be satisfied for points in theneighborhood of z =
Revised: Therefore, the reconstruction identity in equation (16) cannot be satisfied for points in the neighborhood of z =

Original: [ z max , · · · , z max ] .
Revised: [ z max , · · · , z max ] .

Original: Therefore, θ ij < [Equation] 1 max .
Revised: Therefore, θ ij < 1 z k − κ − 1 max .

Original: We can consider other vetries ofthe hypercube Z and conclude that | θ ij | <
Revised: We can consider other vetries of the hypercube Z and conclude that | θ ij | <

Original: [Figure]
Revised: [Equation]

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose the observational data is generated from equation (1) and interventional datais generated from equation (5) .
Revised: Suppose the observational data is generated from equation (1) and interventional data is generated from equation (5) .

Original: If t Assumption 1- 4 are satisfied, then the solution to equation (6)and (7) satisfy ˆ z k
Revised: If t Assumption 1- 4 are satisfied, then the solution to equation (6) and (7) satisfy ˆ z k

Original: = az
Revised: = az

Original: i + b and ˆ z − k =
Revised: i + b and ˆ z − k =

Original: Az − i + c , where ˆ z − k denotes the estimate of the latentsother than k and z − i denotes the vector of true latents other than i .
Revised: Az − i + c , where ˆ z − k denotes the estimate of the latents other than k and z − i denotes the vector of true latents other than i .

Original: Proof.
Revised: Proof.

Original: First note that since Assumptions 1-3 hold, we can continue to use the result from Theorem1.
Revised: First note that since Assumptions 1-3 hold, we can continue to use the result from Theorem 1.

Original: From Theorem 1, it follows that the estimated latents ˆ z are an affine function of the true z .ˆ
Revised: From Theorem 1, it follows that the estimated latents ˆ z are an affine function of the true z .

Original:  z i = a t z + b, ∀ z ∈ Z ∪Z i .
Revised: ˆ z i = a t z + b, ∀ z ∈ Z ∪Z i .

Original: From the constraint in equation (7) it follows that for all z ∈ Z i , ˆ z i = z † .
Revised: From the constraint in equation (7) it follows that for all z ∈ Z i , ˆ z i = z † .

Original: We write z ∈
Revised: We write z ∈

Original: Z i as [ z ∗ , z − i ] .
Revised: Z i as [ z ∗ , z − i ] .

Original: We consider a z ∈
Revised: We consider a z ∈

Original: Z i such that z − i is in the interior of ˜
Revised: Z i such that z − i is in the interior of ˜

Original: Z i .
Revised: Z i .

Original: We canwrite ˆ z i = a i z ∗ + a t − i z − i + b , where a − i is the vector of the values of coefficients in a other thanthe coefficient of i th dimension, a i is i th component of a , z − i is the vector of values in z other thanz i .
Revised: We can write ˆ z i = a i z ∗ + a t − i z − i + b , where a − i is the vector of the values of coefficients in a other than the coefficient of i th dimension, a i is i th component of a , z − i is the vector of values in z other than z i .

Original: We use these expressions to carry out the following simplification.
Revised: We use these expressions to carry out the following simplification.

Original: [Equation]
Revised: [Equation]

Original: Consider another data point z ′ ∈
Revised: Consider another data point z ′ ∈

Original: Z i from the same interventional distribution such that z ′ − i =
Revised: Z i from the same interventional distribution such that z ′ − i =

Original: z − i + θe j , where e j is vector with one in j th coordinate and zero everywhere else.
Revised: z − i + θe j , where e j is vector with one in j th coordinate and zero everywhere else.

Original: Since the point isfrom the same interventional distribution z ′ i =
Revised: Since the point is from the same interventional distribution z ′ i =

Original: z ∗ .
Revised: z ∗ .

Original: From Assumption 4, we know that there exists asmall enough θ such that z ′ −
Revised: From Assumption 4, we know that there exists a small enough θ such that z ′ −

Original: i is in Z − i .
Revised: i is in Z − i .

Original: For z ′ − i we have
Revised: For z ′ − i we have

Original: [Equation]
Revised: [Equation]

Original: We take a difference of the two equations (21) and (22) to get
Revised: We take a difference of the two equations (21) and (22) to get

Original: [Equation]
Revised: [Equation]

Original: From the above we get that a j = 0 .
Revised: From the above we get that a j = 0 .

Original: We can repeat the above argument for all j and get thata − i = 0 .
Revised: We can repeat the above argument for all j and get that a − i = 0 .

Original: B Related Works
Revised: B Related Works

Original: Our work relates to multiple threads of work in identifiable representation learning.
Revised: Our work relates to multiple threads of work in identifiable representation learning.

Original: We discuss themin groups based on the type of information they leverage for representation identification.
Revised: We discuss them in groups based on the type of information they leverage for representation identification.

Original: • Time-series datasets.
Revised: • Time-series datasets.

Original: Several works have leveraged the structure of latent variables in time-series data to achieve identification.
Revised: Several works have leveraged the structure of latent variables in time-series data to achieve identification.

Original: The canonical data generating process in these works follows x t ← g ( z t ) and the latents z t evolve under a structured time-series.
Revised: The canonical data generating process in these works follows x t ← g ( z t ) and the latents z t evolve under a structured time-series.

Original: Early works in this area consider non-stationary evolution of latents (i
Revised: Early works in this area consider non-stationary evolution of latents (i

Original: e. assuming no dependence between time frames) (Hyvarinen and Morioka, 2016) and then came the models that considered stationary Markovian evolution (Hyvarinen and Morioka, 2017).
Revised: e. assuming no dependence between time frames) (Hyvarinen and Morioka, 2016) and then came the models that considered stationary Markovian evolution (Hyvarinen and Morioka, 2017).

Original: In recent years, these models have been generalized significantly in works like Lachapelle et al.
Revised: In recent years, these models have been generalized significantly in works like Lachapelle et al.

Original:  Ahuja et al. ); Lippe et al.
Revised: ; Ahuja et al. (2021); Lippe et al.

Original: • Contrastive observation-based datasets.
Revised: • Contrastive observation-based datasets.

Original: In another family of works includingZimmermann et al.
Revised: In another family of works including Zimmermann et al.

Original: ; Von Kügelgen et al.
Revised: ; Von Kügelgen et al.

Original: ; Brehmer et al.
Revised: ; Brehmer et al.

Original: Locatelloet al.
Revised: ; Locatello et al.

Original: (2020), one assumes access to contrastive observation pairs ( x, ˜ x ) .
Revised: (2020), one assumes access to contrastive observation pairs ( x, ˜ x ) .

Original: For instance, animage and its rotated version can serve as contrastive observation pairs ( x, ˜ x ) (Zimmermannet al., 2021).
Revised: For instance, an image and its rotated version can serve as contrastive observation pairs ( x, ˜ x ) (Zimmermann et al., 2021).

Original: In works such as Brehmer et al.
Revised: In works such as Brehmer et al.

Original: (2022), the pair ( x, ˜ x ) corresponds to a datapoint pre and post-intervention on the latents.
Revised: (2022), the pair ( x, ˜ x ) corresponds to a data point pre and post-intervention on the latents.

Original: The data generation process is similar totime-series datasets in several aspects but there are a few key differences, including (a) thepoints ( x, ˜ x ) are not necessarily ordered by time, and (b) there may not exist any causalconnections between the latents associated with x and ˜ x , unlike those in time-series datasets(e.g., Lachapelle et al.
Revised: The data generation process is similar to time-series datasets in several aspects but there are a few key differences, including (a) the points ( x, ˜ x ) are not necessarily ordered by time, and (b) there may not exist any causal connections between the latents associated with x and ˜ x , unlike those in time-series datasets (e.g., Lachapelle et al.

Original: ; Lippe et al.
Revised: ; Lippe et al.

Original: (2022)).• Auxiliary information datasets.
Revised: • Auxiliary information datasets.

Original: In the third line of work (Khemakhem et al., 2020a,b;Ahuja et al., 2022b), one assumes access to the high-dimensional observation x
Revised: In the third line of work (Khemakhem et al., 2020a,b; Ahuja et al., 2022b), one assumes access to the high-dimensional observation x

Original: (e.g., animage) and some auxiliary information u .
Revised: (e.g., an image) and some auxiliary information u .

Original: If u is the label of the image, we obtain standardsupervised learning datasets.
Revised: If u is the label of the image, we obtain standard supervised learning datasets.

Original: If u = ˜ x is the positive pair (e.g., rotation of the image), weobtain contrastive observation-based datasets.
Revised: If u = ˜ x is the positive pair (e.g., rotation of the image), we obtain contrastive observation-based datasets.

Original: If u is the time stamp and previous timeinformation x t − 1 , then we obtain the time-series datasets.
Revised: If u is the time stamp and previous time information x t − 1 , then we obtain the time-series datasets.

Original: This line of work (Khemakhemet al., 2020a) often relies on strong assumptions on the interaction between latents andauxiliary information(e.g., latents are independent conditioned on auxiliary information) toguarantee provable identification.
Revised: This line of work (Khemakhem et al., 2020a) often relies on strong assumptions on the interaction between latents and auxiliary information(e.g., latents are independent conditioned on auxiliary information) to guarantee provable identification.

Original: Finally we contrast our work with the existing works discussed above in terms of the type of information we leverage for representation identification: we leverage interventional knowledge (level-two knowledge in Pearl’s ladder of causation (Bareinboim et al., 2022)) while most existingworks leverage either observational data (level-one knowledge) (e.g. Khemakhem et al., 2020a) orcounterfactual information (level-three knowledge) (e.g. Brehmer et al., 2022).
Revised: Finally we contrast our work with the existing works discussed above in terms of the type of information we leverage for representation identification: we leverage interventional knowledge (level-two knowledge in Pearl’s ladder of causation (Bareinboim et al., 2022)) while most existing works leverage either observational data (level-one knowledge) (e.g. Khemakhem et al., 2020a) or counterfactual information (level-three knowledge) (e.g. Brehmer et al., 2022).

Original: Specifically in thiswork, we focus on studying “to what extent can we identify the latent causal variables if the datacomprises different interventional distributions?”
Revised: Specifically in this work, we focus on studying “to what extent can we identify the latent causal variables if the data comprises different interventional distributions?”

Original: This question about causal representation learningshares the same spirit with causal discovery using interventional data, where we seek to understandhow different interventional distributions help identify the underlying causal graph (Yang et al., 2018);both tasks rely on interventional data though targeting different causal tasks.
Revised: This question about causal representation learning shares the same spirit with causal discovery using interventional data, where we seek to understand how different interventional distributions help identify the underlying causal graph (Yang et al., 2018); both tasks rely on interventional data though targeting different causal tasks.

Original: In contrast to our work, most existing works have leveraged other types of information forrepresentation identification.
Revised: In contrast to our work, most existing works have leveraged other types of information for representation identification.

Original: For example, Brehmer et al.
Revised: For example, Brehmer et al.

Original: (2022) assume that the representationlearner observes a pair of data points both pre- and post-intervention, and the data generationprocess therein requires the noise to be set to the same realization across the pair of points atall the nodes except the intervened nodes.
Revised: (2022) assume that the representation learner observes a pair of data points both pre- and post-intervention, and the data generation process therein requires the noise to be set to the same realization across the pair of points at all the nodes except the intervened nodes.

Original: Therefore, they leverage counterfactual information(level-three knowledge) for representation identification.
Revised: Therefore, they leverage counterfactual information (level-three knowledge) for representation identification.

Original: As another set of examples, Lippe et al.(2022); Lachapelle et al.
Revised: As another set of examples, Lippe et al. ; Lachapelle et al.

Original: (2022) leverage pre- and post-intervention observations in adjacent timeframes to study the causal relationships between the latents.
Revised: (2022) leverage pre- and post-intervention observations in adjacent time frames to study the causal relationships between the latents.

Original: Other works (e.g. Khemakhem et al.(2020a,b); Ahuja et al.
Revised: Other works (e.g. Khemakhem et al. (2020a,b); Ahuja et al.

Original: (2022b)) directly work with observational data, i.e., level-one knowledge,and do not require or take advantage of interventional data.
Revised: (2022b)) directly work with observational data, i.e., level-one knowledge, and do not require or take advantage of interventional data.

Original: However, these works often achieveidentification guarantees by making strong assumptions on the structure of the underlying causalconnections between the latents, relying on observations of auxiliary information such as the label, orcapitalizing on parametric assumptions on the distribution of the latent.
Revised: However, these works often achieve identification guarantees by making strong assumptions on the structure of the underlying causal connections between the latents, relying on observations of auxiliary information such as the label, or capitalizing on parametric assumptions on the distribution of the latent.