Original: P ROPERTIES FROM MECHANISMS : AN EQUIVARIANCE PERSPECTIVE ON IDENTIFIABLE REPRESENTATION LEARNING
Revised: P ROPERTIES FROM MECHANISMS : AN EQUIVARIANCE PERSPECTIVE ON IDENTIFIABLE REPRESENTATION LEARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A key goal of unsupervised representation learning is “inverting” a data generating process to recover its latent properties.
Revised: A key goal of unsupervised representation learning is “inverting” a data generating process to recover its latent properties.

Original: Existing work that provably achieves this goal relies on strong assumptions on relationships between the latent variables (e.g., independence conditional on auxiliary information).
Revised: Existing work that provably achieves this goal relies on strong assumptions on relationships between the latent variables (e.g., independence conditional on auxiliary information).

Original: In this paper, we take a very different perspective on the problem and ask, “Can we instead identify latent properties by leveraging knowledge of the mechanisms that govern their evolution?”
Revised: In this paper, we take a very different perspective on the problem and ask, “Can we instead identify latent properties by leveraging knowledge of the mechanisms that govern their evolution?”

Original: We provide a complete characterization of the sources of non-identiﬁability as we vary knowledge about a set of possible mechanisms.
Revised: We provide a complete characterization of the sources of non-identiﬁability as we vary knowledge about a set of possible mechanisms.

Original: In particular, we prove that if we know the exact mechanisms under which the latent properties evolve, then identiﬁcation can be achieved up to any equivariances that are shared by the underlying mechanisms.
Revised: In particular, we prove that if we know the exact mechanisms under which the latent properties evolve, then identiﬁcation can be achieved up to any equivariances that are shared by the underlying mechanisms.

Original: We generalize this characterization to settings where we only know some hypothesis class over possible mechanisms, as well as settings where the mechanisms are stochastic.
Revised: We generalize this characterization to settings where we only know some hypothesis class over possible mechanisms, as well as settings where the mechanisms are stochastic.

Original: We demonstrate the power of this mechanismbased perspective by showing that we can leverage our results to generalize existing identiﬁable representation learning results.
Revised: We demonstrate the power of this mechanismbased perspective by showing that we can leverage our results to generalize existing identiﬁable representation learning results.

Original: These results suggest that by exploiting inductive biases on mechanisms, it is possible to design a range of new identiﬁable representation learning approaches.
Revised: These results suggest that by exploiting inductive biases on mechanisms, it is possible to design a range of new identiﬁable representation learning approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern unsupervised representation learning techniques can generate images of our world with intricate detail (e.g. Karras et al., 2019;
Revised: Modern unsupervised representation learning techniques can generate images of our world with intricate detail (e.g. Karras et al., 2019;

Original: Song et al., 2020; Razavi et al., 2019) and yet, the latent representations from which these images are generated remain entangled and challenging to interpret (Sch ¨ olkopf et al., 2021;
Revised: Song et al., 2020; Razavi et al., 2019), and yet, the latent representations from which these images are generated remain entangled and challenging to interpret (Sch ¨ olkopf et al., 2021;

Original: Locatello et al., 2019).
Revised: Locatello et al., 2019).

Original: At the same time, the success of pre-trained transformers (Devlin et al., 2018; Brown et al., 2020) shows that advances in our ability to disentangle the underlying generative factors can lead to dramatic improvements in the sample complexity of downstream supervised tasks (Bengio & LeCun, 2007; Bengio et al., 2013).
Revised: At the same time, the success of pre-trained transformers (Radford et al., 2021; Brown et al., 2020) shows that advances in our ability to disentangle the underlying generative factors can lead to dramatic improvements in the sample complexity of downstream supervised tasks (Bengio & LeCun, 2007; Bengio et al., 2013).

Original: In order to consistently replicate this success, we need methods that can reliably invert the data generating process.
Revised: In order to consistently replicate this success, we need methods that can reliably invert the data generating process.

Original: This is a challenging task because unsupervised representation learning with independent and identically distributed (IID) data is hopelessly unidentiﬁed : even in the nice case in which observations are generated via some p ( x | z ) , there exists an inﬁnitely large class of possible latent distributions ˜ p
Revised: This is a challenging task because unsupervised representation learning with independent and identically distributed (IID) data is hopelessly unidentiﬁed : even in the nice case in which observations are generated via some p ( x | z ) , there exists an inﬁnitely large class of possible latent distributions ˜ p

Original: ( z | x ) that are consistent with the observed marginal distribution, p ( x ) , and only one of them corresponds to the true latent distribution p ( z | x ) (Locatello et al., 2019; Khemakhem et al., 2020a).
Revised: ( z | x ) that are consistent with the observed marginal distribution, p ( x ) , and only one of them corresponds to the true latent distribution p ( z | x ) (Locatello et al., 2019; Khemakhem et al., 2020a).

Original: If we want to build systems which are able to provably identify 1 the true latent variables z that generated our observed data x = g ( z ) for some observation model g , then we need to exploit structural assumptions that constrain this set of possible distributions.
Revised: If we want to build systems which are able to provably identify 1 the true latent variables z that generated our observed data x = g ( z ) for some observation model g , then we need to exploit structural assumptions that constrain this set of possible distributions.

Original: Most of the prior work that can provide such guarantees was developed in the independent component analysis (ICA) literature.
Revised: Most of the prior work that can provide such guarantees was developed in the independent component analysis (ICA) literature.

Original: The key ICA assumption is that the latent factors are (conditionally) independent and non-Gaussian.
Revised: The key ICA assumption is that the latent factors are (conditionally) independent and non-Gaussian.

Original: Then, if the observation model, g : Z → X is linear and invertible, z = g − 1 ( x ) is identiﬁed (Comon, 1994), and for nonlinear g there are a number of recent approaches that leverage non-stationarity in the distributions over z to identify g − 1 (Hyvarinen & Morioka, 2016; 2017; Hyvarinen et al., 2019;

Original: Encoder
Revised: Encoder

Original: Decoder
Revised: Decoder

Original: Mechanism Latent representation m 1 ( z 1 ) m 2 ( z 2 ) ˜
Revised: Mechanism Latent representation m 1 ( z 1 ) m 2 ( z 2 ) ˜

Original: g ∘ m 1 ∘ ˜ g −1 ( x 1 ) g −1 ( x 1 ) g ( z 1 ) g −1 ( x 2 )
Revised: g ∘ m 1 ∘ ˜ g −1 ( x 1 ) g −1 ( x 1 ) g ( z 1 ) g −1 ( x 2 )

Original: g ( z 2 ) g ( z 3 ) m 2 (˜ z 2 ) ˜ g −11 ( x 2 ) ˜ g
Revised: g ( z 2 ) g ( z 3 ) m 2 (˜ z 2 ) ˜ g −11 ( x 2 ) ˜ g

Original: 1 ∘ m ∘ (˜ z 2 ) = xm 2 (˜ z 2 )
Revised: 1 ∘ m ∘ (˜ z 2 ) = xm 2 (˜ z 2 )

Original: ˜
Revised: ˜

Original: g −12 ( x 2 ) ˜ g
Revised: g −12 ( x 2 ) ˜ g

Original: 2 ∘ m ∘ (˜ z 2 ) ≠ x 3 Observation identity xxx
Revised: 2 ∘ m ∘ (˜ z 2 ) ≠ x 3 Observation identity x [Figure] x

Revised: Then, if the observation model, g

Revised: Z → X is linear and invertible, z = g − 1 ( x ) is identiﬁed (Comon, 1994), and for nonlinear g there are a number of recent approaches that leverage non-stationarity in the distributions over z to identify g − 1 (Hyvarinen & Morioka, 2016; 2017; Hyvarinen et al., 2019;

Original: Khemakhem et al., 2020a).
Revised: Khemakhem et al., 2020a).

Original: These results give a tantalizing demonstration that representation learning with identiﬁcation guarantees is possible, but the requirement that the latent factors are statistically (conditionally) independent is limiting 2 (Higgins et al., 2018; Sch ¨ olkopf et al., 2021).
Revised: These results give a tantalizing demonstration that representation learning with identiﬁcation guarantees is possible, but the requirement that the latent factors are statistically (conditionally) independent is limiting 2 (Higgins et al., 2018; Sch¨olkopf et al., 2021).

Original: In this paper we take a very different approach.
Revised: In this paper we take a very different approach.

Original: We study how the mechanisms that govern an environment’s evolution constrain the set of possible latent representations that are consistent with the data.
Revised: We study how the mechanisms that govern an environment’s evolution constrain the set of possible latent representations that are consistent with the data.

Original: As a simple concrete example, consider the bouncing ball environment shown in Figure 1.
Revised: As a simple concrete example, consider the bouncing ball environment shown in Figure 1.

Original: The latent state can be completely described by a vector, z , containing the position, velocity and acceleration of the balls 3 , and given this latent state, the images, x , shown in Figure 1, are produced via some rendering engine g : Z → X .
Revised: The latent state can be completely described by a vector, z , containing the position, velocity and acceleration of the balls 3 , and given this latent state, the images, x , shown in Figure 1, are produced via some rendering engine g : Z → X .

Original: Our task is to leverage sequences of observations x 1 , . . .
Revised: Our task is to leverage sequences of observations x 1 , . . .

Original: , x T and knowledge of m to recover z 1 , . .
Revised: , x T and knowledge of m to recover z 1 , . .

Original: . , z T .
Revised: . , z T .

Original: If we can show that this task has a unique solution that is consistent with the observations and mechanisms, then the problem is identiﬁed.
Revised: If we can show that this task has a unique solution that is consistent with the observations and mechanisms, then the problem is identiﬁed.

Original: Our main result is that when we know the true mechanism, m , the system is identiﬁed up to any equivariances of the mechanism; or equivalently, in Section 2.2 we prove that we can identify z up to any invertible transformation a : Z → Z that commutes with m , such that the composition m ◦ a = a ◦ m .
Revised: Our main result is that when we know the true mechanism, m , the system is identiﬁed up to any equivariances of the mechanism; or equivalently, in Section 2.2 we prove that we can identify z up to any invertible transformation a : Z → Z that commutes with m , such that the composition m ◦ a = a ◦ m .

Original: For example, in the bouncing balls environment shown in Figure 1, the laws of physics are equivariant with respect to your choice of units of measurement—changing from representing z in meters to inches leaves the output of the mechanism unchanged up to a corresponding unit change— and hence we can only hope to identify z up to some scaling function a ( z ) which corresponds to an arbitrary choice of units of measurement.
Revised: For example, in the bouncing balls environment shown in Figure 1, the laws of physics are equivariant with respect to your choice of units of measurement—changing from representing z in meters to inches leaves the output of the mechanism unchanged up to a corresponding unit change— and hence we can only hope to identify z up to some scaling function a ( z ) which corresponds to an arbitrary choice of units of measurement.

Original: Interestingly, when the environment evolves according to multiple known mechanisms, the sources of non-identiﬁability are even further constrained: such a system is identiﬁed up to equivariances that are shared by all n mechanisms.
Revised: Interestingly, when the environment evolves according to multiple known mechanisms, the sources of non-identiﬁability are even further constrained: such a system is identiﬁed up to equivariances that are shared by all n mechanisms.

Original: Perfectly knowing the true mechanisms and when they are applied is unlikely, but in Section 2.3 we show that we can relax that assumption to a setting where we instead know a hypothesis class of possible mechanisms that could have been applied.
Revised: Perfectly knowing the true mechanisms and when they are applied is unlikely, but in Section 2.3 we show that we can relax that assumption to a setting where we instead know a hypothesis class of possible mechanisms that could have been applied.

Original: This weaker assumption leads to an additional source of non-identiﬁability: the mechanisms in our hypothesis class can imitate each other if there exists an invertible transformation a such that for any two mechanisms m 1 and m 2 in our hypothesis class, m 2 = a − 1 ◦ m 1 ◦ a .
Revised: This weaker assumption leads to an additional source of non-identiﬁability: the mechanisms in our hypothesis class can imitate each other if there exists an invertible transformation a such that for any two mechanisms m 1 and m 2 in our hypothesis class, m 2 = a − 1 ◦ m 1 ◦ a .

Original: For example, if we are in a setting where our hypothesis class includes both a product mechanism, m 1 ( z ) = (cid:81)
Revised: For example, if we are in a setting where our hypothesis class includes both a product mechanism, m 1 ( z ) = (cid:81)

Original: i z i , and a sum mechanism, m 2 ( z ) =
Revised: i z i , and a sum mechanism, m 2 ( z ) =

Original: (cid:80)
Revised: (cid:80)

Original: i z i , then m 1 can imitate m 2 if a ( z ) =
Revised: i z i , then m 1 can imitate m 2 if a ( z ) =

Original: exp( z ) , since (cid:80) i z i =
Revised: exp( z ) , since (cid:80) i z i =

Original: log
Revised: log

Original: (cid:81) i exp( z i )) .
Revised: (cid:81) i exp( z i )) .

Original: As before, this result is complete, in the sense that these two sources of non-identiﬁability—equivariance and imitation—are the only sources of non-identiﬁability in such systems.
Revised: As before, this result is complete, in the sense that these two sources of non-identiﬁability—equivariance and imitation—are the only sources of non-identiﬁability in such systems.

Original: This gives us a natural way of thinking about the way knowledge of deterministic mechanisms constrains a representation learning task: with complete knowledge, the only source of non-identiﬁability is any equivariances inherent in the mechanism, but by allowing a hypothesis class of possible mechanisms, we introduce potential non-identiﬁability via imitation.
Revised: This gives us a natural way of thinking about the way knowledge of deterministic mechanisms constrains a representation learning task: with complete knowledge, the only source of non-identiﬁability is any equivariances inherent in the mechanism, but by allowing a hypothesis class of possible mechanisms, we introduce potential non-identiﬁability via imitation.

Original: That said, the relationship between the size of the hypothesis class and the size of the set of a ’s that commute via imitation is not necessarily monotonic: for environments governed by multiple mechanisms, a larger hypothesis class can lead to fewer imitations.
Revised: That said, the relationship between the size of the hypothesis class and the size of the set of a ’s that commute via imitation is not necessarily monotonic: for environments governed by multiple mechanisms, a larger hypothesis class can lead to fewer imitations.

Original: Section 3, shows that we can derive analogous results for stochastic mechanisms, m ( z, U ) , where the mechanism deﬁnes a conditional distribution p ( z t +1 | z t ) .
Revised: Section 3, shows that we can derive analogous results for stochastic mechanisms, m ( z, U ) , where the mechanism deﬁnes a conditional distribution p ( z t +1 | z t ) .

Original: This generalization gives us a way of comparing our mechanism-based perspective with existing identiﬁability results.
Revised: This generalization gives us a way of comparing our mechanism-based perspective with existing identiﬁability results.

Original: We demonstrate this in Section 4 by showing that it is possible to view the distributional assumptions made in Klindt et al.
Revised: We demonstrate this in Section 4 by showing that it is possible to view the distributional assumptions made in Klindt et al.

Original: (2020) as a particular choice of mechanism, and by doing so, we can leverage our theory to give alternative proofs of these results.
Revised: (2020) as a particular choice of mechanism, and by doing so, we can leverage our theory to give alternative proofs of these results.

Original: This strategy required weaker distributional assumptions, thereby generalizing their result.
Revised: This strategy required weaker distributional assumptions, thereby generalizing their result.

Original: Finally, we give a mechanism-based perspective on the related work in
Revised: Finally, we give a mechanism-based perspective on the related work in

Original: Section 5 and Section 6 concludes with a discussion of the open problems that need to be addressed in order to reliably leverage this approach in practice.
Revised: Section 5 and Section 6 concludes with a discussion of the open problems that need to be addressed in order to reliably leverage this approach in practice.

Original:  M ECHANISM BASED IDENTIFICATION 2. D ATA GENERATION PROCESS
Revised:  M ECHANISM BASED IDENTIFICATION 2. D ATA GENERATION PROCESS

Original: The state of the system at time t ∈ { 1 , · · · , T } is given by z t ∈ Z ⊆ R d .
Revised: The state of the system at time t ∈ { 1 , · · · , T } is given by z t ∈ Z ⊆ R d .

Original: At each time t , we observe g ( z t ) =
Revised: At each time t , we observe g ( z t ) =

Original: x t ∈ X ⊆ R n , which is some transformation of the latent z t .
Revised: x t ∈ X ⊆ R n , which is some transformation of the latent z t .

Original: We can think of g : Z → X as a function that transforms the (typically low dimensional) state variables to the (typically high dimensional) observed variables; for example, in the bouncing ball environment described in the introduction, g is the rendering engine that produces the images shown in Figure 1.
Revised: We can think of g : Z → X as a function that transforms the (typically low dimensional) state variables to the (typically high dimensional) observed variables; for example, in the bouncing ball environment described in the introduction, g is the rendering engine that produces the images shown in Figure 1.

Original: We assume g is injective 4 with respect to R n —i.e. g ( z 1 )
Revised: We assume g is injective 4 with respect to R n —i.e. g ( z 1 )

Original: =
Revised: =

Original: g ( z 2 ) implies z 1 = z 2 ; or equivalently, any change to the underlying state, z , is reﬂected in some pixel-level change to the observation x —and we make g bijective by restricting its inverse g − 1 to any x on the data manifold which we denote X (i.e. X is the image of g ).
Revised: g ( z 2 ) implies z 1 = z 2 ; or equivalently, any change to the underlying state, z , is reﬂected in some pixel-level change to the observation x —and we make g bijective by restricting its inverse g − 1 to any x on the data manifold which we denote X (i.e. X is the image of g ).

Original: The state transition from time t to t + 1 is governed by a mechanism m t : Z → Z .
Revised: The state transition from time t to t + 1 is governed by a mechanism m t : Z → Z .

Original: There may be multiple mechanisms in a given environment.
Revised: There may be multiple mechanisms in a given environment.

Original: For example in Figure 1 the transition from z 1 to z 2 does not involve any collisions so the state evolves according to Newton’s ﬁrst law of motion (Newton, 1687); the transition from z 2 to z 3 involves a collision between two of the balls that is described by Newton’s third law.
Revised: For example in Figure 1 the transition from z 1 to z 2 does not involve any collisions so the state evolves according to Newton’s ﬁrst law of motion (Newton, 1687); the transition from z 2 to z 3 involves a collision between two of the balls that is described by Newton’s third law.

Original: Together m t and g describe the data-generation from time t to t
Revised: Together m t and g describe the data-generation from time t to t

Original: + 1 as follows,
Revised: + 1 as follows,

Original: [Equation]
Revised: [Equation]

Original: If we ﬁx the initial conditions, z 0 , this data generation process is deterministic.
Revised: If we ﬁx the initial conditions, z 0 , this data generation process is deterministic.

Original: In Section 2.2, we provide results when the underlying mechanism is known and in Section 2.3, we we extend those results to the case when the mechanisms are not known.
Revised: In Section 2.2, we provide results when the underlying mechanism is known and in Section 2.3, we we extend those results to the case when the mechanisms are not known.

Original: In Section 3, we extend our results to stochastic mechanisms, m ( Z t , U ) that take samples from some distribution U ∼ Uniform (0 , 1) as input.
Revised: In Section 3, we extend our results to stochastic mechanisms, m ( Z t , U ) that take samples from some distribution U ∼ Uniform (0 , 1) as input.

Original: These stochastic mechanisms can represent any conditional distribution P ( Z t +1 | Z t ) over states.
Revised: These stochastic mechanisms can represent any conditional distribution P ( Z t +1 | Z t ) (Austin (2015, Lemma 3.1))

Original: 2. I DENTIFYING ENCODERS WHEN THE UNDERLYING MECHANISM IS KNOWN
Revised: 2. I DENTIFYING ENCODERS WHEN THE UNDERLYING MECHANISM IS KNOWN

Original: We begin in the simplest version of the system described by equation (1): assume that at each time t the same mechanism m : Z → Z is used, and we know this mechanism.
Revised: We begin in the simplest version of the system described by equation (1): assume that at each time t the same mechanism m : Z → Z is used, and we know this mechanism.

Original: From these assumptions we can derive an identity that describes how the observations x t and x t +1 are related,
Revised: From these assumptions we can derive an identity that describes how the observations x t and x t +1 are related,

Original: [Equation]
Revised: [Equation]

Original: It may be helpful to think of this identity as describing an autoencoder where we require that the encoder g − 1 ( x t ) inverts the data generating process to produce some latent z t from x t , and that the decoder has to reproduce x t +1 from m ( z t ) ; i.e. the representation transformed by the mechanism.
Revised: It may be helpful to think of this identity as describing an autoencoder where we require that the encoder g − 1 ( x t ) inverts the data generating process to produce some latent z t from x t , and that the decoder has to reproduce x t +1 from m ( z t ) ; i.e. the representation transformed by the mechanism.

Original: Importantly, this identity describes the true data generating process, rather than some model of it.
Revised: Importantly, this identity describes the true data generating process, rather than some model of it.

Original: Our hypothesis class over the possible encoder / decoder functions is the set, G , of all bijective functions from Z → X .
Revised: Our hypothesis class over the possible encoder / decoder functions is the set, G , of all bijective functions from Z → X .

Original: By assuming that bijectivity we are essentially assuming that the reconstuction task is solved: 6 G is the set of all autoencoders that perfectly reproduce the data, such that for any x on the data manifold X , and any encoder / decoder pair ( ˜ g − 1 , ˜ g ) with ˜ g ∈ G , their composition is the identity function, x = ˜ g ◦
Revised: By assuming that bijectivity we are essentially assuming that the reconstuction task is solved: 5 G is the set of all autoencoders that perfectly reproduce the data, such that for any x on the data manifold X , and any encoder / decoder pair ( ˜ g − 1 , ˜ g ) with ˜ g ∈ G , their composition is the identity function, x = ˜ g ◦

Original: ˜
Revised: ˜

Original: g − 1 ◦ x .
Revised: g − 1 ◦ x .

Original: Because we assumed that the true g is bijective, we know that it is in our search space, G .
Revised: Because we assumed that the true g is bijective, we know that it is in our search space, G .

Original: We can constrain this set using our knowledge of the mechanism by only considering solutions that also satisfy the identity given in equation 2, such that for every pair of observations ( x t , x t +1 ) , an analogous identity holds,
Revised: We can constrain this set using our knowledge of the mechanism by only considering solutions that also satisfy the identity given in equation 2, such that for every pair of observations ( x t , x t +1 ) , an analogous identity holds,

Original: [Equation]
Revised: [Equation]

Original: Now suppose that we have access to observations x t from the entire set X , then the above identities have to hold for all x t , ∈ X with corresponding x t +1 from equation 2, so we can conclude that the following functions are equal,
Revised: Now suppose that we have access to observations x t from the entire set X , then the above identities have to hold for all x t , ∈ X with corresponding x t +1 from equation 2, so we can conclude that the following functions are equal,

Original: [Equation]
Revised: [Equation]

Original: This relationship will hold for any of the possible decoders ˜ g
Revised: This relationship will hold for any of the possible decoders ˜ g

Original: (and corresponding encoders ˜ g − 1 ) that are observationally equivalent given our assumptions.
Revised: (and corresponding encoders ˜ g − 1 ) that are observationally equivalent given our assumptions.

Original: We denote the set of all such decoders,
Revised: We denote the set of all such decoders,

Original: G id = { ˜ g | ˜ g ∈ G , g ◦ m ◦ g − 1 = ˜
Revised: G id = { ˜ g | ˜ g ∈ G , g ◦ m ◦ g − 1 = ˜

Original: g ◦ m ◦ ˜ g − 1 } .
Revised: g ◦ m ◦ ˜ g − 1 } .

Original: If G id = { g } then we have shown that the problem is exactly identiﬁed, which means that if we manage to ﬁnd a ˜ g that satisﬁes equation 2, then ˜ g
Revised: If G id = { g } then we have shown that the problem is exactly identiﬁed, which means that if we manage to ﬁnd a ˜ g that satisﬁes equation 2, then ˜ g

Original: = g ; but if G id , also includes other functions
Revised: = g ; but if G id , also includes other functions

Original: ˜ g (cid:54) =
Revised: ˜ g (cid:54) =

Original: g , then these functions are the sources of non-identiﬁability.
Revised: g , then these functions are the sources of non-identiﬁability.

Original: Equivariant mechanisms To see an example of a setting where the problem is not exactly identiﬁed, consider a mechanism which is equivariant with respect to some bijective transformation a : Z → Z .
Revised: Equivariant mechanisms To see an example of a setting where the problem is not exactly identiﬁed, consider a mechanism which is equivariant with respect to some bijective transformation a : Z → Z .

Original: A mechanism m is said to be equivariant w.r.t a if a ◦ m = m ◦ a .
Revised: A mechanism m is said to be equivariant w.r.t a if a ◦ m = m ◦ a .

Original: For example, a mechanism derived from Newtonian mechanics may be equivariant with respect to your choice of units of measurement, such that m ( cz ) =
Revised: For example, a mechanism derived from Newtonian mechanics may be equivariant with respect to your choice of units of measurement, such that m ( cz ) =

Original: cm ( z ) for some scaling constant c .
Revised: cm ( z ) for some scaling constant c .

Original: Similarly, if a mechanism transforms sets of items, any permutation of z would lead to a corresponding permutation of the mechanism’s output.
Revised: Similarly, if a mechanism transforms sets of items, any permutation of z would lead to a corresponding permutation of the mechanism’s output.

Original: If we have a mechanism that is equivariant with respect to some transformation a (where a is not identity map), that implies that there exists a function ˜ g (cid:54)
Revised: If we have a mechanism that is equivariant with respect to some transformation a (where a is not identity map), that implies that there exists a function ˜ g (cid:54)

Original: =
Revised: =

Original: g in G id , so the problem is not exactly identiﬁed.
Revised: g in G id , so the problem is not exactly identiﬁed.

Original: We can see this as follows,
Revised: We can see this as follows,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality uses the fact that a − 1 ◦ a is the identity function, the second applies the deﬁnition of equivariance, and the ﬁnal equality deﬁnes ˜ g :=
Revised: where the ﬁrst equality uses the fact that a − 1 ◦ a is the identity function, the second applies the deﬁnition of equivariance, and the ﬁnal equality deﬁnes ˜ g :=

Original: g ◦ a − 1 and ˜ g − 1 :
Revised: g ◦ a − 1 and ˜ g − 1 :

Original: = a ◦ g − 1 .
Revised: = a ◦ g − 1 .

Original: This shows that if the mechanism is equivariant, an encoder, ˜ g − 1 , can output a transformed ˜ z
Revised: This shows that if the mechanism is equivariant, an encoder, ˜ g − 1 , can output a transformed ˜ z

Original: = a ( z ) , and the decoder, ˜ g , inverts this transformation before producing its output, thereby leaving the observed variables, x , unchanged.
Revised: = a ( z ) , and the decoder, ˜ g , inverts this transformation before producing its output, thereby leaving the observed variables, x , unchanged.

Original: If we denote the set of all equivariances of the mechanism
Revised: If we denote the set of all equivariances of the mechanism

Original: E = { a | a is a bijection , a ◦ m = m ◦ a } , then we can deﬁne the set of all such sources of non-identiﬁcation as, G eq
Revised: E = { a | a is a bijection , a ◦ m = m ◦ a } , then we can deﬁne the set of all such sources of non-identiﬁcation as, G eq

Original: =
Revised: =

Original: { ˜ g | ˜ g
Revised: { ˜ g | ˜ g

Original: = g ◦ a − 1 , a ∈ E} .
Revised: = g ◦ a − 1 , a ∈ E} .

Original: This is a natural source of non-identiﬁcation: given that we are relying on the mechanism to constrain the encoder ˜ g − 1 , it is unsurprising we cannot prevent transformations that are not affected by the mechanism.
Revised: This is a natural source of non-identiﬁcation: given that we are relying on the mechanism to constrain the encoder ˜ g − 1 , it is unsurprising we cannot prevent transformations that are not affected by the mechanism.

Original: The more interesting observation, which we will show below, is that this set is the only source of non-identiﬁcation when the mechanism is known, and hence we recover the true z up to equivariances in the mechanism.
Revised: The more interesting observation, which we will show below, is that this set is the only source of non-identiﬁcation when the mechanism is known, and hence we recover the true z up to equivariances in the mechanism.

Original: To state this theorem, we ﬁrst deﬁne the notion of identiﬁability up to an equivalence class deﬁned by a family of bijections A , where a ∈ A is a map a : Z → Z .
Revised: To state this theorem, we ﬁrst deﬁne the notion of identiﬁability up to an equivalence class deﬁned by a family of bijections A , where a ∈ A is a map a : Z → Z .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Identiﬁability up to A .
Revised: Identiﬁability up to A .

Original: If the learned encoder ˜ g − 1 and the true encoder g − 1 are related by some bijection a ∈ A , such that ˜ g − 1 = a ◦ g − 1 (or equivalently ˜
Revised: If the learned encoder ˜ g − 1 and the true encoder g − 1 are related by some bijection a ∈ A , such that ˜ g − 1 = a ◦ g − 1 (or equivalently ˜

Original: g = g ◦ a − 1 ), then ˜ g − 1 is said to learn g − 1 up to bijections in A .
Revised: g = g ◦ a − 1 ), then ˜ g − 1 is said to learn g − 1 up to bijections in A .

Original: We denote this ˜ g − 1 ∼ A g − 1 .
Revised: We denote this ˜ g − 1 ∼ A g − 1 .

Original: Suppose, for example, A is a family of a permutations.
Revised: Suppose, for example, A is a family of a permutations.

Original: Identiﬁability up to A implies that the true latent variables will be recovered, but that they would not necessarily be ordered in the same way as they were in the original data generation process.
Revised: Identiﬁability up to A implies that the true latent variables will be recovered, but that they would not necessarily be ordered in the same way as they were in the original data generation process.

Original: In this setting where the mechanism, m , is known, the following theorem shows that A is just E , the set of equivariances of m .
Revised: In this setting where the mechanism, m , is known, the following theorem shows that A is just E , the set of equivariances of m .

Original: Theorem 1.
Revised: Theorem 1.

Original: If the data generation process follows equation 2, the encoders that solve the observation identity in equation 4 identify true encoder g − 1 up to equivariances of m ( ˜ g − 1 ∼ E g − 1 ).
Revised: If the data generation process follows equation 2, the encoders that solve the observation identity in equation 4 identify true encoder g − 1 up to equivariances of m ( ˜ g − 1 ∼ E g − 1 ).

Original: To prove the above theorem, we need to establish that the set of solutions to the identity in equationand the set of maps derived from equivariances are equal G id = G eq .
Revised: To prove the above theorem, we need to establish that the set of solutions to the identity in equationand the set of maps derived from equivariances are equal G id = G eq .

Original: The proof is given in Section
Revised: The proof is given in Section

Original: A.1 of the appendix.
Revised: A.1 of the appendix.

Original: From Theorem 1, we can derive a number of observations.
Revised: From Theorem 1, we can derive a number of observations.

Original: First, notice that if we have a standard autoencoder 7 then the mechanism is the identity map, m ( z ) =
Revised: First, notice that if we have a standard autoencoder 6 then the mechanism is the identity map, m ( z ) =

Original: z , and its set of equivariances E is any invertible function a , and hence Theorem 1 shows that the encoder is essentially unconstrained.
Revised: z , and its set of equivariances E is any invertible function a , and hence Theorem 1 shows that the encoder is essentially unconstrained.

Original: However, if the mechanism is any non-trivial function, m ( z (cid:48) ) (cid:54) =
Revised: However, if the mechanism is any non-trivial function, m ( z (cid:48) ) (cid:54) =

Original: z (cid:48) for some z (cid:48) , then the space of possible encoders is signiﬁcantly reduced to just those invertible transformations that commute with m .
Revised: z (cid:48) for some z (cid:48) , then the space of possible encoders is signiﬁcantly reduced to just those invertible transformations that commute with m .

Original: If the system involves multiple known mechanisms, { m 1 , . .
Revised: If the system involves multiple known mechanisms, { m 1 , . .

Original: , m T } , where at each time z t +1 = m t ( z t ) , then the encoder is even further constrained.
Revised: , m T } , where at each time z t +1 = m t ( z t ) , then the encoder is even further constrained.

Original: Deﬁne the set of all the mechanisms that are used at least once in the evolution of the system as M ∗ = ∪
Revised: Deﬁne the set of all the mechanisms that are used at least once in the evolution of the system as M ∗ = ∪

Original: Tt =1 { m t } .
Revised: Tt =1 { m t } .

Original: Suppose
Revised: Suppose

Original: E i denotes the equivariances of m i ∈ M ∗ .
Revised: E i denotes the equivariances of m i ∈ M ∗ .

Original: Deﬁne the equivariances that are shared across all the mechanisms as E ∗ =
Revised: Deﬁne the equivariances that are shared across all the mechanisms as E ∗ =

Original: ∩ i E i ;
Revised: ∩ i E i ;

Original: Corollary 1.
Revised: Corollary 1.

Original: If the data generation process follows equation 1, then the encoders that satisfy observation identity in equation 4 for all m ∈ M ∗ identify true encoder g − 1 up to the equivariances shared across all the mechanisms in M ∗ , E ∗ ( ˜ g − 1 ∼ E ∗ g − 1 ).
Revised: If the data generation process follows equation 1, then the encoders that satisfy observation identity in equation 4 for all m ∈ M ∗ identify true encoder g − 1 up to the equivariances shared across all the mechanisms in M ∗ , E ∗ ( ˜ g − 1 ∼ E ∗ g − 1 ).

Original: The proof of the above claim is in Section A.3.
Revised: The proof of the above claim is in Section A.3.

Original: This corollary implies a blessing that comes with more complex environments: if an object is transformed by multiple mechanisms which are diverse (in the sense that they share few equivariances), then it becomes easier to identify.
Revised: This corollary implies a blessing that comes with more complex environments: if an object is transformed by multiple mechanisms which are diverse (in the sense that they share few equivariances), then it becomes easier to identify.

Original: Given access to inputs and outputs of a mechanism, if we cannot tell apart whether some transformation was applied to the input or the output, then the mechanism is equivariant with respect to the transformation.
Revised: Given access to inputs and outputs of a mechanism, if we cannot tell apart whether some transformation was applied to the input or the output, then the mechanism is equivariant with respect to the transformation.

Original: Together
Revised: Together

Original: Theorem 1 and Corollary 1, state that we can learn to invert the data generation process but we cannot distinguish latents that were transformed by equivariances shared across the mechanisms.
Revised: Theorem 1 and Corollary 1, state that we can learn to invert the data generation process but we cannot distinguish latents that were transformed by equivariances shared across the mechanisms.

Original: 2.2. I DENTIFYING ENCODERS FOR AFFINE MECHANISMS
Revised: 2.2. I DENTIFYING ENCODERS FOR AFFINE MECHANISMS

Original: Theorem 1 shows us that an encoder ˜ g − 1 is identiﬁed up to any equivariances of the known mechanism, but given some mechanism, it does not tell us what equivariances it may exhibit.
Revised: Theorem 1 shows us that an encoder ˜ g − 1 is identiﬁed up to any equivariances of the known mechanism, but given some mechanism, it does not tell us what equivariances it may exhibit.

Original: This section gives an example of how one might go about ﬁnding all sources of equivariance for a given mechanism.
Revised: This section gives an example of how one might go about ﬁnding all sources of equivariance for a given mechanism.

Original: We derive the equivariances for afﬁne mechanisms, and in doing so we show conditions under which afﬁne mechanisms lead to identiﬁcation up to some ﬁxed offset.
Revised: We derive the equivariances for afﬁne mechanisms, and in doing so we show conditions under which afﬁne mechanisms lead to identiﬁcation up to some ﬁxed offset.

Original: Afﬁne mechanisms are broadly applicable because with a sufﬁciently short time interval, they approximate a wide variety of physical systems as the Euler discretization of some linear ordinary differential equation.
Revised: Afﬁne mechanisms are broadly applicable because with a sufﬁciently short time interval, they approximate a wide variety of physical systems as the Euler discretization of some linear ordinary differential equation.

Original: In such systems, the mechanism is given by m ( z ) =
Revised: In such systems, the mechanism is given by m ( z ) =

Original: Mz t + b t where M ∈ R d × d is an invertible diagonalizable matrix (with eigendecomposition given as M = S Λ S − 1 , where S is the matrix of eigenvectors and Λ is a diagonal matrix of eigenvalues), b t ∈ R
Revised: Mz t + b t where M ∈ R d × d is an invertible diagonalizable matrix (with eigendecomposition given as M = S Λ S − 1 , where S is the matrix of eigenvectors and Λ is a diagonal matrix of eigenvalues), b t ∈ R

Original: d is the offset parameter at time t , and the analog of equation 2 is,
Revised: d is the offset parameter at time t , and the analog of equation 2 is,

Original: [Equation]
Revised: [Equation]

Original: t ) . We search for some encoder ˜ g − 1 such that this relationship holds for all x and t .
Revised: We search for some encoder ˜ g − 1 such that this relationship holds for all x and t .

Original: Deﬁne an offset function o ( z ) =
Revised: Deﬁne an offset function o ( z ) =

Original: z + p , where the offset function shifts the latent by a vector p .
Revised: z + p , where the offset function shifts the latent by a vector p .

Original: Deﬁne O to be the set of all the offset functions.
Revised: Deﬁne O to be the set of all the offset functions.

Original: We show that the encoder is identiﬁed up to O when we have at least two distinct offset terms b t and a regularity condition (Assumption 2).
Revised: We show that the encoder is identiﬁed up to O when we have at least two distinct offset terms b t and a regularity condition (Assumption 2).

Original: Assumption 1.
Revised: Assumption 1.

Original: In the data generation process in equation 1, we set m ( z t ) =
Revised: In the data generation process in equation 1, we set m ( z t ) =

Original: Mz t + b t , where M is invertible and diagonalizable.
Revised: Mz t + b t , where M is invertible and diagonalizable.

Original: We assume that the offset b t takes at least d + 1 distinct values, which we denote by { b 1 , · · · , b d +1 } .
Revised: We assume that the offset b t takes at least d + 1 distinct values, which we denote by { b 1 , · · · , b d +1 } .

Original: The set { b 2 − b 1 , · · · , b d +1 − b 1 } of vectors is linearly independent.
Revised: The set { b 2 − b 1 , · · · , b d +1 − b 1 } of vectors is linearly independent.

Original: Assumption 2.
Revised: Assumption 2.

Original: a : Z → Z is analytic and satisﬁes the following assumption.
Revised: a : Z → Z is analytic and satisﬁes the following assumption.

Original: For each component i ∈ { 1 , · · · , d } of a i ( z ) and each b ∈ R d , deﬁne the set S ij = { θ | ∇ j a i ( z + b )
Revised: For each component i ∈ { 1 , · · · , d } of a i ( z ) and each b ∈ R d , deﬁne the set S ij = { θ | ∇ j a i ( z + b )

Original: = ∇ j a i ( z ) + ∇ 2 j a i ( θ ) b, z ∈ R d } .
Revised: = ∇ j a i ( z ) + ∇ 2 j a i ( θ ) b, z ∈ R d } .

Original: Each set S ij has a non-zero Lebesgue measure in R d .
Revised: Each set S ij has a non-zero Lebesgue measure in R d .

Original: We conjecture that the regularity condition holds for all analytic functions and is thus not needed.
Revised: We conjecture that the regularity condition holds for all analytic functions and is thus not needed.

Original: Since we do not have a proof of this claim, we include it as an assumption.
Revised: Since we do not have a proof of this claim, we include it as an assumption.

Original: Theorem 2.
Revised: Theorem 2.

Original: If the data generation process follows equation 1 with afﬁne mechanisms, m ( z ) =
Revised: If the data generation process follows equation 1 with afﬁne mechanisms, m ( z ) =

Original: Mz t + b t , Assumptions 1, 2 hold, the eigenvalues of the mechanism M are all distinct, and each component of S − 1 ( b i − b j ) is non-zero for some i (cid:54)
Revised: Mz t + b t , Assumptions 1, 2 hold, the eigenvalues of the mechanism M are all distinct, and each component of S − 1 ( b i − b j ) is non-zero for some i (cid:54)

Original: =
Revised: =

Original: j , then the encoders that solve the observation identity in equation 4 identify true encoder g − 1 up to offsets O such that ˜ g − 1 ∼ O g − 1 .
Revised: j , then the encoders that solve the observation identity in equation 4 identify true encoder g − 1 up to offsets O such that ˜ g − 1 ∼ O g − 1 .

Original: The proof is given in Section A.4 in the Appendix.
Revised: The proof is given in Section A.4 in the Appendix.

Original: The above theorem shows the power of using multiple mechanisms.
Revised: The above theorem shows the power of using multiple mechanisms.

Original: It can be shown that if there is only one mechanism, then we cannot do better than linear identiﬁcation.
Revised: It can be shown that if there is only one mechanism, then we cannot do better than linear identiﬁcation.

Original: However, if we use two mechanisms as is the case in the above theorem, the constraint of shared equivariances (Theorem 1 and Corollary 1) enforces almost exact identiﬁability (only offset-based errors remain).
Revised: However, if we use two mechanisms as is the case in the above theorem, the constraint of shared equivariances (Theorem 1 and Corollary 1) enforces almost exact identiﬁability (only offset-based errors remain).

Original: 2. I DENTIFYING ENCODERS WHEN THE MECHANISMS ARE NOT KNOWN
Revised: 2. I DENTIFYING ENCODERS WHEN THE MECHANISMS ARE NOT KNOWN

Original: We have seen in the previous section that with complete knowledge of the mechanisms under which a system evolves, we can learn an encoder up to equivariances.
Revised: We have seen in the previous section that with complete knowledge of the mechanisms under which a system evolves, we can learn an encoder up to equivariances.

Original: In practice, however, we are unlikely to have such complete knowledge.
Revised: In practice, however, we are unlikely to have such complete knowledge.

Original: In this section, the system still evolves according to some deterministic mechanism, z t +1 ← m t ( z t ) , but we assume that you only know some hypothesis class
Revised: In this section, the system still evolves according to some deterministic mechanism, z t +1 ← m t ( z t ) , but we assume that you only know some hypothesis class

Original: M of possible mechanisms which could have been used, without knowing which m t ∈ M is used at every time step.
Revised: M of possible mechanisms which could have been used, without knowing which m t ∈ M is used at every time step.

Original: A candidate solution now needs to propose both an encoder
Revised: A candidate solution now needs to propose both an encoder

Original: ˜ g − 1 and a mechanism ˜ m t ∈ M for every ( x t , x t +1 ) pair such that,
Revised: ˜ g − 1 and a mechanism ˜ m t ∈ M for every ( x t , x t +1 ) pair such that,

Original: [Equation]
Revised: [Equation]

Original: As before, this relationship holds for all x t ∈ X , where x t +1 is generated from m t , so any candidate solution that is consistent with the x ’s that we observe must satisfy
Revised: As before, this relationship holds for all x t ∈ X , where x t +1 is generated from m t , so any candidate solution that is consistent with the x ’s that we observe must satisfy

Original: [Equation]
Revised: [Equation]

Original: We partition the hypothesis class of mechanisms, M = M ∗ ∪ M (cid:48) , into the mechanisms that are used at least once in the evolution of the system, M ∗ ( M ∗ = ∪
Revised: We partition the hypothesis class of mechanisms, M = M ∗ ∪ M (cid:48) , into the mechanisms that are used at least once in the evolution of the system, M ∗ ( M ∗ = ∪

Original: Tt =1 { m t } ), and mechanisms which are hypothesized but not used, M (cid:48) .
Revised: Tt =1 { m t } ), and mechanisms which are hypothesized but not used, M (cid:48) .

Original: We deﬁne the set of all decoders ˜
Revised: We deﬁne the set of all decoders ˜

Original: g
Revised: g

Original: (with corresponding encoders ˜ g − 1 ) that solve equation 6 across all the time steps as as ˜ G id
Revised: (with corresponding encoders ˜ g − 1 ) that solve equation 6 across all the time steps as as ˜ G id

Original: = { ˜ g | ˜ g is a bijection , for each m t ∈
Revised: = { ˜ g | ˜ g is a bijection , for each m t ∈

Original: M ∗ , ∃ ˜ m t ∈ M , such that g ◦ m t ◦
Revised: M ∗ , ∃ ˜ m t ∈ M , such that g ◦ m t ◦

Original: g
Revised: g

Original: − 1 = ˜
Revised: − 1 = ˜

Original: g ◦ ˜ m t ◦ ˜ g − 1 } .
Revised: g ◦ ˜ m t ◦ ˜ g − 1 } .

Original: This set looks very much like the set G id that we deﬁned in Section 2.2, but the fact that we have to select ˜ m ∈ M rather than knowing the true m implies a new source of non-identiﬁability: imitator mechanisms.
Revised: This set looks very much like the set G id that we deﬁned in Section 2.2, but the fact that we have to select ˜ m ∈ M rather than knowing the true m implies a new source of non-identiﬁability: imitator mechanisms.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Equivariances and imitators w.r.t M .
Revised: Equivariances and imitators w.r.t M .

Original: Deﬁne a set of functions ˜ E that satisfy commutativity w.r.t the set of mechanisms M in the following sense.
Revised: Deﬁne a set of functions ˜ E that satisfy commutativity w.r.t the set of mechanisms M in the following sense.

Original: The set ˜ E comprises of all the bijections, a ( · ) , that satisfy the following condition.
Revised: The set ˜ E comprises of all the bijections, a ( · ) , that satisfy the following condition.

Original: If for each m 1 ∈ M ∗ , ∃ m 2 ∈ M such that a ◦ [Equation] , then a ∈ ˜ E .
Revised: If for each m 1 ∈ M ∗ , ∃ m 2 ∈ M such that a ◦ m 1 = m 2 ◦ a , then a ∈ ˜ E .

Original: The set ˜ E consists of two types of elements.
Revised: The set ˜ E consists of two types of elements.

Original: We illustrate this through an example of set M =
Revised: We illustrate this through an example of set M =

Original: M ∗ = { m 1 , m 2 } .
Revised: M ∗ = { m 1 , m 2 } .

Original: If a is a bijection that commutes with both m 1 and m 2 , i.e., a ◦ m 1 = m 1 ◦ a
Revised: If a is a bijection that commutes with both m 1 and m 2 , i.e., a ◦ m 1 = m 1 ◦ a

Original: and a ◦ m 2 =
Revised: and a ◦ m 2 =

Original: m 2 ◦ a , then a ∈ ˜ E .
Revised: m 2 ◦ a , then a ∈ ˜ E .

Original: From this we can see that ˜ E consists of elements in the intersection of the equivariances of the respective mechanisms.
Revised: From this we can see that ˜ E consists of elements in the intersection of the equivariances of the respective mechanisms.

Original: Alternatively, if a satisﬁes a ◦ m 1 = m 2 ◦ a and a ◦ m 2 = m 1 ◦ a , then we say m 2 “imitates” m 1 and vice-versa because you can produce m 1 ’s output from m 2 for any z using the following relationship m 1 = a − 1 ◦ m 2 ◦ a .
Revised: Alternatively, if a satisﬁes a ◦ m 1 = m 2 ◦ a and a ◦ m 2 = m 1 ◦ a , then we say m 2 “imitates” m 1 and vice-versa because you can produce m 1 ’s output from m 2 for any z using the following relationship m 1 = a − 1 ◦ m 2 ◦ a .

Original: Further simplifcation of this yields that a 2 = a ◦ a is an equivariance of both m 1 and m 2 .
Revised: Further simplifcation of this yields that a 2 = a ◦ a is an equivariance of both m 1 and m 2 .

Original: This example shows that when we know the list of mechanisms M = M ∗
Revised: This example shows that when we know the list of mechanisms M = M ∗

Original: but do not know which mechanism is used, the set ˜ E can be expressed in terms of the equivariances of the mechanisms.
Revised: but do not know which mechanism is used, the set ˜ E can be expressed in terms of the equivariances of the mechanisms.

Original: For further details see the Appendix
Revised: For further details see the Appendix

Original: Section A.11.
Revised: Section A.11.

Original: Deﬁne the set of maps that are identiﬁed up to ˜ E as ˜ G eq = { ˜ g | ˜ g
Revised: Deﬁne the set of maps that are identiﬁed up to ˜ E as ˜ G eq = { ˜ g | ˜ g

Original: = g ◦ a − 1 , a ∈ ˜ E} .
Revised: = g ◦ a − 1 , a ∈ ˜ E} .

Original: Theorem 3.
Revised: Theorem 3.

Original: If the data generation process follows equation 1, then the set of all the encoders that satisfy equation 6 identify true encoders g − 1 up to equivariances and imitators w.r.t M , ˜ E
Revised: If the data generation process follows equation 1, then the set of all the encoders that satisfy equation 6 identify true encoders g − 1 up to equivariances and imitators w.r.t M , ˜ E

Original: ( ˜ g − 1 ∼ ˜
Revised: ( ˜ g − 1 ∼ ˜

Original: E g − 1 ).
Revised: E g − 1 ).

Original: To prove the above theorem, we follow a similar strategy as in Theorem 1 and establish ˜ G id = ˜ G eq .
Revised: To prove the above theorem, we follow a similar strategy as in Theorem 1 and establish ˜ G id = ˜ G eq .

Original: The proof of the above is in Section A.5 of the Appendix.
Revised: The proof of the above is in Section A.5 of the Appendix.

Original: Equivariances and imitators play similar roles in the way that they relax constraints on the encoder ˜ g − 1 —any bijection that commutes with either is a source of non-identiﬁability—but they are different from the perspective of how we should think about designing representation learning algorithms.
Revised: Equivariances and imitators play similar roles in the way that they relax constraints on the encoder ˜ g − 1 —any bijection that commutes with either is a source of non-identiﬁability—but they are different from the perspective of how we should think about designing representation learning algorithms.

Original: Recall that M is composed of two sets of mechanisms, M = M ∗ ∪ M (cid:48) , mechanisms in M ∗ that are used at least once in the evolution of the environment and those mechanisms in M (cid:48) which are hypothesized but never used.
Revised: Recall that M is composed of two sets of mechanisms, M = M ∗ ∪ M (cid:48) , mechanisms in M ∗ that are used at least once in the evolution of the environment and those mechanisms in M (cid:48) which are hypothesized but never used.

Original: Equivariances are dictated only by M ∗ , which characterizes the evolution of the environment.
Revised: Equivariances are dictated only by M ∗ , which characterizes the evolution of the environment.

Original: Any increases to the number of distinct mechanisms in M ∗ will potentially decrease the number of equivariances shared by all mechanisms.
Revised: Any increases to the number of distinct mechanisms in M ∗ will potentially decrease the number of equivariances shared by all mechanisms.

Original: This can only be achieved by modifying the environment in some way, either through an explicit intervention that modiﬁes its mechanisms or by collecting data from multiple environments with diverse set of mechanisms.
Revised: This can only be achieved by modifying the environment in some way, either through an explicit intervention that modiﬁes its mechanisms or by collecting data from multiple environments with diverse set of mechanisms.

Original: For example, in the bouncing balls example given in
Revised: For example, in the bouncing balls example given in

Original: Figure 1, one could change the environment by varying the mass of the balls or observing it under different gravity conditions; or one could intervene by, say, changing the shape of balls in the system such that you get a different bouncing mechanism.
Revised: Figure 1, one could change the environment by varying the mass of the balls or observing it under different gravity conditions; or one could intervene by, say, changing the shape of balls in the system such that you get a different bouncing mechanism.

Original: Imitators, by contrast, are a function of both the mechanisms M ∗ that were used and those that were hypothesized, M (cid:48) .
Revised: Imitators, by contrast, are a function of both the mechanisms M ∗ that were used and those that were hypothesized, M (cid:48) .

Original: An imitator is just some mechanism that can imitate another via some bijection, so one would expect that as we grow the number of mechanisms in M , the size of the set of imitators can only grow; but interestingly, this is not always the case for mechanisms from M ∗ .
Revised: An imitator is just some mechanism that can imitate another via some bijection, so one would expect that as we grow the number of mechanisms in M , the size of the set of imitators can only grow; but interestingly, this is not always the case for mechanisms from M ∗ .

Original: Recall that any a ∈ ˜ E produces an encoder of the form ˜ g − 1 = a ◦ g − 1 , so the same transformation has to be used for all imitations and equivariances among the mechanisms in M ∗ .
Revised: Recall that any a ∈ ˜ E produces an encoder of the form ˜ g − 1 = a ◦ g − 1 , so the same transformation has to be used for all imitations and equivariances among the mechanisms in M ∗ .

Original: Because of this, it is possible that increasing the size of M ∗ reduces the number of imitators.
Revised: Because of this, it is possible that increasing the size of M ∗ reduces the number of imitators.

Original: For example, if there is some mechanism, m i , that does not commute with any non-trivial a in ˜ E (either by imitation or equivariance), then adding m i to M ∗ will make the problem exactly identiﬁed.
Revised: For example, if there is some mechanism, m i , that does not commute with any non-trivial a in ˜ E (either by imitation or equivariance), then adding m i to M ∗ will make the problem exactly identiﬁed.

Original: On the other hand, growing the size of the set of unused mechanisms, M (cid:48) , can result in signiﬁcant non-identiﬁability.
Revised: On the other hand, growing the size of the set of unused mechanisms, M (cid:48) , can result in signiﬁcant non-identiﬁability.

Original: For example, if
Revised: For example, if

Original: M consisted of a ﬂexible class of functions (e.g. a multi-layer perceptron) then it would be easy to construct imitators of the form m 1 = a − 1 ◦ m 2 ◦ a .
Revised: M consisted of a ﬂexible class of functions (e.g. a multi-layer perceptron) then it would be easy to construct imitators of the form m 1 = a − 1 ◦ m 2 ◦ a .

Original: Illustrating Theorem 3.
Revised: Illustrating Theorem 3.

Original: We consider the same setting as in Theorem 2.
Revised: We consider the same setting as in Theorem 2.

Original: For each t ∈ { 1 , · · · , d +1 } , m ( z t ) =
Revised: For each t ∈ { 1 , · · · , d +1 } , m ( z t ) =

Original: Mz t + b t and x t = g
Revised: Mz t + b t and x t = g

Original: ( z t ) .
Revised: ( z t ) .

Original: We only know that the mechanism is afﬁne and only the offset b t is changing, but parameters M and b t are not known.
Revised: We only know that the mechanism is afﬁne and only the offset b t is changing, but parameters M and b t are not known.

Original: Let us construct the set ˜ E corresponding to the above setting.
Revised: Let us construct the set ˜ E corresponding to the above setting.

Original: We can show that the set ˜ E consists of afﬁne functions (See the Appendix A.6 for details).
Revised: We can show that the set ˜ E consists of afﬁne functions (See the Appendix A.6 for details).

Original: From Theorem 3, we can thus conclude that for this data generation process even with very little knowledge of the mechanism, we get linear identiﬁability .
Revised: From Theorem 3, we can thus conclude that for this data generation process even with very little knowledge of the mechanism, we get linear identiﬁability .

Original: This is weaker than the offset based identiﬁability in Theorem 2, but there we were required to know the entire afﬁne mechanism.
Revised: This is weaker than the offset based identiﬁability in Theorem 2, but there we were required to know the entire afﬁne mechanism.

Original:  S TOCHASTIC MECHANISMS
Revised:  S TOCHASTIC MECHANISMS

Original: The results thus far relied on the assumption that the evolution of a system could be described in terms of deterministic mechanisms.
Revised: The results thus far relied on the assumption that the evolution of a system could be described in terms of deterministic mechanisms.

Original: This deterministic approach models settings where the full latent state is observable (via some unknown encoder g − 1 ) at a short enough time interval that there is no uncertainty about the system’s evolution.
Revised: This deterministic approach models settings where the full latent state is observable (via some unknown encoder g − 1 ) at a short enough time interval that there is no uncertainty about the system’s evolution.

Original: To generalize to cases where there is some uncertainty about the latent state’s evolution, we now develop analogous identiﬁcation results for stochastic mechanisms that induce conditional distributions over latent states.
Revised: To generalize to cases where there is some uncertainty about the latent state’s evolution, we now develop analogous identiﬁcation results for stochastic mechanisms that induce conditional distributions over latent states.

Original: The systems evolves as
Revised: The systems evolves as

Original: [Equation]
Revised: [Equation]

Original: where each U t is noise with each component sampled independently from standard uniform distribution Uniform [0 , 1] , Z 1 ∼ P Z , m t : Z× [0 , 1] d → Z , and the decoder g : Z → X is a diffeomorphism (i.e. a smooth bijection with an invertible Jacobian, see deﬁnition A.1 (Kass & Vos, 2011)).
Revised: where each U t is noise with each component sampled independently from standard uniform distribution Uniform [0 , 1] , Z 1 ∼ P Z , m t : Z× [0 , 1] d → Z , and the decoder g : Z → X is a diffeomorphism (i.e. a smooth bijection with an invertible Jacobian, see deﬁnition A.1 (Kass & Vos, 2011)).

Original: In this section, we will focus on the case where the true mechanism is unknown; the case where mechanism is known is a special case with no imitator, i.e., ˜ m t = m t for all t .
Revised: In this section, we will focus on the case where the true mechanism is unknown; the case where mechanism is known is a special case with no imitator, i.e., ˜ m t = m t for all t .

Original: The goal is to search for an encoder ˜ g − 1 , which is a diffeomorphism, that generates ˆ X t +

Revised: The goal is to search for an encoder ˜ g − 1 , which is a diffeomorphism, that generates ˆ X t +

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: g ◦ ˜ m
Revised: g ◦ ˜ m

Original: t (˜ g − 1 ( x t ) , ˆ U t ) , where ˆ U t is a random vector with each component from Uniform [0 , 1] .
Revised: t (˜ g − 1 ( x t ) , ˆ U t ) , where ˆ U t is a random vector with each component from Uniform [0 , 1] .

Original: In the deterministic case, we had required that any candidate encoder, ˜ g − 1 , was point-wise consistent with the pairs of observations ( x t +1 , x t ) .
Revised: In the deterministic case, we had required that any candidate encoder, ˜ g − 1 , was point-wise consistent with the pairs of observations ( x t +1 , x t ) .

Original: Here, encoders are only required to match the observed conditional distributions.
Revised: Here, encoders are only required to match the observed conditional distributions.

Original: An encoder that is consistent with the observed data can be used to generate ˆ X t +1 such that the distribution of ˆ
Revised: An encoder that is consistent with the observed data can be used to generate ˆ X t +1 such that the distribution of ˆ

Original: X t +1 | X t
Revised: X t +1 | X t

Original: = x t matches the distribution of X t +1 | X t = x t for all x t ∈ X ,
Revised: = x t matches the distribution of X t +1 | X t = x t for all x t ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: Now, deﬁne the set of all candidate decoders ˜ g (with corresponding encoders ˜ g − 1 ) that solve the above equation 8 as G sid .
Revised: Now, deﬁne the set of all candidate decoders ˜ g (with corresponding encoders ˜ g − 1 ) that solve the above equation 8 as G sid .

Original: We can extend the notion of equivariance and imitation to the stochastic case by replacing equality in value by equality in distribution, and show that, as before, these are the only sources of non-identiﬁability.
Revised: We can extend the notion of equivariance and imitation to the stochastic case by replacing equality in value by equality in distribution, and show that, as before, these are the only sources of non-identiﬁability.

Original: In the special case where m t is known, ˜ E s , deﬁned below, only contains maps that result from equivariance.
Revised: In the special case where m t is known, ˜ E s , deﬁned below, only contains maps that result from equivariance.

Original: We continue to use the M ∗ – set of mechanisms that are used at least once in the evolution of the system and M – hypothesis class of all the mechanisms.
Revised: We continue to use the M ∗ – set of mechanisms that are used at least once in the evolution of the system and M – hypothesis class of all the mechanisms.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Equivariance and imitators in distribution w.r.t M .
Revised: Equivariance and imitators in distribution w.r.t M .

Original: Deﬁne a set of functions
Revised: Deﬁne a set of functions

Original: E s that satisfy commutativity w.r.t the set of distributions M in the following sense.
Revised: E s that satisfy commutativity w.r.t the set of distributions M in the following sense.

Original: The set E s comprises all the diffeomorphisms a that satisfy the following condition: a ∈ E s if and only if for each m ∈ M ∗ , ∃ m (cid:

Revised: The set E s comprises all the diffeomorphisms a that satisfy the following condition: a ∈ E s if and only if for each m ∈ M ∗ , ∃ m (cid:

Original: ) ∈
Revised: ) ∈

Original: M such that for all z ∈ Z , a ◦ m ( z, U )
Revised: M such that for all z ∈ Z , a ◦ m ( z, U )

Original: d = m (cid:48) ( a ( z ) , U ) , where each component of U is sampled independently from Uniform [0 , 1] .
Revised: d = m (cid:48) ( a ( z ) , U ) , where each component of U is sampled independently from Uniform [0 , 1] .

Original: Deﬁne the set of maps that identify true g up to E s as G seq = { ˜ g | ˜ g
Revised: Deﬁne the set of maps that identify true g up to E s as G seq = { ˜ g | ˜ g

Original: = g ◦ a − 1 , a ∈ E s }
Revised: = g ◦ a − 1 , a ∈ E s }

Original: Theorem 4.
Revised: Theorem 4.

Original: If the data generation process follows equation 7, then the set of encoders that solve stochastic observation identity equation 8 identify the true g − 1 up to the equivariances and imitators in distribution w.r.t M , E s ( ˜ g − 1 ∼ E s g − 1 ).
Revised: If the data generation process follows equation 7, then the set of encoders that solve stochastic observation identity equation 8 identify the true g − 1 up to the equivariances and imitators in distribution w.r.t M , E s ( ˜ g − 1 ∼ E s g − 1 ).

Original: To prove the above theorem, we follow a similar strategy as in Theorem 1 and establish G s id = G s eq .
Revised: To prove the above theorem, we follow a similar strategy as in Theorem 1 and establish G sid = G seq .

Original: The proof of the above is provided in Section A.7 in the Appendix.
Revised: The proof of the above is provided in Section A.7 in the Appendix.

Original: Theorem 4 is consistent with the results that we developed for the deterministic case (Theorem 3), but because the mechanism is allowed to be stochastic, it allows us to generalize known results based on distributional assumptions; we give an example of this in the next section.
Revised: Theorem 4 is consistent with the results that we developed for the deterministic case (Theorem 3), but because the mechanism is allowed to be stochastic, it allows us to generalize known results based on distributional assumptions; we give an example of this in the next section.

Original:  A MECHANISM - BASED PERSPECTIVE ON EXISTING RESULTS
Revised:  A MECHANISM - BASED PERSPECTIVE ON EXISTING RESULTS

Original: Our primary motivation for understanding how mechanistic knowledge can aid identiﬁcation, is to develop methods that do not require independence assumptions over the latent variables.
Revised: Our primary motivation for understanding how mechanistic knowledge can aid identiﬁcation, is to develop methods that do not require independence assumptions over the latent variables.

Original: However, independence assumptions are not incompatible with the mechanism-based perspective: they simply deﬁne a particular kind of mechanism, which then implies identiﬁcation up to the mechanism’s associated equivariances and imitators.
Revised: However, independence assumptions are not incompatible with the mechanism-based perspective: they simply deﬁne a particular kind of mechanism, which then implies identiﬁcation up to the mechanism’s associated equivariances and imitators.

Original: We demonstrate this by re-deriving recent identiﬁcation results from Klindt et al.
Revised: We demonstrate this by re-deriving recent identiﬁcation results from Klindt et al.

Original: (2020) using the mechanisms implied by their respective distributional assumptions.
Revised: (2020) using the mechanisms implied by their respective distributional assumptions.

Original: We begin by describing the data generation process used by Klindt et al.
Revised: We begin by describing the data generation process used by Klindt et al.

Original: (2020) as a stochastic mechanism of the form of equation 7.
Revised: (2020) as a stochastic mechanism of the form of equation 7.

Original: For each t ∈ { 1 , · · · , T }
Revised: For each t ∈ { 1 , · · · , T }

Original: [Equation]
Revised: [Equation]

Original: where f is an inverse CDF such that each component of V t ∈ R d is sampled from the generalized
Revised: where f is an inverse CDF such that each component of V t ∈ R d is sampled from the generalized

Original: Laplace distribution centred at zero with norm parameter α (cid:54) = 2 , Z 1 ∼ P Z .
Revised: Laplace distribution centred at zero with norm parameter α (cid:54) = 2 , Z 1 ∼ P Z .

Original: Next, we want to use
Revised: Next, we want to use

Original: Theorem 4 to derive all the solutions to the observation identity in equation 8.
Revised: Theorem 4 to derive all the solutions to the observation identity in equation 8.

Original: Theorem 5.
Revised: Theorem 5.

Original: If the data generation process follows equation 9, and P Z and the parameters deﬁning f are known (same assumption as in Klindt et al.
Revised: If the data generation process follows equation 9, and P Z and the parameters deﬁning f are known (same assumption as in Klindt et al.

Original: , then the solution to the stochastic observation identity equation 8 leads to identifying the true representations up to permutation, sign-ﬂip and offset.
Revised: , then the solution to the stochastic observation identity equation 8 leads to identifying the true representations up to permutation, sign-ﬂip and offset.

Original: The proof is given in Section A.8 in the Appendix.
Revised: The proof is given in Section A.8 in the Appendix.

Original: The above theorem generalizes Theorem 1 from
Revised: The above theorem generalizes Theorem 1 from

Original: Klindt et al.
Revised: Klindt et al.

Original: (2020) as we do not require α < 2 and rather we work with α (cid:54) = 2 .
Revised: (2020) as we do not require α < 2 and rather we work with α (cid:54) = 2 .

Original: Alternatively, analogous results to those in Klindt et al.
Revised: Alternatively, analogous results to those in Klindt et al.

Original: Hyvarinen & Morioka (2017) can be derived in settings where we do not know the distribution of V t but instead assume that we observe X t often enough that the difference between Z t and Z t +1 is small.
Revised: Hyvarinen & Morioka (2017) can be derived in settings where we do not know the distribution of V t but instead assume that we observe X t often enough that the difference between Z t and Z t +1 is small.

Original: In particular, suppose that the data generation process follows equation 9 except each component of V t is an i.i.d.
Revised: In particular, suppose that the data generation process follows equation 9 except each component of V t is an i.i.d.

Original: draw from a nonGaussian with zero mean and | V t | < δ .
Revised: draw from a nonGaussian with zero mean and | V t | < δ .

Original: Then as δ → 0 the true latent is identiﬁed up to permutation, sign-ﬂip and offset.
Revised: Then as δ → 0 the true latent is identiﬁed up to permutation, sign-ﬂip and offset.

Original: See Section A.9 for details.
Revised: See Section A.9 for details.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Non-linear independent component analysis (ICA) is a highly unidentiﬁed problem; several works (Hyv¨arinen & Pajunen, 1999; Locatello et al., 2019) have shown that it is impossible to invert the data generation process without placing restrictions on the data and models.
Revised: Non-linear independent component analysis (ICA) is a highly unidentiﬁed problem; several works (Hyv¨arinen & Pajunen, 1999; Locatello et al., 2019) have shown that it is impossible to invert the data generation process without placing restrictions on the data and models.

Original: In recent years, a lot of progress has been made on the problem of non-linear identiﬁcation.
Revised: In recent years, a lot of progress has been made on the problem of non-linear identiﬁcation.

Original: Hyvarinen & Morioka (2016; 2017) provide the ﬁrst proofs for identiﬁcation in non-linear ICA.
Revised: Hyvarinen & Morioka (2016; 2017) provide the ﬁrst proofs for identiﬁcation in non-linear ICA.

Original: Hyvarinen & Morioka (2016) showed that if the latent variables are mutually independent, with each component evolving in time following a non-stationary time series without temporal dependence, then non-linear identiﬁcation is possible.
Revised: Hyvarinen & Morioka (2016) showed that if the latent variables are mutually independent, with each component evolving in time following a non-stationary time series without temporal dependence, then non-linear identiﬁcation is possible.

Original: Hyvarinen & Morioka (2017) showed that non-linear identiﬁcation is also possible if the latent variables are mutually independent, with each component evolving in time following a stationary time series with temporal dependence.
Revised: Hyvarinen & Morioka (2017) showed that non-linear identiﬁcation is also possible if the latent variables are mutually independent, with each component evolving in time following a stationary time series with temporal dependence.

Original: In H¨alv¨a & Hyvarinen (2020), the authors combine non-stationarity (Hyvarinen & Morioka, 2016) and temporal dependency (Hyvarinen &
Revised: In H¨alv¨a & Hyvarinen (2020), the authors combine non-stationarity (Hyvarinen & Morioka, 2016) and temporal dependency (Hyvarinen &

Original: Morioka, 2017) and extend identiﬁability guarantees in somewhat more general settings.
Revised: Morioka, 2017) and extend identiﬁability guarantees in somewhat more general settings.

Original: Khemakhem et al.
Revised: Khemakhem et al.

Original: (2020a);
Revised: (2020a);

Original: Hyvarinen et al.
Revised: Hyvarinen et al.

Original: ; Khemakhem et al.
Revised: ; Khemakhem et al.

Original: (2020b) further generalized the previous results; in these works instead of using time the authors require observation of auxiliary information.
Revised: (2020b) further generalized the previous results; in these works instead of using time the authors require observation of auxiliary information.

Original: Klindt et al.
Revised: Klindt et al.

Original: departs from other non-linear ICA works as it explicitly exploits the sparsity in the transitions of the latent variables (further details on Klindt et al.
Revised: departs from other non-linear ICA works as it explicitly exploits the sparsity in the transitions of the latent variables (further details on Klindt et al.

Original: (2020) can be found in the previous section.).
Revised: (2020) can be found in the previous section.).

Original: In Zimmermann et al.
Revised: In Zimmermann et al.

Original: , the authors show that minimizing contrastive losses commonly used in self-supervised learning can also guarantee identiﬁcation provided the data (contrastive pairs) follow a speciﬁc choice of data generation process (e.g., contrastive pair is generated from a von Mises-Fisher distribution).
Revised: , the authors show that minimizing contrastive losses commonly used in self-supervised learning can also guarantee identiﬁcation provided the data (contrastive pairs) follow a speciﬁc choice of data generation process (e.g., contrastive pair is generated from a von Mises-Fisher distribution).

Original: In another line of work Locatello et al.
Revised: In another line of work Locatello et al.

Original: Shu et al.
Revised: Shu et al.

Original: (2019), the authors study the role of weak supervision in assisting disetanglement.
Revised: (2019), the authors study the role of weak supervision in assisting disetanglement.

Original: In a recent work, Gresele et al.
Revised: In a recent work, Gresele et al.

Original: (2021), propose to add new form of constraints to non-linear ICA.
Revised: (2021), propose to add new form of constraints to non-linear ICA.

Original: The constraint is based on the observation that the decoder g that gives rise to the image x is composed of simpler functions that are mutually algorithmically independent; the authors exploit this inductive bias on the structure of g to invert the data generation process.
Revised: The constraint is based on the observation that the decoder g that gives rise to the image x is composed of simpler functions that are mutually algorithmically independent; the authors exploit this inductive bias on the structure of g to invert the data generation process.

Original: In another recent work von K ¨ ugelgen et al.
Revised: In another recent work von K ¨ ugelgen et al.

Original: , the authors consider models where the latent variables are divided into two blocks – content and style block, where the latents are not necessarily independent.
Revised: , the authors consider models where the latent variables are divided into two blocks – content and style block, where the latents are not necessarily independent.

Original: Using data augmentation on the style latents, the authors show block-wise identiﬁcation for content block.
Revised: Using data augmentation on the style latents, the authors show block-wise identiﬁcation for content block.

Original: In Section 4 we argued the existing distributional assumptions could be interpreted as particular choices of stochastic mechanisms; for more details, see Table 1 in the Appendix, where we describe the form of the respective mechanisms.
Revised: In Section 4 we argued the existing distributional assumptions could be interpreted as particular choices of stochastic mechanisms; for more details, see Table 1 in the Appendix, where we describe the form of the respective mechanisms.

Original: In short, prior work has focused on identiﬁcation guarantees under assumptions on the dependence between the different random variables, which is in sharp contrast to our approach, which focuses on identiﬁcation under varying degrees of the knowledge of mechanisms that govern latent dynamics.
Revised: In short, prior work has focused on identiﬁcation guarantees under assumptions on the dependence between the different random variables, which is in sharp contrast to our approach, which focuses on identiﬁcation under varying degrees of the knowledge of mechanisms that govern latent dynamics.

Original: Equivariance There is signiﬁcant recent interest in leveraging equivariance assumptions to design more efﬁcient deep network architectures; for a recent survey, see Bronstein et al.
Revised: Equivariance There is signiﬁcant recent interest in leveraging equivariance assumptions to design more efﬁcient deep network architectures; for a recent survey, see Bronstein et al.

Original: The general recipe of this line of work is to design functions (deep network architectures) that enforce equivariances.
Revised: The general recipe of this line of work is to design functions (deep network architectures) that enforce equivariances.

Original: Our setting inverts this recipe, in that we have some known function, m ( z ) , and we are interested in ﬁnding all of its equivariances.
Revised: Our setting inverts this recipe, in that we have some known function, m ( z ) , and we are interested in ﬁnding all of its equivariances.

Original: The relationship between distributions and their equivariances has a long history in the statistics literature (see e.g. Eaton, 1989, and the references therein).
Revised: The relationship between distributions and their equivariances has a long history in the statistics literature (see e.g. Eaton, 1989, and the references therein).

Original: Our characterization of stochastic equivariances was inspired by the functional representations given in Bloem-Reddy & Teh (2020).
Revised: Our characterization of stochastic equivariances was inspired by the functional representations given in Bloem-Reddy & Teh (2020).

Original: Finally, the importance of group symmetries in representation learning was discussed in Higgins et al.
Revised: Finally, the importance of group symmetries in representation learning was discussed in Higgins et al.

Original: Higgins et al.
Revised: Higgins et al.

Original: focus on the relationship between the symmetries of the environment and a model’s representations, whereas we focus on how symmetries in the environment’s transition function constrain the representation.
Revised: focus on the relationship between the symmetries of the environment and a model’s representations, whereas we focus on how symmetries in the environment’s transition function constrain the representation.

Original: The two perspectives are complementary, and in future work we hope to unify them.
Revised: The two perspectives are complementary, and in future work we hope to unify them.

Original:  D ISCUSSION AND LIMITATIONS
Revised:  D ISCUSSION AND FUTURE WORK

Original: This paper has presented the ﬁrst systematic study of how mechanisms governing the dynamics of high-level variables can be used to identify these variables from low-level observations, and up to what equivariances, which depend on the mechanisms.
Revised: This paper has presented the ﬁrst systematic study of how mechanisms governing the dynamics of high-level variables can be used to identify these variables from low-level observations, and up to what equivariances, which depend on the mechanisms.

Original: We show that this perspective is both powerful—yielding signiﬁcant constraints in the space of possible representation—and that it generalizes many known approaches.
Revised: We show that this perspective is both powerful— yielding signiﬁcant constraints in the space of possible representation—and that it generalizes many known approaches.

Original: There are, however, a number of open problems that need to be addressed to make this approach practical.
Revised: Moving forward, a natural direction is to build methods founded on this theory.

Original: First, we have not speciﬁed a particular loss function to select ˜ g ; theobservation identity immediately implies a reconstruction-based loss, but other losses such as contrastive losses may be more beneﬁcial.
Revised: We describe two natural losses based on the identity ˜ g ◦ m ◦ ˜ g − 1 ( x t ) = x t +1 . i)

Original: Second, there is not yet a general approach to enforcing invertibility while mapping between different sized spaces, so maintaining bijectivity is a challenge. And ﬁnally, while we provide a characterization of identiﬁability when mechanisms are not known, a natural follow-up question is how to use those insights to build methods that allow one to learn both the mechanism and the representation.
Revised: Loss using observations:

Original: We are working on a follow-up based on the above directions.
Revised: We minimize next observation prediction error min

Original: In Appendix Section A.12 we share our initial results on 3d identiﬁcation dataset (Zimmermann et al., 2021){+[Equation] where H , M is the hypothesis class of functions for decoder and mechanisms respectively. ii) Loss using latents: Alternatively, one could re-write the identity as g − 1 ◦ x t +
 = m ◦ g − 1 ◦ x t and use square error or contrastive loss given as min ˜ g ∈H , ˜ m ∈M (cid:80) t E (cid:104) − log (cid:16)+}
Revised: {+[Equation] where H , M is the hypothesis class of functions for decoder and mechanisms respectively. ii) Loss using latents: Alternatively, one could re-write the identity as g − 1 ◦ x t +
 = m ◦ g − 1 ◦ x t and use square error or contrastive loss given as min ˜ g ∈H , ˜ m ∈M (cid:80) t E (cid:104) − log (cid:16)+}

Revised: [Equation]

Revised: where τ represents other time instances,

Revised: i.e., τ (cid:54) =

Revised: t + 1 .

Revised: The positive pair in the contrastive loss is formed by the adjacent time instances and the negative pair is formed by non-adjacent time instances.

Revised: In Appendix Section A.12 we share our initial results for the contrastive approach applied to 3d identiﬁcation dataset (Zimmermann et al., 2021) .

Original: A A PPENDIX A. P ROOF OF T HEOREM
Revised: A A PPENDIX A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: First we show that G id ⊆ G eq .
Revised: First we show that G id ⊆ G eq .

Original: Consider a ˜ g ∈ G id .
Revised: Consider a ˜ g ∈ G id .

Original: For each x ∈ X g [Equation]
Revised: For each x ∈ X g ◦ m ◦ g − 1 ( x ) = ˜

Original:  ˜ g − 1 ◦ (cid:16) g ◦ m ◦ g − 1 ( x ) (cid:17) =[Equation]
Revised: g ◦ m ◦ ˜ g − 1 ( x ) ˜ g − 1 ◦ (cid:16) g ◦ m ◦ g − 1 ( x ) (cid:17) =

Original:  (cid:16) [Equation]
Revised: ˜ g − 1 ◦ (cid:16) ˜

Original: g  − 1 ( x ) (cid:17) (Compose ˜ g − 1 with both sides) ˜
Revised: g ◦ m ◦ ˜ g − 1 ( x ) (cid:17) (Compose ˜ g − 1 with both sides) ˜

Original: g − 1 ◦ (cid:16) g ◦ m ◦ g − 1 ( x ) (cid:17)
Revised: g − 1 ◦ (cid:16) g ◦ m ◦ g − 1 ( x ) (cid:17)

Original: = m ◦ ˜ g −
Revised: = m ◦ ˜ g −

Original: [Equation]
Revised: [Equation]

Original: Since g is invertible we can substitute x in the above equation with x = g ( z ) and obtain for
Revised: Since g is invertible we can substitute x in the above equation with x = g ( z ) and obtain for

Original: each z ∈ Z
Revised: each z ∈ Z

Original: [Equation]
Revised: [Equation]

Original: Deﬁne ˜ g − 1 ◦ g = a .
Revised: Deﬁne ˜ g − 1 ◦ g = a .

Original: Observe that a is invertible and from equation 10 we gather that a ∈ E .
Revised: Observe that a is invertible and from equation 10 we gather that a ∈ E .

Original: Also, since ˜ g = g ◦ a − 1 , we can conclude that ˜ g ∈ G eq , which proves the ﬁrst part of the claim.
Revised: Also, since ˜ g = g ◦ a − 1 , we can conclude that ˜ g ∈ G eq , which proves the ﬁrst part of the claim.

Original: For the second part, we show that G eq ⊆
Revised: For the second part, we show that G eq ⊆

Original: G id .
Revised: G id .

Original: Consider a ˜ g ∈ G eq = { ˜ g | ˜
Revised: Consider a ˜ g ∈ G eq = { ˜ g | ˜

Original: g = g ◦ a − 1 , a ∈ E} .
Revised: g = g ◦ a − 1 , a ∈ E} .

Original: By deﬁnition, can express ˜ g
Revised: By deﬁnition, can express ˜ g

Original: = g ◦ a − 1 .
Revised: = g ◦ a − 1 .

Original: For each x ∈ X we write
Revised: For each x ∈ X we write

Original: [Equation]
Revised: [Equation]

Original: Observe that ˜ g is both a bijection and satisﬁes the identity in equation 4.
Revised: Observe that ˜ g is both a bijection and satisﬁes the identity in equation 4.

Original: Therefore, ˜ g ∈ G id .
Revised: Therefore, ˜ g ∈ G id .

Original: This proves the second part of the claim.
Revised: This proves the second part of the claim.

Original: Therefore, G id = G eq .
Revised: Therefore, G id = G eq .

Original: A. L INEAR MECHANISM M AND LINEAR G .
Revised: A. L INEAR MECHANISM M AND LINEAR G .

Original: The focus of the main text is on nonlinear identiﬁcation, but in this section we show how the analysis for general functions g and m applies to the special case when g and m are linear and afﬁne maps respectively.
Revised: The focus of the main text is on nonlinear identiﬁcation, but in this section we show how the analysis for general functions g and m applies to the special case when g and m are linear and afﬁne maps respectively.

Original: This special case is useful both as a concrete example, and as a stepping stone to
Revised: This special case is useful both as a concrete example, and as a stepping stone to

Original: Theorem 2 (nonlinear g with afﬁne m ) which will reuse some of the same proof strategies.
Revised: Theorem 2 (nonlinear g with afﬁne m ) which will reuse some of the same proof strategies.

Original: We write the data generation process as follows.
Revised: We write the data generation process as follows.

Original: [Equation]
Revised: [Equation]

Original: where M ∈ R d × d is a diagonalizable matrix describing the mechanism, b ∈ R
Revised: where M ∈ R d × d is a diagonalizable matrix describing the mechanism, b ∈ R

Original: d is the offset parameter,
Revised: d is the offset parameter,

Original: G ∈ R d × d is an invertible matrix determining how the data transforms from latent space to the observable space.
Revised: G ∈ R d × d is an invertible matrix determining how the data transforms from latent space to the observable space.

Original: We write the eigendecomposition of M as follows M = S Λ S − 1 , where S is the matrix of eigenvectors and Λ is a diagonal matrix of the eigenvalues.
Revised: We write the eigendecomposition of M as follows M = S Λ S − 1 , where S is the matrix of eigenvectors and Λ is a diagonal matrix of the eigenvalues.

Original: On the same lines as the equation 2, we can obtain an identity between x t and x t +1 as follows.
Revised: On the same lines as the equation 2, we can obtain an identity between x t and x t +1 as follows.

Original: [Equation]
Revised: [Equation]

Original: If the learner knows M and b , it tries to solve for an invertible ˜ G that satisﬁes
Revised: If the learner knows M and b , it tries to solve for an invertible ˜ G that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Theorem 6.
Revised: Theorem 6.

Original: If the data generation process follows equation 11 and the eigenvalues of the mechanism
Revised: If the data generation process follows equation 11 and the eigenvalues of the mechanism

Original: M are all distinct and each component of the vector S − 1 b is non-zero, then the only solution to the identity in equation 13 is the true mechanism G .
Revised: M are all distinct and each component of the vector S − 1 b is non-zero, then the only solution to the identity in equation 13 is the true mechanism G .

Original: Proof.
Revised: Proof.

Original: We take the difference of the equations 12 and 13 to get the following condition.
Revised: We take the difference of the equations 12 and 13 to get the following condition.

Original: For each x t ∈ R d
Revised: For each x t ∈ [Equation] d

Original: [Equation]
Revised: [Equation]

Original: Because, equations 12 and 13 hold for all x t , we can substitute x t = 0 in the above to get
Revised: Because, equations 12 and 13 hold for all x t , we can substitute x t = 0 in the above to get

Original: [Equation]
Revised: [Equation]

Original: We plug the above condition in equation 15 back into equation 14 to get the following condition.
Revised: We plug the above condition in equation 15 back into equation 14 to get the following condition.

Original: For each x t ∈ R d
Revised: For each x t ∈ R d

Original: [Equation]
Revised: [Equation]

Original: If equation 16 holds for d linearly independent vectors x t ∈ R d , then we can conclude that,
Revised: If equation 16 holds for d linearly independent vectors x t ∈ R d , then we can conclude that,

Original: [Equation]
Revised: [Equation]

Original: Let A =
Revised: Let A =

Original: G − 1 ˜ G .
Revised: G − 1 ˜ G .

Original: We substitute A and the eigendecomposition of M ( M = S Λ S − 1 , where Λ = diag ( λ 1 , · · · , λ d ) ) in equation 17 to get
Revised: We substitute A and the eigendecomposition of M ( M = S Λ S − 1 , where Λ = diag ( λ 1 , · · · , λ d ) ) in equation 17 to get

Original: [Equation]
Revised: [Equation]

Original: where C = S [Equation]
Revised: where C = S − 1 AS )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We further simplify equation 18 and compare each element of the matrix Λ C and C Λ to get the following condition.
Revised: We further simplify equation 18 and compare each element of the matrix Λ C and C Λ to get the following condition.

Original: For all i, j ∈ { 1 , · · · , d }
Revised: For all i, j ∈ { 1 , · · · , d }

Original: [Equation]
Revised: [Equation]

Original: Consider the above equation 19 for i (cid:54) =
Revised: Consider the above equation 19 for i (cid:54) =

Original: j to get
Revised: j to get

Original: [Equation]
Revised: [Equation]

Original: (we use the assumption that λ
Revised: (we use the assumption that λ

Original: i (cid:54)
Revised: i (cid:54)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From the above, it follows that C is a diagonal matrix.
Revised: From the above, it follows that C is a diagonal matrix.

Original: We obtain an expression for A in terms of C matrix below.
Revised: We obtain an expression for A in terms of C matrix below.

Original: [Equation]
Revised: [Equation]

Original: From equation 15 we get that
Revised: From equation 15 we get that

Original: [Equation]
Revised: [Equation]

Original: multiply
Revised: multiply

Original: [Equation]
Revised: [Equation]

Original:  Since C is a diagonal matrix, we can simplify the above condition further to get ( c ii − 1)( S − 1 b ) i = 0 .
Revised: [Equation] Since C is a diagonal matrix, we can simplify the above condition further to get ( c ii − 1)( S − 1 b ) i = 0 .

Original: Since ( S − 1 b ) i (cid:54) = 

Revised: Since ( S − 1 b ) i (cid:54) =

Original: = ⇒ c ii = 1 , we obtain C
Revised: = ⇒ c ii = 1 , we obtain C

Original: = I = ⇒
Revised: = I = ⇒

Original: G − 1 ˜ G =
Revised: G − 1 ˜ G =

Original: I = ⇒ G =
Revised: I = ⇒ G =

Original: ˜ G .
Revised: ˜ G .

Original: A. P ROOF OF C OROLLARY
Revised: A. P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: This proof follows essentially the same strategy as the proof of Theorem 1 with a set of mechanisms, M ∗ instead of a single mechanism m ; we include it for completeness.
Revised: This proof follows essentially the same strategy as the proof of Theorem 1 with a set of mechanisms, M ∗ instead of a single mechanism m ; we include it for completeness.

Original: First we show that G ∗ id ⊆ G ∗ eq .
Revised: First we show that G ∗ id ⊆ G ∗ eq .

Original: Consider a ˜ g ∈ G ∗ id .
Revised: Consider a ˜ g ∈ G ∗ id .

Original: For each x ∈ X and for each m ∈ M ∗
Revised: For each x ∈ X and for each m ∈ M ∗

Original: [Equation]
Revised: [Equation]

Original: 23) and by following the same steps as the proof of Theorem 1, we can show that,
Revised: 23) and by following the same steps as the proof of Theorem 1, we can show that,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As before, deﬁne ˜ g − 1 ◦ g = a .
Revised: As before, deﬁne ˜ g − 1 ◦ g = a .

Original: Observe that a is invertible and from equation 24 we gather that a ∈ E ∗ .
Revised: Observe that a is invertible and from equation 24 we gather that a ∈ E ∗ .

Original: Also, since ˜ g = g ◦ a − 1 , we can conclude that ˜ g ∈ G ∗ eq , which proves the ﬁrst part of the claim.
Revised: Also, since ˜ g = g ◦ a − 1 , we can conclude that ˜ g ∈ G ∗ eq , which proves the ﬁrst part of the claim.

Original: In the second part, we need to show that G ∗ eq ⊆
Revised: In the second part, we need to show that G ∗ eq ⊆

Original: G ∗ id .
Revised: G ∗ id .

Original: Consider a ˜ g ∈ G ∗ eq
Revised: Consider a ˜ g ∈ G ∗ eq

Original: = { ˜ g | ˜ g
Revised: = { ˜ g | ˜ g

Original: = g ◦ a − 1 , a ∈ E ∗ } .
Revised: = g ◦ a − 1 , a ∈ E ∗ } .

Original: By deﬁnition, can express ˜ g
Revised: By deﬁnition, can express ˜ g

Original: = g ◦ a − 1 .
Revised: = g ◦ a − 1 .

Original: For each x ∈ X and for each m ∈ M ∗ we write,
Revised: For each x ∈ X and for each m ∈ M ∗ we write,

Original: [Equation]
Revised: [Equation]

Original: since a commutes with each m ∈ M ) =
Revised: since a commutes with each m ∈ M ) =

Original: [Equation]
Revised: [Equation]

Original: all m [Equation]
Revised: all m (25)

Original: Observe that ˜ g is both a bijection and satisﬁes the observation identity in equation 4.
Revised: Observe that ˜ g is both a bijection and satisﬁes the observation identity in equation 4.

Original: Therefore, ˜ g ∈ G ∗ id .
Revised: Therefore, ˜ g ∈ G ∗ id .

Original: This proves the second part of the claim.
Revised: This proves the second part of the claim.

Original: Therefore, G ∗ id = G ∗ eq .
Revised: Therefore, G ∗ id = G ∗ eq .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We showed in Theorem 1 that the only source of non-identiﬁability are the bijections, a , in the set E ; our task here is to explicitly ﬁnd all of these bijections for afﬁne mechanisms.
Revised: We showed in Theorem 1 that the only source of non-identiﬁability are the bijections, a , in the set E ; our task here is to explicitly ﬁnd all of these bijections for afﬁne mechanisms.

Original: If a ∈ E , then it satisﬁes a ◦ m = m ◦ a .
Revised: If a ∈ E , then it satisﬁes a ◦ m = m ◦ a .

Original: We replace m with the afﬁne mechanism to obtain the following condition.
Revised: We replace m with the afﬁne mechanism to obtain the following condition.

Original: For each z ∈ R d
Revised: For each z ∈ R d

Original: [Equation]
Revised: [Equation]

Original: [Equation] Next, recall that a : R d → R d ; we take gradient of the function in the LHS and RHS of the above equation 26 separately w.r.t z .
Revised: Next, recall that a : R d → R d ; we take gradient of the function in the LHS and RHS of the above equation 26 separately w.r.t z .

Original: Consider the j th component of a ( Mz + b ) denoted as a j ( Mz + b ) .
Revised: Consider the j th component of a ( Mz + b ) denoted as a j ( Mz + b ) .

Original: We ﬁrst take the gradient of a j ( Mz + b ) w.r.t z
Revised: We ﬁrst take the gradient of a j ( Mz + b ) w.r.t z

Original: [Equation]
Revised: [Equation]

Original: where y =
Revised: where y =

Original: Mz + b , ∇ y a j ( y ) is the gradient of a j w.r.t y and dydz denotes the Jacobian of y w.r.t z .
Revised: Mz + b , ∇ y a j ( y ) is the gradient of a j w.r.t y and dydz denotes the Jacobian of y w.r.t z .

Original: We simplify the above further to get
Revised: We simplify the above further to get

Original: [Equation]
Revised: [Equation]

Original: We can write the above for each component of a as follows.
Revised: We can write the above for each component of a as follows.

Original: [Equation]
Revised: [Equation]

Original: where J ( Mz + b ) is the Jacobian of a computed at Mz + b .
Revised: where J ( Mz + b ) is the Jacobian of a computed at Mz + b .

Original: Next, we take the gradient of the j th component of the RHS in equation 26 and let m j denote the j th column of M ,
Revised: Next, we take the gradient of the j th component of the RHS in equation 26 and let m j denote the j th column of M ,

Original: [Equation]
Revised: [Equation]

Original: We can write the above for each component in the RHS of equation 26 as follows
Revised: We can write the above for each component in the RHS of equation 26 as follows

Original: [Equation]
Revised: [Equation]

Original: We equate the gradient of LHS and RHS in equation 26 using the expressions derived in equationand equation 31 to obtain
Revised: We equate the gradient of LHS and RHS in equation 26 using the expressions derived in equationand equation 31 to obtain

Original: [Equation]
Revised: [Equation]

Original: We write the same expression at another offset b (cid:48) (cid:54) =
Revised: We write the same expression at another offset b (cid:48) (cid:54) =

Original: b below
Revised: b below

Original: [Equation]
Revised: [Equation]

Original: Taking the difference of equation 32 and equation 33 we get M T J T ( Mz + b ) =
Revised: Taking the difference of equation 32 and equation 33 we get M T J T ( Mz + b ) =

Original: M T J T ( Mz + b
Revised: M T J T ( Mz + b

Original: [Equation]
Revised: [Equation]

Original: Since M is invertible
Revised: Since M is invertible

Original: [Equation]
Revised: [Equation]

Original: we get J ( Mz + b ) =
Revised: we get J ( Mz + b ) =

Original: J ( Mz + b (cid:48) ) .
Revised: J ( Mz + b (cid:48) ) .

Original: Consider row j of this identity.
Revised: Consider row j of this identity.

Original: For each z ∈ R d
Revised: For each z ∈ R d

Revised: [Equation]

Revised: [Equation]

Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: where ∇ 2 a j is the Hessian of a j and ∇ 2 k a j ( θ k ) corresponds to the k th row of the Hessian matrix.
Revised: where ∇ 2 a j is the Hessian of a j and ∇ 2 k a j ( θ k ) corresponds to the k th row of the Hessian matrix.

Original: Note that in the above expansion there is a different θ k for each row (mean value theorem applied to each component of ∇ a j yields a different point θ k on the line joinining ˜ z and
Revised: Note that in the above expansion there is a different θ k for each row (mean value theorem applied to each component of ∇ a j yields a different point θ k on the line joinining ˜ z and

Original: ˜ z
Revised: ˜ z

Original: + b − b (cid:48) .
Revised: + b − b (cid:48) .

Original: From
Revised: From

Original: Assumption 2 and based on the fact that M is invertible, it follows that ∇ 2 k a j ( θ k )( b − b (cid:48) )
Revised: Assumption 2 and based on the fact that M is invertible, it follows that ∇ 2 k a j ( θ k )( b − b (cid:48) )

Original: = 

Revised: =

Original: over a measurable set.
Revised: over a measurable set.

Original: Since a j is analytic ∇ 2 k a j ( z )( b − b (cid:48) ) is also analytic.
Revised: Since a j is analytic ∇ 2 k a j ( z )( b − b (cid:48) ) is also analytic.

Original: Therefore, from (Mityagin, 2015), we can conclude that ∇ 2 k a j ( z )( b − b (cid:48) )
Revised: Therefore, from (Mityagin, 2015), we can conclude that ∇ 2 k a j ( z )( b − b (cid:48) )

Original: = 

Revised: =

Original: for all z .
Revised: for all z .

Original: We can make the same argument for each component k and conclude that ∇ 2 a j ( z )( b − b (cid:48) )
Revised: We can make the same argument for each component k and conclude that ∇ 2 a j ( z )( b − b (cid:48) )

Original: = 0 .
Revised: = 0 .

Original: From Assumption 1, it follows that ∇ 2 a j ( z )( b j − b 1 ) = 0 for all j ∈ { 2 , · · · , d + 1 } and since the set { b 2 − b 1 , · · · , b d +1 − b 1 } is linearly independent ∇ 2 a j ( z ) = 0 for all z .
Revised: From Assumption 1, it follows that ∇ 2 a j ( z )( b j − b 1 ) = 0 for all j ∈ { 2 , · · · , d + 1 } and since the set { b 2 − b 1 , · · · , b d +1 − b 1 } is linearly independent ∇ 2 a j ( z ) = 0 for all z .

Original: This implies a ( z ) =
Revised: This implies a ( z ) =

Original: Az + p .
Revised: Az + p .

Original: Plug a ( z ) =
Revised: Plug a ( z ) =

Original: Az + p into a ( Mz + b ) =
Revised: Az + p into a ( Mz + b ) =

Original: Ma ( z ) + b to get
Revised: Ma ( z ) + b to get

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We write the same expression for offset b (cid:48)
Revised: We write the same expression for offset b (cid:48)

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We take the difference of equation 35 and equation 36 to get
Revised: We take the difference of equation 35 and equation 36 to get

Original: [Equation]
Revised: [Equation]

Original: Substitute z = 0 in equation 35 to get
Revised: Substitute z = 0 in equation 35 to get

Original: [Equation]
Revised: [Equation]

Original: Substitute the above condition in equation 38 into equation 35 to get the following.
Revised: Substitute the above condition in equation 38 into equation 35 to get the following.

Original: For each z
Revised: For each z

Original: [Equation]
Revised: [Equation]

Original: We can now leverage the proofs from the linear setting in Section A.2.
Revised: We can now leverage the proofs from the linear setting in Section A.2.

Original: The above equation 39 is the same as equation 17 and the equation 37 is the same as equation 22, with b replaced by b − b (cid:48) .
Revised: The above equation 39 is the same as equation 17 and the equation 37 is the same as equation 22, with b replaced by b − b (cid:48) .

Original: Following the same analysis as before, we get that latent variables are exactly identiﬁed; we show all the steps below for completeness.
Revised: Following the same analysis as before, we get that latent variables are exactly identiﬁed; we show all the steps below for completeness.

Original: By choosing d linearly independent z and substituting in equation 39 we get the following,
Revised: By choosing d linearly independent z and substituting in equation 39 we get the following,

Original: [Equation]
Revised: [Equation]

Original: where C = S − 1 AS , M = S Λ S − 1 , Λ = diag (cid:16) λ 1 , · · · , λ d (cid:17) .
Revised: where C = S − 1 AS , M = S Λ S − 1 , Λ = diag (cid:16) λ 1 , · · · , λ d (cid:17) .

Original: We further simplify equationand compare each element of the matrix Λ C and C Λ to get the following condition.
Revised: We further simplify equationand compare each element of the matrix Λ C and C Λ to get the following condition.

Original: For all i, j ∈ { 1 , · · · , d }
Revised: For all i, j ∈ { 1 , · · · , d }

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Consider the above equation 41 for i (cid:54) =
Revised: Consider the above equation 41 for i (cid:54) =

Original: j to get
Revised: j to get

Original: [Equation]
Revised: [Equation]

Original: we use the assumption that λ
Revised: we use the assumption that λ

Original: i (cid:54)
Revised: i (cid:54)

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: From the above, it follows that C is a diagonal matrix.
Revised: From the above, it follows that C is a diagonal matrix.

Original: We obtain an expression for A in terms of C matrix below.
Revised: We obtain an expression for A in terms of C matrix below.

Original: [Equation]
Revised: [Equation]

Original: From equation 37 we get that
Revised: From equation 37 we get that

Original: [Equation]
Revised: [Equation]

Original: [Table] S −[Equation]
Revised: Left multiply S −

Original: [Equation]
Revised: [Equation]

Original: Since C is a diagonal matrix, we can simplify the above condition further to get ( c ii − 1)( S − 1 ( b − b (cid:48) ))
Revised: Since C is a diagonal matrix, we can simplify the above condition further to get ( c ii − 1)( S − 1 ( b − b (cid:48) ))

Original: i = 0 .
Revised: i = 0 .

Original: Since ( S − 1 ( b − b (cid:48) ))
Revised: Since ( S − 1 ( b − b (cid:48) ))

Original: i (cid:54) = 

Revised: i (cid:54) =

Original: = ⇒ c ii = 1 , we obtain C
Revised: = ⇒ c ii = 1 , we obtain C

Original: = I = ⇒
Revised: = I = ⇒

Original: A
Revised: A

Original: = I = ⇒ a ( z )
Revised: = I = ⇒ a ( z )

Original: =
Revised: =

Original: z + p .
Revised: z + p .

Original: This proves that the latents are identiﬁed up to an offset.
Revised: This proves that the latents are identiﬁed up to an offset.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that ˜ G id ⊆ ˜ G eq .
Revised: We ﬁrst show that ˜ G id ⊆ ˜ G eq .

Original: Consider a ˜ g ∈ ˜
Revised: Consider a ˜ g ∈ ˜

Original: G id .
Revised: G id .

Original: We rewrite equation 6 below.
Revised: We rewrite equation 6 below.

Original: For all x ∈ X
Revised: For all x ∈ X

Original: [Equation]
Revised: [Equation]

Original: Since g is bijective, we can write x = g ( z ) to get
Revised: Since g is bijective, we can write x = g ( z ) to get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the above equality holds for all z ∈ Z we can conclude that
Revised: Since the above equality holds for all z ∈ Z we can conclude that

Original: [Equation]
Revised: [Equation]

Original: The above conclusion in equation 47 holds for all m t ∈ M ∗ .
Revised: The above conclusion in equation 47 holds for all m t ∈ M ∗ .

Original: Therefore, a in equation 47 and { ˜ m t } Tt =1 (where ˜ m t ∈ M ) together satisfy the condition that for all m ∈ M ∗ , a ◦ m = ˜ m ◦ a , where ˜ m ∈ M .
Revised: Therefore, a in equation 47 and { ˜ m t } Tt =1 (where ˜ m t ∈ M ) together satisfy the condition that for all m ∈ M ∗ , a ◦ m = ˜ m ◦ a , where ˜ m ∈ M .

Original: We can rewrite ˜ g − 1 ◦ g = a as ˜ g
Revised: We can rewrite ˜ g − 1 ◦ g = a as ˜ g

Original: = g ◦ a − 1 .
Revised: = g ◦ a − 1 .

Original: From this it follows that ˜ g ∈ ˜ G eq .
Revised: From this it follows that ˜ g ∈ ˜ G eq .

Original: This proves the ﬁrst part of the theorem.
Revised: This proves the ﬁrst part of the theorem.

Original: Now let us consider the second part of the theorem.
Revised: Now let us consider the second part of the theorem.

Original: Consider a ˜ g ∈ ˜ G eq .
Revised: Consider a ˜ g ∈ ˜ G eq .

Original: We can write ˜ g
Revised: We can write ˜ g

Original: = g ◦ a − 1 , where a ∈ ˜ E .
Revised: = g ◦ a − 1 , where a ∈ ˜ E .

Original: At time t , some mechanism m t ∈ M ∗ is used to transform the latents.
Revised: At time t , some mechanism m t ∈ M ∗ is used to transform the latents.

Original: Since a ∈ ˜ E , select the mechanism ˜ m t ∈ M for which a ◦ m t =
Revised: Since a ∈ ˜ E , select the mechanism ˜ m t ∈ M for which a ◦ m t =

Original: ˜ m
Revised: ˜ m

Original: t ◦ a and as a consequence
Revised: t ◦ a and as a consequence

Original: [Equation]
Revised: [Equation]

Original: In the ﬁrst equality above, we use a ◦ m t = ˜ m t ◦ a and in the second equality we use the deﬁnition of ˜ g .
Revised: In the ﬁrst equality above, we use a ◦ m t = ˜ m t ◦ a and in the second equality we use the deﬁnition of ˜ g .

Original: We can repeat the above exercise for all t and corresponding m t using the same a .
Revised: We can repeat the above exercise for all t and corresponding m t using the same a .

Original: Therefore, ˜ g is in ˜ G id .
Revised: Therefore, ˜ g is in ˜ G id .

Original: This shows the second part of the theorem, i.e., ˜ G eq ⊆
Revised: This shows the second part of the theorem, i.e., ˜ G eq ⊆

Original: ˜ G id .
Revised: ˜ G id .

Original: A. L EVERAGING T HEOREM 3 WHEN MECHANISM IS LINEAR AND g IS NON - LINEAR
Revised: A. L EVERAGING T HEOREM 3 WHEN MECHANISM IS LINEAR AND g IS NON - LINEAR

Original: We write the data generation process as follows.
Revised: We write the data generation process as follows.

Original: For each t ∈ { 1 , · · · , d + 1 } z t +1 =
Revised: For each t ∈ { 1 , · · · , d + 1 } z t +1 =

Original: M t z t + b t , 
Revised: M t z t + b t , x t

Original: [Equation]
Revised: [Equation]

Original: g ( z t ) , (49)
Revised: g ( [Equation] t ) , (49)

Original: Let us construct the set ˜ E corresponding to the above setting.
Revised: Let us construct the set ˜ E corresponding to the above setting.

Original: For each z ∈ R d ,
Revised: For each z ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where ( M, b ) and ( M, b (cid:48) ) are the true mechanisms and ( M (cid:48) , b (cid:48) ) and ( M (cid:48) , ˜ b (cid:48) ) are the imtitating mechanisms chosen by the learner, J is the Jacobian of a .
Revised: where ( M, b ) and ( M, b (cid:48) ) are the true mechanisms and ( M (cid:48) , b (cid:48) ) and ( M (cid:48) , ˜ b (cid:48) ) are the imtitating mechanisms chosen by the learner, J is the Jacobian of a .

Original: Note here the learner only exploits the knowledge that b changes to ˜ b , which is why it keeps M (cid:48) ﬁxed and only changes the offset.
Revised: Note here the learner only exploits the knowledge that b changes to ˜ b , which is why it keeps M (cid:48) ﬁxed and only changes the offset.

Original: We take the difference of the RHS in the above two equations to get
Revised: We take the difference of the RHS in the above two equations to get

Original: [Equation]
Revised: [Equation]

Original: Since M is invertible we get J ( Mz + b ) − J ( Mz + b (cid:48) )
Revised: Since M is invertible we get J ( Mz + b ) − J ( Mz + b (cid:48) )

Original: = 

Revised: =

Original: for all z .
Revised: for all z .

Original: We can follow the same justiﬁcation as was used in equation 33 to conclude that J ( z ) is constant and a is thus an afﬁne map.
Revised: We can follow the same justiﬁcation as was used in equation 33 to conclude that J ( z ) is constant and a is thus an afﬁne map.

Original: We substitue the afﬁne map a ( z ) =
Revised: We substitue the afﬁne map a ( z ) =

Original: Az + p back into equation 50 to get the following.
Revised: Az + p back into equation 50 to get the following.

Original: For all z
Revised: For all z

Original: [Equation]
Revised: [Equation]

Original: Substitute z = 0 to get b (cid:48) ,j =
Revised: Substitute z = 0 to get b (cid:48) ,j =

Original: Ab j + p − M (cid:48) p .
Revised: Ab j + p − M (cid:48) p .

Original: Substitute this condition back into the above equation, we get AM =
Revised: Substitute this condition back into the above equation, we get AM =

Original: M (cid:48)
Revised: M (cid:48)

Original: A = ⇒ M (cid:48) =
Revised: A = ⇒ M (cid:48) =

Original: AMA − 1 .
Revised: AMA − 1 .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Before stating the proof of Theorem 4, we state two existing results that we use.
Revised: Before stating the proof of Theorem 4, we state two existing results that we use.

Original: Result 1.
Revised: Result 1.

Original: (Change of variables formula (DeGroot, 2012))
Revised: (Change of variables formula (DeGroot, 2012))

Original: Given a continuous random variable
Revised: Given a continuous random variable

Original: X ∈ R d with pdf p X and its transformation Y = f
Revised: X ∈ R d with pdf p X and its transformation Y = f

Original: ( X ) , where f : R d → R d is a diffeomorphism,then p Y ( f ( x )) | det ( J f ( x ))
Revised: ( X ) , where f : R d → R d is a diffeomorphism,then p Y ( f ( x )) | det ( J f ( x ))

Original: | =
Revised: | =

Original: p X ( x ) , where J f is the Jacobian of f computed at x .
Revised: p X ( x ) , where J f is the Jacobian of f computed at x .

Original: Lemma 1.
Revised: Lemma 1.

Original: If X and Y are two continuous random variables that take values in R d that are equal in distribution, i.e., X d = Y .
Revised: If X and Y are two continuous random variables that take values in R d that are equal in distribution, i.e., X d = Y .

Original: If f : R d → R d is a diffeomorphism, then f ( X ) d
Revised: If f : R d → R d is a diffeomorphism, then f ( X ) d

Original: = f ( Y ) .
Revised: = f ( Y ) .

Original: Proof.
Revised: Proof.

Original: Since X and Y are equal in distribution, they have the same pdfs, i.e. p X ( x ) =
Revised: Since X and Y are equal in distribution, they have the same pdfs, i.e. p X ( x ) =

Original: p Y ( x ) for all x ∈ R d .
Revised: p Y ( x ) for all x ∈ R d .

Original: We can use the change of variables formula in Resut 1 above to get the following.
Revised: We can use the change of variables formula in Resut 1 above to get the following.

Original: Let W = f
Revised: Let W = f

Original: ( X ) , p X ( f − 1 ( w ))
Revised: ( X ) , p X ( f − 1 ( w ))

Original: | det ( J f − 1 ( w ))
Revised: | det ( J f − 1 ( w ))

Original: | = p W ( w ) and let V = f ( Y ) , p Y ( f − 1 ( v ))
Revised: | = p W ( w ) and let V = f ( Y ) , p Y ( f − 1 ( v ))

Original: | det ( J f − 1 ( v )) | = p V ( v ) .
Revised: | det ( J f − 1 ( v )) | = p V ( v ) .

Original: Comparing the two expressions when w = v we get p W ( w )
Revised: Comparing the two expressions when w = v we get p W ( w )

Original: =
Revised: =

Original: p V ( w ) .
Revised: p V ( w ) .

Original: This proves the result.
Revised: This proves the result.

Original: We stated Result 1 and Lemma 1 for continuous random variables.
Revised: We stated Result 1 and Lemma 1 for continuous random variables.

Original: When the random variables are discrete, Lemma 1 holds for any function f .
Revised: When the random variables are discrete, Lemma 1 holds for any function f .

Original: Lemma 2.
Revised: Lemma 2.

Original: (Kass & Vos, 2011)
Revised: (Kass & Vos, 2011)

Original: If f : Z → Z and g : Z → Z are diffeomorphisms, then f ◦ g is a diffeomorphism.
Revised: If f : Z → Z and g : Z → Z are diffeomorphisms, then f ◦ g is a diffeomorphism.

Original: Proof.
Revised: Proof.

Original: We ﬁrst show that G sid ⊆
Revised: We ﬁrst show that G sid ⊆

Original: G seq .
Revised: G seq .

Original: From the observation identity in equation 8 we get that ˜ g, { ˜ m t } Tt =1 satisfy the following for all t ∈ { 1 , · · · , T }
Revised: From the observation identity in equation 8 we get that ˜ g, { ˜ m t } Tt =1 satisfy the following for all t ∈ { 1 , · · · , T }

Original: [Equation]
Revised: [Equation]

Original: In the second step in the above equation
Revised: In the second step in the above equation

Original: [Equation]
Revised: [Equation]

Original: we transformed the random variables in the ﬁrst step using the same transform ˜ g − 1 .
Revised: we transformed the random variables in the ﬁrst step using the same transform ˜ g − 1 .

Original: ˜ g − 1 is a diffeomorphism; we compose both sides of the ﬁrst step LHS and
Revised: ˜ g − 1 is a diffeomorphism; we compose both sides of the ﬁrst step LHS and

Original: RHS with ˜ g − 1 .
Revised: RHS with ˜ g − 1 .

Original: We use Lemma 1 to get from the ﬁrst step to the second step in the above equation equation 53.
Revised: We use Lemma 1 to get from the ﬁrst step to the second step in the above equation equation 53.

Original: In the above equation x t is a ﬁxed value and the only source of randomness is from
Revised: In the above equation x t is a ﬁxed value and the only source of randomness is from

Original: U t in LHS and ˆ U t in the RHS.
Revised: U t in LHS and ˆ U t in the RHS.

Original: We substitute x t = g ( z t ) to further simplify the above expression in equation
Revised: We substitute x t = g ( z t ) to further simplify the above expression in equation

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substitute a = ˜ g − 1 ◦ g in the above to get the following
Revised: Substitute a = ˜ g − 1 ◦ g in the above to get the following

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: From Lemma 2 it follows that a in the above is a diffeomorphism.
Revised: From Lemma 2 it follows that a in the above is a diffeomorphism.

Original: Since we assume that ∪ Tt =1 { m t } =
Revised: Since we assume that ∪ Tt =1 { m t } =

Original: M ∗ it follows that a in equation 47 and { ˜ m t } Tt =1 (where ˜ m t ∈ M ) together satisfy the condition for membership in E s .
Revised: M ∗ it follows that a in equation 47 and { ˜ m t } Tt =1 (where ˜ m t ∈ M ) together satisfy the condition for membership in E s .

Original: Since ˜ g = g ◦ a − 1 we obtain that ˜ g ∈ G seq .
Revised: Since ˜ g = g ◦ a − 1 we obtain that ˜ g ∈ G seq .

Original: We now show that G seq ⊆
Revised: We now show that G seq ⊆

Original: G sid .
Revised: G sid .

Original: Consider a ˜ g ∈ G seq .
Revised: Consider a ˜ g ∈ G seq .

Original: We use ˜ g
Revised: We use ˜ g

Original: = g ◦ a − 1 , where a ∈ E s to simplify the following random variable pdf
Revised: = g ◦ a − 1 , where a ∈ E s to simplify the following random variable pdf

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since a ∈ E s we have
Revised: Since a ∈ E s we have

Original: [Equation]
Revised: [Equation]

Original: From Lemma 2 it follows that ˜ g is a diffeomorphism.
Revised: From Lemma 2 it follows that ˜ g is a diffeomorphism.

Original: From Lemma 1 it follows that
Revised: From Lemma 1 it follows that

Original: [Equation]
Revised: [Equation]

Original: Combining equation 56 and equation 57 and using the fact that ˆ U t d = U t we get
Revised: Combining equation 56 and equation 57 and using the fact that ˆ U t d = U t we get

Original: [Equation]
Revised: [Equation]

Original: From the deﬁnition of E s it follows with the same choice of a the condition continues to hold for all m t ∈ M ∗ .
Revised: From the deﬁnition of E s it follows with the same choice of a the condition continues to hold for all m t ∈ M ∗ .

Original: Therefore, ˜ g ∈ G sid .
Revised: Therefore, ˜ g ∈ G sid .

Original: This proves the second part of the theorem.
Revised: This proves the second part of the theorem.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: In Theorem 4, we showed that all the solutions to the observation identity in equation 8 can be characterized in terms of the equivariances in distribution deﬁned by the set E s .
Revised: In Theorem 4, we showed that all the solutions to the observation identity in equation 8 can be characterized in terms of the equivariances in distribution deﬁned by the set E s .

Original: Let us analyze the set E s for the class of mechanisms considered in Klindt et al.
Revised: Let us analyze the set E s for the class of mechanisms considered in Klindt et al.

Original: Consider a a ∈ E s .
Revised: Consider a a ∈ E s .

Original: For each z ∈ R d
Revised: For each z ∈ R d

Original: [Equation]
Revised: [Equation]

Original: Deﬁne ˆ Y = a ( z ) + ˆ V .
Revised: Deﬁne ˆ Y = a ( z ) + ˆ V .

Original: Since ˆ V d = V
Revised: Since ˆ V d = V

Original: we write the probability density function (pdf) of ˆ Y as
Revised: we write the probability density function (pdf) of ˆ Y as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Deﬁne Y =
Revised: Deﬁne Y =

Original: a ( z + V ) .
Revised: a ( z + V ) .

Original: a : R d → R d is a diffeomorphism.
Revised: a : R d → R d is a diffeomorphism.

Original: We use the change of variables result (Result 1) to write the pdf Y as follows.
Revised: We use the change of variables result (Result 1) to write the pdf Y as follows.

Original: For each y ∈ R d
Revised: For each y ∈ R d

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where J ( a − 1 ( y )) is the Jacobian of a computed at a − 1 ( y ) , and det is the determinant.
Revised: where J ( a − 1 ( y )) is the Jacobian of a computed at a − 1 ( y ) , and det is the determinant.

Original: We substitute equation 60 and equation 61 in the equivariance condition in equation 59 to obtain the following.
Revised: We substitute equation 60 and equation 61 in the equivariance condition in equation 59 to obtain the following.

Original: For each y ∈ R d
Revised: For each y ∈ R d

Original: [Equation]
Revised: [Equation]

Original: where w = a − 1 ( y ) .
Revised: where w = a − 1 ( y ) .

Original: In the above we equated the conditionals for each z , we now equate the marginals.
Revised: In the above we equated the conditionals for each z , we now equate the marginals.

Original: [Equation]
Revised: [Equation]

Original: We follow Klindt et al.
Revised: We follow Klindt et al.

Original: and assume ˆ Z d = Z and ˆ V d = V .
Revised: and assume ˆ Z d = Z and ˆ V d = V .

Original: Therefore, Z + V d
Revised: Therefore, Z + V d

Original: = ˆ Z
Revised: = ˆ Z

Original: + ˆ V .
Revised: + ˆ V .

Original: We use this condition to restate equation 63 as
Revised: We use this condition to restate equation 63 as

Original: [Equation]
Revised: [Equation]

Original: where W = Z + V .
Revised: where W = Z + V .

Original: We translate equation 64 into the condition on the pdfs as follows.
Revised: We translate equation 64 into the condition on the pdfs as follows.

Original: For each w ∈ 
Revised: For each w ∈ R

Original: [Equation]
Revised: [Equation]

Original: Substituting the above equation equation 65 into equation 62 we get
Revised: Substituting the above equation equation 65 into equation 62 we get

Original: [Equation]
Revised: [Equation]

Original: where in the last condition in the above expression we exploit the fact that f V is a generalized
Revised: where in the last condition in the above expression we exploit the fact that f V is a generalized

Original: Laplacian distribution.
Revised: Laplacian distribution.

Original: From Mazur-Ulam theorem Nica (2013) it follows that a is afﬁne.
Revised: From Mazur-Ulam theorem Nica (2013) it follows that a is afﬁne.

Original: We now write a as a matrix A with offset vector q and simplify the condition in equation 59.
Revised: We now write a as a matrix A with offset vector q and simplify the condition in equation 59.

Original: For each z ∈ R d we have
Revised: For each z ∈ R d we have

Original: [Equation]
Revised: [Equation]

Original: Since A is a square matrix and AA T it follows that
Revised: Since A is a square matrix and AA T it follows that

Original: A T A
Revised: A T A

Original: = I .
Revised: = I .

Original: Therefore, A is an orthonormal matrix.
Revised: Therefore, A is an orthonormal matrix.

Original: Observe that all the elements of ˆ V are independent.
Revised: Observe that all the elements of ˆ V are independent.

Original: Since AV d = ˆ V it follows that all the elements of AV are also independent.
Revised: Since AV d = ˆ V it follows that all the elements of AV are also independent.

Original: Deﬁne AV = Q .
Revised: Deﬁne AV = Q .

Original: Observe that A is an orthonormal matrix that is multiplied with a vector V with all independent elements (each of which is non-Gaussian as α (cid:54) = 2 ) and outputs a vector that has all independent components.
Revised: Observe that A is an orthonormal matrix that is multiplied with a vector V with all independent elements (each of which is non-Gaussian as α (cid:54) = 2 ) and outputs a vector that has all independent components.

Original: From Theorem 11 in Comon (1994) we get that A is a composition of permutation and scaling.
Revised: From Theorem 11 in Comon (1994) we get that A is a composition of permutation and scaling.

Original: Since A is also orthonormal, each term in the diagonal scaling matrix can only be 1 or − 1 .
Revised: Since A is also orthonormal, each term in the diagonal scaling matrix can only be 1 or − 1 .

Original: Therefore, A = ΠΛ , where Π is a permutation matrix and Λ is a diagonal matrix with +1 , − 1 elements.
Revised: Therefore, A = ΠΛ , where Π is a permutation matrix and Λ is a diagonal matrix with +1 , − 1 elements.

Original: Finally, a ( z ) =
Revised: Finally, a ( z ) =

Original: ΠΛ z + q .
Revised: ΠΛ z + q .

Original: A. A LTERNATIVE IDENTIFICATION RESULT FOR SMALL TRANSITIONS
Revised: A. A LTERNATIVE IDENTIFICATION RESULT FOR SMALL TRANSITIONS

Original: In this section, we analyze time series models similar to one in Hyvarinen & Morioka (2017) under the condition that the transitions are small in magnitude to arrive at permutation and scaling based identiﬁcation.
Revised: In this section, we analyze time series models similar to one in Hyvarinen & Morioka (2017) under the condition that the transitions are small in magnitude to arrive at permutation and scaling based identiﬁcation.

Original: Let us analyze the set E s for this class of mechanisms.
Revised: Let us analyze the set E s for this class of mechanisms.

Original: We assume that the learner knows that the mechanism is additive, and that the noise components are all independent.
Revised: We assume that the learner knows that the mechanism is additive, and that the noise components are all independent.

Original: In the analysis below we consider bijections that are analytic (each component of the bijection is an analytic function).
Revised: In the analysis below we consider bijections that are analytic (each component of the bijection is an analytic function).

Original: Consider an a ∈ E s
Revised: Consider an a ∈ E s

Original: [Equation]
Revised: [Equation]

Original: We write the ﬁrst-order approximation of the above identity below
Revised: We write the ﬁrst-order approximation of the above identity below

Original: [Equation]
Revised: [Equation]

Original: where V d (cid:54) =
Revised: where V d (cid:54) =

Original: ˆ V .
Revised: ˆ V .

Original: Note that the set of solutions a to equation 68 and equation 69 become equal in the limit of δ → 0 , where δ is the bound on each component of | V | .
Revised: Note that the set of solutions a to equation 68 and equation 69 become equal in the limit of δ → 0 , where δ is the bound on each component of | V | .

Original: We analyze the solution to equation 69 below.
Revised: We analyze the solution to equation 69 below.

Original: [Equation]
Revised: [Equation]

Original: Since J ( z ) V d = ˆ V and each component of ˆ V is independent, we can deduce that all the components of J ( z ) V are independent as well.
Revised: Since J ( z ) V d = ˆ V and each component of ˆ V is independent, we can deduce that all the components of J ( z ) V are independent as well.

Original: From Theorem 11 in Comon (1994), we can deduce that J ( z ) is composed of permutation times a diagonal matrix.
Revised: From Theorem 11 in Comon (1994), we can deduce that J ( z ) is composed of permutation times a diagonal matrix.

Original: Since the matrix is orthonormal, each scaling component can only be ± 1 .
Revised: Since the matrix is orthonormal, each scaling component can only be ± 1 .

Original: We can apply this same analysis at another point ˜
Revised: We can apply this same analysis at another point ˜

Original: z in the neighborhood of z and continue to ﬁnd that the jacobian matrix is a permutation times diagonal matrix (that describes sign ﬂips).
Revised: z in the neighborhood of z and continue to ﬁnd that the jacobian matrix is a permutation times diagonal matrix (that describes sign ﬂips).

Original: Note that the permutation matrix times scaling used to express the Jacobian cannot change between the points ˜ z and z (if it does change then that violates the Jacobian’s continuity).
Revised: Note that the permutation matrix times scaling used to express the Jacobian cannot change between the points ˜ z and z (if it does change then that violates the Jacobian’s continuity).

Original: Since the
Revised: Since the

Original: Jacobian is equal to a ﬁxed permutation times a ﬁxed scaling matrix over a neighborhood, we can extend this to the entire space (here we use the fact that the a is analytic and Mityagin (2015)).
Revised: Jacobian is equal to a ﬁxed permutation times a ﬁxed scaling matrix over a neighborhood, we can extend this to the entire space (here we use the fact that the a is analytic and Mityagin (2015)).

Original: As a result, a is of the form ΠΛ z + q , where Π is a permutation matrix, Λ is a diagonal matrix.
Revised: As a result, a is of the form ΠΛ z + q , where Π is a permutation matrix, Λ is a diagonal matrix.

Original: A. A NALYZING AUXILIARY INFORMATION MODELS
Revised: A. A NALYZING AUXILIARY INFORMATION MODELS

Original: We now discuss how our machinery can be used in models when auxiliary information is available (Khemakhem et al.
Revised: We now discuss how our machinery can be used in models when auxiliary information is available (Khemakhem et al.

Original: (2020a))
Revised: (2020a))

Original: to arrive at permutation and scaling based identiﬁcation.
Revised: to arrive at permutation and scaling based identiﬁcation.

Original: We deﬁne the data generation process compatible with Khemakhem et al.
Revised: We deﬁne the data generation process compatible with Khemakhem et al.

Original: (2020a).
Revised: (2020a).

Original: Suppose the latent Z is generated from a mechanism m : O × [0 , 1] d that takes as input some observed auxiliary information
Revised: Suppose the latent Z is generated from a mechanism m : O × [0 , 1] d that takes as input some observed auxiliary information

Original: O and uniform independent noise vector U ∈ [0 , 1] d :
Revised: O and uniform independent noise vector U ∈ [0 , 1] d :

Original: Z ← m ( O, U ) ,
Revised: Z ← m ( O, U ) ,

Original: X ← g ( Z ) , (71) where g : Z → X is a bijection.
Revised: X ← g ( Z ) , (71) where g : Z → X is a bijection.

Original: Suppose the learner knows the mechanism.
Revised: Suppose the learner knows the mechanism.

Original: The learner selects ˜
Revised: The learner selects ˜

Original: g and outputs
Revised: g and outputs

Original: ˆ X = ˜
Revised: ˆ X = ˜

Original: g ◦ m ( o, ˜ U ) , where ˜ U is a random vector with each component sampled independently from Uniform [0 , 1] .
Revised: g ◦ m ( o, ˜ U ) , where ˜ U is a random vector with each component sampled independently from Uniform [0 , 1] .

Original: The learner’s goal is to match
Revised: The learner’s goal is to match

Original: [Equation]
Revised: [Equation]

Original: for all possible observations o ∈ O .
Revised: for all possible observations o ∈ O .

Original: Deﬁne the set of solutions to equation equation 72 as G oid .
Revised: Deﬁne the set of solutions to equation equation 72 as G oid .

Original: Consider bijection a s.t.
Revised: Consider bijection a s.t.

Original: the following identity holds for all o ∈ O
Revised: the following identity holds for all o ∈ O

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the set of all bijections a that satisfy the condition in equation equation 73 as E oeq .
Revised: Deﬁne the set of all bijections a that satisfy the condition in equation equation 73 as E oeq .

Original: The set E oeq consists of intersection of measure preserving transformations of Z | O
Revised: The set E oeq consists of intersection of measure preserving transformations of Z | O

Original: = o .
Revised: = o .

Original: Deﬁne the set of ˜ g
Revised: Deﬁne the set of ˜ g

Original: that are identiﬁable up to E oeq as G oeq =
Revised: that are identiﬁable up to E oeq as G oeq =

Original: { ˜ g | ˜
Revised: { ˜ g | ˜

Original: g = g ◦ a − 1 , a ∈ E oeq } .
Revised: g = g ◦ a − 1 , a ∈ E oeq } .

Original: Theorem 7.
Revised: Theorem 7.

Original: If the data is generated as described in equation 71, then the set of solutions to the identity in equation 72 identify true g up to intersection of all the measure preserving maps in E o eq ( ˜ g − 1 ∼ E oeq g − 1 )
Revised: If the data is generated as described in equation 71, then the set of solutions to the identity in equation 72 identify true g up to intersection of all the measure preserving maps in E oeq ( ˜ g − 1 ∼ E oeq g − 1 )

Original: Proof.
Revised: Proof.

Original: Consider a ˜ g ∈ G oid .
Revised: Consider a ˜ g ∈ G oid .

Original: [Equation]
Revised: [Equation]

Original: From the above it follows that ˜ g ∈ G oeq .
Revised: From the above it follows that ˜ g ∈ G oeq .

Original: Consider a ˜ g ∈ G oeq .
Revised: Consider a ˜ g ∈ G oeq .

Original: [Equation]
Revised: [Equation]

Original: From the above it follows that ˜ g ∈ G oid .
Revised: From the above it follows that ˜ g ∈ G oid .

Original: This completes the proof.
Revised: This completes the proof.

Original: Let us consider additive mechanisms of the form m ( o, U )
Revised: Let us consider additive mechanisms of the form m ( o, U )

Original: =
Revised: =

Original: ¯ m ( o ) + U , where U has independent components.
Revised: ¯ m ( o ) + U , where U has independent components.

Original: Suppose the learner knows that the mechanism is additive and the noise has independent components.
Revised: Suppose the learner knows that the mechanism is additive and the noise has independent components.

Original: The learner solves the following identity.
Revised: The learner solves the following identity.

Original: [Equation]
Revised: [Equation]

Original: Suppose that the absolute value of each component of U is really small and bounded by δ .
Revised: Suppose that the absolute value of each component of U is really small and bounded by δ .

Original: We can use the ﬁrst-order Taylor expansion and obtain
Revised: We can use the ﬁrst-order Taylor expansion and obtain

Original: [Equation]
Revised: [Equation]

Original: where J is the Jacobian of a .
Revised: where J is the Jacobian of a .

Original: Suppose the noise has zero mean.
Revised: Suppose the noise has zero mean.

Original: Take the expectation w.r.t.
Revised: Take the expectation w.r.t.

Original: U and ˜
Revised: U and ˜

Original: U on the two sides respectively to get a ( ¯ m ( o ))
Revised: U on the two sides respectively to get a ( ¯ m ( o ))

Original: =
Revised: =

Original: ¯ m (cid:48)
Revised: ¯ m (cid:48)

Original: ( o ) .
Revised: ( o ) .

Original: Substitute this back into the equation we get that J ( ¯ m ( o ))
Revised: Substitute this back into the equation we get that J ( ¯ m ( o ))

Original: U d = ˜
Revised: U d = ˜

Original: U .
Revised: U .

Original: We can now follow the analysis that we carried out after equation 69 and conclude that a is equal to permutation times a scaling matrix along with some offset.
Revised: We can now follow the analysis that we carried out after equation 69 and conclude that a is equal to permutation times a scaling matrix along with some offset.

Original: A. A NALYZING ˜ E WHEN M = M ∗
Revised: A. A NALYZING ˜ E WHEN M = M ∗

Original: In this section, we analyze imitators when we know the set of mechanisms that are deployed, we do not know which mechanism is used when.
Revised: In this section, we analyze imitators when we know the set of mechanisms that are deployed, we do not know which mechanism is used when.

Original: If a ∈ ˜ E and M = M ∗ .
Revised: If a ∈ ˜ E and M = M ∗ .

Original: From the deﬁnition of a , it follows that for each m ∈ M ∗ , ∃ m (cid:48) ∈
Revised: From the deﬁnition of a , it follows that for each m ∈ M ∗ , ∃ m (cid:48) ∈

Original: M ∗ such that a ◦ m = m (cid:48) ◦ a .
Revised: M ∗ such that a ◦ m = m (cid:48) ◦ a .

Original: We claim that two distinct m ∈ M ∗ cannot share the same m (cid:48)
Revised: We claim that two distinct m ∈ M ∗ cannot share the same m (cid:48)

Original: (imitator).
Revised: (imitator).

Original: Suppose there was a common m (cid:48)
Revised: Suppose there was a common m (cid:48)

Original: imitating m and ˜ m .
Revised: imitating m and ˜ m .

Original: [Equation]
Revised: [Equation]

Original: We take the difference of the above two equations to get
Revised: We take the difference of the above two equations to get

Original: [Equation]
Revised: [Equation]

Original: Since a is a bijection, we can conclude that m = ˜ m , which is a contradiction of the fact that m and ˜ m are distinct.
Revised: Since a is a bijection, we can conclude that m = ˜ m , which is a contradiction of the fact that m and ˜ m are distinct.

Original: This claim implies that for a given a there is an injective map from M ∗ to M ∗ .
Revised: This claim implies that for a given a there is an injective map from M ∗ to M ∗ .

Original: If the set M ∗ is ﬁnite, then from Pigeonhole principle it follows that this injective map is a bijection.
Revised: If the set M ∗ is ﬁnite, then from Pigeonhole principle it follows that this injective map is a bijection.

Original: Let us index the mechanism M ∗ = { m 1 , · · · , m n } .
Revised: Let us index the mechanism M ∗ = { m 1 , · · · , m n } .

Original: We call the bijection map π : M ∗ → M ∗
Revised: We call the bijection map π : M ∗ → M ∗

Original: Consider the element i .
Revised: Consider the element i .

Original: We claim ∃ l ∈ { 1 , · · · , n } π l ( i ) =
Revised: We claim ∃ l ∈ { 1 , · · · , n } π l ( i ) =

Original: i .
Revised: i .

Original: We write the chain starting from i as i → π ( i ) →
Revised: We write the chain starting from i as i → π ( i ) →

Original: π 2 ( i ) , · · · , π k ( i ) .
Revised: π 2 ( i ) , · · · , π k ( i ) .

Original: Since the chain ( π ( i ) → π 2 ( i ) , · · · , π k ( i ) ) has n steps there have to be at least two elements that are equal.
Revised: Since the chain ( π ( i ) → π 2 ( i ) , · · · , π k ( i ) ) has n steps there have to be at least two elements that are equal.

Original: Suppose p > q and π p ( i ) =
Revised: Suppose p > q and π p ( i ) =

Original: π q ( i 
Revised: π q ( i )

Original: [Equation]
Revised: [Equation]

Original: In the above at each step we use the fact that π is a bijection and that shows the claim that π l ( i ) =
Revised: In the above at each step we use the fact that π is a bijection and that shows the claim that π l ( i ) =

Original: i .
Revised: i .

Original: We now use this observation to carry out the following simpliﬁcation
Revised: We now use this observation to carry out the following simpliﬁcation

Original: [Equation]
Revised: [Equation]

Original: Substituting the second equation m π ( i ) into ﬁrst, and then the third m π 2 ( i ) and so on we get
Revised: Substituting the second equation m π ( i ) into ﬁrst, and then the third m π 2 ( i ) and so on we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, for each m i , ∃ k such that a k is its equivariance.
Revised: Therefore, for each m i , ∃ k such that a k is its equivariance.

Original: To summarize, if a ∈ ˜ E and M = M ∗ , where M is a ﬁnite set, then for each mechanism m ∈ M , ∃ k ∈ { 1 , · · · , |M|} such that a 
Revised: To summarize, if a ∈ ˜ E and M = M ∗ , where M is a ﬁnite set, then for each mechanism m ∈ M , ∃ k ∈ { 1 , · · · , |M|} such that a k is its equivariance.

Original: A.T RANSLATING IDENTITY INTO LOSS FUNCTIONS AND PRELIMINARY EXPERIMENTS
Revised: A.

Revised: T RANSLATING IDENTITY INTO LOSS FUNCTIONS AND PRELIMINARY EXPERIMENTS

Original: We describe twonatural loss choicesthat fall out based on the identitythat we want to solve ˜ g ◦ m ◦ ˜ g − 1 ( x t ) =
Revised: We restate the two loss choices based on our identity ˜ g ◦ m ◦ ˜ g − 1 ( x t ) =

Original: x t +1 .
Revised: x t +1 below.

Original: • Autoencoder based loss.
Revised: • Loss based on observations.

Original: The identity above immediately implies an autoencoder-style algorithm where one minimizes a reconstruction loss of the form
Revised: The identity above immediately implies an autoencoder-style algorithm where one minimizes a reconstruction loss of the form

Original: [Equation]
Revised: [Equation]

Revised: where H

Revised: is

Revised: the

Revised: hypothesis

Original: where H is the hypothesis class of functions for ˜ g and M is the hypothesis class of mechanisms.
Revised: class of functions for ˜ g and M is the hypothesis class of mecha nisms.

Original: • Contrastive loss.
Revised: • Loss based on latents.

Original: Alternatively, one could re-write the observation identity as g − 1 ◦ x t +1 = m ◦
Revised: Alternatively, one could re-write the observation identity as g − 1 ◦ x t +1 = m ◦

Original: g − 1 ◦ x t and use a contrastive loss as follows
Revised: g − 1 ◦ x t and use a contrastive loss as follows

Original: [Equation]
Revised: [Equation]

Original: where τ represents other time instances
Revised: where τ represents other time instances

Original: , i
Revised: , i

Original: [Equation]

Original: e., τ (cid:54) =
Revised: e., τ (cid:54) =

Original: t + 1 .
Revised: t + 1 .

Original: The positive pair in the contrastive loss is formed by the adjacent time instances and the negative pair is formed by non-adjacent time instances.
Revised: The positive pair in the contrastive loss is formed by the adjacent time instances and the negative pair is formed by non-adjacent time instances.

Original: Identifying which of these two losses works better in practice is an important empirical question.
Revised: Identifying which of these two losses works better in practice is an important empirical question.

Original: Note how in both the losses above, we have explicitly not enforced invertibility for the learned g.
Revised: Note how in both the losses above, we have explicitly not enforced invertibility for the learned g.

Original: We present our initial experiments on 3dIdent dataset from [Zimmerman et al. 2021], using the contrastive loss described above.
Revised: We present our initial experiments on 3dIdent dataset from [Zimmerman et al. 2021], using the contrastive loss described above.

Original: With contrastive pairs generated by a (ﬁxed) random orthogonal matrix U applied to the latents, we obtain the following values for linear disentanglement score ( Rof the predictions of the true representation using a linear model).
Revised: With contrastive pairs generated by a (ﬁxed) random orthogonal matrix U applied to the latents, we obtain the following values for linear disentanglement score ( Rof the predictions of the true representation using a linear model).

Original: We 
Revised: We report median scores overseeds.

Original: • Standard contrastive learning.
Revised: • Standard contrastive learning.

Original: Linear disentanglement score of 0.
Revised: Linear disentanglement score of 0.

Original: • Contrastive leveraging with exact mechanism knowledge.
Revised: • Contrastive leveraging with exact mechanism knowledge.

Original: Leveraging the true U as the mechanism in the contrastive loss achieves a median score of0.76.
Revised: Leveraging the true U as the mechanism in the contrastive loss achieves a median score of

Revised: [Table]

Original:  Contrastive leveraging with some knowledge of mechanism If we use a random orthogonal matrix ˜ U (cid:54) =
Revised: • Contrastive leveraging with some knowledge of mechanism If we use a random orthogonal matrix ˜ U (cid:54) =

Original: U , that achieves a median score of 0.64.
Revised: U , that achieves a median score of 0.64.

Original: The random matrix can be interpreted as sampling a random m from the hypothesis class M ; it seems likely that better results are possible by optimizing over M .
Revised: The random matrix can be interpreted as sampling a random m from the hypothesis class M ; it seems likely that better results are possible by optimizing over M .