Revised: Extended Abstract Track

Original: Homomorphism AutoEncoder — Learning Group Structured Representations from Observed Transitions
Revised: Homomorphism AutoEncoder — Learning Group Structured Representations from Observed Transitions

Original: Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane
Revised: Editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Arianna Di Bernardo, Nina Miolane

Original: Abstract
Revised: Abstract

Original: It is crucial for agents, both biological and artificial, to acquire world models that veridically represent the external world and how it transforms under the agent’s own actions.
Revised: It is crucial for agents, both biological and artificial, to acquire world models that veridically represent the external world and how it transforms under the agent’s own actions.

Original: We consider the case where such transformations are a subset of the group structuring the world state space.
Revised: We consider the case where such transformations are a subset of the group structuring the world state space.

Original: We use tools from representation learning and group theory to learn latent representations that account for both sensory information and the actions that alters it during interactions.
Revised: We use tools from representation learning and group theory to learn latent representations that account for both sensory information and the actions that alters it during interactions.

Original: We introduce the Homomorphism AutoEncoder (HAE), an autoencoder equipped with a learned group representation linearly acting on its latent space trained on 2-step transitions to implicitly enforce the group homomorphism property on the action representation.
Revised: We introduce the Homomorphism AutoEncoder (HAE), an autoencoder equipped with a learned group representation linearly acting on its latent space trained on 2-step transitions to implicitly enforce the group homomorphism property on the action representation.

Original: Compared to existing work, our approach makes fewer assumptions on the group representation and on which transformations the agent can sample from.
Revised: Compared to existing work, our approach makes fewer assumptions on the group representation and on which transformations the agent can sample from.

Original: We motivate our method theoretically, and demonstrate empirically that it can learn the correct representation of the groups and the topology of the environment.
Revised: We motivate our method theoretically, and demonstrate empirically that it can learn the correct representation of the groups and the topology of the environment.

Original: We also compare its performance in trajectory prediction with previous methods.
Revised: We also compare its performance in trajectory prediction with previous methods.

Original: 1. Introduction
Revised: 1. Introduction

Original: Developmental psychologist Piaget (1964) states that a representation on an object is an understanding of how it transforms under different interactions instead of a copy of it.
Revised: Developmental psychologist Piaget (1964) states that a representation on an object is an understanding of how it transforms under different interactions instead of a copy of it.

Original: These representations of the world are learned from interaction starting from the first stage of development: the sensorimotor phase and that these representations of the world are an understanding of how the world transforms under performed and imagined interventions.
Revised: These representations of the world are learned from interaction starting from the first stage of development: the sensorimotor phase, and that these representations of the world are an understanding of how the world transforms under performed and imagined interventions.

Original: The concept of group, as a mathematical structure, is pervasive in the description of the states and properties of the world.
Revised: The concept of group, as a mathematical structure, is pervasive in the description of the states and properties of the world.

Original: In physics, Noether’s theorem shows the continuous symmetries of a system correspond to conserved quantities (Thompson and Cook, 1995).
Revised: In physics, Noether’s theorem shows the continuous symmetries of a system correspond to conserved quantities (Thompson and Cook, 1995).

Original: Conversely, conserved quantities are associated with symmetries that only act on them while keeping everything else constant.
Revised: Conversely, conserved quantities are associated with symmetries that only act on them while keeping everything else constant.

Original: With the aim of representation learning being to recover these quantities from observed stimuli (Bengio et al., 2012), it is appropriate for the representation to transport the existing symmetries of the world states into the model.
Revised: With the aim of representation learning being to recover these quantities from observed stimuli (Bengio et al., 2012), it is appropriate for the representation to transport the existing symmetries of the world states into the model.

Original: Such a group-structured representation satisfies an equivariance property between the true generating factors and the learned representations.
Revised: Such a group-structured representation satisfies an equivariance property between the true generating factors and the learned representations.

Original: To observe these symmetries, an agent can perform motor interactions to intervene on the world’s state.
Revised: To observe these symmetries, an agent can perform motor interactions to intervene on the world’s state.

Original: These interactions also organize in a group which can be used to parametrize the symmetries of the real world.
Revised: These interactions also organize in a group which can be used to parametrize the symmetries of the real world.

Original: While one can mathematically create infinitely many such representations, an agent with bounded computational abilities needs to choose one allowing efficient manipulation, interpretation and prediction of changes in its environment.
Revised: While one can mathematically create infinitely many such representations, an agent with bounded computational abilities needs to choose one allowing efficient manipulation, interpretation and prediction of changes in its environment.

Original: A representational property compatible with this desideratum is disentanglement (Bengio et al., 2012;
Revised: A representational property compatible with this desideratum is disentanglement (Bengio et al., 2012;

Original: Kulkarni et al., 2015) which states the latent representation decomposes into parts reflecting the different interpretable properties of the environment that the agent can modify independently.
Revised: Kulkarni et al., 2015) which states the latent representation decomposes into parts reflecting the different interpretable properties of the environment that the agent can modify independently.

Original: While a group theoretic account of disentangled representations has been proposed by
Revised: While a group theoretic account of disentangled representations has been proposed by

Original: Higgins et al.
Revised: Higgins et al.

Original: without proposing a learning algorithm, few attempts have followed (Caselles-Dupr´e et al., 2019; Quessard et al., 2020), all assuming that the agent is able to separately intervene on independent factors of variation.
Revised: without proposing a learning algorithm, few attempts have followed (Caselles-Dupr´e et al., 2019; Quessard et al., 2020), all assuming that the agent is able to separately intervene on independent factors of variation.

Original: In this work, we propose the Homomorphism Autoencoder (HAE) framework to jointly learn a group representation of transitions between world states, as well as a group-structured disentangled representation of observations with minimal assumptions on the group or the actions the agent can sample from.
Revised: In this work, we propose the Homomorphism Autoencoder (HAE) framework to jointly learn a group representation of transitions between world states, as well as a group-structured disentangled representation of observations with minimal assumptions on the group or the actions the agent can sample from.

Original: We show theoretically and experimentally that the HAE learns the group structure of the set of transitions.
Revised: We show theoretically and experimentally that the HAE learns the group structure of the set of transitions.

Original: In addition, the HAE learns to separate the pose of an object from the identity of acted-on objects, which can be identified as orbits of the non-transitive group action.
Revised: In addition, the HAE learns to separate the pose of an object from the identity of acted-on objects, which can be identified as orbits of the non-transitive group action.

Original: 2. Background 2.1. Group-Structured Representation Learning
Revised: 2. Background 2.1. Group-Structured Representation Learning

Original: Following the group theoretic formalism introduced by Higgins et al.
Revised: Following the group theoretic formalism introduced by Higgins et al.

Original: (we refer to
Revised: (we refer to

Original: Appendix A for background on groups), we assume the set of observations O ⊂ R m is obtained from a set of world states W through an unknown generative process b :
Revised: Appendix A for background on groups), we assume the set of observations O ⊂ R m is obtained from a set of world states W through an unknown generative process b :

Original: W → O , that we assume to be injective for the scope of this work.
Revised: W → O , that we assume to be injective for the scope of this work.

Original: An inference process h : O → Z maps observations to their vector representations in Z ⊂ R k .
Revised: An inference process h : O → Z maps observations to their vector representations in Z ⊂ R k .

Original: A group of symmetries G structures the world states set by its action · W : G × W → W .
Revised: A group of symmetries G structures the world states set by its action · W : G × W → W .

Original: The different semantic properties of the world can each be thought of as induced by the  action of a subgroup in the direct product decomposition of G =
Revised: The different semantic properties of the world can each be thought of as induced by the Extended Abstract Track action of a subgroup in the direct product decomposition of G =

Original: G 1 × ...
Revised: G 1 × ...

Original: × G n .
Revised: × G n .

Original: Where, each subgroup only transforms a specific latent property while keeping all others constant.
Revised: Where, each subgroup only transforms a specific latent property while keeping all others constant.

Original: We focus in this manuscript on developing how we learn group structured representation and we leave discussion around disentangling the representation to the Appendices A.2 and C.
Revised: We focus in this manuscript on developing how we learn group structured representation and we leave discussion around disentangling the representation to the Appendices A.2 and C.

Original: A representation is a group-structured representation if the diagram in Figure 1 commutes.
Revised: A representation is a group-structured representation if the diagram in Figure 1 commutes.

Original: It then satisfies:
Revised: It then satisfies:

Original: There is a (non-trivial) action of G on Z , i
Revised: There is a (non-trivial) action of G on Z , i

Original: e., · Z : G ×
Revised: e., · Z : G ×

Original: Z
Revised: Z

Original: → Z. 2.
Revised: → Z. 2.

Original: The composition f = h ◦ b : W → Z is equivariant, meaning that transformations of W are reflected on Z , i.e., ∀ g ∈ G, w ∈ W,
Revised: The composition f = h ◦ b : W → Z is equivariant, meaning that transformations of W are reflected on Z , i.e., ∀ g ∈ G, w ∈ W,

Original: [Equation]
Revised: [Equation]

Original: A group action · Z : G × Z → Z induces a group homomorphism ρ : G → Sym ( Z ) where Sym ( Z ) is the group of invertible mappings from Z to itself (more in Appendix A.1).
Revised: A group action · Z : G × Z → Z induces a group homomorphism ρ : G → Sym ( Z ) where Sym ( Z ) is the group of invertible mappings from Z to itself (more in Appendix A.1).

Original: We require the group action · Z on Z to be linear, in which case, the induced homomorphism ρ : G → GL ( Z ) is called a group representation.
Revised: We require the group action · Z on Z to be linear, in which case, the induced homomorphism ρ : G → GL ( Z ) is called a group representation.

Original: 3. HAE Architecture and Derivation 3.1. The Homomorphism Autoencoder (HAE)
Revised: 3. HAE Architecture and Derivation 3.1. The Homomorphism Autoencoder (HAE)

Original: To jointly learn the latent representation h of the observations and the group representation ρ , we introduce the HAE, described in Figure1.
Revised: To jointly learn the latent representation h of the observations and the group representation ρ , we introduce the HAE, described in Figure

Original: An autoencoder equipped with a learnable mapping ρ : G → GL ( Z ) which maps an observed transition g to an invertible matrix ρ ( g ).
Revised: An autoencoder equipped with a learnable mapping ρ : G → GL ( Z ) which maps an observed transition g to an invertible matrix ρ ( g ).

Original: The obtained matrix transforms encoding vectors of observations
Revised: The obtained matrix transforms encoding vectors of observations

Original: z i =
Revised: z i =

Original: h
Revised: h

Original: ( o i ) to predict the encoding of future images.
Revised: ( o i ) to predict the encoding of future images.

Original: The latent prediction is evaluated on both the latent space through the latent prediction loss and on the image space through the reconstruction loss.
Revised: The latent prediction is evaluated on both the latent space through the latent prediction loss and on the image space through the reconstruction loss.

Original: γ is a scalar hyperparameter.
Revised: γ is a scalar hyperparameter.

Original: [Equation]
Revised: [Equation]

Original: We obtain the mapping ρ by first mapping g to the linear algebra
Revised: We obtain the mapping ρ by first mapping g to the linear algebra

Original: g = R k × k through a neural network ϕ which is then composed with the matrix exponential to obtain invertible matrices (See Appendix D.2.6).
Revised: g = R k × k through a neural network ϕ which is then composed with the matrix exponential to obtain invertible matrices (See Appendix D.2.6).

Original: We define the losses used throughout.
Revised: We define the losses used throughout.

Original: The latent prediction loss uses the group representation action to predict the evolution of stimuli encodings.
Revised: The latent prediction loss uses the group representation action to predict the evolution of stimuli encodings.

Original: [Equation]
Revised: [Equation]

Original: The reconstruction loss estimates the evolution of the observations from the predicted evolution of encodings.
Revised: The reconstruction loss estimates the evolution of the observations from the predicted evolution of encodings.

Original: The reconstruction loss also evaluates the reconstruction of the initial observation like a standard autoencoder.
Revised: The reconstruction loss also evaluates the reconstruction of the initial observation like a standard autoencoder.

Original: [Equation]
Revised: [Equation]

Original: We theoretically show in 3.2 how the proposed architecture leads to learning a groupstructured representation ( ρ, h ).
Revised: We theoretically show in 3.2 how the proposed architecture leads to learning a groupstructured representation ( ρ, h ).

Revised: Extended Abstract Track

Original: 3.2. The HAE Learns Group-Structured Representations
Revised: 3.2. The HAE Learns Group-Structured Representations

Original: In this section, we provide theoretical insights on learning group-structured representations and how the two-step HAE architecture achieves that with minimal assumptions.
Revised: In this section, we provide theoretical insights on learning group-structured representations and how the two-step HAE architecture achieves that with minimal assumptions.

Original: The one step latent prediction loss is simply enforcing the commutative diagram in Figure 1.
Revised: The one step latent prediction loss is simply enforcing the commutative diagram in Figure 1.

Original: With the assumption that the group G is a compact Lie Group, it admits a faithful group representation ρ ∗ that we can assume is the one acting on the world states W .
Revised: With the assumption that the group G is a compact Lie Group, it admits a faithful group representation ρ ∗ that we can assume is the one acting on the world states W .

Original: If we assume ρ ∗ is given on Z , then minimizing L 1 pred ( ρ ∗ , h ) is enough to learn a group-structured representation (See Appendix B Proposition 10).
Revised: If we assume ρ ∗ is given on Z , then minimizing L 1 pred ( ρ ∗ , h ) is enough to learn a group-structured representation (See Appendix B Proposition 10).

Original: However, when ρ ∗ is not known and a group representation ρ of G needs to be learned over a space of arbitrary mappings, minimizing L 1 pred ( ρ, h ) can lead to the trivial representation (See Appendix B Proposition 11).
Revised: However, when ρ ∗ is not known and a group representation ρ of G needs to be learned over a space of arbitrary mappings, minimizing L 1 pred ( ρ, h ) can lead to the trivial representation (See Appendix B Proposition 11).

Original: The reconstruction loss of the initial observation helps avoid the representation collapse into a trivial solution by ensuring h is not constant for a given fixed group representation ρ 

Revised: The reconstruction loss of the initial observation helps avoid the representation collapse into a trivial solution by ensuring h is not constant for a given fixed group representation ρ

Original: (See Appendix B Proposition 12).
Revised: (See Appendix B Proposition 12).

Original: Although we found using L 2 rec ( ρ, h ) works better than
Revised: Although we found using L 2 rec ( ρ, h ) works better than

Original: L 0 rec ( ρ, h ) when jointly learning ( ρ, h ).
Revised: L 0 rec ( ρ, h ) when jointly learning ( ρ, h ).

Original: We now present the main theoretical result of the paper: The HAE through enforcing the 2-step latent prediction loss and the observations reconstruction (enforcing h is injective) learns a group-structured representation.
Revised: We now present the main theoretical result of the paper: The HAE through enforcing the 2-step latent prediction loss and the observations reconstruction (enforcing h is injective) learns a group-structured representation.

Original: Proposition 1 Assume ( ρ, h ) minimizes L 2 pred ( ρ, h ) and h is injective, then ρ is a nontrivial group representation and ( ρ, h ) is a group-structured representation.
Revised: Proposition 1 Assume ( ρ, h ) minimizes L 2 pred ( ρ, h ) and h is injective, then ρ is a nontrivial group representation and ( ρ, h ) is a group-structured representation.

Original: 4. Results and Discussion
Revised: 4. Results and Discussion

Original: [Figure]
Revised: [Figure]

Original: We consider a toy dataset (See Appendix D.1) of shapes translated cyclically on a black background.
Revised: We consider a toy dataset (See Appendix D.1) of shapes translated cyclically on a black background.

Original: The first experiment (detailed in Appendix D.2) looks at the learned representation when the dataset is generated by the transitive action of translations on the ellipse shape.
Revised: The first experiment (detailed in Appendix D.2) looks at the learned representation when the dataset is generated by the transitive action of translations on the ellipse shape.

Original: The resulting manifold can be seen Figure 2 to match the expected 2-Torus
Revised: The resulting manifold can be seen Figure 2 to match the expected 2-Torus

Original: S 1 × S 1 and the group representation learned corresponds to the independent rotation matrices Figure 2.
Revised: S 1 × S 1 and the group representation learned corresponds to the independent rotation matrices Figure 2.

Original: An advantage of our proposed group representation is it gives access to the  group’s Lie Algebra where composition is a simple addition (See Appendix D.2.6).
Revised: An advantage of our proposed group representation is it gives access to the Extended Abstract Track group’s Lie Algebra where composition is a simple addition (See Appendix D.2.6).

Original: Enforcing the disentangled structure is detailed in Appendix C. We also show in Figure 10 how the consistency obtained in the learned group structured representations are useful for the rollout prediction of states after a long succession of actions (detailed in Appendix D.3).
Revised: Enforcing the disentangled structure is detailed in Appendix C. We also show in Figure 10 how the consistency obtained in the learned group structured representations are useful for the rollout prediction of states after a long succession of actions (detailed in Appendix D.3).

Original: We compare the performance of our model to another group structured approach (Quessard et al., 2020) (labelled ‘Rotations’) and to an unstructured joint encoder of state and action.
Revised: We compare the performance of our model to another group structured approach (Quessard et al., 2020) (labelled ‘Rotations’) and to an unstructured joint encoder of state and action.

Original: Finally, we show how in the case of multiple shapes being acted on by the same translation group, the HAE learns to separate the different shapes into orbits along a new representational unit, resulting in 3 parallel identical manifolds Figure 11.
Revised: Finally, we show how in the case of multiple shapes being acted on by the same translation group, the HAE learns to separate the different shapes into orbits along a new representational unit, resulting in 3 parallel identical manifolds Figure 11.

Original: Acknowledgments
Revised: Acknowledgments

Original: Hamza Keurti is grateful to CLS for generous funding support.
Revised: Hamza Keurti is grateful to CLS for generous funding support.

Original: The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting
Revised: The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting

Original: Hsiao-Ru Pan.
Revised: Hsiao-Ru Pan.

Original: Further this work was supported by the Swiss National Science Foundation (B.F.G. CRSII5-173721 and 315230 189251), ETH project funding (B.F.G. ETH-20 19-01), the Human Frontiers Science Program (RGY0072/2019).
Revised: Further this work was supported by the Swiss National Science Foundation (B.F.G. CRSII5-173721 and 315230 189251), ETH project funding (B.F.G. ETH-20 19-01), the Human Frontiers Science Program (RGY0072/2019).

Revised: Extended Abstract Track

Original: Appendix A. Background A.1. Group Theory
Revised: Appendix A. Background A.1. Group Theory

Original: In this section, we provide an overview of group theory concepts exploited in this work.
Revised: In this section, we provide an overview of group theory concepts exploited in this work.

Original: Definition 2 (Group)
Revised: Definition 2 (Group)

Original: A set G is a group if it is equipped with a binary operation · :
Revised: A set G is a group if it is equipped with a binary operation · :

Original: G × G → G and if the group axioms are satisfied
Revised: G × G → G and if the group axioms are satisfied

Original: Associativity: ∀ a, b, c ∈ G , ( a · b ) · c = a · ( b · c ) 2.
Revised: Associativity: ∀ a, b, c ∈ G , ( a · b ) · c = a · ( b · c ) 2.

Original: Identity: There exists e ∈ G such that ∀ a ∈ G , a · e = e · a = a . 3.
Revised: Identity: There exists e ∈ G such that ∀ a ∈ G , a · e = e · a = a . 3.

Original: Inverse: ∀ a ∈ G , there exists b ∈ G such that a · b = b · a = e .
Revised: Inverse: ∀ a ∈ G , there exists b ∈ G such that a · b = b · a = e .

Original: This inverse is denoted a − 1 .
Revised: This inverse is denoted a − 1 .

Original: We are often interested in sets of transformations, which respect a group structure, but are applied to objects that are not necessarily group elements.
Revised: We are often interested in sets of transformations, which respect a group structure, but are applied to objects that are not necessarily group elements.

Original: This can be studied through group actions, which describe how groups act on other mathematical entities.
Revised: This can be studied through group actions, which describe how groups act on other mathematical entities.

Original: Definition 3 (Group Action) Given a group G and a set X , a group action is a function · X : G × X
Revised: Definition 3 (Group Action) Given a group G and a set X , a group action is a function · X : G × X

Original: → X such that the following conditions are satisfied.
Revised: → X such that the following conditions are satisfied.

Original: Identity: If e ∈ G is the identity element, then e · X x = x , ∀ x ∈ X . 2.
Revised: Identity: If e ∈ G is the identity element, then e · X x = x , ∀ x ∈ X . 2.

Original: Compatibility: ∀ g, h ∈ G and ∀ x ∈ X , g · X ( h · X x ) =
Revised: Compatibility: ∀ g, h ∈ G and ∀ x ∈ X , g · X ( h · X x ) =

Original: (( g · h ) · X x )
Revised: (( g · h ) · X x )

Original: Induced homomorphism.
Revised: Induced homomorphism.

Original: The group action · X : G × X →
Revised: The group action · X : G × X →

Original: X induces a group homomorphism ρ · X : G → Sym ( X ).
Revised: X induces a group homomorphism ρ · X : G → Sym ( X ).

Original: (where Sym ( X ) is the group of all invertible transformations of X ) through:
Revised: (where Sym ( X ) is the group of all invertible transformations of X ) through:

Original: [Equation]
Revised: [Equation]

Original: The group homomorphism property of ρ · X comes from the group action axioms of · X :
Revised: The group homomorphism property of ρ · X comes from the group action axioms of · X :

Original: [Equation]
Revised: [Equation]

Original: So ρ · X ( id ) = id X .
Revised: So ρ · X ( id ) = id X .

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Equality over all of X leads to equality of the functions: ρ · X ( g 1 · g 2 ) =
Revised: Equality over all of X leads to equality of the functions: ρ · X ( g 1 · g 2 ) =

Original: ρ · X
Revised: ρ · X

Original: [Equation]
Revised: [Equation]

Original: g1 ) ◦ ρ · X
Revised: g

Original: [Equation]
Revised: [Equation]

Original:  g 2 ).
Revised: ◦ ρ · X ( g 2 ).

Original: In what follows, we are interested in linear group actions in which case the acted on space is a vector space V and the induced homomorphism ρ maps G to the group GL ( V ) of invertible linear transformations of V .
Revised: In what follows, we are interested in linear group actions in which case the acted on space is a vector space V and the induced homomorphism ρ maps G to the group GL ( V ) of invertible linear transformations of V .

Original: This mapping is called a group representation.
Revised: This mapping is called a group representation.

Original: Actions of this type have been studied extensively in representation theory.
Revised: Actions of this type have been studied extensively in representation theory.

Revised: Extended Abstract Track

Original: Definition 4 (Group Representation)
Revised: Definition 4 (Group Representation)

Original: Let G be a group and V a vector space.
Revised: Let G be a group and V a vector space.

Original: A representation is a function ρ : G → GL ( V ) such that ∀ g, h ∈ G , one has ρ ( g ) ρ ( h ) =
Revised: A representation is a function ρ : G → GL ( V ) such that ∀ g, h ∈ G , one has ρ ( g ) ρ ( h ) =

Original: ρ ( g · h ) .
Revised: ρ ( g · h ) .

Original: Note that such definition is not restricted to finite dimensional vector spaces, however we will limit our study to this case, such that representations are appropriately described by mappings from G to a space of square matrices.
Revised: Note that such definition is not restricted to finite dimensional vector spaces, however we will limit our study to this case, such that representations are appropriately described by mappings from G to a space of square matrices.

Original: Definition 5 (Lie Group)
Revised: Definition 5 (Lie Group)

Original: A Lie Group G is a nonempty set satisfying the following conditions:
Revised: A Lie Group G is a nonempty set satisfying the following conditions:

Original: • G is a group.
Revised: • G is a group.

Original: • G is a smooth manifold.
Revised: • G is a smooth manifold.

Original: • The group operation · : G × G → G and the inverse map .
Revised: • The group operation · : G × G → G and the inverse map .

Original: − 1 : G → G are smooth.
Revised: − 1 : G → G are smooth.

Original: We limit ourselves to the study of linear Lie Groups, Lie groups that are matrix groups.
Revised: We limit ourselves to the study of linear Lie Groups, Lie groups that are matrix groups.

Original: The tangent space to a Lie Group at the identity forms a Lie Algebra.
Revised: The tangent space to a Lie Group at the identity forms a Lie Algebra.

Original: A Lie Algebra g is a vector space equipped with a bilinear map [ ., . ] : g × g → g called the Lie Bracket.
Revised: A Lie Algebra g is a vector space equipped with a bilinear map [ ., . ] : g × g → g called the Lie Bracket.

Original: We will not introduce the Lie Bracket as we do not make use of it.
Revised: We will not introduce the Lie Bracket as we do not make use of it.

Original: The Lie Algebra somehow describes most of everything happening in its Lie Group.
Revised: The Lie Algebra somehow describes most of everything happening in its Lie Group.

Original: This connection is established through the exponential map.
Revised: This connection is established through the exponential map.

Original: Definition 6 (Exponential Map)
Revised: Definition 6 (Exponential Map)

Original: The exponential map exp : g → G is defined for matrix Lie Groups by the series:
Revised: The exponential map exp : g → G is defined for matrix Lie Groups by the series:

Original: [Equation]
Revised: [Equation]

Original: The exponential map is not always surjective.
Revised: The exponential map is not always surjective.

Original: However if we only consider groups that are connected and compact, the exponential is surjective, which justifies our parametrization of the group representation through:
Revised: However if we only consider groups that are connected and compact, the exponential is surjective, which justifies our parametrization of the group representation through:

Original: [Equation]
Revised: [Equation]

Original: Where ϕ is a trainable arbitrary mapping.
Revised: Where ϕ is a trainable arbitrary mapping.

Original: Group action types
Revised: Group action types

Original: The effect of a group action on a base space X varies according to the properties of the homomorphism defined by the group action
Revised: The effect of a group action on a base space X varies according to the properties of the homomorphism defined by the group action

Original: [Equation]
Revised: [Equation]

Original: We introduce two types of actions:
Revised: We introduce two types of actions:

Original: Definition 7 (Transitive Group Action)
Revised: Definition 7 (Transitive Group Action)

Original: The action of G on X is transitive if X forms a single orbit .
Revised: The action of G on X is transitive if X forms a single orbit .

Original: in other words, ∀ x, y ∈ X, ∃ g ∈ G ; g · x = y .
Revised: in other words, ∀ x, y ∈ X, ∃ g ∈ G ; g · x = y .

Revised: Extended Abstract Track

Original: Definition 8 (Faithful Group Action)
Revised: Definition 8 (Faithful Group Action)

Original: The action of G on X is faithful if the homomorphism G
Revised: The action of G on X is faithful if the homomorphism G

Original: → Sym ( X ) corresponding to the action is bijective (an isomorphism).
Revised: → Sym ( X ) corresponding to the action is bijective (an isomorphism).

Original: In that case, ∀ g 1 ̸ = g 2 ∈ G, ∃ x ∈ X ; [Equation]
Revised: In that case, ∀ g 1 ̸ = g 2 ∈ G, ∃ x ∈ X ; g 1 · x ̸ = g 2 · x .

Original: We also define the orbits by a group action:
Revised: We also define the orbits by a group action:

Original: Definition 9 (Orbit by a Group Action)
Revised: Definition 9 (Orbit by a Group Action)

Original: The orbit of an element x ∈ X by the action · X of a group G is the set
Revised: The orbit of an element x ∈ X by the action · X of a group G is the set

Original: [Equation]
Revised: [Equation]

Original: When the action of G is transitive on X , then X is the single orbit by the action of G :
Revised: When the action of G is transitive on X , then X is the single orbit by the action of G :

Original: [Equation]
Revised: [Equation]

Original: Such is the case for our experiments using a single shape.
Revised: Such is the case for our experiments using a single shape.

Original: We also explore the case where the action is not transitive in the multi shape experiment visualized in Figure 11.
Revised: We also explore the case where the action is not transitive in the multi shape experiment visualized in Figure 11.

Original: A.2. Group-Structured Disentangled Representation Learning
Revised: A.2. Group-Structured Disentangled Representation Learning

Original: The group-structured representation is defined in Section 2.1.
Revised: The group-structured representation is defined in Section 2.1.

Original: We proceed to define groupstructured disentangled representations.
Revised: We proceed to define groupstructured disentangled representations.

Original: The group-structured representation is disentangled with regard to the group decomposition G =
Revised: The group-structured representation is disentangled with regard to the group decomposition G =

Original: G 1 × ...
Revised: G 1 × ...

Original: × G n if it satisfies this additional condition:
Revised: × G n if it satisfies this additional condition:

Original: Z can be written as a product of spaces Z
Revised: Z can be written as a product of spaces Z

Original: = Z 1 × ...
Revised: = Z 1 × ...

Original: × Z n or as a direct sum of subspaces Z
Revised: × Z n or as a direct sum of subspaces Z

Original: = Z 1 ⊕ ...
Revised: = Z 1 ⊕ ...

Original: ⊕ Z n such that each subgroup G i acts non trivially on Z i and acts trivially on Z j for j ̸
Revised: ⊕ Z n such that each subgroup G i acts non trivially on Z i and acts trivially on Z j for j ̸

Original: = i .
Revised: = i .

Original: If we require the group action · Z on Z to be linear, then the disentanglement of the group action on Z is expressed as:
Revised: If we require the group action · Z on Z to be linear, then the disentanglement of the group action on Z is expressed as:

Original: (Linear)
Revised: (Linear)

Original: There exists a decomposition Z
Revised: There exists a decomposition Z

Original: = Z 1 ⊕ ...
Revised: = Z 1 ⊕ ...

Original: ⊕ Z n and a decomposition of the group representation
Revised: ⊕ Z n and a decomposition of the group representation

Original: [Equation]
Revised: [Equation]

Original: where each ρ i : G i → GL ( Z i ) is a subrepresentation.
Revised: where each ρ i : G i → GL ( Z i ) is a subrepresentation.

Original: The action on Z can then be written
Revised: The action on Z can then be written

Original: [Equation]
Revised: [Equation]

Original: for g = ( g 1 , ..., g n ) ∈ G and z = z 1 ⊕ ... ⊕ z n ∈ Z .
Revised: for g = ( g 1 , ..., g n ) ∈ G and z = z 1 ⊕ ... ⊕ z n ∈ Z .

Original: Clearly each subgroup G i acts trivially on Z j , j ̸
Revised: Clearly each subgroup G i acts trivially on Z j , j ̸

Original: = i .
Revised: = i .

Revised: Extended Abstract Track

Original: A.3. Observing Symmetries
Revised: A.3. Observing Symmetries

Original: Caselles-Dupr´e et al.
Revised: Caselles-Dupr´e et al.

Original: (2019) proved that learning such group-structured representations requires observing the group elements that transition one state of the world into another instead of isolated observations.
Revised: (2019) proved that learning such group-structured representations requires observing the group elements that transition one state of the world into another instead of isolated observations.

Original: Similarly to past works (Quessard et al., 2020; CasellesDupr´e et al., 2019), we suggest using the interventions of an agent on its environment as a means to probe the symmetries of the environment.
Revised: Similarly to past works (Quessard et al., 2020; CasellesDupr´e et al., 2019), we suggest using the interventions of an agent on its environment as a means to probe the symmetries of the environment.

Original: However, we do not assume that these interventions only act on one latent at a time.
Revised: However, we do not assume that these interventions only act on one latent at a time.

Original: Instead, the agent intervenes through motor signals which are parametrized by the joints positions and it is not known a priori when an intervention only modifies one true latent while keeping others constant.
Revised: Instead, the agent intervenes through motor signals which are parametrized by the joints positions and it is not known a priori when an intervention only modifies one true latent while keeping others constant.

Original: The parametrization of the observed group elements is also expressed in the joints space instead of along the true latents.
Revised: The parametrization of the observed group elements is also expressed in the joints space instead of along the true latents.

Original: For a given group element, we will denote ˜ g , its parametrization in the latents space, and g its parametrization in the joints space.
Revised: For a given group element, we will denote ˜ g , its parametrization in the latents space, and g its parametrization in the joints space.

Original: And we assume there exists a deterministic mapping between parametrizations φ : ˜ g (cid:55)→ g .
Revised: And we assume there exists a deterministic mapping between parametrizations φ : ˜ g (cid:55)→ g .

Original: For instance, when a person moves a chalk along a blackboard, the chalk describes a 2D movement overparametrized by the rotation angles of the joints of the arm.
Revised: For instance, when a person moves a chalk along a blackboard, the chalk describes a 2D movement overparametrized by the rotation angles of the joints of the arm.

Original: We propose to learn the inference process h and a disentangled group representation ρ using the HAE, described in section 3.1.
Revised: We propose to learn the inference process h and a disentangled group representation ρ using the HAE, described in section 3.1.

Original: A.4. Lie Groups and the Exponential Map
Revised: A.4. Lie Groups and the Exponential Map

Original: We assume the group G is a connected compact Lie group, the observed interventions are a discrete subset sampled from G .
Revised: We assume the group G is a connected compact Lie group, the observed interventions are a discrete subset sampled from G .

Original: When a group G is also a differentiable manifold, it is called a Lie Group.
Revised: When a group G is also a differentiable manifold, it is called a Lie Group.

Original: The tangent space to the group G at the identity forms a Lie Algebra g : A vector space equipped with a bilinear product, the Lie bracket.
Revised: The tangent space to the group G at the identity forms a Lie Algebra g : A vector space equipped with a bilinear product, the Lie bracket.

Original: We will leverage a convenient property of Lie Groups and their Lie Algebras that is the group can be studied through its tangent space.
Revised: We will leverage a convenient property of Lie Groups and their Lie Algebras that is the group can be studied through its tangent space.

Original: Indeed, the matrix exponential, called the exponential map in this setting, transports elements from the Lie Algebra to the Lie Group.
Revised: Indeed, the matrix exponential, called the exponential map in this setting, transports elements from the Lie Algebra to the Lie Group.

Original: Under certain assumptions, for instance if G is connected compact, which we assume, the exponential map is surjective and therefore the whole group G can be described from the tangent space at its identity.
Revised: Under certain assumptions, for instance if G is connected compact, which we assume, the exponential map is surjective and therefore the whole group G can be described from the tangent space at its identity.

Original: The exponential of an arbitrary matrix A is given by the series e A
Revised: The exponential of an arbitrary matrix A is given by the series e A

Original: = (cid:80) ∞
Revised: = (cid:80) ∞

Original: k =0 

Revised: k =0

Original: k A k .
Revised: k A k .

Original: We construct the learnable group representation ρ as the composition of the exponential map with an arbitrary mapping to the Lie Algebra.
Revised: We construct the learnable group representation ρ as the composition of the exponential map with an arbitrary mapping to the Lie Algebra.

Original: [Equation]
Revised: [Equation]

Original: We show how access to the group algebra can be leveraged in appendix D.2.6.
Revised: We show how access to the group algebra can be leveraged in appendix D.2.6.

Original: Appendix B. Theoretical results B.1. Propositions and Proofs
Revised: Appendix B. Theoretical results B.1. Propositions and Proofs

Original: First we prove the main theoretical result of the paper, then proceed to provide other propositions and their proofs.
Revised: First we prove the main theoretical result of the paper, then proceed to provide other propositions and their proofs.

Original: Proof [Proposition 1] Given that the group G is supposed compact, it admits a group representation by the Peter-Weyl theorem .
Revised: Proof [Proposition 1] Given that the group G is supposed compact, it admits a group representation by the Peter-Weyl theorem .

Original: We can therefore assume that the true state space W is acted on linearly by G through its representation ρ ∗ .
Revised: We can therefore assume that the true state space W is acted on linearly by G through its representation ρ ∗ .

Original: As such the inverse of the generating process b − 1 and ρ ∗ verify L 2 pred ( ρ ∗ , b − 1 ) = 0.
Revised: As such the inverse of the generating process b − 1 and ρ ∗ verify L 2 pred ( ρ ∗ , b − 1 ) = 0.

Revised: Extended Abstract Track

Original: Assume h is injective, guaranteed by a minimization of the 0-step reconstruction loss.
Revised: Assume h is injective, guaranteed by a minimization of the 0-step reconstruction loss.

Original: Assume ( ρ, h ) minimizes L 2 pred ( ρ, h ) therefore L 2 pred ( ρ, h ) = 0.
Revised: Assume ( ρ, h ) minimizes L 2 pred ( ρ, h ) therefore L 2 pred ( ρ, h ) = 0.

Original: Then for all observed 2-step transitions ( o t , g t , o t +1 , g t +1 , o t +2 ) — note that observed transitions ( o t , g t , o t +1 ) correspond to an action on the true world states w t
Revised: Then for all observed 2-step transitions ( o t , g t , o t +1 , g t +1 , o t +2 ) — note that observed transitions ( o t , g t , o t +1 ) correspond to an action on the true world states w t

Original: +1 =
Revised: +1 =

Original: g t · W w t — ( ρ, h ) verifies:
Revised: g t · W w t — ( ρ, h ) verifies:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Let us prove ρ is a group representation, meaning it verifies ρ ( g 2 g 1 )
Revised: Let us prove ρ is a group representation, meaning it verifies ρ ( g 2 g 1 )

Original: =
Revised: =

Original: ρ ( g 2 ) ρ ( g 1 ) , ∀ g 1 , g 2 ∈
Revised: ρ ( g 2 ) ρ ( g 1 ) , ∀ g 1 , g 2 ∈

Original: G .
Revised: G .

Original: Let g 1 , g 2 , g 3 ∈ G such that g 3 =
Revised: Let g 1 , g 2 , g 3 ∈ G such that g 3 =

Original: g 2 g 1 .
Revised: g 2 g 1 .

Original: Let w 1 , w 2 , w 3 which verify w 2 = g 1 · W w 1 , w 3 = g 2 · W w 2 , w 3 = g 3 · W w 1 .
Revised: Let w 1 , w 2 , w 3 which verify w 2 = g 1 · W w 1 , w 3 = g 2 · W w 2 , w 3 = g 3 · W w 1 .

Original: Therefore, the associated transitions ( o 1 , g 1 , o 2 ), ( o 2 , g 2 , o 3 ) and ( o 1 , g 3 , o 3 ) can be observed.
Revised: Therefore, the associated transitions ( o 1 , g 1 , o 2 ), ( o 2 , g 2 , o 3 ) and ( o 1 , g 3 , o 3 ) can be observed.

Original: Assume, we observe the 2-step transitions ( o t , g t , o t +1 , g t +1 , o t +2 ) and ( o t ′ , g t ′ , o t ′
Revised: Assume, we observe the 2-step transitions ( o t , g t , o t +1 , g t +1 , o t +2 ) and ( o t ′ , g t ′ , o t ′

Original: +1 , g t ′
Revised: +1 , g t ′

Original: +1 , o t ′ +2 )
Revised: +1 , o t ′ +2 )

Original: such that:
Revised: such that:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: L 2 pred = 0 gives for the transition at t : ρ ( g t +1 ) ρ ( g t )
Revised: L 2 pred = 0 gives for the transition at t : ρ ( g t +1 ) ρ ( g t )

Original: h ( o t )
Revised: h ( o t )

Original: =
Revised: =

Original: h ( o t +2 )
Revised: h ( o t +2 )

Original: therefore ρ ( g 2 ) ρ ( g 1 ) h ( o 1 ) =
Revised: therefore ρ ( g 2 ) ρ ( g 1 ) h ( o 1 ) =

Original: h ( o 3 ).
Revised: h ( o 3 ).

Original: And for the transition at t ′ :
Revised: And for the transition at t ′ :

Original: ρ ( g t ′ ) h
Revised: ρ ( g t ′ ) h

Original: ( o t ′ )
Revised: ( o t ′ )

Original: =
Revised: =

Original: h ( o t ′ +1 )
Revised: h ( o t ′ +1 )

Original: therefore ρ ( g 3 ) h ( o 1 ) =
Revised: therefore ρ ( g 3 ) h ( o 1 ) =

Original: h ( o 3 )
Revised: h ( o 3 )

Original: Therefore we have ρ ( g 2 ) ρ ( g 1 )
Revised: Therefore we have ρ ( g 2 ) ρ ( g 1 )

Original: h ( o 1 )
Revised: h ( o 1 )

Original: =
Revised: =

Original: ρ ( g 3 ) h
Revised: ρ ( g 3 ) h

Original: ( o 1 ) or ρ ( g 2 ) ρ ( g 1 ) h
Revised: ( o 1 ) or ρ ( g 2 ) ρ ( g 1 ) h

Original: ( o 1 ) =
Revised: ( o 1 ) =

Original: ρ ( g 2 g 1 ) h
Revised: ρ ( g 2 g 1 ) h

Original: ( o 1 )
Revised: ( o 1 )

Original: With this equality verified over the set O tr of first observations o 1 in the training set — meaning we observe two successions of the action of group elements ( g 1 , g 2 ) and g 3 for different values of the starting observation o 1 — by assuming h ( O ) ⊆ span ( h ( O tr )), we get that ρ ( g 2 ) ρ ( g 1 ) =
Revised: With this equality verified over the set O tr of first observations o 1 in the training set — meaning we observe two successions of the action of group elements ( g 1 , g 2 ) and g 3 for different values of the starting observation o 1 — by assuming h ( O ) ⊆ span ( h ( O tr )), we get that ρ ( g 2 ) ρ ( g 1 ) =

Original: ρ ( g 2 g 1 ) over h ( O ) (equality of linear mappings over vectors that span a vector subspace).
Revised: ρ ( g 2 g 1 ) over h ( O ) (equality of linear mappings over vectors that span a vector subspace).

Original: Therefore the subrepresentation of ρ over span ( h ( O )) is a group representation of G .
Revised: Therefore the subrepresentation of ρ over span ( h ( O )) is a group representation of G .

Original: The injectivity assumption ensure h ( O ) does not collapse to a single element.
Revised: The injectivity assumption ensure h ( O ) does not collapse to a single element.

Original: Let us show h is a group-structured representation.
Revised: Let us show h is a group-structured representation.

Original: We have for every observed transition ( o t , g t , o t +1 ):
Revised: We have for every observed transition ( o t , g t , o t +1 ):

Revised: Extended Abstract Track

Original: [Equation]
Revised: [Equation]

Original: by the generative model assumptions
Revised: by the generative model assumptions

Original: [Equation]
Revised: [Equation]

Original: Also o t = b ( w t ), finally
Revised: Also o t = b ( w t ), finally

Original: [Equation]
Revised: [Equation]

Original: Proposition 10 Assume we observed the action of the group G on each point of the observation space.
Revised: Proposition 10 Assume we observed the action of the group G on each point of the observation space.

Original: Assume h minimizes L 1 pred ( ρ ∗ , h ) then h is a group-structured representation, meaning h ◦ b is equivariant.
Revised: Assume h minimizes L 1 pred ( ρ ∗ , h ) then h is a group-structured representation, meaning h ◦ b is equivariant.

Original: Proof [Proposition 10] Given that the group G is supposed compact, it admits a group representation by the Peter-Weyl theorem .
Revised: Proof [Proposition 10] Given that the group G is supposed compact, it admits a group representation by the Peter-Weyl theorem .

Original: We can therefore assume that the true state space W is acted on linearly by G through its representation ρ ∗ .
Revised: We can therefore assume that the true state space W is acted on linearly by G through its representation ρ ∗ .

Original: As such the inverse of the generating process b − 1 and ρ ∗ verify L 2 pred ( ρ ∗ , b − 1 ) = 0.
Revised: As such the inverse of the generating process b − 1 and ρ ∗ verify L 2 pred ( ρ ∗ , b − 1 ) = 0.

Original: As a consequence, h also achieves zero loss such that 
Revised: As a consequence, h also achieves zero loss such that (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: (cid:89) i =

Revised: [Equation]

Revised: ρ

Original: such that for all ( j, t ) h 
Revised: [Equation] ( g [Equation] h ( o

Original:  ( o 
Revised: [Equation] ) ||such that for all ( j, t ) h ( o t + j ) = j − 1 (cid:89) i = ρ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: g

Original: [Equation]
Revised: [Equation]

Original: =
Revised: + i

Original: [Equation]
Revised: [Equation]

Original: − 1 (cid:89)
Revised: h ( o

Original: [Equation]
Revised: [Equation]

Original: In particular for j =
Revised: In particular for j =h

Original: [Equation]
Revised: [Equation]

Original: ) by the generative model assumptions
Revised: ) by the generative model assumptions

Original: [Equation]
Revised: [Equation]

Original: Also o t = b ( w t ) such that
Revised: Also o t = b ( w t ) such that

Original: [Equation]
Revised: [Equation]

Original: Proposition 11 The trivial group representation ρ =
Revised: Proposition 11 The trivial group representation ρ =

Original: I (that always maps to the identity matrix)
Revised: I (that always maps to the identity matrix)

Original: combined with a constant h is a zero of the prediction loss L 1 pred ( ρ, h ) .
Revised: combined with a constant h is a zero of the prediction loss L 1 pred ( ρ, h ) .

Original: Proposition 12 Assume the data samples at least once all points of the observation space.
Revised: Proposition 12 Assume the data samples at least once all points of the observation space.

Original: If h minimizes L Nrec ( ρ 0 , h ) then h is injective.
Revised: If h minimizes L Nrec ( ρ 0 , h ) then h is injective.

Revised: Extended Abstract Track

Original: Proof [Proposition 12] Assume the encoder h and the decoder d minimize the 0-step reconstruction.
Revised: Proof [Proposition 12] Assume the encoder h and the decoder d minimize the 0-step reconstruction.

Original: ∀ o, o ′ ∈
Revised: ∀ o, o ′ ∈

Original: O, such that o ̸
Revised: O, such that o ̸

Original: = o ′ .
Revised: = o ′ .

Original: d ( h ( o ))
Revised: d ( h ( o ))

Original: =
Revised: =

Original: o and d ( h ( o ′ ))
Revised: o and d ( h ( o ′ ))

Original: = o ′
Revised: = o ′

Original: Therefore d ( h ( o )) ̸
Revised: Therefore d ( h ( o )) ̸

Original: = d ( h ( o ′ )) .
Revised: = d ( h ( o ′ )) .

Original: Finally h ( o ) ̸
Revised: Finally h ( o ) ̸

Original: = h ( o ′ )
Revised: = h ( o ′ )

Original: Therefore h is injective.
Revised: Therefore h is injective.

Original: Appendix C. Disentanglement C.1. Disentanglement through Structural Constraints
Revised: Appendix C. Disentanglement C.1. Disentanglement through Structural Constraints

Original: As expressed in Appendix A.2 the disentanglement condition for a linear action on Z defined through its group representation ρ , is a decomposition of both the representation space
Revised: As expressed in Appendix A.2 the disentanglement condition for a linear action on Z defined through its group representation ρ , is a decomposition of both the representation space

Original: Z = (cid:76) n 1 Z i and the group representation
Revised: Z = (cid:76) n 1 Z i and the group representation

Original: ρ = (cid:76) ni
Revised: ρ = (cid:76) ni

Original: =

Revised: =

Original: ρ i .
Revised: ρ i .

Original: Where the subgroup representations ρ
Revised: Where the subgroup representations ρ

Original: i are representations of the subgroups G i on the subspaces Z i .
Revised: i are representations of the subgroups G i on the subspaces Z i .

Original: Following that the group G is decomposed in the true latent’s parametrization, the observed group representation is disentangled with regard to the group decomposition if in matrix form, the group representation of any group element g = φ (˜ g ) is a block-diagonal matrix of the subgroups representations:
Revised: Following that the group G is decomposed in the true latent’s parametrization, the observed group representation is disentangled with regard to the group decomposition if in matrix form, the group representation of any group element g = φ (˜ g ) is a block-diagonal matrix of the subgroups representations:

Original: [Equation]
Revised: [Equation]

Original: We can therefore constrain our trainable group representation in the space of matrices of the block diagonal form given in equation 2.
Revised: We can therefore constrain our trainable group representation in the space of matrices of the block diagonal form given in equation 2.

Original: This requires prior knowledge of:
Revised: This requires prior knowledge of:

Original: • The number of groups in the decomposition.
Revised: • The number of groups in the decomposition.

Original: • The dimension of each subgroup representation dim ( Z i ).
Revised: • The dimension of each subgroup representation dim ( Z i ).

Original: We start by assuming knowledge of this information, however we hope that in future works, one would be able to search for the best decomposition.
Revised: We start by assuming knowledge of this information, however we hope that in future works, one would be able to search for the best decomposition.

Original: However, we do not assume prior knowledge of the decomposition.
Revised: However, we do not assume prior knowledge of the decomposition.

Original: Meaning that given an observed transition g , we do not have access to the decomposition φ − 1 ( g )
Revised: Meaning that given an observed transition g , we do not have access to the decomposition φ − 1 ( g )

Original: =
Revised: =

Original: ˜
Revised: ˜

Original: g
Revised: g

Original: =
Revised: =

Original: (˜ g 1 , ..., ˜ g n ) along the group decomposition
Revised: (˜ g 1 , ..., ˜ g n ) along the group decomposition

Original: G = G 1 × ...
Revised: G = G 1 × ...

Original: × G n .
Revised: × G n .

Original: Which means we are learning representations of the form given in equation 3.
Revised: Which means we are learning representations of the form given in equation 3.

Original: [Equation]
Revised: [Equation]

Original: While we do not prove that this block diagonal constraint leads to disentanglement, we show through experiments that HAE learns a group-structured representation ( ρ, h ) that is disentangled with regard to the true factors and takes the form in equation 2.
Revised: While we do not prove that this block diagonal constraint leads to disentanglement, we show through experiments that HAE learns a group-structured representation ( ρ, h ) that is disentangled with regard to the true factors and takes the form in equation 2.

Original: C.2. Disentanglement through Sparsity Loss
Revised: C.2. Disentanglement through Sparsity Loss

Original: To drop the assumption of prior knowledge of the number of subgroups in the representation as well as the dimension for their representation, we propose a sparsity loss (Equation 4) on the group representation that prefers block-diagonal patterns (Example in Figure 3) by minimizing the terms of ( ρ ij ( g t )) ij that do not belong to allowed non-zero terms.
Revised: To drop the assumption of prior knowledge of the number of subgroups in the representation as well as the dimension for their representation, we propose a sparsity loss (Equation 4) on the group representation that prefers block-diagonal patterns (Example in Figure 3) by minimizing the terms of ( ρ ij ( g t )) ij that do not belong to allowed non-zero terms.

Original: See
Revised: See

Original: Appendix C.2 for a brief discussion.
Revised: Appendix C.2 for a brief discussion.

Original: The loss is inspired from works on sparsity inducing losses (Bach et al., 2011).
Revised: The loss is inspired from works on sparsity inducing losses (Bach et al., 2011).

Original: [Equation]
Revised: [Equation]

Original: The model is then trained on the composite loss:
Revised: The model is then trained on the composite loss:

Original: [Equation]
Revised: [Equation]

Original: We can then choose a representation space Z
Revised: We can then choose a representation space Z

Original: = R d with d large enough to accommodate the total dimension of the group representation, extra dimensions are then trivialized by
Revised: = R d with d large enough to accommodate the total dimension of the group representation, extra dimensions are then trivialized by

Original: L sparse .
Revised: L sparse .

Original: Figure 3 shows how the term associated with each value of i induces a direct sum ofsubrepresentations ρ = ρ 1 ⊕ ρ 2 with a different split of the total representation dimension.
Revised: Figure 3 shows how the term associated with each value of i induces a direct sum ofsubrepresentations ρ = ρ 1 ⊕ ρ 2 with a different split of the total representation dimension.

Original: Combining patterns produces any possible decomposition into a direct sum of subrepresentations.
Revised: Combining patterns produces any possible decomposition into a direct sum of subrepresentations.

Original: See Appendix D.2.7 for associated results.
Revised: See Appendix D.2.7 for associated results.

Original: Appendix D. Experiments D.1. Data
Revised: Appendix D. Experiments D.1. Data

Original: We use a subset of the dSprites dataset consisting only of the ellipse at a fixed scale and orientation with varying x and y positions, there are 32 equally spaced positions for each.
Revised: We use a subset of the dSprites dataset consisting only of the ellipse at a fixed scale and orientation with varying x and y positions, there are 32 equally spaced positions for each.

Original: We consider that the ellipse is acted on by the group G =
Revised: We consider that the ellipse is acted on by the group G =

Original: G x × G y of cyclic translations, where the sprite warps to the opposite extremity when it reaches an extremal position.
Revised: G x × G y of cyclic translations, where the sprite warps to the opposite extremity when it reaches an extremal position.

Original: For each image o 1 we sample group elements ˜ g 

Revised: For each image o 1 we sample group elements ˜ g

Original: = (˜ g x , ˜ g y ) uniformly from a square around  identity spanning the range (cid:74) − 10 ,(cid:75) .
Revised: = (˜ g x , ˜ g y ) uniformly from a square around Extended Abstract Track identity spanning the range (cid:74) − 10 ,(cid:75) .

Original: We assume the agent observes g 1 = φ 45 (˜ g 1 ) = √ 22 (˜ g x − ˜ g y , ˜
Revised: We assume the agent observes g 1 = φ 45 (˜ g 1 ) = √ 22 (˜ g x − ˜ g y , ˜

Original: g
Revised: g

Original: x + ˜ g y ).
Revised: x + ˜ g y ).

Original: We obtain the first transition ( o 1 , g 1 , o 2 ), we do the same for o 2 to get 2-step transitions ( o 1 , g 1 , o 2 , g 2 , o 3 ).
Revised: We obtain the first transition ( o 1 , g 1 , o 2 ), we do the same for o 2 to get 2-step transitions ( o 1 , g 1 , o 2 , g 2 , o 3 ).

Original: D.2. Learning a Group-Structured Disentangled Representation D.2.1. Setup
Revised: D.2. Learning a Group-Structured Disentangled Representation D.2.1. Setup

Original: We consider a subset of the dsprites dataset (Matthey et al., 2017) where a fixed scale and orientation ellipse is acted on by the group of 2D cyclic translations G = C x × C y .
Revised: We consider a subset of the dsprites dataset (Matthey et al., 2017) where a fixed scale and orientation ellipse is acted on by the group of 2D cyclic translations G = C x × C y .

Original: The corresponding transition dataset contains tuples ( o 1 , g 1 , o 2 , ..., g n − 1 , o n ), where the observations o i are 64 × 64 pixels and the transitions are given by g i =
Revised: The corresponding transition dataset contains tuples ( o 1 , g 1 , o 2 , ..., g n − 1 , o n ), where the observations o i are 64 × 64 pixels and the transitions are given by g i =

Original: φ π/ 4 (˜ g i ) where ˜ g i
Revised: φ π/ 4 (˜ g i ) where ˜ g i

Original: =
Revised: =

Original: (˜ g xi , ˜ g yi ) parametrizes the displacement along x and y , and φ π/ 4 is the rotation of the 2D plane by 45 deg.
Revised: (˜ g xi , ˜ g yi ) parametrizes the displacement along x and y , and φ π/ 4 is the rotation of the 2D plane by 45 deg.

Original: D.2.2. Hyperparameters Model architecture
Revised: D.2.2. Hyperparameters Model architecture

Original: We use a symmetrical architecture for the encoder and decoder, which we summarize in Table 1.
Revised: We use a symmetrical architecture for the encoder and decoder, which we summarize in Table 1.

Original: The network was trained on the combined loss:
Revised: The network was trained on the combined loss:

Original: [Equation]
Revised: [Equation]

Original: Where we use the Binary Cross Entropy loss for the reconstruction term instead of the
Revised: Where we use the Binary Cross Entropy loss for the reconstruction term instead of the

Original: Mean Squared Error as it is better behaved during training.
Revised: Mean Squared Error as it is better behaved during training.

Original: [Table]
Revised: [Table]

Original: Training hyperparameters
Revised: Training hyperparameters

Original: We trained the network using the hyperparameters summarized in Table 2.
Revised: We trained the network using the hyperparameters summarized in Table 2.

Original: D.2.3. Visualization
Revised: D.2.3. Visualization

Original: We obtained the manifold in Figure 2 by projecting the 4D representation vector for each image in the dataset on a random 2D plane through Random Matrix Projection.
Revised: We obtained the manifold in Figure 2 by projecting the 4D representation vector for each image in the dataset on a random 2D plane through Random Matrix Projection.

Original: We chose the projection with the most explainable visualization.
Revised: We chose the projection with the most explainable visualization.

Original: [Table]
Revised: [Table]

Original: D.2.4. Results Learned data representation
Revised: D.2.4. Results Learned data representation

Original: We visualize the learned 4-dimensional encodings of the whole dataset through 2-dimensional random matrix projections, and selected a projection matrix showing a discernable projected manifold (Figure 2 Left).
Revised: We visualize the learned 4-dimensional encodings of the whole dataset through 2-dimensional random matrix projections, and selected a projection matrix showing a discernable projected manifold (Figure 2 Left).

Original: The learned manifold corresponds to the expected latent space topology S 1 × S 1 .
Revised: The learned manifold corresponds to the expected latent space topology S 1 × S 1 .

Original: Learned Group Representation ρ
Revised: Learned Group Representation ρ

Original: We then evaluate the learned matrices for the identity id = (0 , 0), the generating elements of each subgroup ˜1 x
Revised: We then evaluate the learned matrices for the identity id = (0 , 0), the generating elements of each subgroup ˜1 x

Original: = (1 , 0) , ˜1 y
Revised: = (1 , 0) , ˜1 y

Original: =
Revised: =

Original: (0 , 1) and their inverses ˜ g − 1 x =
Revised: (0 , 1) and their inverses ˜ g − 1 x =

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Revised: g

Original: [Equation]
Revised: [Equation]

Original: y = (0 , − 1), Figure 2.
Revised: y = (0 , − 1), Figure 2.

Original: Note that the corresponding observed actions g presented to the group representation ρ are in the same order: (0 , 0) , 
Revised: Note that the corresponding observed actions g presented to the group representation ρ are in the same order: (0 , 0) , √

Original: [Equation]
Revised: [Equation]

Original: The matrices obtained, for example, actions in Figure 2, show that ρ (0) =
Revised: The matrices obtained, for example, actions in Figure 2, show that ρ (0) =

Original: I 4 and that representations for elements belonging to subgroups of the decomposition
Revised: I 4 and that representations for elements belonging to subgroups of the decomposition

Original: G = C x × C y follow the disentangled group representation predicted in subsection C.1 where actions on the same subgroup have representations that act on the same subspace while fixing the other subspace.
Revised: G = C x × C y follow the disentangled group representation predicted in subsection C.1 where actions on the same subgroup have representations that act on the same subspace while fixing the other subspace.

Original: In addition, the blocks correspond to 2D rotation matrices which are of the form:
Revised: In addition, the blocks correspond to 2D rotation matrices which are of the form:

Original: [Equation]
Revised: [Equation]

Original: The angle of rotation of an elementary step corresponds to the number of equally spaced true latent values for each subgroup: 2 π/ 32, with cos(2 π/ 32) ≈ 0 . 981 and sin(2 π/ 32) ≈ 0 .
Revised: The angle of rotation of an elementary step corresponds to the number of equally spaced true latent values for each subgroup: 2 π/ 32, with cos(2 π/ 32) ≈ 0 . 981 and sin(2 π/ 32) ≈ 0 .

Original: D.2.5. Representation of a Neighbourhood of Identity
Revised: D.2.5. Representation of a Neighbourhood of Identity

Original: We show in Figure 4 the group representation ρ ( g ) for g in a grid around identity.
Revised: We show in Figure 4 the group representation ρ ( g ) for g in a grid around identity.

Original: D.2.6. Lie Algebra and Latent Traversal
Revised: D.2.6. Lie Algebra and Latent Traversal

Original: We show how learning a mapping to the group algebra can be leveraged to navigate the group and the data manifold.
Revised: We show how learning a mapping to the group algebra can be leveraged to navigate the group and the data manifold.

Original: We remind that ρ = exp ◦ ϕ , where ϕ maps to the algebra g of the group G , and exp is the matrix exponential which gives a connection between the algebra and the lie group.
Revised: We remind that ρ = exp ◦ ϕ , where ϕ maps to the algebra g of the group G , and exp is the matrix exponential which gives a connection between the algebra and the lie group.

Original: The mapping ϕ
Revised: The mapping ϕ

Original: = ϕ 1 ⊕ ϕ 2 and the group representation ρ = ρ 1 ⊕ ρ 2 are constrained on the space of block diagonal matrices of the form M = M 1 ⊕ M 2 each of dimension 2 × 2.
Revised: = ϕ 1 ⊕ ϕ 2 and the group representation ρ = ρ 1 ⊕ ρ 2 are constrained on the space of block diagonal matrices of the form M = M 1 ⊕ M 2 each of dimension 2 × 2.

Original: However since each block is made of elements from a 1 D subgroup of GL 2 ( R ): SO (2), its algebra is the 1 D subalgebra of M 2 ( R ) of skew-symmetric matrices.
Revised: However since each block is made of elements from a 1 D subgroup of GL 2 ( R ): SO (2), its algebra is the 1 D subalgebra of M 2 ( R ) of skew-symmetric matrices.

Original: We find this subspace by performing a PCA over each of the sets { ϕ 1 ( g t ) } t and { ϕ 2 ( g t ) } t for a batch { g t } t of
Revised: We find this subspace by performing a PCA over each of the sets { ϕ 1 ( g t ) } t and { ϕ 2 ( g t ) } t for a batch { g t } t of

Revised: Extended Abstract Track

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (cid:74) − 2 ,(cid:75) ×
Revised: (cid:74) − 2 ,(cid:75) ×

Original: (cid:74) − 2 ,(cid:75)  observed transitions.
Revised: (cid:74) − 2 ,(cid:75) Extended Abstract Track observed transitions.

Original: The first component for each block E 1 and E 2 corresponds to the only base vector for that subalgebra.
Revised: The first component for each block E 1 and E 2 corresponds to the only base vector for that subalgebra.

Original: We find:
Revised: We find:

Original: E 1 ⊕ 0 2 , 2 =   − 0 .
Revised: E 1 ⊕ 0 2 , 2 =   − 0 .

Original: 04 − 0 .
Revised: 04 − 0 .

Original: 65 00 .− 0 . 04 

Revised: 65 00 .− 0 . 04

Original:  0  0  0 2 , 2 ⊕ E 2 ≈   0  0  0 0 0 .
Revised:  0  0  0 2 , 2 ⊕ E 2 ≈   0  0  0 0 0 .

Original: 01 − 0 .0 0 0 .0 .
Revised: 01 − 0 .0 0 0 .0 .

Original: 
Revised: 

Original: We obtain the figure 5 by linearly traversing the subalgebras through t ( E 1 ⊕ 0 2 , 2 ) and t (0 2 , 2 ⊕ E 2 ) for equally spaced values of t ∈ (cid:74) 0 , 9 (cid:75) and passing it to the matrix exponential which yields invertible matrices of the form R 1 ,t = e tE 1 ⊕ I 2 and R 2 ,t
Revised: We obtain the figure 5 by linearly traversing the subalgebras through t ( E 1 ⊕ 0 2 , 2 ) and t (0 2 , 2 ⊕ E 2 ) for equally spaced values of t ∈ (cid:74) 0 , 9 (cid:75) and passing it to the matrix exponential which yields invertible matrices of the form R 1 ,t = e tE 1 ⊕ I 2 and R 2 ,t

Original: = I 2 ⊕ e tE 2 .
Revised: = I 2 ⊕ e tE 2 .

Original: We encode an arbitrary initial observation to obtain its representation vector z , and traverse the latent space through R i,t z .
Revised: We encode an arbitrary initial observation to obtain its representation vector z , and traverse the latent space through R i,t z .

Original: We decode the obtained vectors to obtain the predicted images.
Revised: We decode the obtained vectors to obtain the predicted images.

Original: The group algebra offers a smooth parametrization of the group and consequently of the data manifold and enables the prediction of observations evolution in the absence of performed actions.
Revised: The group algebra offers a smooth parametrization of the group and consequently of the data manifold and enables the prediction of observations evolution in the absence of performed actions.

Original: Indeed, in the above example, all transformations can be obtained in the form exp( t 1 E 1 ) ⊕ exp( t 2 E 2 ) for t 1 , t 2 ∈ R .
Revised: Indeed, in the above example, all transformations can be obtained in the form exp( t 1 E 1 ) ⊕ exp( t 2 E 2 ) for t 1 , t 2 ∈ R .

Original: D.2.7. Sparsity Based Disentanglement
Revised: D.2.7. Sparsity Based Disentanglement

Original: We show in Figure 6 the learned group representation for the same dataset used in the experiment described in Appendix D.2.1 where instead of enforcing a block diagonal structure, it emerges under the influence of the sparsity inducing loss described in Appendix C.2.
Revised: We show in Figure 6 the learned group representation for the same dataset used in the experiment described in Appendix D.2.1 where instead of enforcing a block diagonal structure, it emerges under the influence of the sparsity inducing loss described in Appendix C.2.

Original: We do not assume knowledge of the representation dimension and we set it to a value high enough (8 in the example) and we find the HAE learns the disentangled representation found in the main experiment and trivializes the extra dimensions.
Revised: We do not assume knowledge of the representation dimension and we set it to a value high enough (8 in the example) and we find the HAE learns the disentangled representation found in the main experiment and trivializes the extra dimensions.

Original: D.2.8. Additional Experiment
Revised: D.2.8. Additional Experiment

Original: We consider a subset of the dataset consisting of all variations of the heart under a fixed scale.
Revised: We consider a subset of the dataset consisting of all variations of the heart under a fixed scale.

Original: As such the heart is acted on by the group G =
Revised: As such the heart is acted on by the group G =

Original: G θ × G x × G y =
Revised: G θ × G x × G y =

Original: C 39 × C 32 × C 32 .
Revised: C 39 × C 32 × C 32 .

Revised: Extended Abstract Track

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We train a similar model to the one described in the section D.2.2 by changing the latent space to 6 dimensions, the group representation is fixed of the form ρ = ρ 1 ⊕ ρ 2 ⊕ ρeach of dimension 2.
Revised: We train a similar model to the one described in the section D.2.2 by changing the latent space to 6 dimensions, the group representation is fixed of the form ρ = ρ 1 ⊕ ρ 2 ⊕ ρeach of dimension 2.

Original: We expect a disentangled representation space Z
Revised: We expect a disentangled representation space Z

Original: = Z 1 ⊕ Z 2 ⊕ Z 3 .
Revised: = Z 1 ⊕ Z 2 ⊕ Z 3 .

Original: For the visualization of the learned representation manifold Figure 7, we visualize each subspace
Revised: For the visualization of the learned representation manifold Figure [Figure]

Original:  Z i separately by only varying one generative factor and keeping all else fixed.
Revised: we visualize each subspace Z i separately by only varying one generative factor and keeping all else fixed.

Original: We also visualized the learned representations for a subset of transitions corresponding to the elementary generative transitions for each subgroup in Figure 8.
Revised: We also visualized the learned representations for a subset of transitions corresponding to the elementary generative transitions for each subgroup in Figure [Figure]

Original: D.3. Rollouts prediction
Revised: D.3. Rollouts prediction

Original: One important application of learning structured representation is to predict how the observations would change given sequences of actions.
Revised: One important application of learning structured representation is to predict how the observations would change given sequences of actions.

Original: We compare HAE to two other approaches of modeling the dynamics in the latent space: (1) Unstructured : z t +1 = h ( z t , g t ), where h is a learnable function.
Revised: We compare HAE to two other approaches of modeling the dynamics in the latent space: (1) Unstructured : z t +1 = h ( z t , g t ), where h is a learnable function.

Original: Similar approaches have been widely adapted in recent model-based deep RL methods (Ha and Schmidhuber, 2018; Schrittwieser et al., 2020).
Revised: Similar approaches have been widely adapted in recent model-based deep RL methods (Ha and Schmidhuber, 2018; Schrittwieser et al., 2020).

Original: (2) Rotations : z t +

Revised: (2) Rotations : z t +

Original: =
Revised: =

Original: R g z t , where R g = (cid:81) i,j G ( i, j, θ ij,g ) and G ( i, j, θ ij,g ) are the Givens rotation matrices.
Revised: R g z t , where R g = (cid:81) i,j G ( i, j, θ ij,g ) and G ( i, j, θ ij,g ) are the Givens rotation matrices.

Original: This approach was proposed by Quessard et al.
Revised: This approach was proposed by Quessard et al.

Original: and was shown to be capable of learning group-structured representations when the actions are sampled along the true latents.
Revised: and was shown to be capable of learning group-structured representations when the actions are sampled along the true latents.

Original: We evaluate the methods in an offline setting, where we train each method on a given set of 2-step trajectories and test their generalization ability on a hold-out set of 128-
Revised: We evaluate the methods in an offline setting, where we train each method on a given set of 2-step trajectories and test their generalization ability on a hold-out set of 128-

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: step trajectories.
Revised: step trajectories.

Original: Figures 10 and 9 shows the reconstruction loss for each method on the test trajectories.
Revised: Figures 10 and 9 shows the reconstruction loss for each method on the test trajectories.

Original: Figure 10 considers rollouts of actions sampled in a discrete grid around the identity while Figure 9 considers sparse actions sampled along a single property as was performed in Quessard et al.
Revised: Figure 10 considers rollouts of actions sampled in a discrete grid around the identity while Figure 9 considers sparse actions sampled along a single property as was performed in Quessard et al.

Original: Our result suggests that when the actions sampled are not sparse (i.e., each action may involve changes in multiple generating factors),  the Rotations method may perform worse than the Unstructured method while HAE can outperform them significantly.
Revised: Our result suggests that when the actions sampled are not sparse (i.e., each action may involve changes in multiple generating factors), Extended Abstract Track the Rotations method may perform worse than the Unstructured method while HAE can outperform them significantly.

Original: In our experiment, we allow the agent to control the object in the dSprite dataset withactions.
Revised: In our experiment, we allow the agent to control the object in the dSprite dataset withactions.

Original: Namely, translation in the x-y axes, rotation in both directions (clockwise, counterclockwise), and idle.
Revised: Namely, translation in the x-y axes, rotation in both directions (clockwise, counterclockwise), and idle.

Original: Each action corresponds to an increment/decrement in one of the generating factors of the dataset, except for the identity, which does nothing.
Revised: Each action corresponds to an increment/decrement in one of the generating factors of the dataset, except for the identity, which does nothing.

Original: Additionally, we use the heart shape from the dataset to fully utilize the orientation latent factor as other shapes present rotational symmetry.
Revised: Additionally, we use the heart shape from the dataset to fully utilize the orientation latent factor as other shapes present rotational symmetry.

Original: We train each method with a set of pre-generated set of 2-step trajectories and evaluate on a hold-out pre-generated set of 128-step trajectories.
Revised: We train each method with a set of pre-generated set of 2-step trajectories and evaluate on a hold-out pre-generated set of 128-step trajectories.

Original: For each trajectory, we begin by sampling a random initial state (x,y position and orientation) from all possible states.
Revised: For each trajectory, we begin by sampling a random initial state (x,y position and orientation) from all possible states.

Original: After that, we sample actions uniformly from the 7 possible actions at each step until the number of steps is satisfied.
Revised: After that, we sample actions uniformly from the 7 possible actions at each step until the number of steps is satisfied.

Original: For the Rotations method (Quessard et al., 2020) and HAE, each action is represented using a matrix ρ ( g ), and the transition in the latent space is simply z t +

Revised: For the Rotations method (Quessard et al., 2020) and HAE, each action is represented using a matrix ρ ( g ), and the transition in the latent space is simply z t +

Original: = ρ ( g t )
Revised: = ρ ( g t )

Original: z t .
Revised: z t .

Original: For the
Revised: For the

Original: Unstructured method, we use a 2 layer MLP of size [128, 128] to model the transition by z t +1 =
Revised: Unstructured method, we use a 2 layer MLP of size [128, 128] to model the transition by z t +1 =

Original: f θ ( z t , g t ), where we concatenate the latent vector z t and the one-hot encoding of the action a t .
Revised: f θ ( z t , g t ), where we concatenate the latent vector z t and the one-hot encoding of the action a t .

Original: The reconstruction loss is the same for all three methods, as described in Section 3.2.
Revised: The reconstruction loss is the same for all three methods, as described in Section 3.2.

Original: For
Revised: For

Original: HAE, we additionally add the latent prediction loss
Revised: HAE, we additionally add the latent prediction loss

Original: L pred as described in Section 3.2.
Revised: L pred as described in Section 3.2.

Original: We increase γ to 1600 which we found to be more stable when matrices are directly parameterized instead of mapped from MLPs.
Revised: We increase γ to 1600 which we found to be more stable when matrices are directly parameterized instead of mapped from MLPs.

Original: For the Rotations method, an additional entanglement loss L ent is required to encourage each matrix to act on the fewest dimensions of the latent space, which is equal to
Revised: For the Rotations method, an additional entanglement loss L ent is required to encourage each matrix to act on the fewest dimensions of the latent space, which is equal to

Original: [Equation]
Revised: [Equation]

Original: β ) | θ gi,j |with θ gα,β = max i,j | θ gi,j | .
Revised: =( α,β ) | θ gi,j |with θ gα,β = max i,j | θ gi,j | .

Original: For the Unstructured method, we only use the reconstruction loss and no additional terms.
Revised: For the Unstructured method, we only use the reconstruction loss and no additional terms.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: D.4. Multi-objects
Revised: D.4. Multi-objects

Original: We consider the subset of the dSprites dataset consisting of three shapes (heart, square and ellipse) acted on by the previous group of 2D cyclic translations G = C x × C y .
Revised: We consider the subset of the dSprites dataset consisting of three shapes (heart, square and ellipse) acted on by the previous group of 2D cyclic translations G = C x × C y .

Original: The action of the group is not transitive , as it describes a separate orbit for each shape (see appendix A).
Revised: The action of the group is not transitive , as it describes a separate orbit for each shape (see appendix A).

Original: Indeed, no intervention transforms a shape into another: the model is trained on sequences of 3 images containing the same shape.
Revised: Indeed, no intervention transforms a shape into another: the model is trained on sequences of 3 images containing the same shape.

Original: In this experiment, we augment the representation space by one dimension on which the group representation acts trivially.
Revised: In this experiment, we augment the representation space by one dimension on which the group representation acts trivially.

Original: We therefore learn 5-dimensional encodings of the observations 2 × 2 D subspaces acted on by the subrepresentations of the group G and 1 representation unit trivially acted on.
Revised: We therefore learn 5-dimensional encodings of the observations 2 × 2 D subspaces acted on by the subrepresentations of the group G and 1 representation unit trivially acted on.

Original: Our results in Figure 11 show that the model not only learns the representation of the cyclic translation group action shared among shapes, but also learns to separate the representation of observations by shape along the last G -invariant representation unit, giving rise to three identical manifolds.
Revised: Our results in Figure 11 show that the model not only learns the representation of the cyclic translation group action shared among shapes, but also learns to separate the representation of observations by shape along the last G -invariant representation unit, giving rise to three identical manifolds.

Original: This is reminiscent of the two-streams hypothesis of visual processing (Goodale and Milner, 1992), the ”What” pathway processes information related to object identity, while the ”Where” pathway processes information related to the object pose, or the required motor action for manipulation.
Revised: This is reminiscent of the two-streams hypothesis of visual processing (Goodale and Milner, 1992), the ”What” pathway processes information related to object identity, while the ”Where” pathway processes information related to the object pose, or the required motor action for manipulation.

Original: Figure 11 is obtained by projecting the 5D encodings of the dataset on a 2D space through Random Matrix Projection.
Revised: Figure 11 is obtained by projecting the 5D encodings of the dataset on a 2D space through Random Matrix Projection.

Original: Note that although the three different tori appear of different sizes, it is only an artifact of the projection while the tori are identical.
Revised: Note that although the three different tori appear of different sizes, it is only an artifact of the projection while the tori are identical.

Original: D.5. Unsupervised Identity Separation from Intervention
Revised: D.5. Unsupervised Identity Separation from Intervention

Original: A more thorough analysis of the experiment can be found in Appendix D.4.
Revised: A more thorough analysis of the experiment can be found in Appendix D.4.

Original: Appendix E. Link to Neuroscience
Revised: Appendix E. Link to Neuroscience

Original: It is interesting to note that similar cyclic embeddings have been reported in neuroscience, for example, in the hippocampus of mice where toroid-like embeddings encode the animals’ head-direction (Chaudhuri et al., 2019).
Revised: It is interesting to note that similar cyclic embeddings have been reported in neuroscience, for example, in the hippocampus of mice where toroid-like embeddings encode the animals’ head-direction (Chaudhuri et al., 2019).

Original: In fact, our cyclic shape translation task restricted to a single dimension could be viewed as an agent horizontally rotating its head while observing its environment.
Revised: In fact, our cyclic shape translation task restricted to a single dimension could be viewed as an agent horizontally rotating its head while observing its environment.

Original: Appendix F. Limitations
Revised: Appendix F. Limitations

Original: One limitation of our theoretical analysis is the deterministic nature of our model, the intrinsically non-linear nature of the problem would make the theoretical analysis more challenging in the stochastic setting.
Revised: One limitation of our theoretical analysis is the deterministic nature of our model, the intrinsically non-linear nature of the problem would make the theoretical analysis more challenging in the stochastic setting.

Original: On the experimental side, we solely assessed the extraction and internal representation of geometric action structure, we leave it to future work to test if the same principled HAE approach generalizes to learn other, non cyclic, facets of the structure of the external world. 
Revised: On the experimental side, we solely assessed the extraction and internal representation of geometric action structure, we leave it to future work to test if the same principled HAE approach generalizes to learn other, non cyclic, facets of the structure of the external world. Extended Abstract Track

Original: F.1. Computational resources
Revised: F.1. Computational resources

Original: The experiments were performed on an NVIDIA GeForce RTX 3090 and A100 GPUs.
Revised: The experiments were performed on an NVIDIA GeForce RTX 3090 and A100 GPUs.

Original: The training of our optimal models run for approximately 20 mins.
Revised: The training of our optimal models run for approximately 20 mins.

Original: Appendix G. Third-Party Software G.1. Deep Learning Framework
Revised: Appendix G. Third-Party Software G.1. Deep Learning Framework

Original: To implement our architecture we used the deep learning framework PyTorch.
Revised: To implement our architecture we used the deep learning framework PyTorch.

Original: Paszke et al.
Revised: Paszke et al.

Original: G.2. Hyperparameter Search
Revised: G.2. Hyperparameter Search

Original: We used the hyperparameter search utility provided in the hypnettorch project https:// github.com/chrhenning/hypnettorch/tree/master/hypnettorch/hpsearch to perform a random grid search.
Revised: We used the hyperparameter search utility provided in the hypnettorch project https:// github.com/chrhenning/hypnettorch/tree/master/hypnettorch/hpsearch to perform a random grid search.

Original: G.3. Dataset
Revised: G.3. Dataset

Original: In the presented experiments, we used the dSprites dataset Matthey et al.
Revised: In the presented experiments, we used the dSprites dataset Matthey et al.

Original: The dSprites dataset is an image dataset of white sprites on a black background, varying in shape (heart, ellipse, square), in scale (6 values), in orientation (39 values, cyclic), in x and y position (32 values each).
Revised: The dSprites dataset is an image dataset of white sprites on a black background, varying in shape (heart, ellipse, square), in scale (6 values), in orientation (39 values, cyclic), in x and y position (32 values each).

Original: We consider all factors besides shape to be cyclic, in particular for the x and y positions, we ”glued” opposite borders of images into a torus.
Revised: We consider all factors besides shape to be cyclic, in particular for the x and y positions, we ”glued” opposite borders of images into a torus.

Original: The resolution of the images is 64 × 64 pixels.
Revised: The resolution of the images is 64 × 64 pixels.

Original: Appendix H. Societal Impact
Revised: Appendix H. Societal Impact

Original: This work proposes new findings in basic research.
Revised: This work proposes new findings in basic research.

Original: To the best of our knowledge, this work does not have immediate applications with a negative societal impact.
Revised: To the best of our knowledge, this work does not have immediate applications with a negative societal impact.