Original: D YNAMICS - AWARE E MBEDDINGS
Revised: D YNAMICS - AWARE E MBEDDINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper we consider self-supervised representation learning to improve sample efﬁciency in reinforcement learning (RL).
Revised: In this paper we consider self-supervised representation learning to improve sample efﬁciency in reinforcement learning (RL).

Original: We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences.
Revised: We propose a forward prediction objective for simultaneously learning embeddings of states and action sequences.

Original: These embeddings capture the structure of the environment’s dynamics, enabling efﬁcient policy learning.
Revised: These embeddings capture the structure of the environment’s dynamics, enabling efﬁcient policy learning.

Original: We demonstrate that our action embeddings alone improve the sample efﬁciency and peak performance of model-free RL on control from low-dimensional states.
Revised: We demonstrate that our action embeddings alone improve the sample efﬁciency and peak performance of model-free RL on control from low-dimensional states.

Original: By combining state and action embeddings, we achieve efﬁcient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.
Revised: By combining state and action embeddings, we achieve efﬁcient learning of high-quality policies on goal-conditioned continuous control from pixel observations in only 1-2 million environment steps.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, there has been a lot of excitement around end-to-end model-free reinforcement learning for control, both in simulation (Lillicrap et al., 2015; Andrychowicz et al., 2018; Haarnoja et al., 2018b; Fujimoto et al., 2018) and on real hardware (Kalashnikov et al., 2018; Haarnoja et al., 2018c).
Revised: In recent years, there has been a lot of excitement around end-to-end model-free reinforcement learning for control, both in simulation (Lillicrap et al., 2015; Andrychowicz et al., 2018; Haarnoja et al., 2018b; Fujimoto et al., 2018) and on real hardware (Kalashnikov et al., 2018; Haarnoja et al., 2018c).

Original: In this paradigm, we simultaneously learn intermediate representations and policies by maximizing rewards provided by environment.
Revised: In this paradigm, we simultaneously learn intermediate representations and policies by maximizing rewards provided by environment.

Original: End-to-end learning has one indisputable advantage: since every component of the system is optimized for the end objective, there are no sub-optimal modules that limit best-case performance by losing task-relevant information.
Revised: End-to-end learning has one indisputable advantage: since every component of the system is optimized for the end objective, there are no sub-optimal modules that limit best-case performance by losing task-relevant information.

Original: Learning only from the target task is however a double-edged sword.
Revised: Learning only from the target task is however a double-edged sword.

Original: When the end objective provides only weak signal for learning, a policy with a poor representation may require many samples to learn a better one.
Revised: When the end objective provides only weak signal for learning, a policy with a poor representation may require many samples to learn a better one.

Original: By contrast, a policy with a good representation may be able to rapidly ﬁt a simple function of that representation even with weak signal.
Revised: By contrast, a policy with a good representation may be able to rapidly ﬁt a simple function of that representation even with weak signal.

Original: [Figure]
Revised: [Figure]

Original: The agent (blue dot) can move continuously left and right to reach the goal (gold star).
Revised: The agent (blue dot) can move continuously left and right to reach the goal (gold star).

Original: Consider the environment shown in Figure 1, and two representations of its state: coordinates and pixels.
Revised: Consider the environment shown in Figure 1, and two representations of its state: coordinates and pixels.

Original: As a function of the agent’s x coordinate, the value function is simple and smooth.
Revised: As a function of the agent’s x coordinate, the value function is simple and smooth.

Original: The coordinate representation has structure which is useful for learning about the task; namely, points which are close in L 2 distance have similar values.
Revised: The coordinate representation has structure which is useful for learning about the task; namely, points which are close in L 2 distance have similar values.

Original: By contrast, a pixel representation of the agent’s state (below, blue) is practically a one-hot vector.
Revised: By contrast, a pixel representation of the agent’s state (below, blue) is practically a one-hot vector.

Original: Two states whose x coordinates differ by one unit have pixels exactly as different as states which differ by 100 units.
Revised: Two states whose x coordinates differ by one unit have pixels exactly as different as states which differ by 100 units.

Original: This illustrates the importance of good representations and the potential of representation learning to aid RL.
Revised: This illustrates the importance of good representations and the potential of representation learning to aid RL.

Original: We propose a self-supervised objective for learning embeddings of states and action sequences such that a pair of states or action sequences will be close together if they have similar outcomes.
Revised: We propose a self-supervised objective for learning embeddings of states and action sequences such that a pair of states or action sequences will be close together if they have similar outcomes.

Original: This objective simultaneously trains a smooth embedding space for states and a temporally abstract action space for control which is taskindependent and generalizes across goals and objects.
Revised: This objective simultaneously trains a smooth embedding space for states and a temporally abstract action space for control which is taskindependent and generalizes across goals and objects.

Original: We demonstrate the effectiveness of our representation learning objective by training the twin delayed deep deterministic policy gradient algorithm (TD3) (Fujimoto et al., 2018) with learned action and state spaces.
Revised: We demonstrate the effectiveness of our representation learning objective by training the twin delayed deep deterministic policy gradient algorithm (TD3) (Fujimoto et al., 2018) with learned action and state spaces.

Original: With a learned representation of temporally abstract actions, our method exhibits improved sample efﬁciency compared to state-of-the-art RL methods on control tasks, with larger gains on more complex environments.
Revised: With a learned representation of temporally abstract actions, our method exhibits improved sample efﬁciency compared to state-of-the-art RL methods on control tasks, with larger gains on more complex environments.

Original: When additionally combined with our learned state representation, our method allows TD3 to scale to pixel observations.
Revised: When additionally combined with our learned state representation, our method allows TD3 to scale to pixel observations.

Original: We demonstrate good
Revised: We demonstrate good

Original: [Figure]
Revised: [Figure]

Original: performance on a simple family of goal-conditioned 2D control tasks within a few million environment steps without adjusting any TD3 hyperparameters.
Revised: performance on a simple family of goal-conditioned 2D control tasks within a few million environment steps without adjusting any TD3 hyperparameters.

Original: This stands in contrast to end-to-end model-free
Revised: This stands in contrast to end-to-end model-free

Original: RL from pixels, which requires extensive tuning (Lillicrap et al., 2015) and on the order ofmillion environment steps 1 (Barth-Maron et al., 2018).
Revised: RL from pixels, which requires extensive tuning (Lillicrap et al., 2015) and on the order ofmillion environment steps 1 (Barth-Maron et al., 2018).

Original:  D YNAMICS - AWARE EMBEDDINGS 2. N OTATION
Revised:  D YNAMICS - AWARE EMBEDDINGS 2. N OTATION

Original: We consider the framework of reinforcement learning in Markov decision processes (MDPs).
Revised: We consider the framework of reinforcement learning in Markov decision processes (MDPs).

Original: 2 We denote the state of an environment (e.g. joint angles of a robot or pixels) by s ∈ S , and we assume that the states given by the environment satisfy the Markov property.
Revised: 2 We denote the state of an environment (e.g. joint angles of a robot or pixels) by s ∈ S , and we assume that the states given by the environment satisfy the Markov property.

Original: We refer to a sequence of actions { a 1 , ..., a k } ∈ A k using the shorthand a k .
Revised: We refer to a sequence of actions { a 1 , ..., a k } ∈ A k using the shorthand a k .

Original: We use s (cid:48) ∼ T ( s, a ) to refer to the environment’s (stochastic) transition function, and overload it to accept sequences of actions: s t + k ∼ T ( s t , a kt ) .
Revised: We use s (cid:48) ∼ T ( s, a ) to refer to the environment’s (stochastic) transition function, and overload it to accept sequences of actions: s t + k ∼ T ( s t , a kt ) .

Original: 2. M ODEL AND LEARNING OBJECTIVE
Revised: 2. M ODEL AND LEARNING OBJECTIVE

Original: We propose that a good representation for reinforcement learning should represent states or actions close together if they have similar outcomes (resulting trajectories).
Revised: We propose that a good representation for reinforcement learning should represent states or actions close together if they have similar outcomes (resulting trajectories).

Original: This allows the agent to generalize from a small number of samples since each sample accurately reﬂects the value of all the states or actions in its neighborhood.
Revised: This allows the agent to generalize from a small number of samples since each sample accurately reﬂects the value of all the states or actions in its neighborhood.

Original: In a Markov decision process the outcome of taking an action a in a state s is summarized by the distribution of resulting states p ( s (cid:48) | s, a ) =
Revised: In a Markov decision process the outcome of taking an action a in a state s is summarized by the distribution of resulting states p ( s (cid:48) | s, a ) =

Original: T ( s, a ) .
Revised: T ( s, a ) .

Original: Therefore we construct a method which embeds states and actions such that nearby embeddings have similar distributions of next states.
Revised: Therefore we construct a method which embeds states and actions such that nearby embeddings have similar distributions of next states.

Original: Our method, which we call Dynamics-aware Embedding (DynE), learns encoders e s and e a which embed a state and action sequence into latent spaces z s ∈
Revised: Our method, which we call Dynamics-aware Embedding (DynE), learns encoders e s and e a which embed a state and action sequence into latent spaces z s ∈

Original: Z s and z a ∈ Z a respectively.
Revised: Z s and z a ∈ Z a respectively.

Original: These encodings are optimized to form a maximally compressed representation of the sufﬁcient statistics of p ( s (cid:48) | s, a k ) such that p ( s (cid:48) | s, a k )
Revised: These encodings are optimized to form a maximally compressed representation of the sufﬁcient statistics of p ( s (cid:48) | s, a k ) such that p ( s (cid:48) | s, a k )

Original: ≈ p ( s (cid:48)
Revised: ≈ p ( s (cid:48)

Original: | z s , z a ) .
Revised: | z s , z a ) .

Original: We approximate this by maximizing the following objective:
Revised: We approximate this by maximizing the following objective:

Original: [Equation]
Revised: [Equation]

Original: where z s ∼ e s ( s ) , z a ∼ e a ( a k ) , and ρ π is the distribution of transitions under a behavior policy π .
Revised: where z s ∼ e s ( s ) , z a ∼ e a ( a k ) , and ρ π is the distribution of transitions under a behavior policy π .

Original: The DynE objective is similar to a β -VAE (Higgins et al., 2017a) for s (cid:48)
Revised: The DynE objective is similar to a β -VAE (Higgins et al., 2017a) for s (cid:48)

Original: but with a different variational family; like a β -VAE, it forms a variational lower bound on p ( s (cid:48) )
Revised: but with a different variational family; like a β -VAE, it forms a variational lower bound on p ( s (cid:48) )

Original: when β
Revised: when β

Original: = γ = 1 .
Revised: = γ = 1 .

Original: Where a variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014) or β -VAE chooses the variational family to be Q = { q ( z | s (cid:48) ) } , we use a factored latent space { z s , z a } and independent posterior approximations given the previous state and the action: Q = { ( q ( z s | s ) , q ( z a | a k )) } .
Revised: Where a variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014) or β -VAE chooses the variational family to be Q = { q ( z | s (cid:48) ) } , we use a factored latent space { z s , z a } and independent posterior approximations given the previous state and the action: Q = { ( q ( z s | s ) , q ( z a | a k )) } .

Original: This factorization yields separate encoders for states and actions where the state encoder’s output is valid for any action and vice versa.
Revised: This factorization yields separate encoders for states and actions where the state encoder’s output is valid for any action and vice versa.

Original: The DynE objective can also be interpreted in the information bottleneck (IB) framework (Tishby et al., 2000).
Revised: The DynE objective can also be interpreted in the information bottleneck (IB) framework (Tishby et al., 2000).

Original: In the IB framework term (1) is the prediction objective and terms (2) and (3) regularize the latent representation to remove all extraneous information.
Revised: In the IB framework term (1) is the prediction objective and terms (2) and (3) regularize the latent representation to remove all extraneous information.

Original: Our construction is nearly identical to the approximate information bottleneck proposed by Alemi et al.
Revised: Our construction is nearly identical to the approximate information bottleneck proposed by Alemi et al.

Original: (2016), with the main difference being the factorization of the representation into separate state and action components.
Revised: (2016), with the main difference being the factorization of the representation into separate state and action components.

Original: In our experiments we use an isotropic Normal distribution for p ( s (cid:48) | z s , z a ; θ ) such that term (1) reduces to (cid:107) f
Revised: In our experiments we use an isotropic Normal distribution for p ( s (cid:48) | z s , z a ; θ ) such that term (1) reduces to (cid:107) f

Original: ( z s , z a ; θ ) − s (cid:48)
Revised: ( z s , z a ; θ ) − s (cid:48)

Original: (cid:107) 22 where f computes the mean.
Revised: (cid:107) 22 where f computes the mean.

Original: We use diagonal-covariance Normal distributions for e s and e a such that { µ s , σ 2 s } =
Revised: We use diagonal-covariance Normal distributions for e s and e a such that { µ s , σ 2 s } =

Original: e s ( s ) , { µ a , σ 2 a } =
Revised: e s ( s ) , { µ a , σ 2 a } =

Original: e a ( a k ) , z s ∼ N ( µ s , σ 2 s ) , and z a ∼ N ( µ a , σ 2 a ) .
Revised: e a ( a k ) , z s ∼ N ( µ s , σ 2 s ) , and z a ∼ N ( µ a , σ 2 a ) .

Original: The behavior policy we use for data collection is π = Unif ( A ) .
Revised: The behavior policy we use for data collection is π = Unif ( A ) .

Original:  U SING LEARNED EMBEDDINGS FOR REINFORCEMENT LEARNING 
Revised:  U SING LEARNED EMBEDDINGS FOR REINFORCEMENT LEARNING

Original: D ECODING TO RAW ACTIONS
Revised: D ECODING TO RAW ACTIONS

Original: In order to be useful for RL, the abstract action space produced by the encoder must be decodeable to raw actions in the environment.
Revised: In order to be useful for RL, the abstract action space produced by the encoder must be decodeable to raw actions in the environment.

Original: Since the mapping from action sequences to high-level actions is many-to-one, inverting it is nontrivial.
Revised: Since the mapping from action sequences to high-level actions is many-to-one, inverting it is nontrivial.

Original: We simplify this ill-posed problem by deﬁning an objective with a single optimum.
Revised: We simplify this ill-posed problem by deﬁning an objective with a single optimum.

Original: Once the action encoder e a is fully trained, we hold it ﬁxed and train an action decoder
Revised: Once the action encoder e a is fully trained, we hold it ﬁxed and train an action decoder

Original: d a to minimize
Revised: d a to minimize

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term of this objective ensures that the action decoder d is a one-sided inverse of e a ; that is, e a ( d a ( z a )) =
Revised: The ﬁrst term of this objective ensures that the action decoder d is a one-sided inverse of e a ; that is, e a ( d a ( z a )) =

Original: z
Revised: z

Original: a but d a ( e a ( a 1 , ..., a k )) (cid:54) =
Revised: a but d a ( e a ( a 1 , ..., a k )) (cid:54) =

Original: a 1 , ..., a k .
Revised: a 1 , ..., a k .

Original: The second term of the loss ensures that d a is in particular the minimum-norm one-sided inverse of e a and gives the objective for the output of d a a single minimum.
Revised: The second term of the loss ensures that d a is in particular the minimum-norm one-sided inverse of e a and gives the objective for the output of d a a single minimum.

Original: Out of all the action sequences which have the same outcome, the minimum-norm sequence is desireable as it leads to trajectories which are smooth and consume less energy.
Revised: Out of all the action sequences which have the same outcome, the minimum-norm sequence is desireable as it leads to trajectories which are smooth and consume less energy.

Original: We choose λ to be small (e.g. 10 − 2 ) to ensure that the reconstruction criterion dominates the optimization.
Revised: We choose λ to be small (e.g. 10 − 2 ) to ensure that the reconstruction criterion dominates the optimization.

Original: 3. E FFICIENT RL WITH TEMPORAL ABSTRACTION
Revised: 3. E FFICIENT RL WITH TEMPORAL ABSTRACTION

Original: Once equipped with a decoder which maps from high-level actions to sequences of raw actions, we train a high-level policy that solves a task by selecting high-level actions.
Revised: Once equipped with a decoder which maps from high-level actions to sequences of raw actions, we train a high-level policy that solves a task by selecting high-level actions.

Original: In this section we extend the deterministic policy gradient (Silver et al., 2014) family of algorithms to work with temporallyextended actions while maintaining off-policy updates and learning from every environment step.
Revised: In this section we extend the deterministic policy gradient (Silver et al., 2014) family of algorithms to work with temporallyextended actions while maintaining off-policy updates and learning from every environment step.

Original: This allows our method to achieve superior sample efﬁciency when working with high-level actions.
Revised: This allows our method to achieve superior sample efﬁciency when working with high-level actions.

Original: In particular, we extend the twin delayed deep deterministic policy gradient (TD3) algorithm (Fujimoto et al., 2018) to work with the DynE representation of actions to form an algorithm we call DynE-TD3.
Revised: In particular, we extend the twin delayed deep deterministic policy gradient (TD3) algorithm (Fujimoto et al., 2018) to work with the DynE representation of actions to form an algorithm we call DynE-TD3.

Original: We ﬁrst describe why DPG requires modiﬁcations to accommodate temporally-abstracted actions.
Revised: We ﬁrst describe why DPG requires modiﬁcations to accommodate temporally-abstracted actions.

Original: One simple approach to combining DynE with DPG would be to incorporate the k -step DynE action space into the environment to form a new MDP.
Revised: One simple approach to combining DynE with DPG would be to incorporate the k -step DynE action space into the environment to form a new MDP.

Original: This MDP allows the use of DPG without modiﬁcation; however, it only emits observations once every k timesteps.
Revised: This MDP allows the use of DPG without modiﬁcation; however, it only emits observations once every k timesteps.

Original: As a result, after N steps in the original environment, the deterministic policy µ and critic function Q can only be trained on
Revised: As a result, after N steps in the original environment, the deterministic policy µ and critic function Q can only be trained on

Original: N/k observations.
Revised: N/k observations.

Original: This has a substantial impact on sample efﬁciency when measured in the original environment.
Revised: This has a substantial impact on sample efﬁciency when measured in the original environment.

Original: Instead we require an algorithm which can perform updates to the policy µ and critic Q for every environment step.
Revised: Instead we require an algorithm which can perform updates to the policy µ and critic Q for every environment step.

Original: To do this, we train both µ and Q in the abstract action space with minor changes to their updates.
Revised: To do this, we train both µ and Q in the abstract action space with minor changes to their updates.

Original: We distinguish these functions which use DynE actions from their raw equivalents by adding a superscript DynE, i.e. µ DynE and Q DynE .
Revised: We distinguish these functions which use DynE actions from their raw equivalents by adding a superscript DynE, i.e. µ DynE and Q DynE .

Original: We augment the critic function with an additional input, i , which represents the number of steps 0 ≤ i < k of the current embedded action z that have already been executed.
Revised: We augment the critic function with an additional input, i , which represents the number of steps 0 ≤ i < k of the current embedded action z that have already been executed.

Original: This forms the DynE-TD3 critic:
Revised: This forms the DynE-TD3 critic:

Original: [Equation]
Revised: [Equation]

Original: In plain language, the value of being on step i of abstract action e t is the value of ﬁnishing the remaining ( k − i ) steps of z t and then continuing on following the policy.
Revised: In plain language, the value of being on step i of abstract action e t is the value of ﬁnishing the remaining ( k − i ) steps of z t and then continuing on following the policy.

Original: This is similar to the idea of k -step returns (Sutton & Barto, 2018), but with a variable k which depends on the step within the current plan.
Revised: This is similar to the idea of k -step returns (Sutton & Barto, 2018), but with a variable k which depends on the step within the current plan.

Revised: Whereas k -step returns would typically require an off-policy correction such as

Revised: Retrace (Munos et al., 2016), conditioning on z t and i determines all k − i actions in the return.

Revised: In effect, they remain a single action, making the update valid off policy.

Original: The DynE critic is trained by minimizing the Bellman error implied by the equation above.
Revised: The DynE critic is trained by minimizing the Bellman error implied by eq.

Original: To update the policy we follow the standard DPG technique of using the gradient of the critic.
Revised: To update the policy we follow the standard DPG technique of using the gradient of the critic.

Original: We modify the algorithm to take into account that i = 

Revised: We modify the algorithm to take into account that i =

Original: at the time of issuing a new high-level action.
Revised: at the time of issuing a new high-level action.

Original: The gradient of the return with respect to the policy parameters is then
Revised: The gradient of the return with respect to the policy parameters is then

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: given that data was collected according to a behavior policy π .
Revised: given that data was collected according to a behavior policy π .

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Successor representations, an inspiration for this work, represent a state by the expected rate of future visits to other states (Dayan, 1993; Kulkarni et al., 2016b; Barreto et al., 2017).
Revised: Successor representations, an inspiration for this work, represent a state by the expected rate of future visits to other states (Dayan, 1993; Kulkarni et al., 2016b; Barreto et al., 2017).

Original: Successor representations have been demonstrated to be an effective model of animal and human learning (Momennejad et al., 2017; Stachenfeld et al., 2017).
Revised: Successor representations have been demonstrated to be an effective model of animal and human learning (Momennejad et al., 2017; Stachenfeld et al., 2017).

Original: They are also one of the earliest realizations of the idea of representing each state by its future.
Revised: They are also one of the earliest realizations of the idea of representing each state by its future.

Original: Whereas successor representations learn future occupancy maps for a particular policy, we learn an embedding space where states are close together if they have similar outcomes for any policy.
Revised: Whereas successor representations learn future occupancy maps for a particular policy, we learn an embedding space where states are close together if they have similar outcomes for any policy.

Original: Several papers have proposed using (variational) auto-encoders to learn embeddings for observations (Lange & Riedmiller, 2010; Van Hoof et al., 2016; Higgins et al., 2017b;
Revised: Several papers have proposed using (variational) auto-encoders to learn embeddings for observations (Lange & Riedmiller, 2010; Van Hoof et al., 2016; Higgins et al., 2017b;

Original: Caselles-Dupré et al., 2018); unlike our work, these models operate on a single observation at a time and do not depend on the environment dynamics.
Revised: Caselles-Dupré et al., 2018); unlike our work, these models operate on a single observation at a time and do not depend on the environment dynamics.

Original: Forward prediction has also been used as an auxiliary task to speed
Revised: Forward prediction has also been used as an auxiliary task to speed

Original: RL training (Jaderberg et al., 2016).
Revised: RL training (Jaderberg et al., 2016), and Jonschkowski et al.

Revised: (2017) learn representations which adhere to physical constraints.

Original: Ghosh et al.
Revised: Ghosh et al.

Original: (2018) propose to learn state embeddings using the action distribution of a goal-conditioned policy; however, their technique depends on already having a successful policy.
Revised: (2018) propose to learn state embeddings using the action distribution of a goal-conditioned policy; however, their technique depends on already having a successful policy.

Original: Other work has proposed to use mutual information maximization to learn embeddings which facilitate exploration via intrinsic motivation (Kim et al., 2018).
Revised: Other work has proposed to use mutual information maximization to learn embeddings which facilitate exploration via intrinsic motivation (Kim et al., 2018).

Original: Similarly to this work, hierarchical reinforcement learning seeks to learn temporal abstractions.
Revised: Similarly to this work, hierarchical reinforcement learning seeks to learn temporal abstractions.

Original: These abstractions are variously deﬁned as skills (Florensa et al., 2017;
Revised: These abstractions are variously deﬁned as skills (Florensa et al., 2017;

Original: Hausman et al., 2018), options (Sutton et al., 1999; Bacon et al., 2017), or goal-directed sub-policies (Kulkarni et al., 2016a;
Revised: Hausman et al., 2018), options (Sutton et al., 1999; Bacon et al., 2017), or goal-directed sub-policies (Kulkarni et al., 2016a;

Original: Vezhnevets et al., 2017).
Revised: Vezhnevets et al., 2017).

Original: Most closely related are SeCTAR (Co-Reyes et al., 2018) and HIRO (Nachum et al., 2018).
Revised: Most closely related are SeCTAR (Co-Reyes et al., 2018) and HIRO (Nachum et al., 2018).

Original: SeCTAR simultaneously learns a generative model of future states and a low-level policy which can reach those states.
Revised: SeCTAR simultaneously learns a generative model of future states and a low-level policy which can reach those states.

Original: HIRO learns a representation of goals such that a high-level policy can induce any action in a low-level policy.
Revised: HIRO learns a representation of goals such that a high-level policy can induce any action in a low-level policy.

Original: Unlike this work, both SeCTAR and
Revised: Unlike this work, both SeCTAR and

Original: HIRO learn state-dependent low-level policies, not action representations.
Revised: HIRO learn state-dependent low-level policies, not action representations.

Original: Furthermore SeCTAR assumes the reward function is given ahead of time, and HIRO’s off-policy performance depends on an approximate re-labeling of action sequences to train the high-level policy.
Revised: Furthermore SeCTAR assumes the reward function is given ahead of time, and HIRO’s off-policy performance depends on an approximate re-labeling of action sequences to train the high-level policy.

Original: Also related are methods which attempt to learn embeddings of single actions to enable efﬁcient learning in very large action spaces (Dulac-Arnold et al., 2015; Chandak et al., 2019).
Revised: Also related are methods which attempt to learn embeddings of single actions to enable efﬁcient learning in very large action spaces (Dulac-Arnold et al., 2015; Chandak et al., 2019).

Original: In particular,
Revised: In particular,

Original: Chandak et al.
Revised: Chandak et al.

Original: (2019) learns a latent space of actions based on the effects of an action on the environment.
Revised: (2019) learns a latent space of actions based on the effects of an action on the environment.

Original: However, their latent spaces are for a single action and they do not consider learned state representations.
Revised: However, their latent spaces are for a single action and they do not consider learned state representations.

Original: Another related direction is learning embeddings of one or more actions from demonstrations (Tennenholtz & Mannor, 2019); this embedded action space builds in prior knowledge from the demonstrator and can allow faster learning.
Revised: Another related direction is learning embeddings of one or more actions from demonstrations (Tennenholtz & Mannor, 2019); this embedded action space builds in prior knowledge from the demonstrator and can allow faster learning.

Original:  R EPRESENTATION E XPERIMENTS
Revised:  R EPRESENTATION E XPERIMENTS

Original: In this section we empirically investigate how the learned DynE representations reshape the problem of reinforcement learning.
Revised: In this section we empirically investigate how the learned DynE representations reshape the problem of reinforcement learning.

Original: First we make a connection between temporal abstraction and exploration, revealing that DynE actions result in better state coverage.
Revised: First we make a connection between temporal abstraction and exploration, revealing that DynE actions result in better state coverage.

Original: Then we probe the relationship between
Revised: Then we probe the relationship between

Original: DynE state embeddings and the task value function.
Revised: DynE state embeddings and the task value function.

Original: 5. T EMPORAL ABSTRACTION AND EXPLORATION
Revised: 5. T EMPORAL ABSTRACTION AND EXPLORATION

Original: When embedding an action sequence, the DynE objective seeks to preserve information about the outcome of that action sequence (i.e. the change in state), but minimize information about the original action sequence.
Revised: When embedding an action sequence, the DynE objective seeks to preserve information about the outcome of that action sequence (i.e. the change in state), but minimize information about the original action sequence.

Original: As shown in Appendix C, this leads to a representation where all action sequences which have similar outcomes embed close together.
Revised: As shown in Appendix D, this leads to a representation where all action sequences which have similar outcomes embed close together.

Original: We propose that this temporally abstract action space, where actions correspond to multi-step outcomes, allows random actions to explore the environment more efﬁciently.
Revised: We propose that this temporally abstract action space, where actions correspond to multi-step outcomes, allows random actions to explore the environment more efﬁciently.

Original: We empirically validate the exploration beneﬁts of the temporally abstract DynE actions.
Revised: We empirically validate the exploration beneﬁts of the temporally abstract DynE actions.

Original: Figureshows that uniformly sampling a DynE action results in a nearly uniform distribution over the states reachable within k steps.
Revised: Figureshows that uniformly sampling a DynE action results in a nearly uniform distribution over the states reachable within k steps.

Original: Over the course of an entire episode, selecting DynE actions uniformly at random reaches faraway states more often than random exploration with raw actions.
Revised: Over the course of an entire episode, selecting DynE actions uniformly at random reaches faraway states more often than random exploration with raw actions.

Original: Appendix E shows the qualitative difference between random trajectories in the raw and DynE action spaces.
Revised: Appendix F shows the qualitative difference between random trajectories in the raw and DynE action spaces, and

Revised: Appendix C studies the impact of varying k on the performance of a learned policy.

Original: [Figure]
Revised: [Figure]

Original: 5. S TATE REPRESENTATIONS
Revised: 5. S TATE REPRESENTATIONS

Original: The DynE objective compresses states while preserving information about the outcome of taking any action in that state.
Revised: The DynE objective compresses states while preserving information about the outcome of taking any action in that state.

Original: If this compression is successful, states which have similar outcomes will be close together in embedding space.
Revised: If this compression is successful, states which have similar outcomes will be close together in embedding space.

Original: In an MDP, two states which have identical successor states have values which differ by at most the range of the reward function r max − r min .
Revised: In an MDP, two states which have identical successor states have values which differ by at most the range of the reward function r max − r min .

Original: While in general states which lead to merely similar successors may have arbitrarily different value, we suggest that in many tasks of interest, similar successors may entail similar value.
Revised: While in general states which lead to merely similar successors may have arbitrarily different value, we suggest that in many tasks of interest [Figure] similar successors may entail similar value [Figure]

Original: We investigate whether the DynE state embedding leads to neighborhoods with similar value in the
Revised: We investigate whether the DynE state embedding leads to neighborhoods with similar value in the

Original: Reacher Vertical environment.
Revised: Reacher Vertical environment.

Original: We collect 10K states from a random policy in the environment and perform dimensionality reduction on three representations of those states: the DynE embedding of state images, low-dimensional joint states, and pixels.
Revised: We collect 10K states from a random policy in the environment and perform dimensionality reduction on three representations of those states: the DynE embedding of state images, low-dimensional joint states, and pixels.

Original: Figure 4 shows the results of this dimensionality reduction, in which every point is colored by its value under a fully-trained TD3 policy on the low-d states.
Revised: Figure 4 shows the results of this dimensionality reduction, in which every point is colored by its value under a fully-trained TD3 policy on the low-d states.

Original: DynE embeddings have neighborhoods with more similar values than states or pixels.
Revised: DynE embeddings have neighborhoods with more similar values than states or pixels.

Original:  R EINFORCEMENT LEARNING EXPERIMENTS
Revised:  R EINFORCEMENT LEARNING EXPERIMENTS

Original: In this section we assess the effectiveness of the DynE representations for deep RL, individually analyzing the contributions of the action and state representations before combining them.
Revised: In this section we assess the effectiveness of the DynE representations for deep RL, individually analyzing the contributions of the action and state representations before combining them.

Original: First we evaluate the DynE action space on a set of six tasks with low-dimensional state observations, testing its usefulness across a set of tasks and object interactions.
Revised: First we evaluate the DynE action space on a set of six tasks with low-dimensional state observations, testing its usefulness across a set of tasks and object interactions.

Original: Then, we test the DynE state space on a set of three tasks with pixel observations.
Revised: Then, we test the DynE state space on a set of three tasks with pixel observations.

Original: Finally, we combine DynE actions with DynE observations, verifying that the two learned representations are complementary.
Revised: Finally, we combine DynE actions with DynE observations, verifying that the two learned representations are complementary.

Original: Appendix B provides a full description of hyperparameters and model architectures, and all of the code for DynE is available on GitHub at https://github.com/dyne-submission/ dynamics-aware-embeddings .
Revised: Appendix B provides a full description of hyperparameters and model architectures, and all of the code for DynE is available on GitHub at https://github.com/dyne-submission/ dynamics-aware-embeddings .

Original: Environments We use six continuous control tasks from two families implemented in the MuJoCo simulator (Todorov et al., 2012) to evaluate our method.
Revised: Environments We use six continuous control tasks from two families implemented in the MuJoCo simulator (Todorov et al., 2012) to evaluate our method.

Original: Within each family, the task and observation space change but the robot being controlled stays roughly the same, allowing us to test the transferrability of the DynE action space between tasks.
Revised: Within each family, the task and observation space change but the robot being controlled stays roughly the same, allowing us to test the transferrability of the DynE action space between tasks.

Original: The Reacher family consists of three of tasks which involve controlling a 2D, 2DoF arm to interact with various objects.
Revised: The Reacher family consists of three of tasks which involve controlling a 2D, 2DoF arm to interact with various objects.

Original: The 7DoF family of tasks from
Revised: The 7DoF family of tasks from

Original: OpenAI Gym (Brockman et al., 2016) is quite difﬁcult, featuring three tasks in which a 3D, 7DoF arm must use different end effectors to push or throw various objects to randomly-generated goal positions.
Revised: OpenAI Gym (Brockman et al., 2016) is quite difﬁcult, featuring three tasks in which a 3D, 7DoF arm must use different end effectors to push or throw various objects to randomly-generated goal positions.

Original: Images and detailed descriptions of both families of tasks are available in Appendix A.
Revised: Images and detailed descriptions of both families of tasks are available in Appendix A.

Original: 6. L OW - DIMENSIONAL STATES
Revised: 6. L OW - DIMENSIONAL STATES

Original: For training the DynE action representation we use 100K steps with a uniformly random behavior policy in the simplest environment in each family with no reward or other supervisory signal.
Revised: For training the DynE action representation we use 100K steps with a uniformly random behavior policy in the simplest environment in each family with no reward or other supervisory signal.

Original: As this
Revised: As this

Original: DynE pretraining is unsupervised and only occurs once for each family of environments, the x axis on these training curves refers only to the samples used to train the policy.
Revised: DynE pretraining is unsupervised and only occurs once for each family of environments, the x axis on these training curves refers only to the samples used to train the policy.

Original: 3 We then transfer this action representation to all three environments in the family.
Revised: 3 We then transfer this action representation to all three environments in the family.

Original: When training DynE-TD3 we use all of the default hyperparameters from the TD3 implementation across all environments.
Revised: When training DynE-TD3 we use all of the default hyperparameters from the TD3 implementation across all environments.

Original: We directly test the impact of switching from raw to DynE actions by comparing TD3 to DynE-TD3.
Revised: We directly test the impact of switching from raw to DynE actions by comparing TD3 to DynE-TD3.

Original: For completeness we compare with two additional state-of-the-art model-free methods: soft actorcritic (SAC) (Haarnoja et al., 2018b;c) and proximal policy optimization (PPO){+actor
+}
Revised: For completeness we compare with two additional state-of-the-art model-free methods: soft {+actor
+}

Original: (Schulman et al., 2017). We also compare with soft actor-critic with latent space policies (SAC-LSP) (Haarnoja et al., 2018a), an innovative hierarchical method which transforms a low-level action space into an abstract
Revised: On all environments except the simplest (Reacher Vertical) shifting the DynE-TD3 plot by 100K steps does not affect the ordering of the results.

Original: [Figure]
Revised: [Figure]

Original:  one by training an invertible low-level policy.
Revised: critic (SAC) (Haarnoja et al., 2018b;c) and proximal policy optimization (PPO) (Schulman et al., 2017). We also compare with soft actor-critic with latent space policies (SAC-LSP) (Haarnoja et al., 2018a), an innovative hierarchical method which transforms a low-level action space into an abstract one by training an invertible low-level policy.

Original: In all cases we use the ofﬁcial implementations 456 and the MuJoCo hyperparameters used by the authors.
Revised: In all cases we use the ofﬁcial implementations 456 and the MuJoCo hyperparameters used by the authors.

Original: We also attempted to compare with the hierarchical method by Nachum et al.
Revised: We also attempted to compare with the hierarchical method by Nachum et al.

Original: (2018), but after several emails with the authors and dozens of experiments we were unable to get it to converge on tasks other than those in their paper.
Revised: (2018), but after several emails with the authors and dozens of experiments we were unable to get it to converge on tasks other than those in their paper.

Original: Results Figure 5 shows the results of these experiments.
Revised: Results Figure 5 shows the results of these experiments.

Original: Most signiﬁcantly, they show that switching from the raw action space (TD3 curve) to the DynE action space results in faster training and allows TD3 to solve the difﬁcult 7DoF suite of tasks.
Revised: Most signiﬁcantly, they show that switching from the raw action space (TD3 curve) to the DynE action space results in faster training and allows TD3 to solve the difﬁcult 7DoF suite of tasks.

Original: We see that the DynE action space generalizes across several tasks with the same robot, even when interacting with objects unseen during training.
Revised: We see that the DynE action space generalizes across several tasks with the same robot, even when interacting with objects unseen during training.

Original: It is especially worth noting that the gains from DynE increase as the tasks become harder, maintaining convergence, stability, and low variance in the face of high-dimensional control with difﬁcult exploration.
Revised: It is especially worth noting that the gains from DynE increase as the tasks become harder, maintaining convergence, stability, and low variance in the face of high-dimensional control with difﬁcult exploration.

Original: Since SAC-LSP (Haarnoja et al., 2018a) performs similarly but worse than
Revised: Since SAC-LSP (Haarnoja et al., 2018a) performs similarly but worse than

Original: SAC we test it only on the simpler Reacher family of tasks; meanwhile, the PPO curves do not enter the frame on the Reacher family of tasks due to its poor sample efﬁciency.
Revised: SAC we test it only on the simpler Reacher family of tasks; meanwhile, the PPO curves do not enter the frame on the Reacher family of tasks due to its poor sample efﬁciency.

Original: 6. P IXELS
Revised: 6. P IXELS

Original: Using the Reacher family of environments we evaluate several state representations by their effectiveness for policy learning with TD3.
Revised: Using the Reacher family of environments we evaluate several state representations by their effectiveness for policy learning with TD3.

Original: We evaluate two established methods for learning representations from single images.
Revised: We evaluate two established methods for learning representations from single images.

Original: “DARLA” is the Disentangled Representation Learning Agent proposed by Higgins et al.
Revised: “DARLA” is the Disentangled Representation Learning Agent proposed by Higgins et al.

Original: (2017b)
Revised: (2017b)

Original: with the denoising autoencoder loss, which is referred to in that work as β -VAE DAE .
Revised: with the denoising autoencoder loss, which is referred to in that work as β -VAE DAE .

Original: “VAE” is a standard variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014), which has previously been found to learn effective representations for control (Van Hoof
Revised: “VAE” is a standard variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014), which has previously been found to learn effective representations for control (Van Hoof

Original: et al., 2016)
Revised: et al., 2016)

Original: ; it is equivalent to DARLA with the pixel-space loss and β = 1 .
Revised: ; it is equivalent to DARLA with the pixel-space loss and β = 1 .

Original: Since these representations operate on a single frame at a time, we apply them to the most recent four frames independently and then concatenate the embeddings before feeding them to the policy.
Revised: Since these representations operate on a single frame at a time, we apply them to the most recent four frames independently and then concatenate the embeddings before feeding them to the policy.

Original: These representations have compressed latent spaces, but they encode no knowledge of the environment’s dynamics, allowing us to evaluate the importance of incorporating the dynamics into our embeddings.
Revised: These representations have compressed latent spaces, but they encode no knowledge of the environment’s dynamics, allowing us to evaluate the importance of incorporating the dynamics into our embeddings.

Original: Next we evaluate representation learning methods whose objectives incorporate the dynamics.
Revised: Next we evaluate representation learning methods whose objectives incorporate the dynamics.

Original: “SDynE,” for State DynE, is the DynE state embedding e s , and “SA-DynE” combines the DynE state and action representations.
Revised: “SDynE,” for State DynE, is the DynE state embedding e s , and “SA-DynE” combines the DynE state and action representations.

Original: “S-Deterministic” and “SA-Deterministic” are ablations of the corresponding DynE methods which have the same forward-prediction objective but no KL or noise on the latent representations.
Revised: “S-Deterministic” and “SA-Deterministic” are ablations of the corresponding DynE methods which have the same forward-prediction objective but no KL or noise on the latent representations.

Original: Comparing the DynE methods to their respective ablations reveals the contribution of explicitly introducing a compression objective to the latent space.
Revised: Comparing the DynE methods to their respective ablations reveals the contribution of explicitly introducing a compression objective to the latent space.

Original: For training all of the learned representations we use a dataset of 100K steps in each environment from a uniformly random policy.
Revised: For training all of the learned representations we use a dataset of 100K steps in each environment from a uniformly random policy.

Original: In every case we train TD3 with the learned representations using all of the default hyperparameters from the ofﬁcial TD3 implementation.
Revised: In every case we train TD3 with the learned representations using all of the default hyperparameters from the ofﬁcial TD3 implementation.

Original: We compare these representation learning methods with TD3 trained from pixels.
Revised: We compare these representation learning methods with TD3 trained from pixels.

Original: As there are no experiments on pixels in the TD3 paper, we performed extensive search over network architectures and hyperparameters.
Revised: As there are no experiments on pixels in the TD3 paper, we performed extensive search over network architectures and hyperparameters.

Original: We included in our search the conﬁgurations used in the pixel experiments of DDPG (Lillicrap et al., 2015) as well as those used in successful discrete-action RL works from pixels (Schulman et al., 2017; Kostrikov, 2018; Espeholt et al., 2018).
Revised: We included in our search the conﬁgurations used in the pixel experiments of DDPG (Lillicrap et al., 2015) as well as those used in successful discrete-action RL works from pixels (Schulman et al., 2017; Kostrikov, 2018; Espeholt et al., 2018).

Original: Results Figure 6 shows the results of these experiments.
Revised: Results Figure 6 shows the results of these experiments.

Original: We ﬁnd that the single-image methods are unable to solve any of the three tasks from pixels; TD3 from pixels diverges in all cases, while VAE and DARLA learn gradually at best.
Revised: We ﬁnd that the single-image methods are unable to solve any of the three tasks from pixels; TD3 from pixels diverges in all cases, while VAE and DARLA learn gradually at best.

Original: If simply reducing the dimension of the states were sufﬁcient to enable effective policy training, we would expect good performance from these methods.
Revised: If simply reducing the dimension of the states were sufﬁcient to enable effective policy training, we would expect good performance from these methods.

Original: S-DynE and S-Deterministic, which incorporate the dynamics into their representation learning objectives, perform far better.
Revised: S-DynE and S-Deterministic, which incorporate the dynamics into their representation learning objectives, perform far better.

Original: The minimality imposed by the DynE objective allows S-DynE and SA-Dyne to outperform their deterministic ablations.
Revised: The minimality imposed by the DynE objective allows S-DynE and SA-Dyne to outperform their deterministic ablations.

Original: SA-DynE learns rapidly and reliably, ﬁnding behaviors which qualitatively solve all three tasks.
Revised: SA-DynE learns rapidly and reliably, ﬁnding behaviors which qualitatively solve all three tasks.

Original: The improvement of SA-DynE over S-DynE shows that the state and action representations are complementary.
Revised: The improvement of SA-DynE over S-DynE shows that the state and action representations are complementary.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work we proposed a method, Dynamics-aware Embedding (DynE), that jointly learns embedded representations of states and actions for reinforcement learning.
Revised: In this work we proposed a method, Dynamics-aware Embedding (DynE), that jointly learns embedded representations of states and actions for reinforcement learning.

Original: Our experiments reveal that DynE action embeddings lead to more efﬁcient exploration, resulting in more sample efﬁcient learning on complex tasks, while DynE state embeddings allow unmodiﬁed model-free RL algorithms to scale to pixel observations.
Revised: Our experiments reveal that DynE action embeddings lead to more efﬁcient exploration, resulting in more sample efﬁcient learning on complex tasks, while DynE state embeddings allow unmodiﬁed model-free RL algorithms to scale to pixel observations.

Original: When combined, the DynE state and action embeddings result in stable, sample-efﬁcient learning of high-quality policies from pixels.
Revised: When combined, the DynE state and action embeddings result in stable, sample-efﬁcient learning of high-quality policies from pixels.

Original: A. P IXELS
Revised: A. P IXELS

Original: We use full-color images rendered at 256x256 and resized to 64x64 pixels.
Revised: We use full-color images rendered at 256x256 and resized to 64x64 pixels.

Original: In order to allow the agents to perceive motion, we stack the current frame with the three most recent frames, resulting in an observation of dimension 12x64x64.
Revised: In order to allow the agents to perceive motion, we stack the current frame with the three most recent frames, resulting in an observation of dimension 12x64x64.

Original: A PPENDIX B H YPERPARAMETERS AND D YN E TRAINING
Revised: A PPENDIX B H YPERPARAMETERS AND D YN E TRAINING

Original: For DynE-TD3 we use all of the default hyperparameters from the TD3 code 7 across all tasks.
Revised: For DynE-TD3 we use all of the default hyperparameters from the TD3 code 7 across all tasks.

Original: For all experiments we choose the dimension of the DynE action space to be equal to the dimension of a single action in the environment.
Revised: For all experiments we choose the dimension of the DynE action space to be equal to the dimension of a single action in the environment.

Original: We set the number of actions in the DynE space to be k = 4 for all experiments except Thrower-v2 , for which we use k = 8 .
Revised: We set the number of actions in the DynE space to be k = 4 for all experiments except Thrower-v2 , for which we use k = 8 .

Original: We use the Adam optimizer (Kingma &
Revised: We use the Adam optimizer (Kingma &

Original: Ba, 2014) with learning rate 10 − 4 .
Revised: Ba, 2014) with learning rate 10 − 4 .

Original: All our experiments used recent-model NVidia GPUs.
Revised: All our experiments used recent-model NVidia GPUs.

Original: Training on states When computing log-likelihoods we divide by the number of dimensions in the state in an attempt to make the correct settings of γ invariant to the observation dimension; the same result could be achieved by multiplying the values of γ that we report by the state dimension and changing the learning rate.
Revised: Training on states When computing log-likelihoods we divide by the number of dimensions in the state in an attempt to make the correct settings of γ invariant to the observation dimension; the same result could be achieved by multiplying the values of γ that we report by the state dimension and changing the learning rate.

Original: With that scaling we set we set our hyperparameters
Revised: With that scaling we set we set our hyperparameters

Original: γ = λ = 10 −across all environments.
Revised: γ = λ = 10 −across all environments.

Original: We concatenate all the joint angles and velocities to use as the states during representation learning.
Revised: We concatenate all the joint angles and velocities to use as the states during representation learning.

Original: We preprocess the s, s (cid:48) pairs by ﬁrst taking the difference ∆ s
Revised: We preprocess the s, s (cid:48) pairs by ﬁrst taking the difference ∆ s

Original: = s (cid:48) − s and then whitening so that ∆ s has zero mean and unit variance in each dimension.
Revised: = s (cid:48) − s and then whitening so that ∆ s has zero mean and unit variance in each dimension.

Original: This preprocessing encourages the encoder to represent both position and velocity in the latent space; the scales of these two components are quite different.
Revised: This preprocessing encourages the encoder to represent both position and velocity in the latent space; the scales of these two components are quite different.

Original: We use fully-connected networks for the action encoder e a and the conditional state predictor f .
Revised: We use fully-connected networks for the action encoder e a and the conditional state predictor f .

Original: Each function has two hidden layers of 400 units.
Revised: Each function has two hidden layers of 400 units.

Original: Training this model should take 5-10 minutes on GPU.
Revised: Training this model should take 5-10 minutes on GPU.

Original: Training on pixels We train a DynE model for each environment, taking in a stack of frames and a sequence of k = 4 actions and predicting future states.
Revised: Training on pixels We train a DynE model for each environment, taking in a stack of frames and a sequence of k = 4 actions and predicting future states.

Original: To speed training we predict only the two latest frames of the future state (i.e. the picture of the world at time t + k and t + k − 1 ) instead of all four.
Revised: To speed training we predict only the two latest frames of the future state (i.e. the picture of the world at time t + k and t + k − 1 ) instead of all four.

Original: When doing RL we take the state encoder e s from this model and use it to preprocess all states from the environment.
Revised: When doing RL we take the state encoder e s from this model and use it to preprocess all states from the environment.

Original: We set the dimension of the state embedding z s to 100.
Revised: We set the dimension of the state embedding z s to 100.

Original: We did not try other options, and given the sensitivity of RL to state dimension a smaller setting would very likely yield faster learning.
Revised: We did not try other options, and given the sensitivity of RL to state dimension a smaller setting would very likely yield faster learning.

Original: We set β
Revised: We set β

Original: = γ = 1 , at which setting DynE is optimizing a variational lower bound on p ( s t + k | s t , a k ) .
Revised: = γ = 1 , at which setting DynE is optimizing a variational lower bound on p ( s t + k | s t , a k ) .

Original: We recommend ensuring that the predictions (not generations) from the model are correctly rendering all the task-relevant objects; if β and γ are too high, the model may incur lower loss by ignoring details in the image.
Revised: We recommend ensuring that the predictions (not generations) from the model are correctly rendering all the task-relevant objects; if β and γ are too high, the model may incur lower loss by ignoring details in the image.

Original: We use cyclic KL annealing (Liu et al., 2019) to improve convergence over a wide range of settings.
Revised: We use cyclic KL annealing (Liu et al., 2019) to improve convergence over a wide range of settings.

Original: We use the DCGAN architecture (Radford et al., 2015) for the image encoder e s and the predictor f .
Revised: We use the DCGAN architecture (Radford et al., 2015) for the image encoder e s and the predictor f .

Original: The action encoder e a is fully connected with two hidden layers of 400 units.
Revised: The action encoder e a is fully connected with two hidden layers of 400 units.

Original: Training this model takes 1-2 hours on GPU.
Revised: Training this model takes 1-2 hours on GPU.

Original: A PPENDIX C V ISUALIZING THE D YN E ACTION SPACE
Revised: A PPENDIX C V ARYING LEVELS OF TEMPORAL ABSTRACTION

Revised: We study the impact of varying k , the level of temporal abstraction in the DynE action space.

Revised: We ﬁnd that increasing k improves performance and learning speed up to a point; beyond this point, performance degrades.

Revised: The optimal setting of k will depend on the environment dynamics.

Revised: We expect that environments with very slow dynamics will beneﬁt from a greater degree of temporal abstraction.

Original:  To better understand the structure in the DynE action embedding space, we visualize the relationship between the outcome of a sequence of actions and the DynE embedding of those actions.
Revised: [Figure] A PPENDIX D V ISUALIZING THE D YN E ACTION SPACE To better understand the structure in the DynE action embedding space, we visualize the relationship between the outcome of a sequence of actions and the DynE embedding of those actions.

Original: In particular, when embedding an action sequence, the DynE objective seeks to preserve information about the outcome of that action sequence (i.e. the change in state), but minimize information about the original action sequence.
Revised: When embedding an action sequence, the DynE objective seeks to preserve information about the outcome of that action sequence (i.e. the change in state), but minimize information about the original action sequence.

Original: Therefore we should see that all action sequences which have similar outcomes embed close together, regardless of the actions along the way.
Revised: Therefore we should see that all action sequences which have similar outcomes embed close together, regardless of the actions along the way.

Original: We investigate this by plotting the 2D

Original: DynE embedding of 10K action sequences and coloring them by their outcome under the environment dynamics.

Original: If the DynE embedding depends on the actions within the sequence and not just the outcome, some sequences with similar outcomes (colors) will be embedded far apart.

Original: Figure 9 shows the result of thisexperiment in a simple Point environment with an easy-to-visualize 2D ( x, y ) state.
Revised: Figure 10 investigates this in a simple Point environment with an easy-to-visualize 2D ( x, y ) state.

Original: For this simple problem, we see that all pairs of action sequences a k 1 and a k 2 with similar outcomes are close together in the embedding space.
Revised: For this simple problem, we see that all pairs of action sequences a k 1 and a k 2 with similar outcomes are close together in the embedding space.

Original: The correspondence between the two spaces appears to remain strong for high-dimensional and nonlinear environments, but is much harder to render in two dimensions.
Revised: The correspondence between the two spaces appears to remain strong for high-dimensional and nonlinear environments, but is much harder to render in two dimensions.

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX D E XTENDED RESULTS
Revised: A PPENDIX E E XTENDED RESULTS

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: A PPENDIX E E XPLORATION WITH RAW AND D YN E ACTION SPACES
Revised: A PPENDIX F E XPLORATION WITH RAW AND D YN E ACTION SPACES

Original: [Figure]
Revised: [Figure]