Original: E FFICIENT E MPOWERMENT E STIMATION FOR U NSUPERVISED S TABILIZATION
Revised: E FFICIENT E MPOWERMENT E STIMATION FOR U NSUPERVISED S TABILIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Intrinsically motivated artiﬁcial agents learn advantageous behavior without externally-provided rewards.
Revised: Intrinsically motivated artiﬁcial agents learn advantageous behavior without externally-provided rewards.

Original: Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright positions, which is a prototypical intrinsically motivated behavior for upright standing and walking.
Revised: Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright positions, which is a prototypical intrinsically motivated behavior for upright standing and walking.

Original: That follows from the coincidence between the objective of stabilization and the objective of empowerment.
Revised: This follows from the coincidence between the objective of stabilization and the objective of empowerment.

Original: Unfortunately, sample-based estimation of this kind of mutual information is challenging.
Revised: Unfortunately, sample-based estimation of this kind of mutual information is challenging.

Original: Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased,  have high sample complexity, and are unstable in training.
Revised: Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased, unstable in training, and have high sample complexity.

Original: In this work, we propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel, which allows us to efﬁciently calculate an unbiased estimator of empowerment by convex optimization.
Revised: In this work, we propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel, which allows us to efﬁciently calculate an unbiased estimator of empowerment by convex optimization.

Original: We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of our method by comparing it to the existing VLB approaches.
Revised: We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of our method by comparing it to the existing VLB approaches.

Original: Speciﬁcally, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function and allows estimation of empowerment from images.
Revised: Speciﬁcally, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images.

Original: Consequently, our method opens a path to wider and easier adoption of empowerment for many applications
Revised: Consequently, our method opens a path to wider and easier adoption of empowerment for various applications.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Intrinsic motivation allows artiﬁcial and biological agents to acquire useful behaviours without external knowledge (Barto et al.
Revised: Intrinsic motivation allows artiﬁcial and biological agents to acquire useful behaviours without external knowledge (Barto et al.

Original: ; Chentanez et al.
Revised: ; Chentanez et al.

Original: ; Schmidhuber (2010); Barto (2013);
Revised: ; Schmidhuber (2010); Barto (2013);

Original: Oudeyer et al.
Revised: Oudeyer et al.

Original: In the framework of reinforcement learning (RL), this external knowledge is usually provided by an expert through a task-speciﬁc reward, which is optimized by an artiﬁcial agent towards a desired behavior (Mnih et al.
Revised: In the framework of reinforcement learning (RL), this external knowledge is usually provided by an expert through a task-speciﬁc reward, which is optimized by an artiﬁcial agent towards a desired behavior (Mnih et al.

Original: (2013); Schulman et al.
Revised: (2013); Schulman et al.

Original: In contrast, an intrinsic reward can arise solely from the interaction between the agent and environment, which eliminates the need for domain knowledge and reward engineering in some settings (Mohamed & Rezende (2015);
Revised: In contrast, an intrinsic reward can arise solely from the interaction between the agent and environment, which eliminates the need for domain knowledge and reward engineering in some settings (Mohamed & Rezende (2015);

Original: Houthooft et al.
Revised: Houthooft et al.

Original: ; Pathak et al.
Revised: ; Pathak et al.

Original: Previously, it was shown that maximizing mutual information (Cover & Thomas (2012)) between an agent’s actuators and sensors can guide the agent towards states in the environment which have a higher potential to achieve a larger number of eventual goals (Klyubin et al.
Revised: Previously, it was shown that maximizing mutual information (Cover & Thomas (2012)) between an agent’s actuators and sensors can guide the agent towards states in the environment with higher potential to achieve a larger number of eventual goals (Klyubin et al.

Original: (2005); Wissner-Gross & Freer (2013)).
Revised: (2005); Wissner-Gross & Freer (2013)).

Original: Maximizing this kind of mutual information is known as the empowerment principle (Klyubin et al.
Revised: Maximizing this kind of mutual information is known as the empowerment principle (Klyubin et al.

Original: ; Salge et al.
Revised: ; Salge et al.

Original: Empowerment is a function of the state.

Original: Previously, it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment in various dynamical control systems (Jung et al.
Revised: Previously, it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment in various dynamical control systems (Jung et al.

Original: ; Salge et al.
Revised: ; Salge et al.

Original: ; Karl et al.
Revised: ; Karl et al.

Original: In this application, ascending the gradient of the empowerment function coincides with the objective of optimal control for stabilization at an unstable equilibrium (Strogatz (2018)), which is an important task for both engineering (Todorov (2006)) and biological 1 systems; we refer to this as the essential property of empowerment.
Revised: In this application, ascending the gradient of the empowerment function coincides with the objective of optimal control for stabilization at an unstable equilibrium (Strogatz (2018)), which is an important task for both engineering (Todorov (2006)) and biological 2 systems; we refer to this as the essential property of empowerment.

Original: It follows from the aforementioned prior works that a plausible estimate of the empowerment function should possess this essential property.
Revised: It follows from the aforementioned prior works that a plausible estimate of the empowerment function should possess this essential property.

Original: Empowerment has been found to be useful for a broad spectrum of applications, including: unsupervised skill discovery (Sharma et al.
Revised: Empowerment has been found to be useful for a broad spectrum of applications, including: unsupervised skill discovery (Sharma et al.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: ; Gregor et al.
Revised: ; Gregor et al.

Original: ; Karl et al.
Revised: ; Karl et al.

Original: ; Campos et al.
Revised: ; Campos et al.

Original: (2020); human-agent coordination (Salge & Polani (2017); Guckelsberger et al.
Revised: (2020); human-agent coordination (Salge & Polani (2017); Guckelsberger et al.

Original: ; assistance (Yuqing et al.
Revised: ; assistance (Du et al.

Original: (2021)); and stabilization (Tiomkin et al.
Revised: (2021)); and stabilization (Tiomkin et al.

Original: However, estimation of mutual information is hard (McAllester & Statos (2018)).

Original: Fortunately, recent advances in deep learning (Goodfellow et al.

Original: (2016)) have brought new opportunities for the estimation of mutual information from samples.

Original: Past work has utilized variational lower bounds (VLBs) (Poole et al.
Revised: Past work has utilized variational lower bounds (VLBs) (Poole et al.

Original: ; Alemi et al.
Revised: ; Alemi et al.

Original: ; Tschannen et al.
Revised: ; Tschannen et al.

Original: ; Mohamed & Rezende (2015)) to achieve an estimate of the empowerment.
Revised: ; Mohamed & Rezende (2015)) to achieve an estimate of the empowerment.

Original: However, VLB approaches to empowerment in dynamical control systems (Sharma et al.
Revised: However, VLB approaches to empowerment in dynamical control systems (Sharma et al.

Original: ; Achiam et al.
Revised: ; Achiam et al.

Original: (2018)) have high sample complexity, are often unstable in training, and may be biased.
Revised: (2018)) have high sample complexity, are often unstable in training, and may be biased.

Original: Moreover, it was not previously studied if empowerment estimators learned via VLBs possess the essential properties of empowerment.
Revised: Moreover, it was not previously studied if empowerment estimators learned via VLBs possess the essential properties of empowerment.

Original: In this work, we introduce a new method, Latent Gaussian Channel Empowerment (Latent GCE), for empowerment estimation and utilize the above-mentioned property as an “indicator” for the quality of the estimation.
Revised: In this work, we introduce a new method, Latent Gaussian Channel Empowerment (Latent-GCE), for empowerment estimation and utilize the above-mentioned property as an “indicator” for the quality of the estimation.

Original: Speciﬁcally, we propose a particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories.
Revised: Speciﬁcally, we propose a particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories.

Original: This representation admits an efﬁcient estimation of empowerment by convex optimization (Cover & Thomas (2012)), both from raw state and from images.
Revised: This representation admits an efﬁcient estimation of empowerment by convex optimization (Cover & Thomas (2012)), both from raw state and from images.

Original: We propose an algorithm for simultaneous estimation and maximization of empowerment using standard RL algorithms such as Proximal Policy Optimization(PPO) (Schulman et al.
Revised: We propose an algorithm for simultaneous estimation and maximization of empowerment using standard RL algorithms such as Proximal Policy Optimization (Schulman et al.

Original: and Soft Actor-Critic(SAC) (Haarnoja et al.
Revised: and Soft Actor-Critic (Haarnoja et al.

Original: We test our method on the task of unsupervised stabilization of dynamical systems with solely intrinsic reward, showing our estimator exhibits essential properties of the empowerment function.
Revised: We test our method on the task of unsupervised stabilization of dynamical systems with solely intrinsic reward, showing our estimator exhibits essential properties of the empowerment function.

Original: We demonstrate the advantages of our method through comparisons to the existing state-of-the-art empowerment estimators in different dynamical systems from the OpenAI Gym simulator (Brockman et al.
Revised: We demonstrate the advantages of our method through comparisons to the existing state-of-the-art empowerment estimators in different dynamical systems from the OpenAI Gym simulator (Brockman et al.

Original: We ﬁnd that our method (i) has a lower sample complexity, (ii) is more stable in training, (iii) possesses the essential properties of the empowerment function, and (iv) allows us to accurately estimate empowerment from images.
Revised: We ﬁnd that our method (i) has a lower sample complexity, (ii) is more stable in training, (iii) possesses the essential properties of the empowerment function, and (iv) allows us to accurately estimate empowerment from images.

Original: In addition, we provide a novel benchmark between different estimators of empowerment, speciﬁcally comparing their essential properties.

Original: We hope such a review of the existing methods for empowerment estimation will help push this research direction.
Revised: We hope such a review of the existing methods for empowerment estimation will help push this research direction.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we review the necessary background for our method, consisting of the reinforcement learning setting, various empowerment estimators, and the Gaussian channel capacity.
Revised: In this section, we review the necessary background for our method, consisting of the reinforcement learning setting, various empowerment estimators, and the Gaussian channel capacity.

Original: We also review the underlying components in relevant prior work which we use for comparison to our method.
Revised: We also review the underlying components in relevant prior work which we use for comparison to our method.

Original: 2. R EINFORCEMENT L EARNING
Revised: 2. R EINFORCEMENT L EARNING

Original: The reinforcement learning (RL) setting is modeled as an inﬁnite-horizon Markov Decision Process (MDP) deﬁned by: the state space S , the action space A , the transition probabilities p ( s (cid:48) | s, a ) , the initial state distribution p 0 ( s ) , the reward function r ( s, a ) ∈ R , and the discount factor γ .
Revised: The reinforcement learning (RL) setting is modeled as an inﬁnite-horizon Markov Decision Process (MDP) deﬁned by: the state space S , the action space A , the transition probabilities p ( s (cid:48) | s, a ) , the initial state distribution p 0 ( s ) , the reward function r ( s, a ) ∈ R , and the discount factor γ .

Original: The goal of RL is to ﬁnd an optimal control policy π ( a | s ) that maximizes the expected return, i.e.[-max π E s 0 ∼ p 0 ,a t ∼ π,s t +1 ∼ p (cid:2)(cid:80) ∞ t =
-]
Revised: The goal of RL is to ﬁnd an optimal control policy π ( a | s ) that maximizes the expected return, i.e.[-max π E s 0 ∼ p 0 ,a t ∼ π,s t +1 ∼ p (cid:2)(cid:80) ∞ t =
-]

Original: γ t r ( s t , a t ) (cid:3) (Sutton & Barto (2018)).
Revised: [Equation] (Sutton & Barto (2018)) [Equation]

Original: 2. E MPOWERMENT
Revised: 2. E MPOWERMENT

Original: Interaction between an agent and its environment is plausibly described by the perception-action cycle (PAC), (Tishby & Polani (2011)), where the agent observes the state of the environment via its sensors and responds with its actuators.
Revised: Interaction between an agent and its environment is plausibly described by the perception-action cycle (PAC), (Tishby & Polani (2011)), where the agent observes the state of the environment via its sensors and responds with its actuators.

Original: The maximal information rate from actuators to sensors is an inherent property of PAC which characterizes the empowerment of the agent, formally deﬁned below.
Revised: The maximal information rate from actuators to sensors is an inherent property of PAC which characterizes the empowerment of the agent, formally deﬁned below.

Original: Empowerment (Klyubin et al.
Revised: Empowerment (Klyubin et al.

Original: (2005)) is deﬁned by the maximal mutual information rate (Cover &
Revised: (2005)) is deﬁned by the maximal mutual information rate (Cover &

Original: Thomas (2012)) between the agent’s sensor observations o ∈ O and actuators a ∈ A given the current state s ∈ S .
Revised: Thomas (2012)) between the agent’s sensor observations o ∈ O and actuators a ∈ A given the current state s ∈ S .

Original: It is fully speciﬁed by a ﬁxed probability distribution of sensor observations conditioned on the actuators and the state, p ( O | A, s ) , denoted by “channel”, and by a free probability of the actuator signals conditioned on the state, ω
Revised: It is fully speciﬁed by a ﬁxed probability distribution of sensor observations conditioned on the actuators and the state, p ( O | A, s ) , denoted by “channel”, and by a free probability of the actuator signals conditioned on the state, ω

Original: ( A | s ) , denoted by “source”.
Revised: ( A | s ) , denoted by “source”.

Original: The empowerment E is the capacity of the channel, which is found via optimization of the source distribution ω ( A | s ) :
Revised: The empowerment E is the capacity of the channel, which is found via optimization of the source distribution ω ( A | s ) :

Original: [Equation]
Revised: [Equation]

Original: where I [ O ; A | s ] is the mutual information, reﬂecting the difference between the entropy of the sensor inputs, H ( O | s ) and the corresponding conditional entropy, H ( O | A, s ) 2 .
Revised: where I [ O ; A | s ] is the mutual information, reﬂecting the difference between the entropy of the sensor inputs, H ( O | s ) and the corresponding conditional entropy, H ( O | A, s ) 3 .

Original: In this work, we follow the original empowerment formulation (Klyubin et al.
Revised: In this work, we follow the original empowerment formulation (Klyubin et al.

Original: (2005)), where A is an action sequence of length T starting from the state s and O is the resulting observation sequence.
Revised: (2005)), where A is an action sequence of length T starting from the state s and O is the resulting observation sequence.

Original: However, our method is applicable to other choices of observation and control sequences.
Revised: However, our method is applicable to other choices of observation and control sequences.

Original: Remark 1 The source distribution in Equation 1, ω ∗ ( A | s ) , is used only for the estimation of empowerment.
Revised: Remark 1 The source distribution in Equation 1, ω ∗ ( A | s ) , is used only for the estimation of empowerment.

Original: Another policy, π ( a | s ) , is calculated using empowerment as an intrinsic reward by either reinforcement learning (Karl et al.
Revised: Another policy, π ( a | s ) , is calculated using empowerment as an intrinsic reward by either reinforcement learning (Karl et al.

Original: or a 1-step greedy algorithm (Jung et al.
Revised: or a 1-step greedy algorithm (Jung et al.

Original: Salge et al.
Revised: Salge et al.

Original: to reach the states with maximal empowerment.
Revised: to reach the states with maximal empowerment.

Original: Strictly speaking, the source distribution ω maximizes the mutual information, (i.e. estimates the empowerment), while the policy distribution π maximizes expected accumulated empowerment along trajectories.
Revised: Strictly speaking, the source distribution ω maximizes the mutual information, (i.e. estimates the empowerment), while the policy distribution π maximizes expected accumulated empowerment along trajectories.

Original: In general, the capacity of an arbitrary channel for a continuous random variable is unknown except for a few special cases such as the Gaussian linear channel, as reviewed below.
Revised: In general, the capacity of an arbitrary channel for a continuous random variable is unknown except for a few special cases such as the Gaussian linear channel, as reviewed below.

Original: As a result, past works have largely relied on variational lower bounds instead.
Revised: As a result, past works have largely relied on variational lower bounds instead.

Original: 2.2. E MPOWERMENT BY V ARIATIONAL L OWER B OUNDS
Revised: 2.2. E MPOWERMENT BY V ARIATIONAL L OWER B OUNDS

Original: The mutual information in Equation 1 can be bounded from below as follows (Mohamed & Rezende (2015); Gregor et al. 
Revised: The mutual information in Equation 1 can be bounded from below as follows (Mohamed & Rezende [Equation] ; Gregor et al. [Equation]

Original: ; Karl et al.
Revised: Karl et al.

Revised: [Equation]

Original: ; Poole et al.
Revised: Poole et al.

Original: [Equation]
Revised: [Equation]

Original: where q ( A | O, s ) and ω ( A | s ) are represented by neural networks with parameters φ and ψ respectively.
Revised: where q ( A | O, s ) and ω ( A | s ) are represented by neural networks with parameters φ and ψ respectively.

Original: The lower bound in Equation 2 can be estimated from samples using reinforcement learning with intrinsic reward r ( s t , a t ) =
Revised: The lower bound in Equation 2 can be estimated from samples using reinforcement learning with intrinsic reward r ( s t , a t ) =

Original: (cid:80)
Revised: (cid:80)

Original: Tt log q φ ( a t | o t , s ) − log ω ψ ( a t | s t ) , as detailed in (Gregor et al.
Revised: Tt log q φ ( a t | o t , s ) − log ω ψ ( a t | s t ) , as detailed in (Gregor et al.

Original: ; Karl et al.
Revised: ; Karl et al.

Original: To apply this variational lower bound on empowerment to unsupervised stabilization of dynamical systems one needs to learn three distributions: q φ ( a t | o t , s ) and ω ψ ( a t | s t ) for the lower bound in Equation 2, and π ( a | s ) as the control policy (see Remark 1).
Revised: To apply this variational lower bound on empowerment to unsupervised stabilization of dynamical systems one needs to learn three distributions: q φ ( a t | o t , s ) and ω ψ ( a t | s t ) for the lower bound in Equation 2, and π ( a | s ) as the control policy (see Remark 1).

Original: Developing the mutual information in Equation 1 in the opposite direction, one gets another lower bound on empowerment, an approximation of which was used in (Sharma et al.
Revised: Developing the mutual information in Equation 1 in the opposite direction, one gets another lower bound on empowerment, an approximation of which was used in (Sharma et al.

Original: [Equation]
Revised: [Equation]

Original: For sake of the clarity of the exposition, the above-explained variational empowerment is schematically depicted in Figure 1.
Revised: For sake of the clarity of the exposition, the above-explained variational empowerment is schematically depicted in Figure 1.

Original: The controller policy, π θ ( a | s ) , collects transitions in environment, ( s, a, s (cid:48) ) .
Revised: The controller policy, π θ ( a | s ) , collects transitions in environment, ( s, a, s (cid:48) ) .

Original: The source policy ω ψ ( a | s ) generates the action a at every state s visited by the controller policy, and the environment transits to the next state, s (cid:48)
Revised: The source policy ω ψ ( a | s ) generates the action a at every state s visited by the controller policy, and the environment transits to the next state, s (cid:48)

Original: (yellow block).
Revised: (yellow block).

Original: The inverse channel policy q ψ ( a | s, a (cid:48) ) is regressed to a given s and and s (cid:48) , generating the intrinsic reward,
Revised: The inverse channel policy q ψ ( a | s, a (cid:48) ) is regressed to a given s and and s (cid:48) , generating the intrinsic reward,

Original: C = log ω
Revised: C = log ω

Original: φ ( a | s ) − log q ψ ( a | s, a (cid:48) ) , which is used as the reward to improve the controller policy via reinforcement learning.
Revised: φ ( a | s ) − log q ψ ( a | s, a (cid:48) ) , which is used as the reward to improve the controller policy via reinforcement learning.

Original: This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment.
Revised: This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment.

Original: 2.2. E MPOWERMENT FOR G AUSSIAN C HANNELS
Revised: 2.2. E MPOWERMENT FOR G AUSSIAN C HANNELS

Original: The Gaussian linear channel in our case is given by:
Revised: The Gaussian linear channel in our case is given by:

Original: [Equation]
Revised: [Equation]

Original: where O and A are the observation and actuator sequences with dimensions
Revised: where O and A are the observation and actuator sequences with dimensions

Original: d O × 1 and d A × 1 , respectively; G ( s ) is a matrix with dimensions d O × d A ; and η is multivariate Gaussian noise with dimension
Revised: d O × 1 and d A × 1 , respectively; G ( s ) is a matrix with dimensions d O × d A ; and η is multivariate Gaussian noise with dimension

Original: d
Revised: d

Original: O × 1 .
Revised: O × 1 .

Original: Following Salge et al.
Revised: Following Salge et al.

Original: (2013), we assume the noise is distributed as a Gaussian.
Revised: (2013), we assume the noise is distributed as a Gaussian.

Original: For completeness, Appendix A provides the analytical derivation of G ( s ) in the case of known dynamics for pendulum for the action sequence A with length 3.
Revised: For completeness, Appendix A provides the analytical derivation of G ( s ) in the case of known dynamics for pendulum for the action sequence A with length 3.

Original: In contrast, the current work proposes a method how to estimate G ( s ) for arbitrary dynamics and arbitrary length of the action sequence from samples.
Revised: In contrast, the current work proposes a method how to estimate G ( s ) for arbitrary dynamics and arbitrary length of the action sequence from samples.

Original: The details of the calculation of Gaussian channel capacity are included in Appendix B.
Revised: The details of the calculation of Gaussian channel capacity are included in Appendix B.

Original: The key observation in our method is that we can learn G ( s ) from samples, which allows us to estimate empowerment efﬁciently for arbitrary dynamics, including for high-dimensional dynamical systems or from images (pixels).
Revised: The key observation in our method is that we can learn G ( s ) from samples, which allows us to estimate empowerment efﬁciently for arbitrary dynamics, including for high-dimensional dynamical systems or from images (pixels).

Original: As a result, it is not required to resort to lower bounds as done in prior work.
Revised: As a result, it is not required to resort to lower bounds as done in prior work.

Original: In Section 4, we describe how to learn G ( s ) from samples using deep neural networks.
Revised: In Section 4, we describe how to learn G ( s ) from samples using deep neural networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we review the relevant prior work and state of the art in empowerment estimation, emphasizing the main differences between these existing works and our proposed approach.
Revised: In this section, we review the relevant prior work and state of the art in empowerment estimation, emphasizing the main differences between these existing works and our proposed approach.

Original: Estimation via Gaussian channel.
Revised: Estimation via Gaussian channel.

Original: The representation given by Equation 4 was previously considered in (Salge et al.
Revised: The representation given by Equation 4 was previously considered in (Salge et al.

Original: (2013)), where the channel p ( O | A, s ) was assumed to be known.
Revised: (2013)), where the channel p ( O | A, s ) was assumed to be known.

Original: In our current work, we remove this assumption and extend the applicability of the efﬁcient channel capacity solution given by Equation 11 to arbitrary observations, including images.
Revised: In our current work, we remove this assumption and extend the applicability of the efﬁcient channel capacity solution given by Equation 11 to arbitrary observations, including images.

Original: Variational approaches to estimation.
Revised: Variational approaches to estimation.

Original: The lower bound in Equation 2 is used in (Mohamed &
Revised: The lower bound in Equation 2 is used in (Mohamed &

Original: Rezende (2015); Gregor et al.
Revised: Rezende (2015); Gregor et al.

Original: ; Karl et al.
Revised: ; Karl et al.

Original: (2019)), where different additional assumptions are made.
Revised: (2019)), where different additional assumptions are made.

Original: Speciﬁcally, Mohamed & Rezende (2015) estimates empowerment in open loop, while Gregor et al.
Revised: Speciﬁcally, Mohamed & Rezende (2015) estimates empowerment in open loop, while Gregor et al.

Original: ; Karl et al.
Revised: ; Karl et al.

Original: (2019) provide a solution for closed-loop empowerment, and Karl et al.
Revised: (2019) provide a solution for closed-loop empowerment, and Karl et al.

Original: (2019) assumes differentiability of the dynamics.
Revised: (2019) assumes differentiability of the dynamics.

Original: In our work, we do not assume differentiable dynamics and we do not resort to variational bounds, aiming to achieve a more stable, sample-efﬁcient, and generalizable estimator of empowerment without losing the essential properties of the empowerment function, as explained in the Introduction.
Revised: In our work, we do not assume differentiable dynamics and we do not resort to variational bounds, aiming to achieve a more stable, sample-efﬁcient, and generalizable estimator of empowerment without losing the essential properties of the empowerment function, as explained in the Introduction.

Original: The works by (Mohamed & Rezende (2015); Gregor et al.
Revised: The works by (Mohamed & Rezende (2015); Gregor et al.

Original: (2017)) precede that by (Karl et al.
Revised: (2017)) precede that by (Karl et al.

Original: , and we also experimentally ﬁnd the latter to be more stable.
Revised: , and we also experimentally ﬁnd the latter to be more stable.

Original: Consequently, in Section 5 we provide comparisons to Karl et al.
Revised: Consequently, in Section 5 we provide comparisons to Karl et al.

Original: (2019), which represents the current start of the art in this line of research.
Revised: (2019), which represents the current start of the art in this line of research.

Original: Skill discovery.
Revised: Skill discovery.

Original: An approximation of the lower bound in Equation 3 is successfully used by
Revised: An approximation of the lower bound in Equation 3 is successfully used by

Original: Sharma et al.
Revised: Sharma et al.

Original: (2020) in the context of learning a set of skills.
Revised: (2020) in the context of learning a set of skills.

Original: Another relevant and important work is
Revised: Another relevant and important work is

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: Citing the authors, their work is about “maximizing the empowerment of a hierarchical agent whose action space is the set of skills” .
Revised: Citing the authors, their work is about “maximizing the empowerment of a hierarchical agent whose action space is the set of skills” .

Original: Section 5 provides comparisons between our empowerment estimator and these methods with regards to stability in training, sample efﬁciency, and correctness of the empowerment landscape generated by both methods.
Revised: Section 5 provides comparisons between our empowerment estimator and these methods with regards to stability in training, sample efﬁciency, and correctness of the empowerment landscape generated by both methods.

Original: Although they aim to achieve a different goal from our work – unsupervised skill discovery vs unsupervised stabilization – their objectives are the same type us ours: maximizing mutual information between actuators and sensors.
Revised: Although they aim to achieve a different goal from our work – unsupervised skill discovery vs unsupervised stabilization – their objectives are the same type us ours: maximizing mutual information between actuators and sensors.

Original: In Section 5 we show that the empowerment landscape derived by these methods converge to the true empowerment landscape in simple cases, but fail to converge to a known empowerment landscape in more complicated dynamical systems.
Revised: In Section 5 we show that the empowerment landscape derived by these methods converge to the true empowerment landscape in simple cases, but fail to converge to a known empowerment landscape in more complicated dynamical systems.

Original: These works represent the state of the art in empowerment estimation.
Revised: These works represent the state of the art in empowerment estimation.

Original: We believe a thorough comparison between our method and these works, as provided in Section 5, clearly demonstrate the main properties of our method and clarify our contribution to the improvement of sample-based empowerment estimation in dynamical systems.
Revised: We believe a thorough comparison between our method and these works, as provided in Section 5, clearly demonstrate the main properties of our method and clarify our contribution to the improvement of sample-based empowerment estimation in dynamical systems.

Original: As mentioned in the introduction, empowerment is useful for a broad spectrum of different applications.
Revised: As mentioned in the introduction, empowerment is useful for a broad spectrum of different applications.

Original: In this section, we highlighted the most relevant and updated methods, which we hope are helpful to see the merits and advances of our approach.
Revised: In this section, we highlighted the most relevant and updated methods, which we hope are helpful to see the merits and advances of our approach.

Original: Our improved empowerment estimation possesses the essential properties of the empowerment function: in particular, it converges to the correct empowerment landscape.
Revised: Our improved empowerment estimation possesses the essential properties of the empowerment function: in particular, it converges to the correct empowerment landscape.

Original: The training has improved stability.
Revised: The training has improved stability.

Original: Furthermore, it allows for the estimation of the empowerment function from images, which we demonstrate in Section 5, where we show purely intrinsically motivated (empowerment-based) stabilization of inverted pendulum from images.
Revised: Furthermore, it allows for the estimation of the empowerment function from images, which we demonstrate in Section 5, where we show purely intrinsically motivated (empowerment-based) stabilization of inverted pendulum from images.

Original: Accurate estimation of empowerment from images signiﬁcantly expands the possible breadth of applications for empowerment.
Revised: Accurate estimation of empowerment from images signiﬁcantly expands the possible breadth of applications for empowerment.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: The desiderata for our method as follows:
Revised: The desiderata for our method as follows:

Original: • to learn the channel matrix ( G ( s ) in Equation 4) from samples for arbitrary dynamics.
Revised: • to learn the channel matrix ( G ( s ) in Equation 4 from samples for arbitrary dynamics.

Original: • to utilize general RL optimizers to reliably train intrinsically motivated policies.
Revised: • to utilize general RL optimizers to reliably train intrinsically motivated policies.

Original: • to enable empowerment maximization from arbitrary dynamics using latent representations.
Revised: • to enable empowerment maximization from arbitrary dynamics using latent representations.

Original: These are desirable as: the ability to learn G ( s ) from samples would allow us to estimate empowerment by the efﬁcient line search algorithm in Equation 12, recent advances in RL algorithms can alleviate challenges in stability and sample efﬁciency, and latent representations would allow our method to be applicable even when the observation or underlying dynamics do not satisfy the
Revised: These are desirable as the ability to learn G ( s ) from samples would allow us to estimate empowerment by the efﬁcient line search algorithm in Equation 12, recent advances in RL algorithms can alleviate challenges in stability and sample efﬁciency, and latent representations would allow our method to be applicable even when the observation or underlying dynamics do not satisfy the

Original: Gaussian Linear Channel assumption.
Revised: Gaussian Linear Channel assumption.

Original: In this work, we present Latent Gaussian Channel Empowerment, or Latent-GCE in short.
Revised: In this work, we present Latent Gaussian Channel Empowerment, or Latent-GCE in short.

Original: Schematically, our method is depicted in Figure 2.
Revised: Schematically, our method is depicted in Figure 2.

Original: The controller policy π θ ( a | s ) collects trajectories in the environment, ( s, a, s (cid:48) ) , where s is either a state or image observation.
Revised: The controller policy π θ ( a | s ) collects trajectories in the environment, ( s, a, s (cid:48) ) , where s is either a state or image observation.

Original: The channel matrix G χ ( s t ) predicts the ﬁnal state of a trajectory corresponding to the action sequence a T − 1 t , starting from s t .
Revised: The channel matrix G χ ( s t ) predicts the ﬁnal state of a trajectory corresponding to the action sequence a T − 1 t , starting from s t .

Original: After performing singular value decomposition on G χ ( s t ) , the empowerment value E ( s t ) is estimated by Equation 11 and Equation 12.
Revised: After performing singular value decomposition on G χ ( s t ) , the empowerment value E ( s t ) is estimated by Equation 11 and Equation 12.

Original: The empowerment along trajectories provide intrinsic rewards, denoted by r (cid:48)
Revised: The empowerment along trajectories provide intrinsic rewards, denoted by r (cid:48)

Original: ( τ ) , for the update of the controller policy, π θ , using off-the-shelf RL algorithms.
Revised: ( τ ) , for the update of the controller policy, π θ , using off-the-shelf RL algorithms.

Original: Our method is summarized in high level in Algorithm 1.
Revised: Our method is summarized in high level in Algorithm 1.

Original: 4. E MBEDDING OF O BSERVATIONS AND A CTION S EQUENCES
Revised: 4. E MBEDDING OF O BSERVATIONS AND A CTION S EQUENCES

Original: While the Gaussian Linear Channel deﬁned in Equation 4 provides a good estimate of simple dynamic systems, we want our empowerment estimator to generalize to arbitrary dynamics, including highdimensional systems, and environments where the observations are images.
Revised: While the Gaussian Linear Channel deﬁned in Equation 4 provides a good estimate of simple dynamic systems, we want our empowerment estimator to generalize to arbitrary dynamics, including highdimensional systems, and environments where the observations are images.

Original: Using autoencoders, we embed the observations and action sequences such that their latent vector representations satisfy the
Revised: Using autoencoders, we embed the observations and action sequences such that their latent vector representations satisfy the

Original: Gaussian Linear Channel property.
Revised: Gaussian Linear Channel property.

Original: We use CNN encoder and decoder for image observations, or fully-connected networks for state observations.
Revised: We use CNN encoder and decoder for image observations, or fully-connected networks for state observations.

Original: The encoder-decoder pair for action sequences are also parameterized by fully-connected networks.
Revised: The encoder-decoder pair for action sequences are also parameterized by fully-connected networks.

Original: Let f χ , f − 1 χ , g χ , g − 1 χ denote the encoder-decoder pairs for the observations and action sequences, f χ maps the observations into d f dimensional latentdo 3: { τ k Kk
Revised: Let f χ , f − 1 χ , g χ , g − 1 χ denote the encoder-decoder pairs for the observations and action sequences, f χ maps the observations into d f dimensional latent

Original: ← run policy π } 4: extract tuples (s a H t , s  H from {
Revised: 1: Input: π θ 0 ( a | s ) - control policy, H - prediction horizon.

Original: Kk =  {  for Gaussian } 5: for j = 1 to M do 6: χ ∗ ← arg min χ (cid:80) tuples (cid:13)(cid:13)
Revised: 2: for i = 0 to N do 3: { τ k } Kk =1 ← run policy π θ i ( a | s ) {data collection in environment} 4: extract tuples ( s t , a H − 1 t , s t + H ) from { τ k } K k = {data arrangement for Gaussian channel} 5: for j = 1 to M do 6: χ ∗ ← arg min χ (cid:80) tuples (cid:13)(cid:13)

Original: G χ ( s t ) · a Ht − s t + H (cid:13)(cid:13) 2{projection to Equation 4} 7: end for 8: { σ ∗ i ( s t ) }
Revised: G χ ( s t ) · a Ht − s t + H (cid:13)(cid:13) 2{projection to Equation 4} 7: end for 8: { σ ∗ i ( s t ) }

Original: ki =1 ← SVD (  )) { an alternative explained in Appendix C} 9:
Revised: ki =1 ← SVD ( G χ ∗ ( s t )) {an alternative explained in Appendix C} 9:

Original: E ( s t ) ← 12 (cid:80) ki
Revised: E ( s t ) ← 12 (cid:80) ki

Original: =1 log(1 + σ ∗ i ( s ) p ∗ i ) {Equation 11 and Equation 12} 10: reinforce π
Revised: =1 log(1 + σ ∗ i ( s ) p ∗ i ) {Equation 11 and Equation 12} 10: reinforce π

Original: θ
Revised: θ

Original: i ( a | s ) with r (cid:48) ( τ ) =
Revised: i ( a | s ) with r (cid:48) ( τ ) =

Original: (cid:80) Tt =
Revised: (cid:80) Tt =

Original: 1 γ t − 1 E ( s t ) .
Revised: 1 γ t − 1 E ( s t ) .

Original: {policy update} 11: end for states, and g χ maps the H -step action sequences into d g dimensional latent actions.
Revised: {policy update} 11: end for states, and g χ maps the H -step action sequences into d g dimensional latent actions.

Original: Our Latent
Revised: Our Latent

Original: Gaussian Linear Channel is formulated as follows:
Revised: Gaussian Linear Channel is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: 4. L EARNING C HANNEL M ATRIX G
Revised: 4. L EARNING C HANNEL M ATRIX G

Original: The channel matrix G χ ( f χ ( s t )) has dimensions d f × d g , where d f and d g are the dimensions of latent states and actions as deﬁned in section 4.1.
Revised: The channel matrix G χ ( f χ ( s t )) has dimensions d f × d g , where d f and d g are the dimensions of latent states and actions as deﬁned in section 4.1.

Original: In the Latent Gaussian Linear Channel, as formulated in Equation 5, the ending latent state is approximated by a matrix-vector multiplication between the channel matrix G χ ( f χ ( s t )) and the latent action g χ ( a Ht ) .
Revised: In the Latent Gaussian Linear Channel, as formulated in Equation 5, the ending latent state is approximated by a matrix-vector multiplication between the channel matrix G χ ( f χ ( s t )) and the latent action g χ ( a Ht ) .

Original: The mapping from a latent state to its channel matrix is parameterized by a 3-layer fully-connected neural network.
Revised: The mapping from a latent state to its channel matrix is parameterized by a 3-layer fully-connected neural network.

Original: The output layer of the network contains d f · d g neurons, which is then reshaped into the d f × d g matrix.
Revised: The output layer of the network contains d f · d g neurons, which is then reshaped into the d f × d g matrix.

Original: 4. O VERALL O PTIMIZATION O BJECTIVES
Revised: 4. O VERALL O PTIMIZATION O BJECTIVES

Original: The overall training objective for our Latent-GCE estimation consists of two parts: reconstruction loss and latent prediction loss.
Revised: The overall training objective for our Latent-GCE estimation consists of two parts: reconstruction loss and latent prediction loss.

Original: The reconstruction loss L R is the average L2 distance between the decoded latent representations and the original encoder inputs.
Revised: The reconstruction loss L R is the average L2 distance between the decoded latent representations and the original encoder inputs.

Original: Minimizing L R encourages the autoencoders to learn meaningful latent representations.
Revised: Minimizing L R encourages the autoencoders to learn meaningful latent representations.

Original: The prediction loss L P is the average L2 distance between the matrix-vector product and the true latent ending state.
Revised: The prediction loss L P is the average L2 distance between the matrix-vector product and the true latent ending state.

Original: Minimizing L P forces the latent space to maintain the Gaussian Linear Channel structure:
Revised: Minimizing L P forces the latent space to maintain the Gaussian Linear Channel structure:

Original: [Equation]
Revised: [Equation]

Original: Overall, our Latent-GCE Maximization scheme (Figure 2) trains two sets of parameters, ( θ , χ ), while
Revised: Overall, our Latent-GCE Maximization scheme (Figure 2) trains two sets of parameters, ( θ , χ ), while

Original: VLB methods (Figure 1) generally needs three sets, ( θ , φ , ψ ).
Revised: VLB methods (Figure 1) generally needs three sets, ( θ , φ , ψ ).

Original: The reduced trainable components suggest that our method might be superior in terms of better stability and quicker convergence, which we conﬁrm in our experiment section.
Revised: The reduced trainable components suggest that our method might be superior in terms of better stability and quicker convergence, which we conﬁrm in our experiment section.

Original:  E XPERIMENTS 5. U NSUPERVISED S TABILIZATION OF C LASSIC D YNAMIC S YSTEMS
Revised:  E XPERIMENTS 5. U NSUPERVISED S TABILIZATION OF C LASSIC D YNAMIC S YSTEMS

Original: As discussed in Section 1, a control policy that maximizes agent empowerment should lead to agent stabilization, even at an unstable equilibrium.
Revised: As discussed in Section 1, a control policy that maximizes agent empowerment should lead to agent stabilization, even at an unstable equilibrium.

Original: To evaluate this, we compare our method with two prior works by Mohamed & Rezende (2015) and Karl et al.
Revised: To evaluate this, we compare our method with two prior works by Mohamed & Rezende (2015) and Karl et al.

Original: on their ability to stabilize simple dynamic systems.
Revised: on their ability to stabilize simple dynamic systems.

Original: Speciﬁcally, in the 2D ball-in-box environment, empowerment is highest at the center of the box, where the ball is free to move in all directions; in the pendulum environment, empowerment is highest at the up-right position, where gravity assists in reaching other states.
Revised: Speciﬁcally, in the 2D ball-in-box environment, empowerment is highest at the center of the box, where the ball is free to move in all directions; in the pendulum environment, empowerment is highest at the up-right position, where gravity assists in reaching other states.

Original: We compare average squared distance away from the stable position during policy training.
Revised: We compare average squared distance away from the stable position during policy training.

Original: As shown in Figure 3, all three methods are successful on the simple ball-in-box environment, whereas only our method consistently stabilizes the pendulum at an unstable equilibrium.
Revised: As shown in Figure 3, all three methods are successful on the simple ball-in-box environment, whereas only our method consistently stabilizes the pendulum at an unstable equilibrium.

Original: We refer readers to
Revised: We refer readers to

Original: Appendix D for detailed visualization of pendulum empowerment/policy learning using our method, and Appendix E for results on cart-pole.
Revised: Appendix D for detailed visualization of pendulum empowerment/policy learning using our method, and Appendix E for results on cart-pole.

Original: 5. Q UALITATIVE C ONVERGENCE OF D IFFERENT E STIMATORS
Revised: 5. Q UALITATIVE C ONVERGENCE OF D IFFERENT E STIMATORS

Original: We now qualitatively compare the convergence of different empowerment estimators to the true empowerment function.
Revised: We now qualitatively compare the convergence of different empowerment estimators to the true empowerment function.

Original: Not only do we include solutions by VLB methods that approximate the classic empowerment deﬁnition (Mohamed & Rezende (2015); Karl et al.
Revised: Not only do we include solutions by VLB methods that approximate the classic empowerment deﬁnition (Mohamed & Rezende (2015); Karl et al.

Original: , but we also compare with recent works that employ empowerment-like objectives for skill discovery (Eysenbach et al.
Revised: , but we also compare with recent works that employ empowerment-like objectives for skill discovery (Eysenbach et al.

Original: ; Sharma et al.
Revised: ; Sharma et al.

Original: From Figure 4, we observe that in simple environments such as ball-in-box, all estimators converge to the correct landscape with high empowerment values in the middle and low values near the boundaries, corresponding to previous work (Klyubin et al.
Revised: From Figure 4, we observe that in simple environments such as ball-in-box, all estimators converge to the correct landscape with high empowerment values in the middle and low values near the boundaries, corresponding to previous work (Klyubin et al.

Original: For the pendulum environment, it is clear that while both the method by Karl et al.
Revised: For the pendulum environment, it is clear that while both the method by Karl et al.

Original: and ours succeed in creating results similar to the analytical solution (the ground truth), ours outputs an empowerment landscape that is smoother and more discerning.
Revised: and ours succeed in creating results similar to the analytical solution (the ground truth), ours outputs an empowerment landscape that is smoother and more discerning.

Original: Detailed derivation of the analytical solution is included in Appendix A.
Revised: Detailed derivation of the analytical solution is included in Appendix A.

Original: 5. S TABILITY OF E MPOWERMENT E STIMATIONS
Revised: 5. S TABILITY OF E MPOWERMENT E STIMATIONS

Original: Besides the convergence to correct empowerment values, the stability and sample efﬁciency are also key aspects to determine whether the empowerment estimators are suitable as reliable intrinsic reward signals.
Revised: Besides the convergence to correct empowerment values, the stability and sample efﬁciency are also key aspects to determine whether the empowerment estimators are suitable as reliable intrinsic reward signals.

Original: Speciﬁcally, we quantify the variance of average empowerment values overseeds.
Revised: We compute the variance of average empowerment values over 10 seeds as a measurement of algorithm stability.

Original: Since the outputs by existing methods are on different scales, we instead compute the relative standard deviation.
Revised: Since the outputs by existing methods are on different scales, we instead compare the relative standard deviation.

Original: We show the learning curves in Figure 5. Our method outperforms the existing sample-based methods in that it has lower variance in training and faster convergence to the ﬁnal estimator.
Revised: As shown in Figure 5, our method outperforms the existing sample-based methods in that it has lower variance in training and faster convergence to the ﬁnal estimator.

Original: These properties make our method competent as a reliable unsupervised training signal.
Revised: These properties make our method competent as a reliable unsupervised training signal.

Original: [Figure]
Revised: [Figure]

Original: 5. E MPOWERMENT E STIMATION IN H IGH - DIMENSIONAL S YSTEMS
Revised: 5. E MPOWERMENT E STIMATION IN H IGH - DIMENSIONAL S YSTEMS

Original: In this section, we verify the applicability of Latent-GCE in high-dimensional systems from two aspects: ﬁrst, thelearned latent space should be invertible - able to predict the true underlying dynamics; moreover, the empowerment maximization policy should develop stabilization behaviors.
Revised: We verify the applicability of Latent-GCE in high-dimensional systems from two aspects: ﬁrst, the latent space should be invertible - able to predict the true underlying dynamics; moreover, the empowerment maximization policy should develop stabilizing behaviors.

Original: 5.4. P IXEL - BASED P ENDULUM
Revised: 5.4. P IXEL - BASED P ENDULUM

Original: We use a pendulum environment that supplies 64 × 64 images as observations.
Revised: We use a pendulum environment that supplies 64 × 64 images as observations.

Original: As input to our
Revised: As input to our

Original: Latent-GCE algorithm, we concatenate two consecutive frames so as to capture the angular velocity.
Revised: Latent-GCE algorithm, we concatenate two consecutive frames so as to capture the angular velocity.

Original: In Figure 6, we start the pendulum at its up-right position s 0 .
Revised: In Figure 6, we start the pendulum at its up-right position s 0 .

Original: Using the encoder f χ , we map the images into the latent representation f χ ( s 0 ) .
Revised: Using the encoder f χ , we map the images into the latent representation f χ ( s 0 ) .

Original: Given different action sequences a H 0 , we predict the ending latent state as z t + H
Revised: Given different action sequences a H 0 , we predict the ending latent state as z t + H

Original: =
Revised: =

Original: G χ ( f χ ( s t )) · g χ ( a Ht ) .
Revised: G χ ( f χ ( s t )) · g χ ( a Ht ) .

Original: If our learned Latent-GCE model captures the true system dynamics, we expect the reconstruction of the ending state f − 1 χ
Revised: If our learned Latent-GCE model captures the true system dynamics, we expect the reconstruction of the ending state f − 1 χ

Original: ( z t + H ) to match the ground truth s t + H .
Revised: ( z t + H ) to match the ground truth s t + H .

Original:  Figure 6 veriﬁes that the encoders and latent space learned by our Latent-GCE algorithm indeed captures the underlying dynamics of the pendulum.
Revised: This property holds true as veriﬁed in Figure 6.

Original: More importantly, as shown in Figure 7, the empowerment estimations given by our method is not only close to the analytical solution, but also trains a policy that swing the pendulum up from bottom to balance at the top.
Revised: More importantly, as shown in Figure 7, the empowerment estimations given by our method is not only close to the analytical solution, but also trains a policy that swing the pendulum up from bottom to balance at the top.

Original: On the other hand, VIM by Mohamed & Rezende (2015) which we compare as a baseline, fails to assign a high empowerment estimation for the up-right position.
Revised: On the other hand, VIM by Mohamed & Rezende (2015) which we compare as a baseline, fails to assign a high empowerment estimation for the up-right position.

Original: 5.4. H OPPER
Revised: 5.4. H OPPER

Original: We evaluate the ability of Latent-GCE to stabilize theMuJoCo Hopper environment from OpenAI Gym.
Revised: We evaluate the ability of Latent-GCE to stabilize the Hopper environment from OpenAI Gym.

Original: This environment is more challenging because of higher state dimension and long horizon (64) hopper method
Revised: This environment is more challenging because of higher state dimension and long horizon (64) for empowerment estimation.

Revised: Here we expect an empowerment maximization policy to balance the hopper at its standing position where it is able to move left and right freely.

Revised: In Figure 8 we see our method successfully balances hopper whereas a random policy inevitably makes the hopper fall over.

Original: Our method stabilizes the hopper to a standing position ( z ≈ 1 ).
Revised: Our method stabilizes the hopper to a standing position ( z ≈ 1 ).

Original:  S UMMARY AND D ISCUSSION
Revised:  S UMMARY AND D ISCUSSION

Original: In this work we introduce a new sample-based method for empowerment estimation in dynamical systems.
Revised: In this work we introduce a new sample-based method for empowerment estimation in dynamical systems.

Original: The key component of our method is the representation of dynamical system by the Gaussian channel, which allows  to estimate empowerment by convex optimization.
Revised: The key component of our method is the representation of dynamical system by the Gaussian channel, which allows us to estimate empowerment by convex optimization.

Original: Our estimator converges to true empowerment landscape both from raw states and images.
Revised: Our estimator converges to true empowerment landscape both from raw states and images.

Original: Empowerment is used in various applications and our improved estimator has a potential to improve these applications.
Revised: Empowerment is used in various applications and our improved estimator has a potential to improve these applications.

Original: For example in Appendix F, we show that when the agent is trained with an empowerment-aware objective, it develops distinguishable levels of attention to safety.
Revised: For example in Appendix F, we show that when the agent is trained with an empowerment-aware objective, it develops distinguishable levels of attention to safety.

Original: Moreover, a natural continuation of this work is to learn skills using our representation of empowerment by Gaussian channel, rather than by VLB.
Revised: Moreover, a natural continuation of this work is to learn skills using our representation of empowerment by Gaussian channel, rather than by VLB.

Revised: A CKNOWLEDGMENTS

Revised: This research is supported by Berkeley Deep Drive, Intel, and Amazon through providing AWS cloud computing services.

Original: A A NALYTICAL DERIVATION OF G ( s ) FOR INVERTED PENDULUM
Revised: A A NALYTICAL DERIVATION OF G ( s ) FOR INVERTED PENDULUM

Original: The main steps in the derivation of G ( s ) appear below.
Revised: The main steps in the derivation of G ( s ) appear below.

Original: The full derivation appears in Section 4.1 at
Revised: The full derivation appears in Section 4.1 at

Original: Salge et al.
Revised: Salge et al.

Original: The current state of the pendulum, s t , is given by θ t and ˙ θ t :
Revised: The current state of the pendulum, s t , is given by θ t and ˙ θ t :

Original: [Equation]
Revised: [Equation]

Original: where θ is measured from the upright position.
Revised: where θ is measured from the upright position.

Original: We consider the development of the pendulum for three time steps of duration dt .
Revised: We consider the development of the pendulum for three time steps of duration dt .

Original: The behaviour of the pendulum can be approximate linearly at every time step for small enough dt .
Revised: The behaviour of the pendulum can be approximate linearly at every time step for small enough dt .

Original: Then, the next state of the pendulum is given by:
Revised: Then, the next state of the pendulum is given by:

Original: [Equation]
Revised: [Equation]

Original: where a t is the agent’s action, A t = (cid:20) ˙
Revised: where a t is the agent’s action, A t = (cid:20) ˙

Original: θ t gl sin( θ t ) (cid:21) , and B t =
Revised: θ t gl sin( θ t ) (cid:21) , and B t =

Original: (cid:20) 01 (cid:21) .
Revised: (cid:20) 01 (cid:21) .

Original: For the next time step we need to calculate a new matrix, A , because the non-linearity of the system causes A t + dt to depend on the new state.
Revised: For the next time step we need to calculate a new matrix, A , because the non-linearity of the system causes A t + dt to depend on the new state.

Original: Consequently, A t + dt = (cid:20) ˙
Revised: Consequently, A t + dt = (cid:20) ˙

Original: θ t + dt gl sin( θ t + dt ) (cid:21) .
Revised: θ t + dt g l sin( θ t + dt ) (cid:21) .

Original: A similar matrix can be calculated for A t +2 dt .
Revised: A similar matrix can be calculated for A t +2 dt .

Original: An propagation of pendulum dynamics for 3 time steps allows us to calculate s t +3 dt explicitly, which appears as:
Revised: An propagation of pendulum dynamics for 3 time steps allows us to calculate s t +3 dt explicitly, which appears as:

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: where a t + dt is the action at the next step.
Revised: where a t + dt is the action at the next step.

Original: Our goal is to represent s t +3 dt by:
Revised: Our goal is to represent s t +3 dt by:

Original: [Equation]
Revised: [Equation]

Original: which is achieved by linearisation θ t +3 dt and ˙ θ t +3 dt around a t = 0 .
Revised: which is achieved by linearisation θ t +3 dt and ˙ θ t +3 dt around a t = 0 .

Original: A straightforward calculation shows that the matrix G ( s t ) is given by:
Revised: A straightforward calculation shows that the matrix G ( s t ) is given by:

Original: [Equation]
Revised: [Equation]

Original: This calculation can be continued beyond 3-steps, involving more complicated terms in the expansions.
Revised: This calculation can be continued beyond 3-steps, involving more complicated terms in the expansions.

Original: Given G ( s t ) in Equation 10 one can represent s t +3 dt by Equation 9, where the matrix K does not affect the mutual information Cover & Thomas (2012).
Revised: Given G ( s t ) in Equation 10 one can represent s t +3 dt by Equation 9, where the matrix K does not affect the mutual information Cover & Thomas (2012).

Original: Adding to Equation 9 the normal noise, η , as suggested in Salge et al.
Revised: Adding to Equation 9 the normal noise, η , as suggested in Salge et al.

Original: , we get the representation in Equation 4.
Revised: , we get the representation in Equation 4.

Original: In contrast, we propose to learn G ( s ) from samples via interaction with the environment for an arbitrary dynamics and an arbitrary number of steps (c.f. 3 steps in the above derivation).
Revised: In contrast, we propose to learn G ( s ) from samples via interaction with the environment for an arbitrary dynamics and an arbitrary number of steps (c.f. 3 steps in the above derivation).

Original: Using G ( s ) from Equation 10, we can calculate the analytical (ground truth empowerment landscape), which is shown at Figure
Revised: Using G ( s ) from Equation 10, we can calculate the analytical (ground truth empowerment landscape), which is shown at Figure

Original: B G AUSSIAN C HANNEL C APACITY
Revised: B G AUSSIAN C HANNEL C APACITY

Original: The capacity of the channel in Equation 4 is given by, (Cover & Thomas (2012)):
Revised: The capacity of the channel in Equation 4 is given by, (Cover & Thomas (2012)):

Original: [Equation]
Revised: [Equation]

Original: where σ i ( s ) , p i , k , and P , are the i-th singular value of G ( s ) , the corresponding actuator signal power, the number of nonzero singular values, and the total power of the actuator signal, respectively.
Revised: where σ i ( s ) , p i , k , and P , are the i-th singular value of G ( s ) , the corresponding actuator signal power, the number of nonzero singular values, and the total power of the actuator signal, respectively.

Original: The optimal solution to Equation 11 is given by:
Revised: The optimal solution to Equation 11 is given by:

Original: [Equation]
Revised: [Equation]

Original: where ν is found by a line search (Cover & Thomas (2012)) to satisfy the total power constraint.
Revised: where ν is found by a line search (Cover & Thomas (2012)) to satisfy the total power constraint.

Original: C SVD A LTERNATIVE
Revised: C SVD A LTERNATIVE

Original: The singular values, σ i ( s t ) , can be estimated directly without the singular values decomposition by representing the channel matrix as G χ ( s t )
Revised: The singular values, σ i ( s t ) , can be estimated directly without the singular values decomposition by representing the channel matrix as G χ ( s t )

Original: [Equation]
Revised: [Equation]

Original: ∗ V χ ( s t ) with two additional orthonormality constraints:(cid:13)(cid:13) U χ
Revised: V χ ( s t ) with two additional orthonormality constraints:

Original: [Equation]
Revised: [Equation]

Original: s t ) U † χ

Original: [Equation]

Original: s t ) − I { d s × d s } (cid:13)(cid:13) 2 2 and (cid:13) (cid:13) V χ

Original: ( s t ) V † χ ( s t ) − I { d a ( H − 1) ×

Original: d a ( H − 1) } (cid:13) (cid:13) 2 2 .

Original: In our experiments, we did not experience a bottleneck in calculating the singular values decomposition for both raw states and images.
Revised: In our experiments, we did not experience a bottleneck in calculating the singular values decomposition for both raw states and images.

Original: While we don’t implement this in our experiments, this alternative direct calculation can be applied when SVD is undesired.
Revised: While we don’t implement this in our experiments, this alternative direct calculation can be applied when SVD is undesired.

Original: D T WO PHASES IN EMPOWERMENT - BASED STABILIZATION
Revised: D T WO PHASES IN EMPOWERMENT - BASED STABILIZATION

Original: Empowerment of the pendulum is highest when it is stationary at the up-right position.
Revised: Empowerment of the pendulum is highest when it is stationary at the up-right position.

Original: In this section, we show the evolution of the empowerment landscape, the corresponding control policy and the state distribution as we run Algorithm 1 without any external reward.
Revised: In this section, we show the evolution of the empowerment landscape, the corresponding control policy and the state distribution as we run Algorithm 1 without any external reward.

Original: In Figure 10, we observe that Algorithm 1 includes two phases: empowerment learning and policy learning.
Revised: In Figure 10, we observe that Algorithm 1 includes two phases: empowerment learning and policy learning.

Original: Empowerment converges from epoch 0 to 10 and then, policy converges from epoch 10 to 50.
Revised: Empowerment converges from epoch 0 to 10 and then, policy converges from epoch 10 to 50.

Original: E C ART -P OLE S WING - UP
Revised: [Equation] WING - UP

Original: The cart-pole environment is made up of a stiff pole with one end ﬁxed to a pivot on a cart movable along one axis.
Revised: The cart-pole environment is made up of a stiff pole with one end ﬁxed to a pivot on a cart movable along one axis.

Original: The cart can be pushed left or right within a conﬁned region.
Revised: The cart can be pushed left or right within a conﬁned region.

Original: The state space ofepochepoch- epochepoch- epochepoch- cart-pole has 4 components: cart position, cart velocity, pole angle, pole angular velocity.
Revised: The state space of cart-pole has 4 components: cart position, cart velocity, pole angle, pole angular velocity.

Original: We expect high empowerment when the pole is up right, because starting there, gravity is able to help the system reach a broader set of future states.
Revised: We expect high empowerment when the pole is up right, because starting there, gravity is able to help the system reach a broader set of future states.

Original: To measure whether Algorithm 1 successfully brings the cart-pole to up-right position, we plot the average squared angle to the top against the number of training steps.
Revised: To measure whether Algorithm 1 successfully brings the cart-pole to up-right position, we plot the average squared angle to the top against the number of training steps.

Original: We also include 2 different reward functions that directly aim this objective for comparison.
Revised: We also include 2 different reward functions that directly aim this objective for comparison.

Original: As shown in Figure 11, the empowerment reward, without the need of prior knowledge of the environment, is able to achieve similar ﬁnal performance compared to direct reward signals.
Revised: As shown in Figure 11, the empowerment reward, without the need of prior knowledge of the environment, is able to achieve similar ﬁnal performance compared to direct reward signals.

Original: Moreover, it learns this objective at a speed on par with the dense reward. 
Revised: Moreover, it learns this objective at a speed on par with the dense reward. epochepoch-

Revised: epochepoch- epochepoch-

Original: F E MPOWERMENT E STIMATION FOR S AFETY
Revised: F E MPOWERMENT E STIMATION FOR S AFETY

Original: A useful application of empowerment is its implication of safety for the artiﬁcial agent.
Revised: A useful application of empowerment is its implication of safety for the artiﬁcial agent.

Original: A state is intrinsically safe for an agent when the agent has a high diversity of future states, achievable by its actions.
Revised: A state is intrinsically safe for an agent when the agent has a high diversity of future states, achievable by its actions.

Original: This is because in such states, the agent can take effective actions to prevent undesirable futures.
Revised: This is because in such states, the agent can take effective actions to prevent undesirable futures.

Original: In this context, the higher its empowerment value, the safer the agent is.
Revised: In this context, the higher its empowerment value, the safer the agent is.

Original: In this experiment, we ﬁrst check that our calculation of empowerment matches the speciﬁc design of the environment.
Revised: In this experiment, we ﬁrst check that our calculation of empowerment matches the speciﬁc design of the environment.

Original: Additionally, we show that empowerment augmented reward function can affect the agent’s preference between a shorter but more dangerous path and a longer but safer one.
Revised: Additionally, we show that empowerment augmented reward function can affect the agent’s preference between a shorter but more dangerous path and a longer but safer one.

Original: Environment: a double tunnel environment implemented with to the OpenAI Gym API (Brockman et al., 2016).
Revised: Environment: a double tunnel environment implemented with to the OpenAI Gym API (Brockman et al., 2016).

Original: Agent (blue) is modeled as a ball of radius 1 inside a 20 × 20 box.
Revised: Agent (blue) is modeled as a ball of radius 1 inside a 20 × 20 box.

Original: The box is separated by a thick wall (gray) into top and bottom section.
Revised: The box is separated by a thick wall (gray) into top and bottom section.

Original: Two tunnels connect the top The
Revised: Two tunnels connect the top and bottom of the box in middle is narrower but closer to the goal on right

Original: Control:
Revised: Control:

Original: In each time step, the agent can move at most 0.5 unit length in each of x and y direction.
Revised: In each time step, the agent can move at most 0.5 unit length in each of x and y direction.

Original: If an action takes the agent into the walls, it will be shifted back out to the nearest surface.
Revised: If an action takes the agent into the walls, it will be shifted back out to the nearest surface.

Original: Reset criterion: each episode has a maximum length of 200 steps.
Revised: Reset criterion: each episode has a maximum length of 200 steps.

Original: The environment resets when time runs out or when the agent reaches the goal.
Revised: The environment resets when time runs out or when the agent reaches the goal.

Original: Since the tunnel in the middle is narrower, the agent is relatively less safe there.
Revised: Since the tunnel in the middle is narrower, the agent is relatively less safe there.

Original: The effectiveness of the control of the agent is damped in 2 ways:
Revised: The effectiveness of the control of the agent is damped in 2 ways:

Original: In a particular time step, it’s more likely for the agent to bump into the walls
Revised: In a particular time step, it’s more likely for the agent to bump into the walls.

Original: When this happens, the agent is unable to proceed as far as desired
Revised: When this happens, the agent is unable to proceed as far as desired.

Original: The tunnel is 10 units in length.
Revised: The tunnel is 10 units in length.

Original: When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon.
Revised: When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon.

Original: Thus, the agent has fewer possible future states.
Revised: Thus, the agent has fewer possible future states.

Original: We trained the agent with PPO algorithm (Schulman et al., 2017) from OpenAI baselines (Dhariwal et al., 2017) using an empowerment augmented reward function.
Revised: We trained the agent with PPO algorithm (Schulman et al., 2017) from OpenAI baselines (Dhariwal et al., 2017) using an empowerment augmented reward function.

Original: After a parameter search, we used discount factor γ = 0 .
Revised: After a parameter search, we used discount factor γ = 0 .

Original: 95 over a total of 10 6 steps.
Revised: 95 over a total of 10 6 steps.

Original: The reward function that we choose is:
Revised: The reward function that we choose is:

Original: [Equation]
Revised: [Equation]

Original: where β balances the relative weight between the goal conditioned reward and the intrinsic safety reward.
Revised: where β balances the relative weight between the goal conditioned reward and the intrinsic safety reward.

Original: With high β , we expect the agent to learn a more conservative policy.
Revised: With high β , we expect the agent to learn a more conservative policy.

Revised: [Figure]

Original: The results from the tunnel environment again support our proposed scheme.
Revised: The results from the tunnel environment again support our proposed scheme.

Original: First, the empowerment landscape matches our design of the environment.
Revised: First, the empowerment landscape matches our design of the environment.

Original: Second, high quality empowerment reward successfully alters the behavior of the agent.
Revised: Second, high quality empowerment reward successfully alters the behavior of the agent.

Original: [Figure]

Original: G E XPERIMENT D ETAILS G. U SE OF NEURAL ENCODERS
Revised: G E XPERIMENT D ETAILS G. U SE OF NEURAL ENCODERS

Original: In our state-based classic control environments, the encoder/decoder pairs for observations and actions are set to be the identity map.
Revised: In our state-based classic control environments, the encoder/decoder pairs for observations and actions are set to be the identity map.

Original: In this case, the sequences of actions are simply concatenated.
Revised: In this case, the sequences of actions are simply concatenated.

Original: For the Hopper environment and pixel-based pendulum, encoders and decoders are parameterized by neural networks.
Revised: For the Hopper environment and pixel-based pendulum, encoders and decoders are parameterized by neural networks

Original:  G. D ETAILS ON NEURAL NETWORK LAYERS G.2. CNN FOR IMAGE ENCODING
Revised: . G. D ETAILS ON NEURAL NETWORK LAYERS G.2. CNN FOR IMAGE ENCODING

Original: (h1) 2D convolution: 4 ﬁlters, stride 2, 32 channels, ReLU activation (h2) 2D convolution: 4 ﬁlters, stride 2, 64 channels, ReLU activation (h3) 2D convolution: 4 ﬁlters, stride 2, 128 channel, ReLU activations (h4) 2D convolution: 4 ﬁlters, stride 2, 256 channel, ReLU activations (out) Flatten each sample to a 1D tensor
Revised: (h1) 2D convolution: 4 ﬁlters, stride 2, 32 channels, ReLU activation (h2) 2D convolution: 4 ﬁlters, stride 2, 64 channels, ReLU activation (h3) 2D convolution: 4 ﬁlters, stride 2, 128 channel, ReLU activations (h4) 2D convolution: 4 ﬁlters, stride 2, 256 channel, ReLU activations (out) Flatten each sample to a 1D tensor

Original: G.2. D ECONVOLUTIONAL NET FOR IMAGE RECONSTRUCTION
Revised: G.2. D ECONVOLUTIONAL NET FOR IMAGE RECONSTRUCTION

Original: (h1) Fully connected layer with 1024 neurons, ReLU activation (h1’)
Revised: (h1) Fully connected layer with 1024 neurons, ReLU activation (h1’)

Original: Reshape to 1 × 1 images with 1024 channels (h2) 2D conv-transpose: 5 ﬁlters, stride 2, 128 channels, ReLU activation (h3) 2D convolution: 5 ﬁlters, stride 2, 64 channels, ReLU activation (h4) 2D convolution: 6 ﬁlters, stride 2, 32 channel, ReLU activations (out) 2D convolution: 6 ﬁlters, stride 2, C channel
Revised: Reshape to 1 × 1 images with 1024 channels (h2) 2D conv-transpose: 5 ﬁlters, stride 2, 128 channels, ReLU activation (h3) 2D convolution: 5 ﬁlters, stride 2, 64 channels, ReLU activation (h4) 2D convolution: 6 ﬁlters, stride 2, 32 channel, ReLU activations (out) 2D convolution: 6 ﬁlters, stride 2, C channel

Original: G.2. MLP FOR ACTION SEQUENCE ENCODING
Revised: G.2. MLP FOR ACTION SEQUENCE ENCODING

Original: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h3) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with 32 output neurons
Revised: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h3) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with 32 output neurons

Original: G.2. MLP FOR ACTION SEQUENCE RECONSTRUCTION
Revised: G.2. MLP FOR ACTION SEQUENCE RECONSTRUCTION

Original: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with T × d a neurons, tanh activation then scaled to action space
Revised: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with T × d a neurons, tanh activation then scaled to action space

Original: G.2. MLP FOR TRANSITION MATRIX A
Revised: G.2. MLP FOR TRANSITION MATRIX A

Original: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with d f × d g neurons
Revised: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with d f × d g neurons

Original: G. D YNAMICS -A WARE D ISCOVERY OF S KILLS (DADS)
Revised: G. D YNAMICS -A WARE D ISCOVERY OF S KILLS (DADS)

Original: For our experiments using DADS, we use policy and discriminator architecture of two hidden layers of size 256 with ReLU activations and a latent dimension of 2.
Revised: For our experiments using DADS, we use policy and discriminator architecture of two hidden layers of size 256 with ReLU activations and a latent dimension of 2.

Original: Our discount factor is .
Revised: Our discount factor is .

Original: As per suggestions from the original paper, we use 512 prior samples to approximate the intrinsic reward, run 32 discriminator updates and 128 policy updates per 1000 timesteps; learning rates of 3 × 10 −and batch size of 256 are used throughout.
Revised: As per suggestions from the original paper, we use 512 prior samples to approximate the intrinsic reward, run 32 discriminator updates and 128 policy updates per 1000 timesteps; learning rates of 3 × 10 −and batch size of 256 are used throughout.

Original: We use a replay buffer to help stabilize training of size 20000, used only for policy updates.
Revised: We use a replay buffer to help stabilize training of size 20000, used only for policy updates.

Original: G. H OPPER EXPERIMENT DETAILS
Revised: G. H OPPER EXPERIMENT DETAILS

Original: Each episode is set to a ﬁxed length of 200 steps.
Revised: Each episode is set to a ﬁxed length of 200 steps.

Original: We use PPO as the RL backbone for our control policy.
Revised: We use PPO as the RL backbone for our control policy.

Original: The policy is updated every 1000 steps in the environment, with learning rate of 1e-4 and γ = 0 .
Revised: The policy is updated every 1000 steps in the environment, with learning rate of 1e-4 and γ = 0 .