Original: E FFICIENT E MPOWERMENT E STIMATION FOR U NSUPERVISED S TABILIZATION
Revised: E FFICIENT E MPOWERMENT E STIMATION FOR U NSUPERVISED S TABILIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Intrinsically motivated artiﬁcial agents learn advantageous behavior without externally-provided rewards.
Revised: Intrinsically motivated artiﬁcial agents learn advantageous behavior without externally-provided rewards.

Original: Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright positions, which is a prototypical intrinsically motivated behavior for upright standing and walking.
Revised: Previously, it was shown that maximizing mutual information between agent actuators and future states, known as the empowerment principle, enables unsupervised stabilization of dynamical systems at upright positions, which is a prototypical intrinsically motivated behavior for upright standing and walking.

Original: That follows from the coincidence between the objective of stabilization and the objective of empowerment.
Revised: That follows from the coincidence between the objective of stabilization and the objective of empowerment.

Original: Unfortunately, sample-based estimation of this kind of mutual information is challenging.
Revised: Unfortunately, sample-based estimation of this kind of mutual information is challenging.

Original: Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased, have high sample complexity, and are unstable in training.
Revised: Recently, various variational lower bounds (VLBs) on empowerment have been proposed as solutions; however, they are often biased, have high sample complexity, and are unstable in training.

Original: In this work, we propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel, which allows us to efﬁciently calculate an unbiased estimator of empowerment by convex optimization.
Revised: In this work, we propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel, which allows us to efﬁciently calculate an unbiased estimator of empowerment by convex optimization.

Original: We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of our method by comparing it to the existing VLB approaches.
Revised: We demonstrate our solution for sample-based unsupervised stabilization on different dynamical control systems and show the advantages of our method by comparing it to the existing VLB approaches.

Original: Speciﬁcally, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function and allows, for the ﬁrst time, to estimate empowerment from images.
Revised: Speciﬁcally, we show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function and allows estimation of empowerment from images.

Original: Consequently, our method opens a path to wider and easier adoption of empowerment for many applications
Revised: Consequently, our method opens a path to wider and easier adoption of empowerment for many applications

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Intrinsic motivation allows artiﬁcial and biological agents to acquire useful behaviours without external knowledge (Barto et al.
Revised: Intrinsic motivation allows artiﬁcial and biological agents to acquire useful behaviours without external knowledge (Barto et al.

Original: ; Chentanez et al.
Revised: ; Chentanez et al.

Original: ; Schmidhuber (2010); Barto (2013);
Revised: ; Schmidhuber (2010); Barto (2013);

Original: Oudeyer et al.
Revised: Oudeyer et al.

Original: In the framework of reinforcement learning (RL), this external knowledge is usually provided by an expert through a task-speciﬁc reward, which is optimized by an artiﬁcial agent towards a desired behavior (Mnih et al.
Revised: In the framework of reinforcement learning (RL), this external knowledge is usually provided by an expert through a task-speciﬁc reward, which is optimized by an artiﬁcial agent towards a desired behavior (Mnih et al.

Original: (2013); Schulman et al.
Revised: (2013); Schulman et al.

Original: In contrast, an intrinsic reward can arise solely from the interaction between the agent and environment, which eliminates, in certain scenarios, the need for domain knowledge and reward engineering  (Mohamed & Rezende (2015);
Revised: In contrast, an intrinsic reward can arise solely from the interaction between the agent and environment, which eliminates the need for domain knowledge and reward engineering in some settings (Mohamed & Rezende (2015);

Original: Houthooft et al.
Revised: Houthooft et al.

Original: ; Pathak et al.
Revised: ; Pathak et al.

Original: Previously, it was shown that maximizing mutual information (Cover & Thomas (2012)) between an agent’s actuators and sensors reveals preferable states in the environment that have a higher potential to achieve a larger number of eventual goals (Klyubin et al.
Revised: Previously, it was shown that maximizing mutual information (Cover & Thomas (2012)) between an agent’s actuators and sensors can guide the agent towards states in the environment which have a higher potential to achieve a larger number of eventual goals (Klyubin et al.

Original: (2005); Wissner-Gross & Freer (2013)).
Revised: (2005); Wissner-Gross & Freer (2013)).

Original: Maximizing this kind of mutual information is known as the empowerment principle (Klyubin et al.
Revised: Maximizing this kind of mutual information is known as the empowerment principle (Klyubin et al.

Original: ; Salge et al.
Revised: ; Salge et al.

Original: Empowerment is a function of the state.
Revised: Empowerment is a function of the state.

Original: Previously, it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment which was demonstrated on various dynamical control systems (Jung et al.
Revised: Previously, it was found that an agent maximizing its empowerment converges to an unstable equilibrium of the environment in various dynamical control systems (Jung et al.

Original: ; Salge et al.
Revised: ; Salge et al.

Original: Maximilian et al.
Revised: ; Karl et al.

Original: In this sense, ascending the gradient of the empowerment functiontowards unstable equilibrium coincides with the objective of optimal control for stabilization at  unstable equilibrium (Strogatz (2018)), which is an important task for both engineering (Todorov (2006)) and biological 1 systems.
Revised: In this application, ascending the gradient of the empowerment function coincides with the objective of optimal control for stabilization at an unstable equilibrium (Strogatz (2018)), which is an important task for both engineering (Todorov (2006)) and biological 1 systems; we refer to this as the essential property of empowerment.

Original: It follows from the above mentioned prior workson empowerment that a plausible estimate of the empowerment function should possess this essential property.
Revised: It follows from the aforementioned prior works that a plausible estimate of the empowerment function should possess this essential property.

Original: Empowerment has been found to be useful for a broad spectrum of applications. Among others are unsupervised skill discovery (Sharma et al.
Revised: Empowerment has been found to be useful for a broad spectrum of applications, including: unsupervised skill discovery (Sharma et al.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: ; Gregor et al.
Revised: ; Gregor et al.

Original: Maximilian et al.
Revised: ; Karl et al.

Original: ; Campos et al.
Revised: ; Campos et al.

Original: (2020)), human-agent coordination (Salge & Polani (2017);
Revised: (2020); human-agent coordination (Salge & Polani (2017);

Original: Guckelsberger et al.
Revised: Guckelsberger et al.

Original: , assistance (Yuqing et al.
Revised: ; assistance (Yuqing et al.

Original: (2021)), and stabilization (Tiomkin et al.
Revised: (2021)); and stabilization (Tiomkin et al.

Original: In general, in these applications, the agent maximizes mutual information between its skills and one of: (1) its actuators, which generate either single actions (Maximilian et al.

Original: ; Gregor et al.

Original: or (2) action sequences (Tiomkin & Tishby (2017);

Original: Salge et al.

Original: ; (3) options (Gregor et al.

Original: (2017)); or (4) skills (Sharma et al.

Original: Eysenbach et al.

Original: There exist two possibilities for the estimation and use of empowerment: sequential and concurrent.

Original: In sequential mode, one ﬁrst estimates empowerment before using it and/or its byproducts for downstream tasks (Sharma et al.

Original: Eysenbach et al.

Original: In concurrent mode, one estimates and uses empowerment simultaneously, which is adopted by our current work.

Original: The concurrent mode is suitable for scenarios when the empowerment objective coincides with the control objective, which happens for instance in stabilization.

Original: Estimation of mutual information is hard (McAllester & Statos (2018)).
Revised: However, estimation of mutual information is hard (McAllester & Statos (2018)).

Original: Fortunately, recent advances in deep neural networks (Goodfellow et al.
Revised: Fortunately, recent advances in deep learning (Goodfellow et al.

Original: (2016)) have brought new opportunities for the estimation of mutual information from samples by means of variational lower bounds (VLBs) (Poole et al.
Revised: (2016)) have brought new opportunities for the estimation of mutual information from samples. Past work has utilized variational lower bounds (VLBs) (Poole et al.

Original: ; Alemi et al.
Revised: ; Alemi et al.

Original: ; Tschannen et al.
Revised: ; Tschannen et al.

Original: ; Mohamed & Rezende (2015)).
Revised: ; Mohamed & Rezende (2015)) to achieve an estimate of the empowerment.

Original: However, VLB approaches to empowerment in dynamical control systems (Sharma et al.
Revised: However, VLB approaches to empowerment in dynamical control systems (Sharma et al.

Original: ; Achiam et al.
Revised: ; Achiam et al.

Original: (2018)) have high sample complexity, are often unstable in training, and may be biased.
Revised: (2018)) have high sample complexity, are often unstable in training, and may be biased.

Original: Moreover, it was notstudied previously whether VLBs on empowerment preserve the essential properties of the empowerment function.
Revised: Moreover, it was not previously studied if empowerment estimators learned via VLBs possess the essential properties of empowerment.

Original: In this work, we introduce a new method for empowerment estimation and utilize the above-mentioned property as an "indicator" for the quality of the estimation.
Revised: In this work, we introduce a new method, Latent Gaussian Channel Empowerment (Latent GCE), for empowerment estimation and utilize the above-mentioned property as an “indicator” for the quality of the estimation.

Original: Speciﬁcally, we propose a particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories.
Revised: Speciﬁcally, we propose a particular representation for dynamical control systems using deep neural networks which is learned from state-action trajectories.

Original: This representation admits an efﬁcient estimation of empowerment by convex optimization (Cover & Thomas (2012)), both from raw state and from images.
Revised: This representation admits an efﬁcient estimation of empowerment by convex optimization (Cover & Thomas (2012)), both from raw state and from images.

Original: We propose an algorithm for simultaneous estimation and maximization of empowerment by the standard optimizers such as Proximal Policy Optimization (PPO), (Schulman et al.
Revised: We propose an algorithm for simultaneous estimation and maximization of empowerment using standard RL algorithms such as Proximal Policy Optimization (PPO) (Schulman et al.

Original: and Soft Actor-Critic (SAC) (Haarnoja et al.
Revised: and Soft Actor-Critic (SAC) (Haarnoja et al.

Original: We test our method on the task of unsupervised stabilization of dynamical systems with using only the intrinsic reward, showing our estimator exhibits essential properties of the empowerment function.
Revised: We test our method on the task of unsupervised stabilization of dynamical systems with solely intrinsic reward, showing our estimator exhibits essential properties of the empowerment function.

Original: We believe our experiments clearly convey the gist of our method without losing the generality our estimator beyond the stabilization task; we discuss more general applications in the future directions.

Original: We demonstrate the advantages of our method thorough comparisons to the existing state-of-the-art empowerment estimators in different dynamical systems from the MuJoCo simulator (Todorov et al.
Revised: We demonstrate the advantages of our method through comparisons to the existing state-of-the-art empowerment estimators in different dynamical systems from the OpenAI Gym simulator (Brockman et al.

Original: We found that our method (i) has a lower sample complexity, (ii) is more stable in training, (iii) possesses the essential properties of the empowerment function, and (iv) allows usfor the ﬁrst time to  estimatethe empowermentfunction from images.
Revised: We ﬁnd that our method (i) has a lower sample complexity, (ii) is more stable in training, (iii) possesses the essential properties of the empowerment function, and (iv) allows us to accurately estimate empowerment from images.

Original: In addition, this work provides for a novel benchmark between different estimators of empowerment, speciﬁcally comparing their  properties.
Revised: In addition, we provide a novel benchmark between different estimators of empowerment, speciﬁcally comparing their essential properties.

Original: We hope such a review of the existing methods for empowerment estimation will be helpful to pushfurther this research direction.
Revised: We hope such a review of the existing methods for empowerment estimation will help push this research direction.

Original: The paper is organized as follows.

Original: In Section 2 we review the necessary background on reinforcement learning, empowerment, and relevant existing works.

Original: In Section 4 we explain our method.

Original: In Section 5, we provide numerical simulations of our method and comparison between our method to the existing state-of-the-art.

Original: In Section 6 we summarize the paper, discuss the limitations of our method, and propose the continuation which is beyond the scope of the current work.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: In this section, we review the necessary background for our method, consisting of the reinforcement learning setting, various empowerment estimators, and the Gaussian channel capacity.
Revised: In this section, we review the necessary background for our method, consisting of the reinforcement learning setting, various empowerment estimators, and the Gaussian channel capacity.

Original: We also review the underlying components in relevant prior work which we use for comparison to our method.
Revised: We also review the underlying components in relevant prior work which we use for comparison to our method.

Original: 2. R EINFORCEMENT L EARNING
Revised: 2. R EINFORCEMENT L EARNING

Original: The reinforcement learning (RL) setting is modeled as an inﬁnite-horizon Markov Decision Process (MDP) deﬁned by: the state space S , the action space A , the transition probabilities p ( s (cid:48) | s, a ) , the initial state distribution p 0 ( s ) , the reward function r ( s, a ) ∈ R , and the discount factor γ .
Revised: The reinforcement learning (RL) setting is modeled as an inﬁnite-horizon Markov Decision Process (MDP) deﬁned by: the state space S , the action space A , the transition probabilities p ( s (cid:48) | s, a ) , the initial state distribution p 0 ( s ) , the reward function r ( s, a ) ∈ R , and the discount factor γ .

Original: The goal of RL is to ﬁnd an optimal control policy π ( a | s ) that maximizes the expected return, i.e. max π E s 0 ∼ p 0 ,a t ∼ π,s t +1 ∼ p (cid:2)(cid:80) ∞ t =

Revised: The goal of RL is to ﬁnd an optimal control policy π ( a | s ) that maximizes the expected return, i.e. max π E s 0 ∼ p 0 ,a t ∼ π,s t +1 ∼ p (cid:2)(cid:80) ∞ t =

Original: γ t r ( s t , a t ) (cid:3) (Sutton & Barto (2018)).
Revised: γ t r ( s t , a t ) (cid:3) (Sutton & Barto (2018)).

Original: 2. E MPOWERMENT
Revised: 2. E MPOWERMENT

Original: Interaction  an agent and its environment is plausibly described by the perception-action cycle (PAC), (Tishby & Polani (2011)), where the agent observes the state of the environment via its sensors and responds with its actuators.
Revised: Interaction between an agent and its environment is plausibly described by the perception-action cycle (PAC), (Tishby & Polani (2011)), where the agent observes the state of the environment via its sensors and responds with its actuators.

Original: Simply put, the maximal information rate from actuators to sensors isin an inherent property of PAC which characterizes  empowerment of the agent, formally deﬁned below.
Revised: The maximal information rate from actuators to sensors is an inherent property of PAC which characterizes the empowerment of the agent, formally deﬁned below.

Original: Empowerment (Klyubin et al.
Revised: Empowerment (Klyubin et al.

Original: (2005)) is deﬁned by the maximal mutual information rate (Cover &
Revised: (2005)) is deﬁned by the maximal mutual information rate (Cover &

Original: Thomas (2012)) between agent sensor observations, o ∈ O, and agent actuators, a ∈ A, given the current state, s ∈ S .
Revised: Thomas (2012)) between the agent’s sensor observations o ∈ O and actuators a ∈ A given the current state s ∈ S .

Original: It is fully speciﬁed by a ﬁxed probability distribution of sensor observations conditioned on the actuators and thecurrent state, p ( O | A, s ) , denoted by “channel”, and by a free probability of the actuator signals conditioned on thecurrent state, ω
Revised: It is fully speciﬁed by a ﬁxed probability distribution of sensor observations conditioned on the actuators and the state, p ( O | A, s ) , denoted by “channel”, and by a free probability of the actuator signals conditioned on the state, ω

Original: ( A | s ) , denoted by “source”.
Revised: ( A | s ) , denoted by “source”.

Original: Empowerment, E, is a capacity of the channel,p ( O | A, s ) , which is found by optimization of the source distribution, ω ( A | s ) :
Revised: The empowerment E is the capacity of the channel, which is found via optimization of the source distribution ω ( A | s ) :

Original: [Equation]
Revised: [Equation]

Original: where I [ O ; A | s ] is the mutual information, reﬂecting the difference between the entropy of the sensor inputs, H ( O | s ) and the corresponding conditional entropy, H ( O | A, s ) 2 .
Revised: where I [ O ; A | s ] is the mutual information, reﬂecting the difference between the entropy of the sensor inputs, H ( O | s ) and the corresponding conditional entropy, H ( O | A, s ) 2 .

Original: In this work, we follow the originalformulation of empowerment  (Klyubin et al.
Revised: In this work, we follow the original empowerment formulation (Klyubin et al.

Original: (2005)), where A is an action sequence of length T started at the state s, and O is the resultingsensor observation sequence.
Revised: (2005)), where A is an action sequence of length T starting from the state s and O is the resulting observation sequence.

Original: However, our method is applicable to other combinations of length of observation and control sequences.
Revised: However, our method is applicable to other choices of observation and control sequences.

Original: Remark 1 The source distribution in equation 1, ω ∗ ( A | s ) , is used only for the estimation of empowerment.
Revised: Remark 1 The source distribution in Equation 1, ω ∗ ( A | s ) , is used only for the estimation of empowerment.

Original: Another policy, π ( a | s ) , is calculated using empowerment as an intrinsic reward by either reinforcement learning (Maximilian et al.
Revised: Another policy, π ( a | s ) , is calculated using empowerment as an intrinsic reward by either reinforcement learning (Karl et al.

Original: (2019)) or  1-step greedy algorithm (Jung et al.
Revised: or a 1-step greedy algorithm (Jung et al.

Original: (2011); Salge et al.
Revised: Salge et al.

Original: to reach the states with maximal empowerment, as explained.
Revised: to reach the states with maximal empowerment.

Original: Strictly speaking, the source distribution ω maximizes the mutual information, (i.e. estimates  empowerment), while the policy distribution π maximizes expected accumulated empowerment along trajectories.
Revised: Strictly speaking, the source distribution ω maximizes the mutual information, (i.e. estimates the empowerment), while the policy distribution π maximizes expected accumulated empowerment along trajectories.

Original: In general, the capacity of an arbitrary channel for a continuous random variable is unknown except for a few special cases such as the Gaussian linear channel, as reviewed below.2.2. E MPOWERMENT BY G AUSSIAN C HANNEL C APACITY The Gaussian linear channel in our case is given by: [Equation]
Revised: In general, the capacity of an arbitrary channel for a continuous random variable is unknown except for a few special cases such as the Gaussian linear channel, as reviewed below.

Original: where O and A are the observation and actuator sequences with dimensions

Original: d O × 1 and d A × 1 , respectively; G ( s ) is a matrix with dimensions d O × d A ; and η is multivariate Gaussian noise with dimension

Original: d

Original: O × 1 .

Original: Following Salge et al.

Original: (2013), we assume the noise is distributed by the normal distribution.

Original: For completeness, Appendix A provides the analytical derivation of G ( s ) in the case of known dynamics for pendulum for the action sequence A with length 3.

Original: In contrast, the current work proposes a method how to estimate G ( s ) for arbitrary dynamics and arbitrary length of the action sequence from samples. The capacity of the channel in equation 2 is given by, (Cover & Thomas (2012)): [Equation] where σ i ( s ) , p i , k , and P , are the i-th singular value of G ( s ) , the corresponding actuator signal power, the number of nonzero singular values, and the total power of the actuator signal, respectively.
Revised: As a result, past works have largely relied on variational lower bounds instead.

Original: The optimal solution to equation 3 is given by:

Original: [Equation]

Original: where ν is found by a line search (Cover & Thomas (2012)) to satisfy the total power constraint.

Original: The key observation in our method is that it is possible to learn the matrix G ( s ) from samples, which allows us to estimate empowerment efﬁciently by the line search in equation 4 even for high-dimensional dynamical systems either from raw state or from images (pixels).

Original: In Section 4 we describe how to represent G ( s ) by deep neural networks and train them from data.

Original: The appropriateness of equationfor empowerment estimation has been studied in Salge et al.

Original: (2013), where the dynamics was known and the state was fully observable.

Original: These assumptions allow analytical computation of empowerment for certain dynamical systems, which provides us with the ground truth empowerment landscapes in

Original: Section 5.

Original: 2.2. E MPOWERMENT BY V ARIATIONAL L OWER B OUNDS
Revised: 2.2. E MPOWERMENT BY V ARIATIONAL L OWER B OUNDS

Original: The mutual information in equation 1 can be bounded from below as follows (Mohamed & Rezende (2015); Gregor et al.
Revised: The mutual information in Equation 1 can be bounded from below as follows (Mohamed & Rezende (2015); Gregor et al.

Original: ; Maximilian et al.
Revised: ; Karl et al.

Original: ; Poole et al.
Revised: ; Poole et al.

Original: [Equation]
Revised: [Equation]

Original: O [Equation] where q ( A | O, s ) and ω ( A | s ) are represented by neural networks with parameters φ and ψ respectively.
Revised: where q ( A | O, s ) and ω ( A | s ) are represented by neural networks with parameters φ and ψ respectively.

Original: The lower boundon empowerment in equation 5 can be estimated from samples by reinforcement learning withthe intrinsic reward r  = (cid:80)
Revised: The lower bound in Equation 2 can be estimated from samples using reinforcement learning with intrinsic reward r ( s t , a t ) = (cid:80)

Original: Tt log q φ ( a t | o t , s ) − log ω ψ ( a t | s t ) , as detailed in (Gregor et al.
Revised: Tt log q φ ( a t | o t , s ) − log ω ψ ( a t | s t ) , as detailed in (Gregor et al.

Original: ; Maximilian et al.
Revised: ; Karl et al.

Original: To apply this variational lower bound on empowerment to unsupervised stabilization of dynamical systems one needs to learn three distributions: q φ ( a t | o t , s ) and ω ψ ( a t | s t ) for the lower bound in equation 5, and π ( a | s ) as the control policy, as mentioned in Remark 1.
Revised: To apply this variational lower bound on empowerment to unsupervised stabilization of dynamical systems one needs to learn three distributions: q φ ( a t | o t , s ) and ω ψ ( a t | s t ) for the lower bound in Equation 2, and π ( a | s ) as the control policy (see Remark 1).

Original: Developing the mutual information in equation 1 in the opposite direction, one gets another lower bound on empowerment:
Revised: Developing the mutual information in Equation 1 in the opposite direction, one gets another lower bound on empowerment, an approximation of which was used in (Sharma et al.

Original: [Equation]
Revised: [Equation]

Original: An approximation of the lower bound in equation 6 is used in (Sharma et al.

Original: (2020)), where A is as a skill

Original: z

Original: and O is the next state after executing the skill z from the current state s .

Original: For sake of the clearance of the exposition, the above-explained variational empowerment is schematically depicted in Figure 1.
Revised: For sake of the clarity of the exposition, the above-explained variational empowerment is schematically depicted in Figure 1.

Original: The controller policy, π θ ( a | s ) , collects transitions in environment, ( s, a, s (cid:48) ) .
Revised: The controller policy, π θ ( a | s ) , collects transitions in environment, ( s, a, s (cid:48) ) .

Original: The source policy ω ψ ( a | s ) generates the action a at every state s visited by the controller policy, and the environment transits to the next state, s (cid:48)
Revised: The source policy ω ψ ( a | s ) generates the action a at every state s visited by the controller policy, and the environment transits to the next state, s (cid:48)

Original: (yellow block).
Revised: (yellow block).

Original: The inverse channel policy q ψ ( a | s, a (cid:48) ) is regressed to a given s and and s (cid:48) , generating the intrinsic reward,
Revised: The inverse channel policy q ψ ( a | s, a (cid:48) ) is regressed to a given s and and s (cid:48) , generating the intrinsic reward,

Original: C = log ω
Revised: C = log ω

Original: φ ( a | s ) − log q ψ ( a | s, a (cid:48) ) , which is used as the reward to improve the controller policy via reinforcement learning.
Revised: φ ( a | s ) − log q ψ ( a | s, a (cid:48) ) , which is used as the reward to improve the controller policy via reinforcement learning.

Original: This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment.
Revised: This scheme aims to transfer the intrinsically motivated agent to the state of the maximal empowerment in the environment.

Revised: 2.2. E MPOWERMENT FOR G AUSSIAN C HANNELS

Revised: The Gaussian linear channel in our case is given by:

Revised: [Equation]

Revised: where O and A are the observation and actuator sequences with dimensions

Original: P RIOR A RT
Revised: d O × 1 and d A × 1 , respectively; G ( s ) is a matrix with dimensions d O × d A ; and η is multivariate Gaussian noise with dimension d O × 1 . Following Salge et al.

Revised: (2013), we assume the noise is distributed as a Gaussian.

Revised: For completeness, Appendix A provides the analytical derivation of G ( s ) in the case of known dynamics for pendulum for the action sequence A with length 3.

Revised: In contrast, the current work proposes a method how to estimate G ( s ) for arbitrary dynamics and arbitrary length of the action sequence from samples.

Revised: The details of the calculation of Gaussian channel capacity are included in Appendix B.

Revised: The key observation in our method is that we can learn G ( s ) from samples, which allows us to estimate empowerment efﬁciently for arbitrary dynamics, including for high-dimensional dynamical systems or from images (pixels).

Revised: As a result, it is not required to resort to lower bounds as done in prior work.

Revised: In Section 4, we describe how to learn G ( s ) from samples using deep neural networks.

Revised:  R ELATED W ORK

Original: In this section we review the relevant prior works, emphasizing the main differences between the existing art and the current work.
Revised: In this section, we review the relevant prior work and state of the art in empowerment estimation, emphasizing the main differences between these existing works and our proposed approach.

Original:  The representationof a dynamical system for empowerment estimation given by equation 2 was previously considered in (Salge et al.
Revised: Estimation via Gaussian channel. The representation given by Equation 4 was previously considered in (Salge et al.

Original: (2013)), where the channel, p ( O | A, s ) was assumed to be known.
Revised: (2013)), where the channel p ( O | A, s ) was assumed to be known.

Original: In the current work, we remove this assumption and extend the applicability of the efﬁcient channel capacity solution given by equation 3 to arbitrary observations, including images. 
Revised: In our current work, we remove this assumption and extend the applicability of the efﬁcient channel capacity solution given by Equation 11 to arbitrary observations, including images. Variational approaches to estimation.

Original: The lower bound in equation 5 is used in (Mohamed & Rezende (2015); Gregor et al.
Revised: The lower bound in Equation 2 is used in (Mohamed & Rezende (2015); Gregor et al.

Original: Maximilian et al.
Revised: ; Karl et al.

Original: (2019)), where different additional assumptions are made.
Revised: (2019)), where different additional assumptions are made.

Original: Speciﬁcally, Mohamed & Rezende (2015) estimates empowerment in open loop, while Gregor et al.
Revised: Speciﬁcally, Mohamed & Rezende (2015) estimates empowerment in open loop, while Gregor et al.

Original: ; Maximilian et al.
Revised: ; Karl et al.

Original: (2019) provide a solution for closed-loop empowerment, and Maximilian et al.
Revised: (2019) provide a solution for closed-loop empowerment, and Karl et al.

Original: (2019) assumeson differentiability of the dynamics.
Revised: (2019) assumes differentiability of the dynamics.

Original: In the current work, we do not assume differentiable dynamics and we do not resort to variational bounds, aiming to achieve a more stable, sample-efﬁcient, and generalizable estimator of empowerment without losing the essential properties of the empowerment function, as explained in the Introduction.
Revised: In our work, we do not assume differentiable dynamics and we do not resort to variational bounds, aiming to achieve a more stable, sample-efﬁcient, and generalizable estimator of empowerment without losing the essential properties of the empowerment function, as explained in the Introduction.

Original: The works by (Mohamed & Rezende (2015); Gregor et al.
Revised: The works by (Mohamed & Rezende (2015); Gregor et al.

Original: (2017)) precede that by (Maximilian et al.
Revised: (2017)) precede that by (Karl et al.

Original: Consequently, Section 5 provides comparisons between our empowerment estimator and  the latter representative of this kind of variational lower bound on empowerment. An approximation of the lower bound in equation 6 is successfully used by Sharma et al.
Revised: , and we also experimentally ﬁnd the latter to be more stable. Consequently, in Section 5 we provide comparisons to Karl et al.

Original: (2020) as described above.
Revised: (2019), which represents the current start of the art in this line of research.

Original: Section 5 provides comparisons between our empowerment estimator and
Revised: Skill discovery. An approximation of the lower bound in Equation 3 is successfully used by

Original: Sharma et al.
Revised: Sharma et al.

Original: (2020)with regards to stability intraining, sample efﬁciency, and correctness of the empowerment landscape generated by both methods. Although the work by Sharma et al. (2020) aims to discover adiverse set of skills using different machinery – which is not the main focus of our current work – their objective is of the same type us ours: to maximize the mutual information between actuators and states (or sensor). In Section 5 we show that the empowerment landscape derived by Sharma et al.
Revised: (2020) in the context of learning a set of skills.

Original: (2020) converges to the true empowerment landscape in simple cases, while it does not converge to a known empowerment landscape in more complicated dynamical systems.

Original: Another relevant and important work is Eysenbach et al.
Revised: Another relevant and important work is Eysenbach et al.

Original: Citing the authors, their work is about "maximizing the empowerment of a hierarchical agent whose action space is the set of skills" .
Revised: Citing the authors, their work is about “maximizing the empowerment of a hierarchical agent whose action space is the set of skills” .

Original: Speciﬁcally, they suggest maximizing a variational lower bound in the form ofequation 5 on the  mutual information between skills and states with an additional term of conditional entropy of actions given skills and states, H ( A | S, Z ) , (which is equal tozero for a deterministic policy).
Revised: Section 5 provides comparisons between our empowerment estimator and these methods with regards to stability in training, sample efﬁciency, and correctness of the empowerment landscape generated by both methods. Although they aim to achieve a different goal from our work – unsupervised skill discovery vs unsupervised stabilization – their objectives are the same type us ours: maximizing mutual information between actuators and sensors. In Section 5 we show that the empowerment landscape derived by these methods converge to the true empowerment landscape in simple cases, but fail to converge to a known empowerment landscape in more complicated dynamical systems.

Original: These works represent state-of-the-art empowerment estimation.
Revised: These works represent the state of the art in empowerment estimation.

Original: We believe a thorough comparison between our method and this prior art, provided in Section 5, clearly demonstrate the main properties of our method, and clarify our contribution to the improvement of  empowerment estimation in dynamical systems from samples.
Revised: We believe a thorough comparison between our method and these works, as provided in Section 5, clearly demonstrate the main properties of our method and clarify our contribution to the improvement of sample-based empowerment estimation in dynamical systems.

Original: As mentioned in the introduction, empowerment is useful for a broad spectrum of different applications.
Revised: As mentioned in the introduction, empowerment is useful for a broad spectrum of different applications.

Original: In this section, we highlightedonly the most relevant and updated methods, which we hope are helpful to see the merits and advances of our approach.s
Revised: In this section, we highlighted the most relevant and updated methods, which we hope are helpful to see the merits and advances of our approach.

Original: Our improved empowerment estimation possesses the essential properties of the empowerment function; in particular, it converges to the correct empowerment landscape.
Revised: Our improved empowerment estimation possesses the essential properties of the empowerment function: in particular, it converges to the correct empowerment landscape.

Original: The training has improved stability.
Revised: The training has improved stability.

Original: Furthermore,for the ﬁrst time, it allows for the estimation of the empowerment function from images, which we demonstrate in Section 5, where we show purely intrinsically motivated (empowerment-based) stabilization of inverted pendulum from images.
Revised: Furthermore, it allows for the estimation of the empowerment function from images, which we demonstrate in Section 5, where we show purely intrinsically motivated (empowerment-based) stabilization of inverted pendulum from images.

Original: This is signiﬁcant progress in empowerment estimation which was impossible prior to this work.
Revised: Accurate estimation of empowerment from images signiﬁcantly expands the possible breadth of applications for empowerment.

Original: In the next section, we explain our method, referring to the background reviewed in this section.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: The desiderata for our method as follows:
Revised: The desiderata for our method as follows:

Original: • to learn the channel matrix ( G ( s ) in equation 2) from samples foran arbitrary dynamics.
Revised: • to learn the channel matrix ( G ( s ) in Equation 4) from samples for arbitrary dynamics.

Original: • to utilize an existing optimizer for RL problems, such as SAC and/or PPO, to enable reliable large-scale training of deep neural networks.
Revised: • to utilize general RL optimizers to reliably train intrinsically motivated policies.

Original: • to enable empowerment maximization from raw states and/or images.
Revised: • to enable empowerment maximization from arbitrary dynamics using latent representations.

Original: The rationale of these desired features as follows. The ability to learn G ( s ) from samples would allow us to estimate empowerment by the efﬁcient line research algorithm in equation 4. Training deep neural networks by RL is a hard problem (Mnih et al. Sophisticated optimizers, such PPO and/or SAC, can alleviate the instabilities in training, enhance convergence and exploration.
Revised: These are desirable as: the ability to learn G ( s ) from samples would allow us to estimate empowerment by the efﬁcient line search algorithm in Equation 12, recent advances in RL algorithms can alleviate challenges in stability and sample efﬁciency, and latent representations would allow our method to be applicable even when the observation or underlying dynamics do not satisfy the

Original: Finally, to push forward the entire ﬁeld of intrinsically motivated RL, we hope to demonstrate performance that was previously impossible: a purely intrinsically motivated (via empowerment) agent acting from images (pixels).
Revised: Gaussian Linear Channel assumption. In this work, we present Latent Gaussian Channel Empowerment, or Latent-GCE in short.

Original: Schematically, our method is depicted at Figure 2.
Revised: Schematically, our method is depicted in Figure 2.

Original: The controller policy π θ ( a | s ) collects trajectories in the environment, ( s, a, s (cid:48) ) , where s is either araw state or an image.
Revised: The controller policy π θ ( a | s ) collects trajectories in the environment, ( s, a, s (cid:48) ) , where s is either a state or image observation.

Original: The channel matrix G χ ( s t ) predicts the ﬁnal state of a trajectory corresponding to the action sequence a T − 1 t , started from s t .
Revised: The channel matrix G χ ( s t ) predicts the ﬁnal state of a trajectory corresponding to the action sequence a T − 1 t , starting from s t .

Original: The singular values of G χ ( s t ) are found by singular valuedecomposition. The empowerment, E ( s t ), is estimated by equation 3 and equation 4.
Revised: After performing singular value decomposition on G χ ( s t ) , the empowerment value E ( s t ) is estimated by Equation 11 and Equation 12.

Original: Theestimated values of empowerment  providethe intrinsic reward, demoted by r (cid:48)
Revised: The empowerment along trajectories provide intrinsic rewards, denoted by r (cid:48)

Original: ( τ ) , for the update of the controller policy, π θ( a | s ) ,by RL using Soft Actor Critic (SAC).
Revised: ( τ ) , for the update of the controller policy, π θ , using off-the-shelf RL algorithms.

Original: Pseudo-code summarising all the details of our method is provided below in Algorithm 1. 
Revised: Our method is summarized in high level in Algorithm 1. 4. E MBEDDING OF O BSERVATIONS AND A CTION S EQUENCES

Original: 1: Input: π θ 0 ( a | s ) - control policy, H - prediction horizon.
Revised: While the Gaussian Linear Channel deﬁned in Equation 4 provides a good estimate of simple dynamic systems, we want our empowerment estimator to generalize to arbitrary dynamics, including highdimensional systems, and environments where the observations are images. Using autoencoders, we embed the observations and action sequences such that their latent vector representations satisfy the Gaussian Linear Channel property. We use CNN encoder and decoder for image observations, or fully-connected networks for state observations.

Revised: The encoder-decoder pair for action sequences are also parameterized by fully-connected networks.

Original: 2: for i = 0 to N do 3: { τ k} Kk=1 ← run policy π θ i ( a | s ) {data collection in environment} 4: extract tuples ( st , a H− 1 t , st + H ) from { τ k } K k = {data arrangement for Gaussian channel} 5: for j = 1 to M do 6: χ ∗ ← arg min χ (cid:80) tuples (cid:13)(cid:13)
Revised: Let f χ , f − 1 χ , g χ , g − 1 χ denote the encoder-decoder pairs for the observations and action sequences, f χ maps the observations into d f dimensional latent do 3: { τ k Kk ← run policy π } 4: extract tuples ( s a H t , s

Revised: H from {

Revised: Kk = { for Gaussian

Original:  G χ ( s t ) · a H − 1 t − s t + H (cid:13)(cid:13) 2{projection to equation 2} 7: end for 8: { σ ∗ i ( s t ) } ki =1 ← SingularValueDecomposition ( G χ ∗ (s t )) {an alternative explained below} 9:
Revised: } 5: for j = 1 to M do 6: χ ∗ ← arg min χ (cid:80) tuples (cid:13)(cid:13) G χ ( s t ) · a Ht − s t + H (cid:13)(cid:13) 2{projection to Equation 4} 7: end for 8: { σ ∗ i ( s t ) } ki =1 ← SVD ( )) { an alternative explained in Appendix C} 9:

Original: E ( s t ) ← 12 (cid:80) ki
Revised: E ( s t ) ← 12 (cid:80) ki

Original: =1 log(1 + σ ∗ i ( s ) p ∗ i ) {equation 3 and equation 4} 10: reinforce π
Revised: =1 log(1 + σ ∗ i ( s ) p ∗ i ) {Equation 11 and Equation 12} 10: reinforce π

Original: θ
Revised: θ

Original: i ( a | s ) with r (cid:48) ( τ ) =
Revised: i ( a | s ) with r (cid:48) ( τ ) =

Original: (cid:80) Tt =1 E ( s t ) .
Revised: (cid:80) Tt = 1 γ t − 1 E ( s t ) .

Original: {policy update} 11: end for 
Revised: {policy update} 11: end for states, and g χ maps the H -step action sequences into d g dimensional latent actions. Our Latent Gaussian Linear Channel is formulated as follows: [Equation]

Original: The channel matrix G χ (  s t ) has dimensions d s × d a · ( H − 1) , where d s and d a are the dimensions ofthe state space (or latent representation) and the action space.
Revised: The channel matrix G χ ( f χ ( s t )) has dimensions d f × d g , where d f and d g are the dimensions of latent states and actions as deﬁned in section 4.1.

Original: In line 6, there is  a  multiplication between the matrix, G χ (  s t ) , and the  action sequence, a H − 1 t . = ( a t , a t +1 , . . . , a H − 1 ) .
Revised: In the Latent Gaussian Linear Channel, as formulated in Equation 5, the ending latent state is approximated by a matrix-vector multiplication between the channel matrix G χ ( f χ ( s t )) and the latent action g χ ( a Ht ) .

Original: The singular values, σ i ( s t ) , can be estimated directly without the singular values decomposition by representing the channel matrix as G χ ( s t )
Revised: The mapping from a latent state to its channel matrix is parameterized by a 3-layer fully-connected neural network. The output layer of the network contains d f · d g neurons, which is then reshaped into the d f × d g matrix. 4. O VERALL O PTIMIZATION O BJECTIVES

Original: =
Revised: The overall training objective for our Latent-GCE estimation consists of two parts: reconstruction loss and latent prediction loss. The reconstruction loss L R is the average L2 distance between the decoded latent representations and the original encoder inputs. Minimizing L R encourages the autoencoders to learn meaningful latent representations. The prediction loss L P is the average L2 distance between the matrix-vector product and the true latent ending state.

Original: U χ ( s t ) ∗ Diagm (Σ χ ( s t )) ∗ V χ ( s t ) with two additional orthonormality constraints: (cid:13)(cid:13)
Revised: Minimizing L P forces the latent space to maintain the Gaussian Linear Channel structure:

Original: [Equation]
Revised: [Equation]

Original: 2 2 and (cid:13)(cid:13) V χ ( s t ) V † χ ( s t ) − I { d a ( H − 1) × d a ( H − 1) } (cid:13)(cid:13) 2 2 .
Revised: Overall, our Latent-GCE Maximization scheme (Figure 2) trains two sets of parameters, ( θ , χ ), while VLB methods (Figure 1) generally needs three sets, ( θ , φ , ψ ).

Original: In our experiments, we did not experience a bottleneck in calculating the singular values decomposition for both raw states and images.
Revised: The reduced trainable components suggest that our method might be superior in terms of better stability and quicker convergence, which we conﬁrm in our experiment section.

Original:  But, the alternative direct calculation can be applied when SVD is undesired.
Revised:  E XPERIMENTS 5. U NSUPERVISED S TABILIZATION OF C LASSIC D YNAMIC S YSTEMS

Original: This explains the essence of our algorithm, which can work from images using simple algorithmic modiﬁcations (adding an encoder, a decoder, and a regularize for latent space) as described in Appendix B. Note that VLB methods (Figure 1) comprise of three neural networks, ( θ , φ , ψ ), while our method (Figure 2) only uses two ( θ , χ ), additionally using an efﬁcient 1D line-search in equation 4.
Revised: As discussed in Section 1, a control policy that maximizes agent empowerment should lead to agent stabilization, even at an unstable equilibrium. To evaluate this, we compare our method with two prior works by Mohamed & Rezende (2015) and Karl et al.

Original: Hence our method is simpler, using less “moving parts”, generally promising to be more stable.
Revised: on their ability to stabilize simple dynamic systems.

Original:  We conﬁrm this with the experiments in  the next section.
Revised: Speciﬁcally, in the 2D ball-in-box environment, empowerment is highest at the center of the box, where the ball is free to move in all directions; in the pendulum environment, empowerment is highest at the up-right position, where gravity assists in reaching other states. We compare average squared distance away from the stable position during policy training. As shown in Figure 3, all three methods are successful on the simple ball-in-box environment, whereas only our method consistently stabilizes the pendulum at an unstable equilibrium. We refer readers to

Original:  E XPERIMENTS 5.  C ONVERGENCE OF DIFFERENT ESTIMATORS
Revised: Appendix D for detailed visualization of pendulum empowerment/policy learning using our method, and Appendix E for results on cart-pole. 5. Q UALITATIVE C ONVERGENCE OF D IFFERENT E STIMATORS

Original: In this section we compare the convergence of different empowerment estimators to the true empowerment function.
Revised: We now qualitatively compare the convergence of different empowerment estimators to the true empowerment function.

Revised: Not only do we include solutions by VLB methods that approximate the classic empowerment deﬁnition (Mohamed & Rezende (2015); Karl et al.

Original: We observe that in simple environments such as "ball-in-box", shown in Figure 3, all estimators converge to the correct landscape with high empowerment values in the middle and low values near the boundaries, which correspond to previous results for this environment (Klyubin et al.; Maximilian et al.
Revised: , but we also compare with recent works that employ empowerment-like objectives for skill discovery (Eysenbach et al. ; Sharma et al. From Figure 4, we observe that in simple environments such as ball-in-box, all estimators converge to the correct landscape with high empowerment values in the middle and low values near the boundaries, corresponding to previous work (Klyubin et al.

Original: For the pendulum environment, it is clear that while both  method (b) and ours succeed in creating results similar to the analytical solution in 6, ours outputs an empowerment landscape that is smoother and more discerning.
Revised: For the pendulum environment, it is clear that while both the method by Karl et al. and ours succeed in creating results similar to the analytical solution (the ground truth), ours outputs an empowerment landscape that is smoother and more discerning.

Original: [Figure] Figure 3 shows that the proposed algorithm is superior to the existing sample-based methods with regard to a lower variance in training and its convergence to empowerment.
Revised: Detailed derivation of the analytical solution is included in Appendix A.

Original: 5. T WO PHASES IN EMPOWERMENT - BASED STABILIZATION
Revised: 5. S TABILITY OF E MPOWERMENT E STIMATIONS

Original: Empowerment of the pendulum is highest when it is stationary at the up-right position.
Revised: Besides the convergence to correct empowerment values, the stability and sample efﬁciency are also key aspects to determine whether the empowerment estimators are suitable as reliable intrinsic reward signals.

Original: In this section, we show the evolution of the empowerment landscape, the corresponding control policy and the state distribution as we run Algorithm 1 without any external reward.
Revised: Speciﬁcally, we quantify the variance of average empowerment values overseeds. Since the outputs by existing methods are on different scales, we instead compute the relative standard deviation.

Original: In Figure 4, we observe that Algorithm 1 includes two phases: empowerment learning and policy learning. Empowerment converges from epoch 0 to 10 and then, policy converges from epoch 10 to 50.
Revised: We show the learning curves in Figure 5. Our method outperforms the existing sample-based methods in that it has lower variance in training and faster convergence to the ﬁnal estimator. These properties make our method competent as a reliable unsupervised training signal.

Original: [Figure]
Revised: [Figure]

Original: 5. U NSUPERVISED STABILIZATION BY EMPOWERMENT FROM IMAGES
Revised: 5. E MPOWERMENT E STIMATION IN H IGH - DIMENSIONAL S YSTEMS

Original: In this section we demonstrate for the ﬁrst time unsupervised stabilization by empowerment from images. Simply put, no prior work succeeded to estimate the empowerment function reliably from images for swinging up the pendulum and stabilizing it at the upright position, which has been demonstrated before only by theanalytical empowerment calculation
Revised: In this section, we verify the applicability of Latent-GCE in high-dimensional systems from two aspects: ﬁrst, the learned latent space should be invertible - able to predict the true underlying dynamics; moreover, the empowerment maximization policy should develop stabilization behaviors. 5.4. P IXEL - BASED P ENDULUM We use a pendulum environment that supplies 64 × 64 images as observations.

Original: Salge et al.
Revised: As input to our

Original:  Figure 5 shows the empowerment landscape computed from images by our approach, and the optimal trajectory for swinging up the pendulum from the bottom position to the upright position.
Revised: Latent-GCE algorithm, we concatenate two consecutive frames so as to capture the angular velocity. In Figure 6, we start the pendulum at its up-right position s 0 . Using the encoder f χ , we map the images into the latent representation f χ ( s 0 ) . Given different action sequences a H 0 , we predict the ending latent state as z t + H

Original: The x-axis is θ (angle) and the y-axis is ˙ θ (angular velocity).
Revised: = G χ ( f χ ( s t )) · g χ ( a Ht ) .

Original: The black curve is  the optimal trajectory for swinging up the pendulum, which starts fromthe bottom with θ = π rad and ˙ θ = 0 rad .
Revised: If our learned Latent-GCE model captures the true system dynamics, we expect the reconstruction of the ending state f − 1 χ ( z t + H ) to match the ground truth s t + H . Figure 6 veriﬁes that the encoders and latent space learned by our Latent-GCE algorithm indeed captures the underlying dynamics of the pendulum. More importantly, as shown in Figure 7, the empowerment estimations given by our method is not only close to the analytical solution, but also trains a policy that swing the pendulum up from bottom to balance at the top.

Original: At the beginning the pendulum swings with a [-small amplitude (the dense black dots around θ = π rad , ˙ θ = 
-]
Revised: On the other hand, VIM by Mohamed & Rezende (2015) which we compare as a [-small amplitude (the dense black dots around θ = π rad , ˙ θ = 
-]baseline, fails to assign a high empowerment estimation for the up-right position. 5.4. H OPPER We evaluate the ability of Latent-GCE to stabilize the MuJoCo Hopper environment from OpenAI Gym.

Original: rad s − 1 ), then, once accumulated enough energy, it starts traversing the state space, and arrives at the top position, where it stabilises, (dense black points around θ = 0 rad , ˙ θ = 0 rad s − 1 ).
Revised: This environment is more challenging because of higher state dimension and long horizon (64) for empowerment estimation. Here we expect an empowerment maximization policy to balance the hopper at its standing position where it is able to move left and right freely. In Figure 8 we see our method successfully balances hopper whereas a random policy inevitably makes the hopper fall over. Our method stabilizes the hopper to a standing position ( z ≈ 1 . 1 ).

Original:  S UMMARY AND F UTURE W ORK
Revised:  S UMMARY AND D ISCUSSION

Original: In this work we introduce a new sample-based method for empowerment estimation in dynamical systems.
Revised: In this work we introduce a new sample-based method for empowerment estimation in dynamical systems.

Original: The key component of our method is the representation of dynamical system by the Gaussian channel, which allows to estimate empowerment by convex optimization.
Revised: The key component of our method is the representation of dynamical system by the Gaussian channel, which allows to estimate empowerment by convex optimization.

Original: Our estimator converges to true empowerment landscape both from raw states and images.
Revised: Our estimator converges to true empowerment landscape both from raw states and images.

Original: Empowerment is used in various applications. Our improved estimator has a potential to improve these applications.
Revised: Empowerment is used in various applications and our improved estimator has a potential to improve these applications.

Revised: For example in Appendix F, we show that when the agent is trained with an empowerment-aware objective, it develops distinguishable levels of attention to safety.

Original: Speciﬁcally, the natural continuation of this work is to learn skills using our representation of empowerment by Gaussian channel, rather than by VLB.
Revised: Moreover, a natural continuation of this work is to learn skills using our representation of empowerment by Gaussian channel, rather than by VLB.

Original: A A NALYTICAL DERIVATION OF G ( s ) FOR INVERTED PENDULUM
Revised: A A NALYTICAL DERIVATION OF G ( s ) FOR INVERTED PENDULUM

Original: The main steps in the derivation of G ( s ) appear below.
Revised: The main steps in the derivation of G ( s ) appear below.

Original: The full derivation appears in Section 1 at
Revised: The full derivation appears in Section 4.1 at

Original: Salge et al.
Revised: Salge et al.

Original: The current state of the pendulum, s t , is given by θ t and ˙ θ t :
Revised: The current state of the pendulum, s t , is given by θ t and ˙ θ t :

Original: [Equation]
Revised: [Equation]

Original: where θ is measured from the upright position.
Revised: where θ is measured from the upright position.

Original: We consider the development of the pendulum for three time steps of duration dt .
Revised: We consider the development of the pendulum for three time steps of duration dt .

Original: The behaviour of the pendulum can be approximate linearly at every time step for small enough dt .
Revised: The behaviour of the pendulum can be approximate linearly at every time step for small enough dt .

Original: Then the next state of the pendulum is given by:
Revised: Then, the next state of the pendulum is given by:

Original: [Equation]
Revised: [Equation]

Original: (8) where a t is the agent’s action, A t = (cid:20) ˙
Revised: where a t is the agent’s action, A t = (cid:20) ˙

Original: [Equation]

Original:  gl sin( θ t ) (cid:21) , and B t = (cid:20) 01 (cid:21) .
Revised: θ t gl sin( θ t ) (cid:21) , and B t = (cid:20) 01 (cid:21) .

Original: For the next time step we need to calculate a new matrix, A , because the non-linearity of the system causes A t + dt to depend on the new state.
Revised: For the next time step we need to calculate a new matrix, A , because the non-linearity of the system causes A t + dt to depend on the new state.

Original: Consequently, A t + dt = (cid:20) ˙
Revised: Consequently, A t + dt = (cid:20) ˙

Original: θ t + dt gl sin( θ t + dt ) (cid:21) .
Revised: θ t + dt gl sin( θ t + dt ) (cid:21) .

Original: A similar matrix can be calculated for A t +2 dt .
Revised: A similar matrix can be calculated for A t +2 dt .

Original: An propagation of pendulum dynamics for 3 time steps allows us to calculate s t +3 dt explicitly, which appears as:
Revised: An propagation of pendulum dynamics for 3 time steps allows us to calculate s t +3 dt explicitly, which appears as:

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: where a t + dt is the action at the next step.
Revised: where a t + dt is the action at the next step.

Original: Our goal is to represent s t +3 dt by:
Revised: Our goal is to represent s t +3 dt by:

Original: [Equation]
Revised: [Equation]

Original: which is achieved by linearisation θ t +3 dt and ˙ θ t +3 dt around a t = 0 .
Revised: which is achieved by linearisation θ t +3 dt and ˙ θ t +3 dt around a t = 0 .

Original: A straightforward calculation shows that the matrix G ( s t ) is given by:
Revised: A straightforward calculation shows that the matrix G ( s t ) is given by:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This calculation can be continued beyond 3-steps, involving more complicated terms in the expansions.
Revised: This calculation can be continued beyond 3-steps, involving more complicated terms in the expansions.

Original: Given G ( s t ) in equation 10 one can represent s t +3 dt by equation 9, where the matrix K does not affect the mutual information Cover & Thomas (2012).
Revised: Given G ( s t ) in Equation 10 one can represent s t +3 dt by Equation 9, where the matrix K does not affect the mutual information Cover & Thomas (2012).

Original: Adding to equation 9 the normal noise, η , as suggested in Salge et al.
Revised: Adding to Equation 9 the normal noise, η , as suggested in Salge et al.

Original: , we get the representation in equation 2.
Revised: , we get the representation in Equation 4.

Original: In contrast, we propose to learn G ( s ) from samples via interaction with the environment for an arbitrary dynamics and an arbitrary number of steps (c.f. 3 steps in the above derivation).
Revised: In contrast, we propose to learn G ( s ) from samples via interaction with the environment for an arbitrary dynamics and an arbitrary number of steps (c.f. 3 steps in the above derivation).

Original: Using G ( s ) from equation 10, we can calculate the analytical (ground truth empowerment landscape), which is shown at Figure
Revised: Using G ( s ) from Equation 10, we can calculate the analytical (ground truth empowerment landscape), which is shown at Figure

Original: B G AUSSIAN C HANNEL IN LATENT SPACE
Revised: B G AUSSIAN C HANNEL C APACITY

Original: To learn the empowerment function from pixels, we construct theGaussian linear channel equationin the latent space using an encoder.
Revised: The capacity of the channel in Equation 4 is given by, (Cover & Thomas (2012)): [Equation] where σ i ( s ) , p i , k , and P , are the i-th singular value of G ( s ) , the corresponding actuator signal power, the number of nonzero singular values, and the total power of the actuator signal, respectively. The optimal solution to Equation 11 is given by:

Revised: [Equation]

Original: To verify the correctness of our latent dynamics model, we compare reconstruction of latent predictions with ground truth observations.
Revised: where ν is found by a line search (Cover & Thomas (2012)) to satisfy the total power constraint. C SVD A LTERNATIVE The singular values, σ i ( s t ) , can be estimated directly without the singular values decomposition by representing the channel matrix as G χ ( s t ) [Equation]

Revised: ∗

Original: In inverted pendulum, we use two consecutive images to capture the state information (angle and angular velocity).
Revised: V χ ( s t ) with two additional orthonormality constraints: (cid:13)(cid:13) U χ [Equation] s t ) U † χ

Revised: [Equation]

Revised: s t ) − I { d s × d s } (cid:13)(cid:13) 2 2 and (cid:13) (cid:13) V χ

Revised: ( s t ) V † χ ( s t ) − I { d a ( H − 1) ×

Original: As shown inFigure 7, our model is capable of reconstructing future images from current images and action sequences.
Revised: d a ( H − 1) } (cid:13) (cid:13) 2 2 . In our experiments, we did not experience a bottleneck in calculating the singular values decomposition for both raw states and images. While we don’t implement this in our experiments, this alternative direct calculation can be applied when SVD is undesired. D T WO PHASES IN EMPOWERMENT - BASED STABILIZATION

Revised: Empowerment of the pendulum is highest when it is stationary at the up-right position.

Revised: In this section, we show the evolution of the empowerment landscape, the corresponding control policy and the state distribution as we run Algorithm 1 without any external reward.

Revised: In Figure 10, we observe that Algorithm 1 includes two phases: empowerment learning and policy learning.

Revised: Empowerment converges from epoch 0 to 10 and then, policy converges from epoch 10 to 50.

Original: C C ART -P OLE S WING - UP
Revised: E C ART -P OLE S WING - UP

Original: The cart-pole environment is made up of a stiff pole with one end ﬁxed to a pivot on a cart movable along one axis.
Revised: The cart-pole environment is made up of a stiff pole with one end ﬁxed to a pivot on a cart movable along one axis.

Original: The cart can be pushed left or right within a conﬁned region.
Revised: The cart can be pushed left or right within a conﬁned region.

Original: The state space of  cart-pole has 4 components: cart position, cart velocity, pole angle, pole angular velocity.
Revised: The state space of epochepoch- epochepoch- epochepoch- cart-pole has 4 components: cart position, cart velocity, pole angle, pole angular velocity.

Original: We expect high empowerment when the pole is up right, because starting there, gravity is able to help the system reach a broader set of future states.
Revised: We expect high empowerment when the pole is up right, because starting there, gravity is able to help the system reach a broader set of future states.

Original: To measure whether Algorithm 1 successfully brings the cart-pole to up-right position, we plot the average squared angle to the top against the number of training steps.
Revised: To measure whether Algorithm 1 successfully brings the cart-pole to up-right position, we plot the average squared angle to the top against the number of training steps.

Original: We also include 2 different reward functions that directly aim this objective for comparison.
Revised: We also include 2 different reward functions that directly aim this objective for comparison.

Original: As shown in Figure 8, the empowerment reward, without the need of prior knowledge of the environment, is able to achieve similar ﬁnal performance compared to direct reward signals.
Revised: As shown in Figure 11, the empowerment reward, without the need of prior knowledge of the environment, is able to achieve similar ﬁnal performance compared to direct reward signals.

Original: Moreover, it learns this objective at a speed on par with the dense reward.
Revised: Moreover, it learns this objective at a speed on par with the dense reward.

Original: D E MPOWERMENT E STIMATION FOR S AFETY
Revised: F E MPOWERMENT E STIMATION FOR S AFETY

Original: A useful application of empowerment is its implication of safety for the artiﬁcial agent.
Revised: A useful application of empowerment is its implication of safety for the artiﬁcial agent.

Original: A state is intrinsically safe for an agent when the agent has a high diversity of future states, achievable by its actions.
Revised: A state is intrinsically safe for an agent when the agent has a high diversity of future states, achievable by its actions.

Original: This is because in such states, the agent can take effective actions to prevent undesirable futures.
Revised: This is because in such states, the agent can take effective actions to prevent undesirable futures.

Original: In this context, the higher its empowerment value, the safer the agent is.
Revised: In this context, the higher its empowerment value, the safer the agent is.

Original: In this experiment, we ﬁrst check that our calculation of empowerment matches the speciﬁc design of the environment.
Revised: In this experiment, we ﬁrst check that our calculation of empowerment matches the speciﬁc design of the environment.

Original: Additionally, we show that empowerment augmented reward function can affect the agent’s preference between a shorter but more dangerous path and a longer but safer one.
Revised: Additionally, we show that empowerment augmented reward function can affect the agent’s preference between a shorter but more dangerous path and a longer but safer one.

Original: Environment: a double tunnel environment implemented with to the OpenAI Gym API (Brockman et al., 2016).
Revised: Environment: a double tunnel environment implemented with to the OpenAI Gym API (Brockman et al., 2016).

Original: Agent (blue) is modeled as a ball of radius 1 inside a 20 × 20 box.
Revised: Agent (blue) is modeled as a ball of radius 1 inside a 20 × 20 box.

Original: The box is separated by a thick wall (gray) into top and bottom section the top and bottom of the box
Revised: The box is separated by a thick wall (gray) into top and bottom section.

Original: The tunnel in middle is narrower but closer the goal the one on
Revised: Two tunnels connect the top The

Original: Control:
Revised: Control:

Original: In each time step, the agent can move at most 0.5 unit length in each of x and y direction.
Revised: In each time step, the agent can move at most 0.5 unit length in each of x and y direction.

Original: If an action takes the agent into the walls, it will be shifted back out to the nearest surface.
Revised: If an action takes the agent into the walls, it will be shifted back out to the nearest surface.

Original: Reset criterion: each episode has a maximum length of 200 steps.
Revised: Reset criterion: each episode has a maximum length of 200 steps.

Original: The environment resets when time runs out or when the agent reaches the goal.
Revised: The environment resets when time runs out or when the agent reaches the goal.

Original: Since the tunnel in the middle is narrower, the agent is relatively less safe there.
Revised: Since the tunnel in the middle is narrower, the agent is relatively less safe there.

Original: The effectiveness of the control of the agent is damped in 2 ways:
Revised: The effectiveness of the control of the agent is damped in 2 ways:

Original: In a particular time step, it’s more likely for the agent to bump into the walls.
Revised: In a particular time step, it’s more likely for the agent to bump into the walls

Original: When this happens, the agent is unable to proceed as far as desired.
Revised: When this happens, the agent is unable to proceed as far as desired

Original: The tunnel is 10 units in length.
Revised: The tunnel is 10 units in length.

Original: When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon.
Revised: When the agent is around the middle, it will still be inside the tunnel in the 5-step horizon.

Original: Thus, the agent has fewer possible future states.
Revised: Thus, the agent has fewer possible future states.

Original: We trained the agent with PPO algorithm (Schulman et al., 2017) from OpenAI baselines (Dhariwal et al., 2017) using an empowerment augmented reward function.
Revised: We trained the agent with PPO algorithm (Schulman et al., 2017) from OpenAI baselines (Dhariwal et al., 2017) using an empowerment augmented reward function.

Original: After a parameter search, we used discount factor γ = 0 .
Revised: After a parameter search, we used discount factor γ = 0 .

Original: 95 over a total of 10 6 steps.
Revised: 95 over a total of 10 6 steps.

Original: The reward function that we choose is:
Revised: The reward function that we choose is:

Original: [Equation]
Revised: [Equation]

Original: where β balances the relative weight between the goal conditioned reward and the intrinsic safety reward.
Revised: where β balances the relative weight between the goal conditioned reward and the intrinsic safety reward.

Original: With high β , we expect the agent to learn a more conservative policy.
Revised: With high β , we expect the agent to learn a more conservative policy.

Original: [Figure]

Original: The results from the tunnel environment again support our proposed scheme.
Revised: The results from the tunnel environment again support our proposed scheme.

Original: First, the empowerment landscape matches our design of the environment.
Revised: First, the empowerment landscape matches our design of the environment.

Original: Second, high quality empowerment reward successfully alters the behavior of the agent. 
Revised: Second, high quality empowerment reward successfully alters the behavior of the agent. [Figure]

Original: E E XPERIMENT D ETAILS E. D YNAMICS -A WARE D ISCOVERY OF S KILLS (DADS)
Revised: G E XPERIMENT D ETAILS G. U SE OF NEURAL ENCODERS

Revised: In our state-based classic control environments, the encoder/decoder pairs for observations and actions are set to be the identity map. In this case, the sequences of actions are simply concatenated. For the Hopper environment and pixel-based pendulum, encoders and decoders are parameterized by neural networks.

Revised: G. D ETAILS ON NEURAL NETWORK LAYERS G.2. CNN FOR IMAGE ENCODING

Revised: (h1) 2D convolution: 4 ﬁlters, stride 2, 32 channels, ReLU activation (h2) 2D convolution: 4 ﬁlters, stride 2, 64 channels, ReLU activation (h3) 2D convolution: 4 ﬁlters, stride 2, 128 channel, ReLU activations (h4) 2D convolution: 4 ﬁlters, stride 2, 256 channel, ReLU activations (out) Flatten each sample to a 1D tensor

Revised: G.2. D ECONVOLUTIONAL NET FOR IMAGE RECONSTRUCTION

Revised: (h1) Fully connected layer with 1024 neurons, ReLU activation (h1’)

Revised: Reshape to 1 × 1 images with 1024 channels (h2) 2D conv-transpose: 5 ﬁlters, stride 2, 128 channels, ReLU activation (h3) 2D convolution: 5 ﬁlters, stride 2, 64 channels, ReLU activation (h4) 2D convolution: 6 ﬁlters, stride 2, 32 channel, ReLU activations (out) 2D convolution: 6 ﬁlters, stride 2, C channel

Revised: G.2. MLP FOR ACTION SEQUENCE ENCODING

Revised: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h3) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with 32 output neurons

Revised: G.2. MLP FOR ACTION SEQUENCE RECONSTRUCTION

Revised: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with T × d a neurons, tanh activation then scaled to action space

Revised: G.2. MLP FOR TRANSITION MATRIX A

Revised: (h1) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (h2) Fully connected layer with 512 neurons, ReLU activation (out) Fully conected layer with d f × d g neurons

Revised: G. D YNAMICS -A WARE D ISCOVERY OF S KILLS (DADS)

Original: For our experiments using DADS, we use policy and discriminator architecture of two hidden layers of size 256 with ReLU activations and a latent dimension of 2.
Revised: For our experiments using DADS, we use policy and discriminator architecture of two hidden layers of size 256 with ReLU activations and a latent dimension of 2.

Original: Our discount factor is .
Revised: Our discount factor is .

Original: As per suggestions from the original paper, we use 512 prior samples to approximate the intrinsic reward, run 32 discriminator updates and 128 policy updates per 1000 timesteps; learning rates of 3 × 10 −and batch size of 256 are used throughout.
Revised: As per suggestions from the original paper, we use 512 prior samples to approximate the intrinsic reward, run 32 discriminator updates and 128 policy updates per 1000 timesteps; learning rates of 3 × 10 −and batch size of 256 are used throughout.

Original: We use a replay buffer to help stabilize training of size 20000, used only for policy updates.
Revised: We use a replay buffer to help stabilize training of size 20000, used only for policy updates.