Original: L EARNING THE A RROW OF T IME FOR P ROBLEMS IN R EINFORCEMENT L EARNING
Revised: L EARNING THE A RROW OF T IME FOR P ROBLEMS IN R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We humans have an innate understanding of the asymmetric progression of time, which we use to efﬁciently and safely perceive and manipulate our environment.
Revised: We humans have an innate understanding of the asymmetric progression of time, which we use to efﬁciently and safely perceive and manipulate our environment.

Original: Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.
Revised: Drawing inspiration from that, we approach the problem of learning an arrow of time in a Markov (Decision) Process.

Original: We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.
Revised: We illustrate how a learned arrow of time can capture salient information about the environment, which in turn can be used to measure reachability, detect side-effects and to obtain an intrinsic reward signal.

Original: Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).
Revised: Finally, we propose a simple yet effective algorithm to parameterize the problem at hand and learn an arrow of time with a function approximator (here, a deep neural network).

Original: Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer, and Otto (1998).
Revised: Our empirical results span a selection of discrete and continuous environments, and demonstrate for a class of stochastic processes that the learned arrow of time agrees reasonably well with a well known notion of an arrow of time due to Jordan, Kinderlehrer, and Otto (1998).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The asymmetric progression of time has a profound effect on how we, as agents, perceive, process and manipulate our environment.
Revised: The asymmetric progression of time has a profound effect on how we, as agents, perceive, process and manipulate our environment.

Original: Given a sequence of observations of our familiar surroundings (e.g. as video frames), we possess the innate ability to predict whether the said observations are ordered correctly .
Revised: Given a sequence of observations of our familiar surroundings (e.g. as video frames), we possess the innate ability to predict whether the said observations are ordered correctly .

Original: We use this ability not just to perceive, but also to act: for instance, we know to be cautious about dropping a vase, guided by the intuition that the act of breaking a vase cannot be undone.
Revised: We use this ability not just to perceive, but also to act: for instance, we know to be cautious about dropping a vase, guided by the intuition that the act of breaking a vase cannot be undone.

Original: This profound intuition reﬂects some fundamental properties of the world in which we dwell, and in this work we ask whether and how these properties can be exploited to learn a representation that functionally mimics our understanding of the asymmetric nature of time.
Revised: This profound intuition reﬂects some fundamental properties of the world in which we dwell, and in this work we ask whether and how these properties can be exploited to learn a representation that functionally mimics our understanding of the asymmetric nature of time.

Original: The term Arrow of Time was coined by the British astronomer Eddington (1929) to denote this inherent asymmetry, which he attributed to the non-decreasing nature of the total thermodynamic entropy of an isolated system, as required by the second law of thermodynamics.
Revised: The term Arrow of Time was coined by the British astronomer Eddington (1929) to denote this inherent asymmetry, which he attributed to the non-decreasing nature of the total thermodynamic entropy of an isolated system, as required by the second law of thermodynamics.

Original: Since then, the notion of an arrow of time has been formalized and explored in various contexts, spanning not only physics, but also algorithmic information theory (Zurek, 1989), causal inference (Janzing et al., 2016) and time-series analysis (Janzing, 2010; Bauer et al., 2016).
Revised: Since then, the notion of an arrow of time has been formalized and explored in various contexts, spanning not only physics, but also algorithmic information theory (Zurek, 1989), causal inference (Janzing et al., 2016) and time-series analysis (Janzing, 2010; Bauer et al., 2016).

Original: Broadly, an arrow of time can be thought of as a function that monotonously increases as a system evolves in time.
Revised: Broadly, an arrow of time can be thought of as a function that monotonously increases as a system evolves in time.

Original: Expectedly, the notion of irreversibility plays a central role in the discourse.
Revised: Expectedly, the notion of irreversibility plays a central role in the discourse.

Original: In statistical physics, it is posited that the arrow of time (i.e. entropy production) is driven by irreversible processes (Prigogine, 1978; Seifert, 2012).
Revised: In statistical physics, it is posited that the arrow of time (i.e. entropy production) is driven by irreversible processes (Prigogine, 1978; Seifert, 2012).

Original: To understand how a notion of an arrow of time can be useful in the reinforcement learning context, consider the example of a cleaning robot tasked with moving a box across a room (Amodei et al., 2016).
Revised: To understand how a notion of an arrow of time can be useful in the reinforcement learning context, consider the example of a cleaning robot tasked with moving a box across a room (Amodei et al., 2016).

Original: The optimal way of successfully completing the task might involve the robot doing something disruptive, like knocking a vase over (Fig 1).
Revised: The optimal way of successfully completing the task might involve the robot doing something disruptive, like knocking a vase over (Fig 1).

Original: Now on the one hand, such disruptions – or side-effects – might be difﬁcult to recover from.
Revised: Now on the one hand, such disruptions – or side-effects – might be difﬁcult to recover from.

Original: In the extreme case, they might be virtually irreversible – say when the vase is broken.
Revised: In the extreme case, they might be virtually irreversible – say when the vase is broken.

Original: On the other hand, irreversibility implies that states with a larger number of broken vases tend to occur in the future, and one should therefore expect an arrow of time (as a scalar function of the state) to assign larger values to states with larger number of broken vases.
Revised: On the other hand, irreversibility implies that states with a larger number of broken vases tend to occur in the future, and one should therefore expect an arrow of time (as a scalar function of the state) to assign larger values to states with larger number of broken vases.

Original: An arrow of time should therefore quantify the amount of disorder in the environment, analogous to the entropy for isolated thermodynamical systems.
Revised: An arrow of time should therefore quantify the amount of disorder in the environment, analogous to the entropy for isolated thermodynamical systems.

Original: One application could be to detect and preempt such side-effects, for instance by penalizing policies that signiﬁcantly increment the arrow of time by executing difﬁcult-to-reverse transitions.
Revised: Now, one possible application could be to detect and preempt such side-effects, for instance by penalizing policies that signiﬁcantly increment the arrow of time by executing difﬁcult-to-reverse transitions.

Original: But the utility of an arrow of time is more general: it serves as a directed measure ofreachability. This can
Revised: But the utility of an arrow of time is more general: it serves as a directed measure of

Original: D i s o r d e r
Revised: D i s o r d e r

Original: Time D
Revised: Time D

Original: i s o r d e r
Revised: i s o r d e r

Original: Time
Revised: Time

Original: D i s o r d e r
Revised: D i s o r d e r

Original: Time
Revised: Time

Original: D i s o r d e r
Revised: D i s o r d e r

Original: Time  be seen by observing that it is more difﬁcult to obtain order from disorder : it is, after all, difﬁcult to reach a state with a vase intact from one with it broken, rather than vice versa.
Revised: Time reachability. This can be seen by observing that it is more difﬁcult to obtain order from disorder : it is, after all, difﬁcult to reach a state with a vase intact from one with it broken, rather than vice versa.

Original: In this sense, we may say that a state is relatively unreachable from another state if an arrow of time assigns a lower value to the former.
Revised: In this sense, we may say that a state is relatively unreachable from another state if an arrow of time assigns a lower value to the former.

Original: Further, a directed measure of reachability afforded by an arrow of time can be utilized for deriving an intrinsic reward signal to enable agents to learn complex skills in the absence of external rewards.
Revised: Further, a directed measure of reachability afforded by an arrow of time can be utilized for deriving an intrinsic reward signal to enable agents to learn complex skills in the absence of external rewards.

Original: To see how, consider that an agent tasked with reversing the arrow of time (by creating order from disorder ) must in general learn complex skills to achieve its goal.
Revised: To see how, consider that an agent tasked with reversing the arrow of time (by creating order from disorder ) must in general learn complex skills to achieve its goal.

Original: Indeed, gluing together a broken vase will require the agent to learn an array of complex planning and motor skills, which is the ultimate goal of such intrinsic rewards.
Revised: Indeed, gluing together a broken vase will require the agent to learn an array of complex planning and motor skills, which is the ultimate goal of such intrinsic rewards.

Original: In summary, our contributions are the following.
Revised: In summary, our contributions are the following.

Original: (a) We propose a simple objective to learn an arrow of time for a Markov (Decision) Process in a self-supervised manner, i.e. entirely from sampled environment trajectories and without external rewards.
Revised: (a) We propose a simple objective to learn an arrow of time for a Markov (Decision) Process in a self-supervised manner, i.e. entirely from sampled environment trajectories and without external rewards.

Original: We call the resulting function (acting on the state)
Revised: We call the resulting function (acting on the state)

Original: the h -potential, and demonstrate its utility and caveats for a selection of discrete and continuous environments.
Revised: the h -potential, and demonstrate its utility and caveats for a selection of discrete and continuous environments.

Original: Moreover, we compare the learned h -potential to the free-energy functional of stochastic processes – the latter being a well-known notion of an arrow of time (Jordan et al., 1998).
Revised: Moreover, we compare the learned h -potential to the free-energy functional of stochastic processes – the latter being a well-known notion of an arrow of time (Jordan et al., 1998).

Original: While there exist prior work on detecting the arrow of time in videos (Pickup et al., 2014; Wei et al., 2018) and time-series data (Peters et al., 2009; Bauer et al., 2016), we believe our work to be the ﬁrst towards measuring it in the context of reinforcement learning.
Revised: While there exist prior work on detecting the arrow of time in videos (Pickup et al., 2014; Wei et al., 2018) and time-series data (Peters et al., 2009; Bauer et al., 2016), we believe our work to be the ﬁrst towards measuring it in the context of reinforcement learning.

Original: (b) We critically and transparently discuss the conceptually rich subtleties that arise before an arrow of time can be practically useful in the RL context.
Revised: (b) We critically and transparently discuss the conceptually rich subtleties that arise before an arrow of time can be practically useful in the RL context.

Original: (c) We expose how the notions of reachability, safety and curiosity can be uniﬁed under the common framework afforded by a learned arrow of time.
Revised: (c) We expose how the notions of reachability, safety and curiosity can be uniﬁed under the common framework afforded by a learned arrow of time.

Original:  T HE h -P OTENTIAL
Revised:  T HE h -P OTENTIAL

Original: Motivated by the preceding discussion, our goal is to learn a function that quantiﬁes the amount of disorder in a given environment state, where we say that irreversible state transitions increase disorder .
Revised: Motivated by the preceding discussion, our goal is to learn a function that quantiﬁes the amount of disorder in a given environment state, where we say that irreversible state transitions increase disorder .

Original: In this sense, we seek a function (of the state) that is constant in expectation along fully reversible state transitions, but increase in expectation along state transitions that are less reversible .
Revised: In this sense, we seek a function (of the state) that is constant in expectation along fully reversible state transitions, but increase in expectation along state transitions that are less reversible .

Original: To that end, we begin by formally introducing this function, which we call the h -potential, as the solution to a functional optimization problem.
Revised: To that end, we begin by formally introducing this function, which we call the h -potential, as the solution to a functional optimization problem.

Original: Subsequently, we critically discuss a few conceptual roadblocks that must be cleared before such a function can be useful in the RL setting.
Revised: Subsequently, we critically discuss a few conceptual roadblocks that must be cleared before such a function can be useful in the RL setting.

Original: 2. F ORMALISM
Revised: 2. F ORMALISM

Original: Consider a Markov Decision Process (a MDP, i.e. environment ), and let S and A be its state and action spaces respectively.
Revised: Consider a Markov Decision Process (a MDP, i.e. environment ), and let S and A be its state and action spaces respectively.

Original: A policy π is a mapping from the state space to the space of distributions over actions.
Revised: A policy π is a mapping from the state space to the space of distributions over actions.

Original: Given a state s ∈ S sampled from some initial state distribution p 0 , we may sample an action a ∈ A from the policy π ( a | s ) , which in turn can be used to sample another state s (cid:48) ∈ S from the environment dynamics p ( s (cid:48) | a, s ) .
Revised: Given a state s ∈ S sampled from some initial state distribution p 0 , we may sample an action a ∈ A from the policy π ( a | s ) , which in turn can be used to sample another state s (cid:48) ∈ S from the environment dynamics p ( s (cid:48) | a, s ) .

Original: Iterating N more times for a ﬁxed π , one obtains a sequence of states ( s 0 , ..., s t , ..., s N ) , which is a realization of the Markov chain (a trajectory ) with transition probabilities p π ( s t +1 | s t )
Revised: Iterating N more times for a ﬁxed π , one obtains a sequence of states ( s 0 , ..., s t , ..., s N ) , which is a realization of the Markov chain (a trajectory ) with transition probabilities p π ( s t +1 | s t )

Original: =
Revised: =

Original: (cid:80) a ∈A p ( s t +1 | s t , a ) π ( a | s t ) .
Revised: (cid:80) a ∈A p ( s t +1 | s t , a ) π ( a | s t ) .

Original: We may now deﬁne a function h π : S → R as the solution to the following functional objective:
Revised: We may now deﬁne a function h π : S → R as the solution to the following functional objective:

Original: [Equation]
Revised: [Equation]

Original: where U ( A ) is the uniform distribution over any set A , E t E s t
Revised: where U ( A ) is the uniform distribution over any set A , E t E s t

Original: E s t +1 | s t is the expectation over all state transitions, λ is a scalar coefﬁcient and T [ˆ h ] is a regularizing term that prevents ˆ h from diverging within a ﬁnite domain.
Revised: E s t +1 | s t is the expectation over all state transitions, λ is a scalar coefﬁcient and T [ˆ h ] is a regularizing term that prevents ˆ h from diverging within a ﬁnite domain.

Original: In words: the ﬁrst term on the right hand side of the ﬁrst equation above encourages h π to increase in expectation along the sampled trajectories, whereas the second term controls this increase; the two terms are balanced with a coefﬁcient λ .
Revised: In words: the ﬁrst term on the right hand side of the ﬁrst equation above encourages h π to increase in expectation along the sampled trajectories, whereas the second term controls this increase; the two terms are balanced with a coefﬁcient λ .

Original: Informally: if a state transition s → s (cid:48) is fully reversible , the probability of sampling it equals that of sampling the corresponding reverse transition, s (cid:48)
Revised: Informally: if a state transition s → s (cid:48) is fully reversible , the probability of sampling it equals that of sampling the corresponding reverse transition, s (cid:48)

Original: → s .
Revised: → s .

Original: For such transitions, the pressure on h π to increase along the forward transition ( s → s (cid:48) ) is compensated by the counter-pressure for it to increase along the reverse transition ( s (cid:48)
Revised: For such transitions, the pressure on h π to increase along the forward transition ( s → s (cid:48) ) is compensated by the counter-pressure for it to increase along the reverse transition ( s (cid:48)

Original: → s ), or equivalently, decrease along the forward transition.
Revised: → s ), or equivalently, decrease along the forward transition.

Original: Along such transitions, we should therefore expect h π to remain constant (in expectation).
Revised: Along such transitions, we should therefore expect h π to remain constant (in expectation).

Original: Accordingly, if the forward transition were to be more likely (i.e. if the transition is not fully reversible), we should expect h π to increase (in expectation) in order to satisfy its objective.
Revised: Accordingly, if the forward transition were to be more likely (i.e. if the transition is not fully reversible), we should expect h π to increase (in expectation) in order to satisfy its objective.

Original: The regularizer T must be chosen to suit the problem at hand, and different choices result in solutions that have different characteristics 1 .
Revised: The regularizer T must be chosen to suit the problem at hand, and different choices result in solutions that have different characteristics 1 .

Original: Possible choices for T include (any combination of) the negative of L 2 norm −(cid:107) ˆ h (cid:107) 2 , and/or the following trajectory regularizer:
Revised: Possible choices for T include (any combination of) the negative of L 2 norm −(cid:107) ˆ h (cid:107) 2 , and/or the following trajectory regularizer:

Original: [Equation]
Revised: [Equation]

Original: Intuitively: while the solution h π is required to increase in expectation along trajectories, the trajectory regularizer acts as an contrastive term by penalizing h π for changing at all.
Revised: Intuitively: while the solution h π is required to increase in expectation along trajectories, the trajectory regularizer acts as an contrastive term by penalizing h π for changing at all.

Original: With some effort, the problem deﬁned in Eqn 1 can be approached analytically for toy Markov chains (interested readers may refer to App A for a technical discussion).
Revised: With some effort, the problem deﬁned in Eqn 1 can be approached analytically for toy Markov chains (interested readers may refer to App A for a technical discussion).

Original: However, such analytical treatment becomes infeasible for more complex and larger-scale environments with unknown transition probabilities.
Revised: However, such analytical treatment becomes infeasible for more complex and larger-scale environments with unknown transition probabilities.

Original: To tackle such environments, we will cast the functional optimization problem in Eqn 1 to an optimization problem over the parameters of a deep neural network and solve it for a variety of discrete and continuous environments.
Revised: To tackle such environments, we will cast the functional optimization problem in Eqn 1 to an optimization problem over the parameters of a deep neural network and solve it for a variety of discrete and continuous environments.

Original: 2. S UBTLETIES
Revised: 2. S UBTLETIES

Original: In this section, we discuss two conceptually rich subtleties that determine the conditions under which the learned arrow of time ( h -potential) can be useful in practice.
Revised: In this section, we discuss two conceptually rich subtleties that determine the conditions under which the learned arrow of time ( h -potential) can be useful in practice.

Original: Maxwell’s Demon.
Revised: The Role of a Policy.

Original: The ﬁrst subtlety is rooted in the observation that the trajectories ( s 0 , ..., s N ) are collected by a given but arbitrary policy.
Revised: The ﬁrst subtlety is rooted in the observation that the trajectories ( s 0 , ..., s N ) are collected by a given but arbitrary policy.

Original: However, there may exist policies for which the resulting arrow of time is unnatural, perhaps even misleading.
Revised: However, there may exist policies for which the resulting arrow of time is unnatural, perhaps even misleading.

Original: Consider for instance the actions of a practitioner of Kintsugi, the ancient Japanese art of repairing broken pottery.
Revised: Consider for instance the actions of a practitioner of Kintsugi, the ancient Japanese art of repairing broken pottery.

Original: The corresponding policy, analogous to Maxwell’s demon in classical thermodynamics, might cause the environment to transition from a state where the vase is broken to one where it is not.
Revised: The corresponding policy 2 might cause the environment to transition from a state where the vase is broken to one where it is not.

Original: If we learn the h -potential on such trajectories, it might be the case that counter to our intuition, states with a larger number of broken vases are assigned smaller values (and the vice versa).
Revised: If we learn the h -potential on such trajectories, it might be the case that counter to our intuition, states with a larger number of broken vases are assigned smaller values (and the vice versa).

Original: Now, one may choose to resolve this conundrum by deﬁning:
Revised: Now, one may choose to resolve this conundrum by deﬁning:

Original: [Equation]
Revised: [Equation]

Original: where Π is the set of all policies deﬁned on S , and U (Π) denotes a uniform distribution over Π .
Revised: where Π is the set of all policies deﬁned on S , and U (Π) denotes a uniform distribution over Π .

Original: The resulting function h ∗ =
Revised: The resulting function h ∗ =

Original: arg max {J [ h ] + λ T [ h ] } would characterize the arrow of time with respect to all possible policies, and one would expect that for a vast majority of such policies, the transition from broken vase to a intact vase is rather unlikely and/or requires highly specialized policies.
Revised: arg max {J [ h ] + λ T [ h ] } would characterize the arrow of time with respect to all possible policies, and one would expect that for a vast majority of such policies, the transition from broken vase to a intact vase is rather unlikely and/or requires highly specialized policies.

Original: Unfortunately, determining h ∗ is not feasible for most interesting applications, given the outer expectation over all possible policies.
Revised: Unfortunately, determining h ∗ is not feasible for most interesting applications, given the outer expectation over all possible policies.

Original: As a compromise, we usea (uniformly) random policy to sample trajectories.
Revised: As a compromise, we use (uniformly) random actions to gather trajectories.

Original: The simplicity of the  random policy justiﬁes its adoption, since one would expect a policy resembling Maxwell’s demon to be rather complex and not implementable with random actions.
Revised: The simplicity of the corresponding random policy justiﬁes its adoption, since one would expect a policy resembling (say) a Kintsugi artist to be rather complex and not implementable with random actions.

Original: In this sense, we ensure that the learned arrow of time characterizes the underlying dynamics of the environment, and not the peculiarities of a particular agent 2 .
Revised: In this sense, we ensure that the learned arrow of time characterizes the underlying dynamics of the environment, and not the peculiarities of a particular agent 3 .

Original: The price we pay is the lack of adequate exploration in complex enough environments, although this problem plagues most model-based reinforcement learning approaches 3 (cf.
Revised: The price we pay is the lack of adequate exploration in complex enough environments, although this problem plagues most model-based reinforcement learning approaches 4 (cf.

Original: Ha & Schmidhuber (2018)).
Revised: Ha & Schmidhuber (2018)).

Original: In the following, wewill assumethat π is uniformly random and use h π interchangeably with h .
Revised: In the following, we assume π to be uniformly random and use h π interchangeably with h .

Original: Dissipative Environments.
Revised: Dissipative Environments.

Original: The second subtlety concerns what we require of environments in which the arrow of time is informative.
Revised: The second subtlety concerns what we require of environments in which the arrow of time is informative.

Original: To illustrate the matter, consider the class ofHamiltonian systems 4 , a typical instance of which could be a billiard ball moving on a frictionless arena and bouncing (elastically) off the edges (Bunimovich, 2007).
Revised: To illustrate the matter, consider the class of systems 5 , a typical instance of which could be a billiard ball moving on a frictionless arena and bouncing (elastically) off the edges (Bunimovich, 2007).

Original: The state space comprises the ball’s velocity and its position constrained to a billiard table (without holes!), where the ball is initialized at a random position on the table.
Revised: The state space comprises the ball’s velocity and its position constrained to a billiard table (without holes!), where the ball is initialized at a random position on the table.

Original: For such a system, it can be seen by time-reversal symmetry that when averaged over a large number of trajectories, the state transition s → s (cid:48) is just as likely as the reverse transition s (cid:48)
Revised: For such a system, it can be seen by time-reversal symmetry that when averaged over a large number of trajectories, the state transition s → s (cid:48) is just as likely as the reverse transition s (cid:48)

Original: → s .
Revised: → s .

Original: In this case, recall that the arrow of time is expected to remain constant.
Revised: In this case, recall that the arrow of time is expected to remain constant.

Original: A similar argument can be made for systems that identically follow closed trajectories in their respective state space (e.g. a frictionless and undriven pendulum).
Revised: A similar argument can be made for systems that identically follow closed trajectories in their respective state space (e.g. a frictionless and undriven pendulum).

Original: It follows that the h -potential must remain constant along the trajectory and that the arrow of time is uninformative.
Revised: It follows that the h -potential must remain constant along the trajectory and that the arrow of time is uninformative.

Original: However, for so-called dissipative systems, the notion of an arrow of time is pronounced and well studied (Willems, 1972; Prigogine, 1978).
Revised: However, for so-called dissipative systems, the notion of an arrow of time is pronounced and well studied (Willems, 1972; Prigogine, 1978).

Original: In MDPs, dissipative behaviour may arise in situations where certain transitions are irreversible by design (e.g. bricks disappearing in Atari Breakout), or due to partial observability, e.g. for a damped pendulum, the state space does not track the microscopic processes that give rise to friction.
Revised: In MDPs, dissipative behaviour may arise in situations where certain transitions are irreversible by design (e.g. bricks disappearing in Atari Breakout), or due to partial observability, e.g. for a damped pendulum, the state space does not track the microscopic processes that give rise to friction 6 .

Original: Therefore, a central premise underlying the practical utility of learning the arrow of time is that the considered MDP is indeed dissipative, which we shall assume in the following; in Sec(Fig 5b), we will empirically investigate the case where this assumption is violated.
Revised: Therefore, a central premise underlying the practical utility of learning the arrow of time is that the considered MDP is indeed dissipative, which we shall assume in the following; in Sec 5 (Fig 5b), we will empirically investigate the case where this assumption is violated.

Original:  A PPLICATIONS WITH R ELATED W ORK
Revised:  A PPLICATIONS WITH R ELATED W ORK

Original: In this section, we discuss a few applications of the arrow of time, and illustrate how the h -potential provides a common framework to unify the notions of reachability, safety and curiosity.
Revised: In this section, we discuss a few applications of the arrow of time, and illustrate how the h -potential provides a common framework to unify the notions of reachability, safety and curiosity.

Original: 3. M EASURING R EACHABILITY
Revised: 3. M EASURING R EACHABILITY

Original: Given two states s and s (cid:48) in S , the reachability of s (cid:48) from s measures how difﬁcult it is for an agent at state s to reach state s (cid:48) .
Revised: Given two states s and s (cid:48) in S , the reachability of s (cid:48) from s measures how difﬁcult it is for an agent at state s to reach state s (cid:48) .

Original: The prospect of learning reachability from state-transition trajectories has been explored: in Savinov et al.
Revised: The prospect of learning reachability from state-transition trajectories has been explored: in Savinov et al.

Original: (2018), the approach taken involves learning a logistic regressor network g θ : S × S → [0 , 1] to predict the probability of states s (cid:48) and s being reachable to one another within a certain number of steps (of a random policy), in which case g ( s, s (cid:48) )
Revised: (2018), the approach taken involves learning a logistic regressor network g θ : S × S → [0 , 1] to predict the probability of states s (cid:48) and s being reachable to one another within a certain number of steps (of a random policy), in which case g ( s, s (cid:48) )

Original: ≈ 1 .
Revised: ≈ 1 .

Original: However, the model g is not directed : it does not learn whether s (cid:48) is more likely to follow s , or the vice versa.
Revised: However, the model g is not directed : it does not learn whether s (cid:48) is more likely to follow s , or the vice versa.

Original: Instead, our proposal is to derive a directed measure of reachability from h -potential by deﬁning a function η : S ×S → R such that η ( s, s (cid:48) )
Revised: Instead, our proposal is to derive a directed measure of reachability from h -potential by deﬁning a function η : S ×S → R such that η ( s, s (cid:48) )

Original: ≡ η ( s → s (cid:48) ) :
Revised: ≡ η ( s → s (cid:48) ) :

Original: = h ( s (cid:48) )
Revised: = h ( s (cid:48) )

Original: − h
Revised: − h

Original: ( s ) , where η ( s → s (cid:48) ) measures the reachability of state s (cid:48) from state s .
Revised: ( s ) , where η ( s → s (cid:48) ) measures the reachability of state s (cid:48) from state s .

Original: This inductive bias on η (in form of a functional constraint) induces the following useful properties.
Revised: This inductive bias on η (in form of a functional constraint) induces the following useful properties.

Original: First , consider the case where the transition between states s and s (cid:48) is fully reversible, i.e. when state s is exactly as reachable from state s (cid:48) as is s (cid:48) from s ; we denote such transitions with s ↔ s (cid:48) .
Revised: First , consider the case where the transition between states s and s (cid:48) is fully reversible, i.e. when state s is exactly as reachable from state s (cid:48) as is s (cid:48) from s ; we denote such transitions with s ↔ s (cid:48) .

Original: Now, in expectation, we obtain that h ( s (cid:48) )
Revised: Now, in expectation, we obtain that h ( s (cid:48) )

Original: =
Revised: =

Original: h ( s ) and consequently, η ( s → s (cid:48) )
Revised: h ( s ) and consequently, η ( s → s (cid:48) )

Original: =
Revised: =

Original: η ( s (cid:48) → s )
Revised: η ( s (cid:48) → s )

Original: = 0 .
Revised: = 0 .

Original: But if instead the state s (cid:48) is more likely to follow state s than the vice versa (in expectation over trajectories), we say s (cid:48) is more reachable from s than the vice versa.
Revised: But if instead the state s (cid:48) is more likely to follow state s than the vice versa (in expectation over trajectories), we say s (cid:48) is more reachable from s than the vice versa.

Original: It follows in expectation that h ( s (cid:48) ) >
Revised: It follows in expectation that h ( s (cid:48) ) >

Original: h ( s ) , and consequently, η ( s → s (cid:48) )
Revised: h ( s ) , and consequently, η ( s → s (cid:48) )

Original: > 0 .
Revised: > 0 .

Original: Now the inductive bias on η as a difference of h -potentials automatically implies η ( s (cid:48)
Revised: Now the inductive bias on η as a difference of h -potentials automatically implies η ( s (cid:48)

Original: → s )
Revised: → s )

Original: = − η
Revised: = − η

Original: ( s → s (cid:48) ) < 0 .
Revised: ( s → s (cid:48) ) < 0 .

Original: Second , observe that the reachability measure implemented by η is additive by construction: given a trajectory s 0 → s 1 → s 2 , we have that η ( s 0 → s 2 ) =
Revised: Second , observe that the reachability measure implemented by η is additive by construction: given a trajectory s 0 → s 1 → s 2 , we have that η ( s 0 → s 2 ) =

Original: η ( s 0 → s 1 ) + η ( s 1 → s 2 ) .
Revised: η ( s 0 → s 1 ) + η ( s 1 → s 2 ) .

Original: As a special case, if we have that s 0 ↔ s 1 and s 1 ↔ s 2 – i.e. if η ( s 0 → s 1 )
Revised: As a special case, if we have that s 0 ↔ s 1 and s 1 ↔ s 2 – i.e. if η ( s 0 → s 1 )

Original: =
Revised: =

Original: η ( s 1 → s 2 )
Revised: η ( s 1 → s 2 )

Original: = 

Revised: =

Original: – it identically follows that s 

Revised: – it identically follows that s

Original: ↔ s 2 , i.e. η ( s 0 → s 2 )
Revised: ↔ s 2 , i.e. η ( s 0 → s 2 )

Original: = 0 .
Revised: = 0 .

Original: In this case, the inductive bias enables η to generalize to the transition s 0 ↔ s 2 even if it is never explicitly sampled by the policy.
Revised: In this case, the inductive bias enables η to generalize to the transition s 0 ↔ s 2 even if it is never explicitly sampled by the policy.

Original: Third , η allows for a soft measure of reachability.
Revised: Third , η allows for a soft measure of reachability.

Original: It measures not only whether a state s (cid:48) is reachable from another state s , but also quantiﬁes how reachable the former is from the latter.
Revised: It measures not only whether a state s (cid:48) is reachable from another state s , but also quantiﬁes how reachable the former is from the latter.

Original: As an example, consider a trajectory s 0 → s 1 → ... → s 100 , where the agent breaks one vase at every state transition.
Revised: As an example, consider a trajectory s 0 → s 1 → ... → s 100 , where the agent breaks one vase at every state transition.

Original: If the h -potential increases in constant increments for every vase broken (which we conﬁrm it does in Sec 5), we obtain due to the inductive bias that η ( s 0 → s 100 ) = 100 · η ( s 0 → s 1 ) .
Revised: If the h -potential increases in constant increments for every vase broken (which we conﬁrm it does in Sec 5), we obtain due to the inductive bias that η ( s 0 → s 100 ) = 100 · η ( s 0 → s 1 ) .

Original: This behaviour is sought-after in the context of AI-Safety (Krakovna et al., 2018; Leike et al., 2017).
Revised: This behaviour is sought-after in the context of AI-Safety (Krakovna et al., 2018; Leike et al., 2017).

Original: Nonetheless, one should be careful when interpreting η .
Revised: Nonetheless, one should be careful when interpreting η .

Original: While the above implies that η ( s (cid:48) → s )
Revised: While the above implies that η ( s (cid:48) → s )

Original: =
Revised: =

Original: η ( s → s (cid:48) )
Revised: η ( s → s (cid:48) )

Original: if the transition between states s and s (cid:48) is fully reversible, the converse can only be guaranteed if the Markov process admits a trajectory between s and s (cid:48) in either direction, i.e. if there exists a trajectory that visits both s and s (cid:48)
Revised: if the transition between states s and s (cid:48) is fully reversible, the converse can only be guaranteed if the Markov process admits a trajectory between s and s (cid:48) in either direction, i.e. if there exists a trajectory that visits both s and s (cid:48)

Original: (in any order).
Revised: (in any order).

Original: Observe that this condition much weaker than ergodicity, which requires that the Markov process admit a trajectory from any given state s to all other states s (cid:48) .
Revised: Observe that this condition much weaker than ergodicity, which requires that the Markov process admit a trajectory from any given state s to all other states s (cid:48) .

Original: In fact, the discrete environments we investigate in Sec 5 are non-ergodic.
Revised: In fact, the discrete environments we investigate in Sec 5 are non-ergodic.

Original: 3. D ETECTING AND P ENALIZING S IDE E FFECTS FOR S AFE E XPLORATION
Revised: 3. D ETECTING AND P ENALIZING S IDE E FFECTS FOR S AFE E XPLORATION

Original: The problem of detecting and avoiding side-effects is well known and crucially important for safe exploration (Moldovan & Abbeel, 2012;
Revised: The problem of detecting and avoiding side-effects is well known and crucially important for safe exploration (Moldovan & Abbeel, 2012;

Original: Eysenbach et al., 2017; Krakovna et al., 2018; Armstrong & Levinstein, 2017).
Revised: Eysenbach et al., 2017; Krakovna et al., 2018; Armstrong & Levinstein, 2017).

Original: Broadly, the problem involves detecting and avoiding state transitions that permanently and irreversibly damage the agent or the environment.
Revised: Broadly, the problem involves detecting and avoiding state transitions that permanently and irreversibly damage the agent or the environment.

Original: As such, it is fundamentally related to reachability, as in the agent is prohibited from taking actions that drastically reduce the reachability between the resulting state and some predeﬁned safe state.
Revised: As such, it is fundamentally related to reachability, as in the agent is prohibited from taking actions that drastically reduce the reachability between the resulting state and some predeﬁned safe state.

Original: In Eysenbach et al.
Revised: In Eysenbach et al.

Original: , the authors learn a reset policy responsible for resetting the environment to some initial state after the agent has completed its trajectory.
Revised: , the authors learn a reset policy responsible for resetting the environment to some initial state after the agent has completed its trajectory.

Original: The resulting value function of the reset policy indicates when the actual ( forward ) policy executes an irreversible state transition, but at the cost of the added complexity of training a reset policy.
Revised: The resulting value function of the reset policy indicates when the actual ( forward ) policy executes an irreversible state transition, but at the cost of the added complexity of training a reset policy.

Original: In contrast, Krakovna et al.
Revised: In contrast, Krakovna et al.

Original: (2018) propose to attack the problem by measuring reachability relative to a baseline state.
Revised: (2018) propose to attack the problem by measuring reachability relative to a baseline state.

Original: However, determining it may require counterfactual reasoning, which in turn assumes that the causal model is known.
Revised: However, determining it may require counterfactual reasoning, which in turn assumes that the causal model is known.

Original: We propose to directly use the reachability measure η deﬁned in Section 3.1 to derive a reward term for safe-exploration.
Revised: We propose to directly use the reachability measure η deﬁned in Section 3.1 to derive a reward term for safe-exploration.

Original: Let r t be some external reward at time-step t .
Revised: Let r t be some external reward at time-step t .

Original: The augmented reward is given by:
Revised: The augmented reward is given by:

Original: [Equation]
Revised: [Equation]

Original: where β is a scaling coefﬁcient.
Revised: where β is a scaling coefﬁcient.

Original: In practice, one may replace η with σ ( η ) , where σ is a monotonically increasing transfer function (e.g. a step function).
Revised: In practice, one may replace η with σ ( η ) , where σ is a monotonically increasing transfer function (e.g. a step function).

Original: Intuitively, transitions s → s (cid:48) that are less reversible cause the h -potential to increase, and the resulting reachability measure η ( s → s (cid:48) ) is positive in expectation.
Revised: Intuitively, transitions s → s (cid:48) that are less reversible cause the h -potential to increase, and the resulting reachability measure η ( s → s (cid:48) ) is positive in expectation.

Original: This incurs a penalty (due to the negative sign), which is reﬂected in the value function of the agent.
Revised: This incurs a penalty (due to the negative sign), which is reﬂected in the value function of the agent.

Original: Conversely, transitions that are reversible should have the property that η ( s → s (cid:48) )
Revised: Conversely, transitions that are reversible should have the property that η ( s → s (cid:48) )

Original: = 

Revised: =

Original: (also in expectation), thereby incurring no penalty.
Revised: (also in expectation), thereby incurring no penalty.

Original: 3. R EWARDING C URIOUS B EHAVIOUR
Revised: 3. R EWARDING C URIOUS B EHAVIOUR

Original: In most reinforcement learning applications, the reward function is assumed to be given; however, shaping a good reward function can often prove to be a challenging endeavour.
Revised: In most reinforcement learning applications, the reward function is assumed to be given; however, shaping a good reward function can often prove to be a challenging endeavour.

Original: It is in this context that the notion of curiosity comes to play an important role (Schmidhuber, 2010; Chentanez et al., 2005; Pathak et al., 2017; Burda et al., 2018;
Revised: It is in this context that the notion of curiosity comes to play an important role (Schmidhuber, 2010; Chentanez et al., 2005; Pathak et al., 2017; Burda et al., 2018;

Original: Savinov et al., 2018).
Revised: Savinov et al., 2018).

Original: One typical approach towards encouraging curious behaviour is to seek novel states that surprise the agent (Schmidhuber, 2010;
Revised: One typical approach towards encouraging curious behaviour is to seek novel states that surprise the agent (Schmidhuber, 2010;

Original: Pathak et al., 2017; Burda et al., 2018) and use the error in the agent’s prediction of future states is used as a curiosity reward.
Revised: Pathak et al., 2017; Burda et al., 2018) and use the error in the agent’s prediction of future states is used as a curiosity reward.

Original: This approach is however known to be susceptible to the so-called noisy-TV problem, wherein an uninteresting source of entropy like a noisy-TV can induce a large curiosity bonus because the agent cannot predict its future state.
Revised: This approach is however known to be susceptible to the so-called noisy-TV problem, wherein an uninteresting source of entropy like a noisy-TV can induce a large curiosity bonus because the agent cannot predict its future state.

Original: Savinov et al.
Revised: Savinov et al.

Original: (2018) propose to circumvent the noisy-TV problem by deﬁning novelty in terms of (undirected) reachability, wherein states that are easily reachable from the current state are considered less novel.
Revised: (2018) propose to circumvent the noisy-TV problem by deﬁning novelty in terms of (undirected) reachability, wherein states that are easily reachable from the current state are considered less novel.

Original: The h -potential and the corresponding reachability measure η affords another way of deﬁning a curiosity reward.
Revised: The h -potential and the corresponding reachability measure η affords another way of deﬁning a curiosity reward.

Original: Say an agent’s policy samples a trajectory from state s to s (cid:48) .
Revised: Say an agent’s policy samples a trajectory from state s to s (cid:48) .

Original: Now, recall that η ( s → s (cid:48) ) takes a positive value if state s (cid:48) is reachable from s (with respect to a simple reference policy); we therefore encourage the agent policy to sample trajectories where the η ( s → s (cid:48) ) is negative, i.e. where s (cid:48) is less reachable from s .
Revised: Now, recall that η ( s → s (cid:48) ) takes a positive value if state s (cid:48) is reachable from s (with respect to a simple reference policy); we therefore encourage the agent policy to sample trajectories where the η ( s → s (cid:48) ) is negative, i.e. where s (cid:48) is less reachable from s .

Original: In doing so, we encourage the agent to seek states that are otherwise difﬁcult to reach just by chance, and possibly learn useful skills in the process.
Revised: In doing so, we encourage the agent to seek states that are otherwise difﬁcult to reach just by chance, and possibly learn useful skills in the process.

Original: In other words, we reward the agent for reversing the arrow of time (recall that η ( s → s (cid:48) ) < 0 implies h ( s (cid:48) )
Revised: In other words, we reward the agent for reversing the arrow of time (recall that η ( s → s (cid:48) ) < 0 implies h ( s (cid:48) )

Original: <
Revised: <

Original: h ( s ) ).
Revised: h ( s ) ).

Original: The general form of the corresponding reward is given by:
Revised: The general form of the corresponding reward is given by:

Original: [Equation]
Revised: [Equation]

Original: While the above is independent of the external reward function deﬁned by the environment, the latter might often align with the former: in many environments, the task at hand is to reach the least reachable state.
Revised: While the above is independent of the external reward function deﬁned by the environment, the latter might often align with the former: in many environments, the task at hand is to reach the least reachable state.

Original: This is readily recognized in classical control tasks like Pendulum, Cartpole and Mountain-Car, where the goal state is often the least reachable.
Revised: This is readily recognized in classical control tasks like Pendulum, Cartpole and Mountain-Car, where the goal state is often the least reachable.

Original: However, if the environment’s speciﬁed task requires the agent to inadvertently execute irreversible trajectories, it is possible that our proposed reward is less applicable.
Revised: However, if the environment’s speciﬁed task requires the agent to inadvertently execute irreversible trajectories, it is possible that our proposed reward is less applicable.

Original: To summarize, we used the h -potential to deﬁne a directed measure of reachability (Sec 3.1), which then naturally lead to two applications.
Revised: To summarize, we used the h -potential to deﬁne a directed measure of reachability (Sec 3.1), which then naturally lead to two applications.

Original: In the ﬁrst (Sec 3.2), we obtained a safety penalty by essentially discouraging the agent from increasing the h -potential by executing difﬁcult-to-reverse transitions.
Revised: In the ﬁrst (Sec 3.2), we obtained a safety penalty by essentially discouraging the agent from increasing the h -potential by executing difﬁcult-to-reverse transitions.

Original: In the second (Sec 3.3), we argued that encouraging the agent to decrease the h -potential can provide an useful curiosity (intrinsic) reward signal in the absence of external rewards.
Revised: In the second (Sec 3.3), we argued that encouraging the agent to decrease the h -potential can provide an useful curiosity (intrinsic) reward signal in the absence of external rewards.

Original: In this sense, we have illustrated how the framework of a learned arrow of time (i.e. the h -potential) uniﬁes the notions of reachability, safety, and curiosity.
Revised: In this sense, we have illustrated how the framework of a learned arrow of time (i.e. the h -potential) uniﬁes the notions of reachability, safety, and curiosity.

Original:  A LGORITHM
Revised:  A LGORITHM

Original: In Sec 2, we proposed a general functional objective, and deﬁned the h -potential as the solution to the corresponding functional optimization problem.
Revised: In Sec 2, we proposed a general functional objective, and deﬁned the h -potential as the solution to the corresponding functional optimization problem.

Original: While the problem could be approached analytically with some effort for certain toy Markov chains (see App A), complex environments with unspeciﬁed dynamics require a fundamentally different approach.
Revised: While the problem could be approached analytically with some effort for certain toy Markov chains (see App A), complex environments with unspeciﬁed dynamics require a fundamentally different approach.

Original: We therefore convert the functional optimization problem in Eqn 1 (right) to one over the parameters θ of a deep neural network
Revised: We therefore convert the functional optimization problem in Eqn 1 (right) to one over the parameters θ of a deep neural network

Original: ˆ h θ to obtain the following surrogate problem:
Revised: ˆ h θ to obtain the following surrogate problem:

Original: [Equation]
Revised: [Equation]

Original:  where π is a reference policy, i.e. uniform random, and we denote the solution ˆ h θ ∗ by h .
Revised: [Equation] where π is a reference policy, i.e. uniform random, and we denote the solution ˆ h θ ∗ by h .

Original: To train the network, the expectations are replaced by their sample estimates.
Revised: To train the network, the expectations are replaced by their sample estimates.

Original: As for the regularizer, recall that its purpose was to prevent h from diverging within a ﬁnite domain – this can be achieved by a loss term T (like the trajectory regularizer in Eqn 2), or by a training constraint like early stopping.
Revised: As for the regularizer, recall that its purpose was to prevent h from diverging within a ﬁnite domain – this can be achieved by a loss term T (like the trajectory regularizer in Eqn 2), or by a training constraint like early stopping.

Original: The training algorithm is rather straightforward and can be summarized as follows (please refer to
Revised: The training algorithm is rather straightforward and can be summarized as follows (please refer to

Original: App B for the full algorithm).
Revised: App B for the full algorithm).

Original: We ﬁrst use an ofﬂine reference policy (uniform random, in our experiments) to sample trajectories from the environment.
Revised: We ﬁrst use an ofﬂine reference policy (uniform random, in our experiments) to sample trajectories from the environment.

Original: Next, we sample a batch of uniformly random state transitions and evaluate the objective in Eqn 6 (by replacing expectations by their sample estimates).
Revised: Next, we sample a batch of uniformly random state transitions and evaluate the objective in Eqn 6 (by replacing expectations by their sample estimates).

Original: We regularize the either by adding the trajectory regularizer to the objective or by using early stopping to terminate the training after a ﬁxed number of iterations.
Revised: We regularize the either by adding the trajectory regularizer to the objective or by using early stopping to terminate the training after a ﬁxed number of iterations.

Original: Finally, we optimize the parameters θ of ˆ h θ to maximize the objective at hand.
Revised: Finally, we optimize the parameters θ of ˆ h θ to maximize the objective at hand.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically investigate the h -potential that we obtain with the training procedure described in the previous section.
Revised: In this section, we empirically investigate the h -potential that we obtain with the training procedure described in the previous section.

Original: First, we show in a 2D-world environment that the h -potential learns to measure reachability.
Revised: First, we show in a 2D-world environment that the h -potential learns to measure reachability.

Original: Second, we show that the h -potential can be used to detect sideeffects in the challenging game of Sokoban (Leike et al., 2017).
Revised: Second, we show that the h -potential can be used to detect sideeffects in the challenging game of Sokoban (Leike et al., 2017).

Original: Third, we show on the game of
Revised: Third, we show on the game of

Original: Mountain Car with Friction that the h -potential can learn to capture sailent features of the environment, which can be used to formulate an intrinsic reward.
Revised: Mountain Car with Friction that the h -potential can learn to capture sailent features of the environment, which can be used to formulate an intrinsic reward.

Original: We also demonstrate how the h -potential fails if the environment is not dissipative, i.e. if the friction is turned off.
Revised: We also demonstrate how the h -potential fails if the environment is not dissipative, i.e. if the friction is turned off.

Original: Finally, we show for a particle undergoing Brownian motion under a potential that in expectation over states, the h -potential agrees reasonably well with the Free Energy functional, wherein the latter is known to be an arrow of time (Jordan et al., 1998).
Revised: Finally, we show for a particle undergoing Brownian motion under a potential that in expectation over states, the h -potential agrees reasonably well with the Free Energy functional, wherein the latter is known to be an arrow of time (Jordan et al., 1998).

Original: Moreover in App C, we show results on two additional environments.
Revised: Moreover in App C, we show results on three additional environments.

Original: [Figure]
Revised: [Figure]

Original: ( s t s t + 1 ) [ P o t e n t i a l D i ff e r e n c e ]
Revised: ( s t s t + 1 ) [ P o t e n t i a l D i ff e r e n c e ]

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: h t o f t h e
Revised: h t o f t h e

Original: M o un t a i n
Revised: M o un t a i n

Original: Height
Revised: Height

Original: Measuring Irreversibility.
Revised: Measuring Irreversibility.

Original: The environment considered is a 7 × 7 2D world, where cells can be occupied by the agent, the goal and/or a vase (their respective positions are randomly sampled in each episode).
Revised: The environment considered is a 7 × 7 2D world, where cells can be occupied by the agent, the goal and/or a vase (their respective positions are randomly sampled in each episode).

Original: If the agent enters a cell with a vase in it, the vase disappears without compromising the agent.
Revised: If the agent enters a cell with a vase in it, the vase disappears without compromising the agent.

Original: In Fig 2, we plot the change in h -potential
Revised: In Fig 2, we plot the change in h -potential

Original: (recall that η ( s t → s t +1 )
Revised: (recall that η ( s t → s t +1 )

Original: =
Revised: =

Original: h ( s t +1 ) − h ( s t ) ) to ﬁnd that the breaking of a vase (irreversible) corresponds to the h -potential increasing in steps of roughly constant size (observe that the spikes attain similar heights), whereas the agent moving around (reversible) does not result in it increasing.
Revised: h ( s t +1 ) − h ( s t ) ) to ﬁnd that the breaking of a vase (irreversible) corresponds to the h -potential increasing in steps of roughly constant size (observe that the spikes attain similar heights), whereas the agent moving around (reversible) does not result in it increasing.

Original: This indicates that the h -potential has learned to quantify irreversibility instead of merely detecting it by counting the number of broken vases.
Revised: This indicates that the h -potential has learned to quantify irreversibility instead of merely detecting it by counting the number of broken vases.

Original: In App C.1.1, (a) we further investigate the effect of adding temporally-correlated and TV (uncorrelated) noise to the state and ﬁnd that the h -potential is fairly robust to the latter but might get distracted by the former and (b) verify that an agent trained with the safety penalty in Eqn 4 breaks fewer vases (than without).
Revised: In App C.1.1, (a) we further investigate the effect of adding temporally-correlated and TV (uncorrelated) noise to the state and ﬁnd that the h -potential is fairly robust to the latter but might get distracted by the former and (b) verify that an agent trained with the safety penalty in Eqn 4 breaks fewer vases (than without).

Original: Detecting Side-Effects.
Revised: Detecting Side-Effects.

Original: Sokoban (“warehouse-keeper”) is a challenging puzzle video game, where an agent must push a number of boxes to set goal locations placed on a map.
Revised: Sokoban (“warehouse-keeper”) is a challenging puzzle video game, where an agent must push a number of boxes to set goal locations placed on a map.

Original: The agent may only push boxes (and not pull), rendering certain moves irreversible - for instance, when a box is pushed against a wall.
Revised: The agent may only push boxes (and not pull), rendering certain moves irreversible - for instance, when a box is pushed against a wall.

Original: Moreover, the task of even determining whether a move is irreversible might be nontrivial, making the problem a good test-bed for detecting side-effects (Leike et al., 2017).
Revised: Moreover, the task of even determining whether a move is irreversible might be nontrivial, making the problem a good test-bed for detecting side-effects (Leike et al., 2017).

Original: In Fig 3, we see that the h -potential increases if a box is pushed against a wall (irreversible side-effect) but remains constant if the agent moves about (reversible, even when the agent pushes a box around), demonstrating that the h -potential has indeed learned to detect side-effects.
Revised: In Fig 3, we see that the h -potential increases if a box is pushed against a wall (irreversible side-effect) but remains constant if the agent moves about (reversible, even when the agent pushes a box around), demonstrating that the h -potential has indeed learned to detect side-effects.

Original: For experimental details and additional plots, please refer to App C.1.3.
Revised: For experimental details and additional plots, please refer to App C.1.3.

Original: Obtaining Intrinsic Reward and the Importance of Dissipativity.
Revised: Obtaining Intrinsic Reward and the Importance of Dissipativity.

Original: The environment considered shares its dynamics with the well known (continuous) Mountain-Car environment (Sutton & Barto, 2011), but with a crucial amendment: the car is subject to friction.
Revised: The environment considered shares its dynamics with the well known (continuous) Mountain-Car environment (Sutton & Barto, 2011), but with a crucial amendment: the car is subject to friction.

Original: Friction is required to make the environment dissipative and thereby induce an arrow of time (cf.
Revised: Friction is required to make the environment dissipative and thereby induce an arrow of time (cf.

Original: Sec 2.2).
Revised: Sec 2.2).

Original: Moreover, we initialize the system in a uniform-randomly sampled state to avoid exploration issues.
Revised: Moreover, we initialize the system in a uniform-randomly sampled state to avoid exploration issues (cf.

Original:  In Fig 4, we see that the learned h -potential roughly recovers the terrain from random trajectories (i.e. without external rewards), which can now be used to obtain an intrinsic reward signal.
Revised: App C.3). In Fig 4, we see that the learned h -potential roughly recovers the terrain from random trajectories (i.e. without external rewards), which can now be used to obtain an intrinsic reward signal.

Original: Further, Fig 5b illustrates the importance of dissipation (in this case, induced via friction).
Revised: Further, Fig 5b illustrates the importance of dissipation (in this case, induced via friction).

Original: Detailsare in App C.2.2.
Revised: Details in App C.2.2.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Comparison with the Free-Energy Functional.
Revised: Comparison with the Free-Energy Functional.

Original: The setting considered is that of a particle (a random-walker ) undergoing Brownian motion under the inﬂuence of a potential ﬁeld Ψ( x ) (where x denotes the spatial position).
Revised: The setting considered is that of a particle (a random-walker ) undergoing Brownian motion under the inﬂuence of a potential ﬁeld Ψ( x ) (where x denotes the spatial position).

Original: We denote the probability of ﬁnding the particle at position x at time t by ρ ( x , t ) .
Revised: We denote the probability of ﬁnding the particle at position x at time t by ρ ( x , t ) .

Original: Now, the dynamics of the corresponding time-dependent random variable (i.e. stochastic process) X ( t ) is governed by the stochastic differential equation:
Revised: Now, the dynamics of the corresponding time-dependent random variable (i.e. stochastic process) X ( t ) is governed by the stochastic differential equation:

Original: [Equation]
Revised: [Equation]

Original: where W ( t ) is the standard Wiener process (i.e. d W ( t ) is white-noise) and β − 1 is a temperature parameter.
Revised: where W ( t ) is the standard Wiener process (i.e. d W ( t ) is white-noise) and β − 1 is a temperature parameter.

Original: The Free-Energy functional F is now deﬁned as:
Revised: The Free-Energy functional F is now deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst expectation of the RHS is the energy functional, and the second expectation is the negative entropy.
Revised: where the ﬁrst expectation of the RHS is the energy functional, and the second expectation is the negative entropy.

Original: A celebrated result due to Jordan, Kinderlehrer, and Otto (1998) is that the FreeEnergy is a Lyapunov functional of the dynamics, i.e. it can only decrease with time, thereby deﬁning a notion of an arrow of time.
Revised: A celebrated result due to Jordan, Kinderlehrer, and Otto (1998) is that the FreeEnergy is a Lyapunov functional of the dynamics, i.e. it can only decrease with time, thereby deﬁning a notion of an arrow of time.

Original: Now, to ﬁnd out how well our learned arrow of time agrees with the Free-Energy functional, we train it with realizations of the stochastic process X ( t ) in twodimensions.
Revised: Now, to ﬁnd out how well our learned arrow of time agrees with the Free-Energy functional, we train it with realizations of the stochastic process X ( t ) in twodimensions.

Original: Fig 6 plots the Free-Energy functional F against a linearly adjusted H -functional, deﬁned as: H [ ρ ( · , t )]
Revised: Fig 6 plots the Free-Energy functional F against a linearly adjusted H -functional, deﬁned as: H [ ρ ( · , t )]

Original: = − E
Revised: = − E

Original: x ∼ ρ ( · ,t ) [ h ( x )] .
Revised: x ∼ ρ ( · ,t ) [ h ( x )] .

Original: Indeed, we ﬁnd that up to a linear transform, the Hfunctional (and the corresponding h -potential) agrees reasonably well with the true arrow of time given by the Free-Energy functional F .
Revised: Indeed, we ﬁnd that up to a linear transform, the Hfunctional (and the corresponding h -potential) agrees reasonably well with the true arrow of time given by the Free-Energy functional F .

Original: Crucially, the H -functional is also a Lyapunov functional of the dynamics – implying that in expectation over states, the h -potential functions as an arrow of time.
Revised: Crucially, the H -functional is also a Lyapunov functional of the dynamics – implying that in expectation over states, the h -potential functions as an arrow of time.

Original: Details can be found in App C.3.
Revised: Details can be found in App C.4.

Original: Conclusion.
Revised: Conclusion.

Original: In this work, we approached the problem of learning an arrow of time in a Markov (Decision) Processes.
Revised: In this work, we approached the problem of learning an arrow of time in a Markov (Decision) Processes.

Original: We deﬁned the arrow of time ( h -potential) as a solution to an optimization problem and laid out the conceptual roadblocks that must be cleared before it can be useful in the RL context.
Revised: We deﬁned the arrow of time ( h -potential) as a solution to an optimization problem and laid out the conceptual roadblocks that must be cleared before it can be useful in the RL context.

Original: But once these roadblocks have been cleared, we demonstrated how the notions of reachability, safety and curiosity can be bridged by a common framework of a learned arrow of time.
Revised: But once these roadblocks have been cleared, we demonstrated how the notions of reachability, safety and curiosity can be bridged by a common framework of a learned arrow of time.

Original: Finally, we empirically investigated the strengths and shortcomings of our method on a selection of discrete and continuous environments.
Revised: Finally, we empirically investigated the strengths and shortcomings of our method on a selection of discrete and continuous environments.

Original: Future work could draw connections to algorithmic independence of cause and mechanism (Janzing et al., 2016) and explore applications in causal inference (Janzing, 2010; Peters et al., 2017).
Revised: Future work could draw connections to algorithmic independence of cause and mechanism (Janzing et al., 2016) and explore applications in causal inference (Janzing, 2010; Peters et al., 2017).

Original: A T HEORETICAL A NALYSIS
Revised: A T HEORETICAL A NALYSIS

Original: In this section, we present a theoretical analysis of the optimization problem formulated in Eqn 1, and analytically evaluate the result for a few toy Markov processes to validate that the resulting solutions are indeed consistent with intuition.
Revised: In this section, we present a theoretical analysis of the optimization problem formulated in Eqn 1, and analytically evaluate the result for a few toy Markov processes to validate that the resulting solutions are indeed consistent with intuition.

Original: To simplify the exposition, we consider the discrete case where the state space S of the MDP is ﬁnite.
Revised: To simplify the exposition, we consider the discrete case where the state space S of the MDP is ﬁnite.

Original: Consider a discrete Markov chain with enumerable states s i ∈ S .
Revised: Consider a discrete Markov chain with enumerable states s i ∈ S .

Original: At an arbitrary (but given) timestep t , we let p ti = p ( s t =
Revised: At an arbitrary (but given) timestep t , we let p ti = p ( s t =

Original: s i ) denote the probability that the Markov chain is in state s i , and p t the corresponding vector (over states).
Revised: s i ) denote the probability that the Markov chain is in state s i , and p t the corresponding vector (over states).

Original: With T ij we denote the probability of the Markov chain transitioning from state s i to s j under some policy π , i.e. T ij = p π ( s t +
Revised: With T ij we denote the probability of the Markov chain transitioning from state s i to s j under some policy π , i.e. T ij = p π ( s t +

Original: [Equation]
Revised: [Equation]

Original: s j | s t =
Revised: s j | s t =

Original: s i ) .
Revised: s i ) .

Original: One has the transition rule:
Revised: One has the transition rule:

Original: [Equation]
Revised: [Equation]

Original: where T t is the t -th matrix power of T .
Revised: where T t is the t -th matrix power of T .

Original: Now, we let h i denote the value h π takes at state s i , i.e. h i =
Revised: Now, we let h i denote the value h π takes at state s i , i.e. h i =

Original: h π ( s i ) , and the corresponding vector (over states) becomes h .
Revised: h π ( s i ) , and the corresponding vector (over states) becomes h .

Original: This reduces the expectation of the function (now a vector)
Revised: This reduces the expectation of the function (now a vector)

Original: h w.r.t any state distribution (now also a vector) p to the scalar product p · h .
Revised: h w.r.t any state distribution (now also a vector) p to the scalar product p · h .

Original: In matrix notation, the optimization problem in Eqn 1 simpliﬁes to:
Revised: In matrix notation, the optimization problem in Eqn 1 simpliﬁes to:

Original: [Equation]
Revised: [Equation]

Original: For certain T , the discrete problem in Eqn 10 can be handled analytically.
Revised: For certain T , the discrete problem in Eqn 10 can be handled analytically.

Original: We consider two candidates for T , the ﬁrst being the norm of h , and the second one being the norm of change in h i in expectation along trajectories.
Revised: We consider two candidates for T , the ﬁrst being the norm of h , and the second one being the norm of change in h i in expectation along trajectories.

Original: Proposition 1.
Revised: Proposition 1.

Original: If T ( h ) = − (2 N ) − 1 (cid:107) h (cid:107) 2 , the solution to the optimization problem in Eqn 10 is given by:
Revised: If T ( h ) = − (2 N ) − 1 (cid:107) h (cid:107) 2 , the solution to the optimization problem in Eqn 10 is given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, note that the objective in Eqn 10 becomes:
Revised: First, note that the objective in Eqn 10 becomes:

Original: [Equation]
Revised: [Equation]

Original: To solve the maximization problem, we must differentiate L w.r.t.
Revised: To solve the maximization problem, we must differentiate L w.r.t.

Original: its argument h , and set the resulting expression to zero.
Revised: its argument h , and set the resulting expression to zero.

Original: This yields:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: Now, the summation (over t ) is telescoping, and evaluates to p N − 1 T − p 0 .
Revised: Now, the summation (over t ) is telescoping, and evaluates to p N − 1 T − p 0 .

Original: Substituting p N − 1 with the corresponding expression from Eqn 9 and solving for h , we obtain Eqn 11.
Revised: Substituting p N − 1 with the corresponding expression from Eqn 9 and solving for h , we obtain Eqn 11.

Original: Proposition 1 has an interesting implication: if the Markov chain is initialized at equilibrium, i.e. if p 0 = p 0 T , we obtain that h = 0 identically.
Revised: Proposition 1 has an interesting implication: if the Markov chain is initialized at equilibrium, i.e. if p 0 = p 0 T , we obtain that h = 0 identically.

Original: Given the above, we may now consider as examples the following simple Markov chains.
Revised: Given the above, we may now consider as examples the following simple Markov chains.

Original: [Equation]
Revised: [Equation]

Original: Example 1.
Revised: Example 1.

Original: Consider a Markov chain with two states and reversible transitions, parameterized by α ∈ [0 , 1] such that T 11 =
Revised: Consider a Markov chain with two states and reversible transitions, parameterized by α ∈ [0 , 1] such that T 11 =

Original: T 21 = 1 − α and T 12 = T 

Revised: T 21 = 1 − α and T 12 = T

Original: = α (Fig 7).
Revised: = α (Fig 7).

Original: If p 0 = ( 1 / 2 , 1 / 2 ) , one obtains:
Revised: If p 0 = ( 1 / 2 , 1 / 2 ) , one obtains:

Original: [Equation]
Revised: [Equation]

Original: where γ =
Revised: where γ =

Original: α − 1 / 2 .
Revised: α − 1 / 2 .

Original: To see how, consider that for all N > 0 , one obtains p 0 T N =
Revised: To see how, consider that for all N > 0 , one obtains p 0 T N =

Original: (1 − α, α ) .
Revised: (1 − α, α ) .

Original: Together with Proposition 1, Eqn 14 follows.
Revised: Together with Proposition 1, Eqn 14 follows.

Original: The above example illustrates two things.
Revised: The above example illustrates two things.

Original: On the one hand, if α = 1 / 2 , one obtains a Markov chain with perfect reversibility , i.e. the transition s 1 → s 2 is equally as likely as the transition s 2 → s 1 .
Revised: On the one hand, if α = 1 / 2 , one obtains a Markov chain with perfect reversibility , i.e. the transition s 1 → s 2 is equally as likely as the transition s 2 → s 1 .

Original: In this case, one indeed obtains h ( s 1 )
Revised: In this case, one indeed obtains h ( s 1 )

Original: =
Revised: =

Original: h ( s 2 ) = 0 , as mentioned above.
Revised: h ( s 2 ) = 0 , as mentioned above.

Original: On the other hand, if one sets α = 1 , the transition from s 2 → s 1 is never sampled, and that from s 1 → s 2 is irreversible; consequently, h ( s 2 ) − h ( s 1 ) takes the largest value possible.
Revised: On the other hand, if one sets α = 1 , the transition from s 2 → s 1 is never sampled, and that from s 1 → s 2 is irreversible; consequently, h ( s 2 ) − h ( s 1 ) takes the largest value possible.

Original: Now, while this aligns well with our intuition, the following example exposes a weakness of the L2-norm-penalty used in Proposition 1.
Revised: Now, while this aligns well with our intuition, the following example exposes a weakness of the L2-norm-penalty used in Proposition 1.

Original: [Figure]
Revised: [Figure]

Original: Example 2.
Revised: Example 2.

Original: Consider two Markov chains, both always initialized at s 1 .
Revised: Consider two Markov chains, both always initialized at s 1 .

Original: For the ﬁrst Markov chain, the dynamics admits the following transitions: s 1 → s 2 → s 3 → s 4 , whereas for the second chain, one has s 1 → s 3 → s 2 → s 4 (Fig 8).
Revised: For the ﬁrst Markov chain, the dynamics admits the following transitions: s 1 → s 2 → s 3 → s 4 , whereas for the second chain, one has s 1 → s 3 → s 2 → s 4 (Fig 8).

Original: Now, for both chains and N ≥ 4 , it’s easy to see that ( p 0 T N ) i
Revised: Now, for both chains and N ≥ 4 , it’s easy to see that ( p 0 T N ) i

Original: = 

Revised: =

Original: if i = 4 , but 0 otherwise.
Revised: if i = 4 , but 0 otherwise.

Original: From Eqn 11, one obtains:
Revised: From Eqn 11, one obtains:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The solution for h given by Eqn 15 indeed increases (non-strictly) monotonously with timestep.
Revised: The solution for h given by Eqn 15 indeed increases (non-strictly) monotonously with timestep.

Original: However, we obtain h ( s 2 )
Revised: However, we obtain h ( s 2 )

Original: =
Revised: =

Original: h ( s 3 ) = 0 for both Markov chains.
Revised: h ( s 3 ) = 0 for both Markov chains.

Original: In particular, h does not increase between the s 2 → s 3 transition in the former and the s 3 → s 2 transition in the latter, even though both transitions are irreversible.
Revised: In particular, h does not increase between the s 2 → s 3 transition in the former and the s 3 → s 2 transition in the latter, even though both transitions are irreversible.

Original: It is in general apparent from 1 that the solution for h depends only on the initial and ﬁnal state distribution, and not the intermediate trajectory.
Revised: It is in general apparent from 1 that the solution for h depends only on the initial and ﬁnal state distribution, and not the intermediate trajectory.

Original: Now, consider the following regularizer that penalizes not just the function norm, but the change in h in expectation along trajectories:
Revised: Now, consider the following regularizer that penalizes not just the function norm, but the change in h in expectation along trajectories:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ω is the relative weight of the L2 regularizer.
Revised: where ω is the relative weight of the L2 regularizer.

Original: This leads to the result:
Revised: This leads to the result:

Original: Proposition 2.
Revised: Proposition 2.

Original: The solution to the optimization problem in Eqn 10 with the regularizer in Eqn 16 is the solution to the following matrix-equation:
Revised: The solution to the optimization problem in Eqn 10 with the regularizer in Eqn 16 is the solution to the following matrix-equation:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Analogous to Eqn 12, we may write the objective in Eqn 10 as (by substituting Eqn 16 in
Revised: Analogous to Eqn 12, we may write the objective in Eqn 10 as (by substituting Eqn 16 in

Original: Eqn 10):
Revised: Eqn 10):

Original: [Equation]
Revised: [Equation]

Original: Like in Proposition 1, we maximize it by setting the gradient of L w.r.t.
Revised: Like in Proposition 1, we maximize it by setting the gradient of L w.r.t.

Original: h to zero.
Revised: h to zero.

Original: This yields:
Revised: This yields:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term in the RHS is again a telescoping sum; it evaluates to: p 0 T N − p 

Revised: The ﬁrst term in the RHS is again a telescoping sum; it evaluates to: p 0 T N − p

Original: (cf.
Revised: (cf.

Original: proof of
Revised: proof of

Original: Proposition 1).
Revised: Proposition 1).

Original: The second term can be expressed as (with I as the identity matrix):
Revised: The second term can be expressed as (with I as the identity matrix):

Original: [Equation]
Revised: [Equation]

Original: where the last equality follows from Eqn 9.
Revised: where the last equality follows from Eqn 9.

Original: Substituting the above in Eqn 19 and rearranging terms yields Eqn 17.
Revised: Substituting the above in Eqn 19 and rearranging terms yields Eqn 17.

Original: While Eqn 17 does not yield an explicit expression for h , it is sufﬁcient for analysing individual cases considered in Examples 1 and 2.
Revised: While Eqn 17 does not yield an explicit expression for h , it is sufﬁcient for analysing individual cases considered in Examples 1 and 2.

Original: Example 3.
Revised: Example 3.

Original: Consider the two-state Markov chain in Example 1 (Fig 7) and the associated transition matrix T and initial state distribution p 

Revised: Consider the two-state Markov chain in Example 1 (Fig 7) and the associated transition matrix T and initial state distribution p

Original: = ( 1 / 2 , 1 / 2 ) .
Revised: = ( 1 / 2 , 1 / 2 ) .

Original: Using the regularization scheme in Eqnand the associated solution Eqn 17, one obtains:
Revised: Using the regularization scheme in Eqnand the associated solution Eqn 17, one obtains:

Original: [Equation]
Revised: [Equation]

Original: 23) where:
Revised: 23) where:

Original: [Equation]
Revised: [Equation]

Original: To obtain this result 5 , we use that T t =
Revised: To obtain this result 7 , we use that T t =

Original: T for all t ≥ 1 and truncate the sum without loss of generality at N = 1 .
Revised: T for all t ≥ 1 and truncate the sum without loss of generality at N = 1 .

Original: Like in Example 1, we observe h ( s 1 )
Revised: Like in Example 1, we observe h ( s 1 )

Original: =
Revised: =

Original: h ( s 2 ) =
Revised: h ( s 2 ) =

Original: if α = 1 / 2 for all ω > 0 (i.e. at equilibrium).
Revised: if α = 1 / 2 for all ω > 0 (i.e. at equilibrium).

Original: In addition, if ω ≥ 1 / 2 , it can be shown that h ( s 2 ) − h ( s 1 ) increases monotonously with α and takes the largest possible value at α = 1 .
Revised: In addition, if ω ≥ 1 / 2 , it can be shown that h ( s 2 ) − h ( s 1 ) increases monotonously with α and takes the largest possible value at α = 1 .

Original: We therefore ﬁnd that for the simple two-state Markov chain of Example 1, the regularization in Eqn 16 indeed leads to intuitive behaviour for the respective solution h .
Revised: We therefore ﬁnd that for the simple two-state Markov chain of Example 1, the regularization in Eqn 16 indeed leads to intuitive behaviour for the respective solution h .

Original: Now:
Revised: Now:

Original: Example 4.
Revised: Example 4.

Original: Consider the four-state Markov chain with transitions s 1 → s 2 → s 3 → s 4 (Fig 8) and the corresponding transition matrix T , where T 12 = T 

Revised: Consider the four-state Markov chain with transitions s 1 → s 2 → s 3 → s 4 (Fig 8) and the corresponding transition matrix T , where T 12 = T

Original: = T 

Revised: = T

Original: = T 44 = 1 , T ij = 0 for all other i, j . Set p 

Revised: = T 44 = 1 , T ij = 0 for all other i, j . Set p

Original: = (1 , 0 , 0 , 0) , i.e. the chain is always initialized at s 1 .
Revised: = (1 , 0 , 0 , 0) , i.e. the chain is always initialized at s 1 .

Original: Now, the summation over t in
Revised: Now, the summation over t in

Original: Eqn 17 can be truncated at N = 4 without loss of generality (over N ), given that T t +1 =
Revised: Eqn 17 can be truncated at N = 4 without loss of generality (over N ), given that T t +1 =

Original: T t for all t ≥ 3 .
Revised: T t for all t ≥ 3 .

Original: At ω = 0 , one solution is:
Revised: At ω = 0 , one solution is:

Original: [Equation]
Revised: [Equation]

Original: Further, for all ω ≥ 0 , one obtains h ( s 1 ) <
Revised: Further, for all ω ≥ 0 , one obtains h ( s 1 ) <

Original: h ( s 2 ) <
Revised: h ( s 2 ) <

Original: h ( s 3 ) <
Revised: h ( s 3 ) <

Original: h ( s 4 ) , where the inequality is strict.
Revised: h ( s 4 ) , where the inequality is strict.

Original: This is unlike Eqn 15 where h ( s 2 )
Revised: This is unlike Eqn 15 where h ( s 2 )

Original: =
Revised: =

Original: h ( s 3 ) , and consistent with the intuitive expectation that the arrow of time must increase along irreversible transitions.
Revised: h ( s 3 ) , and consistent with the intuitive expectation that the arrow of time must increase along irreversible transitions.

Original: In conclusion: we ﬁnd that the functional objective deﬁned in Eqn 1 may indeed lead to analytical solutions that are consistent with the notion of an arrow of time in certain toy Markov chains.
Revised: In conclusion: we ﬁnd that the functional objective deﬁned in Eqn 1 may indeed lead to analytical solutions that are consistent with the notion of an arrow of time in certain toy Markov chains.

Original: However, in most interesting real world environments, the transition model T is not known and or or the number of states is infeasibly large, rendering an analytic solution intractable.
Revised: However, in most interesting real world environments, the transition model T is not known and or or the number of states is infeasibly large, rendering an analytic solution intractable.

Original: In such cases, as we see in Section 5, it is possible to parameterize h as a neural network and train the resulting model with stochastic gradient descent to optimize the functional objective deﬁned in Eqn 1.
Revised: In such cases, as we see in Section 5, it is possible to parameterize h as a neural network and train the resulting model with stochastic gradient descent to optimize the functional objective deﬁned in Eqn 1.

Original: B A LGORITHM
Revised: B A LGORITHM

Original: Require: Environment Env , random policy π (cid:93) , trajectory buffer B
Revised: Require: Environment Env , random policy π (cid:93) , trajectory buffer B

Original: Require:
Revised: Require:

Original: Model h θ , regularizer T , optimizer.
Revised: Model h θ , regularizer T , optimizer.

Original: 1: for k = 1 ...
Revised: 1: for k = 1 ...

Original: M do 2:
Revised: M do 2:

Original: B[k, :] ← ( s 0 , ..., s N ) ∼ Env [ π (cid:93) ]
Revised: B[k, :] ← ( s 0 , ..., s N ) ∼ Env [ π (cid:93) ]

Original: { Sample a trajectory of length N with the random policy and write to k -th position in the buffer.
Revised: { Sample a trajectory of length N with the random policy and write to k -th position in the buffer.

Original: } 3: end for 4: loop 5:
Revised: } 3: end for 4: loop 5:

Original: Sample trajectory index k ∼ { 1 , ..., M } and time-step t ∼ { 0 , ..., N − 1 } .
Revised: Sample trajectory index k ∼ { 1 , ..., M } and time-step t ∼ { 0 , ..., N − 1 } .

Original: { In general, one may sample multiple k ’s and t ’s for a larger mini-batch. } 6:
Revised: { In general, one may sample multiple k ’s and t ’s for a larger mini-batch. } 6:

Original: Fetch states s t ← B[k, t] and s t +1 ← B[k, t + 1] from buffer.
Revised: Fetch states s t ← B[k, t] and s t +1 ← B[k, t + 1] from buffer.

Original: Compute loss as L ( θ ) = − [ h θ ( s t +1 ) − h
Revised: Compute loss as L ( θ ) = − [ h θ ( s t +1 ) − h

Original: θ ( s t )] .
Revised: θ ( s t )] .

Original: 8: if using trajectory regularizer then 9:
Revised: 8: if using trajectory regularizer then 9:

Original: Compute regularizer term as [ h θ ( s t +1 ) − h
Revised: Compute regularizer term as [ h θ ( s t +1 ) − h

Original: θ ( s t )] 2 and add to L ( θ ) .
Revised: θ ( s t )] 2 and add to L ( θ ) .

Original: 10: else 11:
Revised: 10: else 11:

Original: Apply the regularizer as required.
Revised: Apply the regularizer as required.

Original: If early-stopping, break out of the loop if necessary.
Revised: If early-stopping, break out of the loop if necessary.

Original: 12: end if 13:
Revised: 12: end if 13:

Original: Compute parameter gradients ∇ θ L ( θ ) and update parameters with the optimizer.
Revised: Compute parameter gradients ∇ θ L ( θ ) and update parameters with the optimizer.

Original: 14: end loop
Revised: 14: end loop

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: t i a l
Revised: t i a l

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: C E XPERIMENTAL D ETAILS
Revised: C E XPERIMENTAL D ETAILS

Original: All experiments were run on a workstation with 40 cores, 256 GB RAM and 2 nVidia GTX 1080Ti.
Revised: All experiments were run on a workstation with 40 cores, 256 GB RAM and 2 nVidia GTX 1080Ti.

Original: C. D ISCRETE E NVIRONMENTS C.1. 2D W ORLD WITH V ASES
Revised: C. D ISCRETE E NVIRONMENTS C.1. 2D W ORLD WITH V ASES

Original: The environment state comprises three 7 × 7 binary images (corresponding to agent, vases and goal), and the vases appear in a different arrangement every time the environment is reset.
Revised: The environment state comprises three 7 × 7 binary images (corresponding to agent, vases and goal), and the vases appear in a different arrangement every time the environment is reset.

Original: The probability of sampling a vase at any given position is set to 1 / 2 .
Revised: The probability of sampling a vase at any given position is set to 1 / 2 .

Original: We use a two-layer deep and 256-unit wide ReLU network to parameterize the h -potential.
Revised: We use a two-layer deep and 256-unit wide ReLU network to parameterize the h -potential.

Original: It is trained on 4096 trajectories of length 128 for 10000 iterations of stochastic gradient descent with
Revised: It is trained on 4096 trajectories of length 128 for 10000 iterations of stochastic gradient descent with

Original: Adam optimizer (learning rate: 0.0001).
Revised: Adam optimizer (learning rate: 0.0001).

Original: The batch-size is set to 128, and we use a weight decay of 0.005 to regularize the model.
Revised: The batch-size is set to 128, and we use a weight decay of 0.005 to regularize the model.

Original: We use a validation trajectory to generate the plots in Fig 10 and 2.
Revised: We use a validation trajectory to generate the plots in Fig 10 and 2.

Original: Moreover, Fig 11 shows histograms of the values taken by h at various time-steps along the trajectory.
Revised: Moreover, Fig 11 shows histograms of the values taken by h at various time-steps along the trajectory.

Original: We learn that h takes on larger values (on average) as t increases.
Revised: We learn that h takes on larger values (on average) as t increases.

Original: To test the robustness of our method, we conduct experiments where the environment state is augmented with one of: (a) a 7 × 7 image with uniform-randomly sampled pixel values ( TV-noise ) and (b) a 7 × 7 image where every pixel takes the value t / 128 , where t is the time-step 6 of the correspond- 
Revised: To test the robustness of our method, we conduct experiments where the environment state is augmented with one of: (a) a 7 × 7 image with uniform-randomly sampled pixel values ( TV-noise ) and (b) a 7 × 7 image where every pixel takes the value t / 128 , where t is the time-step 8 of the correspond- [Figure] a

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: ing state ( Causal Noise ).
Revised: ing state ( Causal Noise ).

Original: Fig 9a and 9b plot the corresponding η = ∆ h along randomly sampled trajectories.
Revised: Fig 9a and 9b plot the corresponding η = ∆ h along randomly sampled trajectories.

Original: Given a learned arrow of time, we now present an experiment where we use it to derive a safeexploration penalty (in addition to the environment reward).
Revised: Given a learned arrow of time, we now present an experiment where we use it to derive a safeexploration penalty (in addition to the environment reward).

Original: To that end, we now consider the situation where the agent’s policy is not random, but specialized to reach the goal state (from its current state).
Revised: To that end, we now consider the situation where the agent’s policy is not random, but specialized to reach the goal state (from its current state).

Original: For both the baseline and the safe agents, every action is rewarded with the change in Manhattan norm of the agent’s position to that of the goal – i.e. an action that moves the agent closer to the goal is rewarded +1 , one that moves it farther away from the goal is penalized − 1 , and one that keeps the distance unchanged is neither penalized nor rewarded ( 0 ).
Revised: For both the baseline and the safe agents, every action is rewarded with the change in Manhattan norm of the agent’s position to that of the goal – i.e. an action that moves the agent closer to the goal is rewarded +1 , one that moves it farther away from the goal is penalized − 1 , and one that keeps the distance unchanged is neither penalized nor rewarded ( 0 ).

Original: Further, every step is penalized by − 0 .
Revised: Further, every step is penalized by − 0 .

Original: 1 (so as to keep the trajectories short), and exceeding the available time limit ( 30 steps) incurs a termination penalty ( − 10 ).
Revised: 1 (so as to keep the trajectories short), and exceeding the available time limit ( 30 steps) incurs a termination penalty ( − 10 ).

Original: In addition, the reward function of the safe agent is augmented with the reachability, i.e. it takes the form described in Eqn 4.
Revised: In addition, the reward function of the safe agent is augmented with the reachability, i.e. it takes the form described in Eqn 4.

Original: We use β = 4 and a transfer function σ such that σ ( η ) = 

Revised: We use β = 4 and a transfer function σ such that σ ( η ) =

Original: if η < 5000 (cf.
Revised: if η < 5000 (cf.

Original: Fig 2), and 1 otherwise.
Revised: Fig 2), and 1 otherwise.

Original: The policy is parameterized by a 3-layer deep 256-unit wide (fully connected) ReLU network and trained via Duelling Double Deep Q-Learning 7 (Van Hasselt et al., 2016; Wang et al., 2015).
Revised: The policy is parameterized by a 3-layer deep 256-unit wide (fully connected) ReLU network and trained via Duelling Double Deep Q-Learning 9 (Van Hasselt et al., 2016; Wang et al., 2015).

Original: The discount factor is set to 0 .
Revised: The discount factor is set to 0 .

Original: 99 and the target network is updated once every 200 iterations.
Revised: 99 and the target network is updated once every 200 iterations.

Original: For exploration, we use a 1 − (cid:15) greedy policy, where (cid:15) is decayed linearly from 1 to 0 .
Revised: For exploration, we use a 1 − (cid:15) greedy policy, where (cid:15) is decayed linearly from 1 to 0 .

Original: 1 in the span of the ﬁrst 10000 iterations.
Revised: 1 in the span of the ﬁrst 10000 iterations.

Original: The replay buffer stores 10000 experiences and the batch-size used is 10 .
Revised: The replay buffer stores 10000 experiences and the batch-size used is 10 .

Original: Fig 12a shows the probability of reaching the goal (in an episode of 30 steps) over the iterations (sample size 100 ), whereas Fig 12b shows the expected number of vases broken per episode (over the same 100 episodes).
Revised: Fig 12a shows the probability of reaching the goal (in an episode of 30 steps) over the iterations (sample size 100 ), whereas Fig 12b shows the expected number of vases broken per episode (over the same 100 episodes).

Original: Both curves are smoothed by a Savitzky-Golay ﬁlter (Savitzky & Golay, 1964) of order 3 and window-size 53 (the original, unsmoothed curves are shaded).
Revised: Both curves are smoothed by a Savitzky-Golay ﬁlter (Savitzky & Golay, 1964) of order 3 and window-size 53 (the original, unsmoothed curves are shaded).

Original: As expected, we ﬁnd that using the safety penalty does indeed result in fewer vases broken, but also makes the task of reaching the goal difﬁcult (we do not ensure that the goal is reachable without breaking vases).
Revised: As expected, we ﬁnd that using the safety penalty does indeed result in fewer vases broken, but also makes the task of reaching the goal difﬁcult (we do not ensure that the goal is reachable without breaking vases).

Original: C.1. 2D W ORLD WITH D RYING T OMATOES
Revised: C.1. 2D W ORLD WITH D RYING T OMATOES

Original: The environment considered comprises a 7 × 7 2D world where each cell is initially occupied by watered tomato plant 8 .
Revised: The environment considered comprises a 7 × 7 2D world where each cell is initially occupied by watered tomato plant 10 .

Original: The agent waters the cell it occupies, restoring the moisture level of the plant in the said cell to 100% .
Revised: The agent waters the cell it occupies, restoring the moisture level of the plant in the said cell to 100% .

Original: However, for each step the agent does not water a plant, it loses some moisture (by 2% of maximum in our experiments).
Revised: However, for each step the agent does not water a plant, it loses some moisture (by 2% of maximum in our experiments).

Original: If a plant loses all moisture, it is considered dead and no amount of watering can resurrect it.
Revised: If a plant loses all moisture, it is considered dead and no amount of watering can resurrect it.

Original: The state-space comprises two 7 × 7 images: the ﬁrst image is an indicator of the agent’s position, whereas the pixel values of the second image quantiﬁes the amount of moisture held by the plant 9 at the corresponding location.
Revised: The state-space comprises two 7 × 7 images: the ﬁrst image is an indicator of the agent’s position, whereas the pixel values of the second image quantiﬁes the amount of moisture held by the plant 11 at the corresponding location.

Original: [Figure]
Revised: [Figure]

Original: We show that it is possible to recover an intrinsic reward signal that coincides well with one that one might engineer.
Revised: We show that it is possible to recover an intrinsic reward signal that coincides well with one that one might engineer.

Original: To that end, we parameterize the h -potential as a two-layer deep 256-unit wide
Revised: To that end, we parameterize the h -potential as a two-layer deep 256-unit wide

Original: ReLU network and train it on 4096 trajectories (generated by a random policy) of length 128 for 10000 iterations of Adam (learning rate: 0.0001).
Revised: ReLU network and train it on 4096 trajectories (generated by a random policy) of length 128 for 10000 iterations of Adam (learning rate: 0.0001).

Original: The batch-size is set to 128 and the model is regularized with the trajectory regularizer ( λ = 0 . 5 ).
Revised: The batch-size is set to 128 and the model is regularized with the trajectory regularizer ( λ = 0 . 5 ).

Original: Unsurprisingly, we ﬁnd that h increases as the plants lose moisture.
Revised: Unsurprisingly, we ﬁnd that h increases as the plants lose moisture.

Original: But conversely, when the agent waters a plant, it causes the h -potential to decrease by an amount that strongly correlates with the amount of moisture the watered plant gains.
Revised: But conversely, when the agent waters a plant, it causes the h -potential to decrease by an amount that strongly correlates with the amount of moisture the watered plant gains.

Original: This can be used to deﬁne a dense reward signal for the agent:
Revised: This can be used to deﬁne a dense reward signal for the agent:

Original: [Equation]
Revised: [Equation]

Original: (26) where we use a momentum of 0 .
Revised: (26) where we use a momentum of 0 .

Original: 95 to evaluate the running average.
Revised: 95 to evaluate the running average.

Original: In Fig 13, we plot for a random trajectory the intrinsic reward ˆ r t against a reference reward, which in this case is the moisture gain of the plant the agent just watered.
Revised: In Fig 13, we plot for a random trajectory the intrinsic reward ˆ r t against a reference reward, which in this case is the moisture gain of the plant the agent just watered.

Original: Further, we observe the reward function dropping signiﬁcantly at around the 90-th iteration - this is precisely when all plants have died.
Revised: Further, we observe the reward function dropping signiﬁcantly at around the 90-th iteration - this is precisely when all plants have died.

Original: This demonstrates that the h -potential can indeed be useful for deﬁning intrinsic rewards.
Revised: This demonstrates that the h -potential can indeed be useful for deﬁning intrinsic rewards [Figure]

Original: C.1. S OKOBAN
Revised: C.1. S OKOBAN

Original:  The environment state comprises ﬁve 10 × 10 binary images, where the pixel value at each location indicates the presence of the agent, a box, a goal, a wall and empty space.
Revised: [Figure] [Figure] The environment state comprises ﬁve 10 × 10 binary images, where the pixel value at each location indicates the presence of the agent, a box, a goal, a wall and empty space.

Original: The layout of all sprites are randomized at each environment reset, under the constraint that the game is still solvable (Schrader, 2018).
Revised: The layout of all sprites are randomized at each environment reset, under the constraint that the game is still solvable (Schrader, 2018).

Original: The h -potential is parameterized by a two-layer deep and 512-unit wide network, which is trained on 4096 trajectories of length 512 for 20000 steps of Adam (learning rate: 0.0001).
Revised: The h -potential is parameterized by a two-layer deep and 512-unit wide network, which is trained on 4096 trajectories of length 512 for 20000 steps of Adam (learning rate: 0.0001).

Original: The batch-size is set to 256 and we use the trajectory regularizer ( λ = 0 . 05 ) to regularize our model.
Revised: The batch-size is set to 256 and we use the trajectory regularizer ( λ = 0 . 05 ) to regularize our model.

Original: C. C ONTINOUS E NVIRONMENTS C.2. U NDER - DAMPED P ENDULUM
Revised: C.1. C ONVEYOR B ELT E NVIRONMENT OF K RAKOVNA ET AL . (2018)

Revised: The environment considered is a 5 × 5 2D world (Fig 16) with an agent, a conveyor belt and a vase that is initially placed on the belt (this is the Vase environment in Krakovna et al.

Revised: If the agent is passive (i.e. it stays put, Fig 17b), the conveyor belt moves the vase one step to the right until it eventually falls off the belt and breaks.

Revised: However, the intended behaviour of a safe ( good ) agent (Fig 17a) is that it removes the vase from the conveyor belt, preventing it from breaking.

Revised: One may also have a malicious agent (Fig 17c) that removes the vase from the belt (e.g. to collect a reward) only to put it back on it again.

Revised: In contrast, an inept agent (Fig 17d) may remove the vase from the belt but irreversibly push it to a corner (like in Sokoban).

Revised: In the following, we investigate the safety-reward that is awarded by the h -potential to each of these policies, which we label good , passive , malicious and inept .

Revised: [Figure]

Revised: To that end, we gather 4096 random trajectories of length 64 each.

Revised: The h -potential is parameterized by a 256-unit wide and 2-layer deep MLP with ReLU activation, and trained with 80000 steps of stochastic gradient descent with Adam Kingma & Ba (2014).

Revised: We use a trajectory regularizer with λ = 0 .

Revised: 01 to regularize the model.

Revised: The state space is a collection of 6 binary images of size 7 × 7 , where in each image the truth value of a pixel marks the presence of a wall, empty space, belt, agent, vase and a broken vase in the corresponding location 12 .

Revised: As the model trains, we track the return it awards to the four policies mentioned above (to aid visualization, we normalize the rewards to have a mean of zero) and plot the result in Fig 18.

Revised: We ﬁnd that the h -potential rewards the good policy, but penalizes the passive , malicious and inept policies.

Revised: This can be contrasted with the predeﬁned safety performance measure, which assigns a safety score of +50 to the good and inept agents, 0 to the malicious agent and − 50 to the passive agent 13 (larger score is safer).

Revised: Our method therefore learns that pushing the vase to a corner is no less irreversible than breaking it ( malicious and passive ) and penalizes the inept policy accordingly.

Revised: [Figure]

Revised: To conclude, we conﬁrm that the safety reward extracted from the h -potential can enable agents to avoid irreversible behaviour.

Revised: However, while preventing the irreversible is safe in this context, it may not always be the case – we point the reader to the Sushi environment in Krakovna et al.

Revised: for an example 14 .

Revised: 3 Angle

Revised: A n g u l a r

Revised: V

Revised: e

Revised: l

Revised: o c i t y 2.

Revised: 3 Angle [rad] 0.

Revised: h ( x = , v = 0 )

Revised: [ N e g a t i v e P o t .

Revised: a t Z e r o

Revised: V e l o c

Revised: i t y ]

Revised: C. C ONTINUOUS E NVIRONMENTS C.2. U NDER - DAMPED P ENDULUM

Original: Under-damped Pendulum.
Revised: Under-damped Pendulum.

Original: The environment considered simulates an under-damped pendulum, where the state space comprises the angle 10 θ and angular velocity ˙ θ of the pendulum.
Revised: The environment considered simulates an under-damped pendulum, where the state space comprises the angle 15 θ and angular velocity ˙ θ of the pendulum.

Original: The dynamics are governed by the following differential equation where τ is the (time-dependent) torque applied by the agent and m , l , g are constants:
Revised: The dynamics are governed by the following differential equation where τ is the (time-dependent) torque applied by the agent and m , l , g are constants:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] [Figure] We adapt the implementation in OpenAI Gym (Brockman et al., 2016) to add an extra term α ˙ θ to the dynamics to simulate friction.
Revised: We adapt the implementation in OpenAI Gym (Brockman et al., 2016) to add an extra term α ˙ θ to the dynamics to simulate friction.

Original: In our experiments, we set g = 10 , m = l = 1 , α = 0 .
Revised: In our experiments, we set g = 10 , m = l = 1 , α = 0 .

Original: 1 and the torque τ is uniformly sampled iid.
Revised: 1 and the torque τ is uniformly sampled iid.

Original: from the interval [ − 2 , 2] .
Revised: from the interval [ − 2 , 2] .

Original: The h -Potential is parameterized by a two-layer 256-unit wide ReLU network, which is trained on 4096 trajectories of length 256 for 20000 steps of stochastic gradient descent with Adam (learning rate: 0.0001).
Revised: The h -Potential is parameterized by a two-layer 256-unit wide ReLU network, which is trained on 4096 trajectories of length 256 for 20000 steps of stochastic gradient descent with Adam (learning rate: 0.0001).

Original: The batch-size is set to 1024 and we use the trajectory regularizer with λ
Revised: The batch-size is set to 1024 and we use the trajectory regularizer with λ

Original: = 1 to regularize the network.
Revised: = 1 to regularize the network.

Original: Fig 15a plots the learned h -potential (trained with trajectory regularizer) as a function of the state ( θ, ˙ θ ) whereas Fig 15b shows the negative potential for all angles θ at zero angular velocity, i.e. ˙ θ = 0 .
Revised: Fig 19a plots the learned h -potential (trained with trajectory regularizer) as a function of the state ( θ, ˙ θ ) whereas Fig 19b shows the negative potential for all angles θ at zero angular velocity, i.e. ˙ θ = 0 .

Original: We indeed ﬁnd that states in the vicinity of θ = 0 have a larger hpotential, owing to the fact that all trajectories converge to ( θ, ˙ θ ) = 0 for large t due to the dissipative action of friction.
Revised: We indeed ﬁnd that states in the vicinity of θ = 0 have a larger hpotential, owing to the fact that all trajectories converge to ( θ, ˙ θ ) = 0 for large t due to the dissipative action of friction.

Original: C.2. C ONTINUOUS M OUNTAIN C AR
Revised: C.2. C ONTINUOUS M OUNTAIN C AR

Original: The environment 11 considered is a variation of Mountain Car (Sutton & Barto, 2011), where the state-space is a tuple ( x, ˙ x ) of the position and velocity of a vehicle on a mountainous terrain.
Revised: The environment 16 considered is a variation of Mountain Car (Sutton & Barto, 2011), where the state-space is a tuple ( x, ˙ x ) of the position and velocity of a vehicle on a mountainous terrain.

Original: The action space is the interval [ − 1 , 1] and denotes the force f applied by the vehicle.
Revised: The action space is the interval [ − 1 , 1] and denotes the force f applied by the vehicle.

Original: The dynamics of the modiﬁed environment is given by the following equation of motion:
Revised: The dynamics of the modiﬁed environment is given by the following equation of motion:

Original: [Equation]
Revised: [Equation]

Original: (28) where ζ and α are constants set to 0 .
Revised: (28) where ζ and α are constants set to 0 .

Original: 0015 and 0 .
Revised: 0015 and 0 .

Original: 1 respectively, and the velocity ˙ x is clamped to the interval [ − 0 .
Revised: 1 respectively, and the velocity ˙ x is clamped to the interval [ − 0 .

Original: Our modiﬁcation is the last α ˙ x term to simulate friction.
Revised: Our modiﬁcation is the last α ˙ x term to simulate friction.

Original: Further, the initial state ( x, ˙ x ) is sampled uniformly from the state space S =
Revised: Further, the initial state ( x, ˙ x ) is sampled uniformly from the state space S =

Original: [ − 1 . 2 , 0 . 6] × [ − 0 .
Revised: [ − 1 . 2 , 0 . 6] × [ − 0 .

Original: This can potentially be avoided if an exploratory policy is used (instead of the random policy) to gather trajectories, but we leave this for future work.
Revised: This can potentially be avoided if an exploratory policy is used (instead of the random policy) to gather trajectories, but we leave this for future work.

Original: The h -potential is parameterized by a two-layer 256-unit wide ReLU network, which is trained on 4096 trajectories of length 256 for 20000 steps of stochastic gradient descent with Adam (learning rate: 0.0001).
Revised: The h -potential is parameterized by a two-layer 256-unit wide ReLU network, which is trained on 4096 trajectories of length 256 for 20000 steps of stochastic gradient descent with Adam (learning rate: 0.0001).

Original: The batch-size is set to 1024 and we use the trajectory regularizer with λ = 1 .
Revised: The batch-size is set to 1024 and we use the trajectory regularizer with λ = 1 .

Original: C. COMPARISON WITH THE F REE -E NERGY F UNCTIONAL
Revised: C

Revised: L EARNING THE h -P OTENTIAL W HILE S IMULTANEOUSLY E XPLORING THE E NVIRONMENT

Revised: Recall that the policy we have thus-far used to gather the trajectories required to train the h -potential is random (cf.

Revised: Sec 2.2).

Revised: While the use of random policies is ubiquitous in the model-based and related literature (Ha & Schmidhuber, 2018;

Revised: Savinov et al., 2018; Kulkarni et al., 2019; Anand et al., 2019), it typically comes at a price: namely, the lack of adequate exploration in complex enough environments (Ha & Schmidhuber, 2018).

Revised: In this section, we investigate possible strategies towards approaching this problem in the context of learning an h -potential.

Revised: We stress that the results in this section are preliminary and the discussion below is intended to showcase the challenges that lie in the way; much more future work will be needed to holistically address this important issue in a principled manner.

Revised: To proceed, we consider again the environment of Mountain Car with Damping (cf.

Revised: Sec 5 and

Revised: App C.2.2), but with the amendment that the car is initialized in the valley with zero velocity.

Revised: We choose this task because it is small enough for fast iteration and easy visualization, yet it poses a difﬁcult exploration problem for an appropriate choice of environment parameters 17 , given that the car may not climb far out of the valley by simply applying a constant action.

Revised: Likewise, random actions are not enough to adequately explore the state-space in order to learn the h -potential (Fig 20).

Revised: [Figure]

Revised: The approach we describe can be thought of as bootstrapping the h -potential by using an exploratory policy in tandem with a random policy and a trajectory buffer.

Revised: The procedure (adapted from Anonymous (2019)) is as following:

Revised: Initialize a trajectory buffer and ﬁll it with trajectories from a random policy (starting at the environment speciﬁed initial states, i

Revised: e. in the valley)

Revised: [Figure]

Revised: i

Revised: [Figure]

Revised: Train (from scratch)

Revised: the h -potential with transitions available in the trajectory buffer.

Revised: Train (from scratch) an exploratory policy to minimize the h -potential (cf.

Revised: Sec 3.3).

Revised: Use the exploratory policy to transition to a difﬁcult-to-reach state.

Revised: Initialize the random policy at the said state, and use it to gather more trajectories that randomly replace a fraction 18 of the existing trajectories in the buffer.

Revised: Repeat steps 2-6.

Revised: The trajectories gathered in step 5 are mixed with previously gathered trajectories to improve the stability of the training procedure.

Revised: Moreover, note that the h -potential and the exploratory policies are reinitialized and trained from scratch at every iteration (step 6) to counteract the explorationbias detailed below.

Revised: Further, the exploratory policy is parameterized by a NoisyNet (Fortunato et al., 2017) to (locally) aid exploration.

Revised: Unfortunately, the above procedure runs in to the problem that the exploratory policy adapts to the h

Revised: -potential (in step 3), and the h -potential in turn adapts to the exploratory policy (in step 5).

Revised: This circular adaptation, which we call the exploration bias , leads to the situation where the hpotential (correctly) learns that the mountain to the right is difﬁcult to climb (Fig 21a, positions > − 0 . 52 ), whereas the mountain to the right is left unexplored (because it is initially assigned a larger values).

Revised: Consequently, the exploratory policy minimizing the h -potential will focus on climbing the mountain to the right whilst ignoring the one to the left (cf.

Revised: Fig 21a), which in results in more trajectories gathered for the right mountain (and none for the left).

Revised: To side-step the exploration bias , we resort to pre-populating the trajectory buffer with trajectories gathered by a random policy initialized at states reached by exploratory policies trained on random reward potentials.

Revised: Precisely, we replace the the h -potential by a randomly initialized neural network, use it to train the exploratory policy, use the exploratory policy for a random number of steps to transition to a state, use that state as the initial state of a random policy, gather trajectories to populate the buffer, and repeat a few times over (5 in our experiments).

Revised: The result is a trajectory buffer that is populated with a more diverse set of trajectories.

Revised: The intuition behind this heuristic is that a random reward potential will attract the exploratory policy to random locations in the state space, leading to a more diverse set of trajectories.

Revised: That said, we have not studied the efﬁciency of this strategy and its feasibility in other environments.

Revised: [Figure]

Revised: With the trajectory buffer pre-populated, we train the h -potential to obtain the result 19 in Fig 21b, which can be compared to Fig 5a.

Revised: In conclusion, we presented a preliminary investigation of strategies towards attacking the exploration issues that may arise when training the h -potential (or model-based methods in general).

Revised: Concretely, we formulated the strategy of bootstrapping (Anonymous, 2019), but found that it can be subject to what we called exploration bias .

Revised: We side-stepped the exploration bias by using a population of exploratory policies trained to maximize random reward functions, which lead to trajectories diverse enough to train the h -potential for the environment considered.

Revised: We remain optimistic that a well-crafted algorithm combining bootstrapping with random rewards might be fruitful even in complex environments, but leave a thorough investigation to future work.

Revised: C. C OMPARISON WITH THE F REE -E NERGY F UNCTIONAL

Original: The environment state at a given time-step t comprises two scalars, the x 1 ( t ) and x 2 ( t ) coordinates of the particle’s position x ( t ) .
Revised: The environment state at a given time-step t comprises two scalars, the x 1 ( t ) and x 2 ( t ) coordinates of the particle’s position x ( t ) .

Original: Recall that the dynamics is deﬁned by:
Revised: Recall that the dynamics is deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: [Figure] where X ( t ) is the stochastic process associated with the particle’s position x ( t ) .
Revised: (29) where X ( t ) is the stochastic process associated with the particle’s position x ( t ) .

Original: In our experiments, the potential is given by:
Revised: In our experiments, the potential is given by:

Original: [Equation]
Revised: [Equation]

Original: 30) which makes X ( t ) a two dimensional Ornstein-Uhlenbeck process with temperature parameter (cid:112) 2 β − 1 set to 0 .
Revised: (30) which makes X ( t ) a two dimensional Ornstein-Uhlenbeck process with temperature parameter (cid:112) 2 β − 1 set to 0 .

Original: Further, E x ∼ ρ ( · ,t ) [Ψ] (in Eqn 8) is estimated via Monte-Carlo sampling, the differential entropy E x ∼ ρ ( · ,t ) [log ρ ( · , t )] via a non-parametric estimator (Kozachenko & Leonenko, 1987; Kraskov et al., 2004; Gao et al., 2015), and the linear transform coefﬁcients for H via linear regression.
Revised: Further, E x ∼ ρ ( · ,t ) [Ψ] (in Eqn 8) is estimated via Monte-Carlo sampling, the differential entropy E x ∼ ρ ( · ,t ) [log ρ ( · , t )] via a non-parametric estimator (Kozachenko & Leonenko, 1987; Kraskov et al., 2004; Gao et al., 2015), and the linear transform coefﬁcients for H via linear regression.

Original: We train a two-layer deep, 512-unit wide network on 8092 trajectories of length 64 for 20000 steps of stochastic gradient descent with Adam (learning rate: 0.0001).
Revised: We train a two-layer deep, 512-unit wide network on 8092 trajectories of length 64 for 20000 steps of stochastic gradient descent with Adam (learning rate: 0.0001).

Original: The batch-size is set toand the network is regularized by weight decay (with coefﬁcient 0 . 0005 ).
Revised: The batch-size is set toand the network is regularized by weight decay (with coefﬁcient 0 . 0005 ).

Original: Fig 17 shows the learned h -potential as a function of position x .
Revised: Fig 22 shows the learned h -potential as a function of position x .

Original: Fig 6 compares the free-energy functional with the learnt arrow of time given by the linearly scaled H -functional.
Revised: Fig 6 compares the free-energy functional with the learnt arrow of time given by the linearly scaled H -functional.

Original: To obtain the linear scaling parameters for the H , we ﬁnd parameters w and b such that (cid:80)
Revised: To obtain the linear scaling parameters for the H , we ﬁnd parameters w and b such that (cid:80)

Original: Nt =0 ( wH [ ρ ( · , t )]
Revised: Nt =0 ( wH [ ρ ( · , t )]

Original: + b − F [ ρ ( · , t )]) 2 is minimized (constraining w to be positive), i.e. by solving a linear regression problem.
Revised: + b − F [ ρ ( · , t )]) 2 is minimized (constraining w to be positive), i.e. by solving a linear regression problem.

Original: Finally, Fig 17 plots h as a function of state x ∈ R 2 , whereas Fig 6 shows that after appropriate (linear) scaling, the learned
Revised: Finally, Fig 22 plots h as a function of state x ∈ R 2 , whereas Fig 6 shows that after appropriate (linear) scaling, the learned

Original: H largely agrees with the true F .
Revised: H largely agrees with the true F .

Original: The linear adjustment is done to account for the arbitrary scaling of H and F ; the crucial detail is that H is also a Lyapunov functional of the dynamics, i.e. it decreases montonously with time.
Revised: The linear adjustment is done to account for the arbitrary scaling of H and F ; the crucial detail is that H is also a Lyapunov functional of the dynamics, i.e. it decreases montonously with time.

Original: This arbitrariness results from various aspects.
Revised: This arbitrariness results from various aspects.

Original: First, observe that while the dynamics in Eqn 29 is invariant to a constant shift in potential Ψ , the Free Energy functional F is not – from Eqn 8, we see that adding a constant to the potential Ψ results in the same constant being added to F (for all t ).
Revised: First, observe that while the dynamics in Eqn 29 is invariant to a constant shift in potential Ψ , the Free Energy functional F is not – from Eqn 8, we see that adding a constant to the potential Ψ results in the same constant being added to F (for all t ).

Original: This justiﬁes adding a constant shift to H as appropriate.
Revised: This justiﬁes adding a constant shift to H as appropriate.

Original: Moreover, the scale of h -potential is controlled by the regularizing coefﬁcient λ , which is arbitrary with respect to the scale of the Free
Revised: Moreover, the scale of h -potential is controlled by the regularizing coefﬁcient λ , which is arbitrary with respect to the scale of the Free

Original: Energy functional F .
Revised: Energy functional F .