Original: Explore and Control with Adversarial Surprise
Revised: Explore and Control with Adversarial Surprise

Original: Abstract
Revised: Abstract

Original: Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-speciﬁed rewards.
Revised: Reinforcement learning (RL) provides a framework for learning goal-directed policies given user-speciﬁed rewards.

Original: However, since  rewards can be sparse and task-speciﬁc, we are interested in the problem of learning without rewards, where agents must discover useful behaviors in the absence of domain-speciﬁc incentives.
Revised: However, since designing rewards often requires substantial engineering effort, we are interested in the problem of learning without rewards, where agents must discover useful behaviors in the absence of task-speciﬁc incentives.

Original: Intrinsic motivation is a family of unsupervised RL techniques which develop general objectives for an RL agent to optimize that lead to better exploration or the discovery of skills.
Revised: Intrinsic motivation is a family of unsupervised RL techniques which develop general objectives for an RL agent to optimize that lead to better exploration or the discovery of skills.

Original: In this paper, we propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences.
Revised: In this paper, we propose a new unsupervised RL technique based on an adversarial game which pits two policies against each other to compete over the amount of surprise an RL agent experiences.

Original: The policies each take turns controlling the agent.
Revised: The policies each take turns controlling the agent.

Original: The Explore policy maximizes entropy, putting the agent into surprising or unfamiliar situations.
Revised: The Explore policy maximizes entropy, putting the agent into surprising or unfamiliar situations.

Original: Then, the Control policy takes over and seeks to recover from those situations by minimizing entropy.
Revised: Then, the Control policy takes over and seeks to recover from those situations by minimizing entropy.

Original: The game harnesses the power of multi-agent competition to drive the agent to seek out increasingly surprising parts of the environment while learning to gain mastery over them, leading tobetter exploration and the emergence of complex skills.
Revised: The game harnesses the power of multi-agent competition to drive the agent to seek out increasingly surprising parts of the environment while learning to gain mastery over them. We show empirically that our method leads to the emergence of complex skills by exhibiting clear phase transitions.

Original: Theoretically, we show  that under certain assumptions, this game pushes the agent tofully explore the latent state space of stochastic, partially-observed environments, whereas prior techniques will not.
Revised: Furthermore, we show both theoretically –via a latent state space coverage argument– and empirically that our method has the potential to be applied to the exploration of stochastic, partially-observed environments.

Original: Empirically, we demonstrate thateven with no external rewards, Adversarial Surprise learns more complex behaviors, and explores more effectively than competitive baselines, outperforming intrinsic motivation methods based on active inference, novelty-seeking (Random Network Distillation 
Revised: We show that Adversarial Surprise learns more complex behaviors, and explores more effectively than competitive baselines, outperforming intrinsic motivation methods based on active inference, novelty-seeking (Random Network Distillation (RND)), and multi-agent unsupervised RL (Asym- Equal contribution 1 UC Berkeley 2 Google Research,

Original: (RND)){+Accepted at the ICML 
+}
Revised: {+Accepted at the ICML 
+}

Original: , and multi-agent unsupervised RL (Asymmetric Self-Play (ASP
Revised: Workshop on Unsupervised metric Self-Play (ASP)) in MiniGrid, Atari and

Revised: VizDoom environments.

Original: 1. Introduction
Revised: 1. Introduction

Original: Despite promising results across a number of domains (e.g., (Berner et al., 2019; Kober et al., 2013; Levine et al., 2016;
Revised: Despite promising results across a number of domains (e.g., (Berner et al., 2019; Kober et al., 2013; Levine et al., 2016;

Original: Vinyals et al., 2019)), a major challenge in reinforcement learning (RL) is that the effectiveness of current methods depends heavily on task-speciﬁc rewards (Riedmiller et al., 2018; Vecerik et al., 2017).
Revised: Vinyals et al., 2019)), a major challenge in reinforcement learning (RL) is that the effectiveness of current methods depends heavily on task-speciﬁc rewards (Riedmiller et al., 2018; Vecerik et al., 2017).

Original: For many tasks,  rewards are sparse, making learning slow and difﬁcult (Riedmiller et al., 2018; Vecerik et al., 2017).
Revised: For many tasks, designing dense and informative rewards require signiﬁcant engineering effort, and sparse rewards make learning slow and difﬁcult (Riedmiller et al., 2018; Vecerik et al., 2017).

Original: Yet humans and animals easily learn from their own experience without being constantly told what to do.
Revised: Yet humans and animals easily learn from their own experience without being constantly told what to do.

Original: Therefore, signiﬁcant prior research has focused on developing a class of unsupervised RL methods that optimize for intrinsic motivation (IM) – alternative objectives (such as seeking novelty) that incentivize the agent to autonomously explore and learn about the world without being entirely dependent on explicit goals and task incentives.
Revised: Therefore, signiﬁcant prior research has focused on developing a class of unsupervised RL methods that optimize for intrinsic motivation (IM) – alternative objectives (such as seeking novelty) that incentivize the agent to autonomously explore and learn about the world without being entirely dependent on explicit goals and task incentives.

Original: Many prior works have approached the problem of unsupervised RL as one of pure novelty-seeking, training agents to optimize for surpriseor prediction error in the hopes of facilitating better exploration, which can be helpful for learning downstream tasks.
Revised: Many prior works have approached the problem of unsupervised RL as one of pure novelty-seeking, training agents to optimize for surprise in the hopes of facilitating better exploration, which can be helpful for learning downstream tasks.

Original: While better exploration is a laudable goal, such objectives may be ill-suited to stochastic environments, because they can become distracted by inherently high-entropy elements; this is known as the “noisy TV” problem  (Schmidhuber, 2010).
Revised: However, current novelty-seeking methods don’t always produce the desired explorative behavior. In stochastic environments for example, such methods can suffer from the “noisy TV” problem where the agent becomes distracted by inherently high-entropy elements (Schmidhuber, 2010).

Original: In contrast,  active inference methods (Friston, 2009; Friston et al., 2009; Berseth et al., 2019) deal with highly entropic environments by attempting to minimize surprise.
Revised: In contrast, some biologically-inspired approaches like active inference suggest that complex human-like behaviors are obtained by minimizing the expected surprise on future observations (Friston, 2009; Friston et al., 2009; Berseth et al., 2019).

Original: However, these techniques suffer from the “dark room” problem (Friston et al., 2012); that is, they may get stuck because they prefer to stay in highly predictable, unsurprising states.
Revised: However, in the absence of an informative observation prior that guide the agent towards interesting behaviors, such methods suffer from the “dark room” problem (Friston et al., 2012): if what the agent can observe from the environment is not surprising, the agent will not be incentivized to learn any complex behavior.

Original: This is especially problematic in partially-observed environments.
Revised: This is especially problematic in partially-observed environments.

Original: Yet humans  maintain a balance between optimizing for both novelty and familiarity.
Revised: Yet humans seem to always maintain a tension between opti- mizing for both novelty and familiarity.

Original: Thus, we propose that the role of unsupervised RL is more nuanced: beyond just exploring to ﬁnd surprising states, the goal should be to also achieve meaningful control over a diverse set of states.

Original: For example, a child in a play room does not just try to toss their toys on the ﬂoor in every possible pattern, but instead tries to stack them together, ﬁnd new uses for parts, or combine them in various ways.
Revised: For example, a child in a play room does not just try to toss their toys on the ﬂoor in every possible pattern, but instead tries to stack them together, ﬁnd new uses for parts, or combine them in various ways.

Original: In the same way, an effective unsupervised RLagent should aim to explore the state space while still learning increasingly complex and interesting behaviors. The Explore policy acts ﬁrst, and tries to put the agent into surprising, high entropy states. On its turn, the Control policy tries to minimize surprise by ﬁnding familiar, lowentropy, predictable states. Figure (b) shows an example rollout of a game.
Revised: In the same way, we suggest that the right objective for unsupervised RL should be to actively maintaining a tension between exploration and control.

Original: In the Explore policy’s ﬁrst turn, it attempts to maximize entropy by remaining in the ﬁrst room with ﬂashing lights (a noisy TV state). When the Control policy takes over, it is able to take an action to control the environment, by ﬂipping the switch to stop the lights from ﬂashing. Therefore, when the Explore policy acts again (c) it moves into the next room; this is predicted by our theoretical results, which show that the Explore policy must maximize entropy over the true state space. During the 
Revised: Such method would have the potential to produce increasingly complex and interesting behaviors without the limitations of optimizing for novelty or familiarity alone.

Original: Control policy’s next turn it remains in the dark room.

Original: This process continues until all of the rooms have been visited.

Original: In this paper, we frame the quest for both exploration and control as an adversarial game between two policies, which each take turns sequentially acting for the same RL agent.
Revised: In this paper we maintain a tension between exploration and control by formulating an adversarial game between two policies, which each take turns sequentially acting for the same RL agent.

Original: The Explore policy is novelty-seeking, and attempts to maximize surprise(or observation prediction error) over the course of the episode, putting the agent into a diverse range of novel states.
Revised: The Explore policy is novelty-seeking, and attempts to maximize surprise over the course of the episode, putting the agent into a diverse range of novel states.

Original: In turn, the Control policy must minimize surprise, by learning to manipulate its environment in order to return to safe and predictable states.
Revised: In turn, the Control policy must minimize surprise, by learning to manipulate its environment in order to return to safe and predictable states.

Original: When combined, the two adversaries engage in an arms race, repeatedly putting the agent into challenging new situations, then attempting to gain control of those situations.
Revised: When combined, the two adversaries engage in an arms race, repeatedly putting the agent into challenging new situations, then attempting to gain control of those situations.

Original: Figure 1 shows an illustration of the method, including a sample interaction.
Revised: Figure 1 shows an illustration of the method, including a sample interaction.

Original: Rather than simply adding noise to the environment, the Explore policy learns to adapt to the Control policy, and  search for increasingly challenging situations from which the Control policy must recover.
Revised: Rather than simply adding noise to the environment, the Explore policy learns to adapt to the Control policy, and to search for increasingly challenging situations from which the Control policy must recover.

Original: Thus our method, Adversarial Surprise (AS), leverages the power of multi-agent training to generate a curriculum of increasingly challenging exploration and control problems.
Revised: Thus our method, Adversarial Surprise (AS), leverages the power of multi-agent training to generate a curriculum of increasingly challenging exploration and control problems.

Revised: We show empirically that the competition between the two policies produces the emergence of increasingly complex observable behaviors in the agent by exhibiting clear phase transitions.

Original: Both our theoretical and empirical results demonstrate that while other IM and unsupervised RL techniques can fail in stochastic, partially-observed environments, AS learns to both fully explore the state space, and control it.
Revised: Furthermore, we show theoretically and empirically that the increasingly complex control and exploration strategies found by the competing policies enable the efﬁcient exploration of stochastic, partially-observable environments that previous methods optimising for novelty or familiarity alone fail to explore.

Original: Our main contribution is the Adversarial Surprise (AS) algorithm, a method that frames the unsupervised RL problem as an adversarial game between an Explore and Control policy.
Revised: Our main contribution is the Adversarial Surprise (AS) algorithm, an unsupervised RL method to actively maintain a tension between exploration and control which leads to the emergence of increasingly complex behaviors.

Original: We derive our method, and provide a theoretical analysis in the recently introduced Block MDP setting (Du et al., 2019) which shows  that our method maximizes state coverage in partially-observed, stochastic environments.
Revised: We derive our method, and perform a theoretical analysis in the recently introduced Block MDP setting (Du et al., 2019) which shows –via a latent space coverage argument– that our method can be applied to the exploration of partiallyobserved, stochastic environments.

Original: We present a practical instantiation based on deep reinforcement learning, and provide an empirical evaluation that compares our approach to recent intrinsic motivation and unsupervised RL techniques.
Revised: We present a practical instantiation based on deep reinforcement learning, and provide an empirical evaluation that compares our approach to recent intrinsic motivation and unsupervised RL techniques.

Original: Our results demonstrate that our method learns complex behaviors in the absence of any external rewards.
Revised: Our results demonstrate that our method induces the emergence of complex behaviors that can be used for both control and exploration.

Original: In tasks involving both distracting stochastic elements and complex environment dynamics, Adversarial Surprise outperforms both surprise-minimizing techniques like SMiRL (Berseth et al., 2019), and noveltyseeking methods like Random Network Distillation (RND) (Burda et al., 2018).
Revised: As suggested by our theoretical results, we show empirically that our method can be applied to the exploration of partially-observable, stochastic environments, outperforming previous methods like Random Network Distillation (RND) (Burda et al., 2018), Suprise-Minimization

Original:  We also show that AS learns more quickly than a recently-proposed unsupervised RL method based on multi-agent competition, Asymmetric Self Play (ASP) (Sukhbaatar et al., 2017; OpenAI et al., 2021).
Revised: RL (SMiRL) (Berseth et al., 2019) and Asymmetric Self Play (ASP). We also show that AS produces more meaningful behaviors in VizDoom ( ? ) and Atari (Bellemare et al., 2013) without any external reward.

Original: AS learns to explore more effectively than these prior works, takes more actions to control its environment, and learns complex behavior purely through optimizing its intrinsic motivation.

Original: 2. Related Work
Revised: 2. Related Work

Original: Novelty-seeking intrinsic motivation: One of  the most frequently studied forms of intrinsic motivation is noveltyseeking, or curiosity (e.g., (Bellemare et al., 2016;
Revised: Our method enables the emergence of complex skills without supervision and thus enters into the budding ﬁeld of unsupervised RL that we brieﬂy survey below. A common strategy in this ﬁeld is to formulate a task-agnostic objective that uses environment statistics only and to derive an intrinsic reward that enables exact or approximate optimization of this objective using standard RL algorithms. A recurrent question in this ﬁeld is the applicability of the skills learned.

Original: Pathak et al., 2017; Schmidhuber, 1991; Still and Precup, 2012)). A novelty-seeking agent is motivated to explore its environment, through incentives to maximize model uncertainty (Still and Precup, 2012; Shyam et al., 2019; Pathak et al., 2019), entropy (Eysenbach et al., 2018; Hazan et al., 2019;
Revised: We will show that our method has the potential to be applied to the exploration of stochastic, partially-observable environments. Novelty-seeking intrinsic motivation: One of the most frequently studied forms of intrinsic motivation is noveltyseeking, or curiosity, which can be formulated as maximizing the information gain on the agent’s dynamics model of the environment (Houthooft et al., 2016; Still & Precup, 2012; Bellemare et al., 2016; Pathak et al., 2017; Schmidhuber, 1991).

Original: Lee et al., 2019; Pong et al., 2019), prediction error (Pathak et al., 2017; Yamamoto and Ishikawa, 2010), or surprise (Achiam and Sastry, 2017; Schmidhuber, 1991; Sun et al., 2011).
Revised: Common intrinsic rewards used to approximate curiosity include surprise (Achiam & Sastry, 2017; Schmidhuber, 1991; Yamamoto & Ishikawa, 2010; Pathak et al., 2017; Burda et al., 2018) and ensemble disagreement (Shyam et al., 2019; Pathak et al., 2019).

Original:  However, na¨ıvely maximizing uncertainty, predic- tion error, or entropy leads to methods that are vulnerable to the noisy TV problem, where the agent becomes distracted by inherently high-entropy, unpredictable elements in the environment, such as white noise (e.g. static on a TV)
Revised: A novelty-seeking agent is motivated to explore its environment. However, na ¨ ıvely maximizing surprise leads to methods that are vulnerable to the noisy TV problem, where the agent becomes distracted by inherently high-entropy, unpredictable elements in the environment, such as white noise (e.g. static on a TV)

Original: (Schmidhuber, 2010).
Revised: (Schmidhuber, 2010).

Original: In this case, a curious agent will not be able to learn meaningful behaviours.
Revised: In this case, a curious agent will not be able to learn meaningful behaviours.

Original: We will show that AS overcomes this problem, and fully explores the state space even when there are highly stochastic elements.
Revised: We will show that AS overcomes this problem, and fully explores the state space even when there are highly stochastic elements.

Original: Surprise minimization and the free energy principle:
Revised: Surprise minimization and the free energy principle:

Original: Rather than maximizing surprise, the free energy principle (Faraji et al., 2018; Friston, 2009; Friston et al., 2009; 2016;
Revised: Rather than maximizing surprise, the free energy principle (Faraji et al., 2018; Friston, 2009; Friston et al., 2009; 2016;

Original: Ueltzh¨offer, 2018) proposes that biological agents actually minimize  surprise(or state prediction error) by seeking familiar, stable states, and controlling their environment to make it more predictable.
Revised: Ueltzh¨offer, 2018) proposes that biological agents actually minimize long-term average surprise by seeking familiar, stable states, and controlling their environment to make it more predictable.

Original: Inspired by this idea, Berseth et al.
Revised: Inspired by this idea, Berseth et al.

Original: (2019) built a surprise-minimizing RL agent, SMiRL, which outperforms curiosity-maximizing methods like ICM (Pathak et al., 2017) in high-entropy environments.
Revised: (2019) built a surprise-minimizing RL agent, SMiRL, which outperforms curiosity-maximizing methods like ICM (Pathak et al., 2017) in high-entropy environments.

Original: However, in partially observed or low-entropy environments, surprise minimization is vulnerable to the dark room problem: if the agent can simply stay in a highly-predictable part of the environment where nothing happens, it will (Friston et al., 2012).
Revised: However, in partially observed or low-entropy environments, surprise minimization is vulnerable to the dark room problem: if the agent can simply stay in a highly-predictable part of the environment where nothing happens, it will (Friston et al., 2012).

Original: In this scenario, a surprise-minimizing agent cannot learn meaningful behaviors either.
Revised: In this scenario, a surprise-minimizing agent cannot learn meaningful behaviors either.

Original: AS is designed to avoid this problem, because the Explore policy will not allow the Control policy to remain in a dark room.
Revised: AS is designed to avoid this problem, because the Explore policy will not allow the Control policy to remain in a dark room.

Original: Mutual information and empowerment:

Original: Alternative approaches to intrinsic motivation may be based on mutual information (e.g. (Houthooft et al., 2016)).

Original: For example, the goal of empowerment (Klyubin et al., 2005; Salge et al., 2014b) is to maximize the mutual information between the agent’s actions and its future states.
Revised: Empowerment: The goal of empowerment (Klyubin et al., 2005; Salge et al., 2014b) is to maximize the mutual information between the agent’s actions and its future states.

Original: Empowerment encourages agents to “keep their options open” by exploring many states, while still maintaining a high degree of control in those states.
Revised: Empowerment encourages agents to “keep their options open” by exploring many states, while still maintaining a high degree of control in those states.

Original: Maximizing measures of empowerment between agents has also been explored (Guckelsberger et al., 2016; 2018; Jaques et al., 2019).

Original: However, calculating empowerment in high dimensional environments is intractable, leading to various methods for approximating it (e.g., (Karl et al., 2015; Salge et al., 2014a; Zhao et al., 2020; de Abril and Kanai, 2018;
Revised: However, calculating empowerment in high dimensional environments is intractable, leading to various methods for approximating it (e.g., (Karl et al., 2015; Salge et al., 2014a; Zhao et al., 2020; de Abril & Kanai, 2018;

Original: Zhang et al., 2020)) for example, using variational inference (Mohamed and Rezende, 2015).
Revised: Zhang et al., 2020; Jaques et al., 2019; Mohamed & Rezende, 2015)).

Original: Unfortunately, these methods can also be difﬁcult to get working with high dimensional function approximation (Gregor et al., 2016).
Revised: Unfortunately, these methods can also be difﬁcult to get working with high dimensional function approximation (Gregor et al., 2016).

Original: In contrast, we show that Adversarial Surprise works with deep neural networks applied to pixel inputs in Atari environments.
Revised: In contrast, we show that Adversarial Surprise works with deep neural networks applied to pixel inputs in Atari environments.

Original: Multi-agent curriculum generation: Multi-agent competition can provide a mechanism for driving RL agents to automatically learn increasingly complex behavior (Leibo et al., 2019).
Revised: Emergence in multi-agent setting: Multi-agent competition can provide a mechanism for driving RL agents to automatically learn increasingly complex behavior (Leibo et al., 2019).

Original: As each agent adapts, it makes the learning problem for the other agent increasingly difﬁcult, leading to the emergence of an automatic curriculum of challenging learning tasks (Baker et al., 2019; Dennis et al., 2020; Xu et al., 2020).
Revised: As each agent adapts, it makes the learning problem for the other agent increasingly difﬁcult, leading to the emergence of an automatic curriculum of challenging learning tasks (Baker et al., 2019; Dennis et al., 2020; Xu et al., 2020; ? )

Original: For example, Schmidhuber (1997) proposed having two classiﬁers compete by repeatedly selecting examples which they can classify but which the other cannot.
Revised: For example, Schmidhuber (1997) proposed having two classiﬁers compete by repeatedly selecting examples which they can classify but which the other cannot.

Original: Asymmetric Self Play (ASP) (Sukhbaatar et al., 2017; OpenAI et al., 2021) rewards one agent, Alice, for executing the shortest trajectory that a second agent, Bob, cannot copy (or reverse).
Revised: Asymmetric Self Play (ASP) (Sukhbaatar et al., 2017; OpenAI et al., 2021) rewards one agent, Alice, for executing the shortest trajectory that a second agent, Bob, cannot copy (or reverse).

Original: ASP is limited to training goal-conditioned policies, and is thus less applicable to standard RL tasks where the reward-maximizing states are not known a priori. Adversarial Surprise does not have this limitation, and instead serves as a general algorithm that can speed learning across a variety of contexts.
Revised: Similarly to ASP, Adversarial Surprise is formulated as an adversarial game between two policies. However, unlike ASP our method is formulated in terms of general information theoretic quantities which make it more generally applicable.

Original: More importantly, we showin this work that ASP can fail to explore stochastic environments, because Alice can easily produce a random goal state which Bob is not able to reproduce.
Revised: For example, we show that ASP can fail to explore stochastic environments, because Alice can easily produce a random goal state which Bob is not able to reproduce.

Original: In contrast, AS works well in stochastic environments, outperforming ASP.
Revised: In contrast, AS works well in stochastic environments, outperforming ASP.

Original: 3. Background
Revised: 3. Background

Original: Partially Observed Markov Decision Process:
Revised: Partially Observed Markov Decision Process:

Original: A
Revised: A

Original: POMDP is a tuple ( S , A , T , O , r, γ ) , where s ∈ S are states, a ∈ A are actions, r ( a, s ) is the reward function, and γ ∈ [0 , 1) is a discount factor.
Revised: POMDP is a tuple ( S , A , T , O , r, γ ) , where s ∈ S are states, a ∈ A are actions, r ( a, s ) is the reward function, and γ ∈ [0 , 1) is a discount factor.

Original: The environment is partially observed, so the agent cannot observe the true state s , but rather observes o ∼ p
Revised: The environment is partially observed, so the agent cannot observe the true state s , but rather observes o ∼ p

Original: ( O | s ) .
Revised: ( O | s ) .

Original: At each timestep t , the agent selects an action a t according to its policy π ( a t | o t ) , receives reward r ( a t , s t ) , and the environment transitions to the next state according to T ( s t +1 | s t , a t ) .
Revised: At each timestep t , the agent selects an action a t according to its policy π ( a t | o t ) , receives reward r ( a t , s t ) , and the environment transitions to the next state according to T ( s t +1 | s t , a t ) .

Original: We are interested in stochastic environments, in which the emission distribution distribution is inherently entropic for some states, i.e.
Revised: We are interested in stochastic environments, in which the emission distribution distribution is inherently entropic for some states, i.e.

Original: ∃ s : H ( p ( O | s )) > 0 .
Revised: ∃ s : H ( p ( O | s )) > 0 .

Original: Block Markov Decision Process: A BMDP (Du et al., 2019) is a POMDP with an additional disjointness assumption: for any s, s (cid:48) ∈ S , s (cid:54) =
Revised: Block Markov Decision Process: A BMDP (Du et al., 2019) is a POMDP with an additional disjointness assumption: for any s, s (cid:48) ∈ S , s (cid:54) =

Original: s (cid:48)
Revised: s (cid:48)

Original: ⇒ supp ( p ( O | s ))
Revised: ⇒ supp ( p ( O | s ))

Original: ∩ supp ( p ( O | s ))
Revised: ∩ supp ( p ( O | s ))

Original: = ∅ .
Revised: = ∅ .

Original: Our theoretical results show that the
Revised: Our theoretical results show that the

Original: AS algorithm fully covers the latent state space of a large family of BMDPs.
Revised: AS algorithm fully covers the latent state space of a large family of BMDPs.

Original: Intrinsic motivation: IM can either be used in combination with a task objective, in which case intrinsic motivation serves to facilitate exploration, or by itself, in which case the agent receives no external task rewards, and aims only to maximise its intrinsic objective, leading it to learn skills that may potentially be useful for downstream tasks.
Revised: Intrinsic motivation: IM can either be used in combination with a task objective, in which case intrinsic motivation serves to facilitate exploration, or by itself, in which case the agent receives no external task rewards, and aims only to maximise its intrinsic objective, leading it to learn skills that may potentially be useful for downstream tasks.

Original: In this paper, we study how an agent can learn skilled behaviour without rewards.
Revised: In this paper, we study how an agent can learn skilled behaviour without rewards.

Original: Therefore, we consider agents that seek to optimize cumulative intrinsic reward over the episode:
Revised: Therefore, we consider agents that seek to optimize cumulative intrinsic reward over the episode:

Original: [Equation]{+R = (cid:80) Tt =
+}
Revised: {+R = (cid:80) Tt =
+}

Original:  Surprise minimizing agents (SMiRL): The free energy principle (Faraji et al., 2018; Friston, 2009; Friston et al., 2009; 2016; Ueltzh¨offer, 2018) suggests that biological agents may minimize surprise, or state entropy, in order to remain in safe and stable states.
Revised: γ t r i ( a t , s t ) . Surprise minimizing agents (SMiRL): The free energy principle (Faraji et al., 2018; Friston, 2009; Friston et al., 2009; 2016; Ueltzh¨offer, 2018) suggests that biological agents may minimize surprise, or state entropy, in order to remain in safe and stable states.

Original: Minimising surprise for an RL agent requires keeping track of the state history of the agent via a density model, p θ ( s ) , which bounds the state marginal density of the policy, d π , given states seen so far in an episode, τ t = { s 0 , ...
Revised: Minimising surprise for an RL agent can be done by keeping track of the state history of the agent via a density model, p θ ( s ) , which bounds the state marginal density of the policy, d π , given states seen so far in an episode, τ t = { s 0 , ...

Original: s t } (Berseth et al., 2019).
Revised: s t } .

Original:  Berseth et al.
Revised: Indeed, Berseth et al.

Original: (2019) shows that we can use the intrinsic reward r i ( s t )
Revised: (2019) shows that we can use the intrinsic reward r i ( s t )

Original: = log p θ ( s ) to minimize the entropy of the state marginal distribution H ( d π ( s )) .
Revised: = log p θ ( s ) to minimize the entropy of the state marginal distribution H ( d π ( s )) : [Equation]

Revised: where the bound becomes tight as log p θ ( s ) → d π ( s t ) ; that is, as the density model approaches the true state marginal density.

Original: A SMiRL agent trained with this objective learns emergent behaviors to reduce entropy in stochastic environments—such as stable walking robots or playing Tetris—even in the absence of any external reward.
Revised: A SMiRL agent trained with this objective learns emergent behaviors to reduce entropy in stochastic environments—such as stable walking robots or playing Tetris—even in the absence of any external reward.

Original: However, when applied to partially observed environments, the agent is susceptible to the dark room problem; rather than learning to control the environment, it can simply control its observations by remaining in unsurprising parts of the environment.
Revised: However, when applied to partially observed environments, the agent is susceptible to the dark room problem; rather than learning to control the environment, it can simply control its observations by remaining in unsurprising parts of the environment.

Original: Or, simply turning to look at a wall.
Revised: Or, simply turning to look at a wall.

Original: In our method, we build on surprise minimization, incorporating it into a two player game that alleviates this shortcoming.
Revised: In our method, we build on surprise minimization, incorporating it into a two player game that alleviates this shortcoming.

Original: 4. Adversarial Surprise
Revised: 4. Adversarial Surprise

Original: The goal of Adversarial Surprise (AS) is to facilitate exploring and learning to control stochastic, partially-observed environments.
Revised: The goal of Adversarial Surprise (AS) is to produce complex behaviors that can be used for exploration and control.

Original: To this end, AS pits two policies against each other in a two-player competition over the amount of surprise an RL agent experiences.
Revised: To this end, AS pits two policies against each other in a twoplayer competition over the amount of surprise an RL agent experiences.

Original: Speciﬁcally, we learn an Explore policy, π E , and a Control policy, π C .
Revised: Speciﬁcally, we learn an Explore policy, π E , and a Control policy, π C .

Original: The goal of the Control policy is to minimize its own surprise, or observation entropy, using a learned model p θ ( o ) .
Revised: The goal of the Control policy is to minimize its own surprise, or observation entropy, using a learned model p θ ( o ) .

Original: However, the Explore policy’s goal is to maximize the surprise that the Control policy experiences.
Revised: However, the Explore policy’s goal is to maximize the surprise that the Control policy experiences.

Original: The policies take turns taking actions for the agent, switching back and forth throughout the episode. The policy controlling the RL agent change every k steps, such that: [Equation]
Revised: The policies take turns taking actions for the agent, switchAlgorithm

Original: Each policy is given several steps to act because it enables it to reach states that will be challenging for the other policy to recover from, thus facilitating learning more complex and long-term exploration and control behaviors (see Figure 1).

Original: To estimate surprise, the policies share a learned density model, which estimates the likelihood of experiencing observation o , p θ ( o ) .

Original: Because the Control policy is surpriseminimizing, its reward is r iC ( s t )

Original: = log p θ ( o t ) , which resembles SMiRL (Berseth et al., 2019), except using the observation in place of the state.

Original: The goal of the Explore policy is to maximize the observation surprise of the RL agent when the Control policy is in control.

Original: This creates an adversarial game, in which the Explore policy attempts to ﬁnd surprising situations with which to expose the Control

Original: Randomly initialize φ E and φ Cfor episode = 0 , ..., Mdo Initialize θ, R i = 0 , β ← {} , explore turn =
Revised: Randomly initialize φ E and φ C episode = 0 , ..., M Initialize θ, R i = 0 , β ← {} , explore turn =

Original: True, t C =
Revised: True, t C =

Original: k , s 0 ∼ p ( s 0 ) , o 0 ∼ p ( O 0 | s 0 )for t = 0, ..., Tdo if explore turnthen a t ∼ π E ( o t , h Et ) else a t ∼ π C ( o t , 
Revised: k , s 0 ∼ p ( s 0 ) , o 0 ∼ p ( O 0 | s 0 ) t ← 0 T explore turn a t ∼ π E ( o t , h Et ) *Explore a t ∼ π C ( o t , h Ct )

Original: h Ct ) end if s t +1 ∼ T ( s t +1 | s t , a t ) , o t +1 ∼ p ( O t +1 | 
Revised: *Control s t +1 ∼ T ( s t +1 | s t , a t ) , o t +1 ∼ p ( O t +1 | s t +1 )

Original: s t +1 ) r it 
Revised: *Environment step r it =

Original: = log p θ ( o t +1 ) if not explore turn and t − t C > k/ 2 then
Revised: log p θ ( o t +1 ) *Compute intrinsic reward not explore turn and t − t C > k/ 2 R i

Original: R i = R i + r 
Revised: = R i + r i β

Original: i end if β = β ∪ { o t , a t , o t +1 } if t ==
Revised: = β ∪ { o t , a t , o t +1 } *Update buffer t ==

Original: t Cthen explore turn = not explore turn  t C
Revised: t C explore turn = not explore turn *Switch turns t C

Original: = t C + 2 kend if θ t +1 =
Revised: = t C + 2 k θ t +1 =

Original: MLE update ( β, θ t ) end for φ E = RL update ( β, − R i ) φ C = RL update ( β, R i ) end for policy, and the Control policy’s objective is to recover from them.
Revised: MLE update ( β, θ t ) *Fit density model φ E = RL update ( β, − R i ) * Train Explore policy π E with reward − R [Equation]

Revised: update

Original: Therefore, the Explore policy’s reward is based on the surprise for the observations of the Control policy.
Revised: [Equation] Train Control policy π C with reward R i ing back and forth throughout the episode. The policy controlling the RL agent change every k steps, such that:

Revised: [Equation]

Revised: Each policy is given several steps to act because it enables it to reach states that will be challenging for the other policy to recover from, thus facilitating learning more complex and long-term exploration and control behaviors (see Figure 1).

Original: Assume that the Control policy’s turn begins at timestep t C , and it receives a total reward of R i
Revised: To estimate surprise, we learn a density model which estimates the agent’s likelihood of experiencing observation o , p θ ( o ) . Because the Control policy is surprise-minimizing, its reward is r iC ( s t ) = log p θ ( o t ) , which resembles SMiRL (Berseth et al., 2019), except using the observation in place of the state. The goal of the Explore policy is to maximize the observation surprise of the RL agent when the Control policy is in control.

Revised: This creates an adversarial game, in which the Explore policy attempts to ﬁnd surprising situations with which to expose the Control policy, and the

Original:  = (cid:80) t C + k t = t C γ k r
Revised: Control policy’s objective is to recover from them. Therefore, the Explore policy’s reward is based on the surprise for the observations of the Control policy. Assume that the Control policy’s turn begins at timestep t C , and it receives a total reward of R i = (cid:80) t C + k t = t C γ k r

Original: i ( a t , s t ) for that turn.
Revised: i ( a t , s t ) for that turn.

Original: Then, the Explore policy’s reward is − R i , and is applied to the last timestep of the Explore policy’s turn (i.e. timestep t C − 1 ).
Revised: Then, the Explore policy’s reward is − R i , and is applied to the last timestep of the Explore policy’s turn (i.e. timestep t C − 1 ).

Original: The full training procedure for Adversarial Surprise is given in Algorithm 1.
Revised: The full training procedure for Adversarial Surprise is given in Algorithm 1.

Original: Thus, Adversarial Surprise deﬁnes the following adversarial game between the two agents:
Revised: Thus, Adversarial Surprise deﬁnes the following adversarial game between the two policies:

Original: [Equation]
Revised: [Equation]

Original: where the Explore policy can only effect p ( o t ) through the ﬁnal state that it produces at the end of its turn, which is the initial state for the Control policy. 
Revised: where the Explore policy can only effect p ( o t ) through the ﬁnal state that it produces at the end of its turn, which is the initial state for the Control policy. We show in the

Revised: Appendix the equivalent of eq.

Original: We show in the Appendix that the objective of the Explore policy approaches maximizing the entropy of the Control policy’s observations:
Revised: 1 in the partially observable setting. As a corrolary, the objective of the Explore policy approaches maximizing the entropy of the Control policy’s observations:

Original: [Equation]
Revised: [Equation]

Original: Analogously, the Control policy’s goal is to minimize entropy:
Revised: Analogously, the Control policy’s goal is to minimize entropy:

Original: [Equation]
Revised: [Equation]

Original: Implementation details: We parameterize the policies for the Explore and Control policy using deep neural networks (NN) with parameters φ E and φ C , respectively.
Revised: Implementation details: We parameterize the policies for the Explore and Control policy using deep neural networks (NN) with parameters φ E and φ C , respectively.

Original: We policy is based on a convolutional NN which conditions on a stack the last 4 observation frames.
Revised: We policy is based on a convolutional NN which conditions on a stack the last 4 observation frames.

Original: The networks are trained using Proximal Policy Optimization (PPO) (Schulman et al., 2017); further details and hyperparameters are given in the Appendix.
Revised: The networks are trained using Proximal Policy Optimization (PPO) (Schulman et al., 2017); further details and hyperparameters are given in the Appendix.

Original: Following (Berseth et al., 2019), theshared density model p θ ( o ) is re-initialized each episode and trained using maximum likelihood estimation (MLE) to ﬁt the observations of both agents within a single episode, which are stored in a buffer β .
Revised: Following (Berseth et al., 2019), the density model p θ ( o ) is re-initialized each episode and trained using maximum likelihood estimation (MLE) to ﬁt the observations of the agent within a single episode, which are stored in a buffer β .

Original: The density model is either represented using a Gaussian distribution as in (Berseth et al., 2019), or using independent categorical distributions.
Revised: The density model is either represented using a Gaussian distribution as in (Berseth et al., 2019), or using independent categorical distributions.

Original: We have found it helpful to only compute the surprise reward using observations from the second half of the Control policy’s turn; this gives the agent greater ability to take actions that may lead to initial surprise, but reduce entropy over the long term.
Revised: We have found it helpful to only compute the surprise reward using observations from the second half of the Control policy’s turn; this gives the agent greater ability to take actions that may lead to initial surprise, but reduce entropy over the long term.

Original: We also experiment with when to reset the buffer β ; we ﬁnd that resetting the buffer after each round (after the Explore policy and Control policy each take one turn) can sometimes improve performance.
Revised: We also experiment with when to reset the buffer β ; we ﬁnd that resetting the buffer after each round (after the Explore policy and Control policy each take one turn) can sometimes improve performance.

Original: Finally, we allow the
Revised: Finally, we allow the

Original: Explore and Control policys to act for a different number of timesteps, tuning the emphasis on exploration or control depending on the environment.
Revised: Explore and Control policys to act for a different number of timesteps, tuning the emphasis on exploration or control depending on the environment.

Original: 5. Adversarial Surprise maximizes state coverage in Block MDPs
Revised: 5. Adversarial Surprise maximizes state coverage in Block MDPs

Original: In this section, we show that our method covers the state space under some restrictions on the structure of the
Revised: In this section, we show that our method covers the state space under some restrictions on the structure of the

Original: POMDP, and the density of states with low observation entropy.
Revised: POMDP, and the density of states with low observation entropy.

Original: Full proofs are in the Appendix.
Revised: Full proofs are in the Appendix.

Original: We deﬁne the marginal observation distribution as: d π ( o ) =
Revised: We deﬁne the marginal observation distribution as: d π ( o ) =

Original: (1 − γ ) (cid:80) ∞
Revised: (1 − γ ) (cid:80) ∞

Original: t =0 γ t p ( o t = o ) .
Revised: t =0 γ t p ( o t = o ) .

Original: We are interested in maximizing the state marginal entropy d π ( s ) by using a density model of the observation, p θ ( o ) .
Revised: We are interested in maximizing the state marginal entropy d π ( s ) by using a density model of the observation, p θ ( o ) .

Original: To that end, we prove the following lemm (see Appendix):
Revised: To that end, we prove the following lemma in the Appendix:

Original: Lemma 1.
Revised: Lemma 1.

Original: The cumulative surprise measured by the observation density model p θ ( o ) forms an upper bound of the observation marginal entropy H ( d π ( o )) , which becomes tight when the observation density model ﬁts the observation marginal d [Equation]
Revised: The cumulative surprise measured by the observation density model p θ ( o ) forms an upper bound of the observation marginal entropy H ( d π ( o )) , which becomes tight when the observation density model ﬁts the observation marginal d π ( o ) : − E π (cid:80) ∞ t =0 log p θ ( o t ) ≥ H ( d π ( o ))

Original: We start with an assumption on the structure of the POMDP:
Revised: We start with an assumption on the structure of the POMDP:

Original: Assumption 1 (Block MDP) .
Revised: Assumption 1 (Block MDP (BMDP) (Misra et al., 2020)) .

Original: We suppose that every two different states have disjoint emission supports: for any s, s [Equation] ∈ S , s (cid:54) =
Revised: We suppose that every two different states have disjoint emission supports: for any s, s (cid:48) ∈ S , s (cid:54) =

Original: s (cid:48)
Revised: s (cid:48)

Original: ⇒ supp ( p ( O | s ))
Revised: ⇒ supp ( p ( O | s ))

Original: ∩ supp ( p ( O | s ))
Revised: ∩ supp ( p ( O | s ))

Original: = ∅
Revised: = ∅

Original: Under this assumption, we show a useful relation between observation marginal entropy and state marginal entropy:
Revised: Under this assumption, we show a useful relation between observation marginal entropy and state marginal entropy:

Original: Lemma 2.
Revised: Lemma 2.

Original: In a block MDP (BMDP) (Misra et al., 2020), we can decompose the observation marginal entropy:H ( d [Equation] =
Revised: In a BMDP, we can decompose the observation marginal entropy:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: See  Appendix for the proof.
Revised: See the Appendix for the proof.

Original: Equation 5 shows that maximizing entropy in the marginal observation distribution d π ( o ) amounts to maximizing two terms: the emission entropy H ( p ( O | S )) , and the state marginal entropy H ( d π ( s )) .
Revised: Equation 6 shows that maximizing entropy in the marginal observation distribution d π ( o ) amounts to maximizing two terms: the emission entropy H ( p ( O | S )) , and the state marginal entropy H ( d π ( s )) .

Original: Suppose that we have a small number of latent states with rich observations, i.e. where the entropy of the emission distribution is very high: ∃ s : H ( p ( O | S = s ))
Revised: Suppose that we have a small number of latent states with rich observations, i.e. where the entropy of the emission distribution is very high: ∃ s : H ( p ( O | S = s ))

Original: (cid:29) log |S| .
Revised: (cid:29) log |S| .

Original: We can think of these states as “ noisy TVs ”.
Revised: We can think of these states as “ noisy TVs ”.

Original: In this case, if we are trying to maximize the marginal observation entropy, we have:
Revised: In this case, if we are trying to maximize the marginal observation entropy, we have:

Original: [Equation]
Revised: [Equation]

Original: The RHS is maximized by taking:
Revised: The RHS is maximized by taking:

Original: [Equation]
Revised: [Equation]

Original: This shows that conventional methods that focus on maximizing entropy over observations can trivially maximize d π ( o ) by remaining in  state s .
Revised: This shows that conventional methods that focus on maximizing entropy over observations can trivially maximize d π ( o ) by remaining in a state with rich observation.

Original: We will show that Adversarial Surprise is not subject to this problem.
Revised: We will show that Adversarial Surprise is not subject to this problem.

Original: To this end, we deﬁne a (semiquasi)metric on the latent state space:
Revised: To this end, we deﬁne a (semiquasi)metric on the latent state space:

Original: ˜ d ( s, s (cid:48) ) =
Revised: ˜ d ( s, s (cid:48) ) =

Original: min { k : ∃ π, P πk ( s (cid:48) | s )
Revised: min { k : ∃ π, P πk ( s (cid:48) | s )

Original: = 1 } , where by convention P π 0 ( s | s )
Revised: = 1 } , where by convention P π 0 ( s | s )

Original: = 1 for all s .
Revised: = 1 for all s .

Original: In other words, ˜ d ( s, s (cid:48) )
Revised: In other words, ˜ d ( s, s (cid:48) )

Original: =
Revised: =

Original: k if there is a policy that reaches s (cid:48) from s in k steps with probability 1.
Revised: k if there is a policy that reaches s (cid:48) from s in k steps with probability 1.

Original: We symmetrize this metric by deﬁning the following semimetric: d ( s, s (cid:48) )
Revised: We symmetrize this metric by deﬁning the following semimetric: d [Equation]

Original: =
Revised: s

Original: max { ˜ d ( s, s (cid:48)

Original: [Equation]
Revised: [Equation]

Original: , ˜ d ( s (cid:48) , s ) }
Revised: s [Equation] [Equation]

Original: We now give a formal deﬁnition of dark rooms.
Revised: We now give a formal deﬁnition of dark rooms.

Original: We say that a state s is a dark room if it has minimal emission entropy:
Revised: We say that a state s is a dark room if it has minimal emission entropy:

Original: [Equation]
Revised: [Equation]

Original:  O | S = s )) .
Revised: S H ( p ( O | S = s )) .

Original: We will use the following assumption about the density of dark rooms in the latent state space:
Revised: We will use the following assumption about the density of dark rooms in the latent state space:

Original: Assumption 2.
Revised: Assumption 2.

Original: We  suppose that for every state s , there is a dark room such that d ( s, s (cid:48) ) ≤ T2 .
Revised: We make three assumptions concerning the density of dark rooms: [(a)]We suppose that for every state s , there is a dark room such that d ( s, s (cid:48) ) ≤ T .

Original: That is, the set of dark rooms is a T2 -cover of the state space with respect to d .
Revised: That is, the set of dark rooms is a T -cover of the state space with respect to d .

Revised: We suppose that for every state s , there is a dark room such that P πT ( s (cid:48) | s ) = 1 , that is a dark room can be reached in exactly T steps.

Revised: We suppose that for any state s and any dark room s (cid:48) , if ˜ d ( s, s (cid:48) ) ≤ T , then d ( s, s (cid:48) ) ≤ T , that is if we can reach a dark room from a state s in less than T steps, then we can also reach s from this dark room is less than T steps.

Original: We can now state our main result:
Revised: We can now state our main result:

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumptions 1 and 2 the Markov chain induced by the following AS game:
Revised: Under Assumptions 1 and 2 the Markov chain induced by the following AS game:

Original: [Equation]
Revised: [Equation]

Original: T2 -covers the  state space, i.e., for all states s , there is a state s (cid:48)such that d π ( s (cid:48) ) >
Revised: T -covers the latent state space, i.e., for all states s , there is a state s (cid:48)

Original:  and d ( s, s (cid:48) ) ≤ T2 , where d π is the  marginal  induced by the game between the Explore ( π E ) and Control ( π C ) agents.
Revised: such that d π ( s (cid:48) ) > 0 and d ( s, s (cid:48) ) ≤ T , where d π is the state marginal distribution induced by the game between the Explore ( π E ) and Control ( π C ) policies.

Original: Assumption 2 guarantees that for any states that the Explore policy reaches, the Control policy can ﬁnd a low-emissionentropy state within its turn, such that H ( p ( O | s )) is minimized.
Revised: Assumption 2 guarantees that for any states that the Explore policy reaches, the Control policy can ﬁnd a low-emissionentropy state within its turn, such that H ( p ( O | s )) is minimized.

Original: Thus, from the perspective of the Explore policy, the ﬁrst term in its objective in Eq.
Revised: Thus, from the perspective of the Explore policy, the ﬁrst term in its objective in Eq.

Original: 5 is minimized, and it must focus on maximizing the second term, H ( d π ( s )) .
Revised: 6 is minimized, and it must focus on maximizing the second term, H ( d π ( s )) .

Original: In order to maximize entropy over the state marginal distribution d π ( s ) , the Explore policy must fully explore the state space.
Revised: In order to maximize entropy over the state marginal distribution d π ( s ) , the Explore policy must fully explore the state space.

Original: 6. Experimental results
Revised: 6. Experimental results

Original: In this section we present experimental results designed to answer the following three questions:Q1.
Revised: In this section we present experimental results designed to answer the following four questions:

Original: Exploration and state coverage: how well does AS explore the underlying state space in a stochastic, partially-observed world, as compared to alternative methods?
Revised: Exploration and state coverage: how well does AS explore the underlying state space in a stochastic, partially-observed world, as compared to alternative methods?

Original: Given our theoretical results in Section 5, we hypothesize that methods based on novelty-seeking will become distracted by noisy elements, while AS will fully explore the environment.
Revised: Given our theoretical results in Section 5, we hypothesize that methods based on novelty-seeking will become distracted by noisy elements, while AS will fully explore the environment.

Original: We will use the number of rooms visited in a navigation task as a measurement of state coverage.
Revised: We will use the number of rooms visited in a navigation task as a measurement of state coverage.

Original: Q2. Control: will AS learn to take actions to control its environment, and recover from surprising situations?
Revised: Control: will AS learn to take actions to control its environment, and recover from surprising situations?

Original: We measure control as the number of actions taken that cause changes to elements in the environment, such as ﬂipping a switch to stop ﬂashing lights.
Revised: We measure control as the number of actions taken that cause changes to elements in the environment, such as ﬂipping a switch to stop ﬂashing lights.

Original: Q3.
Revised: Emergence of complexity: is AS able to produce an arms race between the Control and Explore policies that leads to the agent’s acquisition of increasingly complex observable behaviors? If this is the case, we expect to observe the alternation of relatively long learning phases, where the two policies are competing without visible change in the agent’s behavior, and relatively short phase transition that separate two clearly distinguishable behavior.

Original: No-reward learning: will AS enable the agent to learn meaningful behaviors in the absence of any external reward?
Revised: No-reward learning: will AS enable the agent to learn meaningful behaviors in the absence of any external reward?

Original: To assess this, we train the IM methods using only intrinsic reward, then assess the amount of task reward they obtain in the standard Atari benchmark (Bellemare et al., 2013).
Revised: To assess this, we train the IM methods using only intrinsic reward, then assess the amount of task reward they obtain in the standard Atari benchmark (Bellemare et al., 2013).

Original: While there is no reason to expect AS to always correlate with the objectives in arbitrary MDPs, we expect that the twin goals of maximizing coverage while achieving high control should correlate well with objectives in many reasonable MDPs, particularly video games of the sort present in Atari.
Revised: While there is no reason to expect AS to always correlate with the objectives in arbitrary MDPs, we expect that the twin goals of maximizing coverage while achieving high control should correlate well with objectives in many reasonable MDPs, particularly video games of the sort present in Atari.

Original: Many of these games have a notion of progress, which roughly corresponds to coverage, but at the same time have many dangerous states that could result in ‘death’, which leads to an unexpected jump back to the starting state.
Revised: Many of these games have a notion of progress, which roughly corresponds to coverage, but at the same time have many dangerous states that could result in ‘death’, which leads to an unexpected jump back to the starting state.

Original: Therefore, we hypothesize that AS should, without even being aware of the task reward, perform well in these environments.
Revised: Therefore, we hypothesize that AS should, without even being aware of the task reward, perform well in these environments.

Original: Comparing to prior methods in these domains is interesting, because prior work has variously argued that both novelty-seeking exploration methods (Burda et al., 2018) and surprise-minimization methods (Berseth et al., 2019) should be expected to achieve high scores in these domains.
Revised: Comparing to prior methods in these domains is interesting, because prior work has variously argued that both novelty-seeking exploration methods (Burda et al., 2018) and surprise-minimization methods (Berseth et al., 2019) should be expected to achieve high scores in these domains.

Original: Note that when we assess all three metrics we look at the performance of the agent as a whole, which is jointly controlled by both the Explore and Control policy.
Revised: Note that when we assess all three metrics we look at the performance of the agent as a whole, which is jointly controlled by both the Explore and Control policy.

Revised: We include videos of the AS agent learning to play games with no reward, as well as performing navigation tasks, at https://sites.google.com/view/ adversarial-surprise/home .

Original: Baselines: We compare AS to three competitive unsupervised RL baselines: Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2017) (a state-of-the-art multi-agent curriculum method), Random Network Distillation (RND) (Burda et al., 2018)
Revised: Baselines: We compare AS to three competitive unsupervised RL baselines: Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2017) (a state-of-the-art multi-agent curriculum method), Random Network Distillation (RND) (Burda et al., 2018)

Original: (a state-of-the-art exploration method), and SMiRL (Berseth et al., 2019)
Revised: (a state-of-the-art exploration method), and SMiRL (Berseth et al., 2019)

Original: (a recently proposed method based on surprise minimization).
Revised: (a recently proposed method based on surprise minimization).

Original: These three methods are all related to AS: like RND, the Explore policy aims to maximize surprise, though not instantaneously but rather over the Control policy’s episode; like SMiRL, the Control policy aims to minimize surprise, but starting from the initial states that the Explore policy puts it in, and in a partially observed setting; like asymmetric self-play (ASP), AS consists of a two-player game, though the AS two-player game is symmetric and zero-sum, and based on observational entropy rather than goal-reaching.
Revised: These three methods are all related to AS: like RND, the Explore policy aims to maximize surprise, though not instantaneously but rather over the Control policy’s episode; like SMiRL, the Control policy aims to minimize surprise, but starting from the initial states that the Explore policy puts it in, and in a partially observed setting; like asymmetric self-play (ASP), AS con- sists of a two-player game, though the AS two-player game is symmetric and zero-sum, and based on observational entropy rather than goal-reaching.

Original: All methods usethe same underlying PPO implementation, with hyperparameters given in the Appendix.
Revised: All methods use PPO as RL optimization algorithm, with hyperparameters given in the Appendix.

Original: Environments: To evaluate Q1 and Q2 , we need partiallyobserved environments that present an exploration challenge, and which include stochastic phenomenaSince standard benchmarks do not consistently exhibit these properties, we constructed a custom family of procedurally generated navigation tasks based on Minigrid (Chevalier-Boisvert et al., 2018).
Revised: Environments: To evaluate Q1 and Q2 , we need partiallyobserved environments that present an exploration challenge, and which include stochastic phenomena. Since standard benchmarks do not consistently exhibit these properties, we constructed a custom family of procedurally generated navigation tasks based on Minigrid (Chevalier-Boisvert et al., 2018).

Original: These environments contain rooms that are either empty (dark), or contain stochastic elements such as ﬂashing lights that randomly change color.
Revised: These environments contain rooms that are either empty (dark), or contain stochastic elements such as ﬂashing lights that randomly change color.

Original: They also contain elements such as doors that can be opened, and switches that, when ﬂipped, stop the stochastic elements from changing.
Revised: They also contain elements such as doors that can be opened, and switches that, when ﬂipped, stop the stochastic elements from changing.

Original: An example is shown in Figure 1.
Revised: An example is shown in Figure 1.

Original: As in MiniGrid, the environments are partially observed; the agent only sees a 5x5 window of the true state.
Revised: As in MiniGrid, the environments are partially observed; the agent only sees a 5x5 window of the true state.

Revised: To evaluate Q3 , we choose a relatively simpler version of our custom Minigrid environment that includes only two rooms to clearly distinguish the gain of complexity due to AS from the complexity of the environment.

Original: While these environments allow us to carefully study the effects of partial observability and stochasticity, we would also like to compare to prior work on a standardized benchmark.
Revised: While these environments allow us to carefully study the effects of partial observability and stochasticity, we would also like to compare to prior work on a standardized benchmark.

Original: For this purpose,  we use the Atari Arcade Learning Environment (ALE)
Revised: For this purpose, to evaluate Q4 we use the Atari Arcade Learning Environment (ALE)

Original: (Bellemare et al., 2013) (see Figure 4), which was used by both SMiRL (Berseth et al., 2019) and RND (Burda et al., 2018) to establish their effectiveness.
Revised: (Bellemare et al., 2013) (see Figure 5), which was used by both SMiRL (Berseth et al., 2019) and RND (Burda et al., 2018) to establish their effectiveness, as well as the ViZDoom environment ( ? ).

Original: Due to limited computational resources, we do not conduct experiments in all possible
Revised: Due to limited computational resources, we do not conduct experiments in all possible

Original: Atari games (which is consistent with prior work (Berseth et al., 2019; Burda et al., 2018)), but we show results for each of the games that we test, both in Section 6 and  the Appendix.
Revised: Atari games (which is consistent with prior work (Berseth et al., 2019; Burda et al., 2018)), but we show results for each of the games that we test, both in Section 6.4 and in the Appendix.

Original: environments containing both stochastic elements and dark rooms both (a) cumulatively over training, and (b an episode

Original: (a) shows how well each method works as an exploration bonus to encourage collecting experience from all states, and part (b) provides a measure of how well the asymptotic policy learned by each method continues to explore the state space at convergence.

Original: AS outperforms surprise-minimizing

Original: SMiRL, which minimizes observation surprise by turning to face the wall, or remains in dark rooms.

Original: AS also explores better than the state-of-the-art exploration methods RND and ASP, which both become distracted by noisy elements.

Original: Q1. Exploration and state coverage:
Revised: 6.1. Exploration and state coverage:

Original: Figure 2 shows the results of trainingeach of the agents in the procedurallygenerated navigation environments in terms of the number of rooms the agents learn to visit (our measure of state coverage).
Revised: Figure 2 shows the results of training the AS agent in the procedurally-generated navigation environments in terms of the number of rooms the agent learns to visit (our measure of state coverage).

Original: We measure the number of rooms cumulatively, over the course of training (Figure 2), to assess whether each method will lead the agent to collect experience from all possible states.
Revised: We measure the number of rooms cumulatively, over the course of training, to assess whether each method will lead the agent to collect experience from all possible states.

Original: This measure is relevant to whether the technique can be used as an effective exploration bonus to aid learning a downstream task.
Revised: This measure is relevant to whether the technique can be used as an effective exploration bonus to aid learning a downstream task.

Original: We also measure the number of rooms explored within each episode (Figure 2).
Revised: We also measure the number of rooms explored within each episode.

Original: This allows us to assess whether the asymptotic policy learned by the algorithms continues to explore once it has converged.
Revised: This allows us to assess whether the asymptotic policy learned by the algorithms continues to explore once it has converged.

Original: As predicted by our theoretical analysis, we see that AS learns to more fully explore the environments, visiting signiﬁcantly more rooms over a lifetime(Figure 2) and per episode(Figure 2) than competing methods.
Revised: As predicted by our theoretical analysis, we see that AS learns to more fully explore the environments, visiting signiﬁcantly more rooms over a lifetime and per episode than competing methods.

Original: It learns more quickly and explores more thoroughly than RND, which becomes distracted by the inherently random elements in the environment which lead to high prediction error.
Revised: It learns more quickly and explores more thoroughly than RND, which becomes distracted by the inherently random elements in the environment which lead to high prediction error.

Original: The stochastic elements also hinder learning for ASP, since Alice can easily produce random goals that are difﬁcult for Bob to replicate.
Revised: The stochastic elements also hinder learning for ASP, since Alice can easily produce random goals that are difﬁcult for Bob to replicate.

Original: Finally, we see that SMiRL, which is designed for fully observed environments, does not explore effectively because it suffers from the dark room problem – it prefers to stay within the empty rooms, and not venture into rooms with high-entropy, stochastic elements.number
Revised: Finally, we see that SMiRL, which is designed for fully observed environments, does not explore effectively because it suffers from the dark room problem – it prefers to stay within the empty rooms, and not venture into rooms with high-entropy, stochastic elements.

Original: AS buffer more frequently enables it to exceed even SMiRL in taking actions to control the environment

Original: Q2. Control:
Revised: 6.2. Control:

Original: To measure whether agents learn to control their environment, we investigate how many times agents press switches in the navigation environment that stop the stochastic elements in the environment from changing color.
Revised: To measure whether the agent learn to control their environment, we investigate how many times the agent press switches in the navigation environment that stop the stochastic elements in the environment from changing color.

Original: The results are shown in Figure 3.
Revised: The results are shown in Figure 3.

Original: Since RND has no incentive to learn to control the environment, it never learns to press the switch.
Revised: Since RND has no incentive to learn to control the environment, it never learns to press the switch.

Original: A similar result is observed for ASP, since reducing the entropy would make it easier for the Bob agent to replicate the Alice agent’s ﬁnal state.
Revised: A similar result is observed for ASP, since reducing the entropy would make it easier for the Bob agent to replicate the Alice agent’s ﬁnal state.

Original: Thus, ASP will not always lead agents to learn all possible behaviors relevant to controlling the environment.
Revised: Thus, ASP will not always lead the agent to learn all possible behaviors relevant to controlling the environment.

Original: Both SMiRL
Revised: Both SMiRL

Original: and
Revised: and

Original: AS learn to take actions to reduce entropy.
Revised: AS learn to take actions to reduce entropy.

Original: However, when we train AS by resetting the buffer β used to ﬁt the density model p θ ( o ) after each round (that is, after both the Explore and Control policy have taken one turn), rather than after each episode, we see that AS increases the number of actions it takes to reduce entropy even over SMiRL.
Revised: However, when we train AS by resetting the buffer β used to ﬁt the density model p θ ( o ) after each round (that is, after both the Explore and Control policy have taken one turn), rather than after each episode, we see that AS increases the number of actions it takes to reduce entropy even over SMiRL.

Original: This is likely because resetting the buffer removes any incentive to return to states that the agent has previously seen within its lifetime, and instead gives a stronger incentive to reduce entropy immediately. 
Revised: This is likely because resetting the buffer removes any incentive to return to states that the agent has previously seen within its lifetime, and instead gives a stronger incentive to reduce entropy immediately. 6.3. Emergence of Complexity:

Original: how much of game reward the obtains
Revised: To show that Adversarial Surprise leads to emergence of complexity by phases , we plot the temporal acquisition of two behaviors in order of complexity in the MiniGrid environment. The results are shown in Figure 4. This environment includes a dark room and a noisy room separated by a door. The position of the door changes at each episode.

Revised: Initially, the agent is inside the noisy room and the door is open.

Revised: One episode consists of 96 steps: the Control policy takes control of the agent during 32 steps, then the Explore policy takes control of the agent during 32 steps, ﬁnally the

Revised: Control policy takes control of the agent during 32 steps.

Original: 95% Conﬁdence Interval (CI of three seeds.
Revised: The ﬁrst acquired behavior by the Control policy is identifying where the door is and going to the dark room during the ﬁrst round. It is a short-term suprise minimizing behavior and an agent trained with a SMIRL objective can converge to it. However, the Explore policy learns to go back to the noisy room and to reach the farthest point from the door such that the Control policy does not have the time to reach a state of minimum entropy before the surprise of the agent is computed in the reward. This in turn incentivizes the acquisition of a more complex behavior by the

Original: Since the games reward behavior such as staying alive andlearning to shoot enemies, obtaining higher reward indicates the agent has learned meaningful behaviors. AS outperforms both RND and SMiRL, showing that AS provides a general way to learn useful behaviors across multiple environments, in the absence of external reward.
Revised: Control policy: it learns to go in the dark room and to lock the agent inside by closing the door during the ﬁrst round, making it harder for the Explore policy to learn to reach a state that will surprise the agent during the Control policy’s second round. This behavior reminiscent of Dr. Jekyll and Mr. Hyde highlights the potential of Adversarial Surprise to learn long-term suprise-minimizing behaviors.

Original: Q3.
Revised: No-

Original: No-reward learning: Figure 4 shows how well each of the intrinsic motivation (IM) methods (SMiRL, RND, and AS) can be used to learn interesting behaviors in the absence of external reward.
Revised: Figures 5 and 6 show how well each method can be used to learn interesting behaviors in the absence of external reward.

Original: To assess whether they can learn useful skills, we measure the game reward obtained in several Atari environments.
Revised: To assess whether they can learn useful skills, we measure the game reward obtained in several Atari environments and the VizDoom Take Cover environment.

Original: Because the games reward complex behaviors like shooting  enemies, a high game reward indicates the agent has learned interesting skills, purely from optimizing the instrinsic objective.
Revised: Because the games reward complex behaviors like shooting or avoiding enemies, a high game reward indicates the agent has learned interesting skills, purely from optimizing the instrinsic objective.

Original: Across all three environments, AS performs better than both RND and SMiRL.
Revised: Across the environments, AS performs better than RND, SMiRL, and ASP.

Original: The performance of RND  decreases over time due the bonus from the prediction error shrinking as more states become familiar.
Revised: While RND is effective in some environments, its performance often decreases over time due the bonus from the prediction error shrinking as more states become familiar.

Revised: Further, maximizing novelty in environments like Freeway, Space Invaders, and Doom can lead to the agent dying, corresponding to low reward.

Original: SMiRL performs poorly in these environments,  because it avoids entropy by hiding from enemies.
Revised: SMiRL performs well in Freeway, where minimizing entropy corresponds closely to staying alive and not being hit by cars. However in the other environments, SMiRL performs poorly, because it avoids entropy by hiding from enemies (it prefers to stay in dark rooms when they are available).

Original:  In contrast, AS  obtains higher returns, indicating that optimizing for both exploration and control provides a  useful inductive bias for learning interesting behaviors in the absence of external reward.We include videos of the agents learning to play Atari games with no reward, as well as performing the navigation tasks, at https://sites.google.com/view/ adversarial-surprise/home .
Revised: ASP also performs poorly because it is possible for Alice to quickly reach states which Bob cannot easily replicate, preventing the algorithm from learning meaningful behaviors. In contrast, AS consistently obtains high returns across all environments, indicating that optimizing for both exploration and control provides a broadly useful inductive bias for learning interesting behaviors in the absence of external reward.

Original: 7. Discussion
Revised: 7. Discussion

Original: We proposed Adversarial Surprise as a general approach for unsupervised reinforcement learning.
Revised: We proposed Adversarial Surprise as a general approach for unsupervised reinforcement learning.

Original: Adversarial Surprise corresponds to a two-player adversarial game, in which two policies compete over the amount of surprise, or observation entropy, that an agent experiences.
Revised: Adversarial Surprise corresponds to a two-player adversarial game, in which two policies compete over the amount of surprise, or observation entropy, that an agent experiences.

Original: Reminiscent of Dr.
Revised: Reminiscent of Dr.

Original: Jekyll and Mr. Hyde, the Explore policy acts to expose the Control policy to highly entropic states from which it must recover by learning to manipulate the environment.
Revised: Jekyll and Mr. Hyde, the Explore policy acts to expose the Control policy to highly entropic states from which it must recover by learning to manipulate the environment.

Original: AS provides a novel intrinsic motivation approach that leads to more effective learning in stochastic, partially observed environments.
Revised: We show that AS produce increasingly complex control and exploration strategies and has the potential to be applied to the exploration of stochastic, partially observed environments.

Original: In such environments, prior methods either become distracted by noisy elements, or suffer from the “dark room” problem, in which observation entropy is minimized by simply hiding in a low-entropy part of the state space.
Revised: In such environments, prior methods can become distracted by noisy elements, or suffer from the “dark room” problem, in which observation entropy is minimized by simply hiding in a low-entropy part of the state space.

Original: We show both theoretically and empirically that AS is robust against these issues, and learns to explore the environment more thoroughly, and control it more effectively, than state-of-the-art prior works like RND, ASP, and 
Revised: We show both theoretically and empirically that AS is robust against these issues, and learns to explore the environment more thoroughly, and control it more effectively, than state-of-the-art prior works like RND, ASP, and SMiRL. Future work: Our evaluation of AS focuses on coverage and unsupervised exploration, where we demonstrate that

Original: SMiRL.
Revised: AS improves over pure novelty-seeking and pure surprise minimization methods when the environment exhibits both unpredictable and stochastic components and partial observability. However, the potential value of unsupervised reinforcement learning methods extends more broadly: such methods could be used to acquire skills for downstream task learning, controlling an environment to reach states from which more behaviors could be performed successfully, and other applications. Future work could study how AS and its extensions could enable these applications, for example by collecting data for downstream reward-guided learning. Further, we see a potentially exciting method which combines AS with hierarchical RL, by training a meta-policy to select when to invoke the Explore and Control sub-policies.

Revised: In this way, the meta-policy could explicitly decide when to explore and when to exploit.