Original: C ONTINGENCY -A WARE E XPLORATION IN R EINFORCEMENT L EARNING
Revised: C ONTINGENCY -A WARE E XPLORATION IN R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning.
Revised: This paper investigates whether learning contingency-awareness and controllable aspects of an environment can lead to better exploration in reinforcement learning.

Original: To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE).
Revised: To investigate this question, we consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE).

Original: In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games.
Revised: In this study, we develop an attentive dynamics model (ADM) that discovers controllable elements of the observations, which are often associated with the location of the character in Atari games.

Original: The
Revised: The

Original: ADM is trained in a self-supervised fashion to predict the actions taken by the agent.
Revised: ADM is trained in a self-supervised fashion to predict the actions taken by the agent.

Original: The learned contingency information is used as a part of the state representation for exploration purposes.
Revised: The learned contingency information is used as a part of the state representation for exploration purposes.

Original: We demonstrate that combining A2C with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards.
Revised: We demonstrate that combining actor-critic algorithm with count-based exploration using our representation achieves impressive results on a set of notoriously challenging Atari games due to sparse rewards.

Original: For example, we report a state-of-the-art score of >6600 points on M ONTEZUMA ’ S R EVENGE without using expert demonstrations, explicit high-level information ( e g ., RAM states), or supervised data.
Revised: For example, we report a state-of-the-art score of >11,000 points on M ONTEZUMA ’ S R EVENGE without using expert demonstrations, explicit high-level information ( e g ., RAM states), or supervised data.

Original: Our experiments conﬁrm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations
Revised: Our experiments conﬁrm that contingency-awareness is indeed an extremely powerful concept for tackling exploration problems in reinforcement learning and opens up interesting research questions for further investigations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The success of reinforcement learning (RL) algorithms in complex environments hinges on the way they balance exploration and exploitation .
Revised: The success of reinforcement learning (RL) algorithms in complex environments hinges on the way they balance exploration and exploitation .

Original: There has been a surge of recent interest in developing effective exploration strategies for problems with high-dimensional state spaces and sparse rewards (Schmidhuber, 1991; Oudeyer & Kaplan, 2009; Houthooft et al., 2016;
Revised: There has been a surge of recent interest in developing effective exploration strategies for problems with high-dimensional state spaces and sparse rewards (Schmidhuber, 1991; Oudeyer & Kaplan, 2009; Houthooft et al., 2016;

Original: Bellemare et al., 2016;
Revised: Bellemare et al., 2016;

Original: Osband et al., 2016; Pathak et al., 2017; Plappert et al., 2018; Zheng et al., 2018).
Revised: Osband et al., 2016; Pathak et al., 2017; Plappert et al., 2018; Zheng et al., 2018).

Original: Deep neural nets have seen great success as expressive function approximators within RL and as powerful representation learning methods for many domains.
Revised: Deep neural nets have seen great success as expressive function approximators within RL and as powerful representation learning methods for many domains.

Original: In addition, there have been recent studies on using neural network representations for exploration (Tang et al., 2017; Martin et al., 2017; Pathak et al., 2017).
Revised: In addition, there have been recent studies on using neural network representations for exploration (Tang et al., 2017; Martin et al., 2017; Pathak et al., 2017).

Original: For example, count-based exploration with neural density estimation (Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017) presents one of the state-of-the-art techniques on the most challenging
Revised: For example, count-based exploration with neural density estimation (Bellemare et al., 2016; Tang et al., 2017; Ostrovski et al., 2017) presents one of the state-of-the-art techniques on the most challenging

Original: Atari games with sparse rewards.
Revised: Atari games with sparse rewards.

Original: Despite the success of recent exploration methods, it is still an open question on how to construct an optimal representation for exploration.
Revised: Despite the success of recent exploration methods, it is still an open question on how to construct an optimal representation for exploration.

Original: For example, the concept of visual similarity is used for learning density models as a basis for calculating pseudo-counts (Bellemare et al., 2016; Ostrovski et al., 2017).
Revised: For example, the concept of visual similarity is used for learning density models as a basis for calculating pseudo-counts (Bellemare et al., 2016; Ostrovski et al., 2017).

Original: However, as Tang et al.
Revised: However, as Tang et al.

Original: (2017) noted, the ideal way of representing states should be based on what is relevant to solving the MDP, rather than only relying on visual similarity.
Revised: (2017) noted, the ideal way of representing states should be based on what is relevant to solving the MDP, rather than only relying on visual similarity.

Original: In addition, there remains another question on whether the representations used for recent exploration work are easily interpretable.
Revised: In addition, there remains another question on whether the representations used for recent exploration work are easily interpretable.

Original: To address these questions, we investigate whether we can learn a complementary and more intuitive and interpretable high-level abstraction that can be very effective in exploration, by using the ideas of contingency awareness and controllable dynamics.
Revised: To address these questions, we investigate whether we can learn a complementary and more intuitive and interpretable high-level abstraction that can be very effective in exploration, by using the ideas of contingency awareness and controllable dynamics.

Original: The key idea that we focus on in this work is the notion of contingency awareness (Watson, 1966;
Revised: The key idea that we focus on in this work is the notion of contingency awareness (Watson, 1966;

Original: Bellemare et al., 2012) — the agent’s understanding of the environmental dynamics and recognizing that some aspects of the dynamics are under the agent’s control.
Revised: Bellemare et al., 2012) — the agent’s understanding of the environmental dynamics and recognizing that some aspects of the dynamics are under the agent’s control.

Original: Intuitively speaking, this can represent the segmentation mask of the agent operating in the 2D or 3D environments (yet one can think of more abstract and general state spaces).
Revised: Intuitively speaking, this can represent the segmentation mask of the agent operating in the 2D or 3D environments (yet one can think of more abstract and general state spaces).

Original: In this study, we investigate the concept of contingency awareness based on self-localization , i.e ., the awareness of where the agent is located in the abstract state space.
Revised: In this study, we investigate the concept of contingency awareness based on self-localization , i.e ., the awareness of where the agent is located in the abstract state space.

Original: We are interested in discovering parts of the world that are directly dependent on the agent’s immediate action, which often reveal the agent’s approximate location.
Revised: We are interested in discovering parts of the world that are directly dependent on the agent’s immediate action, which often reveal the agent’s approximate location.

Original: For further motivation on the problem, we note that contingency awareness is a very important concept in neuroscience and psychology.
Revised: For further motivation on the problem, we note that contingency awareness is a very important concept in neuroscience and psychology.

Original: In other words, being self-aware of one’s location is an important property within many observed intelligent organisms and systems.
Revised: In other words, being self-aware of one’s location is an important property within many observed intelligent organisms and systems.

Original: For example, recent breakthroughs in neuroscience, such as Nobel Prize winning work on the grid cells (Moser et al., 2015; Banino et al., 2018), show that organisms that perform very well in spatially-challenging tasks are self-aware of their location.
Revised: For example, recent breakthroughs in neuroscience, such as Nobel Prize winning work on the grid cells (Moser et al., 2015; Banino et al., 2018), show that organisms that perform very well in spatially-challenging tasks are self-aware of their location.

Original: This allows rats to navigate, remember paths to visited places and important sub-goals, and ﬁnd shortcuts.
Revised: This allows rats to navigate, remember paths to visited places and important sub-goals, and ﬁnd shortcuts.

Original: In addition, the notion of contingency awareness has been shown as an important factor in developmental psychology (Watson, 1966;
Revised: In addition, the notion of contingency awareness has been shown as an important factor in developmental psychology (Watson, 1966;

Original: Baeyens et al., 1990).
Revised: Baeyens et al., 1990).

Original: We can think of self-localization (and more broadly self-awareness) as a principled and fundamental direction towards intelligent agents.
Revised: We can think of self-localization (and more broadly self-awareness) as a principled and fundamental direction towards intelligent agents.

Original: Based on these discussions, we hypothesize that contingency awareness can be an extremely powerful mechanism for tackling exploration problems in reinforcement learning.
Revised: Based on these discussions, we hypothesize that contingency awareness can be a powerful mechanism for tackling exploration problems in reinforcement learning.

Original: We consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE).
Revised: We consider an instantiation of this hypothesis evaluated on the Arcade Learning Element (ALE).

Original: For example, in the context of 2D Atari games, contingency-awareness roughly corresponds to understanding the notion of controllable entities ( e.g ., the player’s avatar), which Bellemare et al.
Revised: For example, in the context of 2D Atari games, contingency-awareness roughly corresponds to understanding the notion of controllable entities ( e.g ., the player’s avatar), which Bellemare et al.

Original: (2012) refer to as contingent regions .
Revised: (2012) refer to as contingent regions .

Original: More concretely, as shown in Figure 1, in the game F REEWAY , only the chicken sprite is under the agent’s control and not the multiple moving cars; therefore the chicken’s location should be an informative element for exploration (Bellemare et al., 2012; Pathak et al., 2017).
Revised: More concretely, as shown in Figure 1, in the game F REEWAY , only the chicken sprite is under the agent’s control and not the multiple moving cars; therefore the chicken’s location should be an informative element for exploration (Bellemare et al., 2012; Pathak et al., 2017).

Original: In this study, we also investigate whether contingency awareness can be learned without any external annotations or supervision.
Revised: In this study, we also investigate whether contingency awareness can be learned without any external annotations or supervision.

Original: For this, we provide an instantiation of an algorithm for automatically learning such information and using it for improving exploration on a 2D ALE environment (Bellemare et al., 2013).
Revised: For this, we provide an instantiation of an algorithm for automatically learning such information and using it for improving exploration on a 2D ALE environment (Bellemare et al., 2013).

Original: Concretely, we employ an attentive dynamics model (ADM) to predict the agent’s action chosen between consecutive states.
Revised: Concretely, we employ an attentive dynamics model (ADM) to predict the agent’s action chosen between consecutive states.

Original: It allows us to approximate the agent’s position in 2D environments, but unlike other approaches such as (Bellemare et al., 2012), it does not require any additional supervision to do so.
Revised: It allows us to approximate the agent’s position in 2D environments, but unlike other approaches such as (Bellemare et al., 2012), it does not require any additional supervision to do so.

Original: The ADM learns in an online fashion with pure observations as the agent’s policy is updated and does not require hand-crafted features, an environment simulator, or supervision labels for training.
Revised: The ADM learns in an online fashion with pure observations as the agent’s policy is updated and does not require hand-crafted features, an environment simulator, or supervision labels for training.

Original: In experimental evaluation, our methods signiﬁcantly improve the performance of A2C on hardexploration Atari games in comparison with competitive methods such as density-based exploration (Bellemare et al., 2016; Ostrovski et al., 2017) and SimHash (Tang et al., 2017).
Revised: In experimental evaluation, our methods signiﬁcantly improve the performance of A2C on hardexploration Atari games in comparison with competitive methods such as density-based exploration (Bellemare et al., 2016; Ostrovski et al., 2017) and SimHash (Tang et al., 2017).

Original: We report very strong results on sparse reward games, including the state-of-the-art performance on the notoriously difﬁcult
Revised: We report very strong results on sparse reward games, including the state-of-the-art performance on the notoriously difﬁcult

Original: M ONTEZUMA ’ S R EVENGE , without using expert demonstrations or explicit high-level information.
Revised: M ONTEZUMA ’ S R EVENGE when combining our proposed exploration strategy with PPO (Schulman et al., 2017), without using expert demonstrations or explicit high-level information.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We demonstrate the importance of learning contingency awareness for efﬁcient exploration in challenging sparse-reward RL problems.
Revised: • We demonstrate the importance of learning contingency awareness for efﬁcient exploration in challenging sparse-reward RL problems.

Original: • We develop a novel instance of attentive dynamics model using contingency and controllable dynamics to provide robust localization abilities across the most challenging Atari environments.
Revised: • We develop a novel instance of attentive dynamics model using contingency and controllable dynamics to provide robust localization abilities across the most challenging Atari environments.

Original: • We achieve a strong performance on difﬁcult sparse-reward Atari games, including the state-ofthe-art score on notoriously challenging M ONTEZUMA ’ S R EVENGE .
Revised: • We achieve a strong performance on difﬁcult sparse-reward Atari games, including the state-ofthe-art score on notoriously challenging M ONTEZUMA ’ S R EVENGE .

Original: Overall, we believe that our experiments conﬁrm the hypothesis that contingency awareness is an extremely powerful concept for tackling exploration problems in reinforcement learning, which opens up interesting research questions for further investigations.
Revised: Overall, we believe that our experiments conﬁrm the hypothesis that contingency awareness is an extremely powerful concept for tackling exploration problems in reinforcement learning, which opens up interesting research questions for further investigations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Biological Motivation.
Revised: Biological Motivation.

Original: The discovery of grid cells (Moser et al., 2015) motivates working on agents that are self-aware of their location.
Revised: The discovery of grid cells (Moser et al., 2015) motivates working on agents that are self-aware of their location.

Original: Banino et al.
Revised: Banino et al.

Original: (2018) emphasize the importance of self-localization and train a neural network which learns a similar mechanism to grid cells when performing tasks related to spatial navigation.
Revised: (2018) emphasize the importance of self-localization and train a neural network which learns a similar mechanism to grid cells when performing tasks related to spatial navigation.

Original: The presence of grid cells was correlated with high performance.
Revised: The presence of grid cells was correlated with high performance.

Original: A set of potential approaches to self-localization ranges from ideas speciﬁc to a given environment, e.g .,
Revised: A set of potential approaches to self-localization ranges from ideas speciﬁc to a given environment, e.g .,

Original: SLAM (Durrant-Whyte & Bailey, 2006), to methods with potential generalizability.
Revised: SLAM (Durrant-Whyte & Bailey, 2006), to methods with potential generalizability.

Original: Although grid cells seem tailored to 2D or 3D problems that animals encounter in their life, it is speculated that their use can be extended to more abstract spaces. 
Revised: Although grid cells seem tailored to 2D or 3D problems that animals encounter in their life, it is speculated that their use can be extended to more abstract spaces. [Figure]

Original: Self-supervised Dynamics Model and Controllable Dynamics.
Revised: Self-supervised Dynamics Model and Controllable Dynamics.

Original: Several works have used forward and/or inverse dynamics models of the environment (Oh et al., 2015;
Revised: Several works have used forward and/or inverse dynamics models of the environment (Oh et al., 2015;

Original: Agrawal et al., 2016; Shelhamer et al., 2017).
Revised: Agrawal et al., 2016; Shelhamer et al., 2017).

Original: Pathak et al.
Revised: Pathak et al.

Original: (2017) employ a similar dynamics model to learn a feature representation[Figure] of states that captures controllable aspects of the environment.
Revised: (2017) employ a similar dynamics model to learn a feature representation of states that captures controllable aspects of the environment.

Original: This dense representation is used to design a curiosity-driven intrinsic reward.
Revised: This dense representation is used to design a curiosity-driven intrinsic reward.

Original: The idea of learning representation on the relevant aspects of the environment by learning auxiliary tasks is also explored in (Jaderberg et al., 2017; Bengio et al., 2017; Sawada, 2018).
Revised: The idea of learning representation on the relevant aspects of the environment by learning auxiliary tasks is also explored in (Jaderberg et al., 2017; Bengio et al., 2017; Sawada, 2018).

Original: Our presented approach is different as we focus on explicitly discovering controllable aspects using an attention mechanism, resulting in better interpretability.
Revised: Our presented approach is different as we focus on explicitly discovering controllable aspects using an attention mechanism, resulting in better interpretability.

Original: Exploration and Intrinsic Motivation.
Revised: Exploration and Intrinsic Motivation.

Original: The idea of providing an exploration bonus reward depending on the state-action visit-count was proposed by Strehl & Littman (2008) (MBIE-EB), originally under a tabular setting.
Revised: The idea of providing an exploration bonus reward depending on the state-action visit-count was proposed by Strehl & Littman (2008) (MBIE-EB), originally under a tabular setting.

Original: Later it has been combined with different techniques to deal with high-dimensional state spaces.
Revised: Later it has been combined with different techniques to deal with high-dimensional state spaces.

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2016) use a Context-Tree Switching (CTS) density model to derive a state pseudo-count , whereas Ostrovski et al.
Revised: (2016) use a Context-Tree Switching (CTS) density model to derive a state pseudo-count , whereas Ostrovski et al.

Original: (2017) use PixelCNN as a state density estimator.
Revised: (2017) use PixelCNN as a state density estimator.

Original: Martin et al.
Revised: Martin et al.

Original: (2017) also construct a visitation density model over a compressed feature space rather than the raw observation space.
Revised: (2017) also construct a visitation density model over a compressed feature space rather than the raw observation space.

Original: Alternatively, Tang et al.
Revised: Alternatively, Tang et al.

Original: (2017) proposed a localitysensitive hashing (LSH) method to cluster states and maintain a state-visitation counter based on a form of similarity between frames.
Revised: (2017) proposed a localitysensitive hashing (LSH) method to cluster states and maintain a state-visitation counter based on a form of similarity between frames.

Original: We train an agent with a similar count-based exploration bonus, but the way of maintaining state counter seems relatively simpler in that key feature information ( i.e ., controllable region) is explicitly extracted from the observation and directly used for counting states.
Revised: We train an agent with a similar count-based exploration bonus, but the way of maintaining state counter seems relatively simpler in that key feature information ( i.e ., controllable region) is explicitly extracted from the observation and directly used for counting states.

Original: Another popular family of exploration strategies in RL uses intrinsic motivation (Schmidhuber, 1991;
Revised: Another popular family of exploration strategies in RL uses intrinsic motivation (Schmidhuber, 1991;

Original: Singh et al., 2004; Oudeyer & Kaplan, 2009; Barto, 2013).
Revised: Singh et al., 2004; Oudeyer & Kaplan, 2009; Barto, 2013).

Original: These methods encourage the agent to look for something surprising in the environment which motivates its search for novel states, such as surprise (Achiam & Sastry, 2017) and curiosity (Pathak et al., 2017; Burda et al.,  2018), or via feature control (Jaderberg et al., 2017; Dilokthanakul et al., 2017) or diversity (Eysenbach et al., 2018).
Revised: These methods encourage the agent to look for something surprising in the environment which motivates its search for novel states, such as surprise (Achiam & Sastry, 2017), curiosity (Pathak et al., 2017; Burda et al., 2018a), and diversity (Eysenbach et al., 2018), or via feature control (Jaderberg et al., 2017; Dilokthanakul et al., 2017).

Original:  A PPROACH 3. D ISCOVERING C ONTINGENCY VIA A TTENTIVE D YNAMICS M ODEL
Revised:  A PPROACH 3. D ISCOVERING C ONTINGENCY VIA A TTENTIVE D YNAMICS M ODEL

Original: To discover the region of the observation  is controllable by the agent, we develop an instance of attentive dynamics model (ADM) based on inverse dynamics f inv .
Revised: To discover the region of the observation that is controllable by the agent, we develop an instance of attentive dynamics model (ADM) based on inverse dynamics f inv .

Original: The model takes two consecutive input frames (observations) s t − 1 , s t ∈ S as input and aims to predict the action ( a t  ∈ A ) taken by the agent to move from s t − 1 to s t :
Revised: The model takes two consecutive input frames (observations) s t − 1 , s t ∈ S as input and aims to predict the action ( a t − 1 ∈ A ) taken by the agent to transition from s t − 1 to s t :

Original: [Equation]
Revised: [Equation]

Original: Our key intuition is that the inverse dynamics model should attend to the most relevant part of the observation, which is controllable by the agent, to be able to classify the actions.
Revised: Our key intuition is that the inverse dynamics model should attend to the most relevant part of the observation, which is controllable by the agent, to be able to classify the actions.

Original: We determine whether each region in a H × W grid is controllable, or in other words, useful for predicting the agent’s action, by using a spatial attention mechanism (Bahdanau et al., 2015; Xu et al., 2015).
Revised: We determine whether each region in a H × W grid is controllable, or in other words, useful for predicting the agent’s action, by using a spatial attention mechanism (Bahdanau et al., 2015; Xu et al., 2015).

Original: An overview of the model is shown in Figure 1.
Revised: An overview of the model is shown in Figure 1.

Original: Model.
Revised: Model.

Original: To perform action classiﬁcation, we ﬁrst compute a convolutional feature map φ s t = φ ( s t ) ∈
Revised: To perform action classiﬁcation, we ﬁrst compute a convolutional feature map φ st = φ ( s t ) ∈

Original: R H × W × K based on the observation s t using a convolutional neural network φ .
Revised: R H × W × K based on the observation s t using a convolutional neural network φ .

Original: We estimate a set of logit (score) vectors, denoted e t ( i, j ) ∈ R |A| , for action classiﬁcation from each grid cell ( i, j ) of the convolutional feature map.
Revised: We estimate a set of logit (score) vectors, denoted e t ( i, j ) ∈ R |A| , for action classiﬁcation from each grid cell ( i, j ) of the convolutional feature map.

Original: The local convolution features and feature difference for consecutive frames are fed into a shared multi-layer perceptron (MLP) to derive the logits as:
Revised: The local convolution features and feature difference for consecutive frames are fed into a shared multi-layer perceptron (MLP) to derive the logits as:

Original: [Equation]
Revised: [Equation]

Original: j ); φ st ( i, j ) (cid:3)(cid:17) ∈ R |A| .

Original: We then compute an attention mask α t ∈ R
Revised: We then compute an attention mask α t ∈ R

Original: H
Revised: H

Original: × W corresponding to frame t , which indicates the controllable parts of the observation s t .
Revised: × W corresponding to frame t , which indicates the controllable parts of the observation s t .

Original: Such attention masks are computed via a separate MLP from the features of each region ( i, j ) , and then converted into a probability distribution using softmax or sparsemax operators (Martins & Astudillo, 2016): 
Revised: Such attention masks are computed via a separate MLP from the features of each region ( i, j ) , and then converted into a probability distribution using softmax or sparsemax operators (Martins & Astudillo, 2016): [Equation] where

Original: [Equation]
Revised: [Equation]

Original: so that (cid:80) i,j α t ( i, j ) = 1 .
Revised: so that (cid:80) i,j α t ( i, j ) = 1 .

Original: The sparsemax operator is similar to softmax but yields a sparse attention, leading to more stable performance.
Revised: The sparsemax operator is similar to softmax but yields a sparse attention, leading to more stable performance.

Original: Finally, the logits e t ( i, j ) from all regions are linearly combined using the attention probabilities α t :
Revised: Finally, the logits e t ( i, j ) from all regions are linearly combined using the attention probabilities α t :

Original: [Equation]
Revised: [Equation]

Original: Training.
Revised: Training.

Original: The model can be optimized with the standard cross-entropy loss L action ( a ∗ t  , (cid:98) a t  ) with respect to the ground-truth action a ∗ t  ∈
Revised: The model can be optimized with the standard cross-entropy loss L action ( a ∗ t − 1 , (cid:98) a t − 1 ) with respect to the ground-truth action a ∗ t − 1 ∈

Original: A that the agent actually has taken.
Revised: A that the agent actually has taken.

Original: Based on this formulation, the attention probability α t ( i, j ) should be high only on regions ( i, j ) that are predictive of the agent’s actions.
Revised: Based on this formulation, the attention probability α t ( i, j ) should be high only on regions ( i, j ) that are predictive of the agent’s actions.

Original: Our formulation enables learning to localize controllable entities in a fully unsupervised way without any additional supervisory signal, unlike some prior work ( e.g ., Bellemare et al. that adopts simulators to collect extra supervisory labels.
Revised: Our formulation enables learning to localize controllable entities in a fully unsupervised way without any additional supervisory signal, unlike some prior work ( e.g ., (Bellemare et al., 2012)) that adopts simulators to collect extra supervisory labels.

Original: Optimizing the parameters of ADM on on-policy data is challenging for several reasons.
Revised: Optimizing the parameters of ADM on on-policy data is challenging for several reasons.

Original: First, the ground-truth action may be unpredictable for given pairs of frames, leading to noisy labels.
Revised: First, the ground-truth action may be unpredictable for given pairs of frames, leading to noisy labels.

Original: For example, the actions taken in uncontrollable situations do not have any effect ( e.g ., when the agent is in the middle of jumping in M ONTEZUMA ’ S R EVENGE ).
Revised: For example, the actions taken in uncontrollable situations do not have any effect ( e.g ., when the agent is in the middle of jumping in M ONTEZUMA ’ S R EVENGE ).

Original: Second, since we train the ADM online along with the policy, the training examples are not independently and identically distributed, and the data distribution can shift dramatically over time.
Revised: Second, since we train the ADM online along with the policy, the training examples are not independently and identically distributed, and the data distribution can shift dramatically over time.

Original: Third, the action distribution from the agent’s policy can run into a low entropy 2 , being biased towards certain actions.
Revised: Third, the action distribution from the agent’s policy can run into a low entropy 2 , being biased towards certain actions.

Original: These issues may prevent the ADM from generalization to novel observations, which hurts exploration.
Revised: These issues may prevent the ADM from generalization to novel observations, which hurts exploration.

Original: Generally, we prefer models that quickly adapt to the policy and learn to localize the controllable regions in a robust manner.
Revised: Generally, we prefer models that quickly adapt to the policy and learn to localize the controllable regions in a robust manner.

Original: To mitigate the aforementioned issues, we adopt a few additional objective functions.
Revised: To mitigate the aforementioned issues, we adopt a few additional objective functions.

Original: We encourage the attention distribution to attain a high entropy by including an attention entropy regularization loss , i.e ., L ent = −H ( α t ) .
Revised: We encourage the attention distribution to attain a high entropy by including an attention entropy regularization loss , i.e ., L ent = −H ( α t ) .

Original: This term penalizes over-conﬁdent attention masks, making the attention closer to uniform whenever action prediction is not possible.
Revised: This term penalizes over-conﬁdent attention masks, making the attention closer to uniform whenever action prediction is not possible.

Original: We also train the logits corresponding to each grid cell independently using a separate cross-entropy loss, p ( (cid:98) a i,jt  | e t ( i, j ))
Revised: We also train the logits corresponding to each grid cell independently using a separate cross-entropy loss: p ( (cid:98) a i,jt − 1 | e t ( i, j ))

Original: =
Revised: =

Original: softmax( e t ( i, j )) .
Revised: softmax( e t ( i, j )) .

Original: These additional cross-entropy losses, denoted L i,j cell , allow the model to learn from novel observations even when attention fails to perform well at ﬁrst.
Revised: These additional cross-entropy losses, denoted L i,j cell , allow the model to learn from novel observations even when attention fails to perform well at ﬁrst.

Original: The entire training objective becomes
Revised: The entire training objective becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where λ ent is a mixing hyperparameter.
Revised: where λ ent is a mixing hyperparameter.

Original: 3. C OUNT - BASED E XPLORATION WITH C ONTINGENT R EGIONS
Revised: 3. C OUNT - BASED E XPLORATION WITH C ONTINGENT R EGIONS

Original: One natural way to take advantage of discovered contingent regions for exploration is count-based exploration.
Revised: One natural way to take advantage of discovered contingent regions for exploration is count-based exploration.

Original: The ADM can be used to localize the controllable entity ( e.g ., the agent’s avatar) from an observation s t experienced by the agent.
Revised: The ADM can be used to localize the controllable entity ( e.g ., the agent’s avatar) from an observation s t experienced by the agent.

Original: In 2D environments, a natural discretization ( x, y ) =
Revised: In 2D environments, a natural discretization ( x, y ) =

Original: argmax ( j,i ) α t ( i, j ) provides a good approximation of the agent’s location within the current observation 3 .
Revised: argmax ( j,i ) α t ( i, j ) provides a good approximation of the agent’s location within the current observation 3 .

Original: This provides a key piece of information about the current state of the agent.
Revised: This provides a key piece of information about the current state of the agent.

Original: Inspired by previous work (Bellemare et al., 2016; Tang et al., 2017), we add an exploration bonus of r + to the environment reward, where r + ( s ) = 1 / (cid:112) #( ψ ( s )) and #( ψ ( s )) denotes the visitation count of the (discrete) mapped state ψ ( s ) , which consists of the contingent region ( x, y ) .
Revised: Inspired by previous work (Bellemare et al., 2016; Tang et al., 2017), we add an exploration bonus of r + to the environment reward, where r + ( s ) = 1 / (cid:112) #( ψ ( s )) and #( ψ ( s )) denotes the visitation count of the (discrete) mapped state ψ ( s ) , which consists of the contingent region ( x, y ) .

Original: We want to ﬁnd a policy π that maximizes the expected discounted sum of environment rewards r ext plus count-based exploration rewards r + , denoted R = E π (cid:2) (cid:80) t γ t ( β 1 r ext ( s t , a t ) + β 2 r
Revised: We want to ﬁnd a policy π that maximizes the expected discounted sum of environment rewards r ext plus count-based exploration rewards r + , denoted R = E π (cid:2) (cid:80) t γ t ( β 1 r ext ( s t , a t ) + β 2 r

Original: + ( s t )) (cid:3) , where β 1 , β 2 ≥ 0 are hyperparameters that balance the weight of environment reward and the exploration bonus.
Revised: + ( s t )) (cid:3) , where β 1 , β 2 ≥ 0 are hyperparameters that balance the weight of environment reward and the exploration bonus.

Original: For every state s t encountered at time step t , we increase the counter value #( ψ ( s t )) byduring training.
Revised: For every state s t encountered at time step t , we increase the counter value #( ψ ( s t )) byduring training.

Original: The overall procedure of exploration algorithm is summarized in Algorithm 1 in the
Revised: The overall procedure of exploration algorithm is summarized in Algorithm 1 in the

Original: Appendix.
Revised: Appendix.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In the experiments below we investigate the following key questions:
Revised: In the experiments below we investigate the following key questions:

Original: • Does the contingency awareness in terms of self-localization provide a useful state abstraction for exploration
Revised: • Does the contingency awareness in terms of self-localization provide a useful state abstraction for exploration?

Original: • How well can an unsupervised model discover the ground truth abstract states?
Revised: • How well can an unsupervised model discover the ground truth abstract states?

Original: • How well does the proposed exploration strategy perform against other exploration methods?
Revised: • How well does the proposed exploration strategy perform against other exploration methods?

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTS WITH A2C

Original: We evaluate the proposed exploration strategy on several difﬁcult exploration Atari 2600 games from the Arcade Learning Environment (ALE) (Bellemare et al., 2013).
Revised: We evaluate the proposed exploration strategy on several difﬁcult exploration Atari 2600 games from the Arcade Learning Environment (ALE) (Bellemare et al., 2013).

Original: We focus on 8 Atari games including F REEWAY , F ROSTBITE , H ERO , P RIVATE E YE , M ONTEZUMA ’ S R EVENGE , Q BERT , S EAQUEST , and V ENTURE .
Revised: We focus on 8 Atari games including F REEWAY , F ROSTBITE , H ERO , P RIVATE E YE , M ONTEZUMA ’ S R EVENGE , Q BERT , S EAQUEST , and V ENTURE .

Original: In such games an agent without an effective exploration strategy can often converge to a suboptimal policy.
Revised: In such games an agent without an effective exploration strategy can often converge to a suboptimal policy.

Original: For example, as depicted in Figure 2, the Advantage Actor-Critic (A2C) baseline (Mnih et al., 2016) achieves a reward close to 0 on M ONTEZUMA ’ S R EVENGE , V ENTURE ,
Revised: For example, as depicted in Figure 2, the Advantage Actor-Critic (A2C) baseline (Mnih et al., 2016) achieves a reward close to 0 on M ONTEZUMA ’ S R EVENGE , V ENTURE ,

Original: F REEWAY , F ROSTBITE , and P RIVATE E YE , even after 100 M steps of training.
Revised: F REEWAY , F ROSTBITE , and P RIVATE E YE , even after 100 M steps of training.

Original: By contrast, our proposed technique, which augments A2C with count-based exploration with the location information learned by the attentive dynamics model, denoted A2C+CoEX (CoEX stands for “Contingency-aware
Revised: By contrast, our proposed technique, which augments A2C with count-based exploration with the location information learned by the attentive dynamics model, denoted A2C+CoEX (CoEX stands for “Contingency-aware

Original: Exploration”), signiﬁcantly outperforms the A2C baseline on 6 out of the 8 games.
Revised: Exploration”), signiﬁcantly outperforms the A2C baseline on 6 out of the 8 games.

Original: We compare our proposed A2C+CoEX technique against the following baselines:
Revised: We compare our proposed A2C+CoEX technique against the following baselines:

Original: • A2C : an implementation adopted from OpenAI baselines (Dhariwal et al., 2017) using the default hyperparameters, which serves as the building block of our more complicated baselines.
Revised: • A2C : an implementation adopted from OpenAI baselines (Dhariwal et al., 2017) using the default hyperparameters, which serves as the building block of our more complicated baselines.

Original: • A2C+Pixel-SimHash : Following (Tang et al., 2017), we map 52 × 52 gray-scale observations to 128 -bit binary codes using random projection followed by quantization (Charikar, 2002).
Revised: • A2C+Pixel-SimHash : Following (Tang et al., 2017), we map 52 × 52 gray-scale observations to 128 -bit binary codes using random projection followed by quantization (Charikar, 2002).

Original: Then, we add a count-based exploration bonus based on quantized observations.
Revised: Then, we add a count-based exploration bonus based on quantized observations.

Revised: As control experiment, we

Revised: also evaluate

Original: As control experiment, we also evaluate A2C+CoEX+RAM ∗ , our contingency-aware exploration method with the ground truth location information from game RAM, which roughly shows upperbound performance of our proposed method.
Revised: A2C+ CoEX +RAM ∗ , our contingency-aware exploration method with the ground truth location information from game RAM, which roughly shows upperbound performance of our proposed method.

Revised: [Table]

Revised: [Table]

Original: 4. I MPLEMENTATION D ETAILS
Revised: 4. I MPLEMENTATION D ETAILS

Original: For the A2C (Mnih et al., 2016) algorithm, we use 16 parallel actors to collect the agent’s experience, with 5-step rollout, which yields a minibatch of size 80 for on-policy transitions.
Revised: For the A2C (Mnih et al., 2016) algorithm, we use 16 parallel actors to collect the agent’s experience, with 5-step rollout, which yields a minibatch of size 80 for on-policy transitions.

Original: We use the lastobservation frames stacked as input, each of which is resized to 84 × 84 and converted to grayscale as in (Mnih et al., 2015; 2016).
Revised: We use the lastobservation frames stacked as input, each of which is resized to 84 × 84 and converted to grayscale as in (Mnih et al., 2015; 2016).

Original: We set the end of episode to be when the game ends, rather than when the agent loses a life.
Revised: We set the end of episode to be when the game ends, rather than when the agent loses a life.

Original: More implementation details can be found in Appendix A and B.
Revised: More implementation details can be found in Appendix A and B.

Original: For the ADM, we take observation frames of size 160 × 160 as input (resized from the raw observation of size 210 × 160 ).
Revised: For the ADM, we take observation frames of size 160 × 160 as input (resized from the raw observation of size 210 × 160 ).

Original: 4 We employ a 4-layer convolutional neural network that produces a feature map[Table] [Table] φ ( s t ) with spatial grid size of H × W = 9 × 9 .
Revised: 5 We employ a 4-layer convolutional neural network that produces a feature map φ ( s t ) with spatial grid size of H × W = 9 × 9 .

Original: As a result, the prediction of location coordinates lies in the 9 × 9 grid.
Revised: As a result, the prediction of location coordinates lies in the 9 × 9 grid.

Original: In some environments, the contingent regions within the visual observation alone are not sufﬁcient to determine the exact location of the agent within the game; for example, the coordinate alone is not enough to distinguish between different rooms in H ERO , M ONTEZUMA ’ S R EVENGE , P RIVATE E YE , etc.
Revised: In some environments, the contingent regions within the visual observation alone are not sufﬁcient to determine the exact location of the agent within the game; for example, the coordinate alone is not enough to distinguish between different rooms in H ERO , M ONTEZUMA ’ S R EVENGE , P RIVATE E YE , etc.

Original: Therefore, we introduce a discrete context representation c ∈ Z that summarizes the high-level visual context in which the agent currently lies.
Revised: Therefore, we introduce a discrete context representation c ∈ Z that summarizes the high-level visual context in which the agent currently lies.

Original: We use a simple clustering method similar to (Kulis & Jordan, 2012) which we refer to as observation embedding clustering that clusters the random projection vector of the input frames as in (Tang et al., 2017), so that different contexts are assigned to different clusters.
Revised: We use a simple clustering method similar to (Kulis & Jordan, 2012) which we refer to as observation embedding clustering that clusters the random projection vector of the input frames as in (Tang et al., 2017), so that different contexts are assigned to different clusters.

Original: We explain this heuristic approach more in detail in the Appendix D.
Revised: We explain this heuristic approach more in detail in the Appendix D.

Original: In sparse reward problems, the act of collecting a reward is rare but frequently instrumental for the future states of the environment.
Revised: In sparse reward problems, the act of collecting a reward is rare but frequently instrumental for the future states of the environment.

Original: The cumulative reward, R t = (cid:80) t − 1 t (cid:48) =
Revised: The cumulative reward, R t = (cid:80) t − 1 t (cid:48) =

Original: r ext ( s t (cid:48) , a t (cid:48) ) from the beginning of the episode up to the current step t , can provide a useful high-level behavioral context , because collecting rewards can trigger signiﬁcant changes to the agent’s state and as a result the optimal behavior can change as well.
Revised: r ext ( s t (cid:48) , a t (cid:48) ) from the beginning of the episode up to the current step t , can provide a useful high-level behavioral context , because collecting rewards can trigger signiﬁcant changes to the agent’s state and as a result the optimal behavior can change as well.

Original: In this sense, the agent should revisit the previously visited location for exploration when the context changes.
Revised: In this sense, the agent should revisit the previously visited location for exploration when the context changes.

Original: For example, in M ONTEZUMA ’ S R EVENGE , if the agent is in the ﬁrst room and the cumulative reward up to now is 0, we know the agent has not picked up the key and the optimal policy is to reach the key.
Revised: For example, in M ONTEZUMA ’ S R EVENGE , if the agent is in the ﬁrst room and the cumulative reward up to now is 0, we know the agent has not picked up the key and the optimal policy is to reach the key.

Original: However, if the cumulative reward in the ﬁrst room is 100, it means the agent has picked up the key and the next optimal goal is to open a door and move on to the next room.
Revised: However, if the cumulative reward in the ﬁrst room is 100, it means the agent has picked up the key and the next optimal goal is to open a door and move on to the next room.

Original: Therefore, we could include the cumulative reward as a part of state abstraction for exploration, which leads to empirically better performance.
Revised: Therefore, we could include the cumulative reward as a part of state abstraction for exploration, which leads to empirically better performance.

Original: To sum up, for the purpose of count-based exploration, we utilize the location ( x, y ) of the controllable entity ( i.e ., the agent) in the current observation discovered by ADM (Section 3.1), a context representation c ∈ Z that denotes the high level visual context, and cumulative environment reward R ∈ Z that represents the exploration behavioral state.
Revised: To sum up, for the purpose of count-based exploration, we utilize the location ( x, y ) of the controllable entity ( i.e ., the agent) in the current observation discovered by ADM (Section 3.1), a context representation c ∈ Z that denotes the high level visual context, and cumulative environment reward R ∈ Z that represents the exploration behavioral state.

Original: In such setting, we may denote ψ ( s ) =
Revised: In such setting, we may denote ψ ( s ) =

Original: ( x, y, c, R ) .
Revised: ( x, y, c, R ) .

Original: 4. P ERFORMANCE OF C OUNT - BASED E XPLORATION
Revised: 4. P ERFORMANCE OF C OUNT - BASED E XPLORATION

Original: Figure 2 shows the learning curves of the proposed methods on 8 Atari games.
Revised: Figure 2 shows the learning curves of the proposed methods on 8 Atari games.

Original: The performance of our method A2C+CoEX and A2C+CoEX+RAM as well as baselines A2C and A2C+Pixel-SimHash are summarized in Table 1.
Revised: The performance of our method A2C+CoEX and A2C+CoEX+RAM as well as baselines A2C and A2C+Pixel-SimHash are summarized in Table 1.

Original: In order to ﬁnd a balance between environment reward and exploration bonus reward, we perform a hyper-parameter search for the proper weight of environment reward βand exploration reward β 2 for A2C+CoEX+RAM, as well as for A2C+CoEX.
Revised: In order to ﬁnd a balance between environment reward and exploration bonus reward, we perform a hyper-parameter search for the proper weight of environment reward βand exploration reward β 2 for A2C+CoEX+RAM, as well as for A2C+CoEX.

Original: The hyper-parameters for the two ended up being the same, which is consistent with our results.
Revised: The hyper-parameters for the two ended up being the same, which is consistent with our results.

Original: For fair comparison, we also search for the proper weight of environment reward for A2C baseline.
Revised: For fair comparison, we also search for the proper weight of environment reward for A2C baseline.

Original: The best hyper-parameters for each game are shown in Table 4 in Appendix B.
Revised: The best hyper-parameters for each game are shown in Table 5 in Appendix B.

Original: Compared to the vanilla A2C, the proposed exploration strategy improves the score on all the hardexploration games.
Revised: Compared to the vanilla A2C, the proposed exploration strategy improves the score on all the hardexploration games.

Original: In Table 1, if representation ( x, y, c, R ) is perfect, A2C+CoEX+RAM achieves a signiﬁcant improvement over A2C by encouraging the agent to visit novel locations, and could nearly solve these hard exploration games, especially the games with sparse reward, as training goes on.
Revised: In Table 1, if representation ( x, y, c, R ) is perfect, A2C+CoEX+RAM achieves a signiﬁcant improvement over A2C by encouraging the agent to visit novel locations, and could nearly solve these hard exploration games, especially the games with sparse reward, as training goes on.

Original: Furthermore, A2C+CoEX using representation learned with our proposed attentive dynamics model and observation embedding clustering also outperforms the A2C baseline.
Revised: Furthermore, A2C+CoEX using representation learned with our proposed attentive dynamics model and observation embedding clustering also outperforms the A2C baseline.

Original: Especially on F REEWAY , F ROSTBITE , H ERO , M ONTEZUMA ’ S R EVENGE , Q BERT and S EAQUEST , the performance is comparable with A2C+CoEX+RAM, which demonstrates the usefulness of ADM.
Revised: Especially on F REEWAY , F ROSTBITE , H ERO , M ONTEZUMA ’ S R EVENGE , Q BERT and S EAQUEST , the performance is comparable with A2C+CoEX+RAM, which demonstrates the usefulness of ADM.

Original: Comparison to other count-based exploration methods .
Revised: Comparison to other count-based exploration methods .

Original: Table 2 contrasts the proposed method with previous state of the art results.
Revised: Table 2 contrasts the proposed method with previous state of the art results.

Original: We outperform the other methods on 5 out of 8 games.
Revised: We outperform the other methods on 5 out of 8 games.

Original: DQNPixelCNN is the strongest alternative achieving a state of the art performance on some of the most difﬁcult sparse-reward games.
Revised: DQNPixelCNN is the strongest alternative achieving a state of the art performance on some of the most difﬁcult sparse-reward games.

Original: We argue that using Q-learning as the base learner with DQNPixelCNN makes the direct comparison with A2C+CoEX not completely adequate.
Revised: We argue that using Q-learning as the base learner with DQNPixelCNN makes the direct comparison with A2C+CoEX not completely adequate.

Original: Please note that the closest count-based alternative to A2C+CoEX would be A3C+ (Bellemare et al., 2016), which augments A3C (Mnih et al., 2016) with exploration bonus derived from pseudo-count, because
Revised: Please note that the closest count-based alternative to A2C+CoEX would be A3C+ (Bellemare et al., 2016), which augments A3C (Mnih et al., 2016) with exploration bonus derived from pseudo-count, because

Original: A2C and A3C share the similar policy learning method.
Revised: A2C and A3C share the similar policy learning method.

Original: With that in mind, one can observe a clear improvement of A2C+CoEX over A3C+ on all of the Atari games.
Revised: With that in mind, one can observe a clear improvement of A2C+CoEX over A3C+ on all of the Atari games.

Original: 4. A NALYSIS OF A TTENTIVE D YNAMICS M ODEL
Revised: 4. A NALYSIS OF A TTENTIVE D YNAMICS M ODEL

Original: We also analyze the performance of the inverse dynamics model that learns the controllable dynamics of the environment.
Revised: We also analyze the performance of the inverse dynamics model that learns the controllable dynamics of the environment.

Original: As a performance metric, we report the average distance between the ground-truth agent location ( x ∗ , y ∗ ) and the predicted location ( x, y ) within the 9 × 9 grid: (cid:107) ( x, y ) − ( x ∗ , y ∗ ) (cid:107) 2 .
Revised: As a performance metric, we report the average distance between the ground-truth agent location ( x ∗ , y ∗ ) and the predicted location ( x, y ) within the 9 × 9 grid: (cid:107) ( x, y ) − ( x ∗ , y ∗ ) (cid:107) 2 .

Original: The ground-truth location of the agent is extracted from RAM 5 , then rescaled so that the observationimage frame ﬁts into the 9 × 9 grid.
Revised: The ground-truth location of the agent is extracted from RAM 6 , then rescaled so that the observation

Original: Figure 3 shows the result on 4 Atari games (M ONTEZUMA ’ S R EVENGE , S EAQUEST , H ERO , V ENTURE ).
Revised: Figure 3 shows the result on 4 Atari games (M ONTEZUMA ’ S R EVENGE , S EAQUEST , H ERO , V ENTURE ).

Original: The inverse dynamic model is able to quickly capture the location of the agent without any supervision of localization and despite the agent constantly visiting new places.
Revised: The inverse dynamic model is able to quickly capture the location of the agent without any supervision of localization and despite the agent constantly visiting new places.

Original: Typically the predicted location is on average 1 or 2 grid cells away from the ground-truth location.
Revised: Typically the predicted location is on average 1 or 2 grid cells away from the ground-truth location.

Original: Whenever a novel scene is encountered ( e.g ., the second room in M ONTEZUMA ’ S R EVENGE at around 10M frames), the average distance temporarily increases but quickly drops again as the model learns the new room.
Revised: Whenever a novel scene is encountered ( e.g ., the second room in M ONTEZUMA ’ S R EVENGE at around 10M steps), the average distance temporarily increases but quickly drops again as the model learns the new room.

Original: We provide with the video of playing agents and localization information.
Revised: We provide with the video of playing agents and localization information.

Original: 4. A NALYSIS OF O BSERVATION E MBEDDING C LUSTERING
Revised: 4. A NALYSIS OF O BSERVATION E MBEDDING C LUSTERING

Original: For games such as M ONTEZUMA ’ S R EVENGE and V ENTURE , there is a change of high-level visual contexts ( i.e ., rooms in Atari games).
Revised: For games such as M ONTEZUMA ’ S R EVENGE and V ENTURE , there is a change of high-level visual contexts ( i.e ., rooms in Atari games).

Original: To make the agent aware of it, we obtain a representation of theA high-level context and use it for exploration.
Revised: To make the agent aware of it, we obtain a representation of the high-level context and use it for exploration.

Original: The high-level visual contexts are different from each other (different layouts, objects, colors, etc.)
Revised: The high-level visual contexts are different from each other (different layouts, objects, colors, etc.)

Original: , so the embedding generated by a random projection is quite distinguishable and clustering is accurate and robust.
Revised: , so the embedding generated by a random projection is quite distinguishable and clustering is accurate and robust.

Original: For evaluation, given an observation in Atari games, we get the ground truth room number from
Revised: For evaluation, given an observation in Atari games, we get the ground truth room number from

Original: RAM, and discrete representation ( i.e ., which cluster it is assigned to)
Revised: RAM, and discrete representation ( i.e ., which cluster it is assigned to)

Original: based on the embedding from our random projection and we compare the discrete representation to the ground truth room number.
Revised: based on the embedding from our random projection and we compare the discrete representation to the ground truth room number.

Original: Adjusted Rand Index (ARI) (Rand, 1971) measures the similarity between these two data clusterings.
Revised: Adjusted Rand Index (ARI) (Rand, 1971) measures the similarity between these two data clusterings.

Original: The ARI may only yield a value between 0 and 1, and is exactly 1 when the clusterings are identical.
Revised: The ARI may only yield a value between 0 and 1, and is exactly 1 when the clusterings are identical.

Original: The curves of Adjusted Rand Index are shown in Figure 4.
Revised: The curves of Adjusted Rand Index are shown in Figure 4.

Original: For M ONTEZUMA ’ S R EVENGE and
Revised: For M ONTEZUMA ’ S R EVENGE and

Original: V ENTURE , the discrete representation as room number is roughly as good as the ground truth.
Revised: V ENTURE , the discrete representation as room number is roughly as good as the ground truth.

Original: For
Revised: For

Original: H ERO and P RIVATE E YE , because there are many rooms quite similar with each other, it is more challenging to cluster the embedding perfectly.
Revised: H ERO and P RIVATE E YE , because there are many rooms quite similar with each other, it is more challenging to cluster the embedding perfectly.

Original: From the samples shown in Figure 6 in Appendix D, we can see reasonable performances of the clustering method on all these games.
Revised: From the samples shown in Figure 7 in Appendix D, we can see reasonable performances of the clustering method on all these games.

Original: 4. D ISCUSSIONS AND F UTURE W ORK
Revised: 4. E XPERIMENTS WITH PPO

Revised: We evaluate the proposed exploration algorithm on M ONTEZUMA ’ S R EVENGE using a sticky actions environment setup (Machado et al., 2017) identical to the setup found in (Burda et al., 2018b).

Revised: The agent is trained with Proximal Policy Optimization (PPO) (Schulman et al., 2017).

Revised: The method, denoted PPO+CoEX, executes 128 parallel actors and achieves a score of 11,540 at 500M environment steps (2 billion frames), when averaged over 3 runs.

Revised: The learning curve is illustrated in Figure 5.

Revised: Since the vanilla PPO baseline achieves a score of 1,797, 8 this result is not solely due to the beneﬁts of PPO.

Revised: Table 3 presents a clearly increasing trend of the score suggesting algorithm’s stability on top of the high performance.

Revised: The implementation details are included in Appendix E.

Original:  This paper investigates whether discovering controllable dynamics via an attentive dynamics model (ADM) can help exploration in challenging sparse reward environments.
Revised: [Table] D ISCUSSIONS AND F UTURE W ORK This paper investigates whether discovering controllable dynamics via an attentive dynamics model (ADM) can help exploration in challenging sparse reward environments.

Original: We demonstrate the effectiveness of this approach by achieving signiﬁcant improvements on notoriously difﬁcult video games.
Revised: We demonstrate the effectiveness of this approach by achieving signiﬁcant improvements on notoriously difﬁcult video games.

Original: That being said, we acknowledge that our approach has certain limitations.
Revised: That being said, we acknowledge that our approach has certain limitations.

Original: Our currently presented instance of state abstraction method mainly focuses on controllable dynamics and employs a simple clustering scheme to abstract  uncontrollable elements of the scene.
Revised: Our currently presented instance of state abstraction method mainly focuses on controllable dynamics and employs a simple clustering scheme to abstract away uncontrollable elements of the scene.

Original: In more general setting, one can imagine using attentive (forward or inverse) dynamics models to learn an effective and compact abstraction of the controllable and uncontrollable dynamics as well, but we leave this to future work.
Revised: In more general setting, one can imagine using attentive (forward or inverse) dynamics models to learn an effective and compact abstraction of the controllable and uncontrollable dynamics as well, but we leave this to future work.

Original: Key elements of the current ADM method include the use of spatial attention and modelling of the dynamics.
Revised: Key elements of the current ADM method include the use of spatial attention and modelling of the dynamics.

Original: These ideas can be generalized by a set of attention-based dynamics models (ADM) operating in forward, inverse, or combined mode.
Revised: These ideas can be generalized by a set of attention-based dynamics models (ADM) operating in forward, inverse, or combined mode.

Original: Such models could use attention over a lowerdimensional embedding that corresponds to an intrinsic manifold structure from the environment ( i.e ., intuitively speaking, this also corresponds to being self-aware of (e.g., locating) where the agent is in the abstract state space ).
Revised: Such models could use attention over a lowerdimensional embedding that corresponds to an intrinsic manifold structure from the environment ( i.e ., intuitively speaking, this also corresponds to being self-aware of (e.g., locating) where the agent is in the abstract state space ).

Original: Our experiments with the inverse dynamics model suggest that the mechanism does not have to be perfectly precise, allowing for some error in practice.
Revised: Our experiments with the inverse dynamics model suggest that the mechanism does not have to be perfectly precise, allowing for some error in practice.

Original: We speculate that mapping to such subspace could be obtained by techniques of embedding learning.
Revised: We speculate that mapping to such subspace could be obtained by techniques of embedding learning.

Original: We note that RL environments with different visual characteristics may require different forms of attention-based techniques and properties of the model ( e.g ., partial observability).
Revised: We note that RL environments with different visual characteristics may require different forms of attention-based techniques and properties of the model ( e.g ., partial observability).

Original: Even though this paper focuses on 2D video games, we believe that the presented high-level ideas of learning contingency-awareness (with attention and dynamics models) are more general and could be applicable to more complex 3D environments with some extension.
Revised: Even though this paper focuses on 2D video games, we believe that the presented high-level ideas of learning contingency-awareness (with attention and dynamics models) are more general and could be applicable to more complex 3D environments with some extension.

Original: We leave this as future work.
Revised: We leave this as future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a method of providing contingency-awareness through an attentive dynamics model (ADM).
Revised: We proposed a method of providing contingency-awareness through an attentive dynamics model (ADM).

Original: It enables approximate self-localization for an RL agent in 2D environments (as a speciﬁc perspective).
Revised: It enables approximate self-localization for an RL agent in 2D environments (as a speciﬁc perspective).

Original: The agent is able to estimate its position in the space and therefore beneﬁts from a compact, informative representation of the world.
Revised: The agent is able to estimate its position in the space and therefore beneﬁts from a compact, informative representation of the world.

Original: This idea combined with a variant of count-based exploration achieves strong results in various sparse reward Atari games.
Revised: This idea combined with a variant of count-based exploration achieves strong results in various sparse reward Atari games.

Original: Furthermore, we report state-of-the-art results of 6600+ points on the infamously challenging M ONTEZUMA ’ S R EVENGE without using expert demonstrations or supervision.
Revised: Furthermore, we report state-of-the-art results of >11,000 points on the infamously challenging M ONTEZUMA ’ S R EVENGE without using expert demonstrations or supervision.

Original: Though in this work we focus mostly on 2D environments in the form of sparse-reward Atari games, we view our presented high-level concept and approach as a stepping stone towards more universal algorithms capable of similar abilities in various RL environments.
Revised: Though in this work we focus mostly on 2D environments in the form of sparse-reward Atari games, we view our presented high-level concept and approach as a stepping stone towards more universal algorithms capable of similar abilities in various RL environments.

Original: A PPENDIX A S UMMARY OF T RAINING A LGORITHM
Revised: A PPENDIX A S UMMARY OF T RAINING A LGORITHM

Original: Initialize parameter θ ADM for attentive dynamics model f ADM
Revised: Initialize parameter θ ADM for attentive dynamics model f ADM

Original: Initialize parameter θ A2C for actor-critic network
Revised: Initialize parameter θ A2C for actor-critic network

Original: Initialize parameter θ c for context embedding projector if applicable (which is not trainable)
Revised: Initialize parameter θ c for context embedding projector if applicable (which is not trainable)

Original: Initialize transition buffer E ← ∅ for each iteration do (cid:66) Collect on-policy transition samples, distributed over K parallel actors for each step t do s t ← Observe state a t ∼ π θ ( a t
Revised: Initialize transition buffer E ← ∅ for each iteration do (cid:66) Collect on-policy transition samples, distributed over K parallel actors for each step t do s t ← Observe state a t ∼ π θ ( a t

Original: | s t ) s t +1 , r ext t ← Perform action a t in the environment (cid:66) Compute the contingent region information α t +1 ← Compute the attention map of s t +1 using f ADM c ( s t +1 ) ← Compute the observation embedding cluster of s t +1 (Algorithm 2) (cid:66)
Revised: | s t ) s t +1 , r ext t ← Perform action a t in the environment (cid:66) Compute the contingent region information α t +1 ← Compute the attention map of s t +1 using f ADM c ( s t +1 ) ← Compute the observation embedding cluster of s t +1 (Algorithm 2) (cid:66)

Original: Increment state visitation counter based on the representation ψ ( s t +1 )
Revised: Increment state visitation counter based on the representation ψ ( s t +1 )

Original: ← ( argmax ( i,j ) α t +1 ( i, j ) , c
Revised: ← ( argmax ( i,j ) α t +1 ( i, j ) , c

Original: ( s t +1 ) , (cid:98) (cid:80) tk =

Revised: ( s t +1 ) , (cid:98) (cid:80) tk =

Original: r ext k (cid:99) ) #( ψ ( s t +1 ))
Revised: r ext k (cid:99) ) #( ψ ( s t +1 ))

Original: ← #( ψ ( s t +1 ))
Revised: ← #( ψ ( s t +1 ))

Original: +r + t ← 1 √ #( ψ ( s t +1 ))
Revised: +r + t ← 1 √ #( ψ ( s t +1 ))

Original: Store transition E ← E ∪ (cid:8) ( s t , a t , s t +1 , β 1 clip( r ext t , − 1 , 1) + β 2 r + t ) (cid:9) end for (cid:66) Perform actor-critic using on-policy samples in E θ A2C ← θ A2C − η ∇ θ A2C L A2C (cid:66)
Revised: Store transition E ← E ∪ (cid:8) ( s t , a t , s t +1 , β 1 clip( r ext t , − 1 , 1) + β 2 r + t ) (cid:9) end for (cid:66) Perform actor-critic using on-policy samples in E θ A2C ← θ A2C − η ∇ θ A2C L A2C (cid:66)

Original: Train the attentive dynamics network using on-policy samples in E θ ADM ← θ ADM − η ∇ θ ADM L ADM
Revised: Train the attentive dynamics network using on-policy samples in E θ ADM ← θ ADM − η ∇ θ ADM L ADM

Original: Clear transition buffer E ← ∅ end
Revised: Clear transition buffer E ← ∅ end

Original: for
Revised: for

Original: The learning objective L ADM is from Equation (5).
Revised: The learning objective L ADM is from Equation (5).

Original: The objective L A2C of Advantage Actor-Critic (A2C) is as in (Mnih et al., 2016; Dhariwal et al., 2017):
Revised: The objective L A2C of Advantage Actor-Critic (A2C) is as in (Mnih et al., 2016; Dhariwal et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where R nt =
Revised: where R nt =

Original: (cid:80) n − 1 i =

Revised: (cid:80) n − 1 i =

Original: γ i r t
Revised: γ i r t

Original: + i + γ n V
Revised: + i + γ n V

Original: θ
Revised: θ

Original: ( s t + n ) is the n -step bootstrapped return and α is a weight for the standard entropy regularization loss term H t ( π θ ) .
Revised: ( s t + n ) is the n -step bootstrapped return and α is a weight for the standard entropy regularization loss term H t ( π θ ) .

Original: We omit the subscript as θ = θ A2C when it is clear.
Revised: We omit the subscript as θ = θ A2C when it is clear.

Original: B A RCHITECTURE AND H YPERPARAMETER D ETAILS
Revised: B A RCHITECTURE AND H YPERPARAMETER D ETAILS

Original: The architecture details of the inverse dynamics network, A2C policy network, and hyper-parameters are as follows.
Revised: The architecture details of the inverse dynamics network, A2C policy network, and hyper-parameters are as follows.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: C E XPERIMENT WITH RAM INFORMATION
Revised: C E XPERIMENT WITH RAM INFORMATION

Original: In order to understand the performance of exploration with perfect representation, we extract the ground truth location of agent and room number from RAM, and then run count-based exploration with the perfect ( x, y, c, R ) .
Revised: In order to understand the performance of exploration with perfect representation, we extract the ground truth location of agent and room number from RAM, and then run count-based exploration with the perfect ( x, y, c, R ) .

Original: Figure 5 shows the learning curves of the experiments; we could see
Revised: Figure 6 shows the learning curves of the experiments; we could see

Original: A2C+CoEX+RAM acts as an upper bound performance of our proposed method.
Revised: A2C+CoEX+RAM acts as an upper bound performance of our proposed method.

Original: D O BSERVATION E MBEDDING C LUSTERING
Revised: D O BSERVATION E MBEDDING C LUSTERING

Original: We describe the detail of a method to obtain the observation embedding.
Revised: We describe the detail of a method to obtain the observation embedding.

Original: Given an observation of shape (84 , 84 , 3) , we ﬂatten the observation and project it to an embedding of dimension 64 .
Revised: Given an observation of shape (84 , 84 , 3) , we ﬂatten the observation and project it to an embedding of dimension 64 .

Original: We randomly initialize the parameter of fully-connected layer for projection, and keep the values unchanged during the training to make the embedding stationary.
Revised: We randomly initialize the parameter of fully-connected layer for projection, and keep the values unchanged during the training to make the embedding stationary.

Original: For the embedding of these observations, we cluster them based on a threshold value τ .
Revised: For the embedding of these observations, we cluster them based on a threshold value τ .

Original: The value of τ for each game with change of rooms is listed in Table 4.
Revised: The value of τ for each game with change of rooms is listed in Table 5.

Original: If the distance between the current embedding and the center mean( c ) of one cluster c is less than the threshold, we assign this embedding into this cluster and update the center as the mean value of all embedding belonging to this cluster.
Revised: If the distance between the current embedding and the center mean( c ) of one cluster c is less than the threshold, we assign this embedding into this cluster and update the center as the mean value of all embedding belonging to this cluster.

Original: If the distance between the current embedding and the center of any cluster is larger than the threshold, we create a new cluster and this embedding is assigned to this new cluster.
Revised: If the distance between the current embedding and the center of any cluster is larger than the threshold, we create a new cluster and this embedding is assigned to this new cluster.

Original: Algorithm 2 Clustering Observation Embedding
Revised: Algorithm 2 Clustering Observation Embedding

Original: Initialize parameter θ c for context embedding projector if applicable (which is not trainable)
Revised: Initialize parameter θ c for context embedding projector if applicable (which is not trainable)

Original: Initialize threshold τ for clustering
Revised: Initialize threshold τ for clustering

Original: Initialize clusters set C ← ∅ for each observation s do (cid:66) Get embedding of the observation from the random projection v ← f θ c ( s ) (cid:66) Find a cluster to which the current embedding ﬁts, if any
Revised: Initialize clusters set C ← ∅ for each observation s do (cid:66) Get embedding of the observation from the random projection v ← f θ c ( s ) (cid:66) Find a cluster to which the current embedding ﬁts, if any

Original: Find a cluster c ∈ C such that (cid:107) mean( c ) − v (cid:107) ≤ τ , or NIL if there is no such if c (cid:54) =
Revised: Find a cluster c ∈ C such that (cid:107) mean( c ) − v (cid:107) ≤ τ , or NIL if there is no such if c (cid:54) =

Original: NIL then c ← c ∪ v else
Revised: NIL then c ← c ∪ v else

Original: In Figure 6 we also show the samples of observation in each cluster, and we could see observations of the same room are assigned to the same cluster, and different clusters corresponds to different rooms.
Revised: In Figure 7 we also show the samples of observation in each cluster, and we could see observations of the same room are assigned to the same cluster, and different clusters corresponds to different rooms.

Original: E A BLATION S TUDY ON A TTENTIVE D YNAMICS M ODEL
Revised: E I MPLEMENTATION D ETAILS FOR E XPERIMENT PPO+C O EX

Revised: We use the environment with sticky actions ( i.e ., randomly repeating previous actions) as suggested by (Machado et al., 2017), identical to the one used in Exploration by Random Network Distillation (Burda et al., 2018b).

Revised: In this setup, the agent repeats the previous action with probability 0.25, preventing the algorithm from simply memorizing the correct sequence of actions and relying on determinism.

Revised: The policy network architecture and Attentive Dynamic Network architecture are the same as in our A2C+CoEX experiments (Table 4).

Revised: The remaining hyperparameters are summarized in Table 6.

Revised: [Table]

Revised: We empirically found that rescaling the one-hot encoding of the correct action label before feeding it to the ADM is helpful.

Revised: For example, when a t = (0 , 1 , 0 , . . . , 0) is reweighted to a reweighted t =

Revised: . , 0) , it leads to good location prediction and excellent end performance of the agent.

Revised: We are investigating how such modiﬁcation impacts the behavior of the ADM and what are other potential improvements that can produce even better results.

Revised: F A BLATION S TUDY ON A TTENTIVE D YNAMICS M ODEL

Original: We conduct a simple ablation study on the learning objectives of ADM, described in Equation (5).
Revised: We conduct a simple ablation study on the learning objectives of ADM, described in Equation (5).

Original: We evaluate the performance of ADM when trained on the same trajectory data under different combinations of loss terms, simulating batches of on-policy transition data to be replayed.
Revised: We evaluate the performance of ADM when trained on the same trajectory data under different combinations of loss terms, simulating batches of on-policy transition data to be replayed.

Original: The sample trajectory was obtained from an instance of A2C+CoEX+RAM and kept same across all the runs, which allows a fair comparison between different variants.
Revised: The sample trajectory was obtained from an instance of A2C+CoEX+RAM and kept same across all the runs, which allows a fair comparison between different variants.

Original: We compare the following four methods:
Revised: We compare the following four methods:

Original: • ADM (action) : train ADM using L action only • ADM (action, cell) : train ADM using L action and L cell • ADM (action, ent) : train ADM using L action and L ent • ADM (action, cell, ent) : train ADM using all losses ( L action , L cell , L ent )
Revised: • ADM (action) : train ADM using L action only • ADM (action, cell) : train ADM using L action and L cell • ADM (action, ent) : train ADM using L action and L ent • ADM (action, cell, ent) : train ADM using all losses ( L action , L cell , L ent )

Original: Figure 7 shows the average distance between the ground-truth agent location and the predicted one by
Revised: Figure 8 shows the average distance between the ground-truth agent location and the predicted one by

Original: ADM during the early stages of training.
Revised: ADM during the early stages of training.

Original: On M ONTEZUMA ’ S R EVENGE , there is only little difference between the variants although the full model worked slightly better on average.
Revised: On M ONTEZUMA ’ S R EVENGE , there is only little difference between the variants although the full model worked slightly better on average.

Original: On F REEWAY , the effect of loss terms is more clear; in the beginning the agent tends to behave suboptimally by taking mostly single action only (UP out of three action choices — UP, DOWN, and NO-OP), hence very low entropy H ( π ( ·| s )) , which can confuse the ADM of telling which part is actually controllable as the action classiﬁer would give correct answer regardless of attention.
Revised: On F REEWAY , the effect of loss terms is more clear; in the beginning the agent tends to behave suboptimally by taking mostly single action only (UP out of three action choices — UP, DOWN, and NO-OP), hence very low entropy H ( π ( ·| s )) , which can confuse the ADM of telling which part is actually controllable as the action classiﬁer would give correct answer regardless of attention.

Original: We can observe additional loss terms help the model quickly correct attention to localize the controllable object among the uncontrollable clutters with better stability.
Revised: We can observe additional loss terms help the model quickly correct attention to localize the controllable object among the uncontrollable clutters with better stability.

Original: In another ablation study, we compare the end performance of the A2C+CoEX agent with the ADM jointly trained under different loss objectives on these two games.
Revised: In another ablation study, we compare the end performance of the A2C+CoEX agent with the ADM jointly trained under different loss objectives on these three games (M ONTEZUMA ’ S R EVENGE ,

Revised: F REEWAY and S EAQUEST ).

Original: In our experiments the variant with full ADM worked best.
Revised: In our experiments, the variant with full ADM worked best on M ONTEZUMA ’ S R EVENGE and F REEWAY .

Original: The minimal training objective of ADM ( i.e ., L action ) also solely works reasonably well, but with combination of other loss terms we can attain more stable performance.
Revised: The minimal training objective of ADM ( i.e ., L action ) also solely works reasonably well, but with combination of other loss terms we can attain more stable performance.

Original: [Table]
Revised: [Table]

Original: F A BLATION S TUDY ON THE S TATE R EPRESENTATION
Revised: G A BLATION S TUDY ON THE S TATE R EPRESENTATION

Original: We present a result of additional ablation study on the state representation ψ ( s ) used in count-based exploration.
Revised: We present a result of additional ablation study on the state representation ψ ( s ) used in count-based exploration.

Original: The following baselines are considered:
Revised: The following baselines are considered:

Original: • A2C+CoEX ( c ) : Uses only the context embedding for exploration,
Revised: • A2C+CoEX ( c ) : Uses only the context embedding for exploration,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e ., ψ ( s ) =
Revised: e ., ψ ( s ) =

Original: ( c ) .
Revised: ( c ) .

Original: • A2C+CoEX
Revised: • A2C+CoEX

Original: ( c, R ) : Uses only the context embedding and the cumulative reward for exploration without contingent region information, i.e ., ψ ( s ) =
Revised: ( c, R ) : Uses only the context embedding and the cumulative reward for exploration without contingent region information, i [Table] e ., ψ ( s ) =

Original: ( c, R ) .
Revised: ( c, R ) .

Original: • A2C+CoEX
Revised: • A2C+CoEX

Original: ( x, y, c ) :
Revised: ( x, y, c ) :

Original: Uses the contingent region information ( x, y ) as well as the context embedding c , however without the cumulative reward component, i.e ., ψ ( s ) =
Revised: Uses the contingent region information ( x, y ) as well as the context embedding c , however without the cumulative reward component, i.e ., ψ ( s ) =

Original:  x, y, c ) .
Revised: ( x, y, c ) .

Original: One can also consider another baseline similar to A2C+CoEX ( c, R ) with ψ ( s ) =
Revised: One can also consider another baseline similar to A2C+CoEX ( c, R ) with ψ ( s ) =

Original: ( x, y, c, R ) where the location information ( x, y ) is replaced with random coordinates uniformly sampled from the grid.
Revised: ( x, y, c, R ) where the location information ( x, y ) is replaced with random coordinates uniformly sampled from the grid.

Original: It ablates the learned contingent regions.
Revised: It ablates the learned contingent regions.

Original: However, we found that it performs similarly to the presented A2C+CoEX( c, R ) baseline.
Revised: However, we found that it performs similarly to the presented A2C+CoEX( c, R ) baseline.

Original: The experimental results are summarized in Table 5 and Figure 9.
Revised: The experimental results are summarized in Table 7 and Figure 10.

Original: The variants without contingent regions ( i.e ., A2C+CoEX( c ) and A2C+CoEX( c, R ) performed signiﬁcantly worse in most of the games than A2C+CoEX ( x, y, c ) and A2C+CoEX ( x, y, c, R ) giving almost no improvement over the A2C baseline.
Revised: The variants without contingent regions ( i.e ., A2C+CoEX( c ) and A2C+CoEX( c, R ) performed signiﬁcantly worse in most of the games than A2C+CoEX ( x, y, c ) and A2C+CoEX ( x, y, c, R ) giving little improvement over the A2C baseline.

Original: Most notably, in the games with the hardest exploration such as M ONTEZUMA ’ S R EVENGE and V ENTURE the performance is not better than the vanilla A2C or a random policy, achieving the score as low as zero.
Revised: Most notably, in the games with the hardest exploration such as M ONTEZUMA ’ S R EVENGE and V ENTURE the performance is hardly better than the vanilla A2C or a random policy, achieving the score as low as zero.

Original: The variants with contingent region information worked best and comparable to each other.
Revised: The variants with contingent region information worked best and comparable to each other.

Original: We observe that using the cumulative reward (total score) for exploration gives a slight improvement on some environments.
Revised: We observe that using the cumulative reward (total score) for exploration gives a slight improvement on some environments.

Original: These results support the effectiveness of the learned contingency-awareness information in count-based exploration.
Revised: These results support the effectiveness of the learned contingency-awareness information in count-based exploration.