Original: E PISODIC C URIOSITY THROUGH R EACHABILITY
Revised: E PISODIC C URIOSITY THROUGH R EACHABILITY

Original: A BSTRACT
Revised: A BSTRACT

Original: Rewards are sparse in the real world and most today’s reinforcement learning algorithms struggle with such sparsity.
Revised: Rewards are sparse in the real world and most today’s reinforcement learning algorithms struggle with such sparsity.

Original: One solution to this problem is to allow the agent to create rewards for itself — thus making rewards dense and more suitable for learning.
Revised: One solution to this problem is to allow the agent to create rewards for itself — thus making rewards dense and more suitable for learning.

Original: In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus.
Revised: In particular, inspired by curious behaviour in animals, observing something novel could be rewarded with a bonus.

Original: Such bonus is summed up with the real task reward — making it possible for RL algorithms to learn from the combined reward.
Revised: Such bonus is summed up with the real task reward — making it possible for RL algorithms to learn from the combined reward.

Original: We propose a new curiosity method which uses episodic memory to form the novelty bonus.
Revised: We propose a new curiosity method which uses episodic memory to form the novelty bonus.

Original: To determine the bonus, the current observation is compared with the observations in memory.
Revised: To determine the bonus, the current observation is compared with the observations in memory.

Original: Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory — which incorporates rich information about environment dynamics.
Revised: Crucially, the comparison is done based on how many environment steps it takes to reach the current observation from those in memory — which incorporates rich information about environment dynamics.

Original: This allows us to overcome the known “couch-potato” issues of prior work — when the agent ﬁnds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences.
Revised: This allows us to overcome the known “couch-potato” issues of prior work — when the agent ﬁnds a way to instantly gratify itself by exploiting actions which lead to hardly predictable consequences.

Original: We test our approach in visually rich 3 D environments in VizDoom , DMLab and MuJoCo .
Revised: We test our approach in visually rich 3 D environments in VizDoom , DMLab and MuJoCo .

Original: In navigational tasks from VizDoom and DMLab , our agent outperforms the state-of-the-art curiosity method ICM.
Revised: In navigational tasks from VizDoom and DMLab , our agent outperforms the state-of-the-art curiosity method ICM.

Original: In MuJoCo , an ant equipped with our curiosity module learns locomotion out of the ﬁrst-person-view curiosity only.
Revised: In MuJoCo , an ant equipped with our curiosity module learns locomotion out of the ﬁrst-person-view curiosity only.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many real-world tasks have sparse rewards.
Revised: Many real-world tasks have sparse rewards.

Original: For example, animals searching for food may need to go many miles without any reward from the environment.
Revised: For example, animals searching for food may need to go many miles without any reward from the environment.

Original: Standard reinforcement learning algorithms struggle with such tasks because of reliance on simple action entropy maximization as a source of exploration behaviour.
Revised: Standard reinforcement learning algorithms struggle with such tasks because of reliance on simple action entropy maximization as a source of exploration behaviour.

Original: Multiple approaches were proposed to achieve better explorative policies.
Revised: Multiple approaches were proposed to achieve better explorative policies.

Original: One way is to give a reward bonus which facilitates exploration by rewarding novel observations.
Revised: One way is to give a reward bonus which facilitates exploration by rewarding novel observations.

Original: The reward bonus is summed up with the original task reward and optimized by standard RL algorithms.
Revised: The reward bonus is summed up with the original task reward and optimized by standard RL algorithms.

Original: Such an approach is motivated by neuroscience studies of animals: an animal has an ability to reward itself for something novel – the mechanism biologically built into its dopamine release system.
Revised: Such an approach is motivated by neuroscience studies of animals: an animal has an ability to reward itself for something novel – the mechanism biologically built into its dopamine release system.

Original: How exactly this bonus is formed remains an open question.
Revised: How exactly this bonus is formed remains an open question.

Original: Many modern curiosity formulations aim at maximizing “surprise” — inability to predict the future.
Revised: Many modern curiosity formulations aim at maximizing “surprise” — inability to predict the future.

Original: This approach makes perfect sense but, in fact, is far from perfect.
Revised: This approach makes perfect sense but, in fact, is far from perfect.

Original: To show why, let us consider a thought experiment.
Revised: To show why, let us consider a thought experiment.

Original: Imagine an agent is put into a 3 D maze.
Revised: Imagine an agent is put into a 3 D maze.

Original: There is a precious goal somewhere in the maze which would give a large reward.
Revised: There is a precious goal somewhere in the maze which would give a large reward.

Original: Now, the agent is also given a remote control to a
Revised: Now, the agent is also given a remote control to a

Original: TV and can switch the channels.
Revised: TV and can switch the channels.

Original: Every switch shows a random image (say, from a ﬁxed set of images).
Revised: Every switch shows a random image (say, from a ﬁxed set of images).

Original: The curiosity formulations which optimize surprise would rejoice because the result of the channel switching action is unpredictable.
Revised: The curiosity formulations which optimize surprise would rejoice because the result of the channel switching action is unpredictable.

Original: The agent would be drawn to the TV instead of looking for a goal in the environment (this was indeed observed in (Burda et al., 2018)).
Revised: The agent would be drawn to the TV instead of looking for a goal in the environment (this was indeed observed in (Burda et al., 2018a)).

Original: So, should we call the channel switching behaviour curious?
Revised: So, should we call the channel switching behaviour curious?

Original: Maybe, but it is unproductive for the original sparsereward goal-reaching task.
Revised: Maybe, but it is unproductive for the original sparsereward goal-reaching task.

Original: What would be a deﬁnition of curiosity which does not suffer from such “couch-potato” behaviour?
Revised: What would be a deﬁnition of curiosity which does not suffer from such “couch-potato” behaviour?

Original: We propose a new curiosity deﬁnition based on the following intuition.
Revised: We propose a new curiosity deﬁnition based on the following intuition.

Original: If the agent knew the observation after changing a TV channel is only one step away from the observation before doing that — it probably would not be so interesting to change the channel in the ﬁrst place (too easy).
Revised: If the agent knew the observation after changing a TV channel is only one step away from the observation before doing that — it probably would not be so interesting to change the channel in the ﬁrst place (too easy).

Original: This intuition can be formalized as giving a reward only for those observations which take some effort to
Revised: This intuition can be formalized as giving a reward only for those observations which take some effort to

Original: [Figure]
Revised: [Figure]

Original: reach (outside the already explored part of the environment).
Revised: reach (outside the already explored part of the environment).

Original: The effort is measured in the number of environment steps.
Revised: The effort is measured in the number of environment steps.

Original: To estimate it we train a neural network approximator: given two observations, it would predict how many steps separate them.
Revised: To estimate it we train a neural network approximator: given two observations, it would predict how many steps separate them.

Original: The concept of novelty via reachability is illustrated in Figure 1.
Revised: The concept of novelty via reachability is illustrated in Figure 1.

Original: To make the description above practically implementable, there is still one piece missing though.
Revised: To make the description above practically implementable, there is still one piece missing though.

Original: For determining the novelty of the current observation, we need to keep track of what was already explored in the environment.
Revised: For determining the novelty of the current observation, we need to keep track of what was already explored in the environment.

Original: A natural candidate for that purpose would be episodic memory: it stores instances of the past which makes it easy to apply the reachability approximator on pairs of current and past observations.
Revised: A natural candidate for that purpose would be episodic memory: it stores instances of the past which makes it easy to apply the reachability approximator on pairs of current and past observations.

Original: Our method works as follows.
Revised: Our method works as follows.

Original: The agent starts with an empty memory at the beginning of the episode and at every step compares the current observation with the observations in memory to determine novelty.
Revised: The agent starts with an empty memory at the beginning of the episode and at every step compares the current observation with the observations in memory to determine novelty.

Original: If the current observation is indeed novel — takes more steps to reach from observations in memory than a threshold — the agent rewards itself with a bonus and adds the current observation to the episodic memory.
Revised: If the current observation is indeed novel — takes more steps to reach from observations in memory than a threshold — the agent rewards itself with a bonus and adds the current observation to the episodic memory.

Original: The process continues until the end of the episode, when the memory is wiped clean.
Revised: The process continues until the end of the episode, when the memory is wiped clean.

Original: We benchmark our method on a range of tasks from visually rich 3 D environments VizDoom , DMLab and MuJoCo .
Revised: We benchmark our method on a range of tasks from visually rich 3 D environments VizDoom , DMLab and MuJoCo .

Original: We conduct the comparison with other methods — including the state-of-the-art curiosity method ICM
Revised: We conduct the comparison with other methods — including the state-of-the-art curiosity method ICM

Original: (Pathak et al., 2017) — under the same budget of environment interactions.
Revised: (Pathak et al., 2017) — under the same budget of environment interactions.

Original: First, we use the VizDoom environments from prior work to establish that our re-implementation of the ICM baseline is correct — and also demonstrate at least 2 times faster convergence of our method with respect to the baseline.
Revised: First, we use the VizDoom environments from prior work to establish that our re-implementation of the ICM baseline is correct — and also demonstrate at least 2 times faster convergence of our method with respect to the baseline.

Original: Second, in the randomized procedurally generated environments from
Revised: Second, in the randomized procedurally generated environments from

Original: DMLab our method turns out to be more robust to spurious behaviours than the method
Revised: DMLab our method turns out to be more robust to spurious behaviours than the method

Original: ICM: while the baseline learns a persistent ﬁring behaviour in navigational tasks (thus creating interesting pictures for itself), our method learns a reasonable explorative behaviour.
Revised: ICM: while the baseline learns a persistent ﬁring behaviour in navigational tasks (thus creating interesting pictures for itself), our method learns a reasonable explorative behaviour.

Original: In terms of quantitative evaluation, our method reaches the goal at least 2 times more often in the procedurally generated test levels in DMLab with a very sparse reward.
Revised: In terms of quantitative evaluation, our method reaches the goal at least 2 times more often in the procedurally generated test levels in DMLab with a very sparse reward.

Original: Third, when comparing the behaviour of the agent in the complete absence of rewards, our method covers at least 4 times more area (measured in discrete ( x, y ) coordinate cells) than the baseline ICM.
Revised: Third, when comparing the behaviour of the agent in the complete absence of rewards, our method covers at least 4 times more area (measured in discrete ( x, y ) coordinate cells) than the baseline ICM.

Original: Fourth, we demonstrate that our curiosity bonus does not signiﬁcantly deteriorate performance of the plain PPO algorithm (Schulman et al., 2017) in two tasks with dense reward in DMLab .
Revised: Fourth, we demonstrate that our curiosity bonus does not signiﬁcantly deteriorate performance of the plain PPO algorithm (Schulman et al., 2017) in two tasks with dense reward in DMLab .

Original: Finally, we demonstrate that an ant in a MuJoCo environment can learn locomotion purely from our curiosity reward computed based on the ﬁrst-person view.
Revised: Finally, we demonstrate that an ant in a MuJoCo environment can learn locomotion purely from our curiosity reward computed based on the ﬁrst-person view.

Original: The implementation of our method is publicly available 1 .
Revised: The implementation of our method is publicly available 1 .

Original:  E PISODIC C URIOSITY
Revised:  E PISODIC C URIOSITY

Original: We consider an agent which interacts with an environment.
Revised: We consider an agent which interacts with an environment.

Original: The interactions happen at discrete time steps over the episodes of limited duration T .
Revised: The interactions happen at discrete time steps over the episodes of limited duration T .

Original: At each time step t , the environment provides the agent with an observation o t from the observational space
Revised: At each time step t , the environment provides the agent with an observation o t from the observational space

Original: O
Revised: O

Original: (we consider images), samples an action a t from a set of actions
Revised: (we consider images), samples an action a t from a set of actions

Original: A using a probabilistic policy π ( o t ) and receives a scalar reward r t ∈ R together with the new observation
Revised: A using a probabilistic policy π ( o t ) and receives a scalar reward r t ∈ R together with the new observation

Original: o t +1 and an end-of-episode indicator.
Revised: o t +1 and an end-of-episode indicator.

Original: The goal of the agent is to optimize the expectation of the discounted sum of rewards during the episode S
Revised: The goal of the agent is to optimize the expectation of the discounted sum of rewards during the episode S

Original: = (cid:80) t γ t r t .
Revised: = (cid:80) t γ t r t .

Original: In this work we primarily focus on the tasks where rewards r t are sparse — that is, zero for most of the time steps t .
Revised: In this work we primarily focus on the tasks where rewards r t are sparse — that is, zero for most of the time steps t .

Original: Under such conditions commonly used RL algorithms (e.g., PPO Schulman et al.
Revised: Under such conditions commonly used RL algorithms (e.g., PPO Schulman et al.

Original: [Figure]
Revised: [Figure]

Original: oooooooopositive negative aaaaaaa
Revised: oooooooopositive negative aaaaaaa

Original: (2017)) do not work well.
Revised: (2017)) do not work well.

Original: We further introduce an episodic curiosity (EC) module which alleviates this problem.
Revised: We further introduce an episodic curiosity (EC) module which alleviates this problem.

Original: The purpose of this module is to produce a reward bonus b t which is further summed up with the task reward r t to give an augmented reward (cid:98) r t =
Revised: The purpose of this module is to produce a reward bonus b t which is further summed up with the task reward r t to give an augmented reward (cid:98) r t =

Original: r t + b t .
Revised: r t + b t .

Original: The augmented reward has a nice property from the RL point of view — it is a dense reward.
Revised: The augmented reward has a nice property from the RL point of view — it is a dense reward.

Original: Learning with such reward is faster, more stable and often leads to better ﬁnal performance in terms of the cumulative task reward S .
Revised: Learning with such reward is faster, more stable and often leads to better ﬁnal performance in terms of the cumulative task reward S .

Original: In the following section we describe the key components of our episodic curiosity module.
Revised: In the following section we describe the key components of our episodic curiosity module.

Original: 2. E PISODIC C URIOSITY M ODULE
Revised: 2. E PISODIC C URIOSITY M ODULE

Original: The episodic curiosity (EC) module takes the current observation o as input and produces a reward bonus b .
Revised: The episodic curiosity (EC) module takes the current observation o as input and produces a reward bonus b .

Original: The module consists of both parametric and non-parametric components.
Revised: The module consists of both parametric and non-parametric components.

Original: There are two parametric components: an embedding network E : O → R n and a comparator network C : R n × R n → [0 , 1] .
Revised: There are two parametric components: an embedding network E : O → R n and a comparator network C : R n × R n → [0 , 1] .

Original: Those parametric components are trained together to predict reachability as parts of the reachability network — shown in Figure 2.
Revised: Those parametric components are trained together to predict reachability as parts of the reachability network — shown in Figure 2.

Original: There are also two non-parametric components: an episodic memory buffer M and a reward bonus estimation function B .
Revised: There are also two non-parametric components: an episodic memory buffer M and a reward bonus estimation function B .

Original: The high-level overview of the system is shown in Figure 3.
Revised: The high-level overview of the system is shown in Figure 3.

Original: Next, we give a detailed explanation of all the components.
Revised: Next, we give a detailed explanation of all the components.

Original: Embedding and comparator networks.
Revised: Embedding and comparator networks.

Original: Both networks are designed to function jointly for estimating within- k -step-reachability of one observation o i from another observation o j as parts of a reachability network R ( o i , o j )
Revised: Both networks are designed to function jointly for estimating within- k -step-reachability of one observation o i from another observation o j as parts of a reachability network R ( o i , o j )

Original: =
Revised: =

Original: C ( E ( o i ) , E ( o j )) .
Revised: C ( E ( o i ) , E ( o j )) .

Original: This is a siamese architecture similar to (Zagoruyko & Komodakis, 2015).
Revised: This is a siamese architecture similar to (Zagoruyko & Komodakis, 2015).

Original: The architecture is shown in Figure 2.
Revised: The architecture is shown in Figure 2.

Original: R-network is a classiﬁer trained with a logistic regression loss: it predicts values close to 0 if probability of two observations being reachable from one another in k steps is low, and values close to 1 when this probability is high.
Revised: R-network is a classiﬁer trained with a logistic regression loss: it predicts values close to 0 if probability of two observations being reachable from one another in k steps is low, and values close to 1 when this probability is high.

Original: Inside the episodic curiosity the two networks are used separately to save up both computation and memory.
Revised: Inside the episodic curiosity the two networks are used separately to save up both computation and memory.

Original: Episodic memory.
Revised: Episodic memory.

Original: The episodic memory buffer M stores embeddings of past observations from the current episode, computed with the embedding network E .
Revised: The episodic memory buffer M stores embeddings of past observations from the current episode, computed with the embedding network E .

Original: The memory buffer has a limited capacity K to avoid memory and performance issues.
Revised: The memory buffer has a limited capacity K to avoid memory and performance issues.

Original: At every step, the embedding of the current observation might be added to the memory.
Revised: At every step, the embedding of the current observation might be added to the memory.

Original: What to do when the capacity is exceeded?
Revised: What to do when the capacity is exceeded?

Original: One solution we found working well in practice is to substitute a random element in memory with the current element.
Revised: One solution we found working well in practice is to substitute a random element in memory with the current element.

Original: This way there are still more fresh elements in memory than older ones, but the older elements are not totally neglected.
Revised: This way there are still more fresh elements in memory than older ones, but the older elements are not totally neglected.

Original: Reward bonus estimation module.
Revised: Reward bonus estimation module.

Original: The purpose of this module is to check for reachable observations in memory and if none are found — assign larger reward bonus to the current time step.
Revised: The purpose of this module is to check for reachable observations in memory and if none are found — assign larger reward bonus to the current time step.

Original: The check is done by comparing embeddings in memory to the current embedding via comparator network.
Revised: The check is done by comparing embeddings in memory to the current embedding via comparator network.

Original: Essentially, this check insures that no observation in memory can be reached by taking only a few actions from the current state — our characterization of novelty.
Revised: Essentially, this check insures that no observation in memory can be reached by taking only a few actions from the current state — our characterization of novelty.

Original: 2. B ONUS C OMPUTATION A LGORITHM .
Revised: 2. B ONUS C OMPUTATION A LGORITHM .

Original: At every time step, the current observation o goes through the embedding network producing the embedding vector e = E ( o ) .
Revised: At every time step, the current observation o goes through the embedding network producing the embedding vector e = E ( o ) .

Original: This embedding vector is compared with the stored embeddings in the memory buffer M =
Revised: This embedding vector is compared with the stored embeddings in the memory buffer M =

Original: (cid:10) e 1 , . . .
Revised: (cid:10) e 1 , . . .

Original: , e | M | (cid:11) via the comparator network C where | M | is the current number of elements in memory.
Revised: , e | M | (cid:11) via the comparator network C where | M | is the current number of elements in memory.

Original: This comparator network ﬁlls the reachability buffer with values c i
Revised: This comparator network ﬁlls the reachability buffer with values c i

Original: =
Revised: =

Original: C ( e i , e ) , i = 1 , | M | .
Revised: C ( e i , e ) , i = 1 , | M | .

Original: [Figure]
Revised: [Figure]

Original: Then the similarity score between the memory buffer and the current embedding is computed from the reachability buffer as (with a slight abuse of notation)
Revised: Then the similarity score between the memory buffer and the current embedding is computed from the reachability buffer as (with a slight abuse of notation)

Original: [Equation]
Revised: [Equation]

Original: where the aggregation function F is a hyperparameter of our method.
Revised: where the aggregation function F is a hyperparameter of our method.

Original: Theoretically, F = max would be a good choice, however, in practice it is prone to outliers coming from the parametric embedding and comparator networks.
Revised: Theoretically, F = max would be a good choice, however, in practice it is prone to outliers coming from the parametric embedding and comparator networks.

Original: Empirically, we found that 90 -th percentile works well as a robust substitute to maximum.
Revised: Empirically, we found that 90 -th percentile works well as a robust substitute to maximum.

Original: As a curiosity bonus, we take
Revised: As a curiosity bonus, we take

Original: [Equation]
Revised: [Equation]

Original: where α ∈ R + and β ∈ R are hyperparameters of our method.
Revised: where α ∈ R + and β ∈ R are hyperparameters of our method.

Original: The value of α depends on the scale of task rewards — we will discuss how to select it in the experimental section.
Revised: The value of α depends on the scale of task rewards — we will discuss how to select it in the experimental section.

Original: The value of β determines the sign of the reward — and thus could bias the episodes to be shorter or longer.
Revised: The value of β determines the sign of the reward — and thus could bias the episodes to be shorter or longer.

Original: Empirically, β = 0 .
Revised: Empirically, β = 0 .

Original: 5 works well for ﬁxed-duration episodes, and β = 1 is preferred if an episode could have variable length.
Revised: 5 works well for ﬁxed-duration episodes, and β = 1 is preferred if an episode could have variable length.

Original: After the bonus computation, the observation embedding is added to memory if the bonus b is larger than a novelty threshold b novelty .
Revised: After the bonus computation, the observation embedding is added to memory if the bonus b is larger than a novelty threshold b novelty .

Original: This check is necessary for the following reason.
Revised: This check is necessary for the following reason.

Original: If every observation embedding is added to the memory buffer, the observation from the current step will always be reachable from the previous step.
Revised: If every observation embedding is added to the memory buffer, the observation from the current step will always be reachable from the previous step.

Original: Thus, the reward would never be granted.
Revised: Thus, the reward would never be granted.

Original: The threshold b novelty induces a discretization in the embedding space.
Revised: The threshold b novelty induces a discretization in the embedding space.

Original: Intuitively, this makes sense: only “distinct enough” memories are stored.
Revised: Intuitively, this makes sense: only “distinct enough” memories are stored.

Original: As a side beneﬁt, the memory buffer stores information with much less redundancy.
Revised: As a side beneﬁt, the memory buffer stores information with much less redundancy.

Original: We refer the reader to the video 2 which visualizes the curiosity reward bonus and the memory state during the operation of the algorithm.
Revised: We refer the reader to the video 2 which visualizes the curiosity reward bonus and the memory state during the operation of the algorithm.

Original: 2. R EACHABILITY N ETWORK T RAINING
Revised: 2. R EACHABILITY N ETWORK T RAINING

Original: If the full transition graph in Figure 1 was available, there would be no need of a reachability network and the novelty could be computed analytically through the shortest-path algorithm.
Revised: If the full transition graph in Figure 1 was available, there would be no need of a reachability network and the novelty could be computed analytically through the shortest-path algorithm.

Original: However, normally we have access only to the sequence of observations which the agent receives while acting.
Revised: However, normally we have access only to the sequence of observations which the agent receives while acting.

Original: Fortunately, as suggested by (Savinov et al., 2018), even a simple observation sequence graph could still be used for training a reasonable approximator to the real step-distance.
Revised: Fortunately, as suggested by (Savinov et al., 2018), even a simple observation sequence graph could still be used for training a reasonable approximator to the real step-distance.

Original: This procedure is illustrated in Figure 2.
Revised: This procedure is illustrated in Figure 2.

Original: This procedure takes as input a sequence of observations o 1 , . . .
Revised: This procedure takes as input a sequence of observations o 1 , . . .

Original: , o N and forms pairs from those observations.
Revised: , o N and forms pairs from those observations.

Original: The pairs ( o i , o j ) where | i − j | ≤ k are taken as positive (reachable) examples while the pairs with | i − j | > γk become negative examples.
Revised: The pairs ( o i , o j ) where | i − j | ≤ k are taken as positive (reachable) examples while the pairs with | i − j | > γk become negative examples.

Original: The hyperparameter γ is necessary to create a gap between positive and negative examples.
Revised: The hyperparameter γ is necessary to create a gap between positive and negative examples.

Original: In the end, the network is trained with logistic regression loss to output the probability of the positive (reachable) class.
Revised: In the end, the network is trained with logistic regression loss to output the probability of the positive (reachable) class.

Original: In our work, we have explored two settings for training a reachability network: on random policy and together with the task-solving policy (online training).
Revised: In our work, we have explored two settings for training a reachability network: on random policy and together with the task-solving policy (online training).

Original: The ﬁrst version generally follows the training protocol proposed by (Savinov et al., 2018).
Revised: The ﬁrst version generally follows the training protocol proposed by (Savinov et al., 2018).

Original: We put the agent into exactly the same conditions where it will be eventually tested: same episode duration and same action set.
Revised: We put the agent into exactly the same conditions where it will be eventually tested: same episode duration and same action set.

Original: The agent takes
Revised: The agent takes

Original: [Figure]
Revised: [Figure]

Original: random actions from the actions set.
Revised: random actions from the actions set.

Original: Given the environment interaction budget ( 2 . 5 M repeated steps in DMLab , 300 K 4 -repeated steps in VizDoom ), the agent ﬁlls in the replay buffer with observations coming from its interactions with the environment, and forms training pairs by sampling from this replay buffer randomly.
Revised: Given the environment interaction budget ( 2 . 5 M repeated steps in DMLab , 300 K 4 -repeated steps in VizDoom ), the agent ﬁlls in the replay buffer with observations coming from its interactions with the environment, and forms training pairs by sampling from this replay buffer randomly.

Original: The second version collects the data on-policy, and re-trains the reachability network every time after a ﬁxed number of environment interactions is performed.
Revised: The second version collects the data on-policy, and re-trains the reachability network every time after a ﬁxed number of environment interactions is performed.

Original: We provide the details of R-network training in the supplementary material.
Revised: We provide the details of R-network training in the supplementary material.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: We test our method in multiple environments from VizDoom (Kempka et al., 2016), DMLab (Beattie et al., 2016) and MuJoCo (Todorov et al., 2012; Schulman et al., 2015).
Revised: We test our method in multiple environments from VizDoom (Kempka et al., 2016), DMLab (Beattie et al., 2016) and MuJoCo (Todorov et al., 2012; Schulman et al., 2015).

Original: The experiments in VizDoom allow us to verify that our re-implementation of the previous state-of-the-art curiosity method
Revised: The experiments in VizDoom allow us to verify that our re-implementation of the previous state-of-the-art curiosity method

Original: ICM (Pathak et al., 2017) is correct.
Revised: ICM (Pathak et al., 2017) is correct.

Original: The experiments in DMLab allow us to extensively test the generalization of our method as well as baselines — DMLab provides convenient procedural level generation capabilities which allows us to train and test RL methods on hundreds of levels.
Revised: The experiments in DMLab allow us to extensively test the generalization of our method as well as baselines — DMLab provides convenient procedural level generation capabilities which allows us to train and test RL methods on hundreds of levels.

Original: The experiments in MuJoCo allow us to show the generality of our method.
Revised: The experiments in MuJoCo allow us to show the generality of our method.

Original: Due to space limits, the
Revised: Due to space limits, the

Original: MuJoCo experiments are described in the supplementary material.
Revised: MuJoCo experiments are described in the supplementary material.

Original: The examples of tasks are shown in Figure 4.
Revised: The examples of tasks are shown in Figure 4.

Original: Environments.
Revised: Environments.

Original: Both VizDoom and DMLab environments provide rich maze-like 3 D environments.
Revised: Both VizDoom and DMLab environments provide rich maze-like 3 D environments.

Original: The observations are given to the agent in the form of images.
Revised: The observations are given to the agent in the form of images.

Original: For VizDoom , we use 84 ×grayscale images as input.
Revised: For VizDoom , we use 84 ×grayscale images as input.

Original: For DMLab , we use 84 × 84 RGB images as input.
Revised: For DMLab , we use 84 × 84 RGB images as input.

Original: The agent operates with a discrete action set which comprises different navigational actions
Revised: The agent operates with a discrete action set which comprises different navigational actions

Original: Basic RL algorithm.
Revised: Basic RL algorithm.

Original: We choose the commonly used PPO algorithm from the open-source implementation 3 as our basic RL algorithm.
Revised: We choose the commonly used PPO algorithm from the open-source implementation 3 as our basic RL algorithm.

Original: The policy and value functions are represented as CNNs to reduce number of hyperparameters — LSTMs are harder to tune and such tuning is orthogonal to the contribution of the paper.
Revised: The policy and value functions are represented as CNNs to reduce number of hyperparameters — LSTMs are harder to tune and such tuning is orthogonal to the contribution of the paper.

Original: We apply PPO to the sum of the task reward and the bonus reward coming from speciﬁc curiosity algorithms.
Revised: We apply PPO to the sum of the task reward and the bonus reward coming from speciﬁc curiosity algorithms.

Original: The hyperparameters of the PPO algorithm are given in the supplementary material.
Revised: The hyperparameters of the PPO algorithm are given in the supplementary material.

Original: We use only two sets of hyperparameters: one for all VizDoom environments and the other one for all DMLab environments.
Revised: We use only two sets of hyperparameters: one for all VizDoom environments and the other one for all DMLab environments.

Original: Baseline methods.
Revised: Baseline methods.

Original: The simplest baseline for our approach is just the basic RL algorithm applied to the task reward.
Revised: The simplest baseline for our approach is just the basic RL algorithm applied to the task reward.

Original: As suggested by the prior work and our experiments, this is a relatively weak baseline in the tasks where reward is sparse.
Revised: As suggested by the prior work and our experiments, this is a relatively weak baseline in the tasks where reward is sparse.

Original: As the second baseline, we take the state-of-the-art curiosity method ICM
Revised: As the second baseline, we take the state-of-the-art curiosity method ICM

Original: (Pathak et al., 2017).
Revised: (Pathak et al., 2017).

Original: As follows from the results in (Pathak et al., 2017; Fu et al., 2017), ICM is superior to methods
Revised: As follows from the results in (Pathak et al., 2017; Fu et al., 2017), ICM is superior to methods

Original: VIME (Houthooft et al., 2016), #Exploration (Tang et al., 2017) and EX 2 (Fu et al., 2017) on the curiosity tasks in visually rich 3 D environments.
Revised: VIME (Houthooft et al., 2016), #Exploration (Tang et al., 2017) and EX 2 (Fu et al., 2017) on the curiosity tasks in visually rich 3 D environments.

Original: Finally, as a sanity check, we introduce a novel baseline method which we call Grid Oracle.
Revised: Finally, as a sanity check, we introduce a novel baseline method which we call Grid Oracle.

Original: Since we can access current ( x, y ) coordinates of the agent in all environments, we are able
Revised: Since we can access current ( x, y ) coordinates of the agent in all environments, we are able

Original: to directly
Revised: to directly

Original: [Figure]
Revised: [Figure]

Original: discretize the world in 2 D cells and reward the agent for visiting as many cells as possible during the episode (the reward bonus is proportional to the number of cells visited).
Revised: discretize the world in 2 D cells and reward the agent for visiting as many cells as possible during the episode (the reward bonus is proportional to the number of cells visited).

Original: At the end of the episode, cell visit counts are zeroed.
Revised: At the end of the episode, cell visit counts are zeroed.

Original: The reader should keep in mind that this baseline uses privileged information not available to other methods (including our own method EC).
Revised: The reader should keep in mind that this baseline uses privileged information not available to other methods (including our own method EC).

Original: While this privileged information is not guaranteed to lead to success in any particular RL task, we do observe this baseline to perform strongly in many tasks, especially in complicated DMLab environments.
Revised: While this privileged information is not guaranteed to lead to success in any particular RL task, we do observe this baseline to perform strongly in many tasks, especially in complicated DMLab environments.

Original: The Grid
Revised: The Grid

Original: Oracle baseline has two hyperparameters: the weight for combining Grid Oracle reward with the task reward and the cell
Revised: Oracle baseline has two hyperparameters: the weight for combining Grid Oracle reward with the task reward and the cell

Original: Hyperparameter tuning.
Revised: Hyperparameter tuning.

Original: As DMLab environments are procedurally generated, we perform tuning on the validation set, disjoint with the training and test sets.
Revised: As DMLab environments are procedurally generated, we perform tuning on the validation set, disjoint with the training and test sets.

Original: The tuning is done on one of the environments and then the same hyperparameters are re-used for all other environments.
Revised: The tuning is done on one of the environments and then the same hyperparameters are re-used for all other environments.

Original: VizDoom environments are not procedurally generated, so there is no trivial way to have proper training/validation/test splits — so we tune on the same environment (as typical in the prior RL work for the environments without splits).
Revised: VizDoom environments are not procedurally generated, so there is no trivial way to have proper training/validation/test splits — so we tune on the same environment (as typical in the prior RL work for the environments without splits).

Original: When tuning, we consider the mean ﬁnal reward of 10 training runs with the same set of hyperparameters as the objective — thus we do not perform any seed tuning.
Revised: When tuning, we consider the mean ﬁnal reward of 10 training runs with the same set of hyperparameters as the objective — thus we do not perform any seed tuning.

Original: All hyperparameter values are listed in the supplementary material.
Revised: All hyperparameter values are listed in the supplementary material.

Original: Note that although bonus scalar α depends on the range of task rewards, the environments in VizDoom and DMLab have similar ranges within each platform — so our approach with re-using α for multiple environments works.
Revised: Note that although bonus scalar α depends on the range of task rewards, the environments in VizDoom and DMLab have similar ranges within each platform — so our approach with re-using α for multiple environments works.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we describe the speciﬁc tasks we are solving and experimental results for all considered methods on those tasks.
Revised: In this section, we describe the speciﬁc tasks we are solving and experimental results for all considered methods on those tasks.

Original: There are 4 methods to report: PPO, PPO + ICM, PPO + Grid Oracle and PPO + EC (our method).
Revised: There are 4 methods to report: PPO, PPO + ICM, PPO + Grid Oracle and PPO + EC (our method).

Original: First, we test static-maze goal reaching in VizDoom environments from prior work to verify that our baseline re-implementation is correct.
Revised: First, we test static-maze goal reaching in VizDoom environments from prior work to verify that our baseline re-implementation is correct.

Original: Second, we test the goal-reaching behaviour in procedurally generated mazes in DMLab .
Revised: Second, we test the goal-reaching behaviour in procedurally generated mazes in DMLab .

Original: Third, we train no-reward (pure curiosity) maze exploration on the levels from DMLab and report Grid Oracle reward as an approximate measure of the maze coverage.
Revised: Third, we train no-reward (pure curiosity) maze exploration on the levels from DMLab and report Grid Oracle reward as an approximate measure of the maze coverage.

Original: Finally, we demonstrate that our curiosity bonus does not signiﬁcantly deteriorate performance in two dense reward tasks in DMLab .
Revised: Finally, we demonstrate that our curiosity bonus does not signiﬁcantly deteriorate performance in two dense reward tasks in DMLab .

Original: All the experiments were conducted under the same environment interaction budget for all methods (R-network pre-training is included in this budget).
Revised: All the experiments were conducted under the same environment interaction budget for all methods (R-network pre-training is included in this budget).

Original: The videos of all trained agents in all environments are available online 4 .
Revised: The videos of all trained agents in all environments are available online 4 .

Original: For additional experiments we refer the reader to the supplementary material: there we show that
Revised: For additional experiments we refer the reader to the supplementary material: there we show that

Original: R-network can successfully generalize between environments, demonstrate stability of our method to hyperparameters and present an ablation study.
Revised: R-network can successfully generalize between environments, demonstrate stability of our method to hyperparameters and present an ablation study.

Original: 4. S TATIC M AZE G OAL R EACHING .
Revised: 4. S TATIC M AZE G OAL R EACHING .

Original: The goal of this experiment is to verify our re-implementation of the baseline method is correct.
Revised: The goal of this experiment is to verify our re-implementation of the baseline method is correct.

Original: We use the MyWayHome task from VizDoom .
Revised: We use the MyWayHome task from VizDoom .

Original: The agent has to reach the goal in a static 3 D maze in the time limit of 525 4 -repeated steps (equivalent to 1 minute).
Revised: The agent has to reach the goal in a static 3 D maze in the time limit of 525 4 -repeated steps (equivalent to 1 minute).

Original: It only gets a reward of +1 when it reaches the goal (episode ends at that moment), the rest of the time the reward is zero.
Revised: It only gets a reward of +1 when it reaches the goal (episode ends at that moment), the rest of the time the reward is zero.

Original: The task has three sub-tasks (following the setup in (Pathak et al., 2017)): “Dense”, “Sparse” and “Very Sparse”.
Revised: The task has three sub-tasks (following the setup in (Pathak et al., 2017)): “Dense”, “Sparse” and “Very Sparse”.

Original: The layout of the maze is demonstrated in Figure 5(c).
Revised: The layout of the maze is demonstrated in Figure 5(c).

Original: The goal is always at the same room but the starting points are different in those sub-tasks.
Revised: The goal is always at the same room but the starting points are different in those sub-tasks.

Original: For the “Dense” subtask, the agent starts in one of the random locations in the maze, some of which are close to the goal.
Revised: For the “Dense” subtask, the agent starts in one of the random locations in the maze, some of which are close to the goal.

Original: In this sub-task, the reward is relatively dense (hence the name): the agent is likely to bump into the
Revised: In this sub-task, the reward is relatively dense (hence the name): the agent is likely to bump into the

Original: [Figure]
Revised: [Figure]

Original: goal by a short random walk.
Revised: goal by a short random walk.

Original: Thus, this is an easy task even for standard RL methods.
Revised: Thus, this is an easy task even for standard RL methods.

Original: The two sub-tasks are harder: the agent starts in a medium-distant room from the goal (“Sparse”) or in a very distant room (“Very Sparse”).
Revised: The two sub-tasks are harder: the agent starts in a medium-distant room from the goal (“Sparse”) or in a very distant room (“Very Sparse”).

Original: Those tasks are hard for standard RL algorithms because the probability of bumping into a rewarding state by a random walk is very low.
Revised: Those tasks are hard for standard RL algorithms because the probability of bumping into a rewarding state by a random walk is very low.

Original: The training curves are shown in Figure 6.
Revised: The training curves are shown in Figure 6.

Original: By analysing them, we draw a few conclusions.
Revised: By analysing them, we draw a few conclusions.

Original: First, our re-implementation of the ICM baseline is correct and the results are in line with those published in (Pathak et al., 2017).
Revised: First, our re-implementation of the ICM baseline is correct and the results are in line with those published in (Pathak et al., 2017).

Original: Second, our method works on-par with the ICM baseline in terms of ﬁnal performance, quickly reaching 100% success rate in all three sub-tasks.
Revised: Second, our method works on-par with the ICM baseline in terms of ﬁnal performance, quickly reaching 100% success rate in all three sub-tasks.

Original: Finally, in terms of convergence speed our algorithm is signiﬁcantly faster than the state-of-the-art method ICM — our method reaches 100% success rate at least 2 times faster.
Revised: Finally, in terms of convergence speed our algorithm is signiﬁcantly faster than the state-of-the-art method ICM — our method reaches 100% success rate at least 2 times faster.

Original: Note that to make the comparison of the training speed fair, we shift our training curves by the environment interaction budget used for training R-network.
Revised: Note that to make the comparison of the training speed fair, we shift our training curves by the environment interaction budget used for training R-network.

Original: 4. P ROCEDURALLY G ENERATED R ANDOM M AZE G OAL R EACHING .
Revised: 4. P ROCEDURALLY G ENERATED R ANDOM M AZE G OAL R EACHING .

Original: In this experiment we aim to evaluate maze goal reaching task generalization on a large scale.
Revised: In this experiment we aim to evaluate maze goal reaching task generalization on a large scale.

Original: We train on hundreds of levels and then test also on hundreds of hold-out levels.
Revised: We train on hundreds of levels and then test also on hundreds of hold-out levels.

Original: We use “Explore
Revised: We use “Explore

Original: Goal Locations Large” (we will denote it “Sparse”) and “Explore Obstructed Goals Large” (we will denote it “Sparse + Doors”) levels in the DMLab simulator.
Revised: Goal Locations Large” (we will denote it “Sparse”) and “Explore Obstructed Goals Large” (we will denote it “Sparse + Doors”) levels in the DMLab simulator.

Original: In those levels, the agent starts in a random location in a randomly generated maze (both layout and textures are randomized at the beginning of the episode).
Revised: In those levels, the agent starts in a random location in a randomly generated maze (both layout and textures are randomized at the beginning of the episode).

Original: Within the time limit of 1800 4 -repeated steps (equivalent to 2 minutes), the agent has to reach the goal as many times as possible.
Revised: Within the time limit of 1800 4 -repeated steps (equivalent to 2 minutes), the agent has to reach the goal as many times as possible.

Original: Every time it reaches a goal, it is respawned into another random location in the maze and has to go to the goal again.
Revised: Every time it reaches a goal, it is respawned into another random location in the maze and has to go to the goal again.

Original: Every time the goal is reached, the agent gets a reward +10 , the rest of the time the reward is zero.
Revised: Every time the goal is reached, the agent gets a reward +10 , the rest of the time the reward is zero.

Original: The second level is a variation of the ﬁrst one with doors which make the paths in the maze longer.
Revised: The second level is a variation of the ﬁrst one with doors which make the paths in the maze longer.

Original: The layouts of the levels are demonstrated in Figure 5(b,c).
Revised: The layouts of the levels are demonstrated in Figure 5(b,c).

Original: We found out that the standard task “Sparse” is actually relatively easy even for the plain PPO algorithm.
Revised: We found out that the standard task “Sparse” is actually relatively easy even for the plain PPO algorithm.

Original: The reason is that the agent starting point and the goal are sampled on the map independently of each other — and sometimes both happen to be in the same room which simpliﬁes the task.
Revised: The reason is that the agent starting point and the goal are sampled on the map independently of each other — and sometimes both happen to be in the same room which simpliﬁes the task.

Original: To test the limits of the algorithms, we create a gap between the starting point and the goal which eliminates same-room initialization.
Revised: To test the limits of the algorithms, we create a gap between the starting point and the goal which eliminates same-room initialization.

Original: We report the results for both the original task “Sparse” and its harder version “Very Sparse”.
Revised: We report the results for both the original task “Sparse” and its harder version “Very Sparse”.

Original: Thus, there are overall three tasks considered in this section: “Sparse”, “Very Sparse” and “Sparse + Doors”.
Revised: Thus, there are overall three tasks considered in this section: “Sparse”, “Very Sparse” and “Sparse + Doors”.

Original: The results demonstrate that our method can reasonably adapt to ever-changing layouts and textures — see Table 1 and training curves in Figure 7.
Revised: The results demonstrate that our method can reasonably adapt to ever-changing layouts and textures — see Table 1 and training curves in Figure 7.

Original: We outperform the baseline method ICM
Revised: We outperform the baseline method ICM

Original: in all three environments using the same environment interaction budget of 20 M 4-repeated steps.
Revised: in all three environments using the same environment interaction budget of 20 M 4-repeated steps.

Original: The environment “Sparse” is relatively easy and all methods work reasonably.
Revised: The environment “Sparse” is relatively easy and all methods work reasonably.

Original: In the “Very Sparse” and “Sparse + Doors” settings our advantage with respect to PPO and ICM is more clear.
Revised: In the “Very Sparse” and “Sparse + Doors” settings our advantage with respect to PPO and ICM is more clear.

Original: On those levels, the visual inspection of the ICM learnt behaviour reveals an important property of this method: it is confused by the ﬁring action and learns to entertain itself by ﬁring until it runs out of ammunition.
Revised: On those levels, the visual inspection of the ICM learnt behaviour reveals an important property of this method: it is confused by the ﬁring action and learns to entertain itself by ﬁring until it runs out of ammunition.

Original: A similar ﬁnding was reported in a concurrent work (Burda et al., 2018): the agent was given an action which switched the content on a TV screen in a maze, along with the movement actions.
Revised: A similar ﬁnding was reported in a concurrent work (Burda et al., 2018a): the agent was given an action which switched the content on a TV screen in a maze, along with the movement actions.

Original: Instead of moving, the agent learns to switch channels forever.
Revised: Instead of moving, the agent learns to switch channels forever.

Original: While one might intuitively accept such “couchpotato” behaviour in intelligent creatures, it does not need to be a consequence of curious behaviour.
Revised: While one might intuitively accept such “couch-potato” behaviour in intelligent creatures, it does not need to be a consequence of curious behaviour.

Original: In particular, we are not observing such dramatic ﬁring behaviour for ourcuriosity formulation:
Revised: In particular, we are not observing such dramatic ﬁring behaviour for our

Original: [Figure]
Revised: [Figure]

Original:  according to Figure 1, an observation after ﬁring is still one step away from the one before ﬁring, so it is not novel (note that ﬁring still could happen in practice because of the entropy term in PPO).
Revised: curiosity formulation: according to Figure 1, an observation after ﬁring is still one step away from the one before ﬁring, so it is not novel (note that ﬁring still could happen in practice because of the entropy term in PPO).

Original: Thus, our formulation turns out to be more robust than ICM’s prediction error in this scenario.
Revised: Thus, our formulation turns out to be more robust than ICM’s prediction error in this scenario.

Original: Note that we do not speciﬁcally look for an action set which breaks the baseline — just use the standard one for DMLab , in line with the prior work (e.g., (Espeholt et al., 2018)).
Revised: Note that we do not speciﬁcally look for an action set which breaks the baseline — just use the standard one for DMLab , in line with the prior work (e.g., (Espeholt et al., 2018)).

Original: The result of this experiment suggests to look more into how methods behave in extremely-sparse reward scenarios.
Revised: The result of this experiment suggests to look more into how methods behave in extremely-sparse reward scenarios.

Original: The limiting case would be no reward at all — we consider it in the next section.
Revised: The limiting case would be no reward at all — we consider it in the next section.

Original: 4. N O R EWARD /A REA C OVERAGE .
Revised: 4. N O R EWARD /A REA C OVERAGE .

Original: This experiment aims to quantitatively establish how good our method is in the scenario when no task reward is given.
Revised: This experiment aims to quantitatively establish how good our method is in the scenario when no task reward is given.

Original: One might question why this scenario is interesting — however, before the task reward is found for the ﬁrst time, the agent lives in the no-reward world.
Revised: One might question why this scenario is interesting — however, before the task reward is found for the ﬁrst time, the agent lives in the no-reward world.

Original: How it behaves in this case will also determine how likely it is to stumble into the task reward in the ﬁrst place.
Revised: How it behaves in this case will also determine how likely it is to stumble into the task reward in the ﬁrst place.

Original: We use one of the DMLab levels — “Sparse” from the previous experiment.
Revised: We use one of the DMLab levels — “Sparse” from the previous experiment.

Original: We modify the task to eliminate the reward and name the new task “No Reward”.
Revised: We modify the task to eliminate the reward and name the new task “No Reward”.

Original: To quantify success in this task, we report the reward coming from Grid Oracle for all compared methods.
Revised: To quantify success in this task, we report the reward coming from Grid Oracle for all compared methods.

Original: This reward provides a discrete approximation to the area covered by the agent while exploring.
Revised: This reward provides a discrete approximation to the area covered by the agent while exploring.

Original: The training curves are shown in Figure 7 and the ﬁnal test results in Table 1.
Revised: The training curves are shown in Figure 7 and the ﬁnal test results in Table 1.

Original: The result of this experiment is that our method and Grid Oracle both work, while the ICM baseline is not working — and the qualitative difference in behaviour is bigger than in the previous experiments.
Revised: The result of this experiment is that our method and Grid Oracle both work, while the ICM baseline is not working — and the qualitative difference in behaviour is bigger than in the previous experiments.

Original: As can be seen from the training curves, after a temporary increase, ICM quality actually decreases over time, rendering a sharp disagreement between the prediction-error-based bonus and the area coverage metric.
Revised: As can be seen from the training curves, after a temporary increase, ICM quality actually decreases over time, rendering a sharp disagreement between the prediction-error-based bonus and the area coverage metric.

Original: By looking at the video 4 , we observe that the ﬁring behaviour of ICM becomes even more prominent, while our method still shows reasonable exploration.
Revised: By looking at the video 4 , we observe that the ﬁring behaviour of ICM becomes even more prominent, while our method still shows reasonable exploration.

Original: Finally, we try to ﬁnd out if the ICM baseline behaviour above is due to the ﬁring action only.
Revised: Finally, we try to ﬁnd out if the ICM baseline behaviour above is due to the ﬁring action only.

Original: Could it learn exploration of randomized mazes if the Fire action is excluded from the actions set?
Revised: Could it learn exploration of randomized mazes if the Fire action is excluded from the actions set?

Original: For that purpose, we create a new version of the task — we call it “No Reward - Fire”.
Revised: For that purpose, we create a new version of the task — we call it “No Reward - Fire”.

Original: This task demonstrates qualitatively similar results to the one with the full action set — see Table 1.
Revised: This task demonstrates qualitatively similar results to the one with the full action set — see Table 1.

Original: By looking at the videos 4 , we hypothesise that the agent can most signiﬁcantly change its current view when it is close to the wall — thus increasing one-step prediction error — so it tends to get stuck near “interesting” diverse textures on the walls.
Revised: By looking at the videos 4 , we hypothesise that the agent can most signiﬁcantly change its current view when it is close to the wall — thus increasing one-step prediction error — so it tends to get stuck near “interesting” diverse textures on the walls.

Original: The results suggest that in an environment completely without reward, the ICM method will exhaust its curiosity very quickly — passing through a sharp peak and then degrading into undesired behaviour.
Revised: The results suggest that in an environment completely without reward, the ICM method will exhaust its curiosity very quickly — passing through a sharp peak and then degrading into undesired behaviour.

Original: This observation raises concerns: what if ICM passes the peak before it reaches the ﬁrst task reward in the cases of real tasks?
Revised: This observation raises concerns: what if ICM passes the peak before it reaches the ﬁrst task reward in the cases of real tasks?

Original: Supposedly, it would require careful tuning per-game.
Revised: Supposedly, it would require careful tuning per-game.

Original: Fur-
Revised: Fur-

Original: [Table]
Revised: [Table]

Original: thermore, in some cases
Revised: thermore, in some cases

Original: [Table]
Revised: [Table]

Original: it would take a lot of time with a good exploration behaviour to reach the ﬁrst reward
Revised: it would take a lot of time with a good exploration behaviour to reach the ﬁrst reward

Original: [Table]
Revised: [Table]

Original: which would require to stay at the top performance for longer — which is problematic for the ICM method but still possible for our method
Revised: which would require to stay at the top performance for longer — which is problematic for the ICM method but still possible for our method

Original: [Table]
Revised: [Table]

Original: 4. D ENSE R EWARD T ASKS .
Revised: 4. D ENSE R EWARD T ASKS .

Original: A desirable property of a good curiosity bonus is to avoid hurting performance in dense-reward tasks (in addition to improving performance for sparse-reward tasks).
Revised: A desirable property of a good curiosity bonus is to avoid hurting performance in dense-reward tasks (in addition to improving performance for sparse-reward tasks).

Original: We test this scenario in two levels in the DMLab simulator: “Rooms Keys Doors Puzzle” (which we denote “Dense 1”) and “Rooms
Revised: We test this scenario in two levels in the DMLab simulator: “Rooms Keys Doors Puzzle” (which we denote “Dense 1”) and “Rooms

Original: Collect Good Objects Train” (which we denote “Dense 2”).
Revised: Collect Good Objects Train” (which we denote “Dense 2”).

Original: In the ﬁrst task, the agent has to collect keys and reach the goal object behind a few doors openable by those keys.
Revised: In the ﬁrst task, the agent has to collect keys and reach the goal object behind a few doors openable by those keys.

Original: The rewards in this task are rather dense (key collection/door opening is rewarded).
Revised: The rewards in this task are rather dense (key collection/door opening is rewarded).

Original: In the second task the agent has to collect good objects (give positive reward) and avoid bad objects (give negative reward).
Revised: In the second task the agent has to collect good objects (give positive reward) and avoid bad objects (give negative reward).

Original: The episode lasts for 900 4 -repeated steps (equivalent to 1 minute) in both tasks.
Revised: The episode lasts for 900 4 -repeated steps (equivalent to 1 minute) in both tasks.

Original: The results show that our method indeed does not signiﬁcantly deteriorate performance of plain PPO in those dense-reward tasks — see Table 1.
Revised: The results show that our method indeed does not signiﬁcantly deteriorate performance of plain PPO in those dense-reward tasks — see Table 1.

Original: The training curves for “Dense 1” are shown in Figureand for “Dense 2” — in the supplementary material.
Revised: The training curves for “Dense 1” are shown in Figureand for “Dense 2” — in the supplementary material.

Original: Note that we use the same bonus weight in this task as in other DMLab tasks before.
Revised: Note that we use the same bonus weight in this task as in other DMLab tasks before.

Original: All methods work similarly besides the Grid Oracle in the “Dense 2” task — which performs slightly worse.
Revised: All methods work similarly besides the Grid Oracle in the “Dense 2” task — which performs slightly worse.

Original: Video inspection 4 reveals that Grid Oracle — the only method which has ground-truth knowledge about area it covers during training — sometimes runs around excessively and occasionally fails to collect all good objects.
Revised: Video inspection 4 reveals that Grid Oracle — the only method which has ground-truth knowledge about area it covers during training — sometimes runs around excessively and occasionally fails to collect all good objects.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Our method is at the intersection of multiple topics: curiosity, episodic memory and temporal distance prediction.
Revised: Our method is at the intersection of multiple topics: curiosity, episodic memory and temporal distance prediction.

Original: In the following, we discuss the relation to the prior work on those topics.
Revised: In the following, we discuss the relation to the prior work on those topics.

Original: Curiosity in visually rich 3D environments.
Revised: Curiosity in visually rich 3D environments.

Original: Recently, a few works demonstrated the possibility to learn exploration behaviour in visually rich 3D environments like DMLab (Beattie et al., 2016) and VizDoom (Kempka et al., 2016).
Revised: Recently, a few works demonstrated the possibility to learn exploration behaviour in visually rich 3D environments like DMLab (Beattie et al., 2016) and VizDoom (Kempka et al., 2016).

Original: (Pathak et al., 2017) trains a predictor for the embedding of the next observation and if the reality is signiﬁcantly different from the prediction — rewards the agent.
Revised: (Pathak et al., 2017) trains a predictor for the embedding of the next observation and if the reality is signiﬁcantly different from the prediction — rewards the agent.

Original: In that work, the embedding is trained with the purpose to be a good embedding for predicting action taken between observations — unlike an earlier work (Stadie et al., 2015) which obtains an embedding from an autoencoder.
Revised: In that work, the embedding is trained with the purpose to be a good embedding for predicting action taken between observations — unlike an earlier work (Stadie et al., 2015) which obtains an embedding from an autoencoder.

Original: It was later shown by (Burda et al., 2018) that the perceptive prediction approach has a downside — the agent could become a “couch-potato” if given an action to switch TV channels.
Revised: It was later shown by (Burda et al., 2018a) that the perceptive prediction approach has a downside — the agent could become a “couch-potato” if given an action to switch TV channels.

Original: This observation is conﬁrmed in our experiments by observing a persistent ﬁring behaviour of the ICM baseline in the navigational tasks with very sparse or no reward.
Revised: This observation is conﬁrmed in our experiments by observing a persistent ﬁring behaviour of the ICM baseline in the navigational tasks with very sparse or no reward.

Original: By contrast, our method does not show this behaviour.
Revised: By contrast, our method does not show this behaviour.

Original: Another work (Fu et al., 2017) trains a temporal distance predictor and then uses this predictor to establish novelty: if the observation is easy to classify versus previous observations, it is novel.
Revised: Another work (Fu et al., 2017) trains a temporal distance predictor and then uses this predictor to establish novelty: if the observation is easy to classify versus previous observations, it is novel.

Original: This method does not use episodic memory, however, and the predictor is used in way which is different from our work.
Revised: This method does not use episodic memory, however, and the predictor is used in way which is different from our work.

Original: General curiosity.
Revised: General curiosity.

Original: Curiosity-based exploration for RL has been extensively studied in the literature.
Revised: Curiosity-based exploration for RL has been extensively studied in the literature.

Original: For an overview, we refer the reader to the works (Oudeyer & Kaplan, 2009; Oudeyer et al., 2007).
Revised: For an overview, we refer the reader to the works (Oudeyer & Kaplan, 2009; Oudeyer et al., 2007).

Original: The most common practical approaches could be divided into three branches: predictionerror-based, count-based and goal-generation-based.
Revised: The most common practical approaches could be divided into three branches: predictionerror-based, count-based and goal-generation-based.

Original: Since the prediction-based approaches were discussed before, in the following we focus on the latter two branches.
Revised: Since the prediction-based approaches were discussed before, in the following we focus on the latter two branches.

Original: The count-based approach suggests to keep visit counts for observations and concentrate on visiting states which has been rarely visited before — which bears distant similarity to how we use episodic memory.
Revised: The count-based approach suggests to keep visit counts for observations and concentrate on visiting states which has been rarely visited before — which bears distant similarity to how we use episodic memory.

Original: This idea is natural for discrete observation spaces and has solid theoretical foundations.
Revised: This idea is natural for discrete observation spaces and has solid theoretical foundations.

Original: Its extension to continuous observation spaces is non-trivial, however.
Revised: Its extension to continuous observation spaces is non-trivial, however.

Original: The notable step in this direction was taken by works (Bellemare et al., 2016; Ostrovski et al., 2017) which introduce a trained observation density model which is later converted to a function behaving similarly to counts.
Revised: The notable step in this direction was taken by works (Bellemare et al., 2016; Ostrovski et al., 2017) which introduce a trained observation density model which is later converted to a function behaving similarly to counts.

Original: The way conversion is done has some similarity to prediction-error-based approaches: it is the difference of the density in the example before and after training of this example which is converted to count.
Revised: The way conversion is done has some similarity to prediction-error-based approaches: it is the difference of the density in the example before and after training of this example which is converted to count.

Original: The experiments in the original works operate on Atari games (Bellemare et al., 2013) and were not benchmarked on visually rich 3 D environments.
Revised: The experiments in the original works operate on Atari games (Bellemare et al., 2013) and were not benchmarked on visually rich 3 D environments.

Original: Another approach (Tang et al., 2017) discretises the continuous observation space by hashing and then uses the count-based approach in this discretised space.
Revised: Another approach (Tang et al., 2017) discretises the continuous observation space by hashing and then uses the count-based approach in this discretised space.

Original: This method is appealing in its simplicity, however, the experiments in (Pathak et al., 2017; Fu et al., 2017) show that it does not perform well in visually rich 3 D environments.
Revised: This method is appealing in its simplicity, however, the experiments in (Pathak et al., 2017; Fu et al., 2017) show that it does not perform well in visually rich 3 D environments.

Original: Another line of work, Novelty Search (Lehman & Stanley, 2011) and its recent follow-up (Conti et al., 2018), proposed maintaining an archive of behaviours and comparing current behaviour to those — however, the comparison is done by euclidean distance and behaviours are encoded using coordinates, while we learn the comparison function and only use pixels.
Revised: Another line of work, Novelty Search (Lehman & Stanley, 2011) and its recent follow-up (Conti et al., 2018), proposed maintaining an archive of behaviours and comparing current behaviour to those — however, the comparison is done by euclidean distance and behaviours are encoded using coordinates, while we learn the comparison function and only use pixels.

Original: Finally, our concept of novelty through reachability is reminiscent of generating the goals which are reachable but not too easy — a well-studied topic in the prior work.
Revised: Finally, our concept of novelty through reachability is reminiscent of generating the goals which are reachable but not too easy — a well-studied topic in the prior work.

Original: The work (Held et al., 2017) uses a GAN to differentiate what is easy to reach from what is not and then generate goals at the boundary.
Revised: The work (Held et al., 2017) uses a GAN to differentiate what is easy to reach from what is not and then generate goals at the boundary.

Original: Another work (Baranes & Oudeyer, 2013) deﬁnes new goals according to the expected progress the agent will make if it learns to solve the associated task.
Revised: Another work (Baranes & Oudeyer, 2013) deﬁnes new goals according to the expected progress the agent will make if it learns to solve the associated task.

Original: The recent work (P ´ er ´ e et al., 2018) learns an embedding for the goal space and then samples increasingly difﬁcult goals from that space.
Revised: The recent work (P ´ er ´ e et al., 2018) learns an embedding for the goal space and then samples increasingly difﬁcult goals from that space.

Original: In a spirit similar to those works, our method implicitly deﬁnes goals that are at least some ﬁxed number of steps away by using the reachability network.
Revised: In a spirit similar to those works, our method implicitly deﬁnes goals that are at least some ﬁxed number of steps away by using the reachability network.

Original: However, our method is easier to implement than other goal-generation methods and quite general.
Revised: However, our method is easier to implement than other goal-generation methods and quite general.

Original: Episodic memory.
Revised: Episodic memory.

Original: Two recent works (Blundell et al., 2016; Pritzel et al., 2017) were inspired by the ideas of episodic memory in animals and proposed an approach to learn the functioning of episodic memory along with the task for which this memory is applied.
Revised: Two recent works (Blundell et al., 2016; Pritzel et al., 2017) were inspired by the ideas of episodic memory in animals and proposed an approach to learn the functioning of episodic memory along with the task for which this memory is applied.

Original: Those works are more focused on repeating successful strategies than on exploring environments — and are not designed to work in the absence of task rewards.
Revised: Those works are more focused on repeating successful strategies than on exploring environments — and are not designed to work in the absence of task rewards.

Original: Temporal distance prediction.
Revised: Temporal distance prediction.

Original: The idea to predict the distance between video frames has been studied extensively.
Revised: The idea to predict the distance between video frames has been studied extensively.

Original: Usually this prediction is an auxiliary task for solving another problem.
Revised: Usually this prediction is an auxiliary task for solving another problem.

Original: (Sermanet et al., 2017) trains an embedding such that closer in time frames are also closer in the embedding space.
Revised: (Sermanet et al., 2017) trains an embedding such that closer in time frames are also closer in the embedding space.

Original: Multiple works (Fu et al., 2017; Savinov et al., 2018; Aytar et al., 2018) train a binary classiﬁer for predicting if the distance in time between frames is within a certain threshold or not.
Revised: Multiple works (Fu et al., 2017; Savinov et al., 2018; Aytar et al., 2018) train a binary classiﬁer for predicting if the distance in time between frames is within a certain threshold or not.

Original: While (Sermanet et al., 2017; Aytar et al., 2018) use only the embedding for their algorithms, (Fu et al., 2017;
Revised: While (Sermanet et al., 2017; Aytar et al., 2018) use only the embedding for their algorithms, (Fu et al., 2017;

Original: Savinov et al., 2018) also use the classiﬁer trained together with the embedding.
Revised: Savinov et al., 2018) also use the classiﬁer trained together with the embedding.

Original: As mentioned earlier, (Fu et al., 2017) uses this classiﬁer for density estimation instead of comparison to episodic memory.
Revised: As mentioned earlier, (Fu et al., 2017) uses this classiﬁer for density estimation instead of comparison to episodic memory.

Original: (Savinov et al., 2018) does compare to the episodic memory buffer but solves a different task — given an already provided exploration video, navigate to a goal — which is complementary to the task in our work.
Revised: (Savinov et al., 2018) does compare to the episodic memory buffer but solves a different task — given an already provided exploration video, navigate to a goal — which is complementary to the task in our work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we propose a new model of curiosity based on episodic memory and the ideas of reachability.
Revised: In this work we propose a new model of curiosity based on episodic memory and the ideas of reachability.

Original: This allows us to overcome the known “couch-potato” issues of prior work and outperform the previous curiosity state-of-the-art method ICM in visually rich 3 D environments from VizDoom and DMLab .
Revised: This allows us to overcome the known “couch-potato” issues of prior work and outperform the previous curiosity state-of-the-art method ICM in visually rich 3 D environments from VizDoom and DMLab .

Original: Our method also allows a MuJoCo ant to learn locomotion purely out of ﬁrst-personview curiosity.
Revised: Our method also allows a MuJoCo ant to learn locomotion purely out of ﬁrst-personview curiosity.

Original: In the future, we want to make policy aware of memory not only in terms of receiving reward, but also in terms of acting.
Revised: In the future, we want to make policy aware of memory not only in terms of receiving reward, but also in terms of acting.

Original: Can we use memory content retrieved based on reachability to guide exploration behaviour in the test time?
Revised: Can we use memory content retrieved based on reachability to guide exploration behaviour in the test time?

Original: This could open opportunities to learn exploration in new tasks in a few-shot style — which is currently a big scientiﬁc challenge.
Revised: This could open opportunities to learn exploration in new tasks in a few-shot style — which is currently a big scientiﬁc challenge.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We would like to thank Olivier Pietquin, Alexey Dosovitskiy, Vladlen Koltun, Carlos Riquelme,
Revised: We would like to thank Olivier Pietquin, Alexey Dosovitskiy, Vladlen Koltun, Carlos Riquelme,

Original: Charles Blundell and Sergey Levine for the valuable discussions about our work.
Revised: Charles Blundell and Sergey Levine for the valuable discussions about our work.

Original: S UPPLEMENTARY M ATERIAL
Revised: S UPPLEMENTARY M ATERIAL

Original: The supplementary material is organized as follows.
Revised: The supplementary material is organized as follows.

Original: First, we describe the MuJoCo locomotion experiments.
Revised: First, we describe the MuJoCo locomotion experiments.

Original: Then we provide training details for R-network.
Revised: Then we provide training details for R-network.

Original: After that, we list hyperparameter values and the details of hyperparameter search for all methods.
Revised: After that, we list hyperparameter values and the details of hyperparameter search for all methods.

Original: Then we show experimental results which suggest that R-network can generalize between environments: we transfer one general Rnetwork from all available DMLab 30 levels to our tasks of interest and also transfer R-networks between single environments.
Revised: Then we show experimental results which suggest that R-network can generalize between environments: we transfer one general Rnetwork from all available DMLab 30 levels to our tasks of interest and also transfer R-networks between single environments.

Original: After that, we present the results from a stability/ablation study which suggests our method is stable with respect to its most important hyperparameters and the components we used in the method are actually necessary for its performance (and measure their inﬂuence).
Revised: After that, we present the results from a stability/ablation study which suggests our method is stable with respect to its most important hyperparameters and the components we used in the method are actually necessary for its performance (and measure their inﬂuence).

Original: Then we demonstrate the robustness of our method to the environments where every state has a stochastic next state.
Revised: Then we demonstrate the robustness of our method to the environments where every state has a stochastic next state.

Original: After that, we discuss computational considerations for our method.
Revised: After that, we discuss computational considerations for our method.

Original: Finally, we provide the training curves for the “Dense 2” task in the main text.
Revised: Finally, we provide the training curves for the “Dense 2” task in the main text.

Original: S MuJoCo A NT L OCOMOTION OUT OF F IRST - PERSON - VIEW C URIOSITY
Revised: S MuJoCo A NT L OCOMOTION OUT OF F IRST - PERSON - VIEW C URIOSITY

Original: Equipped with our curiosity module, a MuJoCo ant has learned 5 to move out of curiosity based on the ﬁrst-person view 6 .
Revised: Equipped with our curiosity module, a MuJoCo ant has learned 5 to move out of curiosity based on the ﬁrst-person view 6 .

Original: First, let us describe the setup:
Revised: First, let us describe the setup:

Original: • Environment: the standard MuJoCo environment is a plane with a uniform or repetitive texture on it — nothing to be visually curious about.
Revised: • Environment: the standard MuJoCo environment is a plane with a uniform or repetitive texture on it — nothing to be visually curious about.

Original: To ﬁx that, we tiled the 400 ×ﬂoor into squares of size 4 × 4 .
Revised: To ﬁx that, we tiled the 400 ×ﬂoor into squares of size 4 × 4 .

Original: Each tile is assigned a random texture from a set oftextures at the beginning of every episode.
Revised: Each tile is assigned a random texture from a set oftextures at the beginning of every episode.

Original: The ant is initialized at a random location in the 200 × 200 central square of the ﬂoor.
Revised: The ant is initialized at a random location in the 200 × 200 central square of the ﬂoor.

Original: The episode lasts for 1000 steps (no action repeat is used).
Revised: The episode lasts for 1000 steps (no action repeat is used).

Original: If the z -coordinate of the center of mass of the ant is above 1 . 0 or below 0 .
Revised: If the z -coordinate of the center of mass of the ant is above 1 . 0 or below 0 .

Original: 2 — the episode ends prematurely (standard termination condition).
Revised: 2 — the episode ends prematurely (standard termination condition).

Original: • Observation space: for computing the curiosity reward, we only use a ﬁrst-person view camera mounted on the ant (that way we can use the same architecture of our curiosity module as in VizDoom and DMLab ).
Revised: • Observation space: for computing the curiosity reward, we only use a ﬁrst-person view camera mounted on the ant (that way we can use the same architecture of our curiosity module as in VizDoom and DMLab ).

Original: For policy, we use the standard body features from Ant-v2 in gym-mujoco 7 (joint angles, velocities, etc.).
Revised: For policy, we use the standard body features from Ant-v2 in gym-mujoco 7 (joint angles, velocities, etc.).

Original: • Action space: standard continuous space from Ant-v2 in gym-mujoco.
Revised: • Action space: standard continuous space from Ant-v2 in gym-mujoco.

Original: • Basic RL solver: PPO (same as in the main text of the paper).
Revised: • Basic RL solver: PPO (same as in the main text of the paper).

Original: • Baselines: PPO on task reward, PPO on task reward plus constant reward 1 at every step as a trivial curiosity bonus (which we denote PPO+1, it optimizes for longer survival).
Revised: • Baselines: PPO on task reward, PPO on task reward plus constant reward 1 at every step as a trivial curiosity bonus (which we denote PPO+1, it optimizes for longer survival).

Original: Second, we present quantitative results for the setting with no task reward after 10 M training steps in Table S1 (the ﬁrst row).
Revised: Second, we present quantitative results for the setting with no task reward after 10 M training steps in Table S1 (the ﬁrst row).

Original: Our method outperforms the baselines.
Revised: Our method outperforms the baselines.

Original: As seen in the videos 8 , PPO (random policy) dies quickly, PPO+1 survives for longer but does not move much and our method moves around the environment.
Revised: As seen in the videos 8 , PPO (random policy) dies quickly, PPO+1 survives for longer but does not move much and our method moves around the environment.

Original: Additionally, we performed an experiment with an extremely sparse task reward — which we call “Escape Circle”.
Revised: Additionally, we performed an experiment with an extremely sparse task reward — which we call “Escape Circle”.

Original: The reward is given as follows: 0 reward inside the circle of radius 10 , and starting from 10 , we give a one-time reward of 1 every time an agent goes through a concentric circle of radius 10 + 0 .
Revised: The reward is given as follows: 0 reward inside the circle of radius 10 , and starting from 10 , we give a one-time reward of 1 every time an agent goes through a concentric circle of radius 10 + 0 .

Original: 5 k (for integer k ≥ 0 ).
Revised: 5 k (for integer k ≥ 0 ).

Original: The results at 10 M training steps are shown in Table S1 (the second row).
Revised: The results at 10 M training steps are shown in Table S1 (the second row).

Original: Our method signiﬁcantly outperforms the baselines (better than the best baseline by a factor of 10 ).
Revised: Our method signiﬁcantly outperforms the baselines (better than the best baseline by a factor of 10 ).

Original: Finally, let us discuss the relation to some other works in the ﬁeld of learning locomotion from intrinsic reward.
Revised: Finally, let us discuss the relation to some other works in the ﬁeld of learning locomotion from intrinsic reward.

Original: The closest work in terms of task setup is the concurrent work (Burda et al., 2018).
Revised: The closest work in terms of task setup is the concurrent work (Burda et al., 2018a).

Original: The authors demonstrate slow motion 9 of the ant learned from pixel-based curiosity only.
Revised: The authors demonstrate slow motion 9 of the ant learned from pixel-based curiosity only.

Original: Other works use state features (joint angles, velocities, etc.) for formulating intrinsic reward, not pixels — which is a different setup.
Revised: Other works use state features (joint angles, velocities, etc.) for formulating intrinsic reward, not pixels — which is a different setup.

Original: One work in this direction is the concurrent work (Eysenbach et al., 2018) — which also contains a good overview of the literature on intrinsic reward from state features.
Revised: One work in this direction is the concurrent work (Eysenbach et al., 2018) — which also contains a good overview of the literature on intrinsic reward from state features.

Original: [Table]
Revised: [Table]

Original: S R EACHABILITY N ETWORK T RAINING D ETAILS
Revised: S R EACHABILITY N ETWORK T RAINING D ETAILS

Original: For training R-network, we use mini-batches of 64 observation pairs (matched within episodes).
Revised: For training R-network, we use mini-batches of 64 observation pairs (matched within episodes).

Original: The training is run for 50 K mini-batch iterations for VizDoom and 200 K mini-batch iterations for
Revised: The training is run for 50 K mini-batch iterations for VizDoom and 200 K mini-batch iterations for

Original: DMLab .
Revised: DMLab .

Original: At the beginning of every pass through the buffer, we re-shufﬂe it.
Revised: At the beginning of every pass through the buffer, we re-shufﬂe it.

Original: We use Adam optimizer with learning rate 10 − 4 .
Revised: We use Adam optimizer with learning rate 10 − 4 .

Original: The R-network uses a siamese architecture with two branches (see Figurein the main text), each branch is Resnet- 18 with 512 outputs, with a fully-connected network applied to the concatenated output of the branches.
Revised: The R-network uses a siamese architecture with two branches (see Figurein the main text), each branch is Resnet- 18 with 512 outputs, with a fully-connected network applied to the concatenated output of the branches.

Original: The fully-connected network has four hidden layers with 512 units, batch-normalization and ReLU is applied after each layer besides the last one, which is a softmax layer.
Revised: The fully-connected network has four hidden layers with 512 units, batch-normalization and ReLU is applied after each layer besides the last one, which is a softmax layer.

Original: Observations are RGB-images with resolution 160 × 120 pixels.
Revised: Observations are RGB-images with resolution 160 × 120 pixels.

Original: For online training of the R-network, we collect the experience and perform training every 720 K 4repeated environment steps.
Revised: For online training of the R-network, we collect the experience and perform training every 720 K 4repeated environment steps.

Original: Every time the experience is collected, we make 10 epochs of training on this experience.
Revised: Every time the experience is collected, we make 10 epochs of training on this experience.

Original: Before every epoch, the data is shufﬂed.
Revised: Before every epoch, the data is shufﬂed.

Original: S H YPERPARAMETERS
Revised: [Equation]

Original: The hyperparameters of different methods are given in Table S2 for VizDoom environment, in Table S3 for DMLab environment, and in Tables S4, S5 for MuJoCo Ant environment.
Revised: The hyperparameters of different methods are given in Table S2 for VizDoom environment, in Table S3 for DMLab environment, and in Tables S4, S5 for MuJoCo Ant environment.

Original: The hyperparameters for DMLab are tuned on the “Sparse” environment for all methods — because all methods work reasonably on this environment (it is unfair to tune a method on an environment where it fails and also unfair to tune different methods on different environments).
Revised: The hyperparameters for DMLab are tuned on the “Sparse” environment for all methods — because all methods work reasonably on this environment (it is unfair to tune a method on an environment where it fails and also unfair to tune different methods on different environments).

Original: We use the PPO algorithm from the open-source implementation 10 .
Revised: We use the PPO algorithm from the open-source implementation 10 .

Original: For implementation convenience, we scale both the bonus and the task reward (with a single balancing coefﬁcient it would not be possible to turn off one of those rewards).
Revised: For implementation convenience, we scale both the bonus and the task reward (with a single balancing coefﬁcient it would not be possible to turn off one of those rewards).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 21e
Revised: 21e

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 78e
Revised: 78e

Original: [Table]
Revised: [Table]

Original: Task reward scale
Revised: Task reward scale

Original: Curiosity bonus scale α
Revised: Curiosity bonus scale α

Original: [Table]
Revised: [Table]

Original: EC memory size - -
Revised: EC memory size - -

Original: EC reward shift β - -
Revised: EC reward shift β - -

Original: EC novelty threshold b novelty - -
Revised: EC novelty threshold b novelty - -

Original: EC aggregation function F - - 10th largest
Revised: EC aggregation function F - - 10th largest

Original: S R- NETWORK GENERALIZATION STUDY
Revised: S R- NETWORK GENERALIZATION STUDY

Original: One of the promises of our approach is its potential ability to generalize between tasks.
Revised: One of the promises of our approach is its potential ability to generalize between tasks.

Original: In this section we verify if this promise holds.
Revised: In this section we verify if this promise holds.

Original: S4. T RAINING R- NETWORK ON ALL DMLab - 30 TASKS
Revised: S4. T RAINING R- NETWORK ON ALL DMLab - 30 TASKS

Original: Could we train a universal R-network for all available levels — and then use this network for all our tasks of interest?
Revised: Could we train a universal R-network for all available levels — and then use this network for all our tasks of interest?

Original: Since different games have different dynamics models, the notion of closely reachable or far observations also changes from game to game.
Revised: Since different games have different dynamics models, the notion of closely reachable or far observations also changes from game to game.

Original: Can R-network successfully handle this variability?
Revised: Can R-network successfully handle this variability?

Original: Table S6 suggests that using a universal R-network slightly hurts the performance compared to using a specialized R-network trained speciﬁcally for the task.
Revised: Table S6 suggests that using a universal R-network slightly hurts the performance compared to using a specialized R-network trained speciﬁcally for the task.

Original: However, it still deﬁnitely helps to get higher reward compared to using the plain PPO.
Revised: However, it still deﬁnitely helps to get higher reward compared to using the plain PPO.

Original: The R-network is trained using 10M environment interactions equally split across all 30 DMLab - 30 tasks.
Revised: The R-network is trained using 10M environment interactions equally split across all 30 DMLab - 30 tasks.

Original: [Table]
Revised: [Table]

Original: S4.
Revised: S4.

Original: [Table]
Revised: [Table]

Original: T RAINING R- NETWORK ON O NE L EVEL AND T ESTING ON A NOTHER
Revised: T RAINING R- NETWORK ON O NE L EVEL AND T ESTING ON A NOTHER

Original: This experiment is similar to the previous one but in a sense is more extreme.
Revised: This experiment is similar to the previous one but in a sense is more extreme.

Original: Instead of training on all levels (including the levels of interest and other unrelated levels), can we train R-network on just one task and use if for a different task?
Revised: Instead of training on all levels (including the levels of interest and other unrelated levels), can we train R-network on just one task and use if for a different task?

Original: Table S7 suggests we can obtain reasonable performance by transferring the R-network between similar enough environments.
Revised: Table S7 suggests we can obtain reasonable performance by transferring the R-network between similar enough environments.

Original: The performance is unsatisfactory only in one case (using the R-network trained on “Dense 2”).
Revised: The performance is unsatisfactory only in one case (using the R-network trained on “Dense 2”).

Original: Our hypothesis is that the characteristics of the environments are sufﬁciently different in that case: single room versus maze, static textures on the walls versus changing textures.
Revised: Our hypothesis is that the characteristics of the environments are sufﬁciently different in that case: single room versus maze, static textures on the walls versus changing textures.

Original: [Table]
Revised: [Table]

Original: S S TABILITY / ABLATION S TUDY
Revised: S S TABILITY / ABLATION S TUDY

Original: The experiments are done both in “No Reward” and “Very Sparse” environments.
Revised: The experiments are done both in “No Reward” and “Very Sparse” environments.

Original: The “No Reward” environment is useful to avoid the situations where task reward would hide important behavioural differences between different ﬂavors of our method (this “hiding” effect can be easily observed for different methods comparison in the dense reward tasks — but the inﬂuence of task reward still remains even in sparser cases).
Revised: The “No Reward” environment is useful to avoid the situations where task reward would hide important behavioural differences between different ﬂavors of our method (this “hiding” effect can be easily observed for different methods comparison in the dense reward tasks — but the inﬂuence of task reward still remains even in sparser cases).

Original: As in the main text, for the “No Reward” task we report the Grid
Revised: As in the main text, for the “No Reward” task we report the Grid

Original: Oracle reward as a discrete approximation to the area covered by the agent trajectories.
Revised: Oracle reward as a discrete approximation to the area covered by the agent trajectories.

Original: S5. P OSITIVE E XAMPLE T HRESHOLD IN R- NETWORK T RAINING
Revised: S5. P OSITIVE E XAMPLE T HRESHOLD IN R- NETWORK T RAINING

Original: Training the R-network requires a threshold k to separate negative from positive pairs.
Revised: Training the R-network requires a threshold k to separate negative from positive pairs.

Original: The trained policy implicitly depends on this threshold.
Revised: The trained policy implicitly depends on this threshold.

Original: Ideally, the policy performance should not be too sensitive to this hyper-parameter.
Revised: Ideally, the policy performance should not be too sensitive to this hyper-parameter.

Original: We conduct a study where the threshold is varied from 2 to 10 actions (as in all experiments before, each action is repeated 4 times).
Revised: We conduct a study where the threshold is varied from 2 to 10 actions (as in all experiments before, each action is repeated 4 times).

Original: Table S8 shows that the EC performance is reasonably robust to the choice of this threshold.
Revised: Table S8 shows that the EC performance is reasonably robust to the choice of this threshold.

Original: [Table]
Revised: [Table]

Original: The EC-module relies on an explicit memory buffer to store the embeddings of past observations and deﬁne novelty.
Revised: The EC-module relies on an explicit memory buffer to store the embeddings of past observations and deﬁne novelty.

Original: One legitimate question is to study the impact of the size of this memory buffer on the performance of the EC-module.
Revised: One legitimate question is to study the impact of the size of this memory buffer on the performance of the EC-module.

Original: As observed in table S9, the memory size has little impact on the performance.
Revised: As observed in table S9, the memory size has little impact on the performance.

Original: [Table]
Revised: [Table]

Original: E NVIRONMENT I NTERACTION B UDGET FOR T RAINING R- NETWORK
Revised: E NVIRONMENT I NTERACTION B UDGET FOR T RAINING R- NETWORK

Original: The sample complexity of our EC method includes two parts: the sample complexity to train the Rnetwork and the sample complexity of the policy training.
Revised: The sample complexity of our EC method includes two parts: the sample complexity to train the Rnetwork and the sample complexity of the policy training.

Original: In the worst case – when the R-network does not generalize across environments – the R-network has to be trained for each environment and the total sample complexity is then the sum of the previous two sample complexities.
Revised: In the worst case – when the R-network does not generalize across environments – the R-network has to be trained for each environment and the total sample complexity is then the sum of the previous two sample complexities.

Original: It is then crucial to see how many steps are needed to train R-network such that it can capture the notion of reachability.
Revised: It is then crucial to see how many steps are needed to train R-network such that it can capture the notion of reachability.

Original: R-network trained using a number of environment steps as low as 1 M already gives good performance, see Table S10.
Revised: R-network trained using a number of environment steps as low as 1 M already gives good performance, see Table S10.

Original: [Table]
Revised: [Table]

Original: I MPORTANCE OF T RAINING D IFFERENT P ARTS OF R- NETWORK
Revised: I MPORTANCE OF T RAINING D IFFERENT P ARTS OF R- NETWORK

Original: The R-network is composed of an Embedding network and a Comparator network.
Revised: The R-network is composed of an Embedding network and a Comparator network.

Original: How important is each for the ﬁnal performance of our method?
Revised: How important is each for the ﬁnal performance of our method?

Original: To establish that, we conduct two experiments.
Revised: To establish that, we conduct two experiments.

Original: First, we ﬁx the Embedding network at the random initialization and train only the Comparator.
Revised: First, we ﬁx the Embedding network at the random initialization and train only the Comparator.

Original: Second, we substitute the Comparator network applied to embeddings e 1 , e 2 with the sigmoid function
Revised: Second, we substitute the Comparator network applied to embeddings e 1 , e 2 with the sigmoid function

Original: [Figure]
Revised: [Figure]

Original: σ ( e T 1 e 2 ) and train only the Embedding.
Revised: σ ( e T 1 e 2 ) and train only the Embedding.

Original: According to the results in Table S11, we get a reasonable performance with a random embedding: the results are still better than the plain PPO (but worse than with the complete R-network).
Revised: According to the results in Table S11, we get a reasonable performance with a random embedding: the results are still better than the plain PPO (but worse than with the complete R-network).

Original: However, without the Comparator the quality drops below the plain PPO.
Revised: However, without the Comparator the quality drops below the plain PPO.

Original: This experiment leads us to two conclusions.
Revised: This experiment leads us to two conclusions.

Original: First, training the Embedding network is desired but not necessary for our method to work.
Revised: First, training the Embedding network is desired but not necessary for our method to work.

Original: Second, using the Comparator is essential and cannot be omitted in the current setup.
Revised: Second, using the Comparator is essential and cannot be omitted in the current setup.

Original: Apparently, predicting reachability requires ﬁne-grained access to both embeddings at the same time — and a simple comparison function does not work.
Revised: Apparently, predicting reachability requires ﬁne-grained access to both embeddings at the same time — and a simple comparison function does not work.

Original: [Table]
Revised: [Table]

Original: S R ANDOMIZED ENVIRONMENTS
Revised: S R ANDOMIZED ENVIRONMENTS

Revised: In the main text of the paper we observed how the ﬁring action confused the surprise-based curiosity method ICM.

Revised: This was a manifestation of the hardness of future prediction performed by ICM.

Revised: Importantly, there could be more than one reason why future prediction is hard (as observed in the concurrent work (Burda et al., 2018b)): partial observability of the environment, insufﬁciently rich future prediction model or randomized transitions in the environment.

Revised: Since our own method EC relies on comparisons to the past instead of predictions of the future, one could expect it to be more robust to those factors (intuitively, comparison to the past is an easier problem).

Original: In this section  we analyze how different methods behave when all the states in the environment provide stochastic next state.
Revised: The goal of this section is to provide additional evidence for that. We are going to experiment with one source of future prediction errors which we have used in the thought experiment from the introduction: environment stochasticity. In particular, we analyze how different methods behave when all the states in the environment provide stochastic next state.

Original: For that, we create versions of the DMLab environments “Sparse” and “Very Sparse” with added strong source of stochasticity: randomized TV on the head-on display of the agent.
Revised: For that, we create versions of the DMLab environments “Sparse” and “Very Sparse” with added strong source of stochasticity: randomized TV on the head-on display of the agent.

Original: It is implemented as follows: the lower right quadrant of the agent’s ﬁrst person view is occupied with random images.
Revised: It is implemented as follows: the lower right quadrant of the agent’s ﬁrst person view is occupied with random images.

Original: We try a few settings:
Revised: We try a few settings:

Original: • “Image Action k ”: there are k images of animals retrieved from the internet, an agent has a special action which changes an image on the TV screen to a random one from this set.
Revised: • “Image Action k ”: there are k images of animals retrieved from the internet, an agent has a special action which changes an image on the TV screen to a random one from this set.

Original: An example is shown in Figure S1(a).
Revised: An example is shown in Figure S1(a).

Original: • “Noise”: at every step a different noise pattern is shown on the TV screen, independently from agent’s actions.
Revised: • “Noise”: at every step a different noise pattern is shown on the TV screen, independently from agent’s actions.

Original: The noise is sampled uniformly from [0 , 255] independently for each pixel.
Revised: The noise is sampled uniformly from [0 , 255] independently for each pixel.

Original: An example is shown in Figure S1(b).
Revised: An example is shown in Figure S1(b).

Original: • “Noise Action”: same as “Noise”, but the noise pattern only changes if the agent uses a special action.
Revised: • “Noise Action”: same as “Noise”, but the noise pattern only changes if the agent uses a special action.

Original: The results at 20 M 4-repeated environment steps are shown in Tables S12, S13.

Original: In almost all cases, the performance of all methods deteriorates because of any source of stochasticity.

Original: However, our method turns out to be reasonably robust to all sources of stochasticity and still outperforms the baselines in all settings.

Original: The videos 11 , 12 demonstrate that our method still explores the maze reasonably well.

Original: [Table]
Revised: [Table]

Original: method is given for reference — it uses privileged information unavailable to other methods.
Revised: method is given for reference — it uses privileged information unavailable to other methods.

Original: Results are averaged over 30 random seeds.
Revised: Results are averaged over 30 random seeds.

Original: No seed tuning is performed.
Revised: No seed tuning is performed.

Original: [Table]
Revised: [Table]

Revised: The results at 20 M 4-repeated environment steps are shown in Tables S12, S13.

Revised: In almost all cases, the performance of all methods deteriorates because of any source of stochasticity.

Revised: However, our method turns out to be reasonably robust to all sources of stochasticity and still outperforms the baselines in all settings.

Revised: The videos 11 , 12 demonstrate that our method still explores the maze reasonably well.

Original: S C OMPUTATIONAL C ONSIDERATIONS
Revised: S C OMPUTATIONAL C ONSIDERATIONS

Original: The most computationally intensive parts of our algorithm are the memory reachability queries.
Revised: The most computationally intensive parts of our algorithm are the memory reachability queries.

Original: Reachabilities to past memories are computed in parallel via mini-batching.
Revised: Reachabilities to past memories are computed in parallel via mini-batching.

Original: We have shown the algorithm to work reasonably fast with a memory size of 200 .
Revised: We have shown the algorithm to work reasonably fast with a memory size of 200 .

Original: For orders of magnitude larger memory sizes, one would need to better parallelize reachability computations — which should in principle be possible.
Revised: For orders of magnitude larger memory sizes, one would need to better parallelize reachability computations — which should in principle be possible.

Original: Memory consumption for the stored memories is very modest ( 400 KB), as we only store 200 of 512 -ﬂoat-embeddings, not the observations [Table]
Revised: Memory consumption for the stored memories is very modest ( 400 KB), as we only store 200 of 512 -ﬂoat-embeddings, not the observations.

Original: As for the speed comparison between different methods [Table] PPO + ICM is 1 .
Revised: As for the speed comparison between different methods, PPO + ICM is 1 .

Original: 09 x slower than PPO and
Revised: 09 x slower than PPO and

Original: PPO + EC (our method) is 1 .
Revised: PPO + EC (our method) is 1 .

Original: 84 x slower than PPO.
Revised: 84 x slower than PPO.

Original: In terms of the number of parameters, R-network brings 13 M trainable variables, while PPO alone was 1 .
Revised: In terms of the number of parameters, R-network brings 13 M trainable variables, while PPO alone was 1 .

Original: 7 M and PPO
Revised: 7 M and PPO

Original: + ICM was 2 M.
Revised: + ICM was 2 M.

Original: That said, there was almost no effort spent on optimizing the pipeline in terms of speed/parameters, so it is likely easy to make improvements in this respect.
Revised: That said, there was almost no effort spent on optimizing the pipeline in terms of speed/parameters, so it is likely easy to make improvements in this respect.

Original: It is quite likely that a resource-consuming Resnet-18 is not needed for the R-network — a much simpler model may work as well.
Revised: It is quite likely that a resource-consuming Resnet-18 is not needed for the R-network — a much simpler model may work as well.

Original: In this paper, we followed the setup for the R-network from prior work (Savinov et al., 2018) because it was shown to perform well, but there is no evidence that this setup is necessary.
Revised: In this paper, we followed the setup for the R-network from prior work (Savinov et al., 2018) because it was shown to perform well, but there is no evidence that this setup is necessary.

Original: S A DDITIONAL DMLab T RAINING C URVES
Revised: S A DDITIONAL DMLab T RAINING C URVES

Original: We show additional training curves from the main text experimental section in Figure S2.

Original: for the DMLab task “Dense 2”.
Revised: for the DMLab task “Dense 2”.

Original: Higher is better.
Revised: Higher is better.

Original: We shift the curves for our method by the number of environment steps used to train R-network — so the comparison between different methods is fair.
Revised: We shift the curves for our method by the number of environment steps used to train R-network — so the comparison between different methods is fair.

Original: We run every method 30 times and show 5 randomly selected runs.
Revised: We run every method 30 times and show 5 randomly selected runs.

Original: No seed tuning is performed.
Revised: No seed tuning is performed.

Revised: We show additional training curves from the main text experimental section in Figure S2.