Original: D IVIDE AND E XPLORE : M ULTI -A GENT S EPARATE E XPLORATION WITH S HARED I NTRINSIC M OTIVATIONS
Revised: D IVIDE AND E XPLORE : M ULTI -A GENT S EPARATE E XPLORATION WITH S HARED I NTRINSIC M OTIVATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: One of the greatest challenges of reinforcement learning is efﬁcient exploration, especially when training signals are sparse or deceptive.
Revised: One of the greatest challenges of reinforcement learning is efﬁcient exploration, especially when training signals are sparse or deceptive.

Original: The main difﬁculty of exploration lies in the size and complexity of the state space, which makes simple approaches such as exhaustive search infeasible.
Revised: The main difﬁculty of exploration lies in the size and complexity of the state space, which makes simple approaches such as exhaustive search infeasible.

Original: Our work is based on two important observations.
Revised: Our work is based on two important observations.

Original: On one hand, modern computing platforms are extremely scalable in terms of number of computing nodes and cores, which can complete asynchronous and well load-balanced computational tasks very fast.
Revised: On one hand, modern computing platforms are extremely scalable in terms of number of computing nodes and cores, which can complete asynchronous and well load-balanced computational tasks very fast.

Original: On the other hand, Divide-and-Conquer is a commonly used technique in computer science to solve similar problems (such as SAT) of doing efﬁcient search in extremely large state space.
Revised: On the other hand, Divide-and-Conquer is a commonly used technique in computer science to solve similar problems (such as SAT) of doing efﬁcient search in extremely large state space.

Original: In this paper, we apply the idea of divide-and-conquer in the context of intelligent exploration.
Revised: In this paper, we apply the idea of divide-and-conquer in the context of intelligent exploration.

Original: The resulting exploration scheme can be combined with various speciﬁc intrinsic rewards designed for the given task.
Revised: The resulting exploration scheme can be combined with various speciﬁc intrinsic rewards designed for the given task.

Original: In our exploration scheme, the learning algorithm can automatically divide the state space into regions, and each agent is assigned to explore one of these regions.
Revised: In our exploration scheme, the learning algorithm can automatically divide the state space into regions, and each agent is assigned to explore one of these regions.

Original: All the agents run asynchronously and they can be deployed onto modern distributed computing platforms.
Revised: All the agents run asynchronously and they can be deployed onto modern distributed computing platforms.

Original: Our experiments show that the proposed method is highly efﬁcient and is able to achieve state-of-the-art results in many RL tasks such as MiniGrid and
Revised: Our experiments show that the proposed method is highly efﬁcient and is able to achieve state-of-the-art results in many RL tasks such as MiniGrid and

Original: Vizdoom.
Revised: Vizdoom.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning is a very important learning framework to train intelligent agents to solve complex tasks (Sutton & Barto, 2018).
Revised: Deep reinforcement learning is a very important learning framework to train intelligent agents to solve complex tasks (Sutton & Barto, 2018).

Original: Despite recent success stories in a variety of real tasks (Silver et al., 2016; Vinyals et al., 2019; Lee et al., 2020), the difﬁculty of exploration remains a major challenge for DRL when the reward is sparse or deceptive (Burda et al., 2018).
Revised: Despite recent success stories in a variety of real tasks (Silver et al., 2016; Vinyals et al., 2019; Lee et al., 2020), the difﬁculty of exploration remains a major challenge for DRL when the reward is sparse or deceptive (Burda et al., 2018).

Original: Recent research in exploration has made signiﬁcant progresses thanks to the idea of intrinsic motivation (Schmidhuber, 2010).
Revised: Recent research in exploration has made signiﬁcant progresses thanks to the idea of intrinsic motivation (Schmidhuber, 2010).

Original: Intrinsic motivation could be viewed as objectives for unsupervised reinforcement learning (Levine, 2020), and their goal is usually not to guide the agent to complete a speciﬁc task, but to help the agent to discover interesting states and trajectories which could help the agent to understand the environment and ﬁnd possible target solutions for speciﬁc tasks.
Revised: Intrinsic motivation could be viewed as objectives for unsupervised reinforcement learning (Levine, 2020), and their goal is usually not to guide the agent to complete a speciﬁc task, but to help the agent to discover interesting states and trajectories which could help the agent to understand the environment and ﬁnd possible target solutions for speciﬁc tasks.

Original: Many previous works on intrinsic motivation (Raileanu & Rockt¨aschel, 2020; Burda et al., 2018; 2019) can be understood as proposing some kind of metrics to measure the novelty of the states, and then using the metric as an intrinsic reward to encourage agents to reach novel states, such as count-based exploration (Bellemare et al., 2016) and ICM (Pathak et al., 2017).
Revised: Many previous works on intrinsic motivation (Raileanu & Rockt¨aschel, 2020; Burda et al., 2018; 2019) can be understood as proposing some kind of metrics to measure the novelty of the states, and then using the metric as an intrinsic reward to encourage agents to reach novel states, such as count-based exploration (Bellemare et al., 2016) and ICM (Pathak et al., 2017).

Original: Despite of the signiﬁcant progresses that have been made, in theory, efﬁcient exploration is still very challenging if not impossible, due to the infeasible size of the state space to be explored.
Revised: Despite of the signiﬁcant progresses that have been made, in theory, efﬁcient exploration is still very challenging if not impossible, due to the infeasible size of the state space to be explored.

Original: Similar difﬁculties are usually seen in the state-space search problems in theoretical computer science.
Revised: Similar difﬁculties are usually seen in the state-space search problems in theoretical computer science.

Original: For example, in SMT or SAT problems (De Moura & Bjørner, 2008), one is required to search for a feasible solution to satisfy certain constraints, and these problems are usually NP complete because of the exponential size of the state space.
Revised: For example, in SMT or SAT problems (De Moura & Bjørner, 2008), one is required to search for a feasible solution to satisfy certain constraints, and these problems are usually NP complete because of the exponential size of the state space.

Original: A very common class of practical algorithms to solve these search problems is divide and conquer.
Revised: A very common class of practical algorithms to solve these search problems is divide and conquer.

Original: The algorithm ﬁrst tries to divide the state space into certain pieces and then search for results in each piece of state space.
Revised: The algorithm ﬁrst tries to divide the state space into certain pieces and then search for results in each piece of state space.

Original: A signiﬁcant beneﬁt of this approach is it can be easily modiﬁed to take advantage of modern distributed/concurrent computing hardware, because it can naturally divide a hard problem into smaller and simpler ones, allowing different computing nodes to complete the tasks asynchronously.
Revised: A signiﬁcant beneﬁt of this approach is it can be easily modiﬁed to take advantage of modern distributed/concurrent computing hardware, because it can naturally divide a hard problem into smaller and simpler ones, allowing different computing nodes to complete the tasks asynchronously.

Original: Divide-and-conquer type of algorithms are among the state-of-the art algorithms to solve SMT or SAT problems.
Revised: Divide-and-conquer type of algorithms are among the state-of-the art algorithms to solve SMT or SAT problems.

Original: In this paper, we take inspiration from divide and conquer algorithms in theoretical computer science and try to adapt them to the exploration problems in RL.
Revised: In this paper, we take inspiration from divide and conquer algorithms in theoretical computer science and try to adapt them to the exploration problems in RL.

Original: Different from traditional exploration mechanisms with a single exploring agent, we train a number of concurrent exploring agents.
Revised: Different from traditional exploration mechanisms with a single exploring agent, we train a number of concurrent exploring agents.

Original: We carefully design our intrinsic reward such that each exploring agent is encouraged to explore only one region in the state space.
Revised: We carefully design our intrinsic reward such that each exploring agent is encouraged to explore only one region in the state space.

Original: In other words, the state space is automatically divided into several components, where each agent is responsible for exploring one of these components.
Revised: In other words, the state space is automatically divided into several components, where each agent is responsible for exploring one of these components.

Original: In order to achieve this, we ﬁrst choose a reward function based on intrinsic motivation, e.g. countbased reward or ICM.
Revised: In order to achieve this, we ﬁrst choose a reward function based on intrinsic motivation, e.g. countbased reward or ICM.

Original: This function is able to output a novelty score for a given transition.
Revised: This function is able to output a novelty score for a given transition.

Original: We initialize K agents, each agent is equipped with its own version of reward function.
Revised: We initialize K agents, each agent is equipped with its own version of reward function.

Original: Then we design a multi-party intrinsic reward for each agent consisting of two parts: (1) a reward to discourage the agent to visit states which has already been explored by other agents.
Revised: Then we design a multi-party intrinsic reward for each agent consisting of two parts: (1) a reward to discourage the agent to visit states which has already been explored by other agents.

Original: (2) a self exploration reward to encourage the agent to visit novel states which has not been visited by itself.
Revised: (2) a self exploration reward to encourage the agent to visit novel states which has not been visited by itself.

Original: Agents run in a fully asynchronous manner, no agent has to wait for other other agent’s progress to take exploration steps.
Revised: Agents run in a fully asynchronous manner, no agent has to wait for other other agent’s progress to take exploration steps.

Original: All agents communicate with each other through shared memory, under the abstraction of a simple read-write shared object, which can be made lock-free, i.e., the learning system can always progress regardless of some computing nodes may delay or fail.
Revised: All agents communicate with each other through shared memory, under the abstraction of a simple read-write shared object, which can be made lock-free, i.e., the learning system can always progress regardless of some computing nodes may delay or fail.

Original: These properties make our algorithm highly scalable.
Revised: These properties make our algorithm highly scalable.

Original: Our experiments show that the proposed method is highly efﬁcient and is able to achieve state-of-the-art results in many RL tasks such as MiniGrid and Vizdoom.
Revised: Our experiments show that the proposed method is highly efﬁcient and is able to achieve state-of-the-art results in many RL tasks such as MiniGrid and Vizdoom.

Original:  R ELATED W ORK Intrinsic Motivation
Revised:  R ELATED W ORK Intrinsic Motivation

Original: The study of intrinsic motivation has been of interest for some time (Schmidhuber, 2010).
Revised: The study of intrinsic motivation has been of interest for some time (Schmidhuber, 2010).

Original: Earlier works on intrinsic motivation tended to analyse with psychology and combined with reinforcement learning in tabular setting (Deci et al., 1981; Singh et al., 2005).
Revised: Earlier works on intrinsic motivation tended to analyse with psychology and combined with reinforcement learning in tabular setting (Deci et al., 1981; Singh et al., 2005).

Original: With the breakthroughs of deep reinforcement learning, there has been a surge of interest in applying intrinsic motivation with deep neural networks for hard exploration RL problems, in which countbased exploration and curiosity-driven exploration are two main promising research branches.
Revised: With the breakthroughs of deep reinforcement learning, there has been a surge of interest in applying intrinsic motivation with deep neural networks for hard exploration RL problems, in which countbased exploration and curiosity-driven exploration are two main promising research branches.

Original: For count-based methods, the agent is encouraged to explore rarely visited states, while for curiositydriven methods, the agent is encouraged to explore via learning the world model.
Revised: For count-based methods, the agent is encouraged to explore rarely visited states, while for curiositydriven methods, the agent is encouraged to explore via learning the world model.

Original: Count (Bellemare et al., 2016) extends count-based methods to non-tabular setting by deriving pseudo-counts with a density model.
Revised: Count (Bellemare et al., 2016) extends count-based methods to non-tabular setting by deriving pseudo-counts with a density model.

Original: Ostrovski et al.
Revised: Ostrovski et al.

Original: (2017) adapts Bellemare et al.
Revised: (2017) adapts Bellemare et al.

Original: (2016) with PixelCNN.
Revised: (2016) with PixelCNN.

Original: RND (Burda et al., 2018) uses random network distillation to measure the familiarity of states, and RGU (Badia et al., 2020) combines RND with an episodic novelty module.
Revised: RND (Burda et al., 2018) uses random network distillation to measure the familiarity of states, and RGU (Badia et al., 2020) combines RND with an episodic novelty module.

Original: ICM (Pathak et al., 2017) builds a forward dynamic model and an inverse dynamic model to measure the curiosity for exploration.
Revised: ICM (Pathak et al., 2017) builds a forward dynamic model and an inverse dynamic model to measure the curiosity for exploration.

Original: RIDE (Raileanu & Rockt ¨ aschel, 2020) has similar networks with ICM while using a different intrinsic reward combined with episodic state visitation counts.
Revised: RIDE (Raileanu & Rockt ¨ aschel, 2020) has similar networks with ICM while using a different intrinsic reward combined with episodic state visitation counts.

Original: AMIGo (Campero et al., 2020) proposes a meta-learning framework for generating adversarially motivated intrinsic goals in curriculum to guide agent exploration.
Revised: AMIGo (Campero et al., 2020) proposes a meta-learning framework for generating adversarially motivated intrinsic goals in curriculum to guide agent exploration.

Original: RAPID (Zha et al., 2021) ranks past trajectories with episodic scores and does imitation learning for distillation.
Revised: RAPID (Zha et al., 2021) ranks past trajectories with episodic scores and does imitation learning for distillation.

Original: AGAC (Flet-Berliac et al., 2021) formulates an actor-critic framework with adversary.
Revised: AGAC (Flet-Berliac et al., 2021) formulates an actor-critic framework with adversary.

Original: Our work is different from above by building a multi-agent learning mechanism to accelerate exploration.
Revised: Our work is different from above by building a multi-agent learning mechanism to accelerate exploration.

Original: Distributed Reinforcement Learning
Revised: Distributed Reinforcement Learning

Original: The scalability of large scale distributed learning in DRL is a main challenge for reinforcement learning.
Revised: The scalability of large scale distributed learning in DRL is a main challenge for reinforcement learning.

Original: Most of previous works scale up deep reinforcement learning with distributed asynchronous SGD (Dean et al., 2012) by distributing RL components.
Revised: Most of previous works scale up deep reinforcement learning with distributed asynchronous SGD (Dean et al., 2012) by distributing RL components.

Original: Gorila (Nair et al., 2015) scales DQN (Mnih et al., 2015) with distributed Q networks and replay buffers.
Revised: Gorila (Nair et al., 2015) scales DQN (Mnih et al., 2015) with distributed Q networks and replay buffers.

Original: A3C (Mnih et al., 2016) uses multiple workers for training and collects all gradients for parameter synchronous.
Revised: A3C (Mnih et al., 2016) uses multiple workers for training and collects all gradients for parameter synchronous.

Original: Unlike A3C, IMPALA (Espeholt et al., 2018) transfers all trajectories to multiple central learners for distributed learning.
Revised: Unlike A3C, IMPALA (Espeholt et al., 2018) transfers all trajectories to multiple central learners for distributed learning.

Original: Ape-X (Horgan et al., 2018) consists of a distributed replay buffers and a synchronous learner for value-based methods.
Revised: Ape-X (Horgan et al., 2018) consists of a distributed replay buffers and a synchronous learner for value-based methods.

Original: SEED RL (Espeholt et al., 2019) and OpenAI Rapid (OpenAI, 2018) apply central GPU based inference for acceleration.
Revised: SEED RL (Espeholt et al., 2019) and OpenAI Rapid (OpenAI, 2018) apply central GPU based inference for acceleration.

Original: For most of the above works, the main parallelism exploited by the distributed framework is data parallelism, e.g. they use multiple nodes for collecting data, and divide minibatches among computing nodes and aggregate on the gradients computed by each node.
Revised: For most of the above works, the main parallelism exploited by the distributed framework is data parallelism, e.g. they use multiple nodes for collecting data, and divide minibatches among computing nodes and aggregate on the gradients computed by each node.

Original: Our model, on the other hand, can be viewed as taking advantage a new parallelism speciﬁc to RL problems — state space parallelism.
Revised: Our model, on the other hand, can be viewed as taking advantage a new parallelism speciﬁc to RL problems — state space parallelism.

Original: Instead of dividing minibatches for the nodes, we let the agents divide the state space into pieces, which greatly improves exploration efﬁciency.
Revised: Instead of dividing minibatches for the nodes, we let the agents divide the state space into pieces, which greatly improves exploration efﬁciency.

Original:  B ACKGROUND 3. P ROBLEM F ORMULATION
Revised:  B ACKGROUND 3. P ROBLEM F ORMULATION

Original: In this paper, we consider an RL problem formulated as a Markov Decision Process(MDP), which is deﬁned as a tuple MDP = { S, A, P, R, γ } , where S is the state space, A is the action space of the agent, P is the transition probabilities which represents the stochastic transition function S ×
Revised: In this paper, we consider an RL problem formulated as a Markov Decision Process(MDP), which is deﬁned as a tuple MDP = { S, A, P, R, γ } , where S is the state space, A is the action space of the agent, P is the transition probabilities which represents the stochastic transition function S ×

Original: A → S ,
Revised: A → S ,

Original: R ( s, a ) is the reward function after the agent executes an action, γ is the discount factor.
Revised: R ( s, a ) is the reward function after the agent executes an action, γ is the discount factor.

Original: The goal is to maximize the expected accumulated rewards R = E ( (cid:80)
Revised: The goal is to maximize the expected accumulated rewards R = E ( (cid:80)

Original: Tt =

Revised: Tt =

Original: γ t r ( s t , a t ))
Revised: γ t r ( s t , a t ))

Original: the agent receives when executing series of optimal actions in episodes with policy π .
Revised: the agent receives when executing series of optimal actions in episodes with policy π .

Original: In our method, other than the rewards r t received from environment at timestep t , our agents also receive a intrinsic reward R t as an guidance for efﬁcient exploration.
Revised: In our method, other than the rewards r t received from environment at timestep t , our agents also receive a intrinsic reward R t as an guidance for efﬁcient exploration.

Original: The intrinsic reward is a parameterized function of state and action R t ( s, a ) =
Revised: The intrinsic reward is a parameterized function of state and action R t ( s, a ) =

Original: f θ ( s, a ) ,where the parameters θ = θ ( M ) depends on the agent’s history or replay buffer M .
Revised: f θ ( s, a ) ,where the parameters θ = θ ( M ) depends on the agent’s history or replay buffer M .

Original: For example, in count-based reward, θ is the number of visits at each state.
Revised: For example, in count-based reward, θ is the number of visits at each state.

Original: Instead of a single exploring agent, our proposed method trains K independent exploring agents.
Revised: Instead of a single exploring agent, our proposed method trains K independent exploring agents.

Original: The exploring agents do not communicate or affect each other directly, however they indirectly affect each other through the intrinsic rewards.
Revised: The exploring agents do not communicate or affect each other directly, however they indirectly affect each other through the intrinsic rewards.

Original: Each agent uses its own history or replay memory to train its own version of f θ (the intrinsic motivation model).
Revised: Each agent uses its own history or replay memory to train its own version of f θ (the intrinsic motivation model).

Original: For any agent, the total intrinsic reward used for agent training is formed by combining all trained reward functions of all the agents.
Revised: For any agent, the total intrinsic reward used for agent training is formed by combining all trained reward functions of all the agents.

Original: 3. C ONCURRENT S HARED O BJECTS
Revised: 3. C ONCURRENT S HARED O BJECTS

Original: A commonly used primitive in concurrent or distributed computing is concurrent shared objects.
Revised: A commonly used primitive in concurrent or distributed computing is concurrent shared objects.

Original: It is basically the user interface of concurrent data structures.
Revised: It is basically the user interface of concurrent data structures.

Original: A concurrent shared objects is composed of several methods which all processes can call.
Revised: A concurrent shared objects is composed of several methods which all processes can call.

Original: The behavior of a shared object is deﬁned by its sequential execution traces.
Revised: The behavior of a shared object is deﬁned by its sequential execution traces.

Original: For example, a concurrent Set object has three methods, add ( v ) , ﬁnd ( v ) and remove ( v ) .
Revised: For example, a concurrent Set object has three methods, add ( v ) , ﬁnd ( v ) and remove ( v ) .

Original: The behavior of this concurrent object is deﬁned by the sequential executions of an ordinary (non-concurrent) Set data type.
Revised: The behavior of this concurrent object is deﬁned by the sequential executions of an ordinary (non-concurrent) Set data type.

Original: Also, a property called ”linearizability” is applied to guarantee consistent usage of the concurrent object.
Revised: Also, a property called ”linearizability” is applied to guarantee consistent usage of the concurrent object.

Original: Linearizability basically says that for any concurrent execution traces of the concurrent object, one can ﬁnd a sequential execution trace which yields exactly the same results and the sequential execution trace preserves the real time ordering of the concurrent execution trace.
Revised: Linearizability basically says that for any concurrent execution traces of the concurrent object, one can ﬁnd a sequential execution trace which yields exactly the same results and the sequential execution trace preserves the real time ordering of the concurrent execution trace.

Original: In other words, all the methods of the concurrent objects can be viewed as executed atomically.
Revised: In other words, all the methods of the concurrent objects can be viewed as executed atomically.

Original: In this paper, we assume all concurrent objects are linearizable.
Revised: In this paper, we assume all concurrent objects are linearizable.

Original:  D IVIDE AND E XPLORE
Revised:  D IVIDE AND E XPLORE

Original: We design a learning mechanism named Divide-and-Explore (D & E) to realize the idea mentioned above.
Revised: We design a learning mechanism named Divide-and-Explore (D & E) to realize the idea mentioned above.

Original: An overview of D & E is illustrated in Fig.1.
Revised: An overview of D & E is illustrated in Fig.1.

Original: The learning framework consists of n independent agents { A 1 , A 2 , ..., A n } , and each agent tries to explore a particular region in the state space.
Revised: The learning framework consists of n independent agents { A 1 , A 2 , ..., A n } , and each agent tries to explore a particular region in the state space.

Original: We let replay memory M i denote the set of all the past experiences of the i th agent
Revised: We let replay memory M i denote the set of all the past experiences of the i th agent

Original: A i in the form ( s it , a it , r it , s it +1 ) .
Revised: A i in the form ( s it , a it , r it , s it +1 ) .

Original: Let M =
Revised: Let M =

Original: ( M 1 , · · · M n ) be the collection of replay memories of all agents.
Revised: ( M 1 , · · · M n ) be the collection of replay memories of all agents.

Original: Note that in practice we usually don’t need to store all the past experiences in the replay memory, but some sufﬁcient statistics of the experiences is enough.
Revised: Note that in practice we usually don’t need to store all the past experiences in the replay memory, but some sufﬁcient statistics of the experiences is enough.

Original: Our algorithm ﬁrst speciﬁes a learnable intrinsic motivation model f θ as a reward function, and θ is the model’s parameters.
Revised: Our algorithm ﬁrst speciﬁes a learnable intrinsic motivation model f θ as a reward function, and θ is the model’s parameters.

Original: For each agent A i , we keep track of a distinct model f i =
Revised: For each agent A i , we keep track of a distinct model f i =

Original: f θ
Revised: f θ

Original: i , where θ i
Revised: i , where θ i

Original: = θ ( M i ) is a function of the replay memory of agent A i .
Revised: = θ ( M i ) is a function of the replay memory of agent A i .

Original: When agent A i takes a transition ( s it , a it , r it , s it +1 ) , for every agent A j , our training architecture computes a intrinsic reward ˆ r i,jt based on the transition and the agent’s f j , as following
Revised: When agent A i takes a transition ( s it , a it , r it , s it +1 ) , for every agent A j , our training architecture computes a intrinsic reward ˆ r i,jt based on the transition and the agent’s f j , as following

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Intuitively, ˆ r i,jt measures the transition’s novelty for agent A j .
Revised: Intuitively, ˆ r i,jt measures the transition’s novelty for agent A j .

Original: If ˆ r i,jt is large, it means that the state s i +1 is rarely visited and not explored fully by Agent A j .
Revised: If ˆ r i,jt is large, it means that the state s i +1 is rarely visited and not explored fully by Agent A j .

Original: The intrinsic motivation model f θ can be selected with any kind of count-based, curiosity-driven and memory-based exploration methods (e.g. Pathak et al.
Revised: The intrinsic motivation model f θ can be selected with any kind of count-based, curiosity-driven and memory-based exploration methods (e.g. Pathak et al.

Original: ; Burda et al.
Revised: ; Burda et al.

Original: according to the speciﬁed task.
Revised: according to the speciﬁed task.

Original: 4. D IVIDE AND E XPLORE R EWARD
Revised: 4. D IVIDE AND E XPLORE R EWARD

Original: Here we deﬁne the D & E intrinsic reward for agent learning as below:
Revised: Here we deﬁne the D & E intrinsic reward for agent learning as below:

Original: [Equation]
Revised: [Equation]

Original: where ω o ( n − 1) + ω s = 1 .
Revised: where ω o ( n − 1) + ω s = 1 .

Original: According to the deﬁnition, the D & E reward aggregates other agents’ intrinsic rewards and the agent’s own intrinsic reward, which encourages the agent to explore the state space where all agents have rarely explored.
Revised: According to the deﬁnition, the D & E reward aggregates other agents’ intrinsic rewards and the agent’s own intrinsic reward, which encourages the agent to explore the state space where all agents have rarely explored.

Original: This mechanism enables full use of all agents’ experience and gets rid of wasting time on fully explored state space so as to improve exploration efﬁciency.
Revised: This mechanism enables full use of all agents’ experience and gets rid of wasting time on fully explored state space so as to improve exploration efﬁciency.

Original: The weights ω o
Revised: The weights ω o

Original: and ω s determine the exploration preference between agent separability and self exploration.
Revised: and ω s determine the exploration preference between agent separability and self exploration.

Original: If ω o is zero, the algorithm turns out to be conventional intrinsic-driven exploration, while if ω s is zero, the algorithm only considers separability and the agent may converge to a local state space where other agents haven’t visited and stop exploring further.
Revised: If ω o is zero, the algorithm turns out to be conventional intrinsic-driven exploration, while if ω s is zero, the algorithm only considers separability and the agent may converge to a local state space where other agents haven’t visited and stop exploring further.

Original: In practice, we set ω o > ω s to encourage agents to explore separately while keeping self exploration.
Revised: In practice, we set ω o > ω s to encourage agents to explore separately while keeping self exploration.

Original: With the D & E intrinsic reward, the total reward is deﬁned as follows:
Revised: With the D & E intrinsic reward, the total reward is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where r it is the task reward provided by the environment, and α it is the weight of D & E intrinsic reward.
Revised: where r it is the task reward provided by the environment, and α it is the weight of D & E intrinsic reward.

Original: Base on the deﬁnition above, we are able to turn a sparse reward task into a dense one, and apply conventional model free reinforcement learning algorithms such as PPO (Schulman et al., 2017) or IMPALA (Espeholt et al., 2018) to train the agent policy.
Revised: Base on the deﬁnition above, we are able to turn a sparse reward task into a dense one, and apply conventional model free reinforcement learning algorithms such as PPO (Schulman et al., 2017) or IMPALA (Espeholt et al., 2018) to train the agent policy.

Original: 4. A UTOMATIC R EWARD W EIGHT D ECAY FOR E XPLOITATION
Revised: 4. A UTOMATIC R EWARD W EIGHT D ECAY FOR E XPLOITATION

Original: While the main goal of Divide-and-Explore is to encourage efﬁcient exploration, we also consider the trade-off between exploration and exploitation.
Revised: While the main goal of Divide-and-Explore is to encourage efﬁcient exploration, we also consider the trade-off between exploration and exploitation.

Original: In conventional model free reinforcement learning, the trade-off is usually controlled in the agent roll out process via (cid:15) -greedy
Revised: In conventional model free reinforcement learning, the trade-off is usually controlled in the agent roll out process via (cid:15) -greedy

Original: (Mnih et al., 2015) or the action probability distribution (Mnih et al., 2016).
Revised: (Mnih et al., 2015) or the action probability distribution (Mnih et al., 2016).

Original: Because in D & E we use intrinsic reward to encourage agent to explore, we reduce the intrinsic reward weight to make agent pay more attention to the trajectories with high extrinsic reward, especially when the agent has trained a lot and has a probability of obtaining some rewards.
Revised: Because in D & E we use intrinsic reward to encourage agent to explore, we reduce the intrinsic reward weight to make agent pay more attention to the trajectories with high extrinsic reward, especially when the agent has trained a lot and has a probability of obtaining some rewards.

Original: The decay of intrinsic reward weight can be formulated as α = d t · α 0 , and d t is decay factor.
Revised: The decay of intrinsic reward weight can be formulated as α = d t · α 0 , and d t is decay factor.

Original: Compared to linear decay, we design an adaptive decay method as follows:
Revised: Compared to linear decay, we design an adaptive decay method as follows:

Original: [Equation]
Revised: [Equation]

Original: in which α 0 > 0 is the initial value of α it , d it is the decay factor of agent
Revised: in which α 0 > 0 is the initial value of α it , d it is the decay factor of agent

Original: A i at time t , N is the number of A i ’s historical episodes and R ij is the return of episodes.
Revised: A i at time t , N is the number of A i ’s historical episodes and R ij is the return of episodes.

Original: We consider more about newly generated episodes by introducing φ ∈ (0 , 1] .
Revised: We consider more about newly generated episodes by introducing φ ∈ (0 , 1] .

Original: U is a hyperparameter, which determines the decay rate.
Revised: U is a hyperparameter, which determines the decay rate, and in general, we let U =

Original: {+(cid:80) Nj =
+}
Revised: {+(cid:80) Nj =
+}

Revised: φ N − j R ∗ .

Original: According to Eq.4, α it has negative correlation with the sum of historical rewards and is adjusted automatically during training, which matches the goal of trade-off.
Revised: According to Eq.4, α it has negative correlation with the sum of historical rewards and is adjusted automatically during training, which matches the goal of trade-off.

Original: 4. F ULLY A SYNCHRONOUS U PDATES WITH C ONCURRENT D ATA S TRUCTURES
Revised: 4. F ULLY A SYNCHRONOUS U PDATES WITH C ONCURRENT D ATA S TRUCTURES

Original: In order to speed up computing and scale our approach to large number of exploring agents, we use non-blocking concurrent computing primitives in our implementation.
Revised: In order to speed up computing and scale our approach to large number of exploring agents, we use non-blocking concurrent computing primitives in our implementation.

Original: In our algorithm, the reward functions f i =
Revised: In our algorithm, the reward functions f i =

Original: f θ
Revised: f θ

Original: i , i = 1 , 2 · · · n is shared across all agents.
Revised: i , i = 1 , 2 · · · n is shared across all agents.

Original: Only agent A i is allowed to modify θ i , but all agents can read θ i .
Revised: Only agent A i is allowed to modify θ i , but all agents can read θ i .

Original: We can store θ =
Revised: We can store θ =

Original: ( θ 1 , θ 2 , · · · θ n ) in a shared concurrent object O .
Revised: ( θ 1 , θ 2 , · · · θ n ) in a shared concurrent object O .

Original: In the terminology of concurrent data structures, we deﬁne O as a concurrent object that allows two operations, read ( i ) which returns a copy of θ i and update ( i, v ) which overwrite current θ i into value v .
Revised: In the terminology of concurrent data structures, we deﬁne O as a concurrent object that allows two operations, read ( i ) which returns a copy of θ i and update ( i, v ) which overwrite current θ i into value v .

Original: A lot of concurrent data structures could be used to implement this concurrent object, such as read-write locks (locking), transactional memory or software transactional memory (lock-free).
Revised: A lot of concurrent data structures could be used to implement this concurrent object, such as read-write locks (locking), transactional memory or software transactional memory (lock-free).

Original: In our work, we use software transactional memory since it is non-blocking, making our algorithm fully asynchronous.
Revised: In our work, we use software transactional memory since it is non-blocking, making our algorithm fully asynchronous.

Original: Our approach is summarized in Algorithm 1.
Revised: Our approach is summarized in Algorithm 1.

Original: All the agents are trained asynchronously in a distributed training framework, and the shared object O is read every L steps.
Revised: All the agents are trained asynchronously in a distributed training framework, and the shared object O is read every L steps.

Original: Besides, there is a special attribute of D & E that the algorithm is considered convergence as long as only one agent accomplishes the task.
Revised: Besides, there is a special attribute of D & E that all agents will be convergence as long as only one agent accomplishes the task.

Original: With the shared intrinsic motivation and learning mechanism, the failure experience of other agents is transferred to the succeeding agent as bad cases to avoid repetition of wrong exploration, which demonstrates the superiority of this algorithm.
Revised: With the shared intrinsic motivation and learning mechanism, the failure experience of other agents is transferred to the succeeding agent as bad cases to avoid repetition of wrong exploration, which demonstrates the superiority of this algorithm.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate D & E on two challenging reinforcement learning research platforms
Revised: In this section, we evaluate D & E on two challenging reinforcement learning research platforms

Original: MiniGrid (Chevalier-Boisvert et al., 2018) and VizDoom (Kempka et al., 2016), and both of them have hard exploration environments with sparse rewards, which are commonly used as benchmarks.
Revised: MiniGrid (Chevalier-Boisvert et al., 2018) and VizDoom (Kempka et al., 2016), and both of them have hard exploration environments with sparse rewards, which are commonly used as benchmarks.

Original: For all our experiments, we compare our approach against various state-of-the-art baselines for hard exploration including Count (Bellemare et al., 2016), ICM (Pathak et al., 2017), RND (Burda et al., 2018), AMIGo (Campero et al., 2020), RIDE (Raileanu & Rockt¨aschel, 2020), RAPID (Zha et al., 2021) and AGAC (Flet-Berliac et al., 2021).
Revised: For all our experiments, we compare our approach against various state-of-the-art baselines for hard exploration including Count (Bellemare et al., 2016), ICM (Pathak et al., 2017), RND (Burda et al., 2018), AMIGo (Campero et al., 2020), RIDE (Raileanu & Rockt¨aschel, 2020), RAPID (Zha et al., 2021) and AGAC (Flet-Berliac et al., 2021).

Original: 5. M INI G RID E XPERIMENTS Environment
Revised: 5. M INI G RID E XPERIMENTS Environment

Original: The MiniGrid environments are a set of procedurally-generated 2D grid worlds with sparse reward.
Revised: The MiniGrid environments are a set of procedurally-generated 2D grid worlds with sparse reward.

Original: Following AGAC (Flet-Berliac et al., 2021)’s setting, we choose three types of tasks for experiments: MultiRoom( MR ), KeyCorrider( KC ) and ObstructedMaze( OM ).
Revised: Following AGAC (Flet-Berliac et al., 2021)’s setting, we choose three types of tasks for experiments: MultiRoom( MR ), KeyCorrider( KC ) and ObstructedMaze( OM ).

Original: In the
Revised: In the

Original: MultiRoom tasks, there are multiple rooms connected one-by-one with doors.
Revised: MultiRoom tasks, there are multiple rooms connected one-by-one with doors.

Original: The agent has to go through these rooms to reach a goal.
Revised: The agent has to go through these rooms to reach a goal.

Original: In the KeyCorrider tasks, several rooms are nested together and the mission of the agent is to ﬁnd a key so as to open a locked room and pick up an object.
Revised: In the KeyCorrider tasks, several rooms are nested together and the mission of the agent is to ﬁnd a key so as to open a locked room and pick up an object.

Original: For the last, the ObstructedMaze tasks are updated versions of KeyCorrider and most challenging wherethe key is hidden in boxes and the door is obstructed by balls. For all tasks, we make use of partial
Revised: For the last, the ObstructedMaze tasks are updated versions of KeyCorrider and most challenging where

Original: - eters 3: for convergence do 4: for all A i i 2 · · · n do asynchronously 5:
Revised: Update calling

Original: Roll do 7:
Revised: 14: end

Original: Compute total rewards

Original: Updateagent
Revised: Update

Original: M Compute parameters based on the memory Update shared object by calling method
Revised: 17: end

Original: i 14: end for 15: for every K steps do 16: Update reward weight α
Revised: 18: end 19: Return the key is hidden in boxes and the door is obstructed by balls.

Original: Eq. 17: end for 18: end for 19: Return the optimal policy π ∗ observations, which means the observation space is a 7x7x3 tensor indicating the information of partially-observable grid cells, and there are 7 actions for executing.
Revised: For all tasks, we make use of partial observations, which means the observation space is a 7x7x3 tensor indicating the information of partially-observable grid cells, and there are 7 actions for executing.

Original: D & E with Count-based Exploration
Revised: D & E with Count-based Exploration

Original: Due to the simplicity of 2D grid environment, we count the states in the form of tabular as exploration bonus in the MiniGrid tasks, which has been used in previous works, e.g. Count (Bellemare et al., 2016), RIDE (Raileanu & Rockt ¨ aschel, 2020) and
Revised: Due to the simplicity of 2D grid environment, we count the states in the form of tabular as exploration bonus in the MiniGrid tasks, which has been used in previous works, e.g. Count (Bellemare et al., 2016), RIDE (Raileanu & Rockt ¨ aschel, 2020) and

Original: AGAC (Flet-Berliac et al., 2021).
Revised: AGAC (Flet-Berliac et al., 2021).

Original: Therefore, We combine count-based exploration with D & E for experiments.
Revised: Therefore, We combine count-based exploration with D & E for experiments.

Original: In detail, we deﬁne the intrinsic reward by:
Revised: In detail, we deﬁne the intrinsic reward by:

Original: [Equation]
Revised: [Equation]

Revised: where I ( · ) is indicator function.

Original: We also introduce an inner-episodic count mask as a multiplier by replacing original reward weight α it with:
Revised: We also introduce an inner-episodic count mask as a multiplier by replacing original reward weight α it with:

Original: [Equation]
Revised: [Equation]

Original: where C i is the trajectory in current episode.
Revised: where C i is the trajectory in current episode.

Original: With this mask, if a state has been visited in the trajectory, the intrinsic reward will be masked, which improve the exploration efﬁciency.
Revised: With this mask, if a state has been visited in the trajectory, the intrinsic reward will be masked, which improve the exploration efﬁciency.

Original: Training
Revised: Training

Original: We use PPO (Schulman et al.
Revised: We use PPO (Schulman et al.

Original: as a standard model free RL algorithm to train all the agents.
Revised: as a standard model free RL algorithm to train all the agents.

Original: For fair comparison, the architecture of actor and critic network we use is the same as AGAC ’s network in Flet-Berliac et al.
Revised: For fair comparison, the architecture of actor and critic network we use is the same as AGAC ’s network in Flet-Berliac et al.

Original: (2021), which consists of 3 convolutional blocks and a fully connected layer for feature processing.
Revised: (2021), which consists of 3 convolutional blocks and a fully connected layer for feature processing.

Original: More concretely, each convolutional block contains a 32-ﬁlter 3x3 convolution with stride size 2, and the hidden size of fully connected layer is 512.
Revised: More concretely, each convolutional block contains a 32-ﬁlter 3x3 convolution with stride size 2, and the hidden size of fully connected layer is 512.

Original: The number of agents in D & E is set to 3 as default for all tasks.
Revised: The number of agents in D & E is set to 3 as default for all tasks.

Original: More hyperparameters can be seen in
Revised: More hyperparameters can be seen in

Original: Appendix A.
Revised: Appendix A.

Original: Results
Revised: Results

Original: We train D & E with various MiniGrid tasks and compare the results with baselines in both exploration efﬁciency and performance.
Revised: We train D & E with various MiniGrid tasks and compare the results with baselines in both exploration efﬁciency and performance.

Original: For exploration efﬁciency, we illustrated the learning curves of D & E and baselines in less than 50M timesteps in Fig.2.
Revised: For exploration efﬁciency, we illustrated the learning [Figure] [Table]

Original: Because of the training con-
Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: curves of D & E and baselines in less than 50M timesteps in Fig.2.

Original: currency in all D & E’s agents, we record the best agent’s performance for D & E with the agent’s timesteps by default.
Revised: Because of the training concurrency in all D & E’s agents, we record the best agent’s performance for D & E with the agent’s timesteps by default.

Original: In all the tasks, D & E achieves state-of-the-art results and the performance boosts with much less timesteps compared to other methods especially AGAC and RAPID.
Revised: In all the tasks, D & E achieves state-of-the-art results and the performance boosts with much less timesteps compared to other methods especially AGAC and RAPID.

Original: Only in the ObstructedMaze-2Q task, the performance of AGAC is comparable with D & E.
Revised: Only in the ObstructedMaze-2Q task, the performance of AGAC is comparable with D & E.

Original: For performance only, we report the ﬁnal performance of all methods in Table 1.
Revised: For performance only, we report the ﬁnal performance of all methods in Table 1.

Original: All baselines are trained in 50M timesteps.
Revised: All baselines are trained in 50M timesteps.

Original: Due to the multi-agent learning mechanism, the total number of timesteps used in D & E is directly proportional to the number of agents.
Revised: Due to the multi-agent learning mechanism, the total number of timesteps used in D & E is directly proportional to the number of agents.

Original: Therefore, we only record the performance of D & E at 16.7M timesteps for a fair comparison.
Revised: Therefore, we only record the performance of D & E at 16.7M timesteps for a fair comparison.

Original: As we can see in the table, D & E performs best in most scenarios while several approaches such as RND, ICM and Count totally failed in all the listed tasks.
Revised: As we can see in the table, D & E performs best in most scenarios while several approaches such as RND, ICM and Count totally failed in all the listed tasks.

Original: For better understanding the exploration efﬁciency, we also plot the state visitation heatmaps of the challenging MultiRoom-N12-S10 task with 1M timesteps in Fig.3.
Revised: For better understanding the exploration efﬁciency, we also plot the state visitation heatmaps of the challenging MultiRoom-N12-S10 task with 1M timesteps in Fig.3.

Original: The policies are trained in a singleton environment.
Revised: The policies are trained in a singleton environment.

Original: While D & E ﬁnds a good trajectory to the last room with few wrong actions,
Revised: While D & E ﬁnds a good trajectory to the last room with few wrong actions,

Original: AGAC spends more time in the journey, and RAPID is stuck in the sixth room.
Revised: AGAC spends more time in the journey, and RAPID is stuck in the sixth room.

Original: [Table]
Revised: [Table]

Original: 5. V IZ D OOM E XPERIMENTS Environment
Revised: 5. V IZ D OOM E XPERIMENTS Environment

Original: The VizDoom environments are a series of Doom-based 3D scenarios for reinforcement learning.
Revised: The VizDoom environments are a series of Doom-based 3D scenarios for reinforcement learning.

Original: We choose the MyWayHome task following AGAC (Flet-Berliac et al., 2021)’s setting.
Revised: We choose the MyWayHome task following AGAC (Flet-Berliac et al., 2021)’s setting.

Original: The MyWayHome task consists of multiple rooms and corridors, which is a 3D navigation task that the agent explores rooms to ﬁnd the object.
Revised: The MyWayHome task consists of multiple rooms and corridors, which is a 3D navigation task that the agent explores rooms to ﬁnd the object.

Original: Different from MiniGrid, the state is an ego-centric visual observation with a size of 42x42x4, and the number of action is 5.
Revised: Different from MiniGrid, the state is an ego-centric visual observation with a size of 42x42x [Table] and the number of action is 5.

Original: D & E with Curiosity-Driven Exploration
Revised: Curiosity-Driven Exploration

Original: In order to demonstrate the generality of D & E and compare with baselines, we get rid of using count-based reward and combine D & E with the curiosity-driven exploration method ICM
Revised: In order to demonstrate the generality of D & E and compare with baselines, we get rid of using count-based reward and combine D & E with the curiosity-driven exploration method ICM

Original: (Pathak et al., 2017).
Revised: (Pathak et al., 2017).

Original: Every agent has its independent
Revised: Every agent has its independent

Original: ICM which calculates intrinsic reward based on transition ( s t , a t , s t +1 ) .
Revised: ICM which calculates intrinsic reward based on transition ( s t , a t , s t +1 ) .

Original: For each agent, the ICM is trained together with its policy and shared across all the agents.
Revised: For each agent, the ICM is trained together with its policy and shared across all the agents.

Original: We set the learnable reward model f
Revised: We set the learnable reward model f

Original: i mentioned in Eq.1 as the ICM model, following ICM ’s deﬁnition with normalization.
Revised: i mentioned in Eq.1 as the ICM model, following ICM ’s deﬁnition with normalization.

Original: Training and Results
Revised: Training and Results

Original: Following MiniGrid experiments, we train D & E in VizDoom with minor changes.
Revised: Following MiniGrid experiments, we train D & E in VizDoom with minor changes.

Original: On one hand, we use the same network architecture of actor and critic with different state input inVizDoom.
Revised: We use the same network architecture of actor and critic with different state input in

Original: On the other hand, we replace count-based exploration with ICM using the original implementation.
Revised: VizDoom, but replace count-based exploration with ICM using the original implementation.

Original: The hyperparameters are reported in Appendix A. We list the comparison results in Table 2 and D & E outperforms other baselines in any selected timesteps.
Revised: The hyperparameters are reported in Appendix A. We list the comparison results in Table 2 and D & E outperforms other baselines in any selected timesteps.

Original: Interestingly, while ICM is slow to solve the task, D & E learns fast and achieves high performance.
Revised: Interestingly, while ICM is slow to solve the task, D & E learns fast and achieves high performance.

Original: Even when we compare D & E at 2M timesteps (6M timesteps in total agents) with ICM at 6M timesteps, the performance of D & E is still better.
Revised: Even when we compare D & E at 2M timesteps (6M timesteps in total agents) with ICM at 6M timesteps, the performance of D & E is still better.

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: In the ablation study, we aim to answer following questions: 1) Is D & E capable to guide agents to explore different regions of state space separately?
Revised: In the ablation study, we aim to answer following questions: 1) Is D & E capable to guide agents to explore different regions of state space separately?

Original: Compared to single-agent setting, is D & E’s learning mechanism a key reason to improve exploration efﬁciency?
Revised: Compared to single-agent setting, is D & E’s learning mechanism a key reason to improve exploration efﬁciency?

Original: How the number of agents inﬂuence the performance?
Revised: How the number of agents inﬂuence the performance?

Original: For question 1, we train D & E on the KeyCorridorS6R3 task with singleton environment, and the number of agents is 3.
Revised: For question 1, we train D & E on the KeyCorridorS6R3 task with singleton environment, and the number of agents is 3.

Original: The exploration process of different agents is shown in Fig.4, in which state visitation count in historical trajectories is accumulated and visualized in the form of heatmap.
Revised: The exploration process of different agents is shown in Fig.4, in which state visitation count in historical trajectories is accumulated and visualized in the form of heatmap.

Original: As we can see in the ﬁgure, it is clear that three agents are exploring different regions of the environment separately, which validates the learning mechanism of D & E.
Revised: As we can see in the ﬁgure, it is clear that three agents are exploring different regions of the environment separately, which validates the learning mechanism of D & E.

Original: For question 2 and 3, we train D & E on the KeyCorridorS6R3 task with various number of agents in singleton environment.
Revised: For question 2 and 3, we train D & E on the KeyCorridorS6R3 task with various number of agents in singleton environment.

Original: The left ﬁgure in Fig.5 shows the best agent’s learning curve under different agent number setting, which conﬁrms that the number of agent brings beneﬁts to the training efﬁciency of the algorithm.
Revised: The left ﬁgure in Fig.5 shows the best agent’s learning curve under different agent number setting, which conﬁrms that the number of agent brings beneﬁts to the training efﬁciency of the algorithm.

Original: In fact, if the number of agents is one, the method is a conventional count-based method.
Revised: In fact, if the number of agents is one, the method is a conventional count-based method.

Original: On the other hand, as shown in the right ﬁgure in Fig.5, we also plot the number of training timesteps required when the return of episode reaches 0.8 and 0.9 for the ﬁrst time.
Revised: On the other hand, as shown in the right ﬁgure in Fig.5, we also plot the number of training timesteps required when the return of episode reaches 0.8 and 0.9 for the ﬁrst time.

Original: While the more agents, the less time algorithm needs to be trained, it shows a diminishing marginal effect.
Revised: While the more agents, the less time algorithm needs to be trained, it shows a diminishing marginal effect.

Original: Thus, the number of agents should be chosen according to the task.
Revised: Thus, the number of agents should be chosen according to the task.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We proposed a brand new algorithm named Divide-and-Explore(D & E) to handle the hard exploration problem in reinforcement learning.
Revised: We proposed a brand new algorithm named Divide-and-Explore(D & E) to handle the hard exploration problem in reinforcement learning.

Original: Inspired by divide-and-conquer, D & E trains multiple concurrent exploring agents, and successfully guides each agent exploring different regions of state space with shared intrinsic motivations while keeping exploring the boundary, which produces stateof-the-art results.
Revised: Inspired by divide-and-conquer, D & E trains multiple concurrent exploring agents, and successfully guides each agent exploring different regions of state space with shared intrinsic motivations while keeping exploring the boundary, which produces stateof-the-art results.

Original: More generally, D & E can be seen as a framework, which can combine existing single-agent exploration methods, e.g. Count, ICM, RIDE and so on, to improve learning efﬁciency.
Revised: More generally, D & E can be seen as a framework, which can combine existing single-agent exploration methods, e.g. Count, ICM, RIDE and so on, to improve learning efﬁciency.

Original: In the future work, we’d like to extend this idea to large scale setting and test it on general RL benchmarks like atari games (Mnih et al., 2015) and MuJoCo tasks (Todorov et al., 2012).
Revised: In the future work, we’d like to extend this idea to large scale setting and test it on general RL benchmarks like atari games (Mnih et al., 2015) and MuJoCo tasks (Todorov et al., 2012).

Original: And we will also explore the application of D & E on 3D navigation in robotics (Zhelo et al., 2018) or playtesting coverage in games (Gordillo et al., 2021).
Revised: And we will also explore the application of D & E on 3D navigation in robotics (Zhelo et al., 2018) or playtesting coverage in games (Gordillo et al., 2021).

Original: A H YPERPARAMETERS
Revised: A H YPERPARAMETERS

Original: The hyperparameters of D & E used in both MiniGrid and VizDoom environments are listed in Table 3.
Revised: The hyperparameters of D & E used in both MiniGrid and VizDoom environments are listed in Table 3.

Original: For Count, RND, ICM, RIDE, we conduct experiments based on the authors’ implementation 1 , while the policy networks are modiﬁed to be the same as D & E. And for other algorithms, we use the authors’ implementations with default hyperparameters, e
Revised: For Count, RND, ICM, RIDE, we conduct experiments based on the authors’ implementation 1 , while the policy networks are modiﬁed to be the same as D & E. And for other algorithms, we use the authors’ implementations with default hyperparameters, e

Original: [Table]
Revised: [Figure]

Original: g. AMIGo 2 , RAPID 3 and AGAC 4 .
Revised: g. AMIGo 2 , RAPID 3 and AGAC 4 .

Original: [Table]
Revised: [Table]

Original: B A DDITIONAL E XPERIMENTS B. HARDER EXPLORATION TASK
Revised: B A DDITIONAL E XPERIMENTS B. HARDER EXPLORATION TASK

Original: There are many harder exploration tasks in MiniGrid that have not been well resolved in previous studies, for example, MiniGrid-KeyCorridorS10R4-v0.
Revised: There are many harder exploration tasks in MiniGrid that have not been well resolved in previous studies, for example, MiniGrid-KeyCorridorS10R4-v0.

Original: The difﬁculty of this task comes from three perspectives.
Revised: The difﬁculty of this task comes from three perspectives.

Original: Firstly, because the agent can only observe a small range of 7 × 7 , large map will confuse corridors and rooms, making it difﬁcult for the agent to know about its own location through observation.
Revised: Firstly, because the agent can only observe a small range of 7 × 7 , large map will confuse corridors and rooms, making it difﬁcult for the agent to know about its own location through observation.

Original: Secondly, the larger environment makes agent’s trajectory longer and the rewards become more sparse, and this reinforces agent’s dependence on intrinsic rewards during training.
Revised: Secondly, the larger environment makes agent’s trajectory longer and the rewards become more sparse [Figure] and this reinforces agent’s dependence on intrinsic rewards during training.

Original: Thirdly, the larger environment makes the connection between rooms more complex, and agent often needs to go through several rooms to ﬁnd the key and the object.
Revised: Thirdly, the larger environment makes the connection between rooms more complex, and agent often needs to go through several rooms to ﬁnd the key and the object.

Original: However, D & E achieves amazing performance as shown in Fig.6, which shows that D & E does enhance ability to solve problems rather than simply improve efﬁciency.
Revised: However, D & E achieves amazing performance as shown in Fig.6, which shows that D & E does enhance ability to solve problems rather than simply improve efﬁciency.

Revised: B. E XPERIMENT DETAILS

Revised: To the results mentioned in Fig.2, we plot the best performance in several independent runs of baseline algorithms, to illustrate the beneﬁts are mainly from the particular intrinsic motivation.

Revised: As shown in Fig.7, compared with independent runs, the interact between different agents and intrinsic reward calculated from all agents’ replay buffer indeed accelerate the exploration efﬁciency.

Original: B. ABLATION EXPERIMENTS FOR DECAY METHOD
Revised: B. ABLATION EXPERIMENTS FOR DECAY METHOD

Original: As a supplement, we also compare the effects of different intrinsic reward weight decay methods.
Revised: As a supplement, we also compare the effects of different intrinsic reward weight decay methods.

Original: We compare the conventional linear decay and adaptive decay in the MiniGrid-KeyCorridorS6R3-v task.
Revised: We compare the conventional linear decay and adaptive decay in the MiniGrid-KeyCorridorS6R3-v task.

Original: The intrinsic reward weight in the former is calculated as:
Revised: The intrinsic reward weight in the former is calculated as:

Original: [Equation]
Revised: [Equation]

Original:  while in the latter it is calculated as Eq.4.
Revised: (7) while in the latter it is calculated as Eq.4.

Original: The results are shown in the left ﬁgure of Fig.7, which validates the beneﬁts of the adaptive decay method.
Revised: The results are shown in the left ﬁgure of Fig.8, which validates the beneﬁts of the adaptive decay method.

Original: Considering the historical episodes’ rewards, agent can make trade-off between exploration and exploitation better, and reduce exploration when it has found good trajectories and obtained high extrinsic rewards, which is similar with human decision.
Revised: Considering the historical episodes’ rewards, agent can make trade-off between exploration and exploitation better, and reduce exploration when it has found good trajectories and obtained high extrinsic rewards, which is similar with human decision.

Original: We plot the curve of decay factor d t , and there are three phases in the training process.
Revised: We plot the curve of decay factor d t , and there are three phases in the training process.

Original: At the beginning, agent rarely ﬁnd a trajectory to complete the task and the decay factor d t is large which means that agent should pay more attention to intrinsic reward and explore as much as possible.
Revised: At the beginning, agent rarely ﬁnd a trajectory to complete the task and the decay factor d t is large which means that agent should pay more attention to intrinsic reward and explore as much as possible.

Original: When train timesteps is larger than 2M, agent has an increasingly high probability of completing tasks, and d t decreases fast.
Revised: When train timesteps is larger than 2M, agent has an increasingly high probability of completing tasks, and d t decreases fast.

Original: After 10M training, agent’s policy converges and it does not need to explore.
Revised: After 10M training, agent’s policy converges and it does not need to explore.