Original: H UMAN - LEVEL A TARI 200 X FASTER
Revised: H UMAN - LEVEL A TARI 200 X FASTER

Original: Anonymous authors Paper under double-blind review
Revised: DeepMind, *Equal contribution {skapturowski,camunez,rayjiang,rakicevic,hado, cblundell,adriap}@deepmind.com

Original: A BSTRACT
Revised: A BSTRACT

Original: The task of building general agents that perform well over a wide range of tasks has been an important goal in reinforcement learning since its inception.
Revised: The task of building general agents that perform well over a wide range of tasks has been an important goal in reinforcement learning since its inception.

Original: The problem has been subject of research of a large body of work, with performance frequently measured by observing scores over the wide range of environments contained in the
Revised: The problem has been subject of research of a large body of work, with performance frequently measured by observing scores over the wide range of environments contained in the

Original: Atari 57 benchmark.
Revised: Atari 57 benchmark.

Original: Agent57 was the first agent to surpass the human benchmark on all 57 games, but this came at the cost of poor data-efficiency, requiring nearly 80 billion frames of experience to achieve.
Revised: Agent57 was the first agent to surpass the human benchmark on all 57 games, but this came at the cost of poor data-efficiency, requiring nearly 80 billion frames of experience to achieve.

Original: Taking Agent57 as a starting point, we employ a diverse set of strategies to achieve a 200-fold reduction of experience needed for all games to outperform the human baseline within our novel agent
Revised: Taking Agent57 as a starting point, we employ a diverse set of strategies to achieve a 200-fold reduction of experience needed for all games to outperform the human baseline within our novel agent

Original: MEME.
Revised: MEME.

Original: We investigate a range of instabilities and bottlenecks we encountered while reducing the data regime, and propose effective solutions to build a more robust and efficient agent.
Revised: We investigate a range of instabilities and bottlenecks we encountered while reducing the data regime, and propose effective solutions to build a more robust and efficient agent.

Original: We also demonstrate competitive performance with high-performing methods such as Muesli and MuZero.
Revised: We also demonstrate competitive performance with high-performing methods such as Muesli and MuZero.

Original: Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under a rapidly-changing policy.
Revised: Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under a rapidly-changing policy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: To develop generally capable agents, the question of how to evaluate them is paramount.
Revised: To develop generally capable agents, the question of how to evaluate them is paramount.

Original: The Arcade
Revised: The Arcade

Original: Learning Environment (ALE) (Bellemare et al., 2013) was introduced as a benchmark to evaluate agents on an diverse set of tasks which are interesting to humans, and developed externally to the
Revised: Learning Environment (ALE) (Bellemare et al., 2013) was introduced as a benchmark to evaluate agents on an diverse set of tasks which are interesting to humans, and developed externally to the

Original: Reinforcement Learning (RL) community.
Revised: Reinforcement Learning (RL) community.

Original: As a result, several games exhibit reward structures which are highly adversarial to many popular algorithms.
Revised: As a result, several games exhibit reward structures which are highly adversarial to many popular algorithms.

Original: Mean and median human normalized scores (HNS) (Mnih et al., 2015) over all games in the ALE have become standard metrics for evaluating deep RL agents.
Revised: Mean and median human normalized scores (HNS) (Mnih et al., 2015) over all games in the ALE have become standard metrics for evaluating deep RL agents.

Original: Recent progress has allowed state-of-the-art algorithms to greatly exceed average human-level performance on a large fraction of the games (Van Hasselt et al., 2016; Espeholt et al., 2018; Schrittwieser et al., 2020).
Revised: Recent progress has allowed state-of-the-art algorithms to greatly exceed average human-level performance on a large fraction of the games (Van Hasselt et al., 2016; Espeholt et al., 2018; Schrittwieser et al., 2020).

Original: However, it has been argued that mean or median HNS might not be well suited to assess generality because they tend to ignore the tails of the distribution (Badia et al., 2019).
Revised: However, it has been argued that mean or median HNS might not be well suited to assess generality because they tend to ignore the tails of the distribution (Badia et al., 2019).

Original: Indeed, most state-of-the-art algorithms achieve very high scores by performing very well on most games, but completely fail to learn on a small number of them.
Revised: Indeed, most state-of-the-art algorithms achieve very high scores by performing very well on most games, but completely fail to learn on a small number of them.

Original: Agent57 (Badia et al., 2020) was the first algorithm to obtain above human-average scores on all
Revised: Agent57 (Badia et al., 2020) was the first algorithm to obtain above human-average scores on all

Original: Atari games.
Revised: Atari games.

Original: However, such generality came at the cost of data efficiency; requiring tens of billions of environment interactions to achieve above average-human performance in some games, reaching a figure of 78 billion frames before beating the human benchmark in all games.
Revised: However, such generality came at the cost of data efficiency; requiring tens of billions of environment interactions to achieve above average-human performance in some games, reaching a figure of 78 billion frames before beating the human benchmark in all games.

Original: Data efficiency remains a desirable property for agents to possess, as many real-world challenges are data-limited by time and cost constraints (Dulac-Arnold et al., 2019).
Revised: Data efficiency remains a desirable property for agents to possess, as many real-world challenges are data-limited by time and cost constraints (Dulac-Arnold et al., 2019).

Original: In this work, we develop an agent that is as general as
Revised: In this work, we develop an agent that is as general as

Original: Agent

Revised: Agent

Original: but that requires only a fraction of the environment interactions to achieve the same result.
Revised: but that requires only a fraction of the environment interactions to achieve the same result.

Original: There exist two main trends in the literature when it comes to measuring improvements in the learning capabilities of agents.
Revised: There exist two main trends in the literature when it comes to measuring improvements in the learning capabilities of agents.

Original: One approach consists in measuring performance after a limited budget of interactions with the environment.
Revised: One approach consists in measuring performance after a limited budget of interactions with the environment.

Original: While this type of evaluation has led to important progress (Espeholt et al., 2018; van Hasselt et al., 2019; Hessel et al., 2021), it tends to disregard problems which are considered too hard to be solved within the allowed budget (Kaiser et al., 2019).
Revised: While this type of evaluation has led to important progress (Espeholt et al., 2018; van Hasselt et al., 2019; Hessel et al., 2021), it tends to disregard problems which are considered too hard to be solved within the allowed budget (Kaiser et al., 2019).

Original: On the other hand, one can aim to achieve a target end-performance with as few interactions aspossible (Silver et al., 2017; 2018; Schmitt et al., 2020).
Revised: On the other hand, one can aim to achieve a target end-performance with as few interactions as

Original: Since our goal is to show that our new agent is as general as Agent57, while being more data-efficient, we focus on the latter approach.

Original: S k ii n g
Revised: S k ii n g

Original: P r i v a t e E y e P
Revised: P r i v a t e E y e P

Original: i
Revised: i

Original: t
Revised: t

Original: f a ll !
Revised: f a ll !

Original: M o n t e z u m a ' s R e v e n g e
Revised: M o n t e z u m a ' s R e v e n g e

Original: S u rr o un d
Revised: S u rr o un d

Original: A s t e r o i d s
Revised: A s t e r o i d s

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: S o l a r i s B o w li n g G r a v
Revised: S o l a r i s B o w li n g G r a v

Original: i t a r H .
Revised: i t a r H .

Original: E .
Revised: E .

Original: R .
Revised: R .

Original: O .
Revised: O .

Original: B e r z e r k F r o s t b i t e T e nn i s A li e n
Revised: B e r z e r k F r o s t b i t e T e nn i s A li e n

Original: T u t a n k h a m
Revised: T u t a n k h a m

Original: R i v e rr a i d
Revised: R i v e rr a i d

Original: M s .
Revised: M s .

Original: P a c - M a n P o n g
Revised: P a c - M a n P o n g

Original: B e a m R i
Revised: B e a m R i

Original: d
Revised: d

Original: e r
Revised: e r

Original: C h
Revised: C h

Original: o pp e r
Revised: o pp e r

Original: C
Revised: C

Original: o mm a n d
Revised: o mm a n d

Original: S e a q u e s t
Revised: S e a q u e s t

Original: D o u b
Revised: D o u b

Original: l
Revised: l

Original: e
Revised: e

Original: D un k
Revised: D un k

Original: F r ee w a y
Revised: F r ee w a y

Original: Sp a c e I n v a d e r s
Revised: Sp a c e I n v a d e r s

Original: B a n k H
Revised: B a n k H

Original: e
Revised: e

Original: i s t
Revised: i s t

Original: V e n t
Revised: V e n t

Original: u
Revised: u

Original: r e
Revised: r e

Original: B a tt l e Z o n e
Revised: B a tt l e Z o n e

Original: A s t e r i x Q * B E R T A
Revised: A s t e r i x Q * B E R T A

Original: m i d a r
Revised: m i d a r

Original: N a m e T h i s
Revised: N a m e T h i s

Original: G a m e
Revised: G a m e

Original: Y a r s ' R e v e n g e
Revised: Y a r s ' R e v e n g e

Original: P h
Revised: P h

Original: o e n i x
Revised: o e n i x

Original: T i m e P il o t
Revised: T i m e P il o t

Original: D e m o n A tt a c k
Revised: D e m o n A tt a c k

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: B r e a k o u t
Revised: B r e a k o u t

Original: V i
Revised: V i

Original: d e o P i n b a ll
Revised: d e o P i n b a ll

Original: F i s h
Revised: F i s h

Original: i n g
Revised: i n g

Original: D e r b y E n d u r o
Revised: D e r b y E n d u r o

Original: S t a r g unn e r
Revised: S t a r g unn e r

Original: A ss a u l t
Revised: A ss a u l t

Original: Z a xx o n
Revised: Z a xx o n

Original: J a m e s b o n d
Revised: J a m e s b o n d

Original: C e n t
Revised: C e n t

Original: i p e d e
Revised: i p e d e

Original: K a n g a r oo A t l a n t i s
Revised: K a n g a r oo A t l a n t i s

Original: U p ' n D o w n
Revised: U p ' n D o w n

Original: D e f e n d e r B
Revised: D e f e n d e r B

Original: o
Revised: o

Original: x
Revised: x

Original: i n g R o b o t a n k
Revised: i n g R o b o t a n k

Original: K un g - F u
Revised: K un g - F u

Original: M a s t e r G o p h e r
Revised: M a s t e r G o p h e r

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: C r a z y C li m b e r K r u ll 1M [Figure] 100M 
Revised: C r a z y C li m b e r K r u ll 1M 10M 100M 1B 10B 100B

Original: [Figure]

Original: 10B 100B

Original: F r a m
Revised: F r a m

Original: e s t o > hu
Revised: e s t o > hu

Original: m a n s c o r e 390M200M
Revised: m a n s c o r e 390M200M

Original: Agent
Revised: Agent

Original: MEME 
Revised: MEME possible (Silver et al., 2017; 2018; Schmitt et al., 2020). Since our goal is to show that our new agent is as general as Agent57, while being more data-efficient, we focus on the latter approach.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: Building off Agent57, we carefully examine bottlenecks which slow down learning and address instabilities that arise when these bottlenecks are removed.
Revised: Building off Agent57, we carefully examine bottlenecks which slow down learning and address instabilities that arise when these bottlenecks are removed.

Original: We propose a novel agent that we call MEME, for MEME is an Efficient Memory-based
Revised: We propose a novel agent that we call MEME, for MEME is an Efficient Memory-based

Original: Exploration agent , which introduces solutions to enable taking advantage of three approaches that would otherwise lead to instabilities: training the value functions of the whole family of policies from Agent57 in parallel, on all policies’ transitions (instead of just the behaviour policy transitions), bootstrapping from the online network, and using high replay ratios.
Revised: Exploration agent , which introduces solutions to enable taking advantage of three approaches that would otherwise lead to instabilities: training the value functions of the whole family of policies from Agent57 in parallel, on all policies’ transitions (instead of just the behaviour policy transitions), bootstrapping from the online network, and using high replay ratios.

Original: These solutions include carefully normalising value functions with differing scales, as well as replacing the Retrace update target (Munos et al., 2016) with a soft variant of Watkins’ Q( λ ) (Watkins & Dayan, 1992) that enables faster signal propagation by performing less aggressive trace-cutting, and introducing a trust-region for value updates.
Revised: These solutions include carefully normalising value functions with differing scales, as well as replacing the Retrace update target (Munos et al., 2016) with a soft variant of Watkins’ Q( λ ) (Watkins & Dayan, 1992) that enables faster signal propagation by performing less aggressive trace-cutting, and introducing a trust-region for value updates.

Original: Moreover, we explore several recent advances in deep learning and determine which of them are beneficial for non-stationary problems like the ones considered in this work.
Revised: Moreover, we explore several recent advances in deep learning and determine which of them are beneficial for non-stationary problems like the ones considered in this work.

Original: Finally, we examine approaches to robustify performance by introducing a policy distillation mechanism that learns a policy head based on the actions obtained from the value network without being sensitive to value magnitudes.
Revised: Finally, we examine approaches to robustify performance by introducing a policy distillation mechanism that learns a policy head based on the actions obtained from the value network without being sensitive to value magnitudes.

Original: Our agent outperforms the human baseline across all 57 Atari games in 390M frames, using two orders of magnitude fewer interactions with the environment than
Revised: Our agent outperforms the human baseline across all 57 Atari games in 390M frames, using two orders of magnitude fewer interactions with the environment than

Original: Agent57 as shown in Figure 1.
Revised: Agent57 as shown in Figure 1.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Large scale distributed agents have exhibited compelling results in recent years.
Revised: Large scale distributed agents have exhibited compelling results in recent years.

Original: Actor-critic (Espeholt et al., 2018; Song et al., 2020) as well as value-based agents (Horgan et al., 2018; Kapturowski et al., 2018) demonstrated strong performance in a wide-range of environments, including the Ataribenchmark.
Revised: Actor-critic (Espeholt et al., 2018; Song et al., 2020) as well as value-based agents (Horgan et al., 2018; Kapturowski et al., 2018) demonstrated strong performance in a wide-range of environments, including the Ataribenchmark.

Original: Moreover, approaches such as evolutionary strategies (Salimans et al., 2017) and large scale genetic algorithms (Such et al., 2017) presented alternative learning algorithms that achieve competitive results on Atari.
Revised: Moreover, approaches such as evolutionary strategies (Salimans et al., 2017) and large scale genetic algorithms (Such et al., 2017) presented alternative learning algorithms that achieve competitive results on Atari.

Original: Finally, search-augmented distributed agents (Schrittwieser et al., 2020;
Revised: Finally, search-augmented distributed agents (Schrittwieser et al., 2020;

Original: Hessel et al., 2021) also hold high performance across many different tasks, and concretely they hold the highest mean and median human normalized scores over the 57 Atari games.
Revised: Hessel et al., 2021) also hold high performance across many different tasks, and concretely they hold the highest mean and median human normalized scores over the 57 Atari games.

Original: However, all these methods show the same failure mode: they perform poorly in hard exploration games, such as
Revised: However, all these methods show the same failure mode: they perform poorly in hard exploration games, such as

Original: Pitfall! , and Montezuma’s Revenge .
Revised: Pitfall! , and Montezuma’s Revenge .

Original: In contrast, Agent57 (Badia et al., 2020) surpassed the human benchmark on all 57 games, showing better general performance.
Revised: In contrast, Agent57 (Badia et al., 2020) surpassed the human benchmark on all 57 games, showing better general performance.

Original: Go-Explore (Ecoffet et al., 2021) similarly achieved such general performance, by relying on coarse-grained state representations via a downscaling function that is highly specific to Atari.
Revised: Go-Explore (Ecoffet et al., 2021) similarly achieved such general performance, by relying on coarse-grained state representations via a downscaling function that is highly specific to Atari.

Original: Learning as much as possible from previous experience is key for data efficiency.
Revised: Learning as much as possible from previous experience is key for data efficiency.

Original: Since it is often desirable for approximate methods to make small updates to the policy (Kakade & Langford, 2002;
Revised: Since it is often desirable for approximate methods to make small updates to the policy (Kakade & Langford, 2002;

Original: Schulman et al., 2015), approaches have been proposed for enabling multiple learning steps over the same batch of experience in policy gradient methods to avoid collecting new transitions for every learning step (Schulman et al., 2017).
Revised: Schulman et al., 2015), approaches have been proposed for enabling multiple learning steps over the same batch of experience in policy gradient methods to avoid collecting new transitions for every learning step (Schulman et al., 2017).

Original: This decoupling between collecting experience and learning occurs naturally in off-policy learning agents with experience replay (Lin, 1992; Mnih et al.,  and Fitted Q Iteration methods (Ernst et al., 2005; Riedmiller, 2005).
Revised: This decoupling between collecting experience and learning occurs naturally in off-policy learning agents with experience replay (Lin, 1992; Mnih et al., 2015) and Fitted Q Iteration methods (Ernst et al., 2005; Riedmiller, 2005).

Original: Multiple approaches for making more efficient use of a replay buffer have been proposed, including prioritized sampling of transitions (Schaul et al., 2016), sharing experience across populations of agents (Schmitt et al., 2020), learning multiple policies in parallel from a single stream of experience (Riedmiller et al., 2018), or reanalyzing old trajectories with the most recent version of a learned model to generate new targets in model-based settings (Schrittwieser et al., 2020; 2021) or to re-evaluate goals (Andrychowicz et al., 2017).
Revised: Multiple approaches for making more efficient use of a replay buffer have been proposed, including prioritized sampling of transitions (Schaul et al., 2016), sharing experience across populations of agents (Schmitt et al., 2020), learning multiple policies in parallel from a single stream of experience (Riedmiller et al., 2018), or reanalyzing old trajectories with the most recent version of a learned model to generate new targets in model-based settings (Schrittwieser et al., 2020; 2021) or to re-evaluate goals (Andrychowicz et al., 2017).

Original: The ATARI100k benchmark (Kaiser et al., 2019) was introduced to observe progress in improving the data efficiency of reinforcement learning agents, by evaluating game scores after 100k agent steps (400k frames).
Revised: The ATARI100k benchmark (Kaiser et al., 2019) was introduced to observe progress in improving the data efficiency of reinforcement learning agents, by evaluating game scores after 100k agent steps (400k frames).

Original: Work on this benchmark has focused on leveraging the use of models (Ye et al., 2021;
Revised: Work on this benchmark has focused on leveraging the use of models (Ye et al., 2021;

Original: Kaiser et al., 2019; Long et al., 2022), unsupervised learning (Hansen et al., 2019; Schwarzer et al., 2021; Srinivas et al., 2020; Liu & Abbeel, 2021), or greater use of replay data (van Hasselt et al., 2019; Kielak, 2020) or augmentations (Kostrikov et al., 2021;
Revised: Kaiser et al., 2019; Long et al., 2022), unsupervised learning (Hansen et al., 2019; Schwarzer et al., 2021; Srinivas et al., 2020; Liu & Abbeel, 2021), or greater use of replay data (van Hasselt et al., 2019; Kielak, 2020) or augmentations (Kostrikov et al., 2021;

Original: Schwarzer et al., 2021)
Revised: Schwarzer et al., 2021)

Original: While we consider this to be an important line of research, this tight budget produces an incentive to focus on a subset of games where exploration is easier, and it is unclear some games can be solved from scratch with such a small data budget.
Revised: While we consider this to be an important line of research, this tight budget produces an incentive to focus on a subset of games where exploration is easier, and it is unclear some games can be solved from scratch with such a small data budget.

Original: Such a setting is likely to prevent any meaningful learning on hard-exploration games, which is in contrast with the goal of our work.
Revised: Such a setting is likely to prevent any meaningful learning on hard-exploration games, which is in contrast with the goal of our work.

Original:  B ACKGROUND : A GENT
Revised:  B ACKGROUND : A GENT

Original: Our work builds on top of Agent57, which combines three main ideas: ( i ) a distributed deep
Revised: Our work builds on top of Agent57, which combines three main ideas: ( i ) a distributed deep

Original: RL framework based on Recurrent Replay Distributed DQN (R2D2) (Kapturowski et al., 2018), ( ii ) exploration with a family of policies and the Never Give Up (NGU) intrinsic reward (Badia et al., 2019), and ( iii ) a meta-controller that dynamically adjusts the discount factor and balances exploration and exploitation during training, by selecting from a family of policies.
Revised: RL framework based on Recurrent Replay Distributed DQN (R2D2) (Kapturowski et al., 2018), ( ii ) exploration with a family of policies and the Never Give Up (NGU) intrinsic reward (Badia et al., 2019), and ( iii ) a meta-controller that dynamically adjusts the discount factor and balances exploration and exploitation during training, by selecting from a family of policies.

Original: Below, we give a general introduction to the problem setting and some of the relevant components of Agent57.
Revised: Below, we give a general introduction to the problem setting and some of the relevant components of Agent57.

Original: Problem definition.
Revised: Problem definition.

Original: We consider the problem of discounted infinite-horizon RL in Markov Decision
Revised: We consider the problem of discounted infinite-horizon RL in Markov Decision

Original: Processes (MDP) (Puterman, 1994).
Revised: Processes (MDP) (Puterman, 1994).

Original: The goal is to find a policy π that maximises the expected sum of discounted future rewards,
Revised: The goal is to find a policy π that maximises the expected sum of discounted future rewards,

Original: E π [ (cid:80) t ≥ 0 γ t r t ] , where γ ∈ [0 , 1) is the discount factor, r t = r ( x t , a t ) is the reward at time t , x t is the state at time t, and a t ∼ π ( a | x t ) is the action generated by following some policy π .
Revised: E π [ (cid:80) t ≥ 0 γ t r t ] , where γ ∈ [0 , 1) is the discount factor, r t = r ( x t , a t ) is the reward at time t , x t is the state at time t, and a t ∼ π ( a | x t ) is the action generated by following some policy π .

Original: In the off-policy learning setting, data generated by a behavior policy µ is used to learn about the target policy π .
Revised: In the off-policy learning setting, data generated by a behavior policy µ is used to learn about the target policy π .

Original: This can be achieved by employing a variant of Q-learning (Watkins &
Revised: This can be achieved by employing a variant of Q-learning (Watkins &

Original: Dayan, 1992) to estimate the action-value function, Q π ( x, a ) =
Revised: Dayan, 1992) to estimate the action-value function, Q π ( x, a ) =

Original: E π [ (cid:80) t ≥ 

Revised: E π [ (cid:80) t ≥

Original: γ t r t | x t = x, a t = a ] .
Revised: γ t r t | x t = x, a t = a ] .

Original: The estimated action-value function can then be used to derive a new policy π ( a | x ) using the ϵ -greedy operator G ϵ (Sutton & Barto, 2018) 1 .
Revised: The estimated action-value function can then be used to derive a new policy π ( a | x ) using the ϵ -greedy operator G ϵ (Sutton & Barto, 2018) 1 .

Original: This new policy can then be used as the target policy for another iteration, repeating the process.
Revised: This new policy can then be used as the target policy for another iteration, repeating the process.

Original: Agent57 uses a deep neural network with parameters θ to estimate action-value functions, Q π ( x, a ; θ ) 2 , trained on return estimates G t derived with Retrace from sequences of off-policy data (Munos et al., 2016).
Revised: Agent57 uses a deep neural network with parameters θ to estimate action-value functions, Q π ( x, a ; θ ) 2 , trained on return estimates G t derived with Retrace from sequences of off-policy data (Munos et al., 2016).

Original: In order to stabilize learning, a target network is used for bootstrapping the return estimates using double Q-learning (Van Hasselt et al., 2016); the parameters of this target network, θ T , are periodically copied from the online network parameters θ (Mnih et al., 2015).
Revised: In order to stabilize learning, a target network is used for bootstrapping the return estimates using double Q-learning (Van Hasselt et al., 2016); the parameters of this target network, θ T , are periodically copied from the online network parameters θ (Mnih et al., 2015).

Original: Finally, a value-function transformation is used to compress the wide range of reward scales present in Atari, as in (Pohlen et al., 2018).
Revised: Finally, a value-function transformation is used to compress the wide range of reward scales present in Atari, as in (Pohlen et al., 2018).

Original: Distributed RL framework.
Revised: Distributed RL framework.

Original: Agent57 is a distributed deep RL agent based on R2D2 that decouples acting from learning.
Revised: Agent57 is a distributed deep RL agent based on R2D2 that decouples acting from learning.

Original: Multiple actors interact with independent copies of the environment and feed trajectories to a central replay buffer.
Revised: Multiple actors interact with independent copies of the environment and feed trajectories to a central replay buffer.

Original: A separate learning process obtains trajectories from this buffer using prioritized sampling and updates the neural network parameters to predict action-values at each state.
Revised: A separate learning process obtains trajectories from this buffer using prioritized sampling and updates the neural network parameters to predict action-values at each state.

Original: Actors obtain parameters from the learner periodically.
Revised: Actors obtain parameters from the learner periodically.

Original: See Appendix E for more details.
Revised: See Appendix E for more details.

Original: Exploration with NGU.
Revised: Exploration with NGU.

Original: Agent57 uses the Never Give Up (NGU) intrinsic reward to encourage exploration.
Revised: Agent57 uses the Never Give Up (NGU) intrinsic reward to encourage exploration.

Original: It aims at learning a family of N = 32 policies which maximize different weightings of the extrinsic reward given by the environment ( r e t ) and the intrinsic reward ( r i t ), r j,t = r e t + β j r i t ( β j ∈ R + , j ∈ { 0 , . ..
Revised: It aims at learning a family of N = 32 policies which maximize different weightings of the extrinsic reward given by the environment ( r et ) and the intrinsic reward ( r it ), r j,t = r et + β j r it ( β j ∈ R + , j ∈ { 0 , . .

Original: , N − 1 } ).
Revised: , N − 1 } ).

Original: The value of β j controls the degree of exploration, with higher valuesencouraging more exploratory behaviors, and each policy in the family is optimized with a different discount factor γ j . The Universal Value Function Approximators (UVFA) framework (Schaul et al.,
Revised: The value of β j controls the degree of exploration, with higher values

Original: Torso
Revised: Torso

Original: LSTM
Revised: LSTM

Original: Q-valueHead PolicyHead
Revised: Q PolicyHead

Original: Q j (a t , x t ) 𝛑 j (a t | x t )
Revised: Q j (a t , x t ) 𝛑 j (a t | x t )

Original: N-mixtures
Revised: N-mixtures

Original: Torso
Revised: Torso

Original: LSTM
Revised: LSTM

Original: Q-valueHead PolicyHead Q 1 (a t , x t ) 𝛑 1 (a t | x t ) Member policy
Revised: Q-valueHead PolicyHead Q 1 (a t , x t ) 𝛑 1 (a t | x t ) Member policy

Original: Q-valueHead PolicyHead Q N (a t , x t ) 𝛑 N (a t | x t ) Member policy 1 … …
Revised: Q-valueHead PolicyHead Q N (a t , x t ) 𝛑 N (a t | x t ) Member policy 1 … …

Original: DuelingHead
Revised: DuelingHead

Original: DuelingHead
Revised: DuelingHead

Original: Q i (a t , x t )
Revised: Q i (a t , x t )

Original: Q e (a t , x t ) +
Revised: Q e (a t , x t ) +

Original: Q j (a t , x t )
Revised: Q j (a t , x t )

Original: Q-value Head
Revised: Q-value Head

Original: N policy family members
Revised: N policy family members

Original: Torso LSTM
Revised: Torso LSTM

Original: Q-valueHead PolicyHead Q 1 (a t , x t ) 𝛑 1 (a t | x t ) N policy family members … Q N (a t , x t ) 𝛑 N (a t | x t )
Revised: Q-valueHead PolicyHead Q 1 (a t , x t ) 𝛑 1 (a t | x t ) N policy family members … Q N (a t , x t ) 𝛑 N (a t | x t )

Original: [Figure]
Revised: [Figure]

Original: Torso LSTM
Revised: Torso LSTM

Original: Q 1 (x t , a t ; θ ) 𝜋 1 (a t | x t )
Revised: Q 1 (x t , a t ; θ ) 𝜋 1 (a t | x t )

Original: N policy family members …
Revised: N policy family members …

Original: Q N (x t , a t ; θ ) 𝜋 N  t | x t )
Revised: Q N (x t , a t ; θ ) 𝜋 N (a t | x t )

Original: PolicyHead 
Revised: PolicyHead Q- encouraging more exploratory behaviors, and each policy in the family is optimized with a different discount factor γ j .

Original:  2015) is employed to efficiently learn Q π j ( x, a ; θ ) = 
Revised: The Universal Value Function Approximators (UVFA) framework (Schaul et al., 2015) is employed to efficiently learn Q π j ( x, a ; θ ) = E π j [ (cid:80) [Figure]

Original: E π j [ (cid:80) t ≥ 0 γ tj r j,t | x t = x, a t = a ] (we use a shorthand notation Q j ( x, a ; θ ) ) for j ∈ { 0 , . . .
Revised: [Equation] [Figure] ,t | x t = x, a t = a ] (we use a shorthand notation Q j ( x, a ; θ ) ) for j ∈ { 0 , . . .

Original: , N − 1 } using a single set of shared parameters θ .
Revised: , N − 1 } using a single set of shared parameters θ .

Original: The policy π j ( a | x ) can then be derived using the ϵ -greedy operator as G ϵ Q j ( x, a ; θ ) .
Revised: The policy π j ( a | x ) can then be derived using the ϵ -greedy operator as G ϵ Q j ( x, a ; θ ) .

Original: We refer the reader to Appendix H for more details.
Revised: We refer the reader to Appendix H for more details.

Original: Meta-controller.
Revised: Meta-controller.

Original: Agent57 introduces an adaptive meta-controller that decides which policies from the family of N policies to use for collecting experience based on their episodic returns.
Revised: Agent57 introduces an adaptive meta-controller that decides which policies from the family of N policies to use for collecting experience based on their episodic returns.

Original: This naturally creates a curriculum over β j and γ j by adapting their value throughout training.
Revised: This naturally creates a curriculum over β j and γ j by adapting their value throughout training.

Original: The optimization process is formulated as a non-stationary bandit problem.
Revised: The optimization process is formulated as a non-stationary bandit problem.

Original: A detailed description about the meta-controller implementation is provided in Appendix F.
Revised: A detailed description about the meta-controller implementation is provided in Appendix F.

Original: Q-function separation.
Revised: Q-function separation.

Original: The architecture of the Q-function in Agent57 is implemented as two separate networks in order to split the intrinsic and extrinsic components.
Revised: The architecture of the Q-function in Agent57 is implemented as two separate networks in order to split the intrinsic and extrinsic components.

Original: The network parameters of
Revised: The network parameters of

Original: Q j ( x, a ; θ e ) and Q j ( x, a ; θ i ) are separate and independently optimized with r ej and r ij , respectively.
Revised: Q j ( x, a ; θ e ) and Q j ( x, a ; θ i ) are separate and independently optimized with r ej and r ij , respectively.

Original: The main motivation behind this decomposition is to allow each network to adapt to the scale and variance associated with their corresponding reward, as well as preventing the gradients of the decomposed intrinsic and extrinsic value function heads from interfering with each other.
Revised: The main motivation behind this decomposition is to allow each network to adapt to the scale and variance associated with their corresponding reward, as well as preventing the gradients of the decomposed intrinsic and extrinsic value function heads from interfering with each other.

Original:  MEME: I MPROVING THE DATA EFFICIENCY OF A GENT
Revised:  MEME: I MPROVING THE DATA EFFICIENCY OF A GENT

Original: This section describes the main algorithmic contributions of the MEME agent, aimed at improving the data efficiency of Agent57.
Revised: This section describes the main algorithmic contributions of the MEME agent, aimed at improving the data efficiency of Agent57.

Original: These contributions aim to achieve faster propagation of learning signals related to rare events ( A ), stabilize learning under differing value scales ( B ), improve the neural network architecture ( C ), and make updates more robust under a rapidly-changing policy ( D ).
Revised: These contributions aim to achieve faster propagation of learning signals related to rare events ( A ), stabilize learning under differing value scales ( B ), improve the neural network architecture ( C ), and make updates more robust under a rapidly-changing policy ( D ).

Original: For clarity of exposition, we label the contributions according to the type of limitation they address.
Revised: For clarity of exposition, we label the contributions according to the type of limitation they address.

Original: A1 Bootstrapping with online network.
Revised: A1 Bootstrapping with online network.

Original: Target networks are frequently used in conjunction with value-based agents due to their stabilizing effect when learning from off-policy data (Mnih et al., 2015; Van Hasselt et al., 2016).
Revised: Target networks are frequently used in conjunction with value-based agents due to their stabilizing effect when learning from off-policy data (Mnih et al., 2015; Van Hasselt et al., 2016).

Original: This design choice places a fundamental restriction on how quickly changes in the Q-function are able to propagate.
Revised: This design choice places a fundamental restriction on how quickly changes in the Q-function are able to propagate.

Original: This issue can be mitigated to some extent by simply updating the target network more frequently, but the result is typically a less stable agent.
Revised: This issue can be mitigated to some extent by simply updating the target network more frequently, but the result is typically a less stable agent.

Original: To accelerate signal propagation while maintaining stability, we use online network bootstrapping, and we stabilize the learning by introducing an approximate trust region for value updates that allows us to filter which samples contribute to the loss.
Revised: To accelerate signal propagation while maintaining stability, we use online network bootstrapping, and we stabilize the learning by introducing an approximate trust region for value updates that allows us to filter which samples contribute to the loss.

Original: The trust region masks out the loss at any timestep for which both of the following conditions hold:
Revised: The trust region masks out the loss at any timestep for which both of the following conditions hold:

Original: [Equation]
Revised: [Equation]

Original: where α is a fixed hyperparameter, G t denotes the return estimate, θ and θ T denote the online and target parameters respectively, and σ j is the standard deviation of the TD-errors (a more precise description of which we defer until B1 ).
Revised: where α is a fixed hyperparameter, G t denotes the return estimate, θ and θ T denote the online and target parameters respectively, and σ j is the standard deviation of the TD-errors (a more precise description of which we defer until B1 ).

Original: Intuitively, we only mask if the current value of the online network is outside of the trust region (Equation 1) and the sign of the TD-error points away from the trust region (Equation 2), as depicted in Figure 3 in red.
Revised: Intuitively, we only mask if the current value of the online network is outside of the trust region (Equation 1) and the sign of the TD-error points away from the trust region (Equation 2), as depicted in Figure 3 in red.

Original: We note that a very similar trust region scheme is used for the value-function in most Proximal Policy Optimization (PPO) implementations (Schulman et al., 2017), though not described in the original paper.
Revised: We note that a very similar trust region scheme is used for the value-function in most Proximal Policy Optimization (PPO) implementations (Schulman et al., 2017), though not described in the original paper.

Original: In contrast, the
Revised: In contrast, the

Original: PPO version instead uses a constant threshold, and thus is not able to adapt to differing scales of value functions.
Revised: PPO version instead uses a constant threshold, and thus is not able to adapt to differing scales of value functions.

Original: [Figure]
Revised: [Figure]

Original: A2 Target computation with tolerance.
Revised: A2 Target computation with tolerance.

Original: Agent57 uses Retrace (Munos et al., 2016) to compute return estimates from off-policy data, but we observed that it tends to cut traces too aggressively when using ϵ -greedy policies thus slowing down the propagation of information into the value function.
Revised: Agent57 uses Retrace (Munos et al., 2016) to compute return estimates from off-policy data, but we observed that it tends to cut traces too aggressively when using ϵ -greedy policies thus slowing down the propagation of information into the value function.

Original: Preliminary experiments showed that data efficiency was improved in many dense-reward tasks when replacing Retrace with Peng’s Q( λ ) (Peng & Williams, 1994), but its lack of off-policy corrections tends to result in degraded performance as data becomes more off-policy (e.g. by increasing the expected number of times that a sequence is sampled from replay, or by sharing data across a family of policies).
Revised: Preliminary experiments showed that data efficiency was improved in many dense-reward tasks when replacing Retrace with Peng’s Q( λ ) (Peng & Williams, 1994), but its lack of off-policy corrections tends to result in degraded performance as data becomes more off-policy (e.g. by increasing the expected number of times that a sequence is sampled from replay, or by sharing data across a family of policies).

Original: This motivates us to propose an alternative return estimator, which we derive from
Revised: This motivates us to propose an alternative return estimator, which we derive from

Original: Q( λ ) (Watkins & Dayan, 1992):
Revised: Q( λ ) (Watkins & Dayan, 1992):

Original: [Equation]
Revised: [Equation]

Original: where (cid:81) k − 1 i =

Revised: where (cid:81) k − 1 i =

Original: λ i ∈
Revised: λ i ∈

Original: [0 , 1] effectively controls how much information from the future is used in the return estimation and is generally used as a trace cutting coefficient to perform off-policy correction.
Revised: [0 , 1] effectively controls how much information from the future is used in the return estimation and is generally used as a trace cutting coefficient to perform off-policy correction.

Original: Peng’s Q( λ ) does not perform any kind of off-policy correction and sets λ i
Revised: Peng’s Q( λ ) does not perform any kind of off-policy correction and sets λ i

Original: = λ , whereas Watkins’
Revised: = λ , whereas Watkins’

Original: Q( λ ) (Watkins & Dayan, 1992) aggressively cuts traces whenever it encounters an off-policy action by using λ i
Revised: Q( λ ) (Watkins & Dayan, 1992) aggressively cuts traces whenever it encounters an off-policy action by using λ i

Original: = λ I a i ∈ argmax a Q ( x i ,a ) , where I denotes the indicator function.
Revised: = λ I a i ∈ argmax a Q ( x i ,a ) , where I denotes the indicator function.

Original: We propose to use a softer trace cutting mechanism by adding a fixed tolerance parameter κ and taking the expectation of trace coefficients under π :
Revised: We propose to use a softer trace cutting mechanism by adding a fixed tolerance parameter κ and taking the expectation of trace coefficients under π :

Original: [Equation]
Revised: [Equation]

Original: Finally, we replace all occurrences of the max operator in Equation 3 with the expectation under π .
Revised: Finally, we replace all occurrences of the max operator in Equation 3 with the expectation under π .

Original: The resulting return estimator, which we denote Soft Watkins Q( λ ) , leads to more transitions being used and increased sample efficiency.
Revised: The resulting return estimator, which we denote Soft Watkins Q( λ ) , leads to more transitions being used and increased sample efficiency.

Original: Note that Watkins Q( λ ) is recovered when setting κ = 0 and π = G ( Q ) .
Revised: Note that Watkins Q( λ ) is recovered when setting κ = 0 and π = G ( Q ) .

Original: B1 Loss and priority normalization.
Revised: B1 Loss and priority normalization.

Original: As we learn a family of Q-functions which vary over a wide range of discount factors and intrinsic reward scales, we expect that the Q-functions will vary considerably in scale.
Revised: As we learn a family of Q-functions which vary over a wide range of discount factors and intrinsic reward scales, we expect that the Q-functions will vary considerably in scale.

Original: This may cause the larger-scale Q-values to dominate learning and destabilize learning of smaller Q-values.
Revised: This may cause the larger-scale Q-values to dominate learning and destabilize learning of smaller Q-values.

Original: This is a particular concern in environments with very small extrinsic reward scales.
Revised: This is a particular concern in environments with very small extrinsic reward scales.

Original: To counteract this effect we introduce a normalization scheme on the TD-errors similar to that used in Schaul et al.
Revised: To counteract this effect we introduce a normalization scheme on the TD-errors similar to that used in Schaul et al.

Original: Specifically, we compute a running estimate of the standard deviation of TD-errors of the online network σ running j as well as a batch standard deviation σ batch j , and compute σ j = max( σ running j , σ batch j , ϵ ) , where ϵ acts as small threshold to avoid amplification of noise past a specified scale, which we fix to 0 .
Revised: Specifically, we compute a running estimate of the standard deviation of TD-errors of the online network σ running j as well as a batch standard deviation σ batch j , and compute σ j = max( σ running j , σ batch j , ϵ ) , where ϵ acts as small threshold to avoid amplification of noise past a specified scale, which we fix to 0 .

Original: 01 in all our experiments.
Revised: 01 in all our experiments.

Original: We then divide the TD-errors by σ j when computing both the loss and priorities.
Revised: We then divide the TD-errors by σ j when computing both the loss and priorities.

Original: As opposed to Schaul et al.
Revised: As opposed to Schaul et al.

Original: we compute the running statistics on the learner, and we use importance sampling to correct the sampling distribution.
Revised: we compute the running statistics on the learner, and we use importance sampling to correct the sampling distribution.

Original: B2 Cross-mixture training.
Revised: B2 Cross-mixture training.

Original: Agent57 only trains the policy j which was used to collect a given trajectory, but it is natural to ask whether data efficiency and robustness may be improved by training all policies at once.
Revised: Agent57 only trains the policy j which was used to collect a given trajectory, but it is natural to ask whether data efficiency and robustness may be improved by training all policies at once.

Original: We propose a training loss L according to the following weighting scheme between the behavior policy loss and the mean over all losses:
Revised: We propose a training loss L according to the following weighting scheme between the behavior policy loss and the mean over all losses:

Original: [Equation]
Revised: [Equation]

Original: where L j denotes the Q-learning loss for policy j , and j µ denotes the index for the behavior policy selected by the meta-controller for the sampled trajectory.
Revised: where L j denotes the Q-learning loss for policy j , and j µ denotes the index for the behavior policy selected by the meta-controller for the sampled trajectory.

Original: We find that an intermediate value for the mixture parameter of η = 0 .
Revised: We find that an intermediate value for the mixture parameter of η = 0 .

Original: 5 tends to work well.
Revised: 5 tends to work well.

Original: To achieve better compute-efficiency we choose to deviate from the original UVFA architecture which fed a 1-hot encoding of the policy index to the LSTM, and instead modify the Q-value heads to output N sets of Q-values, one for each of the members in the family of policies introduced in Section 3.
Revised: To achieve better compute-efficiency we choose to deviate from the original UVFA architecture which fed a 1-hot encoding of the policy index to the LSTM, and instead modify the Q-value heads to output N sets of Q-values, one for each of the members in the family of policies introduced in Section 3.

Original: Therefore, in the end we output values for all combinations of actions and policies (see Figure 2).
Revised: Therefore, in the end we output values for all combinations of actions and policies (see Figure 2).

Original: We note that in this setting, there is also less deviation in the recurrent states when learning across different mixture policies.
Revised: We note that in this setting, there is also less deviation in the recurrent states when learning across different mixture policies.

Original: C1 Normalizer-free torso network.
Revised: C1 Normalizer-free torso network.

Original: Normalization layers are a common feature of ResNet architectures, and which are known to aid in training of very deep networks, but preliminary investigation revealed that several commonly used normalization layers are in fact detrimental to performance in our setting.
Revised: Normalization layers are a common feature of ResNet architectures, and which are known to aid in training of very deep networks, but preliminary investigation revealed that several commonly used normalization layers are in fact detrimental to performance in our setting.

Original: Instead, we employ a variation of the NFNet architecture (Brock et al., 2021) for our policy torso network, which combines a variance-scaling strategy with scaled weight standardization and careful initialization to achieve state-of-the-art performance on ImageNet without the need for normalization layers.
Revised: Instead, we employ a variation of the NFNet architecture (Brock et al., 2021) for our policy torso network, which combines a variance-scaling strategy with scaled weight standardization and careful initialization to achieve state-of-the-art performance on ImageNet without the need for normalization layers.

Original: We adopt their use of stochastic depth (Huang et al., 2016) at training-time but omit the application of ordinary dropout to fully-connected layers as we did not observe any benefit from this form of regularization.
Revised: We adopt their use of stochastic depth (Huang et al., 2016) at training-time but omit the application of ordinary dropout to fully-connected layers as we did not observe any benefit from this form of regularization.

Original: Some care is required when using stochastic depth in conjunction with multi-step returns, as resampling of the stochastic depth mask at each timestep injects additional noise into the bootstrap values, resulting in a higher-variance return estimator.
Revised: Some care is required when using stochastic depth in conjunction with multi-step returns, as resampling of the stochastic depth mask at each timestep injects additional noise into the bootstrap values, resulting in a higher-variance return estimator.

Original: As such, we employ a temporally-consistent stochastic depth mask which remains fixed over the length of each training trajectory.
Revised: As such, we employ a temporally-consistent stochastic depth mask which remains fixed over the length of each training trajectory.

Original: C2 Shared torso with combined loss.
Revised: C2 Shared torso with combined loss.

Original: Agent57 decomposes the combined Q-function into intrinsic and extrinsic components, Q e and Q i , which are represented by separate networks.
Revised: Agent57 decomposes the combined Q-function into intrinsic and extrinsic components, Q e and Q i , which are represented by separate networks.

Original: Such a decomposition prevents the gradients of the decomposed value functions from interfering with each other.
Revised: Such a decomposition prevents the gradients of the decomposed value functions from interfering with each other.

Original: This interference may occur in environments where the intrinsic reward is poorly aligned with the task objective, as defined by the extrinsic reward.
Revised: This interference may occur in environments where the intrinsic reward is poorly aligned with the task objective, as defined by the extrinsic reward.

Original: However, the choice to use separate separate networks comes at an expensive computational cost, and potentially limits sample-efficiency since generic low-level features cannot be shared.
Revised: However, the choice to use separate separate networks comes at an expensive computational cost, and potentially limits sample-efficiency since generic low-level features cannot be shared.

Original: To alleviate these issues, we introduce a shared torso for the two
Revised: To alleviate these issues, we introduce a shared torso for the two

Original: Q-functions while retaining separate heads.
Revised: Q-functions while retaining separate heads.

Original: While the form of the decomposition in Agent57 was chosen so as to ensure convergence to the optimal value-function Q ⋆ in the tabular setting, this does not generally hold under function approximation.
Revised: While the form of the decomposition in Agent57 was chosen so as to ensure convergence to the optimal value-function Q ⋆ in the tabular setting, this does not generally hold under function approximation.

Original: Comparing the combined and decomposed losses we observe a mismatch in the gradients due to the absence of cross-terms Q i ( θ ) ∂Q
Revised: Comparing the combined and decomposed losses we observe a mismatch in the gradients due to the absence of cross-terms Q i ( θ ) ∂Q

Original: e ( θ ) ∂θ and Q e ( θ ) ∂Q i [Equation]
Revised: e ( θ ) ∂θ and Q e ( θ ) ∂Q i ( θ )

Original: ∂θ in the decomposed loss:
Revised: ∂θ in the decomposed loss:

Original: [Equation]
Revised: [Equation]

Original:  Since we use a behavior policy induced by the total Q-function Q = Q e + βQ i rather than the individual components, theory would suggest to use the combined loss instead.
Revised: [Equation] Since we use a behavior policy induced by the total Q-function Q = Q e + βQ i rather than the individual components, theory would suggest to use the combined loss instead.

Original: In addition, from a practical implementation perspective, this switch to the combined loss greatly simplifies the design choices involved in our proposed trust region method described in A1 .
Revised: In addition, from a practical implementation perspective, this switch to the combined loss greatly simplifies the design choices involved in our proposed trust region method described in A1 .

Original: The penalty paid for this choice is that the decomposition of the value function into extrinsic and intrinsic components no longer carries a strict semantic meaning.
Revised: The penalty paid for this choice is that the decomposition of the value function into extrinsic and intrinsic components no longer carries a strict semantic meaning.

Original: Nevertheless we do still retain an implicit inductive bias induced by multiplication of Q i with the intrinsic reward weight β j .
Revised: Nevertheless we do still retain an implicit inductive bias induced by multiplication of Q i with the intrinsic reward weight β j .

Original: D Robustifying behavior via policy distillation.
Revised: D Robustifying behavior via policy distillation.

Original: Schaul et al.
Revised: Schaul et al.

Original: (2022) describe the effect of policy churn, whereby the greedy action of value-based RL algorithms may change frequently over consecutive parameter updates.
Revised: (2022) describe the effect of policy churn, whereby the greedy action of value-based RL algorithms may change frequently over consecutive parameter updates.

Original: This can have a deleterious effect on off-policy correction methods: traces will be cut more aggressively than with a stochastic policy, and bootstrap values will change frequently which can result in a higher variance return estimator.
Revised: This can have a deleterious effect on off-policy correction methods: traces will be cut more aggressively than with a stochastic policy, and bootstrap values will change frequently which can result in a higher variance return estimator.

Original: In addition, our choice of training with temporally-consistent stochastic depth masks can be interpreted as learning an implicit ensemble of Q-functions; thus it is natural to ask whether we may see additional benefit from leveraging the policy induced by this ensemble.
Revised: In addition, our choice of training with temporally-consistent stochastic depth masks can be interpreted as learning an implicit ensemble of Q-functions; thus it is natural to ask whether we may see additional benefit from leveraging the policy induced by this ensemble.

Original: We propose to train an explicit policy head π dist (see Figure 2) via policy distillation to match the ϵ -greedy policy induced by the Q-function.
Revised: We propose to train an explicit policy head π dist (see Figure 2) via policy distillation to match the ϵ -greedy policy induced by the Q-function.

Original: In expectation over multiple gradient steps this should help to smooth out the policy over time, as well as over the ensemble, while being much faster to evaluate than the individual members of the ensemble.
Revised: In expectation over multiple gradient steps this should help to smooth out the policy over time, as well as over the ensemble, while being much faster to evaluate than the individual members of the ensemble.

Original: Similarly to the trust-region described in A1 , we mask the policy distillation loss at any timestep where a KL constraint C KL is violated:
Revised: Similarly to the trust-region described in A1 , we mask the policy distillation loss at any timestep where a KL constraint C KL is violated:

Original: [Equation]
Revised: [Equation]

Original: We use a fixed value of ϵ = 10 − 4 to act as a simple regularizer to prevent the policy logits from growing too large.
Revised: We use a fixed value of ϵ = 10 − 4 to act as a simple regularizer to prevent the policy logits from growing too large.

Original: We then use π ′ dist = softmax ( log [Equation] ) as the target policy in our Soft Watkins
Revised: We then use π ′ dist = softmax ( log π dist τ ) as the target policy in our Soft Watkins

Original: Q( λ ) return estimator, where τ is a fixed temperature parameter.
Revised: Q( λ ) return estimator, where τ is a fixed temperature parameter.

Original: We tested values of τ in the range [0 , 1] and found that any choice of τ in this range yields improvement compared to not using the distilled policy, but values closer to 1 tend to exhibit greater stability, while those closer to 0 tend to learn more efficiently.
Revised: We tested values of τ in the range [0 , 1] and found that any choice of τ in this range yields improvement compared to not using the distilled policy, but values closer to 1 tend to exhibit greater stability, while those closer to 0 tend to learn more efficiently.

Original: We settled on an intermediate τ = 0 .
Revised: We settled on an intermediate τ = 0 .

Original: 25 to help balance between these two effects.
Revised: 25 to help balance between these two effects.

Original: We found that sampling from π ′ dist at behavior-time was less effective in some sparse reward environments, and instead opt to have the agent act according to G ϵ (cid:0) π dist (cid:1).
Revised: We found that sampling from π ′ dist at behavior-time was less effective in some sparse reward environments, and instead opt to have the agent act according to G ϵ (cid:0) π dist (cid:1)

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Methods constituting MEME proposed in Section 4 aim at improving the data efficiency of Agent57, but such efficiency gains must not come at the cost of end performance.
Revised: Methods constituting MEME proposed in Section 4 aim at improving the data efficiency of Agent57, but such efficiency gains must not come at the cost of end performance.

Original: For this reason, we train our agent with a budget of 1B environment frames 3 .
Revised: For this reason, we train our agent with a budget of 1B environment frames 3 .

Original: This budget allows us to validate that the asymptotic performance is maintained, i.e. the agent converges and is stable when improving data efficiency.
Revised: This budget allows us to validate that the asymptotic performance is maintained, i.e. the agent converges and is stable when improving data efficiency.

Original: Hyperparameters have been tuned over a subset of eight games, encompassing games with different reward density, scale and requiring credit assignment over different time horizons: Frostbite ,
Revised: Hyperparameters have been tuned over a subset of eight games, encompassing games with different reward density, scale and requiring credit assignment over different time horizons: Frostbite ,

Original: H.E.R.O. , Montezuma’s Revenge , Pitfall! , Skiing , Solaris , Surround , and Tennis .
Revised: H.E.R.O. , Montezuma’s Revenge , Pitfall! , Skiing , Solaris , Surround , and Tennis .

Original: Specifically, we selected Montezuma’s Revenge , Pitfall! , and H.E.R.O. as these are games exhibiting very sparse and large scale reward signal, requiring long-term credit assignment, while being partially observable.
Revised: Specifically, we selected Montezuma’s Revenge , Pitfall! , and H.E.R.O. as these are games exhibiting very sparse and large scale reward signal, requiring long-term credit assignment, while being partially observable.

Original: Additionally, Solaris and Skiing also require long-term credit assignment, but have medium scale and large negative scale rewards, respectively.
Revised: Additionally, Solaris and Skiing also require long-term credit assignment, but have medium scale and large negative scale rewards, respectively.

Original: Having a high degree of randomness in the observations is also a property of Solaris which our agents should be robust to.
Revised: Having a high degree of randomness in the observations is also a property of Solaris which our agents should be robust to.

Original: Finally, Surround and Tennis can be considered as fully-observable and having small-scale, moderately sparse rewards.
Revised: Finally, Surround and Tennis can be considered as fully-observable and having small-scale, moderately sparse rewards.

Original: In particular, due to the improvements in stability, we find it beneficial to use a higher samples per insert ratio (SPI) than the published values of Agent57 or R2D2.
Revised: In particular, due to the improvements in stability, we find it beneficial to use a higher samples per insert ratio (SPI) than the published values of Agent57 or R2D2.

Original: For all our experiments SPI is fixed to 6 .
Revised: For all our experiments SPI is fixed to 6 .

Original: An ablation on the effect of different SPI values can be found in Appendix L.
Revised: An ablation on the effect of different SPI values can be found in Appendix L.

Original: An exhaustive description of the hyperparameters used is provided in Appendix A, and the network architecture in Appendix B.
Revised: An exhaustive description of the hyperparameters used is provided in Appendix A, and the network architecture in Appendix B.

Original: We report results averaged over six seeds per game for all experiments.
Revised: We report results averaged over six seeds per game for all experiments.

Original: We also show similar results with sticky actions (Machado et al., 2018) in Appendix K. Metrics for previous methods are computed using the mean final score per game reported in their respective publications:
Revised: We also show similar results with sticky actions (Machado et al., 2018) in Appendix K. Metrics for previous methods are computed using the mean final score per game reported in their respective publications:

Original: Agent57 (Badia et al., 2020), MuZero 20B (Schrittwieser et al., 2020), MuZero 200M (Schrittwieser et al., 2021),
Revised: Agent57 (Badia et al., 2020), MuZero 20B (Schrittwieser et al., 2020), MuZero 200M (Schrittwieser et al., 2021),

Original: Muesli (Hessel et al., 2021).
Revised: Muesli (Hessel et al., 2021).

Original: Data efficiency comparisons with Agent57 are based on the results reported by Badia et al.
Revised: Data efficiency comparisons with Agent57 are based on the results reported by Badia et al.

Original: 5. S UMMARY OF RESULTS
Revised: 5. S UMMARY OF RESULTS

Original: In this section, we show that the proposed MEME agent is able to achieve a 200-fold reduction in the number of frames required for all games to surpass the human benchmark.
Revised: In this section, we show that the proposed MEME agent is able to achieve a 200-fold reduction in the number of frames required for all games to surpass the human benchmark.

Original: The last game to surpass  human score with MEME is Pitfall!
Revised: The last game to surpass [Table] human score with MEME is Pitfall!

Original: at 390M frames, as compared to 78B frames required for Skiing , the last game for Agent57.
Revised: at 390M frames, as compared to 78B frames required for Skiing , the last game for Agent57.

Revised: Figure 1 gives further details per game, about the required number of frames to reach the human baseline

Original: [Table]
Revised: [Table]

Original: Figure 1 gives further details per game, about the required number of frames to reach the human baseline, and shows that hard exploration games such as Private Eye , Pitfall!
Revised: and shows that hard exploration games such as Private Eye , Pitfall!

Original: and Montezuma’s
Revised: and Montezuma’s

Original: Revenge pose the hardest challenge for both agents, being among the last ones in which the human baseline is surpassed.
Revised: Revenge pose the hardest challenge for both agents, being among the last ones in which the human baseline is surpassed.

Original: This can be explained by the fact that in these games the agent might need a very large number of episodes before it is able to generate useful experience from which to learn.
Revised: This can be explained by the fact that in these games the agent might need a very large number of episodes before it is able to generate useful experience from which to learn.

Original: Notably, our agent is able to reduce the number of interactions required to outperform the human baseline in each of the 57 Atari games, by 62 × on average.
Revised: Notably, our agent is able to reduce the number of interactions required to outperform the human baseline in each of the 57 Atari games, by 62 × on average.

Original: In addition to the improved sampleefficiency, MEME shows competitive performance compared to the state-of-the-art methods shown in Table 1, while being trained for only 1B frames.
Revised: In addition to the improved sampleefficiency, MEME shows competitive performance compared to the state-of-the-art methods shown in Table 1, while being trained for only 1B frames.

Original: When comparing with state-of-the-art agents such as MuZero (Schrittwieser et al., 2020) or Muesli (Hessel et al., 2021), we observe a similar pattern to that reported by Badia et al.
Revised: When comparing with state-of-the-art agents such as MuZero (Schrittwieser et al., 2020) or Muesli (Hessel et al., 2021), we observe a similar pattern to that reported by Badia et al.

Original: (2020): they achieve very high scores in most games, as denoted by their high mean and median scores, but struggle to learn completely on a few of them (Agarwal et al., 2021).
Revised: (2020): they achieve very high scores in most games, as denoted by their high mean and median scores, but struggle to learn completely on a few of them (Agarwal et al., 2021).

Original: It is important to note that our agent achieves the human benchmark in all games, as demonstrated by its higher scores on the lower percentiles.
Revised: It is important to note that our agent achieves the human benchmark in all games, as demonstrated by its higher scores on the lower percentiles.

Original: 5. A BLATIONS
Revised: 5. A BLATIONS

Original: We analyze the contribution of all the components introduced in Section 4 through ablation experiments on the same subset of eight games.
Revised: We analyze the contribution of all the components introduced in Section 4 through ablation experiments on the same subset of eight games.

Original: We compare methods based on the area under the score curve in order to capture end performance and data efficiency under a single metric 4 .
Revised: We compare methods based on the area under the score curve in order to capture end performance and data efficiency under a single metric 4 .

Original: The magnitude of this quantity varies across games, so we normalize it by its maximum value per game in order to obtain a score between 0 and 1, and report results on the eight ablation games in Figure 4.
Revised: The magnitude of this quantity varies across games, so we normalize it by its maximum value per game in order to obtain a score between 0 and 1, and report results on the eight ablation games in Figure 4.

Original: Appendix J includes full learning curves for all ablation experiments as well as ablations of other design choices (e.g. the number of times each sample is replayed on average, optimizer hyperparameters).
Revised: Appendix J includes full learning curves for all ablation experiments as well as ablations of other design choices (e.g. the number of times each sample is replayed on average, optimizer hyperparameters).

Original: Results in Figure 4 demonstrate that all proposed methods play a role in the performance and stability of the agent.
Revised: Results in Figure 4 demonstrate that all proposed methods play a role in the performance and stability of the agent.

Original: Disabling the trust region and using the target network for bootstrapping ( A1 ) produces the most important performance drop among all ablations, likely due to the target network being slower at propagating information into the targets.
Revised: Disabling the trust region and using the target network for bootstrapping ( A1 ) produces the most important performance drop among all ablations, likely due to the target network being slower at propagating information into the targets.

Original: Interestingly, we have observed that the trust region is beneficial even when using target networks for bootstrapping (c.f. Figure 24 in Appendix
Revised: Interestingly, we have observed that the trust region is beneficial even when using target networks for bootstrapping (c.f. Figure 24 in Appendix

Original: N), which suggests that the trust region may produce an additional stabilizing effect beyond what target networks alone can provide.
Revised: N), which suggests that the trust region may produce an additional stabilizing effect beyond what target networks alone can provide.

Original: Besides having a similar stabilizing effect, policy distillation ( D ) also speeds up convergence on some games and has less tendency to converge to local optima on some others.
Revised: Besides having a similar stabilizing effect, policy distillation ( D ) also speeds up convergence on some games and has less tendency to converge to local optima on some others.

Original: The Soft Watkins’ Q( λ ) loss ( A2 ) boosts data efficiency especially in games with sparse rewards and requiring long-term credit assignment, and we have empirically verified thatit uses longer traces than other losses
Revised: The Soft Watkins’ Q( λ ) loss ( A2 ) boosts data efficiency especially in games with sparse rewards and requiring long-term credit assignment, and we have empirically verified that

Original:  (c.f. Figure 5 in Appendix C).
Revised: [Figure] it uses longer traces than other losses (c.f. Figure 5 in Appendix C).

Original: Cross-mixture training ( B2 ) and the combined loss ( C2 ) tend to provide efficiency gains across most games.
Revised: Cross-mixture training ( B2 ) and the combined loss ( C2 ) tend to provide efficiency gains across most games.

Original: Finally, while we[Figure] observe overall gains in performance from using normalization of TD errors ( B1 ), the effect is less pronounced than that of other improvements.
Revised: Finally, while we observe overall gains in performance from using normalization of TD errors ( B1 ), the effect is less pronounced than that of other improvements.

Original: We hypothesize that the normalization has a high degree of overlap with other regularizing enhancements, such as including the trust region.
Revised: We hypothesize that the normalization has a high degree of overlap with other regularizing enhancements, such as including the trust region.

Original: We further study the benefits of the proposed components in an agent-agnostic context, by examining their performance when using the R2D2 agent (Kapturowski et al., 2018), which we present in
Revised: We further study the benefits of the proposed components in an agent-agnostic context, by examining their performance when using the R2D2 agent (Kapturowski et al., 2018), which we present in

Original: Appendix M.
Revised: Appendix M.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We present a novel agent MEME that outperforms the human-level baseline in a data-efficient manner on all 57 Atari games.
Revised: We present a novel agent MEME that outperforms the human-level baseline in a data-efficient manner on all 57 Atari games.

Original: Our agent outperforms the human baseline across all 57 Atari games in 390M frames, using two orders of magnitude fewer interactions with the environment than Agent57, which leads to a 62 × speed-up on average.
Revised: Our agent outperforms the human baseline across all 57 Atari games in 390M frames, using two orders of magnitude fewer interactions with the environment than Agent57, which leads to a 62 × speed-up on average.

Original: As Atari games are played at 60 FPS, this translates to approximately 75 days of training, compared to more than 41 years of gameplay required by Agent57.
Revised: As Atari games are played at 60 FPS, this translates to approximately 75 days of training, compared to more than 41 years of gameplay required by Agent57.

Original: To achieve this, the agent employs a set of improvements that address the issues apparent in the previous state-of-the art methods.
Revised: To achieve this, the agent employs a set of improvements that address the issues apparent in the previous state-of-the art methods.

Original: Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under a rapidly-changing policy.
Revised: Our contributions aim to achieve faster propagation of learning signals related to rare events, stabilize learning under differing value scales, improve the neural network architecture, and make updates more robust under a rapidly-changing policy.

Original: We ran ablation experiments to evaluate the contribution of each of the algorithmic and network improvements.
Revised: We ran ablation experiments to evaluate the contribution of each of the algorithmic and network improvements.

Original: Introducing online network bootstrapping with a trust-region has the most impact on the performance among these changes, while certain games require multiple improvements to maintain stability and performance.
Revised: Introducing online network bootstrapping with a trust-region has the most impact on the performance among these changes, while certain games require multiple improvements to maintain stability and performance.

Original: The increased stability enables more aggressive optimization, e.g. through higher samples per insert ratios or a centralized bandit that aggregates statistics from all actors, leading to more data-efficient learning.
Revised: The increased stability enables more aggressive optimization, e.g. through higher samples per insert ratios or a centralized bandit that aggregates statistics from all actors, leading to more data-efficient learning.

Original: Although our agent achieves above average-human performance on all 57 Atari games within 390M frames with the same set of hyperparameters, the agent is separately trained on each game.
Revised: Although our agent achieves above average-human performance on all 57 Atari games within 390M frames with the same set of hyperparameters, the agent is separately trained on each game.

Original: An interesting research direction to pursue would be to devise a training scheme such that the agent with the same set of weights can achieve similar performance and data efficiency as the proposed agent on all games.
Revised: An interesting research direction to pursue would be to devise a training scheme such that the agent with the same set of weights can achieve similar performance and data efficiency as the proposed agent on all games.

Original: Furthermore, the improvements that we propose do not necessarily only apply to Agent57, and further study could analyze the impact of these components in other state-of-the-art agents.
Revised: Furthermore, the improvements that we propose do not necessarily only apply to Agent57, and further study could analyze the impact of these components in other state-of-the-art agents.

Original: We also expect that the generality of the agent could be expanded further.
Revised: We also expect that the generality of the agent could be expanded further.

Original: While this work focuses on
Revised: While this work focuses on

Original: Atari due to its wide range of tasks and reward structures, future research is required to analyze the ability of the agent to tackle other important challenges, such as more complex observation spaces (e.g. 3D navigation, multi-modal inputs), complex action spaces, or longer-term credit assignment.
Revised: Atari due to its wide range of tasks and reward structures, future research is required to analyze the ability of the agent to tackle other important challenges, such as more complex observation spaces (e.g. 3D navigation, multi-modal inputs), complex action spaces, or longer-term credit assignment.

Original: All such improvements would lead MEME towards achieving greater generality.
Revised: All such improvements would lead MEME towards achieving greater generality.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: In this manuscript we made additional efforts to make sure that the explanations of the proposed methods are detailed enough to be easily reproduced by the community.
Revised: In this manuscript we made additional efforts to make sure that the explanations of the proposed methods are detailed enough to be easily reproduced by the community.

Original: Regarding the training setup, we detail the distributed setting and the computation resources used in Appendices E and G, respectively.
Revised: Regarding the training setup, we detail the distributed setting and the computation resources used in Appendices E and G, respectively.

Original: The optimiser details are presented in Appendix D, while the comprehensive list of hyperparameters used is given in Appendix A.
Revised: The optimiser details are presented in Appendix D, while the comprehensive list of hyperparameters used is given in Appendix A.

Original: We explain the implementation details of the neural network architecture in Appendix B with the help of Figure 2.
Revised: We explain the implementation details of the neural network architecture in Appendix B with the help of Figure 2.

Original: Finally we give additional details about the components such as the bandit (Appendix F), the NGU and RND intrinsic rewards (Appendix H), target computation (Appendix C) and further clarify the proposed trust region with Figure
Revised: Finally, we give additional details about the components such as the bandit (Appendix F), the NGU and RND intrinsic rewards (Appendix H), target computation (Appendix C) and further clarify the proposed trust region with Figure

Revised: E THICS S TATEMENT

Revised: We would like to note that large scale RL experiments require a significant amount of compute, as outlined in the Appendix G.

Revised: In some cases the results obtained may not justify the incurred compute and environmental costs.

Revised: However, to this end, our paper greatly improves the sample-efficiency of such RL methods, and we expect its development cost and environmental impact to be amortized over many subsequent applications.

Revised: We hope that in the future, researchers will leverage the contributions presented in this paper in their work on large-scale RL and help reduce the environmental impact their research might have.

Original: A H YPER - PARAMETERS
Revised: A H YPER - PARAMETERS

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: B N ETWORK A RCHITECTURE B. T ORSO
Revised: B N ETWORK A RCHITECTURE B. T ORSO

Original: We use a modified version of the NFNet architecture (Brock et al., 2021).
Revised: We use a modified version of the NFNet architecture (Brock et al., 2021).

Original: We use a simplified stem, consisting of a single 7 × 7 convolution with stride 4.
Revised: We use a simplified stem, consisting of a single 7 × 7 convolution with stride 4.

Original: We also forgo bottleneck residual blocks in favor of 2 layers of 3 × 3 convolutions, followed by a Squeeze-Excite block.
Revised: We also forgo bottleneck residual blocks in favor of 2 layers of 3 × 3 convolutions, followed by a Squeeze-Excite block.

Original: In addition, we make some minor modifications to the downsampling blocks.
Revised: In addition, we make some minor modifications to the downsampling blocks.

Original: Specifically, we apply an activation prior to the average pooling and multiply the output by the stride in order to maintain the variance of the activations.
Revised: Specifically, we apply an activation prior to the average pooling and multiply the output by the stride in order to maintain the variance of the activations.

Original: This is then followed by a 3 × 3 convolution.
Revised: This is then followed by a 3 × 3 convolution.

Original: All convolutions use a Scaled Weight Standardization scheme (Qiao et al., 2019).
Revised: All convolutions use a Scaled Weight Standardization scheme (Qiao et al., 2019).

Original: The block section parameters are as follows:
Revised: The block section parameters are as follows:

Original: • num blocks: (2, 3, 4, 4) • num channels: (64, 128, 128, 64) • strides: (1, 2, 2, 2)
Revised: • num blocks: (2, 3, 4, 4) • num channels: (64, 128, 128, 64) • strides: (1, 2, 2, 2)

Original: B. N ON - IMAGE F EATURES
Revised: B. N ON - IMAGE F EATURES

Original: We also feed the following features into the network:
Revised: We also feed the following features into the network:

Original: • previous action (encoded as a size-32 embedding) • previous extrinsic reward • previous intrinsic reward • previous RND component of intrinsic reward • previous Episodic component of intrinsic reward • previous Action Prediction embedding
Revised: • previous action (encoded as a size-32 embedding) • previous extrinsic reward • previous intrinsic reward • previous RND component of intrinsic reward • previous Episodic component of intrinsic reward • previous Action Prediction embedding

Original: These are fed into a single linear layer of size 256 and activation and then concatenated with the output of the torso and input into the recurrent core.
Revised: These are fed into a single linear layer of size 256 and activation and then concatenated with the output of the torso and input into the recurrent core.

Original: B. R ECURRENT C ORE
Revised: B. R ECURRENT C ORE

Original: The recurrent core is composed of single LSTM with hidden size 1024.
Revised: The recurrent core is composed of single LSTM with hidden size 1024.

Original: The output is then concatenated together with the core input and fed into the Action-Value and Policy heads.
Revised: The output is then concatenated together with the core input and fed into the Action-Value and Policy heads.

Original: B. A CTION -V ALUE AND P OLICY H EADS
Revised: B. A CTION -V ALUE AND P OLICY H EADS

Original: We utilize separate dueling heads for the intrinsic and extrinsic components of the value function, and a simple MLP for the policy.
Revised: We utilize separate dueling heads for the intrinsic and extrinsic components of the value function, and a simple MLP for the policy.

Original: All heads use two hidden layers of size 1024 and output size num _ actions × num _ mixtures .
Revised: All heads use two hidden layers of size 1024 and output size num _ actions × num _ mixtures .

Original: C T ARGET C OMPUTATION AND T RACE C OEFFICIENTS
Revised: C T ARGET C OMPUTATION AND T RACE C OEFFICIENTS

Original: As motivated in Section 4 A2 , we introduce Soft Watkins Q( λ ) as a trade-off between aggressive trace cutting used within Retrace and Watkins Q( λ ), and the lack of off-policy correction in Peng’s Q( λ ).
Revised: As motivated in Section 4 A2 , we introduce Soft Watkins Q( λ ) as a trade-off between aggressive trace cutting used within Retrace and Watkins Q( λ ), and the lack of off-policy correction in Peng’s Q( λ ).

Original: We investigate this hypothesis by observing the average trace coefficients for each of the methods in
Revised: We investigate this hypothesis by observing the average trace coefficients for each of the methods in

Original: Figure 5.
Revised: Figure 5.

Original: The lower values of the trace coefficient
Revised: The lower values of the trace coefficient

Original: λ
Revised: λ

Original: lead to more aggressive trace cutting as soon as the data is off-policy.
Revised: lead to more aggressive trace cutting as soon as the data is off-policy.

Original: Conversely, the λ value of 1 signifies no trace cutting whatsoever.
Revised: Conversely, the λ value of 1 signifies no trace cutting whatsoever.

Original: As expected,
Revised: As expected,

Original: Retrace’s trace coefficient is significantly lower than the other methods considered.
Revised: Retrace’s trace coefficient is significantly lower than the other methods considered.

Original: The proposed
Revised: The proposed

Original: Soft Watkins Q( λ ) has a parameter κ that allows us to control the permissiveness of the trace-cutting which in turn affects the final trace coefficient.
Revised: Soft Watkins Q( λ ) has a parameter κ that allows us to control the permissiveness of the trace-cutting which in turn affects the final trace coefficient.

Original: We ran a course sweep of this parameter over the values { 0 , .
Revised: We ran a course sweep of this parameter over the values { 0 , .

Original: and found values of κ = 0 .
Revised: and found values of κ = 0 .

Original: 01 to generally perform best in our setting.
Revised: 01 to generally perform best in our setting.

Original: As can be seen from Figure 5 this choice considerably increases the average trace coefficient of Soft Watkins’
Revised: As can be seen from Figure 5 this choice considerably increases the average trace coefficient of Soft Watkins’

Original: Q ( λ ) relative to κ = 0 , and concentrates the distribution of trace coefficients without causing it to collapse.
Revised: Q ( λ ) relative to κ = 0 , and concentrates the distribution of trace coefficients without causing it to collapse.

Original: D O PTIMIZER
Revised: D O PTIMIZER

Original: We use separate optimizers for each of the RL, Action Prediction, and RND Networks, with the only differences between them being the choice of learning rate and weight decay.
Revised: We use separate optimizers for each of the RL, Action Prediction, and RND Networks, with the only differences between them being the choice of learning rate and weight decay.

Original: The base optimizer
Revised: The base optimizer

Original: [Figure]
Revised: [Figure]

Revised: a

Revised: [Figure]

Original: t r a c e c o e ff i
Revised: t r a c e c o e ff i

Original: c i e n t used is AdamW with Nesterov Momentum.
Revised: c i e n t used is AdamW with Nesterov Momentum.

Original: A linear learning rate warmup is used for the firstupdates, to allow time for the Adam statistics to stabilize.
Revised: A linear learning rate warmup is used for the firstupdates, to allow time for the Adam statistics to stabilize.

Original: Most notably, we employ an adaptive element-wise gradient clipping strategy whereby we maintain running estimates of the mean and standard deviation of the element-wise gradients using an Exponential Moving Average (EMA) with α clip = 0 .
Revised: Most notably, we employ an adaptive element-wise gradient clipping strategy whereby we maintain running estimates of the mean and standard deviation of the element-wise gradients using an Exponential Moving Average (EMA) with α clip = 0 .

Original: 999 , and only clip when the gradient magnitude exceeds the mean by some factor c of the standard deviation.
Revised: 999 , and only clip when the gradient magnitude exceeds the mean by some factor c of the standard deviation.

Original: Concretely, the factor c is determined by specifying a percentile p clip (that we fix to 0 . 99 ) which is then input to the inverse CDF of a standard normal distribution.
Revised: Concretely, the factor c is determined by specifying a percentile p clip (that we fix to 0 . 99 ) which is then input to the inverse CDF of a standard normal distribution.

Original: In general we found this clipping strategy to yield a small but consistent improvement in stability across many environments compared to global norm clipping.
Revised: In general we found this clipping strategy to yield a small but consistent improvement in stability across many environments compared to global norm clipping.

Original: We ablate some of the optimizer choices in Figure 19.
Revised: We ablate some of the optimizer choices in Figure 19.

Original: E D ISTRIBUTED SETTING
Revised: E D ISTRIBUTED SETTING

Original: All experiments are run using a distributed setting.
Revised: All experiments are run using a distributed setting.

Original: The experiment consists of the actors, learner, bandit and evaluators, as well as a replay buffer.
Revised: The experiment consists of the actors, learner, bandit and evaluators, as well as a replay buffer.

Original: The actors and evaluators are the two types of workers that draw samples from the environment.
Revised: The actors and evaluators are the two types of workers that draw samples from the environment.

Original: Since actors collect experience with non-greedy policies, we follow the common practice in this type of agent and report scores from separate evaluator processes that continually execute the greedy policy and whose experience is not added to the replay buffer (Kapturowski et al., 2018).
Revised: Since actors collect experience with non-greedy policies, we follow the common practice in this type of agent and report scores from separate evaluator processes that continually execute the greedy policy and whose experience is not added to the replay buffer (Kapturowski et al., 2018).

Original: Therefore, only the actor workers write to the replay buffer, while the evaluation workers are used purely for reporting the performance.
Revised: Therefore, only the actor workers write to the replay buffer, while the evaluation workers are used purely for reporting the performance.

Original: The evaluation scheme differs from R2D2 (Kapturowski et al., 2018) in that a separate set of eval parameters are maintained, which are computed as an EMA of the online network with η eval = 0 .
Revised: The evaluation scheme differs from R2D2 (Kapturowski et al., 2018) in that a separate set of eval parameters are maintained, which are computed as an EMA of the online network with η eval = 0 .

Original: 995 ; and these eval parameters are continually updated throughout each episode.
Revised: 995 ; and these eval parameters are continually updated throughout each episode.

Original: We observed that the use of these eval parameters provided a consistent performance boost across almost all environments, but we continue to use the online network for the actors in order to obtain more varied experience.
Revised: We observed that the use of these eval parameters provided a consistent performance boost across almost all environments, but we continue to use the online network for the actors in order to obtain more varied experience.

Original: In the replay buffer, we store fixed-length sequences of ( r et , r NGU t , x t , a t ) tuples.
Revised: In the replay buffer, we store fixed-length sequences of ( r et , r NGU t , x t , a t ) tuples.

Original: These sequences never cross episode boundaries.
Revised: These sequences never cross episode boundaries.

Original: For Atari, we apply the standard DQN pre-processing, as used in
Revised: For Atari, we apply the standard DQN pre-processing, as used in

Original: R2D2.
Revised: R2D2.

Original: The replay buffer is split into 8 shards, to improve robustness due to computational constraints, with each shard maintaining an independent prioritisation of the entries.
Revised: The replay buffer is split into 8 shards, to improve robustness due to computational constraints, with each shard maintaining an independent prioritisation of the entries.

Original: We use prioritized experience replay with the same prioritization scheme proposed by Kapturowski et al.
Revised: We use prioritized experience replay with the same prioritization scheme proposed by Kapturowski et al.

Original: (2018) which used a weighted mixture of max and mean TD-errors over the sequence.
Revised: (2018) which used a weighted mixture of max and mean TD-errors over the sequence.

Original: Each of the actor workers writes to a specific shard which is consistent throughout training.
Revised: Each of the actor workers writes to a specific shard which is consistent throughout training.

Original: The replay buffer is implemented using
Revised: The replay buffer is implemented using

Original: Reverb (Cassirer et al., 2021).
Revised: Reverb (Cassirer et al., 2021).

Original: Given a single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates.
Revised: Given a single batch of trajectories we unroll both online and target networks on the same sequence of states to generate value estimates.

Original: These estimates are used to execute the learner update step, which updates the model weights used by the actors, and the exponential moving average (EMA) of the weights used by the evaluator models, as this yields best performance which we report.
Revised: These estimates are used to execute the learner update step, which updates the model weights used by the actors, and the exponential moving average (EMA) of the weights used by the evaluator models, as this yields best performance which we report.

Original: Acting in the environment is accelerated by sending observations from actors and evaluators to a shared server that runs batched inference.
Revised: Acting in the environment is accelerated by sending observations from actors and evaluators to a shared server that runs batched inference.

Original: The remote inference server allows multiple clients such as actor and evaluator workers to connect to it, and executes their inputs as a batch on the corresponding inference models.
Revised: The remote inference server allows multiple clients such as actor and evaluator workers to connect to it, and executes their inputs as a batch on the corresponding inference models.

Original: The actor and evaluator inference model parameters are queried periodically from the learner.
Revised: The actor and evaluator inference model parameters are queried periodically from the learner.

Original: Also, the recurrent state is persisted on the inference server so that the actor does need to communicate it.
Revised: Also, the recurrent state is persisted on the inference server so that the actor does need to communicate it.

Original: However, the episodic memory lookup required to compute the intrinsic reward is performed locally on actors to reduce the communication overhead.
Revised: However, the episodic memory lookup required to compute the intrinsic reward is performed locally on actors to reduce the communication overhead.

Original: At the beginning of each episode, parameters β and γ are queried from the bandit worker, i.e. meta-controller.
Revised: At the beginning of each episode, parameters β and γ are queried from the bandit worker, i.e. meta-controller.

Original: The parameters are selected from a set of coefficients { ( β j , γ j ) } N − 1 j =0 with N = 16 , which correspond to the N -heads of the network.
Revised: The parameters are selected from a set of coefficients { ( β j , γ j ) } N − 1 j =0 with N = 16 , which correspond to the N -heads of the network.

Original: The actors query optimal ( β , γ ) tuples, while the evaluators query the tuple corresponding to the greedy action.
Revised: The actors query optimal ( β , γ ) tuples, while the evaluators query the tuple corresponding to the greedy action.

Original: After each actor episode, the bandit stats are updated based on the episode rewards by updating the distribution over actions, according to
Revised: After each actor episode, the bandit stats are updated based on the episode rewards by updating the distribution over actions, according to

Original: Discounted UCB (Garivier & Moulines, 2011).
Revised: Discounted UCB (Garivier & Moulines, 2011).

Original: The following subsections describe how actors, evaluators, and learner are run in each stage.
Revised: The following subsections describe how actors, evaluators, and learner are run in each stage.

Original: L EARNER
Revised: L EARNER

Original: • Sample a sequence of extrinsic rewards r et , intrinsic rewards r NGU t , observations x t and actions a t , from the replay buffer.
Revised: • Sample a sequence of extrinsic rewards r et , intrinsic rewards r NGU t , observations x t and actions a t , from the replay buffer.

Original: • Use Q-network to learn from ( r et , r NGU t , x, a ) with our modified version of Watkins’ Q( λ ) (Watkins & Dayan, 1992) using the same procedure as in R2D2.
Revised: • Use Q-network to learn from ( r et , r NGU t , x, a ) with our modified version of Watkins’ Q( λ ) (Watkins & Dayan, 1992) using the same procedure as in R2D2.

Original: • Compute the actor model weights and EMA for the evaluator model weights.
Revised: • Compute the actor model weights and EMA for the evaluator model weights.

Original: • Use the sampled sequences to train the action prediction network in NGU.
Revised: • Use the sampled sequences to train the action prediction network in NGU.

Original: • Use the sampled sequences to train the predictor of RND.
Revised: • Use the sampled sequences to train the predictor of RND.

Original: A CTOR
Revised: A CTOR

Original: • Query optimal bandit action ( β , γ ).
Revised: • Query optimal bandit action ( β , γ ).

Original: • Obtain x t from the environment.
Revised: • Obtain x t from the environment.

Original: • Obtain r NGU t and a t from the inference model.
Revised: • Obtain r NGU t and a t from the inference model.

Original: • Insert x t , a t , r NGU t and r et in the replay buffer.
Revised: • Insert x t , a t , r NGU t and r et in the replay buffer.

Original: • Step on the environment with a t .
Revised: • Step on the environment with a t .

Original: E VALUATOR
Revised: E VALUATOR

Original: • Query greedy bandit action ( β , γ ).
Revised: • Query greedy bandit action ( β , γ ).

Original: • Obtain x t from the environment.
Revised: • Obtain x t from the environment.

Original: • Obtain r NGU t and a t from the inference model.
Revised: • Obtain r NGU t and a t from the inference model.

Original: • Step on the environment with a t .
Revised: • Step on the environment with a t .

Original: [Table]
Revised: [Table]

Original: • Periodically checkpoints bandit action values.
Revised: • Periodically checkpoints bandit action values.

Original: • Queried for optimal action by actors.
Revised: • Queried for optimal action by actors.

Original: • Queried for greedy action by evaluators.
Revised: • Queried for greedy action by evaluators.

Original: • Updates the stats when actors pass the episode rewards for a certain action.
Revised: • Updates the stats when actors pass the episode rewards for a certain action.

Original: F B ANDIT I MPLEMENTATION
Revised: F B ANDIT I MPLEMENTATION

Original: While Agent57 maintains a separate bandit for each actor, we instead utilize a centralized bandit worker.
Revised: While Agent57 maintains a separate bandit for each actor, we instead utilize a centralized bandit worker.

Original: The bandit selects between a family of policies generated by tuples of intrinsic reward weight and discount factor ( β i , γ i ) , parameterized as: 
Revised: The bandit selects between a family of policies generated by tuples of intrinsic reward weight and discount factor ( β i , γ i ) , parameterized as: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: i n t r i n s i c r e w a r d w e
Revised: i n t r i n s i c r e w a r d w e

Original: i g h t
Revised: i g h t

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: d i s c o
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: d i s c o un t

Original:  t o r i
Revised: f a c t o r i

Original: [Equation]
Revised: [Equation]

Original: where N is the number of policies, i is the policy index, σ is the sigmoid function, β IM is the maximum intrinsic reward weight, and γ max and γ int are the maximum and minimum discount factors, respectively.
Revised: where N is the number of policies, i is the policy index, σ is the sigmoid function, β IM is the maximum intrinsic reward weight, and γ max and γ int are the maximum and minimum discount factors, respectively.

Original: At the beginning of each episode an actor will sample a policy index with which to act for the duration of the episode.
Revised: At the beginning of each episode an actor will sample a policy index with which to act for the duration of the episode.

Original: At the end of which, the actor will update the bandit with the obtained extrinsic return for that policy.
Revised: At the end of which, the actor will update the bandit with the obtained extrinsic return for that policy.

Original: We use a discounted variant of UCB-Tuned bandit algorithm (Garivier & Moulines (2008) and Auer et al.
Revised: We use a discounted variant of UCB-Tuned bandit algorithm (Garivier & Moulines (2008) and Auer et al.

Original: In practice the bandit hyper-parameters did not seem to be very important.
Revised: In practice the bandit hyper-parameters did not seem to be very important.

Original: We hypothesize that the use of cross-mixture training may reduce the sensitivity of the agent to these parameters, though we have not explored this relationship thoroughly.
Revised: We hypothesize that the use of cross-mixture training may reduce the sensitivity of the agent to these parameters, though we have not explored this relationship thoroughly.

Original: G C OMPUTE R ESOURCES
Revised: G C OMPUTE R ESOURCES

Original: For the experiments we used the TPUv4, with the 2 × 2 × 1 topology used for the learner.
Revised: For the experiments we used the TPUv4, with the 2 × 2 × 1 topology used for the learner.

Original: Acting is accelerated by sending observations from actors to a shared server that runs batched inference using a 1 × 1 × 1 TPUv4, which is used for inference within the actor and evaluation workers.
Revised: Acting is accelerated by sending observations from actors to a shared server that runs batched inference using a 1 × 1 × 1 TPUv4, which is used for inference within the actor and evaluation workers.

Original: On average, the learner performs 3.8 updates per second.
Revised: On average, the learner performs 3.8 updates per second.

Original: The rate at which environment frames are written to the replay buffer by the actors is approximately 12,970 frames per second.
Revised: The rate at which environment frames are written to the replay buffer by the actors is approximately 12,970 frames per second.

Original: Each experiment consists of 64 actors with 2 threads, each of them acting with their own independent instance of the environment.
Revised: Each experiment consists of 64 actors with 2 threads, each of them acting with their own independent instance of the environment.

Original: The collected experience is stored in the replay buffer split in 8 shards, each with independent prioritization.
Revised: The collected experience is stored in the replay buffer split in 8 shards, each with independent prioritization.

Original: This accumulated experience is used by a single learner worker, while the performance is evaluated on 5 evaluator workers.
Revised: This accumulated experience is used by a single learner worker, while the performance is evaluated on 5 evaluator workers.

Original: H I NTRINSIC REWARDS H. R ANDOM N ETWORK D ISTILLATION
Revised: H I NTRINSIC REWARDS H. R ANDOM N ETWORK D ISTILLATION

Original: The Random Network Distillation (RND) intrinsic reward (Burda et al., 2019) is computed by introducing a random, untrained convolutional network g : X → R d , and training a network
Revised: The Random Network Distillation (RND) intrinsic reward (Burda et al., 2019) is computed by introducing a random, untrained convolutional network g : X → R d , and training a network

Original: ˆ g : X → R d to predict the outputs of g on all the observations that are seen during training by minimizing the prediction error err RND ( x t )
Revised: ˆ g : X → R d to predict the outputs of g on all the observations that are seen during training by minimizing the prediction error err RND ( x t )

Original: =
Revised: =

Original: ||
Revised: ||

Original: ˆ g ( x t ; θ ) − g ( x t )
Revised: ˆ g ( x t ; θ ) − g ( x t )

Original: || 2 with respect to θ .
Revised: || 2 with respect to θ .

Original: The intuition is that the prediction error will be large on states that have been visited less frequently by the agent.
Revised: The intuition is that the prediction error will be large on states that have been visited less frequently by the agent.

Original: The dimensionality of the random embedding, d , is a hyperparameter of the algorithm.
Revised: The dimensionality of the random embedding, d , is a hyperparameter of the algorithm.

Original: The RND intrinsic reward is obtained by normalising the prediction error.
Revised: The RND intrinsic reward is obtained by normalising the prediction error.

Original: In this work, we use a slightly different normalization from that reported in Burda et al.
Revised: In this work, we use a slightly different normalization from that reported in Burda et al.

Original: The RND reward at time t is given by
Revised: The RND reward at time t is given by

Original: [Equation]
Revised: [Equation]

Original: where σ e is the running standard deviation of err RND ( x t ) .
Revised: where σ e is the running standard deviation of err RND ( x t ) .

Original: As with the TD-error statistics, we compute σ e on the learner using importance sampling weights to correct the sampling distribution.
Revised: As with the TD-error statistics, we compute σ e on the learner using importance sampling weights to correct the sampling distribution.

Original: H. N EVER G IVE U P
Revised: H. N EVER G IVE U P

Original: The NGU intrinsic reward modulates an episodic intrinsic reward, r episodic t , with a life long signal α t :
Revised: The NGU intrinsic reward modulates an episodic intrinsic reward, r episodic t , with a life long signal α t :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where L is a fixed maximum reward scaling.
Revised: where L is a fixed maximum reward scaling.

Original: The life-long novelty signal is computed using RND with the normalisation:
Revised: The life-long novelty signal is computed using RND with the normalisation:

Original: [Equation]
Revised: [Equation]

Original: 13) where err RND ( x t ) is the prediction error described in Appendix H.1, and µ e and σ e are its running mean and standard deviation, respectively.
Revised: 13) where err RND ( x t ) is the prediction error described in Appendix H.1, and µ e and σ e are its running mean and standard deviation, respectively.

Original: The episodic intrinsic reward at time t is computed according to formula:
Revised: The episodic intrinsic reward at time t is computed according to formula:

Original: [Equation]
Revised: [Equation]

Original: (14) where N k is the set containing the k -nearest neighbors of f ( x t ) in M , c is a constant and K : R p × R p
Revised: (14) where N k is the set containing the k -nearest neighbors of f ( x t ) in M , c is a constant and K : R p × R p

Original: → R + is a kernel function satisfying K ( x, x ) = 1 (which can be thought of as approximating pseudo-counts Badia et al.
Revised: → R + is a kernel function satisfying K ( x, x ) = 1 (which can be thought of as approximating pseudo-counts Badia et al.

Original: Algorithm 1 shows a detailed description of how the episodic intrinsic reward is computed.
Revised: Algorithm 1 shows a detailed description of how the episodic intrinsic reward is computed.

Original: Below we describe the different components used in Algorithm 1:
Revised: Below we describe the different components used in Algorithm 1:

Original: • M : episodic memory containing at time t the previous embeddings { f ( x 0 ) , f ( x 1 ) , . . .
Revised: • M : episodic memory containing at time t the previous embeddings { f ( x 0 ) , f ( x 1 ) , . . .

Original: f ( x t − 1 ) } .
Revised: f ( x t − 1 ) } .

Original: This memory starts empty at each episode • k : number of nearest neighbours • N k = { f ( x i ) } ki =1 : set of k -nearest neighbours of f ( x t ) in the memory M ; we call N k
Revised: This memory starts empty at each episode • k : number of nearest neighbours • N k = { f ( x i ) } ki =1 : set of k -nearest neighbours of f ( x t ) in the memory M ; we call N k

Original: [ i ] = f
Revised: [ i ] = f

Original: ( x i ) ∈ N k for ease of notation • K : kernel defined as K ( x, y )
Revised: ( x i ) ∈ N k for ease of notation • K : kernel defined as K ( x, y )

Original: =
Revised: =

Original: ϵ d 2( x,y )
Revised: ϵ d 2( x,y )

Original: d 2 m + ϵ where ϵ is a small constant, d is the Euclidean distance and d 2 m is a running average of the squared Euclidean distance of the k -nearest neighbors• c : pseudo-counts constant • ξ : cluster distance • s m : maximum similarity • f ( x ) : action prediction network output for observation x as in Badia et al.
Revised: d 2 m + ϵ where ϵ is a small constant, d is the Euclidean distance and d 2 m is a running average of the squared Euclidean distance of the k -nearest neighbors• c : pseudo-counts constant • ξ : cluster distance • s m : maximum similarity • f ( x ) : action prediction network output for observation x as in Badia et al.

Original: I O THER T HINGS WE T RIED I. F UNCTIONAL R EGULARIZATION IN PLACE OF T RUST R EGION
Revised: I O THER T HINGS WE T RIED I. F UNCTIONAL R EGULARIZATION IN PLACE OF T RUST R EGION

Original: Concurrently with the development of our trust region method we experimented with using an explicit
Revised: Concurrently with the development of our trust region method we experimented with using an explicit

Original: L2 regularization term in the loss, acting on the difference between the online and target networks, similar to (Piche et al., 2021).
Revised: L2 regularization term in the loss, acting on the difference between the online and target networks, similar to (Piche et al., 2021).

Original: Prior to implementation of our normalization scheme we found that this method stabilized learning early on in training but was prone to eventually becoming unstable if run for long enough.
Revised: Prior to implementation of our normalization scheme we found that this method stabilized learning early on in training but was prone to eventually becoming unstable if run for long enough.

Original: With normalization this instability did not occur, but sample efficiency was worse compared to the trust region in most instances we observed.[Table]
Revised: With normalization this instability did not occur, but sample efficiency was worse compared to the trust region in most instances we observed.

Original: : M ; k ; f ( x t ) ; c ; ϵ ; ξ ; x m ; d

Original: [Table] m Output: r [Table] t Compute the k -nearest neighbours of f ( x t ) in M and store them in a list N k Create a list of floats d k of size k /*
Revised: Input : M ; k ; f ( x t ) ; c ; ϵ ; ξ ; x m ; d 2 m Output: r episodic t Compute the k -nearest neighbours of f ( x t ) in M and store them in a list N k Create a list of floats d k of size k /*

Original: The list d k will contain the distances between the embedding f ( x t ) and its neighbours N k .
Revised: The list d k will contain the distances between the embedding f ( x t ) and its neighbours N k .

Original: */ for i ∈ { 1 ,...,k } do d k
Revised: */ for i ∈ { 1 ,...,k } do d k

Original: [ i ] ← d 2 ( f ( x t ) ,N k
Revised: [ i ] ← d 2 ( f ( x t ) ,N k

Original: [ i ]) end
Revised: [ i ]) end

Original: Update the

Original: [Table]

Original: d

Original: [Table]

Original:  m with the list of distances d [Table] [Table] the distances d k with the updated moving average d 2 m .
Revised: Update the moving average d 2 m with the list of distances d k /* Normalize the distances d k with the updated moving average d 2 m .

Original: [Table] [Table] the normalized distances d n i.e. they become 0 if too small and 0 k is a list of k zeros. 
Revised: */ d n ← dkd 2 m /* Cluster the normalized distances d n i.e. they become 0 if too small and 0 k is a list of k zeros. */ d

Original: [Table]
Revised: [Table]

Original: * Compute the Kernel values between the embedding
Revised: ← max(

Original: [Table]
Revised: [Table]

Original: ( x
Revised: ξ,

Original: [Table]
Revised: [Table]

Original:  ) and its neighbours N k .[Table] /* Compute the similarity between the embedding f
Revised: * Compute the Kernel values between the embedding f ( x t ) and its neighbours N k .

Original: [Table] ) and its neighbours N k .*/ s [Table]
Revised: */ K v ← ϵ dn + ϵ /* Compute the similarity between the embedding f ( s t ) and its neighbours N k .

Original:  (cid:113)(cid:80) [Table] = [Table]
Revised: */ s ← (cid:113)(cid:80) ki

Original:  [ i ] + c /* Compute the episodic intrinsic reward at [Table] t : [Table]
Revised: =1 K v [ i ] + c /* Compute the episodic intrinsic reward at time t : r it .

Original: it . */ if [Table] m then r episodic t ← 0 else r episodic t ← 1 /s
Revised: */ if x > x m then r episodic t ← 0 else r episodic t ← 1 /s

Original: I.
Revised: I.

Original: A PPROXIMATE T HOMPSON S AMPLING
Revised: A PPROXIMATE T HOMPSON S AMPLING

Original: We considered using an approximate Thompson Sampling scheme scheme similar to Bootstrapped
Revised: We considered using an approximate Thompson Sampling scheme scheme similar to Bootstrapped

Original: DQN (Osband et al., 2016) whereby the stochastic depth mask was fixed for some period of time at inference (such as once per episode, or every 100 timesteps).
Revised: DQN (Osband et al., 2016) whereby the stochastic depth mask was fixed for some period of time at inference (such as once per episode, or every 100 timesteps).

Original: We observed some marginal benefit in certain games, but in our view this difference was not enough to justify the added complexity of implementation.
Revised: We observed some marginal benefit in certain games, but in our view this difference was not enough to justify the added complexity of implementation.

Original: We hypothesize that the added exploration this provides is not significant when a strong intrinsic reward is already present in the learning algorithm, but it may have a larger effect if this is not the case.
Revised: We hypothesize that the added exploration this provides is not significant when a strong intrinsic reward is already present in the learning algorithm, but it may have a larger effect if this is not the case.

Original: I. M IXTURE OF O NLINE AND R EPLAY D ATA
Revised: I. M IXTURE OF O NLINE AND R EPLAY D ATA

Original: We considered using a mixture of online and replay data as was done in the Muesli agent (Hessel et al., 2021).
Revised: We considered using a mixture of online and replay data as was done in the Muesli agent (Hessel et al., 2021).

Original: This was beneficial for overall stability, but it also degraded performance in the harder exploration games such as Pitfall! .
Revised: This was beneficial for overall stability, but it also degraded performance in the harder exploration games such as Pitfall! .

Original: We were not able to find an easy remedy for this so we did not investigate further in this direction.
Revised: We were not able to find an easy remedy for this so we did not investigate further in this direction.

Original: I. E STIMATING Q e ( θ ) + βQ i ( θ )
Revised: I. E STIMATING Q e ( θ ) + βQ i ( θ )

Original: Agent57 uses two neural networks with completely independent weights to estimate Q e and Q i .
Revised: Agent57 uses two neural networks with completely independent weights to estimate Q e and Q i .

Original: As mentioned in the work, this provides the network with more robustness to the different scales and variance that r e and r i have for many tasks.
Revised: As mentioned in the work, this provides the network with more robustness to the different scales and variance that r e and r i have for many tasks.

Original: MEME changes the separation of networks, whereby the Q e and Q i are still estimated separately, but they share a common torso and recurrent core.
Revised: MEME changes the separation of networks, whereby the Q e and Q i are still estimated separately, but they share a common torso and recurrent core.

Original: However, since many of the components we introduce are geared toward improving stability, even this separation may no longer be necessary.
Revised: However, since many of the components we introduce are geared toward improving stability, even this separation may no longer be necessary.

Original: To analyze this we run an experiment where the agent network has a single head that estimates Q e ( θ ) + βQ
Revised: To analyze this we run an experiment where the agent network has a single head that estimates Q e ( θ ) + βQ

Original: i ( θ ) .
Revised: i ( θ ) .

Original: Note that in this case we still estimate N sets of Q-values.
Revised: Note that in this case we still estimate N sets of Q-values.

Original: Indeed, as results of Figure 7 show, we observe similar results as our proposed method.
Revised: Indeed, as results of Figure 7 show, we observe similar results as our proposed method.

Original: This indicates that the inductive bias that was introduced in maintaining separate heads for intrinsic and extrinsic Q -values is no longer important.
Revised: This indicates that the inductive bias that was introduced in maintaining separate heads for intrinsic and extrinsic Q -values is no longer important.

Original: [Figure]
Revised: [Figure]

Original: J A DDITIONAL RESULTS
Revised: J A DDITIONAL RESULTS

Original: S k ii n g
Revised: S k ii n g

Original: P r i v a t e E y e P
Revised: P r i v a t e E y e P

Original: i
Revised: i

Original: t
Revised: t

Original: f a ll !
Revised: f a ll !

Original: M o n t e z u m a ' s R e v e n g e
Revised: M o n t e z u m a ' s R e v e n g e

Original: S u rr o un d
Revised: S u rr o un d

Original: A s t e r o i d s
Revised: A s t e r o i d s

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: S o l a r i s B o w li n g G r a v
Revised: S o l a r i s B o w li n g G r a v

Original: i t a r H .
Revised: i t a r H .

Original: E .
Revised: E .

Original: R .
Revised: R .

Original: O .
Revised: O .

Original: B e r z e r k F r o s t b i t e T e nn i s A li e n
Revised: B e r z e r k F r o s t b i t e T e nn i s A li e n

Original: T u t a n k h a m
Revised: T u t a n k h a m

Original: R i v e rr a i d
Revised: R i v e rr a i d

Original: M s .
Revised: M s .

Original: P a c - M a n P o n g
Revised: P a c - M a n P o n g

Original: B e a m R i
Revised: B e a m R i

Original: d
Revised: d

Original: e r
Revised: e r

Original: C h
Revised: C h

Original: o pp e r
Revised: o pp e r

Original: C
Revised: C

Original: o mm a n d
Revised: o mm a n d

Original: S e a q u e s t
Revised: S e a q u e s t

Original: D o u b
Revised: D o u b

Original: l
Revised: l

Original: e
Revised: e

Original: D un k
Revised: D un k

Original: F r ee w a y
Revised: F r ee w a y

Original: Sp a c e I n v a d e r s
Revised: Sp a c e I n v a d e r s

Original: B a n k H
Revised: B a n k H

Original: e
Revised: e

Original: i s t
Revised: i s t

Original: V e n t
Revised: V e n t

Original: u
Revised: u

Original: r e
Revised: r e

Original: B a tt l e Z o n e
Revised: B a tt l e Z o n e

Original: A s t e r i x Q * B E R T A
Revised: A s t e r i x Q * B E R T A

Original: m i d a r
Revised: m i d a r

Original: N a m e T h i s
Revised: N a m e T h i s

Original: G a m e
Revised: G a m e

Original: Y a r s ' R e v e n g e
Revised: Y a r s ' R e v e n g e

Original: P h
Revised: P h

Original: o e n i x
Revised: o e n i x

Original: T i m e P il o t
Revised: T i m e P il o t

Original: D e m o n A tt a c k
Revised: D e m o n A tt a c k

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: B r e a k o u t
Revised: B r e a k o u t

Original: V i
Revised: V i

Original: d e o P i n b a ll
Revised: d e o P i n b a ll

Original: F i s h
Revised: F i s h

Original: i n g
Revised: i n g

Original: D e r b y E n d u r o
Revised: D e r b y E n d u r o

Original: S t a r g unn e r
Revised: S t a r g unn e r

Original: A ss a u l t
Revised: A ss a u l t

Original: Z a xx o n
Revised: Z a xx o n

Original: J a m e s b o n d
Revised: J a m e s b o n d

Original: C e n t
Revised: C e n t

Original: i p e d e
Revised: i p e d e

Original: K a n g a r oo A t l a n t i s
Revised: K a n g a r oo A t l a n t i s

Original: U p ' n D o w n
Revised: U p ' n D o w n

Original: D e f e n d e r B
Revised: D e f e n d e r B

Original: o
Revised: o

Original: x
Revised: x

Original: i n g R o b o t a n k
Revised: i n g R o b o t a n k

Original: K un g - F u
Revised: K un g - F u

Original: M a s t e r G o p h e r
Revised: M a s t e r G o p h e r

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: C r a z y C li m
Revised: C r a z y C li m

Original: b e r K r u ll
Revised: b e r K r u ll

Original: [Figure]
Revised: [Figure]

Original: F r a m
Revised: F r a m

Original: e s t o > hu
Revised: e s t o > hu

Original: m a n s c o r e 390M
Revised: m a n s c o r e 390M

Original: Agent
Revised: Agent

Original: MEME
Revised: MEME

Original: [Figure]
Revised: [Figure]

Original: S k ii n g
Revised: S k ii n g

Original: S u rr o un d
Revised: S u rr o un d

Original: P r i v a t e E y e
Revised: P r i v a t e E y e

Original: A t l a n t i s
Revised: A t l a n t i s

Original: D o u b
Revised: D o u b

Original: l
Revised: l

Original: e
Revised: e

Original: D un k
Revised: D un k

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: V i
Revised: V i

Original: d e o P i n b a ll
Revised: d e o P i n b a ll

Original: A s t e r o i
Revised: A s t e r o i

Original: d s B o w li n g
Revised: d s B o w li n g

Original: C h
Revised: C h

Original: o pp e r
Revised: o pp e r

Original: C
Revised: C

Original: o mm a n d
Revised: o mm a n d

Original: F r o s t b i t e P i t
Revised: F r o s t b i t e P i t

Original: f a ll !
Revised: f a ll !

Original: B a tt l e Z o n e T e nn
Revised: B a tt l e Z o n e T e nn

Original: i s A s t e r i x
Revised: i s A s t e r i x

Original: B a n k H
Revised: B a n k H

Original: e
Revised: e

Original: i s t P o n g
Revised: i s t P o n g

Original: B e a m R i d e r A li e n
Revised: B e a m R i d e r A li e n

Original: J a m e s b o n d
Revised: J a m e s b o n d

Original: P h
Revised: P h

Original: o e n i x
Revised: o e n i x

Original: M s .
Revised: M s .

Original: P a c - M a n
Revised: P a c - M a n

Original: Sp a c e I n v a d e r s G r a v
Revised: Sp a c e I n v a d e r s G r a v

Original: i t a r Q * B E R T
Revised: i t a r Q * B E R T

Original: T i m e P il o t
Revised: T i m e P il o t

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: T u t a n k h a m
Revised: T u t a n k h a m

Original: B r e a k o u t
Revised: B r e a k o u t

Original: Y a r s ' R e v e n g e B e r z e r k V e n t u r e S
Revised: Y a r s ' R e v e n g e B e r z e r k V e n t u r e S

Original: o l a r
Revised: o l a r

Original: i s
Revised: i s

Original: D e m o n A tt a c k
Revised: D e m o n A tt a c k

Original: F r ee w a y
Revised: F r ee w a y

Original: M o n t e z u m a ' s R e v e n g e
Revised: M o n t e z u m a ' s R e v e n g e

Original: F i s h
Revised: F i s h

Original: i n g
Revised: i n g

Original: D e r b y
Revised: D e r b y

Original: C e n t
Revised: C e n t

Original: i p e d e
Revised: i p e d e

Original: A ss a u l t E n d u r o
Revised: A ss a u l t E n d u r o

Original: N a m e T h i s
Revised: N a m e T h i s

Original: G a m e Z a xx o n
Revised: G a m e Z a xx o n

Original: U p ' n D o w n
Revised: U p ' n D o w n

Original: R i v e rr a i d R o b o t a n k
Revised: R i v e rr a i d R o b o t a n k

Original: S t a r g unn e r
Revised: S t a r g unn e r

Original: K a n g a r oo B o x i n g A m i d a r K r u ll
Revised: K a n g a r oo B o x i n g A m i d a r K r u ll

Original: S e a q u e s t H .
Revised: S e a q u e s t H .

Original: E .
Revised: E .

Original: R .
Revised: R .

Original: O .
Revised: O .

Original: K un g - F u
Revised: K un g - F u

Original: M a s t e r
Revised: M a s t e r

Original: D e f e n d e r G
Revised: D e f e n d e r G

Original: o
Revised: o

Original: p
Revised: p

Original: h e r
Revised: h e r

Original: C r a z y C li m b e r
Revised: C r a z y C li m b e r

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: K S TICKY ACTIONS RESULTS
Revised: K S TICKY ACTIONS RESULTS

Original: This section reproduces the main results of the paper, but enabling sticky actions (Machado et al., 2018) during both training and evaluation.
Revised: This section reproduces the main results of the paper, but enabling sticky actions (Machado et al., 2018) during both training and evaluation.

Original: Since our agent does not exploit the determinism in the original Atari benchmark, it is still able to outperform the human baseline with sticky actions enabled.
Revised: Since our agent does not exploit the determinism in the original Atari benchmark, it is still able to outperform the human baseline with sticky actions enabled.

Original: We observe a slight decrease in mean scores, which we attribute to label corruption in the action prediction loss used to learn the controllable representations used in the episodic reward computation: due to the implementation of sticky actions proposed by Machado et al.
Revised: We observe a slight decrease in mean scores, which we attribute to label corruption in the action prediction loss used to learn the controllable representations used in the episodic reward computation: due to the implementation of sticky actions proposed by Machado et al.

Original: the agent actions are ignored in a fraction of the timesteps.
Revised: the agent actions are ignored in a fraction of the timesteps.

Original: This phenomenon is aggravated by the frame stacking used in the standard Atari pre-processing, as the action being executed in the environment can vary within each stack of frames.
Revised: This phenomenon is aggravated by the frame stacking used in the standard Atari pre-processing, as the action being executed in the environment can vary within each stack of frames.

Original: We hypothesize that the gap between the two versions of the environment would be much smaller with a different implementation of sticky actions that did not corrupt the action labels used by the representation learning module.
Revised: We hypothesize that the gap between the two versions of the environment would be much smaller with a different implementation of sticky actions that did not corrupt the action labels used by the representation learning module.

Original: All reported results are the average over six different random seeds.
Revised: All reported results are the average over six different random seeds.

Original: [Figure]
Revised: [Figure]

Original: P i t
Revised: P i t

Original: f a ll !
Revised: f a ll !

Original: M o n t e z u m a ' s R e v e n g e
Revised: M o n t e z u m a ' s R e v e n g e

Original: H .
Revised: H .

Original: E .
Revised: E .

Original: R .
Revised: R .

Original: O .
Revised: O .

Original: P r i v a t e E y e S k ii n g S
Revised: P r i v a t e E y e S k ii n g S

Original: o
Revised: o

Original: l a r i s
Revised: l a r i s

Original: S e a q u e s t
Revised: S e a q u e s t

Original: A m i d a r B e r z e r k R
Revised: A m i d a r B e r z e r k R

Original: i v e rr a i d
Revised: i v e rr a i d

Original: C h
Revised: C h

Original: o pp e r
Revised: o pp e r

Original: C
Revised: C

Original: o mm a n d
Revised: o mm a n d

Original: G r a v i t a r A s t
Revised: G r a v i t a r A s t

Original: e
Revised: e

Original: r
Revised: r

Original: o i
Revised: o i

Original: d s P o n g
Revised: d s P o n g

Original: N a m e T h i s
Revised: N a m e T h i s

Original: G a m e A li e n
Revised: G a m e A li e n

Original: T u t a n k h a m
Revised: T u t a n k h a m

Original: V e n t
Revised: V e n t

Original: u
Revised: u

Original: r e
Revised: r e

Original: M s .
Revised: M s .

Original: P a c - M a n
Revised: P a c - M a n

Original: F r ee w a y
Revised: F r ee w a y

Original: Sp a c e I n v a d e r s
Revised: Sp a c e I n v a d e r s

Original: S u rr o un d
Revised: S u rr o un d

Original: Q * B E R T B o w li n g
Revised: Q * B E R T B o w li n g

Original: B e a m R i
Revised: B e a m R i

Original: d
Revised: d

Original: e r
Revised: e r

Original: I c e H o c k e y
Revised: I c e H o c k e y

Original: B a n k H
Revised: B a n k H

Original: e
Revised: e

Original: i s t
Revised: i s t

Original: E n d u r o F r o s t b
Revised: E n d u r o F r o s t b

Original: i t e
Revised: i t e

Original: W i z a r d
Revised: W i z a r d

Original: O f W o r
Revised: O f W o r

Original: S t a r g unn e r
Revised: S t a r g unn e r

Original: Y a r s ' R e v e n g e
Revised: Y a r s ' R e v e n g e

Original: F i s h
Revised: F i s h

Original: i n g
Revised: i n g

Original: D e r b y P
Revised: D e r b y P

Original: h
Revised: h

Original: o e n i x
Revised: o e n i x

Original: D e m o n A tt a c k
Revised: D e m o n A tt a c k

Original: B a tt l e Z o n e
Revised: B a tt l e Z o n e

Original: Z a xx o n A ss a u l t
Revised: Z a xx o n A ss a u l t

Original: T i m e P il o t
Revised: T i m e P il o t

Original: B r e a k o u t A s t e r i x D e f e n d e r
Revised: B r e a k o u t A s t e r i x D e f e n d e r

Original: K un g - F u
Revised: K un g - F u

Original: M a s t e r G o p h e r
Revised: M a s t e r G o p h e r

Original: K a n g a r oo B o x i n g
Revised: K a n g a r oo B o x i n g

Original: R o a d R unn e r
Revised: R o a d R unn e r

Original: C e n t
Revised: C e n t

Original: i p e d e
Revised: i p e d e

Original: U p ' n D o w n
Revised: U p ' n D o w n

Original: V i
Revised: V i

Original: d e o P i n b a ll
Revised: d e o P i n b a ll

Original: J a m e s b o n d
Revised: J a m e s b o n d

Original: D o u b
Revised: D o u b

Original: l
Revised: l

Original: e
Revised: e

Original: D un k
Revised: D un k

Original: C r a z y C li m
Revised: C r a z y C li m

Original: b e r K r u ll
Revised: b e r K r u ll

Original: R o b o t a n k
Revised: R o b o t a n k

Original: A t l a n t i s T e nn i s 1M 10M 100M 1B
Revised: A t l a n t i s T e nn i s [Figure] 1B

Original: F r a m
Revised: F r a m

Original: e s t o > hu
Revised: e s t o > hu

Original: m a n s c o r e 505M
Revised: m a n s c o r e 505M

Original: [Figure]
Revised: [Figure]

Original: Metrics for previous methods are computed using the final score per game reported in their respective publications: MuZero (Schrittwieser et al., 2021), Muesli (Hessel et al., 2021).
Revised: Metrics for previous methods are computed using the final score per game reported in their respective publications: MuZero (Schrittwieser et al., 2021), Muesli (Hessel et al., 2021).

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: L E FFECT OF S AMPLES PER I NSERT R ATIO
Revised: L E FFECT OF S AMPLES PER I NSERT R ATIO

Original: Results of the ablation on the amount of replay that the learner performs per sequence of experience that the actors produce can be seen in Figure 15.
Revised: Results of the ablation on the amount of replay that the learner performs per sequence of experience that the actors produce can be seen in Figure 15.

Original: We can see that, while a samples per insert ratio (SPI) of 10 still provides moderate boosts in data efficiency in games such as H
Revised: We can see that, while a samples per insert ratio (SPI) of 10 still provides moderate boosts in data efficiency in games such as H

Original: [Figure]
Revised: [Figure]

Original: E
Revised: E

Original: [Figure]
Revised: [Figure]

Original: R
Revised: R

Original: [Figure]
Revised: [Figure]

Original: O. , Montezuma’s
Revised: O. , Montezuma’s

Original: Revenge and Pitfall!
Revised: Revenge and Pitfall!

Original: , it is not as pronounced as the increase that is seen from SPI of 3 to 6 .
Revised: , it is not as pronounced as the increase that is seen from SPI of 3 to 6 .

Original: This implies that with an SPI of 10 we obtain a much worse return in terms of wall-clock time as we replay more frequently.
Revised: This implies that with an SPI of 10 we obtain a much worse return in terms of wall-clock time as we replay more frequently.

Original: [Figure]
Revised: [Figure]

Revised: Tennis

Original: M R2D2 ABLATIONS
Revised: M R2D2 ABLATIONS

Original: We further study the benefits of the proposed components in an agent-agnostic context, by examining their performance when used with the R2D2 agent (Kapturowski et al., 2018), as shown in Figure 16.
Revised: We further study the benefits of the proposed components in an agent-agnostic context, by examining their performance when used with the R2D2 agent (Kapturowski et al., 2018), as shown in Figure 16.

Original: Similarly to the original ablations of MEME, we can see that the trust region and online bootstrapping constitute the most important component, which once removed deteriorates the performance significantly.
Revised: Similarly to the original ablations of MEME [Figure] we can see that the trust region and online bootstrapping constitute the most important component, which once removed deteriorates the performance significantly.

Original: In general, the rest of the components are beneficial as well, as MEME R2D2 can be seen to obtain human-level performance on the largest number of games; although the set of Atari games used in this ablation is not fully representative as Montezuma’s Revenge , Pitfall!
Revised: In general, the rest of the components are beneficial as well, as MEME R2D2 can be seen to obtain human-level performance on the largest number of games; although the set of Atari games used in this ablation is not fully representative as Montezuma’s Revenge , Pitfall!

Original: and Skiing are not expected to be solvable without intrinsic motivation or the ability to leverage larger discount factors.
Revised: and Skiing are not expected to be solvable without intrinsic motivation or the ability to leverage larger discount factors.

Original: [Figure]
Revised: [Figure]

Original: N
Revised: N

Original: F ULL ABLATIONS
Revised: F ULL ABLATIONS

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: O S CORES PER GAME
Revised: O S CORES PER GAME

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]