Original: Fast and Data-Efﬁcient Training of Rainbow: an Experimental Study on Atari
Revised: Fast and Data-Efﬁcient Training of Rainbow: an Experimental Study on Atari

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Across the Arcade Learning Environment, Rainbow achieves a level of performancecompetitive with humans and modern RL algorithms.
Revised: Across the Arcade Learning Environment, Rainbow achieves a level of performance competitive with humans and modern RL algorithms.

Original: However, attaining this levelof performance requires large amounts of data and hardware resources, makingresearch in this area computationally expensive and use in practical applicationsoften infeasible.
Revised: However, attaining this level of performance requires large amounts of data and hardware resources, making research in this area computationally expensive and use in practical applications often infeasible.

Original: This paper’s contribution is threefold: We (1) propose an improvedversion of Rainbow, seeking to drastically reduce Rainbow’s data, training time,and compute requirements while maintaining its competitive performance; (2) weempirically demonstrate the effectiveness of our approach through experimentson the Arcade Learning Environment, and (3) we conduct a number of ablationstudies to investigate the effect of the individual proposed modiﬁcations.
Revised: This paper’s contribution is threefold: We (1) propose an improved version of Rainbow, seeking to drastically reduce Rainbow’s data, training time, and compute requirements while maintaining its competitive performance; (2) we empirically demonstrate the effectiveness of our approach through experiments on the Arcade Learning Environment, and (3) we conduct a number of ablation studies to investigate the effect of the individual proposed modiﬁcations.

Original: Ourimproved version of Rainbow reaches a median human normalized score close toclassic Rainbow’s, while using 20 times less data and requiring only 7.5 hours oftraining time on a single GPU.
Revised: Our improved version of Rainbow reaches a median human normalized score close to classic Rainbow’s, while using 20 times less data and requiring only 7.5 hours of training time on a single GPU.

Original: We also provide our full implementation includingpre-trained models.
Revised: We also provide our full implementation including pre-trained models.

Original:  Introduction
Revised:  Introduction

Original: In2013, the Deep Q-Networks (DQN) algorithm [Mnih et al., 2013] kicked off a ﬂurry of developments in the classic Atari reinforcement learning (RL) benchmark.
Revised: In the Deep Q-Networks (DQN) algorithm [Mnih et al., 2013] kicked off a ﬂurry of developments in the classic Atari reinforcement learning (RL) benchmark.

Original: In this benchmark, the agent istasked with learning to play Atari 2600 games solely based on pixel inputs and rewards.
Revised: In this benchmark, the agent is tasked with learning to play Atari 2600 games solely based on pixel inputs and rewards.

Original: While recentmethods such as Agent57, R2D2, NGU, and MuZero [Badia et al., 2020a, Kapturowski et al., 2019,
Revised: While recent methods such as Agent57, R2D2, NGU and MuZero [Badia et al., 2020a, Kapturowski et al., 2019,

Original: Badia et al., 2020b, Schrittwieser et al., 2020] can now achieve above-human level performance inmost or all games, a number of critical issues remain.
Revised: Badia et al., 2020b, Schrittwieser et al., 2020] can now achieve above-human level performance in most or all games, a number of critical issues remain.

Original: Among these are the often vast data, time,and compute requirements for training these agents.
Revised: Among these are the often vast data, time, and compute requirements for training these agents.

Original: Agent57, the only RL algorithm to achievesuper-human performance on all 57 Atari games, required in the order of a century (78 billion frames)of gameplay experience in order to beat the last game, Atari Skiing [Badia et al., 2020a].
Revised: Agent57, the only RL algorithm to achieve super-human performance on all 57 Atari games, required in the order of a century (78 billion frames) of gameplay experience in order to beat the last game, Atari Skiing [Badia et al., 2020a].

Original: Rainbow [Hessel et al., 2018], introduced in 2017 and itself based on DQN, represents an importantmilestone in the development of the above-mentioned agents, acting as a foundation for Agent57 andother algorithms [Badia et al., 2020a, Kapturowski et al., 2019].
Revised: Rainbow [Hessel et al., 2018], introduced in 2017 and itself based on DQN, represents an important milestone in the development of the above-mentioned agents, acting as a foundation for Agent57 and other algorithms [Badia et al., 2020a, Kapturowski et al., 2019].

Original: In the past, Rainbow has also servedas a useful baseline for the development of new environments [Cobbe et al., 2020, Nichol et al., 2018].
Revised: In the past, Rainbow has also served as a useful baseline for the development of new environments [Cobbe et al., 2020, Nichol et al., 2018].

Original: In this paper, we revisit the Rainbow algorithm and review some of the original design choices.
Revised: In this paper, we revisit the Rainbow algorithm and review some of the original design choices.

Original: Bycombining Rainbow with some more recent techniques and by simplifying the implementation, weseek to strike a good balance between performance, data efﬁciency, training time, and implementationcomplexity.
Revised: By combining Rainbow with some more recent techniques and by simplifying the implementation, we seek to strike a good balance between performance, data efﬁciency, training time, and implementation complexity.

Original: Aside from the Atari benchmark, reinforcement learning has further achieved remarkable accomplishments in other domains, such as beating the human champions in some of the world’s most difﬁcultboard games [Silver et al., 2016, 2017, 2018], as well as in strategic real-time combat video games [Vinyals et al., 2019b, Berner et al., 2019].
Revised: Aside from the Atari benchmark, reinforcement learning has further achieved remarkable accomplishments in other domains, such as beating the human champions in some of the world’s most difﬁcult board games [Silver et al., 2016, 2017, 2018], as well as in strategic real-time combat video games [Vinyals et al., 2019b, Berner et al., 2019].

Original: Other prominent demonstrations of the effectiveness of
Revised: Other prominent demonstrations of the effectiveness of

Revised: Deep Reinforcement Learning Workshop at the 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

Original: RL are optimizing the energy consumption of data centers [Lazic et al., 2018], dexterous roboticmanipulation of physical objects [Akkaya et al., 2019, Andrychowicz et al., 2020], and designingmore efﬁcient chip layouts [Mirhoseini et al., 2020] and neural architectures [Zoph and Le, 2017,
Revised: RL are optimizing the energy consumption of data centers [Lazic et al., 2018], dexterous robotic manipulation of physical objects [Akkaya et al., 2019, Andrychowicz et al., 2020], and designing more efﬁcient chip layouts [Mirhoseini et al., 2020] and neural architectures [Zoph and Le, 2017,

Original: Pham et al., 2018].
Revised: Pham et al., 2018].

Original: In these domains too, however, RL-based approaches can suffer from the issuesof lacking data efﬁciency and requiring large amounts of compute resources, among others.
Revised: In these domains too, however, RL-based approaches can suffer from the issues of lacking data efﬁciency and requiring large amounts of compute resources, among others.

Original: Indeed,some of the most remarkable results achieved in these tasks required up to hundreds or even thousandsof years worth of simulated environment interaction experience [Vinyals et al., 2019b,a, Berner et al. ,2019, Akkaya et al., 2019].
Revised: Indeed, some of the most remarkable results achieved in these tasks required up to hundreds or even thousands of years worth of simulated environment interaction experience [Vinyals et al., 2019b,a, Berner et al., 2019, Akkaya et al., 2019].

Original: This severely limits the applicability of RL to many real-world tasks,where gathering such large amounts of experience is often infeasible, and the required hardwareresources impose a substantial ﬁnancial burden.
Revised: This severely limits the applicability of RL to many real-world tasks, where gathering such large amounts of experience is often infeasible, and the required hardware resources impose a substantial ﬁnancial burden.

Original: One prominent stream of research that addresses these issues is data-efﬁcient RL.
Revised: One prominent stream of research that addresses these issues is data-efﬁcient RL.

Original: This area refers to aset of algorithmic improvements to the RL agent that aim to enable it to learn from fewer environmentinteraction steps and shorten the training time the RL agent requires in order to learn the desiredbehavior.
Revised: This area refers to a set of algorithmic improvements to the RL agent that aim to enable it to learn from fewer environment interaction steps and shorten the training time the RL agent requires in order to learn the desired behavior.

Original: In recent years, this area has started to receive lots of attention within the RL community.
Revised: In recent years, this area has started to receive lots of attention within the RL community.

Original: Therefore, we discuss the literature on data-efﬁcient RL in more detail in Section 2.
Revised: Therefore, we discuss the literature on data-efﬁcient RL in more detail in Section 2.

Original: Other, more practical approaches attempt to explicitly shorten the time required to reach the learningobjective by speeding up the RL training process itself.
Revised: Other, more practical approaches attempt to explicitly shorten the time required to reach the learning objective by speeding up the RL training process itself.

Original: This can be achieved, for example, byemploying simple tricks that are commonly used in other areas of machine learning but havepreviously rarely been used in RL.
Revised: This can be achieved, for example, by employing simple tricks that are commonly used in other areas of machine learning but have previously rarely been used in RL.

Original: These include techniques to improve hardware utilization, mixedprecision training [Micikevicius et al., 2018] and methods that aim to stabilize the training process.
Revised: These include techniques to improve hardware utilization, mixedprecision training [Micikevicius et al., 2018] and methods that aim to stabilize the training process.

Original: 1.1 Contributions
Revised: 1.1 Contributions

Original: In this work, we integrate techniques from both streams of research into the established Rainbow agentwhile re-evaluating a number of Rainbow’s original design choices.
Revised: In this work, we integrate techniques from both streams of research into the established Rainbow agent while re-evaluating a number of Rainbow’s original design choices.

Original: Among others, we investigatewhat effect a larger batch size, a bigger network, a higher learning rate, spectral normalization, andmixed precision have on the training speed and the RL agent’s overall performance.
Revised: Among others, we investigate what effect a larger batch size, a bigger network, a higher learning rate, spectral normalization, and mixed precision have on the training speed and the RL agent’s overall performance.

Original: While most ofthese tricks/enhancements have been studied individually in the context of RL, to the best of ourknowledge, our work is the ﬁrst to study them jointly.
Revised: While most of these tricks/enhancements have been studied individually in the context of RL, to the best of our knowledge, our work is the ﬁrst to study them jointly.

Original: We discuss their appearance in the literature inmore detail in Section 2.
Revised: We discuss their appearance in the literature in more detail in Section 2.

Original: To study these improvements, we rely on the well-established Atari benchmark.
Revised: To study these improvements, we rely on the well-established Atari benchmark.

Original: The Arcade Learning
Revised: The Arcade Learning

Original: Environment (ALE) has long been an important challenge for RL research [Bellemare et al., 2013,
Revised: Environment (ALE) has long been an important challenge for RL research [Bellemare et al., 2013,

Original: Machado et al., 2018] and has particularly entered the spotlight since the introduction of Deep
Revised: Machado et al., 2018] and has particularly entered the spotlight since the introduction of Deep

Original: Q-Networks (DQN)
Revised: Q-Networks (DQN)

Original: [Mnih et al., 2013, 2015].
Revised: [Mnih et al., 2013, 2015].

Original: DQN combines Q-Learning with deep convolutionalneural networks (CNNs) and experience replay to learn to play Atari games at human-level performance, solely from pixels and with no prior human knowledge.
Revised: DQN combines Q-Learning with deep convolutional neural networks (CNNs) and experience replay to learn to play Atari games at human-level performance, solely from pixels and with no prior human knowledge.

Original: Since its inception, a variety ofenhancements to the original DQN architecture have been proposed [van Hasselt et al., 2016, Wanget al., 2016, Schaul et al., 2016, Sutton and Barto, 2018, Fortunato et al., 2018, Bellemare et al., 2017],many of which have since been integrated into a single uniﬁed algorithm, Rainbow-DQN [Hesselet al., 2018].
Revised: Since its inception, a variety of enhancements to the original DQN architecture have been proposed [van Hasselt et al., 2016, Wang et al., 2016, Schaul et al., 2016, Sutton and Barto, 2018, Fortunato et al., 2018, Bellemare et al., 2017], many of which have since been integrated into a single uniﬁed algorithm, Rainbow-DQN [Hessel et al., 2018].

Original: The introduction of Rainbow represents a critical milestone in the development of RLalgorithms, and it has ever since been one of the best-performing and most established algorithms for
Revised: The introduction of Rainbow represents a critical milestone in the development of RL algorithms, and it has ever since been one of the best-performing and most established algorithms for

Original: Atari.
Revised: Atari.

Original: Overall, we make the following contributions in this paper:
Revised: Overall, we make the following contributions in this paper

Original:  • We propose an improved version of Rainbow, seeking to reduce wall-clock training time and required hardware resources while improving data efﬁciency and maintaining competitiveperformance.
Revised: : • We propose an improved version of Rainbow, seeking to reduce wall-clock training time and required hardware resources while improving data efﬁciency and maintaining competitive performance.

Original: • We empirically demonstrate the effectiveness of our approach on the Atari benchmark,achieving a median human normalized score close to that of classic Rainbow-DQN whileusing 20 times less data.
Revised: • We empirically demonstrate the effectiveness of our approach on the Atari benchmark, achieving a median human normalized score close to that of classic Rainbow-DQN while using 20 times less data.

Original: • We conduct a number of ablation studies to investigate the effect of the proposed modiﬁca- tions individually and in aggregate.
Revised: • We conduct a number of ablation studies to investigate the effect of the proposed modiﬁcations individually and in aggregate.

Original:  Related Work
Revised:  Related Work

Original: This section covers some recent directions of research in deep reinforcement learning that areorthogonal to our work.
Revised: This section covers some recent directions of research in deep reinforcement learning that are orthogonal to our work.

Original: 2.1 Data-efﬁcient RL: the algorithmic solution
Revised: 2.1 Data-efﬁcient RL: the algorithmic solution

Original: One prominent stream of research related to our work is data-efﬁcient RL.
Revised: One prominent stream of research related to our work is data-efﬁcient RL.

Original: Data-efﬁcient RL aims tolearn from fewer environment interaction steps by leveraging the information at the agent’s disposalmore effectively.
Revised: Data-efﬁcient RL aims to learn from fewer environment interaction steps by leveraging the information at the agent’s disposal more effectively.

Original: Recently, this has become an attractive area of research, as current RL algorithmscan be incredibly data-inefﬁcient.
Revised: Recently, this has become an attractive area of research, as current RL algorithms can be incredibly data-inefﬁcient.

Original: In fact, data inefﬁciency has been identiﬁed as one of the majorobstacles towards widespread adoption of RL in the real world [Dulac-Arnold et al., 2019, 2020].
Revised: In fact, data inefﬁciency has been identiﬁed as one of the major obstacles towards widespread adoption of RL in the real world [Dulac-Arnold et al., 2019, 2020].

Original: Different solutions have been proposed to mitigate this limitation.
Revised: Different solutions have been proposed to mitigate this limitation.

Original: Three of the most promising onesare (1) self-supervised learning, (2) model-based RL, and (3) data augmentation.
Revised: Three of the most promising ones are (1) self-supervised learning, (2) model-based RL, and (3) data augmentation.

Original: Self-supervised learning.
Revised: Self-supervised learning.

Original: Self-supervised methods in RL aim to learn more effective representationsof the environment by optimizing a self-supervised auxiliary objective in addition to the primary RLobjective.
Revised: Self-supervised methods in RL aim to learn more effective representations of the environment by optimizing a self-supervised auxiliary objective in addition to the primary RL objective.

Original: Self-supervised learning (SSL) is the driving force behind recent advances in NLP [Devlinet al., 2018, Brown et al., 2020], computer vision [Chen et al., 2020, Grill et al., 2020] and speechrecognition [Baevski et al., 2020].
Revised: Self-supervised learning (SSL) is the driving force behind recent advances in NLP [Devlin et al., 2018, Brown et al., 2020], computer vision [Chen et al., 2020, Grill et al., 2020] and speech recognition [Baevski et al., 2020].

Original: Only recently, SSL has started to receive widespread attentionfrom the RL community and shown to improve data efﬁciency considerably.
Revised: Only recently, SSL has started to receive widespread attention from the RL community and shown to improve data efﬁciency considerably.

Original: Prominent examples are
Revised: Prominent examples are

Original: Unreal [Jaderberg et al., 2017], CURL [Laskin et al., 2020a], SPR [Schwarzer et al., 2020], and SGI[Schwarzer et al., 2021].
Revised: Unreal [Jaderberg et al., 2017], CURL [Laskin et al., 2020a], SPR [Schwarzer et al., 2020], and SGI [Schwarzer et al., 2021].

Original: Model-based methods.
Revised: Model-based methods.

Original: Model-based RL algorithms, in contrast, learn an explicit model of theenvironment and then use this model to derive a good policy by simulating experience, a processknown as planning [Sutton and Barto, 2018].
Revised: Model-based RL algorithms, in contrast, learn an explicit model of the environment and then use this model to derive a good policy by simulating experience, a process known as planning [Sutton and Barto, 2018].

Original: A major advantage, again, is more data-efﬁcientlearning, but these methods have historically proven difﬁcult in environments with large and complexobservation spaces [Moerland et al., 2020].
Revised: A major advantage, again, is more data-efﬁcient learning, but these methods have historically proven difﬁcult in environments with large and complex observation spaces [Moerland et al., 2020].

Original: However, Schrittwieser et al.
Revised: However, Schrittwieser et al.

Original: [2020] recently introduced
Revised: [2020] recently introduced

Original: MuZero, a model-based RL algorithm that achieves state-of-the-art performance on the Arcade
Revised: MuZero, a model-based RL algorithm that achieves state-of-the-art performance on the Arcade

Original: Learning Environment [Bellemare et al., 2013].
Revised: Learning Environment [Bellemare et al., 2013].

Original: Other prominent examples of model-based RLmethods include World Models [Ha and Schmidhuber, 2018], SimPle [Kaiser et al., 2019] and
Revised: Other prominent examples of model-based RL methods include World Models [Ha and Schmidhuber, 2018], SimPle [Kaiser et al., 2019] and

Original: Dreamer [Hafner et al., 2019, 2020].
Revised: Dreamer [Hafner et al., 2019, 2020].

Original: A detailed overview of the model-based RL literature is givenby Moerland et al.
Revised: A detailed overview of the model-based RL literature is given by Moerland et al.

Original: Data augmentation.
Revised: Data augmentation.

Original: Another option to improve data efﬁciency in RL is data augmentation, a wellestablished technique in many ML domains, most notably for image-based tasks [Perez and Wang,2017, Shorten and Khoshgoftaar, 2019].
Revised: Another option to improve data efﬁciency in RL is data augmentation, a wellestablished technique in many ML domains, most notably for image-based tasks [Perez and Wang, 2017, Shorten and Khoshgoftaar, 2019].

Original: Data augmentation aims to artiﬁcially increase the amountof available training data by modifying existing observations in some meaningful way.
Revised: Data augmentation aims to artiﬁcially increase the amount of available training data by modifying existing observations in some meaningful way.

Original: For RL, thissimple technique has been shown to improve data efﬁciency considerably, as demonstrated by RAD[Laskin et al., 2020b] and DrQ [Yarats et al., 2020].
Revised: For RL, this simple technique has been shown to improve data efﬁciency considerably, as demonstrated by RAD [Laskin et al., 2020b] and DrQ [Yarats et al., 2020].

Original: 2.2 Speeding up RL training: the practical solution
Revised: 2.2 Speeding up RL training: the practical solution

Original: Both the approaches discussed in this section, as well as the data-efﬁcient methods from the previoussection, deal with the same fundamental question: how to accelerate the training of reinforcementlearning agents.
Revised: Both the approaches discussed in this section, as well as the data-efﬁcient methods from the previous section, deal with the same fundamental question: how to accelerate the training of reinforcement learning agents.

Original: On the one hand, data-efﬁcient methods implicitly shorten the training time bycutting down the number of required training samples.
Revised: On the one hand, data-efﬁcient methods implicitly shorten the training time by cutting down the number of required training samples.

Original: The methods discussed in this section, on theother hand, seek to shorten the required training time without fundamentally modifying the learningalgorithm.
Revised: The methods discussed in this section, on the other hand, seek to shorten the required training time without fundamentally modifying the learning algorithm.

Original: These methods are not unique to deep RL but rather are generally applicable to many deeplearning algorithms.
Revised: These methods are not unique to deep RL but rather are generally applicable to many deep learning algorithms.

Original: We do not cover approaches that speed up training through parallelization sincethese simply trade hardware requirements for training time and thus do not alleviate the ﬁnancialburden of training large models.
Revised: We do not cover approaches that speed up training through parallelization since these simply trade hardware requirements for training time and thus do not alleviate the ﬁnancial burden of training large models.

Original: Larger batch sizes.
Revised: Larger batch sizes.

Original: For image-based RL tasks, agents are typically trained with relatively smallbatch sizes [Mnih et al., 2015, Hessel et al., 2018].
Revised: For image-based RL tasks, agents are typically trained with relatively small batch sizes [Mnih et al., 2015, Hessel et al., 2018].

Original: However, prior work has demonstrated that, on theprocgen and ALE benchmarks, increasing the batch size can speed up the training time considerably[Cobbe et al., 2020, Stooke and Abbeel, 2018].
Revised: However, prior work has demonstrated that, on the procgen and ALE benchmarks, increasing the batch size can speed up the training time considerably [Cobbe et al., 2020, Stooke and Abbeel, 2018].

Original: Bigger networks.
Revised: Bigger networks.

Original: Compared to other branches of deep learning, the networks employed by deep
Revised: Compared to other branches of deep learning, the networks employed by deep

Original: RL agents are relatively shallow [Mnih et al., 2015].
Revised: RL agents are relatively shallow [Mnih et al., 2015].

Original: Replacing the standard network architecture with bigger networks has proven a fruitful direction in prior work [Espeholt et al., 2018, Cobbe et al. ,2020, Sinha et al., 2020, Bjorck et al., 2021].
Revised: Replacing the standard network architecture with bigger networks has proven a fruitful direction in prior work [Espeholt et al., 2018, Cobbe et al., 2020, Sinha et al., 2020, Bjorck et al., 2021].

Original: Spectral normalization.
Revised: Spectral normalization.

Original: Recently, Gogianu et al.
Revised: Recently, Gogianu et al.

Original: [2021] proposed to apply Spectral Normalization(SN), a technique that originates from the literature on Generative Adversarial Networks (GANs)[Miyato et al., 2018, Kurach et al., 2019], to the RL setting.
Revised: [2021] proposed to apply Spectral Normalization (SN), a technique that originates from the literature on Generative Adversarial Networks (GANs) [Miyato et al., 2018, Kurach et al., 2019], to the RL setting.

Original: The authors observed that a Categorical
Revised: The authors observed that a Categorical

Original: DQN agent (C51) [Bellemare et al., 2017] augmented with SN achieves similar results as a full
Revised: DQN agent (C51) [Bellemare et al., 2017] augmented with SN achieves similar results as a full

Original: Rainbow DQN implementation [Hessel et al., 2018].
Revised: Rainbow DQN implementation [Hessel et al., 2018].

Original: Mixed precision.
Revised: Mixed precision.

Original: Another prominent and well-established option to accelerate the training of neuralnetworks is mixed precision, initially proposed by Micikevicius et al.
Revised: Another prominent and well-established option to accelerate the training of neural networks is mixed precision, initially proposed by Micikevicius et al.

Original: In RL, Lam et al.[2019] and Björck et al.
Revised: In RL, Lam et al. [2019] and Björck et al.

Original: [2021] were the ﬁrst to integrate low/mixed precision and saw substantialimprovements in training time.
Revised: [2021] were the ﬁrst to integrate low/mixed precision and saw substantial improvements in training time.

Original: 2.3 RL & the Atari benchmark: a lasting relationship
Revised: 2.3 RL & the Atari benchmark: a lasting relationship

Original: The Arcade Learning Environment (ALE)
Revised: The Arcade Learning Environment (ALE)

Original: [Bellemare et al., 2013, Machado et al., 2018] is a lastingand indispensable element of the RL researcher’s toolbox.
Revised: [Bellemare et al., 2013, Machado et al., 2018] is a lasting and indispensable element of the RL researcher’s toolbox.

Original: It is also the focus of our work.
Revised: It is also the focus of our work.

Original: Sinceits inception, hundreds of RL algorithms have been developed, and the achieved scores have onlyincreased over time.
Revised: Since its inception, hundreds of RL algorithms have been developed, and the achieved scores have only increased over time.

Original: The ﬁrst model-free architecture to raise public interest on the Atari benchmarkwas Deep Q-Networks (DQN)
Revised: The ﬁrst model-free architecture to raise public interest on the Atari benchmark was Deep Q-Networks (DQN)

Original: [Mnih et al., 2013, 2015].
Revised: [Mnih et al., 2013, 2015].

Original: Several enhancements to DQN have beenproposed.
Revised: Several enhancements to DQN have been proposed.

Original: including Double DQN [van Hasselt et al., 2016], Dueling DQN [Wang et al., 2016],
Revised: including Double DQN [van Hasselt et al., 2016], Dueling DQN [Wang et al., 2016],

Original: Prioritized Experience Replay [Schaul et al., 2016], Multi-step Bootstrapping [Sutton and Barto,2018], Noisy Nets [Fortunato et al., 2018], and Distributional RL [Bellemare et al., 2017].
Revised: Prioritized Experience Replay [Schaul et al., 2016], Multi-step Bootstrapping [Sutton and Barto, 2018], Noisy Nets [Fortunato et al., 2018], and Distributional RL [Bellemare et al., 2017].

Original: Rainbow DQN, the architecture we leverage in this work, integrates all these enhancements into asingle uniﬁed framework and shows that they are largely compatible [Hessel et al., 2018].
Revised: Rainbow DQN, the architecture we leverage in this work, integrates all these enhancements into a single uniﬁed framework and shows that they are largely compatible [Hessel et al., 2018].

Original: A slightlymodiﬁed version of Rainbow introduced by van Hasselt et al.
Revised: A slightly modiﬁed version of Rainbow introduced by van Hasselt et al.

Original: [2019] and referred to as Data-efﬁcient
Revised: [2019] and referred to as Data-efﬁcient

Original: Rainbow trades off reduced computational efﬁciency in exchange for signiﬁcantly improved dataefﬁciency.
Revised: Rainbow trades off reduced computational efﬁciency in exchange for signiﬁcantly improved data efﬁciency.

Original: R2D2 [Kapturowski et al., 2019] and Agent57 [Badia et al., 2020a], distributed RL algorithms, followin the lineage of Rainbow.
Revised: R2D2 [Kapturowski et al., 2019] and Agent57 [Badia et al., 2020a], distributed RL algorithms, follow in the lineage of Rainbow.

Original: Similar distributed training strategies were employed by Gorila [Nair et al.
Revised: Similar distributed training strategies were employed by Gorila [Nair et al., 2015],

Original: ,2015], Ape-X [Horgan et al., 2018], IMPALA [Espeholt et al., 2018] and Reactor [Gruslys et al.
Revised: Ape-X [Horgan et al., 2018], IMPALA [Espeholt et al., 2018] and Reactor [Gruslys et al., 2018].

Original: Like IMPALA, R2D2 also uses recurrent neural networks to take advantage of longer statehistories.
Revised: Like IMPALA, R2D2 also uses recurrent neural networks to take advantage of longer state histories.

Original: Agent57 builds off of the methods developed in Kapturowski et al.
Revised: Agent57 builds off of the methods developed in Kapturowski et al.

Original: [2019] but combinesthem with two intrinsic reward-based exploration mechanisms: the curiosity-based Random Network
Revised: but combines them with two intrinsic reward-based exploration mechanisms: the curiosity-based Random Network

Original: Distillation [Burda et al., 2019] and Never Give Up exploration [Badia et al., 2020b], aimed atincreasing long and short term state-space coverage, respectively.
Revised: Distillation [Burda et al., 2019] and Never Give Up exploration [Badia et al., 2020b], aimed at increasing long and short term state-space coverage, respectively.

Original: Additionally, a UCB bandit [Suttonand Barto, 2018] algorithm is used to control the amount of exploration and the value of the discountfactor in each of the distributed actors.
Revised: Additionally, a UCB bandit [Sutton and Barto, 2018] algorithm is used to control the amount of exploration and the value of the discount factor in each of the distributed actors.

Original:  Preliminaries
Revised:  Preliminaries

Original: In this section, we brieﬂy discuss the theoretical foundations of this work.
Revised: In this section, we brieﬂy discuss the theoretical foundations of this work.

Original: 3.1 The RL framework
Revised: 3.1 The RL framework

Original: Markov decision process.
Revised: Markov decision process.

Original: The fundamental goal of RL is to learn how to behave in an unknown environment in order to maximize a scalar reward signal.
Revised: The fundamental goal of RL is to learn how to behave in an unknown environment in order to maximize a scalar reward signal.

Original: In each step of this sequential decision-makingtask, the agent observes the environment’s current state s t , performs some action a t , transitions to thenext state s t , and obtains a reward r t [Sutton and Barto, 2018].
Revised: In each step of this sequential decision-making task, the agent observes the environment’s current state s t , performs some action a t , transitions to the next state s t , and obtains a reward r t [Sutton and Barto, 2018].

Original: We assume the standard formulationof the Markov Decision Process (MDP) with (cid:104)S , A , R, P (cid:105) where
Revised: We assume the standard formulation of the Markov Decision Process (MDP) with (cid:104)S , A , R, P (cid:105) where

Original: • S , A are state and action space, respectively • R : S ×
Revised: • S , A are state and action space, respectively • R : S ×

Original: A × S → R is the reward function that assigns each transition from state s t to state s t +1 via action a t a real valued reward r t =
Revised: A × S → R is the reward function that assigns each transition from state s t to state s t +1 via action a t a real valued reward r t =

Original: R ( s t , a t , s t +1 )
Revised: R ( s t , a t , s t +1 )

Original: • P : S ×
Revised: • P : S ×

Original: A → [0 , 1] is the transition probability function that speciﬁes a conditional proba-
Revised: A → [0 , 1] is the transition probability function that speciﬁes a conditional probability p

Original: bility p ( s t +1 | s t , a t ) of transitioning into state s t +1 after executing action a t in state s t .
Revised: ( s t +1 | s t , a t ) of transitioning into state s t +1 after executing action a t in state s t .

Original: The Goal of RL.
Revised: The Goal of RL.

Original: The goal the agents pursues is to maximize its return, G t = (cid:80) Tk = t +1 R k , the sumof all collected rewards.
Revised: The goal the agents pursues is to maximize its return, G t = (cid:80) Tk = t +1 R k , the sum of all collected rewards.

Original: To achieve this, it learns a policy π that determines its behavior.
Revised: To achieve this, it learns a policy π that determines its behavior.

Original: The policyπ maps the perceived states to actions a ∼ π ( a | s ) .
Revised: The policy π maps the perceived states to actions a ∼ π ( a | s ) .

Original: The policy π can be either learned directly viathe policy gradient or indirectly via the Bellman equations.
Revised: The policy π can be either learned directly via the policy gradient or indirectly via the Bellman equations.

Original: RL algorithms that directly learn thepolicy are referred to as policy-based methods whereas algorithms that learn value functions areknown as value-based methods [Sutton and Barto, 2018].
Revised: RL algorithms that directly learn the policy are referred to as policy-based methods whereas algorithms that learn value functions are known as value-based methods [Sutton and Barto, 2018].

Original: Bellman equations.
Revised: Bellman equations.

Original: The state-value function v π ( s ) =
Revised: The state-value function v π ( s ) =

Original: E a ∼ π
Revised: E a ∼ π

Original: [ r ( s, a ) + γv π ( s (cid:48) )] and action valuefunction q π ( s, a ) =
Revised: [ r ( s, a ) + γv π ( s (cid:48) )] and action value function q π ( s, a ) =

Original: E [ r ( s, a ) + γ E a (cid:48) ∼ π
Revised: E [ r ( s, a ) + γ E a (cid:48) ∼ π

Original: [ q π ( s (cid:48) , a (cid:48) )]] determine how good it is to be in a certain stateand how good it is to perform a certain given a certain state, respectively [Sutton and Barto, 2018].
Revised: [ q π ( s (cid:48) , a (cid:48) )]] determine how good it is to be in a certain state and how good it is to perform a certain given a certain state, respectively [Sutton and Barto, 2018].

Original: 3.2 Q-learning: from tables to neural networks
Revised: 3.2 Q-learning: from tables to neural networks

Original: Tabular Q-learning Q-Learning is a value-based RL algorithm that explicitly maintains a tableof Q-estimates for state-action pairs [Watkins and Dayan, 1992].
Revised: Tabular Q-learning Q-Learning is a value-based RL algorithm that explicitly maintains a table of Q-estimates for state-action pairs [Watkins and Dayan, 1992].

Original: More generally, it falls undertemporal difference (TD) learning methods as it grounds value estimates for the current time stepin estimates at future steps.
Revised: More generally, it falls under temporal difference (TD) learning methods as it grounds value estimates for the current time step in estimates at future steps.

Original: The Q-estimates are repeatedly updated according to: q ( s, a ) ←q ( s, a ) + α
Revised: The Q-estimates are repeatedly updated according to: q ( s, a ) ← q ( s, a ) + α

Original: [ r + γ max a q ( s (cid:48) , a ) − q ( s, a )] .
Revised: [ r + γ max a q ( s (cid:48) , a ) − q ( s, a )] .

Original: While Q-Learning works well for small problems, storingthe table becomes intractable for larger state/action spaces [Sutton and Barto, 2018].
Revised: While Q-Learning works well for small problems, storing the table becomes intractable for larger state/action spaces [Sutton and Barto, 2018].

Original: Deep Q-Network.
Revised: Deep Q-Network.

Original: DQN, proposed by Mnih et al.
Revised: DQN, proposed by Mnih et al.

Original: [2015], employ neural networks to represent the
Revised: [2015], employ neural networks to represent the

Original: Q-table.
Revised: Q-table.

Original: In contrast to tabular Q-learning, DQN scales large state and action spaces.
Revised: In contrast to tabular Q-learning, DQN scales large state and action spaces.

Original: Furthermore,
Revised: Furthermore,

Original: Mnih et al.
Revised: Mnih et al.

Original: [2015] introduced two additional enhancements to Q-Learning that are essential forgood performance when using function approximation, experience replay as well as a separate targetnetwork that is updated periodically.
Revised: [2015] introduced two additional enhancements to Q-Learning that are essential for good performance when using function approximation, experience replay as well as a separate target network that is updated periodically.

Original: Therefore, the learning objective becomes:
Revised: Therefore, the learning objective becomes:

Original: [Equation]
Revised: [Equation]

Original: where D represents the experience replay buffer, and θ and θ − are the neural networks that parameterize the Q-functions.
Revised: where D represents the experience replay buffer, and θ and θ − are the neural networks that parameterize the Q-functions.

Original:  Our Approach
Revised:  Our Approach

Original: Our main goals in this work are to reduce Rainbow’s considerable hardware, training time, and datarequirements while maintaining a similar level of performance as the original agent.
Revised: Our main goals in this work are to reduce Rainbow’s considerable hardware, training time, and data requirements while maintaining a similar level of performance as the original agent.

Original: We achieve thesegoals by combining a large Q-network architecture with an efﬁcient high-throughput implementationas well as by accelerating and stabilizing training through extensive hyper-parameter tuning and theuse of spectral normalization.
Revised: We achieve these goals by combining a large Q-network architecture with an efﬁcient high-throughput implementation as well as by accelerating and stabilizing training through extensive hyper-parameter tuning and the use of spectral normalization.

Original: Furthermore, we reduce the implementation complexity of Rainbowby removing the distributional RL component.
Revised: Furthermore, we reduce the implementation complexity of Rainbow by removing the distributional RL component.

Original: While distributional RL was essential for goodperformance in Hessel et al.
Revised: While distributional RL was essential for good performance in Hessel et al.

Original: [2018] when training for over 40M frames, our improvements in learningspeed — leading to lower overall required training time — made distributional RL less vital.
Revised: [2018] when training for over 40M frames, our improvements in learning speed — leading to lower overall required training time — made distributional RL less vital.

Original: Our implementation is available at <hidden-for-blind-review> and includes a complete andhighly customizable framework for preprocessing, training, and evaluation.
Revised: Our implementation is available at https://github.com/schmidtdominik/Rainbow and includes a complete and highly customizable framework for preprocessing, training, and evaluation.

Original: We further provideintegrations for OpenAI’s gym , procgen , and gym-retro environments, as well as pretrained modelsfor all 53 tested Atari games.
Revised: We further provide integrations for OpenAI’s gym , procgen , and gym-retro environments, as well as pretrained models for all 53 tested Atari games.

Original: In the rest of this section, we discuss the tricks we apply and describe the individual componentsof our approach.
Revised: In the rest of this section, we discuss the tricks we apply and describe the individual components of our approach.

Original: First, we describe our evaluation methodology.
Revised: First, we describe our evaluation methodology.

Original: Then, we describe the networkarchitecture we employ in this work and show how different variants thereof compare.
Revised: Then, we describe the network architecture we employ in this work and show how different variants thereof compare.

Original: Furthermore,we discuss how we make use of spectral normalization and again compare a few variants.
Revised: Furthermore, we discuss how we make use of spectral normalization and again compare a few variants.

Original: Also, weaddress the tricks we employ to improve the hardware utilization of our agent.
Revised: Also, we address the tricks we employ to improve the hardware utilization of our agent.

Original: 4.1 Evaluation methodology
Revised: 4.1 Evaluation methodology

Original: We evaluate our approach against the same set of 54 Atari games that were used in Hessel et al.[2018], excluding the game Surround as it is not available via OpenAI gym .
Revised: We evaluate our approach against the same set of 54 Atari games that were used in Hessel et al. [2018], excluding the game Surround as it is not available via OpenAI gym .

Original: To this end, we closelyfollow the evaluation procedure from Mnih et al.
Revised: To this end, we closely follow the evaluation procedure from Mnih et al.

Original: [2015] and Hessel et al.
Revised: [2015] and Hessel et al.

Original: All evaluationruns lasted for 500k frames and each individual episode was no longer than 108k frames.
Revised: All evaluation runs lasted for 500k frames and each individual episode was no longer than 108k frames.

Original: The onlymodiﬁcation we made was that we performed the evaluation runs after training had concluded byperiodically saving model snapshots during training and later loading them for evaluation.
Revised: The only modiﬁcation we made was that we performed the evaluation runs after training had concluded by periodically saving model snapshots during training and later loading them for evaluation.

Original: Lastly, we re-evaluated the best-performing snapshot for each game.
Revised: Lastly, we re-evaluated the best-performing snapshot for each game.

Original: Each experiment was performed with threerandom seeds.
Revised: Each experiment was performed with three random seeds.

Original: Due to the high computational cost of training on the whole ALE, we limited our ablation studies to asubset of 5 games (Asterix, Beam Rider, Freeway, Seaquest, and Space Invaders) as recommendedin Machado et al.
Revised: Due to the high computational cost of training on the whole ALE, we limited our ablation studies to a subset of 5 games (Asterix, Beam Rider, Freeway, Seaquest, and Space Invaders) as recommended in Machado et al.

Original: [2018] and Bellemare et al.
Revised: [2018] and Bellemare et al.

Original: This selection includes two human-optimalgames, two score-exploitable games, and one sparse reward game [Taïga et al., 2020].
Revised: This selection includes two human-optimal games, two score-exploitable games, and one sparse reward game [Taïga et al., 2020].

Original: Overall, thisselection intends to capture the large variety of the ALE in a small number of games.
Revised: Overall, this selection intends to capture the large variety of the ALE in a small number of games.

Original: We encountered one issue with comparability that occurs since Hessel et al.
Revised: We encountered one issue with comparability that occurs since Hessel et al.

Original: [2018] trained their agentfor 200M frames while we only trained for 10M frames: the frequency of evaluation snapshots isthe same (every 1M frames), but the total number is not.
Revised: [2018] trained their agent for 200M frames while we only trained for 10M frames: the frequency of evaluation snapshots is the same (every 1M frames), but the total number is not.

Original: This means that for some games wherethe performance ﬂuctuates wildly during training, taking a larger number of agent snapshots couldincrease the likelihood of taking at least one snapshot at a point of good performance.
Revised: This means that for some games where the performance ﬂuctuates wildly during training, taking a larger number of agent snapshots could increase the likelihood of taking at least one snapshot at a point of good performance.

Original: To supportthis hypothesis, we separately stored and evaluated twice the number of agent snapshots (every 500ktraining frames) and observed an approximately 10% increase in apparent performance.
Revised: To support this hypothesis, we separately stored and evaluated twice the number of agent snapshots (every 500k training frames) and observed an approximately 10% increase in apparent performance.

Original: This suggeststhat evaluation results from agents trained on the ALE and evaluated on a lower number of snapshotsthan previous research may underestimate the true performance of their agent.
Revised: This suggests that evaluation results from agents trained on the ALE and evaluated on a lower number of snapshots than previous research may underestimate the true performance of their agent.

Original: 4.2 Larger and deeper Q-Network Architecture
Revised: 4.2 Larger and deeper Q-Network Architecture

Original: First, we replace the small dueling network architecture, as introduced in Mnih et al.
Revised: First, we replace the small dueling network architecture, as introduced in Mnih et al.

Original: [2013] andcombined with dueling DQN in Wang et al.
Revised: [2013] and combined with dueling DQN in Wang et al.

Original: [2016], with the both larger and deeper IMPALA
Revised: [2016], with the both larger and deeper IMPALA

Original: CNN [Espeholt et al., 2018].
Revised: CNN [Espeholt et al., 2018].

Original: More speciﬁcally, we employ the large variant of the IMPALA CNN(see Figure 1 in the supplementary material) with twice the number of channels, as modiﬁed in
Revised: More speciﬁcally, we employ the large variant of the IMPALA CNN (see Figure 1 in the supplementary material) with twice the number of channels, as modiﬁed in

Original: Cobbe et al.
Revised: Cobbe et al.

Original: We additionally add a size 6 × 6 adaptive max-pooling layer between thenetwork’s convolutional and fully connected parts.
Revised: We additionally add a size 6 × 6 adaptive max-pooling layer between the network’s convolutional and fully connected parts.

Original: This simple yet powerful modiﬁcation makesit straightforward to use our implementation with inputs of different resolutions (such as gamesprovided by procgen or gym-retro ) without affecting the number of parameters in the network.
Revised: This simple yet powerful modiﬁcation makes it straightforward to use our implementation with inputs of different resolutions (such as games provided by procgen or gym-retro ) without affecting the number of parameters in the network.

Original: Like Cobbe et al.
Revised: Like Cobbe et al.

Original: [2020] and Espeholt et al.
Revised: [2020] and Espeholt et al.

Original: [2018], we found that using this architecture substantiallyincreased learning speed (both in terms of wall-clock time and training steps), sample efﬁciency, andﬁnal overall performance.
Revised: [2018], we found that using this architecture substantially increased learning speed (both in terms of wall-clock time and training steps), sample efﬁciency, and ﬁnal overall performance.

Original: In our experiments, we compared values of 1, 2 and 4 for the channel multiplier.
Revised: In our experiments, we compared values of 1, 2 and 4 for the channel multiplier.

Original: The results of theseexperiments are shown in Figure 1.
Revised: The results of these experiments are shown in Figure 1.

Original: We found that a value of 2 yielded a good trade-off between data and computational efﬁciency.
Revised: We found that a value of 2 yielded a good trade-off between data and computational efﬁciency.

Original: Consequently, we selected this network architecture for our subsequentruns.
Revised: Consequently, we selected this network architecture for our subsequent runs.

Original: 4.3 Spectral Normalization
Revised: 4.3 Spectral Normalization

Original: Spectral normalization (SN), commonly used to stabilize the training of discriminators in GANs[Miyato et al., 2018], is a method for controlling the Lipschitz constant of linear operators such asconvolutional or dense linear layers in neural networks.
Revised: Spectral normalization (SN), commonly used to stabilize the training of discriminators in GANs [Miyato et al., 2018], is a method for controlling the Lipschitz constant of linear operators such as convolutional or dense linear layers in neural networks.

Original: Spectral normalization can be efﬁcientlyapproximated via the power iteration method [Miyato et al., 2018, Gogianu et al., 2021].
Revised: Spectral normalization can be efﬁciently approximated via the power iteration method [Miyato et al., 2018, Gogianu et al., 2021].

Original: In our experiments, we compared three different variants of applying spectral normalization to theconvolutional layers in the IMPALA CNN:
Revised: In our experiments, we compared three different variants of applying spectral normalization to the convolutional layers in the IMPALA CNN:

Original: • none – no spectral normalization is performed.
Revised: • none – no spectral normalization is performed.

Original: • all – SN is applied to both convolutional layers in all six residual blocks.
Revised: • all – SN is applied to both convolutional layers in all six residual blocks.

Original: • last – SN is applied to all convolutional layers in the ﬁnal two residual blocks.
Revised: • last – SN is applied to all convolutional layers in the ﬁnal two residual blocks.

Original: Furthermore, we experimented with a fourth variant that applied SN to the ﬁnal (noisy) linear layersonly.
Revised: Furthermore, we experimented with a fourth variant that applied SN to the ﬁnal (noisy) linear layers only.

Original: However, we quickly dismissed this variant due to bad overall performance.
Revised: However, we quickly dismissed this variant due to bad overall performance.

Original: We hypothesize thatthe drop in performance resulted from interference between spectral normalization and Noisy-Nets
Revised: We hypothesize that the drop in performance resulted from interference between spectral normalization and Noisy-Nets

Original: DQN exploration.
Revised: DQN exploration.

Original: Figure 2 shows the comparison of the three variants of spectral normalization over ﬁve games.
Revised: Figure 2 shows the comparison of the three variants of spectral normalization over ﬁve games.

Original: Eventhough the last variant was marginally faster in terms of training throughput, we eventually settled onthe all variant as it slightly outperformed the former.
Revised: Even though the last variant was marginally faster in terms of training throughput, we eventually settled on the all variant as it slightly outperformed the former.

Original: Overall, we observed that spectral normalization has the largest effect at the beginning of trainingacross a number of games.
Revised: Overall, we observed that spectral normalization has the largest effect at the beginning of training across a number of games.

Original: It signiﬁcantly reduces the time until initial learning progress is made.
Revised: It signiﬁcantly reduces the time until initial learning progress is made.

Original: This effect was most strongly observable in games such as Breakout , Seaquest and Tennis whereinitial progress can be particularly slow.
Revised: This effect was most strongly observable in games such as Breakout , Seaquest and Tennis where initial progress can be particularly slow.

Original: 4.4 Improving Hardware Utilization
Revised: 4.4 Improving Hardware Utilization

Original: We additionally employ several practical modiﬁcations to the training process that aim to decrease thewall-clock training time — and thus the required computational budget — by maximizing hardwareutilization and training throughput:
Revised: We additionally employ several practical modiﬁcations to the training process that aim to decrease the wall-clock training time — and thus the required computational budget — by maximizing hardware utilization and training throughput:

Original:  Similarly to Cobbe et al.
Revised: • Similarly to Cobbe et al.

Original: , we increase the batch size from 32 to 256.
Revised: [2020], we increase the batch size from 32 to 256.

Original: As suggested

Original: in Stooke and

Original: Abbeel [

Original: , we accordingly adjust the (cid:15) hyper-parameter for the Adam optimizer to 0 .
Revised: As suggested in Stooke and Abbeel [2018], we accordingly adjust the (cid:15) hyper-parameter for the Adam optimizer to 0 .

Original: 005 /b , where b is the batch size.
Revised: 005 /b , where b is the batch size.

Original:  Environment interactions need to be computed sequentially and can thus not be parallelized across the time dimension.Thus
Revised: • Environment interactions need to be computed sequentially and can thus not be parallelized across the time dimension.

Original: , to batch environment simulation, we maintain e = instances of the environment and take one 
Revised: Thus, to batch environment simulation, we maintain e =instances of the environment and take one step of this vectorized environment for every k = 2 training steps.

Original: step of this

Original: vectorized

Original: environment for

Original: every

Original: k = 2 training steps.

Original: In

Original: our implementation

Original: , the parameters

Original: b and e can be

Original: freely chosen

Original: soas to

Original:  maximize GPU and CPU utilization and the agent’s performance
Revised: In our implementation, the parameters b and e can be freely chosen so as to maximize GPU and CPU utilization and the agent’s performance.

Original: The average number of times each environment transition 
Revised: The average number of times each environment transition is sampled from the replay buffer bke = 8 remains ﬁxed by setting k accordingly.

Original: is

Original: sampled

Original: from the replay

Original: buffer

Original: bke =

Original: remains

Original: ﬁxed

Original: by setting

Original: k accordingly • We perform environment simulation steps and training steps in parallel to minimize idle time on the GPU.
Revised: • We perform environment simulation steps and training steps in parallel to minimize idle time on the GPU.

Original: • We use mixed-precision training as provided by PyTorch’s amp package.
Revised: • We use mixed-precision training as provided by PyTorch’s amp package.

Original: In aggregate, these modiﬁcations decrease the training time by a factor of 3.2, from 24 to approximately 7.5 hours, for training on 10M frames on a single Nvidia RTX 3090 GPU.
Revised: In aggregate, these modiﬁcations decrease the training time by a factor of 3.2, from 24 to approximately 7.5 hours, for training on 10M frames on a single Nvidia RTX 3090 GPU.

Original: Interestingly, theincreased batch size also considerably improves the performance on several games.
Revised: Interestingly, the increased batch size also considerably improves the performance on several games.

Original: Prior work hasobserved a similar effect and attributed it to decreased gradient variance and thus more stable trainingbehavior [Cobbe et al., 2020].
Revised: Prior work has observed a similar effect and attributed it to decreased gradient variance and thus more stable training behavior [Cobbe et al., 2020].

Original: Figure 3 shows the individual contributions of the modiﬁcationscompared to baseline.
Revised: Figure 3 shows the individual contributions of the modiﬁcations compared to baseline.

Original: 4.5 Hyperparameter Tuning
Revised: 4.5 Hyperparameter Tuning

Original: Apart from the batch size, network size, and choice of spectral normalized layers, we also performeda hyper-parameter search over learning rates and compared the Huber and mean-squared error (MSE)loss functions.
Revised: Apart from the batch size, network size, and choice of spectral normalized layers, we also performed a hyper-parameter search over learning rates and compared the Huber and mean-squared error (MSE) loss functions.

Original: Unlike Obando-Ceron and Castro [2021] we did not observe better performance through the use of the
Revised: Unlike Obando-Ceron and Castro [2021] we did not observe better performance through the use of the

Original: MSE loss.
Revised: MSE loss.

Original: However, the search over learning rates did prove fruitful.
Revised: However, the search over learning rates did prove fruitful.

Original: Increasing the learning rate by afactor of 4 (to the same value used in DQN) signiﬁcantly speeds up learning in our experiments, bothwith the increased batch size and without.
Revised: Increasing the learning rate by a factor of 4 (to the same value used in DQN) signiﬁcantly speeds up learning in our experiments, both with the increased batch size and without.

Original: In addition, it considerably improves the ﬁnal performancethe agent achieves.
Revised: In addition, it considerably improves the ﬁnal performance the agent achieves.

Original: We hypothesize that the reason Hessel et al.
Revised: We hypothesize that the reason Hessel et al.

Original: [2018] settled on the lower learningrate was that the beneﬁt of better convergence when using a lower learning rate outweighed thedisadvantage of slower learning when training for much longer periods of time.
Revised: [2018] settled on the lower learning rate was that the beneﬁt of better convergence when using a lower learning rate outweighed the disadvantage of slower learning when training for much longer periods of time.

Original: The full set of hyperparameters is listed in Appendix A in the supplementary material.
Revised: The full set of hyperparameters is listed in Appendix A in the supplementary material.

Original: 4.6 Distributional RL
Revised: 4.6 Distributional RL

Original: Finally, in our experiments removing the distributional RL component from Rainbow did not signiﬁcantly affect our agent’s ﬁnal performance or learning speed when trained for 10M frames.
Revised: Finally, in our experiments removing the distributional RL component from Rainbow did not signiﬁcantly affect our agent’s ﬁnal performance or learning speed when trained for 10M frames.

Original: This is inline with observations from the ablation studies in Hessel et al.
Revised: This is in line with observations from the ablation studies in Hessel et al.

Original: [2018] that showed the main beneﬁtof distributional RL materialized only after training for about 40M frames.
Revised: [2018] that showed the main beneﬁt of distributional RL materialized only after training for about 40M frames.

Original: Thus, considering thenegligible performance improvement and substantial implementation complexity, we decided not toinclude distributional RL in our ﬁnal agent.
Revised: Thus, considering the negligible performance improvement and substantial implementation complexity, we decided not to include distributional RL in our ﬁnal agent.

Original: Similarly, the evaluation curves in Hessel et al.
Revised: Similarly, the evaluation curves in Hessel et al.

Original: [2018] indicate that other components of Rainbowsuch as Noisy-Nets DQN and Dueling DQN may only be important when training for long periods.
Revised: [2018] indicate that other components of Rainbow such as Noisy-Nets DQN and Dueling DQN may only be important when training for long periods.

Original: This may suggest that — given future improvements in data efﬁciency, leading to lower required training time — it might be possible to further simplify the Rainbow agent with no penalties toperformance, in particular when working with a limited computational budget.
Revised: This may suggest that — given future improvements in data efﬁciency, leading to lower required training time — it might be possible to further simplify the Rainbow agent with no penalties to performance, in particular when working with a limited computational budget.

Original: Additionally, we investigated replacing the C51 [Bellemare et al., 2017] variant of distributional RLwith QR-DQN [Dabney et al., 2018].
Revised: Additionally, we investigated replacing the C51 [Bellemare et al., 2017] variant of distributional RL with QR-DQN [Dabney et al., 2018].

Original: However, in our experiments, we could not observe a signiﬁcantdifference.
Revised: However, in our experiments, we could not observe a signiﬁcant difference.

Original:  Results
Revised:  Results

Original: Table 1 compares the ﬁnal aggregate scores of our approach to the ones of the original Rainbow-DQNagent, as well as to standard DQN.
Revised: Table 1 compares the ﬁnal aggregate scores of our approach to the ones of the original Rainbow-DQN agent, as well as to standard DQN.

Original: The full scores and learning curves for each of the 53 games areavailable in Table 1 and Figure 2 in the supplementary material.
Revised: The full scores and learning curves for each of the 53 games are available in Table 1 and Figure 2 in the supplementary material.

Original: Each of our training runs took approximately 7.5 hours on a single Nvidia RTX 3090 GPU or 10hours on a single Nvidia 2080Ti GPU.
Revised: Each of our training runs took approximately 7.5 hours on a single Nvidia RTX 3090 GPU orhours on a single Nvidia 2080Ti GPU.

Original: These requirements make performing larger numbers ofexperiments feasible, even with a more modest compute budget.
Revised: These requirements make performing larger numbers of experiments feasible, even with a more modest compute budget.

Original: Our ﬁnal agent was trained for only 10 million environment transitions and achieves a median humannormalized score (HNS) of 205.7, reaching above human-level performance on 39 out of 53 games.
Revised: Our ﬁnal agent was trained for only 10 million environment transitions and achieves a median human normalized score (HNS) of 205.7, reaching above human-level performance on 39 out of 53 games.

Original: In contrast, classic Rainbow achieves a median HNS of 70 (estimated) and 231 after training for 10and 200 million transitions, respectively.
Revised: In contrast, classic Rainbow achieves a median HNS of 70 (estimated) and 231 after training forand 200 million transitions, respectively.

Original: Overall, their score after 200M transitions is equivalentto super-human performance on 40 out of 53 games.
Revised: Overall, their score after 200M transitions is equivalent to super-human performance on 40 out of 53 games.

Original: The exact scores for classic Rainbow at 10Mframes were not published in Hessel et al.
Revised: The exact scores for classic Rainbow at 10M frames were not published in Hessel et al.

Original: Therefore, they were estimated from the providedevaluation curves.
Revised: Therefore, they were estimated from the provided evaluation curves.

Original: In addition, published results in Castro et al.
Revised: In addition, published results in Castro et al.

Original: [2018] for the "Dopamine" Rainbowimplementation serve as another reference point (see Table 1, HNS of 50.4 for 10M and 145.8 for200M).
Revised: [2018] for the "Dopamine" Rainbow implementation serve as another reference point (see Table 1, HNS of 50.4 for 10M and 145.8 for 200M).

Original: In any case, our approach outperforms the scores at 10M by a big margin and achieves scoresmore similar to the ones obtained after 200M steps. 
Revised: In any case, our approach outperforms the scores at 10M by a big margin and achieves scores more similar to the ones obtained after 200M steps. [Figure] [Table] [Figure]

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: Conclusion
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original:  In this paper, we addressed one of the fundamental questions in reinforcement learning: how toaccelerate the training of RL agents?
Revised: [Figure] [Table]  Conclusion In this paper, we addressed one of the fundamental questions in reinforcement learning: how to accelerate the training of RL agents?

Original: To achieve this, we employed a variety of tricks that speedup the RL training procedure and empirically investigated how they affect training speed as well asoverall performance on the Atari benchmark.
Revised: To achieve this, we employed a variety of tricks that speed up the RL training procedure and empirically investigated how they affect training speed as well as overall performance on the Atari benchmark.

Original: While most of these enhancements have been studiedindividually in the context of RL, our work is, to the best of our knowledge, the ﬁrst to study themjointly.
Revised: While most of these enhancements have been studied individually in the context of RL, our work is, to the best of our knowledge, the ﬁrst to study them jointly.

Original: Our improved training procedure signiﬁcantly reduces the training time and data required bythe RL agent while maintaining competitive performance.
Revised: Our improved training procedure signiﬁcantly reduces the training time and data required by the RL agent while maintaining competitive performance.

Original: Furthermore, we conducted a number ofablations to understand what effect the individual modiﬁcations have on the training process.
Revised: Furthermore, we conducted a number of ablations to understand what effect the individual modiﬁcations have on the training process.

Original: Overall,the results we presented may serve as a practical guide for speeding up the training of RL agents.
Revised: Overall, the results we presented may serve as a practical guide for speeding up the training of RL agents.

Revised: Acknowledgments and Disclosure of Funding

Revised: We would like to thank the TU Wien DataLab for providing the majority of the needed compute resources and the CleanRL team for their baseline implementation of DQN [Huang et al., 2021].

Revised: B Hyperparameters B.1 Hyperparameters for Rainbow

Revised: This section lists the hyperparameters used in our experiments.

Revised: Parameters that differ from the ones used in Hessel et al.

Revised: [2018] are marked with an asterisk.

Revised: As in previous work, the unit "frames" refers to the number of environment steps taken by the wrapped environment, including frame-skipping.

Revised: For noisy-nets DQN we implemented the "factorized Gaussian noise" variant, with noise vectors generated on the GPU.

Revised: [Table]

Revised: B.2 Environment pre-processing hyperparameters

Revised: This section lists the settings for preprocessing environments from gym , gym-retro and procgen .

Revised: All environments used a time limit of 108k frames (30 minutes of emulator time).

Revised: Image downscaling was performed with area interpolation.

Revised: For gym environments, we max-pooled consecutive frames and used 0-30 noop actions at the beginning of each episode as in Hessel et al.

Revised: [Table]