M EASURING
P ROGRESS
IN
D EEP
R EINFORCEMENT
L
EARNING S AMPLE E FFICIENCY
Anonymous authors
Paper under double-blind review
A BSTRACT
Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms.
Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efﬁciency.
As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efﬁciency is an important indicator for economically relevant applications of DRL.
We investigate progress in sample efﬁciency on Atari games and continuous control tasks by comparing the number of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications.
We ﬁnd exponential progress in sample efﬁciency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the speciﬁc task and performance level.
I NTRODUCTION
Recent successes of deep reinforcement learning (DRL) in Go (Silver et al., 2016; 2017; 2018) and complex real-time strategy games (Berner et al., 2019; Vinyals et al., 2019) indicate the vast potential for automating complex economically relevant tasks like assembling goods in non-standardized environments or physically assisting humans using DRL.
However, sample efﬁciency is likely to remain an important bottleneck for the economic feasibility of real world applications of DRL.
This is because samples of state transitions in the environment caused by an agent’s action are essential for training DRL agents but automated systems interacting with the real world are often fragile, slow or costly, which makes DRL training in the real world expensive both in terms of money and time (Dulac-Arnold et al., 2019).
As most widely used benchmarks in deep reinforcement learning (DRL) still consist of computer games and simulations where samples can be obtained risk-free, fast and cheaply 1 , progress in DRL does not necessarily translate to future real world applications unless it corresponds to progress in sample efﬁciency or simulation accuracy and transfer learning.
This makes information about progress in sample efﬁciency an important input to researchers studying topics like the future of employment (Frey & Osborne, 2017), the potential for malicious uses of DRL (Brundage et al., 2018) and other potentially transformative impacts of AI systems in the future (Gruetzemacher & Whittlestone, 2019).
B ACKGROUND AND RELATED WORK 2. S AMPLE EFFICIENCY
While research on progress in the ﬁeld of AI used to have a strong focus on benchmark performance metrics in AI (Eckersley & Nasser, 2017), there have been calls to pay more attention to the importance of other metrics: Martinez-Plumed et al.
(2018) enumerate previously neglected dimensions of AI progress and explore the relationship between computing resources and ﬁnal performance in RL.
The wider implications of sample efﬁciency, more speciﬁcally, are explored by Tucker et al.
While their scope is broader than DRL, most of their points do apply to reinforcement learning as a special case of AI technology.
For example, as the authors point out, robotics is an especially important case of a data-limited domain, in which new applications are likely to be unlocked for many actors with improved sample efﬁciency.
While some benchmark papers make explicit quantitative comparisons of sample efﬁciency (or “data efﬁciency”) to isolated previous methods (Hessel et al., 2018; Hafner et al., 2019a; Kaiser et al., 2019; Srinivas et al., 2020), we are not aware of any more systematic investigation of sample efﬁciency in DRL.
2. C OMPUTING POWER AND SCALING L AWS
While DRL and other applications of deep learning have produced many impressive results, this has gone hand in hand with an ever-increasing amount of computing power (compute) being devoted to model training: While compute usage for large AI projects has been in line with Moore’s law prior to 2012 (Sastry et al., 2019), it saw a doubling time of 3.4-months in the 2012 to 2018 period, which is seven times faster than Moore’s law (Amodei & Hernandez, 2018).
Compute usage would not increase if there was nothing to gain: For example, the performance of Transformer-based language models (Vaswani et al., 2017) follows a power law in the number of model parameters when differently sized models are trained on the same dataset indicating that in the current regime, Transformer language models become more sample efﬁcient with more parameters.
Data is still important, as there is another power law relating dataset size for models with the same amount of parameters to performance (Kaplan et al., 2020).
Things likely look similar for the ImageNet benchmark (Deng et al., 2009) where progress has long been accompanied by larger models using a ﬁxed amount of data (Sun et al., 2017) while recent progress relies on additional data (Touvron et al., 2019).
2. A LGORITHMIC PROGRESS AND INCREASED EFFICIENCY
As increased sample usage might make progress in DRL look faster, increased compute usage might do so for machine learning.
Grace (2013) found that hardware improvements account for half of the progress for several problems in the vicinity of AI research like computer chess, computer go and physics simulations.
More recently, Fichte et al.
(2020) observed the same for progress in SAT solvers.
In the realm of deep learning, Hernandez & Brown (2020) investigated improvements in algorithmic efﬁciency by holding ﬁnal performance on various benchmarks constant and analyzing how the number of FLOPS needed to reach that level changed over time.
They ﬁnd that this amount of compute needed to reach the performance of AlexNet (Krizhevsky et al., 2012) on ImageNet went down with a halving time of 16 months over 6 years.
Looking at DawnBench (Coleman et al., 2017) submissions, they also ﬁnd a cost reduction in terms of dollars of 184x 2 between October 2017 and
September 2018 for reaching a ﬁxed performance level on ImageNet.
Another submission managed to reduce the training time by a factor of 808x from 10 days and 4 hours to 18 minutes in the same time interval while still cutting the cost by a factor of over nine.
M ETHODS 3. W E MEASURE GROWTH IN SAMPLE EFFICIENCY BY HOLDING PERFORMANCE CONSTANT
As proposed by Dulac-Arnold et al.
(2019) and similar to work by Hernandez & Brown (2020) on measuring algorithmic progress, we measure progress in sample efﬁciency by comparing the number of samples needed for systems to reach a ﬁxed performance level over time and deﬁne the sample efﬁciency of an algorithm for a given task and performance level as 1/S, where S is the number of samples needed to train to the given performance level on the task: Sample [Equation] Algorithm
[Equation]
Samples Algorithm needs for Score on Task Compared to the dual approach of comparing the performance of different algorithms for a ﬁxed amount of samples, this approach has the advantage of being more comparable between different benchmarks and a lot easier to interpret: The marginal difﬁculty of gaining score at a given performance level could vary a lot between games, and it is unclear whether twice as much score corresponds to twice better performance in any meaningful sense (Hernandez-Orallo, 2020), while 2 This might be 88x, as there is a second initial contribution which cost $1113 rather than $2323. twice as many samples to reach the same level of performance can straightforwardly be interpreted as half the sample efﬁciency.
3. W E CONSIDER BOTH A TARI AND CONTINUOUS CONTROL
We look at sample efﬁciency in three areas: 1) for the median human-normalized score on 57 games in the Arcade Learning Environment (ALE) (Bellemare et al., 2013) using the no-op evaluation protocol and pixel observations; 2) single task performance on continuous control tasks from the rllab-benchmark (Duan et al., 2016) using simulation states 3 as input; and 3) on tasks from the DeepMind Control Suite (Tassa et al., 2018) using pixel observations as input.
We chose these tasks because of their popularity and the existence of some de facto standards on evaluation and reporting protocols, which yields more mutually comparable published results.
The tasks’ popularity also implies attention from the research community, which ensures that they are somewhat representative of overall progress in DRL.
Another advantage is the tasks’ moderate difﬁculty and the wide range of partial solutions between failing the task and solving it optimally.
This allows for a meaningful comparison of a wider set of different algorithms as more algorithms reach the performance threshold, while there is still space for improvement using advanced techniques.
3. W E TREAT EACH PUBLICATION AS A MEASUREMENT OF THE STATE - OF - THE - ART .
For our main results on progress in sample efﬁciency, we treat every considered result as a measurement of the state-of-the-art at publication:
For each publication, we plot the sample usage (grey dot in ﬁgures 2 (a) and 3) and the state-of-the-art in sample efﬁciency at the time (blue stars in ﬁgures 2 (a) and 3) such that results that improve on the state-of-the-art are represented by a grey dot with a blue star in the middle.
Then we ﬁt the blue stars.
Compared to ﬁtting a dataset only containing jumps in the SOTA (grey dots with a blue star), this dampens the effect single outliers have on the ﬁtted model and means that prolonged periods of stagnation are taken into account adequately.
However, our way of ﬁtting the SOTA is sensitive to the precise dates associated with results that did not improve on the SOTA.
Another approach would be to graph the SOTA on a continuous-time axis and approximate the corresponding step function by an exponential.
This solves the sensitivity to measurements that don’t affect the SOTA but comes with two drawbacks: it requires an arbitrary choice of end date that affects the estimated doubling time and leads to high sensitivity to isolated early measurements.
Full results for the estimated doubling times for the different approaches can be found in appendix E.
Compared to the between task/score variation, the effect of the approach is usually small, but we did observe some larger effects when there have only been few and early changes to the SOTA.
In these cases, the models using only jumps predict fast doubling times despite long periods of stagnation, while both other models are more conservative.
On the other hand, the continuous-time model yielded the fastest doubling times for most pixel-based MuJoCo tasks, as it puts a lot of weight on improvements relative to the early, weak D4PG (Barth-Maron et al., 2018) baseline.
3. O UR STUDY IS BASED ON PREVIOUSLY PUBLISHED TRAINING CURVES
Due to the difﬁculty and overhead associated with the independent replication of results in DRL (Islam et al., 2017; Henderson et al., 2018), we decided to base our study on training curves in published papers.
To that extent, we conducted a systematic search for papers tackling the respective benchmarks and included all relevant 4 identiﬁed papers which reported on the metrics under consideration in a sufﬁciently precise way.
A more detailed description of the results we included and the search process we employed to identify them can be found in appendix B. While our approach implies multiple limitations of our study that are discussed in appendix D, such as a biased selection of results and the inability to tune hyperparameters, we believe that the limited methodology still offers substantial value in exploring trends in DRL sample efﬁciency.
3. W E ALSO LOOK AT OTHER METRICS FOR A TARI
In addition to measuring progress in sample efﬁciency, we analyze trends in the number of samples used for overall training of agents and compare progress in state-of-the-art performance given restricted and unrestricted amounts of data to better understand the role using large amounts of samples plays for benchmark performance.
As Martinez-Plumed et al.
(2018) did for compute usage and task performance, we also look at shifts over time in the Pareto front that visualizes the tradeoff between sample usage and median task performance on the ALE.
R ESULTS 4. T HE AMOUNT OF SAMPLES USED TO TRAIN DRL AGENTS ON THE ALE AND THE SPEED AT WHICH SAMPLES ARE GENERATED HAS INCREASED RAPIDLY
Since DQN was ﬁrst trained on the majority of the now standard 57 Atari games in 2015 (Mnih et al., 2015), the amount of samples per game used by the most ambitious projects to train their agents on the ALE has increased by a factor of 450 from 200 million to 90 billion as shown in ﬁgure 1 (a).
This corresponds to a doubling time in sample use of around 7 months.
Converted into real game time, it represents a jump from 38.6 hours (per game) to 47.6 years which was enabled by the fast speed of the simulators and running large amounts of simulations in parallel to reduce the wall-clock time needed for processing that many frames.
In fact, the trend in wall-clock training time is actually reversed as can be seen in table 1: while DQN was trained for a total of 9.5 days, MuZero took only 12 hours of training to process 20 billion frames (Schrittwieser et al., 2019), which represents a speedup in utilized frames per second of 1900 in less than ﬁve years.
This demonstrates that using larger and larger amounts of samples has become a lot more popular as well as feasible over time.
[Figure]
[Table]
Speed factor is the speedup in the amount of frames used per second compared to
. 4. P ERFORMANCE ON THE ALE SEEMS TO GROW EXPONENTIALLY WITH A FIXED AMOUNT OF SAMPLES BUT TWICE AS FAST WHEN SAMPLES ARE LESS RESTRICTED
Training DQN on billions of frames would take prohibitively long (Table 1) and judging from the training curve 5 , it seems unclear whether this would improve performance at all.
This indicates that progress the performance on the ALE has not exclusively been driven by the massively increased amount of training frames.
In fact, we observe two different trends when looking at the stateof-the-art median human-normalized score without restricting the number of frames used and the state-of-the-art performance on 200 million training frames (Figure 1 (b)).
While the exact slopes of the ﬁtted trendlines are fairly uncertain due to the limited amount of data points, especially for the unrestricted benchmark, it seems like progress on the unrestricted benchmarks is around twice as fast.
This can be interpreted as roughly half of progress coming from increased sample use, while the other half comes from a combination of algorithmic improvements and more compute usage 6 .
4. P ROGRESS IN SAMPLE EFFICIENCY ON THE ALE CAN BE FIT BY EXPONENTIAL CURVES WITH DOUBLING TIMES OF 10 TO 18 MONTHS FOR DIFFERENT PERFORMANCE LEVELS
The improvements on 200 million frames indicate improved sample efﬁciency.
Indeed, this is true:
Figure 2 (a) shows the number of frames needed to reach the same median human-normalized score as DQN as grey dots and the state-of-the-art at the respective publication dates as blue stars.
The black line shows the best exponential ﬁt 7 for the blue dots and corresponds to a doubling time in sample efﬁciency of 11 months.
The doubling time estimate does depend on the baseline we compare performance to and is a bit higher when later and stronger results are used (Table 2).
[Table]
As LASER, the last benchmark we consider is atypical in that its training curve is unusually linear, such that its strong performance at 200 million frames does not translate to better sample efﬁciency and as we were not able to obtain training curves for MuZero which achieved a 731% median humannormalized score after 200 million frames in 2019, we might underestimate the actual doubling time.
On the other hand, we did not ﬁnd improvements for the full 57 game benchmark in 2020 and the current SOTA for sample efﬁciency was achieved in 2018, which points in the opposite direction.
Either way, progress in DRL sample efﬁciency on the ALE has likely been faster than Moore’s law.
4. P ROGRESS ON THE ALE IS NONUNIFORM AND STAGNATION CAN OCCUR
Temporary stagnation in progress on sample efﬁciency happened before.
For example, there have been only marginal improvements from C51 over Dueling DDQN despite a more than one and a [Figure] half year period between their publication dates.
This lack of progress in 2016 can clearly be seen on the frames/median human-normalized score Pareto front (Figure 3 (b)).
A possible explanation for the lack of substantial improvement in 2016 could be that it took some time for a breakthrough to unlock a new set of low-hanging fruit to be picked after most had been harvested in 2015.
Indeed, the invention of distributional methods which estimate a distribution of returns in 2017 was rapidly followed by improvements of the approach (Dabney et al., 2018a) and combinations with other methods (Hessel et al., 2018) and enabled a lot of progress.
Interestingly, distributional methods did not seem to play a large role after mid-2018, when methods discarded the distributional approach (Badia et al., 2020) or entirely moved away from approaches based on Q-learning (Schmitt et al., 2019; Schrittwieser et al., 2019).
Unlike distributional approaches, distributed approaches that use multiple actors collecting experience in parallel and are thus able to generate a lot more samples started to gain importance in 2018 (Horgan et al., 2018; Kapturowski et al., 2018).
This can be seen in the expansion of the Pareto front towards the right of ﬁgure 3 (b) in 2018, and is a trend that continued in 2019.
But 2019 also saw increased attention to the left side of the Pareto front, the low sample regime.
Interestingly, the two leftmost points on the Pareto front in 2019 representing a lot of progress essentially use Rainbow DQN (Hessel et al., 2018) which was ﬁrst tested in 2017 with differently tuned hyperparameters (Kielak, 2020).
This suggests that they are more indicative of the renewed attention to training with a severely restricted amount of samples, rather than of conceptual progress in sample efﬁcient learning.
While the only visible progress on the Pareto front in 2020 so far has been in the low-sample regime, this does not mean that progress in reinforcement learning has come to a halt.
Rather, recent breakthroughs have been focused on reaching above 100% humannormalized score on all 57 Atari games, rather than optimising the median while neglecting progress on the hardest games (Badia et al., 2020).
4. P ROGRESS IN SAMPLE EFFICIENCY MIGHT BE EVEN FASTER FOR CONTINUOUS CONTROL
Figure 3 which is structured like ﬁgure 2 (a) shows the number of frames needed to reach 2000 score in the state-based HalfCheetah task from the rllab/gym benchmark and the number of frames needed [Figure] to reach 400 score on the pixel-based Walker Walk task from the DeepMind Control Suite over time.
The SOTA is ﬁtted well by an exponential model with a doubling time of 9 months for HalfCheetah and 5 months for Walker walk.
Averaged over all task/score combinations we considered, the average doubling time was 11 months for state-based continuous control and 5 months for the pixelbased case, the latter of which is faster than the trend in overall sample use on the ALE.
The amount of included results rose in 2019 for both types of tasks 9 , but this does not seem to correspond to faster progress in sample efﬁciency.
As the Control Suite was only established in 2018, the overall time frame we considered was shorter for the pixel-based tasks, such that the low doubling time is less robust than our other results.
Another reason to be careful with extrapolating from these very short doubling times is that a naive extrapolation would predict that 400 score on Walker Walk will be reachable using a single sample within 6 and a half years.
As learning consistent behaviour from a single sample is most likely impossible, the exponential model has to break down rather soon, at least for tasks that are already solvable with relatively few samples.
D ISCUSSION
We found progress in sample efﬁciency to be mainly driven by improvements in off-policy DRL and model-based approaches, as well as the use of auxiliary learning objectives to speed up representation learning.
As exempliﬁed by the improvements in the low sample regime on the ALE from tuning Rainbow for sample efﬁciency, shifting incentives for sample efﬁciency also play an important role for realized progress.
Interestingly, we did not ﬁnd strong evidence for a large contribution of network size to sample efﬁciency.
For a more detailed discussion of factors contributing to progress in sample efﬁciency, consider appendix C.
Together with cost reductions in robotics hardware (Yang et al., 2019a; Rahme, 2020; Ahn et al., 2020) sustained progress in sample efﬁciency might enable ample applications of DRL in the real world.
In addition, we might see compute-powered improvements in simulation quality, more unsupervised pretraining for representation learning on large task-adjacent datasets and further progress in transferring learnt behaviour from simulations to reality (Peng et al., 2018; James et al., 2019; van Baar et al., 2019). These factors might reduce the cost of real world DRL by providing a reasonable starting policy that makes human intervention to reset the learning system (Zhu et al., 2020) or prevent catastrophes less important.
Such a baseline policy could also be obtained from imitation learning if we are able to create systems that robustly interact with the real world by means other than DRL.
Once economically relevant applications of DRL become more widespread, sample efﬁciency is likely to gain economic relevance which would fuel further progress in sample efﬁciency.
However, widespread structurally similar real world applications of DRL could eventually alleviate the need for better sample efﬁciency, either by providing immense datasets that can be used to leverage progress in ofﬂine DRL (Agarwal et al., 2020) or when combined with improved generalization capabilities (Goyal et al., 2019; Sekar et al., 2020).
We want to stress that there are several limitations to our study: the reliance on published training curves can add noise or even bias, as a single noisy outlier can greatly inﬂuence our estimate for the SOTA.
We were also unable to tune hyperparameters for sample efﬁciency which means that we cannot clearly separate more cleverly tuned hyperparameters from more substantial progress in sample efﬁciency.
Furthermore, we focus on only three types of tasks and neither consider progress in robotics or ofﬂine DRL sample efﬁciency, nor general progress in sim2real and transfer learning, all of which might play an important role for real world application of DRL.
In particular, we believe that progress in ofﬂine DRL might be slower, as improved exploration strategies cannot be leveraged for sample efﬁciency in that case.
Lastly, it could be argued that unlocking new capabilities, a dimension we ignored, corresponds to more progress in sample efﬁciency than getting to previously unlocked capabilities slightly faster.
A more detailed discussion of these limitations can be found in appendix D. Furthermore.
there are many other ways in which extrapolating the presented trends could be misleading: for example, progress might speed up a lot once the ﬁrst commercial application of real world DRL create stronger economic incentives for sample efﬁciency or progress might slow down signiﬁcantly as we come closer to fundamental limits on sample efﬁciency.
In particular, naive extrapolation of the trends we observed would lead to instantaneous learning of strong behaviour on some of the tasks we observed, within the decade, which seems highly implausible.
C ONCLUSION
We ﬁnd considerable progress in the sample efﬁciency of DRL at rates comparable to progress in algorithmic efﬁciency in deep learning.
If the trends we observed proved to be robust and continued, the huge amounts of simulated data that are currently necessary to achieve state-of-the-art results in
DRL might not be required for future applications such that training in real world contexts could become feasible.
However, this should be taken with a grain of salt as relying on the extrapolation of trends that have only been observed for a small time frame using noisy data can be problematic.
A reiteration of our study that reproduces results rather than relying on training curves and tunes hyperparameters for sample efﬁciency would ﬁx many of our study’s limitations, albeit at a substantial time investment, given the difﬁculty of reproducing results in DRL, the large number of publications we considered, and the lack of veriﬁed open-source implementations of many SOTA algorithms.
Still, such a study could focus on a subset of results that are easily reproducible or performed well in our study.
Simpler investigations into how sample efﬁciency depends on hyperparameter tuning for different algorithms could already help to clarify the role of tuning incentives.
Similarly, further monitoring of the observed trends’ continuity, as well as research on the drivers of the observed progress and differences between various domains would be of value.
Researchers in DRL can support this effort by reporting the number of samples that were needed by their ﬁnal algorithm to reach salient performance levels. In particular, sample efﬁciency should be reported for performance levels that were used in earlier publication to make the quantiﬁcation of trends more easy.
A CKNOWLEDGMENTS
Hidden for anonymity.
Will be added to camera-ready version
A B ENCHMARK DESCRIPTIONS A. A TARI (ALE)
The Arcade Learning Environment (Bellemare et al., 2013) provides an interface to a variety of
Atari 2600 games that is adapted to the needs of RL research.
When run in real time, the simulator generates 60 frames per second, but more than 6000 frames per second can be created by a single simulator.
Each frame consists of 210x160 pixels on a 128-colour palette, but preprocessing often involves downscaling to 84x84 pixels and conversion to grayscale images (Mnih et al., 2015).
While some authors use prespeciﬁed features of these pixels as the input to their algorithm (Liang et al., 2015), we only consider methods that directly learn from pixel input.
Many of the games cannot be modelled as a Markov Decision Process (Sutton & Barto, 2018), as vital information is often not visible in every frame 10 .
To circumvent this issue, it is common practice to stack multiple frames and use them as a single observation (Mnih et al., 2015).
Still, this does not always yield full observability (Hausknecht & Stone, 2015), such that many of the more successful solution methods employ recurrent neural networks (Schmitt et al., 2019; Kapturowski et al., 2018).
The action space of the environments in the ALE is discrete and consists of up to 18 actions representing different inputs on the original system’s joystick controller.
The initial DRL benchmark set by DQN (Mnih et al., 2013) consisted of seven games in the ALE, but the nature version of the DQN paper reported 49 games in 2015 (Mnih et al., 2015).
Eight additional games were added in the Double DQN paper (van Hasselt et al., 2016) and the resulting 57 games have been the standard suite of games that ambitious projects try to tackle, since.
To make scores comparable between games, they are often reported as human-normalized scores: professional human game testers repeatedly played each game after around 2 hours of practice and the mean score was collected.
Then, the agent’s results are normalized for this mean score to be at 100%, while the average score for random play is at 0%(Mnih et al., 2015).
As the ALE is deterministic, evaluation is often done using a random amount of up to 30 no-op actions at the beginning of each episode to avoid assigning high scores to algorithms that simply overﬁt to deterministic trajectories (Mnih et al., 2015).
We focus on the median humannormalized score for multiple reasons: First, historically results and especially training curves are published more commonly for this metric than for others.
Apart from that, we care about overall progress, such that an aggregate measure is preferable to single-game scores.
Lastly, the mean score is often dominated by a few outlier games and a less robust indicator of improvements.
Still, the median has its own problems: some games are harder to solve than others and median performance is not sensitive to progress on these games.
A. C ONTINUOUS C ONTROL IN M U J O C O
MuJoCo (Todorov et al., 2012) which stands for “Multi-Joint dynamics with Contact” is a physics engine originally built for model-based control.
It enables users to combine bodies with tendons, different types joints and controllable actuators in a fast and parallelizable simulation 12 that can easily be visualized.
In 2015, MuJoCo was used in early projects working on DRL for continuous control with a focus on locomotion tasks where rewards are obtained by making a simple humanoid or other “robot” move (Heess et al., 2015; Lillicrap et al., 2015; Schulman et al., 2015).
In the next year, Duan et al.
(2016) proposed the rllab benchmark to allow for better quantiﬁcation of progress in continuous control.
The benchmark includes adaptions of six locomotion tasks of varying dimensionality and a set of hierarchical tasks that combine locomotion with more abstract higher-level goals like solving a maze.
Adaptions of many of the rllab environments are included in OpenAI gym 13 (Brockman et al., 2016), a popular toolkit that provides a common interface for a variety of
RL benchmarks.
However, even though the documentation 14 frames gym as an attempt to standardize environments used in publications as subtle differences can have big impacts in RL, it is possible that it had the opposite effect: there are multiple versions of the gym environments that might behave differently, even if a baseline algorithm shows similar performance for most but not all environments when trained on different versions 15 and many publications do not specify which of these versions they use (Nachum et al., 2017; Clavera et al., 2018).
Furthermore, at least the “Humanoid” task clearly behaves differently in the rllab implementation and the gym version (Haarnoja et al., 2018).
Scores for different rllab/gym MuJoCo tasks can be on different scales and normalization is rarely used which makes measuring aggregate performance on the benchmark hard to measure.
The incomparability of scores between tasks in the rllab benchmark was one of the motivations behind the introduction of the DeepMind Control suite (Tassa et al., 2018), another interface to MuJoCo in which tasks similar to the ones in rllab are combined with binary rewards depending on proximity to a goal state and evaluation over 1000 steps, such that attainable overall scores always fall in the interval [0,1000].
Despite the comparability of scores, most publications we looked at did not actually provide training curves for the overall average or median performance on the DeepMind control suite.
For rllab/gym, we looked at Walker2d, Humanoid, HalfCheetah, Hopper and Ant where an armless humanoid, a humanoid with arms, a two-legged “cheetah”, a monopod or a quadruped have to move forward.
For the Deep Mind Control Suite, we looked at Walker Walk, Cheetah Run, Finger Spin and
Ball in Cup Catch.
The ﬁrst two are analogous to Walker2D and HalfCheetah, while in Finger Spin a ﬁnger has to rotate a hinged body and Ball in Cup Catch resembles the Cup-and-ball children’s toy.
B D ATA COLLECTION
The publications we considered in our results were identiﬁed in three ways: 1) drawing from our existing domain knowledge, 2) by searching the top 30 entries in google scholar sorted by relevance (over all time, as well as since 2019 and since 2020) that cite either the paper introducing the environment, the particular benchmark, or the ﬁrst major application of DRL to the particular benchmark for results in a comparable setting and 3) by adding all publications for which quantitative results in a comparable setting were cited in one of the publications already on our list and repeating this procedure until no new publications were identiﬁed.
For papers with multiple versions, the publishing date of the ﬁrst version that contained the relevant result is used as the date in all of our plots 16 .
In case the training curve was not taken from the original paper, we considered the date of the ﬁrst preprint of the original paper that contained the same reported main results as the most recent version 17 of the original paper at the publication time of the paper the data was taken from.
We report the number of samples after which the training curve ﬁrst reaches the speciﬁed score level.
In general, we report the current state-of-the-art for a particular task/score combination at the publication date of each considered paper which was published after the ﬁrst publication reaching that performance level on that task.
We did not take the resolution at which training curves were generated into account such that some of our measured values are actually interpolated.
B. A TARI
The reported results for Atari consist of the best trained agent’s results using 30 no-op actions in the beginning for every game.
We considered the median human-normalized score for DQN (Mnih et al., 2015), Double DQN (van Hasselt et al., 2016), A3C (Mnih et al., 2016), Dueling DQN (Wang et al., 2016a), Prioritized double DQN and Prioritized dueling DQN (Schaul et al., 2015),
C (Bellemare et al., 2017), Rainbow (Hessel et al., 2018), IMPALA (Espeholt et al., 2018), QR-DQN (Dabney et al., 2018a), Noisy Dueling DQN (Fortunato et al., 2017), Ape-x (Horgan et al., 2018),
Meta-gradient (Xu et al., 2018), IQN (Dabney et al., 2018b),
REACTOR 500M (Gruslys et al., 2017), R2D2 (Kapturowski et al., 2018), C51-IDS (Nikolov et al., 2018), LASER (Schmitt et al., 2019), FQF (Yang et al., 2019b), MuZero and MuZero Reanalyze (Schrittwieser et al., 2019), as well as Agent57 (Badia et al., 2020) for ﬁgure 1 but excluded results among these that used exactlymillion frames in ﬁgure 1 (a) and only included state-of-the-art performance (on 200 million or an unrestricted amount of frames) in ﬁgure 1 (b).
Any of these results for which we found sufﬁciently detailed training curves for the median human-normalized score in the paper or another publication were included in ﬁgure 2 (a).
These were DQN, DDQN, Prioritized DDQN, Dueling DDQN, C51,
Rainbow, Noisy DQN and Noisy Dueling DQN, QR-DQN, IQN, LASER, REACTOR and A3C.
Of the results that report median human-normalized performance and the amount of frames used for training we initially identiﬁed, PGQ (O’Donoghue et al., 2016), FRODO (Xu et al., 2020) and
C-Trace DQN (Rowland et al., 2020) were excluded from ﬁgure 1 (b) and 2 (a) and A1 due to a lack of clarity about the use of no-op starts but are still used in ﬁgure 1 (b).
ACER (Wang et al., 2016b) is also not included in ﬁgure 2 (a) and A1 due to unclearly marked axes and missing clarity about what a step consists of in their training curve plots.
The performance under the no-op protocol for
UNREAL is taken from the Ape-x paper.
C51-IDS was only evaluated on 55 games, not on all 57.
As this does not strongly affect the median, the results are still included.
Most results were obtained using a single set of hyperparameters for all games, but Ape-x and UNREAL used separately tuned hyperparameters for every game.
This does not affect the doubling times in ﬁgure 2 (a), as UNREAL was excluded because of its evaluation protocol and the Ape-x publication did not provide training curves.
The results reported for MuZero and R2D2 in the respective publications differ from the ones reported in the Agent57 paper and we use the ones from the original publication.
The same goes for the number of samples reported for R2D2, which are reported to be lower in the MuZero publication than in the original one.
We used the training curves provided in the Rainbow paper for A3C, DQN, DDQN, Dueling DQN, Noisy
DQN, C51 and Rainbow and the training curves from the respective paper for Noisy Dueling DQN,
QR-DQN, IQN, REACTOR and LASER to compare algorithm performance.
The performance results in ﬁgure 2 (b) consider the same publications as the previous ﬁgures, as well as Simple (Kaiser et al., 2019), Data-efﬁcient Rainbow (van Hasselt et al., 2019), OTRainbow (Kielak, 2020), Efﬁcient DQN+DrQ (Kostrikov et al., 2020) and MPR (Schwarzer et al., 2020).
Some of the reported results are approximately lower bounds for the true score, as they are derived from training curves and might include non-greedy actions in the evaluation run.
We essentially made an arbitrary choice about the amounts of frames for which we measured performance on the training curves.
Whenever results from training curves and the ﬁnal reported performance for the same amount of used frames conﬂicted, we chose the result with the larger score, which was usually the reported ﬁnal result.
The results using less than 10 million frames come from Kielak (2020) and Kostrikov et al.
and are medians over a subset of 26 instead of the full 57 games
and it is not clear, whether or not no-op actions were used for these results.
In particular, the results for 2017 in that regime are from runs of a slightly simpliﬁed version of Rainbow with the original hyperparameters in 2019 (Kielak, 2020). The median human-normalized scores for Simple (Kaiser et al., 2019) are taken from the MPR paper.
The training durations used in Table 1 are taken from the Ape-x and Muzero papers.
B. M U J O C O - GYM
Due to the variety of results we considered for continuous control, the protocols for hyperparameter tuning can vary considerably between papers.
Most publications don’t tune hyperparameters on a per-task basis 18 which can make results from publications that used a larger variety of tasks seem worse than they would be with more ﬁne-grained parameter tuning.
For the MuJoCo-gym environments, we only consider results that were reported with the v1 version of the respective environment up to 2019 as the earliest publication of the latest result we found for v1 (Abdolmaleki et al., 2018a) came out in December 2018, but include results that use v2 or an ambiguous version from 2019 and 2020.
Over all, we considered TRPO (Schulman et al., 2015),
DDPG (Lillicrap et al., 2015), Q-Prop (Gu et al., 2016), Soft Q-learning (Haarnoja et al., 2017),
ACKTR (Wu et al., 2017), PPO (Schulman et al., 2017), Clipped Action Policy Gradients (Fujita & Maeda, 2018), TD3 (Fujimoto et al., 2018), STEVE (Buckman et al., 2018), SAC (Haarnoja et al., 2018) and Relative Entropy Regularized Policy Iteration (Abdolmaleki et al., 2018a) for gym-v1.
Apart from that we considered Target Distribution Learning (Zhang et al., 2020), MBPO (Janner et al., 2019), POPLIN (Wang & Ba, 2019), SLAC (Lee et al., 2019), DSPG (Shi et al., 2019), TD with CrossRenorm (Bhatt et al., 2019), SAC-NF (Mazoure et al., 2020), Regularized CEM (Boney et al., 2020) , DAC (Zhang & Whiteson, 2019) and SUNRISE (Lee et al., 2020) which all either use gym-v2 or an ambiguous versions of gym.
As the reported results for the Walker task in the ﬁrst version of the TD3 paper are already strong but improved upon by the results in the second version, we include both versions, labelled as TD3v1 and TD3v2.
The results for Soft Q-learning are taken from the SAC paper, the results for MVE are from the
STEVE paper and the results for TRPO, DDPG and ACKTR, as well as the results on Ant for
PPO, are taken from the TD3 Paper (v2).
We used the reported results for DDPG with the original hyperparameters rather than the improved version the authors of the TD3 paper suggest.
However, we also include the tuned version of DDPG from the Q-Propr paper, as it is the ﬁrst reported method that reached a score of 6000 in half cheetah-v1.
We assign the publication date of Q-prop to this method.
The results for PPO on the Humanoid task are from the SAC paper.
The results in regularized CEM and Q-Prop were reported in episodes rather than time steps.
We converted them into time steps by multiplying by the standard episode length of 1000, but some tasks might include non-step based alternative termination conditions, such that these numbers might be an overestimate.
We excluded Infobot (Goyal et al., 2019) and the results by Rajeswaran et al.
(2017) in RBF-policies due to unclarity what an “iteration” on the x-axis of their learning curve constitutes.
We also excluded ESAC (Suri et al., 2020) due to unclear scaling of the x-axis in their plots (for example the distance between 0 and 0.5 was clearly different from the distance from 0.5 to 1 for their Walker results).
Next, Expected Policy Gradients (Ciosek & Whiteson, 2018) was excluded due to unclearly marked axes.
Lastly, we excluded the variant of the (slightly dated) Reinforce algorithm described by Doan et al.
(2020), as this paper had a theoretical focus and their results did not even get close to the score thresholds we looked at.
We were not able to obtain any reasonable measurements for MVE, STEVE and TD3+CrossRenorm due to insufﬁcient resolution around the zero point of the plot on half cheetah for 2000 score.
As the graphs were cut off below the respective score level, we did not include scores for Relative Entropy Regularized Policy Iteration onscore for Ant.
Also, note that not all papers reported results for all the environments we considered.
B. D EEP M IND C ONTROL S UITE ( ON PIXELS )
For the DeepMind Control Suite, we considered D4PG (Barth-Maron et al., 2018), SAC (Haarnoja et al., 2018), MPO (Abdolmaleki et al., 2018b), PlaNet (Hafner et al., 2019a), SLAC (Lee et al., 2019), SAC+AE (Yarats et al., 2019),
Dreamer (Hafner et al., 2019b), CURL (Srinivas et al., 2020),
DRQ (Kostrikov et al., 2020), RAD (Laskin et al., 2020), SUNRISE (Lee et al., 2020) and Dreaming (Okada & Taniguchi, 2020).
We excluded the method described in Grill et al.
(2020) due to uncertainty about the meaning of “learner step” in the reported learning curves.
Also note that not all papers reported results for all the environments we considered.
As the reported results in the ﬁrst version of the SLAC paper are already strong but improved upon by the results in the second version, we include both versions, labelled as SLACv1 and SLACv2.
We used the training curves for CURL that were provided in DRQ, as the axes in the original publication were labelled in an unclear way.
The data for MPO comes from the SLAC paper (v2).
The results for SAC come from the SAC+AE paper.
The data, as well as the publication date for D4PG, were taken from the DeepMind Control Suite paper (as this included D4PG results before the D4PG paper was published).
As the original publication of the PlaNet results reported the median, not the mean of training curve over different runs like most other results, we took the results from the SLAC paper (v2).
Dreamer also reported results for PlaNet, but these seemed incompatible with the median results reported in the PlaNet paper: The 5% quantile for Cheetah run is reported above the ﬁnal performance of D4PG (524 score) at around 600 score starting at around 1000 (1000 step) episodes in the PlaNet paper, while the mean reported in Dreamer consistently stays below the performance of D4PG for the whole 5 million step training. As the scores are bounded below by 0, the underlying results are clearly different.
Next, we assumed that the x-axis for Finger Spin reported in Sunrise is scaled in thousands as for all other DeepMind Control Suites in the corresponding paper, as the results would otherwise be inconsistent with the reported ﬁnal results and multiple orders of magnitude better than any previous results, which was not discussed by the authors and would be very surprising given that we don’t see anything comparable for any other environment.
As the graphs were cut off below the respective score level, we did not include scores for Sunrise forscore on Walker Walk and 400 as well as 600 score on Finger Spin.
Again, ESAC (Suri et al., 2020) was excluded due to unclear scaling of the x-axis.
C M AIN CONTRIBUTORS TO IMPROVED SAMPLE EFFICIENCY C. I MPROVED OFF - POLICY LEARNING
In all three domains we considered, almost all improvements in sample efﬁciency were achieved by off-policy algorithms that can use the same sample for multiple updates and allow for more elaborate exploration strategies.
While we won’t enumerate all of these improvements, two general strategies stand out and played a role in multiple domains: Counteracting maximization bias by employing a second Q-network (van Hasselt et al., 2016; Fujimoto et al., 2018), and improved exploration by using parametric noise (Fortunato et al., 2017) or by encouraging high entropy policies (Haarnoja et al., 2017; 2018).
C. M ODEL - BASED RL
While not yet up to par with DQN performance on the ALE, model-based algorithms have played an important role for improved sample efﬁciency in continuous control tasks and also helped to catalyze recent improvements in the low-sample regime on the ALE (Kaiser et al., 2019).
Salient features of some model-based approaches with strong sample efﬁciency include the use of a low-dimensional latent state space (Lee et al., 2019), as well as splitting the transition model into a deterministic and a stochastic part and using latent overshooting, which incentivizes longer model rollouts to be accurate (Hafner et al., 2019a), or preventing model trajectories to diverge too far from the training distribution by employing a penalty term (Boney et al., 2020) or by using short rollouts starting in states from a replay buffer (Janner et al., 2019).
C. A UXILIARY OBJECTIVES
As sample efﬁciency is a lot better for state-based continuous control than for the pixel-based version on the DeepMind Control Suite, various recent approaches tackling the pixel-based version employed auxiliary objectives to quickly learn a state representation that is amenable to sample efﬁcient learning.
Approaches with strong sample efﬁciency used an autoencoder on frames for representation learning (Yarats et al., 2019) or incentivized Q-values to be similar for different image augmentations applied to the same game frame (Kostrikov et al., 2020).
C. I NCENTIVES FOR SAMPLE EFFICIENCY CAN HAVE A LARGE EFFECT
Often, explicitly tuning hyperparameters for sample efﬁciency can have a large effect as can be seen in the substantial improvements to Rainbow’s performance in the low-data regime where sample efﬁciency for median performance on 27 Atari games improved by more than 10x when hyperparameters were tuned accordingly (van Hasselt et al., 2019; Kielak, 2020).
This was achieved by increasing the frequency of updates to the Q network, relative to the frequency of sample collection, such that comparable gains seem plausible for other approaches using experience replay.
Furthermore, incentives for sample efﬁciency also inﬂuence the focus on sample efﬁciency in algorithm development.
This might in part explain the comparatively fast doubling times for the pixel-based continuous control domain where many recent results (Hafner et al., 2019b; Lee et al., 2019) explicitly focus on sample efﬁciency.
C. T HE ROLE OF COMPUTE
A lot of the previously discussed improvements like updating the Q-network more often or using a second Q-network do require additional compute.
However, these additional requirements do not seem to keep up with the 3.4 month doubling times in compute usage for the largest AI projects, as they are usually way smaller than the increase of a factor of 32x per sample reported in Kielak (2020).
While experiments suggest that increases in network size which exacerbate the compute requirement per network update can lead to improved DRL performance (Espeholt et al., 2018), network size stayed constant for the different versions of DQN that dominate our comparison of sample efﬁciency on the ALE and improved sample efﬁciency did not seem to strongly correspond to larger network size, more generally.
Nevertheless, further research into the role of compute for
RL performance and sample efﬁciency would certainly be of value.
D L IMITATIONS D. W E ONLY CONSIDER THREE TYPES OF RL TASKS
While we cover two major axes of variation within RL tasks by including both pixel-based and statebased tasks as well as discrete action and continuous action tasks, we observe a lot of between-task variation in sample efﬁciency doubling times.
It is very plausible that this variation would be even bigger if other types of problem classes like hard-exploration problems (Ecoffet et al., 2019) were considered as well.
Also, we chose tasks by popularity.
This might introduce bias, as popularity most likely correlates with sustained speed of progress.
D. T HE LACK OF COMPREHENSIVE COLLECTIONS OF BENCHMARK RESULTS IMPLIES THAT WE MIGHT HAVE MISSED RELEVANT RESULTS .
While we conducted a fairly extensive literature review to identify relevant publications, we may have missed some results, as there sadly is no comprehensive repository that collects benchmark results for RL.
We also had to exclude ALE results that did not report median human-normalized score from our analysis.
Still, we were able to look at more than ten data points for most task/score combinations we considered.
Relatedly, our literature review might have identiﬁed more recent nonSOTA results with a higher probability than older non-SOTA results as we searched for the most highly cited papers overall but also the most highly cited papers from the last one and two years to avoid missing improvements on the SOTA that did not yet have had enough time to compete with older papers’ citation counts.
D. R ELYING ON PUBLISHED LEARNING CURVES IMPLIES NOISY RESULTS
Reporting the number of samples after which the training curve ﬁrst reaches the speciﬁed score level introduces some amount of bias and noise as training performance randomly ﬂuctuates.
More generally, the reliance on training curves reported in other publications forced us to interpolate some results when learning curve plots had insufﬁcient resolution and to exclude a few results for which there was insufﬁcient resolution around zero frames or the plot was cut off above the relevant score level.
Also, the exact evaluation protocol used for generating training curves on the ALE was often ambiguous and different publications might have uses subtly different evaluation protocols (Machado et al., 2018).
In particular, we observed that the provided training curves rarely reached the reported maximum performance for Atari.
As this effect is rather consistent between algorithms it cannot simply be explained by noise in the evaluation as this would lead to the reversed effect similarly often.
While the effect is of similar magnitude for different algorithms it is second strongest for DQN, the earliest algorithm we considered, which might inﬂate perceived progress by making the baseline result worse.
Lastly, the possibility of plotting or transcription errors is particularly problematic because we estimate doubling times in SOTA sample efﬁciency and the SOTA could be set to an extreme value for a long time by a noisy outlier which might explain the bad ﬁt and long doubling time for C51, where the measured state-of-the-art only changes once in three years (Figure 4).
Luckily, we can see the state-of-the-art changing regularly in most of our plots which indicates that this problem is not too severe.
D. W E WERE NOT ABLE TO TUNE HYPERPARAMETERS
As we did not replicate any experiments, we were unable to tune different algorithms to reach the respective baseline performance as fast as possible.
This is problematic because the easy generation of samples using additional compute in simulated tasks combined with the usual focus on ﬁnal performance 19 reduces the incentive for sample efﬁciency in published results.
This effect would have little consequence if it was similarly large for all publications but distorts observed trends in sample efﬁciency if the focus on sample efﬁciency increases over time, which has arguably happened.
On the other hand, our trendlines don’t seem to systematically underestimate the sample usage of early results by much, which would be expected if the described effect was large
. D. T HE AMOUNT OF SAMPLES USED BY THE FINAL ALGORITHM UNDERESTIMATES THE AMOUNT OF SAMPLE USED IN RESEARCH AND DEVELOPMENT .
Similar to previous work on algorithmic efﬁciency in machine learning (Hernandez & Brown, 2020), our analysis only considers samples that were used for the training the ﬁnal agent.
However, the number of samples that are used to solve challenging DRL problems can often greatly exceed this number as exempliﬁed by OpenAI ﬁve, for which a rerun with the ﬁnal network architecture took less than a third as long as the initial training during which the network was repeatedly improved (Berner et al., 2019).
More generically, hyperparameter tuning requires samples which are often not reported.
Depending on how much good hyperparameters generalize between tasks, this might be less of an issue for real world applications, as they could be tuned in simulations.
D. S OLVING HARD , PREVIOUSLY UNSOLVED PROBLEMS CAN BE INTERPRETED AS MORE PROGRESS IN SAMPLE EFFICIENCY THAN SOLVING OLD TASKS MORE QUICKLY
As with algorithmic efﬁciency in machine learning (Hernandez & Brown, 2020), one could argue that most progress in sample efﬁciency is actually represented in the ability to solve previously unsolved tasks, rather than solving already solved tasks using fewer samples.
However, the automation of tasks that can currently be solved in simulations like autonomous driving (Sallab et al., 2016), adaptive trafﬁc light control (Liang et al., 2018) and piloting ﬁghter jets (Tucker, 2020) would already come with a lot of economic or military value.
Ultimately, it does not matter, whether we could train a robot to manufacture an abacus or a supercomputer in a simulation, as long as we struggle to train it to reliably manipulate simple objects in the real world (Zhu et al., 2020).
D. W E DO NOT MEASURE PROGRESS IN ROBOTICS , SIM 2 REAL OR TRANSFER LEARNING .
Arguably, progress in robotics and other real world applications of DRL is what we ultimately care about.
However, while there exist benchmark tasks for robotics (Mahmood et al., 2018; Yang et al., 2019a; Ahn et al., 2020), these benchmarks have received little attention: Mahmood et al.
2018 was only cited 31 times and only one of the citing publications contained a result on the benchmarks while he other two benchmark publications were not even cited ten times.
This makes quantifying progress difﬁcult. We also do not assess progress in simulation quality, which would reduce the need for sample efﬁcient algorithms, even if successfully calibrating these simulations is challenging (Irpan, 2019; OpenAI et al., 2020) and does require the use of real transition data as ground truth.
Similarly, we don’t measure progress in sim2real including dynamics randomization and related approaches where an agent is trained on a variety of differently calibrated simulations to increase its robustness to simulation parameters (Peng et al., 2018; James et al., 2019; van Baar et al., 2019) or progress in RL transfer learning and generalization (Goyal et al., 2019; Sekar et al., 2020) which might reduce the need for calibration data.
D. P ROGRESS MIGHT BE SLOWER FOR OFFLINE REINFORCEMENT LEARNING
In the ofﬂine RL setting (Levine et al., 2020; Agarwal et al., 2020), the agent learns from a ﬁxed dataset rather than from its own experience.
As it might be easier to collect data produced by a human operator or an inefﬁcient but save heuristic rather than an untrained agent, ofﬂine learning could play an important role for many real world applications of RL 20 (Agarwal et al., 2020; Fu et al., 2020).
Due to the ﬁxed dataset, improved exploration strategies cannot help with progress in ofﬂine RL such that we expect progress in ofﬂine RL sample efﬁciency to be slower than general progress in RL sample efﬁciency in the long run.
This means we might be overestimating progress relevant for real world applications of DRL if most of these were using the ofﬂine setting.
Further investigations into trends in sample efﬁciency in ofﬂine RL would be valuable but challenging due to the lack of established evaluation protocols (Levine et al., 2020).
E D ETAILED RESULTS
[Table]
[Figure] (continuous until [Table]
[Figure] [Figure] [Table] [Table]