S TRIVING FOR S IMPLICITY IN O FF -P OLICY D EEP R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper advocates the use of ofﬂine (batch) reinforcement learning (RL) to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler deep RL algorithms.
We demonstrate that recent off-policy deep RL algorithms, even when trained solely on all of the replay data of an online DQN agent, can outperform online DQN on Atari 2600 games.
We also present Random Ensemble Mixture (REM), a simple Q -learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q -value estimates.
Using the DQN replay dataset, ofﬂine REM surpasses the gains from online C51 and outperforms ofﬂine QR-DQN.
Furthermore, REM performs comparably to QR-DQN in the online RL setting on Atari 2600 games. The DQN replay dataset can serve as an ofﬂine RL benchmark and will be released.
I NTRODUCTION
Deep neural networks have become a critical component of modern reinforcement learning (RL) (Sutton and Barto, 2018).
The seminal work of Mnih et al.
(2013; 2015) on deep Q -networks (DQN) has demonstrated that it is possible to train neural networks using Q -learning (Watkins and Dayan, 1992) to achieve human-level performance in playing Atari 2600 games (Bellemare et al., 2013) directly from raw pixels.
Recent progress in mastering Go (Silver et al., 2016) and advances in robotic control (Levine et al., 2016; OpenAI et al., 2018; Kalashnikov et al., 2018) present additional supporting evidence for the enormous potential of deep RL.
Off-policy RL algorithms such as Q -learning are attractive because they disentangle data collection and policy optimization and offer more sample efﬁcient solutions than on-policy algorithms (Sutton et al., 2000; Schulman et al., 2015; Mnih et al., 2016).
Importantly, off-policy techniques can leverage the vast amount of existing ofﬂine logged data for real-world applications such as digital advertising (Strehl et al., 2010; Bottou et al., 2013), education (Mandel et al., 2014), and healthcare (Shortreed et al., 2011).
Since online RL is often unsafe to deploy in the real world, ofﬂine RL algorithms are the only feasible solution for many practical decision making problems (Dulac-Arnold et al., 2019).
Nevertheless, off-policy RL algorithms when combined with neural networks can be unstable or even divergent (Baird, 1995; Boyan and Moore, 1995; Tsitsiklis and Van Roy, 1997).
In the absence of theoretical guarantees for off-policy deep RL, recent advances (see Hessel et al.
(2018) for an overview) are largely governed by empirical results on a popular benchmark suite of
Atari 2600 games (Bellemare et al., 2013).
Unfortunately, these empirical results are difﬁcult to reproduce (Henderson et al., 2018) and the contribution of novel RL algorithms is often conﬂated with many other design choices (Clary et al., 2019; Khetarpal et al., 2018).
Given the empirical nature of deep off-policy RL, it is crucial to come up with simpler and reproducible experimental settings and study the relative importance of different components of deep RL algorithms.
It is also important to strive for ﬁnding successful RL algorithms that are as simple as possible.
This paper advocates the use of ofﬂine (batch) RL to help (1) isolate the contributions of exploitation vs. exploration in off-policy deep RL, (2) improve reproducibility of deep RL research, and (3) facilitate the design of simpler off-policy deep RL algorithms.
To this end, we propose a new benchmark for ofﬂine optimization of RL agents on Atari 2600 games using all of the replay data of a DQN agent (Mnih et al., 2015).
Using this benchmark, we investigate the following questions:
Is it possible to train successful Atari agents based solely on ofﬂine data? 2.
Can one design simple and effective alternatives to intricate RL algorithms in the ofﬂine setting?
Are the insights gained from the ofﬂine setting useful for developing effective online algorithms?
[Figure]
The contributions of this paper can be summarized as:
• An ofﬂine RL benchmark is proposed for evaluating and designing RL algorithms on Atarigames without exploration, based on the logged replay data of a DQN agent comprising 50 million (observation, action, reward, next observation) tuples per game.
This reduces the computation cost of the experiments considerably and helps improve reproducibility of deep RL research by standardizing training using a ﬁxed ofﬂine dataset.
The replay dataset used in our experiments will be released to enable ofﬂine optimization of RL algorithms on a common ground.
• Contrary to recent work (Zhang and Sutton, 2017; Fujimoto et al., 2019), we ﬁnd that the logged DQN data is sufﬁcient for optimizing strong Atari agents ofﬂine without any environment interaction.
For instance, QR-DQN (Dabney et al., 2018b) trained ofﬂine on the DQN replay dataset signiﬁcantly outperforms online DQN (Mnih et al., 2015).
• A simple and novel Q -learning algorithm called Random Ensemble Mixture (REM) is presented, which enforces optimal Bellman consistency on random convex combinations of multiple Q -value estimates.
Ofﬂine REM, trained using the DQN replay dataset, outperforms more complex RL algorithms such as ofﬂine QR-DQN and surpasses the gains from online C51 (Bellemare et al., 2017) over online DQN (Figure 1a).
• We use the insights gained from the ofﬂine experiments to develop an online variant of REM, which performs comparably with online QR-DQN (Figure 1b).
O FF - POLICY R EINFORCEMENT L EARNING
An interactive environment in reinforcement learning (RL) is typically described as a Markov decision process (MDP) ( S , A , R, P, γ ) (Puterman, 1994), with a state space S , an action space A , a stochastic reward function R ( s, a ) , transition dynamics P ( s (cid:48) | s, a ) and a discount factor γ ∈ [0 , 1) .
A stochastic policy π ( · | s ) maps each state s ∈ S to a distribution (density) over actions.
For an agent following the policy π , the action-value function, denoted Q π ( s, a ) , is deﬁned as the expectation of cumulative discounted future rewards, i.e.,
Q
[Equation]
The goal of RL is to ﬁnd an optimal policy π ∗ that attains maximum expected return, for which
[Equation] ) for all π, s, a .
The Bellman optimality equations (Bellman, 1957) characterize the optimal policy in terms of the optimal Q -values, denoted Q ∗ =
Q π ∗ , via:
[Equation]
To learn a policy from interaction with the environment, Q -learning (Watkins and Dayan, 1992) iteratively improves an approximate estimate of Q ∗ , denoted Q θ , by repeatedly regressing the
LHS of (2) to target values deﬁned by samples from the RHS of (2).
For large and complex state spaces, approximate Q -values are obtained using a neural network as the function approximator.
To further stabilize optimization, a target network Q θ (cid:48) with frozen parameters may be used for computing the learning target (Mnih et al., 2013).
The target network parameters θ (cid:48) are updated to the current Q -network parameters θ after a ﬁxed number of time steps.
DQN (Mnih et al., 2013; 2015) parameterizes Q θ with a convolutional neural network (LeCun et al., 1998) and uses Q -learning with a target network while following an (cid:15) -greedy policy with respect to Q θ for data collection.
DQN minimizes the TD error using the loss L ( θ ) on mini-batches of agent’s past experience tuples, ( s, a, r, s
[Equation]
) , sampled from an experience replay buffer D (Lin, 1992) collected during training:
[Equation] [Equation]
) where l δ is the Huber loss (Huber, 1964) given by (cid:96) δ ( u ) = (cid:26) 12 ufor | u | ≤ δ
[Equation]
δ ( | u | − 12 δ ) , otherwise. .
Q -learning is an off-policy algorithm (Sutton and Barto, 2018) since the learning target can be computed without any consideration of how the experience was generated.
In ofﬂine (batch) RL (Ernst et al., 2005; Riedmiller, 2005; Lange et al., 2012), we assume access to a ﬁxed ofﬂine dataset of experiences D , without any further interaction with the environment.
A family of recent off-policy deep RL algorithms, which serve as a strong baseline in this paper, include Distributional RL (Bellemare et al., 2017; Jaquette, 1973) methods.
Such algorithms estimate a density over returns for each state-action pair, denoted Z π ( s, a ) , instead of directly estimating the mean Q π ( s, a ) .
Accordingly, one can express a form of distributional Bellman optimality as
Z ∗ ( s, a ) D = r + γZ
[Equation]
( s (cid:48) , argmax
[Equation]
(cid:48)
[Equation]
( s (cid:48) ,
[Equation]
) , where r ∼ R ( s, a ) , s (cid:48) ∼ P ( · | s, a ) , (4) and D = denotes distributional equivalence and Q ∗ ( s (cid:48) , a (cid:48) ) is estimated by taking an expectation with respect to Z ∗ ( s (cid:48) , a (cid:48) ) .
C51 (Bellemare et al., 2017) approximates Z ∗ ( s, a ) by using a categorical distribution over a set of pre-speciﬁed anchor points, and distributional QR-DQN (Dabney et al., 2018b) approximates the return density by using a uniform mixture of K Dirac delta functions, i.e.,
Z
[Equation]
( s,
[Equation]
=
[Equation]
K (cid:88) i =
[Equation]
and Q θ ( s, a ) = 1 K
K (cid:88) i
=
θ
i
( s, a ) .
QR-DQN trains each θ i for 1 ≤ i ≤ K to match the 2 i − 1 2 K -quantile of the target return density (RHS of (4)) using the Huber quantile regression loss (Koenker, 2005; Dabney et al., 2018b).
QR-DQN, albeit complex, outperforms C51 and DQN and obtains state-of-the-art results on Atari 2600 games, among agents that do not exploit n -step updates (Sutton, 1988) and prioritized replay (Schaul et al., 2016).
This paper avoids using n -step updates and prioritized replay to keep the empirical study simple and focused on deep Q -learning algorithms.
O FFLINE O PTIMIZATION OF R EINFORCEMENT L EARNING A GENTS
Modern off-policy deep RL algorithms perform remarkably well on common benchmarks such as the
Atari 2600 games (Bellemare et al., 2013) and continuous control MuJoCo tasks (Todorov et al., 2012).
Such off-policy algorithms (Mnih et al., 2015; Lillicrap et al., 2015) are considered “online” because they alternate between optimizing a policy and using that policy to collect more data.
Typically, these algorithms keep a sliding window of most recent experiences in a ﬁnite replay buffer (Lin, 1992), throwing away stale data to incorporate most fresh (on-policy) experiences.
In principle, off-policy algorithms can learn from data collected by any policy, however, Zhang and Sutton (2017) assert that using a large replay buffer can signiﬁcantly hurt the performance of Q -learning algorithms, since it can delay rare on-policy experiences that are important for policy learning.
This paper revisits ofﬂine off-policy RL and investigates whether off-policy deep RL agents trained solely on ofﬂine data can be successful.
We advocate the use of ofﬂine RL to help isolate an
RL algorithm’s ability to exploit experience and generalize vs. its ability to explore effectively by collecting interesting new data.
The ofﬂine RL setting removes design choices related to the replay buffer and exploration; therefore, it is much simpler to experiment with and reproduce than the typical online off-policy learning.
Also, in the ofﬂine setting, we optimize an objective ( e.g., (3)) over a ﬁxed dataset as opposed to a changing replay buffer of an online agent. In this work, we mainly experiment
C
M
E M O N A TT
AC K
B REA KO U T V E N T
U
RE
Game − 10 3 − 10 2 − 10− 10 0 0 1010 1 10 2 10% I m p
r
o v e m e n t ( L o g s c a l e )
DQN
Random
OfﬂineDQN(Nature)
M O N T E Z U M A ’ S
R EVE N G E S
O L AR I S
D O U B L E D
UN K
K A N G AR OO S K II N G H .
E .
R .
O A M
I D AR
J A M E S B O N
D
C E N T I P E D E
C RA Z Y C L I M
BER A L I E N C AR
N
I
VA
L A TL A N T
I S K R U
LL P O N G T E
NN I S
M S .
P AC - M A
N
F REE W A Y P I TF A LL !
B EA M R I D
ER R
I
VER
R
A I D
Y AR S ’ S R EVE
N G E
J O U R N E
Y
E
S CA P E
F I S H
I N G D
ERB Y
S T AR G UNN
ER Q * B ER T
V
I D E O P
I N
BA LL
R O B O T A N K
K UN G - F U M
A S T ER B O X
I N G
R O A D R UNN
ER
B A N K H E I S T B O W L
I N G P
H
O E
N I X
T U T A N K H
A M B A TTL E Z O N E P OO Y
A N I CE H O C K E Y
N A M E T H
I S
G A M E
U P A N D D O W
N
F R O S T B
I T E
Z
A XX O N A
I R R
A
I D
W I Z
AR D O F
W O R A S T
ER
I X V E N T
U RE
E N D
U
R O
P R
I VA T E E Y
E
B REA KO U T
A S T
ER O I D S G O P
H ER S EA Q U E S T A SS A U LT G RAV
I T
AR T
I M E P I
L O T
C H O PP ER
C O MM
A N D
S P ACE I N VA
D ER S
D E M O N A TT
AC K
B ER Z ER K
E L EVA T O R
A
C T
I O N
Game − 10 3 − 10 2 − 10− 10 0 0 1010 1 10 2 10% I m p
r
o v e m e n t ( L o g s c a l e )
DQN
Random
OfﬂineQR-DQN
[Figure]
with large and diverse datasets motivated by real-world RL problems where we already have access to (usually continually growing) diverse datasets of logged experiences, e.g., historical data from recommender systems (Bottou et al., 2013), database of robotic experiences (Dasari et al., 2019) etc .
To facilitate a study of ofﬂine RL with large and diverse datasets, we train several instances of
DQN (Nature) agents (Mnih et al., 2015) on stochastic version of 60 Atari 2600 games formillion frames each, with a frame skip of 4 (standard protocol) and sticky actions enabled (Machado et al., 2018).
On each game, we train 5 different agents with random initialization, and store all of the tuples of (observation, action, reward, next observation) encountered during training into 5 replay datasets of 50 million tuples each.
These replay datasets are used for training standard off-policy RL agents, ofﬂine, without any interaction with the environment during training.
We perform an online evaluation of the agents in the intervals of 1 million training frames, and report the best evaluation score for each agent, averaged over 5 runs.
3. E XPERIMENTS AND R ESULTS
Given the logged replay data of the DQN agent, it is natural to ask whether training an ofﬂine variant of DQN on this data will recover the performance of online DQN.
Furthermore, whether more recent off-policy algorithms are able to exploit the DQN replay dataset more effectively than DQN.
To investigate these questions, we train DQN (Nature) and distributional QR-DQN (Dabney et al., 2018b) agents, ofﬂine, on the DQN replay dataset for the same number of gradient updates as online
DQN.
We use the hyperparameters provided in the Dopamine baselines (Castro et al., 2018) for a standardized comparison (Section A.4) and report scores using a normalized scale (Section A.3).
We note that the replay datasets include samples from all of the intermediate policies seen during the optimization of the DQN (Nature) agent.
Accordingly, we compare the performance of ofﬂine agents against the best performing policy among the mixture of data collecting policies.
Results.
We ﬁnd that ofﬂine DQN underperforms online DQN on all except a few games while ofﬂine
QR-DQN outperforms ofﬂine DQN and online DQN on most of the games (Figure 2).
C51 (Bellemare et al., 2017) trained ofﬂine using the DQN replay dataset also considerably improves upon ofﬂine
DQN (Figure A.4).
These results suggest that DQN (Nature) is somewhat ineffective at exploiting off-policy data.
However, ofﬂine QR-DQN’s impressive performance demonstrates that it is possible to optimize strong Atari agents completely ofﬂine (see Figure A.6 for learning curves).
3. O FFLINE C ONTINUOUS C ONTROL E XPERIMENTS
In contrast to our ofﬂine results on Atari 2600 games with discrete action spaces, discussed above,
Fujimoto et al.
ﬁnd that standard off-policy deep RL agents are not effective on continuous control problems when trained ofﬂine, even when large and diverse replay datasets are used.
The results of Fujimoto et al.
(2019) are based on the evaluation of a standard continuous control agent, called DDPG (Lillicrap et al., 2015), and other more recent continuous control algorithms such as
TD3 (Fujimoto et al., 2018) and SAC (Haarnoja et al., 2018) are not considered in their study.
Motivated by the so-called ﬁnal buffer setting in Fujimoto et al.
(Section A.2), we train a DDPG agent on continuous control MuJoCo tasks (Todorov et al., 2012) for 1 million time steps and store all of the experienced transitions.
Using this dataset, we train standard off-policy agents including
TD3 and DDPG completely ofﬂine.
Consistent with our ofﬂine results on Atari games, ofﬂine TD signiﬁcantly outperforms the data collecting DDPG agent and ofﬂine DDPG (Figure A.1).
S EEKING S IMPLER O FF -P OLICY RL A LGORITHMS
Section 3.1 ﬁnds that recent distributional RL algorithms such as QR-DQN are effective at exploiting ofﬂine data.
However, these algorithms are complicated as they require a generalization of Bellman equations to distributions (4) and typically minimize a probability divergence measure from a distributional target (Rowland et al., 2018; Dabney et al., 2018b) instead of scalar TD errors.
Moreover, despite recent efforts (Lyle et al., 2019; Bellemare et al., 2019), the source of the gains when using distributional RL algorithms remains unclear.
We investigate whether one can develop simple and effective alternatives to more intricate RL algorithms in the ofﬂine setting.
To this end, we study two deep Q -learning algorithms, Ensemble DQN and REM, which adopt ensembling to improve stability.
4. E NSEMBLE -DQN
Ensemble-DQN is a simple extension of DQN that approximates the Q -values via an ensemble of parameterized Q -functions (Faußer and Schwenker, 2015;
Osband et al., 2016; Anschel et al., 2017).
Each Q -value estimate, denoted Q kθ ( s, a ) , is trained against its own target Q kθ (cid:48)
( s, a ) , similar to Bootstrapped-DQN (Osband et al., 2016).
The Q -functions are optimized using identical minibatches in the same order, starting from different parameter initializations.
The loss L ( θ ) takes the form,
[Equation]
where l δ is the Huber loss.
While Bootstrapped-DQN uses one of the Q -value estimates in each episode to improve exploration, in the ofﬂine setting, we are only concerned with the ability of
Ensemble-DQN to exploit better and use the mean of the Q -value estimates for evaluation.
4. R ANDOM E NSEMBLE M IXTURE (REM)
Increasing the number of models used for ensembling typically improves the performance of supervised learning models (Shazeer et al., 2017).
This raises the question whether one can use an ensemble over an exponential number of Q -estimates in a computationally efﬁcient manner.
Inspired by dropout (Srivastava et al., 2014), we propose Random Ensemble Mixture for RL.
Random Ensemble Mixture (REM) uses multiple parameterized Q -functions to estimate the Q -values, similar to Ensemble-DQN.
The key insight behind REM is that one can think of a convex combination of multiple Q -value estimates as a valid Q -value estimate itself.
This is especially true at the ﬁxed point, where all of the Q -value estimates have converged to an identical Q -function.
Using this insight, we train a family of Q -function approximators deﬁned by mixing probabilities on a ( K − 1) -simplex. [Figure]
[Table] Speciﬁcally, for each mini-batch we randomly draw a categorical distribution α , which deﬁnes a convex combination of the K estimates to approximate the optimal Q -function.
This approximator is trained against its corresponding target to minimize the TD error.
The loss L ( θ ) takes the form,
[Equation]
(7) where P ∆ represents a probability distribution over the standard ( K − 1) -simplex
∆ K − 1 = { α ∈ R K : α 1 + α 2 + · · · + α K = 1 , α k ≥ 0 , k = 1 , . . .
, K } .
REM considers Q -learning as a constraint satisfaction problem based on Bellman optimality constraints (2) and L ( θ ) can be viewed as an inﬁnite set of constraints corresponding to different mixture probability distributions.
For action selection, we use the average of the K value estimates as the Q -function, i.e., Q ( s, a ) = (cid:80) k Q kθ ( s, a ) /K .
REM is easy to implement and analyze (see Proposition 1), and can be viewed as a simple regularization technique for value-based RL.
In our experiments, we use a very simple distribution P ∆ : we ﬁrst draw a set of K values i.i.d.
from Uniform (0, 1) and normalize them to get a valid categorical distribution, i.e., α (cid:48)
k ∼ U (0 , 1) followed by α k
= α (cid:48)
k / (cid:80) α (cid:48) i .
Proposition 1 .
Consider the assumptions: (a) The distribution P ∆ has full support over the entire ( K − 1) -simplex.
(b) Only a ﬁnite number of distinct Q -functions globally minimize the loss in (3) .
(c) Q ∗ is deﬁned in terms of the MDP induced by the data distribution D .
(d) Q ∗ lies in the family of our function approximation.
Then, at the global minimum of L ( θ ) (7) for a multi-head Q -network: (i) Under assumptions (a) and (b), all the Q -heads represent identical Q -functions.
(ii) Under assumptions (a)–(d), the common global solution is Q ∗ .
The proof of (ii) follows from (i) and the fact that (7) is lower bounded by the TD error attained by
Q ∗ .
The proof of part (i) can be found in the supplementary material.
4. E XPERIMENTS AND R ESULTS
Asymptotic performance of ofﬂine agents.
In supervised learning, asymptotic performance matters much more than performance within a ﬁxed budget of gradient updates.
Similarly, for a given sample complexity, we prefer RL algorithms that perform the best as long as the number of gradient updates is feasible.
Since the sample efﬁciency of the ofﬂine agents for a given dataset is ﬁxed, we train them on the DQN replay dataset for ﬁve times as many gradient updates as online DQN.
Comparison with QR-DQN.
QR-DQN modiﬁes the DQN (Nature) architecture to output K values for each action using a multi-head Q -network and replaces RMSProp (Tieleman and Hinton, 2012) with Adam (Kingma and Ba, 2015) for optimization.
To ensure a fair comparison with QR-DQN, we use the same multi-head Q -network as QR-DQN with K = 200 heads (Figure 3), where each head represents a Q -value estimate for REM and Ensemble-DQN.
We also use Adam for optimization.
Additional Baselines.
To isolate the gains due to Adam in QR-DQN and our proposed variants, we compare against a DQN baseline which also uses Adam.
We also evaluate Averaged EnsembleDQN, a variant of Ensemble-DQN proposed by Anschel et al.
(2017), which uses the average of the predicted target Q -values as the Bellman target for training each parameterized Q -function. This
[Figure] baseline determines whether the random combinations of REM provide any signiﬁcant beneﬁt over simply using an ensemble of predictors to stabilize the Bellman target.
REM with separate Q -networks.
To investigate whether the effectiveness of REM is linked to ensembling techniques and dropout, we compare ofﬂine REM with K Q -value estimates computed using K separate Q -networks vs. a multi-head Q -network for K ∈ { 4 , 16 } .
Results.
Figure 4a shows the comparison of the additional baselines with REM and Ensemble-DQN and Table 1 summarizes the asymptotic performance results.
DQN with Adam noticeably bridges the gap in asymptotic performance between QR-DQN and DQN (Nature) in the ofﬂine setting.
Ofﬂine
Ensemble-DQN does not improve upon this strong DQN baseline showing that its naive ensembling approach is inadequate.
Furthermore, Averaged Ensemble-DQN performs only slightly better than
Ensemble-DQN.
In contrast, REM exploits ofﬂine data more effectively than other agents, including
QR-DQN, when trained for more gradient updates.
Surprisingly, using the DQN replay dataset, ofﬂine REM surpasses the gains from online C51, a huge improvement over online DQN (Figure 1a).
Based on above empirical results, we hypothesize that gains from REM are due to the noise from randomly ensembling Q-value estimates leading to more robust training analogous to dropout.
Furthermore, REM with separate Q -networks performs better asymptotically and learns faster than REM with a multi-head Q -network in the ofﬂine setting (Figure A.5).
This consolidates our belief about effectiveness of REM being linked to dropout as separate Q -networks result in more variation in individual Q -estimates leading to more robust training.
4. F ROM O FFLINE TO O NLINE R EINFORCEMENT L EARNING
Can we combine the insights gained from the ofﬂine setting with appropriate design choices ( e.g., exploration, replay buffer) to create effective online methods?
In online RL, learning and data generation are tightly coupled, i.e., an agent that learns faster also collects more relevant training data.
We ran online REM with K separate Q -networks (with K = 4 for computational efﬁciency) because of the improved convergence speed over multi-head REM in the ofﬂine setting.
For data collection, we use (cid:15) -greedy with a randomly sampled Q -estimate from the simplex for each episode, similar to Bootstrapped DQN.
To estimate the gains from the REM objective (7) in the online setting, we also evaluate Bootstrapped-DQN with identical modiﬁcations ( e.g., separate Q -networks) as online REM.
Figure 1b show that REM performs on par with QR-DQN and considerably outperforms
Bootstrapped-DQN (see Figure A.8 for learning curves).
Based on these results, we conclude that the ofﬂine setting can help facilitate the design of effective online off-policy RL algorithms.  O FFLINE [Equation]
S IZE AND C OMPOSITION
Our ofﬂine learning results indicate that 50 million logged experience tuples per game from DQN (Nature) are sufﬁcient to obtain good online performance on most of the Atari 2600 games.
One may argue that the size of the ﬁxed replay and its composition play a key role in our empirical results (De Bruin et al., 2015).
To study the role of the replay buffer size in the success of the ofﬂine agents, we perform an ablation experiment with variable replay buffer size. We conducted ofﬂine experiments with reduced data obtained via randomly subsampling entire trajectories from the logged DQN experiences, thereby maintaining the same data distribution. Figure 5 presents the performance of the ofﬂine REM and QR-DQN agents with N % of the tuples in the DQN replay dataset where N ∈ { 1 , 10 , 20 , 50 , 100 } . As expected, performance tends to increase
[Figure]
as the fraction of data increases.
With N ≥ 10% , REM and QR-DQN still perform comparably to online DQN on most of these games.
However, the performance deteriorates drastically for N = 1% .
To see the effect of quality of ofﬂine dataset, we perform another ablation where we train ofﬂine
REM and QR-DQN on the ﬁrst 20 million frames in the DQN replay dataset which approximate exploration data with suboptimal returns.
Similar to our ofﬂine results with the entire replay dataset, on most Atari games, ofﬂine REM and QR-DQN perform comparably and outperform the best policy amongst the mixture of policies which collected the ﬁrst 20 million frames (Figure 6 and Figure A.9).
D ISCUSSION AND F UTURE W ORK
Since the (observation, action, next observation, reward) tuples in DQN replay dataset are stored in the order they were experienced by online DQN during training, various data collection strategies for benchmarking ofﬂine RL can be induced by subsampling the replay dataset containing 200 million frames.
For example, the ﬁrst k million frames from the DQN replay dataset emulate exploration data with suboptimal returns ( e.g., Figure 6) while the last k million frames are analogous to nearexpert data with stochasticity.
Another option is to randomly subsample the entire dataset to create smaller ofﬂine datasets with sufﬁcient diversity ( e.g., Figure 5).
Based on the popularity and ease of experimentation on Atari games, we hope that the DQN replay dataset can be used for benchmarking ofﬂine RL in addition to continuous control setups Fujimoto et al.
; Kumar et al.
Ofﬂine RL tackles the challenge of squeezing the most performance gains out of a ﬁxed batch of data.
This indicates the utility of ofﬂine RL for creating more sample-efﬁcient RL agents as an online agent should exploit the data collected prior to any given time during online training as much as possible before further collecting new data via exploration.
The proposed ofﬂine benchmark can be utilized for developing better RL algorithms for various data regimes shown in Figure 5.
In our ofﬂine RL setup, the agents exhibit overﬁtting on some games (Figure 6), even with 50 million tuples (Figure A.7), i.e., after a sufﬁciently large number of gradient updates, their performance degrades.
We also observe divergence w.r.t.
Huber loss in our reduced data experiments with N = 1% .
Based on these ﬁndings, we posit that our ofﬂine benchmark can be used for designing techniques akin to regularization for stable off-policy deep RL algorithms with reasonable performance throughout learning (Garcıa and Fernández, 2015), a requirement for many practical RL applications.
Finally, investigating Rainbow components, e.g., prioritized replay (Schaul et al., 2016), n -step updates (Sutton, 1988), double Q -learning (Van Hasselt et al., 2016), etc ., in the ofﬂine RL setting and incorporating them with REM are interesting directions.
Experience replay-based algorithms can be more sample efﬁcient than model-based approaches (Van Hasselt et al., 2019), and using the DQN replay dataset on Atari 2600 games for designing non-parametric replay models (Pan et al., 2018) and parametric world models (Kaiser et al., 2019) is quite promising for improving sample-efﬁciency.
We also leave further investigation of the exploitation ability of distributional RL to future work.
Our work is motivated by whether the complexity of recent off-policy approaches ( e.g., Rainbow (Hessel et al., 2018)) is necessary and is similar in spirit to work by Rajeswaran et al.
(2017), which attempts to demystify the complexity of on-policy deep RL for continuous control.
Our work is mainly related to two subareas of RL known as batch RL and distributional RL.
For the sake of clarity, we refer to batch RL as ofﬂine RL in other sections of the paper.
Batch Reinforcement Learning .
While there has been increasing interest in batch RL (Lange et al., 2012) over the last few years (Jiang and Li, 2016; Farajtabar et al., 2018; Irpan et al., 2019), much of this focussed on off-policy policy evaluation, where the goal is to estimate the performance of a given ﬁxed policy.
Similar to (Ernst et al., 2005; Kalyanakrishnan and Stone, 2007; Jaques et al., 2019), we investigate batch off-policy learning, which requires learning a good policy given a ﬁxed dataset.
In our ofﬂine setup, we only assume access to samples from the behavior policy and focus on Q -learning methods without any importance sampling correction as opposed to (Swaminathan and Joachims, 2015; Liu et al., 2019).
Recent work (Fujimoto et al., 2019; Kumar et al., 2019) documents that standard off-policy methods with ﬁxed datasets fail on continuous control MuJoCo tasks.
Opposed to
Kumar et al.
(2019), we focus on the ofﬂine setting with data collected from a diverse mixture of policies rather than a single Markovian behavior policy.
Furthermore, in contrast to Fujimoto et al.
(2019), our batch learning results on MuJoCo tasks and Atari 2600 games (Section 3) demonstrate that standard off-policy deep RL algorithms ( e.g., TD3 (Fujimoto et al., 2018), QR-DQN (Dabney et al., 2018b)) are quite effective when learning truly off-policy from large and diverse datasets.
Zhang and
Sutton (2017) show that large replay buffers can hurt the performance of simple Q -learning methods with weak function approximators; they attribute this to the “off-policyness” of a large buffer which might delay important transitions for learning.
However, our results reveal that even with uniform sampling, effective deep Q -learning algorithms ( e.g., REM) can exploit completely off-policy logged
DQN data on Atari 2600 games, given sufﬁcient number of gradient updates.
Distributional Reinforcement Learning .
Recently, distributional RL algorithms ( e.g., C51 (Bellemare et al., 2017), QR-DQN (Dabney et al., 2018b), IQN (Dabney et al., 2018a)) gained popularity due to their strong performance on Atari 2600 games.
We observe that distributional algorithms ( e.g., QRDQN, C51) are quite effective at exploitation and develop much simpler REM (Section 4.2), which is as successful as QR-DQN.
In contrast to distributional RL, much of the theoretical results of
Q -learning naturally extends to REM due to its similarity to Q -learning, e.g., in the tabular setting,
REM converges to Q ∗ , while popular distributional methods ( e.g., QR-DQN, C51) need not learn true return statistics corresponding to the underlying MDP (Rowland et al., 2019).
C ONCLUSIONS
This work investigates off-policy deep RL using an ofﬂine RL benchmark on Atari 2600 games based on logged experiences of a DQN agent.
We demonstrate that it is possible to learn policies with high returns that signiﬁcantly outperform the policies used to collect replay data.
We develop REM, a simple variant of ensemble Q -learning, that can effectively exploit large-scale off-policy data and shares much of the success of the more complicated distributional RL algorithms on Atari 2600 games.
The proposed benchmark can serve as a testbed for designing simple, stable and sample-efﬁcient RL algorithms and enable the research community to evaluate off-policy methods on a common ground.
As mentioned by Sutton and Barto (2018, Chapter 11.10): “ The potential for off-policy learning remains tantalizing, the best way to achieve it still a mystery. ”
A. O FFLINE CONTINUOUS CONTROL EXPERIMENTS
We replicated the ﬁnal buffer setup as described by Fujimoto et al.
(2019): We train a DDPG (Lillicrap et al., 2015) agent for 1 million time steps three standard MuJoCo continuous control environments in OpenAI gym
(Todorov et al., 2012; Brockman et al., 2016), adding N (0 , 0 . 5) Gaussian noise to actions for high exploration, and store all experienced transitions.
This collection procedure creates a dataset with a diverse set of states and actions, with the aim of sufﬁcient coverage.
Similar to
Fujimoto et al.
(2019), we train DDPG across 15 seeds, and select the 5 top performing seeds for dataset collection.
[Figure]
Using this logged dataset, we train standard continuous control off-policy actor-critic methods namely DDPG and TD3 (Fujimoto et al., 2018) completely ofﬂine without any exploration.
We also train a Batch-Constrained deep Q -learning (BCQ) agent, proposed by Fujimoto et al.
(2019), which restricts the action space to force the ofﬂine agent towards behaving close to on-policy w.r.t.
a subset of the given data.
We use the open source code generously provided by the authors at https://github.com/sfujim/BCQ and https://github.com/sfujim/TD3 .
We use the hyperparameters mentioned in (Fujimoto et al., 2018; 2019) except ofﬂine TD3 which uses a learning rate of 0.0005 for both the actor and critic.
Figure A.1 show that ofﬂine TD3 signiﬁcantly outperforms the behavior policy which collected the ofﬂine data as well as the ofﬂine DDPG agent.
Noticeably, ofﬂine TD3 also performs comparably to
BCQ, an algorithm designed speciﬁcally to learn from arbitrary, ﬁxed ofﬂine data.
While Fujimoto et al.
(2019) attribute the failure to learn in the ofﬂine setting to extrapolation error ( i.e., the mismatch between the ofﬂine dataset and true state-action visitation of the current policy), our results suggest that failure to learn from diverse ofﬂine data may be linked to extrapolation error for only weak exploitation agents such as DDPG.
A. S CORE N ORMALIZATION
The improvement in normalized performance of an ofﬂine agent, expressed as a percentage, over an online DQN (Nature)
(Mnih et al., 2015) agent is calculated as: 100 × (Score Normalized − 1) where:
[Equation]
Here, the scores are the mean evaluation scores averaged over 5 runs.
We chose not to measure performance in terms of percentage of online DQN scores alone because a tiny difference relative to the random agent on some games can translate into hundreds of percent in DQN score difference.
Additionally, the max is needed since DQN performs worse than a random agent on the games Solaris and Skiing.
A. H YPERPARAMETERS & E XPERIMENT D ETAILS
In our experiments, we used the hyperparameters provided in Dopamine baselines (Castro et al., 2018) and report them for completeness and ease of reproducibility in Table 2.
As mentioned by
Dopamine’s GitHub repository, changing these parameters can signiﬁcantly affect performance, without necessarily being indicative of an algorithmic difference.
We will also open source our code to further aid in reproducing our results.
The Atari environments (Bellemare et al., 2013) used in our experiments are stochastic due to sticky actions (Machado et al., 2018), i.e., there is 25% chance at every time step that the environment will execute the agent’s previous action again, instead of the agent’s new action.
All agents (online or ofﬂine) are compared using the best evaluation score (averaged over 5 runs) achieved during training where the evaluation is done online every training iteration using a (cid:15) -greedy policy with (cid:15)
= 0 .
We report ofﬂine training results with same hyperparameters over 5 random seeds of the DQN replay data collection, game simulator and network initialization.
[Table]
DQN replay dataset collection .
For collecting the ofﬂine data used in our experiments, we use online DQN (Nature) (Mnih et al., 2015) with the RMSprop (Tieleman and Hinton, 2012) optimizer
[Table]
The DQN replay dataset, B DQN , consists of approximately 50 million experience tuples for each run per game corresponds to 200 million frames due to frame skipping of four,
i
[Table]
e., repeating a selected action for four consecutive frames.
Note that the total dataset size is approximately 15 billion tuples ( 50 million tuples agent ∗ 5 agentsgame ∗ 60 games ).
Optimizer related hyperparameters .
For existing off-policy agents, step size and optimizer were taken as published.
Ofﬂine DQN (Adam) and all the ofﬂine agents with multi-head Q -network (Figure 3) use the Adam optimizer (Kingma and Ba, 2015) with same hyperparameters as online QRDQN (Dabney et al., 2018b)
( lr = 0 .
00005 , (cid:15) Adam = 0 .
Note that scaling the loss has the same effect as inversely scaling (cid:15) Adam when using Adam.
Online Agents .
For online REM shown in Figure 1b, we performed hyper-parameter tuning over (cid:15)
Adam in ( 0 . 01 / 32 , 0 .
001 / 32 ) over 5 training games (Asterix, Breakout, Pong, Q*Bert,
Seaquest) and evaluated on the full set of 60 Atari 2600 games using the best setting ( lr = 0 .
00005 , (cid:15)
Adam = 0 .
Online REM uses 4 Q -value estimates calculated using separate Q -networks where each network has the same architecture as originally used by online DQN (Nature).
Similar to
REM, our version of Bootstrapped-DQN also uses 4 separate Q -networks and Adam optimizer with identical hyperaparmeters ( lr = 0 .
00005 , (cid:15)
Adam = 0 .
Wall-clock time for ofﬂine experiments.
The ofﬂine experiments are approximately 3X faster than the online experiments for the same number of gradient steps on a P100 GPU.
In Figure 2 and 4(a), the ofﬂine agents are trained for 5X gradient steps, thus, the experiments are 1.67X slower than running online DQN for 200 million frames (standard protocol).
Furthermore, since the ofﬂine experiments do not require any data generation, using tricks from supervised learning such as using much larger batch sizes than 32 with TPUs / multiple GPUS would lead to a signiﬁcant speed up.
[Table]
S T
D E M O N A TT
AC K
W I Z
AR D O F
W O R
B O W L I N G
Game − 10 3 − 10 2 − 10− 10 0 0 1010 1 10 2 10% I m p
r
o v e m e n t ( L o g s c a l e )
DQN
Random
OfﬂineEnsemble-DQN
S O L AR I S
E L EVA T O R
A
C T
I O N B ER
Z
ER K K A N G AR OO
A TL A N T I S
V
I D E O P
I N
BA LL A M I D AR
C E N T I P E D E
D O U B L E D
UN K
J A M E S B O N
D
S T AR G UNN
ER K R U LL V E N T U RE C AR N
I
VA
L A SS A U LT F REE W A Y
B EA M R I D
ER P OO Y
A N A L I E N R O B O T
A N K
K UN G - F U M
A S T ER P O N G
T U T A N K H
A M H .
E .
R .
O T E
NN
I S P I TF A LL !
R I VER R
A I D Q * B ER T
M S .
P AC - M A
N
B A N K H E I S T
Y AR S ’ S R EVE
N G E
J O U R N E
Y
E
S CA P E
F I S H
I N G D
ERB Y B O X
I N G
C RA Z Y C L I M
BER
R O A D R UNN
ER
B O W L
I N G G O P H
ER
M O N T E Z U M A ’ S
R EVE N G E A
I R R
A
I D S K II N G E N D
U
R O G RAV I T AR
N A M E T H
I S
G A M E
I CE H O C K E Y
S P ACE I N VA
D ER S
B A TTL E Z O N
E
U P A N D D O W
N
T I M E P
I
L O T Z
A XX O N P H O E N
I
X
A S T
ER
I X F R O S T B I T E
P R
I VA T E E Y
E
B REA KO U T
C H O PP ER
C O MM
A N D S EA Q
U E S T A S T
ER O I D S
W I Z
AR D O F
W O R
D E M O N A TT
AC K
Game − 10 3 − 10 2 − 10− 10 0 0 1010 1 10 2 10% I m p
r
o v e m e n t ( L o g s c a l e )
DQN
Random
OfﬂineREM
[Figure]
M O N T E Z U M A ’ S
R EVE N G E S
O L AR I S
D O U B L E D
UN K
K A N G AR OO
A M I D AR H .
E .
R .
O
J A M E S B O N
D S K II N G
C E N T
I P
E D
E A TL
A N T
I S K R U LL
C RA Z Y C L I M
BER
C AR N I
VA
L P O N G T E NN
I S P I TF A LL !
Y AR S ’ S R EVE
N G E
J O U R N E
Y
E
S CA P E
M S .
P AC - M A
N
F I S H
I N G D
ERB Y
B EA M R I D
ER R O B O T A N K
S T
G UNN
ER Q * B ER T A L I E N B
O X
I N G
R O A D R UNN
ER
B A N K H E I S T R
I VER R
A I D F REE W A Y
K UN G - F U M
A S T ER
V
I D E O P
I N
BA LL P H O E N
I X
T U T A N K H
A M B A TTL E Z O N E
I CE H O C K E
Y B O W L
I N G P OO
Y
A N F R O S T B
I T E
Z
A XX O N A
I R R
A
I D
W I Z
AR D O F
W O R A S T
ER
I X V E N T
U RE
U P A N D D O W
N
N A M E T H
I S
G A M E E N D
U R O B REA KO U T S EA Q
U E S T
P R
I VA T E E Y
E
A SS A U LT G O P H ER G RAV I T
AR T
I M E P I
L O T
C H O PP ER
C O MM
A N D
S P ACE I N VA
D ER S
A S T
ER O I D S
D E M O N A TT
AC K
B ER Z ER K
E L EVA T O R
A
C T
I O N
Game
[Table] [Table]
[Table]
K A N G AR OO
V
I D E O P
I N
BA LL A M I D AR
D O U B L E D
UN K
J A M E S B O N
D
A TL A N T
I S C AR
N
I
VA
L K R U LL F REE W A Y
T U T A N K H
A M T E NN I S P
I TF A LL !
P O N G
B A N K H E I S T
J O U R N E
Y
E
S CA P E H .
E .
R .
O
S T AR G UNN
ER
A SS A U LT
M S .
P AC - M A
N
Q * B ER T
Y AR S ’ S R EVE
N G E
R O B O T A N K
F I S H
I N G D
ERB Y
K UN G - F U M
A S T ER
C RA Z Y C L I M
BER B O X
I N G
R O A D R UNN
ER
B EA M R I D ER C E N T
I P
E D E
A L
I E N B O W L
I N G R
I
VER
R
A I D
M O N T E Z U M A ’ S
R EVE N G E P
OO
Y
A N
B A TTL E Z O N
E
A
I R R A
I D E N D
U
R O
N A M E T H
I S
G A M E
U P A N D D O W
N
I CE H O C K E Y P
H O E
N I X
V E N T
U RE
A S T
ER
I X F R O S T B
I T E
Z
A XX O N S
O L AR I S B REA KO U T S
EA Q U E S T
P R
I VA T E E Y
E
G RAV I T AR
A S T
ER O I D S S K II N G
C H O PP ER
C O MM
A N D
S P ACE I N VA
D ER S G O P H ER
W I Z
AR D O F
W O R
D E M O N A TT
AC K
B ER Z ER K T I M E P
I
L O T
E L EVA T O R
A
C T
I O N
[Figure]
[Figure] [Figure]
[Table]
[Figure] [Figure] [Figure]
[Figure] [Figure]
r
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
v e r a g e S c
o
r e
[Figure]
S c
o r e
[Figure]
r a g e S
c
o r e
[Figure]
[Figure]
r a g e S
c
o r e
[Figure]
c
[Figure]
r
[Figure]
r
[Figure]