T HE R EACTOR : A FAST AND SAMPLE - EFFICIENT A CTOR -C RITIC AGENT FOR R EINFORCEMENT L EARNING
Audr¯unas Gruslys,
DeepMind audrunas@google.com Will Dabney, DeepMind wdabney@google.com Mohammad Gheshlaghi Azar,
DeepMind mazar@google.com
Bilal Piot,
DeepMind piot@google.com
Marc G. Bellemare, Google Brain bellemare@google.com Rémi Munos, DeepMind munos@google.com
A BSTRACT
In this work, we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efﬁciency than Prioritized Dueling DQN (Wang et al., 2017) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than
A3C (Mnih et al., 2016).
Our ﬁrst contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting.
The same approach can be used to convert several classes of multi-step policy evaluation algorithms, designed for expected value evaluation, into distributional algorithms.
Next, we introduce the β -leave-one-out policy gradient algorithm, which improves the trade-off between variance and bias by using action values as a baseline.
Our ﬁnal algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efﬁcient replay prioritization.
Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both sample efﬁciency and ﬁnal agent performance.
Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.
I NTRODUCTION
Model-free deep reinforcement learning has achieved several remarkable successes in domains ranging from super-human-level control in video games (Mnih et al., 2015) and the game of Go (Silver et al., 2016; 2017), to continuous motor control tasks (Lillicrap et al., 2015; Schulman et al., 2015).
Much of the recent work can be divided into two categories.
First, those of which that, often building on the DQN framework, act (cid:15) -greedily according to an action-value function and train using minibatches of transitions sampled from an experience replay buffer (Van Hasselt et al., 2016; Wang et al., 2015; He et al., 2017; Anschel et al., 2017).
These value-function agents beneﬁt from improved sample complexity, but tend to suffer from long runtimes (e.g. DQN requires approximately a week to train on Atari).
The second category are the actor-critic agents , which includes the asynchronous advantage actor-critic (A3C) algorithm, introduced by Mnih et al.
These agents train on transitions collected by multiple actors running, and often training, in parallel (Schulman et al., 2017;
Vezhnevets et al., 2017).
The deep actor-critic agents train on each trajectory only once, and thus tend to have worse sample complexity.
However, their distributed nature allows signiﬁcantly faster training in terms of wall-clock time.
Still, not all existing algorithms can be put in the above two categories and various hybrid approaches do exist (Zhao et al., 2016; O’Donoghue et al., 2017;
Gu et al., 2017; Wang et al., 2017).
Data-efﬁciency and off-policy learning are essential for many real-world domains where interactions with the environment are expensive.
Similarly, wall-clock time (time-efﬁciency) directly impacts an algorithm’s applicability through resource costs.
The focus of this work is to produce an agent that is sample- and time-efﬁcient.
To this end, we introduce a new reinforcement learning agent, called
Reactor (Retrace-Actor), which takes a principled approach to combining the sample-efﬁciency of off-policy experience replay with the time-efﬁciency of asynchronous algorithms.
We combine recent advances in both categories of agents with novel contributions to produce an agent that inherits the beneﬁts of both and reaches state-of-the-art performance over 57 Atari 2600 games.
Our primary contributions are (1) a novel policy gradient algorithm, β -LOO, which makes better use of action-value estimates to improve the policy gradient; (2) the ﬁrst multi-step off-policy distributional reinforcement learning algorithm, distributional Retrace( λ ); (3) a novel prioritized replay for off-policy sequences of transitions; and (4) an optimized network and parallel training architecture.
We begin by reviewing background material, including relevant improvements to both value-function agents and actor-critic agents.
In Section 3 we introduce each of our primary contributions and present the Reactor agent.
Finally, in Section 4, we present experimental results on the 57 Atarigames from the Arcade Learning Environment (ALE) (Bellemare et al., 2013), as well as a series of ablation studies for the various components of Reactor.
B ACKGROUND
We consider a Markov decision process (MDP) with state space X and ﬁnite action space A .
A (stochastic) policy π ( ·| x ) is a mapping from states x ∈ X to a probability distribution over actions.
We consider a γ -discounted inﬁnite-horizon criterion, with γ ∈ [0 , 1) the discount factor, and deﬁne for policy π the action-value of a state-action pair ( x, a ) as
[Equation]
where ( { x t } t ≥ 0 ) is a trajectory generated by choosing a in x and following π thereafter, i.e., a t ∼ π ( ·| x t ) (for t ≥ 1 ), and r t is the reward signal.
The objective in reinforcement learning is to ﬁnd an optimal policy π ∗ , which maximises Q π ( x, a ) .
The optimal action-values are given by
Q ∗ ( x, a ) = max π Q π ( x, a ) .
2. V ALUE - BASED ALGORITHMS
The Deep Q-Network (DQN) framework, introduced by Mnih et al. (2015), popularised the current line of research into deep reinforcement learning by reaching human-level, and beyond, performance across 57 Atari 2600 games in the ALE.
While DQN includes many speciﬁc components, the essence of the framework, much of which is shared by Neural Fitted Q-Learning (Riedmiller, 2005), is to use of a deep convolutional neural network to approximate an action-value function, training this approximate action-value function using the Q-Learning algorithm (Watkins & Dayan, 1992) and mini-batches of one-step transitions ( x t , a t , r t , x t +1 , γ t ) drawn randomly from an experience replay buffer (Lin, 1992).
Additionally, the next-state action-values are taken from a target network , which is updated to match the current network periodically.
Thus, the temporal difference (TD) error for transition t used by these algorithms is given by
[Equation]
(1) where θ denotes the parameters of the network and ¯ θ are the parameters of the target network.
Since this seminal work, we have seen numerous extensions and improvements that all share the same underlying framework.
Double DQN (Van Hasselt et al., 2016), attempts to correct for the over-estimation bias inherent in Q-Learning by changing the second term of (1) to
Q ( x t +1 , arg max a (cid:48) ∈A Q ( x t +1 , a (cid:48) ; θ ); ¯ θ ) .
The dueling architecture (Wang et al., 2015), changes the network to estimate action-values using separate network heads V ( x ; θ ) and A ( x, a ; θ ) with
[Equation]
Recently, Hessel et al.
(2017) introduced Rainbow, a value-based reinforcement learning agent combining many of these improvements into a single agent and demonstrating that they are largely complementary.
Rainbow signiﬁcantly out performs previous methods, but also inherits the poorer time-efﬁciency of the DQN framework.
We include a detailed comparison between Reactor and
Rainbow in the Appendix.
In the remainder of the section we will describe in more depth other recent improvements to DQN.
2.1. P RIORITIZED EXPERIENCE REPLAY
The experience replay buffer was ﬁrst introduced by Lin (1992) and later used in DQN (Mnih et al., 2015).
Typically, the replay buffer is essentially a ﬁrst-in-ﬁrst-out queue with new transitions gradually replacing older transitions.
The agent would then sample a mini-batch uniformly at random from the replay buffer.
Drawing inspiration from prioritized sweeping (Moore & Atkeson, 1993), prioritized experience replay replaces the uniform sampling with prioritized sampling proportional to the absolute TD error (Schaul et al., 2016).
Speciﬁcally, for a replay buffer of size N , prioritized experience replay samples transition t with probability P ( t ) , and applies weighted importance-sampling with w t to correct for the prioritization bias, where
[Equation]
Prioritized DQN signiﬁcantly increases both the sample-efﬁciency and ﬁnal performance over DQN on the Atari 2600 benchmarks (Schaul et al., 2015).
2.1. R ETRACE ( λ )
Retrace( λ ) is a convergent off-policy multi-step algorithm extending the DQN agent (Munos et al., 2016).
Assume that some trajectory { x 0 , a 0 , r 0 , x 1 , a 1 , r 1 , . . .
, x t , a t , r t , . . .
, } has been generated according to behaviour policy µ , i.e., a t ∼ µ ( ·| x t ) .
Now, we aim to evaluate the value of a different target policy π , i.e. we want to estimate Q π .
The Retrace algorithm will update our current estimate
Q of Q π in the direction of
[Equation]
where δ πs Q def = r s + γ E π [ Q ( x s +1 , · )]
− Q ( x s , a s ) is the temporal difference at time s under π , and
[Equation]
The Retrace algorithm comes with the theoretical guarantee that in ﬁnite state and action spaces, repeatedly updating our current estimate Q according to (3) produces a sequence of Q functions which converges to Q π for a ﬁxed π or to Q ∗ if we consider a sequence of policies π which become increasingly greedy w.r.t.
the Q estimates (Munos et al., 2016).
2.1. D ISTRIBUTIONAL RL
Distributional reinforcement learning refers to a class of algorithms that directly estimate the distribution over returns, whose expectation gives the traditional value function (Bellemare et al., 2017).
Such approaches can be made tractable with a distributional Bellman equation, and the recently proposed algorithm C 51 showed state-of-the-art performance in the Atari 2600 benchmarks.
Cparameterizes the distribution over returns with a mixture over Diracs centered on a uniform grid,
Q ( x, a ; θ ) =
[Equation]
with hyperparameters v min , v max that bound the distribution support of size N .
2. A CTOR - CRITIC ALGORITHMS
In this section we review the actor-critic framework for reinforcement learning algorithms and then discuss recent advances in actor-critic algorithms along with their various trade-offs.
The asynchronous advantage actor-critic (A3C) algorithm (Mnih et al., 2016), maintains a parameterized policy π ( a | x ; θ ) and value function V ( x ; θ v ) , which are updated with
[Equation]
where,
[Equation]
A3C uses M = 16 parallel CPU workers, each acting independently in the environment and applying the above updates asynchronously to a shared set of parameters.
In contrast to the previously discussed value-based methods, A3C is an on-policy algorithm, and does not use a GPU nor a replay buffer.
Proximal Policy Optimization (PPO) is a closely related actor-critic algorithm (Schulman et al., 2017), which replaces the advantage (7) with,
[Equation]
where ρ t is as deﬁned in Section 2.1.2.
Although both PPO and A3C run M parallel workers collecting trajectories independently in the environment, PPO collects these experiences to perform a single, synchronous, update in contrast with the asynchronous updates of A3C.
Actor-Critic Experience Replay (ACER) extends the A3C framework with an experience replay buffer,
Retrace algorithm for off-policy corrections, and the Truncated Importance Sampling Likelihood
Ratio (TISLR) algorithm used for off-policy policy optimization (Wang et al., 2017).
T HE R EACTOR
The Reactor is a combination of four novel contributions on top of recent improvements to both deep value-based RL and policy-gradient algorithms.
Each contribution moves Reactor towards our goal of achieving both sample and time efﬁciency.
3. β -LOO
The Reactor architecture represents both a policy π ( a | x ) and action-value function Q ( x, a ) .
We use a policy gradient algorithm to train the actor π which makes use of our current estimate Q ( x, a ) of
Q π ( x, a ) .
Let V π ( x 0 ) be the value function at some initial state x 0 , the policy gradient theorem says that ∇ V π ( x 0 ) =
E (cid:2) (cid:80) t γ t (cid:80) a Q π ( x t , a ) ∇ π ( a | x t ) (cid:3) , where ∇ refers to the gradient w.r.t.
policy parameters (Sutton et al., 2000).
We now consider several possible ways to estimate this gradient.
To simplify notation, we drop the dependence on the state x for now and consider the problem of estimating the quantity
[Equation]
In the off-policy case, we consider estimating G using a single action ˆ a drawn from a (possibly different from π ) behaviour distribution ˆ a ∼ µ .
Let us assume that for the chosen action ˆ a we have access to an unbiased estimate R (ˆ a ) of Q π (ˆ a ) .
Then, we can use likelihood ratio (LR) method combined with an importance sampling (IS) ratio (which we call ISLR) to build an unbiased estimate of G :
[Equation]
where V is a baseline that depends on the state but not on the chosen action.
However this estimate suffers from high variance.
A possible way for reducing variance is to estimate G directly from (8) by using the return R (ˆ a ) for the chosen action ˆ a and our current estimate Q of Q π for the other actions, which leads to the so-called leave-one-out (LOO) policy-gradient estimate:
[Equation]
[Equation] [Figure]
This estimate has low variance but may be biased if the estimated Q values differ from Q π .
A better bias-variance tradeoff may be obtained by the more general β -LOO policy-gradient estimate:
[Equation]
where β
= β ( µ, π, ˆ a ) can be a function of both policies, π and µ , and the selected action ˆ a .
Notice that when β
= 1 , (10) reduces to (9), and when β
= 1 /µ (ˆ a ) , then (10) is
[Equation]
This estimate is unbiased and can be seen as a generalization of ˆ G ISLR where instead of using a state-only dependent baseline, we use a state-and-action-dependent baseline (our current estimate Q ) and add the correction term (cid:80) a ∇ π ( a ) Q ( a ) to cancel the bias.
Proposition 1 gives our analysis of the bias of G β -LOO , with a proof left to the Appendix.
Proposition 1.
Assume ˆ a ∼ µ and that E
[ R (ˆ a )]
= Q π (ˆ a ) .
Then, the bias of G β -LOO is (cid:12)(cid:12) (cid:80) a (1 − µ ( a ) β ( a )) ∇ π ( a )
[ Q ( a ) − Q π ( a )] (cid:12)(cid:12) .
Thus the bias is small when β ( a ) is close
to 1 /µ ( a ) , or when the Q -estimates are close to the true
Q π values, and unbiased regardless of the estimates if β ( a ) = 1 /µ ( a ) .
The variance is low when β is small, therefore, in order to improve the bias-variance tradeoff we recommend using the β -LOO estimate with β deﬁned as: β (ˆ a )
= min (cid:0) c, 1 µ (ˆ a ) (cid:1) , for some constant c ≥ 1 .
This truncated 1 /µ coefﬁcient shares similarities with the truncated IS gradient estimate introduced in (Wang et al., 2017) (which we call TISLR for truncated-ISLR):
[Equation]
The differences are: (i) we truncate 1 /µ
(ˆ a )
=
π (ˆ a ) /µ
(ˆ a ) × 1 /π
(ˆ a ) instead of truncating π
(ˆ a ) /µ
(ˆ a ) , which provides an additional variance reduction due to the variance of the LR ∇ log π
(ˆ a ) =
∇ π
(ˆ a ) π (ˆ a ) (since this LR may be large when a low probability action is chosen), and (ii) we use our Q -baseline instead of a V baseline, reducing further the variance of the LR estimate.
3. D ISTRIBUTIONAL R ETRACE
In off-policy learning it is very difﬁcult to produce an unbiased sample R (ˆ a ) of Q π (ˆ a ) when following another policy µ .
This would require using full importance sampling correction along the trajectory.
Instead, we use the off-policy corrected return computed by the Retrace algorithm, which produces a (biased) estimate of Q π (ˆ a ) but whose bias vanishes asymptotically (Munos et al., 2016).
In Reactor, we consider predicting an approximation of the return distribution function from any state-action pair ( x, a ) in a similar way as in Bellemare et al.
The original algorithm C described in that paper considered single-step Bellman updates only.
Here we need to extend this idea to multi-step updates and handle the off-policy correction performed by the Retrace algorithm, as deﬁned in (3).
Next, we describe these two extensions.
Multi-step distributional Bellman operator:
First, we extend C51 to multi-step Bellman backups.
We consider return-distributions from ( x, a ) of the form (cid:80)
i q i ( x, a )
δ z i (where δ z denotes a Dirac in z ) which are supported on a ﬁnite uniform grid { z i } ∈ [ v min , v max ] , z i <
z i +1 , z 1 = v min , z m = v max .
The coefﬁcients q i ( x, a ) (discrete distribution) corresponds to the probabilities assigned to each atom
z i of the grid.
From an observed n -step sequence { x t , a t , r t , x t +1 , . . .
, x t + n } , generated by behavior policy µ (i.e, a s ∼ µ ( ·| x s ) for t ≤ s < t + n ), we build the n -step backed-up return-distribution from ( x t , a t ) .
The n -step distributional Bellman target, whose expectation is (cid:80) t + n [Equation] Q [Equation]
, a ) , is given by:
[Equation]
Since this distribution is supported on the set of atoms { z ni } , which is not necessarily aligned with the grid { z i } , we do a projection step and minimize the KL-loss between the projected target and the current estimate, just as with C51 except with a different target distribution (Bellemare et al., 2017).
Distributional Retrace: Now, the Retrace algorithm deﬁned in (3) involves an off-policy correction which is not handled by the previous n -step distributional Bellman backup.
The key to extending this distributional back-up to off-policy learning is to rewrite the Retrace algorithm as a linear combination of n -step Bellman backups, weighted by some coefﬁcients α n,a .
Indeed, notice that (3) rewrites as
[Equation]
where α n,a = (cid:0) c t +1 . . .
c t + n − 1 (cid:1)(cid:0) π ( a | x t + n ) − I { a = a t + n } c t + n (cid:1) .
These coefﬁcients depend on the degree of off-policy-ness (between µ and π ) along the trajectory.
We have that (cid:80) n ≥ 1 (cid:80) a α n,a = (cid:80) n ≥ 1 (cid:0) c t
+1 . . .
c t + n − 1 (cid:1) (1 − c t + n ) = 1 , but notice some coefﬁcients may be negative.
However, in expectation (over the behavior policy) they are non-negative.
Indeed,
[Equation]
by deﬁnition of the c s coefﬁcients (4).
Thus in expectation (over the behavior policy), the Retrace update can be seen as a convex combination of n -step Bellman updates.
Then, the distributional Retrace algorithm can be deﬁned as backing up a mixture of n -step distributions.
More precisely, we deﬁne the Retrace target distribution as:
[Equation]
where h z i ( x ) is a linear interpolation kernel, projecting onto the support { z i } : h z i ( x )
[Equation]
if z i
[Equation]
1 ≤ x ≤ z [Equation] if z i ≤ x ≤ z i +
[Equation]
if x ≤ z i − 1 or x ≥ z i +
[Equation]
if ( x ≤ v min and z i = v min ) or ( x ≥ v max and z i = v max )
 
We update the current probabilities q ( x t , a t ) by performing a gradient step on the KL-loss [Equation]
[Equation]
Again, notice that some target “probabilities” q ∗ i ( x t , a t ) may be negative for some sample trajectory, but in expectation they will be non-negative.
Since the gradient of a KL-loss is linear w.r.t.
its ﬁrst argument, our update rule (12) provides an unbiased estimate of the gradient of the KL between the expected (over the behavior policy) Retrace target distribution and the current predicted distribution.
Remark: The same method can be applied to other algorithms (such as TB( λ ) (Precup et al., 2000) and importance sampling (Precup et al., 2001))
in order to derive distributional versions of other off-policy multi-step RL algorithms.
3. P RIORITIZED SEQUENCE REPLAY
Prioritized experience replay has been shown to boost both statistical efﬁciency and ﬁnal performance of deep RL agents (Schaul et al., 2016).
However, as originally deﬁned prioritized replay does not handle sequences of transitions and weights all unsampled transitions identically.
In this section we present an alternative initialization strategy, called lazy initialization , and argue that it better encodes prior information about temporal difference errors.
We then brieﬂy describe our computationally efﬁcient prioritized sequence sampling algorithm, with full details left to the appendix.
It is widely recognized that TD errors tend to be temporally correlated, indeed the need to break this temporal correlation has been one of the primary justiﬁcations for the use of experience replay (Mnih et al., 2015).
Our proposed algorithm begins with this fundamental assumption.
Assumption 1.
Temporal differences are temporally correlated, with correlation decaying on average with the time-difference between two transitions.
Prioritized experience replay adds new transitions to the replay buffer with a constant priority, but given the above assumption we can devise a better method.
Speciﬁcally, we propose to add experience to the buffer with no priority , inserting a priority only after the transition has been sampled and used for training.
Also, instead of sampling transitions, we assign priorities to all (overlapping) sequences of length n .
When sampling, sequences with an assigned priority are sampled proportionally to that priority.
Sequences with no assigned priority are sampled proportionally to the average priority of assigned priority sequences within some local neighbourhood.
Averages are weighted to compensate for sampling biases (i.e. more samples are made in areas of high estimated priorities, and in the absence of weighting this would lead to overestimation of unassigned priorities).
The lazy initialization scheme starts with priorities p t corresponding to the sequences { x t , a t , . . .
, x t + n } for which a priority was already assigned.
Then it extrapolates a priority of all other sequences in the following way.
Let us deﬁne a partition ( I i ) i of the states ordered by increasing time such that each cell
I
i contains exactly one state s i with already assigned priority.
We deﬁne the estimated priority ˆ p t to all other sequences as ˆ p t = (cid:80) s
i ∈ J ( t ) w i [Equation] i [Equation]
J [Equation] t ) w i (cid:48) p ( s i ) , where J ( t ) is a collection of contiguous cells ( I i ) containing time t , and w i =
| I i | is the length of the cell
I i containing s i .
For already deﬁned priorities denote ˆ p t = p t .
Cell sizes work as estimates of inverse local density and are used as importance weights for priority estimation.
For the algorithm to be unbiased, partition ( I i )
i must not be a function of the assigned priorities.
So far we have deﬁned a class of algorithms all free to choose the partition ( I i ) and the collection of cells I ( t ) , as long that they satisfy the above constraints.
Figure 4 in the Appendix illustrates the above description.
Now, with probability (cid:15) we sample uniformly at random, and with probability 1 − (cid:15) we sample proportionally to ˆ p t .
We implemented an algorithm satisfying the above constraints and called it
Contextual Priority Tree (CPT).
It is based on AVL trees (Velskii & Landis, 1976) and can execute sampling, insertion, deletion and density evaluation in O (ln( n )) time.
We describe CPT in detail in the Appendix in Section 6.3.
We treated prioritization as purely a variance reduction technique.
Importance-sampling weights were evaluated as in prioritized experience replay, with ﬁxed β = 1 in (2).
We used simple gradient magnitude estimates as priorities, corresponding to a mean absolute TD error along a sequence for
Retrace, as deﬁned in (3) for the classical RL case, and total variation in the distributional Retrace case.
3. A GENT ARCHITECTURE
In order to improve CPU utilization we decoupled acting from learning.
This is an important aspect of our architecture: an acting thread receives observations, submits actions to the environment, and
Sum
[Figure]
Algorithm
Training Time
Type # Workers
DQN 8 days
GPU
Double DQN 8 days GPU Dueling 8 days GPU
Prioritized DQN 8 days
GPU
Rainbow 10 days
GPU
A3C 4 days
CPU
Reactor < 2 days
CPU 10+
Reactor 500m 4 days CPU 10+ Reactor* < 1 day CPU 20+ stores transitions in memory, while a learning thread re-samples sequences of experiences from memory and trains on them (Figure 2, left).
We typically execute 4-6 acting steps per each learning step.
We sample sequences of length n = 33 in batches of 4.
A moving network is unrolled over frames 1-32 while the target network is unrolled over frames 2-33.
We allow the agent to be distributed over multiple machines each containing action-learner pairs.
Each worker downloads the newest network parameters before each learning step and sends delta-updates at the end of it.
Both the network and target network are stored on a shared parameter server while each machine contains its own local replay memory.
Training is done by downloading a shared network, evaluating local gradients and sending them to be applied on the shared network.
While the agent can also be trained on a single machine, in this work we present results of training obtained with either 10 or 20 actor-learner workers and one parameter server.
In Figure 2 (right) we compare resources and runtimes of Reactor with related algorithms.
3.4. N ETWORK ARCHITECTURE
In some domains, such as Atari, it is useful to base decisions on a short history of past observations.
The two techniques generally used to achieve this are frame stacking and recurrent network architectures.
We chose the latter over the former for reasons of implementation simplicity and computational efﬁciency.
As the Retrace algorithm requires evaluating action-values over contiguous sequences of trajectories, using a recurrent architecture allowed each frame to be processed by the convolutional network only once, as opposed to n times times if n frame concatenations were used.
The Reactor architecture uses a recurrent neural network which takes an observation x t as input and produces two outputs: categorical action-value distributions
q i ( x t , a ) ( i here is a bin identiﬁer), and policy probabilities π ( a | x t ) .
We use an architecture inspired by the duelling network architecture (Wang et al., 2015).
We split action-value -distribution logits into state-value logits and advantage logits, which in turn are connected to the same LSTM network (Hochreiter & Schmidhuber, 1997).
Final action-value logits are produced by summing state- and action-speciﬁc logits, as in Wang et al.
Finally, a softmax layer on top for each action produces the distributions over discounted future returns.
The policy head uses a softmax layer mixed with a ﬁxed uniform distribution over actions, where this mixing ratio is a hyperparameter (Wiering, 1999, Section 5.1.3).
Policy and Q-networks have separate LSTMs.
Both LSTMs are connected to a shared linear layer which is connected to a shared convolutional neural network (Krizhevsky et al., 2012).
The precise network speciﬁcation is given in
Table 3 in the Appendix.
Gradients coming from the policy LSTM are blocked and only gradients originating from the Qnetwork LSTM are allowed to back-propagate into the convolutional neural network.
We block gradients from the policy head for increased stability, as this avoids positive feedback loops between π and q i caused by shared representations.
We used the Adam optimiser (Kingma & Ba, 2014), with a learning rate of 5 × 10 − 5 and zero momentum because asynchronous updates induce implicit momentum (Mitliagkas et al., 2016).
Further discussion of hyperparameters and their optimization can be found in Appendix 6.1.
E XPERIMENTAL R ESULTS
We trained and evaluated Reactor on 57 Atari games (Bellemare et al., 2013).
Figure 3 compares the performance of Reactor with different versions of Reactor each time leaving one of the algorithmic improvements out.
We can see that each of the algorithmic improvements (Distributional retrace, betaLOO and prioritized replay) contributed to the ﬁnal results.
While prioritization was arguably the most important component, Beta-LOO clearly outperformed TISLR algorithm.
Although distributional and non-distributional versions performed similarly in terms of median human normalized scores, distributional version of the algorithm generalized better when tested with random human starts (Table 1)
[Table]
A LGORITHM
[Table]
ORMALIZED
[Table]
[Table]
4. C OMPARING TO PRIOR WORK
We evaluated Reactor with target update frequency T update = 1000 , λ = 1 . 0 and β -LOO with β =on 57 Atari games trained on 10 machines in parallel.
We averaged scores over 200 episodes using 30 random human starts and noop starts (Tables 4 and 5 in the Appendix).
We calculated mean and median human normalised scores across all games.
We also ranked all algorithms (including random and human scores) for each game and evaluated mean rank of each algorithm across all 57 Atari games.
We also evaluated mean Rank and Elo scores for each algorithm for both human and noop start settings.
Please refer to Section 6.2 in the Appendix for more details.
Tables 1 & 2 compare versions of our algorithm, 5 with several other state-of-art algorithms across
Atari games for a ﬁxed random seed across all games (Bellemare et al., 2013).
We compare Reactor dueling and dueling and
Each algorithm was exposed to 200 million frames of experience, or 500 million frames when followed by 500 M , and the same pre-processing pipeline including 4 action repeats was used as in the original
DQN paper (Mnih et al., 2015).
In Table 1, we see that Reactor exceeds the performance of all algorithms across all metrics, despite requiring under two days of training.
With 500 million frames and four days training we see Reactor’s performance continue to improve signiﬁcantly.
The difference in time-efﬁciency is especially apparent when comparing Reactor and Rainbow (see Figure 3, right).
Additionally, unlike Rainbow, Reactor does not use Noisy Networks (Fortunato et al., 2017), which was reported to have contributed to the performance gains.
When evaluating under the no-op starts regime (Table 2), Reactor out performs all methods except for Rainbow.
This suggests that Rainbow is more sample-efﬁcient when training and evaluation regimes match exactly, but may be overﬁtting to particular trajectories due to the signiﬁcant drop in performance when evaluated on the random human starts.
Regarding ACER, another Retrace-based actor-critic architecture, both classical and distributional versions of Reactor (Figure 3) exceeded the best reported median human normalized score of 1.
with noop starts achieved in 500 million steps.
C ONCLUSION
In this work we presented a new off-policy agent based on Retrace actor-critic architecture and show that it achieves similar performance as the current state-of-the-art while giving signiﬁcant real-time performance gains.
We demonstrate the beneﬁts of each of the suggested algorithmic improvements, including Distributional Retrace, beta-LOO policy gradient and contextual priority
A PPENDIX
Proposition 1.
Assume ˆ a ∼ µ and that E
[ R (ˆ a )]
= Q π (ˆ a ) .
Then, the bias of G β -LOO is (cid:12)(cid:12) (cid:80) a (1 − µ ( a ) β ( a )) ∇ π ( a )
[ Q ( a ) − Q π ( a )] (cid:12)(cid:12) .
Proof.
The bias of ˆ G β -LOO is
[Equation]
6. H YPERPARAMETER OPTIMIZATION
As we believe that algorithms should be robust with respect to the choice of hyperparameters, we spent little effort on parameter optimization.
In total, we explored three distinct values of learning rates and two values of ADAM momentum (the default and zero) and two values of T update on a subset of 7 Atari games without prioritization using non-distributional version of Reactor.
We later used those values for all experiments.
We did not optimize for batch sizes and sequence length or any prioritization hyperparamters.
6. R ANK AND E LO EVALUATION
Commonly used mean and median human normalized scores have several disadvantages.
A mean human normalized score implicitly puts more weight on games that computers are good and humans are bad at.
Comparing algorithm by a mean human normalized score across 57 Atari games is almost equivalent to comparing algorithms on a small subset of games close to the median and thus dominating the signal.
Typically a set of ten most score-generous games, namely Assault, Asterix,
Breakout, Demon Attack, Double Dunk, Gopher, Pheonix, Stargunner, Up’n Down and Video Pinball can explain more than half of inter-algorithm variance.
A median human normalized score has the opposite disadvantage by effectively discarding very easy and very hard games from the comparison.
As typical median human normalized scores are within the range of 1-2.5, an algorithm which scores zero points on Montezuma’s Revenge is evaluated equal to the one which scores 2500 points, as both performance levels are still below human performance making incremental improvements on hard games not being reﬂected in the overall evaluation.
In order to address both problem, we also evaluated mean rank and Elo metrics for inter-algorithm comparison.
Those metrics implicitly assign the same weight to each game, and as a result is more sensitive of relative performance on very hard and easy games: swapping scores of two algorithms on any game would result in the change of both mean rank and Elo metrics.
We calculated separate mean rank and Elo scores for each algorithm using results of test evaluations with 30 random noop-starts and 30 random human starts (Tables 5 and 4).
All algorithms were ranked across each game separately, and a mean rank was evaluated across 57 Atari games.
For Elo score evaluation algorithm, A was considered to win over algorithm B if it obtained more scores on a given
Atari.
We produced an empirical win-probability matrix by summing wins across all games and used this matrix to evaluate Elo scores.
A ranking difference of 400 corresponds to the odds of winning of 10:1 under the Gaussian assumption.
6. C ONTEXTUAL PRIORITY TREE
Contextual priority tree is one possible implementation of lazy prioritization (Figure 4).
All sequence keys are put into a balanced binary search tree which maintains a temporal order.
An AVL tree (Velskii & Landis (1976)) was chosen due to the ease of implementation and because it is on average more evenly balanced than a Red-Black Tree.
Each tree node has up to two children (left and right) and contains currently stored key and a priority of the key which is either set or is unknown.
Some trees may only have a single child subtree while some may have none.
In addition to this information, we were tracking other summary statistics at each node which was re-evaluated after each tree rotation.
The summary statistics was evaluated by consuming previously evaluated summary statistics of both children and a priority of the key stored within the current node.
In particular, we were tracking a total number of nodes within each subtree and mean-priority estimates updated according to rules shown in Figure 5.
The total number of nodes within each subtree was always known ( c in Figure 5), while mean priority estimates per key ( m in
Figure 5) could either be known or unknown.
If a mean priority of either one child subtree or a key stored within the current node is unknown then it can be estimated to by exploiting information coming from another sibling subtree or a priority stored within the parent node.
Sampling was done by traversing the tree from the root node up while sampling either one of the children subtrees or the currently held key proportionally to the total estimated priority masses contained within.
The rules used to evaluate proportions are shown in orange in Figure 5.
Similarly, probabilities of arbitrary keys can be queried by traversing the tree from the root node towards the child node of an interest while maintaining a product of probabilities at each branching point.
Insertion, deletion, sampling and probability query operations can be done in O(ln(n)) time.
The suggested algorithm has the desired property that it becomes a simple proportional sampling algorithm
once all the priorities are known.
While some key priorities are unknown, they are estimated by using nearby known key priorities (Figure 6).
Each time when a new sequence key is added to the tree, it was set to have an unknown priority.
Any priority was assigned only after the key got ﬁrst sampled and the corresponding sequence got passed through the learner.
When a priority of a key is set or updated, the key node is deliberately removed from and placed back to the tree in order to become a leaf-node.
This helped to set priorities of nodes in the immediate vicinity more accurately by using the freshest information available.
6. N ETWORK ARCHITECTURE
The value of (cid:15) = 0 .
01 is the minimum probability of choosing a random action and it is hard-coded into the policy network.
Figure 7 shows the overall network topology while Table 3 speciﬁes network layer sizes.
L AYER
I NPUT
P ARAMETERS
SIZE
C ONVOLUTIONAL
KERNEL
OUTPUT
STRIDES
WIDTH
CHANNELS
C ONV[84, 84, 1] [8, 8]
C ONCAT R E LU [20, 20, 16]
C ONV[20, 20, 32] [4, 4]
C ONCAT R E LU [9, 9, 32]
C ONV[9, 9, 64] [3, 3]
C ONCAT R E LU [7, 7, 32]
F ULLY CONNECTED
O UTPUT SIZE
L INEAR [7, 7, 64]
C ONCAT R E LU [128]
R ECURRENT
O UTPUT SIZE π
LSTM [256]
L INEAR [128]
C ONCAT R E LU [32]
L INEAR [64] # ACTIONS
S OFTMAX
[# ACTIONS ] # ACTIONS
X (1- (cid:15) )
+ (cid:15) /
# ACTIONS [# ACTIONS ] # ACTIONS
R ECURRENT Q
O UTPUT SIZE
LSTM [256]
V ALUE LOGIT HEAD
O UTPUT SIZE
L INEAR [128]
C ONCAT R E LU [32]
L INEAR [64] # BINS
A DVANTAGE LOGIT HEAD # ACTIONS × # BINS
L INEAR [128]
C ONCAT R E LU [32]
6. C OMPARISONS WITH R AINBOW
In this section we compare Reactor with the recently published Rainbow agent (Hessel et al., 2017).
While ACER is the most closely related algorithmically, Rainbow is most closely related in terms of performance and thus a deeper understanding of the trade-offs between Rainbow and Reactor may beneﬁt interested readers.
There are many architectural and algorithmic differences between Rainbow and Reactor.
We will therefore begin by highlighting where they agree.
Both use a categorical action-value distribution critic (Bellemare et al., 2017), factored into state and state-action logits (Wang et al., 2015),
[Equation]
Both use prioritized replay, and ﬁnally, both perform n -step Bellman updates.
Despite these similarities, Reactor and Rainbow are fundamentally different algorithms and are based upon different lines of research.
While Rainbow uses Q-Learning and is based upon DQN (Mnih et al., 2015), Reactor is an actor-critic algorithm most closely based upon A3C (Mnih et al., 2016).
Each inherits some design choices from their predecessors, and we have not performed an extensive ablation comparing these various differences.
Instead, we will discuss four of the differences we believe are important but less obvious.
First, the network structures are substantially different.
Rainbow uses noisy linear layers and ReLU activations throughout the network, whereas Reactor uses standard linear layers and concatenated
ReLU activations throughout.
To overcome partial observability, Rainbow, inheriting this choice from DQN, uses frame stacking .
On the other hand, Reactor, inheriting its choice from A3C, uses
LSTMs after the convolutional layers of the network.
It is also difﬁcult to directly compare the number of parameters in each network because the use of noisy linear layers doubles the number of parameters, although half of these are used to control noise, while the LSTM units in Reactor require more parameters than a corresponding linear layer would.
Second, both algorithms perform n -step updates, however, the Rainbow n -step update does not use any form of off-policy correction.
Because of this, Rainbow is restricted to using only small values of n (e.g. n = 3 ) because larger values would make sequences more off-policy and hurt performance.
By comparison, Reactor uses our proposed distributional Retrace algorithm for off-policy correction of n -step updates.
This allows the use of larger values of n (e.g. n = 33 ) without loss of performance.
Third, while both agents use prioritized replay buffers (Schaul et al., 2016), they each store different information and prioritize using different algorithms.
Rainbow stores a tuple containing the state x t − 1 , action a t − 1 , sum of n discounted rewards (cid:80) n − 1 k =
r t + k (cid:81) k − 1 m =
γ t + m , product of n discount factors (cid:81) n −k =
γ t + k , and next-state n steps away x t + n − 1 .
Tuples are prioritized based upon the last observed TD error, and inserted into replay with a maximum priority.
Reactor stores length n sequences of tuples ( x t − 1 , a t − 1 , r t , γ t ) and also prioritizes based upon the observed TD error.
However, when inserted into the buffer the priority is instead inferred based upon the known priorities of neighboring sequences.
This priority inference was made efﬁcient using the previously introduced contextual priority tree, and anecdotally we have seen it improve performance over a simple maximum priority approach.
Finally, the two algorithms have different approaches to exploration.
Rainbow, unlike DQN, does not use (cid:15) -greedy exploration, but instead replaces all linear layers with noisy linear layers which induce randomness throughout the network.
This method, called Noisy Networks (Fortunato et al., 2017), creates an adaptive exploration integrated into the agent’s network.
Reactor does not use noisy networks, but instead uses the same entropy cost method used by A3C and many others (Mnih et al., 2016), which penalizes deterministic policies thus encouraging indifference between similarly valued actions.
Because Rainbow can essentially learn not to explore, it may learn to become entirely greedy in the early parts of the episode, while still exploring in states not as frequently seen.
In some sense, this is precisely what we want from an exploration technique, but it may also lead to highly deterministic trajectories in the early part of the episode and an increase in overﬁtting to those trajectories.
We hypothesize that this may be the explanation for the signiﬁcant difference in
Rainbow’s performance between evaluation under no-op and random human starts, and why Reactor does not show such a large difference.
6. A TARI RESULTS
[Table]
C HOPPER C OMMAND
[Table]
C RAZY C LIMBER
[Table]
D EFENDER
[Table]
D EMON A TTACK
[Table]
OUBLE
[Table]
UNK
[Table]
NDURO
[Table]
F ISHING D ERBY
[Table]
F REEWAY
[Table] [Table]
[Table] [Table]
[Table] [Table]
ITFALL !
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
64.2 S EAQUEST 215. 25887.9 S
[Table] -11904.4 S OLARIS 2047.
[Table] [Table]
[Table] [Table]
W IZARD OF W OR
[Table]
Z AXXON
[Table] [Table]
LIMBER
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
S
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
OLARIS
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
S URROUND -
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
T ENNIS -23.
[Table] [Table]
[Table] [Table]
[Table]
23.6 T
[Table]
P ILOT 3568.
[Table]
[Table]
[Table]
18841.5 T UTANKHAM 11.
[Table]
[Table]
U P ’ N D OWN 533. [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] V ENTURE [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
INBALL
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] W IZARD OF W OR 563. [Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
Y
[Table]
EVENGE 3092.
[Table]
[Table]
[Table]
[Table]
[Table]
148855.0 Z AXXON 32.
[Table]
[Table]
[Table]
[Table]
[Table]