Paper:
THE REACTOR: A FAST AND SAMPLE-EFFICIENT ACTOR-CRITIC AGENT FOR REINFORCEMENT LEARNING
In this work, we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN (Wang et al., 2017) and Categorical DQN (Bellemare et al., 2017), while giving better run-time performance than A3C (Mnih et al., 2016). Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms, designed for expected value evaluation, into distributional algorithms. Next, we introduce the β-leave-one-out policy gradient algorithm, which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training.

Review:
This paper proposes a novel reinforcement learning algorithm (« The Reactor ») based on the combination of several improvements to DQN: a distributional version of Retrace, a policy gradient update rule called beta-LOO aiming at variance reduction, a variant of prioritized experience replay for sequences, and a parallel training architecture. Experiments on Atari games show a significant improvement over prioritized dueling networks in particular, and competitive performance compared to Rainbow, at a fraction of the training time.
There are definitely several interesting and meaningful contributions in this submission, and I like the motivations behind them. They are not groundbreaking (essentially extending existing techniques) but are still very relevant to current RL research.
Unfortunately I also see it as a step back in terms of comparison to other algorithms. The recent Rainbow paper finally established a long overdue clear benchmark on Atari. We have seen with the « Deep Reinforcement Learning that Matters » paper how important (and difficult) it is to properly compare algorithms on deep RL problems. I assume that this submission was mostly written before Rainbow came out, and that comparisons to Rainbow were hastily added just before the ICLR deadline: this would explain why they are quite limited, but in my opinion it remains a major issue, which is the main reason why I am advocating for rejection.
More precisely, focusing on the comparison to Rainbow which is the main competitor here, my concerns are the following:
- There is almost no discussion on the differences between Reactor and Rainbow (actually the paper lacks a « related work » section). In particular Rainbow also uses a version of distributional multi-step, which as far as I can tell may not be as well motivated (from a mathematical point of view) as the one in this submission (since it does not correct for the « off-policyness » of the replay data), but still seems to work well on Atari.
- Rainbow is not distributed. This was a deliberate choice by its authors to focus on algorithmic comparisons. However, it seems to me that it could benefit from a parallel training scheme like Reactor’s. I believe a comparison between Reactor and Rainbow needs to either have them both parallelized or none of them (especially for a comparison on time efficiency like in Fig. 2)
- Rainbow uses the traditional feedforward DQN architecture while Reactor uses a recurrent network. It is not clear to which extent this has an impact on the results.
- Rainbow was stopped at 200M steps, at which point it seems to be overall superior to Reactor at 200M steps. The results as presented here emphasize the superiority of Reactor at 500M steps, but a proper comparison would require Rainbow results at 500M steps as well.
In addition, although I found most of the paper to be clear enough, some parts were confusing to me, in particular:
- « multi-step distributional Bellman operator » in 3.2: not clear exactly what the target distribution is. If I understand correctly this is the same as the Rainbow extension, but this link is not mentioned.
- 3.4.1 (network architecture): a simple diagram in the appendix would make it much easier to understand (Table 3 is still hard to read because it is not clear which layers are connected together)
- 3.3 (prioritized sequence replay): again a visual illustration of the partitioning scheme would in my opinion help clarify the approach
A few minor points to conclude:
- In eq. 6, 7 and the rest of this section, A does not depend (directly) on theta so it should probably be removed to avoid confusion. Note also that using the letter A may not be best since A is used to denote an action in 3.1.
- In 3.1: « Let us assume that for the chosen action A we have access to an estimate R(A) of Qπ(A) » => « unbiased estimate »
- In last equation of p.5 it is not clear what q_i^n is
- There is a lambda missing on p.6 in the equation showing that alphas are non-negative on average, just before the min
- In the equation above eq. 12 there is a sum over « i=1 »
- That same equation ends with some h_z_i that are not defined
- In Fig. 2 (left) for Reactor we see one worker using large batches and another one using many threads. This is confusing.
- 3.3 mentions sequences of length 32 but 3.4 says length 33.
- 3.3 says tree operations are in O(n ln(n)) but it should be O(ln(n))
- At very end of 3.3 it is not clear what « total variation » is.
- In 3.4 please specify the frequency at which the learner thread downloads shared parameters and uploads updates
- Caption of Fig. 3 talks about « changing the number of workers » for the left plot while it is in the right plot
- The explanation on what the variants of Reactor (ND and 500M) mean comes after results are shown in Fig. 2.
- Section 4 starts with Fig. 3 without explaining what the task is, how performance is measured, etc. It also claims that Distributional Retrace helps while this is not the case in Fig. 3 (I realize it is explained afterwards, but it is confusing when reading the sentence « We can also see... »). Finally it says priorization is the most important component while the beta-LOO ablation seems to perform just the same.
- Footnote 3 should say it is 200M observations except for Reactor 500M
- End of 4.1: « The algorithms that we compare Reactor against are » => missing ACER, A3C and Rainbow
- There are two references for « Sample efficient actor-critic with experience replay »
- I do not see the added benefit of the Elo computation. It seems to convey essentially the same information as average rank.
And a few typos:
- Just above 2.1.3: « increasing » => increasingly
- In 3.1: « where V is a baseline that depend » => depends
- p.7: « hight » => high, and « to all other sequences » => of all other sequences
- Double parentheses in Bellemare citation at beginning of section 4
- Several typos in appendix (too many to list)
Note: I did not have time to carefully read Appendix 6.3 (contextual priority tree)
Edit after revision: bumped score from 5 to 7 because (1) authors did many improvements to the paper, and (2) their explanations shed light on some of my concerns