Paper:
RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNING
Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay. We study the effects of parameter lag resulting in representational drift and recurrent state staleness and empirically derive an improved training strategy. Using a single network architecture and fixed set of hyperparameters, the resulting agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and matches the state of the art on DMLab-30. It is the first agent to exceed human-level performance in 52 of the 57 Atari games.

Review:
In this submission, the authors investigate using recurrent networks in distributed DQN with prioritized experience replay on the Atari and DMLab benchmarks. They experiment with several strategies to initialize the recurrent state when processing a sub-sequence sampled from the replay buffer: the best one consists in re-using the initial state computed when the sequence was originally played (even if it may now be outdated) but not doing any network update during the first k steps of the sequence (“burn-in” period). Using this scheme with LSTM units on top of traditional convolutional layers, along with a discount factor gamma = 0.997, leads to a significant improvement on Atari over the previous state-of-the-art, and competitive performance on DMLab.
The proposed technique (dubbed R2D2) is not particularly original (it is essentially “just” using RNNs in Ape-X), but experiments are thorough, investigating several important aspects related to recurrence and memory to validate the approach. These findings are definitely quite relevant to anyone using recurrent networks on RL tasks. The results on Atari are particularly impressive and should be of high interest to researchers working on this benchmark. The fact that the same network architecture and hyper-parameters also work pretty well on DMLab is encouraging w.r.t. the generality of the method.
I do have a couple of important concerns though. The first one is that a few potentially important changes were made to the “traditional” settings typically used on Atari, which makes it difficult to perform a fair comparison to previously published results. Using gamma = 0.997 could by itself provide a significant boost, as hinted by results from “Meta-Gradient Reinforcement Learning” (where increasing gamma improved results significantly compared to the usual 0.99). Other potentially impactful changes are the absence of reward clipping (replaced with a rescaling scheme) and episodes not ending with life loss: I am not sure whether these make the task easier or harder, but they certainly change it to some extent (the “despite this” above 5.1 suggests it would be harder, but this is not shown empirically). Fortunately, this concern is partially alleviated by Section 6 that shows feedforward networks do not perform as well as recurrent ones, but this is only verified on 5 games: a full benchmark comparison would have been more reassuring (as well as running R2D2 with more “standard” Atari settings, even if it would mean using different hyper-parameters on DMLab).
The second important issue I see is that the authors do not seem to plan to share their code to reproduce their results. Given how time consuming and costly it is to run such experiments, and all potentially tricky implementation details (especially when dealing with recurrent networks), making this code available would be tremendously helpful to the research community (particularly since this paper claims a new SOTA on Atari). I am not giving too much weight to this issue in my review score since (unfortunately) the ICLR reviewer guidelines do not explicitly mention code sharing as a criterion, but I strongly hope the authors will consider it.
Besides the above, I have a few additional small questions:
1. “We also found no benefit from using the importance weighting that has been typically applied with prioritized replay”: this is potentially surprising since this could be “wrong”, mathematically speaking. Do you think this is because of the lack of stochasticity in the environments? (I know Atari is deterministic, but I am not sure about DMLab)
2. Fig. 3 (left) shows R2D2 struggling on some DMLab tasks. Do you have any idea why? The caption of Table 3 in the Appendix suggests the absence of specific reward clipping may be an issue for some tasks, but have you tried adding it back? I also wonder if maybe training a unique network per task may make DMLab harder, since IMPALA has shown some transfer learning occurring between DMLab tasks? (although the comparison might be to the “deep experts” version of IMPALA — this is not clear in Fig. 3 — in which case this last question would be irrelevant)
3. In Table 1, where do the IMPALA (PBT) numbers on DMLab come from? Looking at the current arxiv version of their paper, their Fig. 4 shows it goes above 70% in mean capped score, while your Table 1 reports only 61.5%. I also can’t find a median score being reported on DMLab in their paper, did you try to compute it from their Fig. 9? And why don’t you report their results on Atari?
4. Table 4’s caption mentions “30 no-op starts” but you actually used the standard “random starts” setting, right? (not a fixed number of 30 no-ops)
And finally a few minor comments / suggestions:
- In the equation at bottom of p. 2, it seems like theta and theta- (the target network) have been accidentally swapped (at least compared to the traditional double DQN formula)
- At top of p. 3 I guess \bar{delta}_i is the mean of the delta_i’s, but then the index i should be removed
- In Fig. 1 (left) please clarify which training phase these stats are computed on (whole training? beginning / middle / end?)
- p. 4, “the true stored recurrent states at each step”: “true” is a bit misleading here as it can be interpreted as “the states one would obtain by re-processing the whole episode from scratch with the current network” => I would suggest to remove it, or to change it (e.g. “previously”). By the way, I think it would have been interesting to also compare to these states recomputed “from scratch”, since they are the actual ground truth.
- I think you should mention in Table 1’s caption that the PBT IMPALA is a single network trained to solve all tasks
- Typo at bottom of p. 7, “Indeed, Table 1 that even...”
Update: score updated to 8 (from 7) following discussion below