Paper: An Extensive Analysis on the Underlying Premises Behind Deep Reinforcement Learning Algorithm Design (ICLR-2024)

Reviewer comment (Theory):
I found the theoretical work limiting as it's only concerned with the linear case and therefore it ignores the many optimisation challenges of Deep RL (see Lyle, 2023 for a good example of the difficulties of pinpointing what exact intervention in the algorithm correlates with performance) which could easily dominate the sample complexity for any given data regime.

Author response:
**3.** *“The reason I cited Schwarzer is because it demonstrates that larger models perform better in the low data regime which goes against the insights from your theoretical analyses. Maybe they are wrong to use Rainbow but I don't think it has any bearing on this specific point.”*

One important point in [1] is that with the same number of parameters used in the prior studies the CNN architecture obtains IQM 0.28 while the ResNet architecture with the same number of parameters obtains 0.55. Thus one of the main gains achieved here (**96%** performance gain over CNN) is **not** due to increased parameters but rather simply using a **better architecture**. 

Again note that the paper [1] uses SPR with **weight decay, annealing, harder resets, receding update horizon, and increasing discount factor**, all of which are quite recent techniques that contribute substantially to increasing sample efficiency. So much so that for instance, as can be seen from Figure 5 of [1], simply not using resets reduces the performance from 1.01 to 0.40. Thus simply using resets causes **152.5% performance gain**. Again we see that simply using standard n reduces the performance to 0.54 from 0.96. Thus simply using receding update horizon contributes by **77% to the performance gain**. Again we see that without annealing the performance is reduced to 0.65 from 0.96, where again simply using annealing causes **47.7% performance gain**. Again without harder resets the performance is reduced to 0.6 from 1.01 thus simply using harder resets achieves **68.3% performance gain**.

Thus looking over all these simple techniques that contribute **substantially** to the performance achieved, even more importantly with a **completely different architecture** with the same number of parameters causing **96%** performance gain, further demonstrates that the performance gain achieved in [1] is contributed by utilizing resetting that contributes **152.5%**, using receding update horizon that contributes **77%**, using a different architecture **96%**, and using harder resets that contributes **68.3%**. Thus, given these it is not true that the paper [1] goes against the insights from our theoretical analyses.

[1] Max Schwarzer, Johan Obando-Ceron, Aaron Courville, Marc G. Bellemare, Rishabh Agarwal, Pablo Samuel Castro. Bigger, Better, Faster: Human-level Atari with human-level efficiency, ICML 2023.

Reviewer comment (Novelty):
While I find the claim that algorithms developed in one data regime won't monotonically transfer to other data regimes is generally true, I don't think it is a novel or surprising observation. There are several papers on tuning Rainbow alone (van Hasselt 2019 with DER, Kielak 2019 with OTR, more recently Schmidt et al) that make this exact point, that various data constraints will require essentially different algorithms.

Author response:
**1.** *”While I find the claim that algorithms developed in one data regime won't monotonically transfer to other data regimes is generally true, I don't think it is a novel or surprising observation. There are several papers on tuning Rainbow alone (van Hasselt 2019 with DER, Kielak 2019 with OTR, more recently Schmidt et al) that make this exact point, that various data constraints will require essentially different algorithms.”*

We would like to kindly highlight that you are **incorrect** on the description of the prior work which we also already cite in our paper [1,2,3]. These studies demonstrate that one cannot use the exact same hyperparameters that are tuned for the 200 million frame training in 100K, and if the hyperparameters are specifically tuned these papers state that the best performing algorithm in the high data regime will be again best performing in the low data regime.

On the other hand, our paper demonstrates that even when hyperparameters are specifically tuned for the data regime one should not and cannot expect the performance profile of an algorithm to monotonically transfer from the high data regime to the low-data regime and vice versa. 


[1] When to use parametric models in reinforcement learning? NeurIPS 2019.

[2] Do recent advancements in model-based deep reinforcement learning really improve data efficiency? CoRR, 2019.

[3] Dominik Schmidt and Thomas Schmied. Fast and Data-Efficient Training of Rainbow: an Experimental Study on Atari