Paper: SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR-2024)

Reviewer comment (Presentation):
Formatting obviously needs to be updated for the code presented in Code 1 and 2 in section 3.4.

Author response:
Thank you for your thoughtful review and constructive feedback on our paper! We appreciate your recognition of our efforts to develop a practical and efficient distributed RL platform for the community. We want to emphasize that this system has been widely adopted within our research group, and we are committed to sharing and refining it further.

Regarding formatting, Code 1 and 2 have been aligned in our revised paper. We will carefully review the paper once more to fix typos.

Reviewer comment (Novelty):
MSRL also introduced a data flow abstraction and supported IMPALA and SeedRL distributed architectures. The worker abstraction in SRL is similar to the "fragment" in MSRL. Environment ring is also introduced in EnvPool. Please provide further information about the novelty of SRL.

Author response:
We respectfully disagree with the reviewer with his/her comments about the contribution and novelty of this paper.

First, we remark that SRL’s contribution is much more than just a proposal of system abstractions. SRL is the first open-source system that can really perform efficient large-scale RL experiments over 15000 cores and reproduce OpenAI’s hide-and-seek project. Besides, SRL offers flexible and extensible interfaces such that users can effortlessly implement a new RL algorithm (see code snippets in Sec3.4 & App.A, various algorithm implementations in Sec.4.2). This paper also performs algorithm evaluation and confirms that insights provided by [1,2] extend to large-scale settings up to 15000 cores, i.e., larger batch sizes are usually beneficial to PPO’s performance. We also want to remark that these experimental and engineering efforts are appreciated by [reviewer Qkzp](https://openreview.net/forum?id=lajn1iROCu&noteId=ITT99z9ky8 ) and [bQuB
](https://openreview.net/forum?id=lajn1iROCu&noteId=gPmZgegemV) . We kindly hope the reviewer can consider these aspects when judging this paper.

Second, we have to emphasize that SRL’s abstraction indeed has novelty even compared with the concept proposed by MSRL. Although MSRL’s design concepts can implement both of SEED and IMPALA architectures, it does not provide additional solutions to *overcome their limitations caused by coupling*, as we emphasized and deeply discussed in Sec 2.2. In contrast, workers in SRL are completely decoupled, allowing massive parallelism and efficient resource utilization under large-scale settings.

[1] Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula

[2] Marcin Andrychowicz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, Raphael Marinier, ¨ Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier ´ Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. 

> Environment ring is also introduced in EnvPool.

We remark that EnvPool runs environments in a thread pool, resembling *parallelism*, while environment ring adopts the idea of *pipelining* between environment simulation and policy inference to reduce idle time.

Reviewer comment (Experiments):
In order to provide a comprehensive understanding of where any gains come from, it would be valuable to include a breakdown ablation study in the evaluation for each optimization and design.

Author response:
Please find the breakdown ablation studies in Appendix C. These results are deferred to appendix due to page limits. In ablation studies, we show the benefits from environment ring, GPU inference, and trainer worker prefetching. Moreover, we also study the effect of each worker type and offer practical recommendations on how to configure SRL to achieve the best practical performance.

Reviewer comment (Experiments):
It would be beneficial to include MSRL in the experiments to evaluate the gain compared to the state-of-the-art solutions. This could provide deeper insights into where SRL stands in relation to other established techniques.

Author response:
We want to emphasize that **MSRL results have been included in the experiments already. Please refer to paragraph 2 in Section 4.1**. 

We want to further clarify that **we have made our best effort to run the open-source code of MSRL. After we carefully checked the implementation, we have fully confirmed that the MSRL codebase cannot be examined in the same testing scenarios (with a similar scale) as we did**. In MSRL, each environment process requires an independent GPU for policy inference because it uses NCCL as the communication backend. Besides, we didn’t find a multi-learner option in the code. (See [code](https://github.com/mindspore-lab/mindrl/blob/master/mindspore_rl/algorithm/impala/impala_trainer.py).) These implementation choices limit the practical scalability of MSRL. In practice, we can utilize at most 8 CPU cores in the single-machine setting (64 cores for other methods) and at most 64 CPU cores in the distributed setting (700 cores for RLlib and 3200 cores for SRL). We believe this may not become  a fair comparison with such a scalability difference. As a result, we did not include MSRL in our main results.

In addition, as a most sincere attempt to conduct a possible comparison between MSRL and our project, we evaluate SRL under the same setting as the  MSRL paper (Section 6.2 Figure 6(a) in MSRL paper). In this setting, MSRL evaluated the PPO algorithm with 320 Mujoco Halfcheetah environments and 24 Nvidia V100 GPUs for inference. From the numbers in the MSRL paper, MSRL can finish one episode of 1000 steps in 3.85s resulting in 83116 FPS. In our experiment, SRL employs the same number of CPU cores and environment instances, but only need 4 Nvidia 3090 GPUs for inference. Even with 6x fewer GPUs, SRL achieves an overall training throughput of 210165 FPS, which is about 2.52x faster than the reported number of MSRL.

Reviewer comment (Evaluation):
Performance results lack error bars.

Author response:
Regarding the error bar, thanks for the suggestions and we have updated the figures in our paper. We want to clarify that we measured FPS within one hour. The number is stable over such a long time span. As a result, the error bars may look negligible.

Reviewer comment (Novelty):
Please clarify or contextualize the novelty of the work (see above).

Author response:
Finally, we would still want to discuss the difference between SRL and classical task-oriented programming. Libraries like Charm++ and Ray usually require a driver process (e.g. trainer) to invoke RPCs on remote workers (e.g. actors) to run the training loop. In contrast, SRL does not need a driver and runs independent workers. This can lead to the following differences in the context of RL. First, RPCs may result in additional communication overhead because we need to send requests, while workers in SRL only transfer data among workers to maximize throughput. Second, it is easier to implement customized dataflow with SRL, e.g., controlling multiple agents with different policies, by composing workers and streams. Achieving this with classical task-oriented programming would require the user to change the driver code, which can be tedious.

Reviewer comment (Evaluation):
Please add error bars to the performance results.

Author response:
Regarding the error bar, thanks for the suggestions and we have updated the figures in our paper. We want to clarify that we measured FPS within one hour. The number is stable over such a long time span. As a result, the error bars may look negligible.