Original: GPU-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning
Revised: ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Deep reinforcement learning (DRL) has revolutionized learning and actuation inapplications such as game playing and robotic control.
Revised: Deep reinforcement learning (DRL) has revolutionized learning and actuation in applications such as game playing and robotic control.

Original: The cost of data collection,i.e., generating transitions from agent-environment interactions, remains a majorchallenge for wider DRL adoption in complex real-world problems.
Revised: The cost of data collection, i.e., generating transitions from agent-environment interactions, remains a major challenge for wider DRL adoption in complex real-world problems.

Original: Following acloud-native paradigm to train DRL agents on a GPU cloud platform is a promisingsolution.
Revised: Following a cloud-native paradigm to train DRL agents on a GPU cloud platform is a promising solution.

Original: In this paper, we present a scalable and elastic library GPU-podracerfor cloud-native deep reinforcement learning, which efﬁciently utilizes millionsof GPU cores to carry out massively parallel agent-environment interactions.
Revised: In this paper, we present a scalable and elastic library ElegantRL-podracer for cloud-native deep reinforcement learning, which efﬁciently supports millions of GPU cores to carry out massively parallel training at multiple levels.

Original: Ata high-level, GPU-podracer employs a tournament-based ensemble scheme toorchestrate the training process on hundreds or even thousands of GPUs, schedulingthe interactions between a leaderboard and a training pool with hundreds of pods.
Revised: At a high-level, ElegantRL-podracer employs a tournament-based ensemble scheme to orchestrate the training process on hundreds or even thousands of GPUs, scheduling the interactions between a leaderboard and a training pool with hundreds of pods.

Original: At a low-level, each pod simulates agent-environment interactions in parallelby fully utilizing nearly 7 , 000 GPU CUDA cores in a single GPU.
Revised: At a low-level, each pod simulates agent-environment interactions in parallel by fully utilizing nearly 7 , 000 GPU CUDA cores in a single GPU.

Original: Our GPUpodracer library features high scalability, elasticity and accessibility by followingthe development principles of containerization, microservices and MLOps.
Revised: Our ElegantRLpodracer library features high scalability, elasticity and accessibility by following the development principles of containerization, microservices and MLOps.

Original: Usingan NVIDIA DGX SuperPOD cloud , we conduct extensive experiments on varioustasks in locomotion and stock trading and show that GPU-podracer outperforms Stable Baseline3 and RLlib, e.g., GPU-podracer achieves nearly linear scaling.
Revised: Using an NVIDIA DGX SuperPOD cloud, we conduct extensive experiments on various tasks in locomotion and stock trading and show that ElegantRL-podracer substantially outperforms RLlib. Our codes are available on GitHub [16].

Original:  Introduction
Revised:  Introduction

Original: Deep reinforcement learning (DRL), which balances the exploration (of uncharted territory) andexploitation (of current information) has revolutionized learning and actuation in applications suchas game playing [23] and robotic control [29].
Revised: [Table] (DRL), which balances the exploration (of uncharted territory) and exploitation (of current information), has revolutionized learning and actuation in applications such as game playing [25] and robotic control [31].

Original: DRL employs a trial-and-error manner to generatetransitions from agent-environment interactions, along with the learning procedure.
Revised: DRL employs a trial-and-error manner to generate transitions from agent-environment interactions, along with the learning procedure.

Original: However, thecost of data collection remains a major challenge for wider DRL adoption in real-world problemswith complex and dynamic environments.
Revised: However, the cost of data collection remains a major challenge for wider DRL adoption in real-world problems with complex and dynamic environments.

Original: Therefore, a compelling solution is massively paralleltraining on hundreds or even thousands of GPUs, say millions of GPU cores.
Revised: Therefore, a compelling solution is massively parallel training on hundreds or even thousands of GPUs, say millions of GPU cores.

Original: Existing DRL frameworks are not satisfactory with respect to scalability and accessibility.
Revised: Existing DRL frameworks are not satisfactory with respect to scalability and accessibility.

Original: As shownin Fig.
Revised: As shown in Fig.

Original: 1, OpenAI Baselines [7], Stable Baselines 3 [19] and 
Revised: OpenAI Baselines [7], Stable Baselines 3 [21] and OpenAI

Original: OpenAI Spinning Up [1] utilize a single
Revised: Spinning Up [1] utilize a single

Original: GPU, while RLlib [13] and rlpyt [24] can support multiple GPUs.
Revised: GPU, while RLlib [14] and rlpyt [26] can support multiple GPUs.

Original: However, there is no existing
Revised: However, there is no existing

Original: DRL framework for a cloud with hundreds or even thousands of GPUs.
Revised: DRL framework for a cloud with hundreds or even thousands of GPUs.

Original: We aim to fully utilize twocore techonlogies: 1).
Revised: We aim to fully utilize two core techonlogies: 1).

Original: GPUDirect technology [25] that provides a path for data to bypass CPUs andtravel on “open highways” offered by GPUs, storage, and networking devices; and 2).
Revised: GPUDirect technology [27] that provides a path for data to bypass CPUs and travel on “open highways” offered by GPUs, storage, and networking devices; and 2).

Original: massivelyparallel simulations using thousands of GPU cores on a single GPU.
Revised: massively parallel simulations using thousands of GPU cores on a single GPU.

Original: On the other hand, the aboveframeworks, except OpenAI Spinning Up serving an educational purpose, involve a steep learningcurve or a lack of customization ﬂexibility, which results in low accessibility.
Revised: On the other hand, the above frameworks, except OpenAI Spinning Up serving an educational purpose, involve a steep learning curve or a lack of customization ﬂexibility, which results in low accessibility.

Original: Scaling out the training process of DRL agents to hundreds or even thousands of GPUs is challengingfor researchers and practitioners.
Revised: Scaling out the training process of DRL agents to hundreds or even thousands of GPUs is challenging for researchers and practitioners.

Original: The cloud-native paradigm aims to scalably and elastically utilizethe cloud computing resources.
Revised: The cloud-native paradigm aims to scalably and elastically utilize the cloud computing resources.

Original: Therefore, we believe it is practically promising to schedule thetraining of DRL agents by following the cloud-native paradigm, such as employing standardizedsoftware stack, e.g., Kubernetes (K8s) [3], and adopting core technologies including containers,microservices, continuous integration (CI) and continuous delivery (CD) [2, 9].
Revised: Therefore, we believe it is practically promising to schedule the training of DRL agents by following the cloud-native paradigm, such as employing standardized software stack, e.g., Kubernetes (K8s) [3], and adopting core technologies including containers, microservices, continuous integration (CI) and continuous delivery (CD) [2, 9].

Original: In this paper, we present a scalable and elastic library GPU-podracer for cloud-native deep reinforcement learning, which efﬁciently utilizes millions of GPU cores to carry out massively parallelagent-environment interactions.
Revised: In this paper, we present a scalable and elastic library ElegantRL-podracer for cloud-native deep reinforcement learning, which efﬁciently utilizes millions of GPU cores to carry out massively parallel training at multiple levels.

Original: At a high-level, GPU-podracer employs a tournament-based ensemble scheme to orchestrate the training process on hundreds or even thousands of GPUs, schedulingthe interactions between a leaderboard and a training pool with hundreds of pods.
Revised: At a high-level, ElegantRL-podracer employs a tournamentbased ensemble scheme to orchestrate the training process on hundreds or even thousands of GPUs, scheduling the interactions between a leaderboard and a training pool with hundreds of pods.

Original: At a low-level,each pod simulates agent-environment interactions in parallel by fully utilizing over 7 , 000 GPUcores in a single GPU.
Revised: At a low-level, each pod simulates agent-environment interactions in parallel by fully utilizing over 7 , GPU cores in a single GPU.

Original: Our GPU-podracer library features high scalability, elasticity and accessibilityby following the development principles of containerization, microservices and MLOps.
Revised: Our ElegantRL-podracer library features high scalability, elasticity and accessibility by following the development principles of containerization, microservices and MLOps.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • We present a scalable and elastic open-source library for cloud-native deep reinforcement learning,GPU-podracer , that can utilize millions of GPU cores to train effective DRL agents for complex real-world problems.
Revised: • We present a scalable and elastic open-source library for cloud-native deep reinforcement learning, ElegantRL-podracer , that can utilize millions of GPU cores to train effective DRL agents for complex real-world problems.

Original: • To accelerate data collection for efﬁcient exploration, we propose a tournament-based ensemble training scheme and employ massive parallel simulations.
Revised: • To accelerate data collection for efﬁcient exploration, we propose a tournament-based ensemble training scheme and employ massive parallel simulations.

Original: • GPU-podracer follows a cloud-native paradigm by realizing the development principles of containerization, microservices and MLOps (e
Revised: • ElegantRL-podracer follows a cloud-native paradigm by realizing the development principles of containerization, microservices and MLOps (e

Original: g., continuous integration and continuous delivery), and achieves high accessibility.
Revised: g., continuous integration and continuous delivery), and achieves high accessibility.

Original: • Using an NVIDIA DGX SuperPOD cloud [25], we conduct extensive experiments on various tasks in locomotion and stock trading and show that GPU-podracer outperforms[-Stable Baseline
 [19]and-] RLlib [13], e.g., GPU-Podracer achieves nearly linear scaling.
Revised: • Using an NVIDIA DGX SuperPOD cloud [27], we conduct extensive experiments on various tasks in locomotion and stock trading and show that ElegantRL-podracer substantially outperforms[-Stable Baseline
 [19]and-] RLlib [14].

Original: The remainder of this paper is organized as follows.
Revised: The remainder of this paper is organized as follows.

Original: Section 2 describes related works.
Revised: Section 2 describes related works.

Original: Section 3presents our design principles.
Revised: Sectionpresents our design principles.

Original: Section 4 describes the GPU-podracer library.
Revised: Section 4 describes the ElegantRL-podracer library.

Original: In Section 5, we presentexperimental results.
Revised: In Section 5, we present experimental results.

Original: We conclude this paper in Section 6.
Revised: We conclude this paper in Section 6.

Original:  Related Works
Revised:  Related Works

Original: We review open-source DRL frameworks/libraries and environment simulation packages.
Revised: We review open-source DRL frameworks/libraries and environment simulation packages.

Original: 2.1 Deep Reinforcement Learning Framework/Libraries
Revised: 2.1 Deep Reinforcement Learning Framework/Libraries

Original: Many open-source DRL frameworks/libraries have been developed in recent years with variedcapabilites.
Revised: Many open-source DRL frameworks/libraries have been developed in recent years with varied capabilites.

Original: OpenAI Spinning Up [1] and Google Dopamine [5] are research frameworks for thefast prototyping of DRL algorithms.
Revised: OpenAI Spinning Up [1] and Google Dopamine [5] are research frameworks for the fast prototyping of DRL algorithms.

Original: They both implement numerous DRL algorithms with simpleand pedagogical codes.
Revised: They both implement numerous DRL algorithms with simple and pedagogical codes.

Original: Stable Baseline

Revised: Stable Baseline

Original: [19] is a stable and efﬁcient DRL library, introducing theparallelism of sampling through vectorized environment.
Revised: [21] is a stable and efﬁcient DRL library, introducing the parallelism of sampling through vectorized environment.

Original: RLlib [13] is a generic DRL library thatderives its strength from Ray communication procotols that enable scalable, distributed training.
Revised: RLlib [14] is a generic DRL library that derives its strength from Ray communication procotols that enable scalable, distributed training.

Original: Podracer [12] from DeepMind is closely related to our GPU-podracer, which also focuses on theefﬁcient usage of large computing resources for training DRL agents.
Revised: Podracer [12] from DeepMind is closely related to our ElegantRL-podracer, which also focuses on the efﬁcient usage of large computing resources for training DRL agents.

Original: However, it is designed for
Revised: However, it is designed for

Original: Google’s tensor processing units (TPUs) that are inaccessible to many researchers and practitioners.
Revised: Google’s tensor processing units (TPUs) that are inaccessible to many researchers and practitioners.

Original: 2.2 Simulation Packages
Revised: 2.2 Simulation Packages

Original: Environment simulation is a critical component of DRL training, and lots of platforms that providevarious task simulations are emerging to close the simulation-to-reality gap.
Revised: Environment simulation is a critical component of DRL training, and lots of platforms that provide various task simulations are emerging to close the simulation-to-reality gap.

Original: OpenAI Gym [4] is afundamental simulation toolkit for DRL research, which deﬁnes a standard interface for follow-upworks.
Revised: OpenAI Gym [4] is a fundamental simulation toolkit for DRL research, which deﬁnes a standard interface for follow-up works.

Original: It includes a collection of benchmark problems, e.g., classic control, Atari games, and 2Dand 3D robots.
Revised: It includes a collection of benchmark problems, e.g., classic control, Atari games, and 2D and 3D robots.

Original: MuJoCo [26] and Isaac Gym [16] are two powerful platforms for robotic simulations.
Revised: MuJoCo [28] and Isaac Gym [18] are two powerful platforms for robotic simulations.

Original: MuJoCo [26] is a popular physics simulator that efﬁciently simulates joint contact models.
Revised: MuJoCo [28] is a popular physics simulator that efﬁciently simulates joint contact models.

Original: Therecently released Isaac Gym [16] is a high-performance simulation environment for physics.
Revised: The recently released Isaac Gym [18] is a high-performance simulation environment for physics.

Original: It enablesthousands of environments running in parallel on a single GPU.
Revised: It enables thousands of environments running in parallel on a single GPU.

Original: FinRL [15] is a new ﬁnance-related DRL platform, which simulates various markets as training environments that are built on historicaldata and live trading APIs.
Revised: FinRL [17, 13] is a new ﬁnance-related DRL platform, which simulates various markets as training environments that are built on historical data and live trading APIs.

Original:  Design Principles and Overview
Revised:  Design Principles and Overview

Original: We aim to develop a user-friendly open-source library that fully exploits cloud resources to train DRLagents.
Revised: We aim to develop a user-friendly open-source library that fully exploits cloud resources to train DRL agents.

Original: The library emphasizes the following design principles:
Revised: The library emphasizes the following design principles:

Original: • Scaling-out : scalability and elasticity.
Revised: • Scaling-out : scalability and elasticity.

Original: • Efﬁciency : low communication overhead, massively parallel simulations and robustness of agents.
Revised: • Efﬁciency : low communication overhead, massively parallel simulations and robustness of agents.

Original: • Accessibility : lightweightness and customization.
Revised: • Accessibility : lightweightness and customization.

Original: For algorithm design, GPU-podracer employs a tournament-based ensemble training  to balanceexploration and exploitation.
Revised: For algorithm design, ElegantRL-podracer employs a tournament-based ensemble training scheme to balance exploration and exploitation.

Original: In contrast to Evolutionary Strategies (ES) [20] where a populationof agents evolve over generations, our tournament-based ensemble training scheme updates agentsasynchronously in parallel, which decouples population evolution and single-agent learning.
Revised: In contrast to Evolutionary Strategies (ES) [22] where a population of agents evolve over generations, our tournament-based ensemble training scheme updates agents asynchronously in parallel, which decouples population evolution and single-agent learning.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the key of the tournament-based ensemble training scheme is the interaction betweena leaderboard and a training pool .
Revised: 2, the key of the tournament-based ensemble training scheme is the interaction between a leaderboard and a training pool .

Original: The training pool contains hundreds of agents (pods) that 1)are trained in an asynchronous manner, and 2) can be initialized with different DRL algorithms forthe ensemble purpose.
Revised: The training pool contains hundreds of agents (pods) that 1) are trained in an asynchronous manner, and 2) can be initialized with different DRL algorithms for an ensemble purpose.

Original: The leaderboard records the agents with high performance and continuallyupdates as more agents (pods) are trained.
Revised: The leaderboard records the agents with high performance and continually updates as more agents (pods) are trained.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the tournament-based ensemble training scheme proceeds as follows:
Revised: 2, the tournament-based ensemble training scheme proceeds as follows:

Original: An orchestrator instantiates a new agent (pod) and put it into a training pool.
Revised: An orchestrator instantiates a new agent (pod) and put it into a training pool.

Original: A generator initializes an agent (pod) with networks and optimizers selected from a leaderboard.
Revised: A generator initializes an agent (pod) with networks and optimizers selected from a leaderboard.

Original: The generator is a class of subordinate functions associated with the leaderboard, which hasdifferent variations to support different evolution strategies.
Revised: The generator is a class of subordinate functions associated with the leaderboard, which has different variations to support different evolution strategies.

Original: An updater determines whether and where to insert an agent into the leaderboard according to its performance, after a pod has been trained for a certain number of steps or certain amount of time.
Revised: An updater determines whether and where to insert an agent into the leaderboard according to its performance, after a pod has been trained for a certain number of steps or certain amount of time.

Original: For system design, GPU-podracer follows the cloud-native paradigm.
Revised: For system design, ElegantRL-podracer follows the cloud-native paradigm.

Original: GPU-podracer achievescontainerization by implementing the tournament-based ensemble training scheme as the synergy ofmicroservices.
Revised: ElegantRL-podracer achieves containerization by implementing the tournament-based ensemble training scheme as the synergy of microservices.

Original: Such a paradigm allows a lightweight usage via simple APIs and a high degree ofcustomization through the ﬂexible cooperation of microservices.
Revised: Such a paradigm allows a lightweight usage via simple APIs and a high degree of customization through the ﬂexible cooperation of microservices.

Original: At a high-level, GPU-podracer has the following capabilities to embody our design principles:
Revised: At a high-level, ElegantRL-podracer has the following capabilities to embody our design principles:

Original: • Asynchronously distributed training is made possible through a training pool.
Revised: • Asynchronously distributed training is made possible through a training pool.

Original: GPU-podracercan scale out to hundreds or even thousands of computing nodes and elastically adjust the number of agents according to the available computing resources.
Revised: ElegantRLpodracer can scale out to hundreds or even thousands of computing nodes and elastically adjust the number of agents according to the available computing resources.

Original: • Tournament-based ensemble training is made possible through a leaderboard.
Revised: • Tournament-based ensemble training is made possible through a leaderboard.

Original: Tournament- based training scheme decouples the agent learning and population evolution to achieve low communication overhead between pods.
Revised: Tournamentbased training scheme decouples the agent learning and population evolution to achieve low communication overhead between pods.

Original: Ensembling many DRL algorithms increases efﬁciency by exploiting agent robustness and diversity.
Revised: Ensembling many DRL algorithms increases efﬁciency by exploiting agent robustness and diversity.

Original: • Cloud-nativity is made possible with the containerization, microservices, and MLOps adherence.
Revised: • Cloud-nativity is made possible with the containerization, microservices, and MLOps adherence.

Original: MLOps achieves continuous training/integration/delivery (CT /CI/CD) by exploiting the Kurbernetes (K8s) [3] software for automated cloud orchestration.
Revised: MLOps achieves continuous training/integration/delivery (CT/CI/CD) by exploiting the Kurbernetes (K8s) [3] software for automated cloud orchestration.

Original:  GPU-Podracer: Scalable and Elastic Cloud-native Library
Revised:  ElegantRL-Podracer: Scalable and Elastic Cloud-native Library

Original: In this section, we propose a scalable and elastic cloud-native library, called GPU-Podracer .
Revised: In this section, we propose a scalable and elastic cloud-native library, called ElegantRL-podracer .

Original: We ﬁrstdescribe its key components and then present its features.
Revised: We ﬁrst describe its key components and then present its features.

Original: 4.1 Ensemble Training Using Microservices
Revised: 4.1 Ensemble Training Using Microservices

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, the ensemble training scheme exploits the synergy of the following microservices:orchestrator, leaderboard (including updater and generator), and agents (pods) in the training pool,where each microservice maps to a container 
Revised: 2, the ensemble training scheme exploits the synergy of the following microservices: orchestrator, leaderboard (including updater and generator), and agents (pods) in the training pool, where each microservice maps to a container .

Original: Orchestrator : An orchestrator monitors the available computing resources and determines the number of pods in the training pool.
Revised: Orchestrator : An orchestrator monitors the available computing resources and determines the number of pods in the training pool.

Original: When K8s signals that the workload is light, the orchestrator generates a set of new pods and insert them into the training pool.
Revised: When K8s signals that the workload is light, the orchestrator generates a set of new pods and insert them into the training pool.

Original: When a training objective (i e., target rewards) is achieved, the orchestrator will terminate the training process.
Revised: When a training objective (i.e., target rewards) is achieved, the orchestrator will terminate the training process.

Original: Leaderboard : A leaderboard records a set of candidate agents with high performance, say cumula- tive reward.
Revised: Leaderboard : A leaderboard records a set of candidate agents with high performance, say cumulative reward.

Original: An updater updates the candidate agents to the leaderboard, while a generator instantiatesa new pod by referring to candidate agents.
Revised: An updater updates the candidate agents to the leaderboard, while a generator instantiates a new pod by referring to candidate agents.

Original: The leaderboard may also track other information, such as the covariance matrix, mean, variance, etc, which helps the generator to adaptively allocate computing resources to highly potential candidate agents.
Revised: The leaderboard may also track other information, such as the covariance matrix, mean, variance, etc, which helps the generator to adaptively allocate computing resources to highly potential candidate agents.

Original: • Updater : An updater receives a trained agent (pod)and may insert its training ﬁles (includingactor network, critic network, optimizer parameters, replay buffer (for off-policy algorithms)) into
Revised: : An updater receives a trained agent (pod) insert its training ﬁles ( network critic network

Original:  theleaderboard if it has high performance.
Revised: optimizer parameters buffer off ) into the

Original: • Generator : A generator generates training ﬁles for a newly created pod.
Revised: • Generator : A generator generates training ﬁles for a newly created pod.

Original: The generator may perform a mutation of the candidate agents to increase the diversity.
Revised: The generator may perform a mutation of the candidate agents to increase the diversity.

Original: 4.2 Agent Learning Using Microservices
Revised: 4.2 Agent Learning Using Microservices

Original: GPU-podracer maps the training process of an agent to a pod , which is the smallest deployable unitin K8s [3].
Revised: ElegantRL-podracer maps the training process of an agent to a pod , which is the smallest deployable unit in K8s [3].

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, GPU-podracer separates an agent learning into three microservices:worker, learner and evaluator.
Revised: 3, ElegantRL-podracer separates an agent learning into three microservices: worker, learner and evaluator.

Original: Worker : A worker generates batches of transitions from interactions between the agent and thebatched environment.
Revised: Worker : A worker generates batches of transitions from interactions between the actor and the batched environment.

Original: A batched environment consists of multiple independent sub-environments.
Revised: A batched environment consists of multiple independent sub-environments.

Original: Each actor collects transitions from the sub-environments of the batched environment in parallel toaccelerate the data collection process.
Revised: Each actor collects transitions from the sub-environments of the batched environment in parallel to accelerate the data collection process.

Original: Learner :
Revised: Learner :

Original: A learner fetches a batch of transitions from the replay buffer to train the neural networks.
Revised: A learner fetches a batch of transitions from the replay buffer to train the neural networks.

Original: GPU-podracer trains multiple learners in parallel and fuses the networks by aggregating networkparameters .
Revised: ElegantRL-podracer trains multiple learners in parallel and fuses the networks by aggregating network parameters .

Original: In this way, GPU-podracer experiments much less communication overhead thandistributed SGD in RLlib [13].
Revised: In this way, ElegantRL-podracer experiments much less communication overhead than distributed SGD in RLlib [14].

Original: Evaluator : An evaluator continuously evaluates a pod and records its performance and correspondingnetworks during the training process.
Revised: Evaluator : An evaluator continuously evaluates a pod and records its performance and corresponding networks during the training process.

Original: Commonly used performance metrics are the mean and varianceof the episodic reward.
Revised: Commonly used performance metrics are the mean and variance of the episodic reward.

Original: Note that the evaluator effectively mitigates the performance loss caused byeither overﬁtting or early stopping.
Revised: Note that the evaluator effectively mitigates the performance loss caused by either overﬁtting or early stopping.

Original: 4.3 Features of GPU-Podracer
Revised: 4.3 Features of ElegantRL-podracer

Original: GPU-podracer achieves several features that facilitate the implementation of a lightweight andpowerful training process on a GPUcloud.
Revised: ElegantRL-podracer achieves several features that facilitate the implementation of a lightweight and powerful training process on a GPU

Revised: cloud.

Original: Scalable parallelism : The multi-level parallelism of GPU-podracer leads to high scalability.
Revised: Scalable parallelism : The multi-level parallelism of ElegantRL-podracer leads to high scalability.

Original: • Agent parallelism : The agents in the training pool are parallel, thus can easily scale out to alarge number.
Revised: • Agent parallelism : The agents in the training pool are parallel, thus can easily scale out to a large number.

Original: The asynchronous training of parallel agents can also reduce the frequency of agent-to-agent communication.
Revised: The asynchronous training of parallel agents can also reduce the frequency of agent-to-agent communication.

Original: • Learner parallelism : An agent employs multiple learners to train the neural networks in parallel, and then fuse the networks parameters to obtain a result agent, instead of using distributed SGD.
Revised: • Learner parallelism : An agent employs multiple learners to train the neural networks in parallel, and then fuse the networks parameters to obtain a result agent, instead of using distributed SGD.

Original: Such a model fusion through network parameters involves a much lower frequency communication as the fusion process only happens at the end of an epoch.
Revised: Such a model fusion through network parameters involves a much lower frequency communication as the fusion process only happens at the end of an epoch.

Original: • Worker parallelism : An agent utilizes multiple rollout workers to sample trajectories in parallel.
Revised: • Worker parallelism : An agent utilizes multiple rollout workers to sample transitions in parallel.

Original: Elastic resource allocation :
Revised: Elastic resource allocation :

Original: The elasticity is critical for cloud-level applications as it helps users adapt to the changes in cloud resources and prevent over-provisioning and under-provisioning of resources [17, 11].
Revised: The elasticity is critical for cloud-level applications as it helps users adapt to the changes in cloud resources and prevent over-provisioning and under-provisioning of resources [19, 11].

Original: GPU-podracer can elastically allocate the number of agents (pods) by employing an orchestrator to monitor the available computing resources and the current training status.
Revised: ElegantRL-podracer can elastically allocate the number of agents (pods) by employing an orchestrator to monitor the available computing resources and the current training status.

Original: Cloud-oriented optimizations : GPU-podracer co-locates microservices on GPUs to accelerate the parallel computation on both data collection and model training.
Revised: Cloud-oriented optimizations : ElegantRL-podracer co-locates microservices on GPUs to accelerate the parallel computation on both data collection and model training.

Original: For the data transfer and storage,
Revised: For the data transfer and storage,

Original: GPU-podracer represents data as tensors to speedup the communication and allocates the shared replay buffer on the contiguous memory of GPUs to increase the addressing speed.
Revised: ElegantRL-podracer represents data as tensors to speedup the communication and allocates the shared replay buffer on the contiguous memory of GPUs to increase the addressing speed.

Original: Continuous training (CT) pipeline : Continuous training, which is a part of the MLOps practice,seeks to automatically and continuously retrain the model to adapt to changes that might occurin the data.
Revised: Continuous training (CT) pipeline : Continuous training, which is a part of the MLOps practice, seeks to automatically and continuously retrain the model to adapt to changes that might occur in the data.

Original: GPU-podracer performs the CT of a DRL agent by automating a lightweight DRLtraining pipeline, which is composed of microservices.
Revised: ElegantRL-podracer performs the CT of a DRL agent by automating a lightweight DRL training pipeline, which is composed of microservices.

Original: Users can conduct different experiments andhyper-parameter search by modifying workers, learners and other microservices.
Revised: Users can conduct different experiments and hyper-parameter search by modifying workers, learners and other microservices.

Original: Continuous integration/delivery (CI/CD) : GPU-podracer enables a robust CI/CD for users toexplore new ideas by modifying existing microservices or build new microservices.
Revised: Continuous integration/delivery (CI/CD) : ElegantRL-podracer enables a robust CI/CD for users to explore new ideas by modifying existing microservices or build new microservices.

Original: The microservicesare loosely coupled, such that the change of one microservice will not break existing ones.
Revised: The microservices are loosely coupled, such that the change of one microservice will not break existing ones.

Original: Alsomodularity allows for more comprehensive search over the experiment space, e.g., instead of designingone experiment at a time, we could theoretically be able to test c 1 × c 2 × · × c n experiments in anautomated fashion, where n is the number of components for a DRL algorithm and
Revised: Also modularity allows for more comprehensive search over the experiment space, e.g., instead of designing one experiment at a time, we could theoretically be able to test c 1 × c 2 × · × c n experiments in an automated fashion, where n is the number of components for a DRL algorithm and

Original: c i is the numberof optional microservices for component i , i = 1 , ..., n :
Revised: c i is the number of optional microservices for component i , i = 1 , ..., n :

Original: • Environment variation : GPU-podracer supports any environment written in gym-style and provides a class PreprocessVecEnv to convert a normal environment to a batch mode.
Revised: • Environment variation : ElegantRL-podracer supports any environment written in gym-style and provides a class PreprocessVecEnv to convert a normal environment to a batch mode.

Original: • Algorithm variation : GPU-podracer can realize different DRL algorithms through combinationsof worker and learner variants.
Revised: • Algorithm variation : ElegantRL-podracer can realize different DRL algorithms through combinations of worker and learner variants.

Original: Currently, GPU-podracer supports ﬁne-tuned standard DRLalgorithms, including DQN-series [18, 27], DDPG [14], TD3 [8], SAC [10] and PPO [21].
Revised: Currently, ElegantRL-podracer supports ﬁne-tuned standard DRL algorithms, including DQN-series [20, 29], DDPG [15], TD3 [8], SAC [10], and PPO [23].

Original: New algorithms may be used as long as they adhere to the agent interface.• Evolution variation : GPU-podracer allows users to customize the evaluator, updater and generator in the leaderboard to decide how to evaluate , where to update , and what to generate .
Revised: New algorithms may be used as long as they adhere to the agent interface. • Evolution variation : ElegantRL-podracer allows users to customize the evaluator, updater and generator in the leaderboard to decide how to evaluate , where to update , and what to generate .

Original:  Performance Evaluation
Revised:  Performance Evaluation

Original: In the section, we describe the cloud platform in our experiments and the performance of GPUpodracer on various tasks from robotic control to stock trading task.
Revised: In the section, we describe the cloud platform in our experiments and the performance of ElegantRLpodracer on various tasks from robotic control to stock trading task.

Original: 5.1 Experiment Platform: GPU Cloud
Revised: 5.1 Experiment Platform: GPU Cloud

Original: All experiments were executed using NVIDIA DGX-2 servers [6] in an NVIDIA DGX SuperPODcloud [25], a cloud-native infrastructure.
Revised: All experiments were executed using NVIDIA DGX-2 servers [6] in a DGX SuperPOD cloud [27], a cloud-native infrastructure.

Original: EachNVIDIA DGX-2 server contains 8 [-NVIDIA A
-] GPUs and 320 GB GPU memory in total, and also has 128 CPU cores of Dual AMD Rome 7742running at 2.25GHz.
Revised: Each DGX-2 server contains 8 [-NVIDIA A
-]A100 GPUs and 320 GB GPU memory in total, and also has 128 CPU cores of Dual AMD Rome 7742 running at 2.25GHz.

Original: Among the8 NVIDIA A100 GPUs in oneNVIDIA DGX-2 server, any two A100 GPUs are connected with each other through12 NVLinks, providing 600 Gbps of full-duplexbandwidth [6].
Revised: Among the A100 GPUs in one DGX-2 server, any two A100 GPUs are connected with each other through NVLinks, providing 600 Gbps of full-duplex bandwidth [6].

Original: 5.2 Robotic Control Tasks
Revised: 5.2 Robotic Control Tasks

Original: Ant and humanoid are two canonical robotic control tasks that simulate an ant and a humanoid,respectively, where each task has both MuJoCo [26] version and Isaac Gym [16] version, as shown in
Revised: Ant and humanoid are two canonical robotic control tasks that simulate an ant and a humanoid, respectively, where each task has both MuJoCo [28] version and Isaac Gym [18] version, as shown in

Original: Fig.
Revised: Fig.

Original: The ant task is a simple environment to simulate due to its stability in the initial state, while thehumanoid task is often used as a testbed for locomotion learning.
Revised: The ant task is a simple environment to simulate due to its stability in the initial state, while the humanoid task is often used as a testbed for locomotion learning.

Original: Even though the implementationsof MuJoCo [26] and Isaac Gym [16] are slightly different, the objective of both is to have the agentmove forward as fast as possible.
Revised: Even though the implementations of MuJoCo [28] and Isaac Gym [18] are slightly different, the objective of both is to have the agent move forward as fast as possible.

Original: The state space, action space and reward function are given in Table 1.
Revised: The state space, action space and reward function are given in Table 1.

Original: We select the same tasks from the two platforms in order to show that 1) GPU-podracer cansupport different simulator platforms, and 2) the potential of massively parallel simulations in the
Revised: We select the same tasks from the two platforms in order to show that 1) ElegantRL-podracer can support different simulator platforms, and 2) the potential of massively parallel simulations in the

Original: DRL training by comparing the CPU-based MuJoCo [26] with the GPU-based Isaac Gym [16].
Revised: DRL training by comparing the CPU-based MuJoCo [28] with the GPU-based Isaac Gym [18].

Original: Compared methods : On one DGX-2 server, we compare GPU-podracer with RLlib [13], sinceboth support multiple GPUs.
Revised: Compared methods : On one DGX-2 server, we compare ElegantRL-podracer with RLlib [14], since both support multiple GPUs.

Original: We used PPO [21] in GPU-podracer, while inRLlib [13] we used the
Revised: ElegantRL-podracer used PPO [23] from ElegantRL [16], while in

Original:  Decentralized Distributed Proximal Policy Optimization (DD-PPO)[28] algorithm that scales well tomultiple GPUs. For fair comparison, we keep all adjustable parameters and computing resources thesame, such as the depth and width of neural networks, total training steps, number of workers, and GPU and CPU resources. Performance metrics : We employ two different metrics to evaluate the agent’s performance:
Revised: RLlib [14] we used the Decentralized Distributed Proximal Policy Optimization (DD-PPO)

Original: Tasks
Revised: Tasks

Original: State space S
Revised: State space S

Original: Action space A
Revised: Action space A

Original: Reward r ( s, a, s (cid:48) )
Revised: Reward r ( s, a, s (cid:48) )

Original: Ant [26, 16]
Revised: Ant [28, 18]

Original: Body height and rotation 8 controllable joints
Revised: Body height and rotation 8 controllable joints

Original: Alive bonus
Revised: Alive bonus

Original: Velocity and angular velocity
Revised: Velocity and angular velocity

Original: Running speed
Revised: Running speed

Original: Joint angles
Revised: Joint angles

Original: Standing and Heading
Revised: Standing and Heading

Original: Forces, etc.
Revised: Forces, etc.

Original: Contact forces
Revised: Contact forces

Original: Humanoid [26, 16]
Revised: Humanoid [28, 18]

Original: Body height and rotation 17 joints for MuJoCo
Revised: Body height and rotation 17 joints for MuJoCo

Original: Alive bonus
Revised: Alive bonus

Original: Velocity and angular velocity 21 joints for Isaac gym
Revised: Velocity and angular velocity 21 joints for Isaac gym

Original: Running speed
Revised: Running speed

Original: Joint angles
Revised: Joint angles

Original: Standing and Heading
Revised: Standing and Heading

Original: Forces, etc.
Revised: Forces, etc.

Original: Contact forces
Revised: Contact forces

Original: Stock trading [15]
Revised: Stock trading [17]

Original: Balance, Shares
Revised: Balance, Shares

Original: Buy
Revised: Buy

Original: Change of account value
Revised: Change of account value

Original: Close prices
Revised: Close prices

Original: Sell
Revised: Sell

Original: Technical indicators
Revised: Technical indicators

Original: Hold 
Revised: Hold algorithm that scales well to multiple GPUs. For fair comparison, we keep all adjustable parameters and computing resources the same, such as the depth and width of neural networks, total training steps/time, number of workers, and GPU and CPU resources. Speciﬁcally, we use a batch size of 1024, learning rate of 0.001, and a replay buffer size of 4096 across tasks. Performance metrics : We employ two different metrics to evaluate the agent’s performance:

Original: • Episodic reward vs. training time (wall-clock time) : we measure the episodic reward at different training time, which can be affected by the convergence rate, communication overhead, scheduling efﬁciency, etc.
Revised: • Episodic reward vs. training time (wall-clock time) : we measure the episodic reward at different training time, which can be affected by the convergence speed, communication overhead, scheduling efﬁciency, etc.

Original: • Episodic reward vs. training step : from the same testings, we also measure the episodic rewardat different training steps.
Revised: • Episodic reward vs. training step : from the same testings, we also measure the episodic reward at different training steps.

Original: This result can be used to investigate the massive parallel simulation capability of GPUs, and also check the algorithm’s convergence rate.
Revised: This result can be used to investigate the massive parallel simulation capability of GPUs, and also check the algorithm’s performance.

Original: During the training process, we store the snapshots of the policy networks at different training time,say approximately every 800 seconds.

Original: Then, for each snapshot of a policy network, we evaluate it32 times to obtain 32 episodic rewards and report thecorresponding average episodic reward andstandard deviation.
Revised: During the training process, we evaluate the agent 32 times to obtain 32 episodic rewards and report the average episodic reward and standard deviation.

Original: Performance analysis : For the four tasks in Fig.
Revised: For the four tasks in Fig.

Original: 4, we terminate the training processes at 10 , 000 s ( ≈2 . 8 hours), 40 , 000 s ( ≈ 11 . 1 hours) , 22 , 000 s ( ≈ 6 . 1 hours) and 28 , 000 s ( ≈ 7 . 8 hours), respectively.
Revised: 4, we terminate the training processes at 8 , 000 s ( ≈ 2 . 2 hours), 32 , 000 s ( ≈ 8 . 9 hours), 25 , 000 s ( ≈ 6 . 9 hours) and 9 , 000 s ( ≈ 2 . 5 hours), respectively.

Original: Correspondingly, GPU-podracer has run 2 .

Original: 4 × 10 7 steps, 3 . 3 × 10 7 steps, 18 × 10 7 steps and 28 × 10 7steps, respectively, while RLlib has run 1 . 6 × 10 7 steps, 2 .

Original: ×

Original: [-10 7 steps, 5 . 
-]
Revised: [-10 7 steps, 5 . 
-]

Original: × 10 7 steps and 11 .

Original: 5 × 10 7steps, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, we can see that given the same training time, GPU-podracer achieves higherepisodic reward than RLlib.
Revised: 5, we can see that given the same training time, ElegantRL-podracer achieves substantially higher episodic rewards than RLlib.

Original: Forthe Isaac Gymenvironments in particular, the corresponding episodicrewards have been tripled.
Revised: For Isaac Gym in particular, the corresponding episodic rewards have been nearly doubled.

Revised: In the task ant (Isaac Gym), RLlib needs approximately 7 . 0 hours to achieve a reward 9 , 000 , while Elegant-podracer only needs approximately 1 .

Revised: 4 hours to get the same reward, which is 5 × faster.

Original: {+We run 2 . 
+}
Revised: {+We run 2 . 
+}

Revised: × 10 7 steps, 2 . 5 × 10 7 steps, 4 × 10 8 steps and 8 × 10 8 steps, respectively.

Original: Take a closer look at Fig.
Revised: Take a closer look at Fig.

Original: 6, we can see that  RLlib has not convergedyet within the given training time (due to the limitation of cloud usage), e.g, ≈ 6 . 1 hours
Revised: 6, we can see that ElegantRL-podracer achieves higher episodic rewards than RLlib in all four tasks. A possible reason is the tournament-based ensemble training scheme guide a population of agents update toward a direction with higher rewards.

Original: and ≈ 7 .
Revised: [Figure]

Original: 8hours, respectively. However, we believe it is not practically viable to use over 20 hours to train an RLlib agent for those two tasks.
Revised: [Figure]

Original: 5.3 Stock Trading Task
Revised: 5.3 Stock Trading Task

Original: Finance is a promising and challenging real-world application of DRL algorithms [15].
Revised: Finance is a promising and challenging real-world application of DRL algorithms.

Original: We apply
Revised: We apply

Original: GPU-podracer to a stock trading task as an example  to show its potential in quantitative ﬁnance.
Revised: ElegantRL-podracer to a stock trading task as an example [17, 13] to show its potential in quantitative ﬁnance.

Original: Stock trading task : we aim to train a DRL agent that decides where to trade , at what price and whatquantity in a stock market, thus the objective of the problem is to maximize the expected return and minimize the risk.
Revised: Stock trading task : we aim to train a DRL agent that decides where to trade , at what price and what quantity in a stock market, thus the objective of the problem is to maximize the expected return and minimize the risk.

Original: We model the stock trading task as a Markov Decision Process (MDP) as in FinRL[15], where the state space, action space and reward function are given in Table 1.
Revised: We model the stock trading task as a Markov Decision Process (MDP) as in FinRL [17, 13], where the state space, action space and reward function are given in Table 1.

Original: Data pre-processing : We select the NASDAQ-100 constituent stocks as our stock pool and use theminute-level dataset for our experiment.
Revised: Data pre-processing : We select the NASDAQ-100 constituent stocks as our stock pool and use the minute-level dataset for our experiment.

Original: For the data preparation, we download the raw data from the
Revised: For the data preparation, we download the raw data from the

Original: Compustat database through the Wharton Research Data Services (WRDS) [22].
Revised: Compustat database through the Wharton Research Data Services (WRDS) [24].

Original: Next, we process itto an open-high-low-close-volume (OHLCV) format, and extract technical indicators.
Revised: Next, we process it to an open-high-low-close-volume (OHLCV) format and extract technical indicators.

Original: Finally, wefollow a training-backtesting pipeline and split the dataset into two sets: the data from 01/01/2016 to05/12/2019 for training, and the data from 05/13/2019 to 05/26/2021 for backtesting.
Revised: Finally, we follow a training-backtesting pipeline and split the dataset into two sets: the data from 01/01/2016 to 05/25/2020 for training, and the data from 05/26/2020 to 05/26/2021 for backtesting.

Original: Performance metrics : Weﬁrst evaluate training performance and testing performance, respectively.
Revised: Performance metrics : We evaluate trading performance and training performance, respectively.

Original: Then, ﬁve common metrics are used to quantify the trading performance:
Revised: Five common metrics are used to quantify the trading performance:

Original: • Cumulative return :  the ratio between the ﬁnal capital and the initial capital.
Revised: • Cumulative return : subtracting the initial value from the ﬁnal portfolio value, then dividing by the initial value.

Original: • Annual return  :the geometric average amount of money earned by the agent each year.• Annual volatility : the annual standard deviation of the return.
Revised: • Annual return and volatility : geometric average return in a yearly sense, and the deviation. • Maximum drawdown : the maximum observed loss from a historical peak to a trough of a portfolio, before a new peak is achieved.

Revised: It is an indicator of downside risk over a time period.

Original: • Sharpe ratio : the difference between theannualized risk-free rate and annualized return, standardized (divided) by the annualized volatility. • Max drawdown : the  maximum percentage loss during the trading period.
Revised: • Sharpe ratio : the average return earned in excess of the risk-free rate per unit of volatility. • Calmar ratio : the fund’s average compounded annual rate of return versus its maximum drawdown.

Original: For the training performance, we use the same metrics with the robotic control problems inthe Section 5.2.
Revised: For the training performance, we use the metric, episodic reward vs training time (wall-clock time), in Section 5.2.

Revised: We record the required training time for reaching a speciﬁc cumulative return.

Original: For the stock trading problem, we reserve a time periodthat is not used for training, butonly testing to evaluate generalization performance.
Revised: We reserve a time period not used for training but only testing to evaluate generalization performance for the stock trading problem.

Original: Since the agent cannot access the testing datasetduring the training, we store the model snapshots at different training times, say every 100 seconds.
Revised: Since the agent cannot access the testing dataset during the training, we store the model snapshots at different training times, say every 100 seconds.

Original: We then later we use each snapshot model to perform inference on the testing dataset to obtain thecumulative return.
Revised: Later, we use each snapshot model to perform inference on the testing dataset to obtain the cumulative return.

Original: Compared methods : We compare GPU-podracer with[-a vanilla implementation without ensembletraining, FinRL [15], Stable Baseline
 [19], and-] RLlib [13].
Revised: Compared methods : We compare ElegantRL-podracer with[-a vanilla implementation without ensembletraining, FinRL [15], Stable Baseline
 [19], and-] RLlib [14] with the same setup in Section 5.2.

Original: Invesco QQQ ETF is used to representthe market performance.
Revised: Invesco QQQ ETF is the benchmark to represent the market performance.

Original: There are in total 80 A100 GPUs assigned to our usage.
Revised: There are in total 80 A100 GPUs assigned to our usage.

Original: For the ensembletraining scheme of GPU-podracer, we use three DRL algorithms, PPO [21], SAC [10] and TD3[8] with ratio 1 : 1 : 1 . We used PPO [21] in vanilla GPU-podracer. For RLlib [13], we used the Decentralized Distributed Proximal Policy Optimization (DD-PPO) [28] algorithm that scales well tomultiple GPUs.
Revised: From Fig.

Original: For fair comparison, we keep all adjustable parameters and computing resources thesame, such as the depth and width of neural networks, total training steps, number of workers, and

Original: GPU and CPU resources.

Original: Trading performance : From Fig.

Original: 7, all DRL agents are able to achieve a better performance thanthe market baseline with respect to the cumulative return, demonstrating the algorithm’s effectiveness.
Revised: 7, all DRL agents can achieve a better performance than the market benchmark with respect to the cumulative return, demonstrating the algorithm’s effectiveness.

Original: According to Table 2, we observe that GPU-podracer has cumulative return 362.408%, annualreturn 111.549%, and Sharpe ratio 2.42, which outperforms other methods in all metrics in terms of
Revised: According to Table 2, we observe that ElegantRL-podracer has cumulative return 104.743%, annual return 103.591%, [Figure] [Figure] [Figure]

Revised: C u m u l a t

Revised: i v e R e t

Original: [Table]
Revised: u r n ElegantRL-Podracer RLlib QQQ   80 # GPUs T r a i n i n g T i m e ( s )

Revised: ElegantRL-Podracer-1.7 RLlib-1.

Revised: ElegantRL-Podracer-1.8 RLlib-1.

Revised: Cumul.

Revised: return

Revised: Annualreturn

Revised: Annualvolatility

Revised: Max.

Revised: drawdown

Revised: Sharperatio

Revised: Calmarratio

Revised: ElegantRL-podracer

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: proﬁtability.
Revised: [Table] [Table] [Table] [Table]

Revised: Invesco QQQ ETF

Revised: [Table]

Original: Moreover, GPU-podracer also shows outstanding stability during the backtesting period:it achieves a low annual volatility 33.427% and max drawdown -15.874%.
Revised: [Table] [Table] and Sharpe ratio 2.20, which outperforms RLlib substantially. However, ElegantRL-podracer is not as stable as RLlib during the backtesting period: it achieves an annual volatility 35.357%, max.

Original: The high proﬁtability andstability of GPU-podracer demonstrate the effectiveness of our tournament-based ensemble training,which can train a more proﬁtable and robust agent in thecomplex stock market.
Revised: drawdown -17.187%, and Calmar ratio 6.02. There are two possible reasons to account for such an instability: 1) the reward design in the stock trading environment is mainly related to the cumulative return, thus leading the agent to take less care of the risk; 2) ElegantRL-podracer holds a large amount of funds around 2021-03, as shown in Fig. 7, which naturally leads to a larger slip.

Original: Training efﬁciency : We compare the training efﬁciency of GPU-podracer with RLlib [13] on avarying number of GPUs, i.e., 8, 16, 32, and 80 GPUs.
Revised: We compare the training performance on a varying number of GPUs, i.e., 8, 16, 32, and 80. We measure the required training time to obtain two cumulative returns 1.7 and 1.8, respectively.

Original: In Fig.
Revised: In Fig.

Original: 7, both GPU-podracer and RLlib [13]can achievea higher cumulative return at the same training time as the number of GPUs increases,which directly demonstrates the advantage of cloud computing resources on the DRL training.
Revised: 7, both ElegantRL-podracer and RLlib [14] requires less training time to achieve the same cumulative return as the number of GPUs increases, which directly demonstrates the advantage of cloud computing resources on the DRL training.

Original: For GPU-podracer, the speed-up of increasing the numbers of GPUs scales near linearly . For GPUpodracer with 80 GPUs, it has a much steeper curve and reaches a cumulative return of 4.0 at 800seconds.
Revised: For ElegantRL-podracer with 80 GPUs, it requires (1900s, 2200s) to reach cumulative returns of 1.7 and 1.8.

Original: GPU-podracer with 32 and 16 GPUs need 2 , 200 seconds and 3 , 200 seconds to achieve thesame cumulative return.
Revised: ElegantRL-podracer with 32 and 16 GPUs need (2400s, 2800s) and (3400s, 4000s) to achieve the same cumulative returns.

Original: Such a linear scaling demonstrates the high scalability of GPU-podracerand the effectiveness of our cloud-oriented optimizations.
Revised: It demonstrates the high scalability of ElegantRL-podracer and the effectiveness of our cloud-oriented optimizations.

Original: For the experiments using RLlib [13],increasing the number of GPUs produces relatively similar cumulative return curves and not muchspeed-up.
Revised: For the experiments using RLlib [14], increasing the number of GPUs does not lead to much speed-up.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: In this paper, we have introduced GPU-podracer , a scalable and elastic library for cloud-native deepreinforcement learning.
Revised: In this paper, we have introduced ElegantRL-podracer , a scalable and elastic library for cloud-native deep reinforcement learning.

Original: To efﬁciently utilize millions of GPU cores for DRL training, we ﬁrst proposea tournament-based ensemble training scheme to orchestrate the training process on hundreds of GPUs, and then enable massively parallel simulation on thousands of GPU cores in a single GPU.
Revised: To efﬁciently utilize millions of GPU cores for DRL training, we ﬁrst propose a tournament-based ensemble training scheme to orchestrate the training process on hundreds of GPUs, and then enable massively parallel simulation on thousands of GPU cores in a single GPU.

Original: Moreover, we follow the cloud-native paradigm to schedule the training of DRL agents by adheringto containerization, microservices, and MLOps.
Revised: Moreover, we follow the cloud-native paradigm to schedule the training of DRL agents by adhering to containerization, microservices, and MLOps.

Original: Thus, GPU-podracer realizes the design principles inthe respect of scaling-out , efﬁciency , and accessibility .
Revised: Thus, ElegantRL-podracer realizes the design principles in the respect of scaling-out , efﬁciency , and accessibility .

Original: By presenting GPU-podracer to the DRL community, we hope that GPU-podracer can help addressthe data collection bottleneck using the manycore GPU architecture and apply DRL algorithms tocomplex real-world problems. 
Revised: By presenting ElegantRL-podracer to the DRL community, we hope that ElegantRL-podracer can help address the data collection bottleneck using the manycore GPU architecture and apply DRL algorithms to complex real-world problems. Acknowledgement This research used computational resources of the GPU cloud platform [27] provided by the IDEA Research institute.