L ARGE B ATCH S IMULATION FOR D EEP R EINFORCEMENT L EARNING
A BSTRACT
We accelerate deep reinforcement learning-based training in visually complex 3D environments by two orders of magnitude over prior work, realizing end-to-end training speeds of over 19,000 frames of experience per second on a single GPU and up to 72,000 frames per second on a single eight-GPU machine.
The key idea of our approach is to design a 3D renderer and embodied navigation simulator around the principle of “batch simulation”: accepting and executing large batches of requests simultaneously.
Beyond exposing large amounts of work at once, batch simulation allows implementations to amortize in-memory storage of scene assets, rendering work, data loading and synchronization costs across many simulation requests, dramatically improving the number of simulated agents per
GPU and overall simulation throughput.
To balance DNN inference and training costs with faster simulation, we also build a computationally efﬁcient policy DNN that maintains high task performance and modify training algorithms to maintain sample efﬁciency when training with large mini-batches.
By combining batch simulation and DNN performance optimizations, we demonstrate that PointGoal navigation agents can be trained in complex 3D environments on a single GPU in .5 days to 97% of the accuracy of agents trained on a prior state-of-the-art system using a 64-GPU cluster over three days.
We provide open-source reference implementations of our batch 3D renderer and simulator to facilitate incorporation of these ideas into RL systems
I NTRODUCTION
Speed matters.
It is now common for modern reinforcement learning (RL) algorithms leveraging deep neural networks (DNNs) to require billions of samples of experience from simulated environments (Wijmans et al., 2020;
Petrenko et al., 2020; OpenAI et al., 2019; Silver et al., 2017; Vinyals et al., 2019).
For embodied AI tasks such as visual navigation, where the ultimate goal for learned policies is deployment in the real world, learning from realistic simulations is important for successful transfer of learned policies to physical robots.
In these cases simulators must render detailed 3D scenes and simulate agent interaction with complex environments (Kolve et al., 2017; Dosovitskiy et al., 2017; Savva et al., 2019; Xia et al., 2020; Gan et al., 2020).
Evaluating and training a DNN on billions of simulated samples is computationally expensive.
For instance, the DD-PPO system (Wijmans et al., 2020) used 64 GPUs over three days to learn from 2.5 billion frames of experience and achieve near-perfect PointGoal navigation in 3D scanned environments of indoor spaces. At an even larger distributed training scale, OpenAI Five used over 50,000 CPUs and 1000 GPUs to train Dota 2 agents (OpenAI et al., 2019).
Unfortunately, experiments at this scale are out of reach for most researchers. This problem will only grow worse as the ﬁeld explores more complex tasks in more detailed environments.
Many efforts to accelerate deep RL focus on improving the efﬁciency of DNN evaluation and training – e.g ., by “centralizing” computations to facilitate efﬁcient batch execution on GPUs or TPUs (Espeholt et al., 2020; Petrenko et al., 2020) or by parallelizing across GPUs (Wijmans et al., 2020).
However, most RL platforms still accelerate environment simulation by running many copies of off-the-shelf, unmodiﬁed simulators , such as simulators designed for video game engines (Bellemare et al., 2013; Kempka et al., 2016; Beattie et al., 2016; Weihs et al., 2020), on large numbers [Figure] of CPUs or GPUs.
This approach is a simple and productive way to improve simulation throughput, but it makes inefﬁcient use of computation resources.
For example, when rendering complex environments (Kolve et al., 2017; Savva et al., 2019; Xia et al., 2018), a single simulator instance might consume gigabytes of GPU memory, limiting the total number of instances to far below the parallelism afforded by the machine.
Further, running many simulator instances (in particular when they are distributed across machines) can introduce overhead in synchronization and communication with other components of the RL system.
Inefﬁcient environment simulation is a major reason RL platforms typically require scale-out parallelism to achieve high end-to-end system throughput.
In this paper, we crack open the simulation black box and take a holistic approach to co-designing a 3D renderer, simulator, and RL training system. Our key contribution is batch simulation for RL: designing high-throughput simulators that accept large batches of requests as input (aggregated across different environments, potentially with different assets) and efﬁciently execute the entire batch at once.
Exposing work en masse facilitates a number of optimizations: we reduce memory footprint by sharing scene assets (geometry and textures) across rendering requests (enabling orders of magnitude more environments to be rendered simultaneously on a single GPU), amortize rendering work using GPU commands that draw triangles from multiple scenes at once, hide latency of scene
I/O, and exploit batch transfer to reduce data communication and synchronization costs between the simulator, DNN inference, and training.
To further improve end-to-end RL speedups, the DNN workload must be optimized to match high simulation throughput, so we design a computationally efﬁcient policy DNN that still achieves high task performance in our experiments.
Large-batch simulation increases the number of samples collected per training iteration, so we also employ techniques from large-batch supervised learning to maintain sample efﬁciency in this regime.
We evaluate batch simulation on the task of PointGoal navigation (Anderson et al., 2018) in 3D scanned Gibson and Matterport3D environments, and show that end-to-end optimization of batched rendering, simulation, inference, and training yields a 110 × speedup over state-of-the-art prior systems, while achieving 97% of the task performance for depth-sensor-driven agents and 91% for RGB-camera-driven agents.
Concretely, we demonstrate sample generation and training at over 19,000 frames of experience per second on a single GPU.
In real-world terms, a single GPU is capable of training a virtual agent on 26 years of experience in a single day.
This new performance regime signiﬁcantly improves the accessibility and efﬁciency of RL research in realistic 3D environments, and opens new possibilities for more complex embodied tasks in the future.
R ELATED W ORK
Systems for high-performance RL.
Existing systems for high-performance RL have primarily focused on improving the efﬁciency of DNN components of the workload (policy inference and optimization) and use a simulator designed for efﬁcient single agent simulation as a black box.
For example, Impala and Ape-X used multiple worker processes to asynchronously collect experience for a centralized learner (Espeholt et al., 2018; Horgan et al., 2018).
SEED RL and Sample Factory built upon this idea and introduced inference workers that centralize network inference, thereby allowing it to be accelerated by GPUs or TPUs (Espeholt et al., 2020; Petrenko et al., 2020).
DD-PPO proposed a synchronous distributed system for similar purposes (Wijmans et al., 2020).
A number of efﬁcient implementations of these ideas have been proposed as part of RL frameworks or in other deep learning libraries (Liang et al., 2018; Stooke & Abbeel, 2019; K¨uttler et al., 2019).
We extend the idea of centralizing inference and learning to simulation by cracking open the simulator black box and designing a new simulation architecture for RL workloads.
Our large-batch simulator is a drop-in replacement for large numbers of (non-batched) simulation workers, making it synergistic with existing asynchronous and synchronous distributed training schemes.
It reduces the number of processes and communication overhead needed for asynchronous methods and eliminates separate simulation worker processes altogether for synchronous methods.
We demonstrate this by combining our system with DD-PPO (Wijmans et al., 2020).
Concurrently with our work, CuLE, a GPU-accelerated reimplementation of the Atari Learning Environment (ALE), demonstrates the beneﬁts of centralized batch simulation (Dalton et al., 2020).
While both our work and CuLE enable wide-batch execution of their respective simulation workloads, our focus is on high-performance batch rendering of complex 3D environments. This involves optimizations (GPU-driven pipelined geometry culling, 3D asset sharing, and asynchronous data transfer) not addressed by CuLE due to the simplicity of rendering Atari-like environments.
Additionally, like CuLE, we observe that the large training batches produced by batch simulation reduce RL sample efﬁciency. Our work goes further and leverages large-batch optimization techniques from the supervised learning literature to mitigate the loss of sample efﬁciency without shrinking batch size.
Large mini-batch optimization.
A consequence of large batch simulation is that more experience is collected between gradient updates.
This provides the opportunity to accelerate learning via large mini-batch optimization.
In supervised learning, using large mini-batches during optimization typically decreases the generalization performance of models (Keskar et al., 2017).
Goyal et al.
(2017) demonstrated that model performance can be improved by scaling the learning rate proportionally with the batch size and “warming-up” the learning rate at the start of training.
You et al.
(2017) proposed an optimizer modiﬁcation, LARS, that adaptively scales the learning rate at each layer, and applied it to SGD to improve generalization further.
In reinforcement learning and natural language processing, the Adam optimizer (Kingma & Ba, 2015) is often used instead of SGD.
Lamb
(You et al., 2020) combines LARS (You et al., 2017) with Adam (Kingma & Ba, 2015).
We do not ﬁnd that large mini-batch optimization harms generalization in reinforcement learning, but we do ﬁnd it decreases sample efﬁciency.
We adapt the techniques proposed above – learning rate scaling (You et al., 2017) and the Lamb optimizer (You et al., 2020) – to improve sample efﬁciency.
Simulators for machine learning.
Platforms for simulating realistic environments for model training fall into two broad categories: those built on top of pre-existing game engines (Kolve et al., 2017; Dosovitskiy et al., 2017; Lee et al., 2019; Gan et al., 2020; James et al., 2020), and those built from scratch using open-source 3D graphics and physics libraries (Savva et al., 2017; 2019; Xia et al., 2018; 2020; Xiang et al., 2020; Zeng et al., 2020).
While improving simulator performance has been a focus of this line of work, it has been evaluated in a narrow sense ( i.e . frame rate benchmarks for predetermined agent trajectories), not accounting for the overall performance of end-to-end RL training.
We instead take a holistic approach to co-design rendering and simulation modules and their interfaces to the RL training system, obtaining signiﬁcant gains in end-to-end throughput over the state of the art.
S YSTEM D ESIGN & I MPLEMENTATION
Batch simulation accelerates rollout generation during RL training by processing many simulated environments simultaneously in large batches.
Fig.
2 illustrates how batch simulation interacts with policy inference to generate rollouts.
Simulation for sensorymotor agents, such as the PointGoal navigation task targeted by our implementation, can be separated into two tasks: determining the next environment state given an agent’s actions and rendering its sensory observations.
Therefore, our design utilizes two components: a batch simulator that performs geodesic distance and navigation mesh (Snook, 2000) computations on the CPU, and a batch renderer that renders complex 3D environments on the GPU.
During rollout generation, batches of requests are passed between these components – given N agents, the simulator produces a batch of N environment states.
Next, the renderer processes the batch of environment states by simultaneously rendering N frames and exposing the result directly in GPU memory.
Agent observations (from both the simulator and the renderer) are then provided as a batch to policy inference to determine the next actions for the N agents.
Learning +
Inference
[Figure]
Renderer
[Figure]
The key idea is that the batch simulator and renderer implementations (in addition to the DNN workload) take responsibility for their own parallelization .
Large batch sizes (values of N on the order of hundreds to thousands of environments) provide opportunities for implementations to efﬁciently utilize parallel execution resources (e.g., GPUs) as well as amortize processing, synchronization and data communication costs across many environments.
The remainder of this section describes the design and key implementation details of our system’s batch simulator and batch renderer, as well as contributions that improve the efﬁciency of policy inference and optimization in this regime.
3. B ATCH E NVIRONMENT S IMULATION
Our CPU-based batch simulator executes geodesic distance and navigation mesh computations in parallel for a large batch of environments.
Due to differences in navigation mesh complexity across environments, the time to perform simulation may differ per environment.
This variance is the source of workload imbalance problems in parallel synchronous RL systems (Wijmans et al., 2020;
Savva et al., 2019) and one motivation for recent asynchronous designs (Petrenko et al., 2020; Espeholt et al., 2020; 2018).
To ensure good workload balance, our batch simulator operates on large batches that contain signiﬁcantly more environments than the number of available CPU cores and dynamically schedules work onto cores using a pool of worker threads (simulation for each environment is carried out sequentially).
Worker threads report simulation results into a designated per-environment slot in a results buffer that is communicated to the renderer via a single batched request when all environment simulation for a batch is complete.
To minimize CPU memory usage, the simulator only loads navigation meshes and does not utilize the main rendering assets.
3. B ATCH R ENDERING
A renderer for producing RL agent observations in scanned real-world environments must efﬁciently synthesize many low-resolution renderings (e.g., 64 × 64 pixels) of scenes featuring high-resolution textures and complex meshes.
3 Low-resolution output presents challenges for GPU acceleration.
Rendering images one at a time produces too little rendering work to efﬁciently utilize a modern GPU rendering pipeline’s parallel processing resources.
Rendering many environments concurrently but individually (e.g., from different worker threads or processes) exposes more rendering work to the GPU, but incurs the overhead of sending the GPU many ﬁne-grained rendering commands.
To address the problem of rendering many small images efﬁciently, our renderer combines the GPU commands required to render observations for an entire simulation batch of N environments into a single rendering request to the GPU – effectively drawing the entire batch as a single large frame (individual environment observations are tiles in the image).
This approach exposes large amounts of rendering work to the GPU and amortizes GPU pipeline conﬁguration and rendering overhead over an entire batch.
Our implementation makes use of modern GPU pipeline features (Khronos Group, 2017) that allow rendering tasks that access different texture and mesh assets to proceed as part of a single large operation (avoiding GPU pipeline ﬂushes due to pipeline state reconﬁguration). Scene asset sharing.
Efﬁciently utilizing a GPU requires batches to be large (we use N up to 1024). However, geometry and texture assets for a single environment may be gigabytes in size, so naively loading unique assets for each environment in a large batch would exceed available GPU memory.
Our implementation allows multiple environments in a batch to reference the same 3D scene assets in GPU memory.
Speciﬁcally, our system materializes K unique assets in GPU memory ( K (cid:28) N ) and constructs batches of N environments that reference these assets.
Asset reuse decreases the diversity of training experiences in a batch, so to preserve diversity we limit the ratio of N to K in any one batch to 32, and continuously rotate the set of K assets in GPU memory.
The renderer refreshes the set of K assets by asynchronously loading new scene assets into GPU memory during the main rollout generation and learning loop.
As episodes complete, new environments are constructed to reference the newly loaded assets, and assets no longer referenced by active environments are removed from GPU memory.
This design allows policy optimization to learn from an entire dataset of assets without exceeding GPU memory or incurring the latency costs of frequent asset loading.
Pipelined geometry culling. When rendering detailed geometry to low-resolution images, most scene triangles cover less than one pixel.
As a result, rendering performance is determined by the rate the GPU’s rasterization hardware processes triangles, not the rate the GPU can shade covered pixels.
To reduce the number of triangles the GPU pipeline must process, the renderer uses idle GPU cores to identify and discard geometry that lies outside the agent’s view—a process known as frustum culling (Akenine-M¨oller et al., 2018).
Our implementation pipelines frustum culling operations (implemented using GPU compute shaders) with rendering for different environments in a batch.
This pipelined design increases GPU utilization by concurrently executing culling work on the GPU’s programmable cores and rendering work on the GPU’s rasterization hardware.
3. P OLICY DNN A RCHITECTURE
High-throughput batch simulation creates a need for high-throughput policy DNN inference.
Therefore, we develop a policy DNN architecture designed to achieve an efﬁcient balance between high task performance and low computational cost.
Prior work in PointGoal navigation (Wijmans et al., 2020) used a policy DNN design where a visual encoder CNN processes an agent’s visual sensory information followed by an LSTM (Hochreiter & Schmidhuber, 1997) that determines the policy’s actions.
Our policy DNN uses this core design augmented with several performance optimizations.
First, we reduce DNN effective input resolution from 128 × 128 (Wijmans et al., 2020) to 64 × 64.
Beyond this simple optimization, we choose a shallow visual encoder CNN – a nine-layer ResNet (He et al., 2016) (ResNet18 with every other block removed), rather than the 50 layer (or larger)
ResNets used by prior work.
To counteract reduced task performance from the ResNet’s relatively low capacity, all stages include Squeeze-Excite (SE) blocks (Hu et al., 2018) with r =16 .
Additionally, we use a SpaceToDepth stem (Ridnik et al., 2020), which we ﬁnd performs equally to the standard
Conv+MaxPool stem while using less GPU memory and compute.
Finally, we avoid the use of normalization layers in the ResNet as these require spatial reductions over the feature maps, preventing layer-fusion optimizations.
Instead, the CNN utilizes Fixup Initialization (Zhang et al., 2019) to improve training stability.
Fixup Initialization replaces expensive normalization layers after each convolution with cheap elementwise multiplication and addition.
3. L ARGE M INI -B ATCH P OLICY O PTIMIZATION
In on-policy reinforcement learning, policy optimization utilizes trajectories of experience to reduce bias and for backpropagation-through-time.
When generating trajectories of length L with a simulation batch size of N , a rollout will have N × L steps of experience.
Therefore, a consequence of simulation with large N is that more experience is collected per rollout.
Large N presents the opportunity to utilize large mini-batches to improve the throughput of policy optimization; however, throughput must be balanced against generalization and sample efﬁciency to ensure that reduced task performance does not offset the throughput gains.
Although large minibatch training is known to hurt generalization in supervised learning (Keskar et al., 2017), we do not see evidence of this for RL.
Conversely, we doA ﬁnd that sample efﬁciency for PointGoal navigation is harmed by naively increasing N .
Fortunately, we are able to mitigate this loss of sample efﬁciency using techniques for improving generalization from the large mini-batch optimization literature.
First, we scale the learning rate by (cid:113) BB base , where B base =256 and B , the training batch size, is N × L divided by the number of mini-batches per training iteration.
We ﬁnd it beneﬁcial to use the scaled learning rate immediately instead of ‘warming-up’ the learning rate (Goyal et al., 2017).
Second, we use and adapt the Lamb optimizer (You et al., 2020).
Lamb is a modiﬁcation to Adam (Kingma & Ba, 2015) that applies LARS (You et al., 2017) to the step direction estimated by Adam to better handle high learning rates.
Since the Adam optimizer is often used with PPO (Schulman et al.,
, Lamb is a natural choice.
Given the Adam step direction s ( k ) t for weights θ ( [Equation] t
[Equation]
where η t is the learning rate and λ is the weight decay coefﬁcient.
We set φ ( || θ ( k ) t || ) as min {|| θ ( k ) t || , 10 . 0 } and introduce an additional clip on the trust ratio r ( k ) t :
[Equation]
We ﬁnd the exact value of ρ to be ﬂexible (we observed similar training with ρ ∈ { 10 − 2 , 10 − 3 , 10 − 4 } ) and also observed that this clip is only inﬂuential at the start of training, suggesting that there is an initialization scheme where it is unnecessary.
R ESULTS
We evaluate the impact of our contributions on end-to-end training speed and task performance by training PointGoal navigation agents in the complex Gibson (Xia et al., 2018) and Matterport3D (Chang et al., 2017) environments.
The fastest published end-to-end training performance in these environments is achieved with the synchronous RL implementation presented with DD-PPO (Wijmans et al., 2020).
Therefore, both our implementation and the baselines we compare against are synchronous PPO-based RL systems.
4. E XPERIMENTAL S ETUP
PointGoal navigation task.
We train and evaluate agents via the same procedure as Wijmans et al.
(2020): agents are trained for PointGoalNav (Anderson et al., 2018) with either a Depth sensor or an RGB camera.
Depth agents are trained on Gibson-2plus (Xia et al., 2018) and, consistent with
Wijmans et al. (2020), RGB agents are also trained on Matterport3D (Chang et al., 2017). RGB camera simulation requires textures for the renderer, increasing the GPU memory consumed by each scene signiﬁcantly. Both classes of agent are trained on 2.5 billion simulated samples of experience.
Agents are evaluated on the Gibson dataset (Xia et al., 2018). We use two metrics: Success, whether or not the agent reached the goal, and SPL (Anderson et al., 2018), a measure of both Success and efﬁciency of the agent’s path. We perform policy evaluation using Habitat-Sim (Savva et al., 2019), unmodiﬁed for direct comparability to prior work. Batch Processing Simulator ( BPS ).
We provide an RL system for learning PointGoalNav built around the batch simulation techniques and system-wide optimizations described in Section 3.
The remainder of the paper refers to this system as BPS (Batch Processing Simulator).
To further accelerate the policy DNN workload, BPS uses half-precision inference and mixed-precision training.
Baseline.
The primary baseline for this work is Wijmans et al. (2020)’s open-source PointGoalNav implementation, which uses Habitat-Sim (Savva et al., 2019) – the prior state of the art in highperformance simulation of realistic environments such as Gibson.
Unlike BPS , multiple environments are simulated simultaneously using parallel worker processes that render frames at 256 ×pixels before downsampling to 128 × 128 for the visual encoder. The fastest published conﬁguration uses a ResNet50 visual encoder.
Subsequent sections refer to this implementation as W IJMANS 20 .
Ablations.
As an additional baseline, we provide W IJMANS ++ , which uses the optimized SEResNet9-based policy DNN (including performance optimizations and resolution reduction relative to W IJMANS 20 ) developed for BPS , but otherwise uses the same system design and simulator as W IJMANS 20 (with a minor modiﬁcation to not load textures for Depth agents).
W IJMANS ++ serves to isolate the impact of two components of BPS :
ﬁrst, the low-level DNN efﬁciency improvements, and, more importantly, the performance of batch simulation versus W IJMANS 20 ’s independent simulation worker design.
Additionally, to ablate the effect of our encoder CNN architecture optimizations, we include a variant of BPS , BPS - R 50 , that uses the same ResNet50 visual encoder and input resolution as W IJMANS 20 , while maintaining the other of optimizations BPS .
Multi-GPU training.
To support multi-GPU training, all three systems replace standard PPO with
DD-PPO (Wijmans et al., 2020).
DD-PPO scales rollout generation and policy optimization across all available GPUs, scaling the number of environments simulated and the number of samples gathered between training iterations proportionally.
We report results with eight GPUs.
[Table]
[Table]
Determining batch size.
The per-GPU batch size, N , controls a trade-off between memory usage, sample efﬁciency, and speed.
For BPS , N designates the batch size for simulation, inference, and training.
For W IJMANS 20 and W IJMANS ++ , N designates the batch size for inference and training, as well as the number of simulation processes.
W IJMANS 20 sets N = 4 for consistency with Wijmans et al.
To maximize performance of single-GPU runs, BPS uses the largest batch size that ﬁts in GPU memory, subject to the constraint that no one scene asset can be shared by more than 32 environments in the batch.
In eight-GPU conﬁgurations, DD-PPO scales the number of parallel rollouts with the number of GPUs, so to maintain reasonable sample efﬁciency BPS limits perGPU batch size to N = 128 , with K = 4 active scenes per GPU.
W IJMANS ++ Depth experiments use N =64 (limited by system memory due to N separate processes running Habitat-Sim).
Batch size in
W IJMANS ++ RGB experiments is limited by GPU memory ( N ranges from 6 to 20 depending on the
GPU).
Appendix B provides the batch sizes used in all experiments.
Benchmark evaluation.
We report end-to-end performance benchmarks in terms of average frames per second (FPS) achieved by each system.
We measure FPS as the number of samples of experience processed over 16,000 inference batches divided by the time to complete rollout generation and training for those samples.
In experiments that run at 128 × 128 pixel sensor resolution, rendering occurs at 256 × 256 and is downsampled for the policy DNN to match the behavior of W IJMANSregardless of system, while 64 × 64 resolution experiments render without downsampling.
Results are reported across three models of NVIDIA GPUs: Tesla V100, GeForce RTX 2080Ti, and GeForce
RTX 3090. (The different GPUs are also accompanied by different CPUs, see Appendix C.)
4. E ND - TO -E ND T RAINING S PEED
Single-GPU performance.
On a single GPU, BPS trains agents 45 × (9000 vs. 190 FPS, Tesla V100) to 110 × (19900 vs. 180 FPS, RTX 3090) faster than W IJMANS 20 (Table 1).
The greatest speedup was achieved using the RTX 3090, which trains Depth agents at 19,900 FPS and RGB agents at 13,300 FPS – a 110 × and 95 × increase over W IJMANS 20 , respectively.
This 6000 FPS performance drop from Depth to RGB is not caused by the more complex rendering workload, because the addi- [Figure] i [Figure]
e r i s
[Figure]
r ) [Figure] [Figure] tional cost of fetching RGB textures is masked by the dominant cost of geometry processing.
Instead, due to memory constraints, BPS must reduce the batch size ( N ) for RGB tasks, reducing the performance of all components (further detail in Section 4.4) To assess how much of the BPS speedup is due to the SE-ResNet9 visual encoder and lower input resolution, we also compare BPS - R 50 and W IJMANS 20 , which have matching encoder architecture and resolution. For Depth agents training on the the RTX 3090, BPS - R 50 still achieves greater than 10 × performance improvement over W IJMANS 20 (2,300 vs. 180 FPS), demonstrating the beneﬁts of batch simulation even in DNN heavy workloads.
BPS - R 50 is only 6 × faster than W IJMANSon the RTX 2080Ti, since the ResNet50 encoder’s larger memory footprint requires batch size to be reduced from N = 128 on the RTX 3090 (24 GB RAM) to N = 64 on the RTX 2080Ti (11 GB RAM). Similarly, increasing DNN input resolution increases memory usage, forcing batch size to be decreased and reducing performance (Table A1). The BPS batch simulation architecture is signiﬁcantly faster than the W IJMANS ++ design that uses multiple worker processes.
When training Depth agents, BPS outperforms W IJMANS ++ by 4.5 × to 7.8 × , with a greater speedup of 6 × to 13 × for RGB agents.
Since BPS and W IJMANS ++ use the same policy DNN and input resolution, this comparison isolates the performance advantage of batch simulation and rendering against an optimized version of the multiple-worker-process-based design:
W IJMANS ++ is up to 15 × faster than W IJMANS 20 .
The relative speedup of BPS for RGB agents is larger because W IJMANS ++ does not share environment assets between simulator instances. Textures needed for RGB rendering signiﬁcantly increase the memory footprint of each simulator instance and limit W IJMANS ++ to as few as N =6 workers (compared to N =64 for Depth agents).
Conversely, BPS shares 3D assets across environments and maintains a batch size at least N =128 for RGB agents.
Multi-GPU performance.
BPS achieves high end-to-end throughput when running in eight-GPU conﬁgurations: up to 72,000 FPS for Depth agents on eight RTX 2080Ti.
Relative to W IJMANS 20 ,
BPS is 29 × to 34 × faster with eight Telsa V100s and 45 × faster with eight RTX 2080Ti.
These speedups are lower than the single-GPU conﬁgurations, because BPS reduces the per-GPU batch size in eight-GPU conﬁgurations to avoid large aggregate batches that harm sample efﬁciency.
This leads to imperfect multi-GPU scaling for BPS : for Depth agents, each RTX 2080Ti is approximately 4000 FPS slower in an eight-GPU conﬁguration than in a single-GPU conﬁguration.
Eight-GPU scaling for Depth is lower on the Tesla V100s (3.7 × ) compared to the 2080Ti (5.6 × ) because larger batch sizes are needed to utilize the large number of parallel compute units on the Tesla V100.
4. P OLICY T ASK P ERFORMANCE
To understand how the system design and visual encoder architecture of BPS impact learning, we evaluate the task performance of agents trained with BPS in an eight-GPU conﬁguration with aggregate batch size of N =1024.
For Depth agents, the reduction in encoder CNN depth results in a 1% and 3% decrease in SPL on Val and Test respectively with a negligible Success change on Val and a 0.9 Success decrease on Test (Table 2, row 1 vs. 2 ).
For RGB agents, BPS suffers a performance loss of 3.8/1.3 SPL/Success on Val and 8.3/2.0 SPL/Success on Test (Table 2, row 3 vs. 4 ).
Despite this performance reduction, the RGB agent trained by BPS would have won the 2019 Habitat challenge by 4 SPL and is only beaten by W IJMANS 20 ’s ResNet50-based policy on Test.
SPL vs. training time.
BPS signiﬁcantly outperforms the baselines in terms of wall-clock training time to reach a given SPL.
After 10 hours of training on a single RTX 3090, BPS reaches over 80% SPL (on Val) while W IJMANS 20 and W IJMANS ++ reach only 40% and 65% SPL respectively (Fig. 3). Furthermore, BPS converges within 1% of peak SPL at approximately 20 hours; conversely, neither baseline reaches convergence within 48 hours.
BPS converges to a lower ﬁnal SPL in Fig.
3 than Table 2, likely due to the tested single-GPU conﬁguration differing in batch size and scene asset swapping frequency compared to the eight-GPU conﬁguration used to produce Table 2. Effect of batch size. The end-to-end training efﬁciency of BPS is dependent on batch size ( N ): larger N will increase throughput and reduce wall-clock time to reach a given number of samples, but may harm sample efﬁciency and ﬁnal task performance at convergence.
We evaluate this relationship by training Depth agents with BPS across a range of N .
As shown in Fig.
4, all experiments converge within 1% of the peak SPL achieved; however, N =256 halves total throughput compared to N =1024 (the setting used elsewhere in the paper for eight-GPU conﬁgurations).
At the high end, N =4096 yields slightly worse SPL than N =1024 and is only 20% faster.
Larger batch sizes also require more memory for rollout storage and training, which is prohibitive for RGB experiments that require signiﬁcant GPU memory for texture assets.
In terms of sample efﬁciency alone, Fig.
A shows that smaller batch sizes have a slight advantage (without considering training speed).
4. R UNTIME B REAKDOWN
[Table]
(Depth) (RGB) (Depth) (Depth)
Fig.
5 provides a breakdown of time spent in each of the main components of the BPS system ( µ s per frame).
Nearly 60% of BPS runtime on the RTX 3090 GPU (for both Depth and RGB ) is spent in DNN inference and training, even when rendering complex 3D environments and using a small, low-cost policy DNN.
This demonstrates the high degree of simulation efﬁciency achieved by BPS .
Furthermore, the results in Table A2 for BPS - R 50 show that, with the larger visual encoder, over 90% of per-frame time (on Depth tasks) is spent in the DNN workload (70% on learning).
Batch size ( N ) heavily impacts DNN performance. DNN operations for Depth ( N =1024) are 2 × faster than RGB ( N =256) on the RTX 3090, because RGB must use a smaller batch size to ﬁt texture assets in GPU memory.
The larger batch size improves GPU utilization for all system components.
A similar effect is visible when comparing the single-GPU and eight-GPU V100 breakdowns.
BPS reduces the per-GPU batch size from N =1024 to N =128 in eight-GPU experiments to maintain an aggregate batch size of 1024 for sample efﬁciency.
Further work in policy optimization to address this learning limitation would improve multi-GPU scaling by allowing larger aggregate batch sizes.
D ISCUSSION
We demonstrated that architecting an RL training system around the idea of batch simulation can accelerate learning in complex 3D environments by one to two orders of magnitude over prior work.
With these efﬁciency gains, agents can be trained with billions of simulated samples from complex environments in about a day using only a single GPU.
We believe these fast turnaround times stand to make RL in realistic simulated environments accessible to a broad range of researchers, increase the scale and complexity of tasks and environments that can be explored, and facilitate new studies of how much visual realism is needed to learn a given task (e.g., dynamic lighting, shadows, custom augmentations).
To facilitate such efforts, our system is available open-source at https://github.com/shacklettbp/bps-nav .
More generally, this work demonstrates the value of building RL systems around components that have been speciﬁcally designed for RL workloads, not repurposed from other application domains.
We believe this philosophy should be applied to other components of future RL systems, in particular to new systems for performing physics simulation in complex environments.
A CKNOWLEDGMENTS
This work was supported in part by NSF, DARPA, ONR YIP, ARO PECASE, Intel and Facebook. EW is supported in part by an ARCS fellowship. We thank NVIDIA for GPU equipment donations. We also thank the Habitat team for helpful discussions and their support of this project
A A DDITIONAL R ESULTS A. F LEE AND E XPLORE T ASKS ON AI2-THOR D ATASET
To demonstrate batch simulation and rendering on additional tasks besides PointGoal navigation,
BPS also supports the Flee (ﬁnd the farthest valid location from a given point) and Explore (visit as much of an area as possible) tasks.
We evaluate BPS ’s performance on these tasks on the AI2-THOR (Kolve et al., 2017) dataset to additionally show how batch rendering performs on assets with less geometric complexity than the scanned geometry in Gibson and Matterport3D.
Table A3 shows the learned task performance and end-to-end training speed of BPS on these two tasks for Depth -sensor-driven agents.
For both tasks, BPS outperforms its results on PointGoal navigation by around 5000 frames per second, largely due to the signiﬁcantly reduced geometric complexity of the AI2-THOR dataset versus Gibson.
Additionally, the Explore task slightly outperforms the Flee task by 600 FPS on average due to a simpler simulation workload, because no geodesic distance computation is necessary.
A. S TANDALONE B ATCH R ENDERER P ERFORMANCE
To evaluate the absolute performance of BPS ’s batch renderer independently from other components of the system, Fig. A2 shows the performance of the standalone renderer on the “Stokes” scene from [Table] [Figure]
[Figure]
{+of 
+}
For lower batch sizes, increasing resolution has a minimal performance impact, because the GPU still isn’t fully utilized. As resolution increases with larger batches, the relative decrease in performance from higher resolution increases. the Gibson dataset using a set of camera positions taken from a training run. A batch size ofachieves a 3.7x performance increase over a batch size of 1, which emphasizes the fact that much of the end to end speedup provided by batch rendering comes from the performance beneﬁts of larger inference and training batches made possible by the batch renderer’s 3D asset sharing.
Fig.
A2 also demonstrates that the batch renderer can maintain extremely high performance (approximately 23,000 FPS) at much higher resolutions than used in the RL tasks presented in this work.
While this may be useful for tasks requiring higher resolution inputs, considerable advancements would need to be made in DNN performance to handle these high resolution frames at a comparable framerate to the renderer.
A. L AMB O PTIMIZER A BLATION S TUDY
To demonstrate the beneﬁt provided by the Lamb optimizer with regard to sample efﬁciency, Fig. A shows a comparison between the Lamb optimizer used by BPS and the Adam optimizer used by W IJMANS 20 and W IJMANS
++ .
The training setup for these two optimizers is identical, with the exception of the removal of learning rate scaling for Adam, as this causes training to diverge.
The beneﬁts of Lamb are most pronounced early in training, allowing Lamb to reach within 0.7% SPL of convergence after just 1 billion samples of experience (while Adam trails Lamb by 1.5% at the same point).
As training progresses, the difference shrinks as Adam slowly converges for a ﬁnal difference of 0.6% SPL after 2.5 billion frames. [Figure]
B E XPERIMENT AND T RAINING A DDITIONAL D ETAILS
Complete PointGoal navigation description.
We train and evaluate agents via the same procedure as Wijmans et al.
Speciﬁcally, agents are trained for PointGoalNav (Anderson et al., 2018) where the agent is tasked with navigating to a point speciﬁed relative to its initial location.
Agents are equipped with a GPS+Compass sensor (providing the agent with its position and orientation relative to the starting position) and either a Depth sensor or RGB camera.
The agent has access tolow-level actions, forward (0.25m), turn left (10 ◦ ), turn right (10 ◦ ), and stop . Agents are evaluated on the Gibson dataset (Xia et al., 2018). We use two metrics to evaluate the agents: Success, whether or not the agent called stop within 0.2m of the goal, and SPL (Anderson et al., 2018), a measure of both Success and efﬁciency of the agent’s path. During evaluation, the agent does not have access to reward.
Half-precision inference and mixed-precision training.
We perform inference in half precision for all components except the action distribution.
We train in mixed precision (Jia et al., 2018), utilizing the Apex library in O2 mode.
We use half precision for all computations except the action distribution and losses.
Additionally, The optimizer still utilizes single precision for all computations and applies gradients to a single-precision copy of the weights.
Training hyper-parameters Our hyper-parameters for eight-GPU runs are given in Table A4.
We additionally employ a gradual learning rate decay where we decay the learning rate from its scaled value back to the base value over the ﬁrst half of training.
We use a cosine schedule.
We ﬁnd it necessary to set ρ =1 . 0 for the bias parameters, ﬁxup parameters, and layer-norm parameters of the network, making the optimizer for these parameters equivalent to AdamW (Kingma &
Ba, 2015; Loshchilov & Hutter, 2018).
We also use L2 weight-decay both to add back regularization lost by removing normalization layers and to stabilize Lamb; we use λ =10 − 2 .
We ﬁnd one epoch of PPO with two mini-batches to be sufﬁcient (instead of two epochs with two mini-batches), thus effectively doubling the learning speed.
We also evaluated one mini-batch, but found two to be beneﬁcial while also having little penalty on overall training speed.
C B ENCHMARKING A DDITIONAL D ETAILS
Pretrained benchmarking.
A pretrained DNN is used when benchmarking to avoid frequent environment resets at the start of training.
Benchmarking hyper-parameters.
Table A5 shows the setting for hyper-parameters that impact system throughput.
GPU details We report FPS results on three models of NVIDIA GPUs: Tesla V100, GeForce RTX 2080 TI, and GeForce RTX 3090.
We demonstrate scaling to multiple GPUs with eight GPU conﬁgurations for all but the RTX 3090.
Single GPU and eight GPU results are benchmarked on the same machines; however single GPU conﬁgurations are limited to 12 cores and 64 GB of RAM as this is a reasonable conﬁguration for a single GPU workstation. [Table] et al [Table]
[Table]
CPU details .
Each GPU conﬁguration also uses different CPU conﬁgurations based on hardware access.
Tesla V100 benchmarking was done with 2x Intel Xeon E5-2698 v4 (a DGX-1 station).
RTX 2080 TI benchmarking was done with 2x Intel Xeon Gold 6226.
RTX 3090 benchmarking was done with with 1x Intel i7-5820k.
On all CPUs, we disable Hardware P-State (HWP) (where applicable) and put software P-State in performance mode.
Our CPU load on simulation worker cores is inherently sporadic and we ﬁnd that certain CPUs are unable to change clock frequencies fast enough to not incur a considerable performance penalty when allowed to enter a power saving state.
[Table]
[Figure]