DD-PPO: L EARNING N EAR -P ERFECT P OINT G OAL N AVIGATORS FROM 2.5 B ILLION F RAMES
Anonymous authors
Paper under double-blind review
A BSTRACT
We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments.
DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever ‘stale’), making it conceptually simple and easy to implement.
In our experiments on training virtual robots to navigate in Habitat-Sim (Savva et al., 2019), DD-PPO exhibits near-linear scaling – achieving a speedup of 107x on 128 GPUs over a serial implementation.
We leverage this scaling to train an agent for 2.5 Billion steps of experience (the equivalent of 80 years of human experience) – over 6 months of
GPU-time training in under 3 days of wall-clock time with 64 GPUs.
This massive-scale training not only sets the state of art on Habitat Autonomous
Navigation Challenge 2019, but essentially ‘solves’ the task – near-perfect autonomous navigation in an unseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor.
Fortuitously, error vs computation exhibits a power-law-like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under 1 day with 8 GPUs).
Finally, we show that the scene understanding and navigation policies learned can be transferred to other navigation tasks – the analog of ‘ImageNet pre-training + task-speciﬁc ﬁne-tuning’ for embodied AI.
Our model outperforms
ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models + code will be publicly available).
I NTRODUCTION
Recent advances in deep reinforcement learning (RL) have given rise to systems that can outperform human experts at variety of games (Silver et al., 2017; Tian et al., 2019; OpenAI, 2018).
These advances, even more-so than those from supervised learning, rely on signiﬁcant numbers of training samples, making them impractical without large-scale, distributed parallelization.
Thus, scaling RL via multi-node distribution is of importance to AI – that is the focus of this work.
Several works have proposed systems for distributed RL (Heess et al., 2017; Liang et al., 2018;
Tian et al., 2019; Silver et al., 2016; OpenAI, 2018; Espeholt et al., 2018).
These works utilize two core components: 1) workers that collect experience (‘rollout workers’), and 2) a parameter server that optimizes the model.
The rollout workers are then distributed across, potentially, thousands of CPUs 1 .
However, synchronizing thousands of workers introduces signiﬁcant overhead (the parameter server must wait for the slowest worker, which can be costly as the number of workers grows).
To combat this, they wait for only a few rollout workers, and then asynchronously optimize the model.
However, this paradigm – of a single parameter server and thousands of (typically CPU) workers – appears to be fundamentally incompatible with the needs of modern computer vision and robotics communities.
Over the last few years, a large number of works have proposed training virtual robots (or ‘embodied agents’ ) in rich 3D simulators before transferring the learned skills to reality (Beattie et al., 2016; Chaplot et al., 2017; Das et al., 2018; Gordon et al., 2018; Anderson et al., 2018b;
Wijmans et al., 2019; Savva et al., 2019).
Unlike Gym or Atari, 3D simulators require GPU acceleration, and, consequently, the number of workers is greatly limited ( 2 5 to 8 vs. 2 12 to 15 ).
The desired agents operate from high dimensional inputs (pixels) and, consequentially, use deep networks (ResNet50) that strain the parameter server.
Thus, there is a need to develop a distributed architecture.
[Figure]
Contributions.
We propose a simple, synchronous, distributed RL method scales well.
We call this method Decentralized Distributed Proximal Policy Optimization (DD-PPO) as it is decentralized (has no parameter server), distributed (runs across many different machines), and we use it to scale
Proximal Policy Optimization (Schulman et al., 2017).
In DD-PPO, each worker alternates between collecting experience in a resource-intensive and GPU accelerated simulated environment and optimizing the model.
This distribution is synchronous – there is an explicit communication stage where workers synchronize their updates to the model (the gradients).
To avoid delays due to stragglers, we propose a preemption threshold where the experience collection of stragglers is forced to end early once a pre-speciﬁed percentage of the other workers ﬁnish collecting experience.
All workers then begin optimizing the model.
We characterize the scaling of DD-PPO by the steps of experience per second with N workers relative to 1 worker.
We consider two different workloads, 1) simulation time is roughly equivalent for all environments, and 2) simulation time can vary dramatically due to large differences in environment complexity.
Under both workloads, we ﬁnd that DD-PPO scales near-linearly .
While we only examined our method with PPO, other on-policy RL algorithms can easily be used and we believe the method is general enough to be adapted to off -policy RL algorithms.
We leverage these large-scale engineering contributions to answer a key scientiﬁc question arising in embodied navigation.
Mishkin et al.
(2019) benchmarked classical (mapping + planning) and learning-based methods for agents with RGB-D and GPS+Compass sensors on PointGoal Navigation (Anderson et al., 2018a)
( PointGoalNav ), see Fig. 1, and showed that classical methods outperform learning-based.
However, they trained for ‘only’ 5 million steps of experience.
Savva et al.
(2019) then scaled this training to 75 million steps and found that this trend reverses – learningbased outperforms classical, even in unseen environments !
However, even with an order of magnitude more experience (75M vs 5M), they found that learning had not yet saturated.
This begs the question – what are the fundamental limits of learnability in PointGoalNav ?
Is this task entirely learnable?
We answer this question afﬁrmatively via an ‘existence proof’.
Utilizing DD-PPO, we ﬁnd that agents continue to improve for a long time (Fig. 1) – not only setting the state of art in Habitat Autonomous Navigation Challenge 2019 (Savva et al., 2019), but essentially ‘solving’ PointGoalNav (for agents with GPS+Compass).
Speciﬁcally, these agents 1) almost always reach the goal (failing on 1/1000 val episodes on average), and 2) reach it nearly as efﬁciently as possible – nearly matching (within 3% of) the performance of a shortest-path oracle !
It is worth stressing how uncompromising that comparison is – in a new environment, an agent navigating without a map traverses a path nearly matching the shortest path on the map.
This means there is no scope for mistakes of any kind – no wrong turn at a crossroad, no back-tracking from a
Rollout worker
Parameter Server
Rollout
Parameters
Exp
[Figure]
Optimization
Asynchronous Communcation
Synchronous Communcation
Asynchronous
Synchronous via Distributed Data Parallel
GPU Process
CPUProcess
Gradient Gradient
Gradient Gradient
Legend
Exp
[Figure]
Optimization
Exp
[Figure]
Optimization
Exp
[Figure]
Optimization
Rollout worker
Rollout
Parameters
Rollout worker
Parameter Server
Rollout
Parameters
Exp
[Figure]
Optimization
Asynchronous Communcation
Synchronous Communcation
Asynchronous
Synchronous via Distributed Data Parallel
GPU Process
CPUProcess
Gradient Gradient
Gradient Gradient
Legend
[Figure]
Optimization
[Figure]
Optimization
[Figure]
Rollout
Parameters dead-end, no exploration or deviation of any kind from the shortest-path.
hypothesis is that the model learns to exploit the statistical regularities in the ﬂoor-plans of indoor environments (apart ments ofﬁces in
The more challenging navigating without GPS+Compass progress but
Finally, we show that the scene understanding and navigation policies learned on PointGoalNav can be transferred to other tasks (Flee and Explore (Gordon et al., 2019))
– the analog of ‘ImageNet pre-training + task-speciﬁc ﬁne-tuning’ for Embodied AI.
Our models are able to rapidly learn these new tasks (outperforming ImageNet pre-trained CNNs) and can be utilized as near-perfect neural
PointGoal controllers , a universal resource for other high-level navigation tasks (Anderson et al., 2018b; Das et al., 2018).
We will make code and trained models publicly available.
P RELIMINARIES : RL AND PPO
Reinforcement learning (RL) is concerned with decision making in Markov decision processes.
In a partially observable MDP (POMDP), the agent receives an observation that does not fully specify the state ( s t ) of the environment, o t ( e.g . an egocentric RGB image), takes an action a t , and is given a reward r t .
The objective is to maximize cumulative reward over an episode , Formally, let τ be a sequence of ( o t , a t , r t ) where a t ∼ π ( · | o t ) , and s t +1 ∼ T ( s t , a t ) .
For a discount factor γ , which balances the trade-off between exploration and exploitation, the optimal policy, π ∗ , is speciﬁed by
[Equation]
One technique to ﬁnd π ∗ is Proximal Policy Optimization (PPO) (Schulman et al., 2017), an on-policy algorithm in the policy-gradient family.
Given a θ -parameterized policy π θ and a set of trajectories collected with it (commonly referred to as a ‘rollout’), PPO updates π θ as follows.
Let ˆ A t =
R t − (cid:98) V t , be the estimate of the advantage, where R t = (cid:80)
Ti = t γ i − t r i , and ˆ V t is the expected value of R t , and r t ( θ ) =
π θ ( a t | o t ) π θt ( a t | o t ) be the ratio of the probability of the action a t under the current policy and the policy used to collect the rollout.
The parameters are then updated by maximizing
[Equation]
[Table]
[Equation]
[Table] [Equation] This clipped objective keeps this ratio within (cid:15) and functions as a trust-region optimization method; allowing for the multiple gradient updates using the rollout, thereby improving sample efﬁciency.
D ECENTRALIZED D ISTRIBUTED P ROXIMAL P OLICY O PTIMIZATION
In reinforcement learning, the dominant paradigm for distribution is asynchronous (see Fig. 2).
Asynchronous distribution is notoriously difﬁcult – even minor errors can result in opaque crashes – and the parameter server and rollout workers necessitate separate programs.
Policy <left> h<forward> h<left> <forward>
h Ta T- <stop>
Policy
Policy Policy
Policy a
CNN CNN CNN
CNN CNN
V 1 aha
V
Vh Ta TV TV T
In supervised learning, however, synchronous distributed training via data parallelism (Hillis &
Steele Jr, 1986) dominates.
As a general abstraction, this method implements the following: at step k , worker n has a copy of the parameters, θ kn , calculates the gradient, ∂θ kn , and updates θ via θ k +1 n = ParamUpdate (cid:16) θ kn , AllReduce (cid:0)
∂θ k 1 , . .
, ∂θ kN (cid:1) (cid:17) =
ParamUpdate (cid:16) θ kn , 1 N
N (cid:88) i =
∂θ ki (cid:17) , (3) where ParamUpdate is any ﬁrst-order optimization technique ( e.g . gradient descent) and AllReduce performs a reduction ( e.g . mean) over all copies of a variable and returns the result to all workers.
Distributed DataParallel scales very well (near-linear scaling up to 32,000 GPUs (Kurth et al., 2018)), and is reasonably simple to implement (all workers synchronously running identical code).
We adapt this to on-policy RL as follows:
At step k , a worker n has a copy of the parameters θ kn ; it gathers experience (rollout) using π θ kn , calculates the parameter-gradients ∇ θ via any policy-gradient method ( e.g . PPO), synchronizes these gradients with other workers, and updates the model:
[Equation]
A key challenge to using this method in RL is variability in experience collection run-time.
In supervised learning, all gradient computations take approximately the same time.
In RL, some resourceintensive environments can take signiﬁcantly longer to simulate.
This introduces signiﬁcant synchronization overhead as every worker must wait for the slowest to ﬁnish collecting experience.
To combat this, we introduce a preemption threshold where the rollout collection stage of these stragglers is preempted (forced to end early) once some percentage, p % , (we ﬁnd 60% to work well) of the other workers are ﬁnished collecting their rollout; thereby dramatically improving scaling.
We weigh all worker’s contributions to the loss equally and limit the minimum number of steps before preemption to one-fourth the maximum to ensure all environments contribute to learning.
While we only examined our method with PPO, other on-policy RL algorithms can easily be used and we believe the method can be adapted to off -policy RL algorithms.
Off-policy RL algorithms also alternate between experience collection and optimization, but differ in how experience is collected/used and the parameter update rule.
Our adaptations simply add synchronization to the optimization stage and a preemption to the experience collection stage.
Implementation.
We leverage PyTorch’s (Paszke et al., 2017) DistributedDataParallel to synchronize gradients, and TCPStore – a simple distributed key-value storage – to track how many workers have ﬁnished collecting experience.
See Apx. E for a detailed description with code.
E XPERIMENTAL S ETUP : P OINT G OAL N AVIGATION , A GENTS , S IMULATOR
PointGoal Navigation ( PointGoalNav ).
An agent is initialized at a random starting position and orientation in a new environment and asked to navigate to target coordinates speciﬁed relative to the agents position; no map is available and the agent must navigate using only its sensors – in our case
RGB-D (or RGB ) and GPS+Compass (providing current position and orientation relative to start).
The evaluation criteria for an episode is as follows (Anderson et al., 2018a):
Let S indicate ‘success’ (did the agent stop within 0.2 meters of the target?), l be the length of the shortest path between start
Path Length SPL =
S l max( l,p ) .
It is worth stressing that SPL is a highly punitive metric – to achieve
SPL = 1 , the agent (navigating without the map) must match the performance of the shortest-path oracle that has access to the map!
There is no scope for any mistake – no wrong turn at a crossroad, no back-tracking from a dead-end, no exploration or deviation from the shortest path.
In general, this may not even be possible in a new environment (certainly not if an adversary designs the map).
Agent.
As in Savva et al.
(2019), the agent has 4 actions, stop , which indicates the agent has reached the goal, move forward (0.25m), turn left ( 10 ◦ ), and turn right ( 10 ◦ ).
It receives 256x256 sized images and uses the GPS+Compass to compute target coordinates relative to its current state.
The
RGB-D agent is limited to only Depth as Savva et al.
(2019) found this to perform best.
Our agent architecture (Fig. 3) has two main components – a visual encoder and a policy network.
The visual encoder is based on either ResNet (He et al., 2016) or SE (Hu et al., 2018)-ResNeXt
(Xie et al., 2017) with the number of output channels at every layer reduced by half.
We use a ﬁrst layer of 2x2-AvgPool to reduce resolution (essentially performing low-pass ﬁltering + down-sampling) – we ﬁnd this to have no impact on performance while allowing faster training.
From our initial experiments, we found it necessary to replace every BatchNorm layer (Ioffe & Szegedy, 2015) with
GroupNorm (Wu & He, 2018) to account for highly correlated inputs seen in on-policy RL.
The policy is parameterized by a 2 -layer LSTM with a 512 -dimensional hidden state.
It takes three inputs: the previous action, the target relative to the current state, and the output of the visual encoder.
The LSTM’s output is used to produce a softmax distribution over the action space and an estimate of the value function.
See Appendix C for full details.
Training.
We use PPO with Generalized Advantage Estimation (Schulman et al., 2015).
We set the discount factor γ to 0 .
99 and the GAE parameter τ to 0 .
Each worker collects (up to) 128 frames of experience from 4 agents running in parallel (all in different environments) and then performsepochs of PPO with 2 mini-batches per epoch.
We use Adam (Kingma & Ba, 2014) with a learning rate of 2 . 5 × 10 − 4 .
Unlike popular implementations of PPO, we do not normalize advantages as we ﬁnd this leads to instabilities.
We use DD-PPO to train with 64 workers on 64 GPUs.
The agent receives terminal reward r T = 2 .
5 SPL, and shaped reward r t ( a t , s t ) = − ∆ geo dist − 0 .
01 , where ∆ geo dist is the change in geodesic distance to the goal by performing action a t in state s t .
Simulator+Datasets.
Our experiments are conducted using Habitat, a 3D simulation platform for embodied AI research (Savva et al., 2019).
Habitat is a modular framework with a highly performant and stable simulator, making it an ideal framework for simulating billions of steps of experience.
We experiment with several different sources of data.
First, we utilize the training data released as part of the Habitat Challenge 2019, consisting of 72 scenes from the Gibson dataset (Xia et al., 2018).
We then augment this with all 90 scenes in the Matterport3D dataset (Chang et al., 2017) to create a larger training set (note that Matterport3D meshes tend to be larger and of better quality).
Furthermore, Savva et al.
(2019) curated the Gibson dataset by rating every mesh reconstruction on a quality scale of 0 to 5 and then ﬁltered all splits such that each only contains scenes with a rating of 4 or above (Gibson-4+), leaving all scenes with a lower rating previously unexplored.
We examine training on the 332 scenes from the original train split with a rating of 2 or above (Gibson-2+).
B ENCHMARKING : H OW DOES DD-PPO SCALE ?
In this section, we examine how DD-PPO scales under two different workload regimes – homogeneous (every environment takes approximately the same amount of time to simulate) and heterogeneous (different environments can take orders of magnitude more/less time to simulate).
We examine the number of steps of experience per second with N workers relative to 1 worker.
We compare different values of the preemption threshold p % .
We benchmark training our ResNet50 PointGoalNav agent with Depth on a cluster with Nvidia V100 GPUs and NCCL2.4.7 with Inﬁniband interconnect.
Homogeneous.
To create a homogeneous workload, we train on scenes from the Gibson dataset, which require very similar times to simulate agent steps.
As shown in Fig.
4 (left), DD-PPO exhibits near-linear scaling (linear = ideal) for preemption thresholds larger than 50%, achieving a 196x speed up with 256 GPUs relative to 1 GPU and an 7.3x speed up with 8 GPUs relative to 1.
[Figure]
Heterogeneous.
To create a heterogeneous workload, we train on scenes from both Gibson and
Matterport3D. Unlike Gibson, MP3D scenes vary signiﬁcantly in complexity and time to simulate – the largest contains 8GB of data while the smallest is only 135MB.
DD-PPO scales poorly at a preemption threshold of 100% (no preemption) due to the substantial straggler effect (one rollout taking substantially longer than the others); see Fig.
4 (right).
However, with a preemption threshold of 80% or 60%, we achieve near-identical scaling to the homogeneous workload!
We found no degradation in performance of models trained with any of these values for the preemption threshold despite learning in large scenes occurring at a lower frequency.
M ASTERING P OINT G OAL N AVIGATION WITH GPS+C OMPASS
In this section, we answer the following questions: 1) What are the fundamental limits of learnability in PointGoalNav navigation?
2) Do more training scenes improve performance?
3) Do better visual encoders improve performance?
4) Is PointGoalNav ‘solvable’ when navigating from RGB instead of Depth ? 5)
What are the open/unsolved problems – speciﬁcally, how does navigation without
GPS+Compass perform? 6) Can agents trained for PointGoalNav be transferred to new tasks?
Agents continue to improve for a long time.
Using DD-PPO, we train agents for 2.5 Billion steps of experience with 64 Tesla V100 GPUs in 2.75 days – 180 GPU-days of training, the equivalent of 80 years of human experience (assuming 1 human second per step).
As a comparison, Savva et al.
(2019) reached 75 million steps (an order of magnitude more than prior work) in 2.5 days using GPUs – at that rate, it would take them over a month (wall-clock time) to achieve the scale of our study.
Fig.
1 shows the performance of an agent with RGB-D and GPS+Compass sensors, utilizing an SE-ResNeXt50 visual encoder, trained on Gibson-2+ – it does not saturate before 1 billion steps 3 , suggesting that previous studies were incomplete by 1-2 orders of magnitude .
Fortuitously, error vs computation exhibits a power-law-like distribution; 90% of peak performance is obtained relatively early (100M steps) and relatively cheaply (in 0.1 day with 64 GPUs and in 1 day with 8 GPUs 4 ).
Also noteworthy in Fig.
1 is the strong generalization (train to val) and corresponding lack of overﬁtting.
Increasing training data helps.
Tab. 1 presents results with different training datasets and visual encoders for agent with RGB-D and GPS+Compass.
Our most basic setting (ResNet50, Gibson-4+ training) already achieves SPL of 0.922 (val), 0.917 (test), which nearly misses (by 0.003) the top of the leaderboard for the Habitat Challenge 2019 RGB-D track 5 .
Next, we increase the size of the training data by adding in all Matterport3D scenes and see an improvement of ∼ 0.03 SPL – to 0.
(val), 0.941 (test).
Next, we compare training on Gibson-4+ and Gibson-2+.
Recall that Gibson- { 2, 3 } corresponds to poorly reconstructed scenes (see Fig. 10).
A priori, it is unclear whether the net effect of this addition would be positive or negative; adding them provides diverse experience to the agent, however, it is poor quality data.
We ﬁnd a potentially counter-intuitive result – adding poor 3D reconstructions to the train set improves performance on good reconstructions in val/test by ∼ 0.
SPL – from 0.922 (val), 0.917 (test) to 0.956 (val), 0.944 (test).
Our conjecture is that training on poor (Gibson- { 2,3 } ) and good (4+) reconstructions leads to robustness in representations learned.
[Table]
Better visual encoders and more parameters help.
Using a better visual encoder, SE (Hu et al.
2018)-ResNeXt50 (Xie et al., 2017) instead of ResNet50, improves performance by 0.003 SPL (Tab. 1).
Adding capacity to the visual encoder (SE-ResNeXt101 vs SE-ResNeXt50) and navigation policy (1024-d vs 512-d LSTM) further improves performance by 0.010 SPL.
PointGoalNav ‘solved’ with RGB-D and GPS+Compass.
Our best agent – SE-ResNeXt101 + 1024-d LSTM trained on Gibson-2+ – achieves SPL of 0.969 (val), 0.948 (test), which not only sets the state of art on the Habitat Challenge 2019 RGB-D track but is also within 3 - 5% of the shortest-path oracle 6 .
Given the challenges with achieving near-perfect SPL in new environments, it is important to dig deeper.
Fig.
12 shows (a) distribution of episode lengths in val and (b) SPL vs episode length.
We see that while the dataset is dominated by short episodes (2-12m), the performance of the agent is remarkably stable over long distances and average SPL is not necessarily inﬂated.
Our hypothesis is the agent has learned to exploit the structural regularities in layouts of real indoor environments.
One (admittedly imperfect) way to test this is by training a Blind agent with only a GPS+Compass sensor.
Fig.
12 shows that this agent is able to handle short-range navigation (which primarily involve turning to face the target and walking straight) but performs very poorly on longer trajectories – SPL of 0.3 ( Blind ) vs 0.95 ( RGB-D ) at 20-25m navigation.
Thus, structural regularities, in part, explain performance for short-range navigation.
For long-range navigation, the RGB-D agent is extracting overwhelming signal from its Depth sensor.
We repeat this analysis on two additional navigation datasets proposed by Chaplot et al. (2019) – longer episodes and ‘harder’ episodes (more navigation around obstacles) – and ﬁnd similar trends (Fig. 13). This discussion continues in Apx.
A. Performance with RGB is also improved.
So far we studied RGB-D as this performed best in Savva et al.
We now study RGB (with SE-ResNeXt50 encoder).
We found it crucial to train on
Gibson-2+ and all of Matterport3D, ensuring diversity in both layouts (Gibson-2+) and appearance (Matterport3D), and to channel-wise normalize RGB (subtract by mean and divide by standard deviation) as our networks lack BatchNorm.
Performance improves dramatically from 0.57 (val), 0.
(test) SPL in Savva et al.
to near-perfect success 0.991 (val), 0.977 (test) and high SPL 0.
(val), 0.920 (test).
While SPL is considerably lower than the Depth agent, (0.929 vs 0.959), interestingly, the RGB agent still reaches the goal a similar percentage of the time (99.1% vs 99.9%).
This agent achieves state-of-art on the Habitat Challenge 2019 RGB track (rank 2 entry has 0.89 SPL).
No GPS+Compass remains unsolved.
Finally, we examine if we also achieve better performance on the signiﬁcantly more challenging task of navigation from RGB without GPS+Compass.
Atmillion steps (an amount equivalent to Savva et al.
, the agent achieves 0 SPL.
By training to 2.5 billion steps, we make some progress and achieve 0.15 SPL.
While this is a substantial improvement, the task continues to remain an open frontier for research in embodied AI.
Transfer Learning.
We examine transferring our agents to the following tasks (Gordon et al., 2019)
– Flee The agent maximizes its geodesic distance from its starting location.
Let s t be the agent’s position at time t , and Max ( s 0 ) denote the maximum distance over all reachable points, then the agent maximizes D T =
Geo ( s T , s 0 ) /Max
( s 0 ) .
The reward is r t =
( D t − D t − 1 ) .
– Exploration The agent maximizes the number of locations (speciﬁed by 1m cubes) visited.
Let | Visited t | denote the number of location visited at time t , then the agent maximizes | Visited T | .
The reward is r t = 0 .
25( | Visited t | − | Visited t − 1 | ) .
[Figure]
r f
[Figure]
r
[Figure]
c
[Figure]
( h i ghe r
i s
[Figure]
tt e r)
[Figure]
We use a PointGoalNav -trained agent with RGB and GPS+Compass, remove the GPS+Compass, and transfer to these tasks under ﬁve different settings:
– Scratch.
All parameters (visual encoder + policy) are trained from scratch for each new task.
Improvements over this baseline demonstrate beneﬁts
of
transfer learning.
– ImageNetEncoder-ScratchPolicy.
The visual encoder is initialized with ImageNet pre-trained weights and frozen; the navigation policy is trained from scratch.
– PointGoalNav Encoder-ScratchPolicy.
The visual encoder is initialized from PointGoalNav and frozen; the navigation policy is trained from scratch.
– PointGoalNav Encoder-FinetunePolicy.
Both visual encoder and policy parameters are initialized from PointGoalNav (critic layers are reinitialized).
Encoder is frozen, policy is ﬁne-tuned.
– ∇ Neural Controller
We treat
our agent as a differentiable neural controller , a closed-loop lowlevel controller than
can
navigate to
a speciﬁed coordinate
We
utilize
this controller in a new
task by
training
a light-weight
high level planner that predicts a goal-coordinate (at each time-step) for the controller to navigate to.
Since the controller is fully differentiable, we can backprop through it.
We freeze the controller, train the planner+controller system with PPO for the new
task.
The planner
is a 2-layer LSTM and shares the (frozen) visual encoder with the controller.
Fig.
5 shows performance vs. experience results (higher is better).
Nearly all methods outperform learning from scratch, establishing the value of transfer learning.
PointGoalNav pre-trained visual encoders dramatically outperforms ImageNet pre-trained ones, indicating that the agent has learned generally useful scene understanding.
For both tasks, ﬁne-tuning an existing policy allows it to rapidly learn the new task, indicating that the agent has learned general navigation skills.
∇ Neural
Controller outperforms PointGoalNav Encoder-ScratchPolicy on Flee and is competitive on Exploration, indicating that the agent can indeed be ‘controlled’ or directed to target locations by a planner.
Overall, these results demonstrate that our trained model is useful for more than just PointGoalNav .
R ELATED W ORK
Visual Navigation.
Visual navigation in indoor environments has been the subject of many recent works (Gupta et al., 2017; Das et al., 2018; Anderson et al., 2018b; Savva et al., 2019; Mishkin et al., 2019).
Our primary contribution is DD-PPO, thus we discuss other distributed works.
In the general case, computation in reinforcement learning (RL) in simulators can be broken down into 4 roles: 1) Simulation: Takes actions performed by the agent as input, simulates the new state, returns observations, reward, etc . 2)
Inference: Takes observations as input and utilizes the agent policy to return actions, value estimate, etc . 3) Learner: Takes rollouts as input and computes gradients to update the policy’s parameters.
Parameter server/master: Holds the source of truth for the policy’s parameters and coordinates workers.
Synchronous RL.
Synchronous RL systems utilize a single processes to perform all four roles; this design is found in RL libraries like OpenAI Baselines (Dhariwal et al., 2017) and PytorchRL (Kostrikov, 2018).
This method is limited to a single nodes worth of GPUs.
Synchronous Distributed RL.
The work most closely related to DD-PPO is that of Stooke &
Abbeel (2018), who also propose to scale synchronous RL by replicating this simulation/inference/learner process across multiple GPUs and then synchronize gradients with AllReduce ; they experiment with Atari and ﬁnd it not effective however.
We hypothesize that this is due to a subtle difference – this distribution design relies on a single worker collecting experience from multiple environments, stepping through them in lock step .
This introduces signiﬁcant synchronization and communication costs as every step in the rollout must be synchronized across as many as 64 processes (possible because each environment is resource-light, e.g . Atari).
For instance, taking 1 step in 8 parallel pong environments takes approximately the same wall-clock time as 1 pong environment, but it takes 10 times longer to take 64 steps in lock-step; thus gains from parallelization are washed out due to the lock-step synchronization.
In contrast, we study resource-intensive environments, where only 2 or 4 environments per worker is possible, and ﬁnd this technique to be effective.
We also propose an adaption to mitigate the straggler effect – preempting the experience collection (rollout) of stragglers and then beginning optimization.
This improves scaling for homogeneous workloads and dramatically improves scaling for heterogeneous workloads.
Asynchronous Distributed RL.
Existing public frameworks for asynchronous distributed reinforcement learning (Heess et al., 2017; Liang et al., 2018; Espeholt et al., 2018) use a single (CPU-only) process to perform the simulation and inference roles (and then replicate this process to scale).
A separate process asynchronously performs the learner and parameter server roles (note its not clear how to use more than one these processes as it holds the source of truth for the parameters). Adapting these methods to the resource-intensive environments studied in this work ( e.g . Habtiat (Savva et al., 2019)) encounters the following issues: 1) Limiting the inference/simulation processes to
CPU-only is untenable (deep networks and need for GPU-accelerated simulation). While the inference/simulation processes could be moved to the GPU, this would be ineffective for the following: GPUs operate most efﬁciently with large batch sizes each inference
/simulation process would have a batch size of 1) CUDA runtime requires ∼ 600MB
of GPU memory per process, and only one
CUDA kernel (function that runs on the GPU) can executed by the GPU at a time.
These issue contribute and lead to low GPU utilization.
In contrast, DD-PPO utilizes a single process per GPU and batches observations from multiple environments for inference.
The single process learner/- parameter server is limited to a single node’s worth of GPUs.
While this not a limitation for small networks and low dimensional inputs, our agents take high dimensional inputs ( e.g . a Depth sensor) and utilize large neural networks (ResNet50), thereby requiring considerable computation to compute gradients.
In contrast, DD-PPO has no parameter server and every GPU computes gradients, supporting even very large networks (SE-ResNeXt101).
Related work continues in Apx.
B.
A A DDITIONAL ANALYSIS AND DISCUSSION
In this section, we continue the analysis of our agent and examine differences in its behavior from a classical, hand-designed agent – the map-and-plan baseline agent proposed in Gupta et al.
[Figure]
Error recovery.
Given the near-perfect performance of our agent (on average), we explicitly examine if it is able to recover from its own navigation errors.
Fig.
6 column 3 shows several examples of error recovery, including several well executed backtracks (video: https://www.youtube.com/
watch?v=a8AugVLSJ50 ), indicating that the agent is effective at recovering from its own navigation errors.
Next, we look at the statistics of non-perfect (SPL < 0.99) episodes on the longer validation episodes proposed in Chaplot et al.
Nonperfect episodes make up the majority of episodes (54%, see
Fig. 7) with an average SPL of 0.85 (99.0% success) – compared to 0.92 SPL (99.5% success) over all episodes.
Thus there are many episodes where the agent makes signiﬁcant deviation from the shortest path and reaches the goal (a 15% deviation on long trajectories ( > 10m) is signiﬁcant).
When does the agent fail?
Column 2 in Fig.
6 shows that the agent performs poorly when the ratio of the geodesic distance to goal and euclidean distance to goal.
However, the agent is able to eventually overcome this failure mode and reach the goal in most cases.
Row 1 column 1 in Fig.
6 shows that the agent fails or performs poorly when it needs to go slightly up/down stairs.
The data-set generation process used in Savva et al.
(2019) only guarantees a start and goal pair won’t be on different ﬂoors, but there remains a possibility that the agent will need to traverse the stairs slightly.
However, these situations are rare, and, in general, the stairs should be avoided.
Furthermore, the GPS sensor provides location in 2D, not 3D.
The remaining failure cases of column 1 in Fig.
6 show that a singular location in one environment acts as a sink for the agent (once it enters this location, it is almost never able to leave it).
At this location, there is a large hole in the mesh (an entire wall is missing).
Utilizing visual encoders that explicitly handle missing values may allow the agent to overcome this failure mode.
Differences from a classical agent.
We compare the behavior of our agent with the classical mapand-plan baseline agent proposed in Gupta et al.
This agent achieves 0.92 val (0.89 test)
SPL with 0.976 success.
By comparing and contrasting qualitative behaviors, we can determine what behaviors learning-based methods enable.
We make the following observation.
The learned agent is able to recover from unexpected collisions without hurting SPL.
The mapand-plan baseline agent incorporates a speciﬁc collision recovery behavior where, after repeated collisions, the agent turns around and backs up 1.25m. This behavior brings the obstacle into view, maps it, and then allows the agent to create a plan to avoid it.
In contrast, our agent is able to navigate around unseen obstacles without such a large impact on SPL.
Determining the set of action sequences and heuristics necessary to do this is what learning enables.
B R ELATED W ORK C ONTINUED
Straggler Effect Mitigation.
In supervised learning, the straggler effect is commonly caused by heterogeneous hardware or hardware failures.
Chen et al.
(2016) propose a pool of b “back-up” workers (there are N + b workers total) and perform the parameter update once N workers ﬁnish.
In comparison, their method a) requires a parameter server, and b) discards all work done by the stragglers.
Chen et al.
(2018) propose to dynamically adjust the batch size of each worker such that all workers perform their forward and backward pass in the same amount of time.
Our method aims to reduce variance in experience collection times.
While DD-PPO does dynamically adjust a worker’s batch size, this is a necessary side-effect of on-policy RL.
Distributed Synchronous SGD.
Data parallelism is a common paradigm in high performance computing (Hillis & Steele Jr, 1986).
In this paradigm, parallelism is achieved by workers performing the same work on different data.
This paradigm can be naturally adapted to supervised deep learning (Chen et al., 2016).
Works have used this to achieve state-of-the-art results in tasks ranging from computer vision (Goyal et al., 2017;
He et al., 2017) to natural language processing (Peters et al., 2018; Devlin et al., 2018; Ott et al., 2019).
Furthermore, multiple deep learning frameworks provide simple-to-use wrappers supporting this parallelism model (Paszke et al., 2017; Abadi et al., 2015;
Sergeev & Balso, 2018).
We adapt this framework to reinforcement learning.
C A GENT D ESIGN
In this section, we outline the exact agent design we use.
We break the agent into three components: a visual encoder, a goal encoder, and a navigation policy.
Visual Encoder.
Out visual encoder uses one of three different backbones, ResNet50 (He et al., 2016), Squeeze-Excite(SE) (Hu et al.
, 2018)-ResNeXt50 (Xie et al., 2017), and SE-ResNeXt101.
For all backbones, we reduce the number of output channels at each layer by half.
We also add a 2 x 2 -AvgPool before each backbone so that the effective resolution is 128 x 128 .
Given these modiﬁcations, each backbone produces a 1024 x 4 x 4 feature map.
We then convert this to a 128 x 4 x 4 feature map with a 3 x 3 -Conv .
We replace every BatchNorm layer with GroupNorm (Wu & He, 2018) to account for the highly correlated trajectories seen in on-policy RL and massively distributed training.
Goal encoder.
Habitat (Savva et al., 2019) provides the vector pointing to the goal in ego-centric polar coordinates.
We convert this to magnitude and a unit vector, i.e .
[d, θ ] to [d, cos( θ ) , sin( θ ) ] , to account for the discontinuity at the x -axis in polar coordinates.
We pass the goal vector to a fully connected layer, resulting in a 32 -dimensional representation.
Navigation Policy.
Our navigation policy takes the 64 x 4 x 4 feature map from the visual encoder, ﬂattens it, and then converts the 2048-d vector to the same size as the hidden size via a fully-connected layer.
It then concatenates this vector with output of the goal encoder, and a 32 -dimensional embedding of the previous action taken (or the start-token in the case of the ﬁrst action) and then passes this to a 2 -layer LSTM with either a 512 -dimensional or 1024 -dimensional hidden dimension. The
[Figure] output of the LSTM is used as input to a fully connected layer, resulting in a soft-max distribution of the action space and an estimate of the value function.
D A DDITIONAL SCALING DETAILS
We use the following procedure for benchmarking the throughput of our proposed DD-PPO: Each optimizer selects 4 scenes at random and then performs the process of collecting experience and optimizing the model based on that experience 10 times.
We calculate throughput as the total number of steps of experience collected over the last 5 rollout/optimizing steps divided by the amount of time taken.
We repeat this procedure over 10 different random seeds (we use the same random seeds for all variations of number
E DD-PPO I MPLEMENTATION
Utilizing Distributed Data Parallel in supervised learning is straightforward as frameworks such as PyTorch (Paszke et al., 2017) provide a simple wrapper.
The recommended way to use these wrappers is to ﬁrst write training code that runs on a single GPU and then enable distributed training via the wrapper.
We follow a similar approach.
Given an implementation of
PPO that runs on one GPU we create a decentralized distributed variant by adding gradient synchronization, leveraging highly performant code written for this purpose in popular deep-learning frameworks, e.g. tf.distribute.
MirroredStrategy in TensorFlow (Abadi et al., 2015) and torch.nn.parallel.
DistributedDataParallel in PyTorch.
Note that care must be taken to synchronize any training or rollout statistics between workers – in most cases these can also be synchronized via AllReduce .
We track how many workers have ﬁnished the experience collection stage with a distributed keyvalue storage – we use PyTorch’s torch.distributed.
TCPStore , however almost any distributed key-value storage would be sufﬁcient. [Table]
See Fig.
9 for an example implementation which adds 1) gradient synchronization via torch.nn.parallel.
DistributedDataParallel , and 2) preempts stragglers by tracking the number of workers have ﬁnished the experience collection stage with a torch.distributed.
TCPStore .
F T RANSFER EXPERIMENTS ADDITIONAL DETAILS
For the transfer learning experiments, we utilize the same PPO hyper-parameters as the
PointGoalNav experiments.
We use DD-PPO to train with 8 workers on 8 GPUs.
We train our agents on Gibson-4+ and evaluate on the Habitat Challenge 2019 Validation scene and starting locations (the goal location is simply discarded).
The ImageNet encoder is trained using the same hyper-parameters and training procedure as Xie et al.
(2017) with no data-augmentation.
G N EURAL CONTROLLER ADDITIONAL DETAILS
The planner for neural controller used in Sec.
6 shares the same architecture as our agent’s policy, but utilizes a 512-d hidden state.
It takes as input the previous action of the controller (or the start token), and the output of the visual encoder (which is shared with the controller).
The output of the LSTM is then used to produced an estimate of the value function and a 3-dimensional vector specifying the PointGoal in magnitude and unit direction vector format.
The magnitude competent is passed through an ELU activation and offset by 0.75.
Each component of the unit direction vector is passed through a tanh activation – note that we do not re-normalize this vector have a length ofas we ﬁnd doing so both unnecessary and harder to optimize.
[Table] [Table]
their
[Table]
num
[Table]
workers
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
) model.to(device)#Addgradientsynchronization to the model model = torch.nn.parallel.
DistributedDataParallel( model , [device], device ) while not_converged(): num_workers_done.set("done", "0") for step in range (max_experience_steps): collect_step(model)
#Preemptstragglers if ( int (num_workers_done.get("done"))
>
preemption_threshold * world_size and step >= max_experience_steps / 4 ): break # Mark that a worker is done collecting experience num_workers_done.add("done", 1) # Update the model using PPO for _ in range (n_ppo_epochs): for _ in range (n_ppo_batch): batch = get_batch() loss =
evaluate(model , batch) loss.backward() #DistributedDataParallel automatically # performs an AllReduce on all gradients # during the backward call.
# If this wasn’t being used , here is where # calls to AllReduce on the gradients would # be made.
step
[Figure] [Figure]
with GPS+Compass on the Habitat Challenge 2019 (Savva et al., 2019).
Gibson (Xia et al.
, 2018)-4+ refers to the subset of Gibson train scenes with a quality rating of 4 or better.
Gibson-4+ and MP3D refers to training on both Gibson-4+ and all of Matterport3D. [-Gibson-
-]{+Gibson
+}
+ refers to training on the subset of Gibson train scenes with a quality rating of 2 or better.
[Figure] [Figure]
ghe r
[Figure]
episodes within each geodesic distance bin.