Original: Variable Experience Rollout: Learning Robust Skills for Embodied Rearrangement
Revised: Variable Experience Rollout: Learning Robust Skills for Embodied Rearrangement

Original: Abstract
Revised: Abstract

Original: We present Variable Experience Rollout ( VER ), a technique for efﬁciently scalingbatched on-policy reinforcement learning in heterogenous environments (wheredifferent environments take vastly different times for generating rollouts) to many GPUs residing on, potentially, many machines.
Revised: We present Variable Experience Rollout ( VER ), a technique for scaling batchedon-policy reinforcement learning in heterogenous environments (where differentenvironments take vastly different times for generating rollouts).

Original: VER combines the strengths ofand blurs the line between synchronous and asynchronous on-policy RL methods( SyncOnRL  and  AsyncOnRL , respectively). Speciﬁcally, it learns from on-policyexperience (like SyncOnRL ) and has no synchronization points (like AsyncOnRL ),enabling high throughput.
Revised: VER combines thestrengths of and blurs the line between synchronous ( SyncOnRL ) and asynchronous( AsyncOnRL ) on-policy RL methods – specifically, it learns from on-policy experience but has no synchronization points, enabling high throughput.

Original: We ﬁnd that VER leads to signiﬁcant and consistent speed-ups across a broadrange of embodied navigation and mobile manipulation tasks in photorealistic 3Dsimulation environments.
Revised: We find that VER leads to significant and consistent speed-ups across a broadrange of embodied navigation and mobile manipulation tasks in photorealistic 3Dsimulation environments.

Original: Speciﬁcally, for PointGoal navigation and ObjectGoalnavigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO,the current state of art for distributed SyncOnRL , with similar sample efﬁciency.
Revised: Specifically, for PointGoal navigation and ObjectGoalnavigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) over DD-PPO, thecurrent state of art for distributed SyncOnRL .

Original: For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup)on 8 GPUs than DD-PPO.
Revised: For mobile manipulation tasks (openfridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup)on 1 GPU and 200% faster (3x speedup) on 8 GPUs with similar or better sampleefficiency.

Original: Compared to SampleFactory (the current state-of-the-art AsyncOnRL ), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup)on 8 GPUs with better sample efﬁciency.
Revised: Compared to SampleFactory ( AsyncOnRL ), VER matches its speed on1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efficiency.

Original: We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB).
Revised: We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB).

Original: We ﬁnd a surprisingemergence of navigation in skills that do not ostensible require any navigation.
Revised: We find a surprisingemergence of navigation in skills that do not ostensible require any navigation.

Original: Speciﬁcally, the Pick skill involves a robot picking an object from a table.
Revised: Specifically, the pick skill involves a robot picking an object from a table.

Original: Duringtraining the robot was always spawned close to the table and never needed tonavigate.
Revised: Duringtraining the robot was always spawned close to the table and never needed tonavigate.

Original: However, we ﬁnd that if base movement is part of the action space, therobot learns to navigate then pick an object in new environments with 50% success,demonstrating surprisingly high out-of-distribution generalization.
Revised: However, we find that if base movement is part of the action space, therobot learns to navigate then pick an object in new environments with 50% success,demonstrating surprisingly high out-of-distribution generalization.

Original:  Introduction
Revised:  Introduction

Original: Scaling matters.
Revised: Scaling matters.

Original: Progress towards buildingembodied intelligent agents that are capable of performing goal driven tasks has been driven, in part, by training large neural networks in photo-realistic 3D environments with deep reinforcement learning (RL) for (up to) billions of steps of experience [Wijmanset al., 2020, Maksymets et al., 2021, Mezghani et al., 2021, Ramakrishnan et al., 2021, Miki et al.,2022].
Revised: Progress towards building intelligent agents that are capable of performing goaldriven tasks has been driven, in part, by training large neural networks in photo-realistic 3D environments with deep reinforcement learning (RL) for (up to) billions of steps of experience [Wijmanset al., 2020, Maksymets et al., 2021, Mezghani et al., 2021, Ramakrishnan et al., 2021, Miki et al.,2022].

Original: To enable this scale, RL systems must be able to efﬁciently utilize the available resources ( e.g . GPUs), and scale to multiple machines all while maintaining sample-efﬁcient learning.
Revised: To enable this scale, RL systems must be able to efficiently utilize their resources, and scale tomultiple machines all while maintaining sample-efficient learning.

Original: One promising class of techniques to achieve this scale is batched on-policy RL.
Revised: One promising class of techniques to achieve this scale is batched on-policy RL.

Original: These methods collect experience from many( N ) environments simultaneously using the policy and update it with this cumulative experience.
Revised: These methodscollect experience from many environments simultaneously using the policy and update it with thiscumulative experience.

Original: They are broadly divided into two classes: synchronous ( SyncOnRL )and asynchronous ( AsyncOnRL ).
Revised: These methods are broadly divided into two classes: synchronous ( SyncOnRL )and asynchronous ( AsyncOnRL ).

Original: SyncOnRL contains two  synchronization points: ﬁrst the policy is {+executed for the entire batch ( o t → a t ) Bb =
 (Fig. 1 A), then actions are executed in all+}
Revised: SyncOnRL contains two potential synchronization points: first thepolicy is {+executed for the entire batch ( o t → a t ) Bb =
 (Fig. 1 A), then actions are executed in all+}

Original: [Figure]
Revised: [Figure]

Original: AsyncOnRL
Revised: AsyncOnRL

Original: Learning Learning

Original: [Figure]
Revised: [Figure]

Original: Simulation
Revised: Simulation

Original: D
Revised: D

Original: E +30% executed for the batch (  t a t )
Revised: E +30% environments, ( s t → s +1 , +1 )

Original: Bb =1 1 (Fig.
Revised: Bb =1 1 (Fig.

Original: 1 A), then actions are executed in all environments,( s t , a t → s t +1 , o t +1 )

Original: [-Bb =
-]
Revised: [-Bb =
-]

Original: (Fig. 1 B), until Tsteps have been collected from all N environments.
Revised: 1 B), until T collected from all N environments.

Original: This ( T, N ) -shaped batch of experience is used to update the policy (Fig. 1 C).
Revised: This ( T, N ) -shaped batch of experience is used to update the policy (Fig. 1 C).

Original: Synchronizationreduces throughput as the system spends signiﬁcant (sometimesthe most) time  waiting for the slowestenvironment to ﬁnish. This is the straggler effect [Petrini et al., 2003, Dean and Ghemawat, 2004].
Revised: The synchronization points reduce throughput due to the straggler effect, where the system spendssignificant (sometimes most) time idling, waiting for the slowest worker.

Original: AsyncOnRL removes these synchronization points, thereby mitigating the straggler effect and improving throughput.
Revised: AsyncOnRL remove these synchronization points, thereby mitigating the straggler effect and improvingthroughput.

Original: Actions are taken as soon as they are computed, a t → o t +1 (Fig. 1 D), the nextaction is computed as soon as the observation is ready, o t → a t (Fig. 1 E), and the policy is updatedas soon as enough experience is collected.
Revised: Actions are taken as soon as they are computed, a t → o t +1 (Fig. 1 D), the next actionis computed as soon as the observation is ready, o t → a t (Fig. 1 E), and the policy is updated assoon as enough experience is collected.

Original: However, AsyncOnRL systems are not able to ensure thatall experience has been collected by only the current policy and thus must consume near -policydata.
Revised: However, AsyncOnRL systems are not able to ensure thatall experience has been collected by only the current policy and thus must work with near -policydata.

Original: This reduces sample efﬁciency [Liu et al., 2020].
Revised: This reduces sample efficiency [Liu et al., 2020].

Original: Thus, status quo leaves us with an unpleasanttradeoff – high sample-efﬁciency with low throughput or high throughput with low sample-efﬁciency.
Revised: Thus, status quo leaves us with an unpleasanttradeoff – on-policy data with low throughput or high throughput with low sample-efficiency.

Original: In this work, we propose Variable Experience Rollout ( VER ).
Revised: In this work, we propose Variable Experience Rollout ( VER ).

Original: VER combines the strengths of and blursthe line between SyncOnRL and AsyncOnRL .
Revised: VER combines the strengths of and blursthe line between SyncOnRL and AsyncOnRL .

Original: Like SyncOnRL , VER collects experience with the currentpolicy and then updates it.
Revised: Like SyncOnRL , VER collects experience with the currentpolicy and then updates it.

Original: Like AsyncOnRL , VER does not have synchronization points – it computesnext actions, steps environments, and updates the policy as soon as possible.
Revised: Like AsyncOnRL , VER does not have synchronization points – it computesnext actions, steps environments, and updates the policy as soon as possible.

Original: The inspiration for
Revised: The inspiration for

Original: VER comes from two key observations:
Revised: VER comes from two key observations:

Original: 1) AsyncOnRL mitigates the straggler effect by implicitly collecting a variable amount of experience
Revised: 1) AsyncOnRL mitigates the straggler effect by implicitly collecting a variable amount of experience

Original: from each environment –  more from fast-to-simulate environments and less from slow ones.
Revised: from different environments – collecting more from fast environments and less from slow ones.

Original: 2) Both SyncOnRL and AsyncOnRL use a ﬁxed rollout length, T steps of experience.
Revised: 2) Both SyncOnRL and AsyncOnRL collect rollouts of experience with a fixed number of steps per

Original:  Our key insight 
Revised: environment, e.g . ( T, N ) 2 . Our key insight is that an equal distribution of T steps may simplify an

Original: is that while a ﬁxed rollout length may simplify an implementation, it is not a requirement for RL.
Revised: implementation but is not a requirement for RL.

Original: These two key observations naturally lead us to variable experience rollout ( VER ), i.e . collectingrollouts with a variable number of steps.
Revised: This insight naturally leads us to variable experience rollouts ( VER ), i

Original: VER adjusts the rollout length for each environment basedon its simulation speed.
Revised: e .

Original: It explicitly collects more experience from fast-to-simulate environmentsand less from slow ones (Fig. 1).
Revised: collecting rollouts with a variable number of steps across different environments.

Original: The result is an RL system that overcomes the straggler effectand maintains sample-efﬁciency by learning from on-policy data.
Revised: The result is an RL system that overcomes the straggler effect

Original: VER focuses on efﬁciently utilizing a single GPU. To enable efﬁcient scaling to multiple GPUs, wecombine VER with the decentralized distributed method proposed in [Wijmans et al., 2020].
Revised: and maintains sample-efficiency.

Original: First, we evaluate VER on well-established embodied navigation tasks using Habitat 1.0 [Savva et al.
Revised: First we evaluate VER on well-established embodied navigation tasks using Habitat 1.0 [Savva et al.

Original: ,2019] on 8 GPUs. VER trains PointGoal navigation [Anderson et al., 2018] 60% faster  than Decentralized Distributed PPO (DD-PPO)
Revised: VER trains PointGoal navigation [Anderson et al., 2018] 60% faster (1.6x speedup) than Decentralized Distributed PPO (DD-PPO)

Original: [Wijmans et al., 2020], the current state-of-the-art for distributedon-policy RL, with the same sample efﬁciency.
Revised: [Wijmans et al., 2020], the current state-of-the-art fordistributed on-policy RL, with the same sample efficiency.

Original: For ObjectGoal navigation [Batra et al., 2020b], anactive area of research, VER is 100% faster than DD-PPO with(slightly) better sample efﬁciency.
Revised: For ObjectGoal navigation [Batra et al. ,2020b], an active area of research, VER trains 100% faster than DD-PPO with better sample efficiency.

Original: Next, we evaluate VER on the recently introduced (and signiﬁcantly more challenging) GeometricGoalrearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].
Revised: Next, we evaluate VER on the recently introduced (and significantly more challenging) GeometricGoalrearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].

Original: In GeoRearrange, a virtualrobot is spawned in a new environment and asked to rearrange a set of objects from their initial todesired coordinates.
Revised: In GeoRearrange, a virtualrobot is spawned in a new environment and asked to rearrange a set of objects from their initial todesired coordinates as efficiently as possible.

Original: These environments have highly variable simulation time (physics simulation time increases if the robot bumps into something) and require GPU-acceleration (for photo-realisticrendering), limiting the number of environments that can be run in parallel.
Revised: These environments have highly variable simulation time(physics simulation time increases if the robot bumps into something) and require GPU-acceleration(for photo-realistic rendering), limiting the number of environments that can be run in parallel.

Original: On 1 GPU, VER is 150% faster (2.5x speedup) than DD-PPO with the same sample efﬁciency.
Revised: On 1 GPU, VER is 150% faster (2.5x speedup) than DD-PPO with the same sample efficiency.

Original: VERis as fast as SampleFactory [Petrenko et al., 2020], the state-of-the-art AsyncOnRL , with the samesample efﬁciency.
Revised: VERis as fast as SampleFactory [Petrenko et al., 2020], the fastest single machine AsyncOnRL , with thesame sample efficiency.

Original: VER is as fast as AsyncOnRL in pure throughput; this is a surprisingly strongresult.
Revised: VER is as fast as AsyncOnRL in pure compute-efficiency, which is a high bar.

Original: AsyncOnRL never stops collecting experience and should, in theory, be a strict upper boundon performance. VER is able to match AsyncOnRL for environments that heavily utilize the GPU forrendering, like Habitat. In AsyncOnRL , learning, inference, and rendering contend the GPU whichreduces throughput.
Revised: We then combine VER with DD-PPO to scale to multiple GPUs.

Original: In VER , inference and rendering contend for the GPU while learning does not.
Revised: VER achieves better scaling than

Original: On 8 GPUs, VER achieves better scaling than DD-PPO, achieving a 6.7x speed-up  (vs. 6x for DD-PPO)due to lower variance in experience collection time between GPU-workers.
Revised: DD-PPO, achieving a 6.7x speed-up on 8 GPUs (vs. 6x for DD-PPO) due to lower variance inexperience collection time between GPU-workers.

Original: Due to this efﬁcientmulti-GPU scaling, VER is 70% faster(1.7x speedup) than SampleFactory on 8 GPUs and has bettersample efﬁciency as it learns from on-policy data.
Revised: Due to highly efficient multi-GPU scaling, VER is70% faster than SampleFactory on 8 GPUs and more sample-efficient.

Original: Finally, we leverage these SysML contributions to study open research questions posed in prior work.
Revised: We leverage these SysML contributions to study open research questions posed in prior work.

Original: Speciﬁcally, we train RL policies for mobile manipulation skills ( Navigate , Pick , Place , etc .)
Revised: Specifically, we train RL policies for mobile manipulation skills (navigation, pick, place, etc .)

Original: andchain them via a task planner.
Revised: andchain them via a task planner.

Original: Szot et al.
Revised: Szot et al.

Original: [2021] called this approach TP-SRL and identiﬁed a critical‘handoff problem’ – downstream skills are set up for failure by small errors made by upstream skills( e.g . the Pick skill failing because the navigation skill stopped the robot  a bit too far from the object ).
Revised: [2021] called this approach TP-SRL and identified a critical‘handoff problem’ – downstream skills are set up for failure by small errors made by upstream skills( e.g ., the robot stopping a bit too far to pick up an object).

Original: We demonstrate a number of surprising ﬁndings when TP-SRL is scaled via VER .
Revised: We demonstrate a number of surprising findings when TP-SRL is scaled via VER .

Original: Most importantly,we ﬁnd the emergence of navigation when skills that do not ostensibly require navigation ( e.g .pick) are trained with navigation actions enabled.
Revised: Most importantly,we find the emergence of navigation when skills that do not ostensibly require navigation ( e.g .pick) are trained with navigation actions enabled.

Original: In principle, Pick and Place policies do not needto navigate during training since the objects are always in arm’s reach, but in practice they learnto navigate to recover from their mistakes and this results in strong out-of-distribution test-timegeneralization.
Revised: In principle, pick and place policies do not needto navigate during training since the objects are always in arm’s reach, but in practice they learnto navigate to recover from their mistakes and this results in strong out-of-distribution test-timegeneralization.

Original: Speciﬁcally, TP-SRL without a navigation skill achieves 50% success on NavPickand 20% success on a NavPickNavPlace task simply because the Pick and Place skills have learnedto navigate (sometimes across the room!).
Revised: Specifically, TP-SRL without a navigation skill achieves 50% success on NavPickand 20% success on a NavPickNavPlace task simply because the pick and place skills have learnedto navigate (sometimes across the room).

Original: TP-SRL with a Navigate skill performs even stronger:90% on NavPickNavPlace and 32% on 5 successive NavPickNavPlaces (called Tidy House in Szotet al.
Revised: TP-SRL with a navigation skill performs even stronger:90% on NavPickNavPlace and 32% on 5 successive NavPickNavPlaces (called Tidy House in Szotet al.

Original: [2021]), which are +32% and +30% absolute improvements over Szot et al.[2021], respectively.
Revised: [2021]), which are +32% and +30% absolute improvements over Szot et al.

Original: Prepare Groceries and Set Table, which both require interaction with articulated receptacles (fridge,drawer), remain as open problems(5% and 0% Success, respectively) and are the next frontiers.
Revised: Prepare Groceries and Set Table, which both require interaction with articulated receptacles (fridge, drawer),remain as open problems and are the next frontiers.

Original:  VER : Variable Experience Rollout
Revised:  VER : Variable Experience Rollout

Original: The key challenge that any batched on-policy RL technique needs to address is variability of simulation time for the environments in a batch.
Revised: The key challenge that any large-batch RL technique needs to address is variability of simulationtime for the environments in a batch.

Original: There are two primary sources of this variability:action-level and episode-level.
Revised: There are two primary sources of this variability: action-leveland episode-level.

Original: The amount of time needed to simulate an action within an environment varies depending on the the speciﬁc action, the state of the robot, and the environment( e.g . simulating the robot navigating on a clear ﬂoor is much faster than simulating the robot’s armcolliding with objects).
Revised: The amount of time needed to simulate an action within an environment variesdepending on the the specific action, the state of the robot, and the environment ( e.g . simulating therobot navigating on a clear floor is much faster than simulating the robot’s arm colliding with objects).

Original: The amount of time needed to simulate an entire episode also varies environment to environment irrespective of action-level variability ( e.g . rendering images takes longer forvisually-complex scenes, simulating physics takes longer for scenes with a large number of objects).
Revised: The amount of time needed to simulate an entire episode also varies environment to environmentirrespective of action-level variability ( e.g . rendering images can take longer for visually-complexscenes, simulating physics can take longer for scenes with a large number of objects).

Original: 2.1 Action-Level Straggler Mitigation
Revised: 2.1 Action-Level Straggler Mitigation

Original: We mitigate the action-level straggler effect by applying the experience collection method of
Revised: We mitigate the action-level straggler effect by applying the experience collection technique of

Original: AsyncOnRL to SyncOnRL .
Revised: AsyncOnRL to SyncOnRL .

Original: We represent this visually in Fig. A3 and describe it in text bellow.
Revised: There are two components, policy workers and actor workers.

Original: Environment workers receive the next action  and step the environment, EnvStep, e.g . s t , a t →s t +1 , o t +1 , r t .
Revised: Actor workers receive the next action in their environment and step the environment ( a t → o t +1 ).

Original: They write the outputs of the environment (observations, reward, etc .) into preallocated CPU shared memory for consumption by inference workers.
Revised: They write the outputs of the environment (observations, reward, and whether or not the episodeterminated) into pre-allocated CPU shared memory for consumption by the policy worker.

Original: Inference workers receivebatches of steps of experience from environment workers.
Revised: Policy workers receive steps of experience from actor workers.

Original: They performinference with the current policy to select the next action and send it to the environment worker usingpre-allocated CPU shared memory.
Revised: They perform inference with thecurrent policy to select the next action and send it to the actor worker using pre-allocated CPU sharedmemory.

Original: After inference they store experience for learning in shared GPUmemory.
Revised: After inference policy workers store experience for learning in shared GPU memory.

Original: Inference workers use dynamic batching and perform inference on all outstanding inferencerequests instead of waiting for a ﬁxed number of requests to arrive 2 .
Revised: policyworkers use dynamic batching and only process steps after a threshold or a time-limit is reached.

Original: This allows us to leverage thebeneﬁts of batching without introducing synchronization points between environment workers.
Revised: This experience collection technique is similar to that of HTS-RL [Liu et al., 2020] ( SyncOnRL )and SampleFactory Petrenko et al.

Original: compute
Revised: [2020] ( AsyncOnRL ). Unlike both, we do not overlap experiencecollection with learning. This has various system benefits, including reducing GPU memory usageand reducing GPU driver context switching, which improves usage.

Original: N=
Revised: In both SyncOnRL and AsyncOnRL , the experience used for learning consists of N sets of T stepsof experience, an ( T, N ) -shaped batch. In SyncOnRL these N sets are all collected with the currentpolicy, this the environment-level straggler effect. AsyncOnRL mitigates this by relaxing the constraintthat experience must be strictly on-policy, and thereby implicitly changes the experience collectionrate for each environment.

Original: N=
Revised: We instead relax the constraint that we must use N sets of T steps. Specifically, VER collects T × Nsteps of experience from N environments without a constraint on how many steps of experience arecollected from each environment (see Fig. 2 (A)). This explicitly varies the experience collection

Original: N=
Revised: N=

Original: N=    (A) 3 0 13 0 1 05 7 2 

Revised: N= N= N=   (A) 3 0 13 0 1 05 7 2

Original: N= N=
Revised: N= N=

Original: N= N=
Revised: N= N=

Original: N=0 (Navigating)
Revised: N=0 (Navigating)

Original: N=1 (Picking) (B)
Revised: N=1 (Picking) (B)

Original: Packed Sequence Format 3 0 13 0 10 1 0 0 0 0 1 1 1 2 2 2 

Revised: Packed Sequence Format 3 0 13 0 10 1 0 0 0 0 1 1 1 2 2 2

Original: T=0 T=1 T=

Revised: T=0 T=1 T=

Original: T=3 N=
Revised: T=3 N=

Original: N=
Revised: N=

Original: N= N=
Revised: N= N=

Original: T= T=
Revised: T= T=

Original: T= T=
Revised: T= T=

Original: [Figure]
Revised: [Figure]

Original: This experience collection technique is similar to that of HTS-RL
Revised: rate for each environment – in effect, collecting more experience from environments that are fast tosimulate.

Original: [Liu et al., 2020] ( SyncOnRL )and SampleFactory [ Petrenko et al., 2020] ( AsyncOnRL ). Unlike both, we do not overlap experiencecollection with learning. This has various system beneﬁts, including reducing GPU memory usageand reducing GPU driver contention.
Revised: Compared to SyncOnRL , VER does not suffer from the environment-level straggler effect.

Original: More details are available in Appendix A. 2.2 Environment-Level Straggler Mitigation In both SyncOnRL and AsyncOnRL , the data used for learning consists of N rollouts of equal- T stepsof experience, an ( T, N ) -shaped batch. In SyncOnRL these N sets are all collected with the currentpolicy, this leads to the environment-level straggler effect.
Revised: Compared to AsyncOnRL , VER does not require correction for stale data.

Original: AsyncOnRL mitigates this by relaxing theconstraint that experience must be strictly on-policy, and thereby implicitly changes the experiencecollection rate for each environment.

Original: Variable Experience Rollout ( VER ). We instead relax the constraint that we must use N rollouts ofequal- T steps. Speciﬁcally, VER collects T × N steps of experience from N environments withouta constraint on how many steps of experience are collected from each environment.
Revised: One subtle design choice is the following – when VER finishes a ( T × N ) rollout collection, there will be(slow) environments that haven’t completed simulation yet.

Original: This explicitlyvaries the experiencecollection rate for each environment – ineffect, collecting more experience fromenvironments that are fast to simulate. Consider the 4 environments shows in Fig. 2A.
Revised: Instead of discarding that data, we chooseto collect this experience in the next rollout.

Original: The length ofthe each step representation the wall-clock time taken to collect it, some steps are fast, some are slow. VER collects more experiencefrom environment 0 as it is fastest to step and less from 1, the slowest.
Revised: This experience is at most 1 policy-update old, containsat most N − 1 steps, and we find this choice leads to speed gains without any sample-efficiency loss.

Original: Learning mini-batch creation.
Revised: Learning mini-batch creation.

Original: VER is designed with recurrent policies in mind because memoryis key in long-range and partially observable tasks like HAB.

Original: When training  recurrent policies, wemust create mini-batches of experience with sequences for back-propagation-through-time.
Revised: When training a recurrent policy, we must create batches ofexperience with sequences for back-propagation-through-time (BPTT).

Original: Normally B mini-batches are constructed by spitting the  N environments’ experience into B (T , N / B )-sizedmini-batches.
Revised: Normally, the ( T, N ) shapedbatch of experience is divided between the mini-batches along the environment ( N ) axis.

Original: A similar procedure would result in mini-batches of different sizes with VER .
Revised: A similarprocedure would result in mini-batches of different sizes for VER .

Original: Thiswould harm optimization because learning rate and optimization mini-batch size are intertwined andautomatically adjusting the learning rate is an open question [Goyal et al., 2017, You et al., 2020].
Revised: We instead divide the sequences ofexperience (splitting sequences if needed).

Original: To under VER ’s mini-batching, ﬁrst note that there are two reasons for the start of a new sequenceof experience: rollout starts (Fig.
Revised: First, note that there are two reasons for the start of a new sequence of experience. Rollout starts(Fig.

Original: 2B, step 0) and episode starts (Fig.
Revised: 2 (B), step 0) and episode starts (Fig.

Original: 2B, a step after a bar).
Revised: 2 (B), a step after a bar).

Original: Thesetwo boundaries types are independent – episodes can end at any arbitrary step within the rollout andthen that environment will reset and start a new episode.
Revised: These two boundaries typesare independent – episodes can start at any arbitrary step within the rollout.

Original: Thus when we collect experience from Nenvironments, we will have K ≥ N sequences to divide between the mini-batches.
Revised: Thus when we collectexperience from N environments, we will have K ≥ N sequences to divide between the mini-batches.

Original: We distributedthese K sequences between the mini-batches.
Revised: We distributed these K sequences between the mini-batches.

Original: We randomly order the sequences, then the ﬁrst T × N / Bsteps are the ﬁrst mini-batch,  the next T × N / B to  the second, etc .
Revised: We use a greedy algorithm that seeks to1) maximizes the number of environments in each mini-batch, 2) minimizes the number of sequencesthat need to be cut, and 3) cuts the shortest sequences first.

Original: See Fig.
Revised: See Fig.

Original: 2B for an example.
Revised: 2 (B) for an example.

Original: Batching computation for learning.
Revised: Batching computation for learning.

Original: The mini-batches constructed from the algorithm above havesequences with variable length.
Revised: The mini-batches constructed from the algorithm above havesequences with variable length.

Original: To batch the computation of these sequences we use cuDNN’s
Revised: To batch the computation of these sequences we use cuDNN’s

Original: PackedSequence data model.
Revised: PackedSequence data model.

Original: This data model represents a set ofvariable-length sequences (Fig.
Revised: This data model represents a set of sequences with variable length(Fig.

Original: 2Cleft) such that all sequence-elements at a  time-step arecontiguous in memory (Fig.
Revised: 2 (C) left) such that all elements for a given time-step are in a contiguous block of memory(Fig.

Original: 2C right)
Revised: 2 (C) right)

Original: – enablingbatched computation on each time-step forcomponents with a temporal dependence, e.g . the RNN –and that all elements across all time-steps are also contiguous – enabling batched computation acrossall time-steps for network components that don’t have a temporal dependence, e.g . the visual encoder.
Revised: – this enables efficient batched computation on each time-step. Further, it uses acontiguous block of memory for all elements. This enables batched computation on all elements fornetwork components that don’t have a temporal aspect, such the visual encoder.

Original: During experience collection we write experience into a linear buffer in GPU memory and then arrangeeach mini-batch as a PackedSequence .
Revised: During experience collection we write experience into a linear buffer. After experience collectionwe arrange each mini-batch as a PackedSequence .

Original: This takes less than 10 milliseconds (per learning phase);orders of magnitude less than experience collection ( ∼ 3s) or learning ( ∼ 1.5s) in our experiments. Learning method.
Revised: This takes less than 10 milliseconds (per rolloutphase); orders of magnitude less than experience collection ( ∼ 3s) or learning ( ∼ 1.5s).

Original: The experience and mini-batches generated from VER are well-suited for use with

Original: RL methods that use on-policy data [Sutton and Barto, 1992].

Original: We use Proximal Policy Optimization(PPO)

Original: [Schulman et al., 2017] as it is known to work well for embodied AI tasks and recurrent policies.

Original: 2.3 Multiple GPUs
Revised: 2.3 Multiple GPUs

Original: We leverage the decentralized distributed training architecture from Wijmans et al. [2020] to scale VERto multiple GPUs (residing on a single or multiple nodes).
Revised: We apply the decentralized distributed method from DD-PPO [Wijmans et al., 2020] to scale oursystem to multiple GPUs.

Original: In this architecture, each GPU both collectsexperience and learns from that experience.
Revised: We adapt and build upon the straggler preemption method presentedtherein. During experience collection, we track when each step of experience was added tothe learning buffer. We then compute the average time to collect a step of experience for eachenvironment.

Original: During learning, gradients from each GPU-worker areaveraged with an AllReduce operation.
Revised: Finally, we compute the optimal number of steps of experience to collect for thatrollout and use it to decide when to preempt on the next rollout. We find this method to be morerobust than the fixed threshold used in DD-PPO as it is able to adapt to changes in simulation timethroughout learning. Unlike DD-PPO we don’t reduce the learning batch-size for workers that don’tcollect the full amount of experience. We fill the rollout with experience from the previous rollout.

Original: We further describe our implementation in the Appendix.
Revised: This improves performance when using cuDNN as this remembers what kernels worked best for eachbatch-size. When we copy experience from a previous rollout, we don’t recompute the return. Ineffect, we perform extra epochs of PPO on this experience. This comes with no effective computation cost since the result is that all GPUs have the same batch size instead of some having a smaller batchsize.

Revised: In practice we never see the faction of previous experience exceed 1%.

Original:  Embodied Navigation: Benchmarking
Revised:  Embodied Navigation: Benchmarking

Revised: Task

Revised: DD-PPO

Original:  First, we benchmark VER on the embodied navigation [Anderson et al., 2018] tasks in Habitat1.0 [Savva et al., 2019] – PointNav [Anderson et al., 2018] and ObjectNav [Batra et al., 2020b].
Revised: VER PointGoalNav ObjectNav First, we benchmark VER on the embodied navigation [Anderson et al., 2018] tasks in Habitat1.0 [Savva et al., 2019] – PointNav [Anderson et al.

Revised: and ObjectNav [Batra et al., 2020b].

Original: Ourgoal here is simply to show training speed-ups in well-studied tasks (and in the case of PointNav, awell-saturated task with no room left for accuracy improvements).
Revised: Our goalhere is simply to show training speed-up in wellstudied tasks (and in the case of PointNav, a wellsaturated task with no room left for accuracy improvements).

Original: We present accuracy improvementsand in-depth analysis on challenging rearrangement tasks in Section 6.Task [Table]
Revised: We present accuracy improvementsand in-depth analysis on rearrangements tasks in Section 6.

Original: [Figure]
Revised: [Figure]

Original: efﬁciency than DD-PPO, indicating that performance is not negatively impacted by the non-uniform sampling of experience from environments.

Original: Shading is a 95% conﬁdence interval over 3 seeds.

Original: For both tasks, we use standard architectures from
Revised: For both tasks, we use standard architectures from

Original: Habitat Baselines [Savva et al., 2019, Wijmans et al.
Revised: Habitat Baselines [Savva et al., 2019, Wijmans et al.

Original: ,2020] – the ResNet18 encoder and a 2 layer LSTM.
Revised: ,2020] – the ResNet18 encoder and a 2 layer LSTM.

Original: Following Ye et al.
Revised: Following Ye et al.

Original: [2020, 2021], we add Action
Revised: [2020, 2021], we add Action

Original: Conditional Contrastive Coding [Guo et al., 2018],using the hyper-parameters from Guo et al.
Revised: Conditional Contrastive Coding [Guo et al., 2018],using the hyper-parameters from Guo et al.

Original: PointNav.
Revised: PointNav.

Original: We train PointNav agents with one RGBcamera on the HM3D dataset [Ramakrishnan et al.
Revised: We train PointNav agents with an RGBcamera on the HM3D dataset [Ramakrishnan et al.

Original: ,2021] for 1.85 billion steps of experience on 8 GPUs.
Revised: ,2021] for 1.85 billion steps of experience on 8 GPUs.

Original: We study the RGB setting (and not Depth ) becausethis is the more challenging version of the task andthus we expect it to be more sensitive to possibledifferences in the training system.
Revised: We examine RGB as this the more challenging versionof the task and thus we expect it to be more sensitiveto possible differences in the training system.

Original: We examine VERalong two axes: 1) training throughput – the numberof samples of experience per second (SPS) the systemcollects and learns from, 2) sample efﬁciency.

Original: [-On 
-]
Revised: [-On 
-]

Original: GPUs, VER trains agents 60% faster than DD-PPO,from 3065 SPS to 5325 SPS (Table 1), with similarsample efﬁciency (Fig. 3).
Revised: VERtrains agents 60% faster than DD-PPO (Table 1) withsimilar sample efficiency (Fig. 3).

Original: ObjectNav.
Revised: ObjectNav.

Original: We train ObjectNav agents with one RGBand one Depth camera on the MP3D [Chang et al. ,2017] dataset for 600 million steps of experience on8{+on 
+} GPUs.
Revised: We train ObjectNav agents with RGB and Depth cameras on the MP3D+[Chang et al., 2017] dataset for 600 million steps of experience {+on 
+} GPUs.

Original: VER trains agents 100% faster than DD-PPO,1021 to 2019 (Table 1), withslightly better sampleefﬁciency (Fig. 3).
Revised: VER trains agents 100% faster than DD-PPO with better sample efficiency.

Original: There are two effects that enablebetter sample efﬁciency with VER .
Revised: There are twoeffects that enable better sample efficiency with VER .

Original: First, we performadditional epochs of PPO on experience from the last rollout when a GPU-worker is preempted.
Revised: First, we perform additional epochs of PPO onexperience from the last rollout when a GPU-worker is preempted.

Original: Second, the variable experience rollout mechanism results in a natural curriculum.
Revised: Second, the variable experiencerollout mechanism results in a natural curriculum.

Original: Environmentsare often faster to simulate when they are easier ( i.e . a smaller home), so more experience will becollected in these easier cases.
Revised: Environments are often faster to simulate whenthey are easier ( i.e . a smaller home), so more experience will be collected in these easier cases.

Original:  Embodied Rearrangement: Task, Agent, and Training
Revised:  Embodied Rearrangement: Task, Agent, and Training

Original: Next, we use VER to study the recently introduced (and more challenging) GeometricGoal rearrangement rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].
Revised: Next, we use VER to study the recently introduced (and more challenging) GeometricGoal rearrangement rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].

Original: Task.
Revised: Task.

Original: In GeoRearrange, an agent is initialized in an unknown environment and tasked with rearrangingobjects in its environment.
Revised: In GeoRearrange, an agent is initialized in a novel unseen environment and tasked with rearranging its environment.

Original: The task is speciﬁed as a set of coordinate pairs { ( Pose Initial , Pose Final ) } Oo =1 .
Revised: The task is specified as a set of coordinate pair { ( Pose Initial , Pose Final ) } Oo =1 .

Original: The agent must bring each object at Pose Initial to Pose Final where Pose is the initial or desired centerof-mass location for the object(s).
Revised: The agent must bring each object at Pose Initial to Pose Final .

Original: We use the Home Assistant Benchmark (HAB) which consists of3 scenarios of increasing difﬁculty: Tidy House, Prepare Groceries, and Set Table.
Revised: HAB consists of 3 scenarios of increasingdifficulty: Tidy House, Prepare Groceries, and Set Table.

Original: Simulation.
Revised: Simulation.

Original: We use the Habitat simulatorwith the ReplicaCAD Dataset [Savva et al., 2019, Szotet al., 2021].
Revised: We use the Habitat simulator [Savva et al., 2019, Szot et al., 2021].

Original: The robot policy operates at 30 Hz and physics is simulated at 120 Hz.
Revised: The robot policyoperates at 30 Hz and physics is simulated at 120 Hz.

Original: Agent.
Revised: Agent.

Original: The agent is embodied as a Fetch robot with a 7-DOF arm.
Revised: The agent is embodied as a Fetch robot with a 7-DOF arm.

Original: The arm is controlled via jointvelocities.
Revised: The arm is controlled via jointvelocities.

Original: At every time step the policy predicts a delta in motor position for joint in the  arm.
Revised: At every time step the policy predicts a delta in motor position for each of the 7 joints inthe arm.

Original: We ﬁndjoint velocity control equally easy to learn but faster to simulate than the end-effector control used in Szot et al.
Revised: We find joint velocity control equally easy to learn but faster to simulate than the end-effectorcontrol used in Szot et al.

Original: The arm is equipped with a suction gripper.
Revised: The arm is equipped with a suction gripper.

Original: The agent must control the arm such thatthe gripper is in contact with the object to grasp and then activate the gripper.
Revised: The agent must controlthe arm such that the gripper is in contact with the object to grasp and then activate the gripper.

Original: The object is droppedonce the gripper is deactivated.
Revised: Theobject is dropped once the gripper is deactivated.

Original: This is more realistic than the ‘magic’ grasp action used in Szot et al.[2021].
Revised: This is more realistic than the ‘magic’ grasp actionused in Szot et al.

Original: The robot base(navigation) is controlled by the policy commanding a desired linear speed andangular velocity.
Revised: The robot base is controlled by the policy commanding a desired linearspeed and angular velocity.

Original: The robot is equipped with one Depth camera attached to its head, proprioceptivesensors that provide the joint positions of its arm, and a GPS+Compass sensor that provides its headingand location relative to its initial location.
Revised: The robot is equipped with a singular Depth camera attached to its head.

Original: The policy models a t ∼ π ( · | s t − 1 ) instead of a t ∼ π ( · |s t ) .
Revised: The policy models a t ∼ π ( · | s t − 1 ) instead of a t ∼ π ( · | s t ) .

Original: This is  more realistic and enables physics and rendering to be overlapped [Szot et al., 2021]. We build upon the TaskPlanning-SkillRL
Revised: This is both more realistic and enablesphysics and rendering to be overlapped [Szot et al., 2021]. [Figure]

Original: (TP-SRL) method proposed in Szot et al. TP-SRLis a hierarchical method for GeoRearrange that decomposes the task into a series of skills – Navigate , Pick , Place , and { Open , Close } × { Cabinet, Fridge } .
Revised: build upon

Original: Skills are controlled via a skill-policy (learnedwith RL) and chained together via a task planner. One of the key challenges is the ‘handoff problem’– downstream skills are setup for failure due to slight errors made by the upstream skill. We give allskill policies access tonavigation actions toallow themto correct for these errors.
Revised: TaskPlanning- for GeoRearrange task into a series of and { to to them

Original: Architecture. All skill policies  share the same architecture.
Revised: Our skill policies all share the same architecture.

Original: We use [-ResNet
-] [He et al., 2016]to process the 128 × 128 visual input.
Revised: use [-ResNet
-]ResNet18 [He et al to process the 128 × 128 visual input.

Original: Following Wijmans et al.
Revised: Following Wijmans et al.

Original: [2020], we reduce with width ofthe network by half and use GroupNorm [Wu and He, 2018].
Revised: [2020], we reduce with width ofthe network by half and use GroupNorm [Wu and He, 2018].

Original: We also apply some of the recentadvancements from ConvNeXt [Liu et al., 2022].
Revised: We also apply some of the recentadvancements from ConvNeXt [Liu et al., 2022].

Original: We usea patch-ify stem, dedicated down-samplestages, layer scale[Touvron et al., 2021], and dilated convolutions [Yu and Koltun, 2015] (thismimics larger kernel convolutions without increasing computation).
Revised: We use patch-ify stem, dedicated down-samplestages, layer scale and dilated convolutions (this mimics larger kernel convolutions without increasingcomputation).

Original: The visual embedding is thencombined with the proprioceptive observations and previous action, and then processed with a 2-layer 
Revised: The visual embedding is then combined with the state observations and previous actionand then processed with a 2-layer LSTM

Original: LSTM [Hochreiter and Schmidhuber, 1997].
Revised: [Hochreiter and Schmidhuber, 1997].

Original: The output of the LSTM is used to predict the actiondistribution and value function.
Revised: The output of the LSTM is used to predict the action distribution and value function.

Original: Actions are sampled from independent Gaussian distributions.
Revised: Actions are sampled multivariate Gaussian distribution with a diagonal covariance matrix.

Original: Training.
Revised: Training.

Original: We train agents using VER and PPO [Schulman et al., 2017] with Generalized Advantage Estimation [Schulman et al., 2016].
Revised: We train agents using VER and Proximal Policy Optimization [Schulman et al., 2017] with Generalized Advantage Estimation [Schulman et al., 2016].

Original: We use a minimum entropy constraint with a learnedcoefﬁcient [Haarnoja et al., 2018] as we ﬁnd this to be more stable given our diverse set ofskills than a ﬁxed coefﬁcient.
Revised: We use an entropy constraint with alearned coefficient [Haarnoja et al., 2018] as we find this to be more stable given our diverse set ofskills.

Original: Formally, let H ( π ) be the entropy of the  policy, we then minimizeα ( λ − [[  H ( π)]] sg ) − [[α ]] sg H ( π  ) where [[ · ]] sg is the stop gradient operator.
Revised: Formally, let H ( π ) be the entropy of the current policy, we then minimize − [[ α ]] sg H ( π ) +α ( λ + [[ H ( π )]] sg ) where [[ · ]] sg is the stop gradient operator.

Original: We set the target entropy,λ , to zero for all tasks.
Revised: We set λ to zero for all tasks.

Original: We use the Adam optimizer [Kingma and Ba, 2015] with an initial learningrate of 2 . 5 × 10 − 4 and decay it to zero with a cosine schedule.
Revised: We usethe Adam optimizer [Kingma and Ba, 2015] with an initial learning rate of 2 . 5 × 10 − 4 and decaythe learning rate to zero using a cosine schedule.

Original: To correct for  biased sampling in VER ,we use truncated importance sampling weighting [Espeholt et al., 2018] with a maximum of 1.0.
Revised: To correct for the biased sampling of VER , we usetruncated importance sampling weighting [Espeholt et al., 2018] with a maximum of 1.0.

Original:  Embodied Rearrangement: Benchmarking
Revised:  Embodied Rearrangement: Benchmarking

Revised: We examine VER along two axes: 1) training throughput – the number of samples of experience persecond (SPS) the system collects and learns from, 2) sample efficiency.

Original: In this section, we benchmark VER for training open-fridge policies because this task involvesinteraction of the robot with an articulated object (the fridge) and represents a challenging case for thetraining system due to large variability in physics time.
Revised: In this section, we reportbenchmarking results for the open-fridge policy because this task involves a complex interaction ofthe robot with an articulated object (the fridge) and represents a challenging case for the trainingsystem due to large variability in physics time.

Original: In Section 6, we analyze the task performance,which requires all skills.
Revised: In Section 6, we analyze the task performance, whichrequires all skills.

Original: For all systems, we set the number of environments, N , to 16  per GPU.
Revised: All systems use 16 environments per GPU.

Original: 5.1 System throughout
Revised: 5.1 System throughout

Original: {+RNN HTS-RL No VER +Overlap No VER VER ×  501 620 ✓ - 
+}
Revised: {+RNN HTS-RL No VER +Overlap No VER VER ×  501 620 ✓ - 
+}

Original:  VER  is 150% faster than DD-PPO (Table 2); an even larger difference than in the simpler navigationtasks we studied before.
Revised: [Table] We compare the provided HTS-RL [Liu et al., 2020]implementation 3 with our re-implementation, VERminus variable experience rollouts and overlappedexperience collection and learning (No VER +Overlap). No VER +Overlap is 110% faster than HTS-RL (Table 2).

Revised: Key differences in our implementation are fast userspacemutexes (futexes) in shared memory (vs. spin locks),pre-allocated pinned memory for CPU to GPU transfers(vs.

Revised: allocating for each transfer), and GPU shared

Revised: [Table]

Revised: memory to send experience from policy workers tolearn and weights from learner to policy workers(vs.

Revised: CPU shared memory).

Revised: There has no significant change in system SPS if we remove overlappedexperience collection and learning (Table 2, No VER +Overlap vs. No VER ), while doing so simplifiesthe implementation, and reduces GPU memory usage ( ∼ 2 GB).

Original: DD-PPO has no mechanism to mitigate the action-level or episode-levelstraggler effects.
Revised: We note that overlapped experiencecollection and learning does have uses, i.e . for CPU simulation or policies with significant CPUcomponents, but it isn’t necessary when both the policy and simulator make heavy use of the GPU. No VER is a ‘steel-manned’ baseline for VER and benefits from all our micro-optimizations. The onlydifference is whether we collect a variable experience rollout or not. VER is 30% faster than No VER .

Original: In their absence, DD-PPO has similar throughput as VER (Table 2, Max).
Revised: Both in our HTS-RL comparison (Table 2) and SyncOnRL and AsyncOnRL comparison (Table 3). [Table]

Original: Underthese effects, DD-PPO’s throughput reduces 150% compared to 20% for VER (Table 2, Mean vs. Max).
Revised: [Table] [Table] VER is 150% faster than DDPPO [Wijmans et al., 2020]

Original: Variable experience rollouts are effective.
Revised: (Table 3),the fastest SyncOnRL implementationfor training agents with recurrent policies. DD-PPO has no mechanism tomitigate the action-level or episodelevel straggler effects.

Original: We compare VER with  VER minus variable experiencerollouts (No VER ). No VER is a ‘steel-manned’ baseline for VERand beneﬁts from all our microoptimizations. VER is 30% faster than No VER (Table 2).
Revised: Early in training when the agent is not interacting with objects, DD-PPO has similar throughput as VER (Table 3, Max). Once the robot begins bumping objects the throughput of DD-PPO reduces by 150% compared to 20% for VER (Table 3, Mean vs. Max).

Original: VER closes the gap to AsyncOnRL .
Revised: VER closes the gap to AsyncOnRL .

Original: On 1 GPU, VER is as fast as SampleFactory [Petrenko et al.
Revised: On 1 GPU VER is as fast as SampleFactory [Petrenko et al.

Original: ,2020], the fastest single machine AsyncOnRL .
Revised: ,2020], the fastest single machine AsyncOnRL .

Original: Intuitively AsyncOnRL should be a strict upper-boundon performance – it never stops collecting experience while VER does.
Revised: Intuitively AsyncOnRL should be an upper-bound onperformance – it never stops collecting experience while VER does.

Original: However this doesn’t take intoaccount the realities of hardware.
Revised: However this doesn’t take intoaccount the realities of hardware.

Original: Recall that we are training an agent with a large visual encoder.
Revised: Recall that we are training an agent with a large visual encoder.

Original: This means that updating the parameters of the agent takes a large amount of time ( ∼ 150ms per [Figure] mini-batch of size 1024 on a V100).
Revised: This means that updating the parameters of the agent takes a large amount of time ( ∼ 150ms permini-batch of size 1024 on a V100).

Original: Further, Habitat uses the GPU for rendering.
Revised: Further, Habitat uses the GPU for rendering.

Original: The use of the GPUfor both rendering and learning simultaneously results in  GPU driver contention.
Revised: The use of the GPU for both rendering and learning simultaneously results in context switches by the GPU driver.

Original: In SampleFactory,learning time and experience collection time areroughly double that of VER .[Figure] [Table]
Revised: In SampleFactory, learning time and experience collection time are double that of VER .

Original: Multi-GPU scaling.
Revised: Multi-GPU scaling.

Original: VERhas better multi-GPU scaling than DD-PPO,achieving a 6.7x speed-up on 8 GPUscompared{+
 GPUs compared+} to 6x.
Revised: VER better multi-GPU scaling than DD-PPO, achieving a 6.7x speed-up on {+
 GPUs compared+} to 6x.

Revised: DD-PPO introduces a third straggler effect, the GPU-worker-level, whichtheir straggler preemption seeks to mitigate.

Revised: While that method does work, it isn’t perfect and scalingwould be better if it wasn’t needed.

Original: The rollout collection time in VER is lower variance,which improves scaling.
Revised: Rollouts in VER are lower variance, which improves scaling.

Original: On 2 GPUs, SampleFactory is 12% faster than VER .
Revised: On2 GPUs, SampleFactory is 12% faster than VER .

Original: Here one GPU is used for learning+inference and the other is used forrendering.
Revised: In this case, one GPU is used for learning+inferenceand the other is used for rendering.

Original: This creates a nice divisionof work and doesn’t result in costly GPU contention.
Revised: This creates a nice division of work and doesn’t result in costlycontext switches.

Original: On 4 and 8 GPUs however, the single GPUused for learning in SampleFactory is the bottleneck and VER has higher throughput (nearly 100%faster on 8 GPUs).
Revised: On 4 and 8 GPUs however, the single GPU used for learning in SampleFactoryis the bottleneck and VER has higher throughput (nearly 100% faster 8 GPUs).

Original: While it is possible to implement multi-GPU learning for AsyncOnRL , it is lefttothe user to balance the number of GPUs used for experience collection and learning and this balanceis static.
Revised: While it is possibleto implement multi-GPU learning for AsyncOnRL to overcome this problem, it is left to the user tobalance the number of GPUs used for experience collection and learning.

Original: VER automatically balances GPU-time used for experience and learning continuously.
Revised: VER automatically balances GPU-time used for experience and learning.

Original: 5.2 Sample Efﬁciency
Revised: 5.2 Sample Efficiently

Revised: [Figure]

Original: Next we examine sample efﬁciency of the training systems.
Revised: Next we examine sample efficiency of the training systems.

Original: VER has either identical (1, 2, and 

Revised: VER has either identical (1, 2, and

Original: GPUs), Fig. 4, or better sample efﬁciency (8 GPUs) than DD-PPO.
Revised: GPUs), Fig. 4) or better sample efficiency (8 GPUs) than DD-PPO.

Original: Interestingly, SampleFactoryhas similar sample efﬁciency with 1 GPU (Fig. 4) and is more stable.
Revised: Interestingly, SampleFactoryhas similar sample efficiency with 1 GPU (Fig. 4) and is more stable.

Original: This is because reducing thenumber of GPUs also reduces the batch-size and PPO is known to not be batch-size invariant [Hiltonet al., 2021].
Revised: This is because reducing thenumber of GPUs also reduces the batch-size and PPO is known to not be batch-size invariant [Hiltonet al., 2021].

Original: We believe the stale data serves a similar function to PPO-EWMA [Hilton et al., 2021],which uses an exponential moving average of the policy weights for the trust region.
Revised: We believe the stale data serves a similar function to PPO-EMWA [Hilton et al., 2021],which uses an exponential moving average of the policy weights for the importance sampling terms.

Original: On 2, 4, and [-
-] GPUs, VER has better sample efﬁciency than SampleFactory.
Revised: On 2, 4, and [-
-]8 GPUs, VER has better sample efficiency.

Original: Combing the ﬁndings of throughputand sample efﬁciency, we ﬁnd that VER always achieves a given performance threshold in the leastamount of training time (outright or a tie) (Fig. 4 Lower).

Original:  Embodied Rearrangement: Analysis of Learned Skills
Revised:  Embodied Rearrangement: Analysis of Learned Skills

Revised: [Figure]

Original: We examine the performance of TP-SRL on the Home Assistant Benchmark (HAB)
Revised: Next we examine the performance of TP-SRL on the Home Assistant Benchmark (HAB)

Original: [Szot et al.
Revised: [Szot et al.

Original: We examine both skill policies trained with the full action space and the limited, per-skill
Revised: We examine both skill policies trained with the full action space and the limited and per-skill

Original: [Figure]
Revised: [Figure]

Original: speciﬁc action space used in  et al.
Revised: specific action space used in Szot et al.

Original: Each skill is trained with VER for 500 million steps
Revised: Each skills is trained with VER for 500 million stepsof experience on 8 GPUs.

Original: This  less than  per 
Revised: This takes less than 2 days per skill.

Original: 6.1 Performance on HAB
Revised: 6.1 Performance on HAB

Original: We ﬁnd that  skills with navigation actions improve performance on the full task but does not changeperformance on the skill’s train task ( e.g . Pick achieves 90% success with and without during training).
Revised: We find that using skills with navigation improves performance on the full task but does not changeperformance on the skill’s train task ( e.g . pick achieves 90% Success regardless).

Revised: Skills withnavigation are able to overcome errors of the navigation policy.

Original: Fig.
Revised: Fig.

Original: 5 shows smaller drops in performance between every interaction (there is a navigation betweeneach interaction) demonstrating that skills with navigation effectively correct for handoff errors.
Revised: 5 shows smaller drops inperformance between every interaction (there is a navigation between each interaction).

Original: Thisis impactful after place as the navigation policy tends to make more errors when navigating to thenext location after place.
Revised: This isimpactful after place as the navigation policy tends to make more errors when navigating to the nextlocation after place.

Original: On Tidy House, full task performance improves from 2% Success to 32%.
Revised: On Tidy House, full task performance improves from 2% Success to 32%.

Original: On Prepare Groceries (which requires picking/placing from/into the fridge) and Set Table (whichrequires opening the fridge/cabinet and then picking from it) performance improves slightly.
Revised: On Prepare Groceries (which requires picking/placing from/into the fridge) and Set Table (whichrequires opening the fridge or cabinet and then picking from it) performance improves slightly.

Original: 0%to 5% on Prepare Groceries and 0% to 7.5% on the ﬁrst pick+place on Set Table.
Revised: 0%to 5% on Prepare Groceries and 0% to 7.5% on the first Pick+Place on Set Table.

Original: Both these tasksremain as open problems and the next frontier.
Revised: Both these tasksremain as open problems and the next frontier.

Original: 6.2 Emergent Navigation
Revised: 6.2 Emergent Navigation

Revised: [Figure]

Original: Next we examine if the skill policies are able to navigate to correct for highly out-of-distributioninitial location.
Revised: Next we examine the extent that the skill policies are able to use navigation to correct for out-ofdistribution initial locations.

Original: We examine TP-SRL(NoNav), which that omits the navigation skill.
Revised: We examine an extreme case and construct a version of the TP-SRLagent that omits the Nav skill – TP-SRL(NoNav).

Original: In this agent allnavigation is done by skill policies that ostensibly never needed to navigate during training.
Revised: In this agent all navigation is done by skill policiesthat ostensibly never needed to navigate during training.

Original: We ﬁnd emergent navigation in both the Pick and Place policies.
Revised: We find emergent navigation in both the pick and place policies.

Original: TP-SRL(NoNav) achieves 50%success on NavPick (Fig. 5, Tidy House interaction 1), and 20% success on NavPickNavPlace (Fig. 5,
Revised: On Tidy house, the pick skillssuccessfully navigates and picks up the object 50% of the time on Tidy House (Fig. 5).

Original: Tidy House interaction 2).
Revised: The placepolicy successfully navigates and places the object at the desired location 40% of the time (Fig. The limiting factor for the place policy’s navigation performance is that it often mistakenlydrops the object when navigating, not that it doesn’t know where to go or how to avoid obstacles.

Original: The latter is on-par with the TaskPlanning+SensePlanAct (TP-SPA)classical baseline and signiﬁcantly better than the MonolithicRL baseline in Szot et al.
Revised: This is on-par with the TaskPlanning+SensePlanAct (TP-SPA) classical baseline and significantlybetter than the MonolithicRL baseline in Szot et al.

Original: The Pick and Place policies were trained on tasks that requires no navigation but both are capable ofnavigation.
Revised: The pick and place policy were trained on a task that requires no navigation but both are capable ofnavigation.

Original: We provided examples of both the training task for Pick and Place , and TP-SRL(NoNav)on Tidy House in the supplementary materials.
Revised: We provided examples of both the training task for pick and place, and TP-SRL(NoNav)on Tidy House in the supplementary materials.

Original: On Prepare Groceries and Set Table, the navigation performance of these policies is worse (-34%
Revised: On Prepare Groceries and Set Table, the navigation performance of these policies is worse.

Original: Success and -45% Success on the ﬁrst interaction, respectively).
Revised: Prepare

Original: Prepare Groceries requires pickingfrom the fridge, which is challenging and requires navigation that doesn’t accidentally close the door. Set Table requires opening the cabinet and then picking, which introduces an additional OpenCabskill and requiresmore precise navigation and picking from Pick .
Revised: Groceries requires picking from the fridge while Set Table requires opening the cabinet and thenpicking. This requires navigation that does not accidentally bump the fridge door/cabinet draw andclose it.

Original: Performance is non-zero(15% and4% on the ﬁrst interaction, respectively) in both scenarios; indicting that the skill policies are capableof navigating even in these scenarios, albeit less successfully than in Tidy House.
Revised: Performance is non-zero in both these scenarios however, indicting that the skill policies arecapable of navigating even in these scenarios; albeit less successfully than in Tidy House.

Original: We hypothesize that the Pick and Place polices learned navigation because this was useful early intraining.
Revised: We hypothesize that the pick and place polices learned navigation because this was useful early intraining.

Original: Early in training the policies have yet to learn that only minimal navigation is needed to complete the task.
Revised: Early in training the policies have yet to learn that only minimal navigation is needed to complete the task.

Original: Therefore the policy will sometimes cause itself to move away from the pickobject/place location and will navigate back.
Revised: Therefore the policy will sometimes cause itself to move away from the pickobject/place location and will navigate back.

Original: Navigation is then not forgotten as the policy converges.
Revised: Navigation is then not forgotten as the policy converges.

Original: We examined the behavior of a Pick policy early in training and found that it does tend to move awayfrom the object it needs to pick up and sometimes moves back.
Revised: We examined the behavior of a pick policy early in training and found that it does tend to move awayfrom the object it needs to pick up and sometimes moves back.

Original: Although the magnitude of navigationis small and  quite infrequent, so the degree of generalization is high.
Revised: Although the magnitude of navigationis small and navigation quite infrequent, so the degree of generalization is high.

Original: This result, and  higher performance on HAB, highlights that it may not always be beneﬁcial to remove‘unneeded’ actions.
Revised: This result, and the higher performance on HAB, highlights that it may not always be beneficial toremove ‘unneeded’ actions.

Original:  Szot et al.
Revised: Presumably Szot et al.

Original: [2021] removed navigation where possible to improve sample efﬁciencyand training throughput 3 . Our experiments corroborate this; training without navigation improvesboth sample efﬁciency and training speed.
Revised: [2021] removed navigation where possible toimprove sample efficiency and/or training throughput – in our own experiments, training withoutnavigation both improves sample efficiency and throughput.

Original: However, by enabling navigation and allowing the agentto learn how to (not) use it, we arrived upon emergent navigation and improved HAB performance.
Revised: However, by enabling navigation andallowing the agent to learn that only minimal navigation is needed itself, we arrived upon emergentnavigation and improved both full task performance.

Original:  Related Work
Revised:  Related Work

Original: AsyncOnRL methods provide high-throughput on-policy reinforcement learning [Espeholt et al., 2018,
Revised: AsyncOnRL systems provide high-throughput on-policy reinforcement learning [Espeholt et al., 2018,

Original: Petrenko et al., 2020].
Revised: Petrenko et al., 2020].

Original: However, they have reduced sample efﬁciency as they must correct for nearpolicy, or ‘stale’, data.
Revised: However, they have reduced sample efficiency as they must correct for nearpolicy, or ‘stale’, data.

Original: Few support multi-GPU learning and, when they do, the user must manuallybalance compute between learning and experience collection [Espeholt et al., 2020].
Revised: Few support multi-GPU learning and, when they do, the user must manuallybalance compute between learning and experience collection [Espeholt et al., 2020].

Original: VER achieves thesame throughput on 1-GPU while learning with on-policy data, has better sample efﬁciency, supportsmulti-GPU learning, and automatically balances compute between learning and simulation.
Revised: VER achieves thesame throughput on 1-GPU while learning with on-policy data, has better sample efficiency, supportsmulti-GPU learning, and automatically balances compute between learning and simulation.

Original: SyncOnRL .
Revised: SyncOnRL Systems.

Original: Closely related to our work, HTS-RL [Liu et al., 2020] also use the same experiencecollection techniques as AsyncOnRL to mitigate the action-level straggler effect.Unfortunately inefﬁciencies in the provided implementation prevents a meaningful direct comparison.
Revised: Closely related to our work, HTS-RL [Liu et al., 2020] also use the sameexperience collection techniques as AsyncOnRL to mitigate the action-level straggler effect.

Original: In Appendix E weshow that our re-implementation is 110% faster (2.1x speedup) and thus instead compare to strongerbaseline of No VER in the main text.

Original: We propose a novel mechanism, variable experience rollouts, tomitigate the episode-level straggler effect and thereby close the gap to AsyncOnRL .
Revised: Wepropose a novel mechanism, variable experience rollouts, to mitigate the episode-level straggler effect.

Original: We use and buildupon Decentralized Distributed PPO (DD-PPO)
Revised: We use and build upon Decentralized Distributed PPO (DD-PPO)

Original: [Wijmans et al., 2020], which proposed a distributedmulti-GPU method based on data parallelism [Hillis and Steele Jr, 1986].
Revised: [Wijmans et al., 2020], whichproposed a distributed multi-GPU method based upon data parallelism [Hillis and Steele Jr, 1986].

Original: Batched simulators simulate multiple agents (in multiple environments)simultaneously and areresponsible for their own parallelization [Shacklett et al., 2021, Petrenko et al., 2021, Freeman et al. ,2021, Makoviychuk et al., 2021].
Revised: Batched simulators simulate multiple agents (in multiple environments) and are responsible for theirown parallelization [Shacklett et al., 2021, Petrenko et al., 2021, Freeman et al., 2021, Makoviychuket al., 2021].

Original: While these systems offer impressive performance, none currentlysupport a benchmark like HAB (which combines physics and photo-realism) nor  the ﬂexibility of 
Revised: While these systems offer impressive performance, none currently support a benchmarklike HAB (which combines physics and photo-realism) nor do any offer the flexibility of Habitat,

Original: Habitat, AI2Thor [Kolve et al., 2017], or ThreeDWorld [Gan et al., 2020].
Revised: AI2Thor [Kolve et al., 2017], or ThreeDWorld [Gan et al., 2020].

Original: VER enables researchersto ﬁrst explore promising directions using existing simulators and then build batched simulatorswith the knowledged gained from their ﬁndings.
Revised: VER enables researchers to firstexplore promising directions using existing simulators and then build batched simulators.

Original:  Societal Impact, Limitations, and Conclusion
Revised:  Societal Impact, Limitations, and Conclusion

Original: Our main application result is trained using the ReplicaCAD dataset [Szot et al., 2021], which islimited to only US apartments, and this may have negative societal impacts for deployed assistants.
Revised: Our main application result is trained using the ReplicaCAD dataset [Szot et al., 2021], which islimited to only US apartments, and this may have negative societal impacts for deployed assistants.

Original: VER was designed and evaluated for tasks with both GPU simulation and large neural networks.
Revised: VER was designed and evaluated for tasks with both GPU simulation and large neural networks.

Original: Fortasks with CPU simulation and smaller networks, we expect it to improve upon SyncOnRL but itmay have less throughput than AsyncOnRL and overlapping experience collection and learning wouldlikely be beneﬁcial. Our implementation supports overlapping learning and collecting 1 rollout, butmore overlap may be beneﬁcial.
Revised: Fortaks with CPU simulation and smaller networks, we expect it to improve upon SyncOnRL but it mayhave less throughput than AsyncOnRL and overlapping experience collection and learning wouldlikely be beneficial 4 .

Original: The TP-SRL agent we build upon requires oracle knowledge, e.g .that the cabinet must be opened before picking.
Revised: The TP-SRL agent we build upon requires oracle knowledge, i.e . that the cabinetmust be opened before picking.

Original: We have presented Variable Experience Rollout ( VER ).
Revised: We have presented Variable Experience Rollout ( VER ).

Original: VER combines the strengths of and blurs theline between SyncOnRL and AsyncOnRL .
Revised: VER combines the strengths of and blurs theline between SyncOnRL and AsyncOnRL .

Original: Its trains agents for embodied navigation tasks in Habitat1.0 60%-100% faster (1.6x to 2x speedup) than DD-PPO with similar  sample efﬁciency – saving 19. GPU-days on PointNav and 28 GPU-day for ObjectNav per seed in our experiments.
Revised: Its trains agents for embodied navigation tasks in Habitat1.0 60%-100% faster (1.6x to 2x speedup) than DD-PPO with similar or better sample efficiency –saving 19.2 GPU-days on PointNav and 28 GPU-day for ObjectNav.

Original: On the recently introduced (and more challenging) embodied rearrangement tasks in Habitat 2.0, VER trains agents 
Revised: On the recently introduced (andmore challenging) embodied rearrangement tasks in Habitat 2.0, VER trains agents 150% faster than

Original: 150% faster than DD-PPO and is fast as SampleFactory ( AsyncOnRL ) on 1 GPU.
Revised: DD-PPO and is fast as SampleFactory ( AsyncOnRL ) on 1 GPU.

Original: On 8 GPUs, VER is 
Revised: On 8 GPUs, VER is 180% faster than

Original: 180% faster than DD-PPO and 70% faster than SampleFactory with better sample efﬁciency – saving 32 GPU-days vs. DD-PPO and 11.2 GPU-days vs. SampleFactory per skill in our experiments.
Revised: DD-PPO and 70% faster than SampleFactory with better sample efficiency – saving 32 GPU-daysper skill vs. DD-PPO and 11.2 GPU-days vs. SampleFactory.

Original: We use VER to study rearrangement.
Revised: We use VER to study rearrangement.

Original: We ﬁnd the emergence of navigation in policies that ostensibly require no navigation when given access to navigation actions.
Revised: We find the emergence of navigation in policies that ostensibly require no navigation when givenaccess to navigation actions.

Original: This results in strong progress on Tidy House (+30% success) and shows that it may not always be advantageous to remove ‘unneeded actions’
Revised: This results in strong progress on Tidy House (+30% success). Thisresults highlights that it may not always be advantageous to limit a policy’s action space.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section
Revised: [Yes] See Section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the
Revised: [Yes] See the

Original: supplemental material
Revised: supplemental material

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] See supplementary materials
Revised: [Yes] See supplementary materials

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We use 95% bootstrapped conﬁdence intervals.
Revised: [Yes] We use 95% bootstrapped confidence intervals.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] .
Revised: [Yes] .

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See supplementary materials
Revised: [Yes] See supplementary materials

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [Yes] See supplementary materials
Revised: [Yes] See supplementary materials

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [Yes] See supplementary materials
Revised: [Yes] See supplementary materials

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Systems ConsiderationsA.1 Memory beneﬁts of not overlapping experience collection and learning

Original: VER does not overlap experience collection and learning.

Original: This allows the main process to act as aninference worker during experience collection and a learner during learning, saving 1 CUDA context( ∼ 1GB of GPU memory).

Original: Unlike AsyncOnRL , VER is able to use GPU shared memory to transfer experience from inferenceworkers to learner.

Original: This is because VER uses (at least) half the GPU memory for storing rollouts.

Original: AsyncOnRL must maintain at least 1 set of rollout buffers for storing the rollout currently being usedfor learning an another set to store the next rollout being collected.

Original: In practice, they maintain morethan this minimum 2x to enable faster training.

Original: A.2 Graphics Processing Units (GPUs)

Original: The application of graphics processing units (GPUs) to training neural networks has led to manyadvances in artiﬁcial intelligence.

Original: In reinforcement learning, GPUs are used for policy inference,learning the policy, and rendering visual observations like Depth or RGB .

Original: Given the importance of

Original: GPUs in reinforcement learning, there are several important attributes of them to understand.1) GPUs are not optimized for multiple processes using them concurrently.

Original: They are unable toefﬁciently execute multiple processes at once (if at all).

Original: While sometimes concurrent use is the onlyoption, a system should whenever possible have a single process that has exclusive use of the GPUand uses it fully.

Original: GPU drivers have different compute modes for graphics ( i.e . rendering) andcompute ( i.e . neural network inference).

Original: Current GPU drivers have to perform an expensive contextswitch whenever alternating between these modes.

Original: Thus a system should minimize the number oftimes the GPU must switch between compute and graphics.

Original: The compute mode CUDA context forprocess and each CUDA context requires a large amount of memory.

Original: Thus a system should use theminimal number of CUDA contexts possible.

Original: B Skill Deployment

Original: We use the same skills and deployement procedure as Szot et al.

Original: [2021] with two changes to thenavigation skill: 1) We have a dedicated stop action for navigation instead of using a t = 0 as stop.

Original: We ﬁnd this to be more robust.

Original: We add a force penalty to training, this makes the navigation skillless likely to bump into things by accident.

Original: However, this also makes it stop immediately if it everstarts in contact with the environment.

Original: During evaluation, we mask its prediction of stop if the targetobject is more than 2 meters away.

Original: This information is part of its sensor suite, so this does not requireany privileged information.

Original: On Set Table, we add an additional stage to the planner that calls navigation again after open cabinetas the open cabinet skill with base movement tends to move away from the cabinet.

Original: C Videos

Original: In the supplement, we include the following videos.

Original: Video 1 (1-tp-srl-no-nav.mp4)

Original: This is an example of TP-SRL(NoNav)+All Base Movement on Tidy

Original: House.

Original: The objects to pick are have a white wireframe box drawn around them and at their placelocations there is another copy of that object.

Original: The grey sphere on the ground denotes where thenavigation skill was trained to navigate to.

Original: This information is not shown to the policy.

Original: The policy makes errors and picks up the wrong object

Original: Video 2 (

Original: 2-tp-srl.mp

Original: ) This is an example of

Original: TP

Original: -SRL+All Base Movement on Tidy House.

Original: Video

Original: 3 (3-pick-*.mp4)

Original: Examples of the pick policy

Original: on

Original: its

Original: training task

Original: Note the difference inspawn distance between this and Video 1.

Original: Video 3 (3-place-*.mp4) Examples of the place policy on its training task.

Original: Optimizer

Original: Adam [Kingma and Ba, 2015]

Original: Initial

Original: [Table]

Original: Rate 2 .

Original: 5 ×

Original: [Table]

Original: [Table]

Original: D Hyperparamaters

Original: The hyperparameters for our rearrangement skills are in Table A1.

Original: For the benchmark comparisons, we set the rollout length T to 128 for methods that require this, thatway all methods collect the same amount of experience per rollout.

Original: For SampleFactory, we use abatch size of 1024 on 1 GPU, 2048 on 2, and 4096 on 4 and 8 (this the GPU memory limit).

Original: We dono use a learned entropy coefﬁcient for benchmarking as not all frameworks support this.

Original: We use aﬁxed value of 10 − 4 as this works well for Open Fridge.

Original: E HTS-RL Comparison

Original: HTS-RL [Liu et al., 2020] uses the sampling method of AsyncOnRL to collect experience for SyncOnRL(the same as No VER and VER ).

Original: It further uses delayed gradients to enable overlapped experience andlearning.

Original: Unfortunately the provided implementation 4 has inefﬁciencies that limit its throughput.

Original: To demonstrate this, we add the same style of overlapped experience collection and learning to No VERand compare the provided HTS-RL implementation,

Original: HTS-RL (Provided), with our re-implementation,

Original: HTS-RL (Ours).

Original: Our implementation 110% faster than the provided implementation (Table A2).

Original: Keydifferences in our implementation are fast userspace mutexes (futexes) in shared memory for synchronization (vs. spin locks), pre-allocated pinned memory for CPU to GPU transfers (vs. allocatingfor each transfer), and GPU shared memory to send experience from inference workers to learn andweights from learner to inference workers (vs. CPU shared memory).

Original: [Figure]

Original: S u cc e ss ( H i g

Original: h e r

Original: i s

Original: B e tt e r )

Original: DD-PPO

Original: [Figure]

Original: VER (Ours)

Original: [Figure]

Original: [Figure]

Original: SampleFactory

Original: [-GPUs 
-]
Revised: [-GPUs 
-]

Original: [Figure]

Original: [Figure]

Original: u cc e ss

Original: ( H i g h e

Original: r

Original: i s

Original: B e tt e

Original: [Figure]

Original: [Figure]

Original: Even with the reduction of sample efﬁciency when increasing the number of GPUs, increasing the the number of GPUs always reduces the time to reach convergence.

Original: [Figure]

Original: 6x

Original: [Figure]

Original: We also ﬁnd that given the efﬁciency of our implementation, there is no signiﬁcant change insystem SPS if we remove overlapped experience collection and learning (Table A2, HTS-RL (Ours)vs.

Original: No VER ) for Habitat-style tasks.

Original: Removing this reduces GPU memory usage ( ∼ 2 GB in ourexperiments, this will be larger for larger networks).

Original: We note that overlapped experience collectionand learning does have uses, i.e . for CPU simulation or policies with signiﬁcant CPU components,but it isn’t necessary when both the policy and simulator make heavy use of the GPU.

Original: F VER Inﬂight actions

Original: One subtle design choice is the following – when VER ﬁnishes a T × N experience collection, therewill be (slow) environments that haven’t completed simulation yet.

Original: Instead of discarding that data,we choose to collect this experience in the next rollout.

Original: We ﬁnd this choice leads to speed gainswithout any sample-efﬁciency loss.

Original: [Figure]

Original: G DD-PPO Continued

Original: This is a synchronous operation and thus introduces a GPU-worker-level straggler effect.

Original: Wijmanset al.

Original: [2020] mitigate this effect by preempting stagglers – stopping collecting experience early andproceeding to learning – after a ﬁxed number of GPU-workers have ﬁnished experience collection.

Original: We instead approximate the optimal preemption for each experience collection phase.

Original: Given thelearning time, LT, and a function Time ( S ) that returns the time needed to collect S steps of experience,the optimal number of steps S to collect before preempting stragglers is

Original: [Equation]

Original: For LT, we record the time from the last iteration as this doesn’t change between iterations.

Original: Weapproximate Time ( S ) using the average step time to receive a step of experience from each environment (this includes both inference time and simulation time) from the previous rollout.

Original: This will be apoor approximation in some circumstances

Original: but we ﬁnd it to work well in practice.

Original: We improve upon sample efﬁciency of DD-PPO by ﬁlling the preempted rollouts with experiencefrom the previous rollout.

Original: We perform extra epochs of PPO on this now ‘stale’ data instead ofcorrecting for the off-policy return as we ﬁnd this simpler and effective.

Original: This comes with no effectivecomputation cost since the result is that all GPUs have the same batch size instead of some havinga smaller batch size.

Original: H HAB Tasks

Original: In Tidy House, the agent is tasked with moving 5 objects from their initial locations to their ﬁnallocations.

Original: The objects are rarely in containers ( i.e . fridge or cabinet drawer) and when they are, thecontainers are already opened.

Original: In Prepare Groceries, the agent must move 3 objects from the kitchencounter into the open fridge (or open fridge to counter).

Original: This stresses picking and placing in thefridge, which is challenging.

Original: In Set Table, the agent must move 1 object from the closed kitchencabinet drawer to the table and 1 object from the closed fridge to the table.

Original: This requires opening thecabinet and fridge, and picking from these challenging receptacles.

Original: I Datasets

Original: ReplicaCAD [Szot et al., 2021] – Creative Commons Attribution 4.0 International (CC BY 4.0)license.

Original: This dataset was artist constructed.

Original: YCB dataset [Calli et al., 2015] – Creative Commons Attribution 4.0 International (CC BY 4.0).

Original: Thisdataset contains 3D scans of generic objects.

Original: There is no PII.

Original: Matterport 3D [Chang et al., 2017] – http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf .

Original: Consent was given by the owners of the space.

Original: Habitat Matterport Research Dataset [Ramakrishnan et al., 2021] – https://matterport.com/matterport-end-user-license-agreement-academic-use-model-data Consent was given bythe owners of the space and any PII was blurred.