Original: VER : Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement
Revised: VER : Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement

Original: Abstract
Revised: Abstract

Original: We present Variable Experience Rollout ( VER ), a technique for efﬁciently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many
Revised: We present Variable Experience Rollout ( VER ), a technique for efﬁciently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many

Original: GPUs residing on, potentially, many machines.
Revised: GPUs residing on, potentially, many machines.

Original: VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods ( SyncOnRL and AsyncOnRL , respectively).
Revised: VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods ( SyncOnRL and AsyncOnRL , respectively).

Original: Speciﬁcally, it learns from on-policy experience (like SyncOnRL ) and has no synchronization points (like AsyncOnRL ) enabling high throughput.
Revised: Speciﬁcally, it learns from on-policy experience (like SyncOnRL ) and has no synchronization points (like AsyncOnRL ) enabling high throughput.

Original: We ﬁnd that VER leads to signiﬁcant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments.
Revised: We ﬁnd that VER leads to signiﬁcant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments.

Original: Speciﬁcally, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art for distributed SyncOnRL , with similar sample efﬁciency.
Revised: Speciﬁcally, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art for distributed SyncOnRL , with similar sample efﬁciency.

Original: For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup) on 8 GPUs than DD-PPO.
Revised: For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup) on 8 GPUs than DD-PPO.

Original: Compared to SampleFactory (the current state-of-the-art
Revised: Compared to SampleFactory (the current state-of-the-art

Original: AsyncOnRL ), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efﬁciency.
Revised: AsyncOnRL ), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efﬁciency.

Original: We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB).
Revised: We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB).

Original: We ﬁnd a surprising emergence of navigation in skills that do not ostensible require any navigation.
Revised: We ﬁnd a surprising emergence of navigation in skills that do not ostensible require any navigation.

Original: Speciﬁcally, the Pick skill involves a robot picking an object from a table.
Revised: Speciﬁcally, the Pick skill involves a robot picking an object from a table.

Original: During training the robot was always spawned close to the table and never needed to navigate.
Revised: During training the robot was always spawned close to the table and never needed to navigate.

Original: However, we ﬁnd that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high out-of-distribution generalization.
Revised: However, we ﬁnd that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high out-of-distribution generalization.

Original:  Introduction
Revised:  Introduction

Original: Scaling matters.
Revised: Scaling matters.

Original: Progress towards building embodied intelligent agents that are capable of performing goal driven tasks has been driven, in part, by training large neural networks in photo-realistic 3D environments with deep reinforcement learning (RL) for (up to) billions of steps of experience
Revised: Progress towards building embodied intelligent agents that are capable of performing goal driven tasks has been driven, in part, by training large neural networks in photo-realistic 3D environments with deep reinforcement learning (RL) for (up to) billions of steps of experience

Original: [Wijmans et al., 2020, Maksymets et al., 2021, Mezghani et al., 2021, Ramakrishnan et al., 2021, Miki et al., 2022].
Revised: [Wijmans et al., 2020, Maksymets et al., 2021, Mezghani et al., 2021, Ramakrishnan et al., 2021, Miki et al., 2022].

Original: To enable this scale, RL systems must be able to efﬁciently utilize the available resources ( e.g .
Revised: To enable this scale, RL systems must be able to efﬁciently utilize the available resources ( e.g .

Original: GPUs), and scale to multiple machines all while maintaining sample-efﬁcient learning.
Revised: GPUs), and scale to multiple machines all while maintaining sample-efﬁcient learning.

Original: One promising class of techniques to achieve this scale is batched on-policy RL.
Revised: One promising class of techniques to achieve this scale is batched on-policy RL.

Original: These methods collect experience from many ( N ) environments simultaneously using the policy and update it with this cumulative experience.
Revised: These methods collect experience from many ( N ) environments simultaneously using the policy and update it with this cumulative experience.

Original: They are broadly divided into two classes: synchronous ( SyncOnRL )
Revised: They are broadly divided into two classes: synchronous ( SyncOnRL )

Original: [Figure]
Revised: [Figure]

Original: AsyncOnRL
Revised: AsyncOnRL

Original: Learning Learning
Revised: Learning Learning

Original: Policy Inference
Revised: Policy Inference

Original: Simulation
Revised: Simulation

Original: D
Revised: D

Original: E +30% and synchronization points: is executed for the entire
Revised: E +30% and synchronization points: is executed for the entire

Original: ( o t → a t )
Revised: ( o t → a t )

Original: Bb =1 1 (Fig.
Revised: Bb =1 1 (Fig.

Original: 1 A), then actions are executed in all environments, ( s a
Revised: 1 A), then actions are executed in all environments, ( s a

Original: → s
Revised: → s

Original: + +
Revised: + +

Original: Bb =1 (Fig.
Revised: Bb =1 (Fig.

Original: 1 B), until T been collected from all N environments.
Revised: 1 B), until T been collected from all N environments.

Original: This ( T, N ) -shaped batch of experience is used to update the policy (Fig. 1 C).
Revised: This ( T, N ) -shaped batch of experience is used to update the policy (Fig. 1 C).

Original: Synchronization reduces throughput as the system spends signiﬁcant (sometimes the most) time waiting for the slowest environment to ﬁnish.
Revised: Synchronization reduces throughput as the system spends signiﬁcant (sometimes the most) time waiting for the slowest environment to ﬁnish.

Original: This is the straggler effect [Petrini et al., 2003, Dean and Ghemawat, 2004].
Revised: This is the straggler effect [Petrini et al., 2003, Dean and Ghemawat, 2004].

Original: AsyncOnRL removes these synchronization points, thereby mitigating the straggler effect and improving throughput.
Revised: AsyncOnRL removes these synchronization points, thereby mitigating the straggler effect and improving throughput.

Original: Actions are taken as soon as they are computed, a t → o t +1 (Fig. 1 D), the next action is computed as soon as the observation is ready, o t → a t (Fig. 1 E), and the policy is updated as soon as enough experience is collected.
Revised: Actions are taken as soon as they are computed, a t → o t +1 (Fig. 1 D), the next action is computed as soon as the observation is ready, o t → a t (Fig. 1 E), and the policy is updated as soon as enough experience is collected.

Original: However, AsyncOnRL systems are not able to ensure that all experience has been collected by only the current policy and thus must consume near -policy data.
Revised: However, AsyncOnRL systems are not able to ensure that all experience has been collected by only the current policy and thus must consume near -policy data.

Original: This reduces sample efﬁciency [Liu et al., 2020].
Revised: This reduces sample efﬁciency [Liu et al., 2020].

Original: Thus, status quo leaves us with an unpleasant tradeoff – high sample-efﬁciency with low throughput or high throughput with low sample-efﬁciency.
Revised: Thus, status quo leaves us with an unpleasant tradeoff – high sample-efﬁciency with low throughput or high throughput with low sample-efﬁciency.

Original: In this work, we propose Variable Experience Rollout ( VER ).
Revised: In this work, we propose Variable Experience Rollout ( VER ).

Original: VER combines the strengths of and blurs the line between SyncOnRL and AsyncOnRL .
Revised: VER combines the strengths of and blurs the line between SyncOnRL and AsyncOnRL .

Original: Like SyncOnRL , VER collects experience with the current policy and then updates it.
Revised: Like SyncOnRL , VER collects experience with the current policy and then updates it.

Original: Like AsyncOnRL , VER does not have synchronization points – it computes next actions, steps environments, and updates the policy as soon as possible.
Revised: Like AsyncOnRL , VER does not have synchronization points – it computes next actions, steps environments, and updates the policy as soon as possible.

Original: The inspiration for
Revised: The inspiration for

Original: VER comes from two key observations:
Revised: VER comes from two key observations:

Original: 1) AsyncOnRL mitigates the straggler effect by implicitly collecting a variable amount of experience from each environment – more from fast-to-simulate environments and less from slow ones.
Revised: 1) AsyncOnRL mitigates the straggler effect by implicitly collecting a variable amount of experience from each environment – more from fast-to-simulate environments and less from slow ones.

Original: Both SyncOnRL and AsyncOnRL use a ﬁxed rollout length, T steps of experience.
Revised: Both SyncOnRL and AsyncOnRL use a ﬁxed rollout length, T steps of experience.

Original: Our key insight is that while a ﬁxed rollout length may simplify an implementation, it is not a requirement for RL.
Revised: Our key insight is that while a ﬁxed rollout length may simplify an implementation, it is not a requirement for RL.

Original: These two key observations naturally lead us to variable experience rollout ( VER ),
Revised: These two key observations naturally lead us to variable experience rollout ( VER ),

Original: i.e . collecting rollouts with a variable number of steps.
Revised: i.e . collecting rollouts with a variable number of steps.

Original: VER adjusts the rollout length for each environment based on its simulation speed.
Revised: VER adjusts the rollout length for each environment based on its simulation speed.

Original: It explicitly collects more experience from fast-to-simulate environments and less from slow ones (Fig. 1).
Revised: It explicitly collects more experience from fast-to-simulate environments and less from slow ones (Fig. 1).

Original: The result is an RL system that overcomes the straggler effect and maintains sample-efﬁciency by learning from on-policy data.
Revised: The result is an RL system that overcomes the straggler effect and maintains sample-efﬁciency by learning from on-policy data.

Original: VER focuses on efﬁciently utilizing a single GPU.
Revised: VER focuses on efﬁciently utilizing a single GPU.

Original: To enable efﬁcient scaling to multiple GPUs, we combine VER with the decentralized distributed method proposed in [Wijmans et al., 2020].
Revised: To enable efﬁcient scaling to multiple GPUs, we combine VER with the decentralized distributed method proposed in [Wijmans et al., 2020].

Original: First, we evaluate VER on well-established embodied navigation tasks using Habitat 1.0 [Savva et al., 2019] on 8 GPUs.
Revised: First, we evaluate VER on well-established embodied navigation tasks using Habitat 1.0 [Savva et al., 2019] on 8 GPUs.

Original: VER trains PointGoal navigation [Anderson et al., 2018] 60% faster than Decentralized Distributed PPO (DD-PPO)
Revised: VER trains PointGoal navigation [Anderson et al., 2018] 60% faster than Decentralized Distributed PPO (DD-PPO)

Original: [Wijmans et al., 2020], the current state-of-the-art for distributed on-policy RL, with the same sample efﬁciency.
Revised: [Wijmans et al., 2020], the current state-of-the-art for distributed on-policy RL, with the same sample efﬁciency.

Original: For ObjectGoal navigation [Batra et al., 2020b], an active area of research, VER is 100% faster than DD-PPO with (slightly) better sample efﬁciency.
Revised: For ObjectGoal navigation [Batra et al., 2020b], an active area of research, VER is 100% faster than DD-PPO with (slightly) better sample efﬁciency.

Original: Next, we evaluate VER on the recently introduced (and signiﬁcantly more challenging) GeometricGoal rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].
Revised: Next, we evaluate VER on the recently introduced (and signiﬁcantly more challenging) GeometricGoal rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].

Original: In GeoRearrange, a virtual robot is spawned in a new environment and asked to rearrange a set of objects from their initial to
Revised: In GeoRearrange, a virtual robot is spawned in a new environment and asked to rearrange a set of objects from their initial to

Original: [Figure]
Revised: [Figure]

Original: desired coordinates.
Revised: desired coordinates.

Original: These environments have highly variable simulation time (physics simulation time increases if the robot bumps into something) and require GPU-acceleration (for photo-realistic rendering), limiting the number of environments that can be run in parallel.
Revised: These environments have highly variable simulation time (physics simulation time increases if the robot bumps into something) and require GPU-acceleration (for photo-realistic rendering), limiting the number of environments that can be run in parallel.

Original: On 1 GPU, VER is 150% faster (2.5x speedup) than DD-PPO with the same sample efﬁciency.
Revised: On 1 GPU, VER is 150% faster (2.5x speedup) than DD-PPO with the same sample efﬁciency.

Original: VER is as fast as SampleFactory [Petrenko et al., 2020], the state-of-the-art AsyncOnRL , with the same sample efﬁciency.
Revised: VER is as fast as SampleFactory [Petrenko et al., 2020], the state-of-the-art AsyncOnRL , with the same sample efﬁciency.

Original: VER is as fast as AsyncOnRL in pure throughput; this is a surprisingly strong result.
Revised: VER is as fast as AsyncOnRL in pure throughput; this is a surprisingly strong result.

Original: AsyncOnRL never stops collecting experience and should, in theory, be a strict upper bound on performance.
Revised: AsyncOnRL never stops collecting experience and should, in theory, be a strict upper bound on performance.

Original: VER is able to match AsyncOnRL for environments that heavily utilize the GPU for rendering, like Habitat.
Revised: VER is able to match AsyncOnRL for environments that heavily utilize the GPU for rendering, like Habitat.

Original: In AsyncOnRL , learning, inference, and rendering contend the GPU which reduces throughput.
Revised: In AsyncOnRL , learning, inference, and rendering contend the GPU which reduces throughput.

Original: In VER , inference and rendering contend for the GPU while learning does not.
Revised: In VER , inference and rendering contend for the GPU while learning does not.

Original: On 8 GPUs, VER achieves better scaling than DD-PPO, achieving a 6.7x speed-up (vs. 6x for DD-PPO) due to lower variance in experience collection time between GPU-workers.
Revised: On 8 GPUs, VER achieves better scaling than DD-PPO, achieving a 6.7x speed-up (vs. 6x for DD-PPO) due to lower variance in experience collection time between GPU-workers.

Original: Due to this efﬁcient multi-GPU scaling, VER is 70% faster (1.7x speedup) than SampleFactory on 8 GPUs and has better sample efﬁciency as it learns from on-policy data.
Revised: Due to this efﬁcient multi-GPU scaling, VER is 70% faster (1.7x speedup) than SampleFactory on 8 GPUs and has better sample efﬁciency as it learns from on-policy data.

Original: Finally, we leverage these SysML contributions to study open research questions posed in prior work.
Revised: Finally, we leverage these SysML contributions to study open research questions posed in prior work.

Original: Speciﬁcally, we train RL policies for mobile manipulation skills ( Navigate , Pick , Place , etc .) and chain them via a task planner.
Revised: Speciﬁcally, we train RL policies for mobile manipulation skills ( Navigate , Pick , Place , etc .) and chain them via a task planner.

Original: Szot et al.
Revised: Szot et al.

Original: [2021] called this approach TP-SRL and identiﬁed a critical ‘handoff problem’ – downstream skills are set up for failure by small errors made by upstream skills ( e.g . the Pick skill failing because the navigation skill stopped the robot a bit too far from the object ).
Revised: [2021] called this approach TP-SRL and identiﬁed a critical ‘handoff problem’ – downstream skills are set up for failure by small errors made by upstream skills ( e.g . the Pick skill failing because the navigation skill stopped the robot a bit too far from the object ).

Original: We demonstrate a number of surprising ﬁndings when TP-SRL is scaled via VER .
Revised: We demonstrate a number of surprising ﬁndings when TP-SRL is scaled via VER .

Original: Most importantly, we ﬁnd the emergence of navigation when skills that do not ostensibly require navigation ( e.g . pick) are trained with navigation actions enabled.
Revised: Most importantly, we ﬁnd the emergence of navigation when skills that do not ostensibly require navigation ( e.g . pick) are trained with navigation actions enabled.

Original: In principle, Pick and Place policies do not need to navigate during training since the objects are always in arm’s reach, but in practice they learn to navigate to recover from their mistakes and this results in strong out-of-distribution test-time generalization.
Revised: In principle, Pick and Place policies do not need to navigate during training since the objects are always in arm’s reach, but in practice they learn to navigate to recover from their mistakes and this results in strong out-of-distribution test-time generalization.

Original: Speciﬁcally, TP-SRL without a navigation skill achieves 50% success on NavPick and 20% success on a NavPickNavPlace task simply because the Pick and Place skills have learned to navigate (sometimes across the room!).
Revised: Speciﬁcally, TP-SRL without a navigation skill achieves 50% success on NavPick and 20% success on a NavPickNavPlace task simply because the Pick and Place skills have learned to navigate (sometimes across the room!).

Original: TP-SRL with a Navigate skill performs even stronger: 90% on NavPickNavPlace and 32% on 5 successive NavPickNavPlaces (called Tidy House in Szot et al.
Revised: TP-SRL with a Navigate skill performs even stronger: 90% on NavPickNavPlace and 32% on 5 successive NavPickNavPlaces (called Tidy House in Szot et al.

Original: [2021]), which are +32% and +30% absolute improvements over Szot et al.
Revised: [2021]), which are +32% and +30% absolute improvements over Szot et al.

Original: [2021], respectively.
Revised: [2021], respectively.

Original: Prepare Groceries and Set Table, which both require interaction with articulated receptacles (fridge, drawer), remain as open problems (5% and 0% Success, respectively) and are the next frontiers.
Revised: Prepare Groceries and Set Table, which both require interaction with articulated receptacles (fridge, drawer), remain as open problems (5% and 0% Success, respectively) and are the next frontiers.

Original:  VER : Variable Experience Rollout
Revised:  VER : Variable Experience Rollout

Original: The key challenge that any batched on-policy RL technique needs to address is variability of simulation time for the environments in a batch.
Revised: The key challenge that any batched on-policy RL technique needs to address is variability of simulation time for the environments in a batch.

Original: There are two primary sources of this variability: action-level and episode-level.
Revised: There are two primary sources of this variability: action-level and episode-level.

Original: The amount of time needed to simulate an action within an environment varies depending on the the speciﬁc action, the state of the robot, and the environment ( e.g . simulating the robot navigating on a clear ﬂoor is much faster than simulating the robot’s arm colliding with objects).
Revised: The amount of time needed to simulate an action within an environment varies depending on the the speciﬁc action, the state of the robot, and the environment ( e.g . simulating the robot navigating on a clear ﬂoor is much faster than simulating the robot’s arm colliding with objects).

Original: The amount of time needed to simulate an entire episode also varies environment to environment irrespective of action-level variability ( e.g . rendering images takes longer for visually-complex scenes, simulating physics takes longer for scenes with a large number of objects).
Revised: The amount of time needed to simulate an entire episode also varies environment to environment irrespective of action-level variability ( e.g . rendering images takes longer for visually-complex scenes, simulating physics takes longer for scenes with a large number of objects).

Original: N=
Revised: N=

Original: N= N=
Revised: N= N=

Original: N=   (A) 3 0 13 0 1 05 7 2 

Revised: N=   (A) 3 0 13 0 1 05 7 2

Original: N= N=
Revised: N= N=

Original: N= N=
Revised: N= N=

Original: N=0 (Navigating)
Revised: N=0 (Navigating)

Original: N=1 (Picking) (B)
Revised: N=1 (Picking) (B)

Original: Packed Sequence Format 3 0 13 0 10 1 0 0 0 0 1 1 1 2 2 2 

Revised: Packed Sequence Format 3 0 13 0 10 1 0 0 0 0 1 1 1 2 2 2

Original: T=0 T=1 T=

Revised: T=0 T=1 T=

Original: T=3 N=
Revised: T=3 N=

Original: N=
Revised: N=

Original: N= N=
Revised: N= N=

Original: T= T=
Revised: T= T=

Original: T=
Revised: T=

Original: T= (C)
Revised: T= (C)

Original: [Figure]
Revised: [Figure]

Original: 2.1 Action-Level Straggler Mitigation
Revised: 2.1 Action-Level Straggler Mitigation

Original: We mitigate the action-level straggler effect by applying the experience collection method of
Revised: We mitigate the action-level straggler effect by applying the experience collection method of

Original: AsyncOnRL to SyncOnRL .
Revised: AsyncOnRL to SyncOnRL .

Original: We represent this visually in Fig.
Revised: We represent this visually in Fig.

Original: 2 and describe it in text bellow.
Revised: 2 and describe it in text bellow.

Original: Environment workers receive the next action and step the environment, EnvStep, e.g . s t , a t → s t +1 , o t +1 , r t .
Revised: Environment workers receive the next action and step the environment, EnvStep, e.g . s t , a t → s t +1 , o t +1 , r t .

Original: They write the outputs of the environment (observations, reward, etc .) into preallocated CPU shared memory for consumption by inference workers.
Revised: They write the outputs of the environment (observations, reward, etc .) into preallocated CPU shared memory for consumption by inference workers.

Original: Inference workers receive batches of steps of experience from environment workers.
Revised: Inference workers receive batches of steps of experience from environment workers.

Original: They perform inference with the current policy to select the next action and send it to the environment worker using pre-allocated CPU shared memory.
Revised: They perform inference with the current policy to select the next action and send it to the environment worker using pre-allocated CPU shared memory.

Original: After inference they store experience for learning in shared GPU memory.
Revised: After inference they store experience for learning in shared GPU memory.

Original: Inference workers use dynamic batching and perform inference on all outstanding inference requests instead of waiting for a ﬁxed number of requests to arrive 2 .
Revised: Inference workers use dynamic batching and perform inference on all outstanding inference requests instead of waiting for a ﬁxed number of requests to arrive 2 .

Original: This allows us to leverage the beneﬁts of batching without introducing synchronization points between environment workers.
Revised: This allows us to leverage the beneﬁts of batching without introducing synchronization points between environment workers.

Original: This experience collection technique is similar to that of HTS-RL
Revised: This experience collection technique is similar to that of HTS-RL

Original: [Liu et al., 2020] ( SyncOnRL ) and SampleFactory [
Revised: [Liu et al., 2020] ( SyncOnRL ) and SampleFactory [

Original: Petrenko et al., 2020] ( AsyncOnRL ).
Revised: Petrenko et al., 2020] ( AsyncOnRL ).

Original: Unlike both, we do not overlap experience collection with learning.
Revised: Unlike both, we do not overlap experience collection with learning.

Original: This has various system beneﬁts, including reducing GPU memory usage and reducing GPU driver contention.
Revised: This has various system beneﬁts, including reducing GPU memory usage and reducing GPU driver contention.

Original: More details are available in Appendix A.
Revised: More details are available in Appendix A.

Original: 2.2 Environment-Level Straggler Mitigation
Revised: 2.2 Environment-Level Straggler Mitigation

Original: In both SyncOnRL and AsyncOnRL , the data used for learning consists of N rollouts of equal- T steps of experience, an ( T, N ) -shaped batch.
Revised: In both SyncOnRL and AsyncOnRL , the data used for learning consists of N rollouts of equal- T steps of experience, an ( T, N ) -shaped batch.

Original: In SyncOnRL these N sets are all collected with the current policy, this leads to the environment-level straggler effect.
Revised: In SyncOnRL these N sets are all collected with the current policy, this leads to the environment-level straggler effect.

Original: AsyncOnRL mitigates this by relaxing the constraint that experience must be strictly on-policy, and thereby implicitly changes the experience collection rate for each environment.
Revised: AsyncOnRL mitigates this by relaxing the constraint that experience must be strictly on-policy, and thereby implicitly changes the experience collection rate for each environment.

Original: Variable Experience Rollout ( VER ).
Revised: Variable Experience Rollout ( VER ).

Original: We instead relax the constraint that we must use N rollouts of equal- T steps.
Revised: We instead relax the constraint that we must use N rollouts of equal- T steps.

Original: Speciﬁcally, VER collects T × N steps of experience from N environments without a constraint on how many steps of experience are collected from each environment.
Revised: Speciﬁcally, VER collects T × N steps of experience from N environments without a constraint on how many steps of experience are collected from each environment.

Original: This explicitly varies the experience collection rate for each environment – in effect, collecting more experience from environments that are fast to simulate.
Revised: This explicitly varies the experience collection rate for each environment – in effect, collecting more experience from environments that are fast to simulate.

Original: Consider the 4 environments shows in Fig.
Revised: Consider the 4 environments shows in Fig.

Original: 3A. The length of the each step representation the wall-clock time taken to collect it, some steps are fast, some are slow.
Revised: 3A. The length of the each step representation the wall-clock time taken to collect it, some steps are fast, some are slow.

Original: VER collects more experience from environment 0 as it is fastest to step and less from 1, the slowest.
Revised: VER collects more experience from environment 0 as it is fastest to step and less from 1, the slowest.

Original: Learning mini-batch creation.
Revised: Learning mini-batch creation.

Original: VER is designed with recurrent policies in mind because memory is key in long-range and partially observable tasks like HAB.
Revised: VER is designed with recurrent policies in mind because memory is key in long-range and partially observable tasks like HAB.

Original: When training recurrent policies, we must create mini-batches of experience with sequences for back-propagation-through-time.
Revised: When training recurrent policies, we must create mini-batches of experience with sequences for back-propagation-through-time.

Original: Normally
Revised: Normally

Original: B mini-batches are constructed by spitting the N environments’ experience into B ( T , N / B )-sized mini-batches.
Revised: B mini-batches are constructed by spitting the N environments’ experience into B ( T , N / B )-sized mini-batches.

Original: A similar procedure would result in mini-batches of different sizes with VER .
Revised: A similar procedure would result in mini-batches of different sizes with VER .

Original: This would harm optimization because learning rate and optimization mini-batch size are intertwined and automatically adjusting the learning rate is an open question [Goyal et al., 2017, You et al., 2020].
Revised: This would harm optimization because learning rate and optimization mini-batch size are intertwined and automatically adjusting the learning rate is an open question [Goyal et al., 2017, You et al., 2020].

Original: To under VER ’s mini-batching, ﬁrst note that there are two reasons for the start of a new sequence of experience: rollout starts (Fig.
Revised: To under VER ’s mini-batching, ﬁrst note that there are two reasons for the start of a new sequence of experience: rollout starts (Fig.

Original: 3B, step 0) and episode starts (Fig.
Revised: 3B, step 0) and episode starts (Fig.

Original: 3B, a step after a bar).
Revised: 3B, a step after a bar).

Original: These two boundaries types are independent – episodes can end at any arbitrary step within the rollout and then that environment will reset and start a new episode.
Revised: These two boundaries types are independent – episodes can end at any arbitrary step within the rollout and then that environment will reset and start a new episode.

Original: Thus when we collect experience from N environments, we will have K ≥ N sequences to divide between the mini-batches.
Revised: Thus when we collect experience from N environments, we will have K ≥ N sequences to divide between the mini-batches.

Original: We distributed these K sequences between the mini-batches.
Revised: We distributed these K sequences between the mini-batches.

Original: We randomly order the sequences, then the ﬁrst T × N / B steps are the ﬁrst mini-batch, the next T × N / B to the second, etc .
Revised: We randomly order the sequences, then the ﬁrst T × N / B steps are the ﬁrst mini-batch, the next T × N / B to the second, etc .

Original: See Fig.
Revised: See Fig.

Original: 3B for an example.
Revised: 3B for an example.

Original: Batching computation for learning.
Revised: Batching computation for learning.

Original: The mini-batches constructed from the algorithm above have sequences with variable length.
Revised: The mini-batches constructed from the algorithm above have sequences with variable length.

Original: To batch the computation of these sequences we use cuDNN’s
Revised: To batch the computation of these sequences we use cuDNN’s

Original: PackedSequence data model.
Revised: PackedSequence data model.

Original: This data model represents a set of variable-length sequences (Fig.
Revised: This data model represents a set of variable-length sequences (Fig.

Original: 3C left) such that all sequence-elements at a time-step are contiguous in memory (Fig.
Revised: 3C left) such that all sequence-elements at a time-step are contiguous in memory (Fig.

Original: 3C right) –
Revised: 3C right) –

Original: enabling batched computation on each time-step for components with a temporal dependence, e.g . the RNN – and that all elements across all time-steps are also contiguous – enabling batched computation across all time-steps for network components that don’t have a temporal dependence, e.g . the visual encoder.
Revised: enabling batched computation on each time-step for components with a temporal dependence, e.g . the RNN – and that all elements across all time-steps are also contiguous – enabling batched computation across all time-steps for network components that don’t have a temporal dependence, e.g . the visual encoder.

Original: During experience collection we write experience into a linear buffer in GPU memory and then arrange each mini-batch as a PackedSequence .
Revised: During experience collection we write experience into a linear buffer in GPU memory and then arrange each mini-batch as a PackedSequence .

Original: This takes less than 10 milliseconds (per learning phase); orders of magnitude less than experience collection ( ∼ 3s) or learning ( ∼ 1.5s) in our experiments.
Revised: This takes less than 10 milliseconds (per learning phase); orders of magnitude less than experience collection ( ∼ 3s) or learning ( ∼ 1.5s) in our experiments.

Original: Learning method.
Revised: Learning method.

Original: The experience and mini-batches generated from VER are well-suited for use with
Revised: The experience and mini-batches generated from VER are well-suited for use with

Original: RL methods that use on-policy data [Sutton and Barto, 1992].
Revised: RL methods that use on-policy data [Sutton and Barto, 1992].

Original: We use Proximal Policy Optimization (PPO)
Revised: We use Proximal Policy Optimization (PPO)

Original: [Schulman et al., 2017] as it is known to work well for embodied AI tasks and recurrent policies.
Revised: [Schulman et al., 2017] as it is known to work well for embodied AI tasks and recurrent policies.

Original: Inﬂight actions One subtle design choice is the following – when VER ﬁnishes a T × N experience collection, there will be (slow) environments that haven’t completed simulation yet.
Revised: Inﬂight actions One subtle design choice is the following – when VER ﬁnishes a T × N experience collection, there will be (slow) environments that haven’t completed simulation yet.

Original: Instead of discarding that data, we choose to collect this experience in the next rollout.
Revised: Instead of discarding that data, we choose to collect this experience in the next rollout.

Original: We ﬁnd this choice leads to speed gains without any sample-efﬁciency loss.
Revised: We ﬁnd this choice leads to speed gains without any sample-efﬁciency loss.

Original: 3 Multiple GPUs
Revised: 3 Multiple GPUs

Original: We leverage the decentralized distributed training architecture from Wijmans et al.
Revised: We leverage the decentralized distributed training architecture from Wijmans et al.

Original: [2020] to scale
Revised: [2020] to scale

Original: VER to multiple GPUs (residing on a single or multiple nodes).
Revised: VER to multiple GPUs (residing on a single or multiple nodes).

Original: In this architecture, each GPU both collects experience and learns from that experience.
Revised: In this architecture, each GPU both collects experience and learns from that experience.

Original: During learning, gradients from each
Revised: During learning, gradients from each

Original: GPU-worker are averaged with an AllReduce operation.
Revised: GPU-worker are averaged with an AllReduce operation.

Original: This is a synchronous operation and thus introduces a GPU-worker-level straggler effect.
Revised: This is a synchronous operation and thus introduces a GPU-worker-level straggler effect.

Original: Wijmans et al.
Revised: Wijmans et al.

Original: [2020] mitigate this effect by preempting stagglers – stopping collecting experience early and proceeding to learning – after a ﬁxed number of GPU-workers have ﬁnished experience collection.
Revised: [2020] mitigate this effect by preempting stagglers – stopping collecting experience early and proceeding to learning – after a ﬁxed number of GPU-workers have ﬁnished experience collection.

Original: We instead approximate the optimal preemption for each experience collection phase.
Revised: We instead approximate the optimal preemption for each experience collection phase.

Original: Given the learning time, LT, and a function Time ( S ) that returns the time needed to collect S steps of experience, the optimal number of steps S to collect before preempting stragglers is max S S / ( Time ( S )+ LT ) , s.t. S ≤ ( T · N · #GPUs ) .
Revised: Given the learning time, LT, and a function Time ( S ) that returns the time needed to collect S steps of experience, the optimal number of steps S to collect before preempting stragglers is max S S / ( Time ( S )+ LT ) , s.t. S ≤ ( T · N · #GPUs ) .

Original: For LT, we record the time from the last iteration as this doesn’t change between iterations.
Revised: For LT, we record the time from the last iteration as this doesn’t change between iterations.

Original: We estimate Time ( S ) using the average step time to receive a step of experience from each environment (this includes both inference time and simulation time) from the previous rollout.
Revised: We estimate Time ( S ) using the average step time to receive a step of experience from each environment (this includes both inference time and simulation time) from the previous rollout.

Original: This will be a poor estimation in some circumstances
Revised: This will be a poor estimation in some circumstances

Original: but we ﬁnd it to work well in practice.
Revised: but we ﬁnd it to work well in practice.

Original: We improve upon sample efﬁciency of DD-PPO by ﬁlling the preempted rollouts with experience from the previous rollout.
Revised: We improve upon sample efﬁciency of DD-PPO by ﬁlling the preempted rollouts with experience from the previous rollout.

Original: We perform extra epochs of PPO on this now ‘stale’ data instead of correcting for the off-policy return as we ﬁnd this simpler and effective.
Revised: We perform extra epochs of PPO on this now ‘stale’ data instead of correcting for the off-policy return as we ﬁnd this simpler and effective.

Original: This comes with no effective computation cost since the maximum batch size per GPU is unchanged.
Revised: This comes with no effective computation cost since the maximum batch size per GPU is unchanged.

Original:  Embodied Navigation: Benchmarking
Revised:  Embodied Navigation: Benchmarking

Original: First, we benchmark VER on the embodied navigation [Anderson et al., 2018] tasks in Habitat 1.0 [Savva et al., 2019] – PointNav [Anderson et al., 2018] and ObjectNav [Batra et al., 2020b].
Revised: First, we benchmark VER on the embodied navigation [Anderson et al., 2018] tasks in Habitat 1.0 [Savva et al., 2019] – PointNav [Anderson et al., 2018] and ObjectNav [Batra et al., 2020b].

Original: Our goal here is simply to show training speed-ups in well-studied tasks (and in the case of PointNav, a well-saturated task with no room left for accuracy improvements).
Revised: Our goal here is simply to show training speed-ups in well-studied tasks (and in the case of PointNav, a well-saturated task with no room left for accuracy improvements).

Original: We present accuracy improvements and in-depth analysis on challenging rearrangement tasks in Section 6.
Revised: We present accuracy improvements and in-depth analysis on challenging rearrangement tasks in Section 6.

Original: For both tasks, we use standard architectures from Habitat Baselines [Savva et al., 2019, Wijmans et al., 2020] – the ResNet18 encoder and a 2 layer LSTM.
Revised: For both tasks, we use standard architectures from Habitat Baselines [Savva et al., 2019, Wijmans et al., 2020] – the ResNet18 encoder and a 2 layer LSTM.

Original: Following Ye et al.
Revised: Following Ye et al.

Original: [2020, 2021], we add
Revised: [2020, 2021], we add

Original: Action Conditional Contrastive Coding [Guo et al., 2018].
Revised: Action Conditional Contrastive Coding [Guo et al., 2018].

Original: PointNav.
Revised: PointNav.

Original: We train PointNav agents with one RGB camera on the HM3D dataset [Ramakrishnan et al., 2021] for 1.85 billion steps of experience on 8 GPUs.
Revised: We train PointNav agents with one RGB camera on the HM3D dataset [Ramakrishnan et al., 2021] for 1.85 billion steps of experience on 8 GPUs.

Original: We study the RGB setting (and not Depth ) because this is the more challenging version of the task and thus we expect it to be more sensitive to
Revised: We study the RGB setting (and not Depth ) because this is the more challenging version of the task and thus we expect it to be more sensitive to

Original: [Figure]
Revised: [Figure]

Original: possible differences in the training system.
Revised: possible differences in the training system.

Original: We examine VER along two axes: 1) training throughput – the number of samples of experience per second (SPS) the system collects and learns from, 2) sample efﬁciency.
Revised: We examine VER along two axes: 1) training throughput – the number of samples of experience per second (SPS) the system collects and learns from, 2) sample efﬁciency.

Original: On 8 GPUs, VER trains agents 60% faster than DD-PPO, from 3065 SPS to 5325 SPS (Fig. 4A), with similar sample efﬁciency (Fig. 4B).
Revised: On 8 GPUs, VER trains agents 60% faster than DD-PPO, from 3065 SPS to 5325 SPS (Fig. 4A), with similar sample efﬁciency (Fig. 4B).

Original: ObjectNav.
Revised: ObjectNav.

Original: We train ObjectNav agents with one RGB and one Depth camera on the MP3D [Chang et al., 2017] dataset for 600 million steps of experience on 8 GPUs.
Revised: We train ObjectNav agents with one RGB and one Depth camera on the MP3D [Chang et al., 2017] dataset for 600 million steps of experience on 8 GPUs.

Original: VER trains agents 100% faster than
Revised: VER trains agents 100% faster than

Original: DD-PPO, 1021 to 2019 (Fig. 4A), with slightly better sample efﬁciency (Fig.
Revised: DD-PPO, 1021 to 2019 (Fig. 4A), with slightly better sample efﬁciency (Fig.

Original: 4B).
Revised: 4B).

Original: Due to improved throughput and sample efﬁciency, VER uses 4.3x less compute than DD-PPO to reach the same success (Fig. 4C).
Revised: Due to improved throughput and sample efﬁciency, VER uses 4.3x less compute than DD-PPO to reach the same success (Fig. 4C).

Original: There are two effects that enable better sample efﬁciency with VER .
Revised: There are two effects that enable better sample efﬁciency with VER .

Original: First, we perform additional epochs of PPO on experience from the last rollout when a GPU-worker is preempted.
Revised: First, we perform additional epochs of PPO on experience from the last rollout when a GPU-worker is preempted.

Original: Second, the variable experience rollout mechanism results in a natural curriculum.
Revised: Second, the variable experience rollout mechanism results in a natural curriculum.

Original: Environments are often faster to simulate when they are easier ( i.e . a smaller home), so more experience will be collected in these easier cases.
Revised: Environments are often faster to simulate when they are easier ( i.e . a smaller home), so more experience will be collected in these easier cases.

Original:  Embodied Rearrangement: Task, Agent, and Training
Revised:  Embodied Rearrangement: Task, Agent, and Training

Original: Next, we use VER to study the recently introduced (and more challenging) GeometricGoal rearrangement rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].
Revised: Next, we use VER to study the recently introduced (and more challenging) GeometricGoal rearrangement rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].

Original: Task.
Revised: Task.

Original: In GeoRearrange, an agent is initialized in an unknown environment and tasked with rearranging objects in its environment.
Revised: In GeoRearrange, an agent is initialized in an unknown environment and tasked with rearranging objects in its environment.

Original: The task is speciﬁed as a set of coordinate pairs { ( Pose Initial , Pose Final ) } Oo =1 .
Revised: The task is speciﬁed as a set of coordinate pairs { ( Pose Initial , Pose Final ) } Oo =1 .

Original: The agent must bring each object at Pose Initial to Pose Final where Pose is the initial or desired centerof-mass location for the object(s).
Revised: The agent must bring each object at Pose Initial to Pose Final where Pose is the initial or desired centerof-mass location for the object(s).

Original: We use the Home Assistant Benchmark (HAB) which consists of 3 scenarios of increasing difﬁculty: Tidy House, Prepare Groceries, and Set Table.
Revised: We use the Home Assistant Benchmark (HAB) which consists of 3 scenarios of increasing difﬁculty: Tidy House, Prepare Groceries, and Set Table.

Original: In Tidy House, the agent is tasked with moving 5 objects from their initial locations to their ﬁnal locations.
Revised: In Tidy House, the agent is tasked with moving 5 objects from their initial locations to their ﬁnal locations.

Original: The objects are rarely in containers ( i.e . fridge or cabinet drawer) and when they are, the containers are already opened.
Revised: The objects are rarely in containers ( i.e . fridge or cabinet drawer) and when they are, the containers are already opened.

Original: In Prepare Groceries, the agent must move 3 objects from the kitchen counter into the open fridge (or open fridge to counter).
Revised: In Prepare Groceries, the agent must move 3 objects from the kitchen counter into the open fridge (or open fridge to counter).

Original: This stresses picking and placing in the fridge, which is challenging.
Revised: This stresses picking and placing in the fridge, which is challenging.

Original: In Set Table, the agent must move 1 object from the closed kitchen cabinet drawer to the table and 1 object from the closed fridge to the table.
Revised: In Set Table, the agent must move 1 object from the closed kitchen cabinet drawer to the table and 1 object from the closed fridge to the table.

Original: This requires opening the cabinet and fridge, and picking from these challenging receptacles.
Revised: This requires opening the cabinet and fridge, and picking from these challenging receptacles.

Original: Simulation.
Revised: Simulation.

Original: We use the Habitat simulator with the ReplicaCAD Dataset [Savva et al., 2019, Szot et al., 2021].
Revised: We use the Habitat simulator with the ReplicaCAD Dataset [Savva et al., 2019, Szot et al., 2021].

Original: The robot policy operates at 30 Hz and physics is simulated at 120 Hz.
Revised: The robot policy operates at 30 Hz and physics is simulated at 120 Hz.

Original: Agent.
Revised: Agent.

Original: The agent is embodied as a Fetch robot with a 7-DOF arm.
Revised: The agent is embodied as a Fetch robot with a 7-DOF arm.

Original: The arm is controlled via joint velocities.
Revised: The arm is controlled via joint velocities.

Original: At every time step the policy predicts a delta in motor position for joint in the arm.
Revised: At every time step the policy predicts a delta in motor position for joint in the arm.

Original: We ﬁnd joint velocity control equally easy to learn but faster to simulate than the end-effector control used in
Revised: We ﬁnd joint velocity control equally easy to learn but faster to simulate than the end-effector control used in

Original: Szot et al.
Revised: Szot et al.

Original: The arm is equipped with a suction gripper.
Revised: The arm is equipped with a suction gripper.

Original: The agent must control the arm such that the gripper is in contact with the object to grasp and then activate the gripper.
Revised: The agent must control the arm such that the gripper is in contact with the object to grasp and then activate the gripper.

Original: The object is dropped once the gripper is deactivated.
Revised: The object is dropped once the gripper is deactivated.

Original: This is more realistic than the ‘magic’ grasp action used in Szot et al.
Revised: This is more realistic than the ‘magic’ grasp action used in Szot et al.

Original: The robot base (navigation) is controlled by the policy commanding a desired linear speed and
Revised: The robot base (navigation) is controlled by the policy commanding a desired linear speed and

Original: [Table]
Revised: [Table]

Original: angular velocity.
Revised: angular velocity.

Original: The robot is equipped with one Depth camera attached to its head, proprioceptive sensors that provide the joint positions of its arm, and a GPS+Compass sensor that provides its heading and location relative to its initial location.
Revised: The robot is equipped with one Depth camera attached to its head, proprioceptive sensors that provide the joint positions of its arm, and a GPS+Compass sensor that provides its heading and location relative to its initial location.

Original: The policy models a t ∼ π ( · | s t − 1 ) instead of a t ∼ π ( · | s t ) .
Revised: The policy models a t ∼ π ( · | s t − 1 ) instead of a t ∼ π ( · | s t ) .

Original: This is more realistic and enables physics and rendering to be overlapped [Szot et al., 2021].
Revised: This is more realistic and enables physics and rendering to be overlapped [Szot et al., 2021].

Original: We build upon the TaskPlanning-SkillRL
Revised: We build upon the TaskPlanning-SkillRL

Original: (TP-SRL) method proposed in Szot et al.
Revised: (TP-SRL) method proposed in Szot et al.

Original: TP-SRL is a hierarchical method for GeoRearrange that decomposes the task into a series of skills – Navigate ,
Revised: TP-SRL is a hierarchical method for GeoRearrange that decomposes the task into a series of skills – Navigate ,

Original: Pick , Place , and { Open , Close } × { Cabinet, Fridge } .
Revised: Pick , Place , and { Open , Close } × { Cabinet, Fridge } .

Original: Skills are controlled via a skill-policy (learned with RL) and chained together via a task planner.
Revised: Skills are controlled via a skill-policy (learned with RL) and chained together via a task planner.

Original: One of the key challenges is the ‘handoff problem’ – downstream skills are setup for failure due to slight errors made by the upstream skill.
Revised: One of the key challenges is the ‘handoff problem’ – downstream skills are setup for failure due to slight errors made by the upstream skill.

Original: We give all skill policies access to navigation actions to allow them to correct for these errors.
Revised: We give all skill policies access to navigation actions to allow them to correct for these errors.

Original: Architecture.
Revised: Architecture.

Original: All skill policies share the same architecture.
Revised: All skill policies share the same architecture.

Original: We use ResNet

Revised: We use ResNet

Original: [He et al., 2016] to process the 128 × 128 visual input.
Revised: [He et al., 2016] to process the 128 × 128 visual input.

Original: Following Wijmans et al.
Revised: Following Wijmans et al.

Original: [2020], we reduce with width of the network by half and use GroupNorm [Wu and He, 2018].
Revised: [2020], we reduce with width of the network by half and use GroupNorm [Wu and He, 2018].

Original: We also apply some of the recent advancements from ConvNeXt [Liu et al., 2022].
Revised: We also apply some of the recent advancements from ConvNeXt [Liu et al., 2022].

Original: We use a patch-ify stem, dedicated down-sample stages, layer scale [Touvron et al., 2021], and dilated convolutions [Yu and Koltun, 2015] (this mimics larger kernel convolutions without increasing computation).
Revised: We use a patch-ify stem, dedicated down-sample stages, layer scale [Touvron et al., 2021], and dilated convolutions [Yu and Koltun, 2015] (this mimics larger kernel convolutions without increasing computation).

Original: The visual embedding is then combined with the proprioceptive observations and previous action, and then processed with a 2-layer
Revised: The visual embedding is then combined with the proprioceptive observations and previous action, and then processed with a 2-layer

Original: LSTM [Hochreiter and Schmidhuber, 1997].
Revised: LSTM [Hochreiter and Schmidhuber, 1997].

Original: The output of the LSTM is used to predict the action distribution and value function.
Revised: The output of the LSTM is used to predict the action distribution and value function.

Original: Actions are sampled from independent Gaussian distributions.
Revised: Actions are sampled from independent Gaussian distributions.

Original: Training.
Revised: Training.

Original: We train agents using VER and PPO [Schulman et al., 2017] with Generalized Advantage Estimation [Schulman et al., 2016].
Revised: We train agents using VER and PPO [Schulman et al., 2017] with Generalized Advantage Estimation [Schulman et al., 2016].

Original: We use a minimum entropy constraint with a learned coefﬁcient [Haarnoja et al., 2018] as we ﬁnd this to be more stable given our diverse set of skills than a ﬁxed coefﬁcient.
Revised: We use a minimum entropy constraint with a learned coefﬁcient [Haarnoja et al., 2018] as we ﬁnd this to be more stable given our diverse set of skills than a ﬁxed coefﬁcient.

Original: Formally, let H ( π ) be the entropy of the policy, we then minimize α ( λ − [[ H ( π )]] sg  ) − [[ α ]] sg H ( π ) where [[ · ]] sg is the stop gradient operator.
Revised: Formally, let H ( π ) be the entropy of the policy, we then minimize α ([[ H ( π )]] sg − λ ) − [[ α ]] sg H ( π ) where [[ · ]] sg is the stop gradient operator.

Original: We set the target entropy, λ , to zero for all tasks.
Revised: We set the target entropy, λ , to zero for all tasks.

Original: We use the Adam optimizer [Kingma and Ba, 2015] with an initial learning rate of 2 . 5 × 10 − 4 and decay it to zero with a cosine schedule.
Revised: We use the Adam optimizer [Kingma and Ba, 2015] with an initial learning rate of 2 . 5 × 10 − 4 and decay it to zero with a cosine schedule.

Original: To correct for biased sampling in VER , we use truncated importance sampling weighting [Espeholt et al., 2018] with a maximum of 1.0.
Revised: To correct for biased sampling in VER , we use truncated importance sampling weighting [Espeholt et al., 2018] with a maximum of 1.0.

Original:  Embodied Rearrangement: Benchmarking
Revised:  Embodied Rearrangement: Benchmarking

Original: In this section, we benchmark VER for training open-fridge policies because this task involves interaction of the robot with an articulated object (the fridge) and represents a challenging case for the training system due to large variability in physics time.
Revised: In this section, we benchmark VER for training open-fridge policies because this task involves interaction of the robot with an articulated object (the fridge) and represents a challenging case for the training system due to large variability in physics time.

Original: In Section 6, we analyze the task performance, which requires all skills.
Revised: In Section 6, we analyze the task performance, which requires all skills.

Original: For all systems, we set the number of environments, N , to 16 per GPU.
Revised: For all systems, we set the number of environments, N , to 16 per GPU.

Original: 5.1 System throughout
Revised: 5.1 System throughout

Original: VER is 150% faster than DD-PPO (Table 1); an even larger difference than in the simpler navigation tasks we studied before.
Revised: VER is 150% faster than DD-PPO (Table 1); an even larger difference than in the simpler navigation tasks we studied before.

Original: DD-PPO has no mechanism to mitigate the action-level or episode-level straggler effects.
Revised: DD-PPO has no mechanism to mitigate the action-level or episode-level straggler effects.

Original: In their absence, DD-PPO has similar throughput as VER (Table 1, Max).
Revised: In their absence, DD-PPO has similar throughput as VER (Table 1, Max).

Original: Under these effects, DD-PPO’s throughput reduces 150% compared to 20% for VER (Table 1, Mean vs. Max).
Revised: Under these effects, DD-PPO’s throughput reduces 150% compared to 20% for VER (Table 1, Mean vs. Max).

Original: Variable experience rollouts are effective.
Revised: Variable experience rollouts are effective.

Original: We compare VER with VER minus variable experience rollouts (No VER ).
Revised: We compare VER with VER minus variable experience rollouts (No VER ).

Original: No VER is a ‘steel-manned’ baseline for VER and beneﬁts from all our microoptimizations.
Revised: No VER is a ‘steel-manned’ baseline for VER and beneﬁts from all our microoptimizations.

Original: VER is 30% faster than No VER (Table 1).
Revised: VER is 30% faster than No VER (Table 1).

Original: VER closes the gap to AsyncOnRL .
Revised: VER closes the gap to AsyncOnRL .

Original: On 1 GPU, VER is as fast as SampleFactory [Petrenko et al., 2020], the fastest single machine AsyncOnRL .
Revised: On 1 GPU, VER is as fast as SampleFactory [Petrenko et al., 2020], the fastest single machine AsyncOnRL .

Original: Intuitively AsyncOnRL should be a strict upper-bound on performance – it never stops collecting experience while VER does.
Revised: Intuitively AsyncOnRL should be a strict upper-bound on performance – it never stops collecting experience while VER does.

Original: However this doesn’t take into
Revised: However this doesn’t take into

Original: [Figure]
Revised: [Figure]

Original: account the realities of hardware.
Revised: account the realities of hardware.

Original: Recall that we are training an agent with a large visual encoder.
Revised: Recall that we are training an agent with a large visual encoder.

Original: This means that updating the parameters of the agent takes a large amount of time ( ∼ 150ms per mini-batch of size 1024 on a V100).
Revised: This means that updating the parameters of the agent takes a large amount of time ( ∼ 150ms per mini-batch of size 1024 on a V100).

Original: Further, Habitat uses the GPU for rendering.
Revised: Further, Habitat uses the GPU for rendering.

Original: The use of the GPU for both rendering and learning simultaneously results in GPU driver contention.
Revised: The use of the GPU for both rendering and learning simultaneously results in GPU driver contention.

Original: In SampleFactory, learning time and experience collection time are roughly double that of VER .
Revised: In SampleFactory, learning time and experience collection time are roughly double that of VER .

Original: Multi-GPU scaling.
Revised: Multi-GPU scaling.

Original: VER has better multi-GPU scaling than DD-PPO, achieving a 6.7x speed-up on 8 GPUs compared to 6x.
Revised: VER has better multi-GPU scaling than DD-PPO, achieving a 6.7x speed-up on 8 GPUs compared to 6x.

Original: The rollout collection time in VER is lower variance, which improves scaling.
Revised: The rollout collection time in VER is lower variance, which improves scaling.

Original: On 2 GPUs, SampleFactory is 12% faster than VER .
Revised: On 2 GPUs, SampleFactory is 12% faster than VER .

Original: Here one GPU is used for learning+inference and the other is used for rendering.
Revised: Here one GPU is used for learning+inference and the other is used for rendering.

Original: This creates a nice division of work and doesn’t result in costly
Revised: This creates a nice division of work and doesn’t result in costly

Original: GPU contention.
Revised: GPU contention.

Original: On 4 and 8 GPUs however, the single GPU used for learning in SampleFactory is the bottleneck and VER has higher throughput (nearly 100% faster on 8 GPUs).
Revised: On 4 and 8 GPUs however, the single GPU used for learning in SampleFactory is the bottleneck and VER has higher throughput (nearly 100% faster on 8 GPUs).

Original: While it is possible to implement multi-GPU learning for AsyncOnRL , it is left to the user to balance the number of GPUs used for experience collection and learning.
Revised: While it is possible to implement multi-GPU learning for AsyncOnRL , it is left to the user to balance the number of GPUs used for experience collection and learning.

Original: VER balances GPU-time between these automatically.
Revised: VER balances GPU-time between these automatically.

Original: 5.2 Sample and Compute Efﬁciency
Revised: 5.2 Sample and Compute Efﬁciency

Original: Next we examine sample efﬁciency of the training systems.
Revised: Next we examine sample efﬁciency of the training systems.

Original: On 1 GPU, VER has slightly worse sample efﬁciency than DD-PPO (Fig. 5).
Revised: On 1 GPU, VER has slightly worse sample efﬁciency than DD-PPO (Fig. 5).

Original: We suspect that this could be ﬁxed by hyper-parameter choice.
Revised: We suspect that this could be ﬁxed by hyper-parameter choice.

Original: On 2 and 4 GPUs, VER has identical sample efﬁciency and better sample efﬁciency on 8 GPUs, Fig.
Revised: On 2 and 4 GPUs, VER has identical sample efﬁciency and better sample efﬁciency on 8 GPUs, Fig.

Original: Interestingly, SampleFactory has similar sample efﬁciency with 1 GPU (Fig. 5) and is more stable.
Revised: Interestingly, SampleFactory has similar sample efﬁciency with 1 GPU (Fig. 5) and is more stable.

Original: This is because reducing the number of GPUs also reduces the batch-size and PPO is known to not be batch-size invariant [Hilton et al., 2021].
Revised: This is because reducing the number of GPUs also reduces the batch-size and PPO is known to not be batch-size invariant [Hilton et al., 2021].

Original: We believe the stale data serves a similar function to
Revised: We believe the stale data serves a similar function to

Original: PPO-EWMA
Revised: PPO-EWMA

Original: [Hilton et al., 2021], which uses an exponential moving average of the policy weights for the trust region.
Revised: [Hilton et al., 2021], which uses an exponential moving average of the policy weights for the trust region.

Original: On 2, 4, and 8 GPUs, VER has better sample efﬁciency than SampleFactory.
Revised: On 2, 4, and 8 GPUs, VER has better sample efﬁciency than SampleFactory.

Original: Combing the ﬁndings of throughput and sample efﬁciency, we ﬁnd that VER always achieves a given performance threshold in the least amount of training time (outright or a tie) (Fig. 5 Lower).
Revised: Combing the ﬁndings of throughput and sample efﬁciency, we ﬁnd that VER always achieves a given performance threshold in the least amount of training time (outright or a tie) (Fig. 5 Lower).

Original:  Embodied Rearrangement: Analysis of Learned Skills
Revised:  Embodied Rearrangement: Analysis of Learned Skills

Original: We examine the performance of TP-SRL on the Home Assistant Benchmark (HAB)
Revised: We examine the performance of TP-SRL on the Home Assistant Benchmark (HAB)

Original: [Szot et al., 2021].
Revised: [Szot et al., 2021].

Original: We examine both skill policies trained with the full action space and the limited, per-skill speciﬁc action space used in Szot et al.
Revised: We examine both skill policies trained with the full action space and the limited, per-skill speciﬁc action space used in Szot et al.

Original: Each skill is trained with VER for 500 million steps of experience on 8 GPUs.
Revised: Each skill is trained with VER for 500 million steps of experience on 8 GPUs.

Original: This takes less than 2 days per skill.
Revised: This takes less than 2 days per skill.

Original: 6.1 Performance on HAB
Revised: 6.1 Performance on HAB

Original: We ﬁnd that skills with navigation actions improve performance on the full task but does not change performance on the skill’s train task ( e.g . Pick achieves 90% success with and without during training).
Revised: We ﬁnd that skills with navigation actions improve performance on the full task but does not change performance on the skill’s train task ( e.g . Pick achieves 90% success with and without during training).

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 6 shows smaller drops in performance between every interaction (there is a navigation between each interaction) demonstrating that skills with navigation effectively correct for handoff errors.
Revised: 6 shows smaller drops in performance between every interaction (there is a navigation between each interaction) demonstrating that skills with navigation effectively correct for handoff errors.

Original: This is impactful after place as the navigation policy tends to make more errors when navigating to the next location after place.
Revised: This is impactful after place as the navigation policy tends to make more errors when navigating to the next location after place.

Original: On Tidy House, full task performance improves from 2% Success to 32%
Revised: On Tidy House, full task performance improves from 2% Success to 32%

Original: On Prepare Groceries (which requires picking/placing from/into the fridge) and Set Table (which requires opening the fridge/cabinet and then picking from it) performance improves slightly.
Revised: On Prepare Groceries (which requires picking/placing from/into the fridge) and Set Table (which requires opening the fridge/cabinet and then picking from it) performance improves slightly.

Original: 0% to 5% on Prepare Groceries and 0% to 7.5% on the ﬁrst pick+place on Set Table.
Revised: 0% to 5% on Prepare Groceries and 0% to 7.5% on the ﬁrst pick+place on Set Table.

Original: Both these tasks remain as open problems and the next frontier.
Revised: Both these tasks remain as open problems and the next frontier.

Original: 6.2 Emergent Navigation
Revised: 6.2 Emergent Navigation

Original: Next we examine if the skill policies are able to navigate to correct for highly out-of-distribution initial location.
Revised: Next we examine if the skill policies are able to navigate to correct for highly out-of-distribution initial location.

Original: We examine TP-SRL(NoNav), which that omits the navigation skill.
Revised: We examine TP-SRL(NoNav), which that omits the navigation skill.

Original: In this agent all navigation is done by skill policies that ostensibly never needed to navigate during training.
Revised: In this agent all navigation is done by skill policies that ostensibly never needed to navigate during training.

Original: We ﬁnd emergent navigation in both the Pick and Place policies.
Revised: We ﬁnd emergent navigation in both the Pick and Place policies.

Original: TP-SRL(NoNav) achieves 50% success on NavPick (Fig. 6, Tidy House interaction 1), and 20% success on NavPickNavPlace (Fig. 6,
Revised: TP-SRL(NoNav) achieves 50% success on NavPick (Fig. 6, Tidy House interaction 1), and 20% success on NavPickNavPlace (Fig. 6,

Original: Tidy House interaction 2).
Revised: Tidy House interaction 2).

Original: The latter is on-par with the TaskPlanning+SensePlanAct (TP-SPA) classical baseline and signiﬁcantly better than the MonolithicRL baseline in Szot et al.
Revised: The latter is on-par with the TaskPlanning+SensePlanAct (TP-SPA) classical baseline and signiﬁcantly better than the MonolithicRL baseline in Szot et al.

Original: The Pick and Place policies were trained on tasks that requires no navigation but both are capable of navigation.
Revised: The Pick and Place policies were trained on tasks that requires no navigation but both are capable of navigation.

Original: We provided examples of both the training task for Pick and Place , and TP-SRL(NoNav) on Tidy House in the supplementary materials.
Revised: We provided examples of both the training task for Pick and Place , and TP-SRL(NoNav) on Tidy House in the supplementary materials.

Original: On Prepare Groceries and Set Table, the navigation performance of these policies is worse (-34%
Revised: On Prepare Groceries and Set Table, the navigation performance of these policies is worse (-34%

Original: Success and -45% Success on the ﬁrst interaction, respectively).
Revised: Success and -45% Success on the ﬁrst interaction, respectively).

Original: Prepare Groceries requires picking from the fridge, which is challenging and requires navigation that doesn’t accidentally close the door.
Revised: Prepare Groceries requires picking from the fridge, which is challenging and requires navigation that doesn’t accidentally close the door.

Original: Set Table requires opening the cabinet and then picking, which introduces an additional OpenCab skill and requires more precise navigation and picking from Pick .
Revised: Set Table requires opening the cabinet and then picking, which introduces an additional OpenCab skill and requires more precise navigation and picking from Pick .

Original: Performance is non-zero (15% and 4% on the ﬁrst interaction, respectively) in both scenarios; indicting that the skill policies are capable of navigating even in these scenarios, albeit less successfully than in Tidy House.
Revised: Performance is non-zero (15% and 4% on the ﬁrst interaction, respectively) in both scenarios; indicting that the skill policies are capable of navigating even in these scenarios, albeit less successfully than in Tidy House.

Original: We hypothesize that the Pick and Place polices learned navigation because this was useful early in training.
Revised: We hypothesize that the Pick and Place polices learned navigation because this was useful early in training.

Original: Early in training the policies have yet to learn that only minimal navigation is needed to complete the task.
Revised: Early in training the policies have yet to learn that only minimal navigation is needed to complete the task.

Original: Therefore the policy will sometimes cause itself to move away from the pick object/place location and will navigate back.
Revised: Therefore the policy will sometimes cause itself to move away from the pick object/place location and will navigate back.

Original: Navigation is then not forgotten as the policy converges.
Revised: Navigation is then not forgotten as the policy converges.

Original: We examined the behavior of a Pick policy early in training and found that it does tend to move away from the object it needs to pick up and sometimes moves back.
Revised: We examined the behavior of a Pick policy early in training and found that it does tend to move away from the object it needs to pick up and sometimes moves back.

Original: Although the magnitude of navigation is small and quite infrequent, so the degree of generalization is high.
Revised: Although the magnitude of navigation is small and quite infrequent, so the degree of generalization is high.

Original: This result, and higher performance on HAB, highlights that it may not always be beneﬁcial to remove ‘unneeded’ actions.
Revised: This result, and higher performance on HAB, highlights that it may not always be beneﬁcial to remove ‘unneeded’ actions.

Original: Szot et al.
Revised: Szot et al.

Original: [2021] removed navigation where possible to improve sample efﬁciency and training throughput 3 .
Revised: [2021] removed navigation where possible to improve sample efﬁciency and training throughput 3 .

Original: Our experiments corroborate this; training without navigation improves both sample efﬁciency and training speed.
Revised: Our experiments corroborate this; training without navigation improves both sample efﬁciency and training speed.

Original: However, by enabling navigation and allowing the agent to learn how to (not) use it, we arrived upon emergent navigation and improved HAB performance.
Revised: However, by enabling navigation and allowing the agent to learn how to (not) use it, we arrived upon emergent navigation and improved HAB performance.

Original:  Related Work
Revised:  Related Work

Original: AsyncOnRL methods provide high-throughput on-policy reinforcement learning [Espeholt et al., 2018,
Revised: AsyncOnRL methods provide high-throughput on-policy reinforcement learning [Espeholt et al., 2018,

Original: Petrenko et al., 2020].
Revised: Petrenko et al., 2020].

Original: However, they have reduced sample efﬁciency as they must correct for nearpolicy, or ‘stale’, data.
Revised: However, they have reduced sample efﬁciency as they must correct for nearpolicy, or ‘stale’, data.

Original: Few support multi-GPU learning and, when they do, the user must manually balance compute between learning and experience collection [Espeholt et al., 2020].
Revised: Few support multi-GPU learning and, when they do, the user must manually balance compute between learning and experience collection [Espeholt et al., 2020].

Original: VER achieves the same throughput on 1-GPU while learning with on-policy data, has better sample efﬁciency, supports multi-GPU learning, and automatically balances compute between learning and simulation.
Revised: VER achieves the same throughput on 1-GPU while learning with on-policy data, has better sample efﬁciency, supports multi-GPU learning, and automatically balances compute between learning and simulation.

Original: SyncOnRL .
Revised: SyncOnRL .

Original: HTS-RL [Liu et al., 2020] also use the experience collection techniques as AsyncOnRL to mitigate the action-level straggler effect.
Revised: HTS-RL [Liu et al., 2020] also use the experience collection techniques as AsyncOnRL to mitigate the action-level straggler effect.

Original: Unfortunately inefﬁciencies in the provided implementation prevent a meaningful direct comparison and hide the full effectiveness of this technique.
Revised: Unfortunately inefﬁciencies in the provided implementation prevent a meaningful direct comparison and hide the full effectiveness of this technique.

Original: In
Revised: In

Original: Appendix E we show that our re-implementation is 110% faster (2.1x speedup) and thus instead compare to the stronger baseline of No VER in the main text.
Revised: Appendix E we show that our re-implementation is 110% faster (2.1x speedup) and thus instead compare to the stronger baseline of No VER in the main text.

Original: We propose a novel mechanism, variable experience rollouts, to mitigate the episode-level straggler effect and thereby close the gap to
Revised: We propose a novel mechanism, variable experience rollouts, to mitigate the episode-level straggler effect and thereby close the gap to

Original: AsyncOnRL .
Revised: AsyncOnRL .

Original: We use and build upon Decentralized Distributed PPO (DD-PPO)
Revised: We use and build upon Decentralized Distributed PPO (DD-PPO)

Original: [Wijmans et al., 2020], which proposed a distributed multi-GPU method based on data parallelism [Hillis and Steele Jr, 1986].
Revised: [Wijmans et al., 2020], which proposed a distributed multi-GPU method based on data parallelism [Hillis and Steele Jr, 1986].

Original: Batched simulators simulate multiple agents (in multiple environments) simultaneously and are responsible for their own parallelization [Shacklett et al., 2021, Petrenko et al., 2021, Freeman et al., 2021, Makoviychuk et al., 2021].
Revised: Batched simulators simulate multiple agents (in multiple environments) simultaneously and are responsible for their own parallelization [Shacklett et al., 2021, Petrenko et al., 2021, Freeman et al., 2021, Makoviychuk et al., 2021].

Original: While these systems offer impressive performance, none currently support a benchmark like HAB (which combines physics and photo-realism) nor the ﬂexibility of
Revised: While these systems offer impressive performance, none currently support a benchmark like HAB (which combines physics and photo-realism) nor the ﬂexibility of

Original: Habitat, AI2Thor [Kolve et al., 2017], or ThreeDWorld [Gan et al., 2020].
Revised: Habitat, AI2Thor [Kolve et al., 2017], or ThreeDWorld [Gan et al., 2020].

Original: VER enables researchers to ﬁrst explore promising directions using existing simulators and then build batched simulators with the knowledged gained from their ﬁndings.
Revised: VER enables researchers to ﬁrst explore promising directions using existing simulators and then build batched simulators with the knowledged gained from their ﬁndings.

Original:  Societal Impact, Limitations, and Conclusion
Revised:  Societal Impact, Limitations, and Conclusion

Original: Our main application result is trained using the ReplicaCAD dataset [Szot et al., 2021], which is limited to only US apartments, and this may have negative societal impacts for deployed assistants.
Revised: Our main application result is trained using the ReplicaCAD dataset [Szot et al., 2021], which is limited to only US apartments, and this may have negative societal impacts for deployed assistants.

Original: VER was designed and evaluated for tasks with both GPU simulation and large neural networks.
Revised: VER was designed and evaluated for tasks with both GPU simulation and large neural networks.

Original: For tasks with CPU simulation and smaller networks, we expect it to improve upon SyncOnRL
Revised: For tasks with CPU simulation and smaller networks, we expect it to improve upon SyncOnRL

Original: but it may have less throughput than AsyncOnRL and overlapping experience collection and learning would likely be beneﬁcial.
Revised: but it may have less throughput than AsyncOnRL and overlapping experience collection and learning would likely be beneﬁcial.

Original: Our implementation supports overlapping learning and collecting 1 rollout, but more overlap may be beneﬁcial.
Revised: Our implementation supports overlapping learning and collecting 1 rollout, but more overlap may be beneﬁcial.

Original: The TP-SRL agent we build upon requires oracle knowledge, e.g . that the cabinet must be opened before picking.
Revised: The TP-SRL agent we build upon requires oracle knowledge, e.g . that the cabinet must be opened before picking.

Original: We have presented Variable Experience Rollout ( VER ).
Revised: We have presented Variable Experience Rollout ( VER ).

Original: VER combines the strengths of and blurs the line between SyncOnRL and AsyncOnRL .
Revised: VER combines the strengths of and blurs the line between SyncOnRL and AsyncOnRL .

Original: Its trains agents for embodied navigation tasks in Habitat 1.0 60%-100% faster (1.6x to 2x speedup) than DD-PPO with similar sample efﬁciency – saving 19.
Revised: Its trains agents for embodied navigation tasks in Habitat 1.0 60%-100% faster (1.6x to 2x speedup) than DD-PPO with similar sample efﬁciency – saving 19.

Original: GPU-days on PointNav and 28 GPU-day for ObjectNav per seed in our experiments.
Revised: GPU-days on PointNav and 28 GPU-day for ObjectNav per seed in our experiments.

Original: On the recently introduced (and more challenging) embodied rearrangement tasks in Habitat 2.0, VER trains agents 150% faster than DD-PPO and is fast as SampleFactory ( AsyncOnRL ) on 1 GPU.
Revised: On the recently introduced (and more challenging) embodied rearrangement tasks in Habitat 2.0, VER trains agents 150% faster than DD-PPO and is fast as SampleFactory ( AsyncOnRL ) on 1 GPU.

Original: On 8 GPUs, VER is 180% faster than DD-PPO and 70% faster than SampleFactory with better sample efﬁciency – saving 32 GPU-days vs. DD-PPO and 11.2 GPU-days vs. SampleFactory per skill in our experiments.
Revised: On 8 GPUs, VER is 180% faster than DD-PPO and 70% faster than SampleFactory with better sample efﬁciency – saving 32 GPU-days vs. DD-PPO and 11.2 GPU-days vs. SampleFactory per skill in our experiments.

Original: We use
Revised: We use

Original: VER to study rearrangement.
Revised: VER to study rearrangement.

Original: We ﬁnd the emergence of navigation in policies that ostensibly require no navigation when given access to navigation actions.
Revised: We ﬁnd the emergence of navigation in policies that ostensibly require no navigation when given access to navigation actions.

Original: This results in strong progress on Tidy House (+30% success) and shows that it may not always be advantageous to remove ‘unneeded actions’.
Revised: This results in strong progress on Tidy House (+30% success) and shows that it may not always be advantageous to remove ‘unneeded actions’.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: The Georgia Tech effort was supported in part by NSF, ONR YIP, and ARO
Revised: The Georgia Tech effort was supported in part by NSF, ONR YIP, and ARO

Original: PECASE.
Revised: PECASE.

Original: EW is supported in part by an ARCS fellowship.
Revised: EW is supported in part by an ARCS fellowship.

Original: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
Revised: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.

Original: A Systems Considerations A.1 Memory beneﬁts of not overlapping experience collection and learning

Original: VER does not overlap experience collection and learning.

Original: This allows the main process to act as an inference worker during experience collection and a learner during learning, saving 1 CUDA context ( ∼ 1GB of GPU memory).

Original: Unlike AsyncOnRL , VER is able to use GPU shared memory to transfer experience from inference workers to learner.

Original: This is because VER uses (at least) half the GPU memory for storing rollouts.

Original: AsyncOnRL must maintain at least 1 set of rollout buffers for storing the rollout currently being used for learning an another set to store the next rollout being collected.

Original: In practice, they maintain more than this minimum 2x to enable faster training.

Original: A.2 Graphics Processing Units (GPUs)

Original: The application of graphics processing units (GPUs) to training neural networks has led to many advances in artiﬁcial intelligence.

Original: In reinforcement learning, GPUs are used for policy inference, learning the policy, and rendering visual observations like Depth or RGB .

Original: Given the importance of

Original: GPUs in reinforcement learning, there are several important attributes of them to understand.

Original: 1) GPUs are not optimized for multiple processes using them concurrently.

Original: They are unable to efﬁciently execute multiple processes at once (if at all).

Original: While sometimes concurrent use is the only option, a system should whenever possible have a single process that has exclusive use of the GPU and uses it fully.

Original: GPU drivers have different compute modes for graphics ( i.e . rendering) and compute ( i.e . neural network inference).

Original: Current GPU drivers have to perform an expensive context switch whenever alternating between these modes.

Original: Thus a system should minimize the number of times the GPU must switch between compute and graphics.

Original: The compute mode CUDA context for process and each CUDA context requires a large amount of memory.

Original: Thus a system should use the minimal number of CUDA contexts possible.

Original: B Skill Deployment

Original: We use the same skills and deployement procedure as Szot et al.

Original: [2021] with two changes to the navigation skill: 1) We have a dedicated stop action for navigation instead of using a t = 0 as stop.

Original: We ﬁnd this to be more robust.

Original: We add a force penalty to training, this makes the navigation skill less likely to bump into things by accident.

Original: However, this also makes it stop immediately if it ever starts in contact with the environment.

Original: During evaluation, we mask its prediction of stop if the target object is more than 2 meters away.

Original: This information is part of its sensor suite, so this does not require any privileged information.

Original: On Set Table, we add an additional stage to the planner that calls navigation again after open cabinet as the open cabinet skill with base movement tends to move away from the cabinet.

Original: C Videos

Original: In the supplement, we include the following videos.

Original: Video 1 (1-tp-srl-no-nav.mp4)

Original: This is an example of TP-SRL(NoNav)+All Base Movement on Tidy House.

Original: The objects to pick are have a white wireframe box drawn around them and at their place locations there is another copy of that object.

Original: The grey sphere on the ground denotes where the navigation skill was trained to navigate to.

Original: This information is not shown to the policy.

Original: The policy makes errors and picks up the wrong object Video 2 (2-tp-srl.mp4)

Original: This is an example of TP-SRL+All Base Movement on Tidy House.

Original: Video 3 (3-pick-*.mp4) Examples of the pick policy on its training task.

Original: Note the difference in spawn distance between this and Video 1.

Original: Video 3 (3-place-*.mp4)

Original: Examples of the place policy on its training task.

Original: D Hyperparamaters

Original: For CPC|A, we use the hyperparameters from Guo et al.

Original: The hyperparameters for our rearrangement skills are in Table A1.

Original: Optimizer

Original: Adam [Kingma and Ba, 2015]

Original: Initial Learning Rate 2 .

Original: 5 × 10 −

Original: Final Learning Rate

Original: [Table]

Original: [Table]

Original: For the benchmark comparisons

Original: [Table]

Original: we set the rollout length T to 128 for methods that require this, that way all methods collect the same amount of experience per rollout.

Original: For SampleFactory, we use a batch size of 1024 on 1 GPU, 2048 on 2, and 4096 on 4 and 8 (this the GPU memory limit).

Original: We do no use a learned entropy coefﬁcient for benchmarking as not all frameworks support this.

Original: We use a ﬁxed value of 10 − 4 as this works well for Open Fridge.

Original: E HTS-RL Comparison

Original: HTS-RL [Liu et al., 2020] uses the sampling method of AsyncOnRL to collect experience for SyncOnRL (the same as No VER and VER ).

Original: It further uses delayed gradients to enable overlapped experience and learning.

Original: Unfortunately the provided implementation 4 has inefﬁciencies that limit its throughput.

Original: To demonstrate this, we add the same style of overlapped experience collection and learning to No VER and compare the provided HTS-RL implementation, HTS-RL (Provided), with our re-implementation,

Original: HTS-RL (Ours).

Original: Our implementation 110% faster than the provided implementation (Table A2).

Original: Key differences in our implementation are fast userspace mutexes (futexes) in shared memory for synchronization (vs. spin locks), pre-allocated pinned memory for CPU to GPU transfers (vs. allocating for each transfer), and GPU shared memory to send experience from inference workers to learn and weights from learner to inference workers (vs. CPU shared memory).

Original: We also ﬁnd that given the efﬁciency of our implementation, there is no signiﬁcant change in system SPS if we remove overlapped experience collection and learning (Table A2, HTS-RL (Ours) vs. No VER ) for Habitat-style tasks.

Original: Removing this reduces GPU memory usage ( ∼ 2 GB in our experiments, this will be larger for larger networks).

Original: We note that overlapped experience collection

Original: [Figure]

Original: i

Original: [Figure]

Original: i

Original: [Figure]

Original: Even with the reduction of sample efﬁciency when increasing the number of GPUs, increasing the the number of GPUs always reduces the time to reach convergence.

Original: and learning does have uses, i.e . for CPU simulation or policies with signiﬁcant CPU components, but it isn’t necessary when both the policy and simulator make heavy use of the GPU.

Original: F Datasets

Original: ReplicaCAD [Szot et al., 2021] – Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Original: This dataset was artist constructed.

Original: YCB dataset [Calli et al., 2015] – Creative Commons Attribution 4.0 International (CC BY 4.0).

Original: This dataset contains 3D scans of generic objects.

Original: There is no PII.

Original: Matterport 3D [Chang et al., 2017] – http://kaldir.vc.in.tum.de/matterport/MP_TOS.pdf .

Original: Consent was given by the owners of the space.

Original: Habitat Matterport Research Dataset [Ramakrishnan et al., 2021] – https://matterport.com/ matterport-end-user-license-agreement-academic-use-model-data .

Original: Consent was given