Original: Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
Revised: Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract: Deep reinforcement learning algorithms require large and diverse datasets in order to learn successful policies for perception-based mobile navi- gation.
Revised: Abstract: Deep reinforcement learning algorithms require large and diverse datasets in order to learn successful policies for perception-based mobile navi- gation.

Original: However, gathering such datasets with a single robot can be prohibitivelyexpensive.
Revised: However, gathering such datasets with a single robot can be prohibitivelyexpensive.

Original: Collecting data with multiple different robotic platforms with possibly different dynamics is a more scalable approach to large-scale data collection.
Revised: Collecting data with multiple different robotic platforms with possibly different dynamics is a more scalable approach to large-scale data collection.

Original: But how can deep reinforcement learning algorithms leverage such heterogeneous datasets?
Revised: But how can deep reinforcement learning algorithms leverage such heterogeneous datasets?

Original: In this work, we propose a deep reinforcement learning algorithm withhierarchically integrated models (HInt).
Revised: In this work, we propose a deep reinforcement learning algorithm withhierarchically integrated models (HInt).

Original: At training time, HInt learns separate perception and dynamics models, and at test time, HInt integrates the two mod- els in a hierarchical manner and plans actions with the integrated model.
Revised: At training time, HInt learns separate perception and dynamics models, and at test time, HInt integrates the two mod- els in a hierarchical manner and plans actions with the integrated model.

Original: This method of planning with hierarchically integrated models allows the algorithm to train on datasets gathered by a variety of different platforms, while respecting the physical capabilities of the deployment robot at test time.
Revised: This method of planning with hierarchically integrated models allows the algorithm to train on datasets gathered by a variety of different platforms, while respecting the physical capabilities of the deployment robot at test time.

Original: Our mobile navigation experiments show that HInt outperforms conventional hierarchical policies andsingle-source approaches.
Revised: Our mobile navigation experiments show that HInt outperforms conventional hierarchical policies andsingle-source approaches.

Original: [Figure]
Revised: [Figure]

Original:  Introduction
Revised:  Introduction

Original: Machine learning provides a powerful tool for enabling robots to acquire control policies by learningdirectly from experience.
Revised: Machine learning provides a powerful tool for enabling robots to acquire control policies by learningdirectly from experience.

Original: One of the key guiding principles behind recent advances in machinelearning is to leverage large datasets.
Revised: One of the key guiding principles behind recent advances in machinelearning is to leverage large datasets.

Original: In previous works in deep reinforcement learning for roboticcontrol, the most common approach is to collect data from a single robot, train a policy in an end-toend fashion and deploy the policy on the same data-collection platform.
Revised: In previous works in deep reinforcement learning for roboticcontrol, the most common approach is to collect data from a single robot, train a policy in an end-toend fashion and deploy the policy on the same data-collection platform.

Original: This approach necessitatescollecting a large and diverse dataset for every model of robot we wish to deploy on, which canpresent a signiﬁcant practical obstacle, since there exists a plethora of different platforms in thereal world.
Revised: This approach necessitatescollecting a large and diverse dataset for every model of robot we wish to deploy on, which canpresent a signiﬁcant practical obstacle, since there exists a plethora of different platforms in thereal world.

Original: What if we could instead leverage datasets collected by a variety of different robots inour training procedure?
Revised: What if we could instead leverage datasets collected by a variety of different robots inour training procedure?

Original: An ideal method could use data from any platform that provides useful knowledge about the problem.
Revised: An ideal method could use data from any platform that provides useful knowledge about the problem.

Original: As an example, it is expensive and time-consuming to gather a largedataset for visual navigation on a drone, due to on-board battery constraints.
Revised: As an example, it is expensive and time-consuming to gather a largedataset for visual navigation on a drone, due to on-board battery constraints.

Original: In comparison, it ismuch easier to collect data on a robot car.
Revised: In comparison, it ismuch easier to collect data on a robot car.

Original: Because knowledge about the visual features of obstaclescan be shared across vision-based mobile robots, making use of visual data collected by a car to traina control policy for a drone could signiﬁcantly reduce the amount of data needed from the drone.
Revised: Because knowledge about the visual features of obstaclescan be shared across vision-based mobile robots, making use of visual data collected by a car to traina control policy for a drone could signiﬁcantly reduce the amount of data needed from the drone.

Original: Unfortunately, data from such heterogeneous platforms presents a challenge: different platformshave different physical capabilities and action spaces.
Revised: Unfortunately, data from such heterogeneous platforms presents a challenge: different platformshave different physical capabilities and action spaces.

Original: In order to leverage such heterogeneous data,we must properly account for the underlying dynamics of the data collection platform.
Revised: In order to leverage such heterogeneous data,we must properly account for the underlying dynamics of the data collection platform.

Original: To learn from multiple sources of data, previous works have utilized hierarchical policies [1, 2, 3].
Revised: To learn from multiple sources of data, previous works have utilized hierarchical policies [1, 2, 3].

Original: In this type of method, a high-level and a low-level policy are trained separately.
Revised: In this type of method, a high-level and a low-level policy are trained separately.

Original: At test time,the actions generated by the high-level policy are used as waypoints for the low-level policy.
Revised: At test time,the actions generated by the high-level policy are used as waypoints for the low-level policy.

Original: Byseparating the policy into two parts, these algorithm are able to utilize data from multiple sourcesin the high-level policy, while representing robot-speciﬁc information such as the dynamics in thelow-level policy.
Revised: Byseparating the policy into two parts, these algorithm are able to utilize data from multiple sourcesin the high-level policy, while representing robot-speciﬁc information such as the dynamics in thelow-level policy.

Original: One drawback of these methods, however, is that the low-level policy is unableto communicate any robot-speciﬁc information to the high-level policy.
Revised: One drawback of these methods, however, is that the low-level policy is unableto communicate any robot-speciﬁc information to the high-level policy.

Original: This makes it possible forthe high-level policy to command waypoints that are impossible for the robot to physically achieve,leading to poor performance and possibly dangerous outcomes.
Revised: This makes it possible forthe high-level policy to command waypoints that are impossible for the robot to physically achieve,leading to poor performance and possibly dangerous outcomes.

Original: The key idea in our approach, illustrated in Fig.
Revised: The key idea in our approach, illustrated in Fig.

Original: 1, is to instead learn hierarchical models , and tointegrate the hierarchical models for planning.
Revised: 1, is to instead learn hierarchical models , and tointegrate the hierarchical models for planning.

Original: At training time, we learn a perception model thatreasons about interactions with the world, using our entire multi-robot dataset, and a dynamicsmodel speciﬁc to the deployment robot, using only data from that robot.
Revised: At training time, we learn a perception model thatreasons about interactions with the world, using our entire multi-robot dataset, and a dynamicsmodel speciﬁc to the deployment robot, using only data from that robot.

Original: At test time, the perceptionand dynamics models are combined to form a single integrated model, which a planner uses tochoose the actions.
Revised: At test time, the perceptionand dynamics models are combined to form a single integrated model, which a planner uses tochoose the actions.

Original: Such hierarchically integrated models can leverage data from multiple robotsfor the perception layer, while also reasoning about the physical capabilities of the deploymentrobot during planning.
Revised: Such hierarchically integrated models can leverage data from multiple robotsfor the perception layer, while also reasoning about the physical capabilities of the deploymentrobot during planning.

Original: This is because when the algorithm uses the hierarchically integrated modelto plan, the dynamics model can “hint” to the perception model about the robot-speciﬁc physicalcapabilities of the deployment robot, and the planner can only select behaviors that the deploymentrobot can actually execute (according to the dynamics model).
Revised: This is because when the algorithm uses the hierarchically integrated modelto plan, the dynamics model can “hint” to the perception model about the robot-speciﬁc physicalcapabilities of the deployment robot, and the planner can only select behaviors that the deploymentrobot can actually execute (according to the dynamics model).

Original: In contrast to hierarchical policies,which may produce waypoint commands that are dynamically infeasible for the deployment robotto achieve, our hierarchically integrated models take the robot’s physical capabilities into account,and only permit physically feasible plans.
Revised: In contrast to hierarchical policies,which may produce waypoint commands that are dynamically infeasible for the deployment robotto achieve, our hierarchically integrated models take the robot’s physical capabilities into account,and only permit physically feasible plans.

Original: The primary contribution of this work is HInt— hierarchically integrated models for acquiringimage-based control policies from heterogeneous datasets.
Revised: The primary contribution of this work is HInt— hierarchically integrated models for acquiringimage-based control policies from heterogeneous datasets.

Original: We demonstrate that HInt successfullylearns policies from heterogeneous datasets in real-world navigation tasks, and outperforms methodsthat use only one data source or use conventional hierarchical policies.
Revised: We demonstrate that HInt successfullylearns policies from heterogeneous datasets in real-world navigation tasks, and outperforms methodsthat use only one data source or use conventional hierarchical policies.

Original:  Related Work
Revised:  Related Work

Original: Prior work demonstrated end-to-end learning for vision-based control for a wide variety of applications, from video games [4, 5] and manipulation [6, 7] to navigation [8, 9].
Revised: Prior work demonstrated end-to-end learning for vision-based control for a wide variety of applications, from video games [4, 5] and manipulation [6, 7] to navigation [8, 9].

Original: However, these approaches typically require a large amount of diverse data [10], which hinders the adoption of thesealgorithms for real-world robot learning.
Revised: However, these approaches typically require a large amount of diverse data [10], which hinders the adoption of thesealgorithms for real-world robot learning.

Original: One approach for overcoming these data constraints is tocombine data from multiple robots; While prior methods have addressed collective learning, theytypically assume that the robots are the same [11], have similar underlying dynamics [12], or thedata is from expert demonstrations [13, 14, 15].
Revised: One approach for overcoming these data constraints is tocombine data from multiple robots; While prior methods have addressed collective learning, theytypically assume that the robots are the same [11], have similar underlying dynamics [12], or thedata is from expert demonstrations [13, 14, 15].

Original: Our approach learns from off-policy data gatheredby robots with a variety of underlying dynamics.
Revised: Our approach learns from off-policy data gatheredby robots with a variety of underlying dynamics.

Original: Prior methods have also transferred skills fromsimulation, where data is more plentiful [16, 17, 18, 19, 20, 21, 22].
Revised: Prior methods have also transferred skills fromsimulation, where data is more plentiful [16, 17, 18, 19, 20, 21, 22].

Original: In contrast, our method doesnot require any simulation, and instead is aimed at leveraging heterogeneous real-world data sources,though it could be extended to incorporate simulated data as well.
Revised: In contrast, our method doesnot require any simulation, and instead is aimed at leveraging heterogeneous real-world data sources,though it could be extended to incorporate simulated data as well.

Original: Prior work has also investigated learning hierarchical vision-based control policies for applicationssuch as autonomous ﬂight [23, 2] and driving [1, 24, 3, 25].
Revised: Prior work has also investigated learning hierarchical vision-based control policies for applicationssuch as autonomous ﬂight [23, 2] and driving [1, 24, 3, 25].

Original: One advantage of these conventionalhierarchy approaches is that many can leverage heterogeneous datasets [26, 12, 23].
Revised: One advantage of these conventionalhierarchy approaches is that many can leverage heterogeneous datasets [26, 12, 23].

Original: However, evenif each module is perfectly accurate , these conventional hierarchy approaches can still fail becausethe low-level policy cannot communicate the robot’s capabilities and limitations to the high-levelpolicy.
Revised: However, evenif each module is perfectly accurate , these conventional hierarchy approaches can still fail becausethe low-level policy cannot communicate the robot’s capabilities and limitations to the high-levelpolicy.

Original: In contrast, because HInt learns hierarchical models, and performs planning on the integratedmodels at test time, HInt is able to jointly reason about the capabilities of the robot and the perceptualfeatures of the environment.
Revised: In contrast, because HInt learns hierarchical models, and performs planning on the integratedmodels at test time, HInt is able to jointly reason about the capabilities of the robot and the perceptualfeatures of the environment.

Original: End-to-end algorithms that can leverage datasets from heterogeneous platforms have also been investigated by prior work; however, these methods typically require on-policy data, are evaluated invisually simplistic domains, or have only been demonstrated in simulation [27, 28, 29].
Revised: End-to-end algorithms that can leverage datasets from heterogeneous platforms have also been investigated by prior work; however, these methods typically require on-policy data, are evaluated invisually simplistic domains, or have only been demonstrated in simulation [27, 28, 29].

Original: In contrast,
Revised: In contrast,

Original: HInt works with real-world off-policy datasets because at the core of HInt are predictive models,which can be trained using standard supervised learning.
Revised: HInt works with real-world off-policy datasets because at the core of HInt are predictive models,which can be trained using standard supervised learning.

Original:  HInt: Hierarchically Integrated Models
Revised:  HInt: Hierarchically Integrated Models

Original: Our goal is to learn image-based mobile navigation policies that can leverage data from heterogeneous robotic platforms.
Revised: Our goal is to learn image-based mobile navigation policies that can leverage data from heterogeneous robotic platforms.

Original: The key contribution of our approach, shown in Fig.
Revised: The key contribution of our approach, shown in Fig.

Original: 2, is to learn separatehierarchical models at training time, and combine these models into a single integrated model forplanning at test time.
Revised: 2, is to learn separatehierarchical models at training time, and combine these models into a single integrated model forplanning at test time.

Original: The two hierarchical models include a high-level, shared perception model,and a low-level, robot-speciﬁc dynamics model.
Revised: The two hierarchical models include a high-level, shared perception model,and a low-level, robot-speciﬁc dynamics model.

Original: The perception model can be trained using datagathered by a variety of different robots, all with possibly different dynamics, while the robotspeciﬁc dynamics model is trained only using data from the deployment robot.
Revised: The perception model can be trained using datagathered by a variety of different robots, all with possibly different dynamics, while the robotspeciﬁc dynamics model is trained only using data from the deployment robot.

Original: At test time, becausethe output predictions of the dynamics model —which are kinematic poses —are the input actionsfor the perception model, the dynamics and perception models can be combined into a single integrated model.
Revised: At test time, becausethe output predictions of the dynamics model —which are kinematic poses —are the input actionsfor the perception model, the dynamics and perception models can be combined into a single integrated model.

Original: The separate hierarchical model training allows our method to leverage datasetsgathered by heterogeneous platforms, while the integrated planning enables our approach to directlymap raw sensory inputs to robot actions that are dynamically feasible for the deployment robot.
Revised: The separate hierarchical model training allows our method to leverage datasetsgathered by heterogeneous platforms, while the integrated planning enables our approach to directlymap raw sensory inputs to robot actions that are dynamically feasible for the deployment robot.

Original: Inthe following sections, we will describe how HInt trains a perception model and a dynamics model,combines these models into a single integrated model in order to perform planning, and concludewith an algorithm summary.
Revised: Inthe following sections, we will describe how HInt trains a perception model and a dynamics model,combines these models into a single integrated model in order to perform planning, and concludewith an algorithm summary.

Original: 3.1 Perception Model
Revised: 3.1 Perception Model

Original: The perception model is a neural network predictive model f PER ✓ ( o t ,  p t : t + H ) !
Revised: The perception model is a neural network predictive model f PER ✓ ( o t ,  p t : t + H ) !

Original: ˆ r t : t + H —shownin Fig. 2— parameterized by model parameters ✓ .
Revised: ˆ r t : t + H —shownin Fig. 2— parameterized by model parameters ✓ .

Original: f
Revised: f

Original: PER ✓ takes as input the current image observation o t and a sequence of H future changes in poses  p t : t
Revised: PER ✓ takes as input the current image observation o t and a sequence of H future changes in poses  p t : t

Original: + H =
Revised: + H =

Original: (  p t ,  p t +1 , ...,  p t + H  1 ) , andpredicts H future rewards ˆ r t : t
Revised: (  p t ,  p t +1 , ...,  p t + H  1 ) , andpredicts H future rewards ˆ r t : t

Original: + H =
Revised: + H =

Original: (ˆ r t , ˆ r t +1 , ..., ˆ r t + H  1 ) .
Revised: (ˆ r t , ˆ r t +1 , ..., ˆ r t + H  1 ) .

Original: The pose p characterizes the kinematicconﬁguration of the robot.
Revised: The pose p characterizes the kinematicconﬁguration of the robot.

Original: In our experiments, we used the robot’s x position, y position, and yaworientation as the pose, though 3D conﬁgurations that include height, roll, and pitch are also possible.
Revised: In our experiments, we used the robot’s x position, y position, and yaworientation as the pose, though 3D conﬁgurations that include height, roll, and pitch are also possible.

Original: This kinematic conﬁguration provides a dynamics-agnostic interface between the perceptionmodel and the dynamics model.
Revised: This kinematic conﬁguration provides a dynamics-agnostic interface between the perceptionmodel and the dynamics model.

Original: The perception model is trained using the heterogeneous dataset
Revised: The perception model is trained using the heterogeneous dataset

Original: D PER by minimizing the objective:
Revised: D PER by minimizing the objective:

Original: [Equation]
Revised: [Equation]

Original: The perception model can be trained with a heterogeneous dataset consisting of data gathered by avariety of robots, all with possibly different underlying dynamics.
Revised: The perception model can be trained with a heterogeneous dataset consisting of data gathered by avariety of robots, all with possibly different underlying dynamics.

Original: The only requirement for the datais that the recorded sensors (e.g., camera) are approximately the same, and that we can calculate thechange in pose—position and orientation—between each sequential datapoint, which could be doneusing visual or mechanical odometry, as well as the reward, which could be collected via onboardsensors or manual labeling.
Revised: The only requirement for the datais that the recorded sensors (e.g., camera) are approximately the same, and that we can calculate thechange in pose—position and orientation—between each sequential datapoint, which could be doneusing visual or mechanical odometry, as well as the reward, which could be collected via onboardsensors or manual labeling.

Original: This ability to train using datasets gathered by heterogeneous platformsis crucial because the perception model is an image-based neural network, which requires largeamounts of data in order to effectively generalize.
Revised: This ability to train using datasets gathered by heterogeneous platformsis crucial because the perception model is an image-based neural network, which requires largeamounts of data in order to effectively generalize.

Original: This model is trained using data gathered by a variety of robots that have the same image observations, but potentially different dynamics.
Revised: This model is trained using data gathered by a variety of robots that have the same image observations, but potentially different dynamics.

Original: When deploying, the dynamics and perception models are combined into a single integrated model that is used for planning and executing actions that maximize reward.
Revised: When deploying, the dynamics and perception models are combined into a single integrated model that is used for planning and executing actions that maximize reward.

Original: 3.2 Dynamics Models
Revised: 3.2 Dynamics Models

Original: [Figure]
Revised: [Figure]

Original: The dynamics model is a neural network predictive model
Revised: The dynamics model is a neural network predictive model

Original: [Equation]
Revised: [Equation]

Original: —shown in
Revised: —shown in

Original: Fig.
Revised: Fig.

Original: by model takes as input the current robot state s t anda sequence of H future actions a t : t
Revised: by model takes as input the current robot state s t anda sequence of H future actions a t : t

Original: + H =
Revised: + H =

Original: ( a t , a t +1 , ..., a t + H  1 ) , and predicts H future changes inposes ˆ  p t :
Revised: ( a t , a t +1 , ..., a t + H  1 ) , and predicts H future changes inposes ˆ  p t :

Original: t + H
Revised: t + H

Original: =
Revised: =

Original: ( ˆ  p t , ˆ  p t +1 , ..., ˆ  p t + H  1 ) .
Revised: ( ˆ  p t , ˆ  p t +1 , ..., ˆ  p t + H  1 ) .

Original: The state s can include any sensor information, suchas the robot’s battery charge or velocity, that may help the model to make more accurate predictionsof future poses.
Revised: The state s can include any sensor information, suchas the robot’s battery charge or velocity, that may help the model to make more accurate predictionsof future poses.

Original: The dynamics model is trained using the respective robot-speciﬁc dataset D DYN byminimizing the objective:
Revised: The dynamics model is trained using the respective robot-speciﬁc dataset D DYN byminimizing the objective:

Original: [Equation]
Revised: [Equation]

Original: Although the dynamics model must only be trained using data collected by the deployment robot,the dynamics model dataset can be signiﬁcantly smaller compared to the perception model dataset,because the dynamics model inputs are lower dimensional by orders of magnitude.
Revised: Although the dynamics model must only be trained using data collected by the deployment robot,the dynamics model dataset can be signiﬁcantly smaller compared to the perception model dataset,because the dynamics model inputs are lower dimensional by orders of magnitude.

Original: Furthermore, thedynamics model dataset does not need a reward signal, allowing for easier data collection.
Revised: Furthermore, thedynamics model dataset does not need a reward signal, allowing for easier data collection.

Original: Though,because perception and dynamics are decoupled in HInt, the dynamics model can accurately modelsystems whose dynamics depend on low-dimensional state information only.
Revised: Though,because perception and dynamics are decoupled in HInt, the dynamics model can accurately modelsystems whose dynamics depend on low-dimensional state information only.

Original: 3.3 Planning and Control
Revised: 3.3 Planning and Control

Original: 136 In order to perform planning and control at test time, we ﬁrst combine the perception and dynamicsmodels into a single integrated model
Revised: 136 In order to perform planning and control at test time, we ﬁrst combine the perception and dynamicsmodels into a single integrated model

Original: [Equation]
Revised: [Equation]

Original: COM
Revised: COM

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: shown in Fig.
Revised: shown in Fig.

Original: These models can be combined without any additional training because the outputof the dynamics model—changes in kinematic poses—is also the input to the perception model.
Revised: These models can be combined without any additional training because the outputof the dynamics model—changes in kinematic poses—is also the input to the perception model.

Original: This integrated model is essential because it enables the planner to holistically reason about theentire system.
Revised: This integrated model is essential because it enables the planner to holistically reason about theentire system.

Original: In contrast, in conventional hierarchical control methods, where the high-level policyoutputs a goal for the low-level policy, the high-level policy could output a dynamically infeasiblereference trajectory for the low-level controller.
Revised: In contrast, in conventional hierarchical control methods, where the high-level policyoutputs a goal for the low-level policy, the high-level policy could output a dynamically infeasiblereference trajectory for the low-level controller.

Original: Our approach would not suffer from this failure casebecause, with integrated planning, the perception model can only take as input dynamically-feasibletrajectories that are output by the dynamics model.
Revised: Our approach would not suffer from this failure casebecause, with integrated planning, the perception model can only take as input dynamically-feasibletrajectories that are output by the dynamics model.

Original: We then plan at each time step for the action sequence that maximizes reward according to theintegrated model by solving the following optimization:
Revised: We then plan at each time step for the action sequence that maximizes reward according to theintegrated model by solving the following optimization:

Original: [Equation]
Revised: [Equation]

Original: Here, R is a user-deﬁned task-speciﬁc reward function, and g t is a user-speciﬁed goal.
Revised: Here, R is a user-deﬁned task-speciﬁc reward function, and g t is a user-speciﬁed goal.

Original: We follow the framework of model predictive control, where at each time step, the robot calculates thebest sequence of actions for the next H steps, executes the ﬁrst action in the sequence, and thenrepeats the process at the next time step.
Revised: We follow the framework of model predictive control, where at each time step, the robot calculates thebest sequence of actions for the next H steps, executes the ﬁrst action in the sequence, and thenrepeats the process at the next time step.

Original: The action sequence that approximately maximizes theobjective in Eqn.
Revised: The action sequence that approximately maximizes theobjective in Eqn.

Original: 3 can be computed using any optimization method.
Revised: 3 can be computed using any optimization method.

Original: In our implementation, we employ stochastic zeroth-order optimization, which selects the best sequence among a set of randomlygenerated trajectories, as is common in model-based RL [30, 31].
Revised: In our implementation, we employ stochastic zeroth-order optimization, which selects the best sequence among a set of randomlygenerated trajectories, as is common in model-based RL [30, 31].

Original: Speciﬁcally, we used either thecross-entropy method (CEM) [32] or MPPI [33], depending on the computational constraints of theplatform.
Revised: Speciﬁcally, we used either thecross-entropy method (CEM) [32] or MPPI [33], depending on the computational constraints of theplatform.

Original: While this approach does result in a limitation for long-horizon reasoning, our main focusis to perform short-horizon navigation.
Revised: While this approach does result in a limitation for long-horizon reasoning, our main focusis to perform short-horizon navigation.

Original: 3.4 Algorithm Summary
Revised: 3.4 Algorithm Summary

Original: 159 We now brieﬂy summarize how our HInt system operates during training and deployment, as shownin Fig.
Revised: 159 We now brieﬂy summarize how our HInt system operates during training and deployment, as shownin Fig.

Original: 1 and Fig.
Revised: 1 and Fig.

Original: During training, we ﬁrst gather perception data using a number of platforms.
Revised: During training, we ﬁrst gather perception data using a number of platforms.

Original: For each of theseplatforms, we save the onboard observations o , change in poses  p , and rewards r into a shareddataset D PER (see §B.1), and use this dataset to train the perception model f
Revised: For each of theseplatforms, we save the onboard observations o , change in poses  p , and rewards r into a shareddataset D PER (see §B.1), and use this dataset to train the perception model f

Original: PER ✓ (Eqn. 1) (see §B.2).
Revised: PER ✓ (Eqn. 1) (see §B.2).

Original: Then, using the robot we will deploy at test time, we gather dynamics data by having the robot act inthe real world and recording the robot’s onboard states s , actions a , and change in poses  p into thedataset D DYN (see §C.1); we use this dataset to train the dynamics model f DYN  (Eqn. 2) (see §C.2).
Revised: Then, using the robot we will deploy at test time, we gather dynamics data by having the robot act inthe real world and recording the robot’s onboard states s , actions a , and change in poses  p into thedataset D DYN (see §C.1); we use this dataset to train the dynamics model f DYN  (Eqn. 2) (see §C.2).

Original: When deploying HInt, we ﬁrst combine the perception model f PER ✓ and dynamics model f DYN  intoa single model f COM ✓ ,  .
Revised: When deploying HInt, we ﬁrst combine the perception model f PER ✓ and dynamics model f DYN  intoa single model f COM ✓ ,  .

Original: The robot then plans a sequence of actions that maximizes reward (Eqn.
Revised: The robot then plans a sequence of actions that maximizes reward (Eqn.

Original: 3),executes the ﬁrst action, and repeat this planning process at each time step until the task is complete, as is standard in model-based RL with model-predictive control [30, 31] (see §D).
Revised: 3),executes the ﬁrst action, and repeat this planning process at each time step until the task is complete, as is standard in model-based RL with model-predictive control [30, 31] (see §D).

Original:  Experiments
Revised:  Experiments

Original: We now present an experimental evaluation of our hierarchically integrated models algorithm in thecontext of real world mobile robot navigation tasks.
Revised: We now present an experimental evaluation of our hierarchically integrated models algorithm in thecontext of real world mobile robot navigation tasks.

Original: Videos and additional details about the datasources, models, training procedures, and planning can be found in the Appendix section or on theproject website: https://sites.google.com/view/multi-robot-drl
Revised: Videos and additional details about the datasources, models, training procedures, and planning can be found in the Appendix section or on theproject website: https://sites.google.com/view/multi-robot-drl

Original: In our experiments, we aim to answer the following questions:
Revised: In our experiments, we aim to answer the following questions:

Original: Q1 : Does leveraging data collected by other robots, in addition to data from the deployment robot,
Revised: Q1 : Does leveraging data collected by other robots, in addition to data from the deployment robot,

Original: improve performance compared to only learning from data collected by the deployment robot?
Revised: improve performance compared to only learning from data collected by the deployment robot?

Original: Q2 : Does HInt’s integrated model planning approach result in better performance compared to
Revised: Q2 : Does HInt’s integrated model planning approach result in better performance compared to

Original: conventional hierarchy approaches?
Revised: conventional hierarchy approaches?

Original: In order to separately examine these questions, we investigate Q1 by training the perception modulewith multiple different real-world data sources, including data from different environments and
Revised: In order to separately examine these questions, we investigate Q1 by training the perception modulewith multiple different real-world data sources, including data from different environments and

Original: dif-
Revised: dif-

Original: ferent platforms, and evaluating on a single real-world robot.
Revised: ferent platforms, and evaluating on a single real-world robot.

Original: To examine Q2 , we deploy a sharedperception module to systems with different low-level dynamics
Revised: To examine Q2 , we deploy a sharedperception module to systems with different low-level dynamics

Original: 4.1 Q1:
Revised: 4.1 Q1:

Original: Comparison to Single
Revised: Comparison to Single

Original: -Source Models 

Revised: -Source Models

Original: In this experiment, we deployed a robot in a number of visually diverse environments, includingones in which that robot itself has not collected any data .
Revised: In this experiment, we deployed a robot in a number of visually diverse environments, includingones in which that robot itself has not collected any data .

Original: Our hypothesis is that HInt, which canmake use of heterogeneous datasets gathered by multiple robots, will outperform methods that canonly train using data gathered by the deployment robot.
Revised: Our hypothesis is that HInt, which canmake use of heterogeneous datasets gathered by multiple robots, will outperform methods that canonly train using data gathered by the deployment robot.

Original: Perception data was collected in three different environments using three different platforms (Fig.
Revised: Perception data was collected in three different environments using three different platforms (Fig.

Original: 3):a Yujin Kobuki robot in an indoor ofﬁce environment (3.7 hours), a Clearpath Jackal in an outdoorurban environment (3.5 hours), and a person recording video with a hand-held camera in an industrial environment (1.2 hours).
Revised: 3):a Yujin Kobuki robot in an indoor ofﬁce environment (3.7 hours), a Clearpath Jackal in an outdoorurban environment (3.5 hours), and a person recording video with a hand-held camera in an industrial environment (1.2 hours).

Original: The deployment robot is the Clearpath Jackal.
Revised: The deployment robot is the Clearpath Jackal.

Original: The same dataset thatprovided the Jackal perception data as described above was also used for the Jackal dynamics data.
Revised: The same dataset thatprovided the Jackal perception data as described above was also used for the Jackal dynamics data.

Original: The robot’s objective is to drive towards a goal location while avoiding collisions and minimizingaction magnitudes.
Revised: The robot’s objective is to drive towards a goal location while avoiding collisions and minimizingaction magnitudes.

Original: More speciﬁcally, the reward r is  1 for a collision and 0 otherwise, the goal gspeciﬁes a GPS location, \ ( ˆ  p t
Revised: More speciﬁcally, the reward r is  1 for a collision and 0 otherwise, the goal gspeciﬁes a GPS location, \ ( ˆ  p t

Original: + h , g t ) denotes the angle in radians formed by ˆ  p t
Revised: + h , g t ) denotes the angle in radians formed by ˆ  p t

Original: + h and g t withrespect to the robot’s current position, and the reward function used for planning is
Revised: + h and g t withrespect to the robot’s current position, and the reward function used for planning is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We compared HInt, which learns from all the data sources, with a variant of HInt which onlylearned from a single data source.
Revised: We compared HInt, which learns from all the data sources, with a variant of HInt which onlylearned from a single data source.

Original: Both approaches use the same training parameters and neural
Revised: Both approaches use the same training parameters and neural

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: s t r i a l network architecture for the integrated model in order to make the most fair comparison, with theonly difference between the data with which they were trained.
Revised: l network architecture for the integrated model in order to make the most fair comparison, with theonly difference between the data with which they were trained.

Original: In our implementation, HInt andthe single data source approach both build upon the method proposed in Kahn et al [34].
Revised: In our implementation, HInt andthe single data source approach both build upon the method proposed in Kahn et al [34].

Original: Kahn etal [34] also uses a vision-based predictive model to perform planning, but differs from HInt in thefollowing ways: (i) only data gathered by the deployment robot can be used for training and (ii) theintegrated perception and dynamics model is trained end-to-end.
Revised: Kahn etal [34] also uses a vision-based predictive model to perform planning, but differs from HInt in thefollowing ways: (i) only data gathered by the deployment robot can be used for training and (ii) theintegrated perception and dynamics model is trained end-to-end.

Original: Fig.
Revised: Fig.

Original: 4 shows the results comparing HInt to the single data source approach.
Revised: 4 shows the results comparing HInt to the single data source approach.

Original: In all environments 1 , ourapproach is more successful in reaching the goal.
Revised: In all environments 1 , ourapproach is more successful in reaching the goal.

Original: Note that even when the single data source methodis trained and deployed in the same environment, HInt still performs better, because learning-basedmethods beneﬁt from large and diverse datasets.
Revised: Note that even when the single data source methodis trained and deployed in the same environment, HInt still performs better, because learning-basedmethods beneﬁt from large and diverse datasets.

Original: Furthermore, the row labeled “Industrial” illustrateswell how HInt can beneﬁt from data collected with other platforms: although the Jackal robot hadnever seen the industrial setting during training, the training set did include data collected by aperson with a video camera in this setting.
Revised: Furthermore, the row labeled “Industrial” illustrateswell how HInt can beneﬁt from data collected with other platforms: although the Jackal robot hadnever seen the industrial setting during training, the training set did include data collected by aperson with a video camera in this setting.

Original: The increase in performance from including this data(“Kobuki + Jackal + Human”) shows that the Jackal robot was able to effectively integrate this data into its visual navigation strategy.
Revised: The increase in performance from including this data(“Kobuki + Jackal + Human”) shows that the Jackal robot was able to effectively integrate this data into its visual navigation strategy.

Original: Fig.
Revised: Fig.

Original: 5 shows ﬁrst-person images of HInt successfully navigatingin both the urban and industrial environments.
Revised: 5 shows ﬁrst-person images of HInt successfully navigatingin both the urban and industrial environments.

Original: 4.2 Q2: Comparison to
Revised: 4.2 Q2: Comparison to

Original: Conventional Hierarchy 

Revised: Conventional Hierarchy

Original: In this experiment, we deployed the perception module onto robots with different low-level dynamics, including ones that were not seen during the collection of the perception data.
Revised: In this experiment, we deployed the perception module onto robots with different low-level dynamics, including ones that were not seen during the collection of the perception data.

Original: Our hypothesis isthat our integrated HInt approach will outperform conventional hierarchy approaches, because it isable to jointly reason about perception and dynamics.
Revised: Our hypothesis isthat our integrated HInt approach will outperform conventional hierarchy approaches, because it isable to jointly reason about perception and dynamics.

Original: We compared our integrated approach with the most commonly used hierarchical approach, in whichthe perception model is used to output desired waypoints that are then passed to a low-level controller [1, 23, 24, 2, 3].
Revised: We compared our integrated approach with the most commonly used hierarchical approach, in whichthe perception model is used to output desired waypoints that are then passed to a low-level controller [1, 23, 24, 2, 3].

Original: In our prediction- and planning-based framework, this modular approach isimplemented by ﬁrst planning over a sequence of positions  p ⇤ t : t + H by using a kinematic visionbased model, and then planning over a sequence of actions a ⇤ t : t + H
Revised: In our prediction- and planning-based framework, this modular approach isimplemented by ﬁrst planning over a sequence of positions  p ⇤ t : t + H by using a kinematic visionbased model, and then planning over a sequence of actions a ⇤ t : t + H

Original: so as to minimize the trackingerror against these planned positions using a robot-speciﬁc dynamics model.
Revised: so as to minimize the trackingerror against these planned positions using a robot-speciﬁc dynamics model.

Original: This baseline represents a clean “apples-to-apples” comparison between our approach – which directly combines boththe dynamic and kinematic models into a single model – and a conventional pipelined approach thatseparates vision-based kinematic planning with low-level trajectory tracking.
Revised: This baseline represents a clean “apples-to-apples” comparison between our approach – which directly combines boththe dynamic and kinematic models into a single model – and a conventional pipelined approach thatseparates vision-based kinematic planning with low-level trajectory tracking.

Original: Both our method andthis baseline employ the same neural network architectures for the vision and dynamics models, andtrain them on the same data, thus providing for a controlled comparison that isolates the questionof whether our end-to-end approach improves over a conventional pipelined hierarchical approach.
Revised: Both our method andthis baseline employ the same neural network architectures for the vision and dynamics models, andtrain them on the same data, thus providing for a controlled comparison that isolates the questionof whether our end-to-end approach improves over a conventional pipelined hierarchical approach.

Original: The planning process for the hierarchical baseline can be expressed as the following two-step optimization:
Revised: The planning process for the hierarchical baseline can be expressed as the following two-step optimization:

Original: [Equation]
Revised: [Equation]

Original: We trained the perception model with three data sources from different robots (“Kobuki + Jackal+ Human” in Fig. 4), one of which is a Clearpath Jackal robot.
Revised: We trained the perception model with three data sources from different robots (“Kobuki + Jackal+ Human” in Fig. 4), one of which is a Clearpath Jackal robot.

Original: We then deployed this perceptionmodel on the Jackal robot.
Revised: We then deployed this perceptionmodel on the Jackal robot.

Original: The robot’s objective is to go towards a goal location while avoidingcollisions, using the same reward function for planning as Eqn. 5.
Revised: The robot’s objective is to go towards a goal location while avoidingcollisions, using the same reward function for planning as Eqn. 5.

Original: We also evaluated our method on a Parrot Bebop drone, shown in Fig.
Revised: We also evaluated our method on a Parrot Bebop drone, shown in Fig.

Original: 3, which was not included inthe data collection process for the perception model.
Revised: 3, which was not included inthe data collection process for the perception model.

Original: Note that this generalization to a new platformwas only possible because the drone’s image observations lies in-distribution within the perceptiontraining data.
Revised: Note that this generalization to a new platformwas only possible because the drone’s image observations lies in-distribution within the perceptiontraining data.

Original: The drone’s objective is to avoid collisions with minimal turning.
Revised: The drone’s objective is to avoid collisions with minimal turning.

Original: The reward functionfor planning is
Revised: The reward functionfor planning is

Original: [Equation]
Revised: [Equation]

Original: The results in Fig.
Revised: The results in Fig.

Original: 6 compare HInt against the conventional hierarchy approach on a Jackal robotwith its normal dynamics and with its steering limited to 40% of its full steering range, as well as ona Bebop drone ﬂying at low speed (8 degrees forward tilt) and high speed (16 degrees forward tilt).
Revised: 6 compare HInt against the conventional hierarchy approach on a Jackal robotwith its normal dynamics and with its steering limited to 40% of its full steering range, as well as ona Bebop drone ﬂying at low speed (8 degrees forward tilt) and high speed (16 degrees forward tilt).

Original: While the two approaches were able to achieve similar performance when evaluated on the normal
Revised: While the two approaches were able to achieve similar performance when evaluated on the normal

Original: Jackal and the low speed drone, our approach outperformed conventional hierarchy on the limitedsteering Jackal and on the high speed drone.
Revised: Jackal and the low speed drone, our approach outperformed conventional hierarchy on the limitedsteering Jackal and on the high speed drone.

Original: The poor performance of the conventional hierarchybaseline in these experiments can be explained by the inability of the conventional higher-levelplanner – which plans kinematic paths based on visual observations – to account for the differentdynamics limitations of each platform.
Revised: The poor performance of the conventional hierarchybaseline in these experiments can be explained by the inability of the conventional higher-levelplanner – which plans kinematic paths based on visual observations – to account for the differentdynamics limitations of each platform.

Original: This can be particularly problematic near obstacles, wherethe kinematic model might decide that a last-minute turn to avoid an obstacle may be feasible, whenin fact the dynamics of the current robot make it impossible.
Revised: This can be particularly problematic near obstacles, wherethe kinematic model might decide that a last-minute turn to avoid an obstacle may be feasible, whenin fact the dynamics of the current robot make it impossible.

Original: This is not an issue for the standard
Revised: This is not an issue for the standard

Original: Jackal robot and the low speed drone, which are both able to make sharp turns.
Revised: Jackal robot and the low speed drone, which are both able to make sharp turns.

Original: However, the limited
Revised: However, the limited

Original: [Figure]
Revised: [Figure]

Original: steering Jackal was unable to physically achieve the sharp turns commanded by the higher-levelmodel in the baseline method.
Revised: steering Jackal was unable to physically achieve the sharp turns commanded by the higher-levelmodel in the baseline method.

Original: Similarly, due to the aerodynamics of high-speed ﬂight, the Bebopdrone was also unable to achieve the sharp turns commanded by the baseline higher-level planner,leading to collision (see, e.g., the lower-right plot in Fig. 6).
Revised: Similarly, due to the aerodynamics of high-speed ﬂight, the Bebopdrone was also unable to achieve the sharp turns commanded by the baseline higher-level planner,leading to collision (see, e.g., the lower-right plot in Fig. 6).

Original: In contrast, in HInt, because the lowlevel model is able to inform the high-level model about the physical capabilities of the robot, theintegrated model could correctly deduce that avoiding the obstacles required starting the turn earlierwhen controlling a less maneuverable robot, which allowed for successful collision avoidance.
Revised: In contrast, in HInt, because the lowlevel model is able to inform the high-level model about the physical capabilities of the robot, theintegrated model could correctly deduce that avoiding the obstacles required starting the turn earlierwhen controlling a less maneuverable robot, which allowed for successful collision avoidance.

Original: We also evaluated HInt and conventional hierarchy on a visual navigation task in simulation involving robot cars with more drastic dynamical differences, such as limited steering, right turn only, and0.25 seconds lag.
Revised: We also evaluated HInt and conventional hierarchy on a visual navigation task in simulation involving robot cars with more drastic dynamical differences, such as limited steering, right turn only, and0.25 seconds lag.

Original: More details can be found in the Appendix (see §E).
Revised: More details can be found in the Appendix (see §E).

Original: In both the real world andsimulation, we showed that conventional hierarchy can fail when deployed on dynamical systemsthat are not a part of the perception dataset.
Revised: In both the real world andsimulation, we showed that conventional hierarchy can fail when deployed on dynamical systemsthat are not a part of the perception dataset.

Original: This is because the higher level perception-based policycan set waypoints for the lower level dynamics-based policy that are outside the physical capabilitiesof the robot.
Revised: This is because the higher level perception-based policycan set waypoints for the lower level dynamics-based policy that are outside the physical capabilitiesof the robot.

Original: In contrast, HInt’s integrated planning approach enables the dynamics model to informthe perception model about which maneuvers are feasible.
Revised: In contrast, HInt’s integrated planning approach enables the dynamics model to informthe perception model about which maneuvers are feasible.

Original:  Discussion
Revised:  Discussion

Original: We presented HInt, a deep reinforcement learning algorithm with hierarchically integrated models.
Revised: We presented HInt, a deep reinforcement learning algorithm with hierarchically integrated models.

Original: The hierarchical training procedure enables the perception model to be trained on heterogeneousdatasets, which is crucial for learning image-based neural network control policies, while the integrated model at test time ensures the perception model only considers trajectories that are dynamically feasible.
Revised: The hierarchical training procedure enables the perception model to be trained on heterogeneousdatasets, which is crucial for learning image-based neural network control policies, while the integrated model at test time ensures the perception model only considers trajectories that are dynamically feasible.

Original: Our experiments show that HInt can outperform both single-source and conventionalhierarchy methods on real-world navigation tasks.
Revised: Our experiments show that HInt can outperform both single-source and conventionalhierarchy methods on real-world navigation tasks.

Original: One of the key algorithmic ideas of HInt is that planning at test time enables us to directly use theintegrated model without any additional training.
Revised: One of the key algorithmic ideas of HInt is that planning at test time enables us to directly use theintegrated model without any additional training.

Original: However, one of the main limitations is that ourapproach can still suffer if the outputs of one model are out-of-distribution for the inputs of thesubsequent model.
Revised: However, one of the main limitations is that ourapproach can still suffer if the outputs of one model are out-of-distribution for the inputs of thesubsequent model.

Original: Bayesian neural networks and other approaches could provide a principled wayto cope with these intermediate out-of-distribution activations.
Revised: Bayesian neural networks and other approaches could provide a principled wayto cope with these intermediate out-of-distribution activations.

Original: We believe that solving these andother challenges is crucial for enabling robot learning platforms to learn from large visual datasetswhile still combining perception and control, and that HInt is a promising step towards this goal.
Revised: We believe that solving these andother challenges is crucial for enabling robot learning platforms to learn from large visual datasetswhile still combining perception and control, and that HInt is a promising step towards this goal.