Original: Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
Revised: Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots

Original: Anonymous Author(s)

Original: AfﬁliationAddress email Abstract: Deep reinforcement learning algorithms require large and diverse datasets in order to learn successful policies for perception-based mobile navi- gation.
Revised: Abstract: Deep reinforcement learning algorithms require large and diverse datasets in order to learn successful policies for perception-based mobile navigation.

Original: However, gathering such datasets with a single robot can be prohibitivelyexpensive.
Revised: However, gathering such datasets with a single robot can be prohibitively expensive.

Original: Collecting data with multiple different robotic platforms with possibly different dynamics is a more scalable approach to large-scale data collection.
Revised: Collecting data with multiple different robotic platforms with possibly different dynamics is a more scalable approach to large-scale data collection.

Original: But how can deep reinforcement learning algorithms leverage such heterogeneous datasets?
Revised: But how can deep reinforcement learning algorithms leverage such heterogeneous datasets?

Original: In this work, we propose a deep reinforcement learning algorithm withhierarchically integrated models (HInt).
Revised: In this work, we propose a deep reinforcement learning algorithm with hierarchically integrated models (HInt).

Original: At training time, HInt learns separate perception and dynamics models, and at test time, HInt integrates the two mod- els in a hierarchical manner and plans actions with the integrated model.
Revised: At training time, HInt learns separate perception and dynamics models, and at test time, HInt integrates the two models in a hierarchical manner and plans actions with the integrated model.

Original: This method of planning with hierarchically integrated models allows the algorithm to train on datasets gathered by a variety of different platforms, while respecting the physical capabilities of the deployment robot at test time.
Revised: This method of planning with hierarchically integrated models allows the algorithm to train on datasets gathered by a variety of different platforms, while respecting the physical capabilities of the deployment robot at test time.

Original: Our mobile navigation experiments show that HInt outperforms conventional hierarchical policies andsingle-source approaches.[Figure]
Revised: Our mobile navigation experiments show that HInt outperforms conventional hierarchical policies and single-source approaches.

Original:  Introduction
Revised:  Introduction

Original: Machine learning provides a powerful tool for enabling robots to acquire control policies by learningdirectly from experience.
Revised: Machine learning provides a powerful tool for enabling robots to acquire control policies by learning directly from experience.

Original: One of the key guiding principles behind recent advances in machinelearning is to leverage large datasets.
Revised: One of the key guiding principles behind recent advances in machine learning is to leverage large datasets.

Original: In previous works in deep reinforcement learning for roboticcontrol, the most common approach is to collect data from a single robot, train a policy in an end-toend fashion and deploy the policy on the same data-collection platform.
Revised: In previous works in deep reinforcement learning for robotic control, the most common approach is to collect data from a single robot, train a policy in an end-toend fashion, and deploy the policy on the same data-collection platform.

Original: This approach necessitatescollecting a large and diverse dataset for every model of robot we wish to deploy on, which canpresent a signiﬁcant practical obstacle, since there exists a plethora of different platforms in thereal world.
Revised: This approach necessitates collecting a large and diverse dataset for every model of robot we wish to deploy on, which can present a signiﬁcant practical obstacle, since there exists a plethora of different platforms in the real world.

Original: What if we could instead leverage datasets collected by a variety of different robots inour training procedure?
Revised: What if we could instead leverage datasets collected by a variety of different robots in our training procedure?

Original: An ideal method could use data from any platform that provides useful knowledge about the problem.
Revised: An ideal method could use data from any platform that provides useful knowledge about the problem.

Original: As an example, it is expensive and time-consuming to gather a largedataset for visual navigation on a drone, due to on-board battery constraints.
Revised: As an example, it is expensive and time-consuming to gather a large dataset for visual navigation on a drone, due to on-board battery constraints.

Original: In comparison, it ismuch easier to collect data on a robot car.
Revised: In comparison, it is much easier to collect data on a robot car.

Original: Because knowledge about the visual features of obstaclescan be shared across vision-based mobile robots, making use of visual data collected by a car to traina control policy for a drone could signiﬁcantly reduce the amount of data needed from the drone.
Revised: Because knowledge about the visual features of obstacles can be shared across vision-based mobile robots, making use of visual data collected by a car to train a control policy for a drone could signiﬁcantly reduce the amount of data needed from the drone.

Original: Unfortunately, data from such heterogeneous platforms presents a challenge: different platformshave different physical capabilities and action spaces.
Revised: Unfortunately, data from such heterogeneous platforms presents a challenge: different platforms have different physical capabilities and action spaces.

Original: In order to leverage such heterogeneous data,we must properly account for the underlying dynamics of the data collection platform.
Revised: In order to leverage such heterogeneous data, we must properly account for the underlying dynamics of the data collection platform.

Original: To learn from multiple sources of data, previous works have utilized hierarchical policies [1, 2, 3].
Revised: To learn from multiple sources of data, previous works have utilized hierarchical policies [1, 2, 3].

Original: In this type of method, a high-level and a low-level policy are trained separately.
Revised: In this type of method, a high-level and a low-level policy are trained separately.

Original: At test time,the actions generated by the high-level policy are used as waypoints for the low-level policy.
Revised: At test time, the actions generated by the high-level policy are used as waypoints for the low-level policy.

Original: Byseparating the policy into two parts, these algorithm are able to utilize data from multiple sourcesin the high-level policy, while representing robot-speciﬁc information such as the dynamics in thelow-level policy.
Revised: By separating the policy into two parts, these algorithm are able to utilize data from multiple sources in the high-level policy, while representing robot-speciﬁc information such as the dynamics in the low-level policy.

Original: One drawback of these methods, however, is that the low-level policy is unableto communicate any robot-speciﬁc information to the high-level policy.
Revised: One drawback of these methods, however, is that the low-level policy is unable to communicate any robot-speciﬁc information to the high-level policy.

Original: This makes it possible forthe high-level policy to command waypoints that are impossible for the robot to physically achieve,leading to poor performance and possibly dangerous outcomes.
Revised: This makes it possible for the high-level policy to command waypoints that are impossible for the robot to physically achieve, leading to poor performance and possibly dangerous outcomes.

Original: The key idea in our approach, illustrated in Fig.
Revised: The key idea in our approach, illustrated in Fig.

Original: 1, is to instead learn hierarchical models , and tointegrate the hierarchical models for planning.
Revised: 1, is to instead learn hierarchical models , and to integrate the hierarchical models for planning.

Original: At training time, we learn a perception model thatreasons about interactions with the world, using our entire multi-robot dataset, and a dynamicsmodel speciﬁc to the deployment robot, using only data from that robot.
Revised: At training time, we learn a perception model that reasons about interactions with the world, using our entire multi-robot dataset, and a dynamics model speciﬁc to the deployment robot, using only data from that robot.

Original: At test time, the perceptionand dynamics models are combined to form a single integrated model, which a planner uses tochoose the actions.
Revised: At test time, the perception and dynamics models are combined to form a single integrated model, which a planner uses to choose the actions.

Original: Such hierarchically integrated models can leverage data from multiple robotsfor the perception layer, while also reasoning about the physical capabilities of the deploymentrobot during planning.
Revised: Such hierarchically integrated models can leverage data from multiple robots for the perception layer, while also reasoning about the physical capabilities of the deployment robot during planning.

Original: This is because when the algorithm uses the hierarchically integrated modelto plan, the dynamics model can “hint” to the perception model about the robot-speciﬁc physicalcapabilities of the deployment robot, and the planner can only select behaviors that the deploymentrobot can actually execute (according to the dynamics model).
Revised: This is because when the algorithm uses the hierarchically integrated model to plan, the dynamics model can “hint” to the perception model about the robot-speciﬁc physical capabilities of the deployment robot, and the planner can only select behaviors that the deployment robot can actually execute (according to the dynamics model).

Original: In contrast to hierarchical policies,which may produce waypoint commands that are dynamically infeasible for the deployment robotto achieve, our hierarchically integrated models take the robot’s physical capabilities into account,and only permit physically feasible plans.
Revised: In contrast to hierarchical policies, which may produce waypoint commands that are dynamically infeasible for the deployment robot to achieve, our hierarchically integrated models take the robot’s physical capabilities into account, and only permit physically feasible plans.

Original: The primary contribution of this work is HInt— hierarchically integrated models for acquiringimage-based control policies from heterogeneous datasets.
Revised: The primary contribution of this work is HInt— hierarchically integrated models for acquiring image-based control policies from heterogeneous datasets.

Original: We demonstrate that HInt successfullylearns policies from heterogeneous datasets in real-world navigation tasks, and outperforms methodsthat use only one data source or use conventional hierarchical policies.
Revised: We demonstrate that HInt successfully learns policies from heterogeneous datasets in real-world navigation tasks, and outperforms methods that use only one data source or use conventional hierarchical policies.

Original:  Related Work
Revised:  Related Work

Original: Prior work demonstrated end-to-end learning for vision-based control for a wide variety of applications, from video games [4, 5] and manipulation [6, 7] to navigation [8, 9].
Revised: Prior work demonstrated end-to-end learning for vision-based control for a wide variety of applications, from video games [4, 5] and manipulation [6, 7] to navigation [8, 9].

Original: However, these approaches typically require a large amount of diverse data [10], which hinders the adoption of thesealgorithms for real-world robot learning.
Revised: However, these approaches typically require a large amount of diverse data [10], which hinders the adoption of these algorithms for real-world robot learning.

Original: One approach for overcoming these data constraints is tocombine data from multiple robots; While prior methods have addressed collective learning, theytypically assume that the robots are the same [11], have similar underlying dynamics [12], or thedata is from expert demonstrations [13, 14, 15].
Revised: One approach for overcoming these data constraints is to combine data from multiple robots; While prior methods have addressed collective learning, they typically assume that the robots are the same [11], have similar underlying dynamics [12], or the data is from expert demonstrations [13, 14, 15].

Original: Our approach learns from off-policy data gatheredby robots with a variety of underlying dynamics.
Revised: Our approach learns from off-policy data gathered by robots with a variety of underlying dynamics.

Original: Prior methods have also transferred skills fromsimulation, where data is more plentiful [16, 17, 18, 19, 20, 21, 22].
Revised: Prior methods have also transferred skills from simulation, where data is more plentiful [16, 17, 18, 19, 20, 21, 22].

Original: In contrast, our method doesnot require any simulation, and instead is aimed at leveraging heterogeneous real-world data sources,though it could be extended to incorporate simulated data as well.
Revised: In contrast, our method does not require any simulation, and instead is aimed at leveraging heterogeneous real-world data sources, though it could be extended to incorporate simulated data as well.

Original: Prior work has also investigated learning hierarchical vision-based control policies for applicationssuch as autonomous ﬂight [23, 2] and driving [1, 24, 3, 25].
Revised: Prior work has also investigated learning hierarchical vision-based control policies for applications such as autonomous ﬂight [23, 2] and driving [1, 24, 3, 25].

Original: One advantage of these conventionalhierarchy approaches is that many can leverage heterogeneous datasets [26, 12, 23].
Revised: One advantage of these conventional hierarchy approaches is that many can leverage heterogeneous datasets [26, 12, 23].

Original: However, evenif each module is perfectly accurate , these conventional hierarchy approaches can still fail becausethe low-level policy cannot communicate the robot’s capabilities and limitations to the high-levelpolicy.
Revised: However, even if each module is perfectly accurate , these conventional hierarchy approaches can still fail because the low-level policy cannot communicate the robot’s capabilities and limitations to the high-level policy.

Original: In contrast, because HInt learns hierarchical models, and performs planning on the integratedmodels at test time, HInt is able to jointly reason about the capabilities of the robot and the perceptualfeatures of the environment.
Revised: In contrast, because HInt learns hierarchical models, and performs planning on the integrated models at test time, HInt is able to jointly reason about the capabilities of the robot and the perceptual features of the environment.

Original: End-to-end algorithms that can leverage datasets from heterogeneous platforms have also been investigated by prior work; however, these methods typically require on-policy data, are evaluated invisually simplistic domains, or have only been demonstrated in simulation [27, 28, 29].
Revised: End-to-end algorithms that can leverage datasets from heterogeneous platforms have also been investigated by prior work; however, these methods typically require on-policy data, are evaluated in visually simplistic domains, or have only been demonstrated in simulation [27, 28, 29].

Original: In contrast,
Revised: In contrast,

Original: HInt works with real-world off-policy datasets because at the core of HInt are predictive models,which can be trained using standard supervised learning.
Revised: HInt works with real-world off-policy datasets because at the core of HInt are predictive models, which can be trained using standard supervised learning.

Original:  HInt: Hierarchically Integrated Models
Revised:  HInt: Hierarchically Integrated Models

Original: Our goal is to learn image-based mobile navigation policies that can leverage data from heterogeneous robotic platforms.
Revised: Our goal is to learn image-based mobile navigation policies that can leverage data from heterogeneous robotic platforms.

Original: The key contribution of our approach, shown in Fig.
Revised: The key contribution of our approach, shown in Fig.

Original: 2, is to learn separatehierarchical models at training time, and combine these models into a single integrated model forplanning at test time.
Revised: 2, is to learn separate hierarchical models at training time, and combine these models into a single integrated model for planning at test time.

Original: The two hierarchical models include a high-level, shared perception model,and a low-level, robot-speciﬁc dynamics model.
Revised: The two hierarchical models include a high-level, shared perception model, and a low-level, robot-speciﬁc dynamics model.

Original: The perception model can be trained using datagathered by a variety of different robots, all with possibly different dynamics, while the robotspeciﬁc dynamics model is trained only using data from the deployment robot.
Revised: The perception model can be trained using data gathered by a variety of different robots, all with possibly different dynamics, while the robotspeciﬁc dynamics model is trained only using data from the deployment robot.

Original: At test time, becausethe output predictions of the dynamics model —which are kinematic poses —are the input actionsfor the perception model, the dynamics and perception models can be combined into a single integrated model.
Revised: At test time, because the output predictions of the dynamics model —which are kinematic poses —are the input actions for the perception model, the dynamics and perception models can be combined into a single integrated model.

Original: The separate hierarchical model training allows our method to leverage datasetsgathered by heterogeneous platforms, while the integrated planning enables our approach to directlymap raw sensory inputs to robot actions that are dynamically feasible for the deployment robot.

Original: Inthe following sections, we will describe how HInt trains a perception model and a dynamics model,combines these models into a single integrated model in order to perform planning, and concludewith an algorithm summary.
Revised: In the following sections, we will describe how HInt trains a perception model and a dynamics model, combines these models into a single integrated model in order to perform planning, and conclude with an algorithm summary.

Original: 3.1 Perception Model
Revised: 3.1 Perception Model

Original: The perception model is a neural network predictive model f PER ✓ ( o t ,  p t : t + H ) !
Revised: The perception model is a neural network predictive model f PER ✓ ( o t ,  p t : t + H ) !

Original: ˆ r t : t + H —shownin Fig. 2— parameterized by model parameters ✓ .
Revised: ˆ r t : t + H —shown in Fig. 2— parameterized by model parameters ✓ .

Original: f
Revised: f

Original: PER ✓ takes as input the current image observation o t and a sequence of H future changes in poses  p t : t
Revised: PER ✓ takes as input the current image observation o t and a sequence of H future changes in poses  p t : t

Original: + H =
Revised: + H =

Original: (  p t ,  p t +1 , ...,  p t + H  1 ) , andpredicts H future rewards ˆ r t : t
Revised: (  p t ,  p t +1 , ...,  p t + H  1 ) , and predicts H future rewards ˆ r t : t

Original: + H =
Revised: + H =

Original: (ˆ r t , ˆ r t +1 , ..., ˆ r t + H  1 ) .
Revised: (ˆ r t , ˆ r t +1 , ..., ˆ r t + H  1 ) .

Original: The pose p characterizes the kinematicconﬁguration of the robot.
Revised: The pose p characterizes the kinematic conﬁguration of the robot.

Original: In our experiments, we used the robot’s x position, y position, and yaworientation as the pose, though 3D conﬁgurations that include height, roll, and pitch are also possible.
Revised: In our experiments, we used the robot’s x position, y position, and yaw orientation as the pose, though 3D conﬁgurations that include height, roll, and pitch are also possible.

Original: This kinematic conﬁguration provides a dynamics-agnostic interface between the perceptionmodel and the dynamics model.
Revised: This kinematic conﬁguration provides a dynamics-agnostic interface between the perception model and the dynamics model.

Original: The perception model is trained using the heterogeneous dataset
Revised: The perception model is trained using the heterogeneous dataset

Original: D PER by minimizing the objective:
Revised: D PER by minimizing the objective:

Original: [Equation]
Revised: [Equation]

Original: The perception model can be trained with a heterogeneous dataset consisting of data gathered by avariety of robots, all with possibly different underlying dynamics.
Revised: The perception model can be trained with a heterogeneous dataset consisting of data gathered by a variety of robots, all with possibly different underlying dynamics.

Original: The only requirement for the datais that the recorded sensors (e.g., camera) are approximately the same, and that we can calculate thechange in pose—position and orientation—between each sequential datapoint, which could be doneusing visual or mechanical odometry, as well as the reward, which could be collected via onboardsensors or manual labeling.
Revised: The only requirement for the data is that the recorded sensors (e.g., camera) are approximately the same, and that we can calculate the change in pose—position and orientation—between each sequential datapoint, which could be done using visual or mechanical odometry, as well as the reward, which could be collected via onboard sensors or manual labeling.

Original: This ability to train using datasets gathered by heterogeneous platformsis crucial because the perception model is an image-based neural network, which requires largeamounts of data in order to effectively generalize.
Revised: This ability to train using datasets gathered by heterogeneous platforms is crucial because the perception model is an image-based neural network, which requires large amounts of data in order to effectively generalize.

Original: This model is trained using data gathered by a variety of robots that have the same image observations, but potentially different dynamics.

Original: When deploying, the dynamics and perception models are combined into a single integrated model that is used for planning and executing actions that maximize reward.

Original: 3.2 Dynamics Models
Revised: 3.2 Dynamics Models

Original: [Figure] The dynamics model is a neural network predictive model [Equation]
Revised: The dynamics model is a neural network predictive model f DYN  ( s t , a t : t + H ) !

Original:  —shown in
Revised: ˆ  p t : t + H —shown in

Original: Fig. 
Revised: Fig. 2— parameterized by model parameters  .

Original: by model takes as input the current robot state s t anda sequence of H future actions a t : t
Revised: f DYN  takes as input the current robot state s t and a sequence of H future actions a t : t

Original: + H =
Revised: + H =

Original: ( a t , a t +1 , ..., a t + H  1 ) , and predicts H future changes inposes ˆ  p t : 
Revised: ( a t , a t +1 , ..., a t + H  1 ) , and predicts H future changes in poses ˆ  p t : t

Original: t + H =
Revised: + H =

Original: ( ˆ  p t , ˆ  p t +1 , ..., ˆ  p t + H  1 ) .
Revised: ( ˆ  p t , ˆ  p t +1 , ..., ˆ  p t + H  1 ) .

Original: The state s can include any sensor information, suchas the robot’s battery charge or velocity, that may help the model to make more accurate predictionsof future poses.
Revised: The state s can include any sensor information, such as the robot’s battery charge or velocity, that may help the model to make more accurate predictions of future poses.

Original: The dynamics model is trained using the respective robot-speciﬁc dataset D DYN byminimizing the objective:
Revised: The dynamics model is trained using the respective robot-speciﬁc dataset D DYN by minimizing the objective:

Original: [Equation]
Revised: [Equation]

Original: Although the dynamics model must only be trained using data collected by the deployment robot,the dynamics model dataset can be signiﬁcantly smaller compared to the perception model dataset,because the dynamics model inputs are lower dimensional by orders of magnitude.
Revised: Although the dynamics model must only be trained using data collected by the deployment robot, the dynamics model dataset can be signiﬁcantly smaller compared to the perception model dataset, because the dynamics model inputs are lower dimensional by orders of magnitude.

Original: Furthermore, thedynamics model dataset does not need a reward signal, allowing for easier data collection.
Revised: Furthermore, the dynamics model dataset does not need a reward signal, allowing for easier data collection.

Original: Though,because perception and dynamics are decoupled in HInt, the dynamics model can accurately modelsystems whose dynamics depend on low-dimensional state information only.
Revised: Though, because perception and dynamics are decoupled in HInt, the dynamics model can accurately model systems whose dynamics depend on low-dimensional state information only.

Original: 3.3 Planning and Control
Revised: 3.3 Planning and Control

Original: 136 In order to perform planning and control at test time, we ﬁrst combine the perception and dynamicsmodels into a single integrated model 
Revised: In order to perform planning and control at test time, we ﬁrst combine the perception and dynamics models into a single integrated model f

Original: [Equation]
Revised: [Equation]

Original: COM
Revised:  ( [Equation]

Original: [Equation]
Revised: [Equation]

Revised: t + H

Original: [Equation]
Revised: [Equation]

Original: shown in Fig.
Revised: shown in Fig.

Original: These models can be combined without any additional training because the outputof the dynamics model—changes in kinematic poses—is also the input to the perception model.
Revised: These models can be combined without any additional training because the output of the dynamics model—changes in kinematic poses—is also the input to the perception model.

Original: This integrated model is essential because it enables the planner to holistically reason about theentire system.
Revised: This integrated model is essential because it enables the planner to holistically reason about the entire system.

Original: In contrast, in conventional hierarchical control methods, where the high-level policyoutputs a goal for the low-level policy, the high-level policy could output a dynamically infeasiblereference trajectory for the low-level controller.
Revised: In contrast, in conventional hierarchical control methods, where the high-level policy outputs a goal for the low-level policy, the high-level policy could output a dynamically infeasible reference trajectory for the low-level controller.

Original: Our approach would not suffer from this failure casebecause, with integrated planning, the perception model can only take as input dynamically-feasibletrajectories that are output by the dynamics model.
Revised: Our approach would not suffer from this failure case because, with integrated planning, the perception model can only take as input dynamically-feasible trajectories that are output by the dynamics model.

Original: We then plan at each time step for the action sequence that maximizes reward according to theintegrated model by solving the following optimization:
Revised: We then plan at each time step for the action sequence that maximizes reward according to the integrated model by solving the following optimization:

Original: [Equation]
Revised: [Equation]

Original: Here, R is a user-deﬁned task-speciﬁc reward function, and g t is a user-speciﬁed goal.
Revised: Here, R is a user-deﬁned task-speciﬁc reward function, and g t is a user-speciﬁed goal.

Original: We follow the framework of model predictive control, where at each time step, the robot calculates thebest sequence of actions for the next H steps, executes the ﬁrst action in the sequence, and thenrepeats the process at the next time step.
Revised: We follow the framework of model predictive control, where at each time step, the robot calculates the best sequence of actions for the next H steps, executes the ﬁrst action in the sequence, and then repeats the process at the next time step.

Original: The action sequence that approximately maximizes theobjective in Eqn.
Revised: The action sequence that approximately maximizes the objective in Eqn.

Original: 3 can be computed using any optimization method.
Revised: 3 can be computed using any optimization method.

Original: In our implementation, we employ stochastic zeroth-order optimization, which selects the best sequence among a set of randomlygenerated trajectories, as is common in model-based RL [30, 31].
Revised: In our implementation, we employ stochastic zeroth-order optimization, which selects the best sequence among a set of randomly generated trajectories, as is common in model-based RL [30, 31].

Original: Speciﬁcally, we used either thecross-entropy method (CEM) [32] or MPPI [33], depending on the computational constraints of theplatform.
Revised: Speciﬁcally, we used either the cross-entropy method (CEM) [32] or MPPI [33], depending on the computational constraints of the platform.

Original: While this approach does result in a limitation for long-horizon reasoning, our main focusis to perform short-horizon navigation.
Revised: While this approach does result in a limitation for long-horizon reasoning, our main focus is to perform short-horizon navigation.

Original: 3.4 Algorithm Summary
Revised: 3.4 Algorithm Summary

Original: 159 We now brieﬂy summarize how our HInt system operates during training and deployment, as shownin Fig.
Revised: We now brieﬂy summarize how our HInt system operates during training and deployment, as shown in Fig.

Original: 1 and Fig.
Revised: 1 and Fig.

Original: During training, we ﬁrst gather perception data using a number of platforms.
Revised: During training, we ﬁrst gather perception data using a number of platforms.

Original: For each of theseplatforms, we save the onboard observations o , change in poses  p , and rewards r into a shareddataset D PER (see §B.1), and use this dataset to train the perception model f 
Revised: For each of these platforms, we save the onboard observations o , change in poses  p , and rewards r into a shared dataset D PER (see §B.1), and use this dataset to train the perception model f PER ✓

Original: PER ✓ (Eqn. 1) (see §B.2).
Revised: [Equation] 1) (see §B.2).

Original: Then, using the robot we will deploy at test time, we gather dynamics data by having the robot act inthe real world and recording the robot’s onboard states s , actions a , and change in poses  p into thedataset D DYN (see §C.1); we use this dataset to train the dynamics model f DYN  (Eqn. 2) (see §C.2).
Revised: Then, using the robot we will deploy at test time, we gather dynamics data by having the robot act in the real world and recording the robot’s onboard states s , actions a , and change in poses  p into the dataset D DYN (see §C.1); we use this dataset to train the dynamics model f DYN  (Eqn. 2) (see §C.2).

Original: When deploying HInt, we ﬁrst combine the perception model f PER ✓ and dynamics model f DYN  intoa single model f COM ✓ ,  .
Revised: When deploying HInt, we ﬁrst combine the perception model f PER ✓ and dynamics model f DYN  into a single model f COM ✓ ,  .

Original: The robot then plans a sequence of actions that maximizes reward (Eqn. 3),executes the ﬁrst action, and repeat this planning process at each time step until the task is complete, as is standard in model-based RL with model-predictive control [30, 31] (see §D).
Revised: The robot then plans a sequence of actions that maximizes reward (Eqn. 3), executes the ﬁrst action, and repeat this planning process at each time step until the task is complete, as is standard in model-based RL with model-predictive control [30, 31] (see §D).

Original:  Experiments
Revised:  Experiments

Original: We now present an experimental evaluation of our hierarchically integrated models algorithm in thecontext of real world mobile robot navigation tasks.
Revised: We now present an experimental evaluation of our hierarchically integrated models algorithm in the context of real world mobile robot navigation tasks.

Original: Videos and additional details about the datasources, models, training procedures, and planning can be found in the Appendix section or on theproject website: https://sites.google.com/view/multi-robot-drl
Revised: Videos and additional details about the data sources, models, training procedures, and planning can be found in the Appendix section or on the project website: https://sites.google.com/berkeley.edu/hint-public

Original: In our experiments, we aim to answer the following questions:
Revised: In our experiments, we aim to answer the following questions:

Original: Q1 : Does leveraging data collected by other robots, in addition to data fromthe deployment robot,
Revised: Q1 : Does leveraging data collected by other robots, in addition to data from

Original:  improve performance compared to only learning from data collected by the deployment robot?
Revised: the deployment robot, improve performance compared to only learning from data collected by the deployment robot?

Original: Q2 : Does HInt’s integrated model planning approach result in better performance compared to conventional hierarchy approaches?
Revised: Q2 : Does HInt’s integrated model planning approach result in better performance compared to conventional hierarchy approaches?

Original: In order to separately examine these questions, we investigate Q1 by training the perception modulewith multiple different real-world data sources, including data from different environments and dif- ferent platforms, and evaluating on a single real-world robot.
Revised: In order to separately examine these questions, we investigate Q1 by training the perception module with multiple different real-world data sources, including data from different environments and different platforms, and evaluating on a single real-world robot.

Original: To examine Q2 , we deploy a sharedperception module to systems with different low-level dynamics
Revised: To examine Q2 , we deploy a shared perception module to systems with different low-level dynamics.

Original: 4.1 Q1: Comparison to Single -Source Models[-
-]
Revised: 4.1 Q1: Comparison to Single-Source Models[-
-]

Original: In this experiment, we deployed a robot in a number of visually diverse environments, includingones in which that robot itself has not collected any data .
Revised: In this experiment, we deployed a robot in a number of visually diverse environments, including ones in which that robot itself has not collected any data .

Original: Our hypothesis is that HInt, which canmake use of heterogeneous datasets gathered by multiple robots, will outperform methods that canonly train using data gathered by the deployment robot.
Revised: Our hypothesis is that HInt, which can make use of heterogeneous datasets gathered by multiple robots, will outperform methods that can only train using data gathered by the deployment robot.

Original: Perception data was collected in three different environments using three different platforms (Fig. 3):a Yujin Kobuki robot in an indoor ofﬁce environment (3.7 hours), a Clearpath Jackal in an outdoorurban environment (3.5 hours), and a person recording video with a hand-held camera in an industrial environment (1.2 hours).
Revised: Perception data was collected in three different environments using three different platforms (Fig. 3): a Yujin Kobuki robot in an indoor ofﬁce environment (3.7 hours), a Clearpath Jackal in an outdoor urban environment (3.5 hours), and a person recording video with a hand-held camera in an industrial environment (1.2 hours).

Original: The deployment robot is the Clearpath Jackal.
Revised: The deployment robot is the Clearpath Jackal.

Original: The same dataset thatprovided the Jackal perception data as described above was also used for the Jackal dynamics data.
Revised: The same dataset that provided the Jackal perception data as described above was also used for the Jackal dynamics data.

Original: The robot’s objective is to drive towards a goal location while avoiding collisions and minimizingaction magnitudes.
Revised: The robot’s objective is to drive towards a goal location while avoiding collisions and minimizing action magnitudes.

Original: More speciﬁcally, the reward r is  1 for a collision and 0 otherwise, the goal gspeciﬁes a GPS location, \ ( ˆ  p t
Revised: More speciﬁcally, the reward r is  1 for a collision and 0 otherwise, the goal g speciﬁes a GPS location, \ ( ˆ  p t

Original: + h , g t ) denotes the angle in radians formed by ˆ  p t
Revised: + h , g t ) denotes the angle in radians formed by ˆ  p t

Original: + h and g t withrespect to the robot’s current position, and the reward function used for planning is
Revised: + h and g t with respect to the robot’s current position, and the reward function used for planning is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We compared HInt, which learns from all the data sources, with a variant of HInt which onlylearned from a single data source.
Revised: We compared HInt, which learns from all the data sources, with a variant of HInt which only learned from a single data source.

Original: Both approaches use the same training parameters and neural[Figure] [Figure] l network architecture for the integrated model in order to make the most fair comparison, with theonly difference between the data with which they were trained.
Revised: Both approaches use the same training parameters and neural network architecture for the integrated model in order to make the most fair comparison, with the only difference between the data with which they were trained.

Original: In our implementation, HInt andthe single data source approach both build upon the method proposed in Kahn et al [34].
Revised: In our implementation, HInt and the single data source approach both build upon the method proposed in Kahn et al [34].

Original: Kahn etal [34] also uses a vision-based predictive model to perform planning, but differs from HInt in thefollowing ways: (i) only data gathered by the deployment robot can be used for training and (ii) theintegrated perception and dynamics model is trained end-to-end. 
Revised: Kahn et al [34] also uses a vision-based predictive model to perform planning, but differs from HInt in the following ways: (i) only data gathered by the deployment robot can be used for training and (ii) the integrated perception and dynamics model is trained end-to-end. [Figure] [Figure] I ndu s t

Revised: r i a l

Original: Fig.
Revised: Fig.

Original: 4 shows the results comparing HInt to the single data source approach.
Revised: 4 shows the results comparing HInt to the single data source approach.

Original: In all environments 1 , ourapproach is more successful in reaching the goal.
Revised: In all environments 1 , our approach is more successful in reaching the goal.

Original: Note that even when the single data source methodis trained and deployed in the same environment, HInt still performs better, because learning-basedmethods beneﬁt from large and diverse datasets.
Revised: Note that even when the single data source method is trained and deployed in the same environment, HInt still performs better, because learning-based methods beneﬁt from large and diverse datasets.

Original: Furthermore, the row labeled “Industrial” illustrateswell how HInt can beneﬁt from data collected with other platforms: although the Jackal robot hadnever seen the industrial setting during training, the training set did include data collected by aperson with a video camera in this setting.
Revised: Furthermore, the row labeled “Industrial” illustrates well how HInt can beneﬁt from data collected with other platforms: although the Jackal robot had never seen the industrial setting during training, the training set did include data collected by a person with a video camera in this setting.

Original: The increase in performance from including this data(“Kobuki + Jackal + Human”) shows that the Jackal robot was able to effectively integrate this data into its visual navigation strategy.
Revised: The increase in performance from including this data (“Kobuki + Jackal + Human”) shows that the Jackal robot was able to effectively integrate this data into its visual navigation strategy.

Original: Fig.
Revised: Fig.

Original: 5 shows ﬁrst-person images of HInt successfully navigatingin both the urban and industrial environments.
Revised: 5 shows ﬁrst-person images of HInt successfully navigating in both the urban and industrial environments.

Original: 4.2 Q2: Comparison to 
Revised: 4.2 Q2: Comparison to Conventional Hierarchy

Original: [-Conventional Hierarchy 
-]
Revised: [-Conventional Hierarchy 
-]

Original: In this experiment, we deployed the perception module onto robots with different low-level dynamics, including ones that were not seen during the collection of the perception data.
Revised: In this experiment, we deployed the perception module onto robots with different low-level dynamics, including ones that were not seen during the collection of the perception data.

Original: Our hypothesis isthat our integrated HInt approach will outperform conventional hierarchy approaches, because it isable to jointly reason about perception and dynamics.
Revised: Our hypothesis is that our integrated HInt approach will outperform conventional hierarchy approaches, because it is able to jointly reason about perception and dynamics.

Original: We compared our integrated approach with the most commonly used hierarchical approach, in whichthe perception model is used to output desired waypoints that are then passed to a low-level controller [1, 23, 24, 2, 3].
Revised: We compared our integrated approach with the most commonly used hierarchical approach, in which the perception model is used to output desired waypoints that are then passed to a low-level controller [1, 23, 24, 2, 3].

Original: In our prediction- and planning-based framework, this modular approach isimplemented by ﬁrst planning over a sequence of positions  p ⇤ t : t + H by using a kinematic visionbased model, and then planning over a sequence of actions a ⇤ t : t + H
Revised: In our prediction- and planning-based framework, this modular approach is implemented by ﬁrst planning over a sequence of positions  p ⇤ t : t + H by using a kinematic visionbased model, and then planning over a sequence of actions a ⇤ t : t + H

Original: so as to minimize the trackingerror against these planned positions using a robot-speciﬁc dynamics model.
Revised: so as to minimize the tracking error against these planned positions using a robot-speciﬁc dynamics model.

Original: This baseline represents a clean “apples-to-apples” comparison between our approach – which directly combines boththe dynamic and kinematic models into a single model – and a conventional pipelined approach thatseparates vision-based kinematic planning with low-level trajectory tracking.
Revised: This baseline represents a clean “apples-to-apples” comparison between our approach – which directly combines both the dynamic and kinematic models into a single model – and a conventional pipelined approach that separates vision-based kinematic planning with low-level trajectory tracking.

Original: Both our method andthis baseline employ the same neural network architectures for the vision and dynamics models, andtrain them on the same data, thus providing for a controlled comparison that isolates the questionof whether our end-to-end approach improves over a conventional pipelined hierarchical approach.
Revised: Both our method and this baseline employ the same neural network architectures for the vision and dynamics models, and train them on the same data, thus providing for a controlled comparison that isolates the question of whether our end-to-end approach improves over a conventional pipelined hierarchical approach.

Original: The planning process for the hierarchical baseline can be expressed as the following two-step optimization:
Revised: The planning process for the hierarchical baseline can be expressed as the following two-step optimization:

Original: [Equation]
Revised: [Equation]

Original: We trained the perception model with three data sources from different robots (“Kobuki + Jackal+ Human” in Fig. 4), one of which is a Clearpath Jackal robot.
Revised: We trained the perception model with three data sources from different robots (“Kobuki + Jackal + Human” in Fig. 4), one of which is a Clearpath Jackal robot.

Original: We then deployed this perceptionmodel on the Jackal robot.
Revised: We then deployed this perception model on the Jackal robot.

Original: The robot’s objective is to go towards a goal location while avoidingcollisions, using the same reward function for planning as Eqn. 5.
Revised: The robot’s objective is to go towards a goal location while avoiding collisions, using the same reward function for planning as Eqn. 5.

Original: We also evaluated our method on a Parrot Bebop drone, shown in Fig.
Revised: We also evaluated our method on a Parrot Bebop drone, shown in Fig.

Original: 3, which was not included inthe data collection process for the perception model.
Revised: 3, which was not included in the data collection process for the perception model.

Original: Note that this generalization to a new platformwas only possible because the drone’s image observations lies in-distribution within the perceptiontraining data.
Revised: Note that this generalization to a new platform was only possible because the drone’s image observations lies in-distribution within the perception training data.

Original: The drone’s objective is to avoid collisions with minimal turning.
Revised: The drone’s objective is to avoid collisions with minimal turning.

Original: The reward functionfor planning is
Revised: The reward function for planning is

Original: [Equation]
Revised: [Equation]

Original: The results in Fig.
Revised: The results in Fig.

Original: 6 compare HInt against the conventional hierarchy approach on a Jackal robotwith its normal dynamics and with its steering limited to 40% of its full steering range, as well as ona Bebop drone ﬂying at low speed (8 degrees forward tilt) and high speed (16 degrees forward tilt).
Revised: 6 compare HInt against the conventional hierarchy approach on a Jackal robot with its normal dynamics and with its steering limited to 40% of its full steering range, as well as on a Bebop drone ﬂying at low speed (8 degrees forward tilt) and high speed (16 degrees forward tilt).

Original: While the two approaches were able to achieve similar performance when evaluated on the normal
Revised: While the two approaches were able to achieve similar performance when evaluated on the normal

Original: Jackal and the low speed drone, our approach outperformed conventional hierarchy on the limitedsteering Jackal and on the high speed drone.
Revised: Jackal and the low speed drone, our approach outperformed conventional hierarchy on the limited steering Jackal and on the high speed drone.

Original: The poor performance of the conventional hierarchybaseline in these experiments can be explained by the inability of the conventional higher-levelplanner – which plans kinematic paths based on visual observations – to account for the differentdynamics limitations of each platform.
Revised: The poor performance of the conventional hierarchy baseline in these experiments can be explained by the inability of the conventional higher-level planner – which plans kinematic paths based on visual observations – to account for the different dynamics limitations of each platform.

Original: This can be particularly problematic near obstacles, wherethe kinematic model might decide that a last-minute turn to avoid an obstacle may be feasible, whenin fact the dynamics of the current robot make it impossible.
Revised: This can be particularly problematic near obstacles, where the kinematic model might decide that a last-minute turn to avoid an obstacle may be feasible, when in fact the dynamics of the current robot make it impossible.

Original: This is not an issue for the standard
Revised: This is not an issue for the standard

Original: Jackal robot and the low speed drone, which are both able to make sharp turns.
Revised: Jackal robot and the low speed drone, which are both able to make sharp turns.

Original: However, the limited[Figure] steering Jackal was unable to physically achieve the sharp turns commanded by the higher-levelmodel in the baseline method.
Revised: However, the limited steering Jackal was unable to physically achieve the sharp turns commanded by the higher-level model in the baseline method.

Original: Similarly, due to the aerodynamics of high-speed ﬂight, the Bebopdrone was also unable to achieve the sharp turns commanded by the baseline higher-level planner,leading to collision (see, e.g., the lower-right plot in Fig. 6).
Revised: Similarly, due to the aerodynamics of high-speed ﬂight, the Bebop drone was also unable to achieve the sharp turns commanded by the baseline higher-level planner, leading to collision (see, e.g., the lower-right plot in Fig. 6).

Original: In contrast, in HInt, because the lowlevel model is able to inform the high-level model about the physical capabilities of the robot, theintegrated model could correctly deduce that avoiding the obstacles required starting the turn earlierwhen controlling a less maneuverable robot, which allowed for successful collision avoidance.
Revised: In contrast, in HInt, because the lowlevel model is able to inform the high-level model about the physical capabilities of the robot, the integrated model could correctly deduce that avoiding the obstacles required starting the turn earlier when controlling a less maneuverable robot, which allowed for successful collision avoidance.

Revised: [Figure]

Original: We also evaluated HInt and conventional hierarchy on a visual navigation task in simulation involving robot cars with more drastic dynamical differences, such as limited steering, right turn only, and0.25 seconds lag.
Revised: We also evaluated HInt and conventional hierarchy on a visual navigation task in simulation involving robot cars with more drastic dynamical differences, such as limited steering, right turn only, and 0.25 seconds lag.

Original: More details can be found in the Appendix (see §E).
Revised: More details can be found in the Appendix (see §E).

Original: In both the real world andsimulation, we showed that conventional hierarchy can fail when deployed on dynamical systemsthat are not a part of the perception dataset.
Revised: In both the real world and simulation, we showed that conventional hierarchy can fail when deployed on dynamical systems that are not a part of the perception dataset.

Original: This is because the higher level perception-based policycan set waypoints for the lower level dynamics-based policy that are outside the physical capabilitiesof the robot.
Revised: This is because the higher level perception-based policy can set waypoints for the lower level dynamics-based policy that are outside the physical capabilities of the robot.

Original: In contrast, HInt’s integrated planning approach enables the dynamics model to informthe perception model about which maneuvers are feasible.
Revised: In contrast, HInt’s integrated planning approach enables the dynamics model to inform the perception model about which maneuvers are feasible.

Original:  Discussion
Revised:  Discussion

Original: We presented HInt, a deep reinforcement learning algorithm with hierarchically integrated models.
Revised: We presented HInt, a deep reinforcement learning algorithm with hierarchically integrated models.

Original: The hierarchical training procedure enables the perception model to be trained on heterogeneousdatasets, which is crucial for learning image-based neural network control policies, while the integrated model at test time ensures the perception model only considers trajectories that are dynamically feasible.
Revised: The hierarchical training procedure enables the perception model to be trained on heterogeneous datasets, which is crucial for learning image-based neural network control policies, while the integrated model at test time ensures the perception model only considers trajectories that are dynamically feasible.

Original: Our experiments show that HInt can outperform both single-source and conventionalhierarchy methods on real-world navigation tasks.
Revised: Our experiments show that HInt can outperform both single-source and conventional hierarchy methods on real-world navigation tasks.

Original: One of the key algorithmic ideas of HInt is that planning at test time enables us to directly use theintegrated model without any additional training.
Revised: One of the key algorithmic ideas of HInt is that sharing perception data from heterogeneous platforms could improve generalization and reduce the amount of training data needed from the deployment robot. As with all machine learning methods, our approach generalizes to situations that are within the distribution of the training data. So if training data comes from a variety of ground robots with similar cameras, we expect good generalization to other ground robots with similar cameras, but not, for example, to walking robots. We conjecture that this problem can be solved by learning from a bigger and more diverse dataset.

Original: However, one of  the main limitations is thatourapproach can still suffer if the outputs of one model are out-of-distribution for the inputs of thesubsequent model.
Revised: An exciting direction for future work could be to scale up our experimental evaluation to a much larger number and diversity of platforms (e.g. training the perception model with data from 1000 different cars, drones, and legged robots). Such an approach could lead to a general and robust perception system that can be directly deployed on a wide range of mobile robots without needing to collect additional perception data from the deployment robot. We believe that enabling robot platforms to learn from large and diverse datasets is essential for the success of robot learning, and that HInt is a promising step towards this goal. Acknowledgments

Original: Bayesian neural networks and other approaches could provide a principled wayto cope with these intermediate out-of-distribution activations.
Revised: We thank Anusha Nagabandi and Simin Liu for insightful discussions, and anonymous reviewers at RAIL for feedback on early drafts on the paper.

Original: We believe that solving these andother challenges is crucial for enabling robot learning platforms to learn from large visual datasetswhile still combining perception and control, and that HInt is a promising step towards this goal.
Revised: This research was supported by DARPA Assured Autonomy and ARL DCIST CRA W911NF-17-2-0181. KK and GK are supported by the NSF GRFP.