Original: L EARNING E XPLORATION P OLICIES FOR N AVIGATION
Revised: L EARNING E XPLORATION P OLICIES FOR N AVIGATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Numerous past works have tackled the problem of task-driven navigation.
Revised: Numerous past works have tackled the problem of task-driven navigation.

Original: But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention.
Revised: But, how to effectively explore a new environment to enable a variety of down-stream tasks has received much less attention.

Original: In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards.
Revised: In this work, we study how agents can autonomously explore realistic and complex 3D environments without the context of task-rewards.

Original: We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms.
Revised: We propose a learning-based approach and investigate different policy architectures, reward functions, and training paradigms.

Original: We ﬁnd that use of policies with spatial memory that are bootstrapped with imitation learning and ﬁnally ﬁnetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments.
Revised: We ﬁnd that use of policies with spatial memory that are bootstrapped with imitation learning and ﬁnally ﬁnetuned with coverage rewards derived purely from on-board sensors can be effective at exploring novel environments.

Original: We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques.
Revised: We show that our learned exploration policies can explore better than classical approaches based on geometry alone and generic learning-based exploration techniques.

Original: Finally, we also show how such task-agnostic exploration can be used for down-stream tasks.
Revised: Finally, we also show how such task-agnostic exploration can be used for down-stream tasks.

Original: Videos are available at: https://sites.google.com/view/exploration-for-nav/.
Revised: Videos are available at: https://sites.google.com/view/exploration-for-nav/.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Imagine your ﬁrst day at a new workplace.
Revised: Imagine your ﬁrst day at a new workplace.

Original: If you are like most people, the ﬁrst task you set for yourself is to become familiar with the ofﬁce so that the next day when you have to attend meetings and perform tasks, you can navigate efﬁciently and seamlessly.
Revised: If you are like most people, the ﬁrst task you set for yourself is to become familiar with the ofﬁce so that the next day when you have to attend meetings and perform tasks, you can navigate efﬁciently and seamlessly.

Original: To achieve that goal, you explore your ofﬁce without the task context of target locations you have to reach and build a generic understanding of space.
Revised: To achieve that goal, you explore your ofﬁce without the task context of target locations you have to reach and build a generic understanding of space.

Original: This step of task-independent exploration is quite critical yet often ignored in current approaches for navigation.
Revised: This step of task-independent exploration is quite critical yet often ignored in current approaches for navigation.

Original: When it comes to navigation, currently there are two paradigms: (a) geometric reconstruction and path-planning based approaches (Hartley & Zisserman, 2003; Thrun et al., 2005; LaValle, 2006), and (b) learning-based approaches (Mirowski et al., 2017; Gupta et al., 2017; Savinov et al., 2018; Zhu et al., 2017).
Revised: When it comes to navigation, currently there are two paradigms: (a) geometric reconstruction and path-planning based approaches (Hartley & Zisserman, 2003; Thrun et al., 2005; LaValle, 2006), and (b) learning-based approaches (Mirowski et al., 2017; Gupta et al., 2017; Savinov et al., 2018; Zhu et al., 2017).

Original: SLAM-based approaches, ﬁrst build a map and then use localization and path planning for navigation.
Revised: SLAM-based approaches, ﬁrst build a map and then use localization and path planning for navigation.

Original: In doing this, one interesting question that is often overlooked is: How does one build a map?
Revised: In doing this, one interesting question that is often overlooked is: How does one build a map?

Original: How should we explore the environment to build this map?
Revised: How should we explore the environment to build this map?

Original: Current approaches either use a human operator to control the robot for building the map ( e.g. Thrun et al.
Revised: Current approaches either use a human operator to control the robot for building the map ( e.g. Thrun et al.

Original: or use heuristics such as frontier-based exploration (Yamauchi, 1997).
Revised: or use heuristics such as frontier-based exploration (Yamauchi, 1997).

Original: On the other hand for approaches that use learning, most only learn policies for speciﬁc tasks (Zhu et al., 2017; Mirowski et al., 2017; Gupta et al., 2017), or assume environments have already been explored (Savinov et al., 2018).
Revised: On the other hand for approaches that use learning, most only learn policies for speciﬁc tasks (Zhu et al., 2017; Mirowski et al., 2017; Gupta et al., 2017), or assume environments have already been explored (Savinov et al., 2018).

Original: Moreover, in context of such learning based approaches, the question of exploration is not only important at test time, but also at train time.
Revised: Moreover, in context of such learning based approaches, the question of exploration is not only important at test time, but also at train time.

Original: And once again, this question is largely ignored.
Revised: And once again, this question is largely ignored.

Original: Current approaches either use sample inefﬁcient random exploration or make impractical assumptions about full map availability for generating supervision from optimal trajectories.
Revised: Current approaches either use sample inefﬁcient random exploration or make impractical assumptions about full map availability for generating supervision from optimal trajectories.

Original: It is clear that a big bottleneck for both these navigation paradigms is an exploration policy: a taskindependent policy that explores the environment to either build a map or sample trajectories for learning a navigation policy in a sample-efﬁcient manner.
Revised: It is clear that a big bottleneck for both these navigation paradigms is an exploration policy: a taskindependent policy that explores the environment to either build a map or sample trajectories for learning a navigation policy in a sample-efﬁcient manner.

Original: But how do we learn this task-independent policy?
Revised: But how do we learn this task-independent policy?

Original: What should be the reward for such a policy?
Revised: What should be the reward for such a policy?

Original: First possible way is to not use learning and use heuristic based approaches (Yamauchi, 1997).
Revised: First possible way is to not use learning and use heuristic based approaches (Yamauchi, 1997).

Original: However, there are four issues with non-learning based approaches: (a) these approaches are brittle and fail when there is noise in ego-estimation or localization; (b) they make strong assumptions about free-space/collisions and fail to generalize when navigation requires interactions such as opening doors etc.
Revised: However, there are four issues with non-learning based approaches: (a) these approaches are brittle and fail when there is noise in ego-estimation or localization; (b) they make strong assumptions about free-space/collisions and fail to generalize when navigation requires interactions such as opening doors etc.

Original: ; (c) they fail to capture semantic priors that can reduce search-space signiﬁcantly; and (d) they heavily rely on specialized sensors such as range scanners.
Revised: ; (c) they fail to capture semantic priors that can reduce search-space signiﬁcantly; and (d) they heavily rely on specialized sensors such as range scanners.

Original: Another possibility is to learn exploration policies on training environments.
Revised: Another possibility is to learn exploration policies on training environments.

Original: One way is to use reinforcement learning (RL) with intrinsic rewards.
Revised: One way is to use reinforcement learning (RL) with intrinsic rewards.

Original: Examples of intrinsic rewards can be “curiosity” where prediction error is used as reward signal or “diversity rewards” which discourages the agent from revisiting the same states.
Revised: Examples of intrinsic rewards can be “curiosity” where prediction error is used as reward signal or “diversity rewards” which discourages the agent from revisiting the same states.

Original: While this seems like an effective reward, such approaches are still sample inefﬁcient due to blackbox reward functions that cannot be differentiated to compute gradients effectively.
Revised: While this seems like an effective reward, such approaches are still sample inefﬁcient due to blackbox reward functions that cannot be differentiated to compute gradients effectively.

Original: So, what would be an effective way to learn exploration policies for navigation?
Revised: So, what would be an effective way to learn exploration policies for navigation?

Original: In this paper, we propose an approach for learning policies for exploration for navigation.
Revised: In this paper, we propose an approach for learning policies for exploration for navigation.

Original: We explore this problem from multiple perspectives: (a) architectural design, (b) reward function design, and (c) reward optimization.
Revised: We explore this problem from multiple perspectives: (a) architectural design, (b) reward function design, and (c) reward optimization.

Original: Speciﬁcally, from perspective of reward function and optimization, we take the alternative paradigm and use supervision from human explorations in conjunction with intrinsic rewards.
Revised: Speciﬁcally, from perspective of reward function and optimization, we take the alternative paradigm and use supervision from human explorations in conjunction with intrinsic rewards.

Original: We notice that bootstrapping of learning from small amount of human supervision aids learning of semantics ( e.g. doors are pathways).
Revised: We notice that bootstrapping of learning from small amount of human supervision aids learning of semantics ( e.g. doors are pathways).

Original: It also provides a good initialization for learning using intrinsic rewards.
Revised: It also provides a good initialization for learning using intrinsic rewards.

Original: From the perspective of architecture design, we explore how to use 3D information and use it efﬁciently while doing exploration.
Revised: From the perspective of architecture design, we explore how to use 3D information and use it efﬁciently while doing exploration.

Original: Finally, we study proxy rewards that characterize coverage and demonstrate how this reward outperforms other rewards such as curiosity.
Revised: Finally, we study proxy rewards that characterize coverage and demonstrate how this reward outperforms other rewards such as curiosity.

Original: Finally, we show how experience gathered from our learned exploration policies improves performance at down-stream navigation tasks.
Revised: Finally, we show how experience gathered from our learned exploration policies improves performance at down-stream navigation tasks.

Original: Code will be made available upon acceptance.
Revised: Code will be made available upon acceptance.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work on learning exploration policies for navigation in real world scenes is related to active
Revised: Our work on learning exploration policies for navigation in real world scenes is related to active

Original: SLAM in classical robotics, and intrinsic rewards based exploration in reinforcement learning.
Revised: SLAM in classical robotics, and intrinsic rewards based exploration in reinforcement learning.

Original: As we study the problem of navigation, we also draw upon recent efforts that use learning for this problem.
Revised: As we study the problem of navigation, we also draw upon recent efforts that use learning for this problem.

Original: We survey related efforts in these three directions.
Revised: We survey related efforts in these three directions.

Original: Navigation in Classical Robotics.
Revised: Navigation in Classical Robotics.

Original: Classical approaches to navigation operate by building a map of the environment, localizing the agent in this map, and then planning paths to convey the agent to desired target locations.
Revised: Classical approaches to navigation operate by building a map of the environment, localizing the agent in this map, and then planning paths to convey the agent to desired target locations.

Original: Consequently, the problems of mapping, localization and path-planning have been very thoroughly studied (Hartley & Zisserman, 2003; Thrun et al., 2005; LaValle, 2006).
Revised: Consequently, the problems of mapping, localization and path-planning have been very thoroughly studied (Hartley & Zisserman, 2003; Thrun et al., 2005; LaValle, 2006).

Original: However, most of this research starts from a human-operated traversal of the environment, and falls under the purview of passive SLAM.
Revised: However, most of this research starts from a human-operated traversal of the environment, and falls under the purview of passive SLAM.

Original: Active SLAM, or how to automatically traverse a new environment for building spatial representations is much less studied.
Revised: Active SLAM, or how to automatically traverse a new environment for building spatial representations is much less studied.

Original: Cadena et al.
Revised: Cadena et al.

Original: (2016) present an excellent review of active SLAM literature, we summarize some key efforts here.
Revised: (2016) present an excellent review of active SLAM literature, we summarize some key efforts here.

Original: Past works have formulated active SLAM as Partially Observable Markov Decision Processes (POMDPs) (MartinezCantin et al., 2009), or as choosing actions that reduce uncertainty in the estimated map (Carrillo et al., 2012).
Revised: Past works have formulated active SLAM as Partially Observable Markov Decision Processes (POMDPs) (MartinezCantin et al., 2009), or as choosing actions that reduce uncertainty in the estimated map (Carrillo et al., 2012).

Original: While these formulations enable theoretical analysis, they crucially rely on sensors to build maps and localize.
Revised: While these formulations enable theoretical analysis, they crucially rely on sensors to build maps and localize.

Original: Thus, such approaches are highly susceptible to measurement noise.
Revised: Thus, such approaches are highly susceptible to measurement noise.

Original: Additionally, such methods treat exploration purely as a geometry problem, and entirely ignore semantic cues for exploration such as doors.
Revised: Additionally, such methods treat exploration purely as a geometry problem, and entirely ignore semantic cues for exploration such as doors.

Original: Learning for Navigation.
Revised: Learning for Navigation.

Original: In order to leverage such semantic cues for navigation, recent works have formulated navigation as a learning problem (Zhu et al., 2017; Gupta et al., 2017; Mirowski et al., 2017;
Revised: In order to leverage such semantic cues for navigation, recent works have formulated navigation as a learning problem (Zhu et al., 2017; Gupta et al., 2017; Mirowski et al., 2017;

Original: Savinov et al., 2018).
Revised: Savinov et al., 2018).

Original: A number of design choices have been investigated.
Revised: A number of design choices have been investigated.

Original: For example, these works have investigated different policy architectures for representing space: Zhu et al.
Revised: For example, these works have investigated different policy architectures for representing space: Zhu et al.

Original: (2017) use feed-forward networks, Mirowski et al.
Revised: (2017) use feed-forward networks, Mirowski et al.

Original: (2017) use vanilla neural network memory, Gupta et al.
Revised: (2017) use vanilla neural network memory, Gupta et al.

Original: (2017) use spatial memory and planning modules, and Savinov et al.
Revised: (2017) use spatial memory and planning modules, and Savinov et al.

Original: (2018) use semi-parametric topological memory.
Revised: (2018) use semi-parametric topological memory.

Original: Different training paradigms have also been explored: Gupta et al.
Revised: Different training paradigms have also been explored: Gupta et al.

Original: (2017) learn to imitate behavior of an optimal expert, Mirowski et al.
Revised: (2017) learn to imitate behavior of an optimal expert, Mirowski et al.

Original: (2017) and Zhu et al.
Revised: (2017) and Zhu et al.

Original: (2017) use extrinsic reward based reinforcement learning, while Savinov et al.
Revised: (2017) use extrinsic reward based reinforcement learning, while Savinov et al.

Original: (2018) use self-supervision.
Revised: (2018) use self-supervision.

Original: In our work here, we build on insights from these past works.
Revised: In our work here, we build on insights from these past works.

Original: Our policy architecture and reward deﬁnition use spatial memory, and we imitate human exploration demonstrations to boot-strap policy learning.
Revised: Our policy architecture and reward deﬁnition use spatial memory, and we imitate human exploration demonstrations to boot-strap policy learning.

Original: However, in crucial distinction, instead of studying goal-directed navigation (either in the form of going to a particular goal location, or object of interest), we study the problem of autonomous exploration of novel environments in a task-agnostic manner.
Revised: However, in crucial distinction, instead of studying goal-directed navigation (either in the form of going to a particular goal location, or object of interest), we study the problem of autonomous exploration of novel environments in a task-agnostic manner.

Original: In doing so, unlike past works, we do not assume access to human demonstrations in the given novel test environment like Savinov et al.
Revised: In doing so, unlike past works, we do not assume access to human demonstrations in the given novel test environment like Savinov et al.

Original: (2018), nor do we assume availability of millions of samples of experience or reward signals in the novel test environment like Zhu et al.
Revised: (2018), nor do we assume availability of millions of samples of experience or reward signals in the novel test environment like Zhu et al.

Original: (2017) or Mirowski et al.
Revised: (2017) or Mirowski et al.

Original: Moreover, we do not depend on extrinsically deﬁned reward signals for training.
Revised: Moreover, we do not depend on extrinsically deﬁned reward signals for training.

Original: We derive them using on-board sensors.
Revised: We derive them using on-board sensors.

Original: This makes our proposed approach amenable to real world deployment.
Revised: This makes our proposed approach amenable to real world deployment.

Original: Learning based
Revised: Learning based

Original: [Figure]
Revised: [Figure]

Original: #$%&' ())
Revised: #$%&' ())

Original: =
Revised: =

Original: [Figure]
Revised: [Figure]

Original: (- $./ )− ,(- $ ) ! "
Revised: (- $./ )− ,(- $ ) ! "

Original: #$%&11 )
Revised: #$%&11 )

Original: [Figure]
Revised: [Figure]

Original: −
Revised: −

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: +
Revised: +

Original: [Figure]
Revised: [Figure]

Original: Logits
Revised: Logits

Original: [Figure]
Revised: [Figure]

Original: 89 $ - $ - $./ :
Revised: 89 $ - $ - $./ :

Original: [Figure]
Revised: [Figure]

Original: approaches (Gandhi et al., 2017; Sadeghi & Levine, 2017) have also been used to learn low-level collision avoidance policies.
Revised: approaches (Gandhi et al., 2017; Sadeghi & Levine, 2017) have also been used to learn low-level collision avoidance policies.

Original: While they do not use task-context, they learn to move towards open space, without the intent of exploring the whole environment.
Revised: While they do not use task-context, they learn to move towards open space, without the intent of exploring the whole environment.

Original: Intrinsic Rewards for Exploration.
Revised: Intrinsic Rewards for Exploration.

Original: A notable few past works share a similar line of thought, and investigate exploration in context of reinforcement learning (Schmidhuber, 1991; Stadie et al., 2015; Pathak et al., 2017; Fu et al., 2017; Lopes et al., 2012; Chentanez et al., 2005).
Revised: A notable few past works share a similar line of thought, and investigate exploration in context of reinforcement learning (Schmidhuber, 1991; Stadie et al., 2015; Pathak et al., 2017; Fu et al., 2017; Lopes et al., 2012; Chentanez et al., 2005).

Original: These works design intrinsic reward functions to capture novelty of states or state-action transitions.
Revised: These works design intrinsic reward functions to capture novelty of states or state-action transitions.

Original: Exploration policies are then learned by optimizing these reward functions using reinforcement learning.
Revised: Exploration policies are then learned by optimizing these reward functions using reinforcement learning.

Original: Our work is most similar to these works.
Revised: Our work is most similar to these works.

Original: However, we focus on this problem in the speciﬁc context of navigation in complex and realistic 3D environments, and propose specialized policy architectures and intrinsic reward functions.
Revised: However, we focus on this problem in the speciﬁc context of navigation in complex and realistic 3D environments, and propose specialized policy architectures and intrinsic reward functions.

Original: We experimentally demonstrate that these specializations improve performance, and how learning based exploration techniques may not be too far from real world deployment.
Revised: We experimentally demonstrate that these specializations improve performance, and how learning based exploration techniques may not be too far from real world deployment.

Original: Jayaraman & Grauman (2018) use a related reward function (pixel reconstruction) to learn policies to look around (and subsequently solve tasks).
Revised: Jayaraman & Grauman (2018) use a related reward function (pixel reconstruction) to learn policies to look around (and subsequently solve tasks).

Original: However, they do it in context of 360 ◦ images, and their precise reward can’t be estimated intrinsically.
Revised: However, they do it in context of 360 ◦ images, and their precise reward can’t be estimated intrinsically.

Original: System Identiﬁcation.
Revised: System Identiﬁcation.

Original: Finally, the general idea of exploration prior to goal-driven behavior, is also related to the classical idea of system identiﬁcation (Ljung, 1987).
Revised: Finally, the general idea of exploration prior to goal-driven behavior, is also related to the classical idea of system identiﬁcation (Ljung, 1987).

Original: Recent interest in this idea with end-to-end learning Yu et al.
Revised: Recent interest in this idea with end-to-end learning Yu et al.

Original: (2017); Duan et al.
Revised: (2017); Duan et al.

Original: ; Mishra et al.
Revised: ; Mishra et al.

Original: (2018) has been shown to successfully adapt to novel mazes.
Revised: (2018) has been shown to successfully adapt to novel mazes.

Original: In comparison, we tackle navigation in complex and realistic 3D environments, with very long spatio-temporal dependencies.
Revised: In comparison, we tackle navigation in complex and realistic 3D environments, with very long spatio-temporal dependencies.

Original:  A PPROACH
Revised:  A PPROACH

Original: Let us consider a mobile agent that is equipped with an RGB-D camera and that can execute basic movement macro-actions.
Revised: Let us consider a mobile agent that is equipped with an RGB-D camera and that can execute basic movement macro-actions.

Original: This agent has been dropped into a novel environment that it has never been in before.
Revised: This agent has been dropped into a novel environment that it has never been in before.

Original: We want to learn an exploration policy π e that enables this agent to efﬁciently explore this new environment.
Revised: We want to learn an exploration policy π e that enables this agent to efﬁciently explore this new environment.

Original: A successful exploration of the new environment should enable the agent to accomplish down-stream tasks in this new environment efﬁciently.
Revised: A successful exploration of the new environment should enable the agent to accomplish down-stream tasks in this new environment efﬁciently.

Original: We formulate estimation of π e as a learning problem.
Revised: We formulate estimation of π e as a learning problem.

Original: We design a reward function that is estimated entirely using on-board sensors, and learn π e using RL.
Revised: We design a reward function that is estimated entirely using on-board sensors, and learn π e using RL.

Original: Crucially, π e is learned on a set of environments E train , and tested on a held-out set of environments E test , i.e. , E train (cid:84) E test = φ .
Revised: Crucially, π e is learned on a set of environments E train , and tested on a held-out set of environments E test , i.e. , E train (cid:84) E test = φ .

Original: The key novelties in our work are: a) the design of the policy π e , b) the design of the reward function
Revised: The key novelties in our work are: a) the design of the policy π e , b) the design of the reward function

Original: R , and c) use of human demonstrations to speed up training of π e .
Revised: R , and c) use of human demonstrations to speed up training of π e .

Original: The design of the policy and the reward function depend on a rudimentary map of the environment that we maintain over time.
Revised: The design of the policy and the reward function depend on a rudimentary map of the environment that we maintain over time.

Original: We ﬁrst describe the map construction procedure, and then detail the aforementioned aspects.
Revised: We ﬁrst describe the map construction procedure, and then detail the aforementioned aspects.

Original: 3. M AP C ONSTRUCTION
Revised: 3. M AP C ONSTRUCTION

Original: As the agent moves around, it uses its depth camera to build and update a map of the world around it.
Revised: As the agent moves around, it uses its depth camera to build and update a map of the world around it.

Original: This is done by maintaining an estimate of the agent’s location over time and projecting 3D points observed in the depth image into an allocentric map of the environment.
Revised: This is done by maintaining an estimate of the agent’s location over time and projecting 3D points observed in the depth image into an allocentric map of the environment.

Original: More speciﬁcally, let us assume that the agent’s estimate of its current location at time step t is ˆ x t , and that the agent starts from origin ( i.e. ˆ x 0 = 0 ).
Revised: More speciﬁcally, let us assume that the agent’s estimate of its current location at time step t is ˆ x t , and that the agent starts from origin ( i.e. ˆ x 0 = 0 ).

Original: When the agent executes an action a t at time step t , it updates its position estimate through a known transition function f , i.e. , ˆ x t +1 = f
Revised: When the agent executes an action a t at time step t , it updates its position estimate through a known transition function f , i.e. , ˆ x t +1 = f

Original: (ˆ x t , a t ) .
Revised: (ˆ x t , a t ) .

Original: The depth image observation at time step t + 1 , D t +1 is back-projected into 3D points using known camera intrinsic parameters.
Revised: The depth image observation at time step t + 1 , D t +1 is back-projected into 3D points using known camera intrinsic parameters.

Original: These 3D points are transformed using the estimate ˆ x t +1 , and projected down to generate a 2D map of the environment (that tracks what parts of space are known to be traversable or known to be non-traversable or unknown).
Revised: These 3D points are transformed using the estimate ˆ x t +1 , and projected down to generate a 2D map of the environment (that tracks what parts of space are known to be traversable or known to be non-traversable or unknown).

Original: Note that, because of slippage in wheels etc. , ˆ x t +1 may not be the same as the true relative location of the agent from start of episode x t +1 .
Revised: Note that, because of slippage in wheels etc. , ˆ x t +1 may not be the same as the true relative location of the agent from start of episode x t +1 .

Original: This leads to aliasing in the generated map.
Revised: This leads to aliasing in the generated map.

Original: We do not use expensive non-linear optimization (bundle adjustment) to improve our estimate ˆ x t +1 , but instead rely on learning to provide robustness against this mis-alignment.
Revised: We do not use expensive non-linear optimization (bundle adjustment) to improve our estimate ˆ x t +1 , but instead rely on learning to provide robustness against this mis-alignment.

Original: We use the new back-projected 3D points to update the current map M t and to obtain an updated map M t +1 .
Revised: We use the new back-projected 3D points to update the current map M t and to obtain an updated map M t +1 .

Original: 3. P OLICY A RCHITECTURE
Revised: 3. P OLICY A RCHITECTURE

Original: Before we describe our policy architecture, let’s deﬁne what are the features that a good exploration policy needs
Revised: Before we describe our policy architecture, let’s deﬁne what are the features that a good exploration policy needs

Original: : (a) good exploration of a new environment requires an agent to meaningfully move around by detecting and avoiding obstacles; (b) good policy also requires the agent to identify semantic cues such as doors that may facilitate exploration; (c) ﬁnally, it requires the agent to keep track of what parts of the environment have or have not been explored, and to estimate how to get to parts of the environment that may not have been explored.
Revised: : (a) good exploration of a new environment requires an agent to meaningfully move around by detecting and avoiding obstacles; (b) good policy also requires the agent to identify semantic cues such as doors that may facilitate exploration; (c) ﬁnally, it requires the agent to keep track of what parts of the environment have or have not been explored, and to estimate how to get to parts of the environment that may not have been explored.

Original: This motivates our policy architecture.
Revised: This motivates our policy architecture.

Original: We fuse information from RGB image observations and occupancy grid-based maps.
Revised: We fuse information from RGB image observations and occupancy grid-based maps.

Original: Information from the RGB image allows recognition of useful semantic cues.
Revised: Information from the RGB image allows recognition of useful semantic cues.

Original: While information from the occupancy maps allows the agent to keep track of parts of the environment that have or have not been explored and to plan paths to unexplored regions without bumping into obstacles.
Revised: While information from the occupancy maps allows the agent to keep track of parts of the environment that have or have not been explored and to plan paths to unexplored regions without bumping into obstacles.

Original: The policy architecture is shown in Fig.
Revised: The policy architecture is shown in Fig.

Original: We describe it in detail here:
Revised: We describe it in detail here:

Original: Information from RGB images : RGB images are processed through a CNN.
Revised: Information from RGB images : RGB images are processed through a CNN.

Original: We use a ResNet18 CNN that has been pre-trained on the ImageNet classiﬁcation task, and can identify semantic concepts in images.
Revised: We use a ResNet18 CNN that has been pre-trained on the ImageNet classiﬁcation task, and can identify semantic concepts in images.

Original: Information from Occupancy Maps : We derive an occupancy map from past observations (as described in Sec. 3.1), and use it with a ResNet-18 CNN to extract features.
Revised: Information from Occupancy Maps : We derive an occupancy map from past observations (as described in Sec. 3.1), and use it with a ResNet-18 CNN to extract features.

Original: To simplify learning, we do not use the allocentric map, but transform it into an egocentric map using the estimated position ˆ x t (such that the agent is always at the center of the map, facing upwards).
Revised: To simplify learning, we do not use the allocentric map, but transform it into an egocentric map using the estimated position ˆ x t (such that the agent is always at the center of the map, facing upwards).

Original: This map canonicalization aids learning.
Revised: This map canonicalization aids learning.

Original: It allows the CNN to not only detect unexplored space but to also locate it with respect to its current location.
Revised: It allows the CNN to not only detect unexplored space but to also locate it with respect to its current location.

Original: Additionally, we use two such egocentric maps, a coarse map that captures information about a 40 m × 40 m area around the agent, and a detailed map that describes a 4 m × 4 m neighborhood.
Revised: Additionally, we use two such egocentric maps, a coarse map that captures information about a 40 m × 40 m area around the agent, and a detailed map that describes a 4 m × 4 m neighborhood.

Original: Some of these design choices were inspired by recent work from Gupta et al.
Revised: Some of these design choices were inspired by recent work from Gupta et al.

Original: (2017), Parisotto & Salakhutdinov (2018) and Henriques & Vedaldi (2018), though we make some simpliﬁcations.
Revised: (2017), Parisotto & Salakhutdinov (2018) and Henriques & Vedaldi (2018), though we make some simpliﬁcations.

Original: Recurrent Policy : Information from the RGB image and maps is fused and passed into an RNN.
Revised: Recurrent Policy : Information from the RGB image and maps is fused and passed into an RNN.

Original: Recurrence can allow the agent to exhibit coherent behavior.
Revised: Recurrence can allow the agent to exhibit coherent behavior.

Original: 3. C OVERAGE R EWARD
Revised: 3. C OVERAGE R EWARD

Original: We now describe the intrinsic reward function that we use to train our exploration policy.
Revised: We now describe the intrinsic reward function that we use to train our exploration policy.

Original: We derive this intrinsic rewards from the map M t , by computing the coverage.
Revised: We derive this intrinsic rewards from the map M t , by computing the coverage.

Original: Coverage C
Revised: Coverage C

Original: ( M t ) is deﬁned as the total area in the map that is known to be traversable or known to be non-traversable.
Revised: ( M t ) is deﬁned as the total area in the map that is known to be traversable or known to be non-traversable.

Original: Reward
Revised: Reward

Original: R covint ( t ) at time step t is obtained via gain in coverage in the map: C ( M t +1 ) − C
Revised: R covint ( t ) at time step t is obtained via gain in coverage in the map: C ( M t +1 ) − C

Original: ( M t ) .
Revised: ( M t ) .

Original: Intuitively, if the current observation adds no obstacles or free-space to the map then it adds no information and hence no reward is given.
Revised: Intuitively, if the current observation adds no obstacles or free-space to the map then it adds no information and hence no reward is given.

Original: We also use a collision penalty, that is estimated using the bump sensor,
Revised: We also use a collision penalty, that is estimated using the bump sensor,

Original: R collint ( t ) =
Revised: R collint ( t ) =

Original: − Bump ( t + 1) , where Bump ( t + 1) denotes if a collision occurred while executing action a t .
Revised: − Bump ( t + 1) , where Bump ( t + 1) denotes if a collision occurred while executing action a t .

Original: R covint and R collint are combined to obtain the total reward.
Revised: R covint and R collint are combined to obtain the total reward.

Original: 3. T RAINING P ROCEDURE
Revised: 3. T RAINING P ROCEDURE

Original: Finally, we describe how we optimize the policy.
Revised: Finally, we describe how we optimize the policy.

Original: Navigating in complex realistic 3D houses, requires long-term coherent behavior over multiple time steps, such as existing a room, going through doors, going down hallways.
Revised: Navigating in complex realistic 3D houses, requires long-term coherent behavior over multiple time steps, such as existing a room, going through doors, going down hallways.

Original: Such long-term behavior is hard to learn using reinforcement learning, given sparsity in reward.
Revised: Such long-term behavior is hard to learn using reinforcement learning, given sparsity in reward.

Original: This leads to excessively large sample complexity for learning.
Revised: This leads to excessively large sample complexity for learning.

Original: To overcome this large sample complexity, we pre-train our policy to imitate human demonstrations of how to explore a new environment.
Revised: To overcome this large sample complexity, we pre-train our policy to imitate human demonstrations of how to explore a new environment.

Original: We do this using trajectories collected from AMT workers as they were answering questions in House3D (Das et al., 2018).
Revised: We do this using trajectories collected from AMT workers as they were answering questions in House3D (Das et al., 2018).

Original: We ignore the question that was posed to the human, and treat these trajectories as a proxy for how a human will explore a previosuly unseen environment.
Revised: We ignore the question that was posed to the human, and treat these trajectories as a proxy for how a human will explore a previosuly unseen environment.

Original: After this phase of imitation learning, π e is further trained via policy gradients (Williams, 1992).
Revised: After this phase of imitation learning, π e is further trained via policy gradients (Williams, 1992).

Original: using proximal policy optimization (PPO) from (Schulman et al., 2017).
Revised: using proximal policy optimization (PPO) from (Schulman et al., 2017).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The goal of this paper is to build agents that can autonomously explore novel complex 3D environments.
Revised: The goal of this paper is to build agents that can autonomously explore novel complex 3D environments.

Original: We want to understand this in context of the different choices we made in our design, as well as how our design compares to alternate existing techniques for exploration.
Revised: We want to understand this in context of the different choices we made in our design, as well as how our design compares to alternate existing techniques for exploration.

Original: While coverage of the novel environment is a good task-agnostic metric, we also design experiments to additionally quantify the utility of generic task-agnostic exploration for downstream tasks of interest.
Revised: While coverage of the novel environment is a good task-agnostic metric, we also design experiments to additionally quantify the utility of generic task-agnostic exploration for downstream tasks of interest.

Original: We ﬁrst describe our experimental setup that consists of complex realistic 3D environments and emphasizes the study of generalization to novel environments.
Revised: We ﬁrst describe our experimental setup that consists of complex realistic 3D environments and emphasizes the study of generalization to novel environments.

Original: We then describe experiments that measure task-agnostic exploration via coverage.
Revised: We then describe experiments that measure task-agnostic exploration via coverage.

Original: And ﬁnally, we present experiments where we use different exploration schemes for downstream navigation tasks.
Revised: And ﬁnally, we present experiments where we use different exploration schemes for downstream navigation tasks.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: We conducted our experiments on the House3D simulation environment Wu et al.
Revised: We conducted our experiments on the House3D simulation environment Wu et al.

Original: House 3D is based on realistic apartment layouts from the SUNCG dataset Song et al.
Revised: House 3D is based on realistic apartment layouts from the SUNCG dataset Song et al.

Original: and
Revised: and

Original: simulates ﬁrst-person observations and actions of a robotic agent embodied in these apartments.
Revised: simulates ﬁrst-person observations and actions of a robotic agent embodied in these apartments.

Original: We usehouses each for training and testing.
Revised: We usehouses each for training and testing.

Original: These sets are sampled from the respective sets in House 3D, and do not overlap.
Revised: These sets are sampled from the respective sets in House 3D, and do not overlap.

Original: That is, testing is done on a set of houses not seen during training.
Revised: That is, testing is done on a set of houses not seen during training.

Original: This allows us to study generalization, i.e. , how well our learned policies perform in novel, previously unseen houses.
Revised: This allows us to study generalization, i.e. , how well our learned policies perform in novel, previously unseen houses.

Original: We made one customization to the House 3D environment: by default doors in House 3D are rendered but not used for collision checking.
Revised: We made one customization to the House 3D environment: by default doors in House 3D are rendered but not used for collision checking.

Original: We modiﬁed House 3D to also not render doors, in addition to not using them for collision checking.
Revised: We modiﬁed House 3D to also not render doors, in addition to not using them for collision checking.

Original: Observation Space.
Revised: Observation Space.

Original: We assume that the agent has an RGB-D camera with a ﬁeld of view of 60 ◦ , and a bump sensor.
Revised: We assume that the agent has an RGB-D camera with a ﬁeld of view of 60 ◦ , and a bump sensor.

Original: The RGB-D camera returns a regular RGB image and a depth image that records the distance (depth information is clipped at 3m ) of each pixel from the camera.
Revised: The RGB-D camera returns a regular RGB image and a depth image that records the distance (depth information is clipped at 3m ) of each pixel from the camera.

Original: The bump sensor returns if a collision happened while executing the previous action.
Revised: The bump sensor returns if a collision happened while executing the previous action.

Original: Action Space.
Revised: Action Space.

Original: We followed the same action space as in EmbodiedQA (Das et al., 2018), with 6 motion primitives: move forward 0 .
Revised: We followed the same action space as in EmbodiedQA (Das et al., 2018), with 6 motion primitives: move forward 0 .

Original: 25m , move backward 0 .
Revised: 25m , move backward 0 .

Original: 25m , strafe left 0 .
Revised: 25m , strafe left 0 .

Original: 25m , strafe right 0 .
Revised: 25m , strafe right 0 .

Original: 25m , turn left 9 ◦ , and turn right 9 ◦ .
Revised: 25m , turn left 9 ◦ , and turn right 9 ◦ .

Original: Extrinsic Environmental Reward.
Revised: Extrinsic Environmental Reward.

Original: We do not use any  reward signals from the environment. We reiterate, we do not use any externally
Revised: We do not use any externally speciﬁed reward signal from the environment: R ext ( t )

Original: [Figure]
Revised: [Figure]

Original: Intrinsic Reward.
Revised: Intrinsic Reward.

Original: As described in Sec. 3.3, the intrinsic reward of the agent is based upon the map that it constructs as it moves around (as described in Sec. 3.1), and readings from the bump sensor.
Revised: As described in Sec. 3.3, the intrinsic reward of the agent is based upon the map that it constructs as it moves around (as described in Sec. 3.1), and readings from the bump sensor.

Original: Reward R int ( t ) = αR cov int ( t )+ βR coll int ( t ) .
Revised: Reward R int ( t ) = αR cov int ( t )+ βR coll int ( t ) .

Original: Here R cov int is the coverage reward, and R coll int is the collision reward.
Revised: Here R cov int is the coverage reward, and R coll int is the collision reward.

Original: α, β are hyper-parameters to trade-off how aggressive the agent is.
Revised: α, β are hyper-parameters to trade-off how aggressive the agent is.

Original: Training.
Revised: Training.

Original: As described in Sec. 3.4, we train policies using imitation of human trajectories and RL.
Revised: As described in Sec. 3.4, we train policies using imitation of human trajectories and RL.

Original: Imitation from Human Exploration Trajectories.
Revised: Imitation from Human Exploration Trajectories.

Original: We leverage human exploration trajectories
Revised: We leverage human exploration trajectories

Original: collected from
Revised: collected from

Original: AMT workers as they answered questions for the
Revised: AMT workers as they answered questions for the

Original: EmbodiedQA
Revised: EmbodiedQA

Original: task (Das et al., 2018).
Revised: task (Das et al., 2018).

Original: We ignore the question that was posed to the AMT worker, and train our policy to simply mimic the actions that the humans took.
Revised: We ignore the question that was posed to the AMT worker, and train our policy to simply mimic the actions that the humans took.

Original: As the humans were trying to answer these questions in previously unseen environments, we assume that these trajectories largely exhibit exploration behavior.
Revised: As the humans were trying to answer these questions in previously unseen environments, we assume that these trajectories largely exhibit exploration behavior.

Original: We used a total of
Revised: We used a total of

Original: human
Revised: human

Original: trajectories for this
Revised: trajectories for this

Original: imitation.
Revised: imitation.

Original: Reinforcement Learning : After learning via imitation, we further train the policy using reinforcement learning on the training houses.
Revised: Reinforcement Learning : After learning via imitation, we further train the policy using reinforcement learning on the training houses.

Original: We use PPO (Schulman et al., 2017) to optimize the intrinsic reward deﬁned above.
Revised: We use PPO (Schulman et al., 2017) to optimize the intrinsic reward deﬁned above.

Original: At the start of each episode, the agent is initialized at a random location inside one of the training houses.
Revised: At the start of each episode, the agent is initialized at a random location inside one of the training houses.

Original: Each episode is run for 500 time-steps.
Revised: Each episode is run for 500 time-steps.

Original: We run a total of 6400 episodes which amounts to a total of 3.2M steps of experience.
Revised: We run a total of 6400 episodes which amounts to a total of 3.2M steps of experience.

Original: Baselines.
Revised: Baselines.

Original: We next describe the various baseline methods that we experimented with.
Revised: We next describe the various baseline methods that we experimented with.

Original: We implemented a classical baseline that purely reasons using geometry and a learning baseline that uses curiosity for exploration.
Revised: We implemented a classical baseline that purely reasons using geometry and a learning baseline that uses curiosity for exploration.

Original: Frontier-based Exploration.
Revised: Frontier-based Exploration.

Original: As a classical baseline, we experimented with frontier-based exploration (Yamauchi, 1997).
Revised: As a classical baseline, we experimented with frontier-based exploration (Yamauchi, 1997).

Original: This is a purely geometric method that utilizes the built map M t .
Revised: This is a purely geometric method that utilizes the built map M t .

Original: Every iteration, it samples a point in currently unexplored space, and plans a path towards it from the current location (unobserved space is assumed to be free).
Revised: Every iteration, it samples a point in currently unexplored space, and plans a path towards it from the current location (unobserved space is assumed to be free).

Original: As the plan is executed, both the map and the plan are updated.
Revised: As the plan is executed, both the map and the plan are updated.

Original: Once the chosen point is reached, this process is repeated.
Revised: Once the chosen point is reached, this process is repeated.

Original: Curiosity-based Exploration.
Revised: Curiosity-based Exploration.

Original: The next baseline we tried was curiosity-based exploration.
Revised: The next baseline we tried was curiosity-based exploration.

Original: In particular, we use the version proposed by (Pathak et al., 2017) that uses prediction error of a forward model as reward.
Revised: In particular, we use the version proposed by (Pathak et al., 2017) that uses prediction error of a forward model as reward.

Original: We use the modiﬁcations proposed by (Burda et al., 2018), and only train a forward model.
Revised: We use the modiﬁcations proposed by (Burda et al., 2018), and only train a forward model.

Original: We prevent degeneracy in forward model by learning it in the ﬁxed feature space of a ResNet-18 model that has been pre-trained on the ImageNet classiﬁcation task.
Revised: We prevent degeneracy in forward model by learning it in the ﬁxed feature space of a ResNet-18 model that has been pre-trained on the ImageNet classiﬁcation task.

Original: 4. C OVERAGE Q UALITY
Revised: 4. C OVERAGE Q UALITY

Original: We ﬁrst measure exploration by itself, by measuring the true coverage of the agent.
Revised: We ﬁrst measure exploration by itself, by measuring the true coverage of the agent.

Original: We compute the true coverage using the map as described in Sec. 3.1, except for using the true location of the agent rather than the estimated location ( i.e. x t instead of ˆ x t ).
Revised: We compute the true coverage using the map as described in Sec. 3.1, except for using the true location of the agent rather than the estimated location ( i.e. x t instead of ˆ x t ).

Original: We study the following three scenarios:
Revised: We study the following three scenarios:

Original: [Figure]
Revised: [Figure]

Original: Without Estimation Noise: We ﬁrst evaluate the performance of the agent that is trained and tested without observation noise, that is, ˆ x t = x t .
Revised: Without Estimation Noise: We ﬁrst evaluate the performance of the agent that is trained and tested without observation noise, that is, ˆ x t = x t .

Original: Note that this setting is not very realistic as there is always observation error in an agent’s estimate of its location.
Revised: Note that this setting is not very realistic as there is always observation error in an agent’s estimate of its location.

Original: It is also highly favorable to the frontier-based exploration agent, which very heavily relies on the accuracy of its maps.
Revised: It is also highly favorable to the frontier-based exploration agent, which very heavily relies on the accuracy of its maps.

Original: Fig. 2(left) presents the performance of different policies for this scenario.
Revised: Fig. 2(left) presents the performance of different policies for this scenario.

Original: We ﬁrst note that the curiosity based agent (‘IL + RL with Curiosity’) explores better than a random exploration policy (that executes a random action at each time step).
Revised: We ﬁrst note that the curiosity based agent (‘IL + RL with Curiosity’) explores better than a random exploration policy (that executes a random action at each time step).

Original: However, this curiosity based agent is worse than a RGB only version of our method, an RGB only policy that is trained with our coverage reward (‘IL + RL with RGB’).
Revised: However, this curiosity based agent is worse than a RGB only version of our method, an RGB only policy that is trained with our coverage reward (‘IL + RL with RGB’).

Original: Our full system (‘IL + RL with Map + RGB’) that also uses maps as input performs even better.
Revised: Our full system (‘IL + RL with Map + RGB’) that also uses maps as input performs even better.

Original: Frontier-based exploration has the best performance in this scenario.
Revised: Frontier-based exploration has the best performance in this scenario.

Original: As noted, this is to expect as this method gets access to perfect, fully-registered maps, and employs optimal path planning algorithm to move from one place to another.
Revised: As noted, this is to expect as this method gets access to perfect, fully-registered maps, and employs optimal path planning algorithm to move from one place to another.

Original: It is also worth noting that, it is extremely hard to use such classical techniques in situations where we only have a RGB images.
Revised: It is also worth noting that, it is extremely hard to use such classical techniques in situations where we only have a RGB images.

Original: In contrast, learning allows us to easily arrive at policies that can explore using only RGB images at test time.
Revised: In contrast, learning allows us to easily arrive at policies that can explore using only RGB images at test time.

Original: With Estimation Noise: We next describe a more realistic scenario that has estimation noise, i.e.
Revised: With Estimation Noise: We next describe a more realistic scenario that has estimation noise, i.e.

Original: ˆ x t +1 is estimated using a noisy dynamics function.
Revised: ˆ x t +1 is estimated using a noisy dynamics function.

Original: In particular, we add truncated Gaussian noise to the transition function f at each time step.
Revised: In particular, we add truncated Gaussian noise to the transition function f at each time step.

Original: This noise compounds over time.
Revised: This noise compounds over time.

Original: Fig.
Revised: Fig.

Original: 2(center) presents the coverage area at the end of episode (1000 time steps) as a function of the amount of noise introduced 1 .
Revised: 2(center) presents the coverage area at the end of episode (1000 time steps) as a function of the amount of noise introduced 1 .

Original: When the system suffers from sensor noise, the performance of the frontier-based exploration method drops rapidly.
Revised: When the system suffers from sensor noise, the performance of the frontier-based exploration method drops rapidly.

Original: In contrast, our learning-based agent that wasn’t even trained with any noise continues to performs well.
Revised: In contrast, our learning-based agent that wasn’t even trained with any noise continues to performs well.

Original: Even at relatively modest noise of 4% our learning based method already does better than the frontier-based agent.
Revised: Even at relatively modest noise of 4% our learning based method already does better than the frontier-based agent.

Original: We additionally note that our agent can even be trained when the intrinsic reward estimation itself suffers from state estimation noise: for instance performance with 10% estimation noise (for intrinsic reward computation and map construction) is 98 .
Revised: We additionally note that our agent can even be trained when the intrinsic reward estimation itself suffers from state estimation noise: for instance performance with 10% estimation noise (for intrinsic reward computation and map construction) is 98 .

Original: 9 m 2 , only a minor degradation from 117 .
Revised: 9 m 2 , only a minor degradation from 117 .

Original: 4 m 2 (10% estimation noise for map construction at test time only).
Revised: 4 m 2 (10% estimation noise for map construction at test time only).

Original: Geometry and Affordance Mismatch: Next, to emphasize the utility of learning for this task, we experiment with a scenario where we explicitly create a mismatch between geometry and affordance of the environment.
Revised: Geometry and Affordance Mismatch: Next, to emphasize the utility of learning for this task, we experiment with a scenario where we explicitly create a mismatch between geometry and affordance of the environment.

Original: We do this by rendering doors, but not using them for collision checking ( i.e. the default House 3D environment).
Revised: We do this by rendering doors, but not using them for collision checking ( i.e. the default House 3D environment).

Original: This setting helps us investigate if learning based techniques go beyond simple geometric reasoning in any way.
Revised: This setting helps us investigate if learning based techniques go beyond simple geometric reasoning in any way.

Original: Fig. 2(right) presents performance curves.
Revised: Fig. 2(right) presents performance curves.

Original: We see that there is a large drop in performance for the frontier-based agent.
Revised: We see that there is a large drop in performance for the frontier-based agent.

Original: This is because it is not able to distinguish doors from other obstacles, leading to path planning failures.
Revised: This is because it is not able to distinguish doors from other obstacles, leading to path planning failures.

Original: However, there is a relatively minor drop in performance of our learning-based agent.
Revised: However, there is a relatively minor drop in performance of our learning-based agent.

Original: This is because it can learn about doors (how to identify them in RGB images and the fact that they can be traversed) from human demonstrations and experience during reinforcement learning.
Revised: This is because it can learn about doors (how to identify them in RGB images and the fact that they can be traversed) from human demonstrations and experience during reinforcement learning.

Original: 4.2. A BLATION S TUDY
Revised: 4.2. A BLATION S TUDY

Original: We also conducted ablations of our method to identify what parts of our proposed technique contribute to the performance.
Revised: We also conducted ablations of our method to identify what parts of our proposed technique contribute to the performance.

Original: We do these ablations in the setting without any estimation noise, and use coverage as the metric.
Revised: We do these ablations in the setting without any estimation noise, and use coverage as the metric.

Original: Imitation Learning: We check if pre-training with imitation learning is useful for this task.
Revised: Imitation Learning: We check if pre-training with imitation learning is useful for this task.

Original: We test this by comparing to the models that were trained only with RL using the coverage reward.
Revised: We test this by comparing to the models that were trained only with RL using the coverage reward.

Original: Plots in Fig.
Revised: Plots in Fig.

Original: 3 shows performance of agents with following combinations:
Revised: 3 shows performance of agents with following combinations:

Original: RL : policy trained with PPO only, IL : policy trained with imitation learning, Map : policy uses constructed maps as input, RGB : policy uses RGB images as input.
Revised: RL : policy trained with PPO only, IL : policy trained with imitation learning, Map : policy uses constructed maps as input, RGB : policy uses RGB images as input.

Original: In all settings, pre-training with imitation learning helps improve performance, though training with RL improves coverage further.
Revised: In all settings, pre-training with imitation learning helps improve performance, though training with RL improves coverage further.

Original: Also, policies trained using RL have a fairly large variance.
Revised: Also, policies trained using RL have a fairly large variance.

Original: Imitation learning also helps reduce the variance.
Revised: Imitation learning also helps reduce the variance.

Original: RGB Observations and Map: Fig.
Revised: RGB Observations and Map: Fig.

Original: 3 (left) and Fig.
Revised: 3 (left) and Fig.

Original: (right) respectively show that both
Revised: (right) respectively show that both

Original: RGB images and map inputs improve performance.
Revised: RGB images and map inputs improve performance.

Original: 4. U SING E XPLORATION FOR D OWNSTREAM T ASK
Revised: 4. U SING E XPLORATION FOR D OWNSTREAM T ASK

Original: [Figure]
Revised: [Figure]

Original: Now that we have established how to explore well, we next ask if task-agnostic exploration is even useful for downstream tasks.
Revised: Now that we have established how to explore well, we next ask if task-agnostic exploration is even useful for downstream tasks.

Original: We show this in context of goal-driven navigation.
Revised: We show this in context of goal-driven navigation.

Original: We execute the learned exploration policy π e (for 1500 time steps) to explore a new house, and collect experience (set of images I along with their pose p ).
Revised: We execute the learned exploration policy π e (for 1500 time steps) to explore a new house, and collect experience (set of images I along with their pose p ).

Original: Once this exploration has ﬁnished, the agent is given different navigation tasks.
Revised: Once this exploration has ﬁnished, the agent is given different navigation tasks.

Original: These navigation tasks put the agent at an arbitrary location in the environment, and give it a goal image that it needs to reach.
Revised: These navigation tasks put the agent at an arbitrary location in the environment, and give it a goal image that it needs to reach.

Original: The agent then needs to use the experience of the environment acquired during exploration to efﬁciently navigate to the desired target location.
Revised: The agent then needs to use the experience of the environment acquired during exploration to efﬁciently navigate to the desired target location.

Original: Efﬁciency of navigation on these test queries, measures the utility of exploration.
Revised: Efﬁciency of navigation on these test queries, measures the utility of exploration.

Original: This evaluation requires a navigation policy π n , that uses the exploration experience and the goal image to output actions that can convey the agent to the desired target location.
Revised: This evaluation requires a navigation policy π n , that uses the exploration experience and the goal image to output actions that can convey the agent to the desired target location.

Original: We opt for a simple policy π n .
Revised: We opt for a simple policy π n .

Original: π n ﬁrst localizes the target image using nearest neighbor matching to the set of collected images I (in
Revised: π n ﬁrst localizes the target image using nearest neighbor matching to the set of collected images I (in

Original: ImageNet pre-trained ResNet-18 feature space).
Revised: ImageNet pre-trained ResNet-18 feature space).

Original: It then plans a path to the this estimated target location using a occupancy map computed from I .
Revised: It then plans a path to the this estimated target location using a occupancy map computed from I .

Original: We do this experiment in the setting without any state estimation noise.
Revised: We do this experiment in the setting without any state estimation noise.

Original: We independently measure the effectiveness of exploration data for a) localization of the given target images, and b) path planning efﬁciency in reaching desired locations.
Revised: We independently measure the effectiveness of exploration data for a) localization of the given target images, and b) path planning efﬁciency in reaching desired locations.

Original: Localization performance:
Revised: Localization performance:

Original: We measure the distance between the agent’s estimate of the goal image location, and the true goal image location.
Revised: We measure the distance between the agent’s estimate of the goal image location, and the true goal image location.

Original: Fig.
Revised: Fig.

Original: 4 (top) plots the success at localization as a function of the success threshold (distance at which we consider a localization as correct).
Revised: C.5 (top) plots the success at localization as a function of the success threshold (distance at which we consider a localization as correct).

Original: We report top-1 and top-5 success rates.
Revised: We report top-1 and top-5 success rates.

Original: We compare to the random exploration baseline which serves to measure the hardness of the task.
Revised: We compare to the random exploration baseline which serves to measure the hardness of the task.

Original: We see our exploration scheme performs well.
Revised: We see our exploration scheme performs well.

Original: Path planning efﬁciency: Next, we measure how efﬁciently desired goal locations can be reached.
Revised: Path planning efﬁciency: Next, we measure how efﬁciently desired goal locations can be reached.

Original: We measure performance using the SPL metric as described by (Anderson et al., 2018)
Revised: We measure performance using the SPL metric as described by (Anderson et al., 2018)

Original: (described in the appendix, higher is better).
Revised: (described in the appendix, higher is better).

Original: We compare against a baseline that does not have any prior experience in this environment and derives all map information on the ﬂy from goal driven behavior (going to the desired test location).
Revised: We compare against a baseline that does not have any prior experience in this environment and derives all map information on the ﬂy from goal driven behavior (going to the desired test location).

Original: Once again this serves as a measure of the hardness of the topology of the underlying environment.
Revised: Once again this serves as a measure of the hardness of the topology of the underlying environment.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 4 (bottom), using exploration data from our policy improves efﬁciency of paths to reach target locations.
Revised: C.5 (bottom), using exploration data from our policy improves efﬁciency of paths to reach target locations.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we motivated the need for learning explorations policies for navigation in realistic, 3D environments.
Revised: In this paper, we motivated the need for learning explorations policies for navigation in realistic, 3D environments.

Original: We showed how to design and train such policies, and how experience gathered from such policies enables better performance for downstream tasks.
Revised: We showed how to design and train such policies, and how experience gathered from such policies enables better performance for downstream tasks.

Original: We think we have just scratched the surface of this problem, and hope this inspires future research towards semantic exploration using more expressive policy architectures, reward functions and training techniques.
Revised: We think we have just scratched the surface of this problem, and hope this inspires future research towards semantic exploration using more expressive policy architectures, reward functions and training techniques.

Original: A PPENDIX A SPL M ETRIC
Revised: A PPENDIX A SPL M ETRIC

Original: As deﬁned by Anderson et al.
Revised: As deﬁned by Anderson et al.

Original: , Success weighted by (normalized inverse)
Revised: , Success weighted by (normalized inverse)

Original: Path Length or SPL is computed as follows.
Revised: Path Length or SPL is computed as follows.

Original: Here, (cid:96) i is the shortest-path distance between the starting and goal position, p i is the length of the path actually executed by the agent, and S i is a binary variable that indicates if the agent succeeded.
Revised: Here, (cid:96) i is the shortest-path distance between the starting and goal position, p i is the length of the path actually executed by the agent, and S i is a binary variable that indicates if the agent succeeded.

Original: SPL is computed over N trials as follows:
Revised: SPL is computed over N trials as follows:

Original: [Equation]
Revised: [Equation]

Original: A PPENDIX B E XAMPLES OF P OLICY I NPUTS AND M APS
Revised: A PPENDIX B E XAMPLES OF P OLICY I NPUTS AND M APS

Original: We show an example of the policy inputs (RGB image and maps in two different scales) at time t and t + 10 in Fig.
Revised: We show an example of the policy inputs (RGB image and maps in two different scales) at time t and t + 10 in Fig.

Original: A1.
Revised: B.1.

Original: Green area means the known(seen) traversable area, blue area means known non-traversable area, and the white area means unknown area.
Revised: Green area means the known(seen) traversable area, blue area means known non-traversable area, and the white area means unknown area.

Original: Fig.
Revised: Fig.

Original: A2 shows how the map evolves and the agent moves as it explores a new house.
Revised: B.2 shows how the map evolves and the agent moves as it explores a new house.

Original: [Figure]
Revised: [Figure]

Revised: (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l)

Revised: A PPENDIX C E XPERIMENTAL D ETAILS C. E NVIRONMENT D ETAILS

Revised: In this paper, we simulated the agent in House3D environment Wu et al.

Revised: and used 20 houses for training and 20 new houses for testing.

Revised: Fig.

Revised: C.1 shows the distribution of the total traversable area of these houses.

Revised: Fig.

Revised: C.2 shows some examples top-down views for training and testing houses (white is traversable, black is occupied).

Revised: [Figure]

Revised: C. M AP R ECONSTRUCTION

Revised: Given a sequence of camera poses (extrinsics) ( [ R 1 , t 1 ] , [ R 2 , t 2 ] , ..., [ R N , t N ] ), and the corresponding depth images ( D 1 , D 2 , ..., D N ) at each time step as well as the camera intrinsics K ∈ R 3 × 3 , we can reconstruct the point cloud of the scene based on the principle of mulitple-view geometry (Hartley & Zisserman, 2003).

Revised: We can convert the points from pixel space to the camera space and then to the world space.

Revised: More speciﬁcally, the formulas to achieve this can be summarized as follows:

Revised: [Equation]

Revised: where S is the total number of pixels in each depth image, x ij ∈ R 3 is the homogeneous coordinates for j th pixel on i th depth image

Revised: D i , λ ij ∈ R is the depth value of the j th pixel on i th depth image

Revised: D i , and w ij ∈ R 4 is the homogeneous coordinates for the corresponding point in the world coordinate system.

Revised: We can get w ij from x ij based on Equation (2).

Revised: And we can merge points via Equation (3).

Revised: C. T RAINING D ETAILS

Revised: The occupancy map generated by the agent itself uses a resolution of 0 .

Revised: 05 m. Our policy uses a coarse map, and a detailed map.

Revised: The coarse map captures information about a 40 m × 40 m area around the agent, at a resolution of of 0 .

Revised: 5 m .

Revised: The occupancy map is down-sampled from 800 ×to 80 × 80 to generate the coarse map that is fed into the policy network.

Revised: The detailed map captures information about a 4 m × 4 m area around the agent at a 0 .

Revised: 05 m resolution.

Revised: RGB images are rendered at 224 × 224 resolution in House3D and re-sized into 80 × 80 before they are fed into the policy network.

Revised: The size of the last fully-connected layer in ResNet18 architecture is modiﬁed to 128.

Revised: Outputs of the three ResNet18 networks are concatenated into a 384-dimensional vector.

Revised: This is transformed into a 128-dimensional vector via a fully-connected layer.

Revised: Next, it’s fed into a single-layer RNN(GRU) layer with 128 hidden layer size.

Revised: The output of RNN layer is followed by two heads.

Revised: One head has two fully-connected layers ( 128 − 32 − 6 ) as the policy head.

Revised: The other head has two fully-connected layers ( 128 − 32 − 1 ) as the value head.

Revised: We use ELU as the nonlinear activation function for the fully-connected layers after ResNet18.

Revised: Each episode consists of 500 steps in training.

Revised: RNN time sequence length is 20 .

Revised: Coverage area

Revised: C ( M t ) is measured by the number of covered grids in the occupancy map.

Revised: Coefﬁcient α for the coverage reward R cov int ( t ) is 0 .

Revised: 0005 , coefﬁcient β for R coll int ( t ) is 0 .

Revised: PPO entropy loss coefﬁcient is 0 .

Revised: Network is optimized via Adam optimizer with a learning rate of 0 . 00001 .

Original: [Figure] 
Revised: [Figure] Fig.

Revised: C.3 shows the exploration performance of different algorithms for 2000 steps.

Revised: The trends and conclusions made from 1000 step runs continue to hold, for all settings presented in the main paper (no noise, 14% noise, and geometry-affordance mismatch).

Revised: We also include additional suggested baselines.

Revised: Straight (Fig. C.3a) is a baseline where the agent moves along a straight line and executes a random number of 9 ◦ turns when a collision occurs.

Revised: One Step Greedy (Fig. C.3a) is a baseline where the agent greedily takes the action that maximize the coverage increment in the next step.

Revised: This greedy policy quickly gets stuck in a local region.

Revised: Collision Penalty Only (IL+RL with MAP+RGB)

Revised: (Fig.

Revised: C.4a) is a baseline where the policy network uses exactly same input and architecture as our method, but is trained with collision penalty reward only (no coverage reward).

Revised: We can see that agents trained with purely collision reward behave similar to a random policy.

Revised: Our(Coverage reward only, no collision penalty)

Revised: (Fig.

Revised: C.4b) is our policy architecture trained with only coverage reward and without the collision penalty.

Revised: We can see that collision penalty improves the overall exploration performance.

Revised: [Figure]

Revised: Extrinsic Reward Yielding Objects (Fig.

Revised: C.4c).

Revised: The extrinsic reward is setup as follows: we randomly generated a number of locations evenly distributed across the traversable area of the houses, where the agent will get a positive reward if the agent is close to any of these locations.

Revised: Once the agent gets a reward from a location, this location will be no longer taken into account for future reward calculation.

Revised: We tried two settings where we place 1 or 4 reward-yielding objects per m(Fig.

Revised: C.4c).

Revised: We can see that our coverage map reward provides a better reward signal and in fact can be estimated intrinsically without needing to instrument the environment.

Revised: C. I MITATION L EARNING D ETAILS

Revised: We use behavioral cloning technique (Argall et al., 2009; Michalski et al., 1998; Abbott, 2007) to imitate the human behaviors in exploring the environments.

Revised: We got the human demonstration

Revised: [Figure]

Revised: trajectories from (Das et al., 2018).

Revised: We ignored the question-answering part of the data and only used the exploration trajectories.

Revised: We cleaned up the data by removing the trajectories that have less than 100 time steps.

Revised: The trajectories are then converted into short sequences of trajectory segments ( ( s i , a i , s i +1 , a i +1 , ..., s i + T , a i + T ) , where T is based on the RNN sequence length).

Revised: The policy is pretrained with behavioral cloning by imitating actions ( a i , a i +1 , ..., a i + T ) from the human demonstrations given the states ( s i , s i +1 , ..., s i + T ) .

Revised: C. N AVIGATION T ASK D ETAILS

Revised: To show the beneﬁt of the exploration policy, we tested it based on two metrics.

Revised: The ﬁrst one is the localization performance.

Revised: We let each policy run for 1500 time steps.

Revised: And then we reset the agent to 50 random poses (positions and orientations) in each testing house (20 testing houses in total) and get the RGB camera view.

Revised: We use ResNet18 as the feature extractor and K-nearest neighbors to determine whether the agent has seen the target image during the exploration stage.

Revised: We counted the success rate of localizing the target images.

Revised: Fig.

Revised: C.5 shows the localization success rate of different policies.

Revised: We see that our learned policy gives highest success rate compared to the curiosity-driven policy and random policy.

Revised: The second one is the path planning efﬁciency.

Revised: We use the cases where the target images are successfully localized ( < 5 m) to calculate the SPL metric.

Revised: Table C.1 shows the path planning efﬁciency via SPL.

Revised: The curiosity-driven agent takes action solely based on the network policy output.

Revised: And the other two agents take actions based on the shortest-path motion planning algorithm.

Revised: [Figure]

Revised: [Table]