Original: ROS-HPL: R OBOTIC O BJECT S EARCH WITH H IERARCHICAL P OLICY L EARNING AND I NTRINSIC -E XTRINSIC M ODELING
Revised: ROS-HPL: R OBOTIC O BJECT S EARCH WITH H IERARCHICAL P OLICY L EARNING AND I NTRINSIC -E XTRINSIC M ODELING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite signiﬁcant progress in Robotic Object Search (ROS) over the recent years with deep reinforcement learning based approaches, the sparsity issue in reward setting as well as the lack of interpretability of the previous ROS approaches leave much to be desired.
Revised: Despite signiﬁcant progress in Robotic Object Search (ROS) over the recent years with deep reinforcement learning based approaches, the sparsity issue in reward setting as well as the lack of interpretability of the previous ROS approaches leave much to be desired.

Original: We present a novel policy learning approach for ROS, based on a hierarchical and interpretable modeling with intrinsic/extrinsic reward setting, to tackle these two challenges.
Revised: We present a novel policy learning approach for ROS, based on a hierarchical and interpretable modeling with intrinsic/extrinsic reward setting, to tackle these two challenges.

Original: More speciﬁcally, we train the low-level policy by deliberating between an action that achieves an immediate sub-goal and the one that is better suited for achieving the ﬁnal goal.
Revised: More speciﬁcally, we train the low-level policy by deliberating between an action that achieves an immediate sub-goal and the one that is better suited for achieving the ﬁnal goal.

Original: We also introduce a new evaluation metric, namely the extrinsic reward, as a harmonic measure of the object search success rate and the average steps taken.
Revised: We also introduce a new evaluation metric, namely the extrinsic reward, as a harmonic measure of the object search success rate and the average steps taken.

Original: Experiments conducted with multiple settings on the House3D environment validate and show that the intelligent agent, trained with our model, can achieve a better object search performance (higher success rate with lower average steps, measured by SPL: Success weighted by inverse Path Length).
Revised: Experiments conducted with multiple settings on the House3D environment validate and show that the intelligent agent, trained with our model, can achieve a better object search performance (higher success rate with lower average steps, measured by SPL: Success weighted by inverse Path Length).

Original: In addition, we conduct studies w.r.t.
Revised: In addition, we conduct studies w.r.t.

Original: the parameter that controls the weighted overall reward from intrinsic and extrinsic components.
Revised: the parameter that controls the weighted overall reward from intrinsic and extrinsic components.

Original: The results suggest it is critical to devise a proper trade-off strategy to perform the object search well.
Revised: The results suggest it is critical to devise a proper trade-off strategy to perform the object search well.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Robotic Object Search (ROS) is a task where an intelligent agent (a.k.a. robot) is expected to take reasonable steps to approach the user-speciﬁed object in an unknown indoor environment.
Revised: Robotic Object Search (ROS) is a task where an intelligent agent (a.k.a. robot) is expected to take reasonable steps to approach the user-speciﬁed object in an unknown indoor environment.

Original: ROS is an essential capability for assistant robots and could serve as the enabling step for other tasks, such as the Embodied Question Answering (Das et al., 2018a).
Revised: ROS is an essential capability for assistant robots and could serve as the enabling step for other tasks, such as the Embodied Question Answering (Das et al., 2018a).

Original: Recently, (deep) reinforcement learning (RL) has demonstrated its power at enabling robots with autonomous behaviors (Arulkumaran et al., 2017), such as navigating over an unknown environment (Mirowski et al., 2016; Zhu et al., 2017), manipulating objects with robot’s end effectors (Gu et al., 2017; Popov et al., 2017; Rajeswaran et al., 2017), and motion planning (Chen et al., 2017; Everett et al., 2018).
Revised: Recently, (deep) reinforcement learning (RL) has demonstrated its power at enabling robots with autonomous behaviors (Arulkumaran et al., 2017), such as navigating over an unknown environment (Mirowski et al., 2016; Zhu et al., 2017), manipulating objects with robot’s end effectors (Gu et al., 2017; Popov et al., 2017; Rajeswaran et al., 2017), and motion planning (Chen et al., 2017; Everett et al., 2018).

Original: Under the RL setting, a robot learns behavioral policy by maximizing the expected rewards that are estimated by interacting with the environment physically and/or virtually.
Revised: Under the RL setting, a robot learns behavioral policy by maximizing the expected rewards that are estimated by interacting with the environment physically and/or virtually.

Original: The estimated rewards serve as the reinforcement signals for robot to update its policy.
Revised: The estimated rewards serve as the reinforcement signals for robot to update its policy.

Original: A well-known challenge to train agent to perform ROS with RL is the sparse reward issue, due to the fact that the environment and/or the location of the target object are typically unknown.
Revised: A well-known challenge to train agent to perform ROS with RL is the sparse reward issue, due to the fact that the environment and/or the location of the target object are typically unknown.

Original: With welldeﬁned reward functions, such as the ones in Atari games (Mnih et al., 2015), the agents are shown to achieve extremely promising performance.
Revised: With welldeﬁned reward functions, such as the ones in Atari games (Mnih et al., 2015), the agents are shown to achieve extremely promising performance.

Original: However, it is a well-known challenge to deﬁne the reward function under the real-world scenarios(Abbeel & Ng, 2004).
Revised: However, it is a well-known challenge to deﬁne the reward function under the real-world scenarios(Abbeel & Ng, 2004).

Original: Typically, for the real-world applications such as object search or target-driven visual navigation, prior research prefers to construct the reward function in terms of the distance between the robot’s current location and the target location under the assumption that the full information of the environment is known (Mousavian et al., 2018;
Revised: Typically, for the real-world applications such as object search or target-driven visual navigation, prior research prefers to construct the reward function in terms of the distance between the robot’s current location and the target location under the assumption that the full information of the environment is known (Mousavian et al., 2018;

Original: Wang et al., 2018b;a).
Revised: Wang et al., 2018b;a).

Original: Given an unknown environment, a straightforward way is to set a high reward when robot reaches the ﬁnal goal state while at all other intermediate states, the reward is either zero or with a small negative value (Zhu et al., 2017).
Revised: Given an unknown environment, a straightforward way is to set a high reward when robot reaches the ﬁnal goal state while at all other intermediate states, the reward is either zero or with a small negative value (Zhu et al., 2017).

Original: More recently, Ye et al.
Revised: More recently, Ye et al.

Original: (2018b) presented a relatively denser reward function which is based on the bounding box of the target object from robot’s detection system, the reward is still not deﬁned among the situations that the target object is not detected.
Revised: (2018b) presented a relatively denser reward function which is based on the bounding box of the target object from robot’s detection system, the reward is still not deﬁned among the situations that the target object is not detected.

Original: Thus, learning in a sparse reward setting (reward is only deﬁned for a small subset of states) is a fundamental challenge for RL agents to perform ROS well, especially dealing with real-world scenarios and in complex environments.
Revised: Thus, learning in a sparse reward setting (reward is only deﬁned for a small subset of states) is a fundamental challenge for RL agents to perform ROS well, especially dealing with real-world scenarios and in complex environments.

Original: To overcome the sparsity issue in reward settings, hierarchical RL has shown its superiority under the sparse reward setting (Kulkarni et al., 2016; Le et al., 2018; Levy et al., 2018).
Revised: To overcome the sparsity issue in reward settings, hierarchical RL has shown its superiority under the sparse reward setting (Kulkarni et al., 2016; Le et al., 2018; Levy et al., 2018).

Original: It aims to learn multiple layers of policies, in which the higher layer breaks down the task into several easier sub-tasks and proposes corresponding sub-goals for the lower layer to achieve.
Revised: It aims to learn multiple layers of policies, in which the higher layer breaks down the task into several easier sub-tasks and proposes corresponding sub-goals for the lower layer to achieve.

Original: This kind of hierarchical policy is biologically plausible and intuitive.
Revised: This kind of hierarchical policy is biologically plausible and intuitive.

Original: For human beings, when asked to accomplish a complicated task, we rarely plan over the atomic steps directly towards the target goal.
Revised: For human beings, when asked to accomplish a complicated task, we rarely plan over the atomic steps directly towards the target goal.

Original: Instead, we plan through a sequence of semantic meaningful sub-goals.
Revised: Instead, we plan through a sequence of semantic meaningful sub-goals.

Original: Motivated by this observation, we put forward a novel two-layer hierarchical policy learning paradigm to deal with the sparse reward challenge.
Revised: Motivated by this observation, we put forward a novel two-layer hierarchical policy learning paradigm to deal with the sparse reward challenge.

Original: One way to contruct the hierarchical policy learning is by adopting the option-critic framework (Bacon et al., 2017) to generate the sub-goals automatically.
Revised: One way to contruct the hierarchical policy learning is by adopting the option-critic framework (Bacon et al., 2017) to generate the sub-goals automatically.

Original: By specifying the number of desired sub-tasks or sub-goals, the option-critic framework can automatically abstract the whole action sequences into the speciﬁed number of the sub-goals.
Revised: By specifying the number of desired sub-tasks or sub-goals, the option-critic framework can automatically abstract the whole action sequences into the speciﬁed number of the sub-goals.

Original: However, the automatically generated sub-goals typically lack semantically meaningful senses, thus are difﬁcult for humans to understand and hurts the overall system’s interpretability.
Revised: However, the automatically generated sub-goals typically lack semantically meaningful senses, thus are difﬁcult for humans to understand and hurts the overall system’s interpretability.

Original: In the other way pf constructing the hierarchical policy learning, as shown in work (Kulkarni et al., 2016; Das et al., 2018b), the sub-goal space could be designated from human knowledge with semantic meaningful terms.
Revised: In the other way of constructing the hierarchical policy learning, as shown in work (Kulkarni et al., 2016; Das et al., 2018b), the sub-goal space could be designated from human knowledge with semantic meaningful terms.

Original: Constructing the hierarchy following human knowledge is intuitive and interpretable.
Revised: Constructing the hierarchy following human knowledge is intuitive and interpretable.

Original: Whenever the high-level layer proposes a sub-goal from the sub-goal space, the lowlevel layer then aims to achieve it without considering the overall goal again.
Revised: Whenever the high-level layer proposes a sub-goal from the sub-goal space, the lowlevel layer then aims to achieve it without considering the overall goal again.

Original: However, although their results are promising for certain tasks, it is worth to mention that there is no guarantee the prior human knowledge is always perfect.
Revised: However, although their results are promising for certain tasks, it is worth to mention that there is no guarantee the prior human knowledge is always perfect.

Original: The low-level layer policy which focuses only on achieving the sub-goal proposed by the high-level layer policy without keeping the ﬁnal goal in mind, may drift away from being ultimately successful.
Revised: The low-level layer policy which focuses only on achieving the sub-goal proposed by the high-level layer policy without keeping the ﬁnal goal in mind, may drift away from being ultimately successful.

Original: This is equivalent to the scenario that without the long-term goal to provide human beings a clear direction and focus down the road, our short-term achievements are not likely to add up to something substantial.
Revised: This is equivalent to the scenario that without the long-term goal to provide human beings a clear direction and focus down the road, our short-term achievements are not likely to add up to something substantial.

Original: Taking the advantages from these two ways, we put forward a system with a higher level of generalization capability upon previous work (especially, Bacon et al.
Revised: Taking the advantages from these two ways, we put forward a system with a higher level of generalization capability upon previous work (especially, Bacon et al.

Original: (2017) and Kulkarni et al.
Revised: (2017) and Kulkarni et al.

Original: and present a novel framework dubbed as ROS-HPL : Robotic Object Search with Hierarchical Policy Learning and Intrinsic-Extrinsic Modeling, to mitigate the aforementioned challenges (see Fig.left).
Revised: and present a novel framework dubbed as ROS-HPL : Robotic Object Search with Hierarchical Policy Learning and Intrinsic-Extrinsic Modeling, to mitigate the aforementioned challenges (see Fig.left).

Original: Our contributions could be summarized in three-fold.
Revised: Our contributions could be summarized in three-fold.

Original: First, we fully utilize the prior knowledge provided by human to make the hierarchy interpretable.
Revised: First, we fully utilize the prior knowledge provided by human to make the hierarchy interpretable.

Original: Our high-level layer plans over a sub-goal space from human knowledge in order to achieve the ﬁnal goal.
Revised: Our high-level layer plans over a sub-goal space from human knowledge in order to achieve the ﬁnal goal.

Original: Second, when a sub-goal is proposed by the high-level layer, the low-level layer learns the action policy by deliberating between achieving the immediate sub-goal and the action that is helpful for achieving the ﬁnal goal.
Revised: Second, when a sub-goal is proposed by the high-level layer, the low-level layer learns the action policy by deliberating between achieving the immediate sub-goal and the action that is helpful for achieving the ﬁnal goal.

Original: Third, we build our framework with a novel intrinsic and extrinsic rewards setting.
Revised: Third, we build our framework with a novel intrinsic and extrinsic rewards setting.

Original: Though the rewards in this paper are designed for object search, the method itself is general and can be applied to other tasks.
Revised: Though the rewards in this paper are designed for object search, the method itself is general and can be applied to other tasks.

Original: To validate ROS-HPL , we conduct extensive sets of experiments on the House3D (Wu et al., 2018) simulation environment, and report the observed results that validate the efﬁciency and efﬁcacy of the proposed system over other state-of-the-art systems for object search.
Revised: To validate ROS-HPL , we conduct extensive sets of experiments on the House3D (Wu et al., 2018) simulation environment, and report the observed results that validate the efﬁciency and efﬁcacy of the proposed system over other state-of-the-art systems for object search.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: First, we deﬁne the robotic object search (ROS) task.
Revised: First, we deﬁne the robotic object search (ROS) task.

Original: Formally speaking, when a target object is speciﬁed and provided with an image, the robot is asked to search and approach the object from its random starting position.
Revised: Formally speaking, when a target object is speciﬁed and provided with an image, the robot is asked to search and approach the object from its random starting position.

Original: The RGB image from the robot’s on-board camera is the only source of information for decision making.
Revised: The RGB image from the robot’s on-board camera is the only source of information for decision making.

Original: None of the environment information, such as the map of the environment or the location of the target object could be accessed.
Revised: None of the environment information, such as the map of the environment or the location of the target object could be accessed.

Original: Once the area of the target object in the robot’s viewpoint (the image captured by its camera) is larger than a predeﬁned threshold, the agent stops and we consider it as a success.
Revised: Once the area of the target object in the robot’s viewpoint (the image captured by its camera) is larger than a predeﬁned threshold, the agent stops and we consider it as a success.

Original: In this work, we present a novel two-layer hierarchical policy for the robot to perform the object search task, motivated by how human beings  conduct object search.
Revised: In this work, we present a novel two-layer hierarchical policy for the robot to perform the object search task, motivated by how human beings typically conduct object search.

Original: In the following sections, we ﬁrst describe the hierarchy of policies.
Revised: In the following sections, we ﬁrst describe the hierarchy of policies.

Original: Then we introduce two kinds of reward functions, i.e. extrinsic rewards and intrinsic rewards, and we make use of these two reward functions to formulate the solution.
Revised: Then we introduce two kinds of reward functions, i.e. extrinsic rewards and intrinsic rewards, and we make use of these two reward functions to formulate the solution.

Original: Finally, we describe the network architecture adopted for learning the two-layer hierarchical policy.
Revised: Finally, we describe the network architecture adopted for learning the two-layer hierarchical policy.

Original: 2. H IERARCHY OF P OLICIES
Revised: 2. H IERARCHY OF P OLICIES

Original: Our hierarchical policy has two levels, a high-level policy π h and a low-level policy π l .
Revised: Our hierarchical policy has two levels, a high-level policy π h and a low-level policy π l .

Original: At time step t , the robot takes the image captured by its camera as the current state s t .
Revised: At time step t , the robot takes the image captured by its camera as the current state s t .

Original: Given a target object or goal g , the high-level layer proposes a sub-goal sg t ∼ π h ( sg | s t , g ) and the low-level layer takes over the control.
Revised: Given a target object or goal g , the high-level layer proposes a sub-goal sg t ∼ π h ( sg | s t , g ) and the low-level layer takes over the control.

Original: The low-level layer then draws an atomic action a t ∼ π l ( a | s t , g, sg t ) to perform.
Revised: The low-level layer then draws an atomic action a t ∼ π l ( a | s t , g, sg t ) to perform.

Original: The robot will receive a new image/state s t +1 .
Revised: The robot will receive a new image/state s t +1 .

Original: The low-level layer repeats N t times till 1) the low-level layer terminates itself achieving the termination signal term ( s t + N t , g, sg t ) ; 2) the low-level layer achieves the sub-goal sg t .
Revised: The low-level layer repeats N t times till 1) the low-level layer terminates itself achieving the termination signal term ( s t + N t , g, sg t ) ; 2) the low-level layer achieves the sub-goal sg t .

Original: Either way, the low-level layer terminates at state s t + N t , and then returns the control back to the high-level layer, and the high-level layer proposes another sub-goal.
Revised: Either way, the low-level layer terminates at state s t + N t , and then returns the control back to the high-level layer, and the high-level layer proposes another sub-goal.

Original: This process repeats until 1) the goal g is achieved, i.e. the robot ﬁnds the target object successfully; 2) a predeﬁned maximum number of atomic actions has been performed.
Revised: This process repeats until 1) the goal g is achieved, i.e. the robot ﬁnds the target object successfully; 2) a predeﬁned maximum number of atomic actions has been performed.

Original: For ROS, we deﬁne the sub-goal space as { approach obj | obj is visible in the robot’s current view } .
Revised: For ROS, we deﬁne the sub-goal space as { approach obj | obj is visible in the robot’s current view } .

Original: We argue three reasons for the sub-goal space deﬁnition, a) approaching an object that shows in the robot’s view is a more general and relatively trainable task shown by Ye et al.
Revised: We argue three reasons for the sub-goal space deﬁnition, a) approaching an object that shows in the robot’s view is a more general and relatively trainable task shown by Ye et al.

Original: (2018a).
Revised: (2018a).

Original: It also aligns well with the goal of the hierarchical reinforcement learning by breaking down the task into several easier sub-tasks; b) approaching a related object may increase the probability of seeing the target object.
Revised: It also aligns well with the goal of the hierarchical reinforcement learning by breaking down the task into several easier sub-tasks; b) approaching a related object may increase the probability of seeing the target object.

Original: As soon as the target object is captured in the robot’s current view, the task becomes an object approaching task; c) as also suggested by Kulkarni et al.
Revised: As soon as the target object is captured in the robot’s current view, the task becomes an object approaching task; c) as also suggested by Kulkarni et al.

Original: (2016), specifying sub-goals over entities and relations can provide an efﬁcient space for exploration in a complex environment.
Revised: (2016), specifying sub-goals over entities and relations can provide an efﬁcient space for exploration in a complex environment.

Original: Moreover, in case there is no object visible in the robot’s current view, we supplement one additional candidate sub-goal: non-goal-driven exploration.
Revised: Moreover, in case there is no object visible in the robot’s current view, we supplement one additional candidate sub-goal: non-goal-driven exploration.

Original: The atomic action space for the low-level layer is adopted with navigation in mind, namely { move forward / backward / left / right, turn left / right } .
Revised: The atomic action space for the low-level layer is adopted with navigation in mind, namely { move forward / backward / left / right, turn left / right } .

Original: 2. E XTRINSIC R EWARDS AND I NTRINSIC R EWARDS
Revised: 2. E XTRINSIC R EWARDS AND I NTRINSIC R EWARDS

Original: We deﬁne two kinds of reward functions.
Revised: We deﬁne two kinds of reward functions.

Original: The extrinsic rewards r e are deﬁned for our object search task, thus are goal dependent.
Revised: The extrinsic rewards r e are deﬁned for our object search task, thus are goal dependent.

Original: Further, we also introduce the intrinsic rewards r
Revised: Further, we also introduce the intrinsic rewards r

Original: i for the low-level sub-tasks.
Revised: i for the low-level sub-tasks.

Original: The intrinsic rewards are hereby sub-goal dependent.
Revised: The intrinsic rewards are hereby sub-goal dependent.

Original: We specify the two reward functions respectively as follows.
Revised: We specify the two reward functions respectively as follows.

Original: Extrinsic rewards r e .
Revised: Extrinsic rewards r e .

Original: Without loss of generality, to encourage the robot to ﬁnish the object search task, we provide a high extrinsic reward (in practice, 100 ) when the robot reaches the ﬁnal goal state.
Revised: Without loss of generality, to encourage the robot to ﬁnish the object search task, we provide a high extrinsic reward (in practice, 100 ) when the robot reaches the ﬁnal goal state.

Original: At all other intermediate states, the extrinsic rewards are set to 0 .
Revised: At all other intermediate states, the extrinsic rewards are set to 0 .

Original: Formally, r et ( s t − 1 , a t − 1 , s t , g ) =
Revised: Formally, r et ( s t − 1 , a t − 1 , s t , g ) =

Original: 100 if and only if s t is a goal state, otherwise r et ( s t − 1 , a t − 1 , s t , g ) = 0 .
Revised: 100 if and only if s t is a goal state, otherwise r et ( s t − 1 , a t − 1 , s t , g ) = 0 .

Original: Intrinsic rewards r i .
Revised: Intrinsic rewards r i .

Original: To facilitate the robot perform the sub-task, i.e. approaching the object speciﬁed in the sub-goal sg which shows in the robot’s current view, we adopt the following reward function (Ye et al., 2018a) as the intrinsic rewards.
Revised: To facilitate the robot perform the sub-task, i.e. approaching the object speciﬁed in the sub-goal sg which shows in the robot’s current view, we adopt the following reward function (Ye et al., 2018a) as the intrinsic rewards.

Original: To be speciﬁc, let ar t be the area of the object in robot’s view at time step t (known as s t ), the intrinsic reward r it ( s t − 1 , a t − 1 , s t , sg ) =
Revised: To be speciﬁc, let ar t be the area of the object in robot’s view at time step t (known as s t ), the intrinsic reward r it ( s t − 1 , a t − 1 , s t , sg ) =

Original: ar t
Revised: ar t

Original: if and only if ar t > ar t − 1 , ar t − 2 , ...ar 0 , otherwise r it ( s t − 1 , a t − 1 , s t , sg )
Revised: if and only if ar t > ar t − 1 , ar t − 2 , ...ar 0 , otherwise r it ( s t − 1 , a t − 1 , s t , sg )

Original: = 0 .
Revised: = 0 .

Original: Note that ar t can be easily calculated based on the robot’s detection outputs, e.g. the size of the detected bounding box.
Revised: Note that ar t can be easily calculated based on the robot’s detection outputs, e.g. the size of the detected bounding box.

Original: In practice, we normalize ar t to the range of [0 , 100] .
Revised: In practice, we normalize ar t to the range of [0 , 100] .

Original: 2. M ODEL F ORMULATION
Revised: 2. M ODEL F ORMULATION

Original: We formulate the ROS task in terms of the two rewards introduced in Sec. 2.2.
Revised: We formulate the ROS task in terms of the two rewards introduced in Sec. 2.2.

Original: When the robot starts from an initial state s 0 , it proposes a sub-goal sg 0 aiming to achieve the ﬁnal goal
Revised: When the robot starts from an initial state s 0 , it proposes a sub-goal sg 0 aiming to achieve the ﬁnal goal

Original: g (locating and approaching the target object).
Revised: g (locating and approaching the target object).

Original: To achieve the ﬁnal goal, we can optimize the discounted cumulative extrinsic rewards, expected over all trajectories starting at state s 0 and sub-goal sg 0 , which is E [ (cid:80) ∞ t =

Revised: To achieve the ﬁnal goal, we can optimize the discounted cumulative extrinsic rewards, expected over all trajectories starting at state s 0 and sub-goal sg 0 , which is E [ (cid:80) ∞ t =

Original: γ t r et +1 | s 0 , g, sg 0 ] .
Revised: γ t r et +1 | s 0 , g, sg 0 ] .

Original: If and only if the robot takes minimal steps to the goal state, the discounted cumulative extrinsic rewards are thus maximized.
Revised: If and only if the robot takes minimal steps to the goal state, the discounted cumulative extrinsic rewards are thus maximized.

Original: The discounted cumulative extrinsic rewards is also known as the state action value Q eh (Sutton &
Revised: The discounted cumulative extrinsic rewards is also known as the state action value Q eh (Sutton &

Original: Barto, 2018) for our high-level layer, i.e. E [ (cid:80) ∞ t =0 γ t r e t +1 | s 

Revised: Barto, 2018) for our high-level layer, i.e. E [ (cid:80) ∞ t =0 γ t r e t +1 | s

Original: = s, g = g, sg 0 =
Revised: = s, g = g, sg 0 =

Original: sg ] =
Revised: sg ] =

Original: Q e h ( s, g, sg ) .
Revised: Q e h ( s, g, sg ) .

Original: Following the option-critic framework (Bacon et al., 2017), we unroll the Q eh
Revised: Following the option-critic framework (Bacon et al., 2017), we unroll the Q eh

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: ) as,
Revised: ) as,

Original: [Equation]
Revised: [Equation]

Original: where the state action value Q el ( s, g, sg, a ) for our low-level layer is the discounted cumulative extrinsic rewards after taking action a under the state s , goal g and sub-goal sg .
Revised: where the state action value Q el ( s, g, sg, a ) for our low-level layer is the discounted cumulative extrinsic rewards after taking action a under the state s , goal g and sub-goal sg .

Original: Given the transition probability P ( s (cid:48) | s, a ) which denotes the probability of being state s (cid:48) after taking action a at state s ,
Revised: Given the transition probability P ( s (cid:48) | s, a ) which denotes the probability of being state s (cid:48) after taking action a at state s ,

Original: Q el ( s, g, sg, a ) can be further formulated as,
Revised: Q el ( s, g, sg, a ) can be further formulated as,

Original: [Equation]
Revised: [Equation]

Original: We parameterize π l ( a | s, g, sg ) and term ( s, g, sg ) as θ π l and θ t respectively.
Revised: We parameterize π l ( a | s, g, sg ) and term ( s, g, sg ) as θ π l and θ t respectively.

Original: According to Bacon et al.
Revised: According to Bacon et al.

Original: (2017), θ π l and θ t can be updated by Equation 3 and 4 to optimize Q e h ( s, g, sg ) .
Revised: (2017), θ π l and θ t can be updated by Equation 3 and 4 to optimize Q e h ( s, g, sg ) .

Original: We refer interested readers to Bacon et al.
Revised: We refer interested readers to Bacon et al.

Original: (2017) for more details.
Revised: (2017) for more details.

Original: [Equation]
Revised: [Equation]

Original: In addition, we adopt Q-leaning method (Mnih et al., 2015) to learn Q eh ( s, g, sg ) and generate subgoal sg = arg max sg Q eh ( s, g, sg ) .
Revised: In addition, we adopt Q-learning method (Mnih et al., 2015) to learn Q eh ( s, g, sg ) and generate subgoal sg = arg max sg Q eh ( s, g, sg ) .

Original: We parameterize Q eh ( s, g, sg ) with θ he and update its value towards the 1 -step extrinsic return R e 1 =
Revised: We parameterize Q eh ( s, g, sg ) with θ he and update its value towards the 1 -step extrinsic return R e 1 =

Original: r e ( s, a, s (cid:48) , g )
Revised: r e ( s, a, s (cid:48) , g )

Original: + γU
Revised: + γU

Original: θ he ( g, sg, s (cid:48) ) , and consequently θ he can be updated as follows.
Revised: θ he ( g, sg, s (cid:48) ) , and consequently θ he can be updated as follows.

Original: [Equation]
Revised: [Equation]

Original: For the low-level policy layer, when we build the hierarchical policy from the sub-goal space, we inherently assume that achieving the sub-goal is helpful for achieving the ﬁnal goal.
Revised: For the low-level policy layer, when we build the hierarchical policy from the sub-goal space, we inherently assume that achieving the sub-goal is helpful for achieving the ﬁnal goal.

Original: We deﬁne the sub-goal as approaching a related object by doing so the robot is more likely to observe the target one.
Revised: We deﬁne the sub-goal as approaching a related object by doing so the robot is more likely to observe the target one.

Original: Without loss of generality, we introduce α ∈ [0 , 1] as the proportion of how much effort the low-level layer puts in achieving the sub-goal.
Revised: Without loss of generality, we introduce α ∈ [0 , 1] as the proportion of how much effort the low-level layer puts in achieving the sub-goal.

Original: Meanwhile, the low-level layer keeps (1 − α ) of its effort for achieving the ﬁnal goal, which yields a α mixed overall loss.
Revised: Meanwhile, the low-level layer keeps (1 − α ) of its effort for achieving the ﬁnal goal, which yields a α mixed overall loss.

Original: Similarly, to learn the desired policy for the low-level layer, we optimize the discounted cumulative hybrid rewards, E [ (cid:80) ∞ t =

Revised: Similarly, to learn the desired policy for the low-level layer, we optimize the discounted cumulative hybrid rewards, E [ (cid:80) ∞ t =

Original: γ t ((1 − α )
Revised: γ t ((1 − α )

Original: r et +

Revised: r et +

Original: + αr it +1 ) | s 0 , g, sg 0 , a 0 ] .
Revised: + αr it +1 ) | s 0 , g, sg 0 , a 0 ] .

Original: We denote it as Q l ( s, g, sg, a ) , and further represent it by Q el ( s, g, sg, a ) and Q il ( s, g, sg, a ) as in Equation 6.
Revised: We denote it as Q l ( s, g, sg, a ) , and further represent it by Q el ( s, g, sg, a ) and Q il ( s, g, sg, a ) as in Equation 6.

Original: [Equation]
Revised: [Equation]

Original: As a result, to optimize Q l ( s, g, sg, a ) , we can optimize Q el ( s, g, sg, a ) and Q il ( s, g, sg, a ) one by one.
Revised: As a result, to optimize Q l ( s, g, sg, a ) , we can optimize Q el ( s, g, sg, a ) and Q il ( s, g, sg, a ) one by one.

Original: With the policy gradient method, θ π l can be updated using Equation 7 and Equation 8 respectively (Mnih et al., 2016).
Revised: With the policy gradient method, θ π l can be updated using Equation 7 and Equation 8 respectively (Mnih et al., 2016).

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that V el ( s, g, sg )
Revised: Note that V el ( s, g, sg )

Original: =
Revised: =

Original: (cid:80) a π l ( a | s, g, sg ) Q el ( s, g, sg, a ) by deﬁnition, therefore V el ( s, g, sg ) =
Revised: (cid:80) a π l ( a | s, g, sg ) Q el ( s, g, sg, a ) by deﬁnition, therefore V el ( s, g, sg ) =

Original: Q eh ( s, g, sg ) according to Equation 1.
Revised: Q eh ( s, g, sg ) according to Equation 1.

Original: Thus, the update of θ π l in Equation 7 for optimizing
Revised: Thus, the update of θ π l in Equation 7 for optimizing

Original: Q el ( s, g, sg, a ) is exactly the same as that in Equation 3 for optimizing Q eh ( s, g, sg ) .
Revised: Q el ( s, g, sg, a ) is exactly the same as that in Equation 3 for optimizing Q eh ( s, g, sg ) .

Original: At last, we combine the two updates of θ π l according to Equation 6.
Revised: At last, we combine the two updates of θ π l according to Equation 6.

Original: Equation 9 gives the ﬁnal update.
Revised: Equation 9 gives the ﬁnal update.

Original: [Equation]
Revised: [Equation]

Original: To conclude, we adopt Equation 4, 5 and 9 to update θ t , θ he and θ π l for learning term ( s, g, sg ) ,
Revised: To conclude, we adopt Equation 4, 5 and 9 to update θ t , θ he and θ π l for learning term ( s, g, sg ) ,

Original: Q eh ( s, g, sg ) and π l ( a | s, g, sg ) respectively.
Revised: Q eh ( s, g, sg ) and π l ( a | s, g, sg ) respectively.

Original: 2. N ETWORK A RCHITECTURE
Revised: 2. N ETWORK A RCHITECTURE

Original: Since the image captured by the robot’s camera serves as the robot’s current state, we adopt deep neural networks to handle this high dimensional inputs and to approximate term ( s, g, sg ) , Q eh ( s, g, sg ) and π l ( a | s, g, sg ) .
Revised: Since the image captured by the robot’s camera serves as the robot’s current state, we adopt deep neural networks to handle this high dimensional inputs and to approximate term ( s, g, sg ) , Q eh ( s, g, sg ) and π l ( a | s, g, sg ) .

Original: To update the model parameters according to Equation 4, 5 and 9, we still need the value of V e h ( s, g ) , Q e l ( s, g, sg, a ) , Q i l
Revised: To update the model parameters according to Equation 4, 5 and 9, we still need the value of V e h ( s, g ) , Q e l ( s, g, sg, a ) , Q i l

Original: ( s, g, sg, a ) and V
Revised: ( s, g, sg, a ) and V

Original: i l ( s, g, sg ) .
Revised: i l ( s, g, sg ) .

Original: Here, we use deep neural networks to approximate V il ( s, g, sg ) , and estimate the remaining (Equation 10).
Revised: Here, we use deep neural networks to approximate V il ( s, g, sg ) , and estimate the remaining (Equation 10).

Original: [Equation]
Revised: [Equation]

Original: Fig.
Revised: Fig.

Original: 2 illustrates our network architecture.
Revised: 2 illustrates our network architecture.

Original: We ﬁrst extract multiple feature representations from the encoder-decoder network (shown in Fig.
Revised: We ﬁrst extract multiple feature representations from the encoder-decoder network (shown in Fig.

Original: 2 upper right corner) for the image inputs.
Revised: 2 upper right corner) for the image inputs.

Original: Our highlevel network takes the outputs from the encoder as the feature representations of the robot’s current view and the target object image accordingly.
Revised: Our highlevel network takes the outputs from the encoder as the feature representations of the robot’s current view and the target object image accordingly.

Original: The two feature representations are then fed into two different fully connected layers respectively and the outputs are then concatenated into a joint vector before attaching another fully connected layer to generate an embedding fusion.
Revised: The two feature representations are then fed into two different fully connected layers respectively and the outputs are then concatenated into a joint vector before attaching another fully connected layer to generate an embedding fusion.

Original: We feed the embedding fusion into one additional fully connected layer to approximate Q eh ( s, g, sg ) .
Revised: We feed the embedding fusion into one additional fully connected layer to approximate Q eh ( s, g, sg ) .

Original: For low-level model, we take the outputs of the two decoders (a semantic segmentation decoder and a depth prediction decoder) as the feature representations of the robot’s current view.
Revised: For low-level model, we take the outputs of the two decoders (a semantic segmentation decoder and a depth prediction decoder) as the feature representations of the robot’s current view.

Original: Based on the goal (i.e. the target object) and the sub-goal proposed by the high-level network, we generate two attention masks using the semantic segmentation module.
Revised: Based on the goal (i.e. the target object) and the sub-goal proposed by the high-level network, we generate two attention masks using the semantic segmentation module.

Original: Note that if the sub-goal is proposed as the candidate non-goal-driven exploration, there is no sub-goal attention in hope that the robot can learn to explore.
Revised: Note that if the sub-goal is proposed as the candidate non-goal-driven exploration, there is no sub-goal attention in hope that the robot can learn to explore.

Original: We take the two attention masks, as well as the depth map as the inputs to our low-level network, which shares a similar architecture as our high-level network except that the low-level network has three branches attached after the embedding fusion for approximating π l ( a | s, g, sg ) ,
Revised: We take the two attention masks, as well as the depth map as the inputs to our low-level network, which shares a similar architecture as our high-level network except that the low-level network has three branches attached after the embedding fusion for approximating π l ( a | s, g, sg ) ,

Original: V il ( s, g, sg ) and term ( s, g, sg ) respectively.
Revised: V il ( s, g, sg ) and term ( s, g, sg ) respectively.

Original: Each branch consists of two fully connected layers.
Revised: Each branch consists of two fully connected layers.

Original: We follow Equation 4, 5 and 9 to learn term ( s, g, sg ) , Q eh ( s, g, sg ) and π l ( a | s, g, sg ) .
Revised: We follow Equation 4, 5 and 9 to learn term ( s, g, sg ) , Q eh ( s, g, sg ) and π l ( a | s, g, sg ) .

Original: In addition, to learn V il ( s, g, sg ) , we update the estimated value towards the 1 -step intrinsic return
Revised: In addition, to learn V il ( s, g, sg ) , we update the estimated value towards the 1 -step intrinsic return

Original: R i 1 =
Revised: R i 1 =

Original: r i ( s, a, s (cid:48) , sg ) + γV θ li ( s (cid:48) , g, sg ) .
Revised: r i ( s, a, s (cid:48) , sg ) + γV θ li ( s (cid:48) , g, sg ) .

Original: As a result, the parameter θ li could then be updated as, θ li ← θ li − ∇ θ li [ R i 1 − V θ li ( s, g, sg )] 2 .
Revised: As a result, the parameter θ li could then be updated as, θ li ← θ li − ∇ θ li [ R i 1 − V θ li ( s, g, sg )] 2 .

Original:  E XPERIMENTS 3. D ATASET
Revised:  E XPERIMENTS 3. D ATASET

Original: We validate our framework on the simulation platform House3D (Wu et al., 2018).
Revised: We validate our framework on the simulation platform House3D (Wu et al., 2018).

Original: House3D consists of rich indoor environments with diverse layouts for a virtual robot to navigate.
Revised: House3D consists of rich indoor environments with diverse layouts for a virtual robot to navigate.

Original: In each indoor environment, a variety of objects are scattered at many locations, such as television, sofa, desk .
Revised: In each indoor environment, a variety of objects are scattered at many locations, such as television, sofa, desk .

Original: While navigating, the robot has a ﬁrst-person view RGB image as its observation.
Revised: While navigating, the robot has a ﬁrst-person view RGB image as its observation.

Original: The simulator also provides the robot with the ground truth semantic segmentation and depth map corresponding to the RGB image.
Revised: The simulator also provides the robot with the ground truth semantic segmentation and depth map corresponding to the RGB image.

Original: The RGB images, as well as the semantic segmentation and depth maps can be used as the training data to learn the encoder-decoder network (shown in Fig.
Revised: The RGB images, as well as the semantic segmentation and depth maps can be used as the training data to learn the encoder-decoder network (shown in Fig.

Original: 2 upper right corner) for the feature representation extraction as we mentioned in Sec. 2.4.
Revised: 2 upper right corner) for the feature representation extraction as we mentioned in Sec. 2.4.

Original: In addition, the trained model, speciﬁcally the semantic segmentation prediction, can be used as the robot’s detection system.
Revised: In addition, the trained model, speciﬁcally the semantic segmentation prediction, can be used as the robot’s detection system.

Original: To validate our proposed method in learning hierarchical policy for object search, we conduct the experiments in indoor environments where the objects’ placements are in accordance with the realworld scenario.
Revised: To validate our proposed method in learning hierarchical policy for object search, we conduct the experiments in indoor environments where the objects’ placements are in accordance with the realworld scenario.

Original: For example, the television is placed close to the sofa , and is likely occluded by the sofa at many viewpoints.
Revised: For example, the television is placed close to the sofa , and is likely occluded by the sofa at many viewpoints.

Original: In such a way, to search the target object television , the robot could approach sofa ﬁrst to increase the likelihood of seeing the television .
Revised: In such a way, to search the target object television , the robot could approach sofa ﬁrst to increase the likelihood of seeing the television .

Original: We consider discrete actions for the robot to navigate in this environment.
Revised: We consider discrete actions for the robot to navigate in this environment.

Original: Speciﬁcally, the robot moves forward / backward / left / right 0 .
Revised: Speciﬁcally, the robot moves forward / backward / left / right 0 .

Original: 2 meters, or rotates 90 degrees every time.
Revised: 2 meters, or rotates 90 degrees every time.

Original: It also discretizes each environment into a certain number of reachable locations, as shown in Fig.
Revised: It also discretizes each environment into a certain number of reachable locations, as shown in Fig.

Original: 3. E XPERIMENTAL S ETTING
Revised: 3. E XPERIMENTAL S ETTING

Original: We compare the following methods and variants:
Revised: We compare the following methods and variants:

Original: R ANDOM .
Revised: R ANDOM .

Original: At each time step, the robot ignores its observation and performs a random action.
Revised: At each time step, the robot ignores its observation and performs a random action.

Original: H IGH - LEVEL ONLY .
Revised: H IGH - LEVEL ONLY .

Original: We remove our low-level network and train the high-level network to predict the state action value Q e ( s, g, a ) with the DQN method (Mnih et al., 2015).
Revised: We remove our low-level network and train the high-level network to predict the state action value Q e ( s, g, a ) with the DQN method (Mnih et al., 2015).

Original: L OW - LEVEL ONLY .
Revised: L OW - LEVEL ONLY .

Original: Only the low-level network is trained to generate an action policy for object search without hierarchy.
Revised: Only the low-level network is trained to generate an action policy for object search without hierarchy.

Original: Since there is no sub-goal provided to the low-level network, we further block the sub-goal input channel.
Revised: Since there is no sub-goal provided to the low-level network, we further block the sub-goal input channel.

Original: A recent work from Ye et al.
Revised: A recent work from Ye et al.

Original: (2018a) shows that it serves as a stronger baseline than Zhu et al.
Revised: (2018a) shows that it serves as a stronger baseline than Zhu et al.

Original: O PTION -C RITIC (Bacon et al., 2017).
Revised: O PTION -C RITIC (Bacon et al., 2017).

Original: This method aims to build the hierarchy automatically.
Revised: This method aims to build the hierarchy automatically.

Original: It learns the desired policy by maximizing the discounted cumulative extrinsic rewards, while no intrinsic rewards are involved.
Revised: It learns the desired policy by maximizing the discounted cumulative extrinsic rewards, while no intrinsic rewards are involved.

Original: When the high-level network proposes a sub-goal (option), the corresponding option policy is learned by maximizing the extrinsic rewards and acting towards the goal.
Revised: When the high-level network proposes a sub-goal (option), the corresponding option policy is learned by maximizing the extrinsic rewards and acting towards the goal.

Original: The option policy stops according to the predicted termination signal.
Revised: The option policy stops according to the predicted termination signal.

Original: Here, the sub-goal doesn’t have much interpretability, it can be seen as a temporal abstraction.
Revised: Here, the sub-goal doesn’t have much interpretability, it can be seen as a temporal abstraction.

Original: H RL (Kulkarni et al., 2016).
Revised: H RL (Kulkarni et al., 2016).

Original: Different from O PTION -C RITIC , this method works on both extrinsic rewards and intrinsic rewards.
Revised: Different from O PTION -C RITIC , this method works on both extrinsic rewards and intrinsic rewards.

Original: The high-level network maximizes the discounted cumulative extrinsic rewards to reach the goal, while the low-level network maximizes the discounted cumulative intrinsic rewards to achieve the sub-goal that is proposed by the high-level network.
Revised: The high-level network maximizes the discounted cumulative extrinsic rewards to reach the goal, while the low-level network maximizes the discounted cumulative intrinsic rewards to achieve the sub-goal that is proposed by the high-level network.

Original: The low-level network stops either when the sub-goal is achieved, or a predeﬁned maximum number of steps have been performed.
Revised: The low-level network stops either when the sub-goal is achieved, or a predeﬁned maximum number of steps have been performed.

Original: H RL WITH STOP (Das et al., 2018b).
Revised: H RL WITH STOP (Das et al., 2018b).

Original: This method is similar to H RL , while the low-level network has an additional action stop in its action space.
Revised: This method is similar to H RL , while the low-level network has an additional action stop in its action space.

Original: When the low-level network produces the stop action, it stops and returns the control to the high-level network.
Revised: When the low-level network produces the stop action, it stops and returns the control to the high-level network.

Original: Otherwise, the low-level network continues until it has performed a predeﬁned maximum number of steps.
Revised: Otherwise, the low-level network continues until it has performed a predeﬁned maximum number of steps.

Original: Note that in Das et al.
Revised: Note that in Das et al.

Original: (2018b), the authors adopted an imitation learning method to warm-start the hierarchical policy learning.
Revised: (2018b), the authors adopted an imitation learning method to warm-start the hierarchical policy learning.

Original: For fair comparison, we don’t provide any imitation signals in our experiments, and we learn the policy purely through the reinforcement learning method.
Revised: For fair comparison, we don’t provide any imitation signals in our experiments, and we learn the policy purely through the reinforcement learning method.

Original: O UR M ETHOD follows Sec 2.
Revised: O UR M ETHOD follows Sec 2.

Original: To identify the role of the α value, we compare multiple variants of our method, namely setting α = 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 respectively, where the higher α value denotes the more effort the low-level network devotes to achieving the sub-goal.
Revised: To identify the role of the α value, we compare multiple variants of our method, namely setting α = 0 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 respectively, where the higher α value denotes the more effort the low-level network devotes to achieving the sub-goal.

Original: It is also worth to mention that when setting α = 0 , our method shares a similar idea with O PTION -C RITIC .
Revised: It is also worth to mention that when setting α = 0 , our method shares a similar idea with O PTION -C RITIC .

Original: However, our low-level network still learns to approximate the state value V il ( s, g, sg ) which is supervised by the intrinsic rewards.
Revised: However, our low-level network still learns to approximate the state value V il ( s, g, sg ) which is supervised by the intrinsic rewards.

Original: When setting α = 1 , our method is similar to H RL .
Revised: When setting α = 1 , our method is similar to H RL .

Original: Yet our low-level network learns a termination signal term ( s, g, sg ) , which is supervised by the extrinsic rewards.
Revised: Yet our low-level network learns a termination signal term ( s, g, sg ) , which is supervised by the extrinsic rewards.

Original: Experiments are conducted under two settings:
Revised: Experiments are conducted under two settings:

Original: S ETTING -A. To compare the efﬁcacy in learning the object search policy, we train each method to learn a policy for searching a speciﬁc target object.
Revised: S ETTING -STSE (Singe Target in Single Environment). To compare the efﬁcacy in learning the object search policy, we train each method to learn a policy for searching a speciﬁc target object.

Original: Without loss of generality, we randomly choose 1 target object from a speciﬁc environment.
Revised: Without loss of generality, we randomly choose 1 target object from a speciﬁc environment.

Original: We evaluate each method in terms of the network for searching the speciﬁc target object.
Revised: We evaluate each method in terms of the network for searching the speciﬁc target object.

Original: S ETTING -B.
Revised: S ETTING -MTME (Multiple Targets in Multiple Environments).

Original: To further evaluate, we train every method under different environments to search different target objects.
Revised: To further evaluate, we train every method under different environments to search different target objects.

Original: To be speciﬁc, we randomly choose 24 target objects in 4 different environments ( 6 each) as our training data.
Revised: To be speciﬁc, we randomly choose 24 target objects in 4 different environments ( 6 each) as our training data.

Original: We evaluate each method in terms of the network with thedifferent environment-target object pairs being taken as the inputs randomly.
Revised: We evaluate each method in terms of the network with the 24 different environment-target object pairs being taken as the inputs randomly.

Original: For each experiment, we periodically increase the distance between the target object and the robot’s starting position during training time which is also known as the curriculum learning paradigm.
Revised: For each experiment, we periodically increase the distance between the target object and the robot’s starting position during training time which is also known as the curriculum learning paradigm.

Original: During testing time, we randomly choose 100 starting positions.
Revised: During testing time, we randomly choose 100 starting positions.

Original: We set the maximum number of all atomic action steps to 1000 .
Revised: We set the maximum number of all atomic action steps to 1000 .

Original: For method H RL and H RL WITH STOP , we set the maximum number of each low-level steps to 50 to ensure the proposed sub-goals can be achieved.
Revised: For method H RL and H RL WITH STOP , we set the maximum number of each low-level steps to 50 to ensure the proposed sub-goals can be achieved.

Original: The robot stops either it reaches the goal state (success case) or it runs out of the maximum number of all atomic action steps (failure case).
Revised: The robot stops either it reaches the goal state (success case) or it runs out of the maximum number of all atomic action steps (failure case).

Original: We implement our algorithm using Tensorﬂow toolbox and conduct all the experiments with Nvidia V100 GPUs and 16 Intel Xeon E5-2680 v4 CPU cores.
Revised: We implement our algorithm using Tensorﬂow toolbox and conduct all the experiments with Nvidia V100 GPUs and 16 Intel Xeon E5-2680 v4 CPU cores.

Original: 3. E XPERIMENTAL R ESULTS AND D ISCUSSION
Revised: 3. E XPERIMENTAL R ESULTS AND D ISCUSSION

Original: Since we formulate the object search problem as maximizing the discounted cumulative extrinsic rewards, we take the Average discounted cumulative extrinsic Rewards (AR) as one of the evaluation metrics, calculated by:[Equation] i = ∞ (cid:88) t = [Equation]
Revised: Since we formulate the object search problem as maximizing the discounted cumulative extrinsic rewards, we take the Average discounted cumulative extrinsic Rewards (AR) as one of the evaluation metrics, calculated by:

Original: et

Original: [Equation]
Revised: [Equation]

Original: N N (cid:88) i = 1 ( success ) [Equation] 100 , (12) where γ ∈ (0 , 1] is the discount factor.
Revised: where γ ∈ (0 , 1] is the discount factor.

Original: From the perspective of the evaluation metric, it can also be seen as a trade-off between the success rate metric and the average steps metric.
Revised: From the perspective of the evaluation metric, it can also be seen as a trade-off between the success rate metric and the average steps metric.

Original: With the higher
Revised: With the higher

Original: [Figure]
Revised: [Figure]

Original: value of γ , the average steps metric weighs more, and vice versa.
Revised: value of γ , the average steps metric weighs more, and vice versa.

Original: In our experiments, we set γ = 0 .
Revised: In our experiments, we set γ = 0 .

Original: In addition, we also report the following widely used evaluation metrics.
Revised: In addition, we also report the following widely used evaluation metrics.

Original: Success Rate (SR).
Revised: Success Rate (SR).

Original: Average
Revised: Average

Original: Steps over all successful cases (AS).
Revised: Steps over all successful cases (AS).

Original: Success weighted by inverse Path Length (SPL) (Anderson et al., 2018a), which is calculated as 1 N (cid:80) Ni 
Revised: Success weighted by inverse Path Length (SPL) (Anderson et al., 2018a), which is calculated as 1 N (cid:80) Ni =1 S

Original: =1 S i 
Revised: i l

Original: [Figure] l i ,p i ) .
Revised: i max ( l i ,p i ) .

Original: Here, S i is the binary indicator of success in episode i , l i and p i are the lengths of the shortest path and the path actually taken by the robot.
Revised: Here, S i is the binary indicator of success in episode i , l i and p i are the lengths of the shortest path and the path actually taken by the robot.

Original: We adopt the number of the action steps as the path length.
Revised: We adopt the number of the action steps as the path length.

Original: As a result, SPL also trades-off success rate against average steps.
Revised: As a result, SPL also trades-off success rate against average steps.

Original: Fig.
Revised: Fig.

Original: 4 reports the training performance (i.e. AR and SPL) of our method with different α values under S ETTING -A. From Fig.
Revised: 4 reports the training performance (i.e. AR and SPL) of our method with different α values under S ETTING -STSE. From Fig.

Original: 4, we observe that the model with α = 0 .
Revised: 4, we observe that the model with α = 0 .

Original: 8 achieves much better performance, indicating that a proper trade-off between achieving the goal and the sub-goal is needed for our hierarchical method to perform the object search well.
Revised: 8 achieves much better performance, indicating that a proper trade-off between achieving the goal and the sub-goal is needed for our hierarchical method to perform the object search well.

Original: When α = 0 , the model performs worst.
Revised: When α = 0 , the model performs worst.

Original: It is because that the low-level layer needs to choose the actions that are suited for achieving the goal, and as a result the learning process still suffers from the sparse extrinsic rewards setting.
Revised: It is because that the low-level layer needs to choose the actions that are suited for achieving the goal, and as a result the learning process still suffers from the sparse extrinsic rewards setting.

Original: When α = 1 , the model performs much better at the earlier training stage where the target object is closer to robot’s starting position.
Revised: When α = 1 , the model performs much better at the earlier training stage where the target object is closer to robot’s starting position.

Original: We explain it as when α = 1 , the navigation actions generated by the low-level network are fully driven by the intrinsic rewards in order to achieve the sub-goals.
Revised: We explain it as when α = 1 , the navigation actions generated by the low-level network are fully driven by the intrinsic rewards in order to achieve the sub-goals.

Original: When the target object is nearby, the proposed sub-goals are more likely to be consistent with the ﬁnal goal and thus achieving the sub-goals is helpful for achieving the goal.
Revised: When the target object is nearby, the proposed sub-goals are more likely to be consistent with the ﬁnal goal and thus achieving the sub-goals is helpful for achieving the goal.

Original: While the target object is far away, the proposed sub-goals are much more noisy.
Revised: While the target object is far away, the proposed sub-goals are much more noisy.

Original: Then focus only on achieving the sub-goals without keeping the ﬁnal goal in mind may hurt the performance.
Revised: Then focus only on achieving the sub-goals without keeping the ﬁnal goal in mind may hurt the performance.

Original: [Table]
Revised: [Table]

Original: We set α = 0 .
Revised: We set α = 0 .

Original: 8 in the following experiments.
Revised: 8 in the following experiments.

Original: Table 1 shows the comparison of all methods under both settings.
Revised: Table 1 shows the comparison of all methods under both settings.

Original: Table 1 (left) shows that under S ETTING -A, our method ( α = 0 . 8 ) outperforms all previous methods in terms of SR, SPL, AR, and fairly in terms of AS since H IGHLEVEL ONLY method doesn’t perform well in general as its low SR, SPL and AR suggest.
Revised: Table 1 (left) shows that under S ETTING -STSE, our method ( α = 0 . 8 ) outperforms all previous methods in terms of SR, SPL, AR, and fairly in terms of AS since H IGHLEVEL ONLY method doesn’t perform well in general as its low SR, SPL and AR suggest.

Original: Moreover, we also observe that O PTION -C RITIC doesn’t perform well as other hierarchical methods (H RL , H RL WITH
Revised: Moreover, we also observe that O PTION -C RITIC doesn’t perform well as other hierarchical methods (H RL , H RL WITH

Original: STOP and our method), indicating that the prior knowledge provided by human is vital in guiding the robot to perform the challenge tasks.
Revised: STOP and our method), indicating that the prior knowledge provided by human is vital in guiding the robot to perform the challenge tasks.

Original: Compared to H RL WITH STOP , H RL doesn’t stop the robot at a more valuable state timely and for which the performance is worse.
Revised: Compared to H RL WITH STOP , H RL doesn’t stop the robot at a more valuable state timely and for which the performance is worse.

Original: However, simply add a stop action in the atomic action space is not much efﬁcient when compared with our method.
Revised: However, simply add a stop action in the atomic action space is not much efﬁcient when compared with our method.

Original: Under the more challenge S ETTING -B, though our method has a slightly lower SR and higher AS from Table(right), our method achieves higher SPL and AR that both trade-off the SR against AS, which also fairly demonstrates the superiority of our method.
Revised: Under the more challenge S ETTING -MTME, though our method has a slightly lower SR and higher AS from Table 1 (right), our method achieves higher SPL and AR that both trade-off the SR against AS, which also fairly demonstrates the superiority of our method.

Original: We further depict sample qualitative results in Fig.
Revised: We further depict sample qualitative results in Fig.

Original: 1 (right) and Fig. 3, which show that our method yields a more concise trajectory compare to other methods for the ROS task.
Revised: 1 (right) and Fig. 3, which show that our method yields a more concise trajectory compare to other methods for the ROS task.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work is closely related to two major research thrusts: hierarchical reinforcement learning and target-driven visual navigation.
Revised: Our work is closely related to two major research thrusts: hierarchical reinforcement learning and target-driven visual navigation.

Original: Hierarchical reinforcement learning.
Revised: Hierarchical reinforcement learning.

Original: Previous work has studied hierarchical reinforcement learning in many different ways.
Revised: Previous work has studied hierarchical reinforcement learning in many different ways.

Original: One is to come up with efﬁcient methods to accelerate the learning process of the general hierarchical reinforcement learning scheme.
Revised: One is to come up with efﬁcient methods to accelerate the learning process of the general hierarchical reinforcement learning scheme.

Original: As in Nachum et al.
Revised: As in Nachum et al.

Original: (2018b), the authors introduced an off-policy correction method.
Revised: (2018b), the authors introduced an off-policy correction method.

Original: Levy et al.
Revised: Levy et al.

Original: (2017) and Levy et al.
Revised: (2017) and Levy et al.

Original: (2018) proposed to use Hindsight Experience Replay to facilitate learning at multiple time scales.
Revised: (2018) proposed to use Hindsight Experience Replay to facilitate learning at multiple time scales.

Original: Though these methods’ performance are impressive, they typically assume the sub-goal space for the higher level policy is a subspace of the state space.
Revised: Though these methods’ performance are impressive, they typically assume the sub-goal space for the higher level policy is a subspace of the state space.

Original: However, in the ROS task, the RL system takes the image as the state representation, these methods are not directly applicable since the higher layer can hardly propose an image as a sub-goal for the lower layer to achieve.
Revised: However, in the ROS task, the RL system takes the image as the state representation, these methods are not directly applicable since the higher layer can hardly propose an image as a sub-goal for the lower layer to achieve.

Original: Other methods designate a separate sub-goal space for hierarchical reinforcement learning.
Revised: Other methods designate a separate sub-goal space for hierarchical reinforcement learning.

Original: For example, Kulkarni et al.
Revised: For example, Kulkarni et al.

Original: (2016) deﬁned the sub-goal space in the space of entities and relations, such as the “reach” relation they used for their Atari game experiment.
Revised: (2016) deﬁned the sub-goal space in the space of entities and relations, such as the “reach” relation they used for their Atari game experiment.

Original: Sub-tasks and their relations were provided as inputs in Andreas et al.
Revised: Sub-tasks and their relations were provided as inputs in Andreas et al.

Original: (2017) and Sohn et al.
Revised: (2017) and Sohn et al.

Original: Closer related to our work,
Revised: Closer related to our work,

Original: Das et al.
Revised: Das et al.

Original: (2018b) adopted { exit-room, ﬁnd-room, ﬁnd-object, answer } as the sub-goal space to learn a hierarchical policy for the Embodied Question Answering task.
Revised: (2018b) adopted { exit-room, ﬁnd-room, ﬁnd-object, answer } as the sub-goal space to learn a hierarchical policy for the Embodied Question Answering task.

Original: For the same task, Gordon et al.
Revised: For the same task, Gordon et al.

Original: (2018) chose { navigate, scan, detect, manipulate, answer } as the possible sub-tasks, while the reinforcement learning methods were mainly applied for learning high-level policy.
Revised: (2018) chose { navigate, scan, detect, manipulate, answer } as the possible sub-tasks, while the reinforcement learning methods were mainly applied for learning high-level policy.

Original: More recently,
Revised: More recently,

Original: Gordon et al.
Revised: Gordon et al.

Original: (2019) integrated symbolic planning as an additional sub-task.
Revised: (2019) integrated symbolic planning as an additional sub-task.

Original: On the other side, attempts have been made to learn the sub-tasks automatically.
Revised: On the other side, attempts have been made to learn the sub-tasks automatically.

Original: These sub-tasks are referred to as temporal abstractions.
Revised: These sub-tasks are referred to as temporal abstractions.

Original: Bacon et al.
Revised: Bacon et al.

Original: (2017) proposed the option-critic framework to autonomously discover the speciﬁed number of temporal abstractions.
Revised: (2017) proposed the option-critic framework to autonomously discover the speciﬁed number of temporal abstractions.

Original: Osa et al.
Revised: Osa et al.

Original: (2019) learned the temporal abstractions through advantage-weighted information maximization.
Revised: (2019) learned the temporal abstractions through advantage-weighted information maximization.

Original: Nachum et al.
Revised: Nachum et al.

Original: (2018a) addressed the sub-goal representation learning problem.
Revised: (2018a) addressed the sub-goal representation learning problem.

Original: With the learned representation, their hierarchical policies are shown to approach the optimal performance within a bounded error.
Revised: With the learned representation, their hierarchical policies are shown to approach the optimal performance within a bounded error.

Original: Motivated by these works, we fully utilize the human speciﬁed sub-goal space to make the hierarchy better interpretable.
Revised: Motivated by these works, we fully utilize the human speciﬁed sub-goal space to make the hierarchy better interpretable.

Original: Meanwhile, to avoid being misled by the possibly imperfect human knowledge, we also leverage the beneﬁts from the automatic temporal abstraction learning methods, which yields a hybrid one.
Revised: Meanwhile, to avoid being misled by the possibly imperfect human knowledge, we also leverage the beneﬁts from the automatic temporal abstraction learning methods, which yields a hybrid one.

Original: Target-driven visual navigation.
Revised: Target-driven visual navigation.

Original: Deep reinforcement learning has been studied extensively for the target-driven visual navigation task.
Revised: Deep reinforcement learning has been studied extensively for the target-driven visual navigation task.

Original: Mirowski et al.
Revised: Mirowski et al.

Original: (2016) proposed learning depth prediction and action policy jointly for visual navigation.
Revised: (2016) proposed learning depth prediction and action policy jointly for visual navigation.

Original: Zhu et al.
Revised: Zhu et al.

Original: (2017) introduced a target-driven deep reinforcement learning framework to enable a robot to reach an image-speciﬁed location.
Revised: (2017) introduced a target-driven deep reinforcement learning framework to enable a robot to reach an image-speciﬁed location.

Original: The Embodied Question Answering or Interactive Question Answering task is introduced and addressed in
Revised: The Embodied Question Answering or Interactive Question Answering task is introduced and addressed in

Original: Das et al.
Revised: Das et al.

Original: (2018a),Das et al.
Revised: (2018a),Das et al.

Original: (2018b),Gordon et al.
Revised: (2018b),Gordon et al.

Original: , Gordon et al.
Revised: , Gordon et al.

Original: (2019), Yu et al.
Revised: (2019), Yu et al.

Original: and Wijmans et al.
Revised: and Wijmans et al.

Original: (2019), where the robot needs to navigate an environment to answer an unconstrained natural language question.
Revised: (2019), where the robot needs to navigate an environment to answer an unconstrained natural language question.

Original: Gupta et al.
Revised: Gupta et al.

Original: (2017), Ye et al.
Revised: (2017), Ye et al.

Original: (2018b), Mousavian et al.
Revised: (2018b), Mousavian et al.

Original: (2018) and Ye et al.
Revised: (2018) and Ye et al.

Original: (2018a) focused on the robotic object search and/or object approaching task.
Revised: (2018a) focused on the robotic object search and/or object approaching task.

Original: Anderson et al.
Revised: Anderson et al.

Original: (2018b),Wang et al. (2018b) and Wang et al.
Revised: (2018b),Wang et al. (2018b) and Wang et al.

Original: (2018a) focused on natural language guided visual navigation.
Revised: (2018a) focused on natural language guided visual navigation.

Original: Among all of these cited work, most of them plan over the atomic actions for navigation (Mirowski et al., 2016; Zhu et al., 2017; Das et al., 2018a; Wijmans et al., 2019; Gupta et al., 2017; Ye et al., 2018b; Mousavian et al., 2018;
Revised: Among all of these cited work, most of them plan over the atomic actions for navigation (Mirowski et al., 2016; Zhu et al., 2017; Das et al., 2018a; Wijmans et al., 2019; Gupta et al., 2017; Ye et al., 2018b; Mousavian et al., 2018;

Original: Ye et al., 2018a; Anderson et al., 2018b;
Revised: Ye et al., 2018a; Anderson et al., 2018b;

Original: Wang et al., 2018a) without hierarchical modeling, and thus lack interpretability.
Revised: Wang et al., 2018a) without hierarchical modeling, and thus lack interpretability.

Original: While Gordon et al.
Revised: While Gordon et al.

Original: (2018) and Gordon et al.
Revised: (2018) and Gordon et al.

Original: (2019) studied hierarchical policies, they manually constructed the hierarchy with high-level and low-level controllers, and a standard reinforcement learning method is applied on high-level layer to decide which low-level controller to invoke.
Revised: (2019) studied hierarchical policies, they manually constructed the hierarchy with high-level and low-level controllers, and a standard reinforcement learning method is applied on high-level layer to decide which low-level controller to invoke.

Original: Das et al.
Revised: Das et al.

Original: (2018b) also studied hierarchical reinforcement learning to generate explainable hierarchical policies.
Revised: (2018b) also studied hierarchical reinforcement learning to generate explainable hierarchical policies.

Original: Along with these mentioned methods that build upon human speciﬁed sub-goal space, we further consider the likelihood that these manual sub-goals may not lead to the optimal steps towards the ﬁnal goal, and we model it explicitly in our framework.
Revised: Along with these mentioned methods that build upon human speciﬁed sub-goal space, we further consider the likelihood that these manual sub-goals may not lead to the optimal steps towards the ﬁnal goal, and we model it explicitly in our framework.

Original: More importantly, many of the previous works assume that the agent can access the full information of the environments during the training time, either by deﬁning the reward function with the distance between the agent’s current location and the target location (Mousavian et al., 2018;
Revised: More importantly, many of the previous works assume that the agent can access the full information of the environments during the training time, either by deﬁning the reward function with the distance between the agent’s current location and the target location (Mousavian et al., 2018;

Original: Wang et al., 2018b;a),
Revised: Wang et al., 2018b;a),

Original: and/or adopting shortest path as the supervised signal for pre-training (Das et al., 2018a;b;
Revised: and/or adopting shortest path as the supervised signal for pre-training (Das et al., 2018a;b;

Original: Gordon et al., 2018; Anderson et al., 2018b; Wang et al., 2018a; Yu et al., 2019).
Revised: Gordon et al., 2018; Anderson et al., 2018b; Wang et al., 2018a; Yu et al., 2019).

Original: Nevertheless, for applications in real-world environments, collecting all the information is unarguably expensive and sometimes impossible.
Revised: Nevertheless, for applications in real-world environments, collecting all the information is unarguably expensive and sometimes impossible.

Original: We would like to stress upon the point that our model does not assume any environment information accessible even during the training process, which levels up the ROS’ difﬁculty.
Revised: We would like to stress upon the point that our model does not assume any environment information accessible even during the training process, which levels up the ROS’ difﬁculty.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we present a novel two-layer hierarchical policy learning framework that builds on intrinsic and extrinsic rewards.
Revised: In this paper, we present a novel two-layer hierarchical policy learning framework that builds on intrinsic and extrinsic rewards.

Original: The framework fully utilizes the prior knowledge provided by human to make the hierarchy scheme interpretable.
Revised: The framework fully utilizes the prior knowledge provided by human to make the hierarchy scheme interpretable.

Original: When the high-level layer plans over the human speciﬁed sub-goal space to achieve the goal, the low-level layer plans over the atomic actions by taking both the sub-goal proposed by the high-level layer and the goal into the consideration.
Revised: When the high-level layer plans over the human speciﬁed sub-goal space to achieve the goal, the low-level layer plans over the atomic actions by taking both the sub-goal proposed by the high-level layer and the goal into the consideration.

Original: The framework is general and we validate it in the context of the object search task.
Revised: The framework is general and we validate it in the context of the object search task.

Original: The empirical experiments on House3D platform demonstrate the efﬁcacy and efﬁciency of our proposed framework.
Revised: The empirical experiments on House3D platform demonstrate the efﬁcacy and efﬁciency of our proposed framework.

Original: The current work also opens several avenues for future study.
Revised: The current work also opens several avenues for future study.

Original: First, while we adopt a ﬁxed α value in our experiments to indicate the importance of the sub-goals compared to the goal, the α value can be adaptively updated while approaching different sub-goals or even be learned from common-sense knowledge.
Revised: First, while we adopt a ﬁxed α value in our experiments to indicate the importance of the sub-goals compared to the goal, the α value can be adaptively updated while approaching different sub-goals or even be learned from common-sense knowledge.

Original: We intend to extend our work towards integrating top-down human knowledge together with the human speciﬁed sub-goal space to facilitate the object search task with higher efﬁciency.
Revised: We intend to extend our work towards integrating top-down human knowledge together with the human speciﬁed sub-goal space to facilitate the object search task with higher efﬁciency.

Original: A N ETWORK A RCHITECTURE AND T RAINING D ETAILS
Revised: A N ETWORK A RCHITECTURE AND T RAINING D ETAILS

Original: Feature representation network.
Revised: Feature representation network.

Original: The network is shown in Fig.
Revised: The network is shown in Fig.

Original: 5 upper right corner.
Revised: 5 upper right corner.

Original: It is trained to predict the semantic segmentation and depth map from a single RGB image.
Revised: It is trained to predict the semantic segmentation and depth map from a single RGB image.

Original: The network builds upon Chen et al.
Revised: The network builds upon Chen et al.

Original: (2018) with an additional decoder branch to output the depth map.
Revised: (2018) with an additional decoder branch to output the depth map.

Original: The two decoder branches share the the same architecture except that one decoder branch pipelines to a classiﬁcation output layer for semantic segmentation and the other adopts a regression output layer for depth prediction.
Revised: The two decoder branches share the the same architecture except that one decoder branch pipelines to a classiﬁcation output layer for semantic segmentation and the other adopts a regression output layer for depth prediction.

Original: We adopt the cross entropy loss function for semantic segmentation and mean-square error loss function for depth prediction and train the network with the images sampled fromenvironments from House3D (Wu et al.,
Revised: We adopt the cross entropy loss function for semantic segmentation and mean-square error loss function for depth prediction and train the network with the images sampled fromenvironments from House3D (Wu et al.,

Original: 2018).The trained network is then deployed for extracting the feature representations and the semantic segmentation prediction is further adopted as the robot’s detection system.
Revised: 2018).The trained network is then deployed for extracting the feature representations and the semantic segmentation prediction is further adopted as the robot’s detection system.

Original: High-level network.
Revised: High-level network.

Original: The network is shown in Fig.
Revised: The network is shown in Fig.

Original: 5 upper part.
Revised: 5 upper part.

Original: The encoder of the feature representation network outputs a 256 dimensional feature for either target object image or the robot’s current observation image.
Revised: The encoder of the feature representation network outputs a 256 dimensional feature for either target object image or the robot’s current observation image.

Original: For each input stream, we concatenate the features of 4 history frames and then project the 1024 dimensional vector down to a 512 dimensional vector.
Revised: For each input stream, we concatenate the features of 4 history frames and then project the 1024 dimensional vector down to a 512 dimensional vector.

Original: The two 512 dimensional vectors from both input streams are then concatenated and aggregated into a singedimensional joint vector.
Revised: The two 512 dimensional vectors from both input streams are then concatenated and aggregated into a singedimensional joint vector.

Original: The joint vector is further projected into 78 state action values.
Revised: The joint vector is further projected into 78 state action values.

Original: ( 78 is the number of the objects that being of our interest.)
Revised: ( 78 is the number of the objects that being of our interest.)

Original: Low-level network.
Revised: Low-level network.

Original: The network is shown in Fig.
Revised: The network is shown in Fig.

Original: 5 lower part.
Revised: 5 lower part.

Original: We extract the two attention masks from the predicted semantic segmentation with the semantic label of the target object and the object speciﬁed in the sub-goal.
Revised: We extract the two attention masks from the predicted semantic segmentation with the semantic label of the target object and the object speciﬁed in the sub-goal.

Original: The two attention masks and the predicted depth map are then resized to 10 by 10 .
Revised: The two attention masks and the predicted depth map are then resized to 10 by 10 .

Original: For each input stream, we concatenate the features of 4 history frames and then project the 400 dimensional vector down to a 512 dimensional vector.
Revised: For each input stream, we concatenate the features of 4 history frames and then project the 400 dimensional vector down to a 512 dimensional vector.

Original: The three 512 dimensional vectors from three input streams are then concatenated and aggregated into a single 512 dimensional joint vector.
Revised: The three 512 dimensional vectors from three input streams are then concatenated and aggregated into a single 512 dimensional joint vector.

Original: The joint vector is further passed through three branches.
Revised: The joint vector is further passed through three branches.

Original: For all three branches, we ﬁrst project the 512 dimensional joint vector into a 20 dimensional vector.
Revised: For all three branches, we ﬁrst project the 512 dimensional joint vector into a 20 dimensional vector.

Original: In the ﬁrst branch, we further project the 20 dimensional vector into 6 atomic action policy outputs, i.e. probability over atomic actions.
Revised: In the ﬁrst branch, we further project the 20 dimensional vector into 6 atomic action policy outputs, i.e. probability over atomic actions.

Original: In the second branch, we project the 20 dimensional vector into a single state value output.
Revised: In the second branch, we project the 20 dimensional vector into a single state value output.

Original: In the third branch, we project the 20 dimensional vector into a single signal output as termination, i.e. probability to terminate.
Revised: In the third branch, we project the 20 dimensional vector into a single signal output as termination, i.e. probability to terminate.

Original: We train this hierarchical network with a RMSProp optimizer of learning rate 1 × 10 − 4 , and we reduce the learning rate by an order of magnitude every 1000 episodes.
Revised: We train this hierarchical network with a RMSProp optimizer of learning rate 1 × 10 − 4 , and we reduce the learning rate by an order of magnitude every 1000 episodes.

Original: B Q UALITATIVE R ESULTS
Revised: B Q UALITATIVE R ESULTS

Original: Fig.
Revised: Fig.

Original: 6 shows some qualitative results of our method.
Revised: 6 shows some qualitative results of our method.

Original: The robot can only access the ﬁrst-person view
Revised: The robot can only access the ﬁrst-person view

Original: RGB images.
Revised: RGB images.

Original: We show the top-down 2D map for better visualization.
Revised: We show the top-down 2D map for better visualization.

Original: From Fig.
Revised: From Fig.

Original: 6, we can see that the robot chooses to approach a related object ﬁrst before detecting the target object.
Revised: 6, we can see that the robot chooses to approach a related object ﬁrst before detecting the target object.

Original: Once the target object is detected, the robot then opts to approach the target object immediately.
Revised: Once the target object is detected, the robot then opts to approach the target object immediately.

Original: ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓
Revised: ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓ ⇓⇓

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: b)
Revised: b)

Original: [Figure]
Revised: [Figure]