Original: L EARNING AND P LANNING WITH A S EMANTIC M ODEL
Revised: L EARNING AND P LANNING WITH A S EMANTIC M ODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI.
Revised: Building deep reinforcement learning agents that can generalize and adapt to unseen environments remains a fundamental challenge for AI.

Original: This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities.
Revised: This paper describes progresses on this challenge in the context of man-made environments, which are visually diverse but contain intrinsic semantic regularities.

Original: We propose a hybrid model-based and model-free approach, LEArning and Planning with Semantics (LEAPS) , consisting of a multi-target sub-policy that acts on visual inputs, and a
Revised: We propose a hybrid model-based and model-free approach, LEArning and Planning with Semantics (LEAPS) , consisting of a multi-target sub-policy that acts on visual inputs, and a

Original: Bayesian model over semantic structures.
Revised: Bayesian model over semantic structures.

Original: When placed in an unseen environment, the agent plans with the semantic model to make high-level decisions, proposes the next sub-target for the sub-policy to execute, and updates the semantic model based on new observations.
Revised: When placed in an unseen environment, the agent plans with the semantic model to make high-level decisions, proposes the next sub-target for the sub-policy to execute, and updates the semantic model based on new observations.

Original: We perform experiments in visual navigation tasks using House3D, a 3D environment that contains diverse human-designed indoor scenes with real-world objects.
Revised: We perform experiments in visual navigation tasks using House3D, a 3D environment that contains diverse human-designed indoor scenes with real-world objects.

Original: LEAPS outperforms strong baselines that do not explicitly plan using the semantic content.
Revised: LEAPS outperforms strong baselines that do not explicitly plan using the semantic content.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep reinforcement learning (DRL) has undoubtedly witnessed strong achievements in recent years (Silver et al., 2016; Mnih et al., 2015; Levine et al., 2016).
Revised: Deep reinforcement learning (DRL) has undoubtedly witnessed strong achievements in recent years (Silver et al., 2016; Mnih et al., 2015; Levine et al., 2016).

Original: However, training an agent to solve tasks in a new unseen scenario, usually referred to as its generalization ability , remains a challenging problem (Geffner, 2018;
Revised: However, training an agent to solve tasks in a new unseen scenario, usually referred to as its generalization ability , remains a challenging problem (Geffner, 2018;

Original: Lake et al., 2017).
Revised: Lake et al., 2017).

Original: In model-free RL, the agent is trained to reactively make decisions from the observations, e.g., ﬁrst-person view, via a black-box policy approximator.
Revised: In model-free RL, the agent is trained to reactively make decisions from the observations, e.g., ﬁrst-person view, via a black-box policy approximator.

Original: However the generalization ability of agents trained by model-free RL is limited, and is even more evident on tasks that require extensive planning (Tamar et al., 2016; Kansky et al., 2017).
Revised: However the generalization ability of agents trained by model-free RL is limited, and is even more evident on tasks that require extensive planning (Tamar et al., 2016; Kansky et al., 2017).

Original: On the other hand, model-based RL learns a dynamics model, predicting the next observation when taking an action.
Revised: On the other hand, model-based RL learns a dynamics model, predicting the next observation when taking an action.

Original: With the model, sequential decisions can be made via planning.
Revised: With the model, sequential decisions can be made via planning.

Original: However, learning a model for complex tasks and with high dimensional observations, such as images, is challenging.
Revised: However, learning a model for complex tasks and with high dimensional observations, such as images, is challenging.

Original: Current approaches for learning action-conditional models from video are only accurate for very short horizons (Finn & Levine, 2017; Ebert et al., 2017;
Revised: Current approaches for learning action-conditional models from video are only accurate for very short horizons (Finn & Levine, 2017; Ebert et al., 2017;

Original: Oh et al., 2015).
Revised: Oh et al., 2015).

Original: Moreover, it is not clear how to efﬁciently adapt such models to changes in the domain.
Revised: Moreover, it is not clear how to efﬁciently adapt such models to changes in the domain.

Original: In this work, we aim to improve the generalization of RL agents in domains that involve highdimensional observations.
Revised: In this work, we aim to improve the generalization of RL agents in domains that involve highdimensional observations.

Original: Our insight is that in many realistic settings, building a pixel-accurate model of the dynamics is not necessary for planning high-level decisions.
Revised: Our insight is that in many realistic settings, building a pixel-accurate model of the dynamics is not necessary for planning high-level decisions.

Original: There are semantic structures and properties that are shared in real-world man-made environments.
Revised: There are semantic structures and properties that are shared in real-world man-made environments.

Original: For example, rooms in indoor scenes are often arranged by their mutual functionality ( e.g . bathroom next to bedroom, dining room next to kitchen).
Revised: For example, rooms in indoor scenes are often arranged by their mutual functionality (e.g. , bathroom next to bedroom, dining room next to kitchen).

Original: Similarly, objects in rooms are placed at locations of practical signiﬁcance ( e.g . nightstand next to bed, chair next to table).
Revised: Similarly, objects in rooms are placed at locations of practical signiﬁcance (e.g. , nightstand next to bed, chair next to table).

Original: Humans often make use of such structural priors when exploring a new scene, or when making a high-level plan of actions in the domain.
Revised: Humans often make use of such structural priors when exploring a new scene, or when making a high-level plan of actions in the domain.

Original: However, pixel-level details are still necessary for carrying out the high-level plan.
Revised: However, pixel-level details are still necessary for carrying out the high-level plan.

Original: For example, we need high-ﬁdelity observations to locate and interact with objects, open doors, etc.
Revised: For example, we need high-ﬁdelity observations to locate and interact with objects, open doors, etc.

Original: Based on this observation, we propose a hybrid framework, LEArning and Planning with Semantics (LEAPS) , which consists of a model-based component that works on the semantic level to pursue a high-level target, and a model-free component that executes the target by acting on pixel-level inputs.
Revised: Based on this observation, we propose a hybrid framework, LEArning and Planning with Semantics (LEAPS) , which consists of a model-based component that works on the semantic level to pursue a high-level target, and a model-free component that executes the target by acting on pixel-level inputs.

Original: Concretely, we (1) train model-free multi-target subpolicies in the form of neural networks that take the ﬁrst-person views as input and sequentially execute sub-targets towards the ﬁnal goal; (2) build a semantic model in the form of a latent variable model that only takes semantic signals , i.e., low-dimensional binary vectors, as input and is dynamically updated to plan the next sub-target.
Revised: Concretely, we (1) train model-free multi-target subpolicies in the form of neural networks that take the ﬁrst-person views as input and sequentially execute sub-targets towards the ﬁnal goal; (2) build a semantic model in the form of a latent variable model that only takes semantic signals , i.e., low-dimensional binary vectors, as input and is dynamically updated to plan the next sub-target.

Original: LEAPS has following advantages: (1) via model-based planning, generalization ability is improved; (2) by learning the prior distribution of the latent variable model, we capture the semantic consistency among the environments; (3) the semantic model can be efﬁciently updated by posterior inference when the agent is exploring the unseen environment, which is effective even with very few exploration experiences thanks to the Bayes rule; and (4) the semantic model is lightweight and fully interpretable.
Revised: LEAPS has following advantages: (1) via model-based planning, generalization ability is improved; (2) by learning the prior distribution of the latent variable model, we capture the semantic consistency among the environments; (3) the semantic model can be efﬁciently updated by posterior inference when the agent is exploring the unseen environment, which is effective even with very few exploration experiences thanks to the Bayes rule; and (4) the semantic model is lightweight and fully interpretable.

Original: Our approach requires observations that are composed of both pixel-level data and a list of semantic properties of the scene.
Revised: Our approach requires observations that are composed of both pixel-level data and a list of semantic properties of the scene.

Original: In general, automatically extracting high-level semantic structure from data is difﬁcult.
Revised: In general, automatically extracting high-level semantic structure from data is difﬁcult.

Original: As a ﬁrst step, in this work we focus on domains where obtaining semantics is easy.
Revised: As a ﬁrst step, in this work we focus on domains where obtaining semantics is easy.

Original: In particular, we considerreal-world environmentsfor which  strong object detectorsare available (He et al., 2017).
Revised: In particular, we consider environments which resemble the real-world and have strong object detectors available (He et al., 2017).

Original: An example of such environments is House3D which contains 45k real-world 3D scenes (Wu et al., 2018).
Revised: An example of such environments is House3D which contains 45k human-designed 3D scenes (Wu et al., 2018).

Original: House3D provides a diverse set of scene layouts, object types, sizes and connectivity, which all conform to a consistent “natural” semantics.
Revised: House3D provides a diverse set of scene layouts, object types, sizes and connectivity, which all conform to a consistent “natural” semantics.

Original: Within these complex scenes, we tackle navigation tasks within novel indoor scenes.
Revised: Within these complex scenes, we tackle navigation tasks within novel indoor scenes.

Original: Note that this problem is extremely challenging as the agent needs to reach far-away targets which can only be completed effectively if it can successfully reason about the overall structure of the new scenario.
Revised: Note that this problem is extremely challenging as the agent needs to reach far-away targets which can only be completed effectively if it can successfully reason about the overall structure of the new scenario.

Original: Lastly, we emphasize that although we consider navigation as a concrete example in this work, our approach is general and can be applied to other tasks for which semantic structures and signals are available
Revised: Lastly, we emphasize that although we consider navigation as a concrete example in this work, our approach is general and can be applied to other tasks for which semantic structures and signals are available

Original: Our extensive experiments show that our LEAPS framework outperforms strong model-free RL approaches, even when the semantic signals are given as input to the policy.
Revised: Our extensive experiments show that our LEAPS framework outperforms strong model-free RL approaches, even when the semantic signals are given as input to the policy.

Original: Furthermore, the relative improvements of LEAPS over baselines become more signiﬁcant when the targets are further away from the agent’s birthplace, indicating the effectiveness of planning on the learned semantic model.
Revised: Furthermore, the relative improvements of LEAPS over baselines become more signiﬁcant when the targets are further away from the agent’s birthplace, indicating the effectiveness of planning on the learned semantic model.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Most deep RL agents are tested in the same training environments (Mirowski et al., 2016), disregarding generalization.
Revised: Most deep RL agents are tested in the same training environments (Mirowski et al., 2016), disregarding generalization.

Original: While limited, robust training approaches have been proposed to enforce an agent’s generalization ability, such as domain randomization (Tobin et al., 2017) and data augmentation by generating random mazes for training (Oh et al., 2017; Parisotto & Salakhutdinov, 2017).
Revised: While limited, robust training approaches have been proposed to enforce an agent’s generalization ability, such as domain randomization (Tobin et al., 2017) and data augmentation by generating random mazes for training (Oh et al., 2017; Parisotto & Salakhutdinov, 2017).

Original: In our work, we use a test set of novel unseen environments, where an agent cannot resort to memorization or simple pattern matching to solve the task.
Revised: In our work, we use a test set of novel unseen environments, where an agent cannot resort to memorization or simple pattern matching to solve the task.

Original: [Table]
Revised: Meta

Original:  has shown promising results for fast adaptation to novel environments.
Revised: -learning has shown promising results for fast adaptation to novel environments.

Original: Methods include learning a good initialization for gradient descent (Finn et al., 2017) or learning a neural network that can adapt its policy during exploration (Duan et al., 2016;
Revised: Methods include learning a good initialization for gradient descent (Finn et al., 2017) or learning a neural network that can adapt its policy during exploration (Duan et al., 2016;

Original: Mishra et al., 2017).
Revised: Mishra et al., 2017).

Original: We propose to learn a Bayesian model over the semantic level and infer the posterior structure via the
Revised: We propose to learn a Bayesian model over the semantic level and infer the posterior structure via the

Original: Bayes rule.
Revised: Bayes rule.

Original: Our approach (1) can work even without any exploration steps in a new environment and (2) is interpretable and can be potentially combined with any graph-based planning algorithm.
Revised: Our approach (1) can work even without any exploration steps in a new environment and (2) is interpretable and can be potentially combined with any graph-based planning algorithm.

Original: Our work can be viewed as a special case of hierarchical reinforcement learning (HRL).
Revised: Our work can be viewed as a special case of hierarchical reinforcement learning (HRL).

Original: Unlike other approaches (Vezhnevets et al., 2017; Bacon et al., 2017), in our work high-level planning is performed based on the semantic signals.
Revised: Unlike other approaches (Vezhnevets et al., 2017; Bacon et al., 2017), in our work high-level planning is performed based on the semantic signals.

Original: With orders of magnitudes fewer parameters, our approach is easier to learn compared to recurrent controllers.
Revised: With orders of magnitudes fewer parameters, our approach is easier to learn compared to recurrent controllers.

Original: LEAPS assumes a discrete semantic signal in addition to the continuous state.
Revised: LEAPS assumes a discrete semantic signal in addition to the continuous state.

Original: A similar assumption is also adopted in (Zhang et al., 2018), where the discrete signals are called “attributes” and used for planning to solve compositional tasks within the same fully observable environment.
Revised: A similar assumption is also adopted in (Zhang et al., 2018), where the discrete signals are called “attributes” and used for planning to solve compositional tasks within the same fully observable environment.

Original: (Riedmiller et al., 2018) use additional discrete signals to tackle the sparse reward problem.
Revised: (Riedmiller et al., 2018) use additional discrete signals to tackle the sparse reward problem.

Original: The schema network (Kansky et al., 2017) further assumes that even the continuous visual signal can be completely represented in a binary form and therefore directly runs logical reasoning on the binary states.
Revised: The schema network (Kansky et al., 2017) further assumes that even the continuous visual signal can be completely represented in a binary form and therefore directly runs logical reasoning on the binary states.

Original: For evaluating our approach, we focus on the problem of visual navigation, which has been studied extensively (Leonard & Durrant-Whyte, 1992).
Revised: For evaluating our approach, we focus on the problem of visual navigation, which has been studied extensively (Leonard & Durrant-Whyte, 1992).

Original: Classical approaches build a 3D map of the scene using SLAM, which is subsequently used for planning (Fox et al., 2005).
Revised: Classical approaches build a 3D map of the scene using SLAM, which is subsequently used for planning (Fox et al., 2005).

Original: More recently, end-to-end approaches have been applied to tackle various domains, such as maze (Mirowski et al., 2016), indoor scenes (Zhu et al., 2017) and Google street view (Mirowski et al., 2018).
Revised: More recently, end-to-end approaches have been applied to tackle various domains, such as maze (Mirowski et al., 2016), indoor scenes (Zhu et al., 2017) and Google street view (Mirowski et al., 2018).

Original: Evidently, navigation performance deteriorates as the agent’s distance from the target increases (Zhu et al., 2017; Wu et al., 2018).
Revised: Evidently, navigation performance deteriorates as the agent’s distance from the target increases (Zhu et al., 2017; Wu et al., 2018).

Original: To aid navigation and boost performance, auxiliary tasks (Mirowski et al., 2016; Jaderberg et al., 2016) are often introduced during training.
Revised: To aid navigation and boost performance, auxiliary tasks (Mirowski et al., 2016; Jaderberg et al., 2016) are often introduced during training.

Original: Another direction for visual navigation is to use a recurrent neural network and represent the memory in the form of a 2D spatial map (Khan et al., 2018;
Revised: Another direction for visual navigation is to use a recurrent neural network and represent the memory in the form of a 2D spatial map (Khan et al., 2018;

Original: Parisotto & Salakhutdinov, 2017; Tamar et al., 2016; Gupta et al., 2017)
Revised: Parisotto & Salakhutdinov, 2017; Tamar et al., 2016; Gupta et al., 2017)

Original: such that a differentiable planning computation can be performed on the spatial memory.
Revised: such that a differentiable planning computation can be performed on the spatial memory.

Original: Our approach considers more general graph structures beyond dense 2D grids and captures relationships between semantic signals, which we utilize as an informative latent structure in semantically rich environments like House3D.
Revised: Our approach considers more general graph structures beyond dense 2D grids and captures relationships between semantic signals, which we utilize as an informative latent structure in semantically rich environments like House3D.

Original: Similar to our work, Savinov et al .
Revised: Similar to our work, Savinov et al .

Original: (Savinov et al., 2018) constructs a graph of nodes corresponding to different locations of the environment.
Revised: (Savinov et al., 2018) constructs a graph of nodes corresponding to different locations of the environment.

Original: However, they rely on a pre-exploration step within the test scene and build the graph completely from the pixel space.
Revised: However, they rely on a pre-exploration step within the test scene and build the graph completely from the pixel space.

Original: In LEAPS, we use semantic knowledge and learn a prior over the semantic structures that are shared across real-world scenes.
Revised: In LEAPS, we use semantic knowledge and learn a prior over the semantic structures that are shared across real-world scenes.

Original: This allows us to directly start solving for the task at hand without any exploratory steps.
Revised: This allows us to directly start solving for the task at hand without any exploratory steps.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We assume familiarity with standard DRL notations.
Revised: We assume familiarity with standard DRL notations.

Original: Complete deﬁnitions are in Appendix A.
Revised: Complete deﬁnitions are in Appendix A.

Original: Environment: We consider a contextual Markov decision process (Hallak et al., 2015)
Revised: Environment: We consider a contextual Markov decision process (Hallak et al., 2015)

Original: E ( c ) deﬁned by E ( c ) =
Revised: E ( c ) deﬁned by E ( c ) =

Original: ( S , A , P ( s (cid:48) | s, a ; c ) , r ( s, a ; c )) .
Revised: ( S , A , P ( s (cid:48) | s, a ; c ) , r ( s, a ; c )) .

Original: Here c represents the objects, layouts and any other semantic information describing the environment, and is sampled from C , the distribution of possible semantic scenarios.
Revised: Here c represents the objects, layouts and any other semantic information describing the environment, and is sampled from C , the distribution of possible semantic scenarios.

Original: For example, c can be intuitively understood as encoding the complete map for navigation, or the complete object and obstacle layouts in robotics manipulations, not known to the agent in advance, and we refer to them as the context .
Revised: For example, c can be intuitively understood as encoding the complete map for navigation, or the complete object and obstacle layouts in robotics manipulations, not known to the agent in advance, and we refer to them as the context .

Original: Semantic Signal: At each time step, the agent observes from s a tuple ( s o , s s ) , which consists of: (1) a high-dimensional observation s o , e.g . the ﬁrst person view image, and (2) a low-dimensional discrete semantic signal s s , which encodes semantic information.
Revised: Semantic Signal: At each time step, the agent observes from s a tuple ( s o , s s ) , which consists of: (1) a high-dimensional observation s o , e.g., the ﬁrst person view image, and (2) a low-dimensional discrete semantic signal s s , which encodes semantic information.

Original: Such signals are common in AI, e.g . in robotic manipulation tasks s s indicates whether the robot is holding an object; for games it is the game status of a player; in visual navigation it indicates whether the agent reached a landmark; while in the AI planning literature, s s is typically a list of predicates that describe binary properties of objects.
Revised: Such signals are common in AI, e.g., in robotic manipulation tasks s s indicates whether the robot is holding an object; for games it is the game status of a player; in visual navigation it indicates whether the agent reached a landmark; while in the AI planning literature, s s is typically a list of predicates that describe binary properties of objects.

Original: We assume s s is provided by an oracle function, which can either be directly provided by the environment or extracted by some semantic extractor.
Revised: We assume s s is provided by an oracle function, which can either be directly provided by the environment or extracted by some semantic extractor.

Original: Generalization: Let µ ( a |{ s ( t ) } t ; θ ) denote the agent’s policy parametrized by θ conditioned on the previous states { s ( t ) } t .
Revised: Generalization: Let µ ( a |{ s ( t ) } t ; θ ) denote the agent’s policy parametrized by θ conditioned on the previous states { s ( t ) } t .

Original: The objective of generalization is to train a policy on training environments
Revised: The objective of generalization is to train a policy on training environments

Original: E train such that the accumulative reward R ( µ ( θ ); c ) on test set E test is maximized.
Revised: E train such that the accumulative reward R ( µ ( θ ); c ) on test set E test is maximized.

Original:  L EARNING AND P LANNING WITH A S EMANTIC M ODEL
Revised:  L EARNING AND P LANNING WITH A S EMANTIC M ODEL

Original: The key motivation of LEAPS is the fact that while each environment can be different in visual appearances, there are structural similarities between environments that can be captured as a probabilistic graphical model over the semantic information.
Revised: The key motivation of LEAPS is the fact that while each environment can be different in visual appearances, there are structural similarities between environments that can be captured as a probabilistic graphical model over the semantic information.

Original: On a high level, we aim to learn a Bayesian model M (cid:63) ( D , c ) that captures the semantic properties of the context c , from the agent’s exploration experiences D .
Revised: On a high level, we aim to learn a Bayesian model M (cid:63) ( D , c ) that captures the semantic properties of the context c , from the agent’s exploration experiences D .

Original: Given a new environment E ( c (cid:48) ) , the agent computes the posterior P ( c (cid:48) |D (cid:48) , M (cid:63) ) for the unknown context c (cid:48) via the learned model M (cid:63) and its current experiences D (cid:48) .
Revised: Given a new environment E ( c (cid:48) ) , the agent computes the posterior P ( c (cid:48) |D (cid:48) , M (cid:63) ) for the unknown context c (cid:48) via the learned model M (cid:63) and its current experiences D (cid:48) .

Original: This allows the agent to plan according to its belief of c (cid:48) to reach the goal more effectively.
Revised: This allows the agent to plan according to its belief of c (cid:48) to reach the goal more effectively.

Original: Thanks to the Bayes rule, this formulation allows probabilistic inference even with limited (or even no) exploration experiences.
Revised: Thanks to the Bayes rule, this formulation allows probabilistic inference even with limited (or even no) exploration experiences.

Original: Learning an accurate and complete Bayesian model M (cid:63) ( D , c ) can be challenging.
Revised: Learning an accurate and complete Bayesian model M (cid:63) ( D , c ) can be challenging.

Original: We learn an approximate latent variable model M ( y, z ; ψ ) parameterized by ψ with observation variable y and latent variable z
Revised: We learn an approximate latent variable model M ( y, z ; ψ ) parameterized by ψ with observation variable y and latent variable z

Original: that only depend on the semantic signal s s .
Revised: that only depend on the semantic signal s s .

Original: Suppose we have K different semantic signals T 1 , . .
Revised: Suppose we have K different semantic signals T 1 , . .

Original: , T K and s s ∈ { 0 , 1 } K where s s ( T k ) denotes whether the k th signal T k (e.g., landmarks in navigation) is reached or not.
Revised: , T K and s s ∈ { 0 , 1 } K where s s ( T k ) denotes whether the k th signal T k (e.g., landmarks in navigation) is reached or not.

Original: Assuming T 1 is the ﬁnal goal of the task, from any state s , we want to reach some ﬁnal state s (cid:48) with s (cid:48) s ( T 1 )
Revised: Assuming T 1 is the ﬁnal goal of the task, from any state s , we want to reach some ﬁnal state s (cid:48) with s (cid:48) s ( T 1 )

Original: = 1 .
Revised: = 1 .

Original: In this work, we consider navigation as a concrete example, which can be represented as reaching a state where a desired semantic signal becomes ‘true’.
Revised: In this work, we consider navigation as a concrete example, which can be represented as reaching a state where a desired semantic signal becomes ‘true’.

Original: We exploit the fact that navigation to a target can be decomposed into reaching several way points on way to the target, and therefore can be guided by planning on the semantic signals, i.e., arrival at particular way points.
Revised: We exploit the fact that navigation to a target can be decomposed into reaching several way points on way to the target, and therefore can be guided by planning on the semantic signals, i.e., arrival at particular way points.

Original: 4. T HE S EMANTIC M ODEL
Revised: 4. T HE S EMANTIC M ODEL

Original: Note that there can be 2 K different values for s s .
Revised: Note that there can be 2 K different values for s s .

Original: For efﬁcient computation, we assume independence between different semantic signals T k : we use a binary variable
Revised: For efﬁcient computation, we assume independence between different semantic signals T k : we use a binary variable

Original: z i,j to denote whether some state s (cid:48) with s (cid:48) s ( T j )
Revised: z i,j to denote whether some state s (cid:48) with s (cid:48) s ( T j )

Original: = 1 can be “ directly reached ”, i.e., by a few exploration steps, from some state s with s s ( T i )
Revised: = 1 can be “ directly reached ”, i.e., by a few exploration steps, from some state s with s s ( T i )

Original: = 1 , regardless of other signals T k (cid:54)∈ { T i , T j } .
Revised: = 1 , regardless of other signals T k (cid:54)∈ { T i , T j } .

Original: In addition, we also assume reversibility , i.e., z i,j =
Revised: In addition, we also assume reversibility , i.e., z i,j =

Original: z j,i , so only K ( K − 1) / 2 latent variables are needed.
Revised: z j,i , so only K ( K − 1) / 2 latent variables are needed.

Original: Before entering the unknown environment, the agent does not know the true value of z i,j , but holds some prior belief P ( z i,j ) , deﬁned by z
Revised: Before entering the unknown environment, the agent does not know the true value of z i,j , but holds some prior belief P ( z i,j ) , deﬁned by z

Original: i,j ∼ Bernoulli( ψ prior i,j ) , where ψ prior i,j is some parameter to be learned.
Revised: i,j ∼ Bernoulli( ψ prior i,j ) , where ψ prior i,j is some parameter to be learned.

Original: After some exploration steps, the agent receives a noisy observation y i,j of z
Revised: After some exploration steps, the agent receives a noisy observation y i,j of z

Original: i,j , i.e., whether a state s (cid:48) with s (cid:48) s ( T j )
Revised: i,j , i.e., whether a state s (cid:48) with s (cid:48) s ( T j )

Original: = 1 is reached.
Revised: = 1 is reached.

Original: We deﬁne the observation model P ( y i,j | z i,j ) as follows:
Revised: We deﬁne the observation model P ( y i,j | z i,j ) as follows:

Original: [Equation]
Revised: [Equation]

Original: At any time step, the agent hold an overall belief P ( z |Y ) of the semantic structure of the unknown environment, based on its experiences Y , namely the samples of y .
Revised: At any time step, the agent hold an overall belief P ( z |Y ) of the semantic structure of the unknown environment, based on its experiences Y , namely the samples of y .

Original: 4. C OMBINING THE S EMANTIC M ODEL WITH M ULTI -T ARGET S UB - POLICIES
Revised: 4. C OMBINING THE S EMANTIC M ODEL WITH M ULTI -T ARGET S UB - POLICIES

Original: Multi-target sub-policies: With our semantic model, we correspondingly learn multi-target subpolicies µ ( a |{ s ( t ) o } t ; T i , θ ) taking s o as input such that µ ( T i , θ ) is particularly trained for sub-target
Revised: Multi-target sub-policies: With our semantic model, we correspondingly learn multi-target subpolicies µ ( a |{ s ( t ) o } t ; T i , θ ) taking s o as input such that µ ( T i , θ ) is particularly trained for sub-target

Original: T i , i.e., reaching a state s (cid:48) with s (cid:48) s ( T i )
Revised: T i , i.e., reaching a state s (cid:48) with s (cid:48) s ( T i )

Original: = 1 .
Revised: = 1 .

Original: Hence the semantic model can be treated as a modelbased planning module that picks an intermediate sub-target for the sub-policies to execute so that the ﬁnal target T 1 can be reached with the highest probability.
Revised: Hence the semantic model can be treated as a modelbased planning module that picks an intermediate sub-target for the sub-policies to execute so that the ﬁnal target T 1 can be reached with the highest probability.

Original: Learning the multi-target sub-policies can be accomplished by any standard deep RL method on E train .

Original: Inference and planning on M : We assume the agent explores the current environment for a short horizon of N steps and receives semantic signals s s (1) , . . .
Revised: Inference and planning on M : We assume the agent explores the current environment for a short horizon of N steps and receives semantic signals s s (1) , . . .

Original: , s s ( N ) .
Revised: , s s ( N ) .

Original: Then we compute the bit-OR operation over these binary vectors
Revised: Then we compute the bit-OR operation over these binary vectors

Original: B = s s (1) OR . . .
Revised: B = s s (1) OR . . .

Original: OR s s ( N ) .
Revised: OR s s ( N ) .

Original: By the reversibility assumption, for T i and T j with B ( T i ) =
Revised: By the reversibility assumption, for T i and T j with B ( T i ) =

Original: B ( T j ) = 1 , we know that T i and T j are “directly reachable” for each other, namely a sample of y i,j = 1 , and otherwise y i,j = 0 .
Revised: B ( T j ) = 1 , we know that T i and T j are “directly reachable” for each other, namely a sample of y i,j = 1 , and otherwise y i,j = 0 .

Original: Combining all the history samples of y and the current batch from B as Y , we can perform posterior inference P ( z |Y ) by the Bayes rule.
Revised: Combining all the history samples of y and the current batch from B as Y , we can perform posterior inference P ( z |Y ) by the Bayes rule.

Original: By the independence assumption, we can individually compute the belief of each latent variable z i,j , denoted by ˆ z i,j = P ( z i,j |Y i,j ) .
Revised: By the independence assumption, we can individually compute the belief of each latent variable z i,j , denoted by ˆ z i,j = P ( z i,j |Y i,j ) .

Original: Given the current beliefs ˆ z
Revised: Given the current beliefs ˆ z

Original: i,j , the current semantic signals s s and the goal T 1 , we search for an optimal plan τ ∗ = { τ 0 , τ 1 , . . .
Revised: i,j , the current semantic signals s s and the goal T 1 , we search for an optimal plan τ ∗ = { τ 0 , τ 1 , . . .

Original: , τ m − 1 , τ m } , where τ  m = 1 , so that the joint belief along the path from some current signal to the goal is maximized:
Revised: , τ m − 1 , τ m } , where τ i ∈ { 1 . . . K } denotes an index of concepts and particularly τ m = 1 , so that the joint belief along the path from some current signal to the goal is maximized:

Original: [Equation]
Revised: [Equation]

Original: After obtaining τ (cid:63) , we execute the sub-policy for the next sub-target T τ (cid:63) 1 , and then repeatedly update the model and replan every N steps.
Revised: After obtaining τ (cid:63) , we execute the sub-policy for the next sub-target T τ (cid:63) 1 , and then repeatedly update the model and replan every N steps.

Original: 4. L EARNING THE S EMANTIC M ODEL
Revised: 4. L EARNING THE S EMANTIC M ODEL

Original: The model parameters ψ have two parts: ψ prior for the prior of z and ψ obs for the noisy observation y .
Revised: The model parameters ψ have two parts: ψ prior for the prior of z and ψ obs for the noisy observation y .

Original: Note that ψ obs is related to the performance of the sub-policies µ ( θ ) : if µ ( θ ) has a high success rate for reaching sub-targets, ψ obs should be low; when µ ( θ ) is poor, ψ obs should be higher (cf.
Revised: Note that ψ obs is related to the performance of the sub-policies µ ( θ ) : if µ ( θ ) has a high success rate for reaching sub-targets, ψ obs should be low; when µ ( θ ) is poor, ψ obs should be higher (cf.

Original: Eq.
Revised: Eq.

Original: Learning ψ prior : We learn ψ prior from E train .
Revised: Learning ψ prior : We learn ψ prior from E train .

Original: During training, for each pair of semantic signals T i and T j , we run random explorations from some state s with s ( T i ) = 1 .
Revised: During training, for each pair of semantic signals T i and T j , we run random explorations from some state s with s ( T i ) = 1 .

Original: If eventually we reach some state s (cid:48) with s (cid:48)
Revised: If eventually we reach some state s (cid:48) with s (cid:48)

Original: ( T j )
Revised: ( T j )

Original: = 1 , we consider T i and T j are reachable and
Revised: = 1 , we consider T i and T j are reachable and

Original: therefore a positive sample z i,j = 1 ; otherwise a negative sample
Revised: therefore a positive sample z i,j = 1 ; otherwise a negative sample

Original: z
Revised: z

Original: i,j = 0 .
Revised: i,j = 0 .

Original: Suppose Z denotes the samples we obtained for z from E train .
Revised: Suppose Z denotes the samples we obtained for z from E train .

Original: We run maximum likelihood estimate for ψ prior by maximizing L MLE ( ψ prior ) =
Revised: We run maximum likelihood estimate for ψ prior by maximizing L MLE ( ψ prior ) =

Original: P ( Z| ψ prior ) .
Revised: P ( Z| ψ prior ) .

Original: Learning ψ obs : There is no direct supervision for ψ obs .
Revised: Learning ψ obs : There is no direct supervision for ψ obs .

Original: However, we can evaluate a particular value of ψ obs by policy evaluation on the validation environments E valid .
Revised: However, we can evaluate a particular value of ψ obs by policy evaluation on the validation environments E valid .

Original: We optimize the accumulative reward L valid ( ψ obs ) =
Revised: We optimize the accumulative reward L valid ( ψ obs ) =

Original: E E ( c ) ∈E valid
Revised: E E ( c ) ∈E valid

Original: [ R ( µ ( θ ) , M ( ψ ); c )] , with the semantic model M ( ψ ) .
Revised: [ R ( µ ( θ ) , M ( ψ ); c )] , with the semantic model M ( ψ ) .

Original: Analytically optimizing L valid is hard.
Revised: Analytically optimizing L valid is hard.

Original: Instead, we apply local search in practice to ﬁnd the optimal ψ obs .
Revised: Instead, we apply local search in practice to ﬁnd the optimal ψ obs .

Revised: 4. L EARNING THE LEAPS AGENT

Revised: The LEAPS agent consists of two parts, the multi-target sub-policy µ ( T i , θ ) and the semantic model

Revised: M ( ψ ) .

Revised: Learning the multi-target sub-policies can be accomplished by any standard deep RL method on E train .

Revised: For the semantic model, learning ψ prior does not depend on the sub-policies and can be reused even with different sub-policies; ψ obs depends on the sub-policies so it should be learned after µ ( T i , θ ) is obtained.

Original:  R OOM N AV : A 3D N AVIGATION T ASK FOR RL G ENERALIZATION
Revised:  R OOM N AV : A 3D N AVIGATION T ASK FOR RL G ENERALIZATION

Original: RoomNav is a concept driven navigation task based on the House3D environment (Wu et al., 2018).
Revised: RoomNav is a concept driven navigation task based on the House3D environment (Wu et al., 2018).

Original: In RoomNav, the agent is given a concept target, i.e., a room type, and needs to navigate to ﬁnd the target room.
Revised: In RoomNav, the agent is given a concept target, i e., a room type, and needs to navigate to ﬁnd the target room.

Original: RoomNav pre-selected a ﬁxed set of target room types and provides a training set ofhouses, a testing set of 50 houses and a small validation set of 20 houses.
Revised: RoomNav pre-selected a ﬁxed set of target room types and provides a training set ofhouses, a testing set of 50 houses and a small validation set of 20 houses.

Original: Semantic signals: We choose the K = 8 most common room types as our semantic signals, such that s s ( T i ) denotes whether the agent is currently in a room with type T i 1 .
Revised: Semantic signals: We choose the K = 8 most common room types as our semantic signals, such that s s ( T i ) denotes whether the agent is currently in a room with type T i 1 .

Original: When given a target T i , reaching a state s with s s ( T i ) = 1 becomes our ﬁnal goal.
Revised: When given a target T i , reaching a state s with s s ( T i ) = 1 becomes our ﬁnal goal.

Original: House3D provides bounding boxes for rooms, which can be directly used as the oracle for semantic signals.
Revised: House3D provides bounding boxes for rooms, which can be directly used as the oracle for semantic signals.

Original: We can also train a room type classiﬁer to extract the semantics signals, which is evaluated in  Sec. 6.
Revised: But in practice, we only use these oracle signals to train a room type detector and use this detector to extract semantic information during evaluation. Details can be found in the beginning part of Sec. 6.

Original: The semantic model and sub-policies: In navigation, the reachability variable z i,j can naturally represent the connectivity between room type T i and room type T j 2 .
Revised: The semantic model and sub-policies: In navigation, the reachability variable z i,j can naturally represent the connectivity between room type T i and room type T j 2 .

Original: We run random explorations in training houses between rooms to collect samples for learning ψ prior .
Revised: We run random explorations in training houses between rooms to collect samples for learning ψ prior .

Original: For learning ψ obs , we perform a grid search and evaluate on the validation set.
Revised: For learning ψ obs , we perform a grid search and evaluate on the validation set.

Original: For sub-policies, we learn target driven LSTM policies by A3C (Mnih et al., 2016) with shaped reward on E train .
Revised: For sub-policies, we learn target driven LSTM policies by A3C (Mnih et al., 2016) with shaped reward on E train .

Original: More details are in Appendix.
Revised: More details are in Appendix.

Original: E.
Revised: G.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we experiment on RoomNav and try to answer the following questions: (1) Does the learned prior distribution capture meaningful semantic consistencies?
Revised: In this section, we experiment on RoomNav and try to answer the following questions: (1) Does the learned prior distribution capture meaningful semantic consistencies?

Original: (2) Does our LEAPS agent generalize better than the model-freel RL agent that only takes image input? 
Revised: (2) Does our LEAPS agent generalize better than the model-freel RL agent that only takes image input? (3) Our LEAPS

Revised: As the number of planning computations, i.e., H/N , increases (from left to right), LEAPS agent outperforms baselines more.

Revised: LEAPS also has higher relative improvements, i.e., 40% to 180% , for targets requiring more semantic planning computations (i.e., plan-steps > 2).

Original: (3) Our LEAPS agent takes additional semantic signals as input.
Revised: agent takes additional semantic signals as input.

Original: How does LEAPS compare to other model-free RL approaches that also take the semantic signals as part of the inputs but in a different way from our semantic model?
Revised: How does LEAPS compare to other model-free RL approaches that also take the semantic signals as part of the inputs but in a different way from our semantic model?

Original: For example, what about replacing our semantic model with a complicated RNN controller?
Revised: For example, what about replacing our semantic model with a complicated RNN controller?

Original: (4) House3D provides labels for rooms (although noisy).
Revised: (4) What do our LEAPS agents perform under some other metric considering the episode length? We consider a recently proposed metric, SPL (Anderson et al., 2018) in Sec. 6.4.

Original: Can our approach still work if we extract the semantic signals from a trained neural classiﬁer for room types?
Revised: Semantic Signals: All the semantic signals fed to our semantic model at test time are extracted by a CNN room type detector, which is trained on the (noisy) oracle semantic information provided by the House3D on E train and validated on E valid . During training, all the approaches directly use the oracle semantic signals.

Revised: More details are in Appendix J and Appendix D.

Original: 6. T HE L EARNED P RIOR OF THE S EMANTIC M ODEL
Revised: 6. T HE L EARNED P RIOR OF THE S EMANTIC M ODEL

Original: We visualize the learned prior P ( z | ψ prior ) in Fig.
Revised: We visualize the learned prior P ( z | ψ prior ) in Fig.

Original: 1 with 3 room types and their most and least likely connected rooms.
Revised: 1 with 3 room types and their most and least likely connected rooms.

Original: The learned prior indeed captures reasonable relationships: bathroom is likely to connect to a bedroom; kitchen is often near a dining room while garage is typically outdoor
Revised: The learned prior indeed captures reasonable relationships: bathroom is likely to connect to a bedroom; kitchen is often near a dining room while garage is typically outdoor.

Original: 6. C OMPARISON WITH M ODEL -F REE RL B ASELINES
Revised: 6. C OMPARISON WITH M ODEL -F REE RL B ASELINES

Original: We followthe evaluation process in (Wu et al., 2018) and measure the testing success rate  on E test .
Revised: We follow measure the testing success rate of different agents under various horizon lengths on E test .

Original: More details are in Appendix C and D.
Revised: More details are in Appendix C and F.

Original: We compare our LEAPS agent with two baselines (1) random policy (denoted by “random”) and (2) model-free RL agent that only takes in image input s o and executes µ
Revised: We compare our LEAPS agent with two baselines (1) random policy (denoted by “random”) and (2) model-free RL agent that only takes in image input s o and executes µ

Original: ( T i , θ ) throughout the episode (denoted by “pure µ ( θ ) ”).
Revised: ( T i , θ ) throughout the episode (denoted by “pure µ ( θ ) ”).

Original: For LEAPS agent, we set N = 30 , i e., update the semantic model every 30 steps.
Revised: For LEAPS agent, we set N = 30 , i.e., update the semantic model every 30 steps.

Original: We experiment ondifferent horizons H = 300 , 500 , 1000 and evaluate the success rate and relative improvements of our LEAPS agent over the baselines in Fig.
Revised: We experiment on horizons H = 300 , 500 ,and evaluate the success rate and relative improvements of our LEAPS agent over the baselines in Fig.

Original: As the number of planning computations, H/N , increases, our LEAPS agent outperforms the baselines more signiﬁcantly in success rate.
Revised: As the number of planning computations, H/N , increases, our LEAPS agent outperforms the baselines more signiﬁcantly in success rate.

Revised: Note that since targets of plan-steps 1 do not require any planning computations, hence it is as expected that LEAPS does not improve much over the pure policy.

Original: Note that the best relative improvements are achieved for targets neither too faraway nor too close, i e., optimal plan steps equal to 3 or 4.
Revised: The best relative improvements are achieved for targets neither too faraway nor too close, i.e., plan steps equal to 3 or 4.

Original: Interestingly, we observe that there is a small success rate increase for targets that are 5 plan steps away.
Revised: Interestingly, we observe that there is a small success rate increase for targets that are 5 plan steps away.

Original: We suspect that this is because it is rare to see houses that has a diameter of 5 in the semantic model (imagine a house where you need to go through 5 rooms to reach a place).
Revised: We suspect that this is because it is rare to see houses that has a diameter of 5 in the semantic model (imagine a house where you need to go through 5 rooms to reach a place).

Original: Such houses may have structural properties that makes navigation easier.
Revised: Such houses may have structural properties that makes navigation easier.

Original: Fig.
Revised: Fig.

Original: 2 shows an example of a success trajectory of our LEAPS agent.
Revised: 2 shows an example of a success trajectory of our LEAPS agent.

Original: We visualize the progression of the episode, describe the plans and show the updated graph after exploration.
Revised: We visualize the progression of the episode, describe the plans and show the updated graph after exploration.

Original: 6. C OMPARING TO S EMANTIC -A WARE A GENTS WITHOUT A G RAPH R EPRESENTATION
Revised: 6. C OMPARING TO S EMANTIC -A WARE A GENTS WITHOUT A G RAPH R EPRESENTATION

Original: Here we consider two semantic-aware agents that also takes the semantic signals as input.
Revised: Here we consider two semantic-aware agents that also takes the semantic signals as input.

Original: Note that even though the LSTM controller has two orders of magnitudes more parameters than our semantic model M , our LEAPS agent still performs better, especially for faraway targets

Original: Semantic augmented agents: We train new sub-policies µ s ( θ s ) taking both s o and s s as input.
Revised: Semantic augmented agents: We train new sub-policies µ s ( θ s ) taking both s o and s s as input.

Original: HRL agents with a RNN controller: Note that updating and planning on M (Eq. 2) only depend on (1) the current semantic signal s s , (2) the target T i , and (3) the accumulative bit-
Revised: HRL agents with a RNN controller: Note that updating and planning on M (Eq. 2) only depend on (1) the current semantic signal s s , (2) the target T i , and (3) the accumulative bit-

Original: OR feature B .
Revised: OR feature B .

Original: Hence, we ﬁxed the same set of sub-policies µ ( θ ) used by our LEAPS agent, and train an LSTM controller with 50 hidden units on E train that takes all the necessary semantic information, and produce a sub-target every N steps.
Revised: Hence, we ﬁxed the same set of sub-policies µ ( θ ) used by our LEAPS agent, and train an LSTM controller with 50 hidden units on E train that takes all the necessary semantic information, and produce a sub-target every N steps.

Original: Training details are in Appendix G. Note that the only difference between our LEAPS agent and this HRL agent is the representation of the planning module.
Revised: Training details are in Appendix I. Note that the only difference between our LEAPS agent and this HRL agent is the representation of the planning module.

Original: The LSTM controller has access to exactly the same semantic information as our model M and uses a much more complicated neural model.
Revised: The LSTM controller has access to exactly the same semantic information as our model M and uses a much more complicated neural model.

Original: Thus we expect it to perform competitively to our LEAPS agent.
Revised: Thus we expect it to perform competitively to our LEAPS agent.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: 4, where our LEAPS agent outperforms both baselines.
Revised: 4, where our LEAPS agent outperforms both baselines.

Original: The semantic augmented policy µ s ( θ s ) does not improve much on the original µ ( θ ) .
Revised: The semantic augmented policy µ s ( θ s ) does not improve much on the original µ ( θ ) .

Original: For the HRL agent with an
Revised: For the HRL agent with an

Original: LSTM controller, the LEAPS agent achieves higher relative improvements for faraway targets, and also has the following advantages: (1) M can be learned more efﬁciently with much fewer parameters: an LSTM with 50 hidden units has 
Revised: LSTM controller, the LEAPS agent achieves higher relative improvements for targets requiring more planning computations (i.e., plan-steps > 1), and also has the following advantages: (1) M can be learned more efﬁciently with much fewer parameters: an LSTM with 50 hidden units has over 10parameters while M ( ψ ) only has 38 parameters 3 ; (2) M can adapt to new sub-policies µ ( θ (cid:48) ) with little ﬁne-tuning ( ψ prior remains unchanged) while the LSTM controller needs to re-train; (3) the model M and the planning procedure are fully interpretable.

Original: over 10 4 parameters while M (
Revised: 6. E VALUATION U NDER M ETRICS C ONSIDERING E PISODE L ENGTH

Original: ψ ) only has 38 parameters M can adapt to new sub policies
Revised: In the previous evaluation, we only consider the metric success rate under different horizons.

Original: µ θ (cid:48) )
Revised: Indeed, another informative metric will be the episode length.

Original: with little ﬁnetuning
Revised: There are two important factors: (1) we would expect a better navigation agent to ﬁnish the semantic task in a shorter amount of actions; (2) we should assign more credits to the agent when it ﬁnishes a hard episode while less credits when ﬁnishing an easy one.

Original: ψ prior remains unchanged while the LSTM controller needs to re-
Revised: Recently, there is a new evaluation metric proposed for embodied navigation agent by Anderson et al.

Original: train; (3) the model M and the planning procedure are fully interpretable
Revised: (2018) capturing both these two factors, i.e., the Success weighted by Path

Original: In previous experiments we use the bounding box labels from House3D as the semantic signals.
Revised: Length (SPL) metric.

Original: Now we train a CNN room type classiﬁer instead to extract semantic signals from visual input.
Revised: SPL is a function considering both success rate and the path length to reach the goal from the starting point deﬁned by 1 N (cid:80) i S i

Original: The CNN classiﬁer are trained on E train and validated on E valid .
Revised: L i max ( L i ,P i ) , where N is total episodes evaluated, S i indicates whether the episode is success or not, L i is the ground truth shortest path distance in the episode,

Original: More details are in Appendix H.
Revised: P i is the number of steps the agent actually took.

Original: We compare our LEAPS agent with the semantic-aware agentsusing a CNN semantic extractor to produce s s . The results are shown in Fig.
Revised: We evaluate the performance of LEAPS agents against all baseline agents in the metric of both success rate and SPL in Figure 5.

Original: where we also include a reference LEAPS agent using the labels from House3D (purple).
Revised: Our LEAPS agent has the highest average SPL (rightmost column) with a big margin over all baseline agents in all the cases.

Original: Note that all these agents are trained with ground truth semantic signals while only use the CNN extractor during test phase.
Revised: Notably, the margin in SPL is much more signiﬁcant than the margin in pure success rate. More importantly, as the horizon increases, namely, more planning computations allowed, the SPL margin of LEAPS over the best remaining baselines strictly increases. This again indicates the effectiveness of our semantic model and shows that it indeed helps solve harder tasks, i e., ﬁnding those faraway targets requiring more planning computations.

Original: Our LEAPS agent still outperforms all the baselines and has a comparable performance comparing to  the reference LEAPS agent using the ground truth signals.
Revised: We also notice that in shorter horizons ( H ≤ 500 ), for plan-steps equal to 4, LEAPS agents have the highest success rate but relatively lower SPL. This is because that our LEAPS agent updates the semantic model every ﬁxed N = 30 steps.

Original: C ONCLUSION AND F UTURE W ORK
Revised: This relatively low update frequency may potentially increase the episode length to reach a goal that requires more planning computations. However, when the horizon is long, i.e., allowing enough planning computations, our LEAPS agents signiﬁcantly outperform all the baselines in SPL metric. It will be helpful if the LEAPS agent can learn to update the semantic model instead of updating it in a ﬁxed frequency. We leave this to future works.

Original: In this work we proposed LEAPS to improve generalization of RL agents in unseen environments with diverse room layouts and object arrangements, while the underlying semantic information is shared with the environments in which the agent is trained on.
Revised: In this work, we proposed LEAPS to improve generalization of RL agents in unseen environments with diverse room layouts and object arrangements, while the underlying semantic information is [Table] LEAPS 57.2 / 61.

Original: We adopt a graphical model over semantic signals, which are low-dimensional binary vectors.
Revised: Horizon H =random 24.3 / 17. pure µ ( θ ) 60.8 / 58. aug.

Original: During evaluation, starting from a prior obtained from the training set, the agent plans on model, explores the unknown environment, and keeps updating the semantic model after new information arrives.
Revised: µ S ( θ ) 61.3 / 50. RNN control. LEAPS 66.4 / 58. obtained from the training set, the agent plans on model, explores the unknown environment, and keeps updating the semantic model after new information arrives.

Original: For exploration, sub-policies that focus on multiple targets are pre-trained to execute primitive actions from visual input.
Revised: For exploration, sub-policies that focus on multiple targets are pre-trained to execute primitive actions from visual input.

Original: The semantic model in LEAPS is lightweight, interpretable and can be updated dynamically with little explorations.
Revised: The semantic model in LEAPS is lightweight, interpretable and can be updated dynamically with little explorations.

Original: As illustrated in the House3D environment, LEAPS works well for environments with semantic consistencies – typical of realistic domains.
Revised: As illustrated in the House3D environment, LEAPS works well for environments with semantic consistencies – typical of realistic domains.

Original: On random environments, e.g., random mazes, LEAPS degenerates to exhaustive search.
Revised: On random environments, e g., random mazes, LEAPS degenerates to exhaustive search.

Original: Our approach is general and can be applied to other tasks, such as robotics manipulations where semantic signals can be status of robot arms and object locations, or video games where we can plan on semantic signals such as the game status or current resources.
Revised: Our approach is general and can be applied to other tasks, such as robotics manipulations where semantic signals can be status of robot arms and object locations, or video games where we can plan on semantic signals such as the game status or current resources.

Original: In future work we will investigate models for more complex semantic structures.
Revised: In future work we will investigate models for more complex semantic structures

Original: A C OMPLETE N OTATIONS AND D EFINITIONS
Revised: A C OMPLETE N OTATIONS AND D EFINITIONS

Original: Environment: We consider a contextual Markov Decision Process (Hallak et al., 2015)
Revised: Environment: We consider a contextual Markov Decision Process (Hallak et al., 2015)

Original: E ( c ) deﬁned by E ( c ) =
Revised: E ( c ) deﬁned by E ( c ) =

Original: ( S , A , P ( s (cid:48) | s, a ; c ) , r ( s, a ; c )) , where S is the state space and A is the action space.
Revised: ( S , A , P ( s (cid:48) | s, a ; c ) , r ( s, a ; c )) , where S is the state space and A is the action space.

Original: c represents the objects, layouts and any other semantic information describing the environment, and is sampled from C , the distribution of possible semantic scenarios.
Revised: c represents the objects, layouts and any other semantic information describing the environment, and is sampled from C , the distribution of possible semantic scenarios.

Original: r ( s, a ; c ) denotes the reward function while P ( s (cid:48) | s, a ; c ) describes transition probability conditioned on c .
Revised: r ( s, a ; c ) denotes the reward function while P ( s (cid:48) | s, a ; c ) describes transition probability conditioned on c .

Original: For example, c can be intuitively understood as encoding the complete map for navigation, or the complete object and obstacle layouts in robotics manipulations, not known to the agent in advance, and we refer to them as the context .
Revised: For example, c can be intuitively understood as encoding the complete map for navigation, or the complete object and obstacle layouts in robotics manipulations, not known to the agent in advance, and we refer to them as the context .

Original: Semantic Signal: At each time step, the agent’s observation is a tuple ( s o , s s ) , which consists of: (a) a high-dimensional observation s o , e.g . the ﬁrst person view image, and (b) a low-dimensional semantic signal s s , which encodes semantic information.
Revised: Semantic Signal: At each time step, the agent’s observation is a tuple ( s o , s s ) , which consists of: (a) a high-dimensional observation s o , e.g. the ﬁrst person view image, and (b) a low-dimensional semantic signal s s , which encodes semantic information.

Original: Such low-dimensional discrete signals are commonly used in AI, e.g . in robotic manipulation tasks s s indicates whether the robot is holding an object; for games it is the game status of a player; in visual navigation it indicates whether the agent reached a landmark; while in the AI planning literature, s s is typically a list of predicates that describe binary properties of objects.
Revised: Such low-dimensional discrete signals are commonly used in AI, e.g. in robotic manipulation tasks s s indicates whether the robot is holding an object; for games it is the game status of a player; in visual navigation it indicates whether the agent reached a landmark; while in the AI planning literature, s s is typically a list of predicates that describe binary properties of objects.

Original: We assume s s is provided by an oracle function, which can either be directly provided by the environment or extracted by some semantic extractor.
Revised: We assume s s is provided by an oracle function, which can either be directly provided by the environment or extracted by some semantic extractor.

Original: Generalization: Let µ ( a |{ s ( t ) } t ; θ ) denote the agent’s policy parametrized by θ conditioned on the previous states { s ( t ) } t and R ( µ ( θ ); c ) denote the accumulative reward of µ ( θ ) in E ( c ) .
Revised: Generalization: Let µ ( a |{ s ( t ) } t ; θ ) denote the agent’s policy parametrized by θ conditioned on the previous states { s ( t ) } t and R ( µ ( θ ); c ) denote the accumulative reward of µ ( θ ) in E ( c ) .

Original: The objective is to ﬁnd the best policy that maximizes the expected accumulative reward E c ∼C
Revised: The objective is to ﬁnd the best policy that maximizes the expected accumulative reward E c ∼C

Original: [ R ( µ ( θ ); c, T i )] .
Revised: [ R ( µ ( θ ); c, T i )] .

Original: In practice, we sample a disjoint partition of a training set E train =
Revised: In practice, we sample a disjoint partition of a training set E train =

Original: { E ( c i ) } i and a testing set
Revised: { E ( c i ) } i and a testing set

Original: E test =
Revised: E test =

Original: { E ( c j ) } j , where { c i } and { c j } are samples from C .
Revised: { E ( c j ) } j , where { c i } and { c j } are samples from C .

Original: We train µ ( θ ) with a shaped reward r train only on E train , and measure the empirical generalization performance of the learned policy on E test with the original unshaped reward (e.g., binary reward of success or not).
Revised: We train µ ( θ ) with a shaped reward r train only on E train , and measure the empirical generalization performance of the learned policy on E test with the original unshaped reward (e.g., binary reward of success or not).

Original: B E NVIRONMENT D ETAILS
Revised: B E NVIRONMENT D ETAILS

Original: In RoomNav the 8 targets are: kitchen, living room, dining room, bedroom, bathroom, ofﬁce, garage and outdoor.
Revised: In RoomNav the 8 targets are: kitchen, living room, dining room, bedroom, bathroom, ofﬁce, garage and outdoor.

Original: We inherit the success measure of “see” from (Wu et al., 2018): the agent needs to see some corresponding object for at least 450 pixels in the input frame and stay in the target area for at least 3 time steps.
Revised: We inherit the success measure of “see” from (Wu et al., 2018): the agent needs to see some corresponding object for at least 450 pixels in the input frame and stay in the target area for at least 3 time steps.

Original: For the binary signal s s , we obtain from the bounding box information for each room provided from
Revised: For the binary signal s s , we obtain from the bounding box information for each room provided from

Original: SUNCG dataset (Song et al., 2017), which is very noisy.
Revised: SUNCG dataset (Song et al., 2017), which is very noisy.

Original: Originally the House3D environment supports 13 discrete actions.
Revised: Originally the House3D environment supports 13 discrete actions.

Original: Here we reduce it to 9 actions: large forward, forward, left-forward, right-forward, large left rotate, large right rotate, left rotate, right rotate and stay still.
Revised: Here we reduce it to 9 actions: large forward, forward, left-forward, right-forward, large left rotate, large right rotate, left rotate, right rotate and stay still.

Original: C E VALUATION D ETAILS
Revised: C E VALUATION D ETAILS

Original: We following the evaluation setup from (Wu et al., 2018) and measure the success rate on E test over 5750 test episodes, which consists of 5000 random generated conﬁgurations and 750 specialized for faraway targets to increase the conﬁdence of measured success rate.
Revised: We following the evaluation setup from (Wu et al., 2018) and measure the success rate on E test over 5750 test episodes, which consists of 5000 random generated conﬁgurations and 750 specialized for faraway targets to increase the conﬁdence of measured success rate.

Original: These 750 episodes are generated such that for each plan-distance, there are at least 500 evaluation episodes.
Revised: These 750 episodes are generated such that for each plan-distance, there are at least 500 evaluation episodes.

Original: Each test episode has a ﬁxed conﬁguration for a fair comparison between different approaches, i.e., the agent will always start from the same location with the same target in that episode.
Revised: Each test episode has a ﬁxed conﬁguration for a fair comparison between different approaches, i.e., the agent will always start from the same location with the same target in that episode.

Original: Note that we always ensure that (1) the target is connected to the birthplace of the agent, and (2) the the birthplace of the agent is never within the target room.
Revised: Note that we always ensure that (1) the target is connected to the birthplace of the agent, and (2) the the birthplace of the agent is never within the target room.

Original: D V ISUALIZATION D ETAILS
Revised: D E VALUATION R ESULTS U SING G ROUND T RUTH S EMANTIC S IGNALS

Revised: In the experiment sections, we use a CNN detector to extract the semantic signals at test time. Here we also evaluate the performance of all the approaches when using the ground truth signal from the oracle provided by the House3D environment. The results are in Figure 6, where we also include the

Revised: LEAPS agent using CNN detector as a references.

Revised: Generally, using both the ground truth signal and using the CNN detector yield comparable overall performances in both metrics of success rate and

Revised: SPL.

Revised: They all consistently outperform all the baseline methods, which indicates that our Bayesian model is indeed robust over semantic signals.

Revised: One interesting observation is that there are many cases, using CNN detector produces better results than using the ground truth signals.

Revised: We hypothesis that this is because the semantic labels in House3D is noisy and therefore a well-trained CNN detector will not be inﬂuenced by the noisy labels at test time.

Revised: [Table]

Revised: aug

Revised: [Table]

Revised: RNN

Revised: [Table]

Revised: [Table]

Revised: LEAPS

Revised: [Table]

Revised: true

Revised: [Table]

Revised: LEAPS

Revised: [Table]

Revised: Horizon H =random 24.3 / 17.

Revised: pure

Revised: [Table]

Revised: aug

Revised: [Table]

Revised: RNN

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: LEAPS (CNN) 66.4 / 58.

Revised: E E VALUATION D ETAILS ON E PISODE L ENGTH

Revised: We illustrate the ground truth shortest distance information as well as the average episode length of success episodes for all the approaches.

Revised: The results are shown in Figure 7.

Revised: Note that the average ground truth shortest path is around 46.86 steps.

Revised: Considering the fact agent has 9 actions per step as well as the strong partial observability, this indicates that our benchmark semantic navigation task is indeed challenging.

Revised: F V ISUALIZATION D ETAILS

Original: For conﬁdence interval of the measured success rate, we computed it by ﬁtting a binomial distribution.
Revised: For conﬁdence interval of the measured success rate, we computed it by ﬁtting a binomial distribution.

Original: For optimal plan steps, we ﬁrstly extract all the room locations, and then construct a graph where a vertex is a room while an edge between two vertices is the shortest distance between these two rooms.
Revised: For optimal plan steps, we ﬁrstly extract all the room locations, and then construct a graph where a vertex is a room while an edge between two vertices is the shortest distance between these two rooms.

Original: After obtaining the graph and a birthplace of the agent, we compute shortest path from the birthplace to the target on this graph to derive the optimal plan steps. 
Revised: After obtaining the graph and a birthplace of the agent, we compute shortest path from the birthplace to the target on this graph to derive the optimal plan steps. [Table] LEAPS 158.

Original: E D ETAILS FOR L EARNING N EURAL S UB -P OLICIES
Revised: G D ETAILS FOR L EARNING N EURAL S UB -P OLICIES

Original:  We utilize the same policy architecture as (Wu et al., 2018).
Revised: Hyperparameters: We utilize the same policy architecture as (Wu et al., 2018).

Original: It was mentioned in (Wu et al., 2018) that using segmentation mask + depth signals as input leads to relatively better performances for policy learning.
Revised: It was mentioned in (Wu et al., 2018) that using segmentation mask + depth signals as input leads to relatively better performances for policy learning.

Original: So we inherit this setting here.
Revised: So we inherit this setting here.

Revised: In the original House3D paper, a gated attention module is used to incorporate the target instruction.

Revised: Here, since we only have K =different sub-policies, we simply train an individual policy for each target and we empirically observe that this leads to better performances.

Original: We run A3C with γ = 0 .
Revised: We run A3C with γ = 0 .

Original: 97 , batch size 64, learning rate 0.001 with Adam, weight decay 10 − 5 , entropy bonus 0.1.
Revised: 97 , batch size 64, learning rate 0. with Adam, weight decay 10 − 5 , entropy bonus 0.1.

Original: We backprop through at most 30 time steps.
Revised: We backprop through at most 30 time steps.

Original: We also compute the squared l 2 norm of logits and added to the loss with a coefﬁcient 0.01.
Revised: We also compute the squared l 2 norm of logits and added to the loss with a coefﬁcient 0.01.

Original: We also normalize the advantage to mean 0 and standard deviation 1.
Revised: We also normalize the advantage to mean 0 and standard deviation 1.

Revised: Reward shaping: We used a shaped reward function similar to (Wu et al., 2018): the reward at each time step is computed by the difference of shortest paths in meters from the agent’s location to the goal after taking a action.

Revised: We also add a time penalty of 0.1 and a collision penalty of 0.3.

Revised: When the agent reaches the goal, the success reward is 10.

Original:  We run a curriculum learning by increasing the maximum of distance between agent’s birth meters and target by 3 meters every 10000 iterations.
Revised: Curriculum learning: We run a curriculum learning by increasing the maximum of distance between agent’s birth meters and target by 3 meters every 10000 iterations.

Original: We totally run 60000 training iterations and use the ﬁnal model as our learned policy µ ( θ ) .
Revised: We totally run 60000 training iterations and use the ﬁnal model as our learned policy µ ( θ ) .

Original: In the original House3D paper, a gated attention module is used to incorporate the target instruction.

Original: Here, since we only have K = 8 different sub-policies, we simply train an individual policy for each target and we empirically observe that this leads to better performances.

Original: F D ETAILS FOR L EARNING THE S EMANTIC M ODEL
Revised: H D ETAILS FOR L EARNING THE S EMANTIC M ODEL

Original: After evalution on the validation set, we choose to run random exploration for 300 steps to collect a sample of z .
Revised: After evalution on the validation set, we choose to run random exploration for 300 steps to collect a sample of z .

Original: For a particular environment, we collect totally 50 samples for each z i,j .
Revised: For a particular environment, we collect totally 50 samples for each z i,j .

Original: For all i (cid:54) =
Revised: For all i (cid:54) =

Original: j ,we set ψ obs i,j, 0 = 0 . 001 and ψ obs i,j, 1 = 0 .
Revised: j ,

Original: G A DDITIONAL D ETAILS FOR T RAINING S EMANTIC -A WARE P OLICIES
Revised: I A DDITIONAL D ETAILS FOR T RAINING S EMANTIC -A WARE P OLICIES

Original: For the LSTM controller, we ran A2C with batch size 32, learning rate 0.001 with adam, weight decay 0.00001, gamma 0 .
Revised: For the LSTM controller, we ran A2C with batch size 32, learning rate 0.001 with adam, weight decay 0.00001, gamma 0 .

Original: 99 , entropy bonus 0.01 and advantage normalization.
Revised: 99 , entropy bonus 0.01 and advantage normalization.

Original: The reward function is designed as follows: for every subtask it propose, it gets a time penalty of 0.1; when the agent reach the target, it gets a success bonus of 2.
Revised: The reward function is designed as follows: for every subtask it propose, it gets a time penalty of 0.1; when the agent reach the target, it gets a success bonus of 2.

Original: The input of the LSTM controller consists of (1) s s ( t ) ( K bits), (2) B ( K bits), (3) last subtask T k , and (4) the ﬁnal target T i .
Revised: The input of the LSTM controller consists of (1) s s ( t ) ( K bits), (2) B ( K bits), (3) last subtask T k , and (4) the ﬁnal target T i .

Original: We convert T i and T k to a one-hot vector and combine the other two features to feed into the LSTM.
Revised: We convert T i and T k to a one-hot vector and combine the other two features to feed into the LSTM.

Original: Hence the input dimension of LSTM controller is 4 K , namely 32 in
Revised: Hence the input dimension of LSTM controller is 4 K , namely 32 in

Original: RoomNav.
Revised: RoomNav.

Original: For the semantic augmented LSTM policy, µ s ( θ s ) , we ﬁrstly use the CNN extract visual features from s o and combine the input semantic features and the visual features as the combined input to the
Revised: For the semantic augmented LSTM policy, µ s ( θ s ) , we ﬁrstly use the CNN extract visual features from s o and combine the input semantic features and the visual features as the combined input to the

Original: LSTM in the policy.
Revised: LSTM in the policy.

Original: H A DDITIONAL D ETAILS FOR T RAINING THE CNN S EMANTIC E XTRACTOR
Revised: J A DDITIONAL D ETAILS FOR T RAINING THE CNN S EMANTIC E XTRACTOR

Original: We noticed that in order to have a room type classiﬁer, only using the single ﬁrst person view image is not enough.
Revised: We noticed that in order to have a room type classiﬁer, only using the single ﬁrst person view image is not enough.

Original: For example, the agent may face towards a wall, which is not informative, but is indeed inside the bedroom (and the bed is just behind).
Revised: For example, the agent may face towards a wall, which is not informative, but is indeed inside the bedroom (and the bed is just behind).

Original: So we take the panoramic view as input, which consists of 4 images, s 1 o , . . .
Revised: So we take the panoramic view as input, which consists of 4 images, s 1 o , . . .

Original: , s 4 o with different ﬁrst person view angles.
Revised: , s 4 o with different ﬁrst person view angles.

Original: The only exception is that for target “outdoor”, we notice that instead of using a panoramic view, simply keeping the recent 4 frames in the trajectory leads to the best prediction accuracy.
Revised: The only exception is that for target “outdoor”, we notice that instead of using a panoramic view, simply keeping the recent 4 frames in the trajectory leads to the best prediction accuracy.

Original: We use an CNN feature extractor to extract features f ( s io ) by applying CNN layers with kernel size 3, strides [1 , 1 , 1 , 2 , 1 , 2 , 1 , 2 , 1 , 2] and channels [4 , 8 , 16 , 16 , 32 , 32 , 64 , 64 , 128 , 256] .
Revised: We use an CNN feature extractor to extract features f ( s io ) by applying CNN layers with kernel size 3, strides [1 , 1 , 1 , 2 , 1 , 2 , 1 , 2 , 1 , 2] and channels [4 , 8 , 16 , 16 , 32 , 32 , 64 , 64 , 128 , 256] .

Original: We also use relu activation and batch norm.
Revised: We also use relu activation and batch norm.

Original: Then we compute the attention weights over these 4 visual features by l i = f
Revised: Then we compute the attention weights over these 4 visual features by l i = f

Original: ( s io ) W T 1 W 2 (cid:2) f
Revised: ( s io ) W T 1 W 2 (cid:2) f

Original: ( s 1 o ) , . . .
Revised: ( s 1 o ) , . . .

Original: , f ( s 4 o ) (cid:3) and a i = softmax ( l i ) .
Revised: , f ( s 4 o ) (cid:3) and a i = softmax ( l i ) .

Original: Then we compute the weighted average of these four frames g =
Revised: Then we compute the weighted average of these four frames g =

Original: (cid:80)
Revised: (cid:80)

Original: i a i f ( s io ) and feed it to a single layer perceptron with 32 hidden units.
Revised: i a i f ( s io ) and feed it to a single layer perceptron with 32 hidden units.

Original: For each semantic signal, we generate 15k positive and 15k negative training data from E train and use Adam optimizer with learning rate 5 e - 4 , weight decay 1 e - 5 , batch size 256 and gradient clip of 5.
Revised: For each semantic signal, we generate 15k positive and 15k negative training data from E train and use Adam optimizer with learning rate 5 e - 4 , weight decay 1 e - 5 , batch size 256 and gradient clip of 5.

Original: We keep the model that has the best prediction accuracy on E valid .
Revised: We keep the model that has the best prediction accuracy on E valid .

Original: For a smooth prediction during testing, we also have a hard threshold and ﬁltering process on the
Revised: For a smooth prediction during testing, we also have a hard threshold and ﬁltering process on the

Original: CNN outputs: s s ( T i ) will be 1 only if the output of CNN has conﬁdence over 0.85 for consecutively 3 steps.
Revised: CNN outputs: s s ( T i ) will be 1 only if the output of CNN has conﬁdence over 0.85 for consecutively 3 steps.