Original: L EARNING TO R EASON : D ISTILLING H IERARCHY VIA S ELF -S UPERVISION AND R EINFORCEMENT L EARNING Anonymous authors
Revised: L EARNING TO R EASON : D ISTILLING H IERARCHY VIA S ELF -S UPERVISION AND R EINFORCEMENT L EARNING Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task ﬂexibly.
Revised: We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task ﬂexibly.

Original: Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by self-supervision and reinforcement learning.
Revised: Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by self-supervision and reinforcement learning.

Original: The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables.
Revised: The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables.

Original: The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention.
Revised: The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention.

Original: Because the reasoning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training.
Revised: Because the reasoning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training.

Original: We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments.
Revised: We demonstrate the proposed framework can learn compact representations (3- and 1-dimensional latent states and commands for a humanoid with 197- and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments.

Original: The supplementary video is available at: https://bit.ly/2rwIfQn
Revised: The supplementary video is available at: https://bit.ly/2rwIfQn

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) aims to compute the optimal control policy while an agent interacts with the environment.
Revised: Reinforcement learning (RL) aims to compute the optimal control policy while an agent interacts with the environment.

Original: Recent advances in deep learning enable RL frameworks to utilize deep neural networks to efﬁciently represent and learn a policy having a ﬂexible and expressive structure.
Revised: Recent advances in deep learning enable RL frameworks to utilize deep neural networks to efﬁciently represent and learn a policy having a ﬂexible and expressive structure.

Original: As a result, we’ve been witnessing RL agents that already achieved or even exceeded human-level performances in particular tasks (Mnih et al., 2015; Silver et al., 2017).
Revised: As a result, we’ve been witnessing RL agents that already achieved or even exceeded human-level performances in particular tasks (Mnih et al., 2015; Silver et al., 2017).

Original: The core of intelligence, however, is not just to learn a policy for a particular problem instance, but to solve various multiple tasks or immediately adapt to a new task.
Revised: The core of intelligence, however, is not just to learn a policy for a particular problem instance, but to solve various multiple tasks or immediately adapt to a new task.

Original: Given that a huge computational burden makes it unrealistic to learn an individual policy for each task, an agent should be able to reason about its action.
Revised: Given that a huge computational burden makes it unrealistic to learn an individual policy for each task, an agent should be able to reason about its action.

Original: If predictions about consequences of actions are available, e.g., by using an internal model (Ha &
Revised: If predictions about consequences of actions are available, e.g., by using an internal model (Ha &

Original: Schmidhuber, 2018; Kaiser et al., 2019), an intelligent agent can plan a sequence of its actions.
Revised: Schmidhuber, 2018; Kaiser et al., 2019), an intelligent agent can plan a sequence of its actions.

Original: Involving planning procedures in a control policy could provide adaptiveness to an agent, but it is often not trivial to learn such a prediction & planning framework:
Revised: Involving planning procedures in a control policy could provide adaptiveness to an agent, but it is often not trivial to learn such a prediction & planning framework:

Original: First, it is difﬁcult to obtain the exact internal dynamic model directly represented in high-dimensional state (observation) space.
Revised: First, it is difﬁcult to obtain the exact internal dynamic model directly represented in high-dimensional state (observation) space.

Original: Model errors inevitably become larger in the high-dimensional space and are accumulated along the prediction/planning horizon.
Revised: Model errors inevitably become larger in the high-dimensional space and are accumulated along the prediction/planning horizon.

Original: This prohibits planning methods from producing a valid prediction and so a sensible plan.
Revised: This prohibits planning methods from producing a valid prediction and so a sensible plan.

Original: Second, and perhaps more importantly, planning methods cannot help but relying on some dynamic programming or search procedures, which quickly become intractable for problems with high degrees of freedom (DOFs) because the size of search space grows exponentially with DOFs, i.e., the curse of dimensionality (LaValle, 2006).
Revised: Second, and perhaps more importantly, planning methods cannot help but relying on some dynamic programming or search procedures, which quickly become intractable for problems with high degrees of freedom (DOFs) because the size of search space grows exponentially with DOFs, i.e., the curse of dimensionality (LaValle, 2006).

Original: Crucial evidence found in the cognitive science ﬁeld is that there exists a certain type of hierarchical structure in the humans’ motor control scheme addressing the aforementioned fundamental difﬁculty (Todorov & Ghahramani, 2003; Todorov, 2004).
Revised: Crucial evidence found in the cognitive science ﬁeld is that there exists a certain type of hierarchical structure in the humans’ motor control scheme addressing the aforementioned fundamental difﬁculty (Todorov & Ghahramani, 2003; Todorov, 2004).

Original: Such a hierarchical structure is known to utilize two levels of parallel control loops, operating in different time scales; in a coarser scale, the high-level loop generates task-relevant commands for the agent to perform a given task, and then in a ﬁner time scale, the low-level loop maps those commands into control signals while actively reacting to disturbances that the high-level loop could not consider (e.g., the spinal cord)
Revised: Such a hierarchical structure is known to utilize two levels of parallel control loops, operating in different time scales; in a coarser scale, the high-level loop generates task-relevant commands for the agent to perform a given task, and then in a ﬁner time scale, the low-level loop maps those commands into control signals while actively reacting to disturbances that the high-level loop could not consider (e.g., the spinal cord)

Original: (Todorov
Revised: (Todorov

Original: Ghahramani, 2003).
Revised: Ghahramani, 2003).

Original: Because the low-level loop does not passively generate control signals from high-level commands, the high-level loop is able to focus only on the task-relevant aspects of the environment dynamics that can be represented in a low-dimensional form.
Revised: Because the low-level loop does not passively generate control signals from high-level commands, the high-level loop is able to focus only on the task-relevant aspects of the environment dynamics that can be represented in a low-dimensional form.

Original: Consequently, this hierarchical structure allows us for efﬁciently predicting and planning the future states to compute the commands.
Revised: Consequently, this hierarchical structure allows us for efﬁciently predicting and planning the future states to compute the commands.

Original: High-level Planner 𝑞𝑞 ( 𝒉𝒉 | 𝒔𝒔 )
Revised: High-level Planner 𝑞𝑞 ( 𝒉𝒉 | 𝒔𝒔 )

Original: 𝒔𝒔 𝒔𝒔 𝒂𝒂
Revised: 𝒔𝒔 𝒔𝒔 𝒂𝒂

Original: 𝒉𝒉 Low-level control policy 𝜋𝜋 ( 𝒂𝒂 | 𝒔𝒔 , 𝒉𝒉 )
Revised: 𝒉𝒉 Low-level control policy 𝜋𝜋 ( 𝒂𝒂 | 𝒔𝒔 , 𝒉𝒉 )

Original: Environment
Revised: Environment

Original: Internal Model 𝑝𝑝 𝜙𝜙 ( 𝒔𝒔 | 𝒉𝒉 ) � r �𝒔𝒔 � r � r �𝒔𝒔 �𝒔𝒔 ..
Revised: Internal Model 𝑝𝑝 𝜙𝜙 ( 𝒔𝒔 | 𝒉𝒉 ) � r �𝒔𝒔 � r � r �𝒔𝒔 �𝒔𝒔 ..

Original: T
Revised: T

Original: T T
Revised: T T

Original: Motivated by this evidence, we propose a framework, termed "DISH", that DIStills a Hierarchical structure for reasoning and control.
Revised: Motivated by this evidence, we propose a framework, termed "DISH", that DIStills a Hierarchical structure for reasoning and control.

Original: As depicted in Fig.
Revised: As depicted in Fig.

Original: 1, the proposed framework has two levels of hierarchy.
Revised: 1, the proposed framework has two levels of hierarchy.

Original: The high-level loop represents an agent’s current state as a lowdimensional latent state and generates/reasons task-relevant high-level commands by predicting and planning the future in the latent space.
Revised: The high-level loop represents an agent’s current state as a lowdimensional latent state and generates/reasons task-relevant high-level commands by predicting and planning the future in the latent space.

Original: The low-level loop receives the high-level commands as well as the current states and maps them into the high-dimensional control signal.
Revised: The low-level loop receives the high-level commands as well as the current states and maps them into the high-dimensional control signal.

Original: Two different types of learning are required to build such a framework: (i) a lowdimensional latent representation for an internal model should be obtained from agent’s own experiences via self-supervised learning ; (ii) a control policy should be learned while interacting with the environment via reinforcement learning .
Revised: Two different types of learning are required to build such a framework: (i) a lowdimensional latent representation for an internal model should be obtained from agent’s own experiences via self-supervised learning ; (ii) a control policy should be learned while interacting with the environment via reinforcement learning .

Original: We combined these two learning problems by transforming a multitask RL problem into generative model learning using the control-inference duality (Levine, 2018; Todorov, 2008; Rawlik et al., 2012).
Revised: We combined these two learning problems by transforming a multitask RL problem into generative model learning using the control-inference duality (Levine, 2018; Todorov, 2008; Rawlik et al., 2012).

Original: In this perspective, an agent equipped with a low-level control policy is viewed as a generative model that outputs trajectories according to high-level commands.
Revised: In this perspective, an agent equipped with a low-level control policy is viewed as a generative model that outputs trajectories according to high-level commands.

Original: Reasoning the high-level commands is then considered as a posterior inference problem; we introduce a low-dimensional internal model to make this inference tractable.
Revised: Reasoning the high-level commands is then considered as a posterior inference problem; we introduce a low-dimensional internal model to make this inference tractable.

Original: We demonstrate that the proposed framework can learn the compact representation (3-dimensional latent states for a humanoid robot having 90-dimensional states) and the control policy while solving a small number of imitation tasks, and the learned planning and control scheme is immediately applicable to new tasks, e.g., navigation through a cluttered environment.
Revised: We demonstrate that the proposed framework can learn the compact representation (3-dimensional latent states for a humanoid robot having 90-dimensional states) and the control policy while solving a small number of imitation tasks, and the learned planning and control scheme is immediately applicable to new tasks, e.g., navigation through a cluttered environment.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Hierarchical RL: To apply task-speciﬁc policies learned from individual RL problems to various tasks, hierarchical structures are often considered where each learned policy serves as a low-level controlller, i.e., as a "skill", and a high-level controller selects which skills to perform in the context the agent lies at (Peng et al., 2018; 2019; Merel et al., 2019a; Lee et al., 2019).
Revised: Hierarchical RL: To apply task-speciﬁc policies learned from individual RL problems to various tasks, hierarchical structures are often considered where each learned policy serves as a low-level controlller, i.e., as a "skill", and a high-level controller selects which skills to perform in the context the agent lies at (Peng et al., 2018; 2019; Merel et al., 2019a; Lee et al., 2019).

Original: Peng et al.
Revised: Peng et al.

Original: (2018; 2019) trained robust control policies for imitating a broad range of example motion clips and integrated multiple skills into a composite policy capable of executing various tasks.
Revised: (2018; 2019) trained robust control policies for imitating a broad range of example motion clips and integrated multiple skills into a composite policy capable of executing various tasks.

Original: Merel et al. (2019a) similarly trained many imitation policies and utilized them as individual skills that a high-level controller chooses based on the visual inputs.
Revised: Merel et al. (2019a) similarly trained many imitation policies and utilized them as individual skills that a high-level controller chooses based on the visual inputs.

Original: Lee et al.
Revised: Lee et al.

Original: (2019) included transition policies which help the agent smoothly switch between the skills.
Revised: (2019) included transition policies which help the agent smoothly switch between the skills.

Original: Another line of approaches is using continuous-valued latent variables to represent skills (Co-Reyes et al., 2018;
Revised: Another line of approaches is using continuous-valued latent variables to represent skills (Co-Reyes et al., 2018;

Original: Gupta et al., 2018;
Revised: Gupta et al., 2018;

Original: Eysenbach et al., 2019;
Revised: Eysenbach et al., 2019;

Original: Florensa et al., 2017; Hausman et al., 2018).
Revised: Florensa et al., 2017; Hausman et al., 2018).

Original: Co-Reyes et al.
Revised: Co-Reyes et al.

Original: (2018) proposed an autoencoder-like framework where an encoder compresses trajectories into latent variables, a state decoder reconstructs trajectories, and a policy decoder provides a control policy to follow the reconstructed trajectory.
Revised: (2018) proposed an autoencoder-like framework where an encoder compresses trajectories into latent variables, a state decoder reconstructs trajectories, and a policy decoder provides a control policy to follow the reconstructed trajectory.

Original: Gupta et al.
Revised: Gupta et al.

Original: Eysenbach et al.
Revised: Eysenbach et al.

Original: ; Florensa et al.
Revised: ; Florensa et al.

Original: (2017) also introduced latent variables to efﬁciently represent various policies.
Revised: (2017) also introduced latent variables to efﬁciently represent various policies.

Original: Instead of using one static latent variable, Merel et al.
Revised: Instead of using one static latent variable, Merel et al.

Original: (2019b) proposed a framework that encodes expert’s demonstrations as latent trajectories and infers a latent trajectory from an unseen skill for one-shot imitation.
Revised: (2019b) proposed a framework that encodes expert’s demonstrations as latent trajectories and infers a latent trajectory from an unseen skill for one-shot imitation.

Original: Haarnoja et al.
Revised: Haarnoja et al.

Original: (2018a) proposed a hierarchical structure for RL problems where marginalization of low-level actions provides a new system for high-level action.
Revised: (2018a) proposed a hierarchical structure for RL problems where marginalization of low-level actions provides a new system for high-level action.

Original: In their framework, policies at all levels can be learned with different reward functions such that a high-level policy becomes easier to be optimized from the marginalization.
Revised: In their framework, policies at all levels can be learned with different reward functions such that a high-level policy becomes easier to be optimized from the marginalization.

Original: Note that the above hierarchical RL approaches train the high-level policy by solving another
Revised: Note that the above hierarchical RL approaches train the high-level policy by solving another

Original: RL problem; because the individual skill or the latent variables compress dynamics of the agent, variations of them provide efﬁcient exploration for the high-level RL.
Revised: RL problem; because the individual skill or the latent variables compress dynamics of the agent, variations of them provide efﬁcient exploration for the high-level RL.

Original: Our framework also considers low-dimensional and continuous latent trajectories to represent various policies.
Revised: Our framework also considers low-dimensional and continuous latent trajectories to represent various policies.

Original: Rather than learning a high-level policy, however, our framework learns an internal model with which the high-level module performs reasoning; the agent can efﬁciently reason its high-level commands by searching the low-dimensional latent space with the learned internal model.
Revised: Rather than learning a high-level policy, however, our framework learns an internal model with which the high-level module performs reasoning; the agent can efﬁciently reason its high-level commands by searching the low-dimensional latent space with the learned internal model.

Original: The learned planning/control structure is then directly applicable to new sets of tasks the agent hasn’t met during training.
Revised: The learned planning/control structure is then directly applicable to new sets of tasks the agent hasn’t met during training.

Original: Only a few recent works (Hafner et al., 2019;
Revised: Only a few recent works (Hafner et al., 2019;

Original: Sharma et al., 2019) incorporated reasoning processes into high-level modules, but neither of them exploits low-dimensional latent space for planning (Sharma et al., 2019) nor low-dimensional commands (Hafner et al., 2019).
Revised: Sharma et al., 2019) incorporated reasoning processes into high-level modules, but neither of them exploits low-dimensional latent space for planning (Sharma et al., 2019) nor low-dimensional commands (Hafner et al., 2019).

Original: Our ablation study in Section 4.1 shows the effectiveness of utilizing both latent states and commands and, to our best knowledge, DISH is the ﬁrst framework doing so.
Revised: Our ablation study in Section 4.1 shows the effectiveness of utilizing both latent states and commands and, to our best knowledge, DISH is the ﬁrst framework doing so.

Original: Model-based RL & Learning to Plan: Model-based RL algorithms attempt to learn the agent’s dynamics and utilize the planning and control methods to perform tasks (Williams et al., 2017;
Revised: Model-based RL & Learning to Plan: Model-based RL algorithms attempt to learn the agent’s dynamics and utilize the planning and control methods to perform tasks (Williams et al., 2017;

Original: Deisenroth et al., 2015; Chua et al., 2018).
Revised: Deisenroth et al., 2015; Chua et al., 2018).

Original: Williams et al.
Revised: Williams et al.

Original: ; Chua et al.
Revised: ; Chua et al.

Original: (2018) utilized deep neural networks to model the dynamics and adopted the model predictive control method on the learned dynamics; Deisenroth et al.
Revised: (2018) utilized deep neural networks to model the dynamics and adopted the model predictive control method on the learned dynamics; Deisenroth et al.

Original: (2015) used the Gaussian processes as system dynamics, which leads to the efﬁcient and stable policy search.
Revised: (2015) used the Gaussian processes as system dynamics, which leads to the efﬁcient and stable policy search.

Original: Though these methods have shown impressive results, they are not directly applicable to systems having high DOFs because high-dimensional modeling is hard to be exact and even advanced planning and control methods are not very scalable to such systems.
Revised: Though these methods have shown impressive results, they are not directly applicable to systems having high DOFs because high-dimensional modeling is hard to be exact and even advanced planning and control methods are not very scalable to such systems.

Original: One exceptional work was proposed by Ha & Schmidhuber (2018), where the variational autoencoder and the recurrent neural network are combined to model the dynamics of the observation.
Revised: One exceptional work was proposed by Ha & Schmidhuber (2018), where the variational autoencoder and the recurrent neural network are combined to model the dynamics of the observation.

Original: They showed that a simple linear policy w.r.t the low-dimensional latent state can control the low
Revised: They showed that a simple linear policy w.r.t the low-dimensional latent state can control the low

Original: DOFs agent, but (i) high-DOFs systems require a more complicated policy structure to output highdimensional actions and (ii) reasoning (or planning) by predicting the future is essential to solve a set of complex tasks.
Revised: DOFs agent, but (i) high-DOFs systems require a more complicated policy structure to output highdimensional actions and (ii) reasoning (or planning) by predicting the future is essential to solve a set of complex tasks.

Original: On the other hand, Ha et al.
Revised: On the other hand, Ha et al.

Original: (2018a;b) trained the low-dimensional latent dynamics from expert’s demonstrations and generated motion plans using the learned dynamics; the high-dimensional motion plans were able to be computed efﬁciently, but the control policy for executing those plans was not considered.
Revised: (2018a;b) trained the low-dimensional latent dynamics from expert’s demonstrations and generated motion plans using the learned dynamics; the high-dimensional motion plans were able to be computed efﬁciently, but the control policy for executing those plans was not considered.

Original: Some recent works have attempted to build the policy network in such way that resembles the advanced planning and optimal control methods: Tamar et al.
Revised: Some recent works have attempted to build the policy network in such way that resembles the advanced planning and optimal control methods: Tamar et al.

Original: (2016) encoded the value iteration procedures into the network; Okada et al.
Revised: (2016) encoded the value iteration procedures into the network; Okada et al.

Original: ; Amos et al.
Revised: ; Amos et al.

Original: (2018) wired the network so as to resemble the path-integral control and the iterative LQR methods, respectively.
Revised: (2018) wired the network so as to resemble the path-integral control and the iterative LQR methods, respectively.

Original: The whole policy networks are trained end-to-end and, interestingly, system dynamics and a cost function emerge during the learning procedure.
Revised: The whole policy networks are trained end-to-end and, interestingly, system dynamics and a cost function emerge during the learning procedure.

Original: However, these methods were basically designed just to mimic the expert’s behaviors, i.e., addressing inverse RL problems, and also tried to ﬁnd the control policy directly in the (possibly high-dimensional) state space.
Revised: However, these methods were basically designed just to mimic the expert’s behaviors, i.e., addressing inverse RL problems, and also tried to ﬁnd the control policy directly in the (possibly high-dimensional) state space.

Original:  DISH: DIS TILLING H IERARCHY FOR P LANNING AND C ONTROL 3. M ULTITASK RL AS L ATENT V ARIABLE M ODEL L EARNING
Revised:  DISH: DIS TILLING H IERARCHY FOR P LANNING AND C ONTROL 3. M ULTITASK RL AS L ATENT V ARIABLE M ODEL L EARNING

Original: Suppose that a dynamical system with states s ∈ S is controlled by actions a ∈ A , where the states evolve with the stochastic dynamics p ( s k +

Revised: Suppose that a dynamical system with states s ∈ S is controlled by actions a ∈ A , where the states evolve with the stochastic dynamics p ( s k +

Original: |
Revised: |

Original: s k , a k ) from the initial states p ( s 1 ) .
Revised: s k , a k ) from the initial states p ( s 1 ) .

Original: Let ˜ r k
Revised: Let ˜ r k

Original: ( s k , a k ) denote a reward function that the agent wants to maximize with the control policy π θ ( a k | s k ) .
Revised: ( s k , a k ) denote a reward function that the agent wants to maximize with the control policy π θ ( a k | s k ) .

Original: Reinforcement learning problems are then formulated as the following optimization problem:
Revised: Reinforcement learning problems are then formulated as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where the controlled trajectory distribution q θ is given by:
Revised: where the controlled trajectory distribution q θ is given by:

Original: [Equation]
Revised: [Equation]

Original: By introducing an artiﬁcial binary random variable o t , called the optimality variable , whose emission probability is given by exponential of a state-dependent reward, i.e. p ( O k = 1 | s k ) =
Revised: By introducing an artiﬁcial binary random variable o t , called the optimality variable , whose emission probability is given by exponential of a state-dependent reward, i.e. p ( O k = 1 | s k ) =

Original: exp ( r k ( s k )) , and by deﬁning an appropriate action prior p ( a ) and corresponding the uncontrolled trajectory distribution, p ( s 1: K , a 1: K ) ≡ p ( s 1 ) (cid:81)
Revised: exp ( r k ( s k )) , and by deﬁning an appropriate action prior p ( a ) and corresponding the uncontrolled trajectory distribution, p ( s 1: K , a 1: K ) ≡ p ( s 1 ) (cid:81)

Original: Kk =1 p ( s k +1 | s k , a k ) p ( a k ) , we can view the above RL problem as a probabilistic inference problem for a graphical model in Fig 2(a).
Revised: Kk =1 p ( s k +1 | s k , a k ) p ( a k ) , we can view the above RL problem as a probabilistic inference problem for a graphical model in Fig 2(a).

Original: The objective of such an
Revised: The objective of such an

Original: 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐚𝐚 𝐚𝐚𝑂 (a) 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐚𝐚 𝐚𝐚𝑂 𝐡𝐡 𝐡𝐡𝑂 (b) 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐡𝐡 𝐡𝐡𝑂 (c) 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐳𝐳 𝐳𝐳𝑂 𝐡𝐡 𝐡𝐡𝑂 (d) inference problem is to ﬁnd the optimal variational parameter, θ , such that the controlled trajectory distribution q θ ( s 1: K , a 1: K ) ﬁts the posterior distribution p ( s 1: K , a 1: K | O 1: K = 1) best.
Revised: 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐚𝐚 𝐚𝐚𝑂 (a) 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐚𝐚 𝐚𝐚𝑂 𝐡𝐡 𝐡𝐡𝑂 (b) 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐡𝐡 𝐡𝐡𝑂 (c) 𝑂𝑂 𝑂𝑂𝑂 𝐬𝐬 𝐬𝐬𝑂 𝐳𝐳 𝐳𝐳𝑂 𝐡𝐡 𝐡𝐡𝑂 (d) inference problem is to ﬁnd the optimal variational parameter, θ , such that the controlled trajectory distribution q θ ( s 1: K , a 1: K ) ﬁts the posterior distribution p ( s 1: K , a 1: K | O 1: K = 1) best.

Original: More detailed derivations of this duality can be found in Appendix A.2 or in the tutorial paper (Levine, 2018).
Revised: More detailed derivations of this duality can be found in Appendix A.2 or in the tutorial paper (Levine, 2018).

Original: Rather than solving one particular task, i.e., one reward function, agents are often required to perform various tasks.
Revised: Rather than solving one particular task, i.e., one reward function, agents are often required to perform various tasks.

Original: Let T be a set of tasks, and π θ ∗ t ( a t | s t ) be the optimal policy for t th task, i.e.,
Revised: Let T be a set of tasks, and π θ ∗ t ( a k | s k ) be the optimal policy for t th task, i.e.,

Original: [Equation]
Revised: [Equation]

Original: For high DOF systems, where policies π θ t represent a mapping from a high-dimensional state space to a high-dimensional action space, individually optimizing each policy is computationally too expensive.
Revised: For high DOF systems, where policies π θ t represent a mapping from a high-dimensional state space to a high-dimensional action space, individually optimizing each policy is computationally too expensive.

Original: Instead of doing so, we can assume that tasks the agent needs to perform require similar solution properties and consequently the optimal policies have some sort of common structures.
Revised: Instead of doing so, we can assume that tasks the agent needs to perform require similar solution properties and consequently the optimal policies have some sort of common structures.

Original: We can then introduce a low-dimensional latent variable h ( t ) that, by compressing a particular aspect of π θ t over all the policies, each policy can be conditioned on as π θ ( a k | s k , h ( t ) ) .
Revised: We can then introduce a low-dimensional latent variable h ( t ) that, by compressing a particular aspect of π θ t over all the policies, each policy can be conditioned on as π θ ( a k | s k , h ( t ) ) .

Original: Such a hierarchical structure is depicted as Fig. 2(b), where h can be interpreted as high-level commands .
Revised: Such a hierarchical structure is depicted as Fig. 2(b), where h can be interpreted as high-level commands .

Original: We can then deﬁne the uncontrolled and the task t ’s controlled trajectory distributions as
Revised: We can then deﬁne the uncontrolled and the task t ’s controlled trajectory distributions as

Original: [Equation]
Revised: [Equation]

Original: receptively.
Revised: receptively.

Original: In other words, the control policy π θ is shared across all the tasks, actively mapping high-level commands h , into actual actions, a .
Revised: In other words, the control policy π θ is shared across all the tasks, actively mapping high-level commands h , into actual actions, a .

Original: Only high-level commands vary with the given task speciﬁcations.
Revised: Only high-level commands vary with the given task speciﬁcations.

Original: In the perspective of control as inference , a corresponding inference problem now has two parts: one for the policy parameter θ and another for the task-speciﬁc commands h .
Revised: In the perspective of control as inference , a corresponding inference problem now has two parts: one for the policy parameter θ and another for the task-speciﬁc commands h .

Original: Note that, if a high-level policy ¯ π θ ( h | s ) is used to compute high-level commands, the learning problem then becomes the standard Hierarchical RL (HRL).
Revised: Note that, if a high-level policy ¯ π θ ( h | s ) is used to compute high-level commands, the learning problem then becomes the standard Hierarchical RL (HRL).

Original: We instead introduce a reasoning module to generate high-level commands which infers the optimal h for a given task t and a current state s by predicting futures.
Revised: We instead introduce a reasoning module to generate high-level commands which infers the optimal h for a given task t and a current state s by predicting futures.

Original: As often used in many HRL methods, the high-level module of the proposed framework operates in a coarser time scale than the low-level policy does.
Revised: As often used in many HRL methods, the high-level module of the proposed framework operates in a coarser time scale than the low-level policy does.

Original: Similar to the latent model learning in Appendix A.3 and the control-inference duality in Appendix
Revised: Similar to the latent model learning in Appendix A.3 and the control-inference duality in Appendix

Original: A.2, we can derive the following lower-bound of optimality likelihood L ( t ) for a task t :
Revised: A.2, we can derive the following lower-bound of optimality likelihood L ( t ) for a task t :

Original: [Equation]
Revised: [Equation]

Original: where τ ≡ ( s 1: K , a 1: K , h 1: K ) .
Revised: where τ ≡ ( s 1: K , a 1: K , h 1: K ) .

Original: This suggests a novel learning scheme of the hierarchical policy in Equation 5: (i) For a given task t and a ﬁxed low-level policy π θ , high-level commands h k are computed via variational inference.
Revised: This suggests a novel learning scheme of the hierarchical policy in Equation 5: (i) For a given task t and a ﬁxed low-level policy π θ , high-level commands h k are computed via variational inference.

Original: This inference procedure q ( h | s ) should take predictions about future rewards into account to generate h , which can be interpreted as planning.
Revised: This inference procedure q ( h | s ) should take predictions about future rewards into account to generate h , which can be interpreted as planning.

Original: To do so, we build an internal model via self-supervised learning and perform planning with the internal model.
Revised: To do so, we build an internal model via self-supervised learning and perform planning with the internal model.

Original: (ii) With the planning module equipped, a low-level policy π θ ( a | s , h ) generates control actions a as in RL problems, which can be trained using standard deep RL algorithms (Schulman et al., 2017; Haarnoja et al., 2018b).
Revised: (ii) With the planning module equipped, a low-level policy π θ ( a | s , h ) generates control actions a as in RL problems, which can be trained using standard deep RL algorithms (Schulman et al., 2017; Haarnoja et al., 2018b).

Original: 3. S ELF - SUPERVISED L EARNING OF I NTERNAL M ODEL
Revised: 3. S ELF - SUPERVISED L EARNING OF I NTERNAL M ODEL

Original: The role of q ( h | s ) is to compute the high-level commands that would lead to maximum accumulated rewards in the future; as shown in Equation 6, this infers the commands that maximizes the likelihood of optimality variables when O 1: K = 1 were observed.
Revised: The role of q ( h | s ) is to compute the high-level commands that would lead to maximum accumulated rewards in the future; as shown in Equation 6, this infers the commands that maximizes the likelihood of optimality variables when O 1: K = 1 were observed.

Original: Given that the ELBO gap is the KL-divergence between the posterior and variational distributions, it is obvious that more exact variational inference will make the lower bound tighter, thereby directly leading to the agent’s better performance as well as the better policy learning.
Revised: Given that the ELBO gap is the KL-divergence between the posterior and variational distributions, it is obvious that more exact variational inference will make the lower bound tighter, thereby directly leading to the agent’s better performance as well as the better policy learning.

Original: What would the exact posterior be like?
Revised: What would the exact posterior be like?

Original: Fig. 2(c) shows the graphical model of the inference problem that q ( h | s ) should address, which is obtained by marginalizing actions from Fig.
Revised: Fig. 2(c) shows the graphical model of the inference problem that q ( h | s ) should address, which is obtained by marginalizing actions from Fig.

Original: 2(b); as also shown in (Haarnoja et al., 2018a), sucha marginalization results in a new system with new control input h , thus the inference problem in this level is again the RL/OC problem.
Revised: 2(b); as also shown in (Haarnoja et al., 2018a), such marginalization results in a new system with new control input h , thus the inference problem in this level is again the RL/OC problem.

Original: To get the command at the moment, h 1 , the inference procedure should compute the posterior command trajectories h 1: K by considering the dynamics and observations (the optimality variables), and marginalize the future commands h 2: K out.
Revised: To get the command at the moment, h 1 , the inference procedure should compute the posterior command trajectories h 1: K by considering the dynamics and observations (the optimality variables), and marginalize the future commands h 2: K out.

Original: Though the dimensionality of h is much lower than that of a , this inference problem is still not trivial to solve by two reasons: (i) The dynamics of states p θ ( s (cid:48) | s , h ) =
Revised: Though the dimensionality of h is much lower than that of a , this inference problem is still not trivial to solve by two reasons: (i) The dynamics of states p θ ( s (cid:48) | s , h ) =

Original: (cid:82) p ( s (cid:48) | s , a ) π θ ( a | s , h )
Revised: (cid:82) p ( s (cid:48) | s , a ) π θ ( a | s , h )

Original: d a contains the environment component of which information can be obtained only through expensive interactions with the environment.
Revised: d a contains the environment component of which information can be obtained only through expensive interactions with the environment.

Original: (ii)
Revised: (ii)

Original: One might consider building a surrogate model p φ ( s (cid:48) | s , h ) via supervised learning with transition data obtained during low-level policy learning and utilizing the learned model for inference.
Revised: One might consider building a surrogate model p φ ( s (cid:48) | s , h ) via supervised learning with transition data obtained during low-level policy learning and utilizing the learned model for inference.

Original: However, learning high-dimensional transition model is hard to be accurate and the inference (planning) in high-dimensional space is intractable because of, e.g., the curse of dimensionality (Ha et al., 2018a).
Revised: However, learning high-dimensional transition model is hard to be accurate and the inference (planning) in high-dimensional space is intractable because of, e.g., the curse of dimensionality (Ha et al., 2018a).

Original: However, we can reasonably assume that conﬁgurations that should be considered from planning form some sort of low-dimensional manifold in the original space (Vernaza & Lee, 2012), and the closed-loop system with high-level commands provides stochastic dynamics on that manifold.
Revised: However, we can reasonably assume that conﬁgurations that should be considered from planning form some sort of low-dimensional manifold in the original space (Vernaza & Lee, 2012), and the closed-loop system with high-level commands provides stochastic dynamics on that manifold.

Original: That is, a high-dimensional transition model in Fig. 2(c) can be represented as a latent variable model (LVM) in Fig. 2(d).
Revised: That is, a high-dimensional transition model in Fig. 2(c) can be represented as a latent variable model (LVM) in Fig. 2(d).

Original: Once this low-dimensional representation is obtained, any motion planning or inference algorithms can solve the variational inference problem very efﬁciently with the vastly restricted search space.
Revised: Once this low-dimensional representation is obtained, any motion planning or inference algorithms can solve the variational inference problem very efﬁciently with the vastly restricted search space.

Original: Our framework collects the trajectories from low-level policies and utilize them to learn a LVM for inference, which is formulated as a maximum likelihood estimation (MLE) problem.
Revised: Our framework collects the trajectories from low-level policies and utilize them to learn a LVM for inference, which is formulated as a maximum likelihood estimation (MLE) problem.

Original: Suppose that we have collected a set of state trajectories and latent commands { s ( n ) 1: K , h ( n ) 1: K } n =1 ,...,N .
Revised: Suppose that we have collected a set of state trajectories and latent commands { s ( n ) 1: K , h ( n ) 1: K } n =1 ,...,N .

Original: We then formulate the MLE problem as:
Revised: We then formulate the MLE problem as:

Original: [Equation]
Revised: [Equation]

Original: As in Fig. 2(d), the states are assumed to be emerged from a latent dynamical system, where a latent state trajectory, z 1: K , lies on a low-dimensional latent space Z :
Revised: As in Fig. 2(d), the states are assumed to be emerged frwwwwwwwwwwom a latent dynamical system, where a latent state trajectory, z 1: K , lies on a low-dimensional latent space Z :

Original: [Equation]
Revised: [Equation]

Original: In particular, we consider the state space model where latent states follow stochastic transition dynamics with h as inputs, i.e., the prior p φ ( z 1: K | h 1: K ) is a probability measure of a following system:
Revised: In particular, we consider the state space model where latent states follow stochastic transition dynamics with h as inputs, i.e., the prior p φ ( z 1: K | h 1: K ) is a probability measure of a following system:

Original: [Equation]
Revised: [Equation]

Original: and also a conditional likelihood of a state trajectory is assumed to be factorized along the time axis as: s k ∼ N ( µ φ ( z k ) , Σ φ ( z k ))
Revised: and also a conditional likelihood of a state trajectory is assumed to be factorized along the time axis as: s k ∼ N ( µ φ ( z k ) , Σ φ ( z k ))

Original: ∀ k.
Revised: ∀ k.

Original: The resulting sequence modeling is a self-supervised learning problem that has been extensively studied recently (Karl et al., 2017; Krishnan et al., 2017; Fraccaro et al., 2017; Ha et al., 2018b).
Revised: The resulting sequence modeling is a self-supervised learning problem that has been extensively studied recently (Karl et al., 2017; Krishnan et al., 2017; Fraccaro et al., 2017; Ha et al., 2018b).

Original: In particular, we adopt the idea of Adaptive path-integral autoencoder in (Ha et al., 2018b), where the variational distribution is parameterized by the controls, u , and an initial distribution, q 0 , i.e., the proposal q u ( z [0 ,T ] ) is a probability measure of a following system:
Revised: In particular, we adopt the idea of Adaptive path-integral autoencoder in (Ha et al., 2018b), where the variational distribution is parameterized by the controls, u , and an initial distribution, q 0 , i.e., the proposal q u ( z [0 ,T ] ) is a probability measure of a following system:

Original: [Equation]
Revised: [Equation]

Original: Compared to the original formulation in (Ha et al., 2018b), the probability model here is conditioned on the commands, h 1: K , making the learning problem conditional generative model learning (Sohn et al., 2015).
Revised: Compared to the original formulation in (Ha et al., 2018b), the probability model here is conditioned on the commands, h 1: K , making the learning problem conditional generative model learning (Sohn et al., 2015).

Original: 1 Note that it is also possible to ﬁrst obtain a low-dimensional representation considering each state (not trajectory) independently and then ﬁt their dynamics using RNNs like World Model (Ha & Schmidhuber, 2018), or to stack two consecutive observations and learn the dynamical model considering the stacked data as one observation like E2C (Watter et al., 2015).
Revised: 1 Note that it is also possible to ﬁrst obtain a low-dimensional representation considering each state (not trajectory) independently and then ﬁt their dynamics using RNNs like World Model (Ha & Schmidhuber, 2018), or to stack two consecutive observations and learn the dynamical model considering the stacked data as one observation like E2C (Watter et al., 2015).

Original: However, Ha et al.
Revised: However, Ha et al.

Original: (2018b) showed that the representations learned from the short horizon data easily fail to extract enough temporal information and a latent dynamical model suitable for planning can be well-obtained only when considering long trajectories.
Revised: (2018b) showed that the representations learned from the short horizon data easily fail to extract enough temporal information and a latent dynamical model suitable for planning can be well-obtained only when considering long trajectories.

Original: 3. R EASONING (P LANNING ) WITH L EARNED I NTERNAL M ODEL
Revised: 3. R EASONING (P LANNING ) WITH L EARNED I NTERNAL M ODEL

Original: Once the LVM is trained, a planning module can efﬁciently explore the state space S through the latent state z and infer the latent commands h 1: K that are likely to result in high rewards; in particular, we adopt a simple particle ﬁlter algorithm for inference, because it is known to perform well with non-linear and non-Gaussian systems (Ha et al., 2018a; Piche et al., 2019).
Revised: Once the LVM is trained, a planning module can efﬁciently explore the state space S through the latent state z and infer the latent commands h 1: K that are likely to result in high rewards; in particular, we adopt a simple particle ﬁlter algorithm for inference, because it is known to perform well with non-linear and non-Gaussian systems (Ha et al., 2018a; Piche et al., 2019).

Original: Particle ﬁltering, which is also called the sequential Monte-Carlo, utilizes a set of samples and their weights to represent a posterior trajectory distribution.
Revised: Particle ﬁltering, which is also called the sequential Monte-Carlo, utilizes a set of samples and their weights to represent a posterior trajectory distribution.

Original: Starting from the initial state, it propagates a set of samples according to the dynamics (Equation 9) and updates the weights using the observation likelihood as w (cid:48)
Revised: Starting from the initial state, it propagates a set of samples according to the dynamics (Equation 9) and updates the weights using the observation likelihood as w (cid:48)

Original: ∝ w × p ( O k = 1 |
Revised: ∝ w × p ( O k = 1 |

Original: s k ) .
Revised: s k ) .

Original: It also resamples the low-weighted particles to maintain the effective sample size.
Revised: It also resamples the low-weighted particles to maintain the effective sample size.

Original: In the perspective of this work, this procedure can be view as the agent simulating multiple future state trajectories with the internal model, assigning each of them according to the reward, and reasoning the command that leads to the best-possible future.
Revised: In the perspective of this work, this procedure can be viewed as the agent simulating multiple future state trajectories with the internal model, assigning each of them according to the reward, and reasoning the command that leads to the best-possible future.

Original: The detailed explanation is elaborated in Appendix A.4 and in Algorithm 2.
Revised: The detailed explanation is elaborated in Appendix A.4 and in Algorithm 2.

Original: Note that for the more complex environment, we can also iterate the whole procedure multiple times to compute a better command, then the planning algorithm becomes the adaptive path integral method (Kappen & Ruiz, 2016; Williams et al., 2017;
Revised: Note that for the more complex environment, we can also iterate the whole procedure multiple times to compute a better command, then the planning algorithm becomes the adaptive path integral method (Kappen & Ruiz, 2016; Williams et al., 2017;

Original: Ha et al., 2018b).
Revised: Ha et al., 2018b).

Original: If the resampling procedure is eliminated, it is equivalent to the widely-used cross entropy method (Hafner et al., 2019).
Revised: If the resampling procedure is eliminated, it is equivalent to the widely-used cross entropy method (Hafner et al., 2019).

Original: Any other inference/planning algorithms compatible with the graphical model of Fig. 2(d)
Revised: Any other inference/planning algorithms compatible with the graphical model of Fig. 2(d)

Original: can be also used.
Revised: can be also used.

Original: 3. M AIN L EARNING A LGORITHM
Revised: 3. M AIN L EARNING A LGORITHM

Original: The overall learning procedure is summarized in Algorithm 1.
Revised: The overall learning procedure is summarized in Algorithm 1.

Original: The procedure consists of an outer internal model learning loop and an inner policy update loop.
Revised: The procedure consists of an outer internal model learning loop and an inner policy update loop.

Original: During the policy update stage (inner loop), the algorithm samples a task, solves the sampled task by using the hierarchical policy, and collects trajectories into the experience buffer.
Revised: During the policy update stage (inner loop), the algorithm samples a task, solves the sampled task by using the hierarchical policy, and collects trajectories into the experience buffer.

Original: At each time step, the low-level policy decides actions the agent takes under the high-level commands determined by the planning module equipped with the
Revised: At each time step, the low-level policy decides actions the agent takes under the high-level commands determined by the planning module equipped with the

Original: [Figure]
Revised: [Figure]

Original: internal latent model.
Revised: internal latent model.

Original: Using trajectory data in the buffer, the low-level policy is updated via a deep RL algorithm (e.g., policy gradient methods).
Revised: Using trajectory data in the buffer, the low-level policy is updated via a deep RL algorithm (e.g., policy gradient methods).

Original: After the low-level policy update, DISH collects another rollouts by random sampling a latent variable h , and the internal model is learned via self-supervised learning.
Revised: After the low-level policy update, DISH collects another rollouts by random sampling a latent variable h , and the internal model is learned via self-supervised learning.

Original: These two learning procedures are then iterated for L times.
Revised: These two learning procedures are then iterated for L times.

Original: Note that, for complex systems, tasks can be selected more carefully (at line 4) for the better learning landscape; for example, in earlier phases where the agent couldn’t yet learn a valid policy and/or an internal model, the agent can ﬁrst learn them through imiation learning of expert’s demonstrations (Peng et al., 2018) or play data (Lynch et al., 2019) or through intrinsic motivations to acquire useful skills by itself (Sharma et al., 2019).
Revised: Note that, for complex systems, tasks can be selected more carefully (at line 4) for the better learning landscape; for example, in earlier phases where the agent couldn’t yet learn a valid policy and/or an internal model, the agent can ﬁrst learn them through imiation learning of expert’s demonstrations (Peng et al., 2018) or play data (Lynch et al., 2019) or through intrinsic motivations to acquire useful skills by itself (Sharma et al., 2019).

Original: As more challenging tasks are gradually provided to the agent, the internal model (or the reasoning module) is learned to cover wider ranges of state space for those tasks and the low-level policy is trained such that it can be operated with more complicated high-level commands.
Revised: As more challenging tasks are gradually provided to the agent, the internal model (or the reasoning module) is learned to cover wider ranges of state space for those tasks and the low-level policy is trained such that it can be operated with more complicated high-level commands.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: In this section, we demonstrate the effectiveness of the proposed framework on performing planning and control for the high dimensional humanoid model (Peng et al., 2018) which has 197 state features and 36 action parameters, simulated by 1.2kHz Bullet physics engine (Coumans et al., 2013).
Revised: In this section, we demonstrate the effectiveness of the proposed framework on performing planning and control for the high dimensional humanoid model (Peng et al., 2018) which has 197 state features and 36 action parameters, simulated by 1.2kHz Bullet physics engine (Coumans et al., 2013).

Original: The low-level control policy and the internal latent model are trained through the imitation learning, where three locomotion data from the Carnegie Mellon University motion capture (CMU mocap) database are used as target motions of imitation.
Revised: The low-level control policy and the internal latent model are trained through the imitation learning, where three locomotion data from the Carnegie Mellon University motion capture (CMU mocap) database are used as target motions of imitation.

Original: The control policy is trained with the DeepMimic imitation reward (Peng et al., 2018) by using proximal policy optimization (PPO)
Revised: The control policy is trained with the DeepMimic imitation reward (Peng et al., 2018) by using proximal policy optimization (PPO)

Original: (Schulman et al., 2017), while the internal model is learned to maximize the likelihood of experience data (i.e. Equation 7) by using the APIAE approach (Ha et al., 2018b).
Revised: (Schulman et al., 2017), while the internal model is learned to maximize the likelihood of experience data (i.e. Equation 7) by using the APIAE approach (Ha et al., 2018b).

Original: The internal model of DISH is constructed to have a 3-dimensional latent state and a 1-dimensional latent command for all experiments.
Revised: The internal model of DISH is constructed to have a 3-dimensional latent state and a 1-dimensional latent command for all experiments.

Original: The low-level policy and the internal model are operated in different time scales, 30Hz and 1Hz, respectively.
Revised: The low-level policy and the internal model are operated in different time scales, 30Hz and 1Hz, respectively.

Original: The learned hierarchical model is then evaluated on trajectory following and navigation tasks in
Revised: The learned hierarchical model is then evaluated on trajectory following and navigation tasks in

Original: Section 4.1 and 4.2, respectively.
Revised: Section 4.1 and 4.2, respectively.

Original: For planning and execution, the model predictive control (MPC) scheme with particle ﬁltering (A.4) is used; a 5-second trajectory is planned and the ﬁrst reasoned high-level command is applied to the low-level policy at 1Hz and 4Hz for each task.
Revised: For planning and execution, the model predictive control (MPC) scheme with particle ﬁltering (A.4) is used; a 5-second trajectory is planned and the ﬁrst reasoned high-level command is applied to the low-level policy at 1Hz and 4Hz for each task.

Original: We refer to the appendix for the reward functions, hyperparmeters, and network architectures (A. and A.6), task conﬁgurations (A.7), and more experimental results (A.8).
Revised: We refer to the appendix for the reward functions, hyperparmeters, and network architectures (A. and A.6), task conﬁgurations (A.7), and more experimental results (A.8).

Original: Our TensorFlow (Abadi et al., 2015) implementation will be made available in the ﬁnal manuscript.
Revised: Our TensorFlow (Abadi et al., 2015) implementation will be made available in the ﬁnal manuscript.

Original: The videos of the training procedure and the resulting policy are available at: https://bit.ly/2rwIfQn
Revised: The videos of the training procedure and the resulting policy are available at: https://bit.ly/2rwIfQn

Original: 4. A BLATION S TUDY : L EARNING H IERARCHICAL S TRUCTURE
Revised: 4. A BLATION S TUDY : L EARNING H IERARCHICAL S TRUCTURE

Original: In the ﬁrst experiment, we examine how effectively the proposed framework learns and exploits the internal model.
Revised: In the ﬁrst experiment, we examine how effectively the proposed framework learns and exploits the internal model.

Original: To investigate the effectiveness of each component introduced, we conduct ablation studies by considering three baselines: (i) sas (cid:48) that does not have neither the hierarchical structure nor LVMs (Fig. 2(a)), (ii) shs (cid:48) that utilizes the hierarchical policy but doesn’t learn the low-dimensional latent dynamics (Fig. 2(c)), and (iii) zaz (cid:48) that considers the latent dynamics but 
Revised: To investigate the effectiveness of each component introduced, we conduct ablation studies by considering three baselines: (i) sas (cid:48) that does not have neither the hierarchical structure nor LVMs (Fig. 2(a)), (ii) shs (cid:48) that utilizes the hierarchical policy but doesn’t learn the lowdimensional latent dynamics (Fig. 2(c)), and (iii) zaz (cid:48) that considers the latent dynamics but without [Table]

Original: [Table]
Revised: [Table]

Original: without the hierarchical structure (no latent commands, a LVM version of Fig. 2(a) 2 ).
Revised: the hierarchical structure (no latent commands, a LVM version of Fig. 2(a) 2 ).

Original: Given the rollouts { τ ( i ) } = { s ( i ) 1: K , a ( i ) 1: K , h ( i ) 1: K } , learning sas (cid:48) and shs (cid:48) is simply supervised learning problems.
Revised: Given the rollouts { τ ( i ) } = { s ( i ) 1: K , a ( i ) 1: K , h ( i ) 1: K } [Table] learning sas (cid:48) and shs (cid:48) are simply supervised learning problems.

Original: For the zaz (cid:48) model, the variational autoencoder (VAE) approach (Kingma & Welling, 2013) is taken to train mappings between the observation and  latent space, and then the latent dynamics is trained via supervised learning, following the idea of (Ha & Schmidhuber, 2018).
Revised: For the zaz (cid:48) model, the variational autoencoder (VAE) approach (Kingma & Welling, 2013) is taken to train mappings between the observation and the latent space, and then the latent dynamics is trained via supervised learning, following the idea of (Ha & Schmidhuber, 2018).

Original: Note that most
Revised: Note that most

Original: HRL frameworks can be categorized as either zaz (cid:48)
Revised: HRL frameworks can be categorized as either zaz (cid:48)

Original: e.g., (Ha & Schmidhuber, 2018; Hafner et al., 2019) or shs (cid:48)
Revised: e.g., (Ha & Schmidhuber, 2018; Hafner et al., 2019) or shs (cid:48)

Original: e.g., (Sharma et al., 2019).
Revised: e.g., (Sharma et al., 2019).

Original: The similar network structures are used for the baselines; implementation details of the baselines also can be found in A.6.
Revised: The similar network structures are used for the baselines; implementation details of the baselines also can be found in A.6.

Revised: Table 1 summarizes the different features of the models with the related works.

Original: Figs. 3(a) and 3(b) show the learned latent space colored by the moving-averaged angular velocity of the groundtruth motion.
Revised: Figs. 3(a) and 3(b) show the learned latent space colored by the moving-averaged angular velocity of the ground truth motion.

Original: In the case of DISH, the latent states forms a manifold of a cylindrical shape in 3-dimensional space where the locomotion phase and the angular velocity are well encoded along the manifold.
Revised: In the case of DISH, the latent state forms a manifold of a cylindrical shape in 3-dimensional space where the locomotion phase and the angular velocity are well encoded along the manifold.

Original: In contrast, the latent state structure of the zaz (cid:48) model does not capture the phase information and failed to construct  periodic manifold, which prevents a valid latent dynamics from being learned.
Revised: In contrast, the latent state structure of the zaz (cid:48) model does not capture the phase information and failed to construct a periodic manifold, which prevents a valid latent dynamics from being learned.

Original: Figs. 3(c) and 3(d) show the rollout trajectories from each internal model colored by the values of high-level commands, h .
Revised: Figs. 3(c) and 3(d) show the rollout trajectories from each internal model colored by the values of high-level commands, h .

Original: The high-level commands of DISH are learned to control the heading direction of the humanoid so that the agent can make the structural exploration in the conﬁguration space.
Revised: The high-level commands of DISH are learned to control the heading direction of the humanoid so that the agent can make the structural exploration in the conﬁguration space.

Original: The shs (cid:48) model, on the other hand, fails to learn a valid controlled dynamics (since its space is too large) and consequently just generates noisy trajectories.
Revised: The shs (cid:48) model, on the other hand, fails to learn a valid controlled dynamics (since its space is too large) and consequently just generates noisy trajectories.

Original: To quantitatively evaluate the reasoning performance of DISH and its ability to ﬂexibly perform different tasks, we compare DISH to the baseline models on three trajectory following tasks: going straight, turning left and right.
Revised: To quantitatively evaluate the reasoning performance of DISH and its ability to ﬂexibly perform different tasks, we compare DISH to the baseline models on three trajectory following tasks: going straight, turning left and right.

Original: Table 1 reports the RMS errors for reconstruction and differences between the reference, planned, and executed trajectories.
Revised: Table 2 reports the RMS errors for reconstruction and differences between the reference, planned, and executed trajectories.

Original: There are three things we can observe from the table: (i)
Revised: There are three things we can observe from the table: (i)

Original: Although sas (cid:48) had the lowest reconstruction error, the computed action from its internal model even cannot make the humanoid walk.
Revised: Although sas (cid:48) has the lowest reconstruction error, the computed action from its internal model even cannot make the humanoid walk.

Original: This is because the humanoid has a highly unstable dynamics and reasoning of the high-dimensional action can notbe accurate enough to stabilize the humanoid dynamics, i.e., searching over the 36-dimensional action space with the limited number of particles (1024 in this case) is not feasible.
Revised: This is because the humanoid has a highly unstable dynamics and reasoning of the high-dimensional action is not accurate enough to stabilize the humanoid dynamics, i.e., searching over the 36-dimensional action space with the limited number of particles (1024 in this case) is not feasible.

Original: For the same reason, zaz (cid:48) also fails to let the humanoid walk.
Revised: For the same reason, zaz (cid:48) also fails to let the humanoid walk.

Original: (ii) Only the models considering the hierarchical policies can make the humanoid walk, and the DISH framework generates the most executable and valuable plans; the humanoid with the shs (cid:48) model walks just in random directions rather than following a planned trajectory (see Fig. 3(d)), which implies that the high-level command h does not provide any useful information regarding the navigation.
Revised: (ii) Only the models considering the hierarchical policies can make the humanoid walk, and the DISH framework generates the most executable and valuable plans; the humanoid with the shs (cid:48) model walks just in random directions rather than following a planned trajectory (see Fig. 3(d)), which implies that the high-level command h does not provide any useful information regarding the navigation.

Original: (iii)
Revised: (iii)

Original: By iterating the low-level policy and the internal model learning further, DISH+ becomes able to reason better plans as well as execute them better.
Revised: By iterating the low-level policy and the internal model learning further, DISH+ becomes able to reason better plans as well as execute them better.

Original: Further analysis can be found in
Revised: Further analysis can be found in

Original: A.
Revised: A.

Revised: [Figure]

Original: 4. P LANNING AND C ONTROL WITH L EARNED H IERARCHY
Revised: 4. P LANNING AND C ONTROL WITH L EARNED H IERARCHY

Original: In the second experiment, we further demonstrate the capability of DISH framework to perform navigation tasks in cluttered environments (shown in Fig. 4).
Revised: In the second experiment, we further demonstrate the capability of DISH framework to perform navigation tasks in cluttered environments (shown in Fig. 4).

Original: Since the humanoid character with the baseline models either kept falling or failed to walk in a desired direction, we omit the comparisons with the baselines in this task.
Revised: Since the humanoid character with the baseline models either kept falling or failed to walk in a desired direction, we omit the comparisons with the baselines in this task.

Original: The navigation reward is designed as a sum of two components: penalty for distance from the goal and penalty fora collision with obstacles.
Revised: The navigation reward is designed as a sum of two components: penalty for distance from the goal and penalty for collision with obstacles.

Original: As shown in Figs. 4(c) and 4(d) as well as in the supplementary video, the humanoid equipped with the DISH policy is able to not only escape a bug trap that cannot be overcome with greedy algorithms (i.e. without planning),[Figure] but also navigate through obstacle regions successfully.
Revised: As shown in Figs. 4(c) and 4(d) as well as in the supplementary video, the humanoid equipped with the DISH policy is able to not only escape a bug trap that cannot be overcome with greedy algorithms (i.e. without planning), but also navigate through obstacle regions successfully.

Original: Note that, unlike the HRL algorithms, the proposed hierarchical policy trained using the imitation tasks can be directly applied to the navigation tasks.
Revised: Note that, unlike the HRL algorithms, the proposed hierarchical policy trained using the imitation tasks can be directly applied to the navigation tasks.

Original: It shows the generalization power of reasoning process; utilizing the internal model and the command-conditioned policy, the agent becomes able to plan and control its motions to adapt to new tasks and environments.
Revised: It shows the generalization power of reasoning process; utilizing the internal model and the command-conditioned policy, the agent becomes able to plan and control its motions to adapt to new tasks and environments.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a framework to learn a hierarchical policy for an RL agent.
Revised: We proposed a framework to learn a hierarchical policy for an RL agent.

Original: In the proposed policy, the high-level loop plans the agent’s motion by predicting its low-dimensional "task-speciﬁc" futures and the low-level loop maps the high-level commands into actions while actively reacting to the environment using its own state feedback loop.
Revised: In the proposed policy, the high-level loop plans the agent’s motion by predicting its low-dimensional "task-speciﬁc" futures and the low-level loop maps the high-level commands into actions while actively reacting to the environment using its own state feedback loop.

Original: This sophisticated separation was able to emerge because two loops operated in different scales; the high-level planning loop only focuses on taskspeciﬁc low-dimensional aspects in a coarser timescale, which enables it to plan relatively long-term futures.
Revised: This sophisticated separation was able to emerge because two loops operated in different scales; the high-level planning loop only focuses on taskspeciﬁc low-dimensional aspects in a coarser time scale, which enables it to plan relatively long-term futures.

Original: In order to learn the internal model for planning, we took advantage of recent advances in self-supervised learning of sequential data, while the low-level control policy is learned using a deep
Revised: In order to learn the internal model for planning, we took advantage of recent advances in self-supervised learning of sequential data, while the low-level control policy is learned using a deep

Original: RL algorithm.
Revised: RL algorithm.

Original: By alternately optimizing both the LVM and the policy, the proposed framework was able to construct a meaningful internal model as well as a versatile control policy.
Revised: By alternately optimizing both the LVM and the policy, the proposed framework was able to construct a meaningful internal model as well as a versatile control policy.

Original: As future works, it would be interesting to incorporate visual inputs into the high-level reasoning module as suggested by Merel et al. (2019a).
Revised: As future works, it would be interesting to incorporate visual inputs into the high-level reasoning module as suggested by Merel et al. (2019a).

Original: Though only continuous latent variables were considered in our framework, utilizing discrete variables such as a notion of logics or modes (Toussaint et al., 2018) also seems to be a promising direction.
Revised: Though only continuous latent variables were considered in our framework, utilizing discrete variables such as a notion of logics or modes (Toussaint et al., 2018) also seems to be a promising direction.

Original: Lastly, besides imitation of experts, an agent should be able to learn from play data (Lynch et al., 2019) or from its own intrinsic motivation (Sharma et al., 2019).
Revised: Lastly, besides imitation of experts, an agent should be able to learn from play data (Lynch et al., 2019) or from its own intrinsic motivation (Sharma et al., 2019).

Original: A A PPENDIX A. C ONTROL -I NFERENCE D UALITY
Revised: A A PPENDIX A. C ONTROL -I NFERENCE D UALITY

Original: One theoretical concept this work extensively takes advantage of is the duality between optimal control (OC) and probabilistic inference (Levine, 2018; Todorov, 2008; Rawlik et al., 2012).
Revised: One theoretical concept this work extensively takes advantage of is the duality between optimal control (OC) and probabilistic inference (Levine, 2018; Todorov, 2008; Rawlik et al., 2012).

Original: The idea is that, if we consider an artiﬁcial binary observation whose emission probability is given by the exponential of a negative cost, an OC problem can be reformulated as an equivalent inference problem.
Revised: The idea is that, if we consider an artiﬁcial binary observation whose emission probability is given by the exponential of a negative cost, an OC problem can be reformulated as an equivalent inference problem.

Original: In this case, the objective is to ﬁnd the trajectory or control policy that maximizes the likelihood of the observations along the trajectory.
Revised: In this case, the objective is to ﬁnd the trajectory or control policy that maximizes the likelihood of the observations along the trajectory.

Original: One advantage of this perspective is that in order to solve the OC or RL problems, we can adopt any powerful and ﬂexible inference methods, e.g., the expectation propagation (Toussaint, 2009), the particle ﬁltering (Ha et al., 2018a; Piche et al., 2019), or the inference for Gaussian processes (Mukadam et al., 2018).
Revised: One advantage of this perspective is that in order to solve the OC or RL problems, we can adopt any powerful and ﬂexible inference methods, e.g., the expectation propagation (Toussaint, 2009), the particle ﬁltering (Ha et al., 2018a; Piche et al., 2019), or the inference for Gaussian processes (Mukadam et al., 2018).

Original: In addition to utilizing efﬁcient inference methods, this work also enjoys the duality to transform a multi-task RL problem into a generative model learning problem, which enables an agent to distill a low-dimensional representation and a versatile control policy in a combined framework.
Revised: In addition to utilizing efﬁcient inference methods, this work also enjoys the duality to transform a multi-task RL problem into a generative model learning problem, which enables an agent to distill a low-dimensional representation and a versatile control policy in a combined framework.

Original: A. R EINFORCEMENT L EARNING AS P ROBABILISTIC I NFERENCE
Revised: A. R EINFORCEMENT L EARNING AS P ROBABILISTIC I NFERENCE

Original: For easier reference, we restate the RL problem and the controlled trajectory distribution here:
Revised: For easier reference, we restate the RL problem and the controlled trajectory distribution here:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (12) respectively.
Revised: (12) respectively.

Original: It is well known in the literature that the above optimization (Equation 11) also can be viewed as a probabilistic inference problem for a certain type of graphical models (Levine, 2018;
Revised: It is well known in the literature that the above optimization (Equation 11) also can be viewed as a probabilistic inference problem for a certain type of graphical models (Levine, 2018;

Original: Todorov, 2008; Rawlik et al., 2012).
Revised: Todorov, 2008; Rawlik et al., 2012).

Original: Suppose we have an artiﬁcial binary random variable o t , called the optimality variable , whose emission probability is given by exponential of a state-dependent reward,
Revised: Suppose we have an artiﬁcial binary random variable o t , called the optimality variable , whose emission probability is given by exponential of a state-dependent reward,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.,
Revised: e.,

Original: [Equation]
Revised: [Equation]

Original: 13) and the action prior p ( a k ) deﬁnes the uncontrolled trajectory distribution (see also Fig.
Revised: 13) and the action prior p ( a k ) deﬁnes the uncontrolled trajectory distribution (see also Fig.

Original: 2(a)):
Revised: 2(a)):

Original: [Equation]
Revised: [Equation]

Original: Then we can derive the evidence lower-bound (ELBO) for the variational inference:
Revised: Then we can derive the evidence lower-bound (ELBO) for the variational inference:

Original: [Equation]
Revised: [Equation]

Original: The ELBO maximization in Equation 15 becomes equivalent to the reinforcement learning in
Revised: The ELBO maximization in Equation 15 becomes equivalent to the reinforcement learning in

Original: Equation 11 by choosing an action prior p ( a k ) and parameterized policy family π θ ( a k | s k ) to match ˜ r k = r k − log π θ p 3 .
Revised: Equation 11 by choosing an action prior p ( a k ) and parameterized policy family π θ ( a k | s k ) to match ˜ r k = r k − log π θ p 3 .

Original: Similar to Equation 19, the above maximization means to ﬁnd the control policy π θ resulting in the variational distribution that best approximates the posterior trajectory distribution when all the optimality variables were observed p ( s 1: K , a 1: K | O 1: K = 1) .
Revised: Similar to Equation 19, the above maximization means to ﬁnd the control policy π θ resulting in the variational distribution that best approximates the posterior trajectory distribution when all the optimality variables were observed p ( s 1: K , a 1: K | O 1: K = 1) .

Original: A. S ELF -S UPERVISED L EARNING OF L ATENT D YNAMICAL M ODELS
Revised: A. S ELF -S UPERVISED L EARNING OF L ATENT D YNAMICAL M ODELS

Original: Self-supervised learning is an essential approach that allows an agent to learn underlying dynamics from sequential high-dimensional sensory inputs.
Revised: Self-supervised learning is an essential approach that allows an agent to learn underlying dynamics from sequential high-dimensional sensory inputs.

Original: The learned dynamical model can be utilized to predict and plan the future state of the agent.
Revised: The learned dynamical model can be utilized to predict and plan the future state of the agent.

Original: By assuming that observations were emerged from the low-dimensional latent states, the learning problems are formulated as latent model learning, which includes an intractable posterior inference of latent states for given input data (Karl et al., 2017;
Revised: By assuming that observations were emerged from the low-dimensional latent states, the learning problems are formulated as latent model learning, which includes an intractable posterior inference of latent states for given input data (Karl et al., 2017;

Original: Krishnan et al., 2017; Fraccaro et al., 2017; Ha et al., 2018b).
Revised: Krishnan et al., 2017; Fraccaro et al., 2017; Ha et al., 2018b).

Original: Suppose that a set of observation sequences { s ( n ) 1: K } n =1 ,...,N is given, where s ( n ) 1: K ≡ { s k ; ∀ k = 1 , ..., K } ( n ) are i.i.d.
Revised: Suppose that a set of observation sequences { s ( n ) 1: K } n =1 ,...,N is given, where s ( n ) 1: K ≡ { s k ; ∀ k = 1 , ..., K } ( n ) are i.i.d.

Original: sequences of observation that lie on (possibly high-dimensional) data space S .
Revised: sequences of observation that lie on (possibly high-dimensional) data space S .

Original: The goal of the self-supervised learning problem of interest is to build a probabilistic model that well describes the given observations.
Revised: The goal of the self-supervised learning problem of interest is to build a probabilistic model that well describes the given observations.

Original: The problem is formulated as a maximum likelihood estimation (MLE) problem by parameterizing a probabilistic model with φ :
Revised: The problem is formulated as a maximum likelihood estimation (MLE) problem by parameterizing a probabilistic model with φ :

Original: [Equation]
Revised: [Equation]

Original: For latent dynamic models, we assume that the observations are emerged from a latent dynamical system, where a latent state trajectory, z 1: K ≡ { z k ; ∀ k ∈ 1 , ..., K } , lies on a (possibly lowdimensional) latent space Z :
Revised: For latent dynamic models, we assume that the observations are emerged from a latent dynamical system, where a latent state trajectory, z 1: K ≡ { z k ; ∀ k ∈ 1 , ..., K } , lies on a (possibly lowdimensional) latent space Z :

Original: [Equation]
Revised: [Equation]

Original: (17) where p φ ( s 1: K | z 1: K ) and p φ ( z 1: K ) are called a conditional likelihood and a prior distribution, respectively.
Revised: (17) where p φ ( s 1: K | z 1: K ) and p φ ( z 1: K ) are called a conditional likelihood and a prior distribution, respectively.

Original: Since the objective function (Equation 16) contains the intractable integration, it cannot be optimized directly.
Revised: Since the objective function (Equation 16) contains the intractable integration, it cannot be optimized directly.

Original: To circumvent the intractable inference, a variational distribution q ( · ) is introduced and then a surrogate loss function L ( q, φ ; s 1: K ) , which is called the evidence lower bound (ELBO), can be considered alternatively:
Revised: To circumvent the intractable inference, a variational distribution q ( · ) is introduced and then a surrogate loss function L ( q, φ ; s 1: K ) , which is called the evidence lower bound (ELBO), can be considered alternatively:

Original: [Equation]
Revised: [Equation]

Original: where q ( · ) can be any probabilistic distribution over Z of which support includes that of p θ ( · ) .
Revised: where q ( · ) can be any probabilistic distribution over Z of which support includes that of p θ ( · ) .

Original: Note that the gap between the log-likelihood and the ELBO is the Kullback-Leibler (KL) divergence between q ( z ) and the posterior p θ ( z 1: K | s 1: K ) :
Revised: Note that the gap between the log-likelihood and the ELBO is the Kullback-Leibler (KL) divergence between q ( z ) and the posterior p θ ( z 1: K | s 1: K ) :

Original: [Equation]
Revised: [Equation]

Original: One of the most general approaches is the expectation-maximization (EM) style optimization where, alternately, (i) E-step denotes an inference procedure where an optimal variational distribution q ∗ is computed for given φ and (ii) M-step maximizes the ELBO w.r.t.
Revised: One of the most general approaches is the expectation-maximization (EM) style optimization where, alternately, (i) E-step denotes an inference procedure where an optimal variational distribution q ∗ is computed for given φ and (ii) M-step maximizes the ELBO w.r.t.

Original: model parameter φ for given q ∗ .
Revised: model parameter φ for given q ∗ .

Original: Note that if we construct the whole inference and generative procedures as one computational graph, all the components can be learned by efﬁcient end-to-end training (Karl et al., 2017; Krishnan et al., 2017; Fraccaro et al., 2017; Ha et al., 2018b).
Revised: Note that if we construct the whole inference and generative procedures as one computational graph, all the components can be learned by efﬁcient end-to-end training (Karl et al., 2017; Krishnan et al., 2017; Fraccaro et al., 2017; Ha et al., 2018b).

Original: In p articular, Ha et al.
Revised: In p articular, Ha et al.

Original: (2018b) proposed the adaptive path-integral autoencoder (APIAE), a framework that utilizes the optimal control method; this framework is suitable to this work because we want to perform the planning in the learned latent space.
Revised: (2018b) proposed the adaptive path-integral autoencoder (APIAE), a framework that utilizes the optimal control method; this framework is suitable to this work because we want to perform the planning in the learned latent space.

Original: APIAE considers the state-space model in which the latent states are governed by a stochastic dynamical model, i.e., the prior p φ ( z 1: K ) is a probability measure of a following system:
Revised: APIAE considers the state-space model in which the latent states are governed by a stochastic dynamical model, i.e., the prior p φ ( z 1: K ) is a probability measure of a following system:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Additionally, a conditional likelihood of sequential observations is factorized along the time axis:
Revised: Additionally, a conditional likelihood of sequential observations is factorized along the time axis:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If the variational distribution is parameterized by the control input u 1: K − 1 and the initial state distribution q 0 as:
Revised: If the variational distribution is parameterized by the control input u 1: K − 1 and the initial state distribution q 0 as:

Original: [Equation]
Revised: [Equation]

Original: the ELBO can be written in the following form:
Revised: the ELBO can be written in the following form:

Original: [Equation]
Revised: [Equation]

Original: Then, the problem of ﬁnding the optimal variational parameters u ∗ and q ∗ 0 (or equivalently, the best approximate posterior) can be formulated as a stochastic optimal control (SOC) problem:
Revised: Then, the problem of ﬁnding the optimal variational parameters u ∗ and q ∗ 0 (or equivalently, the best approximate posterior) can be formulated as a stochastic optimal control (SOC) problem:

Original: [Equation]
Revised: [Equation]

Original: 24) where V ( z 1: K ) ≡ − log p 0 ( z
Revised: 24) where V ( z 1: K ) ≡ − log p 0 ( z

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: K k =
Revised: K k =

Original: [Equation]
Revised: [Equation]

Original: log p φ ( s k | z k ) serves as a state cost of the SOC problem.
Revised: log p φ ( s k | z k ) serves as a state cost of the SOC problem.

Original: Ha et al.
Revised: Ha et al.

Original: (2018b) constructed the differentiable computational graph that resembles the path-integral control procedure to solve the above SOC problem, and trained the whole architecture including the latent dynamics, p 0 ( z ) , f φ ( z ) and σ φ ( z ) , and the generative network, p φ ( s | z ) through the end-to-end training.
Revised: (2018b) constructed the differentiable computational graph that resembles the path-integral control procedure to solve the above SOC problem, and trained the whole architecture including the latent dynamics, p 0 ( z ) , f φ ( z ) and σ φ ( z ) , and the generative network, p φ ( s | z ) through the end-to-end training.

Original: A. P LANNING BY P ARTICLE F ILTERING Algorithm 2 Particle Filtering with Internal Model for Planning
Revised: A. P LANNING BY P ARTICLE F ILTERING Algorithm 2 Particle Filtering with Internal Model for Planning

Original: 1: Initialize ∀ i ∈ { 1 , ..., N particle } :
Revised: 1: Initialize ∀ i ∈ { 1 , ..., N particle } :

Original: z ( i ) 1 ∼ q φ ( ·| s : cur ) and w ( i ) 1 = 1 /N particle 2: for k = 2 , ..., K plan do 3: for i = 1 , ..., N particle do 4:
Revised: z ( i ) 1 ∼ q φ ( ·| s : cur ) and w ( i ) 1 = 1 /N particle 2: for k = 2 , ..., K plan do 3: for i = 1 , ..., N particle do 4:

Original: z ( i ) k
Revised: z ( i ) k

Original: = f φ ( z ( i ) k − 1 ) + σ φ ( z ( i ) k − 1 ) (cid:16) h
Revised: = f φ ( z ( i ) k − 1 ) + σ φ ( z ( i ) k − 1 ) (cid:16) h

Original: ( i ) k − 1 + w ( i ) k − 1 (cid:17) , w ( i ) k − 1 ∼ N (0 , I ) 5: s ( i ) k ∼ N (cid:16) µ φ ( z ( i ) k ) ,
Revised: ( i ) k − 1 + w ( i ) k − 1 (cid:17) , w ( i ) k − 1 ∼ N (0 , I ) 5: s ( i ) k ∼ N (cid:16) µ φ ( z ( i ) k ) ,

Original: Σ φ ( z ( i ) k ) (cid:17) .
Revised: Σ φ ( z ( i ) k ) (cid:17) .

Original: 6: w ( i ) k
Revised: 6: w ( i ) k

Original: = w ( i ) k − 1 exp( r k ( s ( i ) k ))
Revised: = w ( i ) k − 1 exp( r k ( s ( i ) k ))

Original: 7: end for 8: w ( i ) k
Revised: 7: end for 8: w ( i ) k

Original: = w ( i ) k / (cid:80)
Revised: = w ( i ) k / (cid:80)

Original: j w ( j ) k , ∀ i ∈ { 1 , ..., N particle } 9:
Revised: j w ( j ) k , ∀ i ∈ { 1 , ..., N particle } 9:

Original: Resample { z ( i ) 1: k , w ( i ) 1: k } if (cid:16)(cid:80) i ( w ( i ) k ) 2 (cid:17) − 1 < N particle /

Revised: Resample { z ( i ) 1: k , w ( i ) 1: k } if (cid:16)(cid:80) i ( w ( i ) k ) 2 (cid:17) − 1 < N particle /

Original: : end for 11: return h ∗ 1 = (cid:80) i
Revised: : end for 11: return h ∗ 1 = (cid:80) i

Original: w ( i ) K plan w ( i )(cid:46) h ∗ k
Revised: w ( i ) K plan w ( i )(cid:46) h ∗ k

Original: = (cid:80) i
Revised: = (cid:80) i

Original: w ( i ) K plan w ( i ) k , ∀ k
Revised: w ( i ) K plan w ( i ) k , ∀ k

Original: = 1 , .., K MPC for general MPC cases
Revised: = 1 , .., K MPC for general MPC cases

Original: At each time step δt , the high-level planner takes the current state as an argument and required to output the commands by predicting the future trajectory and corresponding reward r k ( · ) .
Revised: At each time step δt , the high-level planner takes the current state as an argument and required to output the commands by predicting the future trajectory and corresponding reward r k ( · ) .

Original: We adopted the particle ﬁlter algorithm to perform such the reasoning and the pseudo code is shown in
Revised: We adopted the particle ﬁlter algorithm to perform such the reasoning and the pseudo code is shown in

Original: Algorithm 2.
Revised: Algorithm 2.

Original: The particle ﬁlter algorithm attempts to represent the posterior distribution using a set of samples.
Revised: The particle ﬁlter algorithm attempts to represent the posterior distribution using a set of samples.

Original: The algorithm ﬁrst samples N particle initial latent states using the inference network (which is a part of the learned internal model) and assigns the same weights for them.
Revised: The algorithm ﬁrst samples N particle initial latent states using the inference network (which is a part of the learned internal model) and assigns the same weights for them.

Original: During the forward recursion, the particles are propagated using the latent dynamics of the internal model (line 4), and the corresponding conﬁgurations are generated through the learned model (line 5).
Revised: During the forward recursion, the particles are propagated using the latent dynamics of the internal model (line 4), and the corresponding conﬁgurations are generated through the learned model (line 5).

Original: The weights of all particles are then updated based on the reward of the generated conﬁgurations (line 6 and 8); i.e., the particles that induce higher reward values will get higher weights.
Revised: The weights of all particles are then updated based on the reward of the generated conﬁgurations (line 6 and 8); i.e., the particles that induce higher reward values will get higher weights.

Original: If only a few samples have weights effectively, i.e., if the weights collapse, the algorithm resamples the particles from the current approximate posterior distribution to maintain the effective sample size (line 9).
Revised: If only a few samples have weights effectively, i.e., if the weights collapse, the algorithm resamples the particles from the current approximate posterior distribution to maintain the effective sample size (line 9).

Original: After the forward recursion over the planning horizon, the optimal commands are computed as a linear combination of the initial disturbances; i.e., it is given by the expected disturbance under the posterior transition dynamics (Kappen & Ruiz, 2016).
Revised: After the forward recursion over the planning horizon, the optimal commands are computed as a linear combination of the initial disturbances; i.e., it is given by the expected disturbance under the posterior transition dynamics (Kappen & Ruiz, 2016).

Original: [Figure]
Revised: [Figure]

Original: For the training algorithm for low-level policy network ( π θ ), we extend motion imitation approach (Peng et al., 2018) to multi-task scheme; we construct value networks parameterized by neural network with size [197 , 1024 , 512 , 1] for each task (three in our experiments), and the low-level policy network (actor network) taking a state feature s and a latent variable h as inputs to determine an action a as illustrated in Fig.
Revised: For the training algorithm for low-level policy network ( π θ ), we extend motion imitation approach (Peng et al., 2018) to multi-task scheme; we construct value networks parameterized by neural network with size [197 , 1024 , 512 , 1] for each task (three in our experiments), and the low-level policy network (actor network) taking a state feature s and a latent variable h as inputs to determine an action a as illustrated in Fig.

Original: The imitation reward is given as following:
Revised: The imitation reward is given as following:

Original: [Equation]
Revised: [Equation]

Original: where q t and p t represent angle and global position while ˆ represent those of the reference
Revised: where q t and p t represent angle and global position while ˆ represent those of the reference

Original: [Equation]
Revised: [Equation]

Original: As reference motion data, three motion capture clips, turning left ( t = [1 , 0 , 0] ), going straight ( t = [0 , 1 , 0] ), turning right ( t = [0 , 0 , 1] ) from the Carnegie Mellon University motion capture (CMU mocap) database are utilized.
Revised: As reference motion data, three motion capture clips, turning left ( t = [1 , 0 , 0] ), going straight ( t = [0 , 1 , 0] ), turning right ( t = [0 , 0 , 1] ) from the Carnegie Mellon University motion capture (CMU mocap) database are utilized.

Original: Following the reference, PPO with same hyperparameters is used for RL algorithm.
Revised: Following the reference, PPO with same hyperparameters is used for RL algorithm.

Original: Since the internal model does not exist at the ﬁrst iteration ( l = 1 ), the high-level planner is initialized by q φ ( h | s ; t ) =
Revised: Since the internal model does not exist at the ﬁrst iteration ( l = 1 ), the high-level planner is initialized by q φ ( h | s ; t ) =

Original: w T t where w =
Revised: w T t where w =

Original: [ − 1 , 0 , 1] .
Revised: [ − 1 , 0 , 1] .

Original: After the ﬁrst iteration, high-level planner computes a command h t that makes the model to best follow the horizontal position of the reference motion for 5 seconds ( δt = 0 .
Revised: After the ﬁrst iteration, high-level planner computes a command h t that makes the model to best follow the horizontal position of the reference motion for 5 seconds ( δt = 0 .

Original: 1 s and K plan = 50 ).
Revised: 1 s and K plan = 50 ).

Original: The corresponding reward function is as following:
Revised: The corresponding reward function is as following:

Original: [Equation]
Revised: [Equation]

Original: (26) where p h,k is the horizontal components of position vector at time step k .
Revised: 26) where p h,k is the horizontal components of position vector at time step k .

Original: A. T RAINING I NTERNAL M ODELS
Revised: A. T RAINING I NTERNAL M ODELS

Original: Internal models of DISH is trained to maximize the ELBO in Equation 23 by using the APIAE approach (Ha et al., 2018b) with hyperparameters as following: 3 adaptations ( R = 4 ), 10 time steps ( K = 10 ), 32 samples ( L = 32 ), and time interval of 0.1s ( δt = 0 . 1 ).
Revised: Internal models of DISH is trained to maximize the ELBO in Equation 23 by using the APIAE approach (Ha et al., 2018b) with hyperparameters as following: 3 adaptations ( R = 4 ), 10 time steps ( K = 10 ), 32 samples ( L = 32 ), and time interval of 0.1s ( δt = 0 . 1 ).

Original: The network architectures of transition network and inference network are shown in Fig 6.
Revised: The network architectures of transition network and inference network are shown in Fig 6.

Original: For the baselines, the transition functions, f φ ( x k +1 | x k , y k ) , were parameterized by neural networks having same architectures as DISH except for the input variables as shown in Table 2.
Revised: For the baselines, the transition functions, f φ ( x k +1 | x k , y k ) , were parameterized by neural networks having same architectures as DISH except for the input variables as shown in Table 3.

Original: The loss
Revised: The loss

Original: [Figure]
Revised: [Figure]

Original: function for baseline is as following:
Revised: function for baseline is as following:

Original: [Equation]
Revised: [Equation]

Original: where ˜ s 

Revised: where ˜ s

Original: = s 1 , ˜ z 1 is latent state for s 1 inferred by VAE, and g ( · ) is learned generative network of
Revised: = s 1 , ˜ z 1 is latent state for s 1 inferred by VAE, and g ( · ) is learned generative network of

Original: VAE.
Revised: VAE.

Original: A. T ASK C ONFIGURATIONS
Revised: A. T ASK C ONFIGURATIONS

Original: Trajectory Following Tasks:
Revised: Trajectory Following Tasks:

Original: Planning reward r t penalizes the distance between the horizontal position of the root of humanoid character p rk and the that of reference trajectory ¯ p k :
Revised: Planning reward r t penalizes the distance between the horizontal position of the root of humanoid character p rk and the that of reference trajectory ¯ p k :

Original: [Equation]
Revised: [Equation]

Original: Navigation Tasks: Planning cost r t penalizes the distance between the horizontal position of the root of humanoid character p rk and the that of the goal p goal and the collision with obstacles, while giving a reward on arrival:
Revised: Navigation Tasks: Planning cost r t penalizes the distance between the horizontal position of the root of humanoid character p rk and the that of the goal p goal and the collision with obstacles, while giving a reward on arrival:

Original: [Equation]
Revised: [Equation]

Original: ( IS _ REACHED ) .
Revised: ( IS _ REACHED ) .

Original: A. F URTHER R ESULTS
Revised: A. F URTHER R ESULTS

Original: Table 3 reports the RMS between reference, planned, and executed trajectories for each tasks.
Revised: Table 4 reports the RMS between reference, planned, and executed trajectories for each tasks.

Original: As illustrated in the table, DISHs showed the best performance.
Revised: As illustrated in the table, DISHs showed the best performance.

Original: Although shs (cid:48) sometimes showed smaller errors for the difference between the planed and reference trajectories, the errors between the reference and executed trajectory of DISHs are always smallest.
Revised: Although shs (cid:48) sometimes showed smaller errors for the difference between the planed and reference trajectories, the errors between the reference and executed trajectory of DISHs are always smallest.

Original: This demonstrates that DISHs best learn the internal dynamics of the humanoid, making the most valid predictions for future motion.
Revised: This demonstrates that DISHs best learn the internal dynamics of the humanoid, making the most valid predictions for future motion.

Original: [Table]
Revised: [Table]

Original: Comparing DISH ( L = 1 ) and DISH+ ( L = 2 ), we can observe that DISH outperforms in the turning tasks while showing the worse performance in going straight.
Revised: Comparing DISH ( L = 1 ) and DISH+ ( L = 2 ), we can observe that DISH outperforms in the turning tasks while showing the worse performance in going straight.

Original: This is because the high-level planner is initialized to output only one of {− 1 , 0 , 1 } (as shown in Appendix A.5), so the corresponding low-level policy of DISH is trained only around h ∈ {− 1 , 0 , 1 } rather than along the continuous h values.
Revised: This is because the high-level planner is initialized to output only one of {− 1 , 0 , 1 } (as shown in Appendix A.5), so the corresponding low-level policy of DISH is trained only around h ∈ {− 1 , 0 , 1 } rather than along the continuous h values.

Original: As a result, the DISH agent is only good at radical turns (not smooth turns), making it difﬁcult to stabilize its heading direction properly.
Revised: As a result, the DISH agent is only good at radical turns (not smooth turns), making it difﬁcult to stabilize its heading direction properly.

Original: The ability to turn smoothly is obtained in the next iteration where the proper reasoning module is equipped, thus, although it lost some ability to turn fast, the
Revised: The ability to turn smoothly is obtained in the next iteration where the proper reasoning module is equipped, thus, although it lost some ability to turn fast, the

Original: DISH+ agent achieves the better ability to walk straight and the increased average performance (see
Revised: DISH+ agent achieves the better ability to walk straight and the increased average performance (see

Original: Table 1).
Revised: Table 2).

Original: Fig.
Revised: Fig.

Original: 7 shows rollout samples by varying the control values.
Revised: 7 shows rollout samples by varying the control values.

Original: Except for DISHs, the generated trajectories are very noisy, which indicates that the baseline internal models are not suitable for planning the future movements of the humanoid.
Revised: Except for DISHs, the generated trajectories are very noisy, which indicates that the baseline internal models are not suitable for planning the future movements of the humanoid.

Original: [Figure]
Revised: [Figure]