Paper:
We present a hierarchical planning and control framework that enables an agent to perform various tasks and adapt to a new task flexibly. Rather than learning an individual policy for each particular task, the proposed framework, DISH, distills a hierarchical policy from a set of tasks by self-supervision and reinforcement learning. The framework is based on the idea of latent variable models that represent high-dimensional observations using low-dimensional latent variables. The resulting policy consists of two levels of hierarchy: (i) a planning module that reasons a sequence of latent intentions that would lead to optimistic future and (ii) a feedback control policy, shared across the tasks, that executes the inferred intention. Because the reasoning is performed in low-dimensional latent space, the learned policy can immediately be used to solve or adapt to new tasks without additional training. We demonstrate the proposed framework can learn compact representations (3-and 1-dimensional latent states and commands for a humanoid with 197-and 36-dimensional state features and actions) while solving a small number of imitation tasks, and the resulting policy is directly applicable to other types of tasks, i.e., navigation in cluttered environments. The supplementary video is available at: https://bit.ly/2rwIfQn 1 INTRODUCTION Reinforcement learning (RL) aims to compute the optimal control policy while an agent interacts with the environment. Recent advances in deep learning enable RL frameworks to utilize deep neural networks to efficiently represent and learn a policy having a flexible and expressive structure. As a result, we've been witnessing RL agents that already achieved or even exceeded human-level performances in particular tasks (Mnih et al., 2015;Silver et al., 2017). The core of intelligence, however, is not just to learn a policy for a particular problem instance, but to solve various multiple tasks or immediately adapt to a new task. Given that a huge computational burden makes it unrealistic to learn an individual policy for each task, an agent should be able to reason about its action. If predictions about consequences of actions are available, e.g., by using an internal model (Ha &  Schmidhuber, 2018; Kaiser et al., 2019), an intelligent agent can plan a sequence of its actions. Involving planning procedures in a control policy could provide adaptiveness to an agent, but it is often not trivial to learn such a prediction & planning framework: First, it is difficult to obtain the exact internal dynamic model directly represented in high-dimensional state (observation) space. Model errors inevitably become larger in the high-dimensional space and are accumulated along the prediction/planning horizon. This prohibits planning methods from producing a valid prediction and so a sensible plan. Second, and perhaps more importantly, planning methods cannot help but relying on some dynamic programming or search procedures, which quickly become intractable for problems with high degrees of freedom (DOFs) because the size of search space grows exponentially with DOFs, i.e., the curse of dimensionality (LaValle, 2006).
Crucial evidence found in the cognitive science field is that there exists a certain type of hierarchical structure in the humans' motor control scheme addressing the aforementioned fundamental difficulty (Todorov & Ghahramani, 2003;Todorov, 2004). Such a hierarchical structure is known to utilize two levels of parallel control loops, operating in different time scales; in a coarser scale, the high-level loop generates task-relevant commands for the agent to perform a given task, and then in a finer time scale, the low-level loop maps those commands into control signals while actively reacting to disturbances that the high-level loop could not consider (e.g., the spinal cord) (Todorov 1 Under review as a conference paper at ICLR 2020& Ghahramani, 2003). Because the low-level loop does not passively generate control signals from high-level commands, the high-level loop is able to focus only on the task-relevant aspects of the environment dynamics that can be represented in a low-dimensional form. Consequently, this hierarchical structure allows us for efficiently predicting and planning the future states to compute the commands.
. .

Review:
*Edit: original score was a weak reject (3), updating to a weak accept (6) in light of revisions.*
This work implements a hierarchical control scheme for a high-dimensional control problem (locomotion using a humanoid body).  The hierarchy consists of a high-level module that plans in an abstract space of "intention", and the intention variables serve as inputs, along with state, to a low-level controller that actually executes the movements.  The premise is that a lower-level controller should be usable for multiple tasks, and should be able to be commanded by a lower-dimensional intention input.  I find the basic ideas presented clear, the literature reviewed reasonably well, and the motivation and setting to be very interesting.  The video summary is valuable.
My main concerns have to do with presentation, but I think they are relatively significant concerns.  As the draft currently stands, I would, somewhat regrettably, be inclined to reject the submission (marginally).  I think revisions could seriously improve this paper and incline me towards acceptance.
Algorithm 1 indicates that the learning of the low-level controller will be done jointly with the learning of the latent model and planning using the high-level, learned intention space.  In the experiment, it is indicated that the low-level controller is pretrained.  This points to a couple issues that are not clear in the draft:
(1) Presumably this pretraining is necessary and things do not work without it.  Indeed, it is hard to imagine that the movements will be well grounded to human motion capture movements without this pretraining.  Does the algorithm work as written or is pretraining a fundamentally essential step?  There are no settings, even toy settings, where the algorithm as written is shown to be effective.
(2) The authors should be clearer how they conduct the pretraining which involves learning the low-level controller.  
(3) I'm not clear how updating the low-level controller is effective in the algorithm.  While I understand why it makes sense to plan in the intention space of the pre-trained controller, and I understand why learning a model is a core part of planning, it would seem like fine-tuning the low-level controller could make the movements deviate considerably from the initial movement space and maybe even eliminate the ability of the low-level policy to express movements that are not used early in training. So essentially, while the planning in the low-D space makes sense and the learning of the model makes sense, the low-level controller update seems possibly to not make sense, and there aren't experiments showing that step helps. 
Is it just a coincidence that the intention space (h) is one-of-three and the low-d state space (z) is 3-dimensional as well?  Or are these both selected with sort of going straight vs turning left or right in mind?
The experiment section is generally very unclear, though details are made a little clearer from the video.  In the paper, there are a few points that need to be clearer: 
(1) "ref", "plan", and  "true" are not well defined and it is unclear what these distances in Table 1 refer to precisely.  Clearly introduce what each of these refers to.  The authors simply say that there are imitation tasks but do not walk through what these terms refer to. 
(2) In 4.1, the different structures are not adequately introduced.  The pointers to the figure 2 diagrams are essential, but there is no pointer for zaz', the pointers are only in the table (not in the text), there are grammar issues in the text and the text could be verbally clearer about the variants.
(3) shs' setting is a bit unclear. Basically, clarify briefly how planning is performed in this case.   Is a forward model still trained, but the model opperates with the full state space?  If so, presumably the forward model is much worse and then the planning approach is correspondingly bad, hence the poor rollouts? 
(4) 4.2 is I think obviously inadequately described in the text and I can only assume was the result of rushing for the deadline?  The second experiment is essentially not presented in the text all aside from a still image.  
(5) And for all of the experiments that rely on planning with a particle filter, details such as how reliable the filter is in generating useful control, how many samples are required, and possibly elements of compute speed would make much clearer how well the approach actually works.  Does the choice of planner matter at all?  A common, albeit relatively weak, baseline planning approach is CEM...would CEM work here?  I'd like to understand if the choice of particle filter is the author's default choice, which is fine if so, or if there is a positive assertion being made that the particle filter is particularly valuable.
I hope the authors will generally improve the exposition in the experiment section (4) during the revisions.
Overall, I find the paper well motivated in framing the problem (i.e. using model-based approaches to control the latent space of a low-level controller).  I also appreciate the scale of the problem (humanoid control is challenging, so this is not a toy problem).  I find the results a bit unclear, perhaps due to hurriedness in writing, so I find them a bit difficult to fully appreciate.  Nevertheless, the core contribution that I take away from this work is that there is a value to learning the low-dimensional state representation (z, via the LVM), relative to planning using a forward model on the full state (?...I'm still unclear on the presentation of this result, due to unclear exposition). Slightly more broadly, this is a good demonstration of using a planner jointly with a learned high-level command/intention representation, for a high-dimensional problem. 
If I've understood this correctly, I'd be reasonably interested in this result. If the authors can both clarify the core results and communicate that the choices made in the algorithm are well thought through, I would be happy to adjust my score.
Relatively minor:
Abstract says 90-dimensional humanoid system, but later it is stated "34 degrees of freedom,
197 state features, and 36 action parameters". Where the 90 dimensions comes from is unclear.  Often people refer to number of actuators or DoFs.  Please adjust this or be more explicit.
In equation 9, f() is not very clearly specified. Is f() a nonlinear function (e.g. a neural network) or is it a linear function? It seems like it might as well be a linear function, since the authors propose to learn a latent dynamics model that is nonlinearly related to the state.
Typos in Fig 3 caption.