Paper:
HIERARCHICAL VISUOMOTOR CONTROL OF HUMANOIDS
We aim to build complex humanoid agents that integrate perception, motor control, and memory. In this work, we partly factor this problem into low-level motor control from proprioception and high-level coordination of the low-level skills informed by vision. We develop an architecture capable of surprisingly flexible, task-directed motor control of a relatively high-DoF humanoid body by combining pre-training of low-level motor controllers with a high-level, task-focused controller that switches among low-level sub-policies. The resulting system is able to control a physically-simulated humanoid body to solve tasks that require coupling visual perception from an unstabilized egocentric RGB camera during locomotion in the environment. Supplementary video link 1

Review:
1) Summary
The authors propose an interesting hierarchical reinforcement learning method that makes use of visual inputs as well as proprioception for locomotion of humanoid agents. The low-level controllers make use of “motion capture” data and are expected to form a set of movement primitives that can be used by a higher-level controller that has vision and memory. Their method is tested on a variety of tasks and different choices of low-level controller are explored.
2) Pros
+ Combining vision, memory, and motor control
+ Allows the high-level controller to operate at a coarser time scale
+ The set of low-level movement primitives can be extended by using more mocap data
3) Cons
- No comparison to earlier work
- Highly unnatural motions even though it makes use of mocap data
- Sample inefficient: more than 1 billion time-steps to train the high-level controller
4) Comments
Showing that the agent can provide suitable solutions for these tasks using raw vision input is indeed interesting, however it is not clear what the main contribution of the paper is as the authors fail to compare their results with earlier work. It would be useful if the authors could cover the related work in more depth in order to motivate their method and contrast it with the existing solutions. As an example, DeepLoco (Peng et al. 2017) solves a similar problem in which they use an egocentric heightmap instead of direct visual input, hence a formal consideration of the trade-offs would be informative.
In addition, the appeal of using hierarchical reinforcement learning is to divide up the task into easier chunks that can be solved easier, however it is not obvious how well this method succeeds at this task, keeping in mind that the high-level controller takes in the order of 1 billion time-steps to learn most tasks (5 billion in the case of “Heterogeneous Forage”).
In the end, an ablation study could be useful since the authors make plenty of novel design decision, yet their effect on the final performance is not clear.
6) Questions
- Is is possible to entirely remove proprioception from the input to the high-level controller or at least use just a small portion of it? How do the results compare in this case?
- How robust is “cold-switching” between control fragments? Is it possible to transition between most fragments without losing balance or does the high-level controller have to be extremely careful as to which combination it should use? The former case would suggest that this method is indeed useful as a hierarchical method. However the latter case might imply that the hierarchical method is failing and the higher level controller’s task has not been made much easier than the original problem itself.
- Table 2 describes the mocap clips used to train the low-level controllers in each task. What is the effect of choosing different sets of motions? Specifically how well does the steerable controller work if walking motions were used for the “Go To Target” and “Walls” tasks rather than running motions? Presumably, this can result in a more flexible controller which allows sharper turns without loosing balance.
- The network in Figure A.1 gets the last action as an input. Why is this required? Especially since the LSTM unit can learn to remember any information related to the previous actions.
- How does the supervised pre-training described in section 2.1 effect the training of low-level controllers? Is it used as a speed-up mechanism or a way of escaping local minima?
- In section 2.1 the authors mention that the episodes are “terminated when the pose deviates too far from the trajectory”. I believe this termination criteria was not present in the earlier works (Peng et al. 2018), then what is the effect of adding such a criterion? Can this make the learned agent less robust as it will not learn to recover from larger perturbations?
7) Conclusion
The method and the results are interesting but further comparison with existing work is required.