Paper:
The successful application of flexible, general learning algorithms to real-world robotics applications is often limited by their poor data-efficiency. To address the challenge, domains with more than one dominant task of interest encourage the sharing of information across tasks to limit required experiment time. To this end, we investigate compositional inductive biases in the form of hierarchical policies as a mechanism for knowledge transfer across tasks in reinforcement learning (RL). We demonstrate that this type of hierarchy enables positive transfer while mitigating negative interference. Furthermore, we demonstrate the benefits of additional incentives to efficiently decompose task solutions. Our experiments show that these incentives are naturally given in multitask learning and can be easily introduced for single objectives. We design an RL algorithm that enables stable and fast learning of hierarchical policies and the effective reuse of both behavior components and transition data across tasks in an off-policy setting for complex, real-world domains. Finally, we evaluate our algorithm in simulated environments as well as physical robot experiments and demonstrate substantial improvements in data data-efficiency over competitive baselines.

Review:
While this paper has some interesting experiments. I am quite confused about what exactly the author are claiming is the core contribution of their work. To me the proposed approach does not seem particularly novel and the idea that hierarchy can be useful for multi-task learning is also not new. While it is possible that I am missing something, I have tried going through the paper a few times and the contribution is not immediately obvious. The two improvements in section 3.2 seem quite low level and are only applicable to this particular approach to hierarchical RL. Additionally, it is very much not clear why someone, for example, would select the approach of this paper in comparison to popular paradigms like Option-Critic and Feudal Networks. 
The authors mention that Feudal approaches "employ different rewards for different levels of the hierarchy rather than optimizing a single objective for the entire model as we do." Why reward decomposition at the lower levels is a problem instead of a feature isn't totally clear, but this criticism does not apply to Option-Critic models. For Option-Critic models the authors claim that "Rather than the additional inductive bias of temporal abstraction, we focus on the investigation of composition as type of hierarchy in the context of single and multitask learning while demonstrating
the strength of hierarchical composition to lie in domains with strong variation in the objectives such as in multitask domains." First of all, I should point out that [1] looked at applying Option-Critic in a many task setting and found both that there was an advantage to hierarchy and an advantage to added depth of hierarchy. Additionally, it is well known that Option-Critic approaches (when unregularized) tend to learn options that terminate every step [2].  So, if you generically apply Option-Critic, it would in fact be possible to disentangle the inductive bias of hierarchy from the inductive bias of temporal abstraction by using options that always terminate. 
In comparison to past frameworks, the approach of this paper seems less theoretically motivated. It certainly does not seem justified to me to just assume this framework and disregard past successful approaches even as a comparison. While the experiments show the value of hierarchy, they do not show the value of this particular method of creating hierarchy. The feeling I get is that the authors are trying to make their experiments less about what they are proposing in this paper and more about empirical insights about the nature of hierarchy overall. If this is the case, I feel like the empirical results are not novel enough to create value for the community and too tied to a particular approach to hierarchy which does not align with much of the past work on HRL. 
[1] "Learning Abstract Options". Matthew Riemer, Miao Liu, and Gerald Tesauro. NeurIPS-18. 
[2] "When Waiting is not an Option: Learning Options with a Deliberation Cost" Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. AAAI-18.