Paper:
Hierarchical reinforcement learning has demonstrated significant success at solving difficult reinforcement learning (RL) tasks. Previous works have motivated the use of hierarchy by appealing to a number of intuitive benefits, including learning over temporally extended transitions, exploring over temporally extended periods, and training and exploring in a more semantically meaningful action space, among others. However, in fully observed, Markovian settings, it is not immediately clear why hierarchical RL should provide benefits over standard "shallow" RL architectures. In this work, we isolate and evaluate the claimed benefits of hierarchical RL on a suite of tasks encompassing locomotion, navigation, and manipulation. Surprisingly, we find that most of the observed benefits of hierarchy can be attributed to improved exploration, as opposed to easier policy learning or imposed hierarchical structures. Given this insight, we present exploration techniques inspired by hierarchy that achieve performance competitive with hierarchical RL while at the same time being much simpler to use and implement.

Review:
This is an interesting paper, as it tries to understand the role of hierarchical methods (such as Options, higher level controllers etc) in RL. The core contribution of the paper is understand and evaluate the claimed benefits often proposed by hierarchical methods, and finds that the core benefit in fact comes from exploration. The paper studies hierarchical methods to eventually draw the conclusion that HRL in fact leads to better exploration based behaviour in complex tasks. 
While the conclusion suggesting HRL leading to better exploration seems interesting, I am not sure whether this is in fact too surprising? For example, the Options framework has been fundamentally proposed and shows benefits in terms of transfer learning and faster exploration. Options has been fundamentally argued to lead to faster exploration (for example, when the goal changes in the four rooms task). Therefore, isn't the conclusion that this paper draws already known? Maybe the paper studies HRL methods in different experimental settings, not considerered in existing HRL or Options based frameworks before, but I would imagine similar conclusions can be drawn if a pure HRL system is studied?
The paper mainly does an experimental ablation study for HRL systems, and draws the conclusion of faster exploration led by HRL methods. However, I am not sure whether this conclusion, while assummably true as argued by previous methods too, is convincing enough based on the proposed suite of tasks? 
It might have been more useful if there were similar theoretical contributions or analysis were also made for a stronger claim for HRL methods. Without lack of theoretical analysis or proofs, and simply based on the ablation studies as in this paper, I am not sure whether this paper is yet ready for acceptance in a venue like ICLR. 
In the tasks considered, also there is no termination condition being learnt, and in fact the HRL systems studied here terminates every c time steps (as in HIRO and most of other HRL papers). This might be a significant limitation though. If we learn the termination condition, as in Option-Critic, would we expect similar behaviour in performance and can attribute the benefits of HRL only to exploration?
There are a vast amount of approaches based on identifying bottleneck states, and often they have shown better performance in terms of transfer learning. While in these approaches, often the benefits are claimed to be in terms of both exploration and transfer learning, this paper seems to contradict that? 
Experimentally, there are only few specific tasks considered, like the variations of Ant tasks like AntPush and AntMaze. I am not convinced from the set of experimental results that they are sufficient enough to draw the conclusion that HRL methods only excel due to better exploration. 
Why is the conventional Four Rooms domain ignored? What if we take the Four Rooms domain, change the goal states - I would expect the paper to do such analysis on the range of HRL methods, and then propose a convincing argument. 
I think overall the paper needs more work, before such conclusion can be drawn overall. It seems to me like a strong claim that the benefits of HRL is only due to exploration. The paper does not do enough experimental abltation studies to strengthen the claim, mainly lacking fundamental HRL task setups. It does not do a theoretical analysis studying HRL methods and their benefits either. Overall, I think with these contributions, if similar behaviour persists, then it would make a more convincing argument. 
There are a lot of theoretical papers studying provably sufficient exploration methods. Perhaps such approaches can also be taken here to study HRL methods and whether they provably lead to faster exploration? 
Experimentally, I would expect a more wide range of task setups and domains to be studied - since this is mainly a paper based on experimental studies and trying to draw conclusion for existing HRL methods without proposing new approaches. I think without these carefully studied experiments, the conclusions are over-claimed.