Paper:
When mastering a complex manipulation task, humans often decompose the task into sub-skills of their body parts, practice the sub-skills independently, and then execute the sub-skills together. Similarly, a robot with multiple end-effectors can perform a complex task by coordinating sub-skills of each end-effector. To realize temporal and behavioral coordination of skills, we propose a hierarchical framework that first individually trains sub-skills of each end-effector with skill behavior diversification, and learns to coordinate end-effectors using diverse behaviors of the skills. We demonstrate that our proposed framework is able to efficiently learn sub-skills with diverse behaviors and coordinate them to solve challenging collaborative control tasks such as picking up a long bar, placing a block inside a container while pushing the container with two robot arms, and pushing a box with two ant agents.

Review:
The paper presents a hierarchical reinforcement learning method for coordination of multiple cooperative agents with pre-learned adaptable skills. These skills are learned via a maximum entropy objective where diversity of behaviour given each skill is maximised and controlled via a latent conditioning vector. This allows controllability of the variation in skill execution by the meta-policy via changing the skill-specific latent vectors. The paper presents empirical results in manipulation (pick-push-place and moving a long bar by coordinating two Jaco arms) and locomotion (two Ants pushing a large block to a goal location). The method proposed outperforms the baselines reported. 
Overall, this paper addresses an interesting problem and can be impactful with the caveat for some clarifications and analysis. Given that the authors address my concerns, I would be willing to increase my score.
The main novelty of this work lies in how one can learn sub-skills that can be leveraged and adapted for down-stream tasks. The problem setting used to test the method is a multi-agent setting where it is crucial that skills are adapted to enable cooperation. I found the environments and the problem setting generally interesting and important for testing the proposed method. I have a few concerns that I listed below:
1) I found the notations at times inconsistent and confusing. It would have helped to see some more details on the diagram (Figure 2) to understand how everything fits together. 
2) The set of skills for the two agents are selected by the meta-policy in every T_low steps. It looks like in the Jaco environments T_low = 1. Can you comment on this? This seems slightly concerning since it seems like the meta-controller is treating these skills as primitive actions rather than temporally extended behaviour.
3) Looking at the training curves in Figure 4, there seems to be a really high variance in performance of the method. Can you comment on this as this seems concerning. Could you run more seeds to improve this?
4) It is nice to see in section 4.5 how the hyper-parameters balancing diversity in combination with external reward (equation 4) is tuned and how sensitive that is to achieving adaptability for downstream tasks. The only criticism I have is that it is difficult to understand from "Episode reward" on y-axis what the success rate is (similar to Figure 4)? It would’ve been nice to report results in a consistent way throughout the paper for these environments. 
5) Given that all the tasks in the experiments are cooperative multi-agent settings, I would have liked to see more in depth discussion regarding alternative multi-agent methods. The multi-agent baseline provided (which is using a decentralized policy with a shared critic, inspired by Lowe et al., 2017) seems fair, but I wonder if there has been more recent work in this direction that could have been highlighted?