Paper:
Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manuallydesigned reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing the entropy of the goal distribution together with goal reaching performance, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. Skew-Fit enables self-supervised agents to autonomously choose and practice reaching diverse goals. We show that, under certain regularity conditions, our method converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that it can learn a variety of manipulation tasks from images, including opening a door with a real robot, entirely from scratch and without any manually-designed reward function.

Review:
Summary : 
The paper proposes an exploratory objective that can maximize state coverage in RL. They show that a formal objective for maximizing state coverage is equivalent to maximizing the entropy of a goal distribution. The core idea is to propose a method to maximize entropy of a goal distribution, or a state distribution since goals are full states. They show that the proposed method to maximize the state or goal distribution can lead to diverse exploration behaviour sufficient for solving complex image based manipulation tasks. 
Comments and Questions : 
	- The core idea is to maximize the entropy of the state visitation frequency H(s). It is not clear from the paper whether the authors talk about the normalized discounted weighting of states (a distribution) or the stationary distribution? The entropy of the state visitation distribution only deals with valid states - but I am not sure what it means to maximize the entropy of this term exactly in terms of exploration, since it is neither the discounted weighting of states or the stationary distribution for an infinite horizon task? 
	- The authors do mention that maximizing the entropy of H(s) is not sufficient - so instead suggests for maxmizing entropy of H(s|g). But why is this even sufficient for exploration - if I do not consider new tasks at test time but only the training task? How is this a sufficient exploration objective? Furthermore, since it is the conditional entropy given goal states, the fundamental idea of this is not clear from the paper. 
	- Overall, I am not convinced that an objective based on H(s|g) is equivalent to an maximizing H(s), and why is this even a good objective for exploration? The meaning of H(s) to me is a bit vague from the text (due to reasons above) and therefore H(s|g) does not convince to be a good exploration objective either?
	- The paper then talks about the MI(S;G) to be maximized for exploration - what does this MI formally mean? I understand the breakdown from equation 1, but why is this a sufficient exploration objective? There are multiple ideas introduced at the same time - the MI(s;g) and talking about test time and training time exploration - but the idea itself is not convincing for a sufficient exploration objective. In light of this, I am not sure whether the core idea of the paper is convincing enough to me. 
	- I think the paper needs more theoretical insights and details to show why this form of objective based on the MI(s;g) is good enough for exploration. Theoretically, there are a lot of details missing from the paper, and the paper simply proposes the idea of MI(s;g) and talks about formal or computationally tractable ways of computing this term. While the proposed solutuon to compute MI(s;g) seems reasonable, I don't think there is enough contribution or details as to why is maximizing H(s) good for exploration in the first place.
	- Experimentally, few tasks are proposed comparing skew-fit with other baselines like HER and AutoGoal GAN - but the differences in all the results seem negligible (example : Figure 5). 
	- I am not sure why the discussion of goal conditioned policies is introduced rightaway. To me, a more convincing approach would have been to first discuss why H(s) and the entropy of this is good for exploration (discounted weighting or stationary state distribution and considering episodic and  infinite horizon tasks). If H(s) is indeed a difficult or not sufficient term to maximize the entropy for, then it might make sense to introduce goal conditioned policies? Following then, it might be convincing to discuss why goal conditioned policies are indeed required, and then tractable ways of computing MI(s;g). 
	- Experimentally, I think the paper needs significantly more work - especially considering hard exploration tasks (it might be simple setups too like mazes to begin with), and then to propose a set of new experimental results, without jumping directly to image based tasks as discussed here and then comparing to all the goal conditioned policy baselines. 
Overall, I would recommend to reject this paper, as I am not convinced by the proposed solution, and there are lot of theoretical details missing from the paper. It skips a lot of theoretical insights required to propose a new exploration based objective, and the paper proposes a very specific solution for a set a very specific set of experimental setups.