Paper:
Meta-learning algorithms use past experience to learn to quickly solve new tasks. In the context of reinforcement learning, meta-learning algorithms acquire reinforcement learning procedures to solve new problems more efficiently by utilizing experience from prior tasks. The performance of meta-learning algorithms depends on the tasks available for meta-training: in the same way that supervised learning generalizes best to test points drawn from the same distribution as the training points, meta-learning methods generalize best to tasks from the same distribution as the meta-training tasks. In effect, meta-reinforcement learning offloads the design burden from algorithm design to task design. If we can automate the process of task design as well, we can devise a meta-learning algorithm that is truly automated. In this work, we take a step in this direction, proposing a family of unsupervised meta-learning algorithms for reinforcement learning. We motivate and describe a general recipe for unsupervised meta-reinforcement learning, and present an instantiation of this approach. Our conceptual and theoretical contributions consist of formulating the unsupervised meta-reinforcement learning problem and describing how task proposals based on mutual information can be used to train optimal metalearners. Our experimental results indicate that unsupervised meta-reinforcement learning effectively acquires accelerated reinforcement learning procedures without the need for manual task design and these procedures exceed the performance of learning from scratch.

Review:
Summary: this paper claims to design an unsupervised meta-learning algorithm that does automatically design a task distribution for the target task. The conceptual idea is to propose a task based on mutual information and to train the optimal meta-learner. They also use experiments to show the effectiveness of the proposed approach. 
Overall Comments:
I would think this paper requires a major revision. It is written in a very confusing way. Many terms are directly used without a definition. The problem is also not clearly defined. I have tried to understand everything, but I have to give up in Section 3. Overall, I do not think this paper is ready for publication.
Detailed comments:
	• It would benefit a lot if you can clearly define the original meta-learning procedure and then compare that with the one proposed in this paper.
	• Define ”hand-specified” distribution. This word does not make sense if you claim this is the difference between the meta-learning procedure proposed in this paper and the original meta-learning algorithm. In this paper, you used p(z) to specify a task. I would think p(z) is also “hand-specified”.
	• I am not very sure by what you mean for “task-proposal procedure”, “goal-proposal procedure”
	• In the first paragraph of the intro: what do you mean by “specifying a task distribution is tedious”, is specifying p(z) also “tedious”
	• 2nd paragraph of intro: “automate the meta-training process by removing the need for hand-designed meta-training tasks”. Again, why p(z) is not “hand-designed”
	• Why compare with the original meta-RL algorithm on p(z) is not fair? 
	• What do you mean by “acquire reinforcement learning procedures”?
	• “Environment”, “task” are not clear when they first appear
	• The word “learn” is used everywhere, and is confusing. E.g. what do you mean by “learn new tasks”, “learn a learning algorithm f”, “learn an optimal policy”, “learn a task distribution” …
	• “Reward functions induced by p(z) and r_z(s,a)”: isn’t r_z(s,a) already a reward function? What is “induced”?
	• What is “meta-training” time?
	• What is “no free lunch theorem”?
	• The “controlled-MDP” setting is actually much easier: perhaps you just need to learn the probability distribution. Then for every r_z, we just solve it. Why not compare with this simple algorithm?
	• “Regret” is not defined when it first appears
	• “The task distribution is defined by a latent variable z and a reward function r_z”: why “distribution” is defined by an r.v.?
	• In (2), “regret” should be the (cost of the algorithm) - (the total cost of an optimal policy) — it is not hitting time
	• (3) is confusing, no derivation is given
	• Based on the usual definition of “regret”, how can a “policy” have low regret? Any fixed “policy” would have linear regret …