From: ICLR.cc/2019/Conference/Paper132/Area_Chair1
metareview: This paper introduces unsupervised meta-learning algorithms for RL. Major concerns of the paper include: 1. Lack of clarity. The presentation of the method can be improved. 2. The motivation and justification of applying unsupervised meta-learning needs to be strengthened. More discussions and better motivating examples may be useful. 3. Experimental details are not sufficient and comparisons may not be sufficient to support the aim. Overall, this paper cannot be accepted yet. 
confidence: 5: The area chair is absolutely certain
recommendation: Reject
title: Improvement needed

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer5
title: Some suggestions
comment: I would suggest some experiments that might address the concerns that AnonReviewer3 and I have regarding “inductive bias.” 
- Consider two tasks M1 and M2 with the same dimensionality but different dynamics. Get a set of reward functions from M1 and then try to use them to meta-train on M2. This might illustrate a mismatch in good inductive biases.
- (Less important. An enhancement and to some, clarification.) Consider an additional set of test tasks in Ant with a goal velocity (and obviously, use the same rewards you used in goal position to meta-train). If your claim is correct, we should expect a speed up in both sets as they rely on the same dynamics. 

In addition, I would suggest including a simple motivating example to illustrate why knowing the dynamics might provide a better learning recipe for arbitrary reward function, especially for a model-free method like PG.

I applaud the authors’ continued effort but I feel reluctant to change my assessment.

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer5
title: Some major flaws still remain
comment: I still find some of your claims hard to follow in the revised version but I am pleased to see the effort you have shown: I worried that my original critique might be seen as too discouraging. 

> Firstly, the exposition is hard to follow. For example, the "Task acquisition via random discriminators" subsection, without first mentioning DIAYN, seems out-of-context: what is D_phi_rand(z|s)? a joint distribution of (reward function, state) makes no sense. It only makes sense when there is a stochastic process, e.g. MDP coupled with a policy. 
I still find this section confusing but I also think _my_ initial interpretation (above) is incorrect. Suppose that the support for z has k many values. Is it correct to say that you initialize a random (neural network) parameter phi — this gives a random discriminator D_phi over k classes — then you use _the score of the z-th class_ as reward when z is sampled, i.e., r_z(s) = log(D_phi(s))[z]? I mean that phi is not modified and s here is not observed in running a policy, possibly derived from optimizing for some metric as in DIYAN. Is this correct? If so, I am very confused by your presentation of “[g]iven a uniformly distributed random variable z ~ p(z), we define a random discriminator as a parametric function D_phi(z|s).” The notation of D makes it look like a distribution, in which case your presentation strongly suggests D(z|s) = D(z) = 1/k given z ~ uniform(k), whereas a random parameter phi will likely give a non-uniform scores over z (I assume that this is what you want). (That is also why I originally thought you are doing something like DIYAN here.) You might want to put more thought into the presentation of this idea. I think this idea has potential, especially when you include the sample (and computational) complexity during meta-training in your analysis. 

> It seems to want to suggest it through a kind of "duality" between skills/policies and rewards/tasks
At some level, I feel like this is a rebranding of DIYAN but you have addressed this somewhat in the revision and your response pointing out that DIYAN returns “skills” whereas you are more interested using the reward functions (also computed in DIYAN). But at a deeper level — I don’t know if you agree — this is due to the downstream choice of MAML+PG which _requires_ reward functions. This feels almost like a shortcoming of restricting yourself to MAML+PG. Moreover, wouldn’t this commitment mean that after extracting the reward functions from DIYAN, you will have to train all over _again_ with PG (as it is very hard to train off-policy). Of course, the parameters obtained would presumably be different from the theta_i from DIYAN (and I see that you tried to illustrate this by the experimental comparison with DIYAN-finetuning). But I think an analysis of these overheads is absolutely necessary (also pointed out by AnonReviewer3).

To summarize, my current understanding is that you are claiming that we can benefit — learn a given task faster — from exploring the environment dynamics without a reward function. And the current paper concerns how to incorporate such information _in the framework of meta-learning_. I think you misunderstood my original comment “[t]he restriction of tasks to different reward functions is limiting.” On the contrary I completely agree with your assessment that many interesting tasks share the same transition dynamics and this setting is definitely of value from a RL perspective. However, I made the comment from the perspective of your claim of this being an unsupervised approach to meta-reinforcement learning which seems demand much more evidence. But then if we consider the problem in the more limited scope, it seems necessary to compare, including overhead, with RL methods that also incorporate transition information without rewards (which precludes model-free methods). Maybe you can make your case by comparing computational complexities (as you suggested in the reply)?

Pro
- (Additional) Numerical experiments with relevant baselines.
- Applying meta-learning methods to incorporate information about dynamics into model-free RL methods.

Con
- The claims tend to be too board and presentation lacks rigor/precision.
    - This is not exactly an (general) _approach_ to meta-reinforcement learning, rather a particular approach based on meta-RL to a setting in reinforcement learning where environment dynamics can be accessed before a test task (as specified by a test reward) is given, i.e., “unsupervised exploration.” Certainly, there is merit to consider meta-learning methods in this setting.
    - Analysis of overhead before meta-testing should be included.
- I still don’t think model-based methods should be dismissed easily as, again, they seem the most natural to take advantage of the proposed setting.

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer3
title: Some concerns addressed but not enough
comment: Some of my concerns have been addressed on the clarity with a focus on the experiments and some major typos. However I follow AnonReviewer5 as I think the paper needs a major rewriting to improve clarity and be much more formal in the writing. For this reasons I maintain my current score.

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer3
title: Thank you
comment: Thank you for the answer.

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer5
title: Both the exposition and the formulation leave much to be desired
review: The paper considers a particular setting of so-called meta-reinforcement learning (meta-RL) where there is a distribution over reward functions (the transition function is fixed) and with some access to this distribution, the goal is to produce a learning algorithm that  "learns well" on the distribution. At training time, a sample of reward functions are drawn and an algorithm returns a learning program that at test time, can reinforcement learn the test environment as specified by a test reward function. The paper proposes to generate a set of training reward functions instead of relying on some "manual" specification, thus the term "unsupervised" in the title. It also proposes an algorithm, basing on a recent work in skill discovery (DIAYN, not yet peer-reviewed), to find such reward functions. 

Firstly, the exposition is hard to follow. For example, the "Task acquisition via random discriminators" subsection, without first mentioning DIAYN, seems out-of-context: what is D_phi_rand(z|s)? a joint distribution of (reward function, state) makes no sense. It only makes sense when there is a stochastic process, e.g. MDP coupled with a policy. 

Secondly, the reasoning is very informal based on a vague vocabulary (not trying to pick on the authors, these are not uncommon in deep learning literature) are used without rigorous arguments. Section 3.1 brought up a natural objection -- I applaud the authors' self-critique -- based on "no free lunch theorem," but it dismisses it via "the specific choice for the unsupervised learning procedure and meta-learning algorithm can easily impose an inductive bias" without specifying what (and how) choice leads to what "inductive biases." This is crucial as the authors seem to suggest that although the "inductive bias" is important -- task design expresses such -- an unsupervised method, which requires no supervision, can do as well. 

Thirdly, the baseline comparison seems inappropriate to me. The "fair" baseline the authors proposed was to RL a test task from scratch. But this is false as the meta-RL agent enjoys access to the transition dynamics (controlled Markov process, CMP in the paper) during the so-called meta-training (before meta-testing on the test task). In fact, a more appropriate baseline would be initialize an RL agent with with a correct model (if sample complexity in training is not a concern, which seems to be the case as it was never addressed in the paper) or a model estimated from random sample transitions (if we are mindful of sample complexity which seems more reasonable to me). One may object that a (vanilla) policy-gradient method cannot incorporate an environment model but why should we restrict ourselves to these model-free methods in this setting where the dynamics can be accessed during (meta-)training?

Pros:
1. It connects skill discovery and meta-RL. Even though such connection was not made explicitly clear in the writing, its heavy reliance on a recent (not yet peer-reviewed) paper suggests such. It seems to want to suggest it through a kind of "duality" between skills/policies and rewards/tasks (z in the paper denotes the parameter of a reward function and also the parameter of policy). But are there any difference between the two settings? 

Cons:
1. The writing is imprecise and often hard to follow.
2. The setting considered is not well motivated. How does an unsupervised method provide the task distribution before seeing any tasks? 
3. The restriction of tasks to different reward functions made the proposed baseline seem unfairly weak.

In summary, I could not recommend accepting this work as it stands. I sincerely hope that the authors will be more precise in their future writing and focus on articulating and testing their key hypotheses.
rating: 3: Clear rejection
confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer1
title: Review
review: The authors propose a framework for unsupervised meta-reinforcement learning. This aims to perform meta-learning in reinforcement learning context without an specification of the meta tasks being pre-specified. The authors propose two algorithms to acquire the task distributions (unsupervised). In particular the better performing approach relies on the recently introduced DIAYN algorithm.  Experiments are presented on several simple benchmark datasets.

The authors propose an interesting formulation of a useful problem: finding tasks automatically that aid meta-learning. To the best of my knowledge this is indeed a novel idea and indeed an important one. On the other hand the authors only take relatively early steps towards solving this task and the discussion of what is a good unsupervised task selection is underwhelming. Indeed one is not left of a clear idea of what kind of inductive biases would be a valid approach to this problem and why the authors consider specifically the two approaches described.    

For the experiments it seems a lot of the key improvements come from the DIAYN algorithm. The experiments are also presented on relatively toy tasks and mainly compare to RL from scratch approaches.  It would be interesting to see the effectiveness of these methods on harder problems. For the experiments I would be interested to know if one could compare directly to using DIAYN as in the original Eysenbach et al for example as an initialization. 

Overall the paper presents several interesting results and I think the high level formulation could have some potential impacts, although the limits  of such an approach are not completely clear and whether it can be effective on complex tasks is not fully known yet.

rating: 6: Marginally above acceptance threshold
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: ICLR.cc/2019/Conference/Paper132/AnonReviewer3
title: An interresting but rushed paper
review: *Summary:* The present paper proposes to use  Model Agnostic Meta Learning to (meta)-learn in an unsupervised fashion the reward function of a Markov decision process in the context of Reinforcement Learning (RL). The distribution of tasks corresponds to a distribution of reward functions which are created thanks to random discriminators or diversity driven exploration.

*Clarity:* The goal is well stated but the presentation of the method is confusing.

There is a  constant switch between caligraphic and roman D. Could you homogenize the notations?

Could you keep the same notation for the MDP (eg in the introduction and 3.5, the discount factor disappeared)

In the introduction the learning algorithm takes a MDP in mathcal{T} and return a policy. In the remaining of the paper mathcal{D} is used. Could you clarify? I guess this is because only the reward of the MDP is meta-learned, which is itself based on D_phi?

you choose r = log(Discriminator). Could you explain this choice? Is there alternative choices?

In subsection 3.4, why the p in the reward equation?

Algorithm 1 is not clear at all and needs to be rewritten:
   - Could you specify the stopping criterion for MAML you used?
   - Could you number the steps of the algorithm?

Concerning the experiments:

In my opinion the picture of the dataset ant and cheeta is irrelevant and could be removed for more explainations of the method.

It would be very nice to have color-blind of black and white friendly graphs.

In the abstract, I don't think the word demonstrate should be used about the experimental result. As pointed out in the experimental section the experiment are here rather to give additional insight on why and when the proposed method works well.

Your method learns faster than RL from scratch on the proposed dataset in terms of iteration. What about monitoring the reward in terms of time, including the meta-learning step. Is there any important constant overhead in you the proposed method? How does the meta training time impact the training time? Do you have examples of datasets where the inductive bias is not useful or worst than RL from scratch? If yes could you explain why the method is not as good as RL from scratch?

The presentation of the result is weird.
Why figure 4 does not include the ant dataset? Why no handcrafted misspecified on 2D navigation?
Figure 3 and 4 could be merged since many curves are in common.

How have you tuned your hyperparameters of each methods? Could you put in appendix the exact protocol you used, specifying the how hyperparameters of the whole procedured are chosen, what stopping criterion are used, for the sake of reproducibility. A an internet link to a code repository used to produce the graphs would be very welcome in the final version if accepted.

In the conclusion, could you provide some of the questions raised?

*Originality and Significance:* As I'm not an expert in the field, it is difficult for me to evaluate the interest of the RL comunity in this work. Yet to the best of my knowledge the work presented is original, but the lack of clarity and ability to reproduce the results might hinder the impact of the paper. 

Typos:
Eq (2) missing a full stop
Missing capital at ´´ update´´  in algorithm 1
End of page 5, why the triple dots?

rating: 4: Ok but not good enough - rejection
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper