Paper:
Deep reinforcement learning has enabled robots to complete complex tasks in simulation. However, the resulting policies do not transfer to real robots due to model errors in the simulator. One solution is to randomize the simulation environment, so that the resulting, trained policy achieves high performance in expectation over a variety of configurations that could represent the real-world. However, the distribution over simulator configurations must be carefully selected to represent the relevant dynamic modes of the system, as otherwise it can be unlikely to sample challenging configurations frequently enough. Moreover, the ideal distribution to improve the policy changes as the policy (un)learns to solve tasks in certain configurations. In this paper, we propose to use an inexpensive, kernel-based summarization method method that identifies configurations that lead to diverse behaviors. Since failure modes for the given task are naturally diverse, the policy trains on a mixture of representative and challenging configurations, which leads to more robust policies. In experiments, we show that the proposed method achieves the same performance as domain randomization in simple cases, but performs better when domain randomization does not lead to diverse dynamic modes.

Review:
This paper presents a new method for learning diverse policies that can potentially transfer better to new environments. The proposed method aims to find simulation configurations that lead to diverse behaviors using “submodular optimizations” technique; an idea stemmed from past data summarization methods.  
Pros:
-The paper deals with an important problem in RL which is learning robust policies that can transfer better
- Simple idea based on prior information theory literature is proposed. 
- Good incorporation of past methods for improving robustness in RL.
Cons:
>>The paper has provided weak evidence and to support the effectiveness and significance of the proposed approach. The current analysis and experimental evaluations are not convincing.
Relevant baselines are missed and limited tasks are explored:
- A relevant prior work of Pinto et al. (2017) is not considered in the baselines for comparison.  
- Only two tasks are considered in experiments which are not representative of the effectiveness of the proposed approach and does not provide convincing evidence that the proposed method performs better than prior works.
-In the HalfCheetah task only the performance of random baseline is shown and comparison with EpOpt is not reported
-Results of the base DDPG policy is reported in the experiments (Fig 3 and Table 1). For a thorough comparison these missing results are necessary. 
>>Correct citation to the prior work of “domain randomization” is necessary:
- The second paragraph of related work section (first paragraph of page 2) explains that the idea of domain randomization has been first used in computer vision by Tobin et al. 2017. The idea of domain randomization was first proposed and deployed on a real robot platform in Sadeghi and Levine 2016 (Sadeghi, Fereshteh, and Sergey Levine. "CAD2RL: Real single-image flight without a single real image." arXiv preprint arXiv:1611.04201 (2016).) and was later used in Tobin et al. 2017 (this is also explained in the Tobin et al. 2017). Adding proper citation to Sadeghi and Levine 2016 is necessary. 
>> Relevant prior works are missed:
- The proposed idea in the paper is relevant to multi-task RL (e.g. Teh Y, Bapst V, Czarnecki WM, Quan J, Kirkpatrick J, Hadsell R, Heess N, Pascanu R. Distral: Robust multitask reinforcement learning. InAdvances in Neural Information Processing Systems 2017 (pp. 4496-4506). Adding related discussion about prior multi-task RL methods is highly recommended for improving the paper.
- Citating several relevant prior RL methods that deal with transfer learning is missed. A few examples are:
Rusu AA, Vecerik M, Rothörl T, Heess N, Pascanu R, Hadsell R. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286. 2016 Oct 13.
Rusu AA, Rabinowitz NC, Desjardins G, Soyer H, Kirkpatrick J, Kavukcuoglu K, Pascanu R, Hadsell R. Progressive neural networks. arXiv preprint arXiv:1606.04671. 2016 Jun 15.
Rusu AA, Colmenarejo SG, Gulcehre C, Desjardins G, Kirkpatrick J, Pascanu R, Mnih V, Kavukcuoglu K, Hadsell R. Policy distillation. arXiv preprint arXiv:1511.06295. 2015 Nov 19.
>> The format of the paper needs to be improved. 
- The introduction section is rather incomplete and does not properly motivate the problem and the proposed solution.
-The organization of the experimental section should be improved. It is hard to follow the experiments and find the key experimental results in the current version of the paper.