Paper:
ENVIRONMENT PROBING INTERACTION POLICIES
A key challenge in reinforcement learning (RL) is environment generalization: a policy trained to solve a task in one environment often fails to solve the same task in a slightly different test environment. A common approach to improve interenvironment transfer is to learn policies that are invariant to the distribution of testing environments. However, we argue that instead of being invariant, the policy should identify the specific nuances of an environment and exploit them to achieve better performance. In this work, we propose the "Environment-Probing" Interaction (EPI) policy, a policy that probes a new environment to extract an implicit understanding of that environment's behavior. Once this environmentspecific information is obtained, it is used as an additional input to a task-specific policy that can now perform environment-conditioned actions to solve a task. To learn these EPI-policies, we present a reward function based on transition predictability. Specifically, a higher reward is given if the trajectory generated by the EPI-policy can be used to better predict transitions. We experimentally show that EPI-conditioned task-specific policies significantly outperform commonly used policy generalization methods on novel testing environments.

Review:
The submission presents a reinforcement learning method for exploring/probing the environment to determine an environment’s properties and exploit these during later tasks. The method relies on jointly learning an embedding that simplifies prediction of future states and a policy that maximises a curiosity/intrinsic motivation like reward to learn to explore areas where the prediction model underperforms. In particular, the reward is based on the difference between prediction based on the learned embedding and prediction based on a prior collected dataset, such that the reward optimises to collect data with a large difference between the prediction accuracy of both models. The subsequently frozen policy and embedding are then used in other domains in a system identification like manner with the embedding utilised as input for a final task policy. The method is evaluated on a striker and hopper environment with varying dynamics parameters and shown to outperform a broad set of baselines. 
In particular the broad set of baselines and small performed ablation study on the proposed method are quite interesting and beneficial for understanding the approach. However, the ablation study could be in more detail with respect to the additional training variations (Section 4.1.3; e.g. without all training tricks). Additionally, information about the baselines should be extended in the appendix as e.g. different capacities alone could have an impact where the performances of diff. algorithms are comparably similar. In particular, additional information about the training procedure for the UP-OSI (Yu et al 2017) baseline is required as the original approach relies on iterative training and it is unclear if the baseline implementation follows the original implementation (similar to Section 4.1.3.). 
Overall the submission provides an interesting new direction on learning system identification approaches, that while quite similar to existing work (Yu et al 2017), provides increased performance on two benchmark tasks. The contribution of the paper focuses on detailed evaluation and, overall, beneficial details of the proposed method. The novelty of the submission is however limited and highly similar to current methods.
Minor issues:
- Related work on learning system identification:
Learning to Perform Physics Experiments via Deep Reinforcement Learning
Misha Denil, Pulkit Agrawal, Tejas D Kulkarni, Tom Erez, Peter Battaglia, Nando de Freitas