Paper:
We present an adversarial exploration strategy, a simple yet effective imitation learning scheme that incentivizes exploration of an environment without any extrinsic reward or human demonstration. Our framework consists of a deep reinforcement learning (DRL) agent and an inverse dynamics model contesting with each other. The former collects training samples for the latter, and its objective is to maximize the error of the latter. The latter is trained with samples collected by the former, and generates rewards for the former when it fails to predict the actual action taken by the former. In such a competitive setting, the DRL agent learns to generate samples that the inverse dynamics model fails to predict correctly, and the inverse dynamics model learns to adapt to the challenging samples. We further propose a reward structure that ensures the DRL agent collects only moderately hard samples and not overly hard ones that prevent the inverse model from imitating effectively. We evaluate the effectiveness of our method on several OpenAI gym robotic arm and hand manipulation tasks against a number of baseline models. Experimental results show that our method is comparable to that directly trained with expert demonstrations, and superior to the other baselines even without any human priors.

Review:
The paper proposes a novel exploration strategy for self-supervised imitation learning. An inverse dynamics model is trained on the trajectories collected from a RL-trained policy. The policy is rewarded for generating trajectories on which the inverse dynamics model (IDM) currently works poorly, i.e. on which IDM predicts actions that are far (in terms of mean square error) from the actions performed by the policy. This adversarial training is performed in purely self-supervised way. The evaluation is performed by one-shot imitation of an expert trajectory using the IDM: the action is predicted from the current state of the environment and the next state in the expert’s trajectory. Experimental evaluation shows that the proposed method is superior to baseline exploration strategies for self-supervised imitation learning, including random and curiosity-based exploration. 
Overall, I find the idea quite appealing. I am not an expert in the domain and can not make comments on the novelty of the approach. I found the writing mostly clear, except for the following issues:
- the introduction has not made it crystal clear that the considered paradigm is different from e.g. DAGGER and GAIL in that expert demonstrations are used at the inference time. A much wider audience is familiar with the former methods, and this distinction should have be explained more clearly.
- Section 4.2.: “As opposed to discrete control domains, these tasks are especially challenging, as the sample complexity grows in continuous control domains.” - this sentence did not make sense to me. It basically says continuous control is challenging because it is challenging. 
- I did not understand the stabilization approach. How exactly Equation (7) forces the policy to produce “not too hard” training examples for IDM? Figure 4 shows that it is on the opposite examples with small L_I that are avoided by using \delta > 0.
- Table 1 - it is a bit counterintuitive that negative numbers are better than positive numbers here. Perhaps instead of policy’s deterioration you could report the relative change, negative when the performance goes down and positive otherwise?
I do have concerns regarding the experimental evaluation:
- the “Demos” baseline approach should be explained in the main text! In Appendix S.7 I see that 1000 human demonstrations were used for training. Why 1000, and not 100 and not 10000?  How would the results change? This needs to be discussed. Without discussing this it is really unclear how the proposed method can outperform “Demos”, which it does pretty often.
- it is commendable that 20 repetitions of each experiment were performed, but I am not sure if it is ever explained in the paper what exactly the upper and lower boundaries in the figures mean. Is it the standard deviation? A confidence interval? Can you comment on the variance of the proposed approach, which seems to be very high, especially when I am looking at high-dimensional fetch-reach results?
- the results of “HandReach” experiments, where the proposed method works much worse than “Demos” are not discussed in the text at all
- overall, there is no example of the proposed method making a difference between a “working” and “non-working” system, compared to “Curiosity” and “Random”. I am wondering if improvements from 40% to 60% in such cases are really important. In 7 out of 9 plots the performance of the proposed method is less than 80% - not very impressive. "Demos" baseline doesn't perform much better, but what would happen with 10000 demonstrations?
- there is no comparison to behavioral cloning, GAIL, IRL. Would these methods perform better than learning IDM like "Demos" does?
I think that currently the paper is slightly below the threshold, due to evaluation issues discussed above and overall low performance of the proposed algorithm. I am willing to reconsider my decision if these issues are addressed.