Paper:
Deep reinforcement learning (RL) policies are known to be vulnerable to adversarial perturbations to their observations, similar to adversarial examples for classifiers. However, an attacker is not usually able to directly modify another agent's observations. This might lead one to wonder: is it possible to attack an RL agent simply by choosing an adversarial policy acting in a multi-agent environment so as to create natural observations that are adversarial? We demonstrate the existence of adversarial policies in zero-sum games between simulated humanoid robots with proprioceptive observations, against state-of-the-art victims trained via self-play to be robust to opponents. The adversarial policies reliably win against the victims but generate seemingly random and uncoordinated behavior. We find that these policies are more successful in high-dimensional environments, and induce substantially different activations in the victim policy network than when the victim plays against a normal opponent. Fine-tuning protects a victim against a specific adversary, but the attack method can be successfully reapplied to find a new adversarial policy. Videos are available at https://attackingrl.github.io/.

Review:
The authors start with the valid observation that for controlled systems the threat model of explicitly flipping pixels or otherwise changing the inputs directly is less relevant than the case of identifying bad inputs in the environments (which the authors rename "natural observations") that cause the controller under attack to leave its stable region and end up in bad states for itself, unable to recover sometimes. This paper is an empirical exploration of this phenomenon, using as examples trained agents in a Gym environment and involving two humanoid robot simulations interacting with each other in various ways. The original policies are taken from previous training, so the main experimental contribution here is to solve an RL problem on the attacker's side to try to find the bad regions in the victim's control space, going via the means of the observation-based coupling.
I think these are timely issues. I agree that, to the extent that DRL policies are being seriously considered as candidates for various practical problems, we should seriously ask questions about the (lack of) stability of solutions. 
I also think that the paper would be stronger if the authors considered the rich history in control design, formal methods and other communities where this approach to counter-example based thinking is fairly standard and in particular where the use of optimization methods to find 'bugs' has been studied in some depth. 
So, for instance, the authors give us useful quantification to support the observation that the process of attacker RL finds ways to put the victim outside its stable region where it is naturally unstable and ineffective. However, they do not observe that there is a literature on systematically performing 'controller testing' via optimization using various techniques. Here are just a few examples to show the diversity:
Ghosh, S., Berkenkamp, F., Ranade, G., Qadeer, S., & Kapoor, A. (2018, May). Verifying controllers against adversarial examples with Bayesian optimization. In 2018 IEEE International Conference on Robotics and Automation (ICRA) (pp. 7306-7313). IEEE.
Ravanbakhsh, H., & Sankaranarayanan, S. (2016, October). Robust controller synthesis of switched systems using counterexample guided framework. In 2016 international conference on embedded software (EMSOFT) (pp. 1-10). IEEE.
The authors might find it interesting to note that this approach of viewing the control problem 'backwards' to generate new instances has a deeper history in control theory (of course without discussion of NNs etc), e.g., 
Doyle, J., Primbs, J. A., Shapiro, B., & Nevistic, V. (1996, December). Nonlinear games: examples and counterexamples. In Proceedings of 35th IEEE Conference on Decision and Control (Vol. 4, pp. 3915-3920). IEEE.
All that said, I find the results presented here to be plausible and pointing in the desired direction for exploring how to robustify DRL. The authors defer training using these examples to future work but I think the paper would be more self-contained if that were actually demonstrated here already. For reasons I mention above, the existence of this kind of weakness in controllers is not really surprising to people who have thought deeply about control, and it is only to be expected that NN based parameterisations of control would only be even more vulnerable. So, the more satisfying result would be the positive one that shows how to train DRL to be (more) robust under such attacks.