Paper:
Deep reinforcement learning agents are known to be vulnerable to adversarial attacks. In particular, recent studies have shown that attacking a few key steps is effective for decreasing the agent's cumulative reward. However, all existing attacking methods find those key steps with human-designed heuristics, and it is not clear how more effective key steps can be identified. This paper introduces a novel reinforcement learning framework that learns key steps through interacting with the agent. The proposed framework does not require any human heuristics nor knowledge, and can be flexibly coupled with any white-box or black-box adversarial attack scenarios. Experiments on benchmark Atari games across different scenarios demonstrate that the proposed framework is comparable and could be superior to existing methods for learning key steps.

Review:
This paper proposes to learn the ‘key-steps’ at which to to apply an adversarial attack on a reinforcement learning agent. The framing of this problem is a Lagrangian relaxation of a constrained minimization problem which takes the form of an RL problem itself, where the attacking agent’s reward is the negative reward of the target agents plus a penalty (lambda, hyper-parameter) for choosing to attack. The attacking agent’s action space is binary, attack or no attack.
The RL approach is compared with a random attack policy and two heuristic methods for attacking agents in games on the Atari benchmark.
The setting addressed in this work, where the attacker only learns whether/when to attack or not is a greatly simplified version of the full problem. It is (in my opinion, but feel free to correct me) likely that the type of perturbation also being learned has even greater potential, but is (obviously) much harder.
Additionally, although the authors mention this as future work, I think the co-training setting is particularly interesting. As the authors suggest, this could lead to more robust target agents, but additionally I wonder if the type and difficulty of attacks would vary as the target agent trains.
Besides the (picky) complaints, I think the problem formulation is quite reasonable. Again, I find the larger problem extremely interesting, but perhaps this is far too intractable right now. The formulation is straightforward, so while I recognize it as a contribution, it is not a particularly large one.
On the other hand, the experimental results appear somewhat weak to me. 
If you vary the lambda parameter (as in Table 1, but for all games) you should be able to get similar line plots in Figure 5&6 for the RL approach, which would give a much better comparison for the trade-offs as you get for the heuristic methods. I think this comparison would be very interesting and strengthen the existing results in those figures.
The results, currently, do not appear very significant because (1) the gap between the RL solution and the heuristics is very small and (2) these *appear* to be single runs without standard deviations displayed. Can you argue for why these results *are* in fact significant (statistically or otherwise)?
Another question raised by the results is how performance of the attacking policy varies with training. The authors point out that the training is quite small compared to the target policy, but is that because it has already found the best solution it can in that time? How would it improve with more training?
Small note on Figure 7, I think the point for these would be better made by normalizing the respective histograms.
Update:
Thank you for your responses and updating with new results. I think these provide a much better picture of the performance of the RL-based system. Although I am revising my score upward, I still think this is generally a rejection. Obviously I still think the full problem is interesting, but even the key step identification problem would be publishable if the performance was improved or further analysis helped me understand why RL is not doing better (since the heuristics are after all just heuristics). Good luck on future versions of this work.