  To improve policy robustness of deep reinforcement learning agents, a line of
recent works focus on producing disturbances of the environment. Existing
approaches of the literature to generate meaningful disturbances of the
environment are adversarial reinforcement learning methods. These methods set
the problem as a two-player game between the protagonist agent, which learns to
perform a task in an environment, and the adversary agent, which learns to
disturb the protagonist via modifications of the considered environment. Both
protagonist and adversary are trained with deep reinforcement learning
algorithms. Alternatively, we propose in this paper to build on gradient-based
adversarial attacks, usually used for classification tasks for instance, that
we apply on the critic network of the protagonist to identify efficient
disturbances of the environment. Rather than learning an attacker policy, which
usually reveals as very complex and unstable, we leverage the knowledge of the
critic network of the protagonist, to dynamically complexify the task at each
step of the learning process. We show that our method, while being faster and
lighter, leads to significantly better improvements in policy robustness than
existing methods of the literature.
