Adversarial Training Blocks Generalization in Neural Policies
Abstract
Deep neural networks have made it possible for reinforcement learning algorithms to learn from raw high dimensional inputs.
This jump in the progress has caused deep reinforcement learning algorithms to be deployed in many different ﬁelds from ﬁnancial markets to biomedical applications.
While the vulnerability of deep neural networks to imperceptible speciﬁcally crafted perturbations has also been inherited by deep reinforcement learning agents, several adversarial training methods have been proposed to overcome this vulnerability.
In this paper we focus on state-of-the-art adversarial training algorithms and investigate their robustness to semantically meaningful natural perturbations ranging from changes in brightness to rotation.
We conduct several experiments in the OpenAI Atari environments, and ﬁnd that state-of-the-art adversarially trained neural policies are more sensitive to natural perturbations than vanilla trained agents.
We believe our investigation lays out intriguing properties of adversarial training and our observations can help build robust and generalizable neural policies.
Introduction
The advancements in deep neural networks lead to wide spread of applications from image classiﬁcation Krizhevsky et al.
(2012) to natural language processing Sutskever et al.
(2014), and from speech recognition Hannun et al.
(2014) to self learning systems Mnih et al.
Yet the ability to learn from raw high dimensional data brought some of the drawbacks inherited from deep neural networks to deep neural policies.
Szegedy et al.
(2014) showed that speciﬁcally crafted imperceptible perturbations can lead to misclassiﬁcation in image classiﬁcation.
After this initial work a new research area emerged to investigate the vulnerabilities of deep neural networks against speciﬁcally crafted adversarial examples.
While various works studied many different ways to compute these examples Carlini & Wagner (2017);
Madry et al.
; Goodfellow et al.
; Kurakin et al.
several works focused on studying ways to increase the robustness against such speciﬁcally crafted perturbations, based on training with the existence of such perturbations Madry et al.
; Tramèr et al.
Goodfellow et al.
; Xie & Yuille (2020).
As image classiﬁcation suffered from this vulnerability towards worst-case distributional shift in the input a series of work conducted in deep reinforcement learning showed that deep neural policies are also susceptible towards the speciﬁcally crafted imperceptible perturbations Huang et al.
Kos & Song (2017); Pattanaik et al.
Lin et al.
While one line of work put effort on exploring these vulnerabilities in deep neural policies Korkmaz (2020a, 2021a,d,b,c), another line in parallel focused making them robust and reliable via adversarial training Pinto et al.
Mandlekar et al.
; Zhang et al.
To be able to build generalizable and robust deep neural policies, in this paper we approach the problem of performance degradation with respect to observed input from a wider perspective of distributional shift and make the following contributions:
• We conduct an investigation on the robustness of state-of-the-art adversarially trained deep neural policies against various types of distributional shift in the input. • We perform several experiments in the OpenAI Atari baselines.
• We compare the performance drop of the vanilla trained agents with the state-of-the-art adversarially trained agents against natural semantically meaningful perturbations.
• We ﬁnd that vanilla trained agents are more robust against natural semantically meaningful perturbations than the state-of-the-art adversarially trained agents.
Background 2.1 Adversarial Examples
Adversarial examples were introduced in computer vision by Szegedy et al.
based on producing perturbations via a box constrained optimization method.
Goodfellow et al.
(2015) proposed a fast method called the fast gradient sign method (FGSM) to produce adversarial examples based on the linearization of the cost function used to train the network at the input sample point.
[Equation]
where x represents the input, y represents the labels and J ( x, y ) represents the cost function used to train the deep neural network.
Kurakin et al.
(2016) propose the iterative version of the fast gradient sign method inside an (cid:15) -ball.
[Equation]
This method is also known as projected gradient descent (PGD) as proposed by Madry et al.
2.2 Deep Reinforcement Learning and Adversaries
Initially adversarial examples were introduced to the deep reinforcement learning domain by Huang et al.
(2017) and Kos & Song (2017) concurrently by utilizing FGSM as proposed by Goodfellow et al. (2015).
Korkmaz (2020a) proposed to optimize the adversarial example search via Nesterov momentum.
Several studies have been conducted to make deep reinforcement learning policies more robust to such speciﬁcally crafted malicious examples.
Mandlekar et al.
(2017) use adversarial examples produced via modifying the environment by taking the gradient of the cost function with respect to input at training time to regularize the policy in an attempt to increase robustness.
Pinto et al.
(2017) model the interaction between the perturbation maker and the agent as a zero-sum
Markov game, and propose a joint training algorithm to improve robustness against an adversary that aims to minimize the expected cumulative reward of the agent.
Gleave et al.
(2020) model the relationship between the adversary and the agent as a zero-sum game where the agent is limited to taking natural actions in the environment rather than minimal (cid:96) p -norm bounded perturbations, and proposed a self-playing approach to gain robustness against such an adversary.
Finally, Zhang et al.
(2020) propose a modiﬁed MDP called a state-adversarial MDP with the aim of obtaining theoretically principled robust polices.
Recently, Korkmaz (2021d,a) demonstrated the vulnerabilities of state-of-the-art adversarial training techniques in deep reinforcement learning.
In our work, we further will show that adversarial training causes problems on the generalization capabilities of deep reinforcement learning agents.
Experimental Setup
In our experiments we use OpenAI Brockman et al.
Atari baselines Bellemare et al.
Our models are trained with DDQN Wang et al.
(2016) and Zhang et al.
We test trained policies averaged over 10 episodes.
We measure the performance drop of the agent as,
[Equation]
where Score clean represents the score from a clean run of the game where no perturbations are introduced to the agent’s observations, Score ﬁxedmin represents the minimum score available for a given game, and Score adv represents a run of the game where perturbations are introduced to agent’s observation system.
Adversarially Trained Models under Natural Perturbations
In this paper we focus on investigating adversarial training and its effects on generalization of neural policies.
In particular, we focus on the state-of-the-art adversarial training method proposed by Zhang et al.
In this paper the authors propose to model the observation perturbations as a modiﬁed MDP, which they refer to as a state-adversarial MDP, with the aim of making the agent more robust towards natural measurement errors or adversarial noises.
The authors test their adversarially trained models under the PGD attack proposed by Madry et al.
In our paper we test adversarially trained DDQN models proposed by Zhang et al.
(2020) under several natural perturbations proposed by Korkmaz (2020b).
In particular, we introduce several minimal natural perturbations (e.g. rotation, brightness, contrast and compression artifacts) to deep reinforcement learning policy observation systems and investigate the performance drop of the trained policies.
Note that these natural modiﬁcations are quite small and some of the perturbations are not even recognizable by human perception (see Figure 1).
Most importantly, we compare the proposed state-of-the-art adversarially trained agents with vanilla trained agents under the natural semantically meaningful perturbations and ﬁnd that vanilla trained models are more robust than adversarially trained models against many natural perturbations including brightness, rotation, compression artifacts, and contrast.
While we observe this particular sensitivity increase to natural perturbations in adversarially trained models in baselines like Atari environments where the generalization capabilities of deep reinforcement learning agents is not the primary concern, this sensitivity increase caused by adversarial training might cause severe problems for the environments where generalization is essential for learning reasonable policies.
In particular, Cobbe et al.
(2019) environment is more challenging and focused on testing generalization capabilities compared to Atari Baselines.
The observation on the limitations of adversarial trained neural policies towards such natural perturbations implies that adversarially trained neural policies are going to experience challenges in obtaining reasonable policies in these type of environments proposed by Cobbe et al.
The fundamental takeaway from these results is the fact that current state-of-the-art proposed solutions to resolve the sensitivity towards imperceptible perturbations indeed do not solve this problem and make it even worse in the case of more natural perturbations.
Conclusion
In this paper we focused on the generalization capabilities of vanilla trained and the state-of-the-art adversarially trained neural policies.
In particular, we tested adversarially trained neural policies under semantically meaningful natural perturbations and we found that vanilla trained deep neural policies are more robust against natural perturbations than adversarially trained deep neural policies.
We further argue that this kind of sensitivity increase towards natural perturbations in adversarially trained models can hurt generalization.
We believe our study provides a holistic view on the robustness of adversarial training for deep neural policies and can contribute to designing resilient and robust self learning systems for future work.