In deep reinforcement learning (RL), adversarial attacks can trick an agent into unwanted states and disrupt training. We propose a system called Robust Student-DQN (RS-DQN), which permits online robustness training alongside Q networks, while preserving competitive performance. We show that RS-DQN can be combined with (i) state-of-the-art adversarial training and (ii) provably robust training to obtain an agent that is resilient to strong attacks during training and evaluation. To ensure Reinforcement Learning (RL) agents behave reliably in the wild, it is important to consider settings where an adversary aims to interfere with the decisions of the agent. Most recent progress in RL has focused on handling continuous states via neural networks [1] [2] [3] . However, small perturbations to the input of neural networks can yield vastly different outputs [4] . These perturbations are also applicable to neural networks deployed in RL, leading to possible security risks [5] . Existing work in the field of robust RL has focused on small, physically plausible perturbations, discrete states [6, 7] , or settings with few inputs [8] [9] [10] [11] [12] . These approaches, however, do not scale to handling gradient-based attacks on large images as studied in supervised classification [4, 13, 14] . In this work we present a new approach for training RL systems to be more robust against adversarial perturbations. The key idea, shown in Fig. 1 , is to split the standard DQN architecture into a policy (student) network S and a Q network in a way which enables us to robustly train the policy network S and use it for exploration, while at the same time preserving the standard way of training the Q network. We then show how to naturally incorporate state-of-the-art defenses developed in supervised deep learning to the setting of reinforcement learning, by training the student network in two ways: (i) via adversarial training with methods such as PGD [14] where we generate adversarial states that decrease the chance the optimal action is selected and use them to train the policy network, and (ii) via provably robust training with symbolic methods [15, 16] which guarantee the network will select the right action in a given state despite any possible perturbation (within a range) of that state. • A novel deep RL algorithm, RS-DQN, designed to be defended with state-of-the-art adversarial training as well as provably robust training. • We show that when no attack is present, RS-DQN and DQN obtain similar scores, while in the presence of attacks, undefended DQNs fail while RS-DQN remains robust. • An evaluation which demonstrates that RS-DQN can produce an agent that is certifiably robust to ±1 pixel intensity changes on Atari games with scores comparable to DQN. arXiv:1911.00887v2 [cs.LG] 7 Nov 2019 train train Adversary sample from buffer D store in buffer D state action S 1 Initialize a state s 0 , weights θ Q , θ S 2 for i = 0, . . . do 3 Pick action a according to exploration strategy with network Q(·; θ Q ) or S(·; θ S ) 4 Play the game (s i+1 , r) := G(s i , a) 5 Store (s i , a, r, s i+1 ) in D 6 if i mod m == 0 then 7 θ − Q := θ Q 8 end if 9 Sample a batch D from D 10 Train the underlying Q−Network: 11 L(θ Q ) := (s,a,r,s )∈D (Y (θ − Q ) − Q(s; θ Q ) a ) 2 12 θ Q := θ Q − η Q ∇ θ Q L(θ Q ) 13 Train the student S from Q: 14 L(θ S ) := (s,a,r,s )∈D L D (s, θ Q , θ S ) 15 θ S := θ S − η S ∇ θ S L(θ S ) 16 end for No defense Adversarial Training Provably Robust Training
We will compare against the state-of-the-art REF in our experiments.
Online Robustness Training for Deep Reinforcement Learning
ArXiv
ArXiv
Computer Science Mathematics