D ISTILLED A GENT DQN FOR P ROVABLE A DVERSARIAL R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
As deep neural networks have become the state of the art for solving complex reinforcement learning tasks, susceptibility to perceptual adversarial examples have become a concern.
The transferability of adversarial examples is known to enable attacks capable of tricking the agent into bad states.
In this work we demonstrate a simple poisoning attack able to keep deep RL from learning, and into fooling it when trained with defense methods commonly used for classiﬁcation tasks.
We then propose an algorithm called DadQN, based on deep Q-networks, which enables the use of stronger defenses, including defenses enabling the ﬁrst ever on-line robustness certiﬁcation of a deep RL agent.
I NTRODUCTION
To ensure Reinforcement Learning (RL) behaves reliably in the real world, it is important to consider settings where an adversary aims to actively hinder the learning process of the agent.
While prior work has explored robustness in the setting of discrete RL (Morimoto & Doya, 2005; Boyan & Moore, 1995), most recent progress in the ﬁeld has focused on dealing with continuous states by using neural networks (Sutton et al., 2000; Mnih et al., 2013; Peters & Schaal, 2008).
In this work we present a new approach for training RL systems to be more (provably) robust.
The key idea is to decouple the DQN network architecture into a policy (student) network S and a Q-network in a way which allows us to robustly train the policy network and use it for exploration while at the same time preserving the standard way in which the Q network is trained.
We then show how to naturally incorporate state of the art defenses, developed in the context of deep supervised, learning to the reinforcement learning setting by training the student network in two ways: (i) via adversarial training with methods such as FGSM where we generate adversarial states that decrease the chance the optimal action is selected, and (ii) via provably robust training with symbolic methods which guarantee the network will select the right action in a given state despite any possible perturbation (within a range) of that state.
Contributions Our main contributions are:
The ﬁrst deep RL algorithm, DadQN, designed to be adversarially defended with state-ofthe-art adversarial training as well as provably robust training into our RL algorithm.
To our best knowledge, this is the ﬁrst time, on-line robustness certiﬁcation has been achieved with a deep RL agent.
• An attack, illustrated in Figure 1, that hinders learning in modern deep RL algorithms.
• An evaluation demonstrating that DadQN can defend against adversarial attacks when defenses against DQNs catastrophically fail, while being of comparable performance when adversarial attacks are not present.  R ELATED W ORK At roughly the same time as deep reinforcement learning methods were ﬁrst being developed Schulman et al.
; Mnih et al.
(2013) it was noticed by Goodfellow et al.
that in the [Figure]
setting of supervised learning, classiﬁers were susceptible to adversarial attacks by introducing imperceptible perturbations to the inputs.
At the same time, it was determined that training using these adversarially crafted images could serve as a way to defend against them.
Later, Carlini &
Wagner (2016) determined that many defensive techniques were insufﬁcient. A particularly successful defense is that of Madry et al. This method uses adversarial examples generated by the PGD attack as training examples (instead of the original examples). While it is experimentally effective, it provides no guarantees about the resulting network’s robustness.
Carlini & Wagner (2017) introduced further method for evaluating the robustness of neural networks.
Katz et al.
(2017) provided the ﬁrst system which could (for very small networks) determine local-robustness, that is, at test time the network is proven to be either (cid:15) -robust for an individual image.
Gehr et al.
(2018) used abstract interpretation to scale the local robustness analysis by sacriﬁcing completeness.
Raghunathan et al.
(2018) introduced a technique to train networks to be certiﬁably robust (though again limited to small networks).
Multiple recent techniques have been developed since to train increasingly larger networks to be certiﬁably robust including Wong & Kolter (2018); Mirman et al.
; Dvijotham et al.
; Wong et al.
These techniques represent different points in the spectrum of classiﬁcation accuracy, certiﬁable robustness, experimental robustness, and training speed.
While Pattanaik et al.
(2017) used adversarial examples to train an agent and improve its experimental robustness, Behzadan & Munir (2017b) noticed that often doing this prevents learning.
Behzadan & Munir (2017a) further demonstrated that this effect could be intentionally ampliﬁed to prevent learning intentionally on the part of an adversary.
Gu et al.
(2018) demonstrated a technique for training the A3C algorithm to be adversarially robust by training a parallel adversarial actor-critic pair for each protagonist actor-critic pair.
The analysis of their technique is limited to noisy environment adversaries, and not intentional agent-aware adversaries such as FGSM.
Similarly, Ferdowsi et al.
(2018) created an adaptive adversary to improve robustness.
Pinto et al.
ﬁrst used adversarial training to improve robustness of deep reinforcement learning algorithms, speciﬁcally TRPO.
Mandlekar et al.
(2017) demonstrated techniques to construct physically plausible perturbations adversarially to improve deep RL.
Gu et al.
(2018) then demonstrated how adversarial training could also be used to improve A3C.
Papernot et al.
ﬁrst proposed that distilling a neural network by training the same architecture with the probability vector outputs of the original network might act as a defense against adversarial examples.
While Carlini & Wagner (2016) found that defensive distillation is insufﬁcient to protect against adversarial examples for supervised learning, our algorithm uses distillation in combination with more effective defenses to protect reinforcement learning.
Chen et al.
(2017) use a pre-deﬁned rule-based teacher policy to assist a deep RL system.
They introduce a new DQN algorithm which predicts when to consult the teacher and how to learn from the teacher’s experiences.
In our setting there is no known external teacher.
Lin et al.
(2017a) use distillation to improve reinforcement learning itself by introducing the collaborative asynchronous advantage actor-critic algorithm (cA3C).
Their method allows for knowledge transfer between agents in cases where multiple agents are playing simultaneously in different environments with potentially different tasks.
To do this, they use a “deep alignment network” which learns to transfer the outputs from a teacher network to a student network.
Unlike our system, their system only improves agents learning from different tasks, and does not provide a method to defend the learned policy.
Shashua & Mannor (2017) train a DQN, that avoids the worst case outcome and performs well in the presence of a non-deterministic environments by using an extended Kalman-Filter.
However, their Kalman-update requires multiplication of matrices of size quadratic in the total number of weights and thus does not scale to neural networks that play Atari-Games with the screen input.
On the other hand, we optimize a policy to generalize to non-game effecting perturbations in a way which we show is scalable to very large neural networks.
We furthermore demonstrate how to do this in a way that increases provability.  B ACKGROUND The goal of reinforcement learning is to determine a behavior policy 2 π ( a | s ) for a given game G : State × Action → State × R , that maximizes the expected discounted sum of future rewards Q π ( s ) =
E [ (cid:80) ∞ t =1 γ t r t | s 1 = s, a 1 = a, π ] , where γ ∈ [0 , 1] is a discount factor and s i +1 , r i =
G ( s i , a i ) .
The objective of Q-Learning (Watkins & Dayan, 1992) is to learn the function Q ∗ ( s ) a = max π Q π ( s ) a and construct a policy by greedily picking its optimal action.
3. D EEP Q L EARNING
In Deep Q-Learning (Mnih et al., 2015; 2013) the goal is to progressively approximate Q ∗ by weights θ i for a neural network Q ( s ; θ i ) a at iteration i .
Samples are generated and stored in an experience replay buffer D i = { e 1 , . . .
, e i } where e i =
( s i , a i , r i , s i +1 ) is generated using a sufﬁciently random agent based on the approximation Q ( s ; θ i ) .
Integral to the DQN algorithm is the use of a network with lagging weights θ − i
= θ k (cid:98) ik (cid:99) for some k .
The loss at iteration i is described by
[Equation]
A few iterations of stochastic gradient descent are used to minimize L i and produce θ i +1 , although often multiple exploration steps are used ﬁrst.
3. A DVERSARIAL A TTACK
Unlike traditional reinforcement learning, the presence of adversarial attacks introduces the complication that the game has some level of knowledge about the agent playing the game and can act in speciﬁc ways knowing its behavior.
The environment that our algorithm is designed to address is given by our deﬁnition of an agent aware game .
Agent Aware Game
In agent aware reinforcement learning, the game G not only has access to the agent’s choices, but also has access to the learning system and the agent itself.
Thus, an adversarial agent’s aim is to play the game in such a way as to poison learning, but also potentially play differently when it is aware if the RL system is training or testing.
We consider only manipulative agent aware games, deﬁned to be games
[Equation]
which are composed of a core game H : State × Action → State × R and a manipulative adversary
M : State × Agent → State such that G ( s i , a i , π ) =
( M ( s − i +1 , π ) , r i ) where ( s − i +1 , r i ) =
H ( s i , a i ) , and such that for any π where s i +
=
M ( s − i +1 , π ) it is true that ∀ a.H ( s i +1 , a )
=
H ( s − i +1 , a ) .
In other words, manipulation does not effect how the core game works or is played.
The attacks we consider are L ∞ (cid:15) -attacks (not to be confused with the (cid:15) used for greedy exploration).
Speciﬁcally M ( s, π ) ∈ B (cid:15) ( s ) =
{ s (cid:48)
| || s − s (cid:48)
|| ∞ ≤ (cid:15) } , that is, B (cid:15) ( s ) is an (cid:15) sized L ∞ ball.
We consider both, test time attacks, and attacks that change their behavior when the agent is training.
3. T ESTING A TTACKS
Huang et al.
(2017) attack a network policy trained by reinforcement learning, at testing time and ﬁnd that traditional untargeted attacks such as FGSM are sufﬁcient to signiﬁcantly reduce the performance of the policy produced by DQN, A3C and TRPO.
Lin et al.
(2017b) used targeted attacks to “enchant” the agent into performing speciﬁc actions.
We evaluate only with untargeted FGSM attacks.
3. T RAINING A TTACKS
Attacking the training procedure such that the model never learns is known as poisoning Yang et al.
(2017); Steinhardt et al.
(2017); Biggio et al.
Behzadan & Munir (2017a) introduce an attack used to prevent the DQN from ever learning the correct Q function.
In their attack, they do not assume the attacker has direct access to θ Q and instead train a version of it, θ Q (cid:48) , in parallel.
They learn an “adversarial policy” which picks an action a (cid:48) far from the optimal one and then they perturb the observed next-state s (cid:48) to cause Q ( s (cid:48) ; θ − Q (cid:48) ) to be maximized (they use the non Double DQN variant) for the action a (cid:48) in hopes the attack transfers to the intended DQN.
We introduce an attack, Untargeted Q-Poisoning (UQP), which does not need to train additional networks, and which has access to the agent’s network.
We allow the attack the to switch its behavior when the agent is being tested in order to simulate the inevitable asymmetry between production and development environments and common subsequent dysfunction it is known to cause.
In UQP during training time, an attack state is chosen in the style of FGSM to reinforce the decision of the agent policy, thus often creating an illusion of highly successful training.
During test time, standard FGSM is used where the “classiﬁcation” is chosen by the policy.
We deﬁne it as a manipulative adversary: [Equation]
H
[Equation]
g, π ( s )))
π ( s ) is learning and g = arg max a π ( s ) a [Equation] otherwise and g = arg max a π ( s ) a where H is the cross entropy between the probability distribution π ( s ) , and the optimal action (encoded as one hot vector) from that probability distribution.
This attack is visualized in Figure 1.
Here the adversary is aware of whether the agent is training or being actively used, and is able to change the agent’s perception of the stock histories. When it is training, it might show the agent stock values with less noise than normal.
When the agent is then being actively used to trade, it would use the standard FGSM attack to convince it to buy when it should sell.
Intuitively, by using the negative of the gradient during training, the UQP attack increases the over-estimation of the value of an action, which Van Hasselt et al.
(2016) has determined to decrease learning performance. The negative of the gradient has the added beneﬁt of counteracting adversarial training.
While UQP attack is aware of whether the agent is testing or training, our results show that it is only necessary to use during training to derail the performance of a DQN, and thus is functionally a poisoning attack.
D ISTILLED A GENT DQN (D AD QN)
In contrast to deep supervised learning, deep reinforcement learning is more resource intensive task, and has less stable training dynamics, even for seemingly simple problems.
As such, it is important that any proposed defense mechanism is efﬁcient and usable in an online setting and induce as little perturbation as possible to standard network training algorithm.
Furthermore, while some progress has been made towards making deep RL experimentally robust, so far there has been less work on certiﬁable robustness.
Towards this, we propose a method to leverage the base Q learning algorithm’s ability to learn the correct Q function given a sufﬁciently random exploration agent (Even-Dar & Mansour, 2002; Bertsekas, 2008; Tsitsiklis, 1994; Watkins & Dayan, 1992).
Rusu et al.
ﬁrst described the method of improving the learned policy in Deep-Q Learning using Policy Distillation (PD).
In this technique a Q -approximation is ﬁrst learned by the standard
DQN algorithm.
New games are then played using Q as a greedy policy, and the states s are recorded.
A student network S is then trained on these states to mimic the behavior of Q .
Much like PD, our method involves distilling a policy network from Q .
Standard PD only uses the distilled policy during test time, and has so far only been tested in this way.
Here we introduce DadQN which trains the student policy S at the same time as it is learning the Q network and uses the student policy instead for exploration.
By decoupling the policy network from the Q network, we are able to train the policy with additional defensive constraints and use it for the exploration agent in addition to testing with it.
Given a loss L D and learning rates α Q , α S ∈ (0 , 1) our algorithm is as follows:
Algorithm
Initialize a state s , weights θ Q , θ S for i do do
Pick
Play the
Store if i mod k [-== 
-]=
then [Equation] end if
Pick a batch D U
Train the underlying Q:
[Equation]
Train the student S from
[Equation]
end for end for
Here Y ≡ Q ( s (cid:48) ; θ − Q ) argmax (cid:48) a Q ( s (cid:48) ; θ Q ) (cid:48) a is the Double DQN (Van Hasselt et al., 2016) next Q estimate.
The intuition behind this algorithm is that the loss for the Q function has not been changed and thus after being trained on sufﬁciently many random paths, the Q function will approach Q ∗ regardless of what the student learns.
Assuming the student will learn no matter what it is initialized with, the student should be able to handle the concept-shift of Q learning and incorporate the provided constraint.
Presumably, the additional constraints allow the student to achieve a better score when playing, so the student should explore higher reward paths.
While L D could potentially be any optimizable function, we evaluate with either one of the defensive loss described in Section 4.1, or an undefended mean squared error (MSE) loss L MSE .
While
Rusu et al.
(2015) observed that KL divergence works better than MSE and negative-log-likelihood experimentally, we found MSE to be sufﬁcient, simple and common enough for our evaluation.
Wang et al.
(2015) observed better learning performance by introducing a speciﬁc Q network architecture, the Dueling DQN, which is split into two components: an advantage network A ( s ; θ Q ) : R m computing the relative advantage of the m actions, and a value network V ( s ; θ Q ) : R .
These are combined as Q ( s ; θ Q ) =
V ( s ; θ Q ) + ( A ( s ; θ Q ) − 1 |A| (cid:80)
a ∈A A ( s ; θ Q ) a ) , where A denotes the action space.
The advantage and value are deﬁned to share early network layers.
We note that the output of A alone is sufﬁcient to replicate a greedy policy based on Q .
However, we observe that the value network may be able to discriminate between states earlier on in the learning than the advantage network.
We thus train both the advantage network and value network of the student from the Q independently with the same loss, deﬁning L MSE, Dueling for use as L D in Algorithm 1 as follows:
[Equation]
The Exploration Agent Originally, randomization in exploration was accomplished for the DQN using (cid:15) -greedy search (Watkins, 1989).
Here, the action arg max a S ( s ; θ S ) a is used with probability 1 − p and an action is picked uniformly with probability p at every time step (we use p instead of (cid:15) so to not overload (cid:15) used in attacks).
If p ≥ c for a constant c > 0 , then standard Q learning is known to converge.
p is decreased linearly with the number of frames played until it reaches a ﬁxed c .
While this method has good theoretical guarantees, Fortunato et al.
(2017) noticed improvements by letting the Q-network learn a mean and variance on noise for the weights in its dense layers.
We experiment with the same idea here, however only the student network S learns with noise.
We notice that because the student network lags behind the Q network, it might require more diverse samples even when Q has sufﬁciently learned the correct behavior.
We thus introduce an exploration noise constant η ≥ 1 , which we multiply with the learned weight variances at exploration time.
4. D EFENDING D AD QN
The primary motivation for decoupling the DQN network into a policy-student network S and a
Q -network is to allow one to leverage additional constraints on the student network without affecting the learning of the correct Q function (provided that exploration is sufﬁciently noisy).
We address the problem of adversarial robustness in reinforcement learning by improving S ’s robustness.
Speciﬁcally, we split L D into a defense α and a defense loss L O .
The defense α uses θ S and the sample s to produce a defense goal s d (which could be a concrete or symbolic) which is then passed to L O .
[Equation]
Adversarial Training A variety of techniques have been developed for increasing the robustness of neural networks, typically by training with adversarial examples (Tram`er et al., 2017; Shaham et al., 2015; Madry et al., 2018).
Rather than providing the Q network with adversarial examples as in Mandlekar et al.
, we provide them (in this case, as concrete defense goals) s d to the student:
[Equation]
Here, FGSM (cid:15) produces a concrete adversarial example s d in the (cid:15) sized L ∞ ball around s that makes the best action in s as least likely to be selected in s d as possible (that is, it is an untargeted attack).
Finally, we use the traditional MSE loss function:
[Equation]
In practice, we use this loss with probability p , the rest of the time using the non-attacked point s instead of s d .
For Dueling DQNs we calculate the advantage and value separately as in Equation 1.
Provable Robustness Training
In addition to adversarial training, we train and certify the robustness of our networks with the DiffAI framework introduced by Mirman et al.
(2018) using its Interval domain.
This framework has been shown capable of training networks on the scale of DQNs used in Atari with minimal speed and memory overheads over undefended SGD.
Additionally, it has been shown to decrease the accuracy of networks very little, which is essential to reinforcement learning where unstable dynamics can be an issue.
Formally, we ﬁrst create the abstract (symbolic) defense goal s d = B (cid:15) ( s ) .
We then use DiffAI to soundly propagate the defense goal s d through the network S (via symbolic computation), obtaining a symbolic element g f as a result.
Finally, DiffAI deﬁnes a differentiable loss L I : Interval × N → R which takes as input the ﬁnal element g f and a target (action in our case).
The loss has the property that for some target t if L I ( g f , t ) ≤ 1 then ∀ s (cid:48) ∈ B (cid:15) ( s ) .
arg max a S ( s (cid:48) ; θ S ) a = t .
That is, in this case, we have proved that any element inside the (cid:15) sized L ∞ ball around s will be classiﬁed to the action t by the student network.
We deﬁne our defensive loss as a combination of this loss and the adversarial loss described earlier for a concrete α :
[Equation]
where λ ≥ 0 is the constant with which we want to prioritize the DiffAI loss.
E XPERIMENTAL E VALUATION
We now present our experimental results comparing DadQN to existing methods.
We note that while it may be beneﬁcial to intentionally train one’s own DQN by simulating an adversarially attacking manipulator (in an attempt to defend it), there are learning time attacks which prevent this from being effective.
Thus, central to our analysis is the notion that the agent does not know whether it will be attacked during training and with what attack, or whether it will be attacked during testing and with what attack.
We show that in each situation, DadQN is capable of providing a stronger or equally strong a defense as existing work, and virtually always avoids the worst case failures.
We also show the ﬁrst improvement in provable robustness in DQNs by using DiffAI as a training defense.
5. E XPERIMENTAL S ETUP
We tested with 3 Atari games (Bellemare et al., 2013) from the OpenAI Gym (Brockman et al., 2016):
RoadRunner (RR), Pong and Boxing.
Every 10 episodes we play a validation game.
In a these games [Table] [Table] we verify the decision made by the network using the Box domain.
We binary-search for the largest (cid:15) such that the decision is safe.
Validation games alternate between never attacking, or attacking with some probability every timestep.
Each experiment was run for 4 million frames.
To compare DadQN with DQNs, we implemented a variety of extensions known to increase training performance when enabled in concert as in RainbowDQN (Hessel et al., 2017): Priority Replay (Schaul et al., 2015), DoubleDQN (Van Hasselt et al., 2016), DuelingDQN (Wang et al., 2015), and
NoisyNet (Fortunato et al., 2017) using noise constant η = 2 as described in Section 4.
During validation episodes we disable noise due to NoisyNet and use (cid:15) -greedy exploration with (cid:15) = 0 .
Due to limited resources, we choose parameters for the three games that yielded faster training than those reported by RainbowDQN 3 .
All parameters can be found in Tables S1 and S2 in the Appendix.
We implemented both DadQN and DQN in PyTorch (Paszke et al., 2017) asynchronously (Mnih et al., 2016).
Depending on the parameters and the hardware one run took between 4 to 30 hours, and the majority took about 20 hours.
We evaluated using Nvidia 1080Tis and Nvidia k80s, with otherwise modern hardware.
On our fastest machine, using a Nvidia 1080Ti, our implementation of
DQN plays 266.9 frames per second and the DadQN algorithm 266.0 frames.
Attacks & Defenses We consider both untargeted FGSM, as well as our UQP attack.
For all uses of FGSM we use (cid:15) = 0 . 004 .
During validation games we either do not use any attacks whatsoever as in rows preceeded by “none”, or we use FGSM with probability 0 .
4 or 1 . 0 , written as FGSM p =0 .or
FGSM p =1 . When the UQP attack is used during validation games, it is functionally equivalent to the FGSM attack, and thus for simplicity we only write FGSM for testing attacks. For standard
Q-Networks the semantics of attacking and defending with FGSM are similar but not quite the same: when attacked ( TrAtk ), the environment produces adversarial examples which are then stored in the replay buffer and are seen many times in training.
When used as a defense ( Def ), the perturbation is applied to a training example when it is sampled from the replay buffer, thus each perturbation will only be seen once.
We apply our attacks and defenses to the 4-stacks of consecutive frames used for training DQNs on Atari.
While single frame attacks conceptually ﬁt the setting, we are primarily evaluating the effectivity of defenses and thus choose pick the most efﬁcient attck – which is attacking 4-stacks (as used in Behzadan & Munir (2017b)).
5. P OISONING AS AN A TTACK
Table 1 shows the average ﬁnal episode score (weighted by number of frames) from the 15 consecutive best (by sum score) validation episodes during training. See Appedix B for the details of this weighting. [Table]
[Figure] [Table] The leftmost DQN column ﬁrst shows these scores without any attack on DQNs for reference.
When attacking DQN during training we observe signiﬁcant drop in score for unattacked validation games.
We can see from this table that UQP is a strong attack against DQN’s as the presence of the attack (i) impacts undefended training performance and (ii) signiﬁcantly impacts the performance of the agent in unattacked and attacked games during testing.
To exemplify (i), Figure 2 shows the training reward for Boxing in the presence of an attack.
5. U TILITY OF DEFENDING WITH D AD QN
Table 2 shows the average ﬁnal episode score (weighted by number of frames) using the network weights from the best validation game during training over 15 validation episodes.
We can see that attacking a DQN which has never encountered adversarial examples before ( none column under DQN) drastically lowers its score.
When attacked by the environment ( TrAtk ), whether intentionally or not, the agent learns to play the game, although not as well as without an attack or the DadQN agent that has been defended, or DadQN being defended and attacked.
We also see that a DQN using the previously deﬁned FGSM training defense ( Def ) - perturbations happen after sampling from the replay buffer) does not learn how to play the game whatsoever.
We hypothesize that the attack variant places the adversarial examples in the replay buffer making it easier for the agent to learn since it then encounters the same perturbations more frequently.
DadQN trained with the FGSM defense performs nearly as well as baseline DQN without attacks, and much better than baseline with attacks.
In addition to increasing test time attack robustness, we can see in Table 1 that DadQN trained without an explicit defense is robust to poisoning.
Rewards for DadQN were nearly always greater than DQN whether playing ﬁnally in an attacked or unattacked game.
Using the FGSM Defense ampliﬁed the effect to recover nearly the full performance of an unattacked DQN.
5. T OWARDS P RACTICAL P ROVABLE R OBUSTNESS OF DQN S WITH D AD QN
For DiffAI we explored a slightly different domain for each game.
For Roadrunner we use λ =
.with a non-defensive α ( s, θ S )
=
s and L O, MSE .
Pong and Boxing instead use λ = 0 .
01 and (cid:15) = 0 .
with α 0 . 004 .
For boxing, we use an attack probability of 0 . 4 .
Knowing that DiffAI can decrease classiﬁcation accuracy, some loss in score was expected.
However, the last column of
Table 1 demonstrates that DiffAI used in this was still a powerful defense against test the test time
FGSM attack the UQP attack compared to otherwise undefended DQNs.
Table 3 demonstrates the average maximum (cid:15) which could be proven robust using DiffAI over the best (by aggregate score) 15 consecutive validation games.
From this table, we can see that the best (cid:15) that DiffAI was able to prove without using any provability training was a tenth the size of the largest which could be found in DiffAI defended networks.
Without any defense, the baseline DQN is not robust, and thus has a provability (cid:15) of less than a ten-thousandth the minimally normally expressible perturbation in 8-bit images (this would be (cid:15) ∼ 0 . 004 ).
DiffAI on the other hand comes signiﬁcantly closer than any system we are aware of, at nearly 25% this value
A H YPER -P ARAMETERS
[Table]
[Table]
B W EIGHING OF S CORES & A DDITIONAL E XPERIMENTAL D ATA
Some games, such as roadrunner, have multiple levels which is a non trivial problem for DQNs.
As such, when the agents achieved a certain score, it then proceeded to play episodes it wasnt as adept at.
These episodes achieved a low score and thus decrease the aggregates (such as mean or sum) over multiple episodes.
It would be unfair to compare directly against an agent which had in fact never reached the second level. Thus we introduce a weighing scheme to combat this [Table] The values in Table 1 are calculated by max
[Equation]
episode score[ i ] · episode frames[ i ] · A [ i ]
[Equation]
where E is the number of validation episodes played, episode score[ i ] is the score earned in validation episode i , episode frames[ i ] is the number of frames played in validation episode i and A [ i ] is an indicator whether the validation game was played with or without attacks.
Table 2 uses the same weighing but over 15 new games, played from the best set of network weights.
To illustrate the importance of this weighing and show that it does not impact the qualitative interpretation of the results we provide unweighted scores for Table 1 in Tables S3 and S4.
Table S shows
[Equation]
where e (cid:48) is the validation episode that maximized Equation B for the corresponding value.
Table S on the other hand shows
[Table]
[Table]
episode score[ i ] · A [ i ] (cid:80)
[Table]
=
[Table]
A [ i ] .
For Table S4 observe the same trends as for Table 1.
However in Table S3 we see that the scores for
RoadRunner especially are a lot lower for DadQN.
The reason for this is that the agent reaches the second level, in which it does poorly; thus decreasing the score.
C UQP AGAINST A CTOR -C RITIC
Here we demonstrate that UQP is an affective attack against two actor-critic reinforcement learning algorithms for neural networks, A3C introduced by Mnih et al.
(2016) (implemented synchronously) and PPO introduced by Schulman et al.
We implement our attack on top of the framework provided by Kostrikov (2018), using the learning parameters as suggested for each algorithm.
In order to demonstrate the weakness of these algorithms to this attack, we only test with the attack applied to the last image in the stack rather than all four images in the stack.
In these attacks, unless otherwise speciﬁed we use an attack (cid:15) of 3 out of the possible range of [0 , 255] .
For training, we either use no attack (shown in red), or the UQP attack with 4 iterations with step size 2 projecting back into the L ∞ ball after every iteration as in PGD (shown in blue the ﬁgures).
Figures S1, S3, S5 and S3 show the training scores sampled frequently, with the number of training steps in the x-axis and the score on the y-axis.
The solid lines correspond to the average score from the last 10 episodes (unweighted), whereas the same-colored regions surrounding them show the max and min score of a game from the last 10 episodes.
For testing, we either use no attack, or the PGD attack with 16 iterations with step size 2 projecting back into the L ∞ ball after every iteration as in PGD.
In Figures S2, S4, S6, and S4 show the testing scores sampled signiﬁcantly less frequently, at the number of training steps in the x-axis and the score on the y-axis.
Each point correspond to the average score from 10 testing episodes (unweighted) at that time-step.
From these results it can clearly be seen that UQP is able to both prevent both actor-critic methods from learning to play the training set, and is able to keep the agent from being as robust when it is tested against a training set.
1M 2M 3M 4M 10k 20k 30k 40k 50k
[Figure]
[Figure]
2M 3M 4M 10k 20k 30k 40k 50k
[Figure]
[Figure]
[Figure]
2M 3M 4M 10k 20k 30k 40k 50k
[Figure]
[Figure]
[Figure]
2M 3M 4M
[Figure]
[Figure]
[Figure]
[Figure]