P OLICY S MOOTHING FOR P ROVABLY R OBUST R EINFORCEMENT L EARNING
A BSTRACT
The study of provable adversarial robustness for deep neural networks (DNNs) has mainly focused on static supervised learning tasks such as image classiﬁcation.
However, DNNs have been used extensively in real-world adaptive tasks such as reinforcement learning (RL), making such systems vulnerable to adversarial attacks as well.
Prior works in provable robustness in RL seek to certify the behaviour of the victim policy at every time-step against a non-adaptive adversary using methods developed for the static setting.
But in the real world, an
RL adversary can infer the defense strategy used by the victim agent by observing the states, actions, etc.
from previous time-steps and adapt itself to produce stronger attacks in future steps (e.g., by focusing more on states critical to the agent’s performance).
We present an efﬁcient procedure, designed speciﬁcally to defend against an adaptive RL adversary, that can directly certify the total reward without requiring the policy to be robust at each time-step.
Focusing on randomized smoothing based defenses, our main theoretical contribution is to prove an adaptive version of the Neyman-Pearson Lemma – a key lemma for smoothingbased certiﬁcates – where the adversarial perturbation at a particular time can be a stochastic function of current and previous observations and states as well as previous actions.
Building on this result, we propose policy smoothing where the agent adds a Gaussian noise to its observation at each time-step before passing it through the policy function.
Our robustness certiﬁcates guarantee that the ﬁnal total reward obtained by policy smoothing remains above a certain threshold, even though the actions at intermediate time-steps may change under the attack.
We show that our certiﬁcates are tight by constructing a worst-case scenario that achieves the bounds derived in our analysis.
Our experiments on various environments like Cartpole, Pong, Freeway and Mountain Car show that our method can yield meaningful robustness guarantees in practice.
I NTRODUCTION
Deep neural networks (DNNs) have been widely employed for reinforcement learning (RL) problems as they enable the learning of policies directly from raw sensory inputs, like images, with minimal intervention from humans.
From achieving super-human level performance in video-games (Mnih et al., 2013; Schulman et al., 2015; Mnih et al., 2016), Chess (Silver et al., 2017) and Go (Silver et al., 2016) to carrying out complex real-world tasks, such as controlling a robot (Levine et al., 2016) and driving a vehicle (Bojarski et al., 2016), deep-learning based algorithms have not only established the state of the art, but also become more effortless to train.
However, DNNs have been shown to be susceptible to tiny malicious perturbations of the input designed to completely alter their predictions (Szegedy et al., 2014; Madry et al., 2018; Goodfellow et al., 2015).
In the RL setting, an attacker may either directly corrupt the observations of an RL agent (Huang et al., 2017;
Behzadan & Munir, 2017; Pattanaik et al., 2018) or act adversarially in the environment (Gleave et al., 2020) to signiﬁcantly degrade the performance of the victim agent.
Most of the adversarial defense literature has focused mainly on classiﬁcation tasks (Kurakin et al., 2017; Buckman et al., 2018;
Guo et al., 2018; Dhillon et al., 2018; Li & Li, 2017; Grosse et al., 2017; Gong et al., 2017).
In this paper, we study a defense procedure for RL problems that is provably robust against normbounded adversarial perturbations of the observations of the victim agent.
Problem setup.
A reinforcement learning task is commonly described as a game between an agent and an environment characterized by the Markov Decision Process (MDP) M
= ( S, A, T, R, γ ) , where S is a set of states, A is a set of actions, T is the transition probability function, R is the onestep reward function and γ ∈ [0 , 1] is the discount factor.
However, as described in Section 3, our analysis applies to an even more general setting than MDPs.
At each time-step t , the agent makes an observation o t
= o
( s t ) ∈ R d which is a probabilistic function of the current state of the environment, picks an action a t ∈ A and receives an immediate reward R t
= R ( s t , a t ) .
We deﬁne an adversary as an entity that can corrupt the agent’s observations of the environment by augmenting them with a perturbation (cid:15) t at each time-step t which can depend on the states, actions, observations, etc., generated so far.
We use (cid:15) =
( (cid:15) 1 , (cid:15) 2 , . . . )
to denote the entire sequence of adversarial perturbations.
The goal of the adversary is to minimize the total reward obtained by the agent policy π while keeping the overall (cid:96) 2 -norm of the perturbation within a budget B .
Formally, the adversary seeks to optimize the following objective:
[Equation]
Note that the size of the perturbation (cid:15) t in each time-step t need not be the same and the adversary may choose to distribute the budget B over different time-steps in a way that allows it to produce a stronger attack.
Also, our formulation accounts for cases when the agent may only partially observe the state of the environment, making M a Partially Observable Markov Decision Process (POMDP).
Objective.
Our goal in provably robust RL is to design a policy π such that the total reward in the presence of a norm-bounded adversary is guaranteed to remain above a certain threshold, i.e.,
[Equation]
In other words, no norm-bounded adversary can lower the expected total reward of the policy π below a certain threshold.
In our discussion, we restrict out focus to ﬁnite-step games that end after t time-steps.
This is a reasonable approximation for inﬁnite games with γ < 1 , as for a sufﬁciently large t , γ t becomes negligibly small.
For games where γ = 1 , R t must become sufﬁciently small after a ﬁnite number of steps to keep the total reward ﬁnite.
Step-wise vs. episodic certiﬁcates .
Previous works on robust RL have sought to certify the behaviour of the policy function at each time-step of an episode, e.g., the output of a Deep Q-Network (L¨utjens et al., 2019) and the action taken for a given state (Zhang et al., 2020).
Ensuring that the behaviour of the policy remains unchanged in each step can also certify that the ﬁnal total reward remains the same under attack.
However, if the per-step guarantee fails at even one of the intermediate steps, the certiﬁcate on the total reward becomes vacuous or impractical to compute (as noted in Appendix E of Zhang et al.
Our approach gets around this issue by directly certifying the ﬁnal total reward for the entire episode without requiring the policy to be provably robust at each intermediate step.
Also, the threat-model we consider is more general as we allow the adversary to choose the size of the perturbation for each time-step.
Thus, our method can defend against more sophisticated attacks that focus more on states that are crucial for the victim agent’s performance.
Technical contributions.
In this paper, we study a defense procedure based on “randomized smoothing” (Cohen et al., 2019;
L´ecuyer et al., 2019; Li et al., 2019; Salman et al., 2019) since at least in “static” settings, its robustness guarantee scales up to high-dimensional problems and does not need to make stringent assumptions about the model.
We ask: can we utilize the beneﬁts of randomized smoothing to make a general high-dimensional RL policy provably robust against adversarial attacks?
The answer to this question turns out to be non-trivial as the adaptive nature of the adversary in the RL setting makes it difﬁcult to apply certiﬁcates from the static setting.
For example, the (cid:96) 2 -certiﬁcate by Cohen et al.
(2019) critically relies on the clean and adversarial distributions [Figure] being isometric Gaussians (Figure 1-a).
However, in the RL setting, the adversarial perturbation in one step might depend on states, actions, observations, etc., of the previous steps, which could in turn depend on the random Gaussian noise samples added to the observations in these steps.
Thus, the resulting adversarial distribution need not be isometric as in the static setting (Figure 1-(b-d)).
For more details on this example, see Appendix B.
Our main theoretical contribution is to prove an adaptive version of the Neyman-Pearson lemma (Neyman & Pearson, 1992) to produce robustness guarantees for RL.
We emphasize that this is not a straightforward extension (refer to Appendix D, E and F for the entire proof).
To prove this fundamental result, we ﬁrst eliminate the effect of randomization in the adversary (Lemma 1) by converting a general adversary to one where the perturbation at each time-step is a deterministic function of the previous states, actions, observations, etc., and showing that the modiﬁed adversary is as strong as the general one.
Then, we prove the adaptive Neyman-Pearson lemma where we show that, in the worst-case, the deterministic adversary can be converted to one that uses up the entire budget B in the ﬁrst coordinate of the perturbation in the ﬁrst time-step (Lemma 3).
Finally, we derive the robustness guarantee under an isometric Gaussian smoothing distribution (Theorem 1).
In section A, we establish the tightness of our certiﬁcates by constructing the worst-case environmentpolicy pair which attains our derived bounds.
More formally, out of all the environment-policy pairs that achieve a certain total reward with probability p , we show a worst-case environment-policy pair and a corresponding adversary such that the probability of achieving the same reward under the presence of the adversary is minimum.
A discussion on the Neyman-Pearson lemma in the context of randomized smoothing is available in Appendix C.
Building on these theoretical results, we propose Policy Smoothing , a simple model-agnostic randomized-smoothing based technique that can provide certiﬁed robustness without increasing the computational complexity of the agent’s policy.
Our main contribution is to show that by augmenting the policy’s input by a random smoothing noise, we can achieve provable robustness guarantees on the total reward under a norm-bounded adversarial attack (Section 4.2).
Policy Smoothing does not need to make assumptions about the agent’s policy function and is also oblivious to the workings of RL environment.
Thus, this method can be applied to any RL setting without having to make restrictive assumptions on the environment or the agent.
In section 3, we model the entire adversarial RL process under Policy Smoothing as a sequence of interactions between a system A , which encapsulates the RL environment and the agent, and a system B , which captures the addition of the adversarial perturbation and the smoothing noise to the observations.
Our theoretical results do not require these systems to be Markovian and can thus have potential applications in real-time decision-making processes that do not necessarily satisfy the Markov property.
Empirical Results.
We use four standard Reinforcement Learning benchmark tasks to evaluate the effectiveness of our defense and the signiﬁcance of our theoretical results: the Atari games ‘Pong’ and ‘Freeway’ (Mnih et al., 2013) and the classical ‘Cartpole’ and ‘Mountain Car’ control environments (Barto et al., 1983; Moore, 1990) – see Figure 3.
We ﬁnd that our method provides highly nontrivial certiﬁcates.
In particular, on at least two of the tasks, ‘Pong’ and ‘cartpole’, the provable lower bounds on the average performances of the defended agents, against any adversary, exceed the observed average performances of undefended agents under a practical attack.
P RIOR W ORK
Adversarial RL.
Adversarial attacks on RL systems have been extensively studied in recent years.
DNN-based policies have been attacked by either directly corrupting their inputs (Huang et al., 2017; Behzadan & Munir, 2017; Pattanaik et al., 2018) or by making adversarial changes in the environment (Gleave et al., 2020).
Empirical defenses based on adversarial training, whereby the dynamics of the RL system is augmented with adversarial noise, have produced good results in practice (Kamalaruban et al., 2020; Vinitsky et al., 2020).
Zhang et al.
propose training policies together with a learned adversary in an online alternating fashion to achieve robustness to perturbations of the agent’s observations.
Robust RL.
Prior work by L¨utjens et al.
(2019) has proposed a ‘certiﬁed’ defense against adversarial attacks to observations in deep reinforcement learning, particularly for Deep Q-Network agents.
However, that work essentially only guarantees the stability of the network approximated Q-value at each time-step of an episode.
By contrast, our method provides a bound on the expected true reward of the agent under any norm-bounded adversarial attack.
Zhang et al.
(2020) certify that the action in each time-step remains unchanged under an adversarial perturbation of ﬁxed budget for every time-step.
This can guarantee that the ﬁnal total reward obtained by the robust policy remains the same under attack.
However, this approach would not be able to yield any robustness certiﬁcate if even one of the intermediate actions changed under attack.
Our approach gets around this difﬁculty by directly certifying the total reward, letting some of the intermediate actions of the robust policy to potentially change under attack.
For instance, consider an RL agent playing Atari Pong.
The actions taken by the agent when the ball is close to and approaching the paddle are signiﬁcantly more important than the ones when the ball is far away or retreating from the paddle.
By allowing some of the intermediate actions to potentially change, our approach can certify for larger adversarial budgets and provide a more ﬁne-grained control over the desired total-reward threshold.
Moreover, we study a more general threat model where the adversary may allocate different attack budgets for each time-step focusing more on the steps that are crucial for the agent’s performance, e.g., attacking a Pong agent when the ball is close to the paddle.
Provable Robustness in Static Settings: Notable provable robustness methods in static settings are based on interval-bound propagation (Gowal et al., 2018;
Huang et al., 2019; Dvijotham et al., 2018;
Mirman et al., 2018), curvature bounds (Wong & Kolter, 2018; Raghunathan et al., 2018;
Chiang et al., 2020; Singla & Feizi, 2019; 2020; 2021), randomized smoothing (Cohen et al., 2019; L ´ ecuyer et al., 2019; Li et al., 2019;
Salman et al., 2019; Levine & Feizi, 2021), etc.
Certiﬁed robustness has also been extended to problems with structured outputs such as images and sets (Kumar & Goldstein, 2021).
Focusing on Gaussian smoothing, Cohen et al.
(2019) showed that if a classiﬁer outputs a class with some probability under an isometric Gaussian noise around an input point, then it will output that class with high probability at any perturbation of the input within a particular (cid:96) 2 distance.
Kumar et al.
(2020) showed how to certify the expectation of softmax scores of a neural network under Gaussian smoothing by using distributional information about the scores.
P RELIMINARIES AND N OTATIONS
We model the ﬁnite-step adversarial RL framework as a t -round communication between two systems A and B (Figure 2).
System A represents the RL game.
It contains the environment M and the agent, and when run independently, simulates the interactions between the two for some given policy π .
At each time-step i , it generates a token τ i from some set T , which is a tuple of the current state s i and its observation o i , the action a
i − 1 in the previous step (and potentially some other objects that we ignore in this discussion), i.e., τ i =
( s i , a i − 1 , o i , . . . )
∈ S ×
A × R d × . . .
= T .
For the ﬁrst step, replace the action in τ 1 with some dummy element ∗ from the action space A .
System B comprises of the adversary and the smoothing distribution which generate an adversarial perturbation (cid:15) i and a smoothing noise vector δ i , respectively, at each time-step i , the sum of which is denoted by an offset η i = (cid:15)
i + δ i ∈ R d .
When both systems are run together in an interactive fashion, in each round i , system A generates τ i as a probabilistic function of τ 1 , η 1 , τ 2 , η 2 , . . .
, τ i − 1 , η i − 1 , i.e., τ i : ( T × R d ) i − 1 → ∆( T ) .
τ 1 is sampled from a ﬁxed distribution.
It passes τ i to B , which generates (cid:15) i as a probabilistic function of { τ j , η j } i − 1 j =
and τ i , i.e., (cid:15) i : ( T × R d ) i − 1 × T → ∆( R d ) and adds a noise vector δ i sampled independently from the smoothing distribution to obtain η i .
It then passes η i to A for the next round.
After running for t steps, a deterministic or random 0/1-function h is computed over all the tokens and offsets generated.
We are interested in bounding the probability with which h outputs 1 as a function of the adversarial budget B .
In the RL setting, h could be a function indicating whether the total reward is above a certain threshold or not.
P ROVABLY R OBUST RL 4. A DAPTIVE N EYMAN -P EARSON L EMMA
Let X be the random variable representing the tuple z = ( τ 1 , η 1 , τ 2 , η 2 , . . .
t , η t ) ∈ ( T × R d ) t when there is no adversary, i.e., (cid:15) i = 0 and η i =
δ i is sampled directly form the smoothing distribution P .
Let Y be the random variable representing the same tuple in the presence of a general adversary (cid:15) satisfying (cid:107) (cid:15) (cid:107) 2 ≤ B .
Thus, if h ( X ) = 1 with some probability p , we are interested in deriving a lower-bound on the probability of h ( Y )
= 1 as a function of p and B .
Let us now deﬁne a deterministic adversary (cid:15) dt for which the adversarial perturbation at each step is a deterministic function of the tokens and offsets of the previous steps and the token generated in the current step.
i.e., (cid:15) dti : ( T × R d ) i − 1 × T → R d .
Let Y dt be its corresponding random variable.
Then, we have the following lemma that converts a probabilistic adversary into a deterministic one.
Lemma 1 ( Reduction to Deterministic Adversaries).
For any general adversary (cid:15) and an Γ ⊆ ( T × R d ) t , there exists a deterministic adversary (cid:15) dt such that,
[Equation]
where Y dt is the random variable for the distribution deﬁned by the adversary (cid:15) dt .
This lemma says that for any adversary (deterministic or random) and a subset Γ of the space of z , there exists a deterministic adversary which assigns a lower probability to Γ than the general adversary.
In the RL setting, this means that the probability with which a smoothed policy achieves a certain reward value under a general adversary is lower-bounded by the probability of the same under a deterministic adversary.
The intuition behind this lemma is that out of all the possible values that the internal randomness of the adversary may assume, there exists a sequence of values that assigns the minimum probability to Γ (over the randomness of the environment, policy, smoothing noise, etc.).
We defer the proof to the appendix.
Next, we formulate an adaptive version of the Neyman-Pearson lemma for the case when the smoothing distribution P is an isometric Gaussian N (0 , σ 2 I ) .
If we applied the classical Neyman-Pearson lemma on the distributions of X and Y dt , it will give us a characterization of the worst-case 0/ function among the class of functions that achieve a certain probability p of being 1 under the distribution of X that has the minimum probability of being 1 under Y dt .
Let µ X and µ Y dt be the probability density function of X and Y dt , respectively.
Lemma 2 ( Neyman-Pearson Lemma, 1933).
If Γ Y dt = { z ∈ ( T × R d ) t | µ Y dt ( z ) ≤ qµ X ( z ) } for some q ≥ 0 and P [ h ( X ) = 1] ≥ P [ X ∈ Γ Y dt ] , then P [ h ( Y dt ) = 1] ≥ P [ Y dt ∈ Γ Y dt ] .
For an arbitrary element h in the class of functions H p
= { h | P [ h ( X ) = 1] ≥ p } , construct the set Γ Y dt for an appropriate value of q for which P [ X ∈ Γ Y dt ] = p .
Now, consider a function h (cid:48) which is 1 if its input comes from Γ Y dt and 0 otherwise.
Then, the above lemma says that the function h (cid:48) has the minimum probability of being 1 under Y dt , i.e.,
[Equation]
This gives us the worst-case function that achieves the minimum probability under an adversarial distribution.
However, in the adaptive setting, Γ Y dt could be a very complicated set and obtaining an expression for P [ Y dt ∈ Γ Y dt ] might be difﬁcult.
To simplify our analysis, we construct a structured deterministic adversary (cid:15) st which exhausts its entire budget in the ﬁrst coordinate of the ﬁrst perturbation vector, i.e., (cid:15)
st 1 = ( B, 0 , . .
. , 0) and (cid:15) sti =
(0 , 0 , . . . , 0) for i > 1 .
Let Y st be the corresponding random variable and µ Y st its density function.
We formulate the following adaptive version of the Neyman-Pearson lemma:
Lemma 3 ( Adaptive Neyman-Pearson Lemma).
If Γ Y st = { z ∈ ( T × R d ) t | µ Y st ( z ) ≤ qµ X ( z ) } for some q ≥ 0 and P [ h ( X ) = 1] ≥ P [ X ∈ Γ Y st ] , then P [ h ( Y dt ) = 1] ≥ P [ Y st ∈
Γ
Y st ] .
The key difference from the classical version is that the worst-case set we construct in this lemma is for the structured adversary and the ﬁnal inequality relates the probability of h outputting 1 under the adaptive adversary to the probability that the structured adversary assigns to the worst-case set.
It says that for the appropriate value of q for which P
[ X ∈ Γ Y st ] = p , any function h ∈ H p outputs 1 with at least the probability that Y st assigns to Γ Y st .
It shows that over all possible functions in
H p and over all possible adversaries (cid:15) , the indicator function 1 z ∈ Γ Yst and the structured adversary capture the worst-case scenario where probability of h being 1 under the adversarial distribution is the minimum.
Since both Y st and X are just isometric Gaussian distribution with the same variance σ 2 centered at different points on the ﬁrst coordinate of η 1 , the set Γ Y st is the set of all tuples z for which { η 1 } 1 is below a certain threshold.
1 We use lemmas 1 and 3 to derive the ﬁnal bound on the probability of h ( Y )
= 1 in the following theorem, the proof of which is deferred to the appendix.
Theorem 1 ( Robustness Guarantee).
For an isometric Gaussian smoothing noise with variance σ 2 , if P [ h ( X ) = 1] ≥ p , then:
[Equation]
where Φ is the standard normal CDF.
The above analysis can be adapted to obtain an upper-bound on P [ h ( Y )
= 1] of Φ(Φ − 1 ( p ) + B/σ ) .
4. P OLICY S MOOTHING
Building on these results, we develop policy smoothing , a simple model-agnostic randomizedsmoothing based technique that can provide certiﬁed robustness without increasing the computational complexity of the agent’s policy.
Given a policy π , we deﬁne a smoothed policy ¯ π as:
[Equation] where δ t ∼ N (0 , σ 2 I ) .
Our goal is to certify the expected sum of the rewards collected over multiple time-steps under policy
¯ π .
We modify the technique developed by Kumar et al.
(2020) to certify the expected class scores of a neural network by using the empirical cumulative distribution function (CDF) of the scores under the smoothing distribution to work for the RL setting.
This approach utilizes the fact that the expected value of a random variable X representing a class score under a Gaussian N (0 , σ 2 I ) smoothing noise can be expressed using its CDF F ( . )
as below:
[Equation]
[Figure] Given m samples { x i } mi =1 of the random variable X , let us deﬁne its empirical CDF at a point x ,
F m ( x ) =
|{ x i | x i ≤ x }| /m , as the fraction of samples that are less than or equal to x .
Using
F m ( x ) , the Dvoretzky–Kiefer–Wolfowitz inequality can produce high-conﬁdence bounds on the true CDF of X .
It says that with probability 1 − α , for α ∈ (0 , 1] , the true CDF F ( x ) is in the range [ F ( x ) , F ( x )] , where F ( x ) =
F m ( x ) − (cid:112) ln(2 /α ) / 2 m and F ( x ) =
F m ( x ) + (cid:112) ln(2 /α ) / 2 m .
For an adversarial perturbation of (cid:96) 2 -size B , the result of Cohen et al.
(2019) bounds the CDF within [Φ(Φ − 1 ( F ( x )) − B/σ ) , Φ(Φ − 1 ( F ( x )) + B/σ )] , which in turn bounds E
[ X ] using equation (2).
In the RL setting, we can model the total reward as a random variable and obtain its empirical CDF by playing the game using policy ¯ π .
As above, we can bound the CDF F ( x ) of the total reward in a range [ F ( x ) , F ( x )] using the empirical CDF.
Applying Theorem 1, we can bound the CDF within [Φ(Φ − 1 ( F ( x )) − B/σ ) , Φ(Φ − 1 ( F ( x ))
+ B/σ )] for an (cid:96) 2 adversary of size B .
The function h in
Theorem 1 could represent the CDF F ( x ) by indicating whether the total reward computed for an input
z ∈
( T × R d ) t is below a value x .
Finally, equation (2) puts bounds on the expected total reward under an adversarial attack.
E XPERIMENTS 5. E NVIRONMENTS AND S ETUP
We tested on four standard environments: the classical cortrol problems ‘Cartpole’ and ‘Mountain
Car’ and the Atari games ‘Pong’ and ‘Freeway.’
We consider three tasks which use a discrete action space (‘Cartpole’ and the two Atari games) as well as one task that uses a continuous action space (‘Mountain Car’).
For the discrete action space tasks, we use a standard Deep Q-Network (DQN) (Mnih et al., 2013) model, while for ‘Mountain Car’, we use Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016).
As is common in DQN and DDPG, our agents choose actions based on multiple frames of observations.
In order to apply a realistic threat model, we assume that the adversary acts on each frame only once when it is ﬁrst observed.
The adversarial distortion is then maintained when the same frame is used in future time-steps.
In other words, we consider the observation at time step o t (discussed in
Section 3) to be only the new observation at time t : this means that the adversarial/noise perturbation η t , as a ﬁxed vector, continues to be used to select the next action for several subsequent time-steps.
This is a realistic model because we are assuming that the adversary can affect the agent’s observation of states, not necessarily the agent’s memory of previous observations.
As in other works on smoothing-based defenses (e.g., Cohen et al.
, we add noise during training as well as at test time.
We use DQN and DDPG implementations from the popular stable-baselines3 package (Rafﬁn et al., 2019): hyperparameters are provided in the appendix.
In experiments, we report and certify for the total non-discounted ( γ = 1) reward.
In ‘Cartpole’, the observation vector consists of four kinematic features.
We use a simple MLP model for the Q-network, and tested two variations: one in which the agent uses ﬁve frames of observation, and one in which the agent uses only a single frame (shown in the appendix).
In order to show the effectiveness of our technique on tasks involving high-dimensional state observations, we chose two tasks (‘Pong’ and ‘Freeway’) from the Atari environment, where state observations are image frames, observed as 84 × 84 pixel greyscale images.
For ‘Pong’, we test on a “one-round” variant of the original environment.
In our variant, the game ends after one player, either the agent or the opponent, scores a goal: the reward is then either zero or one.
Note that this is not a one-timestep episode: it takes typically on the order of 100 timesteps for this to occur.
Results for a full Pong game are presented in the appendix: as explained there, we ﬁnd that the certiﬁcates unfortunately do not scale with the length of the game.
For the ‘Freeway’ game, we play on ‘Hard’ mode and end the game after 250 timesteps.
In order to test on an environment with a continuous action space, we chose the ‘Mountain Car’ environment.
Note that previous certiﬁcation results for reinforcement learning, which certify actions at individual states rather than certifying the overall reward (Zhang et al., 2020) cannot be applied to continuous action state problems.
In this environment, the observation vector consists of two kinematic features (position and velocity), and the action is one continuous scalar (acceleration).
As in ‘Cartpole’, we use ﬁve observation frames and a simple MLP policy.
We use a slight variant of the original environment: we do not penalize for fuel cost so the reward is a boolean representing whether or not the car reaches the destination in the time allotted (999 steps).
5. R ESULTS
Certiﬁed lower bounds on the expected total reward, as a function of the total perturbation budget, are presented in Figure 4.
For tasks with zero-one total reward (‘Pong’ and ‘Mountain Car’), the function to be smoothed represents the total reward: h ( · ) =
R where R is equal to 1 if the agent wins the round, and 0 otherwise.
To compute certiﬁcates on games with continuous scores (‘Cartpole’ and ‘Freeway’), we use CDF smoothing (Kumar et al., 2020): see appendix for technical details.
In order to evaluate the robustness of both undefended and policy-smoothed agents, we developed an attack tailored to the threat model deﬁned in 1, where the adversary makes a perturbation to state observations which is bounded over the entire episode .
For DQN agents, as in L¨utjens et al.
(2019), we perturb the observation o such that the perturbation-induced action a (cid:48) :
= arg max a Q ( o + (cid:15) t , a ) minimizes the (network-approximated) Q-value of the true observation Q ( o, a (cid:48) ) .
However, in order to conserve adversarial budget, we only attack if the gap between attacked q-value Q ( o, a (cid:48) ) and the clean q-value max a Q ( o, a ) is sufﬁciently large, exceeding a preset threshold λ Q .
In practice, this allows the attacker to concentrate the attack budget only on the time-steps which are critical to the agent’s performance.
When attacking DDPG, where both a Q-value network and a policy network π are trained and the action is taken according to π , we instead minimize Q ( o, π ( o + (cid:15) t ))
+ λ (cid:107) (cid:15) t (cid:107)where the hyperparameter
λ plays an analogous role in focusing perturbation budget on “important” steps, as judged by the effect on the approximated Q-value.
Empirical results are presented in Figure 5.
We see that the attacks are effective on the undefended agents (red, dashed lines).
In fact, from comparing Figures 4 and 5, we see that, for the Pong and Cartpole environments, the undefended performance under attack is worse than the certiﬁed lower bound on the performance of the policy-smoothed agents under any possible attack: our certiﬁcates are the clearly non-vacuous for these environments.
Further details on the attack optimizations are provided in the appendix.
We also present an attempted empirical attack on the smoothed agent, adapting techniques for attacking smoothed classiﬁers from Salman et al.
(solid blue lines).
We observed that our model was highly robust to this attack – signiﬁcantly more robust than guaranteed by our certiﬁcate.
However, it is not clear whether this is due to looseness in the certiﬁcate or to weakness of the attack: the signiﬁcant practical challenges to attacking smoothed agents are also discussed in the appendix.
C ONCLUSION
In this work, we extend randomized smoothing to design a procedure that can make any reinforcement learning agent provably robust against adversarial attacks without signiﬁcantly increasing the complexity of the agent’s policy.
We show how to adapt existing theory on randomized smoothing from static tasks such as classiﬁcation, to the dynamic setting of RL.
By proving an adaptive version of the celebrated Neyman-Pearson Lemma, we show that by adding Gaussian smoothing noise to the input of the policy, one can certiﬁably defend it against norm-bounded adversarial perturbations of its input.
The policy smoothing technique and its theory covers a wide range of adversaries, policies and environments.
Our analysis is tight, meaning that the certiﬁcates we achieve are best possible unless restrictive assumptions about the RL game are made.
In our experiments, we show that our method provides meaningful guarantees on the robustness of the defended policies and the total reward they achieve even in the worst case is higher than an undefended policy.
In the future, the introduction of randomized smoothing to RL could inspire the design of provable robustness techniques for control problems in dynamic real-world environments and multi-agent RL settings.
R EPRODUCIBILITY
We supplement our work with accompanying code for reproducing the experimental results, as well as pre-trained models for a selection of the experiments.
Details about setting hyper-parameters and the environments we test are included in the appendix.
Proofs for our theoretical results (lemmas and main theorem) are also provided in the appendix.
E THICS S TATEMENT
We present a method to make RL models provably robust under adversarial perturbations.
We do not foresee any immediate ethical concerns associated with our work.
A CKNOWLEDGEMENTS
This project was supported in part by NSF CAREER AWARD 1942230, a grant from NIST 60NANB20D134, HR001119S0026-GARD-FP-052, HR00112090132, ONR YIP award N0001422-1-2271, Army Grant W911NF2120076.
A T IGHTNESS OF THE C ERTIFICATE
Here, we present a worst-case environment-policy pair that achieves the bound in Theorem 1, showing that our robustness certiﬁcate is in fact tight.
For a given environment M = ( S, A, T, R, γ ) and a policy π , let p be a lower-bound on the probability that the total reward obtained by policy π under
Gaussian smoothing (no adversary) with variance σ 2 is above a certain threshold ν , i.e.,
P (cid:34) t (cid:88) i =
[Equation]
(cid:35)
[Equation]
Let H p be the class of all such environment-policy pairs that cross this reward threshold with probability at least p .
We construct an environment-policy pair ( M (cid:48) , π (cid:48) ) that achieves the reward threshold ν with probability Φ(Φ − 1 ( p ) − B/σ ) under the structured adversary (cid:15) st .
Note that, this does not mean that (cid:15) st is the strongest possible adversary for a general environment-policy pair.
It only shows that the performance of policy π (cid:48) in environment M (cid:48) under the adversary (cid:15) st is a lower-bound on the performance of a general environment-policy pair under a general adversary.
Consider a one-step game with environment M (cid:48)
= ( S, A, T (cid:48) , R (cid:48) , γ ) with a deterministic observation function o of the state-space and a policy π (cid:48)
such that π (cid:48) returns an action a 1 ∈ A if the ﬁrst coordinate of o ( s 1 ) + ηis at most ω
= { o ( s 1 ) } 1 + σ Φ − 1 ( p ) and another action a 2 ∈ A otherwise.
Here { o ( s 1 ) } 1 represents the ﬁrst coordinate of o ( s 1 ) .
The environment offers a reward ν if the action in the ﬁrst step is a 1 and 0 when it is a 2 .
The game terminates immediately.
The probability of the reward being above ν is equal to the probability of the action being a 1 .
When η 1 is sampled from the Gaussian distribution, this probability is equal to Φ(( ω − { o ( s 1 ) } 1 ) /σ )
= p .
Therefore, ( M (cid:48) , π (cid:48) ) ∈
H p .
Under the presence of the structured adversary (cid:15) st deﬁned in Section 4.1, this probability after smoothing becomes Φ(( ω − { o ( s 1 ) } 1 − B ) /σ )
=
Φ(Φ − 1 ( p ) − B/σ ) , which is same as the bound in Theorem 1.
B S TATIC V S . A DAPTIVE S ETTING
In this section, we illustrate the difference between the adversarial distributions in the static setting and the adaptive setting.
Naively, one might assume that smoothing-based robustness guarantees can be applied directly to reinforcement learning, by adding noise to observations.
For example, it seems plausible to use Cohen et al.
’s (cid:96) 2 certiﬁcate Cohen et al.
(2019), which relies on the overlap in the distributions of isometric Gaussians with different means, by simply adding Gaussian noise to each observation (Figure 1-a).
However, as we demonstrate with a toy example in Figure 1- (b-d), the Cohen et al.
certiﬁcate cannot be applied directly to the RL setting, because adding noise to sequential observations does not result in an isometric Gaussian distribution over the space of observations.
This is because the adversarial offset to later observations may be conditioned on the noise added to previous observations.
In 1-(b-d), we consider a two-step episode, and for simplicity, we consider a case where the ground-truth observations at each step are ﬁxed.
At step 1, the noised distributions of the clean observation o 1 and the adversarially-perturbed observation o (cid:48)are both Gaussians and overlap substantially, similar to in the standard classiﬁcation setting (panel b).
However, we see in panel (c) that the adversarial perturbation (cid:15) 2 added to o 2 can depend on the smoothed value of o (cid:48) 1 .
This is because the agent may leak information about the observation that it receives after smoothing ( o 1 + η 1 ) to the adversary, for example through its choice of actions.
After smoothing is performed on o 2 , the adaptive nature of the adversary causes the distribution of smoothed observations to no longer be an isometric Gaussian in the adversarial case (panel d).
The standard certiﬁcation results therefore cannot be applied.
C N EYMAN –P EARSON LEMMA [1993] IN S MOOTHING
In the context of randomized smoothing, the Neyman–Pearson lemma produces the worst-case decision boundary of a classiﬁer based on the estimated probability of the top class under the smoothing distribution.
It says that this boundary is a region where the ratio of the probability density functions of the smoothing distributions at the clean input and the perturbed input is a constant.
When the two distributions are isometric Gaussians, as is the case in static settings like image classiﬁcation, this boundary takes the form of a hyper-plane (see Appendix A of Cohen et al.
However, in the dynamic setting of RL, the smoothing distribution after adding the adversarial perturbation may not be isometric even if the smoothing noise at each time-step was sampled from an isometric Gaussian distribution (see ﬁgure 1, section ‘Technical contributions’ and Appendix A).
So, we formulate and prove an adaptive version of the Neyman-Pearson lemma to obtain provable robustness in RL through randomized smoothing.
D P ROOF OF L EMMA
Statement: For any general adversary (cid:15) and an Γ ⊆ ( T × R d ) t , there exists a deterministic adversary (cid:15) dt such that,
[Equation]
where Y dt is the random variable for the distribution deﬁned by the adversary (cid:15) dt .
Proof.
Consider a time-step j such that ∀ i < j, (cid:15) i is a deterministic function of τ 1 , η 1 , τ 2 , η 2 , . .
, τ i − 1 , η i − 1 , τ i .
Let H =
{ z | z 1 = τ 1 , z 2 = η 1 , z 3 = τ 2 , z 4 = η 2 , . .
z 2 j − 1 = a j } be the set of points whose ﬁrst 2 j − 1 coordinates are ﬁxed to an arbitrary set of values τ 1 , η 1 , τ 2 , η 2 , . . .
, τ j .
In the space deﬁned by H , (cid:15) 1 , . . .
, (cid:15) j − 1 are ﬁxed vectors in R d and (cid:15)
j is sampled from a ﬁxed distribution over the vectors with (cid:96) 2 -norm at most B jr .
Let Y γ H be the random variable representing the distribution over points in H deﬁned by the adversary for which (cid:15)
j = γ , such that (cid:107) γ (cid:107) 2 ≤ B jr .
Deﬁne an adversary (cid:15) (cid:48) , such that, (cid:15) (cid:48)
i = (cid:15) i , ∀ i (cid:54) =
j . Set (cid:15) (cid:48)
j to the vector γ that minimizes the probability that Y γ H assigns to Γ ∩ H , i.e.,
[Equation]
The adversary (cid:15) (cid:48) behaves as (cid:15) up to step j − 1 .
At step j , it sets (cid:15) (cid:48)
j to the γ that minimizes the probability it assigns to Γ ∩ H , based on the values τ 1 , η 1 , τ 2 , η 2 , . .
τ j .
After that, it mimics (cid:15) till the last time-step t .
Therefore, for a given tuple ( z 1 , z 2 , . .
z 2 j − [Equation] ) =
( τ 1 , η 1 , τ 2 , η 2 , . . .
, τ j ) ,
[Equation]
Since both adversaries are same up to step j − 1 , their respective distributions over z 1 , z 2 , . .
z 2 j −remains same as well.
Therefore, integrating both sides of the above inequality over the space of all tuples ( z 1 , z 2 , . .
z 2 j − 1 ) , we have:
[Equation]
where Y (cid:48) is the random variable corresponding to (cid:15) (cid:48) .
Thus, we have constructed an adversary where the ﬁrst j adversarial perturbations are a deterministic function of the τ i s and η i s of the previous rounds.
Applying the above step sufﬁciently many times we can construct a deterministic adversary (cid:15) dt represented by the random variable Y dt such that
[Equation]
E P ROOF OF L EMMA
Lemma 3 states that the structured adversary characterises the worst-case scenario.
Before proving this lemma, let us ﬁrst show that any deterministic adversary can be converted to one that uses up the entire budget of B without increasing the probability it assigns to h being one in the worst-case.
For each step i , let us deﬁne a used budget B i u = (cid:107) ( (cid:15) 1 , (cid:15) 2 , . . .
, (cid:15) i − 1 ) (cid:107) 2 as the norm of the perturbations of the previous steps and a remaining budget B ir =
(cid:112) B 2 − ( B iu ) 2 as an upper-bound on the norm of the perturbations of the remaining steps.
Note that, B 1 u = 0 and B 1 r = B .
Consider a version ˜ (cid:15) dt of the deterministic adversary that uses up the entire available budget B by scaling up (cid:15) dtt such that its norm is equal to B tr , i.e., setting it to (cid:15) dtt B tr / (cid:107) (cid:15) dtt (cid:107) 2 .
Let ˜ Y
dt be the random variable representing ˜ (cid:15) dt .
Lemma.
If Γ ˜ Y dt = { z ∈ ( T × R d ) t | µ ˜ Y
dt ( z ) ≤ qµ X ( z ) } for some q ≥ 0 and P [ h ( X ) = 1] ≥
P [ X ∈ Γ ˜
Y dt ] , then P [ h ( Y dt )
= 1] ≥ P
[ ˜ Y dt ∈ Γ ˜ Y dt ] .
Proof.
Consider Γ Y dt
= { z ∈
( T × R d ) t | µ Y dt ( z ) ≤ q (cid:48) µ X ( z ) } for some q (cid:48) ≥ 0 , such that, P [ X ∈ Γ Y dt ] = p for some lower-bound p on P [ h ( X ) = 1] .
Then, by the Neyman-Pearson Lemma we have that,
[Equation]
Now consider a space H in ( T × R d ) t where all but the last element of the tuple z are ﬁxed, i.e.,
H =
{ z | z 1 = τ 1 , z 2 = η 1 , z 3 = τ 2 , z 4 = η 2 , . .
z 2 t − 1 = τ t } Since, (cid:15) dt is a deterministic adversary where each (cid:15) dti is a deterministic function of the previous τ i s and η i s, each (cid:15) dti is also ﬁxed in H .
Therefore, in H , both µ X and µ Y dt are two isometric Gaussians in the space of the η i s and the set H ∩ Γ Y dt is a hyperplane.
The probability assigned by Y dt to H ∩ Γ Y dt is proportional to the distance of the center of the corresponding Gaussian.
In the construction of ˜ (cid:15) dt , this distance can only increase, therefore,
[Equation]
Now, consider a function h Γ Y dt ( z ) which outputs one if z ∈ Γ Y dt and zero otherwise.
Construct the set Γ ˜ Y
{+dt = { z ∈ ( T × R d ) t | µ ˜ Y dt ( z ) ≤ qµ X ( z ) } for some q ≥ 
 such that,+}
P
[Equation]
X ∈
[Equation]
X ) = 1] .
Then, by the Neyman-Pearson Lemma, we have,
[Equation]
or, P [ ˜ Y dt ∈ Γ Y dt ] ≥ P [ ˜ Y dt ∈ Γ ˜ Y dt
[Equation]
(from deﬁnition of h Γ Ydt ) or, P [ Y dt ∈ Γ Y dt ] ≥ P [ ˜ Y dt ∈ Γ ˜
Y dt ] or, P [ h ( Y dt )
= 1] ≥ P [ ˜ Y dt ∈ Γ ˜
Y dt ] , (from the above two inequalities) proving the statement of the lemma.
Now, we prove lemma 3 below:
Statement: If Γ Y st = { z ∈ ( T × R d ) t | µ Y st ( z ) ≤ qµ X ( z ) } for some q ≥ 0 and P [ h ( X ) = 1] ≥ P [ X ∈ Γ Y st ] , then P [ h ( Y dt )
= 1] ≥ P
[ Y st ∈ Γ Y st ] .
Proof.
Construct the set Γ ˜ Y dt as deﬁned in the above lemma for a q ≥ 0 such that P [ X ∈ Γ ˜
Y dt ] = p , for some lower-bound p on P [ h ( X ) = 1] .
Then,
[Equation]
Now consider the structured adversary (cid:15) st in which (cid:15) st 1 = ( B, 0 , . .
. , 0) and (cid:15) sti =
(0 , 0 , . . . , 0) for i > 1 .
Deﬁne the set Γ Y st
= { z ∈ ( T × R d ) t | µ Y st ( z ) ≤ qµ X ( z ) } for the same q as above.
Then, we can show that:
P [ ˜ Y dt ∈ Γ
˜ Y dt ] = P [ Y st ∈
Γ Y st ] , and 2.
P [ X ∈ Γ ˜ Y
[Equation]
which, in turn, prove the statement of the lemma.
Let N and N (cid:15) i represent Gaussian distributions centered at origin and (cid:15)
i respectively.
Then, we can write µ X and µ Y as below:
[Equation]
where µ T i is the conditional probability distribution of token τ i given the previous tokens and offsets.
Therefore,
[Equation]
Consider a round j ≤ t such that ˜ (cid:15) dti = 0 , ∀ i > j + 1 and ˜ (cid:15) dtj +1 =
( B j +1 r , 0 , . . . , 0) .
We can always ﬁnd such a j as we always have ˜ (cid:15) dtt +1 =
( B t +1 r , 0 , . . . , 0) , since B t +1 r = 0 .
Note that,
[Equation]
j
+
[Equation]
(cid:17) 2 and in turn ˜ (cid:15) dtj +1 are functions of τ 1 , η 1 , τ 2 , η 2 , . . .
, τ j and not τ j +1 .
Let H =
{ z | z 1 = τ 1 , z 2 = η 1 , z 3 = τ 2 , z 4 = η 2 , . .
z 2 j − 1 = τ j } be the set of points whose ﬁrst 2 j − 1 coordinates are ﬁxed to an arbitrary set of values τ 1 , η 1 , τ 2 , η 2 , . . .
, τ j .
For points in
H , all ˜ (cid:15) dti for i ≤ j + 1 are ﬁxed and for i > j + 1 are set to zero.
Let ˜ Y
dt H denote the random variable representing the distribution of points in H deﬁned by the adversary ˜ (cid:15) dt
(corresponding random variable ˜ Y dt ).
In the space of η j , η j +1 , . . .
, η t , this is an isometric Gaussian centered at (˜ (cid:15) dtj , ˜ (cid:15) dtj +1 , 0 , . . . , 0) .
Therefore, Γ ∩ H is given by
[Equation]
for some constant β dependent on η 1 , ˜ (cid:15) dt 1 , . . .
, η j − 1 , ˜ (cid:15) dtj − 1 , σ and t .
The probability assigned by the
Gaussian random variable Y H to the half-space deﬁned by (3) is proportional to the distance of the center of the Gaussian from the hyper-plane in (3), which is equal to:
[Equation]
where the equality follows from:
[Equation]
(from deﬁnition of B ir )
[Equation]
Now, consider an adversary ˜ (cid:15) dt (cid:48) such that ˜ (cid:15) dt (cid:48)
i
[Equation]
= ( B jr , 0 , . .
. , 0) , and ˜ (cid:15) dti = 0 , ∀ i > j .
Let ˜ Y dt (cid:48) be the corresponding random variable.
Deﬁne Γ ˜
Y dt (cid:48) similar to Γ ˜ Y
dt .
Then, Γ ˜ Y dt (cid:48) ∩ H is given by
[Equation]
which is obtained by replacing ˜ (cid:15) dtj with ( B jr , 0 , . .
. , 0) and ˜ (cid:15)
dtj +1 with (0 , 0 , . .
. , 0) in inequality (3) about the origin.
Deﬁne ˜
Y dt (cid:48)
H similar to ˜ Y
dt H , and just like ˜ Y dt H , the distribution of ˜ Y dt (cid:48)
H is also an isometric Gaussian, but is centered at (( B jr , 0 , . . .
The probability assigned by this Gaussian distribution to Γ ˜
Y dt (cid:48) ∩H is proportional to the distance of its center to the hyper-plane deﬁning the region in (4), which is equal to (( B jr ) 2 − β ) /B
jr .
Therefore,
[Equation]
The key intuition behind this step is that, for isometric Gaussian smoothing distribution, the worstcase probability assigned by the adversarial distribution only depends on the magnitude of the perturbation and not its direction.
Figure 6 illustrates this property for a two-dimensional input space.
Since both adversaries are same up to step j − 1 , their respective distributions over z 1 , z 2 , . .
z 2 j −remains same as well
, i.e., p ˜ Y dt ( z 1 , z 2 , . . .
, z
2 j − 1 ) =
p ˜
Y dt (cid:48)
( z 1 , z 2 , . . .
, z
2 j − 1 ) .
Integrating over the space of all tuples ( z 1 , z 2 , . .
z 2 j − 1 ) , we have:
[Equation]
Since the distribution deﬁned by X (with no adversary) over the space of η i s is a Gaussian centered at origin whose distance to both Γ ˜ Y
dt ∩ H and Γ ˜
Y dt (cid:48)
∩ H is the same (equal to − β/B jr ), it assigns the same probability to both (3) and (4).
Therefore,
[Equation]
Thus, we have constructed an adversary with one less non-zero (cid:15) i .
Applying, this step sufﬁciently many times we can obtain the adversary (cid:15) st
such that,
[Equation]
] = P [ Y st ∈ Γ Y st ] and P [ X ∈ Γ ˜ Y dt ] = P [ X ∈ Γ Y st ] which completes the proof.
F P ROOF OF T HEOREM
Statement: For an isometric Gaussian smoothing noise with variance σ 2 , if P [ h ( X ) = 1] ≥ p , then: P
[Equation]
Proof.
Deﬁne Γ Y =
{ z ∈ ( T × R d ) t | µ Y ( z ) ≤ qµ X ( z ) } for an appropriate q such that P [ X ∈ Γ Y ] = p .
Then, by the Neyman-Pearson lemma, we have P [ h ( Y ) = 1] ≥ P
[ Y ∈ Γ Y ] .
Applying lemma 1, we know that there exists a deterministic adversary (cid:15) dt represented by random variable
Y dt , such that,
[Equation]
Now deﬁne a function h Γ Y ( z ) = 1 { z ∈ Γ Y } and a set Γ Y dt
= { z ∈
( T × R d ) t | µ Y dt ( z ) ≤ q (cid:48) µ X ( z ) } for an appropriate q (cid:48) > 0 , such that, P [ X ∈ Γ Y dt ]
= P [ h Γ Y ( X )
= 1] = p .
Applying the Neyman- Pearson lemma again, we have:
[Equation]
or, P [
[Equation]
P [
[Equation]
] (from deﬁnition of h Γ Y ) or, P [ h ( Y ) = 1] ≥ P [ Y dt ∈ Γ Y dt ] (from inequality (5))
Deﬁne h
Γ Ydt ( z ) =
1 { z ∈ Γ Ydt } .
For the structured adversary (cid:15) st represented by Y st , deﬁne Γ Y st = { z ∈ ( T × R d ) t | µ Y st ( z ) ≤ q (cid:48)(cid:48) µ X ( z ) } for an appropriate q (cid:48)(cid:48) > 0 , such that, P [ X ∈ Γ Y st ] =
P [ h Γ Ydt ( X ) = 1] = p .
Applying lemma 3, we have:
[Equation]
(from deﬁnition of h Γ Ydt )
[Equation]
(since P [ h ( Y ) = 1] ≥ P [ Y dt ∈ Γ Y dt ] ) Γ Y st is deﬁned as the set of points z which satisfy:
[Equation]
or
[Equation]
for some constant β .
This is the set of all tuples z where the ﬁrst coordinate of η 1 is below a certain threshold γ .
Since P [ X ∈ Γ Y st ] = p , Φ( γ/σ ) = p = ⇒ γ
= σ Φ − 1 ( p ) .
Therefore,
[Equation]
G A DDITIONAL C ARTPOLE R ESULTS
We performed two additional experiment on Cartpole: we tested at larger noise levels, ( σ = 0 . 6 and 0 . 8 ) and we tested a variant of the agent architecture.
Speciﬁcally, in addition to the agent shown in the main text, which uses ﬁve frames of observation, we also tested an agent which uses only a single frame.
Unlike the Atari environment, the task is in fact solvable (in the non-adversarial case) using only one frame: the observation vector represents the complete system state.
We computed certiﬁcates for the policy-smoothed version of this model, and tested attacks on the undefended version.
(We did not test attacks on the smoothed single-frame variant).
As we see in Figure 7, we achieve non-vacuous certiﬁcates in both settings (i.e, at large perturbation sizes, the smoothed agent is guaranteed to be more robust than the empirical robustness of a non-smoothed agent).
However, observe that the undefended agent in the multi-frame setting is much more vulnerable to adversarial attack.
This is likely because the increased number of total features (20 vs. four) introduces more complexity of the Q-network, making it more vulnerable against adversarial attack.
H F ULL P ONG G AME
In Figure 8, we explore a failure case of our technique: we fail to produce non-vacuous certiﬁcates for a full Pong game, where the game ends after either player scores 21 goals.
In particular, while, for the one-round Pong game, the smoothed agent is provably more robust than the empirical performance of the undefended agent, this is clearly not the case for the full game.
To understand why our certiﬁcate is vacuous here, note that in the the “worst-case” environment that our certiﬁcate assumes, any perturbation will (maximally) affect all future rewards.
However, in the multi-round Pong game, each round of the game is only loosely coupled to the previous rounds (the ball momentum – but not position – as well as the paddle positions are retained).
Therefore, any perturbation can only have a very limited effect on the total reward.
Another way to think about this is to recall that in smoothing-based certiﬁcates, the noise added to each feature is proportional to the total perturbation budget of the adversary.
In this sort of serial game, the perturbation budget required to attack the average reward scales with the (square root of the) number of rounds, but the noise tolerance of the agent does not similarly scale.
I T RAINING AND C LEAN T EST R ESULTS
In Figure 9, we present the clean (non-attacked) test performance for the experiments presented in the main text, as a function of the smoothing noise σ .
In Figure 10, we present the clean training (i.e., validation round) performance as a function of the training time step and the smoothing noise σ .
Note that early stopping was applied: the model from the best validation round was kept, and only replaced if a strictly better validation performance was recorded later.
• For Cartpole: logs were not kept after the ﬁrst time an evaluation round had a perfect average score of 200 (this is because the “best model” was saved for this evaluation, and it would be impossible to beat this score, so training was not continued).
However, for other tasks (i.e. mountain car) logs continued after a perfect evaluation round.
• For Freeway: as mentioned in Appendix Section L, we trained 5 times at each noise level, and kept the best of all 5 models.
All 5 training curves are shown here for each noise level.
J C OMPLETE A TTACK R ESULTS
In Figures 11 and 12, we report the empirical robustness under attack for all tested values of λ Q : in the main text, we show only the result for the λ Q that represents the strongest attack.
Figurealso shows the attacks on smoothed agents for all smoothing noises.
All attack results are means over 1000 episodes (except for Mountain Car results, where 250 episodes were used) and error bars represent the standard error of the mean.
K E MPIRICAL A TTACK D ETAILS
Our empirical attack on (undefended) RL observations for DQN is described in Algorithm 1.
To summarize, the core of the attack is a standard targeted L 2 PGD attack on the Q-value function.
However, because we wish to “save” our total perturbation budget B for use in later steps, some modiﬁcations are made.
First, we only target actions a for which the clean-observation Q-value is sufﬁciently below (by a gap given by the parameter λ Q ) the Q-value of the ‘best’ action, which would be taken in the absence of adversarial attack.
Among these possible Targets, we ultimately choose whichever action will maximally decrease the Q-value, and which the agent can be successfully be induced to choose within the adversarial budget B .
If no such action exists, then the original observation will be returned, and the entire budget will be saved.
In order to preserve budget, the
PGD optimization is stopped as soon as the “decision boundary” is crossed.
We use a constant step size η .
In order to deal with the variable budget B , we optimize of a number of iterations which is a constant multiple ν of Bη .
For most environments, there is some context used by the Q-value function (i.e, the previous frames) which is carried over from previous steps, but is not directly being attacked in this round.
We need both the clean version of the context, C , in order to evaluate the “ground-truth” values of the Qvalue function under various actions; as well as the “dirty” version of the context, C (cid:48) , based on the adversarial observations which have already been fed to the agent, in order to run the attack optimization.
Our attack for DDPG is described in Algorithm 2.
Here, we use the policy π to determine what action a the agent will take when it observes a corrupted observation o (cid:48)
(with corrupted context
C (cid:48) ), and use the Q-value function supplied by the DDPG algorithm to determine the “value” of that action on the ground-truth observation o .
Because our goal is to minimize this value, this amounts to minimizing Q ( C ; o, π ( C (cid:48) ; o (cid:48) )) .
In order to ensure that a large amount of L 2 “budget” is only used when the Q value can be substantially minimized, we include a regularization term λ (cid:107) o − o (cid:48) (cid:107) 22 .
Attacks on smoothed agents are described in Appendix M. [Table] MLP MLP
[Table] [Table] of the Pong environment. Note that on image data (i.e., Pong), we do not consider integrality constraints on the observations; however, we do incorporate box constraints on the pixel values.
We also incorporate box constraints on the kinematic quantities when attacking Mountain Car, but not when attacking Cartpole: the distinction is that the constraints in Mountain Car represent artiﬁcial constraints on the kinematics [i.e., the velocity of the car is arbitrarily clipped], while the constraints in Cartpole arise naturally from the problem setup.
L E NVIRONMENT DETAILS AND H YPERPARAMETERS
For Atari games, we use the “NoFrameskip-v0” variations of these environments with the standard “AtariPreprocessing” wrapper from the OpenAI Gym (Brockman et al., 2016) package: this provides
This environment also injects non-determinism into the originally-deterministic Atari games, by adding randomized “stickiness” to the agent’s choice of actions – without this, the state-observation robustness problem could be trivially solved by memorizing a winning sequence of actions, and ignoring all observations at test-time.
Due to instability in training, for the freeway environment, we trained each model ﬁve times, and selected the base model based on the performance of validation runs.
See training hyperparameters,
Tables 1 and 2.
For attack hyperparameters, see Table 3 and 4.
[Table]
[Table]
[Table]
Input: Q-value function Q , clean prior observation context C , adversarial prior observation context C (cid:48) , observation o , budget B , Q-value threshold λ Q , step size η , step multiplier ν
Output: Attacked observation o worst , remaining budget B (cid:48) .
Q clean := max a ∈ A Q ( C ; o, a )
Targets := { a ∈ A | Q ( C ; o, a ) ≤ Q clean − λ Q }
Q worst := Q clean o worst :
= o for a ∈ Targets do o (cid:48) :
= o inner: for i in 1, ..., (cid:98) νBη (cid:99) do if arg max a (cid:48) Q ( C (cid:48) ; o (cid:48) , a (cid:48) )
=
a then if Q ( C ; o, a ) < Q worst then o worst :=
o (cid:48)
Q worst := Q ( C ; o, a ) endbreak inner end
D := ∇ o (cid:48) log([ SoftMax ( Q ( C (cid:48) ; o (cid:48) , · )] a ) o (cid:48) :
= o (cid:48)
+ ηD (cid:107) D (cid:107)if (cid:107) o (cid:48)
− o (cid:107) 2 > B then o (cid:48) :
= o +
B (cid:107) o (cid:48) − o (cid:107) 2 ( o (cid:48)
− o ) end end end return o worst , (cid:112) B 2 − (cid:107) o worst − o (cid:107)
Input: Q-value function Q , policy π , clean prior observation context C , adversarial prior observation context C (cid:48) , observation o , budget B , weight parameter λ , step size η , step count τ
Output: Attacked observation o worst , remaining budget B (cid:48) .
o (cid:48) : = o for i in 1, ..., τ do D := ∇ o (cid:48) [ Q ( C ; o, π ( C (cid:48) ; o (cid:48) ))
+ λ (cid:107) o (cid:48)
− o (cid:107) 22 ] if (cid:107) D (cid:107) 2 (cid:107) o (cid:48) (cid:107) 2 ≤ 0 .
001 then break end o (cid:48) :
= o (cid:48)
+ ηD (cid:107) D (cid:107)if (cid:107) o (cid:48)
− o (cid:107) 2 > B then o (cid:48) :
= o +
B (cid:107) o (cid:48) − o (cid:107) 2 ( o (cid:48) − o ) end end return o (cid:48) , (cid:112) B 2 − (cid:107) o (cid:48)
− o (cid:107)
In order to attack smoothed agents, we adapted Algorithms 1 and 2 using techniques suggested by Salman et al. (2019) for attacking smoothed classiﬁers.
In particular, whenever the Q-value function is evaluated or differentiated, we instead evaluate/differentiate the mean output under m = smoothing perturbations. Following Salman et al. (2019), we use the same noise perturbation vectors at each step during the attack.
In the multi-frame case, for the “dirty” context C (cid:48) , we include the actually-realized smoothing perturbations used by the agents for previous steps.
However, when determining the “clean” Q-values Q ( C ; o, a ) , for the “clean” context C , we use the unperturbed previous state observations: we then take the average over m smoothing perturbations of both C and o to determine the clean Q-values.
This gives an unbiased estimate for the Q-values of an undisturbed smoothed agent in this state.
When attacking DDPG, in evaluating Q ( C ; o, π ( C (cid:48) ; o (cid:48) ) , we average over smoothing perturbations for both o and o (cid:48) , in addition to C : this is because both π and Q are trained on noisy samples.
Note that we use independently-sampled noise perturbations on o (cid:48) and o .
Our attack does not appear to be successful, compared with the lower bound given by our certiﬁcate (Figures 12).
One contributing factor may be that attacking a smoothed agent is more difﬁcult that attacking a smoothed classiﬁer , for the following reason: a smoothed classiﬁer evaluates the expected output at test time, while a smoothed agent does not.
Thus, while the average Q-value for the targeted action might be greater than the average Q-value for the clean action, the actual realization will depend on the speciﬁc realization of the random smoothing vector that the agent actually uses.
N R UNTIMES AND C OMPUTATIONAL E NVIRONMENT
Each experiment is run on an NVIDIA 2080 Ti GPU.
Typical training times are shown in Table 5.
Typical clean evaluation times are shown in Table 6.
Typical attack times are shown in Table 7.
[Table]
[Table]
O CDF S MOOTHING D ETAILS
Due to the very general form of our certiﬁcation result ( h ( · ) , as a 0/1 function, can represent any outcome [Table] and we can bound the lower-bound the probability of this outcome) [Table]
there are a variety of [Table] ways we can use the basic result to compute a certiﬁcate for an entire episode entire game. In the main text, we introduce CDF smoothing Kumar et al.
(2020) as one such option.
In CDF smoothing for any threshold value x , we can deﬁne h x ( · ) as an indicator function for the event that the total episode reward is greater than x .
Then, by the deﬁnition of the CDF function, the expectation of h x ( · ) is equal to 1 − F ( x ) , where F ( · ) is the CDF function of the reward.
Then our lower-bound on the expectation of h x ( · ) under adversarial attack is in fact an upper-bound on F ( x ) : combining this with Equation 2 in the main text,
[Equation]
provides a lower bound on the total expectation of the reward under adversarial perturbation.
However, in order to perform this integral from empirical samples, we must bound F ( x ) at all points: this requires ﬁrst upper-bounding the non-adversarial CDF function at all x , before applying our certiﬁcate result.
Following Kumar et al.
, we accomplish this using the Dvoretzky–Kiefer–Wolfowitz inequality (for the Full Pong environment.)
In the case of the Cartpole environment, we explore a different strategy: note that the reward at each timestep is itself a 0/1 function, so we can deﬁne h t ( · ) as simply the reward at timestep t .
We can then apply our certiﬁcate result at each timestep independently, and take a sum.
Note that this requires estimating the average reward at each step independently: we use the Clopper-Pearson method (following Cohen et al.
(2019)), and in order to certify in total to the desired 95% conﬁdence bound, we certify each estimate to (100 - 5/T)% conﬁdence, where T is the total number of timesteps per episode (= 200).
However, note that, in the particular case of the cartpole environment, h t ( · ) = 1 if and only if we have “survived” to time-step t : in other words, h t ( · ) is simply an indicator function for the total reward being ≥ t .
Therefore in this case, this independent estimation method is equivalent to CDF smoothing, just using Clopper-Pearson point-estimates of the CDF function rather than the Dvoretzky–Kiefer–Wolfowitz inequality.
In practice, we ﬁnd that this produced slightly better certiﬁcates for this task.
(Figure 13)
P E NVIRONMENT L ICENSES
OpenAI Gym Brockman et al.
(2016) is Copyright 2016 by OpenAI and provided under the MIT
License.
The stable-baselines3 packageRafﬁn et al.
(2019) is Copyright 2019 by Antonin Rafﬁn and also provided under the MIT License.