CROP: C ERTIFYING R OBUST P OLICIES
FOR
R EINFORCEMENT L EARNING THROUGH F UNCTIONAL
S MOOTHING
Anonymous authors
Paper under double-blind review
A BSTRACT
We present the ﬁrst framework of Certifying Robust Policies (CROP) for reinforcement learning against adversarial state perturbations.
We propose two particular types of robustness certiﬁcation criteria: robustness of per-state actions and lower bound of cumulative rewards .
Speciﬁcally, we develop a local smoothing algorithm that uses a policy derived from Q-functions smoothed with Gaussian noise over each encountered state to guarantee the robustness of actions taken along this trajectory.
Next, we develop a global smoothing algorithm for certifying the robustness of a ﬁnite-horizon cumulative reward under adversarial state perturbations.
Finally, we propose a local smoothing approach that makes use of adaptive search in order to obtain tight certiﬁcation bounds for the reward.
We use the proposed RL robustness certiﬁcation framework to evaluate methods that have previously been shown to yield empirically robust RL, including adversarial training and several forms of regularization, on three representative Atari games.
We show that RegPGD, RegCVX and RadialRL achieve high certiﬁed robustness among these.
Furthermore, by evaluating these algorithms against adversarial attacks, we demonstrate that our certiﬁcations are often tight.
All experiment results are available at website https://crop-leaderboard .
me .
I NTRODUCTION
Reinforcement learning (RL) has been widely applied to different applications, such as robotics (Kober et al., 2013; Deisenroth et al., 2013; Polydoros & Nalpantidis, 2017), autonomous driving vehicles (Shalev-Shwartz et al., 2016; Sallab et al., 2017), and trading (Deng et al., 2016; Almahdi & Yang, 2017; Ye et al., 2020).
However, recent studies have shown that learning algorithms are vulnerable to adversarial attacks (Goodfellow et al., 2014;
Kurakin et al., 2016; Moosavi-Dezfooli et al., 2016; Jia & Liang, 2017; Eykholt et al., 2018), and a range of attacks have also been proposed against the input states and trained policies of RL (Huang et al., 2017; Kos & Song, 2017; Lin et al., 2017; Behzadan & Munir, 2017a).
As more and more safety-critical applications are being deployed in real-world (Christiano et al., 2016; Fisac et al., 2018; Cheng et al., 2019; eop), how to test and improve their robustness before massive production is of great importance.
To defend against adversarial attacks in RL, different empirical defenses have been proposed (Mandlekar et al., 2017; Behzadan & Munir, 2017b; Pattanaik et al., 2018;
Fischer et al., 2019; Zhang et al., 2020; Oikarinen et al., 2020; Donti et al., 2020;
Shen et al., 2020; Eysenbach & Levine, 2021).
In particular, adversarial training (Kos & Song, 2017; Behzadan & Munir, 2017b; Pattanaik et al., 2018) and regularized RL algorithms by enforcing the smoothness of the trained models (Shen et al., 2020;
Zhang et al., 2020) have been studied to improve the robustness of trained policies.
However, several strong adaptive attacks have been proposed against these empirical defenses (Gleave et al., 2019;
Hussenot et al., 2019; Russo & Proutiere, 2019) and it is important to provide robustness certiﬁcation for a given learning algorithm to end such repeated game between attackers and defenders.
To provide robustness certiﬁcation , several studies have been conducted on classiﬁcation.
For instance, both deterministic (Ehlers, 2017; Katz et al., 2017; Cheng et al., 2017; Tjeng et al., 2017; Weng et al., 2018; Zhang et al., 2018; Singh et al., 2019; Gehr et al., 2018; Wong & Kolter, 2018; Raghunathan et al., 2018) and probabilistic approaches (Lecuyer et al., 2019; Cohen et al., 2019; Lee et al., 2019;
Salman et al., 2019; Carmon et al., 2019; Jeong & Shin, 2020) have been explored to provide a lower bound of classiﬁcation accuracy given bounded adversarial perturbation.
Considering the sequential decision making property of RL, which makes it more challenging to be directly certiﬁed compared to classiﬁcation, in this paper we ask: How to provide efﬁcient and effective robustness certiﬁcation for RL algorithms?
What criteria should be used to certify the robustness of RL algorithms?
Differently than classiﬁcation which involves one-step prediction only, RL algorithms provide both action prediction and reward feedback, making what to certify and how to certify robustness of RL challenging.
In this paper we focus on Q-learning and propose two robustness certiﬁcation criteria: per-state action stability and lower bound of perturbed cumulative reward .
In particular, to certify the per-state action stability , we propose the local smoothing on each input state and therefore derive the certiﬁed radius for perturbation at each state, under which the action prediction will not be altered.
To certify the lower bound of perturbed cumulative reward , we propose both global smoothing over the whole ﬁnite state trajectory to obtain the expectation or percentile bounds given trajectories smoothed with sampled noise sequences; and local smoothing to calculate an absolute lower bound based on our adaptive search algorithm.
We leverage our framework to test six empirically robust RL algorithms on three representative Atari games.
We show that the certiﬁed robustness depends on both the algorithm and the game properties.
For instance, RadialRL (Oikarinen et al., 2020) is the most certiﬁably robust method on Freeway.
In addition, based on the per-state certiﬁcation, we observe that for some games such as Pong, some states are more certiﬁably robust and such pattern is periodic.
This means given the information of which states are more vulnerable, it is possible to design robust algorithms to speciﬁcally focus on these vulnerable states.
Based on the lower bound of perturbed cumulative reward, we show that our certiﬁcation is tight by comparing our bounds with adversarial attacks for the games.
Technical Contributions.
In this paper, we take the ﬁrst step towards providing robustness certiﬁca - tion for Q-learning.
We make contributions on both theoretical and empirical fronts.
• We propose the ﬁrst framework for certifying the robustness of Q-learning algorithms.
• We propose two robustness certiﬁcation criteria for Q-learning algorithms, together with corresponding certiﬁcation algorithms based on global and local smoothing strategies.
• We theoretically prove the certiﬁcation radius for input state and lower bound of perturbed cumulative reward under bounded adversarial state perturbations.
• We conduct extensive experiments to provide certiﬁcation for six empirically robust RL algorithms on three Atari games.
We provide several interesting observations which would further inspire the development of robust RL algorithms.
P RELIMINARIES
Q-learning and Deep Q-Networks (DQNs).
Markov decision processes (MDPs) are at the core of
RL.
Our focus is on discounted discrete-time MDPs, which are deﬁned by tuple ( S , A , R, P, γ, d 0 ) , where S is a set of states (each with dimensionality N ), A represents a set of discrete actions,
R : S × A → R is the reward function, and P : S ×
A → P ( S ) is the transition function with P ( · ) deﬁning the set of probability measures, γ ∈ [0 , 1] is the discount factor, and d 0 ∈ P ( S ) is the distribution over the initial state.
At time step t , the agent is in the state s t ∈ S .
After choosing action a t ∈ A , the agent transitions to the next state s t +1 ∼ P ( s t , a t ) and receives reward R ( s t , a t ) .
The goal is to learn a policy π : S → P ( A ) that maximizes the expected cumulative reward E [ (cid:80) t γ t r t ] .
Q-learning (Watkins & Dayan, 1992) learns an action-value function (Q-function), Q (cid:63) ( s, a ) , which is the maximum expected cumulative reward the agent can achieve after taking action a in state s : Q (cid:63) ( s, a )
=
R ( s, a ) + γ
E s (cid:48) ∼ P ( s,a ) [max a (cid:48) Q (cid:63) ( s (cid:48) , a (cid:48) )] .
In deep Q-Networks (DQNs) (Mnih et al., 2013), Q (cid:63) is approximated using a neural network parametrized by θ , i.e. , Q π ( s, a ; θ ) ≈ Q (cid:63) ( s, a ) .
Let ρ ∈ P ( S × A ) be the observed distribution deﬁned over states s and actions a , the network can be trained via minimizing loss function L ( θ ) =
E ( s,a ) ∼ ρ,s (cid:48) ∼ P ( s,a ) (cid:104)
( R ( s, a ) + γ max a (cid:48)
Q π ( s (cid:48) , a (cid:48) ; θ ) − Q π ( s, a ; θ )) 2 (cid:105) .
The greedy policy π is deﬁned as taking the action with highest Q
π value in each state s : π ( s ) =
arg max a ∈A Q π ( s, a ) .
Certiﬁed Robustness for Classiﬁers via Randomized Smoothing.
Randomized smoothing (Cohen et al., 2019) has been proposed to provide probabilistic certiﬁed robustness for classiﬁcation models.
It achieves state-of-the-art certiﬁed robustness on large-scale datasets such as ImageNet under (cid:96) 2 -bounded constraints (Salman et al., 2019; Yang et al., 2020).
In particular, given a base model and a test instance, a smoothed model is constructed by outputting the most probable prediction over different Gaussian perturbed inputs.
R OBUSTNESS C ERTIFICATION IN Q- LEARNING
In this section, we ﬁrst introduce the threat model, followed by two robustness certiﬁcation criteria for the Q-learning algorithm: per-state action and cumulative reward .
We consider the standard adversarial setting in Q-learning (Huang et al., 2017; Kos & Song, 2017; Zhang et al., 2020), where the adversary can apply (cid:96) 2 -bounded perturbation B ε = { δ ∈ R n | (cid:107) δ (cid:107) 2 ≤ ε } to input state observations of the agent during decision (test) time to cause the policy to select suboptimal actions.
The agent observes the perturbed state and takes action a (cid:48)
=
π ( s + δ ) , following policy π .
Following the
Kerckhoff’s principle (Shannon, 1949), we consider a worst-case adversary who applies adversarial perturbations to every state at decision time.
Our analysis and methods are generalizable to other (cid:96) p norms following (Yang et al., 2020; Lecuyer et al., 2019).
3. R OBUSTNESS C ERTIFICATION FOR Q- LEARNING WITH D IFFERENT C RITERIA
To provide the robustness certiﬁcation for Q-learning, we propose two certiﬁcation criteria: per-state action robustness and lower bound of the cumulative reward .
Robustness Certiﬁcation for Per-State Action.
We ﬁrst aim to explore the robustness (stability/consistency) of the per-state action given adversarially perturbed input states.
Deﬁnition 1 (Certiﬁcation for per-state action) .
Given a trained network Q π with policy π , we deﬁne the robustness certiﬁcation for per-state action as the maximum perturbation magnitude ¯ ε , such that for any perturbation δ ∈ B ¯ ε , the predicted action under the perturbed state will be the same as the action taken in the clean environment, i.e. , π ( s + δ ) =
π ( s ) , ∀ δ ∈ B ¯ ε .
Robustness Certiﬁcation for Cumulative Reward.
Given that the cumulative reward is important for RL, here in addition to the per-state action, we also deﬁne the robustness certiﬁcation regarding the cumulative reward under input state perturbation.
Deﬁnition 2 (Cumulative reward) .
Let P : S ×
A → P ( S ) be the transition function of the environment with P ( · ) deﬁning the set of probability measures.
Let R, d 0 , γ, Q π , π be the reward function, initial state distribution, discount factor, a given trained Q-network, and the corresponding greedy policy as introduced in Section 2.
J ( π ) represents the cumulative reward and J ε ( π ) represents the perturbed cumulative reward under perturbations δ t ∈ B ε at each time step t :
[Equation]
where s t +1 ∼ P ( s t , a t ) , s 0 ∼ d 0 , and
J
[Equation]
R
[Equation]
s
[Equation]
, where s t +1 ∼ P ( s t , π ( s t + δ t )) , s 0 ∼ d 0 .
The randomness of J ( π ) arises from the environment dynamics, while that of J ε ( π ) includes additional randomness from the perturbations { δ t } .
Since for a sufﬁciently large ﬁnite horizon H we can approximate J ( π ) and J ε ( π ) to arbitrary precision, we henceforth focus on a ﬁnite horizon H .
Deﬁnition 3 (Robustness certiﬁcation for cumulative reward) .
The robustness certiﬁcation for cumulative reward is the lower bound of perturbed cumulative reward J such that J ≤ J ε ( π ) under perturbation in B ε = { δ ∈ R n | (cid:107) δ (cid:107) 2 ≤ ε } applied to all time steps.
We will provide details on the certiﬁcation of per-state action in Section 4 and the certiﬁcation of cumulative reward in Section 5 based on different smoothing strategies and certiﬁcation methods.
R OBUSTNESS C ERTIFICATION S TRATEGIES FOR P ER -S TATE A CTION
In this section, we discuss the robustness certiﬁcation for per-state action , aiming to calculate a lower bound of maximum perturbation magnitude ¯ ε in Deﬁnition 1.
4. C ERTIFICATION FOR P ER -S TATE A CTION VIA A CTION -V ALUE F UNCTIONAL S MOOTHING
Let Q π be the action-value function given by the trained network Q with policy π .
We derive a smoothed function (cid:101) Q π through per-state local smoothing .
Speciﬁcally, at each time step t , for each action a ∈ A , we draw random noise from a Gaussian distribution N (0 , σ 2 I N ) to smooth Q π ( · , a ) .
[Equation]
Lemma 1 (Lipschitz continuity of the smoothed value function) .
Given the action-value function
Q π : S × A → [ V min , V max ] , the smoothed function (cid:101) Q π with smoothing parameter σ is L-Lipschitz continuous with L = V max − V min σ (cid:112) 2 / π w.r.t.
the state input.
The proof is given in Appendix A.1.
Leveraging the Lipschitz continuity in Lemma 1, we derive the following theorem for certifying the robustness of per-state action.
Theorem 1.
Let Q π : S ×
A → [ V min , V max ] be a trained value network, (cid:101)
Q π be the smoothed func- tion with (2).
At time step t with state s t , we can compute the lower bound r t of maximum perturbation magnitude ¯ ε ( s t ) (i.e., r t ≤ ¯ ε ( s t ) , ¯ ε deﬁned in Deﬁnition 1) for locally smoothed policy ˜ π :
[Equation]
where Φ − 1 is the inverse CDF function, a 1 is the action with the highest (cid:101) Q
π value at state s t , and a 2 is the runner-up action.
We name the lower bound r t as certiﬁed radius for the state s t .
The proof is omitted to Appendix A.2.
The theorem provides a certiﬁed radius r t for per-state action given smoothed policy: As long as the perturbation is bounded by r t , i.e. , (cid:107) δ t (cid:107) 2 ≤ r t , the action does not change: ˜ π ( s t + δ t ) =
˜
π
( s t ) .
To achieve high certiﬁed robustness for per-state action, Theoremimplies a tradeoff between value function smoothness and the margin between the values of top two actions: If a larger smoothing parameter σ is applied, the action-value function would be smoother and therefore more stable; however, it would shrink the margin between the top two action values leading to smaller certiﬁed radius.
Thus, there exists a proper smoothing parameter to balance the tradeoff, which depends on the actual environments and algorithms.
4. CROP-L O A CT : L OCAL R ANDOMIZED S MOOTHING FOR C ERTIFYING P ER -S TATE A CTION
Next we introduce the algorithm to achieve the certiﬁcation for per-state action.
Given a Q π network, we apply (2) to derive a smoothed network (cid:101) Q π .
At each state s t , we obtain the greedy action ˜ a t w.r.t.
(cid:101) Q π , and then compute the certiﬁed radius r t .
We present the complete algorithm in Appendix B.1.
There are some additional challenges in smoothing the value function and computing the certiﬁed radius in Q-learning compared with the standard classiﬁcation task (Cohen et al., 2019).
Challenge 1:
In classiﬁcation, the output range of the conﬁdence [0 , 1] is known a priori; however, in Q-learning, for a given Q π , its range [ V min , V max ] is unknown.
Challenge 2:
In the classiﬁcation task, the lower and upper bounds of the top two classes’ prediction probabilities can be directly computed via the conﬁdence interval base on multinomial proportions (Goodman, 1965).
For Q-networks, the outputs are not probabilities and calculating the multinomial proportions becomes challenging.
Pre-processing.
To address Challenge 1 , we estimate the output range [ V min , V max ] of a given network Q π based on a ﬁnite set of valid states S sub ⊆ S .
In particular, we craft a sufﬁciently large set S sub to estimate V min and V max for Q π on S , which can be used later in per-state smoothing. The details for estimating V min and V max are deferred to Appendix B.1.
Certiﬁcation.
To smooth a given state s t , we use Monte Carlo sampling (Cohen et al., 2019) to sample noise applied to s t , and then estimate the corresponding smoothed value function (cid:101) Q π at s t with (2).
In particular, we sample m Gaussian noise ∆ i ∼ N (0 , σ 2 I N ) , clip the Q-network output to ensure that it falls within the range [ V min , V max ] , and then take the average of the output to obtain the smoothed action prediction based on (cid:101) Q π .
We then employ Theorem 1 to compute the certiﬁed radius r t .
We omit the detailed inference procedure to Appendix B.1.
To address Challenge 2 , we leverage
Hoeffding’s inequality (Hoeffding, 1994) to compute a lower bound of (cid:101) Q π ( s t , a 1 ) and an upper bound of (cid:101) Q π ( s t , a 2 ) with one-sided conﬁdence level parameter α given the top two actions a 1 and a 2 .
When the former is higher than the latter, we can certify a positive radius for the given state s t .
R OBUSTNESS C ERTIFICATION S TRATEGIES FOR THE C UMULATIVE R EWARD
In this section, we present robustness certiﬁcation strategies for the cumulative reward.
The goal is to provide the lower bounds for the perturbed cumulative reward in Deﬁnition 2.
In particular, we propose both global smoothing and local smoothing strategies to certify the perturbed cumulative reward.
In the global smoothing, we view the whole state trajectory as a function to smooth, which would lead to relatively loose certiﬁcation bound.
We then propose the local smoothing by smoothing each state individually to obtain the absolute lower bound.
5. C ERTIFICATION OF C UMULATIVE R EWARD BASED ON G LOBAL S MOOTHING
In contrast to Section 4 where we perform per-state smoothing to achieve the certiﬁcation for per-state action, here, we aim to perform global smoothing on the state trajectory by viewing the entire trajectory as a function.
In particular, we ﬁrst derive the expectation bound of the cumulative reward based on global smoothing by estimating the Lipschitz constant for the cumulative reward w.r.t.
the trajectories.
Since the Lipschitz estimation in the expectation bound is algorithm agnostic and could lead to loose estimation bound, we subsequently propose a more practical and tighter percentile bound .
Deﬁnition 4 ( σ -randomized trajectory and σ -randomized policy) .
Given a state trajectory ( s 0 , s 1 , . . .
, s H − 1 ) of length H where s t +1 ∼ P ( s t , π ( s t )) , s 0 ∼ d 0 , with π the greedy policy of the action-value function Q π , we derive a σ -randomized trajectory as ( s (cid:48) 0 , s (cid:48) 1 , . .
, s (cid:48)
H − 1 ) , where s (cid:48) t +1 ∼ P ( s (cid:48) t , π ( s (cid:48) t +∆ t )) , ∆ t ∼ N (0 , σ 2 I N ) , and s (cid:48)
= s 0 ∼ d 0 .
We correspondingly deﬁne a σ -randomized policy π (cid:48) based on π in the following form: π (cid:48) ( s t ) :
= π ( s t + ∆ t ) where ∆ t ∼ N (0 , σ 2 I N ) .
Let the operator ⊕ concatenates given input states or noise that are added to each state.
The sampled noise sequence is denoted by ∆ =
⊕ H − 1 t =
∆ t , where ∆ t ∼ N (0 , σ 2 I N ) .
Deﬁnition 5 (Perturbed return function) .
Let R, P, γ, d 0 be the reward function, transition function, discount factor, and initial state distribution in Deﬁnition 2.
We deﬁne a bounded perturbed return function F π : R H × N → [ J min , J max ] representing cumulative reward with potential perturbation δ :
[Equation]
where s t +1 ∼ P ( s t , π ( s t + δ t )) , s 0 ∼ d 0 .
We can see when there is no perturbation ( δ t = 0 ), F π ( ⊕ H − 1 t =0 0 ) =
J ( π ) ; when there are adversarial perturbations δ t ∈ B ε at each time step, F π ( ⊕ H − 1 t =
δ t ) =
J ε ( π ) , i.e. , perturbed cumulative reward .
Mean Smoothing: Expectation bound.
Here we propose to sample noise sequences ∆ to perform global smoothing for the entire state trajectory, and calculate the lower bound of the expected perturbed cumulative reward E ∆ [ J ε ( π (cid:48) )] under all possible (cid:96) 2 -bounded perturbations within magnitude ε .
The expectation is over the noise sequence ∆ involved in the σ -randomized policy π (cid:48) in Deﬁnition 4.
Lemma 2 (Lipschitz continuity of smoothed perturbed return function) .
Let F be the perturbed return function function deﬁned in (4), the smoothed perturbed return function (cid:101) F π is ( J max − J min ) σ
[Equation]
2 / πLipschitz continuous, where (cid:101) F π (cid:0) ⊕ H
[Equation]
t
[Equation]
(cid:1) :
= [Equation]
Theorem 2 (Expectation bound) .
Let J E =
[Equation]
Lε √ H , where L = ( J max
[Equation]
[-=
-]J
[Equation]
2 / π .
Then J E ≤ E [ J ε ( π (cid:48) )] .
Proof Sketch.
We ﬁrst derive the equality between expected perturbed cumulative reward E
[ J ε ( π (cid:48) )] and the smoothed perturbed return function (cid:101) F π ( ⊕ H − 1 t =
δ t ) .
Thus, to lower bound the former, it sufﬁces to lower bound the latter, which can be calculated leveraging the Lipschitz continuity of (cid:101) F in Lemma 2 (proved in Appendix A.3), noticing that the distance between ⊕ H − 1 t =0 0 and the adversarial perturbations ⊕ H − 1 t =
δ t is bounded by ε √ H .
The complete proof is omitted to Appendix A.4.
We obtain J min and J max in Lemma 2 from game speciﬁcations which can be loose in practice.
Thus the Lipschitz constant L estimation is coarse and mean smoothing is usually loose.
We next present a method that circumvents estimating the Lipschitz constant and provides a tight percentile bound.
Percentile Smoothing: Percentile bound.
We now propose to apply percentile smoothing to smooth the perturbed cumulative reward and obtain the lower bound of the p -th percentile of J ε ( π (cid:48) ) , where π (cid:48) is a σ -randomized policy deﬁned in Deﬁnition 4.
[Equation]
Theorem 3 (Percentile bound) .
Let J p
= (cid:101) F p (cid:48)
[Equation] (cid:0) ⊕
[Equation] , where p (cid:48) :
=
Φ (cid:0) Φ − 1 ( p ) − ε √ H / σ (cid:1) .
Then
J p ≤ the p -th percentile of J ε ( π (cid:48) ) .
The proof is provided in Appendix A.5 based on Chiang et al.
There are several other advantages of percentile smoothing over mean smoothing.
First, the certiﬁcation given by percentile smoothing is among the cumulative rewards of the sampled σ -randomized trajectories and is therefore achievable by a real-world policy, while the expectation bound is less likely to be achieved in practice given the loose Lipschitz bound.
Second, for a discrete function such as perturbed return function, the output of mean smoothing is continuous w.r.t.
σ , while the results given by percentile smoothing remain discrete.
Thus, the percentile smoothed function preserves properties of the base function before smoothing, and shares similar interpretation, e.g. , the number of rounds that the agent wins.
Third, taking p = 50% in percentile smoothing leads to the median smoothing which achieves additional properties such as robustness to outliers (Manikandan, 2011).
The detailed algorithm CROP-GR E , including the inference procedure, the estimation of (cid:101) F π , the algorithm for computing the empirical order statistics, and the conﬁguration of algorithm parameters J min , J max , are deferred to Appendix B.2.
5. C ERTIFICATION OF C UMULATIVE R EWARD BASED ON L OCAL S MOOTHING
Though global smoothing provides efﬁcient and practical bounds for the perturbed cumulative reward, such bounds are still loose as they involve smoothing the entire trajectory at once.
In this section, we aim to provide a tighter lower bound for J ε (˜ π ) by performing local smoothing .
Given a trajectory of H time steps which is guided by the locally smoothed policy ˜ π , we can compute the certiﬁed radius at each time step according to Theorem 1, which can be denoted as r 0 , r 1 , . . .
, r H − 1 .
Recall that when the perturbation magnitude ε < r t , the optimal action a t at time step t will remain unchanged.
This implies that when ε < min H − 1 t =0 r t , none of the actions in the entire trajectory will be changed, and therefore the lower bound of the cumulative reward when ε < min H − 1 t =
r t is the return of the current trajectory in a deterministic environment.
Increasing ε has two effects.
First, the total number of time steps where the action is susceptible to change will increase; second, at each time step, the action can change from the best to the runner-up or the rest.
We next introduce an extension of certiﬁed radius r t to characterize the two effects.
Theorem 4.
Let ( r 1 t , . . .
, r |A|− 1 t ) be a sequence of certiﬁed radii for state s t at time step t , where r kt denotes the radius such that if ε < r kt , the possible action at time step t will belong to the actions corresponding to top k action values of (cid:101) Q at state s t .
The deﬁnition of r t in Theorem 1 is equivalent to r 1 t here.
The radii can be computed similarly as follows:
[Equation]
where a 1 is the action of the highest (cid:101) Q value at state s t and a k +1 is the ( k + 1) -th best action.
We additionally deﬁne r 0 t ( s t ) = 0 , which is also compatible with the deﬁnition above.
We defer the proof in Appendix A.6.
With Theorem 4, for any given ε , we can compute all possible actions under perturbations in B ε .
This allows an exhaustive search to traverse all trajectories satisfying that all certiﬁed radii along the trajectory are smaller than ε .
Then, we can conclude that
J ε (˜ π ) is lower bounded by the minimum return over all these possible trajectories.
5.2. CROP-L O R E : L OCAL SMOOTHING FOR C ERTIFIED R EWARD
Given a policy π in a deterministic environment, let the initial state be s 0 , we propose CROP-L O R E to certify the lower bound of J ε (˜ π ) .
At a high-level, CROP-L O R E exhaustively explores new trajectories leveraging Theorem 4 with priority queue and effectively updates the lower bound of cumulative reward J by expanding a trajectory tree dynamically.
The algorithm returns a collection of pairs { ( ε i , J ε i ) } | C | i =1 sorted in ascending order of ε i , where | C | is the length of the collection.
For any ( ε i , J ε i ) in the collection, our algorithm ensures that as long as the perturbation magnitude ε ≤ ε i , the cumulative reward J ε (˜ π ) ≥ J ε i .
The algorithm is shown in Algorithm 3 in Appendix B.3.
Algorithm Description.
The method starts from the base case: when perturbation magnitude ε = 0 , the lower bound of cumulative reward J is exactly the benign reward.
The method then gradually increases the perturbation magnitude ε (later we will explain how a new ε is determined).
Along the increase of ε , the perturbation may cause the policy π to take different actions at some time steps, thus resulting in new trajectories.
Thanks to the local smoothing, the method leverages Theoremto ﬁgure out the exhaustive list of possible actions under current perturbation magnitude ε , and effectively explore these new trajectories by formulating them as expanded branches of a trajectory tree.
Once all new trajectories are explored, the method examines all leaf nodes of the tree and ﬁgures out the minimum reward among them, which is the new lower bound of cumulative reward J under this new ε .
To mitigate the explosion of branches, CROP-L O R E proposes several optimization tricks.
In the following, we will brieﬂy introduce the trajectory exploration and expansion and growth of
[Figure]
perturbation magnitude steps, as well as the optimization tricks. More algorithm details are deferred to Appendix B.3, where we also provide an analysis on the time complexity of the algorithm.
In trajectory exploration and expansion , CROP-L O R E organizes all possible trajectories in the form of search tree and progressively grows it.
For each node (representing a game state), leveraging
Theorem 4, we compute a non-decreasing sequence { r k ( s ) } |A|− 1 k =
representing the required perturbation radii for π to choose each alternative action.
Suppose the current ε satisﬁes r i ( s ) ≤ ε < r i +1 ( s ) , we can grow ( i +1) branches from current state s corresponding to the original action and i alternative actions since ε ≥ r j ( s ) for 1 ≤ j ≤ i .
We expand the tree branches using depth-ﬁrst search (Tarjan, 1972).
In perturbation magnitude growth , when all trajectories for perturbation magnitude ε are explored, we increase ε to seek certiﬁcation under larger perturbations.
This is achieved by preserving a priority queue (van Emde Boas, 1977) of the critical ε ’s that we will expand on.
Concretely, along the trajectory, at each tree node, we search for the possible actions and store actions corresponding to { r k ( s ) } |A|− 1 k
= i +1 into the priority queue, since these actions are exactly those need to be explored when ε grows.
We repeat the procedure of perturbation magnitude growth ( i.e. , popping out the head element from the queue) and trajectory exploration and expansion ( i.e. , exhaustively expand all trajectories given the perturbation magnitude) until the priority queue becomes empty or the perturbation magnitude ε reaches the predeﬁned threshold.
Additionally, we adopt a few optimization tricks commonly used in search algorithms to reduce the complexity of the algorithm, such as pruning and the memorization technique (Michie, 1968). More potential improvements are discussed in Appendix B.3.
E XPERIMENTS
In this section, we present evaluation for the proposed robustness certiﬁcation framework CROP.
Concretely, we apply our three certiﬁcation algorithms (CROP-L O A CT , CROP-GR E , and CROP-L O R E ) to certify six RL methods (baseline methods StdTrain (Mnih et al., 2013), GaussAug (Kos & Song, 2017), AdvTrain (Behzadan & Munir, 2017b), and SOTA methods RegPGD , RegCVX (Zhang et al., 2020), RadialRL (Oikarinen et al., 2020)) on two high-dimensional games (Pong and Freeway) and one low dimensional game (CartPole).
As a summary, we ﬁnd that (1) RegCVX, RegPGD, and RadialRL achieve high certiﬁed robustness in different games; (2) Large smoothing variance can help to improve certiﬁed robustness signiﬁcantly on Freeway, while a more careful selection of the smoothing parameter is needed in Pong; (3) For methods that demonstrate high certiﬁed robustness, our certiﬁcation of the cumulative reward is tight.
We defer the detailed descriptions of the game environments to Appendix D.1 and the introduction to the RL methods and implementation details to Appendix D.2.
More interesting results and discussions are omitted to Appendix E and our leaderboard, including results on CartPole (Appendix E.6) and Breakout (Appendix E.7) and three more algorithms (CARRL,
NoisyNet, and GradDQN).
As the ﬁrst work providing robustness certiﬁcation for RL, we expect more RL algorithms and games will be certiﬁed under our framework in future work.
6. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR P ER - STATE A CTION
In this subsection, we provide the robustness certiﬁcation evaluation for per-state action.
Experimental Setup and Metrics.
We evaluate the locally smoothed policy ˜ π derived in (2).
We follow Theorem 1 and report the certiﬁed radius r t at each step t .
We additionally compute certiﬁed ratio η at radius r , representing the percentage of time steps that can be certiﬁed, formally denoted as η r = 1 / H (cid:80) Ht =1 1 { r t ≥ r } . More details are omitted to Appendix D.4.
[Figure] [Figure]
Evaluation Results of CROP-
L O A CT .
We present the certiﬁcation comparison in Figure 1 and the benign performance under different smoothing variances in Figure 2; we omit details of certiﬁed ratio to Appendix E.1.
On Freeway , we evaluate with a large range of smoothing parameter σ up to 1 . 0 , since Freeway can tolerate large noise as shown in Figure 2.
Results under even larger σ are deferred to Appendix E.2, showing that the certiﬁed robustness can be further improved for the empirically robust methods.
From Figure 1, we see that
RadialRL consistently achieves the highest certiﬁed radius across all σ ’s. This is because RadialRL explicitly optimizes over the worst-case perturbations.
StdTrain, GaussAug, and AdvTrain are not as robust, and increasing σ will not make a difference.
In Pong , RegCVX is the most certiﬁably robust, which may be due to the hardness of optimizing the worst-case perturbation in Pong.
More interesting, all methods present similar periodic patterns for the certiﬁed radius on different states, which would inspire further robust training methods to take the “conﬁdent state” ( e.g. , when the ball is ﬂying towards the paddle) into account.
We illustrate the frames with varying certiﬁed radius in Appendix E.3.
Overall, the certiﬁed robustness of these methods largely matches empirical observations (Behzadan & Munir, 2017b;
Oikarinen et al., 2020; Zhang et al., 2020).
6. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR C UMULATIVE R EWARD
Here we will discuss the evaluation for the robustness certiﬁcation regarding cumulative reward in Section 5.
We show the evaluation results for both CROP-GR E and CROP-L O R E .
Evaluation Setup and Metrics.
We evaluate the σ -randomized policy π (cid:48) derived in Deﬁnitionfor CROP-GR E and the locally smoothed policy ˜ π derived in (2) for CROP-L O R E .
We compute the expectation bound J E , percentile bound J p , and absolute lower bound J following Theorem 2, Theorem 3, and Section 5.2.1.
To validate the tightness of the bounds, we additionally perform empirical attacks.
More rationales see Appendix D.3. The detailed parameters are omitted to Appendix D.4.
Evaluation of CROP-GR E .
We present the certiﬁcation of reward in Figure 3 and mainly focus on analyzing percentile bound.
We present the bound w.r.t.
different attack magnitude ε under different smoothing parameter σ .
For each σ , there exists an upper bound of ε that can be certiﬁed, which is positively correlated with σ .
Details see Appendix B.2.
We observe similar conclusion as in the per-state action certiﬁcation that RadialRL is most certiﬁably robust for Freeway, while RegCVX and RegPGD are most robust for Pong although the highest robustness is achieved at different σ .
Evaluation of CROP-L O R E .
In Figure 3, we note that in Freeway , RadialRL is the most robust, followed by RegCVX, then RegPGD, AdvTrain, and GaussAug.
As for Pong , RegCVX outperforms
RadialRL and RegPGD.
The remaining methods are not clearly ranked.
6. D ISCUSSION ON E VALUATION R ESULTS
Impact of Smoothing Parameter σ .
We draw similar conclusions regarding the impact of smoothing variance from the results given by CROP-L O A CT and CROP-GR E .
In Freeway , as σ increases, the robustness of StdTrain, GaussAug, and AdvTrain barely increases, while that of RegPGD,
RegCVX, and RadialRL steadily increases.
In Pong , a σ in the range 0 .
03 shall be suitable for almost all methods.
More explanations from the perspective of benign performance and certiﬁed results will be provided in Appendix E.4.
For CROP-L O R E , there are a few different conclusions.
On Freeway , it is clear that under larger attack magnitude ε , larger smoothing parameter σ always secures higher lower bound J .
In Pong , CROP-L O R E can only achieve non-zero certiﬁcation with small σ under a small range of attack magnitude, implying the difﬁculty of establishing non-trivial certiﬁcation for these methods on Pong.
We defer details on the selection of σ to Appendix E.4.
Tightness of the certiﬁcation J E , J p , and J .
We compare the empirical cumulative rewards achieved under PGD attacks with our certiﬁed lower bounds.
First, the empirical results are consistently lower bounded by our certiﬁcations, validating the correctness of our bounds.
Regarding the tightness, the improved J p is much tighter than the loose J E , supported by discussions in Section 5.1.
The tightness of J can be reﬂected by the zero gap between the certiﬁcation and the empirical result under a wide range of attack magnitude ε .
Furthermore, the certiﬁcation for RegCVX and RegPGD are quite tight for Freeway under large attack magnitudes, and RadialRL demonstrates its superiority on Freeway.
Additionally, different methods may achieve the same empirical results under attack, yet their certiﬁcations differ tremendously, indicating the importance of the robustness certiﬁcation .
Game Properties.
The certiﬁed robustness of any method on Freeway is much higher than that on
Pong, indicating that Freeway is a more stable game than Pong, also shown in Mnih et al. (2015).
Comparison of Different Certiﬁcation Criteria.
Conclusions given by our different certiﬁcation criteria are largely consistent.
We provide detailed discussions of the advantages , limitations , and possible extensions of our certiﬁcation methods in Appendix C.
R ELATED W ORK
We brieﬂy review several categories of RL methods that demonstrate empirical robustness.
Randomization methods (Tobin et al., 2017; Akkaya et al., 2019) were ﬁrst proposed to encourage exploration.
Under the adversarial training framework, Kos & Song (2017) and Behzadan & Munir (2017b) show that re-training with random noise and FGSM perturbations increases the agent’s resilience.
SA-DQN (Zhang et al., 2020) is a regularization based method that encourages the top- 1 action to stay unchanged under perturbation.
Radial-RL (Oikarinen et al., 2020) minimizes an adversarial loss function that incorporates the upper bound of the perturbed loss.
CARRL (Everett et al., 2021) computes the lower bounds of action-values under potential perturbation and selects actions according to the worst-case bound, but it relies on linear bounds (Weng et al., 2018) and is only suitable for low-dimensional environments.
These empirically robust RL methods cannot provide robustness certiﬁcation.
To the best of our knowledge, this is the ﬁrst work that provides robustness certiﬁcation for RL methods.
Broader discussions and comparisons of related work are in Appendix F.
Conclusions.
To provide the robustness certiﬁcation for RL methods, we propose a general framework CROP, aiming to provide certiﬁcation based on two criteria.
Our evaluations show that certain empirically robust RL methods are certiﬁably robust for speciﬁc games.
Ethics Statement.
In this paper, we prove the ﬁrst framework for certifying the robustness of
RL algorithms against evasion attacks.
We aim to certify and therefore potentially improve the trustworthiness of machine learning models, and we do not expect any ethics issues raised by our work.
Reproducibility Statement.
All theorem statements are substantiated with rigorous proofs in our
Appendix.
We have upload the source code as the supplementary material for reproducibility purpose.
A P ROOFS A. P ROOF OF L EMMA
We recall Lemma 1:
Lemma 1 (Lipschitz continuity of the smoothed value function) .
Given the action-value function
Q π : S × A → [ V min , V max ] , the smoothed function (cid:101) Q π with smoothing parameter σ is L-Lipschitz continuous with L = V max − V min σ (cid:112) 2 / π w.r.t.
the state input.
Proof.
To prove Lemma 1, we leverage the technique in the proof for Lemma 1 of Salman et al.
(Salman et al., 2019) in their Appendix A.
For each action a ∈ A , our smoothed value function is
[Equation]
Taking the gradient w.r
[Equation]
t
[Equation]
s
[Equation]
we obtain
[Equation]
For any unit direction u
[Equation]
we have
[Equation]
π
Thus, (cid:101) Q π is L -Lipschitz continuous with L = V max − V min σ (cid:112) 2 / π w.r
[Equation]
t. the state input.
A. P ROOF OF T HEOREM
We recall Theorem 1:
Theorem 1.
Let Q π : S ×
A → [ V min , V max ] be a trained value network, (cid:101)
Q π be the smoothed func- tion with (2).
At time step t with state s t , we can compute the lower bound r t of maximum perturbation magnitude ¯ ε ( s t ) (i.e., r t ≤ ¯ ε ( s t ) , ¯ ε deﬁned in Deﬁnition 1) for locally smoothed policy ˜ π :
[Equation]
6) where Φ − 1 is the inverse CDF function
, a 1 is the action with the highest (cid:101) Q
π value at state s t , and a 2 is the runner-up action.
We name the lower bound r t as certiﬁed radius for the state s t .
We ﬁrst present a lemma that can help in the proof of Theorem 1.
Lemma 3.
Let Φ be the CDF of a standard normal distribution, the mapping η a
( s
[Equation] [Equation]
σ ·
[Equation]
-Lipschitz continuous.
The lemma can be proved following the same technique as the proof for Lemma 1 in Appendix A.1.
The detailed proof can be referred to in the proof for Lemma 2 of Salman et al.
Salman et al.
(2019) in their Appendix A. We next show how to leverage Lemma 3 to prove Theorem 1.
Proof.
[Proof for Theorem 1] Let the perturbation be δ t , based on the Lipschitz continuity of the mapping η , we have
[Equation]
Suppose that under perturbation δ t , the action selection would be misled in the sense that the smoothed value for the original action a 1 is lower than that of another action a 2 , i.e. , (cid:101) Q
π ( s t + δ t , a 1 ) ≤ (cid:101) Q π ( s t + δ t , a 2 ) .
Then, based on the monotonicity of η , we have
[Equation]
Summing up (7), (8), and (9), we obtain
[Equation]
which is a lower bound of the maximum perturbation magnitude ¯ ε ( s t ) that can be tolerated at state s t .
Hence, when r t takes the value of the computed lower bound, it satisﬁes the condition that r t ≤ ¯ ε ( s t ) .
A. P ROOF OF L EMMA
We recall Lemma 2:
Lemma 2 (Lipschitz continuity of smoothed perturbed return function) .
Let F be the perturbed return function function deﬁned in (4), the smoothed perturbed return function (cid:101) F π is ( J max − J min ) σ (cid:112) 2 / πLipschitz continuous, where (cid:101) F π (cid:0) ⊕ H
[Equation]
1 t
[Equation]
δ
[Equation]
(cid:1) :
= [Equation]
(cid:1)
[Equation]
Proof.
The proof can be done in a similar fashion as the proof for Lemma 1 in Appendix A.1.
Compared with the smoothed value network where the expectation is taken over the sampled states, here, similarly, the smoothed perturbed return function is derived by taking the expectation over sampled σ -randomized trajectories.
The difference is that the output range of the Q-network is [ V min , V max ] , while the output range of the perturbed return function is [ J min , J max ] .
Thus, the smoothed perturbed return function (cid:101) F is ( J max − J min ) σ (cid:112)
2 / π -Lipschitz continuous.
A. P ROOF OF T HEOREM
We recall the deﬁnition of (cid:101) F π as well as Theorem 2:
[Equation]
Theorem 2 (Expectation bound) .
Let J E
=
(cid:101) F π (cid:0)
⊕
H − 1 t =0 0 (cid:1) − Lε √ H , where L =
( J max − J min ) σ (cid:112) 2 / π .
Then J E ≤ E [ J ε ( π (cid:48) )] .
Proof.
We note the following equality
[Equation]
where (a) comes from the deﬁnition of the smoothed perturbed return function (cid:101) F π , (b) is due to the deﬁnition of the σ -randomized policy π (cid:48) , and (c) arises from the deﬁnition of the perturbed cumulative reward J Hε .
Thus, the expected perturbed cumulative reward E (cid:2) J Hε ( π (cid:48) )
(cid:3) is equivalent to the smoothed perturbed return function (cid:101) F π (cid:0)
⊕ H − 1 t =
δ t (cid:1) .
Furthermore, since the distance between the all-zero ⊕ H − 1 t =0 0 and the adversarial perturbations ⊕ H − 1 t =
δ t is bounded by ε √ H , leveraging the
Lipschitz smoothness of (cid:101) F in Lemma 2, we obtain the lower bound of the expected perturbed cumulative reward E (cid:2)
J Hε ( π (cid:48) )
(cid:3) as (cid:101) F π (cid:0) ⊕ H − 1 t =0 0 (cid:1) − Lε √ H .
A. P ROOF OF T HEOREM
We recall the deﬁnition of (cid:101) F pπ as well as Theorem 3:
[Equation]
Theorem 3 (Percentile bound) .
Let J p
= (cid:101) F p (cid:48)
π (cid:0) ⊕
H − 1 t =0 0 (cid:1) , where p (cid:48) :
=
Φ (cid:0) Φ − 1 ( p ) − ε √ H / σ (cid:1) .
Then
J p ≤ the p -th percentile of J ε ( π (cid:48) ) .
Proof.
To prove Theorem 3, we leverage the technique in the proof for Lemma 2 of Chiang et al.
(Chiang et al., 2020) in their Appendix B.
For brevity, we abbreviate δ :=
⊕ H − 1 t =
δ t and redeﬁne the plus operator such that δ + ζ :=
⊕ H − 1 t =
( δ t + ζ t ) .
Then, similar to Lemma 3, we have the conclusion that
[Equation]
is 1 -Lipschitz continuous, where ζ ∼ N (0 , σ 2 I H × N ) .
Thus, under the perturbations δ t ∈ B ε for t = 0 . . .
H − 1 , we have
[Equation]
( 0 ) (cid:105)(cid:17) + ε √ H σ (Since (cid:107) δ (cid:107) 2 ≤ ε √ H )
=
Φ − 1 ( p (cid:48) )
+ ε √ H σ (By deﬁnition of (cid:101) F pπ ) =
Φ − 1 ( p ) (By deﬁnition of p (cid:48) ) .
Since Φ − 1 monotonically increase, this implies that P (cid:104) F π ( δ + ζ ) ≤ (cid:101) F p (cid:48) π ( 0 ) (cid:105) ≤ p .
According to the deﬁnition of (cid:101) F pπ in (11), we see that (cid:101) F p (cid:48) π ( 0 ) ≤ (cid:101) F pπ ( δ ) , i.e. , J p ≤ the p -th percentile of J ε ( π (cid:48) ) .
Hence, the theorem is proved.
A. P ROOF OF T HEOREM
We recall Theorem 4:
Theorem 4.
Let ( r 1 t , . . .
, r |A|− 1 t ) be a sequence of certiﬁed radii for state s t at time step t , where r kt denotes the radius such that if ε < r kt , the possible action at time step t will belong to the actions corresponding to top k action values of (cid:101) Q at state s t .
The deﬁnition of r t in Theorem 1 is equivalent to r 1 t here.
The radii can be computed similarly as follows:
[Equation]
where a 1 is the action of the highest (cid:101) Q value at state s t and a k +1 is the ( k + 1) -th best action.
We additionally deﬁne r 0 t ( s t ) = 0 , which is also compatible with the deﬁnition above.
Proof.
Replacing a 2 with a k +1 in the proof for Theorem 1 in Appendix A.2 directly leads to Theorem 4.
B A DDITIONAL D ETAILS OF C ERTIFICATION S TRATEGIES
In this section
[Table]
we cover the concrete details regarding the implementation of our three certiﬁcation strategies, as a complement to the high-level ideas introduced back in Section 4 and Section
[Table]
B. D ETAILED A LGORITHM OF CROP-L O A CT
[Table]
(cid:46) certification failure cert ← False r ← undefined 12 return (cid:101) Q π , a 1 , cert , r
We present the concrete algorithm of CROP-L O
A CT in Algorithm 1 for the procedures introduced in Section 4.2.
For each given state s t , we ﬁrst perform Monte Carlo sampling (Cohen et al., 2019;
Lecuyer et al., 2019) to achieve local smoothing.
Based on the smoothed value function (cid:101) Q π , we then compute the robustness certiﬁcation for per-state action, i.e. , the certiﬁed radius r t at the given state s t , following Theorem 1. Detailed Inference Procedure.
During inference, we invoke the model with m samples of Gaussian noise at each time step, and use the averaged Q value on these m noisy samples to obtain the greedy action selection and compute the certiﬁcation.
This procedure is similar to Predict in Cohen et al.
, but since RL involves multiple step decisions, we do not have the “abstain” decision as in Predict in Cohen et al.
i.e. , CROP-L O A CT will take the greedy action at all steps no matter whether the action can be certiﬁed or not. Estimation of the Algorithm Parameters V min , V max . Our estimate is obtained via sampling the trajectories and calculating the Q values associated with the state-action pairs along these trajectories. Since we perform clipping using the obtained bounds, the certiﬁcation is sound.
The cost for estimating V min and V max is not high given that they can be directly obtained via sampling as concretely described above.
Furthermore, we can balance the trade-off between the accuracy and efﬁciency of the estimation, since for any conﬁguration of V min and V max , the certiﬁcation will invariantly be sound (reason above).
We present the concrete algorithm of CROP-GR E in Algorithm 2 for the procedures introduced in Section 5.1.
Similarly to CROP-L O A CT , the algorithm also consists of two parts: performing smoothing and computing certiﬁcation, where we compute both the expectation bound J E and the percentile bound J p .
Step 1: Global smoothing.
We adopt Monte Carlo sampling (Cohen et al., 2019; Lecuyer et al., 2019) to estimate the smoothed perturbed return function (cid:101) F by sampling multiple σ -randomized [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
trajectories via drawing m noise sequences.
For each noise sequence ζ ∼ N (0 , σ 2 I H × N ) , we apply noise ζ t to the input state s t sequentially, and obtain the sum of the reward
J Ci = (cid:80) H − 1 t =
re t as the return for this σ -randomized trajectory.
We then aggregate the smoothed perturbed return values { J Ci } mi =1 via mean smoothing and percentile smoothing.
Step 2: Certiﬁcation for perturbed cumulative reward.
First, we compute the expectation bound
J E using Theorem 2.
Since the smoothed perturbed return function (cid:101) F is obtained based on m sampled noise sequences, we use Hoeffding’s inequality (Hoeffding, 1994) to compute the lower bound of the random variable (cid:101) F (cid:0) ⊕ H − 1 t =0 0 (cid:1) with a conﬁdence level α .
We then calculate the lower bound of (cid:101) F (cid:0) ⊕ H − 1 t =
δ t (cid:1) under all possible (cid:96) 2 -bounded perturbations δ t ∈ B ε leveraging the smoothness of (cid:101) F .
We then compute the percentile bound J p using Theorem 3.
We let J Ci be sorted increasingly, and perform normal approximations (Stein et al., 1972) to compute the largest empirical order statistic
J Ck such that P (cid:104)
J p ≥ J Ck (cid:105) ≥
1 − α .
The empirical order statistic J Ck is then used as the proxy of
J p under α conﬁdence level.
We next provide detailed explanations for C OMPUTE O RDER S TATS , which aims to compute the order k using binomial formula plus normal approximation. Estimation of (cid:101) F π (deﬁned in Lemma 2) .
For computing the expectation bound ( i.e. , the lower bound of E ∆ [ J ε ( π (cid:48) )] ), an intermediate step is to estimate (cid:101) F π .
However, the accuracy of the estimation does not inﬂuence the soundness of the lower bound calculation.
The reason is given below. The number of sampled randomized trajectories m controls the trade-off between the estimation efﬁciency and accuracy , as well as the tightness of the derived lower bound. Concretely, a small m would provide high efﬁciency, low estimation accuracy, and loose lower bound; but the lower bound is always sound , since our algorithm explicitly accounts for the inaccuracy associated with m via leveraging the Hoeffding’s inequality in line 9 in Algorithm 2. Details of C OMPUTE O RDER S TATS .
We consider the sorted sequence J C 1 ≤ J C 2 ≤ · · · ≤ J Cm .
We additionally set J C 0 =
−∞ and J C m +1 = ∞ .
Our goal is to ﬁnd the largest k such that
P [Equation]
J p ≥ J Ck (cid:105) ≥ 1 − α .
We evaluate the probability explicitly as follows:
[Equation]
Thus, the condition P (cid:104)
J p ≥ J C k (cid:105) ≥ 1 − α is equivalent to
[Equation]
Given large enough m , the LHS of (12) can be approximated via a normal distribution with mean equal to mp (cid:48) and variance equal to mp (cid:48) (1 − p (cid:48) ) .
Concretely, we perform binary search to ﬁnd the largest k that satisﬁes the constraint.
We ﬁnally explain the upper bound of ε that can be certiﬁed for each given smoothing variance.
In practical implementation, for a given sampling number m and conﬁdence level parameter α , if p (cid:48) is too small, then the condition (12) may not be satisﬁed even for k = 1 .
This implies that the existence of an upper bound of ε that can be certiﬁed for each smoothing parameter σ , recalling that p (cid:48) :
=
[Equation]
Detailed Inference Procedure.
Instead of sampling m noisy states per time step as in CROPL O A CT , we sample m randomized trajectories in CROP-GR E . For each sampled randomized trajectory, at each time step in the trajectory, we invoke the model with 1 sample of Gaussian noise. Given the m randomized trajectories and the cumulative reward for each of them, we compute the certiﬁcation using Theorem 2 and Theorem 3.
The Algorithm Parameters J min , J max .
We use the default values of J min , J max in the game speciﬁcations rather than estimate them, which are independent with the trained models.
Thus there is no computational cost at all for obtaining the two parameters.
As mentioned under Theoremin Section 5.1, using these default values may induce a loose bound in practice, so we further propose the percentile smoothing that aims to eliminate the dependency of our certiﬁcation on J min and J max , thus achieving a tighter bound.
B. D ETAILED A LGORITHMS OF CROP
-L O R E
In the following, we will explain the trajectory exploration and expansion , the growth of perturbation magnitude , and optimization tricks in details.
Trajectory Exploration and Expansion.
CROP-L O R E organizes all possible trajectories in the form of a search tree and progressively grows it.
Each node of the tree represents a state, and the depth of the node is equal to the time step of the corresponding state in the trajectory.
The root node (at depth 0 ) represents the initial state s 0 .
For each node, leveraging Theorem 4, we compute a non-decreasing sequence { r k ( s ) } |A|− 1 k =0 corresponding to required perturbation radii for π to choose each alternative action (the subscript t is omitted for brevity).
Suppose the current ε satisﬁes r i ( s ) ≤ ε < r i +1 ( s ) .
We grow ( i + 1) branches from current state s corresponding to the original action and i alternative actions since ε ≥ r j ( s ) for 1 ≤ j ≤ i .
For nodes on the newly expanded branch, we repeat the same procedure to expand the tree with depth-ﬁrst search (Tarjan, 1972) until the terminal state of the game is reached or the node depth reaches H .
As we expand, we keep the record of cumulative reward for each trajectory and update the lower bound J when reaching the end of the trajectory if necessary.
Perturbation Magnitude Growth.
When all trajectories for perturbation magnitude ε are explored, we need to increase ε to seek for certiﬁcation under larger perturbations.
Luckily, since the action space is discrete, we do not need to examine every ε ∈ R + (which is infeasible) but only need to examine the next ε where the chosen action in some step may change.
We leverage priority queue (van
Emde Boas, 1977) to effectively ﬁnd out such next “critical” ε .
Concretely, along the trajectory exploration, at each tree node, we search for the possible actions and store actions corresponding to { r k ( s ) } |A|− 1 k
= i +1 into the priority queue, since these actions are exactly those need to be explored when ε grows.
After all trajectories for ε are fully explored, we pop out the head element from the priority queue as the next node to expand and the next perturbation magnitude ε to grow.
We repeat this process until the priority queue becomes empty or the perturbation magnitude ε reaches the predeﬁned threshold.
Additional Optimization.
We adopt some additional optimization tricks to reduce the complexity of the algorithm.
First, for games with no negative reward, we perform pruning to limit the tree size—if the cumulative reward leading to the current node already reaches the recorded lower bound, we can perform pruning, since the tree that follows will not serve to update the lower bound.
This largely reduces the potential search space.
We additionally adopt the memorization (Michie, 1968) technique which is commonly applied in search algorithms.
We point out a few more potential
[Figure]
take possible actionsa _ list ← a _ list ∪ { a }else (cid:46) store impossible actions in queue for later expansion p _ que. push (( s,a,r,J cur )) return a _ list [Table]
improvements.
First , with more speciﬁc knowledge of the game mechanisms, the search algorithm can be further optimized.
Take the Pong game as an example, given the horizontal speed of the ball, we can compress the time steps where the ball is ﬂying between the two paddles, thus reducing the computation.
Second , empirical attacks may be efﬁciently incorporated into the algorithm framework to provide upper bounds that help with pruning.
Detailed Modules in Algorithm 3.
The function G ET A CTION computes the possible actions at a given state s under the limit ε , while the procedure E XPAND accomplishes the task of expanding upon a given node/state.
The main part of the algorithm involves a loop that repeatedly selects the next element from the priority queue, i.e. , a node associated with an ε value, to expand upon. Time Complexity.
The time complexity of CROP-L O R E is O ( H | S explored | × (log | S explored | + | A | T )) , where | S explored | is the number of explored states throughout the search procedure, which is no larger than cardinality of state set, H is the horizon length, | A | is the cardinality of action set, and T is the time complexity of performing local smoothing.
The main bottleneck of the algorithm is the large number of possible states, which is in the worst case exponential to state dimension.
However, to provide a sound worst-case certiﬁcation agnostic to game properties, exploring all possible states may be inevitable.
Detailed Inference Procedure and Additional Clariﬁcations.
In CROP-L O R E , the inference procedure per step is exactly the same as in CROP-L O A CT .
We obtain the certiﬁcation using the adaptive search algorithm and Theorem 4.
We clarify that the algorithm only requires access to the environment such that it can obtain the reward and next state via interacting with the environment E ( i.e. , taking action a at state s and obtain next action s (cid:48) and reward r ), but does not require access to an oracle transition function Γ .
We further emphasize that access to the environment that supports back-tracking is already sufﬁcient for conducting our adaptive search algorithm.
C D ISCUSSION ON THE C ERTIFICATION M ETHODS
We discuss the advantages and limitations of our certiﬁcation methods, as well as possible direct extensions , hoping to pave the way for future research along similar directions.
CROP-L O A CT .
The algorithm provides state-wise robustness certiﬁcation in terms of the stability/consistency of the per-state action.
It treats each time step independently , smooths the given state at the given time step, and provides the corresponding certiﬁcation.
Thus, one potential extension is to expand the time window from one time step to a consecutive sequence of several time steps, and provide certiﬁcation for the sequences of actions for the given window of states.
CROP-GR E .
As explained in Section 5, the expectation bound J E is too loose to have any practical usage.
In comparison, percentile bound J p is much tighter and practical.
However, one limitation of J p is that there exists an upper bound of the attack magnitude ε that can be certiﬁed for each σ , as explained in Appendix B.2.
For attack magnitudes that exceed the upper bound, we can obtain no useful information via this certiﬁcation (though the upper bound is usually sufﬁciently large).
One drawback of the CROP-GR E certiﬁcation is that Lemma 2 requires knowing the noise added to each step beforehand so as to generate m randomized trajectories given the known noise sequence { ζ t } H − 1 t =0 .
Thus, it cannot be applied against an adaptive attacker.
We note that our CROP-L O A CT and CROP-L
O R E does not have this limitation.
CROP-L O R E .
The algorithm provides the absolute lower bound J of the cumulative reward for any ﬁnite-horizon trajectory with a given initial state.
The advantage of the algorithm is that J is an absolute lower bound that bounds the worst-case situation (apart from the exceptions due to the probabilistic conﬁdence α ), rather than the statistical lower bounds J E and J p that characterize the statistical properties of the random variable J ε .
We clarify one potential pitfall in understanding J : what CROP-L O R E certiﬁes is the lower bound for the trajectory with a given initial state s 0 ∼ d 0 , rather than all possible states in d 0 .
This is because our search starts from a root node of the tree, which is set to the ﬁxed given state s 0 .
Regarding the conﬁdence of our probabilistic certiﬁcation, we clarify that we consider the independent multiple-test.
Concretely, due to the independence of decision making errors, the conﬁdence is (1 − α ) N , where
N is the maximum number of possible attacked states explored by CROP-L O R E .
We remark that in practice, the attacker usually has the ability to perturb only a limited number of steps, i.e. , N is typically small.
Therefore, the conﬁdence is non-trivial.
The main limitation is the high time complexity of the algorithm (see Appendix B.3).
The algorithm has exponential time complexity in worst case despite the existence of several optimizations.
Therefore, it is not suitable for games with a large action set, or when the horizon length is set too long.
Extension to Policy-based Methods. Though we speciﬁcally study the robustness certiﬁcation in Q-learning in this paper, our two certiﬁcation criteria and three certiﬁcation strategies can be readily extended to policy-based methods.
The intuition is that, instead of smoothing the value function in the Q-learning setting, we directly smooth the policy function in policy-based methods.
With the smoothing module replaced and the theorems updated, other technical details in the algorithms for certifying the per-state action and the cumulative reward would then be similar.
D A DDITIONAL E XPERIMENTAL D ETAILS D. D ETAILS OF THE G AME E NVIRONMENT
We experiment with two Atari-2600 games in OpenAI Gym (Brockman et al., 2016) on top of the
Arcade Learning Environment (Bellemare et al., 2013).
The states in the games are high dimensional color images ( 210 × 160 × 3 ) and the actions are discrete actions that control the agent to accomplish certain tasks.
Concretely, we use the NoFrameskip-v4 version for our experiments, where the randomness that inﬂuences the environment dynamics can be fully controlled by setting the random seed of the environment at the beginning of an episode.
D. D ETAILS OF THE RL M ETHODS
We introduce the details of the six RL methods we evaluated.
Speciﬁcally, for each method, we ﬁrst discuss the algorithm, and then present the implementation details for it.
StdTrain (Mnih et al., 2013): StdTrain (naturally trained DQN model) is an algorithm based on
Q-learning, in which DQN is used to reprent Q-value function and TD-loss is used to optimize DQN.
Our StdTrain model is implemented with Double DQN (Van Hasselt et al., 2016) and Prioritized
Experience Replay Schaul et al. (2015).
GaussAug (Behzadan & Munir, 2017b):
The algorithm of GaussAug is similar with AtdTrain, in which appropriate Gaussian random noises added to states during training.
As for implementation, our GaussAug model is based on the same architecture with the same set of basic training techniques of StdTrain, and adds Gaussian random noise with σ = 0 .
005 , which has the best testing performance under attacks compared with other σ , to all the frames during training.
AdvTrain (Behzadan & Munir, 2017b): Instead of using the original observation to train Double
DQN (Van Hasselt et al., 2016), AdvTrain generates adversarial perturbations and applies the perturbations to part of or all of the frames when training.
In our case, we add the perturbations generated by 5 -step PGD attack to 50% of the frames to make a balance between between stable training and effectiveness of adversarial training.
RegPGD and RegCVX (Zhang et al., 2020): Instead of utilizing adversarial noise to train models directly like AdvTrain, RegPGD and RegCVX regularize the loss function with the help of adversarial noise during training in order to train empirically robust reinforcement learning agents.
In our experiment settings, which is the same as RegPGD and RegCVX (Zhang et al., 2020), models consider adversarial noise with (cid:96) ∞ -norm ε
= 1 / 255 when solving the maximization for the regularizer and minimize the original TD-loss concurrently.
RadialRL (Oikarinen et al., 2020):
With the similar idea of adding regularization during training of
RegPGD and RegCVX, RadialRL compute the worst-case loss instead of regularizing of the bound of the difference of policy distribution under original states and adversarial states.
In our case, the
RadialRL model uses a linearly increasing perturbation magnitude ε (from 0 to 1 / 255 ) to compute the adversarial loss.
CARRL (Everett et al., 2021): Unlike other methods which try to improve robustness of RL-agents during training, CARRL enhances RL-agents’ robustness during testing time.
It computes the lower bound of each action’s Q value at each step and take the one with the highest lower bound conservatively.
Our CARRL model evaluated in the environment of CartPole uses ε = 0 .
1 when computing the lower bound of Q value, which has the best empirical performance under noises and attacks.
NoisyNet (Fortunato et al., 2017): Instead of the conventional exploration heuristics for DQN and
Dueling agents which is ε -greedy algorithms, NoisyNet adds parametric noise to DQN’s weights, which is claimed to aid efﬁcient exploration and yield higher scores than StdTrain.
Our NoisyNet with the same basic architecture of StdTrain model adds Gaussian random noise with σ = 0 .
5 to its weights and samples during training and testing time.
GradDQN (Pattanaik et al., 2018): GradDQN is a variation of AdvTrain, which utilizes conditional value of risk (CVaR) as optimization criteria for robust control instead standard expected long term return.
This method maximizes the expected return over worst α percentile of returns, which can prevent the adversary from possible bad states.
In our case, we generate the adversarial states with 10-step attack by calculating the direction of gradient and sampling in that direction from beta 1 ,distribution in each step.
For StdTrain, RegPGD, RegCVX, RadialRL, we directly evaluate the trained models provided by the authors 1 .
D. R ATIONALES FOR S ETTINGS AND E XPERIMENTAL D ESIGNS
Finite Horizon Setting.
In this paper, we focus on a ﬁnite horizon setting, mainly for the evaluation purpose—we would need to compute the (certiﬁed and empirical) cumulative rewards from ﬁnite rollouts.
We clarify that our criteria, theorem, and algorithm are applicable to the inﬁnite horizon case as well.
The Role of Practical Attacks in Our Evaluation.
The goal of this paper is to provide certiﬁcation that comes with theoretical guarantees, and therefore following the standard certiﬁed robustness literatures (Cohen et al., 2019; Li et al., 2020; Salman et al., 2019; Jeong & Shin, 2020) there is no need to evaluate the empirical attacks, since the certiﬁcation always holds as long as the attacks satisfy certain conditions regardless of the actual attack algorithms.
The attacks we evaluated in our paper were only for the demonstration purpose and in the hope of providing an example.
D. D ETAILED E VALUATION S ETUP
Evaluation Setup for CROP-L O A CT .
We report results averaged over 10 episodes and set the length of the horizon H = 500 .
At each time step, we sample m = 10 , 000 noisy states for smoothing.
When applying Hoeffding’s inequality, we adopt the conﬁdence level parameter α = 0 .
Since the input state observations for the two Atari games are in image space, we rescale the input states such that each pixel falls into the range [0 , 1] .
When adding Gaussian noise to the rescaled states, we sample Gaussian noise of zero mean and different levels of variances.
Concretely, the standard deviation σ is selected among { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 03 , 0 . 05 , 0 . 1 , 0 . 5 , 0 .
We evaluate different parameters for different games.
Evaluation Setup for CROP-GR E .
We sample m = 10 , 000 σ -randomized trajectories, each of which has length H = 500 , and conduct experiments with the same set of smoothing parameters as in the setup of CROP-L O A CT .
When accumulating the reward, we set the discount factor γ = 1 . 0 with no discounting.
We take α = 0 .
as the conﬁdence level when applying Hoeffding’s inequality in the expectation bound J E , and α = 0 .
as the conﬁdence level for C OMPUTE O RDER S TATS in the percentile bound J p .
For the validation of tightness, concretely, we carry out a 10 -step PGD attack with (cid:96) 2 radius within the perturbation bound at all time steps during testing to evaluate the tightness of our certiﬁcation.
Evalution Setup for CROP-L O R E .
We set the horizon length as H = 200 and sample m = 10 , 000 noisy states with the same set of smoothing variance as in the setup of CROP-L O A CT .
Similarly, we adopt γ = 1 .
as the discount factor and α = 0 .
as the conﬁdence level when applying Hoeffding’s inequality.
When evaluating Freeway, we modify the reward mechanism so that losing one ball incurs a zero score rather than a negative score.
This modiﬁcation enables the pruning operation and therefore facilitates the search algorithm, yet still respects the goal of the game.
We also empirically attack the policy ˜ π to validate the tightness of the certiﬁcation via the same PGD attack as described above.
For each set of experiments, we run the attack one time with the same initial state as used for computing the lower bound in Algorithm 3.
D. E XPERIMENTAL S ETUP FOR C ART P OLE
We additionally experiment with CartPole-v0 in OpenAI Gym (Brockman et al., 2016) on top of the
Arcade Learning Environment (Bellemare et al., 2013).
We introduce the experimental setup below.
Details of the CartPole Game Environment.
The state in the CartPole game is a low dimensional vector of length 4 and the action ∈ { move right , move left } .
The goal of the game is to balance the rod on the cart, and one reward point is earned at each timestep when the rod is upright.
Implementation Details of the RL Methods on CartPole.
In addition to the previous 8 RL
Methods in Pong and Freeway, we evaluate another RL algorithm CARRL (Everett et al., 2021), which is claimed to be robust in low dimensional games like CartPole.
This method relies on linear bounds output by the Q network and selects the action whose lower bound of the Q value is the highest.
Evaluation Setup for CROP-L O A CT on CartPole.
We report results averaged over 10 episodes and set the length of the horizon H = 200 .
At each time step, we sample m = 10 , 000 noisy states [Table] [Figure] [Figure]
for smoothing. When applying Hoeffding’s inequality, we adopt the conﬁdence level parameter α = 0 .
We do not performing rescaling on the state observations.
When adding Gaussian noise to states,we sample Gaussian noise of zero mean and different levels of variances.
Concretely, the standard deviation σ is selected among { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 03 , 0 . 05 , 0 . 1 } .
Evaluation Setup for CROP-GR E on CartPole.
We sample m = 10 , 000 σ -randomized trajectories, each of which has length H = 200 , and conduct experiments with the same set of smoothing parameters as in the setup of CROP-L O A CT .
All other experiment settings are the same as Pong and
Freeway in Appendix D.4.
Evalution Setup for CROP-L O R E on CartPole.
Since the reward of CartPole game is quite dense, we set the horizon length as H = 10 and sample m = 10 , 000 noisy states with the same set of smoothing variance as in the setup of CROP-L O A CT .
All other experiment settings are the same as Pong and Freeway in Appendix D.4.
E A DDITIONAL E VALUATION R ESULTS AND D ISCUSSIONS E. R OBUSTNESS C ERTIFICATION FOR P ER - STATE A CTION – C ERTIFIED R ATIO
We present the robustness certiﬁcation for per-state action in terms of the certiﬁed ratio η r w.r.t.
the certiﬁed radius r in Figure 4.
From the ﬁgure, we see that RadialRL is the most certiﬁably robust method on Freeway, followed by RegCVX and RegPGD; while on Pong, RegCVX is the most robust.
These conclusions are highly consistent with those in Section 6.1 as observed from Figure 1.
Comparing the curves for Freeway and Pong, we note that Freeway not only achieves larger certiﬁed radius overall , it also more frequently attains large certiﬁed radius.
E. R ESULTS OF F REEWAY UNDER L ARGER S MOOTHING P ARAMETER σ
Since the robustness of the methods RadialRL, RegCVX, and RegPGD on Freeway have been shown to improve with the increase of the smoothing parameter σ in Section 6, we subsequently evaluate their performance under even larger σ values.
We present the certiﬁcation results in Figure 5 and their benign performance in Figure 6.
[Figure]
First of all, as Figure 6 reveals, RadialRL, RegCVX, and
RegPGD can tolerate quite large noise—the benign performance of these methods does not drop even for σ up to 4 . 0 .
For the remaining three methods, the magnitude of noise they can tolerate is much smaller, as already presented in Figure 2 in the main paper.
On the other hand, although all the three methods RadialRL, RegCVX, and
RegPGD invariably attain good benign performance as σ grows, their certiﬁed robustness does not equally increase.
As Figure 5 shows, the certiﬁed robustness of RegPGD increases the fastest, while that of RadialRL drops a little.
The observation corresponds exactly to our discussion regarding the tradeoff between value function smoothness and the margin between the values to the top two actions in Section 4.
E. P ERIODIC P ATTERNS FOR P ER -S TATE R OBUSTNESS
We speciﬁcally study the periodic patterns for per-state robustness in Pong and present the results in Figure 7.
In the game frames, our agent controls the green paddle on the right.
Whenever the opponent (the orange paddle on the left) misses catching a ball, we earn a point.
We present two periods (frame 320-400 and frame 400-480) where our agent each earns a point in the ﬁgure above.
The total points increase from 2 to 3 , and from 3 to 4 , respectively.
We note that Pong is a highly periodic game, with each period consisting of the following four game stages corresponding to distinct stages of the certiﬁed robustness : 1) A ball is initially ﬁred and set ﬂying towards the opponent.
In this stage, the certiﬁed radius remains low, since our agent does not need to take any critical action.
The opponent catches the ball and reverses the direction of the ball.
In this stage, the certiﬁed radius gradually increases, since the moves of our agent serve as the preparation to catch the ball.
The ball bounces from the ground, ﬂies towards our agent, gets caught, and bounces back.
In this stage, the certiﬁed radius increases rapidly, since every move is critical in determining whether we will be able to catch the ball, i.e. , whether our point (action value) will increase.
The opponent misses the ball, and the ball disappears from the game view.
In this stage, the certiﬁed radius drastically plummets to a low value, since there is no clear signal ( i.e. , ball) in the game for the agent to take any determined action.
The illustration not only helps us understand the semantic meaning of the certiﬁed radius, but can further provide guidance on designing empirically robust RL algorithms that leverage the information regarding the high/low certiﬁed radius of the states.
E. D ETAILED D ISCUSSIONS AND S ELECTION OF S MOOTHING P ARAMETER σ
We ﬁrst take Freeway as an example to concretely explain the impact of smoothing variance from the perspective of benign performance and certiﬁed results, and then introduce the guidelines for the selection of σ based on different certiﬁcation criteria.
On Freeway , as σ increases, the benign performance of the locally smoothed policy ˜ π of StdTrain,
GaussAug, and AdvTrain decrease rapidly in Figure 2.
Also, the certiﬁed radius of these methods barely gets improved and remains low even as σ increases.
For the percentile bound J p , these three methods similarly suffer a drastic decrease of cumulative reward as σ grows.
This is however not the case for RegPGD, RegCVX, and RadialRL, for which not only the benign performance do not experience much degradation for σ as large as 1 . 0 , but the certiﬁed radius even steadily increases.
In addition, in terms of the percentile bound J p , as σ grows, even we increase the attack ε simultaneously,
J p almost does not decrease.
This indicates that larger smoothing variance can bring more robustness to RegPGD, RegCVX, and RadialRL.
CROP-L O A CT .
We discuss how to ﬁnd the sweet spot that enables high benign performance and robustness simultaneously.
As σ increases, the benign performance of different methods generally decreases due to the noise added, while not all methods decrease at the same rate.
On Freeway ,
StdTrain and GaussAug decrease the fastest, while RegPGD, RegCVX, and RadialRL do not degrade much even for σ as large as 4 . 0 .
Referring back to Figure 1 and Figure 5, we see that the certiﬁed radius of StdTrain and GaussAug remains low even though σ increases, thus the best parameters for these two methods are both around 0 .
For RegPGD and RegCVX, their certiﬁed radius instead steadily increases as σ increases to 4 . 0 , indicating that larger smoothing variance can bring more robustness to these two methods without sacriﬁcing benign performance, offering certiﬁed radius larger than 2 . 0 .
As to RadialRL, its certiﬁed radius reaches the peak at σ = 1 .
and then decreases under larger σ .
Thus, 1 . 0 is the best smoothing parameter for RadialRL which offers a certiﬁed radius of 1 . 6 .
Generally speaking, it is feasible to select an appropriate smoothing parameter σ for all methods to increase their robustness, and robust methods will beneﬁt more from this by admitting larger smoothing variances and achieving larger certiﬁed radius.
CROP-GR E .
We next discuss how to leverage Figure 3 to assist with the selection of σ .
For a given ε or a known range of ε , we can compare between the lower bounds corresponding to different σ and select the one that gives the highest lower bound.
For example, if we know that ε is small in advance, we can go for a relatively small σ which retains a quite high lower bound.
If we know that ε will be large, we will instead choose among the larger σ ’s, since a larger ε will not fall in the certiﬁable ranges of smaller σ ’s.
According to this guideline, we are able to obtain lower bounds of quite high value for RadialRL, RegCVX, and RegPGD on Freeway under a larger range of attack ε .
CROP-L O R E .
Apart from the conclusion that larger smoothing parameter σ often secures higher lower bound J , we point out that smaller smoothing variances may be able to lead to a higher lower bound than larger smoothing variances for a certain range of ε .
This is almost always true for very small ε since smaller σ is sufﬁcient to deal with the weak attack without sacriﬁcing much empirical performance, e.g. , in Figure 3, when ε = 0 .
001 , RegCVX on Freeway can achieve J = 3 at σ = 0 .
005 while only J = 1 for large σ .
This can also happen to robust methods at large ε , e.g. , when ε = 1 . 2 , RadialRL on Freeway achieves J = 2 at σ = 0 .
75 while only J = 1 at σ = 1 . 0 .
Another case is when σ is large enough such that further increasing σ will not bring additional robustness, e.g. , in Figure 5, when ε = 1 . 5 , RadialRL on Freeway achieves J = 1 at σ = 1 . 5 while only J =at σ = 4 . 0 .
E. C OMPUTATIONAL C OST
Our experiments are conducted on GPU machines, including GeForce RTX 3090, GeForce RTX 2080 Ti, and GeForce RTX 1080 Ti.
The running time of m = 10000 forward passes with sampled states for per-state smoothing ranges from 2 seconds to 9 seconds depending on the server load.
Thus, for one experiment of CROP-L O A CT with trajectory length H = 500 and 10 repeated runs, the running time ranges from 2 .
5 hours to 12 .
5 hours.
CROP-GR E is a little more time-consuming than
CROP-L O A CT , but the running time is still in the same magnitude.
For trajectory length H = [Table] and sampling number m = 10000 , most of our experiments ﬁnish within 3 hours.
For CROP-L O R [Table] speciﬁcally, the running time depends on the eventual size of the search tree, and therefore differs tremendously for different methods and different smoothing variances, ranging from a few hours toto 5 days.
E. D ISCUSSION ON E VALUATION R ESULTS OF C ART P OLE
We provide the evaluation results for CROP on CartPole, comparing nine RL algorithms: the six algorithms evaluated in our main paper (StdTrain, GaussAug, AdvTrain, RegPGD, RegCVX, and
RadialRL), as well as three additional algorithms (CARRL, NoisyNet, and GradDQN).
The detailed descriptions of these algorithms are provided in Appendix D.2.
We present the evaluation results in Figure 8.
Impact of Smoothing Parameter σ .
In CartPole game, we draw similar conclusions regarding the impact of smoothing variance from the results given by CROP-L O
A CT , CROP-GR E and CROPL O R E .
For small σ , different methods are indistinguishable.
As σ increases, CARRL demonstrates its advantage as σ increases, but AdvTrain and GradDQN tolerate large noise.
One of differences when evaluating with CROP-GR E and CROP-L O R E is that RadialRL can hold a high performance and even sometimes can be better than CARRL.
Another one is that the lower bound for GradDQN is lower than all other methods in CROP-L O R E .
Tightness of the certiﬁcation J E , J p and J .
In CartPole, we also compare the empirical cumulative rewards achieved under PGD attacks with our certiﬁed lower bounds J E , J p and J .
Demonstrated by Figure 8c, ﬁrst of all, the correctness of our bounds is validated because the empirical results are consistently lower bounded by our certiﬁcations.
Compared with the loose expectation bound J E , the improved percentile bound J p is much tighter.
Compared with these two methods, the absolute lower bound J is even tighter, especially noticing the zero gap between the certiﬁcation and the empirical result when the attack magnitude is not too large.
Finally, methods with the same empirical results under attack may achieve very different certiﬁed lower bounds J E , J p and J , showing the importance of certiﬁcation.
Game Properties.
Different from Pong and Freeway, CartPole is a game with low dimension states , which leads to some different observations.
First, StdTrain can tolerate noise well in CartPole.
Second, CARRL demonstrates its advantage in low dimensional game, which is consistent with the empirical observations in Everett et al.
E.
M ORE R ESULTS FOR CROP-L O R E ON A TARI G AME B REAKOUT
We provide the results for CROP-L O R E on a more complicated Atari game Breakout. Concretely, we present the certiﬁed cumulative reward lower bound computed via our CROP-L O R E algorithm in Figure 9, as well as the empirical cumulative reward obtained under practical attacks in Table 1. We obtain similar conclusions based on the new game and will describe the detailed observations as below. First , Figure 9 shows that GaussAug is the most certiﬁably robust, followed by AdvTrain; while StdTrain and RegPGD are shown to be non-robust for this game, whose lower bounds remainfor all smoothing parameters σ we evaluate.
Second , the robustness ranking given by our certiﬁcation
[Figure]
[Figure]
[Figure]
[Figure]
is aligned with the ranking under practical attacks.
From Table 1, we see that GaussAug can achieve much higher cumulative reward under larger perturbation magnitudes ε compared with other RL methods, which is consistent with the conclusion drawn from Figure 9.
Third , our certiﬁcation is relatively tight, especially for larger perturbation magnitudes.
Notably, when σ = 0 . 05 , our lower bounds exactly match the attack results.
F A B ROADER D ISCUSSION ON R ELATED W ORK F. E VASION A TTACKS IN RL
We consider the adversarial attacks on state observations, where the attacker aims to add perturbations to the state during test time, so as to achieve certain adversarial goals , such as misleading the action selection and minimizing the cumulative reward .
We discuss the related works that fall into these two categories below.
Misleading the Action Selection.
It has been shown that different methods—applying random noise to simply interfere with the action selection Kos & Song (2017), or adopting adversarial attacks ( e.g. , FGSM (Goodfellow et al., 2014) and CW attacks (Carlini & Wagner, 2017)) to deliberately alter the probability of action selection—are effective to different degrees.
More concretely, when manipulating the action selection probability, some works aim to reduce the probability of selecting the optimal action (Huang et al., 2017; Kos & Song, 2017; Behzadan & Munir, 2017b), while others try to increase the probability of selecting the worst action (Lin et al., 2017; Pattanaik et al., 2018).
Minimizing the Cumulative Reward.
ATLA (Zhang et al., 2021) and PA-AD (Sun et al., 2021) consider an optimal adversary under the SA-MDP framework (Zhang et al., 2020), which aims to lead to minimal value functions under bounded state perturbations.
To ﬁnd this optimal adversary ( i.e. , the optimal adversarial state perturbation), ATLA (Zhang et al., 2021) proposes to train an adversary whose action space is the perturbation set in the state space, while PA-AD (Sun et al., 2021) further decouples the problem of ﬁnding state perturbations into ﬁnding policy perturbations plus ﬁnding the state that achieves the lowest value policy, thus addressing the challenge of large state space.
F. R OBUST RL
Distributionally Robust RL.
Nilim & El Ghaoui (2005) and Iyengar (2005) consider the problem of distributionally robust RL, where there is normally an uncertainty set with prior information regarding the distribution of the uncertain parameters.
Iyengar (2005) directly put constraints on environmental dynamics and reward functions by assuming that they take values in an uncertainty set.
Another line of research assumes the environmental dynamics and reward function as the uncertain parameters and are sampled from an uncertain distribution in the uncertain set.
The uncertainty set is in general state-wise independent, i.e. , “s-rectangularity” in Nilim & El Ghaoui (2005).
Both types aim to derive a policy by solving a max-min problem, with the former taking the minimum directly over the parameters, while the latter taking the minimum over the distribution.
Empirically Robust RL against Evasion Attacks.
We have brieﬂy introduced related work on empirically robust RL in Section 7 in the main paper.
We emphasize two main differences between the contributions of these works and our CROP.
First , these robust RL methods only provide empirical robustness against perturbed state inputs during test time, but cannot provide theoretical guarantees for the performance of the trained models under any bounded perturbations; while our CROP framework can provide practically computable certiﬁed robustness w.r.t.
two robustness certiﬁcation criteria (per-state action stability and cumulative reward lower bound), i.e. , for a given RL algorithm, we can compute certiﬁcations for it that indicate its robustness.
Second , most of these Robust RL methods focus on only the per-state decision, while we additionally consider the certiﬁcation of trajectories to obtain the lower bound of cumulative rewards.
Below, we provide a more comprehensive review of the categories of RL methods that demonstrate empirical robustness against evasion attacks.
Randomization methods (Tobin et al., 2017; Akkaya et al., 2019) were ﬁrst proposed to encourage exploration.
This type of method was later systematically studied for its potential to improve model robustness.
NoisyNet (Fortunato et al., 2017) adds parametric noise to the network’s weight during training, providing better resilience to both training-time and test-time attacks (Behzadan & Munir,
2017b; 2018), also reducing the transferability of adversarial examples, and enabling quicker recovery with fewer number of transitions during phase transition.
Under the adversarial training framework, Kos & Song (2017) and Behzadan & Munir (2017b) show that re-training with random noise and FGSM perturbations increases the resilience against adversarial examples.
Pattanaik et al.
(2018) leverage attacks using an engineered loss function speciﬁcally designed for RL to signiﬁcant increase the robustness to parameter variations.
RS-DQN (Fischer et al., 2019) is an imitation learning based approach that trains a robust student-DQN in parallel with a standard DQN in order to incorporate the constrains such as SOTA adversarial defenses (Madry et al., 2017; Mirman et al., 2018).
SA-DQN (Zhang et al., 2020) is a regularization based method that adds regularizers to the training loss function to encourage the top- 1 action to stay unchanged under perturbation.
Built on top of the neural network veriﬁcation algorithms (Gowal et al., 2018;
Weng et al., 2018),
Radial-RL (Oikarinen et al., 2020) proposes to minimize an adversarial loss function that incorporates the upper bound of the perturbed loss, computed using certiﬁed bounds from veriﬁcation algorithms.
CARRL (Everett et al., 2021) aims to compute the lower bounds of action-values under potential perturbation and select actions according to the worst-case bound, but it relies on linear bounds (Weng et al., 2018) and is only suitable for low-dimensional environments.
F. R OBUSTNESS C ERTIFICATION FOR RL
Despite the abundant literature in robustness certiﬁcation in supervised learning, there is almost no work on robustness certiﬁcation for RL, given the unclear criteria and the intrinsic difﬁculty of the task.
In this part, we discuss another concurrent work Kumar et al. (2021), which also aims to provide provable robustness for RL. In terms of the threat model , both Kumar et al. and CROP consider the type of adversary that can perturb the state observations of the agent during test time.
In our work, we consider a perturbation budget per time step following previous works (Huang et al., 2017; Behzadan & Munir, 2017b; Kos & Song, 2017; Pattanaik et al., 2018; Zhang et al., 2020), while they assume a budget for the entire episode.
The two perspectives are closely related.
Regarding the certiﬁcation criteria , we certify both per-state action stability and cumulative reward lower bound.
Although they also consider the cumulative reward in their certiﬁcation goal, they formulate the certiﬁcation as a classiﬁcation problem—certifying whether the cumulative reward is above a threshold or not, in contrast to directly certifying the lower bound as in our case.
For the certiﬁcation technique , both works are developed based on randomized smoothing proposed in supervised learning (Cohen et al., 2019).
We propose a global smoothing technique (CROP-GR E ), as well as an adaptive search algorithm coupled with local smoothing (CROP-L O R E ) to achieve the certiﬁcation of cumulative reward; while they propose an adaptive version of the Neyman-Pearson lemma, which is similar with our global smoothing.
Note that we have analyzed in our paper and show that our local smoothing approach is much more sophisticated and tight than the global smoothing one.