CROP: C ERTIFYING R OBUST P OLICIES FOR R EINFORCEMENT L EARNING THROUGH F UNCTIONAL S MOOTHING
A BSTRACT
As reinforcement learning (RL) has achieved great success and been even adopted in safety-critical domains such as autonomous vehicles, a range of empirical studies have been conducted to improve its robustness against adversarial attacks.
However, how to certify its robustness with theoretical guarantees still remains challenging.
In this paper, we present the ﬁrst uniﬁed framework CROP (Certifying Robust
Policies for RL) to provide robustness certiﬁcation on both action and reward levels.
In particular, we propose two robustness certiﬁcation criteria: robustness of per-state actions and lower bound of cumulative rewards .
We then develop a local smoothing algorithm for policies derived from Q-functions to guarantee the robustness of actions taken along the trajectory; we also develop a global smoothing algorithm for certifying the lower bound of a ﬁnite-horizon cumulative reward, as well as a novel local smoothing algorithm to perform adaptive search in order to obtain tighter reward certiﬁcation.
Empirically, we apply CROP to evaluate several existing empirically robust RL algorithms, including adversarial training and different robust regularization, in four environments (two representative Atari games,
Highway, and CartPole).
Furthermore, by evaluating these algorithms against adversarial attacks, we demonstrate that our certiﬁcations are often tight.
All experiment results are available at website https://crop-leaderboard .
github .
io .
I NTRODUCTION
Reinforcement learning (RL) has been widely applied to different applications, such as robotics (Kober et al., 2013; Deisenroth et al., 2013; Polydoros & Nalpantidis, 2017), autonomous driving vehicles (Shalev-Shwartz et al., 2016; Sallab et al., 2017), and trading (Deng et al., 2016; Almahdi & Yang, 2017; Ye et al., 2020).
However, recent studies have shown that learning algorithms are vulnerable to adversarial attacks (Goodfellow et al., 2014;
Kurakin et al., 2016; Moosavi-Dezfooli et al., 2016; Jia & Liang, 2017; Eykholt et al., 2018), and a range of attacks have also been proposed against the input states and trained policies of RL (Huang et al., 2017; Kos & Song, 2017; Lin et al., 2017; Behzadan & Munir, 2017a).
As more and more safety-critical applications are being deployed in real-world (Christiano et al., 2016; Fisac et al., 2018; Cheng et al., 2019; eop), how to test and improve their robustness before massive production is of great importance.
To defend against adversarial attacks in RL, different empirical defenses have been proposed (Mandlekar et al., 2017; Behzadan & Munir, 2017b; Pattanaik et al., 2018;
Fischer et al., 2019; Zhang et al., 2020; Oikarinen et al., 2020; Donti et al., 2020;
Shen et al., 2020; Eysenbach & Levine, 2021).
In particular, adversarial training (Kos & Song, 2017; Behzadan & Munir, 2017b; Pattanaik et al., 2018) and regularized RL algorithms by enforcing the smoothness of the trained models (Shen et al., 2020;
Zhang et al., 2020) have been studied to improve the robustness of trained policies.
However, several strong adaptive attacks have been proposed against these empirical defenses (Gleave et al., 2019;
Hussenot et al., 2019; Russo & Proutiere, 2019) and it is important to provide robustness certiﬁcation for a given learning algorithm to end such repeated game between attackers and defenders.
To provide robustness certiﬁcation , several studies have been conducted on classiﬁcation.
For instance, both deterministic (Ehlers, 2017; Katz et al., 2017; Cheng et al., 2017; Tjeng et al., 2017; Weng et al., 2018; Zhang et al., 2018; Singh et al., 2019; Gehr et al., 2018; Wong & Kolter, 2018; Raghunathan et al., 2018) and probabilistic approaches (Lecuyer et al., 2019; Cohen et al., 2019; Lee et al., 2019;
Salman et al., 2019; Carmon et al., 2019; Jeong & Shin, 2020) have been explored to provide a lower bound of classiﬁcation accuracy given bounded adversarial perturbation.
Considering the sequential decision making property of RL, which makes it more challenging to be directly certiﬁed compared to classiﬁcation, in this paper we ask: How to provide efﬁcient and effective robustness certiﬁcation for RL algorithms?
What criteria should be used to certify the robustness of RL algorithms?
Different from classiﬁcation which involves one-step prediction only, RL algorithms provide both action prediction and reward feedback, making what to certify and how to certify robustness of RL challenging.
In this paper we focus on Q-learning and propose two certiﬁcation criteria: per-state action stability and lower bound of perturbed cumulative reward .
In particular, to certify the per-state action stability , we propose the local smoothing on each input state and therefore derive the certiﬁed radius for perturbation at each state, within which the action prediction will not be altered.
To certify the lower bound of cumulative reward , we propose both global smoothing over the ﬁnite trajectory to obtain the expectation or percentile bounds given trajectories smoothed with sampled noise sequences; and local smoothing to calculate an absolute lower bound based on our adaptive search algorithm.
We leverage our framework to test nine empirically robust RL algorithms on multiple RL environments.
We show that the certiﬁed robustness depends on both the algorithm and the environment properties.
For instance, RadialRL (Oikarinen
et al.,
) is the most certiﬁably robust method on Freeway.
In addition, based
on
the
per-state certiﬁcation, we observe that for some environments
such as
Pong, some states are more certiﬁably robust and such pattern is periodic.
Given the information of which states are more vulnerable, it is possible to design robust algorithms to speciﬁcally focus on these vulnerable states.
Based on the lower bound of perturbed cumulative reward, we show that our certiﬁcation is tight by comparing our bounds with empirical results under adversarial attacks.
Technical Contributions.
In this paper, we take an important step towards providing robustness certiﬁcation for Q-learning.
We make contributions on both theoretical and empirical fronts.
• We propose a framework for certifying the robustness of Q-learning algorithms, which is notably the ﬁrst that provides the robustness certiﬁcation w.r.t.
the cumulative reward.
• We propose two robustness certiﬁcation criteria for Q-learning algorithms, together with corresponding certiﬁcation algorithms based on global and local smoothing strategies.
• We theoretically prove the certiﬁcation radius for input state and lower bound of perturbed cumulative reward under bounded adversarial state perturbations.
• We conduct extensive experiments to provide certiﬁcation for nine empirically robust RL algorithms on multiple RL environments.
We provide several interesting observations which would further inspire the development of robust RL algorithms.
P RELIMINARIES
Q-learning and Deep Q-Networks (DQNs).
Markov decision processes (MDPs) are at the core of
RL.
Our focus is on discounted discrete-time MDPs, which are deﬁned by tuple ( S , A , R, P, γ, d 0 ) , where S is a set of states (each with dimensionality N ), A represents a set of discrete actions,
R : S ×
A → R is the reward function, and P : S ×
A → P ( S ) is the transition function with
P ( · ) deﬁning the set of probability measures, γ ∈ [0 , 1] is the discount factor, and d 0 ∈ P ( S ) is the distribution over the initial state.
At time step t , the agent is in the state s t ∈ S .
After choosing action a t ∈ A , the agent transitions to the next state s t +1 ∼ P ( s t , a t ) and receives reward R ( s t , a t ) .
The goal is to learn a policy π : S → P ( A ) that maximizes the expected cumulative reward E [ (cid:80) t γ t r t ] .
Q-learning (Watkins & Dayan, 1992) learns an action-value function (Q-function), Q (cid:63) ( s, a ) , which is the maximum expected cumulative reward the agent can achieve after taking action a in state s : Q (cid:63) ( s, a )
=
R ( s, a ) + γ
E s (cid:48) ∼ P ( s,a ) [max a (cid:48) Q (cid:63) ( s (cid:48) , a (cid:48) )] .
In deep Q-Networks (DQNs) (Mnih et al., 2013), Q (cid:63) is approximated using a neural network parametrized by θ , i.e. , Q π ( s, a ; θ ) ≈ Q (cid:63) ( s, a ) .
Let ρ ∈ P ( S × A ) be the observed distribution deﬁned over states s and actions a , the network can be trained via minimizing loss function L ( θ ) =
E ( s,a ) ∼ ρ,s (cid:48) ∼ P ( s,a ) (cid:104)
( R ( s, a ) + γ max a (cid:48)
Q π ( s (cid:48) , a (cid:48) ; θ ) − Q π ( s, a ; θ )) 2 (cid:105) .
The greedy policy π is deﬁned as taking the action with highest Q
π value in each state s : π ( s ) =
arg max a ∈A Q π ( s, a ) .
Certiﬁed Robustness for Classiﬁers via Randomized Smoothing.
Randomized smoothing (Cohen et al., 2019) has been proposed to provide probabilistic certiﬁed robustness for classiﬁcation.
It achieves state-of-the-art certiﬁed robustness on large-scale datasets such as ImageNet under (cid:96) 2bounded constraints (Salman et al., 2019; Yang et al., 2020).
In particular, given a base model and a test instance, a smoothed model is constructed by outputting the most probable prediction over different Gaussian perturbed inputs.
R OBUSTNESS C ERTIFICATION IN Q- LEARNING
In this section, we ﬁrst introduce the threat model, followed by two robustness certiﬁcation criteria for the Q-learning algorithm: per-state action and cumulative reward .
We consider the standard adversarial setting in Q-learning (Huang et al., 2017; Kos & Song, 2017; Zhang et al., 2020), where the adversary can apply (cid:96) 2 -bounded perturbation B ε = { δ ∈ R n | (cid:107) δ (cid:107) 2 ≤ ε } to input state observations of the agent during decision (test) time to cause the policy to select suboptimal actions.
The agent observes the perturbed state and takes action a (cid:48)
=
π ( s + δ ) , following policy π .
Following the
Kerckhoff’s principle (Shannon, 1949), we consider a worst-case adversary who applies adversarial perturbations to every state at decision time.
Our analysis and methods are generalizable to other (cid:96) p norms following (Yang et al., 2020; Lecuyer et al., 2019).
3. R OBUSTNESS C ERTIFICATION FOR Q- LEARNING WITH D IFFERENT C RITERIA
To provide the robustness certiﬁcation for Q-learning, we propose two certiﬁcation criteria: per-state action robustness and lower bound of the cumulative reward .
Robustness Certiﬁcation for Per-State Action.
We ﬁrst aim to explore the robustness (stability/consistency) of the per-state action given adversarially perturbed input states.
Deﬁnition 1 (Certiﬁcation for per-state action) .
Given a trained network Q π with policy π , we deﬁne the robustness certiﬁcation for per-state action as the maximum perturbation magnitude ¯ ε , such that for any perturbation δ ∈ B ¯ ε , the predicted action under the perturbed state will be the same as the action taken in the clean environment, i.e. , π ( s + δ ) =
π ( s ) , ∀ δ ∈ B ¯ ε .
Robustness Certiﬁcation for Cumulative Reward.
Given that the cumulative reward is important for RL, here in addition to the per-state action, we also deﬁne the robustness certiﬁcation regarding the cumulative reward under input state perturbation.
Deﬁnition 2 (Cumulative reward) .
Let P : S ×
A → P ( S ) be the transition function of the environment with P ( · ) deﬁning the set of probability measures.
Let R, d 0 , γ, Q π , π be the reward function, initial state distribution, discount factor, a given trained Q-network, and the corresponding greedy policy as introduced in Section 2.
J ( π ) represents the cumulative reward and J ε ( π ) represents the perturbed cumulative reward under perturbations δ t ∈ B ε at each time step t :
J ( π ) := ∞ (cid:88) t = γ t R ( s t , π ( s t )) , where s t +1 ∼ P ( s t , a t ) , s 0 ∼ d 0 , and
J ε ( π ) := ∞ (cid:88) t = γ t R ( s t , π ( s t + δ t )) , where s t +1 ∼ P ( s t , π ( s t + δ t )) , s 0 ∼ d 0 .
The randomness of J ( π ) arises from the environment dynamics, while that of J ε ( π ) includes additional randomness from the perturbations { δ t } .
We focus on a ﬁnite horizon H in this paper, where a sufﬁciently large H can approximate J ( π ) and J ε ( π ) to arbitrary precision when γ < 1 .
Deﬁnition 3 (Robustness certiﬁcation for cumulative reward) .
The robustness certiﬁcation for cumulative reward is the lower bound of perturbed cumulative reward J such that J ≤ J ε ( π ) under perturbation in B ε = { δ ∈ R n | (cid:107) δ (cid:107) 2 ≤ ε } applied to all time steps.
We will provide details on the certiﬁcation of per-state action in Section 4 and the certiﬁcation of cumulative reward in Section 5 based on different smoothing strategies and certiﬁcation methods.
R OBUSTNESS C ERTIFICATION S TRATEGIES FOR P ER -S TATE A CTION
In this section, we discuss the robustness certiﬁcation for per-state action , aiming to calculate a lower bound of maximum perturbation magnitude ¯ ε in Deﬁnition 1.
4. C ERTIFICATION FOR P ER -S TATE A CTION VIA A CTION -V ALUE F UNCTIONAL S MOOTHING
Let Q π be the action-value function given by the trained network Q with policy π .
We derive a smoothed function (cid:101) Q π through per-state local smoothing .
Speciﬁcally, at each time step t , for each action a ∈ A , we draw random noise from a Gaussian distribution N (0 , σ 2 I N ) to smooth Q π ( · , a ) .
(cid:101) Q π ( s t , a ) :=
E ∆ t ∼N (0 ,σ 2 I N ) Q π ( s t + ∆ t , a ) ∀ s t ∈ S , a ∈ A , and ˜ π ( s t ) := argmax a (cid:101) Q π ( s t , a ) ∀ s t ∈ S .
Lemma 1 (Lipschitz continuity of the smoothed value function) .
Given the action-value function
Q π : S × A → [ V min , V max ] , the smoothed function (cid:101) Q π with smoothing parameter σ is L-Lipschitz continuous with L = V max − V min σ (cid:112) 2 / π w.r.t.
the state input.
The proof is given in Appendix A.1.
Leveraging the Lipschitz continuity in Lemma 1, we derive the following theorem for certifying the robustness of per-state action.
Theorem 1.
Let Q π : S ×
A → [ V min , V max ] be a trained value network, (cid:101)
Q π be the smoothed func- tion with (2).
At time step t with state s t , we can compute the lower bound r t of maximum perturbation magnitude ¯ ε ( s t ) (i.e., r
[Equation]
≤ ¯ ε ( s t ) , ¯ ε deﬁned in Deﬁnition 1) for locally smoothed policy ˜ π :
[Equation]
where Φ − 1 is the inverse CDF function, a 1 is the action with the highest (cid:101) Q
π value at state s t , and a 2 is the runner-up action.
We name the lower bound r t as certiﬁed radius for the state s t .
The proof is omitted to Appendix A.2.
The theorem provides a certiﬁed radius r t for per-state action given smoothed policy: As long as the perturbation is bounded by r t , i.e. , (cid:107) δ t (cid:107) 2 ≤ r t , the action does not change: ˜ π ( s t + δ t ) =
˜
π
( s t ) .
To achieve high certiﬁed robustness for per-state action, Theoremimplies a tradeoff between value function smoothness and the margin between the values of top two actions: If a larger smoothing parameter σ is applied, the action-value function would be smoother and therefore more stable; however, it would shrink the margin between the top two action values leading to smaller certiﬁed radius.
Thus, there exists a proper smoothing parameter to balance the tradeoff, which depends on the actual environments and algorithms.
4. CROP-L O A CT : L OCAL R ANDOMIZED S MOOTHING FOR C ERTIFYING P ER -S TATE A CTION
Next we introduce the algorithm to achieve the certiﬁcation for per-state action.
Given a Q π network, we apply (2) to derive a smoothed network (cid:101) Q π .
At each state s t , we obtain the greedy action ˜ a t w.r.t.
(cid:101) Q π , and then compute the certiﬁed radius r t .
We present the complete algorithm in Appendix B.1.
There are some additional challenges in smoothing the value function and computing the certiﬁed radius in Q-learning compared with the standard classiﬁcation task (Cohen et al., 2019).
Challenge 1:
In classiﬁcation, the output range of the conﬁdence [0 , 1] is known a priori; however, in Q-learning, for a given Q π , its range [ V min , V max ] is unknown.
Challenge 2:
In the classiﬁcation task, the lower and upper bounds of the top two classes’ prediction probabilities can be directly computed via the conﬁdence interval base on multinomial proportions (Goodman, 1965).
For Q-networks, the outputs are not probabilities and calculating the multinomial proportions becomes challenging.
Pre-processing.
To address Challenge 1 , we estimate the output range [ V min , V max ] of a given network Q π based on a ﬁnite set of valid states S sub ⊆ S .
In particular, we craft a sufﬁciently large set S sub to estimate V min and V max for Q π on S , which can be used later in per-state smoothing.
The details for estimating V min and V max are deferred to Appendix B.1.
Certiﬁcation.
To smooth a given state s t , we use Monte Carlo sampling (Cohen et al., 2019) to sample noise applied to s t , and then estimate the corresponding smoothed value function (cid:101) Q π at s t with (2).
In particular, we sample m Gaussian noise ∆ i ∼ N (0 , σ 2 I N ) , clip the Q-network output to ensure that it falls within the range [ V min , V max ] , and then take the average of the output to obtain the smoothed action prediction based on (cid:101) Q π .
We then employ Theorem 1 to compute the certiﬁed radius r t .
We omit the detailed inference procedure to Appendix B.1.
To address Challenge 2 , we leverage
Hoeffding’s inequality (Hoeffding, 1994) to compute a lower bound of (cid:101) Q π ( s t , a 1 ) and an upper bound of (cid:101) Q π ( s t , a 2 ) with one-sided conﬁdence level parameter α given the top two actions a 1 and a 2 .
When the former is higher than the latter, we can certify a positive radius for the given state s t .
R OBUSTNESS C ERTIFICATION S TRATEGIES FOR THE C UMULATIVE R EWARD
In this section, we present robustness certiﬁcation strategies for the cumulative reward.
The goal is to provide the lower bounds for the perturbed cumulative reward in Deﬁnition 2.
In particular, we propose both global smoothing and local smoothing strategies to certify the perturbed cumulative reward.
In the global smoothing, we view the whole state trajectory as a function to smooth, which would lead to relatively loose certiﬁcation bound.
We then propose the local smoothing by smoothing each state individually to obtain the absolute lower bound.
5. C ERTIFICATION OF C UMULATIVE R EWARD BASED ON G LOBAL S MOOTHING
In contrast to Section 4 where we perform per-state smoothing to achieve the certiﬁcation for per-state action, here, we aim to perform global smoothing on the state trajectory by viewing the entire trajectory as a function.
In particular, we ﬁrst derive the expectation bound of the cumulative reward based on global smoothing by estimating the Lipschitz constant for the cumulative reward w.r.t.
the trajectories.
Since the Lipschitz estimation in the expectation bound is algorithm agnostic and could lead to loose estimation bound, we subsequently propose a more practical and tighter percentile bound .
Deﬁnition 4 ( σ -randomized trajectory and σ -randomized policy) .
Given a state trajectory ( s 0 , s 1 , . . .
, s H − 1 ) of length H where s t +1 ∼ P ( s t , π ( s t )) , s 0 ∼ d 0 , with π the greedy policy of the action-value function Q π , we derive a σ -randomized trajectory as ( s (cid:48) 0 , s (cid:48) 1 , . .
, s (cid:48)
H − 1 ) , where s (cid:48) t +1 ∼
P ( s (cid:48) t , π ( s (cid:48) t
+∆ t )) , ∆ t ∼ N (0 , σ 2 I N ) , and s (cid:48)
= s 0 ∼ d 0 .
We correspondingly deﬁne a σ -randomized policy π (cid:48) based on π in the following form: π (cid:48) ( s t ) :
= π ( s t + ∆ t ) where ∆ t ∼ N (0 , σ 2 I N ) .
Let the operator ⊕ concatenates given input states or noise that are added to each state.
The sampled noise sequence is denoted by ∆ =
⊕ H − 1 t =
∆ t , where ∆ t ∼ N (0 , σ 2 I N ) .
Deﬁnition 5 (Perturbed return function) .
Let R, P, γ, d 0 be the reward function, transition function, discount factor, and initial state distribution in Deﬁnition 2.
We deﬁne a bounded perturbed return function F π : R H × N → [ J min , J max ] representing cumulative reward with potential perturbation δ :
[Equation]
where s t +1 ∼ P ( s t , π ( s t + δ t )) , s 0 ∼ d 0 .
We can see when there is no perturbation ( δ t = 0 ), F π ( ⊕ H − 1 t =0 0 ) =
J ( π ) ; when there are adversarial perturbations δ t ∈ B ε at each time step, F π ( ⊕ H − 1 t =
δ t ) =
J ε ( π ) , i.e. , perturbed cumulative reward .
Mean Smoothing: Expectation bound.
Here we propose to sample noise sequences ∆ to perform global smoothing for the entire state trajectory, and calculate the lower bound of the expected perturbed cumulative reward E ∆ [ J ε ( π (cid:48) )] under all possible (cid:96) 2 -bounded perturbations within magnitude ε .
The expectation is over the noise sequence ∆ involved in the σ -randomized policy π (cid:48) in Deﬁnition 4.
Lemma 2 (Lipschitz continuity of smoothed perturbed return function) .
Let F be the perturbed return function function deﬁned in (4), the smoothed perturbed return function (cid:101) F π is ( J max − J min ) σ
[Equation]
2 / πLipschitz continuous, where (cid:101) F π (cid:0) ⊕
[Equation]
Theorem 2 (Expectation bound) .
Let J E
=
(cid:101)
[Equation]
Lε √ H , where L =
( J max
[Equation]
2 / π .
Then J E ≤ E [ J ε ( π (cid:48) )] .
Proof Sketch.
We ﬁrst derive the equality between expected perturbed cumulative reward E
[ J ε ( π (cid:48) )] and the smoothed perturbed return function (cid:101) F π ( ⊕ H − 1 t =
δ t ) .
Thus, to lower bound the former, it sufﬁces to lower bound the latter, which can be calculated leveraging the Lipschitz continuity of (cid:101) F in Lemma 2 (proved in Appendix A.3), noticing that the distance between ⊕ H − 1 t =0 0 and the adversarial perturbations ⊕ H − 1 t =
δ t is bounded by ε √ H .
The complete proof is omitted to Appendix A.4.
We obtain J min and J max in Lemma 2 from environment speciﬁcations which can be loose in practice.
Thus the Lipschitz constant L estimation is coarse and mean smoothing is usually loose.
We next present a method that circumvents estimating the Lipschitz constant and provides a tight percentile bound.
Percentile Smoothing: Percentile bound.
We now propose to apply percentile smoothing to smooth the perturbed cumulative reward and obtain the lower bound of the p -th percentile of J ε ( π (cid:48) ) , where π (cid:48) is a σ -randomized policy deﬁned in Deﬁnition 4.
[Equation]
Theorem 3 (Percentile bound) .
Let J p
= (cid:101) F p (cid:48)
π (cid:0) ⊕
H − 1 t =
[Equation]
0 (cid:1) , where p (cid:48) :
=
Φ (cid:0) Φ − 1 ( p ) − ε √ H / σ (cid:1) .
Then
J p ≤ the p -th percentile of J ε ( π (cid:48) ) .
The proof is provided in Appendix A.5 based on Chiang et al.
There are several other advantages of percentile smoothing over mean smoothing.
First, the certiﬁcation given by percentile smoothing is among the cumulative rewards of the sampled σ -randomized trajectories and is therefore achievable by a real-world policy, while the expectation bound is less likely to be achieved in practice given the loose Lipschitz bound.
Second, for a discrete function such as perturbed return function, the output of mean smoothing is continuous w.r.t.
σ , while the results given by percentile smoothing remain discrete.
Thus, the percentile smoothed function preserves properties of the base function before smoothing, and shares similar interpretation, e.g. , the number of rounds that the agent wins.
Third, taking p = 50% in percentile smoothing leads to the median smoothing which achieves additional properties such as robustness to outliers (Manikandan, 2011).
The detailed algorithm CROP-GR E , including the inference procedure, the estimation of (cid:101) F π , the calculation of the empirical order statistics, and the conﬁguration of algorithm parameters J min , J max , are deferred to Appendix B.2.
5. C ERTIFICATION OF C UMULATIVE R EWARD BASED ON L OCAL S MOOTHING
Though global smoothing provides efﬁcient and practical bounds for the perturbed cumulative reward, such bounds are still loose as they involve smoothing the entire trajectory at once.
In this section, we aim to provide a tighter lower bound for J ε (˜ π ) by performing local smoothing .
Given a trajectory of H time steps which is guided by the locally smoothed policy ˜ π , we can compute the certiﬁed radius at each time step according to Theorem 1, which can be denoted as r 0 , r 1 , . . .
, r H − 1 .
Recall that when the perturbation magnitude ε < r t , the optimal action a t at time step t will remain unchanged.
This implies that when ε < min H − 1 t =0 r t , none of the actions in the entire trajectory will be changed, and therefore the lower bound of the cumulative reward when ε < min H − 1 t =
r t is the return of the current trajectory in a deterministic environment.
Increasing ε has two effects.
First, the total number of time steps where the action is susceptible to change will increase; second, at each time step, the action can change from the best to the runner-up or the rest.
We next introduce an extension of certiﬁed radius r t to characterize the two effects.
Theorem 4.
Let ( r 1 t , . . .
, r |A|− 1 t ) be a sequence of certiﬁed radii for state s t at time step t , where r kt denotes the radius such that if ε < r kt , the possible action at time step t will belong to the actions corresponding to top k action values of (cid:101) Q at state s t .
The deﬁnition of r t in Theorem 1 is equivalent to r 1 t here.
The radii can be computed similarly as follows:
[Equation]
where a 1 is the action of the highest (cid:101) Q value at state s t and a k +1 is the ( k + 1) -th best action.
We additionally deﬁne r 0 t ( s t ) = 0 , which is also compatible with the deﬁnition above.
We defer the proof in Appendix A.6.
With Theorem 4, for any given ε , we can compute all possible actions under perturbations in B ε .
This allows an exhaustive search to traverse all trajectories satisfying that all certiﬁed radii along the trajectory are smaller than ε .
Then, we can conclude that
J ε (˜ π ) is lower bounded by the minimum return over all these possible trajectories.
5.2. CROP-L O R E : L OCAL SMOOTHING FOR C ERTIFIED R EWARD
Given a policy π in a deterministic environment, let the initial state be s 0 , we propose CROP-L O R E to certify the lower bound of J ε (˜ π ) .
At a high-level, CROP-L O R E exhaustively explores new trajectories leveraging Theorem 4 with priority queue and effectively updates the lower bound of cumulative reward J by expanding a trajectory tree dynamically.
The algorithm returns a collection of pairs { ( ε i , J ε i ) } | C | i =1 sorted in ascending order of ε i , where | C | is the length of the collection.
For all ε (cid:48) , let i be the largest integer such that ε i ≤ ε (cid:48) <
ε i +1 , then as long as the perturbation magnitude ε ≤ ε (cid:48) , the cumulative reward J ε (˜ π ) ≥ J ε i .
The algorithm is shown in Algorithm 3 in Appendix B.3.
Algorithm Description.
The method starts from the base case: when perturbation magnitude ε = 0 , the lower bound of cumulative reward J is exactly the benign reward.
The method then gradually increases the perturbation magnitude ε (later we will explain how a new ε is determined).
Along the increase of ε , the perturbation may cause the policy π to take different actions at some time steps, thus resulting in new trajectories.
Thanks to the local smoothing, the method leverages Theoremto ﬁgure out the exhaustive list of possible actions under current perturbation magnitude ε , and effectively explore these new trajectories by formulating them as expanded branches of a trajectory tree.
Once all new trajectories are explored, the method examines all leaf nodes of the tree and ﬁgures out the minimum reward among them, which is the new lower bound of cumulative reward J under this new ε .
To mitigate the explosion of branches, CROP-L O R E proposes several optimization tricks.
[Figure]
In the following, we will brieﬂy introduce the trajectory exploration and expansion and growth of perturbation magnitude steps, as well as the optimization tricks.
More algorithm details are deferred to Appendix B.3, where we also provide an analysis on the time complexity of the algorithm.
In trajectory exploration and expansion , CROP-L O R E organizes all possible trajectories in the form of search tree and progressively grows it.
For each node (representing a state), leveraging Theorem 4, we compute a non-decreasing sequence { r k ( s ) } |A|− 1 k =0 representing the required perturbation radii for π to choose each alternative action.
Suppose the current ε satisﬁes r i ( s ) ≤ ε < r i +1 ( s ) , we can grow ( i + 1) branches from current state s corresponding to the original action and i alternative actions since ε ≥ r j ( s ) for 1 ≤ j ≤ i .
We expand the tree branches using depth-ﬁrst search (Tarjan, 1972).
In perturbation magnitude growth , when all trajectories for perturbation magnitude ε are explored, we increase ε to seek certiﬁcation under larger perturbations.
This is achieved by preserving a priority queue (van Emde Boas, 1977) of the critical ε ’s that we will expand on.
Concretely, along the trajectory, at each tree node, we search for the possible actions and store actions corresponding to { r k ( s ) } |A|− 1 k
= i +1 into the priority queue, since these actions are exactly those need to be explored when ε grows.
We repeat the procedure of perturbation magnitude growth ( i.e. , popping out the head element from the queue) and trajectory exploration and expansion ( i.e. , exhaustively expand all trajectories given the perturbation magnitude) until the priority queue becomes empty or the perturbation magnitude ε reaches the predeﬁned threshold.
Additionally, we adopt a few optimization tricks commonly used in search algorithms to reduce the complexity of the algorithm, such as pruning and the memorization technique (Michie, 1968).
More potential improvements are discussed in Appendix B.3.
So far, we have presented all our certiﬁcation methods.
In Appendix C, we further discuss the advantages , limitations , and extensions of these methods, and provide more detailed analysis to help with understanding.
E XPERIMENTS
In this section, we present evaluation for the proposed robustness certiﬁcation framework CROP.
Concretely, we apply our three certiﬁcation algorithms (CROP-L O A CT , CROP-GR E , and CROP-L O R E ) to certify nine RL methods ( StdTrain (Mnih et al., 2013), GaussAug (Kos & Song, 2017), AdvTrain (Behzadan & Munir, 2017b), SA-MDP (PGD,CVX) (Zhang et al., 2020), RadialRL (Oikarinen et al., 2020), CARRL (Everett et al., 2021), NoisyNet (Fortunato et al., 2017), and GradDQN (Pattanaik et al., 2018)) on two high-dimensional Atari games (Pong and Freeway), one low dimensional control environment (CartPole), and an autonomous driving environment (Highway).
As a summary, we ﬁnd that (1) SA-MDP (CVX), SA-MDP (PGD), and RadialRL achieve high certiﬁed robustness in different environments; (2) Large smoothing variance can help to improve certiﬁed robustness signiﬁcantly on Freeway, while a more careful selection of the smoothing parameter is needed in
Pong; (3) For methods that demonstrate high certiﬁed robustness, our certiﬁcation of the cumulative reward is tight.
We defer the detailed descriptions of the environments to Appendix D.1 and the introduction to the RL methods and implementation details to Appendix D.2.
More interesting results and discussions are omitted to Appendix E and our leaderboard, including results on CartPole (Appendix E.6) and Highway (Appendix E.7).
As a foundational work providing robustness certiﬁcation for RL, we expect more RL algorithms and RL environments will be certiﬁed under our framework in future work.
6. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR P ER - STATE A CTION
5e-3 5e-2 1e-1 5e-1 7.5e-1 1.0    c u m u l a t
i v e r e w a r d J
Freeway 1e-3 5e-3 1e-2 3e-2 5e-2 1e-1 12.
Pong
StdTrainGaussAugAdvTrainSA-MDP (PGD)
SA-MDP (CVX)
RadialRLNoisyNetGradDQN
In this subsection, we provide the robustness certiﬁcation evaluation for per-state action.
Experimental Setup and Metrics.
We evaluate the locally smoothed policy ˜ π derived in (2).
We follow Theorem 1 and report the certiﬁed radius r t at each step t .
We additionally compute certiﬁed ratio η at radius r , representing the percentage of time steps that can be certiﬁed, formally denoted as η r = 1 / H (cid:80)
Ht =1 1 { r t ≥ r } .
More details are omitted to Appendix D.4.
Evaluation Results of CROP-L O A CT .
We present the certiﬁcation comparison in Figure 1 and the benign performance under different smoothing variances in Figure 2; we omit details of certiﬁed ratio to Appendix E.1.
On Freeway , we evaluate with a large range of smoothing parameter σ up to 1 . 0 , since Freeway can tolerate large noise as shown in Figure 2.
Results under even larger σ are deferred to Appendix E.2, showing that the certiﬁed robustness can be further improved for the empirically robust methods.
From Figure 1, we see that RadialRL consistently achieves the highest certiﬁed radius across all σ ’s.
This is because RadialRL explicitly optimizes over the worst-case perturbations.
StdTrain, GaussAug, and AdvTrain are not as robust, and increasing σ will not make a difference.
In Pong , SA-MDP (CVX) is the most certiﬁably robust, which may be due to the hardness of optimizing the worst-case perturbation in Pong.
More interesting, all methods present similar periodic patterns for the certiﬁed radius on different states, which would inspire further robust training methods to take the “conﬁdent state” ( e.g. , when the ball is ﬂying towards the paddle) into account.
We illustrate the frames with varying certiﬁed radius in Appendix E.3.
Overall, the certiﬁed robustness of these methods largely matches empirical observations (Behzadan & Munir, 2017b;
Oikarinen et al., 2020; Zhang et al., 2020).
6. E VALUATION OF R OBUSTNESS C ERTIFICATION FOR C UMULATIVE R EWARD
Here we will discuss the evaluation for the robustness certiﬁcation regarding cumulative reward in Section 5.
We show the evaluation results for both CROP-GR E and CROP-L O R E .
Evaluation Setup and Metrics.
We evaluate the σ -randomized policy π (cid:48) derived in Deﬁnitionfor CROP-GR E and the locally smoothed policy ˜ π derived in (2) for CROP-L O R E .
We compute the expectation bound J E , percentile bound J p , and absolute lower bound J following Theorem 2, Theorem 3, and Section 5.2.1.
To validate the tightness of the bounds, we additionally perform empirical attacks.
More rationales see Appendix D.3.
The detailed parameters are omitted to Appendix D.4.
Evaluation of CROP-GR E .
We present the certiﬁcation of reward in Figure 3 and mainly focus on analyzing percentile bound.
We present the bound w.r.t.
different attack magnitude ε under different smoothing parameter σ .
For each σ , there exists an upper bound of ε that can be certiﬁed, which is positively correlated with σ .
Details see Appendix B.2.
We observe similar conclusion as in the per-state action certiﬁcation that RadialRL is most certiﬁably robust for Freeway, while SA-MDP (CVX,PGD) are most robust for Pong although the highest robustness is achieved at different σ .
Evaluation of CROP-L O R E .
In Figure 3, we note that in Freeway , RadialRL is the most robust, followed by SA-MDP (CVX), then SA-MDP (PGD), AdvTrain, and GaussAug.
As for Pong , SAMDP (CVX) outperforms RadialRL and SA-MDP (PGD).
Remaining methods are not clearly ranked.
6. D ISCUSSION ON E VALUATION R ESULTS
Impact of Smoothing Parameter σ .
We draw similar conclusions regarding the impact of smoothing variance from the results given by CROP-L O A CT and CROP-GR E .
In Freeway , as σ increases, the robustness of StdTrain, GaussAug, and AdvTrain barely increases, while that of SA-MDP (PGD),
SA-MDP (CVX), and RadialRL steadily increases.
In Pong , a σ in the range 0 .
03 shall be suitable for almost all methods.
More explanations from the perspective of benign performance and certiﬁed results will be provided in Appendix E.4.
For CROP-L O R E , there are a few different conclusions.
On Freeway , it is clear that under larger attack magnitude ε , larger smoothing parameter σ always secures higher lower bound J .
In Pong , CROP-L O R E can only achieve non-zero certiﬁcation with small σ under a small range of attack magnitude, implying the difﬁculty of establishing nontrivial certiﬁcation for these methods on Pong.
We defer details on the selection of σ to Appendix E.4.
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Tightness of the certiﬁcation J E , J p , and J .
We compare the empirical cumulative rewards achieved under PGD attacks with our certiﬁed lower bounds.
First, the empirical results are consistently lower bounded by our certiﬁcations, validating the correctness of our bounds.
Regarding the tightness, the improved J p is much tighter than the loose J E , supported by discussions in Section 5.1.
The tightness of J can be reﬂected by the zero gap between the certiﬁcation and the empirical result under a wide range of attack magnitude ε .
Furthermore, the certiﬁcation for SA-MDP (CVX,PGD) are quite tight for Freeway under large attack magnitudes, and RadialRL demonstrates its superiority on Freeway.
Additionally, different methods may achieve the same empirical results under attack, yet their certiﬁcations differ tremendously, indicating the importance of the robustness certiﬁcation .
Game Properties.
The certiﬁed robustness of any method on Freeway is much higher than that on
Pong, indicating that Freeway is a more stable game than Pong, also shown in Mnih et al. (2015).
R ELATED W ORK
We brieﬂy review several RL methods that demonstrate empirical robustness.
Kos & Song (2017) and
Behzadan & Munir (2017b) show that adversarial training can increase the agent’s resilience.
SADQN (Zhang et al., 2020) leverages regularization to encourage the top- 1 action to stay unchanged under perturbation.
Radial-RL (Oikarinen et al., 2020) minimizes an adversarial loss function that incorporates the upper bound of the perturbed loss.
CARRL (Everett et al., 2021) computes the lower bounds of Q values under perturbation for action selection, but it is only suitable for low-dimensional environments.
Most of these works only provide empirical robustness, with Zhang et al.
(2020) and Fischer et al.
(2019) additionally provide robustness certiﬁcates at state level.
To the best of our knowledge, this is the ﬁrst work providing robustness certiﬁcation for the cumulative reward of RL methods.
Broader discussions and comparisons of related work are in Appendix F.
Conclusions.
To provide the robustness certiﬁcation for RL methods, we propose a general framework CROP, aiming to provide certiﬁcation based on two criteria.
Our evaluations show that certain empirically robust RL methods are certiﬁably robust for speciﬁc RL environments. A CKNOWLEDGMENTS
This work is partially supported by the NSF grant No.1910100, NSF CNS 20-46726 CAR, Alfred P.
Sloan Fellowship, and Amazon Research Award.
Ethics Statement.
In this paper, we prove the ﬁrst framework for certifying the robustness of
RL algorithms against evasion attacks.
We aim to certify and therefore potentially improve the trustworthiness of machine learning models, and we do not expect any ethics issues raised by our work.
Reproducibility Statement.
All theorem statements are substantiated with rigorous proofs in our
Appendix.
We have upload the source code as the supplementary material for reproducibility purpose.
A P ROOFS A. P ROOF OF L EMMA
We recall Lemma 1:
Lemma 1 (Lipschitz continuity of the smoothed value function) .
Given the action-value function
Q π : S × A → [ V min , V max ] , the smoothed function (cid:101) Q π with smoothing parameter σ is L-Lipschitz continuous with L = V max − V min σ (cid:112) 2 / π w.r.t.
the state input.
Proof.
To prove Lemma 1, we leverage the technique in the proof for Lemma 1 of Salman et al.
(2019) in their Appendix A.
For each action a ∈ A , our smoothed value function is
[Equation]
Taking the gradient w
[Equation]
r
[Equation]
t. s
[Equation]
we obtain
[Equation]
For any unit direction u
[Equation]
we have
[Equation]
σ
[Equation]
π
Thus, (cid:101)
[Table]
A. P ROOF OF T HEOREM
We recall Theorem 1:
Theorem 1.
Let Q π : S ×
A → [ V min , V max ] be a trained value network, (cid:101)
Q π be the smoothed func- tion with (2).
At time step t with state s t , we can compute the lower bound r t of maximum perturbation magnitude ¯ ε ( s t ) (i.e., r t ≤ ¯ ε ( s t ) , ¯ ε deﬁned in Deﬁnition 1) for locally smoothed policy ˜ π :
[Equation]
where Φ − 1 is the inverse CDF function, a 1 is the action with the highest (cid:101) Q
π value at state s t , and a 2 is the runner-up action.
We name the lower bound r t as certiﬁed radius for the state s t .
We ﬁrst present a lemma that can help in the proof of Theorem 1.
Lemma 3.
Let Φ be the CDF of a standard normal distribution, the mapping η a ( s ) :
= σ · Φ − 1 (cid:16)
[Equation]
is
[Equation]
Lipschitz continuous.
The lemma can be proved following the same technique as the proof for Lemma 1 in Appendix A.1.
The detailed proof can be referred to in the proof for Lemma 2 of Salman et al.
in their
Appendix A. We next show how to leverage Lemma 3 to prove Theorem 1.
Proof for Theorem 1.
Let the perturbation be δ t , based on the Lipschitz continuity of the mapping η , we have
[Equation]
Suppose that under perturbation δ t , the action selection would be misled in the sense that the smoothed value for the original action a 1 is lower than that of another action a 2 , i.e. , (cid:101) Q
π ( s t + δ t , a 1 ) ≤ (cid:101) Q π ( s t + δ t , a 2 ) .
Then, based on the monotonicity of η , we have
[Equation]
Summing up (7), (8), and (9), we obtain
[Equation]
which is a lower bound of the maximum perturbation magnitude ¯ ε ( s t ) that can be tolerated at state s t .
Hence, when r t takes the value of the computed lower bound, it satisﬁes the condition that r t ≤ ¯ ε ( s t ) .
A. P ROOF OF L EMMA
We recall Lemma 2:
Lemma 2 (Lipschitz continuity of smoothed perturbed return function) .
Let F be the perturbed return function function deﬁned in (4), the smoothed perturbed return function (cid:101) F π is ( J max − J min ) σ (cid:112) 2 / πLipschitz continuous, where (cid:101) F π (cid:0)
⊕ H − 1 t =
δ t (cid:1) :=
[Equation]
Proof.
The proof can be done in a similar fashion as the proof for Lemma 1 in Appendix A.1.
Compared with the smoothed value network where the expectation is taken over the sampled states, here, similarly, the smoothed perturbed return function is derived by taking the expectation over sampled σ -randomized trajectories.
The difference is that the output range of the Q-network is [ V min , V max ] , while the output range of the perturbed return function is [ J min , J max ] .
Thus, the smoothed perturbed return function (cid:101) F is ( J max − J min )
σ
[Equation]
A. P ROOF OF T HEOREM
We recall the deﬁnition of (cid:101) F π as well as Theorem 2:
[Equation]
Theorem 2 (Expectation bound) .
Let J E
=
(cid:101) F π (cid:0)
⊕
H − 1 t =0 0 (cid:1) − Lε √ H , where L =
( J max − J min ) σ (cid:112) 2 / π .
Then J E ≤ E [ J ε ( π (cid:48) )] .
Proof.
We note the following equality
[Equation]
where (a) comes from the deﬁnition of the smoothed perturbed return function (cid:101) F π , (b) is due to the deﬁnition of the σ -randomized policy π (cid:48) , and (c) arises from the deﬁnition of the perturbed cumulative reward J Hε .
Thus, the expected perturbed cumulative reward E (cid:2) J Hε ( π (cid:48) )
(cid:3) is equivalent to the smoothed perturbed return function (cid:101) F π (cid:0)
⊕ H − 1 t =
δ t (cid:1) .
Furthermore, since the distance between the all-zero ⊕ H − 1 t =0 0 and the adversarial perturbations ⊕ H − 1 t =
δ t is bounded by ε √ H , leveraging the
Lipschitz smoothness of (cid:101) F in Lemma 2, we obtain the lower bound of the expected perturbed cumulative reward E (cid:2)
J Hε ( π (cid:48) )
(cid:3) as (cid:101) F π (cid:0) ⊕ H − 1 t =0 0 (cid:1) − Lε √ H .
A. P ROOF OF T HEOREM
We recall the deﬁnition of (cid:101) F pπ as well as Theorem 3:
[Equation]
Theorem 3 (Percentile bound) .
Let J p
= (cid:101) F p (cid:48)
π (cid:0) ⊕
H − 1 t =0 0 (cid:1) , where p (cid:48) :
=
Φ (cid:0) Φ − 1 ( p ) − ε √ H / σ (cid:1) .
Then
J p ≤ the p -th percentile of J ε ( π (cid:48) ) .
Proof.
To prove Theorem 3, we leverage the technique in the proof for Lemma 2 of Chiang et al.
(2020) in their Appendix B.
For brevity, we abbreviate δ :=
⊕ H − 1 t =
δ t and redeﬁne the plus operator such that δ + ζ :=
⊕ H − 1 t =
( δ t + ζ t ) .
Then, similar to Lemma 3, we have the conclusion that
[Equation]
is 1 -Lipschitz continuous, where ζ ∼ N (0 , σ 2 I H × N ) .
Thus, under the perturbations δ t ∈ B ε for t = 0 . . .
H − 1 , we have
[Equation]
+
[Equation] [Equation]
(Since (cid:107) δ (cid:107) 2 ≤ ε √ H )
[Equation]
(By deﬁnition of (cid:101) F pπ )
[Equation]
(By deﬁnition of p (cid:48) ) .
Since Φ − 1 monotonically increase, this implies that P (cid:104) F π ( δ + ζ ) ≤ (cid:101) F p (cid:48) π ( 0 ) (cid:105) ≤ p .
According to the deﬁnition of (cid:101) F pπ in (11), we see that (cid:101) F p (cid:48) π ( 0 ) ≤ (cid:101) F pπ ( δ ) , i.e. , J p ≤ the p -th percentile of J ε ( π (cid:48) ) .
Hence, the theorem is proved.
A. P ROOF OF T HEOREM
We recall Theorem 4:
Theorem 4.
Let ( r 1 t , . . .
, r |A|− 1 t ) be a sequence of certiﬁed radii for state s t at time step t , where r kt denotes the radius such that if ε < r kt , the possible action at time step t will belong to the actions corresponding to top k action values of (cid:101) Q at state s t .
The deﬁnition of r t in Theorem 1 is equivalent to r 1 t here.
The radii can be computed similarly as follows:
[Equation]
where a 1 is the action of the highest (cid:101) Q value at state s t and a k +1 is the ( k + 1) -th best action.
We additionally deﬁne r 0 t ( s t ) = 0 , which is also compatible with the deﬁnition above.
Proof.
Replacing a 2 with a k +1 in the proof for Theorem 1 in Appendix A.2 directly leads to Theorem 4.
B A DDITIONAL D ETAILS OF C ERTIFICATION S TRATEGIES
In this section, we cover the concrete details regarding the implementation of our three certiﬁcation strategies, as a complement to the high-level ideas introduced back in Section 4 and Section 5.
B. D ETAILED A LGORITHM OF CROP-L O A CT
[Table]
[Table] [Table]
[Table] [Table]
[Table]
We present the concrete algorithm of CROP-L O
A CT in Algorithm 1 for the procedures introduced in Section 4.2.
For each given state s t , we ﬁrst perform Monte Carlo sampling (Cohen et al., 2019;
Lecuyer et al., 2019) to achieve local smoothing.
Based on the smoothed value function (cid:101) Q π , we then compute the robustness certiﬁcation for per-state action, i.e. , the certiﬁed radius r t at the given state s t , following Theorem 1.
Detailed Inference Procedure.
During inference, we invoke the model with m samples of Gaussian noise at each time step, and use the averaged Q value on these m noisy samples to obtain the greedy action selection and compute the certiﬁcation.
This procedure is similar to P REDICT in Cohen et al.
, but since RL involves multiple step decisions, we do not take the “abstain” decision as in P REDICT in Cohen et al.
; instead, CROP-L O A CT will take the greedy action at all steps no matter whether the action can be certiﬁed or not.
Estimation of the Algorithm Parameters V min , V max .
Our estimate is obtained via sampling the trajectories and calculating the Q values associated with the state-action pairs along these trajectories.
Since we perform clipping using the obtained bounds, the certiﬁcation is sound.
The cost for estimating V min and V max is essentially associated with the number of sampled trajectories, the number of steps in each trajectory, the number of sampled Gaussian noise m per step, and the cost of doing one forward pass of the Q network; thus, the estimation can be done with low cost.
Furthermore,
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
we can balance the trade-off between the accuracy and efﬁciency of the estimation, and for any conﬁguration of V min and V max , the certiﬁcation will invariably be sound (reason above).
B. D ETAILED A LGORITHM OF CROP-GR E
We present the concrete algorithm of CROP-GR E in Algorithm 2 for the procedures introduced in Section 5.1.
Similarly to CROP-L O A CT , the algorithm also consists of two parts: performing smoothing and computing certiﬁcation, where we compute both the expectation bound J E and the percentile bound J p .
Step 1: Global smoothing.
We adopt Monte Carlo sampling (Cohen et al., 2019; Lecuyer et al., 2019) to estimate the smoothed perturbed return function (cid:101) F by sampling multiple σ -randomized trajectories via drawing m noise sequences.
For each noise sequence ζ ∼ N (0 , σ 2 I H × N ) , we apply noise ζ t to the input state s t sequentially, and obtain the sum of the reward
J Ci = (cid:80) H − 1 t =
re t as the return for this σ -randomized trajectory.
We then aggregate the smoothed perturbed return values { J Ci } mi =1 via mean smoothing and percentile smoothing.
Step 2: Certiﬁcation for perturbed cumulative reward.
First, we compute the expectation bound
J E using Theorem 2.
Since the smoothed perturbed return function (cid:101) F is obtained based on m sampled noise sequences, we use Hoeffding’s inequality (Hoeffding, 1994) to compute the lower bound of the random variable (cid:101) F (cid:0) ⊕ H − 1 t =0 0 (cid:1) with a conﬁdence level α .
We then calculate the lower bound of (cid:101) F (cid:0) ⊕ H − 1 t =
δ t (cid:1) under all possible (cid:96) 2 -bounded perturbations δ t ∈ B ε leveraging the smoothness of (cid:101) F .
We then compute the percentile bound J p using Theorem 3.
We let J Ci be sorted increasingly, and perform normal approximations (Stein et al., 1972) to compute the largest empirical order statistic
J Ck such that P (cid:104)
J p ≥ J Ck (cid:105) ≥
1 − α .
The empirical order statistic J Ck is then used as the proxy of
J p under α conﬁdence level.
We next provide detailed explanations for C OMPUTE O RDER S TATS , which aims to compute the order k using binomial formula plus normal approximation.
Estimation of (cid:101) F π (deﬁned in Lemma 2) .
For computing the expectation bound ( i.e. , the lower bound of E ∆ [ J ε ( π (cid:48) )] ), an intermediate step is to estimate (cid:101) F π (as shown in line 8 of Algorithm 2).
We emphasize that the accuracy of the estimation does not inﬂuence the soundness of the lower bound calculation.
The reason is given below.
The number of sampled randomized trajectories m controls the trade-off between the estimation efﬁciency and accuracy , as well as the tightness of the derived lower bound.
Concretely, a small m would provide high efﬁciency, low estimation accuracy, and loose lower bound; but the lower bound is always sound , since our algorithm explicitly accounts for the inaccuracy associated with m via leveraging the Hoeffding’s inequality in line 9 in Algorithm 2.
Details of C OMPUTE O RDER S TATS .
We consider the sorted sequence J C 1 ≤ J C 2 ≤ · · · ≤ J Cm .
We additionally set J C 0 =
−∞ and J Cm +1 = ∞ .
Our goal is to ﬁnd the largest k such that
P (cid:104)
J p ≥ J Ck (cid:105) ≥
1 − α .
We evaluate the probability explicitly as follows:
[Equation]
Thus, the condition P (cid:104)
J p ≥ J Ck
[Equation]
≥ 1 − α is equivalent to
[Equation]
Given large enough m , the LHS of (12) can be approximated via a normal distribution with mean equal to mp (cid:48) and variance equal to mp (cid:48) (1 − p (cid:48) ) .
Concretely, we perform binary search to ﬁnd the largest k that satisﬁes the constraint.
We ﬁnally explain the upper bound of ε that can be certiﬁed for each given smoothing variance.
In practical implementation, for a given sampling number m and conﬁdence level parameter α , if p (cid:48) is too small, then the condition (12) may not be satisﬁed even for k = 1 .
This implies that the existence of an upper bound of ε that can be certiﬁed for each smoothing parameter σ , recalling that p (cid:48) :
=
Φ (cid:0) Φ − 1 ( p ) − ε √ H / σ (cid:1) .
Detailed Inference and Certiﬁcation Procedures.
We next provide detailed descriptions of the inference and certiﬁcation procedures, respectively.
Inference procedure.
We deploy the σ -randomized policy π (cid:48) as deﬁned in Deﬁnition 4 in Section 5.1.
That is, for each observed state, we sample one time of Gaussian noise ∆ t to add to s t , and take action according to this single randomized observation a t =
π ( s t + ∆ t ) .
Thus, in deployment time, the ∆ -randomized policy π (cid:48) executes on one rollout—at each step it takes one observation and chooses one action (following the procedure described above).
Certiﬁcation procedure.
We sample m randomized trajectories in CROP-GR E instead of sampling m noisy states per time step as in CROP-L O A CT .
For each sampled randomized trajectory, at each time step in the trajectory, we invoke the model with 1 sample of Gaussian noise (as shown in (4)).
Given the m randomized trajectories and the cumulative reward for each of them, we compute the certiﬁcation using Theorem 2 and Theorem 3.
The Algorithm Parameters J min , J max .
We use the default values of J min , J max in the game speciﬁcations rather than estimate them, which are independent with the trained models.
Thus there is no computational cost at all for obtaining the two parameters.
As mentioned under Theoremin Section 5.1, using these default values may induce a loose bound in practice, so we further propose the percentile smoothing that aims to eliminate the dependency of our certiﬁcation on J min and J max , thus achieving a tighter bound.
B. D ETAILED A LGORITHMS OF CROP-L O R E
In the following, we will explain the trajectory exploration and expansion , the growth of perturbation magnitude , and optimization tricks in details.
Trajectory Exploration and Expansion.
CROP-L O R E organizes all possible trajectories in the form of a search tree and progressively grows it.
Each node of the tree represents a state, and the depth of the node is equal to the time step of the corresponding state in the trajectory.
The root node (at depth 0 ) represents the initial state s 0 .
For each node, leveraging Theorem 4, we compute a non-decreasing sequence { r k ( s ) } |A|− 1 k =0 corresponding to required perturbation radii for π to choose each alternative action (the subscript t is omitted for brevity).
Suppose the current ε satisﬁes r i ( s ) ≤ ε < r i +1 ( s ) .
We grow ( i + 1) branches from current state s corresponding to the original action and i alternative actions since ε ≥ r j ( s ) for 1 ≤ j ≤ i .
For nodes on the newly expanded branch, we repeat the same
[Figure]
continue
[Figure]
possible actionsa _ list ← a _ list ∪ { a }else (cid:46) store impossible actions in queue for later expansion p
_ que.
push (( s,a,r,J cur )) return a _ list
[Table] [Table]
[Table] [Table]
procedure to expand the tree with depth-ﬁrst search (Tarjan, 1972) until the terminal state of the game is reached or the node depth reaches H .
As we expand, we keep the record of cumulative reward for each trajectory and update the lower bound J when reaching the end of the trajectory if necessary.
Perturbation Magnitude Growth.
When all trajectories for perturbation magnitude ε are explored, we need to increase ε to seek for certiﬁcation under larger perturbations.
Luckily, since the action space is discrete, we do not need to examine every ε ∈ R + (which is infeasible) but only need to examine the next ε where the chosen action in some step may change.
We leverage priority queue (van
Emde Boas, 1977) to effectively ﬁnd out such next “critical” ε .
Concretely, along the trajectory exploration, at each tree node, we search for the possible actions and store actions corresponding to { r k ( s ) } |A|− 1 k
= i +1 into the priority queue, since these actions are exactly those need to be explored when ε grows.
After all trajectories for ε are fully explored, we pop out the head element from the priority queue as the next node to expand and the next perturbation magnitude ε to grow.
We repeat this process until the priority queue becomes empty or the perturbation magnitude ε reaches the predeﬁned threshold.
Additional Optimization.
We adopt some additional optimization tricks to reduce the complexity of the algorithm.
First, for environments with no negative reward, we perform pruning to limit the tree size—if the cumulative reward leading to the current node already reaches the recorded lower bound, we can perform pruning, since the tree that follows will not serve to update the lower bound.
This largely reduces the potential search space.
We additionally adopt the memorization (Michie, 1968) technique which is commonly applied in search algorithms.
We point out a few more potential improvements.
First , with more speciﬁc knowledge of the game mechanisms, the search algorithm can be further optimized.
Take the Pong game as an example, given the horizontal speed of the ball, we can compress the time steps where the ball is ﬂying between the two paddles, thus reducing the computation.
Second , empirical attacks may be efﬁciently incorporated into the algorithm framework to provide upper bounds that help with pruning.
Time Complexity.
The time complexity of CROP-L O R E is O
( H | S explored | × (log | S explored | + | A | T )) , where | S explored | is the number of explored states throughout the search procedure, which is no larger than cardinality of state set, H is the horizon length,
| A | is the cardinality of action set, and
T is the time complexity of performing local smoothing.
The main bottleneck of the algorithm is the large number of possible states, which is in the worst case exponential to state dimension.
However, to provide a sound worst-case certiﬁcation agnostic to game properties, exploring all possible states may be inevitable.
Detailed Inference and Certiﬁcation Procedures and Important Modules in Algorithm 3.
We next provide detailed descriptions of the inference and certiﬁcation procedures, respectively, along with other important modules.
Inference procedure.
In CROP-L O R E , we deploy the locally smoothed policy ˜ π as deﬁned in (2) in Section 4.1.
Concretely, for each observed state s t , we sample a batch of Gaussian noise to add to s t , and take action according to the computed mean Q value on the batch of randomized observations.
(Actually, the inference procedure per step is exactly the same as in CROP-L O
A CT described in
Appendix B.1.)
Certiﬁcation procedure.
We obtain the certiﬁcation via Theorem 4 and the adaptive search algorithm, which outputs a collection of pairs { ( ε i , J ε i ) } | C | i =1 sorted in ascending order of ε i , where | C | is the length of the collection.
The interpretation for this collection of pairs is provided below.
For all ε (cid:48) , let i be the largest integer such that ε i ≤ ε (cid:48) <
ε i +1 , then as long as the perturbation magnitude ε ≤ ε (cid:48) , the cumulative reward J ε (˜ π ) ≥ J ε i .
This is supported by the fact that all certiﬁed radii associated with all nodes in the entire expanded tree compose a discrete set of ﬁnite cardinality; and the perturbation value between two adjacent certiﬁed radii in the returned collection will not lead to a different tree from the tree corresponding to the largest smaller perturbation magnitude.
This actually is also the inspiration for the certiﬁcation design.
Important modules.
The function G ET A CTION computes the possible actions at a given state s under the limit ε , while the procedure E XPAND accomplishes the task of expanding upon a given node/state.
The main part of the algorithm involves a loop that repeatedly selects the next element from the priority queue, i.e. , a node associated with an ε value, to expand upon.
Additional Clariﬁcations.
We clarify that the algorithm only requires access to the environment such that it can obtain the reward and next state via interacting with the environment E ( i.e. , taking action a at state s and obtain next action s (cid:48) and reward r ), but does not require access to an oracle transition function Γ .
We further emphasize that access to the environment that supports back-tracking is already sufﬁcient for conducting our adaptive search algorithm.
C D ISCUSSION ON THE C ERTIFICATION M ETHODS
We discuss the advantages and limitations of our certiﬁcation methods, as well as possible direct extensions , hoping to pave the way for future research along similar directions.
We also provide more detailed analysis that help with the understanding of our algorithms.
CROP-L O A CT .
The algorithm provides state-wise robustness certiﬁcation in terms of the stability/consistency of the per-state action.
It treats each time step independently , smooths the given state at the given time step, and provides the corresponding certiﬁcation.
Thus, one potential extension is to expand the time window from one time step to a consecutive sequence of several time steps, and provide certiﬁcation for the sequences of actions for the given window of states.
CROP-GR E .
As explained in Section 5, the expectation bound J E is too loose to have any practical usage.
In comparison, percentile bound J p is much tighter and practical.
However, one limitation of J p is that there exists an upper bound of the attack magnitude ε that can be certiﬁed for each σ , as explained in Appendix B.2.
For attack magnitudes that exceed the upper bound, we can obtain no useful information via this certiﬁcation (though the upper bound is usually sufﬁciently large).
One limitation of CROP-GR E .
Lemma 2 requires knowing the noise added to each step beforehand so as to generate m randomized trajectories given the known noise sequence { δ t } H − 1 t =0 .
Thus, it cannot be applied against an adaptive attacker.
We clarify that our CROP-L O A
CT and CROP-L O R E does not have such limitation.
CROP-L O R E .
The algorithm provides the absolute lower bound J of the cumulative reward for any ﬁnite-horizon trajectory with a given initial state.
The advantage of the algorithm is that J is an absolute lower bound that bounds the worst-case situation (apart from the exceptions due to the probabilistic conﬁdence α ), rather than the statistical lower bounds J E and J p that characterize the statistical properties of the random variable J ε .
One potential pitfall in understanding J .
What CROP-L O R E certiﬁes is the lower bound for the trajectory with a given initial state s 0 ∼ d 0 , rather than all possible states in d 0 .
This is because our search starts from a root node of the tree, which is set to the ﬁxed given state s 0 .
Conﬁdence of CROP-L O R E .
Regarding the conﬁdence of our probabilistic certiﬁcation, we clarify that we consider the independent multiple-test.
Concretely, due to the independence of decision making errors, the conﬁdence is (1 − α ) N , where N is the maximum number of possible attacked states explored by CROP-L O R E .
Formally,
Pr[ CROP-L O R E certiﬁcation holds ] =
[Equation]
Pr[ not make error on s | not make error on s pre ]
[Equation]
all
[Equation]
all
[Equation] [Equation]
In the above equation, we can see that (i) leverages the independence which in turn gives a bound of form (1 − α ) N .
This is because in each step, the event of “certiﬁcation does not hold” is independent since we sample Gaussian noise independently .
Leveraging such independence, we obtain a conﬁdence lower bound of (1 − α ) N .
From (ii), we see that the conﬁdence is only related to the number of possible attacked states N .
This is because for unattackable states, the certiﬁcation deterministically holds since there is no attack at current step.
Therefore, we only need to count the conﬁdence intervals from all possibly attackable steps instead of all states explored.
We remark that in practice, the attacker usually has the ability to perturb only a limited number of steps, i.e. , N is typically small.
Therefore, the conﬁdence is non-trivial.
Main limitation of CROP-L O R E .
The main limitation is the high time complexity of the algorithm (details see Appendix B.3).
The algorithm has exponential time complexity in worst case despite the existence of several optimizations.
Therefore, it is not suitable for environments with a large action set, or when the horizon length is set too long.
Extension to Policy-based Methods.
Though we speciﬁcally study the robustness certiﬁcation in
Q-learning in this paper, our two certiﬁcation criteria and three certiﬁcation strategies can be readily extended to policy-based methods.
The intuition is that, instead of smoothing the value function in the Q-learning setting, we directly smooth the policy function in policy-based methods.
With the smoothing module replaced and the theorems updated, other technical details in the algorithms for certifying the per-state action and the cumulative reward would then be similar.
D A DDITIONAL E XPERIMENTAL D ETAILS D. D ETAILS OF THE A TARI G AME E NVIRONMENT
We experiment with two Atari-2600 environments in OpenAI Gym (Brockman et al., 2016) on top of the Arcade Learning Environment (Bellemare et al., 2013).
The states in the environments are high dimensional color images ( 210 × 160 × 3 ) and the actions are discrete actions that control the agent to accomplish certain tasks.
Concretely, we use the NoFrameskip-v4 version for our experiments, where the randomness that inﬂuences the environment dynamics can be fully controlled by setting the random seed of the environment at the beginning of an episode.
D. D ETAILS OF THE RL M ETHODS
We introduce the details of the nine RL methods we evaluated.
Speciﬁcally, for each method, we ﬁrst discuss the algorithm, and then present the implementation details for it.
StdTrain (Mnih et al., 2013): StdTrain (naturally trained DQN model) is an algorithm based on
Q-learning, in which DQN is used to reprent Q-value function and TD-loss is used to optimize DQN.
Our StdTrain model is implemented with Double DQN (Van Hasselt et al., 2016) and Prioritized
Experience Replay (Schaul et al., 2015).
GaussAug (Behzadan & Munir, 2017b):
The algorithm of GaussAug is similar with AtdTrain, in which appropriate Gaussian random noises added to states during training.
As for implementation, our GaussAug model is based on the same architecture with the same set of basic training techniques of StdTrain, and adds Gaussian random noise with σ = 0 .
005 , which has the best testing performance under attacks compared with other σ , to all the frames during training.
AdvTrain (Behzadan & Munir, 2017b): Instead of using the original observation to train Double
DQN (Van Hasselt et al., 2016), AdvTrain generates adversarial perturbations and applies the perturbations to part of or all of the frames when training.
In our case, we add the perturbations generated by 5 -step PGD attack to 50% of the frames to make a balance between between stable training and effectiveness of adversarial training.
SA-MDP (PGD) and SA-MDP (CVX) (Zhang et al., 2020): Instead of utilizing adversarial noise to train models directly like AdvTrain, SA-MDP (PGD) and SA-MDP (CVX) regularize the loss function with the help of adversarial noise during training in order to train empirically robust RL agents.
In our experiment settings, which is the same as SA-MDP (PGD) and SA-MDP (CVX) (Zhang et al., 2020), models consider adversarial noise with (cid:96) ∞ -norm ε
= 1 / 255 when solving the maximization for the regularizer and minimize the original TD-loss concurrently.
RadialRL (Oikarinen et al., 2020):
With the similar idea of adding regularization during training of
SA-MDP (PGD) and SA-MDP (CVX), RadialRL compute the worst-case loss instead of regularizing of the bound of the difference of policy distribution under original states and adversarial states.
In our case, the RadialRL model uses a linearly increasing perturbation magnitude ε (from 0 to 1 / 255 ) to compute the adversarial loss.
CARRL (Everett et al., 2021): Unlike other methods which try to improve robustness of RL agents during training, CARRL enhances RL agents’ robustness during testing time.
It computes the lower bound of each action’s Q value at each step and take the one with the highest lower bound conservatively.
Given that the lower bound is derived via neural network veriﬁcation methods (Gowal et al., 2018; Weng et al., 2018), CARRL can only be applied to low dimensional environments (Everett et al., 2021).
Thus, we only evaluate CARRL on CartPole and Highway, where we set the bound of the (cid:96) 2 norm of the perturbation be ε = 0 .
1 and ε = 0 .
05 respectively
when computing the lower bound of Q value, which demonstrate the best empirical performance under noise and empirical attacks.
NoisyNet (Fortunato et al., 2017): Instead of the conventional exploration heuristics for DQN and
Dueling agents which is ε -greedy algorithms, NoisyNet adds parametric noise to DQN’s weights, which is claimed to aid efﬁcient exploration and yield higher scores than StdTrain.
Concretely, our
NoisyNet shares the same basic architecture with StdTrain, and samples the network weights during training and testing, where the weights are sampled from a Gaussian distribution with σ = 0 .
GradDQN (Pattanaik et al., 2018): GradDQN is a variation of AdvTrain, which utilizes conditional value of risk (CVaR) as optimization criteria for robust control instead standard expected long term return.
This method maximizes the expected return over worst α percentile of returns, which can prevent the adversary from possible bad states.
In our case, we generate the adversarial states with 10-step attack by calculating the direction of gradient and sampling in that direction from beta 1 ,distribution in each step.
For StdTrain, SA-MDP (PGD), SA-MDP (CVX), RadialRL, we directly evaluate the trained models provided by the authors 1 .
D. R ATIONALES FOR S ETTINGS AND E XPERIMENTAL D ESIGNS
Finite Horizon Setting.
In this paper, we focus on a ﬁnite horizon setting, mainly for the evaluation purpose—we would need to compute the (certiﬁed and empirical) cumulative rewards from ﬁnite rollouts.
We clarify that our criteria, theorem, and algorithm are applicable to the inﬁnite horizon case as well.
The Role of Practical Attacks in Our Evaluation.
The goal of this paper is to provide certiﬁcation that comes with theoretical guarantees, and therefore following the standard certiﬁed robustness literatures (Cohen et al., 2019; Li et al., 2020; Salman et al., 2019; Jeong & Shin, 2020), there is no need to evaluate the empirical attacks, since the certiﬁcation always holds as long as the attacks satisfy certain conditions regardless of the actual attack algorithms.
The attacks we evaluated in our paper were only for the demonstration purpose and in the hope of providing an example.
D. D ETAILED E VALUATION S ETUP FOR A TARI G AMES
We introduce the detailed evaluation setups for the three certiﬁcation methods corresponding to the experiments in Section 6.
Evaluation Setup for CROP-L O A CT .
We report results averaged over 10 episodes and set the length of the horizon H = 500 .
At each time step, we sample m = 10 , 000 noisy states for smoothing.
When applying Hoeffding’s inequality, we adopt the conﬁdence level parameter α = 0 .
Since the input state observations for the two Atari games are in image space, we rescale the input states such that each pixel falls into the range [0 , 1] .
When adding Gaussian noise to the rescaled states, we sample Gaussian noise of zero mean and different variances.
Concretely, the standard deviation σ is selected among { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 03 , 0 . 05 , 0 . 1 , 0 . 5 , 0 .
We evaluate different parameters for different environments.
Evaluation Setup for CROP-GR E .
We sample m = 10 , 000 σ -randomized trajectories, each of which has length H = 500 , and conduct experiments with the same set of smoothing parameters as in the setup of CROP-L O A CT .
When accumulating the reward, we set the discount factor γ = 1 . 0 with no discounting.
We take α = 0 .
as the conﬁdence level when applying Hoeffding’s inequality in the expectation bound J E , and α = 0 .
as the conﬁdence level for C OMPUTE O RDER S TATS in the percentile bound J p .
For the validation of tightness, concretely, we carry out a 10 -step PGD attack with (cid:96) 2 radius within the perturbation bound at all time steps during testing to evaluate the tightness of our certiﬁcation.
Evalution Setup for CROP-L O R E .
We set the horizon length as H = 200 and sample m = 10 , 000 noisy states with the same set of smoothing variance as in the setup of CROP-L O A CT .
Similarly, we adopt γ = 1 .
as the discount factor and α = 0 .
as the conﬁdence level when applying Hoeffding’s inequality.
When evaluating Freeway, we modify the reward mechanism so that losing one ball incurs a zero score rather than a negative score.
This modiﬁcation enables the pruning operation and therefore facilitates the search algorithm, yet still respects the goal of the game.
We also empirically attack the policy ˜ π to validate the tightness of the certiﬁcation via the same PGD attack as described above.
For each set of experiments, we run the attack one time with the same initial state as used for computing the lower bound in Algorithm 3.
D. E XPERIMENTAL S ETUP FOR C ART P OLE
We additionally experiment with CartPole-v0 in OpenAI Gym (Brockman et al., 2016) on top of the
Arcade Learning Environment (Bellemare et al., 2013).
We introduce the experimental setup below.
Details of the CartPole Environment.
The state in the CartPole game is a low dimensional
Implementation Details of the RL Methods on CartPole.
In addition to the eight RL Methods evaluated on Pong and Freeway as shown in Section 6, we evaluate another RL algorithm CARRL (Everett et al., 2021), which is claimed to be robust in low dimensional games like CartPole.
This method relies on linear bounds output by the Q network and selects the action whose lower bound of the Q value is the highest.
More detailed introduction to the algorithm and description of the implementation details are provided in Appendix D.2.
Evaluation Setup for CROP-L O A CT on CartPole.
We report results averaged over 10 episodes and set the length of the horizon H = 200 .
At each time step, we sample m = 10 , 000 noisy states for smoothing.
When applying Hoeffding’s inequality, we adopt the conﬁdence level parameter α = 0 .
We do not perform rescaling on the state observations.
When adding Gaussian noise to states, we sample Gaussian noise of zero mean and different variances.
Concretely, the standard deviation σ is selected among { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 03 , 0 . 05 , 0 . 1 } .
Evaluation Setup for CROP-GR E on CartPole.
We sample m = 10 , 000 σ -randomized trajectories, each of which has length H = 200 , and conduct experiments with the same set of smoothing parameters as in the setup of CROP-L O A CT .
All other experiment settings are the same as Pong and
Freeway in Appendix D.4.
Evalution Setup for CROP-L O R E on CartPole.
Since the reward of CartPole game is quite dense, we set the horizon length as H = 10 and sample m = 10 , 000 noisy states with the same set of smoothing variance as in the setup of CROP-L O A CT .
All other experiment settings are the same as Pong and Freeway in Appendix D.4.
D. E XPERIMENTAL S ETUP FOR H IGHWAY
In addition to the previous environments in OpenAI Gym (Brockman et al., 2016) on the Arcade
Learning Environment (Bellemare et al., 2013), we also evaluate our methods in the Highway environment (Leurent, 2018).
Speciﬁcally, we test out algorithms in the highway-fast-v0 environment.
Details of the highway Environment.
The state in the highway-fast-v0 environment is a 5 ×matrix, where each line represents a feature vectors of either the ego vehicle or other vehicles closest to the ego vehicle.
The feature vector for each vehicle has the form of [ x, y, vx, vy, 1] , corresponding to the two-dimensional positions and velocities and an additional indicator value.
The actions are high-level control actions among { lane left , idle , lane right , faster , slower }.
The reward mechanism in this environment is associated with the status of the ego vehicle; more concretely, it gives higher reward for the vehicle when it stays at the rightmost lane, moves at high speed, and does not crash into other vehicles.
Evaluation Setup for CROP-L O A CT on highway.
We report results averaged over 10 episodes and set the length of the horizon H = 30 , which is the maximum length of an episode in the environment conﬁguration.
At each time step, we sample m = 10 , 000 noisy states for smoothing.
When applying Hoeffding’s inequality, we adopt the conﬁdence level parameter α = 0 .
We do not perform rescaling on the state observations.
When adding Gaussian noise to states, we sample
Gaussian noise of zero mean and different variances.
Concretely, the standard deviation σ is selected among { 0 . 005 , 0 . 05 , 0 . 1 , 0 . 5 , 0 .
Evaluation Setup for CROP-GR E on highway.
We sample m = 10 , 000 σ -randomized trajectories (each of which has length H = 30 ) and conduct experiments with the same set of smoothing parameters as in the setup of CROP-L O A CT .
All other experiment settings are the same as Pong and
Freeway in Appendix D.4.
Evalution Setup for CROP-L O R E on highway.
Since the reward of highway game is also quite dense, we set the horizon length as H = 10 as in the CartPole environment and sample m = 10 ,noisy states with the same set of smoothing variance as in the setup of CROP-L O A CT .
All other experiment settings are the same
Pong and Freeway in Appendix D.4.
[Table]
[Figure] [Figure]
E A DDITIONAL E VALUATION R ESULTS AND D ISCUSSIONS E.
R OBUSTNESS C ERTIFICATION FOR P ER - STATE
A CTION – C ERTIFIED R ATIO
We present the robustness certiﬁcation for per-state action in terms of the certiﬁed ratio η r w.r.t.
the certiﬁed radius r in Figure 4.
From the ﬁgure, we see that RadialRL is the most certiﬁably robust method on Freeway, followed by SA-MDP (CVX) and SA-MDP (PGD); while on Pong, SA-MDP (CVX) is the most robust.
These conclusions are highly consistent with those in Section 6.1 as observed from Figure 1.
Comparing the curves for Freeway and Pong, we note that Freeway not only achieves larger certiﬁed radius overall , it also more frequently attains large certiﬁed radius.
E. R ESULTS OF F REEWAY UNDER L ARGER S MOOTHING P ARAMETER σ
Since the robustness of the methods RadialRL, SA-MDP (CVX), and SA-MDP (PGD) on Freeway have been shown to improve with the increase of the smoothing parameter σ in Section 6, we subsequently evaluate their performance under even larger σ values.
We present the certiﬁcation results in Figure 5 and their benign performance in Figure 6.
[Figure]
First of all, as Figure 6 reveals, RadialRL, SA-MDP (CVX), and SA-MDP (PGD) can tolerate quite large noise—the benign performance of these methods does not drop even for σ up to 4 . 0 .
For the remaining three methods, the magnitude of noise they can tolerate is much smaller, as already presented in Figure 2 in the main paper.
On the other hand, although all the three methods RadialRL, SA-MDP (CVX), and SA-MDP (PGD) invariably attain good benign performance as σ grows, their certiﬁed robustness does not equally increase.
As Figure 5 shows, the certiﬁed robustness of SA-MDP (PGD) increases the fastest, while that of RadialRL drops a little.
The observa- tion corresponds exactly to our discussion regarding the tradeoff between value function smoothness and the margin between the values to the top two actions in Section 4.
E. P ERIODIC P ATTERNS FOR P ER -S TATE R OBUSTNESS
We speciﬁcally study the periodic patterns for per-state robustness in Pong and present the results in Figure 7.
In the game frames, our agent controls the green paddle on the right.
Whenever the opponent (the orange paddle on the left) misses catching a ball, we earn a point.
We present two periods (frame 320-400 and frame 400-480) where our agent each earns a point in the ﬁgure above.
The total points increase from 2 to 3 , and from 3 to 4 , respectively.
We note that Pong is a highly periodic game, with each period consisting of the following four game stages corresponding to distinct stages of the certiﬁed robustness : 1) A ball is initially ﬁred and set ﬂying towards the opponent.
In this stage, the certiﬁed radius remains low, since our agent does not need to take any critical action.
The opponent catches the ball and reverses the direction of the ball.
In this stage, the certiﬁed radius gradually increases, since the moves of our agent serve as the preparation to catch the ball.
The ball bounces from the ground, ﬂies towards our agent, gets caught, and bounces back.
In this stage, the certiﬁed radius increases rapidly, since every move is critical in determining whether we will be able to catch the ball, i.e. , whether our point (action value) will increase.
The opponent misses the ball, and the ball disappears from the game view.
In this stage, the certiﬁed radius drastically plummets to a low value, since there is no clear signal ( i.e. , ball) in the game for the agent to take any determined action.
The illustration not only helps us understand the semantic meaning of the certiﬁed radius, but can further provide guidance on designing empirically robust RL algorithms that leverage the information regarding the high/low certiﬁed radius of the states.
E. D ETAILED D ISCUSSIONS AND S ELECTION OF S MOOTHING P ARAMETER σ
We ﬁrst take Freeway as an example to concretely explain the impact of smoothing variance from the perspective of benign performance and certiﬁed results, and then introduce the guidelines for the selection of σ based on different certiﬁcation criteria.
On Freeway , as σ increases, the benign performance of the locally smoothed policy ˜ π of StdTrain,
GaussAug, and AdvTrain decrease rapidly in Figure 2.
Also, the certiﬁed radius of these methods barely gets improved and remains low even as σ increases.
For the percentile bound J p , these three methods similarly suffer a drastic decrease of cumulative reward as σ grows.
This is however not the case for SA-MDP (PGD), SA-MDP (CVX), and RadialRL, for which not only the benign performance do not experience much degradation for σ as large as 1 . 0 , but the certiﬁed radius even steadily increases.
In addition, in terms of the percentile bound J p , as σ grows, even we increase the attack ε simultaneously, J p almost does not decrease.
This indicates that larger smoothing variance can bring more robustness to SA-MDP (PGD), SA-MDP (CVX), and RadialRL.
CROP-L O A CT .
We discuss how to ﬁnd the sweet spot that enables high benign performance and robustness simultaneously.
As σ increases, the benign performance of different methods generally decreases due to the noise added, while not all methods decrease at the same rate.
On Freeway ,
StdTrain and GaussAug decrease the fastest, while SA-MDP (PGD), SA-MDP (CVX), and RadialRL do not degrade much even for σ as large as 4 . 0 .
Referring back to Figure 1 and Figure 5, we see that the certiﬁed radius of StdTrain and GaussAug remains low even though σ increases, thus the best parameters for these two methods are both around 0 .
For SA-MDP (PGD) and SA-MDP (CVX), their certiﬁed radius instead steadily increases as σ increases to 4 . 0 , indicating that larger smoothing variance can bring more robustness to these two methods without sacriﬁcing benign performance, offering certiﬁed radius larger than 2 . 0 .
As to
RadialRL, its certiﬁed radius reaches the peak at σ = 1 .
and then decreases under larger σ .
Thus, 1 . 0 is the best smoothing parameter for RadialRL which offers a certiﬁed radius of 1 . 6 .
Generally speaking, it is feasible to select an appropriate smoothing parameter σ for all methods to increase their robustness, and robust methods will beneﬁt more from this by admitting larger smoothing variances and achieving larger certiﬁed radius.
CROP-GR E .
We next discuss how to leverage Figure 3 to assist with the selection of σ .
For a given ε or a known range of ε , we can compare between the lower bounds corresponding to different σ and select the one that gives the highest lower bound.
For example, if we know that ε is small in advance, we can go for a relatively small σ which retains a quite high lower bound.
If we know that ε will be large, we will instead choose among the larger σ ’s, since a larger ε will not fall in the certiﬁable ranges of smaller σ ’s.
According to this guideline, we are able to obtain lower bounds of quite high value for RadialRL, SA-MDP (CVX), and SA-MDP (PGD) on Freeway under a larger range of attack ε .
CROP-L O R E .
Apart from the conclusion that larger smoothing parameter σ often secures higher lower bound J , we point out that smaller smoothing variances may be able to lead to a higher lower bound than larger smoothing variances for a certain range of ε .
This is almost always true for very small ε since smaller σ is sufﬁcient to deal with the weak attack without sacriﬁcing much empirical performance, e.g. , in Figure 3, when ε = 0 .
001 , SA-MDP (CVX) on Freeway can achieve J = 3 at σ = 0 .
005 while only J = 1 for large σ .
This can also happen to robust methods at large ε , e.g. , when ε = 1 . 2 , RadialRL on Freeway achieves J = 2 at σ = 0 .
75 while only J = 1 at σ = 1 . 0 .
Another case is when σ is large enough such that further increasing σ will not bring additional robustness, e.g. , in Figure 5, when ε = 1 . 5 , RadialRL on Freeway achieves J = 1 at σ = 1 . 5 while only J =at σ = 4 . 0 .
E. C OMPUTATIONAL C OST
Our experiments are conducted on GPU machines, including GeForce RTX 3090, GeForce RTX 2080 Ti, and GeForce RTX 1080 Ti.
The running time of m = 10000 forward passes with sampled states for per-state smoothing ranges from 2 seconds to 9 seconds depending on the server load.
Thus, for one experiment of CROP-L O A CT with trajectory length H = 500 and 10 repeated runs, the running time ranges from 2 .
5 hours to 12 .
5 hours.
CROP-GR E is a little more time-consuming than
CROP-L O A CT , but the running time is still in the same magnitude.
For trajectory length H =and sampling number m = 10000 , most of our experiments ﬁnish within 3 hours.
For CROP-L O R E speciﬁcally, the running time depends on the eventual size of the search tree, and therefore differs tremendously for different methods and different smoothing variances, ranging from a few hours toto 5 days.
E. D ISCUSSION ON E VALUATION R ESULTS OF C ART P OLE
We provide the evaluation results for CROP on CartPole, comparing nine RL algorithms: the six algorithms evaluated in our main paper (StdTrain, GaussAug, AdvTrain, SA-MDP (PGD), SA-MDP (CVX), and RadialRL), as well as three additional algorithms (CARRL, NoisyNet, and GradDQN).
The detailed descriptions of these algorithms are provided in Appendix D.2.
We present the evaluation results in Figure 8.
Impact of Smoothing Parameter σ .
In CartPole game, we draw similar conclusions regarding the impact of smoothing variance from the results given by CROP-L O
A CT , CROP-GR E and CROP-
L O R E .
For small σ , different methods are indistinguishable.
As σ increases, CARRL demonstrates its advantage as σ increases, but AdvTrain and GradDQN tolerate large noise.
One of differences when evaluating with CROP-GR E and CROP-L O R E is that RadialRL can hold a high performance and even sometimes can be better than CARRL.
Another one is that the lower bound for GradDQN is lower than all other methods in CROP-L O R E .
Tightness of the certiﬁcation J E , J p and J .
In CartPole, we also compare the empirical cumulative rewards achieved under PGD attacks with our certiﬁed lower bounds J E , J p and J .
Demonstrated by Figure 8c, ﬁrst of all, the correctness of our bounds is validated because the empirical results are consistently lower bounded by our certiﬁcations.
Compared with the loose expectation bound J E , the improved percentile bound J p is much tighter.
Compared with these two methods, the absolute lower bound J is even tighter, especially noticing the zero gap between the certiﬁcation and the empirical result when the attack magnitude is not too large.
Finally, methods with the same empirical results under attack may achieve very different certiﬁed lower bounds J E , J p and J , showing the importance of certiﬁcation.
Environment Properties.
Different from Pong and Freeway, CartPole is an environment with low dimension states , in contrast to the high dimensional Atari games evaluated in Section 6.
This gives rise to several outcomes.
First, StdTrain can tolerate noise well in CartPole.
Second,
CARRL demonstrates its advantage in low dimensional game, which is consistent with the empirical observations in Everett et al.
E. D ISCUSSION ON E VALUATION R ESULTS OF H IGHWAY
We next present the evaluation results for CROP on an autonomous driving environment Highway, whose state dimension is larger than CartPole but smaller than Atari games.
We compare nine RL algorithms as introduced previously.
We present the evaluation results in Figure 9.
Impact of Smoothing Parameter σ .
In highway, CARRL demonstrates its advantage when σ is small, which is supported by results in all three certiﬁcations CROP-L O A CT , CROP-GR E , and
CROP-L O R E .
For action certiﬁcation ( i.e. , CROP-L O A CT ), SA-MDP (PGD) can tolerate a large range of σ , meaning that the action selection of model trained with SA-MDP (PGD) algorithm is more consistent than other methods.
For reward certiﬁcation ( i.e. , CROP-GR E and CROP-L O R E ), we derive the conclusion that RadialRL outperforms than other methods when σ is large.
Tightness of the certiﬁcation J E , J p and J .
In highway, we compare the empirical cumulative rewards achieved under PGD attacks with our certiﬁed lower bounds J E , J p , and J .
The correctness of our bounds is validated because the empirical results are consistently lower bounded by our certiﬁcations, as shown in Figure 9c.
As for the tightness of reward certiﬁcation methods, we draw similar conclusion as other environments— J is the most tight, followed by J p ; and J E is much looser than J p and J E .
We also notice the non-negligible gap between certiﬁed lower bounds and the empirical rewards demonstrated by RadialRL and CARRL under large smoothing parameters.
This may imply that there exists large scopes for improving the attack method, or there is potential to further tighten the certiﬁed lower bounds.
F A B ROADER D ISCUSSION ON R ELATED W ORK F. E VASION A TTACKS IN RL
We consider the adversarial attacks on state observations, where the attacker aims to add perturbations to the state during test time, so as to achieve certain adversarial goals , such as misleading the action selection and minimizing the cumulative reward .
We discuss the related works that fall into these two categories below.
Misleading the Action Selection.
It has been shown that different methods—applying random noise to simply interfere with the action selection (Kos & Song, 2017), or adopting adversarial attacks ( e.g. , FGSM (Goodfellow et al., 2014) and CW attacks (Carlini & Wagner, 2017)) to deliberately alter the probability of action selection—are effective to different degrees.
More concretely, when manipulating the action selection probability, some works aim to reduce the probability of selecting
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table]
attack ε attack ε attack ε attack ε attack ε attack ε the optimal action (Huang et al., 2017; Kos & Song, 2017; Behzadan & Munir, 2017b), while others try to increase the probability of selecting the worst action (Lin et al., 2017; Pattanaik et al., 2018).
Minimizing the Cumulative Reward.
ATLA (Zhang et al., 2021) and PA-AD (Sun et al., 2021) consider an optimal adversary under the SA-MDP framework (Zhang et al., 2020), which aims to lead to minimal value functions under bounded state perturbations.
To ﬁnd this optimal adversary ( i.e. , the optimal adversarial state perturbation), ATLA (Zhang et al., 2021) proposes to train an adversary whose action space is the perturbation set in the state space, while PA-AD (Sun et al., 2021) further decouples the problem of ﬁnding state perturbations into ﬁnding policy perturbations plus ﬁnding the state that achieves the lowest value policy, thus addressing the challenge of large state space.
F. R OBUST RL
Distributionally Robust RL.
Nilim & El Ghaoui (2005) and Iyengar (2005) consider the problem of distributionally robust RL, where there is normally an uncertainty set with prior information regarding the distribution of the uncertain parameters.
Iyengar (2005) directly put constraints on environmental dynamics and reward functions by assuming that they take values in an uncertainty set.
Another line of research assumes the environmental dynamics and reward function as the uncertain parameters and are sampled from an uncertain distribution in the uncertain set.
The uncertainty set is in general state-wise independent, i.e. , “s-rectangularity” in Nilim & El Ghaoui (2005).
Both types aim to derive a policy by solving a max-min problem, with the former taking the minimum directly over the parameters, while the latter taking the minimum over the distribution.
Empirically Robust RL against Evasion Attacks.
We have brieﬂy introduced related work on empirically robust RL in Section 7 in the main paper.
We emphasize two main differences between the contributions of these works and our CROP.
First , most of these robust RL methods only provide empirical robustness against perturbed state inputs during test time, but cannot provide theoretical guarantees for the performance of the trained models under any bounded perturbations; while our
CROP framework can provide practically computable certiﬁed robustness w.r.t.
two robustness certiﬁcation criteria (per-state action stability and cumulative reward lower bound), i.e. , for a given
RL algorithm, we can compute certiﬁcations for it that indicate its robustness.
(We will discuss a few more related work that can provide certiﬁed robustness in Appendix F.3.)
Second , most of these Robust RL methods focus on only the per-state decision, while we additionally consider the certiﬁcation of trajectories to obtain the lower bound of cumulative rewards.
Below, we provide a more comprehensive review of the categories of RL methods that demonstrate empirical robustness against evasion attacks.
Randomization methods (Tobin et al., 2017; Akkaya et al., 2019) were ﬁrst proposed to encourage exploration.
This type of method was later systematically studied for its potential to improve model robustness.
NoisyNet (Fortunato et al., 2017) adds parametric noise to the network’s weight during training, providing better resilience to both training-time and test-time attacks (Behzadan & Munir, 2017b; 2018), also reducing the transferability of adversarial examples, and enabling quicker recovery with fewer number of transitions during phase transition.
Under the adversarial training framework, Kos & Song (2017) and Behzadan & Munir (2017b) show that re-training with random noise and FGSM perturbations increases the resilience against adversarial examples.
Pattanaik et al.
(2018) leverage attacks using an engineered loss function speciﬁcally designed for RL to signiﬁcant increase the robustness to parameter variations.
RS-DQN (Fischer et al., 2019) is an imitation learning based approach that trains a robust student-DQN in parallel with a standard DQN in order to incorporate the constrains such as SOTA adversarial defenses (Madry et al., 2017; Mirman et al., 2018).
SA-DQN (Zhang et al., 2020) is a regularization based method that adds regularizers to the training loss function to encourage the top- 1 action to stay unchanged under perturbation.
Built on top of the neural network veriﬁcation algorithms (Gowal et al., 2018;
Weng et al., 2018),
Radial-RL (Oikarinen et al., 2020) proposes to minimize an adversarial loss function that incorporates the upper bound of the perturbed loss, computed using certiﬁed bounds from veriﬁcation algorithms.
CARRL (Everett et al., 2021) aims to compute the lower bounds of action-values under potential perturbation and select actions according to the worst-case bound, but it relies on linear bounds (Weng et al., 2018) and is only suitable for low-dimensional environments.
F. R OBUSTNESS C ERTIFICATION FOR RL
Despite the abundant literature in robustness certiﬁcation in supervised learning, there is almost no work on robustness certiﬁcation for RL, given the unclear criteria and the intrinsic difﬁculty of the task.
In this part, we ﬁrst introduce another two works that can achieve robustness certiﬁcation at state-level, and then another concurrent work Kumar et al.
(2021) which also aims to provide provable robustness regarding the cumulative reward for RL.
Robustness Certiﬁcation on State Level.
Fischer et al.
and Zhang et al.
(2020) have discussed the state level robustness certiﬁcation as well, but the way they obtain the certiﬁcation is different from our CROP.
Concretely, we achieve robustness by probabilistic certiﬁcation via randomized smoothing (Cohen et al., 2019), while Fischer et al.
and Zhang et al.
(2020) directly achieve robustness by deterministic certiﬁcation via leveraging the neural network veriﬁcation techniques (Zhang et al., 2018; Xu et al., 2020; Mirman et al., 2018).
Despite these works on state-level robustness certiﬁcation for RL, we are the ﬁrst to provide the certiﬁcation of cumulative rewards.
We emphasize that the certiﬁcation of lower bound of the cumulative reward in RL is more challenging than the per-state certiﬁcation considering the dynamic nature of RL.
Our main contribution of the paper indeed mainly focuses on providing an efﬁcient adaptive search based algorithm CROP-L O R E to certify the lower bound of the cumulative reward together with rigorous analysis of the certiﬁcation.
Robustness Certiﬁcation for Cumulative Rewards.
We compare our robustness certiﬁcation for cumulative rewards with another concurrent work Kumar et al.
In terms of the threat model , both Kumar et al.
and CROP consider the type of adversary that can perturb the state observations of the agent during test time.
In our CROP, we consider a perturbation budget per time step following previous works (Huang et al., 2017; Behzadan & Munir, 2017b; Kos & Song, 2017; Pattanaik et al., 2018; Zhang et al., 2020), while they assume a budget for the entire episode.
The two perspectives are closely related.
Regarding the certiﬁcation criteria , we certify both per-state action stability and cumulative reward lower bound .
Although they also consider the cumulative reward in their certiﬁcation goal, they formulate the certiﬁcation as a classiﬁcation problem—certifying whether the cumulative reward is above a threshold or not, in contrast to directly certifying the lower bound as in our case.
For the certiﬁcation technique , both works are developed based on randomized smoothing proposed in supervised learning (Cohen et al., 2019).
We propose a global smoothing technique (CROP-GR E ), as well as an adaptive search algorithm coupled with local smoothing (CROP-L O R E ) to achieve the certiﬁcation of cumulative reward; while they propose an adaptive version of the Neyman-Pearson lemma, which is similar with our global smoothing.
Among the three algorithms (CROP-GR E ,
CROP-L O R E , and Kumar et al.
, CROP-GR E cannot defend against an adaptive adversary (as concretely stated in Appendix C) while the other two can; speciﬁcally, CROP-L O R E is much more sophisticated and tight than the global smoothing ones, which is an important contribution in our work.