Efﬁcient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recent studies reveal that a well-trained deep reinforcement learning (RL) policycan be particularly vulnerable to adversarial perturbations on input observations.
Therefore, it is crucial to train RL agents that are robust against any attackswith a bounded budget.
Existing robust training methods in deep RL either treatcorrelated steps separately, ignoring the robustness of long-term reward, or trainthe agents and RL-based attacker together, doubling the computational burdenand sample complexity of the training process.
In this work, we propose a strongand efﬁcient robust training framework for RL, named Worst-case-aware Robust
RL (WocaR-RL), that directly estimates and optimizes the worst-case reward of apolicy under bounded ` p attacks without requiring extra samples for learning anattacker.
Experiments on multiple environments show that WocaR-RL achievesstate-of-the-art performance under various strong attacks, and obtains signiﬁcantlyhigher training efﬁciency than prior state-of-the-art robust training methods.
Introduction
Deep reinforcement learning (DRL) has achieved impressive results by using deep neural networks(DNN) to learn complex policies in high-dimensional environments.
However, well-trained DNNsmay drastically fail under adversarial perturbations of the input [1, 5].
Therefore, before deploying
DRL policies to real-life applications, there is a crucial need to improve the robustness of deep policiesagainst adversarial attacks, especially worst-case attacks that maximally depraves the performance oftrained agents [41].
A line of regularization-based robust methods [52, 32, 39] focuses on improving the robustness of the DNN itself and regularizes the policy networkto output similar actions under bounded state perturbations.
However, different from supervised learning problems, the vulnerability of a deep policycomes not only from the DNN approximator, but also from the dynamicsof the RL environment [50].
These regularization-based methods neglectthe intrinsic vulnerability of policies under the environment dynamics, andthus may still fail under strong attacks [41].
For example, in the go-hometask shown in Figure 1, both the green policy and the red policy arrivehome without rock collision, when there is no attack.
However, althoughregularization-based methods may ensure a minor action change under astate perturbation, the red policy may still be susceptible to a low rewardunder attacks, as a very small divergence can lead it to the bomb.
On the contrary, the green policy ismore robust to adversarial attacks since it stays away from the bomb.
Therefore, besides promotingthe robustness of DNN approximators (such as the policy network), it is also important to learn apolicy with stronger intrinsic robustness.
There is another line of work considering the long-term robustness of a deep policy under strongadversarial attacks.
In particular, it is theoretically proved [52, 41] that the strongest (worst-case)attacker against a policy can be learned as an RL problem, and training the agent under such alearned attacker can result in a robust policy.
Zhang et al.
[50] propose the Alternating Training with
Learned Adversaries (ATLA) framework, which alternately trains an RL agent and an RL attacker.
Sun et al.
[41] further propose PA-ATLA, which alternately trains an agent and the proposed moreefﬁcient PA-AD RL attacker, obtaining state-of-the-art robustness in many MuJoCo environments.
However, training an RL attacker requires extra samples from the environment, and the attacker’s
RL problem may even be more difﬁcult and sample expensive to solve than the agent’s original RLproblem [50, 41], especially in large-scale environments such as Atari games with pixel observations.
Therefore, although ATLA and PA-ATLA are able to achieve high long-term reward under attacks,they double the computational burden and sample complexity to train the robust agent.
The above analysis of existing literature suggests two main challenges in improving the adversarialrobustness of DRL agents: (1) correctly characterizing the long-term reward vulnerability of an
RL policy, and (2) efﬁciently training a robust agent without requiring much more effort thanvanilla training.
To tackle these challenges, in this paper, we propose a generic and efﬁcient robusttraining framework named Worst-case-aware Robust RL (WocaR-RL) that estimates and improves thelong-term robustness of an RL agent.
WocaR-RL has 3 key mechanisms.
First, WocaR-RL introduces a novel worst-attack Bellmanoperator which uses existing off-policy samples to estimate the lower bound of the policy value underthe worst-case attack.
Compared to prior works [50, 41] which attempt to learn the worst-case attackby RL methods, WocaR-RL does not require any extra interaction with the environment.
Second,using the estimated worst-case policy value, WocaR-RL optimizes the policy to select actions thatnot only achieve high natural future reward, but also achieve high worst-case reward when there areadversarial attacks.
Therefore, WocaR-RL learns a policy with less intrinsic vulnerability.
Third,
WocaR-RL regularizes the policy network with a carefully designed state importance weight.
Asa result, the DNN approximator tolerates state perturbations, especially for more important stateswhere decisions are crucial for future reward.
The above 3 mechanisms can also be interpreted froma geometric perspective of adversarial policy learning, as detailed in Appendix B.
Our contributions can be summarized as below.
(1) We provide an approach to estimate the worstcase value of any policy under any bounded ` p adversarial attacks.
This helps evaluate the robustnessof a policy without learning an attacker which requires extra samples and exploration.
(2) Wepropose a novel and principled robust training framework for RL, named Worst-case-aware
Robust
RL (WocaR-RL) , which characterizes and improves the worst-case robustness of an agent.
WocaRRL can be used to robustify existing DRL algorithms (e.g. PPO [38], DQN [31]).
(3) We showby experiments that WocaR-RL achieve improved robustness against various adversarial attacksas well as higher efﬁciency , compared with state-of-the-art (SOTA) robust RL methods in many
MuJoCo and Atari games.
For example, compared to the SOTA algorithm PA-ATLA-PPO [41] in the
Walker environment, we obtain 20% more worst-case reward (under the strongest attack algorithm),with only about 50% training samples and 50% running time.
Moreover, WocaR-RL learns moreinterpretable “robust behaviors” than PA-ATLA-PPO in Walker as shown in Figure 2.
[Figure]
Related Work
Defending against Adversarial Perturbations on State Observations.
(1) Regularization-basedmethods [52, 39, 32] enforce the policy to have similar outputs under similar inputs, which achievescertiﬁable performance for DQN in some Atari games.
But in continuous control tasks, these methodsmay not reliably improve the worst-case performance.
A recent work by Korkmaz [20] points outthat these adversarially trained models may still be sensible to new perturbations.
(2) Attack-drivenmethods train DRL agents with adversarial examples.
Some early works [21, 4, 28, 33] apply weak orstrong gradient-based attacks on state observations to train RL agents against adversarial perturbations.
Zhang et al.
[50] propose Alternating Training with Learned Adversaries (ATLA), which alternatelytrains an RL agent and an RL adversary and signiﬁcantly improves the policy robustness in continuouscontrol games.
Sun et al.
[41] further extend this framework to PA-ATLA with their proposed moreadvanced RL attacker PA-AD.
Although ATLA and PA-ATLA achieve strong empirical robustness,they require training an extra RL adversary that can be computationally and sample expensive.
There is another line of work studying certiﬁable robustness of RL policies.
Several works [26, 32, 8]computed lower bounds of the action value network Q ⇡ to certify robustness of action selection atevery step.
However, these bounds do not consider the distribution shifts caused by attacks, so someactions that appear safe for now can lead to extremely vulnerable future states and low long-termreward under future attacks.
Moreover, these methods cannot apply to continuous action spaces.
Kumar et al.
and Wu et al.[22, 48] both extend randomized smoothing [6] to derive robustnesscertiﬁcates for trained policies.
But these works mostly focus on theoretical analysis, and effectiverobust training approaches rather than robust training.
Adversarial Defenses against Other Adversarial Attacks.
Besides observation perturbations,attacks can happen in many other scenarios.
For example, the agent’s executed actions can beperturbed [49, 43, 44, 23].
Moreover, in a multi-agent game, an agent’s behavior can create adversarialperturbations to a victim agent [12].
Pinto et al.
[34] model the competition between the agent andthe attacker as a zero-sum two-player game, and train the agent under a learned attacker to tolerateboth environment shifts and adversarial disturbances.
We point out that although we mainly considerstate adversaries, our WocaR-RL can be extended to action attacks as formulated in Appendix C.5.
Note that we focus on robustness against test-time attacks, different from poisoning attacks whichalter the RL training process [3, 19, 40, 54, 35].
Safe RL and Risk-sensitive RL.
There are several lines of work that study RL under safety/riskconstraints [17, 10, 9, 2, 45] or under intrinsic uncertainty of environment dynamics [25, 29].
However,these works do not deal with adversarial attacks, which can be adaptive to the learned policy.
Morecomparison between these methods and our proposed method is discussed in Section 4.
Preliminaries and Background
Reinforcement Learning (RL).
An RL environment is modeled by a Markov Decision Process(MDP), denoted by a tuple M = h S , A , P, R,  i , where S is a state space, A is an action space, P :
S ⇥ A !
( S ) is a stochastic dynamics model 1 , R :
S ⇥ A !
R is a reward function and  2 [0 , 1) is 115 a discount factor.
An agent takes actions based on a policy ⇡ :
S !
( A ) .
For any policy, its natural 116 performance can be measured by the value function V ⇡ ( s ) :
= E P, ⇡ [
P 1 t =
t R ( s t , a t ) | s 0 = s ] ,and the action value function Q ⇡ ( s, a ) :=
E P, ⇡ [
P 1 t =
t R ( s t , a t ) | s 0 =
s, a 0 = a ] .
We call V ⇡the natural value and Q ⇡ the natural action value in contrast to the values under attacks, as will beintroduced in Section 4.
Deep Reinforcement Learning (DRL).
In large-scale problems, a policy can be parameterized bya neural network.
For example, value-based RL methods (e.g. DQN [31]) usually ﬁt a Q network andtake the greedy policy ⇡ ( s ) = argmax a Q ( s, a ) .
In actor-critic methods (e.g. PPO [38]), the learnerdirectly learns a policy network and a critic network.
In practice, an agent usually follows a stochasticpolicy during training that enables exploration, and executes a trained policy deterministically intest-time, e.g. the greedy policy learned with DQN.
Throughout this paper, we use ⇡ ✓ to denote thetraining-time stochastic policy parameterized by ✓ , while ⇡ denotes the trained deterministic policythat maps a state to an action.
Test-time Adversarial Attacks.
After training, the agent is deployed into the environment andexecutes a pre-trained ﬁxed policy ⇡ .
An attacker/adversary, during the deployment of the agent, may perturb the state observation of the agent/victim at every time step with a certain attack budget ✏ .
Notethat the attacker only perturbs the inputs to the policy, and the underlying state in the environmentdoes not change.
This is a realistic setting because real-world observations can come from noisysensors or be manipulated by malicious attacks.
For example, an auto-driving car receives sensoryobservations; an attacker may add imperceptible noise to the camera, or perturb the GPS signal,although the underlying environment (the road) remains unchanged.
In this paper, we consider the ` pthread model which is widely used in adversarial learning literature: at step t , the attacker alters theobservation s t into ˜ s t 2 B ✏ ( s t ) , where B ✏ ( s t ) is a ` p norm ball centered at s t with radius ✏ .
The abovesetting ( ` p constrained observation attack) is the same with many prior works [18, 33, 52, 50, 41].
Worst-case-aware Robust RL
In this section, we present Worst-case-aware Robust RL (WocaR-RL) , a generic framework that canbe fused with any DRL approach to improve the adversarial robustness of an agent.
We will introducethe three key mechanisms in WocaR-RL: worst-attack value estimation, worst-case-aware policyoptimization, and value-enhanced state regularization, respectively.
Then, we will illustrate how toincorporate these mechanisms into existing DRL algorithms to improve their robustness.
Mechanism 1: Worst-attack Value Estimation
Traditional RL aims to learn a policy with the maximal value V ⇡ .
However, in a real-world problemwhere observations can be noisy or even adversarially perturbed, it is not enough to only considerthe natural value V ⇡ and Q ⇡ .
As motivated in Figure 1, two policies with similar natural rewardscan get totally different rewards under attacks.
To comprehensively evaluate how good a policy isin an adversarial scenario and to improve its robustness, we should be aware of the lowest possiblelong-term reward of the policy when its observation is adversarially perturbed with a certain attackbudget ✏ at every step (with an ` p attack model introduced in Section 3).
The worst-case value of a policy is, by deﬁnition, the cumulative reward obtained under the optimalattacker.
As justiﬁed by prior works [52, 41], for any given victim policy ⇡ and attack budget ✏ > 0 ,there exists an optimal attacker, and ﬁnding the optimal attacker is equivalent to learning the optimalpolicy in another MDP.
We denote the optimal (deterministic) attacker’s policy as h ⇤ .
However,learning such an optimal attacker by RL algorithms requires extra interaction samples from theenvironment, due to the unknown dynamics.
Moreover, learning the attacker by RL can be hard andexpensive, especially when the state observation space is high-dimensional.
Instead of explicitly learning the optimal attacker with a large amount of samples, we propose todirectly estimate the worst-case cumulative reward of the policy by characterizing the vulnerabilityof the given policy.
We ﬁrst deﬁne the worst-attack action value of policy ⇡ as Q ⇡ ( s, a ) :=
E P [
P 1 t =
t R ( s t , ⇡ ( h ⇤ ( s t )))
| s
=
s, a 0 = a ] .
The worst-attack value V ⇡ can be deﬁned usingh ⇤ in the same way, as shown in Deﬁnition A.1 in Appendix A.
Then, we introduce a novel operator
T ⇡ , namely the worst-attack Bellman operator , deﬁned as below.
Deﬁnition 4.1 (Worst-attack Bellman Operator) .
For MDP M , given a ﬁxed policy ⇡ and attackradius ✏ , deﬁne the worst-attack Bellman operator T ⇡ as
[Equation]
where 8 s 2 S , A adv ( s, ⇡ ) is deﬁned as
[Equation]
Here A adv ( s 0 , ⇡ ) denotes the set of actions an adversary can mislead the victim ⇡ into selecting byperturbing the state s 0 into a neighboring state ˜ s 2 B ✏ ( s 0 ) .
This hypothetical perturbation to the futurestate s 0 is the key for characterizing the worst-case long-term reward under attack.
The followingtheorem associates the worst-attack Bellman operator and the worst-attack action value.
Theorem 4.2 (Worst-attack Bellman Operator and Worst-attack Action Value) .
For any given policy⇡ , T ⇡ is a contraction whose ﬁxed point is Q ⇡ , the worst-attack action value of ⇡ under any ` pobservation attacks with radius ✏ .
Theorem 4.2 proved in Appendix A suggests that the lowest possible cumulative reward of a policy under bounded observation attacks can be computed by worst-attack Bellman operator.
Thecorresponding worst-attack value V ⇡ can be obtained by V ⇡ ( s ) =
min a 2 A adv ( s, ⇡ ) Q ⇡ ( s, a ) .
How to Compute A adv .
To obtain A adv ( s, ⇡ ) , we need to identify the actions that can be theoutputs of the policy ⇡ when the input state s is perturbed within B ✏ ( s ) .
This can be solved by commonly-used convex relaxation of neural networks [14, 53, 47, 51, 13], where layer-wise lowerand upper bounds of the neural network are derived.
That is, we calculate ⇡ and ⇡ such that⇡ ( s )  ⇡ (ˆ s )  ⇡ ( s ) , 8 ˆ s 2 B ✏ ( s ) .
With such a relaxation, we can obtain a superset of A adv , namelyˆ A adv . Then, the ﬁxed point of Equation (1) with A adv being replaced by ˆ A adv becomes a lowerbound of the worst-attack action value.
For a continuous action space, ˆ A adv ( s, ⇡ ) contains actionsbounded by ⇡ ( s ) and ⇡ ( s ) .
For a discrete action space, we can ﬁrst compute the maximal and minimalprobabilities of taking each action, and derive the set of actions that are likely to be selected.
Thecomputation of ˆ A adv is not expensive, as there are many efﬁcient convex relaxation methods [30, 51]which compute ⇡ and ⇡ with only constant-factor more computations than directly computing ⇡ ( s ) .
Experiment in Section 5 veriﬁes the efﬁciency of our approach. More implementation details andexplanations are provided in Appendix C.1.
Estimating Worst-attack Value.
Note that the worst-attack Bellman operator T ⇡ is similar tothe optimal Bellman operator T ⇤ , although it uses min a 2 A adv instead of max a 2 A .
Therefore, once 194 we identify
A adv as introduced above, it is straightforward to compute the worst-attack actionvalue using Bellman backups.
To model the worst-attack action value, we train a network namedworst-attack critic , denoted by Q ⇡ , where  is the parameterization.
Concretely, for any mini-batch{ s t , a t , r t , s t +1 } Nt =1 , Q ⇡ is optimized by minimizing the following estimation loss:
[Equation]
⇡ ( s t +1 , ˆ a ) .
For a discrete action space, A adv is a discrete set and solving y t is straightforward.
For a continuousaction space, we use gradient descent to approximately ﬁnd ˆ a that minimizes Q ⇡ ( s t +1 , ˆ a ) .
Since A advis in general small, this minimization is usually easy to solve.
In our experiments in MuJoCo, we ﬁndthat 50-step gradient descent already converges to a good solution with little computational cost.
Differences with Worst-case Value Estimation in Related Work.
Our proposed worst-attack Bellman operator is different from the worst-case Bellman operator in the literature of risk-sensitive
RL [17, 10, 42, 9, 2, 45], whose goal is to avoid unsafe trajectories under the intrinsic uncertaintiesof the MDP.
These inherent uncertainties of the environment are independent of the learned policy.
In contrast, our focus is to defend against adversarial perturbations created by malicious attackersthat can be adaptive to the policy.
The GWC reward proposed by [32] also estimates the worst-casereward of a policy under state perturbations.
But their evaluation is based on a greedy strategy andrequires interactions with the environment, which is different from our estimation.
Mechanism 2: Worst-case-aware Policy Optimization
So far we have introduced how to evaluate the worst-attack value of a policy by learning a worst-attackcritic.
Inspired by the actor-critic framework, where the actor policy network ⇡ ✓ is optimized towardsa direction that the critic value increases the most, we can regard worst-attack critic as a special criticthat directs the actor to increase the worst-attack value.
That is, we encourage the agent to select anaction with a higher worst-attack action value, by minimizing the worst-attack policy loss below:
[Equation]
where Q ⇡ is the worst-attack critic learned via L est introduced in Equation (3).
Note that L wst is ageneral form, while the detailed implementation of the worst-attack policy optimization can varydepending on the architecture of ⇡ ✓ in the base RL algorithm (e.g. PPO has a policy network, while
DQN acts using the greedy policy induced by a Q network).
In Appendix C.2 and Appendix C.3, weillustrate how to implement L wst for PPO and DQN as two examples.
The proposed worst-case-aware policy optimization has several merits compared to prior ATLA [50]and PA-ATLA [41] methods which alternately train the agent and an RL attacker.
(1) Learning theoptimal attacker
h
⇤ requires collecting extra samples using the current policy (on-policy estimation).
In contrast, Q ⇡ can be learned using off-policy samples, e.g., historical samples in the replay buffer,and thus is more suitable for training where the policy changes over time.
( Q ⇡ depends on the currentpolicy via the computation of A adv .)
(2) We properly exploit the policy function that is being trainedby computing the set of possibly selected actions ˆ A adv for any state.
In contrast, ATLA [50] learns anattacker by treating the current policy as a black box, ignoring the intrinsic properties of the policy.
PA-ATLA [41], although assumes white-box access to the victim policy, also needs to explore andlearn from extra on-policy interactions.
The attacker trained with DRL methods, namely ˆ h
⇤ , is not guaranteed to converge to an optimal solution, such that the performance of ⇡ estimated under ˆ h ⇤can be overly optimistic.
Our estimation, as mentioned in Mechanism 1, computes a lower bound of Q ⇡ and thus can better indicate the robustness of a policy.
Mechanism 3: Value-enhanced State Regularization
As discussed in Section 1, the vulnerability of a deep policy comes from both the policy’s intrinsicvulnerability with the RL dynamics and the DNN approximator.
The ﬁrst two mechanisms of WocaRRL mainly focus on the policy’s intrinsic vulnerability, i.e., let the policy select actions that are lessvulnerable to possible attacks in all future steps.
However, if a bounded state perturbation can causethe network to output a very different action, then the A adv set will be large and Q ⇡ can thus be low.
Therefore, it is also important to encourage the trained policy to output similar actions for the cleanstate s and any ˜ s 2 B ✏ ( s ) , as is done in prior work [52, 39, 8].
But different from these prior methods, we note that different states should be treated differently.
Some states are “critical” where selecting a bad action will result in catastrophic consequences.
Forexample, when the agent gets close to the bomb in Figure 1, we should make the network moreresistant to adversarial state perturbations.
To differentiate states based on their impacts on futurereward, we propose to measure the importance of states with Deﬁnition 4.3 below.
Deﬁnition 4.3 (State Importance Weight) .
Deﬁne state importance weight of s 2 S for policy ⇡ as
[Equation]
To justify whether Deﬁnition 4.3 can characterize state importance,we train a DQN network in an Atari game Pong, and show the stateswith the highest weight and the lowest weight in Figure 3, amongmany state samples.
We can see that the state with higher weight in
Figure 3(left) is indeed crucial for the game, as the green agent paddleis close to the ball.
Conversely, a less-important state in Figure 3(right)does not have signiﬁcantly different future rewards under differentactions.
Computing w ( s ) is easy in a discrete action space, while in acontinuous action space, one can use gradient descent to approximatelyﬁnd the maximal and the minimal Q values for a state.
By incorporating the state importance weight w ( s ) , we regularize the policy network and let it paymore attention to more crucial states, by minimizing the following loss:
[Equation]
where Dist can be any distance measure between two distributions (e.g., KL-divergence).
Minimizing
L reg can result in a smaller A adv , and thus the worst-attack value will be closer to the natural value.
WocaR-RL: A Generic Robust Training Framework
[Figure]
So far we have introduced three key mechanismsand their loss functions, L est in Equation (3),
L wst in Equation (4) and L reg in Equation (6).
Then, our robust training framework WocaRRL combines these losses with any base RLalgorithm.
To be more speciﬁc, as shown in
Figure 4, for any base RL algorithm that trainspolicy ⇡ ✓ using loss L RL , we learn an extraworst-attack critic network Q ⇡ by minimizing
[Equation]
(7) and combine L wst and L reg with L RL to optimize ⇡ ✓ by minimizing
[Equation]
where  wst and  reg are hyperparameters balancing between natural performance and robustness.
Note that Q ⇡  is trained together but independently with ⇡ ✓ using historical transition samples, so
WocaR-RL does not require extra samples from the environment.
WocaR-RL can also be interpretedfrom a geometric perspective based on prior RL polytope theory [7, 41] as detailed in Appendix B.
Our WocaR-RL is a generic robust training framework that can be used to robustify existing DRLalgorithms.
We provide two case studies: (1) combining WocaR-RL with a policy-based algo- rithm PPO [38], namely WocaR-PPO , and (2) combining WocaR-RL with a value-based algorithm
DQN [31], namely WocaR-DQN .
The pseudocodes of WocaR-PPO and WocaR-DQN are illustratedin Appendix C.2 and Appendix C.3.
The application of WocaR-RL to other DRL methods is thenstraightforward, since most DRL methods are either policy-based or value-based.
Next, we showby experiments that WocaR-PPO and WocaR-DQN achieve state-of-the-art robustness with superiorefﬁciency, in various continuous control tasks and video game environments.
We also empiricallyverify the effectiveness of each of the 3 mechanisms of WocaR-RL in Section 5.2.
Experiments and Discussion
In this section, our experimental evaluations on various MuJoCo and Atari environments aim to studythe following questions: (1) Can WocaR-RL learn policies with better robustness under existingstrong adversarial attacks?
(2) Can WocaR-RL maintain natural performance when improvingrobustness?
(3) Can WocaR-RL learn more efﬁciently during robust training?
(4) Is each mechanismin WocaR-RL effective ?
Problem (1), (2) and (3) are answered in Section 5.1 with detailed empiricalresults, and problem (4) is studied in Section 5.2 via ablation experiments.
5.1 Experiments and Evaluations
Environments.
Following most prior works [52, 50, 32] and the released implementation, we applyour WocaR-RL to PPO [38] on 4 MuJoCo tasks with continuous action spaces, including Hopper,
Walker2d, Halfcheetah and Ant, and to DQN [31] agents on 4 Atari games including Pong, Freeway,
BankHeist and RoadRunner, which have high dimensional pixel inputs and discrete action spaces.
Baselines and Implementation.
We compare our algorithm with several state-of-the-art robusttraining methods, including (1) SA-PPO/SA-DQN [52]: regularizing policy networks by convexrelaxation.
(2) ATLA-PPO [50]: alternately training an agent and an RL attacker.
(3) PA-ATLAPPO [41]: alternately training an agent and a more advanced RL attacker PA-AD.
(4) RADIALPPO/RADIAL-DQN [32]: optimizing policy network by designed adversarial loss functions based onrobustness bounds.
SA and RADIAL have both PPO and DQN versions, which are compared withour WocaR-PPO and WocaR-DQN.
But ATLA and PA-ATLA do not provide DQN versions, sincealternately training on DQN can be expensive as explained in the original papers [41].
(PA-ATLA hasan
A2C version, which we compare in Appendix D.2.)
Therefore, we reproduce their ATLA-PPOand PA-ATLA-PPO results and compare them with our WocaR-PPO.
More implementation andhyperparameter details are provided in Appendix D.1.
Case I: Robust PPO for MuJoCo Continuous Control
Evaluation Metrics.
To reﬂect both the natural performance and robustness of trained agents, wereport the average episodic rewards under no attack and against various attacks.
For a comprehensiverobustness evaluation, we attack the trained robust models with multiple existing attack methods,including: (1) MaxDiff [52] (maximal action difference), (2) Robust Sarsa (RS) [52] (attacking witha robust action-value function), (3) SA-RL [52] (ﬁnding the optimal state adversary) and (4) PA-AD[41] (the existing strongest attack by learning the optimal policy adversary with RL).
For a clearcomparison, we use the same attack radius ✏ as in most baselines [52, 50, 41].
Performance and Robustness of WocaR-PPO Figure 5 (left four columns) shows performancecurves during training under four different adversarial attacks.
Among all four attack algorithms,
WocaR-PPO converges much faster than baselines, and often achieves the best asymptotic robustperformance, especially under the strongest PA-AD attack.
It is worth emphasizing that sincewe train a robust agent without explicitly learning an RL attacker, our method not only obtainsstronger robustness and much higher efﬁciency, but also a more general defense: WocaR-PPOobtains comprehensively superior performance against a variety of attacks compared against existing
SOTA algorithms based on learned attackers (ATLA-PPO, PA-ATLA-PPO).
Additionally, in ourexperiments, WocaR-PPO learns relatively more universal defensive behaviors as shown in Figure 2,which can physically explain why our algorithm can defend against diverse attacks.
We providepolicy demonstrations in multiple tasks in our supplementary materials.
The comparison of natural performance and the worst-case performance appears in Figure 5 (right).
We see that WocaR-PPO maintains competitive natural rewards under no attack compared with otherbaselines, which demonstrates that our algorithm gains more robustness without losing too muchnatural performance.
The full results of baselines and our algorithm under different attack evaluationsare provided by Table 2 in Appendix D.2 (including performance under random attacks).
[Figure]
a l f c h ee t
a
[Figure]
Efﬁciency of Training WocaR-PPO.
The learning curves in Figure 5 (left) directly show thesample efﬁciency of WocaR-PPO.
Following the optimal settings provided in [52, 50, 32], our methodtakes 50% training steps required by RADIAL-PPO and ATLA methods on Hopper, Walker2d, and
Halfcheetah because RADIAL-PPO needs more steps to ensure convergence and ATLA methodsrequire additional adversary training steps.
When solving high dimensional environments like Ant,
WocaR-PPO only requires 75% steps compared with all other baselines to converge.
We also provideadditional results of baselines using the same training steps as WocaR-PPO in Appendix D.3.2.
In terms of time efﬁciency, WocaR-PPO saves 50% training time for convergence on Hopper,
Walker2d, and Halfcheetah, and 32% time on Ant compared with the SOTA method.
Therefore,
WocaR-PPO achieves both higher computational efﬁciency and higher sample efﬁciency than SOTAbaselines.
Detailed costs in time and sampling are in Appendix D.3.3.
Case II: Robust DQN for Atari Video Games
Evaluation Metrics.
Since Atari games have pixel state spaces and discrete action spaces, theapplicable attacking algorithms also differ from those in MuJoCo tasks.
We include the followingcommon attacks: (1) 10-step untargeted PGD (projected gradient descent) attack,
(2) MinBest [18],which minimizes the probability of choosing the “best” action, (3) PA-AD [41], as the state-of-the-art
RL-based adversarial attack algorithm.
Performance and Robustness of WocaR-DQN.
Table 1 presents the results on four Atari gamesunder attack radius ✏ = 3 / 255 , while results and analysis under smaller attack radius 1 / 255 are in
Appendix D.2.
We can see that our WocaR-DQN consistently outperforms baselines under MinBestand PA-AD attacks in all environments, with a signiﬁcant advance under the strongest (worst-case)
PA-AD attacks compared with other robust agents.
Under PGD attacks, WocaR-DQN performscomparably with the state-of-the-art in Freeway and Pong (which are simpler games) and gains higherrewards than other agents in BankHeist and Roadrunner.
Since SA-DQN and RADIAL-DQN focuson bounding and smoothing the policy network and do not consider the policy’s intrinsic vulnerability,they are robust under the PGD attack but still vulnerable against the stronger PA-AD attack.
Efﬁciency of Training WocaR-DQN.
The total training time for SA-DQN, RADIAL-DQN, andour WocaR-DQN are roughly 35, 17, and 18 hours, respectively.
All baselines are trained for 6million frames on the same hardware.
Therefore, WocaR-DQN is 49% faster (and is more robust)than SA-DQN.
Compared to the more advanced baseline RADIAL-DQN, although WocaR-DQN is5% slower, it achieves better robustness (539% higher reward than RADIAL-DQN in RoadRunner).
[Table]
[Figure]
[Table]
5.2 Verifying Effectiveness of WocaR-RL
Now we dive deeper into the algorithmic design and verify the effectiveness of WocaR-RL by ablationstudies on WocaR-PPO.
[Figure]
(1) Worst-attack value estimation.
We show the learned worst-attack value estimation, Q ⇡ , duringthe training process in Figure 9c, in comparison with the actual reward under the strongest attack (PAAD [41]) in Figure 9b.
The pink curves in both plots suggest that our worst-attack value estimationmatches the trend of actual worst-case reward under attacks, although the network estimated valueand the real reward have different scales due to the commonly-used reward normalization for learningstability.
Therefore, the proposed worst-attack value estimation ( L est ) is effective in the experiments.(2) Worst-case-aware policy optimization.
Compared to vanilla PPO and SA-PPO, we can see that
WocaR-PPO improves the worst-attack value and the worst-case reward during training , suggestingthe effectiveness of our worst-attack value improvement ( L wst ).
The comparison of natural rewards,as well as curves in other environments, are provided in Appendix D.3.4.
Moreover, the adjustableweight  wst in Equation (8) controls the trade-off between natural value and worst-attack value inpolicy optimization.
When  wst is high, the policy pays more attention to its worst-attack value.
Appendix D.3.5 veriﬁes that WocaR-RL, with different values of weight  wst , produces differentrobustness and natural performance while consistently dominating other robust agents.(3)
Value-enhanced state regularization.
We compare the performance of the original WocaR-PPOto variants of WocaR-PPO without the state importance weight w ( s ) in Figure 6c.
The results of SAPPO with w ( s ) also indicate the effectiveness of the state importance.
Detailed ablation studies formore environments are shown in Appendix D.3.6, which verify that the weighted state regularizationis beneﬁcial to enhancing the robustness of RL agents.
Without w ( s ) , our algorithm also achievessimilar or better performance than baselines, but including this inexpensive technique w ( s ) gives
WocaR-RL a greater advantage, especially under learned strong attacks SA-RL and PA-AD.
Conclusion
This paper proposes a robust RL training framework, WocaR-RL, that evaluates and improves the longterm robustness of a policy via worst-attack value estimation, worst-case-aware policy optimization,value-enhanced state regularization.
Compared with state-of-the-art robust RL approaches, WocaRRL achieves better robustness, efﬁciency and interpretability in experiments.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See our abstract and Section 1.
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitations in
Appendix E.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We
discuss the potential negative societal impacts in Appendix E.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] Our paper conforms to the guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Ap-
pendix A.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] For each empirical result, we report error bars.
(d) Did you include the total amount of compute and the type of resources used (e g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] See efﬁciency analysis in Appendix D.3.3.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]