Efﬁcient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning
Abstract
Recent studies reveal that a well-trained deep reinforcement learning (RL) policy can be particularly vulnerable to adversarial perturbations on input observations.
Therefore, it is crucial to train RL agents that are robust against any attacks with a bounded budget.
Existing robust training methods in deep RL either treat correlated steps separately, ignoring the robustness of long-term rewards, or train the agents and RL-based attacker together, doubling the computational burden and sample complexity of the training process.
In this work, we propose a strong and efﬁcient robust training framework for RL, named Worst-case-aware Robust RL (WocaR-RL), that directly estimates and optimizes the worst-case reward of a policy under bounded (cid:96) p attacks without requiring extra samples for learning an attacker.
Experiments on multiple environments show that WocaR-RL achieves state-ofthe-art performance under various strong attacks, and obtains signiﬁcantly higher training efﬁciency than prior state-of-the-art robust training methods.
The code of this work is available at https://github.com/umd-huang-lab/WocaR-RL .
Introduction
Deep reinforcement learning (DRL) has achieved impressive results by using deep neural networks (DNN) to learn complex policies in large-scale tasks.
However, well-trained DNNs may drastically fail under adversarial perturbations of the input [1, 6].
Therefore, before deploying DRL policies to real-life applications, it is crucial to improve the robustness of deep policies against adversarial attacks, especially worst-case attacks that maximally depraves the performance of trained agents [42]
A line of regularization-based robust methods [54, 33, 40] focuses on improving the robustness of the DNN itself and regularizes the policy network to output similar actions under bounded state perturbations.
However, different from supervised learning problems, the vulnerability of a deep policy comes not only from the DNN approximator, but also from the dynamics of the RL environment [52].
These regularization-based methods neglect the intrinsic vulnerability of policies under the environment dynamics, and thus may still fail under strong attacks [42].
For example, in the go-home task shown in Figure 1, both the green policy and the red policy arrive home without rock collision, when there is no attack.
However, although regularization-based methods may ensure a minor action change under a state perturbation, the red policy may still be susceptible to a low reward under attacks, as a very small divergence can lead it to the bomb.
On the contrary, the green policy is more robust to adversarial attacks since it stays away from the bomb.
Therefore, besides promoting the robustness of DNN approximators (such as the policy network), it is also important to learn a policy with stronger intrinsic robustness.
There is another line of work considering the long-term robustness of a deep policy under strong adversarial attacks.
In particular, it is theoretically proved [54, 42] that the strongest (worst-case) attacker against a policy can be learned as an RL problem, and training the agent under such a learned attacker can result in a robust policy.
Zhang et al.
[52] propose the Alternating Training with
Learned Adversaries (ATLA) framework, which alternately trains an RL agent and an RL attacker.
Sun et al.
[42] further propose PA-ATLA, which alternately trains an agent and the proposed more efﬁcient PA-AD RL attacker, obtaining state-of-the-art robustness in many MuJoCo environments.
However, training an RL attacker requires extra samples from the environment, and the attacker’s
RL problem may even be more difﬁcult and sample expensive to solve than the agent’s original RL problem [52, 42], especially in large-scale environments such as Atari games with pixel observations.
Therefore, although ATLA and PA-ATLA are able to achieve high long-term reward under attacks, they double the computational burden and sample complexity to train the robust agent.
The above analysis of existing literature suggests two main challenges in improving the adversarial robustness of DRL agents: (1) correctly characterizing the long-term reward vulnerability of an
RL policy, and (2) efﬁciently training a robust agent without requiring much more effort than vanilla training.
To tackle these challenges, in this paper, we propose a generic and efﬁcient robust training framework named Worst-case-aware Robust RL (WocaR-RL) that estimates and improves the long-term robustness of an RL agent.
WocaR-RL has 3 key mechanisms.
First, WocaR-RL introduces a novel worst-attack Bellman operator which uses existing off-policy samples to estimate the lower bound of the policy value under the worst-case attack.
Compared to prior works [52, 42] which attempt to learn the worst-case attack by RL methods, WocaR-RL does not require any extra interaction with the environment.
Second, using the estimated worst-case policy value, WocaR-RL optimizes the policy to select actions that not only achieve high natural future reward, but also achieve high worst-case reward when there are adversarial attacks.
Therefore, WocaR-RL learns a policy with less intrinsic vulnerability.
Third,
WocaR-RL regularizes the policy network with a carefully designed state importance weight.
As a result, the DNN approximator tolerates state perturbations, especially for more important states where decisions are crucial for future reward.
The above 3 mechanisms can also be interpreted from a geometric perspective of adversarial policy learning, as detailed in Appendix B.
Our contributions can be summarized as below.
(1) We provide an approach to estimate the worstcase value of any policy under any bounded (cid:96) p adversarial attacks.
This helps evaluate the robustness of a policy without learning an attacker which requires extra samples and exploration.
(2) We propose a novel and principled robust training framework for RL, named Worst-case-aware Robust
RL (WocaR-RL) , which characterizes and improves the worst-case robustness of an agent.
WocaRRL can be used to robustify existing DRL algorithms (e.g. PPO [39], DQN [32]).
(3) We show by experiments that WocaR-RL achieve improved robustness against various adversarial attacks as well as higher efﬁciency , compared with state-of-the-art (SOTA) robust RL methods in many
MuJoCo and Atari games.
For example, compared to the SOTA algorithm PA-ATLA-PPO [42] in the
Walker environment, we obtain 20% more worst-case reward (under the strongest attack algorithm), with only about 50% training samples and 50% running time.
Moreover, WocaR-RL learns more interpretable “robust behaviors” than PA-ATLA-PPO in Walker as shown in Figure 2.
[Figure]
attacks, it learns to jump with one leg, which is counter-intuitive and may indicate some level of overﬁtting to a speciﬁc attacker.
In contrast, our WocaR-RL agent learns to lower down its body, which is more intuitive and interpretable.
The full agent trajectories in Walker and other environments are provided in supplementary materials as GIF ﬁgures.
Related Work
Defending against Adversarial Perturbations on State Observations.
(1) Regularization-based methods [54, 40, 33] enforce the policy to have similar outputs under similar inputs, which achieves certiﬁable performance for DQN in some Atari games.
But in continuous control tasks, these methods may not reliably improve the worst-case performance.
A recent work by Korkmaz [21] points out that these adversarially trained models may still be sensible to new perturbations.
(2) Attack-driven methods train DRL agents with adversarial examples.
Some early works [22, 4, 29, 34] apply weak or strong gradient-based attacks on state observations to train RL agents against adversarial perturbations.
Zhang et al.
[52] propose Alternating Training with Learned Adversaries (ATLA), which alternately trains an RL agent and an RL adversary and signiﬁcantly improves the policy robustness in continuous control games.
Sun et al.
[42] further extend this framework to PA-ATLA with their proposed more advanced RL attacker PA-AD.
Although ATLA and PA-ATLA achieve strong empirical robustness, they require training an extra RL adversary that can be computationally and sample expensive.
There is another line of work studying certiﬁable robustness of RL policies.
Several works [27, 33, 9] computed lower bounds of the action value network Q π to certify robustness of action selection at every step.
However, these bounds do not consider the distribution shifts caused by attacks, so some actions that appear safe for now can lead to extremely vulnerable future states and low long-term reward under future attacks.
Moreover, these methods cannot apply to continuous action spaces.
Kumar et al.
and Wu et al.[23, 49] both extend randomized smoothing [7] to derive robustness certiﬁcates for trained policies.
But these works mostly focus on theoretical analysis, and effective robust training approaches rather than robust training.
Adversarial Defenses against Other Adversarial Attacks.
Besides observation perturbations, attacks can happen in many other scenarios.
For example, the agent’s executed actions can be perturbed [50, 44, 45, 24].
Moreover, in a multi-agent game, an agent’s behavior can create adversarial perturbations to a victim agent [13].
Pinto et al.
[35] model the competition between the agent and the attacker as a zero-sum two-player game, and train the agent under a learned attacker to tolerate both environment shifts and adversarial disturbances.
We point out that although we mainly consider state adversaries, our WocaR-RL can be extended to action attacks as formulated in Appendix C.5.
Note that we focus on robustness against test-time attacks, different from poisoning attacks which alter the RL training process [3, 20, 41, 56, 36].
Safe RL and Risk-sensitive RL.
There are several lines of work that study RL under safety/risk constraints [18, 11, 10, 2, 46] or under intrinsic uncertainty of environment dynamics [26, 30].
However, these works do not deal with adversarial attacks, which can be adaptive to the learned policy.
More comparison between these methods and our proposed method is discussed in Section 4.
Preliminaries and Background
Reinforcement Learning (RL).
An RL environment is modeled by a Markov Decision Process (MDP), denoted by a tuple M = (cid:104)S , A , P, R, γ (cid:105) , where S is a state space, A is an action space, P :
S×A → ∆( S ) is a stochastic dynamics model 2 , R : S×A
→ R is a reward function and γ ∈ [0 , 1) is a discount factor.
An agent takes actions based on a policy π : S → ∆( A ) .
For any policy, its natural performance can be measured by the value function V π ( s ) :
= E P,π [ (cid:80) ∞ t =
γ t R ( s t , a t ) | s 0 = s ] , and the action value function Q π ( s, a ) :
= E P,π [ (cid:80) ∞ t =
γ t R ( s t , a t ) | s 0 =
s, a 0 = a ] .
We call V π the natural value and Q π the natural action value in contrast to the values under attacks, as will be introduced in Section 4.
Deep Reinforcement Learning (DRL).
In large-scale problems, a policy can be parameterized by a neural network.
For example, value-based RL methods (e.g. DQN [32]) usually ﬁt a Q network and take the greedy policy π ( s ) = argmax a Q ( s, a ) .
In actor-critic methods (e.g. PPO [39]), the learner directly learns a policy network and a critic network.
In practice, an agent usually follows a stochastic policy during training that enables exploration, and executes a trained policy deterministically in test-time, e.g. the greedy policy learned with DQN.
Throughout this paper, we use π θ to denote the training-time stochastic policy parameterized by θ , while π denotes the trained deterministic policy that maps a state to an action.
Test-time Adversarial Attacks.
After training, the agent is deployed into the environment and executes a pre-trained ﬁxed policy π .
An attacker/adversary, during the deployment of the agent, may perturb the state observation of the agent/victim at every time step with a certain attack budget (cid:15) .
Note that the attacker only perturbs the inputs to the policy, and the underlying state in the environment does not change.
This is a realistic setting because real-world observations can come from noisy sensors or be manipulated by malicious attacks.
For example, an auto-driving car receives sensory observations; an attacker may add imperceptible noise to the camera, or perturb the GPS signal, although the underlying environment (the road) remains unchanged.
In this paper, we consider the (cid:96) p thread model which is widely used in adversarial learning literature: at step t , the attacker alters the observation s t into ˜ s t ∈ B (cid:15) ( s t ) , where B (cid:15) ( s t ) is a (cid:96) p norm ball centered at s t with radius (cid:15) .
The above setting ( (cid:96) p constrained observation attack) is the same with many prior works [19, 34, 54, 52, 42].
Worst-case-aware Robust RL
In this section, we present Worst-case-aware Robust RL (WocaR-RL) , a generic framework that can be fused with any DRL approach to improve the adversarial robustness of an agent.
We will introduce the three key mechanisms in WocaR-RL: worst-attack value estimation, worst-case-aware policy optimization, and value-enhanced state regularization, respectively.
Then, we will illustrate how to incorporate these mechanisms into existing DRL algorithms to improve their robustness.
Mechanism 1: Worst-attack Value Estimation
Traditional RL aims to learn a policy with the maximal value V π .
However, in a real-world problem where observations can be noisy or even adversarially perturbed, it is not enough to only consider the natural value V π and Q π .
As motivated in Figure 1, two policies with similar natural rewards can get totally different rewards under attacks.
To comprehensively evaluate how good a policy is in an adversarial scenario and to improve its robustness, we should be aware of the lowest possible long-term reward of the policy when its observation is adversarially perturbed with a certain attack budget (cid:15) at every step (with an (cid:96) p attack model introduced in Section 3).
The worst-case value of a policy is, by deﬁnition, the cumulative reward obtained under the optimal attacker.
As justiﬁed by prior works [54, 42], for any given victim policy π and attack budget (cid:15) > 0 , there exists an optimal attacker, and ﬁnding the optimal attacker is equivalent to learning the optimal policy in another MDP.
We denote the optimal (deterministic) attacker’s policy as h ∗ .
However, learning such an optimal attacker by RL algorithms requires extra interaction samples from the environment, due to the unknown dynamics.
Moreover, learning the attacker by RL can be hard and expensive, especially when the state observation space is high-dimensional.
Instead of explicitly learning the optimal attacker with a large amount of samples, we propose to directly estimate the worst-case cumulative reward of the policy by characterizing the vulnerability of the given policy.
We ﬁrst deﬁne the worst-attack action value of policy π as Q π ( s, a ) :=
E P [ (cid:80) ∞ t =
γ t R ( s t , π ( h ∗ ( s t )))
| s
=
s, a 0 = a ] .
The worst-attack value V π can be deﬁned using h ∗ in the same way, as shown in Deﬁnition A.1 in Appendix A.
Then, we introduce a novel operator
T π , namely the worst-attack Bellman operator , deﬁned as below.
Deﬁnition 4.1 (Worst-attack Bellman Operator) .
For MDP M , given a ﬁxed policy π and attack radius (cid:15) , deﬁne the worst-attack Bellman operator T π as
[Equation]
where ∀ s ∈ S , A adv ( s, π ) is deﬁned as
[Equation]
Here A adv ( s (cid:48) , π ) denotes the set of actions an adversary can mislead the victim π into selecting by perturbing the state s (cid:48) into a neighboring state ˜ s ∈ B (cid:15) ( s (cid:48) ) .
This hypothetical perturbation to the future state s (cid:48) is the key for characterizing the worst-case long-term reward under attack.
The following theorem associates the worst-attack Bellman operator and the worst-attack action value.
Theorem 4.2 (Worst-attack Bellman Operator and Worst-attack Action Value) .
For any given policy π , T π is a contraction whose ﬁxed point is Q π , the worst-attack action value of π under any (cid:96) p observation attacks with radius (cid:15) .
Theorem 4.2 proved in Appendix A suggests that the lowest possible cumulative reward of a policy under bounded observation attacks can be computed by worst-attack Bellman operator.
The corresponding worst-attack value V π can be obtained by V π ( s ) =
min a ∈A
adv ( s,π ) Q π ( s, a ) .
How to Compute A adv .
To obtain A adv ( s, π ) , we need to identify the actions that can be the outputs of the policy π when the input state s is perturbed within B (cid:15) ( s ) .
This can be solved by commonly-used convex relaxation of neural networks [15, 55, 48, 53, 14], where layer-wise lower and upper bounds of the neural network are derived.
That is, we calculate π and π such that π ( s ) ≥ π (ˆ s ) ≥ π ( s ) , ∀ ˆ s ∈ B (cid:15) ( s ) .
With such a relaxation, we can obtain a superset of A adv , namely ˆ A adv .
Then, the ﬁxed point of Equation (1) with A adv being replaced by ˆ A adv becomes a lower bound of the worst-attack action value.
For a continuous action space, ˆ A adv ( s, π ) contains actions bounded by π ( s ) and π ( s ) .
For a discrete action space, we can ﬁrst compute the maximal and minimal probabilities of taking each action, and derive the set of actions that are likely to be selected.
The computation of ˆ A adv is not expensive, as there are many efﬁcient convex relaxation methods [31, 53] which compute π and π with only constant-factor more computations than directly computing π ( s ) .
Experiment in Section 5 veriﬁes the efﬁciency of our approach, where we use a well-developed toolbox auto _ LiRPA [51] to calculate the convex relaxation.
More implementation details and explanations are provided in Appendix C.1.
Estimating Worst-attack Value.
Note that the worst-attack Bellman operator T π is similar to the optimal Bellman operator T ∗ , although it uses min a ∈A adv instead of max a ∈A .
Therefore, once we identify A adv as introduced above, it is straightforward to compute the worst-attack action value using Bellman backups.
To model the worst-attack action value, we train a network named worst-attack critic , denoted by Q πφ , where φ is the parameterization.
Concretely, for any mini-batch { s t , a t , r t , s t +1 } Nt =1 , Q πφ is optimized by minimizing the following estimation loss:
[Equation]
For a discrete action space, A adv is a discrete set and solving y t is straightforward.
For a continuous action space, we use gradient descent to approximately ﬁnd the minimizer ˆ a .
Since A adv is in general small, this minimization is usually easy to solve.
In MuJoCo, we ﬁnd that 50-step gradient descent already converges to a good solution with little computational cost, as detailed in Appendix D.3.3.
Differences with Worst-case Value Estimation in Related Work.
Our proposed worst-attack
Bellman operator is different from the worst-case Bellman operator in the literature of risk-sensitive
RL [18, 11, 43, 10, 2, 46], whose goal is to avoid unsafe trajectories under the intrinsic uncertainties of the MDP.
These inherent uncertainties of the environment are independent of the learned policy.
In contrast, our focus is to defend against adversarial perturbations created by malicious attackers that can be adaptive to the policy.
The GWC reward proposed by [33] also estimates the worst-case reward of a policy under state perturbations.
But their evaluation is based on a greedy strategy and requires interactions with the environment, which is different from our estimation.
Mechanism 2: Worst-case-aware Policy Optimization
So far we have introduced how to evaluate the worst-attack value of a policy by learning a worst-attack critic.
Inspired by the actor-critic framework, where the actor policy network π θ is optimized towards a direction that the critic value increases the most, we can regard worst-attack critic as a special critic that directs the actor to increase the worst-attack value.
That is, we encourage the agent to select an action with a higher worst-attack action value, by minimizing the worst-attack policy loss below:
[Equation]
where Q πφ is the worst-attack critic learned via L est introduced in Equation (3).
Note that L wst is a general form, while the detailed implementation of the worst-attack policy optimization can vary depending on the architecture of π θ in the base RL algorithm (e.g. PPO has a policy network, while
DQN acts using the greedy policy induced by a Q network).
In Appendix C.2 and Appendix C.3, we illustrate how to implement L wst for PPO and DQN as two examples.
The proposed worst-case-aware policy optimization has several merits compared to prior ATLA [52] and PA-ATLA [42] methods which alternately train the agent and an RL attacker.
(1) Learning the optimal attacker h ∗ requires collecting extra samples using the current policy (on-policy estimation).
In contrast, Q πφ can be learned using off-policy samples, e.g., historical samples in the replay buffer, and thus is more suitable for training where the policy changes over time.
( Q πφ depends on the current policy via the computation of A adv .)
(2) We properly exploit the policy function that is being trained by computing the set of possibly selected actions ˆ A adv for any state.
In contrast, ATLA [52] learns an attacker by treating the current policy as a black box, ignoring the intrinsic properties of the policy.
PA-ATLA [42], although assumes white-box access to the victim policy, also needs to explore and learn from extra on-policy interactions.
The attacker trained with DRL methods, namely ˆ h ∗ , is not guaranteed to converge to an optimal solution, such that the performance of π estimated under ˆ h ∗ can be overly optimistic.
Our estimation, as mentioned in Mechanism 1, computes a lower bound of
Q π and thus can better indicate the robustness of a policy.
Mechanism 3: Value-enhanced State Regularization
As discussed in Section 1, the vulnerability of a deep policy comes from both the policy’s intrinsic vulnerability with the RL dynamics and the DNN approximator.
The ﬁrst two mechanisms of WocaRRL mainly focus on the policy’s intrinsic vulnerability, i.e., let the policy select actions that are less vulnerable to possible attacks in all future steps.
However, if a bounded state perturbation can cause the network to output a very different action, then the A adv set will be large and Q π can thus be low.
Therefore, it is also important to encourage the trained policy to output similar actions for the clean state s and any ˜ s ∈ B (cid:15) ( s ) , as is done in prior work [54, 40, 9].
But different from these prior methods, we note that different states should be treated differently.
Some states are “critical” where selecting a bad action will result in catastrophic consequences.
For example, when the agent gets close to the bomb in Figure 1, we should make the network more resistant to adversarial state perturbations.
To differentiate states based on their impacts on future reward, we propose to measure the importance of states with Deﬁnition 4.3 below.
Deﬁnition 4.3 (State Importance Weight) .
Deﬁne state importance weight of s ∈ S for policy π as
[Equation]
[Equation]
To justify whether Deﬁnition 4.3 can characterize state importance, we train a DQN network in an Atari game Pong, and show the states with the highest weight and the lowest weight in Figure 3, among many state samples.
We can see that the state with higher weight in
Figure 3(left) is indeed crucial for the game, as the green agent paddle is close to the ball.
Conversely, a less-important state in Figure 3(right) does not have signiﬁcantly different future rewards under different actions.
Computing w ( s ) is easy in a discrete action space, while in a continuous action space, one can use gradient descent to approximately ﬁnd the maximal and the minimal Q values for a state.
Similar to the computation of Equation (3) with a continuous action space, we ﬁnd that a 50-step gradient descent works well in experiments.
By incorporating the state importance weight w ( s ) , we regularize the policy network and let it pay more attention to more crucial states, by minimizing the following loss:
[Equation]
where Dist can be any distance measure between two distributions (e.g., KL-divergence).
Minimizing
L reg can result in a smaller A adv , and thus the worst-attack value will be closer to the natural value.
WocaR-RL: A Generic Robust Training Framework
[Figure]
So far we have introduced three key mechanisms and their loss functions, L est in
Equation (3), L wst in Equation (4) and
L reg in Equation (6).
Then, our robust training framework WocaR-RL combines these losses with any base RL algorithm.
To be more speciﬁc, as shown in Figure 4, for any base RL algorithm that trains policy π θ using loss L RL , we learn an extra worstattack critic network Q πφ by minimizing
[Equation]
and combine L wst and L reg with L RL to optimize π θ by minimizing
[Equation]
where κ wst and κ reg are hyperparameters balancing between natural performance and robustness.
Note that Q πφ is trained together but independently with π θ using historical transition samples, so
WocaR-RL does not require extra samples from the environment.
WocaR-RL can also be interpreted from a geometric perspective based on prior RL polytope theory [8, 42] as detailed in Appendix B.
Our WocaR-RL is a generic robust training framework that can be used to robustify existing DRL algorithms.
We provide two case studies: (1) combining WocaR-RL with a policy-based algorithm PPO [39], namely WocaR-PPO , and (2) combining WocaR-RL with a value-based algorithm
DQN [32], namely WocaR-DQN .
The pseudocodes of WocaR-PPO and WocaR-DQN are illustrated in Appendix C.2 and Appendix C.3.
The application of WocaR-RL to other DRL methods is then straightforward, since most DRL methods are either policy-based or value-based.
Next, we show by experiments that WocaR-PPO and WocaR-DQN achieve state-of-the-art robustness with superior efﬁciency, in various continuous control tasks and video game environments.
We also empirically verify the effectiveness of each of the 3 mechanisms of WocaR-RL and their weights by ablation study in Section 5.2.
Experiments and Discussion
In this section, our experimental evaluations on various MuJoCo and Atari environments aim to study the following questions: (1) Can WocaR-RL learn policies with better robustness under existing strong adversarial attacks?
(2) Can WocaR-RL maintain natural performance when improving robustness?
(3) Can WocaR-RL learn more efﬁciently during robust training?
(4) Is each mechanism in WocaR-RL effective ?
Problem (1), (2) and (3) are answered in Section 5.1 with detailed empirical results, and problem (4) is studied in Section 5.2 via ablation experiments.
5.1 Experiments and Evaluations
Environments.
Following most prior works [54, 52, 33] and the released implementation, we apply our WocaR-RL to PPO [39] on 4 MuJoCo tasks with continuous action spaces, including Hopper,
Walker2d, Halfcheetah and Ant, and to DQN [32] agents on 4 Atari games including Pong, Freeway,
BankHeist and RoadRunner, which have high dimensional pixel inputs and discrete action spaces.
Baselines and Implementation.
We compare our algorithm with several state-of-the-art robust training methods, including (1) SA-PPO/SA-DQN [54]: regularizing policy networks by convex relaxation.
(2) ATLA-PPO [52]: alternately training an agent and an RL attacker.
(3) PA-ATLAPPO [42]: alternately training an agent and a more advanced RL attacker PA-AD.
(4) RADIALPPO/RADIAL-DQN [33]: optimizing policy network by designed adversarial loss functions based on robustness bounds.
SA and RADIAL have both PPO and DQN versions, which are compared with our WocaR-PPO and WocaR-DQN.
But ATLA and PA-ATLA do not provide DQN versions, since alternately training on DQN can be expensive as explained in the original papers [42].
(PA-ATLA has an A2C version, which we compare in Appendix D.2.)
Therefore, we reproduce their ATLA-PPO and PA-ATLA-PPO results and compare them with our WocaR-PPO.
More implementation and hyperparameter details are provided in Appendix D.1.
Case I: Robust PPO for MuJoCo Continuous Control
Evaluation Metrics.
To reﬂect both the natural performance and robustness of trained agents, we report the average episodic rewards under no attack and against various attacks.
For a comprehensive robustness evaluation, we attack the trained robust models with multiple existing attack methods, including: (1) MaxDiff [54] (maximal action difference), (2) Robust Sarsa (RS) [54] (attacking with a robust action-value function), (3) SA-RL [54] (ﬁnding the optimal state adversary) and (4) PA-AD [42] (the existing strongest attack by learning the optimal policy adversary with RL).
For a clear comparison, we use the same attack radius (cid:15) as in most baselines [54, 52, 42].
Performance and Robustness of WocaR-PPO Figure 5 (left four columns) shows performance curves during training under four different adversarial attacks.
Among all four attack algorithms,
WocaR-PPO converges much faster than baselines, and often achieves the best asymptotic robust performance, especially under the strongest PA-AD attack.
It is worth emphasizing that since we train a robust agent without explicitly learning an RL attacker, our method not only obtains
[Figure]
ee t
[Figure]
stronger robustness and much higher efﬁciency, but also a more general defense: WocaR-PPO obtains comprehensively superior performance against a variety of attacks compared against existing
SOTA algorithms based on learned attackers (ATLA-PPO, PA-ATLA-PPO).
Additionally, in our experiments, WocaR-PPO learns relatively more universal defensive behaviors as shown in Figure 2, which can physically explain why our algorithm can defend against diverse attacks.
We provide policy demonstrations in multiple tasks in our supplementary materials.
The comparison of natural performance and the worst-case performance appears in Figure 5 (right).
We see that WocaR-PPO maintains competitive natural rewards under no attack compared with other baselines, which demonstrates that our algorithm gains more robustness without losing too much natural performance.
The full results of baselines and our algorithm under different attack evaluations are provided by Table 2 in Appendix D.2 (including performance under random attacks).
Efﬁciency of Training WocaR-PPO.
The learning curves in Figure 5 (left) directly show the sample efﬁciency of WocaR-PPO.
Following the optimal settings provided in [54, 52, 33], our method takes 50% training steps required by RADIAL-PPO and ATLA methods on Hopper, Walker2d, and
Halfcheetah because RADIAL-PPO needs more steps to ensure convergence and ATLA methods require additional adversary training steps.
When solving high dimensional environments like Ant,
WocaR-PPO only requires 75% steps compared with all other baselines to converge.
We also provide additional results of baselines using the same training steps as WocaR-PPO in Appendix D.3.2.
In terms of time efﬁciency, WocaR-PPO saves 50% training time for convergence on Hopper,
Walker2d, and Halfcheetah, and 32% time on Ant compared with the SOTA method.
Therefore,
WocaR-PPO achieves both higher computational efﬁciency and higher sample efﬁciency than SOTA baselines.
Detailed costs in time and sampling are in Appendix D.3.3.
Case II: Robust DQN for Atari Video Games
Evaluation Metrics.
Since Atari games have pixel state spaces and discrete action spaces, the applicable attacking algorithms also differ from those in MuJoCo tasks.
We include the following common attacks: (1) 10-step untargeted PGD (projected gradient descent) attack, (2) MinBest [19], which minimizes the probability of choosing the “best” action, (3) PA-AD [42], as the state-of-the-art
RL-based adversarial attack algorithm.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
Performance and Robustness of WocaR-DQN.
Table 1 presents the results on four Atari games under attack radius (cid:15)
= 3 / 255 , while results and analysis under smaller attack radius 1 / 255 are in
Appendix D.2.
We can see that our WocaR-DQN consistently outperforms baselines under MinBest and PA-AD attacks in all environments, with a signiﬁcant advance under the strongest (worst-case)
PA-AD attacks compared with other robust agents.
Under PGD attacks, WocaR-DQN performs comparably with the state-of-the-art in Freeway and Pong (which are simpler games) and gains higher rewards than other agents in BankHeist and Roadrunner.
Since SA-DQN and RADIAL-DQN focus on bounding and smoothing the policy network and do not consider the policy’s intrinsic vulnerability, they are robust under the PGD attack but still vulnerable against the stronger PA-AD attack.
Efﬁciency of Training WocaR-DQN.
The total training time for SA-DQN, RADIAL-DQN, and our WocaR-DQN are roughly 35, 17, and 18 hours, respectively.
All baselines are trained formillion frames on the same hardware.
Therefore, WocaR-DQN is 49% faster (and is more robust) than SA-DQN.
Compared to the more advanced baseline RADIAL-DQN, although WocaR-DQN is 5% slower, it achieves better robustness (539% higher reward than RADIAL-DQN in RoadRunner).
5.2 Verifying Effectiveness of WocaR-RL
Now we dive deeper into the algorithmic design and verify the effectiveness of WocaR-RL by ablation studies on WocaR-PPO.
(1) Worst-attack value estimation.
We show the learned worst-attack value estimation, Q πφ , during the training process in Figure 6a and 6c, in comparison with the actual reward under the strongest attack (PA-AD [42]) in Figure 6b and 6d.
The
pink curves in both plots suggest that our worstattack value estimation matches the trend of actual worst-case reward under attacks, although the network estimated value and the real reward have different scales due to the commonly-used reward normalization for learning stability.
Therefore, the effectiveness of our proposed worst-attack value estimation ( L est ) is veriﬁed.
(2) Worst-case-aware policy optimization.
Compared to vanilla PPO and SA-PPO, we can see that WocaR-PPO improves the worst-attack value and the worst-case reward during training , suggesting the effectiveness of our worst-attack value improvement ( L wst ).
The comparison of natural rewards,
[Figure]
as well as curves in other environments, are provided in Appendix D.3.4.
Moreover, the adjustable weight κ wst in Equation (8) controls the trade-off between natural value and worst-attack value in policy optimization.
When κ wst is high, the policy pays more attention to its worst-attack value.
Appendix D.3.5 veriﬁes that WocaR-RL, with different values of weight κ wst , produces different robustness and natural performance while consistently dominating other robust agents.
(3) Value-enhanced state regularization.
We conduct ablation experiments to analyze the effect of two techniques: our proposed state importance weight w ( s ) and the state regularization loss L reg [54].
In Figure 7a, we compare the performance of the original WocaR-PPO to a variant of WocaR-PPO without the state importance weight w ( s ) on Halfcheetah, which visually indicates that w ( s ) can help agents boost the robustness.
Since SA-PPO [54] also uses a state regularization technique, the improvement of SA-PPO added with w ( s ) also show the universal effectiveness of our state importance.
Without w ( s ) , our algorithm also achieves similar or better performance than baselines, but including this inexpensive technique w ( s ) gives WocaR-RL a greater advantage, especially under learned strong attacks SA-RL and PA-AD.
Figure 7b presents the performance of ATLA methods and our algorithm without L reg on Hopper, which veriﬁes that WocaR-PPO also yields the superior performance when removing the regularization technique.
And the comparison between WocaR-PPO and WocaR-PPO without L reg demonstrates that the weighted state regularization is beneﬁcial to enhancing the robustness in our algorithm.
Detailed ablation studies for w ( s ) and L reg on four
MuJoCo environments are shown in Appendix D.3.6.
Conclusion and Discussion
This paper proposes a robust RL training framework, WocaR-RL, that evaluates and improves the longterm robustness of a policy via worst-attack value estimation, worst-case-aware policy optimization, and value-enhanced state regularization.
Different from recent state-of-the-art adversarial training methods [42, 52] which train an extra adversary to improve the robustness of an agent, we directly estimate and improve the lower bound of the agent’s cumulative reward.
As a result, WocaR-RL not only achieves better robustness than state-of-the-art robust RL approaches, but also halves the total sample complexity and computation complexity, in a wide range of Atari and MuJoCo tasks.
There are several aspects to improve or extend the current approach.
First, the proposed worst-attack
Bellman operator in theory gives the exact worst-case value of a policy under (cid:96) p bounded attacks.
But in practice, it is hard to compute the set
A adv directly, so we use convex relaxation to obtain a superset of it, ˆ A adv .
As a result, the ﬁxed point of worst-attack Bellman operator with A adv being replaced by ˆ A adv is a lower bound of the worst-case value.
Then, our algorithm increases the worst-case value by improving its lower bound, as visualized and explained in Figure 8 in Appendix B. Therefore, one potential way of further improving the robustness is using a tighter relaxation.
In addition, this paper only considers the (cid:96) p threat model as is common in most related works.
But in real-world applications, other attack models could exist (e.g. patch attacks [5]), and improving the robustness of
RL agents in these scenarios is another important research direction.
Acknowledgments
This work is supported by DOD-ONR-Ofﬁce of Naval Research, DOD-DARPA-Defense Advanced
Research Projects Agency Guaranteeing AI Robustness against Deception (GARD), and Adobe,
Capital One and JP Morgan faculty fellowships.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] See our abstract and Section 1.
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitations in Appendix E. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss the potential negative societal impacts in Appendix E. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Our paper conforms to the guidelines. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix A. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] For each empirical result, we report error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See efﬁciency analysis in Appendix D.3.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material:
Efﬁcient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning
A Theoretical Analysis
Similar to the worst-attack action value, we can deﬁne the worst-attack value as below:
Deﬁnition A.1 (Worst-attack Value) .
For a given policy π , deﬁne the worst-attack value of π as
V
[Equation]
(9) where h ∗ is the optimal attacker which minimizes the victim’s cumulative reward under the (cid:15) constraint.
Proof of Theorem 4.2.
First, we show that T π is a contraction.
For any two Q functions Q 1 : S × A → R and Q 2 : S ×
A → R , we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The second inequality comes from the fact that,
[Equation]
The operator T π satisﬁes,
[Equation]
so it is a contraction in the sup-norm.
Recall the deﬁnition of worst-attack action value:
[Equation]
where h ∗ is the optimal attacker which minimizes the victim’s cumulative reward under the (cid:15) constraint.
That is, the optimal attacker h ∗ lets the agent select the worst possible action among all achievable actions in A adv .
Hence, we have Q π ( s, a ) =
T π Q π ( s, a ) .
Therefore, Q π ( s, a ) is the ﬁxed point of the Bellman operator T π .
[Figure]
B Geometric Understanding of WocaR-RL B.1 A Closer Look at Robust RL
In real-world applications where observations may be noisy or perturbed, it is important to ensure that the agent not only makes good decisions, but also makes safe decisions.
Existing Robust RL Approaches.
There are many existing robust training methods for RL, and we summarize the common ideas as the following two categories.
(1) Lipschitz-driven methods: encourage the policy to output similar actions for any pair of clean state and perturbed state, i.e., min θ max s ∈S , ˜ s ∈B (cid:15) ( s ) Dist ( π θ
( s ) , π θ (˜ s )) , where Dist can be any distance metric.
Therefore, the policy function (network) has small local Lipschitz constant at each clean state.
Note that this idea is similar to many certiﬁable robust training methods [14] in supervised learning.
For example, Fischer et al.
[9] achieve provable robustness for DQN by applying the DiffAI [31] approach, so that the DQN agent selects the same action for any element inside B (cid:15) ( s ) .
Zhang et al.
[54] propose to minimize the total variance between π ( s ) and π (˜ s ) using convex relaxations of
NNs.
Although Lipschitz-driven methods are relatively efﬁcient in training, they usually treat all states equally, and do not explicitly consider long-term rewards.
Therefore, it is hard to obtain a non-vacuous reward certiﬁcation, especially in continuous-action environments.
(2) Attack-driven methods: train the agent under adversarial attacks, which is analogous to Adversarial
Training (AT) [28].
However, different from AT, a PGD attacker may not induce a robust policy in an RL problem due to the uncertainty and complexity of the environment.
Zhang et al.
[52] propose to alternately train an agent and an RL-based “optimal” adversary, so that the agent can adapt to the worst-case input perturbation.
Therefore, attack-driven method can be formulated as max θ V π θ .
Zhang et al.
[52] and a follow-up work by Sun et al.
[42] apply the alternate training approach and obtain state-of-the-art robust performance.
However, learning the optimal attacker using
RL algorithms doubles the learning complexity and the required samples, making it hard to apply these methods to large-scale problems.
Moreover, although these attack-driven methods improve the worst-case performance of an agent, the natural reward can be sacriﬁced.
Note that we discuss methods that improve the robustness of deep policies during training.
Therefore, the focus is different from some important works [27, 49, 23] that directly use non-robust policies and execute them in a robust way.
Our Motivation: Geometric Understanding of Robust RL.
The robustness of a learned RL policy can be understood from a geometric perspective.
Dadashi et al.
[8] point out that the value functions of all policies in a ﬁnite MDP form a polytope, as shown by the grey area in Figure 8.
Sun et al.
[42] further ﬁnd that V ˜ π , possible values of a policy π under all (cid:15) -constrained (cid:96) p perturbations, also form a polytope (pink area in Figure 8), which we refer to as the value perturbation polytope .
Recall that in robust RL, we pursue a high natural value V π , and a high worst-case value V π which is the lower leftmost vertex of the value perturbation polytope.
A vulnerable policy that outputs a different action for a perturbed state as a larger value perturbation polytope.
Lipschitz-driven methods, as Figure 8(a) shows, attempts to shrink the size of the value perturbation polytope, but does not necessarily result in a high V π .
Attack-driven methods, as Figure 8 shows, improves V π , but have no control over the size of the value perturbation polytope, and may not obtain a high natural value V π .
Our Proposed Robust RL Principle.
In contrast to prior Lipschitz-driven methods and Attackdriven methods, we propose to both “lift the position” and “shrink the size” of the value perturbation polytope.
To achieve the above principle in an efﬁcient way, we propose to (1) directly estimate and optimize the worst-case value of a policy without training the optimal attacker (worst-attack value estimation and worst-case-aware policy optimization mechanisms of WocaR-RL), and (2) regularize the local Lipschitz constants of the policy with value-enhanced weights (value-enhanced state regularization mechanism of WocaR-RL).
See Section 4 for more details of the proposed algorithm.
C Algorithm Details C.1 Computing A adv by Network Bounding Techniques
Recall that A adv ( s, π ) =
{ a ∈ A : ∃ ˜ s ∈ B (cid:15) ( s )
s.t.
π (˜ s )
= a } is the set of actions that π may be misled to select in state s .
Computing the exact A adv is difﬁcult due to the complexity of neural networks, so we use relaxations of network such as Interval Bound Propagation (IBP) [48, 15] to approximately calculate A adv .
A Brief Introduction to Convex Relaxation Methods.
Convex relaxation methods are techniques to bound a neural network that provide the upper and lower bound of the neural network output given a bounded l p perturbation to the input.
In particular, we take l ∞ as an example, which has been studied extensively in prior works.
Formally, let f θ be a real-valued function parameterized by a neural network θ , and let f θ ( s ) denote the output of the neural network with the input s .
Given an l ∞ perturbation budget (cid:15) , convex relaxation method outputs ( f θ ( s ) , f θ ( s )) such that
[Equation]
Recall that we use π θ to denote the parameterized policy being trained that maps a state observation to a distribution over the action space, and π denotes the deterministic policy reﬁned from π θ with π ( s ) = argmax a ∈A π θ ( a | s ) .
A adv ( s, π ) contains actions that could be selected by π (with the highest probability in π θ ’s output) when s is perturbed within a (cid:15) -radius ball.
Our goal is to approximately identify a superset of A adv ( s, π ) , i.e., ˆ A adv ( s, π ) , via the convex relaxation of networks introduced above.
Computing A adv in Continuous Action Space.
The most common policy parameterization in a continuous action space is through a Gaussian distribution.
Let µ θ ( s ) be the mean of Gaussian computed by π θ ( s ) , then π = µ ( s ) .
Therefore, we can use network relaxation to compute an upper bound and a lower bound of µ θ with input B (cid:15) ( s ) .
Then, ˆ A adv ( s, π ) =
[ µ θ ( s ) , µ θ ( s )] , i.e., a set of actions that are coordinate-wise bounded by µ θ ( s ) and µ θ ( s ) .
For other continuous distributions, e.g., Beta distribution, the computation is similar, as we only need to ﬁnd the largest and smallest actions.
In summary, we can compute ˆ A adv ( s, π ) =
[ π θ ( s ) , π θ ( s )] .
Computing A adv in Discrete Action Space.
For a discrete action space, the output of π θ is a categorical distribution, and π selects the action with the highest probability.
Or equivalently, in value-based algorithms like DQN, the Q network (can be regarded as π θ ) outputs the Q estimates for each action, and π selects the action with the highest Q value.
In this case, we can compute the upper and lower bound of π θ in every dimension (corresponding to an action), denoted as a i , a i , ∀ 1 ≤ i ≤ |A| .
Then, an action a i ∈ A is in ˆ A adv if for all 1 ≤ j ≤ |A| , j (cid:54)
=
i , we have a i > a j .
Implementation details of A adv For a continuous action space, interval bound propagation (IBP) is the cheapest method to implement convex relaxation.
We use IBP+Backward relaxation provided by auto_LiRPA library, following [54] to efﬁciently produce tighter bounds A adv for the policy networks π theta .
For a discrete action space, we compute the layer-wise output bounds for the Q-network by applying robustness veriﬁcation algorithms from [33].
C.2 Worst-case-aware Robust PPO (WocaR-PPO)
In policy-based DRL methods [38, 25, 39] such as PPO, the actor policy π θ is optimized so that it increases the probability of selecting actions with higher critic values.
Therefore, we combine our worst-attack critic and the original critic function, and optimize π θ such that both the natural value ( L RL ) and the worst-attack action value Q πφ ( L wst ) can be increased ( L RL and L wst ).
At the same time, π θ is also regularized by L reg .
We provide the full algorithm of WocaR-PPO in Algorithm 1 and highlight the differences with the prior method SA-PPO.
WocaR-PPO needs to train an additional worst-attack critic Q πφ to provide the robust-PPO-clip objective.
The perturbation budget (cid:15) t increases slowly during training.
The implementation of L reg is the same as the SA-regularizer [54].
For computing the state importance weight w s t , because there is no Q-value network in PPO, we provide a different formula to measure the state importance without extra calculation (Line 11 in Algorithm 1).
C.3 Worst-case-aware Robust DQN (WocaR-DQN)
For value-based DRL methods [32, 16, 47] such as DQN, a Q network is learned to evaluate the natural action value.
Although the policy is not directly modeled by a network, the Q network induces a greedy policy by π ( s ) = argmax a Q ( s, a ) .
To distinguish the acting policy and the natural action value, we keep the original Q network, and learn a new Q network that serves as a robust policy.
This new Q network is called a robust Q network , denoted by Q r , which is used to take greedy actions a = π ( s ) :
= argmax a Q r ( s, a )
In addition to the original vanilla Q network Q v and the robust Q network Q r , we learn the worst-attack critic network Q πφ , which evaluates the worst-attack action value of the greedy policy induced by Q r .
Then, we update Q r by assigning higher values for actions with both high natural Q value and high worst-attack action value ( L RL and L wst ), while enforcing the network to output the same action under bounded state perturbations ( L reg ).
WocaR-DQN is presented in Algorithm 2.
WocaR-DQN trains three Q-value functions including a vanilla Q network, a worst-case Q network, and a robust Q network.
The worst-case Q Q πφ is learned to estimate the worst-case performance and the robust Q is updated using the vanilla value and worst-case value together.
Moreover, a target Q network is used as the original DQN implementation, to compute the target value when updating the vanilla Q network (Line 8 to 10 in Algorithm 2).
To learn the worst-case critic Q πφ , we select the worst-attack action from the estimated possible perturbed action set ˆ A adv to compute the worst-case TD loss L est (Line 11 to 15).
The implementation of L reg is the same as the SA-regularizer [54], where the robust Q network is regularized.
To update the robust Q, we use a special y ri which combines the target Q Q v (cid:48) and Q r for the next state to compute the TD loss, and minimize the L reg weighted by the state importance w ( s i )
(Line 16 to 17).
In
WocaR-DQN, we use an increasing (cid:15) t schedule and a more slowly increasing worst-case schedule κ wst ( t ) for robust Q training.
C.4 Worst-case-aware Robust A2C (WocaR-A2C)
We also provide WocaR-A2C based on A2C implementation in Algorithm 3.
Differ from the original
A2C, WocaR-A2C needs to learn an additional Q πφ similar to WocaR-PPO.
To learn Q πφ , we compute the output bounds for the policy network π θ π under (cid:15) -bounded perturbations and then select the worst action ˆ a t +1 to calculate the TD-loss L est (Line 6 to 9).
The solutions for state importance weight w ( s t ) and regularization L reg are same as WocaR-PPO
(Line 10-11).
To learn the policy network π θ π , we minimize the Q πφ value together with the original actor loss (Line 12).
C.5 Extension to Action Attacks
Although our paper mainly focuses on state attack, our proposed techniques and algorithms based on the worst-attack Bellman operator can be easily extended to action attack, which is another threat model studied in previous works [35, 44, 45].
In fact, for action attack, we even do not need to apply
|D| | k | τ k ∈D | τ k | (cid:88) t
= (cid:16) V θ V ( s t
− ˆ R t (cid:17)
Calculate a the
= (cid:26) t + γQ πφ ( s t +1 , ˆ a t +
[Equation]
) for non-terminal s t + 10:
Update parameters of worst-attack critic network φ by minimizing the TD-error ( L est ):
[Equation]
For each state s t , calculate a state importance weight w s t by V θ V ( s t )
− min a Q πφ ( s t , a ) for s t 12:
Solve the value-enhanced state regularization loss by SGLD (Stochastic gradient Langevin dynamics [12]) (from SA-PPO [54]):
[Equation]
Update the policy network by minimizing the Robust-PPO-Clip objective (via ADAM):
[Equation]
where ρ θ (cid:48)
[Equation]
( a
[Equation]
:=
[Equation]
, g
[Equation]
:= clip
[Equation]
14: end for
IBP for the worst-attack Bellman backup.
We could just simply replace A adv with the set of actions that the agent could take under attack, then the rest of the algorithms will follow the exact same as the ones presented here.
Computing bounds of robust action-value function:
For each action a in action space A , calculate the output bounds of robust action-value function Q r under (cid:15) t -bounded perturbations using IBP to input
s i +1 :
Q l ( s t +1 , a, (cid:15) t ) and
Q u ( s t +1 , a, (cid:15) t ) .
Find the possible perturbed action set:
For every action a ∈ A , if Q u ( s t +1 , a, (cid:15) t ) > Q l ( s t +1 , a (cid:48) , (cid:15) t ) , ∀ a (cid:48)
∈ A , then add a in the perturbed action set ˆ A adv 13:
Calculate the worst-attack action: ˆ a i +1 = arg min a ∈
ˆ
A adv
Q πφ ( s i +1 , a ) .
Set y i
= (cid:26) r i for terminal s i +1 r
[Equation]
) for non-terminal s i + 15:
Compute TD-loss for worst-attack critic: L est =
( y i − Q πφ ( s i , a i ; φ )) 2 and perform a gradient descent step with respect to the parameters φ 16:
Calculate the state importance w s i for each s i by normalizing max a Q v ( s t , a ) − min a Q v ( s t , a ) 17:
Update the robust Q function Q r based on the modiﬁed TD-Loss and value-enhanced state regularization:
[Equation]
where y ri =
r i + γ max a (cid:48) (cid:104) κ wst ( t ) Q v (cid:48)
( s i +1 , a (cid:48) ; θ ) + (1 − κ wst ( t ))
Q πφ ( s i +1 , a (cid:48) ; θ ) (cid:105) if s i + is a non-terminal state, otherwise y ri = r
i 18: end for
D Experiment Details and Additional Results D.1 Implementation Details
For reproducibility, the reported results are selected from 30 agents for different training methods with medium performance due to the high variance in RL training.
D.1.1 PPO in MuJoCo
(a) PPO Baselines
Vanilla PPO
We use the optimal hyperparameters from [54] with the original fully connected (MLP) structure as the policy network for vanilla PPO training on all environments.
On Hopper,
Walker2d and Halfcheetah, we train for 2 million steps (976 iterations) , and 10 million steps ( iterations) on Ant to ensure convergence, which are consistent with other baselines (except ATLA methods).
[Equation] [Equation]
V
[Equation] [Equation]
[Equation] [Equation]
A
[Equation]
Calculate the satisﬁed ˆ a t +
= min
a
Q πφ s with the worst-attack critic network
Q
Compute
Set y t
(cid:26) for terminal s t +1 r t + γQ πφ ( s t +1 , ˆ a t +1 ) terminal s
[Equation]
arg
[Equation]
(cid:88)
[Equation]
For each state s t , calculate a state importance weight w ( s t ) by V θ V ( s t )
− min a Q πφ ( s t , a ) for s t 11:
Solve the value-enhanced state regularization loss [53] by SGLD (Stochastic gradient Langevin dynamics [12]):
[Equation]
Update the policy network by (via ADAM)
[Equation]
13: end for
SA-PPO
We use the hyperparameters using a grid search and solve the regularizer using convex relaxation with the IBP+Backward scheme to solve the regularizer.
The regularization parameter kappa is chosen in { 0 . 01 , 0 . 03 , 0 . 1 , 0 . 3 , 1 . 0 } .
ATLA-PPO
The hyperparameters for both policy and adversary are tuned for vanilla PPO with
LSTM models.
A larger entropy bonus coefﬁcient is set to allow sufﬁcient exploration.
We set
N v = N π = 1 for all experiments.
We train 2441 iterations for Hopper, Walker2d, and Halfcheetah as well as 4882 iterations for Ant.
PA-ATLA-PPO
We use the hyperparameters similar to ATLA-PPO and conduct a grid search for a part of adversary hyperparameters including the learning rate and the entropy bonus coefﬁcient.
RADIAL-PPO RADIAL-PPO applies the same value of hyperparameters from [33].
We train agents with the same iterations aligning vanilla PPO for fair comparison.
(b) PPO Attackers
For Random and MaxDiff attack, we directly use the implementation from [52].
The reported rewards under RS attack are from 30 trained robust value function, which is used to attack agents.
For SA-RL attack, a grid search of the optimal hyperparameters for each robust agents is conducted to ﬁnd the strongest attacker.
The strength of the regularization κ is set as 1 × 10 − 6 to 1.
For PA-AD attack, the adversaries are trained by PPO with a grid search of hyperparameters to obtain the strongest adversary.
For different types of RL-based attacks, we respectively train 100 adversaries and report the worst rewards among all trained adversaries.
(c) WocaR-PPO
We use the same LSTM structure (single layer with 64 hidden neurons as in vanilla PPO agents.
With a grid search experiment, we ﬁnd the optimal hyperparameters for WocaRPPO.
Specially, we use PGD to compute bounds for the policy network and convex relaxation to solve the state regularization.
The number of WocaR-PPO training steps in all environments are the same as those in vanilla PPO.
We tune the adjustable weight κ wst and increase κ wst from 0 to the target value.
For Hopper, Walker2d and Halfcheetah, κ wst is linearly increasing and we set the target value as 0.8.
For Ant, we choose the exponential increase and the target value as 0.5.
D.1.2 DQN in Atari
[Figure]
Vanilla DQN
We follow [54] and [33] in hyperparameters and network structures for vanilla
DQN training.
The implementation of all our baselines applies Double DQN [17] and Prioritized
Experience Replay [37].
For each Atari environment without framestack, we normalize the pixel values to [0 , 1] and clip rewards to [ − 1 , +1] .
For reliably convergence, we run 6 × 10 6 steps for all baselines on all environments.
Additionally, we use a replay buffer with a
capacity of 5 × 10 .
During testing, we evaluate agents without epsilon greedy exploration for 1000 episodes.
SA-DQN SA-DQN use the same settings of network structures and hyperparameters as in vanilla DQN.
The regularization parameter κ is chosen from 0 .
02 and the schedule of (cid:15)
during training also follows [54].
RADIAL-DQN
Following the original implementation from [33], we reproduce the results of RADIAL-DQN with our environment settings.
(b) DQN Attackers
For PGD attacks, we apply 10-step untargeted PGD attacks.
We also try 50-step PGD attacks, but we ﬁnd that the rewards of robust agents do not further reduce.
For
MinBest attacks,
we use
FGSM
to compute state perturbations following
For PA-AD attacks, the PA-AD attackers are learned with the ACKTR algorithm.
We use a learning rate 0 . 0001 and train the attackers for 5 million frames.
(c) WocaR-DQN For WocaR-DQN, we keep the same network architectures and hyperparameters as in vanilla DQN agents.
During training, we set the adjustable weight κ wst as 0 for the ﬁrst 2 × 10steps, and then exponentially increase it from 0 to 0.5 for 4 × 10 6 steps.
D.2 Additional Experiment Results on Robustness Performance
MuJoCo Experiments We reported all results in Table 2 including episode rewards of well-trained robust models under various adversarial attacks.
Under this full adversarial evaluation, we provide a robustness comparison between baselines and our algorithm from a comprehensive angle.
We report the attack performance under a common chosen perturbation budget (cid:15) following [54, 52].
Results in all four MuJoCo environments show that our WocaR-PPO is the most robust method.
We emphasize that Table 2 reports the ﬁnal performance of all robust training baselines after convergence, but some baselines takes much more steps than our WocaR-PPO.
Table 5 in Appendix D.3.2 compares all methods under the same number of training steps, where WocaR-PPO outperforms baselines more signiﬁcantly.
[Table]
Atari Experiments In Table
[Table]
we present performance based on DQN on four Atari environments under 1/255 and 3/255 (cid:15) attack.
Under (cid:15) of 1/255, our WocaR-DQN achieves competitive performance under PGD attacks and outperforms all baselines under MinBest and PA-AD attacks, which shows better robustness of WocaR-DQN under weaker attacks.
Based on vanilla A2C
[Table]
we implement SA-A2C[54] and PA-ATLA-A2C[42] as robust baselines.
We implement WocaR-A2C to compare with ATLA methods on Atari.
In Table
[Table]
under any (cid:15) value
[Table]
our
WocaR-A2C outperforms other robust baselines across different attacks.
We can conclude that our method considerably enhance more robustness than ATLA methods on Atari.
[Table]
[Figure]
a
[Figure] [Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table]
D.3 Additional Evaluation and Ablation Studies D.3.1 Robustness Evaluation Using Multiple (cid:15)
To study how WocaR-PPO performs under attacks with different value of (cid:15) , Figure 9 shows the evaluation of our algorithms under different (cid:15) attacks compared with the baselines in Hopper and
Walker2d.
We can conclude that our robustly trained model universally and signiﬁcantly outperforms other robust agents considering various attack budget (cid:15) .
D.3.2 Additional Evaluation on Sample Efﬁciency
In Table 5, we report the performance of WocaR-PPO and all robust PPO baselines using the same training steps.
We ﬁnd that under limited training steps, ATLA-PPO, PA-ATLA-PPO and RADIALPPO obtain sub-optimal robustness, which suggests that these methods are more sample-hungry.
In contrast, WocaR-PPO converges under fewer steps and achieves best performance with a large advantage, which shows the higher efﬁciency of WocaR-PPO.
D.3.3 Additional Results of Time Efﬁciency
We show the training efﬁciency of WocaR-PPO from three aspects including time, training iterations, and sampling in MuJoCo environments by comparing with SA-PPO and state-of-the-art methods
ATLA-PPO, PA-ATLA-PPO, and RADIAL-PPO in Table 6.
For a fair comparison, we use the same
GeForce RTX 1080 Ti GPUs to train all the robust agents.
It needs to mention that in continuous action spaces when estimating the worst-case value, we solve min ˆ a ∈ ˆ A adv Q πφ ( s t +1 , ˆ a ) using 50-step gradient descent.
The running time of this 50-step
[Table] [Table]
gradient descent is about 1.68 seconds per batch with batch size 128.
In total, this gradient descent computation takes 18% of the total training time, thus it is not the computation bottleneck.
Without training with an adversary, our algorithm requires much less (only 50% or 75%) steps to reliably converge.
WocaR-PPO only takes less than half of time for low-dimensional environments to converge compared to ATLA methods and RADIAL-PPO.
In high-dimensional environments like
Ant, we only need 4 hours for training, while ATLA methods require at least 7 hours.
When solving harder tasks, the efﬁciency advantage of WocaR-PPO is more obvious.
D.3.4 Effectiveness of Worst-attack Policy Optimization
In addition to Figure 6, we show the learning curves in Walker2d and Ant in Figure 10 to verify the effectiveness of worst-attack value estimation and worst-case-aware policy optimization.
Figure 10(a) and (d) show the natural rewards of agents during training without attacks.
The actual worst-attack rewards in Figure 10(b) and (e) refer to the the reward obtained by the agents under PA-AD attack [42] which is the existing strongest attacking algorithm.
To study the worst-case performance during training, We evaluate PPO, SA-PPO and WocaR-PPO agents after every 20 iterations using all types of attacks and report the worst-case rewards for each checkpoint.
We also present the trend of the estimated worst-case values during training in Figure 10(c) and (f), which are tested by the trained worst-attack value functions Q πφ .
We observe from the curves that our worst-attack critic estimation matches the trend of actual worstattack rewards.
Also, the increases of estimated worst-attack values and actual worst-attack rewards of
WocaR-PPO show that our WocaR-RL signiﬁcantly improves the robustness of agents by enhancing worst-attack values.
[Figure] [Figure]
a
[Figure]
s
[Figure]
D.3.5 Trade-off between Natural Performance and Robustness
As mentioned in Section 5.
[Figure]
the adjustable weight κ wst controls the trade-off between natural performance and robustness.
To discuss the effect of κ wst , we train agents using WocaR-PPO in
Hopper, Walker2d
[Figure]
and Halfcheetah with uniformly sampled 40 different values of weight κ wst in range (0 , 1] .
Figure 11 plots the worst-case performance and natural performance of robust training baselines and 10 agents trained by WocaR-PPO with various values of κ wst .
We can see that when reward under worst-case perturbations increases, it leads to a reduction of the natural reward.
The choice of the worst-case value’s weight κ wst is to control the trade-off between the ﬁnal natural performance and robustness.
It does not affect the convergence of the algorithm.
When we increase the weight of worst-case values κ wst , the reward under worst-case perturbations increases, but it leads to a reduction of the natural reward.
Equally, when κ wst is set close to 0, the algorithm is similar to standard training, where the policy achieves high reward under no attack, but extremely low reward under attacks.
Hence, κ wst is necessary for our algorithm to balance these two kinds of performance.
In practice, one can adjust κ wst according to their preferences to robustness and natural performance.
We report the results in Table 2 with signiﬁcant better worst-case robustness and comparable natural performance compared with baselines.
WocaR-PPO can always ﬁnd policies which dominate other robust agents.
[Figure] [Figure]
D.3.6 Additional Ablation Studies
We provide full ablation experimental results for the state importance weight w ( s ) and the regularization loss L reg [54] on four MuJoCo environments.
For the state importance weight w ( s )
[Figure]
we compare the performance between the original WocaR-PPO and WocaR-PPO without w ( s ) in Figure 12.
Additionally, we also equip SA-PPO with w ( s ) to show the universal applicability of this design.
In all four MuJoCo environments, we can see that with w ( s ) , both WocaR-PPO and SA-PPO get boosted robustness, verifying the effectiveness of the state importance weight.
For the state regularization loss L reg , Figure 13 veriﬁes that L reg enhances the robustness of WocaRPPO, since the performance of WocaR-PPO drops without L reg .
On the other hand, Figurealso compares the performance of ATLA methods and our algorithm without L reg (note that ATLA methods also regularizes the PPO policies during training).
The results indicate that the decisive contribution of WocaR-PPO to robustness improving comes from the worst-attack-aware policy optimization.
These ablation studies demonstrate that all the techniques are beneﬁcial for robustness improvement and further show that our worst-case-aware training performs better than training with attackers.
E Potential Societal Impacts
This work focuses on improving the robustness of deep RL agents, which can make RL models more reliable in high-stakes applications.
Although it is generally positive for the community to build more robust agents, such robust agents may also bring some potentially negative impacts, including the possibility of robust robots replacing some occupations and causing mass unemployment.