W HO I S THE S TRONGEST E NEMY ? T OWARDS O PTIMAL AND E FFICIENT E VASION A TTACKS IN D EEP RL Anonymous authors
Paper under double-blind review
A BSTRACT
Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents.
However, ﬁnding the optimal adversary is challenging, in terms of both whether we can ﬁnd the optimal attack and how efﬁciently we can ﬁnd it.
Existing works on adversarial RL either use heuristics-based methods that may not ﬁnd the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can ﬁnd the optimal adversary but may become intractable in a large state space.
This paper introduces a novel attacking method to ﬁnd the optimal attacks through collaboration between a designed function named “actor” and an RL-based learner named “director”.
The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions.
Our proposed algorithm, PA-AD, is theoretically optimal and signiﬁcantly more efﬁcient than prior RL-based works in environments with large state spaces.
Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in various Atari and
MuJoCo environments.
By applying PA-AD to adversarial training, we achieve state-of-the-art empirical robustness in multiple tasks under strong adversaries.
I NTRODUCTION
Deep Reinforcement Learning (DRL) has achieved incredible success in many applications.
However, recent works (Huang et al., 2017; Pattanaik et al., 2018) reveal that a well-trained RL agent may be vulnerable to test-time evasion attacks , making it risky to deploy RL models in high-stakes applications.
As in most related works, we consider a state adversary which adds imperceptible noise to the observations of an agent such that its cumulative reward is reduced during test time.
In order to understand the vulnerability of an RL agent and to improve its certiﬁed robustness, it is important to evaluate the worst-case performance of the agent under any adversarial attacks with certain constraints.
In other words, it is crucial to ﬁnd the strongest/optimal adversary that can minimize the cumulative reward gained by the agent with ﬁxed constraints, as motivated in a recent paper by Zhang et al.
Therefore, we focus on the following question:
Given an arbitrary attack radius (budget) (cid:15) for each step of the deployment, what is the worstcase performance of an agent under the strongest adversary?
[Figure]
Finding the strongest adversary in RL is challenging.
Many existing attacks (Huang et al., 2017; Pattanaik et al., 2018) are based on heuristics, crafting adversarial states at every step independently, although steps are interrelated in contrast to image classiﬁcation tasks.
These heuristic methods can often effectively reduce the agent’s reward, but are not guaranteed to achieve the strongest attack under a given budget.
This type of attack is “myopic” since it does not plan for the future.
Figure 1 shows an intuitive example, where myopic adversaries only prevent the agent from selecting the best action in the current step, but the strongest adversary can strategically “lead” the agent to a trap, which is the worst event for the agent.
Achieving computational efﬁciency arises as another challenge in practice, even if the strongest adversary can be found in theory.
A recent work (Zhang et al., 2020a) points out that learning the optimal state adversary is equivalent to learning an optimal policy in a new Markov Decision Process (MDP).
A follow-up work (Zhang et al., 2021) shows that the learned adversary signiﬁcantly outperforms prior adversaries in MuJoCo games.
However, the state space and the action space of the new
MDP are both as large as the state space in the original environment, which can be high-dimensional in practice.
For example, video games and autonomous driving systems use images as observations.
In these tasks, learning the state adversary directly becomes computationally intractable.
To overcome the above two challenges, we propose a novel attack method called Policy Adversarial Actor Director (PA-AD) , where we design a “director” and an “actor” that collaboratively ﬁnds the optimal state perturbations.
In PA-AD, a director learns an MDP named Policy Adversary
MDP (PAMDP) , and an actor is embedded in the dynamics of PAMDP.
At each step, the director proposes a perturbing direction in the policy space, and the actor crafts a perturbation in the state space to lead the victim policy towards the proposed direction.
Through a trail-and-error process, the director can ﬁnd the optimal way to cooperate with the actor and attack the victim policy.
Theoretical analysis shows that the optimal policy in PAMDP induces an optimal state adversary.
The size of PAMDP is generally smaller than the adversarial MDP deﬁned by Zhang et al.
and thus is easier to be learned efﬁciently using off-the-shelf RL algorithms.
With our proposed director-actor collaborative mechanism , PA-AD outperforms state-of-the-art attacking methods on various types of environments, and improves the robustness of many DRL agents by adversarial training.
Summary of Contributions
(1) We establish a theoretical understanding of the optimality of evasion attacks from the perspective of policy perturbations, allowing a more efﬁcient implementation of optimal attacks.
(2) We introduce a Policy Adversary MDP (PAMDP) model, whose optimal policy induces the optimal state adversary under any attacking budget (cid:15) .
(3) We propose a novel attack method, PA-AD, which efﬁciently searches for the optimal adversary in the PAMDP.
PA-AD is a general method that works on stochastic and deterministic victim policies, vectorized and pixel state spaces, as well as discrete and continuous action spaces.
(4) Empirical study shows that PA-AD universally outperforms previous attacking methods in various environments, including Atari games and MuJoCo tasks.
PA-AD achieves impressive attacking performance in many environments using very small attack budgets, (5) Combining our strong attack PA-AD with adversarial training, we signiﬁcantly improve the robustness of RL agents, and achieve the state-of-the-art robustness in many tasks
P RELIMINARIES AND N OTATIONS
The Victim RL Agent
In RL, an agent interacts with an environment modeled by a Markov
Decision Process (MDP) denoted as a tuple M = (cid:104)S , A , P, R, γ (cid:105) , where S is a state space with cardinality |S| , A is an action space with cardinality |A| , P : S
×
A → ∆( S ) is the transition function 1 , R : S ×
A → R is the reward function, and γ ∈ (0 , 1) is the discount factor.
In this paper, we consider a setting where the state space is much larger than the action space, which arises in a wide variety of environments.
For notation simplicity, our theoretical analysis focuses on a ﬁnite
MDP, but our algorithm applies to continuous state spaces and continuous action spaces, as veriﬁed in experiments.
The agent takes actions according to its policy , π : S → ∆( A ) .
We suppose the victim uses a ﬁxed policy π with a function approximator (e.g. a neural network) during test time.
We denote the space of all policies as Π , which is a Cartesian product of |S| simplices.
The value of a policy π ∈ Π for state s ∈ S is deﬁned as V π ( s ) =
E π,P [ (cid:80)
∞
t =
γ t R ( s t , a t ) | s 0 = s ] .
Evasion Attacker
Evasion attacks are test-time attacks that aim to reduce the expected total reward gained by the agent/victim.
As in most literature (Huang et al., 2017; Pattanaik et al., 2018;
Zhang et al., 2020a), we assume the attacker knows the victim policy π (white-box attack).
However, the attacker does not know the environment dynamics, nor does it have the ability to change the environment directly.
The attacker can observe the interactions between the victim agent and the environment, including states, actions and rewards.
We focus on a typical state adversary (Huang et al., 2017; Zhang et al., 2020a), which perturbs the state observations returned by the environment before the agent observes them.
Note that the underlying states in the environment are not changed.
Formally, we model a state adversary by a function h which perturbs state s ∈ S into ˜ s :
= h ( s ) , so that the input to the agent’s policy is ˜ s instead of s .
In practice, the adversarial perturbation is usually under certain constraints.
In this paper, we consider the common (cid:96) p threat model (Goodfellow et al., 2015): ˜ s should be in B (cid:15) ( s ) , where B (cid:15) ( s ) denotes an (cid:96) p norm ball centered at s with radius (cid:15) ≥ 0 , a constant called the budget of the adversary for every step.
With the budget constraint, we deﬁne the admissible state adversary and the admissible adversary set as below.
Deﬁnition 1 ( Set of Admissible State Adversaries H (cid:15) ) .
A state adversary h is said to be admissible if ∀ s ∈ S , we have h ( s ) ∈ B (cid:15) ( s ) .
The set of all admissible state adversaries is denoted by H (cid:15) .
Then the goal of the attacker is to ﬁnd an adversary h ∗ in H (cid:15) that maximally reduces the cumulative reward of the agent.
In this work, we propose a novel method to learn the optimal state adversary through the identiﬁcation of an optimal policy perturbation deﬁned and motivated in the next section.
U NDERSTANDING O PTIMAL A DVERSARY VIA P OLICY P ERTURBATIONS
In this section, we ﬁrst motivate our idea of interpreting evasion attacks as perturbations of policies, then discuss how to efﬁciently ﬁnd the optimal state adversary via the optimal policy perturbation.
Evasion Attacks Are Perturbations of Policies Although existing literature usually considers state-attacks and actionattacks separately, we point out that evasion attacks, either applied to states or actions, are essentially equivalent to perturbing the agent’s policy π into another policy π h in the policy space Π .
For instance, as shown in Figure 2, if the adversary h alters state s into state ˜ s , the victim selects an action ˜ a based on π ( ·| ˜ s ) .
This is equivalent to directly perturbing π ( ·| s ) to π h ( ·| s ) :
=
π ( ·| ˜ s ) .
(See Appendix A for more detailed analysis including action adversaries.)
In this paper, we aim to ﬁnd the optimal state adversary through the identiﬁcation of the “optimal policy perturbation”, which has the following merits .
(1) π h ( ·| s ) usually lies in a lower dimensional space than h ( s ) for an arbitrary state s ∈ S .
For example, in Atari games, the action space is discrete and small (e.g. |A| = 18 ), while a state is a high-dimensional image.
Then the state perturbation h ( s ) is an image, while π h ( ·| s ) is a vector of size |A| .
(2) It is easier to characterize the optimality of a policy perturbation than a state perturbation.
How a state perturbation changes the value of a victim policy depends on both the victim policy network and the environment dynamics.
In contrast, how a policy perturbation changes the victim value only depends on the environment.
Our Theorem 4 in Section 3 and Theorem 12 in Appendix B both provide insights about how V π changes as π changes continuously.
(3) Policy perturbation captures the essence of evasion attacks, and uniﬁes state and action attacks.
Although this paper focuses on state-space adversaries, the learned “optimal policy perturbation” can also be used to conduct action-space attacks against the same victim.
Characterizing the Optimal Policy Adversary As depicted in Figure 3, the policy perturbation serves as a bridge connecting the perturbations in the state space and the value space.
Our goal is to ﬁnd the optimal state adversary by identifying the optimal “policy adversary”.
We ﬁrst deﬁne an Admissible Adversarial Policy Set (Adv-policy-set)
B H(cid:15)
( π ) ⊂
Π as the set of policies perturbed from π by all admissible state adversaries h ∈ H (cid:15) .
In other words, when a state adversary perturbs states within an (cid:96) p norm ball B (cid:15) ( · ) , the victim policy is perturbed within B H(cid:15) ( π ) .
Deﬁnition 2 ( Admissible Adversarial Policy Set (Adv-policy-set)
B H(cid:15) ( π ) ) .
For an MDP M , a ﬁxed victim policy π , we deﬁne the admissible adversarial policy set (Adv-policy-set) w.r.t.
π , denoted by B H(cid:15) ( π ) , as the set of policies that are perturbed from π by all admissible adversaries, i.e.,
[Equation]
Remarks (1) B H(cid:15) ( π ) is a subset of the policy space Π
and it surrounds the victim π , as shown in Figure 3(middle).
In the same MDP, B H(cid:15) ( π ) varies for different victim π or different attack budget (cid:15) .
In Appendix B, we characterize the topological properties of B H(cid:15) ( π ) .
We show that for a continuous function π (e.g., neural network), B H(cid:15) ( π ) is connected and compact, and the value functions generated by all policies in the Adv-policy-set B H(cid:15) ( π ) form a polytope (Figure 3(right)), following the polytope theorem by Dadashi et al.
𝑠 𝑆 ℎ(𝑠)
𝜋 𝜋 !
Π 𝑉 𝑉 " 𝑉 " !
End-to-end
SA-RL:
An RL problem in a large MDP
PA-AD (ours):
An RL problem in a small MDP
An optimization problem
The D irector The A ctor (Policy network)
Environment 𝑩 𝝐𝑯 (𝝅) 𝑉 !
∗ 𝑉 ! "
∗ 𝑩 𝝐 (𝒔)
Victim 𝝅
Given that the Adv-policy-set B H(cid:15) ( π ) contains all the possible policies the victim may execute under admissible state perturbations, we can characterize the optimality of a state adversary through the lens of policy perturbations.
Recall that the attacker’s goal is to ﬁnd a state adversary
h ∗ ∈ H (cid:15) that minimizes the victim’s expected total reward.
From the perspective of policy perturbation, the attacker’s goal is to perturb the victim’s policy to another policy π h ∗ ∈ B H(cid:15)
( π ) with the lowest value.
Therefore, we can deﬁne the optimal state adversary and the optimal policy adversary as below.
Deﬁnition 3 ( Optimal State Adversary h ∗ and Optimal Policy Adversary π h ∗ ) .
For an MDP
M , a ﬁxed policy π , and an admissible adversary set H (cid:15) with attacking budget (cid:15) ,
(1) an optimal state adversary h ∗ satisﬁes h ∗ ∈ argmin h ∈ H (cid:15) V π h ( s ) , ∀ s ∈ S , which leads to (2) an optimal policy adversary π h ∗ satisﬁes π h ∗ ∈ argmin π h ∈B H(cid:15)
( π ) V π h ( s ) , ∀ s ∈ S .
Recall that π h is the perturbed policy caused by adversary h , i.e., π h ( ·| s ) =
π ( ·| h ( s )) , ∀ s ∈ S .
Deﬁnition 3 implies an equivalent relationship between the optimal state adversary and the optimal policy adversary: an optimal state adversary leads to an optimal policy adversary, and any state adversary that leads to an optimal policy adversary is optimal.
Theorem 19 in Appendix D.1 shows that there always exists an optimal policy adversary for a ﬁxed victim π , and learning the optimal policy adversary is an RL problem.
(A similar result have been shown by Zhang et al.
(2020a) for the optimal state adversary, while we focus on the policy perturbation.)
Due to the equivalence, if one ﬁnds an optimal policy adversary π h ∗ , then the optimal state adversary can be found by executing targeted attacks with target policy π h ∗ .
However, directly ﬁnding the optimal policy adversary in the Adv-policy-set B H(cid:15)
( π ) is challenging since B H(cid:15)
( π ) is generated by all admissible state adversaries in H (cid:15) and is hard to compute.
To address this challenge, we ﬁrst get insights from theoretical characterizations of the Adv-policy-set B H(cid:15)
( π ) .
Theorem 4 below shows that the “outermost boundary” of B H(cid:15) ( π ) always contains an optimal policy adversary.
Intuitively, a policy π (cid:48) is in the outermost boundary of B H(cid:15)
( π ) if and only if no policy in B H(cid:15) ( π ) is farer away from π than π (cid:48) in the direction π (cid:48)
− π .
Therefore, if an adversary can perturb a policy along a direction, it should push the policy as far away as possible in this direction under the budget constraints.
Then, the adversary is guaranteed to ﬁnd an optimal policy adversary after trying all the perturbing directions.
In contrast, such a guarantee does not exist for state adversaries, justifying the beneﬁts of considering policy adversaries.
Our proposed algorithm in Section 4 applies this idea to ﬁnd the optimal attack: an RL-based director searches for the optimal perturbing direction, and an actor is responsible for pushing the policy to the outermost boundary of B H(cid:15)
( π ) with a given direction.
Theorem 4.
For an MDP M , a ﬁxed policy π , and an admissible adversary set H (cid:15) , deﬁne the outermost boundary of the admissible adversarial policy set B H(cid:15)
( π ) w.r.t π as
[Equation]
Then there exists a policy ˜ π ∈ ∂ π B H(cid:15) ( π ) , such that ˜ π is the optimal policy adversary w.r.t.
π .
Theorem 4 is proven in Appendix B.3, and we visualize the outermost boundary in Appendix B.5.
PA-AD: O PTIMAL AND E FFICIENT E VASION A TTACK
In this section, we ﬁrst formally deﬁne the optimality of an attack algorithm and discuss some existing attack methods.
Then, based on the theoretical insights in Section 3, we introduce our algorithm,
Policy Adversarial Actor Director (PA-AD) that has an optimal formulation and is efﬁcient to use.
Although many attack methods for RL agents have been proposed (Huang et al., 2017; Pattanaik et al., 2018; Zhang et al., 2020a), it is not yet well-understood how to characterize the strength and the optimality of an attack method.
Therefore, we propose to formulate the optimality of an attack algorithm, which answers the question “whether the attack objective ﬁnds the strongest adversary”.
Deﬁnition 5 (Optimal Formulation of Attacking Algorithm) .
An attacking algorithm Algo is said to have an optimal formulation iff for any MDP M , policy π and admissible adversary set H (cid:15) under attacking budget (cid:15) , the set of optimal solutions to its objective, H Algo (cid:15) , is a subset of the optimal adversaries against π , i.e., H Algo (cid:15)
[Equation]
Many heuristic-based attacks, although are empirically effective and efﬁcient, do not meet the requirements of optimal formulation.
In Appendix D.3, we categorize existing heuristic attack methods into four types, and theoretically prove that there exist scenarios where these heuristic methods may not ﬁnd the strongest adversary.
A recent paper (Zhang et al., 2021) proposes to learn the optimal state adversary using RL methods, which we will refer to as SA-RL in our paper for simplicity.
SA-RL can be viewed as an “end-to-end” RL attacker, as it directly learns the optimal state adversary such that the value of the victim policy is minimized.
The formulation of SA-RL satisﬁes Deﬁnition 5 and thus is optimal.
However, SA-RL learns an MDP whose state space and action space are both the same as the original state space.
If the original state space is high-dimensional (e.g. images), learning a good policy in the adversary’s
MDP may become computationally intractable, as empirically shown in Section 6.
Can we address the optimal attacking problem in an efﬁcient manner?
SA-RL treats the victim and the environment together as a black box and directly learns a state adversary. But if the victim policy is known to the attacker (e.g. in adversarial training), we can exploit the victim model and simplify the attacking problem while maintaining the optimality.
Therefore, we propose a novel algorithm, Policy Adversarial Actor Director (PA-AD) , that has optimal formulation and is generally more efﬁcient than SA-RL.
PA-AD decouples the whole attacking process into two simpler components: policy perturbation and state perturbation, solved by a “director” and an “actor” through collaboration.
The director learns the optimal policy perturbing direction with RL methods, while the actor crafts adversarial states at every step such that the victim policy is perturbed towards the given direction.
Compared to the black-box SA-RL, PA-AD is a white-box attack, but works for a broader range of environments more efﬁciently.
Note that PA-AD can be used to conduct black-box attack based on the transferability of adversarial attacks (Huang et al., 2017), although it is out of the scope of this paper. Appendix F.2 provides a comprehensive comparison between PA-AD and SA-RL in terms of complexity, optimality, assumptions and applicable scenarios.
[Figure]
An end-to-end RL attacker directly learns a policy to generate state perturbations, but is inefﬁcient in large-state- environments.
In contrast, our PA-AD solves the attack problem with a combination of an RL-based director and a non-RL actor, so that PA-AD achieves both optimality and efﬁciency.
Formally, for a given victim policy π , our proposed PA-AD algorithm solves a Policy Adversary
MDP (PAMDP) deﬁned in Deﬁnition 6.
An actor denoted by g is embedded in the dynamics of the
PAMDP, and a director searches for an optimal policy ν ∗ in the PAMDP.
Deﬁnition 6 ( Policy Adversary MDP (PAMDP) (cid:99) M ) .
Given an MDP M = (cid:104)S , A , P, R, γ (cid:105) , a ﬁxed stochastic victim policy π , an attack budget (cid:15) ≥ 0 , we deﬁne a Policy Adversarial MDP (cid:99) M
= (cid:104)S , (cid:98) A , (cid:98) P, (cid:98) R, γ (cid:105) , where the action space is (cid:98) A :=
{ d ∈ [0 , 1] |A| , (cid:80) |A|
i =1 d i = 0 } , and ∀ s, s (cid:48) ∈ S , ∀ (cid:98) a ∈ (cid:98) A , (cid:98) P ( s (cid:48) | s, (cid:98) a ) =
(cid:88) a ∈A π ( a | g ( (cid:98) a, s )) P ( s (cid:48) | s, a ) , (cid:98) R ( s, (cid:98) a )
= − (cid:88) a ∈A π ( a | g ( (cid:98) a, s ))
R ( s, a ) , where g is the actor function deﬁned as
[Equation]
= argmax
[Equation]
(cid:98) a (cid:107) .
[Equation]
G )
If the victim policy is deterministic , i.e., π D := argmax a π ( a | s ) , (subscript D stands for deterministic), the action space of PAMDP is (cid:98)
A D := A , and the actor function g D is
[Equation]
Detailed deﬁnition of the deterministic-victim version of PAMDP is in Appendix C.1.
A key to PA-AD is the director-actor collaboration mechanism.
The input to director policy ν is the current state s in the original environment, while its output (cid:98) a is a signal to the actor denoting “which direction to perturb the victim policy into”.
(cid:98) A is designed to contain all “perturbing directions” in the policy space.
That is, ∀ (cid:98) a ∈ (cid:98) A , there exists a constant θ 0 ≥ 0 such that ∀ θ ≤ θ 0 , π ( ·| s ) + θ (cid:98) a (cid:107) (cid:98) a (cid:107) belongs to the simplex ∆( A ) .
The actor g takes in the state s and director’s direction (cid:98) a and then computes a state perturbation within the attack budget.
Therefore, the director and the actor together induce a state adversary: h ( s ) :
=
g ( ν ( s ) , s ) , ∀ s ∈ S .
The deﬁnition of PAMDP is slightly different for a stochastic victim policy and a deterministic victim policy, as described below.
For a stochastic victim π , the director’s action (cid:98) a ∈ (cid:98) A is designed to be a unit vector lying in the policy simplex, denoting the perturbing direction in the policy space.
The actor, once receiving the perturbing direction (cid:98) a , will “push” the policy as far as possible by perturbing s to g ( (cid:98) a, s ) ∈ B (cid:15) ( s ) , as characterized by the optimization problem ( G ).
In this way, the policy perturbation resulted by the director and the actor is always in the outermost boundary of B H(cid:15)
( π ) w.r.t.
the victim π , where the optimal policy perturbation can be found according to Theorem 4.
For a deterministic victim π D , the director’s action (cid:98) a ∈ (cid:98) A D can be viewed as a target action in the original action space, and the actor conducts targeted attacks to let the victim execute (cid:98) a , by forcing the logit corresponding to the target action to be larger than the logits of other actions.
In both the stochastic-victim and deterministic-victim case, PA-AD has an optimal formulation as stated in Theorem 7 (proven in Appendix D.2).
Theorem 7 ( Optimality of PA-AD ) .
For any MDP M , any ﬁxed victim policy π , and any attack budget (cid:15) ≥ 0 , an optimal policy ν ∗ in (cid:99) M induces an optimal state adversary against π in M .
That is, the formulation of PA-AD is optimal, i.e., H PA-AD ⊆
H ∗ (cid:15) .
Efﬁciency of PA-AD
As commonly known, the sample complexity and computational cost of learning an MDP usually grow with the cardinalities of its state space and action space.
Both SARL and PA-AD have state space S , the state space of the original MDP.
But the action space of
SA-RL is also S , while our PA-AD has action space R |A| for stochastic victim policies, or A for deterministic victim policies.
In most DRL applications, the state space (e.g., images) is much larger than the action space, then PA-AD is generally more efﬁcient than SA-RL as it learns a smaller MDP
[Table]
[Table]
The attacking procedure is illustrated in Algorithm 1.
At step t , the director observes a state s t , and proposes a policy perturbation (cid:98) a t , then the actor searches for a state perturbation to meet the policy perturbation.
Afterwards, the victim acts with the perturbed state ˜ s t , then the director updates its policy based on the opposite value of the victim’s reward.
Note that the actor solves a constrained optimization problem, ( G D ) or ( G ).
Problem ( G D ) is similar to a targeted attack in supervised learning, while the stochastic version ( G ) can be approximately solved with a Lagrangian relaxation.
In Appendix C.2, we provide our implementation details for solving the actor’s optimization, which empirically achieves state-of-the-art attack performance as veriﬁed in Section 6.
Extending to Continuous Action Space
Our PA-AD can be extended to environments with continuous action spaces, where the actor minimizes the distance between the policy action and the target action, i.e., argmax s (cid:48) ∈ B (cid:15) ( s ) (cid:107) π ( s (cid:48) ) − (cid:98) a (cid:107) .
More details and formal deﬁnitions of the variant of
PA-AD in continuous action space are provided in Appendix C.3.
In Section 6, we show experimental results in MuJoCo tasks, which have continuous action spaces.
R ELATED W ORK Heuristic-based Evasion Attacks on States
There are many works considering evasion attacks on the state observations in RL.
Huang et al.
ﬁrst propose to use FGSM (Goodfellow et al., 2015) to craft adversarial states such that the probability that the agent selects the “best” action is minimized.
The same objective is also used in a recent work by Korkmaz (2020), which adopts a
Nesterov momentum-based optimization method to further improve the attack performance.
Pattanaik et al.
(2018) propose to lead the agent to select the “worst” action based on the victim’s Q function and use gradient descent to craft state perturbations.
Zhang et al.
(2020a) deﬁne the concept of a state-adversarial MDP (SAMDP) and propose two attack methods: Robust SARSA and Maximal Action Difference.
The above heuristic-based methods are shown to be effective in many environments, although might not ﬁnd the optimal adversaries, as proven in Appendix D.3.
RL-based Evasion Attacks on States
As discussed in Section 4, SA-RL (Zhang et al., 2021) uses an end-to-end RL formulation to learn the optimal state adversary, which achieves state-ofthe-art attacking performance in MuJoCo tasks.
For a pixel state space, an end-to-end RL attacker may not work as shown by our experiment in Atari games (Section 6).
Russo & Proutiere (2021) propose to use feature extraction to convert the pixel state space to a small state space and then learn an end-to-end RL attacker.
But such feature extractions require expert knowledge and can be hard to obtain in many real-world applications.
In contrast, our PA-AD works for both pixel and vector state spaces and does not require expert knowledge.
Other Works Related to Adversarial RL
There are many other papers studying adversarial RL from different perspectives, including limited-steps attacking (Lin et al., 2017; Kos & Song, 2017), multi-agent scenarios (Gleave et al., 2020), limited access to data (Inkawhich et al., 2020), and etc.
Adversarial action attacks (Xiao et al., 2019; Tan et al., 2020; Tessler et al., 2019; Lee et al., 2021) are developed separately from state attacks; although we mainly consider state adversaries, our PA-AD can be extended to action attacks as formulated in Appendix A. Poisoning (Behzadan & Munir, 2017; Huang & Zhu, 2019; Sun et al., 2021; Zhang et al., 2020b; Rakhsha et al., 2020) is another type of adversarial attacks that manipulates the training data, different from evasion attacks that deprave a well-trained policy.
Training a robust agent is the focus of many recent works (Pinto et al., 2017; Fischer et al., 2019; L¨utjens et al., 2020; Oikarinen et al., 2020; Zhang et al., 2020a; 2021).
Although our main goal is to ﬁnd a strong attacker, we also show by experiments that our proposed attack method signiﬁcantly improves the robustness of RL agents by adversarial training.
E XPERIMENTS
In this section, we show that PA-AD produces stronger evasion attacks than state-of-the-art attack algorithms on various OpenAI Gym environments, including Atari and MuJoCo tasks.
Also, our experiment justiﬁes that PA-AD can evaluate and improve the robustness of RL agents.
Baselines and Performance Metric
We compare our proposed attack algorithm with existing evasion attack methods, including MinBest (Huang et al., 2017) which minimizes the probability that the agent chooses the “best” action, MinBest +Momentum (Korkmaz, 2020) which uses Nesterov momentum to improve the performance of MinBest, MinQ (Pattanaik et al., 2018) which leads
[Table]
the agent to select actions with the lowest action values based on the agent’s Q network, Robust
SARSA (RS) (Zhang et al., 2020a) which performs the MinQ attack with a learned stable Q network,
MaxDiff (Zhang et al., 2020a) which maximizes the KL-divergence between the original victim policy and the perturbed policy, as well as SA-RL (Zhang et al., 2021) which directly learns the state adversary with RL methods.
We consider state attacks with (cid:96) ∞ norm as in most literature (Zhang et al., 2020a; 2021).
Appendix E.1 provides hyperparameter settings and implementation details.
PA-AD Finds the Strongest Adversaries in Atari Games
We ﬁrst evaluate the performance of
PA-AD against well-trained DQN (Mnih et al., 2015) and A2C (Mnih et al., 2016) victim agents on Atari games with pixel state spaces.
The observed pixel values are normalized to the range of [0 , 1] .
SA-RL and PA-AD adversaries are learned using the ACKTR algorithm (Wu et al., 2017) with the same number of steps.
(Appendix E.1 shows hyperparameter settings.)
Table 1 presents the experiment results, where PA-AD signiﬁcantly outperforms all baselines against both DQN and A2C victims.
In contrast, SA-RL does not converge to a good adversary in the tested Atari games with the same number of training steps as PA-AD, implying the importance of sample efﬁciency.
Surprisingly, using a relatively small attack budget (cid:15) , PA-AD leads the agent to the lowest possible reward in many environments such as Pong, RoadRunner and Tutankham, whereas other attackers may require larger attack budget to achieve the same attack strength.
Therefore, we point out that vanilla RL agents are extremely vulnerable to carefully learned adversarial attacks.
Even if an RL agent works well under naive attacks, a carefully learned adversary can let an agent totally fail with the same attack budget, which stresses the importance of evaluating and improving the robustness of RL agents using the strongest adversaries.
Our further investigation in Appendix F.3 shows that RL models can be generally more vulnerable than supervised classiﬁers, due to the different loss and architecture designs.
In Appendix E.2.1, we show more experiments with various selections of the budget (cid:15) , where one can see PA-AD reduces the average reward more than all baselines over varying (cid:15) ’s in various environments.
PA-AD Finds the Strongest Adversaries MuJoCo Tasks
We further evaluate PA-AD on MuJoCo games, where both state spaces and action spaces are continuous.
We use the same setting with
Zhang et al.
(2021), where both the victim and the adversary are trained with PPO (Schulman et al., 2017).
During test time, the victim executes a deterministic policy, and we use the deterministic
[Table]
version of PA-AD with a continuous action space, as discussed in Section 4 and Appendix C.3.
We use the same attack budget (cid:15) as in Zhang et al.
(2021) for all MuJoCo environments.
Results in Table 2 show that PA-AD reduces the reward much more than heuristic methods, and also outperforms SA-RL in most cases.
In Ant, our PA-AD achieves much stronger attacks than SA-RL, since PA-AD is more efﬁcient than SA-RL when the state space is large.
Admittedly, PA-AD requires additional knowledge of the victim model, while SA-RL works in a black-box setting. Therefore, SA-RL is more applicable to black-box scenarios with a relatively small state space, whereas PA-AD is more applicable when the attacker has access to the victim (e.g. in adversarial training as shown in Table 3).
Appendix E.2.3 provides more empirical comparison between SA-RL and PA-AD, which shows that PA-AD converges faster, takes less running time, and is less sensitive to hyperparameters than SA-RL by a proper exploitation of the victim model.
[Table]
Training and Evaluating Robust Agents
A natural application of PA-AD is to evaluate the robustness of a known model, or to improve the robustness of an agent via adversarial training, where the attacker has white-box access to the victim.
Inspired by ATLA (Zhang et al., 2021) which alternately trains an agent and an SA-RL attacker, we propose PA-ATLA, which alternately trains an agent and a PA-AD attacker.
In Table 3, we evaluate the performance of PA-ATLA for a PPO agent (namely PA-ATLA-PPO) in MuJoCo tasks, compared with state-of-the-art robust training methods, SA-PPO (Zhang et al., 2020a) and ATLA-PPO (Zhang et al., 2021) 2 .
From the table, we make the following observations. (1) Our PA-AD attacker can signiﬁcantly reduce the reward of previous “robust” agents.
Take the Ant environment as an example, although SA-PPO and ATLA-PPO agents gain 2k+ and 3k+ rewards respectively under SA-RL, the previously strongest attack, our PAAD still reduces their rewards to about -1.3k and 200+ with the same attack budget.
Therefore, we emphasize the importance of understanding the worst-case performance of RL agents, even robustlytrained agents.
(2) Our PA-ATLA-PPO robust agents gain noticeably higher average rewards across attacks than other robust agents , especially under the strongest PA-AD attack.
Under the SA-RL attack, PA-ATLA-PPO achieves comparable performance with ATLA-PPO, although ATLA-PPO agents are trained to be robust against SA-RL.
Due to the efﬁciency of PA-AD, PA-ATLA-PPO requires fewer training steps than ATLA-PPO, as justiﬁed in Appendix E.2.4.
The results of attacking and training robust models in Atari games are in Appendix E.2.5 and E.2.6, where PA-ATLA improves the robustness of Atari agents against strong attacks with (cid:15) as large as 3 / 255 .
C ONCLUSION
In this paper, we propose an attack algorithm called PA-AD for RL problems, which achieves optimal attacks in theory and signiﬁcantly outperforms prior attack methods in experiments.
PA-AD can be used to evaluate and improve the robustness of RL agents before deployment.
A potential future direction is to use our formulation for robustifying agents under both state and action attacks.
E THICS S TATEMENT
Despite the rapid advancement of interactive AI and ML systems using RL agents, the learning agent could fail catastrophically in the presence of adversarial attacks, exposing a serious vulnerability in current RL systems such as autonomous driving systems, market-making systems, and security monitoring systems.
Therefore, there is an urgent need to understand the vulnerability of an RL model, otherwise, it may be risky to deploy a trained agent in real-life applications, where the observations of a sensor usually contain unavoidable noise.
Although the study of a strong attack method may be maliciously exploited to attack some RL systems, it is more important for the owners and users of RL systems to get aware of the vulnerability of their RL agents under the strongest possible adversary.
As the old saying goes, “if you know yourself and your enemy, you’ll never lose a battle”.
In this work, we propose an optimal and efﬁcient algorithm for evasion attacks in Deep RL (DRL), which can signiﬁcantly inﬂuence the performance of a well-trained DRL agent, by adding small perturbations to the state observations of the agent.
Our proposed method can automatically measure the vulnerability of an RL agent, and discover the “ﬂaw” in a model that might be maliciously attacked.
We also show in experiments that our attack method can be applied to improve the robustness of an RL agent via robust training.
Since our proposed attack method achieves state-of-the-art performance, the RL agent trained under our proposed attacker could be able to “defend” against any other adversarial attacks with the same constraints.
Therefore, our work has the potential to help combat the threat to high-stakes systems.
A limitation of PA-AD is that it requires the “attacker” to know the victim’s policy, i.e., PA-AD is a white-box attack.
If the attacker does not have full access to the victim, PA-AD can still be used based on the transferability of adversarial attacks (Huang et al., 2017), although the optimality guarantee does not hold in this case.
However, this limitation only restricts the ability of the malicious attackers.
In contrast, PA-AD should be used when one wants to evaluate the worst-case performance of one’s own RL agent, or to improve the robustness of an agent under any attacks, since PA-AD produces strong attacks efﬁciently.
In these cases, PA-AD does have white-box access to the agent.
Therefore, PA-AD is more beneﬁcial to defenders than attackers.
R EPRODUCIBILITY S TATEMENT
For theoretical results , we provide all detailed technical proofs and lemmas in Appendix.
In Appendix A, we analyze the equivalence between evasion attacks and policy perturbations.
In Appendix B, we theoretically prove some topological properties of the proposed Adv-policy-set, and derive Theorem 4 that the outermost boundary of B H(cid:15) ( π ) always contains an optimal policy perturbation.
In Appendix D, we systematically characterize the optimality of many existing attack methods.
We theoretically show (1) the existence of an optimal adversary, (2) the optimality of our proposed PA-AD, and (3) the optimality of many heuristic attacks, following our Deﬁnition 5 in
Section 4.
For experimental results , the detailed algorithm description in various types of environments is provided in Appendix C. In Appendix E, we illustrate the implementation details, environment settings, hyperparameter settings of our experiments.
Additional experimental results show the performance of our algorithm from multiple aspects, including hyperparameter sensitivity, learning efﬁciency, etc.
In addition, in Appendix F, we provide some detailed discussion on the algorithm design, as well as a comprehensive comparison between our method and prior works.
The source code and running instructions for both Atari and MuJoCo experiments are in our supplementary materials.
We also provide trained victim and attacker models so that one can directly test their performance using a test script we provide.
Appendix: Who
Is
the
Strongest Enemy? Towards Optimal and Efﬁcient Evasion Attacks in Deep RL
A R ELATIONSHIP BETWEEN E VASION A TTACKS AND P OLICY P ERTURBATIONS .
As mentioned in Section 2, all evasion attacks can be regarded as perturbations in the policy space.
To be more speciﬁc, we consider the following 3 cases, where we assume the victim uses policy π .
Case 1 (attack on states) : deﬁne the state adversary as function h such that ∀ s ∈ S
[Equation]
(For simplicity, we consider the attacks within a (cid:15) -radius norm ball.)
In this case, for all s ∈ S , the victim samples action from π h ( ·| s ) =
π ( ·| h ( s ))
=
π (˜ s ) , which is equivalent to the victim executing a perturbed policy π h ∈ Π .
Case 2 (attack on actions for a deterministic π ) : deﬁne the action adversary as function h
( A ) :
S ×
A → A , and ∀ s ∈ S , a ∈ A
[Equation]
(cid:107) ≤ (cid:15) } .
In this case, there exists a policy π h ( A ) such that π h ( A ) ( s )
=
h ( A ) ( a | s ) =
˜ a , which is equivalent to the victim executing policy π h
( A ) ∈ Π .
Case 3 (attack on actions for a stochastic π ) : deﬁne the action adversary as function h
( A ) :
S×A →
A , and ∀ s ∈ S , a ∈
A h ( A ) ( a | s ) =
˜ a such that {(cid:107) π ( ·| s ) − Pr ( ·| s ) (cid:107) ≤ (cid:15) } , where Pr (˜ a | s ) denotes the probability that the action is perturbed into ˜ a .
In this case, there exists a policy π h ( A ) such that π h ( A ) ( s ) =
Pr ( ·| s ) , which is equivalent to the victim executing policy π h
( A ) ∈ Π .
Most existing evasion RL works (Huang et al., 2017; Pattanaik et al., 2018; Zhang et al., 2020a; 2021) focus on state attacks, while there are also some works (Tessler et al., 2019; Tan et al., 2020) studying action attacks.
For example, Tessler et al.
(Tessler et al., 2019) consider Case 2 and Caseabove and train an agent that is robust to action perturbations.
These prior works study either state attacks or action attacks, considering them in two different scenarios.
However, the ultimate goal of robust RL is to train an RL agent that is robust to any threat models.
Otherwise, an agent that is robust against state attacks may still be ruined by an action attacker.
We take a step further to this ultimate goal by proposing a framework, policy attack, that uniﬁes observation attacks and action attacks.
Although the focus of this paper is on state attacks, we would like to point out that our proposed method can also deal with action attacks (the director proposes a policy perturbation direction, and an actor perturbs the action accordingly).
It is also an exciting direction to explore hybrid attacks (multiple actors conducting states perturbations and action perturbations altogether, directed by a single director.)
Our policy perturbation framework can also be easily incorporated in robust training procedures, as an agent that is robust to policy perturbations is simultaneously robust to both state attacks and action attacks.
B T OPOLOGICAL P ROPERTIES OF THE A DMISSIBLE A DVERSARIAL P OLICY S ET
As discussed in Section 3, ﬁnding the optimal state adversary in the admissible adversary set H (cid:15) can be converted to a problem of ﬁnding the optimal policy adversary in the Adv-policy-set B H(cid:15)
( π ) .
In this section, we characterize the topological properties of B H(cid:15)
( π ) , and identify how the value function changes as the policy changes within B H(cid:15)
( π ) .
In Section B.1, we show that under the settings we consider, B H(cid:15) ( π ) is a connected and compact subset of Π .
Then, Section B.2, we deﬁne some additional concepts and re-formulate the notations.
In Section B.3, we prove Theorem 4 in Section 3 that the outermost boundary of B H(cid:15) ( π ) always contains an optimal policy perturbation.
In Section B.4, we prove that the value functions of policies in B H(cid:15) ( π ) (or more generally, any connected and compact subset of Π ) form a polytope.
Section B. shows an example of the polytope result with a 2-state MDP, and Section B.5 shows examples of the outermost boundary deﬁned in Theorem 4.
B. T HE S HAPE OF A DV - POLICY - SET B H(cid:15) ( π )
It is important to note that B H(cid:15)
( π ) is generally connected and compact as stated in the following lemma.
Lemma 8 ( B H(cid:15) ( π ) is connected and compact ) .
Given an MDP M , a policy π that is a continuous mapping, and admissible adversary set H (cid:15) :
= { h : h ( s ) ∈ B (cid:15) ( s ) , ∀ s ∈ S} (where (cid:15) > 0 is a constant), the admissible adversarial policy set B H(cid:15)
( π ) is a connected and compact subset of Π .
Proof of Lemma 8.
For an arbitrary state s ∈ S , an admissible adversary h ∈ H (cid:15) perturbs it within an (cid:96) p norm ball B (cid:15) ( s ) , which is connected and compact.
Since π is a continuous mapping, we know π ( s ) is compact and connected.
Therefore, B H(cid:15) ( π ) as a Cartesian product of a ﬁnite number of compact and connected sets, is compact and connected.
B. A DDITIONAL N OTATIONS AND D EFINITIONS FOR P ROOFS
We ﬁrst formally deﬁne some concepts and notations.
For a stationary and stochastic policy π : S → ∆( A ) , we can deﬁne the state-to-state transition function as
[Equation]
and the state reward function as
[Equation]
Then the value of π , denoted as V π , can be computed via the Bellman equation
[Equation]
We further use Π s i to denote the projection of Π into the simplex of the i -th state, i.e., the space of action distributions at state s i .
Let f v : Π → R |S| be a mapping that maps policies to their corresponding value functions.
Let
V = f v (Π) be the space of all value functions.
Dadashi et al.
(Dadashi et al., 2019) show that the image of f v applied to the space of policies, i.e., f v (Π) , form a (possibly non-convex) polytope as deﬁned below.
Deﬁnition 9 ( (Possibly non-convex) polytope ) .
A is called a convex polytope iff there are k ∈ N points x 1 , x 2 , · · · , x k ∈ R n such that A =
Conv ( x 1 , · · · , x k ) .
Furthermore, a (possibly nonconvex) polytope is deﬁned as a ﬁnite union of convex polytopes.
And a more general concept is (possibly non-convex) polyhedron, which might not be bounded.
Deﬁnition 10 ( (Possibly non-convex) polyhedron ) .
A is called a convex polyhedron iff it is the intersection of k ∈ N half-spaces ˆ B 1 , ˆ B 2 , · · · , ˆ B k , i.e., A = ∩ ki =1 ˆ B i .
Furthermore, a (possibly non-convex) polyhedron is deﬁned as a ﬁnite union of convex polyhedra.
In addition, let Y πs 1 , ··· ,s k be the set of policies that agree with π on states s 1 , · · · , s k .
Dadashi et al.
(Dadashi et al., 2019) also prove that the values of policies that agree on all but one state s , i.e., f v ( Y π S\{ s } ) , form a line segment, which can be bracketed by two policies that are deterministic on s .
Our Lemma 14 extends this line segment result to our setting where policies are restricted in a subset of policies.
B. P ROOF OF T HEOREM 4: B OUNDARY C ONTAINS O PTIMAL P OLICY P ERTURBATIONS
Lemma 4 in Dadashi et al.
(2019) shows that policies agreeing on all but one state have certain monotone relations.
We restate this result in Lemma 11 below.
Lemma 11 (Monotone Policy Interpolation) .
For any π 0 , π 1 ∈ Y π S\{ s } that agree with π on all states except for s ∈ S , deﬁne a function l : [0 , 1] → V as l ( α ) =
f v ( απ 1 + (1 − α ) π 0 ) .
Then we have
(1) l (0) (cid:60) l (1) or l (1) (cid:60) l (0) ( (cid:60) stands for element-wise greater than or equal to); (2)
If l (0) = l (1) , then l ( α ) =
l (0) , ∀ α ∈ [0 , 1] ; (3)
If l (0) (cid:54) =
l (1) , then there is a strictly monotonic rational function ρ : [0 , 1] → R , such that l ( α )
=
ρ ( α ) l (1) + (1 − ρ ( α )) l (0) .
More intuitively, Lemma 11 suggests that the value of π α :
= απ 1 + (1 − α )
π 0 changes (strictly) monotonically with α , unless the values of π 0 , π 1 and π α are all equal.
With this result, we can proceed to prove Theorem 4.
Proof of Theorem 4.
We will prove the theorem by contradiction.
Suppose there is a policy ˆ π ∈ B H(cid:15) ( π ) such that ˆ π / ∈ ∂ π B H(cid:15) ( π ) and f v (ˆ π ) =
V ˆ π < V ˜ π , ∀ ˜ π ∈ B H(cid:15) ( π ) , i.e., there is no optimal policy adversary on the outermost boundary of B H(cid:15)
( π ) .
Then according to the deﬁnition of ∂ π B H(cid:15) ( π ) , there exists at least one state s ∈ S such that we can ﬁnd another policy π (cid:48)
∈ B H(cid:15) ( π ) agreeing with ˆ π on all states except for s , where π (cid:48)
( s ) satisﬁes
[Equation]
for some scalar α ∈ (0 , 1) .
Then by Lemma 11, either of the following happens:
(1) f v ( π ) (cid:31) f v (ˆ π ) (cid:31) f v ( π (cid:48) ) .
(2) f v ( π ) =
f v (ˆ π ) =
f v ( π (cid:48) ) ;
Note that f v (ˆ π ) (cid:31) f v ( π ) is impossible because we have assumed ˆ π has the lowest value over all policies in B H(cid:15)
( π ) including π .
If (1) is true, then π (cid:48) is a better policy adversary than ˆ π in B H(cid:15)
( π ) , which contradicts with the assumption.
If (2) is true, then π (cid:48) is another optimal policy adversary.
By recursively applying the above process to π (cid:48) , we can ﬁnally ﬁnd an optimal policy adversary on the outermost boundary of B H(cid:15) ( π ) , which also contradicts with our assumption.
In summary, there is always an optimal policy adversary lying on the outermost boundary of B H(cid:15)
( π ) .
B. P ROOF OF T HEOREM 12: V ALUES OF P OLICIES IN A DMISSIBLE A DVERSARIAL P OLICY S ET F ORM A P OLYTOPE
We ﬁrst present a theorem that describes the “shape” of the value functions generated by all admissible adversaries (admissible adversarial policies).
Theorem 12 ( Policy Perturbation Polytope ) .
For a ﬁnite MDP M , consider a policy π and an
Adv-policy-set B H(cid:15)
( π ) .
The space of values (a subspace of R |S| ) of all policies in B H(cid:15) ( π ) , denoted by V B H(cid:15) ( π ) , is a (possibly non-convex) polytope.
In the remaining of this section, we prove a more general version of Theorem 12 as below.
Theorem 13 ( Policy Subset Polytope ) .
For a ﬁnite MDP M , consider a connected and compact subset of Π , denoted as T .
The space of values (a subspace of R |S| ) of all policies in T , denoted by
V T , is a (possibly non-convex) polytope.
According to Lemma 8, B H(cid:15) ( π ) is a connected and compact subset of Π , thus Theorem 12 is a special case of Theorem 13.
Additional Notations To prove Theorem 13, we further deﬁne a variant of Y πs 1 , ··· ,s k as T πs 1 , ··· ,s k , which is the set of policies that are in T and agree with π on states s 1 , · · · , s k , i.e.,
[Equation]
Note that different from B H(cid:15) ( π ) , T is no longer restricted under an admissible adversary set and can be any connected and compact subset of Π .
The following lemma shows that the values of policies in T that agree on all but one state form a line segment.
Lemma 14.
For a policy π ∈ T and an arbitrary state s ∈ S , there are two policies in ∂ π T π S\{ s } , namely π − s , π + s , such that ∀ π (cid:48)
∈ T π S\{ s } ,
[Equation]
where (cid:52) denotes element-wise less than or equal to (if a (cid:52) b , then a i ≤ b i for all index i ).
Moreover, the image of f v restricted to T π S\{ s } is a line segment.
Proof of Lemma 14.
Lemma 5 in Dadashi et al.
(2019) has shown that f v is inﬁnitely differentiable on Π , hence we know f v ( T π S\{ s } ) is compact and connected.
According to Lemma 4 in Dadashi et al.
(2019), for any two policies
π 1 , π 2 ∈ Y π S\{ s } , either f v ( π 1 ) (cid:52) f v ( π 2 ) , or f v ( π 2 ) (cid:52) f v ( π 1 ) (there exists a total order).
The same property applies to T π S\{ s } since T π S\{ s } is a subset of Y π S\{ s } .
Therefore, there exists π − s and π + s that achieve the minimum and maximum over all policies in
T π S\{ s } .
Next we show π − s and π + s can be found on the outermost boundary of T π S\{ s } .
Assume π + s / ∈ ∂
π T π S\{ s } , and for all ˜ π ∈ T π S\{ s } , f v (˜ π ) ≺ f v ( π + s ) .
Then we can ﬁnd another policy π (cid:48) ∈
∂ π T π S\{ s } such that π + s = απ + (1 − α ) π (cid:48) for some scalar α ∈ (0 , 1) .
Then according to
Lemma 11, f v ( π (cid:48) )
(cid:60) f v ( π + s ) , contradicting with the assumption.
Therefore, one should be able to ﬁnd a policy on the outermost boundary of T π S\{ s } whose value dominates all other policies.
And similarly, we can also ﬁnd π − s on ∂ π T π S\{ s } .
Furthermore, f v ( T π S\{ s } ) is a subset of f v ( Y π S\{ s } ) since T π S\{ s } is a subset of Y π S\{ s } .
Given that f v ( Y π S\{ s } ) is a line segment, and f v ( T π S\{ s } ) is connected, we can conclude that f v ( T π S\{ s } ) is also a line segment.
Next, the following lemma shows that π + s and π − s and their linear combinations can generate values that cover the set f v ( T π S\{ s } ) .
Lemma 15.
For a policy π ∈ T , an arbitrary state s ∈ S , and π + s , π − s deﬁned in Lemma 14, the following three sets are equivalent:
(1) f v ( T π S\{ s } ) ; (2) f v (cid:0) closure ( T π S\{ s } ) (cid:1) , where closure ( · ) is the convex closure of a set; (3) { f v ( απ + s + (1 − α ) π − s ) | α ∈ [0 , 1] } ; (4) { αf v ( π + s ) + (1 − α )
f v ( π − s ) | α ∈ [0 , 1] } ;
Proof of Lemma 15.
We show the equivalence by showing (1) ⊆
(4) ⊆
(3) ⊆ (2) ⊆
(1) as below.
(2) ⊆ (1): For any π 1 , π 2 ∈ T π S\{ s } , without loss of generality, suppose f v ( π 1 ) (cid:52) f v ( π 2 ) .
According to Lemma 11, for any α ∈ [0 , 1] , f v ( π 1 ) (cid:52) απ 1 + (1 − α )
π 2 (cid:52) f v ( π 2 ) .
Therefore, any convex combinations of policies in T π S\{ s } has value that is in the range of f v ( T π S\{ s } ) .
So the values of policies in the convex closure of T π S\{ s } do not exceed f v ( T π S\{ s } ) , i.e., (2) ⊆
(3) ⊆ (2): Based on the deﬁnition, απ + s + (1 − α )
π − s ∈ closure ( T π S\{ s } ) , so (3) ⊆ (2).
(4) ⊆ (3):
According to Lemma 11, there exists a strictly monotonic rational function ρ : [0 , 1] → R , such that
[Equation]
Therefore, due to intermediate value theorem, for α ∈ [0 , 1] , ρ ( α ) takes all values from 0 to 1.
So (4) =
(1) ⊆ (4): Lemma 14 shows that f v ( T π S\{ s } ) is a line segment bracketed by f v ( π + s ) and f v ( π − s ) .
Therefore, for any π (cid:48) ∈ T π S\{ s } , its value is a convex combination of f v ( π + s ) and f v ( π − s ) .
Next, we show that the relative boundary of the value space constrained to T πs 1 , ··· ,s k is covered by policies that dominate or are dominated in at least one state.
The relative interior of set A in B is deﬁned as the set of points in A that have a relative neighborhood in A ∩ B , denoted as relint B A .
The relative boundary of set A in B , denoted as ∂ B A , is deﬁned as the set of points in A that are not in the relative interior of A , i.e., ∂ B A =
A \ relint B A .
When there is no ambiguity, we omit the subscript of ∂ to simplify notations.
In addition, we introduce another notation F πs 1 , ··· ,s k :=
V π + span
( C πk +1 , · · · , C π |S| ) , where C πi stands for the i -th column of the matrix ( I − γP π ) − 1 .
Note that F πs 1 , ··· ,s k is the same with H πs 1 , ··· ,s k in Dadashi et al.
Dadashi et al.
and we change H to F in order to distinguish from the admissible adversary set H (cid:15) deﬁned in our paper.
Lemma 16.
For a policy π ∈ T , k ≤ |S| , and a set of policies T πs 1 , ··· ,s k that agree with π on s 1 , · · · , s k (perturb π only at s k +1 , · · · , s |S| ), deﬁne V t := f v ( T πs 1 , ··· ,s k ) .
Deﬁne two sets of policies
X
[Equation] [Equation]
{ π (cid:48)
[Equation]
T πs 1 , ··· ,s k : π (cid:48) ( ·| s ) =
π
+ s ( ·| s ) } , and X − s :=
{ π (cid:48) ∈ T πs 1 , ··· ,s k : π (cid:48) ( ·| s ) =
π − s ( ·| s ) } .
We have that the relative boundary of V t in F πs 1 , ··· ,s k is included in the value functions spanned by policies in T πs 1 , ··· ,s k ∩ ( X + s j ∪ X − s j ) for at least one s / ∈ { s 1 , · · · , s k } , i.e., ∂ V t ⊂
[Equation]
Proof of Lemma 16.
We ﬁrst prove the following claim:
Claim 1: For a policy π 0 ∈ T πs 1 , ··· ,s k , if ∀ j ∈ { k + 1 , · · · , |S|} , (cid:64) π (cid:48) ∈ closure ( T πs 1 , ··· ,s k ) ∩ ( X + s j ∪
X − s j ) such that f v ( π (cid:48) )
=
f v ( π 0 ) , then f v ( π 0 ) has a relative neighborhood in V t ∩ F πs 1 , ··· ,s k .
First, based on Lemma 14 and Lemma 15, we can construct a policy ˆ π ∈ closure ( T πs 1 , ··· ,s k ) such that f v (ˆ π ) =
f v ( π 0 ) through the following steps:
Algorithm 2: Constructing ˆ π 1 Set π k = π2 for j = k + 1 , · · · , |S| do
Find π + s j , π − s j ∈ T π j − 1 S\{ s j }
Find π j = ˆ α j π
+ s j + (1 − ˆ α j ) π − s j such that f v ( π j ) =
f v ( π j − 1 )
Return ˆ π = π |S|
Denote the concatenation of α j ’s as a vector ˆ α :=
[ˆ α k +1 , · · · , ˆ α |S| ] .
According to the assumption that ∀ j ∈ { k + 1 , · · · , |S|} , (cid:64) π (cid:48) ∈ closure ( T π s 1 , ··· ,s k ) ∩ ( X + s j ∪ X − s j ) such that f v ( π (cid:48) )
=
f v ( π 0 ) , we have ˆ α j / ∈ { 0 , 1 } , ∀ j = k + 1 , · · · , |S| .
Then, deﬁne a function φ : (0 , 1) |S|− k
→ V t such that φ ( α ) =
f v ( π α ) , where (cid:40)
[Equation] [Equation]
if j ∈ { k + 1 , · · · , |S|} π
[Equation]
( ·| s j )
=
ˆ π ( ·| s j ) otherwise
Then we have that
φ is continuously differentiable.
φ (ˆ α ) =
f v (ˆ π ) .
∂φ∂α j is non-zero at ˆ α (because of Lemma 11 (3)).
∂φ∂α j is along the i -the column of ( I − γP ˆ π ) − 1 (see Lemma 3 in Dadashi et al.
Dadashi et al.
Therefore, by the inverse theorem function, there is a neighborhood of φ ( α ) =
f v (ˆ π ) in the image space.
Now we have proved Claim 1.
As a result, for any policy π 0 ∈ T πs 1 , ··· ,s k , if f v ( π 0 ) is in the relative boundary of V t in F πs 1 , ··· ,s k , then ∃ j ∈ { k + 1 , · · · , |S|} , π (cid:
) ∈ closure ( T πs 1 , ··· ,s k ) ∩ ( X + s j ∪ X − s j ) such that f v ( π (cid:48) )
=
f v ( π 0 ) .
Based on Lemma 15, we can also
ﬁnd π (cid:48)(cid:48) ∈ T πs 1 , ··· ,s k ∩ ( X + s j ∪ X − s j ) such that f v ( π (cid:48)(cid:48) )
=
f v ( π 0 ) .
So Lemma 16 holds.
Now, we are ﬁnally ready to prove Theorem 13.
Proof of Theorem 13.
We will show that ∀{ s 1 , · · · , s k } ⊆ S , the value V t = f v ( T πs 1 , ··· ,s k ) is a polytope.
We prove the above claim by induction on the cardinality of the number of states k .
In the base case where k = |S| , V t = { f v ( π ) } is a polytope.
Suppose the claim holds for k + 1 , then we show it also holds for k , i.e., for a policy π ∈ Π , the value of T πs 1 , ··· ,s k ⊆
Y πs 1 , ··· ,s k ⊆ Π for a polytope.
According to Lemma 16, we have
[Equation]
where ∂ V t denotes the relative boundary of V t in F πs 1 , ··· ,s k ; F + s j and F − s j are two afﬁne hyperplanes of F πs 1 , ··· ,s k , standing for the value space of policies that agree with π + s j and π − s j in state s j respectively.
Then we can get
V t = f v ( T πs 1 , ··· ,s k ) is closed as T πs 1 , ··· ,s k is compact and f v is continuous.
∂ V t ⊂ (cid:83) |S| j = k +
( F + s j ∪ F − s j )) , a ﬁnite number of afﬁne hyperplanes in F πs 1 , ··· ,s k . 3.
V t ∩ F + s j (or V t ∩ F − s j ) is a polyhedron by induction assumption.
Hence, based on Proposition 1 by Dadashi et al.
Dadashi et al.
(2019), we get V t is a polyhedron.
Since V t ⊆
V is bounded, we can further conclude that V t is a polytope.
Therefore, for an arbitrary connected and compact set of policies T ⊆ Π , let π ∈ T be an arbitrary policy in T , then f v ( T ) =
f v ( T π ∅ ) is a polytope.
B. E XAMPLES OF THE O UTERMOST B OUNDARY
See Figure 5 for examples of the outermost boundary for different B H(cid:15) ( π ) ’s.
[Figure]
B. A N E XAMPLE OF T HE P OLICY P ERTURBATION P OLYTOPE
An example is given by Figure 6, where we deﬁne an MDP with 2 states and 3 actions.
We train an DQN agent with one-hot encodings of the states and then randomly perturb the states within an (cid:96) ∞ ball with (cid:15) = 0 . 8 .
By sampling 5M random policies and 100K random perturbations, we visualize the value space of approximately the whole policy space Π and the admissible adversarial policy set B H(cid:15)
( π ) , both of which are polytopes (boundaries are ﬂat).
A learning agent searches for the optimal policy π ∗ whose value is the upper right vertex of the larger blue polytope, while the attacker attempts to ﬁnd an optimal adversary h ∗ , which perturbs a given clean policy π to the worst perturbed policy π h ∗ whose value is the lower left vertex of the smaller green polytope.
This also justiﬁes the fact that learning an optimal adversary is as difﬁcult as learning an optimal policy in an
RL problem.
The example MDP M ex :
[Equation] [Equation]
ˆ r = [ − 0 .
1 , − 1 ., 0 . 1 , 0 . 4 , 1 . 5 , 0 .
ˆ P =
The base/clean policy π : π ( a
[Equation]
|
[Equation] [Equation]
215 , π ( a 2 | s 1 ) = 0 .
429 , π ( a 3 | s 1 )
= 0 .π
( a
[Equation]
| s 2 )
= 0 .
271 , π ( a 2 | s 2 )
= 0 .
592 , π ( a 3 | s 2 )
= 0 .
C E XTENTIONS AND A DDITIONAL D ETAILS OF O UR A LGORITHM C. A TTACKING A D ETERMINISTIC V ICTIM P OLICY
For a deterministic victim π D = argmax a π ( a | s ) , we deﬁne Deterministic Policy Adversary MDP (D-PAMDP) as below, where a subscript D is added to all components to distinguish them from their stochastic counterparts.
In D-PAMDP, the director proposes a target action (cid:98)
a D ∈
A (=: (cid:98) A D ) , and the actor tries its best to let the victim output this target action.
Deﬁnition 17 ( Deterministic Policy Adversary MDP (D-PAMDP) ) .
Given an MDP M = (cid:104)S , A , P, R, γ (cid:105) , a ﬁxed and deterministic victim policy π D , we deﬁne a Deterministic Policy Adversarial MDP (cid:99) M D
=
(cid:104)S , (cid:98)
A D , (cid:98) P D , (cid:98) R D , γ (cid:105) , where the action space is (cid:98)
A D = (cid:98)
A D , and ∀ s, s (cid:48) ∈
S , ∀ (cid:98) a ∈ A ,
[Equation]
P
[Equation]
s
[Equation] [Equation]
The actor function g is deﬁned as
[Equation]
The optimal policy of D-PAMDP is an optimal adversary against π D as proved in Appendix D.2.
C. I MPLEMENTATION D ETAILS OF PA-AD
To address the actor function g (or g D ) deﬁned in ( G ) and ( G D ), we let the actor maximize objectives
J D and J within the B (cid:15) ( · ) ball around the original state, for a deterministic victim and a stochastic victim, respectively.
Below we explicitly deﬁne J D and J .
Actor Objective for Deterministic Victim For the deterministic variant of PA-AD, the actor function ( G D ) is simple and can be directly solved to identify the optimal adversary.
Concretely, we deﬁne the following objective
[Equation]
which can be realized with the multi-class classiﬁcation hinge loss.
In practice, a relaxed crossentropy objective can also be used to maximize π ( (cid:98) a | s (cid:48) ) .
Actor Objective for Stochastic Victim Different from the deterministic-victim case, the actor function for a stochastic victim deﬁned in ( G ) requires solving a more complex optimization problem with a non-convex constraint set, which in practice can be relaxed to ( ˆ J )
(a Lagrangian relaxation) to efﬁciently get an approximation of the optimal adversary.
[Equation]
where CosineSim in the second refers to the cosine similarity function; the ﬁrst term measures how far away the policy is perturbed from the victim policy; λ is a hyper-parameter controlling the tradeoff between the two terms.
Experimental results show that our PA-AD is not sensitive to the value of λ .
In our reported results in Section 6, we set λ as 1.
Appendix E.2.2 shows the evaluation of our algorithm using varying λ ’s.
The procedure of learning the optimal adversary is depicted in Algorithm 3, where we simply use the Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2015) to approximately solve the actor’s objective, although more advanced solvers such as Projected Gradient Decent (PGD) can be applied to further improve the performance.
Experiment results in Section 6 verify that the above FGSMbased implementation achieves state-of-the-art attack performance.
What is the Inﬂuence of the Relaxation in ( ˆ J ) ?
First, it is important that the relaxation is only needed for a stochastic victim.
For a deterministic victim, which is often the case in practice, the actor solves the original unrelaxed objective.
Second, as we will discuss in the next paragraph, the optimality of both SA-RL and PA-AD is regarding the formulation.
That is, SA-RL and PA-AD formulate the optimal attack problem as an
MDP whose optimal policy is the optimal adversary.
However, in a large-scale task, deep RL algorithms themselves usually do not converge to the globally optimal policy and exploration becomes the main challenge.
Thus, when the adversary’s MDP is large, the suboptimality caused by the RL solver due to exploration difﬁculties could be much more severe than the suboptimality caused by
1 Input: Initialization of director’s policy ν ; victim policy π ; budget (cid:15) ; start state s2 for t = 0 , 1 , 2 , ... do
Director samples a policy perturbing direction (cid:98) a t ∼ ν ( ·| s t ) if Victim is deterministic then # for a deterministic victim, J D () is deﬁned in Equation ( J D )
Actor computes the gradient of its objective ∇ δ J D ( s t + δ ; (cid:98) a t , s t ) else # for a stochastic victim, J () is deﬁned in Equation ( ˆ J )
Actor computes the gradient of its objective ∇ δ J ( s t + δ ; (cid:98) a t , s t )
Actor
=
+ δ ) t r t t t t buffer the relaxation of the formulation.
The comparison between SA-RL and PA-AD in our experiments can justify that the size of the adversary MDP has a larger impact than the relaxation of the problem on the ﬁnal solution found by the attackers.
Third, in Appendix F.1, we empirically show that with the relaxed objective, PA-AD can still ﬁnd the optimal attacker in 3 example environments.
Optimality in Formulation v.s. Approximated Optimality in Practice PA-AD has an optimal formulation, as the optimal solution to its objective (the optimal policy in PAMDP) is always an optimal adversary (Theorem 7).
Similarly, the previous attack method SA-RL has an optimal solution since the optimal policy in the adversary’s
MDP is also an optimal adversary.
However, in practice where the environments are in a large scale and the number of samples is ﬁnite, the optimal policy is not guaranteed to be found by either PA-AD and SA-RL with deep RL algorithms.
Therefore, for practical consideration, our goal is to search for a good solution or approximate the optimal solution using optimization techniques (e.g. actor-critic learning, one-step FGSM attack, Lagrangian relaxation for the stochastic-victim attack).
In experiments (Section 6), we show that our implementation universally ﬁnds stronger attackers than prior methods, which veriﬁes the effectiveness of both our theoretical framework and our practical implementation.
C. V ARIANTS F OR E NVIRONMENTS WITH C ONTINUOUS A CTION S PACES
Although the analysis in the main paper focuses on an MDP whose action space is discrete, our algorithm also extends to a continuous action space as justiﬁed in our experiments.
C.3. F OR A D ETERMINISTIC V ICTIM
In this case, we can still use the formulation D-PAMDP, but a slightly different actor function
[Equation]
C.3. F OR A S TOCHASTIC V ICTIM
Different from a stochastic victim in a discrete action space whose actions are sampled from a categorical distribution, a stochastic victim in a continuous action space usually follows a parametrized probability distribution with a certain family of distributions, usually Gaussian distributions.
In this case, the formulation of PAMDP in Deﬁnition 6 is impractical.
However, since the mean of a Gaussian distribution has the largest probability to be selected, one can still use the formulation in ( G CD ), while replacing π D ( s (cid:48) ) with the mean of the output distribution.
Then, the director and the actor can collaboratively let the victim output a Gaussian distribution whose mean is the target action.
If higher accuracy is needed, we can use another variant of PAMDP, named Continuous
Policy Adversary MDP (C-PAMDP) that can also control the variance of the Gaussian distribution.
Deﬁnition 18 ( Continuous Policy Adversary MDP (C-PAMDP) ) .
Given an MDP M = (cid:104)S , A , P, R, γ (cid:105) where A is continuous, a ﬁxed and stochastic victim policy π , we deﬁne a Con- tinuous Policy Adversarial MDP (cid:99) M C
= (cid:104)S , (cid:98) A C , (cid:98) P C , (cid:98) R C , γ (cid:105) , where the action space is (cid:98)
A D =
A , and ∀ s, s (cid:48) ∈ S , ∀ (cid:98) a ∈ A , (cid:98) P ( s (cid:48)
[Equation]
[Equation]
The actor function g is deﬁned as
[Equation]
where σ is a hyper-parameter, and N denotes a multivariate Gaussian distribution.
In short, Equation ( G C ) encourages the victim to output a distribution that is similar to the target distribution.
The hyperparameter σ controls the standard deviation of the target distribution.
One can set σ to be small in order to let the victim execute the target action (cid:98) a with higher probabilities.
D C HARACTERIZE O PTIMALITY OF E VASION A TTACKS
In this section, we provide a detailed characterization for the optimality of evasion attacks from the perspective of policy perturbation, following Deﬁnition 5 in Section 4.
Section D.1 establishes the existence of the optimal policy adversary which is deﬁned in Section 3.
Section D.2 then provides a proof for Theorem 7 that the formulation of PA-AD is optimal.
We also analyze the optimality of heuristic attacks in Section D.3.
D. E XISTENCE OF A N O PTIMAL P OLICY A DVERSARY
Theorem 19 (Existence of An Optimal Policy Adversary) .
Given an MDP M = (cid:104)S , A , P, R, γ (cid:105) , and a ﬁxed stationary policy π on M , let H (cid:15) be a non-empty set of admissible state adversaries and
B H(cid:15)
( π ) be the corresponding Adv-policy-set, then there exists an optimal policy adversary π h ∗ ∈
B H(cid:15)
( π ) such that π h ∗ ∈ argmin
π h ∈B H(cid:15) ( π ) V π h M ( s ) , ∀ s ∈ S .
Proof.
We prove Theorem 19 by constructing a new MDP corresponding to the original MDP M and the victim π .
Deﬁnition 20 ( Policy Perturbation MDP ) .
For a given MDP M , a ﬁxed stochastic victim policy π , and an admissible state adversary set H (cid:15) , deﬁne a policy perturbation MDP as M P
= (cid:104)S , A P , P P , R P , γ (cid:105) , where A P = ∆( A ) , and ∀ s ∈ S , a P ∈ A P ,
[Equation]
otherwise
[Equation]
Then we can prove Theorem 19 by proving the following lemma.
Lemma 21.
The optimal policy in M P is an optimal policy adversary for π in M .
Let N P denote the set of deterministic policies in M P .
According to the traditional MDP theory (Puterman, 1994), there exists a deterministic policy that is optimal in M P .
Note that H (cid:15) is non-empty, so there exists at least one policy in M P with value ≥ −∞ , and then the optimal policy should have value ≥ −∞ .
Denote this optimal and deterministic policy as ν ∗ P ∈ N P .
Then we write the Bellman equation of ν ∗ P , i.e.,
[Equation]
Note that ν ∗ P ( s ) is a distribution on action space, ν ∗ P ( a | s ) is the probability of a given by distribution ν ∗ ( s ) .
Multiply both sides of Equation (6) by − 1 , and we obtain
[Equation]
In the original MDP M , an optimal policy adversary (if exists) π h ∗ for π should satisfy
[Equation]
By comparing Equation (7) and Equation (8) we get the conclusion that ν ∗ P is an optimal policy adversary for π in M .
D. P ROOF OF T HEOREM 7: O PTIMALITY OF O UR PA-AD
In this section, we provide theoretical proof of the optimality of our proposed evasion RL algorithm
PA-AD.
D.2. O PTIMALITY OF PA-AD FOR A S TOCHASTIC V ICTIM
We ﬁrst build a connection between the PAMDP (cid:99) M deﬁned in Deﬁnition 6 (Section 4) and the policy perturbation MDP deﬁned in Deﬁnition 20 (Appendix D.1).
A deterministic policy ν in the PAMDP (cid:99) M can induce a policy ˆ ν P in M P in the following way:
[Equation]
= π ( ·| g ( ν ( s ) , s )) , ∀ s ∈ S .
More importantly, the values of ν and (cid:98) ν P in (cid:99) M and M P are equal because of the formulations of the two MDPs, i.e., (cid:98) V ν = V (cid:98)
ν P P , where (cid:98) V and V P denote the value functions in (cid:99) M and V P respectively.
Proposition 22 below builds the connection of the optimality between the policies in these two
MDPs.
Proposition 22.
An optimal policy in (cid:99) M induces an optimal policy in M P .
Proof of Proposition 22.
Let ν ∗ be an deterministic optimal policy in (cid:99) M , and it induces a policy in
M P , namely (cid:98) ν P .
Let us assume (cid:98)
ν P is not an optimal policy in M P , hence there exists a policy ν ∗ P in M P s.t.
V ν ∗ P P ( s ) >
V (cid:98) ν
P P ( s ) for at least one s ∈ S .
And based on Theorem 4, we are able to ﬁnd such a ν ∗ P whose corresponding policy perturbation is on the outermost boundary of B ( π ) , i.e., ν ∗ ∈ ∂ π B H(cid:15)
( π ) .
Then we can construct a policy ν (cid:48) in (cid:99) M such that ν (cid:48) ( s ) =
ν ∗ P ( s ) − π ( s ) , ∀ s ∈ S .
And based on
Equation ( G ), π ( ·| g ( ν (cid:48) ( s ) , s )) is in ∂ π B ( π ( s )) for all s ∈ S .
According to the deﬁnition of ∂ π , if two policy perturbations perturb π in the same direction and are both on the outermost boundary, then they are equal.
Thus, we can conclude that π ( g ( ν (cid:48) ( s ) , s ))
=
ν ∗ P ( s ) , ∀ s ∈ S .
Then we obtain (cid:98) V ν (cid:48)
( s
) =
V ν ∗ P P ( s ) , ∀ s ∈ S
Now we have conditions:
(1) (cid:98) V ν ∗ ( s ) =
V (cid:98)
ν P P ( s ) , ∀ s ∈ S ; (2) V ν ∗ P P ( s ) >
V (cid:98) ν
P P ( s ) for at least one s ∈ S ; (3) ∃ ν (cid:48)
such that (cid:98) V ν (cid:48) ( s ) =
V ν ∗ P P ( s ) , ∀ s ∈ S .
From (1), (2) and (3), we can conclude that (cid:98) V ν (cid:48) ( s ) > (cid:98)
V ν ∗ ( s ) for at least one s ∈ S , which conﬂicts with the assumption that ν ∗ is optimal in (cid:99) M .
Therefore, Proposition 22 is proven.
Proposition 22 and Lemma 21 together justiﬁes that the optimal policy of (cid:99) M , namely ν ∗ , induces an optimal policy adversary for π in the original M .
Then, if the director learns the optimal policy in (cid:99) M , then it collaborates with the actor and generates the optimal state adversary h ∗ by h ∗ ( s ) =
g ( ν ∗ ( s ) , s ) , ∀ s ∈ S .
D.2. O PTIMALITY OF O UR PA-AD FOR A D ETERMINISTIC V ICTIM
In this section, we show that the optimal policy in D-PAMDP (the deterministic variant of PAMDP deﬁned in Appendix C.1) also induces an optimal policy adversary in the original environment.
Let π D be a deterministic policy reduced from a stochastic policy π , i.e.,
[Equation]
Note that in this case, the Adv-policy-set B H (cid:15) ( π ) is not connected as it contains only deterministic policies.
Therefore, we re-formulate the policy perturbation MDP introduced in Appendix D.1 with a deterministic victim as below:
Deﬁnition 23 ( Deterministic Policy Perturbation MDP ) .
For a given MDP M , a ﬁxed deterministic victim policy π , and an admissible adversary set H (cid:15) , deﬁne a deterministic policy perturbation
MDP as M DP = (cid:104)S , A DP , P DP , R DP , γ (cid:105) , where A DP = A , and ∀ s ∈ S , a DP ∈ A DP ,
R DP ( s, a DP ) := { − R ( s, a DP ) if ∃ h ∈ H (cid:15) s.t.
a DP ( s ) =
π D ( h ( s ))
−∞ otherwise
[Equation] [Equation]
M DP can be viewed as a special case of M P where only deterministic policies have ≥ −∞ values.
Therefore Theorem 19 and Lemma 21 also hold for deterministic victims.
Next we will show that an optimal policy in (cid:99) M D induces an optimal policy in M DP .
Proposition 24.
An optimal policy in (cid:99) M D induces an optimal policy in M DP .
Proof of Proposition 24.
We will prove Proposition 24 by contradiction.
Let ν ∗ be an optimal policy in (cid:99) M D , and it induces a policy in M DP , namely (cid:98)
ν DP .
Let us assume (cid:98)
ν DP is not an optimal policy in M DP , hence there exists a deterministic policy ν ∗ DP in M DP s.t.
V ν ∗ DP DP ( s ) >
V (cid:98)
ν DP DP ( s ) for at least one s ∈ S .
Without loss of generality, suppose
V ν ∗ DP DP ( s 0 ) >
V (cid:98)
ν DP DP ( s 0 ) .
Next we construct another policy ν (cid:48) in (cid:99) M D by setting ν (cid:48) ( s ) =
ν ∗ DP ( s ) , ∀ s ∈ S .
Given that ν ∗ DP is deterministic, ν (cid:48) is also a deterministic policy.
So we use ν ∗ DP ( s ) and ν (cid:48)
( s ) to denote the action selected by ν ∗ DP and ν (cid:48) respectively at state s .
For an arbitrary state s i , let a i :=
ν ∗ DP ( s i ) .
Since ν ∗ DP is the optimal policy in M DP , we get that there exists a state adversary h ∈ H (cid:15) such that π D ( h ( s i ))
= a i , or equivalently, there exists a state ˜ s i ∈ B (cid:15) ( s i ) such that argmax a ∈A π (˜ s i ) = a i .
Then, the solution to the actor’s optimization problem ( G D ) given direction a i and state s i , denoted as ˜ s ∗ , satisﬁes
[Equation]
and we can get
[Equation] [Equation]
Given that argmax a ∈A π ( a i | ˜ s i )
=
a i , we obtain argmax a ∈A π ( a i | ˜ s ∗ )
=
a i , and hence π D ( g D ( a i , s i ))
= a i .
Since this relation holds for an arbitrary state s , we can get
[Equation]
Also, we have ∀ s ∈ S
[Equation]
Therefore,
[Equation]
Then we have
[Equation]
which gives (cid:98) V
[Equation]
<
[Equation]
there is a contradiction.
Combining the results of Proposition 24 and Lemma
[Equation]
for a deterministic victim
[Equation]
the optimal policy in D-PAMDP gives an optimal adversary for the victim.
D. O PTIMALITY OF H EURISTIC - BASED A TTACKS
There are many existing methods of ﬁnding adversarial state perturbations for a ﬁxed RL policy, most of which are solving some optimization problems deﬁned by heuristics.
Although these methods are empirically shown to be effective in many environments, it is not clear how strong these adversaries are in general.
In this section, we carefully summarize and categorize existing heuristic attack methods into 4 types, and then characterize their optimality in theory.
D.3. TYPE I - M INIMIZE T HE B EST (M IN B EST )
A common idea of evasion attacks in supervised learning is to reduce the probability that the learner selects the “correct answer” Goodfellow et al. (2015).
Prior works Huang et al.
; Kos & Song (2017); Korkmaz (2020) apply a similar idea to craft adversarial attacks in RL, where the objective is to minimize the probability of selecting the “best” action, i.e.,
[Equation]
(I) where a + is the “best” action to select at state s .
Huang et al. Huang et al.
(2017) deﬁne a + as argmax a ∈A Q π ( s, a ) for DQN, or argmax a ∈A π ( a | s ) for TRPO and A3C with a stochastic π .
Since the agent’s policy π is usually well-trained in the original MDP, a + can be viewed as (approximately) the action taken by an optimal deterministic policy π ∗ ( s ) .
Lemma 25 ( Optimality of MinBest ) .
Denote the set of optimal solutions to objective (I) as
H MinBest .
There exist an MDP M and an agent policy π , such that H MinBest does not contain an optimal adversary h ∗ , i.e., H MinBest ∩ H ∗ (cid:15) = ∅ .
Proof of Lemma 25.
We prove this lemma by constructing the following MDP such that for any victim policy, there exists a reward conﬁguration in which MinBest attacker is not optimal.
Here, let r 1 = r
( s 4 | s 2 , a 1 ) , r 2 = r
( s 5 | s 2 , a 2 ) , r 3 = r
( s 3 | s 1 , a 2 ) .
Assuming all the other rewards are zero, transition dynamics are deterministic, and states s 3 , s 4 , s 5 are the terminal states.
For the sake of simplicity, we also assume that the discount factor here γ = 1 .
Now given a policy π such that π ( a 1 | s 1 ) =
β 1 and π ( a 1 | s 2 )
=
β 2 ( β 1 , β 2 ∈ [0 , 1] ), we could ﬁnd r 1 , r 2 , r 3 such that the following constraints hold:
[Equation]
Now we consider the Adv-policy-set
[Equation]
Under these three linear constraints, the policy given by MinBest attacker satisﬁes that π h MinBest ( a 1 | s 1 ) = β 1 − (cid:15) 1 , and π h MinBest ( a 1 | s 2 ) = β 2 − (cid:15) 2 .
On the other hand, we can ﬁnd another admissible policy adversary π h ∗ ( a 1 | s 1 ) =
β 1 + (cid:15) 1 , and π h ∗ ( a 1 | s 2 )
=
β 2 − (cid:15) 2 .
Now we show that V π h ∗ ( s 1 ) <
V π h MinBest ( s 1 ) , and thus MinBest attacker is not optimal.
[Equation]
Therefore
[Equation]
< 0 Because of the constraint (19)
[Equation] [Equation]
D.3. TYPE II - M AXIMIZE T HE W ORST (M AX W ORST )
Pattanaik et al.
(Pattanaik et al., 2018) point out that only preventing the agent from selecting the best action does not necessarily result in a low total reward.
Instead, Pattanaik et al.
(Pattanaik et al., 2018) propose another objective function which maximizes the probability of selecting the worst action, i.e.,
[Equation]
where a − refers to the “worst” action at state s .
Pattanaik et al.(Pattanaik et al., 2018) deﬁne the “worst” action as the actions with the lowest Q value, which could be ambiguous, since the Q function is policy-dependent.
If a worst policy π − ∈ argmin π V π ( s ) , ∀ s ∈ S is available, one can use a − =
argmin Q π − ( s, a ) .
However, in practice, the attacker usually only has access to the agent’s current policy π , so it can also choose a − =
argmin Q π ( s, a ) .
Note that these two selections are different, as the agent’s policy π is usually far away from the worst policy.
Lemma 26 ( Optimality of MaxWorst ) .
Denote the set of optimal solutions to objective (II) as
H MaxWorst , which include both versions of MaxWorst attacker formulations as we discussed above.
Then there exist an MDP M and an agent policy π , such that H MaxWorst contains a non-optimal adversary h ∗ , i.e., H MaxWorst (cid:54)⊂ H ∗ (cid:15) .
Proof of Lemma 26.
Case I: Using current policy to compute the target action
We prove this lemma by constructing the MDP in Figure 8 such that for any victim policy, there exists a reward conﬁguration in which MaxWorst attacker is not optimal.
Here, let r 1 = r
( s 11 | s 1 , a 1 ) , r 2 = r
( s 12 | s 1 , a 2 ) , r 3 = r
( s 21 | s 2 , a 1 ) , r 4 = r
( s 22 | s 2 , a 2 ) .
Assuming all the other rewards are zero, transition dynamics are deterministic, and states s 11 , s 12 , s 21 , s 22 are the terminal states.
For the sake of simplicity, we also assume that the discount factor here γ = 1 .
Now given a policy π such that π ( a 1 | s 0 ) =
β 0 , π ( a 1 | s 1 )
=
β 1 , and π ( a 2 | s 2 ) =
β 2 ( β 0 , β 1 , β 2 ∈ [0 , 1] ), consider the Adv-policy-set
[Equation]
We could ﬁnd r 1 , r 2 , r 3 , r 4 such that the following linear constraints hold:
[Equation]
Now, given these constraints, the perturbed policy given by MaxWorst attaker satisﬁes π h MaxWorst ( a 1 | s 0 ) =
β 0 − (cid:15) 0 , π h MaxWorst ( a 1 | s 1 )
=
β 1 − (cid:15) 1 , and π h MaxWorst ( a 1 | s 2 ) =
β 2 − (cid:15) 2 .
However, consider another perturbed policy π h ∗ in Adv-policy-set such that π h ∗ ( a 1 | s 0 )
=
β 0 + (cid:15) 0 , π h ∗ ( a 1 | s 1 )
=
β 1 − (cid:15) 1 , and π h ∗ ( a 1 | s 2 ) =
β 2 − (cid:15) 2 .
We will prove that V π
[Equation]
s
[Equation]
) <
V
[Equation]
s
[Equation] [Equation]
and thus MaxWorst attacker is not optimal.
On the one hand,
V
[Equation]
On the other hand,
[Equation]
Therefore,
[Equation]
Case compute the
Same as before, we construct a MDP where H MaxWorst contains a non-optimal adversary.
Let r 1 = r
( s 1 | s 0 , a 1 ) , r 2 = r
( s 2 | s 0 , a 2 ) , r 3 = r
( s 3 | s 0 , a 3 ) .
Assuming all the other rewards are zero, transition dynamics are deterministic, and states s 1 , s 2 , s 3 are the terminal states.
For the sake of simplicity, we also assume that the discount factor here γ = 1 .
Let pi be the given policy such that π ( a 1 | s 0 )
=
β 1 and π ( a 2 | s 0 )
=
β 2 .
Now without loss of generality, we assume r 1 > r 2 > r 3 ( ∗ ) .
Then the worst policy π (cid:48) satisﬁes that π (cid:48) ( a 3 | s 0 )
= 1 . Consider the Adv-policy-set B H(cid:15)
( π ) =
(cid:110) π (cid:48)
∈ Π (cid:12)(cid:12)(cid:12) (cid:107) π (cid:48) ( ·| s 0 ) − π ( ·| s 0 ) (cid:107) 1 < (cid:15) (cid:111) .
Then H MaxWorst = (cid:110) π (cid:48) ∈ Π (cid:12)(cid:12)(cid:12) π (cid:48)
( a 3 | s 0 )
=
(1 − β 1 − β 2 ) + (cid:15) (cid:111) .
Now consider two policies π h 1 , π h 2 ∈ H MaxWorst , where π h 1 ( a 1 | s 0 )
=
β 1 , π h 1 ( a 2 | s 0 )
=
β 2 − (cid:15) ,
π h 2 ( a 1 | s 0 )
=
β 1 − (cid:15) , π h 2 ( a 2 | s 0 ) =
β 2 .
Then V π h 1 ( s 0 ) −
V π h 2 ( s 0 ) =
(cid:15) ( r 1 − r 2 ) > 0 .
Therefore, π h 1 ∈ H MaxWorst
but it’s not optimal.
D.3. TYPE III - M INIMIZE Q V ALUE (M IN Q).
Another idea of attacking Pattanaik et al.
; Zhang et al.
(2020a) is to craft perturbations such that the agent selects actions with minimized Q values at every step, i.e.,
[Equation]
III [Equation] where ˆ Q is the approximated Q function of the agent’s original policy.
For example, Pattanaik et al.
Pattanaik et al.
(2018) directly use the agent’s Q network (of policy π ), while the Robust SARSA (RS) attack proposed by Zhang et al.
Zhang et al.
(2020a) learns a more stable Q network for the agent’s policy π .
Note that in practice, this type of attack is usually applied to deterministic agents (e.g., DQN, DDPG, etc), then the objective becomes argmin h ∈ H (cid:15)
ˆ Q π ( s, π h ( s )) , ∀ s ∈ S Pattanaik et al.
; Zhang et al.
(2020a)
; Oikarinen et al.
In this case, the MinQ attack is equivalent to the MaxWorst attack with the current policy as the target.
Lemma 27 ( Optimality of MinQ ) .
Denote the set of optimal solutions to objective (III) as H MinQ , which include both versions of MinQ attacker formulations as we discussed above.
Then there exist an MDP M and an agent policy π , such that H MinQ contains a non-optimal adversary h ∗ , i.e.,
H MinQ (cid:54)⊂ H ∗ (cid:15) .
Proof of Lemma 26.
Case I:
For a deterministic victim
In the deterministic case
[Equation]
ˆ Q π ( s, a ) | s ) , ∀ s ∈ S (III D )
In this case, the objective is equivalent to objective (II), thus Lemma 27 holds.
Case II:
stochastic
victim
In this case, we consider the MDP in Figure 8 and condition (27) to (30).
Then the MinQ objective gives π h MinQ ( a 1 | s 0 ) =
β 0 − (cid:15) 0 , π h MinQ ( a 1 | s 1 ) =
β 1 − (cid:15) 1 , and π h MinQ ( a 1 | s 2 ) =
β 2 − (cid:15) 2 .
According to the proof of the ﬁrst case of Lemma 26, π h MinQ =
π h MaxWorst is not an optimal adversary.
Thus Lemma 27 holds.
D.3. TYPE IV - M AXIMIZE D IFFERENCE
[Equation] M AX D IFF ).
The MAD attack proposed by Zhang et al.
(Zhang et al., 2020a) is to maximize the distance between the perturbed policy π h and the clean policy π , i.e., [Equation]
[Equation]
where TV denotes the total variance distance between two distributions.
In practical implementations, the TV distance can be replaced by the KL-divergence, as D TV [ π h ( ·| s )
||
π ( ·| s )] ≤ (D KL [ π h ( ·| s )
||
π ( ·| s )]) 2 .
This type of attack is inspired by the fact that if two policies select actions with similar action distributions on all the states, then the value of the two policies is also small (see Theorem 5 in Zhang et al. (2020a)).
Lemma 28 ( Optimality of MaxDiff ) .
Denote the set of optimal solutions to objective (IV) as
H MaxDiff .
There exist an MDP M and an agent policy π , such that H MaxDiff contains a non-optimal adversary h ∗ , i.e., H MaxDiff (cid:54)⊂ H ∗ (cid:15) .
Proof of Lemma 28.
The proof follows from the proof of lemma 25.
In the MDP we constructed, π (cid:48)
=
β 1 − (cid:15) 1 , π h MinBest ( a 1 | s 2 ) =
β 2 − (cid:15) 2 is one of the policies that has the maximum KL divergence from the victim policy within Adv-policy-set.
However, as we proved in 25, this is not the optimally perturbed policy.
Therefore, MaxDiff attacker may not be optimal.
E A DDITIONAL E XPERIMENT D ETAILS AND R ESULTS
In this section, we provide details of our experimental settings and present additional experimental results.
Section E.1 describes our implementation details and hyperparameter settings for Atari and
MuJoCo experiments.
Section E.2 provide additional experimental results, including experiments with varying budgets ( (cid:15) ) in Section E.2.1, more comparison between SA-RL and PA-AD in terms of convergence rate and sensitivity to hyperparameter settings as in Section E.2.3, robust training in
MuJoCo games with fewer training steps in Section E.2.4, attacking performance on robust models in Atari games in Section E.2.5, as well as robust training results in Atari games in Section E.2.6.
E. I MPLEMENTATION D ETAILS E.1. A TARI E XPERIMENTS
In this section we report the conﬁgurations and hyperparameters we use for DQN, A2C and ACKTR in Atari environments.
We use GeForce RTX 2080 Ti GPUs for all the experiments.
DQN Victim We compare PA-AD algorithm with other attacking algorithms on 7 Atari games.
For DQN, we take the softmax of the Q values Q ( s, · ) as the victim policy π ( ·| s ) as in prior works (Huang et al., 2017).
For these environments, we use the wrappers provided by stablebaselines (Hill et al., 2018), where we clip the environment rewards to be − 1 and 1 during training and stack the last 4 frames as the input observation to the DQN agent.
For the victim agent, we implement Double Q learning (Hado Van Hasselt, 2016) and prioritized experience replay (Tom Schaul & Silver, 2016).
The clean DQN agents are trained for 6 million frames, with a learning rate 0 .and
the same network architecture and hyperparameters as the ones used in Mnih et al.
In addition, we use a replay buffer of size 5 × 10 5 .
Prioritized replay buffer sampling is used with α = 0 .
6 and β increases from 0 .
4 to 1 linearly during training.
During evaluation, we execute the agent’s policy without epsilon greedy exploration for 1000 episodes.
A2C Victim For the A2C victim agent, we also use the same preprocessing techniques and convolutional layers as the one used in Mnih et al.
Besides, values and policy network share the same CNN layers and a fully-connected layer with 512 hidden units.
The output layer is a categorical distribution over the discrete action space.
We use 0.0007 as the initial learning rate and apply linear learning rate decay, and we train the victim A2C agent for 10 million frames.
During evaluation, the A2C victim executes a stochastic policy (for every state, the action is sampled from the categorical distribution generated by the policy network).
Our implementation of A2C is mostly based on an open-source implementation by Kostrikov Kostrikov (2018).
ACKTR Adversary To train the director of PA-AD and the adversary in SA-RL, we use
ACKTR (Wu et al., 2017) with the same network architecture as A2C. We train the adversaries of PA-AD and SA-RL for the same number of steps for a fair comparison.
For the DQN victim, we use a learning rate 0.0001 and train the adversaries for 5 million frames.
For the A2C victim, we use a learning rate 0.0007 and train the adversaries for 10 million frames.
Our implementation of
ACKTR is mostly based on an open-source implementation by Kostrikov Kostrikov (2018).
Heuristic Attackers For the MinBest attacker, we following the algorithm proposed by Huang et al.
(2017) which uses FGSM to compute adversarial state perturbations.
The MinBest + Mo- mentum attacker is implemented according to the algorithm proposed by Korkmaz (2020), and we set the number of iterations to be 10, the decaying factor µ to be 0.5 (we tested 0 .
5 , 0 .and found 0.5 is relatively better while the difference is minor).
Our implementation of the MinQ attacker follows the gradient-based attack by Pattanaik et al.
and we also set the number of iterations to be 10.
For the MaxDiff attacker, we refer to Algorithm 3 in Zhang et al.
(2020a) with the number of iterations equal to 10.
In addition, we implement a random attacker which perturbs state s to ˜ s
= s + (cid:15) sign( µ ) , where µ is sampled from a standard multivariate Gaussian distribution with the same dimension as s .
E.1. M U J O C O E XPERIMENTS
For four OpenAI Gym MuJoCo continuous control environments, we use PPO with the original fully connected (MLP) structure as the policy network to train the victim policy.
For robustness evaluations, the victim and adversary are both trained using PPO with independent value and policy optimizers.
We complete all the experiments on MuJoCo using 32GB Tesla V100 .
PPO Victim We directly use the well-trained victim model provided by Zhang et al.
(2020a).
PPO Adversary Our PA-AD adversary is trained by PPO and we use a grid search of a part of adversary hyperparameters (including learning rates of the adversary policy network and policy network, the entropy regularization parameter and the ratio clip (cid:15) for PPO) to train the adversary as powerful as possible.
The reported optimal attack result is from the strongest adversary among all 50 trained adversaries.
Other Attackers For Robust Sarsa (RS) attack, we use the implementation and the optimal RS hyperparameters from Zhang et al.
(2020a) to train the robust value function to attack the victim.
The reported RS attack performance is the best one over the 30 trained robust value functions.
For MaxDiff attack, the maximal action difference attacker is implemented referring to Zhang et al.
(2020a).
For SA-RL attacker, following Zhang et al.
(2021), the hyperparameters is the same as the optimal hyperparameters of vanilla PPO from a grid search.
And the training steps are set for different environments.
For the strength of SA-PPO regularization κ , we choose from 1 × 10 − 6 to 1 and report the worst-case reward.
Robust Training For ATLA Zhang
et al.
(2021), the hyperparameters for both victim policy and adversary remain the same as those in vanilla PPO training.
To ensure sufﬁcient exploration, we run a small-scale grid search for the entropy bonus coefﬁcient for agent and adversary.
The experiment results show that a larger entropy bonus coefﬁcient allows the agent to learn a better policy for the continual-improving adversary.
In robust training experiments, we use larger training steps in all the MuJoCo environments to guarantee policy convergence.
We train 5 million steps in Hopper,
Walker, and HalfCheetah environments and 10 million steps for Ant.
For reproducibility, the ﬁnal results we reported are the experimental performance of the agent with medium robustness fromagents training with the same hyperparameter set.
E. A DDITIONAL E XPERIMENT R ESULTS E.2. A TTACKING P ERFORMANCE WITH V ARIOUS B UDGETS
In Table 1, we report the performance of our PA-AD attacker under a chosen epsilon across different environments.
To see how PA-AD algorithm performs across different values of (cid:15) ’s, here we select three Atari environments each for DQN and A2C victim agents and plot the performance of PA-AD under various (cid:15) ’s compared with the baseline attackers in Figure 10.
We can see from the ﬁgures that our PA-AD universally outperforms baseline attackers concerning various (cid:15) ’s.
In Table 2, we provide the evaluation results of PA-AD under a commonly unused epsilon in four
MuJoCo experiments (Zhang et al.
(2020a; 2021)) to show that PA-AD attacker also has the best attacking performance compared with other attackers under different (cid:15) ’s in Figure 11.
[Figure] [Figure]
[Figure] [Figure]
E.2. H YPERPARAMETER T EST
In our Actor-Director Framework, solving an optimal actor is a constraint optimization problem.
Thus, in our algorithm, we instead use Lagrangian relaxation for the actor’s constraint optimization.
In this section, we report the effects of different choices of the relaxation hyperparameter λ on the ﬁnal performance of our algorithm.
Although we set λ by default to be 1 and keep it ﬁxed throughout all of the other experiments, here we ﬁnd that in fact, difference choice of λ has only minor impact on the performance of the attacker.
This result demonstrates that our PA-AD algorithm is robust to different choices of relaxation hyperparameters.
[Table]
{+Reward 
+}
{+±96 ±λ = 0 .− 19 ±16 ±λ = 0 .− 
 ±17 ±λ = 0 .− 20 ±19 ±λ = 0 .− 19 ±14 ±λ = 1 .−+}
[Table]
λ = [Table] [Table] −
[Table]
=
[Table]
−
[Table]
±
[Table]
±
[Table]
λ =
[Table]
−
[Table]
±
[Table]
±λ =
[Table]
.−
[Table]
±
[Table]
±λ = 0 .− ±872 ±λ = 1 .− 2580 ±
[Table]
±λ
{+= 2 .− 
+}
±795 ±λ
= 5 .− 2425 ±
[Table]
±
[Table]
E.
[Table]
E MPIRICAL C OMPARISON BETWEEN PA-AD AND SA-RL
In this section, we provide more empirical comparison between PA-AD and SA-RL.
Note that PAAD and SA-RL are different in terms of their applicable scenarios: SA-RL is a black-box attack methods, while PA-AD is a white-box attack method.
When the victim model is known, we can see that by a proper exploitation of the victim model, PA-AD demonstrates better attack performance, higher sample and computational efﬁciency, as well as higher scalability.
Appendix F.2 shows detailed theoretical comparison between SA-RL and PA-AD.
PA-AD has better convergence property than SA-RL.
In Figure 12, we plot the learning curves of SA-RL and PA-AD in the CartPole environment and the Ant environment.
Compared with SARL attacker, PA-AD has a higher attacking strength in the beginning and converges much faster.
In
Figure 12b, we can see that PA-AD has a “warm-start” (the initial reward of the victim is already signiﬁcantly reduced) compared with SA-RL attacker which starts from scratch.
This is because
PA-AD always tries to maximize the distance between the perturbed policy and the original victim policy in every step according to the actor function ( G ).
So in the beginning of learning, PA-AD works similarly to the MaxDiff attacker, while SA-RL works similarly to a random attacker.
We also note that although PA-AD algorithm is proposed particularly for environments that have state spaces much larger than action spaces, in CartPole where the state dimensions is fewer than the number of actions, PA-AD still works better than SA-RL because of the distance maximization.
[Figure]
PA-AD is more computationally efﬁcient than SA-RL.
Our experiments in Section 6 show that
PA-AD converges to a better adversary than SA-RL given the same number of training steps, which veriﬁes the sample efﬁciency of PA-AD.
Another aspect of efﬁciency is based on the computational resources, including running time and required memory.
For RL algorithms, the computation cost comes from the interaction with the environment (the same for SA-RL and PA-AD) and the policy/value update.
If the state space S is higher-dimensional than the action space A , then SA-RL requires a larger policy network than PA-AD since SA-RL has a higher-dimensional output, and thus SA-RL has more network parameters than PA-AD, which require more memory cost and more computation operations.
On the other hand, PA-AD requires to solve an additional optimization problem deﬁned by the actor objective ( G ) or ( G D ).
In our implementation, we use FGSM which only requires one-step gradient computation and is thus efﬁcient.
But if more advanced optimization algorithms (e.g. PGD) are used, more computations may be needed.
In summary, if S is much larger than A , PA-AD is more computational efﬁcient than SA-RL; if A is much larger than S , SARL is more efﬁcient than PA-AD; if the sizes of S and A are similar, PA-AD may be slightly more expensive than SA-RL, depending on the optimization methods selected for the actor.
To verify the above analysis, we compare computational training time for training SA-RL and PAAD attackers, which shows that PA-AD is more computationally efﬁcient.
Especially on the environment with high-dimensional states like Ant, PA-AD takes signiﬁcantly less training time than
SA-RL (and ﬁnds a better adversary than SA-RL), which quantiﬁes the efﬁciency of our algorithm in empirical experiments.
[Equation] [Table] PA-AD is less sensitive to hyperparameters settings than SA-RL. In addition to better ﬁnal attacking results and convergence property, we also observe that PA-AD is much less sensitive to hyerparameter settings compared to SA-RL.
On the Walker environment, we run a grid search over 216 different conﬁgurations of hyperparameters, including actor learning rate, critic learning rate, entropy regularization coefﬁcient, and clipping threshold in PPO.
Here for comparison we plot two histograms of the agent’s ﬁnal attacked results across different hyperparameter conﬁgurations.
[Figure]
The perturbation radius is set to be 0.05, for which the mean reward reported by Zhang et al.
(2020a) is 1086.
However, as we can see from this histogram, only one out of the 216 conﬁgurations of SARL achieves an attacking reward within the range 1000-2000, while in most hyperparameter settings, the mean attacked return lies in the range 4000-4500.
In contrast, about 10% hyperparameter settings of PA-AD algorithm are able to reduce the reward to 500-1000, and another 10% settings could reduce the reward to 1000-2000.
Therefore, the performance of PA-AD attacker is generally better and more robust across different hyperparameter conﬁgurations than SA-RL.
E.2. R OBUST T RAINING E FFICIENCY ON M U J O C O BY PA-ATLA
In the ATLA process proposed by Zhang et al.
(2021), one alternately trains an agent and an adversary.
As a result, the agent policy may learn to adapt to the speciﬁc type of attacker it encounters during training.
In Table 3, we present the performance of our robust training method PA-ATLAPPO compared with ATLA-PPO under different types of attacks during testing.
ATLA-PPO uses SA-RL to train the adversary, while PA-ATLA-PPO uses PA-AD to train the adversary during alternating training.
As a result, we can see that ATLA-PPO models perform better under the SA-RL attack, and PA-ATLA-PPO performs better under the PA-AD attack.
However, the advantage of ATLA-PPO over PA-ATLA-PPO against SA-RL attack is much smaller than the advantage of PAATLA-PPO over ATLA-PPO against PA-AD attack.
In addition, our PA-ATLA-PPO models significantly outperform ATLA-PPO models against other heuristic attack methods, and achieve higher average rewards across all attack methods.
Therefore, PA-ATLA-PPO is generally more robust than ATLA-PPO.
Furthermore, the efﬁciency of training an adversary could be the bottleneck in the ATLA Zhang et al.
(2021) process for practical usage.
Appendix E.2.3 suggests that our PA-AD generally converges faster than SA-RL.
Therefore, when the computation resources are limited, PA-ATLA-PPO can train robust agents faster than ATLA-PPO.
We conduct experiments on continuous control environments to empirically show the efﬁciency comparison between PA-ATLA-PPO and ATLA-PPO.
In Table [Table] we show the robustness performance of two ATLA methods with 2 million training steps for Hopper, Walker and Halfcheetah and 5 million steps for Ant (Compared with results in Table 3, we have reduced training steps by half or more).
It can be seen that our PA-ATLA-PPO models still signiﬁcantly outperform the original ATLA-PPO models under different types of attacks.
More importantly, our PA-ATLA-PPO achieves higher robustness under SA-RL attacks in Walker and Ant, suggesting the efﬁciency and effectiveness of our method. [Table]
E.2. A TTACKING R OBUSTLY T RAINED A GENTS ON A TARI
In this section, we show the attack performance of our proposed algorithm PA-AD against DRL agents that are trained to be robust by prior works (Zhang et al., 2020a; Oikarinen et al., 2020) in
Atari games.
Zhang et al.
(2020a) propose SA-DQN , which minimizes the action change under possible state perturbations within (cid:96) p norm ball, i.e., to minimize the extra loss
[Equation]
where θ refers to the Q network parameters, a ∗ ( s )
= argmax a Q θ ( a | s ) , and c is a small constant.
Zhang et al.
(2020a) solve the above optimization problem by a convex relaxation of the Q network, which achieves 100% action certiﬁcation (i.e. the rate that action changes with a constrained state perturbation) in Pong and Freeway, over 98% certiﬁcation in BankHeist and over 47% certiﬁcation in RoadRunner under attack budget (cid:15) = 1 / 255 .
[Table] 620 ±
Oikarinen et al.
(2020) propose another robust training method named RADIAL-RL.
By adding a adversarial loss to the classical loss of the RL agents, and solving the adversarial loss with interval bound propagation, the proposed RADIAL-DQN and RADIAL-A3C achieve high rewards in Pong, Freeway, BankHeist and RoadRunner under attack budget (cid:15) = = 3 /
Implementation of the Robust Agents and Environments.
We directly use the trained SA-DQN agents provided by Zhang et al.
(2020a), as well as RADIAL-DQN and RADIAL-A3C agents provided by Oikarinen et al.
During test time, the agents take actions deterministically.
In order to reproduce the results in these papers, we use the same environment conﬁgurations as in Zhang et al.
(2020a) and Oikarinen et al. (2020), respectively.
But note that the environment conﬁgurations of SA-DQN and RADIAL-RL are simpler versions of the traditional Atari conﬁgurations we use (described in Appendix E.1.1).
Both SA-DQN and RADIAL-RL use a single frame instead of the stacking as 4 frames.
Moreover, SA-DQN restricts the number of actions as 6 (4 for Pong) in each environment, although the original environments have 18 actions (6 for Pong).
The above simpliﬁcations in environments can make robust training easier since the dimensionality of the input space is much smaller, and the number of possible outputs is restricted.
Attack Methods In experiments, we ﬁnd that the robust agents are much harder to attack than vanilla agents in Atari games, as claimed by the robust training papers (Zhang et al., 2020a; Oikarinen et al., 2020).
A reason is that Atari games have discrete action spaces, and leading an agent to make a different decision at a state with a limited perturbation could be difﬁcult.
Therefore, we use a 30-step Projected Gradient Descent for all attack methods (with step size (cid:15)/ 10 ), including
MinBest (Huang et al., 2017) and our PA-AD which use FGSM for attacking vanilla models.
Note that the PGD attacks used by Zhang et al.
(2020a) and Oikarinen et al.
(2020) in their experiments are the same as the MinBest-PGD attack we use.
For our PA-AD, we use PPO to train the adversary since PPO is relatively stable.
The learning rate is set to be 5e − 4 , and the clip threshold is 0.1.
Note that SA-DQN, RADIAL-DQN and RADIAL-A3C agents all take deterministic actions, so we use the deterministic formulation of PA-AD as described in Appendix C.1.
In our implementation, we simply use a CrossEntropy loss for the actor as in Equation (38).
[Equation]
Experiment Results In Table 7, we reproduce the results reported by Zhang et al.
(2020a) and Oikarinen et al.
(2020), and demonstrate the average rewards gained by these robust agents under different attacks in RoadRunner and BankHeist.
Note that SA-DQN is claimed to be robust to attacks with budget (cid:15)
= 1 / 255 , and RADIAL-DQN and RADIAL-A3C are claimed to be relatively robust against up to (cid:15)
= 3 / 255 attacks.
( (cid:96) ∞ is used in both papers.)
So we use the same (cid:15) ’s for these agents in our experiments.
It can be seen that compared with vanilla agents in Table 1, SA-DQN, RADIAL-DQN and RADIALA3C are more robust due to the robust training processes.
However, in some environments, PA-AD can still decrease the rewards of the agent signiﬁcantly.
For example, in RoadRunner with (cid:15) = 3 / 255 , RADIAL-DQN gets 1k+ reward against our PA-AD attack, although RADIAL-DQN under other attacks can get 10k+ reward as reported by Oikarinen et al.
In contrast, we ﬁnd that RADIAL-A3C is relatively robust, although the natural rewards gained by RADIAL-A3C are not as high as RADIAL-DQN and SA-DQN.
Also, as SA-DQN achieves over 98% action certiﬁcation in BankHeist, none of the attackers is able to noticeably reduce its reward with (cid:15) = 1 / 255 .
Therefore, our PA-AD can approximately evaluate the worst-case performance of an RL agent under attacks with ﬁxed constraints, i.e., PA-AD can serve as a “detector” for the robustness of RL agents.
For agents that perform well under other attacks, PA-AD may still ﬁnd ﬂaws in the models and decrease their rewards; for agents that achieve high performance under PA-AD attack, they are very likely to be robust against other attack methods.
E.2. I MPROVING R OBUSTNESS ON A TARI BY PA-ATLA
Note that different from SA-DQN (Zhang et al., 2020a) and RADIAL-RL (Oikarinen et al., 2020) discussed in Appendix E.2.5, we use the traditional Atari conﬁgurations (Mnih et al., 2015) without any simpliﬁcation (e.g. disabling frame stacking, or restricting action numbers).
We aim to improve the robustness of the agents in original Atari environments, as in real-world applications, the environments could be complex and unchangeable.
Baselines We propose PA-ATLA-A2C by combining our PA-AD and the ATLA framework proposed by Zhang et al.
We implement baselines including vanilla A2C, adversarially trained A2C (with MinBest (Huang et al., 2017) and MaxDiff (Zhang et al., 2020a) adversaries attacking 50 frames).
SA-A2C (Zhang et al., 2020a) is implemented using SGLD and convex relaxations in Atari environments.
In Table [Table] naive adversarial training methods have unreliable performance under most strong attacks and SA-A2C is ineffective under PA-AD strongest attack.
To provide evaluation using different (cid:15) , we provide the attack rewards of all robust models with different attack budgets (cid:15) .
Under all attacks with different (cid:15) value, PA-ATLA-A2C models outperform all other robust models and achieve consistently better average rewards across attacks.
We can observe that our PA-ATLA-A2C training method can considerably enhance the robustness in Atari environments
[Table]
F A DDITIONAL D ISCUSSION OF O UR A LGORITHM F. O PTIMALITY OF O UR R ELAXED O BJECTIVE FOR S TOCHASTIC V ICTIMS
Proof of Concept: Optimality Evaluation in
A
We implemented and tested heuristic attacks and our PA-AD in the example 2-state MDP used in
Figure 6, and visualize the results in Figure 14.
For simplicity, assume the adversaries can perturb only perturb π at s 1 within a (cid:96) 2 norm ball of radius 0.2.
And we let all adversaries perturb the policy directly based on their objective functions.
As shown in Figure 14a, all possible ˜ π ( s 1 ) ’s form a disk in the policy simplex, and executing above methods, as well as our PA-AD, leads to 4 different policies on this disk.
All these computed policy perturbations are on the boundary of the policy perturbation ball, justifying our Theorem 4.
[Figure] As our theoretical results suggest, the resulted value vectors lie on a line segment shown in Figure 14b and a zoomed in version Figure 14c, where one can see that MinBest, MaxWorst and MAD all fail to ﬁnd the optimal adversary (the policy with lowest value).
On the contrary, our PA-AD ﬁnds the optimal adversary that achieves the lowest reward over all policy perturbations.
For Continuous MDP: Optimality Evaluation in CartPole and MountainCar
We provided a comparison between SA-RL and PA-AD in the CartPole environment in Figure 15, where we can see the SA-RL and PA-AD converge to the same result (the learned SA-RL adversary and PA-AD adversary have the same attacking performance).
CartPole has a 4-dimensional state space, and contains 2 discrete actions.
Therefore since SA-RL has an optimal formulation, we expect SA-RL to converge to the optimal adversary in a small MDP like CartPole.
Then the result in Figure 15 suggests that our PA-AD algorithm, although with a relaxation in the actor optimization, also converges to the optimal adversary with even a faster rate than SA-RL (the reason is explained in Appendix E.2.3).
In addition to CartPole, we also run experiments in MountainCar with a 2-dimensional state space against a DQN victim.
The SA-RL attacker reduces the victim reward to -128, and our PA-AD attacker reduces the victim reward to -199.45 within the same number of training steps.
Note that the lowest reward in MountainCar is -200, so our PA-AD indeed converges to a near-optimal adversary, while SA-RL fails to converge to a near-optimal adversary.
This is because MountainCar is an environment with relatively spare rewards.
The actor in PA-AD utilizes our Theorem 4 and only focuses on perturbations in the outermost boundary, which greatly reduces the exploration burden in solving an RL problem .
In contrast, SA-RL directly uses RL algorithms to learn the perturbation, and thus it has difﬁculties in converging to the optimal solution.
F. M ORE C OMPARISON BETWEEN SA-RL AND PA-AD
We provide a more detailed comparison between SA-RL and PA-AD from the following multiple aspects to claim our contribution.
1. Size of the Adversary MDP
Suppose the original MDP has size |S| , |A| for its state space and action space, respectively.
Both
PA-AD and SA-RL construct an adversary’s
MDP and search for the optimal policy in it.
But the adversary’s MDPs for PA-AD and SA-RL have different sizes.
PA-AD : state space is of size |S| , action space is of size R |A|− 1 for a stochastic victim, or |A| for a deterministic victim.
SA-RL : state space is of size |S| , action space is of size |S| .
2. Learning Complexity and Efﬁciency
When the state space is larger than the action space, which is very common in RL environments, PAAD solves a smaller MDP than SA-RL and thus more efﬁcient.
In environments with pixel-based states, SA-RL becomes computationally intractable, while PA-AD still works.
It is also important to note that the actor’s argmax problem in PA-AD further accelerates the convergence, as it rules out the perturbations that do not push the victim policy to its outermost boundary.
Our experiment and analysis in Appendix E.2.3 verify the efﬁciency advantage of our PA-AD compared with SA-RL, even in environments with small state spaces.
3. Optimality
PA-AD : (1) the formulation is optimal for a deterministic victim policy; (2) for a stochastic victim policy, the original formulation is optimal, but in practical implementations, a relaxation is used which may not have optimality guarantees.
SA- RL : the formulation is optimal.
Note that both SA-RL and PA-AD require training an RL attacker, but the RL optimization process may not converge to the optimal solution, especially in deep RL domains.
Therefore, SA-RL and
PA-AD are both approximating the optimal adversary in practical implementations.
4. Knowledge of the Victim
PA-AD : needs to know the victim policy (white-box).
Note that in a black-box setting, PA-AD can still be used based on the transferability of adversarial attacks in RL agents, as veriﬁed by Huang et al.
But the optimality guarantee of PA-AD does not hold in the black-box setting.
SA-RL : does not need to know the victim policy (black-box).
It should be noted that the white-box setting is realistic and helps in robust training: (1) The white-box assumption is common in existing heuristic methods.
(2) It is always a white-box process to evaluate and improve the robustness of a given agent, for which PA-AD is the SOTA method.
As discussed in our Ethics Statement, the ultimate goal of ﬁnding the strongest attacker is to better understand and improve the robustness of RL agents.
During the robust training process, the victim is the main actor one wants to train, so it is a white-box setting.
The prior robust training art ATLA (Zhang et al., 2021) uses the black-box attacker SA-RL, despite the fact that it has white-box access to the victim actor.
Since SA-RL does not utilize the knowledge of the victim policy, it usually has to deal with a more complex MDP and face converging difﬁculties.
In contrast, if one replaces SA-RL with our PA-AD, PA-AD can make good use of the victim policy and ﬁnd a stronger attacker with the same training steps as SA-RL, as veriﬁed in our Section 6 and Appendix E.2.6. 5. Applicable Scenarios
SA-RL is a good choice if (1) the action space is much larger than the state space in the original MDP, or the state space is small and discrete; (2) the attacker wants to conduct black-box attacks. PA-AD is a good choice if (1) the state space is much larger than the state space in the original
MDP; (2) the victim policy is known to the attacker; (3) the goal is to improve the robustness of one’s own agent via adversarial training.
In summary, as we discussed in Section 4, there is a trade-off between efﬁciency and optimality in evasion attacks in RL.
SA-RL has an optimal RL formulation, but empirical results show that
SA-RL usually do not converge to the optimal adversary in a continuous state space, even in a lowdimensional state space (e.g. see Appendix F.1 for an experiment in MountainCar).
Therefore, the difﬁculty of solving an adversary’s MDP is the bottleneck for ﬁnding the optimal adversary.
Our
PA-AD, although may sacriﬁce the theoretical optimality in some cases, greatly reduces the size and the exploration burden of the attacker’s RL problem (can also be regarded as trading some estimation bias off for lower variance).
Empirical evaluation shows our PA-AD signiﬁcantly outperforms SARL in a wide range of environments.
Though PA-AD requires to have access to the victim policy, PA-AD solves a smaller-sized RL problem than SA-RL by utilizing the victim’s policy and can be applied on evaluating/improving the robustness of RL policy.
It is possible to let PA-AD work in a black-box setting based on the transferability of adversarial attacks.
For example, in a black-box setting, the attacker can train a proxy agent in the same environment, and use PA-AD to compute a state perturbation for the proxy agent, then apply the state perturbation to attack the real victim agent.
This is out of the scope of this paper, and will be a part of our future work.
F. V ULNERABILITY OF RL A GENTS
It is commonly known that neural networks are vulnerable to adversarial attacks (Goodfellow et al., 2015).
Therefore, it is natural that deep RL policies, which are modeled by neural networks, are also vulnerable to adversarial attacks (Huang et al., 2017).
However, there are few works discussing the difference between deep supervised classiﬁers and DRL policies in terms of their vulnerabilities.
In this section, we take a step further and investigate the vulnerability of DRL agents, through a comparison with standard adversarial attacks on supervised classiﬁers.
Our main conclusion is that commonly used deep RL policies can be instrinsically much more vulnerable to small-radius adversarial attacks.
The reasons are explained below.
1. Optimization process
Due to the different loss functions that RL and supervised learning agents are trained on, the size of robustness radius of an RL policy is much smaller than that of a vision-based classiﬁer.
On the one hand, computer vision-based image classiﬁers are trained with cross-entropy loss.
Therefore, the classiﬁer is encouraged to make the output logit of the correct label to be larger than the logits of other labels to maximize the log probability of choosing the correct label.
On the other hand, RL agents, in particular DQN agents, are trained to minimize the Bellman Error instead.
Thus the agent is not encouraged to maximize the absolute difference between the values of different actions.
Therefore, if we assume the two networks are lipschitz continuous and their lipschitz constants do not differ too much, it is clear that a supervised learning agent has a much larger perturbation radius than an RL agent.
To prove our claim empirically, we carried out a simple experiment, we compare the success rate of target attacks of a well-trained DQN agent on Pong with an image classiﬁer trained on the
CIFAR-10 dataset with similar network architecture.
For a fair comparison, we use the same image preprocessing technique, which is to divide the pixel values by 255 and no further normalization is applied.
On both the image-classiﬁer and DQN model, we randomly sample a target label other than the model predicted label and run the same 100-step projected gradient descent (PGD) attack to minimize the cross-entropy loss between the model output and the predicted label.
We observe that for a perturbation radius of 0.005 ( l ∞ norm), the success rate of a targeted attack for the image classiﬁer is only 15% , whereas the success rate of a targeted attack for the DQN model is 100% .
This veriﬁes our claim that a common RL policy is much more vulnerable to small-radius adversarial attacks than image classiﬁers.
2. Network Complexity
In addition, we also want to point out that the restricted network complexity of those commonly used deep RL policies could play an important role here.
Based on the claim by Madry et al.
(2018), a neural network with greater capacity could have much better robustness, even when trained with only clean examples.
But for the neural network architectures commonly used in RL applications, the capacity of the networks is very limited compared to SOTA computer vision applications.
For example, the commonly used DQN architecture proposed in Mnih et al.
(2015) only has 3 convolutional layers and 2 fully connected layers.
But in vision tasks, a more advanced and deeper structure (e.g. ResNet has 100 layers) is used.
Therefore, it is natural that the perturbation radius need for attacking an RL agent is much smaller than the common radius studied in the supervised evasion attack and adversarial learning literature.