V ULNERABILITY -A WARE P OISONING M ECHANISM FOR O NLINE RL WITH U NKNOWN D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
Poisoning attacks on Reinforcement Learning (RL) systems could take advantage of RL algorithm’s vulnerabilities and cause failure of the learning.
However, prior works on poisoning RL usually either unrealistically assume the attacker knows the underlying Markov Decision Process (MDP), or directly apply the poisoning methods in supervised learning to RL.
In this work, we build a generic poisoning framework for online RL via a comprehensive investigation of heterogeneous poisoning models in RL.
Without any prior knowledge of the MDP, we propose a strategic poisoning algorithm called Vulnerability-Aware Adversarial Critic Poison (VA2C-P), which works for most policy-based deep RL agents, closing the gap that no poisoning method exists for policy-based RL agents.
VA2C-P uses a novel metric, stability radius in RL, that measures the vulnerability of RL algorithms.
Experiments on multiple deep RL agents and multiple environments show that our poisoning algorithm successfully prevents agents from learning a good policy or teaches the agents to converge to a target policy, with a limited attacking budget.
I NTRODUCTION
Although reinforcement learning (RL), especially deep RL, has been successfully applied in various ﬁelds, the security of RL techniques against adversarial attacks is not yet well understood.
In realworld scenarios, including high-stakes ones such as autonomous driving vehicles and healthcare systems, a bad decision may lead to a tragic outcome.
Should we trust the decision made by an
RL agent?
How easy is it for an adversary to mislead the agent?
These questions are crucial to ask before deploying RL techniques in many applications.
In this paper, we focus on poisoning attacks , which occur during the training and inﬂuence the learned policy.
Since training RL is known to be very sample-consuming, one might have to constantly interact with the environment to collect data, which opens up a lot of opportunities for an attacker to poison the training samples collected.
Therefore, understanding poisoning mechanisms and studying the vulnerabilities in RL are crucial to provide guidance for defense methods.
However, existing works on adversarial attacks in RL mainly study the test-time evasion attacks (Chen et al., 2019) where the attacker crafts adversarial inputs to fool a well-trained policy, but does not cause any change to the policy itself.
Motivated by the importance of understanding RL security in the training process and the scarcity of relevant literature, in this paper, we investigate how to poison
RL agents and how to characterize the vulnerability of deep RL algorithms.
In general, RL agents learn in an “online” manner.
At every iteration, the agent rolls out experience from the environment with its policy, and uses the experience to improve its policy, then uses the new policy to roll out new experience, etc.
Poisoning in online RL is signiﬁcantly different from poisoning in classic supervised learning (SL), even online SL, and could be more difﬁcult due to the following challenges.
Challenge I – Future Data Unavailable in Online RL.
Poisoning approaches in SL (Mu˜noz-Gonz´alez et al., 2017; Wang & Chaudhuri, 2018) usually require the access to the whole training dataset, so the attacker can decide the optimal poisoning strategy before the learning starts.
However, in online
RL, the training data (trajectories) are generated by the agent while it is learning.
Although the optimal poison should work in the long run, the attacker can only access and change the data in the current iteration, since the future data is not yet generated.
# $ :& =# ) : & =!
+ ! ", # $ :& =
# $ :& =# ) :& =# ) :& =# $ :& =
Challenge II – Data Samples No Longer
i.i.d..
It is well-known that in RL, data samples (state-action transitions) are no longer i.i.d., which makes learning challenging, since we should consider the longterm reward rather than the immediate result.
However, we notice that data samples being not i.i.d.
also makes poisoning attacks challenging.
For example, an attacker wants to reduce the agent’s total reward in a task shown as Figure 1; at state s 1 , the attacker ﬁnds that a 1 is less rewarding than a 0 ; if the attacker only looks at the immediate reward, he will lure the agent into choosing a 1 .
However, following a 1 ﬁnally leads the agent to swhich has a much higher reward.
Challenge III – Unknown Dynamics of Environment.
Although Challenge I and II can be partially addressed by predicting the future trajectories or steps, it requires prior knowledge on the dynamics of the underlying MDP.
Many existing poisoning RL works (Rakhsha et al., 2020; Ma et al., 2019) assume the attacker has perfect knowledge of the MDP, then compute the optimal poisoning.
However, in many real-world environments, knowing the dynamics of the MDP is difﬁcult.
Although the attacker could potentially interact with the environment to build an estimate of the environment model, the cost of interacting with the environment could be unrealistically high, market making (Spooner et al., 2018) for instance.
In this paper, we study a more realistic scenario where the attacker does not know the underlying dynamics of MDP, and can not directly interact with the environment, either.
Thus, the attacker learns the environment only based on the agent’s experience.
In this paper, we systematically investigate poisoning in RL by considering all the aforementioned
RL-speciﬁc challenges.
Previous works either do not address any of the challenges or only address some of them.
Behzadan & Munir (2017) achieve policy induction attacks for deep Q networks (DQN).
However, they treat output actions of DQN similarly to labels in SL, and do not consider
Challenge II that the current action will inﬂuence future interactions.
Ma et al.
(2019) propose a poisoning attack for model-based RL, but they suppose the agent learns from a batch of given data, not considering Challenge I. Rakhsha et al.
(2020) study poisoning for online RL,
but they require perfect knowledge of the MDP dynamics, which is unrealistic as stated in Challenge III.
Summary of Contributions.
(1) We propose a practical poisoning algorithm called VulnerabilityAware Adversarial Critic Poison (VA2C-P) that works for deep policy gradient learners without any prior knowledge of the environment.
To the best of our knowledge, VA2C-P is the ﬁrst practical algorithm that poisons policy-based deep RL methods .
(2) We introduce a novel metric, called stability radius, to characterize the stability of RL algorithms, measuring and comparing the vulnerabilities of RL algorithms in different scenarios.
(3) We conduct a series of experiments for various environments and state-of-the-art deep policy-based RL algorithms, which demonstrates RL agents’ vulnerabilities to even weaker attackers with limited knowledge and attack budget.
R ELATED W ORK
The main focus of this paper is on poisoning RL, an emerging area in the past few years.
We survey related works of adversarial attacks in SL and evasion attacks in RL in Appendix A, as they are out of the scope of this paper.
Targeted Poisoning Attacks for RL.
Most RL poisoning researches work on targeted poisoning, also called policy teaching, where the attacker leads the agent to learn a pre-deﬁned target policy.
Policy teaching can be achieved by manipulating the rewards (Zhang & Parkes, 2008; Zhang et al., 2009) or dynamics (Rakhsha et al., 2020) of the MDP.
However, they require the attackers to not only have prior knowledge of the environments (e.g., the dynamics of the MDP), but also have the ability to alter the environment (e.g. change the transition probabilities), which are often unrealistic or difﬁcult in practice.
Poisoning RL with Omniscient Attackers.
Most guaranteed poisoning RL literature (Rakhsha et al., 2020; Ma et al., 2019) assume omniscient attackers , who not only know the learner’s model, but also know the underlying MDP.
However, as motivated in the introduction, the underlying MDP is usually either unknown or too complex in practice.
Some works poison RL learners by changing the reward signals sent from the environment to the agent.
For example, Ma et al.
(2019) introduce a policy teaching framework for batch-learning model-based agents; Huang & Zhu (2019) propose a reward-poisoning attack model, and provide convergence analysis for Q-learning; Zhang et al.
(2020b) present an adaptive reward poisoning method for Q-learning (while it also extends to DQN) and analyze the safety thresholds of RL; these papers all assume the attacker knows not only the models of the agent, but also the parameters of the underlying MDP, which could be possible in a tabular MDP, but hard to realize in large environments and modern deep RL systems.
On the contrary, we consider non-omniscient attackers who do not know the underlying MDP or environment in this paper.
The non-omniscient attackers can be further divided into two categories: white-box attackers, who know the learner’s model/parameters, and black-box attackers, who do not know the learner’s model/parameters.
They both tap the interactions between the learner and the environment.
Black-box Poisoning for Value-based Learners.
Although there are many successful black-box evasion approaches (Xinghua et al., 2020; Inkawhich et al., 2020), black-box poisoning in RL is rare.
There is a black-box attacking method for a value-based learner (DQN) proposed by Behzadan & Munir (2017), which does not require the attacker to know the learner’s model or the underlying
MDP.
In this work, the attacker induces the DQN agent to output the target action by perturbing the state with Fast Gradient Sign Method (FGSM)
(Goodfellow et al., 2015) in every step.
However, the data-correlation problem of RL (Challenge II) is not considered, and FGSM attack does not work for policy-based methods due to their high stochasticity, as we show in experiments.
In this paper, we propose a new poisoning algorithm for policy-based deep RL agents, which can achieve both non-targeted and targeted attacks.
We do not require any prior knowledge of the environment.
And our algorithm works not only when the attacker knows the learner’s model (whitebox), but also when the leaner’s model is hidden (black-box).
P ROBLEM F ORMULATION FOR P OISONING O NLINE RL 3. N OTATIONS AND P RELIMINARIES
In RL, an agent interacts with the environment by taking actions, observing states and receiving rewards.
The environment is modeled by a Markov Decision Process (MDP), which is denoted by a tuple M = (cid:104)S , A , P, R, γ, µ (cid:105) , where S is the state space, A is the action space, P is the transition kernel, R is the reward function, γ ∈ (0 , 1) is the discount factor, and µ is the initial state distribution.
A trajectory τ ∼ π generated by policy π is a sequence s 1 , a 1 , r 1 , s 2 , a 2 , · · · , where s 1 ∼ µ , a t ∼ π ( a | s t ) , s t +1 ∼ P ( s | s t , a t ) and r t =
R ( s t , a t ) .
The goal of an RL agent is to ﬁnd an optimal policy π ∗ that maximizes the expected total rewards η , which is deﬁned as
[Equation]
We use an overhead check sign ˇ on a variable to denote that the variable is poisoned.
For example, if the attacker perturbs a reward r t , then the poisoned reward is denoted as ˇ r t .
If a policy π is updated with poisoned observation, then it is denoted as ˇ π .
3. T HE P ROCEDURE OF O NLINE L EARNING AND P OISONING
[Figure]
Procedure of Online Learning.
We consider a classical online policy-based RL setting, where the learner iteratively updates its policy π parametrized by θ , through K iterations with the environment.
For notation simplicity, we omit θ and use π k to denote π θ k , the learner’s policy at iteration k .
The online learning process is described as below.
At iteration k = 1 , · · · , K ,
(1) The agent uses the current policy
π k to roll out observation O k
= ( O s k , O a k , O r k ) from environment M k , where O s k
=
[ s 1 , s 2 , · · · ] , O a k =
[ a 1 , a 2 , · · · ] , O r k =
[ r 1 , r 2 , · · · ] are respectively the sequence of states, actions and rewards generated at iteration k .
The agent updates its policy parameters θ with its algorithm f .
Most policy-based algorithms perform on-policy updating, i.e., update policy only by the current observation
O k .
The on-policy update can be then formalized as π k +
= f
( π k , O k ) ≈ argmax π J ( π, π k , O k ) , where J is an objective function deﬁned by algorithm f , e.g., the expected total reward η ( π ) .
Procedure of Online Poisoning.
A poisoning attacker inﬂuences the learner in the training process by perturbing the training data.
In SL, the training data consists of features and labels, and the attacker poisons the training data before the learning starts.
However, the training data in RL is the trajectories a learner rolls out from the environment, i.e., observation O =
( O s , O a , O r ) .
At iteration k , the attacker eavesdrops on the interaction between the learner and the environment, obtains the observation O k , and may poison it into ˇ O k , then send ˇ O k to the learner before policy updating.
1 Procedure 1 illustrates how the online game goes between the learner and the attacker.
Figure 2 visualizes this online learning-poisoning procedure, where we can see that learning and poisoning are convoluted and inter-dependent.
Procedure 1: Flow of Online RL Poisoning
1 Learner initializes its initial policy π θ2 for k = 1 , · · · , K do
Learner rollouts observation (trajectories)
O k in environment M with current policy π k
Attacker may poison the observation O k to ˇ O k
Learner updates its policy: π k +
=
f
( π k , ˇ O k )
≈ argmax π J ( π, π k , ˇ O k )
3. A U NIFIED F ORMULATION FOR P OISONING O NLINE P OLICY - BASED RL. 3.3. A TTACKER ’ S P OISON A IM
As deﬁned in Section 3.1, the observation is a collection of trajectories, consisting of the observed states O s , the executed actions O a or the received rewards O r .
When poisoning the observation, the attacker may only focus on the states, or on the actions, or on the rewards.
We call the quantity being altered as the poison aim of the attacker, denoted by D ∈ {O s , O a , O r } .
For example, D =
O s means the attacker chooses to attack the states.
Distinguishing different poison aims is important, since in real-world applications different poison aims correspond to different behaviors of the attacker.
For example, in an RL-based recommender system, the RL agent recommends an item (i.e., an action) for a user (i.e., a state), and the user may or may not choose to click on the recommended item (i.e., a reward).
An adversary might manipulate the reward (poisoning D
= O r ), e.g., blocking the user’s click from the RL agent or creating a fake click.
An adversary might also alter the state (poisoning D
= O s ), e.g., raising a teenager user’s age which could result in inappropriate recommendations.
An adversary might also change the action (poisoning D
= O a ), e.g., inserting a fake recommendation into the agent’s list of recommendations.
Under different scenarios, the feasibility of poisoning different aims may vary.
Most existing works on poisoning RL only solve one type of poison aim.
Zhang et al.
(2020b); Huang & Zhu (2019) propose to poison rewards, and Behzadan & Munir (2017) assume the attacker poison the states.
However, in our paper, we provide a general solution for any of the poison aims to satisfy the needs in different scenarios.
Our proposed method also supports a ”hybrid” poison aim, where the attacker could switch aims at different iterations, as discussed in Section 5.
3.3. A P OISONING F RAMEWORK FOR RL
We focus on proposing a poisoning mechanism for the above challenging online learning scenario.
We ﬁrst formalize the poisoning attacking at iteration k as a sequential bilevel optimization problem in Problem (Q), and explain the details of the problem in the remaining of this section.
[Equation]
c [Equation] [Table]
[Equation]
((d
[Table]
[Equation]
(a) Attacker’s Weighted Loss.
L A (ˇ π ) measures the attacker’s loss w.r.t.
a poisoned policy π .
As the deﬁnition of poisoning implies, the attacker inﬂuences or misleads the learner’s policy.
λ k : K are the weights of future attacker losses, controlling how much the attacker value the poisoning results in different iterations.
The goal of the attacker is either (1) non-targeted poisoning , which minimizes the expected total rewards of the learner, i.e., L A = η (ˇ π ) , or (2) targeted poisoning , which induces the learner to learn a pre-deﬁned target policy, i.e., L A
=
distance(ˇ π, π † ) , where distance(ˇ π, π † ) can be any distance measure between a learned policy ˇ π and a target policy π † .
Note that the targeted poisoning objective can usually be directly computed with a prior target policy, while non-targeted poisoning has a “reward-minimizing” objective, which is the reverse of the learner’s objective.
Without any prior knowledge of the environment, non-targeted poisoning is usually more difﬁcult, as the attacker needs to ﬁrst learn “what is the worst way” (which is as difﬁcult as a learning problem by a RL agent) and then lead the learner to that way (which is as difﬁcult as a targeted poisoning problem, assuming leading an agent to different policies is roughly equally challenging).
However, most existing poisoning work focus on targeted poisoning, which requires the attacker to know a pre-deﬁned target policy.
Thus in this paper, we make more efforts to solve the reward-minimizing poisoning problem, which may deprave the policy without any prior knowledge.
(b) Imitate the Policy-Based Learner.
To conﬁdently mislead a learner, the attacker needs to predict how the learner will behave under the poison, which can be achieved by imitating the learner using the learner’s observation .
More speciﬁcally, at the j -th iteration, the attacker estimates the learner’s policy to be ˜ π j , called imitating policy .
Then, the attacker predicts the next-policy ˇ π j +1 under poisoned observation, based on the learner’s update rule argmax π J ( π, ˜ π j , ˇ O j | ˇ D j ) , where D ∈ {O s , O a , O r } stands for the poison aim of the poisoning,
ˇ O| ˇ D denotes that O is poisoned into ˇ O given that poison aim D is poisoned into ˇ D .
However, the imitating policy ˜ π may or may not be the same as the actual learner’s policy, depending on the attacker’s knowledge .
As introduced in
Section 2, we deal with both white-box and black-box attackers, and both of them do not know the environment M .
A white-box attacker knows the current and past observations O 1: k , the learner’s algorithm f and policy π , so it can directly copy the policy ˜
π j = π j , ∀ j .
A black-box attacker knows the current and past observations O 1: k , but does not know the learner’s policy π .
In this case, the attacker has to estimate π at every iteration.
Section 4.3 states how to guess π .
(c,d) Limited-budget and Limited-power.
In practice, the ability of an attacker is usually restricted by some constraints.
For the online poisoning problem, we consider attacker’s constraints in two forms: (1) ( attack budget C ) the total number of iterations that the attacker could poison does not exceed C ; (2) ( attack power (cid:15) ) in one iteration, the total change 2 U ( D k , ˇ D k ) between D k and ˇ D k can not be larger than (cid:15) .
Attack power controls the amount of perturbation, as commonly used in the adversarial learning literature.
Attack budget considers the frequency of attack, which is similar to the constraint studied by Wang & Chaudhuri (2018).
Problem (Q) is a generic formulation, covering a variety of poisoning models, and speciﬁes the best poison an attacker can execute.
However, directly solving Problem (Q) is prohibitive, as (1) the future observations O k
+1:
K are unknown when poisoning the k -th iteration, as the attacker has no knowledge of the underlying MDP.
(2) the limited-budget constraint is analogous to (cid:96) 0 -norm regularization, which is generally NP-hard (Nguyen et al., 2019); and (3) minimizing attacker’s loss while maximizing learner’s gain is non-convex minimax optimization, which is a complex problem (Perolat et al., 2015).
In spite of the above difﬁculties, we introduce a practical method to approximately and effectively solve Problem (Q) in Section 4.
VA2C-P: P OISON P OLICY G RADIENT L EARNERS
In this section, we propose a practical and efﬁcient poisoning algorithm called Vulnerability-Aware
Adversarial Critic Poison (VA2C-P) for policy gradient learners.
Without loss of generality, we assume the loss weights λ j = 1 for all j = 1 , · · · , K .
Main Idea.
As discussed in Section 3.3, Problem (Q) is difﬁcult mainly because of the unknown future observations and the limited budget constraint.
In other words, it is hard to exactly determine
(1) what kind of attack beneﬁts the future the most, and (2) which iterations are worth attacking the most.
Thus, we propose to break Problem (Q) into two decisions: when to attack, and how to attack.
The “when to attack” decision allocates the limited budget to iterations which are more likely to be inﬂuenced by the attacker, and the “how to attack” decision utilizes limited power to minimize the attacker’s loss.
We introduce two mechanisms of VA2C-P, vulnerability-awareness and adversarial critic, to make these two decisions respectively.
4. D ECISION 1: W HEN TO A TTACK – V ULNERABILITY -A WARE
To answer the question of when to attack, we identify the iterations under which the learner’s policy gets more depraved by the same level of attack power.
Inspired by the notion of stability in learning theory, which measures how a machine learning algorithm changes due to a small perturbation of the input data, we formally investigate the stability of an RL algorithm, which is the ﬁrst attempt in the existing literature to the best of our knowledge.
Stability of RL Algorithms.
We ﬁrst focus on a single update process of an algorithm f .
An update π (cid:48)
=
f ( π, O ) is stable if a limited-power poisoning attack does not cause any difference on the output policy π (cid:48) .
That is, the learning algorithm produces the same result regardless of the presence of the poison.
More formally, we deﬁne the concept of stability radius of one update in Deﬁnition 1.
Deﬁnition 1 (Stability Radius of One Update) .
For the update of an RL algorithm π (cid:48)
=
f ( π, O ) , with any poison aim D , the δ -stability radius of the update is deﬁned as the minimum poison power needed to cause δ change in policy (called δ -policy-discrepancy) φ δ, D
( f, π, O ) = inf (cid:15) {
∃ ˇ D s.t.
U ( D , ˇ D ) ≤ (cid:15) and d max [ π (cid:48) ||
ˇ π (cid:48) ] > δ, where ˇ π (cid:48) =
f ( π, ˇ O| ˇ D ) } , (1)
Policy discrepancy d max [ π 1 || π 2 ]
= max s d (cid:2) π 1 ( ·| s )
||
π 2 ( ·| s ) (cid:3) , where d [ ·||· ] could be any measure of distribution distance.
Remarks.
(1) The one-update stability radius is w.r.t.
the algorithm f , the old policy π , the clean observation O and the poison aim D .
(2) Poison with power under φ δ, D ( f, π, O ) will not cause the policy distributions to change more than δ .
(3) As shown by Proposition 2 in Appendix D.1, poison with power under φ δ, D ( f, π, O ) will not make the policy value drop more than O (cid:0) δ 2 γ (1 − γ ) − 2 max s,a | A π (cid:48) ( s, a ) | (cid:1) , where A is the advantage function, i.e., A π ( s, a ) = Q π ( s, a ) − V π ( s ) .
Stability radius measures the minimal effort needed to make the poisoned next-policy ˇ π (cid:48) notably different from the clean next-policy π (cid:48) which the learner will get if no poison is applied.
Assuming max s,a | A π (cid:48) ( s, a ) | does not drastically change for the learner’s policy during training, then an attack that causes higher policy discrepancy between π (cid:48) and ˇ π could cause more drop of the policy value.
Therefore, the idea of vulnerability-aware attack is to estimate the vulnerability of each update and attack the most vulnerable ones.
More speciﬁcally, if the attacker ﬁnds an (cid:15) -powered attack results in a policy discrepancy larger than some threshold δ , then it can conclude φ δ ( f, π, O ) ≤ (cid:15) , and the current update is relatively vulnerable.
In practice, we have the ﬁxed budget C instead of δ , so we perform the vulnerability check in an adaptive way: attacking the C iterations where (cid:15) -powered attacks can trigger the highest policy discrepancies.
Section 4.3 introduces an algorithm to realize this idea.
4. D ECISION 2: H OW TO A TTACK – A DVERSARIAL C RITIC
Since “when to attack” decision tackles the limited-budget constraint, now we turn our attention to minimizing the attacker’s loss while satisfying the limited-power constraint.
If the attacker has already decided to poison the k -th iteration due to its high vulnerability, then the decision of how to attack should be made before the learner uses the observation to update the policy.
We relax the original Problem (Q) into Problem (P) as below.
[Equation]
Compared with Problem (Q), Problem (P) does not consider future losses, which require the unavailable future observations.
Instead, Problem (P) ﬁnds a greedy attack ˇ D k to minimize the loss of the immediate next iteration.
The solution to Problem (P) is always feasible to Problem (Q), although might not be optimal.
Appendix F.2 provides details about the optimality of Problem (Q).
In practice, it is also challenging to estimate L A (ˇ π ) .
For targeted attacking, L A =
distance(ˇ π, π † ) is directly computable with a properly deﬁned distance metric, but for non-targeted attacking, the loss
L A = η (ˇ π ) can not be directly computed, since ˇ π is not the behavior policy.
Although one can use importance sampling to evaluate η (ˇ π ) with the current trajectories generated by the learner’s policy, i.e., E τ ∼ π k − 1 [ π k ( τ ) π k − 1 ( τ ) r
( τ )] , it may suffer from a high variance (Schulman et al., 2015b) when there are few trajectories.
To solve this challenge, we introduce another mechanism, adversarial critic.
Adversarial Critic.
Under poisoning, the value network (if any) held by the learner usually fails to ﬁt the correct value of its policy, since it does not observe the real trajectories generated by its policy.
However, the attacker observes the real trajectories before poisoning, and is able to learn the real values to make the attacking stronger.
Inspired by the Actor-Critic method, we propose to let the attacker learn a value function (network)
˜ V ω with observations of the learner, i.e., the attacker learns a critic of the learner’s current policy π .
Then the attacker can use ˜ V ω to design poisoned observations, directing the learner to a decreasing-value direction, which is called Adversarial Critic .
Then, using importance sampling, the attacker’s loss becomes E s,
[Equation]
∼
[Equation]
( s t ) (cid:1) ] , where G is the discounted future reward (cid:80)
Ti = t γ i − t r t .
4. P OISONING A LGORITHM VA2C-P.
do
Break
Obtain
O
Update
Copy
Compute
Update
Algorithm 2 illustrates how an attacker can poison an online RL learner with our proposed VA2C-P, which is corresponding to Line 4 in Procedure 1 shown in Section 3.2.
More implementation details are in Algorithm 3 in Appendix E.
In an iteration, an attacker ﬁrst ﬁnds a good perturbation for observation O with the current attack power (cid:15) , then estimates how much the output policy will change by computing the policy discrep- ancy between the clean next-policy and the poisoned next-policy.
Then the attacker poisons if the policy discrepancy ranks high in the historical policy discrepancies.
In Line 11, we use projected gradient descent to solve Problem P. Computation details are illustrated in Appendix E.
Algorithm 2 covers both white-box and black-box settings, both of which maintain an imitating policy ˜ π to keep track of the learner’s potential status.
As mentioned in Section 3.3, a white-box attacker knows the learner’s current policy, thus he can directly copy the learner’s parameters to his imitating policy (Line 9), then predict the next-policy the learner would get under different observations.
In contrast, a black-box attacker does not know the learner’s policy, but he can update its imitating policy using the same observation as the learner uses (Line 18, 22).
As claimed and veriﬁed by many black-box attacking methods (Behzadan & Munir, 2017), adversarial attacks are usually transferable, i.e., if the attack works on the imitating learner, then the attack is also likely to work for the real learner.
Note that as assumed by Behzadan & Munir (2017), the black-box attacker knows what RL algorithm the learner is using (e.g., PPO, A2C, etc), so that the black-box attacker computes its estimation for the next-policy in Line 10 and Line 12.
E XPERIMENTS
In this section, we evaluate the performance of VA2C-P by poisoning multiple algorithms on various environments.
We demonstrate that VA2C-P can effectively reduce the total reward of a training agent, or force the agent to choose a speciﬁc policy with limited power and budget.
Moreover,
VA2C-P works for heterogeneous poison aims, and works in both white-box and black-box settings.
Experiment Setup.
We choose 4 policy-gradient learning algorithms, including Vanilla Policy Gradient (Sutton et al., 2000), A2C
(Mnih et al., 2016), ACKTR (Wu et al., 2017) and PPO (Schulman et al., 2017).
And we choose 5 Gym (Brockman et al., 2016) environments with increasing difﬁculty levels: CartPole, LunarLander, Hopper, Walker and HalfCheetah.
All results are averaged overrandom seeds.
The total effort U is calculated by the normalized (cid:96) 2 -distance.
See Appendix G.1 for the expression of U , as well as more hyper-parameter settings.
[Figure]
Reward-minimizing Poisoning.
Baselines.
To the best of our knowledge, there is no existing poisoning algorithm against deep policy-gradient algorithms .
Although some evasion methods (Inkawhich et al., 2020) also work for policy-gradient algorithms, evasion is substantially different from poisoning since it does not inﬂuence the policy.
Therefore, to show the effectiveness of VA2C-P, we compare it with 3 baselines: (1) the normal learning with no poison; (2) a random attacker which randomly chooses C iterations, and perturbs the reward to an arbitrary direction by (cid:15) ; and (3) a simpliﬁed version of our algorithm, called Adversarial Critic Poison (AC-P), which decides “how to attack” in the same way as VA2C-P does, but chooses “when to attack” randomly.
Performance.
We ﬁrst show the reward-minimizing performance of VA2C-P on all three types of poison aims, assuming the attacker knows the learner’s model (white-box attack).
Figure 3 shows the rewards of various learners under different kinds of poisoning methods, with different ratios of budget C to the total number of iterations K .
Compared with random poisoning, our proposed
VA2C-P and the simpliﬁed version AC-P make the reward drop more signiﬁcantly , demonstrating the effectiveness of our “how to attack” decisions made by the Adversarial Critic.
VA2C-P further outperforms AC-P in almost all cases, which implies that our “when to attack” decisions based on
Vulnerability-Awareness work well in practice.
An interesting observation is random poisoning not only does not work well in many cases, but sometimes also facilitates the learner (Figure 3h).
This phenomenon is mainly due to the uncertainty of the environment, as pointed out by Challenge I and
Challenge II in Section 1.
Thus, a good poisoning strategy is important
[Figure]
P r
[Figure]
r ti
[Figure]
a r g e t
[Figure]
c ti
[Figure]
Black-box Poisoning.
Figure 3 demonstrates the performance of VA2C-P when the attacker knows the learner’s model.
But when the learner’s model is not available (black-box attack), VA2C-P could also work as shown in Figure 4a, where one can see that black-box poisoning is still effective, although worse than white-box poisoning due to the lack of knowledge .
Hybrid-aim Poisoning.
The experiments shown in Figure 3 assume that the attacker poisons one of the 3 poison aims, O s , O a or O r .
But if the attacker happens to have access to all of these poison aims, it can do better by performing a “hybrid” poisoning, i.e., at each iteration, evaluate the vulnerability of the algorithm w.r.t.
each poison aim with their corresponding attacker power, then choose the one with the highest vulnerability.
Figure 4b shows that adaptive attacking using a hybrid poison aim could signiﬁcantly outperform attacking using a ﬁxed single poison aim.
Targeted Poisoning.
Baselines.
Although targeted RL poisoning is studied by many works (Huang & Zhu, 2019; Rakhsha et al., 2020), few of them work for deep RL.
Despite that no existing work focuses on deep policy-based methods, we transfer the FGSM-based targeted poisoning method proposed for DQN by Behzadan & Munir (2017) to attack policy-based learners as our baseline.
The attacking method is, for each new state, adding a perturbation to it such that the perturbed state is pushed across the decision boundary and towards the target action.
See Appendix G.2 for details.
Performance.
Figure 4c and 4d respectively show the targeted-attack performance comparison on
CartPole and LunarLander, where the target policies are both “always going to the right”.
The y-axes show the proportion of target actions among all actions taken by the learner.
VA2C-P successfully leads the learner to learn the target policy using all types of poison aims with relatively small attack power.
In contrast, FGSM fails to let the learner select the target action in most cases, even with much larger attack power.
The main reason why FGSM works for DQN Behzadan & Munir (2017) but not for policy-gradient methods is, policy-based agents generally learn stochastic policies, and even though FGSM could perturb the state such that the output probability for the target action is 0.51, the agent will still choose other actions with probability 0.49.
Therefore, FGSM cannot easily perform targeted poisoning against policy-based learners.
More experiment results are in Appendix G.3.
Note that our proposed poisoning method can also be extended to off-policy learners, as discussed in Appendix G.4.
C ONCLUSION AND D ISCUSSION
In this paper, we propose VA2C-P, the ﬁrst generic poisoning algorithm for deep policy-gradient online RL methods, which incorporates heterogeneous poisoning models and does not require any prior knowledge of the environment.
Although the effectiveness of VA2C-P is veriﬁed in a wide range of RL algorithms and environments, we acknowledge that poisoning RL in practice is still challenging, due to the relatively high computational burden and the uncertainty of the environments.
These challenges are also exciting opportunities for work.
Appendix: Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics A A DDITIONAL R ELATED W ORKS
Adversarial Attacks in SL.
There have been many works studying adversarial attacking and defending in the past decade (Huang et al., 2011; Steinhardt et al., 2017).
Poisoning, as an important type of adversarial attacking, has been well-studied in the ﬁeld of machine learning (Biggio et al., 2012; Mei & Zhu, 2015), including deep learning (Mu˜noz-Gonz´alez et al., 2017; Shafahi et al., 2018).
Wang & Chaudhuri (2018) consider online learning, which is similar to ours.
However, they focus on supervised learning, where the attacker knows all the data stream, while in our RL setting, the whole training data stream is not available to the attacker (as Challenge I states).
Evasion Attacks in RL.
Recently, adversarial RL is attracting more and more attention, especially evasion attacks at test-time, as summarized by Chen et al.
Most works consider adversarial perturbations on observations, similar to adversarial examples in supervised learning.
Huang et al.
(2017) ﬁrst show that neural network policies are vulnerable to evasion attacks on states, by generating state perturbation with FGSM.
Lin et al.
(2017) consider the data-correlation problem in RL and an attacker with limited ability, and propose a strategical attack method, which perturbs input states only under certain conditions.
In addition, Gleave et al.
(2019) show that in multi-agent RL, choosing an adversarial policy could also negatively affect the victim agent, which is similar to perturbing a state into a novel one.
There are plenty of following works showing the vulnerability of deep RL policies, even against a limited-power black-box attacker Xinghua et al.
(2020); Inkawhich et al.
(2020), Adversarial attacks can be utilized to train robust policies Pinto et al.
; Pattanaik et al.
There is also a line of work focusing on adversarial examples in path-ﬁnding problems Xiang et al.
; Bai et al.
These evasion attacks are created to fool a well-trained policy, but they can not change the policy itself in training time as poisoning does.
B A P OISONING F RAMEWORK FOR RL
In this section, we establish a generic framework of poisoning in online RL, systematically characterizing its challenges and difﬁculties from multiple perspectives - objective of poisoning, various poison aims, and attacker’s knowledge.
Our in-depth comparison with the SL allows a thorough understanding of the additional vulnerability of online RL systems compared with the well-understood
SL systems.
Our framework also provides a clear context to correctly position prior works in the literature as well as to compare our work with existing works.
Compared to the poisoning framework described by Huang & Zhu (2019), we provide a solution for unifying these poison aims in one attack model in this section.
We consider the online learning scenario, where the RL agent (the learner) does not know the dynamics or rewards of the underlying MDP M with state space S , action space A , transition dynamics
P , rewards R and discount factor γ .
Settings and Notations In online RL, the learner interacts with the environment and collects observations.
The learner’s algorithm, denoted by f , iteratively searches for a policy π parametrized by θ , through K interactions with the environment.
Before learning starts, the learner initializes a policy π 1 .
At each iteration k , the learner uses its previous policy π k − 1 to roll out observations O k from the MDP M .
O k is a concatenation of multiple trajectories , denoted as O k
= ( s k , a k , r k ) , where s k =
[ s 1 , s 2 , · · · ] , a k =
[ a 1 , a 2 , · · · ] , r k =
[ r 1 , r 2 , · · · ]] are respectively the sequence of states, actions, and rewards in iteration k .
Then, with the observations O k , the learner updates its policy by attempting to solve argmax π J ( π, π k − 1 , O k ) , where J is the objective function.
The generated policy by the learner’s algorithm π k = f
( π k − 1 , O k ) 4 does not necessarily achieve the maximization of the objective function.
In this paper, an overhead check sign ˇ on a variable always denotes that the variable is poisoned.
For example, if the attacker changes a reward r t , then the poisoned reward is denoted as ˇ r t .
Poison Objective.
We use L A to denote loss function of the poisoning attack, which the attacker attempts to minimize.
The form of L A is determined by its goal, which falls into one of the two categories, non-targeted and targeted poisoning.
In non-targeted poisoning , the attack poisons a policy π to ˇ π to minimize the learner’s expected rewards.
Therefore the poison objective L A is to minimize the learner’s value η (ˇ π ) .
In targeted poisoning , the attack “teaches” the agent to learn a pre-deﬁned target policy π † .
Therefore the poison objective L A is deﬁned as distance 5 (ˇ π, π † ) .
Most existing poison RL researches focus on targeted poisoning (Ma et al., 2019; Rakhsha et al., 2020), and non-targeted poisoning, although discussed by Huang & Zhu (2019), remains relatively untouched.
poison aims.
To inﬂuence the behaviors of the learner, an attacker could inject poison at multiple locations of the learner’s learning process as detailed in Figure 5.
Part of the reason why poisoning in RL is more challenging than in SL is that it involves more poison aims of poisoning, some of which adapt with the environment, increasing the uncertainty.
[Figure]
Poison Aim I – Poison Observation ( O r , O s ) .
The attacker could manipulate the observation of the learner, i.e., change O into ˇ O .
This may happen when the attacker is able to intercept the communication between the learner and the environment, similar to the man-in-the-middle attack in cryptography.
The attacker could target the rewards, called O r -poisoning , studied by Huang & Zhu (2019); or the states, called O s -poisoning , investigated by Behzadan & Munir (2017).
Poison Aim II – Poison MDP ( M R , M P ) .
An attack could directly change the MDP (environment) that the learner is interacting with, i.e., change M into ˇ M .
For example, a seller could inﬂuence the behaviors of customers by changing the prices of products.
The poison of MDP could be injected at the reward model R or the transition dynamics P , respectively denoted as M R -poisoning (studied by Ma et al.
and M P -poisoning (studied by Rakhsha et al.
The analogy of poison
MDP in SL is to manipulate the underlying data distribution of the training data.
Poison Aim III – Poison Executor E a .
The executor of the learner could be poisoned.
For example, an attacker applies a force to the agent, so that the intended action “north” becomes “northeast”.
Pinto et al.
(2017) train a robust RL agent against the executor poisoner.
Denote this type of poisoning as E a -poisoning .
We show in the Appendix C that E a -poisoning is equivalent to directly changing a stored in the observation O =
( s , a , r , d ) .
Attacker’s Knowledge
At the k -th iteration, what an attacker can do depends on its current knowledge set, denoted by K k .
K k could contain the underlying MDP M , the learner’s algorithm f , the learner’s previous policy models θ 1: k − 1 as well as the previous and current observations O 1: k .
An omniscient attacker knows everything , i.e, K
( O ) k
= {M , f, θ 1: k − 1 , O 1: k } .
Most guaranteed policy teaching literature (Rakhsha et al., 2020; Ma et al., 2019) assume omniscient attacker.
However as motivated in the introduction, it is often unrealistic to exactly know the underlying environment.
We discuss two more realistic setting where the attacker only has limited knowledge as follows.
A monitoring attacker has some information but does not assume knowledge of the underlying
MDP M , i.e., K ( M ) k
= { f, θ 1: k − 1 , O 1: k } .
This is especially relevant in applications where learner’s information is not secure (or even open), or an attacker hacks to steal information from the learner.
Monitoring attacker is similar to the white-box attacker in supervised learning.
A tapping attacker has very limited knowledge and knows the observations only, i.e., K ( T ) k
[Figure]
{O 1: k } .
This is widely applicable since the tapping the communication between the learner and the environment is easy.
Tapping attacker is analogous to the black-box attacker in supervised learning.
Behzadan & Munir (2017) consider a tapping attacker, which observes and manipulates the states but does not know the learner’s parameters.
C T ARGET T YPES OF A TTACKING : M ORE D ETAILED E XPLANATION
[Figure]
C. P OISON O BSERVATION
Consider a deep reinforcement learning algorithm, which uses its old policy to generate a series of observations and updates its policy with the collected observations at every iteration.
The observations are stored in a temporary buffer in the form of ( s , a , r , d ) , where s , r , d are returned by the environment, and a is produced by the policy itself.
An attacker could stay in the middle between the learner and the environment and falsify s , r returned by the environment before the learner receives them.
(It is also possible to alter the terminal state ﬂags d , but it is relatively easier for the learner to detect, and its inﬂuence to the learner is not as large as s and r , so we do not discuss this attack type.)
On the other hand, the attacker may also hack the observation buffer to change s , r .
In both cases, poisoning s and r are called O s -poisoning and O r -poisoning respectively.
Note that in the case of hacking observation buffer, the attacker also has the option to manipulate a in O .
But we do not include this kind of attack in observation poisoning, because not like states and rewards, the actions are taken by the learner, so it is easy for the learner to detect the change of action sequence stored in the buffer.
And we will show in Section C.3 that changing a in O is similar to the case of executor poisoning.
C. P OISON MDP
MDP poisoning can be considered as “changing the reality” for the learner.
For example, an attacker decides to increase the reward for state-action pair ( s, a ) by ∆ at iteration t , it then changes the parameter R k ( s, a ) of the environment to ˇ R k ( s, a )
=
R k
( s, a )
+ ∆ .
As a result, whenever the learner visits ( s, a ) in the current iteration, it receives ˇ R k ( s, a ) as its reward.
A more intuitive example is depicted in Figure 7, where we want to train a mouse to ﬁnd the cheese in a maze.
However, if some bad man adds another piece of cheese in the training environment, the mouse is likely to be misled.
Then, in the test time, the “malicious cheese” is removed, but the mouse still wants to ﬁnd the malicious cheese instead of the original one.
Compared with observation poisoning and executor poisoning, MDP Poisoning is more powerful and more difﬁcult to defend against.
An example of MDP poisoning is given by Rakhsha et al.
(2020), where the attacker can force the learner to learn a target policy with guarantees.
[Figure]
C. P OISON E XECUTOR
In the online RL process, the learner learns by taking actions and getting feedbacks of the actions.
If observation poisoning is viewed as changing the feedbacks, then executor poisoning can be viewed as perturbing the actions taken by the learner.
For example, for an auto-driving agent, one can slightly add some force to the steering wheel, so when the agent takes “steer left 90 degrees”, what actually happens is “steer left 100 degrees”.
In this way, the trained policy is biased.
We now show that the aforementioned attack of changing a in O could be converted to executor poisoning.
We describe the following two scenarios respectively for these two poisoning attacks and show they have equal effects.
For simplicity, we only consider one step of the interaction, which can be simply extended to multiple steps.
(1) Poisoning a in O .
For one-step experience, the learner observes state s , takes action a i , receives reward r i
=
R ( s, a i ) and observes a new state s (cid:48)
i ∼ P ( ·| s, a i ) , then the tuple ( s, a i , r i , s (cid:48) i ) will be stored in the observation buffer.
Now the attacker may change the action a i to a j , and the tuple becomes ( s, a j , r i , s (cid:48) i ) .
Finally, the learner regard r i , s (cid:48) i and the following observations as caused by a j , instead of a i .
(2) Poisoning Executor.
In the same environment, suppose the learner observes state s , and takes action a j , then the attacker conducts executor poisoning and changes a j to a i , then the reward and the next state will also change to r i
=
R ( s, a i ) and s (cid:48)
i ∼ P ( ·| s, a i ) .
The learner does not know the falsiﬁcation of action and stores ( s, a j , r i , s (cid:48) i ) to the buffer.
Finally, the attacker will take r i , s (cid:48)
i and the future observations as caused by a j , and updates its policy accordingly.
Based on the descriptions, we can see changing a i to a j in the buffer is equivalent to changing a j to a i on the executor.
If we assume the policy always has non-zero probability on all actions, then for any manipulation on a in O , it is possible to achieve the same effects by attacking the executor.
Hence, given the fact that the former attack can be trivially defended, we only discuss executor poisoning in our poisoning framework.
D R OBUSTNESS AND S TABILITY OF RL A LGORITHMS D. M EASURING V ULNERABILITY OF RL A LGORITHMS
Deﬁnition 1 in Section 4 measures the stability of one update based on policy discrepancy.
But it is not obvious whether the rewards change drastically due to the attacks.
Proposition 2 provides a guarantee on the performance of the poisoned policy ˇ π (cid:48) , compared with the un-poisoned new policy π (cid:48) .
Proposition 2.
For an update of stochastic policy π (cid:48)
=
f ( π, O ) , if its δ -stability radius is ε with the total variance measure, then any poisoning effort smaller than ε on D will cause the expected total reward drop ( η ( π (cid:48) ) − η (ˇ π (cid:48) ) ) by no more than
[Equation]
where γ is the discount factor, g π ( s ) : = (cid:80) ∞ t =0 γ t P ( s t = s | π ) is the discounted visitation frequency, and A is the advantage function, i.e.,
A π ( s, a )
=
Q π ( s, a ) − V π ( s ) .
Proposition 2 shows that if the attack power is within the stability radius, the reward of the poisoned policy will not be inﬂuenced too much.
which also explains the motivation behind our proposed vulnerability-aware attack.
The proof is in Appendix D.4.
With Deﬁnition 1, the one-update stability measure, we are able to formally deﬁne the stability radius of an RL algorithm w.r.t.
an MDP.
Deﬁnition 3 (Stability Radius w.r.t an MDP) .
The δ -stability radius of an algorithm f in an MDP M is deﬁned as the minimal stability radius of all observations drawn from the MDP, and all possible policies in policy space Π .
If f is on-policy, then φ ( f, M ) = min π ∈ Π , O∼ π φ ( f, π, O ) ; if f is off-policy, then φ ( f, M ) = min π ∈ Π , O∼ π φ ( f, π, O ) , where π is the behavior policy;
D. R OBUSTNESS R ADIUS
Different with poisoning, test-time evasion (adversarial examples) misleads the agent by manipulating the states only, since the agent no longer learns from interactions and feedbacks.
Note that although Gleave et al.
(2019) propose an attack called “adversarial policy”, the perturbation does not happen in the policy, but still happens in the input states (observations) of the agent.
To study how robust a trained policy is, we deﬁne the robustness radius with regard to both a single state and the whole environment.
Deﬁnition 4 (Robustness Radius of Policy w.r.t.
a State) .
The robustness radius of a deterministic policy π on a state s is deﬁned as the minimal perturbation of s which changes the output action, i.e.,
[Equation]
Similarly, for any 0 < δ < 1 , the δ -robustness radius of a stochastic policy π on a state s is deﬁned as the minimal perturbation of s which makes the output action distribution disagrees with the original π ( s ) with probability more than δ , i.e.,
[Equation]
where d ( ·|· ) could be any distance measure between two distributions.
Remark.
If we regard the policy as a classiﬁer which “classiﬁes” a state to an action, then the robustness radius of policy deﬁned above is analogous to the robustness radius of classiﬁers deﬁned by Wang et al.
(2017), with an extension to stochastic predictions.
Deﬁnition 5 (Robustness Radius w.r.t an MDP) .
The ( δ -)robustness radius of a policy π in an MDP
M is deﬁned as the maximal robustness radius of all states, i.e., ρ ( π, M ) =
min s ∈S ρ ( π, s )
Remarks.
(1) A deterministic policy is robust against any state perturbation smaller than ρ ( π, M ) .
(2) For a stochastic policy, if its δ -robustness radius in an M is ε , then any state perturbation within ε will cause the expected total reward drop by no more than
[Equation]
which is proven by Theorem 5 in Zhang et al. (2020a).
D. V ULNERABILITY C OMPARISON : D IFFERENCE B ETWEEN SL AND RL
To shed some light on understanding adversarial attacks in RL, we compare SL and RL in terms of their vulnerability to poisoning and adversarial examples.
At test time, a policy network receives states as input, and returns probabilities of choosing each actions as output; a value network receives states (or state-action pairs) as input, and returns the corresponding value as the output.
Thus, test-time RL systems are very similar to SL systems, as one can view the policy networks as classiﬁcation networks, and value networks as regression networks.
However, the key difference between evasion in RL and evasion in SL is, data samples are not independent in RL.
A single adversarial example in SL test dataset may cause at most one misclassiﬁcation instance, whereas an adversarial example in RL may case a drastic change of the gained rewards (e.g., by leading the agent to a ”devastating” or ”absorbing” state).
At training time, SL systems and RL systems are signiﬁcantly different, as Figure 5 shows.
Even when the supervised learner also learns from data streams in an online manner, the training data are independent with the learner’s classiﬁer.
In contrast, the distribution of training data samples changes as the learner updates its policy.
Poisoning attacks against an SL system could alter the decision boundary, so that the learner makes wrong decisions for certain data samples.
For an RL system, poisoning attacks could (1) alter the decision boundary so that the learner chooses bad actions for certain states, and also (2) change the following observations and interactions due to a different selection of action.
In summary, an adversarial attacker may cause higher damages on RL systems than on SL systems, with the same power and budget.
But it does not suggests attacking RL systems is easier than attacking SL systems.
As every coin has two sides, the high uncertainty of the environment may help an attack reduce the learner’s reward, but may also lead the learner to gain higher reward in the future (as shown in Section 5).
Therefore, it is more challenging to successfully attack RL systems than SL systems with a speciﬁc goal.
D. P ROOF OF P ROPOSITION
Proof.
According to the deﬁnition of δ -stability radius, for any poisoning effort within ε , the poisoned policy satisﬁes D max TV [ π (cid:48) ||
ˇ π (cid:48) ] ≤ δ (assume total variance D TV is the distance measure between policy distributions).
We are interested in the difference of expected rewards η ( π (cid:48) )
− η (ˇ π (cid:48) ) .
Deﬁne
[Equation]
( s, a ) , where g is the discounted state visitation frequencies,
i
[Equation]
e.,
[Equation]
Since D max TV [ π (cid:48)
||
ˇ π (cid:48) ] ≤ δ , follow Theorem 1 in paper (Schulman et al., 2015a) [Equation] we can get
[Equation]
So we have
[Equation]
which can be transformed to
[Equation]
We upper bound the term − (cid:80) s ∈S g π (cid:48) ( s ) (cid:80) a ∈A ˇ π (cid:48) ( a | s ) A π (cid:48)
( s, a ) as below. − (cid:88) s ∈S g π (cid:48) ( s ) (cid:88) a ∈A ˇ π (cid:48) ( a | s )
A π (cid:48)
( s, a ) =
(cid:88) s ∈S g π (cid:48)
( s ) (cid:0) − (cid:88) a ∈A
ˇ π (cid:48) ( a | s )
A π (cid:48)
( s, a ) (cid:1) =
(cid:88) s ∈S g π (cid:48)
( s ) (cid:0) − (cid:88) a ∈A
ˇ π (cid:48) ( a | s )
A π (cid:48)
( s, a ) + (cid:88) a ∈A π (cid:48)
( a | s )
A π (cid:48)
( s, a ) − (cid:88) a ∈A π (cid:48)
( a | s )
A π (cid:48)
( s, a ) (cid:1) =
(cid:88) s ∈S g π (cid:48)
( s ) (cid:0) (cid:88) a ∈A
A π (cid:48) ( s, a )( π (cid:48) ( a | s ) − ˇ π (cid:48) ( a | s )) (cid:1) − (cid:88) a ∈A π (cid:48) ( a | s ) A π (cid:48)
( s, a ) (cid:1) ≤ (cid:88) s ∈S g π (cid:48) ( s ) (cid:0) 2 δ max a ∈A | A π (cid:48) ( s, a ) | − (cid:88) a ∈A π (cid:48) ( a | s ) A π (cid:48) ( s, a ) (cid:1)
[Equation]
Combining the above results, we obtain
[Equation]
E A LGORITHM
Assume the learner’s policy π is parametrized by θ .
How to solve the projected gradient descent.
To solve (P), assume ∂θ k ∂ ˇ r exists, one can use projected gradient descent to update r by using the chain rule:
[Equation]
For Vanilla Policy Gradient (VPG) whose update rule is, θ k
= θ k − 1 + α ∇ θ k − 1 ˆ η ( π θ k − 1 , r ) , where
[Equation]
we can derive
[Equation]
(12) and [
[Equation]
Although ∂θ k ∂ ˇ r has a closed-form expression for simple learners like VPG, analytically computing how the poisoned reward inﬂuences the model is challenging for more complicated learners like
PPO, whose update rule is an argmax function.
Therefore, we use the Direct Gradient Method proposed by Yang et al.
(2017) to approximate the gradient by
[Equation]
[Equation]
To show the concrete poisoning process, we assume D =
O r .
Then Algorithm 3 shows the detailed procedure of VA2C-P with a non-targeted goal and a white-box attacker.
For Targeted Poisoning.
In line 16, instead of computing ∂η ∂ ˇ r ( i ) t , we compute ∂ dist ( θ (cid:48) ,θ † ) ∂ ˇ r ( i ) t .
For Black-box Poisoning.
In line 7, instead of getting the learner’s policy model θ k − 1 , we train a policy with the same algorithm of the learner ˜ θ k − 1 ← f (˜ θ k − 2 , O k − 1 ) .
F T HEORETICAL I NTERPRETATION OF THE B I - LEVEL O PTIMIZATION P ROBLEM
In this section, we discuss the problem relaxation made in Section 4.2.
For notation simplicity, we focus on the case D
=
O r .
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Break
Compute ψ k = 1 NT (cid:80) s ( i ) t
d ( π θ k
||
π ˇ θ k ) and add ψ k to Ψ if ψ k is larger than the (cid:98) ( C − c ) /
( K − k ) (cid:99) -th largest element in Ψ then
Attack: replace r with ˇ r in O k and send it back to the learner c ← c +25 Procedure Update( θ, r )
Perform an update with the learner’s algorithm θ (cid:48)
← f ( θ, r )
Compute the attacker’s objective
[Table]
[Equation]
[Table]
return θ (cid:48) , η F. P ROBLEM F ORMS
Suppose the attacker has budget C = K .
Then following the format of Problem equation Q, we could deﬁne the original O r -poisoning optimization problem for the poisoning at the k -th iteration as Problem equation P ∗ .
Note that for notation simplicity, we use the policy parameter θ to denote the policy π θ , so that η ( θ ) :
= η ( π θ ) .
[Equation]
Note that we only optimize on ˇ r k to ˇ r K , since previous k − 1 decisions have already been made at the k -th iteration.
Although an omniscient attacker is able to predict all the observations and solve Problem equation P ∗ directly, for the non-omniscient attacker that we focus on, Problem equation P ∗ is not solvable because of the unknown observations in iteration k + 1 to K .
Hence, as discussed in Section 4, we relax equation P ∗ , a multi-variable optimization problem, to ( K − k + 1) sequential
[Equation]
P
[Equation]
and all the way to argmin
[Equation]
where r j ∼ θ j − 1 , ∀ j = k, · · · , K .
Note that r k is already generated and known for all the above problems.
When solving Problem ( P j ), r j and θ j are also known (determined by θ j − 1 ).
F. T IGHTNESS OF P ROBLEM R ELAXATION
Problem equation P ∗ is a ( K − k + 1) -variable optimization problem with ( K − k + 1) equality constraints and ( K − k + 1) inequality constraints.
And Problem equation P k , · · · , equation P K are ( K − k + 1) single-variable optimization problems respectively with 1 equality constraints and 1 inequality constraints.
The two sets of problems are naturally equivalent if θ k , · · · , θ K , as well as the constraints are independent.
However, due to the online learning process (Figure 6), θ k +1 and r k +1 are all dependent on θ k
and ˇ r k , which makes the relaxation not necessarily optimal.
We call the optimal solution to Problem equation P ∗ as (ˇ r ∗ k , · · · , ˇ r ∗ K ) , and the optimal solutions to
Problem equation P k to equation P K as ˇ r # k , · · · , ˇ r # K .
For simplicity, we assume the environment and the policies are all deterministic.
So that the value of observed reward r j is a deterministic function of θ j − 1 .
We claim that the relaxation does not change the feasibility as stated in Proposition 6.
Proposition 6.
(ˇ r # k , · · · , ˇ r # K ) is a feasible solution to Problem equation P ∗ .
Proof.
Note that r k is known and the same for both equation P ∗ and equation P k .
(1) ˇ r # k is feasible to equation P ∗ because it satisﬁes (cid:107) ˇ r # k − r k (cid:107) ≤ (cid:15) .
For Problem equation P ∗ , with θ k = f
( θ k − 1 , ˇ r # k ) , r k +1 is the same with the r k +1 in Problem P
k +1 .
Since the optimal solution to equation P k +1 , ˇ r # k +1 , satisﬁes (cid:107) ˇ r #
k
+1 − r k +1 (cid:107) ≤ (cid:15) , ˇ r # k +1 is also feasible to equation P ∗ .
(3) By induction, { ˇ r # j } Kj = k all satisfy the constraints in equation P ∗ at the same time, so (ˇ r # k , · · · , ˇ r # K ) is a feasible solution to Problem equation P ∗ .
Note that if the environment or the policy is stochastic, then r j is a random variable sampled from some distribution deﬁned by θ j − 1 .
In this case, the constraints for Problem equation P ∗ should be
Pr( (cid:107) ˇ r j − r j (cid:107) ≤ (cid:15) ) ≥ t, ∀ j = k, · · · , K , where t ∈ [0 , 1] is a threshold probability.
Then, with appropriate t , Proposition 6 also holds.
So far we have shown that the relaxation is feasible, so that the attacker will not plan for a nonfeasible poisoning with VA2C-P. Next, we discuss the optimality of the relaxation.
We ﬁrst make the following assumptions.
Assumption 1 : the learner’s update function f ( θ, ˇ r ) is differentiable w.r.t.
ˇ r and θ .
Assumption 2 : the environment and the policies are all deterministic, and the reward r generated by a policy π θ is differentiable w.r.t.
θ (i.e. R ( s, π θ ( s ) is differentiable w.r.t.
θ ).
Proposition 7.
If Assumption 1 and 2 hold, the necessary condition for (ˇ r # k , · · · , ˇ r # K ) being an optimal solution to Problem equation P ∗ is, for all j = k, · · · , K , [Equation]
[Equation]
Proof.
Consider the case K = 2 for simplicity, and the results naturally extend to a larger K .
At iteration 1, Problem equation P ∗ becomes
[Equation]
where θ 0 and r 1 are known.
The relaxed problems are
[Equation]
where θ 0 and r 1 are known, and
[Equation]
where θ 1 and ˇ r 2 are determined by ˇ r 1 , the solution to equation P 1 .
Suppose ˇ r #1 and ˇ r #2 are the optimal solutions to equation P 1 and equation P 2 .
And we discuss the necessary condition for ˇ r #1 and ˇ r #2 being optimal to equation P 0 .
We can rewrite equation P 1 as
[Equation]
And the Lagrange function of the above problem is
[Equation]
The necessary conditions for ˇ r 1 being optimal are
[Equation]
[Equation]
for some λ 1 and y 1 .
Similarly, for Problem equation P 2 , the necessary conditions of optimality are
[Equation]
for some λ 2 and y 2 . Since ˇ r #1 and ˇ r #2 are the optimal solutions to equation P 1 and equation P 2 , ˇ r #1 and ˇ r #2 satisfy
Equation equation 17 ∼ equation 22.
Expanding equation 17 and equation
[Equation]
we get
[Equation]
For Problem equation P
[Equation]
the Lagrange is
[Equation]
[Equation]
And the necessary conditions for ˇ r 1 , ˇ r 2 being optimal are ∂η ( f ( θ 0 , ˇ r 1 ))
[Equation]
∂η [Equation] λ
[Equation]
y
[Equation]
= [Equation] [Equation] 31) for some λ (cid:48) 1 , λ (cid:48) 2 , y (cid:48) 1 , y (cid:48) 2 (not the same with λ 1 , λ 2 , y 1 , y 2 ).
Expanding equation 26 and equation
[Equation]
we get
[Equation]
Combining equation 23, equation 24, equation 32 and equation 33, we obtain
[Equation]
for some ζ , which is the necessary condition for ˇ r #1 and ˇ r #2 being optimal to Problem equation P 0 .
Intuitively, this condition implies that the gradient of η ( θ 1 ) w.r.t.
ˇ r #1 (the RHS) should be aligned with the gradient of η ( θ 2 ) w.r.t.
ˇ r #1 (the LHS), without considering the inﬂuence of ˇ r 1 to ˇ r 2 .
That is, although ˇ r 1 inﬂuence θ 1 , θ 1 inﬂuences both θ 2 and ˇ r 2 (because ˇ r 2 is generated by π θ 1 ), LHS does not include ∂η ∂θ 2 ∂θ 2 ∂ ˇ r 2 ∂ ˇ r 2 ∂θ 1 ∂θ 1 ∂ ˇ r 1 , which makes equation 34 computable in many cases.
However, for the setting of our VA2C-P (monitoring attacker for online RL), the attacker at iteration k
= 1 does not know the observed reward r 2 of iteration k = 2 , which prevent the agent from conducting the optimal attack.
But equation 34 could help the attacker verify whether a past poison is likely to be optimal for the iterations so far.
For example, if the learner is using VPG, then at iteration k = 2 , the attacker can test whether its previous poison ˇ r 1 did a good job in minimizing η
( θ 1 ) + η ( θ 2 ) by evaluating whether ( I + ∇ 2 θ 1 η ( θ 1 )) ∇ θ 2 η ( θ 2 ) is equal to ∇ θ 1 η ( θ 1 ) .
G E XPERIMENT S ETTINGS AND A DDITIONAL R ESULTS G. D ETAILED E XPERIMENT S ETTINGS
Network Architecture.
For all the learners, we use a two-layer policy network with Tanh as the activation function, where each layer has 64 nodes.
PPO, A2C and ACKTR also have an additional same-sized critic network.
We implement VPG and PPO with PyTorch, and the implementation of
A2C and ACKTR are modiﬁed from the project by Kostrikov (2018).
Hyper-parameters.
In all experiments, the discount factor γ is set to be 0.99.
We run VPG and
PPO for 1000 episodes on every environment, and update the policy after every episode.
For A2C and ACKTR, we use 16 processes to collect observations simultaneously, and update policy everysteps (i.e., each observation O has 80 ( s , a , r ) tuples); learning last for 80000 steps in total.
Distance Measure for Perturbation
The deﬁnition of total effort function U ( · ) plays an important role of understanding the attack power.
Since states, actions and rewards are in different forms and scales, we deﬁne U differently for various poison aims.
Also, note that O is a concatenation of state-action-reward tuples, and its length could vary in different iterations, so we normalize over the length of observation.
For D =
O s , we deﬁne the total effort as
[Equation]
For D =
O a , if the action space is continuous, we deﬁne the total effort as
[Equation]
and if the action space is discrete, we deﬁne the total effort as
[Equation]
For example, in CartPole, the action is either 0 or 1, then the attacker with (cid:15)
= 0 .
1 can ﬂip up to 10% of the actions in one iteration.
For D =
O r , we deﬁne the total effort as
[Equation]
In the supplementary materials we provide the code and instructions, as well as demo videos of poisoning A2C in the Hopper environment, where one can see under the same budget constraints, random poisoning has nearly no inﬂuence the agent’s behaviors, while our proposed VA2C-P successfully prevents the agent from hopping forward.
G. FGSM- BASED P OISONING
The procedure of the FGSM-based targeted poisoning is as follows.
We transfer the method proposed by Behzadan & Munir (2017) from attacking DQN to attacking policy-based algorithms.
Although Behzadan & Munir (2017) assume a black-box setting, we hereby use white-box attack (attacker knows learner’s policy parameters) in order to let FGSM be a stronger baseline.
For step t ,
Step 1: the learner observes s t takes action a t , the environment returns reward r t , state s t +1 ; Step 2: the attacker queries the target policy and gets a adv = π † ( s t +1 ) ; Step 3: the attacker poisons s t +1 by
; Step 4: the attacker sends ˇ s t +1 as s t +1 to the learner.
G. A DDITIONAL E XPERIMENT R ESULTS
Figure 8 shows additional results on various environments and RL algorithms, including both nontargeted poisoning and targeted poisoning
[Figure]
i
[Figure]
r
[Figure] [Figure] [Figure] [Figure]
[Figure] =
, (cid:15)
[Figure]
G. E XTENSION TO O FF - POLICY L EARNERS
Although we focus on on-policy policy gradient learners in this paper, our poisoning method is also applicable to off-policy learners which update their policies using sampled mini-batches from all historical observation (trajectories).
If the adversary can manipulate the mini-batch the learner samples at every step, our proposed poisoning process works as usual.
We implement this idea and test it for one of the state-of-the-art off-policy learning method SAC (Haarnoja et al., 2018), and the results are shown in Figure 9, where VA2C-P signiﬁcantly reduces the reward gained by the learner.
In the other case, if the adversary doesn’t see which mini-batch the learner samples but has access to the buffer, he can still alter or insert some samples to inﬂuence learning.
[Figure]