Controllable Attack and Improved Adversarial Training in Multi-Agent Reinforcement Learning
Abstract
Deep reinforcement learning policies have been shown vulnerable to adversarial attacks due to the inherent frangibility of neural networks.
Current attack methods mainly focus on the adversarial state or action perturbations, where such direct manipulations to a reinforcement learning system may not always be feasible or realizable in the real-world.
In this paper, we consider the more practical adversarial attacks realized through actions by an adversarial agent in the same environment.
It has been shown, in prior work, that a victim agent is vulnerable to behaviors of an adversarial agent who targets to attack the victim, at the cost of introducing perceivable abnormal behaviors for the adversarial agent itself.
To address this, in the first part of this paper, we propose to constrain the state distribution shift caused by the adversarial policy and offer a more controllable attack scheme by building connections among policy space variations, state distribution shift, and the value function difference.
To provide provable defense, we revisit the cycling behavior of common adversarial training methods in Markov game, which has been a well-known issue in general differential games including Generative Adversarial
Networks (GANs) and adversarial training in supervised learning.
We propose to fix the non-converging behavior through a simple timescale separation mechanism.
In sharp contrast to general differential games, where timescale separation may only converge to stationary points, two-timescale training methods in Markov games can converge to the Nash Equilibrium (NE).
Using the Robosumo competition experiments, we demonstrate the controllable attack is much more efficient in the sense that it can introduce much less state distribution shift while achieving the same winning rate with unconstrained attack.
Furthermore, in both Kuhn Poker and Robosumo competition, we verify that the rule of timescale separation leads to stable learning dynamics and less exploitable victim policies.
Introduction
Despite the huge success of deep reinforcement learning (RL) algorithms across various domains [22, 16, 20], it has been shown that deep reinforcement learning policies are highly vulnerable to adversarial attacks.
The most popular attack methods focus on fooling the RL agent by adversarially perturbing the states or observations.
The success of such attacks can be attributed to lack of adversarial robustness in the deep neural network which is used to parametrize the value functions/Qfunctions or the policies of the deep RL algorithms.
One of the earliest work by [9] showed a detailed study of various adversarial attacks on the neural network policies and developed threat models capable of significantly degrading the test-time performance of trained policies with minimal perturbation.
[13] emphasized on enhancing the efficiency of the adversarial attack by minimizing the agent’s reward by only attacking the agent at a small subset of time steps in an episode and in the most significant times.
There has been subsequent research in developing more efficient pixel-based attacks [19, 18, 17].
[26] built the theoretical framework SA-MDP for adversarial state perturbation and proposed the corresponding regularizer for more robust reinforcement learning policies.
Subsequent work [24] improved [26] with the framework of PA-MDP for better efficiency.
However, perturbing states or observations of an agent might not always be feasible/practical in reallife scenarios such as self-driving cars and robotic manipulation, where manipulating the observations may require attacking the communication systems which is hard to achieve.
Consequently, in this paper, we consider the attacks realizable through actions of an adversarial agent in the same environment as the victim.
Along this direction, [6] follows a principled approach towards attacking well-trained RL agents by training adversarial policies of an opponent agent to minimize the expected return of the victim.
The adversarially trained policy efficiently defeats the state-of-the-art agents trained via self-play [2] despite the adversarial policy being trained for less than 3% of the training time steps.
In zero-sum games, the goal to maximize the expected return of the attacker coincides with minimizing the expected return of the victim.
For games, which are not strictly zero-sum, [7] proposed to maximize the expected return of the attacker and minimize the expected return of the victim simultaneously and improved the attack efficiency.
If the attacker is allowed to access the observations and actions of the victim, [25] showed that fitting the environment transition model and the victim policy model can help improve the attack success rate.
However, while seeking the policy to degrade the performance of the victim, the attacker is usually optimized in an unconstrained fashion thereby resulting in adversaries demonstrating abnormal behaviours [6], which could be easily detected by human eyes and thus not consistent with the spirit of imperceivable adversarial examples in supervised learning.
To address this, we propose to constrain the state distribution shift caused by the adversarial policy and offer a more controllable attack scheme by building connections among policy space variations, state distribution shift, and the value function difference.
On the other hand, to promote robustness of the victim policy, we investigate the adversarial training methods and verify the diverging behavior in Markov games, which has been a known issue in general differential games including Generative Adversarial Networks (GANs) [3, 8, 5] and adversarial training in supervised learning.
To fix this, we introduce a simple timescale separation mechanism, which is guaranteed to converge to Nash Equilibrium (NE) with tabular parameterization.
To summarize, this work makes contributions to both attack and defense in the multi-agent reinforcement learning.
For attack, we propose a controllable attack framework to mitigate the large state distribution shift introduced by the unconstrained attack.
For defense, we revisit the non-converging behaviors of the adversarial training even in Markov games and introduce the simple rule of timescale separation, which enjoys both strong theoretical guarantee (converging to NE standing in contrast to local convergence in differential games) and leads to stable training dynamics and much less exploitable victim policy.
Preliminaries
The extension of Markov decision processes (MDPs) with more than one agents is commonly modelled as Markov games [14].
A Markov game with N agents is defined by a tuple <
N, S , {A i } Ni =1 , P, { r i } Ni =1 , ρ, γ > , where S denotes the state space and A
i is the action space for agent i .
The function P controls the state transitions by the current state and one action from each agent: P : S ×
A 1 × · · · × A N → ∆( S ) , where ∆( S ) denotes the set of probability distributions over the state space S .
Given the current state s t and the joint action ( a 1 , . . .
, a N ) , the transition probability to s t +1 is given by P ( s t +1 | s t , a 1 , . . .
, a N ) .
The initial state is sampled from the initial state distribution ρ ∈ ∆( S ) .
Each agent i also has an associated reward function r i : S
×
A
i × · · · × A N → [0 , 1] .
Each agent’s goal is to maximize the γ -discounted expected return
R i =
E [ (cid:80) ∞ t =
γ t r i ( s t , a ti , a t − i )] , where − i is a compact representation of all complementary agents of i .
In the following discussions, we mainly focus on the two-player zero-sum games, where we have two agents labeled as α and µ and the reward satisfies r α + r ν = 0 .
In Markov games, each agent is equipped with a policy π
i : S → ∆( A i ) .
The corresponding policy class is denoted as Π i and the joint policy is defined as π ( a | s ) =
Π Ni =
π i ( a i | s ) .
The value function for the two-player zero-sum game given joint policy ( π α , π ν ) is defined by V s ( π α , π ν ) =
E π α ,π ν [ (cid:80) ∞ t =0 γ t r α ( s t , a t ) | s 0 =
s ] , where player α attempts to maximize the value function and player ν minimizes this value.
We abuse the notation V ρ ( π α , π ν ) :
=
E s ∼ ρ [ V ρ ( π α , π ν )] .
We further define state visitation, which reflects how often the policy visits different states in the state space.
Definition 1.
(Stationary State Visitation) Let d π ∈ ∆( S ) denote the normalized distribution of state visitation by following the joint policy π in the environment:
[Equation]
Controllable Adversarial Attack
The authors of [6] consider the more realistic attack model in which attacks are realized through actions of an adversarial agent in the same environment as the victim.
Although principled, the attacking formula may be too strong as the attacks may no longer be stealthy or imperceptible.
People may argue that once the attacks are non-stealthy, they can be easily recognized and thus no longer be harmful.
Here, we make one essential hypothesis that humans usually detect the presence of adversarial behaviors by tracking the states the multi-agent systems.
Therefore, for the adversarial agent to hide the adversarial behaviors from being detected, we propose to control the perturbation of the system states.
Formally, given a multi-agent system with policies ( π α , π ν ) , we offer our attack objective as the following optimization problem.
[Equation]
The objective (2) is a common attack objective, where the adversarial player aims to minimize the value of the victim.
Apart from the objective, we propose further to constrain the behavior of the adversarial policy π ′ ν so that the induce state visitation of d π α ,π ′ ν stays close to the original d π α ,π ν .[-
-]
However, it is generally difficult to estimate the state visitation for given policies, let alone enforce the constraints in (3).
Therefore, in the remaining section, we will try to find an upper bound for the discrepancy between state visitation.
The key idea is to show that the distance between state visitation can be upper bounded by the distance between policies.
To do so, we firstly define the distance between two policies as follows
Definition 2 (Attack Budget) .
We define the maximum total variation distance D maxTV between two policies as: D maxTV ( π || π ′ )
=
max s D TV ( π ( ·| s )
|| π ′ ( ·| s )) , where D TV ( p || q ) = 12 (cid:80)
i | p i − q i | .
This definition gives a more straight-forward way to control the variation of the policy at every state s .
However, it is unclear how this definition relates to the total variation between state visitations.
We will answer this in the following theorem.
Theorem 1.
For two victim-attacker policy pairs ( π α , π ν ) and ( π α , π ′ ν ) , where D maxTV ( π ν || π ′ ν ) ≤ ϵ , the difference between the state visitation of ( π α , π ν ) and ( π α , π ′ ν ) can be bounded as: ||
d π α ,π ν − d π
[Equation]
This theorem shows a relationship between D max TV ( π ν || π ′ ν ) and || d π α ,π ν − d π α ,π ′ ν || 1 : to control the variation of the state visitation, it suffices to control the policy variation at each state s .
Meanwhile, with the state distribution variation controlled, we derive the following robustness certificate:
Corollary 1 (Robust Certificate) .
Under the same condition of Theorem 1, the following value difference holds | V ρ ( π α , π ν ) − V ρ ( π α , π ′ ν ) | ≤ [Equation]
[Equation]
This corollary suggests that the value function in Markov games naturally satisfies a Lipschitz condition, where the attacker must have enough budget to achieve large performance degradation, standing in sharp contrast with the adversarial attack in supervised learning, where small perturbations suffices to make huge changes to the final outputs.
Formally, this tells us the performance drop is at most
[Equation] the attacker has the attack budget ϵ .
A parameterization that converts the constrained problem to unconstrained.
As shown in
Theorem 1, policy variation upper-bounds the state distribution variation.
Consequently, rather than directly constrain state visitation distribution, we constrain policy variation D maxTV ( π ν
|| π ′ ν ) , which can be realized more efficiently and sufficiently.
We use the idea from conservative policy iteration [10] and convert the constrained problem to the unconstrained by using the following parameterization.
Definition 3 ( ϵ coupling) .
The ϵ coupled policy is constructed for each state s ∈ S
[Equation]
where π ν is the original attacker, which is fixed during attack and π ν ( θ ) parameterized by θ is what the adversary tries to optimize.
With ϵ coupling, it is easy to verify the following proposition.
Proposition 1.
With π ′ ν defined as (4) , it holds for all θ that D maxTV ( π ν || π ′ ν ) ≤ ϵ.
Therefore, the objective (2) can be replaced by min θ V ρ ( π α , (1 − ϵ ′ )
π ν + ϵ ′
π ν ( θ )) , where ϵ ′ = 1 − γ 2 γ ϵ according to theorem 1 and proposition 1.
Improved Adversarial Training with Timescale Separation
Considering the vulnerability of victim policy, effective defense strategies are in urgent need.
There have been prior work on utilizing the attack for re-training to ensure the robustness of the victim policy [6, 7, 25].
However, it has been shown that although re-training against a specific adversarial policy provides robustness against the specific attacker, the performance against benign policies is actually degraded.
Therefore, it is important to ensure the performance of victim against all possible attackers. and adversarial attacker.
To begin with, in this work, we will start from providing defense for the basic unconstrained attack, i.e. ϵ = 1 , for which we will see common adversarial training methods suffer from the non-converging problem.
We leave the defense to attack of different intensities as future works.
Since we only consider the robustness of the victim, we define the exploitability of π α using the following one-side exploitability.
Definition 4 ((One-side) exploitability) .
For a victim policy π α , we meaure the robustness of π α by:
[Equation]
To ensure the worst-case performance against the strongest adversarial policy, the victim should choose the policy according to min π α Expl( π α ) , which is equivalent to max π α min π ν V ρ ( π α , π ν ) , coinciding with finding the Nash Equilibrium in zero-sum Markov games [21].
To motivate the necessity of timescale separation in adversarial training for robust RL policy, we revisit some known issues of naive methods including Gradient Descent Ascent (GDA) and iterative best response (IBR) with both simultaneous and alternate update using a simple normal-form game Rock-Paper-Scissor, which is a Markov game with one step.
For detailed comparison, we investigate four single timescale algorithm and one algorithm with timescale separation in Appendix A. Formally, we propose to improve the adversarial training via timescale separation with Min oracle (shown in algorithm 1), where the attacker takes a min step against the victim in line 3 while the victim takes one gradient update in line 4 .
The min oracle used in algorithm 1 can be implemented with standard reinforcement
Algorithm
π
arg min π ν V ρ
π tα π ν
learning algorithm.
When the game has special structures like extensive-form games or one player has substantially smaller state and action space, such min oracle may be implemented efficiently.
In general, to make one gradient update for player α , agent ν needs to solve the corresponding best response oracle, which is computationally inefficient in practice.
To further fix this issue, we utilize the idea of using a much larger step size for player ν so that when player α performs the gradient update, player ν is already an approximate minimum of V ρ ( π tα , · ) .
Formally, in addition to Algorithm 1, we present an alternative efficient Algorithm 2, where the min oracle is replaced by a gradient update with a larger step and both players perform gradient update independently.
Theoretically, the algorithm is guaranteed to converge to (one-side) Nash Equilibrium by directly adopting the main conclusion of Theorem 1 from [4].
Formal statements are given in Appendix B.
Experimental Setup and Results
Our experiments are based on two common zero-sum games Kuhn Poker [11] and RoboSumo competition [1].
Detailed introductions to the environment are given in Appendix D.
In this section, we seek answers to the following two essential questions.
▷ (1) Can the constrained attack formulation yield less state distribution variations (being more imperceptible) compared with unconstrained attack with the same winning rate?
▷ (2) Will adversarial training with time scale separation achieve more stable learning dynamics and better/less robustness/exploitability compared with only single time scale?
Controllable Adversarial Attack.
To answer our first question, we conduct experiments on the
Robosumo Spider vs Spider environment.
To verify that our constrained attack approach indeed achieves smaller state distribution variations, we fix an unconstrained adversarial policy and a constrained policy with the same winning rate for fair comparisons.
Specifically, we investigate the distribution shift in the victim’s observation part of the state features.
To select important state features, we use the variance-based feature importance method to filter out the unimportant features of the states with small variance.
The feature variance is determined by observing the change in the features over several episodes.
The sorted feature importance is given in Appendix D.2 for the
Robosumo Spider-vs-Spider game.
Figure 1 shows that the constrained adversarial policy induces much smaller state distribution shift while compared to the unconstrained adversarial policy when testing under the same winning rate.
[Figure]
It is clearly that our constrained adversarial policy induces much smaller state distribution shift.
Improved Adversarial Training.
To answer our second question, we compare the learning dynamics and robustness of policies trained by two time scales and single time scales in both Kuhn
Poker and Robosumo Ant-vs-Ant environments.
[-o i t a b ili t y 1e-4 vs 1e-4 1e-4 vs 5e-4 1e-4 vs 1e-3 1e-4 vs 5e-3 1e-4 vs 1e-2 1e-4 vs Min 0 1 2 3 
-]
[-1e
-]
[-o i t a b ili t y 5e-3 vs 5e-3 5e-3 vs 1e-2 5e-3 vs Min 0 1 2 3 
-]
Kuhn Poker.
We implement both our algorithm 1 and 2 with open-source project OpenSpiel [12], where the min oracle is achieved by search in the game tree.
For gradient update, instead of using the vanilla monte-carlo policy gradient, we use both Regret Policy Gradient (RPG) [23] and Advantage
Actor Critic (A2C) [15] methods.
In figure 2, we show exploitability of the victim policy π α averaged over 5 independent seeds, where adversarial training with the min oracle enjoys the fastest convergence, lowest exploitability and variance.
Meanwhile, the victim policy trained with large enough timescale separation parameter η tν /η tα approximate the algorithm with a min oracle.
On the contrary, adversarial policy trained with only single timescale suffers from high exploitability and variance even when the step size is small enough.
Robosumo Competition.
To verify the necessity of timescale separation, apart from the previous relatively simple Kuhn Poker experiments, we also evaluate our proposed methods on Robosumo Ant vs Ant, a high-dimensional, continuous control task, which is much more challenging in terms of both training and evaluation.
For training, although such a min oracle with game tree search in the Kuhn
Poker does not exist in this continuous control task anymore, previous experiments highlight that a large enough timescale separation ratio can very well approximate the algorithm 1 with a min oracle. [Table] 1e [Table]
[Table] [Table] [Table] [Table]
E x p l
o i t a b ili t y 1e-4 vs 1e-4 1e-4 vs 5e-4 1e-4 vs 1e-3 1e-4 vs 5e-3 1e-4 vs 1e-2 1e
[Table]
[Table]
1e
[Table]
[Table]
[Table]
[Table]
[Table]
E x p l
o i t a b ili t y 5e
[Table]
4 vs
[Table]
4 5e
[Table]
4 vs 1e
[Table]
5e
[Table]
[Table]
1e
[Table]
1e
[Table]
[Table]
[Table]
[Table]
[Table]
E x p l
o i t a b ili t
[Table]
1e
[Table]
1e
[Table]
1e
[Table]
[Table]
3 1e
[Table]
vs 1e
[Table]
1e
[Table]
[Table]
1e 0.
[Table]
[Table]
E x p l
o i t a b ili t y 5e
[Table]
3 vs 5e
[Table]
3 5e-3 vs 1e
[Table]
[Table]
{+1e
+}
[Table]
[Table]
[Table]
E
[Table]
p
[Table]
o i t a
[Table]
ili t
[Table]
[Table]
-4 vs 1e-3 1e-4 vs 5e-3 1e-4 vs 1e-2 1e-4 vs Min 0 1 2 3 4 5 Number of episodes 1e6 0.
E x p l
o i t a b ili t y 5e-4 vs 5e-4 5e-4 vs 1e-3 5e-4 vs 5e-3 5e-4 vs 1e-2 5e-4 vs Min 0 1 2 3 4 5 Number of episodes 1e6 0.
E x p l
o i t a b ili t y 1e-3 vs 1e-3 1e-3 vs 5e-3 1e-3 vs 1e-2 1e-3 vs Min 0 1 2 3 4 5 Number of episodes 1e6 0.
E x p l
o i t a b ili t y 5e-3 vs 5e-3 5e-3 vs 1e-2 5e-3 vs Min 0 1 2 3 4 5 Number of episodes 1e6 0.
E x p l
{+o i t a b ili t y a2c Exploitability of player 
+}
1e-2 vs 1e-2 1e-2 vs Min
[Figure]
[Figure]
1e
[Figure]
[Figure]
[Figure]
[Figure]
For evaluation, again due to the lack of an efficient min oracle, we can not track the exploitability of the victim at every round of training.
Instead, we use standard RL algorithm to compute the best response of the final robustified victim policy and keep track of the score of the attacker in figure 3c, where lower winning rate for the attacker indicates better robustness of the victim.
Apart from the exploitability test, we also keep track of the score, V ρ ( π tα , π tν ) during training and the norm of gradient ∇ π α V ρ ( π tα , π tν ) .
From figure 3a and 3b, we verify that single timescale training leads to unstable behaviors, large variance and gradient norm, while the two timescale training leads to quickly converged value and much smaller gradient norm.
On the other hand, victim policy trained by two timescale shows much better robustness when the attacker computes the best response against it
A An Illustrative Example
Example 1.
The zero-sum game Rock-Paper-Scissor includes two players with the same action space
A = { Rock, Paper, Scissor } .
The payoff matrix P is given as P = (cid:34) −−   − (cid:35) for the row player.
The row player has the mixed strategy x ∈ X = ∆( A ) , where x i represents the probability of choosing i th action.
The column player holds a similar mixed strategy y ∈ Y .
The corresponding payoff is given by V ( x, y ) =
x ⊤ P y .
Our objective is given by max x ∈X min y ∈Y x ⊤ P y .
[Figure]
We evaluate the following 5 methods.
• Simultaneous Gradient Descent Ascent (SGDA): y t +
[Equation]
P Y
[Equation]
y t [Equation]
P
[Equation] [Equation]
+ η P y t ) . • Alternate Gradient Descent Ascent (AGDA): y
[Equation]
+ η P y t +1 )
• Simultaneous Iterative Best Response (SIBR):
[Equation]
t • Alternate Iterative Best Response (AIBR):
[Equation]
P
[Equation]
+1 .
• Gradient Ascent with Min oracle (GAMin):
[Equation]
We show the exploitability of the x player during the learning process in figure 4.
It is clear that the first four single timescale training methods (SGDA, AGDA, SIBR, AIBR) fail to achieve low exploitability, while only GAMin achieves the near optimal exploitability 0 .
B Adversarial Training with Two Timescale
Theorem 2.
Fix δ ≥ 0 .
Support both π α and π ν follows the direct parameterization as π α ( a α | s ) =
(1 − δ α ) x s,a α + δ α |A
α | and π ν ( a ν | s ) =
(1 − δ ν ) y s,a ν + δ ν |A
ν | and use the single episode
REINFORCE gradient estimator.
If the step size satisfies η α ≍ δ 10 .
5 , η ν ≍ δ 6 , and the exploration parameters satisfy δ α ≍ δ , δ ν ≍ δ 2 , it is guaranteed that
[Equation]
after T ≤ poly( 1 δ , |S| , |A
α | , |A ν | ) iterations.
C Full Proof C.1 Proof of Theorem
Proof.
Let us firstly review the following facts for any joint policy
π = ( π α , π ν ) , π ′ =
( π α , π ′ ν ) with D maxTV ( π ν || π ′ ν ) ≤ ϵ and the transition matrix P π , where P π ( s ′ , s ) =
(cid:80) a π ( a | s ) P ( s ′ | s, a )
and we use P π ( i, j ) denote P
π ( s i , s j ) .
1 α α π + η tα ∇ π α V ρ π
, π
+1 ν
∇
According to the definition in Equation 1, one can verify that d π satisfies that:
[Equation]
(6) which gives the solution d π = (1 − γ )
( I − γP π ) − 1 ρ .
For P π :
[Equation] [Equation]
(cid:88)
a
[Equation]
[Equation]
a
[Equation]
i
[Equation]
(9) =
For || ( I − γP π ) − 1 || 1 :
||
( I − γP π ) − 1 || 1 = || ∞ (cid:88) k
= ( γP π ) k || 1 ≤ ∞ (cid:88) k
= ||
( γP π ) k
|| 1 ≤ ∞ (cid:88) k =
[Equation]
For || P π − P π
|| 1 :
To begin with, since π α remains unchanged in our proof, let us abuse the notation a little bit and define the marginalized transition P ( s ′ | s, a ν )
= (cid:80) a α π α ( a α | s ) P ( s ′ | s, a α , a ν ) .
We have
[Equation]
Now fix any index j , define m ⊤ i
=
( P ( s i | s j , a kν )) |A
ν | k
=1 , M ⊤ =
( m 1 , · · · , m |S| ) , and n ⊤ =
( π ν ( a kν | s j ) − π ′ ν
( a kν | s j ))
|A
ν | k
=1 .
Then the following holds (cid:88)
i
[Equation]
[Equation]
According to the definition of M , it is easy to check
[Equation]
Therefore [Equation] we conclude that for any fixed index j , we have
[Equation]
which proves || P π − P π ′ || 1 ≤ ϵ .
Now we are ready to prove our theorem
[Equation]
C.2 Proof of Corollary
It is easy to verify that the following holds
[Equation]
Let us define the marginalized reward r π ( s )
= (cid:80) a π ( a | s ) r
( s, a )
[Equation]
and further define the vector notation r ⊤ π = ( r π ( s k ))
|S| k =1 .
Now for the difference of the value function
[Equation]
D Experimental Details D.1 Introduction to the Environment
Kuhn Poker is popular research game, which is extensive-form and zero-sum with discrete observation and action space.
There exists efficient min oracle with game tree search.
For RoboSumo competition, both agents are multi-leg robots and observe the position, velocity and contact forces of joints in their body and the position of their opponent’s joints, which is much more challenging due to the high dimensional observation and action space.
D.2 Feature Importantance in Robosumo Spider vs Spider
We show the important features in the Robosumo Spider vs Spider environment in Table
[Table]