R OBUST R EINFORCEMENT L EARNING ON S TATE O BSERVATIONS WITH L EARNED O PTIMAL A DVERSARY
Anonymous
authors
Paper under double-blind review
A BSTRACT
We study the robustness of reinforcement learning (RL) with adversarially perturbed state observations, which aligns with the setting of many adversarial attacks to deep reinforcement learning (DRL) and is also important for rolling out real-world RL agent under unpredictable sensing noise.
With a ﬁxed agent policy, we demonstrate that an optimal adversary to perturb state observations can be found, which is guaranteed to obtain worst case agent reward.
For DRL settings, this leads to a novel empirical adversarial attack to RL agents via a learned adversary that is much stronger than previous ones.
To enhance the robustness of an agent, we propose a framework of alternating training with learned adversaries (ATLA), which trains an adversary online together with the agent using policy gradient following the optimal adversarial attack framework.
Additionally, inspired by the analysis of state-adversarial Markov decision process (SA-MDP), we show that past states and actions (history) can be useful for learning a robust agent, and we empirically ﬁnd a LSTM based policy can be more robust under adversaries.
Empirical evaluation on a few continuous control environments show that ATLA achieves state-of-the-art performance under strong adversaries.
I NTRODUCTION
Modern deep reinforcement learning agents (Mnih et al., 2015; Levine et al., 2015; Lillicrap et al., 2015; Silver et al., 2016; Fujimoto et al., 2018) typically use neuron networks as function approximators.
Since the discovery of adversarial examples in image classiﬁcation tasks (Szegedy et al., 2013), the vulnerabilities in DRL agents were ﬁrst demonstrated in (Huang et al., 2017; Lin et al., 2017; Kos & Song, 2017) and further developed under more environments and different attack scenarios (Behzadan & Munir, 2017a; Pattanaik et al., 2018; Xiao et al., 2019).
These attacks commonly add imperceptible noises into the observations of states , e.g., the observed environment slightly differs from true environment states.
This raises concerns for using RL in safety-crucial applications such as autonomous driving (Sallab et al., 2017; Voyage, 2019); additionally, the discrepancy between ground-truth states and agent observations also contributes to the “reality gap” - an agent working well in simulated environments may fail in real environments due to noises in observations (Jakobi et al., 1995; Muratore et al., 2019), as real-world sensing contains unavoidable noise (Brooks, 1992).
We classify the weakness of a DRL agent on the perturbations of state observations into two classes: the vulnerability in function approximators , which typically originates from the highly non-linear and blackbox nature of neural networks; and intrinsic weakness of policy : even perfect features for states are extracted, a policy can still make mistakes due to an intrinsic weakness in the policy.
For example, in the deep Q networks (DQNs) for Atari games, a large convolutional neural network (CNN) is used for extracting features from input frames.
To act correctly, the network must extract crucial features: e.g., for the game of Pong, the position and velocity of the ball, which can observed by visualizing convolutional layers (Hausknecht & Stone, 2015; Guo et al., 2014).
Many attacks to the DQN setting add imperceptible noises (Huang et al., 2017; Lin et al., 2017; Kos & Song, 2017;
Behzadan & Munir, 2017a) that exploit the vulnerability of deep neural networks so that they extract wrong features, as we have seen in adversarial examples of image classiﬁcation tasks.
On the other hand, the fragile function approximation is not the only source of the weakness of a RL agent - in a ﬁnite-state markov decision process (MDP), we can use a tabular form of policy and value functions so there is no function approximation error.
The agent can still be vulnerable to small perturbations on g perturbing the observation neighbors
Techniques developed in enhancing the robustness of neural network (NN) classiﬁers can be applied to address the vulnerability in function approximators.
Especially, for environments like Atari games with images as input and discrete actions as outputs, the policy network π θ behaves similar to a classiﬁer in test time.
Thus, Fischer et al.
; Mirman et al.
(2018a) utilized existing certiﬁed adversarial defense (Mirman et al., 2018b; Wong & Kolter, 2018; Gowal et al., 2018) approaches in supervised learning to enhance the robustness of DQN agents.
Another successful approach (Zhang et al., 2020) for both Atari and high-dimensional continuous control environment regularizes the smoothness of the learned policy such that max ˆ s ∈B s D ( π ( s ) , π (ˆ s )) is small for some divergence D and B ( s ) is a neighborhood around s .
This maximization can be solved using a gradient method or convex relaxations of NNs (Salman et al., 2019), and then will be minimized by optimizing θ in a minimax loss function.
Such an adversarial minimax regularization is in the same spirit as the ones used in some adversarial training approaches for (semi-)supervised learning, such as in
TRADES (Zhang et al., 2019) and VAT (Miyato et al., 2015).
However, regularizing the function approximators does not explicitly improve the intrinsic robustness of the policy.
In this paper, we propose an orthogonal approach, alternating training with learned adversaries (ATLA), to enhance the robustness of DRL agents.
We focus on dealing with the intrinsic weakness of the policy by learning an adversary online with the agent during training time, rather than directly regularizing function approximators.
Our main contributions can be summarized as:
• We follow the theory of state-adversarial Markov decision process (SA-MDP) in Zhang et al.
and show how to learn an optimal adversary for perturbing observations.
We demonstrate practical attacks under this formulation and obtain learned adversaries that are signiﬁcantly stronger than previous ones.
• We propose the alternating training with learned adversaries (ATLA) framework to improve the robustness of DRL agents.
The difference between our approach and previous adversarial training approaches is that we use a stronger adversary, which is learned online together with the agent.
• Our analysis on SA-MDP also shows that history can be important for learning a robust agent.
We thus propose to use a LSTM based policy in the ATLA framework and ﬁnd that it is more robust than policy parameterized as regular feedforward NNs.
• We evaluate our approach empirically on four continuous control environments.
We outperform explicit regularization based methods in a few environments, and our approach can also be directly combined with explicit regularizations on function approximators to achieve state-of-the-art results.
R ELATED W ORK
State-adversarial Markov decision process (SA-MDP) (Zhang et al., 2020) characterizes the decision making problem under adversarial attacks on state observations .
Most importantly, the true state in the environment is not perturbed by the adversary under this setting; for example, perturbing pixels in an Atari environment (Huang et al., 2017; Kos & Song, 2017; Lin et al., 2017; Behzadan & Munir, 2017a; Inkawhich et al., 2019) does not change the true location of an object in the game simulator.
In real world settings, SA-MDP can characterize agent performance under natural or adversarial noise from sensor measurements.
For example, GPS sensor readings on a car are naturally noisy, but the ground truth location of the car is not affected by the noise.
Importantly, this setting is different from robust Markov decision process (RMDP)
(Nilim & El Ghaoui, 2004; Iyengar, 2005), where the worst case transition probabilities of the environment are considered.
“Robust reinforcement learning” in some works (Mankowitz et al., 2018; 2019) refer to this different deﬁnition of robustness in RMDP, and should not be confused with our setting of robustness against perturbations on state observations.
Several works proposed methods to learn an adversary online together with an agent.
RARL (Pinto et al., 2017) proposed to train an agent and an adversary under the two-player Markov game (Littman, 1994) setting.
The adversary can change the environment states through actions directly applied to environment.
The goal of RARL is to improve the robustness against environment parameter changes, such as mass, length or friction.
Gleave et al.
(2019) discussed the learning of an adversary using reinforcement learning to attack a victim agent, by taking adversarial actions that changes the environment and consequentially change the observation of the victim agent.
Both
Pinto et al.
; Gleave et al.
(2019) conduct their attack under on the two-player Markov game framework, rather than considering perturbations on state observations.
Besides, Li et al.
(2019) consider a similar Markov game setting in multi-agent RL environments.
The difference between these works and ours can be clearly seen in the setting where the adversary is ﬁxed - under the framework of (Pinto et al., 2017; Gleave et al., 2019), the learning of agent is still a MDP, but in our setting, it becomes a harder POMDP problem (Section 3.2).
Training DRL agents with perturbed state observations from adversaries have been investigated in a few works, sometimes referred to as adversarial training.
Kos & Song (2017); Behzadan & Munir (2017b) used gradient based adversarial attack to DQN agents and put adversarial frames into replay buffer.
This approach is not very successful because for Atari environments the main source of weakness is likely to come from the function approximator, so an adversarial regularization framework such as (Zhang et al., 2020; Qu et al., 2020) which directly control the smoothness of the Q function are more effective.
For lower dimensional continuous control tasks such as the MuJoCo environments, Mandlekar et al.
; Pattanaik et al.
(2018) conducted FGSM and multi-step gradient based attack during training time; however, their main focus was on the robustness against environment parameter changes and only limited evaluation on the adversarial attack setting was conducted with relatively weak adversaries.
Zhang et al.
(2020) systematically tested this approach under newly proposed strong attacks in a few more environments, and found that it cannot reliably improve robustness.
These early adversarial training approaches typically use a gradient basic attack based on a critic function.
Such an attack is usually relatively weak, and not sufﬁcient to lead to a robust policy under stronger attacks.
The robustness of RL has also been investigated from other perspectives.
For example, Tessler et al.
(2019) study MDPs under action perturbations; Tan et al.
(2020) use adversarial training on action space to enhance agent robustness under action perturbations.
Besides, policy teaching (Zhang &
Parkes, 2008; Zhang et al., 2009; Ma et al., 2019) and policy poisoning (Rakhsha et al., 2020; Huang & Zhu, 2019) manipulate the reward or cost signal during agent training time to induce a desired agent policy.
Essentially, policy teaching is a training time “attack” with perturbed rewards from the environments (which can be analogous to data poisoning attacks in supervised learning settings), while our goal is to obtain a robust agent against test time adversarial attacks.
All these settings differ from the setting of perturbing state observations discussed in our paper.
M ETHODOLOGY
In this section, we ﬁrst discuss the case where the agent policy is ﬁxed, and then the case where the adversary is ﬁxed in SA-MDPs.
This allows us to propose an alternating training framework to improve robustness of RL agents under perturbations on state observations.
Notations and Background We use S and A to represent the state space and the action space, respectively; P ( S ) deﬁnes the set of all possible probability measures on S .
We deﬁne a Markov decision process (MDP) as ( S , A , R, p, γ ) , where R : S ×
A × S → R and p : S ×
A → P ( S ) are two mappings represent the reward and transition probability.
The transition probability at time step t can be written as p ( s (cid:48) | s, a ) =
Pr( s t +1 =
s (cid:48) | s t = s, a t = a ) .
Reward function is deﬁned as the expected reward R ( s, a, s (cid:48) ) :
= E [ r t | s t = s, a t = {+a, s t +
 = s (cid:48) ]+} .
γ ∈ [0 , 1] is the discounting factor.
We denote a stationary policy as π : S → P ( A ) which is independent of history.
We denote history h t at time t as { s 0 , a 0 , · · · , s t − 1 , a t − 1 , s t } and H as the set of all histories.
A history-dependent policy is deﬁned as π : H → P ( A ) .
A partially observable Markov decision process (Astrom, 1965) (POMDP) can be deﬁned as a 7-tuple ( S , A , O , Ω , R, p, γ ) where O is a set of observations and Ω is a set of conditional observation probabilities p ( o | s ) .
Unlike MDPs, POMDPs typically require history-dependent optimal policies. [Table]
[Figure]
To study the decision problem under adversary on state observations, we use the framework of state-adversarial
Markov decision process (Zhang et al., 2020) (SA-MDP).
In SA-MDP, an adversary ν : S → P ( S ) is introduced to perturb the input state of an agent; however, the true environment state s is unchanged (Figure 2).
Formally, an SA-MDP is a 6-tuple ( S , A , B , R, p, γ ) where B is a mapping from a state s ∈ S to a set of states B ( s ) ∈ S .
The agent sees the perturbed state ˆ s ∼ ν ( ·| s ) and take the action π ( a | ˆ s ) accordingly.
B limits the power of adversary: supp ( ν ( ·| s ))
∈ B ( s ) .
The goal of SA-MDP is to solve an optimal policy π ∗ under its optimal adversary ν
∗ ( π ∗ ) ; an optimal adversary is deﬁned as an adversary ν ∗ ( π ) such that π achieves the lowest possible expected discounted return (or value) on all states.
Zhang et al.
(2020) did not give an explicit algorithm to solve SA-MDP and found that a stationary optimal policy need not exist.
3. F INDING THE OPTIMAL ADVERSARY UNDER A FIXED POLICY
In this section, we discuss how to ﬁnd an optimal adversary ν for a given policy π .
An optimal adversary leads to the worst case performance under bounded perturbation set B , and is an absolute lower bound of the expected cumulative reward an agent can receive.
It is similar to the concept of “minimal adversarial example” in supervised learning tasks.
We ﬁrst show how to solve the optimal adversary in MDP setting and then apply it to the DRL settings.
A technical lemma (Lemma 1) from Zhang et al.
(2020) shows that, from the adversary’s point of view, a ﬁxed and stationary agent policy π and the environment dynamics can be essentially merged into an MDP with redeﬁned dynamics and reward functions:
Lemma 1 (An extension of Zhang et al.
Given an SA-MDP M =
( S , A , R, B , p, γ ) and a ﬁxed and stationary policy π ( ·|· ) , there exists an MDP ˆ M =
( S , ˆ A , ˆ R, ˆ p, γ ) such that the optimal policy of ˆ M is the optimal adversary ν for SA-MDP given the ﬁxed π , where ˆ A = S , and ˆ R ( s, ˆ a, s (cid:48) ) : = E [ [Equation] s,
[Equation]
a, s
[Equation] [Equation]
for s, s [Equation] ∈ S and ˆ a ∈ B ( s ) ⊂ ˆ A ,
[Equation]
for s, s (cid:48) ∈ S and ˆ a / ∈ B ( s ) .
C is a large negative constant.
[Equation] for s, s [Equation]
and ˆ [Equation] A [Equation]
The intuition behind Lemma 1 is that the adversary’s goal is to reduce the reward earned by the agent.
Thus, when a reward r t is received by the agent at time step t , the adversary receives a negative reward of ˆ r t = − r t . In our proof, we extend (Zhang et al., 2020) slightly to allow r t to [Figure] be a random variable and our proof is given in Appendix A.1.
To prevent the agent from taking actions outside of set B ( s ) , a large negative reward is assigned to these actions such that the optimal adversary cannot take them.
For actions within B ( s ) , we calculate ˆ R ( s, ˆ a, s (cid:48) ) by its deﬁnition, ˆ R ( s, ˆ a, s (cid:48) ) :
= E [ˆ r | s, ˆ a, s (cid:48) ] which yields the term in Lemma 1 (see details in Appendix A.1). After constructing the MDP ˆ M , it is possible to solve an optimal agent ν of ˆ M , which will be the optimal adversary on SA-MDP M given policy π .
For MDPs, under mild regularity assumptions an optimal policy always exists (Puterman, 2014).
In our case, the optimal policy on ˆ M corresponds to an optimal adversary in SA-MDP, which is the worst case perturbation for policy π .
As an illustration, in Figure 1, we show a GridWorld environment.
The red square is the starting point.
The blue circle and green triangles are the target and traps, respectively.
When the agent hits the target, it earns reward +1 and the game stops and it earns reward -1 when it encounters a trap.
We set γ = 0 .
9 for both agent and adversary.
The adversary is allowed to perturb the observation to adjacent cells along four directions: up, down, left, and right.
When there is no adversary, after running policy iteration, the agent can easily reach target and earn a reward of +1, as in Figure 1a.
However, if we train the adversary based on Lemma 1 and apply it to the agent, we are able to make the agent repeatedly encounter a trap.
This leads to −∞ reward for the agent and + ∞ reward for the adversary as shown in Figure 1b.
We now extend this Lemma 1 to the DRL setting.
Since the learning of adversary is equivalent to solving an MDP, we parameterize the adversary as a neural network function and use any popular
DRL algorithm to learn an “optimal” adversary.
Here we quote the word “optimal” as we use function approximator to learn the agent
so it’s no longer optimal, but we emphasize that it follows the SA-MDP framework of solving an optimal adversary.
No existing adversarial attacks follow such a theoretically guided framework.
We show our algorithm in Algorithm 1.
Instead of learning to produce ˆ s ∈ B ( s ) directly, since B ( s ) is usually a small set nearby s (e.g., B = s (cid:48) |(cid:107)
s − s (cid:48)
(cid:107) p ≤ (cid:15)
, our adversary learns a perturbation vector ∆ , and we project s + ∆ to B ( s ) .
The ﬁrst advantage of attacking a policy in this way is that it is strong - as we allow to optimize the adversary in an online loop of interactions with the agent policy and environment, and keep improving the adversary with a goal of receiving as less reward as possible.
It is strong because it follows the theoretical framework of ﬁnding an optimal adversary, rather than using any heuristic to generate a perturbation.
Empirically, in the cases demonstrated in Figure 3, previous strong attacks (e.g., Robust Sarsa attack) can successfully fail an agent and make it stop moving and receive a small positive reward; our learned attack can trick the agent into moving toward the opposite direction of the goal and receive a large negative reward.
We also ﬁnd that this attack can further reduce the reward of robustly trained agents, like SA-PPO (Zhang et al., 2020).
The second advantage of this attack is that it requires no gradient access to the policy itself; in fact, it treats the agent as part of the environment and only needs to run it in a blackbox.
Previous attacks (e.g., Lin et al.
; Pattanaik et al.
Xiao et al.
(2019)) are mostly gradient based approach and need to access the values or gradients to a policy or value function.
Even without access to gradients, the overall learning process is still just a MDP and we can apply any popular modern DRL methods to learn the adversary. Input: Policy π
) 1: initialize ν s parameterized by neural parameters φ , 2: for i 1 to N do 3: D ← Adv Traj ( ν φ , π, B ) # episodes here) 4: φ ← PolicyOptimizer ( D , φ ) 5: end for Function Adv Traj ( ν φ , π, B ) : 6: s ← s# Initial state 7: D ← ∅ 8: for b = 1 to B do 9: ∆ ← sample from ν φ (
| s ) 10: ˆ s ← Proj B s ) s + ∆)
11: a ← sample π s ) 12: obtain reward r next state s (cid:48) from environment given action a 13:
D ← D ∪ ( s, ∆ , − r, s (cid:48) )
14: s ← s (cid:48) 15: end for 16: return D
3. F INDING THE OPTIMAL POLICY UNDER A FIXED ADVERSARY
We now investigate SA-MDP when we ﬁx the adversary ν and ﬁnd an optimal policy.
In Lemma 2, we show that this case SA-MDP becomes a POMDP:
Lemma 2 (Optimal policy under ﬁxed adversary)
Given an SA-MDP M =
( S , A , B , R, p, γ ) and a ﬁxed and stationary adversary ν
(ˆ s | s ) , there exists a POMDP ¯ M = ( S , A , O , Ω , R, p, γ ) such that the optimal policy of ¯ M is the optimal policy π for SA-MDP given the ﬁxed ν , where O = (cid:91) s ∈S { s (cid:48)
[Equation]
To prove Lemma 2, we construct the a POMDP with the observations deﬁned on the support of all ν ( ·| s ) , s ∈ S and the observation process is exactly the process of generating an adversarially perturbed state ˆ s .
This POMDP is functionally identical to the original SA-MDP when ν is ﬁxed.
This lemma unveils the connection between POMDP and SA-MDP: SA-MDP can be seen as a version of “robust” POMDP where the policy needs to be robust under a set of observational processes (adversaries).
SA-MDP is different from robust POMDP (RPOMDP)
(Osogami, 2015; Rasouli &
Saghaﬁan, 2018), which optimizes for the worst case environment transitions.
As a proof of concept, we use a modern POMDP solver, SARSOP (Kurniawati et al., 2008) to solve the GridWorld environment in Figure 1 to ﬁnd a policy that can defeat the adversary.
The POMDP solver produces a ﬁnite state controller (FSC) with 8 states (FSC is an efﬁcient representation of history dependent policies).
This FSC policy can almost eliminate the impact of the adversary and receive close to perfect reward, as shown in Figure 1c.
Unfortunately, unlike MDPs, it is challenging to solve an optimal policy for POMDPs; state-of-theart solvers (Bai et al., 2014; Sunberg & Kochenderfer, 2017) can only work on relatively simple environments which are much smaller than those used in modern DRL.
Thus, we do not aim to solve the optimal policy.
We follow (Wierstra et al., 2007) to use recurrent policy gradient theorem on POMDPs and use LSTM as function approximators for the value and policy networks. We denote h t
= { ˆ s 0 , a 0 , ˆ s 1 , a 1 · · · , ˆ s t } containing all history of states (perturbed states ˆ s in our setting) and actions. The policy π parameterized by θ takes an action a t given all observed history h t , and h t is typically encoded by a recurrent neural network (e.g., LSTM). The recurrent policy gradient theorem (Wierstra et al., 2007) shows that
[Equation]
where N is the number of sampled episodes, T is episode length (for notation similarity, we assume each episode has the same length), and h nt is the history of states for episode n up to time t , and r nt is the reward received for episode n at time t .
We can then extend Eq. 2 to modern DRL algorithms such as proximal policy optimization (PPO), similarly as done in (Azizzadenesheli et al., 2018), by using the following loss function: [Equation]
where A h nt is a baseline advantage function for episode n time step t , which is based on a LSTM value function. (cid:15) is the clipping threshold in PPO. The loss can be optimized via a gradient based optimizer and θ old is the old policy parameter before optimization iterations start.
Although a LSTM or recurrent policy network has been used in the DRL setting in a few other works (Hausknecht & Stone, 2015; Azizzadenesheli et al., 2018), our focus is to improve agent robustness rather than learning a policy purely for POMDPs.
In our empirical evaluation, we will compare feedforward and LSTM policies under our ATLA framework.
3. A LTERNATING T RAINING WITH L EARNED A DVERSARIES (ATLA)
As we have discussed in Section 3.1, we can solve an optimal adversary given any ﬁxed policy.
In our ATLA framework, we train such an adversary online with the agent: we ﬁrst keep the agent and optimize the adversary; the adversary is also parameterized as a neural network.
Then we keep the adversary and optimize the agent.
Both adversary and agent can be updated using a policy gradient algorithm such as PPO.
We show our full algorithm in Algorithm 2. deﬁned in Algorithm
Our algorithm is designed to use a strong and learned adversary that tries to ﬁnd intrinsic weakness of the policy, and to obtain a good reward the policy must learn to defeat such an adversary.
In other words, it attempts to solve the SA-MDP problem directly rather than relying on explicit regularization on the function approximator like the approach in (Zhang et al., 2020).
In our empirical evaluation, we show that such regularization can be unhelpful in some environments and harmful for performance when evaluating the agent without attacks.
The difference between our approach and previous adversarial training approaches such as (Pattanaik et al., 2018) is that we use a stronger adversary, learned online with the agent.
Our empirical evaluation ﬁnds that using such a learned “optimal” adversary in training time allows the agent to learn a robust policy generalized to different types of strong adversarial attacks during test time.
Additionally, it is important to distinguish between the original state s and the perturbed state ˆ s .
We ﬁnd that using s instead of ˆ s to train the advantage function and policy of the agent leads to worse performance, as it does not follow the theoretical framework of SA-MDP.
E XPERIMENTS
“Optimal” attack on DRL agents In section 3.1 we show that it is possible to cast the optimal adversary ﬁnding problem as an MDP problem.
In practice, the environment dynamics are unknown [Table] SA-PPO 3635 ±3589 ± [Table]
±
[Table]
±
[Table]
3283 ±
[Table]
but model-free RL methods can be used to approximately ﬁnd this optimal adversary.
In this section, we use PPO to train an adversary on 4 MuJoCo continuous control environments.
In Table 1 we present results on attacking vanilla PPO and robustly trained SA-PPO (Zhang et al., 2020) agents.
As a comparison, we also report the attack reward of ﬁve other baseline attacks: critic attack is based on (Pattanaik et al., 2018); random attack adds uniform random noise to state observation; MAD (maximal action difference) attack (Zhang et al., 2020) maximizes the differences in action under perturbed state; RS (robust sarsa) attack is based on training robust action-value functions and was the strongest attack proposed in (Zhang et al., 2020). Additionally, we include a recent black-box attack, Snooping attack (Inkawhich et al., 2019).
For all attacks in this paper we consider B ( s ) as a (cid:96) ∞ norm ball around s with radius (cid:15) , set similarly as in (Zhang et al., 2020).
During testing, we run the agents without attacks as well as under attacks for 50 episodes and report the mean and standard deviation of the reward.
In Table 1 our “optimal” attack achieves noticeably lower rewards than all the other ﬁve attacks.
We illustrate a few examples of attacks in Figure 3.
For each method, we train with the best hyper-parameters for at least ﬁve times and choose the median model for this experiment.
For RS and “optimal” attacks, we report the best (lowest) attack reward obtained from different hyper-parameters.
Evaluation of ATLA In this experiment, we study the effectiveness of our proposed ATLA method.
Speciﬁcally, we use PPO as our policy optimizer.
For policy networks, we have two different structures: the original fully connected (MLP) structure, and an LSTM structure which takes historical observations.
The LSTMs are trained using backpropagation through time for up to 100 steps.
In Table 2 we include the following methods for comparisons
: • PPO (vanilla) and PPO (LSTM): PPO with a feedforward NN or LSTM as the policy network.
• SA-PPO (Zhang et al., 2020): the state-of-the-art approach for improving the robustness of DRL in continuous control environments, using a smooth policy regularization on feedforward NNs solved by convex relaxations.
• Adversarial training using critic attack (Pattanaik et al., 2018): a previous work using critic based attack to generate adversarial observations in training time, and train a feedforward NN based agent with this relatively weak adversary.
• ATLA-PPO (MLP) and ATLA-PPO (LSTM):
Our proposed method trained with a feedforward NN (MLP) or LSTM as the policy network.
The agent and adversary are trained using PPO with independent value and policy networks.
For simplicity, we set N π = N
ν
= 1 in all settings.
• ATLA-PPO (LSTM)
+SA reg: Based on ATLA-PPO (LSTM), but with an extra adversarial smoothness constraint similar to those in SA-PPO.
We use a 2-step stochastic gradient Langevin dynamics (SGLD) to solve the minimax loss, as convex relaxations of LSTMs are expensive.
For each model, we report its “natural reward” (without attacks) and best attack reward in Table 2.
The best attack reward is the lowest reward achieved by all six types attacks in Table 1, including our new “optimal” attack (these include hundreds of independent adversaries for attacking a single agent, see Appendix A.2 for more details).
For reproducibility, for each setup we train at least 5 (up to agents, attack all of them and report the one with median robustness.
We include detailed hyperparameters in A.6.
We report natural rewards (no attacks) and the best (lowest) attack rewards among six types of adversarial attacks, including a simple random noise attack, the critic based attack in (Pattanaik et al., 2018), MAD and RS attacks in Zhang et al. Snooping attack proposed in Inkawhich et al. and the optimal attack proposed in this paper.
For each environment, we bold the most robust model. Since both RS attack and our “optimal” attack are parameterized attacks, the “best attack” column represents the worst case agent performance under hundreds of adversaries . See Appendix A.2 for more details
[Table] [Figure] HalfCheetah [Figure]
In Table 2 we can see that vanilla PPO with MLP or LSTM are not robust. For feedforward (MLP) agent policies, critic based adversarial training (Pattanaik et al., 2018) is not very effective under our suite of strong adversaries and is sometimes only slightly better than vanilla PPO.
ATLA-PPO (MLP) outperforms SA-PPO on Hopper and Walker2d and is also competitive on HalfCheetah; for high dimensional environments like Ant, the robust function approximator regularization in SA-PPO is more effective.
For LSTM agent policies, compared to vanilla PPO (LSTM) agents, ATLA-PPO (LSTM) can signiﬁcantly improve agent robustness; a LSTM agent trained without a robust training procedure like ATLA cannot improve robustness. We ﬁnd that LSTM agents tend to be more robust than their MLP counterparts, validating our ﬁndings in Section 3.2. ATLA-PPO (LSTM) is better than SA-PPO on Hopper and Walker2d. In all settings, especially for high dimensional environments like Ant, our ATLA approach that also includes State-Adversarial regularization (ATLA-PPO +SA Reg) outperforms all other baselines, as this combination improves both the intrinsic robustness of policy and the robustness of function approximator.
A robust function approximator can be insufﬁcient For some environments, SA-PPO method has its limitations - even using an increasingly larger regularization parameter κ (which controls how robust the function approximator needs to be), we still cannot reach the same performance as our ATLA agent (Figure 4).
Additionally, when a large regularization is used, agent performance becomes much worse.
In Figure 4, under the largest κ = 1 . 0 , the natural reward ( 1436 ± 96 ) is much lower than other agents reported in Table 2.
A A PPENDIX A. P ROOF OF L EMMA
In Zhang et al. , the authors only consider MDP with deterministic reward for the adversary. Here we extend Lemma 1 in Zhang et al.
(2020) to stochastic reward such that given ( s, a, s (cid:48) )
, the agent’s reward is sampled from a distribution p ( r | s, a, s (cid:48) ) .
For the case of ˆ a ∈ B ( s ) , the reward function deﬁnition is based on the intuition that when the agent receives a reward r at a time step given s, a, s (cid:48) , the adversary’s reward is ˆ r = − r. Note that we now consider r also as a random variable subjected to a conditional distribution p ( r | s, a, s (cid:48) ) . To give the distribution of rewards for adversary p (ˆ r | s, ˆ a, s (cid:48) ) , we follow the conditional probability which marginalizes π : [Equation]
Considering that R ( s, a, s (cid:48) )
=
E [ r | s (cid:48) , a, s ] = − E
[ˆ r | s (cid:48) , a, s ] , and taking an expectation in Eq.
(5) over ˆ r yield the ﬁrst case in Lemma 1:
[Equation]
Then we can see that this ˆ R ( s, ˆ a, s (cid:48) ) has the same form as in Zhang et al.
(2020), so the rest of the proof is the same as the proof in the appendix of Zhang et al.
A.
In Table 2, we only include the best attack rewards (lowest rewards over all attacks).
In Table 3 we list the rewards under each speciﬁc attack.
Note that, Robust Sarsa (RS) attack and our “optimal” policy attack both have hyperparameters.
For RS attack we use the same set of 30 different settings of hyperparameters as in (Zhang et al., 2020) to train a robust value function to attack the network.
The reported RS attack result for each agent is the strongest one over the 30 trained value functions.
For Snooping based attack, we use the “imitator” attack proxy as it was the strongest reported in (Inkawhich et al., 2019), and we attack every step of the agent.
The imitator is a MLP or LSTM network according to agent policy network.
We use the same loss KL divergence function as in the
MAD attacks for this Snooping attack.
We ﬁrst collect state-action pairs for 100 episodes to train the “imitators”, whose network structures are the same as the corresponding agents. In the test time, we ﬁrst run MAD attack on it the “imitator” and then input the generated perturbed observation to the agent in a transfer attack fashion. For our “optimal” policy attack, the hyperparameters are PPO training parameters for the adversary (including the learning rate of the adversary policy network, learning rate of the adversary value network, the entropy regularization parameter and the ratio clip (cid:15) for PPO).
We use a grid search of these hyperparameters to train an adversary that is as strong as possible, resulting in 100 to 200 adversaries produced for each agent.
The reported optimal attack rewards is the lowest reward among all trained adversaries.
Under this comprehensive adversarial evaluation, each agent is tested using hundreds of adversaries and the strongest adversary determines the true robustness of an agent. [Table] A. A GENT P ERFORMANCE DURING T RAINING
In Table 3 we only report the agent performance at the end of training.
In this subsection
[Table]
we evaluate our agent performance during 20%, 40%, 60% and 80% of total training epochs.
The results are presented in Table 4/
[Table]
A. N ETWORK STRUCTURE
For fully connected networks, we use the same network as in (Zhang et al., 2020), which contains 2 hidden layers with 64 hidden neurons each layer, for both policy and value networks, for both the agent and the adversary.
For LSTM agents, we use a single layer LSTM with 64 hidden neurons, along with an input embedding layer projecting state dimension to 64 and an output layer projecting 64 to output dimension.
For LSTM agents, when conducting the “optimal” attack, we also use a
LSTM network for the adversary to ensure the adversary is powerful enough.
A. H YPERPARAMETER FOR THE LEARNING - BASED “ OPTIMAL ” ATTACK
Our “optimal” attacks uses policy gradient methods to learn the optimal adversary during agent testing, and each learning process involves the selection of hyperparameters.
Speciﬁcally, the hyperparameters include the learning rates of the adversary’s policy and value networks, the entropy coefﬁcient, and the annealing of the learning rate.
To reduce search space, for ATLA models, the learning rates of the testing phase adversary’s policy and value networks are chosen ranging from 0.3X to 3X of the learning rates of adversary’s policy and value networks used in training.
For other models trained without an adversary, the learning rates of the testing phase adversary’s policy and value networks are chosen ranging from 0.3X to 3X of the learning rates of the agent’s policy and value networks.
We tested both linearly annealed learning rate and non-annealing learning rate.
The adversary’s entropy coefﬁcient is chosen form 0 and 0.003.
The ﬁnal results reported in all tables are the best (lowest) reward achieved by the “optimal” attacks among all hyperparameter conﬁgurations.
Typically this includes around 100 to 200 different adversaries trained with different hyperparameters.
This guarantees the strength of this attack and allows a comprehensive evaluation of the robustness of all agents.
A. H YPERPARAMETERS FOR ATLA PERFORMANCE EVALUATION
Hyperparameters for PPO (vanilla)
For the Walker2d and Hopper environment, we use the same set of hyperparameters as in (Zhang et al., 2020); the hyperparameters were originally from (Engstrom et al., 2020) and found using a grid search experiment.
We found that this set of hyperparameters work well.
For HalfCheetah and Ant environment, we use a grid search of hyperparameters, including the learning rate of the policy network, learning rate of the value network and the entropy bonus coefﬁcient.
For Hopper, Walker2d and HalfCheetah environments, we train for 2 million steps (2 million environment interactions).
For Ant, we train for 10 million steps.
Training for longer may slightly improve agent performance under no attacks, but has no impact for performance under strong adversarial attacks.
Hyperparameters for PPO (LSTM) For PPO (LSTM), we conduct a smaller scale hyperparameter search.
We search hyperparameter values that are close to the optimal ones found for the PPO vanilla model.
We train these LSTM agents for the same steps as those in vanilla PPO.
Hyperparameters for SA-PPO We use the same value for all hyperparameters as in vanilla PPO except SA-PPO’s extra κ for the strength of SA-PPO regularization.
For κ , we choose from 1 × 10 −to 1 .
We train models with each κ at least ﬁve times and choose the κ value whose median model has the highest worst-case reward under all attacks.
Hyperparameters for ATLA-PPO For ATLA-PPO, we have hyperparameters for both agent and adversary.
We keep all agent hyperparameters the same as those in vanilla MLP/LSTM agents, except for the entropy bonus coefﬁcient.
We ﬁnd that sometimes we need a larger entropy bonus coefﬁcient in ATLA to allow sufﬁcient exploration of the agent, as learning with an adversary is harder than learning in attack-free environments.
For the adversary, we run a small-scale hyperparameter search on the learning rate of adversary policy and value networks, and the entropy bonus coefﬁcient for the adversary.
To reduce the number of hyperparameters for searching, we use values close to those of the agent.
We set N ν = N π = 1 in all experiments and did not tune this hyperparameter.
For ATLA, we train 5 million steps for Hopper, Walker and HalfCheetah and 10 million steps for
Ant.
We ﬁnd that similar to the observations in (Madry et al., 2018), training with an adversary typically requires more steps to converge, however in all our environments the training does reliably converge.
Model selection For each setup, we repeat the experiments using the same set of hyperparameters for at least 5 times, and at most 21 times for some high variance settings.
We then attack all the models using random, critic, MAD and RS attacks.
We use the lowest reward among all attacks as a metric to rank those models.
Then, we select the model with median robustness as our ﬁnal model.
This ﬁnal model is then attacked using the “optimal” attack to further reduce its reward.
The numbers we report in Table 2 are not from the best runs, but the runs with median robustness.
This is done to improve reproducibility as RL training process can have high variance.
[Table]