R OBUST R EINFORCEMENT L EARNING VIA A DVERSARIAL TRAINING WITH L ANGEVIN D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
We re-think the Two-Player Reinforcement Learning (RL) as an instance of a distribution sampling problem in inﬁnite dimensions.
Using the powerful Stochastic
Gradient Langevin Dynamics, we propose a new two-player RL algorithm which is a sampling variant of the two-player policy gradient method.
Our new algorithm consistently outperforms existing baselines, in terms of generalization across differing training and testing conditions, on several MuJoCo environments.
I NTRODUCTION
Reinforcement learning (RL) promise automated solutions to many real-world tasks with beyondhuman performance.
Indeed, recent advances in policy gradient methods (Sutton et al., 2000; Silver et al., 2014; Schulman et al., 2015; 2017) and deep reinforcement learning have demonstrated impressive performance in games (Mnih et al., 2015; Silver et al., 2017), continuous control (Lillicrap et al., 2015), and robotics (Levine et al., 2016) towards this grand challenge.
Despite the success of deep RL, the progress is still upset by the fragility in real-life deployments.
In particular, majority of these methods fail to perform well when there is some difference between training and testing scenarios, thereby posting serious safety and security concerns.
To this end, learning policies that are robust to environmental shifts, mismatched conﬁgurations, and even mismatched control actions is becoming increasingly more important.
A powerful framework to learning robust policies is to interpret the changing of the environment as an adversarial perturbation.
This notion naturally lends itself to a two-player minimax problem involving a pair of agents, a protagonist and an adversary, where the protagonist learns to fulﬁll the original task goals while being robust to the disruptions generated by its adversary.
Two prominent examples along this research vein, differing in how they model the adversary, are the
Robust Adversarial Reinforcement Learning (RARL) (Pinto et al., 2017) and Noisy Robust Markov
Decision Process (NR-MDP) (Tessler et al., 2019).
Despite achieving impressive performance in practice, these existing frameworks heavily rely on heuristic algorithms, and hence, suffer from lack of theoretical guarantees, even in idealistic cases with inﬁnite data as well as computational power.
One critical challenge in robust RL setting is that while maximizing rewards is a well-studied subject in classical/deep RL, the needed two-player minimax version is signiﬁcantly more complicated to solve both in theory and practice.
For instance, Tessler et al.
(2019) prove that it is in fact strictly suboptimal to directly apply (deterministic) policy gradient steps to their NR-MDP max-min objectives.
Owing to the lack of a better algorithm, the policy gradient is nonetheless still employed in their experiments; similar comments also apply to (Pinto et al., 2017).
Our paper precisely bridges this gap between theory and practice in previous works, by proposing the ﬁrst theoretically convergent algorithm for robust RL.
Our key idea is to switch from optimizing the max-min reward to sampling from the optimal randomized policies , which corresponds to ﬁnding a mixed Nash Equilibrium (NE) (Nash et al., 1950) in the max-min objective.
It is a classical fact in game theory that, while deterministic minimax objective is often ill-posed, the mixed NE is well-behaved under very mild assumptions.
Furthermore, algorithmic approaches to ﬁnding mixed NE with ﬁnite strategies have been studied extensively (Freund & Schapire, 1999;
Nemirovski, 2004) and is recently extended to the case of inﬁnite strategies (Hsieh et al., 2019).
In particular, (Hsieh et al., 2019) show that, by using the Stochastic Gradient Langevin Dynamics
(SGLD)
(Welling & Teh, 2011) to take samples from randomized strategies, one can ﬁnd
a mixed
NE of Generative Adversarial Networks (Goodfellow et al., 2014).
Our work introduces the same mixed NE perspective to the max-min objectives in robust RL, and substantiate the ensuing theoretical framework with extensive experimental evidence.
We apply the new sampling framework to the well-known Deep Deterministic Policy Gradient (DDPG) method in the scope of NR-MDP.
We demonstrate that the new algorithm achieves clearly superior performance in its generalization capabilities.
Intriguingly, we also observe that the idea of mixed strategy in single-player RL (i.e., the non-robust or one-player formulation) can lead to substantially more robust policies over the standard DDPG algorithm.
More precisely, we represent the agent’s policy as a distribution over deterministic policies, and aim to sample from the distribution µ ( π ) ∝ exp( − 1 σ J ( π )) where π denotes a deterministic policy, J the associated expected reward, and σ a temperature parameter going to 0 during training.
Our numerical evidence demonstrates that DDPG combined with this sampling approach for the actor update leads to learned policies that generalize better to unseen MDPs, when compared to the state-of-the-art DDPG variant of Tessler et al.
(2019) while using similar computational resources.
B ACKGROUND 2. S TOCHASTIC G RADIENT L ANGEVIN D YNAMICS (SGLD)
For any probability distribution p ( z ) ∝ exp ( − g
z )) , the Stochastic Gradient Langevin Dynamics (SGLD)
Welling & Teh (2011) iterates as
[Equation]
(1) where γ is the step-size, (cid:92) ∇ z g ( z ) is an unbiased estimator of ∇ z g ( z ) , (cid:15) > 0 is a temperature parameter, and ξ k ∼ N (0 , I ) is a standard normal vector, independently drawn across different iterations.
In some cases, the convergence rate of SGLD can be improved by scaling the noise using a positive-deﬁnite symmetric matrix C .
We thus deﬁne a preconditioned variant of the above update equation 1 as follows:
[Equation]
In the experiments, we use a RMSProp-preconditioned version of the SGLD (Li et al., 2016).
2. I NFINITE -D IMENSIONAL B I -L INEAR G AMES
In this section, we review some of the key results from (Hsieh et al., 2019).
We denote the set of all probability measures on Z by P ( Z ) , and the set of all functions on Z by F ( Z ) .
Given a (sufﬁciently regular) function h : Θ × Ω → R , consider the following objective (a two-player game with inﬁnitely many strategies):
[Equation]
A pair ( p ∗ , q ∗ ) achieving the max-min value in equation 3 is called a mixed Nash Equilibrium (NE).
Deﬁne the operator G : P (Ω) → F (Θ) , and its adjoint operator G † : P (Θ) → F (Ω) as follows:
[Equation]
Denoting (cid:104) p, g (cid:105) :
=
E z ∼ p
[ g ( z )] for any probability measure p and function g on Z
, we can write f as f ( p, q ) =
(cid:104) p, Gq (cid:105) =
(cid:10) G † p, q (cid:11) .
Furthermore, the derivative (the analogue of gradient in inﬁnite dimension) of f ( p, q ) with respect to p is simply Gq , and the derivative of f ( p, q ) with respect to q is G † p ; i.e., ∇ p
f
( p, q ) =
Gq , and ∇ q f ( p, q )
=
G † p .
Input: ω 1 , θ 1 ← random initialization, SGLD step-size { γ t } Tt =1 , thermal noise of SGLD { (cid:15) t } Tt =1 , warmup steps for SGLD { K t } Tt =1 , exponential damping factor β , standard normal noise ξ k , ξ (cid:48) k .
for t = 1 , 2 , . . .
, T − 1 do ¯ ω t , ω (1) t ← ω t ; ¯ θ t , θ (1) t ← θ t for k = 1 , 2 , . .
, K t do θ ( k +1)
t ← θ ( k ) t + γ t (cid:104) (cid:92) ∇ θ h ( θ, ω t ) (cid:105) θ = θ ( k ) t + √ 2 γ t (cid:15) t ξ k ω
( k +1) t ← ω ( k )
[Table]
−
[Table]
(cid:104) (cid:92) ∇
[Table]
h (
[Table]
, ω
[Table]
(cid:105)
[Table]
¯ ω
[Table]
←
[Table] [Table]
[Table] [Table]
[Table] [Table]
end for ω t +1 ← (1 − β )
ω t + β
[Table] [Table]
← (
[Table]
+ β ¯ θ t end for
Output: ω T , θ T .
Conceptually, problem (3) can be solved via the so-called inﬁnite-dimensional entropic mirror descent; see Algorithm 2 in the appendix.
Hsieh et al.
(2019) have proved the convergence (to the mixed NE) rate of Algorithm 2.
But this algorithm is inﬁnite-dimensional and requires inﬁnite computational power to implement.
For practical interest, by leveraging the SGLD sampling techniques and using some practical relaxations, Hsieh et al.
(2019) have proposed a simpliﬁed variant of Algorithm 2.
The pseudocode for their resulting algorithm can be found in Algorithm 1.
T WO -P LAYER M ARKOV G AMES
Markov Decision Process: We consider a Markov Decision Process (MDP) represented by
M 1 :=
( S , A , T 1 , γ, P 0 , R 1 ) , where the state and action spaces are denoted by S and A respectively.
We focus on continuous control tasks, where the actions are real-valued, i.e. , A = R d .
T 1 : S × S ×
A → [0 , 1] captures the state transition dynamics, i.e. , T 1 ( s (cid:48) | s, a ) denotes the probability of landing in state s (cid:48) by taking action a from state s .
Here γ is the discounting factor,
P 0 : S → [0 , 1] is the initial distribution over states S , and R 1 : S ×
A → R is the reward.
Two-Player Zero-Sum Markov Games: Consider a two-player zero-sum Markov game Littman (1994); Perolat et al. (2015), where at each step of the game, both players simultaneously choose an action.
The reward each player gets after one step depends on the state and the joint action of both players.
Furthermore, the transition kernel of the game is controlled jointly by both the players.
In this work, we only consider simultaneous games, not the turn-based games.
This game can be described by an MDP M 2 = ( S , A , A (cid:48) , T 2 , γ, R 2 , P 0 ) , where A and A (cid:48) are the continuous set of actions the players can take, T 2 : S ×
A × A (cid:48)
× S → R is the state transition probability, and R 2 : S ×
A × A (cid:48)
→ R is the reward for both players.
Consider an agent executing a policy µ : S → A , and an adversary executing a policy ν : S → A (cid:48) in the environment M .
At each timestep t , both players observe the state s t and take actions a t = µ ( s t ) and a (cid:48) t = ν
( s t ) .
In the zero-sum game, the agent gets a reward r t =
R 2 ( s t , a t , a (cid:48) t ) while the adversary gets a negative reward − r t .
This two-player zero-sum Markov game formulation has been used to model the following robust
RL settings:
• Robust Adversarial Reinforcement Learning (RARL) (Pinto et al., 2017), where the power of the adversary is limited by the action space A (cid:48) of the adversary.
• Noisy Robust Markov Decision Process (NR-MDP)
(Tessler et al., 2019), where A (cid:48)
=
A , T 2 ( s t +1 | s t , a t , a (cid:48) t ) =
T 1 ( s t +1 | s t , ¯ a t ) , and R 2 ( s t , a t , a (cid:48) t )
=
R 1 ( s t , ¯ a t ) , with ¯ a t =
(1 − δ ) a t + δa (cid:48) t , for δ ∈ (0 , 1) .
The power of the adversary is limited by δ .
In our adversarial game, we consider the following performance objective:
[Equation]
where (cid:80) ∞ t =1 γ t − 1 r t be the random cumulative return.
In particular, we consider the parameterized policies { µ θ : θ ∈ Θ } , and { ν ω : ω ∈ Ω } .
By an abuse of notation, we denote J ( θ, ω ) =
J ( µ θ , ν ω ) .
We consider the following objective:
[Equation]
Note that J is non-convex/concave in both θ and ω .
Instead of solving equation 4 directly, we focus on the mixed strategy formulation of equation 4.
In other words, we consider the set of all probability distributions over Θ and Ω , and we search for the optimal distribution that solves the following program:
[Equation]
Then, we can use the techniques from Section 2.2 to solve the above problem.
E XPERIMENTS
In this section, we demonstrate the effectiveness of using inﬁnite-dimensional sampling techniques to solve the robust RL problem.
Two-Player DDPG: As a case study, we consider NR-MDP setting with δ = 0 .
1 (as recommended in Section 6.3 of (Tessler et al., 2019)).
We design a two-player variant of DDPG (Lillicrap et al., 2015) algorithm by adapting the Algorithm 1.
As opposed to standard DDPG, in two-player
DDPG two actor networks output two deterministic policies, the protagonist and adversary policies, denoted by µ θ and ν ω .
The critic is trained to estimate the Q-function of the joint-policy.
The gradients of the protagonist and adversary parameters are given in Proposition 5 of (Tessler et al., 2019).
The resulting algorithm is given in Algorithm 3.
We compare the performance of our algorithm against the baseline algorithm proposed in (Tessler et al., 2019)
(see Algorithm 4).
(Tessler et al., 2019) have suggested a training ratio of 1 : 1 for actors and critic updates.
Note that the action noise is injected while collecting transitions for the replay buffer.
In Fujimoto et al.
, authors noted that the action noise drawn from the OrnsteinUhlenbeck Uhlenbeck & Ornstein (1930) process offered no performance beneﬁts.
Thus we also consider uncorrelated Gaussian noise.
Setup: We evaluate the performance of Algorithm 3 and Algorithm 4 on standard continuous control benchmarks available on OpenAI Gym Brockman et al.
(2016) utilizing the MuJoCo environment Todorov et al.
Speciﬁcally, we benchmark on eight tasks: Walker, Hopper, HalfCheetah, Ant, Swimmer, Reacher, Humanoid, and InvertedPendulum.
Details of these environments can be found in Brockman et al.
(2016) and on the GitHub website.
The Algorithm 3 implementation is based on the codebase from (Tessler et al., 2019).
For all the algorithms, we use a two-layer feedforward neural network structure of (64, 64, tanh) for both actors (agent and adversary) and critic.
The optimizer we use to update the critic is Adam Kingma & Ba (2015) with a learning rate of 10 − 3 .
The target networks are soft-updated with τ = 0 .
For the baseline, the actors are trained with RMSProp optimizer.
For our algorithm, the actors are updated according to Algorithm 1 with warmup steps K t = min (cid:8) 15 , (cid:98) (1 + 10 − 5 ) t (cid:99) (cid:9) , and thermal noise σ t = σ 0 × (1 − 5 × 10 − 5 ) t .
The hyperparameters that are not related to exploration (see
Table 1) are identical to both algorithms that are compared.
And we tuned only the exploration-related hyper-parameters (for both algorithms) by grid search: a) Algorithm 3 with ( σ 0 , σ ) ∈ (cid:8) 10 − 2 , 10 − 3 , 10 − 4 , 10 −
× { b gorithm 4 with σ ∈ { 0 , pair, we identiﬁed the best performing exploration hyperparameter conﬁguration (see Tables 2 and 3).
Each algorithm is trained on 0.5M samples (i.e., 0.5M time steps in the environment).
We run our experiments, for each environment, with 5 different seeds.
The exploration noise is turned off for evaluation.
Evaluation: We evaluate the robustness of both algorithms under different testing conditions, and in the presence of adversarial disturbances in the testing environment.
We train both algorithms with the standard mass variables in OpenAI Gym.
At test time, we evaluate the learned policies by changing the mass values (without adversarial perturbations) and estimating the cumulative rewards.
As shown in Figure 1, our Algorithm 3 outperforms the baseline Algorithm 4 in terms of robustness.
We also evaluate the robustness of the learned policies under both test condition changes, and adversarial disturbances (see Figure 2).
One-Player DDPG: We evaluate the robustness of one-player variants of Algorithm 3, and Algorithm 4, i.e., we consider the NR-MDP setting with δ = 0 .
In this case, we set K t = 1 for
Algorithm 3 (this choice of K t makes the computational complexity of both algorithms equal).
The results are presented in Figures 3 and 4.
Here, we remark that Algorithm 3 with δ = 0 , and K t = 1 is simply the standard DDPG with actor being updated by preconditioned version of SGLD.
Thus we achieve robustness under different testing conditions with just a simple change in the DDPG algorithm and without additional computational cost.
C ONCLUSION
In this work, we studied the robust reinforcement learning problem.
By adapting the approximate inﬁnite-dimensional entropic mirror descent from (Hsieh et al., 2019), we design a robust variant of
DDPG algorithm, under the NR-MDP setting.
In our experiments, we evaluated the robustness of our algorithm on several continuous control tasks, and found that our algorithm clearly outperformed the baseline algorithm from (Tessler et al., 2019).
A A DDITIONAL R ESULTS
The results for the best performing seeds are presented in Figures 5 and 6.
B A LGORITHMS AND H YPERPARAMETER D ETAILS
Most of the values for the hyperparameters (in Table 1) are chosen from Dhariwal et al.
In addition, for Algorithm 3, we set
thermal noise σ t =
σ 0 × (1 − 5 × 10 − 5 ) t , where σ 0 ∈ (cid:8) 10 − 2 , 10 − 3 , 10 − 4 , 10 − 5 (cid:9) .
warmup steps K t = min (cid:8) 15 , (cid:98) (1 + 10 − 5 ) t (cid:99) (cid:9)
The best performing values for every environment are presented in Tables 2 and 3. [Table]
[Table]
Input: Initial distributions p 1 , q 1 , and learning rate η for t = 1 , 2 , . .
{+, T − 1 do p t +
 ( θ ) ∝ exp (cid:16) + η (cid:80) s ≤ t Gq s ( θ ) (cid:17) q t +1 ( ω ) ∝ exp (cid:16) − η (cid:80) s ≤ t G † p s ( ω ) (cid:17)+}
end for
Output: ¯ p T = 1 T (cid:80) Tt =1 p t and ¯ q T = 1 T (cid:80) Tt =
1 q t
[Table]
[Table] grid search (for NR-MDP setting with [Table] =
[Table] [Table]
HalfCheetah
[Table]
Hopper-v (
[Table]
Ant-
[Table] Swimmer-v ( [Table] Reacher-v
[Table] Humanoid-v [Table] InvertedPendulum-
[Table]
Hyperparameters: see Table
Initialize (randomly) policy parameters ω 1 , θ 1 , and Q-function parameter φ .
Initialize the target network parameters ω targ ← ω 1 , θ targ ← θ 1 , and φ targ ← φ .
Initialize replay buffer D .
Initialize m ← 0 ; m (cid:48) ← 0 .
t ← 1 .
repeat
Observe state s , and select actions a = µ θ t ( s ) + ξ ; a (cid:48) =
ν ω t ( s ) + ξ (cid:48) , where ξ, ξ (cid:48) ∼ N (0 , σI )
Execute the action ¯ a = (1 − δ ) a + δa (cid:48) in the environment.
Observe reward r , next state s (cid:48) , and done signal d to indicate whether s (cid:48) is terminal.
Store ( s, ¯ a, r, s (cid:48) , d ) in replay buffer D .
If s (cid:48) is terminal, reset the environment state.
if it’s time to update then for however many updates do ¯ ω t , ω (1) t ← ω t ; ¯ θ t , θ (1) t ← θ t for k = 1 , 2 , . . .
, K t do
Sample a random minibatch of N transitions B = { ( s, ¯ a, r, s (cid:48) , d ) } from D .
Compute targets y ( r, s (cid:48) , d ) =
r + γ (1 − d ) Q φ targ (cid:0) s (cid:48) , (1 − δ ) µ θ targ ( s (cid:48) )
+ δν ω targ ( s (cid:48) )
(cid:1) .
Update critic by one step of (preconditioned) gradient descent using ∇ φ L ( φ ) , where
L (
[Equation]
= 1 N (cid:88) ( s, ¯ a,r,s (cid:48) ,d ) ∈ B ( y ( r, s (cid:48) , d ) −
[Equation]
Compute the (agent and adversary) policy gradient estimates:
[Equation]
←
[Equation]
+ ηC
[Equation]
√
[Equation]
t C
[Equation]
12 ξ , where ξ ∼ N (0 , I )
[Equation]
ω
[Equation]
√ 2 ησ t D
[Equation]
12 ξ (cid:48) , where ξ (cid:48) ∼ N
[Equation]
, I
[Equation]
¯
[Equation]
Update the target networks:
[Equation]
τ ) φ θ targ ← τθ targ + (1 − τ ) θ ( k +1)
t ω targ ← τω targ
(1 − τ ) ω
1) t end for ω
(1 − β ) ω
β ¯ ω
; θ t +1 ← (1 − β ) θ t + β ¯ θ t end for end if until convergence
Output: ω T , θ T .
Algorithm
Hyperparameters: see Table
Initialize (randomly) policy parameters ω 1 , θ 1 , and Q-function parameter φ .
Initialize the target network parameters ω targ ← ω 1 , θ targ ← θ 1 , and φ targ ← φ .
Initialize replay buffer D .
Initialize m ← 0 ; m (cid:48) ← 0 .
t ← 1 .
repeat
Observe state s , and select actions a = µ θ t ( s ) + ξ ; a (cid:48) =
ν ω t ( s ) + ξ (cid:48) , where ξ, ξ (cid:48) ∼ N (0 , σI )
Execute the action ¯ a = (1 − δ ) a + δa (cid:48) in the environment.
Observe reward r , next state s (cid:48) , and done signal d to indicate whether s (cid:48) is terminal.
Store ( s, ¯ a, r, s (cid:48) , d ) in replay buffer D .
If s (cid:48) is terminal, reset the environment state.
if it’s time to update then for however many updates do
Sample a random minibatch of N transitions B = { ( s, ¯ a, r, s (cid:48) , d ) } from D .
Compute targets
[Equation]
Update critic by one step of (preconditioned) gradient descent using ∇ φ L ( φ ) , where
[Equation]
N
[Equation]
Compute the (agent and adversary) policy gradient estimates: (cid:92)
[Equation]
(cid:88)
[Equation]
g ← (cid:104) (cid:92) ∇
[Equation]
J ( θ
[Equation]
ω
[Equation]
←
[Equation]
g
[Equation]
← diag (cid:0)
[Equation]
(cid:15) (cid:1) θ
[Equation]
← θ
[Equation]
ηC
[Equation]
g g (cid:48) ← (cid:104) (cid:92) ∇ ω
[Equation]
( θ
[Equation]
ω
[Equation]
m (cid:48)
[Equation]
(cid:15) (cid:1) ω
[Equation]
← ω
[Equation]
ηD
[Equation]
g (cid:48)
Update the target networks: φ
[Equation]
ω
[Equation]
ω
[Equation]
t ← t + 1 . end for end if until convergence
Output: ω T , θ T .