Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification
Abstract
In the field of reinforcement learning, because of the high cost and risk of policy training in the real world, policies are trained in a simulation environment and transffered to the corresponding real-world environment.
However, differences in the environments lead to model misspecification.
Multiple studies report significant deterioration of policy performance in a real-world environment.
In this study, we focus on scenarios involving a simulation environment with uncertainty parameters and the set of their possible values, called the uncertainty parameter set.
The aim is to optimize the worst-case performance on the uncertainty parameter set to guarantee the performance in the corresponding real-world environment.
To obtain a policy for the optimization, we propose an off-policy actor-critic approach called the Max-Min Twin Delayed Deep Deterministic Policy Gradient algorithm (M2TD3), which solves a max-min optimization problem using a simultaneous gradient ascent descent approach.
Experiments in multi-joint dynamics with contact (MuJoCo) environments show that the proposed method exhibited a worst-case performance superior to several baseline approaches.
Our implementation is publicly available ( https://github.com/akimotolab/M2TD
Introduction
Applications of deep reinforcement learning (DRL) to control tasks that involving interaction with the real world are limited because of the need for interaction between the agent and the real-world environment [37].
This difficulty arises because of high interaction time, agent maintenance costs, and safety issues during the interaction.
Without enough interactions, DRL tends to overfit to specific interaction histories and yields policies with poor generalizability and safety issues [29].
To solve this problem, simulation environments that estimate the characteristics of real-world environments are often employed, which enable enough interactions for effective DRL application.
The policy is trained in the simulation environment and then transferred to the real-world environment [45].
Moreover, because fast and general-purpose simulators, such as Open Dynamics Engine [34], Bullet [3] and multi-joint dynamics with contact (MuJoCo [38]), are available in the field of robotics, the cost of developing a simulation environment can sometimes be significantly lower than that for real-world interactions.
However, because a simulation environment is an estimation of the real-world environment, discrepancies exist between the two [2].
Even if they are entirely similar, discrepancies may still occur, e.g., because of robot degradation over time or weight changes because of part replacements [22].
Studies report significant degradation of policy trained in a simulation environment when transferred to the corresponding real-world environment [8].
The above discrepancies limit the use of simulator environments in practice and, thus, limit the use of DRL.
In this study, to guarantee performance in the corresponding real-world environment, we aim to obtain a policy that maximizes the expected reward under the worst-case scenario in the uncertainty parameter set Ω .
We focus on a scenario where (1) a simulator M ω is available, (2) the model uncertainty is parameterized by ω ∈ Ω , and ω is configurable in the simulation, and (3) the real-world environment is identified with some ω ∗ ∈ Ω , where ω ∗ is fixed during an episode; however, (4) ω ∗ is unknown, is not uniquely determined, or changes from episode to episode 1 . The model of uncertainty described in hypotheses (3) and (4) is referred to as the stationary uncertainty model, in contrast to the time-varying uncertainty model where ω ∗ can vary during an episode [28].
We develop an off-policy deep deterministic actor-critic approach to optimize the worst-case performance, called the Max-Min Twin Delayed Deep Deterministic Policy Gradient Algorithm (M2TD3).
By extending the bi-level optimization formulation of the standard actor-critic method [30], we formulate our problem as a tri-level optimization, where the critic network models a value for each tuple of state, action, and uncertainty parameter, and the actor network models the policy of an agent and an estimate of the worst uncertainty parameter value.
Based on the existing deep deterministic actor-critic approach [19, 7], we design an algorithm to solve the tri-level optimization problem by implementing several technical components to stabilize the training, including multiple uncertainty parameters, their refreshing strategies, the uncertainty parameter sampler for interaction, and the soft-min policy update.
Numerical experiments on 19 MuJoCo tasks reveal the competitive and superior worst-case performance of our proposed approaches compared to those of several baseline approaches.
Preliminaries
We consider an episodic Markov Decision Process (MDP) [6] family M Ω
= {M ω } ω ∈ Ω , where
M ω = ⟨ S, A, p ω , p 0 ω , r ω , γ ⟩ is the MDP with uncertainty parameter ω ∈ Ω .
The state space S and the action space
A are subsets of real-valued vector spaces.
The transition probability density p ω : S ×
A × S → R , the initial state probability density p 0 ω : S → R , and the immediate reward r ω : S ×
A → R depend on ω .
The discount factor is denoted by γ ∈ (0 , 1) .
We focus on an episodic task on M Ω .
Let µ θ : S → A be a deterministic policy parameterized by θ ∈ Θ .
Given an uncertainty parameter ω ∈ Ω , the initial state follows s 0 ∼ p 0 ω .
At each time step t ⩾ 0 , the agent observes state s t , select action a t = µ θ ( s t ) , interacts with the environment, and observes the next state s ′ t ∼ p ω ( · | s t , a t ) , the immediate reward r t = r ω ( s t , a t ) , and the termination flag h t .
Here, h t = 1 if s ′ t is a terminal state or a predefined maximal time step for each past episode; otherwise, h t = 0 .
If h t = 0 , let s t +
= s ′ t ; otherwise, the next state is reset by s t +1 ∼ p 0 ω .
For simplicity, we let q ω
( s t +1 | s t , a t ) be the probability density of s t +
given s t and a t .
The discount reward of the trajectory starting from time step t is R t = (cid:80) k ⩾ 0 γ k r t + k .
The action value function Q µ θ ( s, a, ω ) under ω is the expectation of R t starting with s t =
s and a t = a under ω ; that is, Q µ θ ( s, a, ω )
=
E [ R t | s t = s, a t = a, s t + k +1 ∼ q ω
( · | s t + k , µ θ ( s t + k ))
∀ k ⩾ 0] , where the expectation is taken over s t
+ k +1 for k ⩾ 0 .
Note that we introduce ω to the argument to explain the Q -value dependence on ω ; however, this is essentially the same definition as that for the standard action value function Q µ θ ( s, a ) .
The recursive formula is as follows:
[Equation]
We consider an off-policy reinforcement learning setting, where the agent interacts with M ω using a stochastic behavior policy β : S ×
A → R .
However, the target policy to be optimized, denoted by µ θ , is deterministic.
Under a fixed ω ∈ Ω , the standard objective of off-policy learning is to maximize the expected action value function over the stationary distribution ρ β , such that
[Equation]
Here, ρ β ( s | ω ) =
lim T →∞ 1 T (cid:80) T − 1 t =0 (cid:82) s 0 q tω,β
( s | s 0 )
p 0 ω ( s 0 )d s 0 denotes the stationary distribution under β and a fixed ω ∈ Ω , where the step- t transition probability density q tω,β is defined as q 1 ω,β ( s
[Equation]
| s )
= (cid:82)
a ∈
A q ω ( s ′ | s, a ) β ( a | s ) d a and q tω,β ( s ′ | s ) = (cid:82) ¯ s ∈ S q t − 1 ω,β (¯ s | s ) q 1 ω,β ( s ′ | ¯ s )d¯ s . We assume that the agent can interact with M ω for any ω ∈ Ω during the training and can change ω after every episode, that is, when h t = 1 .
Our objective is to obtain the µ θ that maximizes J ω ( θ ) under the worst environment ω ∈ Ω .
Hence, we tackle the max-min optimization problem requiring max θ ∈ Θ min ω ∈ Ω J ω ( θ ) .
M2TD
We propose an off-policy actor-critic approach to obtain a policy that maximizes the worst-case performance on M Ω , called M2TD3.
Our approach is based on TD3, an off-policy actor-critic algorithm for a deterministic target policy.
In TD3, the critic network Q ϕ parameterized by ϕ ∈ Φ is trained to approximate the Q µ θ of µ θ , whereas the actor network models µ θ with parameter θ ∈ Θ and is trained to maximize J ω ( θ ) .
Moreover, ω is not considered and is fixed during the training.
In this study, to obtain the µ θ that maximizes the performance under the worst ω , we extend TD by formulating the objective as a maximin problem and introducing a simultaneous gradient ascent descent approach.
The main difficulty in the worst-case performance maximization is in estimating the worst-case performance.
Because the objective is expected to be non-concave with respect to (w.r.t.) the uncertainty parameter because of deep actor-critic networks, solving this problem is considered intractable in general [4].
To stabilize the worst-case performance estimation, we introduce various techniques: multiple uncertainty parameters, an uncertainty parameter refresh strategy, and an uncertainty parameter sampler.
Formulation
We can formulate our objective as the following tri-level optimization:
[Equation]
where J and L are defined below.
Note that this tri-level optimization formulation is an extension of the bi-level optimization formulation of the actor-critic approach proposed in [30].
We introduce a probability density α : Ω → R , from which an ω to be used in the interaction during the training phase is drawn for each episode.
The relationship between ˆ ω and α is similar to that between θ and β : namely, ˆ ω and θ are the parameters to be optimized while α and β are introduced for exploration.
Let ρ βα ( s )
=
(cid:82) ω ∈ Ω ρ β ( s | ω ) α ( ω )d ω and ρ βα ( s, a, ω ) =
β ( a | s ) ρ β ( s | ω ) α ( ω ) be the stationary distribution of s and the joint stationary distribution of ( s, a, ω ) , respectively, under α and β .
The critic loss function L ( ϕ ; θ ) is designed to simulate the Q-learning algorithm.
Let T µ θ be the function satisfying
[Equation]
| s, a )d s ′ .
Then, (1) states that Q µ θ is the solution to Q = T µ θ [ Q ] .
Therefore, the critic is trained to minimize the difference between Q ϕ and T µ θ [ Q ϕ ] .
This is achieved by minimizing
[Equation]
The max-min objective function of the actor network
[Equation]
measures the performance of µ θ under the uncertainty parameter ˆ ω ∈ Ω approximated using the critic network Q ϕ instead of the ground truth Q µ θ .
Similar to the standard actor-critic approach, we expect
Q ϕ to approach Q µ θ as the critic loss is minimized.
Therefore, we expect J ( θ, ˆ ω ; ϕ ∗ ) approximates (cid:82) s ∈ S Q µ θ ( s, µ θ ( s ) , ˆ ω ) ρ βα ( s )d s once we obtain ϕ ∗ ≈
argmin ϕ ∈
Φ L ( ϕ ; θ ) .
Particularly, even if Q ϕ ∗ =
Q µ θ , our objective function J ( θ, ˆ ω ; ϕ ∗ ) differs from J ω ( θ ) with ω = ˆ ω in (2), in that the stationary distribution ρ β ( s | ω ) under fixed ω in J ω is replaced with ρ βα ( s ) under ω ∼ α .
However, this change allows us to effectively utilize the replay buffer, which stores the interaction history, to approximate the objective J ( θ, ˆ ω ; ϕ ) .
Moreover, if α is concentrated at ˆ ω ,
J ( θ, ˆ ω ; ϕ ∗ ) coincides with J ω
=ˆ ω ( θ ) .
Algorithmic Framework
The framework of M2TD3 is designed to solve the tri-level optimization (3).
The overall framework of M2TD3 follows that of TD3.
In each episode, an uncertainty parameter ω is sampled from α .
The training agent interacts with the environment M ω using behavior policy β .
At each time step t , the transition ( s t , a t , r t , s ′ t , h t , ω ) is stored in the replay buffer, which is denoted by B .
The critic network Q ϕ is trained at every time step, to minimize (4), with a mini-batch being drawn uniformly randomly from B .
The actor network µ θ and the worst-case uncertainty parameter ˆ ω are trained in every T freq step to optimize (5).
Note that a uniform random sample ( s, a, ω ) taken from B can be regarded as a sample from the stationary distribution ρ βα ( s, a, ω ) .
Similarly, s taken from B is regarded as a sample from ρ β α ( s ) .
These facts allow approximation of the expectations in (4) and (5) using the Monte Carlo method with mini-batch samples uniformly randomly taken from B . An algorithmic overview of the proposed method is summarized in Algorithm 1.
We have underlined the differences from the general off-policy actor-critic method in the episodic settings.
A detailed description of M2TD3 is summarized in Algorithm 2 in Appendix A. As shown from
Algorithm 1, the differences between the proposed method and the general off-policy actor-critic are as follows: (1) the introduction of an uncertainty parameter sampler, (2) definition and updating method of critic, and (3) updating method of actor and uncertainty parameters.
1: Draw uncertainty parameter ω ∼ α2: Observe initial state s ∼ p 0 ω 3: for t = 1 to T max do 4: # interaction 5:
Select action a ∼ β t ( s ) 6:
Interact with M ω with a , observe next state s ′ , immediate reward r and termination flag h 7:
Store transition tuple ( s, a, r, s ′ , h, ω ) in B 8: if h = 1 then 9:
Reset uncertainty parameter ω ∼ α t 10: Observe initial state s ∼ p 0 ω 11: else 12: Update current state s ← s ′ 13: end if 14: # learning 15: Sample mini-batch { ( s i , a i , r i , s ′
i , h i , ω i ) } of M transitions uniform-randomly from B 16:
Update the critic network by optimizing Equation (6) 17:
Update the actor network and the uncertainty parameter by optimizing Equation (7) 18:
Update uncertainty parameter sampler α 19: end for Critic Update The critic network update, namely, minimization of (4), is performed by implementing the TD error-based approach.
The concept is as follows.
Let { ( s i , a i , r i , s ′
i , ω i ) } Mi =
⊂ B be the mini-batch, and let θ t and ϕ t be the actor and critic parameters at time step t , respectively.
For each tuple, T µ θt [ Q ϕ t ]( s i , a i , ω i ) is approximated by y i =
r i + γ · Q ϕ t ( s ′ i , µ θ t ( s ′ i ) , ω i ) .
Then, L ( ϕ t ; θ t ) in (4) is approximated by the mean square error ˜ L ( ϕ t ) , such that
[Equation]
Then, the update follows ϕ t +1 =
ϕ t − λ ϕ ∇ ϕ ˜ L ( ϕ t ) .
We incorporate the techniques introduced in TD3 to stabilize the training: clipped double Q-learning, target policy smoothing regularization, and target networks.
Additionally, we introduce the smoothing regularization for the uncertainty parameter so that the critic is smooth w.r.t.
ˆ ω .
Algorithm 3 in
Appendix A summarizes the critic update and the details of each technique are available in [7].
Actor Update
Updating of the actor as well as the uncertainty parameter, namely, maximin optimization of (5), is performed via simultaneous gradient ascent descent [27], summarized in
Algorithm 4 in Appendix A. Let ϕ t , θ t and ˆ ω t be the critic, policy, and the uncertainty parameters, respectively, at time step t .
Instead of the optimal ϕ ∗ in (3), we use ϕ t as its estimate.
With the mini-batch { s i } Mi =1 ⊂ B , we approximate the max-min objective (5) follows:
[Equation]
We update θ and ˆ ω as θ t +
=
θ t
+ λ θ ∇ θ ˜ J t ( θ t , ˆ ω t ) and ˆ ω t
+
=
ˆ ω t − λ ω ∇ ˆ ω ˜
J t ( θ t , ˆ ω t ) , respectively.
For higher-stability performance, we introduce multiple uncertainty parameters.
The motivation is twofold.
One is to deal with multiple local minima of J t ( θ, ˆ ω ; ϕ ∗ ) w.r.t.
ˆ ω .
As the critic network becomes generally non-convex, there may exist multiple local minima of J t ( θ, ˆ ω ; ϕ ∗ ) w.r.t.
ˆ ω .
Once ˆ ω is stacked at a local minimum point, e.g., ˆ ω ∗ , θ may be trained to be robust around a neighborhood of ˆ ω ∗ and to perform poorly outside that neighborhood.
The other is that the maximin solution ( θ ∗ , ϕ ∗ ) of J t ( θ, ˆ ω ; ϕ ∗ ) is not a saddle point, which occurs when the objective is non-concave in ˆ ω [14].
Here, more than one ˆ ω is necessary to approximate the worstcase performance min ˆ ω J t ( θ, ˆ ω ; ϕ ∗ ) around ( θ ∗ , ϕ ∗ ) and a standard simultaneous gradient ascent descent method fails to converge.
To relax this defect of simultaneous gradient ascent descent methods, we maintain multiple candidates for the worst ˆ ω , denoted by ˆ ω 1 , . . .
, ˆ ω N .
Therefore, replacing min ˆ ω J ( θ, ˆ ω ; ϕ ∗ ) with min k =1 ,...,N J ( θ, ˆ ω k ; ϕ ∗ ) in (5) has no effect on the optimal θ ; however, this change does affect the training behavior.
Our update follows the simultaneous gradient ascent descent on min k =1 ,...,N J t ( θ, ˆ ω k ) : θ ← θ + λ θ ∇ θ (min k =1 ,...,N J t ( θ, ˆ ω k )) and ˆ ω k ← ˆ ω k − λ ω ∇ ˆ ω k (min ℓ =1 ,...,N J t ( θ, ˆ ω ℓ )) for k = 1 , . . .
, N .
Hence, θ is updated against the worst ˆ ω k , and only the worst ˆ ω k is updated because the gradient w.r.t.
the other ˆ ω k is zero.
For multiple uncertainty parameters to be effective, they must be distant from each other.
Moreover, all are expected to be selected as the worst parameters with non-negligible frequencies.
Otherwise, the advantage of having multiple uncertainty parameters is lessened.
From this perspective, we introduce a refreshing strategy for uncertainty parameters.
Namely, we resample ˆ ω k ∼ U (Ω) if one of the following scenarios is observed: there exists ˆ ω ℓ such that the distance d ˆ ω (ˆ ω k , ˆ ω ℓ ) ⩽ d thre ; the frequency p k of ˆ ω k being selected as the worst-case during the actor update is no greater than p thre .
Uncertainty Parameter Sampler
The uncertainty parameter sampler α controls the explorationexploitation trade-off in the uncertainty parameter.
On the one hand, exploration in Ω is necessary to train the critic network.
If the critic network is not well trained over Ω , it is difficult to locate the worst ˆ ω correctly.
On the other hand, for J ( θ, ˆ ω ; ϕ ∗ ) in (5) to coincide with J ω ( θ ) for ω = ˆ ω in (2), we require α to be concentrated at ˆ ω .
Otherwise, the optimal θ for J ( θ, ˆ ω ; ϕ ∗ ) may deviate from that of J ω ( θ ) .
We design α as follows.
For the first T rand steps, we sample the uncertainty parameter ω uniformly randomly on Ω , i.e., α = U (Ω) .
Then, we set α = (cid:80) Nk
=1 p k ·N (ˆ ω k , Σ ω ) , where Σ ω is the predefined covariance matrix.
We decrease Σ ω as the time step increases.
The rationale behind the gradual decrease of Σ ω is that the training of the critic network is still important after T rand to make the estimation of the worst-case uncertainty parameter accurate.
Details are provided in Appendix C.
Related Work
Methods to handle model misspecification include (1) robust policy searching [26, 31, 23, 17, 12, 22], (2) transfer learning [36, 43, 32], (3) domain randomization (DR) [37], and (4) approaches minimizing worst-case sub-optimality gap [5, 21].
Robust policy searching includes methods that aim to obtain the optimal policy under the worst possible disturbance or model misspecification, i.e., maximization of min ω ∈ Ω J ω ( θ ) for an explicitly or implicitly defined Ω .
Transfer learning is a method in which a policy is trained on source tasks and then fine-tuned through interactions while performing the target task.
DR is a method in which a policy is trained on source tasks that are sampled randomly from M Ω .
There are two types of DR approaches: vanilla and guided.
In vanilla DR [37], the policy is trained on source tasks that are randomly sampled from a pre-defined distribution on Ω .
In guided DR [44], the policy is trained on source tasks; however, the ω distribution is guided towards the target task.
Because we do not assume access to the target task, transfer learning and many guided DR approaches are outside the scope of this work.
Some guided DR approaches, such as active domain randomization [24], do not access the target task or consider worst-case optimization either.
Vanilla DR can be applied to the setting considered here.
However, the objective of vanilla DR differs from the present aim, i.e., to maximize the performance averaged over Ω , namely, E ω [ J ω ( θ ) | ω ∼ U
(Ω)] , if the sampling distribution is U (Ω) .
The approaches minimizing the worst-case sub-optimality gap [5, 21] do not optimize the worst-case performance, instead attempt to obtain a policy that generalizes well on the uncertainty parameter set while avoiding too conservative performance, which often attributes to the worst-case performance optimization.
Some robust policy search methods, some adopt an adversarial approach to policy optimization.
For example, Robust Adversarial Reinforcement Learning (RARL) [31] models the disturbance caused by an external force produced by an adversarial agent, and alternatively trains the protagonist and adversarial agents.
Robust Reinforcement Learning (RRL) [26] similarly models the disturbance but has not been applied to the DRL framework.
Minimax Multi-Agent Deep Deterministic Policy
Gradient (M3DDPG) [17] has been designed to obtain a robust policy for multi-agent settings.
This method is applicable to the setting targeted in this study if only two agents are considered.
The above approaches frame the problem as a zero-sum game between a protagonist agent attempting to optimize µ θ and an adversarial agent attempting to minimize the protagonist’s performance, hindering the protagonist by generating the worst possible disturbance.
Adv-Soft Actor Critic (AdvSAC) [13] learns policies that are robust to both internal disturbances in the robot’s joint space and those from other robots.
Recurrent SAC (RSAC) [42] introduces POMDPs to treat the uncertainty parameter as an unobservable state.
A DDPG based approach robustified by applying stochastic gradient Langevin dynamics is proposed under the noisy robust MDP setting [15].
However, it has been reported in [42, 15] that their worst-case performances are sometimes even worse than their baseline non-robust approaches.
State-Adversarial-DRL (SA-DRL) [48], and alternating training with learned adversaries (ATLA) [47] improve the robustness of DRL agents by using an adversary that perturbs the observations in SAMDP framework that characterizes the decision-making problem in an adversarial attack on state observations.
They do not address the model misspecification in the reward and transition functions.
Other approaches attempt to estimate the robust value function, i.e., a value function under the worst uncertainty parameter.
Among them, Robust Dynamic Programming (DP) [12] is a DP approach, while the Robust Options Policy Iteration (ROPI) [23] incorporates robustness into option learning [35], which allows agents to learn both hierarchical policies and their corresponding option sets.
ROPI is a type of Robust MDP approach [39, 20].
Robust Maximum A-posteriori Policy Optimization (R-MPO) [22] incorporates robustness in MPO [1].
Neither Robust MDP nor R-MPO require interaction with M ω for an arbitrary ω ∈ Ω .
This can be advantageous in scenarios where the design of a simulator valid over Ω is tedious.
However, these methods typically require additional assumptions for computating the worst-case for each value function update, such as the finiteness of the state-action space and/or the finiteness of Ω . To apply
R-MPO to our setting, finite scenarios from Ω for training are sampled, but the choice of the training scenarios affects the worst-case performance.
Additionally, to optimize the worst-case performance in the field of offline RL [40, 46, 41], offline
RL attempts to obtain robust measures by introducing the principle of pessimism.
Some studies that introduce the principle of pessimism in the model-free context [40, 41] and in the actor-critic context [46].
However, these do not compete with this study due to differences in motivation and because offline RL requires additional assumptions in the datasets [46, 41] and realizability [40].
Table 5 in Appendix D summarizes the robust policy search method taxonomy.
Our proposed approach, M2TD3, considers the uncertainties in both the reward function and transition probability.
The state and action spaces, as well as the uncertainty set, are assumed to be continuous.
Tablein Appendix D compares the related methods considering their action value function definitions.
Methods that do not explicitly utilize the action value function are interpreted from the objective functions.
Both M3DDPG and M2TD3 maintain the value function for the tuple of ( s, a, ω ) .
However, M3DDPG takes the worst ω ′ on the right-hand side, yielding a more conservative policy than M2TD for our setting because we assume a stationary uncertainty model, whereas M3DDPG assumes a time-varying uncertainty model in nature [28].
The value functions in RARL and M2TD3 are identical if ω is fixed.
Particularly, ω is not introduced to the RARL value function because it does not simultaneously optimize µ θ and the estimate of the worst ω .
RARL repeats the θ and ω optimizations alternatively.
In contrast, M2TD3 optimizes θ and ω simultaneously in a manner similar to the training processes of generative adversarial networks (GANs) [9].
Hence, we require an action value for each ω ∈ Ω .
A difference exists between the optimization strategies of RARL and M2TD3.
As noted in the aforementioned section, both methods attempt to maximize the worst-case performance min ω ∈ Ω J ω ( θ ) .
Conceptually, RARL repeats θ ← argmax θ J ω ( θ ) and ω ← argmin ω J ω
( θ ) .
However, this optimization strategy fails to converge even if the objective function ( θ, ω ) (cid:55)→ J ω ( θ ) is concave-convex.
As an example, consider the function ( x, y ) (cid:55)→ y 2 − x 2 + αxy .
The RARL optimization strategy reads x ← ( α/ 2) y
and y ← − ( α/ 2) x , which causes divergence if α > 2 .
2 Alternating updates are also employed in other approaches such as Adv-SAC, SA-DRL, and ATLA.
These share the same potential issue as RARL.
M2TD3 attempts to alleviate this divergence problem by applying the gradient-based max-min optimization method, which has been employed in GANs and other applications and analyzed for its convergence [25, 18].
Experiments
In this study, we conducted experiments on the optimal control problem using MuJoCo environments.
Hence, we demonstrated the problems of existing methods and assessed the worst-case performance and average performance of the policy trained by M2TD3 for different continuous control tasks.
Baseline Methods
We summarize the baseline methods adapted to our experiment setting, namely,
DR, RARL, and M3DDPG.
DR: The objective of DR is to maximize the expected cumulative reward for the distribution α = U (Ω) of ω .
In each training episode, ω is drawn randomly from U (Ω) , and the agent neglects ω when training and performing the standard DRL.
For a fair comparison, we implemented DR with TD3 as the baseline DRL method.
RARL: We adapted RARL to our scenario by setting µ ˆ ω : s (cid:55)→ ˆ ω to the antagonist policy.
RARL was regarded as optimizing (2) for the worst ω ∈ Ω ; hence, the objective was the same as that of M2TD3.
Particularly, the main technical difference between M2TD3 and RARL is in the optimization strategy as described in Section 4.
The original RARL is implemented with Trust Region Policy Optimization (TRPO) [33], but for a fair comparison, we implemented it with DDPG, denoted as RARL (DDPG).
The experimental results of RARL (TRPO), the original RARL, are provided in Appendix F.
M3DDPG:
By considering a two-agent scenario and a state-independent policy µ ˆ ω : s (cid:55)→ ˆ ω as the opponent-agent’s policy, we adapted M3DDPG to our setting.
M3DDPG is different from M2TD (and M2-DDPG below) even under a state-independent policy as described in Section 4.
Because of the difficulty in replacing DDPG with TD3 in the M3DDPG framework, we used DDPG as the baseline DRL.
Additionally, for comparison, we implemented our approach with DDPG instead of TD3, denoted by M2-DDPG.
We also implemented a variant of M2TD3, denoted as SoftM2TD3, performing a “soft” worst-case optimization to achieve better average performance while considering the worst-case performance.
The detail of SoftM2TD3 is described in Appendix B.
Experiment Setting
We constructed 19 tasks based on six MuJoCo environments and with 1uncertainty parameters, as summarized in Table 4 in Appendix C.
Here, Ω was defined as an interval, which mostly included the default ω values.
To assess the worst-case performance of the given policy µ under ω ∈ Ω , we evaluated the cumulative reward 30 times for each uncertainty parameter value ω 1 , . .
. , ω K ∈ Ω .
Here, R k ( µ ) was defined as the cumulative reward on ω k averaged over 30 trials.
Then, R worst ( µ ) = min 1 ⩽ k ⩽ K R k ( µ ) was [Table] [Figure]
l a t i v
[Figure] r [Figure] measured as an estimate of the worst-case performance of µ on Ω .
We also report the average performance R average ( µ ) = 1 K (cid:80)
Kk =1 R k ( µ ) .
A total of K uncertainty parameters ω 1 , . . .
, ω K for evaluation were drawn as follows: for 1D ω , we chose K = 10 equally spaced points on the 1D interval Ω ; for 2D ω , we chose 10 equally spaced points in each dimension of Ω , thereby obtaining
K = 100 points; and for 3D ω , we chose 10 equally spaced points in each dimension of Ω , thereby obtaining K = 1000 points.
For each approach, we trained the policy 10 times in each environment.
The training time steps T max were set to 2M, 4M, and 5M for the environments with 1D, 2D, and 3D uncertainty parameters, respectively.
The final policies obtained were evaluated for their worst-case performances.
For further details of the experiment settings, please refer to Appendix C
Comparison to Baseline Methods
Table 1 summarizes the worst-case performances of the policies trained by M2TD3, SoftM2TD3, M2-DDPG, M3DDPG, RARL, and DR. See also Table 7 in Appendix E for the results of TD3 trained on the reference uncertainty parameters as baselines and
Figure 2 in Appendix G for the learning curves.
In most cases, M2TD3 and SoftM2TD3 outperformed
DR, and RARL, and M3DDPG.
Figure 1 shows the cumulative rewards of the policies trained by the six approaches for each of the ω values on the Ant 1, InvertedPendulum 1, and Walker 1 environments.
See also Figure 4 in Appendix H for other environments.
Comparison with DR: DR does not attempt to optimize the worst-case performance.
In fact, it showed lower worst-case performance than M2TD3 and SoftM2TD3 in many scenarios because the obtained policy performs poorly on some uncertainty parameters while it performs well on average.
In the results of InvertedPendulum 1 shown in Figure 1, for example, the policy obtained by DR [Table] [Table] exhibited a high performance for a wide range of ω ∈ Ω but performed poorly for small ω .
M2TD outperformed DR and the other baselines on those environments.
However, in some scenarios, such as HumanoidStandup 1, Small HalfCheetah [Table] and Small Hopper 1, DR achieved a better worst-case performance than M2TD3.
This outcome may be because the optimization of worst-case function values is generally more unstable than that of the expected function values.
Comparison with RARL: M2-DDPG outperformed RARL in most environments, with this performance difference originating from the different optimization strategies.
As noted above, the optimization strategy employed in RARL often fails to converge.
Specifically, as shown in Figure 1, RARL failed to optimize the policy not only in the worst-case but also on average in Ant 1 and
Walker 1 environments.
In the InvertedPendulum 1 environment, RARL could train the policy for some uncertainty parameters, but not for the worst-case uncertainty parameter.
Comparison with M3DDPG: In many cases, M2-DDPG outperformed M3DDPG.
Because M3DDPG considers a stronger adversary than necessary (the worst ω for each time step), it was too conservative for this experiment and exhibited lower performance.
Average Performance
Although our objective is maximizing the worst-case performance, the average performance is also important in practice.
Table 2 compares the average performance of the six approaches.
Generally, DR achieved the highest average performance as expected.
However, interestingly, M2TD3 and SoftM2TD3 achieved competitive average performances to DR on several environments such as Hopper 1, 2, and Walker 1, 2.
Moreover, Table 1 compared with Table 2 shows that a few times higher average performance than worst-case performance on several environments such as Ant 3 and Hopper 1–3.
Ablation Study
Ablation studies were conducted on InvertedPendulum 1 with the pole mass as the uncertainty parameters.
M2TD3 with N = 5 was taken as the baseline.
We tested several variants as follows: with different multiple uncertainty parameters ( N = 1 and 10 ), without a distance-based refreshing strategy (w/o DRS), and without a probability-based refreshing strategy (w/o PRS).
Tableshows the results.
The inclusion of multiple uncertainty parameters and the probability-based refresh strategy contributed significantly to the worst-case performance and average performance, implying that both techniques contribute to better estimating the worst uncertainty parameter.
Although DRS had little impact on the performance, prior knowledge in DRS can be implemented by defining the distance and the threshold in the uncertainty parameter task-dependently, which we did not implement here.
Small Uncertainty Set (Limitation of M2TD3)
Small HalfCheetah 1 and Small Hopper 1 were designed to have a smaller uncertainty parameter set than HalfCheetah 1 and Hopper 1 and to reveal the effect of the size of the uncertainty parameter interval.
As expected, a smaller uncertainty parameter set resulted in higher worst-case (Table 1) and average (Table 2) performances, for all approaches.
Because M2TD3 performs the worst-case optimization, it is expected to show better worst-case performance than DR, independently of the size of the uncertainty parameter set.
However, in these environments, DR showed better worst-case performance than that of M2TD3, while M2TD achieved competitive or superior worst-case performance to those of DR on HalfCheetah 1 and Hopper 1.
This may be because the max-min optimization performed by M2TD3 resulted in suboptimal policies.
When the uncertainty parameter set is small, and the performance does not change significantly over the uncertainty parameter set, maximizing the average performance is likely to lead to high worst-case performance.
The sub-optimal results obtained by M2TD3 is then dominated by the results obtained by DR.
Therefore, the maxmin optimization in M2TD3 can be improed.
Evaluation Under Adversarial External Force
Although we developed the proposed approach for the situation that the uncertainty parameter is directly encoded by ω , we can extend it to the situation where the model misspecification is expressed by an external force produced by an adversarial agent as in [31].
The extended approach and its experimental result are given in Appendix J. Superior worst-case performances of M2TD3 over DR, RARL, and M3DDPG were observed for this setting as well.
Conclusion
In this study, we targeted the policy optimization aimed at maximizing the worst-case performance in a predefined uncertainty set.
The list of the contributions are as follows.
(i)
We formulated the off-policy deterministic actor-critic approach to the worst-case performance maximization as a tri-level optimization problem (3).
(ii) We developed the worst-case performance of M2TD3.
The key concepts were the incorporation of the uncertainty parameter into the critic network and use of the simultaneous gradient ascent descent method.
Different technical components were introduced to stabilize the training.
(iii)
We evaluated the worst-case performance of M2TD3 on 19 MuJoCo tasks through comparison with three baseline methods.
Ablation studies revealed the usefulness of each component of M2TD Acknowledgements
This research is partially supported by the JSPS KAKENHI Grant Number 19H04179 and the NEDO
Project Number JPNP18002.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] Please see Section 3 and Section 5.
(b) Did you describe the limitations of your work?
[Yes] Please see paragraph titled "Small Uncertainty Set" in Section 5 and Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Please see Appendix A.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please see Section 5 and Appendix C. (c) Did you report error bars (e.g., w.r.t.
the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix C. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] (b) Did you mention the license of the assets? [Yes] Code licence is part of the repository. (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We include our code in the Appendix A. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] 5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] A Algorithm Details Algorithm 2 summarizes the overall framework of M2TD3.
The training is stalled if the size of the replay buffer is smaller than the minibatch size, i.e., if | B | < M .
Algorithms 3 and 4 show the critic network update and the actor network and uncertainty parameter sampler update, respectively.
AlThough we write the gradient-based update in the form of a mini-batch stochastic gradient update for simplicity, we employ an adaptive approach such as ADAM [16].
We maintain the frequency p k of each uncertainty parameter ˆ ω k being the worst one among ˆ ω 1 , . . .
, ˆ ω N in Algorithm 4.
This is used in two ways: criteria for the refreshing strategy of ˆ ω k in Algorithm 4; and mixture weights for the uncertainty parameter sampler α .
The update of p k follows the exponential moving average with the momentum (1 /T last ) , where T last is the number of steps spent in the last episode ( T last is set to 1000 for the first episode).
The reason behind this design choice is as follows.
The short episode is a meaning that a bad uncertainty parameter ω is used in the last episode.
Because the uncertainty parameter ω used in the interaction is sampled from α , which is a gaussian mixture with components centered at ˆ ω 1 , . . .
, ˆ ω k , it implies that they include a bad uncertainty parameter as well.
Then, there is a high chance that this uncertainty parameter is selected as the worst uncertainty parameter ˆ ω k ′ in Algorithm 4. By setting a greater momentum (1 /T last ) , we can accelerate the approach of p k to I { k = k ′ } , which is 1 if k = k ′ and 0 otherwise. With this fast update of p k , we expect two consequences. First, there are increased chances to sample ω around the worst ˆ ω k ′ .
Second, there are increased chances to refresh the non-worst uncertainty parameters, which leads to more exploration of the worst uncertainty parameter search.
Preliminary experiments have confirmed that this momentum setting leads to a better worst-case performance than a constant momentum.
Our implementation is publicly available ( https://github.com/akimotolab/M2TD3 ).
B Soft-Min Variant: SoftM2TD
In theory, M2TD3 can obtain a policy that exhibits a better worst-case performance than a policy obtained by DR as DR does not explicitly maximize the worst-case performance.
However, we empirically observe that DR sometimes achieves a better worst-case performance than M2TD on environments where the performance does not change significantly over the uncertainty set Ω
1: # initialization 2: Initialize uncertainty parameters ˆ ω i ∈ U (Ω) for i = 1 , . . .
, N 3: Initialize policy parameter θ and critic parameters ϕ 1 , ϕ 2 with random values 4: Initialize target network parameters θ ′ ← θ , ϕ ′ 1 ← ϕ 1 , ϕ ′ 2 ← ϕ5: Initialize frequency parameter p 1 , . . .
, p N = 1 /N 6: Initialize replay buffer B = ∅ 7: # training loop 8: Draw uncertainty parameter ω ∼ α9: Observe initial state s ∼ p 0 ω 10: for t = 1 to T max do 11: # interaction 12:
Select action a ∼ β t ( s ) 13: Interact with M ω with a , observe next state s ′ , immediate reward r and termination flag h 14: Store transition tuple ( s, a, r, s ′ , h, ω ) in B 15: if h = 1 then 16: Reset uncertainty parameter ω ∼ α t 17:
Observe initial state s ∼ p 0 ω 18: else 19: Update current state s ← s ′ 20: end if 21: # learning 22: Sample mini-batch { ( s i , a i , r i , s ′
i , h i , ω i ) } of M transitions uniform-randomly from B 23:
{+Perform Algorithm 3 for critic network update 24: if mod( t, T freq ) = 
+}
then 25:
Perform Algorithm 4 for actor network update and uncertainty parameter sampler update 26:
Update target networks analogously to TD 27: end if 28: end for Algorithm 1: for i = 1 , . . . , M do 2: ˜ a i ← µ θ ′ ( s ′ i )
+ ϵ a , ϵ a ∼ Π
{+a ( N (0 , ˜Σ a )) 
+}
: ˜ ω ′
i ← ω
i + ϵ ω , ϵ ω ∼ Π ω ( N (0 , ˜Σ ω )) 4: y i ← r i
{++ min { Q ϕ ′ 
+}
( s ′ i , ˜ a i , ˜ ω i ) , Q ϕ ′ 2 ( s ′ i , ˜ a i , ˜ ω i ) } 5: end for 6: for j = 1 , 2 do 7:
[Equation]
8: end for (Table 1).
We conjecture that this is because the difficulty in the max-min optimization of M2TD compared to the optimization of the expectation in DR.
To mitigate this issue, we propose a variant of M2TD3, called SoftM2TD3.
The objective of the update of the policy parameter θ is replaced with the following soft-min version:
[Equation]
where w k is the weight for uncertainty parameter ˆ ω k . A greater weight value should be assigned to ˆ ω k with smaller Q-values. We used the frequency p k of ˆ ω k as the worst-case during the actor update, i.e., w k = p k . This update is close to M2TD3 if p = ( p 1 , . .
. , p N ) is close to a one-hot vector and is close to DR if p 1 ≈ · · · ≈ p N , which is the case at the beginning of the training. The objective function of SoftM2TD3 is considered an approximation of the objective function of M2TD3. The difference between M2TD3 and SoftM2TD3 is in the optimization process.
Because the objective function of SoftM2TD3 is closer to that of DR, we expect that the optimization in
1: # maximin update 2: k ′ =
argmin k 1 M (cid:80) Mi
=1 Q ϕ 1 ( s i , µ θ ( s i ) , ˆ ω k ) 3: θ ← θ + λ θ ∇ θ 1 M (cid:80) Mi
=1 Q ϕ 1 ( s i , µ θ ( s i ) , ˆ ω k ′ ) 4: ˆ ω k ′ ← ˆ ω k ′ − λ ω ∇ ˆ ω k ′ 1 M (cid:80) Mi =1 Q ϕ 1 ( s i , µ θ ( s i ) , ˆ ω k ′ ) 5: # uncertainty parameter sampler update with refreshing strategy 6: for k = 1 , . . .
, N do 7: ˆ ω k ∼ U (Ω) if d ω (ˆ ω k , ˆ ω ℓ ) ⩽ d thre for some ℓ ̸ = k 8: ˆ ω k ∼ U (Ω) if p k ⩽ p thre 9: if ˆ ω k is refreshed then 10: p k ← 1 /N 11: else 12: p k ← (1 − 1 /T last )
p k + (1 /T last ) I { k = k ′ } 13: end if 14: end for 15: p k ← p k / (cid:80) nℓ =1 p ℓ for all k = 1 , . . .
, N [Table]
SoftM2TD3 is more efficient than M2TD3, where the effect of the accuracy of the estimation of the worst-case uncertainty parameters is greater than SoftM2TD3.
Table 4 lists the environment we used in our experiments. These environments are created by setting 1 to 3 constants in the original MuJoCo environment to the uncertainty parameters. The uncertainty set is designed as an interval.
Except for Hopper 2 and Hopper 3, the original value is included in the
[Table] [Table]
uncertainty set; hence, the trivial upper bound of the worst-case performance is the best performance of the corresponding MuJoCo environment
[Table]
In all approaches in all environments
[Table]
the same configurations are used for fair comparison as follows.
For DR and RARL, the input to the critic is a ( s, a ) pair, whereas it is a tuple of ( s, a, ω ) for M2TD3,
SoftM2TD3, and M3DDPG.
Except for this point, we used the same network architecture.
The policy and the critic networks are defined as fully connected layers with two hidden layers of size 256.
The uncertainty parameter sampler α t is
[Equation]
where Σ ω is a diagonal matrix, with diagonal elements 0.5 times the lengths of the intervals of the corresponding dimension of Ω , and Π Ω is the projection onto Ω and T rand = 10 5 .
The behavior policy β t is
[Equation]
where Σ a is a diagonal matrix, with diagonal elements are the 0.5 times the length of the intervals of the corresponding dimension of A , and Π A is the projection onto A .
The Σ ω element is designed to decay at each time step to 0.05 when the learning progresses to half of the total.
The minibatch size is M = 100 .
The learning rates for the actor update, the uncertainty parameter update and the critic update are λ θ = λ ω
= λ ϕ = 3 × 10 − 4 .
The noise covariance matrices for the target policy smoothing are ˜Σ a = 2Σ a
and ˜Σ ω = 2Σ ω .
The noise for the target policy smoothing is clipped by Π a and Π ω into the ranges of ± 0 .
25 times the interval lengths of the corresponding dimensions of A and Ω , respectively.
The actor and target network update frequency is T freq = 2 .
The above parameter settings follow TD3 [7].
We used N = 5 uncertainty parameters.
For the refreshing strategy of the uncertainty parameters, we used ℓ 1 -distance as d ω . The distance threshold is d thre = 0 . The frequency threshold is 0 . 05 . Experiments were performed on a machine with two NVIDIA RTX A5000 GPUs, two Intel(R)
Xeon(R) Gold 6230 CPUs, and 192GB memory. D Related Work Table 5 summarizes the robust policy search method taxonomy. Table 6 compares the related methods considering their action value function definitions.
[Table]
E Performance of TD
Table 7 summarizes the worst-case performance and the average performance of the policies obtained by TD3 on the Ant, HalfCheetah, Hopper, HumanoidStandup, InvertedPendulum, and Walker environments with their reference parameters.
The results show that the original TD3 policies learned under the reference parameter cannot be generalized to uncertain parameters set in most scenarios.
These results provide the baseline performances on these environments and show that these environments are non-trivial for the worst-case and the average performance maximization.
F Original RARL
The performance of the original RARL, whose baseline RL approach is TRPO, is shown in Table 8.
RARL (TRPO) exhibited low worst-case performances, similarly to those of RARL (DDPG).
These low performances of RARL (TRPO) may be due to its defect of the optimization strategy and because
TRPO is an on-policy method, which often requires a greater number of interactions than off-policy methods [10].
G Learning Curve
The learning curves for each approach are shown in the Figure 2 and Figure 3.
Figure 2 shows that M2TD3 and SoftM2TD3 tended to perform better in the early stages of learning than DR, even in scenarios where the final worst-case performances of DR, M2TD3, and SoftM2TD3 were competitive. H Cumulative Rewards Under Different Uncertainty Parameters Figures 4 to 6 show the cumulative rewards of the policies trained by the seven approaches for each ω ∈ Ω . These results show that in many scenarios, DR achieved good performances in a wide range of uncertainty parameters.
However, the differences between the worst-case and best-case performances were relatively large for DR, and the worst-case performances were relatively low.
Unlike DR,
[Table]
M2TD3 exhibited smaller performance differences between the worst-case and the best-case, and its worst-case performances were relatively high.
I Multiple Uncertainty Parameter
Figure 7 shows the learning curves for each uncertainty parameter ω ∈ Ω and the uncertainty parameter ˆ ω used to update the actor ( ˆ ω k ′ in Algorithm 4) at each time step during the training.
For each environment, the cumulative rewards under 10 equally spaced uncertainty parameters were evaluated every 1 e 5 time steps.
Hence, the lowest cumulative reward at each time step is a rough estimate of the worst-case performance.
The uncertainty parameter with the lowest Q-value, argmin ˆ ω 1 ,..., ˆ ω k ,...
ˆ ω N J t
( θ, ˆ ω k ) , is chosen for the update of the actor among N worst uncertainty parameter candidates.
This means that the selected worst-case uncertainty parameter can be different from the ground truth worst-case uncertainty parameter because of incomplete optimization for the worst uncertainty parameter, incomplete training of the critic network, and discrepancy between the cumulative reward and the Q-values.
Focusing on the behavior of M2TD3 on the HalfCheetah 1 task (left-most side figures), the cumulative reward (top figure) shows that the uncertainty parameters 0 .
1 (green line), 0 .
5 (orange line), and 4 .(light blue line) alternately came to the bottom during learning, indicating that the worst uncertainty parameter continues to change between these values.
The uncertainty parameter selected during the actor training (bottom figure) were the values around 0 .
{+1 and 4 . 
+}
in the early stages of the training, and the values around 0 .
5 were used in the middle of the training.
This indicates that the algorithm was able to track the change of the worst-case uncertainty parameter during the training.
This behavior of
M2TD3 and SoftM2TD3 is shown on the other tasks.
J Evaluation Under Adversarial External Force
With a small modification of the proposed approaches, we can apply it to the situation conforming to [31], where the model misspecification is expressed by an external force given by an adversarial agent.
In this section, we describe the necessary modification of the proposed approaches and compare the worst
-case performance with baselines
Extension of M2TD
In this setting, we deal with an MDP M = ⟨ S, A p , A a , p, p 0 , r, γ ⟩ , where
A p and A a are the action spaces of the protagonist agent and adversarial agent, respectively, and they are assumed to be continuous. The transition probability density p : S × A p × A a × S → R and immediate reward r : S × A p ×
A a → R are not anymore parameterized by ω but take the action of the adversarial agent as input.
Unless otherwise specified, the other notations are the same as the ones
[Figure]
- c a [Figure] c u m u l a t i v e r e w a r d [Figure]
r
[Figure] t - c a s e c u m u l a t i v e [Figure]
w a r d
[Figure]
s t - c a
[Figure]
c
[Figure]
m u l a t i v e
[Figure]
w a r d
[Figure]
c a
[Figure]
c u m u l a t i v e r e w a r d
[Figure]
in Section 2.
The protagonist and adversarial agents interact with the environment M using stochastic behavior policies β and α , respectively.
Let ρ βα ( s ) =
lim
[Equation]
[Equation]
s
[Equation]
s
[Equation]
srepresent the stationary distribution of s under β and α , where the step- t transition probability density q tα,β is defined as q 1 α,β ( s ′ | s )
= (cid:82) a p ∈
A p (cid:82) a a ∈
A
a q ( s ′ | s, a p , a a ) β ( a p | s )
α ( a a | s )
d a a d a p and q tα,β ( s ′ | s )
= (cid:82) ¯ s ∈ S q t − 1 α,β (¯ s | s ) q 1 α,β ( s ′ | ¯ s )d¯ s .
The joint stationary distribution of ( s, a p , a a ) is defined as ρ βα ( s, a p , a a ) =
β ( a p | s ) α ( a a | s ) ρ βα ( s ) .
We extend the action value function as a function of state s , protagonist agent’s action a p , and adversarial agent’s action a a , namely,
[Equation]
(11) where µ and ξ represent the policies of the protagonist agent and the adversarial agent, respectively, to be trained. We suppose that they are parameterized by θ p and θ a , respectively. [Figure] v e r a g e c u m u l a t i v e r e w a r d
[Figure]
A v e r a g e c u m u l a t
i v e r
[Figure]
w a r d
[Figure]
e r a g e c u m u l a t
i v e r
[Figure]
w a r d
[Figure]
g e c u m u l a t
i v e r
[Figure]
w a r d
[Figure]
[Figure]
(DDPG)
[Figure]
(TRPO)
The objective of M2TD3 is extended as
[Equation]
where the critic loss function L is extended as [Equation] where T µ θp ξ θa is a function satisfying [Equation]
The max-min objective function J of the actor network is defined as
[Equation] [Figure] i v [Figure]
r
[Figure]
a r
[Figure]
l a t i v e r e w a r d
[Figure]
and ρ βα ( s, a p , a a ) and ρ βα ( s ) are approximated by the replay buffer B that stores the trajectories obtained by the interaction using β and α .
Let { ( s i , a pi , a ai , r i , s ′ i ) }
Mi =1 ⊂ B be mini-batch samples taken uniformly randomly from the replay buffer.
The approximated objective function used for the critic update is
[Equation]
(16) where y i = r i + γ · Q ϕ ( s ′ i , µ θ p ( s ′ i ) , ξ θ a ( s ′ i )) . The approximated objective function used for the actor update is
[Equation]
Replacing (6) and (7) with above defined functions, we obtain M2TD3 for this setting.
Experiment
We used the tasks provided in [31] 3 .
For each trained policy, the worst-case performance is estimated by fixing the trained policy of the protagonist agent and training the adversarial agent’s policy to minimize the protagonist agent’s performance.
To evaluate the worstcase performance of each approach, we performed 5 independent training for 2 × 10 6 time steps for each approach.
The trial is indexed as n ∈ { 1 , . .
Then, for each obtained protagonist policy, we trained the adversarial policy for 2 × 10 6 time steps. We performed the adversarial policy training three times, and they are indexed as m ∈ { 1 , 2 , 3 } . During the training of the adversarial policy, we recorded the performance of the protagonist agent under the adversarial policy at time step et ∈ { 10 5 , 2 × 10 5 , . . . , 2 × 10 6 } (every 10 5 time steps), denoted as R ( µ, ξ m,et ) .
Then, the worst-case performance of a protagonist policy µ was estimated by
R worst ( µ ) = min m =1 , 2 , 3 min et ∈{ 10 5 , 2 × 10 5 ,..., 2 × 10 6 } R ( µ, ξ m,et ) . The average and standard error of R worst ( µ ) over 5 trials are reported for each approach.
The parameters and network architecture of the protagonist agent used in all methods and the adversarial agents used in M2TD3, M2-DDPG,
M3DDPG, RARL (TD3), and RARL (TRPO) are the same as in the situation that the uncertainty parameter is directly encoded by ω .
In the situation that the uncertainty parameter was directly encoded by ω , multiple uncertainty parameters were trained, but in this setting, only one adversarial agent was trained.
The adversarial TD3 agents used to estimate worst-case performance were similar to the parameters and network architecture used in [7].
The result is shown in Table 9.
[Table]
M2TD3 showed better worst-case performance than DR in all but the HalfChetahAdv-v1 tasks.
The reference performances were comparable to those of DR.
Compared to those of RARL, M2TD showed competitive or superior performances, both in the worst-case and reference-case.
Comparing
M2-DDPG (the proposed approach based on DDPG instead of TD3) and M3DDPG, we observed similar performances in many scenarios in both reference-case and worst-case, and M2-DDPG significantly outperformed M3DDPG on HopperAdv-v1.
[Figure]
[Figure]
(a) HumanoidStandup
[Figure]
[Figure]