Max-Min Off-Policy Actor-Critic Method Focusing on Worst-Case Robustness to Model Misspecification
Abstract
In the field of reinforcement learning, owing to the high cost and risk of policytraining in the real world, policies trained in a simulation environment are oftentransferred to corresponding real-world environment.
However, the simulationenvironment does not perfectly mimic the real-world environment, leading tomodel misspecification.
Multiple studies report significant deterioration of policyperformance in a real-world environment.
In this study, we focus on scenariosinvolving a simulation environment with uncertainty parameters and the set oftheir possible values, called the uncertainty parameter set.
The aim is to optimizethe worst-case performance on the uncertainty parameter set to guarantee theperformance in the corresponding real-world environment, if it is included inthe uncertainty parameter set.
To obtain a policy that optimizes the worst-caseperformance, we propose an off-policy actor-critic approach called the Max-Min
Twin Delayed Deep Deterministic Policy Gradient Algorithm (M2TD3), whichsolves a max-min optimization problem using a simultaneous gradient ascentdescent approach.
Experiments in Multi-Joint Dynamics with Contact (MuJoCo)environments show that the proposed method exhibited a worst-case performancesuperior to several baseline approaches.
Our implementation is publicly available( https://github.com/anonymous-rev/review ).
Introduction
Applications of Deep Reinforcement Learning (DRL) to control tasks that involving interactionwith the real world are limited due to the need for interaction between the agent and the real-worldenvironment [36].
This difficulty arises because of high interaction time, high agent maintenancecosts and safety issues during the interaction.
Without enough interactions, DRL tends to overfit tospecific interaction histories and to yields policies with poor generalizability and safety issues [28].
To circumvent this difficulty in performing many real-world interactions, simulation environments thatestimate the characteristics of real-world environments are often employed.
In such cases, the policyis trained in the simulation environment and then transferred to the real-world environment [44].
Theuse of a simulation environment enables enough interactions for effective DRL application.
Moreover,because fast and general-purpose simulators such as ODE [33], Bullet [3] and MuJoCo [37] areavailable in the field of robotics, the cost of developing a simulation environment can sometimes besignificantly lower than that for real-world interaction.
However, because a simulation environment is an estimation of the real-world environment, discrepancies exist between the two [2].
Even if thesimulation environment perfectly replicates the real-world environment, discrepancies may still occur;for example, as a result of robot degradation over time or weight changes due to part replacements [22].
Studies report significant degradation of policy trained in a simulation environment when transferred to the corresponding real-world environment [8].
The above discrepancies limit the use of simulatorenvironments in practice, and thus, limit the use of DRL.
In this study, to guarantee performance in the corresponding real-world environment, we aim to obtaina policy that maximizes the expected reward under the worst-case scenario in the uncertainty parameterset Ω .
We focus on a scenario where (1) a simulator M ω is available, (2) the model uncertainty isparameterized by ω ∈ Ω and ω is configurable in the simulation, and (3) the real-world environmentis realized with some ω ∗ ∈ Ω , where ω ∗ is fixed during an episode; however, (4) ω ∗ is unknown orchanges from episode to episode 1 .
We develop an off-policy deep deterministic actor-critic approachto optimize the worst-case performance, called the Max-Min Twin Delayed Deep Deterministic
Policy Gradient Algorithm (M2TD3).
By extending the bi-level optimization formulation of thestandard actor-critic method [29], we formulate our problem as a tri-level optimization, where thecritic network models a value for each tuple of state, action, and uncertainty parameter, and the actornetwork models the policy of an agent and an estimate of the worst uncertainty parameter value.
Based on the existing deep deterministic actor-critic approach [19, 7], we design an algorithm to solvethe tri-level optimization problem by implementing several technical components to stabilize thetraining, including multiple uncertainty parameters, their refreshing strategies, uncertainty parametersampler for interaction, and soft-min policy update.
Numerical experiments on 19 MuJoCo tasksreveal the competitive and superior worst-case performance of our proposed approaches compared toseveral baseline approaches.
Preliminaries
We consider an episodic Markov decision process (MDP) [6] family M Ω
= {M ω } ω ∈ Ω , where
M ω = ⟨ S, A, p ω , p 0 ω , r ω , γ ⟩ is the MDP with uncertainty parameter ω ∈ Ω .
The state space Sand the action space
A are subsets of real-valued vector spaces.
The transition probability densityp ω : S ×
A × S → R , the initial state probability density p 0 ω : S → R and the immediate rewardr ω :
S ×
A → R depend on ω .
The discount factor is denoted by γ ∈ (0 , 1) .
We focus on an episodic task on M Ω .
Let µ θ : S → A be a deterministic policy parameterized byθ ∈ Θ .
Given an uncertainty parameter ω ∈ Ω , the initial state follows s 0 ∼ p 0 ω .
At each time stept ⩾ 0 , the agent observes state s t and select action a t = µ θ ( s t ) , interact with the environment; andobserves the next state s ′ t ∼ p ω ( · | s t , a t ) , the immediate reward r t = r ω ( s t , a t ) , and the terminationflag h t .
Here, h t = 1 if s ′ t is a terminal state or a predefined maximal time step for each past episode;otherwise, h t = 0 .
If h t = 0 , let s t +
= s ′ t ; otherwise, the next state is reset by s t +1 ∼ p 0 ω .
Forsimplicity, we let q ω
( s t +1 | s t , a t ) be the probability density of s t +
given s t and a t .
The discountreward of the trajectory starting from time step t is R t = (cid:80) k ⩾ 0 γ k r t + k .
The action value function Q µ θ ( s, a, ω ) under ω is the expectation of R t starting with s t =
s and a t =a under ω ; that is, Q µ θ ( s, a, ω )
=
E [ R t | s t = s, a t = a, s t + k +1 ∼ q ω
( · | s t + k , µ θ ( s t + k ))
∀ k ⩾ 0] ,where the expectation is taken over s t
+ k +1 for k ⩾ 0 .
Note that we introduce ω to the argument toexplain the Q -value dependence on ω ; however, this is essentially the same definition as that for thestandard action value function Q µ θ ( s, a ) .
The recursive formula is as follows:
[Equation]
We consider an off-policy reinforcement learning setting, where the agent interacts with M ω using astochastic behavior policy β : S ×
A → R .
However, the target policy to be optimized, denoted byµ θ is deterministic.
Under a fixed ω ∈ Ω , the standard objective of off-policy learning is to maximizethe expected action value function over the stationary distribution ρ β , such that
[Equation]
Here, ρ β ( s | ω ) =
lim T →∞ 1 T (cid:80) T − 1 t =0 (cid:82) s 0 q tω,β
( s | s 0 )
p 0 ω ( s 0 )d s 0 denotes the stationary distributionunder β and a fixed ω ∈ Ω , where the step- t transition probability density q tω,β is defined as
[Equation]
We assume that the agent can interact with M ω for any ω ∈ Ω during the training and can changeω after every episode; that is, when h t = 1 .
Our objective is to obtain the µ θ that maximizes J ω ( θ )under the worst environment ω ∈ Ω .
In other words, we tackle the max-min optimization problemrequiring max θ ∈ Θ min ω ∈ Ω J ω ( θ ) .
M2TD3
We propose an off-policy actor-critic approach to obtain a policy that maximizes the worst-caseperformance on M Ω , called M2TD3.
Our approach is based on TD3, an off-policy actor-criticalgorithm for a deterministic target policy.
In TD3, the critic network Q ϕ parameterized by ϕ ∈ Φ istrained to approximate the Q µ θ of µ θ , whereas the actor network models µ θ with parameter θ ∈ Θand is trained to maximize J ω ( θ ) .
Note that ω is not considered and is fixed during the training.
In this study to obtain the µ θ that maximizes the performance under the worst ω , we extend TD3by formulating the objective as a maximin problem and introducing a simultaneous gradient ascentdescent approach.
The main difficulty in the worst-case performance maximization is in the estimation of the worstcase performance.
Because the objective is expected to be non-concave with respect to (w.r.t.)the uncertainty parameter due to deep actor-critic networks, solving such a problem is consideredintractable in general [4].
To stabilize the worst-case performance estimation, we introduce varioustechniques: multiple uncertainty parameters, an uncertainty parameter refresh strategy, and anuncertainty parameter sampler.
Formulation
We can formulate our objective as the following tri-level optimization:
[Equation]
(3) where J and L are defined below.
Note that this tri-level optimization formulation is an extension ofthe bi-level optimization formulation of the actor-critic approach proposed in [29].
We introduce a probability density α : Ω → R , from which an ω to be used in the interaction duringthe training phase is drawn for each episode.
The relationship between ˆ ω and α is similar to thatbetween θ and β : that is, ˆ ω and θ are the parameters to be optimized while α and β are introducedfor exploration.
Let ρ βα ( s )
=
(cid:82) ω ∈ Ω ρ β ( s | ω ) α ( ω )d ω and ρ βα ( s, a, ω ) =
β ( a | s ) ρ β ( s | ω ) α ( ω ) bethe stationary distribution of s and the joint stationary distribution of ( s, a, ω ) , respectively, under αand β .
The critic loss function L ( ϕ ; θ ) is designed to simulate the Q-learning algorithm.
Let T µ θ be thefunctional satisfying T µ θ [ Q ]
[Equation]
s, a,
[Equation]
[Equation] [Equation]
Q
[Equation]
s
[Equation] [Equation]
s ′ ) , ω )
q ω
[Equation]
s ′ | s, a )d s ′ .
Then,(1) states that Q µ θ is the solution to Q = T µ θ [ Q ] .
Therefore, the critic is trained to minimize thedifference between Q ϕ and T µ θ [ Q ϕ ] .
This is achieved by minimizing
[Equation]
The max-min objective function of the actor network
[Equation]
(5) measures the performance of µ θ under the uncertainty parameter ˆ ω ∈ Ω approximated using the criticnetwork Q ϕ instead of the ground truth Q µ θ .
Similar to the standard actor-critic approach, we expect
Q ϕ to approach Q µ θ as the critic loss is minimized.
Therefore, we expect J ( θ, ˆ ω ; ϕ ∗ ) approximates 116 (cid:82) s ∈ S Q µ θ ( s, µ θ ( s ) , ˆ ω ) ρ βα ( s )
d s once we obtain ϕ ∗ ≈
argmin ϕ ∈
Φ L ( ϕ ; θ ) .
Note that, even if Q ϕ ∗ =
Q µ θ , our objective function J ( θ, ˆ ω ; ϕ ∗ ) differs from J ω ( θ ) with ω = ˆ ωin (2), in that the stationary distribution ρ β ( s | ω ) under fixed ω in J ω is replaced with ρ βα ( s ) underω
∼ α .
However, this change allows us to effectively utilize the replay buffer, which stores theinteraction history, to approximate the objective J ( θ, ˆ ω ; ϕ ) .
Moreover, if α is concentrated at ˆ ω ,
J ( θ, ˆ ω ; ϕ ∗ ) coincides with J ω
=ˆ ω ( θ ) .
Algorithmic Framework
The framework of M2TD3 is designed to solve the tri-level optimization(3).
The overall framework of M2TD3 follows that of TD3.
In each episode, an uncertainty parameter ω is sampled from α .
The training agent interacts with the environment M ω using behavior policy β .
At each time step t , the transition ( s t , a t , r t , s ′ t , h t , ω ) is stored in the replay buffer, denoted by B .
The critic network Q ϕ is trained at every time step to minimize (4), with a mini-batch being drawnuniformly randomly from B .
The actor network µ θ and the worst-case uncertainty parameter ˆ ω aretrained in every T freq step to optimize (5).
Note that a uniform random sample ( s, a, ω ) taken from Bcan be regarded as a sample from the stationary distribution ρ βα ( s, a, ω ) .
Similarly, s taken from Bis regarded as a sample from ρ βα ( s ) .
These facts allow approximation of the expectations in (4) and(5) using the Monte Carlo method with mini-batch samples uniformly randomly taken from B .
Thealgorithm of M2TD3 is summarized in Algorithm 1 in Appendix A.
Critic Update
The critic network update, that is, minimization of (4), is performed by implementing the TD error-based approach.
The concept is as follows.
Let { ( s i , a i , r i , s ′
i , ω i ) } Mi =
⊂ B bethe mini-batch, and let θ t and ϕ t be the actor and critic parameters at time step t , respectively.
Foreach tuple, T µ θ [ Q ϕ ]( s i , a i , ω i ) is approximated by y i =
r i + γ · Q ϕ t ( s
[Equation]
i , µ θ t ( s ′ i ) , ω i ) .
Then, (4) isapproximated by the mean square error, such that
[Equation]
Then, the update follows ϕ t +1 =
ϕ t − λ ϕ ∇ ϕ ˜ L ( ϕ t ) .
We incorporate the techniques introduced in TD3 to stabilize the training: clipped double Q-learning,target policy smoothing regularization, and target networks.
We additionally introduce the smoothingregularization for the uncertainty parameter so that the critic is smooth w.r.t.
ˆ ω .
Algorithm 2 in
Appendix A summarizes the critic update and the details of each technique available in [7].
Actor Update
Updating of the actor as well as the uncertainty parameter, that is, maximinoptimization of (5), is performed via simultaneous gradient ascent descent [27], summarized in
Algorithm 3 in Appendix A. Let ϕ t , θ t and ˆ ω t be the critic, policy, and the uncertainty parameters,respectively, at time step t .
Instead of the optimal ϕ ∗ in (3), we use ϕ t as its estimate.
With themini-batch { s i } Mi =1 ⊂ B , we approximate the max-min objective (5) as
[Equation]
We update θ and ˆ ω as θ t +
=
θ t
+ λ θ ∇ θ ˜ J t ( θ t , ˆ ω t ) and ˆ ω t
+
=
ˆ ω t − λ ω ∇ ˆ ω ˜
J t ( θ t , ˆ ω t ) , respectively.
For higher-stability performance, we introduce multiple uncertainty parameters.
The motivation istwofold.
One is to deal with multiple local minima of J t ( θ, ˆ ω ; ϕ ∗ ) w.r.t.
ˆ ω .
As the critic networkbecomes generally non-convex, there may exist multiple local minima of J t ( θ, ˆ ω ; ϕ ∗ ) w.r.t.
ˆ ω .
Onceˆ ω is stacked at a local minimum point, e.g., ˆ ω ∗ , θ may be trained to be robust around a neighborhoodof ˆ ω ∗ and to perform poorly outside that neighborhood.
The other is to deal with the situation that themaximin solution ( θ ∗ , ϕ ∗ ) of J t ( θ, ˆ ω ; ϕ ∗ ) is not a saddle point, which happens when the objectiveis non-concave in ˆ ω [14].
In such a situation, more than one ˆ ω is necessary to approximate theworst-case performance min ˆ ω J t ( θ, ˆ ω ; ϕ ∗ ) around ( θ ∗ , ϕ ∗ ) and a standard simultaneous gradientascent descent method fails to converge.
To relax this defect of simultaneous gradient ascent descentmethods, we maintain multiple candidates for the worst ˆ ω , denoted by ˆ ω 1 , . . .
, ˆ ω N .
Therefore,replacing min ˆ ω J ( θ, ˆ ω ; ϕ ∗ ) with min k =1 ,...,N J ( θ, ˆ ω k ; ϕ ∗ ) in (5) has no effect on the optimal θ ;however, this change does affect the training behavior.
Our update follows the simultaneous gradientascent descent on min k =1 ,...,N J t ( θ, ˆ ω k ) : θ ← θ + λ θ ∇ θ (min k =1 ,...,N J t ( θ, ˆ ω k )) and ˆ ω k ← ˆ ω k − 162 λ ω ∇ ˆ ω k (min ℓ =1 ,...,N J t ( θ, ˆ ω ℓ )) for k = 1 , . . .
, N .
Hence, θ is updated against the worst ˆ ω k and onlythe worst ˆ ω k is updated because the gradient w.r.t.
the other ˆ ω k are all zero.
For multiple uncertainty parameters to be effective, they must be distant from each other.
Moreover,all are expected to be selected as the worst parameters with non-negligible frequencies.
Otherwise,the advantage of having multiple uncertainty parameters is lessened.
From this perspective, weintroduce a refreshing strategy for uncertainty parameters.
That is, we resample ˆ ω k ∼ U (Ω) if one ofthe following scenarios is observed: there exists ˆ ω ℓ such that the distance d ˆ ω (ˆ ω k , ˆ ω ℓ ) ⩽ d thre ; thefrequency p k of ˆ ω k being selected as the worst-case during the actor update is no greater than p thre .
Uncertainty Parameter Sampler
The uncertainty parameter sampler α controls the explorationexploitation trade-off in the uncertainty parameter.
On one hand, exploration in Ω is necessary to train the critic network.
If the critic network is not well trained over Ω , it is difficult to locate theworst ˆ ω correctly.
On the other hand, for J ( θ, ˆ ω ; ϕ ∗ ) in (5) to coincide with J ω ( θ ) for ω = ˆ ω in (2),we require α to be concentrated at ˆ ω .
Otherwise, the optimal θ for J ( θ, ˆ ω ; ϕ ∗ ) may deviate from thatof J ω
( θ ) .
We design α as follows.
For the first T rand steps, we sample the uncertainty parameter ωuniform-randomly on Ω , i.e., α = U (Ω) .
Then, we set α = (cid:80) Nk
=1 p k · N (ˆ ω k , Σ ω ) , where Σ ω is thepredefined covariance matrix.
We decrease Σ ω as the time step increases.
The rational behind thegradual decrease of Σ ω is that the training of the critic network is still important after T rand to makethe estimation of the worst-case uncertainty parameter accurate.
Its detail is provided in Appendix C.
Related Work
Methods to handle model misspecification include (1) robust policy searching [26, 30, 23, 17, 12, 22],(2) transfer learning [35, 42, 31], (3) domain randomization (DR) [36], and (4) approaches minimizingworst-case sub-optimality gap [5, 21].
Robust policy searching includes methods that try to obtainthe optimal policy under the worst possible disturbance or model misspecification, i.e., maximizationof min ω ∈ Ω J ω ( θ ) for an explicitly or implicitly defined Ω .
Transfer learning is a method in which apolicy is trained on source tasks and then fine-tuned through interactions while performing the targettask.
DR is a method in which a policy is trained on source tasks that are sampled randomly from
M Ω .
There are two types of DR approach: vanilla and guided.
In vanilla DR [36], the policy istrained on source tasks that are randomly sampled from a pre-defined distribution on Ω .
In guided
DR [43], the policy is trained on source tasks; however, the ω distribution is guided towards thetarget task.
Because we do not assume access to the target task, transfer learning and many guided
DR approaches are outside the scope of this work.
Some guided DR approaches, such as Active
Domain Randomization [24], do not access the target task; however, they do not consider worst-caseoptimization either.
Vanilla DR can be applied to the setting considered here.
However, the objectiveof vanilla DR differs from the present aim, i.e., to maximize the performance averaged over Ω , that is, 196 E ω
[ J ω ( θ ) | ω ∼ U
(Ω)] , if the sampling distribution is U (Ω) .
Approaches minimizing the worst-casesub-optimality gap [5, 21] do not optimize the worst-case performance, but attempt to obtain a policythat generalizes well on the uncertainty parameter set while avoiding too conservative performancewhich often attributes to the worst-case performance optimization.
Among the existing robust policy search methods, some adopt an adversarial approach to policyoptimization.
For example, Robust Adversarial Reinforcement Learning (RARL) [30] models thedisturbance caused by an external force produced by an adversarial agent, and alternatively trainsthe protagonist and adversarial agents.
Robust Reinforcement Learning (RRL) [26] models thedisturbance in the same manner but has not been applied to the DRL framework.
Minimax MultiAgent
Deep Deterministic Policy Gradient (M3DDPG) [17] has been designed to obtain a robustpolicy for multi-agent settings.
This method is applicable to the setting targeted in this work if onlytwo agents are considered.
The above approaches frame the problem as a zero-sum game betweena protagonist agent attempting to optimize µ θ and an adversarial agent attempting to minimize theprotagonist’s performance, hindering the protagonist by generating the worst possible disturbance.
Adv-Soft Actor Critic(Adv-SAC) [13] learns policies that are robust to both internal disturbances inthe robot’s joint space and disturbances from other robots.
Recurrent SAC (RSAC) [41] introduces
POMDPs to treat the uncertainty parameter as an unobservable state.
A DDPG based approachrobustified by applying stochastic gradient Langevin dynamics is proposed under the noisy robust
MDP setting [15].
However, it has been reported in [41, 15] that their worst-case performancesare sometimes even worse than their baseline non-robust approaches.
State-Adversarial-DRL (SADRL) [47] and alternating training with learned adversaries (ATLA) [46] improve the robustnessof DRL agents by using an adversary that perturbs the observations in a SAMDP framework thatcharacterizes the decision-making problem in an adversarial attack on state observations.
They donot address the model misspecification in the reward and transition functions.
Other approaches attempt to estimate the robust value function, i.e., a value function under the worstuncertainty parameter.
Among them, Robust Dynamic Programming (DP) [12] is a DP approach,while the Robust Options Policy Iteration (ROPI) [23] incorporates robustness into option learning[34], which allows agents to learn both hierarchical policies and their corresponding option sets.
ROPI is a type of Robust MDP approach [38, 20].
Robust Maximum A-posteriori Policy Optimization(R-MPO) [22] incorporates robustness in MPO [1].
Neither Robust MDP nor R-MPO require interaction with M ω for an arbitrary ω ∈ Ω .
This can beadvantageous in scenarios where design of a simulator valid over Ω is tedious.
However, methods ofthis type typically require additional assumptions for computation of the worst-case for each valuefunction update, such as the finiteness of the state-action space and/or the finiteness of Ω .
One canapply R-MPO to our setting by sampling finite scenarios from Ω for training.
However, the choice ofthe training scenarios affect the worst-case performance.
There is also the idea of optimizing worst-case performance in the field of offline RL [39, 45, 40].
Offline RL tries to obtain robust measures by introducing the principle of pessimism.
There are studiesthat introduce the principle of pessimism in the model-free context [39, 40] and in the actor-criticcontext [45].
However, it does not compete with this study due to differences in motivation and thefact that offline RL requires some additional assumptions in the datasets [45, 40] and realizability [39].
Table 5 in Appendix D summarizes the robust policy search method taxonomy.
Our proposedapproach, M2TD3, considers the uncertainties in both the reward function and transition probability.
The state and action spaces, as well as the uncertainty set, are assumed to be continuous.
Table 6 in
Appendix D compares the related methods considering their action value function definitions.
Forthe methods that do not explicitly utilize action value function, those functions are interpreted fromthe objective functions.
Both M3DDPG and M2TD3 maintain the value function for the tuple of( s, a, ω ) .
However, a difference exists in that M3DDPG takes the worst ω ′ on the right-hand side,which yields a more conservative policy than M2TD3 for our setting.
The value functions in RARLand M2TD3 are identical if ω is fixed.
Note that ω is not introduced to the RARL value functionbecause it does not simultaneously optimize µ θ and the estimate of the worst ω .
RARL repeats the θand ω optimizations alternatively.
In contrast, M2TD3 optimizes θ and ω simultaneously in a mannersimilar to the training processes of generative adversarial networks (GANs) [9].
For this purpose, werequire an action value for each ω ∈ Ω .
A difference exists between the optimization strategies of RARL and M2TD3.
As mentioned above,both methods try to maximize the worst-case performance
min ω ∈ Ω J ω ( θ ) .
Conceptually, RARLrepeats θ ← argmax θ J ω ( θ ) and ω ← argmin ω J ω
( θ ) .
However, this optimization strategy fails toconverge even if the objective function ( θ, ω ) (cid:55)→ J ω ( θ ) is concave-convex.
As an example, considerthe function ( x, y ) (cid:55)→ y 2 − x 2 + αxy .
The RARL optimization strategy reads x ← ( α/ 2)
y andy ← − ( α/ 2) x , which causes divergence if α > 2 .
2 Alternating updates are also employed in otherapproaches such as Adv-SAC, SA-DRL, and ATLA.
They all share the same potential issue as RARL.
M2TD3 attempts to alleviate this divergence problem by applying the gradient-based max-minoptimization method, which has been employed in GANs and other applications and analyzed for itsconvergence [25, 18].
Experiments
In this study, we conducted experiments on the optimal control problem using MuJoCo environments.
Hence, we demonstrated the problems of existing methods and assessed the worst-case performanceand average performance of the policy trained by M2TD3 for different continuous control tasks.
Baseline Methods
We summarizes the baseline methods adapted to our experiment setting: namely,
DR, RARL, and M3DDPG.
DR: The objective of DR is to maximize the expected cumulative reward for the distributionα =
U (Ω) of ω .
In each training episode, ω is drawn randomly from U (Ω) and the agent neglects ωwhen training and performing the standard DRL.
For fair comparison, we implemented DR with TD3as the baseline DRL method.
RARL: We adapted RARL to our scenario by setting µ ˆ ω : s (cid:55)→ ˆ ω to the antagonist policy.
In thisscenario, RARL was regarded as optimizing (2) for the worst ω ∈ Ω ; hence, the objective was thesame as that of M2TD3.
Note that the main technical difference between M2TD3 and RARL is inthe optimization strategy as described in Section 4.
The original RARL is implemented with Trust
[Table]
Region Policy Optimization (TRPO)
[32], but for fair comparison, we implemented it with DDPG,denoted as RARL (DDPG).
The experimental results of RARL (TRPO), the original RARL, areprovided in Appendix F.
M3DDPG:
By considering a two-agent scenario and µ ˆ ω : s (cid:55)→ ˆ ω as the opponent-agent’s policy,we adapted M3DDPG to our setting.
Because of the difficulty in replacing DDPG with TD3 in the
M3DDPG framework, we used DDPG as the baseline DRL.
Additionally, we implemented our approach with DDPG instead of TD3, denoted by M2-DDPG, forcomparison purpose.
We also implemented a variant of M2TD3, denoted as SoftM2TD3, performinga “soft” worst-case optimization to achieve a better average performance while considering theworst-case performance.
The detail of SoftM2TD3 is described in Appendix B.
Experiment Setting
We constructed 19 tasks based on 6 MuJoCo environments and with 1-3uncertainty parameters, as summarized in Table 4 in Appendix C.
Here, Ω was defined as an interval,which mostly included the default ω values.
To assess the worst-case performance of the given policy µ under ω ∈ Ω , we evaluated the cumulativereward 30 times for each of uncertainty parameter value ω 1 , . .
. , ω K ∈ Ω .
Here, R k ( µ ) was definedas cumulative reward on ω k averaged over 30 trials.
Then, R worst ( µ ) = min 1 ⩽
k ⩽ K R k ( µ ) wasmeasured as an estimate of the worst-case performance of µ on Ω .
We also report the averageperformance R average ( µ ) = 1 K (cid:80) K k
=1 R k ( µ ) .
A total of K uncertainty parameters ω 1 , . . .
, ω K forevaluation were drawn as follows: for 1D ω , we chose K = 10 equally spaced points on the 1Dinterval Ω ; for 2D ω , we chose 10 equally spaced points in each dimension of Ω , thereby obtaining
K = 100 points; and for 3D ω , we chose 10 equally spaced points in each dimension of Ω therebyobtaining K = 1000 points.
For each approach, we trained the policy ten times in each environment.
The training time steps
T max were set to 2M, 4M, and 5M for the environments with 1D, 2D, and 3D uncertainty parameters,respectively.
The finally obtained policies were evaluated for their worst-case performances.
Forfurther details of the experiment settings, please refer to Appendix C
Comparison to Baseline Methods
Table 1 summarizes the worst-case performances of thepolicies trained by M2TD3, SoftM2TD3, M2-DDPG, M3DDPG, RARL, and DR.. See also Table 7in Appendix E for the results of TD3 trained on the reference uncertainty parameters as baselines, and
Figure 2 in Appendix G for the learning curves.
In most cases, M2TD3 and SoftM2TD3 outperformed
DR and RARL and M3DDPG.
Figure 1 shows the cumulative rewards of the policies trained by thesix approaches for each ω values on the Ant 1, InvertedPendulum 1, and Walker 1 environments.
Seealso Figure 4 in Appendix H for other environments.
[Figure]
Comparison with DR: DR does not attempt to optimize worst-case performance.
In fact, it showedlower worst-case performance than M2TD3 and SoftM2TD3 in many scenarios because the obtainedpolicy performs poorly on some uncertainty parameters while it performs well on average.
One suchexample situation is observed in the results of InvertedPendulum 1 shown in Figure 1, where the policyobtained by DR exhibited a high performance for a wide range of ω ∈ Ω but performed poorly forsmall ω .
M2TD3 outperformed DR as well as the other baselines on such environments.
However, insome scenarios such as HumanoidStandup 1, Small HalfCheetah 1, and Small Hopper 1, DR achieveda better worst-case performance than M2TD3.
This outcome may be because the optimization ofworst-case function values is generally more unstable than optimization of the expected functionvalues.
Comparison with RARL: M2-DDPG outperformed RARL in most environments, with this performance difference originating from the different optimization strategies.
As noted above, theoptimization strategy employed in RARL often fails to converge.
As we see in Figure 1, RARLfailed to optimize the policy not only in the worst-case also on average in Ant 1 environment and
Walker 1 environment.
In the InvertedPendulum 1 environment, RARL could train the policy forsome uncertainty parameter, but not for the worst-case uncertainty parameter.
Comparison with M3DDPG: In many cases, M2-DDPG outperformed M3DDPG.
As discussedabove, M3DDPG considers a stronger adversary than necessary (the worst ω for each time step).
Therefore, the resulting policy was too conservative for this experiment and exhibited lower performance.
Average Performance
Although our objective is maximizing the worst-case performance, theaverage performance is also important in practice.
Table 2 compares the average performance ofthe six approaches.
Generally, DR achieved the highest average performance as expected.
However,interestingly, M2TD3 and SoftM2TD3 achieved competitive average performances to DR on severalenvironments such as Hopper 1, 2, and Walker 1, 2.
Moreover, comparison between Table 1 and
Table 2 show that a few times higher average performance than worst-case performance on severalenvironments such as Ant 3 and Hopper 1–3.
Ablation Study
Ablation studies were conducted on InvertedPendulum 1 with the pole mass as theuncertainty parameters.
M2TD3 with N = 5 was taken as the baseline.
We tested several variants:with different multiple uncertainty parameters ( N = 1 and 10 ), without distance-based refreshingstrategy (w/o DRS) and without probability-based refreshing strategy (w/o PRS).
Table 3 shows theresults.
Clearly, the inclusion of multiple uncertainty parameters as well as the probability-basedrefresh strategy contributed significantly to the worst-case performance and average performance,implying that both techniques contribute to better estimate the worst uncertainty parameter.
Though
DRS had little impact on the performance, one can implement prior knowledge in DRS by definingthe distance and the threshold in the uncertainty parameter task-dependently, which we did not here.
Small Uncertainty Set (Limitation of M2TD3)
Small HalfCheetah 1 and Small Hopper 1 weredesigned to have smaller uncertainty parameter set than HalfCheetah 1 and Hopper 1, and to reveal theeffect of the size of the uncertainty parameter interval.
As expected, a smaller uncertainty parameterset resulted in higher worst-case (Table 1) and average (Table 2) performances, for all approaches.
[Table] [Table]
Because M2TD3 performs the worst-case optimization, it is expected to show better worst-caseperformance than DR independently of the size of the uncertainty parameter set.
However, on theseenvironments, DR showed better worst-case performance than M2TD3, while M2TD3 achievedcompetitive or superior worst-case performance to DR on HalfCheetah 1 and Hopper 1.
This may bebecause the max-min optimization performed by M2TD3 resulted in sub-optimal policies.
Whenthe uncertainty parameter set is small and the performance do not change significantly over theuncertainty parameter set, maximizing the average performance is likely to lead to a high worst-caseperformance.
The sub-optimal results obtained by M2TD3 is then dominated by the results obtainedby DR.
Therefore, there is room for improvement in the maximin optimization in M2TD3.
Evaluation Under Adversarial External Force
Although we developed the proposed approachfor the situation that the uncertainty parameter is directly encoded by ω , we can extend it to thesituation that the model misspecification is expressed by an external force produced by an adversarialagent as in [30].
The extended approach and its experimental result are given in Appendix J. Superiorworst-case performances of M2TD3 over DR, RARL, and M3DDPG were observed for this settingas well.
Conclusion
In this study, we targeted the policy optimization aimed at maximizing the worst-case performancein a predefined uncertainty set.
The list of the contributions are as follows.
(i)
We formulatedthe off-policy deterministic actor-critic approach to the worst-case performance maximization as atri-level optimization problem (3).
(ii) We developed the worst-case performance of M2TD3.
Thekey concepts were incorporation of the uncertainty parameter into the critic network and use of thesimultaneous gradient ascent descent method.
Different technical components were introduced tostabilize the training.
(iii)
We evaluated the worst-case performance of M2TD3 on 19 MuJoCo tasksthrough comparison with three baseline methods.
Ablation studies revealed the usefulness of eachcomponent of M2TD3.
Checklist
claims made abstract and introduction accurately reflect the paper’scontributions and scope
Section 3 and Section 5.
(b) Did you describe the limitations of your work?
[Yes] Please see paragraph titled "SmallUncertainty Set" in Section 5 and Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] Please see
Appendix A.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see Section 5 and Appendix C.
(c) Did you report error bars (e.g., w.r
t. the random seed after running experiments multiple
times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix C.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Code licence is part of the repository.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include our code in the Appendix A.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[-Perform Algorithm 2 for critic network update 24: if mod( t, T freq ) = 
-]
[-a ( N (0 , ˜Σ a )) 
-]
[-+ min { Q ϕ ′ 
-]
[Table]
[-1 and 4 . 
-]