Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We study policy optimization for Markov decision processes (MDPs) with multiplereward value functions, which are to be jointly optimized according to given criteria such as proportional fairness (smooth concave scalarization), hard constraints(constrained MDP) and max-min trade-off.
We propose an Anchor-changing
Regularized Natural Policy Gradient (ARNPG) framework, which can systematically incorporate ideas from well-performing ﬁrst-order methods into the design ofpolicy optimization algorithms for multi-objective MDP problems.
Theoretically,the designed algorithms based on the ARNPG framework achieve ˜ O (1 /T ) globalconvergence with exact gradients.
Empirically, the ARNPG-guided algorithms alsodemonstrate superior performance compared to some existing policy gradient-basedapproaches in both exact gradients and sample-based scenarios
Introduction
In many sequential decision-making scenarios, agents usually face multiple objectives simultaneously.
This motivates the study of reinforcement learning (RL) with multiple reward values V ⇡ 1: m ( ⇢ ) .
Given the achievable region V = { V ⇡ 1: m ( ⇢ ) } ⇡ 2 ⇧ consisting of value vectors achieved by policies in 15 policy class ⇧ , the agent employs certain criteria to reﬂect the system requirement.
For example,1.
Proportional fairness [15]:
Given a 1: m > 0 , ﬁnd v 2 V that
P mi =1 a
i v
i  v i v i  0 , 8 v 0 2 V .2.
Hard constraints [4]: Given b 2: m , maximize v 2 V v 1 , subject to v
i
b
i , 8 i = 2 , . . .
, m .3.
Max-min trade-off [8]: Given c 1: m > 0 , maximize v 2 V min i
[ m ] ( v i /c i ) .
We study policy gradient-based approaches that optimize over parameterized policies ⇧ = { ⇡ ✓ :✓ 2 ⇥ } through policy gradient.
In general, the optimization problems above may not be convex interms of ✓ , not even for single-objective MDPs with direct parameterization by ✓ s,a = ⇡ ✓ ( a | s ) [2].
Due to the non-convexity, O (1 /T ) global convergence of policy gradient-based methods was onlyestablished very recently for single-objective MDPs with exact gradients [2, 22].
These breakthroughresults have motivated the study of policy optimization for multi-objective MDPs, e.g., smoothconcave scalarization [5], constrained MDPs (CMDPs) [12, 33].
However, under the exact gradients scenario, the previous approaches for multi-objective MDPs,either suffer from slow provable O (1 / p T ) global convergence [12], or require extra assumptions[41, 36, 20].
The compactness of ⇥ is assumed in [41], but this assumption forbids a very commonsoftmax parameterization, where ⇥ =
R |S||A| .
The NPG-based methods have been analyzed in[36, 20] under an ergodicity assumption, but such an assumption is not required for NPG in singleobjective MDPs [2], and therefore appears artiﬁcial.
The above criteria for multi-objective MDPs could be viewed as convex optimization problems w.r.t.a value vector v 2 V , for which there are a wide array of well-performing ﬁrst-order methods forconvex optimization problems in general.
It is desirable to take full advantage of such efﬁcientﬁrst-order methods in a uniﬁed and ﬂexible manner when designing policy gradient-based algorithmsfor multi-objective MDPs.
Main contributions
We propose an anchor-changing regularized natural policy gradient (ARNPG) framework inSection 3 that can exploit and integrate ﬁrst-order methods for the design of policy gradient-basedalgorithms for multi-objective MDPs.2.
We demonstrate the strength of the ARNPG framework by designing algorithms for threegeneral criteria: smooth concave scalarization (Section 4.1), constrained MDPs (Section 4.2),and max-min trade-off (Section 4.3).
Under softmax parameterization with exact gradients, the proposed algorithms inherit theadvantages of the integrated ﬁrst-order methods, and are guaranteed to have ˜ O (1 /T ) globalconvergence without further assumptions on the underlying MDP.4.
In addition to the theoretical advantages, we provide the results of extensive experimentationin Section 5 and Appendices A and B which demonstrate that the ARNPG-guided algorithms
provide superior performance in exact gradient and sample-based tabular scenarios, as well as
actor-critic deep RL scenarios, compared to several existing policy gradient-based approaches.
1.1 Related works
Policy gradient (PG)-based methods have drawn much attention recently [1, 22, 11, 16] due to theirsimplicity as well as the potential to generalize to large scale problems.
Despite their non-convexnature, PG-based methods have been shown to converge globally for single-objective MDPs [1, 22].
Their convergence may be further accelerated with appropriate regularization [11, 19], e.g., entropyregularization, but the algorithms only converge to the optimum of the regularized problem instead ofthe desired (unregularized) problem.
This paper considers single-policy multi-objective MDPs, including CMDPs where constraints arespeciﬁed on some objectives.
Global convergence of PG-based approaches in the multi-objective
MDPs has been previously studied.
For smooth concave scalarization, Bai et al.
[5] showed an
O (1 / ✏ 4 ) sample complexity (to achieve ✏ -optimal in expectation) of the policy-gradient methodunder sample-based scenarios.
However, with exact gradients, we are unaware of works with fast˜ O (1 /T ) convergence.
For CMDPs, Ding et al.
[12] have studied a primal-dual NPG algorithmachieving O (1 / p T ) global convergence for both the optimality gap and the constraint violation.
Xuet al.
[33] have proposed a primal approach that reduces constraint violations with a higher prioritythan optimizing objective, and enjoys the same O (1 / p T ) global convergence.
In work conductedconcurrently with ours, [36] and [20] have proposed algorithms that achieve ˜ O (1 /T )
convergencebut with extra ergodicity assumptions.
A general setting of optimizing a concave function of the state-action visitation distribution has beenconsidered in [41].
Though the problem is more general, its gradient estimation is more complicatedthan the canonical policy gradient estimate.
Zhang et al.
[41] showed that the gradient ascentachieves O (1 /T ) global convergence for smooth scalarization with exact gradients, under severalassumptions such as convexity and compactness of the parameter set ⇥ .
Directly viewing the stateaction visitation as the decision variables and imposing equality constraints for their feasibility, asmooth concave scalarization has been studied in [40] and later generalized to the constrained settingin [6].
These two works focus on sample-based scenarios, but due to their primal-dual approach withequality constraints, the convergence rate is only O (1 / p T ) even with exact gradients.
Moreover, thestate-action visitation parameterization is difﬁcult to generalize to larger scale deep RL scenarios.
A more thorough discussion on related works is given in Appendix F.
PreliminariesSystem model
A Markov decision process (MDP) is represented by a tuple ( S , A , P, ⇢ ,  , r ) , where
S is the state space, A the action space, P : S ⇥ A !
( S ) the transition kernel, ⇢ 2  ( S ) the initial state distribution,  2 (0 , 1) the discount factor, and r : S ⇥ A !
[0 , 1] the reward function.
Given any policy ⇡ : S !
( A ) and any reward function r : S ⇥ A !
[0 , 1] , we deﬁne the statevalue function V ⇡ r : S !
[0 , 11   ] , and the state-action value function Q ⇡ r : S ⇥ A !
[0 , 11   ] , as
V ⇡ r ( s ) :
= E [
[Equation]
where expectation E is taken over the random trajectory of the Markov chain induced by the policy⇡ and the transition kernel P .
With a slight abuse of notation, we denote V ⇡ r ( ⇢ ) :
=
E s ⇠ ⇢ [ V ⇡ r ( s )] .
Deﬁne the discounted state-action visitation distribution (state-action visitation for short) of policy⇡ with initial state distribution ⇢ by d ⇡⇢ ( s, a ) :=
(1   ) E s 0 ⇠ ⇢ [
P 1 t =
t P ( s t = s, a t = a | s 0 , ⇡ )] .
Itthen follows that V ⇡ r ( ⇢ ) =
11   h d ⇡⇢ , r i by viewing d ⇡⇢ and r as |S||A| -dimensional vectors indexedby ( s, a ) 2 S ⇥ A .
When it is clear from the context, we denote the state visitation distribution by 92 d ⇡⇢ ( s ) :
=
E s 0 ⇠ ⇢ [(1   )
P 1 t =
t P ( s t = s | s 0 )] , which is the marginal distribution of the state-actionvisitation d ⇡⇢ ( s, a ) , i.e., d ⇡⇢ ( s ) =
P a 2 A d ⇡⇢ ( s, a ) .
We study an MDP with m objectives represented by ( S , A , P, ⇢ ,  , r 1: m ) , where r i : S ⇥ A ! 95 [0 , 1] is the i -th reward function for each
i 2 [ m ] .
For simplicity, denote V ⇡ i ( · ) :
= V ⇡ r i ( · ) and
V ⇡ 1: m ( · ) :
= ( V ⇡ 1 ( · ) , . .
, V ⇡ m ( · )) .
We consider parameterized policies in ⇧ = { ⇡ ✓ : ✓ 2 ⇥ } , where⇥ ⇢ R n is the parameter space.
For example, the softmax policy is ⇡ ✓ ( a | s ) =
exp( ✓ s,a )
P a 0 exp( ✓ s,a 0 ) with⇥ =
R |S||A| ; and neural softmax policy is ⇡ ✓ ( a | s ) =
exp( NN ✓ ( s,a ))
P a 0 exp( NN ✓ ( s,a 0 )) , where NN ✓ is some neuralnetwork parameterized ✓ .
Deﬁne V :=
{ V ⇡ ✓ 1: m ( ⇢ ) : ✓ 2 ⇥ } as the achievable region of value vectors.
The agent wishes to optimize the policy in ⇧ for a given speciﬁc multi-objective criterion on valuevectors in V .
Mirror ascent As one of the most well-known iterative optimization methods, mirror descent(actually ascent in the context of our formulation as a maximization problem) [23, 7] is a generalclass that encompasses many ﬁrst-order methods in convex optimization.
Given a variable x in acompact convex set X ⇢ R n and an ascent direction g 2 R n , the variational representation of themirror ascent update is
[Equation]
(1) where B h ( x || y ) :
= h ( x )  h ( y )  hr h ( y ) , x  y
i is some Bregman divergence generated by a 108 differentiable convex function h :
X !
R .
When analyzing the convergence of ﬁrst-order methods, 109 certain fundamental inequalities are usually established to facilitate the proof.
One such inequality is
[Equation]
which is a critical step in many previous works, e.g., [25, 30, 18].
It is desirable to construct a similar fundamental inequality for multi-objective MDPs that canfacilitate the analysis of convergence.
As we will show in the next section, such an inequality canindeed be established in a new framework, which we refer to as the Anchor-Changing Regularized
Natural Policy Gradient (ARNPG).
Notations Denote KL-divergence between two n -dimensional probability vectors x, y by D ( x || y ) :
=
P ni
=1 x i log( x i /y i ) , which is a widely-used Bregman divergence.
For any policies ⇡ , ⇡ 0 and statevisitation distribution d , deﬁne D d ( ⇡ || ⇡ 0 )
:=
P s 2 S d ( s ) D ( ⇡ ( ·| s ) || ⇡ 0 ( ·| s )) .
A uniform policy is one 118 which chooses actions uniformly at random.
Anchor-changing regularized natural policy gradient
Let us consider a hypothetical mirror ascent update on decision value vector v k 2 V according to (1).
Given an ascent direction ˜ G k along which to improve v k , the updated value vector is
[Equation]
Suppose the value vector v k is achieved by a policy ⇡ ✓ k , i.e., v k = V ⇡ ✓ k 1: m ( ⇢ ) .
Denote the rewardfunction in the ascent direction as ˜ r k
( s, a ) =
h ˜ G k , r 1: m ( s, a ) i .
It follows that h ˜ G k , v k i
= V ⇡ ✓ k ˜ r k ( ⇢ ) .
Note that B h ( v || v k ) in (3) serves the role of a soft constraint on v by keeping v within a vicinity ofv k .
Replacing B ( v || v k ) by D d ⇡✓⇢ ( ⇡ ✓ || ⇡ ✓ k )  will induce a similar soft constraint that prefers the vicinityof the “anchor" policy ⇡ ✓ k .
Therefore we consider replacing the variational update in (3) by
[Equation]
where
[Equation]
ARNPG Motivated by the intuition above, we propose the Anchor-Changing Regularized Natural
Policy Gradient (ARNPG) framework.
At (macro) step k , the ARNPG framework determinesthe reward function in the ascent direction ˜ r k and the anchor policy ⇡ ✓ k , which can exploit wellperformed ﬁrst-order methods in convex optimization literature utilizing the features of the speciﬁccriteria in use.
With ˜ r k and ⇡ ✓ k , we wish to solve for (4) to improve the value vector.
However theoptimal solution ✓ 0 of (4) is generally not determinable explicitly.
ARNPG therefore approachesthe optimal solution via a subroutine that executes a natural policy gradient (NPG) algorithm w.r.t.the KL-regularized value function ˜ V ⇡ ✓ k, ↵ ( ⇢ ) .
We refer to this subroutine, given in Algorithm 1, as
InnerLoop( ˜ r k , ⇡ ✓ k , ↵ , ⌘ , t k ).
It iteratively updates the parameter ✓ ( t ) k for t k (micro) steps according tothe NPG update rule as in (5), where F ⇢ ( ✓ ) † is the Moore-Penrose inverse of the Fisher informationmatrix F ⇢
[Equation] [Equation]
i .
Algorithm
[Equation]
Initialize ✓ (0) k
= ✓ k for t = 0 , 1 , . . .
t k  1 do
[Equation]
Return ✓ ( t k ) k
The choice of the number of iterations in InnerLoop (i.e., t k ) involves a trade-off between thevariational update precision and the overall efﬁciency.
On the one hand, a larger t k leads to a moreaccurate approximation of the optimal solution ✓ 0 to (4), but it may cause the algorithm to spendunnecessary computational resources on the regularized objective ˜ V ⇡ ✓ k, ↵ ( ⇢ ) , instead of on the trueoptimization problem.
On the other hand, a smaller t k saves inner loop iterations but the updatefollows less closely to the underlying mirror-ascent update in improving the value vector.
In ourexperiments, we choose t k within 10 to strike a balance and empirically observe t k > 1 has betterperformance.
We note that when t k = 1 , the gradient r ✓ ˜ V ⇡ ✓ k k, ↵ ( ⇢ ) =
r ✓ ˜ V ⇡ ✓ k ˜ r k ( ⇢ ) , since D d ⇡✓⇢ ( ⇡ ✓ || ⇡ ✓ k ) has zerogradient at ✓ = ✓ k .
The update in (5) reduces to an NPG update on the unregularized value function˜ V ⇡ ✓ ˜ r k ( ⇢ ) .
For single-objective MDPs, it reduces to the canonical NPG method.
3.1 Theoretical guarantee of ARNPG
We now present the main theoretical tool for the analysis of the ARNPG framework.
Recall thediscussion of the fundamental inequality after (2).
Proposition 1 establishes such a fundamentalinequality with controllable approximation error under the softmax policy parameterization, i.e.,⇡ ✓ ( a | s ) =
exp( ✓ s,a )
P a 0 exp( ✓ s,a 0 ) .
In the rest of the paper, we omit ✓ in ⇡ ✓ when it is clear from the context,but it should be noted that all updates of policies are performed on the parameters.
Proposition 1.
Under the softmax parameterization, given ✏ k > 0 , for any ˜ r k , t k 11   log( 5 k ˜ r k k 1 (1   ) 2 ✏ k ) + 1 , ↵ > 0 and ⌘ = 1  ↵ , the update ⇡ k +
InnerLoop ( ⇡ k , ˜ r k , ↵ , ⌘ , t k )satisﬁes
[Equation]
The inequality (6) is critical to the convergence proof.
Its right hand side allows telescoping, whichby summing over k can iteratively cancel the terms D d ⇡⇢ ( ⇡ || ⇡ k ) .
Since t k = ⇥ (log(1 / ✏ k )) it sufﬁcesto use very few iterations in InnerLoop for maintaining precision.
Remark.
It has been shown that for the entropy-regularized MDP, i.e., KL-regularized with the uniformpolicy as the anchor policy, NPG converges linearly (i.e., geometrically fast) to the regularized optimalpolicy [11].
It is natural to anticipate that for the KL-regularized MDP ˜ V ⇡ k, ↵ ( ⇢ ) with anchor ⇡ k , NPGwould similarly converge linearly (i.e., ˜ V ⇡ k k, ↵  ˜ V ⇡ ⇤ k k, ↵  ✏ for t k = ⇥ (log(1 / ✏ )) ) to a correspondingoptimal policy, denoted as ⇡ ⇤ k .
In contrast, the right hand side of inequality (6) has a positive drift↵ D d ⇡⇢ ( ⇡ || ⇡ k +1 )
for any policy ⇡ , which is considerably stronger.
Proof sketch of Proposition 1.
We can show that InnerLoop approximately solves the variationalupdate in (4) with linear convergence as anticipated.
However to establish (6), the difﬁculty lies in theintroduction of positive drift, since V ⇡ ✓ ˜ r k ( ⇢ ) is not concave w.r.t.
✓ and D d ⇡✓⇢ ( ⇡ ✓ || ⇡ ✓ k ) may not be a
Bregman divergence.
We tackle this difﬁculty by showing that optimizing ⇡ ✓ in InnerLoop implicitlyperforms a mirror ascent update for state action visitation d ⇡ ✓ ⇢ .
As demonstrated in the next section, Proposition 1 ensures that the convergence rate of the algorithmsderived from the ARNPG framework is of the same rate as the underlying ﬁrst-order methods withonly extra logarithmic factors.
Theoretical applications
In this section, we apply the ARNPG framework to several important multi-objective MDP scenariosand obtain new policy optimization algorithms by integrating ﬁrst-order methods in convex optimization.
All the theoretical results presented in this section are under the softmax parameterizationwith exact gradients.
However, the obtained algorithms can be implemented in more general settingssuch as neural softmax and sample-based scenarios, as in the next section.
We theoretically establish˜ O (1 /T ) convergence of these algorithms by leveraging the fundamental inequality in Proposition 1.
4.1 Smooth concave scalarization function
We start by considering the following optimization problem
[Equation]
where F is a concave function, and  -smooth w.r.t.
k · k 1 norm, i.e., kr F ( v )  r F ( v 0 )
k
 k v  v 0 k 1 .
Since the set of achievable values V ✓ h 0 , 11   i m , it can be veriﬁed that kr F ( v ) k 1  Lfor some factor L > 0 .
The proportional fair criterion discussed in Section 1 can be approximated by F ( v ) :
=
P mi =
a
i log(  + v i ) , where  > 0 is some constant introduced to circumvent the pathologicalcase v i =
for some i 2 [ m ] .
Under this criterion,  =
P mi =1 a i /  2 and L =
P mi =1 a i /  .
When v is viewed as the decision variable, at macro step k with value vector V ⇡ k 1: m ( ⇢ ) , the ascentdirection in a typical gradient ascent step is the gradient ˜ G k
= r v F ( V ⇡ k 1: m ( ⇢ )) .
This naturallydetermines the reward in the ascent direction as ˜ r k ( s, a ) =
h ˜ G k , r 1: m ( s, a ) i .
Adapting the ARNPGframework to this speciﬁc context, we present the algorithm for solving the program (7) in Algorithm2.
We refer to it as “implicit mirror descent" because the algorithm implicitly employs mirror descent.
Algorithm
Input ⇡ 0 , ↵ , ⌘ , t 0: K  1 , K for k = 0 , 1 , . .
, K  1 do Update ⇡ k +
InnerLoop( ⇡ k , ˜ r k , ↵ , ⌘ , t k )
Return the policy in { ⇡ k } Kk =1 with
the largest F ( V ⇡ k 1: m ( ⇢ ))
Let ⇡ ⇤ be the optimal policy for (7).
Based on Proposition 1, we present the following theorem whichguarantees the convergence of ARNPG-IMD with appropriately selected parameters ⇡ 0 , ↵ , ⌘ , t k .
Theorem 1.
For any K  1 , take uniform policy ⇡ 0 , ↵   (1   ) 3 , ⌘ = 1  ↵ , and t k =d 11   log( 5 LK  log( |A| ) )
+ 1 e .
The optimality gap of ARNPG-IMD (Algorithm 2) satisﬁes
[Equation]
There are a total of K macro steps, and the total number of iterations is T =
P
K  1 k =
t k =⇥
( K 1   log( K )) .
The following corollary provides the convergence rate in terms of T .
Corollary 1.
Under the same conditions as in Theorem 1, the ARNPG-IMD algorithm satisﬁes
F
[Equation]
Remark.
In the absence of knowledge of K , we can select time-varying numbers of InnerLoopiterations, such as t k = ⇥ (log( k )) , and ARNPG-IMD will still have the same ˜ O (1 /T ) convergence.
4.2 Constrained Markov decision process
Another way of trading off the objectives is to optimize one while setting hard constraints on theothers.
This can be formulated as the following constrained MDP (CMDP) problem:
[Equation]
where b 2: m 2 [0 , 11   ] m  1 .
Let ⇡ ⇤ =
⇡ ✓ ⇤ be the optimal policy of the CMDP problem in (9).
Deﬁne the Lagrangian of the CMDP problem as L ( ⇡ ✓ ,  ) =
V ⇡ ✓ 1 ( ⇢ )
+
P mi =2  i ( V ⇡ ✓ i ( ⇢ )
b i ) ,where  i is the Lagrange multiplier (dual variable) corresponding to the constraint V
⇡ ✓ i  b i , for 211 each i 2 [2 : m ] .
The Lagrange dual function max ⇡ L ( ⇡ , · ) is a convex function of dual variables 212   0 .
Denote by  ⇤ the optimal dual variables that minimize the Lagrange dual function.
Weassume  ⇤ is ﬁnite, which is guaranteed by Slater’s condition, i.e., there is some ⇡ ✓ and ⇠ > 0 with
V ⇡ ✓ i ( ⇢ )  b i  ⇠ for any i 2 [2 : m ] .
Note ( ⇡ ⇤ ,  ⇤ ) is a saddle point of the Langrangian L ( ⇡ ,  ) .
Thismotivates the primal-dual approach, which iteratively performs gradient ascent for ⇡ ✓ and gradientdescent for  .
This is suitable for the CMDP setting, since for any ﬁxed  , the Lagrangian L ( ⇡ ,  )corresponds to an MDP for which policy gradient can be employed.
The canonical primal-dual gradient ascent-descent method for constrained convex optimization canonly guarantee O (1 / p T ) convergence, and consequently the primal-dual policy gradient-basedapproach for CMDPs [12] has the same convergence.
Recently, Yu et al.
[38] have proposed a primaldual-based method with O (1 /T ) convergence under the Euclidean setting, i.e., B h ( x || y )
=
12 k x  y k 22 .
Adopting ideas from [38], we next propose the ARNPG with Extra Primal-Dual (ARNPG-EPD)algorithm (Algorithm 3).
To the best of our knowledge, this new primal-dual update appears in the
CMDP-related literature for the ﬁrst time.
Note that b i  V ⇡ i ( ⇢ ) is the amount of constraint violation.
There are two key ideas we adopt from[38].
The ﬁrst is the design of the reward in the ascent direction
[Equation]
where an extra constraint violation term is added to the dual variables.
The second idea is that theupdate of dual variables should not fall below the negative constraint violation (the ﬁrst term in (10)),and it can alleviate the overshooting of dual variables.
The extra constraint violation terms in ˜ r k andthe dual update work jointly to ensure the ˜ O (1 /T ) convergence.
Theorem 2.
For any K  1 and ⌘ 0 2 (0 , 1] , take uniform policy ⇡ 0 , ↵  2 ⌘ 0 m (1   ) 3 , ⌘ = 1  ↵ ,and choose t k = d 11   log( 5 L k K 2 ⌘ 0 m log( |A| ) )
+ 1 e with L k = 1 + ⌘ 0 ( m  1) 1   +
P mi =2  k,i .
The averageoptimality gap and the average constraint violation of ARNPG-EPD (Algorithm 3) satisfy
[Equation]
Input
[Equation]
Initialize  0 ,i = max { ⌘ 0 ( V ⇡
i ( ⇢ )  b i ) , 0 } , 8 i 2 [2 : m ] for k = 0 , 1 , . .
, K  1 do Update ⇡ k +
InnerLoop( ⇡
[Equation]
˜ r
[Equation]
, ↵ , ⌘ , t k )
Update
[Equation] [Equation]
Return: a policy randomly chosen from { ⇡ k } Kk =
Note that the number of micro steps t k is chosen according to the dual variables  k in the previoustheorem.
Denote by T :=
P
[Equation]
t k the total number of iterations.
Corollary 2.
Under the same conditions as in Theorem 2, the ARNPG-EPD algorithm satisﬁes
V ⇡
[Equation]
and
[Equation]
K [Equation] The theorem and corollary establish convergence of the average optimality gap and the averageconstraint violation, in the same manner as many previous works [12, 33, 13, 21] on CMDPs.
However, a guarantee on the last iterate is more preferable.
This drawback is inherited from theprimal-dual algorithm for convex optimization, where the primal-dual algorithm with sublinearconvergence can only be guaranteed on the average solution, as of our knowledge.
Last iterateconvergence is still an on-going open research topic.
4.3 Max-min trade-off criteria
Finally, we consider the max-min trade-off criterion deﬁned as
[Equation]
13) where ⇤ is a subset of the m -dimensional probability simplex  ([ m ]) .
We assume  ( · ,  ) is concaveand  ( v, · ) is convex.
We also assume  is  -smooth w.r.t.
the norm ( v
[Equation]
The max-min criterion mentioned in Section 1 can be represented by  ( v,  ) =
P mi =1 v
i
i /c
i and 250 ⇤ =
([ m ]) .
satisﬁes the concave-convex assumption and is  -smooth w.r.t.
the norm with =
O ( m ) .
Denote F ( v ) :
= min  2 ⇤  ( v,  ) , which is concave but not necessarily smooth.
Thus we cannotapply the ARNPG-IMD algorithm
(Algorithm 2) due to the non-smoothness of F , and the subgradientbased method can only guarantee O (1 / p T ) convergence.
We next integrate the optimistic mirror descent ascent (OMDA) method [30] for solving minimaxoptimization in the ARNPG framework.
Denote the gradients ˜
G  k = r   ( V ˜ ⇡ k 1: m ( ⇢ ) , ˜  k ) and ˜ G vk =
r v  ( V ˜ ⇡ k 1: m ( ⇢ ) , ˜
k ) .
It can be veriﬁed that k ˜ G vk k 1  L for some L due to the smoothness of  .
OMDA performs gradient ascent along the direction ˜ G vk w.r.t.
the value vector, and therefore weconstruct the reward in the ascent direction as ˜ r k
( s, a ) =
h ˜ G vk , r 1: m ( s, a ) i .
OMDA performs mirrordescent along direction
˜
G  k w.r.t.
the dual vector  .
A key ingredient of OMDA is that it updatestwice in each macro step.
ARNPG-OMD adopts this idea and update ( ⇡ ,  ) from the same anchorpoints ( ⇡ k ,  k ) , ﬁrst with ascent direction (˜ r k ,  ˜ G  k ) 2 R 2 m and then a further step with direction(˜ r k +1 ,
˜
G  k +1 ) 2 R 2 m .
We present ARNPG-OMDA in Algorithm 4, and establish the following performance guarantees:
Theorem 3.
For any K  1 , take uniform policy ⇡ 0 , ⌘ 0  16  , ↵  6  (1   ) 3 , ⌘ = 1  ↵ , andt k
= d 11   log( 5 LK 6  log( |A| ) )
+ 1 e .
The ARNPG-OMDA algorithm (Algorithm 4) satisﬁes
[Equation]
Input ⇡ 0 ,  0 , ⌘ 0 , ↵ , ⌘ , t 0: K  1 , K
Initialize ˜ ⇡ 0 = ⇡ 0 and  0 , ˜  0 as uniform distribution on [ m ] for k = 0 , 1 , . .
, K  1 do
Update ˜ ⇡ k
+
InnerLoop( ⇡ k , ˜ r k , ↵ , ⌘ , t k ), ˜
k + arg min
[Equation]
h
[Equation]
G
[Equation]
,  [Equation] +
[Equation]
⌘} Update ⇡ k +
InnerLoop( ⇡ k , ˜
r k +1 , ↵ , ⌘ , t k ),  k + arg min  2 ⇤ { h ˜ G
k +1 ,  i + D
(  ||  k ) ⌘}
Return: a policy randomly chosen from { ˜ ⇡ k } Kk =
Similar to the discussion after Corollary 2, Theorem 3 provides a performance guarantee on theaverage value vector F ( 1 K
P
Kk =1 V ˜ ⇡ k 1: m ( ⇢ )) , which is inherited from the OMDA methods.
Denote thetotal number of iterations by T :=
P
K  1 k =0 2 t k .
Corollary 3.
Under the same conditions as in Theorem 3, ARNPG-OMDA satisﬁes F
V ⇡ ⇤ 1: m ( ⇢ )
F
[Equation]
Empirical evaluation and application
In this section, we present the experimental results on CMDP.
We compare the performance of theproposed ARNPG-EPD algorithm (Algorithm 3) with two benchmarks: NPG-PD [12] and CRPO[33].
Experimental details on CMDP are postponed to Appendix A and further experiments onsmooth concave scalarization and max-min trade-off are presented in Appendix B.
5.1 Tabular CMDP with exact gradients
Recall that under softmax policy with exact gradients, Corollary 2 (Theorem 2) guarantees ˜ O (1 /T )convergence of both performance measures: average optimality gap and average constraint violation.
We compare the proposed ARNPG-EPD with the benchmarks NPG-PD and CRPO under bothperformance measures on a randomly generated CMDP with a single constraint, which are illustratedin Figure 1.
The horizontal axis is the total number of iterations, i.e., including the micro steps in
InnerLoop of ARNPG-EPD.
[Figure]
Figures 1(a) and 1(b) show that both the average optimality gap and the average constraint violationof the ARNPG-EPD algorithm converge faster than those of NPG-PD.
Since the CRPO focuses onthe violated constraint, the policy becomes feasible quickly, though at the cost of an initially slowerconvergence for the optimality gap.
As illustrated in Figures 1(c) and 1(d), the slopes of both theoptimality gap and the constraint violation of the ARNPG-EPD algorithm in the log-log plots areapproximately between -0.9 and -1, indicating a converge rate of ˜ O
(1 /T ) .
5.2 Sample-based tabular CMDP
We next consider the same tabular CMDP described in Section 5.1 without exact policy gradients.
Instead, policy gradients are estimated by samples from a generative model that can generate
[Figure]
The performances of CRPO, NPG-PD, and ARNPG-EPD in the sample-based scenario are shown in
Figure 2.
Figures 2(a) and 2(b) display the averaged performance, while Figures 2(c) and 2(d) displaythe performance of the current iterate (a.k.a.
last-iterate in optimization literature).
It shows that inthis sample-based scenario, ARNPG-EPD achieves higher reward values with faster convergence,while all three algorithms satisfy the constraint after a few iterations.
5.3 Acrobot-v1
To demonstrate the efﬁcacy of ARNPG-EPD on complex tasks, we have conducted experimentson the Acrobot-v1 environment from OpenAI Gym [9].
We follow the same experiment setup in[33], where there is a reward value to maximize, and two cost values to be constrained below somethresholds.
The superior performance of ARNPG-EPD is shown in Figure 3.
[Figure]
Figure 3(a) shows that ARNPG-EPD achieves a higher reward value compared to NPG-PD and
CRPO, while Figures 3(b) and 3(c) demonstrate that the cost values of all three algorithms are belowthe thresholds after a few initial iterations.
We believe the superiority is due to the new primal-dualdesign inspired by [37] (discussed in Section 4.2) and the ﬂexibility of choosing t k in the InnerLoopin the framework.
More experiments with different t k are presented in Appendix A.
Conclusion and future works
We propose an ARNPG framework to systematically integrate well-performing ﬁrst-order methodsinto the design of policy gradient-based algorithms for multi-objective MDPs.
The designed algorithms achieve a global ˜ O (1 /T ) convergence rate under the softmax parameterization with exactgradients, and empirically have satisfactory performance beyond tabular and exact gradient settings.
We believe that ARNPG has potential applications in other scenarios, since the general and ﬂexibleframework allows integration with more advanced ﬁrst-order methods, currently and in the future.
A natural future direction is to extend the theoretical results to more general settings such as functionapproximation and sample-based scenarios.
Viewing ARNPG as a heuristic, the anchor-changingideas can also be applied to policy optimization for multi-agent RL and meta RL.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 5.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Lines 240-245 and 268-269.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] After the deﬁnition of the problems (7),(9),(13)(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C, D,and E.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A and B.
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] We only report the mean in Figure 3 for clarity.
InAppendix A, we will report error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See supplemental material.
(b) Did you mention the license of the assets?
[Yes] See supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]