Constrained Update Projection Approach to Safe Policy Optimization
s
Abstract
Safe reinforcement learning (RL) studies problems where an intelligent agent hasto not only maximize reward but also avoid exploring unsafe areas.
In this study,we propose CUP, a novel policy optimization method based on C onstrained U pdate
P rojection framework that enjoys rigorous safety guarantee.
Central to our CUPdevelopment is the newly proposed surrogate functions along with the performancebound.
Compared to previous safe RL methods, CUP enjoys the beneﬁts of 1)
CUP generalizes the surrogate functions to generalized advantage estimator (GAE),leading to strong empirical performance.
2) CUP uniﬁes performance bounds,providing a better understanding and interpretability for some existing algorithms;3) CUP provides a non-convex implementation via only ﬁrst-order optimizers,which does not require any strong approximation on the convexity of the objectives.
To validate our CUP method, we compared CUP against a comprehensive list ofsafe RL baselines on a wide range of tasks.
Experiments show the effectiveness of
CUP both in terms of reward and safety constraint satisfaction.
Introduction
Reinforcement learning (RL) [Sutton and Barto, 1998] has achieved signiﬁcant successes in manyﬁelds (e.g., [Mnih et al. , 2015; Silver et al. , 2017; OpenAI, 2019; Afsar et al. , 2021]).
However,most RL algorithms improve the performance under the assumption that an agent is free to exploreany behaviors.
In real-world applications, only considering return maximization is not enough, andwe also need to consider safe behaviors.
For example, a robot agent should avoid playing actionsthat irrevocably harm its hardware, and a recommender system should avoid presenting offendingitems to users.
Thus, it is crucial to consider safe exploration for RL, which is usually formulated asconstrained Markov decision processes (CMDP)
[Altman, 1999].
It is challenging to solve CMDP since traditional approaches (e.g., Q-learning [Watkins, 1989] &policy gradient [Williams, 1992]) usually violate the safe exploration constraints, which is undesirablefor safe RL.
Recently, Achiam et al.
[2017]; Yang et al.
; Bharadhwaj et al.
[2021] suggest touse some surrogate functions to replace the objective and constraints.
However, their implementationsinvolve some convex approximations to the non-convex objective and safe constraints, which leads tomany error sources and troubles.
Concretely, Achiam et al.
[2017]; Yang et al.
[2020]; Bharadhwaj etal.
[2021] approximate the non-convex objective (or constraints) with ﬁrst-order or second Taylorexpansion, but their implementations still lack a theory to show the error difference between theoriginal objective (or constraints) and its convex approximations.
Besides, their approaches involvethe inverse of a high-dimension inverse Fisher information matrix, which causes their algorithmsrequire a costly computation for each update when solving high-dimensional RL problems.
To address the above problems, we propose the constrained update projection (CUP) algorithm witha theoretical safety guarantee.
We derive the CUP bases on the newly proposed surrogate functions with respect to objectives and safety constraints, and provide a practical implementation of CUP thatdoes not depend on any convex approximation to adapt high-dimensional safe RL.
Concretely, in Section 3, Theorem 1 shows generalized difference bounds between two arbitrarypolicies for the objective and constraints.
Those bounds provide principled approximations to theobjective and constraints, which are theoretical foundations for us to use those bounds as surrogatefunctions to replace objective and constraints to design algorithms.
Although using difference boundsas surrogate functions to replace the objective has appeared in previous works (e.g., [Schulman etal. , 2015; Achiam et al. , 2017]), Theorem 1 reﬁnes those bounds (or surrogate functions) at leasttwo aspects: (i) Firstly, our rigorous theoretical analysis shows a bound with respect to generalizedadvantage estimator (GAE)
[Schulman et al. , 2016].
GAE signiﬁcantly reduces variance empiricallywhile maintaining a tolerable level of bias, the proposed bound involves GAE is one of the criticalsteps for us to design efﬁcient algorithms.
(ii)
Our new bounds unify the classic result of CPO
Achiam et al.
[2017], i.e., the classic performance bound of CPO is a special case of our bounds.
Although existing work (e.g., Zhang et al.
[2020]; Kang et al.
[2021]) has applied the key idea of
CPO with GAE to solve safe RL problems, their approaches are all empirical and lack a theoreticalanalysis.
Thus, the proposed newly bound partially explains the effectiveness of the above safe
RL algorithms.
Finally, we should emphasize that although GAE has been empirically applied toextensive reinforcement learning tasks, this work is the ﬁrst to show a rigorous theoretical analysis toextend the surrogate functions with respect to GAE.
In Section 4, we provide the necessary details of the proposed CUP.
The CUP contains two steps: itﬁrst performs a policy improvement, which may produce a temporary policy violates the constraint.
Then in the second step, CUP projects the policy back onto the safe region to reconcile the constraintviolation.
Theorem 2 shows the worst-case performance degradation guarantee and approximatesatisfaction of safety constraints of CUP, result shows that with a relatively small parameter thatcontrols the penalty of the distance between the old policy and current policy, CUP shares a desirabletoleration for both policy improvements and safety constraints.
Furthermore, we provide a practicalimplementation of sample-based CUP.
This implementation allows us to use deep neural networks totrain a model, which is an efﬁcient iteration without strongly convex approximation of the objectiveor constraints (e.g., [Achiam et al. , 2017;
Yang et al. , 2020]), and it optimizes the policy accordingto the ﬁrst-order optimizer.
Finally, extensive high-dimensional experiments on continuous controltasks show the effectiveness of CUP where the agent satisﬁes safe constraints.
Preliminaries
Reinforcement learning (RL) [Sutton and Barto, 1998] is often formulated as a Markov decisionprocess (MDP)
[Puterman, 2014] that is a tuple M =
( S , A , P , r, ρ 0 , γ ) .
Here S is state space,
A is action space.
P ( s (cid:48) | s, a ) is probability of state transition from s to s (cid:48)
after playing a .
r ( · ) :
S × S ×
A → R , and r ( s (cid:48) | s, a ) denotes the reward that the agent observes when state transition froms to s (cid:48) after it plays a .
ρ 0 ( · ) : S → [0 , 1] is the initial state distribution and γ ∈ (0 , 1) .
A stationary parameterized policy π θ is a probability distribution deﬁned on S × A , π θ ( a | s ) denotesthe probability of playing a in state s .
We use Π θ to denote the set of all stationary policies, whereΠ θ = { π θ : θ ∈ R p } , and θ is a parameter needed to be learned.
Let P π θ ∈ R |S|×|S| be a statetransition probability matrix, and their components are: P π θ [ s, s (cid:48) ]
= (cid:80) a ∈A π θ ( a | s ) P ( s (cid:48) | s, a ) =:
P π θ ( s (cid:48) | s ) , which denotes one-step state transformation probability from s to s (cid:48) by executing π θ .
Let τ = { s t , a t , r t +1 } t ≥ 0 ∼ π θ be a trajectory generated by π θ , where s 0 ∼ ρ 0 ( · ) , a t ∼ π θ ( ·| s t ) ,s t +1 ∼ P ( ·| s t , a t ) , and r t +
= r
( s t +1 | s t , a t ) .
We use P π θ ( s t = s (cid:48) | s ) to denote the probability ofvisiting the state s (cid:48) after t time steps from the state s by executing π θ .
Due to the Markov property in
MDP, P π θ ( s t =
s (cid:48) | s ) is ( s, s (cid:48) )
-th component of the matrix P
tπ θ , i.e., P π θ ( s t =
s (cid:48) | s )
=
P tπ θ [ s, s (cid:48) ] .
Finally, let d s 0 π θ ( s ) =
(1 − γ ) (cid:80) ∞
t =
γ t P π θ ( s t =
s | s 0 ) be the stationary state distribution of the
Markov chain (starting at s 0 ) induced by policy π θ .
We deﬁne d ρ 0 π θ ( s ) =
E s 0 ∼ ρ 0 ( · ) [ d s
π θ ( s )]
as thediscounted state visitation distribution on initial distribution ρ 0 ( · ) .
The state value function of π θ is deﬁned as V π θ ( s ) =
E π θ [ (cid:80) ∞ t =
γ t r t +1 | s 0 =
s ] , where E π θ [ ·|· ]denotes a conditional expectation on actions which are selected by π θ .
Its state-action valuefunction is Q π θ ( s, a ) =
E π θ [ (cid:80) ∞ t =
γ t r t +1 | s 0 =
s, a 0 = a ] , and advantage function is
A
2.1 Policy Gradient and Generalized Advantage Estimator (GAE)
Policy gradient [Williams, 1992; Sutton et al. , 2000] is widely used to solve policy optimization,which maximizes the expected total reward by repeatedly estimating the gradient g = ∇ J ( π θ ) .
Schulman et al.
[2016] summarize several different related expressions for the policy gradient:
[Equation]
where Ψ t can be total discounted reward of the trajectory, value function, advantage function ortemporal difference (TD) error.
As stated by Schulman et al.
[2016], the choice Ψ t =
A ( s t , a t ) yieldsalmost the lowest possible variance, which is consistent with the theoretical analysis [Greensmithet al. , 2004; Wu et al. , 2018].
Furthermore, Schulman et al.
[2016] propose generalized advantageestimator ( GAE )
ˆ A GAE ( γ,λ ) t ( s t , a t ) to replace Ψ t : for any λ ∈ [0 , 1] ,
[Equation]
where δ Vt = r t +1 + γV ( s t +1 ) − V ( s t ) is TD error, and V ( · ) is an estimator of value function.
GAEis an efﬁcient technique for data efﬁciency and reliable performance of reinforcement learning.
2.2 Safe Reinforcement Learning
Safe RL is often formulated as a constrained MDP (CMDP)
M ∪ C
[Altman, 1999], which isa standard MDP M augmented with an additional constraint set C .
The set C = { ( c i , b i ) } mi =1 ,where c i are cost functions: c i : S ×
A → R , and limits are b i , i = 1 , · , m .
The cost-returnis deﬁned as: J c i ( π θ ) =
E π θ [ (cid:80) ∞ t =
γ t c
i ( s t , a t )] , then we deﬁne the feasible policy set Π C as:Π C =
∩ mi =1 { π θ ∈ Π θ
and J c i ( π θ ) ≤ b i } .
The goal of CMDP is to search the optimal policy π (cid:63) :
[Equation]
Furthermore, we deﬁne value functions, action-value functions, and advantage functions for theauxiliary costs in analogy to V π θ , Q π θ , and A π θ , with c
i replacing r respectively, we denote themas
V c
i π θ , Q c i π θ , and A c
i π θ .
For example, V c i π θ ( s ) =
E π θ [ (cid:80) ∞ t =
γ t c
i ( s t , a t ) | s 0 = s ] .
Without loss ofgenerality, we will restrict our discussion to the case of one constraint with a cost function
c andupper bound b .
Finally, we extend the GAE with respect to auxiliary cost function c :
[Equation]
where δ Ct = r t +1 + γC
( s t +1 ) − C ( s t ) is TD error, and C ( · ) is an estimator of cost function c .
Generalized Policy Performance Difference Bounds
In this section, we show some generalized policy optimization performance bounds for J ( π θ ) and
J c ( π θ ) .
The proposed bounds provide some new surrogate functions with respect to the objective andcost function, which are theoretical foundations for us to design efﬁcient algorithms to improve policyperformance and satisfy constraints.
Before we present the new performance difference bounds, letus revisit a classic performance difference from Kakade and Langford [2002],
[Equation]
Eq.(5) shows a difference between two arbitrary policies π θ and π θ (cid:48) with different parameters θ andθ (cid:48) .
According to (5), we rewrite the policy optimization (3) as follows
[Equation]
However, Eq.(5) or (6) is very intractable for sampling-based policy optimization since it requires thedata comes from the (unknown) policy π θ that needed to be learned.
In this section, we provide abound reﬁnes the result (5), which provide the sights for surrogate functions to solve problem (3).
3.1 Some Additional Notations
We use a bold lowercase letter to denote a vector, e.g., a = ( a 1 , a 2 , · · · , a n ) , and its i -th elementa [ i ] =: a i .
Let ϕ ( · ) :
S → R be a function deﬁned on S , δ ϕt =
r ( s t +1 | s t , a t )
+ γϕ ( s t +1 )
− ϕ ( s t )is TD error with respect to ϕ ( · ) .
For two arbitrary policies π θ and π θ (cid:48) , we denote δ ϕπ θ ,t ( s ) as theexpectation of TD error, and deﬁne ∆ ϕt ( π θ , π θ (cid:48) , s ) as the difference between δ ϕπ θ ,t ( s ) and δ ϕπ θ (cid:48) ,t ( s ) :
[Equation]
[Equation]
Furthermore, we introduce two vectors δ ϕπ θ ,t , ∆ ϕt ( π θ , π θ (cid:48) )
∈ R |S| , and their components are:
[Equation]
Let matrix P (
[Equation]
θ , where λ ∈ [0 , 1] .
It is similar to the normalizeddiscounted distribution d ρ
π θ ( s ) , we extend it to λ -version and denote it as d λπ θ ( s ) :
[Equation]
where ˜ γ = γ (1 − λ ) 1 − γλ , the probability P ( λ ) π θ ( s t = s
| s 0 ) is the ( s 0 , s ) -th component of the matrix product(cid:16) P ( λ )
π θ (cid:17) t .
Finally, we introduce a vector d λπ θ ∈ R |S| , and its components are: d λπ θ [ s ]
= d λπ θ ( s ) .
3.2 Main Results
Theorem 1 (Generalized Policy Performance Difference) .
For any function ϕ ( · ) : S → R , for twoarbitrary policies π θ and π θ (cid:48) , for any p, q ∈ [1 , ∞ ) such that 1 p + 1 q = 1 , we deﬁne two error terms:
[Equation] [Equation]
Then, the following bound with respect to policy performance difference J ( π θ ) − J ( π θ (cid:48) ) holds:
[Equation]
Proof.
See Appendix E.
The bound (10) is well-deﬁned, i.e., if π θ = π θ (cid:48) , all the three terms in Eq.(10) are zero identically.
From Eq.(9), we know the performance difference bound L ϕ, ± p,q ( π θ , π θ (cid:48) )
(10) can be interpreted bytwo distinct difference parts: (i) the ﬁrst difference part, i.e., the expectation ∆ ϕ t ( π θ , π θ (cid:48) , s ) , whichis determined by the difference between TD errors of π θ and π θ (cid:48) ; (ii) the second difference part,i.e., the discounted distribution difference (cid:15) ϕ, ( λ ) p,q,t ( π θ , π θ (cid:48) ) , which is determined by the gap betweenthe normalized discounted distribution of π θ and π θ (cid:48) .
Thus, the difference of both TD errors anddiscounted distribution determine the policy difference J ( π θ ) − J ( π θ (cid:48) ) .
The different choices of p and q lead Eq.(10) to be different bounds.
If p = 1 , q = ∞ , we denote(cid:15) ϕπ θ ,t
=: (cid:107) δ ϕπ θ ,t (cid:107) q = max s t ∈S E a t ∼ π θ ( ·| s t ) ,s t +1 ∼ P ( ·| s t ,a t ) [ | δ ϕt | ] , then, according to Lemma 2 (see
Appendix F.2), when p = 1 , q = ∞ , then error (cid:15) ϕ, ( λ ) p,q,t ( π θ , π θ (cid:48) ) is reduced to:
[Equation]
where D TV ( π θ (cid:48) , π θ )
[ s ] is the total variational divergence between action distributions at state s , i.e.,
[Equation]
Finally, let ϕ =
V π θ (cid:48) , the left side of (10) in Theorem 1 implies a lower bound of performancedifference, which illustrates the worse case of approximation error, we present it in Proposition 1.
Proposition 1.
For any two policies π θ and π θ (cid:48) , let (cid:15)
Vπ θ
( π θ
[Equation]
) =:
[Equation] [Equation]
=
[Equation]
} , then
[Equation]
The reﬁned bound (11) contains GAE technique that signiﬁcantly reduces variance while maintaininga tolerable level of bias empirically [Schulman et al. , 2016], which implies using the bound (11)
asa surrogate function could improve performance potentially for practice.
Although GAE has beenempirically applied to extensive reinforcement learning tasks, to the best of our knowledge, the result(11) is the ﬁrst to show a rigorous theoretical analysis to extend the surrogate functions to GAE .
Remark 1 (Uniﬁcation of [Achiam et al. , 2017]) .
If λ → 0 , then the distribution d λπ θ (cid:48) ( · ) is reducedto d ρ 0 π θ (cid:48) ( · ) and the bound (11) is reduced to
[Equation] [Equation]
which matches the result of [Achiam et al. , 2017, Corollary 1].
That is to say the proposed bound(11) uniﬁes
the classic bound (12)
Let ϕ =
V cπ θ (cid:48)
, Theorem 1 implies an upper bound of cost function as presented in the next Proposition2, we will use it to make guarantee for safe policy optimization.
Proposition 2.
For any two policies π θ and π θ (cid:48) , let (cid:15)
Cπ θ
( π θ (cid:48) )
=: sup t ∈ N + { (cid:15) ϕπ θ ,t : ϕ
=
V cπ θ (cid:48) } , then
[Equation]
where we calculate A GAE ( γ,λ ) π θ (cid:48) ,C ( s, a ) according to the data sampled from π θ (cid:48) and the estimator (4).
All above bound results (11) and (13) can be extended for a total variational divergence to KLdivergence between policies, which are desirable for policy optimization.
Proposition 3.
All the bounds in (11) and (13) hold if we make the following substitution:
[Equation]
where KL( · , · ) is KL-divergence, and KL( π θ (cid:48) , π θ )
[ s ] =
KL( π θ (cid:48) ( ·| s ) , π θ ( ·| s )) .
CUP: Constrained Update Projection
According to the bounds in Proposition 1-3, we develop new surrogate functions to replace theobjective and constraints.
We propose the CUP (constrained update projection) algorithm that is atwo-step approach contains performance improvement and projection .
Due to the limitation of space,we present all the details of the implementation in Appendix C and Algorithm 1.
Step 1: Performance Improvement.
According to Proposition 1 and Proposition 3, for an appropriate coefﬁcient α k , we update policy as:
[Equation]
 
[Equation]
This step is a typical minimization-maximization (MM) algorithm
[Hunter and Lange, 2004], itincludes return maximization and minimization the distance between old policy and new policy.
theexpectation (14) by sample averages according to the trajectories collected by π θ k .
Step 2: Projection.
According to Proposition 2 and Proposition 3, for an appropriate coefﬁcient β k ,we project the policy π θ k +12 onto the safe constraint set,
[Equation]
where D ( · , · ) (e.g., KL divergence or (cid:96) 2 -norm) measures distance between π θ k +12 and π θ ,
[Equation]
Until now, the particular choice of surrogate function is heuristically motivated, we show the worstcase performance degradation guarantee and approximate satisfaction of safety constraints of CUP in
Theorem 2, and its proof is shown in Appendix G.
Theorem 2.
Let χ
k
=
E s ∼ d λπ θ k ( · ) [ KL ( π θ k , π θ k +12 )
[ s ]] , ι =: ˜ γ ( γλ ( |S|− 1)+1) (1 − ˜ γ )(1 − γλ ) .If
π θ k and π θ k +1 aregenerated according to (14)-(15), then the lower bound on policy improvement, and upper bound onconstraint violation are
[Equation]
Remark 2 (Asymptotic Safety Guarantee) .
Let α k → 0 , β k
→ 0 as k → ∞ , Theorem 2 implies amonotonic policy improvement with an asymptotic safety guarantee.
Practical Implementation .
Let { ( s t , a t , r t +1 , c t +1 ) }
Tt =1 ∼ π θ k , we denote the empirical KLdivergence with respect to π θ and π θ (cid:48) as: ˆ D KL ( π θ , π θ (cid:48) )
= 1 T (cid:80) Tt =1 KL( π θ ( a t | s t ) , π θ (cid:48)
( a t | s t )) .
Weupdate performance improvement (14) step as follows,
[Equation]
where ˆ A t is an estimator of A GAE ( γ,λ )
π θ k
( s, a ) .
Then we update the projection step by replacing the distance D by KL-divergence, the next Theorem3 (for its proof, see Appendix C.2) provides a fundamental way for us to solve projection step (15).
Theorem 3.
The constrained problem (37) is equivalent to the following primal-dual problem:
[Equation]
According to Theorem 3, we solve the constraint problem (15) by the following primal-dual approach,
[Equation]
where ˆ C ( π θ , π θ k )
=
ˆ J C +
[Equation]
˜ γ · 1 T (cid:80) Tt =1 π θ ( a t | s t )
π θ k
( a t | s t ) ˆ A Ct + β k (cid:113) ˆ D KL ( π θ k , π θ ) − b, ˆ J C and ˆ A Ct areestimators for cost-return and cost-advantage.
Finally, let
[Equation]
we update the parameters ( θ k +1 , ν k +1 ) as follows,
[Equation]
where η > 0 is step-size, {·} + denotes the positive part, i.e., if x ≤ 0 , { x } + = 0 , else { x } + = x .
We have shown all the details of the implementation in Algorithm 1.
Related Work
Due to the limitation of space, for more discussions, see Appendix B and Table 2.
Local Policy Search and Lagrangian Approach .
A direct way to solve CMDP (3) is to apply localpolicy search
[Peters and Schaal, 2008; Pirotta et al. , 2013] over the policy space Π C , i.e.,
[Equation]
and
[Equation]
where δ is a positive scalar, D ( · , · ) is some distance measure.
For practice, the local policy search (19)is challenging to implement because it requires evaluation of the constraint function c to determinewhether a proposed point π is feasible [Zhang et al. , 2020].
Besides, Li and Belta [2019]; Cheng etal.
[2019]; Liu et al.
[2020] provide a local policy search via the barrier function.
The key idea of theproposed CUP is parallel to Barrier functions.
When updating policy according to samples, localpolicy search (19) requires off-policy evaluation [Achiam et al. , 2017], which is very challenging forhigh-dimension control problem
[Duan et al. , 2016; Yang et al. , 2018, 2021a].
A way to solve CMDP (3) is Lagrangian approach that is also known as primal-dual problem:
[Equation]
Although extensive canonical algorithms are proposed to solve problem (20), e.g., [Liang et al. , 2018;
Tessler et al. , 2019; Paternain et al. , 2019; Le et al. , 2019; Russel et al. , 2020; Xu et al. , 2020; Satijaet al. , 2020; Chen et al. , 2021], the policy updated by Lagrangian approach may be infeasible w.r.t.
[Equation]
This is hazardous in reinforcement learning when one needs to execute the intermediatepolicy (which may be unsafe) during training [Chow et al. , 2018].
Constrained Policy Optimization (CPO) .
Recently, Achiam et al.
[2017] suggest to replace thecost constraint with a surrogate cost function which evaluates the constraint J c ( π θ )
according tothe samples collected from the current policy π θ k , see Eq.(21)-(23), Appendix B.1.
Existing recentworks (e.g., [Achiam et al. , 2017; Vuong et al. , 2019;
Yang et al. , 2020; Han et al. , 2020; Bisi etal. , 2020; Bharadhwaj et al. , 2021]) try to ﬁnd some convex approximations to replace the term
A π θ k ( s, a ) and ¯ D KL ( π θ , π θ k ) in Eq.(21)-(23).
Such ﬁrst-order and second-order approximations turna non-convex problem (21)-(23) to be a convex problem, it seems to make a simple solution, but thisapproach results in many error sources and troubles in practice.
Firstly, it still lacks a theory analysisto show the difference between the non-convex problem (21)-(23) and its convex approximation.
Policy optimization is a typical non-convex problem [Yang et al. , 2021b]; its convex approximationmay introduce some error for its original issue.
Secondly, CPO updates parameters according toconjugate gradient [Süli and Mayers, 2003], and its solution involves the inverse Fisher informationmatrix, which requires expensive computation for each update.
Instead of using a convex approximation for the objective function, the proposed CUP algorithmimproves CPO and PCPO at least two aspects.
Firstly, the CUP directly optimizes the surrogateobjective function via the ﬁrst-order method, and it does not depend on any convex approximation.
Thus, the CUP effectively avoids the expensive computation for the inverse Fisher information matrix.
Secondly, CUP extends the surrogate objective function to GAE.
Although Zhang et al.
[2020]has used the GAE technique in experiments, to the best of our knowledge, it still lacks a rigoroustheoretical analysis involved GAE before we propose CUP.
Experiment
In this section, we aim to answer the following three issues: (I) .
Does CUP satisfy the safetyconstraints in different environments?
Does CUP performs well with different cost limits?
(II) .
Howdoes CUP compare to the state-of-the-art safe RL algorithms?
(III) .
Does CUP play a sensibilityduring the hyper-parameters in the tuning processing?
We train different robotic agents using ﬁve MuJoCo physical simulators [Todorov et al. , 2012] whichare open by OpenAI Gym API [Brockman et al. , 2016], and Safety Gym [Ray et al. , 2019].
Formore details, see Appendix H.2.
Baselines includes CPO [Achiam et al. , 2017], PCPO [Yang et al. ,2020], TRPO Lagrangian (TRPO-L), PPO Lagrangian (PPO-L) and FOCOPS [Zhang et al. , 2020].
TRPO-L and PPO-L are improved by [Chow et al. , 2018; Ray et al. , 2019], which are based on TRPO[Schulman et al. , 2015] and PPO [Schulman et al. , 2017].
These two algorithms use the Lagrangianmethod [Bertsekas, 1997], which applies adaptive penalty coefﬁcients to satisfy the constraint.
[Figure]
n
[Figure]
t r a i n t
[Figure] [Figure]
6.1 Evaluation CUP and Comparison Analysis
We have shown the Learning curves for CUP, and other baselines in Figure 1-2, and Table 1 summarizes the performance of all algorithms.
Results show that CUP quickly stabilizes the constraintreturn around the limit value while converging the objective return to higher values faster.
In mostcases, the traces of constraint from CUP almost coincide with the dashed black line of the limit.
Bycontrast, the baseline algorithms frequently suffer from over or under the correction.
From Figure 1, we know initial policies of the baseline algorithms are not guaranteed to be feasible,such as in Swimmer-v3, while CUP performs the best and keeps safety learning in Swimmer-v3 tasks.
In the HumanoidCircle task, all the algorithms learn steadily to obtain a safe policy, except PPO-L.
Additionally, we observed that CUP brings the policy back to the feasible range faster than otherbaselines in the HumanoidCircle task.
In the Ant-v3 task, only the FOCOPS and the proposed CUPlearn safely, and both CPO and TRPO-L violate the safety constraints signiﬁcantly.
Besides, although
FOCOPS and CUP converge to a safe policy, CUP obtains a better reward performance than FOCOPSin the Ant-v3 task.
The result of Figure 2 is relatively complex, the initial policies of the CPO and
PCPO are not guaranteed to be feasible on both Safexp-PointGoal1-v0 and Safexp-PointButton1-v0.
We think it is not accidental, but it partially provides corroboration of the previous discussions in
Appendix B. Both CPO and PCPO use ﬁrst-order and second-order approximation to approximate anon-convex problem as a convex problem, which inevitably produces a signiﬁcant deviation from theoriginal RL problem, and it is more serious in large-scale and complex control systems.
From Table 1, we know although PPO-L achieves a reward of 35 .
58 ± 5 .
68 outperforms CUP in
Swimmer-v3, PPO-L obtains a cost of 54 .
91 ± 3 . 93 that violates the cost limit of 24 .
5 signiﬁcantly,which implies PPO-L learns a dangerous policy under this setting.
On the other hand, Figure 1has shown that CUP generally gains higher returns than different baselines while enforcing thecost constraint.
Mainly, CUP achieves a reward performance of 2025 .
56 ± 122 . 35 that signiﬁcantlyoutperforms all the baseline algorithms.
Additionally, after equal iterations, CUP performs a greaterspeed of stabilizing the constraint return around the limit value and is quicker to ﬁnd feasible policiesto gain a more signiﬁcant objective return.
[Table] [Table]
[Figure]
6.2 Sensitivity
Analysis for Hyper-Parameters Tuning
Hyper-parameters tuning is necessary to achieve efﬁcient policy improvement and enforce constraints.
We investigate the performance with respect to the parameters: ν , step-size α , and cost limit b .
From Figure 3 (a), we know if the estimated cost under the target threshold b , then ν keeps calm,which implies ν is not activated.
Such an empirical phenomenon gives signiﬁcant expression to the
Ant-v3, Humanoid-v3, and Hopper-v3 environments.
While if the estimated cost exceeds the targetthreshold b , ν will be activated, which requires the agent to play a policy on the safe region.
Thoseempirical results are consistent with the update rule of ν : ν k
+1 =
{ ν k + η ( ˆ J Ck − b ) } + , which impliesthe projection of CUP plays an important role for the agent to learn a safe policy.
Furthermore,
Figure 3 (b) shows that the performance of CUP is still very stable for different settings of α , and theconstraint value of CUP also still ﬂuctuates around the target value.
The different value achievedby CUP in different setting α is affected by the simulated environment and constraint thresholds,which are easy to control.
Finally, Figure 3 (c) shows that CUP learns safe policies stably under thecost limit thresholds.
We compare policy performance and cost under different cost limit settings.
For example, in the Swimmer-v3, we set cost limit b among { 10 , 15 , 20 , 25 , 30 } .
Different cost limitsetting implies different difﬁculty for learning, results show that CUP is scalable to various complextasks, which means CUP is robust to different cost limit settings for various safe RL tasks.
Conclusion
This paper proposes the CUP algorithm with a theoretical safety guarantee.
We derive the CUP basedon the newly proposed surrogate functions with respect to objectives and constraints and provide apractical implementation of CUP that does not depend on any convex approximation.
We compared
CUP against a comprehensive list of safe RL baselines on a wide range of tasks, which shows theeffectiveness of CUP where the agent satisﬁes safe constraints.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] See Abstract and Section 1.(b)
Did you describe the limitations of your work?
[No](c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We ensure our paper to conform to the ethics review guidelines.2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Section 4.(b)
Did you include complete proofs of all theoretical results?
[Yes] See Appendix E andAppendix G.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See the URL in
the supplementary material for the code, and see H for environments of experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix H
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix H
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix H
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We provide the code for our implementation of CUP in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We use open source safe reinforcement learning environments,
see Appendix H.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] Our data does not contain any personally
identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A NotationsA.1 Matrix Index
In this paper, we use a bold capital letter to denote matrix,
e.g., A =
( a i,j ) ∈ R m × n , and its ( i, j ) -thelement denoted as
[Equation]
where 1 ≤ i ≤ m, 1 ≤ j ≤ n .
Similarly, a bold lowercase letter denotes a vector, e.g., a =( a 1 , a 2 , · · · , a n ) ∈ R n , and its i -th element denoted asa
[ i ]
=: a i , where 1 ≤ i ≤ n .
A.2 Key Notations of Reinforcement Learning
For convenience of reference, we list key notations that have be used in this paper
A.2.1 Value Function and Dynamic System of MDP.
r
[Equation]
: r π θ ∈ R |S| is the expected vector reward according to π θ , i.e., their components are: r π
[Equation]
s
[Equation]
R
[Equation]
s
[Equation]
S .
v
[Equation]
, V
[Equation]
( s ) , : v π θ ∈ R |S| is the vector that stores all the state value functions, and its components are: v π θ [ s ]
=
V π θ ( s ) , s ∈ S .
ρ ( · ) , ρ : ρ ( s ) : the initial state distribution of state s ; ρ ∈ R |S| , and ρ [ s ]
= ρ ( s ) .
P
[Equation]
θ :
Single-step state transition matrix by executing π θ .
P π θ ( s (cid:48) | s ) :
Single-step state transition probability from s to s (cid:48) by executing π θ , and it is the ( s, s (cid:48) )
-th component of the matrix P π θ , i.e., P π θ [ s, s (cid:48) ]
= P
[Equation]
| s
[Equation]
P
[Equation]
( s
[Equation]
s (cid:48)
[Equation]
s ) :
The probability of visiting the state s (cid:48) after t time steps from the state s by executing π θ , and it is the ( s, s (cid:48) )
-th component of the matrix P π θ , i.e.,
P tπ θ [ s, s (cid:48) ]
=
P π θ ( s t =
s (cid:48) | s ) .
d s 0 π
[Equation]
( s ) , d
[Equation] [Equation]
( s ) :
The normalized discounted distribution of the future state s encountered starting at s 0 by executing π θ : d s
π θ ( s )
=:
(1 − γ ) (cid:80) ∞
t =
γ t P π θ ( s t =
s | s 0 ) .
Since s 0 ∼ ρ ( · ) , we deﬁne d ρ 0 π θ ( s )
=
E s 0 ∼ ρ ( · ) [ d s
π θ ( s )] .
d ρ 0 π θ :
It stores all the normalized discounted state distributions d ρ 0 π θ ( s ) , ∈ S , i.e., d ρ 0 π θ ∈ R |S| , and its components are: d ρ 0 π θ [ s ] =
d ρ 0 π θ ( s ) .
A.2.2 Extend them to λ -version.
[Equation]
[Equation] [Equation]
r
[Equation] [Equation]
[Equation] [Equation]
π θ , π θ (cid:48) , s ) .
B Additional Discussion about Related Work
This section reviews three typical safe reinforcement learning algorithms: CPO [Achiam et al. , 2017],
PCPO [Yang et al. , 2020] and FOCOPS [Zhang et al. , 2020].
Those algorithms also use new surrogatefunctions to replace the objective and constraints, which resembles the proposed CUP algorithm.
Thegoal is to present the contribution of our work.
B.1 CPO [Achiam
et
al. , 2017]
For a given policy π θ k , CPO updates new policy π θ k +1 as follows:
[Equation]
It is impractical to solve the problem (21) directly due to the computational cost.
[Achiam et al. ,2017] suggest to ﬁnd some convex approximations to replace the term A π θ k ( s, a ) and ¯ D KL ( π θ , π θ k )
Eq.(21)-(23).
Concretely, according to (5), Achiam et al.
[2017] suggest to use ﬁrst-order Taylor expansion of
J ( π θ ) to replace the objective (21) as follows,
[Equation]
Similarly, Achiam et al.
[2017] use the following approximations to turn the constrained policyoptimization (21)-(23) to be a convex problem,
[Equation]
where H is Hessian matrix of ¯ D KL ( π θ , π θ k ) , i.e.,
[Equation]
Eq.(25) is the second-oder approximation of (23).
Let λ (cid:63) , ν (cid:63) is the dual solution of the following problem
[Equation]
where g = ∇
[Equation]
E
[Equation]
g
[Equation]
Ha , s = a (cid:62) H − 1 a , and c =
J c ( π θ k ) − b .
Finally, CPO updates parameters according to conjugate gradient as follows: if approximation to
CPO is feasible:
[Equation]
else,
[Equation]
B
2 PCPO [Yang et al. , 2020]
Projection-Based Constrained Policy Optimization (PCPO) is an iterative method for optimizingpolicies in a two-step process: the ﬁrst step performs a local reward improvement update, while thesecond step reconciles any constraint violation by projecting the policy back onto the constraint set.
Reward Improvement:
[Equation] [Equation]
Projection:
[Equation]
Then, Yang et al.
[2020] follows CPO [Achiam et al. , 2017] uses convex approximation to originalproblem, and calculate the update rule as follows,
[Equation]
where L = I if D is (cid:96) 2 -norm, and L = H if D is KL-divergence.
B.3 FOCOPS [Zhang
et al. ,
Zhang et al.
[2020] propose the First Order Constrained Optimization in Policy Space (FOCOPS)that is a two-step approach.
We present it as follows.
Step1:
Finding the optimal update policy.
Firstly, for a given policy π θ k , we ﬁnd an optimalupdate policy π (cid:63) by solving the optimization problem (21)-(23) in the non-parameterized policyspace.
[Equation]
If π θ k is feasible, then the optimal policy for (26)-(28) takes the following form:
[Equation]
[Equation]
where Z λ,ν ( s ) is the partition function which ensures (29) is a valid probability distribution, λ and νare solutions to the optimization problem:
[Equation]
the term ˜ b =
[Equation]
− γ )
[Equation]
J
[Equation] k )) .
Step 2: Projection
Then, we project the policy found in the previous step back into the parameterizedpolicy space Π θ by solving for the closest policy π θ ∈ Π θ to π (cid:63) in order to obtain π θ k +1 :
[Equation]
T a b l e 2 : C
o m p a r
i s on o f s o m
e s a f e r
e
i n f o r ce m e n t a l go r it h m s .
A l go r it h m
O p ti m i za ti on p r ob l e m
I m p l e m e n t a ti on
R e m a r k
C P O
[ A c h i a m e t a l . , ] π θ k + 1 = a r g m a x π θ ∈ Π θ
E s ∼ d ρ 0 π θ k ( · ) , a ∼ π θ ( ·| s ) (cid:104)
A π θ k ( s , a ) (cid:105) , s .
t .
J c ( π θ k ) + E s ∼ d ρ 0 π θ k ( · ) , a ∼ π θ ( ·| s ) (cid:104)
A c π θ k
( s , a ) (cid:105)
≤ b , ¯ D K L ( π θ , π θ k ) =
E s ∼ d ρ 0 π θ k ( · ) [ K
L ( π θ , π θ k )
[ s ]] ≤ δ .
θ k
+= a r g m a x θ g
(cid:62) (
[Table]
− θ k ) , s .
t .
c + b (cid:62) (
[Table]
− θ k ) ≤ 0 , 1 2 ( θ − θ k ) (cid:62) H (
[Table]
− θ k ) ≤ δ .
C onv e x I m p l e m e n t a ti on
P C P
O [ Y a ng e t a l . ,020 ]
R e w a r d I m p r
ov e m e n t π θ k
+ = a r g m a x π θ ∈ Π θ
E s ∼ d ρ 0 π θ k ( · ) , a ∼ π θ ( ·| s ) (cid:104)
A π θ k ( s , a ) (cid:105) , s .
t .
¯ D K L ( π θ , π θ k ) =
E s ∼ d ρ 0 π θ k ( · ) [ K
L ( π θ , π θ k )
[ s ]] ≤ δ ;
P r o j ec ti on π θ k + 1 = a r g m i n π θ ∈ Π θ
D (cid:18) π θ , π θ k +(cid:19) , s .
t .
J c ( π θ k ) + 1 1 − γ E s ∼ d ρ 0 π θ k ( · ) , a ∼ π θ ( ·| s ) (cid:104)
A c π θ k
( s , a ) (cid:105) ≤ b .
R e w a r d I m p r
ov e m e n t θ k + 12 = a r g m a x θ g (cid:62) ( θ − θ k ) , s .
t .
1( θ − θ k ) (cid:62) H ( θ − θ k ) ≤ δ ;
P r o j ec ti on θ k
+= a r g m i n θ 1 2 ( θ − θ k ) (cid:62) L ( θ − θ k ) , s .
t .
c + b (cid:62) ( θ − θ k ) ≤ 0 .
C onv e x I m p l e m e n t a ti on
F O C
O PS [ Z h a ng e t a l . , ]
O p ti m a l up d a t e po li c y π (cid:63) =
a r g m a x π ∈ Π
E s ∼ d ρ 0 π θ k ( · ) , a ∼ π ( ·| s ) (cid:104)
A π θ k ( s , a ) (cid:105) , s .
t .
J c ( π θ k ) + E s ∼ d ρ 0 π θ k ( · ) , a ∼ π ( ·| s ) (cid:104)
A c π θ k
( s , a ) (cid:105)
≤ b , ¯ D K L ( π θ , π θ k ) =
E s ∼ d ρ 0 π θ k ( · ) [ K
L ( π , π θ k )
[ s ]] ≤ δ ;
P r o j ec ti on π θ k + 1 = a r g m i n π θ ∈ Π θ
E s ∼ d ρ 0 π θ k ( · ) [ K
L ( π θ , π (cid:63) )
[ s ]] .
O p ti m a l upd a t e p o li c y π (cid:63) ( a | s ) =
π θ k ( a | s )
Z λ , ν ( s ) e x p (cid:16) 1 λ (cid:16)
A π θ k ( s , a ) − ν
A c π θ k
( s , a ) (cid:17)(cid:17) ;
P r o j ec ti on θ k
+= a r g m i n θ
E s ∼ d ρ 0 π θ k ( · ) [ K
L ( π θ , π (cid:63) )
[ s ]] .
N onC onv e x I m p l e m e nt a ti on
C U P ( O u r
W o r k )
P o li c
y
I m p r
ov e m e n t π θ k
+ = a r g m a x π θ ∈
Π θ (cid:26)
E s ∼ d λ π θ k ( · ) , a ∼ π θ ( ·| s ) (cid:104)
A
G A E ( γ , λ ) π θ k ( s , a ) (cid:105) −
α k (cid:113) E s ∼ d λ π θ k ( · ) [
K L ( π θ k , π θ )
[ s ]] (cid:27) ,
P r o j ec ti on π θ k + 1 = a r g m i n π θ ∈ Π θ
D (cid:16) π θ , π θ k +(cid:17) , s .
t .
J c ( π θ k ) + 1 1 − ˜ γ
E s ∼ d λ π θ k ( · ) , a ∼ π θ ( ·| s ) (cid:104)
A G A
E ( γ , λ ) π θ k , C ( s , a ) (cid:105) + β k (cid:113) E s ∼ d λ π θ k ( · ) [
K L ( π θ k , π θ )
[ s ]] ≤ b .
P o li c
y
I m p r
ov e m e n t θ k + 12 = a r g m a x θ (cid:40) 1 T
T (cid:88) t =
π θ ( a t | s t ) π θ k ( a t
| s t ) ˆ A t − α (cid:118)(cid:117)(cid:117)(cid:116) 1 T
T (cid:88) t =
K L ( π θ k ( ·| s t ) , π θ ( ·| s t ))
(cid:41) ;
P r o j ec ti on θ k
+= a r g m i n θ 1 T
T (cid:88) t =(cid:26) K L (cid:18)
π θ k + 12 ( ·| s t ) , π θ ( ·| s t ) (cid:19)
+ ν k 1 − γ λ 1 − γ π θ
( a t | s t )
π θ k
( a t | s t ) ˆ A C t (cid:27) .
N onC onv e x I m p l e m e nt a ti on
C Constrained Update Projection Algorithm
C.1 Practical Implementation of Performance Improvement
C.1.1 Sample-based Performance Improvement
Let the trajectory { ( s t , a t , r t +1 , c t +1 ) }
Tt =1 be sampled according to π θ k , then we denote the empirical
KL-divergence with respect to π θ and π θ (cid:48) as follows,
[Equation]
We deﬁned the following ˆ L R ( π θ , π θ k ) ,
[Equation]
where ˆ A t is an estimator of A GAE ( γ,λ )
π θ k
( s, a ) .
The term ˆ L R ( π θ , π θ k ) (30) is an estimator of the nextexpectation that appears in (14)
[Equation]
Then we implement the performance improvement as follows,
[Equation] [Equation]
C.1.2 Clipped Surrogate Objective
How can the implementation (31) take the biggest possible improvement step on a policy using thedata we currently have, without stepping so far that we accidentally cause performance collapse?
Now, we present a clip implementation for policy improvement, which is very efﬁcient in practice.
Instead of the previous policy improvement (14), according to PPO Schulman et al.
[2017], we updatethe policy as follows,
[Equation]
where the the objective L clip is deﬁned as follows,
[Equation]
(cid:15) is a hyperparameter which roughly says how far away the policy π θ k +12 is allowed to go from thecurrent policy π θ k .
The objective L clip ( s, a, π θ , π θ k , (cid:15) ) is complex, we present the insights of thisclip mechanism Schulman et al.
[2017] to make CUP learn stably.
Positive GAE:
A
[Equation]
( s, a ) > 0 .
Firstly, we consider the positive advantage, which implies theobjective L clip ( s, a, π θ , π θ k , (cid:15) ) reduces to
[Equation]
Since A GAE ( γ,λ ) π θ k
( s, a ) > 0 , to improve the performance, we need to increase π θ .
The min {·} operatordetermines the quantization how much the CUP improves.
If the policy improves too much such that
[Equation]
The min {·} operator hit the objective with a ceiling of (1 + (cid:15) )
A GAE ( γ
[Equation]
λ ) π θ k
( s, a ) .
The clip techniquerequires CUP learns a policy π θ k +12 does not beneﬁt by going far away from the current policy π θ k .
Negative GAE: A GAE ( γ,
[Equation]
( s, a ) < 0 .
Let us consider the negative advantage, which implies theobjective L clip ( s, a, π θ , π θ k , (cid:15) ) reduces to
[Equation]
Since A GAE ( γ,λ ) π θ k
( s, a ) < 0 , to improve the performance, we need to decrease the policy π θ ( a | s ) .
Themax {·} operator determines the quantization how much the CUP improves.
If the policy decreasetoo much such that
[Equation]
The max {·} operator hit the objective with a ceiling of (1 − (cid:15) )
A
[Equation]
( s, a ) .
Thus, similar to thepositive GAE the clip technique requires CUP learns a policy π θ k +12 does not beneﬁt by going faraway from the current policy π θ k .
C.1.3 Learning from Sampling
To short the expression, we introduce a function g ( (cid:15), A ) as follows,g ( (cid:15), A )
= (cid:26) (1 + (cid:15) )
A A ≥
[Equation]
(1 + (cid:15) )
A A < 0 .
Then we rewrite the objective (32) as follows,
[Equation]
Recall the trajectory { ( s t , a t , r t +1 , c t +1 ) }
Tt =1 be sampled according to π θ k , we deﬁned the followingˆ L clip ( π θ , π θ k ) ,
[Equation]
where ˆ A t is an estimator of A GAE ( γ,λ )
π θ k
( s, a ) .
The term ˆ L clip ( π θ , π θ k , (cid:15) )
(36) is an estimator of thenext expectation that appears in (35).
Then we implement the performance improvement as follows,
[Equation]
i.e., we obtain the parameter θ k + 12 according to
[Equation]
where η 1 is step-size.
C.2 Practical Implementation of Projection
Recall (15), we introduce the new surrogate function with respected to cost function as follows,
[Equation]
where β is adaptive to the term
[Equation]
Now, we rewrite the projection step (15)as follows,
[Equation]
We update the projection step (15) by replacing the distance function D ( · , · ) by KL-divergence, andwe solve the constraint problem (15) by the primal-dual approach.
Theorem 4.
The constrained problem (37) is equivalent to the following primal-dual problem:
[Equation]
Proof.
This result is a direct application of [Boyd and Vandenberghe, 2004, Chapter 5.9], and wealso present it in D.1.
Firstly, we notice if D (cid:16) · , π θ k +12 (cid:17) is KL divergence or (cid:96) 2 -norm, then theconstrained problem (37) is a convex problem 1 .
In fact, for a given policy π θ k +12 , D (cid:16) · , π θ k +12 (cid:17) isconvex over the policy Π θ , and C π θ (cid:48) ( · , β ) is also convex over the policy Π θ .
Additionally, Slater’scondition alway holds since C π θ (cid:48)
( π θ (cid:48) , β ) = 0 .
According to Theorem 4, we turn the projection step (37) as the following unconstrained problem,
[Equation] [Equation]
In our implementation, we use KL-divergence as the distance D ( · , · ) to measure the differencebetween two policies, then
[Equation]
39) which implies we can rewrite the problem (38) as follows,
[Equation]
Furthermore
[Equation]
we update the projection step as follows,
[Equation]
where
[Equation]
ˆ J C and ˆ
A
Ct are estimators for cost-return and cost-advantage
[Equation]
Remark 3 (Track for Learning ν ) .
Particularly, after some simple algebra
[Equation]
we obtain the derivationof ˆ L c ( · ) with respect to ν as follows,
[Equation]
But recall (14) is a minimization-maximization iteration,
i
[Equation]
e., we require to minimize the distance 600 E
[Equation]
) KL ( π θ , π θ k ) [ s ] , which implies π θ is close to π θ k .
Thus it is reasonable to consider
[Equation]
Thus, in practice, we update ν following a simple way
[Equation]
Finally, we obtain the parameters ( θ k +1 , ν k +1 ) as follows,
[Equation]
where {·} + denotes the positive part, i.e., if x ≤ 0 , { x } + = 0 , else { x } + = x .
We have shown allthe details of the implementation in Algorithm 1.
Algorithm
Initialize: policy network parameters θ 0 ; value network parameter ω 0 ; cost value function parameter ν 0 , step-size ν 0 ;
Hyper-parameters: trajectory horizon T ; discount rate γ ; episode number M, N , mini-batch size
B , positive constant α, η ; for k = 0 , 1 , 2 , . . .
do
Collect batch data of M episodes of horizon T in ∪ Mi =1 ∪ Tt =0 {
( s i,t , a i,t , r i,t +1 , c
i,t +1 ) } according to current policy π θ k ;
Estimate c -return by discount averaging on each episode: ˆ J Ci = (cid:80) Tt =
γ t c
i,t +1 ;
Compute TD errors ∪ Mi =1 ∪ Tt =
{ δ i,t } , cost TD errors ∪ Mi =1 ∪ Tt =0 { δ Ci,t } : δ i,t
= r
i,t + γV ω
k
( s i,t ) − V ω
k
( s i,t − 1 ) , δ Ci,t
= c
i,t + γV C ν k ( s i,t ) − V C ν k
( s i,t − 1 );
Compute GAE: ∪ Mi =1 ∪ Tt =
{ ˆ A i,t , ˆ A Ci,t } : ˆ A i,t = (cid:80)
Tj = t ( γλ )
j − t δ i,j , ˆ A Ci,t = (cid:80)
Tj = t ( γλ )
j − t δ Ci,j ;
Compute target function for value function and cost value function as follows,
V target i,t
= ˆ
A i,t + V ω k ( s i,t ) , V target ,C i,t = ˆ A Ci,t + V C
ν
k
( s i,t );
Store data: D k = ∪ Mi =1 ∪ Tt =0 (cid:110) ( a i,t , s i,t , ˆ A
i,t , ˆ A Ci,t , V target i,t , V target ,C i,t ) (cid:111) ; π old ← π θ k ;
Policy Improvement
for i = 0 , 1 , 2 , . . .
, M do θ k +
12 = arg max θ (cid:40)
T
T (cid:88) t =
π θ ( a i,t | s i,t ) π old ( a i,t | s i,t )
ˆ A i,t − g (cid:32) (cid:15), 1 T
T (cid:88) t
= ˆ
A
i,t (cid:33)(cid:41) ; end for π old ← π θ k +12 ;
Projection ν k +
=
( ν k + η ( ˆ J Ci − b ))
+ ; for i = 0 , 1 , 2 , . .
, M do θ k +1 =
arg min θ
T
T (cid:88) t = (cid:26)
KL( π θ old ( ·| s i,t ) , π θ ( ·| s i,t ))
+ ν k 1 − γλ 1 − γ π θ
( a i,t | s i,t )
π θ k
( a i,t | s i,t ) ˆ A Ci,t (cid:27) ; end for for each mini-batch { ( a j , s j , ˆ A j , ˆ A Cj , V target j , V target ,C j ) } of size B from D k do ω
k +1 =
arg
min ω
B (cid:88)
j = (cid:0) V ω ( s j ) − V target j (cid:1) 2 ,
ν k
+1 =
arg min ν
B (cid:88)
j = (cid:16) V c
ν
( s j ) − V target ,C j (cid:17) 2 ; end for end for
D Preliminaries
In this section, we introduce some new notations and results about convex optimization, statedistribution, policy optimization and λ -returns.
D.1 Strong Duality via Slater’s Condition
We consider a convex optimization problem:
[Equation]
) where the functions f 0 , f 1 , · · · , f m are convex, and h 1 , · · · , h p are afﬁne.
We denote by D thedomain of the problem (which is the intersection of the domains of all the functions involved), and by
X ⊂ D its feasible set.
To the problem we associate the Lagrangian L : R n × R m × R p
→ R , with values
[Equation]
The dual function is g : R m × R p
→ R , with values
[Equation]
L ( x [Equation] λ, ν ) .
[Equation]
The associated dual problem is
[Equation]
Slater’s condition .
We say that the problem satisﬁes Slater’s condition if it is strictly feasible, that is:
[Equation]
Theorem 5 (Strong duality via Slater condition) .
If the primal problem (8.1) is convex, and satisﬁesthe weak Slater’s condition, then strong duality holds, that is, p (cid:63) =
d
[Equation]
We omit the proof of Theorem 5, for more discussions, please refer to [Boyd and Vandenberghe,2004, Chapter 5.9].
D.2 State Distribution
We use P π θ ∈ R |S|×|S| to denote the state transition matrix by executing π θ , and their componentsare:
[Equation]
which denotes one-step state transformation probability from s to s (cid:48) .
We use P π θ ( s t = s | s 0 ) to denote the probability of visiting s after t time steps from the initial states 0 by executing π θ .
Particularly, we notice if t = 0 , s t (cid:54) =
s 0 , then P π θ ( s t = s | s 0 )
= 0 , i.e.,
P π θ ( s t = s | s 0 )
= 0 , t = 0 and s (cid:54) =
s 0 .
Then for any initial state s 0 ∼ ρ ( · ) , the following holds,
[Equation]
Recall d s 0 π θ ( s ) denotes the normalized discounted distribution of the future state s encountered startingat s 0 by executing π θ ,
[Equation]
(cid:88)
[Equation]
= s | s
[Equation]
Furthermore, since s 0 ∼ ρ 0 ( · ) , we deﬁne
[Equation]
as the discounted state visitation distribution over the initial distribution ρ 0 ( · ) .
We use d ρ 0 π θ ∈ R |S| tostore all the normalized discounted state distributions, and its components are:
[Equation]
We use ρ 0 ∈ R |S| to denote initial state distribution vector, and their components are:
[Equation]
Then, we rewrite d ρ 0 π θ as the following matrix version,
[Equation]
D.3 Objective of MDP
Recall τ = { s t , a t , r t +1 } t ≥ 0 ∼ π θ , according to τ , we deﬁne the expected return J ( π θ | s 0 ) asfollows,
[Equation]
where R ( τ ) = (cid:80) t ≥
γ t r t +1 , and the notation J ( π θ | s 0 ) is “conditional” on s 0 is to emphasize thetrajectory
τ starting from s 0 .
Since s 0 ∼ ρ 0 ( · ) , we deﬁne the objective of MDP as follows,
[Equation]
The goal of reinforcement learning is to solve the following optimization problem:
[Equation]
[Equation]
D.4 λ -Return
Let B π θ be the Bellman operator :
[Equation]
where r π θ ∈ R |S| is the expected reward according to π θ , i.e., their components are:
[Equation]
Let v π θ ∈ R |S| be a vector that stores all the state value functions, and its components are:
[Equation]
Then, according to Bellman operator (57), we rewrite Bellman equation [Bellman, 1957] as thefollowing matrix version:
[Equation]
Furthermore, we deﬁne λ - Bellman operator B λπ θ as follows,
[Equation]
which implies
[Equation]
59) where
[Equation]
Let
[Equation]
where P t +1 π θ [ s, s (cid:48) ] is the ( s, s (cid:48) )
-th component of matrix P t +1 π θ , which is the probability of visiting s (cid:48)after t + 1 time steps from the state s by executing π θ , i.e.,
[Equation]
Thus, we rewrite P ( λ ) π θ ( s (cid:48) | s ) (61) as follows
[Equation]
Remark 4.
Furthermore, recall the following visitation sequence τ = { s t , a t , r t +1 } t ≥ 0 induced byπ θ , it is similar to the probability P π θ ( s t = s (cid:48) | s 0 ) , we introduce P ( λ ) π θ ( s t
= s (cid:48) | s 0 ) as the probabilityof transition from state s to state s (cid:48) after t time steps under the dynamic transformation matrix P ( λ ) π θ .
Then, the following equity holds
[Equation]
Similarly, let
[Equation]
It is similar to normalized discounted distribution d ρ 0 π θ ( s ) , we introduce λ -return version of discountedstate distribution d λπ θ ( s ) as follows: ∀ s ∈ S ,
[Equation]
[Equation] [Equation]
68) where P ( λ )
[Equation]
( s t = s | s 0 ) is the ( s 0 , s ) -th component of the matrix (cid:16)
[Equation]
[Equation]
Similarly, P ( λ ) π θ ( s t = s (cid:48) | s ) is the ( s, s (cid:48) )
-th component of the matrix (cid:16) P ( λ ) π θ (cid:17) t , i.e.,
[Equation]
Finally, we rewrite d ρ 0 ,λ π θ as the following matrix version,
[Equation]
Remark 5 ( λ -Return Version of Bellman Equation) .
According to Bellman equation (58), v π θ isﬁxed point of λ -operator B λπ θ , i.e.,
[Equation]
Recall τ = { s t , a t , r t +1 } t ≥ 0 ∼ π θ , according to (70), the value function of initial state s 0 is
[Equation]
We unroll the expression of (71) repeatedly, then we have
[Equation]
(cid:32)(cid:88)
[Equation]
According to (54) and (72), we have
[Equation]
Finally, we summarize above results in the following Lemma 1.
Lemma 1.
The objective J ( π θ ) (55) can be rewritten as the following version:
[Equation]
E Proof of Theorem 1
We need the following Proposition 4 to prove Theorem 1, which illustrates an identity for the objectivefunction of policy optimization.
Proposition 4.
For any function ϕ ( · ) : S → R , for any policy π θ , for any trajectory satisﬁesτ =
{ s t , a t , r t +1 } t ≥ 0 ∼ π θ , let
[Equation]
then, the objective J ( π θ ) (73) can be rewritten as the following version:
[Equation]
We present the proof of of Proposition 4 at the end of this section, see Section E.2.
We introduce a vector δ ϕπ θ ,t ∈ R |S| and its components are: for any s ∈ S
[Equation]
Then, we rewrite the objective as the following vector version
[Equation]
76) where (cid:104)· , ·(cid:105) denotes inner production between two vectors.
E.1 Proof of Theorem
Theorem 1 (Generalized Policy Performance Difference)
For any function ϕ ( · ) : S → R , for twoarbitrary policy π θ and π θ (cid:48) , for any p, q ∈ [1 , ∞ ) such that 1 p + 1 q = 1 , The following bound holds:
[Equation]
where the terms M ϕ, − p,q,t and M ϕ, + p,q,t are deﬁned in (93)-(94).
Proof.
(of Theorem 1)
We consider two arbitrary policies π θ and π θ (cid:48) with different parameters θ and θ (cid:48) , let
[Equation]
According to (76), we obtain performance difference as follows,
[Equation]
which requires us to consider the boundedness of the difference D ϕ,
( λ ) t
[Equation]
Step 1:
Bound the term D ϕ, ( λ ) t ( π θ , π θ (cid:48) ) (78).
We rewrite the ﬁrst term of (78) as follows,
[Equation] [Equation]
which is bounded by applying Hölder’s inequality to the term (cid:104)
d λπ θ
[Equation]
d λπ
[Equation] [Equation]
,t (cid:105)
[Equation]
we rewrite (80) asfollows,
[Equation]
where p, q ∈ [1 , ∞ ) and 1 p + 1 q = 1 . Let
[Equation]
then we rewrite Eq.(81) as follows,
[Equation]
Let
[Equation]
combining the (78) and (82)
[Equation]
we achieve the boundedness of D ϕt
( π θ , π θ (cid:48) ) as follows
[Equation]
Step 2: Analyze the term M ϕt ( π θ , π θ (cid:48) )
To analyze (84) further
[Equation]
we need to consider the ﬁrst term appears in M ϕt
( π θ , π θ (cid:48) )
[Equation]
We notice the following relationship
[Equation]
[Equation]
which holds since we use importance sampling: for any distribution p ( · ) and q ( · ) , for any randomvariable function f ( · ) ,
[Equation]
According to (85), (87)
[Equation]
we rewrite the term (cid:104)
d λπ θ (cid:48) , δ ϕπ θ ,t (cid:105) in Eq.(83) as follows,
[Equation]
Now, we consider the second term appears in M ϕt
[Equation]
[Equation]
Finally, take the results (88) and (89) to (83), we obtain the difference between (cid:104) d λπ θ (cid:48) , δ ϕπ θ ,t (cid:105) and(cid:104)
d λπ θ (cid:48) , δ ϕπ θ (cid:48) ,t (cid:105) , i.e., we achieve a identity for M ϕt
( π θ , π θ
[Equation]
) (83) as follows,
[Equation]
To simplify expression, we introduce a notation as follows,
[Equation]
) and we use a vector ∆ ϕt ( π θ , π θ (cid:48) )
∈ R |S| to store all the values { ∆ ϕt ( π θ , π
[Equation]
, s ) } s ∈S :
[Equation]
Then we rewrite (cid:104) d λπ θ
[Equation]
, δ ϕπ θ
[Equation]
t (cid:105) −
[Equation]
d λπ θ
[Equation]
, δ ϕπ θ
[Equation]
(90) as follows,
[Equation]
Step 3:
Bound on J ( π θ ) − J ( π θ (cid:48) ) .
Recall (84), taking above result in it, we obtain
[Equation]
Finally
[Equation]
let
[Equation]

[Equation]
and
[Equation] [Equation]

[Equation]
According to (79) and (92)
[Equation]
we achieve the boundedness of performance difference between twoarbitrary policies π θ and π θ (cid:48) :
[Equation]
E.2 Proof of Proposition
Proof.
(of Proposition 4).
Step 1: Rewrite the objective J ( π θ ) in Eq.(73).
We rewrite the discounted distribution d λπ θ (69) as follows,
[Equation]
Let ϕ ( · ) be a real number function deﬁned on the state space S , i.e., ϕ : S → R .
Then we deﬁne avector function φ ( · ) ∈ R |S| to collect all the values { ϕ ( s ) } s ∈S , and its components areφ [ s ] =
ϕ ( s ) , s ∈ S .
Now, we take the inner product between the vector φ and (96), we have
[Equation]
We express the ﬁrst term (cid:104) ρ 0 , φ (cid:105) of (97) as follows,
[Equation] [Equation]
We express the second term (cid:104)
d λπ θ , φ (cid:105) of (97) as follows,
[Equation] [Equation]
We express the third term (cid:104)
˜
γ P ( λ ) π θ d λπ θ
[Equation]
φ (cid:105) of (97) as follows,
[Equation]
According to Lemma 1, put the results (73) and (97) together, we have
[Equation]
where the last equation holds since we unfold (97) according to (98)-(100).
Step 2: Rewrite the term (cid:16)
[Equation]
Then, we unfold the second term of (101) as follows,
[Equation]
Recall the terms P ( λ ) π θ , r ( λ ) π θ [ s ] deﬁned in (60)-(65),
[Equation]
We consider the ﬁrst term R ( λ ) π θ ( s ) of (102) as follows,
[Equation]
We consider the second term ˜ γ (cid:80) s ∈S P ( λ ) π θ ( s (cid:48) | s ) ϕ ( s ) − ϕ ( s ) of (102) as follows,˜
[Equation]
where the equation from Eq.(108) to Eq.(109) holds since: according to (51), we use the followingidentity
[Equation]
Furthermore
[Equation]
take the result (105) and (109) to (104)
[Equation]
we have
[Equation]
the equation from Eq.(109) to Eq.(110) holds since:
[Equation]
; the equation from Eq.(110) to Eq.(111) holds since we use the Markov property of the deﬁnition of
MDP: for each time t ∈ N ,
[Equation]
the equation (112) the following identity:
[Equation]
then
[Equation]
Step 3: Put all the result together.
Finally, let
[Equation]
combining the results (101) and (113), we have
[Equation]
This concludes the proof of Proposition 4.
E.3 Proposition
All above bound results appear in (11) and (13) can be extended for a total variational divergence to
KL-divergence between policies, which are desirable for policy optimization.
We obtain
[Equation]
where KL ( · , · ) is KL-divergence, and
[Equation]
the ﬁrst inequality follows Pinsker’s inequality [Csiszár and Körner, 2011] and the second inequalityfollows Jensen’s inequality.
According to (115), we obtain the next Proposition 3.
Proposition 3 .
All the bounds in (11) and (13) hold if we make the following substitution:
[Equation]
F Lemma 2
In this section, we show Lemma 2 that presents an upper bound to the difference between twoλ -version of normalized discounted distribution.
Before we present our main results, we review thenorms induced by p -norms for matrix.
F.1 Norms Induced by p -norms for Matrix
If the p -norm for vectors (1 ≤ p ≤ ∞ ) is used for both spaces R n and R m , then the correspondingoperator norm is:
[Equation]
These induced norms are different from the "entry-wise" p -norms and the Schatten p -norms formatrices treated below, which are also usually denoted by (cid:107) A (cid:107) p .
In the special cases of p = 1 and p = ∞ , the induced matrix norms can be computed or estimated by
[Equation]
which is simply the maximum absolute column sum of the matrix;
[Equation]
which is simply the maximum absolute row sum of the matrix.
Thus, the following equation holds
[Equation]
F.2 Lemma
Lemma 2.
The divergence between discounted future state visitation distributions, (cid:107) d λπ θ (cid:48) − d λπ
[Equation]
1is bounded as follows,
[Equation]
where D TV ( π θ (cid:48) , π θ )
[ s ] is the total variational divergence between action distributions at state s , i.e.,
[Equation]
Proof.
(of Lemma 2).
Recall Eq.(69), we know,
[Equation]
To short the expression, we introduce some additional notations as follows.
[Equation]
Then, after some simple algebra, the following holds
[Equation]
Furthermore, by left-multiplying by G π θ and right-multiplying by G π θ (cid:48) , we achieve
[Equation]
Grouping all the results from (117)-(119), recall (69),
[Equation]
then we have
[Equation]
Applying (121)
[Equation]
we have
[Equation]
Firstly, we bound the term (cid:107) G π θ (cid:48) (cid:107) 1 as follows,
[Equation]
Thus, recall ˜ γ = γ (1 − λ )
[Equation]
, we obtain
[Equation]
where the last equation holds due to Lemma [Equation] this concludes the proof of Lemma 2 .
Lemma 3.
The term (cid:107) Dd λπ θ (cid:107) 1 is bounded as follows,
[Equation]
Proof.
Now
[Equation]
we analyze (cid:107) Dd λπ θ (cid:107) 1 as follows,
[Equation]
which implies that to bound (cid:107) Dd λπ θ (cid:107)
[Equation]
we need to bound the following difference
[Equation]
Step 1:
[Equation]
Let s
= s
[Equation]
then
[Equation]
Similarly,
[Equation] [Equation]
Firstly
[Equation]
we consider the following term
[Equation]
To short expression, we introduce a new notation as follows,
[Equation] [Equation]
Then, we rewrite (130) as follows,
[Equation]
Furthermore, according to (132), we obtain∞ (cid:88) t = ( γλ
[Equation]
s 1 ∈S
F π θ [ s 1 , s (cid:48) ] (cid:32)(cid:88) a ∈A π θ ( a | s ) P ( s 1 | s, a ) (cid:33) =
(cid:88) s (cid:48) ∈S (cid:88) s 1 ∈S
F π θ (cid:48)
[ s 1 , s (cid:48) ]
(cid:32)(cid:88) a ∈A π θ (cid:48)
( a | s ) P ( s 1 | s, a ) (cid:33) − (cid:88) s (cid:48) ∈S (cid:88) s 1 ∈S
F π θ (cid:48)
[ s 1 , s (cid:48) ]
(cid:32)(cid:88) a ∈A π θ ( a | s ) P ( s 1 | s, a ) (cid:33) + (cid:88) s (cid:48) ∈S (cid:88) s 1 ∈S
F π θ (cid:48)
[ s 1 , s (cid:48) ]
(cid:32)(cid:88) a ∈A π θ ( a | s ) P ( s 1 | s, a ) (cid:33) − (cid:88) s (cid:48) ∈S (cid:88) s 1 ∈S
F π θ [ s 1 , s (cid:48) ] (cid:32)(cid:88) a ∈A π θ ( a | s ) P ( s 1 | s, a ) (cid:33) =
(cid:88) s (cid:48) ∈S (cid:88) s 1 ∈S
F π θ (cid:48)
[ s 1 , s (cid:48) ]
(cid:32)(cid:88) a ∈A (cid:16) π θ (cid:48)
( a | s ) − π θ ( a | s ) (cid:17) P ( s 1 | s, a ) (cid:33) (
[Equation]
which implies that to bound the following difference
[Equation]
we need to bound (133) and (134).
Step 2:
Bound the difference (134).
Due to the simple fact: for any inverse matrices A and B
[Equation]
then the following identity holds
[Equation]
we rewrite the difference F π θ
[Equation]
− F π θ as follows,
[Equation]
Then
[Equation]
we rewrite (134) as the following matrix version
[Equation]
where p π θ ( s ) ∈ R |S| , and
[Equation]
According to (116), we obtain
[Equation]
where in Eq.(135)
[Equation]
we introduce a notation f (cid:62) s ∈ R |S| as follows,
[Equation]
Eq.(136) holds since:
[Equation]
Eq.(137) holds since:
[Equation]
=
[Equation]
(cid:88)
[Equation]
[Equation]
a
[Equation]
s )
| = (cid:88) s ∈S (cid:88)
a
[Equation]
a
[Equation]
) | , where the last equation holds due to the following fact
[Equation]
Thus, the difference (134) is bounded as follows,
[Equation]
Step 3:
Bound the difference (133).
We turn to consider (133):
[Equation]
where the last Eq.(138) holds since:
[Equation]
and
[Equation]
Step 4: Put all the result together.
Finally, according to (133), (134), (137)
[Equation]
and (138)
[Equation]
we obtain
[Equation]
This concludes the result of Lemma
[Equation]
G Proof of Theorem 2
Before we present the main result, we deﬁne some notations.
[Equation]
Proof.
(of Theorem 2)
According to Bregman divergence, if policy π θ k is feasible, policy π θ k +1 is generated according to(15), then the following
[Equation]
implies
[Equation]
According to the asymptotically symmetry of KL divergence if we update the policy within a localregion, then, we have
[Equation]
Furthermore, according to Proposition 1 and Proposition 3, we have
[Equation]
Similarly, according to Proposition 1 and Proposition
[Equation]
and since policy π θ k +1 satisﬁes J
[Equation]
141) and
[Equation] [Equation]
Combining (141)- (143), we have
[Equation] [Equation]
H Experiments
The Python code for our implementation of CUP is provided along with this submission in thesupplementary material.
All experiments were implemented in Pytorch 1.7.0 with CUDA 11.0 and conducted on an Ubuntu20.04.2 LTS (GNU/Linux 5.8.0-59-generic x86 64) with 40 CPU cores (Intel(R) Xeon(R) Silver4210R
CPU @
2.40GHz), 251G memory and 4 GPU cards (GeForce RTX 3080).
The baselinealgorithm FOCOPS based on the open-source https://github.com/ymzhang01/focops , whichwere ofﬁcal code library.
The other baseline algorithms include CPO, TRPO-L,
PPO-L based onhttps://github.com/openai/safety-starter-agents , which published by openai.
H.1 Algorithm Parameters
Hyperparameter
CUP
PPO-L
TRPO-L
CPO
FOCOPS
No. of hidden layers
No. of hidden nodes
Activation tanh tanh tanh tanh tanh
Initial log std -0.
Discount for reward γ 0.
Discount for cost γ C 0.
Batch size
Minibatch size
N/A N/A
No. of optimization epochs
N/A N/A
Maximum episode length
GAE parameter (reward) 0.
GAE parameter (cost) 0.
Learning
[Table]
[Table]
We consider two tasks from MuJoCo [Brockman et al. , 2016]: Walker2d-v3 and Hopper-v3, wherethe setting of cost follows [Zhang et al. , 2020].
For agents move on a two-dimensional plane, the costis calculated as follows,
[Equation]
for agents move along a straight line, the cost is calculated as
[Equation]
where v x , v y are the velocities of the agent in the x and y directions respectively.
H.2.2 Environment 2: Circle.
The Circle Environment follows [Achiam et al. , 2017], and we use open-source implementation ofthe circle environments from https://github.com/ymzhang01/mujoco-circle .
According to
Zhang et al.
[2020], those experiments were implemented in OpenAI Gym [Brockman et al. ,
2016]while the circle tasks in Achiam et al.
[2017] were implemented in rllab [Duan et al. , 2016].
We alsoexcluded the Point agent from the original experiments since it is not a valid agent in OpenAI Gym.
The ﬁrst two dimensions in the state space are the ( x, y ) coordinates of the center mass of the agent,hence the state space for both agents has two extra dimensions compared to the standard Ant-v0 and
Humanoid-v0 environments from OpenAI Gym.
Now, we present some necessary details of this environment taken from [Zhang et al. , 2020].
In the circle tasks, the goal is for an agent to move along the circumference of a circle while remainingwithin a safety region smaller than the radius of the circle.
The exact geometry of the task is shownin Figure 4.
The reward and cost functions are deﬁned as:
[Equation]
where x, y are the positions of the agent on the plane, v x , v y are the velocities of the agent alongthe x and y directions, r is the radius of the circle, and x lim speciﬁes the range of the safety region.
The radius is set to r = 10 for both Ant and Humanoid while x lim is set to 3 and 2.5 for Ant and
Humanoid respectively.
Note that these settings are identical to those of the circle task in Achiam etal.
[2017]; Zhang et al.
H.3 Safety Gym
In Safety Gym environments, the agent perceives the world through a robot’s sensors and interactswith the world through its actuators [Ray et al. , 2019].
In this section, we consider two robots: Pointand Car, where the presentation of those safety environments are taken from [Ray et al. , 2019], formore details, please refer to [Ray et al. , 2019, Page 8–10].
In this section, we experiment with the
Safety Gym environment-builder two tasks: Goal, Button.
H.3.1 Safety Gym Robots
We consider two robots: Point and Car.
All actions for all robots are continuous and linearly scaled to[ − 1 , +1] , which is typical for 3D robot-based RL environments and (anecdotally) improves learningwith neural nets.
Modulo scaling, the action parameterization is based on a mix of hand-tuning and
MuJoCo actuator defaults, and we caution that it is not clear if these choices are optimal.
Somesafe exploration techniques are action-layer interventions, like projecting to the closest predictedsafe action [ ? ], and these methods can be sensitive to action parameterization.
As a result, actionparameterization may merit more careful consideration than is usually given.
Future work on action
[Figure] [Figure]
space design might be to ﬁnd action parameterizations that respect physical measures we careabout—for example, an action space where a ﬁxed distance corresponds to a ﬁxed amount of energy.
Point : A robot constrained to the 2D plane, with one actuator for turning and another for movingforward/backward.
This factored control scheme makes the robot particularly easy to control fornavigation.
Point has a small square in front that makes it easier to visually determine the robot’sdirection and helps the point push a box element that appears in one of our tasks.
Car : The car is a slightly more complex robot that has two independently-driven parallel wheelsand a free-rolling rear wheel.
The car is not ﬁxed to the 2D plane but mostly resides in it.
For thisrobot, both are turning and moving forward/backward require coordinating both of the actuators.
It issimilar in design to simple robots used in education.
H.3. Tasks
[Figure]
Tasks in Safety Gym are mutually exclusive, and an individual environment can only use a singletask.
Reward functions are conﬁgurable, allowing rewards to be either sparse (rewards only obtainedon task completion) or dense (rewards have helpful, hand-crafted shaping terms).
Task details areshown as follows.
Goal : Move the robot to a series of goal positions.
When a goal is achieved, the goal location israndomly reset to someplace new, while keeping the rest of the layout the same.
The sparse rewardcomponent is attained on achieving a goal position (robot enters the goal circle).
The dense rewardcomponent gives a bonus for moving towards the goal (shown in Figure 5).
Button : Press a series of goal buttons.
Several immobile “buttons” are scattered throughout theenvironment, and the agent should navigate to and press (contact) the currently-highlighted button,which is the goal button.
After the agent presses the correct button, the environment will select andhighlight a new goal button, keeping everything else ﬁxed.
The sparse reward component is attained
[Table]
[Figure]
o s t
[Figure]
v e r a g e E p R
[Figure] t [Figure] on pressing the current goal button.
The dense reward component gives a bonus for moving towardsthe current goal button.
We show a visualization in Figure 6).
H.4 Discussions
Results of Figure 7 show that the performance of CUP is still very stable for different settings of (cid:15) .
Additionally, the constraint value of CUP also still ﬂuctuates around the target value.
The differentvalue achieved by CUP in different setting (cid:15) is affected by the simulated environment and constraintthresholds, which are easy to control
The results of Table 4 show that the proposed CUP signiﬁcantly outperforms all the baseline algorithms except on the Safexp-CarGoal1-v0 task.
Notably, on the Safexp-PointButton1-v0 task, CUPachieve 21 .
27 ± 1 .
42 within the safety region, while the best baseline algorithm is CPO that onlyobtains a reward of 17 .
69 ± 1 .
but it violates the cost limit 25 more than a value of 44.
This resultis consistent with the result of Figure 2.
Besides, from Table 4, we know although CPO achieves areward of 33 ± 00 signiﬁcantly outperforms the proposed CUP in Safexp-CarGoal1-v0, CPO needs acost
[Table] [Table]
CUP.
[Figure] [Figure] [Figure]