Towards Safe Reinforcement Learning with a Safety Editor Policy
[Figure]
AffiliationAddress email
UM
SE
Env
Abstract
We consider the safe reinforcement learning (RL) problem of maximizing utilitywith extremely low constraint violation rates.
Assuming no prior knowledge orpre-training of the environment safety model given a task, an agent has to learn,via exploration, which states and actions are safe.
A popular approach in thisline of research is to combine a model-free RL algorithm with the Lagrangianmethod to adjust the weight of the constraint reward relative to the utility rewarddynamically.
It relies on a single policy to handle the conflict between utility andconstraint rewards, which is often challenging.
We present SEditor, a two-policyapproach that learns a safety editor policy transforming potentially unsafe actionsproposed by a utility maximizer policy into safe ones.
The safety editor is trainedto maximize the constraint reward while minimizing a hinge loss of the utilitystate-action values before and after an action is edited.
SEditor extends existingsafety layer designs that assume simplified safety models, to general safe RLscenarios where the safety model can in theory be arbitrarily complex.
As a firstorder method, it is easy to implement and efficient for both inference and training.
On 12 Safety Gym tasks and 2 safe racing tasks, SEditor obtains much a higheroverall safety-weighted-utility (SWU) score than the baselines, and demonstratesoutstanding utility performance with constraint violation rates as low as once per 2ktime steps, even in obstacle-dense environments.
On some tasks, this low violationrate is up to 200 times lower than that of an unconstrained RL method with similar utility performance.
Code will be made public.
Introduction
Safety has been one of the major roadblocks in the way of deploying reinforcement learning (RL) tothe real world.
Although RL for strategy and video games has achieved great successes (Silver et al.
,2016; Vinyals et al., 2019; Berner et al., 2019), the cost of executing actions that lead to catastrophicfailures in these cases is low: at most losing a game.
On the other hand, RL for control has alsomostly been studied in virtual simulators (Brockman et al., 2016; Tassa et al., 2020).
Aside fromthe data collection consideration, to circumvent the safety issue ( e.g. , damages to real robots andenvironments) is another important cause of using these simulators.
When RL is sometimes appliedto real robots (Kim et al., 2004; Levine et al., 2016; OpenAI et al., 2019), data collection and trainingare configured in restrictive settings to ensure safety.
Thus to promote the deployment of RL tomore real-world scenarios, safety is a critical topic.
In this paper, we study a safe RL problem ofmaximizing utility with extremely low constraint violation rates.
There are two general settings of safe RL.
Some existing works make the assumption that theenvironment safety model is a known prior.
The agent is able to query an oracle function to see if anystate is safe or not, without actually visiting that state.
This includes having access to a well calibrateddynamics model (Berkenkamp et al., 2017;
Chow et al., 2018; Thomas et al., 2021) or the set ofsafe/unsafe states (Turchetta et al., 2020; Luo and Ma, 2021;
Li et al., 2021).
Or they assume that apre-training stage of safe/unsafe states or policies is performed on offline data, and then the learnedsafety knowledge is transferred to main tasks (Dalal et al., 2018;
Miret et al., 2020; Thananjeyan et al.,2021).
With these assumptions, they are able to achieve few or even zero constraint violations duringonline exploration.
However, such assumptions also put restrictions on their applicable scenarios.
This paper focuses on another setting where no prior knowledge or pre-training of safety models isassumed.
The agent only gets feedback on which states are unsafe from its exploration experience.
In other words, it has to learn the safety model (implicitly or explicitly) from scratch and theconstraint budget can only be satisfied asymptotically .
Although we cannot completely avoidconstraint violations during online exploration, this setting can be especially helpful for sim-to-realtransfer (Zhao et al., 2020), or when the training process can be protected from damages.
Many priorworks (Ray et al., 2019; Tessler et al., 2019; Bohez et al., 2019; Stooke et al., 2020; Zhang et al.
,2020; Qin et al., 2021) reply on combining a typical model-free RL algorithm with the Lagrangianmethod (Bertsekas, 1999).
This primal-dual optimization dynamically adjusts the weight of theconstraint reward relative to the utility reward, depending on how well the violation rate target isbeing met.
A single policy is then trained from a weighted combination of the utility and constraintrewards.
However, reconciling utility maximization with constraint violation minimization usuallyposes great challenges to this single policy.
In this paper, we present SEditor , a general safe RL approach that decomposes policy learning acrosstwo polices (Figure 1).
The utility maximizer (UM) policy is only responsible for maximizing theutility reward without considering the constraints.
Its output actions are potentially unsafe.
Thesafety editor (SE) policy then transforms these actions into safe ones.
It is trained to maximize theconstraint reward while minimizing a hinge loss of the utility state-action values before and after anaction is edited.
Both UM and SE are trained in an off-policy manner for good sample efficiency.
Our two-policy paradigm is largely inspired by existing safety layer designs (Dalal et al., 2018;
Pham et al., 2018; Cheng et al., 2019; Li et al., 2021) which simplify ( e.g. , to linear or quadratic)environment safety models.
The high-level idea is the same though: modifying a utility-maximizingaction only when necessary.
We evaluate SEditor on 12 Safety Gym (Ray et al., 2019) tasks and 2 safe car racing tasks adaptedfrom Brockman et al.
(2016), targeting at very low violation rates.
SEditor obtains a much higheroverall safety-weighted-utility (SWU) score (defined in Section 4) than four baselines.
It demonstratesoutstanding utility performance with constraint violation rates as low as once per 2k time steps, evenin obstacle-dense environments.
Our results reveal that the two-policy cooperation is critical, whilesimply doubling the size of a single policy network will not lead to comparable results.
The choicesof the action distance function and editing function are also important in certain circumstances.
Insummary, our contributions are:
i)
We extend the existing safety layer works to more general safe RL scenarios where the environ-
ment safety model can in theory be arbitrarily complex.
ii) We present SEditor, a first-order, easy-to-implement approach that is trained by SGD like mostmodel-free RL methods.
It is efficient during both inference and training, as it does not solve amulti-step inner-level optimization problem when transforming an unsafe action into a safe one.iii)
When measuring the distance between an action and its edited version, we show that in somecases the hinge loss of their state-action values is better than the usual L2 distance (Dalal et al.
,2018; Pham et al., 2018; Li et al., 2021) in the action space.
We further show that an additiveaction editing function introduces an effective inductive bias for the safety editor.
iv) Finally, we achieve outstanding utility performance even with an extremely low constraintviolation rate ( 5 × 10 − 4 ) in dense-obstacle environments.
On some tasks, this low violation rate
is up to 200 times lower than that of a unconstrained RL method with similar utility performance
Preliminaries
The safe RL problem can be defined as policy search in a constrained Markov decision process(CMDP)
( S , A , p, r, c ) .
The state space S and action space
A are both assumed to be continuous.
The environment transition function p ( s t +1 | s t , a t ) determines the probability density of reachings t +1 ∈ S after taking action a t
∈ A at state s t ∈ S .
The initial state distribution µ ( s 0 ) deter- 90 mines the probability density of an episode starting at state s 0 .
Both p ( s t + | s t , a t ) and µ ( s 0 ) are 91 usually unknown to the agent.
For every transition ( s t , a t , s t +1 ) , the environment outputs a scalarr ( s t , a t , s t +1 ) which we call the utility reward.
Sometimes one uses the expected reward of taking a tat s t as r ( s t ,
a t ) ≜
E s t +1 ∼ p ( ·| s t ,a t ) r
( s t , a t , s t +1 ) for a simpler notation.
Similarly, the environmentalso outputs a scalar c ( s t , a t ) as the cost.
To unify the reward and cost notations, we define theconstraint reward r c ( s t , a t ) ≜ − c ( s t , a t ) ≤ 0 , and the CMDP becomes ( S , A , p, r, r c ) .
For both 96 utility and constraint rewards, the value is the higher the better .
Finally, we denote the agent’s policyas π ( a t | s t ) which dictates the probability density of taking a t at s t .
For each s t , the utility state value of following π is denoted by V π ( s t ) =
E π,p (cid:80) ∞
t ′ =
t γ t ′ − t r ( s t ′ , a t ′ ) ,and the utility state-action value is denoted by Q π ( s t , a t ) =
r ( s t , a t ) + γ E s t +1 ∼ p
V π ( s t +1 ) , whereγ ∈ [0 , 1) is the discount for future rewards.
Similarly, we can define V πc and Q πc for the constraintreward.
Then we consider the safe RL objective:
[Equation]
where C ≥ 0 is the constraint violation budget.
It might be unintuitive to specify C for a discountedreturn, so one can rewrite
[Equation]
and specify the per-step budget c instead, relating to C by (cid:80) ∞ t =
γ t c = c 1 − γ = C .
Note that c is notstrictly imposed on every step.
Instead, it is only in the average sense (by a discount factor), treatedas the violation rate target .
The Lagrangian method (Bertsekas, 1999) converts the constrained optimization problem Eq.
1 intoan unconstrained one by introducing a multiplier λ :
[Equation]
Intuitively, it dynamically adjusts the weight λ according to how well the constraint state valuesatisfies the budget, by evaluating (approximately) the difference
[Equation]
which is the gradient of λ given π in Eq.
When optimizing π given λ , Eq. 3 becomes
[Equation]
Thus λ can be seen as the weight of the constraint reward when it is combined with the utility rewardto convert multi-objective RL into single-objective RL.
This objective can be solved by typicalmodel-free RL algorithms.
For practical implementations, previous works (Ray et al., 2019; Tessleret al., 2019; Bohez et al., 2019) usually perform gradient ascent on the parameters of π and gradientdescent on λ simultaneously, potentially with different learning rates.
Approach
We consider a pair of cooperative policies.
The first policy utility maximizer (UM) denoted by π ϕ ,optimizes the utility reward by proposing a preliminary action ˆ a ∼ π ϕ ( ·| s ) which is potentially unsafe.
The second policy safety editor (SE) denoted by π ψ , edits the preliminary action by ∆ a ∼ π ψ ( ·| s, ˆ a ) 121 and the result action a = h
(ˆ a, ∆ a ) is output to the environment for a safe interaction, where hrepresents an editing function without learnable parameters.
Note that we allow UM to makepredictions relying only on the state s , while SE has to consider UM’s output.
Together the twopolicies cooperate to maximize the agent’s utility while maintaining a safe condition (Figure 1).
Forsimplicity, we will denote the overall composed policy by π ψ ◦ ϕ ( a | s ) .
The motivation of learning the UM is to generate a preliminary action ˆ a from a (much simplified)single utility objective, without having to resolve the conflict between utility and constraint.
Thisaction serves as a starting point to be tweaked by SE only when safety has not been guaranteed.
SE isexpected to search a safer alternative (via ∆ a ) of ˆ a , hopefully without comprising the utility reward.
In scenarios where a utility maximizing action is often safe already, this will lead to fast learningof SE (because ∆ a ≈ 0 ).
Thus our architecture decomposes a difficult policy learning task intotwo stages, each stage representing a smaller and easier subtask.
Below we show how our trainingobjectives for UM and SE are designed to align with this motivation.
We employ an off-policy actor-critic setting for training the two policies.
Given an overall policy π ψ ◦ ϕ ,we can use typical TD backup to learn Q π ψ ◦ ϕ
( s, a ) and Q π ψ ◦ ϕ c ( s, a ) parameterized as Q ( s, a ; θ ) and
Q c ( s, a ; θ ) respectively, where we use θ to collectively represent the network parameters of the twostate-action values.
Given s t +1 ∼ p ( ·| s t , a t ) and a t +1 ∼ π ψ ◦ ϕ ( ·| s t +1 ) , the Bellman backup operator 138 (point estimate) for the utility state-action value is
[Equation]
with the backup operator for the constraint state-action value defined similarly with r c .
Both Q ( s, a ; θ )and Q c ( s, a, ; θ ) can be learned on transitions ( s t , a t , s t +1 ) sampled from a replay buffer.
For off-policy training of ϕ and ψ , we first transform Eq.
3 into a bi-level optimization surrogate as:
[Equation]
D denotes a replay buffer and Λ π ψ ◦ ϕ is defined by Eq.
We basically have a historical marginal statedistribution for training the policies, but still use the initial state distribution µ for training λ .
Themotivation for this difference is, when tuning λ , we should always care about how well the policysatisfies our constraint budget starting with µ but not with some historical state distribution.
We continue transforming the off-policy objective (Eq. 6, a) into two,
[Equation]
where d ( a, ˆ a ) is a distance function measuring the change from ˆ a to a .
It is not necessary proportionalto ∆ a because the editing function h could be nonlinear.
The role of SE π ψ is to maximizethe constraint reward while minimizing some distance between the actions before and after themodification.
The role of UM π ϕ is to only maximize the utility reward.
However, it only can onlydo so through the lens of SE π ψ .
In other words, SE has actually changed the action space semantics(also MDP) and reshaped the optimization landscape of the utility Q for UM.
Action editing function.
We choose the editing function h to be mainly additive.
Without loss ofgenerality, we assume a bounded action space [ − A, A ] M , and that both ˆ a and ∆ a are already inthis space.
Then we define a = h (ˆ a, ∆ a )
=
min(max(ˆ a + 2∆ a, − A ) , A ) , where min and maxare element-wise.
It follows that a ∈ [ − A, A ] M .
This additive editing function ensures an easieroptimization landscape of SE π ψ because when ˆ a is often safe already, ∆ a is close to 0 (Appendix D).
Distance function.
Prior works on safety layer design, such as Dalal et al.
; Pham et al.
Li et al.
(2021), set the distance function d ( · , · ) as the L2 distance.
Later we will show that L2 is notalways the best option.
Instead, we use the hinge loss of the utility state-action values of ˆ a and a :
[Equation]
This loss is zero if the edited action a already obtains a higher utility state-action value than thepreliminary action ˆ a .
In this case, only the constraint Q is optimized by π ψ .
Otherwise, the inner partof Eq.
7 (b) is recovered as Q ( s, a ; θ ) + λQ c ( s, a ; θ ) , as we can drop the term − Q ( s, ˆ a ; θ ) due to itsgradient w.r.t.
ψ being zero.
Our distance function in the utility Q space is more appropriate than the
L2 distance in the action space, because eventually we care about how the utility changes after theaction is edited.
The L2 distance between the two actions is only an approximation to the change,from the perspective of the Taylor series of Q ( s, a ; θ
Evaluating Λ π ψ ◦ ϕ .
Given a batch of rollout experiences { ( s n , a n ) } N n =1 following π ψ ◦ ϕ , we approxi- 169 mate the gradient of λ (Eq. 4) as
[Equation]
where c is the violation rate target defined in Eq.
Namely, after every rollout, we collect a batch ofconstraint rewards, compare each of them to − c , and use the mean of differences to adjust λ .
Thisapproximation allows us to update λ using mini-batches of data instead of having to wait for wholeepisodes to finish, or having to rely on estimated V π ψ ◦ ϕ c which is usually not accurate.
Training.
To practically train the objectives, we apply SGD to Eq.
6 (b) and Eq.
7 simultaneously,resulting in a first-order method for approximated bi-level optimization (Likhosherstov et al., 2021).
SGD is made possible by applying the re-parameterization trick (Appendix E) to both ˆ a ∼ π ϕ and∆ a ∼ π ψ .
A computational graph of Eq.
7 is illustrated in Figure 2.
We would like to highlightthat a big difference between our SE and the previous safety layers (Dalal et al., 2018; Phamet al., 2018) is that SE directly uses a feedforward prediction ∆ a ∼ π ψ (Eq. 7, a) to replace theoptimization result of the objective of transforming unsafe actions into safe ones.
In contrast, previoussafety layers construct closed-form solutions or inner-level differentiable optimization steps ( e.g. ,quadratic programming (Amos and Kolter, 2017)) for simplified safety models at every inference step.
Our overall approach generalizes to various constraint rewards (safety models) and action distancefunctions.
Finally, to encourage exploration, we add the entropy terms of π ϕ and π ψ into Eq.
7, withtheir weights dynamically adjusted according to two entropy targets following Haarnoja et al.
Experiments
Baselines.
We compare SEditor with four baselines.
- PPO-Lag combines PPO (Schulman et al., 2017) with the Lagrangian method, as done in Ray et al.(2019).
Since PPO is an on-policy algorithm, we expect its sample efficiency to be much lower
than off-policy algorithms.
- FOCOPS (Zhang et al., 2020) is analogous to PPO-Lag with two differences: 1) there is no clippingof the importance ratio, and 2) a KL divergence regularization term with a fixed weight is added tothe policy improvement loss, with an early stopping when this term averaged over a rollout batchdata violates the trust region constraint.
- SAC-actor2x-Lag combines SAC (Haarnoja et al., 2018) with the Lagrangian method.
Similar to
SEditor, SAC-actor2x-Lag trains its policy on states sampled from the replay buffer but trains λ on states generated by the rollout policy .
Its gradient of λ is also estimated by Eq
The policynetwork size is doubled.
This is to match the model capacity of having two actors in SEditor.- SAC serves as an unconstrained optimization baseline to calibrate the utility return.
We choose not to compare with second-order CMDP approaches such as CPO (Achiam et al.,
2017)or PCPO (Yang et al., 2020) because they require (approximately) computing the inverse of the Fisher [Table] [Table] [Table]
[Table]
information matrix, which is prohibitive when the parameter space is large.
Especially for our CNNbased policies, second-order methods are impractical.
To analyze the key components of SEditor
[Table]
we also evaluate two variants of it for ablation studies.- SEditor-L2 defines d ( a, ˆ a ) ≜
∥ a − ˆ a ∥ 2 as done in most prior works of safety layer.- SEditor-overwrite makes SE directly overwrite UM’s action proposal by a = h (ˆ a, ∆ a ) =
∆ a .
All compared approaches including the variants of SEditor, share a common training configuration( e.g. , replay buffer size, mini-batch size, learning rate, etc) as much as possible.
Specific changes aremade to accommodate to particular algorithm properties (Appendix F).
Evaluation metric.
Following Ray et al.
(2019), one method dominates another if “it strictly improveson either return or cost rate and does at least as well on the other”.
Accordingly, we emphasize thatthe utility reward or constraint violation rate should never be compared in isolation, as one couldeasily find a method that optimizes either metric very well.
Particularly in the experiments, we setthe learning rate of the Lagrangian multiplier λ much larger than that of the remaining parameters.
This ensures that any continued constraint violation will lead to a quick increase of λ and drive theviolation rate back to the target level.
This conservative strategy makes some compared methodshave similar constraint violation curves, but vastly different utility performance.
For a quantitativecomparison of their final performance, we compute the safety-weighted-utility (SWU) scores of thecompared methods towards the end of training.
The score is an soft indicator of dominance, and it isa product of two ratios:
[Equation]
The reason for having min { 1 , ·} is because we only care when the violation rate is higher than the 222 target.
In practice, UtilityScore can be the episodic utility return (assuming positive) or the successrate.
We choose UtilityScore SAC as UtilityScore UnconstrainedRL .
In our experiments, both UtilityScoreand ConstraintViolationRate are averaged over the last 110 of the training steps to reduce variances.
Safety Gym tasks.
In Safety Gym (Ray et al., 2019) environments, a robot with lidar sensorsnavigates through cluttered environments to achieve goals.
We use the P OINT and C AR robots in ourexperiments.
Either of them has three tasks: G OAL , B UTTON , and P USH .
Each task has two levels,where level 2 has more obstacles and a larger map size than level 1.
In total we have 3 × 2 × 2 = 12tasks.
Whenever an obstacle is in contact with the robot, a constraint reward of − 1 is given.
Thusthe constraint violation rate can be calculated as the negative average constraint reward.
The utilityreward is calculated as the decrement of the distance between the robot (G OAL and B UTTON ) orbox (P USH ) and the goal at every step.
We define a success as finishing a task within a time limitof 1000 .
We emphasize that the agent has no prior knowledge of which states are unsafe.
The maplayout is randomized at the beginning of each episode.
This randomization poses a great challenge tothe agent because it has to generalize the safety knowledge to new scenarios.
We refer the reader to
Appendix A.1 for more details of the environment and tasks.
For our experiments, we set the constraint violation target c = 5 × 10 − 4 , meaning that the agentis allowed to violate any constraint only once every 2k steps on average.
This threshold is only150 of the threshold c = 0 .
025 used by the original Safety Gym experiments (Ray et al.
, 2019),highlighting the difficulty of our task.
Each compared approach is trained with 3 random seeds.
Thetraining curves are in Figure 3 and the SWU scores are in Table 1.
We see that SEditor obtainsmuch higher SWU scores than the baselines on 7 of the 12 tasks, while being comparable on the rest.
While SAC-actor2x-Lag performs well among the baselines, its double-size policy network does notlead to results comparable to SEditor.
This suggests that SEditor does not simply rely on the largecombined capacity of two policy networks to improve the performance, instead, its framework in
Figure 1 matters.
Both PPO-Lag and FOCOPS missed all the constraint violation rate targets. The
[Figure] unconstrained baseline SAC obtains better success rates than SEditor with the C AR robot, but violatesconstraints much more.
Surprisingly, SAC is worse than SEditor with the P OINT robot regardingsuccess rates, even without constraints.
Finally, we highlight that towards the end of training on C AR B UTTON 1/2, C AR P USH 2, and C AR G OAL 2, SEditor violates constraints up to 200 times lessthan SAC does, while achieving success rates on par with SAC (black curves vs. dashed blue curves)!
To analyze the exploration behaviors of SAC, SAC-actor2x-Lag, PPO-Lag, and SEditor, we visualizetheir rollout trajectories on an example map of P
OINT G OAL 2, at different training stages (thetraining is done on randomized maps but here we only evaluate on one map).
For each approach,the checkpoints at 10%, 20%, 30%, and 50% of the training process are evaluated and 100 rollouttrajectories are generated for each of them.
The trajectories are then drawn on a bird’s-eye sketchmap (Figure 4).
We can see that although SAC is able to find the goal location in the very beginning,it tends to explore more widely regardless of the obstacles, and its final paths ignore obstacles.
[Figure]
SAC-actor2x-Lag learns to respect the constraints after 10% of training, but the obstacles greatlyhinder its exploration: the trajectories are confined in a small region.
As a result, it takes some timefor it to find the goal location.
PPO-Lag is slower at learning the constraints and it even hits obstaclesat 50%.
SEditor is able to quickly explore regions between obstacles and refine the navigation pathspassing them instead of being blocked.
Our ablation studies (Appendix B Figure 8) show that, in this particular safe RL scenario, SEditoris not sensitive to the choice of distance function: SEditor-L2 achieves similar results with SEditor.
However, the editing function does make a difference: SEditor-overwrite is clearly worse than SEditorin terms of utility performance.
This shows that the inductive bias of the edited action a being closeto the preliminary action ˆ a is very effective (see Appendix D).
Our two-stage “propose-and-edit”strategy is more efficient than outputting an action in one shot.
Safe racing tasks.
Our next two tasks, S AFE R ACING and S AFE R ACING O BSTACLE , are adaptedfrom the unconstrained car racing task in Brockman et al.
The goal of either task is to finish aracetrack as fast as possible.
The total reward of finishing a track is 1000 , and it is evenly distributedon the track tiles.
In S AFE R ACING , the car has to stay on the track and receives a constraint rewardof − 1 whenever driving outside of it.
In S AFE R ACING O BSTACLE , the car receives a constraintreward of − 1 if it hits an obstacle on the track, but there is no penalty for being off-track.
An episodefinishes after every track tile is visited by the car, or after 1000 time steps.
The track (length andshape) and obstacles (positions and shapes) are randomly generated for each episode.
As in Safety
Gym, this randomization could expose a never-experienced safety scenario to the agent at any time.
The agent’s observations include a bird’s-eye view image and a car status vector.
Note that thishigh-dimensional input space usually poses great challenges to second-order safe RL methods.
Again,the constraint violation rate can be computed as the negative average constraint reward.
We set theconstraint violation rate target c = 5 × 10 − 4 .
For evaluation, we use undiscounted episode returnas the UtilityScore.
Each compared approach is trained with 9 random seeds.
The training curvesare shown in Figure 5 and SWU scores in Table 1.
SEditor has much higher SWU scores than allbaselines.
Moreover, it is the only one that satisfies the harsh violation rate target towards the end oftraining.
Surprisingly, even with constraints SEditor gets a much better utility return than SAC on
S AFE R ACING .
One reason is that without the out-of-track penalty as racing guidance, the car easilygets lost on the map and collects lots of meaningless timesteps.
Although SAC gets a much higherreturn on S AFE R ACING O BSTACLE , it greatly violates the constraint budget.
Interestingly, the ablation studies show results that are complementary to those on the Safety Gym tasks (Appendix B Figure 9).
Now the action distance function makes a big difference.
Changing it to the L2 distance greatlyimpacts the utility return, especially on S AFE R ACING O BSTACLE where almost no improvement ismade.
This demonstrates that the closeness of two actions does not necessarily reflect the closenessof their state-action values (Appendix C).
Related Work
Safe RL is closely related to multi-objective RL (Roijers et al., 2013), where the agent optimizesa scalarization of multiple rewards given a preference (Van Moffaert et al., 2013), or finds a set ofpolicies covering the Pareto front if no preference is provided (Moffaert and Nowé, 2014).
In ourcase, the preference for the constraint reward is always changing, because we try to maintain it to acertain level instead of maximizing it.
Trust-region methods for solving CMDPs, for example CPO (Achiam et al., 2017), θ -projection (Chowet al., 2019), and PCPO (Yang et al., 2020), make the assumption that both utility and constraintobjectives can be linearized given a small optimization step.
To enforce a small step, a constraint onthe KL divergence between the new and old policies is imposed, where the KL divergence is secondorder approximated.
For solving the surrogate objective at each iteration, the inverse of the Fisherinformation matrix is approximated.
This category of methods often involves high computationalcosts and is only practical for small policy networks but impractical for CNN policies.
In contrast, SEditor is a simple efficient first-order method.
The line of works closest to ours is safety layer (Dalal et al., 2018; Pham et al., 2018; Cheng et al.
,2019; Li et al., 2021).
Although they also transform unsafe actions into safe ones, their safety layersassume simple safety models which can be solved in closed forms or by inner-level optimization( e.g. , quadratic programming (Amos and Kolter, 2017)).
We extend this idea to more general safe RLscenarios without assuming a particular safety model.
Our SE can be also treated as a teacher underthe teacher-student framework, where UM is the student whose actions are corrected by SE.
However,existing works (Turchetta et al., 2020; Langlois and Everitt, 2021) under this framework more or lessrely on various heuristics and task-specific rules, or have access to the simulator’s internal state.
Incomplex realistic environments, such assumptions are invalidated.
Recovery RL (Thananjeyan et al., 2021) also decomposes policy learning across two policies.
Itmakes a hard switch between a task policy and a recovery policy by comparing a pretrained constraintcritic with a manual threshold.
It requires an offline dataset of demonstrations to learn the constraintcritic and fix it during online exploration.
This offline training requirement and the hard switchscheme reduce the flexibility of Recovery RL.
In contrast, SEditor learns the constraint critic online,and adopts a “propose-and-edit” strategy instead of switching between two policies.
Limitations and Conclusions
While SEditor is able to maintain an extremely low constraint violation rate in expectation, oscillationdoes happen occasionally on the training curves around the target threshold (Figure 3).
In general, ifno oracle safety model can be accessed by the agent, avoiding this oscillation issue would be verydifficult.
On one hand, our tasks always generate new random environment layouts for new episodes,and in theory the agent has to generalize its safety knowledge ( e.g. , one obstacle being dangerous atthis location usually implies it being also dangerous even when the background changes).
On theother hand, neural networks have the notorious issue of catastrophic forgetting, meaning that whilethe agent learns some new safety knowledge, it might forget that already learned.
Finally, from apractical point of view, as the violation target reaches 0, it takes an exponential number of samplesto evaluate if the agent truly can meet the target.
These challenging issues are currently ignored by
SEditor and would be interesting future directions.
In summary, we have introduced SEditor, for general safe RL scenarios, that makes a safety editorpolicy transform preliminary actions output by a utility maximizer policy into safe actions.
Inproblems where a utility maximizing action is often safe already, this architecture leads to fasterpolicy learning.
We train this two-policy framework with simple primal-dual optimization, resultingin an efficient first-order approach.
On 14 safe RL tasks with very harsh constraint violation rates,
SEditor achieves an outstanding overall SWU score.
We hope that SEditor can serve as a preliminarystep towards safe RL with harsh constraint budgets.