Effects of Safety State Augmentation on Safe Exploration
Abstract
Safe exploration is a challenging and important problem in model-free reinforcement learning (RL).
Often the safety cost is sparse and unknown, which unavoidablyleads to constraint violations — a phenomenon ideally to be avoided in safetycritical applications.
We tackle this problem by augmenting the state-space with asafety state, which is nonnegative if and only if the constraint is satisfied.
The valueof this state also serves as a distance toward constraint violation, while its initialvalue indicates the available safety budget.
This idea allows us to derive policiesfor scheduling the safety budget during training.
We call our approach Simmer(Safe policy IMproveMEnt for RL) to reflect the careful nature of these schedules.
We apply this idea to two safe RL problems: RL with constraints imposed on anaverage cost, and RL with constraints imposed on a cost with probability one.
Ourexperiments suggest that simmering a safe algorithm can improve safety duringtraining for both settings.
We further show that Simmer can stabilize training andimprove the performance of safe RL with average constraints.
Introduction
Reinforcement learning (RL) is a framework for sequential decision-making that makes minimalprior assumptions about the environment where the agent has to act or make the decisions [43].
The policy for taking the actions is learned through interactions with the environment over time.
RL has seen recent successes in playing video games with a computer [30], board games with ahuman [39] and is on a path toward real-life applications such as video compression [44] and plasmacontrol [18].
There are still, however, some unsolved challenges for a successful deployment of
RL such as efficient learning of constrained or safe Markov Decision Processes (MDPs) [4].
Theconstraints are typically modeled by a discounted sum of nonnegative costs that have to be smallerthan some pre-defined value called the safety budget .
Exploration is a crucial component of RL and is still an active area of research [23, 38, 32].
In thecontext of safe RL, while exploring we do not want to incur the safety cost and constraint violationsmaking exploration a harder task.
We can distinguish two main research directions for minimizingconstraint violation in safe RL: a model-based approach (which includes partially and fully known environments) and a model-free approach.
In the model-free case, which we focus on, the agent would almost certainly visit unsafe regions to learn the safe policy, and therefore it is next to impossible to avoidconstraint violations completely.
The general goal, in this case, is to minimize the number of violationsduring training.
For example, [9] use a conservative safety critic and rejection sampling to choosea “safer” action, [46] design a curriculum learning approach, where the teacher resets the studentviolating safety constraints, and [19] learns to reset the policy if the safety constraint is violated [19].
In this work, we aim to improve model-free safe reinforcement learning by augmenting the state-spacewith one state encapsulating the safety information.
This safety state is initialized with the safety budget.
If during the deployment a safety cost is incurred, we subtract this cost from the safety statethus tracking the remaining safety budget.
The safety constraint now is simply making sure that thesafety state remains nonnegative.
Therefore, the value of the safety state can serve as a measure ofdistance to the unsafe region.
This form of the safety state was proposed in [16] and introduced in
RL [41], which used of the safety state augmentation to derive a safe RL algorithm with probabilityone constraints.
In this paper, we take a closer look at the purposes of the safety state in a broadercontext.
First, we claim that safety state augmentation is often crucial for the averaged constrainedproblem as well and provide examples of such occurrences.
In some particular problems, the useof safety state augmentation may potentially be avoided, however, this can be said about any statein the environment.
Second, our main suggestion is that scheduling the safety budget at differenttraining epochs can improve the algorithm’s performance.
Our claim extends equally over averagedand almost surely constrained reinforcement learning problems.
In particular, we achieve the stateof the art performance on some environments in the safety gym benchmark.
Further, we claim thatscheduling the initial safety budget can lead to reducing safety constraint violations during trainingfor safe RL with probability one constraints.
We design two algorithms automatically tuning thesafety budget, one is based on a classical control engineering PI controller, while the other uses Qlearning to decide on the safety budget.
Related work.
Many of the current safe RL methods are extensions of the most successful RLalgorithms:
Trust region policy optimization (TRPO) [36], proximal policy optimization (PPO)
[37],soft actor-critic (SAC) [21] etc.
Safe versions of TRPO, PPO, and SAC with a Lagrangian approachwere first presented in [35], which is still considered to be one of the major baselines.
A directextension of TRPO by adding constraints to the trust region update was proposed in [2].
Model-basedapproaches were also considered cf.
[34, 14, 25, 28] and most of them took a Bayesian approachto model one-step transitions.
To our best knowledge, the most successful approaches for safereinforcement learning to date are PID-Lagrangian [42] and LAMBDA [5].
The former viewsthe Lagrangian multiplier update as another control problem and employs a PID controller to it(cf.
Specifically, the authors link the multiplier update to integral control and add proportionaland derivative controllers to achieve a superior behavior.
On the other hand, [5] is a model-basedapproach that uses Bayesian world models to enhance safety.
A recent work [27] formulatedsafe RL as inference resulting in a sample efficient off-policy approach.
Some approaches took aslightly different approach to formulate safety, e.g., [29] proposed a two-player framework withthe cooperating task agent and safety agents, [15] proposed a safety layer that would be applied afterthe action is computed using a classical policy, [22] define a probabilistic shield for safety.
We alsonote [50, 24] that consider safe deployment of RL policies in real-life settings.
Other formulations of safe reinforcement learning were considered in the literature.
For example,[12, 14, 49] proposed to use conditional-value-at risk (CVaR) constraints, while [41, 10] proposedto enforce constraints with probability one.
Further, as we discussed above eliminating the numberof constraint violations typically requires strong assumptions, e.g., finite state space [45], [48] [40],knowledge of a partial model [26] or initial safe policy [7].
These results are in the spirit of safe RLwith control-theoretic notions [13, 8, 31, 11, 3, 17, 20], which make significant prior assumptionsto guarantee safety.
Finally, we note that the closest algorithm in the literature to our method is thecurriculum learning approach to Safe RL from [46].
The major difference with our work is its onlinenature, while [46] pre-train teacher’s policies deciding the curriculum, we do not pre-train our safetybudget schedules.
Our approach is preferable when a new task needs to be learned with minimalprior information about it, while the method from [46] is preferable when the policy of choosing theconstraints can be transferred from another task.
We further discuss [46] in Appendix.
Simmer: Safe policy improvement for reinforcement learning2.1 MDP with safety state augmentation
We consider a constrained reinforcement learning setting defined for a Markov Decision Process(MDP)
M = ⟨S , A , P , r, γ r ⟩ with transition probability P : S ×
A × S → [0 , 1] acting on state Sand A spaces, with the reward r : S ×
A × S → R + ( R + stands for the nonnegative orthant [0 , + ∞ ) )and the discount factor γ r ∈ (0 , 1] .
The MDP is endowed with the following optimization problem
[Equation]
, subject to: g
[Equation]
with the time horizon T > 0 , the safety discount factor γ l ∈ (0 , 1] , the safety cost l : S × A × S → 89 R m + , and the inequality should be understood as the element-wise comparison.
The statisticg ( · ) : R m + → R m + is a design choice and there exits many different possibilities (e.g., CVaR, chanceconstraints etc [12]).
In this paper, we consider two most relevant options, from our point of view: a) aconstraint with probability one, i.e., g po ( z ) =
P ( z ≥ 0) − 1 , b) a constraint on average g av ( z ) =
E z .
Similarly to [41], we augment the safety state information into the state-space by introducing thestate z t = γ − t ( d − (cid:80) t − 1 k =
γ kl l ( s k , a k , s k +1 )) , which has the following update z t +1 =
( z t −l ( s t , a t , s t +1 ))
/γ l , with z 0 = d .
Noting that the update is Markovian this state can be easilyaugmented into the MDP.
The variable z t has the interpretation of the remaining safety budget, and bydefinition enforcing the constraint on z T is equivalent to enforcing the constraint on the accumulatedcost.
Now we can rewrite the safe RL problem as follows:
[Equation]
While [41] considered only the case g po ( z ) , we argue that the case of g av ( z ) deserves additionalattention in the context of safety state augmentation.
For completeness, we review the main points ofthe approach [41] in Appendix.
Note that the policy now also depends on the safety state z and thisfeature deserves a more thorough discussion.
2.2 Do we need the safety state?
[Figure]
As a simple demonstration consider the cartoon in Figure 1.
A robot needs to reach the goal whilecrossing the hazards region, which is marked by the red circle, and the safety cost is acquired forevery time unit spent in the region.
Both green and blue paths are safe, i.e., satisfy the constraint.
However, at the crossing of these paths robot needs to know which path it took to this state.
Switchingfrom the green path to the blue one will lead to a constraint violation.
Standard safe RL algorithmswill have trouble with such scenarios while adding the safety state solves the issue.
We confirmthis observation in our experiments.
Let us now discuss how this logic can be mathematically formalized.
This problem is well studiedin the optimal control literature in the context of problems with known dynamics and with terminalor end-point constraints.
In the deterministic case, this problem reduces to the standard maximumprinciple with end-point constraints [47].
Further, [41] showed that having the policy depending onthe remaining safety budget is crucial for safe reinforcement learning with probability one constraints.
The problem was also studied in the context of stochastic diffusions [33], where the authors derivedthe representation of the optimal value function.
In both cases, it was shown that the optimal policydepends on the whole state, which in our case includes the safety state z .
While in some cases onemay not need to augment the safety state, in many situations it is critical.
2.3 Simmering Safe Reinforcement Learning
In this paper, we propose to use the initial safety budget d as another tuning dial for the algorithms.
Specifically, we will exploit the link between the safety budget d with the initial state of the safety state.
We argue that adjustment of d during training from some initial value d start to the target valued target can lead to improved exploration in terms of safety and performance.
Let us now present the mathematical formulation.
At every epoch k we pick a test safety budget d k ,collect the data set D k , compute the returns E D k (cid:80) T − 1 t =
γ tr r ( s t , a t ) and the costs ˆ g D k ( z T ) , wherethe function ˆ g D k ( · ) is the empirical mean or the maximum for the averaged and the probability oneconstrained problems, respectively.
Using this information we aim at solving the following problemat every epoch
but we only apply a pre-determined number of gradient steps):
[Equation]
a
[Equation]
subject to: ˆ
g D k ( z T
[Equation] [Equation]
z
[Equation]
=
d k
[Equation]
3) where s 0 ∈ S 0 and z
=
d k are the initial states of the Saute MDP.
Note that for off-policyalgorithms the data set D k can potentially grow with epochs, while for on-policy algorithms thedata set D k will be emptied on every epoch.
The key steps of our approach are as follows:
• Let there be a finite number of test safety budgets,
i
e., we assume that d k can take the
following values { d 0 , ref , . .
, d K − 1 , ref } with d start = d 0 , ref and d target =
d K − 1 , ref ;
• We assume that the value d start = d 0 , ref is such that the task can be solved;
• At for every k we perform only one epoch of optimization solving Problem 3.
Now we can find d k to achieve our goals and as we will show in the next section the problem of safetyduring training can be formalized as a two-level decision-making problem.
However, we will showthat even a naïve approach of scheduling d k can lead to improvement in performance and safety.
2.4 Application: Simmering for safety during training
Exploration can often lead to constraint violation during training due to the inherent stochasticity ofexploration.
While there is a significant effort in research for safe exploration it typically requiressignificant prior assumptions on MDPs.
We propose to embrace the philosophy of classical reinforcement learning and proceed with minimal assumptions.
We propose to choose the test safety budgetd k using another decision-making problem:
[Equation]
where z kT ( d target ) is the violation of the target constraint obtained as a result of solving one epoch of
Problem 3, the action set is [ − δ d , δ d ] and δ d is pre-determined, clip ( x, y, z ) clips the value x to alower bound y and an upper bound z , i.e., the function returns y if x ≤ y , z if z ≤ x and x otherwise.
The intuition behind our formulation is quite straightforward.
Assume we start with a very strictsafety budget d start training with which does not lead to significant violations with respect to thetarget safety budget d target .
Then by gradually increasing the safety budget d from d start to d target ,we can reduce the number of constraint violations during training.
We see this formulation as the firststep toward eliminating safety violations during training.
We formalized our problem as reinforcement learning over a partially observable process with thenon-stationary observations of the learning process.
This two-level RL problem allows us to usea broad spectrum of tools available in sequential decision-making differing our approach from theexisting in the literature.
However, solving Problem 4 appears to be a daunting task, especially inthe online setting as we intend.
Hence, we will employ heuristic solutions, which may prove moreeffective than the quest to find an optimal solution.
Intuitively, a gradual increase in d k by assigningscheduling increasing reference values (denoted by d ref k ) would less likely lead to constraint violationdue to stricter exploration constraints.
However, fixing the schedule a priori limits the ability ofthe algorithm to react to constraint violations.
To alleviate this issue we propose two approaches: PI
Simmer (PI-controlled safety budget) and Q Simmer (Online Q-learning with non-stationary rewards).
Algorithm 1:
Inputs: { d ref k } N − 1 k =0 - safety budget schedule, hyper-parameters - τ , K p , K i , K aw , δ d .
Set d 0 = d ref0 ; for k = 0 , . . .
, N − 1 do
Perform one epoch of learning for the safe policy with d k and get the statistic ˆ g
( z T ( d k )) ;
Compute and smooth the error term by e k =
d ref k − ˆ g ( z T ( d k )) , w k
=
(1 − τ ) w k − 1 + τ e k ;
Compute action: u raw k
= K p w k
(cid:124) (cid:123)(cid:122) (cid:125) P-Part
+ K i k (cid:88)
i = k − T
i w i (cid:124) (cid:123)(cid:122) (cid:125)
I-Part
+ K
aw ( u k − 1 − u raw k − 1 ) (cid:124) (cid:123)(cid:122) (cid:125)
Anti-windup ;
Set d k +1 =
clip ( u raw k , − δ d , δ d ) + d k ; end
In both cases the intuition is similar.
If the current accumulated costs are well below the safety budgetd ref k , then we are very safe and the safety budget can be further increased.
If the accumulated costsare around the safety budget, then we could stay at the same level or increase the safety budget.
Ifthe current accumulated costs are well above the safety budget d ref k , then the safety budget should bedecreased.
This intuition is based on our previous empirical observations, i.e., lower safety budgetswould often cause lower initial bursts in accumulated safety costs.
We hypothesize that the policy canquickly learn “extremely unsafe” actions thus providing low safety cost bursts for low safety budgets.
PI Simmer.
The idea for the PI controller is quite intuitive it takes the error term e = d ref − c anduses it for action computation.
P stands for the proportional control and links the error terms withactions by multiplying the error term by the gain K .
The proportional part delivers a brute forcecontrol by having a large control magnitude for large errors, but it is not effective if the instantaneouserror values become small.
Proportional control cannot achieve zero error e tracking.
To deliver zeroerror, integral control is typically used, which sums up previous error terms and uses this sum todetermine actions instead of the error.
PI controller can solve many hard control problems, but thereare some implementation and engineering tricks and improvements.
For example, if the action valuesare clipped, integral control can lead to the unwanted phenomenon called integral windup [6].
This isspecifically the case when the errors become large leading to large values of the integral, which inturn leads to saturated actions for a long time limiting the ability to react to the environment.
We willtake the feedback control approach to anti-windup , where the previous saturation errors are fed backand used to determine the current action [6, 1].
Finally, we use a low pass filter for the error terms toavoid reacting to high-frequency fluctuations and acting only on the trends.
A reader more familiarwith optimization literature will recognize this as a Polyak update.
Finally, we propose Algorithm
We provide rules-of-thumb for the choice of hyper-parameters in our ablation studies in Appendix.
Q Simmer.
Consider an MDP with the states { d ref0 , . . .
, d ref K − 1 } , which for simplicity of notationwe denote { 0 , . . .
, K − 1 } , with the actions { a − 1 , a 0 , a +1 } , where the action a +1 moves the states
=
i to the state s = i + 1 , a − 1 moves the state s =
i to the state s =
i − 1 and the action a 0 doesnot transfer the state.
Note that the action a +1 is defined for all i < K − 1 and the action a − 1 isdefined for all i > 0 .
Our design for the rewards of this MDP is guided by the following intuition.
We are not safe
We are borderline safe
We are very safe
[Equation]
5) where o
= ˆ g ( z T ( d k )) is safety violation statistic (over an epoch) of the safe RL algorithm, δ is athreshold for a significant violation.
We use Q-learning update to learn the Q function:
[Equation] [Equation]
where l r is the learning rate and get the action with ε -greedy exploration strategy:
a
[Equation]
= (cid:26) argmax b Q ( s t , b ) with probability ε, random with probability 1 − ε.
We summarize the approach as Algorithm 2.
Algorithm 2: Q-SIMMER
Input: { d ref k } K − 1 k =0 - an a priori chosen state space; hyper-parameters - N , τ , l r , δ , ε ;
Initialize d 0 = d ref for k = 0 , . .
. , N − 1 do
Perform one epoch of learning for the safe policy with d k and get the statistic ˆ g
( z T ( d k )) ;
Update the Q function as in Equation 6 while computing the reward r as in Equation 5;
Compute an action a k as in Equation 7 and compute d k +1 accordingly; end
Experiments
3.1 Baselines, environments and code base
Environments: We use the safe pendulum environment defined in [14], we also use the custom-madesafety gym environment with deterministic constraints, which we call static point goal [49].
In thisenvironment with 46 states and two actions, a large hazard circle is placed before the goal and forcesthe agent to go around it to reach the goal similarly to our cartoon in Figure 1.
We provide additionaldetails in Appendix.
The rest of our tests are performed on the safety gym benchmarks [35].
Baselines:
Lagrangian PPO
L
PPO
, Lagrangian TRPO (L-TRPO) and CPO.
Standard baselines from [35].
PID-Lagrangian
A
recent algorithm
stabilizing
L
PPO
learning
LAMBDA.
A recent model-based technique showing great performance on Safety Gym [5].
Probability one PPO (PO
PPO
PPO-based algorithm with probability one constraints from [41].
Our algorithms:
PI-Simmer - Scheduling safety budget using
PI controller for PO
PPO
Q-Simmer - Scheduling safety budget using Q learning for PO-PPO;
L-PPO w SA, PID-L w SA
- L-PPO (PID-L) solving Problem 2 with safety state augmentation.
Simmer L-PPO - L-PPO w SA and safety budget scheduling.
Code base: Our code is based on two repositories: safety starter agents [35], and PID Lagrangian [42].
We have implemented the safety state augmentation following the description in [41].
We use defaultparameters for both code bases unless stated otherwise.
Computational resources: We performed all computations on a PC equipped with 512GB of RAM,two Intel Xeon E5 CPUs, and four 16GB NVIDIA Tesla V100 GPUs.
3.2 Improving Safety During Training for Pendulum Swing-Up
Improving safety during training is more suited for almost surely safe RL and we will take PO PPO asour baseline.
In this setting, we can aim to reduce the number of individual trajectories that violate theconstraints, and thus we can avoid estimating the statistic g .
For PI Simmer we chose the followinghyper-parameters K = 0 .
01 , K i = 0 .
005 , K
aw = 0 .
01 and τ = 0 .
We also note that exceptfor an initial burst of violations both our approaches manage to keep the number of violations quitelow.
Overall we found that low values for K and K i are beneficial to avoid overreaction to constraintviolations.
In this case, keeping the value of K aw is advisable as the action saturation does not occurtoo often.
Finally, keeping τ close to one will force the controller to react to most of the constraintviolations.
For Q Simmer we chose δ = 1 , τ = 0 .
995 , l r = 0 .
05 , and ε = 0 .
It appears that a fastlearning rate here can allow for learning, sufficiently fast forgetting of past rewards, but also toavoid catastrophic forgetting.
As we consider a finite state MDP we can avoid using sophisticated
[Figure]
schedule
[Figure]
techniques for online learning and use the simplest one — tuning learning rate.
We chose δ and τ toavoid frequent state transitions.
We compare our algorithm to PO PPO in terms of the number of trajectories with constraint violations,and returns, and compare the progression of the schedule d test k .
Naturally, some individual trajectoriesstill violate the constraint, but the number can be significantly reduced using Simmer RL as Figure 2suggests.
While PI Simmer outperforms Q Simmer in these runs, it is worth mentioning that PI
Simmer uses more prior information than Q Simmer.
Indeed, while composing a schedule is not hard,we still have to identify the switch points, which are learned by Q Simmer.
We perform ablationstudies on the parameters and discuss their choice in more detail in Appendix.
3.3 Guiding Exploration by Scheduling Safety Constraints
We now turn our attention to safe RL with constraints imposed on average costs.
We test theperformance of Simmer L-PPO and L-PPO w SA on the swing-up pendulum environment and presenttraining results in Figure 3.
Here we use safety starter agents as a base learner for all algorithms.
[Figure]
The major observations are that L-PPO w SA delivers almost no constraint violations with respect tothe mean cost estimate, and therefore using Q Simmer and PI Simmer becomes redundant.
In themeantime, L-PPO has even trouble converging.
Note that we have used the same hyper-parametersfor all algorithms , which are default parameters in safety starter agents and the learning rate 0 .
While the behavior of the L-PPO algorithm can certainly be improved with tuning, we note thatsimply augmenting the safety state leads to improved performance as well as stability of the algorithm.
Further, we observe that Simmer L-PPO leads to a fewer number of violations, however, the rate ofviolations for the safety budget 35 is fairly similar to L-PPO w SA.
Interestingly, a similar picture occurs with more advanced baselines such as PID Lagrangian [42]and more complicated environments.
Here, we used the static point goal environment designedin [49], where a large static hazard region is placed in front of the goal.
This is similar to ourmotivational example in Figure 1.
The results for PID-L and PID-L with state augmentation aredepicted in Figures 4a and 4b, respectively, suggest that the presence of the safety state stabilizes
[Figure]
training and leads to a more consistent constraint satisfaction for different safety budgets d .
Notethat hyper-parameters for all the runs are the same for both algorithms.
We further apply the naïvesimmer approach to both baselines with results depicted in Figure 4c.
In both cases, the safety budgettakes values 1 , 5 , 10 , 15 , and 20 , and increased after equal time intervals.
Note that in both casesnow training curves are quite stable, although state augmentation delivers an extra boost.
In all ourexperiments we used the same hyper-parameters for all versions of PID-L, i.e., K = 0 .
1 , K i = 0 .
01 ,γ l = 0 . 99 .
Overall, our experiments suggest that safety during training with constraints imposed on averagecosts becomes a much easier problem with safety state augmentation.
Indeed, even intuitively everyoutlier trajectory with a large constraint violation should bias the average cost and should instructthe algorithm not to follow this path.
We note that in both cases above we have sparse costs, i.e.,the agent encounters unsafe regions and incurs costs while navigating toward the goal.
While analgorithm without the safety state will receive information on constraint violation after an episode,the safety state would constantly inform the algorithm of the distance toward a violation.
This isone of the reasons why safety state augmentation can learn the task better.
Note that simmeringadditionally offers fewer constraint violations while training.
These results suggest that simmeringsafe RL together with state augmentation delivers overall safer solutions with stable training curves.
3.4 Tests on safety gym benchmarks
We finally test our approach on more challenging environments from the safety gym benchmark:
PointPush1, PointGoal1 and PointButton1 in Figures 5a, 5b, 5c respectively, and CarPush1, CarGoal
CarButton1 in Figures 6a, 6b,6c, respectively.
Again we use a naïve schedule increasing the safetybudget from 10 to 25 in 5 unit increments.
We observe that Simmer PID-L and PID-L outperformother baselines significantly in terms of the return while delivering safe policies.
Simmer PID-L doesperform very similarly to PID-L in terms of returns but generally outperforms PID-L in terms ofsafety costs and cost rates.
We note that our results for PointGoal1 are consistent with the resultsfrom [42].
Conclusion
We augment the safety information into the state space and show how we can effectively use it toimprove safe exploration.
The safety state is nonnegative if and only if the constraint is satisfied andtherefore it can serve as a distance toward constraint violations.
We argue that the optimal policy
[Figure]
Simmer PID-L outperforms all baselines in terms of the cost rate, which is a measure of safety during training.
At the same time, Simmer PID-L is competitive with PID-L in terms of the returns or safety costs, while significantly outperforming other baselines.
The curves are means and shaded areas are standard deviations computed over five runs with random seeds.
must depend on this information to achieve safe performance.
We validate this argument usingintuitive examples, references to theoretical results, and experiments.
We do not doubt that safetystate augmentation is needed for effective safe RL.
Safety state augmentation and simmering show superior performance on pendulum swing-up andstatic point goal tasks for average constrained problems.
We further tuned PID Lagrangian toshow very strong performance on safety gym benchmarks, which has not been previously reported.
Simmer PID Lagrangian shows competitive performance in terms of returns to PID Lagrangian andoutperforms all baselines (including PID Lagrangian) in terms of the cost rate and the costs.
Further,using state augmentation for the average constrained problems appears to be quite beneficial as well.
Scheduling the safety budget can stabilize safe algorithms without state augmentation, but using bothstate augmentation and simmering improves performance and safety.
We achieve this performanceand safety boost at the expense of sample efficiency since we effectively learn a series of safe policies.
Simmering RL algorithms with probability one constraints can significantly reduce safety violationsduring training.
We illustrate this feature by developing two algorithms for safe learning.
The firstone is based on a PI controller and allows for adjustment of a pre-defined learning schedule dependingon the estimated training cost.
The second one is based on online Q learning and learns to adjust thesafety budget automatically.
Both approaches have some advantages and limitations.
PI simmering ismore effective if there is a reasonable safety budget schedule, while Q simmering requires less priorknowledge to learn.
Our algorithmic development focused on probability one constraints for two reasons.
First, sinceprobability one constraints have to be imposed on all trajectories we can simply count the numberof violations and do need to resort to empirical estimates of the mean cost (as we need in theaverage constrained problem).
Second, it appears that in the average constrained case with “simple”
[Figure] [Figure]
[Figure]
Simmer PID
[Figure]
L
[Figure]
L
[Figure]
L
[Figure]
L
[Figure]
CPO
[Figure]
Simmer PID-L
Safety
[Figure]
PID
[Figure]
L
[Figure]
Simmer PID-L outperforms all baselines in terms of the cost rate, which is a measure of safety during training.
At the same time, Simmer PID-L is competitive with PID-L in terms of the returns or safety costs, while significantly outperforming other baselines.
The curves are means and shaded areas are standard deviations computed over five runs with random seeds.
Note that we have baseline results only for the CarGoal1 environment.
environments (such as pendulum swing-up and static point goal), we can track the constraint quiteefficiently after the initial (and unavoidable) burst in constraint violations.
Thus our experimentssuggest that PI Simmer and Q Simmer are simply redundant for the average constrained RL.
Augmenting the safety state and scheduling the safety budget do not solve all the problems insafe exploration.
First, an initial burst of constraint violations is an inevitable reality of using ourmodel-free approach.
It appears that lowering the initial safety budget lowers the initial burst ofconstraint violations, but does not completely solve the problem.
We do not see how to address thislimitation without making further assumptions on the environment.
Second, it is noticeable that theperformance on the PointPush1 environment is quite noisy for both PID-L and Simmer PID-L. Thisis because some seeds achieve very good performance (return of 15) and some seeds do not (return of5), while the learning curves appear to be stable.
This suggests that the learning procedure is at alocal maximum.
In our point of view, this calls for a more sophisticated algorithm for explorationseeking maximal return.
This may help achieve stable performance in this environment.
Beyondthese limitations, one can also list safety violations caused by variance and disturbances, lack ofprovable safety guarantees, etc.
Studying these limitations is outside of the scope of this paper.
We have tested PI-Simmer and Q-Simmer on a rather simple in terms of the state and action spaces environment.
We foresee that in a more complex setting the main problemwould be learning the agent’s behavior, but our algorithmic development would remain valid.
Nevertheless, it would be interesting to extend this approach to a more complex setting.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 2.•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] We reported the error bars over all the runs across
different seeds
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We use a PC equipped with 512GBof RAM, two Intel Xeon E5 CPUs, and four 16GB NVIDIA Tesla V100 GPUs4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use the code fromsafety starter agents, safety gym by open AI [35], PID Lagrangian method [42].
(b) Did you mention the license of the assets?
[No] All the libraries are under MIT Licence.
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]