Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention
Abstract: Safety has become one of the main challenges of applying deep reinforcement learning to real world systems.
Currently, the incorporation of external knowledge such as human oversight is the only means to prevent the agent from visiting the catastrophic state.
In this paper, we propose MBHI, a novel framework for safe model-based reinforcement learning, which ensures safety in the state-level and can effectively avoid both ”local” and ”non-local” catastrophes.
An ensemble of supervised learners are trained in MBHI to imitate human blocking decisions.
Similar to human decision-making process, MBHI will roll out an imagined trajectory in the dynamics model before executing actions to the environment, and estimate its safety.
When the imagination encounters a catastrophe,
MBHI will block the current action and use an efﬁcient MPC method to output a safety policy.
We evaluate our method on several safety tasks, and the results show that MBHI achieved better performance in terms of sample efﬁciency and number of catastrophes compared to the baselines.
Introduction
Deep reinforcement learning (DRL) is proposed as an automated framework for intelligent decision-making problems, and has shown tremendous progress in various domains such as video games [1, 2], board games [3, 4] and robotic control [5, 6].
However, most of these success were achieved in the idealized simulator where agents can interact with the environment without restriction.
In real-world applications, random exploration may lead to equipment damage or even worse results [7].
Consequently, when applied in the real world, DRL algorithms need to ensure the security throughout the whole learning process, especially in safety-critical scenarios.
Safe Control 𝜋𝜋 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠
Imaginary
Rollout
Learned
Policy �𝜋𝜋
This has encouraged the research on safety in DRL, which aims to maximize the cumulative rewards while minimizing safety violations during the training processes [8, 9].
Safe RL can be divided into two categories: trajectory-based [9, 10, 11] and state-based [12, 13, 14].
The former uses the cost function to evaluate the safety of the whole trajectory, while state-based methods place restrictions on a state-wise basis.
Safe RL algorithms such as CPO [10] and HIRL [12] have shown that it is possible to signiﬁcantly reduce the number of constraint violations in safety-critical tasks.
Unfortunately, most of existing methods have high sample complexity, which means that a large number of interactions with the real world are required, resulting in the increase of the number of catastrophe.
Another shortcoming is that most of existing approaches use the explicitly deﬁned constraint or safety cost function, while safety is subjective and always vague.
For example, in autonomous driving, the overseer brakes when he judges that the car is going to crash.
The overseer makes judgement based on subjective feelings rather than vehicle dynamic model.
Finally, it is more strict and natural to deﬁne safety over the state.
However, most of existing state-based methods are hard to effectively avoid potential disasters in the future [12, 13, 15], or need suboptimal demonstrations [14, 16].
[Figure]
To address the above-described challenges, in this paper, we introduce a novel safety-aware modelbased framework, called Safe Model-based Reinforcement Learning with Human Intervention (MBHI), which incorporates safe RL with modelbased methods and human knowledge.
Intuitively, before excuting an action, people will estimate potential dangers in the future.
In MBHI, an ensemble of deep neural networks is used to model a supervised Blocker to imitate the human overseer or explicit constraints, output the catastrophe probability of the state and evaluate the model uncertainty.
As shown in Figure 1, at each time step, we roll out a trajectory with a certain number of steps in the learned dynamics to check the safety of the agent in the future.
If dangerous, the MPC controller will take over and output a safe action to avoid possible future catastrophes in advance.
Furthermore, safe active learning is integrated to enable grounded exploration only in safe regions, thereby accelerating model training on the premise of safety.
To summarize, the primary contributions of our work are as follows: (1) a novel safe model-based
RL framework with human intervention, (2) safe active learning mechanism that only encourages agents to explore in safe regions, (3) a state-based action correction mechanism that perceive possible catastrophes in the future and avoid them in advance, (4) efﬁcient MPC solving method that outputs a safe alternative policy in continuous action space.
Related Work
Safe RL: An interaction process is called safe if the agent rarely accesses the dangerous state [9, 11, 13].
Introducing external knowledge is the only way to ensure safety [12, 17].
Recently, various kinds of explicit constraints are used to impose some form of safety measures.
To name some, constraints on expectation [9, 10], constraints on uncertainty [7, 18], and Conditional
Value-at-Risk [11, 19, 20].
Unlike the above trajectory-based safety guarantees, [13] and [21] solve an optimization problem in the policy-level to ensure the safety of states.
CARL [22] ﬁrst trains the agent in non-safety-critical environments, and then adapts to the target environment.
In the setting of implicit constraints, HIRL [12] and [15] train a supervised learner to imitate the human’s intervention decisions and use it to evaluate the safety of the current state-action pairs.
DDPGfD [16] and SAVED [14] learn the policy efﬁciently and safely in the sparse reward environment by using suboptimal human demonstrations.
This work also focuses on learning implicit safety constraints from human knowledge.
The main difference of MBHI is that (1) we image in the leaned dynamics to avoid potential hazards in the future, (2) the human knowledge only contains the judgment about catastrophes, not suboptimal demonstrations, and (3) non-safety-critical source environments are not available.
Model-based RL: Model-based RL exhibits low sample complexity and fast convergence.
The learned dynamics model can be used for planning [4, 23], value expansion [24, 25], or imaginary training [26, 27, 28].
In planning approaches, MBMF [29] and PETS [23] use Model Predictive
Control (MPC) to select actions based on the learned model.
MVE [25] and STEVE [24] incorporate value expansion to improve the targets for temporal difference learning.
ME-TRPO [26] and
MBPO [27] are directly trained on imaginary data to accelerate policy learning.
To limit the inﬂuence of the model error, the uncertainty estimation is widely used in MBRL.
For instance, PETS [23] use uncertainty-aware models for trajectory propagation and planning.
STEVE [24] interpolates between different imaginary rollout lengths to favor those targets with lower uncertainty.
M2AC [30] uses the masking mechanism to only use the model when the model error is small.
In this work, STEVE is used as the base learning algorithm, but our framework is generally applicable to other model-based algorithms with uncertainty quantiﬁcation, such as MBPO, M2AC, etc.
We choose STEVE for three main reasons.
Firstly, it explicitly estimates uncertainty for both dynamics and Q-function.
Secondly, if the dynamics model cannot be learned or the Q-value network converges better, STEVE can reduce to model-free algorithm instead of learning from inaccurate imaginary data.
Thirdly, since STEVE is an off-policy algorithm, the safety detection process in
MBHI can be integrated with value expansion at each time-step to reduce calculation amount.
Preliminaries 3.1 Markov Decision Processes
We focus on Markov Decision Process (MDP) with continuous states and action spaces, deﬁned as ( S , A , R, p 0 , T, γ ) , in which S is the state space, A is the action space, R : S ×
A × S → R is the reward function, p 0 : S → R + is the initial state distribution, T : S ×
A → S is the dynamics function, and γ ∈ [0 , 1] is the discount factor.
The agent starts from an initial state s 0 ∼ p 0 , at each time step, the agent chooses an action a t ∈ A in state s t ∈ S according to a policy π θ ( s t ) parameterized with θ , transitions to a successor state s t
+1 =
T ( s t , a t ) and receives a reward r t =
R ( s t , a t , s t +1 ) .
The goal of reinforcement learning is to ﬁnd an optimal policy π θ ∗ that maximize the expected discounted sum of rewards θ ∗ = arg max {+θ E s 0 ∼ p 
+}
[ (cid:80) ∞ t =0 γ t r t ] .
3.2 Stochastic Ensemble Value Expansion
In model-based RL, the agent learns the world model from the replay buffer to approximate the true dynamics function.
In STEVE, the dynamics model consists of the following modules:
[Equation]
Termination model :
[Equation] An ensemble of parameters is maintained to estimate uncertainty in the learned dynamics (i.e. ξ = { ξ i } N T i =1 , φ = { φ i } N R i =1 , ψ = { ψ i } N D i =1 ).
The entire dynamics model is optimized as follows:
[Equation]
where H is the cross-entropy, and the expectation is calculated over data { ( s t , a t , s t +1 , r t , d t ) } .
For actor-critic methods or Q-learning methods, action-value function ˆ Q ϕ is a critical quantity to guide policy updating.
The Temporal-Difference target is written as:
[Equation]
where ϕ − means target network.
STEVE improves TD targets by combining the short-term value estimate using the dynamics and long-term value estimate using ˆ Q ϕ .
An ensemble of Q-functions is also maintained to evaluate the relative uncertainty between the dynamics and Q-function.
Through these uncertainty estimates, STEVE dynamically interpolates between imagined rollouts of different horizon lengths, favoring those candidate targets with lower uncertainty, and give up utilizing them when signiﬁcant errors are introduced.
Thus, the interpolated target T STEV E H is written as
[Equation]
5) where D i =
d t · (cid:81) ij =1 (1 − ˆ D ψ ( s t + j , a t + j , s t + j +1 )) , s t
+ h +
=
ˆ T ξ ( s t + h , a t + h ) .
T h is the value expansion target with length h , T µh and T σ 2 h are the empirical mean and variance for each T h .
If the environment is too complex or noise, STEVE will ignore the dynamics and reduce to model-free algorithm immediately.
Methods
In this section, we present a novel model-based safe RL framework, MBHI, as shown in Figure 2.
In the interaction phase, at each time-step, we use the current policy network π θ to roll out multiple steps in the learned dynamics, and check the safety of the imaginary trajectories with the supervised
Blocker.
If the catastrophe probability of the imaginary future rollout is greater than the threshold, the output action will be intercepted and replaced by the MPC controller.
In policy learning phase, the model-based method is used to update the policy.
Moreover, to overcome catastrophic forgetting problem of the network [12], we split replay buffer into safe and unsafe categories.
4.1 Catastrophe Prediction Network
Similar to HIRL [12], a supervised learner is trained to imitate the human overseer and block actions that are unsafe, but we maintain an ensemble of parameters for the Blocker µ = { µ i } N B i =1 to evaluate the uncertainty.
In concrete terms, the human-imitator method has the following advantages.
Blocker can learn from data logs, human intervention data or explicit safety constraints by supervised learning.
2) Blocker is modular, it can be applied in distributed system and different tasks.
3) As binary network, the output value of the Blocker can be regarded as the distribution distance between the input state and the catastrophe zone.
4) Blocker can provide gradient information.
During human oversight phase, we store ( s t , a t , s t +1 , r t , d t , c t ) at each time-step, where c t is a binary label for whether the current state is dangerous deﬁned by humans.
Thus, the Blocker can be trained in a supervised manner by minimizing the cross entropy loss as:
[Equation]
Each ˆ B µ i is initialized with various weights and trained on different input sequences.
At this stage, for model-based RL, the collected data can also be used to train the dynamics using Eq (2).
For the sake of fairness, in all experiments, we did not pre-train the dynamics model during the human oversight phase.
4.2 Safe Active Exploration
In order to reduce model uncertainty, we aim to query those points with large entropy, or where the disagreement would be most reduced in a posterior model.
In our settings, the dynamics model takes the form of ensemble, and the disagreement can be quantiﬁed by calculating the empirical variance across the output of each model.
Therefore, the optimization objective can be written as:
[Equation]
where c a is the coefﬁcient of the active learning reward and λ is a policy dependent weighting.
The second term of Eq 7 can also be regarded as maximizing the differential entropy of the transition model.
The model outputs is in the form of empirical Gaussian distribution, its differential entropy can be easily obtained, given by H ( x ) = 12 (ln(2 πσ 2 ) + 1) .
Therefore, maximizing variance is equivalent to maximizing model entropy.
It is also possible to maximize the variance of the predicted reward, as proposed in [31], but we ﬁnd it perform poorly in the sparse reward environment.
However, for safe RL, we do not want agents to visit unsafe regions, even if these areas have greater uncertainty.
A simple yet effective method is to mask the exploration reward near the dangerous area.
The catastrophe probability predicted by the Blocker can be regarded as the distribution distance to the unsafe region.
Thus, Eq (7) can be rewritten in the form of safety-aware.
[Equation]
[Equation]
where the exponent α controls the masking rate.
λ determines the level of exploration guided by the learned transition model, it should tend towards stability with the convergence of the policy and the full exploration of the environment.
In this work, we use the uncertainty measure of the dynamics to adjust λ adaptively.
Formally, λ is deﬁned as λ t +
=
[Equation]
)) , where ¯ σ 2 t denotes the mean variance.
This restricts λ to [0 , 0 . 5] and make the exploration to be guided by the reward signal.
4.3 Replacement of Catastrophic Actions
Before the agent execute the action to the environment, a C step trajectory from current state is rolled out in each dynamics.
The catastrophe probability of the imagination is computed as follows.
[Equation]
If p is greater than the safety threshold, it is considered that the agent will have a disaster in a certain number of steps under the current policy, and it is necessary to replace the action in advance.
In
HIRL [12], the author selects the alternative action based on a lookup table or the logit score ranking.
However, these methods are not suitable for continuous action space and ”non-local” catastrophes.
In this work, Model-Predictive Control is used as an expert to output safe actions.
Given a state s t , the MPC prediction horizon H and the action sequence a t : t + H − 1 , each dynamics model is used to predict the resulting trajectories { ( s t
+ i , r t + i − 1 ) } Hi =1 .
At each timestep t , the MPC controller solves the following optimization problem
[Equation]
where ˆ r MPC is the MPC reward.
There are two requirements of it.
First, guide the agent complete the obstacle-independent task.
Second, guide the agent away from the unsafe regions.
In consequence, ˆ r MPC contains the predicted external reward and the predicted disaster probability.
In addition, we further introduce a Leave-One-Out safety estimation to make MPC samples as far away from the unsafe region as possible.
Given the unsafe dataset D us and the MPC sampled state s t + h , the estimator is deﬁned as u t + h = D KL [ p ( ·|D us ∪ s t + h ) || p ( ·|D us )] .
The value of u t + h is close to zero, when p ( ·|D us ∪ s t + h ) and p ( ·|D us ) gradually match each other, which indicates that the sampled state is dangerous.
Therefore, we aim to maximize (cid:80)
Hh =1 u t + h to keep the whole MPC sampled sequence away from danger.
The MPC return can be formulated as
[Equation]
t
[Equation]
h
[Equation]
t
[Equation]
h
[Equation] t [Equation] Finally, the controller executes the ﬁrst action a t to the environment, advances to the next time-step, and recheck the safety of the agent.
[Figure]
To improve the efﬁciency and performance of
MPC, especially in high-dimensional action space, as shown in Figure 3, the gradient of the dynamics is used to guide the update of
CEM sampling distribution.
Each iteration consists of three steps: (1) Sample action sequences from CEM sampling distribution, and calculate MPC return with Eq (11).
(2) Fix the dynamics parameters, and update action sequences via gradient ascent method.
(3) Update the parameters of CEM sampling distribution.
The action replacement mechanism is summarized in algorithm ?? in Appendix ?? .
A negative reward must be given to guide the agent to learn a safe policy.
The interception mechanism in MBHI considers the danger of the current policy in state s t rather than the state-action pair.
Thus, the negative reward should solely depend on the state.
In this work, an intrinsic reward signal is generated based on how far the agent is from the unsafe region to encourage the agent to stay away from these areas.
Speciﬁcally, the output of Blocker is served as this kind of measurement.
Besides, to increase the punishment near the unsafe area, a boundary value Bound is introduced.
When the predicted catastrophe probability is greater than this value, a larger scaling factor is used to increase the negative reward.
The intrinsic reward signal is computed as
[Equation]
else
[Equation]
where c l and c h are scaling factors, which control the penalties in different areas.
Bound is the safety bound, a smaller bound will increase the complexity of the environment and obtain a more conservative policy.
Finally, the overall optimization problem can be written as
[Equation]
where r a is the active learning reward in Eq (8).
The full procedure is outlined in Algorithm ?? in
Appendix ?? .
Experiments 5.1 Experiment Setting
In this section, we evaluate our approach and various baselines over ﬁve safety-critical continuous control tasks.
1) PuckWorld-L & PuckWorld-H: they differ in acceleration.
The catastrophe of
PuckWorld-H is locally avoidable, while PuckWorld-L is not.
The agent is rewarded for reaching the target and is constrained to not touch the barrier.
Reacher: the agent is rewarded for reaching the target without touching the vertical bar.
3) Ant-Limit: the agent needs to move forward between two parallel barriers.
4) Ant-Block: the agent needs to move forward and avoid obstacles in the path.
The last three tasks are adapted from PyBullet’s environments [32].
The visualization and details of these environments are given in Appendix ?? . All experiments are evaluated over 5 random seeds.
We compare our proposed MBHI against both the model-based and model-free baselines, PPO,
DDPG and STEVE, and human intervention method, HIRL [12].
we extend HIRL to the continuous action space and also use STEVE as the basic algorithm.
To ensure a fair comparison, HIRL and
MBHI have the same Blocker and action replacement mechanism.
HIRL also splits the replay buffer like MBHI.
The ﬁnal implementation differences between HIRL and MBHI are summarized as follows: (1) HIRL only receives a negative reward when a catastrophic action is blocked, while penalty is introduced in MBHI in the form of Eq (12); (2) HIRL does not employ active exploration; (3) HIRL does not image in the learned dynamics to detect potential disasters in the future.
In all experiments, we use the same network structure of the policy, value, dynamics and Blocker for all algorithms, the number of ensemble models is 4.
Details can be found in Appendix ?? .
5.2 Comparison of Performance
      ,WHUDWLRQV H    

 
$ Y H U DJH  U H W X U Q (a) 
     ,WHUDWLRQV H    

 
$ Y H U DJH  U H W X U Q (b)     
    ,WHUDWLRQV H    

 
$ Y H U DJH  U H W X U Q (c)       ,WHUDWLRQV H         $ Y H U DJH  U H W X U Q (d)       ,WHUDWLRQV H         $ Y H U DJH  U H W X U Q (e)
Huawei Confidential
MBHI
HIRL
STEVE
DDPG
PPO
The comparison results in Fig 4 demonstrate that the sample efﬁciency of MBHI is better than baselines, especially DDPG and PPO.
MBHI achieves similar or better performances in all tasks
      ,WHUDWLRQV H 

       & X P X O D W L Y H  F D W D V W U RSKH V (a) 
     ,WHUDWLRQV H  
       & X P X O D W L Y H  F D W D V W U RSKH V (b)     
    ,WHUDWLRQV H  
      & X P X O D W L Y H  F D W D V W U RSKH V (c)       ,WHUDWLRQV H   
      & X P X O D W L Y H  F D W D V W U RSKH V (d)       ,WHUDWLRQV H        & X P X O D W L Y H  F D W D V W U RSKH V (e)
Huawei Confidential
MBHI
HIRL
STEVE
DDPG
PPO than baselines.
In PuckWorld-H, the parameters of MBHI and HIRL in safety inspection are exactly the same, but MBHI can converge faster, which illustrates the importance of active exploration.
Frequent action shielding increases the complexity of policy learning.
However, as an off-policy method, we do not ﬁnd perceptible negative effects for MBHI.
The experiment results of STEVE and DDPG in Fig 5 demonstrate that optimal unsupervised behaviors can result in a large number of catastrophes.
Note that DDPG and PPO has fewer catastrophes in some environments because the agent has not learned how to complete the task, and tends to move near the initial state.
For example, in Ant-Block, the agent does not even successfully move to the place where it could encounter the obstacle, which can explain why the number of catastrophes of
PPO suddenly rises in Fig 5e.
Additionally, HIRL is able to successfully resolve locally avoidable catastrophes, but failed in other environments.
By contrast, MBHI effectively avoids both ”local” and ”non-local” catastrophes during training.
In the simple environment like PuckWorld, MBHI almost achieves zero catastrophes.
In tasks other than PuckWorld-H, HIRL cannot effectively save the agent.
However, from Figure 5, we can see that the number of catastrophes in HIRL is signiﬁcantly less than STEVE.
This is because we divide the replay buffer in HIRL into safe and unsafe parts.
Besides, we can also ﬁnd that even though STEVE has converged, the number of catastrophes is still rising.
Our ﬁndings are in line with recent results on catastrophic forgetting [12, 33].
MBHI can well restrain this problem.
5.3 Ablation on Safety Detection Rollout Length
Intuitively, the longer the safety detection distance, the more conservative the policy.
We evaluate different safety detection rollout lengths on PuckWorld-L and
PuckWorld-H.
The result is shown in
Figure 6.
It shows that, surprisingly, when the safety detection distance is sufﬁcient, our method is not so sensitive to the length of safety detection rollouts.
This also proves that safety detection in the imagination can effectively avoid the catastrophe that will happen in advance.
In addition, the long detection distance does not further reduce the cumulative catastrophes signiﬁcantly, but it will introduce unnecessary calculation costs.
5.4 Comparison of Action Replacement Methods
We study the impact of different MPC optimization techniques and reward settings.
The compared optimizers include the uniform random search [29] and the cross-entropy method [23].
The MPC
      ,WHUDWLRQV H    
  
$ Y H U DJH  U H W X U Q / / / /  (a) 
     ,WHUDWLRQV H    
  
$ Y H U DJH  U H W X U Q / / / /  (b) 3XFN:RUOG/RZ 3XFN:RUOG+LJK       & X P X O D W L Y H  F D W D V W U RSKH V / / / /  (
[Figure]
3XFN:RUOG/RZ 3XFN:RUOG+LJK       , Q W H U F HS W L RQ  V X FF H VV  U D W H (a) 3XFN:RUOG/RZ 3XFN:RUOG+LJK 
     , Q W H U F HS W L RQ  V X FF H VV  U D W H (b)
[Figure]
reward settings include three types: 1) the reward ˆ r predicted by the learned reward function; 2) ˆ r and the catastrophe probability ˆ B predicted by the Blocker; 3) ˆ r , ˆ B , and the distribution distance u between MPC samples and unsafe samples.
For the sake of fairness, we also use ˆ r + ˆ B + u as the
MPC reward in Random and CEM.
We sample ten points uniformly around the catastrophe as the initial position, and make the agent move toward the catastrophe at the fastest speed.
In addition, the converged dynamics model and the pre-trained model are used as the simulator of MPC to evaluate the robustness of the proposed method.
The results of the interception success rate are shown in
Figure 7.
It shows that our method has better interception performance and is more robust.
Conclusion
In this paper, we proposed a state-based safety oversight mechanism MBHI, which is based on the model-based RL algorithm with active learning method.
MBHI is forward-looking, it considers the safety of the current policy in a certain time-step in the future, replaces dangerous actions in advance, and realizes the prevention of both ”local” and ”non-local” catastrophes.
We evaluated the proposed algorithm on ﬁve continuous safety control tasks, the experiment results demonstrate that
MBHI can signiﬁcantly reduce catastrophes and accelerate training.
Furthermore, our solution acts directly at the policy level, so it is independent of the learning algorithm and can be plugged into any existing model-based and model-free continuous control algorithms.
MBHI effectively alleviates the problem of ”non-local” catastrophes, but it needs to have a certain understanding of the task.
The safety detection distance needs to be long enough to avoid catastrophes in advance.
Besides, hyperparameters are sensitive to the performance of the Blocker.
Going forward, the safety detection length should be adaptive to the speciﬁc task, rather than set a ﬁxed value.
Moreover, in future research, we plan to study the theoretical guarantee of MBHI and improve the algorithm’s robustness to the hyper-parameters.
In terms of computation, since safety evaluation inevitably increases the cost of training process, we plan to study more effective methods to estimate the potential danger of the agent in the future and to integrate these estimates in model-based policy optimization.
Acknowledgments
We would like to thank Yang Li and our anonymous reviewers for their insightful comments and suggestions.
This work was supported in part by the National Key Research and Development Program of China (grant No. 2019YFB1312600), in part by the National Natural Science Foundation of China (grant No. 52075480), in part by the Key Research and Development Program of Zhejiang
Province, China (grant No. 2021C01008), in part by the High-level Talent Special Support Plan of
Zhejiang Province, China (grant No. 2020R52004) and in part by the Natural Science Foundation of Zhejiang Province, China under Grant Y19E050078.