D O
A NDROIDS
D REAM
OF
E LECTRIC
F ENCES ?
S AFETY -A
WARE
R EINFORCEMENT
L EARNING
WITH
L ATENT S HIELDING
Anonymous authors
Paper under double-blind review
A BSTRACT
The growing trend of ﬂedgling reinforcement learning systems making their way into real-world applications has been accompanied by growing concerns for their safety and robustness.
In recent years, a variety of approaches have been put forward to address the challenges of safety-aware reinforcement learning; however, these methods often either require a handcrafted model of the environment to be provided beforehand, or that the environment is relatively simple and lowdimensional.
We present a novel approach to safety-aware deep reinforcement learning in high-dimensional environments called latent shielding .
Latent shielding leverages internal representations of the environment learnt by model-based agents to “imagine” future trajectories and avoid those deemed unsafe.
We experimentally demonstrate that this approach leads to improved adherence to formallydeﬁned safety speciﬁcations.
I NTRODUCTION
The steady trickle of reinforcement learning (RL) systems making their way out of the lab and into the real world has cast a spotlight on the safety and robustness of RL agents.
The motivation behind this should be relatively easy to grasp: when training an agent in real-world settings, it is desirable that some states are never reached as they could, for instance, cause permanent damage to the hardware the agent is controlling.
We can thus informally deﬁne the notion of safety-aware
RL in terms of the classical RL setup with the added requirement that the number of unsafe states visited be minimised.
Under this deﬁnition, however, it has been found that many state-of-the-art
RL algorithms unnecessarily enter unsafe states despite safe alternatives being available and there being a positive correlation between avoiding such states and reward (Giacobbe et al., 2021).
The ﬁeld of safety-aware RL encompasses a multitude of approaches ranging from constrained policy optimisation (Chow et al., 2017; Achiam et al., 2017; Yang et al., 2020) to safety critics (Srinivasan et al., 2020; Bharadhwaj et al., 2021; Thananjeyan et al., 2021) to meta-learning (Turchetta et al., 2020).
In this work, we focus on a particular family of approaches known as shielding (Alshiekh et al., 2018; Anderson et al., 2020; Giacobbe et al., 2021; ElSayed-Aly et al., 2021; Pranger et al., 2021).
Central to shielding is the notion of a shield , a ﬁlter that checks actions proposed by the agent’s existing policy with reference to a model of the environment’s dynamics and some formal safety speciﬁcation.
The shield overrides actions that may lead to an unsafe state using some other safe (but by no means optimal) policy.
A key advantage of many shielding approaches is that the resulting shielded policies are formally veriﬁable; however, a shortcoming is that they require a model of environmental dynamics - typically handcrafted - to be provided in advance.
Providing such a model may prove difﬁcult for complex real-world environments, with inaccuracies and human biases creeping into handcrafted models.
In this work, we propose a safe RL agent that makes uses of latent shielding , an approach to shielding in environments where a formally-speciﬁed dynamics model is not available in advance.
At an intuitive level, the agent uses a data-driven approach to learn its own latent world model (a component of which is a dynamics model) which is then leveraged by a shield.
The shield then uses the agent’s model to “imagine” trajectories arising from different actions, forcing the agent to avoid those it foresees leading to unsafe states.
In addition, the agent can be trained within its own latent world model thus reducing the number of safety violations seen during training.
Contributions The main contribution of this work is a framework for shielding agents in complex, stochastic and high-dimensional environments without knowledge of environmental dynamics a priori .
We further introduce a new method to aid exploration when training shielded agents.
Though our framework loses the formal safety guarantees associated with traditional symbolic shielding approaches, our experiments illustrate that latent shielding reduces unsafe behaviour during training and achieves testing performance comparable to previous symbolic approaches.
P RELIMINARIES
In this section, we cover some relevant background topics.
We begin by introducing our problem setup for safety-aware RL and give an overview of the speciﬁcation language used in this work.
This is followed by an outline of the latent world model we make use of in this work as well as a discussion on shielding.
2. P ROBLEM S ETUP
We consider an agent interacting with an environment E modelled as a partially observable Markov decision process (POMDP) with states s ∈ S E , observations o t ∈ O E , agent-generated actions
a t ∈
A E and scalar rewards r t ∈ R over discrete time steps t ∈ [0 , 1 , ..., T − 1] .
We assume the environment has been augmented with a labelling function L φ E : S E → { safe, unsafe } that, at each time step, informs us whether a violation has occurred with respect to some formal safety speciﬁcation φ .
For the avoidance of doubt, we deﬁne a violation to have occurred whenever φ does not hold.
This is a weaker assumption than previous works in shielding (which assume access to an abstraction of the environment) and can be thought as a secondary safety-focused reward function with a binary output.
Intuitively, the goal of the agent is to learn a policy π that maximises its expected cumulative reward while minimising the number of violations of φ
. 2. S YNTACTICALLY C O -S AFE L INEAR T EMPORAL L OGIC
In this work, we use syntactically co-safe Linear Temporal Logic (scLTL) (Kupferman & Vardi, 2001) as our speciﬁcation language.
Valid scLTL formulae over some set of atomic propositions
AP can be constructed according to the following grammar:
[Equation]
where d ∈ AP , ¬ (negation) , ∨ (disjunction) , ∧ (conjunction) are the familiar operators from propositional logic, and (cid:13) (next) , ∪ (until) and (cid:5) (eventually) are temporal operators.
We can monitor a co-safe LTL speciﬁcation using a technique known as progression (Bacchus & Kabanza, 2000).
2. R ECURRENT S TATE -S PACE M ODELS
We refer to the predictive model of an environment maintained by a model-based agent as its world model .
World models can be learnt from experience and be used both as a substitute for the environment during training (Ha & Schmidhuber, 2018; Hafner et al., 2021) and for planning at run-time (Hafner et al., 2019b).
Though many realisations of the notion of a world model exist, the world model used in this work is based on the recurrent state-space model (RSSM) proposed by Hafner et al. (2019b).
An RSSM is composed of three key components: a latent dynamics model, a reward model, and an observation model.
These components act on compact states formed from the concatenation of a deterministic latent state h t and stochastic latent state z t .
Latent Dynamics Model
The latent dynamics model is made up of a number of smaller models.
First, the recurrent model h t = f
( h t − 1 , z t − 1 , a t − 1 ) is used to compute the deterministic latent state based on the previous compact state and action.
From h t and the current observation o t , a distribution q ( z t | h t , o t ) over posterior stochastic latent states z t is computed by the representation model .
At the same time, a distribution p (ˆ z t | h t ) over prior stochastic latent states ˆ z t is computed by the transition model , based only on h t .
During training, the transition model attempts to minimise the Kullback Leibler (KL) divergence between the prior and posterior stochastic latent state distributions.
In doing this, the RSSM learns to predict future latent states (using the recurrent and transition models) without access to future observations.
Observation Model
The observation model computes the distribution p (ˆ o t | h t , z t ) over observations
ˆ o t for a particular state.
Though not strictly needed, the observation model can prove useful for visualising predicted future states and providing a richer training signal.
Reward Model
The reward model computes the distribution p (ˆ r t | h t , z t ) over rewards ˆ r t for a particular state.
In practice, the distributions p and q are implemented with neural networks p θ and q θ respectively, parameterised by some set of parameters θ .
These latent dynamics models deﬁne a fully-observable
Markov decision process (MDP) as the latent states in the agent’s own internal model can always be observed by the agent (Hafner et al., 2019a).
We denote the state space of this MDP (comprised of compact latent states) as S I .
2. S HIELDING
The classical formulation of shielding in RL is given by Alshiekh et al.
It assumes access to two ingredients: an LTL safety speciﬁcation and abstraction (a MDP model of the environment that captures the aspects of the environment relevant for planning ahead with respect to the safety speciﬁcation).
These ingredients are used to construct a formally veriﬁable reactive system that monitors the agent’s actions, overriding those which lead to violation states.
Proposed by Giacobbe et al.
(2021), bounded prescience shielding (BPS) avoids the need for handcrafted abstractions by exploiting the fact that some agents are trained in computer simulations.
The shield operates by leveraging access to the program underlying the simulation to look ahead into future states within some ﬁnite horizon.
Using BPS over classical shielding does, however, come with a few disadvantages.
Firstly, it requires access to the simulation at run-time which may prove difﬁcult to provide (especially in cases where running the simulation is computationally expensive).
Moreover, an agent using BPS, even when starting from a safe state, can ﬁnd itself entering unsafe states in cases where the number of steps between a violation being caused by an action and the violation state itself exceeds the shield’s look-ahead horizon.
This is not the case for classical shielding which resembles BPS with an inﬁnite horizon.
2. B OUNDED S AFETY
The notion of safety used by BPS is deﬁned over MDPs.
For an arbitrary MDP with states S and actions A , a bounded trajectory ρ of length H is a sequence of states and actions s 0 a 0 −→ s 1 a 1 −→ . . .
a n − 1 −−−→ s n comprised of no more than H states and with the ﬁnal state s n either being a terminal state or n = H − 1 .
We further denote the set of all ﬁnite trajectories starting from some arbitrary state s ∈ S by (cid:37) ( s ) and the set of all bounded trajectories of length H that start from s by (cid:37) H ( s ) .
We say a bounded trajectory ρ of length H satisﬁes H -bounded safety with respect to safety speciﬁcation φ , written S H ( ρ, φ ) , if and only if for all s i ∈ ρ , L φ E ( s i )
= safe .
Moreover, we can extend the notion of H -bounded safety over the set of policies: a policy π is H -bounded safe with respect to φ , denoted as S H ( π, φ ) , if and only if for all s ∈ S ,
• either there exists some ρ ∈ (cid:37) H ( s ) such that S H ( ρ, φ ) and π ( s 0 ) = a 0 ; • or for all ρ ∈ (cid:37) H ( s ) , ¬ S H ( ρ, φ ) .
In other words, the policy will choose a safe trajectory as long as one exists.
Finally, we formally deﬁne a violation of φ to be inevitable in state s 0 ∈ S if and only if for all ρ ∈ (cid:37) ( s 0 ) , ¬ S H ( ρ, φ ) .
S HIELDED D REAMS
We introduce the notion of latent shielding , a novel class of shielding approaches that replace abstractions used by shields with learned latent dynamics models thus allowing the enforcement of formal safety speciﬁcations while avoiding the need for an explicitly-deﬁned abstraction of the environment.
We further introduce the ﬁrst such approach, approximate bounded prescience shielding , a framework for latent shielding that leverages latent world models learnt by model-based deep RL (DRL) agents.
In this work, our model-based agent of choice is Dreamer (Hafner et al., 2021) 1 , which we modify to incorporate shielding into its data collection, training and deployment phases.
3. S AFETY RSSM
We augment the standard RSSM with a labelling function L φ I : S I
→ { safe, unsafe } which maps latent states s t ∈ S I to whether they correspond to states in violation of φ .
As with the other models,
L φ I is implemented with a neural network with a categorical output l θ also parameterised by θ .
This yields an enhanced RSSM (illustrated in Figure 1) which we will refer to as a safety RSSM (SRSSM) .
We train l θ along with the other components of the SRSSM with the objective
[Equation]
(2) where the ﬁrst three terms are as described in (Hafner et al., 2019a; 2021) and, for convenience, can be found in Appendix B. L violation is a new term that we introduce that acts as a weighted binary cross-entropy loss over predictions by l θ
[Equation]
where D λ ⊆ S × { 0 , 1 } is a dataset of state-label pairs (with 0 corresponding to safe and 1 to unsafe ) and α ∈ [0 , 1] is a weighting term.
The inclusion of the weighting hyperparameter α is motivated by the observation that the number of observations of safe and unsafe states will generally be quite imbalanced.
Thus, to prevent predictions by l θ from collapsing to a single class, we assign a greater weight to the less-represented class.
3. A PPROXIMATE B OUNDED P RESCIENCE S HIELDING
We now integrate the SRSSM as part of a latent shielding approach which we shall refer to as approximate bounded prescience shielding (ABPS) .
Though ABPS is inspired by BPS, it differs in two key aspects: (1) we approximate the labelling function L φ E and environmental dynamics using an
SRSSM; and (2) we sample a ﬁxed number of potential future trajectories as opposed to exhaustively exploring all possibilities.
Thus, our approach can be thought of as an approximation of some “ideal” bounded prescience shield that uses the true environmental dynamics and labelling function.
The advantage of the ﬁrst difference should be obvious: it enables the shield to learn its own abstraction, removing the need for hand-crafting or access to a digital environment’s underlying program.
Why the second difference is advantageous may be slightly less obvious - it’s a heuristic that allows us to increase the horizon
H .
By directing the sampling of trajectories in accordance with states and actions the policy is biased towards (as opposed to uniformly), it may be possible to achieve comparable performance to a standard BPS in foreseeing unsafe states.
The intuition behind this is that by not sampling trajectories the agent is unlikely to take, more of our computational budget can be dedicated into ensuring that the agent’s most likely trajectories are safe.
In other words, we will not spend time planning to correct actions that the agent is unlikely to take.
More formally, our shielded policy π ∗ can be written:
[Equation]
where π ( s t ) is the agent’s policy without shielding; s
t ∈ S
I is some compact latent state; and π alt is an alternative policy that ensures that, if a violation isn’t already inevitable, the agent avoids a predicted unsafe state.
In our implementation, π alt simply considers all the other actions until a safe trajectory is found.
In the event that no safe action is found, the agent takes a random action.
Since the SRSSM models stochasticity, we can sample multiple futures arising from an action being taken and derive probabilistic estimates of whether an action will lead to a violation.
In this way, we can estimate the safety of an action taken in a given state by checking whether the probability of a violation occurring (inferred by sampling) exceeds a ﬁxed threshold (cid:15) (see Algorithm 1).
Moreover, the sampling process can, in practice, be augmented to sample a wider range of trajectories (less likely to be taken by the agent) by adding a small amount of noise to actions suggested by the policy during sampling.
3. T RAINING R EGIME
We extend the training regime proposed in Hafner et al. (2019a) to include the training and application of ABPS.
Though the full training procedure is detailed in Algorithm 2, we provide an overview below.
3.3. O VERVIEW
The training procedure can be split into three phases: data collection, latent world model training and agent training (lines 10-22, 4-6 and 7-8 in Algorithm 2 respectively).
These phases are cycled through until convergence.
Data Collection
In this phase, the agent interacts with the real environment to collect a dataset
D ⊆ S ×
A × O × R × { 0 , 1 } of states, actions, observations, rewards and violations with which a latent world model can be learned.
At the very start of training, we collect trajectories from S seed episodes using a random policy; at all other times, we use the agent’s shielded policy.
An illustration of the components of the agent active during this phase can be found in Figure 4 in Appendix A.
Latent World Model Training The goal of this phase is to improve the model of the world so that the policy has an accurate imagined environment to train in.
To this end, we train the latent world model with respect to the objective in Equation 2 on B data sequences of length L sampled from D .
An illustration of the components of the agent active during this phase can be found in Figure 5 in
Appendix A.
Agent Training Agent training is composed of two steps.
First, starting from states from the same
B data sequences from the latent world model training phase, the agent imagines I trajectories of length H with actions chosen from its current policy.
Next, the unshielded policy π is updated as in
Hafner et al.
(2019a; 2021).
3.3. K EY C HANGES AND C ONTRIBUTIONS
We describe and discuss the key changes we have made that differentiate our agent from previous approaches.
Experience Dataset.
Elements in the experience dataset D now contain a binary variable representing whether a violation has occurred λ t .
Latent Shielding.
Before being sent to the environment E , actions a t generated by the unshielded policy π are routed through our newly-proposed latent shield (described in Algorithm 1).
The shield returns a new approximately H -bounded safe action a (cid:48) t .
Intrinsic Punishment.
Whenever a violation occurs (whether detected by the latent shield or by the environment), we override the environment’s reward function by instead assigning a reward p < 0 .
This discourages the agent’s unshielded policy π from taking actions that lead to unsafe states through the standard RL setup.
Over time, this means that the shield will have to interfere less as π becomes biased away from unsafe states.
This is a standard practice in the shielding literature (Alshiekh et al., 2018), however new to the Dreamer family of agents (Hafner et al., 2019a;b; 2021).
Shield Introduction Schedule
We introduce the novel notion of a shield introduction schedule Π which can enable and disable shielding during training.
The rationale behind this is that a shield backed by an inaccurate world model will incorrectly label some safe states as unsafe, and vice versa.
In some cases, this may prove detrimental to the training process: suppose l θ happens to be initialised in such a way that all states are labelled as unsafe.
As a result, the shield can prevent the agent from exploring the environment and improving its internal model of the environment.
This, in turn, can prevent the labelling function from learning to correctly differentiate safe states from unsafe states.
To give the agent time to learn a good world model before restricting exploration through a learned shield, we augment the training procedure with Π which aims to gradually introduce shielding.
To our knowledge, this is the ﬁrst time such a system has been proposed.
Though there exist a wide range of possible implementations of Π , in this work we use a simple schedule that seems to work well empirically: start training with shielding disabled and enable shielding once the world model loss begins to plateau (see Appendix H for more details).
A detailed comparison of different shield introduction schedules, however, is beyond the scope of this study and left for further work.
E XPERIMENTS
In this section we compare our ABPS agent against Dreamer without shielding (Hafner et al., 2019a; 2021), Dreamer with BPS (Giacobbe et al., 2021), and CPO (Achiam et al., 2017).
We also empirically investigate some aspects of the internal workings of our agent.
A brief overview of the environments used can be found below with more detailed a discussion in Appendix C.
Visual Grid World
The Visual Grid World (VGW) environment is a simple deterministic navigation benchmark with high-dimensional ( 64 × 64 × 3 ) visual observations (see Figure 2) and discrete actions.
The agent’s (in green) task is to navigate to randomly-placed targets (in black) while avoiding any unsafe locations (in red).
We experiment with both ﬁxed and procedurally generated grids.
Cliff Driver The Cliff Driver (CD) environment is a symbolic benchmark with stochastic dynamics and continuous actions.
The agent controls the forward acceleration of a car and is tasked with driving to the edge cliff as quickly as possible without overshooting and falling into the sea.
Stochasticity comes from the fact that at each time step there is a probability p stick that car’s controls become “stuck” (Machado et al., 2018).
4. P ERFORMANCE E VALUATION
We evaluate our agents on three metrics: (1) average reward per episode at test-time; (2) average number of violations per episode at test-time; and (3)
[Table]
total number of violations during training.
Results are calculated by averaging over ﬁve trained versions of each agent.
The latent shield horizon H was 2 and 6 for the VGW and CD environments respectively.
Moreover, since the action space of the CD environment is continuous, we discretise the actions into four bins when performing BPS (see Appendix D.5).
Such modiﬁcations, however, are not needed for our agent.
Full details on training and evaluation can be found in Appendix D.
We also provide further evaluation of our agents on environment-speciﬁc metrics in Appendix F.
As can be seen in Table 4, our agent collected more reward at test-time than both the unshielded and BPS agents in the VGW environment (reward curves during training can be seen in Figure 3 with further training curves showcased in Appendix E).
Moreover, our agent saw a seven-fold reduction in test-time violations compared to the unshielded agent across both the ﬁxed and procedurally generated environments, averaging at less than one violation per episode for the ﬁxed VGW.
This reduction in training violations is even more dramatic when compared to the model-free CPO agent.
Our agent outperformed the other agents at test-time in the most stochastic CD environment ( p stick = 0 . 5 ).
This is possibly due to the SRSSM used by our latent shield being better able to capture non-determinism than BPS.
Nevertheless, the result is still rather impressive given that the agent using BPS sampled 1024 trajectories at every time step whereas our agent only sampled 20.
4. O THER E XPERIMENTS
Qualitative Evaluation of Learned Latent Dynamics We evaluate the quality of each agent’s latent dynamics model by observing its trajectory predictions given a particular starting state and action sequence.
Each agent is given the same starting observation and predeﬁned sequence ofactions.
The agents take these actions in their respective latent world models with the states traversed being decoded for inspection.
Results can be seen in Appendix G and indicate that, qualitatively, our agent’s decoder performed the best, accurately predicting 10 frames into the future.
One possible reason for this observation is that the inclusion of the violation detection objective in the SRSSM encourages the model to focus on accurately capturing violation states.
Another potential factor is that the BPS agent never actually enters unsafe states and thus ﬁnds it difﬁcult to represent them.
Do Shield Introduction Schedules Actually Help?
We perform an ablation study with shield introduction schedules in the VGW environment.
Results can be found in Appendix I and demonstrate decreased performance in the absence of shield introduction schedules.
R ELATED W ORK AND D ISCUSSION
Latent World Models Learning latent world models from visual observations has seen growing interest from the RL community (Wahlstr¨om et al., 2015;
Watter et al., 2015; Racani`ere et al., 2017;
Ha & Schmidhuber, 2018; Hafner et al., 2019a;b; Schrittwieser et al., 2020; Hafner et al., 2021).
One key trend in the literature is that of training agents in their own learned world models (Ha &
Schmidhuber, 2018; Hafner et al., 2019a; 2021).
In this work, we extend the world model formulation used in Hafner et al.
(2019a;b; 2021) to encode a notion of safety into state representations.
Safety By Filtering Actions Overriding unsafe actions with safe ones has been a popular approach to safety-aware RL.
Introduced into the RL scene by Alshiekh et al.
(2018), shielding has received much research interest and has seen applications in real-world settings (Nikou et al., 2021).
Various works have attempted to address some of its shortcomings of the original formulation.
These include allowing the shield to be updated to aid exploration or correct model imperfections (Anderson et al., 2020; Pranger et al., 2021); improving performance in non-deterministic environments (Jansen et al., 2020; Li & Bastani, 2020); and extending shielding to multi-agent RL (ElSayed-Aly et al., 2021).
To our knowledge, few works (Srinivasan et al., 2020; Thananjeyan et al., 2021; Bharadhwaj et al., 2021; Giacobbe et al., 2021) focus on removing the need for handcrafting an abstraction (among the most time-consuming and error-prone aspects of shielding), and only one of them (Giacobbe et al., 2021) achieves this without getting rid of the abstraction altogether, albeit by providing the agent with access to the program that controls the environment.
In contrast, latent shielding tackles the problem head-on by directly learning an abstraction for use by the shield.
In this work, the abstraction we use is an SRSSM, which captures stochasticity by design, a useful property for non-deterministic environments.
Though our latent shield satisﬁes an approximation of H -bounded safety (see Section 2.4) with respect to the learned abstraction, its safety with respect to the true environmental dynamics is not guaranteed and instead relies on the ﬁdelity with which the true dynamics are captured.
Furthermore, unlike for its formally-veriﬁed predecessors, it is a necessary sacriﬁce that the agent visits unsafe states during training in order to learn a notion of safety (unless, of course, pre-training is possible).
Nevertheless, a learned abstraction may be advantageous in settings where handcrafting an abstraction is not feasible and privileged access to some simulation (as in Giacobbe et al.
(2021)) cannot be assumed.
Moreover, latent shielding may provide a greater degree of explainability over model-free methods which get rid of the abstraction altogether (Srinivasan et al., 2020;
Thananjeyan et al., 2021; Bharadhwaj et al., 2021)
- if the shield overrides an action, one can reconstruct the imagined unsafe trajectories that led to the interference.
C ONCLUSIONS
In this paper, we have presented latent shielding , a new framework for shielding DRL agents without the need for a handcrafted abstraction of the environment.
Using this framework, we have designed a novel shield and demonstrated that this method not only leads to improved adherence to safety speciﬁcations on two benchmark environments with respect to an unshielded agent, but also works out-of-the-box on both continuous and discrete environments (unlike its predecessor, BPS).
Furthermore, we have demonstrated for the ﬁrst time that shielding at inappropriate times may adversely impact the performance of model-based DRL agents and showed how this phenomenon can be counteracted using our novel notion of shield introduction schedules .
Our work opens several exciting avenues for future work.
For instance, this work uses a very simple shield introduction schedule; future work may provide a richer investigation into the properties of different schedules.
Moreover, though demonstrating promising empirical results, our realisation of latent shielding loses the formally veriﬁable safety guarantees enjoyed by many symbolic shielding approaches - whether it is possible to construct a veriﬁable implementation of latent shielding, or compensate for the loss of formal guarantees, are open problems.
B D REAMER L OSS F UNCTIONS
Given some dataset of experience D e ⊆ S × O E × R :
[Equation]
where β is a hyperparameter for KL balancing (Hafner et al., 2021).
Here, minimising L observation and L reward corresponds to maximising the log-likelihood of ˆ o and ˆ r respectively; minimising L KL corresponds to minimising the KL divergence between the z and ˆ z .
C E NVIRONMENT D ETAILS C. V ISUAL G RID W ORLD
The Visual Grid World is a simple visual navigation benchmark.
The environment (see Figure 2) is made up of the agent (green circle), a target (black circle) and a number of unsafe locations (red squares).
The actions the agent can take correspond to moving up, down, left, right and staying still; visual observations from the environment are encoded in 64 × 64 × 3 arrays.
The agent’s task is to navigate to the target, avoiding any unsafe locations.
This yields a relatively straightforward safety speciﬁcation:
[Equation]
where agent in red square is true if and only if the agent is in an unsafe location, and episode ended is only true at the end of an episode.
The reward function (see Equation 6) used is also quite simple, with a small penalty term at each time step to encourage movement (though indeed different forms of intrinsic motivation may be also used).
[Equation]
We train and test our models on randomised 4x4 instances of the environment, each with either one or two unsafe states.
C. C LIFF D RIVER
The Cliff Driver environment is a simple symbolic benchmark with stochastic dynamics.
The agent controls a car on a one-dimensional road with continuous actions a t ∈ [ − 1 , 1] which correspond to the acceleration of the car.
The car cannot move backwards and its speed v t is lower-bounded at 0 (thus a t < 0 corresponds to braking as opposed to reversing).
At the end of the road, there is a cliff.
The agent starts each episode stationary at a ﬁxed distance x 0 from the edge of the cliff with its distance at subsequent time steps being written x t .
Furthermore, at each time step, there is a probability p stick that the car’s controls get “stuck” meaning that the current action is ignored and replaced with the previous action.
The agent’s goal is to drive the car to the edge of the cliff as quickly as possible, without overshooting and falling into the sea.
Observations from the environment are given as two-dimensional vectors encoding the distance from the edge of the cliff in one component and the speed of the car in the other.
The safety speciﬁcation can thus be written:
¬ agent fell off cliff ∪ episode ended where agent fell off cliff is true if and only if the agent has overshot the cliff, and episode ended is only true at the end of an episode.
Finally, the reward function is given as:
[Equation]
D E VALUATION D ETAILS D. G ENERAL
Agents were trained from different seeds on a machine with a single NVIDIA RTX 2080 Ti GPU, an AMD Ryzen 7 2700X processor and 64GB of RAM.
All agents were trained using the Adam optimiser (see Appendix D.6 for hyperparameters).
Each agent was trained ﬁve times, from different random seeds (except the CPO agent in the VGW environments, which, due to computational limitations, was only trained from three seeds).
For each environment, the model-based agents were trained for the same number of steps; the CPO agent was allowed to train for more steps (2 × longer for the VGW environments and 5 × longer for the CD environments).
D. E VALUATION P ROTOCOL
Agents trained on the ﬁxed VGW were evaluated over ten 500-step episodes of the same environment conﬁguration they were trained in.
Agents trained on the procedurally generated VGW were evaluated over ten random environment conﬁgurations.
Agents trained on the CD environments were evaluated over ten episodes, each with a different random seed.
D. N ETWORK A RCHITECTURES
We use the network architectures proposed in Hafner et al. (2019a).
The number of nodes in each layer varied depending on the environment and can be found in Appendix D.6.
The size of the fully-connected layers and output of the observation encoders and decoders can be found under Observation Embedding Size ; the size of the fully-connected layers in the reward, value, observation and violation models can be found under NN Hidden Layer Size .
We implement our encoder for symbolic observations from the CD environment as a feed-forward network with three fullyconnected hidden layers and ReLU activations.
For the CPO agent, we use the same observation encoders and policy networks as mentioned above.
D. S HIELD I NTRODUCTION S CHEDULES
For details on how we constructed these schedules, see Appendix H.
Visual Grid World Start with shield disabled.
After 10 episodes (including the 5 seed episodes), enable shielding every third episode.
After 20 episodes, enable shielding every other episode.
After 30 episodes, enable shielding all the time.
Linearly decay the unsafe threshold (cid:15) form 0.5 to 0.
over the entire course of training.
Cliff Driver Start with shield disabled.
After 60 episodes (including the 50 seed episodes), enable shielding all the time.
D. B OUNDED P RESCIENCE S HIELD FOR C ONTINUOUS A CTION S PACES
In our experiments in the CD environment, we make the action space amenable to BPS by discretising actions into four bins.
When the agent outputs a continuous action, we allocate the action to a bin by rounding the action to the nearest action in the set {− 1 , − 0 .
This rounded action is then sent to the shield, which exhaustively searches through the bounded trajectories generated by taking the actions deﬁned in the aforementioned set.
D. H YPERPARAMETERS
Hyperparameter values were kept constant for the base Dreamer model in all agents in each environment.
For the CPO agent, we use the same hyperparameters as Ray et al.
Hyperparameter
Symbol
Value
Grid World
Cliff Driver
Deterministic State SizeStochastic State SizeNN Hidden Layer SizeObservation
Embedding SizeDiscount Factor γ 0.
Action RepeatSeed Episodes
S
Episode Length
T
Batch Size
B
Sequence Length
L
Training Steps
C
Exploration Noise Variance0.
Imagination Horizon
I
KL Balancing Ratio β : (1 − β ) 4: 4:
Violation Balancing Ratio α : (1 − α ) 1: 1:
Model Learning Rate1e- 1ePolicy Learning Rate8e- 8eValue
Learning Rate8e- 8eBit Depth - -
Adam Epsilon1e- 1eAdam Beta0.9, 0.
Latent Shield Horizon
H
Latent Shield Sampled Trajectories
N
[Table]
(cid:15) [Table] [Table] (cid:15)
E T RAINING C URVES
Shaded regions in the plots denote a single standard deviation.
Note that rewards for the shielded agents may look lower due to intrinsic punishment.
That is, the shields can assign punishments (to discourage potentially unsafe behaviour) in addition to the environment’s reward function.
Furthermore, since the CPO agents were were trained for many more steps than the others, we plot their ﬁnal performance as single dotted lines.
E.
V ISUAL G RID W ORLD R EWARD
See Figure 3.
[Figure]
[Figure]
[Figure]
We evaluate the agents on two further environment-speciﬁc evaluation metrics below.
The ﬁrst is the average number of targets reached during a VGW episode. The second is the average time taken to travel 90% of the distance to the edge of the cliff during a CD episode.
[Table]
G Q UALITY OF L ATENT D YNAMICS
H M ETHOD FOR C ONSTRUCTING A S HIELD I NTRODUCTION S CHEDULE
The method we used to construct our shield introduction schedules is as follows:
Train the agent without shielding and plot the violation loss L violation (see Equation 3).
Look for the “elbow” in the plot (see Figure 10 for an example).
Disable the shield until around the time of the “elbow”.
Once the elbow has been reached, enable shielding.
If the violation curve does not plateau as nicely as in Figure 10, consider introducing the shield more gradually (e.g. once every n episodes, where n eventually decays to 1).
I A BLATION S TUDY WITH S HIELD I NTRODUCTION S CHEDULES
We compare the ﬁrst 100 training episodes of our agent in the ﬁxed VGW environment with and without the shield introduction schedule outlined in Appendix D.4.
As in the other experiments, results are calculated over ﬁve trained agents.
Though both agents start at roughly the same reward, the agents with a shield introduction schedule consistently outperforms their counterparts without shield introduction schedules. J A N OTE ON THE T ITLE
Yes, the title of this work is indeed a spin off Do Androids Dream of Electric Sheep? .
Here, androids dreaming of electric fences refers to RL agents dreaming (i.e. using latent imagination) about being punished for entering forbidden parts of an environment’s state space.