Original: S AFE R EINFORCEMENT L EARNING F ROM P IXELS U SING A S TOCHASTIC L ATENT R EPRESENTATION
Revised: S AFE R EINFORCEMENT L EARNING F ROM P IXELS U SING A S TOCHASTIC L ATENT R EPRESENTATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We address the problem of safe reinforcement learning from pixel observations.
Revised: We address the problem of safe reinforcement learning from pixel observations.

Original: Inherent challenges in such settings are (1) a trade-off between reward optimization and adhering to safety constraints, (2) partial observability, and (3) highdimensional observations.
Revised: Inherent challenges in such settings are (1) a trade-off between reward optimization and adhering to safety constraints, (2) partial observability, and (3) highdimensional observations.

Original: We formalize the problem in a constrained, partially observable Markov decision process framework, where an agent obtains distinct reward and safety signals.
Revised: We formalize the problem in a constrained, partially observable Markov decision process framework, where an agent obtains distinct reward and safety signals.

Original: To address the curse of dimensionality, we employ a novel safety critic using the stochastic latent actor-critic (SLAC) approach.
Revised: To address the curse of dimensionality, we employ a novel safety critic using the stochastic latent actor-critic (SLAC) approach.

Original: The latent variable model predicts rewards and safety violations, and we use the safety critic to train safe policies.
Revised: The latent variable model predicts rewards and safety violations, and we use the safety critic to train safe policies.

Original: Using well-known benchmark environments, we demonstrate competitive performance over existing approaches regarding computational requirements, final reward return, and satisfying the safety constraints.
Revised: Using well-known benchmark environments, we demonstrate competitive performance over existing approaches regarding computational requirements, final reward return, and satisfying the safety constraints.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: As reinforcement learning (RL) algorithms are increasingly applied in the real-world (Mnih et al., 2015; Jumper et al., 2021; Fu et al., 2021), their safety becomes ever more important with the increase of both model complexity, and uncertainty.
Revised: As reinforcement learning (RL) algorithms are increasingly applied in the real-world (Mnih et al., 2015; Jumper et al., 2021; Fu et al., 2021), their safety becomes ever more important with the increase of both model complexity, and uncertainty.

Original: Considerable effort has been made to increase the safety of RL (Liu et al., 2021).
Revised: Considerable effort has been made to increase the safety of RL (Liu et al., 2021).

Original: However, major challenges remain that prevent the deployment of RL in the real-world (Dulac-Arnold et al., 2021).
Revised: However, major challenges remain that prevent the deployment of RL in the real-world (Dulac-Arnold et al., 2021).

Original: Most approaches to safe RL are limited to fully observable settings, neglecting issues such as noisy or imprecise sensors.
Revised: Most approaches to safe RL are limited to fully observable settings, neglecting issues such as noisy or imprecise sensors.

Original: Moreover, realistic environments exhibit high-dimensional observation spaces and are largely out of reach for the stateof-the-art.
Revised: Moreover, realistic environments exhibit high-dimensional observation spaces and are largely out of reach for the stateof-the-art.

Original: In this work, we present an effective safe RL approach that handles partial observability with high-dimensional observation spaces in the form of pixel observations.
Revised: In this work, we present an effective safe RL approach that handles partial observability with high-dimensional observation spaces in the form of pixel observations.

Original: In tandem with prior work, we formalize the safety requirements using a constrained Markov decision process (CMDP; Altman, 1999).
Revised: In tandem with prior work, we formalize the safety requirements using a constrained Markov decision process (CMDP; Altman, 1999).

Original: The objective is to learn a policy that maximizes a reward while constraining the expected return of a scalar cost signal below a certain value (Achiam et al., 2017).
Revised: The objective is to learn a policy that maximizes a reward while constraining the expected return of a scalar cost signal below a certain value (Achiam et al., 2017).

Original: According to the reward hypothesis, it could be possible to encode safety requirements directly in the reward signal.
Revised: According to the reward hypothesis, it could be possible to encode safety requirements directly in the reward signal.

Original: However, as argued by Ray et al.
Revised: However, as argued by Ray et al.

Original: (2019), safe RL based only on a scalar reward carries the issue of designing a suitable reward function.
Revised: (2019), safe RL based only on a scalar reward carries the issue of designing a suitable reward function.

Original: In particular, balancing the tradeoff between reward optimization and safety within a single reward is a difficult problem.
Revised: In particular, balancing the tradeoff between reward optimization and safety within a single reward is a difficult problem.

Original: Moreover, over-engineering rewards to complex safety requirements runs the risk of triggering negative side effects that surface after integration into broader system operation (Abbeel & Ng, 2005; Amodei et al., 2016).
Revised: Moreover, over-engineering rewards to complex safety requirements runs the risk of triggering negative side effects that surface after integration into broader system operation (Abbeel & Ng, 2005; Amodei et al., 2016).

Original: Constrained RL addresses this issue via a clear separation of reward and safety.
Revised: Constrained RL addresses this issue via a clear separation of reward and safety.

Original: Reinforcement learning from pixels typically suffers from sample inefficiency, as it requires many interactions with the environment.
Revised: Reinforcement learning from pixels typically suffers from sample inefficiency, as it requires many interactions with the environment.

Original: In the case of safe RL, improving the sample efficiency is especially crucial as each interaction with the environment, before the agent reaches a safe policy, has an opportunity to cause harm (Zanger et al., 2021).
Revised: In the case of safe RL, improving the sample efficiency is especially crucial as each interaction with the environment, before the agent reaches a safe policy, has an opportunity to cause harm (Zanger et al., 2021).

Original: Moreover, to ensure safety, there is an incentive to act pessimistic with regard to the cost (As et al., 2022).
Revised: Moreover, to ensure safety, there is an incentive to act pessimistic with regard to the cost (As et al., 2022).

Original: This conservative assessment of safety, in turn, may yield a lower reward performance than is possible within the safety constraints.
Revised: This conservative assessment of safety, in turn, may yield a lower reward performance than is possible within the safety constraints.

Original: Our contribution.
Revised: Our contribution.

Original: We introduce Safe SLAC, an extension of the Stochastic Latent Actor Critic approach (SLAC; Lee et al., 2020).
Revised: We propose Safe SLAC, an extension of the Stochastic Latent Actor Critic approach (SLAC; Lee et al., 2020) to problems with safety constraints.

Original: SLAC learns a stochastic latent variable model of the environment dynamics, to address the fact that optimal policies in partially observable settings must estimate the underlying state of the environment from the observations.
Revised: SLAC learns a stochastic latent variable model of the environment dynamics, to address the fact that optimal policies in partially observable settings must estimate the underlying state of the environment from the observations.

Original: The model predicts the next observation, the next latent state, and the reward based on the current observation and current latent state.
Revised: The model predicts the next observation, the next latent state, and the reward based on the current observation and current latent state.

Original: This latent state inferred by the model is then used to provide the input for an actor-critic approach (Konda & Tsitsiklis, 1999).
Revised: This latent state inferred by the model is then used to provide the input for an actor-critic approach (Konda & Tsitsiklis, 1999).

Original: The actor-critic algorithm involves learning a critic function that estimates the utility of taking a certain action in the environment, which serves as a supervision signal for the policy, also named the actor.
Revised: This algorithm involves learning a critic function that estimates the utility of taking a certain action in the environment, which serves as a supervision signal for the policy, also named the actor.

Original: The SLAC method has excellent sample efficiency in the safety-agnostic partially observable setting, which renders it a promising candidate to adapt to high-dimensional settings with safety constraints.
Revised: The SLAC method has excellent sample efficiency in the safety-agnostic partially observable setting, which renders it a promising candidate to adapt to high-dimensional settings with safety constraints.

Original: At its core, SLAC is an actor-critic approach, so it carries the potential for a natural extension toconsider safety with a safety critic.
Revised: At its core, SLAC is an actor-critic approach, carrying the potential for a natural extension to safety with a safety critic.

Original: We extend SLAC in three ways to create our safe RL approach for partially observable settings (Safe SLAC): (1) the latent variable model also predicts cost violations, (2) we learn a safety critic that predicts the discounted cost return, and (3) we modify the policy training procedure to optimize a safety-constrained objective by use of a Lagrangian relaxation, solved using dual gradient descent on the primary objective and a Lagrange multiplier to overcome the inherent difficulty of constrained optimization.
Revised: We extend SLAC in three ways to create our safe RL approach under partial observability (Safe SLAC): (1) the latent variable model also predicts cost violations, (2) we learn a safety critic that predicts the discounted cost return, and (3) we modify the policy training procedure to optimize a safety-constrained objective by use of a Lagrangian relaxation, solved using dual gradient descent on the primary objective and a Lagrange multiplier to overcome the inherent difficulty of constrained optimization.

Original: We evaluate Safe SLAC using a set of benchmark environments introduced by Ray et al.
Revised: We evaluate Safe SLAC using a set of benchmark environments introduced by Ray et al.

Original: The empirical evaluation shows competitive results compared with complex state-of-the-art approaches.
Revised: The empirical evaluation shows competitive results compared with complex state-of-the-art approaches.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Established baseline algorithms for safe reinforcement learning in the fully observable setting include constrained policy optimization (CPO; Achiam et al., 2017), as well as TRPOLagrangian (Ray et al., 2019), a cost-constrained variant of the existing trust region policy optimization (TRPO; Schulman et al., 2015).
Revised: Established baseline algorithms for safe reinforcement learning in the fully observable setting include constrained policy optimization (CPO; Achiam et al., 2017), as well as trust region policy optimization (TRPO)-Lagrangian (Ray et al., 2019), a cost-constrained variant of the existing trust region policy optimization (TRPO; Schulman et al., 2015).

Original: While TRPO-Lagrangian uses an adaptive Lagrange multiplier to solve the constrained problem with primal-dual optimization, CPO solves the problem of constraint satisfaction analytically during the policy update.
Revised: While TRPO-Lagrangian uses an adaptive Lagrange multiplier to solve the constrained problem with primal-dual optimization, CPO solves the problem of constraint satisfaction analytically during the policy update.

Original: The method closest related to ours is Lagrangian model-based agent (LAMBDA; As et al., 2022), which also addresses the problem of learning a safe policy from pixel observations under high partial observability.
Revised: The method closest related to ours is Lagrangian model-based agent (LAMBDA; As et al., 2022), which also addresses the problem of learning a safe policy from pixel observations under high partial observability.

Original: LAMBDA uses the partially stochastic dynamics model introduced by (Hafner et al., 2019).
Revised: LAMBDA uses the partially stochastic dynamics model introduced by (Hafner et al., 2019).

Original: The authors take a Bayesian approach on the dynamics model, sampling from the posterior over parameters to obtain different instantiations of this model.
Revised: The authors take a Bayesian approach on the dynamics model, sampling from the posterior over parameters to obtain different instantiations of this model.

Original: For each instantiation, simulated trajectories are sampled.
Revised: For each instantiation, simulated trajectories are sampled.

Original: Then, the worst cost return and best reward return are used to train critic functions that provide a gradient to the policy.
Revised: Then, the worst cost return and best reward return are used to train critic functions that provide a gradient to the policy.

Original: LAMBDA shows competitive performance with baseline algorithms, however, there are two major trade-offs.
Revised: LAMBDA shows competitive performance with baseline algorithms, however, there are two major trade-offs.

Original: First, by taking a pessimistic approach, the learned policy attains a lower cost return than the allowed cost budget.
Revised: First, by taking a pessimistic approach, the learned policy attains a lower cost return than the allowed cost budget.

Original: A less pessimistic approach that uses the entirety of the allowed cost budget may yield a constraint-satisfying policy with a higher reward return.
Revised: A less pessimistic approach that uses the entirety of the allowed cost budget may yield a constraint-satisfying policy with a higher reward return.

Original: Second, the LAMBDA training procedure involves generating many samples from their variable model to estimate the optimistic/pessimistic temporal difference updates.
Revised: Second, the LAMBDA training procedure involves generating many samples from their variable model to estimate the optimistic/pessimistic temporal difference updates.

Original: While the reinforcement learning literature has numerous safety perspectives (Garcıa & Fern´andez, 2015; Pecka & Svoboda, 2014), we focus on constraining the behavior of the agent on expectation.
Revised: While the reinforcement learning literature has numerous safety perspectives (Garcıa & Fern´andez, 2015; Pecka & Svoboda, 2014), we focus on constraining the behavior of the agent on expectation.

Original: A method called shielding ensures safety already during training, using temporal logic specifications for safety (Alshiekh et al., 2018; Jansen et al., 2020).
Revised: A method called shielding ensures safety already during training, using temporal logic specifications for safety (Alshiekh et al., 2018; Jansen et al., 2020).

Original: Such methods, however, require extensive prior knowledge in the form of a (partial) model of the environment (Carr et al., 2022).
Revised: Such methods, however, require extensive prior knowledge in the form of a (partial) model of the environment (Carr et al., 2022).

Original:  C ONSTRAINED PARTIALLY OBSERVABLE M ARKOV DECISION PROCESSES
Revised:  C ONSTRAINED PARTIALLY OBSERVABLE M ARKOV DECISION PROCESSES

Original: In reinforcement learning, an agent learns to sequentially interact with an environment to maximize some signal of utility.
Revised: In reinforcement learning, an agent learns to sequentially interact with an environment to maximize some signal of utility.

Original: This problem setting is typically modeled as a Markov decision process (MDP; Sutton & Barto, 2018), in which the environment is composed of a set of states S and a set of actions A .
Revised: This problem setting is typically modeled as a Markov decision process (MDP; Sutton & Barto, 2018), in which the environment is composed of a set of states S and a set of actions A .

Original: At each timestep t , the agent receives the current environment state s t ∈ S and executes an action a t ∈ A , according to the policy π : a t ∼ π ( a t | s t ) .
Revised: At each timestep t , the agent receives the current environment state s t ∈ S and executes an action a t ∈ A , according to the policy π : a t ∼ π ( a t | s t ) .

Original: This action results in a new state according to the transition dynamics s t + ∼ p
Revised: This action results in a new state according to the transition dynamics s t + ∼ p

Original: ( s t +1 | s t , a t ) and a scalar reward signal r t = r
Revised: ( s t +1 | s t , a t ) and a scalar reward signal r t = r

Original: ( s t , a t ) ∈ R , where r is the reward function.
Revised: ( s t , a t ) ∈ R , where r is the reward function.

Original: The goal is for the agent to learn an optimal policy π ⋆ such that the expectation of discounted, accumulated reward in the environment under that policy is maximized,
Revised: The goal is for the agent to learn an optimal policy π ⋆ such that the expectation of discounted, accumulated reward in the environment under that policy is maximized,

Original: i.e. π ⋆ = arg max π E [ (cid:80) t γ t r t ] with γ ∈ [0 , 1) .
Revised: i.e. π ⋆ = arg max π E [ (cid:80) t γ t r t ] with γ ∈ [0 , 1) .

Original: We use ρ π to denote the distribution over trajectories induced in the environment by a policy π .
Revised: We use ρ π to denote the distribution over trajectories induced in the environment by a policy π .

Original: In a partially observable Markov decision process (POMDP; Kaelbling et al., 1998), the agent cannot observe the true state s t of the MDP and instead receives some observation x t ∈ X that provides partial information about the state, sampled from the observation function x t ∼ X ( s t ) .
Revised: In a partially observable Markov decision process (POMDP; Kaelbling et al., 1998), the agent cannot observe the true state s t of the MDP and instead receives some observation x t ∈ X that provides partial information about the state, sampled from the observation function x t ∼ X ( s t ) .

Original: In this setting, learning a policy π is more difficult than in MDPs since the true state of the underlying
Revised: In this setting, learning a policy π is more difficult than in MDPs since the true state of the underlying

Original: MDP is not known and must be inferred from sequences of observations to allow the policy to be optimal.
Revised: MDP is not known and must be inferred from sequences of observations to allow the policy to be optimal.

Original: Therefore, the optimal policy is a function of a history of observations and actions π ( a t | x 1: t , a 1: t − 1 ) .
Revised: Therefore, the optimal policy is a function of a history of observations and actions π ( a t | x 1: t , a 1: t − 1 ) .

Original: In practice, representing such policy can be infeasible, so the policy is often conditioned on a compact representation of the history.
Revised: In practice, representing such policy can be infeasible, so the policy is often conditioned on a compact representation of the history.

Original: While the reward hypothesis states that desired agent behavior can plausibly be represented in a single scalar reward signal, in practice it can be difficult to define a reward function that balances different objectives (Vamplew et al., 2022; Roy et al., 2022).
Revised: While the reward hypothesis states that desired agent behavior can plausibly be represented in a single scalar reward signal, in practice it can be difficult to define a reward function that balances different objectives (Vamplew et al., 2022; Roy et al., 2022).

Original: The same is true for safety: as argued by Ray et al.
Revised: The same is true for safety: as argued by Ray et al.

Original: (2019), a useful definition of safety is a constraint on the behavior, stating the problem as a constrained MDPs (CMDP; Altman, 1999), with analogous definitions for constrained
Revised: (2019), a useful definition of safety is a constraint on the behavior, stating the problem as a constrained MDPs (CMDP; Altman, 1999), with analogous definitions for constrained

Original: POMDPs (CPOMDP; Isom et al., 2008; Lee et al., 2018; Walraven & Spaan, 2018).
Revised: POMDPs (CPOMDP; Isom et al., 2008; Lee et al., 2018; Walraven & Spaan, 2018).

Original: A scalar cost variable c t ∈ R at each time step of the C(PO)MDP, according to a cost function c t = c ( s t , a t ) , serves as a measure of safety violation.
Revised: A scalar cost variable c t ∈ R at each time step of the C(PO)MDP, according to a cost function c t = c ( s t , a t ) , serves as a measure of safety violation.

Original: The objective of the reinforcement learning problem then changes to constrain the accumulated cost under a given safety threshold d ∈ R :
Revised: The objective of the reinforcement learning problem then changes to constrain the accumulated cost under a given safety threshold d ∈ R :

Original: [Equation]
Revised: [Equation]

Original: Next, we review RL algorithms for POMDPs and CMDPs, laying the foundations for the safe RL algorithm
Revised: Next, we review RL algorithms for POMDPs and CMDPs, laying the foundations for the safe RL algorithm

Original: we propose in Section 5 dedicated to CPOMDPs with unknown environment dynamics.
Revised: we propose in Section 5 dedicated to CPOMDPs with unknown environment dynamics.

Original:  S TOCHASTIC LATENT ACTOR - CRITIC
Revised:  S TOCHASTIC LATENT ACTOR - CRITIC

Original: Soft actor-critic (SAC; Haarnoja et al., 2018a) is an RL approach based on the maximum entropy framework.
Revised: Soft actor-critic (SAC; Haarnoja et al., 2018a) is an RL approach based on the maximum entropy framework.

Original: Besides the traditional reinforcement learning objective of maximizing the reward, it also aims to maximize the entropy of the policy.
Revised: Besides the traditional reinforcement learning objective of maximizing the reward, it also aims to maximize the entropy of the policy.

Original: The sampling of states that allow high-entropy action distributions is maximized.
Revised: The sampling of states that allow high-entropy action distributions is maximized.

Original: This approach can be robust to disturbances in the dynamics of the environment (Eysenbach & Levine, 2022).
Revised: This approach can be robust to disturbances in the dynamics of the environment (Eysenbach & Levine, 2022).

Original: In practice, it can be challenging to determine a weight for the entropy term in the objective a priori, so instead, the entropy is constrained to a minimum value (Haarnoja et al., 2018b).
Revised: In practice, it can be challenging to determine a weight for the entropy term in the objective a priori, so instead, the entropy is constrained to a minimum value (Haarnoja et al., 2018b).

Original: A stochastic policy is particularly interesting to our work since a deterministic policy might be suboptimal in the CPOMDP setting (Kim et al., 2011).
Revised: A stochastic policy is particularly interesting to our work since a deterministic policy might be suboptimal in the CPOMDP setting (Kim et al., 2011).

Original: Stochastic latent actor-critic (SLAC; Lee et al., 2020) is an RL algorithm that addresses partial observability and high-dimensional observations by combining a probabilistic sequential latent variable model with an actor-critic approach.
Revised: Stochastic latent actor-critic (SLAC; Lee et al., 2020) is an RL algorithm that addresses partial observability and high-dimensional observations by combining a probabilistic sequential latent variable model with an actor-critic approach.

Original: The sequential latent variable model aims to infer the true state of the environment by considering the environment as having an unseen true latent state z t and latent dynamics p ( z t +1 | z t , a t ) .
Revised: The sequential latent variable model aims to infer the true state of the environment by considering the environment as having an unseen true latent state z t and latent dynamics p ( z t +1 | z t , a t ) .

Original: This model generates observations x  ∼ p ( x t | z t ) and rewards r t ∼ p ( r t | z t  ) .
Revised: This model generates observations x t ∼ p ( x t | z t ) and rewards r t ∼ p ( r t | z t , a t , z t +1 ) .

Original: By approximating these latent dynamics in the latent variable model using approximate variational inference, the learned model can infer the latent state from previous observations and actions as q ( z t | x 0: t , a 0: t ) .
Revised: By approximating these latent dynamics in the latent variable model using approximate variational inference, the learned model can infer the latent state from previous observations and actions as q ( z t | x 0: t , a 0: t ) .

Original: Consequently, SLAC can be viewed as an adaption of SAC to POMDPs, using the inferred latent state z as input for the critic in SAC.
Revised: Consequently, SLAC can be viewed as an adaption of SAC to POMDPs, using the inferred latent state z as input for the critic in SAC.

Original: Equation (2) describes the latent variable model, which is parameterized by ψ .
Revised: Equation (2) describes the latent variable model M , which is parameterized by ψ .

Original: This model infers a low-dimensional latent state representation z from a history of high-dimensional observations x and actions a gathered through interaction with the environment.
Revised: This model infers a low-dimensional latent state representation z from a history of high-dimensional observations x and actions a gathered through interaction with the environment.

Original: This model is trained by sampling trajectories from the environment and inferring a latent state from each trajectory according to the distributions given in Equation (3).
Revised: This model is trained by sampling trajectories from the environment and inferring a latent state from each trajectory according to the distributions given in Equation (3).

Original: The model uses this inferred latent state to predict the next observation and reward according to the distributions in Equation (3).
Revised: The model uses this inferred latent state to predict the next observation and reward according to the distributions in Equation (3).

Original: [Equation]
Revised: [Equation]

Revised: Hereinafter, we use M to refer to the functions of the model ( p ψ z , p ψ x , p ψ r , q ψ 0 , q ψ z ), and use ψ without the superscript to simplify the notation.

Revised: Furthermore, following Lee et al.

Revised: (2020), in our implementation we decompose the latent variable z in two parts (see Appendix A for more details).

Original: The parameters of the model ψ are trained to optimize the objective in Equation (4), in which D KL is the Kullback-Leibler divergence, an asymmetric measure of the difference between two distributions commonly used in variational inference (Kingma & Welling, 2014):
Revised: The parameters of the model ψ are trained to optimize the objective in Equation (4), in which D KL is the Kullback-Leibler divergence, an asymmetric measure of the difference between two distributions commonly used in variational inference (Kingma & Welling, 2014):

Original: [Equation]
Revised: [Equation]

Original: The resulting inferred latent state, which captures information about previous observations and expected reward in future time steps, serves as the input for the critic in SAC.
Revised: The resulting inferred latent state, which captures information about previous observations and expected reward in future time steps, serves as the input for the critic in SAC.

Original: In practice, the probability distributions in Equations (2) and (3) are set to follow Gaussian distributions with diagonal covariance matrices, where the means and variances are the output of neural networks, optimized using the reparametrization trick (Kingma & Welling, 2014).
Revised: In practice, the probability distributions in Equations (2) and (3) are set to follow Gaussian distributions with diagonal covariance matrices, where the means and variances are the output of neural networks, optimized using the reparametrization trick (Kingma & Welling, 2014).

Original: The neural network that models q ψ ( z 11 | x 1 ) treats a lower-dimensional feature vector, extracted from the highdimensional observation using a convolutional neural network.
Revised: The neural network that models q ψ ( z 11 | x 1 ) treats a lower-dimensional feature vector, extracted from the highdimensional observation using a convolutional neural network.

Original: For further details, we refer the interested reader to Lee et al.
Revised: For further details, we refer the interested reader to Lee et al.

Original: SAC can be brought to the safety-constrained CMDP setting by taking a Lagrangian relaxation of the constrained problem and optimizing the unconstrained dual problem through dual gradient descent. This approach,named SAC-Lagrangian, has been used effectively to  constrain the stepwise cost (Ha et al., 2020) and cost return (Yang et al., 2021).
Revised: We can solve this problem taking a Lagrangian relaxation of the constrained problem and optimizing the unconstrained dual problem through dual gradient descent (Geibel & Wysotzki, 2005; Stooke et al., 2020). SAC-Lagrangian uses this approach, to bring SAC to the safety-constrained CMDP setting, where it may constrain the step-wise cost (Ha et al., 2020) or cost return (Yang et al., 2021).

Original: In the next section, we present our approach to adapt SLAC for the CPOMDP setting.
Revised: In the next section, we present our approach to adapt SLAC for the CPOMDP setting.

Original:  S AFE SLAC
Revised:  S AFE SLAC

Original: The Safe SLAC model is an adaption of SLAC for the safety-constrained setting under partial observability.
Revised: The Safe SLAC model is an adaption of SLAC for the safety-constrained setting under partial observability.

Original: There are three steps to this adaption.
Revised: There are three steps to this adaption.

Original: First, we adapt the latent variable model to include information related to safety in the latent state.
Revised: First, we adapt the latent variable model to include information related to safety in the latent state.

Original: Second, we train a safety critic to predict the expected cost return resulting from a given action.
Revised: Second, we train a safety critic to predict the expected cost return resulting from a given action.

Original: Third, we constrain the policy objective and use its
Revised: Third, we constrain the policy objective and use its

Original: Lagrangian relaxation to train the actor.
Revised: Lagrangian relaxation to train the actor.

Original: Latent variable model with cost prediction.
Revised: Latent variable model with cost prediction.

Original: We include a notion of safety in the latent state by changing the generative part of the model to predict the cost at the next time step, in addition to predicting the observation and reward.
Revised: We include a notion of safety in the latent state by changing the generative part of the model to predict the cost at the next time step, in addition to predicting the observation and reward.

Original: This introduces the conditional probability distribution c t ∼ p ψ  ( c t | z1 t , z 2 t , a t , z1 t +1 , z 2 t +1 ) to the generative component of the model, analogous to the component that predicts reward.
Revised: This introduces the conditional probability distribution c t ∼ p ψ c ( c t | z t , a t , z t +1 ) to the generative component of the model, analogous to the component that predicts reward.

Original: The resulting model objective is given in Equation (5):
Revised: Using the simplified notation of the parameters ( ψ ), the resulting model objective is:

Original: [Equation]
Revised: [Equation]

Original: While the other probability distributions in the model are Gaussian variables with a diagonal covariance matrix, we follow As et al.
Revised: While the other probability distributions in the model are Gaussian variables with a diagonal covariance matrix, we follow As et al.

Original: and use a Bernoulli distribution for the conditional cost distribution, since the cost is binary in the environments we consider.
Revised: and use a Bernoulli distribution for the conditional cost distribution, since the cost is binary in the environments we consider.

Original: Safety critic.
Revised: Safety critic.

Original: As an integral part of Safe SLAC, we train a safety critic.
Revised: As an integral part of Safe SLAC, we train a safety critic.

Original: First, the reward critic
Revised: First, the reward critic

Original: Q r with parameters θ remains unchanged from SLAC and estimates the expected discounted reward return for a given latent state and action pair under the current policy, as well as the expected entropy of the policy after taking the given action in the environment.
Revised: Q r with parameters θ remains unchanged from SLAC and estimates the expected discounted reward return for a given latent state and action pair under the current policy, as well as the expected entropy of the policy after taking the given action in the environment.

Original: To this end, Q r optimizes the soft
Revised: To this end, Q r optimizes the soft

Original: Bellman residual in Equation (6).
Revised: Bellman residual in Equation (6).

Original: [Equation]
Revised: [Equation]

Original: In addition to the reward critic, we train the safety critic Q c with parameters ζ that predicts the expected discounted cost return for a given latent state and action.
Revised: In addition to the reward critic [Equation] we train the safety critic Q c with parameters ζ that predicts the expected discounted cost return for a given latent state and action.

Original: The loss function for the safety critic, given in Equation (8), is similar to Equation (6) but does not include an entropy term.
Revised: The loss function for the safety critic, given in Equation (8) [Equation] is similar to Equation (6) but does not include an entropy term.

Original: [Equation]
Revised: [Equation]

Original: Safe SLAC policy objective.
Revised: Safe SLAC policy objective.

Original: SLAC follows the maximum entropy framework objective (Haarnoja et al., 2018b), using the latent state as input for the critics.
Revised: SLAC follows the maximum entropy framework objective (Haarnoja et al., 2018b), using the latent state as input for the critics.

Original: We adjust this to be constrained to safe behavior.
Revised: We adjust this to be constrained to safe behavior.

Original: The policy objective in safety-indifferent SLAC maximizes both the entropy of the policy, which ensures sufficient exploration, and the soft action-value estimate as predicted by the reward critic Q rθ :
Revised: The policy objective in safety-indifferent SLAC maximizes both the entropy of the policy, which ensures sufficient exploration, and the soft action-value estimate as predicted by the reward critic Q rθ :

Original: [Equation]
Revised: [Equation]

Original: Analogously to the safety critic term in SAC-Lagrangian (Ha et al., 2020), we obtain the objective for the safety-constrained policy by including a term to minimize the expected cost return as estimated by the safety critic Q cζ , weighted by the Lagrange multiplier λ :
Revised: Analogously to the safety critic term in SAC-Lagrangian (Ha et al., 2020), we obtain the objective for the safety-constrained policy by including a term to minimize the expected cost return as estimated by the safety critic Q cζ , weighted by the Lagrange multiplier λ :

Original: [Equation]
Revised: [Equation]

Original: As we discussed in Section 3, it is not practical to condition the policy on the full history.
Revised: As we discussed in Section 3, it is not practical to condition the policy on the full history.

Original: To circumvent this problem, Lee et al.
Revised: To circumvent this problem, Lee et al.

Original: (2020) use a truncated history of observations and actions as input for the policy; however, we find empirically that using the latent state as input for the actor yields better results in our setting.
Revised: (2020) use a truncated history of observations and actions as input for the policy; however, we find empirically that using the latent state as input for the actor yields better results in our setting.

Original: As such, in our training procedure the policy takes as input the current latent state z , which is updated after each environment step.
Revised: As such, in our training procedure the policy takes as input the current latent state z , which is updated after each environment step.

Original: When using the latent state as input for the policy, occurrences of π ϕ in Equations (6), (9) and (10) and Algorithm 1 can instead be read as modeling the distribution π ϕ ( a τ +1 | z τ +1 ) and π ϕ ( a t | z t ) , respectively.
Revised: When using the latent state as input for the policy, occurrences of π ϕ in Equations (6), (9) and (10) and Algorithm 1 can instead be read as modeling the distribution π ϕ ( a τ +1 | z τ +1 ) and π ϕ ( a t | z t ) , respectively.

Original: In the gradient steps, we infer the latent state from a truncated history of observations and actions that is sampled from the replay buffer, rather than from the full history.
Revised: In the gradient steps, we infer the latent state from a truncated history of observations and actions that is sampled from the replay buffer, rather than from the full history.

Revised: Note that Safe SLAC carries the same limitation as SLAC.

Revised: Overall, even in the unconstrained case, conditioning the policy on the latent variable is not guaranteed to result in the optimal policy of the underlying POMDP (see Lee et al., 2020).

Revised: To find an optimal policy, we must condition the policy in the history or a belief state that provides sufficient statistics of the history, which may lead the agent to take actions to gather information (Cassandra et al., 1994; Thrun, 1999).

Original: The entropy and expected cost-return are constrained, therefore, α and λ are adjusted dynamically to incentivize the policy to adhere to the constraints.
Revised: The entropy and expected cost-return are constrained, therefore, α and λ are adjusted dynamically to incentivize the policy to adhere to the constraints.

Original: α is tuned using the procedure and hyperparameters as in (Haarnoja et al., 2018b).
Revised: α is tuned using the procedure and hyperparameters as in (Haarnoja et al., 2018b).

Original: Next, we describe how to adjust λ during training.
Revised: Next, we describe how to adjust λ during training.

Original: 5. S AFETY L AGRANGE MULTIPLIER LEARNING
Revised: 5. S AFETY L AGRANGE MULTIPLIER LEARNING

Original: We use on-policy data to adjust the Lagrange safety multiplier.
Revised: We use on-policy data to adjust the Lagrange safety multiplier.

Original: Previous work in the fully observable setting has adjusted the Lagrange multiplier by estimating the cost return as the mean prediction of the safety critic over states sampled from the replay buffer(Yang et al., 2021).
Revised: Previous work in the fully observable setting has adjusted the Lagrange multiplier by estimating the cost return as the mean prediction of the safety critic over states sampled from the replay buffer(Yang et al., 2021).

Original: It has been shown for Q-learning (Thrun & Schwartz, 1993) and policy gradient methods (Fujimoto et al., 2018), that using function approximators such as neural networks to estimate state and action value functions can result in biased value estimates.
Revised: It has been shown for Q-learning (Thrun & Schwartz, 1993) and policy gradient methods (Fujimoto et al., 2018), that using function approximators such as neural networks to estimate state and action value functions can result in biased value estimates.

Original: While bias in value estimation can result in suboptimal policy learning, the induced bias is doubly important if the safety critic not only serves to provide a gradient to the agent, but its magnitude is also used to determine the Lagrange multiplier.
Revised: While bias in value estimation can result in suboptimal policy learning, the induced bias is doubly important if the safety critic not only serves to provide a gradient to the agent, but its magnitude is also used to determine the Lagrange multiplier.

Original: We have found in our experiments that for the given environments, this bias was sufficient to prevent the learning of safe policies and we were unable to mitigate this using the techniques proposed by Fujimoto et al.
Revised: We have found in our experiments that for the given environments, this bias was sufficient to prevent the learning of safe policies and we were unable to mitigate this using the techniques proposed by Fujimoto et al.

Original: We attribute this to the high degree of partial observability in the environments, the off-policy nature of the learning algorithm, and the use of 0-step temporal difference error for value updating.
Revised: We attribute this to the high degree of partial observability in the environments, the off-policy nature of the learning algorithm, and the use of 0-step temporal difference error for value updating.

Original: Consequently, we adjust the Lagrange multiplier λ based on the real undiscounted cost return incurred in each training episode.
Revised: Consequently, we adjust the Lagrange multiplier λ based on the real undiscounted cost return incurred in each training episode.

Original: This is expressed in the following loss, where λ is changed to minimize the incurred cost return less the allowed cost budget:
Revised: This is expressed in the following loss, where λ is changed to minimize the incurred cost return less the allowed cost budget:

Original: [Equation]
Revised: [Equation]

Original: where ρ is a trajectory and d is the budget.
Revised: where ρ is a trajectory and d is the budget.

Original: Intuitively, the weight of the safety critic in the policy loss is increased when the policy is unsafe and decreased when the policy is safe.
Revised: Intuitively, the weight of the safety critic in the policy loss is increased when the policy is unsafe and decreased when the policy is safe.

Original: In practice, we found this procedure to be stable and effective for a sufficiently low learning rate.
Revised: In practice, we found this procedure to be stable and effective for a sufficiently low learning rate.

Original: We stress that the evaluation episodes are not used for policy learning, including adjustment of the Lagrange multiplier.
Revised: We stress that the evaluation episodes are not used for policy learning, including adjustment of the Lagrange multiplier.

Original: Algorithm Input : Emptyreplay buffer D  Environment E , initialized parameters ψ, θ 1 , θ 2 , ζ and ϕ
Revised: Input : Empty D Input : Environment E Input : ψ

Original: Hyperparameters : W the number warmup environment , W number warmup steps
Revised: Hyperparameter Hyperparameter

Original: Output :Optimized parameters 1: Sample W p transitions from E using a stochastic policy and store thesein D .
Revised: Output : parameters 1: Sample W p transitions from E using a stochastic policy and store these D .

Original: 2: for i=1 to W t do 3:
Revised: 2: for i=1 to W t do 3:

Original: Sample experience from D 4:
Revised: Sample experience from D 4:

Original: Update ψ according to Equation (5) 5: end for 6: while not converged do 7: for each environment step at time t do 8: a t ∼ π ϕ ( a t | x 1: t , a 1: t − 1 ) 9:
Revised: Update ψ according to Equation (5) 5: end for 6: while not converged do 7: for each environment step at time t do 8: a t ∼ π ϕ ( a t | x 1: t , a 1: t − 1 ) 9:

Original: Obtain ( x t +1 , r t +1 , c t +1 ) by executing a t in E 10:
Revised: Obtain ( x t +1 , r t +1 , c t +1 ) by executing a t in E 10:

Original: D ← D ∪ ( x t +1 , a t , r t +1 , c t +1 ) 11:
Revised: D ← D ∪ ( x t +1 , a t , r t +1 , c t +1 ) 11:

Original: Update λ by minimizing Equation (11) 12: end for 13: for each gradient step do 14: ( x 1: τ +1 , a 1: τ , r 1: τ +1 , c 1: τ +1 ) ∼ D 15:
Revised: Update λ by minimizing Equation (11) 12: end for 13: for each gradient step do 14: ( x 1: τ +1 , a 1: τ , r 1: τ +1 , c 1: τ +1 ) ∼ D 15:

Original: Update θ 1 , θ 2 according to Equation (6) 16:
Revised: Update θ 1 , θ 2 according to Equation (6) 16:

Original: Update ψ according to Equation (5) 17:
Revised: Update ψ according to Equation (5) 17:

Original: Update ϕ according to Equation (10) 18:
Revised: Update ϕ according to Equation (10) 18:

Original: Update ζ according to Equation (8) 19: ¯ θ 1 ← νθ [Equation] + [Equation]
Revised: Update ζ according to Equation (8) 19: ¯ θ 1 ← νθ 1 + (1 − ν ) ¯ θ20: ¯ θ

Original: 1 − ν )

Original: ¯

Original: [Equation]
Revised: [Equation]

Original: 20: ¯ θ 2 ← νθ[Equation] + [Equation]
Revised: ← νθ

Original: − ν )

Original: [Equation]
Revised: [Equation]

Original: 21: ¯ ζ ←νζ
Revised: 21: ¯ ζ ←

Original: [Equation]

Original: − ν

Original: [Equation]
Revised: [Equation]

Original: ¯ ζ 22: end for 23: end while
Revised: 22: end for 23: end while

Original: 5. T RAINING PROCEDURE
Revised: 5. T RAINING PROCEDURE

Original: Safe SLAC is described in Algorithm 1.
Revised: Safe SLAC is described in Algorithm 1.

Original: The training procedure starts initializing the replay buffer by taking actions under a stochastic policy for 60k environment steps, with actions sampled from a truncated diagonal Gaussian distribution.
Revised: The training procedure starts initializing the replay buffer by taking actions under a stochastic policy for 60k environment steps, with actions sampled from a truncated diagonal Gaussian distribution.

Original: It then trains an initial latent variable model for 30k steps on sequences uniformly sampled from the replay buffer, minimizing the objective in Equation (5).
Revised: It then trains an initial latent variable model for 30k steps on sequences uniformly sampled from the replay buffer, minimizing the objective in Equation (5).

Original: After this warm-up procedure, we alternate between collecting experiences and training the latent variable model, actor, and critics.
Revised: After this warm-up procedure, we alternate between collecting experiences and training the latent variable model, actor, and critics.

Original: To improve stability during training, we maintain target networks for the critics, which are updated using exponential averaging (Lines 19 to 21).
Revised: To improve stability during training, we maintain target networks for the critics, which are updated using exponential averaging (Lines 19 to 21).

Original: The neural networks that parameterize the model, actor, and critics have the same architecture as in SLAC, with hyperparameters as detailed in Appendix A. Except for the safety Lagrange multiplier, we optimize all parameters using the Adam optimizer (Kingma & Ba, 2014).
Revised: The neural networks that parameterize the model, actor, and critics have the same architecture as in SLAC, with hyperparameters as detailed in Appendix B. Except for the safety Lagrange multiplier, we optimize all parameters using the Adam optimizer (Kingma & Ba, 2014).

Original: Whereas Lee et al.
Revised: Whereas Lee et al.

Original: (2020) indicate that the optimal additive noise added to the observation prediction during latent variable model training is of low variance for most environments, we find better results with a large variance (0.4).
Revised: (2020) indicate that the optimal additive noise added to the observation prediction during latent variable model training is of low variance for most environments, we find better results with a large variance (0.4).

Original: This is consistent with their hypothesis that a strong noise yields better results for environments with large differences between subsequent observations, as is the case in our environments.
Revised: This is consistent with their hypothesis that a strong noise yields better results for environments with large differences between subsequent observations, as is the case in our environments.

Original: We follow both SLAC and LAMBDA in using two action repeats on these environments, meaning that we transform the base environment so that each action executed in it is performed twice before the agent receives a new observation.
Revised: We follow both SLAC and LAMBDA in using two action repeats on these environments, meaning that we transform the base environment so that each action executed in it is performed twice before the agent receives a new observation.

Original: We aggregated both the reward and cost incurred in the transitions induced by these repeated actions to scalar values by summing them. 
Revised: We aggregated both the reward and cost incurred in the transitions induced by these repeated actions to scalar values by summing them. [Figure]

Original:  E MPIRICAL EVALUATION
Revised:  E MPIRICAL EVALUATION

Original: We evaluate our approach on a set of six SafetyGym benchmark environments introduced by Ray et al.
Revised: We evaluate our approach on a set of six SafetyGym benchmark environments introduced by Ray et al.

Original: (2019) as SG6, shown in Figure 1.
Revised: (2019) as SG6, shown in Figure 1.

Original: For all environments, we use the standard cost return budget of 25 per episode.
Revised: For all environments, we use the standard cost return budget of 25 per episode.

Original: As training progresses, we frequently evaluate the performance of the policy in [Figure] evaluation episodes.
Revised: As training progresses, we frequently evaluate the performance of the policy inevaluation episodes.

Original: The mean reward and cost return during these evaluation episodes are shown in Figure 2 for three different random seeds per environment.
Revised: The mean reward and cost return during these evaluation episodes are shown in Figure 2 for three different random seeds per environment.

Original: We calculate the normalized reward and cost return using the performance in the last 5 evaluations, as proposed by Ray et al.
Revised: We calculate the normalized reward and cost return using the performance in the last 5 evaluations, as proposed by Ray et al.

Original: We compare our results with the final results from basic algorithms using sensor observations and with LAMBDA trained on pixel observations, as reported by As et al.
Revised: We compare our results with the final results from basic algorithms, including the unconstrained proximal policy optimization (PPO; Schulman et al., 2017), using sensor observations, and with LAMBDA trained on pixel observations, as reported by As et al.

Original: ; see Section 2 for a description of these algorithms.
Revised: ; see Section 2 for a description of these algorithms.

Original: Figures 2 and 3 compile the results obtained in the SG6 set.
Revised: Figures 2 and 3 compile the results obtained in the SG6 set.

Original: Note that while Figure 3 allows comparison of performance on DoggoGoal1, the normalized rewards are difficult to read due to the large reward return of our unsafe policy.

Original: In Appendix B, we include the same metrics, with DoggoGoal1 excluded for clarity.
Revised: Appendix D presents a qualitative analysis of the results.

Original: Figure 2 shows that Safe SLAC converges on a safe policy for nearly all environments and attains a competitive reward return.
Revised: Figure 2 shows that Safe SLAC converges on a safe policy for nearly all environments and attains a competitive reward return.

Original: Safe SLAC reaches a higher reward return than LAMBDA compared to other baselines such as PointGoal1 and CarGoal1 environments.
Revised: Safe SLAC reaches a higher reward return than LAMBDA compared to other baselines such as PointGoal1 and CarGoal1 environments.

Original: In PointGoal2, we also reach a higher reward return than LAMBDA, but only after 2 million environment steps.
Revised: In PointGoal2, we also reach a higher reward return than LAMBDA, but only after 2 million environment steps.

Original: The results shown for LAMBDA in PointGoal2 used training for only 1 million steps; we reach the same performance as LAMBDA using approximately 40% more samples and continue to improve after this.
Revised: The results shown for LAMBDA in PointGoal2 used training for only 1 million steps; we reach the same performance as LAMBDA using approximately 40% more samples and continue to improve after this.

Original: Safety and reward tradeoff.
Revised: Safety and reward tradeoff.

Original: In the DoggoGoal1 environment, we observe that while the reward return of Safe SLAC is significantly better than for the other algorithms, the mean cost return is slightly larger than the allowed limit and is only better than CPO.
Revised: In the DoggoGoal1 environment, we observe that while the reward return of Safe SLAC is significantly better than for the other algorithms, the mean cost return is slightly larger than the allowed limit and is only better than CPO.

Original: This indicates that the tradeoff between reward and safety is balanced too far towards reward.
Revised: This indicates that the tradeoff between reward and safety is balanced too far towards reward.

Original: Adjusting the learning rate for the safety Lagrange multiplier from the default 2 ∗ 10 − 6 to 6 ∗ 10 − 6 , we obtain the results indicated in Figure 3 as Safe SLAC +, showing significant improvement in safety with an associated decrease in reward return.
Revised: Adjusting the learning rate for the safety Lagrange multiplier from the default 2 ∗ 10 − 6 to 6 ∗ 10 − 6 , we obtain the results indicated in Figure 3 as Safe SLAC+, which increases the reward return in PointGoal2.

Original: From this effect, we conclude that our policy makes a meaningful tradeoff between optimizing reward and cost return, and that the current training procedure for the Lagrange multiplier is sensitive to the learning rate.
Revised: From this effect, we conclude that our policy makes a meaningful tradeoff between optimizing reward and cost return, and that the current training procedure for the Lagrange multiplier is sensitive to the learning rate.

Original: Challenges in strong partial-observability.
Revised: Challenges in strong partial-observability.

Original: Like LAMBDA, our approach fails to make substantial improvement in reward return in PointPush1, although the policy is safe.
Revised: Like LAMBDA, our approach fails to make substantial improvement in reward return in PointPush1, although the policy is safe.

Original: We attribute this to the strong partial observability in this environment: while the box is pushed, the agent cannot observe its surroundings and so must rely entirely on its belief or latent state.
Revised: We attribute this to the strong partial observability due to the pixel observations: while the box is pushed, the agent cannot observe its surroundings and so must rely entirely on its belief or latent state.

Original: This is supported by the fact that the baselines that use sensor observations are able to attain a much higher reward return.
Revised: This is supported by the fact that the baselines that use sensor observations are able to attain a much higher reward return.

Original: Normalized metrics.
Revised: Normalized metrics.

Original: To summarize the performance of our algorithm, we normalize the mean reward return and cost return for each environment at the end of training by dividing them by the performance of the unsafe RL algorithm PPO (Schulman et al., 2015), as suggested by Ray et al.
Revised: To summarize the performance of our algorithm, we normalize the mean reward return and cost return for each environment at the end of training by dividing them by the performance of unconstrained PPO, as proposed by Ray et al.

Original: The resulting normalized metrics, shown in Figure 5, indicate that Safe SLAC improves over the baselines and is competitive with LAMBDA.
Revised: The resulting normalized metrics, shown in Figure 3, indicate that Safe SLAC improves over the baselines and is competitive [Figure] [Figure] [Figure]

Original: We exclude DoggoGoal1 from this figure, as
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: P o i n t P u s h

Original:  Safe SLAC is not safe for our standard learning rate as discussed above.
Revised: [Figure] [Figure] with LAMBDA. We exclude DoggoGoal1 from this figure, as Safe SLAC is not safe for our standard learning rate as discussed above.

Original: The normalized reward return is so large for Safe SLAC that it obscures the performance in other environments.
Revised: The normalized reward return is so large for Safe SLAC that it obscures the performance in other environments.

Original: The same figure with DoggoGoal1 included can be seen in Figure 3. [Figure]
Revised: The same figure with DoggoGoal1 included can be seen in Figure 5 (Appendix C).

Original: PointGoal2, the final results for LAMBDA are calculated after 1 million environment steps, whereas we use 2 million.

Original: History as actor input.
Revised: History as actor input.

Original: As discussed in Section 5, we deviate from the suggestion by Lee et al.
Revised: As discussed in Section 5, we deviate from the suggestion by Lee et al.

Original: (2020) to use the truncated history as input for the actor and instead use the latent state inferred from the full history to select actions.
Revised: (2020) to use the truncated history as input for the actor and instead use the latent state inferred from the full history to select actions.

Original: In Figure 4, we compare the performance of these two policy inputs on the PointGoal2 environment.
Revised: In Figure 4, we compare the performance of these two policy inputs on the PointGoal2 environment.

Original: While both variants converge to a safe policy, the reward performance of the latent state policy is higher than that of the history policy. 
Revised: While both variants converge to a safe policy, the reward performance of the latent state policy is higher than that of the history policy. [Figure] [Figure]

Original: Safe SLAC is less pessimistic than LAMBDA.
Revised: Safe SLAC may be less pessimistic than LAMBDA.

Original: For PointGoal2, LAMBDA was trained only with 1M environment steps.
Revised: For PointGoal2, LAMBDA was trained only with 1M environment steps.

Original: For this computational budget, Safe SLAC is able to find a safe policy, however, we notice it has not converged yet, and the reward return is slightly lower than the final reward [Figure]
Revised: For this computational budget, Safe SLAC is able to find a safe policy, however, we notice it has not converged yet, and the reward return is slightly lower than the final reward return from LAMBDA.

Original: [Figure] return from LAMBDA. Nevertheless, with more time Safe SLAC clearly outperforms LAMBDA, whose reward return in this environment plateaus before reaching 1 million environment steps (Figure 5 from As et al., 2022).
Revised: Nevertheless, with more time Safe SLAC clearly outperforms LAMBDA, whose reward return in this environment plateaus before reaching 1 million environment steps (Figure 5 from As et al., 2022).

Original: In summary, Safe SLAC can keep improving while LAMBDA stagnates as we use more of the allowed cost budget.
Revised: In summary, Safe SLAC can keep improving while LAMBDA stagnates as we use more of the allowed cost budget.

Original: Computational requirements.
Revised: Computational requirements.

Original: Our approach is less complex than LAMBDA, as it does not require costly sampling of Bayesian world models, or generating synthetic experience using the latent variable model.
Revised: Our approach is less complex than LAMBDA, as it does not require costly sampling of Bayesian world models, or generating synthetic experience using the latent variable model.

Original: As a result, the computational requirements are lower.
Revised: Thus, we conjecture that Safe SLAC also has lower computational requirements.

Original: As an example, training our approach for 1 million base environment steps on PointGoal1 takes a total of 10 hours andminutes.
Revised: As an example, training our approach for 1 million base environment steps on PointGoal1 takes a total of 10 hours and 14 minutes.

Original: For contrast, LAMBDA takes 13 hours and 20 minutes as measured on the same hardware.
Revised: For contrast, LAMBDA takes 13 hours and 20 minutes (measured on the same hardware).

Revised: Nevertheless, this improvement might also be due to implementations details.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper proposes Safe SLAC, a safety-constrained RL approach for partially observable settings, which uses a stochastic latent variable model combined with a safety critic.
Revised: This paper proposes Safe SLAC, a safety-constrained RL approach for partially observable settings, which uses a stochastic latent variable model combined with a safety critic.

Original: Empirically, it shows competitive performance with state-of-the-art methods while having a less complex training procedure.
Revised: Empirically, it shows competitive performance with state-of-the-art methods while having a less complex training procedure.

Original: Safe SLAC often derives policies with higher reward returns, while still satisfying the safety constraint.
Revised: Safe SLAC often derives policies with higher reward returns, while still satisfying the safety constraint.

Original: Future work includes using a distributed critic to increase the control over the safety violations (Yang et al., 2022) and improving the optimization of the Lagrange multiplier (Stooke et al., 2020), which is currently sensitive to its learning rate as seen in the DoggoGoal1 experiments.
Revised: Future work includes using a distributed critic to increase the control over the safety violations (Yang et al., 2022) and improving the optimization of the Lagrange multiplier (Stooke et al., 2020), which is currently sensitive to its learning rate as seen in the DoggoGoal1 experiments.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: An anonymized version of the code for our implementation of Safe SLAC is available on GitHub at https://github.com/safe-slac/safe-slac .
Revised: An anonymized version of the code for our implementation of Safe SLAC is available on GitHub at https://github.com/safe-slac/safe-slac .

Original: The repository contains implementations for the Safe SLAC actor, critics, model and training procedure.
Revised: The repository contains implementations for the Safe SLAC actor, critics, model and training procedure.

Original: This includes the necessary code to interface with the publicly available SafetyGym environments using pixel observations, as well as instructions to install the required dependencies and reproduce our experiments.
Revised: This includes the necessary code to interface with the publicly available SafetyGym environments using pixel observations, as well as instructions to install the required dependencies and reproduce our experiments.

Original: In addition, the default hyperparameters are specified in a configuration file
Revised: In addition, the default hyperparameters are specified in a configuration file

Original: A H YPERPARAMETERS
Revised: A L ATENT V ARIABLE M ODEL

Revised: Following the observations from Lee et al. (2020, Appendix B) [Equation]

Revised: we implemented a model composed of two latent variables z 1 and z 2 .

Revised: Equations (12) and (13) provide the detailed description of this model.

Revised: [Equation]

Original: [Table]
Revised: Notice that in this factorization the generative and variational models share some parts, namely Equations (13b) and (13d) are the same as Equations (12b) and (13d), respectively. As Lee et al. (2020) shows, this allow us to compute the KL-divergence D KL ( q ψ ( z t +1 | x t +1 , z t , a t )

Original: [Figure]
Revised: ∥

Original: [Table] [Figure] [Table] [Figure]
Revised: p ψ ( z t +1 | z t , a t )) based only on the z 1 , which simplifies the training process.

Original: [Table] [Figure]
Revised: We refer the reader to Lee et al.

Original: [Table] [Figure] [Table] [Figure]
Revised: (2020) for further details of the implementation.

Original: [Table] [Figure] [Table] [Figure]
Revised: B H YPERPARAMETERS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: C N ORMALIZED RESULTS INCLUDING D OGGO G OAL

Original: [Table] [Figure] [Table]
Revised: PointGoal PointGoal

Original: [Figure]
Revised: CarGoal

Original: [Table] [Figure] [Table]
Revised: PointButton PointPush

Original: [Figure]
Revised: DoggoGoal Mean E nv i r on m e n t 30 NormalizedRewardReturn 0 .0 .0 .0 .

Revised: 6 NormalizedCostReturn

Revised: TaskObjective

Revised: Safety Metric 0 .0 .1 .

Revised: N o r m a li

Revised: ze d M e t

Revised: r

Revised: i c

Revised: SafeSLAC SafeSLAC+

Revised: CPO TRPOLagrangian

Revised: LAMBDA

Revised: D Q UALITATIVE ANALYSIS OF THE RESULTS

Revised: [Figure]

Revised: Using the final policy computed by Safe SLAC, we collected a few trajectories in each environment to make a qualitative analysis.

Revised: Figure 6 shows one time-step for each environment.

Revised: For each environment, we have three videos showing the observation, the latent variable model reconstruction, and an overview of the environment.

Revised: We observe that Safe-SLAC can effectively solve the PointGoal1, PointGoal2, and CarGoal1 tasks.

Revised: In these environments, the agent has an exploration strategy of turning 180 degrees in place when the target is not visible.

Revised: In PointGoal2, we also see the agent carefully crossing spaces between hazards.

Revised: As expected, the agent mostly moves forward in these environments, while the agents using sensor observations also move backward.

Revised: In the PointButton1 environment, we observe that the agent can reach the target a few times but might fail when the goal is not in the field of vision of the agent.

Revised: In the DoggoGoal1 environment, the agent can also reach the goal.

Revised: However, the spaces between the hazards can be small compared to the robot’s size.

Revised: We suspect this makes this task considerably more complex than the remaining tasks in terms of safety.

Revised: Finally, in the PointPush1 environment, the agent goes to the box.

Revised: However, it does not move the box to the goal position.

Revised: This is due to the strong partial observability of this environment since the agent cannot observe the goal when the box is between the agent and the goal position.