Original: Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Revised: Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress
Revised: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps, which inthe real world limit the practicality of these algorithms as these can lead to potentially dangerous behavior.
Revised: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps, which inthe real world limit the practicality of these algorithms as these can lead to potentially dangerous behavior.

Original: Hence safe exploration is a critical issue in applying RLalgorithms in the real world.
Revised: Hence safe exploration is a critical issue in applying RLalgorithms in the real world.

Original: This problem has been recently well studied under the
Revised: This problem has been recently well studied under the

Original: Constrained Markov Decision Process (CMDP) Framework, where in addition tosingle-stage rewards, state transitions receive single-stage costs or penalties as well.
Revised: Constrained Markov Decision Process (CMDP) Framework, where in addition tosingle-stage rewards, state transitions receive single-stage costs or penalties as well.

Original: The prescribed cost functions are responsible for mapping undesirable behaviorat any given time-step to a scalar value.
Revised: The prescribed cost functions are responsible for mapping undesirable behaviorat any given time-step to a scalar value.

Original: Then we aim to find a feasible policythat maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.
Revised: Then we aim to find a feasible policythat maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.

Original: We propose an On-policy Model-based Safe Deep RL algorithm in which welearn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using Lagrangian Relaxation-based Proximal Policy
Revised: We propose an On-policy Model-based Safe Deep RL algorithm in which welearn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using Lagrangian Relaxation-based Proximal Policy

Original: Optimization.
Revised: Optimization.

Original: We use an ensemble of neural networks with different initializationsto tackle epistemic and aleatoric uncertainty issues faced during environment modellearning.
Revised: We use an ensemble of neural networks with different initializationsto tackle epistemic and aleatoric uncertainty issues faced during environment modellearning.

Original: We compare our approach with relevant model-free and model-basedapproaches in Constrained RL using the challenging Safe Reinforcement Learningbenchmark - the Open AI Safety Gym.
Revised: We compare our approach with relevant model-free and model-basedapproaches in Constrained RL using the challenging Safe Reinforcement Learningbenchmark - the Open AI Safety Gym.

Original: We demonstrate that our algorithm is moresample efficient and results in lower cumulative hazard violations as compared toconstrained model-free approaches.
Revised: We demonstrate that our algorithm is moresample efficient and results in lower cumulative hazard violations as compared toconstrained model-free approaches.

Original: Further, our approach shows better rewardperformance than other constrained model-based approach in the literature.
Revised: Further, our approach shows better rewardperformance than other constrained model-based approach in the literature.

Original:  Introduction
Revised:  Introduction

Original: Deep Reinforcement Learning has provided exceptional results both in discrete action domain (Mnihet al.
Revised: Deep Reinforcement Learning has provided exceptional results both in discrete action domain (Mnihet al.

Original: [2013]) and continuous action domain like locomotion tasks ([Haarnoja et al., 2018], Schulmanet al.
Revised: [2013]) and continuous action domain like locomotion tasks ([Haarnoja et al., 2018], Schulmanet al.

Original: [2017], Schulman et al.
Revised: [2017], Schulman et al.

Original: However, most of the RL algorithms perform significantnumber of random exploratory steps during learning as well as deployment which can lead to agentsperforming undesirable and hazardous behaviour.
Revised: However, most of the RL algorithms perform significantnumber of random exploratory steps during learning as well as deployment which can lead to agentsperforming undesirable and hazardous behaviour.

Original: This limits the application of RL algorithms in thereal world.
Revised: This limits the application of RL algorithms in thereal world.

Original: In scenarios like robot navigation (Han et al.
Revised: In scenarios like robot navigation (Han et al.

Original: [2018]), autonomous driving (Kendall et al.[2018]), healthcare (Yu et al.
Revised: [2018]), autonomous driving (Kendall et al.[2018]), healthcare (Yu et al.

Original: [2020]), etc., where RL has potential applications, unsafe behavior canhave hazardous consequences even on human life and property.
Revised: [2020]), etc., where RL has potential applications, unsafe behavior canhave hazardous consequences even on human life and property.

Original: In García et al.
Revised: In García et al.

Original: [2015], the authors have provided a comprehensive survey of several notions ofsafety and the associated problem formulations.
Revised: [2015], the authors have provided a comprehensive survey of several notions ofsafety and the associated problem formulations.

Original: In our work, we focus on a constraint-based notionof safety.
Revised: In our work, we focus on a constraint-based notionof safety.

Original: In Constraint-based RL, the goal is to maximize long-term expected reward returns andkeep the expected cost-returns below a prescribed threshold.
Revised: In Constraint-based RL, the goal is to maximize long-term expected reward returns andkeep the expected cost-returns below a prescribed threshold.

Original: This problem is known as the Safe
Revised: This problem is known as the Safe

Original: Exploration problem.
Revised: Exploration problem.

Original: In [Ray et al., 2019], the authors advocate that safety specifications should beseparate from task performance specifications.
Revised: In [Ray et al., 2019], the authors advocate that safety specifications should beseparate from task performance specifications.

Original: It also helps in formulating Safe Exploration as a
Revised: It also helps in formulating Safe Exploration as a

Original: Constrained Optimization Problem and methods used in the optimization literature [Bertsekas, 1996]can be used to solve this problem.
Revised: Constrained Optimization Problem and methods used in the optimization literature [Bertsekas, 1996]can be used to solve this problem.

Original: Constrained Markov Decision Process (CMDP)
Revised: Constrained Markov Decision Process (CMDP)

Original: [Altman, 1998]provides a framework to keep task performance specifications and safety specifications separately.
Revised: [Altman, 1998]provides a framework to keep task performance specifications and safety specifications separately.

Original: Here in addition to single-stage rewards, state transitions receive single-stage costs as well.
Revised: Here in addition to single-stage rewards, state transitions receive single-stage costs as well.

Original: Theprescribed cost functions are responsible for mapping undesirable behavior at any given time-step toa non-negative scalar value.
Revised: Theprescribed cost functions are responsible for mapping undesirable behavior at any given time-step toa non-negative scalar value.

Original: The existing model-free algorithms used in Constrained-RL setting suffer from low sample efficiencyin terms of environment interactions i.e., these algorithms require large number of environmentinteractions (to converge) that in turn would lead to high number of hazardous actions due to unsafeexploration.
Revised: The existing model-free algorithms used in Constrained-RL setting suffer from low sample efficiencyin terms of environment interactions i.e., these algorithms require large number of environmentinteractions (to converge) that in turn would lead to high number of hazardous actions due to unsafeexploration.

Original: This serves as the motivation for us to use a Model-based approach for Constrained-RL.
Revised: This serves as the motivation for us to use a Model-based approach for Constrained-RL.

Original: Our Contributions: We propose a simple and sample efficient model-based approach for Safe
Revised: Our Contributions: We propose a simple and sample efficient model-based approach for Safe

Original: Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem.
Revised: Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem.

Original: Wehighlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a wayto incorporate that in our setting.
Revised: Wehighlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a wayto incorporate that in our setting.

Original: We demonstrate that our approach is ∼ 3 − 4 times more sampleefficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulativehazard violations by ∼ 60% .
Revised: We demonstrate that our approach is ∼ 3 − 4 times more sampleefficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulativehazard violations by ∼ 60% .

Original: Our approach also outperforms other model-free Constrained Policy
Revised: Our approach also outperforms other model-free Constrained Policy

Original: Optimization (CPO)
Revised: Optimization (CPO)

Original: [Achiam et al., 2017] in terms of constraint satisfaction.
Revised: [Achiam et al., 2017] in terms of constraint satisfaction.

Original: We also compare ourapproach with model-based approach [Sikchi et al., 2021a] and we observe that our approach doesbetter in terms of reward performance and has competitive cost performance as well.
Revised: We also compare ourapproach with model-based approach [Sikchi et al., 2021a] and we observe that our approach doesbetter in terms of reward performance and has competitive cost performance as well.

Original:  Related Work
Revised:  Related Work

Original: Different notions of safety and their mathematical formulations are provided in García et al.
Revised: Different notions of safety and their mathematical formulations are provided in García et al.

Original: There are several works on the lines of formulating the Safe RL problem in the setting of CMDP.
Revised: There are several works on the lines of formulating the Safe RL problem in the setting of CMDP.

Original: In some of the early works, an actor-critic algorithm for CMDP under the long-run average costcriterion is proposed in Borkar [2005] that is however for the case of full state representations.
Revised: In some of the early works, an actor-critic algorithm for CMDP under the long-run average costcriterion is proposed in Borkar [2005] that is however for the case of full state representations.

Original: Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and
Revised: Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and

Original: Lakshmanan [2012] for the long-run average cost setting and in Bhatnagar [2010] for the infinitehorizon discounted cost scenario.
Revised: Lakshmanan [2012] for the long-run average cost setting and in Bhatnagar [2010] for the infinitehorizon discounted cost scenario.

Original: The procedure in the aforementioned references involved forming a
Revised: The procedure in the aforementioned references involved forming a

Original: Lagrangian by relaxing the constraints.
Revised: Lagrangian by relaxing the constraints.

Original: The algorithms in these papers are based on multi-timescalestochastic approximation with updates of the Lagrange parameter performed on the slowest timescale,the policy updates on the medium timescale and the updates of the value function (for a given policy)performed on the fast timescale.
Revised: The algorithms in these papers are based on multi-timescalestochastic approximation with updates of the Lagrange parameter performed on the slowest timescale,the policy updates on the medium timescale and the updates of the value function (for a given policy)performed on the fast timescale.

Original: In more recent work, in Achiam et al.
Revised: In more recent work, in Achiam et al.

Original: [2017], a trust region based constrained policy optimization(CPO) framework is proposed, which involved approximation of the problem using surrogate functionsfor both the objective and the constraints and included a projection step on policy parameters thatneeded backtracking line search, making it complicated and time-consuming.
Revised: [2017], a trust region based constrained policy optimization(CPO) framework is proposed, which involved approximation of the problem using surrogate functionsfor both the objective and the constraints and included a projection step on policy parameters thatneeded backtracking line search, making it complicated and time-consuming.

Original: CPO showed nearconstraint satisfaction in every iteration of the policy updates in standard Mujoco environmentsmodified for safe exploration (Achiam et al.
Revised: CPO showed nearconstraint satisfaction in every iteration of the policy updates in standard Mujoco environmentsmodified for safe exploration (Achiam et al.

Original: but didn’t yield a constraint satisfying policy inchallenging Safe RL benchmark Safety Gym (Ray et al.
Revised: but didn’t yield a constraint satisfying policy inchallenging Safe RL benchmark Safety Gym (Ray et al.

Original: Another work (Yu et al.
Revised: Another work (Yu et al.

Original: [2019])involved using surrogate functions for approximation of both objective and constraint functions.
Revised: [2019])involved using surrogate functions for approximation of both objective and constraint functions.

Original: Theirprocedure involved constructing a sequence of convex optimization problems for which they showedthat the sequence of stationary points converges to the stationary point of the original non-convexproblem.
Revised: Theirprocedure involved constructing a sequence of convex optimization problems for which they showedthat the sequence of stationary points converges to the stationary point of the original non-convexproblem.

Original: In Ray et al.
Revised: In Ray et al.

Original: [2019], Lagrangian relaxation of Constrained RL problem is used and combined with PPO(Schulman et al.
Revised: [2019], Lagrangian relaxation of Constrained RL problem is used and combined with PPO(Schulman et al.

Original: [2017]) to give a PPO-Lagrangian algorithm and with TRPO (Schulman et al.
Revised: [2017]) to give a PPO-Lagrangian algorithm and with TRPO (Schulman et al.

Original: [2015])to give a TRPO-Lagrangian algorithm.
Revised: [2015])to give a TRPO-Lagrangian algorithm.

Original: These algorithms were seen to outperform CPOAchiamet al.
Revised: These algorithms were seen to outperform CPOAchiamet al.

Original: [2017] in terms of constraint satisfaction on Safety Gym on several environments.
Revised: [2017] in terms of constraint satisfaction on Safety Gym on several environments.

Original: Also thesealgorithms are simpler to implement and tune.
Revised: Also thesealgorithms are simpler to implement and tune.

Original: Another Lagrangian-based method, see Tessler et al.[2018], used a penalized reward function for optimizing their agent and showed convergence tooptimal feasible policies using a two-timescale stochastic approximation scheme where the Lagrangemultiplier is updated on a slower timescale as compared to the policy parameters as was the casewith Borkar [2005], Bhatnagar [2010], Bhatnagar and Lakshmanan [2012].
Revised: Another Lagrangian-based method, see Tessler et al.[2018], used a penalized reward function for optimizing their agent and showed convergence tooptimal feasible policies using a two-timescale stochastic approximation scheme where the Lagrangemultiplier is updated on a slower timescale as compared to the policy parameters as was the casewith Borkar [2005], Bhatnagar [2010], Bhatnagar and Lakshmanan [2012].

Original: In Zhang et al.
Revised: In Zhang et al.

Original: [2020],the authors proposed a first order constrained policy optimization (FOCOPS) method that involvedsolving the optimization problem in non-parametric space and projecting it back into the parametricspace.
Revised: [2020],the authors proposed a first order constrained policy optimization (FOCOPS) method that involvedsolving the optimization problem in non-parametric space and projecting it back into the parametricspace.

Original: Approaches in Tessler et al.
Revised: Approaches in Tessler et al.

Original: [2018] and Zhang et al.
Revised: [2018] and Zhang et al.

Original: [2020] performed poorly on Safety Gym.
Revised: [2020] performed poorly on Safety Gym.

Original: In Stooke et al.
Revised: In Stooke et al.

Original: [2020], a PID-based approach to damp oscillations in Lagrangian methods is proposed, which is seen to minimize constraint violations.
Revised: [2020], a PID-based approach to damp oscillations in Lagrangian methods is proposed, which is seen to minimize constraint violations.

Original: In Wen and Topcu [2021], the Cross Entropy method(de Boer et al.
Revised: In Wen and Topcu [2021], the Cross Entropy method(de Boer et al.

Original: [2005]) is used for finding a safe policy and convergence using the ODE method isshown, however, empirical results are shown on a primitive and less challenging environment.
Revised: [2005]) is used for finding a safe policy and convergence using the ODE method isshown, however, empirical results are shown on a primitive and less challenging environment.

Original: In
Revised: In

Original: Dalal et al.
Revised: Dalal et al.

Original: [2018], the authors formulated a state-wise constrained policy optimization problemwhere at each transition a constraint needs to be satisfied and an analytical method for correcting theunsafe action using a safety layer trained using random exploration was proposed.
Revised: [2018], the authors formulated a state-wise constrained policy optimization problemwhere at each transition a constraint needs to be satisfied and an analytical method for correcting theunsafe action using a safety layer trained using random exploration was proposed.

Original: In Chow et al.[2018], the authors proposed constructing Lyapunov functions to guarantee safety of the behaviourpolicy under a CMDP framework.
Revised: In Chow et al.[2018], the authors proposed constructing Lyapunov functions to guarantee safety of the behaviourpolicy under a CMDP framework.

Original: In Chow et al.
Revised: In Chow et al.

Original: [2019], the above Lyapunov based method wasextended to continuous control
Revised: [2019], the above Lyapunov based method wasextended to continuous control

Original: but it’s performance in Safety Gym environment ([Ray et al.
Revised: but it’s performance in Safety Gym environment ([Ray et al.

Original: , 2019])was not good in terms of rewards obtained ([Sikchi et al., 2021b]).
Revised: , 2019])was not good in terms of rewards obtained ([Sikchi et al., 2021b]).

Original: In Liu et al.
Revised: In Liu et al.

Original: [2022], the methodinvolved using tangent space of the constraint manifold to learn a safe policy but is seen to be highlyspecific to the environment used.
Revised: [2022], the methodinvolved using tangent space of the constraint manifold to learn a safe policy but is seen to be highlyspecific to the environment used.

Original: For unconstrained state-of-the-art Model-based RL algorithms in Deisenroth and Rasmussen [2011],
Revised: For unconstrained state-of-the-art Model-based RL algorithms in Deisenroth and Rasmussen [2011],

Original: Luo et al.
Revised: Luo et al.

Original: [2018], Chua et al.
Revised: [2018], Chua et al.

Original: [2018], Kurutach et al.
Revised: [2018], Kurutach et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2015], a comparison ofthe empirical performance on Open AI Gym Brockman et al.
Revised: [2015], a comparison ofthe empirical performance on Open AI Gym Brockman et al.

Original: [2016] is shown in Wang et al.
Revised: [2016] is shown in Wang et al.

Original: Also, Sikchi et al.
Revised: Also, Sikchi et al.

Original: [2021a] propose augmenting planning trajectory with terminal value function toincorporate long-horizon reasoning in model-based methods, since the approaches in Deisenroth and
Revised: [2021a] propose augmenting planning trajectory with terminal value function toincorporate long-horizon reasoning in model-based methods, since the approaches in Deisenroth and

Original: Rasmussen [2011], Luo et al.
Revised: Rasmussen [2011], Luo et al.

Original: [2018], Chua et al.
Revised: [2018], Chua et al.

Original: [2018], Kurutach et al.
Revised: [2018], Kurutach et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2015]plan over a fixed and short horizon to avoid aggregation of error.
Revised: [2015]plan over a fixed and short horizon to avoid aggregation of error.

Original: This requires good approximationof both model as well as value function.
Revised: This requires good approximationof both model as well as value function.

Original: There are also several works that use model-based RL to tackle the problem of safety.
Revised: There are also several works that use model-based RL to tackle the problem of safety.

Original: In Liu et al.[2020], a model based approach is proposed to learn the system dynamics and cost model.
Revised: In Liu et al.[2020], a model based approach is proposed to learn the system dynamics and cost model.

Original: Then rollouts from the learned model are used to optimize policy using a modified cross-entropy based methodwhich involved sampling from a distribution of policies, sorting sample policies based on constraintfunctions and using them to update the policy distribution but their implementation involved a datacollection step using random policy for large number of episodes which itself is risky in real-worldscenarios.
Revised: Then rollouts from the learned model are used to optimize policy using a modified cross-entropy based methodwhich involved sampling from a distribution of policies, sorting sample policies based on constraintfunctions and using them to update the policy distribution but their implementation involved a datacollection step using random policy for large number of episodes which itself is risky in real-worldscenarios.

Original: In Cowen-Rivers et al.
Revised: In Cowen-Rivers et al.

Original: [2020], model dynamics is learned using PILCO (Deisenroth and
Revised: [2020], model dynamics is learned using PILCO (Deisenroth and

Original: Rasmussen [2011]) and instead of the discounted cost constraint function, conditional value at risk(CVaR) based constraint function is used (Uryasev and Rockafellar [2001], Chow and Ghavamzadeh[2014]).
Revised: Rasmussen [2011]) and instead of the discounted cost constraint function, conditional value at risk(CVaR) based constraint function is used (Uryasev and Rockafellar [2001], Chow and Ghavamzadeh[2014]).

Original: In Thomas et al.
Revised: In Thomas et al.

Original: [2022], penalized reward functions are used instead of a separate costfunction, then model of the environment is learned and Soft-actor critic algorithm (Haarnoja et al.[2018]) is used to optimize the policy.
Revised: [2022], penalized reward functions are used instead of a separate costfunction, then model of the environment is learned and Soft-actor critic algorithm (Haarnoja et al.[2018]) is used to optimize the policy.

Original: In this approach safe and unsafe states are also need to bespecified upfront.
Revised: In this approach safe and unsafe states are also need to bespecified upfront.

Original:  Background
Revised:  Background

Original: 3.1 Constrained Markov Decision Process (CMDP)
Revised: 3.1 Constrained Markov Decision Process (CMDP)

Original: A CMDP is denoted by the tuple ( S, A, R, C, γ, µ ) where S denotes the state space, A is the actionspace, R : S ×
Revised: A CMDP is denoted by the tuple ( S, A, R, C, γ, µ ) where S denotes the state space, A is the actionspace, R : S ×

Original: A × S → R is the single-stage reward function, C : S ×
Revised: A × S → R is the single-stage reward function, C : S ×

Original: A × S → R denotes theassociated single-stage cost function (we assume a single constraint function for simplicity here), γis the discount factor and µ signifies the initial state distribution.
Revised: A × S → R denotes theassociated single-stage cost function (we assume a single constraint function for simplicity here), γis the discount factor and µ signifies the initial state distribution.

Original: We assume that both R and C arenon-negative functions.
Revised: We assume that both R and C arenon-negative functions.

Original: By a policy π = ( π 0 , π 1 , . . . ) , we mean a decision rule for selecting actions.
Revised: By a policy π = ( π 0 , π 1 , . . . ) , we mean a decision rule for selecting actions.

Original: It is specified as follows:
Revised: It is specified as follows:

Original: For any k ≥ 0 and s ∈ S , π k ( s ) ∈ P (
Revised: For any k ≥ 0 and s ∈ S , π k ( s ) ∈ P (

Original: s )
Revised: s )

Original: is the probability distribution π k ( s ) △ =
Revised: is the probability distribution π k ( s ) △ =

Original: ( π k ( s, a
Revised: ( π k ( s, a

Original: , a ∈
Revised: , a ∈

Original: A s ))
Revised: A s ))

Original: where π k ( s, a
Revised: where π k ( s, a

Original: is the
Revised: is the

Original: probability
Revised: probability

Original: of picking action
Revised: of picking action

Original: a in state s at instant k under policy π .
Revised: a in state s at instant k under policy π .

Original: In theabove, A ( s ) is the set of feasible actions in state s and so A = ∪ s ∈ S A ( s ) .
Revised: In theabove, A ( s ) is the set of feasible actions in state s and so A = ∪ s ∈ S A ( s ) .

Original: Such a policy is also oftenreferred to as a randomized policy.
Revised: Such a policy is also oftenreferred to as a randomized policy.

Original: A stationary policy is a randomized policy as above except withπ k = π l , ∀ k ̸ = l .
Revised: A stationary policy is a randomized policy as above except withπ k = π l , ∀ k ̸ = l .

Original: Thus, a stationary policy selects actions according to a given distribution regardlessof the instant when an action is chosen according to the given policy.
Revised: Thus, a stationary policy selects actions according to a given distribution regardlessof the instant when an action is chosen according to the given policy.

Original: By an abuse of notation, wedenote a stationary policy as π itself.
Revised: By an abuse of notation, wedenote a stationary policy as π itself.

Original: We shall consider here a class of stationary policies π θ parameterized by a parameter θ .
Revised: We shall consider here a class of stationary policies π θ parameterized by a parameter θ .

Original: Our objectivefunction is defined via the infinite horizon discounted reward criterion where for a given π θ we have
Revised: Our objectivefunction is defined via the infinite horizon discounted reward criterion where for a given π θ we have

Original: [Equation]
Revised: [Equation]

Original: The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:
Revised: The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:

Original: [Equation]
Revised: [Equation]

Original: Then J R ( π θ ) , J C ( π θ ) ∈ R .
Revised: Then J R ( π θ ) , J C ( π θ ) ∈ R .

Original: Let d > 0 denote a prescribed threshold below which we want J C ( π θ )to lie.
Revised: Let d > 0 denote a prescribed threshold below which we want J C ( π θ )to lie.

Original: The constrained optimization problem then is the following:
Revised: The constrained optimization problem then is the following:

Original: [Equation]
Revised: [Equation]

Original: A parameter θ will be called a feasible point if the cost constraint is satisfied for θ , i.e., J C ( π θ ) ≤ d .
Revised: A parameter θ will be called a feasible point if the cost constraint is satisfied for θ , i.e., J C ( π θ ) ≤ d .

Original: 3.2 Lagrangian Relaxation based Proximal Policy Optimization
Revised: 3.2 Lagrangian Relaxation based Proximal Policy Optimization

Original: The Lagrangian of the constrained optimization problem (3) can be written as follows:
Revised: The Lagrangian of the constrained optimization problem (3) can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R + is the Lagrange multiplier and is a positive real number.
Revised: where λ ∈ R + is the Lagrange multiplier and is a positive real number.

Original: In terms of the Lagrangian,the goal is to find a tuple ( θ ∗ , λ ∗ ) of the policy and Lagrange parameter such that
Revised: In terms of the Lagrangian,the goal is to find a tuple ( θ ∗ , λ ∗ ) of the policy and Lagrange parameter such that

Original: [Equation]
Revised: [Equation]

Original: Solving the max-min problem as above is equivalent to finding a global optimal saddle point ( θ ∗ , λ ∗ )such that ∀ ( θ, λ ) , the following holds:
Revised: Solving the max-min problem as above is equivalent to finding a global optimal saddle point ( θ ∗ , λ ∗ )such that ∀ ( θ, λ ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: We assume that θ refers to the parameter of a Deep Neural Network, hence finding such a globallyoptimal saddle point is computationally hard.
Revised: We assume that θ refers to the parameter of a Deep Neural Network, hence finding such a globallyoptimal saddle point is computationally hard.

Original: So our aim is to find a locally optimal saddle pointwhich satisfies (6) in a local neighbourhood H ϵ 1 ,ϵ 2 which is defined as follows:
Revised: So our aim is to find a locally optimal saddle pointwhich satisfies (6) in a local neighbourhood H ϵ 1 ,ϵ 2 which is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: for some ϵ 1 , ϵ 2 > 0 .
Revised: for some ϵ 1 , ϵ 2 > 0 .

Original: Assuming that L ( θ, λ ) is known for every ( θ, λ ) tuple, a gradient searchprocedure for finding a local ( θ ∗ , λ ∗ ) tuple would be the following:
Revised: Assuming that L ( θ, λ ) is known for every ( θ, λ ) tuple, a gradient searchprocedure for finding a local ( θ ∗ , λ ∗ ) tuple would be the following:

Original: [Equation]
Revised: [Equation]

Original: Here [ x ] + denotes max(0 , x ) .
Revised: Here [ x ] + denotes max(0 , x ) .

Original: This operator ensures that the Lagrange multiplier remains nonnegative after each update.
Revised: This operator ensures that the Lagrange multiplier remains nonnegative after each update.

Original: In (9)-(11), η 1 ( n ) , η 2 ( n ) > 0 ∀ n are certain prescribed step-size schedules.
Revised: In (9)-(11), η 1 ( n ) , η 2 ( n ) > 0 ∀ n are certain prescribed step-size schedules.

Original: We assume that the step-sizes η 1 ( n ) , η 2 ( n ) , n ≥ 0 satisfy the regular step-size conditions.
Revised: We assume that the step-sizes η 1 ( n ) , η 2 ( n ) , n ≥ 0 satisfy the regular step-size conditions.

Original: Thus, fori = 1 , 2 , (cid:88)
Revised: Thus, fori = 1 , 2 , (cid:88)

Original: k
Revised: k

Original: η i ( n ) = ∞ , (cid:88) k
Revised: η i ( n ) = ∞ , (cid:88) k

Original: η 2 i ( n ) < ∞ .
Revised: η 2 i ( n ) < ∞ .

Original: Note however that J R ( π θ ) and J C ( π θ ) as specified in(1)-(2) are not a priori known quantities and need to be estimated.
Revised: Note however that J R ( π θ ) and J C ( π θ ) as specified in(1)-(2) are not a priori known quantities and need to be estimated.

Original: We discuss this is detail below.
Revised: We discuss this in detail below.

Original: 3.2.1 Estimation
Revised: 3.2.1 Estimation

Original: We run each episode for T time steps in our experiments.
Revised: We run each episode for T time steps in our experiments.

Original: Let r t +1 ≡ R ( s t , a t , s t +1 ) and c t +1 ≡
Revised: Let r t +1 ≡ R ( s t , a t , s t +1 ) and c t +1 ≡

Original: C ( s t , a t , s t +1 ) , respectively, for simplicity.
Revised: C ( s t , a t , s t +1 ) , respectively, for simplicity.

Original: For each sample path we would have both a rewardreturn as well as a cost return.
Revised: For each sample path we would have both a rewardreturn as well as a cost return.

Original: Let ˆ R t (resp.
Revised: Let ˆ R t (resp.

Original: ˆ C t ) be the reward-to-go (resp.
Revised: ˆ C t ) be the reward-to-go (resp.

Original: cost-to-go) estimate.
Revised: cost-to-go) estimate.

Original: Wecompute ˆ R t and ˆ C t according to: ˆ R t = (cid:80) T − t − 1 k
Revised: Wecompute ˆ R t and ˆ C t according to: ˆ R t = (cid:80) T − t − 1 k

Original: =
Revised: =

Original: γ
Revised: γ

Original: k r t + k +1 , ˆ C t = (cid:80) T − t − 1 k = γ k c t + k +1 , respectively.
Revised: k r t + k +1 , ˆ C t = (cid:80) T − t − 1 k = γ k c t + k +1 , respectively.

Original: We use a neural network parameterized by ψ r to estimate reward signal based value function V
Revised: We use a neural network parameterized by ψ r to estimate reward signal based value function V

Original: Rψ r anda neural network parameterized by ψ c to estimate a cost signal based value function V Cψ c .
Revised: Rψ r anda neural network parameterized by ψ c to estimate a cost signal based value function V Cψ c .

Original: We run oursimulations on N parallel workers and then sample a mini-batch of size M ≤ NT [Schulman et al.,
Revised: We run oursimulations on N parallel workers and then sample a mini-batch of size M ≤ NT [Schulman et al.,

Original: We use mean-squared loss to estimate value functions on sampled mini-batches as follows:
Revised: We use mean-squared loss to estimate value functions on sampled mini-batches as follows:

Original: [Equation]
Revised: [Equation]

Original: Let A Rt , t ≥ 0 , and A Ct , t ≥ 0 , respectively, denote the advantage estimates w.r.t reward and costvalue functions on the sample path.
Revised: Let A Rt , t ≥ 0 , and A Ct , t ≥ 0 , respectively, denote the advantage estimates w.r.t reward and costvalue functions on the sample path.

Original: We compute them using Generalized Advantage Estimation[Schulman et al., 2016] to balance the bias and variance tradeoff of advantage estimates.
Revised: We compute them using Generalized Advantage Estimation[Schulman et al., 2016] to balance the bias and variance tradeoff of advantage estimates.

Original: We can havea range of advantage estimates as under.
Revised: We can havea range of advantage estimates as under.

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: for k > 2 .
Revised: for k > 2 .

Original: The advantage estimate in (14) will have high bias but low variance while estimates in (16)with a higher value of k generally have high variance but low bias.
Revised: The advantage estimate in (14) will have high bias but low variance while estimates in (16)with a higher value of k generally have high variance but low bias.

Original: Let δ Rt
Revised: Let δ Rt

Original: =
Revised: =

Original: r t +1 + V
Revised: r t +1 + V

Original: Rψ r ( s t +1 ) −
Revised: Rψ r ( s t +1 ) −

Original: V
Revised: V

Original: Rψ r ( s t ) and δ C t =
Revised: Rψ r ( s t ) and δ C t =

Original: c t +1 + V Cψ c ( s t +1 ) − V Cψ c ( s t ) , respectively, denote the reward and cost temporaldifferences.
Revised: c t +1 + V Cψ c ( s t +1 ) − V Cψ c ( s t ) , respectively, denote the reward and cost temporaldifferences.

Original: Let ¯ λ be a parameter which adjusts the bias-variance tradeoff.
Revised: Let ¯ λ be a parameter which adjusts the bias-variance tradeoff.

Original: Generalized Advantage
Revised: Generalized Advantage

Original: Estimate [Schulman et al., 2016] for A Rt ,
Revised: Estimate [Schulman et al., 2016] for A Rt ,

Original: A Ct is then given by
Revised: A Ct is then given by

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: respectively.
Revised: respectively.

Original: Now we use PPO clipped objectives [Schulman et al., 2017] for estimation of
Revised: Now we use PPO clipped objectives [Schulman et al., 2017] for estimation of

Original: J R ( π θ ) , J C ( π θ ) as follows:
Revised: J R ( π θ ) , J C ( π θ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where r t ( θ ) =
Revised: where r t ( θ ) =

Original: π θ ( a t | s t ) π θold ( a t | s t ) is the ratio of probability of selecting action a t in state s t under parameterθ as opposed to θ old .
Revised: π θ ( a t | s t ) π θold ( a t | s t ) is the ratio of probability of selecting action a t in state s t under parameterθ as opposed to θ old .

Original: Further, A Rt and A Ct are the estimated advantages based on the reward and costreturns, respectively, by time t (see above) and ϵ is the clip-ratio which clips r t ( θ ) to (1 − ϵ ) if it isless than (1 − ϵ ) and clips to (1 + ϵ ) if it is greater than (1 + ϵ ) .
Revised: Further, A Rt and A Ct are the estimated advantages based on the reward and costreturns, respectively, by time t (see above) and ϵ is the clip-ratio which clips r t ( θ ) to (1 − ϵ ) if it isless than (1 − ϵ ) and clips to (1 + ϵ ) if it is greater than (1 + ϵ ) .

Original: This algorithm restricts the policyparameters to not change significantly between two iterations which helps in avoiding divergence.
Revised: This algorithm restricts the policyparameters to not change significantly between two iterations which helps in avoiding divergence.

Original: This approach is also known as PPO-Lagrangian [Ray et al., 2019].
Revised: This approach is also known as PPO-Lagrangian [Ray et al., 2019].

Original: 3.3 Model-based Constrained RL
Revised: 3.3 Model-based Constrained RL

Original: We formulate a Constrained RL problem (21) using Model-based framework as follows:
Revised: We formulate a Constrained RL problem (21) using Model-based framework as follows:

Original: [Equation]
Revised: [Equation]

Original: , where (21)
Revised: , where (21)

Original: [Equation]
Revised: [Equation]

Original: In the above, P α ( . | s t , a t ) is an α -parameterized environment model, d i is a human prescribed safetythreshold for the i th constraint and Π θ is the set of all θ − parameterized stationary polices.
Revised: In the above, P α ( . | s t , a t ) is an α -parameterized environment model, d i is a human prescribed safetythreshold for the i th constraint and Π θ is the set of all θ − parameterized stationary polices.

Original: Notethat we assume initial state s 0 is sampled from true initial state distribution µ and then s t +1 ∼
Revised: Notethat we assume initial state s 0 is sampled from true initial state distribution µ and then s t +1 ∼

Original: P α ( .
Revised: P α ( .

Original: | s t , a t ) , ∀ t > 0 .
Revised: | s t , a t ) , ∀ t > 0 .

Original: We would use approximation of environment P α to create ‘imaginary’ rolloutsto estimate reward and cost returns required for policy optimization algorithms.
Revised: We would use approximation of environment P α to create ‘imaginary’ rolloutsto estimate reward and cost returns required for policy optimization algorithms.

Original:  Challenges in Environment Model Learning
Revised:  Challenges in Environment Model Learning

Original: In this section we discuss the challenges that commonly arise due to model learning in RL.
Revised: In this section we discuss the challenges that commonly arise due to model learning in RL.

Original: We furtherhighlight the challenge that arises from using environment model approximation in Safe RL settings.
Revised: We furtherhighlight the challenge that arises from using environment model approximation in Safe RL settings.

Original: Handling aleatoric and epistemic uncertainties : Aleatoric Uncertainty refers to thenotion of natural randomness in the system which leads to variability in outcomes of anexperiment.
Revised: Handling aleatoric and epistemic uncertainties : Aleatoric Uncertainty refers to thenotion of natural randomness in the system which leads to variability in outcomes of anexperiment.

Original: This uncertainty is irreducible because it is a natural property of the system.
Revised: This uncertainty is irreducible because it is a natural property of the system.

Original: Hence in such cases, giving measure of uncertainty in model’s prediction is a good practice.
Revised: Hence in such cases, giving measure of uncertainty in model’s prediction is a good practice.

Original: In Lakshminarayanan et al.
Revised: In Lakshminarayanan et al.

Original: 2017], Chua et al.
Revised: 2017], Chua et al.

Original: 2018] uncertainty-aware neural networksare used which gives an idea about uncertainty
Revised: 2018] uncertainty-aware neural networksare used which gives an idea about uncertainty

Original: in prediction as
Revised: in prediction as

Original: well
Revised: well

Original: They learn
Revised: They learn

Original: Gaussian
Revised: Gaussian

Original: distribution parameterized by neural networks.
Revised: distribution parameterized by neural networks.

Original: Epistemic Uncertainty refers to the notion of
Revised: Epistemic Uncertainty refers to the notion of

Original: lack of sufficient knowledge in the model as a result of which the model does not generalize.
Revised: lack of sufficient knowledge in the model as a result of which the model does not generalize.

Original: In Lakshminarayanan et al.
Revised: In Lakshminarayanan et al.

Original: ], an ensemble of uncertainty aware neural networks with
Revised: ], an ensemble of uncertainty aware neural networks with

Original: different initialization is proposed to reduce epistemic uncertainty for fixed data
Revised: different initialization is proposed to reduce epistemic uncertainty for fixed data

Original: For learning environment model we also use an ensemble of n neural networks withdifferent initialization.
Revised: For learning environment model we also use an ensemble of n neural networks withdifferent initialization.

Original: Each neural network’s output parameterizes a multivariate normaldistribution with diagonal covariance matrix.
Revised: Each neural network’s output parameterizes a multivariate normaldistribution with diagonal covariance matrix.

Original: Suppose the i th neural network in the ensembleis parameterized by α i and the mean and standard deviation outputs are µ α i and σ α irespectively.
Revised: Suppose the i th neural network in the ensembleis parameterized by α i and the mean and standard deviation outputs are µ α i and σ α irespectively.

Original: Recall now that if a random vector X ∈ R d is distributed according to themultivariate normal distribution parameterized by ( µ, Σ d × d ) where µ ∈ R d is the meanvector and Σ d × d is a d × d covariance matrix, then the probability density of X is definedas, P ( X )
Revised: Recall now that if a random vector X ∈ R d is distributed according to themultivariate normal distribution parameterized by ( µ, Σ d × d ) where µ ∈ R d is the meanvector and Σ d × d is a d × d covariance matrix, then the probability density of X is definedas, P ( X )

Original: =
Revised: =

Original: 1 (2 π ) d/ 2 | Σ | − 1 / 2 exp ( − 12 ( X − µ ) ′
Revised: 1 (2 π ) d/ 2 | Σ | − 1 / 2 exp ( − 12 ( X − µ ) ′

Original: Σ − 1 ( X − µ ))
Revised: Σ − 1 ( X − µ ))

Original: As a choice of loss function weuse the negative log-likelihood loss for minimization (i.e., minimizing negative log of P(X)).
Revised: As a choice of loss function weuse the negative log-likelihood loss for minimization (i.e., minimizing negative log of P(X)).

Original: For the i th neural network parameterized by α i , the loss function L ( α i ) is given as follows:
Revised: For the i th neural network parameterized by α i , the loss function L ( α i ) is given as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ α i ( s t , a t ) is the mean vector output of the i th neural network and
Revised: where µ α i ( s t , a t ) is the mean vector output of the i th neural network and

Original: Σ α i ( s t , a t ) is thecovariance matrix which is assumed to be a diagonal matrix.
Revised: Σ α i ( s t , a t ) is thecovariance matrix which is assumed to be a diagonal matrix.

Original: Aggregation
Revised: Aggregation

Original: of
Revised: of

Original: Error :
Revised: Error :

Original: In model-based RL, as we move forward along the horizon, the
Revised: In model-based RL, as we move forward along the horizon, the

Original: error due to approximation
Revised: error due to approximation

Original: starts aggregating and predictions
Revised: starts aggregating and predictions

Original: from the
Revised: from the

Original: approximated modeltend to diverge significantly from the true model
Revised: approximated modeltend to diverge significantly from the true model

Original: In order to tackle this problem, most ofthe model-based RL approaches [Deisenroth and Rasmussen, 2011, Kurutach et al., 2018,
Revised: In order to tackle this problem, most ofthe model-based RL approaches [Deisenroth and Rasmussen, 2011, Kurutach et al., 2018,

Original: Janner et al., 2019] use shorter (or truncated) horizon during the policy optimization phaseand achieve similar performance as Model-Free RL approaches.
Revised: Janner et al., 2019] use shorter (or truncated) horizon during the policy optimization phaseand achieve similar performance as Model-Free RL approaches.

Original: We use truncated horizonin our approach.
Revised: We use truncated horizonin our approach.

Original: Implication of using truncated horizon in Constrained RL : When we use truncated
Revised: Implication of using truncated horizon in Constrained RL : When we use truncated

Original: horizon in Constrained RL, it leads to underestimation of cost returns (23) under the current
Revised: horizon in Constrained RL, it leads to underestimation of cost returns (23) under the current

Original: policy and we use the prescribed constraints to threshold the returns obtained.
Revised: policy and we use the prescribed constraints to threshold the returns obtained.

Original: This can
Revised: This can

Original: lead to constraint violations in the real-environment where the cost objective is based on the
Revised: lead to constraint violations in the real-environment where the cost objective is based on the

Original: infinite horizon cost return.
Revised: infinite horizon cost return.

Original: We propose a hyperparameter-based approach to deal with this
Revised: We propose a hyperparameter-based approach to deal with this

Original: problem in the next section
Revised: problem in the next section

Original:  Model-based PPO Lagrangian
Revised:  Model-based PPO Lagrangian

Original: We propose a Model-based algorithm derived from relaxing the Lagrangian (see Algorithm 1) whichalleviates the problem of requirement of a large number of samples as in model-free Lagrangianbased approaches and as a consequence also decreases the cumulative hazard violations.
Revised: We propose a Model-based algorithm derived from relaxing the Lagrangian (see Algorithm 1) whichalleviates the problem of requirement of a large number of samples as in model-free Lagrangianbased approaches and as a consequence also decreases the cumulative hazard violations.

Original: It is difficultto evaluate policy without interacting with the real environment accurately.
Revised: It is difficultto evaluate policy without interacting with the real environment accurately.

Original: 1: Input: Initialize actor neural net parameter θ 0 , critic parameters ψ r 0 , ψ c 0 , ensemble models [ P α i ] ni =1 , Lagrange parameter λ 0 ≥ 0 , cost threshold =
Revised: 1: Input: Initialize actor neural net parameter θ 0 , critic parameters ψ r 0 , ψ c 0 , ensemble models [ P α i ] ni =1 , Lagrange parameter λ 0 ≥ 0 , cost threshold =

Original: d , Environment Horizon = T , Model
Revised: d , Environment Horizon = T , Model

Original: Horizon =
Revised: Horizon =

Original: H 2: for i = 1 , . .
Revised: H 2: for i = 1 , . .

Original: , N training epochs do 3:
Revised: , N training epochs do 3:

Original: Collect data tuples ( s t , a t , s t +1 ) , for the i th ensemble using policy π θ i , i = 1 , . .
Revised: Collect data tuples ( s t , a t , s t +1 ) , for the i th ensemble using policy π θ i , i = 1 , . .

Original: . , n , in the environment for T time steps over multiple episodes | E | 4:
Revised: . , n , in the environment for T time steps over multiple episodes | E | 4:

Original: Train [ P α i ] ni =1 by minimizing (24) w.r.t α i , ∀ i = 1 to n 5: while Performance ratio > 70% do 6: s 0 ∼ µ  7:
Revised: Train [ P α i ] ni =1 by minimizing (24) w.r.t α i , ∀ i = 1 to n 5: while Performance ratio > PR Threshold do 6: s 0 ∼ µ { Note: For first pass we use mix of real data and imaginary rollout data. (See Appendix A) } 7:

Original: Collect data rollouts as a t ∼ π θ i ( .
Revised: Collect data rollouts as a t ∼ π θ i ( .

Original: | s t ) , s t ∼ P α q ( .
Revised: | s t ) , s t ∼ P α q ( .

Original: | s t , a t ) (At each time step ’ q ’ is randomly selected from 1 , . . . , n ) for H time steps ( H < T ) 8:
Revised: | s t , a t ) (At each time step ’ q ’ is randomly selected from 1 , . . . , n ) for H time steps ( H < T ) 8:

Original: Compute J Csample ( π θ t ) = 

Revised: Compute J Csample ( π θ t ) =

Original: | E | (cid:80) Hp =1 γ p C
Revised: | E | (cid:80) Hp =1 γ p C

Original: ( s t , a t ) where | E | is the number of episodes 9:
Revised: ( s t , a t ) where | E | is the number of episodes 9:

Original: Compute advantage, cost-advantage using (17) and (18) respectively 10:
Revised: Compute advantage, cost-advantage using (17) and (18) respectively 10:

Original: Update λ by replacing J C ( π θ ) with J Csample ( π θ i ) and using an appropriate value of β in (26).
Revised: Update λ by replacing J C ( π θ ) with J Csample ( π θ i ) and using an appropriate value of β in (26).

Original: { Multiple gradient updates for actor and critic } 11: for k = 1 , . .
Revised: { Multiple gradient updates for actor and critic } 11: for k = 1 , . .

Original: , K do 12:
Revised: , K do 12:

Original: Compute J R ( π θ k ) , J C ( π θ k ) as in (19) and (20) respectively.
Revised: Compute J R ( π θ k ) , J C ( π θ k ) as in (19) and (20) respectively.

Original: Update parameters θ k using (9) 14:
Revised: Update parameters θ k using (9) 14:

Original: Update critic parameters ψ r and ψ c minimizing (12) and (13) respectively.
Revised: Update critic parameters ψ r and ψ c minimizing (12) and (13) respectively.

Original: 15: end for 16:
Revised: 15: end for 16:

Original: Compute Performance Ratio (PR) using (25) 17: end while 18: end for
Revised: Compute Performance Ratio (PR) using (25) 17: end while 18: end for

Original: For this we compute the Performance Ratio (PR) metric using ensemble models (see Kurutach et al.[2018]) that is defined as follows:
Revised: For this we compute the Performance Ratio (PR) metric using ensemble models (see Kurutach et al.[2018]) that is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where ζ R ( α i , θ t ) = (cid:80) Tt =

Revised: where ζ R ( α i , θ t ) = (cid:80) Tt =

Original: γ t R ( s t , a t , s t +1 ) , s 0 ∼ µ and ∀ t ≥ 0 : s t +1 ∼ P
Revised: γ t R ( s t , a t , s t +1 ) , s 0 ∼ µ and ∀ t ≥ 0 : s t +1 ∼ P

Original: α i ( ·| s t , a t ) , a t ∼π θ t ( ·| s t ) .
Revised: α i ( ·| s t , a t ) , a t ∼π θ t ( ·| s t ) .

Original: This measures the ratio of the number of models in which policy is improved to the totalnumber of models in ensemble ( n ) .
Revised: This measures the ratio of the number of models in which policy is improved to the totalnumber of models in ensemble ( n ) .

Original: If PR > 70% , we continue training using the same model, if notthen we break and re-train our environment model on data collected from the new update policy.
Revised: If PR > PR threshold , we continue training using the samemodel, if not then we break and re-train our environment model on data collected from the newupdate policy.

Original: Another challenge that we encounter is the underestimation of J C ( π θ ) because of using truncatedhorizon (of length H in step 7 of Algorithm 1) to reduce the aggregation of error.
Revised: Another challenge that we encounter is the underestimation of J C ( π θ ) because of using truncatedhorizon (of length H in step 7 of Algorithm 1) to reduce the aggregation of error.

Original: This can potentially lead to constraint violating policies.
Revised: This can potentially lead to constraint violating policies.

Original: So we need to make the safety threshold( d ) stricter.
Revised: So we need to make the safety threshold( d ) stricter.

Original: It might seem that changing this threshold ( d ) proportional to the truncated horizon,i.e., setting d :=
Revised: It might seem that changing this threshold ( d ) proportional to the truncated horizon,i.e., setting d :=

Original: d ∗ HT might work better where H is the truncated horizon and T is the original environment horizon, but we found that this leads to constraint violations as well because we arelearning policy using data from the approximated environment model that would bring in errors.
Revised: d ∗ HT might work better where H is the truncated horizon and T is the originalenvironment horizon, but we found that this leads to constraint violations as well because we arelearning policy using data from the approximated environment model that would bring in errors.

Original: Hence the cost estimate of the policy (Step 8 in Algorithm 1) is error-prone as well.
Revised: Hence the cost estimate of the policy (Step 8 in Algorithm 1) is error-prone as well.

Original: To tacklethis issue, we make safety threshold stricter using a hyperparameter 0 ≤ β < 1 by modifying the
Revised: To tacklethis issue, we make safety threshold stricter using a hyperparameter 0 ≤ β < 1 by modifying the

Original: Lagrange multiplier update as follows:
Revised: Lagrange multiplier update as follows:

Original: [Equation]
Revised: [Equation]

Original: The variation of expected cost returns and reward returns with respect to β is shown in Figure 1 on
Revised: The variation of expected cost returns and reward returns with respect to β is shown in Figure 1 on

Original: Safety gym PointGoal1 [Ray et al., 2019] environment.
Revised: Safety gym PointGoal [Ray et al., 2019] environment.

Original: We can observe that as we reduce β , theexpected cost returns reduce because cost limit becomes stricter but choosing too small a β leads tolow reward returns as well.
Revised: We can observe that as we reduce β , theexpected cost returns reduce because cost limit becomes stricter but choosing too small a β leads tolow reward returns as well.

Original:  Experimental Details and Results
Revised:  Experimental Details and Results

Original: We test our approach on Safety Gym environments 1 - PointGoal1 and CarGoal1 with modifiedstate representations as used in other model-based Safe RL approaches [Liu et al., 2020, Sikchiet al., 2021a] which is more helpful in model-learning.
Revised: We test our approach on Safety Gym environments 1 - PointGoal and CarGoal with modified staterepresentations as used in other model-based Safe RL approaches [Liu et al., 2020, Sikchi et al. ,2021a] which is more helpful in model-learning.

Original: We increase the difficulty of PointGoal1 and CarGoal1 environments by increasing the number of hazards from 10 to 15.
Revised: We increase the difficulty of PointGoal and CarGoalenvironments by increasing the number of hazards from 10 to 15.

Original: In both environments,the aim of robots is to reach the goal position and have as few collisions with hazards as possible.
Revised: In both environments, the aimof robots is to reach the goal position and have as few collisions with hazards as possible.

Original: We compare our approach (MBPPO-Lagrangian) with Unconstrained PPO [Schulman et al.
Revised: Wecompare our approach (MBPPO-Lagrangian) with Unconstrained PPO [Schulman et al.

Original: , 2017],model-free Safe RL approaches including Constrained Policy Optimization [Achiam et al., 2017],
Revised: , 2017],model-free Safe RL approaches including Constrained Policy Optimization [Achiam et al., 2017],

Original: PPO-Lagrangian [Ray et al., 2019] and the model-based approach – safe-LOOP from Sikchi et al.[2021a].
Revised: PPO-Lagrangian [Ray et al., 2019] and the model-based approach – safe-LOOP from Sikchi et al.[2021a].

Original: The code for our approach is available here 2 .
Revised: The code for our approach is available here 2 .

Original: We run each algorithm with 8 random seeds for 450Kenvironment interactions.
Revised: We run each algorithm with 8 random seeds for 450Kenvironment interactions.

Original: The hyper-parameter settings and other experimental details are given in
Revised: The hyper-parameter settings and other experimental details are given in

Original: Appendix A.
Revised: Appendix A.

Original: The performance of our approach and baseline algorithms is shown in Figure 2 for
Revised: The performance of our approach and baseline algorithms is shown in Figure 2 for

Original: PointGoal environment and Figure 3 for CarGoal environment.
Revised: PointGoal environment and Figure 3 for CarGoal environment.

Original: The left part of the plots represent Average Episode Return (average of the episode rewards) ony-axis and total environment interactions on the x-axis.
Revised: The left part of the plots represent Average Episode Return (average of the episode rewards) ony-axis and total environment interactions on the x-axis.

Original: The dashed lines represent performance ofthe PPO-Lagrangian (in black) at convergence which is around 2 million environment interactions.
Revised: The dashed lines represent performance ofthe PPO-Lagrangian (in black) at convergence which is around 2 million environment interactions.

Original: The right part of the plots show Average Episode Cost (average of the episode costs) on y-axis andtotal environment interactions on the x-axis.
Revised: The right part of the plots show Average Episode Cost (average of the episode costs) on y-axis andtotal environment interactions on the x-axis.

Original: The red dashed line in this plot represents cost limitof 18.
Revised: The red dashed line in this plot represents cost limitof 18.

Original: From the plots we can observe that our approach (MBPPO-Lagrangian) converges to thesame level of reward performance as PPO - Lagrangian in just 450K environment interactions andoutperforms the model-based safe-LOOP algorithm
Revised: From the plots we can observe that our approach (MBPPO-Lagrangian) converges to thesame level of reward performance as PPO - Lagrangian in just 450K environment interactions andoutperforms the model-based safe-LOOP algorithm

Original: [Sikchi et al., 2021a].
Revised: [Sikchi et al., 2021a].

Original: Also our approach givesconstraint adhering policies in both the tasks.
Revised: Also our approach givesconstraint adhering policies in both the tasks.

Original: We also measure cumulative hazard violations thatoccur till convergence for PPO-Lagrangian Ray et al.
Revised: We also measure cumulative hazard violations thatoccur till convergence for PPO-Lagrangian Ray et al.

Original: [2019], safe-LOOP
Revised: [2019], safe-LOOP

Original: [Sikchi et al., 2021a] as
Revised: [Sikchi et al., 2021a] as

Original: Cumulative V iolations = (cid:80)
Revised: Cumulative V iolations = (cid:80)

Original: Till convergence [ 1 ( C ( s t , a t ) =
Revised: Till convergence [ 1 ( C ( s t , a t ) =

Original: = 1)] .
Revised: = 1)] .

Original: We use rliable library (See Sec4.3 in Agarwal et al.
Revised: We use rliable library (See Sec4.3 in Agarwal et al.

Original: [2021] for more details) for plotting 95% confidence intervals for cumulativeviolations in the right part of Figure 4 (normalized by cumulative violations in unconstrained PPOin respective tasks) and final policy reward performance in the left part of Figure 4 (normalizedby reward return of the final policy in unconstrained PPO in respective tasks) using mean, medianand inter-quartile mean as aggregate estimates for our approach, PPO-Lagrangian and safe-LOOPbecause these approaches only give constraint satisfying policies at convergence.
Revised: [2021] for more details) for plotting 95% confidence intervals for cumulativeviolations in the right part of Figure 4 (normalized by cumulative violations in unconstrained PPOin respective tasks) and final policy reward performance in the left part of Figure 4 (normalizedby reward return of the final policy in unconstrained PPO in respective tasks) using mean, medianand inter-quartile mean as aggregate estimates for our approach, PPO-Lagrangian and safe-LOOPbecause these approaches only give constraint satisfying policies at convergence.

Original: From figure 4we can observe that our approach is competitive to the model-based safe-LOOP approach [Sikchiet al., 2021a] in terms of cumulative constraint violations but achieves better reward performance atconvergence.
Revised: From figure 4we can observe that our approach is competitive to the model-based safe-LOOP approach [Sikchiet al., 2021a] in terms of cumulative constraint violations but achieves better reward performance atconvergence.

Original:  Conclusions
Revised:  Conclusions

Original: We presented a sample efficient approach for Safe Reinforcement Learning and compared ourapproach with relevant model-free and model-based baselines [Achiam et al., 2017, Sikchi et al.
Revised: We presented a sample efficient approach for Safe Reinforcement Learning and compared ourapproach with relevant model-free and model-based baselines [Achiam et al., 2017, Sikchi et al.

Original: ,2021a, Ray et al., 2019, Schulman et al., 2017].
Revised: ,2021a, Ray et al., 2019, Schulman et al., 2017].

Original: Note that we chose the above baselines because theyare specifically designed to solve constrained optimization problem of the same structure as in 1-3.
Revised: Note that we chose the above baselines because theyare specifically designed to solve constrained optimization problem of the same structure as in 1-3.

Original: In Sikchi et al.
Revised: In Sikchi et al.

Original: [2021a], approximation of the value function is used to provide long-term reasoninginstead of using a truncated horizon.
Revised: [2021a], approximation of the value function is used to provide long-term reasoninginstead of using a truncated horizon.

Original: A limitation of this approach lies in the fact that approximationsof reward and cost value functions are used.
Revised: A limitation of this approach lies in the fact that approximationsof reward and cost value functions are used.

Original: The most challenging part of model-based approaches isto learn the environment model.
Revised: The most challenging part of model-based approaches isto learn the environment model.

Original: The first issue is of computational resources and the time overheadthat is needed.
Revised: The first issue is of computational resources and the time overheadthat is needed.

Original: We provide a comparison of the running time of algorithms and their hardwarerequirements (see Appendix B).
Revised: We provide a comparison of the running time of algorithms and their hardwarerequirements (see Appendix B).

Original: Our algorithm does much better than safe-LOOP [Sikchi et al.
Revised: Our algorithm does much better than safe-LOOP [Sikchi et al.

Original: ,2021a] in terms of running time.
Revised: ,2021a] in terms of running time.

Original: One should note that in safe exploration settings, agents do not explore as much as an unconstrainedagent would do.
Revised: One should note that in safe exploration settings, agents do not explore as much as an unconstrainedagent would do.

Original: This adds to the complexity of model learning in safe exploration problems becauseconstrained exploration leads to limited representation of data points for model learning even withthe same sample size.
Revised: This adds to the complexity of model learning in safe exploration problems becauseconstrained exploration leads to limited representation of data points for model learning even withthe same sample size.

Original: We found this more pronounced in high-dimensional environments like
Revised: We found this more pronounced in high-dimensional environments like

Original: DoggoGoal1 [Ray et al., 2019]
Revised: DoggoGoal1 [Ray et al., 2019]

Original: (See Appendix C).
Revised: (See Appendix C).

Original: Our algorithm depends on Lagrangian-basedapproach which provides a very simplistic way to construct a safe RL algorithm but suffers from theissue of low reward performance compared to unconstrained approaches (see Appendix A for rewardcomparison).
Revised: Our algorithm depends on Lagrangian-basedapproach which provides a very simplistic way to construct a safe RL algorithm but suffers from theissue of low reward performance compared to unconstrained approaches (see Appendix A for rewardcomparison).

Original: Note that we can have a similar model-based approach using TRPO-Lagrangian [Rayet al., 2019]
Revised: Note that we can have a similar model-based approach using TRPO-Lagrangian [Rayet al., 2019]

Original: but it involves an extra overhead of approximating the Fisher Information Matrix (FIM)and hence we chose a PPO-based approach.
Revised: but it involves an extra overhead of approximating the Fisher Information Matrix (FIM)and hence we chose a PPO-based approach.

Original: Also TRPO-Lagrangian and PPO-Lagrangian havesimilar performance on Safety Gym benchmark [Ray et al., 2019].
Revised: Also TRPO-Lagrangian and PPO-Lagrangian havesimilar performance on Safety Gym benchmark [Ray et al., 2019].

Original: Increasing reward performance of
Revised: Increasing reward performance of

Original: Lagrangian-based approaches and devising better ways for model-learning in high-dimensional staterepresentations in safe RL setting where exploration is limited, can be looked atin the future.
Revised: Lagrangian-based approaches and devising better ways for model-learning in high-dimensional staterepresentations in safe RL setting where exploration is limited, can be looked at future.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We describe limitations inConclusion section
Revised: [Yes] We describe limitations in Conclusion section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)
Revised: [N/A]

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have read the ethics review guidelines and ensured that our workconforms to them.
Revised: [Yes] We have read the ethics review guidelines and ensured that our workconforms to them.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)
Revised: [N/A]

Original:  Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide the
Revised: [Yes] We provide the

Original: anonymous url for our code repository
Revised: anonymous url for our code repository

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Hyperparameter details are given in appendix A
Revised: [Yes] Hyperparameter details are given in appendix A

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] We report plots with 95% confidence intervals and use
Revised: [Yes] We report plots with 95% confidence intervals and use

Original: rliable Agarwal et al.
Revised: rliable Agarwal et al.

Original: [2021] library for robust aggregate estimates
Revised: [2021] library for robust aggregate estimates

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We share details of hardware
Revised: [Yes] We share details of hardware

Original: requirement in appendix B
Revised: requirement in appendix B

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite the authors
Revised: [Yes] We cite the authors

Original: of Safety Gym benchmark and their github code repository.
Revised: of Safety Gym benchmark and their github code repository.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Their MIT License which allows
Revised: [Yes] Their MIT License which allows

Original: without restriction, including without limitation the rights to use, copy, modify, merge,publish provided that license notice is included in our code repository
Revised: without restriction, including without limitation the rights to use, copy, modify, merge,publish provided that license notice is included in our code repository

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A](e)

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]