Original: Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Revised: Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Original: Anonymous Author(s)

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps, which inthe real world limit the practicality of these algorithms as these can lead to potentially dangerous behavior.
Revised: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps, which in the real world limit the practicality of these algorithms as these can lead to potentially dangerous behavior.

Original: Hence safe exploration is a critical issue in applying RLalgorithms in the real world.
Revised: Hence safe exploration is a critical issue in applying RL algorithms in the real world.

Original: This problem has been recently well studied under the
Revised: This problem has been recently well studied under the

Original: Constrained Markov Decision Process (CMDP) Framework, where in addition tosingle-stage rewards, state transitions receive single-stage costs or penalties as well.
Revised: Constrained Markov Decision Process (CMDP) Framework, where in addition to single-stage rewards, state transitions receive single-stage costs or penalties as well.

Original: The prescribed cost functions are responsible for mapping undesirable behaviorat any given time-step to a scalar value.
Revised: The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a scalar value.

Original: Then we aim to find a feasible policythat maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.
Revised: Then we aim to find a feasible policy that maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.

Original: We propose an On-policy Model-based Safe Deep RL algorithm in which welearn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using Lagrangian Relaxation-based Proximal Policy
Revised: We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using Lagrangian Relaxation-based Proximal Policy

Original: Optimization.
Revised: Optimization.

Original: We use an ensemble of neural networks with different initializationsto tackle epistemic and aleatoric uncertainty issues faced during environment modellearning.
Revised: We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning.

Original: We compare our approach with relevant model-free and model-basedapproaches in Constrained RL using the challenging Safe Reinforcement Learningbenchmark - the Open AI Safety Gym.
Revised: We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark - the Open AI Safety Gym.

Original: We demonstrate that our algorithm is moresample efficient and results in lower cumulative hazard violations as compared toconstrained model-free approaches.
Revised: We demonstrate that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches.

Original: Further, our approach shows better rewardperformance than other constrained model-based approach in the literature.
Revised: Further, our approach shows better reward performance than other constrained model-based approach in the literature.

Original:  Introduction
Revised:  Introduction

Original: Deep Reinforcement Learning has provided exceptional results both in discrete action domain (Mnihet al. [2013]) and continuous action domain like locomotion tasks ([Haarnoja et al., 2018], Schulmanet al. [2017], Schulman et al.
Revised: Deep Reinforcement Learning has provided exceptional results both in discrete action domain [Mnih et al., 2013] and continuous action domain like locomotion tasks [Haarnoja et al., 2018], [Schulman et al., 2017], [Schulman et al., 2016].

Original: However, most of the RL algorithms perform significantnumber of random exploratory steps during learning as well as deployment which can lead to agentsperforming undesirable and hazardous behaviour.
Revised: However, most of the RL algorithms perform significant number of random exploratory steps during learning as well as deployment which can lead to agents performing undesirable and hazardous behaviour.

Original: This limits the application of RL algorithms in thereal world.
Revised: This limits the application of RL algorithms in the real world.

Original: In scenarios like robot navigation (Han et al. [2018]), autonomous driving (Kendall et al.[2018]), healthcare (Yu et al. [2020]), etc., where RL has potential applications, unsafe behavior canhave hazardous consequences even on human life and property.
Revised: In scenarios like robot navigation [Han et al., 2018], autonomous driving [Kendall et al., 2018], healthcare [Yu et al., 2020], etc., where RL has potential applications, unsafe behavior can have hazardous consequences even on human life and property.

Original: In García et al.
Revised: In García et al.

Original: [2015], the authors have provided a comprehensive survey of several notions ofsafety and the associated problem formulations.
Revised: [2015], the authors have provided a comprehensive survey of several notions of safety and the associated problem formulations.

Original: In our work, we focus on a constraint-based notionof safety.
Revised: In our work, we focus on a constraint-based notion of safety.

Original: In Constraint-based RL, the goal is to maximize long-term expected reward returns andkeep the expected cost-returns below a prescribed threshold.
Revised: In Constraint-based RL, the goal is to maximize long-term expected reward returns and keep the expected cost-returns below a prescribed threshold.

Original: This problem is known as the Safe
Revised: This problem is known as the Safe

Original: Exploration problem.
Revised: Exploration problem.

Original: In [Ray et al., 2019], the authors advocate that safety specifications should beseparate from task performance specifications.
Revised: In [Ray et al., 2019], the authors advocate that safety specifications should be separate from task performance specifications.

Original: It also helps in formulating Safe Exploration as a
Revised: It also helps in formulating Safe Exploration as a

Original: Constrained Optimization Problem and methods used in the optimization literature [Bertsekas, 1996]can be used to solve this problem.
Revised: Constrained Optimization Problem and methods used in the optimization literature [Bertsekas, 1996] can be used to solve this problem.

Original: Constrained Markov Decision Process (CMDP)
Revised: Constrained Markov Decision Process (CMDP)

Original: [Altman, 1998]provides a framework to keep task performance specifications and safety specifications separately.
Revised: [Altman, 1998] provides a framework to keep task performance specifications and safety specifications separately.

Original: Here in addition to single-stage rewards, state transitions receive single-stage costs as well.
Revised: Here in addition to single-stage rewards, state transitions receive single-stage costs as well.

Original: Theprescribed cost functions are responsible for mapping undesirable behavior at any given time-step toa non-negative scalar value.
Revised: The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a non-negative scalar value.

Original: The existing model-free algorithms used in Constrained-RL setting suffer from low sample efficiencyin terms of environment interactions i.e., these algorithms require large number of environmentinteractions (to converge) that in turn would lead to high number of hazardous actions due to unsafeexploration.
Revised: The existing model-free algorithms used in Constrained-RL setting suffer from low sample efficiency in terms of environment interactions i.e., these algorithms require large number of environment interactions (to converge) that in turn would lead to high number of hazardous actions due to unsafe exploration.

Original: This serves as the motivation for us to use a Model-based approach for Constrained-RL.
Revised: This serves as the motivation for us to use a Model-based approach for Constrained-RL.

Original: Our Contributions: We propose a simple and sample efficient model-based approach for Safe
Revised: Our Contributions: We propose a simple and sample efficient model-based approach for Safe

Original: Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem.
Revised: Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem.

Original: Wehighlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a wayto incorporate that in our setting.
Revised: We highlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a way to incorporate that in our setting.

Original: We demonstrate that our approach is ∼ 3 − 4 times more sampleefficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulativehazard violations by ∼ 60% .
Revised: We demonstrate that our approach is ∼ 3 − 4 times more sample efficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulative hazard violations by ∼ 60% .

Original: Our approach also outperforms other model-free Constrained Policy
Revised: Our approach also outperforms other model-free Constrained Policy

Original: Optimization (CPO)
Revised: Optimization (CPO)

Original: [Achiam et al., 2017] in terms of constraint satisfaction.
Revised: [Achiam et al., 2017] in terms of constraint satisfaction.

Original: We also compare ourapproach with model-based approach [Sikchi et al., 2021a] and we observe that our approach doesbetter in terms of reward performance and has competitive cost performance as well.
Revised: We also compare our approach with model-based approach [Sikchi et al., 2021a] and we observe that our approach does better in terms of reward performance and has competitive cost performance as well.

Original:  Related Work
Revised:  Related Work

Original: Different notions of safety and their mathematical formulations are provided in García et al.
Revised: Different notions of safety and their mathematical formulations are provided in García et al.

Original: There are several works on the lines of formulating the Safe RL problem in the setting of CMDP.
Revised: There are several works on the lines of formulating the Safe RL problem in the setting of CMDP.

Original: In some of the early works, an actor-critic algorithm for CMDP under the long-run average costcriterion is proposed in Borkar [2005] that is however for the case of full state representations.
Revised: In some of the early works, an actor-critic algorithm for CMDP under the long-run average cost criterion is proposed in Borkar [2005] that is however for the case of full state representations.

Original: Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and
Revised: Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and

Original: Lakshmanan [2012] for the long-run average cost setting and in Bhatnagar [2010] for the infinitehorizon discounted cost scenario.
Revised: Lakshmanan [2012] for the long-run average cost setting and in Bhatnagar [2010] for the infinite horizon discounted cost scenario.

Original: The procedure in the aforementioned references involved forming a
Revised: The procedure in the aforementioned references involved forming a

Original: Lagrangian by relaxing the constraints.
Revised: Lagrangian by relaxing the constraints.

Original: The algorithms in these papers are based on multi-timescalestochastic approximation with updates of the Lagrange parameter performed on the slowest timescale,the policy updates on the medium timescale and the updates of the value function (for a given policy)performed on the fast timescale.
Revised: The algorithms in these papers are based on multi-timescale stochastic approximation with updates of the Lagrange parameter performed on the slowest timescale, the policy updates on the medium timescale and the updates of the value function (for a given policy) performed on the fast timescale.

Original: In more recent work, in Achiam et al.
Revised: In more recent work, in Achiam et al.

Original: [2017], a trust region based constrained policy optimization(CPO) framework is proposed, which involved approximation of the problem using surrogate functionsfor both the objective and the constraints and included a projection step on policy parameters thatneeded backtracking line search, making it complicated and time-consuming.
Revised: [2017], a trust region based constrained policy optimization (CPO) framework is proposed, which involved approximation of the problem using surrogate functions for both the objective and the constraints and included a projection step on policy parameters that needed backtracking line search, making it complicated and time-consuming.

Original: CPO showed nearconstraint satisfaction in every iteration of the policy updates in standard Mujoco environmentsmodified for safe exploration (Achiam et al. but didn’t yield a constraint satisfying policy inchallenging Safe RL benchmark Safety Gym (Ray et al.
Revised: CPO showed near constraint satisfaction in every iteration of the policy updates in standard Mujoco environments modified for safe exploration [Achiam et al., 2017] but didn’t yield a constraint satisfying policy in challenging Safe RL benchmark Safety Gym [Ray et al., 2019].

Original: Another work (Yu et al. [2019])involved using surrogate functions for approximation of both objective and constraint functions.
Revised: Another work [Yu et al., 2019] involved using surrogate functions for approximation of both objective and constraint functions.

Original: Theirprocedure involved constructing a sequence of convex optimization problems for which they showedthat the sequence of stationary points converges to the stationary point of the original non-convexproblem.
Revised: Their procedure involved constructing a sequence of convex optimization problems for which they showed that the sequence of stationary points converges to the stationary point of the original non-convex problem.

Original: In Ray et al.
Revised: In Ray et al.

Original: [2019], Lagrangian relaxation of Constrained RL problem is used and combined withPPO(Schulman et al.
Revised: [2019], Lagrangian relaxation of Constrained RL problem is used and combined with

Original: [2017]) to give a PPO-Lagrangian algorithm and with TRPO (Schulman et al. [2015])to give a TRPO-Lagrangian algorithm.
Revised: PPO [Schulman et al., 2017] to give a PPO-Lagrangian algorithm and with TRPO [Schulman et al., 2015] to give a TRPO-Lagrangian algorithm.

Original: These algorithms were seen to outperform CPOAchiamet al. [2017] in terms of constraint satisfaction on Safety Gym on several environments.
Revised: These algorithms were seen to outperform CPO[Achiam et al., 2017] in terms of constraint satisfaction on Safety Gym on several environments.

Original: Also thesealgorithms are simpler to implement and tune.
Revised: Also these algorithms are simpler to implement and tune.

Original: Another Lagrangian-based method, see Tessler et al.[2018], used a penalized reward function for optimizing their agent and showed convergence tooptimal feasible policies using a two-timescale stochastic approximation scheme where the Lagrangemultiplier is updated on a slower timescale as compared to the policy parameters as was the casewith Borkar [2005], Bhatnagar [2010], Bhatnagar and Lakshmanan [2012].
Revised: Another Lagrangian-based method, see Tessler et al. [2018], used a penalized reward function for optimizing their agent and showed convergence to optimal feasible policies using a two-timescale stochastic approximation scheme where the Lagrange multiplier is updated on a slower timescale as compared to the policy parameters as was the case with Borkar [2005], Bhatnagar [2010], Bhatnagar and Lakshmanan [2012].

Original: In Zhang et al.
Revised: In Zhang et al.

Original: [2020],the authors proposed a first order constrained policy optimization (FOCOPS) method that involvedsolving the optimization problem in non-parametric space and projecting it back into the parametricspace.
Revised: [2020], the authors proposed a first order constrained policy optimization (FOCOPS) method that involved solving the optimization problem in non-parametric space and projecting it back into the parametric space.

Original: Approaches in Tessler et al.
Revised: Approaches in Tessler et al.

Original: [2018] and Zhang et al.
Revised: [2018] and Zhang et al.

Original: [2020] performed poorly on Safety Gym.
Revised: [2020] performed poorly on Safety Gym.

Original: In Stooke et al.
Revised: In Stooke et al.

Original: [2020], a PID-based approach to damp oscillations in Lagrangian methods is proposed, which is seen to minimize constraint violations.
Revised: [2020], a PID-based approach to damp oscillations in Lagrangian methods is proposed, which is seen to minimize constraint violations.

Original: In Wen and Topcu [2021], the Cross Entropy method(de Boer et al. [2005]) is used for finding a safe policy and convergence using the ODE method isshown, however, empirical results are shown on a primitive and less challenging environment.
Revised: In Wen and Topcu [2021], the Cross Entropy method [de Boer et al., 2005] is used for finding a safe policy and convergence using the ODE method is shown, however, empirical results are shown on a primitive and less challenging environment.

Original: In
Revised: In

Original: Dalal et al.
Revised: Dalal et al.

Original: [2018], the authors formulated a state-wise constrained policy optimization problemwhere at each transition a constraint needs to be satisfied and an analytical method for correcting theunsafe action using a safety layer trained using random exploration was proposed.
Revised: [2018], the authors formulated a state-wise constrained policy optimization problem where at each transition a constraint needs to be satisfied and an analytical method for correcting the unsafe action using a safety layer trained using random exploration was proposed.

Original: In Chow et al.[2018], the authors proposed constructing Lyapunov functions to guarantee safety of the behaviourpolicy under a CMDP framework.
Revised: In Chow et al. [2018], the authors proposed constructing Lyapunov functions to guarantee safety of the behaviour policy under a CMDP framework.

Original: In Chow et al.
Revised: In Chow et al.

Original: [2019], the above Lyapunov based method wasextended to continuous control
Revised: [2019], the above Lyapunov based method was extended to continuous control

Original: but it’s performance in Safety Gym environment ([Ray et al. , 2019])was not good in terms of rewards obtained ([Sikchi et al., 2021b]).
Revised: but it’s performance in Safety Gym environment [Ray et al., 2019] was not good in terms of rewards obtained [Sikchi et al., 2021b].

Original: In Liu et al.
Revised: In Liu et al.

Original: [2022], the methodinvolved using tangent space of the constraint manifold to learn a safe policy but is seen to be highlyspecific to the environment used.
Revised: [2022], the method involved using tangent space of the constraint manifold to learn a safe policy but is seen to be highly specific to the environment used.

Original: For unconstrained state-of-the-art Model-based RL algorithms in Deisenroth and Rasmussen [2011],
Revised: For unconstrained state-of-the-art Model-based RL algorithms in Deisenroth and Rasmussen [2011],

Original: Luo et al.
Revised: Luo et al.

Original: [2018], Chua et al.
Revised: [2018], Chua et al.

Original: [2018], Kurutach et al.
Revised: [2018], Kurutach et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2015], a comparison ofthe empirical performance on Open AI Gym Brockman et al.
Revised: [2015], a comparison of the empirical performance on Open AI Gym Brockman et al.

Original: [2016] is shown in Wang et al.
Revised: [2016] is shown in Wang et al.

Original: Also, Sikchi et al.
Revised: Also, Sikchi et al.

Original: [2021a] propose augmenting planning trajectory with terminal value function toincorporate long-horizon reasoning in model-based methods, since the approaches in Deisenroth and
Revised: [2021a] propose augmenting planning trajectory with terminal value function to incorporate long-horizon reasoning in model-based methods, since the approaches in Deisenroth and

Original: Rasmussen [2011], Luo et al.
Revised: Rasmussen [2011], Luo et al.

Original: [2018], Chua et al.
Revised: [2018], Chua et al.

Original: [2018], Kurutach et al.
Revised: [2018], Kurutach et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2015]plan over a fixed and short horizon to avoid aggregation of error.
Revised: [2015] plan over a fixed and short horizon to avoid aggregation of error.

Original: This requires good approximationof both model as well as value function.
Revised: This requires good approximation of both model as well as value function.

Original: There are also several works that use model-based RL to tackle the problem of safety.
Revised: There are also several works that use model-based RL to tackle the problem of safety.

Original: In Liu et al.[2020], a model based approach is proposed to learn the system dynamics and cost model.
Revised: In Liu et al. [2020], a model based approach is proposed to learn the system dynamics and cost model.

Original: Then rollouts from the learned model are used to optimize policy using a modified cross-entropy based methodwhich involved sampling from a distribution of policies, sorting sample policies based on constraintfunctions and using them to update the policy distribution but their implementation involved a datacollection step using random policy for large number of episodes which itself is risky in real-worldscenarios.
Revised: Then roll outs from the learned model are used to optimize policy using a modified cross-entropy based method which involved sampling from a distribution of policies, sorting sample policies based on constraint functions and using them to update the policy distribution but their implementation involved a data collection step using random policy for large number of episodes which itself is risky in real-world scenarios.

Original: In Cowen-Rivers et al.
Revised: In Cowen-Rivers et al.

Original: [2020], model dynamics is learned using PILCO (Deisenroth and
Revised: [2020], model dynamics is learned using PILCO [Deisenroth and

Original: Rasmussen [2011]) and instead of the discounted cost constraint function, conditional value at risk(CVaR) based constraint function is used (Uryasev and Rockafellar [2001], Chow and Ghavamzadeh[2014]).
Revised: Rasmussen, 2011] and instead of the discounted cost constraint function, conditional value at risk (CVaR) based constraint function is used [Uryasev and Rockafellar, 2001, Chow and Ghavamzadeh, 2014].

Original: In Thomas et al.
Revised: In Thomas et al.

Original: [2022], penalized reward functions are used instead of a separate costfunction, then model of the environment is learned and Soft-actor critic algorithm (Haarnoja et al.[2018]) is used to optimize the policy.
Revised: [2022], penalized reward functions are used instead of a separate cost function, then model of the environment is learned and Soft-actor critic algorithm [Haarnoja et al., 2018] is used to optimize the policy.

Original: In this approach safe and unsafe states are also need to bespecified upfront.
Revised: In this approach safe and unsafe states are also need to be specified upfront.

Original:  Background 
Revised:  Background 3.1 Constrained Markov Decision Process (CMDP)

Original: 3.1 Constrained Markov Decision Process (CMDP)

Original: A CMDP is denoted by the tuple ( S, A, R, C, γ, µ ) where S denotes the state space, A is the actionspace, R : S ×
Revised: A CMDP is denoted by the tuple ( S, A, R, C, γ, µ ) where S denotes the state space, A is the action space, R : S ×

Original: A × S → R is the single-stage reward function, C : S ×
Revised: A × S → R is the single-stage reward function, C : S ×

Original: A × S → R denotes theassociated single-stage cost function (we assume a single constraint function for simplicity here), γis the discount factor and µ signifies the initial state distribution.
Revised: A × S → R denotes the associated single-stage cost function (we assume a single constraint function for simplicity here), γ is the discount factor and µ signifies the initial state distribution.

Original: We assume that both R and C arenon-negative functions.
Revised: We assume that both R and C are non-negative functions.

Original: By a policy π = ( π 0 , π 1 , . . . ) , we mean a decision rule for selecting actions.
Revised: By a policy π = ( π 0 , π 1 , . . . ) , we mean a decision rule for selecting actions.

Original: It is specified as follows:
Revised: It is specified as follows:

Original: For any k ≥ 0 and s ∈ S , π k ( s ) ∈ P ( s ) is the probability distribution π k ( s ) △ =
Revised: For any k ≥ 0 and s ∈ S , π k ( s ) ∈ P ( s ) is the probability distribution π k ( s ) △ =

Original: ( π k ( s, a  , a ∈ A  s ))where π k ( s, a
Revised: ( π k ( s, a ) , a ∈ A ( s ))

Original:  is the probability of picking action a in state s at instant k under policy π .
Revised: where π k ( s, a ) is the probability of picking action a in state s at instant k under policy π .

Original: In theabove, A ( s ) is the set of feasible actions in state s and so A = ∪ s ∈ S A ( s ) .
Revised: In the above, A ( s ) is the set of feasible actions in state s and so A = ∪ s ∈ S A ( s ) .

Original: Such a policy is also oftenreferred to as a randomized policy.
Revised: Such a policy is also often referred to as a randomized policy.

Original: A stationary policy is a randomized policy as above except withπ k = π l , ∀ k ̸ = l .
Revised: A stationary policy is a randomized policy as above except with π k = π l , ∀ k ̸ = l .

Original: Thus, a stationary policy selects actions according to a given distribution regardlessof the instant when an action is chosen according to the given policy.
Revised: Thus, a stationary policy selects actions according to a given distribution regardless of the instant when an action is chosen according to the given policy.

Original: By an abuse of notation, wedenote a stationary policy as π itself.
Revised: By an abuse of notation, we denote a stationary policy as π itself.

Original: We shall consider here a class of stationary policies π θ parameterized by a parameter θ .
Revised: We shall consider here a class of stationary policies π θ parameterized by a parameter θ .

Original: Our objectivefunction is defined via the infinite horizon discounted reward criterion where for a given π θ we have
Revised: Our objective function is defined via the infinite horizon discounted reward criterion where for a given π θ we have

Original: [Equation]
Revised: [Equation]

Original: The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:
Revised: The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:

Original: [Equation]
Revised: [Equation]

Original: Then J R ( π θ ) , J C ( π θ ) ∈ R .
Revised: Then J R ( π θ ) , J C ( π θ ) ∈ R .

Original: Let d > 0 denote a prescribed threshold below which we want J C ( π θ )to lie.
Revised: Let d > 0 denote a prescribed threshold below which we want J C ( π θ ) to lie.

Original: The constrained optimization problem then is the following:
Revised: The constrained optimization problem then is the following:

Original: [Equation]
Revised: [Equation]

Original: A parameter θ will be called a feasible point if the cost constraint is satisfied for θ , i.e., J C ( π θ ) ≤ d .
Revised: A parameter θ will be called a feasible point if the cost constraint is satisfied for θ , i.e., J C ( π θ ) ≤ d .

Original: 3.2 Lagrangian Relaxation based Proximal Policy Optimization
Revised: 3.2 Lagrangian Relaxation based Proximal Policy Optimization

Original: The Lagrangian of the constrained optimization problem (3) can be written as follows:
Revised: The Lagrangian of the constrained optimization problem (3) can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R + is the Lagrange multiplier and is a positive real number.
Revised: where λ ∈ R + is the Lagrange multiplier and is a positive real number.

Original: In terms of the Lagrangian,the goal is to find a tuple ( θ ∗ , λ ∗ ) of the policy and Lagrange parameter such that
Revised: In terms of the Lagrangian, the goal is to find a tuple ( θ ∗ , λ ∗ ) of the policy and Lagrange parameter such that

Original: [Equation]
Revised: [Equation]

Original: Solving the max-min problem as above is equivalent to finding a global optimal saddle point ( θ ∗ , λ ∗ )such that ∀ ( θ, λ ) , the following holds:
Revised: Solving the max-min problem as above is equivalent to finding a global optimal saddle point ( θ ∗ , λ ∗ ) such that ∀ ( θ, λ ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: We assume that θ refers to the parameter of a Deep Neural Network, hence finding such a globallyoptimal saddle point is computationally hard.
Revised: We assume that θ refers to the parameter of a Deep Neural Network, hence finding such a globally optimal saddle point is computationally hard.

Original: So our aim is to find a locally optimal saddle pointwhich satisfies (6) in a local neighbourhood H ϵ 1 ,ϵ 2 which is defined as follows: 
Revised: So our aim is to find a locally optimal saddle point which satisfies (6) in a local neighbourhood H ϵ 1 ,ϵ 2 which is defined as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for some ϵ 1 , ϵ 2 > 0 .
Revised: for some ϵ 1 , ϵ 2 > 0 .

Original: Assuming that L ( θ, λ ) is known for every ( θ, λ ) tuple, a gradient searchprocedure for finding a local ( θ ∗ , λ ∗ ) tuple would be the following:
Revised: Assuming that L ( θ, λ ) is known for every ( θ, λ ) tuple, a gradient search procedure for finding a local ( θ ∗ , λ ∗ ) tuple would be the following:

Original: [Equation]
Revised: [Equation]

Original: Here [ x ] + denotes max(0 , x ) .
Revised: Here [ x ] + denotes max(0 , x ) .

Original: This operator ensures that the Lagrange multiplier remains nonnegative after each update.
Revised: This operator ensures that the Lagrange multiplier remains nonnegative after each update.

Original: In (9)-(11), η 1 ( n ) , η 2 ( n ) > 0 ∀ n are certain prescribed step-size schedules.
Revised: In (9)-(11), η 1 ( n ) , η 2 ( n ) > 0 ∀ n are certain prescribed step-size schedules.

Original: We assume that the step-sizes η 1 ( n ) , η 2 ( n ) , n ≥ 0 satisfy the regular step-size conditions.
Revised: We assume that the step-sizes η 1 ( n ) , η 2 ( n ) , n ≥ 0 satisfy the regular step-size conditions.

Original: Thus, fori = 1 , 2 , (cid:88)
Revised: Thus, for i = 1 , 2 , (cid:88)

Original: k
Revised: k

Original: η i ( n ) = ∞ , (cid:88) k
Revised: η i ( n ) = ∞ , (cid:88) k

Original: η 2 i ( n ) < ∞ .
Revised: η 2 i ( n ) < ∞ .

Original: Note however that J R ( π θ ) and J C ( π θ ) as specified in(1)-(2) are not a priori known quantities and need to be estimated.
Revised: Note however that J R ( π θ ) and J C ( π θ ) as specified in (1)-(2) are not a priori known quantities and need to be estimated.

Original: We discuss this in detail below.
Revised: We discuss this in detail below.

Original: 3.2.1 Estimation
Revised: 3.2.1 Estimation

Original: We run each episode for T time steps in our experiments.
Revised: We run each episode for T time steps in our experiments.

Original: Let r t +1 ≡ R ( s t , a t , s t +1 ) and c t +1 ≡
Revised: Let r t +1 ≡ R ( s t , a t , s t +1 ) and c t +1 ≡

Original: C ( s t , a t , s t +1 ) , respectively, for simplicity.
Revised: C ( s t , a t , s t +1 ) , respectively, for simplicity.

Original: For each sample path we would have both a rewardreturn as well as a cost return.
Revised: For each sample path we would have both a reward return as well as a cost return.

Original: Let ˆ R t (resp.
Revised: Let ˆ R t (resp.

Original: ˆ C t ) be the reward-to-go (resp.
Revised: ˆ C t ) be the reward-to-go (resp.

Original: cost-to-go) estimate.
Revised: cost-to-go) estimate.

Original: Wecompute ˆ R t and ˆ C t according to: ˆ R t =(cid:80) T − t − 1 k =
Revised: We compute ˆ R t and ˆ C t according to: ˆ R t =

Original:  γ k r t + k +1 , ˆ C t = (cid:80) T − t − 1 k = γ k c t + k +1 , respectively.
Revised: (cid:80) T − t − 1 k = γ k r t + k +1 , ˆ C t = (cid:80) T − t − 1 k = γ k c t + k +1 , respectively.

Original: We use a neural network parameterized by ψ r to estimate reward signal based value function V
Revised: We use a neural network parameterized by ψ r to estimate reward signal based value function V

Original: Rψ r anda neural network parameterized by ψ c to estimate a cost signal based value function V Cψ c .
Revised: Rψ r and a neural network parameterized by ψ c to estimate a cost signal based value function V Cψ c .

Original: We run oursimulations on N parallel workers and then sample a mini-batch of size M ≤ NT [Schulman et al.,
Revised: We run our simulations on N parallel workers and then sample a mini-batch of size M ≤ NT [Schulman et al.,

Original: We use mean-squared loss to estimate value functions on sampled mini-batches as follows:
Revised: We use mean-squared loss to estimate value functions on sampled mini-batches as follows:

Original: [Equation]
Revised: [Equation]

Original: Let A Rt , t ≥ 0 , and A Ct , t ≥ 0 , respectively, denote the advantage estimates w.r.t reward and costvalue functions on the sample path.
Revised: Let A Rt , t ≥ 0 , and A Ct , t ≥ 0 , respectively, denote the advantage estimates w.r.t reward and cost value functions on the sample path.

Original: We compute them using Generalized Advantage Estimation[Schulman et al., 2016] to balance the bias and variance tradeoff of advantage estimates.
Revised: We compute them using Generalized Advantage Estimation [Schulman et al., 2016] to balance the bias and variance tradeoff of advantage estimates.

Original: We can havea range of advantage estimates as under.
Revised: We can have a range of advantage estimates as under.

Original: [Equation]
Revised: [Equation]

Original: A

Original: [Equation]

Original: for k > 2 .
Revised: for k > 2 .

Original: The advantage estimate in (14) will have high bias but low variance while estimates in (16)with a higher value of k generally have high variance but low bias.
Revised: The advantage estimate in (14) will have high bias but low variance while estimates in (16) with a higher value of k generally have high variance but low bias.

Original: Let δ Rt
Revised: Let δ Rt

Original: =
Revised: =

Original: r t +1 + V
Revised: r t +1 + V

Original: Rψ r ( s t +1 ) −
Revised: Rψ r ( s t +1 ) −

Original: V
Revised: V

Original: Rψ r ( s t ) and δ C t =
Revised: Rψ r ( s t ) and δ C t =

Original: c t +1 + V Cψ c ( s t +1 ) − V Cψ c ( s t ) , respectively, denote the reward and cost temporaldifferences.
Revised: c t +1 + V Cψ c ( s t +1 ) − V Cψ c ( s t ) , respectively, denote the reward and cost temporal differences.

Original: Let ¯ λ be a parameter which adjusts the bias-variance tradeoff.
Revised: Let ¯ λ be a parameter which adjusts the bias-variance tradeoff.

Original: Generalized Advantage
Revised: Generalized Advantage

Original: Estimate [Schulman et al., 2016] for A Rt ,
Revised: Estimate [Schulman et al., 2016] for A Rt ,

Original: A Ct is then given by
Revised: A Ct is then given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: respectively.
Revised: respectively.

Original: Now we use PPO clipped objectives [Schulman et al., 2017] for estimation of
Revised: Now we use PPO clipped objectives [Schulman et al., 2017] for estimation of

Original: J R ( π θ ) , J C ( π θ ) as follows:
Revised: J R ( π θ ) , J C ( π θ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where r t ( θ ) =
Revised: where r t ( θ ) =

Original: π θ ( a t | s t ) π θold ( a t | s t ) is the ratio of probability of selecting action a t in state s t under parameterθ as opposed to θ old .
Revised: π θ ( a t | s t ) π θold ( a t | s t ) is the ratio of probability of selecting action a t in state s t under parameter θ as opposed to θ old .

Original: Further, A Rt and A Ct are the estimated advantages based on the reward and costreturns, respectively, by time t (see above) and ϵ is the clip-ratio which clips r t ( θ ) to (1 − ϵ ) if it isless than (1 − ϵ ) and clips to (1 + ϵ ) if it is greater than (1 + ϵ ) .
Revised: Further, A Rt and A Ct are the estimated advantages based on the reward and cost returns, respectively, by time t (see above) and ϵ is the clip-ratio which clips r t ( θ ) to (1 − ϵ ) if it is less than (1 − ϵ ) and clips to (1 + ϵ ) if it is greater than (1 + ϵ ) .

Original: This algorithm restricts the policyparameters to not change significantly between two iterations which helps in avoiding divergence.
Revised: This algorithm restricts the policy parameters to not change significantly between two iterations which helps in avoiding divergence.

Original: This approach is also known as PPO-Lagrangian [Ray et al., 2019].
Revised: This approach is also known as PPO-Lagrangian [Ray et al., 2019].

Original: 3.3 Model-based Constrained RL
Revised: 3.3 Model-based Constrained RL

Original: We formulate a Constrained RL problem (21) using Model-based framework as follows:
Revised: We formulate a Constrained RL problem (21) using Model-based framework as follows:

Original: [Equation]
Revised: [Equation]

Original: , where(21)
Revised: where

Original: [Equation]
Revised: [Equation]

Original:  In the above, P α ( . | s t , a t ) is an α -parameterized environment model, d i is a human prescribed safetythreshold for the i th constraint and Π θ is the set of all θ − parameterized stationary polices.
Revised: [Equation] In the above, P α ( . | s t , a t ) is an α -parameterized environment model, d i is a human prescribed safety threshold for the i th constraint and Π θ is the set of all θ − parameterized stationary polices.

Original: Notethat we assume initial state s 0 is sampled from true initial state distribution µ and then s t +1 ∼
Revised: Note that we assume initial state s 0 is sampled from true initial state distribution µ and then s t +1 ∼

Original: P α ( .
Revised: P α ( .

Original: | s t , a t ) , ∀ t > 0 .
Revised: | s t , a t ) , ∀ t > 0 .

Original: We would use approximation of environment P α to create ‘imaginary’ rolloutsto estimate reward and cost returns required for policy optimization algorithms.
Revised: We would use approximation of environment P α to create ‘imaginary’ rollouts to estimate reward and cost returns required for policy optimization algorithms.

Original:  Challenges in Environment Model Learning
Revised:  Challenges in Environment Model Learning

Original: In this section we discuss the challenges that commonly arise due to model learning in RL.
Revised: In this section we discuss the challenges that commonly arise due to model learning in RL.

Original: We furtherhighlight the challenge that arises from using environment model approximation in Safe RL settings.
Revised: We further highlight the challenge that arises from using environment model approximation in Safe RL settings.

Original: Handling aleatoric and epistemic uncertainties : Aleatoric Uncertainty refers to thenotion of natural randomness in the system which leads to variability in outcomes of anexperiment.
Revised: Handling aleatoric and epistemic uncertainties : Aleatoric Uncertainty refers to the notion of natural randomness in the system which leads to variability in outcomes of an experiment.

Original: This uncertainty is irreducible because it is a natural property of the system.
Revised: This uncertainty is irreducible because it is a natural property of the system.

Original: Hence in such cases, giving measure of uncertainty in model’s prediction is a good practice.
Revised: Hence in such cases, giving measure of uncertainty in model’s prediction is a good practice.

Original: In Lakshminarayanan et al.
Revised: In Lakshminarayanan et al.

Original: 2017], Chua et al.
Revised: [2017], Chua et al.

Original: 2018] uncertainty-aware neural networksare used which gives an idea about uncertainty in prediction as well
Revised: [2018] uncertainty-aware neural networks are used which gives an idea about uncertainty in prediction as well.

Original: They learn Gaussian distribution parameterized by neural networks.
Revised: They learn Gaussian distribution parameterized by neural networks.

Original: Epistemic Uncertainty refers to the notion of lack of sufficient knowledge in the model as a result of which the model does not generalize.
Revised: Epistemic Uncertainty refers to the notion of lack of sufficient knowledge in the model as a result of which the model does not generalize.

Original: In Lakshminarayanan et al.
Revised: In Lakshminarayanan et al.

Original: ], an ensemble of uncertainty aware neural networks with different initialization is proposed to reduce epistemic uncertainty for fixed data
Revised: [2017], an ensemble of uncertainty aware neural networks with different initialization is proposed to reduce epistemic uncertainty for fixed data. For

Original: For learning environment model we also use an ensemble of n neural networks withdifferent initialization.
Revised: learning environment model we also use an ensemble of n neural networks with random initialization.

Original: Each neural network’s output parameterizes a multivariate normaldistribution with diagonal covariance matrix.
Revised: Each neural network’s output parameterizes a multivariate normal distribution with diagonal covariance matrix.

Original: Suppose the i th neural network in the ensembleis parameterized by α i and the mean and standard deviation outputs are µ α i and σ α irespectively.
Revised: Suppose the i th neural network in the ensemble is parameterized by α i and the mean and standard deviation outputs are µ α i and σ α i respectively.

Original: Recall now that if a random vector X ∈ R d is distributed according to themultivariate normal distribution parameterized by ( µ, Σ d × d ) where µ ∈ R d is the meanvector and Σ d × d is a d × d covariance matrix, then the probability density of X is definedas, P ( X ) 
Revised: Recall now that if a random vector X ∈ R d is distributed according to the multivariate normal distribution parameterized by ( µ, Σ d × d ) where µ ∈ R d is the mean vector and Σ d × d is a d × d covariance matrix, then the probability density of X is defined as, P ( X ) =

Original: =
Revised: [Equation]

Original: 1 (2 π ) d/ 2 | Σ | − 1 / 2 exp ( − 12 ( X − µ ) ′
Revised: [Equation] | Σ | [Equation] / 2 exp ( − 12 ( X − µ ) ′

Original: Σ − 1 ( X − µ ))
Revised: Σ − 1 ( X − µ ))

Original: As a choice of loss function weuse the negative log-likelihood loss for minimization (i.e., minimizing negative log of P(X)).
Revised: As a choice of loss function we use the negative log-likelihood loss for minimization (i.e., minimizing negative log of P(X)).

Original: For the i th neural network parameterized by α i , the loss function L ( α i ) is given as follows:
Revised: For the i th neural network parameterized by α i , the loss function L ( α i ) is given as follows:

Original: [Equation]
Revised: [Equation]

Revised: µ α i ( s t , a t ) − s t +1 ] T Σ − 1 ( s t , a t )[ µ α i ( s t , a t ) − s t +1 ]+log | Σ α i ( s t , a t ) | , (24)

Original: where µ α i ( s t , a t ) is the mean vector output of the i th neural network and
Revised: where µ α i ( s t , a t ) is the mean vector output of the i th neural network and

Original: Σ α i ( s t , a t ) is thecovariance matrix which is assumed to be a diagonal matrix.
Revised: Σ α i ( s t , a t ) is the covariance matrix which is assumed to be a diagonal matrix.

Revised: Note that, using n neural networks with random initialization tends to have a regularization effect.

Revised: Intuitively, it introduces diversity in learned models to deal with more possible trajectories.

Revised: Also, it has been shown empirically that with an increase in the number of models, the performance tends to improve (See section 6.3 and Figure 4 of Kurutach et al.

Revised: [2018]), but increase in number of model also leads to increase in space complexity.

Original: Aggregation of Error :
Revised: Aggregation of Error :

Original: In model-based RL, as we move forward along the horizon, the error due to approximation starts aggregating and predictions 
Revised: In model-based RL, as we move forward along the horizon, the error due to approximation starts aggregating and predictions from the approximated model tend to diverge significantly from the true model.

Original: from the approximated modeltend to diverge significantly from the true model In order to tackle this problem, most ofthe model-based RL approaches [Deisenroth and Rasmussen, 2011, Kurutach et al., 2018, Janner et al., 2019] use shorter (or truncated) horizon during the policy optimization phaseand achieve similar performance as Model-Free RL approaches.
Revised: In order to tackle this problem, most of the model-based RL approaches [Deisenroth and Rasmussen, 2011, Kurutach et al., 2018, Janner et al., 2019] use shorter (or truncated) horizon during the policy optimization phase and achieve similar performance as Model-Free RL approaches.

Original: We use truncated horizonin our approach.
Revised: We use truncated horizon in our approach.

Original: Implication of using truncated horizon in Constrained RL : When we use truncated horizon in Constrained RL, it leads to underestimation of cost returns (23) under the current policy and we use the prescribed constraints to threshold the returns obtained.
Revised: Implication of using truncated horizon in Constrained RL : When we use truncated horizon in Constrained RL, it leads to underestimation of cost returns (23) under the current policy and we use the prescribed constraints to threshold the returns obtained.

Original: This can lead to constraint violations in the real-environment where the cost objective is based on the infinite horizon cost return.
Revised: This can lead to constraint violations in the real-environment where the cost objective is based on the infinite horizon cost return.

Original: We propose a hyperparameter-based approach to deal with this problem in the next section
Revised: We propose a hyperparameter-based approach to deal with this problem in the next section.

Original:  Model-based PPO Lagrangian
Revised:  Model-based PPO Lagrangian

Original: We propose a Model-based algorithm derived from relaxing the Lagrangian (see Algorithm 1) whichalleviates the problem of requirement of a large number of samples as in model-free Lagrangianbased approaches and as a consequence also decreases the cumulative hazard violations.
Revised: We propose a Model-based algorithm derived from relaxing the Lagrangian (see Algorithm 1) which alleviates the problem of requirement of a large number of samples as in model-free Lagrangian based approaches and as a consequence also decreases the cumulative hazard violations.

Original: It is difficultto evaluate policy without interacting with the real environment accurately.
Revised: It is difficult to evaluate policy without interacting with the real environment accurately.

Original: 1: Input: Initialize actor neural net parameter θ 0 , critic parameters ψ r 0 , ψ c 0 , ensemble models [ P α i ] ni =1 , Lagrange parameter λ 0 ≥ 0 , cost threshold =
Revised: 1: Input: Initialize actor neural net parameter θ 0 , critic parameters ψ r 0 , ψ c 0 , ensemble models [ P α i ] ni =1 , Lagrange parameter λ 0 ≥ 0 , cost threshold =

Original: d , Environment Horizon = T , Model
Revised: d , Environment Horizon = T , Model

Original: Horizon =
Revised: Horizon =

Original: H 2: for i = 1 , . .
Revised: H 2: for i = 1 , . .

Original: , N training epochs do 3:
Revised: , N training epochs do 3:

Original: Collect data tuples ( s t , a t , s t +1 ) , for the i th ensemble using policy π θ i , i = 1 , . .
Revised: Collect data tuples ( s t , a t , s t +1 ) , for the i th ensemble using policy π θ i , i = 1 , . .

Original: . , n , in the environment for T time steps over multiple episodes | E | 4:
Revised: . , n , in the environment for T time steps over multiple episodes | E | 4:

Original: Train [ P α i ] ni =1 by minimizing (24) w.r.t α i , ∀ i = 1 to n 5: while Performance ratio > PR Threshold do 6: s 0 ∼ µ { Note: For first pass we use mix of real data and imaginary rollout data.
Revised: Train [ P α i ] ni =1 by minimizing (24) w.r.t α i , ∀ i = 1 to n 5: while Performance ratio > PR Threshold do 6: s 0 ∼ µ { Note: For first pass we use mix of real data and imaginary rollout data.

Original: (See Appendix A) } 7:
Revised: (See Appendix A) } 7:

Original: Collect data rollouts as a t ∼ π θ i ( .
Revised: Collect data rollouts as a t ∼ π θ i ( .

Original: | s t ) , s t ∼ P α q ( .
Revised: | s t ) , s t ∼ P α q ( .

Original: | s t , a t ) (At each time step ’ q ’ is randomly selected from 1 , . . . , n ) for H time steps ( H < T ) 8:
Revised: | s t , a t ) (At each time step ’ q ’ is randomly selected from 1 , . . . , n ) for H time steps ( H < T ) 8:

Original: Compute J Csample ( π θ t ) = 

Revised: Compute J Csample ( π θ t ) =

Original: | E | (cid:80) Hp =1 γ p C
Revised: | E | (cid:80) Hp =1 γ p C

Original: ( s t , a t ) where | E | is the number of episodes 9:
Revised: ( s t , a t ) where | E | is the number of episodes 9:

Original: Compute advantage, cost-advantage using (17) and (18) respectively 10:
Revised: Compute advantage, cost-advantage using (17) and (18) respectively 10:

Original: Update λ by replacing J C ( π θ ) with J Csample ( π θ i ) and using an appropriate value of β in (26).
Revised: Update λ by replacing J C ( π θ ) with J Csample ( π θ i ) and using an appropriate value of β in (26).

Original: { Multiple gradient updates for actor and critic } 11: for k = 1 , . .
Revised: { Multiple gradient updates for actor and critic } 11: for k = 1 , . .

Original: , K do 12:
Revised: , K do 12:

Original: Compute J R ( π θ k ) , J C ( π θ k ) as in (19) and (20) respectively.
Revised: Compute J R ( π θ k ) , J C ( π θ k ) as in (19) and (20) respectively.

Original: Update parameters θ k using (9) 14:
Revised: Update parameters θ k using (9) 14:

Original: Update critic parameters ψ r and ψ c minimizing (12) and (13) respectively.
Revised: Update critic parameters ψ r and ψ c minimizing (12) and (13) respectively.

Original: 15: end for 16:
Revised: 15: end for 16:

Original: Compute Performance Ratio (PR) using (25) 17: end while 18: end for
Revised: Compute Performance Ratio (PR) using (25) 17: end while 18: end for

Original: For this we compute the Performance Ratio (PR) metric using ensemble models (see Kurutach et al.[2018]) that is defined as follows:
Revised: For this we compute the Performance Ratio (PR) metric using ensemble models (see Kurutach et al. [2018]) that is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where ζ R ( α i , θ t ) = (cid:80) Tt =

Revised: where ζ R ( α i , θ t ) = (cid:80) Tt =

Original: γ t R ( s t , a t , s t +1 ) , s 0 ∼ µ and ∀ t ≥ 0 : s t +1 ∼ P
Revised: γ t R ( s t , a t , s t +1 ) , s 0 ∼ µ and ∀ t ≥ 0 : s t +1 ∼ P

Original: α i ( ·| s t , a t ) , a t ∼π θ t ( ·| s t ) .
Revised: α i ( ·| s t , a t ) , a t ∼ π θ t ( ·| s t ) .

Original: This measures the ratio of the number of models in which policy is improved to the totalnumber of models in ensemble ( n ) .
Revised: This measures the ratio of the number of models in which policy is improved to the total number of models in ensemble ( n ) .

Original: If PR > PR threshold , we continue training using the samemodel, if not then we break and re-train our environment model on data collected from the newupdate policy.
Revised: If PR > PR threshold , we continue training using the same model, if not then we break and re-train our environment model on data collected from the new update policy.

Original: Another challenge that we encounter is the underestimation of J C ( π θ ) because of using truncatedhorizon (of length H in step 7 of Algorithm 1) to reduce the aggregation of error.
Revised: Another challenge that we encounter is the underestimation of J C ( π θ ) because of using truncated horizon (of length H in step 7 of Algorithm 1) to reduce the aggregation of error.

Original: This can potentially lead to constraint violating policies.
Revised: This can potentially lead to constraint violating policies.

Original: So we need to make the safety threshold( d ) stricter.
Revised: So we need to make the safety threshold ( d ) stricter.

Original: It might seem that changing this threshold ( d ) proportional to the truncated horizon,i.e., setting d :=
Revised: It might seem that changing this threshold ( d ) proportional to the truncated horizon, i.e., setting d :=

Original: d ∗ HT might work better where H is the truncated horizon and T is the originalenvironment horizon, but we found that this leads to constraint violations as well because we arelearning policy using data from the approximated environment model that would bring in errors.
Revised: d ∗ HT might work better where H is the truncated horizon and T is the original environment horizon, but we found that this leads to constraint violations as well because we are learning policy using data from the approximated environment model that would bring in errors.

Original: Hence the cost estimate of the policy (Step 8 in Algorithm 1) is error-prone as well.
Revised: Hence the cost estimate of the policy (Step 8 in Algorithm 1) is error-prone as well.

Original: To tacklethis issue, we make safety threshold stricter using a hyperparameter 0 ≤ β < 1 by modifying the
Revised: To tackle this issue, we make safety threshold stricter using a hyperparameter 0 ≤ β < 1 by modifying the

Original: Lagrange multiplier update as follows:
Revised: Lagrange multiplier update as follows:

Original: [Equation]
Revised: [Equation]

Original: The variation of expected cost returns and reward returns with respect to β is shown in Figure 1 on
Revised: The variation of expected cost returns and reward returns with respect to β is shown in Figure 1 on

Original: Safety gym PointGoal [Ray et al., 2019] environment.
Revised: Safety gym PointGoal [Ray et al., 2019] environment.

Original: We can observe that as we reduce β , theexpected cost returns reduce because cost limit becomes stricter but choosing too small a β leads tolow reward returns as well.
Revised: We can observe that as we reduce β , the expected cost returns reduce because cost limit becomes stricter but choosing too small a β leads to low reward returns as well.

Original:  Experimental Details and Results
Revised:  Experimental Details and Results

Original: We test our approach on Safety Gym environments 1 - PointGoal and CarGoal with modified staterepresentations as used in other model-based Safe RL approaches [Liu et al., 2020, Sikchi et al. ,2021a] which is more helpful in model-learning.
Revised: We test our approach on Safety Gym environments 1 - PointGoal and CarGoal with modified state representations as used in other model-based Safe RL approaches [Liu et al., 2020, Sikchi et al., 2021a] which is more helpful in model-learning.

Original: We increase the difficulty of PointGoal and CarGoalenvironments by increasing the number of hazards from 10 to 15.
Revised: We increase the difficulty of PointGoal and CarGoal environments by increasing the number of hazards from 10 to 15.

Original: In both environments, the aimof robots is to reach the goal position and have as few collisions with hazards as possible.
Revised: In both environments, the aim of robots is to reach the goal position and have as few collisions with hazards as possible.

Original: Wecompare our approach (MBPPO-Lagrangian) with Unconstrained PPO [Schulman et al. , 2017],model-free Safe RL approaches including Constrained Policy Optimization [Achiam et al., 2017],
Revised: We compare our approach (MBPPO-Lagrangian) with Unconstrained PPO [Schulman et al., 2017], model-free Safe RL approaches including Constrained Policy Optimization [Achiam et al., 2017],

Original: PPO-Lagrangian [Ray et al., 2019] and the model-based approach – safe-LOOP from Sikchi et al.[2021a].
Revised: PPO-Lagrangian [Ray et al., 2019] and the model-based approach – safe-LOOP from Sikchi et al. [2021a].

Original: The code for our approach is available here 2 .
Revised: The code for our approach is available here 2 .

Original: We run each algorithm with 8 random seeds for 450Kenvironment interactions.
Revised: We run each algorithm with 8 random seeds for 450K environment interactions.

Original: The hyper-parameter settings and other experimental details are given in
Revised: The hyper-parameter settings and other experimental details are given in

Original: Appendix A.
Revised: Appendix A.

Original: The performance of our approach and baseline algorithms is shown in Figure 2 for
Revised: The performance of our approach and baseline algorithms is shown in Figure 2 for

Original: PointGoal environment and Figure 3 for CarGoal environment.
Revised: PointGoal environment and Figure 3 for CarGoal environment.

Original: The left part of the plots represent Average Episode Return (average of the episode rewards) ony-axis and total environment interactions on the x-axis.

Original: The dashed lines represent performance ofthe PPO-Lagrangian (in black) at convergence which is around 2 million environment interactions.

Original: The right part of the plots show Average Episode Cost (average of the episode costs) on y-axis andtotal environment interactions on the x-axis.

Original: The red dashed line in this plot represents cost limitof 18.

Original: From the plots we can observe that our approach (MBPPO-Lagrangian) converges to thesame level of reward performance as PPO - Lagrangian in just 450K environment interactions andoutperforms the model-based safe-LOOP algorithm
Revised: From the plots we can observe that our approach (MBPPO-Lagrangian) converges to the same level of reward performance as PPO - Lagrangian in just 450K environment interactions and outperforms the model-based safe-LOOP algorithm

Original: [Sikchi et al., 2021a].
Revised: [Sikchi et al., 2021a].

Original: Also our approach givesconstraint adhering policies in both the tasks.
Revised: Also our approach gives constraint adhering policies in both the tasks.

Revised: In addition to above, we run our algorithm on various values of β for both

Revised: CarGoal and RC-Car [Ahn, 2019] in similar manner as we did for PointGoal in Figure 1 and present it in Appendix D.

Revised: We also compare our algorithm with safe-LOOP[Sikchi et al., 2021a] on RC-Car environment and present it in Appendix D as well.

Revised: We found both approaches competitive in RC-Car environment, although safe-LOOP [Sikchi et al., 2021a] exhibits higher variance.

Original: We also measure cumulative hazard violations thatoccur till convergence for  PPO-Lagrangian Ray et al.
Revised: We also measure cumulative hazard violations that occur till convergence for MBPPO-L (ours), PPO-Lagrangian Ray et al.

Original: [2019], safe-LOOP
Revised: [2019], safe-LOOP

Original: [Sikchi et al., 2021a] as 
Revised: [Sikchi et al., 2021a] as follows -

Original: Cumulative V iolations = (cid:80) Till convergence [ 1 ( C ( s t , a t ) = = 1)] .
Revised: [Equation]

Original: We use rliable library (See Sec4.3 in Agarwal et al.
Revised: We use rliable library (See Sec 4.3 in Agarwal et al.

Original: [2021] for more details) for plotting 95% confidence intervals for cumulativeviolations in the right part of Figure 4 (normalized by cumulative violations in unconstrained PPOin respective tasks) and final policy reward performance in the left part of Figure 4 (normalizedby reward return of the final policy in unconstrained PPO in respective tasks) using mean, medianand inter-quartile mean as aggregate estimates for our approach, PPO-Lagrangian and safe-LOOPbecause these approaches only give constraint satisfying policies at convergence.
Revised: [2021] for more details) for plotting 95% confidence intervals for cumulative violations in the right part of Figure 4 (normalized by cumulative violations in unconstrained PPO in respective tasks) and final policy reward performance in the left part of Figure 4 (normalized by reward return of the final policy in unconstrained PPO in respective tasks) using mean, median and inter-quartile mean as aggregate estimates for our approach, PPOLagrangian and safe-LOOP because these approaches only give constraint satisfying policies at convergence.

Original: From figure 4we can observe that our approach is competitive to the model-based safe-LOOP approach [Sikchiet al., 2021a] in terms of cumulative constraint violations but achieves better reward performance atconvergence.
Revised: From figure 4 we can observe that our approach is competitive to the model-based safe-LOOP approach [Sikchi et al., 2021a] in terms of cumulative constraint violations but achieves better reward performance at convergence.

Original:  Conclusions
Revised:  Conclusions

Original: We presented a sample efficient approach for Safe Reinforcement Learning and compared ourapproach with relevant model-free and model-based baselines [Achiam et al., 2017, Sikchi et al. ,2021a, Ray et al., 2019, Schulman et al., 2017].
Revised: We presented a sample efficient approach for Safe Reinforcement Learning and compared our approach with relevant model-free and model-based baselines [Achiam et al., 2017, Sikchi et al., 2021a, Ray et al., 2019, Schulman et al., 2017].

Original: Note that we chose the above baselines because theyare specifically designed to solve constrained optimization problem of the same structure as in 1-3.
Revised: Note that we chose the above baselines because they are specifically designed to solve constrained optimization problem of the same structure as in 1-3.

Original: In Sikchi et al.
Revised: In Sikchi et al.

Original: [2021a], approximation of the value function is used to provide long-term reasoninginstead of using a truncated horizon.
Revised: [2021a], approximation of the value function is used to provide long-term reasoning instead of using a truncated horizon.

Original: A limitation of this approach lies in the fact that approximationsof reward and cost value functions are used.
Revised: A limitation of this approach lies in the fact that approximations of reward and cost value functions are used.

Original: The most challenging part of model-based approaches isto learn the environment model.
Revised: The most challenging part of model-based approaches is to learn the environment model.

Original: The first issue is of computational resources and the time overheadthat is needed.
Revised: The first issue is of computational resources and the time overhead that is needed.

Original: We provide a comparison of the running time of algorithms and their hardwarerequirements (see Appendix B).
Revised: We provide a comparison of the running time of algorithms and their hardware requirements (see Appendix B).

Original: Our algorithm does much better than safe-LOOP [Sikchi et al. ,2021a] in terms of running time.
Revised: Our algorithm does much better than safe-LOOP [Sikchi et al., 2021a] in terms of running time.

Original: One should note that in safe exploration settings, agents do not explore as much as an unconstrainedagent would do.
Revised: One should note that in safe exploration settings, agents do not explore as much as an unconstrained agent would do.

Original: This adds to the complexity of model learning in safe exploration problems becauseconstrained exploration leads to limited representation of data points for model learning even withthe same sample size.
Revised: This adds to the complexity of model learning in safe exploration problems because constrained exploration leads to limited representation of data points for model learning even with the same sample size.

Original: We found this more pronounced in high-dimensional environments like
Revised: We found this more pronounced in high-dimensional environments like

Original: DoggoGoal1 [Ray et al., 2019]
Revised: DoggoGoal1 [Ray et al., 2019]

Original: (See Appendix C).
Revised: (See Appendix C).

Original: Our algorithm depends on Lagrangian-basedapproach which provides a very simplistic way to construct a safe RL algorithm but suffers from theissue of low reward performance compared to unconstrained approaches (see Appendix A for rewardcomparison).
Revised: Our algorithm depends on Lagrangian-based approach which provides a very simplistic way to construct a safe RL algorithm but suffers from the issue of low reward performance compared to unconstrained approaches (see Appendix A for reward comparison).

Original: Note that we can have a similar model-based approach using TRPO-Lagrangian [Rayet al., 2019] but it involves an extra overhead of approximating the Fisher Information Matrix (FIM)and hence we chose a PPO-based approach.
Revised: Note that we can have a similar model-based approach using TRPO-Lagrangian [Ray et al., 2019] but it involves an extra overhead of approximating the Fisher Information Matrix (FIM) and hence we chose a PPO-based approach.

Original: Also TRPO-Lagrangian and PPO-Lagrangian havesimilar performance on Safety Gym benchmark [Ray et al., 2019].
Revised: Also TRPO-Lagrangian and PPO-Lagrangian have similar performance on Safety Gym benchmark [Ray et al., 2019].

Original: Increasing reward performance of
Revised: Increasing reward performance of

Original: Lagrangian-based approaches and devising better ways for model-learning in high-dimensional staterepresentations in safe RL setting where exploration is limited, can be looked at  future.
Revised: Lagrangian-based approaches and devising better ways for model-learning in high-dimensional state representations in safe RL setting where exploration is limited, can be looked at in the future.

Original: Checklist
Revised: Acknowledgments and Disclosure of Funding

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work? [Yes] We describe limitations in 
Revised: Ashish K. Jayant was supported in his work through MHRD scholarship by Ministry of Education,

Original: Conclusion section (c) Did you discuss any potential negative societal impacts of your work? [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: Government of India; Indian Institute of Science, Bangalore; and Flipkart Internet Pvt.

Original: them?
Revised: Ltd., Bangalore.

Original: [Yes] We have read the ethics review guidelines and ensured that our workconforms to them. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A]
Revised: S.Bhatnagar was supported in his work through the J

Original: (b) Did you include complete proofs of all theoretical results? [N/A] If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: C.Bose National Fellowship, Science and Engineering Research Board, Government of India; Project No. DST/ICPS/CLUSTER/Data

Original: mental results (either in the supplemental material or as a URL)? [Yes] We provide the 
Revised: Science/2018/General/23 from the Department of Science and Technology.

Original: anonymous url for our code repository (b) Did you specify all the training details(e.g., data splits, hyperparameters, how they
Revised: Government of India; and the Robert Bosch Centre for Cyber Physical Systems, Indian Institute of Science, Bangalore. A Hyper-parameters and finer experimental details

Original: were chosen)? [Yes] Hyperparameter details are given in appendix A
Revised: The hyper-parameters used for our algorithm are shown in Table 1.

Original: (c) Did you report error bars (e.g., with respect to  the random seed after running experi-
Revised: Each episode runs for 750 steps as opposed to 1000 steps in the original version of Safety Gym, hence we chose cost threshold of 18 which is ∼ 75% of 25 used in the official Safety Gym benchmark paper.

Original: ments multiple times)?
Revised: In Goal-based Safety gym environments the aim is to reach the goal position (in green in Figure 1) with as few collisions as possible with hazards. If robot (in red in Figure 1) accesses ’hazard’ positions (in blue in Figure 1), the agent incurs a cost = 1. The ’Point’ robot has steering and throttle as action space while

Original: [Yes] Wereport plots with 95% confidence intervals and use rliable Agarwal et al.
Revised: [Table] ’Car’ robot has differential control. We use Performance Ratio (PR) threshold of 66%.

Original: [2021] library for robust aggregate estimates
Revised: The logic behind this number is that our agent should perform better in more than 50% of the models in the ensemble.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
Revised: In our case we train an ensemble of 8 models out of which we use the best 6 models with minimum validation losses to calculate PR.

Original: [Yes] We share details of hardware
Revised: We want our agent to perform better in at least 4 out of the 6 models that gives us the value as 66% .

Original: requirement in appendix B
Revised: As mentioned in the text we increase the number of hazards in PointGoal1 and CarGoal1 environments from 10 to 15 so as to increase their difficulty level. We run on 8 random seeds which is greater than 4 used in benchmarking previous unconstrained model-based approaches 3 . We use the official implementation of baselines from safe-LOOP 4 , CPOand PPO-Lagrangian 6 , respectively. In all baselines we use the same size of the neural networks as mentioned in Table 1 and the same number of models in ensemble for model learning for safe-LOOP.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: In safe-LOOP, we do not change the originally used planning horizon of H = 8 and cost threshold of 0 in the model optimization part. During first pass of Algorithm 1 (Line 5-17), we use 5% of real environment interaction data and 95% of imaginary rollout data, this is done to make use of real environment data that we collected to train model dynamics. After first pass, policy gets updated and after that we use purely imaginary data since this is an on-policy algorithm until PR goes below 66% . The values used for normalization in plot for Figure 4 are the following: Reward performance of

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: PPO-unconstrained = 20 and Cumulative constraint violations of PPO-unconstrained = 2e5. B Hardware requirements and Running time Minimum 4 GB GPU space is required for running both the model based approaches. We did not

Original: [Yes] We cite the authors
Revised: [Table] use multiprocessing in model-based approaches for data-collection from simulation. However, for model-free approaches we collected simulation data using 8 CPUs at a time. Running time for 450k steps for all baselines is given in Table 2.

Original:  of Safety Gym benchmark and their github code repository.
Revised: Note that for convergence PPO-Lagrangian required 849 ± 12.48 s and CPO required 1162 ± 8.87 s. The running time of our model-based approach can be improved by using parallel workers for data collection but still major part of training is spent in model learning. C Comparison of model learning validation loss (Unconstrained vs Safe RL) and performance in High-dimensional task We compare how model learning validation loss varies in Safe RL setting as opposed to unconstrained

Original: (b) Did you mention the license of the assets?
Revised: RL one. We plot validation loss vs no. of datapoints collected as our model-based algorithm progress (See Figure 6. We observe that validation error converges quickly in unconstrained setting, which is reflective of better exploration in unconstrained setting.

Original: [Yes] Their MIT License which allows without restriction, including without limitation the rights to use, copy, modify, merge,publish provided that license notice is included in our code repository (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: This plot is generated using 8 different initialization of neural network used for model learning. We do this for our modified PointGoal environment.

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: DoggoGoal1 is a challenging environment where a quadruped robot has to reach a goal position without colliding with obstacles.

Original: [N/A](e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: This task has higher dimensional action space (12-dimensional) than

Original: information or offensive content?
Revised: Point and Car robots which just needs steering and throttle in Safety Gym.

Original: [N/A] If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: On this task, even modelfree Lagrangian baselines suffer and get quite low reward performance.

Original: applicable?
Revised: Further, both safe-LOOP

Original: [N/A]
Revised: and

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
Revised: MBPPO-Lagrangian fails to match model-free baseline in terms of reward performance in DoggoGoal environment.

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: It signifies the additional challenge in model learning in case of high-dimensional environments.

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: In RC Car 7 environment, a car has to rotate within a circle with target velocity for earning rewards. If car goes out of circle, it incurs cost. We observe the similar trend as observed in Figure 1 of paper, lower the value of β lower the reward returns as agent explores pessimistically.

Original: [N/A]
Revised: We also ran MBPPO-L(ours) and safe-LOOP [Sikchi et al., 2021a] on RC-Car environment for 150k steps (5 random seeds) only as safe-LOOP has very high running time. We choose cost limit = 5. Results are presented in Figure 10. We can observe that both approaches has similar performance but safe-LOOP exhibits higher variance than our approach.