Original: Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Revised: Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Original: Abstract
Revised: Abstract

Original: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps.
Revised: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps.

Original: In the real world, this can limit the practicality of these algorithms as it can lead to potentially dangerous behavior.
Revised: In the real world, this can limit the practicality of these algorithms as it can lead to potentially dangerous behavior.

Original: Hence safe exploration is a critical issue in applying RL algorithms in the real world.
Revised: Hence safe exploration is a critical issue in applying RL algorithms in the real world.

Original: This problem has been recently well studied under the Constrained Markov Decision Process (CMDP) Framework, where in addition to single-stage rewards, an agent receives single-stage costs or penalties as well depending on the state transitions.
Revised: This problem has been recently well studied under the Constrained Markov Decision Process (CMDP) Framework, where in addition to single-stage rewards, an agent receives single-stage costs or penalties as well depending on the state transitions.

Original: The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a scalar value.
Revised: The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a scalar value.

Original: The goal then is to find a feasible policy that maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.
Revised: The goal then is to find a feasible policy that maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.

Original: We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using the Lagrangian Relaxation-based Proximal Policy
Revised: We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using the Lagrangian Relaxation-based Proximal Policy

Original: Optimization.
Revised: Optimization.

Original: We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning.
Revised: We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning.

Original: We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark - the Open AI Safety Gym.
Revised: We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark - the Open AI Safety Gym.

Original: We demonstrate that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches.
Revised: We demonstrate that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches.

Original: Further, our approach shows better reward performance than other constrained model-based approaches in the literature.
Revised: Further, our approach shows better reward performance than other constrained model-based approaches in the literature.

Original:  Introduction
Revised:  Introduction

Original: Deep Reinforcement Learning has provided exceptional results both in the case of discrete action settings [Mnih et al., 2013] as well as continuous action domains such as locomotion tasks [Haarnoja et al., 2018], [Schulman et al., 2017], [Schulman et al., 2016].
Revised: Deep Reinforcement Learning has provided exceptional results both in the case of discrete action settings [Mnih et al., 2013] as well as continuous action domains such as locomotion tasks [Haarnoja et al., 2018], [Schulman et al., 2017], [Schulman et al., 2016].

Original: However, most of the RL algorithms perform significant number of random exploratory steps during learning as well as deployment which can lead to agents performing undesirable and hazardous behaviour.
Revised: However, most of the RL algorithms perform significant number of random exploratory steps during learning as well as deployment which can lead to agents performing undesirable and hazardous behaviour.

Original: This limits the application of
Revised: This limits the application of

Original: RL algorithms in the real world.
Revised: RL algorithms in the real world.

Original: In scenarios like robot navigation [Han et al., 2018], autonomous driving [Kendall et al., 2018], healthcare [Yu et al., 2020], etc., where RL has potential applications, unsafe behavior can have hazardous consequences even on human life and property.
Revised: In scenarios like robot navigation [Han et al., 2018], autonomous driving [Kendall et al., 2018], healthcare [Yu et al., 2020], etc., where RL has potential applications, unsafe behavior can have hazardous consequences even on human life and property.

Original: In García et al.
Revised: In García et al.

Original: [2015], the authors have provided a comprehensive survey of several notions of safety and the associated problem formulations.
Revised: [2015], the authors have provided a comprehensive survey of several notions of safety and the associated problem formulations.

Original: In our work, we focus on a constraint-based notion of safety.
Revised: In our work, we focus on a constraint-based notion of safety.

Original: In Constraint-based RL, the goal is to maximize long-term expected reward returns and keep the expected cost-returns below a prescribed threshold.
Revised: In Constraint-based RL, the goal is to maximize long-term expected reward returns and keep the expected cost-returns below a prescribed threshold.

Original: This problem is known as the Safe
Revised: This problem is known as the Safe

Original: Exploration problem.
Revised: Exploration problem.

Original: In [Ray et al., 2019], the authors advocate that safety specifications should be separate from task performance specifications.
Revised: In [Ray et al., 2019], the authors advocate that safety specifications should be separate from task performance specifications.

Original: It also helps in formulating Safe Exploration as a
Revised: It also helps in formulating Safe Exploration as a

Original: Constrained Optimization Problem and methods used in the optimization literature [Bertsekas, 1996] can be used to solve this problem.
Revised: Constrained Optimization Problem and methods used in the optimization literature [Bertsekas, 1996] can be used to solve this problem.

Original: Constrained Markov Decision Process (CMDP)
Revised: Constrained Markov Decision Process (CMDP)

Original: [Altman, 1998] provides a framework to keep task performance specifications and safety specifications separate from one another.
Revised: [Altman, 1998] provides a framework to keep task performance specifications and safety specifications separate from one another.

Original: Here in addition to single-stage rewards, state transitions receive single-stage costs as well.
Revised: Here in addition to single-stage rewards, state transitions receive single-stage costs as well.

Original: The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a non-negative scalar value.
Revised: The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a non-negative scalar value.

Original: The existing model-free algorithms used in the Constrained-RL setting suffer from low sample efficiency in terms of environment interactions, i.e., these algorithms require large number of environment interactions (to converge) that in turn would lead to a large number of hazardous actions due to unsafe exploration.
Revised: The existing model-free algorithms used in the Constrained-RL setting suffer from low sample efficiency in terms of environment interactions, i.e., these algorithms require large number of environment interactions (to converge) that in turn would lead to a large number of hazardous actions due to unsafe exploration.

Original: This serves as the motivation for us to use a Model-based approach for
Revised: This serves as the motivation for us to use a Model-based approach for

Original: Constrained-RL.
Revised: Constrained-RL.

Original: Our Contributions: We propose a simple and sample efficient model-based approach for Safe
Revised: Our Contributions: We propose a simple and sample efficient model-based approach for Safe

Original: Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem.
Revised: Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem.

Original: We highlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a way to incorporate that in our setting.
Revised: We highlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a way to incorporate that in our setting.

Original: We demonstrate that our approach is ∼ 3 − 4 times more sample efficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulative hazard violations by ∼ 60% .
Revised: We demonstrate that our approach is ∼ 3 − 4 times more sample efficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulative hazard violations by ∼ 60% .

Original: Our approach also outperforms model-free Constrained
Revised: Our approach also outperforms model-free Constrained

Original: Policy Optimization (CPO)
Revised: Policy Optimization (CPO)

Original: [Achiam et al., 2017] in terms of constraint satisfaction.
Revised: [Achiam et al., 2017] in terms of constraint satisfaction.

Original: We also compare our approach with model-based approach [Sikchi et al., 2021a] and we observe that our approach does better in terms of reward performance and has competitive cost performance as well.
Revised: We also compare our approach with model-based approach [Sikchi et al., 2021a] and we observe that our approach does better in terms of reward performance and has competitive cost performance as well.

Original:  Related Work
Revised:  Related Work

Original: Different notions of safety and their mathematical formulations are provided in García et al.
Revised: Different notions of safety and their mathematical formulations are provided in García et al.

Original: There are several works on the lines of formulating the Safe RL problem in the setting of CMDP.
Revised: There are several works on the lines of formulating the Safe RL problem in the setting of CMDP.

Original: In some of the early works, an actor-critic algorithm for CMDP under the long-run average cost criterion is proposed in Borkar [2005] that is however for the case of full state representations.
Revised: In some of the early works, an actor-critic algorithm for CMDP under the long-run average cost criterion is proposed in Borkar [2005] that is however for the case of full state representations.

Original: Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and Lakshmanan [2012], Bhatnagar et al.
Revised: Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and Lakshmanan [2012], Bhatnagar et al.

Original: [2013] for the long-run average cost setting and in Bhatnagar [2010] for the infinite horizon discounted cost scenario.
Revised: [2013] for the long-run average cost setting and in Bhatnagar [2010] for the infinite horizon discounted cost scenario.

Original: The procedure in the aforementioned references involved forming a Lagrangian by relaxing the constraints.
Revised: The procedure in the aforementioned references involved forming a Lagrangian by relaxing the constraints.

Original: The algorithms in these papers are based on multi-timescale stochastic approximation with updates of the Lagrange parameter performed on the slow timescale, the policy updates on the medium timescale and the updates of the value function (for a given policy) performed on the fast timescale.
Revised: The algorithms in these papers are based on multi-timescale stochastic approximation with updates of the Lagrange parameter performed on the slow timescale, the policy updates on the medium timescale and the updates of the value function (for a given policy) performed on the fast timescale.

Original: In more recent work, in Achiam et al.
Revised: In more recent work, in Achiam et al.

Original: [2017], a trust region based constrained policy optimization (CPO) framework is proposed, which involved approximation of the problem using surrogate functions for both the objective and the constraints and included a projection step on policy parameters that needed backtracking line search, making it complicated and time-consuming.
Revised: [2017], a trust region based constrained policy optimization (CPO) framework is proposed, which involved approximation of the problem using surrogate functions for both the objective and the constraints and included a projection step on policy parameters that needed backtracking line search, making it complicated and time-consuming.

Original: CPO showed near constraint satisfaction in every iteration of the policy updates in standard Mujoco environments modified for safe exploration [Achiam et al., 2017] but didn’t yield a constraint satisfying policy in challenging Safe RL benchmark Safety Gym [Ray et al., 2019].
Revised: CPO showed near constraint satisfaction in every iteration of the policy updates in standard Mujoco environments modified for safe exploration [Achiam et al., 2017] but didn’t yield a constraint satisfying policy in challenging Safe RL benchmark Safety Gym [Ray et al., 2019].

Original: Another work [Yu et al., 2019] involved using surrogate functions for approximation of both objective and constraint functions.
Revised: Another work [Yu et al., 2019] involved using surrogate functions for approximation of both objective and constraint functions.

Original: Their procedure involved constructing a sequence of convex optimization problems for which they showed that the sequence of stationary points converges to the stationary point of the original non-convex problem.
Revised: Their procedure involved constructing a sequence of convex optimization problems for which they showed that the sequence of stationary points converges to the stationary point of the original non-convex problem.

Original: In Ray et al.
Revised: In Ray et al.

Original: [2019], Lagrangian relaxation of the Constrained RL problem is used and combined with PPO [Schulman et al., 2017] to give a PPO-Lagrangian algorithm and with TRPO [Schulman et al., 2015] to give a TRPO-Lagrangian algorithm.
Revised: [2019], Lagrangian relaxation of the Constrained RL problem is used and combined with PPO [Schulman et al., 2017] to give a PPO-Lagrangian algorithm and with TRPO [Schulman et al., 2015] to give a TRPO-Lagrangian algorithm.

Original: These algorithms were seen to outperform CPO [Achiam et al., 2017] in terms of constraint satisfaction on several environments in Safety Gym.
Revised: These algorithms were seen to outperform CPO [Achiam et al., 2017] in terms of constraint satisfaction on several environments in Safety Gym.

Original: Also, these algorithms are simpler to implement and tune.
Revised: Also, these algorithms are simpler to implement and tune.

Original: Another Lagrangian-based method, see Tessler et al.
Revised: Another Lagrangian-based method, see Tessler et al.

Original: [2018], used a penalized reward function for optimizing their agent and showed convergence to optimal feasible policies using a two-timescale stochastic approximation scheme where the Lagrange multiplier is updated on a slower timescale as compared to the policy parameters as was the case with Borkar [2005], Bhatnagar [2010], Bhatnagar and Lakshmanan [2012].
Revised: [2018], used a penalized reward function for optimizing their agent and showed convergence to optimal feasible policies using a two-timescale stochastic approximation scheme where the Lagrange multiplier is updated on a slower timescale as compared to the policy parameters as was the case with Borkar [2005], Bhatnagar [2010], Bhatnagar and Lakshmanan [2012].

Original: In Zhang et al.
Revised: In Zhang et al.

Original: [2020], the authors proposed a first order constrained policy optimization (FOCOPS) method that involved solving the optimization problem in a non-parametric space and then projecting it back into the parametric space.
Revised: [2020], the authors proposed a first order constrained policy optimization (FOCOPS) method that involved solving the optimization problem in a non-parametric space and then projecting it back into the parametric space.

Original: Approaches in Tessler et al.
Revised: Approaches in Tessler et al.

Original: [2018] and Zhang et al.
Revised: [2018] and Zhang et al.

Original: [2020] performed poorly on Safety Gym.
Revised: [2020] performed poorly on Safety Gym.

Original: In Stooke et al.
Revised: In Stooke et al.

Original: [2020], a PID-based approach to damp oscillations in Lagrangian methods is proposed, which is seen to minimize constraint violations.
Revised: [2020], a PID-based approach to damp oscillations in Lagrangian methods is proposed, which is seen to minimize constraint violations.

Original: In Wen and Topcu [2021], the
Revised: In Wen and Topcu [2021], the

Original: Cross Entropy method [de Boer et al., 2005] is used for finding a safe policy and convergence using the ODE method is shown, however, empirical results are presented only on a primitive and less challenging environment.
Revised: Cross Entropy method [de Boer et al., 2005] is used for finding a safe policy and convergence using the ODE method is shown, however, empirical results are presented only on a primitive and less challenging environment.

Original: In Dalal et al.
Revised: In Dalal et al.

Original: [2018], the authors formulated a state-wise constrained policy optimization problem where at each transition a constraint needs to be satisfied and an analytical method for correcting the unsafe action using a safety layer trained using random exploration was proposed.
Revised: [2018], the authors formulated a state-wise constrained policy optimization problem where at each transition a constraint needs to be satisfied and an analytical method for correcting the unsafe action using a safety layer trained using random exploration was proposed.

Original: In Chow et al.
Revised: In Chow et al.

Original: [2018], the authors proposed constructing Lyapunov functions to guarantee safety of the behaviour policy under a CMDP framework.
Revised: [2018], the authors proposed constructing Lyapunov functions to guarantee safety of the behaviour policy under a CMDP framework.

Original: In Chow et al.
Revised: In Chow et al.

Original: [2019], the above Lyapunov based method was extended to continuous control
Revised: [2019], the above Lyapunov based method was extended to continuous control

Original: but it’s performance in Safety Gym environment [Ray et al., 2019] was not good in terms of rewards obtained [Sikchi et al., 2021b].
Revised: but it’s performance in Safety Gym environment [Ray et al., 2019] was not good in terms of rewards obtained [Sikchi et al., 2021b].

Original: In Liu et al.
Revised: In Liu et al.

Original: [2022], the method involved uses the tangent space of the constraint manifold to learn a safe policy but the approach is seen to be highly specific to the environment used.
Revised: [2022], the method involved uses the tangent space of the constraint manifold to learn a safe policy but the approach is seen to be highly specific to the environment used.

Original: For unconstrained state-of-the-art Model-based RL algorithms in Deisenroth and Rasmussen [2011],
Revised: For unconstrained state-of-the-art Model-based RL algorithms in Deisenroth and Rasmussen [2011],

Original: Luo et al.
Revised: Luo et al.

Original: [2018], Chua et al.
Revised: [2018], Chua et al.

Original: [2018], Kurutach et al.
Revised: [2018], Kurutach et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2015], a comparison of the empirical performance on Open AI
Revised: [2015], a comparison of the empirical performance on Open AI

Original: Gym [Brockman et al., 2016] is shown in Wang et al.
Revised: Gym [Brockman et al., 2016] is shown in Wang et al.

Original: Also, Sikchi et al.
Revised: Also, Sikchi et al.

Original: [2021a] propose augmenting the planning trajectory with terminal value function to incorporate long-horizon reasoning in model-based methods, since the approaches in Deisenroth and
Revised: [2021a] propose augmenting the planning trajectory with terminal value function to incorporate long-horizon reasoning in model-based methods, since the approaches in Deisenroth and

Original: Rasmussen [2011], Luo et al.
Revised: Rasmussen [2011], Luo et al.

Original: [2018], Chua et al.
Revised: [2018], Chua et al.

Original: [2018], Kurutach et al.
Revised: [2018], Kurutach et al.

Original: [2018], Heess et al.
Revised: [2018], Heess et al.

Original: [2015] plan over a fixed and short horizon to avoid aggregation of error.
Revised: [2015] plan over a fixed and short horizon to avoid aggregation of error.

Original: This requires good approximation of both model as well as value function.
Revised: This requires good approximation of both model as well as value function.

Original: There are also several works that use model-based RL to tackle the problem of safety.
Revised: There are also several works that use model-based RL to tackle the problem of safety.

Original: In Liu et al.
Revised: In Liu et al.

Original: [2020], a model based approach is proposed to learn the system dynamics and cost model.
Revised: [2020], a model based approach is proposed to learn the system dynamics and cost model.

Original: Then roll-outs from the learned model are used to optimize the policy using a modified cross-entropy based method which involves sampling from a distribution of policies, sorting sample policies based on constraint functions and using them to update the policy distribution.
Revised: Then roll-outs from the learned model are used to optimize the policy using a modified cross-entropy based method which involves sampling from a distribution of policies, sorting sample policies based on constraint functions and using them to update the policy distribution.

Original: However, their implementation involves a data collection step using random policy for large number of episodes which itself is risky in real-world scenarios.
Revised: However, their implementation involves a data collection step using random policy for large number of episodes which itself is risky in real-world scenarios.

Original: In Cowen-Rivers et al.
Revised: In Cowen-Rivers et al.

Original: [2020], model dynamics is learned using PILCO [Deisenroth and Rasmussen, 2011] and instead of the discounted cost constraint function, conditional value at risk (CVaR) based constraint function is used [Uryasev and Rockafellar, 2001, Chow and
Revised: [2020], model dynamics is learned using PILCO [Deisenroth and Rasmussen, 2011] and instead of the discounted cost constraint function, conditional value at risk (CVaR) based constraint function is used [Uryasev and Rockafellar, 2001, Chow and

Original: Ghavamzadeh, 2014].
Revised: Ghavamzadeh, 2014].

Original: In Thomas et al.
Revised: In Thomas et al.

Original: [2022], penalized reward functions are used instead of a separate cost function, then model of the environment is learned and the soft-actor critic algorithm [Haarnoja et al., 2018] is used to optimize the policy.
Revised: [2022], penalized reward functions are used instead of a separate cost function, then model of the environment is learned and the soft-actor critic algorithm [Haarnoja et al., 2018] is used to optimize the policy.

Original: In this approach safe and unsafe states are also needed to be specified upfront.
Revised: In this approach safe and unsafe states are also needed to be specified upfront.

Original:  Background 3.1 Constrained Markov Decision Process (CMDP)
Revised:  Background 3.1 Constrained Markov Decision Process (CMDP)

Original: A CMDP is denoted by the tuple ( S, A, R, C, γ, µ ) where S denotes the state space, A is the action space, R : S ×
Revised: A CMDP is denoted by the tuple ( S, A, R, C, γ, µ ) where S denotes the state space, A is the action space, R : S ×

Original: A × S → R is the single-stage reward function, C : S ×
Revised: A × S → R is the single-stage reward function, C : S ×

Original: A × S → R denotes the associated single-stage cost function (we assume a single constraint function for simplicity here), γ is the discount factor and µ signifies the initial state distribution.
Revised: A × S → R denotes the associated single-stage cost function (we assume a single constraint function for simplicity here), γ is the discount factor and µ signifies the initial state distribution.

Original: We assume that both R and C are non-negative functions.
Revised: We assume that both R and C are non-negative functions.

Original: By a policy π = { π 0 , π 1 , . . . } , we mean a decision rule for selecting actions.
Revised: By a policy π = { π 0 , π 1 , . . . } , we mean a decision rule for selecting actions.

Original: It is specified as follows:
Revised: It is specified as follows:

Original: For any k ≥ 0 and s ∈ S , π k ( s ) ∈ P ( s ) is the probability
Revised: For any k ≥ 0 and s ∈ S , π k ( s ) ∈ P ( s ) is the probability

Original: [Equation]
Revised: [Equation]

Original: where π k ( s, a ) is the probability of picking action a in state s at instant k under policy π .
Revised: where π k ( s, a ) is the probability of picking action a in state s at instant k under policy π .

Original: In the above, A ( s ) is the set of feasible actions in state s and so A = ∪ s ∈ S A ( s ) .
Revised: In the above, A ( s ) is the set of feasible actions in state s and so A = ∪ s ∈ S A ( s ) .

Original: Such a policy is also often referred to as a randomized policy.
Revised: Such a policy is also often referred to as a randomized policy.

Original: A stationary policy is a randomized policy as above except with π k = π l , ∀ k ̸ = l .
Revised: A stationary policy is a randomized policy as above except with π k = π l , ∀ k ̸ = l .

Original: Thus, a stationary policy selects actions according to a given distribution regardless of the instant when an action is chosen according to the given policy.
Revised: Thus, a stationary policy selects actions according to a given distribution regardless of the instant when an action is chosen according to the given policy.

Original: By an abuse of notation, we denote a stationary policy as π itself.
Revised: By an abuse of notation, we denote a stationary policy as π itself.

Original: We shall consider here a class of stationary policies π θ parameterized by a parameter θ .
Revised: We shall consider here a class of stationary policies π θ parameterized by a parameter θ .

Original: Our objective function is defined via the infinite horizon discounted reward criterion where for a given π θ we have
Revised: Our objective function is defined via the infinite horizon discounted reward criterion where for a given π θ we have

Original: [Equation]
Revised: [Equation]

Original: The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:
Revised: The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:

Original: [Equation]
Revised: [Equation]

Original: Then J R ( π θ ) , J C ( π θ ) ∈ R .
Revised: Then J R ( π θ ) , J C ( π θ ) ∈ R .

Original: Let d > 0 denote a prescribed threshold below which we want J C ( π θ ) to lie.
Revised: Let d > 0 denote a prescribed threshold below which we want J C ( π θ ) to lie.

Original: The constrained optimization problem then is the following:
Revised: The constrained optimization problem then is the following:

Original: [Equation]
Revised: [Equation]

Original: A parameter θ will be called a feasible point if the cost constraint is satisfied for θ , i.e., J C ( π θ ) ≤ d .
Revised: A parameter θ will be called a feasible point if the cost constraint is satisfied for θ , i.e., J C ( π θ ) ≤ d .

Original: 3.2 Lagrangian Relaxation based Proximal Policy Optimization
Revised: 3.2 Lagrangian Relaxation based Proximal Policy Optimization

Original: The Lagrangian of the constrained optimization problem (3) can be written as follows:
Revised: The Lagrangian of the constrained optimization problem (3) can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ R + is the Lagrange multiplier and is a positive real number.
Revised: where λ ∈ R + is the Lagrange multiplier and is a positive real number.

Original: In terms of the Lagrangian, the goal is to find a tuple ( θ ∗ , λ ∗ ) of the policy and Lagrange parameter such that
Revised: In terms of the Lagrangian, the goal is to find a tuple ( θ ∗ , λ ∗ ) of the policy and Lagrange parameter such that

Original: [Equation]
Revised: [Equation]

Original: Solving the max-min problem as above is equivalent to finding a global optimal saddle point ( θ ∗ , λ ∗ ) such that ∀ ( θ, λ ) , the following holds:
Revised: Solving the max-min problem as above is equivalent to finding a global optimal saddle point ( θ ∗ , λ ∗ ) such that ∀ ( θ, λ ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: We assume that θ refers to the parameter of a Deep Neural Network, hence finding such a globally optimal saddle point is computationally hard.
Revised: We assume that θ refers to the parameter of a Deep Neural Network, hence finding such a globally optimal saddle point is computationally hard.

Original: So our aim is to find a locally optimal saddle point which satisfies (6) in a local neighbourhood H ϵ 1 ,ϵ 2 which is defined as follows:
Revised: So our aim is to find a locally optimal saddle point which satisfies (6) in a local neighbourhood H ϵ 1 ,ϵ 2 which is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: for some ϵ 1 , ϵ 2 > 0 .
Revised: for some ϵ 1 , ϵ 2 > 0 .

Original: Assuming that L ( θ, λ ) is known for every ( θ, λ ) tuple, a gradient search procedure for finding a local ( θ ∗ , λ ∗ ) tuple would be the following:
Revised: Assuming that L ( θ, λ ) is known for every ( θ, λ ) tuple, a gradient search procedure for finding a local ( θ ∗ , λ ∗ ) tuple would be the following:

Original: [Equation]
Revised: [Equation]

Original: Here [ x ] + denotes max(0 , x ) .
Revised: Here [ x ] + denotes max(0 , x ) .

Original: This operator ensures that the Lagrange multiplier remains nonnegative after each update.
Revised: This operator ensures that the Lagrange multiplier remains nonnegative after each update.

Original: In (9)-(11), η 1 ( n ) , η 2 ( n ) > 0 ∀ n are certain prescribed step-size schedules.
Revised: In (9)-(11), η 1 ( n ) , η 2 ( n ) > 0 ∀ n are certain prescribed step-size schedules.

Original: We assume that the step-sizes η 1 ( n ) , η 2 ( n ) , n ≥ 0 satisfy the regular step-size conditions.
Revised: We assume that the step-sizes η 1 ( n ) , η 2 ( n ) , n ≥ 0 satisfy the regular step-size conditions.

Original: Thus, for i = 1 , 2 , (cid:88)
Revised: Thus, for i = 1 , 2 , (cid:88)

Original: k
Revised: k

Original: η i ( n ) = ∞ , (cid:88) k
Revised: η i ( n ) = ∞ , (cid:88) k

Original: η 2 i ( n ) < ∞ .
Revised: η 2 i ( n ) < ∞ .

Original: Note however that J R ( π θ ) and J C ( π θ ) as specified in (1)-(2) are not a priori known quantities and need to be estimated.
Revised: Note however that J R ( π θ ) and J C ( π θ ) as specified in (1)-(2) are not a priori known quantities and need to be estimated.

Original: We discuss this in detail below.
Revised: We discuss this in detail below.

Original: 3.2.1 Estimation
Revised: 3.2.1 Estimation

Original: We run each episode for T time steps in our experiments.
Revised: We run each episode for T time steps in our experiments.

Original: Let r t +1 ≡ R ( s t , a t , s t +1 ) and c t +1 ≡
Revised: Let r t +1 ≡ R ( s t , a t , s t +1 ) and c t +1 ≡

Original: C ( s t , a t , s t +1 ) , respectively, for simplicity.
Revised: C ( s t , a t , s t +1 ) , respectively, for simplicity.

Original: For each sample path we would have both a reward return as well as a cost return.
Revised: For each sample path we would have both a reward return as well as a cost return.

Original: Let ˆ R t (resp.
Revised: Let ˆ R t (resp.

Original: ˆ C t ) be the reward-to-go (resp.
Revised: ˆ C t ) be the reward-to-go (resp.

Original: cost-to-go) estimate.
Revised: cost-to-go) estimate.

Original: We compute
Revised: We compute

Original: ˆ R t and ˆ C t according to: ˆ R t =
Revised: ˆ R t and ˆ C t according to: ˆ R t =

Original: (cid:80) T − t − 1 k = γ k r t + k +1 , ˆ C t = (cid:80) T − t − 1 k = γ k c t + k +1 , respectively.
Revised: (cid:80) T − t − 1 k = γ k r t + k +1 , ˆ C t = (cid:80) T − t − 1 k = γ k c t + k +1 , respectively.

Original: We use a neural network parameterized by ψ r to estimate reward signal based value function V
Revised: We use a neural network parameterized by ψ r to estimate reward signal based value function V

Original: Rψ r and a neural network parameterized by ψ c to estimate a cost signal based value function V Cψ c .
Revised: Rψ r and a neural network parameterized by ψ c to estimate a cost signal based value function V Cψ c .

Original: We run our simulations on N parallel workers and then sample a mini-batch M of size M ≤ NT [Schulman et al., 2017].
Revised: We run our simulations on N parallel workers and then sample a mini-batch M of size M ≤ NT [Schulman et al., 2017].

Original: We use mean-squared loss to estimate value functions on sampled mini-batches as follows:
Revised: We use mean-squared loss to estimate value functions on sampled mini-batches as follows:

Original: [Equation]
Revised: [Equation]

Original: Let A Rt , t ≥ 0 , and A Ct , t ≥ 0 , respectively, denote the advantage estimates w.r.t reward and cost value functions on the sample path.
Revised: Let A Rt , t ≥ 0 , and A Ct , t ≥ 0 , respectively, denote the advantage estimates w.r.t reward and cost value functions on the sample path.

Original: We compute them using Generalized Advantage Estimation [Schulman et al., 2016] to balance the bias and variance tradeoff of advantage estimates.
Revised: We compute them using Generalized Advantage Estimation [Schulman et al., 2016] to balance the bias and variance tradeoff of advantage estimates.

Original: We can have a range of advantage estimates as under.
Revised: We can have a range of advantage estimates as under.

Original: [Equation]
Revised: [Equation]

Original: for k > 2 .
Revised: for k > 2 .

Original: The advantage estimate in (14) will have high bias but low variance while estimates in (16) with a higher value of k generally have high variance but low bias.
Revised: The advantage estimate in (14) will have high bias but low variance while estimates in (16) with a higher value of k generally have high variance but low bias.

Original: Let δ Rt
Revised: Let δ Rt

Original: =
Revised: =

Original: r t +1 + V
Revised: r t +1 + V

Original: Rψ r ( s t +1 ) −
Revised: Rψ r ( s t +1 ) −

Original: V
Revised: V

Original: Rψ r ( s t ) and δ Ct =
Revised: Rψ r ( s t ) and δ Ct =

Original: c t +1 + V Cψ c ( s t +1 ) − V Cψ c ( s t ) , respectively, denote the reward and cost temporal differences.
Revised: c t +1 + V Cψ c ( s t +1 ) − V Cψ c ( s t ) , respectively, denote the reward and cost temporal differences.

Original: Let ¯ λ be a parameter which adjusts the bias-variance tradeoff.
Revised: Let ¯ λ be a parameter which adjusts the bias-variance tradeoff.

Original: Generalized Advantage
Revised: Generalized Advantage

Original: Estimates [Schulman et al., 2016] for A Rt ,
Revised: Estimates [Schulman et al., 2016] for A Rt ,

Original: A Ct are then given by
Revised: A Ct are then given by

Original: [Equation]
Revised: [Equation]

Original: respectively.
Revised: respectively.

Original: Now we use PPO clipped objectives [Schulman et al., 2017] for estimation of
Revised: Now we use PPO clipped objectives [Schulman et al., 2017] for estimation of

Original: J R ( π θ ) , J C ( π θ ) as follows:
Revised: J R ( π θ ) , J C ( π θ ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where r t ( θ ) =
Revised: where r t ( θ ) =

Original: π θ ( a t | s t ) π θold ( a t | s t ) is the ratio of the probability of selecting action a t in state s t under parameter θ as opposed to θ old .
Revised: π θ ( a t | s t ) π θold ( a t | s t ) is the ratio of the probability of selecting action a t in state s t under parameter θ as opposed to θ old .

Original: Further, A Rt and A Ct are the estimated advantages based on the reward and cost returns, respectively, by time t (see above) and ϵ is the clip-ratio which clips r t ( θ ) to (1 − ϵ ) if it is less than (1 − ϵ ) and clips to (1 + ϵ ) if it is greater than (1 + ϵ ) .
Revised: Further, A Rt and A Ct are the estimated advantages based on the reward and cost returns, respectively, by time t (see above) and ϵ is the clip-ratio which clips r t ( θ ) to (1 − ϵ ) if it is less than (1 − ϵ ) and clips to (1 + ϵ ) if it is greater than (1 + ϵ ) .

Original: This algorithm restricts the policy parameters to not change significantly between two iterations which helps in avoiding divergence.
Revised: This algorithm restricts the policy parameters to not change significantly between two iterations which helps in avoiding divergence.

Original: This approach is referred to as PPO-Lagrangian [Ray et al., 2019].
Revised: This approach is referred to as PPO-Lagrangian [Ray et al., 2019].

Original: 3.3 Model-based Constrained RL
Revised: 3.3 Model-based Constrained RL

Original: We formulate a Constrained RL problem (21) using a model-based framework as follows:
Revised: We formulate a Constrained RL problem (21) using a model-based framework as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the above, P α ( . | s t , a t ) is an α -parameterized environment model, d i is a human prescribed safety threshold for the i th constraint and Π θ is the set of all θ − parameterized stationary polices.
Revised: In the above, P α ( . | s t , a t ) is an α -parameterized environment model, d i is a human prescribed safety threshold for the i th constraint and Π θ is the set of all θ − parameterized stationary polices.

Original: Note that we assume the initial state s 0 is sampled from the true initial state distribution µ
Revised: Note that we assume the initial state s 0 is sampled from the true initial state distribution µ

Original: and then s t +1 ∼ P
Revised: and then s t +1 ∼ P

Original: α ( .
Revised: α ( .

Original: | s t , a t ) , ∀ t > 0 .
Revised: | s t , a t ) , ∀ t > 0 .

Original: We would use approximation of environment P α to create ‘imaginary’ roll-outs to estimate the reward and cost returns required for policy optimization algorithms.
Revised: We would use approximation of environment P α to create ‘imaginary’ roll-outs to estimate the reward and cost returns required for policy optimization algorithms.

Original:  Challenges in Environment Model Learning
Revised:  Challenges in Environment Model Learning

Original: In this section we discuss the challenges that commonly arise due to model learning in RL.
Revised: In this section we discuss the challenges that commonly arise due to model learning in RL.

Original: We further highlight the challenge that arises from using environment model approximation in Safe RL settings.
Revised: We further highlight the challenge that arises from using environment model approximation in Safe RL settings.

Original: Handling aleatoric and epistemic uncertainties : Aleatoric Uncertainty refers to the notion of natural randomness in the system which leads to variability in outcomes of an experiment.
Revised: Handling aleatoric and epistemic uncertainties : Aleatoric Uncertainty refers to the notion of natural randomness in the system which leads to variability in outcomes of an experiment.

Original: This uncertainty is irreducible because it is a natural property of the system
Revised: This uncertainty is irreducible because it is a natural property of the system

Original: Hence in such cases, giving measure of uncertainty in model’s prediction is a good practice.
Revised: Hence in such cases, giving measure of uncertainty in model’s prediction is a good practice.

Original: In Lakshminarayanan et al.
Revised: In Lakshminarayanan et al.

Original: [2017], Chua et al.
Revised: [2017], Chua et al.

Original: [2018] uncertainty-aware neural networks are used which give an idea about uncertainty in prediction as well.
Revised: [2018] uncertainty-aware neural networks are used which give an idea about uncertainty in prediction as well.

Original: They learn Gaussian distribution parameterized by neural networks.
Revised: They learn Gaussian distribution parameterized by neural networks.

Original: Epistemic Uncertainty refers to the notion of lack of sufficient knowledge in the model as a result of which the model does not generalize.
Revised: Epistemic Uncertainty refers to the notion of lack of sufficient knowledge in the model as a result of which the model does not generalize.

Original: In Lakshminarayanan et al.
Revised: In Lakshminarayanan et al.

Original: [2017], an ensemble of uncertainty aware neural networks with different initializations is proposed to reduce epistemic uncertainty for fixed data.
Revised: [2017], an ensemble of uncertainty aware neural networks with different initializations is proposed to reduce epistemic uncertainty for fixed data.

Original: For the learning environment model, we also use an ensemble of n neural networks with random initialization.
Revised: For the learning environment model, we also use an ensemble of n neural networks with random initialization.

Original: Each neural network’s output parameterizes a multivariate normal distribution with diagonal covariance matrix.
Revised: Each neural network’s output parameterizes a multivariate normal distribution with diagonal covariance matrix.

Original: Suppose the i th neural network in the ensemble is parameterized by α i and the mean and standard deviation outputs are µ α i and σ α i respectively.
Revised: Suppose the i th neural network in the ensemble is parameterized by α i and the mean and standard deviation outputs are µ α i and σ α i respectively.

Original: Recall now that if a random vector X ∈ R d is distributed according to the multivariate normal distribution parameterized by ( µ, Σ d × d ) where µ ∈ R d is the mean vector and Σ d × d is a d × d covariance matrix, then the probability density of X is defined as, P ( x ) =
Revised: Recall now that if a random vector X ∈ R d is distributed according to the multivariate normal distribution parameterized by ( µ, Σ d × d ) where µ ∈ R d is the mean vector and Σ d × d is a d × d covariance matrix, then the probability density of X is defined as, P ( x ) =

Original: [Equation]
Revised: [Equation]

Original: As a choice of loss function we use the negative log-likelihood loss for minimization (i.e., minimizing negative log of P ( X ) ).
Revised: As a choice of loss function we use the negative log-likelihood loss for minimization (i.e., minimizing negative log of P ( X ) ).

Original: For the i th neural network parameterized by α i , the loss function L ( α i ) is given as follows:
Revised: For the i th neural network parameterized by α i , the loss function L ( α i ) is given as follows:

Original: [Equation]
Revised: [Equation]

Original: where µ α i ( s t , a t ) is the mean vector output of the i th neural network and
Revised: where µ α i ( s t , a t ) is the mean vector output of the i th neural network and

Original: Σ α i ( s t , a t ) is the covariance matrix which is assumed to be a diagonal matrix.
Revised: Σ α i ( s t , a t ) is the covariance matrix which is assumed to be a diagonal matrix.

Original: Note that, using n neural networks with random initialization tends to have a regularization effect.
Revised: Note that, using n neural networks with random initialization tends to have a regularization effect.

Original: Intuitively, it introduces diversity in learned models to deal with more possible trajectories.
Revised: Intuitively, it introduces diversity in learned models to deal with more possible trajectories.

Original: Also, it has been shown empirically that with an increase in the number of models, the performance tends to improve (See section 6.3 and Figure 4 of Kurutach et al.
Revised: Also, it has been shown empirically that with an increase in the number of models, the performance tends to improve (See section 6.3 and Figure 4 of Kurutach et al.

Original: [2018]), but increase in number of models also leads to increase in space complexity.
Revised: [2018]), but increase in number of models also leads to increase in space complexity.

Original: Aggregation of Error :
Revised: Aggregation of Error :

Original: In model-based RL, as we move forward along the horizon, the error due to approximation starts aggregating and predictions from the approximated model tend to diverge significantly from the true model.
Revised: In model-based RL, as we move forward along the horizon, the error due to approximation starts aggregating and predictions from the approximated model tend to diverge significantly from the true model.

Original: In order to tackle this problem, most of the model-based RL approaches [
Revised: In order to tackle this problem, most of the model-based RL approaches [

Original: Deisenroth and Rasmussen, 2011, Kurutach et al., 2018,
Revised: Deisenroth and Rasmussen, 2011, Kurutach et al., 2018,

Original: Janner et al., 2019]
Revised: Janner et al., 2019]

Original: use shorter (or truncated) horizon during the policy optimization phase and achieve similar performance as Model-Free RL approaches.
Revised: use shorter (or truncated) horizon during the policy optimization phase and achieve similar performance as Model-Free RL approaches.

Original: We use truncated horizon in our approach.
Revised: We use truncated horizon in our approach.

Original: Implication of using truncated horizon in Constrained RL : When we use truncated horizon in Constrained RL,
Revised: Implication of using truncated horizon in Constrained RL : When we use truncated horizon in Constrained RL,

Original: it
Revised: it

Original: leads to
Revised: leads to

Original: underestimation of cost returns (23) under the current policy
Revised: underestimation of cost returns (23) under the current policy

Original: and we use the prescribed constraints to threshold the returns obtained.
Revised: and we use the prescribed constraints to threshold the returns obtained.

Original: This can lead to constraint violations in the real-environment where the cost objective is based on the infinite horizon cost return.
Revised: This can lead to constraint violations in the real-environment where the cost objective is based on the infinite horizon cost return.

Original: We propose a hyperparameter-based approach to deal with this problem in the next section.
Revised: We propose a hyperparameter-based approach to deal with this problem in the next section.

Original:  Model-based PPO Lagrangian
Revised:  Model-based PPO Lagrangian

Original: We propose a model-based algorithm obtained from relaxing the Lagrangian (see Algorithm 1) which alleviates the problem with obtaining a large number of samples in model-free Lagrangian based approaches and as a consequence also decreases the cumulative hazard violations.
Revised: We propose a model-based algorithm obtained from relaxing the Lagrangian (see Algorithm 1) which alleviates the problem with obtaining a large number of samples in model-free Lagrangian based approaches and as a consequence also decreases the cumulative hazard violations.

Original: It is difficult to evaluate the policy without interacting with the real environment accurately.
Revised: It is difficult to evaluate the policy without interacting with the real environment accurately.

Original: 1: Input: Initialize actor neural net parameter θ 0 , critic parameters ψ r 0 , ψ c 0 , ensemble models [ P α i ] ni =1 , Lagrange parameter λ 0 ≥ 0 , cost threshold =
Revised: 1: Input: Initialize actor neural net parameter θ 0 , critic parameters ψ r 0 , ψ c 0 , ensemble models [ P α i ] ni =1 , Lagrange parameter λ 0 ≥ 0 , cost threshold =

Original: d , Environment Horizon = T , Model
Revised: d , Environment Horizon = T , Model

Original: Horizon =
Revised: Horizon =

Original: H 2: for i = 1 , . .
Revised: H 2: for i = 1 , . .

Original: , N training epochs do 3:
Revised: , N training epochs do 3:

Original: Collect data tuples ( s t , a t , s t +1 ) , for the i th ensemble using policy π θ i , i = 1 , . .
Revised: Collect data tuples ( s t , a t , s t +1 ) , for the i th ensemble using policy π θ i , i = 1 , . .

Original: . , n , in the environment for T time steps over multiple ( | E | ) episodes 4:
Revised: . , n , in the environment for T time steps over multiple ( | E | ) episodes 4:

Original: Train [ P α i ] ni =1 by minimizing (24) w.r.t α i , ∀ i = 1 to n 5: while Performance ratio > PR threshold do 6: s 0 ∼ µ { Note: For first pass we use a mix of real data and imaginary roll-out data.
Revised: Train [ P α i ] ni =1 by minimizing (24) w.r.t α i , ∀ i = 1 to n 5: while Performance ratio > PR threshold do 6: s 0 ∼ µ { Note: For first pass we use a mix of real data and imaginary roll-out data.

Original: (See Appendix
Revised: (See Appendix

Original: A) } 7:
Revised: A) } 7:

Original: Collect data roll-outs as a t ∼ π θ i ( ·| s t ) , s t ∼ P α q ( ·| s t , a t )
Revised: Collect data roll-outs as a t ∼ π θ i ( ·| s t ) , s t ∼ P α q ( ·| s t , a t )

Original: (At each time step ’ q ’ is randomly selected from 1 , . . . , n ) for H time steps ( H < T ) 8:
Revised: (At each time step ’ q ’ is randomly selected from 1 , . . . , n ) for H time steps ( H < T ) 8:

Original: Compute J Csample ( π θ t ) = 

Revised: Compute J Csample ( π θ t ) =

Original: | E | (cid:80) Hp =1 γ p C
Revised: | E | (cid:80) Hp =1 γ p C

Original: ( s t , a t ) where | E | is the number of episodes 9:
Revised: ( s t , a t ) where | E | is the number of episodes 9:

Original: Compute advantage, cost-advantage using (17) and (18) respectively 10:
Revised: Compute advantage, cost-advantage using (17) and (18) respectively 10:

Original: Update λ by replacing J C ( π θ ) with J Csample ( π θ i ) and using an appropriate value of β in (26).
Revised: Update λ by replacing J C ( π θ ) with J Csample ( π θ i ) and using an appropriate value of β in (26).

Original: { Multiple gradient updates for actor and critic } 11: for k = 1 , . .
Revised: { Multiple gradient updates for actor and critic } 11: for k = 1 , . .

Original: , K do 12:
Revised: , K do 12:

Original: Compute J R ( π θ k ) , J C ( π θ k ) as in (19) and (20) respectively.
Revised: Compute J R ( π θ k ) , J C ( π θ k ) as in (19) and (20) respectively.

Original: Update parameters θ k using (9) 14:
Revised: Update parameters θ k using (9) 14:

Original: Update critic parameters ψ r and ψ c minimizing (12) and (13) respectively.
Revised: Update critic parameters ψ r and ψ c minimizing (12) and (13) respectively.

Original: 15: end for 16:
Revised: 15: end for 16:

Original: Compute Performance Ratio ( PR ) using (25) 17: end while 18: end for
Revised: Compute Performance Ratio ( PR ) using (25) 17: end while 18: end for

Original: For this we compute the Performance Ratio ( PR ) metric using ensemble models (see Kurutach et al.
Revised: For this we compute the Performance Ratio ( PR ) metric using ensemble models (see Kurutach et al.

Original: [2018]) that is defined as follows:
Revised: [2018]) that is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where ζ R ( α i , θ t ) = (cid:80) Tt =

Revised: where ζ R ( α i , θ t ) = (cid:80) Tt =

Original: γ t R ( s t , a t , s t +1 ) , s 0 ∼ µ and ∀ t ≥ 0 : s t +1 ∼ P
Revised: γ t R ( s t , a t , s t +1 ) , s 0 ∼ µ and ∀ t ≥ 0 : s t +1 ∼ P

Original: α i ( ·| s t , a t ) , a t ∼ π θ t ( ·| s t ) .
Revised: α i ( ·| s t , a t ) , a t ∼ π θ t ( ·| s t ) .

Original: This measures the ratio of the number of models in which policy is improved to the total number of models in ensemble ( n ) .
Revised: This measures the ratio of the number of models in which policy is improved to the total number of models in ensemble ( n ) .

Original: If PR > PR threshold , we continue training using the same model, if not then we break and re-train our environment model on data collected from the new update policy.
Revised: If PR > PR threshold , we continue training using the same model, if not then we break and re-train our environment model on data collected from the new update policy.

Original: Another challenge that we encounter is the underestimation of J C ( π θ ) resulting from using a truncated horizon (of length H in step 7 of Algorithm 1) to reduce the aggregation of error.
Revised: Another challenge that we encounter is the underestimation of J C ( π θ ) resulting from using a truncated horizon (of length H in step 7 of Algorithm 1) to reduce the aggregation of error.

Original: This can potentially lead to constraint violating policies.
Revised: This can potentially lead to constraint violating policies.

Original: So we need to make the safety threshold ( d ) stricter.
Revised: So we need to make the safety threshold ( d ) stricter.

Original: It might seem that changing this threshold ( d ) proportional to the truncated horizon, i.e., setting d :=
Revised: It might seem that changing this threshold ( d ) proportional to the truncated horizon, i.e., setting d :=

Original: d ∗ HT might work better where H is the truncated horizon and T is the original environment horizon, but we found that this leads to constraint violations as well because we are learning policy using data from the approximated environment model that would bring in errors.
Revised: d ∗ HT might work better where H is the truncated horizon and T is the original environment horizon, but we found that this leads to constraint violations as well because we are learning policy using data from the approximated environment model that would bring in errors.

Original: Hence the cost estimate of the policy (Step 8 in Algorithm 1) is error-prone as well.
Revised: Hence the cost estimate of the policy (Step 8 in Algorithm 1) is error-prone as well.

Original: To tackle this issue, we make safety threshold stricter using a hyperparameter 0 ≤ β < 1 by modifying the
Revised: To tackle this issue, we make safety threshold stricter using a hyperparameter 0 ≤ β < 1 by modifying the

Original: Lagrange multiplier update as follows:
Revised: Lagrange multiplier update as follows:

Original: [Equation]
Revised: [Equation]

Original: The variation of expected cost returns and reward returns with respect to β is shown in Figure 1 on
Revised: The variation of expected cost returns and reward returns with respect to β is shown in Figure 1 on

Original: Safety gym PointGoal [Ray et al., 2019] environment.
Revised: Safety gym PointGoal [Ray et al., 2019] environment.

Original: We can observe that as we reduce β , the expected cost returns reduce because cost limit becomes stricter but choosing too small a β leads to low reward returns as well.
Revised: We can observe that as we reduce β , the expected cost returns reduce because cost limit becomes stricter but choosing too small a β leads to low reward returns as well.

Original:  Experimental Details and Results
Revised:  Experimental Details and Results

Original: We test our approach on Safety Gym environments 1 - PointGoal and CarGoal with modified state representations as used in other model-based Safe RL approaches [Liu et al., 2020, Sikchi et al., 2021a] that are more helpful in model-learning.
Revised: We test our approach on Safety Gym environments 1 - PointGoal and CarGoal with modified state representations as used in other model-based Safe RL approaches [Liu et al., 2020, Sikchi et al., 2021a] that are more helpful in model-learning.

Original: We increase the difficulty of PointGoal and CarGoal environments by increasing the number of hazards from 10 to 15.
Revised: We increase the difficulty of PointGoal and CarGoal environments by increasing the number of hazards from 10 to 15.

Original: In both environments, the aim of robots is to reach the goal position and have as few collisions with hazards as possible.
Revised: In both environments, the aim of robots is to reach the goal position and have as few collisions with hazards as possible.

Original: We compare our approach (MBPPO-Lagrangian) with Unconstrained PPO [Schulman et al., 2017], model-free Safe RL approaches including Constrained Policy Optimization [Achiam et al., 2017],
Revised: We compare our approach (MBPPO-Lagrangian) with Unconstrained PPO [Schulman et al., 2017], model-free Safe RL approaches including Constrained Policy Optimization [Achiam et al., 2017],

Original: PPO-Lagrangian [Ray et al., 2019] and the model-based approach – safe-LOOP from Sikchi et al.
Revised: PPO-Lagrangian [Ray et al., 2019] and the model-based approach – safe-LOOP from Sikchi et al.

Original: [2021a].
Revised: [2021a].

Original: The code for our approach is available here 2 .
Revised: The code for our approach is available here 2 .

Original: We run each algorithm with 8 random seeds for 450K environment interactions.
Revised: We run each algorithm with 8 random seeds for 450K environment interactions.

Original: The hyper-parameter settings and other experimental details are given in
Revised: The hyper-parameter settings and other experimental details are given in

Original: Appendix A.
Revised: Appendix A.

Original: The performance of our approach and baseline algorithms is shown in Figure 2 for
Revised: The performance of our approach and baseline algorithms is shown in Figure 2 for

Original: PointGoal environment and Figure 3 for CarGoal environment.
Revised: PointGoal environment and Figure 3 for CarGoal environment.

Original: The
Revised: The

Original: From the plots we can observe that our approach (MBPPO-Lagrangian) converges to the same level of reward performance as PPO - Lagrangian in just 450K environment interactions and outperforms the model-based safe-LOOP algorithm
Revised: From the plots we can observe that our approach (MBPPO-Lagrangian) converges to the same level of reward performance as PPO - Lagrangian in just 450K environment interactions and outperforms the model-based safe-LOOP algorithm

Original: [Sikchi et al., 2021a].
Revised: [Sikchi et al., 2021a].

Original: Also our approach gives constraint adhering policies in both the tasks.
Revised: Also our approach gives constraint adhering policies in both the tasks.

Original: In addition to above, we run our algorithm for various values of β for both CarGoal and RC-Car
Revised: In addition to above, we run our algorithm for various values of β for both CarGoal and RC-Car

Original: [Ahn, 2019] in a similar manner as we did for PointGoal in Figureand present it in Appendix D. We also compare our algorithm with safe-LOOP[Sikchi et al., 2021a] on RC-Car environment and present it in Appendix D as well.
Revised: [Ahn, 2019] in a similar manner as we did for PointGoal in Figureand present it in Appendix D. We also compare our algorithm with safe-LOOP[Sikchi et al., 2021a] on RC-Car environment and present it in Appendix D as well.

Original: We found both approaches competitive in RC-Car environment although safe-LOOP
Revised: We found both approaches competitive in RC-Car environment although safe-LOOP

Original: [Sikchi et al., 2021a] exhibits higher variance.
Revised: [Sikchi et al., 2021a] exhibits higher variance.

Original: We also measure cumulative hazard violations that occur till convergence for MBPPO-L (ours),
Revised: We also measure cumulative hazard violations that occur till convergence for MBPPO-L (ours),

Original: PPO-Lagrangian [Ray et al., 2019], safe-LOOP
Revised: PPO-Lagrangian [Ray et al., 2019], safe-LOOP

Original: [Sikchi et al., 2021a] as followsCumulative
Revised: [Sikchi et al., 2021a] as followsCumulative

Original: [Equation]
Revised: [Equation]

Original: Till convergence
Revised: Till convergence

Original: [Equation]
Revised: [Equation]

Original: We use rliable library (See Sec 4.3 in Agarwal et al.
Revised: We use rliable library (See Sec 4.3 in Agarwal et al.

Original: [2021] for more details) for plotting 95% confidence intervals for cumulative violations in the right part of Figure 4 (normalized by cumulative violations in unconstrained PPO in respective tasks) and final policy reward performance in the left part of Figure 4 (normalized by reward return of the final policy in unconstrained PPO in respective tasks) using mean, median and inter-quartile mean as aggregate estimates for our approach,
Revised: [2021] for more details) for plotting 95% confidence intervals for cumulative violations in the right part of Figure 4 (normalized by cumulative violations in unconstrained PPO in respective tasks) and final policy reward performance in the left part of Figure 4 (normalized by reward return of the final policy in unconstrained PPO in respective tasks) using mean, median and inter-quartile mean as aggregate estimates for our approach,

Original: PPO-Lagrangian and safe-LOOP because these approaches only give constraint satisfying policies at convergence.
Revised: PPO-Lagrangian and safe-LOOP because these approaches only give constraint satisfying policies at convergence.

Original: From figure 4 we can observe that our approach is competitive with regards the model-based safe-LOOP approach [Sikchi et al., 2021a] in terms of cumulative constraint violations but achieves better reward performance at convergence.
Revised: From figure 4 we can observe that our approach is competitive with regards the model-based safe-LOOP approach [Sikchi et al., 2021a] in terms of cumulative constraint violations but achieves better reward performance at convergence.

Original:  Conclusions
Revised:  Conclusions

Original: We presented a sample efficient approach for Safe Reinforcement Learning and compared our approach with relevant model-free and model-based baselines [Achiam et al., 2017, Sikchi et al., 2021a, Ray et al., 2019, Schulman et al., 2017].
Revised: We presented a sample efficient approach for Safe Reinforcement Learning and compared our approach with relevant model-free and model-based baselines [Achiam et al., 2017, Sikchi et al., 2021a, Ray et al., 2019, Schulman et al., 2017].

Original: Note that we chose the above baselines because they are specifically designed to solve a constrained optimization problem of the same structure as (1)-(3).
Revised: Note that we chose the above baselines because they are specifically designed to solve a constrained optimization problem of the same structure as (1)-(3).

Original: In Sikchi et al.
Revised: In Sikchi et al.

Original: [2021a], approximation of the value function is used to provide long-term reasoning instead of using a truncated horizon.
Revised: [2021a], approximation of the value function is used to provide long-term reasoning instead of using a truncated horizon.

Original: A limitation of this approach lies in the fact that approximations of reward and cost value functions are used.
Revised: A limitation of this approach lies in the fact that approximations of reward and cost value functions are used.

Original: The most challenging part of model-based approaches is to learn the environment model.
Revised: The most challenging part of model-based approaches is to learn the environment model.

Original: The first issue is of computational resources and the time overhead that is needed.
Revised: The first issue is of computational resources and the time overhead that is needed.

Original: We provide a comparison of the running time of algorithms and their hardware requirements (see Appendix B).
Revised: We provide a comparison of the running time of algorithms and their hardware requirements (see Appendix B).

Original: Our algorithm does much better than safe-LOOP [Sikchi et al., 2021a] in terms of running time.
Revised: Our algorithm does much better than safe-LOOP [Sikchi et al., 2021a] in terms of running time.

Original: One should note that in safe exploration settings, agents do not explore as much as an unconstrained agent would do.
Revised: One should note that in safe exploration settings, agents do not explore as much as an unconstrained agent would do.

Original: This adds to the complexity of model learning in safe exploration problems because constrained exploration leads to limited representation of data points for model learning even with the same sample size.
Revised: This adds to the complexity of model learning in safe exploration problems because constrained exploration leads to limited representation of data points for model learning even with the same sample size.

Original: We found this more pronounced in high-dimensional environments like
Revised: We found this more pronounced in high-dimensional environments like

Original: DoggoGoal1 [Ray et al., 2019]
Revised: DoggoGoal1 [Ray et al., 2019]

Original: (See Appendix C).
Revised: (See Appendix C).

Original: Our algorithm depends on Lagrangian-based approach which provides a very simplistic way to construct a safe RL algorithm but suffers from the issue of low reward performance compared to unconstrained approaches (see Appendix A for a reward comparison).
Revised: Our algorithm depends on Lagrangian-based approach which provides a very simplistic way to construct a safe RL algorithm but suffers from the issue of low reward performance compared to unconstrained approaches (see Appendix A for a reward comparison).

Original: Note that we can have a similar model-based approach using TRPO-Lagrangian [Ray et al., 2019] but it involves an extra overhead of approximating the Fisher Information Matrix (FIM) and hence we chose a PPO-based approach.
Revised: Note that we can have a similar model-based approach using TRPO-Lagrangian [Ray et al., 2019] but it involves an extra overhead of approximating the Fisher Information Matrix (FIM) and hence we chose a PPO-based approach.

Original: Also TRPO-Lagrangian and PPO-Lagrangian have similar performance on Safety Gym benchmark [Ray et al., 2019].
Revised: Also TRPO-Lagrangian and PPO-Lagrangian have similar performance on Safety Gym benchmark [Ray et al., 2019].

Original: Increasing reward performance of Lagrangian-based approaches and devising better ways for model-learning in high-dimensional state representations in safe RL settings where exploration is limited, can be looked at in the future.
Revised: Increasing reward performance of Lagrangian-based approaches and devising better ways for model-learning in high-dimensional state representations in safe RL settings where exploration is limited, can be looked at in the future.

Original: Moreover, it would be interesting to adapt off-policy natural actor-critic algorithms such as in
Revised: Moreover, it would be interesting to adapt off-policy natural actor-critic algorithms such as in

Original: Bhatnagar et al.
Revised: Bhatnagar et al.

Original: [2009], Diddigi et al.
Revised: [2009], Diddigi et al.

Original: [2022] to the setting of constrained MDPs and study their performance both theoretically and empirically.
Revised: [2022] to the setting of constrained MDPs and study their performance both theoretically and empirically.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Ashish K. Jayant was supported in his work through an MHRD scholarship from the Ministry of
Revised: Ashish K. Jayant was supported in his work through an MHRD scholarship from the Ministry of

Original: Education, Government of India; Indian Institute of Science, Bangalore; and Flipkart Internet Pvt.
Revised: Education, Government of India; Indian Institute of Science, Bangalore; and Flipkart Internet Pvt.

Original: Ltd., Bangalore.
Revised: Ltd., Bangalore.

Original: S. Bhatnagar was supported in his work through the J.C.Bose National Fellowship,
Revised: S. Bhatnagar was supported in his work through the J.C.Bose National Fellowship,

Original: Science and Engineering Research Board, Government of India; a project from the Department of
Revised: Science and Engineering Research Board, Government of India; a project from the Department of

Original: Science and Technology under the ICPS program; a project from DRDO under JATP-CoE; as well as the Robert Bosch Centre for Cyber Physical Systems, Indian Institute of Science, Bangalore.
Revised: Science and Technology under the ICPS program; a project from DRDO under JATP-CoE; as well as the Robert Bosch Centre for Cyber Physical Systems, Indian Institute of Science, Bangalore.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] We describe limitations in Conclusion section (c) Did you discuss any potential negative societal impacts of your work?
Revised: [Yes] We describe limitations in Conclusion section (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] We have read the ethics review guidelines and ensured that our work conforms to them.
Revised: [Yes] We have read the ethics review guidelines and ensured that our work conforms to them.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We provide the github url for our code repository (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] We provide the github url for our code repository (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Hyperparameter details are given in appendix A (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] Hyperparameter details are given in appendix A (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report plots with 95% confidence intervals and use rliable Agarwal et al.
Revised: [Yes] We report plots with 95% confidence intervals and use rliable Agarwal et al.

Original: [2021] library for robust aggregate estimates (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [2021] library for robust aggregate estimates (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We share details of hardware requirement in appendix B
Revised: [Yes] We share details of hardware requirement in appendix B

Original: If you are using existing assets (e [Table] g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite the authors of Safety Gym benchmark and their github code repository.
Revised: [Yes] We cite the authors of Safety Gym benchmark and their github code repository.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Their MIT License which allows without restriction, including without limitation the rights to use, copy, modify, merge, publish provided that license notice is included in our code repository (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] Their MIT License which allows without restriction, including without limitation the rights to use, copy, modify, merge, publish provided that license notice is included in our code repository (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?[N/A]
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: A Hyper-parameters and finer experimental details

Original: The hyper-parameters used for our algorithm are shown in Table 1.

Original: Each episode runs for 750 steps

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: Hidden layers in

Original: [Table]

Original: [Table]

Original: [Table]

Original: Hidden layers in

Original: [Table]

Original: NN

Original: (Hidden layers in Critic NN : no. of nodes

Original: [Table]

Original: [Table]

Original: 3e-

Original: [Table]

Original: , ψ

Original: [Table]

Original: as opposed to 1000 steps in the original version of Safety Gym

Original: [Table]
Revised: [N/A]

Original: hence we chose cost threshold of 18 which is ∼ 75% of 25 used in the official Safety Gym benchmark paper.

Original: In Goal-based Safety gym environments the aim is to reach the goal position (in green in Figure 1) with as few collisions as possible with hazards.

Original: If robot (in red in Figure 1) accesses ’hazard’ positions (in blue in Figure 1)

Original: [Table]

Original: the agent incurs a cost = 1.

Original: The ’Point’ robot has steering and throttle as action space while ’Car’ robot has differential control.

Original: We use Performance Ratio (PR) threshold of 66%.

Original: The logic behind this number is that our agent should perform better in more than 50% of the models in the ensemble.

Original: In our case we train an ensemble of 8 models out of which we use the best 6 models with minimum validation losses to calculate PR.

Original: We want our agent to perform better in at least 4 out of the 6 models that gives us the value as 66% .

Original: As mentioned in the text we increase the number of hazards in PointGoal1 and CarGoal1 environments from 10 to 15 so as to increase their difficulty level.

Original: We run on 8 random seeds which is greater than 4 used in benchmarking previous unconstrained model-based approaches 3 .

Original: We use the official implementation of baselines from safe-LOOP 4 , CPOand PPO-Lagrangian 6 , respectively.

Original: In all baselines we use the same size of the neural networks as mentioned in Table 1 and the same number of models in ensemble for model learning for safe-LOOP.

Original: In safe-LOOP, we do not change the originally used planning horizon of H = 8 and cost threshold of 0 in the model optimization part.

Original: During first pass of Algorithm 1 (Line 5-17), we use 5% of real environment interaction data and 95% of imaginary rollout data, this is done to make use of real environment data that we collected to train model dynamics.

Original: After first pass, policy gets updated and after that we use purely imaginary data since this is an on-policy algorithm until PR goes below 66% .

Original: The values used for normalization in plot for Figure 4 are the following: Reward performance of

Original: PPO-unconstrained = 20 and Cumulative constraint violations of PPO-unconstrained = 2e5.

Original: B Hardware requirements and Running time

Original: Minimum 4 GB GPU space is required for running both the model based approaches.

Original: We did not

Original: [Table]

Original: use multiprocessing in model-based approaches for data-collection from simulation.

Original: However, for model-free approaches we collected simulation data using 8 CPUs at a time.

Original: Running time for 450k steps for all baselines is given in Table 2.

Original: Note that for convergence PPO-Lagrangian required 849 ± 12.48 s and CPO required 1162 ± 8.87 s.

Original: The running time of our model-based approach can be improved by using parallel workers for data collection but still major part of training is spent in model learning.

Original: C Comparison of model learning validation loss (Unconstrained vs Safe RL) and performance in High-dimensional task

Original: We compare how model learning validation loss varies in Safe RL setting as opposed to unconstrained

Original: RL one.

Original: We plot validation loss vs no.

Original: of datapoints collected as our model-based algorithm progress (See Figure 6.

Original: We observe that validation error converges quickly in unconstrained setting, which is reflective of better exploration in unconstrained setting.

Original: This plot is generated using 8 different initialization of neural network used for model learning.

Original: We do this for our modified PointGoal environment.

Original: DoggoGoal1 is a challenging environment where a quadruped robot has to reach a goal position without colliding with obstacles.

Original: This task has higher dimensional action space (12-dimensional) than

Original: Point and Car robots which just needs steering and throttle in Safety Gym.

Original: On this task, even modelfree Lagrangian baselines suffer and get quite low reward performance.

Original: Further, both safe-LOOP

Original: and

Original: MBPPO-Lagrangian fails to match model-free baseline in terms of reward performance in DoggoGoal environment.

Original: It signifies the additional challenge in model learning in case of high-dimensional environments.

Original: [Table]

Original: [Table]

Original: D Effect of β in CarGoal and RC-Car environment and Model-based baselines results on RC-Car

Original: In RC Car 7 environment, a car has to rotate within a circle with target velocity for earning rewards.

Original: If car goes out of circle, it incurs cost.

Original: We observe a similar trend as observed in Figure 1 of the paper, i.e., lower the value of β , lower are the reward returns as the agent explores pessimistically.

Original: We also ran MBPPO-L(our algorithm) and safe-LOOP [Sikchi et al., 2021a] on the RC-Car environment for 150k steps (with 5 random seeds) only as safe-LOOP has a very high running time.

Original: We choose cost limit = 5.

Original: Results are presented in

Original: Figure 10.

Original: We can observe that both approaches have similar performance but safe-LOOP exhibits higher variance than our approach.