Original: Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning
Revised: Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learning

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Reinforcement Learning (RL) agents in the real world must satisfy safety constraints in addition to maximizing a reward objective.
Revised: Reinforcement Learning (RL) agents in the real world must satisfy safety constraints in addition to maximizing a reward objective.

Original: Model-based RL algorithmshold promise for reducing unsafe real-world actions : they may synthesize policiesthat obey all constraints using simulated samples from a learned model.
Revised: Model-based RL algorithms hold promise for reducing unsafe real-world actions: they may synthesize policies that obey all constraints using simulated samples from a learned model.

Original: However,imperfect models can result in real-world constraint violations even for actionsthat are predicted to satisfy all constraints.
Revised: However, imperfect models can result in real-world constraint violations even for actions that are predicted to satisfy all constraints.

Original: We propose CAP, a model-based safe
Revised: We propose CAP, a model-based safe

Original: RL framework that accounts for potential modeling errors by capturing modeluncertainty and adaptively exploiting it to balance the reward and the cost objectives.
Revised: RL framework that accounts for potential modeling errors by capturing model uncertainty and adaptively exploiting it to balance the reward and the cost objectives.

Original: First, CAP inﬂates predicted costs using an uncertainty-based penalty.
Revised: First, CAP inﬂates predicted costs using an uncertainty-based penalty.

Original: Theoretically, we show that policies that satisfy this conservative cost constraintare guaranteed to also be feasible in the true environment.
Revised: Theoretically, we show that policies that satisfy this conservative cost constraint are guaranteed to also be feasible in the true environment.

Original: We further show thatthis guarantees the safety of all intermediate solutions during RL training.
Revised: We further show that this guarantees the safety of all intermediate solutions during RL training.

Original: Further,
Revised: Further,

Original: CAP adaptively tunes this penalty during training using true cost feedback from theenvironment.
Revised: CAP adaptively tunes this penalty during training using true cost feedback from the environment.

Original: We evaluate this conservative and adaptive penalty-based approachfor model-based safe RL extensively on state and image-based environments.
Revised: We evaluate this conservative and adaptive penalty-based approach for model-based safe RL extensively on state and image-based environments.

Original: Ourresults demonstrate substantial gains in sample-efﬁciency while incurring fewerviolations than prior safe RL algorithms.
Revised: Our results demonstrate substantial gains in sample-efﬁciency while incurring fewer violations than prior safe RL algorithms.

Original:  Introduction
Revised:  Introduction

Original: Many applications of reinforcement learning (RL) require the agent to satisfy safety constraints inaddition to the standard goal of maximizing the expected reward.
Revised: Many applications of reinforcement learning (RL) require the agent to satisfy safety constraints in addition to the standard goal of maximizing the expected reward.

Original: For example, in robot locomotion,we may want to impose speed or torque constraints to prevent the robot from damaging itself.
Revised: For example, in robot locomotion, we may want to impose speed or torque constraints to prevent the robot from damaging itself.

Original: Sincethe set of states that violates the imposed constraints is often a priori unknown, a central goal of safereinforcement learning [1, 2] is to learn a reward-maximizing policy that satisﬁes constraints, whileincurring as few constraint violations as possible during the agent’s training process.
Revised: Since the set of states that violates the imposed constraints is often a priori unknown, a central goal of safe reinforcement learning [1, 2] is to learn a reward-maximizing policy that satisﬁes constraints, while incurring as few constraint violations as possible during the agent’s training process.

Original: To reduce the cumulative number of constraint violations during training, a promising approachis to incorporate safety considerations into sample-efﬁcient RL algorithms, such as model-basedreinforcement learning (MBRL) [3, 4].
Revised: To reduce the cumulative number of constraint violations during training, a promising approach is to incorporate safety considerations into sample-efﬁcient RL algorithms, such as model-based reinforcement learning (MBRL) [3, 4].

Original: MBRL refers to RL algorithms that use learned transitionmodels to directly synthesize policies using simulated samples, thereby reducing the number of realsamples needed to train the policy.
Revised: MBRL refers to RL algorithms that use learned transition models to directly synthesize policies using simulated samples, thereby reducing the number of real samples needed to train the policy.

Original: Given the true environment transition model, it would be trivialto synthesize safe policies without any violations, since we could simply simulate a sequence ofactions to evaluate its safety.
Revised: Given the true environment transition model, it would be trivial to synthesize safe policies without any violations, since we could simply simulate a sequence of actions to evaluate its safety.

Original: However, MBRL agents must learn this transition model from ﬁniteexperience, which induces approximation errors.
Revised: However, MBRL agents must learn this transition model from ﬁnite experience, which induces approximation errors.

Original: In this paper, we ask: can safety be guaranteedduring model-based reinforcement learning, despite these model errors?
Revised: In this paper, we ask: can safety be guaranteed during model-based reinforcement learning, despite these model errors?

Original: We prove that this is indeedpossible, and design a practical algorithm that permits model-based safe RL even in high-dimensionalproblem settings.to
Revised: We prove that this is indeed possible, and design a practical algorithm that permits model-based safe RL even in high-dimensional problem settings.

Original: Speciﬁcally, we propose a model-based safe RL framework involving a c onservative and a daptivecost p enalty ( CAP ).
Revised: Speciﬁcally, we propose a model-based safe RL framework involving a c onservative and a daptive cost p enalty ( CAP ).

Original: We build on a basic model-based safe RL framework, which simply executes amodel-free safe RL algorithm inside a learned transition model.
Revised: We build on a basic model-based safe RL framework, which simply executes a model-free safe RL algorithm inside a learned transition model.

Original: We make two important conceptualcontributions to improve this basic approach.
Revised: We make two important conceptual contributions to improve this basic approach.

Original: First, we derive a conservative upper bound on theerror in the policy cost computed according to the learned model.
Revised: First, we derive a conservative upper bound on the error in the policy cost computed according to the learned model.

Original: In particular, we show that thiserror is bounded above by a constant factor of an integral probability metric (IPM) [5] computedover the true and learned transition models.
Revised: In particular, we show that this error is bounded above by a constant factor of an integral probability metric (IPM) [5] computed over the true and learned transition models.

Original: Based on this bound, we propose to inﬂate the costfunction with an uncertainty-aware penalty function.
Revised: Based on this bound, we propose to inﬂate the cost function with an uncertainty-aware penalty function.

Original: We prove that all feasible policies with respectto this conservative cost function, including the optimal feasible policy (with highest task reward),are guaranteed to be safe in the true environment.
Revised: We prove that all feasible policies with respect to this conservative cost function, including the optimal feasible policy (with highest task reward), are guaranteed to be safe in the true environment.

Original: A direct consequence is that we can ensure that allintermediate policies are safe and incur zero safety violations during training.
Revised: A direct consequence is that we can ensure that all intermediate policies are safe and incur zero safety violations during training.

Original: Second, this penalty function, though theoretically optimal, is often too conservative or cannot becomputed for high-dimensional tasks.
Revised: Second, this penalty function, though theoretically optimal, is often too conservative or cannot be computed for high-dimensional tasks.

Original: Therefore, in practice, we propose a heuristic penalty term thatincludes a scale hyperparameter to modulate the degree of conservativeness: higher scales producebehavior that is more averse to risks arising from modeling errors.
Revised: Therefore, in practice, we propose a heuristic penalty term that includes a scale hyperparameter to modulate the degree of conservativeness: higher scales produce behavior that is more averse to risks arising from modeling errors.

Original: Thus, different scales may beappropriate for use with different environments, tasks, and model ﬁdelities.
Revised: Thus, different scales may be appropriate for use with different environments, tasks, and model ﬁdelities.

Original: We observe that thiscrucial scale hyperparameter need not be manually set and frozen throughout training.
Revised: We observe that this crucial scale hyperparameter need not be manually set and frozen throughout training.

Original: Instead, wecan exploit the fact that the policy receives feedback on its true cost value from the environment, toformulate the entire inﬂated cost function as a control plant.
Revised: Instead, we can exploit the fact that the policy receives feedback on its true cost value from the environment, to formulate the entire inﬂated cost function as a control plant.

Original: In this view, the scale hyparparameter isthe control input.
Revised: In this view, the scale hyparparameter is the control input.

Original: Then, we can readily apply existing update rules from the control literature to tunethe scale.
Revised: Then, we can readily apply existing update rules from the control literature to tune the scale.

Original: In particular, we use the integral controller [6], a simple variant of the PID controller, toadaptively update the scale using cost feedback from the environment.
Revised: In particular, we use the proportional-integral (PI) controller [6], a simple variant of the PID controller, to adaptively update the scale using cost feedback from the environment.

Original: Our overall CAP framework incorporates a conservative penalty term into predicted costs in the basicmodel-based safe RL framework, and adapts its scale to ensure the penalty is neither too aggressivenor too modest.
Revised: Our overall CAP framework incorporates a conservative penalty term into predicted costs in the basic model-based safe RL framework, and adapts its scale to ensure the penalty is neither too aggressive nor too modest.

Original: To evaluate CAP, we ﬁrst illustrate its proposed beneﬁts in simple tabular gridworldenvironments using a linear programming-based instantiation of CAP; there, we show that CAPindeed achieves zero training violations and exhibits effective adaptive behavior .
Revised: To evaluate CAP, we ﬁrst illustrate its proposed beneﬁts in simple tabular gridworld environments using a linear programming-based instantiation of CAP; there, we show that CAP indeed achieves zero training violations and exhibits effective adaptive behavior .

Original: For state andimage-based control environments, we evaluate a second instantiation of CAP, using a cost constraintaware variant [7] of cross entropy method (CEM) [8] coupled with state-of-art dynamics models[9, 10] to optimize action sequences.
Revised: For state and image-based control environments, we evaluate a second instantiation of CAP, using a cost constraintaware variant [7] of cross entropy method (CEM) [8] coupled with state-of-art dynamics models [9, 10] to optimize action sequences.

Original: Through extensive experiments, we show that our practicalalgorithms substantially reduce the number of real environment samples and unsafe episodes requiredto learn feasible, high-reward policies compared to model-free baselines as well as ablations of CAP.
Revised: Through extensive experiments, we show that our practical algorithms substantially reduce the number of real environment samples and unsafe episodes required to learn feasible, high-reward policies compared to model-free baselines as well as ablations of CAP.

Original: In summary, our main contributions are:
Revised: In summary, our main contributions are:

Original: • an uncertainty-aware cost penalty function that can guarantee the safety of all training policyiterates• an automatic update rule for dynamically tuning the degree of conservativeness during training.• a linear program formulation of CAP that achieves near-optimal policies in tabular gridworldswhile incurring zero training violation• and ﬁnally, scalable implementations of CAP that learn safe, high-reward actions in continuouscontrol environments with high-dimensional states, including images.
Revised: • an uncertainty-aware cost penalty function that can guarantee the safety of all training policy iterates • an automatic update rule for dynamically tuning the degree of conservativeness during training. • a linear program formulation of CAP that achieves near-optimal policies in tabular gridworlds while incurring zero training violation • and ﬁnally, scalable implementations of CAP that learn safe, high-reward actions in continuous control environments with high-dimensional states, including images.

Original:  Related Work
Revised:  Related Work

Original: Safe RL Our work is broadly related to the safe reinforcement learning and control literature;we refer interested readers to [2, 11] for surveys on this topic.
Revised: Safe RL Our work is broadly related to the safe reinforcement learning and control literature; we refer interested readers to [2, 11] for surveys on this topic.

Original: A popular class of approachesincorporates Lagrangian constraint regularization into the policy updates in policy-gradient algorithms[12, 13, 14, 15, 16, 17, 18].
Revised: A popular class of approaches incorporates Lagrangian constraint regularization into the policy updates in policy-gradient algorithms [12, 13, 14, 15, 16, 17, 18].

Original: These methods build on model-free deep RL algorithms [19, 20], whichare often sample-inefﬁcient, and do not guarantee that intermediate policies during training are safe.
Revised: These methods build on model-free deep RL algorithms [19, 20], which are often sample-inefﬁcient, and do not guarantee that intermediate policies during training are safe.

Original: These safe RL algorithms are therefore liable to perform large numbers of unsafe maneuvers duringtraining.
Revised: These safe RL algorithms are therefore liable to perform large numbers of unsafe maneuvers during training.

Original: Model-Based Safe RL Model-based safe RL approaches, instead, learn to synthesize a policythrough the use of a transition model learned through data.
Revised: Model-Based Safe RL Model-based safe RL approaches, instead, learn to synthesize a policy through the use of a transition model learned through data.

Original: A distinguishing factor among modelbased approaches is their assumption on what is known or safe in the environment.
Revised: A distinguishing factor among modelbased approaches is their assumption on what is known or safe in the environment.

Original: Most worksassume partially known dynamics [21, 22] or safe regions [23, 24, 25, 26], and come with safetyguarantees that are tied to these assumptions.
Revised: Most works assume partially known dynamics [21, 22] or safe regions [23, 24, 25, 26], and come with safety guarantees that are tied to these assumptions.

Original: In comparison, our work targets the more general setting, involving no such prior knowledge.
Revised: In comparison, our work targets the more general setting, involving no such prior knowledge.

Original: In tabular MDP settings, we prove a high probabilityguarantee on the safety of any feasible solution under the conservative objective; we subsequentlyextend this result to ensure the safety of all training episodes.
Revised: In tabular MDP settings, we prove a high probability guarantee on the safety of any feasible solution under the conservative objective; we subsequently extend this result to ensure the safety of all training episodes.

Original: On more complex domains, we provideapproximate and practically effective implementations for high-dimensional inputs, such as images.
Revised: On more complex domains, we provide approximate and practically effective implementations for high-dimensional inputs, such as images.

Original: Our core idea of using uncertainty estimates as penalty terms to avoid unsafe regions has beenexplored in several prior works [27, 21, 28].
Revised: Our core idea of using uncertainty estimates as penalty terms to avoid unsafe regions has been explored in several prior works [27, 21, 28].

Original: However, our work provides the ﬁrst theoreticaltreatment of the uncertainty-based cost penalty that is independent of the type of the cost (e.g., binarycost) and the parametric choice of the transition model.
Revised: However, our work provides the ﬁrst theoretical treatment of the uncertainty-based cost penalty that is independent of the type of the cost (e.g., binary cost) and the parametric choice of the transition model.

Original: Our theoretical analysis is similar to thatof [29], though we extend their results, originally in the ofﬂine constraint-free setting, to the onlineconstrained MDP setting, and introduce a new result guaranteeing safety for all training episodes.
Revised: Our theoretical analysis is similar to that of [29], though we extend their results, originally in the ofﬂine constraint-free setting, to the online constrained MDP setting, and introduce a new result guaranteeing safety for all training episodes.

Original: Furthermore, our framework permits the cost penalty weight to automatically adjust to transitionmodel updates, using environment cost feedback during MBRL training.
Revised: Furthermore, our framework permits the cost penalty weight to automatically adjust to transition model updates, using environment cost feedback during MBRL training.

Original:  Preliminaries
Revised:  Preliminaries

Original: In safe reinforcement learning, one common problem formulation is to consider an inﬁnite-horizonconstrained Markov Decision Process (CMDP) [30] M = ( S , A , T, r, c,  , µ 0 ) .
Revised: In safe reinforcement learning, one common problem formulation is to consider an inﬁnite-horizon constrained Markov Decision Process (CMDP) [30] M = ( S , A , T, r, c, γ, µ 0 ) .

Original: Here, S , A are thestate and action spaces, T ( s [-
-] | s, a ) is the transition distribution, r ( s, a ) is the reward function, c ( s, a )is the cost function,  2 (0 , 1) is the discount factor, and s [-
 ⇠-] µ 0 ( s 0 ) is the initial state distribution;we assume that both r ( s, a ) and c ( s, a ) are bounded.
Revised: Here, S , A are the state and action spaces, T ( s [-
-](cid:48) | s, a ) is the transition distribution, r ( s, a ) is the reward function, c ( s, a ) is the cost function, γ ∈ (0 , 1) is the discount factor, and s [-
 ⇠-]0 ∼ µ 0 ( s 0 ) is the initial state distribution; we assume that both r ( s, a ) and c ( s, a ) are bounded.

Original: A policy ⇡ : S !  ( A ) is a mapping fromstate to distribution over actions.
Revised: A policy π : S → ∆( A ) is a mapping from state to distribution over actions.

Original: Given a ﬁxed policy ⇡ , its state-action occupancy distributionis deﬁned to be ⇢ ⇡ T ( s, a ) :
Revised: Given a ﬁxed policy π , its state-action occupancy distribution is deﬁned to be ρ πT ( s, a ) :

Original: = (1   ) 
Revised: = (1 − γ ) (cid:80) ∞

Original: P 1 t =

Revised: t =

Original:  t Pr ⇡ ( s t = s, a t = a ) , where Pr ⇡ ( s t = s, a t = a ) isthe probability of visiting ( s, a ) at timestep t when executing ⇡ in M starting at s 0 ⇠ µ 0 .
Revised: γ t Pr π ( s t = s, a t = a ) , where Pr π ( s t = s, a t = a ) is the probability of visiting ( s, a ) at timestep t when executing π in M starting at s 0 ∼ µ 0 .

Original: Theobjective in this safe RL formulation is to ﬁnd the optimal feasible policy ⇡ ⇤ that solves the followingconstrained optimization problem:
Revised: The objective in this safe RL formulation is to ﬁnd the optimal feasible policy π ∗ that solves the following constrained optimization problem

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over s 0 ⇠ µ 0 ( · ) , s t ⇠ T ( s t | s t  1 , a t  1 ) , a t ⇠ ⇡ ( · | s t ) , and C is a112 cumulative constraint threshold that should not be exceeded.
Revised: where the expectation is over s 0 ∼ µ 0 ( · ) , s t ∼ T ( s t | s t − 1 , a t − 1 ) , a t ∼ π ( · | s t ) , and C is a cumulative constraint threshold that should not be exceeded.

Original: We say that a policy ⇡ is feasible if itdoes not violate the constraint, and the optimization problem is feasible if there exists at least onefeasible solution (i.e., policy).
Revised: We say that a policy π is feasible if it does not violate the constraint, and the optimization problem is feasible if there exists at least one feasible solution (i.e., policy).

Original: Unlike unconstrained MDPs, constrained MDPs cannot be solved by dynamic programming [31].
Revised: Unlike unconstrained MDPs, constrained MDPs cannot be solved by dynamic programming [31].

Original: Instead, a common approach is to consider the dual of Eq (1) [30]:
Revised: Instead, a common approach is to consider the dual of Eq (1) [30]:

Original: [Equation]
Revised: [Equation]

Original: where ⇢ ( s ) =
Revised: where ρ ( s ) =

Original: P a ⇢ ( s, a ) .
Revised: (cid:80) a ρ ( s, a ) .

Original: The dual problem Eq (2) is a linear program over occupancy distributions,and can be solved using standard LP algorithms; the second constraint deﬁnes the space of validoccupancy distributions by ensuring a “conservation of ﬂow" property among the distributions.
Revised: The dual problem Eq (2) is a linear program over occupancy distributions, and can be solved using standard LP algorithms; the second constraint deﬁnes the space of valid occupancy distributions by ensuring a “conservation of ﬂow" property among the distributions.

Original: Givenits solution ⇢ ⇤ , the optimal policy can be deﬁned as ⇡ ⇤ ( a | s ) =
Revised: Given its solution ρ ∗ , the optimal policy can be deﬁned as π ∗ ( a | s ) =

Original: arg max ⇢ ⇤ ( s, a ) , or equivalently,⇡ ⇤
Revised: arg max ρ ∗ ( s, a ) , or equivalently, π ∗ ( a | s ) =

Original: [Equation] 
Revised: [Equation] [Equation]

Original: P
Revised: / (cid:80) a

Original: [Equation]
Revised: [Equation]

Original:  (if the optimal policy is unique).
Revised: s, a ) (if the optimal policy is unique).

Original: Typically, the transition function T is not known to the agent; thus, the optimal policy ⇡ ⇤ cannot bedirectly computed through LP.
Revised: Typically, the transition function T is not known to the agent; thus, the optimal policy π ∗ cannot be directly computed through LP.

Original: In model-based reinforcement learning (MBRL), the lack of known Tis directly addressed by learning an estimated transition function ˆ T through data D := { ( s, a, r, c, s 0 ) } .
Revised: In model-based reinforcement learning (MBRL), the lack of known T is directly addressed by learning an estimated transition function ˆ T through data D := { ( s, a, r, c, s (cid:48) ) } .

Original: Then, we can deﬁne a surrogate objective to Eq (2) by simply replacing T with ˆ T and solving Eq (2)as before.
Revised: Then, we can deﬁne a surrogate objective to Eq (2) by simply replacing T with ˆ T and solving Eq (2) as before.

Original: Likewise, we can replace J ( ⇡ ) with ˆ J ( ⇡ ) , and J c ( ⇡ ) with ˆ J c ( ⇡ ) , to obtained modelbased objectives in Eq (1).
Revised: Likewise, we can replace J ( π ) with ˆ J ( π ) , and J c ( π ) with ˆ J c ( π ) , to obtained modelbased objectives in Eq (1).

Original: Putting all this together, we may deﬁne a basic model-based safe RLframework [21, 11] that iterates among three steps: (1) solving for ˆ ⇡ ⇤ approximately, (2) collectingdata ( s, a, r, c, s 0 ) from ˆ ⇡ ⇤ , and (3) updating ˆ T using all collected data so far.
Revised: Putting all this together, we may deﬁne a basic model-based safe RL framework [21, 11] that iterates among three steps: (1) solving for ˆ π ∗ approximately, (2) collecting data ( s, a, r, c, s (cid:48) ) from ˆ π ∗ , and (3) updating ˆ T using all collected data so far.

Original: However, at anyﬁxed training iteration, the modeling error may lead to sub-optimal, potentially infeasible ˆ ⇡ ⇤ .
Revised: However, at any ﬁxed training iteration, the modeling error may lead to sub-optimal, potentially infeasible ˆ π ∗ .

Original: Thismotivates our approach, described in the following sections.
Revised: This motivates our approach, described in the following sections.

Original:  CAP: Conservative and Adaptive Penalty
Revised:  CAP: Conservative and Adaptive Penalty

Original: Next, we introduce c onservative and a daptive cost- p enalty (CAP), our proposed uncertainty andfeedback-aware model-based safe RL framework.
Revised: Next, we introduce c onservative and a daptive cost- p enalty (CAP), our proposed uncertainty and feedback-aware model-based safe RL framework.

Original: First, we precisely characterize the downstreameffect of the model prediction error on the cost estimate ˆ J c ( ⇡ ) by providing an upper bound on thetrue cost J ⇤ c ( ⇡ ) , which allows us to derive a penalty function based on the epistemic uncertainty ofthe model.
Revised: First, we precisely characterize the downstream effect of the model prediction error on the cost estimate ˆ J c ( π ) by providing an upper bound on the true cost J ∗ c ( π ) , which allows us to derive a penalty function based on the epistemic uncertainty of the model.

Original: To this end, we adapt the return simulation lemma results in [32, 29] to the cost settingand derive the following upper bound on the true policy cost 11   P [Equation] ⇡ T 
Revised: To this end, we adapt the return simulation lemma results in [32, 29] to the cost setting and derive the following upper bound on the true policy cost 11 − γ (cid:80) s,a ρ πT ( s, a ) c ( s, a ) with respect to the estimated policy cost 11 − γ (cid:80) s,a ρ π ˆ T ( s, a ) c ( s, a ) .

Original: [Equation]

Original: s, a ) c ( s, a ) with respectto the estimated policy cost 11

Original: [Equation]

Original: 4.1 Cost Penalty
Revised: 4.1 Cost Penalty

Original: First, given a policy mapping ⇡ , we deﬁne V ⇡ c : S!
Revised: First, given a policy mapping π , we deﬁne V πc : S

Original:  R such that V ⇡ c ( s ) :
Revised: → R such that V πc ( s ) :

Original: = E ⇡ ,T [ P{+(cid:80) ∞ t =
+}
Revised: = E π,T [ {+(cid:80) ∞ t =
+}

Original: [Equation] |142 s [-
-] = s ] .
Revised: γ t c ( s t , a t ) | s [-
-]0 = s ] .

Original: We make the following assumption on the realizability of V ⇡ c .
Revised: We make the following assumption on the realizability of V πc .

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: There exists a  > 0 and a function class F such that V ⇡ c 2  F for all ⇡ .
Revised: There exists a β > 0 and a function class F such that V πc ∈ β F for all π .

Original: With this assumption, we show that the difference between the estimated and true costs can bebounded by the integral probability metric (IPM) deﬁned by F computed between the true and thelearned transition models.
Revised: With this assumption, we show that the difference between the estimated and true costs can be bounded by the integral probability metric (IPM) deﬁned by F computed between the true and the learned transition models.

Original: Lemma 4.2 (Cost Simulation Lemma and Upper Bound) .
Revised: Lemma 4.2 (Cost Simulation Lemma and Upper Bound) .

Original: Let the F -induced IPM be deﬁned as
Revised: Let the F -induced IPM be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Then, the difference between the expected policy cost computed using T and ˆ T is bounded above:
Revised: Then, the difference between the expected policy cost computed using T and ˆ T is bounded above:

Original: [Equation]
Revised: [Equation]

Original: We provide a proof in Appendix A.
Revised: We provide a proof in Appendix A.

Original: This upper bound illustrates the risk of applying MBRL withoutmodiﬁcation in safety-critical settings.Attaining 11  
Revised: This upper bound illustrates the risk of applying MBRL without modiﬁcation in safety-critical settings.

Original: P [Equation] )  C does not guaranteethat ⇡ will be feasible in the real MDP (i.e., 11   P s,a ⇢ ⇡ T ( s, a ) c ( s, a )  C ) because the vanillamodel-based optimization does not account for the model error’s impact on the policy cost estimation, d F ( ˆ T ( s, a ) , T ( s, a )) .
Revised: Attaining 11 − γ (cid:80) s,a ρ π ˆ T ( s, a ) c ( s, a ) ≤ C does not guarantee that π will be feasible in the real MDP (i.e., 11 − γ (cid:80) s,a ρ πT ( s, a ) c ( s, a ) ≤ C ) because the vanilla model-based optimization does not account for the model error’s impact on the policy cost estimation, βd F ( ˆ T ( s, a ) , T ( s, a )) .

Original: To enable model-based safe RL that can transfer feasibility from the model to the real world, fora ﬁxed learned transition model ˆ T , we seek a cost penalty function u ˆ T : S ⇥ A !
Revised: To enable model-based safe RL that can transfer feasibility from the model to the real world, for a ﬁxed learned transition model ˆ T , we seek a cost penalty function u ˆ T : S ×

Original:  R such thatd F ( ˆ T ( s, a ) , T ( s, a ))
Revised: A → R such that d F [Equation] ˆ T ( s, a ) , T [Equation]

Original:  u ˆ T ( s, a ) , 8 s, a .
Revised: s, a )) ≤ u ˆ T ( s, a ) , ∀ s, a .

Original: If such a function exists, then we can solve the following
Revised: If such a function exists, then we can solve the following

Original: LP:
Revised: LP

Original: [Equation]
Revised: [Equation]

Original: We can guarantee that the solution policy ⇡ of Eq (5) is feasible for T —in particular, note that
Revised: We can guarantee that the solution policy π of Eq (5) is feasible for T —in particular, note that

Original: [Equation]
Revised: [Equation]

Original: However, this result is not useful if we cannot compute
Revised: However, this result is not useful if we cannot compute

Original: d F ( ˆ T ( s, a ) , T ( s, a )) .
Revised: d F ( ˆ T ( s, a ) , T ( s, a )) .

Original: A suitable functionclass for analysis is F = { f : k f k 1  1 } , which typically can be satisﬁed with Assumption 4.1 160 since the per-step cost is bounded.
Revised: A suitable function class for analysis is F = { f : (cid:107) f (cid:107) ∞ ≤ 1 } , which typically can be satisﬁed with Assumption 4. since the per-step cost is bounded.

Original: Then, for the tabular-MDP setting (i.e., ﬁnite state and actionspace), we can in fact obtain a strong probabilistic guarantee on feasibility.
Revised: Then, for the tabular-MDP setting (i.e., ﬁnite state and action space), we can in fact obtain a strong probabilistic guarantee on feasibility.

Original: Theorem 4.3 (Tabular Case High-Probability Feasibility Guarantee) .
Revised: Theorem 4.3 (Tabular Case High-Probability Feasibility Guarantee) .

Original: Assume F = { f : k f k 1  1 }and that Assumption 4.1 holds.
Revised: Assume F = { f : (cid:107) f (cid:107) ∞ ≤ 1 } and that Assumption 4.1 holds.

Original: Deﬁne u
Revised: Deﬁne u

Original: ( s, a ) :
Revised: ( s, a ) :

Original: = q |S| 8 n ( s,a ) ln 4 |S||A|  , where n ( s, a ) is the count of( s, a ) in D and  2 (0 , 1] .
Revised: = (cid:113) |S| 8 n ( s,a ) ln 4 |S||A| δ , where n ( s, a ) is the count of ( s, a ) in D and δ ∈ (0 , 1] .

Original: Then, with probability 1   , a policy that is feasible for Eq (5) is alsofeasible for Eq (2) .
Revised: Then, with probability 1 − δ , a policy that is feasible for Eq (5) is also feasible for Eq (2) .

Original: Furthermore, we can extend this result to guarantee that all intermediate solutions during training aresafe.
Revised: Furthermore, we can extend this result to guarantee that all intermediate solutions during training are safe.

Original: Corollary 4.4 (High-Probability Zero-Training-Violations Guarantee) .
Revised: Corollary 4.4 (High-Probability Zero-Training-Violations Guarantee) .

Original: Assume the same set ofassumptions as Theorem 4.3 and that the training lasts for K episodes.
Revised: Assume the same set of assumptions as Theorem 4.3 and that the training lasts for K episodes.

Original: Then, for any  2 (0 , 1] ,deﬁne u ( s, a ) :
Revised: Then, for any δ ∈ (0 , 1] , deﬁne u ( s, a ) :

Original: = q |S| 8 n ( s,a ) ln 4 K |S||A|  .
Revised: = (cid:113) |S| 8 n ( s,a ) ln 4 K |S||A| δ .

Original: Then, with probability 1   , all intermediate solutions to
Revised: Then, with probability 1 − δ , all intermediate solutions to

Original: Eq (5) are feasible for Eq (2) .
Revised: Eq (5) are feasible for Eq (2) .

Original: Proofs are given in Appendix A. At a high level, Theorem 4.3 follows from observing that d F is thetotal variation distance for the chosen F and applying concentration bound on the estimation error ofˆ T .
Revised: Proofs are given in Appendix A. At a high level, Theorem 4.3 follows from observing that d F is the total variation distance for the chosen F and applying concentration bound on the estimation error of ˆ T .

Original: Then, Corollary 4.4 can be shown by a union-bound argument.
Revised: Then, Corollary 4.4 can be shown by a union-bound argument.

Original: Together, these results suggest that a principled way of incorporating a conservative penalty functioninto the 3-step basic model-based safe RL framework described at the end of Sec.
Revised: Together, these results suggest that a principled way of incorporating a conservative penalty function into the 3-step basic model-based safe RL framework described at the end of Sec.

Original: 3 is to replace theoriginal constrained MDP objective (i.e., Eq (2)) with its conservative variant (i.e., Eq (5)).
Revised: 3 is to replace the original constrained MDP objective (i.e., Eq (2)) with its conservative variant (i.e., Eq (5)).

Original: 4.2 Adaptive Cost Penalty
Revised: 4.2 Adaptive Cost Penalty

Original: The upper bound derived in the previous section can be overly conservative in practice.
Revised: The upper bound derived in the previous section can be overly conservative in practice.

Original: Thus,we derive an adaptive penalty function based on environment feedback to make it more practical.
Revised: Thus, we derive an adaptive penalty function based on environment feedback to make it more practical.

Original: First, we observe that the conservative penalty modiﬁcation described above is not yet enough for apractical algorithm, because the proposed penalty function as in the theorem or the corollary is tooconservative, to the extent that Eq (5) might admit no solutions.
Revised: First, we observe that the conservative penalty modiﬁcation described above is not yet enough for a practical algorithm, because the proposed penalty function as in the theorem or the corollary is too conservative, to the extent that Eq (5) might admit no solutions.

Original: In practice, it is often estimated asu ( s, a ) :=
Revised: In practice, it is often estimated as u ( s, a ) :

Original:  / p n ( s, a ) , where  2 R is some scaling parameter.[Table] [Table]
Revised: = κ/ (cid:112) n ( s, a ) , where κ ∈ R is some scaling parameter.

Original:  , experience [Table] [Table] [Table]
Revised: 1: Inputs: Transition model ˆ T θ , experience buffer D , cost limit C , initial κ value, κ learning rate α 2: with random policy

Original: policy [Table] do 
Revised: do penalty using

Original: [Table]

Original: # Conservative penalty 5:

Original: Train ˆ T ✓ using D 6:

Original: Optimize ⇡ using Eq (5) (LP) or Eq (7) (CCEM) 7:
Revised: Optimize π using Eq (5) (LP) or Eq (7) (CCEM) 7:

Original: Collect trajectory ⌧ := { ( s t , a t , r t , c t , s t +1 ) } and store to buffer D = D [ { ⌧ } 8: # Adaptive penalty 9:
Revised: Collect trajectory τ := { ( s t , a t , r t , c t , s t +1 ) } and store to buffer D = D ∪ { τ } 8: # Adaptive penalty 9:

Original: Compute J c ( ⇡ t ) = P t =

Revised: Compute J c ( π t ) = (cid:80) t =

Original:  t c t 10:
Revised: γ t c t 10:

Original: Update   + ↵ ( J c ( ⇡ t )  C ) 11: end for
Revised: Update κ ← κ + α ( J c ( π t ) − C ) 11: end for

Original: We observe that setting  to a ﬁxed valuethroughout training can lead to poor performance.
Revised: We observe that setting κ to a ﬁxed value throughout training can lead to poor performance.

Original: Different scales may be appropriatefor use with different environments, tasks, andstages of training.
Revised: Different scales may be appropriate for use with different environments, tasks, and stages of training.

Original: If it is set too low, then thecost penalty may not be large enough to ensuresafety.
Revised: If it is set too low, then the cost penalty may not be large enough to ensure safety.

Original: On the other hand, if it is set too large,then the model may be overly conservative, discouraging exploration and leading to traininginstability.
Revised: On the other hand, if it is set too large, then the model may be overly conservative, discouraging exploration and leading to training instability.

Original: To avoid these issues, we propose to adaptivelyupdate  during training.
Revised: To avoid these issues, we propose to adaptively update κ during training.

Original: A key observation wemake is that the effect of a particular  value hason a policy’s true cost in the environment can bemeasured from executing this policy in the realenvironment.
Revised: A key observation we make is that the effect of a particular κ value has on a policy’s true cost in the environment can be measured from executing this policy in the real environment.

Original: Leveraging this insight, we can infact view the co-evolution of the policy and thelearned transition model as a control plant, forwhich the policy cost is the control output; then, can be viewed as its control input.
Revised: Leveraging this insight, we can in fact view the co-evolution of the policy and the learned transition model as a control plant, for which the policy cost is the control output; then, κ can be viewed as its control input.

Original: Now, to set , we employ an integral controller, a simple variant of the widely used PID controller [6] fromclassic control, to incrementally update  based on the current gap between the policy’s true cost andthe cost threshold.
Revised: Now, to set κ , we employ a PI controller, a simple variant of the widely used PID controller [6] from classic control, to incrementally update κ based on the current gap between the policy’s true cost and the cost threshold.

Original: More precisely, we propose the following integral-control update rule:
Revised: More precisely, we propose the following PI control update rule:

Original: [Equation]
Revised: [Equation]

Original: (6) where ↵ is the learning rate.
Revised: where α is the learning rate.

Original: This update rule is intuitive.
Revised: This update rule is intuitive.

Original: Consider the direction of the  updatewhen J c ( ⇡ t ) < C .
Revised: Consider the direction of the κ update when J c ( π t ) < C .

Original: In this case, the update will be negative, which matches our intuition that thecost penalty can be applied less conservatively due to the positive margin to the cost limit C .
Revised: In this case, the update will be negative, which matches our intuition that the cost penalty can be applied less conservatively due to the positive margin to the cost limit C .

Original: Theargument for the case J c ( ⇡ t ) >
Revised: The argument for the case J c ( π t ) >

Original: C is analogous.
Revised: C is analogous.

Original: In high-dimensional environments, as the fullexpected cost cannot be computed exactly, and we instead approximate it using a single episode (i.e.,the current policy ⇡ t rollout in the environment).
Revised: In high-dimensional environments, as the full expected cost cannot be computed exactly, and we instead approximate it using a single episode (i.e., the current policy π t rollout in the environment).

Original: Now, the full CAP approach is described in Algorithm 1.
Revised: Now, the full CAP approach is described in Algorithm 1.

Original: At a high level, CAP extends upon thebasic model-based safe RL framework by (1) solving the conservative LP (Line 7, Eq (5)), and (2)adapting  using integral-control (Lines 10 & 11).
Revised: At a high level, CAP extends upon the basic model-based safe RL framework by (1) solving the conservative LP (Line 7, Eq (5)), and (2) adapting κ using PI control (Lines 10 & 11).

Original: We set the initial value for  using an exponential search mechanism, which we describe in the Appendix.
Revised: We set the initial value for κ using an exponential search mechanism, which we describe in the Appendix.

Original: We validate this LP formulation of CAPusing a gridworld environment in our experiments.
Revised: We validate this LP formulation of CAP using a gridworld environment in our experiments.

Original: 4.3 CAP for high-dimensional states
Revised: 4.3 CAP for high-dimensional states

Original: Note that this tabular LP variant of CAP cannot extend to environments with continuous state andaction spaces, representative of many high-dimensional RL problems of interest (e.g., robotics); theircontinuous nature precludes enumerating all state-action pairs, which is needed to express the linearprogram.
Revised: Note that this tabular LP variant of CAP cannot extend to environments with continuous state and action spaces, representative of many high-dimensional RL problems of interest (e.g., robotics); their continuous nature precludes enumerating all state-action pairs, which is needed to express the linear program.

Original: Therefore, we propose a scalable implementation of CAP amenable to continuous controlproblems.
Revised: Therefore, we propose a scalable implementation of CAP amenable to continuous control problems.

Original: First, we revert back to the policy-based formulation in Eq (1), and deﬁne the followingequivalent objective:
Revised: First, we revert back to the policy-based formulation in Eq (1), and deﬁne the following equivalent objective:

Original: [Equation]
Revised: [Equation]

Original: where u ( s t , a t ) is a heuristic penalty function based on statistics of the learned transition model.
Revised: where u ( s t , a t ) is a heuristic penalty function based on statistics of the learned transition model.

Original: To optimize Eq (7), we employ the constrained cross entropy method (CCEM)
Revised: To optimize Eq (7), we employ the constrained cross entropy method (CCEM)

Original: [7, 33] as our trajectoryoptimizer; the procedure is summarized in Algorithm 2 in the Appendix.
Revised: [7, 33] as our trajectory optimizer; the procedure is summarized in Algorithm 2 in the Appendix.

Original: At a high level, CCEM ﬁrstsamples N action sequences (Line 4) and computes their values and costs (Line 5).
Revised: At a high level, CCEM ﬁrst samples N action sequences (Line 4) and computes their values and costs (Line 5).

Original: Then, if therewere more than E samples that satisfy the constraint, then the E samples with highest rewards areselected (Line 10); otherwise, the E samples with lowest costs are selected (Line 8).
Revised: Then, if there were more than E samples that satisfy the constraint, then the E samples with highest rewards are selected (Line 10); otherwise, the E samples with lowest costs are selected (Line 8).

Original: These selectedelite samples are used to update the sampling distribution (Line 12).
Revised: These selected elite samples are used to update the sampling distribution (Line 12).

Original: This process continues for Iiterations, and the eventual distribution mean is selected as the optimal action sequence (Line 14).
Revised: This process continues for I iterations, and the eventual distribution mean is selected as the optimal action sequence (Line 14).

Original: Next, we specify the choice of transition model and penalty function u ( s, a ) for state-based andvisual observation-based implementations, respectively.
Revised: Next, we specify the choice of transition model and penalty function u ( s, a ) for state-based and visual observation-based implementations, respectively.

Original: For the former, we model the environmenttransition function using an ensemble of size N , { ˆ T i ✓ = N ( µ i ✓ , ⌃ i ✓ ) } Ni =1 [9] and set u ( s, a )=max Ni =  ⌃ i ✓ ( s, a ) 
Revised: For the former, we model the environment transition function using an ensemble of size N , { ˆ T iθ = N ( µ iθ , Σ iθ ) } Ni =1 [9] and set u ( s, a ) = max Ni =1 (cid:13)(cid:13)

Original:  F to be the maximum Frobenius norm of the ensemble standard deviation outputs,as done for ofﬂine RL in [29].
Revised: Σ iθ ( s, a ) (cid:13)(cid:13) F to be the maximum Frobenius norm of the ensemble standard deviation outputs, as done for ofﬂine RL in [29].

Original: Our visual-based implementation builds on top of PlaNet [10], astate-of-art visual model-based transition model; here, we set u ( s, a ) to be the ensemble disagreementof one-step hidden state prediction models [34].
Revised: Our visual-based implementation builds on top of PlaNet [10], a state-of-art visual model-based transition model; here, we set u ( s, a ) to be the ensemble disagreement of one-step hidden state prediction models [34].

Original: See the Appendix for details.
Revised: See the Appendix for details.

Original:  Experiments
Revised:  Experiments

Original: CAP provides a general, principled, and practical framework for applying MBRL to safe RL.
Revised: CAP provides a general, principled, and practical framework for applying MBRL to safe RL.

Original: Tosupport this claim, we comprehensively evaluate CAP against its ablations as well as model-freebaselines in various environments.
Revised: To support this claim, we comprehensively evaluate CAP against its ablations as well as model-free baselines in various environments.

Original: More speciﬁcally, we investigate the following questions:
Revised: More speciﬁcally, we investigate the following questions:

Original: (Q1) Does CAP’s theoretical guarantees approximately hold in tabular environments?
Revised: (Q1) Does CAP’s theoretical guarantees approximately hold in tabular environments?

Original: (Q2) Does CAP improve reward and safety upon its ablations (i
Revised: (Q2) Does CAP improve reward and safety upon its ablations (i

Original: e., ﬁxed  values)?
Revised: e., ﬁxed κ values)?

Original: (Q3)
Revised: (Q3)

Original: Is CAP more sample and cost efﬁcient than state-of-art model-free baselines?
Revised: Is CAP more sample and cost efﬁcient than state-of-art model-free baselines?

Original: (Q4) Can CAP learn safe policies even with high-dimensional inputs, such as images?
Revised: (Q4) Can CAP learn safe policies even with high-dimensional inputs, such as images?

Original: We investigate Q1-2 using a gridworld environment, and Q2-4 on two high-dimensional deep RLbenchmarks.
Revised: We investigate Q1-2 using a gridworld environment, and Q 4 on two high-dimensional deep RL benchmarks.

Original: Our code is included in the supplementary materials.
Revised: Our code is included in the supplementary materials.

Original: 5.1 Gridworld
Revised: 5.1 Gridworld

Original: We begin by validating our theoretical ﬁndings in tabular gridworld, where we can solve the constrained optimization problem (Eq (5)) exactly using standard LP algorithms.
Revised: We begin by validating our theoretical ﬁndings in tabular gridworld, where we can solve the constrained optimization problem (Eq (5)) exactly using standard LP algorithms.

Original: 5.1.1 Environment, Methods, Training Details
Revised: 5.1.1 Environment, Methods, Training Details

Original: We consider a 8 ⇥ 8 gridworld with stochastic transitions; the reward and the cost functions arerandomly generated Bernoulli distributions drawn according to a Beta prior.
Revised: We consider a 8 × 8 gridworld with stochastic transitions; the reward and the cost functions are randomly generated Bernoulli distributions drawn according to a Beta prior.

Original: In addition to CAP, wecompare against CAP ablations with ﬁxed  values of 0 , 0 . 01 , 0 . 05 , and 0 . 1 ;  = 0 corresponds tothe basic MBRL approach without penalty.
Revised: In addition to CAP, we compare against CAP ablations with ﬁxed κ values of 0 , 0 . 01 , 0 . 05 , and 0 . 1 ; κ = 0 corresponds to the basic MBRL approach without penalty.

Original: We also include the oracle LP solution computed usingthe true environment dynamics.
Revised: We also include the oracle LP solution computed using the true environment dynamics.

Original: For each method (except the oracle), the training procedure lasts30 iterations, in which each iterate includes (1) collecting 500 samples using the current LP solution, (2)updating ˆ T , and (3) solving the new conservative LP objective.
Revised: For each method (except the oracle), the training procedure lasts iterations, in which each iterate includes (1) collecting 500 samples using the current LP solution, (2) updating ˆ T , and (3) solving the new conservative LP objective.

Original: See Appendix for more environmentand training details.
Revised: See Appendix for more environment and training details.

Original: 5.1.2 Metrics & Results
Revised: 5.1.2 Metrics & Results

Original: In Figure 1, we illustrate the training curves for the return, cost, the cumulative number of intermediatepolicy solutions that violate the cost threshold.
Revised: In Figure 1, we illustrate the training curves for the return, cost, the cumulative number of intermediate policy solutions that violate the cost threshold.

Original: The ﬁrst two metrics are standard, and the violationmetric measures how safely a method explores.
Revised: The ﬁrst two metrics are standard, and the violation metric measures how safely a method explores.

Original: We additionally illustrate the training evolution of  .
Revised: We additionally illustrate the training evolution of κ .

Original: These curves are the averages taken over the 100 gridworld simulations; we defer standard deviationerror bar to the Appendix for better visualizations except for the kappa curve.
Revised: These curves are the averages taken over the 100 gridworld simulations; we defer standard deviation error bar to the Appendix for better visualizations except for the kappa curve.

Original: As expected, CAP (  = 0 ), due to its asymptotically consistent nature, converges to the oracle astraining continues; however, this comes at the cost of the highest number of training violations,precisely due to the lack of an uncertainty-aware penalty function.
Revised: As expected, CAP ( κ = 0 ), due to its asymptotically consistent nature, converges to the oracle as training continues; however, this comes at the cost of the highest number of training violations, precisely due to the lack of an uncertainty-aware penalty function.

Original: In sharp contrast, CAP is veryclose to the oracle in both reward and cost, and does so without incurring a single violation in all 100trials, as indicated by its ﬂat horizontal line at 0 in the violation plot.
Revised: In sharp contrast, CAP is very close to the oracle in both reward and cost, and does so without incurring a single violation in alltrials, as indicated by its ﬂat horizontal line at 0 in the violation plot.

Original: These results validate our keytheoretical claims that when the cost penalty is applied properly, the resulting policy is guaranteed tobe safe (Theorem 4.3); furthermore, it applies to all intermediate policies during training (Corollary4.4), answering Q1 above.
Revised: These results validate our key theoretical claims that when the cost penalty is applied properly, the resulting policy is guaranteed to be safe (Theorem 4.3); furthermore, it applies to all intermediate policies during training (Corollary 4.4), answering Q1 above.

Original: On the other hand, CAP ablations with ﬁxed  values, though constraint-satisfying at the end, incurhigher number of violations and achieve sub-optimal solutions, validated by their lower returns andconservative costs.
Revised: On the other hand, CAP ablations with ﬁxed κ values, though constraint-satisfying at the end, incur higher number of violations and achieve sub-optimal solutions, validated by their lower returns and conservative costs.

Original: Interestingly, while all these variants on average satisfy the constraint from
Revised: Interestingly, while all these variants on average satisfy the constraint from

Original: Episode 2 and on (i.e., their average costs are below the threshold of 0 .
Revised: Episode 2 and on (i.e., their average costs are below the threshold of 0 .

Original: 1 in the cost plot), their averagenumbers of violations uniformly increase throughout training.
Revised: 1 in the cost plot), their average numbers of violations uniformly increase throughout training.

Original: This suggests that ﬁxed  values arenot robust to random gridworld simulations, as the same value may be too modest for some randomdraws and hence incur violations, and too aggressive for some other draws and lead to suboptimalsolutions.
Revised: This suggests that ﬁxed κ values are not robust to random gridworld simulations, as the same value may be too modest for some random draws and hence incur violations, and too aggressive for some other draws and lead to suboptimal solutions.

Original: Indeed, we observe greater variance in the performance of ﬁxed  ablations than CAP (see
Revised: Indeed, we observe greater variance in the performance of ﬁxed κ ablations than CAP (see

Original: Appendix).
Revised: Appendix).

Original: In contrast, CAP automatically ﬁnds suitable sequences of  s for each simulation, evidenced by thelarge variance the  sequences exhibit over the simulations.
Revised: In contrast, CAP automatically ﬁnds suitable sequences of κ s for each simulation, evidenced by the large variance the κ sequences exhibit over the simulations.

Original: Its zero-violation and lower variancein all metrics suggest that the adaptive penalty mechanism has the additional beneﬁt of beingdistributionally robust to the randomness in the environment distribution.
Revised: Its zero-violation and lower variance in all metrics suggest that the adaptive penalty mechanism has the additional beneﬁt of being distributionally robust to the randomness in the environment distribution.

Original: Finally, the overalldownward oscillating trend indeed reﬂects CAP’s effectiveness at using feedback to optimize rewardand cost simultaneously.
Revised: Finally, the overall downward oscillating trend indeed reﬂects CAP’s effectiveness at using feedback to optimize reward and cost simultaneously.

Original: Together, these ablations answer Q2 afﬁrmatively.
Revised: Together, these ablations answer Q2 afﬁrmatively.

Original: 5.2 High-Dimensional Environments
Revised: 5.2 High-Dimensional Environments

Original: Next, we evaluate CAP’s generality and effectiveness in high-dimensional environments.
Revised: Next, we evaluate CAP’s generality and effectiveness in high-dimensional environments.

Original: We beginby summarizing our experimental setup; details are in the Appendix.
Revised: We begin by summarizing our experimental setup; details are in the Appendix.

Original: 5.2.1 Environments
Revised: 5.2.1 Environments

Original: We consider two deep RL environments, spanning different input modalities, constraint types, andassociated cost types.
Revised: We consider two deep RL environments, spanning different input modalities, constraint types, and associated cost types.

Original: We describe these environments here; see Figure 2 for illustrations.
Revised: We describe these environments here; see Figure 2 for illustrations.

Original: We use
Revised: We use

Original: HalfCheetah [35] as our state-based environment; here, the cost function is the agent’s horizontalvelocity, and we set the cost constraint to be the 50% of the average velocity of an unconstrainedexpert PPO agent, 152.
Revised: HalfCheetah [35] as our state-based environment; here, the cost function is the agent’s horizontal velocity, and we set the cost constraint to be the 50% of the average velocity of an unconstrained expert PPO agent, 152.

Original: For the image-based environment, we use Car-Racing [36].
Revised: For the image-based environment, we use Car-Racing [36].

Original: The tracks arerandomized every episode.
Revised: The tracks are randomized every episode.

Original: The state space is a 64 ⇥ 64 ⇥ 3 top-down view of the car; the actionspace is continuous and 3-dimensional (steering, acceleration, and braking).
Revised: The state space is a 64 × 64 × 3 top-down view of the car; the action space is continuous and 3-dimensional (steering, acceleration, and braking).

Original: A per-step cost of 1 isincurred if any wheels skid from excessive force; the cost limit is 0, indicating that the car shouldnever skid.
Revised: A per-step cost of 1 is incurred if any wheels skid from excessive force; the cost limit is 0, indicating that the car should never skid.

Original: 5.2.2 Baselines
Revised: 5.2.2 Baselines

Original: [Figure]

Original: In these high-dimensional settings, we compareagainst both deep model-free safe RL baselinesas well as CAP ablations.
Revised: In these high-dimensional settings, we compare against both deep model-free safe RL baselines as well as CAP ablations.

Original: To this end, we include PPO-Lagrangian ( PPO-LAG ), which iterates between PPO policy update and cost lagrangian parameter update to simultaneouslyoptimize return and constraint satisfaction; despite its simplicity, PPO-LAG has been shownto be a strong safe RL baseline [13].
Revised: To this end, we include PPO-Lagrangian ( PPO-LAG ), which iterates between PPO policy update and cost lagrangian parameter update to simultaneously optimize return and constraint satisfaction; despite its simplicity, PPO-LAG has been shown to be a strong safe RL baseline [13].

Original: Additionally, we include FOCOPS [28], a state-of-artmodel-free algorithm which uses ﬁrst-order projection methods to ensure that constraint satisfaction minimally deteriorates policy return.
Revised: Additionally, we include FOCOPS [28], a state-of-art model-free algorithm which uses ﬁrst-order projection methods to ensure that constraint satisfaction minimally deteriorates policy return.

Original: Finally,we include PPO [19] in order to provide comparison to an unconstrained method.
Revised: Finally, we include PPO [19] in order to provide comparison to an unconstrained method.

Original: Finally, as in thegridworld experiment, we consider CAP ablations with ﬁxed  values and separately visualize thetraining curves.
Revised: Finally, as in the gridworld experiment, we consider CAP ablations with ﬁxed κ values and separately visualize the training curves.

Original: We use  = 0 , 0 . 1 , 1 , 10 for both HalfCheetah and Car-Racing to include a widerange of magnitudes; in particular,  = 0 corresponds to the basic model-based safe RL approachwithout the conservative penalty.
Revised: We use κ = 0 , 0 . 1 , 1 , 10 for both HalfCheetah and Car-Racing to include a wide range of magnitudes; in particular, κ = 0 corresponds to the basic model-based safe RL approach without the conservative penalty.

Original: 5.2.3 Training Details & Evaluation Metrics
Revised: 5.2.3 Training Details & Evaluation Metrics

Original:  [Table]
Revised: [Figure] [Table]

Original: substantially more sample-efﬁcient with respect to both return and cost.
Revised: substantially more sample-efﬁcient with respect to both return and cost.

Original: In addition, it is much safer during training, as demonstrated bythe signiﬁcantly fewer violations.
Revised: In addition, it is much safer during training, as demonstrated by signiﬁcantly fewer violations

Original: For model-free algorithms, we train using 1Menvironment steps, and for our model-basedalgorithms, we train using 100K steps for
Revised: For model-free algorithms, we train using 1M environment steps, and for our model-based algorithms, we train using 100K steps for

Original: HalfCheetah and 200K steps for Car-Racing.
Revised: HalfCheetah and 200K steps Car-

Original: An episode in both environments is 1000 steps.
Revised: An episode in both environments is 1000 steps.

Original: We report results on Car-Racing at 200k sinceit is more challenging to learn the dynamicsof a visual environment; both model-free andmodel-based methods take more steps to converge in Car-Racing.
Revised: We report results on Car-Racing at 200k since it is more challenging to learn the dynamics of a visual environment; both model-free and model-based methods take more steps to converge in Car-Racing.

Original: As in gridworld, we reportthe training curves of the return, cost, and cumulative episodic violations; they are included inthe Appendix.
Revised: As in gridworld, we report the training curves of the return, cost, and cumulative episodic violations; they are included in the Appendix.

Original: In the main text, we report a numerical “snapshot" version of these curves at theend of training (average over last 10 episodes);for model-free baselines, we also report thesemetrics after 100K/200K steps to have a head-tohead comparison against CAP.
Revised: In the main text, we report a numerical “snapshot" version of these curves at the end of training (average over last 10 episodes); for model-free baselines, we also report these metrics after 100K/200K steps to have a head-tohead comparison against CAP.

Original: For all methods,we report their hyperparameters as well as implementation details in the Appendix.
Revised: For all methods, we report their hyperparameters as well as implementation details in the Appendix.

Original: In Appendix, wealso supply videos of trained CAP agents in Car-Racing.
Revised: In Appendix, we also supply videos of trained CAP agents in Car-Racing.

Original: 5.2.4 Baseline Comparisons Results
Revised: 5.2.4 Baseline Comparisons Results

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: While the most competitive algorithm FOCOPS matches CAP’sreturn and cost with 1 million environmental steps in both environments, CAP requires about 5-10 ⇥fewer steps, demonstrating its sample efﬁciency.
Revised: While the most competitive algorithm FOCOPS matches CAP’s return and cost with 1 million environmental steps in both environments, CAP requires about 5-10 × fewer steps, demonstrating its sample efﬁciency.

Original: Furthermore, the relative performance of CAP at 100K/200K steps is signiﬁcantly better than all model-free algorithms, which have not learned agood policy by that point.
Revised: Furthermore, the relative performance of CAP at 100K/200K steps is signiﬁcantly better than all model-free algorithms, which have not learned a good policy by that point.

Original: This has direct implication for safety.
Revised: This has direct implication for safety.

Original: On the Car-Racing environment,because model-free methods learn much slower, they also spend more training episodes violatingthe constraint.
Revised: On the Car-Racing environment, because model-free methods learn much slower, they also spend more training episodes violating the constraint.

Original: On HalfCheetah, all methods achieve 0 cumulative episodic violations with 100Ksteps (i.e., 100 episodes), but this is because in HalfCheetah the algorithm will not violate the speedconstraint initially because it has not learned the running behavior yet.
Revised: On HalfCheetah, all methods achieve 0 cumulative episodic violations with 100K steps (i.e., 100 episodes), but this is because in HalfCheetah the algorithm will not violate the speed constraint initially because it has not learned the running behavior yet.

Original: It is particularly illuminating to observe the cumulative episodic violations at the end of each method’straining: we see that CAP violates the speed constraint in HalfCheetah for fewer than 2 episodes out ofits 100 training episodes, while all baselines violate this constraint at much higher rates and volumes.
Revised: It is particularly illuminating to observe the cumulative episodic violations at the end of each method’s training: we see that CAP violates the speed constraint in HalfCheetah for fewer than 2 episodes out of its 100 training episodes, while all baselines violate this constraint at much higher rates and volumes.

Original: This conﬁrms that these model-free methods struggle to ensure safety during training regardless ofthe safety of their ﬁnal policy, while CAP is able to minimize violations throughout training.
Revised: This conﬁrms that these model-free methods struggle to ensure safety during training regardless of the safety of their ﬁnal policy, while CAP is able to minimize violations throughout training.

Original: Onthe more challenging image-based Car-Racing environment, CAP cannot avoid training violationsentirely, but manages to signiﬁcantly reduce them compared to the baselines.
Revised: On the more challenging image-based Car-Racing environment, CAP cannot avoid training violations entirely, but manages to signiﬁcantly reduce them compared to the baselines.

Original: These comparisonsprovide strong evidence for Q3 and Q4.
Revised: These comparisons provide strong evidence for Q3 and Q4.

Original: 5.2.5 CAP Ablations Results
Revised: 5.2.5 CAP Ablations Results

Original: The training curves of CAP as well as its ablations are illustrated in Figure 3.
Revised: The training curves of CAP as well as its ablations are illustrated in Figure 3.

Original: Consistent with ourﬁndings in gridworld, setting  to a ﬁxed value is rarely desirable.
Revised: Consistent with our ﬁndings in gridworld, setting κ to a ﬁxed value is rarely desirable.

Original: Setting it too low often leads tosolutions that fail to satisfy constraint, suggested by the high training cost and violations of CAP(  = 0 . 0 , 0 . 1) in both environments; setting it too high often precludes reward learning in the ﬁrstplace, evidenced by the training curves of CAP (  = 10 . 0) in both environments.
Revised: Setting it too low often leads to solutions that fail to satisfy constraint, suggested by the high training cost and violations of CAP ( κ = 0 . 0 , 0 . 1) in both environments; setting it too high often precludes reward learning in the ﬁrst place, evidenced by the training curves of CAP ( κ = 10 . 0) in both environments.

Original: Furthermore, sincethe cost limit is 0 on Car-Racing, exploration will always violate the constraint initially.
Revised: Furthermore, since the cost limit is 0 on Car-Racing, exploration will always violate the constraint initially.

Original: Hence, wecan additionally measure the safe exploration of a method by its slope on the violation curve: thelower the slope, the fewer violations a method incurs as training goes on.
Revised: Hence, we can additionally measure the safe exploration of a method by its slope on the violation curve: the lower the slope, the fewer violations a method incurs as training goes on.

Original: There, we see that CAPhas the ﬂattest violation slope out of all variants that learn policies with non-trivial driving behavior.
Revised: There, we see that CAP has the ﬂattest violation slope out of all variants that learn policies with non-trivial driving behavior.

Original: These results once again show that CAP is preferrable to its ﬁxed-  variants, providing a positiveanswer to Q2.
Revised: These results once again show that CAP is preferrable to its ﬁxed- κ variants, providing a positive answer to Q2.

Original:  Conclusion
Revised:  Conclusion

Original: We have presented CAP, a general model-based safe reinforcement learning framework.
Revised: We have presented CAP, a general model-based safe reinforcement learning framework.

Original: We havederived a linear programming formulation and proven that we can guarantee safety by using aconservative penalty; this penalty is then made adaptive based on environmental feedback to makeit practically useful.
Revised: We have derived a linear programming formulation and proven that we can guarantee safety by using a conservative penalty; this penalty is then made adaptive based on environmental feedback to make it practically useful.

Original: We have validated our theoretical results in a tabular gridworld environment. Furthermore, we demonstrate that CAP can be easily extended to high-dimensional environmentsthrough appropriate choices of optimizer and transition models.
Revised: We have validated our theoretical results in a tabular gridworld environment and demonstrated that CAP can be easily extended to high-dimensional environments through appropriate choices of optimizer and transition models.

Original: In future work, we aim to implement CAP using model-based policy optimization [37, 38] methods, which have shown to attain betterperformance in practice.
Revised: In future work, we aim to implement CAP using modelbased policy optimization [37, 38] methods, which have shown to attain better performance in practice.

Original: Additionally, we believe that using a full PID controller will further improvesafety of CAP, especially on problems where incremental update is not aggressive enough for safety.
Revised: Additionally, we believe that using a full PID controller will further improve safety of CAP, especially on problems where incremental update is not aggressive enough for safety.

Original: Overall, we believe that CAP opens many future directions in making MBRL practically useful forsafe RL. 
Revised: Overall, we believe that CAP opens many future directions in making MBRL practically useful for safe RL. A Proofs In this section [Equation]

Revised: we provide proofs for the theoretical results appeared in Section 4.

Revised: We will restate each of the results and then append their corresponding proof.

Revised: Lemma A.1 (Cost Simulation Lemma and Upper Bound) .

Revised: Let the F -induced IPM be deﬁned as

Revised: [Equation]

Revised: Then, the difference between the expected policy cost computed using T and ˆ T is bounded above:

Revised: [Equation]

Revised: Proof.

Revised: Using the telescoping lemma [

Revised: [Equation]

Revised: [Equation]

Revised: we have that

Revised: [Equation]

Revised: Then, by Assumption 4.

Revised: [Equation]

Revised: we have that

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: Putting everything together, we have that

Revised: [Equation]

Revised: Theorem A.2 (Tabular Case High-Probability Feasibility Guarantee) .

Revised: Assume F = { f : (cid:107) f (cid:107) ∞ ≤ 1 } and that Assumption 4.1 holds.

Revised: Deﬁne u

Revised: ( s, a ) :

Revised: = (cid:113) |S| 8 n ( s,a ) ln 4 |S||A| δ , where n ( s, a ) is the count of ( s, a ) in D and δ ∈ (0 , 1] .

Revised: Then, with probability 1 − δ , a policy that is feasible for Eq (5) is also feasible for Eq (2) .

Revised: Proof.

Revised: In order for a policy that is feasible for Eq (5) is also feasible for Eq (2), we need to have

Revised: [Equation]

Revised: By the lemma above, this is equivalent to having u ( s, a ) ≥ d F ( ˆ T ( s, a ) , T ( s, a )) , ∀ s, a .

Revised: Since, we assume F = { f : (cid:107) f (cid:107) ∞ ≤ 1 } , this implies

Revised: [Equation]

Revised: where the last step follows because ˆ T ( s, a ) and T ( s, a ) are multinomial distributions, which are countable.

Revised: Then, we need

Revised: [Equation]

Revised: By Hoeffding’s inequality and the l 1 concentration bound for multinomial distribution, we have that, for any δ > 0 , we can set u ( s, a ) :

Revised: = (cid:113) |S| 8 n ( s,a ) ln 4 |S||A| δ , then Eq (10) will hold with probability 1 − δ , completing the proof.

Revised: Corollary A.3 (High-Probability Zero-Training-Violations Guarantee) .

Revised: Assume the same set of assumptions as Theorem A.2 and that the training lasts for K episodes.

Revised: Then, for any δ ∈ (0 , 1] , deﬁne u ( s, a ) :

Revised: = (cid:113) |S| 8 n ( s,a )

Revised: ln 4 K |S||A| δ .

Revised: Then, with probability 1 − δ , all intermediate solutions to

Revised: Eq (5) are feasible for Eq (2) .

Revised: Proof.

Revised: Since we want all K intermediate solutions to be feasible with probability 1 − δ , the fault tolerance for any individual intermediate solution is δ/K ; this follows from an union bound argument.

Revised: Therefore, we can adjust the concentration bound from Hoeffding’s inequality by a factor of K and obtain that by setting u ( s, a ) :

Revised: = (cid:113) |S| 8 n ( s,a ) ln 4 K |S||A| δ , with probability 1 − δ , we can guarantee all intermediate solutions to Eq (5) are feasible for Eq (2).

Revised: B CAP with Linear Programming

Revised: This implementation of CAP is described in detail in the main text.

Revised: Here, we describe the exponential search mechanism we use to initialize κ for the very ﬁrst training episode.

Revised: Starting with a high value for κ (e.g., 10), we use it to construct a new constrained optimization problem of form Eq (5) and attempt to solve it.

Revised: If the problem is infeasible, then we halve the value of κ and repeat the process.

Revised: We stop at the ﬁrst value of κ for which the problem is feasible, and this value is taken as the initialized κ value.

Revised: C CAP with Constrained Cross Entropy Method

Revised: In Algorithm 2, we provide the pseudocode for CAP implemented using constrained cross entropy method.

Revised: Here, we reiterate the algorithm description from the main text for completeness.

Revised: At a high level, CCEM ﬁrst samples N action sequences (Line 4) and computes their values and costs (Line 5).

Revised: Then, if there were more than E samples that satisfy the constraint, then the E samples with highest rewards are selected (Line 10); otherwise, the E samples with lowest costs are selected (Line 8).

Revised: These selected elite samples are used to update the sampling distribution (Line 12).

Revised: This process continues for I iterations, and the eventual distribution mean is selected as the optimal action sequence (Line 14).

Revised: D Gridworld Experimental Detail

Revised: The gridworld environment is of size 8 × 8 .

Revised: The action space consists of the four directional primitives:

Revised: Up, Down, Left, Right.

Revised: For each action, there is a 20% chance that slippage occurs and the agent moves in a random direction, introducing stochastic transitions to the environment.

Revised: The reward and the cost functions are randomly generated Bernoulli distributions drawn according to a Beta(1,3) prior.

Revised: Each state has uniform probability of being selected as the initial state for each episode.

Revised: The discount rate is 0 . 99 .

Revised: The cost threshold is kept at 0 .

Revised: 1 for all trials.

Revised: Training lasts 30 episodes, and we use Gurobi [39] as the LP solver in our implementation.

Revised: For the gridworld experiments, we also pursue a more aggressive way of updating κ .

Revised: After observing

Revised: J c ( π t ) for episode t , we set κ :=

Revised: [Equation]

Revised: s,a ) ; this amounts to a proportional PID controller.

Revised: 1: Inputs: Transition model estimate ˆ T θ , experience buffer D , cost limit C 2: CCEM Hyperparameters: Population size N , elite population size E , max iteration I , planning horizon H , initial sampling distribution N ( µ 0 , Σ 0 ) 3: for i = 1 , . .

Revised: , I do 4:

Revised: Sample N action sequences A 1 :

Revised: = { a 1 t } Ht =1 , . . .

Revised: , A N := { a Nt } Ht =1 ∼ N ( µ i − 1 , Σ i − 1 ) 5:

Revised: Evaluate the action sequences using Eq (7) by simulating trajectories in ˆ T θ 6:

Revised: Construct feasible set X :=

Revised: { A n | ˜ J c ( A n ) ≤ C, n ∈

Revised: [ N ] } 7: if |X| <

Revised: E then 8:

Revised: Construct elite set E :=

Revised: { The E sequences out of all { A n } Nn =1 with lowest costs } 9: else 10:

Revised: Construct elite set E :=

Revised: { The E sequences in X with highest rewards } 11: end if 12:

Revised: Compute µ i , Σ

Revised: i using Maximum Likelihood over E 13: end for 14: Outputs: Optimal action sequence { a ∗ 1 , ..., a ∗ H } := µ I

Revised: Gridworld

Revised: D.1 Additional results

Revised: In Figure 4, we illustrate the full version of Figure 1 with one standard deviation error bars added in.

Revised: In Table 2, we also show these results in table format.

Revised: As shown, CAP ablations with ﬁxed κ values exhibit greater variance in their performances over 100 random seeds; this supports the claim in the main text that ﬁxed κ values are more sensitive to the randomness in the environment distribution.

Revised: E High-Dimensional Environments Experimental Detail E.1 Environments

Revised: • Velocity Constrained HalfCheetah:

Revised: The state space is 17-dimensional and the action space is 6-dimensional.

Revised: We use the original environment reward, v − 110 a T a , v is the forward velocity.

Revised: The cost is | v | [28], meaning that there is a direct trade-off between cost and reward.

Revised: The cost limit is set to 152 , half of the average

Revised: speed of an unconstrained PPO expert agent [28]

Revised: [Table]

Revised: [Table]

Revised: • Constrained Car-Racing:

Revised: The state space is a top down image of the car and the surrounding track.

Revised: We downscale the image to 64 by 64 by 3.

Revised: For model-free baselines, we also stack the last 4 frames, as common in reinforcement learning on image based environments.

Revised: The action space is three dimensional, controlling steering, acceleration and braking.

Revised: Each value is continuous and bounded.

Revised: We use an action repeat of 2 to produce a better signal to the model [10].

Revised: We keep the original reward, which incentivizes the agent to drive through as many tiles as possible.

Revised: We use a binary cost that is 1 if the car skids.

Revised: Skidding is a part of the original environment; a wheel skids if it’s force exceeds the friction limit, which is different on grass and road surfaces.

Revised: E.2 Uncertainty Estimators

Revised: State-based environments: We model the environment transition function using an neural ensemble of size N , where network’s output neurons parameterize a Gaussian distribution ˆ T =

Revised: N ( µ ( s t , a t ) , Σ( s t , a t ) [9].

Revised: We set u ( s, a ) = max Ni =1 (cid:13)(cid:13)

Revised: Σ iθ ( s, a ) (cid:13) (cid:13) F to be the maximum Frobenius norm of the ensemble standard deviation outputs, as done for ofﬂine RL in [29].

Revised: Image-based environments

Revised: : We implement PlaNet [10], which models the environment transition function using a latent dynamics model with deterministic and stochastic transition states; we refer interested readers to the original paper for details.

Revised: PlaNet does not provide an uncertainty estimate because it only utilizes a single transition model.

Revised: To obtain an uncertainty estimate, we train a bootstrap ensemble of one-step hidden-state dynamics model as in [34].

Revised: Each one-step model in the ensemble predicts, from each deterministic state h , the next stochastic state.

Revised: We formulate our uncertainty estimator as u ( h, a ) =

Revised: V ar ( µ i ( h, a ) | i =

Revised: [1 ..K ]) , the variance of ensemble predictions { µ i } Ki =1 .

Revised: As in [34], to keep the scale of this uncertainty estimator similar to that of state-based uncertainty estimator, we multiply it by 10000.

Revised: E.3 Network Architecture

Revised: We use a neural network C to approximate the environment’s true cost function.

Revised: When the cost is continuous, the network’s output neurons parameterize a Gaussian distribution and we construct our conservative cost function as C ( s, a ) +

Revised: κu ( s, a ) .

Revised: When the cost is binary, the network outputs a logit and we construct our conservative cost function as 1 [ C ( s, a ) + u ( s, a ) > 0] .

Revised: To apply model free algorithms on imaged-based environments, we used a shared CNN module to encode the image input.

Revised: The network consists of 5 convolutional layers followed by a ReLU non-linearity.

Revised: 4x4 conv, 8, stride3x3 conv, 16, stride

Revised: [Table]

Revised: 3x3 conv, 32,

Revised: [Table]

Revised: 3x3 conv, 64,

Revised: [Table]

Revised: [Table]

Revised: E.4 Hyperparameters

Revised: In Table

Revised: [Table]

Revised: we include the hyperparameters we used for state-based and image-based experiments, respectively.

Revised: [Table]

Revised: [Table]

Revised: cost discount factor

Revised: [Table]

Revised: [Table]

Revised: the cost is binary,

Revised: [Table]

Revised: total cost per

Revised: [Table]

Revised: is directly interpretable

Revised: [Table]

Revised: E.5 Additional results

Revised: In Figure

Revised: [Table]

Revised: we illustrate the training curves of CAP and model free baselines in HalfCheetah and

Revised: Car-Racing.

Revised: For clarity, we focus on the ﬁrst 100K/200K steps.

Revised: The results are also presented in

Revised: Table 1.

Revised: In HalfCheetah, all model free methods have 0 cost violations in the ﬁrst 100K steps, this is because they have not learnt a running gait that can violate the speed costraint.

Revised: On the other hand, CAP is able to quickly a gait and keep cost below the limit, with less than two violations per 100 episodes.

Revised: In CarRacing, all methods have high cost violations because the cost limit is 0.

Revised: An initial random policy will violate the cost constraint and exploration will always risk violation.

Revised: Even still, we see that CAP dominates FOCOPS, obtaining better episode return with lower cost and total violations.

Revised: CAP has more cost violations than PPO-Lagrangian, but we see that this is because

Revised: PPO-Lagrangian degrades to a trivial policy that maintains a stationary position, obtaining negative return with minimal risk of cost violations.

Revised: E.6 Compute resources

Revised: We use a single GTX 2080 Ti with 32 cores to run our experiments, each run takes about 10 hours in clock time.