Original: C ONSERVATIVE S AFETY C RITICS FOR E XPLORATION
Revised: C ONSERVATIVE S AFETY C RITICS FOR E XPLORATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning.
Revised: Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning.

Original: In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration.
Revised: In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration.

Original: We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisﬁed with high probability during training, derive provable convergence bounds for our approach, which is no worse asymptotically than standard RL, and demonstrate the efﬁcacy of the proposed approach on a suite of challenging navigation, manipulation, and locomotion tasks.
Revised: We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisﬁed with high probability during training, derive provable convergence bounds for our approach, which is no worse asymptotically than standard RL, and demonstrate the efﬁcacy of the proposed approach on a suite of challenging navigation manipulation, and locomotion tasks.

Original: Empirically, we show that the proposed approach can achieve competitive task performance while incurring signiﬁcantly lower catastrophic failure rates during training than prior methods.
Revised: Empirically, we show that the proposed approach can achieve competitive task performance while incurring signiﬁcantly lower catastrophic failure rates during training than prior methods.

Original: Videos are at this url https://sites.google.com/view/safe-exploration/
Revised: Videos are at this url https: //sites google com/ view/ conservative -safety-critics/

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning (RL) is a powerful framework for learning-based control because it can enable agents to learn to make decisions automatically through trial and error.
Revised: Reinforcement learning (RL) is a powerful framework for learning-based control because it can enable agents to learn to make decisions automatically through trial and error.

Original: However, in the real world, the cost of those trials – and those errors – can be quite high: a quadruped learning to run as fast as possible, might fall down and crash, and then be unable to attempt further trials due to extensive physical damage.
Revised: However, in the real world, the cost of those trials – and those errors – can be quite high: a quadruped learning to run as fast as possible, might fall down and crash, and then be unable to attempt further trials due to extensive physical damage.

Original: However, learning complex skills without any failures at all is likely impossible.
Revised: However, learning complex skills without any failures at all is likely impossible.

Original: Even humans and animals regularly experience failure, but quickly learn from their mistakes and behave cautiously in risky situations.
Revised: Even humans and animals regularly experience failure, but quickly learn from their mistakes and behave cautiously in risky situations.

Original: In this paper, our goal is to develop safe exploration methods for RL that similarly exhibit conservative behavior, erring on the side of caution in particularly dangerous settings, and limiting the number of catastrophic failures.
Revised: In this paper, our goal is to develop safe exploration methods for RL that similarly exhibit conservative behavior, erring on the side of caution in particularly dangerous settings, and limiting the number of catastrophic failures.

Original: A number of previous approaches have tackled this problem of safe exploration, often by formulating the problem as a constrained Markov decision process (CMDP)
Revised: A number of previous approaches have tackled this problem of safe exploration, often by formulating the problem as a constrained Markov decision process (CMDP)

Original: (Garcıa & Fern ´ andez, 2015; Altman, 1999).
Revised: (Garcıa & Fern´andez, 2015; Altman, 1999).

Original: However, most of these approaches require additional assumptions, like assuming access to a function that can be queried to check if a state is safe (Thananjeyan et al., 2020), assuming access to a default safe controller (Koller et al., 2018; Berkenkamp et al., 2017), assuming knowledge of all the unsafe states (Fisac et al., 2019), and only obtaining safe policies after training converges, while being unsafe during the training process (Tessler et al., 2018;
Revised: However, most of these approaches require additional assumptions, like assuming access to a function that can be queried to check if a state is safe (Thananjeyan et al., 2020), assuming access to a default safe controller (Koller et al., 2018; Berkenkamp et al., 2017), assuming knowledge of all the unsafe states (Fisac et al., 2019), and only obtaining safe policies after training converges, while being unsafe during the training process (Tessler et al., 2018;

Original: Dalal et al., 2018).
Revised: Dalal et al., 2018).

Original: In this paper, we propose a general safe RL algorithm, with bounds on the probability of failures during training.
Revised: In this paper, we propose a general safe RL algorithm, with bounds on the probability of failures during training.

Original: Our method only assumes access to a sparse (e.g., binary) indicator for catastrophic failure , in the standard RL setting.
Revised: Our method only assumes access to a sparse (e.g., binary) indicator for catastrophic failure , in the standard RL setting.

Original: We train a conservative safety critic that overestimates the probability of catastrophic failure, building on tools in the recently proposed conservative Q-learning framework (Kumar et al., 2020) for ofﬂine RL.
Revised: We train a conservative safety critic that overestimates the probability of catastrophic failure, building on tools in the recently proposed conservative Q-learning framework (Kumar et al., 2020) for ofﬂine RL.

Original: In order to bound the likelihood of catastrophic failures at every iteration, we impose a KL-divergence constraint on successive policy updates so that the stationary distribution of states induced by the old and the new policies are not arbitrarily different.
Revised: In order to bound the likelihood of catastrophic failures at every iteration, we impose a KL-divergence constraint on successive policy updates so that the stationary distribution of states induced by the old and the new policies are not arbitrarily different.

Original: Based on the safety critic’s value, we consider a chance constraint denoting probability of failure, and optimize the policy through primal-dual gradient descent.
Revised: Based on the safety critic’s value, we consider a chance constraint denoting probability of failure, and optimize the policy through primal-dual gradient descent.

Original: Our key contributions in this paper are designing an algorithm that we refer to as Conservative Safety
Revised: Our key contributions in this paper are designing an algorithm that we refer to as Conservative Safety

Original: Critics (CSC), that learns a conservative estimate of how safe a state is, using this conservative estimate for safe-exploration and policy updates, and theoretically providing upper bounds on the probability of failures throughout training.
Revised: Critics (CSC), that learns a conservative estimate of how safe a state is, using this conservative estimate for safe-exploration and policy updates, and theoretically providing upper bounds on the probability of failures throughout training.

Original: Through empirical evaluation in ﬁve separate simulated robotic control domains spanning manipulation, navigation, and locomotion, we show that CSC is able to learn effective policies while reducing the rate of catastrophic failures by up to 50% over prior safe exploration methods.
Revised: Through empirical evaluation in ﬁve separate simulated robotic control domains spanning manipulation, navigation, and locomotion, we show that CSC is able to learn effective policies while reducing the rate of catastrophic failures by up to 50% over prior safe exploration methods.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We describe the problem setting of a constrained MDP (Altman, 1999) speciﬁc to our approach and the conservative Q learning (Kumar et al., 2020) framework that we build on in our algorithm.
Revised: We describe the problem setting of a constrained MDP (Altman, 1999) speciﬁc to our approach and the conservative Q learning (Kumar et al., 2020) framework that we build on in our algorithm.

Original: Constrained MDPs.
Revised: Constrained MDPs.

Revised: We take a constrained RL view of safety (Garcıa & Fern´andez, 2015; Achiam et al., 2017), and deﬁne safe exploration as the process of ensuring the constraints of the constrained

Original: A constrained MDP (CMDP)  is a tuple ( S , A , P, R, γ, µ, C ) , where S is the state space, A is the action space, P : S × A × S → [0 , 1] is a transition kernel, R : S ×
Revised: MDP (CMDP) are satisﬁed while exploring the environment to collect data samples. A CMDP is a tuple ( S , A , P, R, γ, µ, C ) , where S is the state space, A is the action space, P : S ×A×S → [0 , 1] is a transition kernel, R : S ×

Original: A → R is a task reward function, γ ∈ (0 , 1) is a discount factor, µ is a starting state distribution, and 
Revised: A → R is a task reward function, γ ∈ (0 , 1) is a discount factor, µ is a starting state distribution, and C

Original: C ={ ( c i : S → { 0 , 1 } , χ i ∈ R ) | i ∈ Z } is a set of (safety) constraints that the agent must satisfy, with constraint functions
Revised: = ( c i : S → { 0 , 1 } , χ i ∈ R ) | i ∈ Z } is a set of (safety) constraints that the agent must satisfy, with constraint functions

Original: c i taking values either 0 ( alive ) or 1 ( failure ) and limits χ i deﬁning the maximal allowable amount of non-satisfaction, in terms of expected probability of failure.
Revised: c i taking values either 0 ( alive ) or 1 ( failure ) and limits χ i deﬁning the maximal allowable amount of non-satisfaction, in terms of expected probability of failure.

Original: A stochastic policy π : S → P ( A ) is a mapping from states to action distributions, and the set of all stationary policies is denoted by Π .
Revised: A stochastic policy π : S → P ( A ) is a mapping from states to action distributions, and the set of all stationary policies is denoted by Π .

Original: Without loss of generality, we can consider a single constraint, where C denotes the constraint satisfaction function C : S → { 0 , 1 } , ( C ≡ 1 { failure } ) similar to the task reward function, and an upper limit χ .
Revised: Without loss of generality, we can consider a single constraint, where C denotes the constraint satisfaction function C : S → { 0 , 1 } , ( C ≡ 1 { failure } ) similar to the task reward function, and an upper limit χ .

Revised: Note that since we assume only a sparse binary indicator of failure from the environment C ( s ) , in purely online training, the agent must fail a few times during training, and hence 0 failures is impossible.

Revised: However, we will discuss how we can minimize the number of failures to a small rate, for constraint satisfaction.

Original: We deﬁne discounted state distribution of a policy π as d π ( s ) =
Revised: We deﬁne discounted state distribution of a policy π as d π ( s ) =

Original: (1 − γ ) (cid:80) ∞
Revised: (1 − γ ) (cid:80) ∞

Original: t =0 γ t P ( s t = s | π ) , the state value function as V πR ( s ) =
Revised: t =0 γ t P ( s t = s | π ) , the state value function as V πR ( s ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ R ( τ ) | s 

Revised: [ R ( τ ) | s

Original: = s ] , the state-action value function as
Revised: = s ] , the state-action value function as

Original: Q πR ( s, a ) =
Revised: Q πR ( s, a ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ R ( τ ) | s 

Revised: [ R ( τ ) | s

Original: = s, a 0 = a ] , and the advantage function as A πR ( s, a ) =
Revised: = s, a 0 = a ] , and the advantage function as A πR ( s, a ) =

Original: Q πR ( s, a ) −
Revised: Q πR ( s, a ) −

Original: V πR ( s ) .
Revised: V πR ( s ) .

Original: We deﬁne similar quantities for the constraint function, as V C , Q C , and A C .
Revised: We deﬁne similar quantities for the constraint function, as V C , Q C , and A C .

Original: So, we have V πR ( µ ) =
Revised: So, we have V πR ( µ ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ (cid:80) ∞ t =0 R ( s t , a t )] and V πC ( µ ) denoting the average episodic failures, which can also be interpreted as expected probability of failure since V πC ( µ ) =
Revised: [ (cid:80) ∞ t =0 R ( s t , a t )] and V πC ( µ ) denoting the average episodic failures, which can also be interpreted as expected probability of failure since V πC ( µ ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ (cid:80) ∞ t =0 C ( s t )]
Revised: [ (cid:80) ∞ t =0 C ( s t )]

Original: =
Revised: =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ 1 { failure } ]
Revised: [ 1 { failure } ]

Original: = P ( failure | µ ) .
Revised: = P ( failure | µ ) .

Original: For policy parameterized as π φ , we denote d π ( s ) as ρ φ ( s ) .
Revised: For policy parameterized as π φ , we denote d π ( s ) as ρ φ ( s ) .

Original: We take a constrained RL view of safety (Garcıa & Fern´andez, 2015; Achiam et al., 2017), and deﬁne safe exploration in the above CMDP as the process of ensuring the constraints of the CMDP are satisﬁed while exploring the environment to collect data samples. Note that since we assume only a sparse binary indicator of failure from the environment C ( s ) ,  in purely online training, the agent must fail a few times during training, and hence 0 failures isimpossible. However, we will discuss how we can minimize the number of failures to a small rate, for constraint satisfaction.
Revised: Note that although C : S → { 0 , 1 } takes on binary values in our setting, the function V πC ( µ ) is a continuous function of the policy π .

Original: Conservative Q Learning.
Revised: Conservative Q Learning.

Original: CQL (Kumar et al., 2020) is a method for ofﬂine/batch RL (Lange et al., 2012; Levine et al., 2020) that aims to learn a Q -function such that the expected value of a policy under the learned Q function lower bounds its true value, preventing over-estimation due to out-ofdistribution actions as a result.
Revised: CQL (Kumar et al., 2020) is a method for ofﬂine/batch RL (Lange et al., 2012; Levine et al., 2020) that aims to learn a Q -function such that the expected value of a policy under the learned Q function lower bounds its true value, preventing over-estimation due to out-ofdistribution actions as a result.

Original: In addition to training Q-functions via standard Bellman error, CQL minimizes the expected Q -values under a particular distribution of actions, µ ( a | s ) , and maximizes the expected Q-value under the on-policy distribution, π ( a | s ) .
Revised: In addition to training Q-functions via standard Bellman error, CQL minimizes the expected Q -values under a particular distribution of actions, µ ( a | s ) , and maximizes the expected Q-value under the on-policy distribution, π ( a | s ) .

Original: CQL in and of itself might lead to unsafe exploration, whereas we will show in Section 3, how the theoretical tool introduced in CQL can be used to devise a safe RL algorithm.
Revised: CQL in and of itself might lead to unsafe exploration, whereas we will show in Section 3, how the theoretical tool introduced in CQL can be used to devise a safe RL algorithm.

Original:  T HE C ONSERVATIVE S AFE -E XPLORATION F RAMEWORK
Revised:  T HE C ONSERVATIVE S AFE -E XPLORATION F RAMEWORK

Original: In this section we describe our safe exploration framework.
Revised: In this section we describe our safe exploration framework.

Original: The safety constraint C ( s ) deﬁned in
Revised: The safety constraint C ( s ) deﬁned in

Original: Section 2 is an indicator of catastrophic failure: C ( s )
Revised: Section 2 is an indicator of catastrophic failure: C ( s )

Original: = 1 when a state s is unsafe and C ( s )
Revised: = 1 when a state s is unsafe and C ( s )

Original: = when it is not, and we ideally desire C ( s ) =
Revised: =when it is not, and we ideally desire C ( s ) =

Original: 0 ∀ s ∈ S that the agent visits.
Revised: 0 ∀ s ∈ S that the agent visits.

Original: Since we do not make any assumptions in the problem structure for RL, we cannot guarantee this, but can at best reduce the probability of failure in every episode.
Revised: Since we do not make any assumptions in the problem structure for RL (for example a known dynamics model), we cannot guarantee this, but can at best reduce the probability of failure in every episode.

Original: So, we formulate the constraint as V πC ( µ ) =
Revised: So, we formulate the constraint as V πC ( µ ) =

Original: E τ ∼ π
Revised: E τ ∼ π

Original: [ (cid:80) ∞ t =0 C ( s t )] ≤ χ , where χ ∈ [0 , 1) denotes probability of failure .
Revised: [ (cid:80) ∞ t =0 C ( s t )] ≤ χ , where χ ∈ [0 , 1) denotes probability of failure .

Original: Our approach is motivated by the insight that by being “conservative” with respect to how safe a state is, and hence by overestimating this probability of failure, we can effectively ensure constrained exploration.
Revised: Our approach is motivated by the insight that by being “conservative” with respect to how safe a state is, and hence by over-estimating this probability of failure, we can effectively ensure constrained exploration.

Original: Figure 1 provides an overview of the approach.
Revised: Figure 1 provides an overview of the approach.

Original: The key idea of our algorithm is to train a conservative safety critic denoted as Q C ( s, a ) , that overestimates how unsafe a particular state is and modiﬁes the exploration strategy to appropriately account for this safety
Revised: The key idea of our algorithm is to train a conservative safety critic denoted as Q C ( s, a ) , that overestimates how unsafe a particular state is and modiﬁes the exploration strategy to appropriately account for this safety

Original: under-estimate (by overestimating the probability of failure).
Revised: under-estimate (by overestimating the probability of failure).

Original: During policy evaluation in the environment, we use the safety critic Q C ( s, a ) to reduce the chance of catastrophic failures by checking whether taking action a in state s has Q C ( s, a ) less than a threshold (cid:15) .
Revised: During policy evaluation in the environment, we use the safety critic Q C ( s, a ) to reduce the chance of catastrophic failures by checking whether taking action a in state s has Q C ( s, a ) less than a threshold (cid:15) .

Original: If not, we re-sample a from the current policy π ( a | s ) .
Revised: If not, we re-sample a from the current policy π ( a | s ) .

Original: We now discuss our algorithm more formally.
Revised: We now discuss our algorithm more formally.

Original: We start by discussing the procedure for learning the safety critic Q C , then discuss how we incorporate this in the policy gradient updates, and ﬁnally discuss how we perform safe exploration (Garcıa & Fern´andez, 2015) during policy execution in the environment.
Revised: We start by discussing the procedure for learning the safety critic Q C , then discuss how we incorporate this in the policy gradient updates, and ﬁnally discuss how we perform safe exploration (Garcıa & Fern ´ andez, 2015) during policy execution in the environment.

Original: Overall objective.
Revised: Overall objective.

Original: Our objective is to learn an optimal policy π ∗ that maximizes task rewards, while respecting the constraint on expected probability of failures.
Revised: Our objective is to learn an optimal policy π ∗ that maximizes task rewards, while respecting the constraint on expected probability of failures.

Original: [Equation]
Revised: [Equation]

Original: We note that although C : S → { 0 , 1 } takes on binary values in our setting (so that we have a general setting that only identiﬁes if a failure has occurred, without assuming access to a shaped continuous function that indicates impending failure), the function V πC ( µ ) is indeed a continuous function of the policy π . In other words, this is equivalent to the value function corresponding to a sparse reward function from the environment, but this doesn’t mean that the value function is discontinuous in π . In our derivation, we use the derivative of V πC ( µ ) with respect to π via Taylor’s theorem, and do not use the derivative of C ( s ) .
Revised: [Equation]

Original: Learning the safety critic.
Revised: Learning the safety critic.

Original: The safety critic Q C is used to obtain an estimate of how unsafe a particular state is, by providing an estimate of probability of failure , that will be used to guide exploration.
Revised: The safety critic Q C is used to obtain an estimate of how unsafe a particular state is, by providing an estimate of probability of failure , that will be used to guide exploration.

Original: We desire the estimates to be “conservative”, in the sense that the probability of failure should be an over-estimate of the actual probability so that the agent can err on the side of caution while exploring.
Revised: We desire the estimates to be “conservative”, in the sense that the probability of failure should be an over-estimate of the actual probability so that the agent can err on the side of caution while exploring.

Original: To train such a critic Q C , we incorporate tools from CQL to estimate Q C through updates similar to those obtained by reversing the sign of α in Equation 2 of CQL( H ) (Kumar et al., 2020).
Revised: To train such a critic Q C , we incorporate tools from CQL to estimate Q C through updates similar to those obtained by reversing the sign of α in Equation 2 of CQL( H ) (Kumar et al., 2020).

Original: This gives us an upper bound on Q C instead of a lower bound, as ensured by CQL.
Revised: This gives us an upper bound on Q C instead of a lower bound, as ensured by CQL.

Original: We denote the over-estimated advantage corresponding to this safety critic as ˆ A C .
Revised: We denote the over-estimated advantage corresponding to this safety critic as ˆ A C .

Original: Formally the safety critic is trained via the following objective, where the objective inside arg min is called CQL ( ζ ) , ζ parameterizes Q C , and k denotes the k th update iteration.
Revised: Formally the safety critic is trained via the following objective, where the objective inside arg min is called CQL ( ζ ) , ζ parameterizes Q C , and k denotes the k th update iteration.

Original: [Equation]
Revised: [Equation]

Original: Here, ˆ B π φ is the empirical Bellman operator discussed in section 3.1 and equation 2 of Kumar et al.
Revised: Here, ˆ B π φ is the empirical Bellman operator discussed in section 3.1 and equation 2 of Kumar et al.

Original: α is a weight that varies the importance of the ﬁrst term in equation 2, and controls the magnitude of value over-estimation, as we now highlight in red above.
Revised: α is a weight that varies the importance of the ﬁrst term in equation 2, and controls the magnitude of value over-estimation, as we now highlight in red above.

Original: For states sampled from the replay buffer D env , the ﬁrst term seeks to maximize the expectation of Q C over actions sampled from the current policy, while the second term seeks to minimize the expectation of Q C over actions sampled from the replay buffer.
Revised: For states sampled from the replay buffer D env , the ﬁrst term seeks to maximize the expectation of Q C over actions sampled from the current policy, while the second term seeks to minimize the expectation of Q C over actions sampled from the replay buffer.

Original: D env can include off-policy data, and also ofﬂine-data (if available).
Revised: D env can include off-policy data, and also ofﬂine-data (if available).

Original: We interleave the gradient descent updates for training of Q C , with gradient ascent updates for policy π φ and gradient descent updates for Lagrange multiplier λ , which we describe next.
Revised: We interleave the gradient descent updates for training of Q C , with gradient ascent updates for policy π φ and gradient descent updates for Lagrange multiplier λ , which we describe next.

Original: Policy learning.
Revised: Policy learning.

Original: Since we want to learn policies that obey the constraint we set in terms of the safety critic, we  solve the objective in equation 1 via a surrogate policy improvement problem:
Revised: Since we want to learn policies that obey the constraint we set in terms of the safety critic, we can solve the objective in equation 1 via: [Equation] We can construct a Lagrangian and solve the policy optimization problem through primal dual gradient descent

Original: [Equation]
Revised: [Equation]

Original: and
Revised: We can apply vanilla policy gradients or some actor-critic style Q-function approximator for optimization. Here, Q C is the safety critic trained through CQL as described in equation 2. We defer speciﬁc implementation details for policy learning to the ﬁnal paragraph of this section. [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: V
Revised: [Table] [Table] [Table] [Table]

Original: [Equation]
Revised: [Table]

Original: Here, we have introduced a D KL constraint to ensure successive policies are close in order to help obtain bounds on the expected failures of the new policy in terms of the expected failures of the old policy in Section 4. We replace the D KL ( π φ old ( ·| s ) || π φ ( ·| s )) term by its second order Taylor expansion (expressed in terms of the Fisher Information Matrix) and enforce the resulting constraint exactly (Schulman et al., 2015a).
Revised: [Table]

Original: For the constraint on V π φ C ( µ ) , we follow the primal-dual optimization method of Lagrange multipliers without making any simpliﬁcations of the constraint term V π φ C ( µ ) . This, as per equation 23 (Appendix) can be rewritten as
Revised: (cid:46)

Original: [Equation]
Revised: [Table]

Original: We replace the true A C by the learned over-estimated ˆ A C , andconsider the Lagrangian dual of this Sample a ∼ π φ old ( s ) . Execute a iff Q C ( s, a ) ≤ (cid:15) . Else, resample a . 7:
Revised: and

Original: Obtain next state s (cid:48) , r = R ( s, a ) , c = C ( s (cid:48) ) .
Revised: [Table]

Original: D env ← D env ∪ { ( s, a, s (cid:48) , r, c ) } (cid:46) If available, D env can be seeded with off-policy/ofﬂine data 9: end for 10: Store the average episodic failures ˆ V π φold C ( µ ) ← (cid:80) Me =1 ˆ V eC 11: for step t in { 1, ..., N } do (cid:46) Policy and Q function updates using D env 12:
Revised: function updates using D env 12:

Original: Gradient ascent on φ and (Optionally) add Entropy regularization (equation 7) 13:
Revised: Gradient ascent on φ and (Optionally) add [Table] regularization ( [Table]

Original: Gradient updates for the Q-function ζ := ζ − ηQ ∇ ζ CQL ( ζ ) 14:
Revised: Gradient updates for the Q [Table] function ζ := ζ − η

Original: Gradient descent step on Lagrange multiplier λ (equation 9) 15: end for 16: φ old ← φ 17: end for constrained problem, which we can solve by alternating gradient descent as shown below.
Revised: [Table]

Original: [Equation] .t. [Equation] We replace V π φold C by its sample estimate ˆ V π φold C and denote χ − V π φold C as χ (cid:48) .
Revised: ∇

Original: Note that χ (cid:48) is independent of parameter φ that is being optimized over.

Original: For notational convenience let λ (cid:48) denote the fraction λ 1 − γ , and deﬁne ˆ A

Original: [Equation]
Revised: [Table]

Original: = [Equation] C . In addition, we can approximate D KL in terms of the Fisher Information Matrix F , where, F can be estimated with samples as
Revised: CQL

Original: [Equation]
Revised: [Table]

Original: Following the steps in the Appendix A.2, we can write the gradient ascent step for φ as

Original: [Equation] Here β j is the backtracking coefﬁcient and we perform backtracking line search with exponential decay. ∇ φ old ˜ J ( φ old ) is calculated as, [Equation]
Revised: ζ ) 14:

Original: For gradient descent with respect to the Lagrange multiplier λ we have, [Equation] η λ is the learning rate. Detailed derivations of the gradient updates are in Appendix A.2.
Revised: Gradient descent step on Lagrange multiplier λ (Appendix A.2) 15: end for 16: φ old ← φ 17: end for

Original: Executing rollouts (i.e., safe exploration).
Revised: Executing rollouts (i.e., safe exploration).

Original: Since we are interested in minimizing the number of constraint violations while exploring the environment, we do not simply execute the learned policy iterate in the environment for active data collection.
Revised: Since we are interested in minimizing the number of constraint violations while exploring the environment, we do not simply execute the learned policy iterate in the environment for active data collection.

Original: Rather, we query the safety critic Q C to obtain an estimate of how unsafe an action is and choose an action that is safe via rejection sampling.
Revised: Rather, we query the safety critic Q C to obtain an estimate of how unsafe an action is and choose an action that is safe via rejection sampling.

Original: Formally, we sample an action a ∼ π φ old ( s ) , and check if Q C ( s, a ) ≤ (cid:15) .
Revised: Formally, we sample an action a ∼ π φ old ( s ) , and check if Q C ( s, a ) ≤ (cid:15) .

Original: We keep re-sampling actions π φ old ( s ) until this condition is met, and once met, we execute that action in the environment.
Revised: We keep re-sampling actions π φ old ( s ) until this condition is met, and once met, we execute that action in the environment.

Original: In practice, we execute this loop for 100 iterations, and choose the action a among all actions in state s for which Q C ( s, a ) ≤ (cid:15) and the value of Q C ( s, a ) is minimum.
Revised: In practice, we execute this loop for 100 iterations, and choose the action a among all actions in state s for which Q C ( s, a ) ≤ (cid:15) and the value of Q C ( s, a ) is minimum.

Original: If no such action a is found that maintains Q C ( s, a ) ≤ (cid:15) , we just choose a for which Q C ( s, a ) is minimum (although above the threshold).
Revised: If no such action a is found that maintains Q C ( s, a ) ≤ (cid:15) , we just choose a for which Q C ( s, a ) is minimum (although above the threshold).

Original: Here, (cid:15) is a threshold that varies across iterations and is deﬁned as (cid:15) =
Revised: Here, (cid:15) is a threshold that varies across iterations and is deﬁned as (cid:15) =

Original: (1 − γ )( χ − ˆ V π φold C ( µ )) where, ˆ V π φold C ( µ ) is the average episodic failures in the previous epoch, denoting a sample estimate of the true V
Revised: (1 − γ )( χ − ˆ V π φold C ( µ )) where, ˆ V π φold C ( µ ) is the average episodic failures in the previous epoch, denoting a sample estimate of the true V

Original: π φold C ( µ ) .
Revised: π φold C ( µ ) .

Original: This value of (cid:15) is theoretically obtained such that Lemma 1 holds.
Revised: This value of (cid:15) is theoretically obtained such that Lemma 1 holds.

Original: In the replay buffer D env , we store tuples of the form ( s, a, s (cid:48) , r, c ) , where s is the previous state, a is the action executed, s (cid:48) is the next state, r is the task reward from the environment, and c = C ( s (cid:48) ) , the constraint value.
Revised: In the replay buffer D env , we store tuples of the form ( s, a, s (cid:48) , r, c ) , where s is the previous state, a is the action executed, s (cid:48) is the next state, r is the task reward from the environment, and c = C ( s (cid:48) ) , the constraint value.

Original: In our setting, c is binary, with 0 denoting a live agent and 1 denoting failure .
Revised: In our setting, c is binary, with 0 denoting a live agent and 1 denoting failure .

Original: Overall algorithm.
Revised: Overall algorithm.

Original: Our overall algorithm, shown in Algorithm 1, executes policy rollouts in the environment by respecting the constraint Q C ( s, a ) ≤ (cid:15) , stores the observed data tuples in the replay buffer D env , and uses the collected tuples to train a safety value function Q C using equation 2, update the policyusing equation 7, andupdate the dual variable λ using equation 9.
Revised: Our overall algorithm, shown in Algorithm 1, executes policy rollouts in the environment by respecting the constraint Q C ( s, a ) ≤ (cid:15) , stores the observed data tuples in the replay buffer D env , and uses the collected tuples to train a safety value function Q C using equation 2, update the policy and the dual variable λ following the optimization objective in equation 6. Implementation details.

Revised: Here, we discuss the speciﬁcs of the implementation for policy optimization.

Revised: We consider the surrogate policy improvement problem Sutton (2020):

Original: T HEORETICAL A NALYSIS
Revised: [Equation] and [Equation]

Revised: Here, we have introduced a D KL constraint to ensure successive policies are close in order to help obtain bounds on the expected failures of the new policy in terms of the expected failures of the old policy in Section 4.

Revised: We replace the D KL ( π φ old ( ·| s ) ||

Revised: π φ ( ·| s )) term by its second order Taylor expansion (expressed in terms of the Fisher Information Matrix F ) and enforce the resulting constraint exactly (Schulman et al., 2015a).

Revised: Following equation 22 (Appendix A.2) we have,

Revised: [Equation]

Revised: We replace the true A C by the learned over-estimated ˆ A C , and consider the Lagrangian dual of this constrained problem, which we can solve by alternating gradient descent as shown below.

Revised: [Equation]

Revised: Note that although we use FIM for the updates, we can also apply vanilla policy gradients or some actor-critic style Q-function approximator to optimize equation 6.

Revised: Detailed derivations of the gradient updates are in Appendix A.2.

Original: In this section, we aim to theoretically analyze our approach, showing that the expected probability of failures is bounded after each policy update throughout the learning process, while ensuring that the convergence rate to the optimal solution is only mildly bottlenecked by the additional safety constraint.
Revised: In this section, we aim to theoretically analyze our approach, showing that the expected probability of failures is bounded after each policy update throughout the learning process, while ensuring that the convergence rate to the optimal solution is only mildly bottlenecked by the additional safety constraint.

Original: Our main result, stated in Theorem 1, bounds the expected probability of failure of the policy that results from Equation 4.
Revised: Our main result, stated in Theorem 1, bounds the expected probability of failure of the policy that results from Equation 5.

Original: To prove this, we ﬁrst state a Lemma that shows that the constraints in Equation 4 are satisﬁed with high probability during the policy updates.
Revised: To prove this, we ﬁrst state a Lemma that shows that the constraints in Equation 5 are satisﬁed with high probability during the policy updates.

Original: Detailed proofs of all the Lemmas and Theorems are in Appendix A.1.
Revised: Detailed proofs of all the Lemmas and Theorems are in Appendix A.1.

Original: Notation.
Revised: Notation.

Original: Let (cid:15) C = max s | E a ∼ π φnew A C ( s, a ) | and ∆ be the amount of overestimation in the expected advantage value generated from the safety critic, E s ∼ ρ φold (cid:48) ,a ∼ π φold [ ˆ A C ( s, a )] as per equation 2, such that ∆ =
Revised: Let (cid:15) C = max s | E a ∼ π φnew A C ( s, a ) | and ∆ be the amount of overestimation in the expected advantage value generated from the safety critic, E s ∼ ρ φold (cid:48) ,a ∼ π φold [ ˆ A C ( s, a )] as per equation 2, such that ∆ =

Original: E s ∼ ρ φold (cid:48) ,a ∼ π φold [ ˆ A C ( s, a ) − A C ( s, a )] .
Revised: E s ∼ ρ φold (cid:48) ,a ∼ π φold [ ˆ A C ( s, a ) − A C ( s, a )] .

Original: Let ζ denote the sampling error in the estimation of V π φold C ( µ ) by its sample estimate ˆ V π φold C ( µ ) and N be the number of samples used in the estimation of V C .
Revised: Let ζ denote the sampling error in the estimation of V π φold C ( µ ) by its sample estimate ˆ V π φold C [Equation] and N be the number of samples used in the estimation of V C .

Original: Let Reg C ( T ) be the total cumulative failures incurred by running Algorithm 1 until T samples are collected from the environment.
Revised: Let Reg C ( T ) be the total cumulative failures incurred by running Algorithm 1 until T samples are collected from the environment.

Original: We ﬁrst show that when using Algorithm 1, we can upper bound the expectation probability of failure for each policy iterate π φ old .
Revised: We ﬁrst show that when using Algorithm 1, we can upper bound the expectation probability of failure for each policy iterate π φ old .

Original: Lemma 1.
Revised: Lemma 1.

Original: If we follow Algorithm 1, during policy updates via Equation 4, the following is satisﬁed with high probability ≥1 − ω [Equation] ∆
Revised: If we follow Algorithm 1, during policy updates via Equation 5, the following is satisﬁed with high probability ≥

Original: [Equation]
Revised: [Equation]

Original: Here, ζ captures sampling error in the estimation of V π φold C ( µ ) and we have ζ ≤ C
Revised: Here, ζ captures sampling error in the estimation of V π φold C ( µ ) and we have ζ ≤ C

Original: [Equation]
Revised: [Equation]

Original: | N | , where C (cid:48) is a constant independent of ω obtained from union bounds and concentration inequalities (Kumar et al., 2020) and N is the number of samples used in the estimation of V C .
Revised: | N | , where C (cid:48) is a constant independent of ω obtained from union bounds and concentration inequalities (Kumar et al., 2020) and N is the number of samples used in the estimation of V C .

Original: This lemma intuitively implies that the constraint on the safety critic in equation 4 is satisﬁed with a high probability, when we note that the RHS can be made small as N becomes large.
Revised: This lemma intuitively implies that the constraint on the safety critic in equation 5 is satisﬁed with a high probability, when we note that the RHS can be made small as N becomes large.

Original: Lemma 1 had a bound in terms of V π φold C ( µ ) for the old policy
Revised: Lemma 1 had a bound in terms of V π φold C ( µ ) for the old policy

Original: π φ old  .
Revised: π φ old , but not for the updated policy π φ new .

Original: We now show that the expected probability of failure for the policy π φ new resulting from solving equation 4, V π φnew C ( µ ) is bounded with a high probability.
Revised: We now show that the expected probability of failure for the policy π φ new resulting from solving equation 5, V π φnew C ( µ ) is bounded with a high probability.

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider policy updates that solve the constrained optimization problem deﬁned in
Revised: Consider policy updates that solve the constrained optimization problem deﬁned in

Original: Equation 4.
Revised: Equation 5.

Original: With high probability ≥ 1 − ω , we have the following upper bound on expected probability of failure V π φnew C ( µ ) for π φ new during every policy update iteration:
Revised: With high probability ≥ 1 − ω , we have the following upper bound on expected probability of failure V π φnew C ( µ ) for π φ new during every policy update iteration:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Since (cid:15) C depends on the new policy π φ new , it can’t be calculated exactly prior to the update.

Original: As we cap Q C ( s, a ) to be ≤ 1 , therefore, the best bound we can construct for (cid:15) C is the trivial bound (cid:15) C ≤ 2 .

Original: Now, in order to have V π φnew C ( µ ) <

Original: χ , we require ∆ > 2 √ 2 δγ 1 − γ + ζ .

Original: To guarantee this, we can obtain a theoretically prescribed minimum value for α as shown in the proof in Appendix A.1.

Original: So far we have shown that, with high probability, we can satisfy the constraint in the objective during policy updates (Lemma 1) and obtain an upper bound on the expected probability of failure of the updated policy π φ new (Theorem 1).
Revised: So far we have shown that, with high probability, we can satisfy the constraint in the objective during policy updates (Lemma 1) and obtain an upper bound on the expected probability of failure of the updated policy π φ new (Theorem 1).

Original: The key insight from Theorem 1 is that if we execute policy π φ new in the environment, the probability of failing is upper-bounded by a small number depending on the speciﬁed safety threshold χ .
Revised: The key insight from Theorem 1 is that if we execute policy π φ new in the environment, the probability of failing is upper-bounded by a small number depending on the speciﬁed safety threshold χ .

Original: Since the probability of failure is bounded, if we execute π φ new for multiple episodes, the total number of failures is bounded as well.
Revised: Since the probability of failure is bounded, if we execute π φ new for multiple episodes, the total number of failures is bounded as well.

Original: We now talk of task performance  and show that incorporating and satisfying safety constraints during learning does not severely affect the convergence rate to the optimal solution for task performance.
Revised: We now bound the task performance in terms of policy return and show that incorporating and satisfying safety constraints during learning does not severely affect the convergence rate to the optimal solution for task performance.

Original: Theorem 2 builds upon and relies on the assumptions in (Agarwal et al., 2019) and extends it to our constrained policy updates in equation 4.
Revised: Theorem 2 builds upon and relies on the assumptions in (Agarwal et al., 2019) and extends it to our constrained policy updates in equation 5.

Original: Theorem 2 (Convergence rate for policy gradient updates with the safety constraint) .
Revised: Theorem 2 (Convergence rate for policy gradient updates with the safety constraint) .

Original: If we run the policy gradient updates through equation 4, for policy π φ , with µ as the starting state distribution, with φ (0) = 0 ,and learning rate η > 0 ,  then for all policy update iterations T > 0 we have, with probability ≥ 1 − ω ,
Revised: If we run the policy gradient updates through equation 5, for policy π φ , with µ as the starting state distribution, with φ (0) = 0 , learning rate η > 0 , and choose α as mentioned in the discussion of Theorem 1, then for all policy update iterations T > 0 we have, with probability ≥ 1 − ω ,

Original: [Equation]
Revised: [Equation]

Original: ηT Since the value of the dual variables λ strictly decreases during gradient descent updates (Algorithm 1), (cid:80) T − 1 t =

Revised: Since the value of the dual variables λ strictly decreases during gradient descent updates (Algorithm 1), (cid:80) T − 1 t =

Original: λ ( t ) is upper-bounded.
Revised: λ ( t ) is upper-bounded.

Original: In addition, if we choose α as mentioned in the discussion of

Original: Theorem 1 (equation 28), we have ∆ > 2 √ 2 δγ 1 − γ + ζ .

Original: Hence, with probability ≥ 1 − ω , we can ensure V [Equation] So, we see that the additional term proportional to K introduced in the convergence rate (compared to (Agarwal et al., 2019)) due to the safety constraint is upper bounded, and can be made small with a high probability by choosing α appropriately, even after accounting for sampling error.
Revised: So, we see that the additional term proportional to K introduced in the convergence rate (compared to (Agarwal et al., 2019)) due to the safety constraint is upper bounded, and can be made small with a high probability by choosing α appropriately.

Original: In addition, we note that the safety threshold χ helps tradeoff the convergence rate by modifying the magnitude of K (a low χ means a stricter safety threshold, and a higher value of K , implying a larger RHS and slower convergence).
Revised: In addition, we note that the safety threshold χ helps tradeoff the convergence rate by modifying the magnitude of K (a low χ means a stricter safety threshold, and a higher value of K , implying a larger RHS and slower convergence).

Original: We discuss some practical considerations of the theoretical results in Appendix A.4.
Revised: We discuss some practical considerations of the theoretical results in Appendix A.4.

Original: So far we have demonstrated that the resulting policy iterates from our algorithm all satisfy the desired safety constraint of the CMDP which allows for a maximum safety violation of χ for every intermediate policy.
Revised: So far we have demonstrated that the resulting policy iterates from our algorithm all satisfy the desired safety constraint of the CMDP which allows for a maximum safety violation of χ for every intermediate policy.

Original: While this result ensures that the probability of failures is bounded, it does not elaborate on the total failures incurred by the algorithm.
Revised: While this result ensures that the probability of failures is bounded, it does not elaborate on the total failures incurred by the algorithm.

Original: In our next result, we show that the cumulative failures until a certain number of samples T of the algorithm grows sublinearly when executing Algorithm 1, provided the safety threshold χ is set in the right way.
Revised: In our next result, we show that the cumulative failures until a certain number of samples T of the algorithm grows sublinearly when executing Algorithm 1, provided the safety threshold χ is set in the right way.

Original: Theorem 3.
Revised: Theorem 3.

Original: [Number of cumulative safety failures grows sublinearly] Let χ in Algorithm 1 be timedependent such that χ t =
Revised: [Number of cumulative safety failures grows sublinearly] Let χ in Algorithm 1 be timedependent such that χ t =

Original: O (1 / √ t ) .
Revised: O (1 / √ t ) .

Original: Then, the total number of cumulative safety violations until when T transition samples have been collected by Algorithm 1, Reg C ( T ) , scales sub-linearly with
Revised: Then, the total number of cumulative safety violations until when T transition samples have been collected by Algorithm 1, Reg C ( T ) , scales sub-linearly with

Original: T , i.e., Reg C ( T ) =
Revised: T , i.e., Reg C ( T ) =

Original: O ( (cid:112) |S||A| T ) 
Revised: O ( (cid:112) |S||A| T ) .

Original: A proof is provided in Appendix A.1.
Revised: A proof is provided in Appendix A.1.

Original: Theorem 3 is in many ways similar to a typical regret bound for exploration (Russo, 2019; Jaksch et al., 2010), though it measures the total number of safety violations.
Revised: Theorem 3 is in many ways similar to a typical regret bound for exploration (Russo, 2019; Jaksch et al., 2010), though it measures the total number of safety violations.

Original: This means that training with Algorithm 1 will converge to a “safe” policy that incurs no failures at a quick, O ( √ T ) rate.
Revised: This means that training with Algorithm 1 will converge to a “safe” policy that incurs no failures at a quick, O ( √ T ) rate.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Through experiments on continuous control environments of varying complexity, we aim to empirically evaluate the agreement between empirical performance and theoretical guidance by understanding the following questions:
Revised: Through experiments on continuous control environments of varying complexity, we aim to empirically evaluate the agreement between empirical performance and theoretical guidance by understanding the following questions:

Original: • How safe is CSC in terms of constraint satisfaction during training?
Revised: • How safe is CSC in terms of constraint satisfaction during training?

Original: • How does learning of safe policies trade-off with task performance during training?
Revised: • How does learning of safe policies trade-off with task performance during training?

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: Environments.
Revised: Environments.

Original: In each environment, shown in Figure 2, we deﬁne a task objective that the agent must achieve and a criteria for catastrophic failure .
Revised: In each environment, shown in Figure 2, we deﬁne a task objective that the agent must achieve and a criteria for catastrophic failure .

Original: The goal is to solve the task without dying.
Revised: The goal is to solve the task without dying.

Original: In point agent/car navigation avoiding traps , the agent must navigate a maze while avoiding traps.
Revised: In point agent/car navigation avoiding traps , the agent must navigate a maze while avoiding traps.

Original: The agent has a health counter that decreases every timestep that it spends within a trap.
Revised: The agent has a health counter that decreases every timestep that it spends within a trap.

Original: When the counter hits 0 , the agent gets trapped and dies .
Revised: When the counter hits 0 , the agent gets trapped and dies .

Original: In Panda push without toppling , a 7-DoF Franka
Revised: In Panda push without toppling , a 7-DoF Franka

Original: Emika Panda arm must push a vertically placed block across the table to a goal location without the block toppling over.
Revised: Emika Panda arm must push a vertically placed block across the table to a goal location without the block toppling over.

Original: Failure is deﬁned as when the block topples.
Revised: Failure is deﬁned as when the block topples.

Original: In Panda push within boundary , the Panda arm must be controlled to push a block across the table to a goal location without the block going outside a rectangular constraint region.
Revised: In Panda push within boundary , the Panda arm must be controlled to push a block across the table to a goal location without the block going outside a rectangular constraint region.

Original: Failure occurs when the block center of mass ( ( x, y ) position) move outside the constraint region.
Revised: Failure occurs when the block center of mass ( ( x, y ) position) move outside the constraint region.

Original: In Laikago walk without falling , an 18-DoF
Revised: In Laikago walk without falling , an 18-DoF

Original: Laikago quadruped robot must walk without falling.
Revised: Laikago quadruped robot must walk without falling.

Original: The agent is rewarded for walking as fast as possible (or trotting) and failure occurs when the robot falls.
Revised: The agent is rewarded for walking as fast as possible (or trotting) and failure occurs when the robot falls.

Original: Since quadruped walking is an extremely challenging task, for all the baselines, we initialize the agent’s policy with a controller that has been trained to keep the agent standing, while not in motion.
Revised: Since quadruped walking is an extremely challenging task, for all the baselines, we initialize the agent’s policy with a controller that has been trained to keep the agent standing, while not in motion.

Original: Baselines and comparisons.
Revised: Baselines and comparisons.

Original: We compare CSC to three prior methods: constrained policy optimization ( CPO ) (Achiam et al., 2017), a standard unconstrained RL method (Schulman et al., 2015a) which we call Base (comparison with SAC (Haarnoja et al., 2018) in Appendix Figure 7), an algorithm similar to Base, called BaseShaped that modiﬁes the reward R ( s, a ) as R ( s, a ) − PC ( s ) where P = 10 and C ( s ) is 1 when a failure occurs and is 0 otherwise.
Revised: We compare CSC to three prior methods: constrained policy optimization ( CPO ) (Achiam et al., 2017), a standard unconstrained RL method (Schulman et al., 2015a) which we call Base (comparison with SAC (Haarnoja et al., 2018) in Appendix Figure 7), an algorithm similar to Base, called BaseShaped that modiﬁes the reward R ( s, a ) as R ( s, a ) − PC ( s ) where P = 10 and C ( s ) is 1 when a failure occurs and is 0 otherwise.

Original: We also consider a method that extends Leave No Trace (Eysenbach et al., 2017) to our setting, which we refer to as Q ensembles .
Revised: We also consider a method that extends Leave No Trace (Eysenbach et al., 2017) to our setting, which we refer to as Q ensembles .

Original: This last comparison is the most similar to our approach, in that it also implements a safety critic (adapted from LNT’s backward critic), but instead of using our conservative updates, the safety critic uses an ensemble for epistemic uncertainty estimation, as proposed by Eysenbach et al.
Revised: This last comparison is the most similar to our approach, in that it also implements a safety critic (adapted from LNT’s backward critic), but instead of using our conservative updates, the safety critic uses an ensemble for epistemic uncertainty estimation, as proposed by Eysenbach et al.

Original: There are other safe RL approaches which we cannot compare against, as they make multiple additional assumptions, such as the availability of a function that can be queried to determine if a state is safe or not Thananjeyan et al.
Revised: There are other safe RL approaches which we cannot compare against, as they make multiple additional assumptions, such as the availability of a function that can be queried to determine if a state is safe or not Thananjeyan et al.

Original: (2020), availability of a default safe policy for the task Koller et al.
Revised: (2020), availability of a default safe policy for the task Koller et al.

Original: ; Berkenkamp et al.
Revised: ; Berkenkamp et al.

Original: (2017), and prior knowledge of the location of unsafe states (Fisac et al., 2019).
Revised: (2017), and prior knowledge of the location of unsafe states (Fisac et al., 2019).

Original: In addition to the baselines (Figure 3), we analyze variants of our algorithm with different safety thresholds through ablation studies (Figure 4).
Revised: In addition to the baselines (Figure 3), we analyze variants of our algorithm with different safety thresholds through ablation studies (Figure 4).

Original: We also analyze CSC and the baselines by seeding with a small amount of ofﬂine data in the Appendix A.10.
Revised: We also analyze CSC and the baselines by seeding with a small amount of ofﬂine data in the Appendix A.10.

Original: 5. E MPIRICAL R ESULTS
Revised: 5. E MPIRICAL R ESULTS

Original: Comparable or better performance with signiﬁcantly lower failures during training.
Revised: Comparable or better performance with signiﬁcantly lower failures during training.

Original: In Figure 3, we observe that CSC has signiﬁcantly lower average failures per episode, and hence lower cumulative failures during the entire training process.
Revised: In Figure 3, we observe that CSC has signiﬁcantly lower average failures per episode, and hence lower cumulative failures during the entire training process.

Original: Although the failures are signiﬁcantly lower for our method, task performance and convergence of average task rewards is comparable to or better than all prior methods, including the Base method, corresponding to an unconstrained RL algorithm.
Revised: Although the failures are signiﬁcantly lower for our method, task performance and convergence of average task rewards is comparable to or better than all prior methods, including the Base method, corresponding to an unconstrained RL algorithm.

Original: While the CPO and Q-ensembles baselines also achieve near 0 average failures eventually, we see that CSC achieves this very early on during training.
Revised: While the CPO and Q-ensembles baselines also achieve near 0 average failures eventually, we see that CSC achieves this very early on during training.

Original: CSC trades off performance with safety constraint satisfaction, based on the safety-threshold χ .
Revised: CSC trades off performance with safety constraint satisfaction, based on the safety-threshold χ .

Original: In Figure 4, we plot variants of our method with different safety constraint thresholds χ .
Revised: In Figure 4, we plot variants of our method with different safety constraint thresholds χ .

Original: Observe that: (a) when the threshold is set to a lower value (stricter constraint), the number of avg.
Revised: Observe that: (a) when the threshold is set to a lower value (stricter constraint), the number of avg.

Original: failures per episode decreases in all the environments, and (b) the convergence rate of the task reward is lower when the safety threshold is stricter.
Revised: failures per episode decreases in all the environments, and (b) the convergence rate of the task reward is lower when the safety threshold is stricter.

Original: These observations empirically complement our theoretical guarantees in Theorems 1 and 2.
Revised: These observations empirically complement our theoretical guarantees in Theorems 1 and 2.

Original: We note that there are quite a few failures even in the case where χ = 0 . 0 , which is to be expected in practice because in the initial stages of training there is high function approximation error in the learned critic Q C .
Revised: We note that there are quite a few failures even in the case where χ = 0 . 0 , which is to be expected in practice because in the initial stages of training there is high function approximation error in the learned critic Q C .

Original: However, we observe that the average episodic failures quickly drop below the speciﬁed threshold after about 500 episodes of training.
Revised: However, we observe that the average episodic failures quickly drop below the speciﬁed threshold after about 500 episodes of training.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We discuss prior safe RL and safe control methods under three subheadings
Revised: We discuss prior safe RL and safe control methods under three subheadings

Original: Assuming prior domain knowledge of the problem structure.
Revised: Assuming prior domain knowledge of the problem structure.

Original: Prior works have attempted to solve safe exploration in the presence of structural assumptions about the environment or safety structures.
Revised: Prior works have attempted to solve safe exploration in the presence of structural assumptions about the environment or safety structures.

Original: For example, Koller et al.
Revised: For example, Koller et al.

Original: ; Berkenkamp et al.
Revised: ; Berkenkamp et al.

Original: (2017) assume access to a safe set of environment states and a default safe policy, while in Fisac et al.
Revised: (2017) assume access to a safe set of environment states, and a default safe policy, while in Fisac et al.

Original: ; Dean et al.
Revised: ; Dean et al.

Original: (2019), knowledge of system dynamics is assumed and (Fisac et al., 2019) assume access to a distance metric on the state space.
Revised: (2019), knowledge of system dynamics is assumed and (Fisac et al., 2019) assume access to a distance metric on the state space.

Original: SAVED
Revised: SAVED

Original: (Thananjeyan et al., 2020) learns a kernel density estimate over unsafe states, and assumes access to a set of user demonstrations and a user speciﬁed function that can be queried to determine whether a state is safe or not.
Revised: (Thananjeyan et al., 2020) learns a kernel density estimate over unsafe states, and assumes access to a set of user demonstrations and a user speciﬁed function that can be queried to determine whether a state is safe or not.

Original: In contrast to these approaches, our method does not assume any prior knowledge from the user, or domain knowledge of the problem setting, except a binary signal from the environment indicating when a catastrophic failure has occurred.
Revised: In contrast to these approaches, our method does not assume any prior knowledge from the user, or domain knowledge of the problem setting, except a binary signal from the environment indicating when a catastrophic failure has occurred.

Original: Assuming a continuous safety cost function.
Revised: Assuming a continuous safety cost function.

Original: CPO (Achiam et al., 2017), and (Chow et al., 2019) assume a cost function can be queried from the environment at every time-step and the objective is to keep the cumulative costs within a certain limit.
Revised: CPO (Achiam et al., 2017), and (Chow et al., 2019) assume a cost function can be queried from the environment at every time-step and the objective is to keep the cumulative costs within a certain limit.

Original: This assumption limits the generality of the method in scenarios where only minimal feedback, such as binary reward feedback is provided (additional details in section A.3).
Revised: This assumption limits the generality of the method in scenarios where only minimal feedback, such as binary reward feedback is provided (additional details in section A.3).

Original: (Stooke et al., 2020) devise a general modiﬁcation to the Lagrangian by incorporating two additional terms in the optimization of the dual variable.
Revised: (Stooke et al., 2020) devise a general modiﬁcation to the Lagrangian by incorporating two additional terms in the optimization of the dual variable.

Original: SAMBA (Cowen-Rivers et al., 2020) has a learned GP dynamics model and a continuous constraint cost function that encodes safety.
Revised: SAMBA (Cowen-Rivers et al., 2020) has a learned GP dynamics model and a continuous constraint cost function that encodes safety.

Original: The objective is to minimize task cost function while maintaining the CVAR α of cumulative costs below a threshold.
Revised: The objective is to minimize task cost function while maintaining the CVAR α of cumulative costs below a threshold.

Original: In the work of Dalal et al.
Revised: In the work of Dalal et al.

Original: ; Paternain et al.
Revised: ; Paternain et al.

Original: (2019b;a); Grbic & Risi (2020), only the optimal policy is learned to be safe, and there are no safety constraint satisfactions during training.
Revised: (2019b;a); Grbic & Risi (2020), only the optimal policy is learned to be safe, and there are no safety constraint satisfactions during training.

Original: In contrast to these approaches, we assume only a binary signal from the environment indicating when a catastrophic failure has occurred.
Revised: In contrast to these approaches, we assume only a binary signal from the environment indicating when a catastrophic failure has occurred.

Original: Instead of minimizing expected costs, our constraint formulation directly seeks to constrain the expected probability of failure.
Revised: Instead of minimizing expected costs, our constraint formulation directly seeks to constrain the expected probability of failure.

Original: Safety through recoverability.
Revised: Safety through recoverability.

Original: Prior works have attempted to devise resetting mechanisms to recover the policy to a base conﬁguration from (near) a potentially unsafe state.
Revised: Prior works have attempted to devise resetting mechanisms to recover the policy to a base conﬁguration from (near) a potentially unsafe state.

Original: LNT (Eysenbach et al., 2017) trains both a forward policy for solving a task, and a reset goal-conditioned policy that kicks in when the agent is in an unsafe state and learns an ensemble of critics, which is substantially more complex than our approach of a learned safety critic, which can give rise to a simple but provable safe exploration algorithm.
Revised: LNT (Eysenbach et al., 2017) trains both a forward policy for solving a task, and a reset goal-conditioned policy that kicks in when the agent is in an unsafe state and learns an ensemble of critics, which is substantially more complex than our approach of a learned safety critic, which can give rise to a simple but provable safe exploration algorithm.

Revised: Concurrently to us, SQRL (Srinivasan et al., 2020) developed an approach also using safety critics such that during the pre-training phase, the agent explores both safe and unsafe states in the environment for training the critic.

Revised: results plots of cumulative failures are

Original: In control theory, a number of prior works have focused on Hamilton-Jacobi-Isaacs (HJI) reachability analysis (Bansal et al., 2017) for providing safety constraint satisfactions and obtaining control inputs for dynamical systems (Herbert et al., 2019;
Revised: In control theory, a number of prior works have focused on Hamilton-Jacobi-Isaacs (HJI) reachability analysis (Bansal et al., 2017) for providing safety constraint satisfactions and obtaining control inputs for dynamical systems (Herbert et al., 2019;

Original: Bajcsy et al., 2019; Leung et al., 2018).
Revised: Bajcsy et al., 2019; Leung et al., 2018).

Original: Our method does not require knowledge of the system dynamics or regularity conditions on the state-space, which are crucial for computing unsafe states using HJI reachability.
Revised: Our method does not require knowledge of the system dynamics or regularity conditions on the statespace, which are crucial for computing unsafe states using HJI reachability.

Original:  D ISCUSSION , L IMITATIONS , AND C ONCLUSION
Revised:  D ISCUSSION , L IMITATIONS , AND C ONCLUSION

Original: We introduced a safe exploration algorithm to learn a conservative safety critic that estimates the probability of failure for each candidate state-action tuple, and uses this to constrain policy evaluation and policy improvement.
Revised: We introduced a safe exploration algorithm to learn a conservative safety critic that estimates the probability of failure for each candidate state-action tuple and uses this to constrain policy evaluation and policy improvement.

Original: We provably demonstrated that the probability of failures is bounded throughout training and provided convergence results showing how ensuring safety does not severely bottleneck task performance.
Revised: We provably demonstrated that the probability of failures is bounded throughout training and provided convergence results showing how ensuring safety does not severely bottleneck task performance.

Original: We empirically validated our theoretical results and showed that we achieve high task performance while incurring low accidents during training.
Revised: We empirically validated our theoretical results and showed that we achieve high task performance while incurring low accidents during training.

Original: While our theoretical results demonstrated that the probability of failures is bounded with a high probability, one limitation is that we still observe non-zero failures empirically even when the threshold χ is set to 0 .
Revised: While our theoretical results demonstrated that the probability of failures is bounded with a high probability, one limitation is that we still observe non-zero failures empirically even when the threshold χ is set to 0 .

Original: This is primarily because of neural network function approximation error in the early stages of training the safety critic, which we cannot account for precisely in the theoretical results, and also due to the fact that we bound the probability of failures, which in practice means that the number of failures is also bounded, but non-zero.
Revised: This is primarily because of neural network function approximation error in the early stages of training the safety critic, which we cannot account for precisely in the theoretical results, and also due to the fact that we bound the probability of failures, which in practice means that the number of failures is also bounded, but non-zero.

Original: We also showed that if we set the constraint threshold in an appropriate time-varying manner, training with CSC incurs cumulative failures that scales at most sub-linearly with the number of transition samples in the environment.
Revised: We also showed that if we set the constraint threshold in an appropriate time-varying manner, training with CSC incurs cumulative failures that scales at most sub-linearly with the number of transition samples in the environment.

Original: Although our approach bounds the probability of failure and is general in the sense that it does not assume access any user-speciﬁed constraint function, in situations where the task is difﬁcult to solve, for example due to stability concerns of the agent, our approach will fail without additional assumptions.
Revised: Although our approach bounds the probability of failure and is general in the sense that it does not assume access any user-speciﬁed constraint function, in situations where the task is difﬁcult to solve, for example due to stability concerns of the agent, our approach will fail without additional assumptions.

Original: In such situations, some interesting future work directions would be to develop a curriculum of tasks to start with simple tasks where safety is easier to achieve, and gradually move towards more difﬁcult tasks, such that the learned knowledge from previous tasks is not forgotten
Revised: In such situations, some interesting future work directions would be to develop a curriculum of tasks to start with simple tasks where safety is easier to achieve, and gradually move towards more difﬁcult tasks, such that the learned knowledge from previous tasks is not forgotten. A CKNOWLEDGEMENT

Revised: We thank Vector Institute, Toronto and the Department of Computer Science, University of Toronto for compute support.

Revised: We thank Glen Berseth and Kevin Xie for helpful initial discussions about the project, Alexandra Volokhova, Arthur Allshire, Mayank Mittal, Samarth Sinha, and Irene Zhang for feedback on the paper, and other members of the UofT CS Robotics Group for insightful discussions during internal presentations and reading group sessions.

Revised: Finally, we are grateful to the anonymous

Revised: ICLR 2021 reviewers for their feedback in helping improve the paper.

Original:  A. P ROOFS OF ALL THEOREMS AND LEMMAS
Revised: A A PPENDIX A. P ROOFS OF ALL THEOREMS AND LEMMAS

Original: Note.
Revised: Note.

Original: During policy updates via Equation 4, the D KL constraint is satisﬁed with high probability if we follow Algorithm 1.
Revised: During policy updates via Equation 5, the D KL constraint is satisﬁed with high probability if we follow Algorithm 1.

Revised: Following the steps in the Appendix A.2, we can write the gradient ascent step for φ as

Original: This follows from the update equation 7 as  we incorporate backtracking line search to ensure that the D KL constraint is satisﬁed exactly.
Revised: [Equation] F can be estimated with samples as [Equation] Here β j is the backtracking coefﬁcient and we perform backtracking line search with exponential decay.

Original: Let us revisit the update equation
Revised: ∇ φ old ˜ J ( φ old ) is calculated as,

Original: [Equation]
Revised: [Equation]

Original: After every update, we check if ¯ D KL ( φ || φ old ) ≤ δ , and if not we decay β j = β j (1 − β j ) j , set j ← j + 1 and repeat for L steps until ¯ D KL ≤ δ is satisﬁed.
Revised: After every update, we check if ¯ D KL ( φ || φ old ) ≤ δ , and if not we decay β j = β j (1 − β j ) j , set j ← j + 1 and repeat for L steps until ¯ D KL ≤ δ is satisﬁed.

Original: If this is not satisﬁed after L steps, we backtrack, and do not update φ i.e. set φ ← φ old .
Revised: If this is not satisﬁed after L steps, we backtrack, and do not update φ i.e. set φ ← φ old .

Original: Lemma 1.
Revised: Lemma 1.

Original: If we follow Algorithm 1, during policy updates via equation 4, the following is satisﬁed with high probability ≥ 1 − ω[Equation] ∆
Revised: If we follow Algorithm 1, during policy updates via equation 5, the following is satisﬁed with high probability ≥ 1 − ω

Original: [Equation]
Revised: [Equation]

Original: Here, ζ captures sampling error in the estimation of V π φold C ( µ ) and we have ζ ≤ C√
Revised: Here, ζ captures sampling error in the estimation of V π φold C ( µ ) and we have ζ ≤ C

Original: [Equation]
Revised: [Equation]

Revised: /ω )

Original: | N | , where
Revised: | N | , where

Original: C is a constant and N is the number of samples used in the estimation of V C .
Revised: C is a constant and N is the number of samples used in the estimation of V C .

Original: Proof.
Revised: Proof.

Original: Based on line 6 of Algorithm 1, for every rollout { ( s, a ) } , the following holds:
Revised: Based on line 6 of Algorithm 1, for every rollout { ( s, a ) } , the following holds:

Original: [Equation]
Revised: [Equation]

Original: We note that we can only compute a sample estimate ˆ V π φold C ( µ ) instead of the true quantity V C which can introduce sampling error in practice.
Revised: We note that we can only compute a sample estimate ˆ V π φold C ( µ ) instead of the true quantity V C which can introduce sampling error in practice.

Original: In order to ensure that ˆ V π φold C ( µ ) is not much lesser than V π φold C ( µ ) , we can obtain a bound on their difference.
Revised: In order to ensure that ˆ V π φold C ( µ ) is not much lesser than V π φold C ( µ ) , we can obtain a bound on their difference.

Original: Note that if ˆ V π φold C ( µ ) ≥
Revised: Note that if ˆ V π φold C ( µ ) ≥

Original: V π φold C ( µ ) , the
Revised: V π φold C ( µ ) , the

Original: Lemma holds directly, so we only need to consider the less than case.
Revised: Lemma holds directly, so we only need to consider the less than case.

Original: Let ˆ V π φold C [Equation] =
Revised: Let ˆ V π φold C ( µ ) =

Original: [Equation] µ ) − ζ .
Revised: V π φold C ( µ ) − ζ .

Original: With high probability ≥ 1 − ω , we can ensure ζ ≤ C (cid:48) [Equation]
Revised: With high probability ≥ 1 − ω , we can ensure ζ ≤ C (cid:48) √ log(1 /ω )

Original:  where C (cid:48) is a constant independent of ω (obtained from union bounds and concentration inequalities) and N is the number of samples used in the estimation of V C .
Revised: | N | , where C (cid:48) is a constant independent of ω (obtained from union bounds and concentration inequalities) and N is the number of samples used in the estimation of V C .

Original: In addition, our estimate of
Revised: In addition, our estimate of

Original: E s ∼ ρ φ old ,a ∼ π φ (cid:104)
Revised: E s ∼ ρ φold ,a ∼ π φ (cid:104)

Original: ˆ A C
Revised: ˆ A C

Original: ( s, a ) (cid:105) is an overestimate of the true E s ∼ ρ φ old ,a ∼ π φ [ A C ( s, a )] , and we denote their difference by ∆ .
Revised: ( s, a ) (cid:105) is an overestimate of the true E s ∼ ρ φold [Equation] a ∼ π φ [ [Equation]

Revised: a )] , and we denote their difference by ∆ .

Original: So, with high probability ≥ 1 − ω , we have
Revised: So, with high probability ≥ 1 − ω , we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider policy updates that solve the constrained optimization problem deﬁned in equation 4.
Revised: Consider policy updates that solve the constrained optimization problem deﬁned in equation 5.

Original: With high probability ≥ 1 − ω , we have the following upper bound on expected probability of failure V π φ new C ( µ ) for π φ new during every policy update iteration
Revised: With high probability ≥ 1 − ω , we have the following upper bound on expected probability of failure V π φnew C [Equation] µ ) for π φ new during every policy update iteration

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:15) C = max s | E a ∼ π φnew A C ( s, a ) | and ∆ is the overestimation in E s ∼ ρ φold (cid:48) ,a ∼ π φold [ A C ( s, a )] due to CQL.
Revised: Here, (cid:15) C = max s | E a ∼ π φnew A C ( s, a ) | and ∆ is the overestimation in E s ∼ ρ φold (cid:48) ,a ∼ π φold [ A C ( s, a )] due to CQL.

Original: Proof.
Revised: Proof.

Original: C ( s ) denotes the value of the constraint function from the environment in state s .
Revised: C ( s ) denotes the value of the constraint function from the environment in state s .

Original: This is analogous to the task reward function R ( s, a ) .
Revised: This is analogous to the task reward function R ( s, a ) .

Original: In our case C ( s ) is a binary indicator of whether a catastrophic failure has occurred, however the analysis we present holds even when C ( s ) is a shaped continuous cost function.
Revised: In our case C ( s ) is a binary indicator of whether a catastrophic failure has occurred, however the analysis we present holds even when C ( s ) is a shaped continuous cost function.

Original: [Equation]
Revised: [Equation]

Original: Let V π φ R ( µ ) denotes the discounted task rewards obtained in expectation by executing policy π φ for one episode, and let V π φ C ( µ ) denote the corresponding constraint values.
Revised: Let V π φ R ( µ ) denotes the discounted task rewards obtained in expectation by executing policy π φ for one episode, and let V π φ C ( µ ) denote the corresponding constraint values.

Original: [Equation]
Revised: [Equation]

Original:  From the TRPO (Schulman et al., 2015a) and CPO (Achiam et al., 2017) papers, following similar derivations [Equation] we obtain the following bounds
Revised: [Equation] From the TRPO (Schulman et al., 2015a) and CPO (Achiam et al., 2017) papers, following similar derivations, we obtain the following bounds

Original: [Equation]
Revised: [Equation]

Original: Here, A π φ R is the advantage function corresponding to the task rewards and (cid:15) R [Equation] max s | E [Equation]
Revised: Here, A π φ R is the advantage function corresponding to the task rewards and (cid:15) R = max s | E a ∼ π

Original: ∼

Original: [Equation]
Revised: [Equation]

Original: φ
Revised: A

Original: [Equation]
Revised: [Equation]

Original: s [Equation] a ) | .
Revised: R ( s, a ) | .

Original: D TV is the total variation distance.
Revised: D TV is the total variation distance.

Original: We also have,
Revised: We also have,

Original: [Equation]
Revised: [Equation]

Original: Here,
Revised: Here,

Original: Aπ
Revised: A

Original: [Equation]
Revised: [Equation]

Original: C is the advantage function corresponding to the costs and (cid:15) C = max
Revised: is the advantage function corresponding to the costs and (cid:15) C = max

Original: [Equation]
Revised: [Equation]

Original:  ) | .
Revised: E [Equation] ( s, a ) | .

Original: In our case, A C is deﬁned in terms of the safety Q function
Revised: In our case, A C is deﬁned in terms of the safety Q function

Original: Q C ( s, a ) , and CQL can bound its expectation directly.
Revised: Q C ( s, a ) , and CQL can bound its expectation directly.

Original: To see this, note that, by deﬁnition 
Revised: To see this, note that, by deﬁnition E s

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Here, the RHS is precisely the term in equation 2 of (Kumar et al., 2020) that is bounded by CQL.
Revised: Here, the RHS is precisely the term in equation 2 of (Kumar et al., 2020) that is bounded by CQL.

Original: We get an overstimated advantage ˆ A C ( s, a ) from training the safety critic Q C through updates in equation 2.
Revised: We get an overstimated advantage ˆ A C ( s, a ) from training the safety critic Q C through updates in equation 2.

Original: Let ∆ denote the expected magnitude of over-estimate
Revised: Let ∆ denote the expected magnitude of over-estimate

Revised: E s ∼

Revised: [Equation]

Revised: ,a ∼

Revised: [Equation]

Original: E
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ] + ∆ , where ∆ is positive.
Revised: s, a ) (cid:105) = E s ∼ ρ φold ,a ∼ π φ [ A C ( s, a )] + ∆ , where ∆ is positive.

Original: Note that replacing A C , by its over-estimate ˆ A C , the inequality in 17 above still holds.
Revised: Note that replacing A C , by its over-estimate ˆ A C , the inequality in 16 above still holds.

Original: Using Pinsker’s inequality, we can convert the bounds in terms of D KL instead of DTV ,
Revised: Using Pinsker’s inequality, we can convert the bounds in terms of D KL instead of D

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By Jensen’s inequality,
Revised: By Jensen’s inequality,

Original: [Equation]
Revised: [Equation]

Original: So, we can replace the E [ D TV ( p || q )] terms in the bounds by (cid:112) E [ D KL ( p || q )] .
Revised: So [Equation] we can replace the E [ D TV ( p || q )] terms in the bounds by (cid:112) E [ D KL ( p || q )] .

Original: Then, inequationbecomes,
Revised: Then, inequationbecomes,

Original: [Equation]
Revised: [Equation]

Original: Re-visiting our objective in equation 4,
Revised: Re-visiting our objective in equation 5,

Original: [Equation]
Revised: [Equation]

Original: From inequation 20 we note that instead of of constraining V π φ C ( µ ) we can constrain an upper bound on this.
Revised: From inequation 19 we note that instead of of constraining V π φ C ( µ ) we can constrain an upper bound on this.

Original: Writing the constraint in terms of the current policy iterate π φ old using equation 20,
Revised: Writing the constraint in terms of the current policy iterate π φ old using equation 19,

Original: [Equation]
Revised: [Equation]

Original: As there is already a bound on D KL ( π φ old ( ·| s )
Revised: As there is already a bound on D KL ( π φ old ( ·| s )

Original: ||
Revised: ||

Original: π φ ( ·| s ))] , getting rid of the redundant term
Revised: π φ ( ·| s ))] [Equation] getting rid of the redundant term

Original: [Equation]
Revised: [Equation]

Original: we deﬁne the following optimization problem, which we actually optimize for 
Revised: we deﬁne the following optimization problem [Equation] which we actually optimize for [Equation]

Original: [Equation]
Revised: [Equation]

Original: Upper bound on expected probability of failures.
Revised: Upper bound on expected probability of failures.

Original: If π φ new is updated using equation 4, then we have the following upper bound on V π φnew C
Revised: If π φ new is updated using equation 5, then we have the following upper bound on V π φnew C

Original: [Equation]
Revised: [Equation]

Original: If we ensure V π φold C
Revised: V π φnew C

Original: [Equation]
Revised: [Equation]

Revised: If we ensure V π φold C

Original: [Equation]
Revised: [Equation]

Original: χ holds by following Algorithm 1,we have the following upper bound on V π φnew C
Revised: holds by following Algorithm 1,we have the following upper bound on V π φnew C

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Here, (cid:15) C = max s | E a ∼ π φnew A π φold C ( s, a ) | .
Revised: Here, (cid:15) C = max s | E a ∼ π φnew A π φold C ( s, a ) | .

Original: Now, instead of A C ( s, a ) , we have an over-estimated advantage estimate ˆ A C ( s, a ) obtained by training the safety critic Q C through CQL as in equation 2.
Revised: Now, instead of A C ( s, a ) , we have an over-estimated advantage estimate ˆ A C ( s, a ) obtained by training the safety critic Q C through CQL as in equation 2.

Original: Let ∆ denote the expected magnitude of over-estimate E s ∼ ρ φold ,a ∼ π φ 
Revised: Let ∆ denote the expected magnitude of over-estimate E s ∼ ρ φold ,a ∼ π φ (cid:104)

Original: [Equation]
Revised: ˆ A C ( s, a ) (cid:105) =

Original:  ∼ [Equation] a )] + ∆ , where ∆ is positive.
Revised: E s ∼ ρ φold ,a ∼ π φ [ A C ( s, a )] + ∆ , where ∆ is positive.

Original: From Lemma 1, we are able to ensure the following with high probability ≥ 1 − ω
Revised: From Lemma 1, we are able to ensure the following with high probability ≥ 1 − ω

Original: [Equation]
Revised: [Equation]

Original: By combining this with the upper bound on V π φnew C ( µ ) from inequality 24, we obtain with probability ≥ 
Revised: By combining this with the upper bound on V π φnew C ( µ ) from inequality 23, we obtain with probability ≥ 1 − ω

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since (cid:15) C depends on the optimized policy π φ new , it can’t be calculated exactly prior to the update.
Revised: Since (cid:15) C depends on the optimized policy π φ new , it can’t be calculated exactly prior to the update.

Original: As we cap Q C ( s, a ) to be ≤ 1 , therefore, the best bound we can construct for (cid:15) C is the trivial bound (cid:15) C ≤ 2 .
Revised: As we cap Q C ( s, a ) to be ≤ 1 , therefore, the best bound we can construct for (cid:15) C is the trivial bound (cid:15) C ≤ 2 .

Original: Now, in order to have V π φnew C ( µ ) <
Revised: Now, in order to have V π φnew C ( µ ) <

Original: χ , we require ∆ > 2 √ 2 δγ 1 − γ + (1 − γ ) ζ .
Revised: χ , we require ∆ > 2 √ 2 δγ 1 − γ + (1 − γ ) ζ .

Original: To guarantee this, replacing ∆ by the exact overestimation term from CQL, we have the following condition onα : α > G c,T
Revised: To guarantee this, replacing ∆ by the exact overestimation term from CQL, we have the following condition on

Original: [Equation]
Revised: [Equation]

Original: Here, G c,T is a constant depending on the concentration properties of the safety constraint function
Revised: Here, G c,T is a constant depending on the concentration properties of the safety constraint function

Original: C ( s, a ) and the state transition operator T ( s (cid:48) | s, a ) (Kumar et al., 2020).
Revised: C ( s, a ) and the state transition operator T ( s (cid:48) | s, a ) (Kumar et al., 2020).

Original: φ old (cid:48) denotes the parameters of the policy π in the iteration before φ old .
Revised: φ old (cid:48) denotes the parameters of the policy π in the iteration before φ old .

Original: Now, with probability ≥ 1 − ω , we have ζ ≤ C(cid:48) √ log( [Equation] ω
Revised: Now, with probability ≥ 1 − ω , we have ζ ≤ C

Original: [Equation]
Revised: [Equation]

Original: | N | . So, if α is chosen as follows 
Revised: So, if α is chosen as follows α > G c,

Original: [Equation]
Revised: [Equation]

Original: Then with probability ≥ 1 − ω , we will have,
Revised: Then with probability ≥ 1 − ω , we will have,

Original: [Equation]
Revised: [Equation]

Original: In the next theorem, we show that the convergence rate to the optimal solution is not severely affected due to the safety constraint satisfaction guarantee, and gets modiﬁed by addition of an extra bounded term.
Revised: In the next theorem, we show that the convergence rate to the optimal solution is not severely affected due to the safety constraint satisfaction guarantee, and gets modiﬁed by addition of an extra bounded term.

Original: Theorem 2.
Revised: Theorem 2.

Original: If we run the policy gradient updates through equation 4, for policy π φ , with µ as the starting state distribution, with φ (0) = 0 , and learning rate η > 0 , then for all policy update iterations T > 0 we have, with probability ≥ 1 − ω ,
Revised: If we run the policy gradient updates through equation 5, for policy π φ , with µ as the starting state distribution, with φ (0) = 0 , and learning rate η > 0 , then for all policy update iterations T > 0 we have, with probability ≥ 1 − ω ,

Original: [Equation]
Revised: [Equation]

Original:  Since the value of the dual variables λ strictly decreases during gradient descent updates (Algorithm 1), (cid:80) T − 1 t =

Revised: ηT Since the value of the dual variables λ strictly decreases during gradient descent updates (Algorithm 1), (cid:80) T − 1 t =

Original: λ ( t ) is upper-bounded.
Revised: λ ( t ) is upper-bounded.

Original: In addition, if we choose α as mentioned in the discussion of
Revised: In addition, if we choose α as mentioned in the discussion of

Original: Theorem 1, we have ∆ > 2 √ 2 δγ 1 − γ + ζ .
Revised: Theorem 1, we have ∆ > 2 √ 2 δγ 1 − γ + ζ .

Original: Hence, with probability ≥ 1 − ω , we can ensure that
Revised: Hence, with probability ≥ 1 − ω , we can ensure that

Original: [Equation]
Revised: [Equation]

Original: ηT where K ≤ (1 − χ ) +√ 2 δγ (1 − γ )
Revised: ηT where K ≤ (1 − χ ) +√ 2 δγ (1 − γ )

Original: Proof.
Revised: Proof.

Original: Let superscript ( t ) denote the t th policy update iteration.
Revised: Let superscript ( t ) denote the t th policy update iteration.

Original: We follow the derivation in Lemma 5.2 of (Agarwal et al., 2019) but replace A ( s, a ) with our modiﬁed advantage estimator ˆ A ( t ) ( s, a ) =
Revised: We follow the derivation in Lemma 5.2 of (Agarwal et al., 2019) but replace A ( s, a ) with our modiﬁed advantage estimator ˆ A ( t ) ( s, a ) =

Original: A ( t ) R ( s, a ) − λ ( t )
Revised: A ( t ) R ( s, a ) − λ ( t )

Original: A C ( s, a ) .
Revised: A C ( s, a ) .

Original: The quantity log Z t ( s ) is deﬁned in terms of A ( t ) R as
Revised: The quantity log Z t ( s ) is deﬁned in terms of A ( t ) R as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We deﬁne an equivalent alternate quantity based on ˆ A ( t ) log ˆ Z t ( s )
Revised: We deﬁne an equivalent alternate quantity based on ˆ A ( t ) log ˆ Z t ( s )

Original: = log (cid:88)
Revised: = log (cid:88)

Original: a

Original: [Equation]
Revised: [Equation]

Original: = log (cid:88)

Original: a

Original: [Equation]

Original: ) ≥

Original: [Equation]

Original: a

Original: [Equation]

Original: − γ ))

Original: [-= 
-]
Revised: [-= 
-]

Original: − λ ( t ) η 1 − γ (cid:88) a [Equation] For simplicity, consider softmax policy parameterization (equivalent results hold under the function approximation regime as shown in (Agarwal et al., 2019)), where we deﬁne the policy updates with the modiﬁed advantage function ˆ A ( t ) to take the form:
Revised: For simplicity, consider softmax policy parameterization (equivalent results hold under the function approximation regime as shown in (Agarwal et al., 2019)), where we deﬁne the policy updates with the modiﬁed advantage function ˆ A ( t ) to take the form:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Here, ˆ Z t ( s ) =
Revised: Here, ˆ Z t ( s ) =

Original: (cid:80) a ∈A π ( t ) ( a | s ) exp( η ˆ A ( t ) ( s, a ) / (1 − γ )) .
Revised: (cid:80) a ∈A π ( t ) ( a | s ) exp( η ˆ A ( t ) ( s, a ) / (1 − γ )) .

Original: Note that our actual policy updates (with backtracking line search) are almost equivalent to this when η is small .
Revised: Note that our actual policy updates (with backtracking line search) are almost equivalent to this when η is small .

Original: For the sake of notational convenience, we will denote log ˆ Z t ( s )+ λ ( t ) η 1 − γ (cid:80) a π ( t ) ( a | s ) A ( t ) C ( s, a ) as G t ( s ) .
Revised: For the sake of notational convenience, we will denote log ˆ Z t ( s )+ λ ( t ) η 1 − γ (cid:80) a π ( t ) ( a | s ) A ( t ) C ( s, a ) as G t ( s ) .

Original: We have G t ( s ) ≥from equation 31.
Revised: We have G t ( s ) ≥from equation 30.

Original: We consider the performance improvement lemma (Kakade & Langford, 2002) with respect to the task advantage function A ( t ) R ( s, a ) and express it in terms of the modiﬁed advantage function ˆ A ( t ) ( s, a ) =
Revised: We consider the performance improvement lemma (Kakade & Langford, 2002) with respect to the task advantage function A ( t ) R ( s, a ) and express it in terms of the modiﬁed advantage function ˆ A ( t ) ( s, a ) =

Original: A ( t ) R ( s, a ) − λ
Revised: A ( t ) R ( s, a ) − λ

Original: t ) A C ( s, a ) .
Revised: t ) A C ( s, a ) .

Original: Let µ be the starting state distribution of the MDP, and d ( t ) denote the stationary distribution of states induced by policy π in the t th iteration.
Revised: Let µ be the starting state distribution of the MDP, and d ( t ) denote the stationary distribution of states induced by policy π in the t th iteration.

Original: [Equation]
Revised: [Equation]

Original: We note that G t ( s ) ≥ 

Revised: We note that G t ( s ) ≥

Original: from equation 31.
Revised: from equation 30.

Original: We now prove a result upper bounding the difference between the optimal task value for any state distribution ρ and the task value at the t th iteration for the same state distribution.
Revised: We now prove a result upper bounding the difference between the optimal task value for any state distribution ρ and the task value at the t th iteration for the same state distribution.

Original: Sub-optimality gap.
Revised: Sub-optimality gap.

Original: The difference between the optimal value function and the current value function estimate is upper bounded.
Revised: The difference between the optimal value function and the current value function estimate is upper bounded.

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Using equation 32 with d (cid:63) as the starting state distribution µ , we have:
Revised: Using equation 31 with d (cid:63) as the starting state distribution µ , we have:

Original: [Equation]
Revised: [Equation]

Original: which gives us a bound on E s ∼ d (cid:63) log G t ( s ) .
Revised: which gives us a bound on E s ∼ d (cid:63) log G t ( s ) .

Original: Using the above equation and that V ( t +1) ( ρ ) ≥ V ( t ) ( ρ ) (as V ( t +1) ( s ) ≥ V ( t ) ( s ) for all states s ), we have:
Revised: Using the above equation and that V ( t +1) ( ρ ) ≥ V ( t ) ( ρ ) (as V ( t +1) ( s ) ≥ V ( t ) ( s ) for all states s ), we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Here, ∆ denotes the CQL overestimation penalty, and we have used the fact that each term of (cid:16)
Revised: Here, ∆ denotes the CQL overestimation penalty, and we have used the fact that each term of (cid:16)

Original: [Equation]
Revised: [Equation]

Original:  bounded by
Revised: upper bounded by

Original: [Equation]
Revised: [Equation]

Original: from Lemma 1, so the difference is upper-bounded by 2( χ + ζ − ∆ (1 − γ
Revised: from Lemma 1, so the difference is upper-bounded by 2( χ + ζ − ∆ ( [Equation] γ

Original: [Equation]
Revised: [Equation]

Original: By choosing α as in equation 27, we have ∆ > 2 √
Revised: By choosing α as in equation 26, we have ∆ > 2 √

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, we obtain the relation
Revised: Hence, we obtain the relation

Original: We also observe that 2( χ −
Revised: We also observe that 2( χ −

Original: [Equation]
Revised: [Equation]

Original: So, we have the following result for convergence rate
Revised: So, we have the following result for convergence rate

Original: [Equation]
Revised: [Equation]

Original: Again, with probability ≥ 1 − ω , we can ensure ζ ≤ C (cid:48) √ log(1 /ω [Equation] | N | .
Revised: Again, with probability ≥ 1 − ω , we can ensure ζ ≤ C (cid:48) √ log(1 /ω ) | N | .

Original: Overall, choosing the value of α from equation 28, we have ∆ > 2 √ 2 δγ 1 − γ
Revised: Overall, choosing the value of α from equation 27, we have ∆ > 2 √ 2 δγ 1 − γ

Original: + (1 − γ ) ζ .
Revised: + (1 − γ ) ζ .

Original: So, − ∆ <
Revised: So, [Equation] ∆ <

Original: [Equation]
Revised: [Equation]

Original: √ 2 δγ 1 − γ − (1 − γ ) ζ .
Revised: √ 2 δγ 1 − γ − (1 − γ ) ζ .

Original: Hence, with probability ≥ 1 − ω , we can ensure that
Revised: Hence, with probability ≥ 1 − ω , we can ensure that

Original: [Equation]
Revised: [Equation]

Original: where,
Revised: where,

Original: [Equation]
Revised: [Equation]

Original: So far we have demonstrated that the resulting policy iterates from our algorithm all satisfy the desired safety constraint of the CMDP which allows for a maximum safety violation of χ for every intermediate policy.
Revised: So far we have demonstrated that the resulting policy iterates from our algorithm all satisfy the desired safety constraint of the CMDP which allows for a maximum safety violation of χ for every intermediate policy.

Original: While this guarantee ensures that the probability of failures is bounded, it does not elaborate on the total failures incurred by the algorithm.
Revised: While this guarantee ensures that the probability of failures is bounded, it does not elaborate on the total failures incurred by the algorithm.

Original: In our next result, we show that the cumulative failures until a certain number of iterations T of the algorithm grows sublinearly when executing Algorithm 1, provided the safety threshold χ is set in the right way.
Revised: In our next result, we show that the cumulative failures until a certain number of iterations T of the algorithm grows sublinearly when executing Algorithm 1, provided the safety threshold χ is set in the right way.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let χ in Algorithm 1 be time-dependent such that χ t =
Revised: Let χ in Algorithm 1 be time-dependent such that χ t =

Original: O (1 / √ t ) .
Revised: O (1 / √ t ) .

Original: Then, the total number of cumulative safety violations until when T transition samples have been collected by
Revised: Then, the total number of cumulative safety violations until when T transition samples have been collected by

Original: Algorithm 1, Reg C ( T ) , scales sub-linearly with T , i.e., Reg C ( T ) =
Revised: Algorithm 1, Reg C ( T ) , scales sub-linearly with T , i.e., Reg C ( T ) =

Original: O ( (cid:112) |S||A| T ) .
Revised: O ( (cid:112) |S||A| T ) .

Original: Proof.
Revised: Proof.

Original: From Theorem 1, with probability ≥ 1 − ω
Revised: From Theorem 1, with probability ≥ 1 − ω

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Instead of using φ new , for ease of notation in this analysis, let us write this in terms of subscript t such that π t denotes the policy at the t th iteration.
Revised: Instead of using φ new , for ease of notation in this analysis, let us write this in terms of subscript t such that π t denotes the policy at the t th iteration.

Original: We can write the bound on ζ more explicitly as,ζ
Revised: We can write the bound on ζ more explicitly as,

Original: [Equation]
Revised: [Equation]

Original: Here, N t ( s, a ) denotes the total number of times ( s, a ) is seen, and can be written as N t ( s, a )
Revised: Here, N t ( s, a ) denotes the total number of times ( s, a ) is seen, and can be written as N t ( s, a )

Original: = (cid:80) tj =1 n j ( s, a ) , where n j ( s, a ) denotes total number of times ( s, a ) is seen in episode j .
Revised: = (cid:80) tj =1 n j ( s, a ) , where n j ( s, a ) denotes total number of times ( s, a ) is seen in episode j .

Original: T = (cid:80) s,a N k ( s, a ) denotes the total number of samples collected so far.
Revised: T = (cid:80) s,a N k ( s, a ) denotes the total number of samples collected so far.

Original: Let k be the number of episodes elapsed when this happens.
Revised: Let k be the number of episodes elapsed when this happens.

Original: The safety threshold χ can be varied at every iteration, such that it decreases as χ t ∝ 1 √ t .
Revised: The safety threshold χ can be varied at every iteration, such that it decreases as χ t ∝ 1 √ t .

Original: So, we have with probability ≥ 1 − ω
Revised: So, we have with probability ≥ 1 − ω

Original: V
Revised: V

Original: [Equation]

Original: [Equation]

Original: γ

Original: [Equation]
Revised: [Equation]

Original: where ζ t ≤ E s ∼d
Revised: where ζ t ≤ E s ∼

Original: [Equation]
Revised: [Equation]

Original: [Equation] ∼ [Equation] Now, we note that V π t C denotes the expected probability of episodic failures by executing policy π t (as described in section 2 of the main paper).
Revised: Now, we note that V π t C denotes the expected probability of episodic failures by executing policy π t (as described in section 2 of the main paper).

Original: Let us consider that a policy is rolled out for one episode after every training iteration to collect data (exploration).
Revised: Let us consider that a policy is rolled out for one episode after every training iteration to collect data (exploration).

Original: Hence the total cumulative safety failures when T transition samples have been collected by Algorithm 1 is:
Revised: Hence the total cumulative safety failures when T transition samples have been collected by Algorithm 1 is:

Original: [Equation]
Revised: [Equation]

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we used the concavity of square root function to obtain (cid:80) s,a (cid:112) N k
Revised: we used the concavity of square root function to obtain (cid:80) s,a (cid:112) N k

Original: ( s, a ) ≤ (cid:112) |S||A| (cid:113)(cid:80) s,a N k ( s, a ) and the deﬁnition T
Revised: ( s, a ) ≤ (cid:112) |S||A| (cid:113)(cid:80) s,a N k ( s, a ) and the deﬁnition T

Original: =
Revised: =

Original: (cid:80) s,a N k ( s, a )
Revised: (cid:80) s,a N k ( s, a )

Original: A. D ERIVATION OF THE POLICY UPDATE EQUATIONS
Revised: A. D ERIVATION OF THE POLICY UPDATE EQUATIONS

Original: Let a ∈
Revised: Let a ∈

Original: A denote an action, s ∈ S denote a state, π φ ( a | s ) denote a parameterized policy, r ( s, a ) denote a reward function for the task being solved, and τ denote a trajectory of actions by following policy π φ at each state.
Revised: A denote an action, s ∈ S denote a state, π φ ( a | s ) denote a parameterized policy, r ( s, a ) denote a reward function for the task being solved, and τ denote a trajectory of actions by following policy π φ at each state.

Original: To solve the following constrained optimization problem:
Revised: To solve the following constrained optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Here, τ is the trajectory corresponding to an episode.
Revised: Here, τ is the trajectory corresponding to an episode.

Original: The objective is to maximize the cumulative returns while satisfying the constraint.
Revised: The objective is to maximize the cumulative returns while satisfying the constraint.

Original: The constraint says that the agent must never fail during every episode.
Revised: The constraint says that the agent must never fail during every episode.

Original: 1 { failure } = 1 if there is a failure and 1 { failure } = 

Revised: 1 { failure } = 1 if there is a failure and 1 { failure } =

Original: if the agent does not fail.
Revised: if the agent does not fail.

Original: The only way expectation can be 0 for this quantity is if every element is 0 , so the constraint essentially is to never fail in any episode.
Revised: The only way expectation can be 0 for this quantity is if every element is 0 , so the constraint essentially is to never fail in any episode.

Original: Let’s rewrite the objective, more generally as
Revised: Let’s rewrite the objective, more generally as

Original: [Equation]
Revised: [Equation]

Original: We can relax the constraint slightly, by introducing a tolerance parameter χ ≈ 0 .
Revised: We can relax the constraint slightly, by introducing a tolerance parameter χ ≈ 0 .

Original: The objective below tolerates atmost χ failures in expectation.
Revised: The objective below tolerates atmost χ failures in expectation.

Original: Since the agent can fail only once in an episode,
Revised: Since the agent can fail only once in an episode,

Original: V π φ C ( µ ) can also be interpreted as the probability of failure , and the constraint V π φ C ( µ ) ≤ χ says that the probability of failure in expectation must be bounded by χ .
Revised: V π φ C ( µ ) can also be interpreted as the probability of failure , and the constraint V π φ C ( µ ) ≤ χ says that the probability of failure in expectation must be bounded by χ .

Original: So, our objective has a very intuitive and practical interpretation.
Revised: So, our objective has a very intuitive and practical interpretation.

Original: [Equation]
Revised: [Equation]

Original: [Equation] We learn one state value function, V R (corresponding to the task reward), parameterized by θ and one state-action value function Q C (corresponding to the sparse failure indicator), parameterized by ζ .
Revised: We learn one state value function, V R (corresponding to the task reward), parameterized by θ and one state-action value function Q C (corresponding to the sparse failure indicator), parameterized by ζ .

Original: We have a task reward function r ( s, a ) from the environment which is used to learn V R .
Revised: We have a task reward function r ( s, a ) from the environment which is used to learn V R .

Original: For learning Q C , we get a signal from the environment indicating whether the agent is dead (1) or alive (0) i.e. 1 { failure } .
Revised: For learning Q C , we get a signal from the environment indicating whether the agent is dead (1) or alive (0) i.e. 1 { failure } .

Original: The safety critic Q C is used to get an estimate of how safe a particular state is, by providing an estimate of probability of failure , that will be used to guide exploration.
Revised: The safety critic Q C is used to get an estimate of how safe a particular state is, by providing an estimate of probability of failure , that will be used to guide exploration.

Original: We desire the estimates to be conservative, in the sense that the probability of failure should be an over-estimate of the actual probability so that the agent can err in the side of caution while exploring.
Revised: We desire the estimates to be conservative, in the sense that the probability of failure should be an over-estimate of the actual probability so that the agent can err in the side of caution while exploring.

Original: To train such a critic
Revised: To train such a critic

Original: Q C , we incorporate theoretical insights from CQL, and estimate Q C through updates similar to those obtained by ﬂipping the sign of α in equation 2 of the CQL paper (Kumar et al., 2020).
Revised: Q C , we incorporate theoretical insights from CQL, and estimate Q C through updates similar to those obtained by ﬂipping the sign of α in equation 2 of the CQL paper (Kumar et al., 2020).

Original: The motivation for this is to get an upper bound on Q C instead of a lower bound, as guaranteed by CQL.
Revised: The motivation for this is to get an upper bound on Q C instead of a lower bound, as guaranteed by CQL.

Original: We also note that the CQL penalty term (the ﬁrst two terms of equation 2 of the
Revised: We also note that the CQL penalty term (the ﬁrst two terms of equation 2 of the

Original: CQL paper) can be expressed as an estimate for the advantage function of the policy E
Revised: CQL paper) can be expressed as an estimate for the advantage function of the policy E

Original: [Equation]
Revised: [Equation]

Original:  [ A ( s, a )] ,where, A ( s, a ) is the advantage function.
Revised: s [Equation] [ A ( s, a )] ,where, A ( s, a ) is the advantage function.

Original: [Equation]
Revised: [Equation]

Original: Hence, CQL can help provide an upper bound on the advantage function directly.
Revised: Hence, CQL can help provide an upper bound on the advantage function directly.

Original: Although the CQL class of algorithms have been proposed for batch RL, the basic bounds on the value function hold even for online training.
Revised: Although the CQL class of algorithms have been proposed for batch RL, the basic bounds on the value function hold even for online training.

Original: We denote the objective inside arg min as CQL ( ζ ) , where ζ parameterizes Q C , and k denotes the k th update iteration.
Revised: We denote the objective inside arg min as CQL ( ζ ) , where ζ parameterizes Q C , and k denotes the k th update iteration.

Original: [Equation]
Revised: [Equation]

Original: For states sampled from the replay buffer D env , the ﬁrst term seeks to maximize the expectation of Q C over actions sampled from the current policy, while the second term seeks to minimize the expectation of Q C over actions sampled from the replay buffer.
Revised: For states sampled from the replay buffer D env , the ﬁrst term seeks to maximize the expectation of Q C over actions sampled from the current policy, while the second term seeks to minimize the expectation of Q C over actions sampled from the replay buffer.

Original: D env can include off-policy data, and also ofﬂine-data (if available).
Revised: D env can include off-policy data, and also ofﬂine-data (if available).

Original: Let the over-estimated advantage, corresponding to the overestimated critic Q C , so obtained from CQL, be denoted as ˆ A C ( s, a ) , where the true advantage is
Revised: Let the over-estimated advantage, corresponding to the overestimated critic Q C , so obtained from CQL, be denoted as ˆ A C ( s, a ) , where the true advantage is

Original: A C ( s, a ) .
Revised: A C ( s, a ) .

Original: Now [Equation] let ρ φ ( s ) denote the stationary distribution of states induced by policy π φ .
Revised: Now, let ρ φ ( s ) denote the stationary distribution of states induced by policy π φ .

Original: For policy optimization, we have to solve a constrained optimization problem as described below:
Revised: For policy optimization, we have to solve a constrained optimization problem as described below:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: This, as per equation 23 can be rewritten as
Revised: This, as per equation 22 can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Since we are learning an over-estimate of A C through the updates in equation 2, we replace A C by the learned ˆ A C in the constraint above.
Revised: Since we are learning an over-estimate of A C through the updates in equation 2, we replace A C by the learned ˆ A C in the constraint above.

Original: There are multiple ways to solve this constrained optimization problem, through duality.
Revised: There are multiple ways to solve this constrained optimization problem, through duality.

Original: If we consider the Lagrangian dual of this, then we have the following optimization problem, which we can solve approximately by alternating gradient descent.
Revised: If we consider the Lagrangian dual of this, then we have the following optimization problem, which we can solve approximately by alternating gradient descent.

Original: For now, we keep the KL constraint as is, and later use its second order Taylor expansion in terms of the Fisher Information Matrix.
Revised: For now, we keep the KL constraint as is, and later use its second order Taylor expansion in terms of the Fisher Information Matrix.

Original: [Equation]
Revised: [Equation]

Original: We replace V π φold C ( µ ) by its sample estimate ˆ V π φold C ( µ ) and denote χ − ˆ V π φold C ( µ ) as χ (cid:48) .
Revised: We replace V π φold C ( µ ) by its sample estimate ˆ V π φold C ( µ ) and denote χ − ˆ V π φold C ( µ ) as χ (cid:48) .

Original: Note that χ (cid:48) is independent of parameter φ that is being optimized over.
Revised: Note that χ (cid:48) is independent of parameter φ that is being optimized over.

Original: So, the objective becomes
Revised: So, the objective becomes

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: For notational convenience let λ (cid:48) denote the fraction λ 1 − γ .
Revised: For notational convenience let λ (cid:48) denote the fraction λ 1 − γ .

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: in the expectation
Revised: in the expectation

Original: [Equation]
Revised: [Equation]

Original: we replace a ∼ π φ by a ∼ π φ old and account for it by importance weighting of the objective
Revised: we replace a ∼ π φ by a ∼ π φ old and account for it by importance weighting of the objective

Original: [Equation]
Revised: [Equation]

Original: Let us consider max π φ operation and the following gradient necessary for gradient ascent of φ
Revised: Let us consider max π φ operation and the following gradient necessary for gradient ascent of φ

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Here, using slide 20 of Lecture 9 in (Levine, 2018), and the identity ∇ φ π φ = π φ ∇ φ log π φ we have
Revised: Here, using slide 20 of Lecture 9 in (Levine, 2018), and the identity ∇ φ π φ = π φ ∇ φ log π φ we have

Original: [Equation]
Revised: [Equation]

Original: Using slide 24 of Lecture 5 in (Levine, 2018) and estimating locally at φ
Revised: Using slide 24 of Lecture 5 in (Levine, 2018) and estimating locally at φ

Original: = φ old , ∇
Revised: = φ old , ∇

Original: [Equation]
Revised: [Equation]

Original: We note that, E s ∼ ρ φold
Revised: We note that, E s ∼ ρ φold

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: E

Original: [Equation]
Revised: [Equation]

Original: = ∇ φ old J ( φ old ) , the original policy gradient corresponding to task rewards.

Original: So
Revised: s

Original: [Equation]
Revised: [Equation]

Revised: a ) (cid:105)(cid:105) = ∇ φ old J ( φ old ) , the original policy gradient corresponding to task rewards.

Original:  we can write equation 49 as
Revised: So, we can write equation 48 as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In practice, we estimate A π φold R through GAE (Schulman 
Revised: In practice, we estimate A π φold R through GAE (Schulman et

Original: [Equation]
Revised: [Equation]

Original: Let ˆ A π φold ( s, a ) = 
Revised: Let ˆ A π φold [Equation] s, a ) = A

Revised: [Equation]

Original: A π φold R ( s, a ) − λ
Revised: φold R ( s, a ) − λ

Original: [Equation]
Revised: [Equation]

Original: A C ( s, a ) denote the modiﬁed advantage function corresponding to equation
Revised: A C ( s, a ) denote the modiﬁed advantage function corresponding to equation

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: rewriting equations 49 and 54 in terms of ˜ A π φold
Revised: rewriting equations 48 and 53 in terms of ˜ A π φold

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting in equation
Revised: Substituting in equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: As shown in slide 20 of Lecture 9 (Levine, 2018) and (Schulman et al., 2015a), we can approximate D KL in terms of the Fisher Information Matrix F (this is the second order term in the Taylor expansion of KL; note that around φ =
Revised: As shown in slide 20 of Lecture 9 (Levine, 2018) and (Schulman et al., 2015a), we can approximate D KL in terms of the Fisher Information Matrix F (this is the second order term in the Taylor expansion of KL; note that around φ =

Original: φ old , both the KL term and its gradient are 0),
Revised: φ old , both the KL term and its gradient are 0),

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Where, F can be estimated with samples as
Revised: Where, F can be estimated with samples as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: So, ﬁnally, we can write the gradient ascent step for φ as (natural gradient conversion)
Revised: So, ﬁnally, we can write the gradient ascent step for φ as (natural gradient conversion)

Original: [Equation]
Revised: [Equation]

Original: In practice, we perform backtracking line search to ensure the D KL constraint satisfaction.
Revised: In practice, we perform backtracking line search to ensure the D KL constraint satisfaction.

Original: So, we have the following update rule
Revised: So, we have the following update rule

Original: [Equation]
Revised: [Equation]

Original: After every update, we check if ¯ D KL ( φ || φ old ) ≤ δ , and if not we decay β j = β j (1 − β j ) j , set j ← j + 1 and repeat for L steps until ¯ D KL ≤ δ is satisﬁed.
Revised: After every update, we check if ¯ D KL ( φ || φ old ) ≤ δ , and if not we decay β j = β j (1 − β j ) j , set j ← j + 1 and repeat for L steps until ¯ D KL ≤ δ is satisﬁed.

Original: If this is not satisﬁed after L steps, we backtrack, and do not update φ i.e. set φ ← φ old .
Revised: If this is not satisﬁed after L steps, we backtrack, and do not update φ i.e. set φ ← φ old .

Original: For gradient descent with respect to the Lagrange multiplier λ we have (from equation 5),
Revised: For gradient descent with respect to the Lagrange multiplier λ we have (from equation 5),

Original: [Equation]
Revised: [Equation]

Original: Note that in the derivations we have ommitted (cid:80) t in the outermost loop of all expectations, and subscripts (e.g. a t , s t ) in order to avoid clutter in notations.
Revised: Note that in the derivations we have ommitted (cid:80) t in the outermost loop of all expectations, and subscripts (e.g. a t , s t ) in order to avoid clutter in notations.

Original: A. R ELATION TO CPO
Revised: A. R ELATION TO CPO

Original: The CPO paper (Achiam et al., 2017) considers a very similar overall objective for policy gradient updates, with one major difference.
Revised: The CPO paper (Achiam et al., 2017) considers a very similar overall objective for policy gradient updates, with one major difference.

Original: CPO approximates the V π φ C ( µ ) ≤ χ constraint by replacing
Revised: CPO approximates the V π φ C ( µ ) ≤ χ constraint by replacing

Original: V π φ C ( µ ) with its ﬁrst order Taylor expansion and enforces the resulting simpliﬁed constraint exactly in the dual space.
Revised: V π φ C ( µ ) with its ﬁrst order Taylor expansion and enforces the resulting simpliﬁed constraint exactly in the dual space.

Original: On the other hand, we do not make this simpliﬁcation, and use primal-dual optimization to optimize an upper bound on V C through the CQL inspired objective in equation 2.
Revised: On the other hand, we do not make this simpliﬁcation, and use primal-dual optimization to optimize an upper bound on V C through the CQL inspired objective in equation 2.

Original: Doing this and not not making the linearity modiﬁcation allows us to handle sparse (binary) failure indicators from the environment without assuming a continuous safety cost function as done in
Revised: Doing this and not not making the linearity modiﬁcation allows us to handle sparse (binary) failure indicators from the environment without assuming a continuous safety cost function as done in

Original: CPO (Achiam et al., 2017).
Revised: CPO (Achiam et al., 2017).

Original: A. P RACTICAL CONSIDERATIONS
Revised: A. P RACTICAL CONSIDERATIONS

Original: Depending on the value of KL-constraint on successive policies δ , the RHS in Theorem 2 can either be a lower or higher rate than the corresponding problem without safety constraint.
Revised: Depending on the value of KL-constraint on successive policies δ , the RHS in Theorem 2 can either be a lower or higher rate than the corresponding problem without safety constraint.

Original: In particular, let the sampling error ζ
Revised: In particular, let the sampling error ζ

Original: =
Revised: =

Original: 0 , then if δ ≥ (1 − γ ) 4 (2 − χ ) 2 8 γ, the third term is negative.
Revised: 0 , then if δ ≥ (1 − γ ) 4 (2 − χ ) 2 8 γ, the third term is negative.

Original: If we set γ = 0 . 99 and χ = 0 .
Revised: If we set γ = 0 . 99 and χ = 0 .

Original: 05 , then for any δ > 1e-8 , the third term in Theorem 3 will be negative.
Revised: 05 , then for any δ > 1e-8 , the third term in Theorem 3 will be negative.

Original: Also, if α is chosen to be much greater than that in equation 27, the value of ∆ can be arbitrarily increased in principle, and we would be overestimating the value of Q C signiﬁcantly.
Revised: Also, if α is chosen to be much greater than that in equation 26, the value of ∆ can be arbitrarily increased in principle, and we would be overestimating the value of Q C signiﬁcantly.

Original: While increasing ∆ signiﬁcantly will lead to a decrease in the upper bound of V ∗ R ( µ ) − V ( T ) R ( µ ) , but in practice, we would no longer have a practical algorithm.
Revised: While increasing ∆ signiﬁcantly will lead to a decrease in the upper bound of V ∗ R ( µ ) − V ( T ) R ( µ ) , but in practice, we would no longer have a practical algorithm.

Original: This is because, when Q C is overestimated signiﬁcantly, it would be difﬁcult to guarantee that line 9 of Algorithm 1 is satisﬁed, and policy execution will stop, resulting in inﬁnite wall clock time for the algorithm.
Revised: This is because, when Q C is overestimated signiﬁcantly, it would be difﬁcult to guarantee that line 9 of Algorithm 1 is satisﬁed, and policy execution will stop, resulting in inﬁnite wall clock time for the algorithm.

Original: In order to ensure that the above does not happen, in practice we loop over line 6 of Algorithm 1 for a maximum of 100 iterations.
Revised: In order to ensure that the above does not happen, in practice we loop over line 6 of Algorithm 1 for a maximum of 100 iterations.

Original: So, in practice the anytime safety constraint satisfaction of
Revised: So, in practice the anytime safety constraint satisfaction of

Original: Theorem 2 is violated during the early stages of training when the function approximation of Q C is incorrect.
Revised: Theorem 2 is violated during the early stages of training when the function approximation of Q C is incorrect.

Original: However, as we demonstrate empirically, we are able to ensure the guarantee holds during the majority of the training process.
Revised: However, as we demonstrate empirically, we are able to ensure the guarantee holds during the majority of the training process.

Original: A. D ETAILS ABOUT THE ENVIRONMENTS
Revised: A. D ETAILS ABOUT THE ENVIRONMENTS

Original: In each environment, shown in Figure 2, we deﬁne a task objective that the agent must achieve and a criteria for catastrophic failure .
Revised: In each environment, shown in Figure 2, we deﬁne a task objective that the agent must achieve and a criteria for catastrophic failure .

Original: The goal is to solve the task without dying.
Revised: The goal is to solve the task without dying.

Original: In all the environments, in addition to the task reward, the agent only receives a binary signal indicatin whether it is dead i.e. a catastrophic failure has occurred (1) or alive (0).
Revised: In all the environments, in addition to the task reward, the agent only receives a binary signal indicatin whether it is dead i.e. a catastrophic failure has occurred (1) or alive (0).

Original: • Point agent navigation avoiding traps.
Revised: • Point agent navigation avoiding traps.

Original: Here, a point agent with two independent actuators for turning and moving forward/backward must be controlled in a 2D plane to reach a goal (shown in green in Figure 2) while avoiding traps shown in violet circular regions.
Revised: Here, a point agent with two independent actuators for turning and moving forward/backward must be controlled in a 2D plane to reach a goal (shown in green in Figure 2) while avoiding traps shown in violet circular regions.

Original: The agent has a health counter set to 25 for the episode and it decreases by 1 for every timestep that it resides in a trap.
Revised: The agent has a health counter set to 25 for the episode and it decreases by 1 for every timestep that it resides in a trap.

Original: The agent is alive when the health counter is positive, and a catastrophic failure occurs when the counter strikes 0 and the agent dies.
Revised: The agent is alive when the health counter is positive, and a catastrophic failure occurs when the counter strikes 0 and the agent dies.

Original: • Car agent navigation avoiding traps.
Revised: • Car agent navigation avoiding traps.

Original: Similar environment as the above but the agent is a Car with more complex dynamics.
Revised: Similar environment as the above but the agent is a Car with more complex dynamics.

Original: It has two independently controllable front wheels and free-rolling rear wheel.
Revised: It has two independently controllable front wheels and free-rolling rear wheel.

Original: We adapt this environment from (Ray et al., 2019).
Revised: We adapt this environment from (Ray et al., 2019).

Original: • Panda push without toppling.
Revised: • Panda push without toppling.

Original: A Franka Emika Panda arm must push a vertically placed block across the table to a goal location without the block toppling over.
Revised: A Franka Emika Panda arm must push a vertically placed block across the table to a goal location without the block toppling over.

Original: The workspace dimensions of the table are 20cmx40cm and the dimensions of the block are 5cmx5cmx10cm.
Revised: The workspace dimensions of the table are 20cmx40cm and the dimensions of the block are 5cmx5cmx10cm.

Original: The environment is based on Robosuite Zhu et al.
Revised: The environment is based on Robosuite Zhu et al.

Original: and we use Operational Space Control (OSC) to control the end-effevctor velocities of the robot arm.
Revised: and we use Operational Space Control (OSC) to control the end-effevctor velocities of the robot arm.

Original: A catastrophic failure is said to occur is the block topples.
Revised: A catastrophic failure is said to occur is the block topples.

Original: • Panda push within boundary.
Revised: • Panda push within boundary.

Original: A Franka Emika Panda arm must be controlled to push a block across the table to a goal location without the block going outside a rectangular constraint region.
Revised: A Franka Emika Panda arm must be controlled to push a block across the table to a goal location without the block going outside a rectangular constraint region.

Original: Catastrophic failure occurs when the block center of mass ( ( x, y ) position) move outside the constraint region on the table with dimensions 15cmx35cm.
Revised: Catastrophic failure occurs when the block center of mass ( ( x, y ) position) move outside the constraint region on the table with dimensions 15cmx35cm.

Original: The dimensions of the block are 5cmx5cmx10cm.
Revised: The dimensions of the block are 5cmx5cmx10cm.

Original: The environment is based on Robosuite Zhu et al.
Revised: The environment is based on Robosuite Zhu et al.

Original: and we use Operational Space Control (OSC) to control the end-effector velocities of the robot arm.
Revised: and we use Operational Space Control (OSC) to control the end-effector velocities of the robot arm.

Original: • Laikago walk without falling , a Laikago quadruped robot must walk without falling.
Revised: • Laikago walk without falling , a Laikago quadruped robot must walk without falling.

Original: The agent is rewarded for walking as fast as possible (or trotting) and failure occurs when the robot falls.
Revised: The agent is rewarded for walking as fast as possible (or trotting) and failure occurs when the robot falls.

Original: Since this is an extremely challenging task, for all the baselines, we initialize the agent’s policy with a controller that has been trained to keep the agent standing, while not in motion.
Revised: Since this is an extremely challenging task, for all the baselines, we initialize the agent’s policy with a controller that has been trained to keep the agent standing, while not in motion.

Original: The environment is implemented in PyBullet and is based on (Peng et al., 2020).
Revised: The environment is implemented in PyBullet and is based on (Peng et al., 2020).

Original: A. H YPER - PARAMETER DETAILS
Revised: A. H YPER - PARAMETER DETAILS

Original: We chose the learning rate η Q for the safety-critic Q C to be 2 e − 4 after experimenting with 1 e −and 2 e − 4 and observing slightly better results with the latter.
Revised: We chose the learning rate η Q for the safety-critic Q C to be 2 e − 4 after experimenting with 1 e −and 2 e − 4 and observing slightly better results with the latter.

Original: The value of discount factor γ is set to the usual default value 0 .
Revised: The value of discount factor γ is set to the usual default value 0 .

Original: 99 , the learning rate η λ of the dual variable λ is set to 4 e − 2 , the value of δ for the D KL constraint on policy updates is set to 0 .
Revised: 99 , the learning rate η λ of the dual variable λ is set to 4 e − 2 , the value of δ for the D KL constraint on policy updates is set to 0 .

Original: 01 , and the value of α to be 0 .
Revised: 01 , and the value of α to be 0 .

Original: We experimented with three different α values 0 .
Revised: We experimented with three different α values 0 .

Original: 5 , 5 and found nearly same performance across these three values.
Revised: 5 , 5 and found nearly same performance across these three values.

Original: For policy updates, the backtracking co-efﬁcient β (0) is set to 0.7 and the max.
Revised: For policy updates, the backtracking co-efﬁcient β (0) is set to 0.7 and the max.

Original: number of line search iterations L = 20 .
Revised: number of line search iterations L = 20 .

Original: For the Q-ensembles baseline, the ensemble size is chosen to be 20 (as mentioned in the LNT paper), with the rest of the common hyper-parameter values consistent with CSC, for a fair comparison.
Revised: For the Q-ensembles baseline, the ensemble size is chosen to be 20 (as mentioned in the LNT paper), with the rest of the common hyper-parameter values consistent with CSC, for a fair comparison.

Original: All results are over four random seeds.
Revised: All results are over four random seeds.

Original: A. C OMPLETE RESULTS FOR TRADEOFF BETWEEN SAFETY AND TASK PERFORMANCE
Revised: A. C OMPLETE RESULTS FOR TRADEOFF BETWEEN SAFETY AND TASK PERFORMANCE

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. C OMPLETE RESULTS FOR COMPARISON WITH BASELINES
Revised: A. C OMPLETE RESULTS FOR COMPARISON WITH BASELINES

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON BETWEEN TWO UNCONSTRAINED RL ALGORITHMS
Revised: A. C OMPARISON BETWEEN TWO UNCONSTRAINED RL ALGORITHMS

Original: [Figure]
Revised: [Figure]

Original: A. S EEDING THE REPLAY BUFFER WITH VERY FEW SAMPLES
Revised: A. S EEDING THE REPLAY BUFFER WITH VERY FEW SAMPLES

Original: In order to investigate if we can leverage some ofﬂine user-speciﬁed data to lower the number of failures during training even further we seed the replay buffer of CSC and the baselines withtuples in the Car navigation environment.
Revised: In order to investigate if we can leverage some ofﬂine user-speciﬁed data to lower the number of failures during training even further we seed the replay buffer of CSC and the baselines withtuples in the Car navigation environment.

Original: The 1000 tuples are marked as safe or unsafe depending on whether the car is inside a trap location or not in those states.
Revised: The 1000 tuples are marked as safe or unsafe depending on whether the car is inside a trap location or not in those states.

Original: If our method can leverage such manually marked ofﬂine data (in small quantity as this marking procedure is not cheap), then we have a more practical method that can be deployed in situations where the cost of visiting an unsafe state is signiﬁcantly prohibitive.
Revised: If our method can leverage such manually marked ofﬂine data (in small quantity as this marking procedure is not cheap), then we have a more practical method that can be deployed in situations where the cost of visiting an unsafe state is signiﬁcantly prohibitive.

Original: Note that this is different from the setting of ofﬂine/batch RL, where the entire training data is assumed to be available ofﬂine - in this experimental setting we consider very few tuples (only 1000).
Revised: Note that this is different from the setting of ofﬂine/batch RL, where the entire training data is assumed to be available ofﬂine - in this experimental setting we consider very few tuples (only 1000).

Original: Figure 9 shows that our method can successfully leverage this small ofﬂine data to bootstrap the learning of the safety critic and signiﬁcantly lower the average failures.
Revised: Figure 9 shows that our method can successfully leverage this small ofﬂine data to bootstrap the learning of the safety critic and signiﬁcantly lower the average failures.

Original: We attribute this to training the safety critic conservatively through CQL, which is an effective method for handling ofﬂine data.
Revised: We attribute this to training the safety critic conservatively through CQL, which is an effective method for handling ofﬂine data.

Original: [Figure]
Revised: [Figure]

Original: A. S UPPLEMENTARY VIDEOS

Original: We have attached four videos with the supplementary zip showing trained CSC policies on Car navigation, Panda Push without toppling (Panda Topple), Panda Push within Boundary (Panda boundary), and Laikago walkng/running.

Original: In addition, we have also attached a video showing the standing controller used to bootstrap learning for the Laikago environment (for all the baselines and CSC).

Original: The standing controller is also trained purely from RL by using CSC, specifying a standing reference motion for task reward.

Original: We also link the videos in the anonymous website in the abstract, just in case the attached videos in the zip ﬁle cannot be played due to system setting mismatch.

Original: A. C AR N AVIGATION WITH TRAPS USING CONTINUOUS SAFETY SIGNAL
Revised: A. C AR N AVIGATION WITH TRAPS USING CONTINUOUS SAFETY SIGNAL

Original: In this section we consider the case of a continuous safety signal to show that CSC can learn constraints in this setting as well, and minimize failures signiﬁcantly more compared to the baselines.
Revised: In this section we consider the case of a continuous safety signal to show that CSC can learn constraints in this setting as well, and minimize failures signiﬁcantly more compared to the baselines.

Original: The car navigation with traps provides a natural setting for this, because every time the agent enters a trap region, it can receive a penalty (and its health counter decreases by 1), and a catastrophic failure occurs when the health counter drops to 0.
Revised: The car navigation with traps provides a natural setting for this, because every time the agent enters a trap region, it can receive a penalty (and its health counter decreases by 1) and a catastrophic failure occurs when the health counter drops to 0.

Original: By training the safety critic with this continuous failure signal instead of on binary failure signals, we can capture a notion of impending failure and hence aim to be more safe.
Revised: By training the safety critic with this continuous failure signal instead of on binary failure signals, we can capture a notion of impending failure and hence aim to be more safe.

Original: Note that this setting is a strictly easier evaluation setting than what we previously considered with the binary safety signal.
Revised: Note that this setting is a strictly easier evaluation setting than what we previously considered with the binary safety signal.

Original: [Figure]
Revised: [Figure]