Original: DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
Revised: DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Safe reinforcement learning is extremely challenging. Not only must the agentexplore an unknown environment, it must do so while ensuring no safety constraintviolations.
Revised: Safe reinforcement learning is extremely challenging–not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations.

Original: The problem is typically posed as a constrained Markov decisionprocess (MDP) under an unknown model, often with the learning agent havingaccess to a safe suboptimal baseline policy. Recent results obtain an ˜ O ( |S| p |S| K )objective regret in K episodes for an MDP with |S| states, while being safe at alltimes.
Revised: We formulate this safe reinforcement learning (RL) problem using the framework of a ﬁnite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function, where we model the safety requirements as constraints on the expected cumulative costs that must be satisﬁed during all episodes of learning. We propose a model-based safe RL algorithm that we call

Original: The main idea is to combine a reward bonus for exploration (optimism) witha conservative constraint (pessimism).
Revised: Doubly Optimistic and Pessimistic Exploration (DOPE), and show that it achieves an objective regret ˜ O ( |S| p |A| K )

Original: However, the approach is so pessimisticthat empirical results indicate an inordinately long learning process that keepson applying sequences of  the safe baseline policy.
Revised: without violating the safety constraints during learning, where |S| is the number of states, |A| is the number of actions, and K is the number of learning episodes.

Original: Our key insight isthat suchexcessive pessimism hinders exploration, and needs to be combated by optimismwith respect to the model. This insight yields DOPE, which has a double dose ofoptimism with respect tomodel and reward, while being pessimistic with respectto the constraints.
Revised: Our key idea is to combine a reward bonus for exploration (optimism) with a conservative constraint (pessimism), in addition to the standard optimistic model-based exploration.

Original: We show that DOPE reduces the objective regret to ˜ O ( |S| p K ) ,with no constraint violation. Furthermore, we show in empirical studies that DOPEhas a dramatic performance improvement as compared to earlier  approaches.
Revised: DOPE is not only able to improve the objective regret bound, but also shows a signiﬁcant empirical performance improvement as compared to earlier optimism-pessimism approaches.

Original:  Introduction
Revised:  Introduction

Original: Constrained Markov Decision Processes (CMDPs) impose restrictions that pertain to resource orsafety constraints of the system.
Revised: Constrained Markov Decision Processes (CMDPs) impose restrictions that pertain to resource or safety constraints of the system.

Original: For example, the average radiated power in a wireless communicationsystem must be restricted due to user health and battery lifetime considerations or the frequency ofbraking or accelerating in an autonomous vehicle must be kept bounded to ensure passenger comfort.
Revised: For example, the average radiated power in a wireless communication system must be restricted due to user health and battery lifetime considerations or the frequency of braking or accelerating in an autonomous vehicle must be kept bounded to ensure passenger comfort.

Original: Since these systems have complex dynamics a constrained reinforcement learning (CRL) approachis attractive for determining an optimal control policy.
Revised: Since these systems have complex dynamics, a constrained reinforcement learning (CRL) approach is attractive for determining an optimal control policy.

Original: But how do we ensure that safety or resourceavailability constraints are not violated while learning such an optimal control policy?
Revised: But how do we ensure that safety or resource availability constraints are not violated while learning such an optimal control policy?

Original: Our goal is to develop a framework for safe exploration without constraint violation (with highprobability) for solving CMDP problems where the model is unknown.
Revised: Our goal is to develop a framework for safe exploration without constraint violation (with high probability) for solving CMDP problems where the model is unknown.

Original: While there has been muchwork on RL for both the MDP and the CMDP setting, ensuring safe exploration in the CRL settinghas received less attention.
Revised: While there has been much work on RL for both the MDP and the CMDP setting, ensuring safe exploration in the CRL setting has received less attention.

Original: The problem is challenging, since we do not allow constraint violationeither during learning or deployment, while ensuring a low regret in terms of the optimal objective.
Revised: The problem is challenging, since we do not allow constraint violation either during learning or deployment, while ensuring a low regret in terms of the optimal objective.

Original: Our aim is to explore a model-based approach in an episodic setting under which the model (thetransition kernel of the CMDP) is empirically determined as samples from the system are gathered.
Revised: Our aim is to explore a model-based approach in an episodic setting under which the model (the transition kernel of the CMDP) is empirically determined as samples from the system are gathered.

Original: There has been much recent interest in model-based RL approaches to solving a constrained MDP,the most relevant of which we have summarized in Table 1.
Revised: There has been much recent interest in model-based RL approaches to solving a constrained MDP, the most relevant of which we have summarized in Table 1.

Original: The setup is of a ﬁnite horizon episodic
Revised: The setup is of a ﬁnite horizon episodic

Original: CMDP, with a state space of size |S| , an action space of size |A| and a horizon of length H. Regret34 is measured over the ﬁrst K episodes of the algorithm.
Revised: CMDP, with a state space of size |S| , an action space of size |A| and a horizon of length H. Regret is measured over the ﬁrst K episodes of the algorithm.

Original: Both the attained objective and constraintsatisfaction are computed in an expected sense for a given policy.
Revised: Both the attained objective and constraint satisfaction are computed in an expected sense for a given policy.

Original: Allowing constraint violations 
Revised: Allowing constraint violations during learning means that the algorithm suffers both an objective regret and a constraint regret.

Original: [Table]
Revised: [Table]

Original: during learning means that the algorithm suffers both an objective regret and a constraint regret.
Revised: [Table]

Original: Examples of algorithms following this method are OptCMDP, OptCMDP-Bonus [17].
Revised: Examples of algorithms following this method are OptCMDP, OptCMDP-Bonus [18].

Original: The algorithmsuse different ways of incentivizing exploration to obtain samples to build up an empirical systemmodel.
Revised: The algorithms use different ways of incentivizing exploration to obtain samples to build up an empirical system model.

Original: OptCMDP uses the idea of optimism in the face of uncertainty from the model perspectiveand solves an extended linear program to ﬁnd the best model (highest reward) under the samplesgathered thus far.
Revised: OptCMDP uses the idea of optimism in the face of uncertainty from the model perspective and solves an extended linear program to ﬁnd the best model (highest reward) under the samples gathered thus far.

Original: OptCMDP-Bonus uses a different approach of adding a bonus to the reward term toincentivize exploration of state-action pairs that have fewer samples, which we consider as a form ofoptimism from the reward perspective.
Revised: OptCMDP-Bonus uses a different approach of adding a bonus to the reward term to incentivize exploration of state-action pairs that have fewer samples, which we consider as a form of optimism from the reward perspective.

Original: While the objective regret of both these algorithms is ˜ O ( p K ) ,the constraint regret too is of the same order, since constraints may be violated during learning.
Revised: While the objective regret of both these algorithms is ˜ O ( p K ) , the constraint regret too is of the same order, since constraints may be violated during learning.

Original: An algorithm that begins with no knowledge of the model will have exploration steps during learningthat might violate constraints in expectation.
Revised: An algorithm that begins with no knowledge of the model will have exploration steps during learning that might violate constraints in expectation.

Original: Hence, safe RL approaches assume the availability of aninexpert baseline policy that does not violate the constraints, but is insufﬁcient to explore the entirestate action space.
Revised: Hence, safe RL approaches assume the availability of an inexpert baseline policy that does not violate the constraints, but is insufﬁcient to explore the entire state action space.

Original: So it cannot be simply applied until a high-accuracy model is obtained.
Revised: So it cannot be simply applied until a high-accuracy model is obtained.

Original: A heuristicapproach entitled AlwaysSafe [34] assumes a factored CMDP that allows the easy generation of asafe baseline policy.
Revised: A heuristic approach entitled AlwaysSafe [36] assumes a factored CMDP that allows the easy generation of a safe baseline policy.

Original: It then combines the optimism with respect to the model of OptCMDP with aheuristically chosen hardening of constraints.
Revised: It then combines the optimism with respect to the model of OptCMDP with a heuristically chosen hardening of constraints.

Original: This guarantees no constraint violations, but does nothave regret guarantee with respect to the objective.
Revised: This guarantees no constraint violations, but does not have regret guarantee with respect to the objective.

Original: Its empirical performance is variable and its useis limited to factored CMDP problems.
Revised: Its empirical performance is variable and its use is limited to factored CMDP problems.

Original: The OptPessLP algorithm [26] formalizes the idea of coupling optimism and pessimism by startingwith OptCMDPBonus that has an optimistic reward, and systematically applying decreasing levels ofpessimism with respect to the constraint violations.
Revised: The OptPessLP algorithm [28] formalizes the idea of coupling optimism and pessimism by starting with OptCMDPBonus that has an optimistic reward, and systematically applying decreasing levels of pessimism with respect to the constraint violations.

Original: The approach is successful in ensuring the twingoals of a ˜ O ( p K ) objective regret, while ensuring no constraint violations.
Revised: The approach is successful in ensuring the twin goals of a ˜ O ( p K ) objective regret, while ensuring no constraint violations.

Original: However, the authorsdo not present any empirical performance evaluation results.
Revised: However, the authors do not present any empirical performance evaluation results.

Original: When we implemented OptPessLP,we found that the performance is singularly bad in that linear regret persists for a large number ofsamples, and the tapering off to ˜ O
Revised: When we implemented OptPessLP, we found that the performance is singularly bad in that linear regret persists for a large number of samples, and the tapering off to ˜ O

Original: ( p K ) regret behavior does not appear to happen quickly.
Revised: ( p K ) regret behavior does not appear to happen quickly.

Original: Theproblem with this algorithm is that it is so pessimistic with regard to constraints that it heavilydisincentivizes exploration and ends up choosing the base policy for long sequences.
Revised: The problem with this algorithm is that it is so pessimistic with regard to constraints that it heavily disincentivizes exploration and ends up choosing the base policy for long sequences.

Original: The issue upon which algorithm performance depends is the choice of how to combine optimism andpessimism to obtain both order optimal and empirically good objective regret performance, whileensuring no constraint violations happen.
Revised: The issue upon which algorithm performance depends is the choice of how to combine optimism and pessimism to obtain both order optimal and empirically good objective regret performance, while ensuring no constraint violations happen.

Original: Our insight is that optimism with respect to the modelis a key enabler of exploration, and can be coupled with the addition of optimism with respect tothe reward.
Revised: Our insight is that optimism with respect to the model is a key enabler of exploration, and can be coupled with the addition of optimism with respect to the reward.

Original: This double dose of optimism —both with respect to model and reward—could ensurethat pessimistic hardening of constraints does not excessively retard exploration.
Revised: This double dose of optimism —both with respect to model and reward—could ensure that pessimistic hardening of constraints does not excessively retard exploration.

Original: Following thisinsight, we develop DOPE, a doubly optimistic and pessimistic exploration approach.
Revised: Following this insight, we develop DOPE, a doubly optimistic and pessimistic exploration approach.

Original: We are ableto show that DOPE not only attains ˜ O
Revised: We are able to show that DOPE not only attains ˜ O

Original: ( p K ) objective regret behavior with zero constraint regretwith high probability, it also reduces the objective regret bound over OptPessLP by a factor of p |S| .
Revised: ( p K ) objective regret behavior with zero constraint regret with high probability, it also reduces the objective regret bound over OptPessLP by a factor of p |S| .

Original: 72 We conduct performance analysis simulations under representative CMDP problems and show that DOPE easily outperforms all the earlier approaches.
Revised: We conduct performance analysis simulations under representative CMDP problems and show that DOPE easily outperforms all the earlier approaches.

Original: Thus, the idea of double optimism is not onlyvaluable from the order optimal algorithm design perspective, it also shows good empirical regretperformance, indicating the feasibility of utilizing the methodology in real-world systems.
Revised: Thus, the idea of double optimism is not only valuable from the order optimal algorithm design perspective, it also shows good empirical regret performance, indicating the feasibility of utilizing the methodology in real-world systems.

Original: Related
Revised: The code for the experiments in this paper is located at: https://github.com/archanabura/ DOPE-DoublyOptimisticPessimisticExploration

Original:  Work: Constrained RL: Constrained Markov Decision Processes (CMDP) has been an active area ofresearch [2], with applications in domains such as power systems [37, 24], communication networks[3, 36], and robotics [16, 10].
Revised: Related Work: Constrained RL: Constrained Markov Decision Processes (CMDP) has been an active area of research [2], with applications in domains such as power systems [39, 25], communication networks [3, 38], and robotics [17, 11].

Original: CMDP planing problem (with a known model) is typically solved using a linear programming approach or a Lagrangian approach [2].

Original: In [9], the author proposed anactor-critic RL algorithm for learning the asymptotically optimal policy for an inﬁnite horizon averagecost CMDP when the model is unknown.
Revised: In [9], the author proposed an actor-critic RL algorithm for learning the asymptotically optimal policy for an inﬁnite horizon average cost CMDP when the model is unknown.

Original: This approach is also utilized in function approximationsettings with asymptotic local convergence guarantees [8, 11, 38].
Revised: This approach is also utilized in function approximation settings with asymptotic local convergence guarantees [8, 12, 40].

Original: Policy gradient algorithms for CMDPs have also been developed [1,41, 44, 33, 15].
Revised: Policy gradient algorithms for CMDPs have also been developed [1, 44, 47, 35, 16, 27, 49].

Original: However, none these works address the problemof safe exploration to provide guarantees on the constraint violations during learning.
Revised: However, none these works address the problem of safe exploration to provide guarantees on the constraint violations during learning.

Original: Safe Multi-Armed Bandits: The problem of safe exploration in linear bandits with stage-wise safetyconstraint is studied in [4, 23, 29, 31].
Revised: Safe Multi-Armed Bandits: The problem of safe exploration in linear bandits with stage-wise safety constraint is studied in [4, 24, 31, 33].

Original: Linear bandits with more general constraints have also beenstudied [32, 27].
Revised: Linear bandits with more general constraints have also been studied [34, 29].

Original: These works do not consider the more challenging RL setting which involves anunderlying dynamical system with unknown model.
Revised: These works do not consider the more challenging RL setting which involves an underlying dynamical system with unknown model.

Original: Safe Online Convex Optimization: Online convex optimization [19] has been studied with stochastic constraints [42] and adversarial constraints [30, 43, 25].
Revised: Safe Online Convex Optimization: Online convex optimization [20] has been studied with stochastic constraints [45, 10] and adversarial constraints [32, 46, 26].

Original: These allow constraint violation duringlearning and characterize the cumulative amount of violation.
Revised: These allow constraint violation during learning and characterize the cumulative amount of violation.

Original: A safe Frank-Wolf algorithm forconvex optimization with unknown linear stochastic constraints has been studied in [39].
Revised: A safe Frank-Wolf algorithm for convex optimization with unknown linear stochastic constraints has been studied in [41].

Original: However,these too do not consider the RL setting with an unknown model.
Revised: However, these too do not consider the RL setting with an unknown model.

Original: Exploration in Constrained RL: There has been much work in this area with constraint violationsduring learning, including the work discussed in the introduction [17].
Revised: Exploration in Constrained RL: There has been much work in this area with constraint violations during learning, including the work discussed in the introduction [18].

Original: These include [35, 18, 22],which derive bounds either on the objective and constraint regret or on the sample complexity oflearning an ✏ -optimal policy.
Revised: These include [37, 19, 23], which derive bounds either on the objective and constraint regret or on the sample complexity of learning an ✏ -optimal policy.

Original: Other works on safe RL include [40], where a model-free approachis considered, and [5], that pertains to ofﬂine RL.
Revised: Other works on safe RL include [43, 42], where a model-free approach is considered, and [5], that pertains to ofﬂine RL.

Original: These are complementary to our model-basedapproach.
Revised: These are complementary to our model-based approach.

Original: The problem of learning the optimal policy of a CMDP without violating the constraintswas also studied in [45].
Revised: The problem of learning the optimal policy of a CMDP without violating the constraints was also studied in [48].

Original: However, they assume that the model is known and only the cost functionsare unknown, whereas we address more difﬁcult problem with unknown model and cost functions.
Revised: However, they assume that the model is known and only the cost functions are unknown, whereas we address more difﬁcult problem with unknown model and cost functions.

Original: Notations: For any integer M , [ M ] denotes the set { 1 , . .
Revised: Notations: For any integer M , [ M ] denotes the set { 1 , . .

Original: , M } .
Revised: , M } .

Original: For any two real numbers a, b ,a _ b := max { a, b } .
Revised: For any two real numbers a, b , a _ b := max { a, b } .

Original: For any given set X ,  ( X ) denotes the probability simplex over the set X , and|X| denotes the cardinality of the set X .
Revised: For any given set X ,  ( X ) denotes the probability simplex over the set X , and |X| denotes the cardinality of the set X .

Original:  Preliminaries and Problem Formulation 
Revised:  Preliminaries and Problem Formulation 2.1 Constrained Markov Decision Process

Original: 2.1 Constrained Markov Decision Process

Original: We address the safe exploration problem using the framework of episodic Constrained Markov
Revised: We address the safe exploration problem using the framework of episodic Constrained Markov

Original: Decision Process (CMDP) [2].
Revised: Decision Process (CMDP) [2].

Original: We consider a CMDP, denoted as M = h S , A , r, c, P, H, ¯ C i withr =
Revised: We consider a CMDP, denoted as M = h S , A , r, c, P, H, ¯ C i with r =

Original: ( r h )
Revised: ( r h )

Original: Hh =1 , c
Revised: Hh =1 , c

Original: = ( c h )
Revised: = ( c h )

Original: Hh =1 , P
Revised: Hh =1 , P

Original: =
Revised: =

Original: ( P h )
Revised: ( P h )

Original: Hh =1 , where S is the state space, A is the action space, H is theepisode length, r h : S ⇥ A !
Revised: Hh =1 , where S is the state space, A is the action space, H is the episode length, r h : S ⇥ A !

Original: R is the objective cost function at time step h 2 [ H ] , c
Revised: R is the objective cost function at time step h 2 [ H ] , c

Original: h : S ⇥ A !
Revised: h : S ⇥ A !

Original: R112 is the constraint cost function at time step h 2 [ H ] , P h is the transition probability function with
Revised: R is the constraint cost function at time step h 2 [ H ] , P h is the transition probability function with

Original: P h ( s 0 | s, a ) representing the probability of transitioning to state s 0 when action a is taken at state sat time h .
Revised: P h ( s 0 | s, a ) representing the probability of transitioning to state s 0 when action a is taken at state s at time h .

Original: In the RL context, the transition matrix P is also called the model of the CMDP.
Revised: In the RL context, the transition matrix P is also called the model of the CMDP.

Original: Finally,¯ C is a scalar that speciﬁes the safety constraint in terms of the maximum permissible value for theexpected cumulative constraint cost.
Revised: Finally, ¯ C is a scalar that speciﬁes the safety constraint in terms of the maximum permissible value for the expected cumulative constraint cost.

Original: We consider the setting where |S| and |A| are ﬁnite.
Revised: We consider the setting where |S| and |A| are ﬁnite.

Original: Also,117 without loss of generality, we assume that costs r and c are bounded in [0 , 1] .
Revised: Also, without loss of generality, we assume that costs r and c are bounded in [0 , 1] .

Original: A non-stationary randomized policy ⇡ = ( ⇡ h )
Revised: A non-stationary randomized policy ⇡ = ( ⇡ h )

Original: Hh =1 , ⇡ h : S !
Revised: Hh =1 , ⇡ h : S !

Original:  ( A ) speciﬁes the control action to119 be taken at each time step h 

Revised:  ( A ) speciﬁes the control action to be taken at each time step h

Original: [ H ] .
Revised: [ H ] .

Original: In particular, ⇡ h ( s, a ) denotes the probability of taking actiona when the state is s at time step h .
Revised: In particular, ⇡ h ( s, a ) denotes the probability of taking action a when the state is s at time step h .

Original: For an arbitrary cost function l : [ H ] ⇥ S ⇥ A !
Revised: For an arbitrary cost function l : [ H ] ⇥ S ⇥ A !

Original: R , the valuefunction of a policy ⇡ corresponding to time step h 2 [ H ] given a state s 2 S is deﬁned as
Revised: R , the value function of a policy ⇡ corresponding to time step h 2 [ H ] given a state s 2 S is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where a ⌧ ⇠ ⇡ ⌧ ( s ⌧ , · ) , s ⌧ +1 ⇠ P ⌧ ( ·| s ⌧ , a ⌧ ) .
Revised: (1) where a ⌧ ⇠ ⇡ ⌧ ( s ⌧ , · ) , s ⌧ +1 ⇠ P ⌧ ( ·| s ⌧ , a ⌧ ) .

Original: Since we are mainly interested in the value of a policy123 starting from h = 1 , we simply denote V ⇡ l, 1 ( s ; P ) as V ⇡ l ( s ; P ) .
Revised: Since we are mainly interested in the value of a policy starting from h = 1 , we simply denote V ⇡ l, 1 ( s ; P ) as V ⇡ l ( s ; P ) .

Original: For the rest of the paper, we willassume that the initial state is s 1 is ﬁxed.
Revised: For the rest of the paper, we will assume that the initial state is s 1 is ﬁxed.

Original: So, we will simply denote V ⇡ l ( s 1 ; P ) as V ⇡ l ( P ) , when it isclear from the context.
Revised: So, we will simply denote V ⇡ l ( s 1 ; P ) as V ⇡ l ( P ) , when it is clear from the context.

Original: This standard assumption [17, 14] can be made without loss of generality.
Revised: This standard assumption [18, 15] can be made without loss of generality.

Original: The CMDP (planning) problem with a known model P can then be stated as follows:
Revised: The CMDP (planning) problem with a known model P can then be stated as follows:

Original: [Equation]
Revised: [Equation]

Original: We say that a policy ⇡ is a safe policy if V ⇡ c ( P )  ¯ C , i.e., if the expected cumulative constraint costcorresponding to the policy ⇡ is less than the maximum permissible value ¯ C .
Revised: We say that a policy ⇡ is a safe policy if V ⇡ c ( P )  ¯ C , i.e., if the expected cumulative constraint cost corresponding to the policy ⇡ is less than the maximum permissible value ¯ C .

Original: The set of safe policies ,denoted as ⇧ safe , is deﬁned as ⇧ safe = { ⇡ : V ⇡ c ( P )  ¯ C } .
Revised: The set of safe policies , denoted as ⇧ safe , is deﬁned as ⇧ safe = { ⇡ : V ⇡ c ( P )  ¯ C } .

Original: Without loss of generality, we assume thatthe CMDP problem (2) is feasible, i.e., ⇧ safe is non-empty.
Revised: Without loss of generality, we assume that the CMDP problem (2) is feasible, i.e., ⇧ safe is non-empty.

Original: Let ⇡ ⇤ be the optimal safe policy , whichis the solution of (2).
Revised: Let ⇡ ⇤ be the optimal safe policy , which is the solution of (2).

Original: The CMDP (planning) problem is signiﬁcantly different from the standard Markov Decision Process(MDP) (planning) problem [2].
Revised: The CMDP (planning) problem is signiﬁcantly different from the standard Markov Decision Process (MDP) (planning) problem [2].

Original: Firstly, there may not exist an optimal deterministic policy for a
Revised: Firstly, there may not exist an optimal deterministic policy for a

Original: CMDP, whereas the existence of a deterministic optimal policy is well known for a standard MDP.
Revised: CMDP, whereas the existence of a deterministic optimal policy is well known for a standard MDP.

Original: Secondly, there does not exist a Bellman optimality principle or Bellman equation for CMDP.
Revised: Secondly, there does not exist a Bellman optimality principle or Bellman equation for CMDP.

Original: So, thestandard dynamic programming solution approaches which rely on the Bellman equation cannot bedirectly applied to solve the CMDP problem.
Revised: So, the standard dynamic programming solution approaches which rely on the Bellman equation cannot be directly applied to solve the CMDP problem.

Original: There are two standard approaches for solving the CMDP problem, namely the Lagrangian approachand the linear programming (LP) approach.
Revised: There are two standard approaches for solving the CMDP problem, namely the Lagrangian approach and the linear programming (LP) approach.

Original: Both approaches exploit the zero duality gap propertyof the CMDP problem [2] to ﬁnd the optimal policy.
Revised: Both approaches exploit the zero duality gap property of the CMDP problem [2] to ﬁnd the optimal policy.

Original: In this work, we will use the LP approach,consistent with model optimism.
Revised: In this work, we will use the LP approach, consistent with model optimism.

Original: Details of solving (2) using the LP approach are in Appendix A.
Revised: Details of solving (2) using the LP approach are in Appendix A.

Original: 2.2 Reinforcement Learning with Safe Exploration
Revised: 2.2 Reinforcement Learning with Safe Exploration

Original: The goal of the reinforcement learning with safe exploration is to solve (2), but without the knowledgeof the model P a priori.
Revised: The goal of the reinforcement learning with safe exploration is to solve (2), but without the knowledge of the model P a priori.

Original: Hence, the learning algorithm has to perform exploration by employingdifferent policies to learn P .
Revised: Hence, the learning algorithm has to perform exploration by employing different policies to learn P .

Original: However, we also want the exploration for learning to have a safetyguarantee, i.e, the policies employed during learning should belong to the set of safe policies ⇧ safe .
Revised: However, we also want the exploration for learning to have a safety guarantee, i.e, the policies employed during learning should belong to the set of safe policies ⇧ safe .

Original: Since ⇧ safe itself is deﬁned based on the unknown P , the learning algorithm will not know ⇧ safe apriori.
Revised: Since ⇧ safe itself is deﬁned based on the unknown P , the learning algorithm will not know ⇧ safe a priori.

Original: This makes the safe exploration problem extremely challenging.
Revised: This makes the safe exploration problem extremely challenging.

Original: We consider a model-based RL algorithm thatrepeatedly interacts with the environment in an episodicmanner.
Revised: We consider a model-based RL algorithm that interacts with the environment in an episodic manner.

Original: Let ⇡ k
Revised: Let ⇡ k

Original: =
Revised: =

Original: ( ⇡ h,k )
Revised: ( ⇡ h,k )

Original: Hh =1 be the policy employed by the algorithm in episode k .
Revised: Hh =1 be the policy employed by the algorithm in episode k .

Original: At each timestep h 2 [ H ] in an episode k , the algorithm observes state s h,k , selects action a h,k ⇠ ⇡ h,k
Revised: At each time step h 2 [ H ] in an episode k , the algorithm observes state s h,k , selects action a h,k ⇠ ⇡ h,k

Original: ( s h,k , · ) ,152 and incurs the costs r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) .
Revised: ( s h,k , · ) , and incurs the costs r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) .

Original: The next state s h
Revised: The next state s h

Original: +1 ,k is realized according tothe probability vector P ( ·| s h,k , a h,k ) .
Revised: +1 ,k is realized according to the probability vector P ( ·| s h,k , a h,k ) .

Original: As stated before, for simplicity, we assume that the initial stateis ﬁxed for each episode[-k 
-]
Revised: As stated before, for simplicity, we assume that the initial state is ﬁxed for each episode[-k 
-]

Original:  [ K ] := { 1 , . . .
Revised: k 2 [ K ] := { 1 , . . .

Original: , K } , i.e., s 1 ,k
Revised: , K } , i.e., s 1 ,k

Original: = s 1 .
Revised: = s 1 .

Original: We also assume that the maximumpermissible cost ¯ C for any exploration policy is known and it is speciﬁed as part of the learningproblem.
Revised: We also assume that the maximum permissible cost ¯ C for any exploration policy is known and it is speciﬁed as part of the learning problem.

Original: The performance of the RL algorithm is measured using the metric of safe objective regret .
Revised: The performance of the RL algorithm is measured using the metric of safe objective regret .

Original: The safeobjective regret is deﬁned exactly as the standard regret of an RL algorithm for exploration in MDPs[20, 13, 6], but with an additional constraint that the exploration polices should belong to the safe set⇧ safe .
Revised: The safe objective regret is deﬁned exactly as the standard regret of an RL algorithm for exploration in MDPs [21, 14, 6], but with an additional constraint that the exploration polices should belong to the safe set ⇧ safe .

Original: Formally, the safe objective regret R ( K ) after K learning episodes is deﬁned as
Revised: Formally, the safe objective regret R ( K ) after K learning episodes is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Since ⇧ safe is unknown, clearly it is not possible to employ a safe policy without making any additionalassumptions.
Revised: Since ⇧ safe is unknown, clearly it is not possible to employ a safe policy without making any additional assumptions.

Original: We overcome this obvious limitation by assuming that the algorithm has access to asafe baseline policy ⇡ b such that ⇡ b 2 ⇧ safe .
Revised: We overcome this obvious limitation by assuming that the algorithm has access to a safe baseline policy ⇡ b such that ⇡ b 2 ⇧ safe .

Original: We formalize this assumption as follows.
Revised: We formalize this assumption as follows.

Original: Assumption 1 (Safe baseline policy) .
Revised: Assumption 1 (Safe baseline policy) .

Original: The algorithm knows a safe baseline policy ⇡ b such that
Revised: The algorithm knows a safe baseline policy ⇡ b such that

Original: V ⇡ b c ( P ) =
Revised: V ⇡ b c ( P ) =

Original: ¯ C b , where ¯ C b < ¯ C .
Revised: ¯ C b , where ¯ C b < ¯ C .

Original: The value ¯ C b is also known to the algorithm.
Revised: The value ¯ C b is also known to the algorithm.

Original: Remark 2 .
Revised: Remark 2 .

Original: Knowing a safe policy ⇡ b is necessary for solving the safe RL problem because werequire the constraint to always be satisﬁed.
Revised: Knowing a safe policy ⇡ b is necessary for solving the safe RL problem because we require the constraint to always be satisﬁed.

Original: A similar assumption has been used in the case of safeexploration in linear bandits [4, 23, 31], as well as in earlier work on safe RL [34, 26].
Revised: A similar assumption has been used in the case of safe exploration in linear bandits [4, 24, 33], as well as in earlier work on safe RL [36, 28].

Original:  Algorithm and Performance Guarantee
Revised:  Algorithm and Performance Guarantee

Original: DOPE builds on the optimism in the face of uncertainty (OFU) style exploration algorithms for RL[20, 13], using such optimism, both in terms of the model, as well as to provide a reward bonusfor under-explored state-actions.
Revised: DOPE builds on the optimism in the face of uncertainty (OFU) style exploration algorithms for RL [21, 14], using such optimism, both in terms of the model, as well as to provide a reward bonus for under-explored state-actions.

Original: However, a naive OFU-style algorithm may lead to selectingexploration policies that are not in the safe set ⇧ safe .
Revised: However, a naive OFU-style algorithm may lead to selecting exploration policies that are not in the safe set ⇧ safe .

Original: So we modify the selection of exploratory policyby incorporating pessimism in the face of uncertainty (PFU) on the constraints, making DOPE doublyoptimistic and pessimistic in exploration.
Revised: So we modify the selection of exploratory policy by incorporating pessimism in the face of uncertainty (PFU) on the constraints, making DOPE doubly optimistic and pessimistic in exploration.

Original: DOPE operates in episodes, each of length H .
Revised: DOPE operates in episodes, each of length H .

Original: Deﬁne the ﬁltration F k as the sigma algebra generatedby the observations until the end of episode k 

Revised: Deﬁne the ﬁltration F k as the sigma algebra generated by the observations until the end of episode k

Original: [ K ] , i.e., F k = ( s h,k 0 , a h,k 0 , h 

Revised: [ K ] , i.e., F k = ( s h,k 0 , a h,k 0 , h

Original: [ H ] , k 0 

Revised: [ H ] , k 0

Original: [ k ]) .
Revised: [ k ]) .

Original: Letn h,k ( s, a ) =
Revised: Let n h,k ( s, a ) =

Original: P k  1 k 

Revised: P k  1 k

Original: =
Revised: =

Original: 1 { s h,k 

Revised: 1 { s h,k

Original: =
Revised: =

Original: s, a h,k 

Revised: s, a h,k

Original: = a } be the number of times the pair ( s, a ) was observedat time step h until the beginning of episode k .
Revised: = a } be the number of times the pair ( s, a ) was observed at time step h until the beginning of episode k .

Original: Similarly, deﬁne n h,k
Revised: Similarly, deﬁne n h,k

Original: ( s, a, s 0 ) =
Revised: ( s, a, s 0 ) =

Original: P k {+ 1 k 
+}
Revised: P k {+ 1 k 
+}

Original: [Equation]
Revised: =

Original: 1k 0 =1 { s h,k 
 
Revised: 1 { s h,k 
 =

Original: = 180 s, a h,k 

Revised: s, a h,k

Original: = a, s h
Revised: = a, s h

Original: +1 ,k 

Revised: +1 ,k

Original: = s 0 } .
Revised: = s 0 } .

Original: At the beginning of each episode k , DOPE estimates the model asb
Revised: At the beginning of each episode k , DOPE estimates the model as b

Original: P h,k ( s 0 | s, a ) =
Revised: P h,k ( s 0 | s, a ) =

Original: n h,k ( s,a,s 0 ) n h,k
Revised: n h,k ( s, a, s 0 ) /

Original: ( s,a ) 
Revised: ( n h,k ( s, a ) _ 1) .

Original: Similar to OFU-style algorithms, we construct a conﬁdence set P karound b
Revised: Similar to OFU-style algorithms, we construct a conﬁdence set P k around b

Original: P k as P k = \ ( s,a ) 2 S ⇥
Revised: P k as P k = \ ( s,a ) 2 S ⇥

Original: A P k ( s, a ) , where
Revised: A P k ( s, a ) , where

Original: [Equation]
Revised: [Equation]

Revised: where L = log( 2 SAHK  ) , and Var ( b

Revised: P h,k

Original: [Equation]
Revised: [Equation]

Original: where L =log( 2 SAHK  ) , and Var ( b
Revised: s [Equation] =

Original: [Equation]
Revised: [Equation]

Revised: P

Revised: [Equation]

Original: ,k
Revised: ,k

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: | s
Revised: | [Equation] [Equation] [Equation]

Revised: P h,k ( s

Original: [Equation]
Revised: [Equation]

Original: k ( s 0 | s, a )) .
Revised: | s, a )) .

Original: Using theempirical Bernstein inequality, we can show that the true model P is an element of P k for [-anyk 
-]
Revised: Using the empirical Bernstein inequality, we can show that the true model P is an element of P k for [-anyk 
-]any

Original:  [ K ] with probability at least 1  2  (see Appendix C).
Revised: k 2 [ K ] with probability at least 1  2  (see Appendix C).

Original: Similarly, at the beginning of each episode k , DOPE estimates the unknown objective and constraintcosts as ˆ r h,k
Revised: Similarly, at the beginning of each episode k , DOPE estimates the unknown objective and constraint costs as ˆ r h,k

Original: ( s, a ) =

Original: [-P k  1 k 0 =1 r h ( s,a ) { s h,k 
-]
Revised: [-P k  1 k 0 =1 r h ( s,a ) { s h,k 
-]

Original: =

Original: s,a h,k

Original: [Equation]

Original: a } n h,k ( s,a )

Original: [Equation]
Revised: [Equation]

Original: In keeping with OFU, we construct conﬁdence sets R k and C k around ˆ r k and ˆ c k respectively, as
Revised: In keeping with OFU, we construct conﬁdence sets R k and C k around ˆ r k and ˆ c k respectively, as

Original: [Equation]
Revised: [Equation]

Original: 6) where L 0 = 2 log( 6 SAHK  ) , and ˜ r
Revised: where L 0 = 2 log(6 SAHK/  ) , and ˜ r

Original: = (˜ r h )
Revised: = (˜ r h )

Original: Hh =1 ,
Revised: Hh =1 ,

Original: ˜ c
Revised: ˜ c

Original: = (˜ c h )
Revised: = (˜ c h )

Original: Hh
Revised: Hh

Original: =1 .
Revised: =1 .

Original: Using Hoeffding inequality, we canshow that the true costs belong to R k and C k for any k 

Revised: Using Hoeffding inequality, we can show that the true costs belong to R k and C k for any k

Original: [ K ] with probability at least 1   (see191 Appendix C).
Revised: [ K ] with probability at least 1   (see Appendix C).

Original: We deﬁne
Revised: We deﬁne

Original: M k

Original: [Equation]
Revised: [Equation]

Original: P k \ R k \ C k to be the total conﬁdence ball around the empirical 192 transitions and costs.
Revised: k to be the total conﬁdence ball.

Original: It is tempting to use the standard OFU approach for selecting the exploration polices since thisapproach is known to provide sharp regret guarantees for exploration problems in RL.
Revised: It is tempting to use the standard OFU approach for selecting the exploration polices since this approach is known to provide sharp regret guarantees for exploration problems in RL.

Original: The standard
Revised: The standard

Original: OFU approach will ﬁnd the optimistic model P k and optimistic policy ⇡ k , where
Revised: OFU approach will ﬁnd the optimistic model P k and optimistic policy ⇡ k , where

Original: [Equation]
Revised: [Equation]

Original: The OFU problem (7) is feasible since the true model M is an element of M k (with high probability).
Revised: The OFU problem (7) is feasible since the true model M is an element of M k (with high probability).

Original: In particular, ( ⇡ b , P ) and ( ⇡ ⇤ , P ) are feasible solutions of (7).
Revised: In particular, ( ⇡ b , P ) and ( ⇡ ⇤ , P ) are feasible solutions of (7).

Original: Moreover, (7) can be solved efﬁcientlyusing an extended linear programming approach, as described in Appendix B.
Revised: Moreover, (7) can be solved efﬁciently using an extended linear programming approach, as described in Appendix B.

Original: The policy ⇡ k ensuresexploration while satisfying the constraint V ⇡ k c ( P k )  ¯ C .
Revised: The policy ⇡ k ensures exploration while satisfying the constraint V ⇡ k c ( P k )  ¯ C .

Original: However, this naive OFU approachoverlooks the important issue that ⇡ k may not be a safe policy with respect to the true model P .
Revised: However, this naive OFU approach overlooks the important issue that ⇡ k may not be a safe policy with respect to the true model P .

Original: Moreprecisely, it is possible to have V ⇡ k c ( P ) >
Revised: More precisely, it is possible to have V ⇡ k c ( P ) >

Original: ¯ C even though V ⇡ k c ( P k )  ¯ C .
Revised: ¯ C even though V ⇡ k c ( P k )  ¯ C .

Original: So, the standard OFUapproach alone will not give a safe exploration strategy.
Revised: So, the standard OFU approach alone will not give a safe exploration strategy.

Original: In order to ensure that the exploration policy employed at any episode is safe, we add a pessimisticpenalty to the empirical constraint cost to get the pessimistic constraint cost function ¯ c k as
Revised: In order to ensure that the exploration policy employed at any episode is safe, we add a pessimistic penalty to the empirical constraint cost to get the pessimistic constraint cost function ¯ c k as

Original: [Equation]
Revised: [Equation]

Original: (8) where ¯  ph,k ( s, a ) = 
Revised: where ¯  ph,k ( s, a ) = P s

Original: P s 0 2 S  ph,k ( s, a, s 0 ) .
Revised: [Equation] S  ph,k ( s, a, s 0 ) .

Original: Since ¯  ph,k
Revised: Since ¯  ph,k

Original: ( s, a ) is ˜ O
Revised: ( s, a ) is ˜ O

Original: (1 / p n h,k ( s, a )) , ( s, a ) pairs that areless observed have a higher penalty, disincentivizing their exploration.
Revised: (1 / p n h,k ( s, a )) , ( s, a ) pairs that are less observed have a higher penalty, disincentivizing their exploration.

Original: However.
Revised: However.

Original: such a pessimisticpenalty may prevent the exploration that is necessary to learn the optimal policy.
Revised: such a pessimistic penalty may prevent the exploration that is necessary to learn the optimal policy.

Original: To overcome thisissue, we also modify the empirical objective cost function by subtracting a term to incentivizeexploration, to obtain an optimistic objective cost function
Revised: To overcome this issue, we also modify the empirical objective cost function by subtracting a term to incentivize exploration, to obtain an optimistic objective cost function

Original: [Equation]
Revised: [Equation]

Revised: Since ¯  ph,k ( s, a ) is ˜

Revised: O

Revised: [Equation]

Revised: [Equation]

Revised: ,k

Original: Since ¯  p h,k ( s, a ) is ˜ O (1 / p n h,k ( s, a )) , ( s, a ) pairs that are less observed will have a lowered cost toincentivize their exploration.
Revised: [Equation] s, a )) , [Equation] s, a ) pairs that are less observed will have a lowered cost to incentivize their exploration.

Original: We select the policy ⇡ k for episode k by solving the Doubly Optimistic-Pessimistic (DOP) problem:
Revised: We select the policy ⇡ k for episode k by solving the Doubly Optimistic-Pessimistic (DOP) problem:

Original: [Equation]
Revised: [Equation]

Original: Notice that DOPE is doubly optimistic by considering both the optimistic objective cost functionin (9) and the optimistic model P k from the conﬁdence set P k in (10), while being pessimistic on theconstraint in (10).
Revised: Notice that DOPE is doubly optimistic by considering both the optimistic objective cost function in (9) and the optimistic model P k from the conﬁdence set P k in (10), while being pessimistic on the constraint in (10).

Original: Later, in Lemma 18 in the appendix, we prove that ( ⇡ k , P k ) is indeed an optimisticsolution.
Revised: Later, in Lemma 18 in the appendix, we prove that ( ⇡ k , P k ) is indeed an optimistic solution.

Original: This is in contrast with [26], where the optimism is solely limited to the objective cost.
Revised: This is in contrast with [28], where the optimism is solely limited to the objective cost.

Original: Wewill show that our approach carefully balances double optimism and pessimism, yielding a regretminimizing learning algorithm with episodic safe exploration guarantees.
Revised: We will show that our approach carefully balances double optimism and pessimism, yielding a regret minimizing learning algorithm with episodic safe exploration guarantees.

Original: We note that the DOP problem (10) may not be feasible, especially in the ﬁrst few episodes oflearning.
Revised: We note that the DOP problem (10) may not be feasible, especially in the ﬁrst few episodes of learning.

Original: This is because, ¯  ph,k ( s, a ) and  lh,k ( s, a ) may be large during the initial phase of learningso that there may not be a policy ⇡ 0 and a model P 0 2 P k that can satisfy the constraint V ⇡ 

Revised: This is because, ¯  ph,k ( s, a ) and  lh,k ( s, a ) may be large during the initial phase of learning so that there may not be a policy ⇡ 0 and a model P 0 2 P k that can satisfy the constraint V ⇡

Original: ¯ c k
Revised: ¯ c k

Original: ( P 0 )  ¯ C .
Revised: ( P 0 )  ¯ C .

Original: We overcome this issue by employing a safe baseline policy ⇡ b (as deﬁned in Assumption 1) in theﬁrst K o episodes, a value provided by Proposition 4.
Revised: We overcome this issue by employing a safe baseline policy ⇡ b (as deﬁned in Assumption 1) in the ﬁrst K o episodes, a value provided by Proposition 4.

Original: Since ⇡ b is safe by deﬁnition, DOPE ensuressafety during the ﬁrst K o episodes.
Revised: Since ⇡ b is safe by deﬁnition, DOPE ensures safety during the ﬁrst K o episodes.

Original: We will later show that the DOP problem (10) will have a feasiblesolution after the ﬁrst K o episodes (see Proposition 4).
Revised: We will later show that the DOP problem (10) will have a feasible solution after the ﬁrst K o episodes (see Proposition 4).

Original: For any episode k  K o , DOPE employspolicy ⇡ k , which is the solution of (10).
Revised: For any episode k  K o , DOPE employs policy ⇡ k , which is the solution of (10).

Original: We will also show that ⇡ k from (10) (once it becomesfeasible) will indeed be a safe policy (see Proposition 5).
Revised: We will also show that ⇡ k from (10) (once it becomes feasible) will indeed be a safe policy (see Proposition 5).

Original: We present DOPE formally in Algorithm 1.
Revised: We present DOPE formally in Algorithm 1.

Original: 1: Input:  2 (0 , 1) , r, c, ⇡ b , ¯ C b , ¯ C, K o 2: Initialization: n h,k

Original: ( s, a ) =

Original: n h,k ( s, a, s 0 )

Original: = 0 8 s, s 0 2 S, a 2 A,

Original: h 2 [ H ] .

Original: 3: for episodes k = , K do 4: Compute the estimates b P k , ˆ r k , ˆ c k and the conﬁdence set M k according to (4) - (5), and (6).
Revised: conﬁdence set M k according to (4) - (5), and (6).

Original: 5: if k  K o then6:
Revised: : if k  K o then

Original: Select the exploration policy ⇡ k = ⇡
Revised: Select the exploration policy ⇡ k = ⇡

Original: b 7: else 8:
Revised: b 7: else 8:

Original: Select the exploration policy ⇡ k according to (10) 9: end if 10: for h = 1 , 2 , . . .
Revised: Select the exploration policy ⇡ k according to (10) 9: end if 10: for h = 1 , 2 , . . .

Original: , H do 11:
Revised: , H do 11:

Original: Observe state s h,k , select action a h,k ⇠ ⇡ h,k
Revised: Observe state s h,k , select action a h,k ⇠ ⇡ h,k

Original: ( s h,k , · ) , incur the cost r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) , and observe next state s h
Revised: ( s h,k , · ) , incur the cost r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) , and observe next state s h

Original: +1 ,k ⇠ P h ( ·| s h,k , a h,k ) 12:
Revised: +1 ,k ⇠ P h ( ·| s h,k , a h,k ) 12:

Original: Update the counts: n h,k
Revised: Update the counts: n h,k

Original: ( s h,k , a h,k ) n h,k
Revised: ( s h,k , a h,k ) n h,k

Original: ( s h,k , a h,k ) + 1 , n h,k ( s h,k , a h,k , s h +1 ,k ) n h,k
Revised: ( s h,k , a h,k ) + 1 , n h,k ( s h,k , a h,k , s h +1 ,k ) n h,k

Original: ( s h,k , a h,k , s h
Revised: ( s h,k , a h,k , s h

Original: +1 ,k ) +13: end for 14: end for
Revised: +1 ,k ) +13: end for 14: end for

Original: We now present our main result, which shows that the DOPE algorithm achieves
Revised: We now present our main result, which shows that the DOPE algorithm achieves

Original: ˜ O
Revised: ˜ O

Original: ( p K ) regretwithout violating the safety constraints during learning, with high probability.
Revised: ( p K ) regret without violating the safety constraints during learning, with high probability.

Original: Theorem 3.
Revised: Theorem 3.

Original: Fix any  2 (0 , 1) .
Revised: Fix any  2 (0 , 1) .

Original: Consider the DOPE algorithm with K o as speciﬁed in Proposition4.
Revised: Consider the DOPE algorithm with K o as speciﬁed in Proposition 4.

Original: Let { ⇡ k , k 

Revised: Let { ⇡ k , k

Original: [ K ] } be the sequence of policies generated by the DOPE algorithm.
Revised: [ K ] } be the sequence of policies generated by the DOPE algorithm.

Original: Then, withprobability at least 1  5  , ⇡ k 2 ⇧ safe for all k 2 [ K ] .
Revised: Then, with probability at least 1  5  , ⇡ k 2 ⇧ safe for all k 2 [ K ] .

Original: Moreover, with probability at least 1  5  ,233 the regret of the DOPE algorithm satisﬁes
Revised: Moreover, with probability at least 1  5  , the regret of the DOPE algorithm satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  Analysis
Revised:  Analysis

Original: We now provide the technical analysis of DOPE, concluding with the proof outline of Theorem 3.
Revised: We now provide the technical analysis of DOPE, concluding with the proof outline of Theorem 3.

Original: 4.1 Preliminaries
Revised: 4.1 Preliminaries

Original: For an arbitrary policy ⇡ 0 and transition probability function P 0 , deﬁne ✏ ⇡ 0 k ( P 0 ) and
Revised: For an arbitrary policy ⇡ 0 and transition probability function P 0 , deﬁne ✏ ⇡ 0 k ( P 0 ) and

Original: ⌘ ⇡ 0 k ( P 0 ) as 
Revised: ⌘ ⇡ 0 k ( P 0 ) as ✏ ⇡

Original: [Equation]
Revised: [Equation]

Original: Then, it is straightforward to show that (see (23) - (24) in the Appendix) V ⇡ 0 ¯ c k ( P 0 ) =
Revised: Then, it is straightforward to show that (see (23) - (24) in the Appendix) V ⇡ 0 ¯ c k ( P 0 ) =

Original: V ⇡ 

Revised: V ⇡

Original: ˆ c
Revised: ˆ c

Original: k
Revised: k

Original: ( P 0 ) +⌘ ⇡ 0 k ( P ) + ✏ ⇡ 

Revised: ( P 0 ) + ⌘ ⇡ 0 k ( P ) + ✏ ⇡

Original: k ( P 0 )
Revised: k ( P 0 )

Original: , V ⇡ 0 ¯ r k ( P 0 ) =
Revised: , V ⇡ 0 ¯ r k ( P 0 ) =

Original: V ⇡ 0 ˆ r k
Revised: V ⇡ 0 ˆ r k

Original: ( P 0 )
Revised: ( P 0 )

Original:  3 H ¯ C  ¯ C b ⌘ ⇡ 0 k ( P 0 ) 
Revised:  3 H ¯ C  ¯ C b ⌘ ⇡ 0 k ( P 0 )

Original: H ¯ C  ¯ C b ✏ ⇡ 0 k ( P 0 ) .
Revised: H ¯ C  ¯ C b ✏ ⇡ 0 k ( P 0 ) .

Original: The analysis utilizes thisdecomposition of V ⇡ 0 ¯ c k
Revised: The analysis utilizes this decomposition of V ⇡ 0 ¯ c k

Original: ( P 0 ) and V ⇡ 0 ¯ r k
Revised: ( P 0 ) and V ⇡ 0 ¯ r k

Original: ( P 0 ) , and the properties of ✏ ⇡ 0 k ( P 0 ) and ⌘ ⇡ 0 k ( P 0 ) .
Revised: ( P 0 ) , and the properties of ✏ ⇡ 0 k ( P 0 ) and ⌘ ⇡ 0 k ( P 0 ) .

Original: Under the goodevent G deﬁned as in Lemma 12, we can show, V ⇡ {+
+}
Revised: Under the good event G deﬁned as in Lemma 12, we can show, V ⇡ {+
+}

Original: [Equation]
Revised: ˆ c k

Original: ( P 
Revised: ( P 0 )  ⌘ ⇡ 0 k ( P 0 )  V ⇡ 0 c

Original: [Equation]{+( P 0 )  V ⇡ 
 ˆ c k ( P 0 )+ ⌘ ⇡ 0 k ( P 0 ) .+}
Revised: {+( P 0 )  V ⇡ 
 ˆ c k ( P 0 )+ ⌘ ⇡ 0 k ( P 0 ) .+}

Original: 4.2 Feasibility of the OP Problem
Revised: 4.2 Feasibility of the OP Problem

Original: Even though ( ⇡ b , P ) is a feasible solution to the original CMDP problem (2), it may not be afeasible for the DOP problem (10) in the initial phase of learning.
Revised: Even though ( ⇡ b , P ) is a feasible solution to the original CMDP problem (2), it may not be a feasible for the DOP problem (10) in the initial phase of learning.

Original: To see this, note that V ⇡ b ¯ c k ( P ) =
Revised: To see this, note that V ⇡ b ¯ c k ( P ) =

Original: V ⇡ b ˆ c k ( P ) + ⌘ ⇡ b k ( P ) + ✏ ⇡ b k ( P ) , and since V ⇡ b c ( P )  V ⇡ b ˆ c k ( P )  ⌘ ⇡ b k ( P ) under the good event, and
Revised: V ⇡ b ˆ c k ( P ) + ⌘ ⇡ b k ( P ) + ✏ ⇡ b k ( P ) , and since V ⇡ b c ( P )  V ⇡ b ˆ c k ( P )  ⌘ ⇡ b k ( P ) under the good event, and

Original: V ⇡ b c ( P ) =
Revised: V ⇡ b c ( P ) =

Original: ¯ C b , we will have V ⇡ b ¯ c k ( P )  ¯ C if 2 ⌘ ⇡ b k ( P )+ ✏ ⇡ b k ( P )  ( ¯ C  ¯ C b ) .
Revised: ¯ C b , we will have V ⇡ b ¯ c k ( P )  ¯ C if 2 ⌘ ⇡ b k ( P )+ ✏ ⇡ b k ( P )  ( ¯ C  ¯ C b ) .

Original: So, ( ⇡ b , P ) is a feasiblesolution for (10) if 2 ⌘ ⇡ b k ( P ) + ✏ ⇡ b k ( P )  ( ¯ C  ¯ C b ) .
Revised: So, ( ⇡ b , P ) is a feasible solution for (10) if 2 ⌘ ⇡ b k ( P ) + ✏ ⇡ b k ( P )  ( ¯ C  ¯ C b ) .

Original: This sufﬁcient condition may not be satisﬁedfor initial episodes.
Revised: This sufﬁcient condition may not be satisﬁed for initial episodes.

Original: However, since ✏ ⇡ b k ( P ) and ⌘ ⇡ b k ( P ) are decreasing in k , if ( ⇡ b , P ) becomes afeasible solution for (10) at episode k 0 , then it will remain feasible for all episodes k  k 0 .
Revised: However, since ✏ ⇡ b k ( P ) and ⌘ ⇡ b k ( P ) are decreasing in k , if ( ⇡ b , P ) becomes a feasible solution for (10) at episode k 0 , then it will remain feasible for all episodes k  k 0 .

Original: Also,since ¯  ph,k and  lh,k decrease with k , one can expect that (10) becomes feasible after some number ofepisodes.
Revised: Also, since ¯  ph,k and  lh,k decrease with k , one can expect that (10) becomes feasible after some number of episodes.

Original: We use these intuitions, along with some technical lemmas to show the following result.
Revised: We use these intuitions, along with some technical lemmas to show the following result.

Original: Proposition 4.
Revised: Proposition 4.

Original: Under the DOPE algorithm, with a probability greater that 1  5  , ( ⇡ b , P ) is afeasible solution for the OP problem (10) for all
Revised: Under the DOPE algorithm, with a probability greater that 1  5  , ( ⇡ b , P ) is a feasible solution for the DOP problem (10) for all

Original: k  K o , where K o =
Revised: k  K o , where K o =

Original: ˜
Revised: ˜

Original: O ( S 2 AH 4 ( ¯ C  ¯ C b ) 2 ) .
Revised: O ( S 2 AH 4 ( ¯ C  ¯ C b ) 2 ) .

Original: 4.3 Safety Exploration Guarantee
Revised: 4.3 Safety Exploration Guarantee

Original: We show that the DOPE algorithm provides a safe exploration guarantee, i.e., ⇡ k 2 ⇧ safe for allk 2 [ K ] with high probability, where ⇡ k is the exploration policy employed by DOPE in episode k .
Revised: We show that the DOPE algorithm provides a safe exploration guarantee, i.e., ⇡ k 2 ⇧ safe for all k 2 [ K ] with high probability, where ⇡ k is the exploration policy employed by DOPE in episode k .

Original: This is achieved by the carefully designed pessimistic constraint of the DOP problem (10).
Revised: This is achieved by the carefully designed pessimistic constraint of the DOP problem (10).

Original: For any k  K o , since ⇡ k = ⇡ b , and it is safe by Assumption 1.
Revised: For any k  K o , since ⇡ k = ⇡ b , and it is safe by Assumption 1.

Original: For k  K o , (10) is feasibleaccording to Proposition 4.
Revised: For k  K o , (10) is feasible according to Proposition 4.

Original: Since ( ⇡ k , P k ) is the solution of (10), we have V ⇡ k ¯ c k
Revised: Since ( ⇡ k , P k ) is the solution of (10), we have V ⇡ k ¯ c k

Original: ( P k )
Revised: ( P k )

Original: =
Revised: =

Original: V ⇡ k
Revised: V ⇡ k

Original: ˆ c
Revised: ˆ c

Original: k
Revised: k

Original: ( P k ) +⌘ ⇡ k k ( P k ) + ✏ ⇡ k k ( P k )
Revised: ( P k ) + ⌘ ⇡ k k ( P k ) + ✏ ⇡ k k ( P k )

Original:  ¯ C .
Revised:  ¯ C .

Original: This implies that V ⇡ k
Revised: This implies that V ⇡ k

Original: ˆ c k
Revised: ˆ c k

Original: ( P k ) + ⌘ ⇡ k k ( P k )  ¯ C  ✏ ⇡ k k
Revised: ( P k ) + ⌘ ⇡ k k ( P k )  ¯ C  ✏ ⇡ k k

Original: ( P k ) .
Revised: ( P k ) .

Original: We have that
Revised: We have that

Original: V ⇡ k c ( P k )  V ⇡ k ˆ c
Revised: V ⇡ k c ( P k )  V ⇡ k ˆ c

Original: k
Revised: k

Original: ( P k ) + ⌘ ⇡ k k ( P k ) under the good event, and hence, the above equation implies that
Revised: ( P k ) + ⌘ ⇡ k k ( P k ) under the good event, and hence, the above equation implies that

Original: V ⇡ k c ( P k )  ¯ C  ✏ ⇡ k k
Revised: V ⇡ k c ( P k )  ¯ C  ✏ ⇡ k k

Original: ( P k ) , i.e., ⇡ k satisﬁes a tighter constraint with respect to the model P k .
Revised: ( P k ) , i.e., ⇡ k satisﬁes a tighter constraint with respect to the model P k .

Original: However,it is not obvious that the policy ⇡ k will be safe with respect to the true model P because V ⇡ k c ( P ) maybe larger than V ⇡ k c ( P k ) due to the change from P k to P .
Revised: However, it is not obvious that the policy ⇡ k will be safe with respect to the true model P because V ⇡ k c ( P ) may be larger than V ⇡ k c ( P k ) due to the change from P k to P .

Original: We, however, show that V ⇡ k c ( P ) cannot be larger than V ⇡ k c ( P k ) by more than ✏ ⇡
Revised: We, however, show that V ⇡ k c ( P ) cannot be larger than V ⇡ k c ( P k ) by more than ✏ ⇡

Original: k k
Revised: k k

Original: ( P k ) , i.e., V ⇡ k c ( P ) 
Revised: ( P k ) , i.e., V ⇡ k c ( P )

Original: V ⇡ k c ( P k )  ✏ ⇡ k k ( P k ) .
Revised: V ⇡ k c ( P k )  ✏ ⇡ k k ( P k ) .

Original: This will thenimmediately yield that V ⇡ k c ( P )  V ⇡ k c ( P k ) + ✏ ⇡ k k ( P k )  ¯ C ,which is the true safety constraint.
Revised: This will then yield that V ⇡ k c ( P )  V ⇡ k c ( P k ) + ✏ ⇡ k k ( P k )  ¯ C , which is the true safety constraint.

Original: The key idea is in the design of the pessimistic cost function¯ c k ( · , · ) such that its pessimistic effect will balance the change in the value function (from V ⇡ k c
Revised: The key idea is in the design of the pessimistic cost function ¯ c k ( · , · ) such that its pessimistic effect will balance the change in the value function (from V ⇡ k c

Original: ( P k )to V ⇡ k c ( P ) )
Revised: ( P k ) to V ⇡ k c ( P ) )

Original: due to the optimistic selection of the model P k .
Revised: due to the optimistic selection of the model P k .

Original: We formally state the safety guaranteeof DOPE below.
Revised: We formally state the safety guarantee of DOPE below.

Original: Proposition 5.
Revised: Proposition 5.

Original: Let { ⇡ k , k 

Revised: Let { ⇡ k , k

Original: [ K ] } be the sequence of policies generated by the DOPE algorithm.
Revised: [ K ] } be the sequence of policies generated by the DOPE algorithm.

Original: Then ⇡ k is safe for all k 2{+
+} [ K ] , i.e., V ⇡ k c ( P )  ¯ C , for all k 

Revised: Then ⇡ k is safe 8 k {+
+} [ K ] , i.e., V ⇡ k c ( P )  ¯ C , for all k

Original: [ K ] , with a probability greater than1  5  .
Revised: [ K ] , with a probability greater than 1  5  .

Original: 4.4 Regret Analysis
Revised: 4.4 Regret Analysis

Original: The regret analysis for most OFU style RL algorithms follows the standard approach of decomposingthe regret into two terms as R k = V ⇡ k r ( P )  V ⇡ ⇤ r
Revised: The regret analysis for most OFU style RL algorithms follows the standard approach of decomposing the regret into two terms as R k = V ⇡ k r ( P )  V ⇡ ⇤ r

Original: ( P ) =
Revised: ( P ) =

Original: ( V ⇡ k r ( P )  V ⇡ k r ( P k ))
Revised: ( V ⇡ k r ( P )  V ⇡ k r ( P k ))

Original: + ( V ⇡ k r ( P k ) 
Revised: + ( V ⇡ k r ( P k )

Original: V ⇡ ⇤ r ( P )) , where R k denotes the regret in episode k .
Revised: V ⇡ ⇤ r ( P )) , where R k denotes the regret in episode k .

Original: The ﬁrst term is the difference betweenvalue functions of the selected policy ⇡ k with respect to the true model P and optimistic model
Revised: The ﬁrst term is the difference between value functions of the selected policy ⇡ k with respect to the true model P and optimistic model

Original: P k .
Revised: P k .

Original: Bounding this term is the key technical analysis part of most of the OFU algorithms for theunconstrained MDP [20, 12] and also the CMDP [17].
Revised: Bounding this term is the key technical analysis part of most of the OFU algorithms for the unconstrained MDP [21, 13] and also the CMDP [18].

Original: In the standard OFU style analysis for theunconstrained problem, since P 2 P k for all k , it can be easily observed that ( ⇡ ⇤ , P ) is a feasiblesolution for the OFU problem (7) for all k 2 [ K ] .
Revised: In the standard OFU style analysis for the unconstrained problem, since P 2 P k for all k , it can be easily observed that ( ⇡ ⇤ , P ) is a feasible solution for the OFU problem (7) for all k 2 [ K ] .

Original: Moreover, since ( ⇡ k , P k ) is the optimal solution ink th episode, we get V ⇡ k r ( P k )  V ⇡ ⇤ r
Revised: Moreover, since ( ⇡ k , P k ) is the optimal solution in k th episode, we get V ⇡ k r ( P k )  V ⇡ ⇤ r

Original: ( P ) .
Revised: ( P ) .

Original: So, the second term will be non-positive, and hence canbe dropped from the regret analysis.
Revised: So, the second term will be non-positive, and hence can be dropped from the regret analysis.

Original: However, in our setting, the second term can be positive since( ⇡ ⇤ , P ) may not be a feasible solution of the DOP problem (10) due to the pessimistic constraint.
Revised: However, in our setting, the second term can be positive since ( ⇡ ⇤ , P ) may not be a feasible solution of the DOP problem (10) due to the pessimistic constraint.

Original: This necessitates a different approach for bounding the regret.
Revised: This necessitates a different approach for bounding the regret.

Original: Existing work [26] only considers optimismin the objective cost, and hence their proof closely follows that of OptCMDP-Bonus algorithm in [17]with pessimistic constraints.
Revised: Existing work [28] only considers optimism in the objective cost, and hence their proof closely follows that of OptCMDP-Bonus algorithm in [18] with pessimistic constraints.

Original: In analyzing the regret of DOPE, we need to handle the optimism inobjective cost as well as the model in regret terms, along with the pessimistic constraints.
Revised: In analyzing the regret of DOPE, we need to handle the optimism in objective cost as well as the model in regret terms, along with the pessimistic constraints.

Original: This makethe analysis particularly challenging.
Revised: This make the analysis particularly challenging.

Original: The full proof is detailed in the appendix.
Revised: The full proof is detailed in the appendix.

Original:  Experiments
Revised:  Experiments

Original: We now evaluate DOPE via experiments.
Revised: We now evaluate DOPE via experiments.

Original: We have two relevant metrics, namely, (i) objective regret,deﬁned in (3) that measures the optimality gap of the algorithm, and (ii) constraint regret, deﬁnedas
Revised: We have two relevant metrics, namely, (i) objective regret, deﬁned in (3) that measures the optimality gap of the algorithm, and (ii) constraint regret, deﬁned as

Original: P
Revised: P

Original: Kk =1 max { 0 , V ⇡ k c ( P )  ¯ C } , where ⇡ k is the output of the algorithm in question at episode k.
Revised: [Equation] } , where ⇡ k is the output of the algorithm in question at episode k.

Original: This measures the safety gap of the algorithm.
Revised: This measures the safety gap of the algorithm.

Original: Our candidate algorithms are (i) OptCMDP, (ii)
Revised: Our candidate algorithms are (i) OptCMDP, (ii)

Original: AlwaysSafe, (iii) OptPessLP and (iv) DOPE, all described in the introduction.
Revised: AlwaysSafe, (iii) OptPessLP and (iv) DOPE, all described in the introduction.

Original: OptCMDP is expectedto show constraint violations, while the other three should show zero constraint regret.
Revised: OptCMDP is expected to show constraint violations, while the other three should show zero constraint regret.

Original: We considertwo environments here, with a third environment presented in the appendix.
Revised: We consider two environments here, with a third environment presented in the appendix.

Original: AlwaysSafe can directlybe used only with a factored CMDP, and only applies to the ﬁrst environment presented.
Revised: AlwaysSafe can directly be used only with a factored CMDP, and only applies to the ﬁrst environment presented.

Original: We presentboth variants of this algorithm [34].
Revised: We simulate both variants of this algorithm, referred to as AlwaysSafe ⇡ T and AlwaysSafe ⇡ ↵ , respectively [36].

Original: Factored CMDP:
Revised: Factored CMDP:

Original: We ﬁrst consider a CMDP where the safety relavant features of the model can beseparated, as shown in [34].
Revised: We ﬁrst consider a CMDP where the safety relavant features of the model can be separated, as shown in [36].

Original: This CMDP contains states { 1 , 2 , 3 } arranged in a circle, and allows 2actions { 1 , 2 } in each state, to move right or stay put, respectively.
Revised: This CMDP has states { 1 , 2 , 3 } arranged in a circle, and 2 actions { 1 , 2 } in each state, to move right or stay put, respectively.

Original: The transitions move the agentto its right state with probability 1 , if action 1 is taken.
Revised: The transitions move the agent to its right state with probability 1 , if action 1 is taken.

Original: If action 2 is taken, it remains in the samestate with probability 1 .
Revised: If action 2 is taken, it remains in the same state with probability 1 .

Original: Action 1 does not incur any objective cost or constraint cost.
Revised: Action 1 does not incur any objective cost or constraint cost.

Original: Taking action2 gives a objective cost equals to the state number, and a constraint cost of 1 .
Revised: Action 2 incurs an objective cost equals to the state number, and a constraint cost of 1 .

Original: We choose episodelength H = 6 , and constraint as ¯ C = 3 .
Revised: We choose episode length H = 6 , and constraint as ¯ C = 3 .

Original: The structure of this CMDP is such that AlwaysSafe is ableto extract a safe baseline policy from it.
Revised: The structure of this CMDP allows AlwaysSafe to extract a safe baseline policy from it.

Original: Media Streaming CMDP:
Revised: Media Streaming CMDP:

Original: Our second environment represents media streaming to a device froma wireless base station, which provides high and low service rates at different costs.
Revised: Our second environment represents media streaming to a device from a wireless base station, which provides high and low service rates at different costs.

Original: These servicerates have independent Bernoulli distributions, with parameters µ 1 = 0 .
Revised: These service rates have independent Bernoulli distributions, with parameters µ 1 = 0 .

Original: 9 , and µ 2 = 0 .
Revised: 9 , and µ 2 = 0 .

Original: 1 , where µ 1corresponds to the fast service.
Revised: 1 , where µcorresponds to the fast service.

Original: Packets received at the device are stored in a media buffer and playedout according to a Bernoulli process with parameter  .
Revised: Packets received at the device are stored in a media buffer and played out according to a Bernoulli process with parameter  .

Original: We denote the number of incoming packetsinto the buffer is denoted A h , and the number of packets leaving the buffer B h .
Revised: We denote the number of incoming packets into the buffer as A h , and the number of packets leaving the buffer B h .

Original: The media bufferlength is the state and evolves ass h +1 = min { max(0 , s h + A h  B h ) , N } , where N = 20 is the maximum buffer length.
Revised: The media buffer length is the state and evolves as s h +1 = min { max(0 , s h + A h  B h ) , N } , where N = 20 is the maximum buffer length.

Original: The action space is { 1 , 2 } , where action 1 corresponds317 to using the fast service.
Revised: The action space is { 1 , 2 } , where action 1 corresponds to using the fast service.

Original: The objective cost is r ( s, a ) =
Revised: The objective cost is r ( s, a ) =

Original: { s = 0 } , while the constraint cost is318 c ( s, a ) = { a = 1 } , i.e., we desire to minimize the outage cost, while limiting the usage of fastservice.
Revised: { s = 0 } , while the constraint cost is c ( s, a ) = { a = 1 } , i.e., we desire to minimize the outage cost, while limiting the usage of fast service.

Original: We consider a horizon H = 10 , and  ¯ C
Revised: We consider episode length H = 10 , and constraint ¯ C

Original: = H 2 .
Revised: = H 2 .

Original: Experiment Setup: OptCMDP and OptPessLP algorithms have not been implemented earlier.
Revised: Experiment Setup: OptCMDP and OptPessLP algorithms have not been implemented earlier.

Original: Foraccurate comparison, we simulate all the algorithms true to their original formulations of costfunctions and conﬁdence intervals.
Revised: For accurate comparison, we simulate all the algorithms true to their original formulations of cost functions and conﬁdence intervals.

Original: Our experiments are carried out for 20 random runs, and averagedto obtain the regret plots.
Revised: Our experiments are carried out for 20 random runs, and averaged to obtain the regret plots.

Original: For DOPE, we choose K 0 to be as speciﬁed in Proposition 4.
Revised: For DOPE, we choose K 0 to be as speciﬁed in Proposition 4.

Original: Full detailson the algorithm parameters and experiment settings are provided in Appendix E.
Revised: Full details on the algorithm parameters and experiment settings are provided in Appendix E.

Original: Baseline Policies: Both OptPessLP and DOPE require baseline policies.
Revised: Baseline Policies: Both OptPessLP and DOPE require baseline policies.

Original: We select the baselinepolicies as the optimal solutions of the given CMDP with a constraint ¯ C b = 0 .
Revised: We select the baseline policies as the optimal solutions of the given CMDP with a constraint ¯ C b = 0 .

Original: 2 ¯ C .
Revised: 2 ¯ C .

Revised: We choose the same baseline policies for both the algorithms.

Original: This choice is toshowcase the efﬁcacy of DOPE, despite starting with a conservative baseline policy.
Revised: This choice is to showcase the efﬁcacy of DOPE, despite starting with a conservative baseline policy.

Original: Results for Factored CMDP: Fig. 1(a) shows the objective regret of the algorithms in this environment.
Revised: Results for Factored CMDP: Fig. 1(a) shows the objective regret of the algorithms in this environment.

Original: OptCMDP has a good objective regret performance as expected, but shows constraintviolations.
Revised: OptCMDP has a good objective regret performance as expected, but shows constraint violations.

Original: AlwasySafe fails to achieve ˜ O
Revised: AlwasySafe fails to achieve ˜ O

Original: ( p K ) regret in the episodes shown, for both variants of thealgorithm, though the second variant has smaller regret  compared to the ﬁrst.
Revised: ( p K ) regret in the episodes shown for both variants, although the ⇡ ↵ variant has smaller regret as compared to the ⇡ T variant.

Original: OptPessLP takes a longtime to attain p K behavior, which means that is chooses the baseline policy for an extended period,and shows high empirical regret.
Revised: OptPessLP takes a long time to attain p K behavior, which means that is chooses the baseline policy for an extended period, and shows high empirical regret.

Original: This suggests that reward optimism of OptPessLP is insufﬁcientto balance the pessimism in the constraint.
Revised: This suggests that reward optimism of OptPessLP is insufﬁcient to balance the pessimism in the constraint.

Original: DOPE not only achieves the desired ˜ O
Revised: DOPE not only achieves the desired ˜ O

Original: ( p K ) regret, butalso does so in fewer episodes compared to the other two algorithms, despite using the same baselinepolicy as OptPessLP.
Revised: ( p K ) regret, but also does so in fewer episodes compared to the other two algorithms.

Original: Furthermore, it has low empirical regret.
Revised: Furthermore, it has low empirical regret.

Original: Fig. 1(b) shows that the constraintviolation regret is zero for all the episodes of learning for all the safe algorithms, while OptCMDPshows a large constraint violation regret.[Figure] [Figure]
Revised: Fig. 1(b) shows that the constraint violation regret is zero for all the episodes of learning for all the safe algorithms, while OptCMDP shows a large constraint violation regret.

Original: Results for Media Streaming CMDP: Fig.2(a) compares the objective regret across algorithms.
Revised: Results for Media Streaming CMDP: Fig.2(a) compares the objective regret across algorithms.

Original: The value of DOPE over OptPessLP is more apparent here, since, after a linear growth phase, theobjective regret of DOPE changes to a square-root scaling.
Revised: The value of DOPE over OptPessLP is more apparent here. After a linear growth phase, the objective regret of DOPE changes to a square-root scaling.

Original: OptPessLP has not explored sufﬁcientlyat this point, and hence suffers high linear regret.
Revised: OptPessLP has not explored sufﬁciently at this point, and hence suffers high linear regret.

Original: Finally, OptCMDP  hasthe same square-root regretscaling, but is fastest, since it is not constrained by safe exploration.
Revised: Finally, OptCMDP also has square-root regret scaling, but is fastest, since it is not constrained by safe exploration.

Original: Fig.2(b) compares the regretin constraint violation for these algorithms.
Revised: Fig.2(b) compares the regret in constraint violation for these algorithms.

Original: As expected, DOPE and OptPessLP do not violate theconstraint, while the OptCMDP algorithm signiﬁcantly violates the constraints during learning.
Revised: As expected, DOPE and OptPessLP do not violate the constraint, while the OptCMDP algorithm signiﬁcantly violates the constraints during learning.

Original: Effect of Baseline Policy: We compare the objective regret of DOPE under different baseline policiesin Fig.1(c) and Fig.2(c).
Revised: Effect of Baseline Policy: We compare the objective regret of DOPE under different baseline policies in Fig.1(c) and Fig.2(c).

Original: We see that less conservative (but safe) baselines result in lower regret, butthe difference is not excessive, implying that the exact baseline policy chosen is not crucial. 
Revised: We see that less conservative (but safe) baselines result in lower regret, but the difference is not excessive, implying that the exact baseline policy chosen is not crucial. [Figure]

Original:  Summary : DOPE has two valuable properties: ( i ) Faster rate of shift to p K behavior, since thelinear regret phase where it applies the baseline policy is relatively short, and ( ii )
Revised: [Figure] Summary : DOPE has two valuable properties: ( i ) Faster rate of shift to p K behavior, since the linear regret phase where it applies the baseline policy is relatively short, and ( ii )

Original: ˜ O
Revised: ˜ O

Original: ( p K ) regretwith respect to optimal, which together mean that the empirical regret is lower than other approaches.
Revised: ( p K ) regret with respect to optimal, which together mean that the empirical regret is lower than other approaches.

Original: Limitations:
Revised: Limitations:

Original: Our goal is to ﬁnd an RL algorithm with no constraint violation with high probabilityin the tabular setting.
Revised: Our goal is to ﬁnd an RL algorithm with no constraint violation with high probability in the tabular setting.

Original: Since safe exploration is a basic problem, we follow the usual approach in theliterature of ﬁrst establishing the fundamental theory results for the tabular setting [20, 13, 6].
Revised: Since safe exploration is a basic problem, we follow the usual approach in the literature of ﬁrst establishing the fundamental theory results for the tabular setting [21, 14, 6].

Original: Wealso note that most of the existing work on exploration in safe RL is in the tabular setting [17, 35, 26].
Revised: We also note that most of the existing work on exploration in safe RL is in the tabular setting [18, 37, 28].

Original: In our future work, we plan to employ the DOPE approach in a function approximation setting.
Revised: In our future work, we plan to employ the DOPE approach in a function approximation setting.

Original:  Conclusion
Revised:  Conclusion

Original: We considered the safe exploration problem in reinforcement learning, wherein a safety constraintmust be satisﬁed during learning and evaluation.
Revised: We considered the safe exploration problem in reinforcement learning, wherein a safety constraint must be satisﬁed during learning and evaluation.

Original: Earlier approaches to constrained RL have proposedoptimism on the model, optimism on reward, and pessimism on constraints as means of modulatingexploration, but none have shown order optimal regret, no safety violation, and good empiricalperformance simultaneously.
Revised: Earlier approaches to constrained RL have proposed optimism on the model, optimism on reward, and pessimism on constraints as means of modulating exploration, but none have shown order optimal regret, no safety violation, and good empirical performance simultaneously.

Original: We started with the conjecture that double optimism combined withpessimism is the key to attaining the ideal balance for fast and safe exploration, and design DOPE thatcarefully combines these elements.
Revised: We started with the conjecture that double optimism combined with pessimism is the key to attaining the ideal balance for fast and safe exploration, and design DOPE that carefully combines these elements.

Original: We showed that DOPE not only attains order-optimal ˜ O
Revised: We showed that DOPE not only attains order-optimal ˜ O

Original: ( p K )regret without violating safety constraints, but also reduces the best known regret bound by a factorof p |S| .
Revised: ( p K ) regret without violating safety constraints, but also reduces the best known regret bound by a factor of p |S| .

Original: Furthermore, it has signiﬁcantly better empirical performance than existing approaches.
Revised: Furthermore, it has signiﬁcantly better empirical performance than existing approaches.

Original: We367 thus make a case for adoption of the approach for real world use cases and extension to large scale
Revised: We thus make a case for adoption of the approach for real world use cases and extension to large scale

Original: RL problems using function approximation.
Revised: RL problems using function approximation.

Revised:  Acknowledgement

Revised: This work was supported in part by the grants NSF-CAREER-EPCN

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: In Theorem 3 and Section 5.
Revised: In Theorem 3 and Section 5.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: In Section 5.
Revised: In Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Societal Impact and Ethics Statement.
Revised: [Yes] Societal Impact and Ethics Statement.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] Described in Societal Impact and Ethics Statement.
Revised: [Yes] Described in Societal Impact and Ethics Statement.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Assumption 1,and assumptions in statement of Theorem 3.
Revised: [Yes] Assumption 1, and assumptions in statement of Theorem 3.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Appendix. 
Revised: [Yes] Appendix. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results?

Original: [Yes] Supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: [Yes] Supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Section 5, and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [Yes] Section 5, and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Figures contain error bars.
Revised: [Yes] Figures contain error bars.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Appendix E 
Revised: [Yes] Appendix E 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] Code is released as supplemtental material.
Revised: [Yes] Code is released as supplemtental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [N/A]
Revised: [N/A]