Original: DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
Revised: DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning

Original: Abstract
Revised: Abstract

Original: Safe reinforcement learning is extremely challenging–not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations.
Revised: Safe reinforcement learning is extremely challenging–not only must the agent explore an unknown environment, it must do so while ensuring no safety constraint violations.

Original: We formulate this safe reinforcement learning (RL) problem using the framework of a ﬁnite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function, where we model the safety requirements as constraints on the expected cumulative costs that must be satisﬁed during all episodes of learning.
Revised: We formulate this safe reinforcement learning (RL) problem using the framework of a ﬁnite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function, where we model the safety requirements as constraints on the expected cumulative costs that must be satisﬁed during all episodes of learning.

Original: We propose a model-based safe RL algorithm that we call
Revised: We propose a model-based safe RL algorithm that we call

Original: Doubly Optimistic and Pessimistic Exploration (DOPE), and show that it achieves an objective regret ˜ O ( |S| (cid:112) |A| K ) without violating the safety constraints during learning, where |S| is the number of states, |A| is the number of actions, and K is the number of learning episodes.
Revised: Doubly Optimistic and Pessimistic Exploration (DOPE), and show that it achieves an objective regret ˜ O ( |S| p |A| K ) without violating the safety constraints during learning, where |S| is the number of states, |A| is the number of actions, and K is the number of learning episodes.

Original: Our key idea is to combine a reward bonus for exploration (optimism) with a conservative constraint (pessimism), in addition to the standard optimistic model-based exploration.
Revised: Our key idea is to combine a reward bonus for exploration (optimism) with a conservative constraint (pessimism), in addition to the standard optimistic model-based exploration.

Original: DOPE is not only able to improve the objective regret bound, but also shows a signiﬁcant empirical performance improvement as compared to earlier optimism-pessimism approaches.
Revised: DOPE is not only able to improve the objective regret bound, but also shows a signiﬁcant empirical performance improvement as compared to earlier optimism-pessimism approaches.

Original:  Introduction
Revised:  Introduction

Original: Constrained Markov Decision Processes (CMDPs) impose restrictions that pertain to resource or safety constraints of the system.
Revised: Constrained Markov Decision Processes (CMDPs) impose restrictions that pertain to resource or safety constraints of the system.

Original: For example, the average radiated power in a wireless communication system must be restricted due to user health and battery lifetime considerations or the frequency of braking or accelerating in an autonomous vehicle must be kept bounded to ensure passenger comfort.
Revised: For example, the average radiated power in a wireless communication system must be restricted due to user health and battery lifetime considerations or the frequency of braking or accelerating in an autonomous vehicle must be kept bounded to ensure passenger comfort.

Original: Since these systems have complex dynamics, a constrained reinforcement learning (CRL) approach is attractive for determining an optimal control policy.
Revised: Since these systems have complex dynamics, a constrained reinforcement learning (CRL) approach is attractive for determining an optimal control policy.

Original: But how do we ensure that safety or resource availability constraints are not violated while learning such an optimal control policy?
Revised: But how do we ensure that safety or resource availability constraints are not violated while learning such an optimal control policy?

Original: Our goal is to develop a framework for safe exploration without constraint violation (with high probability) for solving CMDP problems where the model is unknown.
Revised: Our goal is to develop a framework for safe exploration without constraint violation (with high probability) for solving CMDP problems where the model is unknown.

Original: While there has been much work on RL for both the MDP and the CMDP setting, ensuring safe exploration in the CRL setting has received less attention.
Revised: While there has been much work on RL for both the MDP and the CMDP setting, ensuring safe exploration in the CRL setting has received less attention.

Original: The problem is challenging, since we do not allow constraint violation either during learning or deployment, while ensuring a low regret in terms of the optimal objective.
Revised: The problem is challenging, since we do not allow constraint violation either during learning or deployment, while ensuring a low regret in terms of the optimal objective.

Original: Our aim is to explore a model-based approach in an episodic setting under which the model (the transition kernel of the CMDP) is empirically determined as samples from the system are gathered.
Revised: Our aim is to explore a model-based approach in an episodic setting under which the model (the transition kernel of the CMDP) is empirically determined as samples from the system are gathered.

Original: There has been much recent interest in model-based RL approaches to solving a constrained MDP, the most relevant of which we have summarized in Table 1.
Revised: There has been much recent interest in model-based RL approaches to solving a constrained MDP, the most relevant of which we have summarized in Table 1.

Original: The setup is of a ﬁnite horizon episodic
Revised: The setup is of a ﬁnite horizon episodic

Original: CMDP, with a state space of size |S| , an action space of size |A| and a horizon of length H. Regret is measured over the ﬁrst K episodes of the algorithm.
Revised: CMDP, with a state space of size |S| , an action space of size |A| and a horizon of length H. Regret is measured over the ﬁrst K episodes of the algorithm.

Original: Both the attained objective and constraint satisfaction are computed in an expected sense for a given policy.
Revised: Both the attained objective and constraint satisfaction are computed in an expected sense for a given policy.

Original: Allowing constraint violations during learning means that the algorithm suffers both an objective regret and a constraint regret.
Revised: Allowing constraint violations during learning means that the algorithm suffers both an objective regret and a constraint regret.

Original: [Table]
Revised: [Table]

Original:  Examples of algorithms following this method are OptCMDP, OptCMDP-Bonus [18].
Revised: [Table] Examples of algorithms following this method are OptCMDP, OptCMDP-Bonus [18].

Original: The algorithms use different ways of incentivizing exploration to obtain samples to build up an empirical system model.
Revised: The algorithms use different ways of incentivizing exploration to obtain samples to build up an empirical system model.

Original: OptCMDP uses the idea of optimism in the face of uncertainty from the model perspective and solves an extended linear program to ﬁnd the best model (highest reward) under the samples gathered thus far.
Revised: OptCMDP uses the idea of optimism in the face of uncertainty from the model perspective and solves an extended linear program to ﬁnd the best model (highest reward) under the samples gathered thus far.

Original: OptCMDP-Bonus uses a different approach of adding a bonus to the reward term to incentivize exploration of state-action pairs that have fewer samples, which we consider as a form of optimism from the reward perspective.
Revised: OptCMDP-Bonus uses a different approach of adding a bonus to the reward term to incentivize exploration of state-action pairs that have fewer samples, which we consider as a form of optimism from the reward perspective.

Original: While the objective regret of both these algorithms is ˜ O ( √ K ) , the constraint regret too is of the same order, since constraints may be violated during learning.
Revised: While the objective regret of both these algorithms is ˜ O ( p K ) , the constraint regret too is of the same order, since constraints may be violated during learning.

Original: An algorithm that begins with no knowledge of the model will have exploration steps during learning that might violate constraints in expectation.
Revised: An algorithm that begins with no knowledge of the model will have exploration steps during learning that might violate constraints in expectation.

Original: Hence, safe RL approaches assume the availability of an inexpert baseline policy that does not violate the constraints, but is insufﬁcient to explore the entire state action space.
Revised: Hence, safe RL approaches assume the availability of an inexpert baseline policy that does not violate the constraints, but is insufﬁcient to explore the entire state action space.

Original: So it cannot be simply applied until a high-accuracy model is obtained.
Revised: So it cannot be simply applied until a high-accuracy model is obtained.

Original: A heuristic approach entitled AlwaysSafe [36] assumes a factored CMDP that allows the easy generation of a safe baseline policy.
Revised: A heuristic approach entitled AlwaysSafe [36] assumes a factored CMDP that allows the easy generation of a safe baseline policy.

Original: It then combines the optimism with respect to the model of OptCMDP with a heuristically chosen hardening of constraints.
Revised: It then combines the optimism with respect to the model of OptCMDP with a heuristically chosen hardening of constraints.

Original: This guarantees no constraint violations, but does not have regret guarantee with respect to the objective.
Revised: This guarantees no constraint violations, but does not have regret guarantee with respect to the objective.

Original: Its empirical performance is variable and its use is limited to factored CMDP problems.
Revised: Its empirical performance is variable and its use is limited to factored CMDP problems.

Original: The OptPessLP algorithm [28] formalizes the idea of coupling optimism and pessimism by starting with OptCMDPBonus that has an optimistic reward, and systematically applying decreasing levels of pessimism with respect to the constraint violations.
Revised: The OptPessLP algorithm [28] formalizes the idea of coupling optimism and pessimism by starting with OptCMDPBonus that has an optimistic reward, and systematically applying decreasing levels of pessimism with respect to the constraint violations.

Original: The approach is successful in ensuring the twin goals of a ˜ O ( √ K ) objective regret, while ensuring no constraint violations.
Revised: The approach is successful in ensuring the twin goals of a ˜ O ( p K ) objective regret, while ensuring no constraint violations.

Original: However, the authors do not present any empirical performance evaluation results.
Revised: However, the authors do not present any empirical performance evaluation results.

Original: When we implemented OptPessLP, we found that the performance is singularly bad in that linear regret persists for a large number of samples, and the tapering off to ˜ O ( √ K ) regret behavior does not appear to happen quickly.
Revised: When we implemented OptPessLP, we found that the performance is singularly bad in that linear regret persists for a large number of samples, and the tapering off to ˜ O ( p K ) regret behavior does not appear to happen quickly.

Original: The problem with this algorithm is that it is so pessimistic with regard to constraints that it heavily disincentivizes exploration and ends up choosing the base policy for long sequences.
Revised: The problem with this algorithm is that it is so pessimistic with regard to constraints that it heavily disincentivizes exploration and ends up choosing the base policy for long sequences.

Original: The issue upon which algorithm performance depends is the choice of how to combine optimism and pessimism to obtain both order optimal and empirically good objective regret performance, while ensuring no constraint violations happen.
Revised: The issue upon which algorithm performance depends is the choice of how to combine optimism and pessimism to obtain both order optimal and empirically good objective regret performance, while ensuring no constraint violations happen.

Original: Our insight is that optimism with respect to the model is a key enabler of exploration, and can be coupled with the addition of optimism with respect to the reward.
Revised: Our insight is that optimism with respect to the model is a key enabler of exploration, and can be coupled with the addition of optimism with respect to the reward.

Original: This double dose of optimism —both with respect to model and reward—could ensure that pessimistic hardening of constraints does not excessively retard exploration.
Revised: This double dose of optimism —both with respect to model and reward—could ensure that pessimistic hardening of constraints does not excessively retard exploration.

Original: Following this insight, we develop DOPE, a doubly optimistic and pessimistic exploration approach.
Revised: Following this insight, we develop DOPE, a doubly optimistic and pessimistic exploration approach.

Original: We are able to show that DOPE not only attains ˜ O ( √ K ) objective regret behavior with zero constraint regret with high probability, it also reduces the objective regret bound over OptPessLP by a factor of (cid:112) |S| .
Revised: We are able to show that DOPE not only attains ˜ O ( p K ) objective regret behavior with zero constraint regret with high probability, it also reduces the objective regret bound over OptPessLP by a factor of p |S| .

Original: We conduct performance analysis simulations under representative CMDP problems and show that DOPE easily outperforms all the earlier approaches.
Revised: We conduct performance analysis simulations under representative CMDP problems and show that DOPE easily outperforms all the earlier approaches.

Original: Thus, the idea of double optimism is not only valuable from the order optimal algorithm design perspective, it also shows good empirical regret performance, indicating the feasibility of utilizing the methodology in real-world systems.
Revised: Thus, the idea of double optimism is not only valuable from the order optimal algorithm design perspective, it also shows good empirical regret performance, indicating the feasibility of utilizing the methodology in real-world systems.

Original: The code for the experiments in this paper is located at: https://github.com/archanabura/
Revised: The code for the experiments in this paper is located at: https://github.com/archanabura/

Original: DOPE-DoublyOptimisticPessimisticExploration
Revised: DOPE-DoublyOptimisticPessimisticExploration

Original: Related Work: Constrained RL: Constrained Markov Decision Processes (CMDP) has been an active area of research [2], with applications in domains such as power systems [39, 25], communication networks [3, 38], and robotics [17, 11].
Revised: Related Work: Constrained RL: Constrained Markov Decision Processes (CMDP) has been an active area of research [2], with applications in domains such as power systems [39, 25], communication networks [3, 38], and robotics [17, 11].

Original: In [9], the author proposed an actor-critic RL algorithm for learning the asymptotically optimal policy for an inﬁnite horizon average cost CMDP when the model is unknown.
Revised: In [9], the author proposed an actor-critic RL algorithm for learning the asymptotically optimal policy for an inﬁnite horizon average cost CMDP when the model is unknown.

Original: This approach is also utilized in function approximation settings with asymptotic local convergence guarantees [8, 12, 40].
Revised: This approach is also utilized in function approximation settings with asymptotic local convergence guarantees [8, 12, 40].

Original: Policy gradient algorithms for CMDPs have also been developed [1, 44, 47, 35, 16, 27, 49].
Revised: Policy gradient algorithms for CMDPs have also been developed [1, 44, 47, 35, 16, 27, 49].

Original: However, none these works address the problem of safe exploration to provide guarantees on the constraint violations during learning.
Revised: However, none these works address the problem of safe exploration to provide guarantees on the constraint violations during learning.

Original: Safe Multi-Armed Bandits: The problem of safe exploration in linear bandits with stage-wise safety constraint is studied in [4, 24, 31, 33].
Revised: Safe Multi-Armed Bandits: The problem of safe exploration in linear bandits with stage-wise safety constraint is studied in [4, 24, 31, 33].

Original: Linear bandits with more general constraints have also been studied [34, 29].
Revised: Linear bandits with more general constraints have also been studied [34, 29].

Original: These works do not consider the more challenging RL setting which involves an underlying dynamical system with unknown model.
Revised: These works do not consider the more challenging RL setting which involves an underlying dynamical system with unknown model.

Original: Safe Online Convex Optimization: Online convex optimization [20] has been studied with stochastic constraints [45, 10] and adversarial constraints [32, 46, 26].
Revised: Safe Online Convex Optimization: Online convex optimization [20] has been studied with stochastic constraints [45, 10] and adversarial constraints [32, 46, 26].

Original: These allow constraint violation during learning and characterize the cumulative amount of violation.
Revised: These allow constraint violation during learning and characterize the cumulative amount of violation.

Original: A safe Frank-Wolf algorithm for convex optimization with unknown linear stochastic constraints has been studied in [41].
Revised: A safe Frank-Wolf algorithm for convex optimization with unknown linear stochastic constraints has been studied in [41].

Original: However, these too do not consider the RL setting with an unknown model.
Revised: However, these too do not consider the RL setting with an unknown model.

Original: Exploration in Constrained RL: There has been much work in this area with constraint violations during learning, including the work discussed in the introduction [18].
Revised: Exploration in Constrained RL: There has been much work in this area with constraint violations during learning, including the work discussed in the introduction [18].

Original: These include [37, 19, 23], which derive bounds either on the objective and constraint regret or on the sample complexity of learning an (cid:15) -optimal policy.
Revised: These include [37, 19, 23], which derive bounds either on the objective and constraint regret or on the sample complexity of learning an ✏ -optimal policy.

Original: Other works on safe RL include [43, 42], where a model-free approach is considered, and [5], that pertains to ofﬂine RL.
Revised: Other works on safe RL include [43, 42], where a model-free approach is considered, and [5], that pertains to ofﬂine RL.

Original: These are complementary to our model-based approach.
Revised: These are complementary to our model-based approach.

Original: The problem of learning the optimal policy of a CMDP without violating the constraints was also studied in [48].
Revised: The problem of learning the optimal policy of a CMDP without violating the constraints was also studied in [48].

Original: However, they assume that the model is known and only the cost functions are unknown, whereas we address more difﬁcult problem with unknown model and cost functions.
Revised: However, they assume that the model is known and only the cost functions are unknown, whereas we address more difﬁcult problem with unknown model and cost functions.

Original: Notations: For any integer M , [ M ] denotes the set { 1 , . .
Revised: Notations: For any integer M , [ M ] denotes the set { 1 , . .

Original: , M } .
Revised: , M } .

Original: For any two real numbers a, b , a ∨ b := max { a, b } .
Revised: For any two real numbers a, b , a _ b := max { a, b } .

Original: For any given set X , ∆( X ) denotes the probability simplex over the set X , and |X| denotes the cardinality of the set X .
Revised: For any given set X ,  ( X ) denotes the probability simplex over the set X , and |X| denotes the cardinality of the set X .

Original:  Preliminaries and Problem Formulation 2.1 Constrained Markov Decision Process
Revised:  Preliminaries and Problem Formulation 2.1 Constrained Markov Decision Process

Original: We address the safe exploration problem using the framework of episodic Constrained Markov
Revised: We address the safe exploration problem using the framework of episodic Constrained Markov

Original: Decision Process (CMDP) [2].
Revised: Decision Process (CMDP) [2].

Original: We consider a CMDP, denoted as M = (cid:104)S , A , r, c, P, H, ¯ C (cid:105) with r =
Revised: We consider a CMDP, denoted as M = h S , A , r, c, P, H, ¯ C i with r =

Original: ( r h )
Revised: ( r h )

Original: Hh =1 , c
Revised: Hh =1 , c

Original: = ( c h )
Revised: = ( c h )

Original: Hh =1 , P
Revised: Hh =1 , P

Original: =
Revised: =

Original: ( P h )
Revised: ( P h )

Original: Hh =1 , where S is the state space, A is the action space, H is the episode length, r h : S ×
Revised: Hh =1 , where S is the state space, A is the action space, H is the episode length, r h : S ⇥ A !

Original: A → R is the objective cost function at time step h ∈
Revised: R is the objective cost function at time step h 2 [ H ] , c

Original: [ H ] , c h : S ×
Revised: h : S ⇥ A !

Original: A → R is the constraint cost function at time step h ∈ [ H ] , P h is the transition probability function with
Revised: R is the constraint cost function at time step h 2 [ H ] , P h is the transition probability function with

Original: P h ( s (cid:48) | s, a ) representing the probability of transitioning to state s (cid:48) when action a is taken at state s at time h .
Revised: P h ( s 0 | s, a ) representing the probability of transitioning to state s 0 when action a is taken at state s at time h .

Original: In the RL context, the transition matrix P is also called the model of the CMDP.
Revised: In the RL context, the transition matrix P is also called the model of the CMDP.

Original: Finally, ¯
Revised: Finally, ¯

Original: C is a scalar that speciﬁes the safety constraint in terms of the maximum permissible value for the expected cumulative constraint cost.
Revised: C is a scalar that speciﬁes the safety constraint in terms of the maximum permissible value for the expected cumulative constraint cost.

Original: We consider the setting where |S| and |A| are ﬁnite.
Revised: We consider the setting where |S| and |A| are ﬁnite.

Original: Also, without loss of generality, we assume that costs r and c are bounded in [0 , 1] .
Revised: Also, without loss of generality, we assume that costs r and c are bounded in [0 , 1] .

Original: A non-stationary randomized policy π = ( π h )
Revised: A non-stationary randomized policy ⇡ = ( ⇡ h )

Original: Hh =1 , π h : S → ∆( A ) speciﬁes the control action to be taken at each time step {+h 
+}
Revised: Hh =1 , ⇡ h : S !  ( A ) speciﬁes the control action to be taken at each time step {+h 
+}

Original: h ∈ [ H ] .
Revised: [ H ] .

Original: In particular, π h ( s, a ) denotes the probability of taking action a when the state is s at time step h .
Revised: In particular, ⇡ h ( s, a ) denotes the probability of taking action a when the state is s at time step h .

Original: For an arbitrary cost function l : [ H ] × S ×
Revised: For an arbitrary cost function l : [ H ] ⇥ S ⇥ A !

Original: A → R , the value function of a policy π corresponding to time step h ∈ [ H ] given a state s ∈ S is deﬁned as
Revised: R , the value function of a policy ⇡ corresponding to time step h 2 [ H ] given a state s 2 S is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where a τ ∼ π τ ( s τ , · ) , s τ +1 ∼ P τ ( ·| s τ , a τ ) .
Revised: (1) where a ⌧ ⇠ ⇡ ⌧ ( s ⌧ , · ) , s ⌧ +1 ⇠ P ⌧ ( ·| s ⌧ , a ⌧ ) .

Original: Since we are mainly interested in the value of a policy starting from h = 1 , we simply denote V πl, 1 ( s ; P ) as V πl ( s ; P ) .
Revised: Since we are mainly interested in the value of a policy starting from h = 1 , we simply denote V ⇡ l, 1 ( s ; P ) as V ⇡ l ( s ; P ) .

Original: For the rest of the paper, we will assume that the initial state is s 1 is ﬁxed.
Revised: For the rest of the paper, we will assume that the initial state is s 1 is ﬁxed.

Original: So, we will simply denote V πl ( s 1 ; P ) as V πl ( P ) , when it is clear from the context.
Revised: So, we will simply denote V ⇡ l ( s 1 ; P ) as V ⇡ l ( P ) , when it is clear from the context.

Original: This standard assumption [18, 15] can be made without loss of generality.
Revised: This standard assumption [18, 15] can be made without loss of generality.

Original: The CMDP (planning) problem with a known model P can then be stated as follows:
Revised: The CMDP (planning) problem with a known model P can then be stated as follows:

Original: [Equation]
Revised: [Equation]

Original: We say that a policy π is a safe policy if V πc ( P ) ≤ ¯ C , i.e., if the expected cumulative constraint cost corresponding to the policy π is less than the maximum permissible value ¯ C .
Revised: We say that a policy ⇡ is a safe policy if V ⇡ c ( P )  ¯ C , i.e., if the expected cumulative constraint cost corresponding to the policy ⇡ is less than the maximum permissible value ¯ C .

Original: The set of safe policies , denoted as Π safe , is deﬁned as Π safe = { π : V πc ( P ) ≤ ¯ C } .
Revised: The set of safe policies , denoted as ⇧ safe , is deﬁned as ⇧ safe = { ⇡ : V ⇡ c ( P )  ¯ C } .

Original: Without loss of generality, we assume that the CMDP problem (2) is feasible, i.e., Π safe is non-empty.
Revised: Without loss of generality, we assume that the CMDP problem (2) is feasible, i.e., ⇧ safe is non-empty.

Original: Let π ∗ be the optimal safe policy , which is the solution of (2).
Revised: Let ⇡ ⇤ be the optimal safe policy , which is the solution of (2).

Original: The CMDP (planning) problem is signiﬁcantly different from the standard Markov Decision Process (MDP) (planning) problem [2].
Revised: The CMDP (planning) problem is signiﬁcantly different from the standard Markov Decision Process (MDP) (planning) problem [2].

Original: Firstly, there may not exist an optimal deterministic policy for a
Revised: Firstly, there may not exist an optimal deterministic policy for a

Original: CMDP, whereas the existence of a deterministic optimal policy is well known for a standard MDP.
Revised: CMDP, whereas the existence of a deterministic optimal policy is well known for a standard MDP.

Original: Secondly, there does not exist a Bellman optimality principle or Bellman equation for CMDP.
Revised: Secondly, there does not exist a Bellman optimality principle or Bellman equation for CMDP.

Original: So, the standard dynamic programming solution approaches which rely on the Bellman equation cannot be directly applied to solve the CMDP problem.
Revised: So, the standard dynamic programming solution approaches which rely on the Bellman equation cannot be directly applied to solve the CMDP problem.

Original: There are two standard approaches for solving the CMDP problem, namely the Lagrangian approach and the linear programming (LP) approach.
Revised: There are two standard approaches for solving the CMDP problem, namely the Lagrangian approach and the linear programming (LP) approach.

Original: Both approaches exploit the zero duality gap property of the CMDP problem [2] to ﬁnd the optimal policy.
Revised: Both approaches exploit the zero duality gap property of the CMDP problem [2] to ﬁnd the optimal policy.

Original: In this work, we will use the LP approach, consistent with model optimism.
Revised: In this work, we will use the LP approach, consistent with model optimism.

Original: Details of solving (2) using the LP approach are in Appendix A.
Revised: Details of solving (2) using the LP approach are in Appendix A.

Original: 2.2 Reinforcement Learning with Safe Exploration
Revised: 2.2 Reinforcement Learning with Safe Exploration

Original: The goal of the reinforcement learning with safe exploration is to solve (2), but without the knowledge of the model P a priori.
Revised: The goal of the reinforcement learning with safe exploration is to solve (2), but without the knowledge of the model P a priori.

Original: Hence, the learning algorithm has to perform exploration by employing different policies to learn P .
Revised: Hence, the learning algorithm has to perform exploration by employing different policies to learn P .

Original: However, we also want the exploration for learning to have a safety guarantee, i.e, the policies employed during learning should belong to the set of safe policies Π safe .
Revised: However, we also want the exploration for learning to have a safety guarantee, i.e, the policies employed during learning should belong to the set of safe policies ⇧ safe .

Original: Since Π safe itself is deﬁned based on the unknown P , the learning algorithm will not know Π safe a priori.
Revised: Since ⇧ safe itself is deﬁned based on the unknown P , the learning algorithm will not know ⇧ safe a priori.

Original: This makes the safe exploration problem extremely challenging.
Revised: This makes the safe exploration problem extremely challenging.

Original: We consider a model-based RL algorithm that interacts with the environment in an episodic manner.
Revised: We consider a model-based RL algorithm that interacts with the environment in an episodic manner.

Original: Let π k
Revised: Let ⇡ k

Original: =
Revised: =

Original: ( π h,k ) Hh =1 be the policy employed by the algorithm in episode k .
Revised: ( ⇡ h,k ) Hh =1 be the policy employed by the algorithm in episode k .

Original: At each time step h ∈ [ H ] in an episode k , the algorithm observes state s h,k , selects action a h,k ∼ π h,k
Revised: At each time step h 2 [ H ] in an episode k , the algorithm observes state s h,k , selects action a h,k ⇠ ⇡ h,k

Original: ( s h,k , · ) , and incurs the costs r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) .
Revised: ( s h,k , · ) , and incurs the costs r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) .

Original: The next state s h
Revised: The next state s h

Original: +1 ,k is realized according to the probability vector P ( ·| s h,k , a h,k ) .
Revised: +1 ,k is realized according to the probability vector P ( ·| s h,k , a h,k ) .

Original: As stated before, for simplicity, we assume that the initial state is ﬁxed for each episodek ∈
Revised: As stated before, for simplicity, we assume that the initial state is ﬁxed for each episode

Original:  [ K ] := { 1 , . . .
Revised: k 2 [ K ] := { 1 , . . .

Original: , K } , i.e., s 1 ,k
Revised: , K } , i.e., s 1 ,k

Original: = s 1 .
Revised: = s 1 .

Original: We also assume that the maximum permissible cost ¯ C for any exploration policy is known and it is speciﬁed as part of the learning problem.
Revised: We also assume that the maximum permissible cost ¯ C for any exploration policy is known and it is speciﬁed as part of the learning problem.

Original: The performance of the RL algorithm is measured using the metric of safe objective regret .
Revised: The performance of the RL algorithm is measured using the metric of safe objective regret .

Original: The safe objective regret is deﬁned exactly as the standard regret of an RL algorithm for exploration in MDPs [21, 14, 6], but with an additional constraint that the exploration polices should belong to the safe set Π safe .
Revised: The safe objective regret is deﬁned exactly as the standard regret of an RL algorithm for exploration in MDPs [21, 14, 6], but with an additional constraint that the exploration polices should belong to the safe set ⇧ safe .

Original: Formally, the safe objective regret R ( K ) after K learning episodes is deﬁned as
Revised: Formally, the safe objective regret R ( K ) after K learning episodes is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Since Π safe is unknown, clearly it is not possible to employ a safe policy without making any additional assumptions.
Revised: Since ⇧ safe is unknown, clearly it is not possible to employ a safe policy without making any additional assumptions.

Original: We overcome this obvious limitation by assuming that the algorithm has access to a safe baseline policy π b such that π b ∈ Π safe .
Revised: We overcome this obvious limitation by assuming that the algorithm has access to a safe baseline policy ⇡ b such that ⇡ b 2 ⇧ safe .

Original: We formalize this assumption as follows.
Revised: We formalize this assumption as follows.

Original: Assumption 1 (Safe baseline policy) .
Revised: Assumption 1 (Safe baseline policy) .

Original: The algorithm knows a safe baseline policy π b such that
Revised: The algorithm knows a safe baseline policy ⇡ b such that

Original: V π b c ( P ) =
Revised: V ⇡ b c ( P ) =

Original: ¯ C b , where ¯ C b < ¯ C .
Revised: ¯ C b , where ¯ C b < ¯ C .

Original: The value ¯ C b is also known to the algorithm.
Revised: The value ¯ C b is also known to the algorithm.

Original: Remark 2 .
Revised: Remark 2 .

Original: Knowing a safe policy π b is necessary for solving the safe RL problem because we require the constraint to always be satisﬁed.
Revised: Knowing a safe policy ⇡ b is necessary for solving the safe RL problem because we require the constraint to always be satisﬁed.

Original: A similar assumption has been used in the case of safe exploration in linear bandits [4, 24, 33], as well as in earlier work on safe RL [36, 28].
Revised: A similar assumption has been used in the case of safe exploration in linear bandits [4, 24, 33], as well as in earlier work on safe RL [36, 28].

Original:  Algorithm and Performance Guarantee
Revised:  Algorithm and Performance Guarantee

Original: DOPE builds on the optimism in the face of uncertainty (OFU) style exploration algorithms for RL [21, 14], using such optimism, both in terms of the model, as well as to provide a reward bonus for under-explored state-actions.
Revised: DOPE builds on the optimism in the face of uncertainty (OFU) style exploration algorithms for RL [21, 14], using such optimism, both in terms of the model, as well as to provide a reward bonus for under-explored state-actions.

Original: However, a naive OFU-style algorithm may lead to selecting exploration policies that are not in the safe set Π safe .
Revised: However, a naive OFU-style algorithm may lead to selecting exploration policies that are not in the safe set ⇧ safe .

Original: So we modify the selection of exploratory policy by incorporating pessimism in the face of uncertainty (PFU) on the constraints, making DOPE doubly optimistic and pessimistic in exploration.
Revised: So we modify the selection of exploratory policy by incorporating pessimism in the face of uncertainty (PFU) on the constraints, making DOPE doubly optimistic and pessimistic in exploration.

Original: DOPE operates in episodes, each of length H .
Revised: DOPE operates in episodes, each of length H .

Original: Deﬁne the ﬁltration F k as the sigma algebra generated by the observations until the end of episode k ∈{+
+}
Revised: Deﬁne the ﬁltration F k as the sigma algebra generated by the observations until the end of episode k {+
+}

Original: [ K ] , i.e., F k = ( s h,k (cid:48) , a h,k (cid:48) , h ∈{+
+} [ H ] , k (cid:48) ∈{+0 
+}
Revised: [ K ] , i.e., F k = ( s h,k 0 , a h,k 0 , h {+
+} [ H ] , k {+0 
+}

Original: [ k ]) .
Revised: [ k ]) .

Original: Let n h,k
Revised: Let n h,k

Original: ( s, a ) =(cid:80)
Revised: ( s, a ) =

Original:  k − 1 k (cid:48){+
+}
Revised: P k  1 k {+
+}

Original: =1 1 { s h,k (cid:48){+
+} =
Revised: = 1 { s h,k {+
+} =

Original: s, a h,k (cid:48){+
+}
Revised: s, a h,k {+
+}

Original: = a } be the number of times the pair ( s, a ) was observed at time step h until the beginning of episode k .
Revised: = a } be the number of times the pair ( s, a ) was observed at time step h until the beginning of episode k .

Original: Similarly, deﬁne n h,k
Revised: Similarly, deﬁne n h,k

Original: ( s, a, s (cid:48) ) = (cid:80) k − 1 k (cid:48){+
+}
Revised: ( s, a, s 0 ) = P k  1 k {+
+}

Original: =1 1 { s h,k (cid:48){+
+} =
Revised: = 1 { s h,k {+
+} =

Original: s, a h,k (cid:48){+
+}
Revised: s, a h,k {+
+}

Original: = a, s h +1 ,k (cid:48){+
+}
Revised: = a, s h +1 ,k {+
+}

Original: = s (cid:48) } .
Revised: = s 0 } .

Original: At the beginning of each episode k , DOPE estimates the model as (cid:98) P h,k
Revised: At the beginning of each episode k , DOPE estimates the model as b

Original:  ( s (cid:48) | s, a ) =
Revised: P h,k ( s 0 | s, a ) =

Original: n h,k ( s, a, s (cid:48) ) /
Revised: n h,k ( s, a, s 0 ) /

Original: ( n h,k ( s, a ) ∨ 1) .
Revised: ( n h,k ( s, a ) _ 1) .

Original: Similar to OFU-style algorithms, we construct a conﬁdence set P k around (cid:98) P k as P k = ∩ ( s,a ) ∈S×A P k ( s, a ) , whereP k
Revised: Similar to OFU-style algorithms, we construct a conﬁdence set P k around b P k as P k = \ ( s,a ) 2 S ⇥ A P k ( s, a ) , where

Original: [Equation]
Revised: [Equation]

Original: where L = log( 2 SAHKδ ) , and Var ( (cid:98) P h,k ( s(cid:48) | s, a ))
Revised: where L = log( 2 SAHK  ) , and Var ( b P h,k [Equation] s

Original:  = (cid:98) Ph,k ( s (cid:48) | s, a )(1 − (cid:98) P h,k ( s (cid:48) | s, a )) .
Revised: [Equation] = [Equation] P

Original: Using the empirical Bernstein inequality, we can show that the true model P is an element of P k for any k ∈
Revised: [Equation] ,k [Equation] s

Original: [ K ] with probability at least 1 − 2 δ (see Appendix C).
Revised: [Equation] | [Equation] [Equation]

Original: Similarly, at the beginning of each episode k , DOPE estimates the unknown objective and constraint costs as ˆ r h,
Revised: [Equation] P h,k ( s

Original: [Equation]
Revised: [Equation]

Original:  s, 
Revised: | s, a )) . Using the empirical Bernstein inequality, we can show that the true model P is an element of P k for any k 2 [ K ] with probability at least 1  2  (see Appendix C). Similarly, at the beginning of each episode k , DOPE estimates the unknown objective and constraint costs as ˆ r h,k

Original: [Equation]
Revised: [Equation]

Original: In keeping with OFU, we construct conﬁdence sets R k and C k around ˆ r k and ˆ c k respectively, as
Revised: In keeping with OFU, we construct conﬁdence sets R k and C k around ˆ r k and ˆ c k respectively, as

Original: [Equation]
Revised: [Equation]

Original: (6) where L (cid:48) = 2 log(6 SAHK/δ ) , and ˜ r
Revised: where L 0 = 2 log(6 SAHK/  ) , and ˜ r

Original: = (˜ r h )
Revised: = (˜ r h )

Original: Hh =1 ,
Revised: Hh =1 ,

Original: ˜ c
Revised: ˜ c

Original: = (˜ c h )
Revised: = (˜ c h )

Original: Hh
Revised: Hh

Original: =1 .
Revised: =1 .

Original: Using Hoeffding inequality, we can show that the true costs belong to R k and C k for any k ∈ [ K ] with probability at least 1 − δ{+
+}
Revised: Using Hoeffding inequality, we can show that the true costs belong to R k and C k for any k {+
+}

Original:  (see Appendix C).
Revised: [ K ] with probability at least 1   (see Appendix C).

Original: We deﬁne M k = P k ∩ R k ∩ C k to be the total conﬁdence ball.
Revised: We deﬁne [Equation] k to be the total conﬁdence ball.

Original: It is tempting to use the standard OFU approach for selecting the exploration polices since this approach is known to provide sharp regret guarantees for exploration problems in RL.
Revised: It is tempting to use the standard OFU approach for selecting the exploration polices since this approach is known to provide sharp regret guarantees for exploration problems in RL.

Original: The standard
Revised: The standard

Original: OFU approach will ﬁnd the optimistic model P k and optimistic policy π k , where
Revised: OFU approach will ﬁnd the optimistic model P k and optimistic policy ⇡ k , where

Original: [Equation]
Revised: [Equation]

Original: The OFU problem (7) is feasible since the true model M is an element of M k (with high probability).
Revised: The OFU problem (7) is feasible since the true model M is an element of M k (with high probability).

Original: In particular, ( π b , P ) and ( π ∗ , P ) are feasible solutions of (7).
Revised: In particular, ( ⇡ b , P ) and ( ⇡ ⇤ , P ) are feasible solutions of (7).

Original: Moreover, (7) can be solved efﬁciently using an extended linear programming approach, as described in Appendix B.
Revised: Moreover, (7) can be solved efﬁciently using an extended linear programming approach, as described in Appendix B.

Original: The policy π k ensures exploration while satisfying the constraint V π k c ( P k ) ≤ ¯ C .
Revised: The policy ⇡ k ensures exploration while satisfying the constraint V ⇡ k c ( P k )  ¯ C .

Original: However, this naive OFU approach overlooks the important issue that π k may not be a safe policy with respect to the true model P .
Revised: However, this naive OFU approach overlooks the important issue that ⇡ k may not be a safe policy with respect to the true model P .

Original: More precisely, it is possible to have V π k c ( P ) >
Revised: More precisely, it is possible to have V ⇡ k c ( P ) >

Original: ¯ C even though V π k c ( P k ) ≤ ¯ C .
Revised: ¯ C even though V ⇡ k c ( P k )  ¯ C .

Original: So, the standard OFU approach alone will not give a safe exploration strategy.
Revised: So, the standard OFU approach alone will not give a safe exploration strategy.

Original: In order to ensure that the exploration policy employed at any episode is safe, we add a pessimistic penalty to the empirical constraint cost to get the pessimistic constraint cost function ¯ c k as
Revised: In order to ensure that the exploration policy employed at any episode is safe, we add a pessimistic penalty to the empirical constraint cost to get the pessimistic constraint cost function ¯ c k as

Original: [Equation][Equation]
Revised: [Equation]

Original: where ¯ β ph,k ( s, a ) =
Revised: where ¯  ph,k ( s, a ) =

Original: (cid:80) s (cid:48) ∈S β ph,k ( s, a, s (cid:48) ) .
Revised: P s [Equation] S  ph,k ( s, a, s 0 ) .

Original: Since ¯ β ph,k ( s, a ) is ˜ O
Revised: Since ¯  ph,k ( s, a ) is ˜ O

Original: (1 / (cid:112) n h,k ( s, a )) , ( s, a ) pairs that are less observed have a higher penalty, disincentivizing their exploration.
Revised: (1 / p n h,k ( s, a )) , ( s, a ) pairs that are less observed have a higher penalty, disincentivizing their exploration.

Original: However.
Revised: However.

Original: such a pessimistic penalty may prevent the exploration that is necessary to learn the optimal policy.
Revised: such a pessimistic penalty may prevent the exploration that is necessary to learn the optimal policy.

Original: To overcome this issue, we also modify the empirical objective cost function by subtracting a term to incentivize exploration, to obtain an optimistic objective cost function
Revised: To overcome this issue, we also modify the empirical objective cost function by subtracting a term to incentivize exploration, to obtain an optimistic objective cost function

Original: [Equation]
Revised: [Equation]

Original: Since ¯ β ph,k ( s, a ) is ˜ O 
Revised: Since ¯  ph,k ( s, a ) is ˜ O [Equation] [Equation]

Revised: ,k

Original: (1 / (cid:112) n h,k ( s, a )) , ( s, a ) pairs that are less observed will have a lowered cost to incentivize their exploration.
Revised: [Equation] s, a )) , [Equation] s, a ) pairs that are less observed will have a lowered cost to incentivize their exploration.

Original: We select the policy π k for episode k by solving the Doubly Optimistic-Pessimistic (DOP) problem:
Revised: We select the policy ⇡ k for episode k by solving the Doubly Optimistic-Pessimistic (DOP) problem:

Original: [Equation]
Revised: [Equation]

Original: Notice that DOPE is doubly optimistic by considering both the optimistic objective cost function in (9) and the optimistic model P k from the conﬁdence set P k in (10), while being pessimistic on the constraint in (10).
Revised: Notice that DOPE is doubly optimistic by considering both the optimistic objective cost function in (9) and the optimistic model P k from the conﬁdence set P k in (10), while being pessimistic on the constraint in (10).

Original: Later, in Lemma 18 in the appendix, we prove that ( π k , P k ) is indeed an optimistic solution.
Revised: Later, in Lemma 18 in the appendix, we prove that ( ⇡ k , P k ) is indeed an optimistic solution.

Original: This is in contrast with [28], where the optimism is solely limited to the objective cost.
Revised: This is in contrast with [28], where the optimism is solely limited to the objective cost.

Original: We will show that our approach carefully balances double optimism and pessimism, yielding a regret minimizing learning algorithm with episodic safe exploration guarantees.
Revised: We will show that our approach carefully balances double optimism and pessimism, yielding a regret minimizing learning algorithm with episodic safe exploration guarantees.

Original: We note that the DOP problem (10) may not be feasible, especially in the ﬁrst few episodes of learning.
Revised: We note that the DOP problem (10) may not be feasible, especially in the ﬁrst few episodes of learning.

Original: This is because, ¯ β ph,k ( s, a ) and β lh,k ( s, a ) may be large during the initial phase of learning so that there may not be a policy π (cid:48) and a model P (cid:48) ∈ P k that can satisfy the constraint V π (cid:48){+⇡ 
 ¯ c k+}
Revised: This is because, ¯  ph,k ( s, a ) and  lh,k ( s, a ) may be large during the initial phase of learning so that there may not be a policy ⇡ 0 and a model P 0 2 P k that can satisfy the constraint V {+⇡ 
 ¯ c k+}

Original: ¯ c k ( P (cid:48) ) ≤ ¯ C .
Revised: ( P 0 )  ¯ C .

Original: We overcome this issue by employing a safe baseline policy π b (as deﬁned in Assumption 1) in the ﬁrst K o episodes, a value provided by Proposition 4.
Revised: We overcome this issue by employing a safe baseline policy ⇡ b (as deﬁned in Assumption 1) in the ﬁrst K o episodes, a value provided by Proposition 4.

Original: Since π b is safe by deﬁnition, DOPE ensures safety during the ﬁrst K o episodes.
Revised: Since ⇡ b is safe by deﬁnition, DOPE ensures safety during the ﬁrst K o episodes.

Original: We will later show that the DOP problem (10) will have a feasible solution after the ﬁrst K o episodes (see Proposition 4).
Revised: We will later show that the DOP problem (10) will have a feasible solution after the ﬁrst K o episodes (see Proposition 4).

Original: For any episode k ≥ K o , DOPE employs policy π k , which is the solution of (10).
Revised: For any episode k  K o , DOPE employs policy ⇡ k , which is the solution of (10).

Original: We will also show that π k from (10) (once it becomes feasible) will indeed be a safe policy (see Proposition 5).
Revised: We will also show that ⇡ k from (10) (once it becomes feasible) will indeed be a safe policy (see Proposition 5).

Original: We present DOPE formally in Algorithm 1.Algorithm
Revised: We present DOPE formally in Algorithm 1.

Original: 4) - (5), and (6).
Revised: conﬁdence set M k according to (4) - (5), and (6).

Original: 5: if k ≤ K o then6:
Revised: : if k  K o then

Original: Select the exploration policy π k = π b 7: else 8:
Revised: Select the exploration policy ⇡ k = ⇡ b 7: else 8:

Original: Select the exploration policy π k according to (10) 9: end if 10: for h = 1 , 2 , . . .
Revised: Select the exploration policy ⇡ k according to (10) 9: end if 10: for h = 1 , 2 , . . .

Original: , H do 11:
Revised: , H do 11:

Original: Observe state s h,k , select action a h,k ∼ π h,k
Revised: Observe state s h,k , select action a h,k ⇠ ⇡ h,k

Original: ( s h,k , · ) , incur the cost r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) , and observe next state s h
Revised: ( s h,k , · ) , incur the cost r h ( s h,k , a h,k ) and c h ( s h,k , a h,k ) , and observe next state s h

Original: +1 ,k ∼ P h ( ·| s h,k , a h,k ) 12:
Revised: +1 ,k ⇠ P h ( ·| s h,k , a h,k ) 12:

Original: Update the counts: n h,k
Revised: Update the counts: n h,k

Original: ( s h,k , a h,k )← n h,k
Revised: ( s h,k , a h,k ) n h,k

Original: ( s h,k , a h,k ) + 1 , n h,k ( s h,k , a h,k , s h +1 ,k )← n h,k
Revised: ( s h,k , a h,k ) + 1 , n h,k ( s h,k , a h,k , s h +1 ,k ) n h,k

Original: ( s h,k , a h,k , s h
Revised: ( s h,k , a h,k , s h

Original: +1 ,k ) + 1 13: end for 14: end for
Revised: +1 ,k ) +13: end for 14: end for

Original: We now present our main result, which shows that the DOPE algorithm achieves ˜ O ( √ K ) regret without violating the safety constraints during learning, with high probability.
Revised: We now present our main result, which shows that the DOPE algorithm achieves ˜ O ( p K ) regret without violating the safety constraints during learning, with high probability.

Original: Theorem 3.
Revised: Theorem 3.

Original: Fix any δ ∈ (0 , 1) .
Revised: Fix any  2 (0 , 1) .

Original: Consider the DOPE algorithm with K o as speciﬁed in Proposition 4.
Revised: Consider the DOPE algorithm with K o as speciﬁed in Proposition 4.

Original: Let { π k , k ∈{+
+} [ K ] } be the sequence of policies generated by the DOPE algorithm.
Revised: Let { ⇡ k , k {+
+} [ K ] } be the sequence of policies generated by the DOPE algorithm.

Original: Then, with probability at least 1 − 5 δ , π k ∈ Π safe for all k ∈ [ K ] .
Revised: Then, with probability at least 1  5  , ⇡ k 2 ⇧ safe for all k 2 [ K ] .

Original: Moreover, with probability at least 1 − 5 δ , the regret of the DOPE algorithm satisﬁes
Revised: Moreover, with probability at least 1  5  , the regret of the DOPE algorithm satisﬁes

Original: [Equation]
Revised: [Equation]

Original:  Analysis
Revised:  Analysis

Original: We now provide the technical analysis of DOPE, concluding with the proof outline of Theorem 3.
Revised: We now provide the technical analysis of DOPE, concluding with the proof outline of Theorem 3.

Original: 4.1 Preliminaries
Revised: 4.1 Preliminaries

Original: For an arbitrary policy π (cid:48) and transition probability function P (cid:48) , deﬁne (cid:15) π (cid:48) k ( P (cid:48) ) andη π (cid:48) k ( P (cid:48) ) as (cid:15) π (cid:48)
Revised: For an arbitrary policy ⇡ 0 and transition probability function P 0 , deﬁne ✏ ⇡ 0 k ( P 0 ) and

Original: [Equation] ( P [Equation] =
Revised: ⌘ ⇡ 0 k ( P 0 ) as ✏ ⇡

Original: [Equation]
Revised: [Equation]

Original: E [Equation] Then, it is straightforward to show that (see (23) - (24) in the Appendix) V π (cid:48) ¯ c k ( P (cid:48) ) =
Revised: Then, it is straightforward to show that (see (23) - (24) in the Appendix) V ⇡ 0 ¯ c k ( P 0 ) =

Original: V π (cid:48){+⇡ 
+}
Revised: V {+⇡ 
+}

Original: ˆ ck ( P (cid:48) ) + η π (cid:48)
Revised: ˆ c

Original: k
Revised: k

Original: ( P  ) + (cid:15) π (cid:48) k ( P(cid:48) ) , V π (cid:48){++ ✏ ⇡ 
+}
Revised: ( P 0 ) + ⌘ ⇡ 0 k ( P ) {++ ✏ ⇡ 
+}

Original: ¯ r k ( P (cid:48) )
Revised: k ( P 0 )

Original: = V π (cid:48)
Revised: , V ⇡ 0 ¯ r k ( P 0 ) =

Original:  ˆ r k
Revised: V ⇡ 0 ˆ r k

Original: ( P (cid:48) )
Revised: ( P 0 )

Original: − 3 H ¯ C − ¯ C b η π (cid:48) k ( P (cid:48) ) −
Revised:  3 H ¯ C  ¯ C b ⌘ ⇡ 0 k ( P 0 )

Original: H ¯ C − ¯ C b (cid:15) π (cid:48) k ( P (cid:48) ) .
Revised: H ¯ C  ¯ C b ✏ ⇡ 0 k ( P 0 ) .

Original: The analysis utilizes this decomposition of V π (cid:48)
Revised: The analysis utilizes this decomposition of V ⇡ 0 ¯ c k

Original: ¯ c k ( P (cid:48) ) and V π (cid:48) ¯ r k
Revised: ( P 0 ) and V ⇡ 0 ¯ r k

Original: ( P (cid:48) ) , and the properties of (cid:15) π (cid:48) k ( P (cid:48) ) and η π (cid:48) k ( P (cid:48) ) .
Revised: ( P 0 ) , and the properties of ✏ ⇡ 0 k ( P 0 ) and ⌘ ⇡ 0 k ( P 0 ) .

Original: Under the good event G deﬁned as in Lemma 12, we can show, V π (cid:48) ˆ c k ( P (cid:48) ){+⇡ 
+}
Revised: Under the good event G deﬁned as in Lemma 12, we can show, V {+⇡ 
+}

Original: − η π
Revised: ˆ c k

Original: [Equation]
Revised: ( P 0 )  ⌘ ⇡ 0 k ( P 0 )  V ⇡ 0 c

Original:  P [Equation] ≤ V [Equation]{+⇡ 
+}
Revised: ( P 0 )  V {+⇡ 
+}

Original:  c[Equation]
Revised: ˆ c

Original: P
Revised: k

Original: [Equation] ) ≤
Revised: ( P 0 )+ ⌘ ⇡ 0 k ( P 0 ) .

Original: [Equation] 4.2 Feasibility of the OP Problem
Revised: 4.2 Feasibility of the OP Problem

Original: Even though ( π b , P ) is a feasible solution to the original CMDP problem (2), it may not be a feasible for the DOP problem (10) in the initial phase of learning.
Revised: Even though ( ⇡ b , P ) is a feasible solution to the original CMDP problem (2), it may not be a feasible for the DOP problem (10) in the initial phase of learning.

Original: To see this, note that V π b ¯ c k ( P ) =
Revised: To see this, note that V ⇡ b ¯ c k ( P ) =

Original: V π b ˆ c k ( P ) + η π b k ( P ) + (cid:15) π b k ( P ) , and since V π b c ( P ) ≥ V π b ˆ c k ( P ) − η π b k ( P ) under the good event, and
Revised: V ⇡ b ˆ c k ( P ) + ⌘ ⇡ b k ( P ) + ✏ ⇡ b k ( P ) , and since V ⇡ b c ( P )  V ⇡ b ˆ c k ( P )  ⌘ ⇡ b k ( P ) under the good event, and

Original: V π b c ( P ) =
Revised: V ⇡ b c ( P ) =

Original: ¯ C b , we will have V π b ¯ c k ( P ) ≤ ¯ C if 2 η π b k ( P )+ (cid:15) π b k ( P ) ≤ ( ¯ C − ¯ C b ) .
Revised: ¯ C b , we will have V ⇡ b ¯ c k ( P )  ¯ C if 2 ⌘ ⇡ b k ( P )+ ✏ ⇡ b k ( P )  ( ¯ C  ¯ C b ) .

Original: So, ( π b , P ) is a feasible solution for (10) if 2 η π b k ( P ) + (cid:15) π b k ( P ) ≤ ( ¯ C − ¯ C b ) .
Revised: So, ( ⇡ b , P ) is a feasible solution for (10) if 2 ⌘ ⇡ b k ( P ) + ✏ ⇡ b k ( P )  ( ¯ C  ¯ C b ) .

Original: This sufﬁcient condition may not be satisﬁed for initial episodes.
Revised: This sufﬁcient condition may not be satisﬁed for initial episodes.

Original: However, since (cid:15) π b k ( P ) and η π b k ( P ) are decreasing in k , if ( π b , P ) becomes a feasible solution for (10) at episode k (cid:48) , then it will remain feasible for all episodes k ≥ k (cid:48) .
Revised: However, since ✏ ⇡ b k ( P ) and ⌘ ⇡ b k ( P ) are decreasing in k , if ( ⇡ b , P ) becomes a feasible solution for (10) at episode k 0 , then it will remain feasible for all episodes k  k 0 .

Original: Also, since ¯ β ph,k and β lh,k decrease with k , one can expect that (10) becomes feasible after some number of episodes.
Revised: Also, since ¯  ph,k and  lh,k decrease with k , one can expect that (10) becomes feasible after some number of episodes.

Original: We use these intuitions, along with some technical lemmas to show the following result.
Revised: We use these intuitions, along with some technical lemmas to show the following result.

Original: Proposition 4.
Revised: Proposition 4.

Original: Under the DOPE algorithm, with a probability greater that 1 − 5 δ , ( π b , P ) is a feasible solution for the DOP problem (10) for all k ≥ K o , where K o =
Revised: Under the DOPE algorithm, with a probability greater that 1  5  , ( ⇡ b , P ) is a feasible solution for the DOP problem (10) for all k  K o , where K o =

Original: ˜
Revised: ˜

Original: O ( S 2 AH 4 ( ¯ C − ¯ C b ) 2 ) .
Revised: O ( S 2 AH 4 ( ¯ C  ¯ C b ) 2 ) .

Original: 4.3 Safety Exploration Guarantee
Revised: 4.3 Safety Exploration Guarantee

Original: We show that the DOPE algorithm provides a safe exploration guarantee, i.e., π k ∈ Π safe for all k ∈ [ K ] with high probability, where π k is the exploration policy employed by DOPE in episode k .
Revised: We show that the DOPE algorithm provides a safe exploration guarantee, i.e., ⇡ k 2 ⇧ safe for all k 2 [ K ] with high probability, where ⇡ k is the exploration policy employed by DOPE in episode k .

Original: This is achieved by the carefully designed pessimistic constraint of the DOP problem (10).
Revised: This is achieved by the carefully designed pessimistic constraint of the DOP problem (10).

Original: For any k ≤ K o , since π k = π b , and it is safe by Assumption 1.
Revised: For any k  K o , since ⇡ k = ⇡ b , and it is safe by Assumption 1.

Original: For k ≥ K o , (10) is feasible according to Proposition 4.
Revised: For k  K o , (10) is feasible according to Proposition 4.

Original: Since ( π k , P k ) is the solution of (10), we have V π k ¯ c k
Revised: Since ( ⇡ k , P k ) is the solution of (10), we have V ⇡ k ¯ c k

Original: ( P k )
Revised: ( P k )

Original: =
Revised: =

Original: V π k ˆ c
Revised: V ⇡ k ˆ c

Original: k
Revised: k

Original: ( P k ) + η π k k ( P k ) + (cid:15)
Revised: ( P k ) + ⌘ ⇡ k k ( P k ) + ✏ ⇡ k k ( P k )

Original: π k k ( P k ) ≤ ¯ C .
Revised:  ¯ C .

Original: This implies that V π k ˆ c k( P k ) +
Revised: This implies that V ⇡ k

Original: η π k k
Revised: ˆ c k

Original: ( P k ) ≤ ¯ C − (cid:15) π k k
Revised: ( P k ) + ⌘ ⇡ k k ( P k )  ¯ C  ✏ ⇡ k k

Original: ( P k ) .
Revised: ( P k ) .

Original: We have that
Revised: We have that

Original: V π k c ( P k ) ≤ V π k ˆ ck ( P k ) +
Revised: V ⇡ k c ( P k )  V ⇡ k ˆ c

Original: η π k k
Revised: k

Original: ( P k )  under the good event, and hence, the above equation implies that
Revised: ( P k ) + ⌘ ⇡ k k ( P k ) under the good event, and hence, the above equation implies that

Original: V π k c ( P k ) ≤ ¯ C − (cid:15)
Revised: V ⇡ k c ( P k )  ¯ C  ✏ ⇡ k k

Original: π k k ( P k ) , i.e., π k satisﬁes a tighter constraint with respect to the model P k .
Revised: ( P k ) , i.e., ⇡ k satisﬁes a tighter constraint with respect to the model P k .

Original: However, it is not obvious that the policy π k will be safe with respect to the true model P because V π k c ( P ) may be larger than V π k c ( P k ) due to the change from P k to P .
Revised: However, it is not obvious that the policy ⇡ k will be safe with respect to the true model P because V ⇡ k c ( P ) may be larger than V ⇡ k c ( P k ) due to the change from P k to P .

Original: We, however, show that V π k c ( P ) cannot be larger than V π k c ( P k ) by more than (cid:15) π k k
Revised: We, however, show that V ⇡ k c ( P ) cannot be larger than V ⇡ k c ( P k ) by more than ✏ ⇡

Original:  ( P k ) , i.e., V π k c ( P ) −
Revised: k k ( P k ) , i.e., V ⇡ k c ( P )

Original: V π k c ( P k ) ≤ (cid:15) π k k ( P k ) .
Revised: V ⇡ k c ( P k )  ✏ ⇡ k k ( P k ) .

Original: This will then yield that V π k c ( P ) ≤ V π k c ( P k ) + (cid:15) π k k ( P k ) ≤ ¯ C , which is the true safety constraint.
Revised: This will then yield that V ⇡ k c ( P )  V ⇡ k c ( P k ) + ✏ ⇡ k k ( P k )  ¯ C , which is the true safety constraint.

Original: The key idea is in the design of the pessimistic cost function ¯ c k ( · , · ) such that its pessimistic effect will balance the change in the value function (from V π k c ( P k ) to V π k c ( P ) ) due to the optimistic selection of the model P k .
Revised: The key idea is in the design of the pessimistic cost function ¯ c k ( · , · ) such that its pessimistic effect will balance the change in the value function (from V ⇡ k c ( P k ) to V ⇡ k c ( P ) ) due to the optimistic selection of the model P k .

Original: We formally state the safety guarantee of DOPE below.
Revised: We formally state the safety guarantee of DOPE below.

Original: Proposition 5.
Revised: Proposition 5.

Original: Let { π k , k ∈{+
+} [ K ] } be the sequence of policies generated by the DOPE algorithm.
Revised: Let { ⇡ k , k {+
+} [ K ] } be the sequence of policies generated by the DOPE algorithm.

Original: Then π k is safe ∀ k ∈{+
+}
Revised: Then ⇡ k is safe 8 k {+
+}

Original: [ K ] , i.e., V π k c ( P ) ≤ ¯ C , for all k ∈{+
+}
Revised: [ K ] , i.e., V ⇡ k c ( P )  ¯ C , for all k {+
+}

Original: [ K ] , with a probability greater than 1 − 5 δ .
Revised: [ K ] , with a probability greater than 1  5  .

Original: 4.4 Regret Analysis
Revised: 4.4 Regret Analysis

Original: The regret analysis for most OFU style RL algorithms follows the standard approach of decomposing the regret into two terms as R k 
Revised: The regret analysis for most OFU style RL algorithms follows the standard approach of decomposing the regret into two terms as R k = V ⇡ k r ( P )  V ⇡ ⇤ r

Original: = V π k r ( P ) − V π ∗ r ( P ) = ( V π k r ( P ) − V π k r ( P k ))
Revised: ( P ) = ( V ⇡ k r ( P )  V ⇡ k r ( P k ))

Original: + ( V π k r ( P k ) −
Revised: + ( V ⇡ k r ( P k )

Original: V π ∗ r ( P )) , where R k denotes the regret in episode k .
Revised: V ⇡ ⇤ r ( P )) , where R k denotes the regret in episode k .

Original: The ﬁrst term is the difference between value functions of the selected policy π k with respect to the true model P and optimistic model
Revised: The ﬁrst term is the difference between value functions of the selected policy ⇡ k with respect to the true model P and optimistic model

Original: P k .
Revised: P k .

Original: Bounding this term is the key technical analysis part of most of the OFU algorithms for the unconstrained MDP [21, 13] and also the CMDP [18].
Revised: Bounding this term is the key technical analysis part of most of the OFU algorithms for the unconstrained MDP [21, 13] and also the CMDP [18].

Original: In the standard OFU style analysis for the unconstrained problem, since P ∈ P k for all k , it can be easily observed that ( π ∗ , P ) is a feasible solution for the OFU problem (7) for all k ∈ [ K ] .
Revised: In the standard OFU style analysis for the unconstrained problem, since P 2 P k for all k , it can be easily observed that ( ⇡ ⇤ , P ) is a feasible solution for the OFU problem (7) for all k 2 [ K ] .

Original: Moreover, since ( π k , P k ) is the optimal solution in k th episode, we get V π k r 
Revised: Moreover, since ( ⇡ k , P k ) is the optimal solution in k th episode, we get V ⇡ k r ( P k )  V ⇡ ⇤ r

Original: ( Pk ) ≤ V π ∗ r ( P ) .
Revised: ( P ) .

Original: So, the second term will be non-positive, and hence can be dropped from the regret analysis.
Revised: So, the second term will be non-positive, and hence can be dropped from the regret analysis.

Original: However, in our setting, the second term can be positive since ( π ∗ , P ) may not be a feasible solution of the DOP problem (10) due to the pessimistic constraint.
Revised: However, in our setting, the second term can be positive since ( ⇡ ⇤ , P ) may not be a feasible solution of the DOP problem (10) due to the pessimistic constraint.

Original: This necessitates a different approach for bounding the regret.
Revised: This necessitates a different approach for bounding the regret.

Original: Existing work [28] only considers optimism in the objective cost, and hence their proof closely follows that of OptCMDP-Bonus algorithm in [18] with pessimistic constraints.
Revised: Existing work [28] only considers optimism in the objective cost, and hence their proof closely follows that of OptCMDP-Bonus algorithm in [18] with pessimistic constraints.

Original: In analyzing the regret of DOPE, we need to handle the optimism in objective cost as well as the model in regret terms, along with the pessimistic constraints.
Revised: In analyzing the regret of DOPE, we need to handle the optimism in objective cost as well as the model in regret terms, along with the pessimistic constraints.

Original: This make the analysis particularly challenging.
Revised: This make the analysis particularly challenging.

Original: The full proof is detailed in the appendix.
Revised: The full proof is detailed in the appendix.

Original:  Experiments
Revised:  Experiments

Original: We now evaluate DOPE via experiments.
Revised: We now evaluate DOPE via experiments.

Original: We have two relevant metrics, namely, (i) objective regret, deﬁned in (3) that measures the optimality gap of the algorithm, and (ii) constraint regret, deﬁned as(cid:80) Kk =1 max { 0 , V π k c ( P ) − ¯ C } , where π k is the output of the algorithm in question at episode k.
Revised: We have two relevant metrics, namely, (i) objective regret, deﬁned in (3) that measures the optimality gap of the algorithm, and (ii) constraint regret, deﬁned as P [Equation] } , where ⇡ k is the output of the algorithm in question at episode k.

Original: This measures the safety gap of the algorithm.
Revised: This measures the safety gap of the algorithm.

Original: Our candidate algorithms are (i) OptCMDP, (ii)
Revised: Our candidate algorithms are (i) OptCMDP, (ii)

Original: AlwaysSafe, (iii) OptPessLP and (iv) DOPE, all described in the introduction.
Revised: AlwaysSafe, (iii) OptPessLP and (iv) DOPE, all described in the introduction.

Original: OptCMDP is expected to show constraint violations, while the other three should show zero constraint regret.
Revised: OptCMDP is expected to show constraint violations, while the other three should show zero constraint regret.

Original: We consider two environments here, with a third environment presented in the appendix.
Revised: We consider two environments here, with a third environment presented in the appendix.

Original: AlwaysSafe can directly be used only with a factored CMDP, and only applies to the ﬁrst environment presented.
Revised: AlwaysSafe can directly be used only with a factored CMDP, and only applies to the ﬁrst environment presented.

Original: We simulate both variants of this algorithm, referred to as AlwaysSafe π T and AlwaysSafe π α , respectively [36].
Revised: We simulate both variants of this algorithm, referred to as AlwaysSafe ⇡ T and AlwaysSafe ⇡ ↵ , respectively [36].

Original: Factored CMDP:
Revised: Factored CMDP:

Original: We ﬁrst consider a CMDP where the safety relavant features of the model can be separated, as shown in [36].
Revised: We ﬁrst consider a CMDP where the safety relavant features of the model can be separated, as shown in [36].

Original: This CMDP has states { 1 , 2 , 3 } arranged in a circle, and 2 actions { 1 , 2 } in each state, to move right or stay put, respectively.
Revised: This CMDP has states { 1 , 2 , 3 } arranged in a circle, and 2 actions { 1 , 2 } in each state, to move right or stay put, respectively.

Original: The transitions move the agent to its right state with probability 1 , if action 1 is taken.
Revised: The transitions move the agent to its right state with probability 1 , if action 1 is taken.

Original: If action 2 is taken, it remains in the same state with probability 1 .
Revised: If action 2 is taken, it remains in the same state with probability 1 .

Original: Action 1 does not incur any objective cost or constraint cost.
Revised: Action 1 does not incur any objective cost or constraint cost.

Original: Action 2 incurs an objective cost equals to the state number, and a constraint cost of 1 .
Revised: Action 2 incurs an objective cost equals to the state number, and a constraint cost of 1 .

Original: We choose episode length
Revised: We choose episode length

Original: H = 6 , and constraint as ¯ C = 3 .
Revised: H = 6 , and constraint as ¯ C = 3 .

Original: The structure of this CMDP allows AlwaysSafe to extract a safe baseline policy from it.
Revised: The structure of this CMDP allows AlwaysSafe to extract a safe baseline policy from it.

Original: Media Streaming CMDP:
Revised: Media Streaming CMDP:

Original: Our second environment represents media streaming to a device from a wireless base station, which provides high and low service rates at different costs.
Revised: Our second environment represents media streaming to a device from a wireless base station, which provides high and low service rates at different costs.

Original: These service rates have independent Bernoulli distributions, with parameters µ 1 = 0 .
Revised: These service rates have independent Bernoulli distributions, with parameters µ 1 = 0 .

Original: 9 , and µ 2 = 0 .
Revised: 9 , and µ 2 = 0 .

Original: 1 , where µcorresponds to the fast service.
Revised: 1 , where µcorresponds to the fast service.

Original: Packets received at the device are stored in a media buffer and played out according to a Bernoulli process with parameter γ.
Revised: Packets received at the device are stored in a media buffer and played out according to a Bernoulli process with parameter  .

Original: We denote the number of incoming packets into the buffer as A h , and the number of packets leaving the buffer B h .
Revised: We denote the number of incoming packets into the buffer as A h , and the number of packets leaving the buffer B h .

Original: The media buffer length is the state and evolves as s h +1 = min { max(0 , s h + A h − B h ) , N } , where N = 20 is the maximum buffer length.The action space is { 1 , 2 } , where action 1 corresponds to using the fast service. The objective cost is r ( s, a ) = 1 { s = 0 } , while the constraint cost is c ( s, a )
Revised: The media buffer length is the state and evolves as s h +1 = min { max(0 , s h + A h  B h ) , N } , where N = 20 is the maximum buffer length.

Original: = 1 {a = 1  } , i.e., we desire to minimize the outage cost, while limiting theusage of fast service.We consider episode length H = 10 , and constraint ¯ C = H 2 .
Revised: The action space is { 1 , 2 } , where action 1 corresponds to using the fast service.

Original: Experiment Setup: OptCMDP and OptPessLP algorithms have not been implemented earlier.

Original: For accurate comparison, we simulate all the algorithms true to their original formulations of cost functions and conﬁdence intervals.

Original: Our experiments are carried out for 20 random runs, and averaged to obtain the regret plots.

Original: For DOPE, we choose K 0 to be as speciﬁed in Proposition 4.

Original: Full details on the algorithm parameters and experiment settings are provided in Appendix E.

Original: Baseline Policies: Both OptPessLP and DOPE require baseline policies.

Original: We select the baseline policies as the optimal solutions of the given CMDP with a constraint ¯ C b = 0 .

Original: 2 ¯ C .

Original: We choose the same baseline policies for both the algorithms.

Original: This choice is to showcase the efﬁcacy of DOPE, despite starting with a conservative baseline policy.

Original: Results for Factored CMDP: Fig. 1(a) shows the objective regret of the algorithms in this environment.

Original: OptCMDP has a good objective regret performance as expected, but shows constraint violations.

Original: AlwasySafe fails to achieve ˜ O ( √ K ) regret in the episodes shown for both variants, although the π α variant has smaller regret as compared to the π T variant.

Original: OptPessLP takes a long time to attain √

Original: K behavior, which means that is chooses the baseline policy for an extended period, and shows high empirical regret.

Original: This suggests that reward optimism of OptPessLP is insufﬁcient to balance the pessimism in the constraint.

Original: DOPE not only achieves the desired ˜ O ( √ K ) regret, but also does so in fewer episodes compared to the other two algorithms.

Original: Furthermore, it has low empirical regret.

Original: Fig. 1(b) shows that the constraint violation regret is zero for all the episodes of learning for all the safe algorithms, while OptCMDP shows a large constraint violation regret.

Original: Results for Media Streaming CMDP: Fig.2(a) compares the objective regret across algorithms.

Original: The value of DOPE over OptPessLP is more apparent here.

Original: After a linear growth phase, the objective regret of DOPE changes to a square-root scaling.

Original: OptPessLP has not explored sufﬁciently at this point, and hence suffers high linear regret.

Original: Finally, OptCMDP also has square-root regret scaling, but is fastest, since it is not constrained by safe exploration.

Original: Fig.2(b) compares the regret in constraint violation for these algorithms.

Original: As expected, DOPE and OptPessLP do not violate the constraint, while the OptCMDP algorithm signiﬁcantly violates the constraints during learning.

Original: Effect of Baseline Policy: We compare the objective regret of DOPE under different baseline policies in Fig.1(c) and Fig.2(c).

Original: We see that less conservative (but safe) baselines result in lower regret, but the difference is not excessive, implying that the exact baseline policy chosen is not crucial.

Original: [Figure]

Original: [Figure]

Original: Summary : DOPE has two valuable properties: ( i ) Faster rate of shift to √ K behavior, since the linear regret phase where it applies the baseline policy is relatively short, and ( ii ) ˜ O ( √ K ) regret with respect to optimal, which together mean that the empirical regret is lower than other approaches.

Original: Limitations:

Original: Our goal is to ﬁnd an RL algorithm with no constraint violation with high probability in the tabular setting.

Original: Since safe exploration is a basic problem, we follow the usual approach in the literature of ﬁrst establishing the fundamental theory results for the tabular setting [21, 14, 6].

Original: We also note that most of the existing work on exploration in safe RL is in the tabular setting [18, 37, 28].

Original: In our future work, we plan to employ the DOPE approach in a function approximation setting.

Original: Conclusion

Original: We considered the safe exploration problem in reinforcement learning, wherein a safety constraint must be satisﬁed during learning and evaluation.

Original: Earlier approaches to constrained RL have proposed optimism on the model, optimism on reward, and pessimism on constraints as means of modulating exploration, but none have shown order optimal regret, no safety violation, and good empirical performance simultaneously.

Original: We started with the conjecture that double optimism combined with pessimism is the key to attaining the ideal balance for fast and safe exploration, and design DOPE that carefully combines these elements.

Original: We showed that DOPE not only attains order-optimal ˜ O ( √ K ) regret without violating safety constraints, but also reduces the best known regret bound by a factor of (cid:112) |S| .

Original: Furthermore, it has signiﬁcantly better empirical performance than existing approaches.

Original: We thus make a case for adoption of the approach for real world use cases and extension to large scale

Original: RL problems using function approximation.

Original: Acknowledgement

Original: This work was supported in part by the grants NSF-CAREER-EPCN 2045783, NSF ECCS 2038963, and ARO W911NF-19-1-0367.

Original: Any opinions, ﬁndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the sponsoring agencies.

Original: Societal Impact and Ethics Statement

Original: Reinforcement learning has much potential for application to a variety of cyber-physical systems, such as the power grid, robotics and other systems where guarantees on the operating region of the system must be met.

Original: Our work provides a theoretical basis for the design of controllers that can be applied in such scenarios.

Original: The approaches presented in the paper were tested on simulated environments, and did not involve any human interaction.

Original: We do not see any ethical concerns with our research approach.

Original: A note of caution with our approach is that the policy generated is only as good as the training environment, and many examples exist wherein the policy generated is optimal according to its training, but violates basic truths known to human operators and could fail quite badly.

Original: Indeed, our approach does not provide sample-path guarantees, and the system could well move into deleterious states for a small fraction of the time, which might be completely unacceptable and trigger hard fail safes, such as breakers in a power system.

Original: Understanding the right application environments with excellent domain knowledge is hence needed before any practical success can be claimed.

Original: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes]

Original: In Theorem 3 and Section 5.

Original: (b) Did you describe the limitations of your work?

Original: [Yes]

Original: In Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Societal Impact and Ethics Statement.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] Described in Societal Impact and Ethics Statement.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Assumption 1, and assumptions in statement of Theorem 3.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Appendix. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results?

Original: [Yes] Supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Section 5, and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] Figures contain error bars.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes]

Original: Appendix

Original: E 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] Code is released as supplemtental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [N/A]

Original: A Linear Programming Method for Solving the CMDP Problem

Original: Here we give a brief description on solving the CMDP problem (2) using the linear programming method when the model P is known.

Original: The details can be found in [18, Section 2].

Original: The ﬁrst step is to reformulate (2) using occupancy measure [2, 50].

Original: For a given policy π and an initial state s 1 , the state-action occupation measure w π for the MDP with model P is deﬁned as

Original: [Equation]

Original: Given the occupancy measure, the policy that generated it can easily be computed as

Original: [Equation]

Original: [Equation]

Original: The occupancy measure of any policy π for an MDP with model P should satisfy the following conditions.

Original: We omit the explicit dependence on π and P from the notation of w for simplicity.

Original: [Equation]

Original: From the above conditions, it is easy to show that (cid:80) s,a w h ( s, a )

Original: = 1 .

Original: So, occupancy measures are indeed probability measures.

Original: Since the set of occupancy measures for a model P , denoted as W ( P ) , is deﬁned by a set of afﬁne constraints, it is straight forward to show that W ( P ) is convex.

Original: We state this fact formally below.

Original: Proposition 6.

Original: The set of occupancy measures for an MDP with model P , denoted as W ( P ) , is convex.

Original: Recall that the value of a policy π for an arbitrary cost function l : S ×

Original: A → R with a given initial state s 1 is deﬁned as V πl ( P ) =

Original: E [ (cid:80)

Original: Hh =1 l ( s h , a h ) | s 1 = s, π, P ] .

Original: It can then be expressed using the occupancy measure as

Original: [Equation]

Original: where w π ( P ) ∈ R SAH with ( s, a, h ) element is given by w πh ( s, a ; P ) and l ∈ R SAH with ( s, a, h ) element is given by l h ( s, a ) .

Original: The CMDP problem (2) can then be written as

Original: [Equation]

Original: Using the properties of the occupancy measures, the reformulated CMDP problem (16) can be rewritten as an LP, where the optimization variables are occupancy measures [50, 18] .

Original: More precisely, the CMDP problem (2) and its equivalent (16) can be written as

Original: [Equation]

Original: 17a

Original: [Equation]

Original: subject to

Original: [Equation]

Original: 17b

Original: [Equation]

Original: From the optimal solution w ∗ of (17), the optimal policy π ∗ for the CMDP problem (2) can be computed using (13).

Original: B Extended Linear Programming Method for Solving (7) and (10)

Original: The OFU problem (7) and the DOP problem (10) may appear much more challenging than the CMDP problem (2) because they involve a minimization over all models in P k , which is non-trivial.

Original: However, ﬁnding the optimistic model (and the corresponding optimistic policy) from a given conﬁdence set is a standard step in OFU style algorithms for exploration in RL [21, 18].

Original: In the case of standard

Original: (unconstrained) MDP, this problem is solved using a approach called extended value iteration [21].

Original: In the case of constrained MDP, (7) (and similarly (10) ) can be solved by an approach called extended linear programming .

Original: The details are given in [18].

Original: We give a brief description below for completeness.

Original: Note that the description below mainly focus on solving (7).

Original: Solving (10) is identical, just by replacing the constraint cost function c h ( · , · ) with pessimist constraint cost function ¯ c h,k ( · , · ) ,

Original: ∀ h ∈

Original: [ H ] , and is mentioned at the end of this subsection.

Original: Deﬁne the state-action-state occupancy measure z π as z πh ( s, a, s (cid:48) ; P ) =

Original: P h ( s (cid:48) | s, a ) w πh ( s, a ; P ) .

Original: The extended LP formulation corresponding to (7) is then given as follows:

Original: [Equation]

Original: The last two conditions ((18f) and (18g)) distinguish the extended LP formulation from the LP formulation for CMDP.

Original: These constraints are based on the Bernstein conﬁdence sets around the empirical model (cid:98) P k .

Original: From the solutions ˜ z ∗ of the extended LP

Original: [Equation]

Original: we can obtain the solution of (7) as

Original: [Equation]

Original: C Useful Technical Results

Original: Here we reproduce the supporting technical results that are required for analyzing our DOPE algorithm.

Original: We begin by stating the following concentration inequality, known as empirical Bernstein inequality [30, Theorem 4].

Original: Lemma 7 (Empirical Bernstein Inequality) .

Original: Let Z =

Original: ( Z 1 , . . .

Original: Z n ) be i.i.d random vector with values in [0 , 1] n , and let δ ∈ (0 , 1) .

Original: Then, with probability at least 1 − δ , it holds that

Original: [Equation]

Original: where V n ( Z ) is the sample variance.

Original: We can get the following result using empirical Bernstein inequality and union bound.

Original: This result is widely used in the literature now, for example see [22, Proof of Lemma 2],

Original: Lemma 8.

Original: With probability at least 1 − 2 δ , for all ( h, s, a, s (cid:48) ) ∈

Original: [ H ] × S ×

Original: A × S , k ∈ [ K ] , we have

Original: [Equation]

Original: Recall (from (4) - (5))

Original: that

Original: [Equation]

Original: s

Original: [Equation] and deﬁne P k = (cid:84) ( h,s,a ) ∈ [ H ] ×S×A P h,k ( s, a ) .
Revised: The objective cost is r ( s, a ) =

Original: Deﬁne the event

Original: [Equation]

Original: Then, using Lemma 8, we can get the following result immediately.

Original: Lemma 9.

Original: Let F p be the event deﬁned as in (20) .

Original: Then, P ( F p ) ≥ 1 − 2 δ .

Original: Deﬁne the events F ck = {∀ ( h, s, a ) : | ˆ c h,k

Original: ( s, a ) − c h

Original: ( s, a )

Original: | ≤ β lh,k

Original: ( s, a ) } , and F rk = {∀ h, s, a : | ˆ r h,k

Original: ( s, a ) − r h

Original: ( s, a )

Original: | ≤ β lh,k

Original: ( s, a ) } , and deﬁne

Original: [Equation]

Original: The following is a standard result, and can be obtained by Hoeffding’s inequality, and using a union bound argument on all h, s, a and all possible values of n h,

Original: k ( s, a ) , for all k ∈ [ K ] .

Original: Lemma 10.

Original: P ( F l ) ≥ 1 − 2 δ .

Original: We now deﬁne the event F w as follows

Original: [Equation]

Original: where w h,j is the occupancy measure corresponding to the policy chosen in episode j .

Original: We have the following result from [14, Corollary E.4.]

Original: Lemma 11 (Corollary E.4., [14]) .

Original: Let F w be the event deﬁned as in (22) .

Original: Then, P ( F w ) ≥ 1 − δ .

Original: We now deﬁne the good event G = F p ∩ F w ∩ F l .

Original: Using union bound, we can show that P ( G ) ≥ 1 − 5 δ .

Original: Since our analysis is based on this good event, we formally state it as a lemma.

Original: Lemma 12.

Original: Let F p is deﬁned as in (20) and F l deﬁned in (21) , F w is deﬁned as in (22) .

Original: Let the good event G = F p ∩ F w ∩ F l .

Original: Then, P ( G ) ≥ 1 − 5 δ .

Original: We will also use the following results for analyzing the performance of our DOPE algorithm.

Original: Lemma 13 (Lemma 36, [18]) .

Original: Under the event F w , K

Original: [Equation]

Original: Lemma 14 (Lemma 37, [18]) .

Original: Under the event F w , K

Original: [Equation]

Original: k , a h,k ) ∨ 1 |F k −(cid:21) ≤ ˜ O

Original: ( SAH 2 ) .

Original: Lemma 15 (Lemma 8,[22]) .

Original: Under the event G , for all k, h, s, a, s (cid:48) , and for all P (cid:48) ∈

Original: P k , there exists constants C 1 , C 2 > 0 such that | P (cid:48) h ( s (cid:48) | s, a )

Original: − P h ( s (cid:48) | s, a ) | ≤ C 1 (cid:113) P h ( s (cid:48) | s,a ) L n h,k

Original: [-( s,a ) ∨ 
-]
Revised: [-( s,a ) ∨ 
-]

Original: + C

Original: L n h,k ( s,a ) ∨ 1 .

Original: Lemma 16 (Value difference lemma) .

Original: Consider two MDPs M =

Original: ( S , A , l, P ) and M (cid:48) =

Original: ( S , A , l (cid:48) , P (cid:48) ) .

Original: For any policy π , state s ∈ S , and time step h ∈

Original: [ H ] , the following relation holds.

Original: [Equation]

Original: D Proof of the Main Results

Original: All the results we prove in this section are conditioned on the good event G deﬁned in Section C.

Original: So, the results hold true with a probability greater than 1 − 5 δ according to Lemma 12.

Original: We will omit stating this conditioning under G in each statement to avoid repetition.

Original: D.1 Proofs of Proposition

Original: First note that

Original: [Equation]

Original: where equations (23)

Original: [Equation]

Original: (24) are due to linearity of expectation

Original: [Equation]

Original: Lemma 17.

Original: Let (cid:15) π (cid:48)

Original: k ( P (cid:48) ) and η π (cid:48) k

Original: ( P (cid:48) ) be as deﬁned in (11) .

Original: Also, let { π k } be the sequence of policies generated by DOPE algorithm.

Original: Then, for any K (cid:48) ≤ K , each of the following relations hold with a probability greater than 1 − 5 δ .

Original: [Equation]

Original: Proof.

Original: [Equation]

Original: Here, we get inequality ( a ) by the deﬁnition of β ph,k (c.f.

Original: To get ( b ) , note that

Original: [Equation]

Original: P

Original: [Equation]

Original: ,k

Original: [Equation]

Original: s

Original: [Equation]

Original: |

Original: [Equation]

Original: s

Original: [Equation]

Original: ,k ,

Original: [Equation]

Original: [Equation]

Original: )) ≤ (cid:113)(cid:80) s

Original: [Equation]

Original: (cid:98) P h,k ( s

Original: [Equation]

Original: | ( s h,k , a h,k )) √ S by Cauchy-Schwarz inequality and (cid:80) s (cid:48) (cid:98) P h,k ( s (cid:48) | ( s h,k , a h,k ))

Original: = 1 .

Original: We get ( c ) using Lemma 13 and Lemma 14.

Original: The other part can also be obtained similarly from Lemma 13.

Original: We now give the proof of Proposition 4.

Original: Proof of Proposition 4 .

Original: First note that even though ( π b , P ) is a feasible solution for the original

Original: CMDP problem (2), it may not feasible for the DOP problem (10).

Original: To see this, note that since

Original: V π b ¯ c k ( P ) =

Original: V π b ˆ c

Original: k

Original: ( P ) + η π b k ( P ) + (cid:15)

Original: π b k ( P ) and V π b ˆ c k ( P ) ≤ V π b c ( P ) + η π b k ( P ) , and V π b c ( P ) =

Original: ¯ C b , we will have V π b ¯ c k ( P ) ≤ ¯ C

Original: if 2 η π b k ( P ) + (cid:15) π b k ( P ) ≤

Original: ( C − ¯ C b ) .

Original: So, ( π b , P ) is a feasible solution for (10) if, 2 η π b k ( P ) + (cid:15)

Original: π b k ( P ) ≤ ( C − ¯ C b ) .

Original: This is a sufﬁcient condition for the feasibility of ( π b , P ) .

Original: This condition may not be satisﬁed in the initial episodes.

Original: However, since η π b k ( P ) and (cid:15) π b k ( P ) are decreasing in k , if ( π b , P ) becomes a feasible solution for (10) at episode k (cid:48) , then it will remain to be a feasible solution for all episodes k ≥ k (cid:48) .

Original: Suppose π k = π b for all k ≤ K (cid:48) .

Original: Also, suppose the above condition is not satisﬁed in the algorithm until episode K (cid:48)

Original: + 1 .

Original: Then, 2 η π b k ( P ) + (cid:15)

Original: π b k ( P ) > C − ¯ C b for all k ≤ K (cid:48) .

Original: So, we should get

Original: K (cid:48) (

Original: [Equation]

Original: <

Original: [Equation]

Original: where the last inequality is from Lemma 17.

Original: However, this inequality is violated for K (cid:48) ≥

Original: ˜ O

Original: ( S 2 AH 4 ( ¯ C − ¯ C b ) 2 ) .

Original: So, ( π b , P ) is a feasible solution for (10) for any episode k ≥ K o =

Original: ˜

Original: O ( S 2 AH 4 ( ¯ C − ¯ C b ) 2 ) provided that π k = π b for all k ≤ K o .

Original: The above result, however, only shows that π b becomes a feasible policy after some ﬁnite number of episodes.

Original: A natural question is, is π b the only feasible policy?

Original: In such a case, the DOPE algorithm may not provide enough exploration to learn the optimal policy.

Original: We alleviate the concerns about the above possible issue by showing that for all k ≥ K o , there exists a feasible solution ( π (cid:48) , P ) for the OP problem (10)

Original: such that w π (cid:48) h

Original: [-( s, a ; P ) > 
-]
Revised: [-( s, a ; P ) > 
-]

Original: for every ( s, a ) ∈ S ×A with w π ∗ h ( s, a ; P ) > 0 .

Original: Informally, this implies that π (cid:48) will visit all state-action pairs that will be visited by the optimal policy π ∗ .

Original: This result can be derived as a corollary for Proposition 4.

Original: D.2 Proof of Proposition

Original: Proof.

Original: For any episode k ≤ K o , we have π k = π b , and it is safe by Assumption 1.

Original: For k ≥ K o , (10) is feasible according to Proposition 4.

Original: Since ( π k , P k ) is the solution of (10), we have V π k ¯ c k

Original: ( P k ) ≤ ¯ C .

Original: We will now show that V π k c ( P ) ≤ ¯ C , conditioned on the good event G .

Original: By the value difference lemma (Lemma 16), we have

Original: [Equation] Here, we get ( a ) by Holder’s inequality inequality. To get ( b ) , we make use of two observations. First, note that (cid:107) V π k c,h +1 ( · ; P ) (cid:107) ∞ ≤ H because the expected cumulative cost cannot be grater than H since | c ( · , · ) | ≤ 1 by assumption.
Revised: { s = 0 } , while the constraint cost is c ( s, a ) = { a = 1 } , i.e., we desire to minimize the outage cost, while limiting the usage of fast service.

Original: Second, under the good event G , (cid:80) s (cid:48)

Original: | P h ( s (cid:48) | s

Original: [Equation]

Original: From (26), we get

Original: [Equation]

Original: where ( c ) is by deﬁnition of good event and ( d ) is from the fact that ( π k , P k ) is the solution of (10).

Original: So, V π k c ( P ) ≤ ¯ C , and hence π k is safe, under the good event G .

Original: So, this statement holds with a probability greater than 1 − 5 δ , according to Lemma 12.

Original: D.3 Proof of Theorem

Original: We ﬁrst prove an important lemma.

Original: Lemma 18 (Optimism) .

Original: Let ( π k , P k ) be the optimal solution corresponding to the DOP problem (10) .

Original: Then,

Original: [Equation]

Original: Proof.

Original: Wewill ﬁrst consider a more general version of the DOP problem (10) as [Equation] subject to [Equation]
Revised: We consider episode length

Original: (27) where we change ¯ r k in (10) to ˜ r k above, with ˜ r kh ( s, a ) = ˆ r h
Revised: H = 10 , and constraint ¯ C

Original: ( s, a ) − 3 bβ lh,k ( s, a ) − bH ¯ β ph,k

Original: ( s, a ) , for b > 0 .

Original: Note that (27) reduces to (10) for b = H ¯ C − ¯ C b and hence it is indeed a general version. Using the occupancy measures w π b h and w π ∗ h , deﬁne a new occupancy measure ˜ w h ( s, a ) = (1 − α k ) w π b h ( s, a ; P ) + α k w π ∗ h ( s, a ; P ) for an α k > 0 .
Revised: = H 2 .

Original: Note that ˜ w is a valid occupancy measure since the set of occupancy measure is convex (c.f. Proposition 6).

Original: Let ˜ π be the policy corresponding to the occupancy measure ˜ w , which can be obtained according to (13) so that ˜ w = w ˜ π .

Original: Claim 1: (˜ π, P ) is a feasible solution for (27) when α k satisﬁes the sufﬁcient condition

Original: [Equation]

Original: Proof of Claim 1: Since value function is a linear function of the occupancy measure, we have

Original: [Equation]

Original: where inequality ( a ) is due to the good event that c is within the conﬁdence interval, V π ˆ c k ( P ) − η πk ( P ) ≤ V πc ( P ) , for any π and inequality ( b ) is due to the fact that V π b c ( P ) =

Original: ¯ C b , and V π ∗ c ( P ) ≤ ¯ C .

Original: For (˜ π k , P ) to be a feasible solution for (27), it must be true that V ˜

Original: π ¯ c k ( P ) ≤ ¯ C .

Original: Hence, it is sufﬁcient to get an α k such that

Original: [Equation] This yields a sufﬁcient condition (28). Note that α k is non-negative because 2 η π b k ( P ) + (cid:15) π b k ( P ) ≤ ¯ C − ¯ C b for k ≥ K o , as shown in the proof of Proposition 4.
Revised: Experiment Setup: OptCMDP and OptPessLP algorithms have not been implemented earlier.

Original: This concludes the proof of Claim 1.

Original: Claim 2: V ˜ π k ˜ r k ( ˜ P k ) ≤ V π ∗ r ( P ) if b satisﬁes the sufﬁcient condition

Original: [Equation]

Original: Proof of Claim 2: Selecting an α k that satisﬁes the condition (28), (˜ π, P ) is a feasible solution of (27).

Original: Since (˜ π k , ˜ P k ) is the optimal solution of (27), we have V ˜ π k ˜ r k ( ˜ P k ) ≤ V

Original: ˜ π ˜ r k ( P ) .

Original: So, it is sufﬁcient to ﬁnd a b such that V ˜

Original: π

Original: ˜ r k ( P ) ≤ V π ∗ r ( P ) . Using the linearity of the value function w.r.t. occupancy measure, this is equivalent to (1 − α k )( V π b ˆ r k ( P ) − [Equation]
Revised: For accurate comparison, we simulate all the algorithms true to their original formulations of cost functions and conﬁdence intervals.

Original: b k

Original: [Equation]

Original: P

Original: [Equation]

Original: −

Original: [Equation]

Original: π b k

Original: [Equation]

Original: P

Original: [Equation]

Original: α k

Original: [Equation]

Original: V π ∗

Original: [Equation]

Original: k ( P ) −

Original: [Equation]

Original: π ∗ k

Original: [Equation]

Original: P )

Original: [Equation]

Original: b(cid:15) π ∗ k

Original: ( P )) ≤

Original: V π ∗ r ( P ) .

Original: Since V π ˆ r k ( P ) − bη πk ( P ) ≤ V π ˆ r k ( P ) − η πk ( P ) ≤ V πr ( P ) for any π under the good event

Original: , it is sufﬁcient if we ﬁnd a b such that (1 − α k )

Original: ( V π b r ( P ) − 2 bη π b k ( P ) − b(cid:15)

Original: π b k ( P ))

Original: + α k

Original: ( V π ∗ r − 2 bη π ∗ k ( P ) − b(cid:15)

Original: π ∗ k

Original: ( P )) ≤

Original: V π ∗ r ( P ) .

Original: This will yield the condition b ≥

Original: [Equation]

Original: Now, we choose α k that satisﬁes the condition (28) as, α

Original: [Equation]

Original: =

Original: [Equation]

Original: k

Original: [Equation]

Original: P

Original: [Equation]

Original: Using this in the previous inequality for b , we get the sufﬁcient condition b ≥ V πbr ( P ) − V π ∗ r ( P )

Original: ¯ C −

Original: [Equation]

Original: C b .

Original: Since V π b r ( P ) ≤ H and V π ∗ r ( P ) ≥ 0 , we get the sufﬁcient condition (29). This concludes the proof of Claim 2. Now, let b =
Revised: Our experiments are carried out for 20 random runs, and averaged to obtain the regret plots.

Original: H ¯ C − ¯ C b .

Original: So, ˜ r k = ¯ r k and (˜ π k , ˜ P k ) =

Original: ( π k , P k ) .

Original: Hence, by Claim 2, we have V π k ¯ r k

Original: ( P k ) ≤

Original: V π ∗ r ( P ) .

Original: Hence, we have the desired result.

Original: We now present the proof of Theorem 3.

Original: Proof of Theorem 3 . The regret for the DOPE algorithm after K episodes can bewritten as, [Equation] We will bound the ﬁrst term in (30) as K
Revised: For DOPE, we choose K 0 to be as speciﬁed in Proposition 4.

Original: [Equation]

Original: where we get the ﬁrst inequality because ( V π k r ( P ) − V π ∗ r ( P )) ≤ H , and the second inequality follows from the bound on K o in Proposition

Original: [Equation]

Original: The second term in (30) can be bounded as K

Original: [Equation]

Original: (32) where ( a ) is due to the fact that V π k ¯ r k ( P k ) ≤ V π ∗ r ( P ) from Lemma 18, ( b ) is due to the fact that | r h ( s, a ) − ˆ r h,k ( s, a ) | ≤ β lh,k

Original: ( s, a )

Original: conditioned on the good event set G (see Lemma 10), ( c ) follows from the deﬁnition of ¯ r k , and ( d ) follows from Lemma 17.

Original: We will now bound the ﬁrst term in (32) as K

Original: [Equation]

Original: Now, for the ﬁrst term in (33), K

Original: [Equation]

Original: where ( e ) is obtained from the value difference lemma (Lemma 16)

Original: [Equation]

Original: and ( f ) is from Lemma 17.

Original: In order to now bound the second term in (33)

Original: [Equation]

Original: we proceed in similar lines to the proof of Lemmafrom [18]

Original: [Equation]

Original: Consider, K

Original: [Equation]

Original: where the last inequality is obtained from Lemma 15.

Original: We will bound the term ( B ) in (35) as

Original: [Equation]

Original: where the ﬁrst inequality is from bounding | V π k ¯ r k ( P k ) − V π k r ( P ) | by H 3 SL ¯ C − ¯ C b .

Original: This is obtained by noting that V π k ¯ r k

Original: [Equation]

Original: P k

Original: [Equation]

Original: =

Original: [Equation]

Original: ˆ

Original: [Equation]

Original: P

Original: [Equation]

Original: −

Original: [Equation]

Original: since (cid:80) Hh =1 (cid:80) s (cid:48)

Original: β ph,k

Original: ( s h,k , a h,k , s (cid:48) )

Original: ≤ HSL , from the deﬁnition.

Original: The second inequality is from

Original: Lemma

Original: [Equation]

Original: We now bound the term ( A ) in (35) as follows.

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: 

Original: 

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: 

Original: 

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Here, ( a ) is obtained by Jensen’s inequality, ( b ) is by cauchy schwartz inequality, ( c ) is from the property of the occupancy measure, i.e., (cid:80) s,a P h ( s

Original: [Equation]

Original: | s

Original: [Equation]

Original: a

Original: [Equation]

Original: ( s, a, P ) =

Original: (cid:80)

Original: a w h +1 ( s (cid:48) , a, P ) , ( d ) is obtained from Lemma 14.

Original: [-To get ( e ) , we use the result from Lemma 
-]
Revised: [-To get ( e ) , we use the result from Lemma 
-]

Original: that V π k ¯ r k ,h +1 ( P k ) ≤ V π k r,

Original: h +1 ( P ) , and hence obtain, ( V π k ¯ r k ,h

Original: +1 ( P k ) − V π k r,h +1 ( P )( s )) 2 ≤ H 3 SL ¯ C − ¯ C b

Original: ( V π k r,h +1 ( P ) − V π k ¯ r k ,h +1 ( P k )( s )) .

Original: We prove ( e ) from Lemma 33 of [18].

Original: The step is to get ( f ) is more involved and we prove it separately in Lemma 20, following a similar result from Lemma 33 of [18].

Original: The inequality ( g ) holds from the fact that √ a + b ≤ √ a + √ b .

Original: Using the above obtained bounds on ( A ) and ( B ) in (35), we get (cid:88) k

Original: [Equation]

Original: − V π k r,h +1 ( · ; P )) (cid:105)| 

Original:  1 /.

Original: Let X = (cid:80) k,h,s,a w π k h ( s, a, P ) |(cid:104) ( P h − P h,k )( .

Original: | s, a )( V π k ¯ r k ,h +1 ( · ; P k ) − V π k r,h +1 ( · ; P ) (cid:105)| .

Original: Then, the above bound takes the form

Original: [Equation]

Original: where a

Original: [Equation]

Original: and b =

Original: [Equation]

Original: [-Now, using the fact that, if 0 ≤ X ≤ a + b √ X , then X ≤ a + b 
-]
Revised: [-Now, using the fact that, if 0 ≤ X ≤ a + b √ X , then X ≤ a + b 
-]

Original: Lemma 38 from [18]), we can obtain the bound (cid:88) k

Original: [Equation]

Original: Substituting the above bound and (34) in (33), we obtain, (cid:88)

Original: [Equation]

Original: Using the above bound in (32), we obtain, (cid:88)

Original: [Equation]

Original: The left hand side of the above inequality is non-negative, since V π k ¯ r k ( P k ) ≤ V π ∗ r ( P ) , from lemma 18, and V π ∗ r ( P ) ≤ V π k r ( P ) , since π ∗ is the optimal policy on P .

Original: This equation is again of the form 0 ≤ X ≤ a + b √ X , where X = (cid:80)

Original: k

Original: V π k r ( P ) − V π k ¯ r k

Original: ( P k ) .

Original: Using the same result we used to get (38), we deduce that X ≤ a + b 2 , and hence,

Original: [Equation]

Original: and hence, from (32), (cid:80) k

Original: [Equation]

Original: Moreover, from proposition (5), we have that π k ∈ Π safe for all k ∈ [ K ] , with probability 1 − 5 δ .

Original: Lemma 19 (Bonus optimism) .

Original: For any ( s, a, h, k ) , conditioned on good event, we have that r h

Original: ( s h , a h ) − ¯ r h,k

Original: ( s h , a h ) + (cid:16) <

Original: P h ( .

Original: | s h , a h ) − P h,k ( .

Original: | s 1 , a 1 ) , V π k r,h +1 ( . ; P ) > (cid:17) ≥ 0 , and, for any π, s, h, k , it holds that V π ¯ r k ,

Original: h ( s ; P k ) ≤ V πr,h ( s ; P ) .

Original: Proof.

Original: Consider

Original: [Equation]

Original: where the last inequality is due to the fact that ¯ C − ¯ C b ≤ H .

Original: Similarly, by value difference lemma 16, we have,

Original: [Equation]

Original: ≤ 0 , where the last inequality is obtained just earlier

Original: [Equation]

Original: Lemma 20.

Original: We prove inequality ( e ) in bounding term ( i ) ,

Original: i

Original: [Equation]

Original: e., we prove that, (cid:88)

Original: [Equation]

Original: Proof.

Original: Let us start with,

Original: [Equation]

Original: [Equation]

Original: where inequality ( a ) is from Lemma 19.

Original: Iterating this relation, for h times, we get,

Original: [Equation]

Original: Summing this relation for h ∈ { 2 , . .

Original: , H }

Original: [Equation]

Original: we get,

Original: [Equation]

Original: Hence, we obtain, (cid:88)

Original: [Equation]

Original: | s h (cid:48) , a h (cid:48) ) , V π k ¯ r k ,h (cid:48)

Original: [-+
-]
Revised: [-+
-]

Original: ( . ; P k ) − V π k r,h (cid:48)

Original: +1 ( . ; P ) > | s 1 , π k , P (cid:105) ≤ H (cid:2)

Original: V π k r, 1 ( P ) − V π k ¯ r k , 1 ( P k ) (cid:3)

Original: +

Original: H (cid:88) h

Original: =

Original: H (cid:88) h (cid:48) =

Original: E (cid:104) | <

Original: ( P h (cid:48) − P h (cid:48) ,k )( .

Original: | s h (cid:48) , a h (cid:48) ) , V π k ¯ r k ,h (cid:48)

Original: [-+
-]
Revised: [-+
-]

Original: ( . ; P k ) − V π k r,h (cid:48)

Original: +1 ( . ; P ) >

Original: || s 1 , π, P (cid:105) ≤ H (cid:2)

Original: V π k r, 1 ( P ) − V π k ¯ r k , 1 ( P k ) (cid:3) + H

Original: H (cid:88) h

Original: =

Original: E (cid:104) | <

Original: ( P h − P h,k )( . | s h , a h ) , V π k ¯ r k ,h +1 ( . ; P k ) − V π k r,h +1 ( . ; P ) > || s 1 , π k , P (cid:105) . E Detailed Description of Experiment Environments and Algorithm Implementation E.1 Experiment Environments
Revised: Full details on the algorithm parameters and experiment settings are provided in Appendix E.

Original: Factored CMDP environment: The factored CMDP is represented in Fig.

Original: The state space is

Original: S = { 1 , 2 , 3 } , and the action space is A = { 1 , 2 } , where action 1 corresponds to moving one step to the right and 2 corresponds to staying put.

Original: Objective cost is r ( s, 1) = 0 , ∀ s ∈ S , and r ( s, 2) =

Original: s, ∀ s ∈ S .

Original: The constraint cost is, c ( s, 1) = 0 , ∀ s ∈ S , and c ( s, 2) = 1 , ∀

Original: (cid:32) 0 10 01 0(cid:33)

Original: 1 00 10 0(cid:33) .

Original: [-2r ( 1 , 1 ) = 
-]
Revised: [-2r ( 1 , 1 ) = 
-]

Original: c ( 1 , 1 ) =

Original: r (2 , 1) =

Original: c (2 , 1) =

Original: r ( 3 , 1 ) =

Original: c ( 3 , 1 ) =

Original: r (1 , 2) =

Original: c (1 , 2) =

Original: r (2 , 2) =

Original: c (2 , 2) =

Original: 1 r (2 , 2) = c (2 , 2) = Media Streaming Environment: Here, we model the media streaming control from a wireless base station. The base station provides two types of service to a device, a fast service and a slow service.
Revised: Baseline Policies: Both OptPessLP and DOPE require baseline policies.

Original: The packets received are stored in a media buffer at the device.

Original: The goal is to minimize the cost of having an empty buffer (which may result in stalling of the video), while keeping the utilization of fast service below certain level.

Original: We denote by A h , the number of incoming packets into the buffer, and by B h , the number of packets leaving the buffer.

Original: The state of the environment, denoted ass h in h th step, is the media buffer length. It evolves as, s h +1 = min { max(0 , s h + A h − B h ) , N } .We consider N = 20 as the maximum buffer length in our experiment.
Revised: We select the baseline policies as the optimal solutions of the given CMDP with a constraint ¯ C b = 0 .

Original: The action space is A = { 1 , 2 } , i.e., the action is to use either fast server 1 or slow server 2 .

Original: We assume that the service rates of the servers have independent Bernoulli distributions, with parameters µ 1 = 0 .

Original: 9 , and µ 2 = 0 .

Original: 1 , where µ 1 corresponds to the fast service.

Original: The media playback at the device is also Bernoulli with parameter γ .

Original: Hence, A h is a random variable with mean either µ 1 or µ 2 depending on the action taken, and B h is a random variable with mean γ .

Original: These components constitute the unknown transition dynamics of our environment.

Original: The objective cost is r ( s, a ) = 1 { s = 0 } .

Original: i.e., it has a value of 1 , when the buffer hits zero, and is zero everywhere else.

Original: Our constraint cost is c ( s, a ) = 1 { a = 1 } , i.e., there is a constraint cost ofwhen the fast service is used, and is zero otherwise.

Original: We then constrain the expected number of times the fast service is used to ¯ C

Original: =

Original: H/ 2 , in a horizon of length H = 10 .

Original: Inventory Control Environment: We consider a single product inventory control problem [7].

Original: Our environment evolves according to a ﬁnite horizon CMDP, with horizon length H = 7 , where each time step

Original: h ∈

Original: [ H ] represents a day of the week.

Original: In this problem, our goal is to maximize the expected total revenue over a week, while keeping the expected total costs in that week below a certain level.

Original: We do not backlog the demands.

Original: The storage has a maximum capacity N = 6 , which means it can store a maximum of 6 items.

Original: We denote by s h , the state of the environment, as the amount of inventory available at h th day.

Original: The action a h is the amount of inventory the agent purchases such that the inventory does not overﬂow.

Original: Thus, the action space A s ∈ { 0 , . .

Original: , N − s } , for the state s .

Original: The exogenous demand is represented by d h , which is a random variable representing the stochastic demand for the inventory on the h th day.

Original: We assume d h to be in { 0 , · · · , N } with distribution [0 . 3 , 0 . 2 , 0 . If the demand is higher than the inventory and supply, the excess demand will not be met. The state evolution then follows as s h +1 = max { 0 , s h + a h − d h } .We deﬁne the rewards and costs as follows.
Revised: 2 ¯ C .

Original: The revenue is generated as, f ( s, a, s (cid:48) )

Original: =

Original: 8( s + a − s (cid:48) ) , when s (cid:48) > 0 , and is 0 otherwise.

Original: The reward obtained in state ( s, a ) is then the expected revenue over all next states s (cid:48) , r ( s, a ) =

Original: E [ f ( s, a, s (cid:48) )] .

Original: The cost associated with the inventory has two components.

Original: Firstly, there is a purchase cost when the inventory is brought in, which is a ﬁxed cost of 4 units, plus a variable cost of 2 a , which increases with the amount of purchase.

Original: Secondly, we also have a non decreasing holding cost s , for storing the inventory.

Original: [-Hence, the cost in ( s, a ) is c ( s, a ) = 
 + 2 a + s .-] We normalize the rewards and costs to be in the range [0 , 1] .
Revised: [-Hence, the cost in ( s, a ) is c ( s, a ) = 
 + 2 a + s .-] We choose the same baseline policies for both the algorithms.

Original: Our goal is to maximize the expected total revenue over a week ( H = 7) , while keeping the expected total costs in that week below a threshold ¯ C = H/ 2 . E.2 Details of the Implementation
Revised: This choice is to showcase the efﬁcacy of DOPE, despite starting with a conservative baseline policy.

Original: We now describe the algorithms described in the introduction.

Original: AlwaysSafe: AlwaysSafe [36] shows empirical results that only depict the expected cost of various policies deduced from their linear programs versus the optimal expected cost.

Original: We note that this comparison is not a reasonable measure of regret, since cumulative regret can be linear even though the expected costs are close.

Original: We consider a factored CMDP environment described previously. For implementing this algorithm, in each episode, we solve the LP4 linear program described in [36], based on the observations. For solving LP4, one also needs an abstract CMDP as described in Section 3 of[36]. We follow their description to construct such a model for the factored CMDP.
Revised: Results for Factored CMDP: Fig. 1(a) shows the objective regret of the algorithms in this environment.

Original: The conﬁdence intervals for

Original: AlwaysSafe algorithm are same as the ones for OptCMDP algorithm from [18] and of DOPE. We notice that the regret of Always safe algorithms indeed grow at a linear rate. OptPessLP: We implement Algorithm 1 from [28]. We choose the baseline policy by solving the corresponding CMDP with a more conservative constraint.
Revised: OptCMDP has a good objective regret performance as expected, but shows constraint violations.

Original: For the factored CMDP and Media

Original: Control environment, we choose the constraint as 0 . 1 ¯ C , and solve the MDP to obtain π b and the corresponding cost ¯ C b . Similarly, for the inventory control environment, we choose the constraint as 0 . 2 ¯ C .
Revised: AlwasySafe fails to achieve ˜ O

Original: These choices of π b are the same for both OptPessLP and DOPE, for a fair comparison. We play π b when the condition in Equation 9 from [28] is met, otherwise we choose the maximizing policy from their linear program. We choose the conﬁdence intervals as speciﬁed in their work, without any scaling. Despite the results suggested by theory, we notice that its empirical performance is very poor in every environment we consider.
Revised: ( p K ) regret in the episodes shown for both variants, although the ⇡ ↵ variant has smaller regret as compared to the ⇡ T variant.

Original: OptCMDP: We implement Algorithm 1 from [18].

Original: This algorithm solves the linear program that minimizes the objective cost with optimism in the model.

Original: Since this algorithm does not consider zero-violation setting, we expect to see constraint violation of the same order as the regret.

Original: We use the conﬁdence intervals as speciﬁed in their work, without any scaling.

Original: DOPE: We implement Algorithm 1 from our work. The choice of baseline policy π b is exactly the same as that for OptPessLP for each environment. π b is played until K 0 episodes, as provided by
Revised: OptPessLP takes a long time to attain p K behavior, which means that is chooses the baseline policy for an extended period, and shows high empirical regret.

Original: Proposition 4.

Original: Then, the algorithm solves the linear program given by Equation (10) to obtain π k in episode k .

Original: The details for the linear program formulations are given in Appendices A and B. DOPE, AlwaysSafe and OptCMDP algorithms use the Extended LP formulations, while OptPessLP uses the regular LP formulation.

Original: For each environment, each of these algorithms are run for 20 random seeds, and are averaged to obtain the regret plots in the ﬁgures.

Original: We show the performance of our DOPE algorithm in Inventory Control Environment. As before, we compare it against the OptCMDP Algorithm 1 in[18], and and OptPess-LP algorithm from [29]. Also, we choose the optimal policy from a conservative constrained problem (with a stricter constraint) as the baseline policy.
Revised: This suggests that reward optimism of OptPessLP is insufﬁcient to balance the pessimism in the constraint.

Original: We use ¯ C b = 0 .

Original: 1 ¯ C . Fig. 4(a) compares the objective regret for the inventory control environment incurred by each algorithm with respect to the number of episodes. As we see in this ﬁgure, in  the initial episodes, the objective regret of DOPE grows linearly with number of episodes.
Revised: DOPE not only achieves the desired ˜ O ( p K ) regret, but also does so in fewer episodes compared to the other two algorithms.

Original: Later, the growth rate of regret changes to square root of number of episodes.
Revised: Furthermore, it has low empirical regret.

Original: We see that this change of behavior happens after K 0 episodes speciﬁed by Proposition 4.
Revised: Fig. 1(b) shows that the constraint violation regret is zero for all the episodes of learning for all the safe algorithms, while OptCMDP shows a large constraint violation regret. Results for Media Streaming CMDP: Fig.2(a) compares the objective regret across algorithms.

Original: Hence, the linear growth rate indeed corresponds to the duration of time in which the base policy is employed. In conclusion, the  regretfor DOPE algorithm depicted in Figures 4(a) matches the result of Theorem 3.
Revised: The value of DOPE over OptPessLP is more apparent here. After a linear growth phase, the objective regret of DOPE changes to a square-root scaling.

Original: Next, the OptPess-LP algorithm performs quite badly in terms of objective regret, as it fails to achieve √ K regret performance within the chosen number of episodes.
Revised: OptPessLP has not explored sufﬁciently at this point, and hence suffers high linear regret.

Original: It thus shows the same issue of excessive pessimism observed in the other environments.

Original: Finally,we observe that the objective regret of OptCMDP  is lower than DOPE. This behavior can be attributed to the fact that in order to perform safe exploration, DOPE includes a pessimistic penalty in the constraint (8).
Revised: Finally, OptCMDP also has square-root regret scaling, but is fastest, since it is not constrained by safe exploration.

Original: [Figure] Fig. 4(b) compares the regret in constraint violation for DOPE, OptPess-LP and OptCMDP algorithms for the inventory control setting.
Revised: Fig.2(b) compares the regret in constraint violation for these algorithms.

Original: Here, we see that DOPE and OptPess-LP do not violate the constraint, while  OptCMDP incurs a regret that grows sublinearly. This ﬁgure shows that DOPE does indeed perform safe exploration as proved, while OptCMDP violates the constraints during learning.
Revised: As expected, DOPE and OptPessLP do not violate the constraint, while the OptCMDP algorithm signiﬁcantly violates the constraints during learning.

Original: Finally, Fig.
Revised: Effect of Baseline Policy: We compare the objective regret of DOPE under different baseline policies in Fig.1(c) and Fig.2(c). We see that less conservative (but safe) baselines result in lower regret, but the difference is not excessive, implying that the exact baseline policy chosen is not crucial. [Figure] [Figure]

Original: (4(c)) compares the optimality regret for various baseline policies.
Revised: Summary : DOPE has two valuable properties: ( i ) Faster rate of shift to p K behavior, since the linear regret phase where it applies the baseline policy is relatively short, and ( ii ) ˜ O

Original: Again, the takeaway here is that a good baseline policy is helpful, although the variation across different baseline policies is not very large.
Revised: ( p K ) regret with respect to optimal, which together mean that the empirical regret is lower than other approaches. Limitations: Our goal is to ﬁnd an RL algorithm with no constraint violation with high probability in the tabular setting. Since safe exploration is a basic problem, we follow the usual approach in the literature of ﬁrst establishing the fundamental theory results for the tabular setting [21, 14, 6].

Revised: We also note that most of the existing work on exploration in safe RL is in the tabular setting [18, 37, 28].

Revised: In our future work, we plan to employ the DOPE approach in a function approximation setting.

Revised:  Conclusion

Revised: We considered the safe exploration problem in reinforcement learning, wherein a safety constraint must be satisﬁed during learning and evaluation.

Revised: Earlier approaches to constrained RL have proposed optimism on the model, optimism on reward, and pessimism on constraints as means of modulating exploration, but none have shown order optimal regret, no safety violation, and good empirical performance simultaneously.

Revised: We started with the conjecture that double optimism combined with pessimism is the key to attaining the ideal balance for fast and safe exploration, and design DOPE that carefully combines these elements.

Revised: We showed that DOPE not only attains order-optimal ˜ O

Revised: ( p K ) regret without violating safety constraints, but also reduces the best known regret bound by a factor of p |S| .

Revised: Furthermore, it has signiﬁcantly better empirical performance than existing approaches.

Revised: We thus make a case for adoption of the approach for real world use cases and extension to large scale

Revised: RL problems using function approximation.

Revised:  Acknowledgement

Revised: This work was supported in part by the grants NSF-CAREER-EPCN

Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Revised: [Yes]

Revised: In Theorem 3 and Section 5.

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes]

Revised: In Section 5.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] Societal Impact and Ethics Statement.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] Described in Societal Impact and Ethics Statement.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] Assumption 1, and assumptions in statement of Theorem 3.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] Appendix. 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results?

Revised: [Yes] Supplemental material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] Section 5, and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] Figures contain error bars.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes]

Revised: Appendix

Revised: E 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [N/A] (b) Did you mention the license of the assets?

Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] Code is released as supplemtental material.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [N/A] 5.

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [N/A]