Original: Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability
Revised: Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Constrained reinforcement learning (RL) is an area of RL whose objective is tofind an optimal policy that maximizes expected cumulative return while satisfyinga given constraint.
Revised: Constrained reinforcement learning (RL) is an area of RL whose objective is tofind an optimal policy that maximizes expected cumulative return while satisfyinga given constraint.

Original: Most of the previous constrained RL works consider expectedcumulative sum cost as the constraint.
Revised: Most of the previous constrained RL works consider expectedcumulative sum cost as the constraint.

Original: However, optimization with this constraintcannot guarantee a target probability of outage event that the cumulative sum costexceeds a given threshold.
Revised: However, optimization with this constraintcannot guarantee a target probability of outage event that the cumulative sum costexceeds a given threshold.

Original: This paper proposes a framework, named Quantile
Revised: This paper proposes a framework, named Quantile

Original: Constrained RL (QCRL), to constrain the quantile of the distribution of cumulativesum cost that is a necessary and sufficient condition to satisfy the outage constraint.
Revised: Constrained RL (QCRL), to constrain the quantile of the distribution of cumulativesum cost that is a necessary and sufficient condition to satisfy the outage constraint.

Original: This is the first work that tackles the issue of applying policy gradient theorem tothe quantile and provides theoretical results for approximating the gradient of thequantile.
Revised: This is the first work that tackles the issue of applying policy gradient theorem tothe quantile and provides theoretical results for approximating the gradient of thequantile.

Original: Based on the derived theoretical results and the technique of the Lagrangemultiplier, we construct a constrained RL algorithm named Quantile Constrained
Revised: Based on the derived theoretical results and the technique of the Lagrangemultiplier, we construct a constrained RL algorithm named Quantile Constrained

Original: Policy Optimization (QCPO).
Revised: Policy Optimization (QCPO).

Original: We use distributional RL with the Large Deviation
Revised: We use distributional RL with the Large Deviation

Original: Principle (LDP) to estimate quantiles and tail probability of cumulative sum costfor the implementation of QCPO.
Revised: Principle (LDP) to estimate quantiles and tail probability of cumulative sum costfor the implementation of QCPO.

Original: The implemented algorithm satisfies the outageprobability constraint after the training period.
Revised: The implemented algorithm learns an optimalpolicy while it keeps satisfying the outage probability constraint during its learningprocess.

Original:  Introduction
Revised:  Introduction

Original: Reinforcement learning (RL) has been developed in the direction of finding an optimal policy thatmaximizes expected cumulative return for a given environment.
Revised: Reinforcement learning (RL) has been developed in the direction of finding an optimal policy thatmaximizes expected cumulative return for a given environment.

Original: Thus, most of the works in RLconsider only rewards given by the environment to optimize the policy.
Revised: Thus, most of the works in RLconsider only rewards given by the environment to optimize the policy.

Original: However, many real-worldcontrol problems impose constraints on the behavior of policy.
Revised: However, many real-worldcontrol problems impose constraints on the behavior of policy.

Original: Constrained RL is an area of RL whoseobjective is to find an optimal policy that maximizes expected cumulative return while satisfying acertain constraint on the cumulative cost.
Revised: Constrained RL is an area of RL whoseobjective is to find an optimal policy that maximizes expected cumulative return while satisfying acertain constraint on the cumulative cost.

Original: A conventional constrained RL problem (ExpCP) can bewritten as
Revised: A conventional constrained RL problem (ExpCP) can bewritten as

Original: Maximize
Revised: Maximize

Original: [Equation]
Revised: [Equation]

Original: subject to
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: (ExpCP) where the cost constraint is that the expectation of the sum of costs is less than or equal to a thresholdparameter d th .
Revised: where the cost constraint is that the expectation of the sum of costs is less than or equal to a thresholdparameter d th .

Original: Note that the threshold d th is set on the average (i.e., expectation) of the sum cost toavoid undesired high-cost events in this formulation.
Revised: Note that the threshold d th is set on the average (i.e., expectation) of the sum cost toavoid undesired high-cost events in this formulation.

Original: If we do not want any event causing a positivecost, d th should be set as a sufficiently small value, i.e., d th ≈ 0 .
Revised: If we do not want any event causing a positivecost, d th should be set as a sufficiently small value, i.e., d th ≈ 0 .

Original: On the other hand, if we canafford events with low cost, we can set d th properly as we desire.
Revised: On the other hand, if we canafford events with low cost, we can set d th properly as we desire.

Original: Most of the previous constrained
Revised: Most of the previous constrained

Original: RL works solved the problem (ExpCP) [1, 21, 20, 9, 17, 8, 18] partly because the constraint on theexpectation of the sum cost in (ExpCP) is well fit with the objective given by the expectation of thesum reward, and this makes the problem amenable.
Revised: RL works solved the problem (ExpCP) [1, 21, 20, 9, 17, 8, 18] partly because the constraint on theexpectation of the sum cost in (ExpCP) is well fit with the objective given by the expectation of thesum reward, and this makes the problem amenable.

Original: However, solving the problem (ExpCP) mayhave an undesirable outcome for real environments that typically need a constrained behavior on the event that the cost exceeds the threshold d th .
Revised: However, solving the problem (ExpCP) may have an undesirable outcome for real environments that typically need a constrained behavior on theevent that the cost exceeds the threshold d th .

Original: For example, in the case of an autonomous driving car,what we want for our sure safety is to control and limit the probability of accident itself.
Revised: For example, in the case of an autonomous driving car,what we want for our sure safety is to control and limit the probability of accident itself.

Original: In the caseof a telecommunication system, what we want to control is the probability of packet loss through thecommunication system.
Revised: In the caseof a telecommunication system, what we want to control is the probability of packet loss through thecommunication system.

Original: These probabilities are called ’outage probability’ in general.
Revised: These probabilities are called ’outage probability’ in general.

Original: Thus, in manyreal-world systems, the system requires a constraint on the outage probability, i.e., the probability ofcritical or unsafe events.
Revised: Thus, in manyreal-world systems, the system requires a constraint on the outage probability, i.e., the probability ofcritical or unsafe events.

Original: In this case, a constraint on the expectation of critical events as in (ExpCP)cannot guarantee the desired target probability of critical events.
Revised: In this case, a constraint on the expectation of critical events as in (ExpCP)cannot guarantee the desired target probability of critical events.

Original: To illustrate this, let us consider thefollowing example.
Revised: To illustrate this, let us consider thefollowing example.

Original: [Figure]
Revised: [Figure]

Original: Consider a simple two-path environment, asshown in Fig.
Revised: Consider a simple two-path environment, asshown in Fig.

Original: 1a.
Revised: 1a.

Original: The objective of the environment is for an agent to reach the goal by drivinga car.
Revised: The objective of the environment is for an agent to reach the goal by drivinga car.

Original: The environment gives a reward when theagent reaches the goal, and gives costs until theagent reaches the goal.
Revised: The environment gives a reward when theagent reaches the goal, and gives costs until theagent reaches the goal.

Original: There exist two pathsto reach the goal and each path has a differentcost distribution.
Revised: There exist two pathsto reach the goal and each path has a differentcost distribution.

Original: Fig.
Revised: Fig.

Original: 1b shows the distributionof the cumulative sum cost (curve) and its mean(vertical line) for each path: blue for path 1, and red for path 2.
Revised: 1b shows the distributionof the cumulative sum cost (curve) and its mean(vertical line) for each path: blue for path 1, and red for path 2.

Original: As we can see Fig.
Revised: As we can see Fig.

Original: 1b, the distributionfor path 1 has a lower average than that for path 2, but has a longer right tail than that for path 2.
Revised: 1b, the distributionfor path 1 has a lower average than that for path 2, but has a longer right tail than that for path 2.

Original: Ifwe use the expectation of cost as a constraint, then following path 1 is a better choice, but in thiscase, the probability of a high cumulative cost (e.g. > 10.0 in Fig.
Revised: Ifwe use the expectation of cost as a constraint, then following path 1 is a better choice, but in thiscase, the probability of a high cumulative cost (e.g. > 10.0 in Fig.

Original: 1b) is higher than following path 2.
Revised: 1b) is higher than following path 2.

Original: If the threshold of 10 represents a catastrophic event, we should constrain the probability of eventsexceeding 10 to a small value.
Revised: If the threshold of 10 represents a catastrophic event, we should constrain the probability of eventsexceeding 10 to a small value.

Original: Thus, solving the problem with the expectation constraint (ExpCP)does not necessarily have precise control over the target outage probability.
Revised: Thus, solving the problem with the expectation constraint (ExpCP) does not necessarily have precisecontrol over the target outage probability.

Original: When the event thatthe cumulative sum cost exceeds d th is a critical unsafe event, this means that the probability ofsuch critical event can occur in high probability even if we solve the constrained problem (ExpCP).
Revised: When the event that the cumulative sum cost exceedsd th is a critical unsafe event, this means that the probability of such critical event can occur in highprobability even if we solve the constrained problem (ExpCP).

Original: Therefore, in this paper, we aim to solve the following constrained RL problem with an outageprobability constraint:
Revised: Therefore, in this paper, we aim tosolve the following constrained RL problem with an outage probability constraint:

Original: [Equation]
Revised: [Equation]

Original: Subject
Revised: Subject

Original: [Equation]
Revised: [Equation]

Original: Our approach to this problem is first to convert the outage probability constraint in (ProbCP) intoa quantile constraint q π 1 − ϵ 

Revised: Our approach to this problem is first to convert the outage probability constraint in (ProbCP) intoa quantile constraint q π 1 − ϵ

Original: ( s 0 ) ≤ d th , which is equivalent to the original probabilistic constraint(details are in Section 3), and then to solve the equivalent optimization:
Revised: ( s 0 ) ≤ d th , which is equivalent to the original probabilistic constraint(details are in Section 3), and then to solve the equivalent optimization:

Original: [Equation]
Revised: [Equation]

Original:  where λ is the Lagrange multiplier, based on policy gradient with parameterized policy.
Revised: (1) where λ is the Lagrange multiplier, based on policy gradient with parameterized policy.

Original: However,we note that the policy gradient theorem, which is the most basic theorem for on-policy RL, cannotbe applied directly to compute the gradient of (1) with respect to (w.r.t.)
Revised: However,we note that the policy gradient theorem, which is the most basic theorem for on-policy RL, cannotbe applied directly to compute the gradient of (1) with respect to (w.r.t.)

Original: policy parameter due tothe quantile term q π 1 − ϵ 0 ( s 0 ) .
Revised: policy parameter due tothe quantile term q π 1 − ϵ 0 ( s 0 ) .

Original: Therefore, we derive theoretical results to approximate the gradient ofthe quantile (in Section 3).
Revised: Therefore, we derive theoretical results to approximate the gradient ofthe quantile (in Section 3).

Original: Then, based on the derived theoretical results and the technique of the
Revised: Then, based on the derived theoretical results and the technique of the

Original: Lagrange multiplier, we construct our algorithm named Quantile Constrained Policy Optimization(QCPO) to solve the outage probability constrained RL problem (in Section 4).
Revised: Lagrange multiplier, we construct our algorithm named Quantile Constrained Policy Optimization(QCPO) to solve the outage probability constrained RL problem (in Section 4).

Original: Here, we usedistributional RL with the Large Deviations Principle (LDP) to estimate quantiles and tail probabilityof cumulative sum cost for implementation of QCPO.
Revised: Here, we usedistributional RL with the Large Deviations Principle (LDP) to estimate quantiles and tail probabilityof cumulative sum cost for implementation of QCPO.

Original: The implemented algorithm satisfies the outageprobability constraint after the training period To the best of our knowledge, this is the first work thattackles the issue of applying the policy gradient theorem to quantile and obtains an (approximate)policy gradient for quantile, and this is one of the main contributions of this paper together with the QCPO algorithm.
Revised: The implemented algorithm learns an optimalpolicy while it keeps satisfying the outage probability constraint during its learning process. To thebest of our knowledge, this is the first work that tackles the issue of applying the policy gradienttheorem to quantile and obtains an (approximate) policy gradient for quantile, and this is one of themain contributions of this paper together with the QCPO algorithm.

Original:  Background and Related Works
Revised:  Background and Related Works

Original: Constrained RL A constrained Markov decision process (CMDP) is defined as a tuple⟨S , A , r, c, M, γ ⟩ , where S is the state space, A is the action space, r : S ×
Revised: Constrained RL A constrained Markov decision process (CMDP) is defined as a tuple⟨S , A , r, c, M, γ ⟩ , where S is the state space, A is the action space, r : S ×

Original: A → R is the reward function, c : S ×
Revised: A → R is the reward function, c : S ×

Original: A → R ≥ 0 is the cost function, M : S ×
Revised: A → R ≥ 0 is the cost function, M : S ×

Original: A × S → [0 , 1] is the state transitionprobability, and γ is the discount factor.
Revised: A × S → [0 , 1] is the state transitionprobability, and γ is the discount factor.

Original: Constrained RL or Safe RL is an area of RL whose objective is to find an optimal policy for agiven CMDP that maximizes the expected return
Revised: Constrained RL or Safe RL is an area of RL whose objective is to find an optimal policy for agiven CMDP that maximizes the expected return

Original: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] while satisfying a constrainton the cumulative sum of cost (cid:80)
Revised: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] while satisfying a constrainton the cumulative sum of cost (cid:80)

Original: ∞ t =

Revised: ∞ t =

Original: γ t c ( s t , a t ) .
Revised: γ t c ( s t , a t ) .

Original: If one wants to constrain the average cost ofthe policy π , then the constraint C π ( s 0 ) :
Revised: If one wants to constrain the average cost ofthe policy π , then the constraint C π ( s 0 ) :

Original: =
Revised: =

Original: E π [ (cid:80) ∞ t =

Revised: E π [ (cid:80) ∞ t =

Original: γ t c ( s t , a t )] ≤ d th can be considered.
Revised: γ t c ( s t , a t )] ≤ d th can be considered.

Original: Onthe other hand, if one wants to constrain the outage behavior of the policy π , then the constraint
Revised: Onthe other hand, if one wants to constrain the outage behavior of the policy π , then the constraint

Original: Pr ( X π ( s 0 ) > d th ) ≤ ϵ 0 should be considered, where X π ( s 0 ) is a random variable defined as
Revised: Pr ( X π ( s 0 ) > d th ) ≤ ϵ 0 should be considered, where X π ( s 0 ) is a random variable defined as

Original: X π ( s 0 ) :
Revised: X π ( s 0 ) :

Original: = (cid:80) ∞
Revised: = (cid:80) ∞

Original: t =

Revised: t =

Original: γ t c ( S t , A t ) with S 

Revised: γ t c ( S t , A t ) with S

Original: = s 0 , A t ∼ π ( ·| S t ) , S t +1 ∼ M ( ·| S t , A t ) for t = 0 , 1 , · · · .
Revised: = s 0 , A t ∼ π ( ·| S t ) , S t +1 ∼ M ( ·| S t , A t ) for t = 0 , 1 , · · · .

Original: Most of the previous constrained RL works considered a constraint on the expectation of cumulativesum cost: C π ( s 0 )
Revised: Most of the previous constrained RL works considered a constraint on the expectation of cumulativesum cost: C π ( s 0 )

Original: ≤ d th [1, 21, 20, 9, 17, 8, 18].
Revised: ≤ d th [1, 21, 20, 9, 17, 8, 18].

Original: In order to solve the constrained optimizationproblem with this expectation-based constraint, researchers considered Lagrangian multiplier methods[1, 14], Lyapunov-based methods [4, 5], projection-based methods [20], safety-layer methods [6].
Revised: In order to solve the constrained optimizationproblem with this expectation-based constraint, researchers considered Lagrangian multiplier methods[1, 14], Lyapunov-based methods [4, 5], projection-based methods [20], safety-layer methods [6].

Original: Quantile (i.e., Value at Risk, VaR) and Conditional Value at Risk (CVaR) are two well-knowntechniques to manage undesirable events in the domain of finance[11].
Revised: Quantile (i.e., Value at Risk, VaR) and Conditional Value at Risk (CVaR) are two well-knowntechniques to manage undesirable events in the domain of finance[11].

Original: In the context of RL, thedefinitions of quantile and CVaR for the distribution of the cumulative sum cost for a given π are givenby q πu ( s 0 ) :
Revised: In the context of RL, thedefinitions of quantile and CVaR for the distribution of the cumulative sum cost for a given π are givenby q πu ( s 0 ) :

Original: = inf { x | Pr ( X π ( s 0 ) ≤ x ) ≥ u } and CVaR πu ( s0 ) : =
Revised: = inf { x | Pr ( X π ( s 0 ) ≤ x ) ≥ u } and CVaR πu ( s

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 0 )] ,respectively.
Revised: respectively.

Original: CVaR was previously used in RL to constrain undesirable events.
Revised: CVaR was previously used in RL to constrain undesirable events.

Original: Chow et al.
Revised: Chow et al.

Original: [3]proposed a trajectory-based CVaR method and provided convergence for their method.
Revised: [3]proposed a trajectory-based CVaR method and provided convergence for their method.

Original: They usedtrajectory-based policy gradient to their Lagrangian and it is simple to compute.
Revised: They usedtrajectory-based policy gradient to their Lagrangian and it is simple to compute.

Original: Like most trajectorybased RL algorithms, however, it suffers from sample inefficiency since it collects a number oftrajectories and updates its parameter once.
Revised: Like most trajectorybased RL algorithms, however, it suffers from sample inefficiency since it collects a number oftrajectories and updates its parameter once.

Original: Recently, Yang et al.
Revised: Recently, Yang et al.

Original: [19] proposed an off-policyalgorithm to satisfy a CVaR constraint, and we refer to the constrained problem as (CVaR-CP).
Revised: [19] proposed an off-policyalgorithm to satisfy a CVaR constraint.

Original: Theexplicit formulation of the CVaR-CP problem is given in Appendix A.3.

Original: They only estimated the meanand variance of the cost distribution using a technique in distributional RL and compute CVaR πu ( s ) 1as CVaR of Gaussian distribution of the estimated mean and variance.
Revised: They only estimated the mean and variance of the costdistribution using a technique in distributional RL and compute CVaR πu ( s ) 1 as CVaR of Gaussiandistribution of the estimated mean and variance.

Original: However, the distributionof X π ( s ) is not Gaussian in general, and the Gaussian approximation  has limited capability tocapture the decay rate of the tail probability, because Gaussian probability density function (PDF)has the form of exp( − βx 2 ) with fixed rate function x 2 .
Revised: However, the distribution of X π ( s ) is not Gaussianin general, and the distribution approximation with a Gaussian distribution has limited capability tocapture the decay rate of the tail probability, because Gaussian probability density function (PDF)has the form of exp( − βx 2 ) with fixed rate function x 2 .

Original: Therefore, this algorithm can yield a poorestimation of the CVaR of the tail, especially for small tail probability, and cannot guarantee to satisfythe CVaR constraint (see Section 5).
Revised: Therefore, this algorithm can yield a poorestimation of the CVaR of the tail, especially for small tail probability, and cannot guarantee to satisfythe CVaR constraint (see Section 5).

Original: Furthermore, note that CVaR and quantile are two differentmeasures for undesirable events, and the choice between the two depends on what we desire.
Revised: Furthermore, note that CVaR and quantile are two differentmeasures for undesirable events, and the choice between the two depends on what we desire.

Original: Forexample, an insurance company prefers the CVaR of undesirable events to determine an insurancepremium.
Revised: Forexample, an insurance company prefers the CVaR of undesirable events to determine an insurancepremium.

Original: On the other hand, a company developing an autonomous driving car system needs thequantile of undesirable events to guarantee the accident probability for safety.
Revised: On the other hand, a company developing an autonomous driving car system needs thequantile of undesirable events to guarantee the accident probability for safety.

Original: Thus, in the contextof safe learning, our work focuses directly on the constraint on quantile , which is an equivalent(i.e., necessary and sufficient) constraint to the outage probability constraint in (ProbCP).
Revised: Thus, in the context of safe learning, our work focuses directly on the constraint on quantile , whichis an equivalent (i.e., necessary and sufficient) constraint to the outage probability constraint in(ProbCP).

Original: To thebest of our knowledge, this is the first work that provides a state-based policy gradient for quantileand required theoretical results regarding the quantile-constrained RL problem.
Revised: To the best of our knowledge, this is the first work that provides a state-based policygradient for quantile and required theoretical results regarding the quantile-constrained RL problem.

Original: Moreover, ourimplementation approximates the tail distribution of X π ( s ) with a Weibull distribution (a particularcase of generalized Gamma distribution), which is general enough to capture various rates of thedecay of the tail probability.
Revised: Moreover, our implementation approximates the tail distribution of X π ( s ) with a Weibull distribution(a particular case of generalized Gamma distribution), which is general enough to capture variousrates of the decay of the tail probability.

Original: Table 1 summarizes the previous constrained RL methods and this paper.
Revised: Table 1 summarizes the previous constrained RL methodsand this paper.

Original: [Table]
Revised: [Table]

Original: Large Deviation Principle (LDP) Large deviation principle (LDP) [7] is a technique for estimating the limiting behavior of a sequence of distributions especially on tail.
Revised: Large Deviation Principle (LDP) Large deviation principle (LDP) [7] is a technique for estimating the limiting behavior of a sequence of distributions especially on tail.

Original: A simple example is the empirical mean ¯ X n = 1 n (cid:80) [-n k =
-]
Revised: A simple example is the empirical mean ¯ X n = 1 n (cid:80) [-n k =
-]nk

Original:  X k of i.i.d.
Revised: =1 X k of i.i.d.

Original: random variables
Revised: random variables

Original: X i .
Revised: X i .

Original: We say that a sequence { ¯ X n } satisfies LDP if the sequence of its log probability distribution 1 n log Pr (cid:0) ¯ X n ∈ Γ (cid:1)satisfies the following condition 1 n log Pr (cid:0) ¯ X n ∈ Γ (cid:1) n →∞ −→ − inf x ∈ Γ
Revised: We say that a sequence { ¯ X n } satisfies LDP if the sequence of its log probability distribution 1 n log Pr (cid:0) ¯ X n ∈ Γ (cid:1)satisfies the following condition 1 n log Pr (cid:0) ¯ X n ∈ Γ (cid:1) n →∞ −→ − inf x ∈ Γ

Original: I ( x ) for some function I ( x ) .
Revised: I ( x ) for some function I ( x ) .

Original: The function I ( x ) satisfying such limiting behavior is called the rate function of ¯ X n .
Revised: The function I ( x ) satisfying such limiting behavior is called the rate function of ¯ X n .

Original: Therate function I ( x ) is also related to the cumulative distribution function (CDF)
Revised: Therate function I ( x ) is also related to the cumulative distribution function (CDF)

Original: F ¯ X n ( x ) since1 − F ¯ X n ( x 0 ) =
Revised: F ¯ X n ( x ) since1 − F ¯ X n ( x 0 ) =

Original: Pr (cid:0) ¯ X n ∈ [ x 0 , ∞ ) (cid:1) ≈ exp (cid:0) − n inf x ∈ [ x 0 , ∞ )
Revised: Pr (cid:0) ¯ X n ∈ [ x 0 , ∞ ) (cid:1) ≈ exp (cid:0) − n inf x ∈ [ x 0 , ∞ )

Original: I ( x ) (cid:1) for some x 0 > E
Revised: I ( x ) (cid:1) for some x 0 > E

Original: [ X ] and sufficiently large n .
Revised: [ X ] and sufficiently large n .

Original: LDP can be applied to finite-state Markov chains and there exists a rate function for agiven Markov chain [7].
Revised: LDP can be applied to finite-state Markov chains and there exists a rate function for agiven Markov chain [7].

Original: In this paper, we consider the tail probability of the distribution of the cumulative sum of costs X π ( s 0 ) =
Revised: In this paper, we consider the tail probability of the distribution of the cumulative sum of costs X π ( s 0 ) =

Original: (cid:80) ∞ t =

Revised: (cid:80) ∞ t =

Original: γ t c ( s t , a t ) .
Revised: γ t c ( s t , a t ) .

Original: Finding its analytic rate function is hard.
Revised: Finding its analytic rate function is hard.

Original: Therefore,we instead approximate the rate function directly as I X π ( s ) ( x ) ≈ ( x/β ( s ))
Revised: Therefore, we insteadapproximate the rate function directly as I X π ( s ) ( x ) ≈ ( x/β ( s ))

Original: α ( s ) with learnable parameters α ( s ) and β ( s ) , which results in a Weibull distribution: 1 − F X π ( s ) ( x ) = exp (cid:8) − ( x/β ( s ))
Revised: α ( s ) with learnable parameters α ( s )and β ( s ) , which results in a Weibull distribution: 1 − F X π ( s ) ( x ) = exp (cid:8) − ( x/β ( s ))

Original: α ( s ) (cid:9) .
Revised: α ( s ) (cid:9) .

Original: We use this distribution to approximate the tail probability of p X π ( s ) ( x ) of X π ( s ) .
Revised: We usethis distribution to approximate the tail probability of p X π ( s ) ( x ) of X π ( s ) .

Original:  Quantile Constrained RL
Revised:  Quantile Constrained RL

Original: In this section, we explain an equivalent form of (ProbCP) that we use to learn an optimal constrainedpolicy under the outage probability constraint and then explain the difficulty of applying the policygradient theorem to optimize the Lagrangian of the equivalent problem.
Revised: In this section, we explain an equivalent form of (ProbCP) that we use to learn an optimal constrainedpolicy under the outage probability constraint and then explain the difficulty of applying the policygradient theorem to optimize the Lagrangian of the equivalent problem.

Original: Finally, we provide theoreticalresults that circumvent this difficulty in Section 3.2.
Revised: Finally, we provide theoreticalresults that circumvent this difficulty in Section 3.2.

Original: 3.1 Motivation: Problem of 
Revised: 3.1 Motivation: Problem of Applying Policy Gradient Theorem to Quantile

Original: Applying Policy Gradient Theorem to Quantile

Original: Solving (ProbCP) with a direct approach is too hard in making a loss function for π based on theoutage probability.
Revised: Solving (ProbCP) with a direct approach is too hard in making a loss function for π based on theoutage probability.

Original: Thus, we convert the probability constrained problem to an equivalent form of aquantile constrained problem:
Revised: Thus, we convert the probability constrained problem to an equivalent form of aquantile constrained problem:

Revised: Maximize E π [ (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Subject to
Revised: Subject to

Original: [Equation]
Revised: [Equation]

Original: where q πu ( s ) = inf { x | Pr ( X π ( s ) ≤ x ) ≥ u } is the u -quantile of the random variable X π ( s ) ofcumulative sum costs: X π ( s ) =
Revised: where q πu ( s ) = inf { x | Pr ( X π ( s ) ≤ x ) ≥ u } is the u -quantile of the random variable X π ( s ) ofcumulative sum costs: X π ( s ) =

Original: (cid:80) ∞ t =

Revised: (cid:80) ∞ t =

Original: γ t c ( S t , A t ) with S 0 = s , A t ∼ π ( ·| S t ) , S t +1 ∼ M ( ·| S t , A t ) ,t = 0 , 1 , 2 , · · · .
Revised: γ t c ( S t , A t ) with S 0 = s , A t ∼ π ( ·| S t ) , S t +1 ∼ M ( ·| S t , A t ) ,t = 0 , 1 , 2 , · · · .

Original: Note that q π 1 − ϵ 0 ( s 0 ) ≤
Revised: Note that q π 1 − ϵ 0 ( s 0 ) ≤

Original: d th is equivalent to Pr [ X π ( s 0 ) >
Revised: d th is equivalent to Pr [ X π ( s 0 ) >

Original: d th ] < ϵ 0 due to thedefinition of quantile.
Revised: d th ] < ϵ 0 due to thedefinition of quantile.

Original: We propose a direct approach to solve the equivalent problem (QuantCP)instead of (ProbCP).
Revised: We propose a direct approach to solve the equivalent problem (QuantCP)instead of (ProbCP).

Original: Although we have an equivalent form of (ProbCP), it is still difficult to solve(QuantCP).
Revised: Although we have an equivalent form of (ProbCP), it is still difficult to solve(QuantCP).

Original: We explain what makes solving the problem (QuantCP) still hard below.
Revised: We explain what makes solving the problem (QuantCP) still hard below.

Original: In the case of (ExpCP), the Lagrange-based optimization of (ExpCP) is given by[Equation] [Equation] − λ
Revised: In the case of (ExpCP), the Lagrange-based optimization of (ExpCP) is given by

Original: [Equation]
Revised: [Equation]

Original: s 0 ) − d th ) , where V π ( s 0 ) =
Revised: th ) , where V π ( s 0 ) =

Original: E π [ (cid:80) ∞ t =

Revised: E π [ (cid:80) ∞ t =

Original: γ t r ( s t , a t )]and C π ( s 0 ) =
Revised: γ t r ( s t , a t )]and C π ( s 0 ) =

Original: E π [ (cid:80) ∞ t =

Revised: E π [ (cid:80) ∞ t =

Original: γ t c ( s t , a t )] .
Revised: γ t c ( s t , a t )] .

Original: Then, due to the form of the cost, the policy gradient theorem[15] can directly be applied, and the gradient the Lagrangian w.r.t.
Revised: Then, due to the form of the cost, the policy gradient theorem[15] can directly be applied, and the gradient the Lagrangian w.r.t.

Original: π is given by the expectation form:
Revised: π is given by the expectation form:

Original: [Equation]
Revised: [Equation]

Original:  (2) where ρ π ( s ) :
Revised: a ) } , (2) where ρ π ( s ) :

Original: = (cid:80) ∞
Revised: = (cid:80) ∞

Original: t =0 γ t Pr ( S t = s | s 0 , π ) , A πr ( s, a ) :
Revised: t =0 γ t Pr ( S t = s | s 0 , π ) , A π r ( s, a ) :

Original: = r ( s, a ) + γ E s ′ ∼ M
Revised: = r ( s, a ) + γ E s ′ ∼ M

Original: [ V π ( s ′ )] − V π ( s ) , and
Revised: [ V π ( s ′ )] − V π ( s ) , and

Original: A πc ( s, a ) :
Revised: A πc ( s, a ) :

Original: =
Revised: =

Original: c ( s, a ) + γ E s ′ ∼ M
Revised: c ( s, a ) + γ E s ′ ∼ M

Original: [ C π ( s ′ )] − C π ( s ) .
Revised: [ C π ( s ′ )] − C π ( s ) .

Original: However, the gradient of the Lagrangian in theequivalent problem of (QuantCP)
Revised: However, the gradient of the Lagrangian in theequivalent problem of (QuantCP)

Original: [Equation]
Revised: [Equation]

Original:  w.r.t.
Revised: (3) w.r.t.

Original: the policy π cannot be expressed as an expectation form:
Revised: the policy π cannot be expressed as an expectation form:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ¯ A πu ( s, a ) :
Revised: where ¯ A πu ( s, a ) :

Original: =
Revised: =

Original: c ( s, a ) + γ E s ′ ∼ M
Revised: c ( s, a ) + γ E s ′ ∼ M

Original: [ q πu ( s ′ )]
Revised: [ q πu ( s ′ )]

Original: − q πu ( s ) .
Revised: − q πu ( s ) .

Original: This is because the u -quantile q πu ( s ) is notthe expectation of the cumulative sum of costs.
Revised: This is because the u -quantile q πu ( s ) is notthe expectation of the cumulative sum of costs.

Original: However, if the u -quantile q πu ( s ) can be written as
Revised: However, if the u -quantile q πu ( s ) can be written as

Original: [Equation]
Revised: [Equation]

Original: for some function ˜ c u ( s, a ) that is independent of the policy π , we can apply the policy gradienttheorem by defining the advantage function for the quantile term:
Revised: for some function ˜ c u ( s, a ) that is independent of the policy π , we can apply the policy gradienttheorem by defining the advantage function for the quantile term:

Original: [Equation]
Revised: [Equation]

Revised: ′ )] −

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This fact motivates us to search for such ˜ c u ( s, a ) .
Revised: This fact motivates us to search for such ˜ c u ( s, a ) .

Original: For this, we first show the existence of a policydependent additional cost ˜ c πu ( s, a ) , and then show that under a mild assumption, the additional costcan be approximated by a cost ˜
Revised: For this, we first show the existence of a policydependent additional cost ˜ c πu ( s, a ) , and then show that under a mild assumption, the additional costcan be approximated by a cost ˜

Original: c π ′ u ( s, a ) for some fixed π ′ independent of π except the requirementmax s KL ( π ′ ( ·| s ) ∥
Revised: c π ′ u ( s, a ) for some fixed π ′ independent of π except the requirementmax s KL ( π ′ ( ·| s ) ∥

Original: π ( ·| s )) ≤ δ .
Revised: π ( ·| s )) ≤ δ .

Original: 3.2 Theoretical Results
Revised: 3.2 Theoretical Results

Original: We here provide theoretical results showing the existence of an additional cost ˜ c πu ( s, a ) and showingthat under a mild assumption, this can be approximated as another cost ˜
Revised: We here provide theoretical results showing the existence of an additional cost ˜ c πu ( s, a ) and showingthat under a mild assumption, this can be approximated as another cost ˜

Original: c π ′ u ( s, a ) for a base policy π ′independent of π only requiring max s KL (cid:0) π ′ ( ·| s ) (cid:13)(cid:13) π ( ·| s ) (cid:1) ≤ δ for some δ > 0 .
Revised: c π ′ u ( s, a ) for a base policy π ′independent of π only requiring max s KL (cid:0) π ′ ( ·| s ) (cid:13)(cid:13) π ( ·| s ) (cid:1) ≤ δ for some δ > 0 .

Original: These theoreticalresults make the quantile constrained policy optimization tractable by enabling application of thepolicy gradient theorem.
Revised: These theoreticalresults make the quantile constrained policy optimization tractable by enabling application of thepolicy gradient theorem.

Original: We assume that the CDF F X π ( s ) ( x ) is strictly increasing on [0 , ∞ ) and it iscontinuously differentiable for all s ∈ S , and the proofs of the theoretical results are in Appendix B.
Revised: We assume that the CDF F X π ( s ) ( x ) is strictly increasing on [0 , ∞ ) and it iscontinuously differentiable for all s ∈ S , and the proofs of the theoretical results are in Appendix B.

Original: We begin with deriving the temporal-difference (TD) relation between the u -quantiles of X π ( s ) at s tand s t +1 .
Revised: We begin with deriving the temporal-difference (TD) relation between the u -quantiles of X π ( s ) at s tand s t +1 .

Original: Theorem 1 states the TD relation for the u -quantile.
Revised: Theorem 1 states the TD relation for the u -quantile.

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumptions 2 and 3 in Appendix B.1 stating the boundedness of quantile andsmoothness of the CDF of X π ( s t ) , the u -quantile of random variable X π ( s t ) satisfies the followingtemporal-difference(TD) relation.
Revised: Under mild assumptions, the u -quantile of random variable X π ( s t ) satisfies thefollowing temporal-difference(TD) relation.

Original: For some constant R and small ϵ > 0 ,
Revised: For some constant R and small ϵ > 0 ,

Original: [Equation]
Revised: [Equation]

Revised: ϵ

Revised: [Equation]

Original:  where µ πu ( s, a, s ′ )
Revised: (7) where µ πu ( s, a, s ′ )

Original: =
Revised: =

Original: p X π ( s ′ ) (cid:16) q πu ( s ) − c
Revised: p X π ( s ′ ) (cid:16) q πu ( s ) − c

Original: ( s,a ) γ (cid:17) (cid:14) γp X π ( s ) ( q πu ( s )) .
Revised: ( s,a ) γ (cid:17) (cid:14) γp X π ( s ) ( q πu ( s )) .

Original: Here, the expectation is for theaction a t ∼ π ( ·| s t ) and the next state s t +1 ∼ M ( ·| s t , a t ) .
Revised: Here, the expectation is for theaction a t ∼ π ( ·| s t ) and the next state s t +1 ∼ M ( ·| s t , a t ) .

Original: ( s t is given.)
Revised: ( s t is given.)

Original: Proof is in Appendix B. Note that for the expectation of cumulative sum cost C π ( s ) considered in(ExpCP), the expectation of TD under the policy π follows
Revised: The mild assumptions and proof are in Appendix B. Note that for the expectation of cumulative sumcost C π ( s ) considered in (ExpCP), the expectation of TD under the policy π follows

Original: [Equation]
Revised: [Equation]

Original: by the Bellman equation for the sum cost.
Revised: by the Bellman equation for the sum cost.

Original: The TD relation (8) for expectation has a similarform to that for the u -quantile (7), but the difference is that (7) is the weighted expectation ofthe TD ( c ( s t , a t ) + γq πu ( s t +1 )
Revised: The TD relation (8) for expectation has a similarform to that for the u -quantile (7), but the difference is that (7) is the weighted expectation ofthe TD ( c ( s t , a t ) + γq πu ( s t +1 )

Original: − q πu ( s t ) ).
Revised: − q πu ( s t ) ).

Original: The numerator p X π ( s t +1 ) (cid:16) q πu ( s t ) − c ( s t ,a t ) γ (cid:17) of the weightµ πu ( s t , a t , s t +1 ) in (7) involves two quantities: 1) a target u -quantile q πu ( s t ) − c
Revised: The numerator p X π ( s t +1 ) (cid:16) q πu ( s t ) − c ( s t ,a t ) γ (cid:17) of the weightµ πu ( s t , a t , s t +1 ) in (7) involves two quantities: 1) a target u -quantile q πu ( s t ) − c

Original: ( s t ,a t ) γ and 2) the PDF ofthe sum of costs
Revised: ( s t ,a t ) γ and 2) the PDF ofthe sum of costs

Original: X π ( s t +1 )
Revised: X π ( s t +1 )

Original: = (cid:80) ∞
Revised: = (cid:80) ∞

Original: k =0 γ k c ( s t + k +1 , a t + k +1 ) starting from state s t +1 .
Revised: k =0 γ k c ( s t + k +1 , a t + k +1 ) starting from state s t +1 .

Original: Here, the valueq πu ( s t ) − c ( s t ,a t ) γ is the target value of the sum of costs (cid:80) ∞ k =

Revised: Here, the valueq πu ( s t ) − c ( s t ,a t ) γ is the target value of the sum of costs (cid:80) ∞ k =

Original: γ k c ( s t + k +1 , a t + k +1 ) from the next states t +1 such that the sum of costs (cid:80) ∞ k =

Revised: γ k c ( s t + k +1 , a t + k +1 ) from the next states t +1 such that the sum of costs (cid:80) ∞ k =

Original: γ k c ( s t + k , a t + k ) for a given pair ( s t , a t ) at t is the u -quantileq πu ( s t ) .Thus, the numerator p X π ( s t +1 ) (cid:16) q πu ( s t ) − c [Equation]
Revised: γ k c ( s t + k , a t + k ) for a given pair ( s t , a t ) at t is the u -quantileq πu ( s t ) .

Original:  s t [Equation] a t ) γ (cid:17) of the weight µ πu ( s t , a t , s t +1 ) in (7) is theprobability of the event that the sum cost starting from ( s t , a t ) becomes the u -quantile q πu ( s t ) at s tfrom the perspective of the next state s t +1 .
Revised: Thus, the numerator p X π ( s t +1 ) (cid:16) q πu ( s t ) − c ( s t ,a t ) γ (cid:17) of the weight µ πu ( s t , a t , s t +1 ) in (7) is theprobability of the event that the sum cost starting from ( s t , a t ) becomes the u -quantile q πu ( s t ) at s tfrom the perspective of the next state s t +1 .

Original: Based on Theorem 1, we obtain the following corollary:
Revised: Based on Theorem 1, we obtain the following corollary:

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the same  assumptions as Theorem 1, the u -quantile q πu ( s t ) of random variable X π ( s t ) is bounded as
Revised: Under the same mild assumptions as Theorem 1, the u -quantile q πu ( s t ) of randomvariable X π ( s t ) is bounded as

Original: [Equation]
Revised: [Equation]

Original: Proof: Note that the term q πu ( s t ) can go outside the expectation in (7) since the expectation is over( a t , s t +1 ) .
Revised: Proof: Note that the term q πu ( s t ) can go outside the expectation in (7) since the expectation is over( a t , s t +1 ) .

Original: From eq.
Revised: From eq.

Original: (3) in Appendix A, the expectation of the numerator of µ πu ( s t , a t , s t +1 )is the same as the the denominator of the weight, i.e., E π (cid:104)
Revised: (3) in Appendix A, the expectation of the numerator of µ πu ( s t , a t , s t +1 )is the same as the the denominator of the weight, i.e., E π (cid:104)

Original: p X π ( s t +1 ) (cid:16) q πu ( s t ) − c
Revised: p X π ( s t +1 ) (cid:16) q πu ( s t ) − c

Original: ( s t ,a t ) γ (cid:17)(cid:105) =
Revised: ( s t ,a t ) γ (cid:17)(cid:105) =

Original: γp X π ( s t ) ( q πu ( s t )) and this leads to E π [ µ πu ( s t , a t , s t +1 )]
Revised: γp X π ( s t ) ( q πu ( s t )) and this leads to E π [ µ πu ( s t , a t , s t +1 )]

Original: = 1 .
Revised: = 1 .

Original: So, we have the claim.
Revised: So, we have the claim.

Original: □
Revised: □

Original: As seen in Corollary 1, the u -quantile at s t can be approximated as a weighted expectation of [Equation] , and the weight is proportional to p X π ( s t +1 ) (cid:16) q πu ( s t ) − c ( s t ,a t ) γ (cid:17) .
Revised: As seen in Corollary 1, the u -quantile at s t can be approximated as a weighted expectation ofc ( s t , a t ) + γq πu ( s t +1 ) , and the weight is proportional to p X π ( s t +1 ) (cid:16) q πu ( s t ) − c ( s t ,a t ) γ (cid:17) .

Original: This meansthat the more probable is the pair ( s t , a t , s t +1 ) to achieve q πu ( s t ) , the higher weight is multipliedto c ( s t , a t )
Revised: This meansthat the more probable is the pair ( s t , a t , s t +1 ) to achieve q πu ( s t ) , the higher weight is multipliedto c ( s t , a t )

Original: + γq πu ( s t +1 ) for approximating q πu ( s t ) .
Revised: + γq πu ( s t +1 ) for approximating q πu ( s t ) .

Original: Furthermore, if we assume that the transitiondynamics of CMDP are deterministic, i.e., s t +1 =
Revised: Furthermore, if we assume that the transitiondynamics of CMDP are deterministic, i.e., s t +1 =

Original: h ( s t , a t ) as in many real-world control problem,we can approximate the u -quantile q πu ( s 0 ) at s 0 as the expectation of the sum of costs under adistorted policy ˜ π u , as stated in the following lemma:
Revised: h ( s t , a t ) as in many real-world control problem,we can approximate the u -quantile q πu ( s 0 ) at s 0 as the expectation of the sum of costs under adistorted policy ˜ π u , as stated in the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose that the state transition dynamics are deterministic, i.e., s t
Revised: Suppose that the state transition dynamics are deterministic, i.e., s t

Original: +1 =
Revised: +1 =

Original: h ( s t , a t ) .
Revised: h ( s t , a t ) .

Original: Then,under the same assumptions as Theorem 1, the u -quantile q πu ( s 0 ) of random variable X π ( s 0 ) is expressed as
Revised: Then,under the same assumptions as Theorem 1, the u -quantile q πu ( s 0 ) of random variable X π ( s 0 ) isexpressed as

Original: [Equation]
Revised: [Equation]

Original: where ˜ π u ( a | s )
Revised: where ˜ π u ( a | s )

Original: =
Revised: =

Original: π ( a | s ) · µ πu ( s, a, h( s, a ))
Revised: π ( a | s ) [Equation] µ πu ( s, a, h

Revised: [Equation]

Original:  ∝ π ( a | s ) · p X π ( h ( s,a )) (cid:16) q πu ( s ) − c ( s,a ) γ (cid:17) .
Revised: s, a [Equation] ∝ π ( a [Equation]

Revised: s

Revised: [Equation]

Revised: p X

Original:  Now, plugging (10) into the quantile term in the Lagrangian (3) of the problem (QuantCP), we mayapply the policy gradient theorem based on the chain rule since the u -quantile q πu ( s 0 ) is expressed asthe expectation of sum of costs.
Revised: [Equation] s [Equation] Now, plugging (10) into the quantile term in the Lagrangian (3) of the problem (QuantCP), we mayapply the policy gradient theorem based on the chain rule since the u -quantile q πu ( s 0 ) is expressed asthe expectation of sum of costs.

Original: However, the gradient of ˜ π u w.r.t.
Revised: However, the gradient of ˜ π u w.r.t.

Original: π for chain rule is too complicateddue to the µ πu term in Lemma 1.
Revised: π for chain rule is too complicateddue to the µ πu term in Lemma 1.

Original: Thus, we find another expectation form of q πu ( s 0 ) using an additionalcost function ˜
Revised: Thus, we find another expectation form of q πu ( s 0 ) using an additionalcost function ˜

Original: c
Revised: c

Original: πu ( s, a ) , as stated in the following theorem:
Revised: πu ( s, a ) , as stated in the following theorem:

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose that the state transition dynamics are deterministic, i.e., s t
Revised: Suppose that the state transition dynamics are deterministic, i.e., s t

Original: +1 =
Revised: +1 =

Original: h ( s t , a t ) .
Revised: h ( s t , a t ) .

Original: Then, the same assumptions as Theorem 1, q πu ( s ) can be expressed as
Revised: Then, the same assumptions as Theorem 1, q πu ( s ) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where ˜ c π u ( s, a ) =
Revised: where ˜ c πu ( s, a ) =

Original: ( µ π u ( s, a, h ( s, a )) − 1) · [ c ( s, a ) + γq π u ( h ( s, a ))] .
Revised: ( µ πu ( s, a, h ( s, a )) − 1) · [ c ( s, a ) + γq πu ( h ( s, a ))] .

Original: Note that the additional cost ˜ c πu ( s t , a t ) in Theorem 2 is a policy-dependent cost function.
Revised: Note that the additional cost ˜ c πu ( s t , a t ) in Theorem 2 is a policy-dependent cost function.

Original: Undera mild assumption, we can find an upper bound of (11) which replaces the policy-dependent costfunction ˜
Revised: Undera mild assumption, we can find an upper bound of (11) which replaces the policy-dependent costfunction ˜

Original: c πu ( s, a ) with another cost ˜ c
Revised: c πu ( s, a ) with another cost ˜ c

Original: π ′ u ( s, a ) for some fixed π ′ independent of π only requiringmax s KL (cid:0) π ′ ( ·| s ) (cid:13)(cid:13) π ( ·| s ) (cid:1) ≤ δ for some δ > 0 .
Revised: π ′ u ( s, a ) for some fixed π ′ independent of π only requiringmax s KL (cid:0) π ′ ( ·| s ) (cid:13)(cid:13) π ( ·| s ) (cid:1) ≤ δ for some δ > 0 .

Original: The assumption is as follows:
Revised: The assumption is as follows:

Original: Assumption 1.
Revised: Assumption 1.

Original: For any given fixed u ∈ (0 , 1) and any policies π and π ′ , there exists a coefficient
Revised: For any given fixed u ∈ (0 , 1) and any policies π and π ′ , there exists a coefficient

Original: C u such that
Revised: C u such that

Original: [Equation]
Revised: [Equation]

Original: Basically, Assumption 1 is that the function ˜ c πu as a function of π is continuous, which is expectedto be satisfied if there is no abrupt change in the associated distributions.
Revised: Basically, Assumption 1 is that the function ˜ c πu as a function of π is continuous, which is expectedto be satisfied if there is no abrupt change in the associated distributions.

Original: With Assumption 1 and
Revised: With Assumption 1 and

Original: Theorem 2, we obtain an expression for the quantile q πu ( s 0 ) as a form of desired expected sum:
Revised: Theorem 2, we obtain an expression for the quantile q πu ( s 0 ) as a form of desired expected sum:

Original: Theorem 3.
Revised: Theorem 3.

Original: Under deterministic dynamics s t +1 =
Revised: Under deterministic dynamics s t +1 =

Original: h ( s t , a t ) , Assumption 1, and the same assumptions as Theorem 1, the u -quantile q πu ( s 0 ) is expressed as the expectation of the sum of actual costand a π -independent additional cost ˜
Revised: h ( s t , a t ) , Assumption 1, and the same assumptions as Theorem 1, the u -quantile q πu ( s 0 ) is expressed as the expectation of the sum of actual costand a π -independent additional cost ˜

Original: c π ′ u ( s, a ) for π ′ satisfying max s KL ( π ′ 
Revised: c π ′ u ( s, a ) for π ′ satisfying max s KL ( π ′ ( ·| s )

Original: [Equation] ) ≤ [Equation]
Revised: ∥ π ( ·| s )) ≤ δ :

Original: [Equation]
Revised: [Equation]

Original: By Theorem 3, we can approximate the u -quantile q π u ( s 0 ) as the expectation of sum of costs plusπ -independent additional costs ˜ c
Revised: By Theorem 3, we can approximate the u -quantile q πu ( s 0 ) as the expectation of sum of costs plusπ -independent additional costs ˜ c

Original: π ′ u ( s, a ) for a base policy π ′ , and this approximation is tighter whenthe distance between the current policy π and the base policy π ′ is smaller.
Revised: π ′ u ( s, a ) for a base policy π ′ , and this approximation is tighter whenthe distance between the current policy π and the base policy π ′ is smaller.

Original: Now, this result enablesus to solve the quantile constrained problem (QuantCP) by applying the policy gradient theorem.
Revised: Now, this result enablesus to solve the quantile constrained problem (QuantCP) by applying the policy gradient theorem.

Original:  Quantile Constrained Policy Optimization
Revised:  Quantile Constrained Policy Optimization

Original: Using the theoretical results in Section 3, we now construct an algorithm named quantile constrainedpolicy optimization (QCPO) to solve (QuantCP) based on an on-policy RL algorithm: PPO [13].
Revised: Using the theoretical results in Section 3, we now construct an algorithm named quantile constrainedpolicy optimization (QCPO) to solve (QuantCP) based on an on-policy RL algorithm: PPO [13].

Original: The
Revised: The

Original: QCPO is a direct method to constrain the outage probability, and consists of three parts: 1) estimationof the u -quantile q πu ( s ) of X π ( s ) for a given policy π , 2) estimation of PDF of X π ( s ) to computethe additional cost ˜
Revised: QCPO is a direct method to constrain the outage probability, and consists of three parts: 1) estimationof the u -quantile q πu ( s ) of X π ( s ) for a given policy π , 2) estimation of PDF of X π ( s ) to computethe additional cost ˜

Original: c π ′ u ( s, a ) , and 3) updating method of the Lagrange multiplier to control the outageprobability.
Revised: c π ′ u ( s, a ) , and 3) updating method of the Lagrange multiplier to control the outageprobability.

Original: We first explain the overall structure of QCPO and the base loss function for policywhich has a similar form to that in [12].
Revised: We first explain the overall structure of QCPO and the base loss function for policywhich has a similar form to that in [12].

Original: Then, we provide a condition for policy improvement for theproposed method.
Revised: Then, we provide a condition for policy improvement for theproposed method.

Original: The implementation of the proposed algorithm is based on the implementation of[14]  and details of the implementation including network structure, loss function
Revised: The implementation of the proposed algorithm is based on the implementation of[14] 2 , and details of the implementation including network structure, loss function, Lagrange updatemethod, and hyper-parameters are in Appendix E.

Original: 4.1 Overall Structure of QCPO
Revised: 4.1 Overall Structure of QCPO

Original: The agent of QCPO uses function approximators for policy π , value function V π ( s ) =
Revised: The agent of QCPO uses function approximators for policy π , value function V π ( s ) =

Original: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] and quantile function q π ( s )
Revised: E π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] and quantile function q π ( s )

Original: = (cid:104)
Revised: = (cid:104)

Original: q πu 1 ( s ) , q πu 2 ( s ) , . . .
Revised: q πu 1 ( s ) , q πu 2 ( s ) , . . .

Original: , q πu nq ( s ) (cid:105) of X π ( s ) =(cid:80) ∞ t =

Revised: , q πu nq ( s ) (cid:105) of X π ( s ) =(cid:80) ∞ t =

Original: γ t c ( s t , a t ) for policy π .
Revised: γ t c ( s t , a t ) for policy π .

Original: These functions are parameterized by deep neural networks withparameters θ , ϕ , and ψ , respectively.
Revised: These functions are parameterized by deep neural networks withparameters θ , ϕ , and ψ , respectively.

Original: We denote θ old , ϕ old , and ψ old as their old parameters.
Revised: We denote θ old , ϕ old , and ψ old as their old parameters.

Original: Note thatthe quantile function q ψ ( s ) outputs n q values and the i -th value represents an estimate of u i -quantileq
Revised: Note thatthe quantile function q ψ ( s ) outputs n q values and the i -th value represents an estimate of u i -quantileq

Original: πu i ( s ) of X π ( s ) for fixed target CDF values (cid:2)
Revised: πu i ( s ) of X π ( s ) for fixed target CDF values (cid:2)

Original: u 1 , u 2 , . .
Revised: u 1 , u 2 , . .

Original: , u n q (cid:3) with u 1 < u 2 < · · · < u n q .
Revised: , u n q (cid:3) with u 1 < u 2 < · · · < u n q .

Original: Inaddition to these parameterized functions, we need another function that approximates the PDFof X π ( s ) on the right tail.
Revised: Inaddition to these parameterized functions, we need another function that approximates the PDFof X π ( s ) on the right tail.

Original: (Here, the right-tail probability most matters since the target outageprobability is typically small.)
Revised: (Here, the right-tail probability most matters since the target outageprobability is typically small.)

Original: [Figure]
Revised: [Figure]

Original: As aforementioned in Section 2, we knowthat X
Revised: As aforementioned in Section 2, we knowthat X

Original: π ( s ) follows LDP with a rate function I X
Revised: π ( s ) follows LDP with a rate function I X

Original: π ( s ) ( x ) .
Revised: π ( s ) ( x ) .

Original: However, finding the ratefunction I X π ( s ) ( x ) in an analytic approach 259 is hard.
Revised: However, finding the ratefunction I X π ( s ) ( x ) in an analytic approach 260 is hard.

Original: Therefore, in QCPO, the agentapproximates the rate function of the form
Revised: Therefore, in QCPO, the agentapproximates the rate function of the form

Original: I X π ( s ) ( x ) ≈
Revised: I X π ( s ) ( x ) ≈

Original: ( x/β ( s ))
Revised: ( x/β ( s ))

Original: α ( s ) and learns the statedependent parameters α ( s ) and β ( s ) by using the quantiles on the right tail approximated by its quantile function for the right tail:q πu nq − k +1 ( s ) , . . . , q πu nq ( s ) .
Revised: α ( s ) and learns the statedependent parameters α ( s ) and β ( s ) by using the quantiles on the right tail approximated by its quantile function for the right tail:q πu nq − k +1 ( s ) , . . . , q πu nq ( s ) .

Original: This approximationof the rate function results in approximationprobability on the right tail as a Weibull distribution, whose tail distribution is 1 − F X π ( s ) ( x ) =
Revised: This approximationof the rate function results in approximationprobability on the right tail as a Weibull distribution, whose tail distribution is 1 − F X π ( s ) ( x ) =

Original: e − ( x/β ( s ))
Revised: e − ( x/β ( s ))

Original: α ( s ) .
Revised: α ( s ) .

Original: In order to obtain the statedependent parameters α ( s ) and β ( s ) for righttail distribution approximation, we again parameterize them with parameters ξ and ζ , respectively.
Revised: In order to obtain the statedependent parameters α ( s ) and β ( s ) for righttail distribution approximation, we again parameterize them with parameters ξ and ζ , respectively.

Original: Fig.
Revised: Fig.

Original: 2 shows both quantile approximation and right-tail approximation of QCPO.
Revised: 2 shows both quantile approximation and right-tail approximation of QCPO.

Original: Note thatour approach actually learns the rate function governing the tail-probability decay rate, whereas theprevious Gaussian approximation [19] on the PDF of X π ( s ) fixes the rate function as quadratic x 2 ,which is not the correct rate function in general.
Revised: Note thatour approach actually learns the rate function governing the tail-probability decay rate, whereas theprevious Gaussian approximation [19] on the PDF of X π ( s ) fixes the rate function as quadratic x 2 ,which is not the correct rate function in general.

Original: The overall procedure of QCPO is as follows: 1) estimate value function V π ( s ) for return andestimate quantile function q πu ( s ) , u ∈ (cid:2)
Revised: The overall procedure of QCPO is as follows: 1) estimate value function V π ( s ) for return andestimate quantile function q πu ( s ) , u ∈ (cid:2)

Original: u 1 , u 2 , . .
Revised: u 1 , u 2 , . .

Original: , u n q (cid:3) for cumulative sum costs, 2) approximatetail distribution p X π ( s ) ( x ) on the right tail using weibull distribution with parameter α ( s ) , β ( s ) , 3)compute additional cost for the base policy π ′ , ˜ c
Revised: , u n q (cid:3) for cumulative sum costs, 2) approximatetail distribution p X π ( s ) ( x ) on the right tail using weibull distribution with parameter α ( s ) , β ( s ) , 3)compute additional cost for the base policy π ′ , ˜ c

Original: π ′ 1 − ϵ 

Revised: π ′ 1 − ϵ

Original: ( s, a ) for quantile advantage
Revised: ( s, a ) for quantile advantage

Original: A π 1 − ϵ 

Revised: A π 1 − ϵ

Original: ( s, a ) :
Revised: ( s, a ) :

Original: =
Revised: =

Original: c ( s, a ) + ˜ c π ′ 1 − ϵ 

Revised: c ( s, a ) + ˜ c π ′ 1 − ϵ

Original: ( s, a ) +
Revised: ( s, a ) +

Original: γq π 1 − ϵ 0 ( s ′ ) − q π 1 − ϵ 0 ( s ) , 4) take policy gradient using the sum of valueadvantage and quantile advantage
Revised: γq π 1 − ϵ 0 ( s ′ ) − q π 1 − ϵ 0 ( s ) , 4) take policy gradient using the sum of valueadvantage and quantile advantage

Original: A r ( s, a ) − λA π 1 − ϵ 0 ( s, a ) , 4) the Lagrange multiplier λ update.
Revised: A r ( s, a ) − λA π 1 − ϵ 0 ( s, a ) , 4) the Lagrange multiplier λ update.

Original: Since QCPO is based on PPO [13], the loss functions for policy and value function are similar tothose of PPO [13].
Revised: Since QCPO is based on PPO [13], the loss functions for policy and value function are similar tothose of PPO [13].

Original: Please see in Appendix E.2and E.3 for detail.
Revised: Please see in Appendix E.2 for detail.

Original: 4.2 Policy Loss Function and Policy Improvement Condition
Revised: 4.2 Policy Loss Function and Policy Improvement Condition

Original: Let us consider the policy loss function of QCPO to solve QuantCP.
Revised: Let us consider the policy loss function of QCPO to solve QuantCP.

Original: The basic loss function of QCPOfor a given Lagrange multiplier λ is given by
Revised: The basic loss function of QCPOfor a given Lagrange multiplier λ is given by

Original: [Equation]
Revised: [Equation]

Original: (14) where
Revised: 14) where

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 17) and π old :
Revised: and π old :

Original: = π θ old is the policy that collects the most recent batch of samples, ρ π old ( s ) := 290 (cid:80) ∞t =
Revised: = π θ old is the policy that collects the most recent batch of samples, ρ π old ( s ) : = 291 (cid:80) ∞

Original: [Equation] Pr ( S t = s | s 0 , π old ) is the stationary state distribution under π old , ˜ C 1 is a constant (see
Revised: t =0 γ t Pr ( S t = s | s 0 , π old ) is the stationary state distribution under π old , ˜ C 1 is a constant (see

Original: Appendix B.5), Now, we consider the relationship between the actually-desired maximization objective L quant ( π, λ ) = 
Revised: Appendix B.5), Now, we consider the relationship between the actually-desired maximization objective L quant ( π, λ ) = V π ( s 0 ) − λ (cid:0) [Equation] s

Original: V π ( s 0 ) − λ (cid:0) q π 1 − ϵ 0 ( s 0 ) − d th (cid:1) in (3) and the practical QCPO objective 
Revised: [Equation] ) − d th (cid:1) in (3) and the practical QCPO objective L π old [Equation]

Revised: θ

Revised: [Equation]

Revised: −

Original: L π old ( θ ) − ˜ C 1 max s KL ( π old (·| s ) ∥
Revised: [Equation] max [Equation] π old (

Revised: [Equation]

Original:  π θ ( ·| s )) in (14).
Revised: | [Equation] ∥ π θ ( ·| s )) in (14).

Original: The relationship between the two is given bythe following theorem.
Revised: The relationship between the two is given bythe following theorem.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let π new :
Revised: Let π new :

Original: = π θ new be the solution of the problem of maximizing
Revised: = π θ new be the solution of the problem of maximizing

Original: [Equation]
Revised: [Equation]

Original: 18) for some constant ˜ C 1 > 0 .
Revised: (18) for some constant ˜ C 1 > 0 .

Original: Then, under deterministic dynamics s t +1 =
Revised: Then, under deterministic dynamics s t +1 =

Original: h ( s t , a t ) , Assumption 1, andthe same assumptions as Theorem 1, the following inequality holds:
Revised: h ( s t , a t ) , Assumption 1, andthe same assumptions as Theorem 1, the following inequality holds:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  for a given Lagrange multiplier λ > 0 , some constant ˜ C 2 and small 0 < ϵ .
Revised: (20) for a given Lagrange multiplier λ > 0 , some constant ˜ C 2 and small 0 < ϵ .

Original: Note that the term ˜ C 2 ϵ 1 − ϵ in (20) is due to our approximation of the quantile as an expected sum toapply policy gradient.
Revised: Note that the term ˜ C 2 ϵ 1 − ϵ in (20) is due to our approximation of the quantile as an expected sum toapply policy gradient.

Original: Therefore, by Theorem 4, when the improvement L π old ( π new ) − L π old ( π old ) −˜ C 1 KL max ( π old || π new ) ( > 0) by policy update from the QCPO loss function is large enough tocompensate for the approximation loss, the desired quantity will also be improved by our policyupdate.
Revised: Therefore, by Theorem 4, when the improvement L π old ( π new ) − L π old ( π old ) −˜ C 1 KL max ( π old || π new ) ( > 0) by policy update from the QCPO loss function is large enough tocompensate for the approximation loss, the desired quantity will also be improved by our policyupdate.

Original: That is, the Lagrangian for the quantile-constrained problem for π new will be higher thanthat for π old .
Revised: That is, the Lagrangian for the quantile-constrained problem for π new will be higher thanthat for π old .

Original:  Experiments
Revised:  Experiments

Original: [Figure]
Revised: [Figure]

Original: We examined the performance [Figure]
Revised: We examined the performance of the proposed

Original: of the proposed QCPO and compared it to that of WCSAC whichuses the CVaR constraint.
Revised: QCPO and compared it to that of WCSAC whichuses the CVaR constraint.

Original: Theenvironments we considered are SimpleButtonEnv, DynamicEnv [19] and GremlinEnv, which arebased on Safety Gym [10], MuJoCo [16], and OpenAI Gym [2].
Revised: The environmentswe considered are DynamicEnv-v0 [19] and SimpleButtonEnv-v0, which are based on Safety Gym [10], MuJoCo [16], and OpenAI Gym [2].

Original: The environmentscan be considered as simplified versions of a real environment of automatic serving robot, and are illustratedin Fig.
Revised: The environments are illustrated in Fig.

Original: The goal of these environments are for a robot (red sphere) to reach a goal (orangesphere wrapped by a grey translucent pillar, or green pillar) while avoiding the non-goal button(orange sphere), hazards (blue circle) or moving gremlins (purple box).
Revised: Thegoal of these environments are for a robot (red sphere) to reach a goal (green pillar for DynamicEnvv0, orange sphere wrapped by a grey translucent pillar for SimpleButtonEnv-v0) while avoidinghazards (blue circle) or the non-goal button (orange sphere).

Original: Once the robot reachesthe current goal, the environments generate the next goal  deterministically (SimpleButtonEnv) orrandomly (DynamicEnv, GremlinEnv), so the task complexity increases in order of SimpleButtonEnv, DynamicEnv and GremlinEnv.
Revised: Once the robot reaches the current goal, the environments generate the next goal randomly (DynamicEnv-v0), or deterministically (SimpleButtonEnv-v0).

Original: When the robot performs an action at time t , it receives a reward 320 (cid:8) [Figure] [Equation] [Figure]
Revised: When the robot performs an action at time t , it receives a reward 319 (cid:8) ∥

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure] [Equation] [Figure] reached , where p t is the position ( x , y ) of the robot at timet and p goal is the current goal position at time t .
Revised: p t +1 − p goal ∥ 2 − ∥ p t − p goal ∥ 2 (cid:9) + 1 goal reached , where p t is the position ( x , y ) of the robot at timet and p goal is the current goal position at time t .

Original: It also receives a cost +1 if the robot touches anon-goal object (the non-goal button, a hazard, or a gremlin), and 0 otherwise.
Revised: It also receives a cost +1 if the robot touches anon-goal object (a hazard or the non-goal button), and 0 otherwise.

Original: Thus, for the robot,it receives higher return when the robot touches more goals in maximum timesteps T = 1000 , andhigher sum of costs when the robot touches one of the other objects more often.
Revised: Thus, for the robot, it receiveshigher return when the robot touches more goals in maximum timesteps T = 1000 , and higher sumof costs when the robot touches one of the other objects more often.

Original: More detailedexplanation of the environments is in Appendix C.
Revised: More detailed explanation of theenvironments is in Appendix D.

Original: 5.2 Empirical Results
Revised: 5.2 Empirical Results

Original: We compared the performance of the proposed algorithm (QCPO) with that of PPO with Lagrangianmultiplier method (PPO_Lag) 2 for (ExpCP) and that of WCSAC [19] 3 for (CVaR-CP) which is
Revised: We compared the performance of the proposed algorithm (QCPO) with that of PPO with Lagrangianmultiplier method (PPO_Lag) 3 for (ExpCP) and that of WCSAC [19] 4 for (ProbCP). We set the projects

Original: 5 Timesteps [-(1.0e+06)5 1020 
-]{+(1.0e+06)
+}
Revised: 5 Timesteps [-(1.0e+06)5 1020 
-]{+(1.0e+06)
+}

Original: R e t u r n A v e r a g e (a) Avg. Return0 1 2 3 4     5 Timesteps (1.0e+06) 0.
Revised: R e t u r n A v e r a g e (a) Average Return     5 Timesteps (1.0e+06) 0.

Original: P r o b O u t a g e
Revised: P r o b O u t a g e

Original: QCPO (Proposed) (15clim_0.1outprob) QCPO (Proposed) (15clim_0.2outprob) PPO_Lag (ExpConst) WCSAC (0.1CVaR) WCSAC (0.2CVaR)[-(b) Outage Prob. 5 Timesteps (1.0e+06)5 
-]
Revised: QCPO (Proposed) (15clim_0.1outprob) QCPO (Proposed) (15clim_0.2outprob) PPO_Lag (ExpConst) WCSAC (0.1CVaR) WCSAC (0.2CVaR)[-(b) Outage Prob. 5 Timesteps (1.0e+06)5 
-]

Original: R e t u r n A v e r a g e (c) Avg. Return 0 1 2 3 4 5 Timesteps (1.0e+06) 0. P r o b O u t a g e QCPO (Proposed) (15clim_0.1outprob) QCPO (Proposed) (15clim_0.2outprob) PPO_Lag (ExpConst) WCSAC (0.1CVaR) WCSAC (0.2CVaR) (d) Outage Prob.
Revised: [Figure]

Original: 0 1 2 3 4     5 Timesteps (1.0e+06)0 26 

Revised: Probability     5 Timesteps (1.0e+06)

Original: R e t u r n A v e r a g e (e) Avg. Return0 1 2 3 4     5 Timesteps (1.0e+06) 0.
Revised: R e t u r n A v e r a g e (c) Average Return     5 Timesteps (1.0e+06) 0.

Original: P r o b O u t a g e
Revised: P r o b O u t a g e

Original: QCPO (Proposed) (15clim_0.1outprob) QCPO (Proposed) (15clim_0.2outprob) PPO_Lag (ExpConst)(0.1Ki) WCSAC (0.1CVaR) WCSAC (0.2CVaR)
Revised: QCPO (Proposed) (15clim_0.1outprob) QCPO (Proposed) (15clim_0.2outprob) PPO_Lag (ExpConst) WCSAC (0.1CVaR) WCSAC (0.2CVaR

Original: [Figure]
Revised: [Figure]

Original: a stricter problem than (ProbCP).

Original: We set the threshold d th = 15 in (ExpCP), (CVaR-CP) and(QuantCP), and the target outage probability ϵ 0 = 0 .

Original: 2 in (CVaR-CP) and (QuantCP).

Original: Fig.
Revised: Fig.

Original: 4 shows the results of the considered algorithms on SimpleButtonEnv, DynamicEnv, and GremlinEnv.
Revised: 4 shows the results of the considered algorithms on DynamicEnv-v0 and SimpleButtonEnv-v0.

Original: All experiments were done with 10 different random seeds, and the real line and theshaded area represent the average and average ± standard deviation, respectively.
Revised: All experiments were done with 10 different random seeds, and the real line and the shaded arearepresent the average and average ± standard deviation, respectively.

Original: PPO with the Lagrangian multiplier method for (ExpCP) (green) keeps the average of the sum cost around the thethreshold d th = 15 well (please see the graph in Appendix D), and its outage probability becomesaround 0 .
Revised: PPO with the Lagrangianmultiplier method for (ExpCP) (green) keeps the average of the sum cost around the the thresholdd th = 15 well (please see the graph in Appendix C), and its outage probability becomes around 0 .

Original: 35 as we can observe in Fig.
Revised: 35as we can observe in Fig.

Original: 4b, 4d, and 4f.
Revised: 4b and 4d.

Original: As aforementioned, the CVaR approach(WCSAC) should satisfy a sufficient condition for satisfying the outage probability constraint in(ProbCP).
Revised: As aforementioned, the CVaR approach (WCSAC) shouldsatisfy a sufficient condition for satisfying the outage probability constraint in (ProbCP).

Original: It is seen that WCSAC ( ϵ 0 = 0 .
Revised: It is seenthat WCSAC ( ϵ 0 = 0 .

Original: 2 (purple), ϵ 0 = 0 .
Revised: 2 (purple), ϵ 0 = 0 .

Original: 1 (red)) achieves lower or similar outageprobability to the threshold ϵ 0 in Fig 4b, but the algorithm does not satisfy the outage probabilityconstraint exactly inFig.
Revised: 1 (red)) achieves lower or similar outage probability to thethreshold ϵ 0 in Fig 4d, but the algorithm does not satisfy the outage probability constraint exactly in

Original: 4d and 4f.
Revised: Fig. 4b.

Original: This means that the Gaussian distribution approximation ofthe distribution of X π ( s ) has limited capability to capture the decay rate of the tail probability.
Revised: This means that the Gaussian distribution approximation of the distribution of X π ( s ) haslimited capability to capture the decay rate of the tail probability.

Original: Onthe other hand, the proposed ( ϵ 0 = 0 .
Revised: On the other hand, the proposed( ϵ 0 = 0 .

Original: 2 (orange), ϵ 0 = 0 .
Revised: 2 (orange), ϵ 0 = 0 .

Original: 1 (blue)) maintains the outage probabilityaround the desired target outage probability very well, as shown in Fig.
Revised: 1 (blue)) maintains the outage probability around the desired target outageprobability very well, as shown in Fig.

Original: 4b, 4d, and 4f.
Revised: 4b and 4d.

Original: Now consider the average return of these algorithms.
Revised: Now consider the average return of these algorithms.

Original: In constrained RL, in general, if an algorithm isallowed to have a higher sum of costs, then it has a higher return.
Revised: In constrained RL, in general, if an algorithm isallowed to have a higher sum of costs, then it has a higher return.

Original: Thus, as seen in Fig.
Revised: Thus, as seen in Fig.

Original: 4b, 4d,and4f, PPO_Lag induces the highest outage probability, so it has the highest average return, as shownin Fig. 4a, 4c, and 4e.
Revised: 4b and 4d, PPO_Lag induces the highest outage probability, so it has the highest average return, as shown in Fig.4a and 4c.

Original: The direct comparison between WCSAC and QCPO is less meaningful in DynamicEnv and GremlinEnv, since WCSAC does not satisfy the outage probability constraint, butit is fair in SimpleButtonEnv because both algorithms satisfy the outage probability constraint.
Revised: The direct comparison between WCSAC and QCPO is less meaningful in DynamicEnv-v0,since WCSAC does not satisfy the outage probability constraint, but it is fair in SimpleButtonEnv-v0because both algorithms satisfy the outage probability constraint.

Original: Asseen in Fig.
Revised: As seen in Fig.

Original: 4a, QCPO achieves higher average return than WCSAC for the same target probabilityconstraint ϵ [-
-] = 0 .
Revised: 4c, QCPO achieveshigher average return than WCSAC for the same target probability constraint ϵ [-
-]0 = 0 .

Original: This is because QCPO satisfies the target outage probability exactly, i.e.,uses the given cost budget fully for a higher return.
Revised: This isbecause QCPO satisfies the target outage probability exactly, i.e., uses the given cost budget fully fora higher return.

Original: We provided more results in Appendix D.
Revised: We provided more results in Appendix D.

Original:  Conclusion
Revised:  Conclusion

Original: We have proposed the framework of quantile-constrained RL to constrain the outage probabilityby adopting a constraint on quantile, which is equivalent to the outage probability constraint.
Revised: We have proposed the framework of quantile-constrained RL to constrain the outage probabilityby adopting a constraint on quantile, which is equivalent to the outage probability constraint.

Original: Wehave investigated issues in applying the policy gradient theorem to the Lagrangian of the quantileconstrained RL problem, and have converted the quantile into an additive form of costs so thatapplication of the policy gradient theorem is feasible.
Revised: Wehave investigated issues in applying the policy gradient theorem to the Lagrangian of the quantileconstrained RL problem, and have converted the quantile into an additive form of costs so thatapplication of the policy gradient theorem is feasible.

Original: Based on our derivation, we have constructedthe QCPO algorithm which uses distributional RL techniques to learn the u -quantile of the cumulativesum cost X π ( s ) , and Weibull distribution to approximate the tail distribution of X π ( s ) .
Revised: Based on our derivation, we have constructedthe QCPO algorithm which uses distributional RL techniques to learn the u -quantile of the cumulativesum cost X π ( s ) , and Weibull distribution to approximate the tail distribution of X π ( s ) .

Original: We alsoproved the policy improvement condition for QCPO and showed that there exists an approximationloss due to our approximation of the quantile.
Revised: We alsoproved the policy improvement condition for QCPO and showed that there exists an approximationloss due to our approximation of the quantile.

Original: Empirical results show that QCPO constrains theoutage probability well as the desired target value.
Revised: Empirical results show that QCPO constrains theoutage probability well as the desired target value.

Original: The meaning of such exact satisfaction of theoutage probability is two-fold: First, the constraint on the outage probability is satisfied to controlthe probability of unsafe events, and second, the exact satisfaction of the cost constraint enablesus to exploit the cost budget fully and obtain a higher return.
Revised: The meaning of such exact satisfaction of theoutage probability is two-fold: First, the constraint on the outage probability is satisfied to controlthe probability of unsafe events, and second, the exact satisfaction of the cost constraint enablesus to exploit the cost budget fully and obtain a higher return.

Original: Empirical results demonstrated theeffectiveness of the proposed scheme.
Revised: Empirical results demonstrated theeffectiveness of the proposed scheme.

Original: Checklist
Revised: Checklist

Original: For all authors...(
Revised: For all authors...

Original: a)
Revised: (a) Do the main

Original: Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: claims made in the abstract and introduction accurately reflect the paper’scontributions and scope

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We mentioned theoretical bound
Revised: [Yes] We mentioned theoretical bound

Original: and approximation loss in Section 3 and 4.
Revised: and approximation loss in Section 3 and 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes] I have read the guidelines.
Revised: [Yes] I have read the guidelines.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Please see Appendix B.
Revised: [Yes] Please seeAppendix B.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Please see Appendix B.
Revised: [Yes] Please see AppendixB.3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] The implemen-
Revised: [Yes] The implemen-

Original: tation code and instructions are uploaded in the supplemental material.
Revised: tation code and instructions are uploaded in the supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Please see Appendix E.
Revised: [Yes] Please see Appendix E.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] Experimental results show mean and standard deviation
Revised: [Yes] Experimental results show mean and standard deviation

Original: with 10 random seeds.
Revised: with 10 random seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] Please see Appendix C.
Revised: [Yes] Please see Appendix C.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Please see footnotes in page 6 and 8.
Revised: [Yes] Please see footnotesin page 6 and 8.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] Please see footnotes in page 6 and 8.
Revised: [Yes] Please see footnotes in page 6 and 8.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: New assets are in the supplemental material.
Revised: New assets are in the supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]