Regret Bounds for Risk-Sensitive Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In safety-critical applications of reinforcement learning such as healthcare androbotics, it is often desirable to optimize risk-sensitive objectives that account fortail outcomes rather than expected reward.
We prove the ﬁrst regret bounds forreinforcement learning under a general class of risk-sensitive objectives includingthe popular CVaR objective.
Our theory is based on a novel characterization of the
CVaR objective as well as a novel optimistic MDP construction.
Introduction
There has been recent interest in risk-sensitive reinforcement learning , which replaces the usualexpected reward objective with one that accounts for variation in possible outcomes.
One of themost popular risk-sensitive objectives is the conditional value-at-risk (CVaR) objective [1, 2, 3, 4],which is the average risk at some tail of the distribution of returns (i.e., cumulative rewards) under agiven policy [5, 6].
More generally, we consider a broad class of objectives in the form of a weightedintegral of quantiles of the return distribution, of which CVaR is a special case.
A key question is providing regret bounds for risk-sensitive reinforcement learning.
While there hasbeen some work studying this question, it has focused on a speciﬁc objective called the entropic riskmeasure [7, 8], leaving open the question of bounds for more general risk-sensitive objectives.
Therehas also been work on optimistic exploration for CVaR [9], but without any regret bounds.
We provide the ﬁrst regret bounds for risk-sensitive reinforcement learning with objectives of form
[Equation]
(1) where Z ( π ) is the random variable encoding the return of policy π , F Z ( π ) is its quantile function(roughly speaking, the inverse CDF), and G is a weighting function over the quantiles.
This classcaptures a broad range of useful objectives, and has been studied in prior work [10, 4].
We focus on the episodic setting, where the agent interacts with the environment, modeled by a
Markov decision process (MDP), over a ﬁxed sequence of episodes.
Its goal is to minimize theregret—i.e., the gap between the objective value it achieves compared to the optimal policy.
Ourapproach is based on the upper conﬁdence bound strategy [11, 12], which makes decisions accordingto an optimistic estimate of the MDP.
We prove that this algorithm (denoted A ) has regret
[Equation]
where T is the length of a single episode, L G is the Lipschitz constant for the weighting function G ,|S| is the number of states in the MDP, |A| is the number of actions, and K is the number of episodes(Theorem 4.1).
Importantly, it achieves the optimal rate √ K achievable for typical expected return objectives (which is a lower bound in our setting since expected return is an objective in the class weconsider, taking G ( τ ) = τ ).
For CVaR objectives, we have L G = 1 /α , where α is the size of the tailconsidered—e.g., when α is small, it averages over outliers with particularly small return.
The main challenge behind proving our result is bounding the gap between the objective value for theestimated MDP and the true MDP.
In particular, even if we have a uniform bound (cid:107) F ˆ Z ( π ) − F Z ( π ) (cid:107) ∞on the CDFs of the estimated return ˆ Z ( π ) and the true return Z ( π ) , we need to translate this to abound on the corresponding objective values.
To do so, we prove that equivalently, we have
[Equation]
This equivalent expression for Φ follows by variable substitution and integration by parts when F Z ( π )is invertible
(so F † Z ( π ) ( τ ) =
F − 1 Z
( π ) ( τ ) ), but the general case requires signiﬁcantly more care.
Weshow that it holds for an arbitrary CDF F Z ( π ) .
In addition to our regret bound, we provide several other useful results for MDPs with risk-sensitiveobjectives.
In particular, optimal policies for risk-sensitive objectives may be non-Markov.
For
CVaR objectives, it is known that the optimal policy only needs to depend on the cumulative returnaccrued so far [13].
We prove that this holds in general for objectives of the form (1) (Theorem 3.1).
Furthermore, the cumulative return so far is a continuous component; we prove that discretizing thiscomponent yields an arbitrarily close approximation of the true MDP (Theorem 3.2).
Related work.
To the best of our knowledge, the only prior work on regret bounds for risk-sensitivereinforcement learning is speciﬁc to the entropic risk objective [7, 8]:
[Equation]
where β ∈ R > 0 is a hyperparameter.
As β → 0 , this objective recovers the expected return objective;for β < 0 , it encourages risk aversion by upweighting negative returns; and for β > 0 , it encouragesrisk seeking behaviors by upweighting positive returns.
This objective is amenable to theoreticalanalysis since the value function satisﬁes a variant of the Bellman equation called the exponential
Bellman equation ; however, it is a narrow family of risk measures and is not widely used in practice.
In contrast, we focus on a much broader class of risk measures including the popular CVaR objective [1], which is used to minimize tail losses.
To the best of our knowledge, we provide the ﬁrstregret bounds for the CVaR objective and for the wide range of objectives given by (1).
Problem Formulation
Markov decision process.
We consider a Markov decision process (MDP) M =
( S , A , D, P, P , T ) ,with ﬁnite state space S , ﬁnite action space A , initial state distribution D ( s ) , ﬁnite time horizon
T , transition probabilities P ( s (cid:48) | s, a ) , and reward measure P R
( s,a ) ; without loss of generality, weassume r ∈ [0 , 1] with probability one.
A history is a sequenceξ ∈ Z =
T (cid:91) t =
Z t where
Z t =
( S × A × R ) t − 1 × S
Intuitively, a history captures the interaction between an agent and M up to step t .
We considerstochastic, time-varying, history-dependent policies π t ( a t | ξ t ) , where t is the time step.
Given π , thehistory Ξ ( π ) t generated by π up to step t is a random variable with probability measure
[Equation]
if t =
[Equation]
1 ) · P R ( s t ,a t ) ( r t ) · P ( s t +1 | s t , a t ) otherwise , where for all τ ∈ [ T ] we use the notationξ τ =
(( s 1 , a 1 , r 1 ) , ..., ( s τ − 1 , a τ − 1 , r τ − 1 ) , s τ ) .
Finally, an episode (or rollout ) is a history ξ ∈ Z T of length T generated by a given policy π .
Bellman equation.
The return of π on step t is the random variable ( Z ( π ) t ( ξ t ))
( ξ T ) =
(cid:80) Tτ =
t r t ,where ξ T ∼ P Ξ ( π ) T ( · | Ξ ( π ) t = ξ t ) —i.e., it is the reward from step t given that the current history is ξ t .
Deﬁning Z ( π ) T +1 ( ξ, s )
= 0 , the distributional Bellman equation [14, 9] is
[Equation]
where S ( ξ ) = s for ξ =
( ..., s ) is the current state in history ξ , and F X is the cumulative distributionfunction (CDF) of random variable X .
Finally, the cumulative return of π is Z ( π ) =
Z ( π ) 1 ( ξ ) , whereξ =
( s ) ∈
Z 1 for s ∼ D is the initial history; in particular, we have
[Equation]
Risk-sensitive objective.
The quantile function of a random variable X is
[Equation]
Note that if F X is strictly monotone, then it is invertible and we have F † X ( τ ) =
F − 1 X ( τ ) .
Now, ourobjective is given by the Riemann-Stieljes
integral
[Equation]
where G ( τ ) is a given CDF over quantiles τ ∈ [0 , 1] .
This objective was originally studied in [15]for the reinforcement learning setting.
For example, choosing G ( τ ) =
min { τ/α, 1 } (i.e., the CDFof the distribution Uniform ([0 , α ]) ) for α ∈ [0 , 1] yields the α -conditional value at risk (CVaR)objective; furthermore, taking α = 1 yields the usual expected cumulative reward objective.
Inaddition, choosing G ( τ ) = 1 ( τ ≤ α ) for α ∈ [0 , 1] yields the α value at risk (VaR) objective.
Otherrisk sensitive-objectives can also be captured in this form, for example the Wang measure [16], andthe cumulative probability weighting (CPW) metric
We call any policy
[Equation]
an optimal policy —i.e., it maximizes the given objective for M .
Assumptions.
First, we have the following assumption on the quantile function for Z ( π ) :
[Equation]
Since T is the maximum reward attainable in an episode, this assumption says that the maximumreward is attained with some nontrivial probability.
This assumption is very minor; for any given
MDP M , we can modify M to include a path achieving reward T with arbitrarily low probability.
Assumption 2.2.
G is L G -Lipschitz continuous for some L G ∈ R > 0 , and G (0) = 0 .
For example, for the α -CVaR objective, we have L G = 1 /α .
Assumption 2.3.
We are given an algorithm for computing π ∗M for a given MDP M .
For CVaR objectives, existing algorithms [13] can compute π ∗M with any desired approximation error.
For completeness, we give a formal description of the procedure in Appendix D. When unambiguous,we drop the dependence on M and simply write π ∗ .
Finally, our goal is to learn while interacting with the MDP M across a ﬁxed number of episodes K .
In particular, at the beginning of each episode k ∈ [ K ] , our algorithm chooses a policy π ( k )
=
A ( H k ) ,where H k = { ξ T,κ } k − 1 κ =1 is the random set of episodes observed so far, to use for the duration ofepisode k .
Then, our goal is to design an algorithm A that aims to minimize regret , which measuresthe expected sub-optimality with respect to π ∗ :
[Equation]
Finally, for simplicity, we assume that the initial state distribution D is known; in practice, we canremove this assumption using a standard strategy.
Optimal Risk-Sensitive Policies
In this section, we characterize properties of the optimal risk-sensitive policy π ∗M .
First, we showthat it sufﬁces to consider policies dependent on the current state and the cumulative rewardsobtained so far, rather than the entire history.
Second, the cumulative reward is a continuous quantity,making it difﬁcult to compute the optimal policy; we prove that discretizing this component doesnot signiﬁcantly reduce the objective value.
For CVaR objectives, these results imply that existingalgorithms can be used to compute the optimal risk-sensitive policy [13].
Augmented state space.
We show there exists an optimal policy π ∗ t ( a t | y t , s t ) that only dependson the current state s t and cumulative reward y t =
J ( ξ t ) =
(cid:80) t − 1 τ =1 r τ obtained so far.
To this end, let
[Equation]
} be the set of length t histories ξ with cumulative reward at most y so far, and current state s .
For anyhistory-dependent policy π , deﬁne the alternative policy ˜ π by
[Equation]
Note that ˜ π only depends on ξ t through y t
= J ( ξ t ) and s t , we can deﬁne ˜
π t ( a t | ξ t ) =
˜ π t ( a t | y t , s t ) .
Theorem 3.1.
For any policy π , we have Φ(˜ π ) =
Φ( π ) .
We give a proof in Appendix A.
In particular, given any optimal policy π ∗ , we have Φ(˜ π ∗ ) =
Φ( π ∗ ) ;thus, we have ˜ π ∗ ∈ arg max π Φ( π ) .
Finally, we note that this result has already been shown for
CVaR objectives [13]; our theorem generalizes the existing result to any risk-sensitive objective thatcan be expressed as a weighted integral of the quantile function.
Augmented MDP.
As a consequence of Theorem 3.1, it sufﬁces to consider the augmented MDP˜ M = ( ˜ S , A , ˜ D, ˜ P, ˜ P , T ) .
First, ˜ S =
S × R is the augmented state space; for a state ( s, y ) ∈ ˜ S , theﬁrst component encodes the current state and the second encodes the cumulative rewards so far.
Theinitial state distribution is a probability measure
[Equation]
where δ 0 is the Dirac delta measure placing all probability mass on y =
(i.e., the cumulative rewardso far is initially zero).
The transitions are given by the product measure
[Equation]
i.e., the second component of the state space is incremented as y (cid:48)
=
y
+ r , where r is the rewardachieved in the original MDP.
Finally, the rewards are now only provided on the ﬁnal step:
[Equation]
otherwise , i.e., the reward at the end of a rollout is simply the cumulative reward so far, as encoded by thesecond component of the state.
By Theorem 3.1, it sufﬁces to compute the optimal policy for ˜ M overhistory-independent policies π t ( a t | ˜ s t ) :
[Equation]
where Π ind is the set of history-independent policies.
Once we have π ∗ ˜ M , we can use it in M bydeﬁning π M ( a | ξ, s ) =
[Equation]
∗ ˜ M (
[Equation]
| J ( ξ ) , s ) .
Discretized augmented MDP.
Planning over ˜ M is complicated by the fact that the second component of its state space is continuous.
Thus, we consider an η -discretization of ˜ M , for some η ∈ R > 0 .
To this end, we modify the reward function so that it only produces rewards in η · N = { η · n | n ∈ N } ,by always rounding the reward up.
Then, sums of these rewards are contained in η · N , so we canreplace the second component of ˜ S with η · N .
In particular, we consider the discretized MDPˆ M =
( ˆ S , A , ˜ D, ˆ P, ˜ P , T ) , where ˆ S = S × ( η · N ) , and transition probability measure
[Equation]
where φ ( r ) =
η · (cid:100) r/η (cid:101) .
That is, P R ( s,a ) is replaced with the pushforward measure P R ( s,a ) ◦ φ − 1 ,which gives reward η · i with probability P R
( s,a ) [ η · ( i − 1) <
r ≤ η · i ] .
Now, we prove that the optimal policy π ∗ ˆ M for the discretized augmented MDP ˆ M achieves objectivevalue close to the optimal policy π ∗M for the original MDP M .
Importantly, we want to considermeasure performance of both policies based on the objective Φ M of the original MDP M .
To do so,we need a way to use π ∗ ˆ M in M .
Note that π ∗ ˆ M depends only on the state ˆ s =
( s, y ) , where s ∈ Sis a state of the original MDP M , and
y ∈ η · N is a discretized version of the cumulative rewardobtained so far.
Thus, we can run π ∗ ˆ M in M by simply rounding the reward r t at each step t up tothe nearest value ˆ r t ∈ η · N at each step—i.e., ˆ r t = φ ( r t ) ; then, we increment the internal state asy t =
y t − 1 + ˆ r t .
We call the resulting policy π M the version of π ∗ ˆ M adapted to M .
Then, our nextresult says that the performance of π M is not too much worse than the performance of π ∗M .
Theorem 3.2.
Let π ∗ ˆ M be the optimal policy for the discretized augmented MDP ˆ M , let π M be thepolicy π ∗ ˆ M adapted to the original MDP M , and let π ∗M be the optimal (history-dependent) policyfor the original MDP M .
Then, we have
[Equation]
We give a proof in Appendix B. Note that we can set η to be sufﬁciently small to achieve any desirederror level (i.e., choose (cid:15)/T , where (cid:15) is the desired error).
The only cost is in computation time.
Note that the number of states in ˆ M is still inﬁnite; however, since the cumulative return satisﬁesy ∈ [0 , H ] , it sufﬁces to take ˆ S = S
× ( (cid:15) · [ (cid:100) H/η (cid:101) ]) ; then, ˆ M has | ˆ S| = |S| · (cid:100) H/η (cid:101) states.
Upper Conﬁdence Bound Algorithm
Here, we present our upper conﬁdence bound (UCB) algorithm (summarized in Algorithm 1).
At ahigh level, for each episode, our algorithm constructs an estimate M ( k ) of the underlying MDP Mbased on the prior episodes i ∈
[ k − 1] ; to ensure exploration, it optimistically inﬂates the estimate ofthe reward probability measure P .
Then, it plans in M ( k ) to obtain an optimistic policy π ( k ) =
π ∗M ( k ) ,and uses this policy to act in the MDP for episode k .
Optimistic MDP.
We deﬁne M ( k ) .
Without loss of generality, we assume S includes a distinguishedstate s ∞ with rewards F R ( s ∞ ,a ) ( r ) =
1 ( r ≥ 1) (i.e., achieve the maximum reward r = 1 withprobability one), and transitions P ( s ∞ | s, a ) =
1 ( s = s ∞ ) and P ( s (cid:48) | s ∞ , a ) =
( s (cid:48)
=
s ∞ ) (i.e.,inaccessible from other states and only transitions to itself).
Our construction of ˆ M ( k ) uses s ∞ foroptimism.
Now, let ˜ M ( k ) be the MDP using the empirical estimates of the transitions and rewards:
[Equation]
i =
T (cid:88) t = 1 ( r ≤ r i,t ) · 1 ( s i,t = s ∧ a i,t = a ) .
Then, let ˆ M ( k ) be the optimistic MDP; in particular, its transitionsˆ P ( k ) ( s (cid:48) | s, a ) =
  1 ( s (cid:48)
=
s ∞ ) if s = s ∞
[Equation]
− (cid:80) s (cid:48)
[Equation]
\{ s
[Equation]
P ( k ) ( s (cid:48)
|
[Equation] [Equation]
) if s (cid:48)
[Equation]
s ∞ max (cid:110) ˜
P ( k ) ( s (cid:48) | s, a ) − (cid:15) ( k ) P ( s, a ) , 0 (cid:111) otherwise transition to the optimistic state s ∞ when uncertain, and its rewards
[Equation]
R
[Equation]
s
[Equation]
a )
[Equation] [Equation]
otherwise optimistically shift the reward CDF downwards.
Here, (cid:15) ( k ) P ( s, a ) and (cid:15) ( k ) R ( s, a ) are deﬁned in
Section 5; intuitively, they are high-probability upper bounds on the errors of the empirical estimates˜ P ( k ) ( · | s, a ) and F ˜ R ( k ) ( s,a ) of the transitions and rewards, respectively.
1: for k ∈ [ K ] do 2:
Compute M ( k ) and π ( k ) =
π ∗M ( k ) using prior episodes { ξ ( i ) | i ∈
[ k − 1] } 3:
Execute π ( k ) in the true MDP M and observe episode ξ ( k ) =
[( s k,t , a k,t , r k,t )]
Tt =1 ∪ [ s k,T +1 ] 4: end for
Theoretical guarantees.
We have the following upper bound on the regret of Algorithm 1.
Theorem 4.1.
Denote Algorithm 1 by A .
For any δ ∈ (0 , 1] , with probability at least 1 − δ , we haveregret( A ) ≤ 4 T 2 · L G · |S| · |A| · (cid:115) [-
 |S [Equation]-]5 |S| · K · log (cid:18) 4 |S| · |A| · K δ (cid:19) =
˜ O ( √ K ) .
We brieﬂy compare our bound to existing ones in the setting of expected return objectives.
Thedependence on the number of episodes K matches existing bounds [11, 12]; since this is optimal inthe setting of expected return, and expected return is a special case of our setting (with G ( τ ) = τ ),our bound is also optimal in K .
In terms of the dependence on the number of states |S| , our bound hasan extra (cid:112) |S| factor compared to the UCRL2 algorithm
[11], and an extra |S| factor compared to theimproved bound of the UCBVI algorithm [12].
In terms of the dependence on the number of actions|A| , our bound has an extra (cid:112) |S| factor compared to both UCRL2 and UCBVI.
Our dependenceon the horizon length T is T 2 , compared to a T 3 / 2 in UCBVI and T in a variant of UCBVI [12]utilizing a carefully designed variance-based bonus.
Proof of Theorem 4.
We prove Theorem 4.1; we defer proofs of several lemmas to Appendix C.
At a high level, the proof proceeds in three steps.
First, we prove our key Lemma 5.1, which expressesthe objective Φ in terms of an integral of the weighted CDF of the return.
This lemma allows us totranslate bounds on the difference between CDFs of the estimated return ˆ Z ( π ) and the true return
Z ( π ) into bounds on the difference between corresponding objective values.
The proof of this lemmais divided into three parts that deal with different sets of points in the domain of the quantile function
F † Z ( π ) : (i) discontinuous; (ii) continuous and strictly monotone; (iii) continuous and non-strictlymonotone.
This result is used throughout the remainder of the proof.
Second, we deﬁne E to be the event where the optimistic estimated MDP ˆ M ( k ) falls into a certainconﬁdence set around the true MDP M for each k ∈
[ K ] ; in Lemma 5.2, we prove that E holds withhigh probability.
Then, in Lemma 5.6, we prove that under event E , the objective values of ˆ M ( k ) and
M are close.
To prove this lemma, we separately show that (i) the objective values of the estimated
MDP ˜ M
( k ) (estimated without optimism) and M are close (Lemma 5.4), and (ii) the objective valuesof ˆ M ( k ) and
˜ M ( k ) are close (Lemma 5.5).
Third, in Lemma 5.7, we prove that under event E , the MDP ˆ M ( k ) is indeed optimistic.
Together,these results imply the regret bound using the standard UCB proof strategy.
We proceed with the proof.
First, we have our key result providing an equivalent expression for Φ :
Lemma 5.1.
We have
[Equation]
Proof.
First, note that by integration by parts, we have
[Equation]
where the last line follows by Assumptions 2.1 & 2.2.
Thus, it sufﬁces to show that
[Equation]
The quantile function F † Z ( π ) is monotonically increasing and left-continuous [18], so this integral isequivalently a Lebesgue-Stieltjes integral [19].
Dividing the unit interval I =
[0 , 1] into disjoint sets
[Equation]
is discontinuous }
[Equation]
continuous and strictly monotone }
[Equation]
is continuous and non-strictly monotone } , then we have
[Equation]
[Equation]
We consider each of the three terms separately and then combine them to ﬁnish the proof.
First term.
Note that
I (1) = { τ (1) i } ∞ i =1 is countable since monotone functions can have countablymany discontinuities.
Also, for each i ∈ N , the measure assigned to τ (1) i by dF † Z ( π ) is
[Equation]
Thus, we have
[Equation]
On the second line, we have used the fact that F Z ( π ) ( x ) = τ (1) i for all x ∈
[ x (1) i , x (1)+ i ) .
To see thisfact, note that since x ≥ x (1) i , by monotonicity of F Z ( π ) , we have F Z ( π ) ( x ) ≥ F Z ( π ) ( x (1) i ) =
τ (1) i .
Furthermore, if F Z ( π ) ( x ) > τ (1) i , then we would have
[Equation]
≤ x
[Equation]
where the ﬁrst inequality follows since F Z ( π ) ( x ) ≥ τ for τ sufﬁciently close to τ (1) i , and the secondsince x ∈ { x (cid:48) ∈ R | F Z ( π ) ( x (cid:48) )
≥ F Z ( π ) ( x ) } .
Since we have assumed x < x (1)+ i , we have acontradiction, so F Z ( π ) ( x ) ≤ τ (1) i .
Thus, it follows that F Z ( π ) ( x ) =
τ (1) i , as claimed.
The third linefollows since
[Equation]
In particular, for any x ∈ F − 1 Z ( π ) ( { τ (1) i } ) , we have F Z ( π ) ( x ) =
τ (1) i , so
[Equation]
Conversely, we have
[Equation]
since F Z ( π ) ( x (cid:48) ) ≤ τ (1) i < τ for all x (cid:48) ≤ x so the inﬁmum must be ≥ x .
These two arguments showthat F − 1 Z ( π ) ( { τ (1) i } ) ⊆
[ x (1) i , x (1)+ i ] .
The fact that [ x (1) i , x (1)+ i ) ⊆
F − 1 Z ( π ) ( { τ (1) i } ) follows by the sameargument as for the second line.
The claim follows.
Finally, the fourth line follows since the sets
F − 1 Z ( π ) ( { τ (1) i } ) are disjoint.
Second term.
For any τ ∈ I (2) , then F − 1 Z ( π ) exists at τ , and we have F † Z ( π ) ( τ ) =
F − 1 Z
( π ) ( τ ) .
Thus, bya substitution τ = F Z ( π ) ( x ) , we have
[Equation]
Third term.
We can divide I (3) into a union of disjoint intervals I (3) =
(cid:83) ∞
i =
I (3) i , where
[Equation]
for some x (3) i ∈ R ; there are only be countably many such intervals (since each one contains a distinctrational number).
Then, we have
[Equation]
since F − 1 Z ( π ) ( I (3) )
= { x (3) i } ∞ i =1 has measure zero according to the Lebesgue measure dx .
Final proof.
Finally, note that F − 1 Z ( π ) ( I (1) ) , F − 1 Z ( π ) ( I (2) ) , and F − 1 Z ( π ) ( I (3) ) cover R and are disjointexcept possibly on a set of measure zero, so
[Equation]
The claim follows.
Next, given δ ∈ R > 0 , deﬁne E to be the event where the following hold:
[Equation]
( k ) R ( s, a ) ( ∀ s ∈ S , a ∈ A ) .
Lemma 5.2.
We have P [ E | { N ( k ) ( s, a ) } k ∈ [ K ] ,s ∈S ,a ∈A ] ≥ 1 − δ .
Next, let ˜ Z ( k,π ) and ˆ Z ( k,π ) be the returns for policy π for ˜ M ( k ) and
ˆ M ( k ) , respectively, let Φ =
Φ M ,˜Φ ( k ) =
Φ ˜ M ( k ) , and ˆΦ ( k ) =
Φ ˆ M ( k ) , and let π ∗ =
π ∗M , ˜ π ( k ) =
π ∗ ˜ M ( k ) , and ˆ π ( k ) =
π ∗ ˆ M ( k ) .
Now, weprove two key results: (i) ˆΦ ( k ) is close to Φ , and (ii) ˆΦ ( k ) is optimistic compared to Φ .
To this end,we have the following key lemma; its proof depends critically on Lemma 5.1.
Lemma 5.3.
Consider MDPs M =
( S , A , D, P, P , T ) and M (cid:48) =
( S , A , D, P (cid:48) , P (cid:48) , T ) , such that(cid:107) P (cid:48) ( · | s, a ) − P ( · | s, a ) (cid:107) 1 ≤ (cid:15) P ( s, a ) and (cid:107) F R (cid:48) ( s,a ) − F R ( s,a ) (cid:107) ∞ ≤ (cid:15) R ( s, a ) .
Then, we have| Φ (cid:48)
( π ) − Φ( π ) | ≤ T · L G · B ( π ) ( ∀ k ∈
[ K ] , π ) , where
B
[Equation]
π
[Equation]
=
[Equation]
( π
[Equation]
T (cid:34) T (cid:88) t = (cid:15) P ( s t , a t ) + (cid:15) R ( s t , a t ) (cid:35) .
Our next lemma characterizes the connection between ˜Φ ( k ) and Φ .
Lemma 5.4.
On event E and conditioned on { N ( k ) ( s, a ) } k ∈
[ K ] ,s ∈S ,a ∈A , we have
[Equation]
where
[Equation]
Proof.
The result follows since on event E and conditioned on { N ( k ) ( s, a ) } k ∈
[ K ] ,s ∈S ,a ∈A , ˜ M ( k )and M satisfy the conditions of Lemma 5.3 for all k ∈
[ K ] .
Our next lemma characterizes the connection between ˆΦ ( k ) and ˜Φ ( k ) .
Lemma 5.5.
For each k ∈ [ K ] and any policy π , we have
[Equation]
Proof.
The result follows since by deﬁnition of ˆ M ( k ) , ˆ M ( k ) and ˜ M ( k ) satisfy the condition of
Lemma 5.3 with (cid:15) P ( s, a ) = 2 | S | · (cid:15) ( k ) P ( s, a ) and (cid:15) R ( s, a ) =
(cid:15) ( k ) R ( s, a ) for all k ∈
[ K ] .
Now, we prove the ﬁrst key claim—i.e., ˆΦ ( k ) is close to Φ .
Lemma 5.6.
On event E , for all k ∈ [ K ] and any policy π , we have| ˆΦ ( k ) ( π
[Equation]
( π
[Equation]
|
[Equation]
S| · B ( k ) ( π ) .
Proof.
Note that
[Equation]
where the second inequality follows by Lemmas 5.4 & 5.5.
Now, we prove the second key claim—i.e., ˆΦ ( k ) is optimistic compared to Φ .
Lemma 5.7.
On event E , we have ˆΦ ( k ) ( π ) ≥ Φ( π ) for all k ∈
[ K ] and all policies π .
With these two key claims, the proof of Theorem 4.1 follows by a standard upper conﬁdence boundargument; we give the proof in Appendix C.4.
Conclusion
We have proposed a novel regret bound for risk sensitive reinforcement learning that applies to a broadclass of objective functions, including the popular conditional value-at-risk (CVaR) objective.
Ourresults recover the usual √ K dependence on the number of episodes, and also highlights dependenceon the Lipschitz constant L G of the integral of the weighting function G used to deﬁne the objective.
Future work includes extending these ideas to the setting of function approximation and understandingwhether alternative exploration strategies such as Thompson sampling are applicable.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
(c) Did you describe the limitations of your work?
[Yes] Assumptions are stated in Sec2.
(d) Did you discuss any potential negative societal impacts of your work?
[N/A]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 5 andsupplement.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Proof of Theorem 3.1
In this section
[Equation]
we prove Theorem 3.1, which says that it sufﬁces to the augmented state space ( y, s )rather than the whole history ξ .
First, we have the following lemma.
Lemma A.1.
For any y ∈ R and s ∈ S , we have
[Equation]
Proof.
Note that
[Equation]
The claim follows by replacing ˜Ξ ( π ) t with Ξ ( π ) t and multiplying by P Ξ ( π ) t [Ξ ( π ) t ∈
Z t ( y, s )] .
Next, let
[Equation]
be the probability of a history achieving current cumulative return at most y and ending in state s .
Lemma A.2.
We have D ( π ) t = D
[Equation]
Proof.
We prove by induction.
The base case t = 1 follows trivially.
For the inductive case, note that
[Equation]
where the ﬁrst line follows by deﬁnition of D ( π ) t +1 , the second by the inductive formula for P Ξ ( π ) t +1 , thethird since J ( ξ ◦ ( a, r, s (cid:48) ))
=
J ( ξ ) + r , and the fourth by summing over s (cid:48) .
Continuing, we have
[Equation]
where the ﬁrst line follows by introducing 1 ( S ( ξ )
= s ) and rearranging, the second by deﬁnition of
Z t , and the third by Lemma A.1.
Continuing, we have
[Equation]
where the ﬁrst line follows since ˜ π is independent of ξ and by rearranging, the second by deﬁnitionof D ( π ) t , the third by induction, and the fourth by deﬁnition of D (˜ π ) t .
Continuing, we have
[Equation]
[Table]
[Equation]
where the ﬁrst line follows by deﬁnition of ˜ π , the second by deﬁnition of Z t , the third by summingover s and rearranging, the fourth by introducing 1 ( s (cid:48)
=
s (cid:48)(cid:48) ) , the ﬁfth by deﬁnition of Z t +1 , the sixthby the inductive formula for P Ξ ( π ) t +1 , and the seventh by the deﬁnition of D (˜ π ) t +1 .
The claim follows.
Now, we prove Theorem 3.1.
By Lemma A.2, we have
[Equation]
Theorem 3.1 follows straightforwardly from this result.
B Proof of Theorem 3.2
We construct a sequence of MDPs M 0 , M 1 , ..., M T , such that M 0 = ˜ M and M T
= ˆ M , andwhere we can bound the incremental errors
[Equation]
noting a policy for one of the MDPs can be used in all the other MDPs.
For each τ ∈ [ T ] , the
MDP M τ discretizes the reward assigned on the t th step of M τ − 1 —more precisely, it discretizesthe transitions since the rewards are only assigned on the last step based on the cumulative rewardrecorded in the second component of the state space.
Formally, M τ is identical to ˜ M , except it usesthe (time-varying) transition probability measure
ˆ P ( τ ) deﬁned by
[Equation]
P
[Equation]
s
[Equation]
s
[Equation]
otherwise .
Then, M τ is identical to M τ − 1 except P R ( s,a ) is replaced with P R ( s,a ) ◦ φ − 1 on step τ .
We provethree lemmas showing a lower bound on the value of a policy π for M τ when adapted to M τ − 1 .
Lemma B.1.
Given τ ∈ [ T ] , let M
=
M τ − 1 and ˆ M = M τ (so compared to M , ˆ M replaces
P R ( s,a ) with P R ( s,a ) ◦ φ − 1 on step τ in its transitions).
Given any policy ˆ π for ˆ M , deﬁne the policy
[Equation]
) for M , where we initialize the (extra) policy internal state α 1 = 0 , and we update α τ +
= φ ( r τ ) − r τon step τ and α t +1 =
α t otherwise.
Then, for all x, y, α ∈ R , for t > τ , we have
[Equation]
and for t ≤ τ , we have
[Equation]
where Z ( π ) t (resp.
, ˆ Z (ˆ π ) t ) is the return of M (resp., ˆ M ) from step t for policy π (resp., ˆ π ).
Proof.
We prove by backwards induction on t .
The base case t =
T follows by deﬁnition (and sincethe reward measure does not change from M to ˆ M ).
For t > τ , we have
[Equation]
where the second line follows by induction and by the deﬁnition of π .
Next, for t =
τ , we have
F
[Equation]
a
[Equation]
r )
[Equation]
where the ﬁrst line uses the update from α t = 0 to α t
+1 =
r − φ ( r ) on this step, the secondline follows by induction and by the deﬁnition of π , the fourth line follows by monotonicity of
F ˆ Z (ˆ π ) t +1 ( s (cid:48) ,y + r ) , and the ﬁfth line follows by a change of variables ρ = φ ( r ) .
For t < τ , we have
[Equation]
Z
[Equation]
where the second line follows by the deﬁnition of π , and the third line follows by induction.
Theclaim follows.
Lemma B.2.
For any monotonically increasing F , we have F † ( F ( x ))
≤ x , and F ( F † ( τ )) ≥ τ .
Proof.
See Proposition 1 in [18].
Lemma B.3.
Let F, G : R → R be monotonically increasing.
If F ( x ) ≤ G ( x + η ) for all x ∈ R ,then we have F † ( τ ) ≥ G † ( τ ) − η for all τ ∈ R .
Proof.
By assumption, G ( x ) ≥ F ( x − η ) .
Substituting x = F † ( τ ) + η into this formula, we obtain
[Equation]
where the second inequality follows by Lemma B.2.
Also by Lemma B.2, since G is monotonicallyincreasing, so is G † , so we can apply G † to each side of the inequality to obtain
[Equation]
where the second inequality follows by Lemma B.2.
The claim follows.
Lemma B.4.
Consider the same setup as in Lemma B.1.
Let ˆ π be a policy for ˆ M , and let π be thepolicy deﬁned in Lemma B.1 that adapts ˆ π to M .
Then, we have
[Equation]
where Φ is the objective for M and ˆΦ is the objective for ˆ M .
Proof.
Let ˆ Z
=
ˆ Z (ˆ π ) 1 ( s 1 , 0) and
Z = Z ( π ) 1 ( s 1 , 0 , 0) .
Applying Lemma B.3 to the inequality in
Lemma B.1, we have
[Equation]
Integrating this inequality, we have
[Equation]
as claimed.
Lemma B.5.
Consider the same setup as in Lemma B.1.
Given any policy π for M , deﬁne the policyˆ π t ( a | s, y, α ) =
π t ( a | s, y + α ) for ˆ M , where we initialize α 1 = 0 , and we update α τ +1 =
r on step τ , where r is a random variablewith probability measure
[Equation]
and α t +1 =
α t otherwise.
Then, for all x, y, α ∈ R , for t > τ , we have
[Equation] [Equation]
, and for t ≤ τ , we have
[Equation]
Proof.
We prove by backwards induction on T .
The base case t =
T follows by deﬁnition.
Fort > τ , we have
[Equation]
where the second line follows by induction and by the deﬁnition of π .
Next, for t = τ
[Equation]
we have
[Equation]
where the second line uses the update from α t = 0 to α t
+1 =
r − ρ on this step, the third line followsby induction and by the deﬁnition of ˆ π , the fourth line follows by monotonicity of F Z ( π ) t
+1 ( s (cid:48) ,y + r ) , andthe ﬁfth line follows by the deﬁnition of conditional probability—in particular,
[Equation]
where in the third line
[Equation]
B i =
( η · ( i − 1) , η · i ] .
For t <
τ
[Equation]
we have
[Equation]
where the second line follows by the deﬁnition of π , and the third line follows by induction.
Theclaim follows.
Next, we prove two lemmas showing a converse—namely, a lower bound on the value of a policy πfor M τ − 1 when adapted to M τ .
Lemma B.6.
Consider the same setup as in Lemma B.1.
Letting π be a policy for M , and ˆ π be thepolicy deﬁned in Lemma B.5 that adapts π to ˆ M .
Then, we have
[Equation]
where ˆΦ is the objective for ˆ M and Φ is the objective for M .
Proof.
Let Z
=
Z
( π )
( s 1 , 0) and
ˆ Z
=
Z
(ˆ π ) 1 ( s 1 , 0 , 0) .
Applying Lemma B.3 to the inequality in
Lemma B.5, we have
[Equation]
Integrating this inequality, we have
[Equation]
as claimed.
Finally, we prove Theorem 3.2.
Let π TT be the optimal policy for M T , and let π Tτ be the policydeﬁned in Lemma
B.4 adapting π Tτ from M τ to M τ − 1 for each τ ∈ [ T ] .
[Equation]
where each inequality follows by Lemma B.4.
Similarly, let π 00 be the optimal policy for M 0 , and letπ 0 τ be the policy deﬁned in Lemma
B.6 adapting π 0 τ − 1 from M τ − 1 to M τ .
Then, we have
[Equation]
where each inequality follows by Lemma B.6.
Furthermore, by optimality of π TT for Φ T , we alsohave
[Equation]
0 T ) ; together, these three inequalities imply
[Equation]
Finally, note that π
= π ∗ ˜
M is the optimal policy for ˜ M = M 0 , and π 0 T =
π ˆ M is π 00 adapted to ˆ M ;also, Φ 0 =
Φ ˜ M is the objective for ˜ M .
Thus, we have
[Equation]
By Theorem 3.1, the optimal policy for ˜ M equals the optimal history-dependent policy for theoriginal MDP M , so the claim follows.
C Proof of Lemmas for Section
C.1 Proof of Lemma 5.
Proof.
First, by the Dvoretzky–Kiefer–Wolfowitz (DKW) inequality and a union bound, for eachk ∈ [ K ] , conditioned on { N ( k ) ( s, a ) } s ∈S ,a ∈A , with probability at least 1 − δ/ (2 K ) , we have(cid:107)
F ˜ R ( k ) ( s,a ) − F R ( s,a ) (cid:107) ∞ ≤ (cid:15) ( k ) R ( s, a ) ( ∀ s ∈ S , a ∈ A ) .
Similarly, by Hoeffding’s inequality, an (cid:96) 1 concentration bound for multinomial distribution, and aunion bound, for each k ∈ [ K ] , conditioned on { N ( k ) ( s, a ) } s ∈S ,a ∈A , we have(cid:107)
˜ P ( k ) ( · | s, a ) − P ( · | s, a ) (cid:107) 1 ≤ (cid:15) ( k ) P ( s, a ) ( ∀ s ∈ S , a ∈ A ) .
Thus, both of these bounds hold for all k ∈
[ K ] with probability at least 1 − δ .
The claim follows.
C.2 Proof of Lemma 5.3
Proof.
First, we prove that for all policies π , we have
[Equation]
To this end, let G ( r ) =
F Z ( π ) t +1 ( s (cid:48) ,y + r ) ( x − r ) ; note that G ( −∞ )
=
1 and G ( ∞ ) = 0 .
Then, byintegration by parts, we have
[Equation]
and similarly for F Z (cid:48)
[Equation]
( s,y ) ( x ) .
Next, note that
[Equation]
=
[Equation]
[Table]
a | s,
[Equation]
) · sup r
[Equation]
R
[Equation]
F R
[Equation]
,a
[Equation]
s,
[Equation]
| ≤ E (cid:20) (cid:15)
[Equation]
( s, a ) + (cid:15) R ( s, a )
[Equation]
sup x (cid:48) ∈ R | F Z (cid:48) ( π )
[Equation]
+
[Equation]
( s (cid:48)
[Equation]
y
+ r
[Equation]
( x (cid:48)
[Equation]
F Z ( π ) t +1 ( s (cid:48) ,y + r ) ( x (cid:48) )
| (cid:21) .
Thus, we have
[Equation]
( s
[Equation]
a
[Equation] [Equation]
=
[Equation]
, where the last step follows by induction.
Finally, we have
[Equation]
where the ﬁrst line follows by Lemma 5.1.
The claim follows since (cid:15) ( π ) 1 equals the desired bound.
C.3 Proof of Lemma 5.
Proof.
First, we prove that F ˆ Z ( k,π ) t
[Equation]
s,y
[Equation]
s,y ) ( x ) .
The case s = s ∞ is straightforward,since its transitions and rewards are equal in M and ˆ M , and it only transitions to itself.
For s (cid:54) =
s ∞ ,we prove by induction on t .
The base case t =
T follows by deﬁnition.
Then, we have
[Equation]
where the second line follows by induction, the third by integration by parts and substituting x (cid:48)
= x − r ,the fourth since F R ( s,a ) ( r ) = 1 = F ˆ R
[Equation]
k ) ( s,a ) ( r ) for r ≥ 1 , and for r < 1 , on event E , we have
[Equation]
and the ﬁfth by integration by parts and substituting r = x (cid:48) − x .
Next, since s (cid:54) = s ∞ , we haveˆ
[Equation]
so we can decompose the summand ˆ P ( k
[Equation]
(cid:48)
=
s ∞ ) in (2)and distribute it across the other summands; in particular, the summands s (cid:48) (cid:54) =
s ∞ become
[Equation]
(3) where the second line follows since F Z ( π ) t +1 ( s ∞ ,y + r ) ( x − r ) ≤ F Z ( π )
[Equation]
+
[Equation]
s
[Equation]
,y + r ) ( x − r ) for all s (cid:48) (cid:54) =
s ∞ ,and since P ( s (cid:48) | s, a ) − ˆ P ( k ) ( s (cid:48) | s, a ) ≥ 0 on event E .
Continuing from (2), we have
F
[Equation]
where the second line follows by distributing the summand s (cid:48)
=
s ∞ and applying (3).
Since M andˆ M have the same initial state distribution, we have F ˆ Z ( k,π ) ( x ) ≤ F Z ( π ) ( x ) .
By Lemma 5.1, we have
[Equation]
where the inequality follows from (4) and since G is monotone.
The claim follows.
C.4 Proof of Theorem 4.1
Proof.
We prove Theorem 4.1.
Note that on event E , we haveregret ( A )
[Equation]
where the ﬁrst inequality follows by Lemma 5.7, the second follows by optimality of ˆ π ( k ) for ˆΦ ( k ) ,and the third by Lemma 5.6.
Furthermore, note that
[Equation]
The claim follows.
D The Optimal Policy for CVaR Objectives
In this section, we describe how to compute the optimal policy for the CVaR objective whenthe MDP is known; this approach is described in detail in [13].
In particular, consider an MDP
M = ( S , A , D, P, P , T ) , and our goal is to compute a policy π that maximizes its CVaR objective.
Step 1: CVaR objective.
We begin by rewriting the CVaR objective in a form that is more amenableto optimization.
First, we have the following key result:
Lemma D.1.
For any random variable Z , we have
[Equation]
where the minimum is achieved by ρ ∗ = VaR ( Z ) .
As a consequence of this lemma, we have
[Equation]
Thus, we have
[Equation]
where
[Equation]
=
[Equation]
J
[Equation]
ρ ) where
[Equation]
The main challenge is evaluating the maximum over π ∈ Π in J ( ρ ) .
To do so, we construct another
MDP whose objective is E Z ( π ) (cid:2) ( Z ( π ) − ρ ) + (cid:3) for the appropriate choice of initial state distribution.
Step 2: Construct alternative MDP.
The MDP we construct is ˜ M =
( ˜ S , A , ˜ D, ˜ P, ˜ R, T ) , wherethe states are ˜ S
= S × R , the (time-varying, deterministic) rewards ˜ R : ˜ S × [ T ] → R are
[Equation]
otherwise , and the transitions are
[Equation]
noting that ˜ P is a (conditional) probability measure since the state space
˜ S includes a continuouscomponent; in practice, we discretize the continuous component of the state space.
Step 3: Value iteration.
Letting S 1 be the random initial state of the original MDP M (withdistribution D ), we have
[Equation]
where ˜ V ( π )is the value function of policy π for MDP ˜ M on step t = 1 .
Thus, we have
[Equation]
where ˜ V ∗ 1 is the value function of the optimal policy for ˜ M .
Intuitively, this strategy works becausethe augmented component of the state space r captures the cumulative reward so far plus its initialvalue − ρ ; then, by the deﬁnition of ˜ R , the reward is r + , which implies that ˜ V ( π ) 1 (( s, − ρ )) is theexpectation of the random variable ( Z ( π ) − ρ ) + .
Thus, we can compute max π ∈ Π E Z ( π ) [( Z ( π ) − ρ ) + ]by performing value iteration on ˜ M to compute ˜ V ( π ) 1 .
In particular, we have
[Equation]
and
[Equation]
for all t ∈ { 1 , ..., T − 1 } .
Then, given an initial state s 1 , we construct state ˜ s
= ( s 1 , − ρ ∗ ) , where
[Equation]
and then acting optimally in ˜ M according to ˜ V ∗ t .
E Experiments
We consider a classic frozen lake problem with a ﬁnite horizon.
The agent moves to a block next to itscurrent state at each timestep t and has a slipping probability of 0.1 in its moving direction if the nextstate is an ice block.
The objective is to maximize the cumulative reward without falling into holes.
The agent needs to choose among paths which correspond to different levels of risk and rewards.
Inother words, the agent should account for the tradeoff between the cumulative reward and risk ofslipping into holes.
We use a map with four paths of the same lengths that have different rewards at the end and different levels of risk of falling into holes.
We consider α ∈ { 0 .
01 } ,which correspond to optimal policies of choosing paths with best possible returns of { 6 , 4 , 2 , 1 } andsuccess probabilities of { 0 . 729 , 0 .
9 , 1 } , respectively (failure corresponds to zero return).
Figure 1 (left) shows the comparison in cumulative regret between our algorithm, UCBVI (whichmaximizes expected returns, not our risk-sensitive objective), and the an algorithm that optimizes ourrisk-sensitive objective but explores in a greedy way (i.e., use the best policy for the current estimated
MDP without any optimism), for α = 0 .
The regret is measured in terms of the CVaR objectivewith respect to the optimal policy for the same CVaR objective.
While UCBVI outperforms greedy,neither of them converge; in contrast, our algorithm converges within 40 episodes.
Figure 1 (right) compares the regret between our algorithm under different values of α using the
CVaR objective.
Note that smaller values of α tend to lead our algorithm to converge more slowly;this result matches our theory since smaller α corresponds to larger L G .
Intuitively, more samplesare needed to get a good estimate of the objective as α becomes small since the CVaR objective is theaverage return over a tiny fraction of samples, causing high variance in our estimate of the objective.