Near-optimal Distributional Reinforcement Learning towards Risk-sensitive Control
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider finite episodic Markov decision processes aiming at the entropic riskmeasure (EntRM) of return for risk-sensitive control.
We identify two properties ofthe EntRM that enable risk-sensitive distributional dynamic programming.
We propose two novel distributional reinforcement learning (DRL) algorithms, includinga model-free one and a model-based one, that implement optimism through twodifferent schemes.
We prove that both of them attain ˜ O
( exp( | β | H ) − 1 | β | H
[Equation]
regret upper bound, where S is the number of states, A the number of states, Hthe time horizon and T the number of total time steps.
It matches RSVI2 proposedin [22] with a much simpler regret analysis.
To the best of our knowledge, this isthe first regret analysis of DRL, which theoretically verifies the efficacy of DRLfor risk-sensitive control.
Finally, we improve the existing lower bound by provinga tighter bound of Ω( exp( βH/ 6) − 1 βH
H √ SAT ) for β > 0 case, which recovers thetight lower bound Ω( H √ SAT ) in the risk-neutral setting.
Introduction
Standard reinforcement learning (RL) [45] seeks to find an optimal policy that maximizes theexpectation of return.
It is also called risk-neutral RL since the objective is the mean functional of the return distribution.
However, in some high-stakes applications including finance [15, 6], medicaltreatment [21] and operations [16] etc, the decision-maker tends to be risk-sensitive with the goal ofmaximizing some risk measure of return distribution.
In this paper we consider the problem of optimizing the exponential risk measure (EntRM) in theepisodic and finite MDP setting for risk-sensitive control. The entropic risk measure can trade-offbetween the expectation and the variance and adjusts the risk-sensitiveness by control a risk parameter(see Equation 1). Ever since the seminal work of [29], risk-sensitive RL based on the EntRM hasbeen applied across a wide range of domains [43, 37, 27].
Most of the existing approaches, however,involve complicated algorithmic design to deal with the non-linearity of the EntRM.
Distributional reinforcement learning (DRL) [4] has demonstrated its superior performance overtraditional methods in some difficult tasks [14, 13] under risk-neutral setting.
Different from thevalue-based approaches, it learns the whole return distribution instead of a real-valued value function.
Given the entire return distribution, it is natural to leverage the distributional information to optimizea risk measure other than expectation [13, 44, 33]. Despite of the intrinsic connection between DRLand risk-sensitive RL, it is surprising that existing works on risk-sensitive control via DRL approaches([13, 34, 1]) lack regret analysis.
Consequently, it is challenging to evaluate and improve these DRLalgorithms in terms of sample-efficiency, which brings about a reasonable question
Can distributional reinforcement learning attain near-optimal regret for risk-sensitive control?
In this work, we answer this question positively by providing two DRL algorithms with provablyregret guarantees.
We devise two novel DRL algorithms with principled exploration schemes forrisk-sensitive control in the tabular MDP setting.
In particular, the proposed algorithms implementthe principle of optimism in the face of uncertainty (OFU) at the distributional level to balance theexploration-exploitation trade-off.
By providing the first regret analysis of DRL, we theoreticallyverifies the efficacy of DRL for risk-sensitive control. Therefore, our work bridge the gap between
DRL and risk-sensitive RL with regard to sample complexity.
Main contributions.
We summarize our main contributions in the following.
We build a risk-sensitive distributional dynamic programming (RS-DDP framework.
To be more specific we choose the entropic risk measure (EntRM) of the return distribution as our objective.
Byidentifying two key properties of EntRM
We establish distributional Bellman optimality equation forrisk-sensitive control.
We propose
two
DRL algorithms that enforce the OFU principle
in a distributional fashion through
two different schemes.
We provide ˜ O ( exp( | β | H ) − 1 | β |
H √ S 2 AK ) regret upper bound, which matchesthe best existing result of RSVI2 in [22].
It is the first regret analysis of DRL algorithm in thefinite episodic MDP in the risk-sensitive setting.
Compared to [22], our algorithm does not involvecomplicated bonus design, and our analysis are conceptually cleaner and easier to interpret.3.
We fill the gaps in the proof of lower bound in [23].
To the best of our knowledge, [23] onlyimplies a lower bound Ω( exp( | β | H/ 2) − 1 | β | √ K ) rather the claimed bound Ω( exp( | β | H/ 2) − 1 | β | √ T ) .
Theresulting lower bound is independent of S and A and is loose with a factor of √ H .
We overcomethese issues by proving a tight lower bound of Ω( exp( βH/ 6) − 1 βH
H √ SAT ) for β > 0 .
Note that thelower bound is tight in the risk-neutral setting ( β → 0 ).
Related work.
Following the paper [4], DRL has witnessed a rapid growth of study in literature[40, 14, 13, 2, 32].
Most of these works focus on improving the performance in the risk-neutralsetting, with a few exceptions [13, 34, 1].
However, none of these works study the sample complexity.
A rich body of work studies risk-sensitive RL with the EntRM [7, 8, 10, 9, 3, 11, 12, 18, 17, 19,24, 28, 30, 33, 35, 36, 38, 39, 42, 43].
In particular, [29] is the first to introduce the ERM as risksensitive objective in MDP.
However, they either assume known transition and reward or considerinfinite-horizon setting without sample-complexity considerations.
Two works are closely related to ours [23, 22] under precisely the same setting.
[23] is the first tostudy the risk-sensitive episodic MDP, which provides the first algorithms and regret guarantees.
Nevertheless, the regret upper bounds contain a dispensable factor of exp( | β | H 2 ) .
Additionally, theirlower bound proof contains mistakes, and the corrected proof suggests a weaker bound.
[22] improvesthe algorithm by removing the additional O (exp( | β | H 2 )) factor.
However, the regret analysis iscomplicated, and the lower bound is not fixed.
A very recent work ([1]) independently proposes arisk-sensitive DDP framework, but their work is fundamentally different from ours.
The risk measureconsidered in [1] is the conditional value at risk (CVaR), and they focus on the infinite horizon setting.
Due to the space limit, we provide detailed comparisons with [23, 22, 1] in Appendix A.
Preliminaries
Notations.
We write [ M : N ] ≜ { M, M + 1 , ..., N } and [ N ] ≜ [1 : N ] for any positive integers
M ≤ N . We adopt the convention that (cid:80) mi = n a i ≜ 0 if n > m and (cid:81) mi = n a i ≜ 1 if n > m .
Weuse I {·} to denote the indicator function.
For any x ∈ R , we define [ x ] + ≜ max { x, 0 } .
We definethe step function with parameter c as ψ c ( x ) ≜
I { x ≥ c } .
Note that ψ c represents the CDF of adeterministic variable taking value c .
We denote by D ([ a, b ]) , D M and D the set of distributionssupported on [ a, b ] , [0 , M ] and the set of all distributions respectively.
For a random variable (r.v.)
X ,we use E [ X ] and V [ X ] to denote its expectation and variance.
For two r.v.s, we denote by X ⊥ Y if
X is independent of Y .
We use ˜ O ( · ) to denote O ( · ) omitting logarithmic factors.
Episodic MDP.
An episodic MDP is identified by M ≜ ( S , A ,
( P h ) h ∈ [ H ] , (
R h ) h ∈ [ H ] , H ) , where
S is the state space, A the action space, P h : S × A× → ∆( S ) the probability transition kernel atstep
h , R h : S ×
A → D ([0 , 1]) the collection of reward distributions at step h and H the length of one episode.
The agent interacts with the environment for K episodes.
At the beginning of episode k , Nature selects an initial state s k 1 arbitrarily.
In step h , the agent takes action a kh and observes randomreward R kh ( s kh , a kh ) ∼ R h ( s kh , a kh ) and reaches the next state s kh +1 ∼ P h ( ·| s kh , a kh ) .
The episodeterminates at H + 1 with R kH {++
+} = 0 , then the agent proceeds to next episode.
For each ( k, h ) ∈ [ K ] × [ H ] , we denote by H kh ≜ (cid:0) s 11 , a 11 , s 12 , a 12 , . . .
, s 1 H , a 1 H , . . .
, s kh , a kh (cid:1) the(random) history up to step h episode k .
We define F k ≜ H k − 1 H as the history up to episodek − 1 .
We describe the interaction between the algorithm and MDP in two levels.
In the level ofepisode, we define an algorithm as a sequence of function A ≜ ( A k ) k ∈
[ K ] , each mapping F k toa policy A k ( F k ) ∈ Π .
We denote by π k ≜
A k ( F k ) the policy at episode k .
In the level of step, a(deterministic) policy π is defined as a sequence of functions π
= ( π h ) h ∈ [ H ] with π h : S → ∆( A ) .
Entropic risk measure.
EntRM is a well-known risk measure in risk-sensitive decision-making,including mathematical finance [25], Markovian decision processes [3].
The EntRM value of a r.v.
X ∼ F with coefficient β ̸ = 0 is defined as
[Equation]
With slight abuse of notations, we write U β ( F ) =
U β ( X ) for X ∼ F .
For β with small absolutevalue, using Taylor’s expansion we have
[Equation]
Hence for a decision-maker who aims at maximizing the EntRM value, she tends to be risk-seeking(favoring high uncertainty in X ) if β > 0 and risk-averse (favoring low uncertainty in X ) if β < 0 .|
β | controls the risk-sensitivity.
It exactly recovers mean as the risk-neutral objective when β → 0 .
Risk-sensitive Distributional Dynamic Programming
[4, 40] has discussed the infinite-horizon distributional dynamic programming in the risk-neutralsetting, which will be referred to as the classical DDP.
There is a big gap between the risk-sensitive
MDP and the risk-neutral one.
In this section, we establish the novel DDP framework for risk-sensitivecontrol.
We start with defining the return for a policy π starting from state-action pair ( s, a ) at step h
[Equation]
Define Y πh ( s ) ≜
Z πh ( s, π h ( s )) .
There are three sources of randomness in Z πh ( s, a ) : the reward
R h ( s, a ) , the transition P π and the next-state return Y {+π h +
+} ( s h +1 ) .
Denote by ν π h ( s ) and η π h ( s, a ) thecumulative distribution function (CDF) corresponding to Y πh ( s ) and Z πh ( s, a ) respectively.
To theend of risk-sensitive control, we define the action-value function of a policy π at step h as Q πh ( s, a ) ≜
U β
( Z πh ( s, a )) , i.e. the EntRM value of the return distribution, for each ( s, a, h ) ∈ S ×
A ×
[ H ] .
Thevalue function is defined as V πh ( s ) ≜ Q πh ( s, π h ( s )) =
U β
( Y πh ( s )) .
We focus on the control setting, in which the goal is to find an optimal policy to maximize the valuefunction, i.e.
[Equation]
We write π = ( π 1 , ..., π H ) to emphasize that it is a multi-stage maximization problem.
Direct searchsuffers exponential computational complexity.
In the risk-neutral case, the principle of optimalityholds, i.e.,the optimal policy of tail sub-problem is the tail optimal policy [5].
Therein the multi-stagemaximization problem can be reduced to a multiple single-stage maximization problem.
However,the principle does not always hold for general risk measures.
For example, the optimal policy for CVaR may be non-Markovian/history-dependent ([41]).
We identify two key properties of EntRM, upon which we retain the principle of optimality.
Lemma 1.
The EntRM satisfies the following properties:
• Additive:
X ⊥ Y ⇒ U
β ( X + Y ) =
U β
( X )
+ U
[Equation]
• Monotonicity-preserving: ∀ F 1 , F
[Equation]
, G
[Equation]
D ,
[Equation] [Equation]
The proof is given in Appendix B.
In particular, the additivity entails that the EntRM value of thecurrent return Z πh ( s, a ) equals the sum of the immediate value of R h ( s, a ) and the value of the futurereturn Y πh ( s [Equation] , i.e.,
[Equation]
The monotonicity-preserving property together with the additivity suggests that the optimal futurereturn Y ∗ h ( s ′ ) consists in the optimal current return Z ∗ h ( s, a )
[Equation]
These observations implies the principle of optimality.
Proposition 1 (Principle of optimality) .
Let π ∗ =
{ π ∗ 1 , π ∗ 2 , ..., π ∗ H } be an optimal policy and assumewhen we visit some state s using policy π at time-step h with positive probability.
Consider thesub-problem defined by the the following maximization problem
[Equation]
Then the truncated optimal policy { π ∗ h , π
[Equation]
h +1 , ..., π ∗ H } is optimal for this sub-problem.
The proof is given in Appendix E. It further induces the distributional Bellman optimality equation.
Proposition 2 (Distributional Bellman optimality equation) .
For arbitrary initial state s 1 , the optimalpolicy ( π ∗ h ) h ∈ [ H ] is given by the following backward recursions:
[Equation]
where f h ( s, a ) is the probability density function of R h ( s, a ) .
Furthermore, the sequence ( η ∗ h ) h ∈
[ H ]and ( ν ∗ h )
h ∈
[ H ] are the sequence of distributions corresponding to the optimal returns at each step.
The proof is given in Appendix E. For simplicity, we define the distributional Bellman operator
B ( P, R ) : D S → D S×A with associated model ( P, R ) =
( P ( s, a ) , R ( s, a ))
( s,a ) ∈S×A as[ B ( P, R )
ν ]( s, a ) ≜ [ Pν ]( s, a ) ∗ f h ( ·| s, a ) , ∀ ( s, a ) ∈ S × A .
Hence we can rewrite Equation 2 in a compact form:
[Equation]
Finally, we define the regret of an algorithm A interacting with an MDP M for K episodes as
Regret ( A , M , K ) ≜
[Equation]
Note that the regret is a random variable since π k is a random quantity.
We denote by
Algorithm
For a better understanding of the readers, we present our algorithms under the assumption thatthe reward is deterministic and known 1 .
The algorithms for the case of random reward are given in Appendix C.
We denote by { r h ( s, a ) } ( s,a,h ) ∈S×A× [ H ] the reward functions.
For the case ofdeterministic reward, the Bellman update in Equation 2 takes the form
[Equation]
since adding a deterministic reward r h ( s, a ) corresponds to shifting the distribution [ P h ν ∗ h
+1 ]
( s, a ) byan amount of r h ( s, a ) .
We thus define the distributional Bellman operator B ( P, R ) :
D S → D S×Awith associated model ( P, r ) =
( P ( s, a ) , r ( s
[Equation]
a
[Equation]
) ( s,a ) ∈S×A as
[Equation]
We propose two DRL algorithms in this section, including a model-free algorithm and a model-basedalgorithm.
We first introduce the M odel- F ree R isk-sensitive O ptimistic D istribution I teration( RODI-MF ) in Algorithm 1.
For completeness, we introduce some additional notations here.
Fortwo CDFs F and G over reals, we define the supremum distance between them ∥ F − G ∥ ∞ ≜sup x | F ( x ) − G ( x ) | .
We define the ℓ 1 distance between two probability mass functions (PMFs)
P and Q as ∥ P − Q ∥ 1 ≜ (cid:80) i | P i − Q i | .
We denote by B ∞ ( F, c ) :
= { G ∈ D |∥ G − F ∥ ∞ ≤ c }
the supremum norm ball centered at F with radius c .
With slight abuse of notations, we denote by
B 1 ( P, c ) the l 1 norm ball centered at P with radius c .
4.1 Algorithm overview
4.1.1 RODI-MF
In each episode, the algorithm includes the planning phase (Line 4-12) and the interaction phase(Line 13-17).
Planning phase.
In a high level, the algorithm implements an optimistic version of approximate
DDP from step H + 1 to step 1 in each episode.
In Line (5-7), it performs sample-based Bellmanupdate.
To make it clear, we introduce the superscript k to the variables of Algorithm 1 in episode k .
For example, η kh denotes η h in episode k .
Specifically, for those visited state-action pairs, we claimthat Line 6 is equivalent to a model-based Bellman update.
Denote by I kh
( s, a ) ≜
I { ( s kh , a kh ) =
( s, a ) } .
Fix a tuple ( s, a, k, h ) such that N kh
( s, a ) ≥ 1 .
We denote by ˆ P kh ( ·| s, a ) the empirical transitionmodel
[Equation]
Observe that for any ν ∈ D S , we have
[Equation]
[Equation]
Hence the update formula in Line 6 of Algorithm 1 can be rewritten as
[Equation]
implying the equivalence to a model-based Bellman update with empirical model ˆ P kh .
Alternatively,the unvisited ( s, a ) remains to be the return distribution corresponding to the highest possible reward H + 1 − h .
The algorithm then computes the optimism constants (Line 8) and enforces OFUthrough the distributional optimism operator c kh
(Line 9) to obtain the optimistically plausible returndistribution η kh .
The choice of c kh will be discussed later.
The optimistic return distributions yields theoptimistic value function, from which the algorithm generates the greedy policy π kh .
The policy π khwill be used in the interaction phase.
Interaction phase.
In Line (15-16), the agent interacts with the environment using policy π andupdates the counts N h based on new observations.
[-h +
-]
MB
We introduce the second algorithm M odel- B ased R isk-sensitive O ptimistic D istribution I teration( RODI-MB ).
Algorithm 2 is a model-based algorithm because it requires to explicitly maintaining theempirical transition model in each episode.
However, it can be reduced to a non-distributional reinforcement learning algorithm that deals with the one-dimensional values instead of the distributions,which saves the computational complexity and space complexity.
Likewise, the algorithm includesthe planning phase (Line 4-10) and the interaction phase (Line 11-15).
Planning phase.
Analogous to Algorithm 1, the algorithm also performs approximate DDP togetherwith the OFU principle.
First, it applies the distributional optimistic operator to the empirical transitionmodel ˆ P kh to get the optimistic transition model ˜ P kh .
Then the algorithm uses ˜ P kh to execute Bellmanupdate to generate the optimistic return distributions η kh .
The remaining steps are the same as Algorithm 1.
Interaction phase.
In Line (13-14), the agent interacts with the environment using policy π k andupdates the counts
N k +1 h and empirical transition model
ˆ P k +1 h based on the new observations.
Algorithm do do 6:
) 10: end for 11:
Receive s k12: for = 1 : H do 13: a π s ) and transit to s
Compute
P
15: end for 16: end for
1 h ( · , · ) ← 1 S 1 for all h [-
-]∈
[ H ] 3: for k = 1 : K do 4:
W kH [-+
-]+1 ( · ) ←5: for h
= H : 1 do 6: ˜ P kh ( · , · ) ← O 1 c kh ( · , · ) ˆ P kh ( · , · ) 7:
J kh ( · , · ) ← e βr h ( · , · ) (cid:104) ˜ P kh W kh {++
+}
(cid:105) ( · , · ) 8:
W kh ( · ) ← max a J kh ( · , a ) 9: end for 10:
Receive s k11: for h = 1 : H do 12: a kh ← arg max a J kh ( s kh , a ) and transit to s kh + 13:
Compute N k +1 h ( · , · ) and ˆ P
k +1 h
( · , · ) 14: end for 15: end for
Equivalence to ROVI .
R isk-sensitive O ptimistic V alue
I teration ( ROVI ) is a non-distributionalalgorithm that deals with the real-valued value function rather than the distribution.
It is motivated bythe exponential Bellman equation proposed by [22].
We define the functional exponential EntRM(EERM)
E β as the EntRM after the exponential transformation
[Equation]
Define the exponential value functions W h ( s ) ≜
E β ( ν h ( s )) and J h ( s, a ) ≜ E β ( η h ( s, a )) for all( s, a, h ) s.
Applying EERM to Equation 3 yields the exponential Bellman equation
[Equation]
To verify the equivalence, it is sufficient to show that J kh in Algorithm 3 corresponds to the exponentialfunction of η kh in Algorithm 2.
Observe that E β is linear in F , hence it follows that
[Equation]
The two algorithms generate the policy sequence in the same way, implying that their trajectories
H KH follow the same distribution.
The formal statement is given in Appendix E.
4.2 Distributional Optimism
It is common to add a bonus to the reward to ensure optimism in the risk-neutral setting.
Specifically,the bonus is closely related to the level of uncertainty, which is quantified by the concentrationinequality.
Yet, this type of optimism cannot be adapted to the distributional setup.
As one of ourtechnical novelty, the distributional optimism is introduced for algorithmic design and regret analysis.
In particular, we specify two types of distributional optimism operators, which map a statisticallyplausible distribution (either the empirical model or the return distribution) to a optimisticallyplausible distribution.
Either of them is applied by Algorithm 2 or Algorithm 1.
Distributional optimism on the return distribution (in Algorithm 1).
For two CDFs F and G ,we say that F is more optimistic than G (w.r.t.
EntRM) if U β ( F ) ≥ U β ( G ) .
This reflects the intuitionthat the more optimistic distribution should own larger EntRM value.
Following [31], we define thedistributional optimism operator O ∞ c : D ([ a, b ]) (cid:55)→ D ([ a, b ]) with level c ∈ (0 , 1) as
[Equation]
+ . The optimistic operator shifts the input F down by at most c over [ a, b ) , and retain the value 1 at b .
Itensures that O ∞ c F remains in D ([ a, b ]) and dominates all the other CDFs in D ([ a, b ]) in the sensethat (O ∞ c F )( x ) ≤ G ( x ) for any G ∈ B ∞ ( F, c ) .
Since EntRM is monotonic, it holds that U
[Equation]
(O ∞ c F ) ≥ U [Equation] ( G ) , ∀ G ∈ B ∞ ( F, c ) . Hence O ∞ c F is the most optimistic distribution in the infinity ball B ∞ ( F, c ) .
In other words, forany CDF F and G satisfying ∥ F − G ∥ ∞ ≤ c , we have O ∞ c G ⪰ F .
When specialized to the returndistributions, we can apply the distributional optimism operator to the estimated return distributionη kh
(Line 9 of Algorithm 1) with the constant c kh to ensure U β ( η kh ( s, a )) ≥ U β ( η ∗ h ( s, a ) .
The constantc kh quantifies uncertainty in the model estimation, i.e., (cid:13)(cid:13)(cid:13) ˆ P kh ( s, a )
− P h
( s, a ) (cid:13)(cid:13)(cid:13) 1 .
Distributional optimism on the model (in Algorithm 2).
Given the model, we consider the optimism among the space of PMFs rather than CDFs.
Using the ℓ 1 concentration inequality [46], we get a concentration bound of the empirical PMF of model: with probability at least 1 − δ ,
[Equation]
We wish to obtain a optimistic transition model ˜ P kh ( s, a ) from the empirical one (cid:98) P kh ( s, a ) . To be morespecific, the return distribution η kh computed from ˜ P kh ( s, a ) and ν kh +1 should be more optimistic than the optimal one η ∗ h ( s, a ) with high probability. We thus define the distributional optimism operator
O
[Equation]
c : D ( S ) (cid:55)→ D ( S ) with level c and future return ν ∈ D S as
[Equation]
The ERM satisfy an interesting property that enables an efficient approach to perform O 1 c (see
Appendix B).
The following holds by using the induction
[Equation]
which verify the optimism of η kh ( s, a ) over η ∗ h ( s, a ) .
Regret Analysis
5.1 Regret upper bounds
Theorem 1 (Regret upper bound of RODI-MF ) .
For any δ ∈ (0 , 1) , with probability 1 − δ , the regretof Algorithm 1 under deterministic reward or Algorithm 4 under random reward is bounded as
Regret ( RODI-
[Equation]
[Equation] [Equation] The proof is given in Appendix D. Theorem 2 (Regret upper bound of RODI-MB / ROVI ) .
For any δ ∈ (0 , 1) , with probability 1 − δ , theregret of Algorithm 1/Algorithm 3 under deterministic reward or Algorithm 4/Algorithm 6 underrandom reward is bounded as
[Equation]
The proof is given in Appendix D.
The above results match the best-known results in [22].
Inparticular, our algorithms attain exponentially improved regret bounds than those of RSVI and RSQin [23] with a factor of exp( | β | H 2 ) .
By choosing | β | =
O (1 /H ) , we can eliminate the exponentialterm and achieve polynomial regret bound akin to the risk-neutral setting.
Compared to the traditional/non-distributional analysis dealing with one-dimensional values, ouranalysis is distribution-centered, called the distributional analysis .
The distributional analysis dealswith the distributions of the return rather than the risk measure values of the return.
For example, itinvolves the operations of the distributions, the optimism between different distributions, the errorcaused by estimation of distribution, etc.
These distributional aspects fundamentally differ from thetraditional analysis that deals with the one-dimensional scalars (value functions).
Now we recap thetechnical novelty of our analysis in the following.
Lipschitz continuity and linearity.
We identify two important properties of EERM that establishesthe regret upper bounds, including the Lipschitz continuity and linearity.
Denote by L M the Lipschitzconstant of the EERM E β : D ([0 , M ]) → R with respect to the infinity norm ∥·∥ ∞ .
Lemma 2provides a tight Lipschitz constant of EERM.
The Lipschitz constant relates the difference betweendistributions to the difference measured by their EERM values.
Lemma 2 (Lipschitz property of EERM) .
E β is Lipschitz continuous with respect to the supremumnorm over D M with L M = exp( | β | M ) − 1 .
Moreover, L M is tight in terms of both | β | and M .
Notice that lim β
→ 0 L M = 0 , which coincides with the fact that lim β → 0 E β = 1 .
The linearity of EERM is a key property that sharpens the regret bounds.
In contrast, EntRM is non-linear in thedistribution, which could induce a factor of exp( | β | H ) when controlling the error propagation acrosstime-steps.
It would further lead to a compounding factor of exp( | β | H 2 ) in the regret bound.
Insummary, the Lipschitz continuity property enables the regret upper bounds of DRL algorithms, andthe linearity tightens the bound.
Distributional optimism.
Another technical novelty in our analysis is the optimism in the face ofuncertainty at the distributional level.
The traditional analysis uses the OFU to construct a sequenceof optimistic value functions.
However, our analysis implements the distributional optimism thatyields a sequence of optimistic return distributions.
In particular, we first define a high probabilityevent, under which the true return distribution concentrates around the estimated one with a certainconfidence radius.
Then we apply the distributional optimism operator to obtain the optimisticallyplausible return distribution and the optimistic EntRM value.
Hence the regret can be bounded by thesurrogate regret, with the optimal EntRM value replaced by
[Equation]
Distributional analysis vs. non-distributional analysis.
When analyzing Algorithm 2/Algorithm3, proving the regret bound of either algorithm suffices due to their equivalence relation. Since
Algorithm 3 is a non-distributional algorithm, one may consider using the standard analysis thatdoes not involve distributions.
However, we show that this induces a factor of 1 | β | exp( | β | H ) , whichexplodes as | β | → 0 .
We overcome this issue by invoking a novel distributional analysis of Algorithm2, leading to the desired factor of 1 | β | (exp( | β | H ) − 1) .
Although we focus on the algorithms for the deterministic reward in the main text, the regret upperbounds also hold for case of random reward.
Algorithm 4, Algorithm 5 and Algorithm 6 correspondsto Algorithm 1, Algorithm 2 and Algorithm 3 respectively (cf.
Appendix C).
5.2 Regret lower bound
We provide more details of the mistakes in the lower bound of [23] in Appendix D.
The proof of [23]reduces the regret lower bound to the two-armed bandit regret lower bound.
Since the two-armedbandit is a special case of MDP with S = 1 , A = 2 and H = 1 , the reduction-based proof only leadsto a lower bound independent of S, A , and H .
Instead, our tight lower bound follows a totally differentroadmap motivated by [20].
[20] proves the tight minimax lower bound H √ SAT for risk-neutral
MDP. However, the generalization to risk-sensitive MDP is non-trivial. The main technical challengeis due to the non-linearity of EntRM.
The proof in [23] heavily relies on the linearity of expectation,allowing the exchange between taking the risk measure (expectation) and the summation. In therisk-sensitive setting, the non-linearity of EntRM requires new proof techniques. Assumption 1.
Assume S ≥ 6 , A ≥ 2 , and there exists an integer d such that S = 3 + A d − 1 A − 1 .
Wefurther assume that H [-(cid:21) 
-]≥ 3 d and ¯ H ≜ H 3 ≥ 1 .
Theorem 3 (Tighter lower bound) .
Assume Assumption 1 holds and β > 0 .
Let ¯ L ≜ (1 − 1 A )( S −3) + 1 A .
Then for any algorithm A , there exists an MDP M A such that for K ≥ 2 exp( β ( H − ¯ H −d ))
¯
H ¯ LA we have
[Equation]
The proof is given in Appendix D. Theorem 3 recovers the tight lower bound for standard episodic
MDP, implying that the exponential dependence on | β | and H in the upper bounds is indispensable. Yet, it is not clear whether a similar lower bound holds for β < 0 , which is left as a future direction.
Conclusion
We propose a risk-sensitive distributional dynamic programming framework.
We devise two novel
DRL algorithms, including a model-free one and a model-based one, which implement the OFUprinciple at the distributional level to balance the exploration and exploitation trade-off under therisk-sensitive setting.
We prove that both attain near-optimal regret upper bounds compared with ourimproved lower bound.
There are several promising future directions.
The current regret upper bound has an additional factor 311 √
HS compared with the lower bound.
It might be possible to remove the factor by designing newalgorithms or improving the analysis.
Besides, it is interesting to extend the DRL algorithm fromtabular MDP to linear function approximation setting.
Finally, it will be meaningful to investigatewhether the DDP framework holds for other risk measures.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately
reflect the
paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]