E FFICIENT R EINFORCEMENT L EARNING IN F ACTORED MDP S WITH A PPLICATION TO C ONSTRAINED RL
A BSTRACT
Reinforcement learning (RL) in episodic, factored Markov decision processes (FMDPs) is studied.
We propose an algorithm called FMDP-BF, whose regret is exponentially smaller than that of optimal algorithms designed for non-factored
MDPs, and improves on the previous FMDP result of Osband & Van Roy (2014b) by a factor of (cid:112) nH |S
i | , where |S i | is the cardinality of the factored state subspace, H is the planning horizon and n is the number of factored transitions.
We also provide a lower bound, which shows near-optimality of our algorithm w.r.t.
timestep T , horizon H and factored state-action subspace cardinality.
Finally, as an application, we study a new formulation of constrained RL, RL with knapsack constraints (RLwK), and provide the ﬁrst sample-efﬁcient algorithm based on FMDP-BF.
I NTRODUCTION
Reinforcement learning (RL) is concerned with sequential decision making problems where an agent interacts with a stochastic environment and aims to maximize its cumulative rewards.
The environment is usually modeled as a Markov Decision Process (MDP) whose transition kernel and reward function are unknown to the agent.
A main challenge of the agent is efﬁcient exploration in the
MDP, so as to minimize its regret or the related sample complexity of exploration.
Extensive study has been done on the tabular case in which almost no prior knowledge is assumed on the MDP dynamics.
The regret or sample complexity bounds typically depend polynomially on the cardinality of state and action spaces (e.g., Strehl et al., 2009; Jaksch et al., 2010; Azar et al., 2017; Dann et al., 2017; Jin et al., 2018; Dong et al., 2019; Zanette & Brunskill, 2019).
Moreover, matching lower bounds (e.g., Jaksch et al., 2010) imply that these results cannot be improved without additional assumptions.
On the other hand, many RL tasks involve large state and action spaces, for which these regret bounds are still excessively large.
In many practical scenarios, one can often take advantage of speciﬁc structures of the MDP to develop more efﬁcient algorithms.
For example, in robotics, the state may be high-dimensional, but the subspaces of the state may evolve independently of others, and only depend on a lowdimensional subspace of the previous state.
Formally, these problems can be described as factored
MDPs (Boutilier et al., 2000; Kearns & Koller, 1999; Guestrin et al., 2003).
Most relevant to the present work is Osband & Van Roy (2014b), who proposed a posterior sampling algorithm and a
UCRL-like algorithm that both enjoy √ T regret, where T is the maximum timestep.
Their regret bounds have a linear dependence on the time horizon and each factored state subspace.
It is unclear whether this bound is tight or not.
In this work, we tackle this problem by proposing algorithms with improved regret bounds, and developing corresponding lower bounds for episodic FMDPs.
We propose a sample- and computationefﬁcient algorithm called FMDP-BF based on the principle of optimism in the face of uncertainty , and prove its regret bounds.
We also provide a lower bound, which implies that our algorithm is near-optimal with respect to the timestep T , the planning horizon H and factored state-action subspace cardinality |X
[ Z i ] | .
As an application, we study a novel formulation of constrained RL, known as RL with knapsack constraints ( RLwK ), which we believe is natural to capture many scenarios in real-life applications.
We apply FMDP-BF to this setting, to obtain a statistically efﬁcient algorithm with a regret bound that is near-optimal in terms of T , S , A , and H .
Our contributions are summarized as follows:
We propose an algorithm for FMDP, and prove its regret bound that improves on the previous result of Osband & Van Roy (2014b) by a factor of (cid:112) nH
|S i | .
We prove a regret lower bound for FMDP, which implies that our regret bound is nearoptimal in terms of timestep T , horizon H and factored state-action subspace cardinality.
We apply FMDP-BF in RLwK, a novel constrained RL setting with knapsack constraints, and prove a regret bound that is near-optimal in terms of T, S , A and H .
P RELIMINARIES
We consider the setting of a tabular episodic Markov decision process (MDP), ( S , A , H, P , R ) , where S is the set of states, A is the action set, H is the number of steps in each episode.
P is the transition probability matrix so that P ( ·| s, a ) gives the distribution over states if action a is taken on state s , and R ( s, a ) is the reward distribution of taking action a on state s with support [0 , 1] .
We use ¯ R ( s, a ) to denote the expectation E
[ R ( s, a )] .
In each episode, the agent starts from an initial state s 1 that may be arbitrarily selected.
At each step h ∈
[ H ] , the agent observes the current state s h ∈ S , takes action a h ∈ A , receives a reward r h sampled from R ( s h , a h ) , and transits to state s h +1 with probability P
( s h
+
| s h , a h ) .
The episode ends when s H +1 is reached.
A policy π is a collection of H policy functions { π h : S → A} h ∈
[ H ] .
We use V πh : S → R to denote the value function at step h under policy π , which gives the expected sum of remaining rewards received under policy π starting from s h
= s , i.e. V πh ( s ) =
E (cid:104)(cid:80) Hh (cid:48)
=
h R ( s h (cid:48) , π h (cid:48)
( s h (cid:48) )) | s h
= s (cid:105) .
Accordingly, we deﬁne Q πh ( s, a ) as the expected Q-value function at step h : Q πh ( s, a ) =
E (cid:104) R ( s h , a h ) + (cid:80) H h (cid:48)
=
h +1 R ( s h (cid:48) , π h (cid:48)
( s h (cid:48) )) | s h
= s, a h = a (cid:105) .
We use V ∗ h and Q ∗ h to denote the optimal value and Q-functions under optimal policy π ∗ at step h .
The agent interacts with the environment for K episodes with policy π k = { π k,h : S → A} h ∈ [ H ] determined before the k -th episode begins.
The agent’s goal is to maximize its cumulative rewards (cid:80)
Kk =1 (cid:80)
Hh =1 r k,h over T = KH steps, or equivalently, to minimize the following expected regret:
[Equation]
where s k, 1 is the initial state of episode k .
2. F ACTORED MDP S
A factored MDP is an MDP whose rewards and transitions exhibit certain conditional independence structures.
We start with the formal deﬁnition of factored MDP (Boutilier et al., 2000; Osband &
Van Roy, 2014b; Xu & Tewari, 2020; Lu & Van Roy, 2019).
Let P ( X , Y ) denote the set of functions that map x ∈ X to the probability distribution on Y .
Deﬁnition 1.
(Factored set)
Let X
= X 1 × · · · × X d be a factored set.
For any subset of indices
Z ⊆ { 1 , 2 , . . .
, d } , we deﬁne the scope set X
[ Z ] := ⊗ i
∈
Z X i .
Further, for any x ∈ X , deﬁne the scope variable x
[ Z ] ∈ X
[ Z ] to be the value of the variables x i ∈
X i with indices i ∈
Z .
If Z is a singleton, we will write x [ i ] for x [ { i } ] .
Deﬁnition 2.
(Factored reward)
The reward function class R ⊂ P ( X , R ) is factored over S ×
A =
X = X 1 × · · · × X d with scopes Z 1 , · · · , Z m if for all R ∈ R , x ∈ X , there exist functions { R i ∈ P ( X [ Z i ] , [0 , 1]) } mi =1 such that r ∼ R ( x ) is equal to 1 m (cid:80) mi =1 r i with each r i ∼ R i
( x [ Z i ]) individually observed.
We use ¯ R i to denote the expectation E
[ R i ] .
Deﬁnition 3.
(Factored transition)
The transition function class P ⊂ P ( X , S ) is factored over
S ×
A = X
= X 1 × · · · × X d and S = S 1 × · · · × S n with scopes Z 1 , · · · , Z n if and only if, for all P ∈ P , x ∈ X , s ∈ S , there exist functions { P j ∈ P ( X [ Z j ] , S j ) } nj =1 such that P ( s | x ) = (cid:81) nj =1 P j ( s [ j ] | x [ Z j ]) .
A factored MDP is an MDP with factored rewards and transitions.
A factored MDP is fully characterized by M = (cid:16) {X i } di =1 ; (cid:8) Z Ri (cid:9) m i =1 ; { R i } mi =1 ; {S j } nj =1 ; (cid:8) Z Pj (cid:9) n j =1 ; { P j } nj =1 ; H (cid:17) , where
X = S × A , { Z Ri } mi =1 and { Z Pj } nj =1 are the scopes for the reward and transition functions, which we assume to be known to the agent.
An excellent example of factored MDP is given by Osband & Van Roy (2014), about a large production line with d machines in sequence with S i possible states for machine i .
Over a single time-step each machine can only be inﬂuenced by its direct neighbors.
For this problem, the scopes Z Ri and
Z Pi of machine i ∈ { 2 , ..., d − 1 } can be deﬁned as { i − 1 , i, i + 1 } , and the scopes of machineand machine d are { 1 , 2 } and { d − 1 , d } respectively.
Another possible example to explain factored
MDP is about robotics.
For a robot, the transition dynamics of its different parts (e.g. its legs and arms) may be relatively independent.
In that case, the factored transition can be deﬁned for each part separately.
For notation simplicity, we use X
[ i : j ] and S [ i : j ] to denote X [ ∪ k = i,...,j Z k ] and ⊗ jk = i S k respectively.
Similarly, We use P [ i : j ] ( s (cid:48)
[ i : j ] | s, a ) to denote (cid:81) jk =
i P ( s (cid:48)
[ k ] | ( s, a )[ Z Pk ]) .
For every
V : S → R and the right-linear operators P , we deﬁne P V ( s, a ) def = (cid:80) s (cid:48) ∈S P ( s (cid:48) | s, a ) V ( s (cid:48) ) .
A state-action pair can be represented as ( s, a ) or x .
We also use ( s, a )[ Z ] to denote the corresponding x [ Z ] for notation convenience.
We mainly focus on the case where the total time step T =
KH is the dominant factor, and assume that T ≥ |X i | ≥ H during the analysis.
R ELATED W ORK
Exploration in Reinforcement Learning Recent years have witnessed a tremendous of work for provably efﬁcient exploration in reinforcement learning, including tabular MDP (e.g., Dann et al., 2017; Azar et al., 2017; Jin et al., 2018; Zanette & Brunskill, 2019), linear RL (e.g., Jiang et al., 2017; Yang & Wang, 2019; Jin et al., 2020; Zanette et al., 2020), and RL with general function approximation (e.g., Osband & Van Roy, 2014a; Ayoub et al., 2020; Wang et al., 2020).
For algorithms in tabular setting, the regret bounds inevitably depend on the cardinality of state-action space, which may be excessively large.
Based on the concept of eluder dimension (Russo & Van Roy, 2013), many recent works proposed efﬁcient algorithms for RL with general function approximation (Osband & Van Roy, 2014a; Ayoub et al., 2020; Wang et al., 2020). Since eluder dimension of the function class for factored MDPs is at most O (cid:0)(cid:80) mi =1 |X [ Z Ri ] | + (cid:80) ni =1 |X
[ Z Pi ] | (cid:1) , it is possible to apply their algorithms and regret bounds to our setting, though the direct application of their algorithms leads to a loose regret bound.
Factored MDP Episodic FMDP was studied by Osband & Van Roy (2014b), in which they proposed both PSRL and UCRL style algorithm with near-optimal Bayesian and frequentist regret bound.
In non-episodic scenarios, Xu & Tewari (2020) recently generalizes the algorithm of Osband & Van Roy (2014b) to the inﬁnite horizon average reward setting.
However, both their results suffer from linear dependence on the horizon (or the diameter) and factored state space’s cardinality.
Concurrent with our work is the recent paper by Tian et al.
(2020), which also applies UCBVI and EULER to factored MDPs.
Compared with their results, we propose a more reﬁned variance decomposition theorem for factored Markov chains (Theorem 1), which results in a better regret by a factor of √ n ; the theorem is also of independent interest with potential use in other problems in factored MDPs.
Furthermore, we formulate the RLwK problem, and provide a sample-efﬁcient algorithm based on our FMDP algorithm.
Constrained MDP and knapsack bandits The knapsack setting with hard constraints has already been studied in bandits with both sample-efﬁcient and computational-efﬁcient algorithms (Badanidiyuru et al., 2013; Agrawal et al., 2016).
This setting may be viewed as a special case of RLwK with H = 1 .
In constrained RL, there is a line of works that focus on soft constraints where the constraints are satisﬁed in expectation or with high probability (Brantley et al., 2020; Zheng & Ratliff, 2020), or a violation bound is established (Efroni et al., 2020; Ding et al., 2020).
RLwK requires stronger constraints that is almost surely satisﬁed during the execution of the agents.
A more related setting is proposed by Brantley et al.
(2020), which studies a sample-efﬁcient algorithm for knapsack episodic setting with hard constraints on all K episodes.
However, we require the constraints to be satisﬁed within each episode , which we believe can better describe the real-world scenarios.
The setting of Singh et al.
(2020) is closer to ours since they are focusing on “every-time” hard constraints, although they consider the non-episodic case.
M AIN R ESULTS
In this section, we introduce our FMDP-BF algorithm, which uses empirical variance to construct a Bernstein-type conﬁdence bound for value estimation.
Besides FMDP-BF, we also propose a simpler algorithm called FMDP-CH with a slightly worse regret, which follows the similar idea of
UCBVI-CH (Azar et al., 2017).
The algorithm and the corresponding analysis are more concise and easy to understand; details are deferred to Section B.
4. E STIMATION E RROR D ECOMPOSITION
Our algorithm will follow the principle of “optimism in the face of uncertainty”.
Like ORLC (Dann et al., 2019) and EULER (Zanette & Brunskill, 2019), our algorithm also maintains both the optimistic and pessimistic estimates of state values to yield an improved regret bound.
We use V k,h and V k,h to denote the optimistic estimation and pessimistic estimation of V ∗ h , respectively.
To guarantee optimism, we need to add conﬁdence bonus to the estimated value function V k,h at each step, so that V k,h ( s ) ≥ V ∗ h ( s ) holds for any k ∈ [ K ] ,
h ∈
[ H ] and s ∈ S .
Suppose ˆ R k,i and ˆ P k,j denote the estimated value of each expected factored reward R i and factored transition probability P j before episode k respectively.
By the deﬁnition of the reward R and the transition P , we use ˆ R def = 1 m (cid:80) mi =1 ˆ R k,i and ˆ P k def = (cid:81) nj =1 ˆ P k,j as the estimation of ¯ R and P .
Following the previous framework, this conﬁdence bonus needs to tightly characterize the estimation error of the one-step backup ¯ R ( s, a )+ P V ∗ h ( s, a ) ; in other words, it should compensate for the estimation errors, (cid:16) ˆ R k − ¯ R (cid:17) ( s, a ) and (cid:16) ˆ P k − P (cid:17) V ∗ h ( s, a ) , respectively.
For the estimation error of rewards (cid:16) ˆ R k − ¯ R (cid:17) ( s k,h , a k,h ) , since the reward is deﬁned as the average of m factored rewards, it is not hard to decompose the estimation error of ¯ R ( s, a ) to the average of the estimation error of each factored rewards.
In that case, we separately construct the conﬁdence bonus of each factored reward ¯ R i .
Suppose CB Rk,Z Ri ( s, a ) is the conﬁdence bonus that compensates for the estimation error
ˆ R k,i − ¯ R i , then we have CB Rk ( s, a ) def = 1 m (cid:80) mi
=1 CB Rk,Z Ri ( s, a ) .
For the estimation error of transition (cid:16) ˆ P k − P (cid:17) V ∗ h
+1 ( s k,h , a k,h ) , the main difﬁculty is that ˆ P k is the multiplication of n estimated transition dynamics ˆ P k,i .
In that case, the estimation error (cid:16) ˆ P k − P (cid:17) V ∗ h
+
( s k,h , a k,h ) may be calculated as the multiplication of n estimation error for each factored transition ˆ P k,i , which makes the analysis much more difﬁcult.
Fortunately, we have the following lemma to address this challenge.
Lemma 4.1.
(Informal) Let the transition function class P ∈ P ( X , S ) be factored over X =
X 1 × · · · × X d , and S = S 1 × · · · × S n with scopes Z P 1 , · · · , Z Pn .
For a given function V : S → R , the estimation error of one-step value | (ˆ P k − P ) V ( s, a ) | can be decomposed by:
[Equation]
Here, β k,h ( s, a ) , formally deﬁned in Lemma E.1, are higher order terms that do not harm the order of the regret.
This lemma allows us to decompose the estimation error (cid:16) ˆ P k − P (cid:17) V ∗ h
+
( s k,h , a k,h )
into an additive form, so we can construct the conﬁdence bonus for each factored transition P j separately.
Let CB Pk,Z Pj ( s, a ) be the conﬁdence bonus for the estimation error (ˆ P k,j − P
[Equation] (cid:16)(cid:81) [Equation]
s, [Equation]
Then, CB [Equation] [Equation] [Equation]
[Equation]
+ η k,h ( s, a ) , where η k,h ( s, a ) collects higher order factors that will be explicitly given later.
Finally, we deﬁne the conﬁdence bonus as the summation of all conﬁdence bonuses for rewards and transition: CB k ( s, a ) =
CB R k ( s, a ) + CB P k ( s, a ) .
4. V ARIANCE OF F ACTORED M ARKOV C HAINS
After the analysis in Section 4.1, the remaining problem is how to deﬁne the conﬁdence bonus
CB Rk,Z Ri ( s, a ) and CB Pk,Z Pj ( s, a ) .
In this subsection, we tackle this problem by deriving the variance decomposition formula for factored MDP.
To begin with, we consider Markov chains with stochastic factored transition and stochastic factored rewards, and deduce the Bellman equation of variance for factored Markov chains.
The analysis shows how to deﬁne the empirical variance in the conﬁdence bonus for factored MDP and gives an upper bound on the summation of per-step variance (Corollary 1.1).
In the Markov chain setting, the reward is deﬁned to be a mapping from S to R .
Suppose J t 1 : t 2 ( s ) denotes the total rewards the agent obtains from step t 1 to step t 2 (inclusively), given that the agent starts from state s in step t 1 .
J t 1 : t 2 is a random variable depending on the randomness of the trajectory from step t 1 to t 2 , and stochastic rewards therein.
Following this deﬁnition of J t 1 : t 2 , we deﬁne J 1: H to be the total reward obtained during one episode.
We use s t to denote the random state that the agent encounters at step t .
We deﬁne ω 2 h ( s ) def = E (cid:104)
( J h : H ( s h ) − V h
( s h ))
2 | s h
= s (cid:105) to be the variance of the total gain after step h , given that s h
= s .
We deﬁne σ 2 R,i ( s ) def =
V [ R i ( ξ )
| ξ = s ] to be the variance of the i -th factored reward, given that the current state is s .
Given the current state s , we deﬁne the variance of the next-state value function w.r.t.
the i -th factored transition as:
[Equation]
E s
[-+
-][Equation]
1 [1: i − 1] (cid:2) V s h +
[ i ] (cid:2) E s h +
[ i +1: n ] [ V h +
( s h +1 )] (cid:3) | s h
= s (cid:3) .
That is, for each given s (cid:48)
[1 : i ] , we ﬁrstly take expectation over all possible values of s (cid:48)
[ i + 1 : n ] w.r.t.
P [ i +1: n ] .
Then, we calculate the variance of transition s (cid:48)
[ i ] ∼ P i ( ·| ( s, a )[ Z Pi ]) given ﬁxed s (cid:48)
[1 : i − 1] .
Finally, we take the expectation of this variance w.r.t.
s (cid:48)
[1 : i − 1] ∼ P [1: i − 1] .
Theorem 1.
For any horizon h ∈ [ H ] , we have ω 2 h ( s ) =
(cid:80) s (cid:48)
P ( s (cid:48) | s ) ω 2 h +1 ( s (cid:48) ) + (cid:80) ni =1 σ 2 P,i,h ( s ) + 1 m 2 (cid:80) mi =1 σ 2 R,i ( s ) .
Theorem 1 generalizes the analysis of Munos & Moore (1999), which deals with non-factored MDPs and deterministic rewards.
From the Bellman equation of variance, we can give an upper bound to the expected summation of per-step variance.
Corollary 1.1.
Suppose the agent takes policy π during an episode.
Let w h
( s, a ) denote the probability of entering state s and taking action a in step h .
Then we have the following inequality:
[Equation]
where σ
[Equation]
is the variance of i -th factored reward given the current state-action pair ( s, a ) , and
[Equation]
=[-E s h +
 [Equation]-]
[Equation]
is the variance of
[Equation]
factored transition given current state s .
This corollary makes it possible to construct conﬁdence bonus with variance for each factored rewards and transition separately.
Please refer to Section F.2 for the detailed proof of Theorem 1 and
Corollary 1.1.
4. A LGORITHM
Our algorithm is formally described in Alg. 1, with a more detailed explanation in Section C. Denote by N k
(( s, a )[ Z ]) the number of steps that the agent encounters ( s, a )[ Z ] during the ﬁrst k episodes.
In episode k , we estimate the mean value of each factored reward R i and each factored transition P i with empirical mean value ˆ R k,i and ˆ P k,i of the previous history data L respectively.
After that, we construct the optimistic MDP ˆ M based on the estimated rewards and transition functions.
For a certain ( s, a ) pair, the transition function and reward function are deﬁned as ˆ R k ( s, a ) = 1 m (cid:80) mi
=1 ˆ R k,i (( s, a )
[ Z Ri ]) and ˆ P k ( s (cid:48) | s, a ) =
(cid:81) nj
=1 ˆ P k,j (cid:0) s (cid:48)
[ j ] | ( s, a ) (cid:2) Z Pj (cid:3)(cid:1) .
Algorithm
Input : δ
L = ∅ , initialize N (( s, a )[ Z i ])
=
for any factored set Z i and
any ( s, a )[ Z i ] ∈ X
[ Z i ] for episode k = 1 , 2 , · · · do
Set V k,H +1 ( s ) =
V k,H +1 ( s ) =
for all s, a .
Estimate the empirical mean ˆ R k and ˆ P k with history data L .
for horizon h = H, H − 1 , ..., 1 do for s ∈ S do for a ∈ A do
Q k,h ( s, a ) = min { H, ˆ R k ( s, a ) + CB k ( s, a ) + ˆ P k V k,h +1 ( s, a ) } 10: end for π k,h ( s ) =
arg max a Q k,h ( s, a )
V k,h ( s ) = max a ∈A Q k,h ( s, a )
V k,h ( s ) = max (cid:110) 0 , ˆ R k ( s, π k,h
( s ))
− CB k ( s, π k,h
( s ))
+ ˆ P k V k,
h +1 ( s, π k,h
( s ))
(cid:111) end for 15: end for
Take action according to π k,h for H steps in this episode.
Update L = L (cid:83) { s k,h , a k,h , r k,h , s k,h +1 } h =1 , 2 ,...,H , and update counter N k − 1 (( s, a )[ Z i ]) .
end for Following the analysis in Section 4.1, we separately construct the conﬁdence bonus of each factored reward R i with the empirical variance: CB Rk,Z Ri ( s, a ) = (cid:114) 2ˆ σ 2 R,k,i ( s,a ) L Ri N k − 1 (( s,a )[ Z Ri ]) + 8 L Ri 3 N k − 1 (( s,a )[ Z Ri ]) , i ∈ [ m ] , where L Ri def = log (cid:0) 18 mT (cid:12)(cid:12)
X
[ Z Ri ] (cid:12)(cid:12) /δ (cid:1) , and ˆ σ 2 R,k,i is the empirical variance of the i -th factored reward R i , i.e. ˆ σ 2 R,k,i ( s, a ) =
[Equation]
We deﬁne L P = log (18 nTSA/δ ) for short.
Following the idea of Lemma 4.1, we separately construct the conﬁdence bonus of scope Z Pi for transition estimation: CB Pk,Z Pi ( s, a ) = [Equation]
[Equation]
, i ∈ [ n ] , where ˆ σ P,k,i ( s, a ) and u k,h,i ( s, a ) are deﬁned later.
η k,h,i ( s, a ) collects the additional bonus terms that do not affect the order of the ﬁnal regret.
The precise expression of η k,h,i ( s, a ) is deferred to Section C.
The deﬁnition of ˆ σ 2 P,k,i ( V k,h +1 , s, a ) corresponds to σ 2 P,
i ( V πh , s, a ) in Corollary 1.1, which can be regarded as the empirical variance of transition P k,i :
[Equation]
To guarantee optimism, we need to use the empirical variance ˆ σ 2 P,k,i ( V ∗ , s, a ) to upper bound the estimation error in the proof.
Since we do not know V ∗ beforehand, we use ˆ σ 2 P,k,i ( V k,h +1 , s, a ) as a surrogate in the conﬁdence bonus.
However, we cannot guarantee that ˆ σ 2 P,k,i ( V ∗ , s, a ) is upper bounded by ˆ σ 2 P,k,i ( V k,h +1 , s, a ) .
To compensate for the error due to the difference between V ∗ h
+ and V k,
h
+1 , we add (cid:113) 2 u k,h,i ( s,a ) L P N k − 1 (( s,a )[ Z Pi ]) to the conﬁdence bonus, where u k,h,i ( s, a ) is deﬁned as:
[Equation]
[ i +1: n ] ∼
[Table]
P k, [ i +1: n ] ( ·| s,a ) (cid:0) V k,
[Table]
+1 − V k,
[Table]
+1 (cid:1) ( s (cid:48) )
(cid:17) 2 (cid:21) .
4. R EGRET
Theorem 2.
Suppose (cid:12)(cid:12) X
[ Z Ri ] (cid:12)(cid:12) ≤ J R , (cid:12)(cid:12)
X
[ Z Pj ] (cid:12)(cid:12) ≤ J P for i ∈ [ m ] , j ∈ [ n ] , then with prob.
1 − δ , the regret of Alg.
1 is O (cid:16) (cid:112) J R T log( mTJ R /δ )
log T + (cid:112) nHJ P T log( nTSA/δ ) log T (cid:17) .
Note that the regret bound does not depend on the cardinalities of state and action spaces, but only has a square-root dependence on the cardinality of each factored subspace X
[ Z i ] .
By leveraging the structure of factored MDP, we achieve regret that scales exponentially smaller compared with that of UCBVI (Azar et al., 2017).
The best previous regret bound for episodic factored
MDP is achieved by Osband & Van Roy (2014b).
When transformed to our setting, it becomes
O (cid:16)(cid:112) J R T log( mTJ R /δ )
+ nH (cid:112) Γ J P T log( nTJ P /δ ) (cid:17) , where Γ is an upper bound of |S j | .
This is worse than our results by a factor of √ nH Γ .
Concurrent to our results, Tian et al.
(2020) also propose efﬁcient algorithms for episodic factored MDP.
When transformed to our setting, their regret bound is ˜ O (cid:16) √ J R T + n √ HJ P T (cid:17) .
Compared with their bounds, we further improve the regret by a factor of √ n with a more reﬁned variance decomposition theorem (Theorem 1).
4. L OWER B OUND
In this subsection, we propose the regret lower bound for factored MDP.
The proof of Theorem 3 is deferred to Section G.
Theorem 3.
Suppose log 2 ( |S i | )
≤ H 2 for any i ∈ [ n ] , the regret of any algorithm on the factored
MDP problem is lower bounded by Ω (cid:16) 1 m (cid:80) mi =1 (cid:113)(cid:12)(cid:12) X
[ Z Ri ] (cid:12)(cid:12) T + 1 n (cid:80) ni
=1 (cid:113)(cid:12)(cid:12)
X [ Z Pi ] (cid:12)(cid:12) HT (cid:17) .
The lower bound of Tian et al.
(2020) is Ω (cid:16) max (cid:110) max i (cid:113)(cid:12)(cid:12)
X
[ Z Ri ] (cid:12)(cid:12) T, max j
(cid:113)(cid:12)(cid:12) X
[ Z Pi ] (cid:12)(cid:12) HT (cid:111)(cid:17) , which is derived from different hard instance construction.
Their lower bound is of the same order with ours, while our bound measures the dependence on all the parameters including the number of the factored transition n and the factored rewards m .
If (cid:12)(cid:12) X
[ Z Ri ] (cid:12)(cid:12) =
J R and (cid:12)(cid:12) X
[ Z Pi ] (cid:12)(cid:12) =
J P , the lower bound turns out to be Ω (cid:16) √ J R T + √ HJ P T (cid:17) , which matches the upper bound in Theoremexcept for a factor of √ n and logarithmic factors.
RL WITH K NAPSACK C ONSTRAINTS
In this section, we study RL with Knapsack constraints, or RLwK, as an application of FMDP-BF.
5. P RELIMINARIES
We generalize bandit with knapsack constraints or BwK (Badanidiyuru et al., 2013; Agrawal et al., 2016) to episodic MDPs.
We consider the setting of tabular episodic Markov decision process, ( S , A , H, P , R, C ) , which adds to an episodic MDP with a d -dimensional stochastic cost vector,
C ( s, a ) .
We use C i ( s, a ) to denote the i -th cost in the cost vector C ( s, a ) .
If the agent takes action a in state s , it receives reward r sampled from R ( s, a ) , together with cost c , before transitioning to the next state s (cid:48) with probability P ( s (cid:48) | s, a ) .
In each episode, the agent’s total budget is B .
We also use B i to denote the total budget of i -th cost.
Without loss of generality, we assume B i ≤ B for all i .
An episode terminates after H steps, or when the cumulative cost (cid:80) h c h,i of any dimension i exceeds the budget B i , whichever occurs ﬁrst.
The agent’s goal is to maximize its cumulative reward (cid:80) Kk =1 (cid:80)
Hh =1 r k,h in K episodes.
5. C OMPARISON WITH O THER S ETTINGS
While RLwK might appear similar to episodic constrained RL (Efroni et al., 2020; Brantley et al., 2020), it is fundamentally different, so those algorithms cannot be applied here.
As discussed in Section 3, the episodic constrained RL setting can be roughly divided into two categories.
A line of works focus on soft constraints where the constraints are satisﬁed in expectation, i.e. (cid:80)
Hh =1 E [ c k,h ] ≤ B .
The expectation is taken over the randomness of the trajectories and the random sample of the costs.
Another line of work focuses on hard constraints in K episodes.
To be more speciﬁc, they assume that the total costs in K episodes cannot exceed a constant vector B , i.e. (cid:80) Kk =1 (cid:80)
Hh =1 c k,h ≤ B .
Once this is violated before episode K 1 < K , the agent will not obtain any rewards in the remaining K − K 1 episodes.
Though both settings are interesting and useful, they do not cover many common situations in constrained RL.
For example, when playing games, the game is over once the total energy or health reduce to 0 .
After that, the player may restart the game (starting a new episode) with full initial energy again.
In robotics, a robot may episodically interact with the environment and learn a policy to carry out a certain task.
The interaction in each episode is over once its energy is used up.
In these two examples, we cannot just consider the expected cost or the cumulative cost across all episodes, but calculate the cumulative cost in every individual episode.
Moreover, in many constrained RL applications, the agent’s optimal action should depend on its remaining budget.
For example, in robotics, the robot should do planning and take actions based on its remaining energy.
However, previous results do not consider this issue, and use policies that map states to actions.
Instead, in RLwK, we need to deﬁne the policy as a mapping from states and remaining budget to actions.
Section H gives further details, including two examples for illustrating the difference between these settings.
5. A LGORITHM
We make the following assumptions about the cost function for simplicity.
Both of them hold if all the stochastic costs are integers with an upper bound.
Assumption 1.
The budget B i as well as the possible value of costs C i ( s, a ) of any state s and action a is an integral multiple of the unit cost 1 m .
Assumption 2.
The stochastic cost C i ( s, a ) has ﬁnite support.
That is, the random variable
C i ( s, a ) can only take at most n possible values.
The reason for Assumption 2 is that we need to estimate the distribution of the cost, instead of just estimating its mean value.
We discuss the necessity of the assumptions and the possible methods for continuous distribution in Section H.3.
From the above discussion, we know that we need to ﬁnd a policy that is a mapping from state and budget to action.
Therefore, it is natural to augment the state with the remaining budget.
It follows that the size of augmented state space is S · ( Bm ) d .
Directly applying UCBVI algorithm (Azar et al., 2017) will lead to a regret of order O (cid:16)(cid:112) HSAT ( Bm ) d (cid:17) .
Our key observation is that the constructed state representation can be represented as a product of subspaces.
Each subspace is relatively independent.
For example, the transition matrix over the original state space S is independent of the remaining budget.
Therefore, the constructed MDP can be formulated as a factored MDP, and the compact structure of the model can reduce the regret signiﬁcantly.
By applying Alg.
1 and Theorem 2 to RLwK, we can reduce the regret to the order of
O (cid:16)(cid:112) HSA (1 + dBm ) T (cid:17) roughly, which is exponentially smaller.
However, the regret still de- pends on the total budget B and the discretization precision m , which may be very large for continuous budget and cost.
Another observation to tackle the problem is that the cost of taking action a on state s only depends on the current state-action pair ( s, a ) , but has no dependence on the remaining budget B .
To be more formal, we have b h
+1 =
b h − c h , where b h is the remaining budget at step h , and c h is the cost suffered in step h .
As a result, we can further reduce the regret to roughly
O (cid:16) √ HdSAT (cid:17) by estimating the distribution of cost function.
A similar model has been discussed in Brunskill et al.
(2009), which is named as noisy offset model.
Our algorithm, which is called FMDP-BF for RLwK, follows the same basic idea of Alg.
We defer the detailed description to Section H to avoid redundance.
The regret can be upper bounded by the following theorem:
Theorem 4.
With prob.
at least 1 − δ , the regret of Alg. 4 is upper bounded by
[Equation]
Compared with the lower bound for non-factored tabular MDP (Jaksch et al., 2010), this regret bound matches the lower bound w.r.t.
S , A , H and T .
There may still be a gap in the dependence of the number of constraints d , which is often much smaller than other quantities.
It should be noted that, though we achieve a near-optimal regret for RLwK, the computational complexity is high, scaling polynomially with the maximum budget B , and exponentially with the number of constraints d .
This is a consequence of the NP-hardness of knapsack problem with multiple constraints (Martello, 1990; Kellerer et al., 2004).
However, since the policy is deﬁned on the state and budget space with cardinality SB d , this computational complexity seems unavoidable.
How to tackle this problem, such as with approximation algorithms, is an interesting future work.
C ONCLUSION
We propose a novel RL algorithm for solving FMDPs with near optimal regret guarantee.
It improves the best previous regret bound by a factor of (cid:112) nH
|S i | .
We also derive a regret lower bound for FMDPs based on the minimax lower bound of multi-armed bandits and episodic tubular MDPs (Jaksch et al., 2010).
Further, we formulate the RL with Knapsack constraints (RLwK) setting, and establish the connections between our results for FMDP and RLwK by providing a sample efﬁcient algorithm based on FMDP-BF in this new setting.
A few problems remain open.
The regret upper and lower bounds have a gap of approximately √ n , where n is the number of transition factors.
For RLwK, it is important to develop a computationally efﬁcient algorithm, or ﬁnd a variant of the hard-constraint formulation.
We hope to address these issues in the future work.
A CKNOWLEDGEMENTS
This work was supported by Key-Area Research and Development Program of Guangdong Province (No. 2019B121204008)], National Key R&D Program of China (2018YFB1402600), BJNSF (L172037) and Beijing Academy of Artiﬁcial Intelligence.
A N OTATIONS
Before presenting the proof
[Table]
we restate the deﬁnition of the following notations.
Symbol
Explanation
[Equation]
The state and action that the agent encounters in episode k and step h
[Equation]
log
[Table]
18 nTSA/
[Table]
[Equation]
log
[Table]
[Equation]
[Table]
[Equation]
[Table]
the number of steps that the agent encounters ( s, a )[ Z ] during the ﬁrst k episodes
[Table]
A shorthand of (cid:80) s
[Table]
∈S P ( s (cid:48) | s, a ) V ( s (cid:48) )
[Table]
[Equation]
[Table]
The empirical variance of reward R
i
[Table]
the next state variance of P V for the transition P
i , i
[Table]
e. E
[Table]
the empirical next state variance of ˆ P k V for the transition ˆ P k,i , i.e. E s (cid:48)
[1: i − 1] ∼ ˆ P k, [1: i ] ( ·| s,a ) (cid:104) V s (cid:48)
[ i ] ∼ ˆ P k,i ( ·| ( s,a )[ Z Pi ]) (cid:16) E s (cid:48)
[ i +1: n ] ∼ ˆ P k, [ i +1: n ] ( ·| s,a ) V ( s (cid:48) )
(cid:17)(cid:105) Ω k,h
The optimism and pessimism event for k, h : (cid:8) V k,h ≥
V ∗ h ≥ V k,h (cid:9) Ω
The optimism and pessimism events for all 1 ≤ k ≤ K, 1 ≤ h ≤ H , i.e. ∪ k,h Ω k,h w k,h,Z ( s, a )
The probability of entering ( s, a )[ Z ] at step h in episode k w k,Z ( s, a ) (cid:80) Hh =
[Equation]
[Equation]
,Z
[Equation]
s, a ) w k,h ( s, a )
The probability of entering ( s, a ) at step h in episode k , i.e. w k,h,Z ( s, a ) with Z = { 1 , 2 , ..., d } w k ( s, a ) (cid:80) Hh =1 w k,h
( s, a )
B O MITTED DETAILS FOR FMDP-CH
In this section, we introduce our algorithm with Hoeffding-type conﬁdence bonus and present the corresponding regret bound.
Our algorithm, which is described in Algorithm 2, is related to UCBVICH algorithm
(Azar et al., 2017), in the sense that Algorithm 2 reduces to UCBVI-CH if we consider a ﬂat MDP with m = n =
d = 1 .
Let N k
(( s, a )[ Z ]) denote the number of steps that the agent encounters ( s, a )
[ Z ] during the ﬁrst k episodes, and N k
(( s, a )[ Z j ] , s j ) denotes the number of steps that the agent transits to a state with s [ j ] =
s j after encountering ( s, a )
[ Z j ] during the ﬁrst k episodes.
In episode k , we estimate the mean value of each factored reward R i and each factored transition P i with empirical mean value ˆ R k,i and ˆ P k,i respectively.
To be more speciﬁc, ˆ R k,i (( s, a )[ Z Ri ]) = (cid:80) t ≤ ( k − 1)
H
[ ( s t ,a t )
[ Z Ri ]=( s,a )[ Z Ri ] ] · r t,i N k − 1 (( s,a )[ Z Ri ]) , where r t,i denotes the reward
R i sampled in step t , and ˆ P k,j (cid:0) s
[ j ] | ( s, a )[ Z Pj ] (cid:1) =
N k − 1 (( s,a )[ Z Pj ] ,s [ j ]) N k − 1 (( s,a )
[ Z Pj ]) .
After that, we construct the optimistic MDP ˆ M based on the estimated rewards and transition functions.
For a certain ( s, a ) pair, the transition function and reward function are deﬁned as ˆ R k ( s, a ) = 1 m (cid:80) mi
=1 ˆ R k,i (( s, a )
[ Z Ri ]) and ˆ P k ( s (cid:48) | s, a ) =
(cid:81) nj
=1 ˆ P k,j (cid:0) s (cid:48)
[ j ] | ( s, a ) (cid:2) Z Pj (cid:3)(cid:1) .
We deﬁne L Ri = log (cid:0) 18 mT (cid:12)(cid:12) X [ Z Ri ] (cid:12)(cid:12) /δ (cid:1) , L P = log (18 nTSA/δ ) and φ k,i ( s, a )
= (cid:113) 4 |S
[Equation]
k −
[Equation]
[Equation]
[Equation]
]) + 4 |S
i | L 3 N k − 1 (( s,a )[ Z P i ]) .
We separately construct the conﬁdence bonus of each fac-
Input : δ , history data L = ∅ , initialize N (( s, a )[ Z i ]) =
0 for any factored set Z i and ( s, a )[ Z i ] ∈ X
[ Z i ] for episode k
= 1 , 2 , ... do
Set ˆ V k,H +1 ( s )
=
for all s .
Estimate ˆ R k,i ( s, a ) with empirical mean value if N k − 1 (( s, a )[ Z Ri ]) > 0 , otherwise ˆ R k,i ( s, a ) = 1 , then calculate ˆ R ( s, a )
= 1 m (cid:80) mi =
ˆ R
i
( s, a )[ Z Ri ])
Let K P = (cid:8) ( s, a ) ∈ S × A , ∪ i ∈ [ n ] N k
(( s, a )[ Z Pi ]) > 0 (cid:9)
Estimate ˆ P k ( ·| s, a ) with empirical mean value for all ( s, a ) ∈ K P for horizon h = H, H − 1 , ..., 1 do for all ( s, a ) ∈ S
× A do 10: if ( s.a ) ∈ K P then ˆ Q k,h ( s, a ) =
min { H, ˆ R k
( s, a ) + CB k
( s, a )
+ ˆ P k ˆ V k,h
+
( s, a ) } else
ˆ Q k,h ( s, a ) =
H end if 15: ˆ V k,h ( s ) =
max a ∈A ˆ Q k,h ( s, a ) end for end for for step h = 1 , · · · , H do
Take action a k,h = arg max a ˆ Q k,h ( s k,h , a ) 20: end for
Update history trajectory
L = L (cid:83) { s k,h , a k,h , r k,h , s k,h +1 } h =1 , 2 ,...,H , and update history counter N k − 1 (( s, a )[ Z i ]) .
end for tored reward R i and factored transition P i in the following way:
[Equation]
We deﬁne the conﬁdence bonus as the summation of all conﬁdence bonus for rewards and transition, i [Equation] e.
[Equation]
We propose the following regret upper bound for Alg. 2.
Theorem 5.
With prob.
1 − δ , the regret of Alg.
2 is upper bounded by
Reg( K
[Equation]
Here O hides the lower order terms with respect to T .
C O MITTED D ETAILS IN S ECTION
In this section, we clarify the omitted details in Section 4.
The detailed algorithm is described in Alg.
we denote N k
(( s, a )[ Z ]) as the number of steps that the agent encounters ( s, a )
[ Z ] during the ﬁrst k episodes, and N k
(( s, a )[ Z j ] , s j ) as the number of steps that the agent transits to a state with s [ j ] =
s j after encountering ( s, a )
[ Z j ] during the ﬁrst k episodes.
In episode k , we estimate the mean value of each factored reward R i and each factored transition P i with empirical mean value ˆ R k,i and ˆ P k,i respectively.
To be more speciﬁc, ˆ R k,i (( s, a )
[ Z R i ]) =
[Equation]
, where r t,i denotes the reward R i sampled in step t
[Equation]
and
[Equation]
The formal deﬁnition of the conﬁdence bonus for Alg. 1 is:
[Equation]
(6) where φ k,i ( s, a ) = (cid:114)
[Equation]
[ Z
[Equation]
[Equation]
4 |S
[Equation]
L P 3 N k −
[Equation] [Equation]
[ Z Pj ]) .
The deﬁnition of η k,h,i ( s, a ) is (cid:115)
16 H 2 L P
N k − 1 (( s, a )[ Z Pi ])
[Equation]
Theorem 6.
(Reﬁned Statement of Theorem 2) With prob.
at least 1 − δ , the regret of Alg.
1 is upper bounded by
[Equation]
For clarity, we also present a cleaner single-term regret bound under a symmetric problem setting.
Suppose M is a set of factored MDP with m = n , |S i | =
S i , |X i
| =
S i
A i and | Z Ri | = | Z Pj | =
ζ for i = 1 , ..., m and j = 1 , ..., n , we write X i =
( S i A i ) ζ and assume that X i ≤ J and S i ≤ Γ .
Corollary 6.1.
Suppose M ∗ ∈ M , with prob.
1 − δ , the regret of FMDP-BF is upper bounded by
O (cid:16)(cid:112) nHJT log( nTSA/δ ) (cid:17) .
The minimax regret bound for non-factored MDP is O (cid:16)(cid:112) HSAT log( SAT/δ ) (cid:17) .
Compared with this result, our algorithm’s regret is exponentially smaller when n and ζ are relatively small.
Under this problem setting, the regret of Osband & Van Roy (2014b) is O (cid:16) nH (cid:112) Γ JT log( nJT ) (cid:17) .
Our results is better by a factor of √ nH Γ .
D H IGH P ROBABILITY E VENTS
In this section, we discuss the high-prob.
events, and assume that these events happen during the proof.
Algorithm
Input : δ
L = ∅ , initialize N (( s, a )[ Z i ])
=
for any factored set Z i and
any ( s, a )[ Z i ] ∈ X
[ Z i ] for episode k = 1 , 2 , · · · do
Set V k,H +1 ( s ) =
V k,H +1 ( s ) =
for all s, a .
Let K = (cid:8) ( s,
a ) ∈ S
×
A : ∩ i =1 ,...,n N k
(( s, a )[ Z Pi ]) > 0 (cid:9)
Estimate ˆ R k,i ( s, a ) as the empirical mean if N k − 1 (( s, a )[ Z Ri ]) > 0 , and 1 otherwise ˆ R
( s, a ) =
1 m (cid:80) mi =1 ˆ R
i
( s, a )[ Z Ri ])
Estimate ˆ P k ( ·| s, a ) with empirical mean value for all ( s, a ) ∈ K for horizon h = H, H − 1 , ..., 1 do 10: for s ∈ S do for a ∈ A do if ( s, a ) ∈ K then
Q k,h ( s, a ) = min { H, ˆ R k ( s, a ) + CB k ( s, a ) + ˆ P k V k,h +1 ( s, a ) } else 15:
Q k,h ( s, a ) =
H end if end for π
[Equation]
max
[Equation]
[Equation]
V
[Equation]
end for end for for step h = 1 , · · · , H do
Take action a k,h = arg max a Q k,h ( s k,h , a ) 25: end for
Update history trajectory
L = L (cid:83) { s k,h , a k,h , r k,h , s k,h +1 } h =1 , 2 ,...,H , and update history counter N k − 1 (( s, a )[ Z i ]) .
end for
Lemma D.1.
(High prob.
event)
With prob.
at least 1 − 2 δ/ 3 , the following events hold for any k, h, s, a : | ˆ
[Equation]
[Equation]
| ˆ P k,i (cid:89)
[Equation]
|
[Equation]
| (ˆ P k,
[Equation]
We deﬁne the above events as Λ 1 , and assume it happens during the proof.
Proof.
By Hoeffding’s inequality and union bounds over all i ∈ [ m ] , step k ∈
[ K ] and ( s, a ) ∈
X [ Z Ri ] , we know that Inq.
7 holds with prob.
1 − δ 9 for any i ∈ [ m ] , k ∈
[ K ] , ( s, a ) ∈ X
[ Z Pi ] .
Similarly, by Hoeffding’s inequality and union bounds over all i ∈ [ n ] , step t and ( s, a ) ∈ X ,
Inq. 8 also holds with prob.
1 − δ 9 for any i, s, a, k .
Inq.
9 is the high probability bound on the Lnorm of the Maximum Likelihood Estimate, which is proved by Weissman et al.
Inq.can be proved with the use of Bernstein inequality and union bound (See Azar et al.
(2017) for a similar derivation).
Inq.
11 and Inq.
12 can be regarded as the summation of martingale difference sequences, which can be derived with the application of Azuma’s inequality.
Finally, we take union bounds over all these inequalities, which indicates that Λ 1 holds with prob.
at least 1 − 2 δ/ 3 .
For the proof of Thm. 2, we also need to consider the following high-prob.
events.
We deﬁne the following events as Λ 2 .
During the proof of Thm. 2, we assume both Λ 1 and Λ 2 happen.
Lemma D.2.
With prob.
at least 1 − δ/ 3 , the following events hold for any k, h, s, a :
[Equation]
[Equation]
Proof.
Inq.
13 can be proved directly by empirical Bernstein inequality.
Now we mainly focus on
Inq.
By Bernstein’s inequality and union bounds over all s
[Equation]
a, k
[Equation]
h
[Equation]
we know that the following inequality holds with prob.
at least 1 − δ 9 .
[Equation]
The last inequality is due to Jensen’s inequality
[Equation]
That is,
[Equation]
where P ( s (cid:48)
[1 : i − 1]) is a shorthand of P ( s (cid:48)
[1 : i − 1] | s, a ) , and C 1 here denotes
[Equation] [Equation]
Inq.
15 follows the same proof of the failure event F N in section B.1 of Dann et al.
E P ROOF OF T HEOREME. E STIMATION E RROR D ECOMPOSITION
Lemma E.1.
The estimation error can be decomposed in the following way:
[Equation]
here V denotes any value function mapping from S to R , e.g. V ∗ h
+1 or ˆ V k,
h +1 − V ∗ h
+
[Equation]
Proof.
Inq.
16 has the same form of Lemma 32 in Li (2009) and Lemma 1 in Osband & Van Roy (2014b).
We mainly focus on Inq.
We can decompose the difference in the following way:
[Equation]
For the last term of Inq.
[Equation]
we have
[Equation]
≤
[Equation]
Where the last inequality is due to Inq.
[Equation]
For the second part of Inq.
[Equation]
we can further decompose the term as:
[Equation]
Following the same decomposition technique
[Equation]
we can prove Inq. 17 by recursively decomposing the second term over all possible n :
[Equation]
[Table]
Lemma E.2.
Under event Λ 1 , then the following Inequality holds:
[Equation]
Proof.
Inq.
20 can be proved by Lemma D.1:
[Equation]
Inq.
21 follows directly by applying Lemma D
[Equation]
1 to Lemma E.1.
[Equation]
Similarly, Inq 22 can be proved by:
[Equation]
E. O PTIMISM
Lemma E.3.
(Optimism)
Under event Λ 1 , ˆ V k,h ( s ) ≥ V ∗ h ( s ) for any k, h, s .
Proof.
We prove the Lemma by induction.
Firstly, for h = H + 1 , the inequality holds trivially since ˆ V k,H +
( s ) =
V ∗ H +1 ( s ) =
ˆ V k,h ( s ) − V ∗ h ( s ) ≥
ˆ R k
( s, π ∗ h ( s ))
+ CB k
( s, π ∗ h ( s ))
+ ˆ P k ˆ V k,h +1 ( s, π ∗ h ( s ))
− ¯ R
( s, π ∗ h ( s ))
− P V ∗ h
+1 ( s, π ∗ h ( s ))
=
ˆ R k
( s, π ∗ h ( s ))
− ¯ R
( s, π ∗ h ( s ))
+ CB k
( s, π ∗ h ( s ))
+ ˆ P k
( ˆ V k,h +1 − V ∗ h +1 )
( s, π ∗ h ( s ))
+ (ˆ P k − P )
V ∗ h
+1 ( s, π ∗ ( s ))
≥ ˆ R k
( s, π ∗ h ( s ))
− ¯ R
( s, π ∗ h ( s ))
+ CB k
( s, π ∗ h [Table]
[Table] (ˆ P k [Table]
P ) V ∗ h
[Table]
( s [Equation] π ∗ (
[Table]
The ﬁrst inequality is due to ˆ V k,h ( s ) ≥ ˆ Q k,h ( s, π ∗ h ( s )) .
The second inequality follows by induction condition
that ˆ V k,h +1 ( s ) ≥ V ∗ h
+1 ( s ) for all s .
The last inequality is due to Inq.
20 and Inq.
22 in
Lemma E.2.
E. P ROOF OF T HEOREM
Now we are ready to prove Thm.
[Equation]
Proof.
(Proof of Thm. 5)
[Equation]
The ﬁrst inequality is due to optimism ˆ V k,h ( s k,h ) ≥
V ∗ h ( s k,h ) .
The ﬁrst equality is due to Bellman equation for V π k h and ˆ V k,h .
For notation simplicity, we deﬁne
[Equation]
Firstly we focus on the upper bound of (cid:16) ˆ P k,h − P (cid:17) (cid:16) ˆ V k,h +1 − V ∗
h +1 (cid:17) (
s k,h , π k,h ( s k,h )) .
We bound this term following the idea of Azar et al.
[Equation]
+
[Equation]
ˆ P
i − P
i (cid:17) ( ·| ( s k,h , a k,h )[ Z Pi ])
[Table]
[Table] ˆ P j − P j (cid:17) ( ·| ( s k,h , a k,h )[ Z Pj [Table]
) (cid:12)(cid:12)(cid:12)≤ n (cid:88) i =   (cid:88) s (cid:48)
[ i ] ∈S [ i ] (cid:115) 2 P i ( s (cid:48)
[ i ] |X
[ Z Pi ])
L P
N k − 1 (( s, a )[ Z Pi ])
+
L P 3 N k − 1 (( s, a )[ Z Pi ]) 
 n (cid:89)
j =1 ,j (cid:54) =
i
P j (cid:16) ˆ V k,h +1 − V ∗ h +1 (cid:17) ( s k,h , a k,h ) + n (cid:88) i = n (cid:88)
j (cid:54)
=
i,j =
H (cid:32)(cid:115) 4 |S i | L P
N k − 1 (( s, a )[ Z Pi ])
+ 4 |S
i | L P 3 N k − 1 (( s, a )[ Z Pi ]) (cid:33) (cid:32)(cid:115)
|S j | L P
N k − 1 (( s, a )[ Z Pj ]) + 4 |S j | L P 3 N k − 1 (( s, a )[ Z Pj ]) (cid:33) ≤ n (cid:88) i = (cid:88) s (cid:48)
[ i ] ∈S [ i ] (cid:115) 2 P i
[Equation]
The ﬁrst inequality is due to Lemma E.1.
The second inequality is because of Lemma D.1, and the last inequality is due to the fact that (cid:12)(cid:12)(cid:12) ˆ V k,h +1 − V ∗ h +1 (cid:12)(cid:12)(cid:12) ∞ ≤ H .
For each i ∈ [ n ] , we consider those s (cid:48)
[ i ] satisfying N k − 1 (( s, a )[ Z Pi ])
P i ( s (cid:48)
[ i ] | ( s k,h , a k,h )
[ Z Pi ]) ≥ 2 n 2 H 2 L P and N k − 1 (( s, a )[ Z Pi ]) P i ( s (cid:48)
[ i ] | ( s k,h , a k,h )
[ Z Pi ]) ≤ 2 n 2 H 2 L P separately.
For those s (cid:48)
[ i ] satisfying N k − 1 (( s, a )[ Z Pi ])
P i ( s (cid:48)
[ i ] | s k,h , a k,h ) ≥ 2 n 2 H 2 L P , the ﬁrst term can be bounded by
[Equation]
=
[Equation]
where the second term can be regarded as a martingale difference sequence, and we denote it as δ 4 k,h .
For those s (cid:48)
[ i ] satisfying N k − 1 ( [Equation] s, a )[ Z Pi ]) P
i ( s (cid:48)
[ i ] | s k,h , a k,h ) ≤ 2 n 2 H 2 L P , the summation can be bounded by
[Equation]
For notation simplicity, we deﬁne δ 5 k,h as:
[Equation]
To sum up
[Equation]
by the above analysis
[Equation]
we prove that
[Equation]
Now we are ready to summarize all the terms in the regret.
Firstly, we recursively calculate the regret for all h ∈
[ H ]
[Equation]
h =
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Then we sum up the regret over k episodes,
[Equation]
δ 2 k,h and δ 4 k,h can be regarded as martingale difference sequence, the summation of which can be bounded by O ( H (cid:112) T log( T )) by Lemma E.2, while δ 1 k,h and δ 3 k,h can also be bounded by
Lemma E.2.
The summation of different terms in δ 1 k,h , δ 3 k,h , δ 4 k,h and δ 5 k,h can be separated into the following categories.
In the following proof, we use C to denote the dependence of other parameters except the counters N k (( s k,h , a k,h )
[ Z i ])
For those terms of the form
[Equation]
we have
[Equation]
The last inequality is due to Cauchy-Schwarz inequality.
This term inﬂuence the main factors in the ﬁnal regret.
For those terms of the form
[Equation]
N k
[Equation]
s k,h ,a k
[Equation]
h ) [ Z i ]) , we have
[Equation]
which has only logarithmic dependence on T .
For those terms of the form
[Equation]
we deﬁne
N k
(( s, a )[ Z i ] , ( s, a )[ Z j ]) as the number of times that agent has encountered ( s, a )
[ Z i ] and ( s, a )[ Z j ] simultaneously for the ﬁrst k episodes.
It is not hard to ﬁnd that
N k
(( s, a )[ Z i ]) ≥ N k
(( s, a )[ Z i ] , ( s, a )[ Z j ]) and N k (( s, a )[ Z j ]) ≥ N k (( s, a )[ Z i ] , ( s, a )
[ Z j ]) .
[Equation]
≤
[Equation]
which also has only logarithmic dependence on T .
For other terms with the form of
C ( N k
(( s k,h ,a k,h )
[ Z i ])) 2 and
C N k (( s k,h ,a k,h )
[ Z i ]) √ N k
(( s k,h ,a k,h )
[ Z j ]) , the summation of these terms has no dependence on T , which is negligible since T is the dominant factor.
By bounding these different kinds of terms with the above methods [Equation] we can ﬁnally show that
[Equation]
Here O hides the lower-order factors w.r.t T .
F P ROOF OF T HEOREMF. E STIMATION E RROR D ECOMPOSITION
Lemma F.1.
Under event Λ 1 and Λ 2 , we have
[Equation]
Proof.
The ﬁrst inequality follows directly by the deﬁnition
that R ( s, a ) = 1 m (cid:80) mi
=1 R i ( s, a ) and
Lemma D.2.
We now prove the second inequality.
By Lemma E.1, we have | (ˆ P k − P
[Equation]
V [Equation] ≤
[Equation]
+
[Equation]
By Inq. 9 in Lemma D.1 and Inq. 14 in Lemma D.2, we have
[Equation]
≤ n
[Equation]
i = 
[Equation]
F. O MITTED PROOF IN S ECTION 4.
Proof.
(Proof of Theorem. 1)
[Equation]
[-V h ( s h )) 
-]
[Equation]
=
[Equation]
[Equation]
Given s h = s, s h
+1 =
s (cid:48) , r h , V h
( s h ) and J h
+1:
H ( s h +1 ) are conditionally independent, thus we have
[Equation]
Therefore, we have
[Equation]
The second equality is due to the fact that V h ( s ) =
¯ R
[Equation]
s [Equation] +
[Equation]
s
[Equation]
[Equation]
For the factored rewards, since the rewards r h,i are conditionally independent give s
[Equation]
we have
[Equation]
σ
[Equation]
R,i
[Equation]
s )
For the factored transition
[Equation]
we decompose the variance in the following way:
[Equation]
 
[Equation]
 
Here ([ s (cid:48) [1] , s (cid:48)(cid:48)
[2 : n ]] denotes the vector s (cid:48)(cid:48) with s (cid:48)(cid:48)
[1] replaced with s (cid:48)
By subtracting σ 2 P,i,h ( s ) for i = 2 , ..., n in the above way, we can show that
[Equation]
Plugging Eqn.
24 back to Eqn. 23, we have
[Equation]
Proof.
(Proof of Corollary 1.1)
We can regard the MDP with given policy π as a Markov chain.
By
Theorem 1, we have
[Equation]
By recursively decomposing the variance until step H , we have:
[Equation]
Since ω 2 h ( s 1 )
=
E (cid:104) ( J h : H ( s h )
− V h ( s ))
| s h
= s (cid:105) ≤ H 2 , we can immediately reach the conclusion.
F. T HE ” GOOD ” S ET C ONSTRUCTION
The construction of the ”good” set is similar with that in Dann et al.
(2017) and Zanette & Brunskill (2019), though we modify it to handle this more complicated factored setting.
The idea is to partition each factored state-action subspace at each episode into two sets, the set of state-action pairs that have been visited sufﬁciently often (so that we can lower bound these visits by their expectations using standard concentration inequalities) and the set of ( s, a ) that were not visited often enough to cause high regret.
That is:
Deﬁnition 4.
(The Good Set)
The set L k,i for factored transition P i is deﬁned as:
[Equation]

[Equation]
The following two Lemmas follow the same idea of Lemma 6 and Lemma 7 in Zanette & Brunskill (2019).
Lemma F.2.
Under event Λ 1 and Λ 2 , if ( s, a )[ Z Pi ] ∈ L i,k , we have
[Equation]
Proof.
By Lemma D.2, we have
[Equation]
Since ( s, a )[ Z Pi ] ∈
L i,k , we have 14 (cid:80) j<k w j,Z Pi ( x ) ≥ H log(18 nX
Pi H/δ ) + H .
That is,
[Equation]
Lemma F.3.
It holds that
[Equation]
Proof.
For those ( s, a )[ Z Pi ] / ∈
L k,i , we have 14 (cid:80) j<k w j,Z Pi ( x ) ≤ H log(10 nX
Pi H/δ )
+ H ≤ 2 H log(10 nX Pi H/δ ) .
That is,
[Equation]
Lemma F.2 shows that we can lower bound the visiting count of a certain ( s, a )[ Z Pi ] if the visiting probability of ( s, a )[ Z Pi ] is sufﬁcient large.
Lemma F.3 shows that those ( s, a )[ Z Pi ] with little visiting probability cause little contribution to the ﬁnal regret.
Lemma F.4.
[Equation]
Proof.
For those ( s, a )[ Z Pi ] ∈ L k,i , we have N k
(( s, a )[ Z Pi ]) ≥ 14 (cid:80)
j<k w j,Z Pi ( s, a ) .
Therefore, we have
[Equation]
Lemma F.5.
For factored set Z Pi of transition
[Equation]
we have:
[Equation]
where X Pi,j = | X
[ Z Pi ∪ Z Pj ] | .
For factored set Z Ri of rewards, similarly we have:
[Equation]
where X R i,j = | X
[ Z R i ∪ Z R j ] | .
Proof.
We only prove the inequalities for the factored set of transition.
The inequalities for the factored set of rewards can be proved in the same manner.
For Inq.
25, we deﬁne X i (( s, a )[ Z Pi ])
= { x ∈ X | x [ Z Pi ] =
( s, a )[ Z Pi ] } , then we have
K (cid:88)
[Equation]
In the ﬁrst equality, we ﬁrstly categorize ( s, a ) based on their value ( s, a )[ Z Pi ] and sum up over all possible choice of ( s, a )[ Z Pi ] , then we sum up the value in each category in the inner summation.
The second equality is due to (cid:80) ( s 1 ,a 1 ) ∈X i (( s,a )[ Z Pi ]) w k,h ( s 1 ,a 1 ) w k,h,ZPi ( s,a ) = 1 .
The ﬁrst inequality is due to
Cauchy-Schwarz inequality.
The second inequality is due to Lemma F.4 and Lemma F.3.
The last inequality is due to the assumption that X Pi ≥ H log(10 nX Pi H/δ ) .
For Inq. 26 and Inq. 27, we deﬁne Z Pi,j = Z Pi ∪ Z Pj .
For the factored set Z Pi,j , similarly we have
K
[Equation]
i
[Equation] By the deﬁnition of Z Pi,j , we know that N k − 1 (( s, a )[ Z Pi ]) ≥
N k − 1 (( s, a )[ Z Pi,j ]) and
N k − 1 (( s, a )[ Z Pj ]) ≥ N k − 1 (( s, a )[ Z Pi,j ]) .
Therefore, we have
[Equation]
The following proof of Inq. 26 and Inq.
27 shares the same idea of the proof of Inq.
F. T ECHNICAL L EMMAS ABOUT V ARIANCE
In this subsection, we prove several technical lemmas about variance.
For notation simplicity, we use E i and E
[ i : j ] as a shorthand of E s (cid:48)
[ i ] ∼ P i ( ·| ( s,a )[ Z Pi ]) and E s (cid:48)
[ i : j ] ∼ P [ i : j ] ( ·| ( s,a )[ Z P [ i : j ] ]) .
Similarly, we use V i and V [ i : j ] as a shorthand of V s (cid:48)
[ i ] ∼ P i ( ·| ( s,a )[ Z Pi ]) and V s (cid:48)
[ i : j ] ∼ P [ i : j ] ( ·| ( s,a )[ Z P [ i : j ] ]) .
For those w.r.t the empirical transition ˆ P k , we use ˆ E k and ˆ V k to denote the corresponding expectation and variance.
For example, E s (cid:48)
[ i ] ∼ ˆ P k,i ( ·| ( s,a )[ Z Pi ]) is denoted as ˆ E k,i .
Lemma F.6.
Under event Λ 1 , Λ 2 , we have:
[Equation]
where V denotes some given function mapping from S to R .
Proof.
[Equation]
We bound Equ. 32 and 33 separately.
For equ.
[Equation]
we have
[Equation]
The last inequality is due to Lemma D.1.
For equ.
[Equation]
given ﬁxed s (cid:48)
[1 : i − 1]
[Equation]
we have
[Equation]
The ﬁrst inequality is due to the deﬁnition of variance.
The last inequality is due to E [ i : n ] V ( s (cid:48) )
+ ˆ E
[ i : n ] V ( s (cid:48) ) ≤ 2 H .
All Equ. 35, 36 and 37 can be bounded with the same manner of Equ. 32.
That is, we ﬁrst bound each term with the L 1 -distance of transition probability multiplying the L ∞ -norm of the value function, then we upper bound the L 1 -distance by Lemma D.1.
This leads to the following results:
[Equation]
This bound doesn’t depend on the given ﬁxed s (cid:48)
[1 : i − 1] .
By taking expectation over s (cid:48)
[1 : i − 1] ∼ P [1: i − 1] ( ·| s, a ) , we have
[Equation]
Combining with Equ. 34, we have
[Equation]
Lemma F.7.
Under event Λ 1 , Λ 2 and Ω , we have
[Equation]
where u k,h,i ( s, a ) is deﬁned in Section 4.3:
[Equation]
Proof.
We can decompose the difference in the following way:
[Equation]
By Lemma F.
[Equation]
we know that
[Equation]
Now we only need to bound ˆ σ 2 P,k,i ( V ∗ h
+1 , s, a ) − 2ˆ σ 2 P,k,i ( V k,
h
+1 , s, a ) .
By Lemma 2 of Azar et al.
[Equation]
we know that for two random variables X ∈ R and Y ∈ R , we have
[Equation]
That is,
[Equation]
The ﬁrst inequality is due to Inq. 38, and the second inequality is due to V X ≤ E X 2 for any random variable X ∈ R .
To sum up, we have
[Equation]
Lemma F.8.
Under event Λ 1 , Λ 2 and Ω , suppose ˜Reg( K ) =
(cid:80)
Kk =1 V k,
[Equation]
s
[Equation]
− V
[Equation]
1 ( s 1 ) , we have:
[Equation]
Proof.
We only prove the ﬁrst inequality in detail.
By replacing V ∗ h +1 with V k,h
+1 , we can prove the second inequality in the same manner.
[Equation]
[Table]
Given ﬁxed
[Table]
we bound the difference of the [Table] [Equation] [Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
The ﬁrst inequality is due to V ∗
h +
( s (cid:48) )
≥ V π k h +
( s (cid:48) ) , and the second inequality is due to
E [ i +1: n ] V ∗ h
+1 ( s (cid:48) )
+ E
[ i +1: n ] V π k h +
( s (cid:48) )
≤ 2 H .
We then take expectation over all s (cid:48)
[1 : i − 1] .
that is
[Equation]
Plugging the inequality into the former equation
[Equation]
we have
[Equation]
For the second inequality, this is because that by lemma E.15 of Dann et al.
(2017), we have (cid:88)
[Equation]
s
[Equation]
(cid:1)
This means that (cid:80) s w k,h ( s )
[ V ∗ h ( s ) − V π k h ( s )]
≤ V ∗ 1 ( s 1 ) − V π k 1 ( s 1 ) for any k, h .
F. O PTIMISM AND P ESSIMISM
Lemma F.9.
Suppose that Λ 1 , Λ 2 and Ω k,h +1 happen, then we have the following inequalities hold for any a ∈ A and s ∈ S :
[Equation]
Proof.
The ﬁrst inequality follows directly by Lemma F.1 and the deﬁnition of CB Ri ( s, a ) .
For the second inequality, by Lemma F.1, we have
[Equation]
φ
[Equation]
s
[Equation]
a
[Equation]
where φ k,i ( s, a ) = (cid:113)
[Equation] [Equation]
Pi ]) .
The ﬁrst inequality is due to V k,h ( s ) ≥
Q k,h ( s, π ∗ ( s )) .
By the deﬁnition of CB Pi ( s, a ) , we have
[Equation]
[Equation]
[Equation]
otherwise
[Equation]
For those 2ˆ σ 2 P,k,i ( V k,h +1 , s, a ) ≤ σ 2 P,i ( V ∗ h
+1 , s, a ) , by Lemma F.
[Equation]
we have
[Equation]
That is,
[Equation]
) ≥
[Table]
[Equation]
≥
[Table]
[Equation]
Combining with Eq.
41, we prove that
[Equation]
The optimism is proved by induction.
Lemma F.10.
(Optimism) Suppose that Λ 1 , Λ 2 and Ω k,h +1 happen, then we have V k,h ≥ V ∗ h .
Proof.
[Equation]
≥
The ﬁrst inequality is due to induction condition that Ω k,h +1 happens.
The last inequality is due to
Lemma F.9.
Lemma F.11.
(Pessimism) Suppose that Λ 1 , Λ 2 and Ω k,h +1 happen, then we have V k,h ≤ V ∗ h .
Proof.
[Equation]
The inequality is due to V k,
h +1 ( s (cid:48) )
≤ V ∗ h
+1 ( s (cid:48) )
since event Ω k,h +1 happens.
By lemma F. [Equation] we have
[Equation]
Therefore, we have
[Equation]
Lemma F.12.
(Optimism and pessimism)
Under event Λ 1 and Λ 2 , we have Ω k,h holds for all k and h .
Proof.
By Lemma F.10 and Lemma F.11, through induction over all possible k, h , we can prove the
Lemma.
F. P ROOF OF T HEOREM
Proof.
We decompose ˜Reg( K ) =
(cid:80)
Kk
[Equation]
[Equation]
s
[Equation]
[Equation] − V π k
[Equation]
s k 1 ,
[Equation]
k
[Equation]
1 ) (cid:1) in the classical way (Azar et al., 2017; Zanette & Brunskill, 2019; Dann et al., 2019), that is
[Equation]
[Equation] [Equation]
[Equation]
We bound Equ. 47, 48, 49 and 50 separately by Lemma F.13, Lemma F.14, Lemma F.15 and
Lemma F.16.
Combining the results of these Lemmas, we have
[Equation]
Here C 1 , C 2 , C 3 denote some constants.
Solving the ˜Reg( K ) in Inq 51, we can show that [Equation] 
[Equation]
where O hides the lower order terms w.r.t T .
By the optimism principle (Lemma F.12), we have V ∗ 1 ( s k, 1 , a k, 1 ) ≤ V k, 1 ( s k, 1 , a k, 1 ) .
This leads to the ﬁnal result:
K
[Equation]
F. B OUNDING THE M AIN T ERMS
Lemma F.13.
Under event Λ 1 , Λ 2 and Ω k,h , suppose ˜Reg( K ) =
(cid:80)
Kk =1 V k, 1 ( s 1 ) − V π k 1 ( s 1 ) , we have
[Equation]
Proof.
By the deﬁnition of CB k ( s, a ) , we have
[Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation]
By Lemma F.5, the upper bound of Eqn.
55, 56 and 57 is O
( T 14 ) , which doesn’t contribute to the main factor in the regret.
We prove the upper bound of Eqn. 54 and Eqn. 58 in detail.
By Lemma F.6, we have
[Equation]
Then Eqn. 54 can be bounded as
[Equation]
Similar with Eqn. 61, the summation of Eqn.
61 is upper bounded by O ( T 1 / 4 ) by Lemma F.5.
For
Eqn. 59, we have (cid:88)
[Equation]
(cid:115)(cid:88)
[Equation]
The ﬁrst inequality is due to ˆ σ 2 R,k,i ( s, a ) ≤ 1 .
The second inequality is due to Cauchy-Schwarz inequality.
By Lemma F.5, the summation can be bounded by 1 m (cid:80) mi =1 (cid:112)
X R i
L R i T log T .
[Table]
[Equation]
The ﬁrst inequality is due to Cauchy-Schwarz inequality.
The second inequality is due to Lemma F.5.
The third inequality is due to Lemma F.8.
The forth inequality is due to σ 2 R,k,i ( s, a ) ≥ 0 , and the last inequality is because of Corollary 1.1.
For Eqn.
58, we have
[Equation]
[Equation]
By Lemma F.20, we know that the summation (cid:80) k,h,s,a w k,h ( s, a ) u k,h,i ( s, a ) is of order
O ( T 12 ) .
This means that Equ. 62 is of order
O
( T 14 ) , which doesn’t contribute to the main term ( O ( √ T ) ).
Lemma F.14.
Under event Λ 1 , Λ 2 and Ω , suppose ˜Reg( K ) =
(cid:80)
[Equation] 1 V k, 1 ( s 1 ) − V
[Equation] [Equation]
s 1 ) , we have
[Equation]
Proof.
By Lemma F.
[Equation]
we have
[Equation]
By Lemma F.
[Equation]
the second term has only logarithmic dependence on T [Equation] which is negligible compared with the main factor.
We mainly focus on the ﬁrst term.
(cid:88) k [Equation] h n
[Equation]
i = [Equation] N
[Equation]
) ≤ (cid:118)(cid:117)(cid:117)(cid:116)(cid:88) k,h (cid:88) ( s
[Equation]
a ) ∈X
[Equation]
a ) n (cid:88) i =
[Equation]
a )
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)(cid:88) k,h [Table] ( s,a ) ∈X n (cid:88) i = w
[Equation]
,h (
[Equation]
a ) L P
N
[Equation]
(( s
[Equation] π ( s )) [Equation] Z Pi ]) ≤ (cid:118)(cid:117)(cid:117)(cid:116)(cid:88)
[Equation]
h (cid:88) (
[Equation]
w
[Equation]
a
[Equation]
(cid:32)m
[Equation]
(cid:88) [Equation]
=
[Equation]
n (cid:88)
[Equation]
=
[Equation]
σ
[Equation]
( V
[Equation]
h
[Equation]
a ) (cid:33) · (cid:118)(cid:117)(cid:117)(cid:116)(cid:88) k,h (cid:88) ( s,a ) ∈X n (cid:88)
i =
w k
[Equation]
h ( s, a ) L P N k − 1 (( s, π ( s )) [ Z Pi ]) ≤ (cid:114)(cid:16) HT + nH
[Table]
˜Reg
[Equation]
The ﬁrst inequality is due to Cauchy-Schwarz inequality.
The second inequality is due to σ 2 R,k,i ( s, a ) ≥ 0 .
For the last inequality, the ﬁrst part is the summation of the variance, which can be bounded by Lemma 1.1 and Lemma F.8, while the second part can be bounded as (cid:80)
i X Pi L P log T by Lemma F.5.
Lemma F.15.
Under event Λ 1 , Λ 2 and Ω , suppose ˜Reg( K ) =
(cid:80)
Kk =1 V k, 1 ( s 1 ) − V π k 1 ( s 1 ) , we have
[Equation]
Proof.
By Lemma E
[Equation]
[Equation]
we can prove that
[Equation]
The second inequality is due to Lemma D.1.
We only focus on the summation of the ﬁrst term, since the summation of other terms has only logarithmic dependence on T by Lemma F.5.
[Equation]
[Table]
[Equation]
The ﬁrst inequality is due to the fact that V k,h ≥ V ∗ h ≥ V k,h .
The second and the third inequality is due to Cauchy-Schwarz inequality.
The last inequality is because of Lemma F.5 and Lemma F.20.
Lemma F.16.
Under event Λ 1 , Λ 2 and Ω , we have
[Equation]
Proof.
By Lemma F.1, we have (cid:88)
[Equation]
The second inequality is due to ˆ σ R,k,i ( s, a ) ≤ 1 .
The last inequality is due to Cauchy-Schwarz inequality.
By Lemma F.5, we know that the summation is of order O (cid:16) 1 m (cid:80) mi =1 (cid:112) TX Ri L Ri [Equation] T
[Equation]
Lemma F.17.
Under event Λ 1 and Λ
[Equation]
we have
[Equation]
The expectation is over all possible trajectories in episode k given s h
= s following policy π k .
Proof.
[Equation]
The second term can be bounded as:
[Equation]
Lemma F.18.
Under event Λ 1 and Λ 2 , we have
[Equation]
Which has only logarithmic dependence on T .
Note that this bound is loose w.r.t parameters such as H ,
|S j | , X Pi and X Ri .
However, it is acceptable since we regard T as the dominant parameter.
This bound doesn’t inﬂuence the dominant factor in the ﬁnal regret.
Proof.
By the deﬁnition of CB k ( s, a ) , CB Rk,i ( s, a ) and
CB Pk,i ( s, a ) , we have
[Equation]

The second inequality is due to ˆ σ 2 R,i ( s, a ) ≤ 1 , ˆ σ 2 P,i ( V k,h +1 , s, a )
≤ H 2 and u k,h,i ( s, a ) ≤ H .
Now we are ready to bound (cid:80)
Kk
[Equation]
Hh
[Equation]
,a
[Equation]
∈X w k, [Equation] , a [Equation]
CB
[Equation]
, a
[Equation]
[Equation]
k
[Equation]
The last inequality is due to Lemma F.5.
Lemma F.19.
Under event Λ 1 and Λ 2 , we have
[Equation]
P [ i +1: n ] ( ·| s,a ) (cid:0) V k,h
+1 − V k,h +1 (cid:1) ( s (cid:48) )
(cid:17) 2 ≤ O (log T ) ,
Here O hides the dependence on other parameters such as H, X Pi , X Ri except T .
Proof.
For notation simplicity, we use E i and E
[ i : j ] as a shorthand of E s (cid:48)
[ i ] ∼ P i ( ·| ( s,a )[ Z Pi ]) and E s (cid:48)
[ i : j ] ∼ P [ i : j ] ( ·| s,a ) .
[Equation]
Deﬁne U k
[Equation]
[Equation]
By Lemma F.17, we have
K [Equation]
[-H (cid:88) h = (cid:88) s,a w k,h +
 ( s, a ) (cid:0) V k, h +1 − V k,h +1 (cid:1) 2 ( s, a ) ≤ (cid:88) k,h,s,a w k,-][Equation]
[Equation]
[-+
 =-][Equation]

[Equation]
h [Equation] (cid:88) s h 1 ,a h [Equation]
Pr( [Equation] h 1 , a h 1 | s h +
= s, a h
+1 =
a ) U
[Equation]
( s h 1 , a h 1 )  
[Equation] k, [Equation] a w
[Equation]
h [Equation] h [Equation]
 [Equation] s h 1 ,a h
[-+
 =-][Equation] [Equation]
h 1 |
[Equation]
h + [Equation] a h + [Equation]
a
[Equation]
h [Equation] a h 1 )  
[Equation]
h
[Equation] (cid:88) s h [Equation] a h
[Equation] [Equation] a h 1 )) [Equation]
H [Equation] h [Equation]
(cid:88) s h 1 ,a hw [Equation] [Equation] a
[Equation] [Equation]
Plugging the deﬁnition of U k ( s, a ) into Equ. 63, we have:
[Equation]
The last inequality is due to Lemma F.5.
We can bound (cid:80) k,h 2 nH 2 (cid:80) s h ,a h w k,h
( s h , a h ) CB 2 k ( s, a ) by
Lemma
F.18.
Summing up over all terms, we can show that
[Equation]
H h
=1 (cid:80) s,a w k,
{+h +
+}
( s, a ) (cid:0) V k,
h +1 − V k,h +1 (cid:1) 2 ( s, a ) is of order O (1) .
Lemma F.20.
Under event Λ 1 and Λ 2 , for any i ∈ [ n ] ,we have
[Equation]
Here O hides the lower order terms w.r.t.
T .
Proof.
For notation simplicity, we use E i and E
[ i : j ] as a shorthand of E s (cid:48)
[ i ] ∼ P i ( ·| ( s,a )[ Z Pi ]) and
E s (cid:48)
[ i : j ] ∼ P [ i : j ] ( ·| ( s,a )[ Z Pi ]) .
For those expectation w.r.t the empirical transition ˆ P k , we use ˆ E k to denote the corresponding expectation.
u k,h,i ( s, a ) is deﬁned as:
[Equation]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Equation]
That is
We can bound Eqn.
70 and Eqn. 71 by Lemma D
[Equation]
For Eqn.
70, we have
[Equation]
The ﬁrst inequality is due to (cid:16) ˆ E
[ i +1: n ] (cid:0) V k,h
[Table]
|S j | L P
N k − 1 (( s, a )[ Z Pj ]) 
 =2 H 2 (cid:88) k,h,s,a w k,h
( s, a ) n (cid:88)
j = i + (cid:115)
|S
j | L P
N k − 1 (( s, a )[ Z Pj ]) ≤
Hn (cid:88)
j = i + (cid:113) [Equation]
log T
Eqn. 69 can be bounded by Lemma F.19, which has only logarithmic dependence on T .
G P ROOF OF T HEOREM
Proof.
We consider the following two hard instances.
The ﬁrst instance is an extension of the hard instance in Jaksch et al.
They proposed a hard instance for non-factored weakly-communicating MDP, which indicates that the lower bound in that setting is Ω( √ DSAT ) .
When transformed to the hard instance for non-factored episodic MDP, it shows a lower bound of order Ω( √ HSAT ) in episodic setting Azar et al.
; Jin et al.
Consider a factored MDP instance with d = m = n and X [ Z Ri ] =
X [ Z Pi ] =
X i =
S i ×
A i , i = 1 , ..., n .
This factored MDP can be decomposed into n independent non-factored MDPs.
By simply setting these n non-factored MDPs to be the construction used in Jaksch et al.
(2010), the regret for each MDP is Ω
( (cid:112) H |X
[ Z Pi ] | T ) .
The total regret is Ω( (cid:80) ni
=1 (cid:112) H |X
[ Z Pi ] | T ) .
Note that in our setting, the reward R = 1 m (cid:80) mi =1 R i is [0 , 1] -bounded.
Therefore, we need to normalize the reward function in the hard instance by a factor of 1 m .
This leads to a ﬁnal lower bound of order Ω( 1 m (cid:80) n i =1 (cid:112) H |X
[ Z P i ] | T ) =
Ω( 1 n (cid:80) n i =1 (cid:112) H |X
[ Z P i ] | T ) .
Similar construction has been used to prove the lower bound for factored weakly-communicating MDP (Xu & Tewari, 2020).
The second hard instance is an extension of the hard instance for stochastic multi-armed bandits.
The lower bound of stochastic multi-armed bandits shows that the regret of a MAB problem with k 0 arms in T 0 steps is lower bounded by Ω( √ k 0 T 0 ) .
Consider a factored MDP instance with d = m = n and X [ Z Ri ] =
X [ Z Pi ] =
X i =
S i ×
A i , i = 1 , ..., n .
There are m independent reward functions, each associated with an independent deterministic transition.
For reward function i , There are log 2 ( |S i | ) levels of states, which form a binary tree of depth log 2 ( |S i | ) .
There are 2 h − 1 states in level h , and thus |S i | − 1 states in total.
Only those states in level log 2 ( |S i | ) have non-zero rewards, the number of which is |S i | 2 .
After taking actions at state s (cid:48) in level log 2 ( |S i | ) , the agent will transits back to state s (cid:48) in level log 2 ( |S i | ) .
That is to say, the agent can enter ”reward states” at least H − log 2 ( |S i | ) ≥ H 2 times in one episodes.
For each reward function i , the instance can be regarded as an MAB problem |S
i A
i |arms running for KH 2 steps 1 , thus the regret for reward i is
Ω
[Equation]
KH
[Equation]
|X
[ Z Ri ] | T ) .
In this construction, the total reward function can be regarded as the average of m independent reward functions of m stochastic MDP.
This indicates that the lower bound is
[Equation]
To sum up, the regret is lower bounded by
[Equation]
which is of the same order as
[Equation]
H O MITTED D ETAILS IN S ECTIONH. S PECIFIC INSTANCES
We further explain the difference with two speciﬁc examples in Fig.
H.1.
No matter which setting the previous work considers, the main idea of the algorithms in Efroni et al.
Brantley et al.
(2020) is to explore the MDP environment, and then ﬁnd a near-optimal policy satisfying that the expected cumulative cost less than a constant vector B 0 , i.e. E [ (cid:80) h ∈
[ H ] c h ] ≤ B 0 .
However, in our setting, the agent has to terminate the interaction once the total costs in this episode exceed budget B .
Because of this difference, their algorithm will converge to an sub-optimal policy with unbounded regret in our setting.
In the ﬁrst MDP instance (Fig.
H.1), the agent starts from state s 0 .
After taking action a 1 , it will transit to s 1 with a deterministic cost c 1 = 0 .
After taking action a 2 , it will transit to s 2 .
The cost of taking a 2 is 0 with prob.
5 , and 1 with prob 0 .
There are no rewards in state s 0 .
In state s 1 and s 2 , the agent will not suffer any costs.
The deterministic rewards are 0 .
5 and 0 .
8 respectively.
s 3 and s 4 are termination states.
The budget B 0 is 0 .
For this MDP instance, the optimal policy is to take action a 1 in s 0 , since the agent can receive total rewards 0 .by taking a 1 .
If taking action a 2 , the agent will terminate at state s 2 with no rewards with prob.
5 , which leads to an expected total rewards of 0 . 4 .
However, if we run the algorithm in Efroni et al.
Brantley et al.
(2020), the algorithm will converge to the policy that always selects action ain s 0 , since the expected cumulative cost of taking a 2 is 0 .
5 ≤ B 0 .
We further show that the policies deﬁned in the previous literature are not expressive enough in our setting.
In the second instance, the agent starts in state s 0 with one action a 0 .
By taking a 0 , the agent transits to s 1 with no rewards.
The cost of taking a 0 is 0 with prob.
5 , and 1 with prob 0 .
In s 1 , the agent needs to decide to take a 1 or a 2 , with deterministic costs of 0 and 0 .
5 respectively.
After taking a 1 , the agent will transits to s 2 , in which it can obtain a reward r 3 = 0 .
While by taking a 2 , the agent can transits to s 3 , and obtain a reward r 4 = 1 .
The budget B = 0 .
In this instance, the action taken in s 1 depends on the remaining budget of the agent.
That is to say, the policy is not expressive enough if it is deﬁned as a mapping from state to action .
Instead, we need to deﬁne it as a mapping from both state and remaining budget to action .
However, previous literature only considers policies on the state space, which cannot deal with this problem.
H. A LGORITHM AND R EGRET
We denote V πh ( s, b ) as the value function in state s at horizon h following policy π , and the agent’s remaining budget is b .
For notation simplicity, we deﬁne P S P C V ( s, a ) =
[Equation]
s
[Equation]
P [Equation] s
[Equation]
| s, a ) P ( C ( s, a ) =
c 0 | s, a ) V ( s (cid:48) , b − c 0 ) .
We use P C,i ( c 0 | s, a ) to denote the ”transition probability” of budget i , i.e. P ( C i ( s, a ) =
c 0 | s, a ) .
The Bellman Equation of our setting is written as:
[Equation]
Suppose N k
( s, a ) denotes the number of times ( s, a ) has been encountered in the ﬁrst k episodes.
We estimate the mean value of r ( s, a ) , the transition matrix P S and P C in the following way: [Equation] s
[Equation]
Following the deﬁnition in the factored MDP setting [Equation] we deﬁne the conﬁdence bonus for rewards and transition respectively (for 0 ≤ i ≤ d ):
[Equation]
where L = log(2 dSAT )
+ d log( mB ) is the logarithmic factors because of union bounds.
The additional d log( mB ) is because that we need to take union bounds over all possible budget b .
This difference compared with factored MDP is mainly due to the noised offset model.
CB Rk ( s, a ) is the conﬁdence bonus for rewards, and ˆ σ R ( s, a ) denotes the empirical variance of reward
R ( s, a ) , which is deﬁned as:
[Equation]
CB Pk, 0 ( s, a ) is the conﬁdence bonus for state transition estimation ˆ P S , and { CB Pk,i ( s, a ) } i =1 ,...,d is the conﬁdence bonus for budget transition estimation { ˆ P C,i } i =1 ,...,d .
ˆ σ 2 P,i ( V k,
[Equation]
[Equation]
s
[Equation]
) is the empirical variance of corresponding transition:
[Equation]
where ˆ P C,
[Equation] [Equation]
= (cid:81)
d 2 i
[Equation]
d 1 ˆ P C,
[Equation]
,i
[Equation]
(cid:113)
u k,h,i ( s,a,
[Equation]
N k
[Equation]
1 ( s
[Equation]
is added to compensate the error due to the difference between V ∗ h
+
and V k,h +1 , where u k,h,i ( s, a ) is deﬁned as:
[Equation]
We calculate the optimistic value function and ﬁnd the optimal policy π via the following value iteration in our algorithm:
[Equation]
Input : δ
Initialize N ( s, a ) = 0 for any ( s, a ) ∈ X for episode k = 1 , 2 , · · · do
Set V k,H +1 ( s, b ) =
V k,H +
( s, b ) =
for all s, a, b .
Let K = { ( s, a ) ∈ S
× A : N k
( s, a ) > 0 } for horizon h = H, H − 1 , ..., 1 do for s ∈ S and all possible budget b from 0 to B do for a ∈ A do if ( s, a ) ∈ K then 10:
Q k,h ( s, a, b ) = min { H, ˆ R k ( s, a ) + CB k ( s, a ) + ˆ P S,k ˆ P C,k V k,h +1 ( s, a, b ) } else
Q k,h ( s, a, b ) =
H end if end for 15: π k,h ( s, b ) =
arg max a Q k,h ( s, a, b )
V k,h ( s, b ) = max a ∈A Q k,h ( s, a, b )
V k,h ( s, b ) = max (cid:110) 0 ,
ˆ R k
( s, π k,h )
− CB k
( s, π k,h , , b )
+ ˆ P k V k,
h +1 ( s, π k,h , b ) (cid:111) end for end for 20: for step h
= 1 , · · · , H do
Take action a k,h = arg max a ˆ Q k,h ( s k,h , a ) end for
Update history trajectory
L = L (cid:83) {
s i , a i , r i , s i +1 } i =1 , 2 ,...,t k end for
Proof.
(Theorem 4)
The proof follows almost the same proof framework of Thm. 2.
The term log( SAT ) + d log( Bm ) is due to a union bound over all possible ( T, s, a ) and budget b .
This difference is because of the additional union bounds over all budget b .
H. D ISCUSSIONS ABOUT A SSUMPTION 1 AND A SSUMPTION
Assumptions 1 and 2 limit our Algorithm 4 to problems with discrete costs.
One may wonder whether it is possible to construct (cid:15) -net for budget B and possible value of the costs when these assumptions don’t hold.
In that case, we only need to estimate the discrete cost distributions on the (cid:15) -net, and then apply Algorithm 4 to tackle the problem.
Unfortunately, we ﬁnd that the (cid:15) -net construction doesn’t work for continuous cost distributions, and it is unlikely to achieve efﬁcient regret guarantee without further assumptions and the modiﬁcation of the basic setting.
If the cost distributions are continuous and the remaining budget can take any value in R , a small perturbation on the remaining budget may totally change the policy in the following steps and the optimal value.
To be more speciﬁc, suppose the agent enters a certain state with remaining budget b .
There are three actions to choose, with the cost of b − (cid:15) , b and b + (cid:15) , respectively.
After suffering the cost, the agent can achieve reward of 0 , 0 .
5 and
respectively.
Note that we can construct such hard instances with (cid:15) extremely small.
For these hard instances, we need to carefully estimate the value function of any remaining budget b ∈ R and the density functions of the costs, after which we can calculate the value through Bellman backup and ﬁnd the optimal policy.
However, estimating the density functions requires inﬁnite number of samples and makes the problem intractable.
In other words, the “non-smoothness” of the value and the policy w.r.t the remaining budget makes the problem difﬁcult for continuous value distribution without further assumptions.
This “non-smoothness” phenomenon also happens in the classical knapsack problem.
There are two possible ways to remove these assumptions and apply our algorithm to continuous cost distributions with (cid:15) -net technique.
The ﬁrst idea is to allow the slight violation of the total budget constraints, with the maximum violation threshold δ , or we assume that the value of any initial state is lipschiz w.r.t the total budget B in a small neighborhood of B (With maximum L ∞ distance δ ).
In that case, we can tolerate the estimation error of each cost function to be at most δH .
(cid:15) -net technique with (cid:15) = δH still works in this case and we can estimate the cost distribution with precision δH .
This modiﬁcation is somewhat reasonable since the agent’s policies are always “smooth” w.r.t the total budget in a small region near B in many real applications such as games and robotics.
The second idea is to consider soft constraints.
That is, when the budget constraints are violated, the agent will suffer a loss that is linear w.r.t the violation of the constraints.
We assume the linear coefﬁcient is relatively large compared with other parameters.
This is also a possible method to remove the non-smoothness w.r.t the total budget, which has wide applications in constrained optimization.