Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning
Anonymous Author(s)
Abstract
In this paper, we study the episodic reinforcement learning (RL) problem modeled by finite-horizon Markov Decision Processes (MDPs) with constraint on thenumber of batches.
The multi-batch reinforcement learning framework, wherethe agent is required to provide a time schedule to update policy before everything, which is particularly suitable for the scenarios where the agent suffersextensively from changing the policy adaptively.
Given a finite-horizon MDPwith S states, A actions and planning horizon H , we achieve near-optimalregret of ˜
O p a
SAH 3 K ln p 1 { δ qq 1 in K episodes using O p H ` log 2 log 2 p K qqbatches with confidence parameter δ .
To our best of knowledge, it is the first˜ O p poly p S, A, H q?
K q regret bound with O p H ` log 2 log 2 p K qq batch complexity.
Meanwhile, we show that to achieve ˜ O p poly p S, A, H q?
K q regret, the numberof batches is at least Ω p H { log A p K q ` log 2 log 2 p K qq , which matches our upperbound up to logarithmic terms.
Our technical contribution are two-fold: 1) a near-optimal design scheme to exploreover the unlearned states; 2) an computational efficient algorithm to explore certaindirections with an approximated transition model.
Introduction
In reinforcement learning (RL), the learning agent interacts with the environment to maximize thetotal reward by making sequential decisions.
The agent typically has to achieve two seemingly verydifferent goals: to try as many actions and reach as many states as possible so as to learn moreinformation about the environment (a.k.a. exploration ) and to follow the policy that collects thehigh rewards according to the learned information (a.k.a. exploitation ).
To address this explorationexploitation dilemma and achieve the near-optimal regret bounds, the agent usually needs to adjusthis/her strategies adaptively based on the historical trajectories and make frequent policy changes[Azar
et al., 2017, Zanette and Brunskill, 2019, Zhang et al., 2020].
On the other hand, however, too much adaptivity requirement usually leads to lower level of parallelism, impeding the large-scale deployment of the RL algorithms (which is often in a distributedmanner).
Frequent policy updates also suffer the cost of re-deploying policies in many practicalapplications.
For example, in medical domains, it often requires complete discussion among manyexperts to change the treatment plans, which is not affordable in terms of both time and monetarycost [Lei et al., 2012, Almirall et al., 2012, 2014]; in RL for hardware placement [Mirhoseini et al.
,2017], rewriting the program into the hardware for too many times is strongly discouraged.
Similarchallenges also arise in applying RL to personalized recommendation system [Yu et al., 2019] anddatabase optimization [Krishnan et al., 2018].
In such cases, the learning agent should minimize the number of policy switches while keeping theregret affordable.
Bai et al.
[2019] first proposed the provably efficient RL algorithms with lowswitching costs under the Q -learning algorithmic framework together with the lazy update techniques.
However, their method needs to actively monitor the data in real time to determine whether a policychange is to be initiated.
In other words, although the number of policy switches by [Bai et al., 2019]is low, the (usually long) time periods when the same policy is used still cannot be parallelized due tothe policy-change trigger in their algorithms which is intrinsically sequential.
In order to address this problem, we propose and study under the framework of multi-batch RL ,where the learning agent has to determine the number of batches and length of each batch before thelearning process starts, 2 and uses as few batches as possible to achieve a low regret.
Multi-batch
RL algorithms can be easily deployed in a distributed fashion as the episodes during the same batchcan be easily and fully parallelized.
The idea of batch learning is also being widely practiced.
Forexample, in medical trials, the medical center usually collects the data during a fixed time periodamong a batch of patients and then designs the experiment for the next phase based on the learnedinformation in previous phases [Lei et al., 2012, Almirall et al., 2012, 2014].
Formally, we define multi-batch RL and batch complexity as below.
Definition 1 (Multi-Batch RL with complexity M ) .
The agent determines a group of lengthst t m u M m “ 1 such that ř
Mm
“ 1 t m “ K before the learning process starts.
For m “ 1 , 2 , . . .
, M , theagent sets a policy π m
and then follows π m for t m episodes.
We highlight that an upper bound for batch complexity implies the same upper bound for globalswitching cost, since each policy switch means a new batch.
It is also worth noting that the proposedbatch RL framework is fully parallelizable during each batch for the applications where dataset comesin batch (e.g., clinical trial).
Like other RL settings, we have the natural and interesting question:
Question 1.
Is it possible to achieve near optimal batch complexity, while keeping the regret˜ O p poly p S, A, H q?
K q .
We provide a positive answer for Question 1, which we state as below.
Theorem 1.
Let 3 ι “ ln p 2 { δ q .
For any episodic MDP, with probability 1 ´ δ , under Algorithm 1 the regret in T episodes is bounded by
[Equation]
and the batch complexity is bounded by O p H ` log 2 log 2 p K qq .
On the other hand, we show a lower bound of batch complexity as below.
Theorem 2.
For any algorithm with O p poly p S, A, H q?
K q regret bound, the batch complexity is atleast Ω p H { log A p K q ` log 2 log 2 p K qq .
Compared to the lower bound of Ω p log 2 log 2 p K qq in [Gao et al., 2019] for multi-armed banditproblem, additional Ω p H { log A p K qq batches are required to explore the structure of the MDP.
Theproof of Theorem 2 is deferred to Appendix B due to space limitation.
Our contribution.
We propose the framework of multi-batch RL, and first achieve O p H `log 2 log 2 p K qq sample complexity bound with the near-optimal ˜ O p?
SAH 3 Kι q regret bound.
Wealso prove that for any algorithm with O p poly p S, A, H q?
K q regret, the global switching costis at least Ω p H { log A p K q ` log 2 log 2 p K qq , which implies a nearly matching lower bound ofΩ p H { log A p K q` log 2 log 2 p K qq for the batch complexity.
We also note that the O p H ` log 2 log 2 p K qqbatch complexity implies an O p H ` log 2 log 2 p K qq bound for the global switching cost, which isalso a near optimal upper bound.
Related Works
Bandit algorithms with limited adaptivity.
Bandit problem with low switching cost is widely studied in past decades [Cesa-Bianchi et al., 2013, Perchet et al., 2016, Gao et al., 2019, Simchi-
Levi and Xu, 2019].
Cesa-Bianchi et al.
[2013] showed an ˜Θ p K 23 q regret bound under adaptiveadversaries and bounded memories.
Perchet et al.
[2016] proved a regret bound of ˜Θ p K 1 1 ´ 21 ´
M q forthe two-armed bandit problem within M batches, and later Gao et al.
[2019] extended their result tothe general A -armed case.
Besides the setting of classical multi-armed bandit problem, other settingshas also been studied, e.g., multinomial bandit problem [Dong et al., 2020] and linear bandit problem[Ruan et al., 2020].
Episodic reinforcement learning with low switching cost.
For model-based algorithms, bydoubling updates, the global switching cost is O p SAH log 2 p K qq while keeping the regret˜ O p? SAKH 3 q Azar et al. For model-free algorithms, Bai et al.
[2019] first studied RL withlow switching cost.
They proposed a Q -learning algorithm with lazy update to achieve ˜ O p?
SAKH 4 qregret bound and O p SAH 3 log p K { A qq local switching cost.
Recently Zhang et al.
[2020] establisheda better regret bound of ˜ O p?
SAKH 3 q
and O p SAH 2 log p K { A qq local switching cost.
Besides, Gaoet al.
[2021] generalized the problem to Linear RL, and established a regret bound of ˜ O p?
d 3 H 4 K qwith O p dH log p K qq global switching cost.
Regret minimization for reinforcement learning.
There is a long line of works devoting to regretminimization for RL problem [Kakade, 2003, Jaksch et al., 2010, Bartlett and Tewari, 2009, Dannet al., 2019, Azar et al., 2017, Jin et al., 2018, Zanette and Brunskill, 2019, Zhang and Ji, 2019, Zhanget al., 2020].
For tabular setting, near optimal regret bound of ˜ O p?
SAH 3 T q has been establishedby [Azar et al., 2017, Zanette and Brunskill, 2019, Zhang et al., 2020] for both model-based andmodel-free algorithms.
However, fewer algorithms focused on the setting of batch RL.
Preliminaries
Episodic reinforcement learning.
M “ x S , A , r, P, s 1 y , where
S ˆ
A is the discretestate-action space, r “ t r h p s, a qu p s,a qP S ˆ A ,h Pr H s is the known 4 reward function, P “t P h p s, a qu p s,a qP S ˆ A ,h Pr H s is the unknown transition model and s 1 is the fixed initial state 5 .
Weassume that the reward function r h p s, a q P r 0 , 1 s for any p h, s, a q .
In each episode, the agent startsat s 1 , then takes actions and transits to the next state step by step, and finally conducts the trajectory tp s h , a h , s h ` 1 qu Hh “ 1 .
The target of the agent is to maximize the accumulative reward function {+
+} ř
Hh “ 1 r h p s h , a h q .
A policy π can be viewed as a series of mappings t π h u Hh “ 1 where π h : S Ñ ∆
A maps s h to adistribution over the action space at the h -th step, where π h p a | s q is the probability taking action a atstate s of the h -th horizon.
Given a policy π , the (optimal) Q -function and value function are given by
[Equation]
Let π p k q denote the policy in the k -th episode.
Then the regret is given by
[Equation]
Notations In this paper, we use E π,p r¨s ( P π,p r¨s ) to denote the expectation (probability) followingpolicy π under transition model p .
In particular, E π r¨s ( P π r¨s ) denotes the expectation (probability) following π under the true transition model P .
We define the general value function
W π p r 1 , p q “ E π,p « H ÿ h “r 1 h p s h , a
[Equation]
q ff .
We use 1 to denote the S -dimensional vector r 1 , 1 , . .
, 1 s J and 1 h,s,a to denote the reward function r 1 such that r 1 h 1 p s 1 , a 1 q “ I rp h, s, a q “ p h 1 , s 1 , a 1 qs .
We also define t d πh p s, a qu p s,a,h q be theoccupancy distribution of π .
That is, d πh p s, a q “ E π r
I
rp s h , a h q “ p s, a qss .
∆ d is used to denotethe d -dimensional simplex.
For two vector x, y with the same dimension, we write x J y as xy forconvenience.
For p P ∆ S and v P R S , we define V p p, v q “ pv 2 ´ p pv q 2 .
For N ě 1 , we use r N s todenote the set r 1 , 2 , . . .
, N s .
Technique Overview
Following the methods in multi-batch bandit learning Perchet et al.
[2016], Gao et al.
[2019], weconstruct our main algorithm using policy elimination.
Like most model-based reinforcement learningmethods, we maintain a confidence region P for the transition model, where the true transition model
P P P with high probability.
Before each batch starts, for a policy π and a reward function u , byextended value iteration (See Algorithm 5 in Appendix C.2), we are able to compute the confidenceinterval r L π p u, P q , U π p u, P qs for the value function of π , where
[Equation]
Here z is a virtual state for the infrequent state-action-state triples (See Function clip in Algorithm 2).
The reason why we give reward 1 for z in computing the upper confidence bound is to encourageexploration to these infrequent state-action-state triples.
By policy elimination we get Π p r, P q “
[Equation]
as the set of survivedpolicies.
The next step is to choose a policy π P Π p r, P q and execute π in the current batch.
Defining P m to be the confidence region for the transition model after the m -th batch and gap m ` 1 “max π P Π p r, P m q p U π p r, P m q ´ L π p r, P m qq , the regret in the m ` 1 -th batch could be bounded byt m ` 1 gap m ` 1 .
Following this idea, our main task is to design efficient exploration policy to reducegap m for each 1 ď m ď M .
Near-optimal design scheme Unlike RL algorithm with limited switching cost, in multi-batchreinforcement learning, the agent can not change the policy adaptively.
As a result, we need to designa policy with proper coverage ratio for all the survived policies.
That is, using the data collectedfollowing this policy, the length of the confidence interval for any survived policy is bounded by auniform threshold.
Using classical regret analysis for tabular RL [Azar et al., 2013, Zanette and Brunskill, 2019], for afixed policy π , the length of confidence interval for π could be roughly bounded by [Equation]
Cauchy 1 sineq
[Equation]
where Var h p s, a q is the variance term with respect to P h,s,a and V ˚ h ` 1 p¨q , and N h p s, a q ě 1 is thecount of p h, s, a q .
Because ř s,a,h d πh p s, a q Var h p s, a q could be uniformly bounded by O p H 2 q using classical analysis,we focus on bounding the term ř s,a,h d πh p s,a q N h p s,a q .
Suppose the policy for current batch is ˜ π .
After thisbatch, we roughly have that N h p s, a q „ d ˜ πh p s, a q .
So it corresponds to find a policy ˜
π P Π p r, P q tominimize max π P Π p r, P q ř h,s,a d πh p s,a q d ˜ πh p s,a q .
For this problem, we have the lemma below.
Lemma 1.
Let d ą 0 be an integer.
Let X Ă p ∆ d q m .
Then there exists a distribution D over X ,such that
[Equation]
where y “ t y i u dmi “ 1 “ E x „ D r x s .
Moreover, if X has a boundary set B X with finite cardinality, wecan find an approximation solution for D in poly p|B X |q time.
Plugging X “ ␣ t d πh p¨ , ¨qu Hh “ 1 | π P Π p r, P q ( , d “ SA and m “ H into Lemma 1, there exists apolicy ˜ π being a mixture of policies in Π p r, P q , such that max π P Π p r, P q ř s,a,h d πh p s,a q d ˜ πh p s,a q “ SAH .
Given the design scheme above, it remains two problems: 1) t d πh p¨ , ¨qu Hh “ 1 is unknown tous; 2) even assuming t d πh p¨ , ¨qu Hh “ 1 is known, it is hard to find ˜ π since the cardinality of
B ␣ t d πh p¨ , ¨qu Hh “ 1 | π P Π p r, P q ( might be exponential in SH .
Constructing tight confidence region For the first problem, we consider to construct a tightconfidence region for the transition model to estimate the occupancy distribution up to a constantratio.
Definition 2.
We say a confidence transition region P “ b h,s,a P h,s,a is tight with respect to p 1 iff(i) p 1 P P ; (ii) e ´ 1 H p 1 h,s,a,s 1 ď p h,s,a,s 1 ď e 1 H p 1 h,s,a,s 1 for any p h, s, a, s 1 q and any p h,s,a P P h,s,a ;(iii) P h,s,a has the form P h,s,a “ t p P ∆ S | a J i p ď
b i , i “ 1 , 2 , ..., m u where m ď poly p SK q .
In model-based reinforcement learning, these conditions are natural and it is easy to construct a tightconfidence region with acceptable error.
Once we have a confidence region which is tight w.r.t.
the true transition model P , for any policy πand p h, s, a q , we can estimate the expected visit count W π p 1 h,s,a q by W π p 1 h,s,a , p q for any p P Pbecause
[Equation]
With W π p 1 h,s,a , p q as approximation of d πh p s, a q , we can continue the analysis above by paying aconstant factor.
To learn such a confidence region, by Bennet’s inequality (Lemma 8), it suffices to visit p h, s, a, s 1 q 6for C
1 H 2 ι for each p h, s, a, s 1 q , where C 1 is an universal constant.
By this idea, we try to visit eachp h, s, a, s 1 q as much as possible.
In the meantime, it is very possible that some p h, s, a, s 1 q tuples areextremely hard to visit.
Fortunately, with proper exploration scheme, we can show that the maximalprobability to visit such tuples is well-bounded, so that these tuples could be ignored by sufferingregret O p?
T q .
Computational efficient design scheme Assume the confidence region P is tight w.r.t.
P .
For thesecond problem, we invoke reward-zero exploration to learn a sub-optimal solution for the problemmin ˜
[Equation]
, P q max
[Equation]
ř h,s,a d πh p s,a q d ˜ πh p s,a q .
Let p P P be fixed and define ˜ d πh p s, a q “ W π p 1 h,s,a , p qbe the approximation for d πh p s, a q .
We define ˜ π
i “ arg max π P Π p r, P
q W π p r i , p q for 1 ď i ď k “ K 3 ,where r ih
[Equation]
min
[Equation]
Note that r i is non-increasing in i .
Let ˜ π be the mixtureof t ˜
π i u ki “ 1 .
For any policy π , we have that
[Equation]
[Equation]
With (4) in hand, max π P Π p r, P q ř h,s,a d πh p s,a q d ˜ πh p s,a q is roughly bounded by O p SAH log p K qq 7 .
Computational efficient constrained exploration Let u, u 1 be two reward functions and P bea set of transition models.
When Π p u, P q is not trivial, it might be hard to find ˜ π P Π p u, P q suchthat ˜ π “ arg max π P Π p u, P
q W π p u 1 , p q for some p P P .
As a trade-off, we turn to find some policy˜ π P Π p u, P q such that W ˜ π p u 1 , p q ě c max π P Π p u, P
q W π p u 1 , p q , where c ą 0 is some universalconstant. The problem turns out to be a RL problem with a soft constraint. For general P , theproblem might be hard to solve. Fortunately, on the benefit of the tight property of P , we can findsuch ˜ π efficiently.
Algorithms
In this section we present our algorithms.
The main algorithm (Algorithm 1) consists of threestages.
In the first two stages, we conduct naive exploration to learn the tuples which are hard tovisit.
After ignoring such tuples, we obtain a tight confidence region.
Given the tight confidenceregion, we conduct policy elimination in the third stage.
The first and second stages contains O p H q batches, and the third stage contains O p log 2 log 2 p K qq batches. So the batch complexity of Algorithm 1 is O p H ` log 2 log 2 p K qq .
Below we describe Raw Exploration (Algorithm 2) and
Policy Elimination (Algorithm 3) in detail. 5.1 Raw Exploration Given a dataset D with counts t N h p s, a, s 1 qu , we define the set of known tuples as tp h, s, a, s 1 q : N h p s, a, s 1 q ě C 1 H 2 [-Ð 288 S 
-]ι u and the left tuples are regarded as infrequent tuples.
In Algorithm 2, we are given a dataset. Then we compute the corresponding confidence region P in Line 20, where α p n, n 1 q “ [Equation]
n [Equation] We conduct exploration layer by layer over policies in Π p r, P q . By visiting each p h, s, a q as much aspossible, we can judge whether a tuple p h, s, a, s 1 q is hard to visit using policies in Π p r, P q .
Given the set of known tuples W , we redirect all tuples not in W to an additional absorbed state zusing clip p¨ , ¨q . Once we prove that the probability of reaching z is small enough for the any optimalpolicy, we can directly learn under the clipped transition model. In Line 6 Algorithm 2, the function Policy Search is invoked. Given any reward u, u 1 , anyconfidence region P and threshold ϵ ą 0 , this function returns a policy ˜ π P Π p u, P q such that
W ˜ π p u 1 , p q ě c max π P Π p r, P q W π p u 1 , p q ´ ϵ with some universal constant c ą 0 . Moreover, when
P is tight w.r.t. the true transition model P after clipping, the time complexity of the algorithm is O p poly p SAHK q log p 1 { ϵ qq . The algorithm and corresponding analysis is postponed to Appendix C.
It is also worth noting that executing each π h,s,a with probability 1 SA can not be regarded as a(history-independent) policy because the agent need to keep in mind which policy is chosen in currentepisode. In contrast, the agent only needs to observe current state to take actions following a policy. To address this problem, we define an operator Sum to take sum over policies under some transitionmodel. Formally, we have the lemma below.
1: Input : reward function u , dataset D , length k ; 2: Initialize: C 1 Ð 200 ; 3: for h “ 1 , 2 , . . .
, H do 4:
P Ð CR p D q ; 5: for p s, a q P S
ˆ A
do 6: π h,s,a Ð Policy Search p 1 h,s,a , Π p u, P qq ; 7: end for 8: p Ð arbitrary element in P ; 9: t ˜ π h , p u Ð Sum ´␣
SA , π h,s,a , p ( p h,s,a q ¯ ; 10: π h be the policy which is the same as ˜ π h in the first h ´ 1 steps, and be the uniformly random policy in the left H ´ h ` 1 steps; 11:
Execute π h for k episodes, and collect the samples as D h ; 12:
D Ð D Y D h ; 13: end for 14: return: t D u ; 15: Function : CR p D q : 16:
N h p s, a, s 1 q Ð count of p h, s, a, s 1 q in D , for all p s, a, s 1 q ; 17:
N h p s, a q Ð max t ř s 1 N h p
s, a, s 1 q , 1 u for all p s, a q ; 18: ˆ p h,s,a,s 1 Ð N h p s,a,s 1 q N h p
s,a q , @p h, s, a, s 1 q ; 19:
W Ð tp h, s, a, s 1 q : N h p s, a, s 1 q ě C 1 H 2 ι u ; 20: ˜ P h,s,a Ð ␣ p P ∆ S | | p s
´ ˆ p h,s,a,s 1 | ď α p N h p s, a q , N h p s, a, s 1 qq , @
s 1 P S ( , @p h, s, a q ; 21:
P h,s,a Ð t clip p p, W q : p P ˜
P
h,s,a u , @p h, s, a q ; 22:
Return : b h,s,a P h,s,a .
23: Function : clip p p, W q 24: p 1 h,s,a,s 1 Ð p h,s,a,s 1 , @p h, s, a, s q P W ; 25: p 1 h,s,a,s 1 Ð 0 , @p h, s, a, s 1 q R W ; 26: p 1 h,s,a, [Equation] Ð
[Equation]
s 1 : p h,s,a,s 1 qR W p h,s,a,s 1 , @p h, s, a q P [Equation] s
[Equation] A ; 27: p 1 h,z,a Ð 1 z , @p h, a q P r H s ˆ A ; 28:
Return : p .
Lemma 2. Let P “ b p h,s,a q P h,s,a be a set of transition models such that P h,s,a Ă ∆ S is convex forany p h, s, a q . Let tp π i , P i qu ni “ 1 be a sequence of policy-transition pairs such that P i P P . For anyt
λ i u ni “ 1 such that {+λ i ě 0 for i ě 
 and ř i λ i+} “ 1 , there exists a policy π and P P P , satisfying that
[Equation] for any p h, s, a q P r H s ˆ S ˆ A . Furthermore, t π, P u could be computed in polynomial time.
Therefore, for any t λ i , π i , P i u ni “ 1 satisfying ř ni “ 1 λ i “ 1 and λ i ě 0 for i ě 1 as input, thereexists t π, P u such that W π p 1 h,s,a , P q “ ř i λ i W π i p 1 h,s,a , P i q and P h,s,a P Convex pt P ih,s,a u ni “ 1 qfor any p h, s, a q P r H s ˆ S ˆ A , where Convex p U q denotes the convex hull of the set U . Then Sumis defined as Sum pt λ i , π i , P i u ni “ 1 q “ t π, P u .
5.2 Policy Elimination
Given the dataset collected in the first two stages, we first compute the known set W .
Unlike
Algorithm 2, we do not update W in the rest time because the first two stages can ensure that theprobability of visiting W C is O p 1 { ? K q .
As mentioned in Section 4, for each batch, we invoke reward-zero exploration to search for the policywith near-optimal coverage.
Based on such a policy, we can provide uniform bound for the lengthof confidence intervals for all survived policies, which enables us to using the batch sizes in banditalgorithms [Perchet et al., 2016, Gao et al., 2019]. Besides, to obtain a better regret bound, we
1: Input: dataset D , length k ; 2: Initialize: D 0 Ð D , P ´ 1 Ð p ∆ S q SA C 1 Ð 100 , v ´ 1 h p s q Ð H
´ h ` 1 , @p h, s q P r H s ˆ S ;
K m Ð
Q
K 1 ´ 12 m
U for m “ 1 , 2 , . . .
, M “ r log 2 log 2 p K q s ; 3: N h p s, a, s 1 q Ð count of p h, s, a, s 1 q in D ; 4: W Ð tp h, s, a, s 1 q : N h p s, a, s 1 q ě C 1 H 2 ι u ; 5: for m “ 0 , 1 , 2 , . . .
, M ´ 1 do 6:
P m Ð P m ´ 1 X CR ˚ ´
D m , D m , W , t v m ´
1 h p s qu p h,s q ¯ ; 7: π m ` 1 Ð Design pp P m q ; 8: if ř mm 1 “ 1 K m 1 ď k then 9:
Execute π m ` 1 for K m ` 1 episodes; 10: else 11:
Execute π m ` 1 for k ´ p
ř mm 1 “ 1 K m 1 q episodes; 12: end if 13:
D m ` 1 Ð the dataset in the p m ` 1 q -th batch; 14:
Update the dataset D m ` 1 Ð D m Y D m ` 1 ; 15: v mh p s q Ð max π,p P P m E π,p ”ř
Hh 1 “ h r h p s h , a h q| s h “ s ı for all p h, s q P r H s ˆ S ; 16: end for 17: Function : CR ˚ p D , D 1 , W , v q : 18: t N h p s, a, s 1 qu Ð counts in D , N h p s, a q Ð max t ř s 1 N h p
s, a, s 1 q , 1 u for all p h, s, a, s 1 q ; 19: ˆ p h,s,a,s 1 Ð N h p s,a,s 1 q N h p
s,a q , @p h, s, a, s 1 q ; 20: t ˇ N h p s, a, s 1 qu Ð counts in D 1 , ˇ N h p s, a q Ð max t ř s 1 ˇ N h p s, a, s 1 q , 1 u for all p h, s, a, s 1 q ; 21: ˇ p h,s,a,s 1 Ð ˇ N h p s,a,s 1 q ˇ N h p s,a q , @p h, s, a, s 1 q ; 22: ˜ P h,s,a Ð !
p P ∆ S | | p s
´ ˆ p h,s,a,s 1 | ď α p N h p s, a q , N h p s, a, s 1 qq , @
s 1 P S ,
|p p ´ ˇ p h,s,a q v | ď α ˚ p ˇ N h p s, a q , ˇ p h,s,a , v q ) , @p h, s, a q ; 23:
P h,s,a Ð t clip p p, W q : p P ˜
P
h,s,a u , @p h, s, a q ; 24:
Return : b h,s,a P h,s,a .
25: Function : Design p
P q : 26: p Ð arbitrary element in P ; 27: for i “ 1 , 2 , ..., K 3 do 28: ˜
d ˜ π j h p s, a q Ð W ˜ π j p 1 h,s,a , p q for 1 ď j ď i ´ 1 and any p h, s, a q ; 29: r ih p s, a q Ð min "
[Equation]
ř
[Equation]
´
[Equation] [Equation]
d
[Equation]
πjh p s,a q ,* , @p h, s, a q ; 30: ˜ π i Ð Policy Search p r
i , P q ; 31: end for 32: t π, p u Ð Sum ´␣
K 3 , ˜ π
i , p ( Ki “¯ ; 33:
Return : π . estimate the optimal value function at the end of each batch, and use it to build a tighter confidenceregion.
As presented in Line 22 Algorithm 3, we use two empirical transition probabilities to constructthe confidence region.
Noting that the samples in the m -th batch is independent of v m ´ 1 , we couldadd a Bernstein-style constraint, where α ˚ p n, p, v q “
[Equation]
V p p,v q
[Equation]
Regret Analysis (Proof Sketch of Theorem 1)
Additional notations In this section, we use N mh p s, a, s 1 q to denote the visit count of p s, a, h, s 1 qafter the m -th batch.
We also define N mh p s, a q “ max t
ř s 1 N mh p s, a, s 1 q , 1 u .
We use t ˇ N mh p s, a, s 1 quto denote the counts of the m -th batch.
Similarly we define ˇ N mh p s, a q “ max t ř s 1 ˇ N mh p s, a, s 1 q , 1 u .
Let W ˚ be the known set after the first two stages.
Let ˆ P mh,s,a,s 1 “ N mh p s,a,s 1 q N mh p s,a q be the empiricaltransition model for 1 ď m ď 2 H ` M .
For 2 H ` 1 ď m ď 2 H ` M , define t ˇ P mh,s,a u be theclipped transition model, i.e., t ˇ P mh,s,a u h,s,a “ clip ˆ!” ˇ N mh p s,a,s 1 q ˇ N
[Equation]
s [Equation] P [Equation]
[Equation]
˙ . The good event Define ¯ P “ clip p P, W ˚ q .
We define G h,s,a,s 1 to be the event where (i)
| ˆ P mh,s,a,s 1 ´
P h,s,a,s 1 | is well bounded for 1 ď m ď 2 H ` M ; (ii) |p ˇ P mh,s,a ´ ¯ P h,s,a q v m ´
1 h ` 1 | is well bounded for2 H ` 1 ď m ď 2 H ` M ; (iii) |p ˇ P mh,s,a ´ ¯ P h,s,a q ¯ V ˚ h ` 1 | is well bounded for 2 H ` 1 ď m ď 2 H ` M ,where ¯ V ˚ is the optimal value function with respect to the transition model ¯ P and reward functionr .
Then the good event G is defined as X h,s,a,s 1 G h,s,a,s 1 .
By Bennet’s inequality (Lemma 8) and
Bernstein’s inequality, we can show that Pr r G s ě 1 ´ 12 S 2 AHKδ .
The formal definition of G ispostpone to Appendix D.1 due to space limitation.
Lemma 3.
Conditioned on G , we have ¯ P P P m for 2 H ď m ď 2 H ` M .
Noting that the batch complexity is bounded by 2 H ` M “ O p H ` log 2 log 2 p K qq , it suffices toprove the regret bound.
We start with counting the regret in the first two stages.
The regret in the firstbatch is bounded by O p H 2 k 1 q trivially.
As for the second batch, we have that
Lemma 4.
Conditioned on G , with probability 1 ´ 4 SAHδ the regret bound in the second batch isbounded by O
[Equation]
To count the regret in the third stage, we first show that the difference between the clipped model andthe original model could be ignored.
Lemma 5.
Conditioned on G , with probability 1 ´ 4 S 2 AH 2 δ , for any optimal policy π ˚ , it holdsthat Pr π ˚ rD
h P r H s , p h, s h , a h , s h ` 1 q R W ˚ s ď O ´
S 3 A 2 H 3 ι k¯
Based on Lemma 5, we further have that
Lemma 6.
Recall that ¯ V ˚ be the optimal value function with respect to the transition model ¯ P andreward function r .
It then holds that ¯ V ˚ 1 p s 1 q ď V ˚ 1 p s 1 q ď ¯ V ˚ 1 p s 1 q ` O
[Equation]
¯
[Equation]
Proof.
The left side is obvious since the reward at z is always 0.
On the other hand, letting π ˚ be anoptimal policy and E be the event where D h P r H s , p h, s h , a h , s h ` 1 q R W ˚ .
Then we have that
[Equation]
Recall that gap m ` 1 : “ max π P Π p r, P m q p
U
π p
P m q ´ L π p P m qq .
For m ě 2 H ` 1 , we have that
Lemma 7.
Conditioned on G , with probability 1 ´ 4 SAHKMδ , it holds thatgap m `
[Equation]
[Equation]
By Lemma 3, 6 and 7, for any 2 H ď m ď 2 H ` K and any π P Π p P m q , we have that
[Equation]
Recall that k 1 “a
SAKι { H , k 2 “ 288 S 3 A 2 H 4 ?
Kι and K m “
Q
K 1 ´ 12 m
U for 1 ď m ď M .
It then holds that K m ´ 2 H ` 1 ? K m ´ 2 H “ ? K for any 2 H ` 1 ď m ď 2 H ` K .
Noting that the regret in them ` 1 -th batch is bounded by K m ` 1 ´ 2 H ¨ gap m ` 1 , and the regret in the 2 H ` 1 -th batch is boundedby K 1 “ O p?
K q , the total regret is bounded by
Regret p K q “ M ¨ O ´a
{+SAH 3 K ln p K q ι 2 ` S 
+}
{+A 98 H 178 ι 58 K 38 ` S 
 A 134 H 334 ln p K q ιK 14 ` S 112 A 92 H 172 ι+} ¯ .
The proof is completed by replacing δ by
[Equation]
Broader Impact This work focus on the theory of multi-batch reinforcement learning, and thebroader impact is not applicable.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A]
The paper is theoretical and there is no numerical experiments.
• Did you include the license to the code and datasets?
[N/A]
The paper is theoretical and there is no numerical experiments.
• Did you include the license to the code and datasets?[N/A]
The paper is theoretical and there is no numerical experiments.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] We provide a near-optimal regret bound for multi-batch
RL
(b) Did you describe the limitations of your work?
[Yes] We focus on studying the tabular
case.
More efforts are required to extend the results to RL with function approximation
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The paper
is theoretical and there is no possible negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] .
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We sketch the proof
in the main body.
The details are postpone to the appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
The paper is
theoretical and there is no numerical experiments.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[N/A]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
The paper is
theoretical and there is no numerical experiments.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
This paper is irrelevant to crowdsourcing or human projects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]