Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning
Abstract
In this paper, we study the episodic reinforcement learning (RL) problem modeled by finite-horizon Markov Decision Processes (MDPs) with constraint on the number of batches.
The multi-batch reinforcement learning framework, where the agent is required to provide a time schedule to update policy before everything, which is particularly suitable for the scenarios where the agent suffers extensively from changing the policy adaptively.
Given a finite-horizon MDP with S states, A actions and planning horizon H , we design a computational efficient algorithm to achieve near-optimal regret of ˜
O p a
SAH 3 K ln p 1 { δ qq 1 in K episodes using O p H ` log 2 log 2 p K qq batches with confidence parameter δ .
To our best of knowledge, it is the first ˜ O p?
SAH 3 K q regret bound with O p H ` log 2 log 2 p K qq batch complexity.
Meanwhile, we show that to achieve ˜ O p poly p S, A, H q?
K q regret, the number of batches is at least Ω p H { log A p K q ` log 2 log 2 p K qq , which matches our upper bound up to logarithmic terms.
Our technical contribution are two-fold: 1) a near-optimal design scheme to explore over the unlearned states; 2) an computational efficient algorithm to explore certain directions with an approximated transition model.
Introduction
In reinforcement learning (RL), the learning agent interacts with the environment to maximize the total reward by making sequential decisions.
The agent typically has to achieve two seemingly very different goals: to try as many actions and reach as many states as possible so as to learn more information about the environment (a.k.a. exploration ) and to follow the policy that collects the high rewards according to the learned information (a.k.a. exploitation ).
To address this explorationexploitation dilemma and achieve the near-optimal regret bounds, the agent usually needs to adjust his/her strategies adaptively based on the historical trajectories and make frequent policy changes [Azar et al., 2017, Zanette and Brunskill, 2019, Zhang et al., 2020].
On the other hand, however, too much adaptivity requirement usually leads to lower level of parallelism, impeding the large-scale deployment of the RL algorithms (which is often in a distributed manner).
Frequent policy updates also suffer the cost of re-deploying policies in many practical applications.
For example, in medical domains, it often requires complete discussion among many experts to change the treatment plans, which is not affordable in terms of both time and monetary cost [Lei et al., 2012, Almirall et al., 2012, 2014]; in RL for hardware placement [Mirhoseini et al., 2017], rewriting the program into the hardware for too many times is strongly discouraged.
Similar challenges also arise in applying RL to personalized recommendation system [Yu et al., 2019] and database optimization [Krishnan et al., 2018].
In such cases, the learning agent should minimize the number of policy switches while keeping the regret affordable.
Bai et al.
[2019] first proposed the provably efficient RL algorithms with low switching costs under the Q -learning algorithmic framework together with the lazy update techniques.
However, their method needs to actively monitor the data in real time to determine whether a policy change is to be initiated.
In other words, although the number of policy switches by [Bai et al., 2019] is low, the (usually long) time periods when the same policy is used still cannot be parallelized due to the policy-change trigger in their algorithms which is intrinsically sequential.
In order to address this problem, we propose and study under the framework of multi-batch RL , where the learning agent has to determine the number of batches and length of each batch before the learning process starts, 2 and uses as few batches as possible to achieve a low regret.
Multi-batch
RL algorithms can be easily deployed in a distributed fashion as the episodes during the same batch can be easily and fully parallelized.
The idea of batch learning is also being widely practiced.
For example, in medical trials, the medical center usually collects the data during a fixed time period among a batch of patients and then designs the experiment for the next phase based on the learned information in previous phases [Lei et al., 2012, Almirall et al., 2012, 2014].
Formally, we define multi-batch RL and batch complexity as below.
Definition 1 (Multi-Batch RL with complexity M ) .
The agent determines a group of lengths t t m u Mm “ 1 such that
[Equation]
before the learning process starts.
For m “ 1 , 2 , . . .
, M , the agent sets a policy π m
and then follows π m for t m episodes.
We highlight that an upper bound for batch complexity implies the same upper bound for global switching cost, since each policy switch means a new batch.
It is also worth noting that the proposed batch RL framework is fully parallelizable during each batch for the applications where dataset comes in batch (e.g., clinical trial).
Like other RL settings, we have the natural and interesting question:
Question 1.
Is it possible to achieve near optimal batch complexity, while keeping the regret ˜ O p?
SAH 3 K q .
We provide a positive answer for Question 1, which we state as below.
Theorem 1.
Let 3 ι “ ln p 2 { δ q .
For any episodic MDP, with probability 1 ´ δ , under Algorithm 1 the regret in T episodes is bounded by
[Equation]
and the batch complexity is bounded by O p H ` log 2 log 2 p K qq .
Moreover, the computational cost of
Algorithm 1 is ˜
O p S 4 AHK 3 ` S 3 A 2 H 2 K 3 q .
On the other hand, we show a lower bound of batch complexity as below.
Theorem 2.
For any algorithm with O p poly p S, A, H q?
K q regret bound, the batch complexity is at least Ω p H { log A p K q ` log 2 log 2 p K qq .
Compared to the lower bound of Ω p log 2 log 2 p K qq in [Gao et al., 2019] for multi-armed bandit problem, additional Ω p H { log A p K qq batches are required to explore the structure of the MDP.
Due to space limitation, we defer the full proofs of Theorem 1 and Theorem 2 to Appendix D and Appendix B respectively.
Our contribution.
We propose the framework of multi-batch RL, and first achieve O p H ` log 2 log 2 p K qq sample complexity bound with the near-optimal ˜ O p ?
SAH 3 Kι q regret bound with an efficient algorithm.
We also prove that for any algorithm with O p poly p S, A, H q?
K q regret, the global switching cost is at least Ω p H { log A p K q ` log 2 log 2 p K qq , which implies a nearly matching lower bound of Ω p H { log 2 p K q ` log 2 log 2 p K qq for the batch complexity.
We also note that the O p H ` log 2 log 2 p K qq batch complexity implies an O p H ` log 2 log 2 p K qq bound for the global switching cost, which is also a near optimal upper bound.
Organization
Related Works
Bandit algorithms with limited adaptivity.
Bandit problem with low switching cost is widely studied in past decades [Cesa-Bianchi et al., 2013, Perchet et al., 2016, Gao et al., 2019, SimchiLevi and Xu, 2019].
Cesa-Bianchi et al.
[2013] showed an ˜Θ p K 23 q regret bound under adaptive adversaries and bounded memories.
Perchet et al.
[2016] proved a regret bound of ˜Θ p K 1 1 ´ 21 ´
M q for the two-armed bandit problem within M batches, and later Gao et al.
[2019] extended their result to the general A -armed case.
Besides the setting of classical multi-armed bandit problem, other settings has also been studied, e.g., multinomial bandit problem [Dong et al., 2020] and linear bandit problem [Ruan et al., 2020].
Episodic reinforcement learning with low switching cost.
For model-based algorithms, by doubling updates, the global switching cost is O p SAH log 2 p K qq while keeping the regret ˜ O p?
SAKH 3 q Azar et al.
For model-free algorithms, Bai et al.
[2019] first studied RL with low switching cost.
They proposed a Q -learning algorithm with lazy update to achieve ˜ O p?
SAKH 4 q regret bound and O p SAH 3 log p K { A qq local switching cost.
Recently Zhang et al.
[2020] established a better regret bound of ˜ O p?
SAKH 3 q
and O p SAH 2 log p K { A qq local switching cost.
Besides, Gao et al.
[2021] generalized the problem to Linear RL, and established a regret bound of ˜ O p?
d 3 H 4 K q with O p dH log p K qq global switching cost.
Recent work Qiao et al.
[2022] achieved
O p HSA log 2 log 2 p K qq switching cost and ˜
O p poly p S, A, H q?
K q regret with a computational inefficient algorithm.
Regret minimization for reinforcement learning.
There is a long line of works devoting to regret minimization for RL problem [Kakade, 2003, Jaksch et al., 2010, Bartlett and Tewari, 2009, Dann et al., 2019, Azar et al., 2017, Jin et al., 2018, Zanette and Brunskill, 2019, Zhang and Ji, 2019, Zhang et al., 2020, Li et al., 2020, Zhang et al., 2021, 2022].
For tabular setting, near optimal regret bound of ˜ O p?
SAH 3 T q has been established by [Azar et al., 2017, Zanette and Brunskill, 2019, Zhang et al., 2020] for both model-based and model-free algorithms.
However, fewer algorithms focused on the setting of multi-batch RL.
Preliminaries
Episodic reinforcement learning.
M “ x S , A , r, P, s 1 y , where
S ˆ
A is the discrete state-action space, r “ t r h p s, a qu p s,a qP S ˆ A ,h Pr H s is the known 4 reward function, P “ t P h p s, a qu p s,a qP S ˆ A ,h Pr H s is the unknown transition model and s 1 is the fixed initial state 5 .
We assume that the reward function r h p s, a q P r 0 , 1 s for any p h, s, a q .
In each episode, the agent starts at s 1 , then takes actions and transits to the next state step by step, and finally conducts the trajectory tp s h , a h , s h ` 1 qu Hh “ 1 .
The target of the agent is to maximize the accumulative reward function[-
-] ř
Hh “ 1 r h p s h , a h q .
A policy π can be viewed as a series of mappings t π h u Hh “ 1 where π h : S Ñ ∆
A maps s h to a distribution over the action space at the h -th step, where π h p a | s q is the probability taking action a at state s of the h -th horizon.
Given a policy π , the (optimal) Q -function and value function are given by
[Equation]
Let π p k q denote the policy in the k -th episode.
Then the regret is given by
[Equation]
Notations In this paper, we use E π,p r¨s ( P π,p r¨s ) to denote the expectation (probability) following policy π under transition model p .
In particular, E π r¨s ( P π r¨s ) denotes the expectation (probability) following π under the true transition model P .
We define the general value function
[Equation]
We use 1 to denote the S -dimensional vector r 1 , 1 , . .
, 1 s J and 1 h,s,a to denote the reward function r 1 such that r 1 h 1 p s 1 , a 1 q “ I rp h, s, a q “ p h [Equation] , s 1 , a 1 qs .
We also define t d πh p s, a qu p s,a, h q be the occupancy distribution of π .
That is, d πh p s, a q “ E π r
I
rp s h , a h q “ p s, a qss .
∆ d is used to denote the d -dimensional simplex.
For two vector x, y with the same dimension, we write x J y as xy for convenience.
For p P ∆ S and v P R S , we define V p p, v q “ pv 2 ´ p pv q 2 .
For N ě 1 , we use r N s to denote the set r 1 , 2 , . . .
, N s .
Technique Overview
In this section, we first introduce the policy elimination framework, which enjoys the near-optimal batch complexity.
Then we summarize the technical challenges to achieve the near-optimal regret bound efficiently under this framework.
At last, we introduce our major technical contributions.
4.1 Policy Elimination Framework
Following the methods in multi-batch bandit learning Perchet et al.
[2016], Gao et al.
[2019], we construct our main algorithm using policy elimination.
Like most model-based reinforcement learning methods, we maintain a confidence region P for the transition model, where the true transition model
P P P with high probability.
Before each batch starts, for a policy π and a reward function u , by extended value iteration (See Algorithm 5 in Appendix C.2), we are able to compute the confidence interval r L π p u, P q , U π p u, P qs for the value function of π , where
[Equation]
Here z is a virtual state for the infrequent state-action-state triples (See Function clip in Algorithm 2).
The reason why we give reward 1 for z in computing the upper confidence bound is to encourage exploration to these infrequent state-action-state triples.
By policy elimination we get Π p r, P q “ !
π ˇˇ
U π p r, P q ě sup π 1 L π 1 p r, P q ) as the set of survived policies.
The next step is to choose a policy π P Π p r, P q and execute π in the current batch.
Defining P m to be the confidence region for the transition model after the m -th batch and gap m ` 1 “ max π P Π p r, P m q p U π p r, P m q ´ L π p r, P m qq , the regret in the m ` 1 -th batch could be bounded by t m ` 1 gap m ` 1 .
Therefore, the main task is to design efficient exploration policy to reduce gap m for each 1 ď m ď M .
Following the policy elimination framework above, we have two major challenges to achieve the near-optimal regret bound with an efficient algorithm. Difficulty in exploration Fix the reward function r and confidence region P . To construct tight confidence interval for every policy π P Π p r, P q , we need to find a policy π P Π p r, P q to collect enough samples for each p h, s, a q .
To address the problem, Qiao et al.
[2022] proposed an algorithm named APEVE, which learns each p h, s, a q triple independently.
More precisely, for each p h, s, a q P r H sˆ S ˆ A , the algorithm searches for a policy π h,s,a to maximize the probability of visiting p h, s, a q over Π p r, P q , and then execute π h,s,a to collect samples for p h, s, a q .
However, this algorithm might be inefficient in sampling, since different horizon-state-action triples may match along with the same exploration policy.
As shown in Qiao et al.
[2022], the regret bound might be sub-optimal with this algorithm.
Therefore, to achieve the near-optimal regret bound, we need to design a new exploration strategy to utilize the correlationship among different horizon-state-action triples.
Difficulty in efficient implementation Because the policy set Π p r, P q might have exponential size, naive enumeration is not applicable to searching for a good exploration policy.
As a consequence, it requires additional efforts to study the structure of Π p r, P q .
For example, when r “ 0 , Π p r, P q is the set of all possible policies.
In this case, we can use extended value iteration (See Algorithm 5) to find the policy which visits p h, s, a q most frequently.
4.3 Key Techniques
Near-optimal design scheme Unlike RL algorithm with limited switching cost, in multi-batch reinforcement learning, the agent can not change the policy adaptively.
As a result, we need to design a policy with proper coverage ratio for all the survived policies.
That is, using the data collected following this policy, the length of the confidence interval for any survived policy is bounded by a uniform threshold. Recall that d πh p s, a q “ E π r I
rp s h , a h q “ p s, a qs . Using classical regret analysis for tabular RL [Azar et al., 2013, Zanette and Brunskill, 2019], for a fixed policy π , the length of confidence interval for π could be roughly bounded by
[Equation]
where Var h p s, a q is the variance term with respect to P h,s,a and V ˚ h ` 1 p¨q , and N h p s, a q ě 1 is the count of p h, s, a q .
Because ř s,a,h d πh p s, a q Var h p s, a q could be uniformly bounded by O p H 2 q using classical analysis, we focus on bounding the term ř s,a,h d πh p s,a q N h p s,a q .
Suppose the policy for current batch is ˜ π .
After this batch, we roughly have that N h p s, a q9 d ˜ πh p s, a q .
So it corresponds to find a policy ˜
π P Π p r, P q to minimize the worst-case coverage number max π P Π p r, P q [Equation] [Equation] q .
For this problem, we have the lemma below, and the proof is deferred to Appendix E.1.
Lemma 1.
Let d ą 0 be an integer.
Let X Ă p ∆ d q m .
Then there exists a distribution D over X , such that
[Equation]
where y “ t y i u dmi “ 1 “ E x „ D r x s .
Moreover, if X has a boundary set B X with finite cardinality, we can find an approximation solution for D in poly p|B X |q time. Plugging X “ ␣ [Equation]
[Equation] [Equation] q ( , d “ SA and m “ H into Lemma 1, there exists a policy ˜
π being a mixture of policies in Π p r, P q , such that max π P Π p r, P q ř s,a,h d πh p s,a q d ˜ πh p s,a q “ SAH .
In this way, we can find the desired exploration policy ˜ π by assuming the knowledge of t d πh p¨ , ¨qu H h “for all π P Π p r, P q .
Given the design scheme above, it remains two problems, for which we present solutions below: 1) t d πh p¨ , ¨qu Hh [Equation] 1 is unknown; 2) even assuming t d πh p¨ , ¨qu Hh “ 1 is known, it is hard to find ˜ π since the cardinality of
[Equation] might be exponential in SH .
Constructing tight confidence region To estimate t d πh p¨ , ¨qu Hh “ 1 , we consider to construct a tight confidence region for the transition model to estimate the occupancy distribution up to a constant ratio.
Definition 2.
We say a confidence transition region P “ b h,s,a P h,s,a is tight with respect to p 1 iff (i) p 1 P P ; (ii) e ´ 1 H p 1 h,s,a,s 1 ď p h,s,a,s 1 ď e 1 H p 1 h,s,a,s 1 for any p h, s, a, s 1 q and any p h,s,a P P h,s,a ; (iii) P h,s,a has the form P h,s,a “ t p P ∆ S | a J i p ď
b i , i “ 1 , 2 , ..., m u where m ď poly p SM q .
In model-based reinforcement learning, these conditions are natural and it is easy to construct a tight confidence region with acceptable error.
Once we have a confidence region which is tight w.r.t.
the true transition model P , for any policy π and p h, s, a q , we can estimate the expected visit count W π p 1 h,s,a q by W π p 1 h,s,a , p q for any p P P because
[Equation]
With W π p 1 h,s,a , p q as approximation of d πh p s, a q , we can continue the analysis above by paying a constant factor.
To learn such a confidence region, by Bennet’s inequality (Lemma 3), it suffices to visit p h, s, a, s 1 qfor C 1 H 2 ι for each p h, s, a, s 1 q , where C 1 is an universal constant.
By this idea, we try to visit each p h, s, a, s 1 q as much as possible.
In the meantime, it is very possible that some p h, s, a, s 1 q tuples are extremely hard to visit.
Fortunately, with proper exploration scheme, we can show that the maximal probability to visit such tuples is well-bounded, so that these tuples could be ignored by suffering regret O p?
T q .
Computational efficient design scheme Assume the confidence region P is tight w.r.t.
P .
We invoke reward-zero exploration to learn a sub-optimal solution for the problem min ˜ π P Π p r, P q max π P Π p r, P q ř
[Equation]
,s,a d
[Equation]
p s,a q d ˜ πh p s,a q .
Let p P P be fixed and define ˜ d πh p s, a q “ W π p 1 h,s,a , p q be the approximation for d πh p s, a q .
We define ˜ π
i “ arg max π P Π p r, P
q W π p r i , p q for 1 ď i ď k “ K 3 , where r ih p s, a q “ min
[Equation]
Let ˜ π be the mixture of t ˜ π
i u ki “ 1 .
For any policy π , we have that ÿ
[Equation]
[Equation]
[Equation]
Here (4) holds by the tightness of P , (5) holds by the fact that r i h p s,
a q ě r k ` 1 h p s, a q “ min
[Equation]
) for any p h, s, a q , and (6) holds by the optimality of ˜ π i for 1 ď i ď k . With (7) in hand, max π P Π p r, P q ř h,s,a d πh p s,a q d ˜ πh p s,a q is roughly bounded by
O p SAH log p K qq 7 , which nearly matches the best worst-case coverage number number of SAH .
Computational efficient constrained exploration Let u, u 1 be two reward functions and P be a set of transition models.
As stated before, for general Π p u, P q , it might be non-trivial to solve the problem ˜ π “ arg max π P Π p u, P
q W π p u 1 , p q for fixed p P P .
As a trade-off, we turn to find some policy ˜ π P Π p u, P q such that W ˜ π p u 1 , p q ě c max π P Π p u, P
q W π p u 1 , p q , where c ą 0 is some universal constant.
The problem turns out to be a RL problem with a soft constraint.
For general Π p u, P q , the problem might be hard to solve.
Fortunately, on the benefit of the tight property of P , we can find such ˜ π efficiently.
Algorithms
In this section we present our algorithms.
The main algorithm (Algorithm 1) consists of three stages.
In the first two stages, we conduct naive exploration to identify the tuples which are hard to visit, which we called infrequent tuples.
In particular, the length of the second stage is slightly larger than that of the first stage, where we use the dataset in the first stage to reduce the regret in the second stage.
In this way, we can bound the regret in the first two stages by ˜ O p?
SAH 3 K q , while the probability of visiting the infrequent tuples is small enough.
After ignoring the infrequent tuples, we could obtain a tight confidence region.
Given the tight confidence region, we compute the confidence region for each policy and conduct policy elimination in the third stage.
The first and second stages contains O p H q batches, and the third stage contains O p log 2 log 2 p K qq batches.
So the batch complexity of Algorithm 1 is O p H ` log 2 log 2 p K qq .
Below we describe Raw Exploration (Algorithm 2) and Policy Elimination (Algorithm 3) in detail.
5.1 Raw Exploration
Given a dataset D with counts t N h p s, a, s 1 qu , we define the set of known tuples as tp h, s, a, s 1 q :
N h p s, a, s 1 q ě C 1 H 2 ι u and the left tuples are regarded as infrequent tuples.
In Algorithm 2, we are given a dataset.
Then we compute the corresponding confidence region P in
Line 20, where α p n, n 1 q “
[Equation]
We conduct exploration layer by layer over policies in the set of survived policies Π p r, P q .
By visiting each p h, s, a q as much as possible, we can judge whether a tuple p h, s, a, s 1 q is hard to visit using policies in Π p r, P q .
Given the set of known tuples W , we redirect all tuples not in W to an additional absorbed state z using clip p¨ , ¨q .
Once we prove that the probability of reaching z is small enough for the any optimal policy, we can directly learn under the clipped transition model.
In Line 6 Algorithm 2, the algorithm Policy Search is invoked.
Given any reward u, u 1 , any confidence region P and threshold ϵ ą 0 , this algorithm returns a policy ˜ π P Π p u, P q such that
W ˜ π
p u 1 , p q ě c max π P Π p u, P
q W π p u 1 , p q ´ ϵ with some universal constant c ą 0 .
Moreover, when
[Table]
min ! 1 d ˜ πh p s,a q , K) .
Actually (7) is sufficient for further analysis.
1: Input : reward function u , dataset D , length k ; 2: Initialize: C 1 Ð 200 ; 3: for h “ 1 , 2 , . . .
, H do 4:
P Ð CR p D q ; 5: for p s, a q P S
ˆ A
do 6: π h,s,a Ð Policy Search p u, 1 h,s,a , P q ; 7: end for 8: p Ð arbitrary element in P ; 9: t ˜ π h , p u Ð Sum ´␣
SA , π h,s,a , p ( p h,s,a q ¯ ; 10: π h be the policy which is the same as ˜ π h in the first h ´ 1 steps, and be the uniformly random policy in the left H ´ h ` 1 steps; 11:
Execute π h for k episodes, and collect the samples as D h ; 12:
D Ð D Y D h ; 13: end for 14: return: t D u ; 15: Function : CR p D q : 16:
N h p s, a, s 1 q Ð count of p h, s, a, s 1 q in D , for all p s, a, s 1 q ; 17:
N h p s, a q Ð max t ř s 1 N h p
s, a, s 1 q , 1 u for all p s, a q ; 18: ˆ p h,s,a,s 1 Ð N h p s,a,s 1 q N h p
s,a q , @p h, s, a, s 1 q ; 19:
W Ð tp h, s, a, s 1 q : N h p s, a, s 1 q ě C 1 H 2 ι u ; 20: ˜ P h,s,a Ð ␣ p P ∆ S | | p s
´ ˆ p h,s,a,s 1 | ď α p N h p s, a q , N h p s, a, s 1 qq , @
s 1 P S ( , @p h, s, a q ; 21:
P h,s,a Ð t clip p p, W q : p P ˜
P
h,s,a u , @p h, s, a q ; 22:
Return : b h,s,a P h,s,a .
23: Function : clip p p, W q 24: p 1 h,s,a,s 1 Ð p h,s,a,s 1 , @p h, s, a, s q P W ; 25: p 1 h,s,a,s 1 Ð 0 , @p h, s, a, s 1 q R W ; 26: p 1 h,s,a,z Ð
ř s 1 : p h,s,a,s 1 qR W p h,s,a,s 1 , @p h, s, a q P r H s
ˆ S ˆ A ; 27: p 1 h,z,a Ð 1 z , @p h, a q P r H s ˆ A ; 28:
Return : p .
P is tight w.r.t.
the true transition model P after clipping, the time complexity of the algorithm is
O p poly p SAHK q log p 1 { ϵ qq .
The algorithm and corresponding analysis is postponed to Appendix C.
It is also worth noting that executing each π h,s,a with probability 1 SA can not be regarded as a (history-independent) policy because the agent need to keep in mind which policy is chosen in current episode.
In contrast, the agent only needs to observe current state to take actions following a policy. To address this problem, we define an operator Sum to take sum over policies under some transition model. Formally, we have the lemma below and postpone the proof to Appendix E.2. Lemma 2.
Let P “ b p h,s,a q P h,s,a be a set of transition models such that P h,s,a Ă ∆ S is convex for any p h, s, a q .
Let tp π i , P i qu ni “ 1 be a sequence of policy-transition pairs such that P i P P .
For any t λ i u ni “ 1 such that λ i ě 0 for i ě
and ř i λ i “ 1 , there exists a policy π and P P P , satisfying that
[Equation]
a , [Equation] (8) for any p h, s, a q P r H s ˆ S ˆ A .
Furthermore, the time complexity to find t π, P u could be bounded by O p nS 3 A 2 H 2 q .
Therefore, for any t λ i , π i , P i u ni “ 1 satisfying ř ni “ 1 λ i “ 1 and λ i ě 0 for i ě 1 as input, there exists t π, P u such that W π p 1 h,s,a , P q “ ř i λ i
W π i p 1 h,s,a , P i q and P h,s,a P Convex pt P ih,s,a u ni “ 1 q for any p h, s, a q P r H s ˆ S ˆ A , where Convex p U q denotes the convex hull of the set U .
Then Sum is defined as Sum pt λ i , π i , P i u ni “ 1 q “ t π, P u .
1: Input: dataset D , length k ; 2: Initialize: D 0 Ð D , P ´ 1 Ð p ∆ S q SA C 1 Ð 100 , v ´ 1 h p s q Ð H
´ h ` 1 , @p h, s q P r H s ˆ S ;
K m Ð
Q
K 1 ´ 12 m
U for m “ 1 , 2 , . . .
, M “ r log 2 log 2 p K q s ; 3: N h p s, a, s 1 q Ð count of p h, s, a, s 1 q in D ; 4: W Ð tp h, s, a, s 1 q : N h p s, a, s 1 q ě C 1 H 2 ι u ; 5: for m “ 0 , 1 , 2 , . . .
, M ´ 1 do 6:
P m Ð P m ´ 1 X CR ˚ ´
D m , D m , W , t v m ´
1 h p s qu p h,s q ¯ ; 7: π m ` 1 Ð Design pp P m q ; 8: if ř mm 1 “ 1 K m 1 ď k then 9:
Execute π m ` 1 for K m ` 1 episodes; 10: else 11:
Execute π m ` 1 for k ´ p
ř mm 1 “ 1 K m 1 q episodes; 12: end if 13:
D m ` 1 Ð the dataset in the p m ` 1 q -th batch; 14:
Update the dataset D m ` 1 Ð D m Y D m ` 1 ; 15: v mh p s q Ð max π,p P P m E π,p ”ř
Hh 1 “ h r h p s h , a h q| s h “ s ı for all p h, s q P r H s ˆ S ; 16: end for 17: Function : CR ˚ p D , D 1 , W , v q : 18: t N h p s, a, s 1 qu Ð counts in D , N h p s, a q Ð max t ř s 1 N h p
s, a, s 1 q , 1 u for all p h, s, a, s 1 q ; 19: ˆ p h,s,a,s 1 Ð N h p s,a,s 1 q N h p
s,a q , @p h, s, a, s 1 q ; 20: t ˇ N h p s, a, s 1 qu Ð counts in D 1 , ˇ N h p s, a q Ð max t ř s 1 ˇ N h p s, a, s 1 q , 1 u for all p h, s, a, s 1 q ; 21: ˇ p h,s,a,s 1 Ð ˇ N h p s,a,s 1 q ˇ N h p s,a q , @p h, s, a, s 1 q ; 22: ˜ P h,s,a Ð !
p P ∆ S | | p s
´ ˆ p h,s,a,s 1 | ď α p N h p s, a q , N h p s, a, s 1 qq , @
s 1 P S ,
|p p ´ ˇ p h,s,a q v | ď α ˚ p ˇ N h p s, a q , ˇ p h,s,a , v q ) , @p h, s, a q ; 23:
P h,s,a Ð t clip p p, W q : p P ˜
P
h,s,a u , @p h, s, a q ; 24:
Return : b h,s,a P h,s,a .
25: Function : Design p
P q : 26: p Ð arbitrary element in P ; 27: for i “ 1 , 2 , ..., K 3 do 28: ˜
d ˜ π j h p s, a q Ð W ˜ π j p 1 h,s,a , p q for 1 ď j ď i ´ 1 and any p h, s, a q ; 29: r ih p s, a q Ð min "
[Equation]
ř
[Equation]
´
[Equation] [Equation]
d
[Equation]
πjh p s,a q ,* , @p h, s, a q ; 30: ˜ π i Ð Policy Search p r, r i , P q ; 31: end for 32: t π, p u Ð Sum ´␣
K 3 , ˜ π
i , p ( Ki “¯ ; 33:
Return : π . 5.2 Policy Elimination Given the dataset collected in the first two stages, we first compute the known set W .
Unlike
Algorithm 2, we do not update W in the rest time because the first two stages can ensure that the probability of visiting W C is O p 1 {?
K q .
As mentioned in Section 4, for each batch, we invoke reward-zero exploration to search for the policy with near-optimal coverage.
Based on such a policy, we can provide uniform bound for the length of confidence intervals for all survived policies, which enables us to using the batch sizes in bandit algorithms [Perchet et al., 2016, Gao et al., 2019].
Besides, to obtain a better regret bound, we estimate the optimal value function at the end of each batch, and use it to build a tighter confidence region.
As presented in Line 22 Algorithm 3, we use two empirical transition probabilities to construct the confidence region.
Noting that the samples in the m -th batch is independent of v m ´ 1 , we could add a Bernstein-style constraint, where α ˚ p n, p, v q “b V p p,v q ι n ` 3 ιn . .
Conclusion
In this paper we study multi-batch reinforcement learning and provide an efficient algorithm to achieve the near-optimal regret bound and batch complexity.
It would be an interesting problem to generalize our results to reinforcement learning with function approximation case, e g., linear MDP.
[-Regret p K q “ M ¨ O ´a SAH 3 K ln p K q ι 2 ` S 
 A 98 H 178 ι 58 K 38 ` S 
 A 134 H 334 ln p K q ιK 14 ` S 112 A 92 H 172 ι ¯ .-]Another important direction is to study the exact batch-regret trade-off for multi-batch reinforcement learning.
Broader Impact This work focus on the theory of multi-batch reinforcement learning, and the broader impact is not applicable
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[N/A]
The paper is theoretical and there is no numerical experiments.
• Did you include the license to the code and datasets?
[N/A]
The paper is theoretical and there is no numerical experiments.
• Did you include the license to the code and datasets?[N/A]
The paper is theoretical and there is no numerical experiments.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We provide a near-optimal regret bound for multi-batch RL (b) Did you describe the limitations of your work?
[Yes] We focus on studying the tabular case.
More efforts are required to extend the results to RL with function approximation (c) Did you discuss any potential negative societal impacts of your work?
[N/A]
The paper is theoretical and there is no possible negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] . 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] We sketch the proof in the main body.
The details are postpone to the appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
The paper is theoretical and there is no numerical experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
The paper is theoretical and there is no numerical experiments.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
This paper is irrelevant to crowdsourcing or human projects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]