Regrets Bounds of Concurrent Thompson Sampling
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We consider the concurrent reinforcement learning problem where n agents simultaneously learn to make decisions in the same environment by sharing experience(i.e.
data) with each other.
Existing works in this emerging area have empiricallydemonstrated that Thompson sampling (TS) based algorithms provide a particularlyattractive alternative for inducing cooperation, because each agent can independently sample a belief environment (and compute a corresponding optimal policy)from the joint posterior computed by aggregating all agents’ data , which inducesdiversity in exploration among agents while beneﬁting shared experience fromall agents.
However, theoretical guarantees in this area remain under-explored; inparticular, no regret bound is known on TS based concurrent RL algorithms.
In this paper, we ﬁll in this gap by considering two settings.
In the ﬁrst, we studythe simple ﬁnite-horizon episodic RL setting, where TS is naturally adapted intothe concurrent setup by having each agent sample from the current joint posteriorat the beginning of each episode.
We establish a ˜ O ( HS q ATn ) per-agent regretbound, where H is the horizon of the episode, S is the number of states, A is thenumber of actions, T is the number of episodes and n is the number of agents.
Inthe second setting, we consider the inﬁnite-horizon RL problem, where a policyis measured by its long-run average reward.
Here, despite not having naturalepisodic breakpoints, we show that by a doubling-horizon schedule, we can adapt
TS to the inﬁnite-horizon concurrent learning setting to achieve a regret bound of˜ O ( DS p ATn ) , where D is the standard notion of diameter of the underlying MDPand T is the number of timesteps.
Note that in both settings, the per-agent regretdecreases at an optimal rate of ⇥ ( 1 p n ) , which manifests the power of cooperationin concurrent RL.
Introduction
In concurrent reinforcement learning [21], many agents interact with (different instances of) the sameenvironment simultaneously and share data with each other in order to jointly learn an optimal policyfor the environment.
This learning scenario has important applications for large-scale automatedsequential decision making systems.
For instance, in recommendation systems [1]) (e.g. movierecommendation, personalized ads recommendation, personalized news/content recommendation), acompany often has a large number of servers – sometimes located in different geographical regions –that serve the entire population of the customer base.
As such, an incoming customer is randomlyrouted to a server (i.e. an agent), which learns to make recommendation decisions in conjunctionwith all the other servers, and the performance judgement is made on the entire set of servers.
Another example of concurrent RL is controlling a team of autonomous vehicles, which, as succinctlysummarized in [6], concerns a scenario where “each agent manages a single vehicle, and again, theagents learn from each other as data is gathered.
The goal could be to optimize a combination of metrics, such as fuel consumption, safety, and satisfaction of transportation objectives.
Exploratoryactions play an important role, and structured diversity of experience may greatly accelerate learning."
This application is particularly relevant and presents an unique opportunity for the emerging era ofautonomous driving.
Other concurrent RL applications include robotics ([7], [21]) and biology([22]).
An important opportunity in the concurrent RL setup –in comparison to the traditional single-agent
RL setup – lies in coordinated exploration: since those agents can share their own learning experiencewith others, the team as a whole should achieve better overall regret performance (or equivalentlybetter per-agent regret performance) if they intelligently coordinate the exploration efforts.
Inparticular, different agents may want to diversify their exploratory actions so as to quickly reduce theteam’s uncertainty over the landscape of the environment in which they all operate.
By sharing theresulting data with each other, there is hope that the team would–in a environment-interaction timeframe much less than that from a single-agent setting–achieve the same level of performance.
When it comes to exploration, a substantial amount of literature has been devoted to designing(provably) efﬁcient algorithms in the single-agent RL setting.
One prominent class of algorithms,based on upper conﬁdence bounds, is known as UCRL (and its variants); (see[10] [3]) and manyreferences citing them and cited by them.
The UCRL family of algorithms embodies the importantphilosophy of "optimism in face of uncertainty" in their design and enjoys optimal theoretical regretguarantees (along with elegant analysis) in various settings, including both ﬁnite-horizon episodic RLand inﬁnite-horizon average-reward RL.
As such, it is perhaps not surprising that UCRL algorithmsare the ﬁrst family of algorithms that have been adapted to the concurrent RL settings [8] and [19].
In particular, concurrent UCRL has been analyzed for the sample complexity performance measure(i.e.
how many samples are needed to learn an ✏ -optimal policy) under both ﬁnite action spacesetting and inﬁnite action space setting.
More speciﬁcally, [8] provided a high-probability boundof ˜
O ( S [-
-]2 A ✏ 3 + SAn ✏ ) for the sample complexity with n agents interacting in the environment.
Theiralgorithm was extended from MBIE(see [23]), with a single agent performing concurrent RL acrossa set of n inﬁnite-horizon MDPs.
The results there show that with sharing samples from copies ofthe same MDP a linear speedup in the sample complexity of learning can be achieved.
But no regretbound was derived there for concurrent RL.
[19] designed an algorithm which extended concurrentexploration to continuous state-action spaces and allowed smooth generalization between concurrent
MDPs under inﬁnite horizon setting.
Likewise, the authors also explored the scenario where onesingle agent interacts with concurrent MDPs at the same time.
Additionally, they deﬁne an errormetric, called TCE (total cost of exploration), and showed an ˜ O (( 1 ✏ + n ) N ( d )) bound for TCE,where N ( d ) is a constant representing the covering number of the MDP.
This is sufﬁcient for an˜ O (( 1 ✏ 2 + n ✏ ) N ( d )) bound for sample complexity.
However, these sample-complexity guarantees notwithstanding, concurrent UCRL algorithms sufferfrom the critical disadvantage of no coordinated exploration: since the upper conﬁdence boundscomputed by aggregating all agents’ data are the same the entire team, each agent would follow theexact same policy, thereby yielding no diversity in exploration.
This point has been observed in [6, 5],which has advocated for designing concurrent RL algorithms based on Thompson sampling [17],[18],[24], [11] [14],[15], also known as posterior sampling reinforcement learning (PSRL).
Unlike UCRL,
PSRL manifests the philosophy of “probabilistic optimism in face of uncertainty", where the agentis uncertain about the transition probabilities and reward functions, and hence at the start of eachepisode, the agent would sample an MDP from the current posterior distribution based on datagathered up to that time and makes decisions according to the policy that is optimal with respect tothe sampled MDP.
In fact, not only PSRL is well-known to outperform UCRL empirically in single-agent RL setting,it also – by its very design – lends itself well to concurrent RL settings: each agent can samplefrom a posterior distribution and explores accordingly, which induces diversity in a coordinatedway.
Building on this insight, [6] and [5] proposed sophisticated Thompson sampling variants thatdemonstrated good empirical performance.
Despite these remarkable empirical results, to date, notheoretical guarantee has been provided on any concurrent Thompson sampling algorithm, simple orsophisticated.
Our goal in this paper is to make initial inroads on the theoretical front and attempt toprove the ﬁrst bounds on simple-but-natural concurrent PSRL algorithms.
1.1 Our Contributions
We establish per-agent regret guarantees for concurrent PSRL algorithms in two settings: one inﬁnite-horizon episodic setting and the other in inﬁnite-horizon average reward setting.
In the ﬁrstsetting, TS is naturally adapted into the concurrent setup by having each agent sample from thecurrent joint posterior at the beginning of each episode.
We establish a ˜ O ( HS q ATn ) per-agent regretbound, where H is the horizon of the episode, S is the number of states, A is the number of actions,
T is the number of episodes and n is the number of agents.
This regret bound further improvesto ˜ O
( H q SATn ) under the special Dirichlet prior.
Note that when n = 1 , both regret bounds aretight(see [13], [18]).
These two regret bounds further highlight that the per-agent regret decreasesat an optimal rate of ⇥ ( 1 p n ) and hence manifests the power of cooperation in concurrent RL: as ateam, the total regret achieved is ˜ O
( HS p nAT ) (and ˜ O
( H p nSAT ) under Dirichlet prior), which isminimax optimal, because that is the optimal regret performance (up to log factors) achievable by asingle-agent for nT episodes that can adapt at each episode 1 .
In the second setting, we consider the inﬁnite-horizon RL problem, where a policy is measuredby its long-run average reward.
Here, despite not having natural episodic breakpoints, we showthat by a doubling-horizon schedule, we can adapt TS to the inﬁnite-horizon concurrent learningsetting to achieve a regret bound of ˜ O ( DS p ATn ) , where D is the standard notion of diameter ofthe underlying MDP and T is the number of timesteps.
Here, we again see that the per-agent regretdecreases at an optimal rate of ⇥ ( 1 p n ) and the fact that a single learning trajectory does not preventthe team of agents from efﬁcient concurrent reinforcement learning.
Due to space limitation, we’llinclude simulation plots in the appendix.
Problem Formulation
We consider the Bayesian regret bound of concurrent Thompson Sampling of Markov decisionprocess in ﬁnite-horizon episodic setting and inﬁnite-horizon setting.
In both settings, we providebounds on the general prior distributions and Dirichlet prior distributions for concurrent Thompson
Sampling of the MDPs.
2.1 Finite-Horizon Episodic Setting
Formally, we consider the problem of learning to optimize a random (parallel) ﬁnite-horizon episodic
Markov decision process M ⇤ =
( S , A , R ⇤ , P ⇤ , H, ⇢ ) over repeated episodes of interaction.
Here
S = { 1 , 2 , . . .
, S } is the state space,
A = { 1 , 2 , . . .
, A } is the action space, H is the horizon ofeach episode, and ⇢ is the initial state distribution for each episode.
At the start of each episodek ( k  1) , each agent will receive the history across all the agents before period k .
And at thestart of episode 1 , all the agents don’t have any information regarding the history.
For every periodh = 1 , 2 , . . .
, H within an episode, each agent p observes state s ph 2 S , selects action a ph 2 A ,receives a reward r h ⇠ R ⇤ ( s ph , a ph ) , where the reward functions r h are assumed to be in [0 , 1] , andtransitions to a new state s ph +1 ⇠ P ⇤ ( s ph , a ph ) .
At time t , agent p chooses a policy ⇡ p , which isa mapping from the information that she has, i.e. H pt , to action a 2 A chosen by agent p .
Foreach agent p , MDP M and policy ⇡ p , the state-action value function for each period h is deﬁnedas Q M ⇡ p ,h ( s, a ) =
E ⇡ p ,M [ P Hj =
h ¯ r M ( s j , a j ) | s h
= s, a h
= a ] , where ¯ r M ( s, a ) =
E [ r | r ⇠ R M ( s, a )] .
The subscript ⇡ p indicates that the actions of agent p over periods h + 1 , . .
, H are chosen accordingto policy ⇡ p .
Let V M ⇡ p ,h ( s ) :
= Q M ⇡ p ,h ( s, ⇡ p ( s, h )) .
We say a policy ⇡ M is optimal for MDP M if⇡ M 2 arg max ⇡ V M ⇡ ,h ( s ) for all s 2 S and h = 1 , 2 , . . .
, H .
Let H t
=
[ np =1 H pt denote the history of observations made by time t (prior to and including time stept ) across all the agents p = 1 , 2 , . . . , n , where H pt is the history of agent p by time step t (include timet ), with t = 1 , 2 , . .
, T and T is the total time horizon.
So t = ( k  1)
H
+ h is the time step at periodh within episode k .
We use s pkh to denote the state of agent p at time t =
( k  1) H + h , i.e. episode k period h .
The RL algorithm we consider here is a deterministic sequence { ⇡ k : k
= 1 , 2 , . . . } offunctions, each mapping H ( k  1) H (i.e. the history of observations prior to episode k ) to a probabilitydistribution ⇡ k ( H ( k  1) H ) over policies, from which each agent samples a policy ⇡ kp for the k -thepisode.
The regret for agent p in episode k is  kp := P s 2 S ⇢ ( s )( V M ⇤ ⇡ ⇤ , 1 ( s )  V M ⇤ ⇡ kp , 1 ( s )) .
Theoverall regret in episode k is  k := P np =
P s 2 S ⇢ ( s )( V M ⇤ ⇡ ⇤ , 1 ( s )  V M ⇤ ( s ) ⇡ kp , 1 ) , where ⇡ ⇤ =
⇡ M ⇤ .
The regret incurred up to time T for policy ⇡ is Regret ( T, ⇡ , M ⇤ ) =
P d T/H e k =  k .
The Bayesianexpected regret for M ⇤ distributed according to the prior  is deﬁned as BayesRegret ( T, ⇡ ,  ) :
= E [ Regret ( T, ⇡ , M ⇤ ) | M ⇤ ⇠  ]
2.2 Inﬁnite-Horizon Setting
We follow the setting in [2] and consider the case where different agents can cooperate.
As in [2] weconsider communicating MDPs with ﬁnite diameter.
We ﬁrst deﬁne ’diameter’, ’communicating’ andother related terms.
Deﬁnition 2.1.
A deterministic policy ⇡ : S ! A is a mapping from state space S to action space A .
Deﬁnition 2.2.
Diameter D ( M ) of an MDP M is deﬁned as the minimum time requiredto go from one state to another in the MDP using some deterministic policy: D ( M ) =
max s {+
+} = s 0 ,s,s 0 2 S , ⇡ : S ! A T ⇡ s ! s 0 .
Here T ⇡ s ! s 0 is the expected number of steps it takes to reach state s 0 152 when starting from state s and using policy ⇡ .
Deﬁnition 2.3.
An MDP is communicating if and only if it has a ﬁnite diameter.
Deﬁnition 2.4.
The gain  ⇡ ( s ) of a policy ⇡ , from starting state s 1 = s , is deﬁned as the inﬁnitehorizon undiscounted average reward given by  ⇡ ( s ) =
E [lim T !
1 1 T P Ti =1 r ( s t , ⇡ ( s t )) | s 1 = s ] ,where s t is the state reached at time t , by executing policy ⇡ .
Lemma 2.1.
(Optimal gain for communicating MDPs).
For a communicating MDP M with diameter
D we have: 1.
([20] Theorem 8.1.2, Theorem 8.3.2)
The optimal gain  ⇤ is state independent andis achieved by a deterministic stationary policy ⇡ ⇤ , i.e. there exists a deterministic policy ⇡ ⇤ suchthat  ⇤ =
max s 0 2 S max ⇡  ⇡ ( s 0 ) =  ⇡ ⇤ ( s ) , 8 s 2 S ; 2.
([4], Theorem 4).
The optimal gain  ⇤satisﬁes the following equations  ⇤ =
min h 2 R S max s,a r ( s, a ) + P Ts,a h  h s = max a r ( s, a ) + P Ts,a h ⇤  h
⇤ s 8 s where the vector h ⇤ =
( h s ) s 2 S satisﬁes the following optimality equation
h ⇤ + ⇤ e = max a 2 A ( r ( s, a ) + P Ts,a h ⇤ ) .
Here h ⇤ is referred to as the bias vector of MDP M satisfying: 164 max
[Equation]
D .
Given the above deﬁnitions and results, we now formally deﬁne the Reinforcement Learning problemwe consider here.
The process proceeds in time steps t = 1 , 2 , . .
, T .
For agent p = 1 , 2 , . . . , n ,each starts from a state s p 1 at time step t = 1 .
In the beginning of each time step t , agent p takesan action a pt 2 A and observes the reward r ( s pt , a pt ) , where s pt is the state of agent p at time t , andalso the next state s pt +1 ⇠ P ( s pt , a pt ) . And again the reward function r is assumed to be in [0 , 1] .
Here r and P are the reward function and the transition model respectively for a communicating MDP M with diameter D .
Each learning agent knows the state-space S , the action space A , aswell as the rewards r ( s, a ) , 8 s 2 S , a 2 A , for the underlying MDP, but not the transition model 173 P or the diameter D .
The agent can use the past observations across all the agents to learn theunderlying MDP model and decide future actions.
The goal for each agent is to maximize totalreward
P Tt =1 r ( s pt , a pt ) , or equivalently minimize the total regret over a time horizon T , deﬁned as R ( T,
[Equation] [Equation] np
[Equation]
Tt =1 r ( s pt , a pt ) , where  ⇤ is the optimal gain of MDP M .
Remark 2.1.
Unlike the ﬁnite-horizon case, we assume that we know the reward functions as in[2].
Since without loss of generality, as long as the reward functions are bounded, we can have thesame bound on the regret.
This is because the unknown transition probability is the main source ofdifﬁculty in the analysis of the regret bound.
Notations We use S to denote the state space, where |S| = S , and A to denote the action space, 182 with |A| =
A .
There are n agents interacting with the environment.
We use r ( s, a ) to denote thereward related to state-action pair ( s, a ) .
In ﬁnite horizon case, H is the horizon of each episode, ⇢is the initial state distribution for each episode.
s pt is the state of of agent p at time step t , and a ptis the action that agent p takes at time step t , where p = 1 , 2 , . . . , n .
In ﬁnite horizon case, we use under the inﬁnite horizon case.
The policy ⇡ p is a mapping from the information set that agent p hasaccess to, to an action a in the action space.
Under ﬁnite horizon, we use s pkh to indicate the state ofagent p in episode k period h , and a pkh to indicate the action that agent p takes in episode k periodh .
We use P ⇤ to denote the transition probability of the true MDP,  ⇤ to denote the optimal gain ofthe true MDP under inﬁnite horizon case.
˜
k is the optimal gain of the sampled MDP in epoch kunder inﬁnite horizon case.
D is the upper bound of the diameter of the MDP considered.
For ﬁnitehorizon case, H pt denotes the information gathered by agent p before time step t , and H t = [ np =1 H ptis the history of observations prior to time step t across all agents.
For inﬁnite horizon case, F pk is theinformation of agent p within and before epoch k , and F k = [ np =1 F pk is the information within and 196 before epoch k across all agents.
Finite-Horizon Episodic Case
Algorithm Here we consider a concurrent version of the PSRL in [18].
At the beginning of eachepisode, each agent samples an MDP independently, where each one computes the optimal policy fortheir sampled MDP, and proceeds with this policy during the rest of the episode.
Then they gatherand update information at the end of each episode (See Algorithm 1).
There are n agents interactingin the environment, each episode is of length H , and the number of total time steps is T .
So there are
K = d T/H e episodes in the environment.
Intuition for ˜ O
( 1 p n ) per-agent regret bound We obtain an ˜ O ( p Tn )
Bayesian regret boundfor ﬁnite-horizon case, with ˜ O
( HS p ATn ) for general prior distribution, and ˜ O
( H p SATn ) for
Dirichlet prior distribution, where n is the number of agents interacting in the environment.
Note thatin the single-agent case, previous literature shows such regret bounds, ( ˜ O ( S p AT ) for general priorin [13] and ˜ O ( p SAT ) for Dirichlet prior in [18]).
This indicates that when n agents interactivelylearn in the same unknown environment over a time horizon of T, the bound scales in a non-linearway as if a single-agent learn over a time horizon of T 0 =
nT , which suggests the optimality ofour bounds here with respect to the number of agents n .
This is also a reﬂection of the power ofconcurrent learning within concurrent RL.
Improvement of Bayesian Regret Under Dirichlet Prior Distribution
The key analysis from˜ O ( HS p AT ) to
˜ O
( H p SAT ) follows the technical notes in [18], where we borrow a technicallemma on Gaussian-Dirichlet concentration bound ([16]).
With this, we are able to show that theregret contribution from the transition probability estimate can concentrate at a rate independent of S .
Finally we use a Sub-Gaussian tail bound to make the improvement.
The proof idea follows [18].
{+Algorithm 1: Concurrent PSRL Data: prior distribution  , H p 
+}
= ; , 8 p 2 [ n ] for episode k = 1 , 2 , . . .
do for p = 1 : n do
Sample MDP M kp ⇠  ( ·|H ( k  1) H ) ;
Compute µ kp 2 arg max µ V M kp µ, 1 ; for period h=1,...,H do take action a pkh = µ kp ( s pkh , h ) ; update H p ( k  1) H + h
= H p ( k  1) H + h  1 [ { s pkh , a pkh , r pkh , s pk ( h +1) } end end
H kH =
[ np =1 H pkH ; Update posterior distribution  ( ·|H kH ) ; end
3.1 Bayesian Regret Bound under General Prior Distribution
Theorem 3.1.
(Bayesian Regret Bound Under General Prior Distribution)
Let M ⇤ be thetrue MDP distributed according to any prior distribution  .
If the number of agents n satsiﬁes n  O ( S 2 AT log( SAnT )) , Then the Bayesian regret for Algorithm (1) is bounded by E [ Regret ( [Equation] [Equation]
,  , n
[Equation]
] =
[Equation]
O ( [Equation] p ATn ) Proof Sketch: We ﬁrst deﬁne a conﬁdence set M k for each episode k as in [3], so that this setcontains both sampled MDP M k and true MDP M ⇤ with high probability.
Then we decompose kp into  kp 1 ( M k 2 M k , M ⇤ 2 M k ) +  kp [ 1 ( M k / 2 M k or M ⇤ / 2 M k )] .
Then we use the 225 fact that when condition on history H ( k  1) H , we have E [ 1 { M k / 2 M k }|H ( k  1) H ] =
E [ 1 { M ⇤ / {+
+}
M k }|H ( k  1) H ] ([13]), so that  kp [ 1 ( M k / 2 M k or M ⇤ / 2 M k )] is bounded by 2 H P ( M ⇤ / 2 M k ) .
As for the ﬁrst part, namely  kp 1 ( M k , M ⇤ 2 M k ) , we found that within the conﬁdence intervals, 228  kp can be controlled with the desired upper bound.
And for  kp [ 1 ( M k / 2 M k or M ⇤ / 2 M k )] , 229 since it’s bounded by 2 H P ( M ⇤ / 2 M k ) , we just need to show that P ( M ⇤ / 2 M k ) is small, which is 230 indeed the case.
We decompose the regret into two parts, with one part negative and the other partpositive.
So we focus on bounding the positive part.
Then we rewrite the positive upper bound viathe Bellman operator, where one part is related to the concentration of reward estimate, and the otherpart is related to the concentration of transition probability estimate.
i.e. BayesRegret ( T, ⇡ ,  ) =
E [ Regret ( T, ⇡ , M ⇤ ) | M ⇤ ⇠  ] =
P d T/H e k =
P np =1 E [
P s 2 S ⇢ ( s )( V M ⇤ ⇡ ⇤ , 1 ( s )  V M ⇤ ⇡ kp , 1 ( s ))
| M ⇤ ⇠  ]
And we can write V M ⇤ ⇡ ⇤ , 1 ( s )  V M ⇤ ( s ) ⇡ kp , [-
-]1 =
V M ⇤ ⇡ ⇤ , 1 ( s )  V M kp ⇡ kp , 1 ( s ) | {z }  opt kp
+ V M kp ⇡ kp , 1 ( s )  V M ⇤ ⇡ kp , 1 ( s ) | {z }  conc kp .
Conditionedupon data before the start of episode k , H ( k  1) H , the true MDP M ⇤ and the sampled M k areidentically distributed.
This means that E [  opt kp ]  0 for all k, p .
Therefore, we just need tobound
[Equation]
For the convenience of notation, we write V kpkh =
V M kp ⇡ kp ,h .
Now we rewrite the regret via the Bellman operator, with w R ( x ) =
¯ r kp ( x )  ˆ r k ( x ) , w Ph ( x ) =( P kp ( x )  ˆ P k ( x ))
T V kpk,h +1 .
Then by some simple derivation we ﬁnd that E [  conc kp |H ( k  1) H ] 
E [
P Hh = [Equation] |
[Equation]
P Hh =1 | w Ph ( x pk,h +1 ) |H
( k  1) H ] .
Based upon this, we prove that with highprobability, both the true MDP and the sampled MDPs are in the conﬁdence sets constructed here.
And also conditional upon the event that both true MDP and sampled MDPs are in the conﬁdence set,the total regret can be controlled.
3.2 Bayesian Regret Bound under Independent Dirichlet Prior Distribution
Theorem 3.2.
(Bayesian Regret Bound Under Dirichlet Prior Distribution)
Let M ⇤ be the true
MDP distributed according to prior  with independent Dirichlet prior over transition probabilities.
Then the Bayesian regret for Algorithm (1) is bounded by
[Equation]
Proof Sketch:
From [18], under Dirichlet prior, by a union bound at each x pkh =
( s, a ) on P ( s, a )and r ( s, a ) , we have that for each p , with probability at least 1  1 nT ,
[Equation]
By pigeonhole principle, when we sum over all the k and p , we can decompose the right hand sideof the above inequality according to the number of times that each state-action pair were visited.
For those state-action pairs which were visited less than twice, since the reward function is within[0 , 1] , the sum of these parts are upper bounded by 2 SA + 1 .
Lemma ( ?? ) in the below guaranteesthat the regret from transition probability estimation error can be bounded at a rate independent of S , and Lemma ( ?? ) further provides a bound for the dominating Gaussian variable on the Dirichletdistribution.
In this way we can get an improved bound of ˜ O
( H p SAnT ) from the case under generalprior distribution.
We leave the proof in the appendix.
Inﬁnite-Horizon Case
Algorithm Algorithm 2 combines the idea of Thompson sampling and double epoch for inﬁnitehorizon case.
Similar to the ﬁnite-horizon episodic case, the agents sample MDP independently at the beginning of each epoch.
The difference is that unlike the ﬁnite-horizon case where eachepisode is of the same length, the inﬁnite horizon adapt a "double epoch" strategy ([3], [2]), wherewe stop the current epoch and move on to the next one immediately as long as the number of visits tosome state-action pair at least doubles.
As in ﬁnite-horizon case, we can also get ˜ O
( 1 p n ) per-agent
Bayesian regret bound as well.
For the similar reason as in the ﬁnite-horizon case, here we also havean improvement from ˜ O
( S p ATn ) to ˜ O ( p SATn ) if we condition on Independent Dirichlet priordistribution instead of on some general prior .
From Finite-Horizon To Inﬁnite Horizon: Doubling Epoch [12] points out the necessity to have"phases" in inﬁnite-horizon problem.
In inﬁnite-horizon case, if the reward of one state is slightlylarger than the reward in the current state, the MDP tends to switch to the other state very easily.
But for ﬁnite-horizon setting, an optimistic algorithm would eventually stop considering switchesbecause eventually, the loss for switching would be assessed to be higher than the potential gain fromswitching.
For this reason, recalculating the optimistic policy in each round may lead to the algorithmto switch states frequently, which can lead to linear regret in the end.
As in [2], we borrow the idea ofdoubling epoch [3], where the next phase starts once the number of visits to some agent’s state-actionpair at least doubles.
Since the total number of visits to any state-action pairs can be no more thannT , the number of epochs is bounded by some constant times SA log( nT ) .
4.1 Bayesian Regret Bound Under General Prior Distribution
for epochs k = 1 , 2 , . . .
do
Sample transition probability vectors: For each s a generate sample probability vectors Q ks,a as follows:
[Equation] do do Observe the transition to the end s pt ,a pt for some p, (i.e. for some ( pt , a pt ) ), then set ⌧ k + t + and break epoch . [Equation]
[Equation]
[Equation]
end end
Theorem 4.1.
(Bayesian Regret Bound Under General Prior Distribution)
For any communicating
MDP M with S states, A actions, diameter D .
Then for T  ⌦ ( SA log 4 ( SATn )) .
Then
BayesRegret
[Equation]
Proof Sketch: We ﬁrst write the total regret as the sum of regrets over all the epochs.
We focus onbounding the regret within any ﬁxed epoch.
Then we decompose the difference between the optimalgain of the true MDP and the reward functions as the sum of two parts: one part is the differencebetween the optimal gain of the true MDP and the optimal gain of the sampled MDP; the other part is the difference between the optimal gain of the sampled MDP and the reward function at somestate-action pair.
Then ﬁnally when we do a sum over all the epochs, we can get the desired regretbound.
As deﬁned before, R ( T, M , n ) =
Tn  ⇤  P np =
P
Tt =1 r ( s pt , a pt ) .
⇤ is the optimal gainof MDP M .
Algorithm (2) proceeds in epochs k = 1 , 2 , . .
, K , where K  SA log( T ) .
Now wedeﬁne R k
= [Equation]
k [Equation]  P
[Equation]
P np =1 r s pt ,a pt , where R k is the regret within epoch k .
So wecan write R k as R k =
[Equation]
where ˜  k is the optimal gain of the sampled MDP ˜ M k .
From Lemma (2.1) we know that for anystate s , optimal policy ˜ ⇡ k for communicating MDP ˜ M k , action a = ˜ ⇡ k
( s ) , ˜
k
= r s,a + ˜ P Ts,
a ˜ h  ˜ h s ,where ˜ P s,a = Q ks,a .
So we have⌧ k +1
[Equation]
X t = ⌧ k n
X p
=
( P s pt ,a pt  1 s pt ) T ˜ h
= ⌧ k +1  X t = ⌧ k n
X p =
( Q ks pt ,a pt  P s pt ,a pt ) T ˜ h + ⌧ k +1  X t = ⌧ k n
X p
=
( P s pt ,a pt  1 s pt ) T ˜ h
4.2 Bayesian Regret Bound Under Dirichlet Prior
Theorem 4.2.
(Bayesian Regret Bound Under Dirichlet Prior)
Let M be the true MDP distributedaccording to prior  with any independent Dirichlet prior over transition probabilities.
Then the
Bayesian regret for Parallel Inﬁnite-Horizon posterior sampling MDP algorithn 2 is bounded by
BayesRegret ( T, M , n,  ) = ˜ O ( D p SATn )
Proof Sketch: We use Lemma (G.2) to prove the concentration bound between sampled transitionprobabilities and the expectation of the true probabilities conditional on the current data.
And notethat one key property of Thompson sampling is that conditional upon the current data, the true MDPand the sampled MDP are identically distributed.
4.3 Worse-case Bound Under Dirichlet Prior
Theorem 4.3.
(Worst-case Inﬁnite-Horizon Regret Bound Under Dirichlet Prior)
With highprobability, Algoithm 5 (provided in Appendix B) is bounded by
[Equation]
Simulation
In this section, we describe our empirical study of concurrent Thompson Sampling on a Markov
Decision Process with a Dirichlet transition function and a normal reward function for both ﬁnitehorizon and inﬁnite-horizon cases.
In particular, we train the concurrent Thompson Sampling agentson randomly sampled MDPs and evaluate the regrets of these agents at the end of each episode/epochon a ﬁxed number of environments sampled from the ground truth distribution.
The simulationsettings are moved to appendix due to page limit.
Some of the experiment plots are shown in
Figure(1a), (1b), (2a) and (2b).
Conclusion
We have extended PSRL from a single agent to multiple agents, where the agents can concurrent witheach other to learn in parallel and share data with each other.
We consider two settings: both ﬁnitehorizon episodic MDP and inﬁnite-horizon MDP.
For state space size S , action space size A , total
[Figure]
[Figure] accumulated time horizon T , episode horizon H , we show a ˜ O ( HS q ATn ) per-agent Bayesian regretbound under general prior distribution; while especially for independent Dirichlet prior distribution,we show that a tighter bound can be achieved as ˜ O
( H q SATn ) .
Likewise, under inﬁnite horizon case,for the general prior distribution, we follow the idea in [2] to achieve a high-probability worst caseper-agent regret bound as ˜ O ( DS q
ATn ) , which is sufﬁcient for ˜ O ( DS q
ATn ) per-agent Bayesianregret bound; and for the inﬁnite-horizon case under dirichlet prior distribution, we combine the ideaof PSRL and double-epoch sampling, so as to obtain a tighter ˜ O
( D q
ATSn ) Bayesian regret bound,with D being the diameter of MDP considered here.
So far we provide a theoretical upper boundfor the concurrent learning under posterior sampling RL setting, and the multiplicative constants inthe bound were just properly chosen but not tuned in an optimistic way.
There is still more work tobe done if we want to put the algorithm into application, including the production of autonomousdriving chip design, etc.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]