Regrets Bounds of Concurrent Thompson Sampling
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the concurrent reinforcement learning problem where n agents simultaneously learn to make decisions in the same environment by sharing experience(i.e.
data) with each other.
Existing works in this emerging area have empiricallydemonstrated that Thompson sampling (TS) based algorithms provide a particularlyattractive alternative for inducing cooperation, because each agent can independently sample a belief environment (and compute a corresponding optimal policy)from the joint posterior computed by aggregating all agents’ data , which inducesdiversity in exploration among agents while benefiting shared experience fromall agents.
However, theoretical guarantees in this area remain under-explored; inparticular, no regret bound is known on TS based concurrent RL algorithms.
In this paper, we fill in this gap by considering two settings.
In the first, we studythe simple finite-horizon episodic RL setting, where TS is naturally adapted intothe concurrent setup by having each agent sample from the current joint posteriorat the beginning of each episode.
We establish a ˜ O ( HS (cid:113) ATn ) per-agent regretbound, where H is the horizon of the episode, S is the number of states, A is thenumber of actions, T is the number of episodes and n is the number of agents.
Inthe second setting, we consider the infinite-horizon RL problem, where a policyis measured by its long-run average reward.
Here, despite not having naturalepisodic breakpoints, we show that by a doubling-horizon schedule, we can adapt
TS to the infinite-horizon concurrent learning setting to achieve a regret bound of˜ O ( DS √ ATn ) , where D is the standard notion of diameter of the underlying MDPand T is the number of timesteps.
Note that in both settings, the per-agent regretdecreases at an optimal rate of Θ( 1 √ n ) , which manifests the power of cooperationin concurrent RL.
Introduction
In concurrent reinforcement learning [21], many agents interact with (different instances of) the sameenvironment simultaneously and share data with each other in order to jointly learn an optimal policyfor the environment.
This learning scenario has important applications for large-scale automatedsequential decision making systems.
For instance, in recommendation systems [1]) (e.g. movierecommendation, personalized ads recommendation, personalized news/content recommendation), acompany often has a large number of servers – sometimes located in different geographical regions –that serve the entire population of the customer base.
As such, an incoming customer is randomlyrouted to a server (i.e. an agent), which learns to make recommendation decisions in conjunctionwith all the other servers, and the performance judgement is made on the entire set of servers.
Another example of concurrent RL is controlling a team of autonomous vehicles, which, as succinctlysummarized in [6], concerns a scenario where “each agent manages a single vehicle, and again, theagents learn from each other as data is gathered.
The goal could be to optimize a combination of metrics, such as fuel consumption, safety, and satisfaction of transportation objectives.
Exploratoryactions play an important role, and structured diversity of experience may greatly accelerate learning."
This application is particularly relevant and presents an unique opportunity for the emerging era ofautonomous driving.
Other concurrent RL applications include robotics ([7], [21]) and biology([22]).
An important opportunity in the concurrent RL setup –in comparison to the traditional single-agent
RL setup – lies in coordinated exploration: since those agents can share their own learning experiencewith others, the team as a whole should achieve better overall regret performance (or equivalentlybetter per-agent regret performance) if they intelligently coordinate the exploration efforts.
Inparticular, different agents may want to diversify their exploratory actions so as to quickly reduce theteam’s uncertainty over the landscape of the environment in which they all operate.
By sharing theresulting data with each other, there is hope that the team would–in a environment-interaction timeframe much less than that from a single-agent setting–achieve the same level of performance.
When it comes to exploration, a substantial amount of literature has been devoted to designing(provably) efficient algorithms in the single-agent RL setting.
One prominent class of algorithms,based on upper confidence bounds, is known as UCRL (and its variants); (see[10] [3]) and manyreferences citing them and cited by them.
The UCRL family of algorithms embodies the importantphilosophy of "optimism in face of uncertainty" in their design and enjoys optimal theoretical regretguarantees (along with elegant analysis) in various settings, including both finite-horizon episodic RLand infinite-horizon average-reward RL.
As such, it is perhaps not surprising that UCRL algorithmsare the first family of algorithms that have been adapted to the concurrent RL settings [8] and [19].
In particular, concurrent UCRL has been analyzed for the sample complexity performance measure(i.e.
how many samples are needed to learn an ϵ -optimal policy) under both finite action spacesetting and infinite action space setting.
More specifically, [8] provided a high-probability boundof ˜
O ( S {+
+} A ϵ 3 + SAnϵ ) for the sample complexity with n agents interacting in the environment.
Theiralgorithm was extended from MBIE(see [23]), with a single agent performing concurrent RL acrossa set of n infinite-horizon MDPs.
The results there show that with sharing samples from copies ofthe same MDP a linear speedup in the sample complexity of learning can be achieved.
But no regretbound was derived there for concurrent RL.
[19] designed an algorithm which extended concurrentexploration to continuous state-action spaces and allowed smooth generalization between concurrent
MDPs under infinite horizon setting.
Likewise, the authors also explored the scenario where onesingle agent interacts with concurrent MDPs at the same time.
Additionally, they define an errormetric, called TCE (total cost of exploration), and showed an ˜ O (( 1 ϵ + n ) N ( d )) bound for TCE,where N ( d ) is a constant representing the covering number of the MDP.
This is sufficient for an˜ O (( 1 ϵ 2 + nϵ ) N ( d )) bound for sample complexity.
However, these sample-complexity guarantees notwithstanding, concurrent UCRL algorithms sufferfrom the critical disadvantage of no coordinated exploration: since the upper confidence boundscomputed by aggregating all agents’ data are the same the entire team, each agent would follow theexact same policy, thereby yielding no diversity in exploration.
This point has been observed in [6, 5],which has advocated for designing concurrent RL algorithms based on Thompson sampling [17],[18],[24], [11] [14],[15], also known as posterior sampling reinforcement learning (PSRL).
Unlike UCRL,
PSRL manifests the philosophy of “probabilistic optimism in face of uncertainty", where the agentis uncertain about the transition probabilities and reward functions, and hence at the start of eachepisode, the agent would sample an MDP from the current posterior distribution based on datagathered up to that time and makes decisions according to the policy that is optimal with respect tothe sampled MDP.
In fact, not only PSRL is well-known to outperform UCRL empirically in single-agent RL setting,it also – by its very design – lends itself well to concurrent RL settings: each agent can samplefrom a posterior distribution and explores accordingly, which induces diversity in a coordinatedway.
Building on this insight, [6] and [5] proposed sophisticated Thompson sampling variants thatdemonstrated good empirical performance.
Despite these remarkable empirical results, to date, notheoretical guarantee has been provided on any concurrent Thompson sampling algorithm, simple orsophisticated.
Our goal in this paper is to make initial inroads on the theoretical front and attempt toprove the first bounds on simple-but-natural concurrent PSRL algorithms.
1.1 Our Contributions
We establish per-agent regret guarantees for concurrent PSRL algorithms in two settings: one infinite-horizon episodic setting and the other in infinite-horizon average reward setting.
In the firstsetting, TS is naturally adapted into the concurrent setup by having each agent sample from thecurrent joint posterior at the beginning of each episode.
We establish a ˜ O ( HS (cid:113) ATn ) per-agent regretbound, where H is the horizon of the episode, S is the number of states, A is the number of actions,
T is the number of episodes and n is the number of agents.
This regret bound further improvesto ˜ O
( H (cid:113) SATn ) under the special Dirichlet prior.
Note that when n = 1 , both regret bounds aretight(see [13], [18]).
These two regret bounds further highlight that the per-agent regret decreasesat an optimal rate of Θ( 1 √ n ) and hence manifests the power of cooperation in concurrent RL: as ateam, the total regret achieved is ˜ O
( HS √ nAT ) (and ˜ O
( H √ nSAT ) under Dirichlet prior), which isminimax optimal, because that is the optimal regret performance (up to log factors) achievable by asingle-agent for nT episodes that can adapt at each episode 1 .
In the second setting, we consider the infinite-horizon RL problem, where a policy is measuredby its long-run average reward.
Here, despite not having natural episodic breakpoints, we showthat by a doubling-horizon schedule, we can adapt TS to the infinite-horizon concurrent learningsetting to achieve a regret bound of ˜ O ( DS √ ATn ) , where D is the standard notion of diameter ofthe underlying MDP and T is the number of timesteps.
Here, we again see that the per-agent regretdecreases at an optimal rate of Θ( 1 √ n ) and the fact that a single learning trajectory does not preventthe team of agents from efficient concurrent reinforcement learning.
Due to space limitation, we’llinclude simulation plots in the appendix.
Problem Formulation
We consider the Bayesian regret bound of concurrent Thompson Sampling of Markov decisionprocess in finite-horizon episodic setting and infinite-horizon setting.
In both settings, we providebounds on the general prior distributions and Dirichlet prior distributions for concurrent Thompson
Sampling of the MDPs.
2.1 Finite-Horizon Episodic Setting
Formally, we consider the problem of learning to optimize a random (parallel) finite-horizon episodic
Markov decision process M ∗ =
( S , A , R ∗ , P ∗ , H, ρ ) over repeated episodes of interaction.
Here
S = { 1 , 2 , . . .
, S } is the state space,
A = { 1 , 2 , . . .
, A } is the action space, H is the horizon ofeach episode, and ρ is the initial state distribution for each episode.
At the start of each episodek ( k ≥ 1) , each agent will receive the history across all the agents before period k .
And at thestart of episode 1 , all the agents don’t have any information regarding the history.
For every periodh = 1 , 2 , . . .
, H within an episode, each agent p observes state s p h ∈ S , selects action a p h ∈ A ,receives a reward r h ∼ R ∗ ( s ph , a ph ) , and transitions to a new state s ph +1 ∼ P ∗ ( s ph , a ph ) .
At time t ,agent p chooses a policy π p , which is a mapping from the information that she has, i.e. H pt , to actiona ∈ A chosen by agent p .
For each agent p , MDP M and policy π p , the state-action value functionfor each period h is defined as Q Mπ p ,h ( s, a ) =
E π p ,M [ (cid:80) Hj =
h ¯ r M ( s j , a j ) | s h
= s, a h
= a ] , where¯ r M ( s, a ) =
E [ r | r ∼ R M ( s, a )] .
The subscript π p indicates that the actions of agent p over periodsh + 1 , . .
, H are chosen according to policy π p .
Let V Mπ p ,h ( s ) :
= Q Mπ p ,h ( s, π p ( s, h )) .
We say apolicy π M is optimal for MDP M if π M ∈ arg max π V Mπ,h ( s ) for all s ∈ S and h = 1 , 2 , . . .
, H .
Let H t
=
∪ np = H pt denote the history of observations made by time t (prior to and including time stept ) across all the agents p = 1 , 2 , . . . , n , where H pt is the history of agent p by time step t (include timet ), with t = 1 , 2 , . .
, T and T is the total time horizon.
So t = ( k − 1)
H
+ h is the time step at periodh within episode k .
We use s pkh to denote the state of agent p at time t =
( k − 1) H + h , i.e. episodek period h .
The RL algorithm we consider here is a deterministic sequence {
π k : k = 1 , 2 , . . . } of functions, each mapping H ( k − 1) H (i.e. the history of observations prior to episode k ) to a probabilitydistribution π k ( H ( k − 1) H ) over policies, from which each agent samples a policy π kp for the k -thepisode.
The regret for agent p in episode k is ∆ kp := (cid:80) s ∈S ρ ( s )( V M ∗ π ∗ , 1 ( s ) − V M ∗ π kp , 1 ( s )) .
Theoverall regret in episode k is ∆ k := (cid:80) np
=1 (cid:80) s ∈S ρ ( s )( V M ∗ π ∗ , 1 ( s ) − V M ∗ ( s ) π kp , 1 ) , where π ∗ =
π M ∗ .
The regret incurred up to time T for policy π is Regret ( T, π, M ∗ )
= (cid:80) ⌈ T/H ⌉ k = ∆ k .
The Bayesianexpected regret for M ∗ distributed according to the prior ϕ is defined as BayesRegret ( T, π, ϕ ) :
= 142 E [ Regret ( T, π, M ∗ ) | M ∗ ∼ ϕ ]
2.2 Infinite-Horizon Setting
We follow the setting in [2] and consider the case where different agents can cooperate.
As in [2] weconsider communicating MDPs with finite diameter.
We first define ’diameter’, ’communicating’ andother related terms.
Definition 2.1.
A deterministic policy π : S → A is a mapping from state space S to action space A .
Definition 2.2.
Diameter D ( M ) of an MDP M is defined as the minimum time requiredto go from one state to another in the MDP using some deterministic policy: D ( M ) =
max s [-
-]̸ = s ′ ,s,s ′ ∈S ,π : S→A T π s → s ′ .
Here T π s → s ′ is the expected number of steps it takes to reach state s ′ 151 when starting from state s and using policy π .
Definition 2.3.
An MDP is communicating if and only if it has a finite diameter.
Definition 2.4.
The gain λ π ( s ) of a policy π , from starting state s 1 = s , is defined as the infinitehorizon undiscounted average reward given by λ π ( s ) =
E [lim T →∞ 1 T (cid:80) Ti =1 r ( s t , π ( s t )) | s 1 = s ] ,where s t is the state reached at time t , by executing policy π .
Lemma 2.1.
(Optimal gain for communicating MDPs).
For a communicating MDP M with diameter
D :1.
([20] Theorem 8.1.2, Theorem 8.3.2)
The optimal gain λ ∗ is state independent and isachieved by a deterministic stationary policy π ∗ , i.e. there exists a deterministic policy π ∗such that
[Equation] Here π ∗ is referred to as an optimal policy for MDP M .2.
([4], Theorem 4).
The optimal gain λ ∗ satisfies the following equations
[Equation]
where the vector h ∗ =
( h s ) s ∈S satisfies the following optimality equation
[Equation]
Here h ∗ is referred to as the bias vector of MDP M satisfying: max s h ∗ s − min s h ∗ s ≤ D .
Given the above definitions and results, we now formally define the Reinforcement Learning problemwe consider here.
The process proceeds in time steps t = 1 , 2 , . .
, T .
For agent p = 1 , 2 , . . . , n , eachstarts from a state s p 1 at time step t = 1 .
In the beginning of each time step t , agent p takes an actiona pt ∈ A and observes the reward r ( s pt , a pt ) , where s pt is the state of agent p at time t , and also the nextstate s pt +1 ∼ P ( s pt , a pt ) .
Here r and P are the reward function and the transition model respectivelyfor a communicating MDP M with diameter D .
Each learning agent knows the state-space S , theaction space A , as well as the rewards r ( s, a ) , ∀ s ∈ S , a ∈ A , for the underlying MDP, but not thetransition model P or the diameter D .
The agent can use the past observations across all the agents tolearn the underlying MDP model and decide future actions.
The goal for each agent is to maximizetotal reward (cid:80)
Tt =1 r ( s pt , a pt ) , or equivalently minimize the total regret over a time horizon T , definedas R ( T, M , n ) =
Tnλ ∗ − (cid:80) np
=1 (cid:80)
Tt =1 r ( s pt , a pt ) , where λ ∗ is the optimal gain of MDP M .
Remark 2.1.
Unlike the finite-horizon case, we assume that we know the reward functions as in[2].
Since without loss of generality, as long as the reward functions are bounded, we can have thesame bound on the regret.
This is because the unknown transition probability is the main source ofdifficulty in the analysis of the regret bound.
Notations We use S to denote the state space, where |S| = S , and A to denote the action space,with |A| =
A .
There are n agents interacting with the environment.
We use r ( s, a ) to denote thereward related to state-action pair ( s, a ) .
In finite horizon case, H is the horizon of each episode, ρis the initial state distribution for each episode.
s pt is the state of of agent p at time step t , and a ptis the action that agent p takes at time step t , where p = 1 , 2 , . . . , n .
In finite horizon case, we use
P ( s, a ) to refer to the transition probability vector related to state-action pair ( s, a ) , and use P s,aunder the infinite horizon case.
The policy π p is a mapping from the information set that agent p hasaccess to, to an action a in the action space.
Under finite horizon, we use s pkh to indicate the state ofagent p in episode k period h , and a pkh to indicate the action that agent p takes in episode k periodh .
We use P ∗ to denote the transition probability of the true MDP, λ ∗ to denote the optimal gain ofthe true MDP under infinite horizon case.
˜ λ
k is the optimal gain of the sampled MDP in epoch kunder infinite horizon case.
D is the upper bound of the diameter of the MDP considered.
For finitehorizon case, H pt denotes the information gathered by agent p before time step t , and H t = ∪ np =1 H ptis the history of observations prior to time step t across all agents.
For infinite horizon case, F pk is theinformation of agent p within and before epoch k , and F k = ∪ np =1 F pk is the information within andbefore epoch k across all agents.
Finite-Horizon Episodic Case
Algorithm Here we consider a concurrent version of the PSRL in [18].
At the beginning of eachepisode, each agent samples an MDP independently, where each one computes the optimal policy fortheir sampled MDP, and proceeds with this policy during the rest of the episode.
Then they gatherand update information at the end of each episode (See Algorithm 1).
There are n agents interactingin the environment, each episode is of length H , and the number of total time steps is T .
So there are
K = ⌈ T/H ⌉ episodes in the environment.
Intuition for ˜ O
( 1 √ n ) per-agent regret bound We obtain an ˜ O ( √ Tn )
Bayesian regret boundfor finite-horizon case, with ˜ O
( HS √ ATn ) for general prior distribution, and ˜ O
( H √ SATn ) for
Dirichlet prior distribution, where n is the number of agents interacting in the environment.
Note thatin the single-agent case, previous literature shows such regret bounds, ( ˜ O ( S √ AT ) for general priorin [13] and ˜ O ( √ SAT ) for Dirichlet prior in [18]).
This indicates that when n agents interactivelylearn in the same unknown environment over a time horizon of T, the bound scales in a non-linearway as if a single-agent learn over a time horizon of T ′ =
nT , which suggests the optimality ofour bounds here with respect to the number of agents n .
This is also a reflection of the power ofconcurrent learning within concurrent RL.
Improvement of Bayesian Regret Under Dirichlet Prior Distribution
The key analysis from˜ O ( HS √ AT ) to
˜ O
( H √ SAT ) follows the technical notes in [18], where we borrow a technicallemma on Gaussian-Dirichlet concentration bound ([16]).
With this, we are able to show that theregret contribution from the transition probability estimate can concentrate at a rate independent of S .
Finally we use a Sub-Gaussian tail bound to make the improvement.
The proof idea follows [18].
[-Data: prior distribution  , H p 
-]
3.1 Bayesian Regret Bound under General Prior Distribution
Theorem 3.1.
(Bayesian Regret Bound Under General Prior Distribution)
Let M ∗ be thetrue MDP distributed according to any prior distribution ϕ .
If the number of agents n satsifiesn ≤ O ( S 2 AT log( SAnT )) , Then the Bayesian regret for Algorithm (1) is bounded by
[Equation]
Proof Sketch: We first define a confidence set M k for each episode k as in [3], so that this setcontains both sampled MDP M k and true MDP M ∗ with high probability.
Then we decompose∆ kp into ∆ kp 1 ( M k ∈ M k , M ∗ ∈ M k ) + ∆ kp [ 1 ( M k / ∈ M k or M ∗ / ∈ M k )] .
Then we use thefact that when condition on history H ( k − 1) H , we have E [ 1 { M k / ∈ M k }|H ( k − 1) H ] =
E [ 1 { M ∗ / [-
-]∈
M k }|H ( k − 1) H ] ([13]), so that ∆ kp [ 1 ( M k / ∈ M k or M ∗ / ∈ M k )] is bounded by 2 H P ( M ∗ / ∈ M k ) .
As for the first part, namely ∆ kp 1 ( M k , M ∗ ∈ M k ) , we found that within the confidence intervals,∆ kp can be controlled with the desired upper bound. And for ∆ kp [ 1 ( M k / ∈ M k or M ∗ / ∈ M k )] , Algorithm
Data: prior distribution ϕ , H p 0 = ∅ , ∀ p ∈ [ n ] for episode k = 1 , 2 , . . . do for p = 1 : n do Sample MDP M kp ∼ ϕ ( ·|H ( k − 1) H ) ; Compute µ kp ∈ arg max µ V M kp µ, 1 ; for period h=1,...,H do take action a pkh = µ kp ( s pkh , h ) ; update H p ( k − 1) H + h
= H p ( k − 1) H + h − 1 ∪ { s pkh , a pkh , r pkh , s pk ( h +1) } end end
H kH =
∪ np =1 H pkH ; Update posterior distribution ϕ ( ·|H kH ) ; end since it’s bounded by 2 H P ( M ∗ / ∈ M k ) , we just need to show that P ( M ∗ / ∈ M k ) is small, which isindeed the case.
We decompose the regret into two parts, with one part negative and the other partpositive.
So we focus on bounding the positive part.
Then we rewrite the positive upper bound viathe Bellman operator, where one part is related to the concentration of reward estimate, and the otherpart is related to the concentration of transition probability estimate.
i.e. BayesRegret ( T, π, ϕ ) =
E [ Regret ( T, π, M ∗ ) | M ∗ ∼ ϕ ] = (cid:80)
⌈ T/H ⌉ k
= (cid:80) np =1 E
[ (cid:80) s ∈S ρ ( s )( V M ∗ π ∗ , 1 ( s ) − V M ∗ π kp , 1 ( s ))
| M ∗ ∼ ϕ ]
And we can write V M ∗ π ∗ , 1 ( s ) − V M ∗ ( s ) π kp , {+
+} =
V M ∗ π ∗ , 1 ( s ) − V M kp π kp , 1 ( s ) (cid:124) (cid:123)(cid:122)
(cid:125) ∆ opt kp + V M kp π kp , 1 ( s ) [Equation] (cid:124) [Equation]
(cid:125) ∆ [Equation] Conditionedupon data before the start of episode k , H ( k − 1) H , the true MDP M ∗ and the sampled M k areidentically distributed.
This means that E [∆ opt kp ] ≤ 0 for all k, p .
Therefore, we just need tobound (cid:80) ⌈ T/H ⌉ k = (cid:80) Pp =1 E [∆ conc kp |H
( k − 1) H ] .
For the convenience of notation, we write V kpkh =
V M kp π kp ,h .
Now we rewrite the regret via the Bellman operator, with w R ( x ) =
¯ r kp ( x ) − ˆ r k ( x ) , w Ph ( x ) =( P kp ( x ) − ˆ P k ( x ))
T V kpk,h +1 .
Then by some simple derivation we find that E [∆ conc kp |H ( k − 1) H ] ≤
E [ (cid:80)
Hh =1 | w R ( x pkh )
| + (cid:80)
Hh =1 | w Ph ( x pk,h +1 ) |H
( k − 1) H ] .
Based upon this, we prove that with highprobability, both the true MDP and the sampled MDPs are in the confidence sets constructed here.
And also conditional upon the event that both true MDP and sampled MDPs are in the confidence set,the total regret can be controlled.
3.2 Bayesian Regret Bound under Independent Dirichlet Prior Distribution
Theorem 3.2.
(Bayesian Regret Bound Under Dirichlet Prior Distribution)
Let M ∗ be the true
MDP distributed according to prior ϕ with independent Dirichlet prior over transition probabilities.
Then the Bayesian regret for Algorithm (1) is bounded by
[Equation]
Proof Sketch:
From [18], under Dirichlet prior, by a union bound at each x pkh =
( s, a ) on P ( s, a )and r ( s, a ) , we have that for each p , with probability at least 1 − 1 nT , E
[Equation]
By pigeonhole principle, when we sum over all the k and p , we can decompose the right hand side ofthe above inequality according to the number of times that each state-action pair were visited.
Forthose state-action pairs which were visited less than twice, since the reward function is within [0 , 1] ,the sum of these parts are upper bounded by 2 SA + 1 .
Lemma (3.1) in the below guarantees thatthe regret from transition probability estimation error can be bounded at a rate independent of S ,and Lemma (3.2) further provides a bound for the dominating Gaussian variable on the Dirichlet distribution.
In this way we can get an improved bound of ˜ O
( H √ SAnT ) from the case under generalprior distribution.
We leave the proof in the appendix.
Lemma 3.1.
(Gaussian-Dirichlet dominance [18]). For all fixed V ∈ [0 , 1] N , α ∈ [0 , ∞ ) N withα T 1 ≥ 2 , if X ∼ N ( α T V/α T 1 , 1 /α T 1 ) and Y = P T V for P ∼ Dirichlet ( α ) , then X ⪰ so Y . Here the order is in terms of stochastic optimism.
Lemma 3.2.
(Sub-Gaussian tail bounds [18]).
Let x 1 , . . .
, x n be independent samples from subGaussian random variables.
Then for any δ > 0 ,
[Equation]  Infinite-Horizon Case Algorithm Algorithm 2 combines the idea of Thompson sampling and double epoch for infinitehorizon case.
Similar to the finite-horizon episodic case, the agents sample MDP independentlyat the beginning of each epoch.
The difference is that unlike the finite-horizon case where eachepisode is of the same length, the infinite horizon adapt a "double epoch" strategy ([3], [2]), wherewe stop the current epoch and move on to the next one immediately as long as the number of visits tosome state-action pair at least doubles.
As in finite-horizon case, we can also get ˜ O
( 1 √ n ) per-agent
Bayesian regret bound as well.
For the similar reason as in the finite-horizon case, here we also havean improvement from ˜ O
( S √ ATn ) to ˜ O ( √ SATn ) if we condition on Independent Dirichlet priordistribution instead of on some general prior .
From Finite-Horizon To Infinite Horizon: Doubling Epoch [12] points out the necessity to have"phases" in infinite-horizon problem.
In infinite-horizon case, if the reward of one state is slightlylarger than the reward in the current state, the MDP tends to switch to the other state very easily.
But for finite-horizon setting, an optimistic algorithm would eventually stop considering switchesbecause eventually, the loss for switching would be assessed to be higher than the potential gain fromswitching.
For this reason, recalculating the optimistic policy in each round may lead to the algorithmto switch states frequently, which can lead to linear regret in the end.
As in [2], we borrow the idea ofdoubling epoch [3], where the next phase starts once the number of visits to some agent’s state-actionpair at least doubles.
Since the total number of visits to any state-action pairs can be no more thannT , the number of epochs is bounded by some constant times SA log( nT ) .
4.1 Bayesian Regret Bound Under General Prior Distribution
Theorem 4.1.
(Bayesian Regret Bound Under General Prior Distribution)
For any communicating
MDP M with S states, A actions, diameter D .
Then for T ≥ Ω( SA log 4 ( SATn )) .
Then
[Equation]
Proof Sketch: We first write the total regret as the sum of regrets over all the epochs.
We focus onbounding the regret within any fixed epoch.
Then we decompose the difference between the optimalgain of the true MDP and the reward functions as the sum of two parts: one part is the differencebetween the optimal gain of the true MDP and the optimal gain of the sampled MDP; the other partis the difference between the optimal gain of the sampled MDP and the reward function at somestate-action pair.
Then finally when we do a sum over all the epochs, we can get the desired regretbound.
As defined before, R ( T, M , n ) =
Tnλ ∗ − (cid:80) np
=1 (cid:80)
Tt =1 r ( s pt , a pt ) .
λ ∗ is the optimal gain of MDP M .
Algorithm (2) proceeds in epochs k = 1 , 2 , . .
, K , where K ≤ SA log( T ) .
Now we define
R k = ( τ k +1 − τ k ) nλ ∗ − (cid:80) τ
k +1 − 1 t
[Equation]
(cid:80) n p =1 r s p t ,a p t , where R k is the regret within epoch k .
So wecan write R k as R k = (cid:80) τ k +1 − 1 t = τ k
[Equation]
where ˜ λ k is the optimal gain of the sampled MDP ˜ M k .
From Lemma (2.1) we know that for anystate s , optimal policy ˜ π k for communicating MDP ˜ M k , action a = ˜ π k ( s ) , ˜ λ
k = r s,a + ˜ P Ts,a ˜ h − ˜ h s ,where ˜ P s,a = Q ks,a . So we have do Sample transition probability vectors: For each s, a generate sample probability vectors Q ks,a as follows:
Q a ∼ | do do
Play action a pt = ˜ π
k Observe Update F pk = F pk ∪ { s pt , a pt , r pt , s pt +1 } end If N t +1 s pt ,a pt ≥ 2 N τ k s pt ,a pt for some p , then set τ k +1 =
t + 1 and break epoch .
F k = ∪ np =1 F pk Update posterior distribution ϕ ( ·|F k ) ; end end τ k +1 − 1 (cid:88) t = τ k n (cid:88) p = (˜ λ k − r s pt ,a pt ) = τ k +1 − 1 (cid:88) t =
τ k n (cid:88) p = ( ˜ P s pt ,a pt − P s pt ,a pt + P s pt ,a pt − 1 s pt ) T ˜ h = τ k +1 − 1 (cid:88) t =
τ k n (cid:88) p = ( ˜ P s pt ,a pt − P s pt ,a pt ) T ˜ h + τ k +1 − 1 (cid:88) t = τ k n (cid:88) p
=
( P s pt ,a pt − 1 s pt ) T ˜ h
= τ k +1 − 1 (cid:88) t =
τ k n (cid:88) p =
( Q ks pt ,a pt − P s pt ,a pt ) T ˜ h + τ k +1 − 1 (cid:88) t =
τ k n (cid:88) p
=
( P s pt ,a pt − 1 s pt ) T ˜ h
We will prove the theorem in the appendix by bounding the two parts of the sum above.4.2 Bayesian Regret Bound Under Dirichlet Prior
Theorem 4.2.
(Bayesian Regret Bound Under Dirichlet Prior)
Let M be the true MDP distributedaccording to prior ϕ with any independent Dirichlet prior over transition probabilities.
Then the
Bayesian regret for Parallel Infinite-Horizon posterior sampling MDP algorithn 2 is bounded by
[Equation]
Proof Sketch: We use Lemma (4.2) to prove the concentration bound between sampled transitionprobabilities and the expectation of the true probabilities conditional on the current data.
And notethat one key property of Thompson sampling is that conditional upon the current data, the true MDPand the sampled MDP are identically distributed.
And similar to the case under general distribution,we use Azuma-Hoeffding inequality (4.1) to bound the term with respect to the difference of trueprobability distribution and the indicator of states.
Lemma 4.1.
( Azuma-Hoeffiding inequality [9]) Let X 1 , X 2 , . . . be a martingale difference sequencewith | X i | ≤ c for all i . Then for all ϵ > 0 and n ∈ N [Equation]
Lemma 4.2. (Transition concentration). Let ˜ P s,a = Q ks,a in Algorithm 2. Let ˆ P s,a = E
[ P s,a |H k 1 ] ,then for any independent prior over rewards with r ∈ [0 , 1] , additive sub-Gaussian noise and anindependent Dirichlet prior over transitions, then for any state-action pair ( s, a ) in episode k , withprobability 1 − δ ,
[Equation]
The proofs of Lemma (4.2) and the theorem are provided in the appendix.
Simulation
In this section, we describe our empirical study of concurrent Thompson Sampling on a Markov
Decision Process with a Dirichlet transition function and a normal reward function for both finitehorizon and infinite-horizon cases.
In particular, we train the concurrent Thompson Sampling agentson randomly sampled MDPs and evaluate the regrets of these agents at the end of each episode/epochon a fixed number of environments sampled from the ground truth distribution.
For the finite-horizon case, we set the number of states of the MDP to S = 5 and actions A = 5 .
Thereward function of each state-action-next state tuple is sampled from the standard normal distribution
N (0 , 1) , and the number of episodes is set to K = 30 and the horizon of each episode to H = 75 .
The time horizon is then T = KH
= 2250 .
For each number of agents, we run the finite-horizon Thompson Sampling algorithm and evaluate the regret for each agent after each episode. After Ttime steps, we compute the average regret of all the episodes to get the Bayesian regret. Similarly, for the infinite-horizon case, the state-space of the environment is also set to S =
A = 5with the reward function for each state-action-next state tuple sampled from N (0 , 1) . The timehorizon T is set to T = 20 , 000 and regret is calculated at the end of each epoch. The Bayesian regretis then computed from the regret at the end of each epoch at the end of T time steps.
Under both settings, the policy at each episode or epoch is computed using value iteration fromtransition probabilities sampled from a Dirichlet distribution with parameters being the number ofvisits to each of the state, action, and next states tuples. The discount for value iteration is set to 0.99and tolerance 0.01. All experiments were run using CPUs, either through a desktop computer or Google Cloud Platform.
Due to page limitation, we put the experiment results in the appendix.
Conclusion
We have extended PSRL from a single agent to multiple agents, where the agents can concurrent witheach other to learn in parallel and share data with each other.
We consider two settings: both finitehorizon episodic MDP and infinite-horizon MDP.
For state space size S , action space size A , totalaccumulated time horizon T , episode horizon H , we show a ˜ O ( HS (cid:113) ATn ) per-agent Bayesian regretbound under general prior distribution; while especially for independent Dirichlet prior distribution,we show that a tighter bound can be achieved as ˜ O
( H (cid:113) SATn ) .
Likewise, under infinite horizon case,for the general prior distribution, we follow the idea in [2] to achieve a high-probability worst caseper-agent regret bound as ˜ O
( DS (cid:113) ATn ) , which is sufficient for ˜ O
( DS (cid:113) ATn ) per-agent Bayesianregret bound; and for the infinite-horizon case under dirichlet prior distribution, we combine the ideaof PSRL and double-epoch sampling, so as to obtain a tighter ˜ O
( D (cid:113) ATSn )
Bayesian regret bound,with D being the diameter of MDP considered here.
So far we provide a theoretical upper boundfor the concurrent learning under posterior sampling RL setting, and the multiplicative constants inthe bound were just properly chosen but not tuned in an optimistic way.
There is still more work tobe done if we want to put the algorithm into application, including the production of autonomousdriving chip design, etc.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions
and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]