Society of Agents: Regret Bounds of Concurrent Thompson Sampling
Abstract
We consider the concurrent reinforcement learning problem where n agents simultaneously learn to make decisions in the same environment by sharing experience with each other.
Existing works in this emerging area have empirically demonstrated that Thompson sampling (TS) based algorithms provide a particularly attractive alternative for inducing cooperation, because each agent can independently sample a belief environment (and compute a corresponding optimal policy) from the joint posterior computed by aggregating all agents’ data , which induces diversity in exploration among agents while beneﬁting shared experience from all agents.
However, theoretical guarantees in this area remain under-explored; in particular, no regret bound is known on TS based concurrent RL algorithms.
In this paper, we ﬁll in this gap by considering two settings.
In the ﬁrst, we study the simple ﬁnite-horizon episodic RL setting, where TS is naturally adapted into the concurrent setup by having each agent sample from the current joint posterior at the beginning of each episode.
We establish a ˜ O ( HS q
ATn ) per-agent regret bound, where H is the horizon of the episode, S is the number of states, A is the number of actions, T is the number of episodes and n is the number of agents.
In the second setting, we consider the inﬁnite-horizon RL problem, where a policy is measured by its long-run average reward.
Here, despite not having natural episodic breakpoints, we show that by a doubling-horizon schedule, we can adapt
TS to the inﬁnite-horizon concurrent learning setting to achieve a regret bound of ˜
O ( DS p ATn ) , where D is the standard notion of diameter of the underlying MDP and T is the number
of timesteps.
Note that in both settings, the per-agent regret decreases at an optimal rate of ⇥ ( 1 p n ) , which manifests the power of cooperation in concurrent RL.
Introduction
In concurrent reinforcement learning [26], many agents interact with (different instances of) the same environment simultaneously and share data with each other in order to jointly learn an optimal policy for the environment.
This learning scenario has important applications for large-scale automated sequential decision making systems.
For instance, in recommendation systems [1]) (e.g. movie recommendation, personalized ads recommendation, personalized news/content recommendation), a company often has a large number of servers – sometimes located in different geographical regions – that serve the entire population of the customer base.
As such, an incoming customer is randomly routed to a server (i.e. an agent), which learns to make recommendation decisions in conjunction with
[Table] [Table]
[Table] [Table]
all the other servers, and the performance judgement is made on the entire set of servers.
Another example is in concurrent personalized promotions, where each agent is actively learning to provision personalized promotions on a product, along with other agents, each of which is performing the same personalized promotions task on a similar product.
As surprising as it sounds, this example has already been done in practice, where Arena Technologies † , a cutting-edge enterprise AI solution provider, deploys sophisticated reinforcement learning agents to provision personalized promotions/pricing to an ensemble of beverages and other consumer packaged goods.
A third example – which exists as a concept but not yet happened– of concurrent RL is controlling a team of autonomous vehicles, which, as succinctly summarized in [9], concerns a scenario where “each agent manages a single vehicle, and again, the agents learn from each other as data is gathered.
The goal could be to optimize a combination of metrics, such as fuel consumption, safety, and satisfaction of transportation objectives.
Exploratory actions play an important role, and structured diversity of experience may greatly accelerate learning.
" This application is particularly relevant and presents an unique opportunity for the emerging era of autonomous driving.
Other concurrent RL applications include robotics ([10], [26]) and biology([27]).
An important opportunity in the concurrent RL setup –in comparison to the traditional single-agent
RL setup – lies in coordinated exploration: since those agents can share their own learning experience with others, the team as a whole should achieve better overall regret performance (or equivalently better per-agent regret performance) if they intelligently coordinate the exploration efforts.
In particular, different agents may want to diversify their exploratory actions so as to quickly reduce the team’s uncertainty over the landscape of the environment in which they all operate.
By sharing the resulting data with each other, there is hope that the team would–in a environment-interaction time frame much less than that from a single-agent setting–achieve the same level of performance.
When it comes to exploration, a substantial amount of literature has been devoted to designing (provably) efﬁcient algorithms in the single-agent RL setting.
One prominent class of algorithms, based on upper conﬁdence bounds, is known as UCRL (and its variants); (see[13] [4]) and many references citing them and cited by them.
The UCRL family of algorithms embodies the important philosophy of "optimism in face of uncertainty" in their design and enjoys optimal theoretical regret guarantees (along with elegant analysis) in various settings, including both ﬁnite-horizon episodic RL and inﬁnite-horizon average-reward RL.
As such, it is perhaps not surprising that UCRL algorithms are the ﬁrst family of algorithms that have been adapted to the concurrent RL settings [11] and [23].
In particular, concurrent UCRL has been analyzed for the sample complexity performance measure (i.e. how many samples are needed to learn an ✏ -optimal policy) under both ﬁnite action space setting and inﬁnite action space setting.
More speciﬁcally, [11] provided a high-probability bound of ˜ O
( S 2 A ✏ 3 + SAn ✏ ) for the sample complexity with n agents interacting in the environment.
Their algorithm was extended from MBIE(see [28]), with a single agent performing concurrent RL across a set of n inﬁnite-horizon MDPs.
The results there show that with sharing samples from copies of the same MDP a linear speedup in the sample complexity of learning can be achieved.
But no regret bound was derived there for concurrent RL.
[23] designed an algorithm which extended concurrent exploration to continuous state-action spaces and allowed smooth generalization between concurrent MDPs under inﬁnite horizon setting.
Likewise, the authors also explored the scenario where one single agent interacts with concurrent MDPs at the same time.
Additionally, they deﬁne an error metric, called TCE (total cost of exploration), and showed an ˜ O (( 1 ✏ + n ) N ( d )) bound for
TCE, where N ( d ) is a constant representing the covering number of the MDP.
This is sufﬁcient for an ˜ O
(( 1 ✏ 2 + n ✏ ) N ( d )) bound for sample complexity.
However, these sample-complexity guarantees notwithstanding, concurrent UCRL algorithms suffer from the critical disadvantage of no coordinated exploration: since the upper conﬁdence bounds computed by aggregating all agents’ data are the same the entire team, each agent would follow the exact same policy, thereby yielding no diversity in exploration.
This point has been observed in [9, 8], which has advocated for designing concurrent RL algorithms based on Thompson sampling [20],[21], [29], [14] [17],[18], also known as posterior sampling reinforcement learning (PSRL).
Unlike UCRL,
PSRL manifests the philosophy of "probabilistic optimism in face of uncertainty", where the agent is uncertain about the transition probabilities and reward functions, and hence at the start of each episode, the agent would sample an MDP from the current posterior distribution based on data gathered up to that time and makes decisions according to the policy that is optimal with respect to the sampled MDP.
[6] and [22] have provided good motivating examples for empirical evidence.
In fact, not only PSRL is well-known to outperform UCRL empirically in single-agent RL setting, it also – by its very design – lends itself well to concurrent RL settings: each agent can sample from a posterior distribution and explores accordingly, which induces diversity in a coordinated way.
Building on this insight, [9] and [8] proposed sophisticated Thompson sampling variants that demonstrated good empirical performance.
Despite these remarkable empirical results, to date, no theoretical guarantee has been provided on any concurrent Thompson sampling algorithm, simple or sophisticated.
Our goal in this paper is to make initial inroads on the theoretical front and attempt to prove the ﬁrst bounds on simple-but-natural concurrent PSRL algorithms.
1.1 Our Contributions
We establish per-agent regret guarantees for concurrent PSRL algorithms in two settings: one in ﬁnite-horizon episodic setting and the other in inﬁnite-horizon average reward setting.
In the ﬁrst setting, TS is naturally adapted into the concurrent setup by having each agent sample from the current joint posterior at the beginning of each episode.
We establish a ˜ O ( HS q
ATn ) per-agent regret bound, where H is the horizon of the episode, S is the number of states, A is the number of actions,
T is the number of episodes and n is the number of agents.
This regret bound further improves to ˜ O
( H q
SATn ) under the special Dirichlet prior.
Note that when n = 1 , both regret bounds are tight(see [16], [21]).
These two regret bounds further highlight that the per-agent regret decreases at an optimal rate of ⇥ ( 1 p n ) and hence manifests the power of cooperation in concurrent RL: as a team, the total regret achieved is ˜ O ( HS p nAT ) (and ˜ O
( H p nSAT ) under Dirichlet prior), which is minimax optimal, because that is the optimal regret performance (up to log factors) achievable by a single-agent for nT episodes that can adapt at each episode † .
In the second setting, we consider the inﬁnite-horizon RL problem, where a policy is measured by its long-run average reward.
Here, despite not having natural episodic breakpoints, we show that by a doubling-horizon schedule, we can adapt TS to the inﬁnite-horizon concurrent learning setting to achieve a regret bound of ˜ O ( DS p ATn ) , where D is the standard notion of diameter of the underlying MDP and T is the number of timesteps.
Here, we again see that the per-agent regret decreases at an optimal rate of ⇥ ( 1 p n ) and the fact that a single learning trajectory does not prevent the team of agents from efﬁcient concurrent reinforcement learning.
Due to space limitation, we’ll include simulation plots in the appendix.
Problem Formulation
We consider the Bayesian regret bound of concurrent Thompson Sampling of Markov decision process in ﬁnite-horizon episodic setting and inﬁnite-horizon setting.
In both settings, we provide bounds on the general prior distributions and Dirichlet prior distributions for concurrent Thompson
Sampling of the MDPs.
2.1 Finite-Horizon Episodic Setting
Formally, we consider the problem of learning to optimize a random (parallel) ﬁnite-horizon episodic
Markov decision process M ⇤ =
( S , A , R ⇤ , P ⇤ , H, ⇢ ) over repeated episodes of interaction.
Here
S = { 1 , 2 , . . .
, S } is the state space,
A = { 1 , 2 , . . .
, A } is the action space, H is the horizon of each episode, and ⇢ is the initial state distribution for each episode.
At the start of each episode k ( k  1) , each agent will receive the history across all the agents before period k .
And at the start of episode 1 , all the agents don’t have any information regarding the history.
For every period h
= 1 , 2 , . .
, H within an episode, each agent p observes state s ph 2 S , selects action a ph 2 A , receives a reward r h ⇠ R ⇤ ( s ph , a ph ) , where the reward functions r h are assumed to be in [0 , 1] , and transitions to a new state s ph +1 ⇠ P ⇤ ( s ph , a ph ) .
At time t , agent p chooses a policy ⇡ p , which is a mapping from the information that she has, i.e. H pt , to action a 2 A chosen by agent p .
For each agent p , MDP M and policy ⇡ p , the state-action value function for each period h is deﬁned as Q M ⇡ p ,h ( s, a ) = E ⇡ p ,M [
P
Hj =
h ¯ r M ( s j , a j ) | s h
= s, a h
= a ] , where ¯ r M ( s, a ) =
E [ r | r ⇠ R M ( s, a )] .
The subscript ⇡ p indicates that the actions of agent p over periods h + 1 , . .
, H are chosen according to policy ⇡ p .
Let V M ⇡ p ,h ( s ) :
= Q M ⇡ p ,h ( s, ⇡ p ( s, h )) .
We say a policy ⇡ M is optimal for MDP M if ⇡ M 2 arg max ⇡ V M ⇡ ,h ( s ) for all s 2 S and h = 1 , 2 , . . .
, H .
Let H t
=
[ np =1 H pt denote the history of observations made by time t (prior to and including time step t ) across all the agents p = 1 , 2 , . . . , n , where H pt is the history of agent p by time step t (include time t ), with t = 1 , 2 , . .
, T and T is the total time horizon.
So t = ( k  1)
H
+ h is the time step at period h within episode k .
We use s pkh to denote the state of agent p at time t =
( k  1) H + h , i.e. episode k period h .
The RL algorithm we consider here is a deterministic sequence { ⇡ k : k
= 1 , 2 , . . .
} of functions, each mapping H ( k  1) H (i.e. the history of observations prior to episode k ) to a probability distribution ⇡ k ( H ( k  1) H ) over policies, from which each agent samples a policy ⇡ kp for the k -th episode.
The regret for agent p in episode k is  kp :=
P s 2 S ⇢ ( s )( V M ⇤ ⇡ ⇤ , 1 ( s )  V M ⇤ ⇡ kp , 1 ( s )) .
The overall regret in episode k is  k :=
P np =
P s 2 S ⇢ ( s )( V M ⇤ ⇡ ⇤ , 1 ( s )  V M ⇤ ( s ) ⇡ kp , 1 ) , where ⇡ ⇤ =
⇡ M ⇤ .
The regret incurred up to time T for policy ⇡ is Regret ( T, ⇡ , M ⇤ ) =
P d T/H e k =  k .
The Bayesian expected regret for M ⇤ distributed according to the prior  is deﬁned as BayesRegret ( T, ⇡ ,  ) :=
E [ Regret ( T, ⇡ , M ⇤ ) | M ⇤ ⇠  ]
2.2 Inﬁnite-Horizon Setting
We follow the setting in [3] and consider the case where different agents can cooperate.
As in [3] we consider communicating MDPs with ﬁnite diameter.
We ﬁrst deﬁne ’diameter’, ’communicating’ and other related terms.
Deﬁnition 2.1.
A deterministic policy ⇡ : S !
A is a mapping from state space S to action space A .
Deﬁnition 2.2.
Diameter D ( M ) of an MDP M is deﬁned as the minimum time required to go from one state to another in the MDP using some deterministic policy: D ( M ) =
max s
= s 0 ,s,s 0 2 S , ⇡ : S !
A T ⇡ s !
s 0 .
Here T ⇡ s !
s 0 is the expected number of steps it takes to reach state s 0 when starting from state s and using policy ⇡ .
Deﬁnition 2.3.
An MDP is communicating if and only if it has a ﬁnite diameter.
Deﬁnition 2.4.
The gain  ⇡ ( s ) of a policy ⇡ , from starting state s 1 = s , is deﬁned as the inﬁnite horizon undiscounted average reward given by  ⇡ ( s ) =
E [lim T !
[Equation]
where s t is the state reached at time t , by executing policy ⇡ .
Lemma 2.1.
(Optimal gain for communicating MDPs).
For a communicating MDP M with diameter
D we have: 1.
([24] Theorem 8.1.2, Theorem 8.3.2)
The optimal gain  ⇤ is state independent and is achieved by a deterministic stationary policy ⇡ ⇤ , i.e. there exists a deterministic policy ⇡
⇤ such that  ⇤ = max s 0 2 S max ⇡  ⇡ ( s 0 ) =
⇡ ⇤ ( s ) , 8 s 2 S ; 2.
([5], Theorem 4).
The optimal gain  ⇤ satisﬁes the following equations  ⇤ =
min h 2 R S max s,a r ( s, a ) + P T s,a h  h s = max a r ( s, a ) +
P Ts,a h ⇤  h
⇤ s 8 s where the vector h
⇤ =
( h s ) s 2 S satisﬁes the following optimality equation
h ⇤ +  ⇤ e = max a 2 A ( r ( s, a ) + P Ts,a h ⇤ ) .
Here h ⇤ is referred to as the bias vector of MDP M satisfying: max s h ⇤ s  min s h
⇤ s  D .
Given the above deﬁnitions and results, we now formally deﬁne the Reinforcement Learning problem we consider here.
The process proceeds in time steps t = 1 , 2 , . .
, T .
For agent p = 1 , 2 , . . . , n , each starts from a state s p 1 at time step t = 1 .
In the beginning of each time step t , agent p takes an action a pt 2 A and observes the reward r ( s pt , a pt ) , where s pt is the state of agent p at time t , and also the next state s pt +1 ⇠ P ( s pt , a pt ) .
And again the reward function r is assumed to be in [0 , 1] .
Here r and P are the reward function and the transition model respectively for a communicating
MDP M with diameter D .
Each learning agent knows the state-space S , the action space A , as well as the rewards r ( s, a ) , 8 s 2 S , a 2 A , for the underlying MDP, but not the transition model P or the diameter D .
The agent can use the past observations across all the agents to learn the underlying MDP model and decide future actions.
The goal for each agent is to maximize total reward
P
Tt =1 r ( s pt , a pt ) , or equivalently minimize the total regret over a time horizon T , deﬁned as
R ( T, M , n ) =
Tn  ⇤
P np =
P
Tt =1 r ( s pt , a pt ) , where  ⇤ is the optimal gain of MDP M .
Remark 2.1.
Unlike the ﬁnite-horizon case, we assume that we know the reward functions as in [3].
Since without loss of generality, as long as the reward functions are bounded, we can have the same bound on the regret.
This is because the unknown transition probability is the main source of difﬁculty in the analysis of the regret bound.
Notations We use S to denote the state space, where |S| = S , and A to denote the action space, with |A| =
A .
There are n agents interacting with the environment.
We use r ( s, a ) to denote the reward related to state-action pair ( s, a ) .
In ﬁnite horizon case, H is the horizon of each episode, ⇢ is the initial state distribution for each episode.
s pt is the state of of agent p at time step t , and a pt is the action that agent p takes at time step t ,
where p = 1 , 2 , . .
. , n .
In ﬁnite horizon case, we use
P ( s, a ) to refer to the transition probability vector related to state-action pair ( s, a ) , and use P s,a under the inﬁnite horizon case.
The policy ⇡ p is a mapping from the information set that agent p has access to, to an action a in the action space.
Under ﬁnite horizon, we use s pkh to indicate the state of agent p in episode k period h , and a pkh to indicate the action that agent p takes in episode k period h .
We use P ⇤ to denote the transition probability of the true MDP,  ⇤ to denote the optimal gain of the true MDP under inﬁnite horizon case.
˜
k is the optimal gain of the sampled MDP in epoch k under inﬁnite horizon case.
D is the upper bound of the diameter of the MDP considered.
For ﬁnite horizon case, H pt denotes the information gathered by agent p before time step t , and H t =
[ np =1 H pt is the history of observations prior to time step t across all agents.
For inﬁnite horizon case, F p k is the information of agent p within and before epoch k , and F k
=
[ np =1 F pk is the information within and before epoch k across all agents.
Finite-Horizon Episodic Case
Algorithm Here we consider a concurrent version of the PSRL in [21].
At the beginning of each episode, each agent samples an MDP independently, where each one computes the optimal policy for their sampled MDP, and proceeds with this policy during the rest of the episode.
Then they gather and update information at the end of each episode (See Algorithm 1).
There are n agents interacting in the environment, each episode is of length H , and the number of total time steps is T .
So there are
K = d T/H e episodes in the environment.
Intuition for ˜ O
( 1 p n ) per-agent regret bound We obtain an ˜ O ( p Tn )
Bayesian regret bound for ﬁnite-horizon case, with ˜ O ( HS p ATn ) for general prior distribution, and ˜ O
( H p SATn ) for
Dirichlet prior distribution, where n is the number of agents interacting in the environment.
Note that in the single-agent case, previous literature shows such regret bounds, ( ˜ O ( S p AT ) for general prior in [16] and
˜ O
( p SAT ) for Dirichlet prior in [21]).
This indicates that when n agents interactively learn in the same unknown environment over a time horizon of T, the bound scales in a non-linear way as if a single-agent learn over a time horizon of T 0 =
nT , which suggests the optimality of our bounds here with respect to the number of agents n .
This is also a reﬂection of the power of concurrent learning within concurrent RL.
Improvement of Bayesian Regret Under Dirichlet Prior Distribution
The key analysis from ˜
O ( HS p ATn ) to
˜ O
( H p SATn ) follows the technical notes in [21], where we borrow a technical lemma on Gaussian-Dirichlet concentration bound ([19]).
With this, we are able to show that the regret contribution from the transition probability estimate can concentrate at a rate independent of S .
Finally we use a Sub-Gaussian tail bound to make the improvement.
The proof idea follows [21].
Decomposition of the Regret: Note that we have
[Equation] [Equation]
And we can write
[Equation]
Data: prior distribution
[ n ] https://www.overleaf.com/project/628438b2462076a6e40eafa episode do do
Sample
Compute do take end end end
Note that conditioned upon any data H ( k  1) H , the true MDP M ⇤ and the sampled M kp are identically distributed.
This means that E [  opt kp ]  0 for all k, p .
Therefore, we just need to bound
P d T/H e k =
P
Pp =1 E
[  conc kp |H ( k  1) H ] .
For the convenience of notation, we write V kpkh =
V M k,p ⇡ kp ,h .
Now we rewrite the regret via the Bellman operator, with w R ( x ) =
¯ r kp ( x )  ˆ r k ( x ) , w Ph ( x ) = ( P kp ( x )  ˆ P k ( x )) T V kpk,h +1 :
E [  conc kp |H ( k  1) H ] = E [(¯ r kp  ¯ r ⇤ ) ( x pk 1 ) + P
[Equation]
( x pk 1 ) T V kpk 2  P ⇤ ( x pk 1 ) V ⇤ k 2 |H ( k  1) H ] = E [(
[Equation]
r
[Equation]
r ⇤ )
[Equation]
x pk 1 ) + ( P
[Equation]
3.1 Bayesian Regret Bound under General Prior Distribution
Theorem 3.1.
(Bayesian Regret Bound Under General Prior Distribution)
Let M ⇤ be the true MDP distributed according to any prior distribution  .
If the number of agents n satsiﬁes n  O ( S 2 AT log( SAnT )) , Then the Bayesian regret for Algorithm (1) is bounded by
[Equation]
Proof Sketch: We ﬁrst deﬁne a conﬁdence set M k for each episode k as in [4], so that this set contains both sampled MDP M k and true MDP M ⇤ with high probability.
Then we decompose  kp into  kp 1 ( M k 2 M k , M ⇤ 2 M k )
+  kp [ 1 ( M k / 2 M k or M ⇤ / 2 M k )] .
Then we use the fact that when condition on history
H ( k  1) H , we have E [ 1 { M k / 2 M k }|H
( k  1) H ] =
E [ 1 { M ⇤ /
M k }|H ( k  1) H ] ([16]), so that  kp [ 1 ( M k / 2 M k or M ⇤ / 2 M k )] is bounded by 2 H P ( M ⇤ / 2 M k ) .
As for the ﬁrst part, namely  kp 1 ( M k , M ⇤ 2 M k ) , we found that within the conﬁdence intervals,  kp can be controlled with the desired upper bound.
And for  kp [ 1 ( M k / 2 M k or M ⇤ / 2 M k )] , since it’s bounded by 2 H P ( M ⇤ / 2 M k ) , we just need to show that P ( M ⇤ / 2 M k ) is small, which is indeed the case.
We decompose the regret into two parts, with one part negative and the other part positive.
So we focus on bounding the positive part.
Then we rewrite the positive upper bound via the Bellman operator, where one part is related to the concentration of reward estimate, and the other part is related to the concentration of transition probability estimate.
i.e. BayesRegret ( T, ⇡ ,  ) =
E [ Regret ( T, ⇡ , M ⇤ ) | M ⇤ ⇠  ] =
[Equation]
And we can write V M ⇤ ⇡
⇤ , 1 ( s )  V M ⇤ ( s ) ⇡ kp , 1 =
V M ⇤ ⇡ ⇤ , 1 ( s )  V M kp ⇡ kp , 1 ( s ) | {z }  opt kp
+ V M kp ⇡ kp , 1 ( s )  V M ⇤ ⇡ kp , 1 ( s ) | {z }  conc kp .
Conditioned upon data before the start of episode k , H ( k  1) H , the true MDP M ⇤ and the sampled M k are identically distributed.
This means that E [  opt kp ]  0 for all k, p .
Therefore, we just need to bound
P d T/H e k =
P
Pp =
[Equation]
E [  conc kp |H
( k  1) H ] .
For the convenience of notation, we write V kpkh =
V M kp ⇡ kp ,h .
Now we rewrite the regret via the Bellman operator, with w R ( x ) =
¯ r kp ( x )  ˆ r k ( x ) , w Ph ( x ) =
( P kp ( x )
ˆ P k ( x ))
T V kpk,h +1 .
Then by some simple derivation we ﬁnd that E
[  conc kp |H ( k  1) H ] 
E [
P
Hh =
[Equation]
|
[Equation] [Equation]
P
Hh =
[Equation]
| w Ph ( x pk,h +1 ) |H
( k  1) H ] .
Based upon this, we prove that with high probability, both the true MDP and the sampled MDPs are in the conﬁdence sets constructed here.
And also conditional upon the event that both true MDP and sampled MDPs are in the conﬁdence set, the total regret can be controlled.
3.2 Bayesian Regret Bound under Independent Dirichlet Prior Distribution
Theorem 3.2.
(Bayesian Regret Bound Under Dirichlet Prior Distribution)
Let M ⇤ be the true
MDP distributed according to prior  with independent Dirichlet prior over transition probabilities.
Then the Bayesian regret for Algorithm (1) is bounded by
BayesRegret (
[Equation]
Proof Sketch:
From [21], under Dirichlet prior, by a union bound at each x pkh =
( s, a ) on P ( s, a ) and r ( s, a ) , we have that for each p , with probability at least 1  1 nT ,
E
[Equation]
By pigeonhole principle, when we sum over all the k and p , we can decompose the right hand side of the above inequality according to the number of times that each state-action pair were visited.
For those state-action pairs which were visited less than twice, since the reward function is within [0 , 1] , the sum of these parts are upper bounded by 2 SA + 1 .
Lemma (D.5) guarantees that the regret from transition probability estimation error can be bounded at a rate independent of S , and Lemma (D.4) further provides a bound for the dominating Gaussian variable on the Dirichlet distribution.
In this way we can get an improved bound of ˜ O
( H p SAnT ) from the case under general prior distribution.
We leave the proof in the appendix.
Inﬁnite-Horizon Case
Algorithm Algorithm 2 combines the idea of Thompson sampling and double epoch for inﬁnite horizon case.
Similar to the ﬁnite-horizon episodic case, the agents sample MDP independently at the beginning of each epoch.
The difference is that unlike the ﬁnite-horizon case where each episode is of the same length, the inﬁnite horizon adapt a "double epoch" strategy ([4], [3]), where we stop the current epoch and move on to the next one immediately as long as the number of visits to some state-action pair at least doubles.
As in ﬁnite-horizon case, we can also get ˜ O
( 1 p n ) per-agent
Bayesian regret bound as well.
For the similar reason as in the ﬁnite-horizon case, here we also have an improvement from ˜ O
( S p ATn ) to ˜ O
( p SATn ) if we condition on Independent Dirichlet prior distribution instead of on some general prior .
From Finite-Horizon To Inﬁnite Horizon: Doubling Epoch [15] points out the necessity to have "phases" in inﬁnite-horizon problem.
In inﬁnite-horizon case, if the reward of one state is slightly larger than the reward in the current state, the MDP tends to switch to the other state very easily.
But for ﬁnite-horizon setting, an optimistic algorithm would eventually stop considering switches because eventually, the loss for switching would be assessed to be higher than the potential gain from switching.
For this reason, recalculating the optimistic policy in each round may lead to the algorithm to switch states frequently, which can lead to linear regret in the end.
As in [3], we borrow the idea of doubling epoch [4], where the next phase starts once the number of visits to some agent’s state-action pair at least doubles.
Since the total number of visits to any state-action pairs can be no more than nT , the number of epochs is bounded by some constant times SA log( nT ) .
do
Sample transition probability generate sample probability vectors as follows:
[Equation]
[Equation]
Compute policy ˜ ⇡
as gain
MDP ˜ M using sample set { Q a
Execute policy ˜ ⇡ k : for time steps t = ⌧ k , ⌧ k +
, until break epoch do for agent p
. , n do (
Play action a pt =
˜ ⇡ k ( s pt ) .
Observe the transition to the next state s pt +1 .
Update F pk =
} end
If N t +1 s pt ,a pt  2 N ⌧ k s pt ,a pt for some p, (i.e. for some ( s pt , a pt ) ), then set ⌧ k +1 = t + 1 and break epoch .
F
= [ np =1 F pk Update posterior distribution
·|F
) ; end end
Theorem 4.1.
(Bayesian Regret Bound Under General Prior Distribution)
For any communicating
MDP M with S states, A actions, diameter D .
Then for T  ⌦ ( SA log 4 ( SATn )) .
Then
BayesRegret ( T, M , n,  )  ˜ O ( DS p ATn )
Proof Sketch: We ﬁrst write the total regret as the sum of regrets over all the epochs.
We focus on bounding the regret within any ﬁxed epoch.
Then we decompose the difference between the optimal gain of the true MDP and the reward functions as the sum of two parts: one part is the difference between the optimal gain of the true MDP and the optimal gain of the sampled MDP; the other part is the difference between the optimal gain of the sampled MDP and the reward function at some state-action pair.
Then ﬁnally when we do a sum over all the epochs, we can get the desired regret bound.
As deﬁned before, R ( T, M , n ) =
Tn  ⇤
P np =
P
Tt =1 r ( s pt , a pt ) .
⇤ is the optimal gain of MDP M .
Algorithm (2) proceeds in epochs k = 1 , 2 , . .
, K , where K  SA log( T ) .
Now we deﬁne R k =
[Equation]
p =1 r s pt ,a pt , where R k is the regret within epoch k .
So we can write R k as R k =
[Equation]
where ˜  k is the optimal gain of the sampled MDP ˜ M k .
From Lemma (2.1) we know that for any state s , optimal policy ˜ ⇡ k for communicating MDP ˜ M k , action a = ˜ ⇡ k
( s ) , ˜
k
= r s,a + ˜ P Ts,
a ˜ h  ˜ h s , where ˜ P s,a = Q ks,a .
So we have
⌧ k +1
X t = ⌧ k n
X p
=
(˜  k  r s pt ,a pt ) =
⌧ k +1
X t = ⌧ k n
X p
=
( ˜ P s pt ,a pt  P s pt ,a pt + P s pt ,a pt  1 s pt ) T ˜
h
=
⌧ k +1
X t = ⌧ k n
X p
=
( ˜ P s pt ,a pt  P s pt ,a pt ) T ˜ h + ⌧ k +1
X t = ⌧ k n
X p
=
( P s pt ,a pt  1 s pt ) T ˜ h
= ⌧ k +1
X t = ⌧ k n
X p
=
( Q ks pt ,a pt  P s pt ,a pt ) T ˜ h + ⌧ k +1
X t = ⌧ k n
X p
=
( P s pt ,a pt  1 s pt ) T ˜ h
4.2 Bayesian Regret Bound Under Dirichlet Prior
Theorem 4.2.
(Bayesian Regret Bound Under Dirichlet Prior)
Let M be the true MDP distributed according to prior  with any independent Dirichlet prior over transition probabilities.
Then the
Bayesian regret for Parallel Inﬁnite-Horizon posterior sampling MDP algorithn 2 is bounded by
BayesRegret ( T, M , n,  ) =
˜ O ( D p SATn )
Proof Sketch: We use Lemma (G.2) to prove the concentration bound between sampled transition probabilities and the expectation of the true probabilities conditional on the current data.
And note that one key property of Thompson sampling is that conditional upon the current data, the true MDP and the sampled MDP are identically distributed.
4.3 Worse-case Bound Under Dirichlet Prior
Theorem 4.3.
(Worst-case Inﬁnite-Horizon Regret Bound Under Dirichlet Prior)
With high probability, Algoithm 5 (provided in Appendix B) is bounded by
[Equation]
Simulation
In this section, we describe our empirical study of concurrent Thompson Sampling on a Markov
Decision Process with a Dirichlet transition function and a normal reward function for both ﬁnitehorizon and inﬁnite-horizon cases.
In particular, we train the concurrent Thompson Sampling agents on randomly sampled MDPs and evaluate the regrets of these agents at the end of each episode/epoch on a ﬁxed number of environments sampled from the ground truth distribution.
The simulation settings are moved to appendix due to page limit.
Some of the experiment plots are shown in
Figure(1a), (1b), (2a) and (2b).
Conclusion
We have extended PSRL from a single agent to multiple agents, where the agents can concurrent with each other to learn in parallel and share data with each other.
We consider two settings: both ﬁnitehorizon episodic MDP and inﬁnite-horizon MDP.
For state space size S , action space size A , total accumulated time horizon T , episode horizon H , we show a ˜ O ( HS q
ATn ) per-agent Bayesian regret bound under general prior distribution; while especially for independent Dirichlet prior distribution, we show that a tighter bound can be achieved as ˜ O
( H q
SATn ) .
Under inﬁnite horizon case, for the general prior distribution, we achieve ˜ O ( DS q
ATn ) per-agent Bayesian regret bound; and for the inﬁnite-horizon case under dirichlet prior distribution, we obtain a ˜ O
( D q
ATSn ) Bayesian regret bound, with D being the diameter of MDP considered here.
We have provided a theoretical upper bound for concurrent learning under posterior sampling RL setting, and the multiplicative constants in the bound were just properly chosen but not tuned in an optimistic way.
For future work, it would
[Figure] [Figure]
be interesting to include communication cost ([2], [30]) and functional approximation in concurrent learning.
One could evaluate this algorithm under model mis-speciﬁcation ([33]).
It is also valuable to explore model-free and worst-case bound settings([33], [32], [25], [7]).
Another extension is to look at heterogeneous reward structure across agents, and compare the dynamic under the same reward structure setting for both PSRL and UCRL.
Acknowledgement
This work was supported in part by NSF award CCF-2106508.
Zhengyuan Zhou acknowledges the generous support from Horizon Robotics faculty research fellowship and New York University’s
Center for Global Economy and Business faculty research grant.
We would like to thank Pratap
Ranade and Engin Ural for inspiring discussions on society of agents that have helped shape and push an ambitious vision of this research agenda, for which this work is only an initial step.