Learning Infinite-Horizon Average-Reward Restless Multi-Action Bandits via Index Awareness
Anonymous Author(s)
AffiliationAddress email
Abstract
We consider the online restless bandits with average-reward and multiple actions,where the state of each arm evolves according to a Markov decision process(MDP), and the reward of pulling an arm depends on both the current state ofthe corresponding MDP and the action taken.
Since finding the optimal controlis typically intractable for restless bandits, existing learning algorithms are oftencomputationally expensive or with a regret bound that is exponential in the numberof arms and states.
In this paper, we advocate index-aware reinforcement learning(RL) solutions to design RL algorithms operating on a much smaller dimensionalsubspace by exploiting the inherent structure in restless bandits.
Specifically, wefirst propose novel index policies to address dimensionality concerns, which areprovably optimal.
We then leverage the indices to develop two low-complexityindex-aware RL algorithms, namely, (i) GM-R2MAB , which has access to a generativemodel; and (ii) UC-R2MAB , which learns the model using an upper confidencestyle online exploitation method.
We prove that both algorithms achieve a sublinear regret that is only polynomial in the number of arms and states.
A keydifferentiator between our algorithms and existing ones stems from the fact that our RL algorithms contain a novel exploitation that leverages our proposed provablyoptimal index policies for decision-makings.
Introduction
Restless multi-armed bandits ( RMABs ) [61] have been used to model a variety of sequential decisionmaking problems such as congestion control [7, 6], job scheduling [48, 32, 12, 63], wireless communication [54, 20, 9, 19, 66], healthcare [22, 13, 41, 44, 45, 37], queueing systems [26, 30, 5, 15, 39, 38]
and cloud computing [16, 49].
In RMABs , there is a collection of N arms, each of which is endowedwith a state that evolves independently accordingly to a two-action Markov Decision Process (MDP)[52].
If the arm is “pulled” or “activated” at any moment in time, then it advances stochasticallyaccording to one transition kernel, and if not, then it advances according to a different kernel.
Rewardsare generated with each transition, and the goal is to maximize the expected total reward over aninfinite horizon, subject to a constraint on the number of arms activated at any moment in time.
A critical limitation of classical RMABs frameworks is that only two actions, either pulled or notpulled, are allowed for each arm.
This is restrictive since the decision maker in many applicationsoften has access to multiple actions for each arm [37, 17].
To this end, we consider an underexamined generalization of RMABs that allows for multiple actions per arm with different degrees ofcosts, which we call the restless multi-action multi-armed bandits , dubbed as R2MAB , in the infinitehorizon average-reward setting.
Our objective is to develop simple reinforcement learning (RL)algorithms to solve R2MAB without knowing the underlying MDPs associated with each arm.
Although RMABs or R2MAB has found its success in many applications as aforementioned, the fundamentaltheoretical understanding of how to develop low computation-complexity and order-of-optimal regret RL algorithms, two of the most important performance metrics for online restless bandits, remains inits infancy so far.
In particular, three important aspects of RL algorithms for RMABs or R2MAB deservespecial attentions:
▷ First, most upper confidence bound (UCB) based policies [42, 57, 43, 56] for RMABs often leveragea heuristic policy in the exploitation phase, e g., constantly pulling one arm, which does not haveany performance guarantee.
Hence, these policies may not perform close to the offline optimum .This is partially due to the fact that finding an optimal solution for RMABs is PSPACE-hard [51],hence infeasible.
The fundamental challenge lies in the explosion of state space and the curse of dimensionality prevents computing optimal policies.
Though Whittle index policy [61] is a powerful tool to address the state space explosion, finding the Whittle index is typically intractable [48]. ▷ Second, existing RL algorithms with a theoretical guarantee of a sub-linear regret upper bound,e.g., colored-UCRL2 [ suffers from an exponential computation complexity
and the regret bound
is exponential in the number of states and arms .
This is due to the fact that it needs to solve Bellmanequations with an exponentially large space set
Likewise, the class of Thompson sampling
based
algorithms [
34] provide theoretical guarantees in the Bayesian setting, where the updates can be computationally expensive, especially when the likelihood functions are complex .▷ Third, although the design of low computation-complexity solutions for online RMABs has beengaining attentions, e.g., [24, 8, 14, 36, 59, 62, 53], many challenges remain unsolved.
For instance,[24, 8, 14, 36, 53] lacked finite-time performance analysis and these multi-timescale stochasticapproximation algorithms often suffer from slow convergence.
[62] focused on the finite-horizonsetting, which makes their approach not directly applicable to ours since the average reward settingstudied in this paper is more challenging to analyze compared to the finite-horizon setting, whichnecessitates different proof techniques.
[59] achieved a low-complexity policy but is constrained to aspecific birth-death Markovian model and is not easy to be directly generalized.
The lack of a fundamental understanding on how to design efficient RL algorithms for R2MAB in theinfinite-horizon average-reward setting that consider the above three aspects in terms of provablyoptimal policy, computation complexity, exponential pre-factor in the regret bound, and sub-linearregret performance guarantees, motivates us to fill the gap by advocating index-aware RL solutionsin this paper.
Specifically, our index-aware RL solutions operate on a much smaller dimensionalsubspace by exploiting the inherent structure encoded in R2MAB problems.
This requires us to firstdesign low-complexity provably optimal index policies for R2MAB , and then RL algorithms thatleverage the structure of provably optimal index policies in the exploitation phase for decisionmakings so as to reduce the high computational complexity and exponential factor in regret analysis.
Our main contributions in this paper are summarized as follows:
• To address the dimensionality concerns in R2MAB , we first develop low-complexity index policiesfor R2MAB when the underlying MDPs are known.
In contrast to the celebrated Whittle indexpolicy, which requires the notoriously difficult-to-verify indexability condition, we bypass the task ofderiving index policies by taking a more general linear programming (LP) approach, and hence iscomputationally efficient.
We show that our proposed index policy is asymptotically optimal.•
We develop two index-aware RL algorithms for infinite-horizon average-reward R2MAB , namely, (i)GM-R2MAB , a generative model based approach that obtains samples initially then creates the model;and (ii) UC-R2MAB , an online approach wherein the model is updated as samples are obtained.
Bothalgorithms follow a two-stage pattern of model construction and an index policy based solution.
The algorithms solve an extended linear programming (ELP) problem from which we construct theprovably optimal index policies to be executed during the exploitation phase in both algorithms.
The key differentiator between our index-aware RL solutions and aforementioned state of the artsstems from two perspectives: (a) our index-aware RL solutions are computationally appealing sinceour index based solutions are merely based on solving an ELP, which is exponentially better thanstate-of-the-art methods such as colored-UCRL2; (b) our index-aware RL solutions contain a novelexploitation phase by leveraging our proposed provably optimal index policy for decision-makings,rather than using a heuristic one or black-box oracle in aforementioned existing algorithms.•
We provide the first-ever regret analysis for infinite-horizon average-reward R2MAB .
We showthat the above key differentiators in the design of GM-R2MAB and UC-R2MAB , contribute to their˜ O ( √ T ) regret, which is only polynomial in the number of arms and states.
It is worth noting that GM-R2MAB and UC-R2MAB achieve low computation complexity and a sub-linear ˜ O ( √ T ) regret witha polynomial pre-factor all at once , while none of the aforementioned state of the arts achieve so.
Model Description and Problem Formulation
We consider the restless multi-action bandits ( R2MAB ) problem in continuous time.
There are a totalof N arms.
Each arm n ∈ N is described by a MDP ( S , A , P n , r n ) , where S is the state space, A isthe action space, P n ( s ′ | s, a ) indicates the probability of reaching state s ′ by taking action a in states , and r n ( s, a ) is the reward of each state-action pair ( s, a ) .
We assume that S and A are finite setswith cardinalities S and A , respectively.
Our results and analysis will still apply when each arm hasits own sets of states and actions; so, without loss of generality, we will simply assume that all armsshare the same state and action sets.
We consider the unichain MDPs in this paper, which in factis known to be necessary to guarantee the existence of stationary policies for the infinite-horizonaverage-reward R2MAB (and MDPs) regardless of initial states [52, 3].
We refer to action a = 0 aspassive action and any action a > 0 as an active action .
Moreover, using the standard terminologyfrom the restless bandits literature [61], we call an arm active when an active action is applied to itand passive otherwise.
A cost is incurred when action a is applied to arm n in state s .
For abuse ofnotation, we denote the cost as a itself.
We assume that the maximum cost to activating arms at anymoment in time is B , which we call the activation budget .
The decision-making scenario is as follows.
At any moment in time t , each arm can be either activeor passive.
When action a n ( t ) = a is applied to arm n in state s n ( t ) =
s , it takes an exponentiallydistributed amount of time to transition to state s ′ with rate P n ( s, a, s ′ ) , ∀ s, s ′ ∈ S , a ∈ A .
Decisionepochs/time are defined as the moments when the state of an arm changes.
A policy determines whatactions to be applied to each arm at each decision epoch, with the restriction that at most B activationbudgets can be used for activating arms.
After receiving an action, each arm incurs an immediatereward r n ( t ) =
r n ( s, a ) , which is a random variable with support [0 , 1] and mean ¯ r n ( s, a ) .
Withoutloss of generality, we further assume that only active arms yield rewards, i.e., r n ( s, 0) = 0 , ∀ s ∈ S .
Let Π be the set of all possible policies for the considered R2MAB problem and π is a feasible policy inΠ , satisfying π ∈ Π : F t (cid:55)→ A N , where F t is the sigma-algebra [55] generated by random variables{ s n ( h ) , a n ( h ) : ∀ n ∈ N , h ≤ t } .
The objective of the decision maker is to maximize the expectedlong-term average reward of activating arms subject to the activation budget constraint, i.e.,
[Equation]
When the underlying MDP (i.e., transition kernel and reward function) of each arm is known, the
R2MAB problem (1) in theory can be solved using relative value iteration [52, 11].
Unfortunately,this approach suffers from the curse of dimensionality [10, 11], and lack of insights for the solutionstructure.
When the underlying MDPs are not known, off-the-shelf RL algorithms are either computationally expensive or without an ˜ O ( √ T ) regret guarantee.
In the remainder of the paper, we first builda low-complexity index based policy for (1) with optimality guarantee when the MDPs are knownin Section 3, and then design index-aware RL algorithms for (1) that not only are computationallyefficient but also achieve an ˜ O ( √ T ) regret when the MDPs are unknown in Section 4.
An Index-based Policy and Asymptotic Optimality
Rather than solving (1) exactly, we instead construct an index based policy for the original R2MAB (1)that we prove to be asymptotically optimal.
To describe our index policy design in Section 3.1, wefirst need to introduce the following linear programming (LP), in which the decision variables are theoccupancy measures [3] of the controlled MDP processes:
[Equation]
to
[Equation]
(cid:88) a ω n ( s, a ) = (cid:88) s ′ (cid:88) a ′ ω n ( s ′ , a ′ ) P n ( s ′ , a ′ , s ) , ∀ n ∈ N , (4) (cid:88) ( s,a ) ∈S×A ω n ( s, a ) = 1 , ω n ( s, a ) ≥ 0 , ∀ n ∈ N , s ∈ S , a ∈ A .
One can arrive at the LP (2)-(5) by replacing all random variables in the relaxed version of (1) with their expected values via introducing a new set of variables ω n ( s, a ) , which are called occupancy measures [3] of the controlled MDP corresponding to arm n .
The description of reducing (1) to the LP (2)-(5) is provided in supplementary materials.
Specifically, the occupancy measure Ω π of a stationary policy π for the N controlled infinite-horizon MDPs is defined as the expected average number of visits to a state-action pair ( s, a ) , i.e.,
[Equation]
It can be easily checked that the occupancy measure satisfies (cid:80) ( s,a ) ω n ( s, a ) = 1 , and henceω n , ∀ n ∈ N is a probability measure.
To this end, (4) represents the fluid transition of the occupancymeasure, which holds due to the ergodic theorem [18] for finite MDPs [58]; and (5) follows the factthat the occupancy measure is a probability measure.
Thus the feasible set of LP (2)-(5) is non-empty.
We denote the optimal solution to the LP (2)-(5) as Ω π ∗ = { ω ∗ n ( s, a ) : n ∈ N , s ∈ S , a ∈ A} , andthe corresponding optimal value as V ∗ :
=
(cid:80) Nn =1 (cid:80) ( s,a ) ω ∗ n ( s, a ) r n ( s, a ) , which serves as an upperbound on the reward of the original R2MAB (1).
Lemma 1.
The optimal value achieved by the LP (2)-(5) is an upper bound of that of the R2MAB (1) .
3.1 An Index Based Policy
Unfortunately, the solution to the LP (2)-(5) does not always provide a feasible decision to the original
R2MAB (1) .
This is because the activation budget constraint in (1) must be met at all time, instead ofjust in the average sense as in (3).
Exacerbating this problem is the fact that the average constraintmay be violated severely during the decision epochs, resulting in poor policy performance.
Weovercome these challenges by introducing a computationally appealing index policy that we proveto be asymptotically optimal.
Specifically, our index policy is derived from the optimal solutionsΩ π ∗ = { ω ∗ n ( s, a ) } and the index assigned to arm n in state s n ( t )
=
s at time t is defined as
[Equation]
where ξ n ( s, a ) ≜
[Equation]
a
[Equation]
ω
[Equation]
n
[Equation] s, [Equation] ) represents the probability of applying action a n ( t )
= a to arm nin state s n ( t ) =
s at time t [3] when (cid:80) a ′ ∈A ω ∗ n ( s, a ′ ) > 0 , and otherwise arm n can be simplymade passive.
Hence, the index of arm n in (6) represents the expected obtained reward of activatingarm n in state s .
To this end, we rank all arms based on their indices (6) in a non-increasing order,and activate the set of highest indexed arms, denoted as N ( t ) ⊂ N , such that the correspondingactivation cost of arms in N ( t ) is within the activation budget B , i.e., (cid:80) n ∈N ( t ) a ⋆n ( s n ( t )) ≤ B .
Herea ⋆n ( s n ( t )) is the action for arm n , which is determined according to the probability ξ n ( s, a ) basedon its current state s n ( t ) =
s at time t .
When multiple arms share the same indices, we randomlyactivate one arm and allocate the remaining activation costs across all possible actions according tothe probability ξ n ( s, a ) .
All remaining arms are kept passive in case they have zero indices.
We callour index based policy, ERC , since it ranks arms by E xpected R eward and pull arms constrained byactivation C ost.
We denote the resultant index based policy as π ⋆ = { π ⋆n , n ∈ N} .
Remark 1.
Unlike Whittle-based index policies [61, 29, 26, 67, 36], our ERC does not require theindexability condition, which is often hard to establish especially when the transition kernel of theunderlying MDP is convoluted [48].
Like Whittle policies, our ERC is computationally efficient since itis merely based on solving a LP.
We remark that [58] also considered average-reward R2MAB , however,only one action can be chosen deterministically for each state with no difference in activation cost.
Hence it cannot be generalized to ours since we consider a randomized policy for each state withheterogeneous activation costs across different actions.
Finally, another line of works [31, 64, 65, 62]designed index policies without indexability requirement for finite-horizon restless bandits, and hencecannot be directly applied to our infinite-horizon average-reward R2MAB .
3.2 Asymptotic Optimality
We now show that ERC is asymptotically optimal in the same asymptotic regime as that in Whittle[61] and many others [60, 58, 67], where all N arms are generalized to N classes, and both thenumber of class- n arms and the activation budget B are scaled by ρ , with their ratio holding constant.
Denote X ρn ( π ⋆ , s, a ; t ) as the number of class- n arms at state s taking action a at time t under ERCπ ⋆ .
We will be interested in the this fluid-scaling process with parameter ρ , and define the expectedlong-term average reward as V ρπ ⋆ :=
lim inf T →∞ 1 T E π ⋆ ( (cid:82) T t =1 (cid:80) Nn =1 (cid:80) ( s,a ) r n ( s, a ) X ρn ( π ⋆ ,s,a ; t ) ρ dt ) .
Our ERC π ⋆ is asymptotically optimal only when V ρπ ⋆ ≥ V ρπ , ∀ π ∈ Π .
Before presenting our mainresult in this section, we first state the following technical condition called “global attractor” [60].
Definition 1.
An equilibrium point X ρ,⋆ /ρ
under ERC π ⋆ is a global attractor for the process
X ρ ( π ⋆ ; t ) /ρ , if, for any initial point X ρ ( π ⋆ ; 0) /ρ , the process X
ρ ( π ⋆ ; t ) /ρ converges to X ρ,⋆ /ρ.
Remark 2.
The global attractor indicates that all trajectories converge to X ρ,⋆ .
Though it may bedifficult to establish analytically that a fixed point is a global attractor for the process [58], suchassumption has been made in [60, 29, 58, 67, 23] and is only verified numerically.
Our experimentalresults in supplementary materials show that such convergence indeed occurs for our ERC π ⋆ .
Theorem 1.
Our ERC π ⋆ is asymptotically optimal under Definition 1, i.e., lim ρ →∞ V ρπ ⋆ − V ρπ opt = 0 ,where π opt represents the optimal policy for the original R2MAB (1) .
Index-aware Reinforcement Learning Solutions
We now consider to learn the R2MAB (1) when the transition kernel P n and reward function r n ,∀ n ∈ N are unknown.
Our goal is to provide low-complexity index-aware RL algorithms anddetermine their finite-time performance measured by the regret [40], which is defined as follows:
Definition 2.
The regret of policy π is defined as the expected gap between the offline optimum, i.e.,the best policy under which both the transition kernels and reward functions are known, and thecumulative reward of the arm selecting algorithm, i.e., Reg ( π, T ) :
=
Tµ opt − E π
[ R ( π, T )] , whereµ ( π ) :
=
lim T →∞ 1 T E [ R ( π, T )]
=
lim T →∞ 1 T E [ (cid:80) Tt =1 (cid:80)
Nn =
1 r n ( t )] is the expected average rewardunder policy π , and µ opt :=
sup π µ ( π ) is the optimal average reward, which is independent of theinitial state for MDPs with finite diameter [52].
4.1 GM-R2MAB : Generative Model Based Index-Aware Reinforcement Learning
We first introduce a generative model based R2MAB learning algorithm called GM-R2MAB , whichcontains two phases: (i) the exploration phase (lines 1-3 in Algorithm 1) and the exploitation phase (lines 4-6 in Algorithm 1).
During the exploration phase, GM-R2MAB samples eachstate-action pair ( s, a ) for J ( T ) times ( J ( T ) to be specified later) for each arm n , counts thenumber of times T n ( s, a, s ′ ) for each transition to the next state s ′ , and constructs an empirical model of transition kernel and reward function, denoted by ˆ P n ( s ′ | s, a )
=
T n ( s,a,s ′ )
J ( T ) andˆ r n ( s, a ) =
[Equation]
T ) (cid:80)
[Equation]
T )
[Equation]
=
[Equation]
s, a
[Equation]
1 ( s n [Equation]
h ) = s, a n ( h ) = a ) , ∀ ( s, a, s ′ ) , n ∈ N , respectively.
Using these empirical estimates, GM-R2MAB creates a set of plausible MDPs such that the transitionkernels and reward functions are close to the true ones, which are defined as
[Equation]
where δ = (cid:113) 1 2 J ( T ) log 2 SANJ ( T ) η for η ∈ (0 , 1) is built using the Hoeffding inequality [46].
In the exploitation phase, GM-R2MAB computes the ERC index policy π ⋆ by performing optimisticplanning.
In other words, GM-R2MAB selects an optimistic transition kernel, an optimistic rewardfunction, and an optimistic policy to maximize the objective function while satisfying the constraints.
More specifically, it can be expressed as the following optimization problem,
[Equation]
To solve (8), GM-R2MAB uses Extended LP (ELP) by leveraging the state-action-state occupancymeasure z n ( s, a, s ′ ) defined as z n ( s, a, s ′ )
=
P n ( s ′ | s, a ) ω n ( s, a ) to express the confidence intervalsof transition probabilities.
The description of ELP is provided in supplementary materials.
Once we compute { z ⋆n , ∀ n } , the probabilities ξ n ( s, a ) and hence the indices I n ( s ) in (6) are recovered fromthe computed occupancy measures as I n ( s ) :
= (cid:80) s
′∈S z ⋆n ( s,a,s ′ )
¯ r n ( s,a ) (cid:80) b ∈A ,s ′∈S
z ⋆n ( s,b,s ′ ) , from which we can constructour ERC index policy π ⋆ , and execute π ⋆ to the end.
4.1.1 Regret Analysis of GM-R2MAB
To characterize the regret, we first introduce the definition of ergodicity coefficient [4, 2].
For easeof readability, we denote the state for all arms as a stacked vector s ∈ S N :=
[ s 1 , s 2 , . . .
, s N ] , thecorresponding actions under policy π ⋆ as π ⋆ ( s ) , and the unknown MDPs as Θ :=
[ θ 1 , θ 2 , . .
, θ N ]with θ n :=
( P n , r n ) .
The transition kernel of the stacked system is then P Θ ( ·| s , π ⋆ ( s )) , ∀ s ∈ S N .
Definition 3.
D P Θ := 1 − min s , s ′ (cid:80) z ∈S N min { P Θ ( z | s , π ⋆ ( s )) , P Θ ( z | s ′ , π ⋆ ( s ′ )) } is defined as theergodicity coefficient of P Θ , and D :=
sup Θ D P Θ as the maximum value.
Theorem 2.
The regret of GM-R2MAB with J ( T ) =
O ( T 1 / 2 ) satisfies:
[Equation]
The regret comes from the explortation and exploitation phases in GM-R2MAB , respectively.
Specifically, the first term O ( SAB √ T ) in (9) is the worst regret from explorations of each state-actionpair under the generative model with J ( T ) time steps for sampling, and the second term comes fromthe policy execution phase of GM-R2MAB due to MDP model mismatch.
The proof of Theorem 2differs from the traditional analysis framework of generative based RL for restless bandits [59, 62],particularly in the way to track regrets due to model mismatch.
We leverage the form of Bellmanequations of long-term average MDPs and transfer the regret to the difference of relative valuefunctions.
This enables us to track the regret relying only on the first moment behavior of transitionkernels.
However, [59] depended on higher order moment behavior of transition kernels to trackregrets.
The higher moment behavior is typically hard to analyze for a general MDP other than thebirth-death process considered in [59].
[62] tracked regrets by leveraging the optimistic properties ofa linear system, which is only applicable to the finite-horizon setting considered in [62].
Remark 3.
We emphasize that although GM-R2MAB has a similar form as the “explore-then-commit”policy [25], a key differentiator between our GM-R2MAB and state-of-the-art methods stems from twoperspectives.
First, GM-R2MAB has access to a generative model and samples are taken initially toestimate a model.
As a result, GM-R2MAB only needs to solve an ELP once to construct a policy,which is computationally efficient.
This is in contrast to the state-of-the-art colored-UCRL2 [50],which needs to solve a recursive Bellman equation to derive the policy.
One contribution here is todetermine the right choice of J ( T ) , which plays a key role in balancing the tradeoff between modelaccuracy and complexity.
Second, GM-R2MAB executes our proposed provably optimal ERC indexpolicy in the exploitation phase rather than using a heuristic one as in state of the arts [42, 57, 43, 56].
This contributes to the polynomial prefactor in the regret, compared to an exponential one in coloredUCRL2.
Finally, we note that such a generative model based approach has also been used in thecontext of CMDPs [27, 28] and restless bandits [59, 62].
However, they considered either a finite-time[27, 62] or a discounted setting [28], and hence cannot be directly applied to the infinite-horizonaverage-reward setting studied in this paper.
Furthermore, [27, 28] focused on the sample complexityanalysis, which is not directly translatable to the regret [21].
Though Restless-UCB [59] operatedin a similar manner as our GM-R2MAB , it depends on the performance of an offline “black-boxed”oracle approximator in the exploitation phase, while our GM-R2MAB leverages an explicit and provablyoptimal ERC index policy.
4.2 UC-R2MAB : Online Index-Aware Reinforcement Learning
The GM-R2MAB approach operates in an “offline” manner in the sense that it first estimates a modelby sampling every state-action pair in the system for a certain number of times, and then computes apolicy to be executed throughout the exploitation phase.
Unfortunately, such an “offline” approachmay not be feasible in many real-world applications since some states may not be reachable withoutexecuting the policy.
To this end, we further develop an “online” approach, dubbed as UC-R2MAB ,via interleaving the process of collecting samples from the environments and model updates, whichis summarized in Algorithm 2.
Specifically, UC-R2MAB operates in an episodic manner where eachepisode consists of H consecutive frames.
Let K be the total number of episodes until time T , hencewe have T = KH .
Denote the k -th episode as H k and let τ k be the time when it starts.
At the beginning of each episode, i.e., τ k , ∀ k , UC-R2MAB estimates the true transition kernel and thetrue reward by the corresponding empirical averages as ˆ P kn ( s ′ | s, a ) =
[Equation]
=
[Equation]
a ) 1 ( s τn ( h ) =
s, a τn ( h ) = a ) , where C k − 1 n ( s, a ) is the number of visitsto state-action pairs ( s, a ) until τ k , and C k − 1 n ( s, a, s ′ ) is the number of transitions from s to s ′ underaction a , satisfying C kn ( s, a ) =
C k − 1 n ( s, a ) + (cid:80)
Hh =1 1 ( s kn ( h ) = s, a kn ( h ) = a ) , and C kn ( s, a, s ′ ) =
C k − 1 n ( s, a, s ′ )
+ (cid:80)
Hh =1 1 ( s kn ( h +1) =
s ′
| s kn ( h ) =
s, a kn ( h ) = a ) , ∀ ( s, a ) ∈ S ×A and ∀ ( s, a, s ′ ) ∈
S ×
A × S , ∀ n , where s kn ( h ) is the state of arm n at the h -th time frame in episode k .
Similar to GM-R2MAB , UC-R2MAB creates a set of plausible MDPs using these empirical estimates foreach episode as in (7) by replacing J ( T ) with C k − 1 n ( s, a ) for each arm n. Thus for η ∈ (0 , 1) , theset of plausible MDPs in the k -th episode is defined as
[Equation]
(10) where δ kn ( s, a ) =
(cid:113)
[Equation]
s
[Equation]
log
[Equation]
is built using the Hoeffding inequality [46].
Once constructing the model, UC-R2MAB computes the ERC index policy π ⋆,k in episode k bysolving an ELP, which is described in supplementary materials.
Specifically, it solves the followingoptimization problem,
[Equation]
from which it recovers the indices I kn ( s ) to construct ERC and then execute the index policy to theend of this episode.
This is the same problem as in (8) except for substituting M with M k .
This algorithm draws inspiration from the infinite-horizon algorithm UCRL [33], which uses thesampled trajectory of each episode to update the plausible MDPs of next episode.
However, thereexists two major differences.
First, UC-R2MAB updates the models at the beginning of each episode,which allows for faster model construction as [27].
Second, UC-R2MAB deploys the proposed ERC ateach episode, and thus results in solving a low-complexity ELP, which is exponentially better thanthat of UCRL-based algorithm [33] that needs to solve extended value iterations.
4.2.1 Regret Analysis of UC-R2MAB
Theorem 3.
The regret of UC-R2MAB satisfies:
[Equation]
Though the design of UC-R2MAB is inspired by UCRL type algorithms, the regret analysis of
UC-R2MAB differs from colored-UCRL2 [50], a state-of-the-art method for online restless bandits.
The major difference comes from the fact that UC-R2MAB leverages the relative value function of Bellman equation for long-term average MDPs to track regrets caused by model mismatch as GM-R2MAB .
There are also recent Thompson sampling based results [2] on characterizing Bayesian regret for
RMABs while using techniques reminiscent of UC-R2MAB .
The sub-linear regret in [2] was achievedby upper bounding the number of episodes to be √ T under a specific requirement for the length ofeach episode.
In contrast, UC-R2MAB directly bounds the sum of regret from each episode withoutsuch a episode length constraint.
Remark 4.
Similar to the state-of-the-art colored-UCRL2 [50] and Thompson sampling basedalgorithms [34, 35, 2], our UC-R2MAB also operates in an episodic manner and achieves an ˜ O ( √ T )regret.
However, the color-UCRL2 is known to be computationally expensive and the regret isexponential in the number of arms and states.
Likewise, Thompson sampling based methods providetheoretical guarantee in the Bayesian settings and need to implement a computationally expensivemethod to update the posterior beliefs due to the complex likelihood functions.
In contrast, our
UC-R2MAB is computationally appealing since it only needs to solve an ELP in each episode, fromwhich UC-R2MAB constructs and implements our proposed ERC index policy.
The regret of UC-R2MABis polynomial in the number of arms and states, which is exponentially better than that of coloredUCRL2.
Finally, we remark that the multiplicative “pre-factor” that goes with the time dependentfunction in the regret of UC-R2MAB is smaller than that of GM-R2MAB .
This is due to the fact that
UC-R2MAB operates in an episodic manner while GM-R2MAB only samples once initially.
This leads toa higher computation complexity of UC-R2MAB , which is O ( NSAK ) compared to that of GM-R2MAB ,which is O ( NSA ) .
Experiments
In this section, we present some of our experimental results.
We also demonstrate the utility of our
GM-R2MAB and UC-R2MAB
by evaluating them under two real-world applications of restless bandits.
5.1 Experiments on Constructed Instance
Instance construction.
We consider a setup with 10 classes of arms.
The state space is S ∈{ 0 , 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 } .
Class- n arm arrives with rate 3 n for n
= 1 , · · · , 10 , and departs witha fixed rate of µ = 20 .
Since Whittle index policy is only designed for RMABs with two actions, weevaluate our algorithms under two settings: (1) two-action setting; and (2) multi-action setting.
Fortwo-action setting, we consider a controlled Markov process in which states evolve as a specificbirth-death process, i.e., state s only transit to s +
or s − 1 with probability P ( s, s + 1) =
λ/ ( λ + µ ) and P ( s, s − 1) =
µ/ ( λ + µ ) .
For active action, class- n arm generates a random rewardr n ( s ) ∼ Ber ( sp n ) , with p n uniformly sampled from [0 . 01 , 0 . 1] .
The activation budget is set to0 .
3 N , where we vary the number of arms N from 50 to 350 .
For the multi-action setting, we considera total of 3 , 5 , 10 actions, and the state of arm n transition to any state in S with a randomizednon-zero transition probability.
We only present results with 10 actions and provide other results insupplementary materials.
Baselines.
We compare our ERC index policy with Whittle index policy (Whittle) [61], a prioritypolicy (Priority) [58] and a lagrangian based policy (Lagrangian) [37].
We compare our RL solutions
GM-R2MAB and UC-R2MAB with a Q-learning based policy (MAIQL)
[36], a Thompson samplingpolicy (TS) [35], Restless-UCB [59], and a set of learning Whittle index based policy including
Fu [24], AB [8] and NeurWin [47].
Note that we do not include results of colored-UCRL2 in ourcomparions since it is known to suffer from a high computation complexity and is outperformedby existing policies, e.g., Restless-UCB [59].
For ease of readability, we only present results oftwo “online” baselines for comparisons.
More experimental results and the parameter settings forbaselines are provided in supplementary materials.
Asymptotic optimality.
We compare the rewards obtained by an index policy with that from thetheoretical upper bound obtained by solving the LP (2)-(5).
We call the ratio between this awarddifference and the number of arms as the optimality gap .
From Figures 1 and 2, we observe that allpolicies are asymptotically optimal, which is consistent with their theoretical performance guarantees.
Our ERC slightly outperforms these baselines in terms of the vanishing speed of optimality gap.
[Figure]
[Figure]
Regret and running time.
The accumulated regrets are presented in Figures 3 and 4, where we usethe Monte Carlo simulation with 1 , 000 independent trials of a single-threaded program on AMD
Ryzen 5800x desktop with 64GB RAM.
For simplicity, we choose 200 arms and a time horizon of
T = 60 , 000 slots.
Each episode consists of 2 , 500 slots.
We observe that UC-R2MAB achieves thelowest cumulative regret and is better than our “offline” method GM-R2MAB .
This is consistent withour theoretical analysis (see Remark 4).
A key observation is that for a large horizon T , our “offline”
GM-R2MAB even outperforms the “online” MAIQL.
This is because GM-R2MAB leverages our proposedprovably optimal ERC index policy in the exploitation phase.
We also compare the average runningtime of these algorithms.
For two-action (10-action) setting, the average running time of GM-R2MAB ,
UC-R2MAB , MAIQL and TS is 86s (144s), 308s (607s), 348s (702s) and 359s (681s), respectively.
Itis clear that our GM-R2MAB and UC-R2MAB are more efficient in running time.
These improvementscome from the intrinsic design of our algorithms that merely need to solve an ELP, while TS needs tosolve Bellman equations.
More results are provided in supplementary materials.
5.2 Experiments on
Real-World Datasets
We demonstrate the utility of GM-R2MAB and UC-R2MAB by evaluating them under two recently studied applications of restless bandits:wireless scheduling with two actions, and tuberculosis care withmultiple actions.
Due to space constraints, we relegate the detaileddescriptions of these two problems to supplementary materials.
Wireless scheduling over fading channels [1].
A wireless clientis modeled as an arm, which has some data to transmit.
Each arm suffers from 1 unit holding costin each time slot until the data is transmitted.
The quality of wireless channel, either good or bad,via which data is transmitted determines the amount of transmitted data and varies over time.
Thegoal is to maximize the negative of total holding cost.
We adopt the settings in [1], where 1 outof 10 arms is activated at any moment in time.
The regret is shown in Figure 5.
It is observedthat UC-R2MAB outperforms all baselines.
Importantly, UC-R2MAB has a sub-linear regret guaranteewhile the best-performing baseline MAIQL lacks of finite-time analysis.
Since the environment isdynamically changing over time, it may be hard to build a perfect simulator and hence UC-R2MAB ispreferable than the offline GM-R2MAB .
Tuberculosis care in India [37].
A health worker takes three actionson 200 patients to improve their adherence over a course of 6 months.
Each action has varying cost and effectiveness: cheap (call patients),semi-expensive (visit patients) and very expensive (escalate patients).
The goal is to maximize patients’ adherence subject to a daily timebudget due to the limited worker time and resources.
We adopt thesettings in [37] with the budget being 20 and the reward defined asadherence level / 3 .
From Figure 6, we again observe that UC-R2MABachieves a sub-linear regret and outperforms all baselines.
Conclusion
In this paper, we developed two low-complexity index-aware RL algorithms, GM-R2MAB and
UC-R2MAB for online infinite-horizon average-reward restless multi-action bandits.
We provedthat both algorithms achieved a sub-linear regret that is only polynomial in the number of armsand states.
Our key design to reduce both the computation complexity and exponential factor inregret analysis is via exploiting the inherent structure encoded in restless bandits and leveraging ourproposed provably optimal index policies for decision-makings.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[N/A]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]