C ONGESTED BANDITS : O PTIMAL ROUTING VIA SHORT - TERM RESETS
Anonymous authors
Paper under double-blind review
A BSTRACT
For trafﬁc routing platforms, the choice of which route to recommend to a user depends on the congestion on these routes – indeed, an individual’s utility depends on the number of people using the recommended route at that instance.
Motivated by this, we introduce the problem of Congested Bandits where each arm’s reward is allowed to depend on the number of times it was played in the past ∆ timesteps.
This dependence on past history of actions leads to a dynamical system where an algorithm’s present choices also affect its future pay-offs, and requires an algorithm to plan for this.
We study the congestion aware formulation in the multi-armed bandit (MAB) setup and in the contextual bandit setup with linear rewards.
For the multi-armed setup, we propose a UCB style algorithm and show that its policy regret scales as ˜ O ( √ K ∆ T ) .
For the linear contextual bandit setup, our algorithm, based on an iterative least squares planner, achieves policy regret ˜
O ( √ dT + ∆) .
From an experimental standpoint, we corroborate the no-regret properties of our algorithms via a simulation study.
I NTRODUCTION
The online multi-armed bandit (MAB) problem and its extensions have been widely studied and used to model many real world scenarios (Robbins, 1952; Auer et al., 2002; 2001).
In the basic MAB setup there are K arms with either stochastic or adversarially chosen reward proﬁles.
The goal is to design an algorithm that achieves a cumulative reward that is as good as that of the best arm in hindsight.
This is quantiﬁed in terms of the regret achieved by the algorithm over T time steps (see
Section 2 for formal deﬁnitions).
In many real world scenarios the MAB setup as described above is not suitable as the reward obtained by playing an arm/action at a given time step may depend on the algorithm’s choices in the previous time steps.
In particular, we are motivated by online routing problems where the reward of suggesting a particular edge to traverse along a path from source to destination often depends on the congestion on that edge.
This congestion is a function of the number of times the particular edge has been recommended earlier (potentially in a time window).
In such scenarios, one would desire algorithms that can compete with the best policy , i.e., the best sequence of actions, in hindsight as compared to a ﬁxed best action.
Classical multi-armed bandit formulation and associated no-regret algorithms (Slivkins, 2019), or their extensions to routing problems (Kalai & Vempala, 2005; Awerbuch & Kleinberg, 2008) do not sufﬁce for the above scenario as they only guarantee competitiveness with respect to the best ﬁxed arm in hindsight.
To overcome these limitations, we propose a new model, viz. , congested bandits which captures the above scenario.
In our proposed model the reward of a given arm at each time step depends on how many times the arm has been played within a given time window of size ∆ .
Hence over time, an arm’s expected reward may decay and reset dynamically.
While our model is motivated by online routing problems, our proposed formulation is very general.
As another example, consider a digital music platform that recommends artists to its end users.
In order to maximize proﬁt the recommendation algorithm may prefer to recommend popular artists, and at the same time the platform may want to promote equity and diversity by highlighting new and emerging artists as well.
This scenario can be model via congested bandits where each artist is an arm and the reward for suggesting an artist is a function of how many times the artist has been recommended in the past time window of length ∆ .
In both the scenarios above, the ability to reset the congestion cost (by simply not playing an arm for ∆ time steps) is a crucial part of the problem formulation.
Our contributions.
We propose and study the congested bandits model with short term resets under a variety of settings and design no-regret algorithms.
In the most basic setup we consider a K -armed stochastic bandit problem where each arm a has a mean reward µ a , and the mean reward obtained by playing arm a at time t equals f cong ( a, h t ) µ a .
Here h t denotes the history of the algorithm’s choices within the last ∆ time steps and f cong is a non-increasing congestion function.
Recall that the algorithm’s goal in this setup is to compete with the best policy in hindsight.
While the above setting can be formulated as a Markov Decision Process (MDP), existing no regret algorithms for
MDPs will incur a regret bound that scales exponential in the parameters (scaling as K ∆ )
(Auer et al., 2009; Jaksch et al., 2010).
Instead, we carefully exploit the problem structure to design an algorithm with near-optimal regret scaling as ˜ O ( √ K ∆ T ) .
Next, we extend our model to the case of online routing with congested edge rewards, again presenting near optimal no-regret algorithms that avoid exponential dependence on the size of the graph in the regret bounds.
We then extend the multi-armed and the congested online routing formulations to a contextual setting where the mean reward for each arm/edge at a given time step equals f cong ( a, h t ) (cid:104) θ ∗ , φ ( x t , a )) (cid:105) , where x t ∈ R d is a context vector and θ ∗ is an unknown parameter.
This extension is inspired from classical work in contextual bandits (Li et al., 2010; Chu et al., 2011) and captures scenarios where users may have different preferences over actions/arms (e.g., a user who wants to avoid routes with tolls).
Solving the contextual case poses signiﬁcant hurdles as a priori it is not even clear whether the setting can be captured via an MDP.
By exploiting the structure of the problem, we present a novel epoch based algorithm that, at each time step, plays a near optimal policy by planning for the next epoch.
Showing that such planning can be done when only given access to the distribution of the contexts is a key technical step in establishing the correctness of the algorithm.
As a result, we obtain algorithms that achieve ˜ O ( √ dT + ∆) regret in the contextual MAB setting.
Finally, using simulations, we perform an empirical evaluation of the effectiveness of our proposed algorithms.
Related work.
Closest to our work are studies on multi-armed bandits with decaying and/or improving costs.
The work of (Levine et al., 2017) proposes the rotting bandits model that has been further studied in (Seznec et al., 2019).
In this model the mean reward of each arm decays in a monotonic way as a function of the number of times the arm has been played in the past.
There is no notion of a short-term reset as in our setting.
As a result it can be shown that a simply greedy policy is optimal in hindsight.
In contrast, in our setting greedy approaches can fail miserably as highlighted in Figure 1.
The work of Heidari et al.
(2016) considers a setting where the mean reward of an arm can either improve or decay as a function of the number of times it has been played.
However, similar to the rotting bandits setup there is no notion of a reset.
Pike-Burke & Gr¨unew¨alder (2019) considers a notion of reset/improvement by allowing the mean reward to depend on the number of time steps since an arm was last played.
Their work considers a Bayesian setup where the mean reward function is modelled by a draw from a Gaussian process.
On the other hand, in our work, the reward is a function of the number of times an arm was played in the past ∆ time steps.
However their regret bounds are either with respect to the instantaneous regret, or with respect to a weaker policy class of d -step look ahead policies.
The notion of instantaneous regret only compares with the class of greedy policies at each timestep as compared to the globally optimal policy.
There also exist works studying the design of online learning algorithms against m -bounded memory adversaries.
The assumption here is that the reward of an action at each time step depends only on the previous m actions.
While this is also true for our setting, in general the regret bounds provided for bounded-memory adversaries are either with respect to a ﬁxed action or a policy class containing policies that do no switch often (Arora et al., 2012;
Anava et al., 2015).
Our problem formulation for the contextual setup is reminiscent of the recent line of works on contextual MDPs (Azizzadenesheli et al., 2016;
Krishnamurthy et al., 2016; Hallak et al., 2015; Modi & Tewari, 2020).
However these works either assume that the context vector is ﬁxed for a given episode (allowing for easier planning), or make strong realizability assumptions on the optimal Q -function.
Finally, our congested bandits formulation of the routing problem is a natural extension of classical work on the online shortest path problem Kalai & Vempala (2005); Awerbuch & Kleinberg (2008); Dani et al.
C ONGESTED MULTI - ARMED BANDIT
In this section, we model the congestion problem in a multi-armed bandit (MAB) framework.
Our setup for congested multi-armed bandits models the congestion phenomenon by allowing the rewards of an arm to depend on the number of times it was played in the past.
Let us consider a MAB setup with K arms, where each arm may represent a possible route.
Let us denote by ∆ the size of the window which affects the reward at the current time step.
For any time t , let h t ∈ H ∆ : =
[ K ] ∆ denote the history of the actions taken by an algorithm 1 in the past ∆ time steps, that is, h t =
[ a t − ∆ , . . . , a t − 1 ] where a τ is the action chosen by the algorithm at time τ .
In order to model congestion arising from repeated plays of a single arm, we consider a function f
cong :
[ K ] × [∆] + → (0 , 1] .
This congestion function takes in two arguments: an arm a
and the number of times this arm was played in the past ∆ time steps, and outputs a value indicating the decay in the reward of arm a arising from congestion.
Protocol for congestion in bandits.
We consider the following online learning protocol for a learner in our congested MAB framework:
At each round t , the learner picks an arm a t ∈
[ K ] and observes reward
[Equation]
where (cid:15) t ∼ N (0 , 1)
[Equation]
Finally, the history changes to h t +1 =
[ h t, 2:∆ , a t ] where we have used the notation h t,i : j to denote the vector [ h t ( i ) , . .
, h t ( j )] and #( h, a ) to denote the number of times the action a was played in history h .
Each arm a is associated with a mean reward vector µ a and the congestion function multiplicatively decreases the reward of that arm.
We assume that the learner does not know the exact form of the function f
cong as well as the the mean vector µ = { µ a } [ K ] .
The objective of the learner is to select the actions a t which minimizes a notion of policy regret, which we deﬁne next.
Policy regret for congested MAB.
In the standard MAB setup, regret compares the cumulative reward of the algorithm to the benchmark of playing the arm with the highest mean reward at all time steps.
However, as described in Section 1, this benchmark is not suitable for our setup.
Indeed, the asymptotically optimal algorithm is one which maximizes the average cumulative reward ρ ∗ : = max alg lim T →∞ 1 T (cid:80)
Tt =1 r ( h t , a t ) , where the action a t is the one chosen by the algorithm alg and history h t is the sequence of actions in the past ∆ time steps.
This asymptotically algorithm corresponds to a stationary policy π ∗ whose selection a t only depends on the history h t .
Accordingly, we consider the following class of stationary policies as the comparator for our regret Π = { π : H ∆ → [ K ] } , with size | Π | =
K K ∆ .
Denote by h πt the history at time t by running policy π up to time t , we deﬁne the policy regret for any algorithm
[Equation]
Input: Congestion window ∆ , conﬁdence parameter δ ∈ (0 , 1) , action set [ K ] , time horizon T .
Initialize: Set t =for episodes e = 1 , . . .
, E do
Initialize episode
Set start time of episode t e = t .
For all actions a and historical count j , set n e ( a, j ) = 0 and N e ( a, j )
=
(cid:80) s<
e n s ( a, j ) .
Set empirical reward estimate for each arm and historical count ˆ r e ( a, j )
= (cid:80) t e − [Equation] = [Equation]
[Equation]
=
[Equation] ] max [Equation] , N e ( a, j )
[Equation]
Compute optimistic policy
Set the feasible rewards
R e = (cid:40) r ∈ [0 , 1] A × (∆+1) | for all ( a, j ) , | r ( a, j ) − ˆ r e ( a, j ) | ≤(cid:115) log( A ∆ t e /δ ) max { 1 , N e ( a, j ) } (cid:41)
Find optimistic policy ˜ π e = arg max π ∈ Π ,r ∈R e ρ ( π, r )
Execute optimistic policy while n e ( a, j ) < max { 1 , N e ( a, j ) } do
Select arm a t = ˜
π e ( s t ) , obtain reward ˆ r t .
Update n e ( a, #( s t , a )) =
n e ( a, #( s t , a )) + 1 . where a t is the action chosen by alg at time t and r ( h t , a t ) = f cong ( h t , ) · µ a t .
This notion of regret is called policy regret (Arora et al., 2012) because the history sequence observed by the algorithm π ∗ and the algorithm alg can be different from each other – this leads to a situation where choosing the same action a at time t can lead to different rewards for the algorithm and the comparator.
2. CARMAB: C ONGESTED MAB ALGORITHM
We now describe our learning algorithm for the congested MAB problem.
At a high level,
CARMAB, detailed in Algorithm 1, is based on a reduction of this problem to a reinforcement learning problem with state space S = H ∆ and action space A =
[ K ] , where the underlying dynamics are known to the learner.
With this reduction, CARMAB deploys an epoch-based strategy which plays a optimistic policy ˜ π e computed from optimistic estimates of the reward function.
Reduction to MDP.
Our congested MAB setup can be viewed as a learning problem in a Markov decision process (MDP) with ﬁnite state and action spaces.
This MDP M mab comprises state space
S = H ∆ and action space A =
[ K ] .
The reward function for this MDP is given by r ( s, a ) =
f cong ( a, #( s, a )) · µ a and the deterministic state transitions are given P ( s (cid:48) | s, a ) =
I [ s (cid:48) =
[ s 2:∆ , a ]] .
Algorithm details.
Our learning algorithm in this MDP is an upper conﬁdence bound (UCB) style algorithm, adapted from the classical UCRL2 (Jaksch et al., 2010) for learning in ﬁnite MDPs.
It splits the time horizon T into a total of E epochs, each of which can be of varying length.
In each episode, for every pair ( a, j ) of action a and historical count j ∈ [∆] + , the algorithm computes the empirical estimate of the rewards ˆ r e ( a, j ) from observations in the past epoch and maintains a feasible set R e of rewards.
This set is constructed such that with high probability, the true reward r belong to this set for each epoch.
Given this set, our algorithm computes the optimistic policy
[Equation]
lim
[Equation]
that is, the policy which achieves the best average expected reward with respect to the optimistic set
R e .
The optimistic policy π e is then deployed in the congested MAB setup till one of the ( a, j ) pair doubles in the number of times it is played and this determines the size of any epoch.
Computing optimistic policy.
In the MDP described above, each deterministic policy π follows a cyclical path since the transition dynamics are deterministic and the state space is ﬁnite.
With this insight, this problem of ﬁnding the optimal policy with highest average reward is equivalent to ﬁnding the maximum mean cycle in a weighted directed graph.
In our simulations, we use Karp’s algorithm (Karp, 1978) for ﬁnding these optimistic policy.
This algorithm runs in time
O
( K ∆+1 ) .
2. R EGRET ANALYSIS FOR CARMAB
In this section, we obtain a bound on the policy regret of the proposed algorithm CARMAB.
Our overall proof strategy is to ﬁrst establish that the MDP M mab has a low diameter (the time taken to move from one state to another in the MDP), then bounding the regret in each episode e of the process and ﬁnally establishing that the total number of episodes E can be at most logarithmic in the time horizon T .
Combining these elements, we establish in the following theorem that the regret of
CARMAB scales as ˜ O ( √ K ∆ T ) with high probability.
Theorem 1 (Regret bound for CARMAB) .
For any conﬁdence δ ∈ (0 , 1) , congestion window ∆ > 0 and time horizon T > ∆ K , the policy regret (1) , of CARMAB is
[Equation]
(cid:18) T ∆ K (cid:19) + c (cid:115)
K ∆ T log (cid:18) ∆ KT δ (cid:19) + c (cid:115)
T log (cid:18)δ (cid:19) with probability at least 1 − δ .
A few comments on the theorem are in order.
Observe that the dominating term in the above regret bound scales as ˜ O ( √ K ∆ T ) in contrast to the classical regret bounds for MAB which have a ˜ O ( √ KT ) dependence.
This additional factor of √ ∆ comes from the fact the stronger notion of policy regret as well as the non-stationary nature of the arm rewards.
Additionally, a na ¨ ıve application of the UCRL2 regret bound to the constructed MDP scales linearly with state space and would correspond to an additional factor of O ( K ∆ ) .
The CARMAB algorithm is able to avoid this exponential dependence by exploiting the underlying structure in the congested MAB problem.
Our regret bound are also minimax optimal – observe that for any constant value of ∆ , the lower bounds from the classical MAB setup immediately imply that the regret of any learner should scale as Ω( √ KT ) , which matches the upper bound in Theorem 1.
We defer the complete proof of this to Appendix A but provide a high-level sketch of the important arguments.
MDP M mab has bounded diameter.
The diameter D of an MDP M measures the number of steps it takes to reach a state s (cid:48) from a state s using an appropriately chosen policy.
The diameter is a measure of the connectedness of the underlying MDP and is commonly studied in the literature on reinforcement learning (Puterman, 2014).
Deﬁnition 1 (Diameter of MDP.) .
Consider the stochastic process induced by the policy π on an
MDP M .
Let τ ( s (cid:48) | s, M , π ) represent the ﬁrst time the policy reaches state s (cid:48) starting from s .
The diameter D of the MDP is D : = max s (cid:54) = s (cid:48)
min π E [ τ ( s (cid:48) | s, M , π )] .
Recall from Section 2.1 that the MDP M mab has deterministic dynamics and state space given by the set of histories H ∆ .
Proposition 2 in Appendix A establishes that the diameter of this MDP is at most the window size ∆ .
With this bound on the diameter, we then show in Lemma 4 that the total regret of the algorithm can be decomposed into a sum of regret terms, one for each episode.
[Equation]
which holds with probability at least 1 − δ
We have used n e ( a, j ) to denote the number of times action a was played by the algorithm when it had a count j in the history and ρ π the average reward of policy π .
Our analysis then proceeds to bound the per-episode regret r e .
Regret for episode e .
In episode e , the set of feasible rewards R e is chosen to ensure that with high probability, the true reward r ∗ ( a, j ) =
f cong ( a, j ) · µ j belongs to this set.
Conditioning on this event, we show that the regret in each episode is upper bounded by the window size ∆ and a scaled ratio of the number of times each action-history ( a, j ) is played, that is, [Equation] ∆
[Equation]
where t e is the time at which episode e starts and N e ( a, j )
= (cid:80) i<
e n i ( a, j ) .
Theorem 1 follows from combining the above with a bound on the total number of episodes E ≤ c · ∆ K log (cid:0) T ∆ K (cid:1) .
2. R OUTING WITH C ONGESTED BANDITS
We now study an extension of the congested MAB setup where the arms correspond to edges on graph G =
( V, E ) with a pre-deﬁned start state s G and goal state t G .
In this setup, at each round t the learner selects an s G - t G path p t on the graph G and receives reward ˜ r ( h t , e i,t ) =
f cong ( e i,t , #( h t , e i,t )) · µ e i,t + (cid:15) t for each e i,t on path p t .
The history changes to h t
+1 =
[ h t, 2:∆ , p t ] .
In comparison to the multi-armed bandit protocol, the learner here selects an s G - t G path on the graph
G , the history at any time t consists of the entire set of paths { p t − ∆ , . . .
, p t − 1 } , and we assume that the congestion function on each edge f cong ( h, e ) depends on the number of times this edge has been used in the past ∆ time steps.
The following theorem generalizes the result from Theorem 1 and shows that a variant of CARMAB has regret ˜ O ( √ T ) for the above s G - t G online problem.
Theorem 2 (Regret bound for CARMAB-st) .
For any conﬁdence δ ∈ (0 , 1) , congestion window ∆ > 0 and time horizon T , the policy regret, of CARMAB-st is R T
[Equation]
CARMAB
[Equation] f cong ) [Equation] with probability at least 1 − δ .
The proof of the above theorem is detailed in Appendix A.3.
L INEAR CONTEXTUAL BANDITS WITH CONGESTION
We now consider the contextual version of the congested bandit problem, where the reward function depends on the choice of arm a as well as an underlying context x ∈ X .While
most of our notation stays the same from the multi-armed bandit setup in Section 2, we introduce the modiﬁcations required to account for the context vectors x t .
We consider the linear contextual bandit problem where the reward function is parameterized as a linear function of a parameter θ ∗ and context-action features φ ( x t , a t ) , that is, r ( h t , a t , x t ; θ ∗ ) :
= (cid:104) θ ∗ , φ ( x t , a t ) (cid:105)
f
cong ( a t , #( h t , a t )) , where we assume that the context-action features φ ( x t , a t )
⊂
R d satisfy (cid:107) φ ( x t , a t ) (cid:107) 2 ≤ 1 and the true parameter (cid:107) θ ∗ (cid:107) 2 ≤ 1 .
In contrast to the bandit setup, we expand our policy class to be dependent on the context as well with Π X : =
{ π : H ∆ × X (cid:55)→ [ K ] } .
In each round of the linear contextual bandits with congestion game, the learner observes context vectors { φ ( x t , a i ) }
[ K ] and selects action a t .
The learner then observes reward ˜ r
( h t , x t , a t ) =
r ( h t , a t , x t ; θ ∗ ) + (cid:15) t and the history changes to h t +1 =
[ h t, 2:∆ , a t ] .
The objective of the learner in the above contextual bandit game is to output a sequence of actions which are competitive with the best policy π ∈ Π X .
Formally, the regret of an algorithm alg is deﬁned to be
[Equation]
In order to provide some intuition about the algorithm, we start with a simple case where all the contexts are known to the learner in advance and later generalize the results to stochastic contexts.
3. W ARM - UP : K NOWN CONTEXTS
In the known context setup, the learner is provided access to a set of contexts { x t } at the start of the online learning game.
Algorithm 2 details our proposed algorithm, CARCB, for this setup.
Algorithm
Input: Congestion window ∆ , congestion function f cong , action set [ K ] , time horizon T , contexts { x t } Tt =
Initialize: Set t = 1 , θ 1 ∼ unif ( B d ) for episodes e = 1 , . . .
, E do
Initialize episode
Set start time of episode t e = t .
Let the steps in this epoch I e =
[ t e , . . .
, t e + 2 e ∆] .
Set the episode policy ˜ π e = arg max π ∈ Π X (cid:80) t ∈
I e \{ t e ,...,t e +∆ } r
( h πt , π ( h πt , x t ) , x t ; θ e )
Execute estimated policy for t = t e , . .
, t e + 2 e ∆ do
Select arm a t = ˜
π e ( h t , x t ) and observe reward ˆ r t .
Update θ e via OLS update θ e +1 =
arg min θ (cid:80) τ
( r τ − (cid:104) θ, φ ( x τ , a τ ) (cid:105) f
cong ( a τ , #( h τ , a τ ))
Algorithm details.
We again divide the total time T into E episodes, where the length of each episode e = 2 e ∆ .
Unlike CARMAB, the algorithm does not maintain any optimistic estimate of the reward parameter θ ∗ but simply updates it via an ordinary least squares (OLS) procedure and executes the policy ˜ π e which maximizes this estimated reward function.
The core idea underlying this algorithm is that as we observe more samples, our estimate θ e converges to θ ∗ and our planner is then able to execute the optimal sequence of actions.
Regret analysis.
To analyze the regret for CARCB, we study the error incurred in estimating the parameter θ e from the reward samples.
To do so, we begin by making the following assumption on the minimum eigenvalue of the sample covariance matrix obtained at any time t e .
Assumption 1.
For t > cd and for any sequence of actions { a 1 , . . .
, a T } , we have λ
min (cid:16) 1 t (cid:80) τ ≤ t φ ( x t , a t ) φ ( x t
a t ) (cid:62) (cid:17)
≥ γ , for some value γ >
Our bound on the regret R T will depend on this minimum eigenvalue γ .
Later when we generalize our setup to the unknown setup, we will show that this assumption holds with high probability for a large class of distributions.
The following theorem shows that the regret bound for CARCB scales as ˜ O ( √ dT + ∆) with high probability.
Proposition 1 (Regret bound; known contexts) .
For any conﬁdence δ ∈ (0 , 1) , congestion window ∆ > 0 and time horizon T > cd , suppose that the sample covariance Σ t satisﬁes Assumption 1.
Then, the policy regret, deﬁned in eq.
(5) , of CARCB with respect to the set Π is
[Equation]
with probability at least 1 − δ where c min = min a,j f cong ( a, j ) .
The proof of the above theorem is deferred to Appendix B. At a high level, the proof proceeds in two steps where ﬁrst it upper bounds the error (cid:107) θ e − θ ∗ (cid:107) 2 for every epoch e and then uses this to bound the deviation of the policy ˜ π e from the optimal choice of policy π ∗ .
In the next section, we generalize this result to the unknown context setup.
3. U NKNOWN STOCHASTIC CONTEXTS
Our stochastic setup assumes that the context vectors { φ ( x t , a t ) } at each time step are sampled i.i.d.
from a known distribution.
We formally state this assumption 3 next.
Assumption 2 (Stochastic contexts from known distribution.)
At each time instance t , the features φ ( x t , a ) for every action a ∈ [ K ] are assumed to be sampled i.i.d.
from the Gaussian distribution
N (¯ x a , Σ a ) , such that α l I (cid:22) Σ a (cid:22) α u I and (cid:107) ¯ x a (cid:107) 2 ≤ 1 .
For large-scale recommendation systems for trafﬁc routing which interact with million of users daily, the above assumption on known distributions is not restrictive at all.
Indeed these systems have a fair understanding of the demographics of the population which interact with it on a daily basis and the real uncertainty is on which person from this population will be using the system at any time.
The algorithm for this setup is similar to the known context scenario where instead of planning with the exact contexts in the optimistic policy computation step, we obtain the episode policy as
[Equation]
where the expectation is taken with respect to the sampling of context.
Our regret bound for this modiﬁed algorithm depends on the mixing time of the policy set Π X in an appropriately deﬁned
Markov chain.
Deﬁnition 2 (Mixing-time of Markov chain) .
For an ergodic discrete time Markov chain M , let d represent an arbitrary starting state distribution and let d ∗ denote the stationary distribution.
The (cid:15) -mixing time τ mix ( (cid:15) ) is deﬁned as
[Equation]
The mixing time of the policy set Π X is given by τ ∗ mix : = max π ∈ Π X max h τ mix ,π ( h ) .
The following theorem establishes the regret bound for the modiﬁed CARCB algorithm, showing that not knowing the context can increase the regret by an additive factor of ˜ O
( (cid:112) ∆ τ ∗ mix · T ) .
Theorem 3 (Regret bound; unknown contexts) .
For any conﬁdence δ ∈ (0 , 1) , congestion window ∆ > 0 and time horizon T , suppose that the context sampling distributions satisfy Assumption 2.
Then, with probability at least 1 − δ , the policy regret of CARCB satisﬁes
[Equation]
A detailed proof of this result is deferred to Appendix B. Observe that the above bound can be seen as a sum of two terms: R T (cid:46) √ dT + (cid:112) ∆ τ ∗ mix T .
The ﬁrst term is a standard regret bound in the d dimensional contextual bandit setup.
The second term, particular to our setup, arises because of the interaction of the congestion window with the unknown stochastic contexts.
In comparison to the bound in Theorem 1, the window size ∆ interacts only additively in the regret bound surprisingly.
The reason for this additive deterioration of regret is that the shared parameter θ ∗ allows us to use data across time steps in our estimation procedure – thus, in effect, the congestion only slows the estimation by a factor of c min which shows up due to the dependence on the minimum eigenvalue.
In order to go from the regret bound in the known context case, Proposition 1, we need to address two key technical challenges: 1) bound the deviation of the reward of policy ˜
π e from the policy which plans with the known sampled contexts, and 2) the context vectors selected by the algorithm φ ( x t , a t ) satisfy the minimum eigenvalue condition in Assumption 1.
Deviation from known contexts.
One way to get around this difﬁculty is to reduce the above problem to the multi-armed bandit on from Section 2.
This simple reduction would lead to a regret bound which scales with the size of the context space |X| which is exponentially large in the dimension d .
Instead of this, we show that the reward obtained by the distribution maximizer ˜
π
e are close to those obtained by the sample maximizer via a concentration argument for random walks on the induced Markov chains.
The key to our analysis is the construction of this random walk using policy ˜ π e and then using the following concentration bound from .
Lemma 1 (Theorem 3.1 in Chung et al.
Let M be an ergodic Markov chain with state space [ n ] and stationary distribution d ∗ .
Let ( V 1 , ..., V t ) denote a t -step random walk on M starting from an initial distribution d on [ n ] .
Let µ = E V ∼ d ∗ [ f ( V )] denote the expected reward over the stationary distribution and X = (cid:80)
i f
( V i ) denote the sum of function on the random walk.
There exists a universal constant c > 0 such that
[Equation]
where the norm (cid:107) d (cid:107) 2 d ∗ : = (cid:80) i
d 2 i d ∗ i .
This concentration bound is not directly applicable to our setup because of two reasons: 1) the constructed Markov chain M ˜ π e might not be ergodic, and 2) the norm (cid:107) d (cid:107) d ∗ might be unbounded in our setup.
By using the fact that the diameter of the MDP M ˜ π e is bounded by ∆ , we use an intermediate policy in the time steps { t e : t e + ∆ } in each episode reach a state starting from which the MDP is shown to be ergodic and have bounded norm (cid:107) d (cid:107) d ∗ .
See Appendix B for details.
Minimum eigenvalue bound.
In order to obtain a regret bound for the stochastic setup, we need to establish that the covariance matrix formed by these context-action features satisﬁes the minimum eigenvalue assumption.
The challenge here is that the the features φ ( x t , a t ) are not independent across time – they are correlated since the algorithm’s choice at time t depends on the history h t which in turn depends on the past features.
In Appendix B we get around this difﬁculty by decoupling these dependencies and showing that even after this decoupling, the random variables still satisfy a sub-exponential moment inequality.
E XPERIMENTAL EVALUATION
In this section, we evaluate both our proposed algorithms, CARMAB and CARCB, in the congested bandit framework and exhibit their no-regret properties.
We generate K arms and assign a base reward of ˆ r j ∈ (0 , 1) to each j ∈ [ K ] .
We draw a noise parameter (cid:15) t,j for every action j and time step t .
We set f cong ( a t , #( h t , a t ))
= 1 / #( h t , a t ) .
We also set the parameter ∆ , which controls the length of the history h t at time t .
Then, the observed reward is ˜ r ( h t , a t )
=
ˆ r at #( h t ,a t ) + (cid:15) t,a t .
We set parameter δ of algorithm CARMAB to 0 .
In terms of distributions, we draw ˆ r j uniformly in (0 , 1) and (cid:15) t,j from N (0 , 0 . 1) .
In Figure 2, we present how the average regret of the algorithm changes as time progresses for K = 4 and different values of the window size ∆ during which congestion occurs.
For evaluating CARCB, again we generate K arms.
For each arm i ∈
[ K ] and each time step t ∈ [ T ] , we draw a random context.
A context x a,t is a vector of 10 numbers which are drawn uniformly in (0 , 1) and then normalized so that the Euclidean norm of the vector is unit.
We also draw the true parameter θ ∗ in the same way.
We assume each arm’s context is available to the algorithm at each time step.
We use the same noise and congestion function as in the previous section.
The observed reward in this setting is:
[Equation]
In Figure 2 we present how the average regret of CARCB changes over time, in a setting with similar
K and ∆ ( K = ∆ = 4 ) and a setting with a number of actions larger than the congestion window ( K = 10 and ∆ = 2 ).
E THICS S TATEMENT
Our contributions in this work are theoretical in nature and we do not see any ethics related issue arising from this work in the foreseeable future.
R EPRODUCIBILITY S TATEMENT
On the theoretical side, we provide detailed proofs for all our theoretical results in the appendix.
For the experimental aspect, we have attached our python code as a supplementary ﬁle.
Furthermore, we have described all necessary hyper parameters and algorithmic details required to reproduce the experiments.