Queue Up Your Regrets: Achieving the Dynamic Capacity Region of Multiplayer Bandits
Abstract
Consider N cooperative agents such that for T turns, each agent n takes an action a n and receives a stochastic reward r n ( a 1 , . .
. , a N ) .
Agents cannot observe the actions of other agents and do not know even their own reward function.
The agents can communicate with their neighbors on a connected graph G with diameter d ( G ) .
We want each agent n to achieve an expected average reward of at least λ n over time, for a given quality of service (QoS) vector λ .
A QoS vector λ is not necessarily achievable.
By giving up on immediate reward, knowing that the other agents will compensate later, agents can improve their achievable capacity region.
Our main observation is that the gap between λ n t and the accumulated reward of agent n , which we call the QoS regret, behaves like a queue.
Inspired by this observation, we propose a distributed algorithm that aims to learn a maxweight matching of agents to actions.
In each epoch, the algorithm employs a consensus phase where the agents agree on a certain weighted sum of rewards by communicating only O ( d ( G )) numbers every turn.
Then, the algorithm uses distributed successive elimination on a random subset of action proﬁles to approximately maximize this weighted sum of rewards.
We prove a bound on the sum of expected QoS regrets of all agents, that holds if λ is a safety margin ε T away from the boundary of the capacity region, where ε T → 0 as T → ∞ .
This bound implies that, for large T , our algorithm can achieve any λ in the interior of the dynamic capacity region, while all agents are guaranteed an empirical average expected QoS regret of ˜ O
(1) over t = 1 , . . .
, T which never exceeds ˜ O (cid:0) √ t (cid:1) for any t .
We then extend our result to time-varying i.i.d.
communication graphs.
Introduction
Consider a group of artiﬁcial cooperative agents interacting in a shared environment.
Each agent has a reward r n ( a ) that is a function of the actions of all agents a and represents a local performance metric.
Examples include wireless devices, robots in a facility, or autonomous vehicles.
The designer wants to program these agents with a distributed protocol that guarantees at least a certain average reward for each agent, which we term its QoS. Wireless devices interact by transmitting and interfering with each other over multiple channels, and the wireless protocol guarantees the advertised throughput for all of them.
Multitasking robots interact by cooperating to make progress on their tasks at a certain rate, following a distributed learning algorithm with this objective in mind.
The main challenge in designing such a distributed protocol or algorithm is that the agents are local and do not know enough about each other to guarantee the QoS of their peers.
In data-driven decision-making, agents do not know their reward function, and can only observe its value at the current action proﬁle a ( t ) , known as bandit feedback.
The agents also do not know the reward functions of their peers, or even their bandit feedback.
As a result, agents have to learn both how to optimize their reward function, and when to prioritize other agents instead.
The QoS is often measured over time.
Throughput in communication networks is measured over seconds, and progress in an automated factory is measured over minutes.
In resource allocation, the optimal policy often combines time-sharing and simultaneous access.
For example, consider four agents and one resource, such that each agent can access the resource or not.
If Agents 1 and 2 access the resource simultaneously, they each get zero reward, and the same is true for Agents 3 and 4.
Otherwise, an agent gets a reward of 1 by accessing the resource.
This simple example arises in many resource allocation applications.
In wireless networks, far away devices can transmit simultaneously on the same channel, but nearby devices have to pick different channels.
In cloud computing, a job that requires little RAM can run simultaneously with a RAM-intensive one, but two
RAM-intensive jobs cannot share the CPU simultaneously.
Similarly, we cannot charge too many electric vehicles that share a transformer simultaneously.
In all these cases, dynamic algorithms can achieve QoS vectors that static multi-agent algorithms cannot, where agents learn when to access, but also, who to access with.
The main objective of this paper is to design a distributed multi-agent algorithm that can achieve any dynamically feasible QoS based only on bandit feedback and minimal communication.
1.1 Our Contributions
We deﬁne the QoS regret of agent n by R n ( t ) = max (cid:110) 0 , λ n t − (cid:80) tτ =
1 r n ( a ( τ )) (cid:111) , given a QoS vector λ .
Our main observation is that R n ( t ) behaves like a queue , which inspires an algorithm that learns to approximate a max-weight matching between agents to actions to guarantee the stability of these “queues”.
A perfect centralized max-weight policy [1] plays the a that maximizes (cid:80)
Nn =1 b n R n ( t ) r n ( a ) for some positive { b n } .
To be able to learn this policy distributedly, our algorithm divides the time horizon of length T into epochs of length E T .
In each epoch, agents use simultaneous distributed averaging (i.e., consensus) steps to agree on (cid:80)
Nn =1 b n R n ( t ) r n ( a ( t + 1)) from recent turns, on a row stochastic communication matrix W .
Then, agents use a distributed and scalable successive elimination to approximately maximize (cid:80)
Nn =1 b n R n ( t ) r n ( a ) .
Our consensus step does not require a doubly stochastic W and converges in ﬁnite time.
We show that for large T , our algorithm can achieve any λ in the interior of the dynamic capacity region, where all agents have an empirical average expected QoS regret of ˜ O
(1) over time which never exceeds ˜ O (cid:0) √ t (cid:1) for any t .
Our algorithm only requires each agent to communicate O ( d ( G ))
numbers each turn, where d ( G ) is the diameter of the communication graph G .
Furthermore, for ﬁnite T , we bound the sum of expected QoS regrets as a function the epoch length E , and characterize the safety margin ε E away from the boundary of the capacity region that is required to make λ feasible.
We then prove QoS regret guarantees for i.i.d.
time-varying stochastic communication graphs, modifying both the bound on the expected sum of QoS regrets and ε E .
For this more complicated case, the communication overhead is O (cid:0) log 2 E (cid:1) per agent per turn.
1.2 Related Work
Our work brings together game theory, learning, and queuing theory.
Aspects of learning in queuing were studied in [2–5].
Queuing games were studied in [6], where each agent chooses which queue to join.
In contrast, our analysis uses the observation that the QoS regret behaves like a queue, but no real backlogs exist.
We consider general discrete games that may have nothing to do with queues.
Multiplayer bandits is an emerging ﬁeld that studies how agents can learn to achieve a common goal [7–10] given bandit feedback of their unknown reward functions.
Most works on multiplayer bandits assume the collision model, in which when multiple agents choose the same arm, they all receive zero reward [11–16].
Instead of the sum of rewards, our objective is to guarantee that the empirical average reward of each agent is at least a certain QoS. Multi-player QoS bandits were studied in [17], assuming the collision model.
However, [17] only achieves QoS vectors λ for which there exists an a such that E { r n ( a ) } ≥ λ n for all n .
Compared to [17], our work considers a general discrete game.
But even for the collision model, for the example on the top of this page, two agents get zero reward in any static solution like [17], while we can achieve λ n = 0 .
5 for all n .
Dynamic multiplayer bandits, where agents can join at different times, were studied in [11, 14, 16] under the collision model.
More related to our work, [18] combined queuing with multiplayer bandits.
In [18], each agent receives packets randomly, and each packet is transmitted to a server of choice, where servers have constant service rates and can only process one packet at a time.
In contrast, our scenario has no actual queues and the “service rates” are the rewards of a general discrete game.
Consequently, we need to learn a max-weight matching where in [18] a maximal matching of agents to servers is optimal.
In power control, [19] can achieve any target QoS vector that is achievable by a static transmission power proﬁle.
The interaction between the wireless links, through the interference, is a game with the transmission powers as the actions.
From this point of view, our work generalizes [19] in two ways: from power control to a general game (but discrete), and from a static to a dynamic solution.
Problem Formulation
We consider a general multiplayer multi-armed bandit game.
At every turn t ≥ 1 , for T turns, each agent n in the set N = { 1 , ..., N } picks an action a n ( t ) ∈ { 0 , . .
, K − 1 } .
The expected reward for agent n is an unknown function of the actions of all agents µ n : { 0 , . .
, K − 1 } N → R .
Each agent n only observes a stochastic reward, r n ( a ( t )) ∈ [0 , 1] , such that µ n ( a ) =
E { r n ( a ) } and z n ( t ) (cid:44) r n ( a ( t )) − µ n ( a ( t )) is independent over time and between agents.
Agents only know the action they played and cannot observe the actions of others.
Our goal is to design a distributed algorithm for the cooperative agents, that dictates the actions a n (1) , . .
. , a n ( T ) of each agent n , such that ideally
[Equation]
i.e., the empirical average of the expected reward of agent n is at least λ n for all n .
The objective in (1) is only possible for certain QoS vectors λ 1 , . .
. , λ N .
The achievability region of the dynamic problem in (1) can be much larger than the static one which includes all λ for which max a min n ( µ n ( a ) − λ n ) ≥ 0 .
As an example, consider a game with N = 2 and K = 2 , such that µ 1 (1 , 0) = 1 , µ 1 (0 , 0)
= µ 1 (0 , 1) = µ 1 (1 , 1) =
and µ 2 (0 , 1) = 1 , µ 2 (0 , 0) = µ 2 (1 , 0)
= µ 2 (1 , 1) = 0 .
Then λ ∗ = max a min n µ n ( a ) = 0 is obtained for any ﬁxed a 1 , a 2 , and no λ 1 , λ 2 > λ ∗ are feasible.
However, if the agents play (1 , 0) in even turns and (0 , 1) in odd ones then 1 T (cid:80) Tt =1 µ 1 ( a ( t ))
= 1 T (cid:80) Tt =
1 µ 2 ( a ( t ))
= 12 > λ ∗ .
This simple example arises often when time-sharing of resources is better than simultaneous access.
In general, the optimal strategies can involve softer time-sharing or time-sharing between groups of agents, based on their characteristics.
This motivates deﬁning the dynamic capacity region [1]:
Deﬁnition 1.
Let ε ≥ 0 .
The approximate dynamic capacity region of the game G = (cid:0)
N , { 0 , . . .
, K − 1 } n ∈N , { µ n } n ∈N (cid:1) is deﬁned as the set:
[Equation]
where p ( a ) is the probability of a and ∆ K N is the K N -th dimensional simplex.
The (exact) dynamic capacity region is then C ( G ) (cid:44) [Equation]
A learning algorithm
that has no knowledge of the game will need time to achieve (1), so we can only hope to achieve (1) up to some T -dependent error, which we term the QoS regret.
Deﬁnition 2.
The QoS regret of agent n at turn t is deﬁned as
[Equation]
The expected QoS regret encodes that an agent “regrets” the total accumulated reward it misses to have an expected empirical reward of at least λ n .
On the other hand, an agent with an empirical average reward of more than λ n does not require anything more so it has zero QoS regret.
Note that if the expected QoS regret of all agents is zero then (1) holds with probability 1.
Our contributions are a novel distributed algorithm that can learn to achieve a target QoS vector, with the following performance guarantees:
Theorem 1.
Let G be a multiplayer multi-armed bandit game, played for T turns.
Let the agents run Algorithm 1 on a communication graph G , with epoch length E T such that E T → ∞ and E
T T → 0 as T → ∞ .
If λ ∈ C ( G ) , then for large enough T , we have 1 T (cid:80) T − 1 t =
E { R n ( t ) } =
O ( E T ) for all n and E { R n ( t ) } =
O (cid:0) √ tE T (cid:1) for all n and t , where the expectation is with respect to the randomness of the algorithm and the rewards.
To achieve this, each agent only needs to communicate O ( d ( G ))
numbers 1 per turn, where d ( G ) is the diameter of G .
The QoS regret guarantees of Theorem 1 are asymptotic in T .
For ﬁnite T , a tradeoff arises between E { R n ( T ) } and the safety margin ε T we have to take from the boundary of C ( G ) to guarantee this E { R n ( T ) } .
Theorem 2 in the next section analyzes this tradeoff explicitly and shows that ε T →as T → ∞ , which implies Theorem 1 above as a special case.
By tuning the epoch length E , we can make ε T smaller for ﬁnite T , at the price of increasing the expected QoS regret, or vice versa.
As long as E T increases with T , then ε T → 0 and our algorithm can achieve any λ ∈ C ( G ) for large enough T .
Hence, if E T = log T (or slower), then Theorem 1 implies that the empirical average expected QoS regret of all agents is ˜ O (1) and no expected QoS regret exceeds ˜ O (cid:0) √ t (cid:1) for any t .
For ﬁnite E , these two bounds slightly improve to O (1) and O (cid:0) √ t (cid:1) respectively, as long
λ ∈ C ε E ( G ) .
In Section 12 in the appendix, we generalize Theorem 1 for a stochastic time-varying i.i.d.
communication graph where only the union of the support set is assumed to be connected.
This requires some slight modiﬁcations to our algorithm, detailed in Algorithm 2, among them using D E
= O (cid:0) log 2 E (cid:1) .
As a result, each agent needs to communicate O (cid:0) log 2 E (cid:1) numbers per turn.
It might look tempting to view (1) as the two-player zero-sum game min p ∈ ∆ KN max n ( λ n − (cid:80) a p ( a ) µ n ( a )) .
Solving zero-sum games with bandit feedback is an active ﬁeld of research [20, Chapter 9].
The idea is that if the column and row players both use a no-regret algorithm, then the empirical distribution of the action proﬁles is an (cid:15) T -Nash equilibrium (NE) with a vanishing (cid:15) T , and the empirical average cost of the row player converges to the value of the game.
The convergence rate is then the regret of the row player, divided by T .
It is an open question whether a convergence rate better than O (cid:16) T − 12 (cid:17) is possible [21,22], which is achievable if both players use EXP3.
To employ this technique in our scenario, one agent will have to operate as a leader that virtualizes the row player that picks p and the column player that picks n .
This leader agent then broadcasts the chosen action proﬁles, which are propagated over G , so each agent sends O ( Nd ( G ))
numbers every turn.
This approach, however, would fail to provide any QoS regret guarantees.
First, the average cost of the row player is not the QoS regret of any of the agents in our game.
Second, the fact that the distribution of play p ( T ) is an (cid:15) T -NE does not provide any
Euclidean distance convergence rate of p ( T ) to the exact NE.
Hence, there is no bound on the incurred QoS regret of p ( T ) .
Remarkably, our queuing approach to solve (1) not only circumvents these issues, but achieves, for any λ ∈ C ( G ) , an ˜ O (1) empirical average expected QoS regret over t = 1 , . . .
, T for all agents, while the expected QoS regret is never worse than ˜ O (cid:0) √ t (cid:1) for any t and n .
Furthermore, our communication overhead is only O ( d ( G )) per agent per turn.
Dynamic Multi-player Bandit Algorithm
To tackle the problem in (1), we exploit the similarity between the QoS regret and a queue.
We can think of a deterministic arrival ﬂow with rate λ n and a service rate of r n ( a ( t )) for agent n at turn t .
Then we can employ ideas from queuing theory to design our algorithm.
In particular, max-weight matching of queues to service rates is a centralized algorithm that achieves optimal throughput [23].
Note that in our case the service rate (the reward) is a function of the actions of all agents.
Hence, our algorithm has to learn to approximate a max-weight matching distributedly for our game model.
Our algorithm, detailed as Algorithm 1, divides the horizon into epochs.
In each epoch, the agents treat their QoS regrets as ﬁxed on their initial value, which is an approximation (Step 1).
Based on these values the agents distributedly learn to maximize (cid:80) Nn =1 b n R n ( t ) µ n ( a ) for some positive { b n } , using successive elimination (see Lemma 3).
Much of the algorithm’s efﬁciency stems from the fact that we do not care what { b n } are, as long as they are positive and constant over epochs.
For stochastic time-varying i.i.d.
communication graphs (Section 12 in the appendix), b n ( t ) will be time-varying as well, and then we need its expectation to be positive and constant over epochs.
Algorithm 1 has a low runtime complexity per turn of O ( ν n D E + S E ) for agent n , which only computes linear combinations of values from its ν n neighbors for D E turns simultaneously, and goes over a list of S E action proﬁles.
Initialization: Set the epoch index to e = 0 and the successive elimination index to l = 0 .
Let W be the communication matrix (Deﬁnition 3).
Let S E = min (cid:8) (cid:100) S
log
E (cid:101) , K N , E (cid:9) , for some S 0 > 0 , be the random subset size and D E > 0 be the delay parameter.
Let b i,j = (cid:8) W D E (cid:9) i,j for all i, j .
Input: λ n ≥ 0 and agent index n
For each epoch e ≥ 0 , each agent n runs:
Compute the QoS regret for epoch e , R en = max (cid:110) 0 , λ n t − (cid:80) tτ =
1 r n ( a ( τ )) (cid:111) .
Randomize actions a 1 n , . .
. , a S E n , uniformly and independently over { 0 , . .
, K − 1 } .
Store their indices in the ordered set B = { 1 , . .
, S E } , where B ( l ) is the l -th element.
Set v i
= ˆ µ i = 0 for all i = 1 , . .
. , S E , set ˜ r n = 1 and T e = eE
+ 1 . 4.
For T e ≤ t ≤ T e + E − 1 ( E turns): (a) Set θ 0 n = ˜
r n R en and θ 0 m = 0 for all m (cid:54)
= n .
(b) Broadcast { ( θ τn , τ ) } D E − 1 τ = to neighbors and receive { ( θ τm , τ ) } D E − 1 τ = from neighbors.
(c) Parallel D E Consensus Steps : Set θ τ +1 n
[Equation]
(d) Agreement : If you know θ τ 1 for D E ≤ τ ≤ 2 D E − 1 then broadcast ( θ τ 1 , τ ) .
(e) If you receive ( θ τ 1 , τ ) for D E ≤ τ ≤ 2 D E − 1 then set the weighted sum of rewards from τ turns ago as S ( t − τ )
=
θ τ 1 .
(f) Play a n ( t ) =
a B ( l ) n and set ˜ r n = r n ( a ( t )) .
(g) Update the number of visits v k ← v k +1 to a k = (cid:0) a k 1 , . .
, a kN (cid:1) for k = B ( l − 2 D E ) .
(h) Compute the conﬁdence radius ρ k =
[Equation]
k
[Equation]
and
[Equation]
k ←
[Equation]
(i) Elimination : If l = |B| then delete from B all i such that U
i < max j ∈B L j , and set l = 0 . (j) Update the successive elimination index l ← l + 1 . 5. Update the epoch index e ← e + 1 .
End
3.1 Communication
To know the reward values of others, agents need to communicate.
We assume that agents can send numbers over a communication graph G , weighted by a row-stochastic communication matrix W :
Deﬁnition 3.
Let G be the adjacency matrix of an undirected connected graph, such that G n,m =if agent n and agent m can communicate or that n = m .
Let ν n be the degree of agent n in G .
Let
W be a matrix such that w n,m >
if and only if G n,m = 1 and (cid:80) Nm =1 w n,m = 1 for all n .
A key design question is how much should agents know about each other to achieve (1).
A maxweight matching at turn t is an a that maximizes (cid:80) Nn =
1 b n R n ( t ) µ n ( a ) for some positive { b n } .
A naive choice is then to guarantee that all agents know all rewards before taking an action.
This will require each agent to send O ( d ( G ) N ) numbers every turn - one for every agent and every turn of a reward value that is still crossing G .
This is infeasible in large-scale networks.
Instead, our algorithm requires a communication overhead of O ( d ( G )) , without sacriﬁcing performance.
This dramatic save in communication is possible since to guarantee stability, agents do not need to have the same coefﬁcient b n in the sum.
Hence, in Algorithm 1, agents run the consensus protocol only until all local averages include all { R en r n ( a ( t )) } n (where R en ≈ R n ( t ) ), even if not with equal b n .
In contrast to the distributed optimization literature [24], we do not need the communication matrix
W to be doubly stochastic, which is hard to guarantee distributedly [25].
Since W is not doubly stochastic, our consensus cannot converge to the average reward.
In fact, since we stop the consensus early after exactly D E turns, it does not converge at all.
However, after our consensus step, each agent has a weighted sum of rewards that includes all agents that have a positive QoS, with weights that depend on W .
In the next D E turns, agents propagate (Steps 4(d) and 4(e))
the weighted sum of rewards that Agent 1 obtained, such that they all eventually know it with zero error.
Our algorithm introduces delays so that agents can agree on (cid:80)
Nn =1 b 1 ,n R en r n ( a ( t + 1)) ,
that
Agent 1 obtained after the distributed averaging step (Steps 4(b) and 4(c)).
If the diameter of G is d ( G ) , then a minimum of d ( G ) turns is necessary to guarantee that all agents contributed to (cid:80)
[Equation]
n R en r n ( a ( t + 1)) .
However, it is unnecessary to continue the consensus after this point, which is a major advantage of our consensus scheme over the classical one.
In Steps 4(b)-4(e), agents pipeline D E simultaneous consensus algorithms that cannot be mixed since they are on the rewards generated at different turns from different action proﬁles.
In a connected graph d ( G ) ≤ N so D E = N is always sufﬁcient.
However, usually, a tighter bound on d ( G ) is available.
For a random Erd ˝ os–R ´ enyi graph G ( N, p ) , d ( G ) =
O (cid:16) log N log( Np ) (cid:17) with high probability [26], and G ( N, p ) is connected with high probability for p =
c log N N with c > 1 .
In our scenario, an apriori bound on (cid:80) Nn =1 b 1 ,n R en r n ( a ) , the effective reward for the successive elimination, is unknown to the agents but is needed to adjust the conﬁdence radius ρ k .
Hence, at the beginning of each epoch
(Step 4(a)), agents start a consensus averaging on { R en } n to obtain (cid:80)
Nn =1 b 1 ,n R en , which bounds (cid:80) Nn =1 b 1 ,n R en r n ( a ) , but varies between epochs.
3.2 Distributed and Scalable Successive Elimination
In the second part of every epoch (Steps 4(f)-4(j)), agents use the values they got in the communication step to run distributed successive elimination on a random subset of the action proﬁles.
In our case, the effective reward of “arm” a at epoch e is (cid:80) Nn
=1 b 1 ,n R en r n ( a ) .
In Step 2, each agent prepares a sequence of S E actions independently and uniformly at random.
This induces a sequence of S E action proﬁles that all agents agree on their indices B = { 1 , . . .
, S E } (which can include duplicates).
Therefore, all agents count the same number of visits v a ( t ) for all a in Step 4(g).
Moreover, all agents agree on the same weighted sum of rewards for the action proﬁle they played 2 D E turns ago, denoted by ˆ µ k for k = B ( l − 2 D E ) .
Hence, all agents compute the same conﬁdence radius ρ k in Step 4(h), for all the action proﬁles indexed in B .
As a result, all agents agree on when to eliminate an action proﬁle from B in Step 4(i) and will skip it in their next cycle over the action proﬁles, which is indexed by l .
Restricting the search to a random subset of action proﬁles is done to improve the scalability of the algorithm, as we discuss in detail in Section 7 of the appendix.
For S E = K N , no randomization is needed as agents can go over all the action proﬁles sequentially.
3.3 QoS Regret Guarantees
Theorem 2 below bounds the expected QoS regrets of the agents.
The proof is given in the appendix.
If the epoch length E T increases with T , then the theorem shows that E { R n ( t ) } =
O (cid:0) √ E T t (cid:1) for any n, t and any λ ∈ C ( G ) .
However, this does not provide a guarantee on the fraction of time during which the QoS regret of a speciﬁc agent is “large”.
A highly ﬂuctuating QoS regret means that an agent has to wait a long time to be “compensated” for when it was patient in favor of others.
Hence, we also provide a general bound on 1 T (cid:80) T − 1 t =
(cid:80) Nn =1 b 1 ,n E { R n ( t ) } , for positive { b 1 ,n } that are speciﬁed below.
Using this bound, the ﬁrst part of Theorem 2 assumes a ﬁxed epoch length E and that λ ∈ C ε E ( G ) for ε E in (5), so λ is far enough from the boundary of C ( G ) .
In this case, we obtain lim sup T →∞ 1 T (cid:80) T − 1 t =
(cid:80) Nn =1 E { R n ( t ) } < ∞ , so the empirical average of the expected QoS regret of any agent is O (1) .
In the queuing literature, these stability notions are known as mean rate stability (i.e., sublinear expected QoS regret) and strong stability.
Theorem 2.
Let G be a multiplayer multi-armed bandit game, played for T turns.
Let λ ∈ C
( G ) .
Let the agents run Algorithm 1 with a row-stochastic W , a connected G with diameter d ( G ) , epochs of length E , a random subset of action proﬁles of size S E ≤ (cid:100) S
0 log E (cid:101) for some S 0 > 0 and
D E ≥ d ( G ) .
Let b i,j = (cid:8) W D E (cid:9) i,j
. Deﬁne the set of approximately optimal action proﬁles
[Equation]
and
[Equation]
Deﬁne
[Equation]
If λ ∈ C ε E ( G ) then the QoS regrets (Deﬁnition 2) satisfy
[Equation]
where E {·} is over the rewards and the random subsets, and b 1 ,n > 0 for all n .
Therefore, for any λ ∈ C ε E ( G ) , Algorithm 1 achieves O (1) empirical average expected QoS regret for all n ,i.e., lim sup T →∞ 1 T (cid:80) T − 1 t =
(cid:80) Nn =1 E { R n ( t ) } < ∞ (strong stability) and E { R n ( t ) } =
O (cid:0) √ t (cid:1) for all n, t .
Furthermore, if we pick E T such that E T → ∞ and E T T → 0 as T → ∞ , then for any λ ∈ C ( G ) , Algorithm 1 achieves 1 T (cid:80) T − 1 t =
(cid:80) Nn =1 E { R n ( t ) } =
O ( E T ) for all n and
E { R n ( t ) } =
O (cid:0) √ E T t (cid:1) for all n, t (mean rate stability).
QoS Regret Stability Analysis
In this section, we break our analysis required to prove Theorem 2 into lemmas and postpone the proofs to the appendix.
The proof technique is inspired by the optimality proof of max-weight matching [23].
However, the QoS regret R n ( t ) does not behave like a standard queue.
The main difference is that the service rate of agent n , which is its reward function, is not selected by a centralized decision-maker but is a function of the actions of the distributed agents.
As opposed to a standard queue, the “backlog” of the QoS regret is not a sufﬁcient statistic, since one has to remember λ n t − (cid:80) tτ
=
1 r n ( τ ) even if R n ( t ) = 0 .
Since λ n t − (cid:80) tτ =
1 r n ( τ ) can get very negative “empty queues” ( R n ( t ) = 0 ) are much more likely in our problem.
As we discuss in Sectionin the appendix, this fact can be used to improve the scalability of our algorithm, since agents with
R n ( t ) = 0 can be thought of as “passive”.
Moreover, our “arrival process” is deterministic, of λ n < 1 reward points every turn for agent n , and the backlog is not an integer.
As detailed next, some of these differences require careful analysis or affect the algorithm design.
For example, while a sublinearly increasing queue (in T ) is not strongly stable, in our case, as long as R n ( T ) T → 0 , agent n will asymptotically achieve its desired QoS in the empirical average sense.
Deﬁnition 4.
Deﬁne the matrix B = W D E , and denote its ( i, j ) element by b i,j .
Deﬁne the following Lyapunov function such that L (0) = 0 and for all t ≥
[Equation]
L ( t ) only uses the coefﬁcients of Agent 1 since all agents, that know their IDs, agree on the weighted sum of rewards that Agent 1 sees.
This forced agreement ensures that all agents will use the exact same weighted sum of rewards.
However, Agent 1 does not know more than others and can be replaced by any of them.
The ﬁrst lemma bounds the Lyapunov drift in each turn.
Lemma 1.
The Lyapunov drift at turn t ≥ 0 satisﬁes:
[Equation]
The proof of Theorem 2 follows by upper bounding E { L ( T ) } − E { L (0) } , where a smaller bound implies smaller QoS regrets.
The upper bound given by Lemma 1 is minimized by the policy
[Equation]
is known as maximum weight matching.
The two main obstacles our algorithm has to overcome to approximate max-weight matching are the distributed nature of the agents, and the unknown reward functions (bandit feedback).
Distributed agents cannot deduce a ∗ ( t + 1) instantly every turn since they need time to agree on the weighted sum of rewards.
For this reason, our algorithm works in epochs.
Working in epochs means that our algorithm treats the QoS regret R n ( t ) as constant on its initial value R en (cid:44) R n ( T e ) throughout epoch e .
The agents can then aim to solve the following problem instead in every epoch
[Equation]
The next lemma bounds the resulting error from solving (10) instead of the max-weight matching.
Lemma 2.
Deﬁne e ( t ) as the epoch such that T e ( t ) ≤ t ≤ T e ( t ) + E − 1 .
Then
T − 1 (cid:88)
[Equation]
In addition to solving (10) instead of the max-weight matching, working in epochs incurs an additive error of O ( ET ) again while replacing R n ( t ) with R en while bounding the drift in (9).
The other loss of Algorithm 1 stems from learning a ∗ e with bandit feedback.
While dividing into epochs creates an additive error of O ( ET ) , learning the optimal action proﬁle a ∗ e creates a multiplicative error, so the larger the QoS regrets R en are, the larger this error is.
For this reason, we need to employ an algorithm that achieves sublinear regret in learning a ∗ e .
Algorithm 1 induces successive elimination on a random subset of S E action proﬁles, with (cid:80)
Nn =1 b 1 ,n R en r n ( a ) as the reward function.
The next lemma bounds the error (regret) of this successive elimination.
Since agents only agree on (cid:80) Nn =1 b 1 ,n R en r n ( a ( t )) after 2 D E turns, our successive elimination suffers from delayed feedback.
While not our focus, we chose successive elimination over UCB since it is more robust to delays, as was shown in [27].
The restriction to a random subset of S E action proﬁles is done to improve scalability, but it leads to a worse regret than in the single-agent standard multi-armed bandits.
Nevertheless, the regret is still sublinear in E and therefore good enough to prove our QoS regret bounds.
Lemma 3.
Let R en = R n ( T e ) and deﬁne the set of approximately optimal action proﬁles as
[Equation] [Equation]
Let a ∗ e ∈
A ∗ e .
Deﬁne the ﬁltration F e = σ (cid:0)(cid:8) a ( t ) , { z n ( t ) } n ∈N | t ≤ T e (cid:9)(cid:1) .
Then Algorithmmaintains
E
[Equation]
where E {· | F e } is over the rewards and the e -th subset of action proﬁles.
Numerical Simulations
Our algorithm is useful in any case where cooperative agents need to achieve a target QoS over time.
Next, we tested our algorithm for wireless channel access and multitasking robots, two applications where smart time-sharing can signiﬁcantly improve performance.
For each experiment, we plot the sum of QoS regrets averaged over 100 realizations with one standard deviation shaded region.
5.1 Wireless Channel Access
Consider N devices and K − 1 channels, such that each device can choose to transmit a packet on channel k ≥ 1 by playing a n = k or not to transmit at all by playing a n = 0 .
We consider a “soft collisions” model where the success probability of transmitting a packet for device n on channel k is p kn = g kn,n 1+ (cid:80) Nm =1 g m,n 1 { a m = k } for some channel gains { g n,m } and (cid:8) g kn,n (cid:9) .
This model generalizes the collision model [11,13, 14], where multiple devices that pick the same channel all receive zero reward.
As in practice, the geometry of the devices dictates how much they interfere with each other.
Fig.
1(a) shows the sum of QoS regrets over time for N = 4 and K = 3 .
We used S E = 128 , d ( G ) = 4 and E = 50 S E .
The channel gains were g 1 , 4 = g 1 , 3 = g 2 , 4 = g 3 , 2 =
and g 1 , 2 = g 3 , 4 = 100 and g 11 , 1 = g 12 , 2 = g 23 , 3 = g 24 , 4 = 1 , such that g n,m = g m,n for all n, m , and g 21 , 1 = g 22 , 2 = g 13 , 3 = g 14 , 4 = 0 .
Hence devices 1 and 2 are in proximity, and devices 3 and 4 are also in proximity, but far away from 1 and 2.
Then, devices 1 and 2 prefer channel k = 1 while devices 3 and 4 prefer channel k = 2 .
The QoS vector was λ n = 0 .
499 for all n , which cannot be achieved with a static solution, but only if devices 1 and 3 transmit simultaneously each on their preferred channel, and in the next turn devices 2 and 4 do the same, and so on in a round-robin manner.
After a while, the QoS regrets stabilized on low values.
Fig. 1(b) shows the sum of QoS regrets over time for N = 100 and K = 2 (one channel).
The channel gains { g n,m } and (cid:8) g kn,n (cid:9) were chosen independently and uniformly at random on [0 , 1] .
We used S E = 10 4 , E = 5 S E , and d ( G ) = 4 .
The QoS of agent n was λ n = 1 .
9 p 1 n − 0 .
02 N , which is 90% more than what an almost perfect time-sharing can achieve.
The stability of the QoS regrets is again evident, in agreement with the theory.
Note that while there are 2 100 action proﬁles, considering only a random subset of 10 4 action proﬁles was enough to achieve this QoS.
5.2 Multitasking Robots
Consider N robots and K = N tasks, such that the n -th robot is in charge of the n -th task.
Each robot n has a skill of s kn for the k -th task.
Each turn, each robot can choose to work on the k -th task by playing a n = k .
The reward of robot n is r n ( a )
=
(cid:80) Nm =1 1 { a m = n } s nm + z n , which models the rate at which this task progresses, where z n is a uniform noise on [ − 0 . 2 , 0 .
2] that is i.i.d.
over time.
Hence, only one robot can measure the progress of a certain task, but the others can still assist.
Fig. 2(a) shows the sum of QoS regrets over time for N = K = 4 .
We used E = 5 K N = 5120 , d ( G ) = 3 .
The QoS vector was λ
= (0 . 8 , 0 . 2 , 0 . 4 , 0 . 4) and the skill matrix was  
[Equation]
[Figure]
[Equation]
[Figure]
2 0 .  (e.g., s 41 = 0 . 2 ).
As a result, λ 1 is larger than the maximal skill of a single robot so the robots had to cooperate on their tasks to achieve this QoS vector.
Additionally, no static solution can support this λ .
Fig. 2(b) shows the sum of QoS regrets for N = 8 and K = 8 .
We used S E = 1000 , d ( G ) =and E = 10 S E , λ
= (0 . 83 , 0 . 79 , 0 .
and picked s k n uniformly and independently at random on [0 . 2 , 1] .
Note that while there are 8 8 action proﬁles, considering only a random subset of 10 3 action proﬁles was enough to achieve this QoS.
In both ﬁgures, the sum of QoS regrets increases slowly with T , which for a traditional queue would mean no strong stability but in our case means that the empirical average rewards converge to λ .
Conclusion
We studied how a team of agents can improve the Quality of Service (QoS) of each by inducing a dynamic behavior between them.
The main observation that inspired our algorithm is that the QoS regret of each agent behaves like a queue.
Then, our distributed algorithm learns to approximate a max-weight matching of agents to actions to maximize the “throughout” going out of these virtual queues.
To that end, our algorithm works in epochs and employs simultaneous consensus averaging and distributed successive elimination on a random subset of action proﬁles and with delayed feedback.
Our main result shows that our algorithm can achieve all QoS vectors in the capacity region for large enough T , where all agents have an empirical average expected QoS regret of ˜ O
(1) over time which never exceeds ˜ O (cid:0) √ t (cid:1) for any t .
Simulations demonstrate the effectiveness of our algorithm for channel access in wireless networks and multitasking robots.
We then extended our result for stochastic communication graphs that are i.i.d.
over time.
The communication graphs create dependencies between the rewards of the “arms” (i.e., action proﬁles) over time.
While we were able to bypass these dependencies for the i.i.d.
case, a Markov graph process, for example, will lead to a restless bandits scenario instead where successive elimination does not work.
While in principle it is easy to replace the successive elimination part of our algorithm, the new bandit algorithm will have to be distributed, ideally with a small communication overhead.
We considered an arbitrary discrete game where an exhaustive search is necessary to ﬁnd an approximately optimal action proﬁle.
However, applications often focus on more structured games.
For example, in congestion games, best-response dynamics are guaranteed to converge to a Nash equilibrium.
Hence, if a congestion game is designed such that all Nash equilibria are efﬁcient, an exhaustive search is not needed.
Alternatively, the agents can directly solve their combinatorial optimization problem as a team, leveraging its structure.
Examples are the assignment problem, unconstrained submodular minimization, and shortest path [28].
Then the challenge is to design a distributed algorithm to solve the combinatorial optimization based only on bandit feedback.
Our work introduces an analogy between multi-agent QoS and queuing.
By further studying this analogy we can leverage more tools from queuing theory to design dynamic distributed learning algorithms with strong performance guarantees.
For example, scheduling algorithms with delay guarantees [29] can help to design QoS multi-agent algorithms with better delay performance.
Another example is to employ the “drift plus penalty” technique [23] to optimize an objective such as the sum of rewards while still guaranteeing some QoS for all agents.