It Doesn’t Get Better and Here’s Why: A Fundamental Drawback in Natural Extensions of UCB to Multi-agent Bandits
Abstract
We identify a fundamental drawback of natural extensions of Upper Conﬁdence
Bound (UCB) algorithms to the multi-agent bandit problem in which multiple agents facing the same explore-exploit problem can share information.
We provide theoretical guarantees that when agents use a natural extension of the UCB sampling rule, sharing information about the optimal option degrades their performance.
For
K the number of agents and T the time horizon, we prove that when agents share information only about the optimal option they suffer an expected group cumulative regret of O
( K log T + K log K ) , whereas when they do not share any information they only suffer a group regret of O
( K log T ) .
Further, while information sharing about all options yields much better performance than with no information sharing, we show that including information about the optimal option is not as good as sharing information only about suboptimal options.
Introduction
Sequential decision making in multi-agent systems is a subclass of collective learning problems with a wide variety of real-world applications ranging from ecology to machine learning systems [17, 4, 5].
A crucial part of developing high performance algorithms for these systems is modifying individual decision-making strategies to improve performance by leveraging shared information.
Natural extensions of Upper Conﬁdence Bound (UCB) algorithms constitute a widely used class of algorithms in multi-agent sequential decision making.
Generally, we expect any well designed multi-agent algorithm to provide better or comparable performance when more information is shared about any and all decisions.
This gives rise to the following question.
Does more information always provide improved or comparable performance?
We answer this question by proving that performance of any natural extension of UCB algorithms degrades when information about the optimal decision is shared.
UCB algorithms, initially proposed by [2], are used in the stochastic multi-armed bandit (MAB) problem [15, 8, 1], a mathematical model that captures the trade-off between exploring and exploiting in sequential decision making under random payoffs.
Consider an agent repeatedly faced with the problem of choosing an option from the same ﬁxed set of options.
At each time step, the agent chooses an option and receives a numerical reward drawn from a probability distribution associated with its chosen option.
The goal of the agent is to maximize the cumulative reward at the end of the decision-making process.
To realize this goal, the agent should choose the optimal option, i.e., the option with the maximum expected reward.
However, the agent is unaware of the statistical properties of probability distributions associated with the options.
So, instead, the agent needs both to choose options with seemingly high expected rewards (exploit) and to choose lesser known options to learn the true expected rewards (explore).
In the UCB algorithm, at each time step t for each option i , the agent constructs an objective function
Q i ( t ) = (cid:98) µ i ( t ) + C i ( t ) , where C i ( t ) =
σ i (cid:115) 2( ξ + 1) log t n i ( t ) .
Here (cid:98) µ i ( t ) is the agent’s estimate of the expected reward of option i and n i ( t ) the number of samples taken from option i at time t .
σ i is the variance of the reward probability distribution associated with option i and ξ is a constant.
The term C i ( t ) denotes the uncertainty associated with the estimate.
At each time step, the agent chooses the option that maximizes the objective function: the corresponding sequence of choices addresses the trade-off between exploiting and exploring.
In natural extensions of UCB algorithms to multi-agent bandit settings in which agents can share information, each agent uses the information it receives from other agents, in addition to the information obtained through its own sampling, to compute the objective functions.
Thus, through information sharing, agents reduce the uncertainty associated with estimates of expected reward of the options.
To meet their goal, agents sample the optimal option much more frequently than they do suboptimal options.
This implies that if they are sharing information about the optimal option, uncertainty associated with the estimate of the optimal option reduces signiﬁcantly as compared to uncertainty associated with the estimates of suboptimal options.
Thus, the C i ( t ) term corresponds to the optimal option becomes signiﬁcantly smaller than the C
i ( t ) terms correspond to suboptimal options.
Since the objective function consists of the sum of estimate and uncertainty, this reduces the number of times agents sample the optimal option as compared to the case in which agents do not share information about the optimal option.
As a result, agents perform better when they do not share any information as compared to the case in which they share information only about the optimal option.
Likewise, while sharing information about all options is better than no sharing, agents perform better when they share information only about suboptimal options.
Related work With a few exceptions, the work related to UCB type sampling rules for multi-agent bandits use natural extensions [3, 6, 10–14].
The papers [10, 11] use a running consensus algorithm to update estimates and provide graph-structure-dependent performance measures that predict the relative performance of agents and networks.
The paper [11] also addresses the case of a constrained reward model in which agents that choose the same option at the same time step receive no reward.
In [14] the authors propose an accelerated consensus procedure in the case that agents know the spectral gap of the communication graph and design a decentralized UCB algorithm based on delayed rewards.
The paper [12] considers the case where agents observe their neighbors according to a modiﬁed Erd ˝ os Rényi communication graph.
In [3] the authors consider that at each time step, agents decide either to sample an option or to broadcast the last received reward to the entire group.
In [13], each agent broadcasts its reward and action to its neighbors when it is exploring.
The paper [6] considers multi-agent bandits with collision where agents that choose the same option at the same time receive zero reward.
The papers that propose exceptions to natural extensions [7, 9] consider sampling protocols where agents imitate the actions of others.
A strategy where agents observe the rewards and choices of their neighbors according to a leader-follower setting is considered in [9].
The paper [7] considers a case where agents copy the actions of their nearest leader.
However, this line of work suffers from the limitation that in order to imitate, each agent is required to possess knowledge about the sampling rules of the agents it is imitating.
Key contributions Our paper is the ﬁrst work to highlight a fundamental drawback of natural extensions of UCB algorithms to multi-agent setting.
Our contributions include the following:
• We show that performance of a group of agents faced with the stochastic bandit problem degrades when they exchange information about the optimal option.
• We prove that agents suffer an expected cumulative group regret of O
( K log T + K log K ) , where expected regret is deﬁned as the expected loss agents suffer by sampling suboptimal options instead of the optimal option.
Here K is the number of agents and T is the time horizon of the decision-making process.
• We show that, while sharing information about all options is signiﬁcantly better than sharing no information, performance of the group further improves when they share information only about suboptimal options.
Mathematical Formulation
We consider the distributed MAB problem with N options and K agents.
The reward of each option i ∈ { 1 , . .
, N } is modeled with a sub-Gaussian random variable X i , which includes widely used distributions such as Bernoulli, Gaussian, and bounded rewards.
Let µ i and σ 2 i be the expected reward and variance proxy associated with option i , respectively.
In this paper we assume that the variance proxy of each option is known to the agents.
Let i ∗ denote the optimal option, i.e., the option with highest expected mean µ i ∗ = max { µ 1 , . . . , µ N } .
Let ∆ i = µ i ∗ − µ i be the expected reward gap between option i ∗ and option i. Let ϕ kt denote the option chosen by agent k at time t .
Deﬁne I { ϕ kt = i } to be an indicator random variable that takes value 1 if agent k chooses option i at time t and 0 otherwise.
Assumption 1 All the agents know the variance proxy σ 2 i of the rewards associated with each option.
Remark 1 Assumption 1 can be relaxed by allowing that the agents only know an upper bound for the variance proxy of all the options.
All results presented in this paper can be generalized to this case by replacing σ i with the upper bound.
Let G ( V , E ) be an undirected graph that encodes the ﬁxed communication network graph among agents.
V is a set of K vertices and E is the set of edges between pairs of vertices.
Without loss of generality, let vertex k ∈ V correspond to agent k .
If there is a communication link e ( k, j ) ∈ E between agents k and j , then agents k and j are neighbors.
Deﬁne d k as the degree of vertex k. This is same as the number of neighbors of agent k .
Deﬁne d avg as the average degree of the graph.
Let I { k,j } be an indicator variable that takes value 1 if agent j is a neighbor of agent k and 0 otherwise.
Let I t { k,j } be an indicator random variable that takes value 1 if agent k receives information, i.e. chosen option and the value of the obtained reward, from agent j at time t and 0 otherwise.
And let I t { k,k = 1 , for all
t , k . Assumption 2 When more than one agent chooses the same option at the same time, the agents receive rewards independently and identically drawn from the probability distribution associated with the chosen option.
Let n ki ( t ) and N ki ( t ) denote the number of times agent k sampled option i and observed rewards from option i , respectively until time t :
[Equation]
Note that the number of observations N ki ( t ) is the sum of the number of samples taken by agent k of option i and the number of times agent k received reward values for option i from its neighbors.
Each agent reduces the uncertainty associated with estimates of expected reward of options by leveraging the information received from neighbors.
Let S ki ( t ) be the sum of reward values until time t received by agent k from sampling option i and receiving reward values from neighbors obtained from option i .
Each agent estimates the expected reward of option i by averaging.
Let (cid:98) µ ki ( t ) be the estimated expected reward of option
i for agent k.
Then we have
[Equation]
The natural extension of UCB sampling rule to multi-agent bandit problem can be stated as follows.
Deﬁnition 1 ( Natural Extension of UCB Sampling Rule)
The sampling rule { ϕ kt } T 1 for agent k at time step t ∈ { 1 , . . .
, T } is deﬁned as
[Equation]
with
[Equation]
where ξ > 1 .
We proceed to show that sharing information about the optimal option degrades performance by considering two cases as follows.
In the ﬁrst case agents do not communicate any information.
This is equivalent to K agents using the single-agent UCB algorithm individually.
In the second case we consider an oracle communication rule where agents broadcast reward values obtained by sampling the optimal option to their neighbors.
We state the oracle communication protocol in Deﬁnition 2.
Deﬁnition 2 (Oracle Communication Rule)
The communication rule for agents k, j ∈ { 1 , . . .
, K }
such that e ( k, j ) ∈ E , at time step t ∈ { 1 , . . .
, T } is deﬁned as
[Equation]
Theoretical Results
In this section we analyze the performance of natural extension of UCB sampling rule (Deﬁnition 1), with the no communication protocol and with the oracle communication protocol deﬁned in
Deﬁnition 2.
Following convention, as a performance measure, we use the expected regret agents incur by sampling suboptimal options.
Let R ( t ) be the cumulative regret of the group at time t. Then the expected group cumulative regret at time t can be given as
[Equation]
We provide upper bounds for the expected cumulative group regret under the two communication protocols.
Theorem 1 (Upper Bound on Group Regret) Consider a multi-agent stochastic bandit problem with T time steps and K agents that communicate over network graph G .
All agents sample according to the rule deﬁned in Deﬁnition 1.
Then the expected group cumulative regret satisﬁes:
With no communication
[Equation]
With oracle communication given in Deﬁnition
[Equation]
The proof of Theorem 1 is given in Appendix A.
In the following corollary, we specialize this result to the case where agents broadcast reward values received from the optimal option to the group.
This corresponds to the case where G is a complete graph.
Corollary 1 (Specialization to complete graph) Let the communication graph G be complete.
With no communication agents suffer a group regret of E ( R ( T ))
=
O ( K log T ) and with oracle communication given in Deﬁnition 2 they suffer a group regret of E ( R ( T ))
=
O ( K log T + + K log K ) .
Proof of Corollary 1 When agents do not communicate the group is equivalent to K individual agents facing the same bandit problem independently.
Thus expected cumulative group regret is the same irrespective of the graph structure.
When the communication graph is complete each agent has
K − 1 number if neighbors.
Thus d avg = K − 1 .
The result follows from Theorem 1.
Remark 2 Additional group regret incurred by sharing information about the optimal option is independent of the time horizon of the decision making process.
The additional regret increases with the average degree of the communication graph.
From Theorem 1 we see that performance of agents degrade when they broadcast reward values received from the optimal option.
Simulation Results
In this section we provide numerical simulations illustrating the results and validating our theoretical claim.
For all the simulations presented in this section, we consider a group of 50 agents ( K = 50 ) and 10 options ( N = 10 ) with Gaussian reward distributions.
We consider a challenging problem in which we let the expected reward value of the optimal option be 11, the expected reward of all other options be 10, and the variance of all options be 1.
We let the communication network graph G be complete.
We provide results with 500 time steps ( T = 500 ) using 1000 Monte Carlo simulations.
[Figure]
The orange solid line in Figure 1(a) shows expected cumulative group regret when agents do not share any information with their neighbors.
This corresponds to the case where K agents are independently faced with the same bandit problem.
The blue dashed line corresponds to the case where agents share rewards values received from the optimal option only.
The ﬁgure illustrates that agents perform better when they do not communicate as compared to the case where they share information about only the optimal option.
Figure 1(b) shows group regret for the cases where agents share information about reward values received from all options (green dotted line) and agents share information about reward values received only from suboptimal options (red dash-dot line).
The ﬁgure illustrates that agents perform better when they only share information about suboptimal options as compared to the case where they share information about all options.
Comparing Figures 1(a) and 1(b) note also that agents perform much better with sharing than without sharing as long they share information on suboptimal options, whether or not they also share information on optimal options.
Discussion and Conclusion
The main result presented in this paper proves that when sampling options according to a natural extension of UCB algorithm, sharing information about the optimal option degrades performance of the agents.
Our work has two major implications for distributed sequential learning algorithms that use natural extensions of UCB.
The ﬁrst implication is the importance of developing communication protocols that minimize the amount of information shared about the optimal option.
Recall that under any efﬁcient sampling rule agents sample suboptimal options only logarithmically in time.
Thus, in cases where communication is costly such protocols have the added advantage of incurring a signiﬁcantly smaller communication cost.
The second implication is that adversaries can decrease the performance of agents by intentionally only sharing information about the optimal option.
In this work we shed light on an inherent drawback of natural extensions of UCB algorithms to multi-agents setting.
We proved that when agent sample options according to a natural extension of UCB, group performance degrades when they share information about the optimal option.
We provided numerical results illustrating that agents perform better when they do not communicate as compared to the case where they share information only about the optimal option.
Similarly, agents perform better when they share information only about suboptimal options as compared to the case where they share information about all options. Acknowledgment
This research has been supported in part by ONR grants N00014-18-1-2873 and N00014-19-1and ARO grant W911NF-18-1-0325.
A Proof of Theorem
We begin by proving a couple of lemmas that are useful in proving Theorem 1.
Lemma 1 (Concentration Bound for Estimate) Let d k be the degree of agent k. For any option i for any ϑ > 0 and some ζ > 1 we have [Equation] where κ =
[Equation]
Proof of Lemma 1 Since X i is a sub-Gaussian random variable with variance proxy σ 2 i , we have
[Equation]
Deﬁne a new random variable such that ∀ τ > 0 .
Then,
[Equation]
Note that I { ϕ jτ = i } , I τ { k,j } are pre-visible variables and hence E ( Y ki ( τ ))
=
E ( Y ki ( τ ) |F τ − 1 ) = 0 .
Let
Z ki ( t ) = (cid:80) tτ
=
1 Y ki ( τ ) .
For any λ > 0 from (1) and (2)
[Equation]
Since I τ { k,j } , I { ϕ jτ = i } are F τ − 1 measurable random variables,
[Equation]
Using the properties of conditional expectations we obtain
E
[Equation]
Thus, we get
E (cid:18) exp
[Equation]
≤ 1 .
Note that we have
[Equation]
Let ζ > 1 .
Then 1 ≤ N ki ( t ) ≤ ζ D t where D t = log(( d k +1) t ) log ζ .
For λ j = 2 σ i (cid:113) κϑ ζ j − 1 / 2 and ζ j − 1 ≤
N ki ( t ) ≤ ζ j we have 2 κϑ
[Equation]
where κ =
[Equation]
Recall from the Markov inequality that P ( Y ≥ a ) ≤ E ( Y ) a for any positive random variable Y .
Thus, from (3) and (4) we have
[Equation]
Then, we get
[Equation]
Substituting Z ki ( t ) = (cid:80) tτ =1 (cid:80) Kj
{+=
+}
( X i − µ i )
I { ϕ jτ = i } I τ { k,j } and from symmetry of the reward distribution we get [Equation] This concludes the proof of Lemma 1. Lemma 2 Let d k be the degree of agent k. For any option i for some ζ > 1 we have
[Equation]
Proof of Lemma 2 Substituting ϑ = 2 σ 2 i ( ξ + 1) log t to Lemma 1 we get
[Equation]
Since C ki ( t ) = (cid:113) 2( ξ +1)log t N ki
( t ) this concludes the proof of Lemma 2.
Lemma 3 Let α, β > 1 and t ∈ { 1 , · · · , T } .
Then,
T (cid:88)
[Equation]
Proof of Lemma 3 Note that
[Equation]
Let z =
αt.
Then we have [Equation]
[-T log αt t β +
-]
β
[Equation]
β 2 z β
[Equation]
α .
Thus,
T (cid:88) t
= log αt t
[Equation]
+
[Equation]
log
[Equation]
log
[Equation]
β
[Equation]
( αT )
[Equation]
−
[Equation]
(cid:21) .
The proof of Lemma 3 follows from (5).
Proof of Theorem 1 Recall that n ki ( t ) and N ki ( t ) denote the number of samples and number of observations taken from option i by agent k until time t. Further, under both no communication and oracle communication protocols, agents do not share information about suboptimal options.
Thus for each suboptimal option i we have n ki ( t ) =
N ki ( t ) , for all k, t , and
K
[Equation]
Here η i ( t ) is a non negative non decreasing function.
Note that we have
[Equation]
Q
[Equation]
When η i ( t ) = (cid:16)
[Equation]
[Equation]
+1)
[Equation]
i ∆ 2 i (cid:17) log t we see that
P [Equation]
, n [Equation]
= 0 .
Thus,
[Equation]
Now we proceed to upper bound the last two summation terms on the right hand side of (6).
First we provide bounds for the case where agents do not communicate.
No communication is equivalent to having no neighbors.
Thus we have d k = 0 , for all k .
Then from Lemma 2 for all options i we have
[Equation]
Thus when agents do not communicate we have
K (cid:88)
k
[Equation]
T
[Equation]
Using the results from Lemma 3 we get
K (cid:88) k
=
[Equation]
Now we proceed to upper bound the last two summations of (6) when agents communicate according to the oracle communication protocol.
Since agents do not share information about suboptimal options for any suboptimal option i we have
[Equation]
Recall that d k is the degree of agent k .
Then for the optimal option we have
[Equation]
Thus when agents only share information about the optimal option we have
[Equation]
Since log is a concave function,
[Equation]
Using the results from (8), (9) and Lemma 3 we get
[Equation]
Recall that E ( R ( T )) =
(cid:80) Ni =1 (cid:80)
Kk =1 ∆ i E [Equation] n ki ( T ) (cid:1) .
Using this result from (7) and substituting for η i ( T ) when agents do not communicate we get
[Equation]
From (10) and substituting for η i ( T ) when agents only share information about the optimal option,
[Equation]
[Equation]
Thus, from (11) and (12) we have, with no communication E ( R ( T ))
=
O ( K log T ) and with oracle communication deﬁned in Deﬁnition 2, E
( R ( T ))
=
O ( K log T + K log( d avg + 1)) .
This concludes the proof of Theorem 1.