A CHIEVING N EAR -O PTIMAL I NDIVIDUAL R EGRET & L OW C OMMUNICATIONS IN M ULTI -A GENT B ANDITS
Anonymous authors
Paper under double-blind review
A BSTRACT
Cooperative multi-agent multi-armed bandits ( CMA2B ) study how distributedagents cooperatively play the same multi-armed bandit game.
Most existing CMA2Bworks focused on maximizing the group performance of all agents—the accumulation of all agents’ individual performance (i.e., individual reward).
However, inmany applications, the performance of the system is more sensitive to the “bad”agent—the agent with the worst individual performance.
For example, in a droneswarm, a “bad” agent may crash into other drones and severely degrade the systemperformance.
In that case, the key of the learning algorithm design is to coordinatecomputational and communicational resources among agents so to optimize theindividual learning performance of the “bad” agent.
In CMA2B , maximizing thegroup performance is equivalent to minimizing the group regret of all agents, andmaximizing the individual performance can be measured by minimizing the maximum (worst) individual regret among agents.
Minimizing the maximum individualregret was largely ignored in prior literature, and currently, there is little workon how to minimize this objective with a low communication overhead.
In thispaper, we propose a near-optimal algorithm on both individual and group regrets, inaddition, we also propose a novel communication module in the algorithm, whichonly needs O (log(log T ))
communication times where T is the number of decisionrounds.
We also conduct simulations to illustrate the advantage of our algorithmby comparing it to other known baselines.
I NTRODUCTION
The stochastic multi-armed bandit problem is a classic sequential decision making problem.
Given
K arms, there is one agent who repeatedly chooses one arm to pull and observes a stochastic rewardfrom the pulled arm in each time slot.
To maximize cumulative reward (or minimize regret which isthe cumulative reward difference between the optimal decision and agent’s choices), the agent needsto pull an arm either with a large empirical mean reward to greedily maximize reward (exploitation),or whose reward estimate is highly uncertain so as to reduce that uncertainty to discover good arms(exploration).
To model many real life applications, e.g., cognitive radio with multiple users (Liu& Zhao, 2010; Jouini et al., 2010; Boursier & Perchet, 2019), clinical trials in multiple labs (Wang,1991), recommendation systems with multiple servers (Agarwal et al., 2008; Li et al., 2010; Landgrenet al., 2016), cooperative source search by multiple robots (Li et al., 2014; Jin et al., 2017), etc., oneneeds to extend the model to allow for more than one agent ( M > 1 ) playing the same multi-armedbandit game.
These agents cooperate with each other to minimize their regrets.
We call this problemthe cooperative multi-agent multi-armed bandits ( CMA2B ) problem and present it formally in §2.
The most common objective of CMA2B is to minimize the aggregate regret among all M agents,dubbed as group regret in this paper.
This objective has been studied in the majority of priorwork (Boursier & Perchet, 2019; Chawla et al., 2020;
Huang et al., 2021; Shi et al., 2021b; Wanget al., 2020a;b).
In addition to group regret, individual performance among agents is another importantmetric that is less studied in prior work on CMA2B .
The performance of each individual agent is criticalin many applications in distributed systems.
For example, in many distributed resource allocationscenarios with different agents in charge of the allocation, overall performance depends on theperformance of the bottleneck agent instead of the aggregate performance of all agents.
This can alsobe seen in a computer network scenario, in which an ISP may apply learning-based algorithms (Maet al., 2010; Jiang et al., 2018) for some networking problems such as shortest path routing, channel
[Table]
selection, etc.
To ensure that the users are served fairly, the underlying algorithms should fairlyprovide approximately equivalent individual performance for each learning agent.
This is equivalentto minimizing the bottleneck agent’s individual regret.
For another thing, in network optimizationliterature, the max-min fairness metric—maximize the minimal individual reward—is widely usedto measure a system’s fairness (Srikant & Ying, 2013, §2.21), such as fair queuing (Demers et al.,1989).
Since the regret is the opposite of reward, optimizing the max-min fairness is also equivalent tominimizing the bottleneck agent’s regret. Other fairness motivation examples can be found in politicalphilosophy (Rawls, 2004).
In this paper, we explicitly take into account the notion of minimizing themaximum individual regret and, for brevity, hereinafter, refer to it as the individual regret .
Another important metric in CMA2B is the communication time of all agents.
For some distributedsystems, e.g., agents are geographically located, communications among agents can be expensive.
Thus it is important to design a cooperative learning algorithm that provides minimal group andindividual regrets, while at the same time, incurs a small communication cost.
In addition, it willbe desirable to have a learning algorithm in which one can tune parameters so as to trade offcommunication times with regret as needed by different applications.
Contributions.
In §3, we present the UCB - TCOM algorithm that achieves not only a near-optimalgroup regret of O (( K/ ∆ 2 ) log T ) but also a near-optimal problem-dependent individual regret of
O (( K/M ∆ 2 ) log T ) with only O (log(log T ))
communication times, where ∆ 2 is the smallest rewardmean gap between arms and T is the number of rounds.
This is the ﬁrst near-optimal algorithmon individual regret with efﬁcient communications: Previous low communication algorithms, e.g.,the leader-follower algorithm
(Wang et al., 2020b), cannot achieve the near-optimal individualregret; and previous near-optimal algorithms on individual regret, e.g., GosInE (Chawla et al.
,2020), required high communication times (see related works below).
UCB - TCOM achieves thenear-optimal individual regret performance by evenly dividing the group regret to all agents.
Toequalize the regrets of all agents, UCB - TCOM directs agents to synchronously pull arms: agentsonly utilize the common reward observations, i.e., those having been broadcast over all agents, tomake decisions.
The communication policy TCOM (Tunable COMmunication) of UCB - TCOM is aparametric meta-algorithm that governs the communication of agents and can be executed on top ofany underlying bandit learning algorithm.
A salient feature of TCOM is that it can be tuned to balanceregret and communication times.
In particular, two parameters in TCOM can be tuned to determine theaggressiveness and frequency of communications among agents.
Our analysis explicitly demonstrateshow communication times can be tuned from 0 to O ( T ) .
Finally, we report numerical results in §5.
Related works.
The most relevant prior literature on CMA2B is summarized in Table 1.
Firstof all, notice all algorithms in Table 1 attain a similar O ( K log T ) regret upper bound.
So, wecompare them from the perspective of individual regret and communication cost.
Previous algorithmsexhibit good individual regret bounds, i.e., Dec_UCB (Zhu et al., 2021a) and GosInE (Chawla et al.
However, the communication times of Dec_UCB are O ( T ) and that of GosInE are at leastΩ(log T ) .
It is worth noting that the leader-follower algorithm ( DPE2 ) (Wang et al., 2020a), wherean agent, i.e., the leader, takes responsibility for exploration, and the other agents, i.e., followers, onlyexploit the arms recommended by the leader incurs constant O (1) communication cost.
However,the leader-follower approach inherently fails to achieve good individual regret—the leader agentincurs almost all of the regret
O
( K log T ) while followers only incur constant regrets.
Lastly,we mention the communication policy ComEx (Madhushani & Leonard, 2021) designed for fullydistributed algorithms achieves a near-optimal group regret of O
( K log T )
with a communicationcost of O (log T ) .
However, ComEx does not guarantee optimal individual regret.
We provide acomprehensive discussion of related work in Appendix A.
M ODELSystem Model.
Consider a multi-armed bandit (MAB) game with K ∈ N + arms.
Each armk ∈ [ K ] 1 is associated with a (1 / 2) -sub-gaussian reward distribution with unknown mean µ ( k ) , e.g.,it can be any distribution whose support is [0 , 1] .
Assume there are M ∈ N + distributed agentsplaying this bandit game in T ∈ N + rounds.
In time slot t ∈ [ T ] , each agent i ∈ [ M ] pulls an arm
A ( i ) t ∈
[ K ] and receives a reward X ( i ) t
( A ( i ) t ) drawn from the reward distribution of arm A ( i ) t .
Whenmore than one agent pulls the same arm, each of them gets an independent reward drawn from thearm’s distribution.
Without loss of generality, assume the K arms are ordered in descending of theirmean rewards, i.e., µ (1) > µ (2) (cid:62) . . .
(cid:62) µ ( K ) .
Denote the reward mean gap as ∆ k := µ (1) − µ ( k )and assume ∆ 2 > 0 , i.e., arm 1 is the unique optimal arm.
Group regret.
This paper uses group regret—the cumulative difference between the optimal policyrewards and an algorithm’s rewards—as the algorithm’s performance metric.
Each agent’s optimalpolicy is to pull arm 1 in all T rounds.
So, under any cooperative algorithm A , the expected groupregret of all M agents is deﬁned as follows,
[Equation]
where the expectation is taken over the randomness of stochastic rewards and algorithm’s (agents’)decisions.
Maximum individual regret
While the group regret in (1) characterizes overall system performance, individual performance of each agent is important as well, and, among all individualregrets of agents, the maximum one is usually more important.
For example, in a drone swarm, thefailure/misbehavior of a single drone, e.g., it crashes into other drones, can dramatically degradethe whole system’s overall performance; in network measurement, the slowest inference enginedetermines how fast the network parameters, e.g., trafﬁc ﬂows and channel bandwidths, are learned.
In the above systems, one can deﬁne the individual regret objective as follows,
[Equation]
Communication times.
Agents in the system cooperate with each other by communicating theirreward observations or reward averages.
Any message (if communicated) is broadcast to all agentsand received within a single time slot.
A broadcast message includes an arm index, the average armreward for all observations since the previous broadcast, and the number of observations.
The totalnumber of messages communicated among these M agents quantiﬁes the communication times ofan algorithm.
We count each broadcast as one message 2 .
Hence, the communication times underalgorithm
A are deﬁned as follows,
[Equation]
Model extensions.
For ease of presentation of the core ideas, we focus on a simple model formulationwhere agents reside on a complete graph (i.e., one clique) and incur no communication delays.
In
Appendix F, we extend the basic model and communication policy TCOM to account for arbitrarycommunication topology and deterministic communication delays.
These demonstrate the generalityof our core algorithmic ideas.
A LGORITHM D ESIGN
In this section, we devise a cooperative learning algorithm, UCB - TCOM , which attains the near-optimalresults of not only group regret but maximum individual regret as well with only O (log(log T )) communication times.
The core of UCB - TCOM is a communication strategy TCOM that can run ontop of diverse bandit learning algorithms in multi-agent systems.
Speciﬁcally, TCOM determines howan agent broadcasts its reward observations to other agents, while the underlying bandit learningalgorithm, e.g., UCB, Thompson sampling, elimination, determines how each agent pulls arms.
3 In§3.1, we present TCOM as the communication module of our algorithm design.
Then, in §3.2, weshow how to integrate TCOM with UCB as the underlying bandit algorithm to obtain UCB - TCOM .
Recall that in CMA2B , the near-optimal group regret Θ(( K/ ∆ 2 ) log T ) (Wang et al., 2020a, §1.2) isthe summation of individual regrets of all agents.
Therefore, minimizing the maximum individualregret is equivalent to equalizing individual regrets of all agents.
Uniformly dividing the groupregret to each individual agent can be achieved by a symmetric learning structure where agentspull same arms in each time slot.
However, in current state-of-the-art algorithms, e.g., the leaderfollower algorithm DPE2 (Wang et al., 2020a), agents usually take different roles—some exploreand other exploit—and thus those algorithms are suboptimal on the individual regret objective.
Foranother thing, some straightforward symmetric learning algorithms, e.g., cooperative UCB (Yanget al., 2022, §III.B) where agents communicate and take the same action in every time slots, requireshigh communication times; most are O ( T ) .
Instead UCB - TCOM proposes a novel communicationpolicy TCOM to maintain a symmetric learning structure with low communication times.
Especially,to maintain the symmetric learning structure, agents only utilize global reward observations.
Here“global” observations refer to those observations that have already been broadcast to all agents,excluding those recent reward observations that have not been shared (i.e., only local to one agent).
3. TCOM : A T UNABLE C OMMUNICATION P OLICY
A key idea of TCOM is based on the observation that the beneﬁt of cooperation in a multi-agent settingcomes primarily from sharing information regarding suboptimal arms instead of the optimal arm.
Speciﬁcally, an agent can avoid pulling a suboptimal arm when she receives external observationsfrom others, while sharing observations from the optimal arm may not necessarily reduce individualagent regret, e.g., sharing information about an optimal arm while other agents still need to exploitthe optimal arm will not decrease regret but merely increase communication times.
So, insteadof paying the cost for communicating the optimal arm observations, an agent can let other agentsexplore the optimal arm by themselves.
Hence, agents can reduce overall communication overheadby refraining from communicating the optimal arm’s rewards while still guaranteeing near-optimalregret.
To implement the above idea, each agent constructs a communication arm set from which
TCOM dynamically excludes the arms that are likely to be optimal.
Then, each agent only sharesobservations about potentially suboptimal arms in the communication arm set.
We explain how toconstruct the communication arm set in §3.1.1.
The communication arm set technique in the previous paragraph reduces communication times to
O (log T ) .
To further reduce communication times, another key idea of TCOM is to aggregate anarm’s multiple observations and send their average at one time.
Naïvely applying a phase-basedcommunication protocol, e.g., the one in (Wang et al., 2020b), to aggregate arm observations doesnot automatically further decrease communication times to O (log(log T )) .
Because the phase-basedprotocol was used to achieve O (log T ) communication times which is already reached via thecommunication arm set technique.
Instead, one needs to break the pre-assigned rigid phased-basedcommunications, i.e., observations of all arms are communicated together at the end of each phase,and allow each arm—depending on how many times it lies inside the communication arm set—to haveits own speciﬁc and dynamic phases for communications.
This idea is implemented by a carefullydesigned observation-buffering technique.
The novel technique can boost the above communicationarm set technique and further reduce communication times to O (log(log T )) .
We formalize this idea
3.1. C ONSTRUCTION OF THE COMMUNICATION ARM SET
In each time slot, each agent determines its communication arm set by comparing its estimatedconﬁdence intervals of the reward means.
Let ˆ n t ( k ) denote the number of times of the global rewardobservations of arm k up to time slot t .
That is, the number of observations that has been broadcast across the whole CMA2B .
Let ˆ µ t ( k ) denote the mean reward estimate of these ˆ n t ( k ) observations.
Then, given the empirical mean ˆ µ t ( k ) and the number of observations ˆ n t ( k ) , we construct a tunableconﬁdence interval for the true reward mean µ ( k ) by Hoeffding’s inequality (Hoeffding, 1994).
Theconﬁdence interval centered at ˆ µ t ( k ) has a width CI t ( k, α ) expressed as
[Equation]
where α ∈ ( −∞ , ∞ ) is a tunable parameter.
The reward mean µ ( k ) lies inside the conﬁdenceinterval (ˆ
µ t ( k ) − CI t ( k, α ) , ˆ µ t ( k )+ CI t ( k, α )) with probability of at least 1 − 2 t − 2 α 2 (see Lemma 1in Appendix B.2).
We deﬁne the tunable upper and lower conﬁdence bounds of ˆ µ t ( k ) as
[Equation]
For any arm k , if there exists another arm k (cid:48) whose upper conﬁdence bound tUCB
t
[Equation] k [Equation] , α ) is greaterthan arm k ’s lower conﬁdence bound tLCB t ( k, α ) , then arm k ’s mean reward estimate is notsigniﬁcantly greater than those of others.
In this case, arm k is potentially identiﬁed as suboptimal,and its new observation (if any) should be broadcast to others.
Formally, given tuning parameter α ,communication arm set C t ( α ) of agent i at time t contains all arms identiﬁed as suboptimal, i.e.,
C [Equation]
[Equation] ∈ [ K ] \ { k } such that tUCB t ( k (cid:48) , α ) >
tLCB t ( k, α ) } .
Remark 1 (Comparison to the candidate arm set of elimination algorithms) .
At the ﬁrst glance,the communication arm set in (3) is similar to the active arm elimination (AAE) policy’s candidatearm set (Auer et al., 2002; Even-Dar et al., 2006; Yang et al., 2022).
However, the two sets haveintrinsically different usages: TCOM constructs the communication arm set to determine how tobroadcast observations, while AAE uses its candidate set to determine how to pull arms.
Remark 2 (Comparsion to ComEx (Madhushani & Leonard, 2020; 2021)) .
The intuition we utilize inthis subsection—communicating the observations of suboptimal arms is more useful than that of theoptimal arm—has also been observed in Madhushani & Leonard (2020; 2021).
Compared to ComEx ,the algorithmic novelties of TCOM are (1) applying conﬁdence intervals to more accurately identifythe suboptimal arms; (2) combining the observation-buffering broadcast with the communicationarm set technique; (3) including tunable parameters with more ﬂexibility.
The detailed algorithmdesign comparisons are deferred to Appendix A.1.
Furthermore, as we mentioned in Table 1, ComExfails to achieve the near-optimal individual regret and its communication times O (log T ) are not asgood as our O (log(log T )) result.
3.1. O BSERVATION -B UFFERING B ROADCAST
While the straightforward immediate broadcast observation sharing method can produce a nearoptimal group regret, its O (log T ) communication times are not efﬁcient.
Prior literature (Desautelset al., 2014; Gao et al., 2019) has shown that near-optimal regret in stochastic bandit games ispreserved in the presence of a constant delay before receiving reward observations (Desautels et al.,2014), and the regret upper bound does not deteriorate too much when delays increase according to ageometric sequence (Gao et al., 2019).
These results suggest that it is not necessary to immediatelycommunicate the latest observations.
Instead, to reduce communication times, the policy canaggregate multiple reward observations from an arm and communicate their sample mean in onemessage.
Combining this mechanism with the designed communication arm set in (3) leads to thecommunication policy TCOM .
Next, we present how the buffering mechanism controls observation aggregation.
Denote by N t ( k )the number of times that an agent pulls arm k when the arm lies inside the communication arm set, i.e.,k ∈ C t ( α ) , up to time t .
Therefore, N t ( k ) is the number of messages agent i would broadcast, eachcontaining a single arm k observation, if one employs immediate broadcast.
Instead of immediatebroadcast, we buffer observations and then broadcast their average when the number of observations(since previous broadcast) increases by a ratio.
For example, if the ratio is 2 , the agent sends armk ’s reward average of the arm’s new observations since last broadcast whenever the counter N t ( k )equals 2 , 4 , 8 , 16 , . .
. , etc, or power of 2 .
More generally, let β ( > 1) denote a ratio parameter thatcontrols how the buffering size of reward observations is increased.
When counter N t ( k ) increasesto (cid:100) βN
τ ( k ) (cid:101) since the last time agent i broadcast arm k ’s average reward, i.e., N t ( k ) (cid:62) (cid:100) βN
τ ( k ) (cid:101) ,then agent i broadcasts a message including the arm index k , the counter N t ( k ) , and the sample mean˜ µ t ( k ) , of this arm’s latest (cid:98) N t ( k )(1 − 1 /β )
(cid:99) observations to other agents.
Due to the space limit, thedetailed discussion of TCOM ’s tunability on parameters α and β is deferred to Appendix D.1.
Update the ˆ µ
ˆ n t ( k
+˜ µ t (
(cid:98) N t
− β ) (cid:99)
ˆ n t ( k )+ (cid:98) N t ( k )(1 − β (cid:99) 17:
Increase n
( k
Update the communication arm set C t ( α ) via (3) on conﬁdence bounds 19: end for
3. UCB - TCOM : A PPLICATION OF TCOM ON THE UCB A LGORITHM
In this section, we demonstrate how to leverage the communication policy TCOM to develop a communication-efﬁcient extension of cooperative UCB .
We call the proposed algorithm UCB - TCOM .
In slot t , agent i chooses the arm with the largest global UCB index, i.e., A t = arg max k UCB t ( k ) ,where UCB t ( k ) :
= ˆ µ t ( k ) + (cid:112) 2 log t/
ˆ n t ( k ) , and observes the arm
’s reward X ( i ) t
( A t ) .
Note the UCBindexes and action A t are the same for all agents; thus we omit their superscript ( i ) .
After pullingarm A t , the agent checks whether the arm is in its communication arm set C t (Line 6); if yes, counter
N t ( A t ) is increased by 1 , and the buffered reward sample mean is updated (Lines 7-8).
When thecounter N t ( A t ) is increased by a factor β since the last time slot that agent
i communicated arm
A t ’s sample mean (Line 10), the agent sends out arm A t ’s sample mean to other agents (Line 11).
Meanwhile, if the agent receives messages from other agents, it updates its estimates accordingly(Lines
The details are presented in Algorithm 1.
T HEORETICAL R ESULTS
In this section, we analyze the regret and communication times of UCB - TCOM (Algorithm 1).
4. R ESULT O VERVIEW
We present near-optimal group and individual regret upper bounds of UCB - TCOM in Theorem 1.
Theorem 1 (Regret upper bounds of UCB - TCOM for α > 1 ) .
When the communication arm setparameter α > 1 4 and buffering-ratio β > 1 , UCB - TCOM (Algorithm 1) attains a near-optimal groupregret upper bound in terms of number of decision rounds T , arms K , and agents M , or formally,
[Equation]
a remnant from bounding a small probability event (see Lemma 2 in
Appendix B.2).
This condition can be relaxed to α > 1 √ 2 via the peeling technique (see Appendix C) where (4)
’s second term will also change accordingly.
This relaxation is also applicable to Theorem 2(iii).
and UCB - TCOM (Algorithm 1) also attains a near-optimal individual regret upper bound, or formally,
[Equation]
Theorem 1’s group regret proof is based on the observation that most suboptimal arm observationsare broadcast when α > 1 (see Theorem 2(iii)), and the fact that under the observation bufferingmechanism the number of observations of suboptimal arms increases by at most a ratio of β betweenbroadcasts.
Theorem 1’s individual regret proof is based on the fact that all M agents are symmetric—they always pull the same arms.
Proofs of Theorem 1 is given in Appendix B.3.
In Theorem 2, we present the communication times of UCB - TCOM .
It shows how parameter αinﬂuences the communication of optimal and suboptimal arms observations.
Theorem 2.
The communication times of UCB - TCOM has the following properties:
(i) When α (cid:54) −√ 2 , no communication occurs among agents.
(ii) When −√ 2 < α < √ 2 and β > 1 , the number of broadcasts of observations of the optimalarm by one agent is O (log(log T )) .
More rigorously, it is less than
[Equation]
(iii) When α > 1 , almost all observations of suboptimal arms—except for a ﬁnite numberindependent of T —are broadcast.
(iv) When α (cid:62) 2 √ 2 µ (1) ∆, almost all observations of the optimal arm—except for a ﬁnite numberthat is independent of T —are broadcast.
The proof of Theorem 2(i) follows from the fact that when α < −√ 2 , the pulled arm’s tLCBindex is greater than those tUCB indexes of all other arms, which violates the communication armset’s condition in (3).
Hence, the pulled arm never lies inside the communication arm set, and thusno communication occurs.
The proof of Theorem 2(ii) is based on the geometric growth of thetime interval between broadcasts and a key observation: the number of broadcasts of the optimalarm’s observations is upper bounded by some suboptimal arm k ’s total number of pulls, that is,ˆ n ( i ) t (1) <
(( √ 2 + α ) / ( √ 2 − α )) 2 ˆ n ( i ) t ( k ) .
Theorems 2(iii) and 2(iv) are derived by excluding somewell-designed small probability events and from the fact that the arm mean rewards lie inside theirtunable conﬁdence intervals with high probability.
The proof of Theorem 2 is given in Appendix B.4.
Remark 3.
Theorem 2’s proof is not a simple extension of the analysis of previous algorithms.
First,
TCOM relies on a group of tunable conﬁdence intervals to identify suboptimal arms and communicatenew observations while ComEx are only based on arm empirical reward mean; a special case oftunable conﬁdence intervals.
Second, the observation-buffering broadcast incurs additional delaysof the global observations sharing which is a unique challenge in TCOM .
We defer the detailedexplanation of both challenges to Appendix B.1.
4. D ISCUSSIONS
(a) Regret optimality of UCB - TCOM .
The following asymptotic lower bound has been establishedin the classic bandits literature (e.g., Lai et al.
[Equation]
where ν k is the reward distribution of arm k and KL is the Kullback-Leibler divergence betweentwo probability distributions.
When the reward distributions are Bernoulli or Gaussian, the regretlower bound can be written as Ω( (cid:80) k> 1 (log T/ ∆ k ))
(Lattimore & Szepesvári, 2020, §16).
Sincethe CMA2B model has the same objective as the centralized (single agent) MAB model, it inheritsthis group regret lower bound (Wang et al., 2020a, §1.2) under any possible communication policies.
Given this lower bound, Theorem 1’s group regret upper bound in (7) is near-optimal, i.e., it is tightup to a constant factor.
Recall that the individual regret in (2) corresponds to the maximal individual regret among M agents.
Since the maximal individual regret is minimized when all agents uniformly pay the regret time, fromthe above group regret lower bound, one can obtain the following individual regret lower bound,
[Equation]
Compared to this lower bound, Theorem 1’s individual regret upper bound in (5) is also near-optimal.(b) Impact of cooperation and number of agents M on group regret.
If there is no cooperationamong the M agents—each agent individually plays the game, then the group regret is lower boundedas follows:
[Equation]
which is M times worse than the group regret achieved by UCB - TCOM in Theorem 1.
This highlightsthe beneﬁts of cooperation.
Furthermore, when the number of rounds T is large, the second termcontaining an M factor in the group regret (4) is negligible.
That is, the group regret of UCB - TCOMdoes not increase rapidly when the number of agents M increases, while it would without cooperation.(c)
Individual regret comparison to the leader-follower algorithm.
Equation (5) shows that theindividual regret of UCB - TCOM deceases as the number of agents M increases.
This is in contrast to
DPE2 (Wang et al., 2020a) based on the leader-follower paradigm, which attributes all explorationcosts to a single leader, and thus DPE2 ’s individual regret is M times worse than that of UCB - TCOM .
In many large-scale distributed systems, e.g., edge computing, database, etc., computational resourcesare often distributed geographically.
The largest individual computational time that these distributednodes (agents) need to pay can be mapped to the individual regret.
From this perspective, UCB - TCOMwith the near-optimal individual regret is preferable to DPE2 especially when M is large.(d)
Communication Times of UCB - TCOM .
Theorem 2(ii) shows that when −√ 2 < α < √ 2 ,the number of broadcasts of the optimal arm’s observations is O (log(log T )) .
Additionally, theobservation buffering mechanism also aggregates the suboptimal arms O (log T ) observations into
O (log(log T )) broadcasts.
So, in this case, UCB - TCOM only incurs O (log(log T )) communicationtime.
Combined the above with Theorem 1 shows that when 1 < α < √ 2 and β > 1 , UCBTCOM enjoys the near-optimal group and individual regret upper bounds with only O (log(log T ))communication times.
We also discuss how parameters α and β of TCOM inﬂuence cooperation among the agents and howthey impact communication times and regret in Appendix D.
N UMERICAL S IMULATIONS
Experiment setup.
Unless otherwise stated, the experiments consist of M = 25 agents and K = 20arms, communication set parameter α = 1 . 2 , buffering ratio β = 2 , and T = 30 , 000 .
Each arm isassociated with a Bernoulli reward random variable whose mean is uniformly randomly taken from
Ad-Clicks (Avito, 2015).
All results are averaged over 50 trials and their standard deviations areplotted as shaded regions.
Comparison to state-of-the-art baselines.
We consider ﬁve baselines: Dec_UCB (Zhu et al.
,2021a),
GosInE (Chawla et al., 2020) DPE2 (Wang et al., 2020a), ComEx (Madhushani & Leonard,2021), and COUCB .
The ﬁrst four are discussed in Table 1.
To compare all algorithms fairly,we implement them using the same undirected complete communication graph where agents canbroadcast its observations to all other agents.
The communication times of GosInE is set to as smallas the algorithm allows.
COUCB is a naïve adaptation of UCB in CMA2B , where each agent runs aninstance of UCB and always immediately broadcasts its new reward observations to other agents.
Comparison results are reported in Figure 1.
Both group and individual regrets are reported in
Figures 1a and 1b as a function of time.
UCB - TCOM performs only slightly worse than the best algorithm ( DPE2 in group regrets, COUCB in individual regrets).
Figure 1c compares the communication
[Figure]
times of these algorithms in the log scale.
It corroborates the fact that UCB - TCOM -1’s
O (log(log T ))
communication times lie between ComEx and GosInE ’s O (log T ) and DPE2 ’s O (1) performance.
The performance of UCB - TCOM in different environments.
In Figures 2a and 2b, we vary thenumber of agents M in { 5 , 25 , 45 , 65 , 85 , 105 } and the number of arms K in { 10 , 20 , 40 , 60 , 80 , 100 }respectively while ﬁxing other default values.
The group regret in Figure 2a is ﬂat (compared to
Figure 2b’s group regret), corroborating that it does not increase rapidly with respect to number ofagents M (discussed in Section 4.2 (b) , also note that the second term in (4) does not depend on
T ).
Its decreasing individual regret curve corroborates the scalability advantage of the multi-agentsystem — the more agents in the system, the smaller the maximum individual cost these agents needto pay (discussed in Section 4.2 (c) ).
Figure 2b shows that both group and individual regrets, andcommunication cost increase linearly as the number of arms K increases, which signiﬁes the lineardependence of regret on the number of arms.
[Figure]
C ONCLUSION
In this paper, we proposed a communication policy TCOM which, combined with diverse single-agentbandit learning algorithms, can be used to devise fully distributed cooperative multi-agent banditlearning algorithms.
We speciﬁcally study the UCB - TCOM algorithm where agents pull arms via
UCB and communicates via TCOM .
We show that UCB - TCOM can attain both near-optimal group andindividual regrets while only incurring O (log(log T )) communication costs.
Extensive simulationsare also reported to support our results.
We note that TCOM can also be tuned to balance the regret and communications.
One interestingfuture work is to systemically study the Pareto frontier of group/individual regrets vs. communicationtimes from the perspective of the upper bounds of TCOM and the intrinsic lower bounds (difﬁculties)of CMA2B as well.
A R ELATED W ORKS
CMA2B with a central server or leader.
Many distributed bandits models (Wang et al., 2020b;a;
Shi & Shen, 2021; Shi et al., 2021a; Mehrabian et al., 2020; Shi et al., 2021b) assume the existenceof a central server or elect a leader among agents.
We choose two mostly related works to discuss.
Wang et al.
(2020b) consider the CMA2B model with a central server.
The server collects information from each agent and then sends its aggregated information back to each agent.
Speciﬁcally,
Wang et al.
(2020b) propose a phased-based elimination algorithm that achieves the near-optimalproblem-independent regret bound O ( (cid:112) KMT log( KM )) with ﬁnite number of communicationtimes (independent of the number of rounds T ).
However, when it comes to near-optimal problemdependent regret upper bound which is the focus of this paper, their algorithm suffers logarithmiccommunication times while our algorithm only incurs O (log(log T )) communications.
Wang et al.(2020a) propose the DPE2 algorithm for CMA2B based on a leader-follower paradigm: The leadertakes the responsibility of exploring while other agents (followers) only exploit arms recommendedby the leader.
Although it achieves the optimal group regret with ﬁnite number of communications,the DPE2 algorithm is inherently unfair — one single agent, the leader, incurs almost all regret
O ( K log T ) while other followers only incur ﬁnite regrets O (1) .
We also note that algorithms ofprevious works heavily rely on a central server (or an elected leader) and thus are vulnerable toattacks on the central server (or a leader), while fully distributed algorithms that this paper studiescan tolerate such attacks.
CMA2B without a central server.
The fully decentralized CMA2B model has been studied in Buccapatnam et al.
; Landgren et al.
; Kolla et al.
Martínez-Rubio et al.
(2019),where they mainly focus on minimizing regrets and seldom quantify their communication times.
Some of these algorithms assume that agents can gossip with each other, e.g., via a graph representedby the gossiping matrix (Martínez-Rubio et al., 2019; Chawla et al., 2020), or via peer-to-peerprotocols (Szorenyi et al., 2013).
In their gossip setting, agents can only communicate with theirneighbors.
This is fairly different from our all-agent broadcast communication (in Appendix F.2,we relax the all-agent broadcast in TCOM to communication with neighbors in the graph).
Recently,
Madhushani & Leonard (2021) proposed a communication protocol, ComEx , which aims to limit thecommunications of optimal arm’s reward observations.
Our tunable communication policy TCOMsubsumes ComEx as a special case (see Remark 1).
Also, ComEx incurs O (log T ) communicationtimes while our algorithm only needs O (log(log T )) to attain the near-optimal regret upper bound.
Beyond the above related work, there are other works on distributed bandits such as federatedbandits (Shi & Shen, 2021;
Shi et al., 2021a; Zhu et al., 2021b; Huang et al., 2021), distributedbandits with collision setting (Boursier & Perchet, 2019; Mehrabian et al., 2020; Wang et al., 2020b;
Shi et al., 2021b), and cooperative pure exploration (Hillel et al., 2013; Tao et al., 2019; Karpov et al.,2020).
The federated bandits (Shi & Shen, 2021; Shi et al., 2021a; Zhu et al., 2021b; Huang et al.,2021) consider heterogeneous local reward distributions, which is different from our homogeneousreward environment.
In distributed bandits with collision setting model (Boursier & Perchet, 2019;
Mehrabian et al., 2020; Wang et al., 2020b; Shi et al., 2021b), when more than one agent chooses thesame arm, each of these agents gets a zero reward, while in our model, each of these agents obtainsan independent reward.
Cooperative pure exploration (best arm identiﬁcation) (Hillel et al., 2013;
Tao et al., 2019; Féraud et al., 2019; Karpov et al., 2020) study how multi-agent cooperation canreduce the sample complexity of identifying the best arm which is a different objective from ourwork on regret minimization. Some of their high-level ideas in communication are similar to TCOM ,e.g., Hillel et al. (2013) utilize the doubling phase for communication and Féraud et al.
(2019)’salgorithm communicates bad arms. However, as we discussed in Appendix B.1, our algorithm isnovel in speciﬁc design detail, and thus introduces unique challenges that cannot be addressed by thetechniques of these known works.
Individual regret in CMA2B With the majority of previous CMA2B works studying the group regretobjective, the individual regrets were also studied sparingly, e.g., in stochastic CMA2B (Zhu et al.
,2021a; Zhu & Liu, 2021; Zhu et al., 2020), in adversarial CMA2B (Bar-On & Mansour, 2019), andin federated bandits (Zhu et al., 2021b).
Among them, Zhu et al.
(2021a) was the most relatedto ours, where they devised algorithms that achieved the near-optimal individual regret with O ( T )communication times, while our algorithms, with near-optimal individual regrets as well, only needs
O (log(log T )) communication times.
A. D ETAIL C OMPARISON TO C OM E X
TCOM shares some high-level ideas from known algorithms.
However, when it comes to the algorithmdesign details, TCOM needs to resolve unique challenges to improve the state-of-the-art result.
Thereare three algorithmic ideas in the design of TCOM which enables improving the state-of-the-art resultin the literature.
In the following, we outline each idea separately and clarify its novelty as comparedto the most relevant prior work.1.
On how to identify suboptimal arms: The most related work with similar high-levelalgorithmic idea is ComEx [Madhushani et.
al., 2021] where arms with suboptimal empiricalreward means are identiﬁed as suboptimal in communication.
In TCOM , however, we utilizea group of tunable conﬁdence intervals, which as compared to the empirical means in
ComEx , provides a clearer separation between optimal and suboptimal arms.
Further, weemphasize that although using conﬁdence intervals in bandits is a classic technique forminimizing regret, TCOM instead uses conﬁdence intervals to decide whether an arm’sobservation should be shared or not.
It is worth noting that the dynamic construction of thecommunication arm set emphasizes the need for different proof techniques than those in
ComEx , which is elaborated in Appendix B.1.2.
On how to buffer observations: Observation buffering (a.k.a.
phase-based communication)has been used in previous works, e.g., [Shi et.
al., 2021], where applying this techniquescan achieve a O (log T ) communication times.
However, TCOM has its own techniquefor construction of communication arm sets (relevant to the ﬁrst algorithmic idea above),which already enjoys a O (log T ) communication times improvement, without using batchedcommunication.
Therefore, simply applying the batched communication technique doesnot automatically improve the communication times of TCOM .
Instead, we design an armspeciﬁc observation-buffering mechanism which allows each arm—depending on whetherit lies in the communication arm set—to have its own speciﬁc and dynamic phases forcommunication.
This carefully designed observation-buffering technique can boost TCOM ’sﬁrst communication arm set technique and further reduce the communication times to
O (log(log T )) .3.
On algorithm design with a tunable parameter: Last, unique to our algorithm design,we introduced two tunable parameters α and β that determine how to add arms into thedynamic communication arm set based on their conﬁdence intervals.
This addition providesgreater ﬂexibility to TCOM , which is explained in details in Appendix D.
B P ROOFS IN A NALYSIS (S ECTION 4)
B. H IGHLIGHT OF A NALYSIS C HALLENGES
TCOM utilizes two techniques: communication arm set construction (§3.1.1) and observation-bufferingbroadcast (§3.1.2).
Each technique alone can reduce the communication times to O (log T ) , and TCOMcombines both to further reduce the communication times to O (log(log T )) .
This O (log(log T ))result comes directly from the TCOM policy design.
The major challenge in analysis is to establishthat—with these reduced and delayed sharing observations due to TCOM —the UCB - TCOM algorithmstill preserves the near-optimal group and individual regrets.
To do so, we show (1) the communicationarm set construction technique can prevent communicating most optimal arm observations whileallow communicating most suboptimal arm observations; (2) the observation sharing delay due tothe observation-buffering broadcast technique (upon the communication arm set technique) does nothave an intrinsic impact on UCB - TCOM ’s group and individual regret performance.(1)
Although the communication arm set construction technique shares the high level idea of ComEx (Madhushani & Leonard, 2021), the analysis of ComEx is not applicable to TCOM .
Because the empirical means utilized for identifying the suboptimal arms in ComEx is only a specialcase of the tunable conﬁdence intervals that TCOM relies on.
Therefore, to show that agents canselectively share suboptimal arms’ observations by the communication arm set technique, we provetwo new results: (a) the optimal arm’s observations are often not broadcast (i.e., the optimal arm isoften not in the communication arm set, Theorem 2(ii)); (b) suboptimal arm observations are almost always broadcast (i.e., the suboptimal arms are often in the set, Theorem 2(iii)).
The (a) is proved viashowing that the number of times that the optimal arm is in the communication arm set is no greaterthan that of an suboptimal arm (multiplied by a constant factor); thus is upper bounded by O (log T ) . This is based on revealing a relation between the bandit arm pull policy and the communication armset construction.
Then (b) is proved via showing whenever an agent pulls a suboptimal arm, the arm isinside the communication arm set with a high probability, and thus the suboptimal arm’s observationsare almost always buffered for later communication.
Our observation-buffering broadcast mechanism shares a high-level idea of doubling phase technique,but its speciﬁc design is different from known ones, and, consequently, its analysis addresses novelchallenges. The doubling phase technique was utilized in CMA2B literature, e.g., Boursier & Perchet(2019) ; Wang et al. (2020b)
; Shi et al.
(2021b), where their arm pull policies within a phase wereeither uniformly pulling each arm (Boursier & Perchet, 2019; Wang et al., 2020a) or sticking to severalarms (Shi et al., 2021b).
However, different from these previous “rigid” arm pull policies, UCB - TCOMapplies the “ﬂexible” UCB arm pull policy.
In order to adapt the doubling phase technique to the UCB policy, we propose the observation-buffering broadcast mechanism which separately buffersthe observation of each arms and respectively communicates each arm’s compound observationswhenever this arm’s observation times is doubled (or generally, increased by a β factor).(2) Since this observation-buffering broadcast is different from known doubling phase algorithms,their existing analysis approaches are not applicable to observation-buffering. Hence, proving that thisbroadcast mechanism does not intrinsically deteriorate the near-optimal group and individual regretsof UCB - TCOM is a unique challenge. More speciﬁcally, we prove that the delay due to bufferingonly makes regrets worse by a constant factor.
This is via showing that the observation-bufferingmechanism separately buffers each arm’s observations, and, therefore, the observation delay of eacharm’s observations (as well as the pulling times of this arm during the delay) can be respectivelyupper bounded by at most a constant factor multiplying this arm’s total previous pulling times (see (9)of Theorem 1’s proof as a formal expression).
B. P RELIMINARIES
In this subsection, we provide several basic results and deﬁnitions that are crucial in the later analysis.
Deﬁnition 1 (Type-I and Type-II decisions) .
An agent makes a Type-I decision in time slot t if inthis time slot all arms’ true reward means µ ( k ) lie inside this agent’s tunable conﬁdence intervals( tLCB t ( k, α ) ,
tUCB t ( k, α )) respectively.
Otherwise, the agent makes a Type-II decision.
Lemma 1.
For any arm k ∈ [ K ] , any agent i ∈ [ M ] , and time slot t ∈ [ T ] , the probability that thearm k ’s true reward mean µ ( k ) lies inside its tunable conﬁdence interval ( tLCB t ( k, α ) , tUCB t ( k, α ))(calculated by agent i ) is no less than 1 − 2 t 1 − 2 α 2 .
Proof of Lemma 1.
We ﬁrst bound the probability that the true reward mean is no less than the tunableupper conﬁdence bound tUCB .
[Equation]
where the inequality (a) is by the Hoeffding’s inequality and that the random variable µ ( k ) − ˆ µ t ( k )is (cid:112) 1 / 4ˆ n t ( k ) -subgaussian.
With a similar derivation, the probability that the true reward mean µ ( k ) is no greater than the tunable lower conﬁdence bound tLCB , i.e., µ ( k ) (cid:54) tLCB t ( k, α ) , can also beupper bounded by t 1 − 2 α 2 .
Therefore, by excluding the above two events, the probability that the truereward mean lies inside the tunable conﬁdence interval is no less than 1 − 2 t 1 − 2 α 2 .
Lemma 2 (Bound the number of times of Type-II decisions) .
When α > 1 , the expected total numberof times of Type-II decisions by any agent at any time slot in the system is ﬁnite.
Speciﬁcally, it is lessthan K 2 α 2 − 1 α 2 − 1 .
Proof of Lemma 2.
That a time slot t is Type-II means that there exists at least one arm whose rewardmean lies outside its tunable conﬁdence interval.
We bound the expected total number of times of
Type-II decisions as follows,
[Equation]
K ]
[Equation]
that µ (
[Equation]
tLCB
[Equation]
tLCB
[Equation]
where the inequality (a) holds for Lemma 1, and the inequality (b) holds because α > 1 (i.e.,1 − 2 α 2 < − 1 ).
Next, we state a standard property of UCB algorithms and provide its proof for completeness.
Lemma 3 (Adapted from (Yang et al., 2022, Lemma 2)) .
If at any time t (cid:54) T agent i ∈ [ M ] by UCBmakes a Type-I decision and pulls arm k , the total number of globally observed rewards of the arm kby agent i up to time t is upper bounded as follows,
[Equation]
Proof of Lemma 3.
Given that the suboptimal arm k is pulled by agent i , we have the followinginequality
[Equation]
Otherwise, we have
[Equation]
which contradicts the pulling of arm k .
Note that inequalities (a) and (c) are from Type-I decision’sdeﬁnition that reward means lie inside conﬁdence interval, and inequality (b) is due to the inverse of(8).
Therefore, (8) holds and rearranging (8) concludes the proof.
B. P ROOFS OF G ROUP R EGRET (T HEOREM 1
Proof of Theorem 1.
To bound the regret, one needs to bound the number of times of pulling suboptimal arms by all M agents in the system.
Let k be the index of a suboptimal arm.
In UCB - TCOM ,the reward observations of pulling arm k are only broadcast (together as reward means) wheneverarm k ’s local observation times is increased by a β ratio.
So, there are delays before receiving otheragents’ observations.
These delays may cause each agent pulling additional number of times of armk .
Next, we show that such delays do not signiﬁcantly increase arm
k ’s overall pulling times in thesystem.
Note that the communication set contains all suboptimal arms when agents make Type-I decisions,in which case all suboptimal arm pulls increase the counter N t ( k ) by 1 .
Adding the other cost in
Type-II decisions (at most K 2 α 2 − 1 α 2 − 1 by Lemma 2 and 2 α 2 − 1 α 2 − 1 for each arm), the total pulling times ofan suboptimal arm k among all M agents are upper bounded by (cid:80)
i ∈
[ M ] N t ( k ) + M 2 α 2 − 1 α 2 − 1 .
Recall that τ t ( k ) denotes the last time slot (before time slot t ) when agent i broadcasts arm k ’sbuffered observations’ average reward.
To bound the ﬁrst term in RHS, we have
[Equation]
where inequality (a) is because the counter N t ( k ) is at most β times greater than its value at the lastbroadcast time slot, and inequality (b) is because (cid:80)
i ∈
[ M ] N τ t ( k ) ( k ) is total global observations ofarm k at time slot t which is less than 8∆ − 2 k log T by Lemma 3.
At last, we sum up the regret cost of pulling suboptimal arms in Type-I decisions and the other costin Type-II decisions (in total at most MK 2 α 2 − 1 α 2 − 1 by Lemma 2 given α > 1 ).
So, the group regret isupper bounded as follows
[Equation]
Notice that executing Algorithm 1, agents have the same arm pull behavior.
So, the group regret areevenly divided into each agent’s individual regret and thus the individual regret is upper bound isupper bounded as follows,
[Equation]
B. P ROOFS OF C OMMUNICATION C OSTS (T HEOREM 2)
Proof of Theorem 2(i).
No occurrence of communication for α (cid:54) −√ 2 is because that the pulledarm A t is never inside the communication set C t ( α ) in (3).
When α (cid:54) −√ 2 , an arm’s tLCB is greaterthan its original UCB index, that is, tLCB t
( k, α )
=
ˆ µ t ( k ) − CI t ( k, α ) (cid:62) ˆ µ t ( k )+ (cid:112) 2 log t/ ˆ n t ( k ) =
UCB t ( k ) ; and its tUCB is smaller than its UCB index, that is, tUCB t ( k, α )
=
ˆ µ t ( k ) + CI t ( k, α ) (cid:54)ˆ µ t ( k ) + (cid:112) 2 log t/
ˆ n t ( k ) =
UCB t ( k ) .
Then, for the pulled arm A t , the inverse of the communicationset’s condition holds:
[Equation]
where the inequality ( a ) is because that the pulled arm A t has the highest UCB index in time slott .
Proof of Theorem 2(ii).
If a reward observation of optimal arm is broadcast by agent i at time slot t ,then there exists at least one suboptimal arm k such that the following two events hold at the sametime:
[Equation]
where the ﬁrst event is from the UCB arm selection policy, and the second event holds because thepulled arm 1 belongs to the communication set.
These two events imply:
[Equation]
After rearranging the inequality, we show that the number of observations of the optimal arm 1 isupper bounded by a factor multiplying the number of observations of a suboptimal arm as follows,
[Equation]
Lemmas 2 and 3 together show that in the UCB - TCOM algorithm the maximum pulling times of anysuboptimal arm k is upper bounded as follows
[Equation]
Substituting the above inequality to (10)’s RHS and choosing k = 2 (because 1 / ∆ 2 k is maximal whenk = 2 ) for any time t , we have
[Equation]
Notice that whenever the optimal arm is pulled and lies inside the communication set, the counterˆ n t (1) increases by 1 .
So, the buffering counter N t (1) is no greater than ˆ n t (1) .
That is,
[Equation]
These N t (1) observations are aggregated and communicated at one time in each phase.
These phases’lengths increases in a geometric sequence with common ratio β .
So, the number of communicatedmessages of the optimal arm 1 is at most
[Equation]
Proof of Theorem 2(iii).
We ﬁrst show, if a suboptimal arm is pulled by a Type-I decision, its rewardobservation will deﬁnitely be broadcast, i.e., the pulled arm is inside the communication set.
Becausethe pulled suboptimal arm k ’s tunable lower conﬁdence bound tLCB ( k, α ) is less than the optimalarm 1 ’s tunable upper conﬁdence bound tUCB (1 , α ) : tLCB ( k, α ) (cid:54) µ k < µ 1 (cid:54) tUCB (1 , α ) .
Lemma 2 shows that when α > 1 , agents make Type-I decisions in almost all time slots (except ﬁnite
Type-II decisions).
Therefore, almost all observations of suboptimal arms are broadcast.
Proof of Theorem 2(iv).
We ﬁrst show that when the optimal arm 1 is pulled and the agent makesa Type-I decision in time slot t , the event that tLCB t (1 , α ) >
tLCB t ( k, α ) , ∀ k (cid:54)
= 1 happens withsmall probability.
[Equation]
where inequality (a) holds because (8) in Lemma 3’s proof shows that when agent i makes a Type-Idecision to pull suboptimal arm k , the conﬁdence interval CI ’s width of this arm’s reward meanis no less than half of the arm’s reward gap,
i.e., (cid:113) 2log t
ˆ n t ( k ) (cid:62) ∆ k 2 , and inequality (b) holds becauseα ∆ k − 2 √ 2 µ (1) > 0 and Hoeffding’s inequality.
Next, we show that the number of times that the optimal arm 1 ’s observations is not broadcast isﬁnite.
E
[Equation]
[Equation] ] 1 { A t = 1 and the obs.
is not broadcast } 

[Equation]
t [Equation]
[ T ] 1 { agent i makes a Type-II decision in time t } 
 + E 
[Equation]
t [Equation] [ T ] 1 { A t = 1 and the obs. is not broadcast and a Type-I decision is made in time t } 

[Equation]
E 
[Equation]
t
[Equation]
[ T ] 1 {
[Equation]
t
[Equation]
1 { tLCB t (1 , α ) >
tUCB t ( k, α ) , ∀ k (cid:54) = 1 | A t = 1 } 

[Equation]
] 1 { t is Type-II }  
[Equation]
E   (cid:88) t ∈
[ T ] 1 { tLCB t (1 , α ) >
tLCB t ( k, α ) , ∀ k {+(cid:54) = 
 | A t = 1 } +}
 [Equation]
E  
[Equation]
[-∀ k (cid:54) = 
 | A-] t
[Equation] [Equation]
] 1 { t is
[Equation]
-II [Equation] MK
[Equation]
α
[Equation]
(cid:90)
[Equation]
where the inequality (a) is from applying (12) to the second term, and the inequality (b) bounds thenumber of times of Type-II decisions by Lemma 2.
C A PPLY P EELING T ECHNIQUE TO R ELAX THE T UNABLE P ARAMETER ’ SC ONSTRAINT
In this section, we apply peeling technique (Bubeck, 2010, §2.2) to relax Lemma 2 (thus Theorems 1and 2(iii))
’s
α > 1 constraint to α > 1 √ 2 .
This ﬁner result helps to move the critical point 1in Figure 3’s α -axis to 1 √ 2 , and extend the red arrow range (with near-optimal group regret and
O (log(log T )) communications) to (cid:16) 1 √ 2 , √ 2 (cid:17) .
Lemma 4 (Bound the number of times of Type-II decisions via peeling technique) .
Lemma 2’scondition α > 1 can be relaxed to α > (cid:113) 12 γ via peeling technique for γ ∈ (0 , 1) .
The total numberof Type-II decisions is no greater than
[Equation]
Proof.
The key idea is applying a peeling argument to enhance Lemma 1’s proof.
Denote γ ∈ (0 , 1) .
For any s (cid:54) t , there exists j ∈ (cid:110) 0 , 1 , . . .
, (cid:100) log t log(1 /γ )
(cid:101) (cid:111) such that γ j t < s (cid:54) γ j +1 t .
Then, we have
P ( µ ( k ) (cid:62) tUCB t ( k, α )) = P (cid:32) µ ( k ) − ˆ µ t ( k ) (cid:62) α (cid:115) log t ˆ n t ( k ) (cid:33)
(cid:54) P (cid:32) ∃ s (cid:54) t : µ ( k ) − ˆ µ t ( k ) (cid:62) α (cid:115)
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
where the inequality (a) is from the maximal Hoeffding’s inequality.
Symmetrically
[Table]
we also have
( µ ( k ) (cid:54) tLCB t ( k, α )) (cid:54) (cid:18) 2 + log t log(1 /γ )
(cid:19) t − 2 α 2 γ .
Together, we have
[Equation]
[Equation] [Equation]
E [ C T ] : (via Theorem 2(i))
O (log(log T ))
(via Theorem 2(ii) and 2(iii))
O (log T ) (via Theorem 2(iv))
E [ R T ( A )] :
O ( MT )
O ( K log T ) (via Theorem 1) α α ∈ (cid:0) 1 , √ 2 (cid:1) , UCB -
O (log(log
)) communications.
Substituting Lemma 2’s proof’s third step with the above tighter inequality, we have
[Equation]
where the last inequality requires the exist of integral, i.e., 2 α 2 γ > 1 , from which one has α > (cid:113) 12 γand it becomes (cid:113) 12 when γ → 1 .
D T UNABILITY OF TCOMD. A LGORITHM D ESIGN D ISCUSSION
The parameter α tunes the aggressiveness of identifying an arm as suboptimal.
When the condition isaggressive ( α is small), the communication arm set C ( i ) t ( α ) is small: most arms inside C ( i ) t ( α ) aresuboptimal with high probability, but there may exist some suboptimal arms outside C ( i ) t ( α ) , i.e.,
C ( i ) t ( α ) fails to cover a subset of suboptimal arms.
Therefore, the observations of those suboptimalarms outside the communication arm set cannot be shared, which causes more explorations on them.
That is, the cooperative algorithm’s regret may be large when a communication arm set with small α .
When the condition is conservative ( α is large), C ( i ) t ( α ) tends to be large: it contains not only mostsuboptimal arms but sometimes the optimal arm as well.
As the observation times of optimal arm arealso broadcast, the communication times would be large when α is large.
The parameter β controls the frequency of communicating an arm’s observation.
When β is large,communication times reduce at the expense of longer delays before other agents receive the observations.
Therefore, the group regret increases.
And vice versa when β decreases.
D. T HEORETICAL D ISCUSSION
Impact of communication parameter α .
(1) Theorem 2(i) shows that when α (cid:54) −√ 2 , there isno communication among agents and thus there are no global observations for agents to optimizetheir decisions.
(2) Theorems 2(iii) and 2(iv) shows that when α > 2 √ 2 µ (1) ∆( > √ 2) almost all
[Figure]
reward observations are broadcast.
Hence the number of global observations that were broadcast issimilar to that of total observations of agents.
(3) Theorem 2(ii) shows that when −√ 2 < α < √ 2 ,the number of broadcasts of the optimal arm’s observations is O (log(log T )) .
Additionally, theobservation buffering mechanism also aggregates the suboptimal arms O (log T ) observations into
O (log(log T )) broadcasts.
So, in this case, UCB - TCOM only incurs O (log(log T )) communicationtime.
(4) Combined with (3), Theorem 1 shows that when 1 < α < √ 2 , UCB - TCOM has near-optimalgroup and individual regret upper bounds with only O (log(log T )) communication cost.
We refer tothe interval (cid:0) 1 , √ 2 (cid:1) as the preferable range for α .
Figure 3 summarizes the impact of parameter αon the regret-communication trade-off.
Impact of observation buffering ratio β .
In addition to α , the β can also be used to tune theregret-communication trade-off.
When the β increases, Theorems 1 shows that both group andindividual regrets increase.
At the same time, note that β is the logarithm base in (6) of Theorem 2(ii),which means the communication times of TCOM decreases as β increases.
Similarly, the regretsdecrease and communication times increases as the ratio β decreases.
D. S IMULATIONS ON T UNABILITY
[Figure] Figure 4 illustrates the impact of communication set parameter α and observation buffering ratioβ on the performance of UCB - TCOM .
From regret aspect, Figure 4a shows that both the group andindividual regrets do not change much as α increases, which conﬁrms the regret upper bounds in
Theorem 1 where α does not appear on the dominating logarithmic term of (4) and (5).
Figure 4bshows that the regrets increase with respect to β , which corresponds to the appearance of β in thedominating logarithmic term of regret bounds in Theorem 1.
From communication aspect, Figure 4aand Figure 4b also corroborate (6) of Theorem 2, where the communication times increases withrespect to α and decreases with respect to β . E A PPLY TCOM TO O THER B ANDITS A LGORITHMS
In this section, we illustrate how TCOM can be applied to other bandits algorithms. Speciﬁcally, wedevise the TS-TCOM algorithm based on the Thompson sampling ( TS ) algorithm (Agrawal & Goyal,2012) and the AAE-TCOM algorithm based on the active arm elimination ( AAE ) algorithm
(Even-Daret al., 2006).
For the ease of presenting TS-TCOM , we restrict our reward distributions to Bernoulli (same as oursimulation setting) and assume the prior of all arms’ rewards are Beta (1 , 1) .
Then, at time t , thearm k ’s reward posterior distribution is Beta (ˆ n t ( k )ˆ µ t ( k ) + 1 , ˆ n t ( k ) + 1) , where the ˆ n t ( k ) and ˆ µ t ( k )are the global pulling times and the global reward mean estimate of arm k respectively.
We present
TS-TCOM in Algorithm 2.
The only difference between TS-TCOM and UCB-TCOM is in Line 4-5.
1: Input: the communication arm set parameter α and buffering ratio β 2: Initialization: ˆ n t ( k ) = 0 , N t ( k ) = 0 , ˆ µ t ( k ) = 0 , τ t ( k ) =3: for each decision round t do 4:
For each arm k ∈ K , sample θ t ( k ) from the Beta (ˆ n t ( k )ˆ µ t ( k ) + 1 , ˆ n t ( k ) + 1) distribution.
Pull arm A t with the highest θ t ( k ) 6:
Observe arm A t ’s reward X t ( A t ) 7:
if A t ∈ C t ( α ) then 8:
Increase N t
( A t ) by9:
Update this phase’s empirical mean ˜ µ t ( A t ) 10: end if 11: if N t ( A t ) (cid:62) (cid:100) βN
τ t ( A t ) ( A t ) (cid:101) then 12:
Broadcast the message (˜ µ t ( A t ) , N t ( A t ) , A t ) 13: τ t ( A t ) ← t 14: end if 15: end for 16: for each newly received message (˜ µ t ( k ) , N t ( k ) , k ) from the past decision round do 17:
Update the empirical mean ˆ µ t ( k ) ← ˆ µ t ( k )ˆ n t ( k )+˜ µ t ( k ) (cid:98) N t ( k )(1 − 1 /β )
(cid:99)
ˆ n t ( k )+ (cid:98) N t ( k )(1 − 1 /β )
(cid:99) 18:
Increase ˆ n t ( k ) by (cid:98) N t ( k )(1 − 1 /β )
(cid:99) 19:
Update the communication arm set C t ( α ) via (3) based on tunable conﬁdence bounds 20: end for communication arm set parameter α and buffering ratio β 2: Initialization: ˆ
0 , ˆ µ t ( τ t (
) =3: for each decision round do 4:
Update candidate arm set D t via (13) 5:
Pull arm A t with the smallest global observation times ˆ n t ( k ) among arms in D t 6:
Observe arm A t ’s reward X t ( A t ) 7:
if A t ∈ C t ( α ) then 8:
Increase N t
( A t ) by9:
Update this phase’s empirical mean ˜ µ t ( A t ) 10: end if 11: if N t ( A t ) (cid:62) (cid:100) βN
τ t ( A t ) ( A t ) (cid:101) then 12:
Broadcast the message (˜ µ t ( A t ) , N t ( A t ) , A t ) 13: τ t ( A t ) ← t 14: end if 15: end for 16: for each newly received message (˜ µ t ( k ) , N t ( k ) , k ) from the past decision round do 17:
Update the empirical mean
[Table]
( k )
[Table]
[Table]
Increase ˆ n t ( k ) by (cid:98) N t ( k )(1 − 1 /β )
(cid:99) 19:
Update the communication arm set C t ( α ) via (3) based on tunable conﬁdence bounds 20: end for
For AAE-TCOM , agents need to construct a global candidate arm set D t which is deﬁned as follows,
[Equation]
, (13) where UCB t ( k ) :
= ˆ µ t ( k ) + (cid:112) 2 log t/
ˆ n t ( k ) is the same as UCB - TCOM , and LCB t ( k ) :
= ˆ µ t ( k ) − 778 (cid:112) 2 log t/ ˆ n t ( k ) .
We note that since both ˆ µ t ( k ) and ˆ n t ( k ) are global, all agents’ candidate arm setare the same.
Then, at each time slot t , agents pull the arm with smallest observation times among allarms in the candidate arm set, i.e., arg min k ∈D t
ˆ n t ( k ) .
We present AAE-TCOM in Algorithm 3.
Thedifference between AAE-TCOM and UCB-TCOM is also in Line 4-5.
In Figure 5, we report the performance comparison between UCB - TCOM , TS-TCOM , and AAE-TCOM .
In Figure 5a and 5b, the AAE-TCOM algorithm has the worse regret performance than UCB - TCOM , and
UCB - TCOM is worse than TS-TCOM .
This observation matches the folklore observation in bandits that
[Figure]
the empirical performance of TS is better than that of UCB , and UCB is better than AAE .
Comparingthe individual regret in Figure 5b to the group regret in Figure 5a shows that the AAE-TCOM and
TS-TCOM algorithms have good individual regret performance.
Figure 5c shows that these threealgorithms need the similar communication times, which validates the communication efﬁciency of
TS-TCOM and AAE-TCOM .
We believe that the TS-TCOM and AAE-TCOM algorithms (like UCB - TCOM ) also enjoy the
O (log(log T )) communication times and near-optimal group and individual regrets.
However, thisrequires new analysis, especially for the Thompson sampling case, because its Bayesian approachis very different from the method utilized in UCB and TS-TCOM needs to validate the symmetriclearning structure.
Studying both new algorithms’ theoretical performance is an interesting futurework.
In this section, we extend the basic CMA2B model in §2 to allow deterministic communicationdelays and also from that the agents can broadcast to any others to the case that agents residing on a network can only communicate with their neighbors (i.e., cannot broadcast).
In the communicationnetwork model, we also allow agents to passing its received messages to other neighbors, i.e., allowmessage-passing.
We show that UCB - TCOM (with minor changes) still enjoys the near-optimal groupand individual regret upper bounds as in Theorem 1 with only constant additional cost (independentof T ).
We also note that the communication times results in Theorem 2 still hold because its proofsare not inﬂuenced by these extensions.
F. D ETERMINISTIC D ELAY
Denote d ∈ N
+ as the deterministic delay of agents’ message broadcasts.
The deterministic delaycan also be relaxed to random delay but with the d as the delay upper bound.
Algorithmically, theglobal UCB utilized for selecting arms to pull should be re-deﬁned to those observations that havebeen broadcast d time slots before .
Because only these reward observations have been received by allagents even with the communication delay and thus agents utilize the same observations to makedecisions.
Theorem 3.
When the communication arm set parameter α > 1 and buffering-ratio β > 1 , given all delays of communication is no greater than d , UCB - TCOM attains a near-optimal group regret upper bound in terms of number of decision rounds T , arms K , and agents M , or formally,
[Equation]
and UCB - TCOM also attains a near-optimal individual regret upper bound, or formally,
[Equation]
Proof of Theorem 3.
The proof procedure of Theorem 1 still applies to Theorem 3.
The only changeis that the inequality (9) should be updated as follows,
[Equation]
where inequality (a) is because the counter N t ( k ) is at most β times greater than its value at the lastbroadcast time slot plus the delay d , and inequality (b) is because (cid:80)
i ∈
[ M ] N τ t ( k ) ( k ) is total globalobservations of arm k at time slot t which is less than 8∆ − 2 k log T by Lemma 3.
With the rest proof the same as Theorem 1’s, UCB - TCOM ’s group regret is upper bounded as follows:
[Equation]
As the symmetry of UCB - TCOM still holds, the individual regret upper bound immediately follows,
[Equation]
F. C OMMUNICATION T OPOLOGY : P EER - TO - PEER AND MESSAGE PASSING COMMUNICATION
We assume the communication network is a connected graph, i.e., there exists a path between anytwo nodes, and the diameter of the graph is D .
In the message-passing protocol (following Dubeyet al.
(2020)’s model), agents i ∈
[ M ] communication via messages ( i, t, ˜ µ it ( A it ) , N it ( A it ) , A it ) .
Thismessage is sent to its neighbors in the graph and then is forwarded to by any agents when receives itto their neighbors until the time slot t
+ D .
Under this protocol, the broadcast message will ﬁnalreach to any agents with a delay at most D .
In algorithmic aspect, the TCOM algorithm still workswith minor modiﬁcations:
Change the Line 11 in Algorithm 1 from broadcast to communication to neighbors;2.
Add a line after Line 11 that an agent should also send all its received messages with timeindex t (cid:48) <
t + D to neighbors on the graph;3.
Line 15 should be changed to receive all messages that have not been received before (i.e.,with unique i, t preﬁx in the message).
With the above modiﬁcations in UCB - TCOM , the algorithm solves a problem equivalent to a CMA2Bwith communication delay D on a complete graph.
Therefore, based on Theorem 3, one obtains thefollowing corollary to bound the group and individual regrets of the modiﬁed UCB - TCOM algorithm.
Corollary 4.
When the communication arm set parameter α > 1 and buffering-ratio β > 1 , given all agents residing on a network with diameter D , UCB - TCOM attains a near-optimal group regret upper bound in terms of number of decision rounds T , arms K , and agents M , or formally,
[Equation]
and UCB - TCOM also attains a near-optimal individual regret upper bound, or formally,
[Equation]