D ISTRIBUTED B ANDIT L EARNING : N EAR -O PTIMAL R EGRET WITH E FFICIENT C OMMUNICATION
A BSTRACT
We study the problem of regret minimization for distributed bandits learning, in which M agents work collaboratively to minimize their total regret under the coordination of a central server.
Our goal is to design communication protocols with near-optimal regret and little communication cost, which is measured by the total amount of transmitted data.
For distributed multi-armed bandits we propose a protocol with near-optimal regret and only O ( M log( MK )) communication cost, where K is the number of arms.
The communication cost is independent of the time horizon T has only logarithmic dependence on the number of arms, and matches the lower bound except for a logarithmic factor.
For distributed d -dimensional linear bandits, we propose a protocol that achieves near-optimal regret and has communication cost of order O (( Md + d log log d ) log T ) , which has only logarithmic dependence on T
I NTRODUCTION
Bandit learning is a central topic in online learning, and has various real-world applications, including clinical trials (Wang, 1991), model selection (Maron & Moore, 1994) and recommendation systems (Agarwal et al., 2009;
Li et al., 2010; Abe et al., 2003).
In many tasks using bandit algorithms, it is appealing to employ more agents to learn collaboratively and concurrently in order to speed up the learning process.
In many other tasks, the sequential decision making is distributed by nature.
For instance, multiple spatially separated labs may be working on the same clinical trial.
In such distributed applications, communication between agents is critical, but may also be expensive or time-consuming.
Another example is a recommendation system deployed on multiple servers to handle high demand.
Since the communication between servers may cause service latency, it would be desirable to design communication strategies without communicating too much.
This motivates us to consider efﬁcient protocols for distributed learning in bandit problems.
A straightforward communication protocol for bandit learning is immediate sharing : each agent shares every new sample immediately with others.
Under this scheme, agents can have good collaborative behaviors close to that in a centralized setting.
However, the amount of communicated data is directly proportional to the total size of collected samples.
When the bandit is played for a long timescale, the cost of communication would render this scheme impractical.
A natural question to ask is: How much communication is actually needed for near-optimal performance?
In this work, we show that the answer is somewhat surprising: The required communication cost has almost no dependence on the time horizon.
In this paper, we consider the distributed learning of stochastic multi-armed bandits (MAB) and stochastic linear bandits.
There are M agents interacting with the same bandit instance in a synchronized fashion.
In time steps t = 1 , · · · , T , each agent pulls an arm and observes the associated reward.
Between time steps, agents can communicate via a server-agent network.
Following the typical formulation of single-agent bandit learning, we consider the task of regret minimization (Lai et al., 1987;
Dani et al., 2008; Bubeck et al., 2012).
The total regret of all agents is used as the performance criterion of a communication protocol.
The communication cost is measured by the total amount of data communicated in the network.
Our goal is to minimize communication cost while maintaining near-optimal performance, that is, regret comparable to the optimal regret of a single agent in MT interactions with the bandit instance.
For multi-armed bandits, we propose the DEMAB protocol, which achieves near-optimal regret.
The amount of transmitted data per agent in DEMAB is independent of T , and is logarithmic with respect to other parameters.
For linear bandits, we propose the DELB protocol, which achieves near-optimal regret, and has communication cost with at most logarithmic dependence on T .
1. P ROBLEM S ETTING
Communication Model
The communication model we consider consists of a server and several agents.
Agents can communicate with the server by sending or receiving packets.
Each data packet contains integers or real numbers.
We deﬁne the communication cost of a protocol as the number of integers or real numbers communicated between server and agents 1 .
We assume that communication between server and agents has zero latency.
Note that protocols in our model can be easily adapted to a network without a server, by designating an agent as the server.
Distributed Multi-armed Bandits In distributed multi-armed bandits, there are M agents, labeled 1 ,..., M .
Each agent is given access to the same stochastic K -armed bandit instance.
Each arm k in the instance is associated with a reward distribution P k .
P k is supported on [0 , 1] with mean µ ( k ) .
Without loss of generality, we assume that arm 1 is the best arm (i.e. µ (1) ≥ µ ( k ) , ∀ k ∈
[ K ] ).
At each time step t = 1 , 2 , ..., T , each agent i chooses an arm a t,i , and receives reward r t,i independently sampled from P a t,i .
The goal of the agents is to minimize their total regret, which is deﬁned as
[Equation] =
[Equation]
Distributed Linear Bandits In distributed linear bandits, the agents are given access to the same d -dimensional stochastic linear bandits instance.
In particular, we assume that at time step t , agents are given an action set D ⊆ (cid:8) x ∈ R
d : (cid:107) x (cid:107) 2 ≤ 1 (cid:9) .
Agent i chooses action x t,i ∈ D and observes reward y t,i .
We assume that the mean of the reward is decided by an unknown parameter θ ∗ ∈ R d : y t,i = x
Tt,i θ ∗ + η t,i , where η t,i ∈ [ − 1 , 1] are independent and have zero mean.
For simplicity, we assume (cid:107) θ ∗ (cid:107) 2 ≤ 1 .
For distributed linear bandits, the cumulative regret is deﬁned as the sum of individual agent’s regrets:
[Equation]
Here, we assume that the action set is ﬁxed.
A more general setting considers a time-varying action set D t .
In both cases, algorithms with O ( d √ T log T ) regret have been proposed (Abbasi-Yadkori et al., 2011), while a regret lower bound of Ω (cid:16) d √ T (cid:17) is shown in Dani et al.
For both distributed multi-armed bandits and distributed linear bandits, our goal is to use as little communication as possible to achieve near-optimal regret.
Since any M -agent protocol running for
[Table]
T steps can be simulated by a single-agent bandit algorithm running for MT time steps, the regret of any protocol is lower bounded by the optimal regret of a single-agent algorithm running for MT time steps.
Therefore, we consider ˜ O ( √ MKT ) regret for multi-armed bandits and
˜ O
( d √ MT ) regret for linear bandits to be near-optimal.
We are mainly interested in the case where the time horizon T is the dominant factor (compared to
M or K ).
Unless otherwise stated, we assume that T > max { M log 2 M K , M, K } in the multi-armed bandits case and T > M in the linear bandits case.
1. O UR C ONTRIBUTION
Now we give an overview of our results.
In both settings, we present communication-efﬁcient protocols that achieve near-optimal regret.
Our results are summarized in Table 1.
Our results are compared with a naive baseline solution called immediate sharing in Table 1: each agent sends the index of the arm he pulled and the corresponding reward he received to every other agent via the server immediately.
This protocol can achieve near-optimal regret for both MAB and linear bandits ( O ( √ MKT ) and ˜ O ( d √ MT ) ), but comes with high communication cost ( O ( M 2 T ) and
O ( M 2 dT ) ).
Distributed MAB For distributed multi-armed bandits, we propose DEMAB (Distributed Elimination for MAB) protocol, which achieves near-optimal regret ( O (cid:16)(cid:112)
MKT log( MK ) (cid:17) ) with
O ( M log( MK )) communication cost.
The communication cost is independent of the number of time steps T and grows only logarithmically w.r.t.
the number of arms.
We also prove the following lower bound:
For any protocol with regret bound o ( M √ KT ) , the expected communication cost is at least Ω( M ) .
That is, in order to achieve near-optimal regret, the communication cost of DEMAB matches our lower bound except for logarithmic factors.
Distributed Linear Bandits We propose DELB (Distributed Elimination for Linear Bandits), an elimination based protocol for distributed linear bandits which achieves near-optimal regret bound ( O (cid:0) d √ MT log T (cid:1) ) with communication cost O (( Md + d log log d ) log T ) .
The communication cost of DELB enjoys nearly linear dependence on both M and d , and has at most logarithmic dependence on T .
For the more general case where the action set is time-varying, we propose
DisLinUCB (Distributed LinUCB) protocol, which achieves near-optimal regret with O (cid:0) M 1 .
5 d 3 (cid:1) communication cost.
R ELATED W ORK
There has been growing interest in bandits problems with multiple players.
One line of research considers the challenging problem of multi-armed bandits with collisions (Rosenski et al., 2016;
Bistritz & Leshem, 2018; Kalathil et al., 2014), in which the reward for an arm is 0 if it is chosen by more than one player.
The task is to minimize regret without communication.
Their setting is motivated by problems in cognitive radio networks, and is fairly different from ours.
Szörényi et al.
(2013) and Korda et al.
(2016) consider the distributed learning of MAB and linear bandits with restriction on the communication network.
Motivated by fully decentralized applications,
Szörényi et al.
(2013) consider P2P communication networks, where an agent can communicate with only two other agents at each time step.
A gossip-based (cid:15) -greedy algorithm is proposed for distributed
MAB.
Their algorithm achieves a speedup linear in M in terms of error rate, but the communication cost is linear in T .
The work of Korda et al.
(2016) uses a gossip protocol for regret minimization in distributed linear bandits.
The main difference between their setting and ours is that each agent is only allowed to communicate with one agent at each time step in Korda et al.
Their algorithm achieves near-optimal (cid:16) O (cid:16) d √ MT log T (cid:17)(cid:17) total regret using O ( Md 2 T ) communication cost.
Another setting in literature concerns about distributed pure exploration in multi-armed bandits (Hillel et al., 2013; Tao et al., 2019), where the communication model is the most similar one to ours.
These works use elimination based protocols for collaborative exploration, and establish tight bounds for communication-speedup tradeoff.
However, their task (speedup in pure exploration) is not directly comparable to ours (i.e. are not reducible to each other).
Moreover, in Hillel et al.
; Tao et al.
(2019), the number of communication rounds is used as the measure of communication, while we use the amount of transmitted data.
M AIN R ESULTS FOR M ULTI - ARMED B ANDITS
In this section, we ﬁrst summarize the single-agent elimination algorithm (Auer & Ortner, 2010), and then present our Distributed Elimination for MAB (DEMAB) protocol.
The regret and communication efﬁciency of the protocol is then analyzed in Sec. 3.3.
A communication lower bound is presented in
Sec. 3.4.
3. E LIMINATION A LGORITHM FOR S INGLE - AGENT MAB
The elimination algorithm (Auer & Ortner, 2010) is a near-optimal algorithm for single-agent MAB.
The agent acts in phases l = 1 , 2 , · · · , and maintains a set of active arms A l .
Initially, A 1 =
[ K ] (all arms).
In phase l , each arm in A l is pulled for Θ(4 l log T ) times; arms with average reward 2 − l lower than the maximum are then eliminated from A l .
For each arm k ∈ [ K ] , deﬁne its suboptimality gap to be ∆ k := µ (1) − µ ( k ) .
In the elimination algorithm, with high probability arm k will be eliminated after approximately l k = log 2 ∆ − 1 k phases, in which it is pulled for at most O (cid:0) ∆ − 2 k log T (cid:1) times.
It follows that regret is O (cid:0)(cid:80) k :∆ k > 0 ∆ − 1 k log T (cid:1) , which is almost instance-optimal.
The regret is O (cid:0) √ KT log T (cid:1) in the worst case.
In order to apply elimination algorithm to our setting and remove the log T dependence in the regret bound, we slightly modify the elimination algorithm:
In phase l , each arm in A l is pulled for Θ(4 l log(4 − l T )) instead of Θ(4 l log( T )) times.
By doing this, the regret bound can be improved from √ KT log T to √ KT log K .
Our DEMAB protocol is based on this modiﬁed elimination algorithm.
See Section 3.2 and Appendix C for detailed explanation.
3. T HE DEMAB P ROTOCOL
The DEMAB protocol executes in two stages.
In the ﬁrst stage, each agent directly runs the singleagent modiﬁed elimination algorithm for D =
(cid:100) T/MK (cid:101) time steps.
The remaining arms of agent i are denoted as A ( i ) .
In D time steps, an agent completes at least l 0 = (cid:98) log
[Equation]
phases.
The purpose of this separate burn-in period is to eliminate the worst arms quickly without communication, so that in the second stage, elimination can begin with a small threshold of O (2 − l 0 ) .
D and l 0 is chosen so that the total regret within the ﬁrst stage is ˜ O (cid:16) √ MT (cid:17) .
1 D =
(cid:100) T/MK (cid:101) , l 0 = (cid:98) log 4 (cid:16)
D 1000 K log( MK ) (cid:17) (cid:99) , m l = (cid:98) 4 l +3 log( MKT · 4 − l ) (cid:99) /
* Stage 1:
Separate Burn-in */ 2 for Agent i = 1 , · · · , M do
Agent i runs single-agent elimination for D time steps, denote remaining arms as A ( i ) /
* Switching:
Random Allocation */ 4 Generate public random numbers r 1 ,..., r K uniformly distributed in [ M ] 5 B ( i ) l 0 +1 = (cid:8) a ∈ A ( i ) | r a
= i (cid:9) , B l 0 +1 =
(cid:83)
i ∈
[ M ] B ( i ) l 0 + /* Stage 2:
Distributed Elimination */ 6 for l = l 0 + 1 , · · · do if N l = |B l | > M then
Agent i sends n ( i l = to server; server broadcasts n max = max i n i ) l if ( n (1) l , ..., n ( M ) l ) is not balanced then
Reallocate //
Adjust B ( i ) l so that their sizes are balanced
Agent i pulls each a ∈ B ( i ) l for m l times, denotes average reward as ˆ u l ( a ) , and then pulls arms in round-robin for ( n max − n ( i ) l ) m l times before the next communication round
Communication round: Agent i sends max a ∈B ( i ) l ˆ u l to server and waits to receive u ∗ l = max a ∈B l ˆ u l from server
Agent i eliminates bad arms: B ( i ) l +1 =
(cid:110)
[Table]
[Table]
: ˆ u
[Table]
[Equation]
a
[Equation]
[Table]
(cid:111) else in l asks M/N
agents
pull it m l N l /M [Table] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
[Table]
Between the two stages, the remaining arms are randomly allocated to agents.
Public randomness is used to allocate the remaining arms to save communication.
Agents ﬁrst generate ( r 1 , · · · , r K ) , K uniformly random numbers in [ M ] , from a public random number generator. Agent i then computes B ( i ) = (cid:8) a ∈ A ( i ) | r a
= i (cid:9) .
By doing so, agent i keeps each arm in A ( i ) with probability 1 /M , and the resulting sets B (1) , · · · , B ( M ) are disjoint.
Meanwhile, every arm in (cid:84) i ∈
[ M ] A ( i ) is kept in
B l 0 +1 = (cid:83) i ∈ [ M ] B ( i ) , so that the best arm remains in B l 0 +1 with high probability 3 .
In the second stage, agents start to simulate a single-agent elimination algorithm starting from phase l 0 + 1 . Initially, the arm set is B l 0 +1 . In phase l , each arm in B l will be pulled for at least m l = (cid:100)
4 l +3 log( MKT · 4 l ) (cid:101) times.
Denote the average reward of arm a in phase l by ˆ u l ( a ) . If ˆ u l ( a ) < max a (cid:48) ∈B l ˆ u l ( a (cid:48) ) − 2 − l , it will be eliminated; the arm set after the elimination is B l +1 .
This elimination in the second stage is performed over M agents in two ways: In distributed mode or in centralized mode .
Let N l = |B
l | be the number of remaining arms at the start of phase l .
If
N l is larger than M , the elimination is performed in distributed mode .
That is, agent i keeps a set of arms B ( i ) l , and pulls each arm in B ( i ) l for m l times in phase l .
Each agent only needs to send the highest average reward to the server, who then computes and broadcasts max a ∈B l ˆ u l ( a ) .
Agent i then eliminates low-rewarding arms from B ( i ) l on its local copy.
When N l ≤ M , the elimination is performed in centralized mode .
That is, B l will be kept and updated by the server 4 .
In phase l , the server assigns an arm in B l to M/N l agents, and asks each of them to pull it m l N l /M times 5 .
The server waits for the average rewards to be reported, and then performs elimination on B l .
One critical issue here is load balancing , especially in distributed mode.
Suppose that n ( i ) l = |B
( i ) l | , n max = max i ∈
[ M ] n ( i ) l .
Then the length of phase l is determined by n max m l .
Agent i would need to keep pulling arms for ( n max − n ( i ) l ) m l times until the start of the next communication round.
This will cause an arm to be pulled for much more than m l times in phase l , and can hurt the performance.
Therefore, it is vital that at the start of phase l , (cid:126)n l := (cid:16) n (1) l , ..., n ( M ) l (cid:17) is balanced 6 .
The subroutine Reallocate is designed to ensure this by reallocating arms when (cid:126)n l is not balanced.
First, the server announces the total number of arms; then, agents with more-than-average number of arms donate surplus arms to the server; the server then distributes the donated arms to the other agents, so that every agent has the same number of arms.
However, calling Reallocate is communicationexpensive: it takes O (min { N l , N l (cid:48) − N l } ) communication cost, where l is the current phase and l (cid:48) is the last phase where Reallocate is called.
Fortunately, since (cid:110) B ( i ) l 0 +1 (cid:111) i ∈
[ M ] are generated randomly, it is unlikely that one of them contain too many good arms or too many bad arms.
By exploiting shared randomness, we greatly reduce the expected communication cost needed for load balancing.
Detailed descriptions of the single-agent elimination algorithm, the Reallocate subroutine, and the DEMAB protocol are provided in Appendix B.
Access to a public random number generator, which is capable of generating and sharing random numbers with all agents with negligible communication cost, is assumed in DEMAB.
This is not a strong assumption, since it is well known that a public random number generator can be replaced by private random numbers with a little additional communication (Newman, 1991).
In our case, only O ( M log T ) additional bits of communication, or O ( M ) additional communication cost, are required for all of our theoretical guarantees to hold.
See Appendix C for detailed discussion.
3. R EGRET AND C OMMUNICATION E FFICIENCY OF DEMAB
In this subsection, we show that the DEMAB protocol achieves near-optimal regret with efﬁcient communication, as captured by the following theorem.
Theorem 1.
The DEMAB protocol incurs O (cid:16)(cid:112) MTK log( MK ) (cid:17) regret, O (cid:0)
M log MKδ (cid:1) communication cost with probability 1 − δ , and O ( M log( MK ))
communication cost in expectation.
The worst-case regret bound above can be improved to an instance-dependent near-optimal regret bound by changing the choice of D and l 0 to 0 .
In that case the communication cost is O
( M log T ) , which is a small increase.
See Theorem 5 in appendix for detailed discussion.
We now give a sketch of the proof of Theorem 1.
Regret In the ﬁrst stage, each agent runs a separate elimination algorithm for D timesteps, which has regret O ( (cid:112) KD log( MK )) .
Total regret for all agents in this stage is O ( M √ KD log D ) =
O ( (cid:112) MT log( MK )) .
After the ﬁrst stage, each agent must have completed at least l 0 phases.
Hence, with high probability, before the second stage, B l 0 +1 =
(cid:83)
i ∈
[ M ] B ( i ) l 0 +1 contains the optimal arm and only arms with suboptimality gap less than 2 − l 0 +1 .
In the second stage, if a ∈ B l , it will be pulled at most 2 m l times in phase l because of our load balancing effort.
Therefore, if arm k has suboptimality gap 0 < ∆ k < 2 − l 0 +1 , it will be pulled for ˜ O (cid:0) ∆ − 2 k (cid:1) times.
It follows that regret in the second stage is O (cid:16) (cid:112) MKT log( MK ) (cid:17) , and that total regret is O (cid:16)(cid:112) MKT log ( MK ) (cid:17) .
Communication In the ﬁrst stage and the random allocation of arms, no communication is needed.
The focus is therefore on the second stage.
During a phase, apart from the potential cost of calling Reallocate , communication cost is O ( M ) .
The communication cost of calling Reallocate in phase l is at most O (min { N l , N l (cid:48) − N l } ) , where l (cid:48) is the last phase where Reallocate is called.
Therefore, total cost for calling
Reallocate in one execution is at most O ( N l 1 ) , where l 1 is the ﬁrst phase in which Reallocate is called.
From the deﬁnition of m l and l 0 , we can see that there are at most L =
O (log( MK )) phases in the second stage.
Therefore in the worst case, communication cost is O ( ML + K ) since
N l 1 ≤ K .
However, in expectation, N l 1 is much smaller than K .
That is to say, Reallocate is called for the ﬁrst time when N l is much smaller that K with high probability.
Since arms are randomly allocated to agents during switching step, when the total number of remaining arms is large enough, (cid:126)n l would be balanced with high probability.
In fact, with probability 1 − δ , N l 1 = O (cid:0) M log MKδ (cid:1) .
Setting δ = 1 /K , we can show that the expected communication complexity is O ( M log MK ) .
3. L OWER B OUND
Intuitively, in order to avoid a Ω (cid:16) M √ KT (cid:17) scaling of regret, Θ( M ) amount of communication cost is necessary; otherwise, most of the agents can hardly do better than a single-agent algorithm.
We prove this intuition in the following theorem.
Theorem 2.
For any protocol with regret bound o ( M √ KT ) , the expected communication cost is at least Ω( M ) .
The theorem is proved using a reduction from single-agent bandits to multi-agent bandits, i.e. a mapping from protocols to single-agent algorithms.
This theorem shows that, in order to achieve near-optimal regret (cid:16) ˜ O (cid:16) √ MKT (cid:17)(cid:17) , the communication cost is at least Ω( M ) .
That is to say, the communication cost of DEMAB protocol matches the lower bound except for a logarithmic factor.
M AIN R ESULTS FOR L INEAR B ANDITS
In this section, we ﬁrst summarize the single-agent elimination algorithm for linear bandits (Lattimore & Szepesvári, 2019, chap. 22), and then present the Distributed Elimination for Linear Bandits (DELB) protocol.
DELB is designed for the case where the action set D is ﬁxed.
Our results for linear bandits with time-varying action set is presented in Sec. 4.4.
For convenience, we assume D is a ﬁnite set, which is without loss of generality from a regret point of view 7 .
4. E LIMINATION A LGORITHM FOR S INGLE - AGENT L INEAR B ANDIT
The elimination algorithm for linear bandits (Lattimore & Szepesvári, 2019) also iteratively eliminates arms from the initial action set.
In phase l , the algorithm maintains an active action set A l .
It computes a distribution π l ( · ) over A l and pulls arms according to π l ( · ) .
A total of n l pulls is made in this phase according to π l ( · ) .
Then, linear regression is used to estimate the expected reward of each arm based on these pulls.
Arms with estimated rewards Θ(2 − l ) lower than the maximum are eliminated at the end of the phase.
To ensure that arms with suboptimality gap Θ(2 − l ) are eliminated in phase l with high probability, the estimation error in phase l needs to be smaller than 2 − l .
On the other hand, to minimize regret, the number of pulls make in phase l needs to be as small as possible, especially when l is small.
Suppose that V l ( π )
=
(cid:80) x ∈A l π ( x ) xx (cid:62) and g l ( π ) =
max x ∈A l x (cid:62) V l ( π ) − 1 x .
By analyzing linear regression (see, e.g., Lattimore & Szepesvári (2019, chap. 21)), one can show that if each arm x ∈ Supp( π l ) is pulled (cid:6) π ( x ) g l ( π )4 l log (cid:0) 1 δ (cid:1)(cid:7) times, the estimation error for any arm x ∈
A l is at most 2 − l with high probability.
Thus, to minimize m l , one should ﬁnd a distribution π ( · ) that minimizes g l ( π ) , a task known as G -optimal design (Pukelsheim, 2006).
It is known that the optimal value for g l ( π ) is d , and that there exists a minimizer π ∗ such that the support set of π ∗ , also called the core set, has cardinality at most d ( d +1) / 2 .
Consequently, only (cid:80) x ∈ Supp( π ∗ l ) (cid:100) π ∗ ( x ) g l ( π ∗ )4 l log( 1 δ ) (cid:101) ≤
O (4 l d log( 1 δ )+ d 2 ) pulls are needed in phase l .
4. T HE DELB P ROTOCOL
In this protocol, we parallelize the data collection part of each phase by sending instructions to agents in a communication-efﬁcient way.
In phase l , the server and all agents ﬁrst locally solve the same
G -optimal design problem on A l , the remaining set of actions.
In our case, only a 2 -approximation to the optimal design is needed.
That is, we only need to ﬁnd π l ( · ) with g ( π ) ≤ 2 d .
By using the
Frank-Wolfe algorithm under appropriate initialization (Todd, 2016, Proposition 3.17), we can ﬁnd such a 2 -approximate solution with a support smaller than ξ = 48 d log log d .
After that, the server assigns arms in A l to agents.
Since the server and the agents obtain the same core set when solving the approximate G -optimal design 8 , the server only needs to send the index among ξ arms to identify and allocate each arm.
After pulling arms, agents send the average reward for each arm to the server, who computes the least squares estimator.
The agents and the server then eliminate low rewarding arms from their local copies of A l .
For notational convenience, we deﬁne V ( π )
=
(cid:80) x ∈D π
( x ) xx (cid:62) , g ( π ) =
max x ∈D x
(cid:62) V ( π ) − 1 x.
Protocol
1 A 1 = D , C 1 =2 for l = 1 , 2 , 3 , ... do / * All
agents and server: Solve a G-optimal design problem
Find distribution π l ( · ) over A l such that: 1.
its support has size at most ξ = 48 d log log d ; 2.
g ( π ) ≤ 2 d /* Server:
Assign pulls and summarize results */
Assign m l ( x ) =
(cid:100) C 1 4 l d [Table] π l ( x ) ln MT (cid:101) pulls for [Table]
x ∈ Supp( π l ) Receive rewards
For each arm in the support of π l ( · , calculate the average reward µ ( x )
Compute 10 X
=
(cid:80) [Equation] Supp( π l ) [Equation]
l ( x )
[Equation]
[Equation]
[Equation]
l
X
Send ˆ θ to all agents /* All agents and server:
Eliminate low-rewarding arms
[Equation] Eliminate low rewarding arms
[Equation]
4. R EGRET AND C OMMUNICATION E FFICIENCY OF DELB
We state our results for the elimination-based protocol for distributed linear bandits.
The full proof is given in Appendix G.
Theorem 3.
The DELB protocol achieves expected regret O (cid:0) d √ TM log T (cid:1) with communication cost O (( Md + d log log d ) log T ) .
Proof sketch:
In round l , the number of pulls is at most 48 d log log d + C 1 4 l d 2 log MT .
Based on the analysis for elimination-based algorithm, we can show that the suboptimality gap (cid:104) θ ∗ , x ∗ − x (cid:105) is at most 2 − l +2 with probability 1 − 1 /MT for any arm x pulled in phase l .
Suppose there are at most
L phases, we can prove that E ( REG ( T )) ≤ (cid:80) Ll
=1 O ( d (cid:112) [Equation]
In each phase, communication cost comes from three parts: assigning arms to agents, receiving average rewards of each arm and sending ˆ θ to agents.
In the ﬁrst and second part, each arm x ∈ Supp( π l ) is designated to as few agents as possible.
We can show that the communication cost of these parts is O ( M + d log log d ) .
In the third part, the cost of sending ˆ θ is Md .
Since l is at most
O (log T ) , the total communication is O
(( Md + d log log d ) log T ) .
4. P ROTOCOL FOR L INEAR B ANDITS WITH T IME - VARYING A CTION S ET
In some previous work on linear bandits (Chu et al., 2011; Abbasi-Yadkori et al., 2011), the action set available at timestep t may depend on t .
That is, players can only choose actions from D t at time t , while regret is deﬁned against the optimal action in D t .
The DELB protocol does not apply in this scenario.
To handle this setting, we propose a different protocol DisLinUCB
(Distributed LinUCB)
based on LinUCB (Abbasi-Yadkori et al., 2011).
In our distributed algorithm, agent i uses all samples available for him to maintain a conﬁdence set C t,i ⊆
R d for the parameter θ ∗ .
In each step, he chooses an optimistic estimate (cid:101) θ t,i = argmax θ ∈C t − 1 ,i (max x ∈D (cid:104) x, θ (cid:105) ) and then chooses action X t,i = argmax x ∈D (cid:68) x, (cid:101) θ t,i (cid:69) , which maximizes the reward according to the estimate ˜ θ t,i .
We denote (cid:80) τ x τ x Tτ and (cid:80) τ x τ y τ as
W and U in our algorithm respectively.
We use W t,i and U t,i to denote the sum calculated using available samples for agent i at time step t .
We construct the conﬁdence set C t,i using W t,i and U t,i :
C t,i = 

[Equation]
(cid:32) det [Equation] (cid:1) [Equation]
[Equation]
det( [Equation] − [Equation]
/δ (cid:33) +
[Equation]   , (1) where V t,i = λI + W t,i , ˆ θ t,i = ( λI + W t,i ) − 1 U t,i , (cid:107) x (cid:107) V :=
x T V x .
Essentially, without synchronization, agents in our protocol execute LinUCB separately.
In a synchronization round, agents share all newly acquired samples with each other.
The remaining question is when to synchronize.
Our key observation, which comes from the analysis in Abbasi-Yadkori et al.
(2011), is that the change in the log-determinant of V t is a good indicator of learning progress.
Based on this observation, we only synchronize when agent i ﬁnds that the log-determinant of V t,i has changed signiﬁcantly since the last synchronization.
We designed this synchronization criterion carefully such that, on the one hand, regret is close to the lower bound and on the other hand, the number of synchronization rounds is small.
The full protocol is described below.
Protocol 3: Distributed Linear UCB (DisLinUCB) 1 for t = 1 , · · · , T do for Agent i = 1 , · · · , M do
Compute W t,i = (cid:80) τ x τ x Tτ and V t,i = I + (cid:80) τ x τ y τ using all available samples
Construct the conﬁdence ellipsoid C t,i using W t,i and ˆ θ t,i = V − 1 t,i U t,i .
( x t,i , ˜ θ t,i ) =
arg max ( x,θ ) ∈D t ×C t,i (cid:104) x, θ (cid:105)
Play t i observe reward y t,i . Update
V V t,i / det V last ) · ( t − t last ) > T log( MT ) / ( dM ) then
Start
a synchronization round t last = t , V last =
I + (cid:80) τ x τ y τ using all available examples
We state the following regret and communication bound for this protocol.
The proof is deferred to
Appendix H.
Theorem 4.
The DisLinUCB protocol can achieve a regret of O (cid:16) d √ MT log 2 ( T ) (cid:17) with
O (cid:0) M 1 .
5 d 3 (cid:1) communication cost.
Although the regret bound is still near-optimal, the communication cost has worse dependencies on
M and d compared to that of DELB.
C ONCLUSIONS AND FUTURE WORK
In this work, we propose communication-efﬁcient protocols with near-optimal regret for both multiarmed bandits and linear bandits problem.
For multi-armed bandits, we propose DEMAB protocol, which achieves (cid:112) MKT log( MK ) regret with M log( MK ) communication cost.
The communication cost of DEMAB protocol matches the lower bound we proposed except for a logarithmic factor.
For distributed linear bandits, our DELB protocol and DisLinUCB protocol achieve near-optimal regret with O (( Md + d log log d ) log T ) and O (cid:0) M 1 .
5 d 3 (cid:1) communication cost respectively.
An interesting open question is proving lower bounds of communication cost in order to achieve nearoptimal regret in linear bandits.
From a reduction from multi-armed bandits to linear bandits, Thm.implies Ω( M ) communication lower bound for linear bandits.
However, there is still a gap between this bound and the communication cost of DELB protocol.
We conjecture that close-to-
Md amount of communication is necessary, which is the case in ofﬂine distributed linear regression (Zhang et al., 2013; Braverman et al., 2016) in the context of achieving an optimal risk rate.
ACKNOWLEDGEMENTS
The authors thank Chi Jin, Chicheng Zhang and Nan Jiang for helpful discussions.
This work is supported by National Basic Research Program of China (973 Program) (grant no. 2015CB352502),
NSFC (61573026), BJNSF (L172037) and Beijing Acedemy of Artiﬁcial Intelligence.
A O RGANIZATION
The appendix is organized as follows.
Sections B-E are mainly about distributed multi-armed bandits.
In Section B, we present omitted details for the DEMAB protocol.
In Section C, we analyze the performance of DEMAB and prove Theorem 1.
We also analyze the number of communicated bits in
DEMAB in this section.
In Section D, we discuss how to remove the usage of public randomness in the DEMAB protocol with little increase in communication cost.
In Section E, we give a proof of the communication lower bound for multi-armed bandit.
Sections F-H are mainly about distributed linear bandits.
In Section F, we present the omitted details of DELB protocol.
In Section G, we prove the regret and communication cost bounds of DELB protocol, and also analyze the number of communicated bits in
DELB.
In Section H, we analyze the performance of the DisLinUCB protocol.
Finally, in Section I, we demonstrate that our protocols can be adapted to the P2P communication networks considered in Korda et al.
(2016), which proves our claim in footnote 2 of the main paper.
B D ETAILED D ESCRIPTION OF DEMAB
In this section, we give a detailed description of the DEMAB protocol and some subroutines used in the protocol. Protocol
1 D =
(cid:100) T/MK (cid:101) , C 2 = 1000 , l 0 = (cid:98) log 4 (cid:16)
D C 2 K log( MK ) (cid:17) (cid:99) , m l = (cid:100) 4 l +3 log( MKT · 4 − l ) (cid:101) /
* Stage 1:
Separate Burn-in */ 2 for agent i = 1 , ..., M do
A ( i ) =
Eliminate ( [ K ] , D ) /
* Switching:
Random Allocation */ 4 Generate random numbers [ M ]
) l 0 +1 = { a A | r a = i } /* Stage 2:
Distributed Elimination */ do
ˆ u l ( a (cid:48) )) to server
Server receives ( ∗ j l u ∗ j,l ) from agent j sends u ∗ l = max j u ∗ j,l to every agent for agent i = 1 , ..., M do
Elimination: B ( i ) l
+1 =
(cid:110)
i ∈ B ( i ) l : ˆ u l ( a ) + 2 − l ≥ u ∗ l (cid:111) else /* centralized mode */
Eliminate: Eliminate executes the single-agent elimination algorithm.
In this function, each agent runs the single-agent elimination algorithm for D time steps, then return the remaining arms.
Protocol 5: Eliminate
Input: A set of arms A 1 , time step D .
1 for l = 1 , ... do for a ∈
A l do
Pull arm a for m l = (cid:100) 4 l +3 log(4 − l MKT ) (cid:101) times, denote average reward as ˆ µ l ( a )
If time step D is reached, go to line
Elimination: A l +1 = (cid:8) a ∈
A l : ˆ µ l ( a ) > max k ∈A l ˆ µ l ( k ) − 2 − l (cid:9)
Return
Reallocate:
In Reallocate , the server announces the average number of arms; agents with morethan-average arms then donate surplus arms to the server; the server then distributes the donated arms to the other agents, so that every agent has nearly the same number of arms.
After calling
Reallocate , (cid:126)n l becomes balanced again.
The function contains the following two parts: One running on the server, and the other running on each agent.
Protocol Input: n (1) l ,..., n ( M ) l 1 ¯ n = (cid:98) (cid:80) Mi =1 n ( i ) l /M (cid:99) 2 Send “reallocation”, ¯ n to every player 3 Receive a set of n ( i ) l − ¯ n arms, A (cid:48)
i , from player i if n ( i ) l > ¯ n ; A temp = (cid:83) i
A (cid:48)
i 4 for i = 1 , ..., M do
If n ( i ) l < ¯ n , send n ( i ) l − ¯ n arms in A temp to player i , and remove them from A temp 6 Send the remaining arms in A temp to players 1 ,..., |A temp | (one each)
Protocol
1 if Receive “reallocation”, ¯ n then if n ( i ) l > ¯ n then
Pick a subset of n ( i ) l − ¯ n arms, A (cid:48)
i , from B ( i ) l , and send them to server
B ( i ) l = B ( i ) l \ A (cid:48)
i else
Wait until receiving A (cid:48)
i from server, B ( i ) l
= B ( i ) l ∪ A (cid:48)
i
Centralize: When the number of arms drops below M , the subroutine Centralize is called, in which agents send their local copy of remaining arms, B ( i ) l , to the server, and server receives
B l = (cid:83) i ∈
[ M ] B ( i ) l .
Protocol
1 for agent i = 1 , ..., M do
Send B ( i ) l to server 3 Server receives B ( i ) l from agent i , and calculates B l = (cid:83) i B ( i ) l
Assignment Strategy:
In centralized mode, server assigns arms to agents in the following way.
Let N l = |B
l | .
If M is exactly divisible by N l , for each arm in B l , server asks M/N l separate agents to play it for (cid:100) m
l N l /M (cid:101) times.
If not, we allocate pulls to agents in the following way: Let p l
=
(cid:100) m
l N l /M (cid:101) denote the average pulls each agent needs to perform.
Our assignment starts from the arm with the smallest index a k 1 and agent 1.
For arm a k j and agent i , if agent i has been assigned p l pulls, we turn to agent i + 1 .
If we have ﬁnished allocating m l pulls for arm a k j , we continue designating arm a k j +1 .
The assignment is ﬁnished until all pulls are scheduled to agents.
C P ROOF OF T HEOREM
In this section, we give a full proof of Theorem 1, which bounds the total regret and communication cost of the DEMAB protocol.
In the analysis below, we will use ˜ B l to represent B (1) l ∪ · · · ∪ B ( M ) l (in the distributed mode) or B l (in the centralized mode).
It refers to the set of remaining arms at the start of the l -th phase at stage 2, either held separately by the agents or held by the server.
Suppose that the protocol terminates when l = l (cid:48) .
We also let N M ( a, t ) =
(cid:80) tj =1 (cid:80) Mi
=
I [ a i,j = a ] be the number of times arm a is pulled before time step t .
Without loss of generality, we assume that arm 1 is the best arm, and deﬁne ∆ k := µ (1) − µ ( k ) .
We ﬁrst state a few facts and lemmas.
Fact 1.
We state some facts regarding the execution of the algorithm.
At line 6 of the server’s part in Reallocate , |A temp | < M ;
After Reallocate is called, (cid:68)(cid:12)(cid:12)(cid:12) B (1) l (cid:12)(cid:12)(cid:12) , · · · , (cid:12)(cid:12)(cid:12) B ( M ) l (cid:12)(cid:12)(cid:12)(cid:69) is balanced; 3.
For any player i , the number of completed phases in stage 1 is at least l 0 =
(cid:106) log 4 (cid:16) D C 2 K log MK (cid:17)(cid:107) , and
at most l D =
(cid:108) log 4 (cid:16) D C (cid:48)
2 log MK (cid:17)(cid:109) .
The number of phases at stage 2 is less than L =
+ 5 log( MK )
=
O (log( MK )) .
Proof.
Let S +
= { i : n ( i ) l ≥ ¯ n } and S − =
{ i : n ( i ) l < ¯ n } .
At line 3 of server’s reallocate code, server receives (cid:80) i ∈ S
+ (cid:16) n ( i ) l − ¯ n (cid:17) arms.
At line 5, (cid:80) i ∈ S − (¯ n − n ( i ) l ) arms are removed.
So at line 6, |A temp | = (cid:88) i ∈ S
+ ( n ( i ) l − ¯ n ) − (cid:88) i ∈ S − (¯ n − n ( i ) l ) = (cid:88) i ∈
[ M ] n ( i ) l − M (cid:98) (cid:88) i ∈
[ M ] n ( i ) l /M (cid:99) <
M.
Let n ( i ) l
= (cid:12)(cid:12)(cid:12) B ( i ) l (cid:12)(cid:12)(cid:12) .
If max i n ( i ) l ≤ 2 min i n ( i ) l , the reallocation procedure will do nothing, and (cid:104) n (1) l , ..., n ( M ) l (cid:105) is by deﬁnition balanced.
If max i n ( i ) l > 2 min i n ( i ) l , then at the end of the reallocation procedure, every player has a new set of arms B ( i ) l such that ¯ n ≤ (cid:12)(cid:12)(cid:12) B ( i ) l (cid:12)(cid:12)(cid:12) ≤ ¯ n + 1 .
This implies that the number of arms is balanced, since when reallocation is called, (cid:80) i n ( i ) l ≥ M .
The length of the l -th phase at stage 1 is at most K (cid:100) 4 l +3 log(4 − l MKT ) (cid:101) .
After l phases, the number of timesteps is at most
[Equation]
Setting C 2 = 1000 and l 0 = (cid:98) log 4 (cid:16)
[Equation]
(cid:17) (cid:99) , one can verify that
[Equation]
Since the total number of timesteps in stage 1 is D , we know that the number of completed phases in stage 1 is at least l 0 .
On the other hand, in l phases, the number of timesteps is at least
[Equation]
By setting C (cid:48) 2 = 512 and l D = (cid:108) log 4 (cid:16)
D C (cid:48) 2 log MK (cid:17)(cid:109) , one can show that 4 l D +3 log(4 − l D MKT ) >
D. Therefore, the number of of completed phases in stage 1 is at most l D .
Suppose that stage 2 has L = 4 + 5 log( MK ) phases, then the total number of phases during stage 1 and stage 2 is at least L + l 0 .
In phase L + l 0 , at least 4 l 0 + L +3 log (cid:0) 4 − ( l 0 + L ) MKT (cid:1) pulls are made.
With some elementary calculation, one can show that the number of arms pulled in phase
L + l 0 is MT ·
[Equation]
, which is even greater than the total number of timesteps MT .
That is to say, the number of phases in stage 2 is less than L .
Via a direct application of Hoeffding’s inequality and union bound, we have the following lemma.
Lemma C.1.
Denote the average rewards computed by agent i in phase l of stage 1 be ˆ µ i,l ( · ) .
At stage 1 , we denote the following even as Λ l : in phase l at stage 1 , for any agent i , for any arm a ∈ A ( i ) l , | ˆ µ i,l ( a ) − µ ( a ) | ≤ 2 − l − 1 .
Then Pr [Λ l ] ≥ 1 − 2 · 4 l / ( MKT ) .
Proof.
For any agent i , any phase l ≤ l D , denote the empirical mean for arm a in phase l by ˆ u i,l ( a ) .
By a direct application of Hoeffding’s bound and union bound, we can observe that for any ﬁxed i, l, a , by Hoeffding’s bound,
[Equation]
A union bound for all arms a ∈ A ( i ) l and all agents i ∈
[ M ] proves the lemma.
Lemma C.2.
For phase l in stage 1, Λ 1 ∧ · · · ∧ Λ l implies the following:
1. 1 ∈
A ( i ) l +1 ; 2.
∀ i ∈
[ M ] , ∀ a ∈ A ( i ) l +1 , µ ( a ) ≥ µ (1) − 2 − l +1 ; 3.
Recall that N M ( k, t ) =
(cid:80) tj
=
(cid:80) Mi
=
1 I [ a i,j = k ] is the number of times arm k is pulled for all agents before time step t .
If l i = (cid:100) log 2 ∆ − 1 i (cid:101) + 1 ≤ l , then N M ( i, D ) ≤ [Equation]
[Equation] log [Equation] ∆
[Equation]
/ ∆ 2 i ;
In particular, let l D be the maximum number of phases for all agents at stage 1.
Λ 1 ∧ · · · ∧ Λ l D also implies the following:
∃ i ∈
[ M ] , 1 ∈ B ( i ) l 0 +1 ; 5. ∀ i ∈
[ M ] , ∀ a ∈ B ( i ) l 0 +1 , µ ( a ) ≥ µ (1) − 2 − l
+1 ;
Proof.
These results are direct implications of lemma C.1.
1. Notice that if Λ l happens [Equation]
[Equation]
Thus, arm 1 will never be eliminated throughout the ﬁrst l phases.
Λ l implies that for any a ∈ A ( i ) l +1 ,
[Equation]
which means
[Equation]
From statement 2 we know that Λ 1 ∧ · · · ∧ Λ l implies that arm i will be eliminated after l i phases, which is less than l .
Therefore, before eliminated, the total number of times arm i is pulled at stageby any agent is at most
[Equation]
for some absolute constant C 5 .
Multiplying by M proves the assertion.
Let A ( i ) denote the remaining arms at the end of stage 1 for agent i .
From statement 1 we know that Λ 1 ∧ · · · ∧ Λ l D implies 1 ∈ A ( i ) for any i ∈ [ M ] .
So at line 5 in protocol 1, arm 1 will be assign to agent r a .
Let A ( i ) denote the remaining arms at the end of stage 1 for agent i .
From statement 2 we know that Λ 1 ∧ · · · ∧ Λ l D implies µ ( a ) ≥ µ (1) − 2 − l ( i ) +1 for any arm in (cid:83)
i ∈
[ M ] A ( i ) .
Here l ( i ) denotes the number of phases for agent i at stage 1.
Since B l 0 +1 =
(cid:83)
i ∈
[ M ] B ( i ) l 0 +1 is a subset of (cid:83) i ∈
[ M ] A ( i ) and
Lemma C.3.
For l > l 0 , denote the following event by Ω l : at stage 2, for any a ∈ B l ,
[Equation]
Then Pr [Ω l ] ≥ 1 − 2 · 4 l / ( M 2 KT ) .
Proof.
Recall that for any l > l 0 , ˆ u l ( a ) is the average of m l independent samples of the reward from arm a .
Therefore, for ﬁxed l and a , by Hoeffding’s inequality,
[Equation]
A union bound for all a ∈ B l proves this lemma.
Lemma C.4.
Recall that ∆ i = µ (1) − µ ( i ) .
If ∆ i > 0 , let l i = (cid:100) log 2 ∆ − 1 i (cid:101)
+ 1 .
Let Λ denote Λ 1 ∧ · · · ∧ Λ l D .
Then at stage 2, Λ ∧ Ω l 0 ∧ · · · ∧ Ω l i implies the following:
[Equation]
where C 4 is a universal constant, and (cid:80) a ∈ [ K ] η ( a ) ≤ M log M .
Proof.
Suppose that | ˜ B l | > M .
By Fact 1, at phase l ≥ l 0 + 1 in stage 2, the sequence (cid:68)(cid:12)(cid:12)(cid:12) B (1) l (cid:12)(cid:12)(cid:12) , · · · , (cid:12)(cid:12)(cid:12) B ( M ) l (cid:12)(cid:12)(cid:12)(cid:69) is balanced.
Therefore, during phase l at stage 2, the number of times an arm in ˜ B l is pulled is at most 2 m l .
If | ˜ B l | ≤ M , an arm in ˜ B l is also pulled at most 2 m l times, unless | ˜ B l |· m l < M .
In that case, a phase only lasts for 1 timestep.
This is possible only when m l < M , which requires l < (cid:98) log 4 M (cid:99) < log M .
We denote the number of times a is pulled in such phases by η ( a ) .
On the other hand, let l i =
(cid:100) log 2 1∆ i (cid:101)
+ 1 .
If l i ≤ l 0 , Λ 2 implies that i / ∈ ˜ B l 0 +1 .
In that case, the number of times arm i is pulled after timestep D is 0 .
Assume Ω l 0 +1 ∧ · · · ∧ Ω l i holds and i ∈ ˜
B l i .
Then
[Equation]
Therefore, Ω l 0 +1 ∧ · · · ∧ Ω l i implies that i / ∈ ˜
B
l i +1 .
In this case, number of times arm i is pulled is
[Equation]
i [Equation] It is not hard to see that (cid:80) a ∈ [ K ] η ( a ) ≤ M log M .
Lemma C.5.
Let l ≥
l
+ 1 , and n l = (cid:12) (cid:12)(cid:12) ˜
B l (cid:12) (cid:12)(cid:12) .
With probability 1 − 2 LKδ , either n l < 21 M log 1 δ or no reallocation is performed before the start of the l -th phase at stage 2.
Proof.
Let Y i,l = I (cid:104) i ∈
˜ B l (cid:105) , X i,j = I
[ r i = j ] , i ∈
[ K ] , j ∈ [ M ] .
Observe that X i,j and Y i (cid:48) ,l are independent.
This is because the elimination process between ˜ B l and ˜ B l +1 uses exactly m l independent samples for each arm; therefore, the conditional probability for remaining has no dependence on which player an arm is assigned to.
Let (cid:126)Y denote { Y i,l , ∀ i, l } .
Since
[Equation]
by Chernoff’s inequality,
[Equation]
Consequently,
Pr
[Equation]
˜ B l (cid:12)(cid:12)(cid:12) 3 M (cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) n l > 21 M log 1 δ 
 ≥ 1 − E (cid:126)Y (cid:20) exp (cid:16) − n l 21 M (cid:17) + exp (cid:16) −
n l 18 M (cid:17)(cid:12)(cid:12)(cid:12) n l > 21 M log 1 δ (cid:21) ≥ 1 − 2 δ.
Note that 23 M (cid:12)(cid:12)(cid:12)
˜ B l (cid:12)(cid:12)(cid:12) < (cid:12)(cid:12)(cid:12) B ( j ) l (cid:12)(cid:12)(cid:12) < 43 M (cid:12)(cid:12)(cid:12)
˜ B l (cid:12)(cid:12)(cid:12) for all j ∈ [ K ] implies that (cid:68)(cid:12)(cid:12)(cid:12) B (1) l (cid:12)(cid:12)(cid:12) , · · · , (cid:12)(cid:12)(cid:12) B ( M ) l (cid:12)(cid:12)(cid:12)(cid:69) is balanced.
Therefore, if n l > 21 M log 1 δ , with probability 1 − 2( l − l 0 )
Kδ , no reallocation will be performed before the l -th phase at stage 2.
Now we are ready to prove Theorem 1.
Theorem 1.
The DEMAB protocol incurs O (cid:16)(cid:112) MTK log( MK ))
(cid:17) regret, O (cid:0)
M log MKδ (cid:1) communication cost with probability 1 − δ , and O ( M log( MK ))
communication cost in expectation.
Proof.
Regret: Let l i =
(cid:100) log 2 ∆ − 1 i (cid:101)
+ 1 .
By Lemma C.2, when Λ 1 ∧ · · · ∧ Λ l i holds,
[Equation]
Therefore, the probability that this does not hold is at most
[Equation]
Thus for any arm i ,
[Equation]
Similarly, if l i ≥ l 0 , when Λ ∧ Ω l 0 +1 ∧ · · · ∧ Ω l i holds,
[Equation]
The probability that this does not hold is at most
Pr
[Equation]
l i +
[Equation]
p
[Equation]
p
[Equation]
Therefore
[Equation]
It follows that
[Equation]
where in the last line we choose (cid:15) 1 = (cid:113) K log( K ) T , and C 6 is some absolute constant.
Similarly,
K (cid:88)
[Equation]
where in the last line we chose (cid:15) 1 = (cid:113) K log( K ) MT .
By our assumption, M log M ≤ (cid:112) MTK log( K ) .
Therefore by the deﬁnition of total regret,
[Equation]
Communication: Total communication in stage 1 is 0 .
We ﬁrst consider the worst case communication cost in stage 2.
Note that during a phase (either in distributed mode or in centralized mode), the communication cost is O ( M ) if reallocation is not performed.
The cost for reallocation at the start of phase l is O (min { n l , n l (cid:48) − n l } ) , where l (cid:48) is the last phase where reallocation is performed.
Summing this over all phases (at most L =
O (log( MK )) ) in stage 2, we conclude that total communication cost for reallocation is O ( ML + K ) .
Deﬁne l 1 to be the ﬁrst phase such that reallocation is performed.
Then, our argument above shows that communication cost is O ( ML + n l 1 ) .
If n l 1 > 21 M log 1 δ , the event in lemma C.5 will be violated for some l .
The probability for that is at most 2 L 2 Kδ .
By resetting δ , we can show that with probability 1 − δ , n l 1 < 21 M log (cid:16) 2 L 2 Kδ (cid:17) .
Therefore, with probability 1 − δ , total communication cost is
[Equation]
In particular, by choosing δ = 1 /K , we can show that expected communication cost is
O ( M log( MK )) .
Theorem 5.
When D = l 0 = 0 in DEMAB, the protocol incurs near-optimal instance-dependent regret O ( (cid:80) k :
∆ k > 0 ∆ − 1 k log T + M log M ) .
With probability 1 − δ the communication cost is
O ( M log( T/δ )) .
The expected communication cost is O
( M log T ) .
Proof.
Regret:
In the case D = 0 , it can be seen from the proof of Theorem 1 that
[Equation]
where (cid:80) i ∈
[ K ] η ( i ) ≤ M log( M ) .
It follows that total expected regret is
[Equation]
Communication:
By the proof of Theorem 1, the worst-case communication cost of this protocol is O
( M log T + K ) .
This is because we need O ( M ) communication at the end of each phase to perform elimination, and at most a total number of O ( K ) additional communication among all phases to perform reallocation.
Let l ∗ be the last complete phase such that n l ∗ +1 > 21 M log( KL (cid:48) /δ ) .
By lemma C.5, no reallocation is needed before phase l ∗ + 1 with probability 1 − δ , so the total communication before phase l ∗ +is O ( ML (cid:48) ) .
From the beginning of phase l ∗ + 1 , the total communication in the following phases is at most
O ( ML (cid:48)
+ M log( KL (cid:48) /δ )) .
Therefore, with probability 1 − δ , the communication cost is O
( M log T + M log( KL (cid:48) /δ )) =
O ( M log( T/δ )) .
Let δ = 1 /K , the expected communication is O
( M log T + ( ML (cid:48)
+ K ) /K )
=
O ( M log T ) .
Finite precision: We now show that only O (log T ) bits are needed for each number sent in
DEMAB.
The integers sent in DEMAB are numbers in { 0 , · · · , K } .
Therefore O (log T ) bits are sufﬁcient for each integer 11 .
In the DEMAB protocol, the only real numbers that are transmitted are the average reward in a phase.
In our proof, it is only required that the average of m l samples is4 l
[Equation]
2 log
[Equation]
[Equation] l ) -subgaussian.
In fact, the average of m l samples is
[Equation]
[Equation]
Therefore, we can use randomized rounding for the average reward with precision (cid:15) = 1 4 TM .
Then, the rounding error for each real number has zero mean, and is
[Equation]
) -subgaussian.
When computing the average of m l samples at phase l , at most M rounding error terms will contribute to it, whose sum is
[Equation]
subgaussian.
Therefore, the concentration inequality in lemma
C.3 and consequently our main theorem still holds.
Apparently log 2 1 (cid:15)
=
O (log( MT ))
=
O (log T ) .
Therefore, expected number of communicated bits is O ( M log( MK ) log T ) .
D R EMOVING P UBLIC R ANDOMNESS
The DEMAB protocol makes use of a public random number generator.
It can be viewed as a sequence of uniformly random bits written on a public blackboard that every agent can read, and reading from this sequence does not require communication.
In practice, this can be approximated by using a pseudorandom number generator with a truly random seed.
In this case, regardless of the amount of public random number used, the communication cost is O ( M ) , which is the cost of broadcasting a short random seed.
However, we can also totally remove the usage of public random numbers.
The role of shared randomness in communication complexity has already been investigated.
It is known that shared randomness can be efﬁciently replaced by private randomness and additional communication, as stated by the Newman’s Theorem Newman (1991).
In our case, the argument is slightly different: we are considering an online learning task instead of function evaluation.
Also, in DEMAB, the communication cost itself depends on the public random bits.
In particular, we show the following theorem.
Theorem 6.
There exists a protocol for distributed MAB that does not use public randomness with expected regret O ( √ MKT log T ) .
It has communication cost bounded by O ( M log( MK ) + K ) (worst case), and expected communication cost O
( M log( MK )) .
Proof.
We make the following modiﬁcations to the original DEMAB protocol.
Instead of using a public random bit string s 12 , we predetermine B strings s 1 ,..., s B (which can be hardcoded in advance), and randomly choose from them.
That is, the server will generate a random number uniformly distributed in [ B ] , and broadcast it to everyone.
The communication cost of doing so will be M (cid:100) log 2 B (cid:101) .
We now analyze how the choice of s 1 , ..., s B affects the performance of the protocol.
In terms of regret bound, observe that for any random string s , the expected regret of any bandits instance X is always O ( √ MKT log T ) .
Therefore, regret bound will not be affected when public randomness is removed.
Now deﬁne f
( X, s ) to be the expected communication cost of the DEMAB protocol using the public random string s and interacting with the multi-armed bandit instance X .
Our analysis for DEMAB tells us that ∃ c 1 , ∀ X ,
E r
[ f ( X, s )] ≤ c 1 M log( MK ) .
Therefore, if we draw i.i.d.
uniform bit strings s 1 , ..., s B ,
[Equation]
i = [Equation] f ( X [Equation]
s
[Equation]
1 M log
[Equation]
MK ) .
We say that a set of bit strings { s 1 , ..., s B } is bad for a bandit X if
[Equation]
i
[Equation]
f
[Equation]
M log( MK ) .
We know that there exists c 2 > 0 such that 0 ≤ f ( X, s i ) ≤ c 2 ( M log( MK ) + K ) .
Therefore, by
Hoeffding’s inequality,
[Equation]
In other words, for ﬁxed X , the probability that we will draw a bad set { s 1 , ..., s B } is exponentially small.
Therefore, for a family of bandits with size Q , the probability for drawing a set of s 1 , ..., s B that is bad for some bandit is at most Q · exp (cid:110) −
[Equation]
If we can show that this quantity is smaller than 1 , it would follow that there exists { s 1 , · · · , s B } such that it is not bad for any bandit in the family.
Now, we consider the following family X of bandits.
For each arm, the expected reward could be a/ ∆ , where a ∈ (cid:8) 0 , 1 , ..., (cid:98) ∆ − 1 (cid:99) (cid:9) .
The reward distribution is Bernoulli.
The size of this family is
Q ≤ (cid:0) ∆ − 1 + 1 (cid:1) K .
Now, consider any other bandit X 1 .
Without loss of generality, we can assume that X 1 is a Bernoulli bandit, and that the expectation of each arm is in [1 / 4 , 3 / 4] 13 .
Apparently we can ﬁnd a bandit X 2 ∈ X such that their expected rewards are ∆ -close in (cid:107) · (cid:107) ∞ .
As a result, the KLdivergence of each arm’s reward in X 1 and X 2 is O (∆ 2 ) .
Let H ( X ) =
{ a 1 , 1 , r 1 , 1 , ..., a T,M , r T,M } be the random history of the DEMAB interacting with the bandit instance X .
Since communication cost is determined given H ( X ) , ∀ s ,
[Equation]
With ∆ =
K − 1 ( MT ) − 0 .
5 , the right-hand-side is O
( M log( MK )) .
Therefore, it sufﬁces to consider the bandit family X .
Therefore, we only need to guarantee that
[Equation]
, where C (cid:48) is a universal constant.
This can be met by setting B = (cid:100) 2 C (cid:48)
K 2 log( MKT ) (cid:101) .
In this case, we can guarantee that there exists a set of bit strings { s 1 , ..., s B } , such that for any bandit instance X , when choosing s randomly from this set, the expectation of f ( X, s ) is O
( M log( MK )) .
The additional communication overhead for generating the random string (in bits) is
[Equation]
Therefore, under our usual assumption that T > max { M, K } , the number of total communicated bits is bounded by O ( M log K + M log log T ) .
In our formulation, we may view log T bits as one packet.
Therefore additional communication cost is O ( M ) .
It follows that total expected communication cost is O ( M log( MK )) .
E P ROOF FOR T HEOREM
Proof.
In order to prove the theorem, we show that for any protocol with communication cost less that M/c ( c is a constant), the regret is at least Ω( M √ KT ) .
First, we list two lemmas that will be used in our proof.
Lemma E.1.
(Lattimore & Szepesvári, 2019, Theorem 9.1)
For K -armed bandits, there is an algorithm with expected regret
[Equation]
Lemma E.2.
(Lattimore & Szepesvári, 2019, Theorem 15.2)
For K -armed bandits, for any algorithm, there exists an instance such that
[Equation]
The original lower bound is proved for Gaussian bandits, which doesn’t ﬁt exactly in our setting.
we modiﬁed the proof to work for Bernoulli bandits, which results in a different constant.
We now prove the theorem’s statement via a reduction from single agent bandit to multi-agent bandit.
That is, we map communication protocols to single-agent algorithms in the following way.
For simplicity, we consider protocols as M blocks of code.
In agent i ’s block, each line could be a local computation, sending a message, or waiting for a message to receive.
Consider a communication protocol with communication cost B ( M ) .
We denote X
i ( i ∈ [ M ] ) to be the number of integers or real numbers that agent i sends or receives throughout a run.
X i is a random variable.
Since expected communication cost is less than M/c ,
[Equation]
Denote S as the set of M/ 2 agents with smaller E X i .
The expected communication cost of any i ∈ S is at most 2 /c .
For any i ∈ S , P ( X i ≥ 1) ≤ E X i ≤ 2 /c .
That is, for any of these agents, the probability of either speaking to or hearing from someone is less than 2 /c .
Suppose that agent j is such an agent.
Then, we can map the communication protocol to a single-agent algorithm by simulating agent j .
The simulation is as follows.
Interacting with single agent bandit with time T , we run the code for agent i in the protocol.
When no communication is needed, we may proceed to the next line of agent i ’s code.
When this line of code sends a message or waits for a message, we terminate the code.
In the rest of the timesteps, we run a single-agent optimal algorithm (the one used to realize lemma E.1).
Then, if agent j ’s code has δ probability of involving in communication, and if agent j ’s regret
REG j ( T ) ≤ A , via this reduction, we can obtain an algorithm for single-agent MAB with expected regret
[Equation]
By lemma 2 , REG ( T ) cannot have a regret upper bound better than (cid:112) T ( K − 1) / 75 .
Therefore
[Equation]
If 38 δ < 1 / 75 , we can show that A = Ω (cid:16) √ KT (cid:17) .
In our case, c = 3000 will sufﬁce.
Since we can show this for any agent in S , we can show that total regret is Ω (cid:16) M √ KT (cid:17) .
F O MITTED D ETAILS OF DELB
Assignment Strategy: At line 4, we assign pulls to agents in the following way.
Let p l
=
(cid:100) (cid:80) x
m l ( x ) /M
(cid:101) denote the average pulls each agent needs to perform.
Our assignment starts from the arm with the largest m l ( x ) and agent 1.
For arm x k and agent i , if agent i has been assigned with p l pulls, we turn to agent i + 1 .
If we have ﬁnished designating m l ( x k ) pulls for arm x k , we continue designating arm x k +1 .
The assignment is ﬁnished until all pulls are scheduled to agents.
Observe that at the start of each phase, each agent has the same A l as the server.
Therefore, at linethey obtain the same π l , with the support size at most d log log d .
In that case, the server only needs to send a index ( O (1) communication cost) over ξ = 48 d log log d arms, instead of a vector ( Ω( d ) communication cost), to identify an arm x ∈ Supp( π l ) .
G P ROOF OF T HEOREM
First, we consider some properties of the elimination based protocol for linear bandits.
Fact 2.
Let T l denote the total number of pulls in the l -th phase, then
[Equation]
where ξ = 48 d log log d .
Proof.
For an arm x in the core set Supp( π l ) , the DELB protocol pulls it (cid:100) C 1 4 l d 2 π l ( x ) log MT (cid:101) times.
Thus the total number of pulls in phase l satisﬁes
[Equation]
and (cid:88) a ∈ Supp( π l )
[Equation]
a ∈ Supp
[Equation]
l
[Equation]
[Equation]
MT
[Equation]
Lemma G.1.
In phase l , with probability 1 − 1 /TM , for any x ∈ D ,
[Equation]
Proof.
First, we construct an (cid:15) l -covering of D with (cid:15) l = 2 − l − 2 .
Denote the center of the covering as
X = { ¯ x 1 , ..., ¯ x Q } .
Here Q satisﬁes Q ≤ 3 d 2 d ( l +2) .
Assume that ˆ θ is calculated from linear regression on x 1 , ... , x t (cid:48) .
For ﬁxed x ∈ D , it is known that (cid:104)
ˆ θ − θ ∗ , x (cid:105) is subgaussian with variance proxy
[Equation]
Therefore with probability 1 − 2 δ ,
[Equation]
Suppose n l pulls are made in phase l .
In our case,
[Equation]
Therefore with probability 1 − 2 δ ,
[Equation]
Choose δ = 1 / (2 TMQ ) .
It can be shown that with C 1 = 600 ,
[Equation]
Therefore with probability 1 − 1 / ( TM )
[Equation]
for all x ∈ X
[Equation]
Now
[Equation]
consider an arbitrary x ∈ D .
There exists ¯ x ∈ X such that (cid:107) x − ¯ x (cid:107) ≤ 2 − l − 2 .
Therefore with probability 1 − 1 /TM , for any x ∈ D ,
[Equation]
· (cid:107) x − ¯ x (cid:107) ≤ 2 − l .
Lemma G.2.
Let x ∗ =
arg max x ∈ D (cid:104) θ ∗ , x (cid:105) be the optimal arm.
Then with probability 1 − log( MT ) / ( TM ) , x ∗ will not be eliminated until the protocol terminates.
Proof.
If x ∗ is eliminated at the end of round l , one of the following must happen: either (1) (cid:12)(cid:12)(cid:12) (cid:104)
ˆ θ − θ ∗ , x ∗ (cid:105) (cid:12)(cid:12)(cid:12) > 2 − l ; or (2) there exists x (cid:54) = x ∗ , (cid:12)(cid:12)(cid:12) (cid:104)
ˆ θ − θ ∗ , x (cid:105) (cid:12)(cid:12)(cid:12) > 2 − l .
Therefore the probability for x ∗ to be eliminated at a particular round is less than 1 − 1 / ( TM ) .
The total number of phases is at most log MT .
Hence a union bound proves the proposition.
Lemma G.3.
Suppose δ = 2 log( TM ) /TM , and ∆ x denotes the suboptimality gap of x , i.e.
∆ x = (cid:104) θ ∗ , x ∗ − x (cid:105) .
For suboptimal x ∈ D , deﬁne l x = inf { l : 8 · 2 − l ≤ ∆ x } .
Then with probability 1 − δ , for any suboptimal x , x (cid:54)∈ A l x .
Proof.
First, let us only consider the case where x ∗ is not eliminated.
That is,
Pr [ ∃ x ∈ D : x ∈
A l x ] ≤ Pr [ x ∗ is eliminated ] + Pr [
∃ x : x ∈ A l x − 1 , x ∈
A l x | x ∗ ∈
A l a ] .
Note that conditioned on x ∗ ∈
A l x , { x ∈ A l x − 1 ∧ x ∈
A l x } implies that at phase l x − 1 , either (cid:12)(cid:12)(cid:12) (cid:104)
ˆ θ − θ ∗ , x (cid:105) (cid:12)(cid:12)(cid:12) > 2 − l x +1 or (cid:12)(cid:12)(cid:12) (cid:104)
ˆ θ − θ ∗ , x ∗ (cid:105) (cid:12)(cid:12)(cid:12) > 2 − l x +1 .
Therefore the probability that there exists such x is less than log( TM ) /TM .
Hence, with probability 1 − 2 log( TM ) /TM , x will be eliminated before phase l x .
We are now ready to prove our main result for DELB.
Theorem 3.
DELB protocol has expected regret O (cid:0) d √ TM log T (cid:1) , and has communication cost
O (( Md + d log log d ) log T ) .
Proof.
Regret: We note that at the start of round l , the remaining arms have suboptimality gap at most 8 · 2 − l .
Suppose that the last ﬁnished phase is L .
Therefore total regret is
[Equation]
Apparently C 1 4 L d 2 log TM ≤ TM .
Therefore
[Equation]
Under our usual assumption that T > M , this can be simpliﬁed to O ( d √ TM log T ) .
Here C 3 and
C 7 are some universal constants.
Communication Cost: Let p l
= (cid:80) x
m l ( x ) /M
denote the average pulls each agent needs to perform.
Observe that for each arm, the number of agents that it is assigned to is at most 1 + (cid:100) m l ( x ) /p
l (cid:101) agents.
Therefore, total communication for scheduling is at most
[Equation]
Similarly, total communication for reporting averages is the same.
The cost for sending ˆ θ is Md .
Hence, communication cost per phase is O ( Md + d log log d ) .
On the other hand, total number of phases is apparently O (log TM ) .
Hence total communication is
[Equation]
Under the assumption that T > M , this can be simpliﬁed to O
( Md + d log log d ) log T ) .
Finite precision:
We now discuss the number of bits needed in DELB.
The integers sent in DELB are less than max { T, ξ } (recall that ξ = 48 d log log d ).
Therefore, every integer can be encoded with
O (log( dT )) bits.
It remains to be proven that transmitting each real number with logarithmic bits is sufﬁcient.
In the DELB protocol, two types of real numbers are transmitted: average of rewards, and entries of ˆ θ .
To transmit real numbers with ﬁnite number of bits, we make the following modiﬁcations to the original protocol: 1.
when transmitting average rewards at line 5 , use randomized rounding with precision (cid:15) 1 = 1 M 2 T ; 2.
after computing ˜ θ
= V − 1 l
X at line 7, let ˆ θ be the entry-wise rounded vector of ˜ θ with (cid:15) 2 = 1 MTd .
It can be seen that we only need to prove that after the modiﬁcations, lemma G.1 still holds.
In each phase, originally µ ( x ) is 1 m l ( x ) -subgaussian, but is only required to be 2 m l ( x ) -subgaussian for (2) to hold.
After the modiﬁcation, the contribution of rounding error to a µ ( x ) comes from at most M independent terms, and is therefore subgaussian with variance proxy 1 MT ≤ 1 m l ( x ) .
Therefore, after the modiﬁcations, the computed µ ( x ) is 2 m l ( x ) -subgaussian; hence, (2) holds.
It follows that with probability 1 − 1 / ( TM ) , for all x ∈ X , (cid:12)(cid:12) (cid:12) (cid:104)
˜
θ − θ ∗ , x (cid:105) (cid:12)(cid:12) (cid:12) ≤ 2 − l − 2 .
Therefore, for any a ∈ D , (cid:12)(cid:12)(cid:12)
[Equation]
− l − 1 .
Combined with the fact that for any x ∈ D ,
[Equation]
we can prove that with probability 1 − 1 / ( TM ) , for all x ∈ X ,
[Equation]
Therefore, after the modiﬁcations, the regret of the protocol is still O (cid:0) d √ TM log T (cid:1) .
The amount of communicated bits is thus
[Equation]
H P ROOF OF T HEOREM
First, let us restate the DisLinUCB protocol in full detail.
1 D =
T log dM )
2 for Agent i = 1 , ..., M do
Set W syn, i 0 , U syn,
i
0 , W new, i
U new,i
0 , t last
0 , V last t = 1 , ..., do i do
Let us then state several lemmas that will be useful in our proof.
Lemma H.1.
For any δ > 0 , with probability 1 − Mδ , θ ∗ always lies in the constructed C t,i for all t and all i .
Proof.
Using Theorem 2 in Abbasi-Yadkori et al.
and
union bound over all agents, we can prove the lemma.
For any positive deﬁnite matrix V 0 ∈ R d × d , any vector x ∈ R d , deﬁne the norm of x w.r.t.
V 0 as (cid:107) x (cid:107) V 0 :
= (cid:112) x T V 0 x .
Lemma H.2.
(Lemma 11 in Abbasi-Yadkori et al.
Let { X t } ∞ t =1 be a sequence in R d , V is a d × d positive deﬁnite matrix and deﬁne V t
= V + (cid:80) t s =
1 X s X (cid:62) s .
Then we have that
[Equation]
Further, if (cid:107) X t (cid:107) 2 ≤ L for all t , then n (cid:88) t = min (cid:110) 1 , (cid:107) X t (cid:107) 2 V − 1 t − 1 (cid:111) ≤ 2 (cid:0) log det (cid:0) V n (cid:1) − log det V (cid:1) ≤ 2 (cid:0)
d log (cid:0)(cid:0) trace( V ) + nL 2 (cid:1) /d (cid:1) − log det V (cid:1) .
Using Lemma H.1, we can bound single step pseudo-regret r t,i .
Lemma H.3.
With probability 1 − Mδ , single step pseudo-regret r t,i = (cid:104) θ ∗ , x ∗ − x t,i (cid:105) is bounded by r t,i ≤(cid:32)(cid:115) 2 log
[Equation] [Equation]
Proof.
Assuming θ ∗ ∈ C t,i ,
[Equation]
Now we are ready to prove Theorem 4.
Theorem 4.
DisLinUCB protocol achieves a regret of O (cid:16) d √ MT log 2 ( T ) (cid:17) with O (cid:0) M 1 .
5 d 3 (cid:1) communication cost.
Proof.
Regret: Set δ = 1 / ( M 2 T ) , the expected regret caused by the failure of Eq.
(3) is at most
MT · 1 / ( MT )
=
O (1) , thus we mainly consider the case where Eq.
(3) holds.
In our protocol, there will be a number of epochs divided by communication rounds.
We denote
V last in epoch p as V p .
Suppose that there are P epochs, then V P will be the matrix with all samples included.
Observe that det
[Equation]
≤
[Equation]
λ
[Equation]
d .
Therefore
[Equation]
Let R :=
(cid:100) d log (cid:0) 1 + MTλd (cid:1) (cid:101) .
It follows that for all but R epochs, we have
[Equation]
We call those satisfying Eq.
4 good epochs.
In these epochs, we can use the argument for theorem 4 in Abbasi-Yadkori et al.
First, we imagine the MT pulls are all made by one agent in a round-robin fashion (i.e. he takes x 1 , 1 , x 1 , 2 ,..., x 1 ,M , x 2 , 1 ,..., x T,M ).
We use ˜ V t,i = λI + (cid:80) { ( p,q ):(
p<t ) ∨ ( p = t
∧ q<i ) } x p,q x Tp,q to denote the V t,i this imaginary agent calculates when he gets to x t,i .
If x t,i is in one of those good epochs(say the j -th epoch), then we can see that
[Equation]
Therefore
[Equation]
[Table]
[Equation]
[Table]
(cid:32)(cid:114)
[Equation]
We can then use the argument for the single agent regret bound and prove regret in these good epochs.
We denote regret in all good epochs as REG good .
Suppose B p means the set of ( t, i ) pairs that belong to epoch p , and P good means the set of good epochs, using lemma H.2, we have
[Equation]
Now we focus on epochs that are not good.
For each bad epoch, suppose at the start of the epoch we have V last .
Suppose that the epoch starts from time step t 0 , and the length of the epoch is n .
Then agent i proceeds as V t 0 ,i , ..., V t 0 + n,i .
Our argument above tells us that regret in this epoch satisﬁes
REG ≤ 2 (cid:16)(cid:112) d log
[Equation]
[Equation] n (cid:88) t =
[Equation]
min
[Equation]
=
[Equation]
(cid:114)
[Equation]
det V last .
Now, for all but 1 agent, n log det V t 0+ n,i det V last < D .
Therefore we can show that
REG ( n ) ≤ O
[Equation]
Since det( V P ) ≤ ( λ + MT/d ) d , we know that the number of such epochs are rare.
(Less than
R = O
( d log MT ) ).
Therefore the second part of the regret is
[Equation]
If we choose D = (cid:16) T log MT dM (cid:17) , then REG ( T ) =
O (cid:16) d √ MT log 2 ( MT ) (cid:17) .
Since T > M , we have
[Equation]
Communication: Let α = (cid:0) DTR (cid:1) 0 .
Apparently there could be at most (cid:100) T/α (cid:101) such epochs that contains more than α time steps.
If the j -th epoch contains less than α time steps, log (cid:16) det V j +1 det V j (cid:17) > Dα .
Since
P
[Equation]
There could be at most (cid:100) RD/α (cid:101)
= (cid:100) RαD (cid:101) epochs with less than α time steps.
Therefore, the total number of epochs is at most
[Equation]
With our choice of D , the right-hand-side is O (cid:0) M 0 .
5 d (cid:1) .
Communication is only required at the end of each epoch, when each agent sends O ( d 2 ) numbers to the server, and then downloads O ( d 2 ) numbers.
Therefore, in each epoch, communication cost is O
( Md 2 ) .
Hence, total communication cost is O (cid:0) M 1 .
5 d 3 (cid:1) .
Finite precision: we now consider the number of bits transmitted in the DisLinUCB protocol.
To that end, we make the following minor modiﬁcation to DisLinUCB.
First, when reward y t,i is observed, we replace it with a random integer in {± 1 } with expectation y t,i .
Second, after line 8 , after x t,i is played, we round each entry of x t,i with precision (cid:15) , and use the rounded vector in the calculation in line 9 .
In this case, each entry of W new,i and U new,i is a multiple of (cid:15) 2 .
Therefore, transmitting them requires O ( d 2 log (cid:15) − 1 ) bits.
The total communication complexity is then O (cid:0) M 1 .
5 d 3 log (cid:15) − 1 (cid:1) bits.
We now discuss how to choose (cid:15) such that regret is not effected.
Deﬁne
[Equation]
Here H is a shorthand for a history ( x 1 , 1 , y 1 , 1 , · · · , x T,M , y T,M ) .
D refers to set of rounded actions.
For every x ∈ D , there exists ¯ x ∈ D such that (cid:107) x − ¯ x (cid:107) ≤ √ d(cid:15) .
Therefore
[Equation]
On the other hand, let H be the (random) history of the DisLinUCB with rounding on action sets
D t , while H is the (random) history of the DisLinUCB with rounding running on action sets D t .
Then at each time step, the mapping from past history to the next action is the same.
Therefore
KL ( H , H ) =
O ( MTd(cid:15) 2 ) .
It follows that
[Equation]
When the action set is D t , no rounding is needed, so the regret analysis for the DisLinUCB protocol without rounding directly follows.
Therefore,
[Equation]
By choosing (cid:15) =
( MT ) − 1 , we can guarantee E [ REG ( H )]
= O (cid:16) d √ MT log 2 T (cid:17) for any action set.
In this case, the total number of communicated bits is O (cid:0) M 1 .
5 d 3 log T (cid:1) .
I DEMAB AND DELB IN P2P C OMMUNICATION N ETWORKS
In this section, we will brieﬂy discuss how to implement our protocols (i.e. DEMAB and DELB) in the P2P communication network considered in Korda et al. (2016).
We show that our protocols can be adopted to P2P networks after little modiﬁcation.
The communication cost will remain the same, while regret bounds would only increase marginally.
In P2P communication networks, an agent can receive information from at most one other agent at a time, which leads to an information delay for each agent.
In order to cope with such delay, we need to extend the length of each communication stage from 1 time step to M time steps, so that agents can complete the communication in turn.
Since there are at most O (log( MK )) communication stages in DEMAB and O (log T ) communication stages in DELB, the extension of communication stages incurs at most O ( M 2 log( MK )) regret in DEMAB and O ( M 2 log T ) regret in DELB for M agents.
When the time horizon T is large (i.e. T > M 3 log M ), the additional term is dominated by
O (cid:0) √ MKT log T (cid:1) and O (cid:0) d √ MT log T (cid:1) .
Another issue for P2P networks is that there is no longer a physical server in the networks.
To solve this problem, we can designate agent 1 as the server: agent 1 will execute both the codes for the server and the codes for an agent.
Speciﬁcally, by saying “extending the length of the communication stage”, we mean that we can use
Procedure 10 and 11 to realize communication subroutines used in our protocols in P2P networks: sending message to the server and receiving messages from the server.
Procedure 4: Server2Agent:
Agent 1 sends message m i to agent i in a P2P network.
1 For the next M − 1 time steps: /* For agent 1: */
Send m i to agent i + 1 at the i -th step
Pull an arbitrary arm at each time step /*
For agent i ( i > 1) :
Receive m i from agent 1 at the ( i − 1) -st step
Pull an arbitrary arm at each time step
Procedure 5: Agent2Server: Agents i ( i > 1) sends message m i to agent 1 in a P2P network.
1 For the next M − 1 time steps: /* For agent 1: */
Receive m i +1 from agent i + 1 at the i -th step
Pulls an arbitrary arm at each time step /* For agent
i ( i > 1) :
Send m i to agent 1 at the ( i − 1) -st step
Pull an arbitrary arm at each time step
I. DEMAB IN P2P N ETWORKS
For distributed DEMAB in a P2P network, we can replace the communication stage in DEMAB (i.e. line 8, 12, 15 of Protocol 1) by Procedure Server2Agent and Agent2Server.
In this way, it costs M time steps instead of a single time step to collect, aggregate, and boardcast information.
We have the following theorem showing the efﬁcacy of distributing DEMAB in a P2P network.
Theorem 7.
The
DEMAB protocol in
P2P networks incurs regret
O (cid:0) √ MKT log T + M 2 log( MK ) (cid:1) , with expected communication cost O
( M log( MK )) .
When T > M 3 log M , the regret bound of this protocol is near-optimal O (cid:0)
√ MKT log T (cid:1) .
Proof.
Regret: We compare DEMAB protocol in P2P net with the original one (i.e. Protocol 1).
The burn-in stage (i.e. Stage 1) of both protocols are the same.
For distributed elimination stage (i.e. Stage 2), the length of each phase in the new protocol is no shorter than that in Protocol 1.
Therefore, the number of phases after phase l 0 + 1 (included) in new protocol is no more than that in
Protocol 1, which is O (log( MK )) .
In each phase starting from phase l 0 + 1 , new protocol needs
O ( M ) additional steps to complete the communication in this phase, incurring O ( M ) additional regret per agent.
Therefore, this protocol incurs O
( M 2 ) additional regret per phase starting from phase l 0 + 1 .
The total regret of this protocol is thereby O (cid:0) √ MKT log T + M 2 log( MK ) (cid:1) .
Communication: We still consider only distributed elimination stage.
There are three communication stages per phase in Protocol 1: Line 8, 12, and 15.
In line 8 of DEMAB protocol, the total communication is O ( M ) since n max is boardcast from the server, and each agent sends n ( i ) l to the server.
We can observe that the communication cost at corresponding place is also O ( M ) by replacing the boardcast with Server2Agent.
In line 12, the communication cost of both protocols is still the same due to the same reason.
In line 15, the new protocol calls Agent2Server which runs for M steps, while the agents report the rewards in the original protocol in a single step.
The communication cost is O ( M ) for both protocols.
In summary, the communication cost of the new protocol is the same as that of Protocol 1, which is
O ( M log( MK )) .
I. DELB IN P2P N ETWORKS
Very similar to the P2P version of DEMAB, we can also distribute DELB to P2P networks by replacing the communication stage of DELB by Server2Agent and Agent2Server.
We have the following theorem for the P2P version of DELB.
Theorem 8.
The DELB protocol in a P2P network has regret O (cid:0) d √ MT log T + M 2 log T (cid:1) with expected communication cost O
( Md + d log log d ) log T ) .
When T > M 3 log M , the regret of
DELB is a near-optimal regret O (cid:0) d √ MT log T (cid:1) .
Proof.
The proof is very similar to the proof of theorem 7.
Note that in the P2P version of DELB, there are O (log T ) communication stages in total, which incurs O ( M 2 log T ) additional regret.
The communication cost of the new protocol is the same as Protocol 2 for the same reason mentioned in the proof of theorem 7.