Near-Optimal Collaborative Learning in Bandits
Abstract
This paper introduces a general multi-agent bandit model in which each agent is facing a finite set of arms and may communicate with other agents through a central controller in order to identify –in pure exploration– or play –in regret minimization– its optimal arm.
The twist is that the optimal arm for each agent is the arm with largest expected mixed reward, where the mixed reward of an arm is a weighted sum of the rewards of this arm for all agents .
This makes communication between agents often necessary.
This general setting allows to recover and extend several recent models for collaborative bandit learning, including the recently proposed federated learning with personalization [30].
In this paper, we provide new lower bounds on the sample complexity of pure exploration and on the regret.
We then propose a near-optimal algorithm for pure exploration.
This algorithm is based on phased elimination with two novel ingredients: a data-dependent sampling scheme within each phase, aimed at matching a relaxation of the lower bound.
Introduction
Collaborative learning is a general machine learning paradigm in which a group of agents collectively train a learning algorithm.
Some recent works have investigated how agents can efficiently perform sequential decision making in a collaborative context [36].
In particular, Shi et al.
propose an interesting setting to tackle collaborative bandit learning when some level of personalization is required [30].
Personalization leads to the twist that each agent should play the best arm in a mixed model which is obtained as a combination of her local model with the local model of other agents.
In this work, we introduce a more general model retaining this idea, that we call the weighted collaborative bandit model.
In this model, there are M agents and a finite number of K arms.
When agent m samples arm k at time t , she gets to observe a local reward X k,m ( t ) , which is drawn from a 1 -sub-Gaussiandistribution of mean µ k,m , independently from past observations and from other agents’ observations.
However, this agent does not necessarily seek to maximize her local reward, but rather some notion of mixed reward , related to the utility of that arm for other agents.
More specifically, we assume that agents share a known weight matrix W ∶= ( w n,m ) n,m ∈ [ 0 , 1 ] M × M , such that ∑ n ∈[ M ] w n,m = for all m ∈ [ M ] , where [ n ] ∶= { 1 , 2 ,...,n } .
The mixed reward at time t for agent m and arm k is defined as a weighted average of the local rewards across all agents X
′ k,m ( t ) ∶=
∑ Mn = 1 w n,m X k,n ( t ) , and its expectation, called the expected mixed reward, is µ ′ k,m ∶=
M ∑ n =w n,m µ k,n .
We denote by k ⋆ m ∶= arg max k ∈
[ K ] µ ′ k,m the arm with largest expected mixed reward for agent m , assumed unique.
Besides the degenerated case in which w n,m = 1 ( n = m ) , in which each agent is solving their own bandit problem in isolation, the agents need to communicate ; i .
e ., to share information about their local rewards to other agents for everyone to be able to estimate their expected mixed rewards.
A strategy for an agent is defined as follows: at each time t , each agent m samples an arm π m ( t ) , based on the available information, and then observes a noisy local reward from this arm.
She has also the option to communicate information ( e .
g ., empirical means of past local observations) to a central controller (or server), which will broadcast this information to all other agents.
Just like in any multi-armed bandit model, several objectives may be considered, either related to maximizing (mixed) rewards –or, equivalently, minimizing regret– or identifying the best arms, while maintaining a reasonably small communication cost.
The weighted collaborative bandit model encompasses different frameworks previously studied in the literature.
Notably, the paper [30] studies a special case in which, given a level of personalization α ∈ [ 0 , 1 ] , the mixed reward is an interpolation between µ k,m and the average of local rewards
M
∑ Mn = 1 µ k,n which amounts to choosing w m,m = α + 1 − αM and w n,m = 1 − αM for n ≠ m .
The authors consider the objective of minimizing the regret while minimizing for the number of communication rounds, under the name federated multi-armed bandit with personalization 2 .
In this paper, we mainly focus on the counterpart pure exploration problem in which agents should collaboratively identify their own optimal arm in terms of expected mixed reward, with high confidence, and using as few exploration rounds as possible.
This extends the well-studied fixed-confidence best arm identification problem [12, 14, 20] to the weighted collaborative bandit setting.
Another related setting is collaborative pure exploration [18, 32, 6, 35], which considers M agents solving the same best arm identification problem.
Most of these papers propose algorithms to solve this problem, while [32] and [21] also prove lower bounds on sample complexity for the fixed-confidence and the fixed-budget best arm identification.
Unlike our framework, [6] consider asynchronous agents, which can only sample at some times, whereas [35, 36] consider agents which can only communicate to some of the other agents.
The goal of collaborative pure exploration is to reduce sample complexity at the cost of some communication rounds.
Our model recovers the synchronous setting when considering µ k,m = µ k,m ′ for all arm k and agents m,m ′ and W = Id M .
Besides collaborative learning, the work of [34] considers a similar weighted model but in a different, contextual bandit setting in which a central controller chooses in each round an arm for the unique agent (corresponding to a sub-population) that arrives, and aim at minimizing regret.
Finally, the paper [29] considers a pure exploration task in which the value of an arm is the weighted average of its utility for M distinct populations (agents).
In this setting w n,m = α n , so that all agents have a common best arm, but the proposed algorithms do not aim at a low communication cost.
Collaborative learning in our general weighted model is also interesting beyond these examples.
In particular, the work of [30] mentioned possible applications to recommendation systems, for which one may want to go beyond uniformly personalized learning.
The “personalization” part means that we favour the local rewards of agent m over other agents’ observations in the identification of optimal arm k ⋆ m .
But the introduction of a general weight matrix W in our framework allows any agent m to consider any linear combination of the other agents’ observations, and then, different degrees of personalization across agents.
Such a setting could also be appropriate for adaptive clinical trials on K therapies, run by M teams who have access to different sub-populations of patients.
In this context, each sub-population is typically aiming at finding their (local) best treatment.
However, solving their best arm identification in isolation may have a large sample complexity.
If one is willing to assume that we have a weight matrix W for which the best mixed arm of each agent coincides with its local best arm, i .
e .
k ⋆ m = arg max k ∈[ K ] µ k,m for any agent m , then solving best arm identification in the weighted collaborative bandit could have a much smaller sample complexity due to the sharing of information, while allowing the different clinical centers to communicate only once in a while.
A first possibility to build such a weight matrix is to rely on a clustering of the different sub-populations so that the µ k,m is supposed to be close to µ k,m ′ (but not necessarily equal) for all k and all agents m,m ′ in the same cluster.
Denoting by C m the cluster to which agent m belongs, by setting w m,n = 1 ( C m =
C n )/∣
C m ∣ we would have the mixed mean of each agent be very close to their local means.
Another possibility is to rely on a similarity function S between sub-populations and define for all n,m w m,n to be proportional to S ( m,n ) .
This similarity function could be obtained prior to the learning phase by computing the similarities between subpopulation biomarkers, for instance.
Related work Collaborative bandit learning has recently sparked wide interest in the multi-armed bandit literature.
While some works do not deal with personalization [11, 31, 26], others have for instance studied the integration of arm features in a modified setting with personalization [19].
An interesting kernelized collaborative pure exploration problem was recently studied by [10].
In their model, both agents and arms are described by feature vectors, and there is a known kernel encoding the similarity between the mean reward of each (agent,arm) pair.
This independent work follows a similar approach as ours and also propose a near-optimal phased elimination algorithm inspired by a lower bound, but the models and related lower bounds are significantly different.
In bandits working in collaboration, the need for a small communication cost indeed makes algorithms based on phased eliminations appealing.
In such algorithms, agent(s) maintain a set of a active arms that are candidate for being optimal, and potentially eliminate arms from this set at the end of each sampling phase.
Adaptivity to the observed rewards (and, in our case, communication) is only needed between sampling phases, which are typically long.
This type of structure has been used in various bandit settings, both for regret minimization or pure exploration objectives [2, 18, 5, 13, 30, 3].
In some of these algorithms, including the one in [30] which motivates this paper, the number of samples gathered from an arm which is active in some phase r is fixed in advance.
We believe that going beyond such a deterministic sampling scheme is crucial to achieve optimal performance with phased algorithms.
In order to achieve (near)-optimality, other phased algorithms rely on computing an oracle allocation from the optimization problem associated with a lower bound on the sample complexity [13, 10].
In these works, based on this allocation, a total number of samples to collect in the current phase is computed, which depends on the identity of the surviving arms.
The distribution of samples across arms for the current round is proportional to the oracle allocation, and is obtained through a rounding procedure.
Compared to these works, our algorithm will show three distinctive features: an allocation inspired by a relaxation of the lower bound, which does not only depend on the identity of surviving arms, and an alternative to the rounding procedure.
Contributions
The authors of [30] exhibit an algorithm using phased elimination for federated bandit learning with personalization, and prove a logarithmic regret bound, whose dependency in the parameters of the problem is conjectured to be sub-optimal.
They also propose a heuristic improvement, based on a more adaptive exploration within each phase.
In this work, we take a step further in identifying the problem-dependent complexity of bandit learning in our novel weighted collaborative bandit model –which includes the setting of [30] as a special case– both from the pure exploration and the regret perspective.
We propose new information theoretic lower bounds, and a recipe to design algorithms (nearly) matching those for pure exploration.
Our main algorithmic contribution is for weighted collaborative best arm identification.
Our phased elimination-based algorithm achieves minimal exploration cost up to some logarithmic multiplicative factors, while using a constant amount of communication rounds.
It relies on a novel data-dependent sampling scheme, which renders its analysis trickier.
The structure of the algorithm can easily be extended to
Top- N identification, that is, where each agent has to identify her own N best arms (instead of her best for N = 1 ) with respect to mixed rewards.
We further compare our novel regret lower bound to what was conjectured in [30] for the particular case of federated learning with personalization.
Notation For both best arm identification and regret minimization, our complexity terms feature the (mixed) gaps of each agent m and arm k , defined by
[Equation]
Collaborative Best-Arm Identification
The goal of collaborative best-arm identification is that each agent m identifies its optimal arm
k ⋆ m by sampling the arms as little as possible and with few communication rounds.
Formally, a collaborative
Best Arm Identification (BAI) algorithm consists of a sampling rule π m for each agent m , such that, at time t , either arm π m ( t ) ∈
[ K ] is sampled by m , or π m ( t ) =
; in that case, instead of picking an arm at time t , we allow the agent to remain idle, and not to select an arm.
3 Similarly to the communication model studied in [32, 30], communication only happens at the end of local sampling rounds for all agents, when all agents are idle at the same time.
Besides the sampling rule, the BAI algorithm uses a stopping rule τ which determines when exploration is over for all agents .
The end of exploration is decided by the central server.
Then, at time τ , each player outputs a guess for its optimal arm with respect to mixed rewards, denoted by ˆ k m .
Our goal is to construct a δ -correct strategy A
=
( π,τ, ˆ k ) , which satisfies, for any model µ ∈ R K × M ,
[Equation]
while achieving a small exploration cost (e.g. in high probability or in expectation)
[Equation]
where N k,m ( t ) ∶=
∑ ts = 1 1 ( π m ( s )=
k ) is the number of selections of arm k by agent m up to time t , and a small communication cost , defined as
[Equation]
where I t is the event that some information is shared between agents at round t .
In our setting, we do not put constraints on the type of information that is exchanged in each communication round –which can be interesting when we consider privacy issues [11, 36]– nor on the lengths of the messages.
Each communication round has a unit cost.
In a communication round, all agents send messages to the central server ( e .
g ., estimates of their local means) and the server can send back arbitrary quantities or instructions ( e . g .
, how many times each arm should be sampled in the next exploration phase, and when to communicate next).
Moreover, contrary to the works of [18, 32] on collaborative learning, we do not look at strategies explicitly minimizing for the number of communication rounds.
Instead, our approach consists in proving a lower bound on the smallest possible exploration cost of a δ -correct algorithm which would communicate at every round ; and then, finding an algorithm for which exploration cost matches this lower bound, while suffering a reasonable communication cost.
Lower Bound
We prove the following lower bound on the exploration cost of an algorithm in which all agents communicate to the central server their latest observation as soon as they received it.
It holds for
Gaussian rewards with variance σ 2 = 1 , meaning that the reward from an arm k observed at time t by agent m will be X k,m ( t ) =
µ k + ε t , where ε t ∼ N( 0 , 1 ) .
We further assume that the weight matrix W satisfies w m,m ≠ 0 for any agent m ∈
[ M ] .
Theorem 1.
Let µ be a fixed matrix of means in R K × M .
For any δ ∈ ( 0 , 1 / 2 ] , let A be a δ -correct algorithm under which each agent communicates each reward to the central server after it is observed, and let us denote for any k ∈
[ K ] , m ∈
[ M ] , τ k,m
∶= E A µ [ N k,m ( τ )] , where τ is the stopping time.
For any m ∈ [ M ] and k ≠ k ⋆ m , it holds that
[Equation]
and therefore E µ [ Exp µ (A)] ≥ T ⋆ W ( µ ′ ) log ( 12 . 4 δ ) , where
[Equation]
⎫⎪⎪⎬⎪⎪⎭ .
The proof, given in Appendix A.1, uses standard change-of-distribution arguments, together with classical results from constrained optimization.
Note that, for M = 1 , we recover the complexity of best arm identification in a Gaussian bandit model [15].
Computing the complexity term The optimization problem which defines T ⋆ W ( µ ) belongs to the family of disciplined convex optimization problems, and can be numerically solved using available solvers, such as CVXPY [1, 9].
We now illustrate, on a small example, the possible reduction in exploration cost that can be obtained by solving weighted collaborative best arm identification instead of M parallel best arm identification problems.
We consider K =
M = 2 and a similarity
S ( 1 , 2 ) = 0 .
9 between the two agents, which yields the following normalized weight matrix
[Equation]
Considering the following matrix of expected rewards
[Equation]
for which arm 1 is the local best arm for both agents and is also their best mixed arm, we obtain
T ⋆ W ( µ ) ≈ 28 .
However, if each agent solves its own best arm identification problem in isolation (which amounts to using W = Id 2 ), the resulting exploration cost scales with T ⋆ Id 2 ( µ ) ≈ 101 > 3 T ⋆ W ( µ ) .
From lower bounds to algorithms In single-agent pure exploration tasks, lower bounds are usually guidelines to design optimal algorithms, as they allow to recover an oracle allocation ( i .
e ., the arg min for t ∈ ( R + ) K,M in the definition of T ⋆ W ( µ ) ) which algorithms can try to achieve by using some tracking [15, 10, 29].
Yet, these approaches may be computationally expensive, as they solve the optimization problem featured in the lower bound in every round.
In the next section, we will propose an alternative approach for our collaborative setting, which exploits the knowledge of the lower bound within a phased elimination algorithm.
This is crucial to maintain a small communication cost and also permit to reduce the computational complexity compared to a pure tracking approach.
Our algorithm will rely on a relaxed complexity term ̃ T ⋆ W ( µ ) , which is within constant factors of T ⋆ W ( µ ) , as proved in Appendix C.1.
Lemma 1.
Introducing the quantity
[Equation]
it holds that ̃ T ⋆ W ( µ ) ≤ T ⋆ W ( µ ) ≤ 2 ̃ T ⋆ W ( µ ) .
Compared to T ⋆ W ( µ ) , a nice feature of ̃ T ⋆ W ( µ ) is that its constraint set does not depend on the knowledge of ( k ⋆ m ) m ∈
[ M ] , which will allow us to design algorithms that do not suffer too much from bad empirical guesses for k ⋆ m in early phases.
We further remark that the computation of ̃ T ⋆ W ( µ ) and that of its associated oracle allocation (see Definition 1) are slightly easier than for T ⋆ W ( µ ) .
Indeed, the optimization problem which defines ̃ T ⋆ W ( µ ) can be decoupled across arms.
Computing for every arm k ∈ [ K ] the vector
[Equation]
we obtain the argmin in ̃ T ⋆ W ( µ ) by setting ( t k,m ) k,m =
(̃ τ km ) k,m .
The computation of ̃ T ⋆ W ( µ ) can therefore be done by solving K disciplined optimization problems ( e
) involving
Definition 1.
For any ∆ ∈ ( R + ) K × M , the oracle ̃P ⋆ ( ∆ ) is
[Equation]
With this notation, observe that ̃ T ⋆ ( µ ′ ) =
∑ k,m τ k,m , where ( τ k,m ) k,m ∈ ̃P ⋆ ( ∆ ′ ) .
The following lemma will be useful to compare values from different oracle problems.
The full lemma along with its proof is available in Appendix C.1.
Lemma 2. Consider ∆ , ∆ ′ ∈ ( R + ) K × M , such that τ ∈ ̃P ⋆ ( ∆ ) and τ ′ ∈ ̃P ⋆ ( ∆ ′ ) .
Moreover, assume that there is a positive constant β such that: ∀ k,m, ∆ ′ k,m ≤ β ∆ k,m .
Then
[Equation]
A Near-Optimal Algorithm For Best Arm Identification
We now introduce an algorithm for collaborative best-arm identification, called W-CPE-BAI for
Weighted Collaborative Phased Elimination, stated as Algorithm 1.
To present an analysis of this algorithm, we assume that, for any k,m ∈
[ K ] ×
[ M ] , µ k,m ∈ [ 0 , 1 ] , and that local rewards ( X k,m ( t )) k,m,t are 1 -sub-Gaussian.
W-CPE-BAI proceeds in phases, indexed by r .
In phase r , we let B m ( r ) be the set of active arms for agent m , and B ( r ) =
∪ m B m ( r ) be the set of arms that are active for at least one agent.
The algorithm maintains proxies for the gaps (̃ ∆ k,m ( r ))
k ∈[ K ] ,m ∈
[ K ] that are halved at the end of each phase for arms that remain active.
At the beginning of each round, the oracle allocation t ( r ) , with respect to the proxy gaps, is computed, as well as the number of new samples d k,m ( r ) that player m should get from arm k in phase r .
d k,m ( r ) is defined such that the total number of selections of arm k by agent m becomes close to (a quantity slightly larger than) t k,m ( r ) log ( 1 / δ ) .
We observe that any arm k ∉ B ( r ) will not get any new samples in phase r , as the proxy gaps (̃ ∆ k,n ( r )) n are identical to those in the previous phase ; therefore t k,n ( r ) =
t k,n ( r − 1 ) .
In contrast to prior works, where the allocation in each round only depends on the identity of the surviving arms and the round index [13, 10], in W-CPE-BAI it also depends on when the arms have been eliminated (which condition the value of their frozen proxy gaps).
After each agent m samples arm k d k,m ( r ) times, they send their local means ˆ µ k,m ( r ) to the central server, which computes the mixed mean estimates ˆ µ ′ k,n ( r ) ∶=
∑ Mm = 1 w n,m ˆ µ k,m ( r ) .
The active sets ( B m ( r )) m ∈
[ M ] of all agents are then updated by removing arms whose mixed means are too small.
As in several prior works [22, 30], we rely on confidence intervals to perform these eliminations.
However, constructing confidence intervals on the mixed means –which are linear combinations of the local means– under our adaptive sampling rule is more challenging than when the number of samples from an active arm in phase r is fixed in advance –which is the case for instance in the algorithm in [30].
The width of our confidence intervals scales with the following quantity
Definition 2.
For any k,m , and round r ≥ 0 , we define
[Equation]
where n k,m ( r ) is the number of times arm k was selected by agent m by the end of phase r (included), and N ↦ β δ ( N ) is a threshold function defined for any N ∈ ( R + ) M .
Leveraging some recent time-uniform concentration inequalities [23], we exhibit below a choice of threshold that yields valid confidence intervals on the mixed means (by "projecting" confidence intervals that can be obtained on local means, see Proposition 24 in [23]).
The fact that the confidence interval depends on the random number of past draws (and not just the index of the round) leads to some non-trivial complication in the analysis, with the introduction of quantities ( d k,m ) k,m .
The proof is given in Appendix C.2, where we also provide an explicit expression of the function g M .
Input: δ ∈ ( 0 , 1 ) , M agents, K arms, weights matrix W
Initialize r ← 0 , ∀ k,m, ̃ ∆ k,m ( 0 ) ← 1 ,n k,m ( 0 ) ← 1 , ∀ m,B m ( 0 ) ← [ K ]
Draw each arm k by each agent m once repeat # Central server
B
[Equation]
r
[Equation]
Compute t ( r ) ←
[Equation] [Equation]
r
[Equation] [Equation]
For any k ∈ [ K ] , compute ( d
[Equation]
,m ( r
[Equation] [Equation]
←
[Equation]
( r )
Send to each agent m ( d k,m ( r ))
k,m and d max ∶= max n ∈[ M ] ∑ k ∈[ K ] d k,n ( r ) # Agent m
Sample arm k ∈ B ( r ) d k,m ( r ) times, so that n k,m ( r ) =
n k,m ( r − 1 ) + d k,m ( r )
Remain idle for d max − ∑ k ∈[ K ] d k,m ( r ) rounds
Send to the server empirical mean ˆ µ k,m ( r ) ∶=
∑ s ≤ n k,m ( r ) X k,m ( s )/ n k,m ( r ) for any k ∈ [ K ] # Central server
Compute the empirical mixed means ( ˆ µ ′ k,m ( r ))
k,m based on ( ˆ µ k,m ( r ))
k,m and W // Update set of candidate best arms for each user for m = 1 to M do
[Equation]
)} end for // Update the gap estimates
For all k,m , ̃ ∆ k,m ( r + 1 ) ← ̃ ∆ k,m ( r ) × ( 1 / 2 ) 1 ( k ∈ B m ( r + 1 )∧∣ B m ( r + 1 )∣> 1 )
r ← r +until ∀ m ∈ [ M ] , ∣ B m ( r )∣ ≤
Output: { k ∈ B m ( r ) ∶ m ∈
[ M ]}
Lemma 3.
Let us define
[Equation]
for any N ∈ ( N ∗ ) M , where g M is some non-explicit function, defined in [23], that satisfies g M ( δ ) ≃ log ( 1 δ ) + M log log ( 1 δ ) .
Then the good event
[Equation]
holds with probability larger than 1 − δ .
From this lemma, it easily follows that W-CPE-BAI is δ -correct for the above choice of threshold function, as, for any agent m , no good arm k ⋆ m can ever be eliminated from B m ( r ) at round r .
The fact that the sample complexity of W-CPE-BAI scales with ̃ T ⋆ ( µ ′ ) on the good event E comes from the interplay between the expression of Ω k,m ( r ) (which, up to the threshold function, is exactly one of the constraints featured in the lower bound) together with the definition of the allocation t ( r ) , which leads to the following crucial result in our analysis
Lemma 4.
On E , ∀ k,m,r ≥ 0 , Ω k,m ( r ) ≤ ̃ ∆ k,m ( r ) .
Proof.
For any round r and arm k , by Algorithm 1 and the definition of oracle t k, ⋅ ( r ) , for any agent m , Ω k,
[Equation]
=
[Equation]
, where when k /∈ B ( r ) , r ′ k ∶= sup { r ′ ≥
∶ k ∈ B ( r ′ )} , and we use the fact that d k,m ( r ) = 0 when k /∈ B ( r ) .
We did not put much emphasis on the way communications are performed between the agents and the central server, as several choices are possible.
The important part is that the server receives all values of the local means ( ˆ µ k,m ( r ))
k,m at the end of round r .
Our suggestion is that the central server maintains the sets ( B m ( r )) m ∈
[ M ] , calls the oracle, and sends to all agents their values of ( d k,m ( r + 1 ))
k,m at the end of each phase r .
In any case, the number of communication rounds in our definition will be equal to the number of phases used by W-CPE-BAI.
All in all, we prove
Theorem 2.
With probability 1 − δ , W-CPE-BAI outputs the optimal arm for each agent with an exploration cost at most
[Equation]
and at most ⌈ log 2 ( 8 / ∆ ′ min )⌉ communication rounds, where ∆ ′ min ∶= min k ∈
[ K ] ,m ∈
[ M ] ∆ ′ k
[Equation]
Proof sketch The detailed proof is given in Appendix B, where we also provide an explicit upper bound on the exploration cost.
We let R denote the (random) number of phases used by the algorithm before stopping.
On the good event E , we can prove that the algorithm never eliminates k ⋆ m therefore
R ∶= max m max k ≠ k
⋆ m R k,m where R k,m is the last phase in which k ∈ B m ( r ) .
Using Lemma 4, we can easily establish that
[Equation]
which satisfies r k,m ≤ log 2 ( 8 / ∆ ′ k,m )
This yields R ≤ log 2 ( 8 / ∆ ′ min ) , and further permits to prove that the proxy gaps can be lower bounded by the true gaps:
[Equation]
See Corollary 2 in Appendix B. Using the monotonicity properties of the oracle that are stated in Lemma 2, we can then establish that the allocation t ( r ) computed from the proxy gaps in the algorithm satisfies
[Equation]
To upper bound the exploration cost, the next step is to relate n k,m ( R ) to the oracle allocations.
To do so, we observe that if R ′ k,m is the last round before R such that d k,m ( r )
≠ 0 (i.e. the last round in which arm k is actually sampled by agent m ; then n k,m ( R ) =
n k,m ( R ′ k,m ) ), we have by definition of the ( d k,m ( r ))
k,m,r that
[Equation]
See Lemma 12 in Appendix B.
We can then upper bound τ ∶= ∑ k,m n k,m ( R ) as follows
[Equation]
and τ
[Equation]
KM , where we use (1) and introduce
[Equation] [Equation]
The end of the proof consists in using the known upper bound on R , and finding an upper bound for the largest τ satisfying the inequality in (2).
Discussion Theorem 2 proves that W-CPE-BAI is matching the exploration lower bound of Theorem 1 in a regime where δ is small, up to multiplicative constants, including a logarithmic term in 1 / ∆ ′ min .
It achieves this using only ⌈ log 2 ( 8 / ∆ ′ min )⌉ communication rounds.
We note that a similar extra multiplicative logarithmic factor is present in the analysis of near-optimal phased algorithms in other contexts [13, 10].
Such a quantity appears as an upper bound on the number of phases, and may be a price to pay for the phased structure.
On the communication cost We argue that the communication cost of W-CPE-BAI is actually of the same order of magnitude as that featured in some related work.
In [30], which is the closest setting to our framework, the equivalent number of communication rounds p needed to solve the regret minimization problem is upper bounded by O ( 2log 2 ( 8 /(√ M ∆ ′ min ))) .
In the setting of collaborative learning –where M agents face the same set of arm distributions and W = Id – [18] in their Theorem 4 .
1 prove that an improvement of multiplicative factor 1 / M on the exploration cost for a traditional best arm identification algorithm can be reached by using at most ⌈ log 2 ( 1 / ∆ min )⌉ communication rounds, where ∆ min is the gap between the best and second best arms.
Experimental validation We propose in Appendix E an empirical evaluation of W-CPE-BAI for the weight matrix w m,n = α 1 ( n = m ) + 1 − αM which corresponds to the setting studied by [30].
In this particular case, we propose as a baseline a counterpart of the regret algorithm of [30] which we call
PF-UCB-BAI.
Our experiments on a synthetic instance show that W-CPE-BAI and PF-UCB-BAI have similar performances in terms of exploration cost and that W-CPE-BAI becomes better when the level of personalization α is smaller than 0 .
Moreover, W-CPE-BAI uses less rounds of communication than PF-UCB-BAI for all values of α .
Finaly, the near-optimality of W-CPE-BAI is empirically observed when compared to an oracle algorithm which has access to the true gaps.
We refer the reader to Appendix E for further details on the optimization libraries that were used.
Remark 1.
The analysis and the structure of Algorithm 1 have the potential to be extended to other pure exploration problems, with similar guarantees.
In Appendix D, we illustrate this claim by tackling Top- N identification.
Regret Lower Bound
In contrast to the BAI setting, [30] considered the objective of minimizing the regret
[Equation]
They provided a conjecture on the lower bound on regret in personalized federated learning [see, 30, Conjecture 1 ].
As mentioned in introduction, their reward model is a special case of ours with weights w m,m = α + 1 − αM and w n,m = 1 − αM for n ≠ m .
In this section, we prove a regret lower bound with general weights that proves in particular Conjecture 1 in [30].
We prove the following result, for an algorithm that selects in each round exactly one arm for each agent, and all agents communicate after each round.
This lower bound holds for Gaussian arms with variance σ 2 = 1 .
Theorem 3.
Any uniformly efficient algorithm 4 in which all agents communicate after each round satisfies
[Equation]
where
[Equation]
We recall that for any agent m , ∆ ′
k ⋆ m ,m
∶= min k ′ ≠ k
⋆ m ∆ ′ k ′ ,m .
Theorem 3 may be viewed as an extension of the lower bound of [25] to the collaborative setting.
Its proof, given in Appendix A.2, uses classical ingredients for regret lower bounds in (single agent) structured bandit models [7, 17].
The generic approaches proposed in [7, 8] for optimal regret minimization in structured bandits could therefore be useful.
However, to turn them into a reasonable algorithm for the collaborative setting, we would need to preserve the phased elimination structure.
Analogous to the relaxation in the BAI setting, we can also define a relaxed complexity term ̃ C M ( µ ) , which does not require the knowledge of the best arms, and is within constant factors of C ( µ ) by the following Lemma 5 (which proof is given in Appendix C.3).
Lemma 5.
Introducing the quantity
[Equation]
it holds that C ⋆ W ( µ ) ≤ ̃ C ⋆ W ( µ ) ≤ 4 C ⋆ W ( µ ) .
The constrained optimization problems governing the regret lower bound are subtly different from those in the BAI setting.
In regret minimization, unknown gaps ( ∆ ′ k,m ) k,m appear both in the constraints and in the objective.
In contrast, in BAI, ( ∆ ′ k,m ) k,m only appear in the constraints.
Due to this difference, designing a similar algorithm for regret minimization leads to an extra multiplicative
O( 1 / ∆ ′ min ) factor in the upper bound.
A very similar issue exists in a different structured bandit problem, bandits on graphs with side observations, where arms are connected through a graph with unweighted edges, and the agent receives the reward associated with the selected arm and its neighbors at a given round.
In [4, Problem P 1 ], authors describe a constrained linear optimization problem which governs the regret lower bound, and face the same issue of scaling in O( 1 / ∆ ′ min ) .
Dealing with this problem would be an interesting subsequent work.
Conclusion
This paper introduced a general framework for collaborative learning in multi-armed bandits.
Our work presents two novel lower bounds: one on the exploration cost for pure exploration, and another on cumulative regret.
The latter permits to prove a prior conjecture on the topic.
Moreover, we propose a phased elimination algorithm for fixed-confidence best arm identification.
This algorithm tracks the optimal allocation from the pure exploration lower bound through a novel approach solving a relaxed optimization problem linked to the lower bound.
The exploration cost of this algorithm is matching the lower bound up to logarithmic factors.
This strategy can be extended to other pure exploration problems, such as Top- N identification, as shown in Appendix D.
As mentioned in introduction, our collaborative setting was motivated by the design of collaborative adaptive clinical trials for personalized drug recommendations, where several patient subpopulations (for instance, representing several subtypes of cancer) are considered and sequentially treated.
However, in practice, especially when dealing with patient data, disclosing the mean response values to the central server should be handled with care to preserve the anonymity of the patients.
A possible solution to overcome this problem would be to carefully combine our algorithm with a data privacy-preserving method, for instance by adding some noise to the data [11].
Our code and run traces are available in an open-source repository.
Acknowledgments and Disclosure of Funding
Clémence Réda was supported by the French Ministry of Higher Education and Research
[ENS.X19RDTME-SACLAY19-22].
The authors acknoweldge the support of the French National
Research Agency under the project [ANR-19-CE23-0026-04] (BOLD).
[-From Equation (3), we get 12 τ k,m ( µ k,m − λ k,m ) 2 > 
-]
[-T 
-]
[-[ M ] , let us prove by induction on ∣ S ∣ ≥ 
-]
[-= 
-]
[-( r )⌉ + ⌈ αMf ( r )⌉ 
-]