Communication Efﬁcient Federated Learning for Generalized Linear Bandits
Abstract
Contextual bandit algorithms have been recently studied under the federated learning setting to satisfy the demand of keeping data decentralized and pushing the learning of bandit models to the client side.
But limited by the required communication efﬁciency, existing solutions are restricted to linear models to exploit their closed-form solutions for parameter estimation.
Such a restricted model choice greatly hampers these algorithms’ practical utility.
In this paper, we take the ﬁrst step to addressing this challenge by studying generalized linear bandit models under the federated learning setting.
We propose a communication-efﬁcient solution framework that employs online regression for local update and ofﬂine regression for global update.
We rigorously proved, though the setting is more general and challenging, our algorithm can attain sub-linear rate in both regret and communication cost, which is also validated by our extensive empirical evaluations.
Introduction
As a classic model for sequential decision making problems, contextual bandit has been widely used for a variety of real-world applications, including recommender systems [19], display advertisement [21] and clinical trials [7].
While most existing bandit solutions are designed under a centralized setting (i.e., data is readily available at a central server), in response to the increasing application scale and public concerns of privacy, there is increasing research effort on federated bandit learning lately [28, 6, 27, 11, 17], where N clients collaborate with limited communication bandwidth to minimize the overall cumulative regret incurred over a ﬁnite time horizon T , while keeping each client’s raw data local.
Compared with standard federated learning [23, 13] that works with ﬁxed datasets, federated bandit learning is characterized by its online interactions with the environment, which continuously provides new data samples to the clients over time.
This brings in new challenges in addressing the conﬂict between the need of timely data/model aggregation for regret minimization and the need of communication efﬁciency with decentralized data.
A carefully designed model update method and communication strategy become vital to strike this balance.
Existing federated bandit learning solutions only partially addressed this challenge by considering simple bandit models, like context-free bandit [27] and contextual linear bandit [28, 6, 17], where closed-form solution for both local and global model update exists.
Therefore, efﬁcient communication for global bandit model update is realized by directly aggregating local sufﬁcient statistics, such that the only concern left is how to control the communication frequency over time horizon T .
However, such a solution framework does not apply to the more complicated bandit models that are often preferred in practice, such as generalized linear bandit (GLB) [8] or neural bandit [30], where only iterative solutions exist for parameter estimation (e.g., gradient-based optimization).
To enable joint model estimation, now the learning system needs to solve distributed optimization for multiple times as new data is collected from the environment, and each requires iterative gradient/model aggregation among clients.
This is much more expensive compared with linear models, and it naturally leads to the question: whether a communication efﬁcient solution to this challenging problem is still possible?
In this paper, we answer this question afﬁrmatively by proposing the ﬁrst provably communication efﬁcient algorithm for federated GLB that only requires ˜ O ( √ T ) communication cost, while still attaining the optimal order of regret.
Our proposed algorithm employs a combination of online and ofﬂine regression, with online regression adjusting each client’s model using its newly collected data, and ofﬂine (distributed) regression occasionally soliciting local gradients from all N clients for joint model estimation when sufﬁcient amount of new data has been accumulated.
In order to balance exploration and exploitation in arm selection, we propose a novel way to construct the conﬁdence set based on the sequence of ofﬂine-and-online model updates that each client has received.
The initialization of online regression with ofﬂine regression introduces dependencies that break the standard martingale argument, which requires proof techniques unique to this paper.
We also explored other non-trivial solution ideas to further justify our current design.
Speciﬁcally, in practice, a common way to update the deployed model for applications with streaming data is to set a schedule and periodically re-train the model using iterative optimization methods.
For comparison, we propose and rigorously analyze a federated GLB algorithm designed based on this idea, as well as a variant that further enables online updates on the clients.
We also consider another solution idea motivated by distributed/batched online convex optimization, which is characterized by lazy online updates over batches of data.
Moreover, extensive empirical evaluations on both synthetic and real-world datasets are performed to validate the effectiveness of our algorithm.
Related Work
GLB, as an important extension of linear bandit models, has demonstrated encouraging performance in modeling binary rewards (such as clicks) that are ubiquitous in real-world applications [18].
The study of GLB under a centralized setting dates back to Filippi et al.
[8], who proposed a UCB-type algorithm that achieved ˜ O
( d √ T ) regret.
Li et al.
[20] later proposed two improvements: a similar
UCB-type algorithm that improves the result of [8] by a factor of O (log T ) , which has been popularly used in practice as it avoids the projection step needed in [8]; and another impractical algorithm that further improves the result by a factor of O ( √ d ) assuming ﬁxed number of arms.
To improve the time and space complexity of the aforementioned GLB algorithms, followup works adopted online regression methods.
In particular, motivated by the online-to-conﬁdence-set conversion technique from [2], Jun et al.
[12] proposed both UCB and Thompsan sampling algorithms with online Newton step, and Ding et al.
[4] proposed a Thompson sampling algorithm with online gradient descent, which, however, requires an additional context regularity assumption to obtain a sub-linear regret.
GLB under federated/distributed setting still remains under-explored.
The most related works are the federated/distributed linear bandits [16, 28, 6, 11, 17].
In these works, thanks to the existence of closed-form solution for linear models, the clients only communicate their local sufﬁcient statistics for global model update.
Korda et al.
[16] considered a peer-to-peer (P2P) communication network and assumed the clients form clusters, i.e., each cluster is associated with a unique bandit problem.
But as they only focused on reducing per-round communication, the communication cost is still linear over time.
Huang et al.
[11] considered a star-shaped communication network as in our paper, but their proposed phase-based elimination algorithm only works in ﬁxed arm set setting.
The closest works to ours are [28, 6, 17], which uses event-triggered communication protocols to obtain sub-linear communication cost over time for federated linear bandit with a time-varying arm set.
Another related line of research is the standard federated learning that considers ofﬂine supervised learning problems [13].
Since its debut in [23], FedAvg has become the most popularly used algorithm for ofﬂine federated learning.
However, despite its popularity, several works [22, 14, 24] identiﬁed that FedAvg suffers from a client-drift problem when the clients’ data are non-IID (which is an important signature of our case), i.e., local iterates in each client drift towards their local minimum.
This leads to a sub-optimal convergence rate of FedAvg: for example, one has to suffer a sub-linear convergence rate for strongly convex and smooth losses, though a linear convergence rate is expected under a centralized setting.
To alleviate this, Pathak and Wainwright [26] proposed an operator splitting procedure to guarantee linear convergence to a neighborhood of the global minimum.
Later,
Mitra et al.
[24] introduced variance reduction techniques to guarantee exact linear convergence to the global minimum.
Preliminaries
In this section, we ﬁrst introduce the general problem formulation of federated bandit learning, and discuss the existing solutions under the linear reward assumption.
Then we formulate the federated
GLB problem considered in this paper, followed by detailed discussions about the new challenges compared with its linear counterpart.
3.1 Federated Bandit Learning
Consider a learning system with 1) N clients responsible for taking actions and receiving corresponding reward feedback from the environment, e.g., each client being an edge device directly interacting with a user, and 2) a central server responsible for coordinating the communication between the clients for joint model estimation.
At each time step t = 1 , 2 , ..., T , all N clients interact with the environment in a round-robin manner, i.e., each client i ∈ [ N ] chooses an arm x t,i from its time-varying candidate set A t,i = { x (1) t,i , x (2) t,i , . . .
, x ( K ) t,i } , where x ( a ) t,i ∈ R
d denotes the context vector associated with the a -th arm for client i at time t .
Without loss of generality, we assume || x ( a ) t,i || 2 ≤ 1 , ∀ i, a, t .
Then client i receives the corresponding reward y t,i ∈ R from the environment, which is drawn from the reward distribution governed by an unknown parameter θ (cid:63) ∈ R d (assume (cid:107)
θ (cid:63) (cid:107) ≤ S ), i.e., y t,i ∼ p θ (cid:63) ( y | x ( a ) t,i ) .
The interaction between the learning system and the environment repeats itself, and the goal of the learning system is to minimize the cumulative (pseudo) regret over all N clients in the ﬁnite time horizon T , i.e., R T =
(cid:80) Tt =1 (cid:80)
Ni =1 r t,i , where r t,i = max x ∈A t,i E
[ y | x ] − E [ y t,i | x t,i ] .
In a federated learning setting, the clients cannot directly communicate with each other, but through the central server, i.e., a star-shaped communication network.
Raw data collected by each client i ∈ [ N ] , i.e., { ( x s,i , y s,i ) } s ∈ [ T ] , is stored locally and cannot be shared with anyone else.
Instead, the clients can only communicate the parameters of the learning algorithm, e.g., models, gradients, or sufﬁcient statistics; and the communication cost is measured by the total number of times data being transferred across the system up to time T , which is denoted as C T .
3.2 Federated Linear Bandit
Prior works have studied communication-efﬁcient federated linear bandit [28, 6], i.e., the reward function is a linear model y t,i =
x (cid:62) t,i θ (cid:63) + η t,i , where η t,i denotes zero-mean sub-Gaussian noise.
Consider an imaginary centralized agent that has direct access to the data of all clients, so that it can compute the global sufﬁcient statistics
[Equation]
Then the cumulative regret incurred by this distributed learning system can match that under a centralized setting, if all N clients select arms based on the global sufﬁcient statistics { A t , b t } .
However, it requires N 2 T communication cost for the immediate sharing of each client’s update to the sufﬁcient statistics with all other clients, which is expensive for most applications.
To ensure communication efﬁciency, prior works like DisLinUCB [28] let each client i maintain a local copy { A t − 1 ,i , b t − 1 ,i } for arm selection, which receives immediate local update using each newly collected data sample, i.e., A t,i =
A t − 1 ,i + x t,i x (cid:62) t,i , b t,i =
b t − 1 ,i + x t,i y t,i .
Then client i checks whether the event ( t − t last ) log( det A t,i det A t last ) >
D is true, where t last denotes the time step of last global update.
If true, a new global update is triggered, such that the server will collect all clients’ local update since t last , aggregate them to compute { A t , b t } , and then synchronize the local sufﬁcient statistics of all clients, i.e., set { A t,i , b t,i } =
{ A t , b t } , ∀ i ∈ [ N ] .
3.3 Federated Generalized Linear Bandit
In this paper, we study federated bandit learning with generalized linear models, i.e., the conditional distribution of reward y given context vector x is drawn from the exponential family [8, 20]:
[Equation]
where τ ∈ R + is a known scale parameter.
Given a function f : R → R , we denote its ﬁrst and second derivatives by ˙ f and ¨ f , respectively.
It is known that ˙ m ( x (cid:62) θ (cid:63) ) =
E [ y | x ] := µ ( x (cid:62) θ (cid:63) ) , which is called the inverse link function, and ¨ m ( x (cid:62) θ (cid:63) ) =
V ( y | x (cid:62) θ (cid:63) ) .
Based on Eq.(1), the reward y t,i observed by client i at time t can be equivalently represented as y t,i = µ ( x (cid:62) t,i θ (cid:63) ) + η t,i , where η t,i denotes the sub-Gaussian noise.
Then we denote the negative log-likelihood of y i,t given x i,t as l ( x (cid:62) t,i θ (cid:63) , y t,i ) =
− log p θ (cid:63) ( y t,i | x t,i ) =
− y t,i x (cid:62) t,i θ (cid:63) + m ( x (cid:62) t,i θ (cid:63) ) .
In addition, we adopt the following two assumptions about the reward, which are standard for GLB [8].
Assumption 1.
The link function µ is continuously differentiable on ( − S, S ) , k µ -Lipschitz on [ − S, S ] , and inf z ∈ [ − S,S ] ˙ µ ( z ) =
c µ > 0 .
Assumption 2.
E [ η t,i |F t,i ] = 0 , ∀ t, i , where F t,i =
σ { x t,i , [ x s,j , y s,j ] ( s,j ): s<t ∩ j = i } denotes the σ -algebra generated by client i ’s previously pulled arms and observed rewards, and max t,i | η t,i | ≤
R max for some constant R max > 0 .
New Challenges Compared with federated linear bandit discussed in Section 3.2, new challenges arise in designing a communication-efﬁcient algorithm for federated GLB due to the absence of a closed form solution:
• Iterative communication for global update: compared with the global update for federated linear bandit that only requires one round of communication to share the sufﬁcient statistics, now it takes multiple iterations of gradient aggregation to obtain converged global optimization.
Moreover, as the clients collect more data samples over time during bandit learning, the required number of iterations for convergence also increases.
• Drifting issue with local update: during local model update, iterative optimization using only local gradient can push the updated model away from the global model,
i
e., forget the knowledge gained during previous communications [15].
Methodology
In this section we propose the ﬁrst algorithm for federated GLB that addresses the aforementioned challenges.
We rigorously prove that it attains sub-linear rate in T for both regret and communication cost.
In addition, we propose and analyze different variants of our algorithm to facilitate understanding of our algorithm design.
4.1 FedGLB-UCB Algorithm
To ensure communication-efﬁcient model updates for federated GLB, we propose to use online regression for local update, i.e., update each client’s local model only with its newly collected data samples, and use ofﬂine regression for global update, i.e., solicit all clients’ local gradients for joint model estimation.
Based on the resulting sequence of ofﬂine-and-online model updates, the conﬁdence ellipsoid for θ (cid:63) is constructed for each client to select arms using the OFUL principle.
We name this algorithm Federated Generalized Linear Bandit with Upper Conﬁdence Bound, or
FedGLB-UCB for short.
We illustrate its key components in Figure 1 and describe its procedures in
Algorithm 1.
In the following, we discuss about each component of FedGLB-UCB in details.
to server, and reset ∆ A t,i =12: Server : compute A t = A t last + (cid:80) Ni {+=
+} ∆
A t,i 13:
Server : perform global update θ t = AGD-Update ( θ t last , J t ) (see Eq.(3) for the choice of
J t ), b t =
b t last + (cid:80) Ni =
∆ A t,i θ t , and set t last = t 14:
Clients ∀ i ∈ [ N ] : set θ t,i =
θ t , A t,i =
A t , b t,i =
b t
• Local update.
As mentioned earlier, iterative optimization over local dataset { ( x s,i , y s,i ) } s ∈ [ t ] leads to the drifting issue that pushes the updated model to the local optimum.
Due to the small size of this local dataset, the conﬁdence ellipsoid centered at the converged model has increased width, which leads to increased regret in bandit learning.
However, as we will prove in Section 4.2, completely disabling local update and restricting all clients to use the previous globally updated model for arm selection is also a bad choice, because the learning system will then need more frequent global updates to adapt to the growing dataset.
To enable local update while alleviating the drifting issue, we adopt online regression in each client, such that the local model estimation θ t,i is only updated for one step using the sample ( x t,i , y t,i ) collected at time t .
Prior works [2, 12] showed that UCB-type algorithms with online regression can attain comparable cumulative regret to the standard UCB-type algorithms [1, 20], as long as the selected online regression method guarantees logarithmic online regret.
As the negative loglikelihood loss deﬁned in Section 3.3 is exp-concave and online Newton step (ONS) is known to attain logarithmic online regret in this case [10, 12], ONS is chosen for the local update of FedGLB-UCB and its description is given in Algorithm 2.
At time step t , after client i pulls an arm x t,i ∈
A t,i and observes the reward y t,i , its model θ t − 1 ,i is immediately updated by the ONS update rule (line 9 in Algorithm 1), where ∇ l ( x (cid:62) t,i θ t − 1 ,i , y t,i ) denotes the gradient w.r.t.
θ t − 1 ,i , and A t,i denotes the covariance matrix for client i at time t .
1: Input: θ t − 1 ,i , A t,i , ∇ l ( x (cid:62) t,i θ t − 1 ,i , y t,i ) 2: θ (cid:48) t,i = θ t − 1 ,i − 1 c µ A − 1 t,i ∇ l ( x (cid:62) t,i θ t − 1 ,i , y t,i ) 3: θ t,i = arg min θ ∈B d ( S )
||
θ − θ (cid:48) t,i ||
A t,i 4: Output: θ t,i • Global update The global update of FedGLB-UCB requires communication among the N clients, which imposes communication cost in two aspects: 1) each global update for federated GLB requires multiple rounds of communication among N clients, i.e., iterative aggregation of local gradients; and 2) global update needs to be performed for multiple times over time horizon T , in order to adapt to the
growing dataset collected by each
client during bandit learning.
Consider
a particular time step t ∈ [ T ] when global update happens, the distributed optimization objective is:
[Equation]
where F t,i ( θ ) = 1 t (cid:80) ts =1 l ( x (cid:62) s,i θ, y s,i ) + λ 2 t
|| θ || 22 denotes the average regularized negative loglikelihood loss for client i ∈
[ N ] , and λ > 0 denotes the regularization parameter.
Based on
Assumption 1, { F t,i ( θ ) } i ∈ [ N ] are λNt -strongly-convex and ( k µ + λNt ) -smooth in θ (proof in Appendix
A), and we denote the unique minimizer of Eq.(2) as ˆ θ MLE t .
In this case, it is known that the number of communication rounds J t required to attain a speciﬁed sub-optimality (cid:15) t , such that
F t ( θ ) − min θ ∈ Θ F t ( θ ) ≤ (cid:15) t , has a lower bound J t
= Ω (cid:16)(cid:112) ( k µ Nt ) /λ + 1 log 1 (cid:15) t (cid:17) [3], which means
J t increases at least at the rate of √ Nt .
This lower bound is matched by the distributed version of accelerated gradient descent (AGD) [25]:
[Equation]
where the superscript ( i ) denotes the i -th iteration of AGD.
In order to minimize the number of communication rounds in one global update, AGD is chosen as the ofﬂine regression method for FedGLB-UCB, and its description is given in Algorithm 3 (subscript t is omitted for simplicity).
However, other federated/distributed optimization methods can be readily used in place of AGD, as our analysis only requires the convergence result of the adopted method.
We should note that (cid:15) t is essential to the regret-communication trade-off during the global update at time t : a larger (cid:15) t leads to a wider conﬁdence ellipsoid, which increases regret, while a smaller (cid:15) t requires more communication rounds J t , which increases communication cost.
In Section 4.2, we will discuss the proper choice of (cid:15) t to attain desired trade-off between the two conﬂicting objectives. Algorithm
1 − • ϑ ( j +1) = θ ( j ) − 1 k µ + λNt ∇ F ( θ ( j ) ) • θ ( j +1) =
(1 − γ j ) ϑ ( j +1) + γ j ϑ ( j ) : Output: arg min θ ∈B d ( S ) (cid:107) g t ( θ ( J +1) )
− g t ( θ ) (cid:107) A − 1 t To reduce the total number of global updates over time horizon T , we adopt the event-triggered communication from [28], such that global update is triggered if the following event is true for any client i ∈ [ N ] (line 8):
[Equation]
(4) where ∆ A t,i denotes client i ’s local update to its covariance matrix since last global update at t last , and D > 0 is the chosen threshold for the event-trigger.
During the global update, the model estimation θ t,i , covariance matrix A t,i and vector b t,i for all clients i ∈
[ N ] will be updated (line 11-14).
We should note that the LHS of Eq.(4) is essentially an upper bound of the cumulative regret that client i ’s locally updated model has incurred since t last .
Therefore, this event-trigger guarantees that a global update only happens when effective regret reduction is possible.
• Arm selection To balance exploration and exploitation during bandit learning, FedGLB-UCB uses the OFUL principle for arm selection [1], which requires the construction of a conﬁdence ellipsoid for each client i .
We propose a novel construction of the conﬁdence ellipsoid based on the sequence of model updates that each client i has received up to time t : basically, there are 1)
one global update at t last , i.e., the joint ofﬂine regression across all clients’ accumulated data till t last : { ( x s,i , y s,i ) } s ∈
[ t last ] ,i ∈
[ N ] , which resets all clients’ local models to θ t last ; and 2) multiple local updates from t last
+ 1 to t , i.e., the online regression on client i ’s own data sequence { ( x s,i , y s,i ) }
s ∈ [ t last +1 ,t ] to get { θ s,i } s ∈
[ t last +1 ,t ] step by step.
This can be more easily understood by the illustration in Figure 1.
The resulting conﬁdence ellipsoid is centered at the ridge regression estimator ˆ θ t,i =
A − 1 t,i b t,i [2, 12], which is computed using the predicted rewards given by the past sequence of model updates { θ t last } ∪ { θ s,i } s ∈
[ t last +1 ,t ] (see the update of b t,i in line 9 and 13 of Algorithm 1).
Then at time step t , client i selects the arm that maximizes the UCB score:
[Equation]
(5) where α t − 1 ,i is the parameter of the conﬁdence ellipsoid given in Lemma 4.2.
Note that compared with standard federated/distributed learning where clients only need to communicate gradients for joint model estimation, in our problem, due to the time-varying arm set, it is also necessary to communicate the conﬁdence ellipsoid among clients, i.e., A t ∈ R d × d and b t ∈ R d (line 14 in
Algorithm 1), as the clients need to be prepared for all possible arms x ∈ R d that may appear in future for the sake of regret minimization.
4.2 Theoretical Analysis
In this section, we construct the conﬁdence ellipsoid based on the ofﬂine-and-online estimators described in Section 4.1.
Then we analyze the cumulative regret and communication cost of FedGLBUCB, followed by theoretical comparisons with its different variants.
• Construction of conﬁdence ellipsoid Compared with prior works that convert a sequence of online regression estimators to conﬁdence ellipsoid [2, 12], our conﬁdence ellipsoid is built on the combination of an ofﬂine regression estimator θ t last for global update, and the subsequent online regression estimators { θ s − 1 ,i } s ∈
[ t last +1 ,t ] for local updates on each client i .
This construction is new and requires proof techniques unique to our proposed solution.
In the following, we highlight the key steps, and refer our readers to the appendix for details.
To simplify the use of notations, we assume without loss of generality that the global update at t last is triggered by the N -th client, such that no more new data will be collected at t last , i.e., the ﬁrst data sample obtained after the global update has index t last
+ 1 .
We start our construction by considering the following loss difference introduced by the global and local model updates:
[Equation]
where the ﬁrst term is the loss difference between the globally updated model θ t
[Equation]
and θ (cid:63) , and the second term is between the sequence of locally updated models { θ s − 1 ,i } s ∈ [ t last +1 ,t ] and θ (cid:63) .
This extends the deﬁnition of online regret used in the construction in [2, 12]; and due to the existence of ofﬂine regression, the obtained upper bounds in Lemma 4.1 are unique to our solution.
Lemma 4.1 (Upper Bound of Loss Difference) .
Denote the sub-optimality of the global model update procedure at time step t last as (cid:15) t last , such that F t last ( θ ) − min θ ∈B d ( S ) F t last ( θ ) ≤ (cid:15) t last .
Then under Assumption 1 and 2, we have
[Equation]
6) where B 1 =
Nt last (cid:15) t last + λ 2 S 2 , and with probability at least 1 − δ , t (cid:88)
[Equation]
where B 2 = 12 c µ (cid:80) ts = t last +1 (cid:107)∇ l ( x (cid:62) s,i θ s − 1 ,i , {+y s,i ) (cid:107) 
+}
[-, y s,i ) (cid:107) 
-] A − 1 s,i + c µ 2 (cid:104) 1 c µ R max (cid:112) d log (1 + Nt last c µ /dλ )
+ 2 log (1 /δ )
+ {+
+}
Nt last (cid:113) 2 k µ λc µ +
[Equation]
last c
[Equation] (cid:15) [Equation] last + (cid:113)
[Equation] respectively. Speciﬁcally, B 1 corresponds to the convergence of the ofﬂine (distributed) optimization in previous global update; B 2 is essentially the online regret upper bound of ONS, with the major difference that it is initialized using the globally updated model θ t last , instead of an arbitrary model as in standard
ONS.
Then due to the c µ -strongly-convexity of l ( z, y ) w.r.t.
z , i.e., l ( x (cid:62) s θ, y s ) − l
( x (cid:62) s θ (cid:63) , y s ) ≥ (cid:2) µ ( x (cid:62) s θ (cid:63) )
− y s (cid:3) x (cid:62) s ( θ − θ (cid:63) )
+ c µ 2 (cid:2) x (cid:62) s ( θ − θ (cid:63) ) (cid:3) 2 , and by rearranging terms in Eq.(6) and Eq.(7), we have: (cid:80) [Equation] (cid:80) Ni [Equation]
[Equation] , and [Equation] ts =
[Equation]
θ
[Equation]
) , whose LHS is quadratic in θ (cid:63) .
To further upper bound the RHS, we should note that the term 2 c µ (cid:80) ts = [Equation] 1 η [Equation]
x [Equation] i ( θ s − [Equation]
i − θ [Equation] ) is standard in [2, 12] as x (cid:62) s,i ( θ s − 1 ,i − θ (cid:63) ) is F s,i -measurable for online estimator θ s − 1 ,i .
However, this is not true for the term
[Equation] ) as the ofﬂine regression estimator θ t last depends on all data samples collected till t last ; and thus we have to develop a different approach to bound it.
This leads to Lemma 4.2 below, which provides the conﬁdence ellipsoid for θ (cid:63) .
Lemma 4.2 (Conﬁdence Ellipsoid of FedGLB-UCB) .
With probability at least 1 − 2 δ , for all t ∈ [ T ] , i ∈ [ N ] ,
[Equation]
where z t,i denotes the vector of predicted rewards [ [Equation]
i
[Equation]
. , x
[Equation]
i
[Equation]
i ]
[Equation] , and β
[Equation]
=
[Equation]
• Regret and communication cost From Lemma 4.2, we can see that α t,i grows at a rate of Nt last √ (cid:15) t last through its dependence on the B 2 term.
To make sure the growth rate of α t,i matches that in standard GLB algorithms [20, 12], we set (cid:15) t last =
[Equation]
2 last , which leads to the following corollary.
Corollary 4.2.1 (Order of β t,i ) .
With (cid:15)
[Equation]
Then using a similar argument as the proof for Theorem 4 of [28], we obtain the following upper bounds on R T and C T for FedGLB-UCB (proof in Appendix D).
Theorem 4.3 (Regret and Communication Cost Upper Bound of FedGLB-UCB) .
Under Assumption 1, 2, and by setting (cid:15) t = 1 N 2 t 2 , ∀ t and D =
T Nd log( NT ) , the cumulative regret R T has upper bound
[Equation]
with probability at least 1 − 2 δ .
The corresponding communication cost C T 1 has upper bound
[Equation]
Theorem 4.3 shows that FedGLB-UCB recovers the standard O (cid:0) d √ NT log( NT ) (cid:1) rate in regret as in the centralized setting, while only incurring a communication cost that is sub-linear in T .
Note that, to obtain O (cid:0) d √ NT log( NT ) (cid:1) regret for federated linear bandit, the DisLinUCB algorithm incurs a communication cost of O ( dN 1 . 5 log( NT ))
[28], which is smaller than that of FedGLB-UCB by a factor of √ NT log( NT ) .
As the frequency of global updates is the same for both algorithms (due to their use of the same event-trigger), this additional communication cost is caused by the iterative optimization procedure for the global update, which is required for GLB model estimation.
Moreover, as we mentioned in Section 4.1, there is not much room for improvement here as the use of AGD already matches the lower bound up to a logarithmic factor.
To facilitate the understanding of our algorithm design and investigate the impact of different components of FedGLB-UCB on its regret and communication efﬁciency trade-off, we propose [Table] and analyze three variants, which are also of independent interest, and report the results in Table 1.
Detailed descriptions, as well as proof for these results can be found in Appendix E. Note that all three variants perform global update according to a ﬁxed schedule S
= { t 1 := (cid:98) TB (cid:99) , t 2 := 2 (cid:98) TB (cid:99) , . .
. , t B := B (cid:98) TB (cid:99)} , where B denotes the total number of global updates speciﬁed in advance to trade-off between R T and C T , and these variants differ in their global and local update strategies.
This comparison demonstrates that our solution is proven to achieve a better regret-communication trade-off against these reasonable alternatives.
For example, when using standard federated learning methods (which assume ﬁxed dataset) for streaming data in real-world applications, it is a common practice to set some ﬁxed schedule to periodically retrain the global model to ﬁt the new dataset, and FedGLB-UCB 1 implements such behaviors.
The design of FedGLB-UCB 3 is motivated by distributed online convex optimization that also deals with streaming data in a distributed setting.
Experiments
We performed extensive empirical evaluations of FedGLB-UCB on both synthetic and real-world datasets, and the results (averaged over 10 runs) are reported in Figure 2.
We included the three variants of FedGLB-UCB (listed in Table 1), One-UCB-GLM, N-UCB-GLM [20] and N-ONS-GLM [12] as baselines, where One-UCB-GLM learns a shared bandit model across all clients, and NUCB-GLM and N-ONS-GLM learn a separated bandit model for each client with no communication.
Additional results and discussions about experiments can be found in Appendix F. • Synthetic Dataset
We simulated the federated GLB setting deﬁned in Section 3.3, with T = 2000 , N = 200 , d = 10 , S = 1 , A t ( K = 25 ) uniformly sampled from a (cid:96) 2 unit sphere, and reward y t,,i ∼ Bernoulli ( µ ( x (cid:62) t,,i θ (cid:63) )) , with µ ( z ) =
(1 + exp( − z )) − 1 .
To compare the algorithms’ R T and C T under different trade-off settings, we run FedGLB-UCB with different threshold value D (logarithmically spaced between 10 − 1 and 10 3 ) and its variants with different number of global updates B .
Note that each dot in the result ﬁgure illustrates the C T (x-axis) and R T (y-axis) that a particular instance of FedGLB-UCB or its variants obtained by time T , and the corresponding value for D or B is labeled next to the dot.
R T of One-UCB-GLM is illustrated as the red horizontal line, and R T of N-UCB-GLM and N-ONS-GLM are labeled on the top of the ﬁgure.
We can observe that for FedGLB-UCB and its variants, R T decreases as C T increases, interpolating between the two extreme cases: independently learned bandit models by N-UCB-GLM, N-ONS-GLM; and the jointly learned bandit model by One-UCB-GLM.
FedGLB-UCB signiﬁcantly reduces C T , while attaining low R T , i.e., its regret is even comparable with One-UCB-GLM that requires at least C T = N 2 T ( 8 × 10 7 in this simulation) for gradient aggregation at each time step.
• Real-world Dataset
The results above demonstrate the effectiveness of FedGLB-UCB when data is generated by a well-speciﬁed generalized linear model.
To evaluate its performance in a more challenging and practical scenario, we performed experiments using real-world datasets: CoverType, MagicTelescope and Mushroom from the UCI Machine Learning Repository [5].
To convert them to contextual bandit problems, we pre-processed these datasets following the steps in prior works [8], with T = 2000 and N = 20 .
Moreover, to demonstrate the advantage of GLB over linear model, we included DisLinUCB [28] as an additional baseline.
Since the parameters being communicated in DisLinUCB and FedGLB-UCB are different, to ensure a fair comparison of C T in this experiment, we measure communication cost (x-axis) by the number of integers or real numbers transferred across the learning system (instead of the frequency of communications).
Note that DisLinUCB has no C T ≥ 3 × 10 6 in Figure 2 because its global update is already happening in every round and cannot be increased further.
As mentioned earlier, due to the difference in messages being sent, the communication in DisLinUCB’s per global update is much smaller than that in FedGLB-UCB.
However, because linear models failed to capture the complicated reward mappings in these three [Figure] t i v e R e g r [Figure]
t
[Figure] synthetic and real world datasets. datasets, we can see that DisLinUCB is clearly outperformed by FedGLB-UCB and its variants. This shows that, by offering a larger variety of modeling choices, e.g., linear, Poisson, logistic regression, etc., FedGLB-UCB has more potential in dealing with the complicated data in real-world applications.
Conclusion
In this paper, we take the ﬁrst step to address the new challenges in communication efﬁcient federated bandit learning beyond linear models, where closed-form solutions do not exist, and propose a solution framework for federated GLB that employs online regression for local update and ofﬂine regression for global update.
For arm selection, we propose a novel conﬁdence ellipsoid construction based on the sequence of ofﬂine-and-online model estimations.
We rigorously prove that the proposed algorithm attains sub-linear rate for both regret and communication cost, and also analyze the impact of each component of our algorithm via theoretical comparison with different variants.
In addition, extensive empirical evaluations are performed to validate the effectiveness of our algorithm.
An important further direction of this work is the lower bound analysis for the communication cost, analogous to the communication lower bound for standard distributed optimization by Arjevani and
Shamir [3].
Moreover, in our algorithm, clients’ locally updated models are not utilized for global model update, so that another interesting direction is to investigate whether using such knowledge, e.g., by model aggregation, can further improve communication efﬁciency. Acknowledgement
This work is supported by NSF grants IIS-2128019, IIS-1838615, IIS-1553568.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] , see Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] , see Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] , the complete proof is given in appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] , in Section(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Technical Lemmas
When applying the following self-normalized bound in the analysis of federated bandit algorithm with event-trigger, a subtle difference from the analysis of standard bandit algorithm is that the sequence of data points used to update each client is controlled by the data-dependent event-trigger, e.g. Eq (4), which introduces dependencies on the future data, and thus breaks the standard argument.
This problem also exists in prior works of distributed linear bandit, but was not addressed rigorously (see
Lemma H.1.
of [28]).
Speciﬁcally, each client i observes the sequence of data points in a different order, i.e., it ﬁrst observes each newly collected local data points from the environment, and then observes (in the form of their gradients) the batch of new data points that other clients have collected at the end of the epoch.
Then, if we consider a data point that is contained in the batch of new data collected by other clients, the index of this data point (as observed by client i ) has dependency all the way to the end of this batch, i.e., its index is only determined after some client triggers the global update.
Therefore, in order to avoid this dependency on future data points, when constructing the ﬁltration, we should avoid including the σ -algebra that ‘cuts a batch in half’, but instead only include the σ -algebra generated by the sequence of data points up to the end of each batch, where we consider each locally observed data point as a batch as well.
Denote the sequence of time indices corresponding to these data points as { t p } p ∈ [ P ] for some P > 0 .
Then the constructed ﬁltration {F t p } p ∈
[ P ] is essentially a batched version of the standard {F t } ∞ t =1 .
The self-normalized bound below still holds, i.e., by changing the stopping time construction from ∪ t ≥ 1 B t ( δ ) to ∪
t ∈{ t p } p ∈
[ P ] B t ( δ ) in the proof of
Theorem 1 in [1], where B t ( δ ) denotes the bad event that the bound does not hold.
Therefore, instead of holding uniformly over all t , the self-normalized bound now only holds for all t ∈ { t p } p ∈
[ P ] , i.e., the sequence of time steps when client i gets updated, which is also what we need.
Lemma A.1 (Vector-valued self-normalized bound (Theorem 1 of [1])) .
Let {F t } ∞ t
=
1 be a ﬁltration.
Let { η t } ∞ t =1 be a real-valued stochastic process such that η t is F t +1 -measurable, and η t is conditionally zero mean R -sub-Gaussian for some R ≥ 0 .
Let { X t } ∞ t =1 be a R d -valued stochastic process such that X t is F t -measurable.
Assume that V is a d × d positive deﬁnite matrix.
For any t > 0 , deﬁne
[Equation]
Then for any δ > 0 , with probability at least 1 − δ ,
[Equation]
Lemma A.2 (Corollary 8 of [2]) .
Under the same assumptions as Lemma A.1, consider a sequence of real-valued variables { Z t } ∞ t =1 such that Z t is F t -measurable.
Then for any δ > 0 , with probability at least 1 − δ , |
[Equation]
Lemma A.3.
Under Assumption 1, F i,t ( θ ) for i = 1 , 2 , . . .
, N is smooth with constant k µ + λNt
Proof.
By Assumption 1, µ ( · ) is Lipschitz continuous with constant k µ , i.e., | µ ( x (cid:62) θ 1 ) − µ ( x (cid:62) θ 2 ) | ≤ k µ | x [Equation] ( θ 1 − θ 2 ) | .
Then we can show that
[Equation]
Therefore, ∇ F i,t ( θ ) is Lipschitz continuous with constant k µ + λNt , and ∇ F t ( θ )
=
1 N (cid:80)
Ni =1 ∇ F i,t ( θ ) is Lipschitz continuous with constant k µ + λNt as well.
Lemma A.4 (Matrix Weighted Cauchy-Schwarz) .
If A ∈ R d × d is a PSD matrix, then x T Ay ≤ (cid:112) x T Ax · y T Ay holds for any vectors x, y ∈ R d .
Proof.
Consider a quadratic function ( x + ty ) T A ( x + ty ) =
x T Ax + 2( x T Ay ) t
+ ( y T Ay ) t 2 for some variable t ∈ R , where x, y ∈ R
d are arbitrary vectors.
Since A is PSD, the value of this quadratic function ( x + ty ) T A ( x + ty ) =
x T Ax + 2( x T Ay ) t
+ ( y T Ay ) t 2 ≥ 0 , ∀ t , which means there can be at most one root.
This is equivalent to saying the discriminant of this quadratic function 4( x T Ay ) 2 − 4 x T Ax · y T Ay ≤ 0 , which ﬁnishes the proof.
Lemma A.5 (Conﬁdence Ellipsoid Centered at Global Model) .
Consider time step t ∈
[ T ] when a global update happens, such that the distributed optimization over N clients is performed to get the globally updated model θ t .
Denote the sub-optimality of the ﬁnal iteration as (cid:15) t , such that
F t
[Equation]
) ≤ (cid:15) t ; then with probability at least 1 − δ , for all t ∈ [ T ] ,
[Equation]
where α t =
Nt (cid:113) 2 k µ λc µ + [Equation] log
[Equation]
Ntc [Equation] [Equation] dλ ))
+ 2 log (1 /δ )
+ (cid:113) λc µ S , and
A [Equation] = [Equation]
Proof.
Recall that the unique minimizer of Eq.(2) is denoted as ˆ θ MLE t , so by taking gradient w.r.t.
θ we have, g t (ˆ θ MLE t ) =
(cid:80) N i =1 (cid:80) t s =1 x s,i y s,i , where we deﬁne g t ( θ ) =
λθ + (cid:80) N i =1 (cid:80) t s =1 µ ( x (cid:62) s,i θ )
x s,i .
First, we start with standard arguments [8, 20] to show that (cid:107) θ t − θ (cid:63) (cid:107)
A t ≤ 1 c µ (cid:107) g t ( θ t ) − g t ( θ (cid:63) )
(cid:107)
A − 1 t .
Speciﬁcally, by Assumption 1 and the Fundamental Theorem of Calculus, we have
[Equation]
where G t = (cid:82) 1 0 ∇ g t ( aθ t + (1 − a ) θ (cid:63) ) da .
Again by Assumption 1, ∇ g t ( θ ) =
λI + (cid:80) ts =1 (cid:80) Ni =1 x s,i x (cid:62) s,i ˙ µ ( x (cid:62) s,i θ ) is continuous, and ∇ g t ( θ ) (cid:60) λI + c µ (cid:80) ts =1 (cid:80)
Ni
=1 x s,i x (cid:62) s,i (cid:31) 0 for θ ∈ B d ( S ) , so G t (cid:31) 0 , i.e., G t is invertible.
Therefore, we have θ
[Equation]
(cid:63) [Equation]
G
[Equation]
1 t [ g t ( θ t ) − g t ( θ (cid:63) )] Note that G t (cid:60) λI + c µ (cid:80) t s =1 (cid:80) N i =1 x s,i x (cid:62) s,i = c µ A t , so G − 1 t (cid:52) 1 c µ A − 1 t .
Hence, (cid:107) θ t − θ (cid:63) (cid:107) A t = (cid:107) G − 1 t [ g t ( θ t ) − g t ( θ (cid:63) )] (cid:107) A t ≤ (cid:107) 1 c µ A − 1 t [ g t ( θ t ) − g
[Equation]
where the ﬁrst term depends on the sub-optimality of the ofﬂine regression estimator θ t to the unique minimizer ˆ θ ( MLE ) t , and the second term is standard for GLB [20].
Recall from Algorithm 3 that θ t = arg min θ ∈B d ( S ) (cid:107) g t ˜ θ t ) − g t ( θ ) (cid:107) A − 1 t , where ˜ θ t denotes the AGD estimator before projection. Therefore, for the ﬁrst term, using triangle inequality and the deﬁnition of g t ( · ) , we have
[Equation]
where the last equality is due to the deﬁnition of F t ( θ ) in Eq.(2).
We can further bound it using the property of Rayleigh quotient and the fact that A t (cid:60) λc µ I , which gives us
[Equation]
Based on Lemma A.3, F t ( θ ) is ( k µ + λNt ) -smooth, which means
[Equation]
where the second inequality is by deﬁnition of (cid:15) t .
Putting everything together, we have the following bound for the ﬁrst term
[Equation]
For the second term, similarly, based on the deﬁnition of g t ( · ) , we have
[Equation]
Then based on the self-normalized bound in Lemma A.1 (Theorem 1 of [1]), we have
[Equation] [Equation]
t , with probability at least 1 − δ .
Substituting the upper bounds for these two terms back into Eq.(8), we have, with probability at least 1 − δ ,
[Equation]
which ﬁnishes the proof.
B Proof of Lemma 4.
Proof.
Denote the two terms for loss difference as A 1 = (cid:80) t last s =1 (cid:80) Ni =1 (cid:2) l ( x (cid:62) s,i θ t last , y s,i ) − l ( x (cid:62) s,i θ (cid:63) , y s,i ) (cid:3) , and A 2 = (cid:80) ts = t last +1 (cid:2) l ( x {+(cid:
 )+} s,i θ s − 1 ,i , y s,i ) − l ( x (cid:62) s,i θ (cid:63) , y s,i ) (cid:3) .
We can upper bound the term A 1 by
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the ﬁrst inequality is because ˆ θ MLE t last minimizes Eq.(2)
[Table]
such that (cid:80) t last s =
[Equation]
l ( x [Equation]
[Table]
[Equation]
y
[Equation]
+
[Table]
[Equation]
[Table]
(cid:62) s [Equation] θ, y
[Equation]
+
[Equation]
θ
[Equation]
22 for any θ ∈ B d ( S ) , and the last inequality is because
[Equation]
deﬁnition.
Now we start with standard arguments [12, 29] in order to bound the term A 2 , which is essentially the online regret of ONS, except that its initial model is the globally updated model θ t last .
First, since l ( z, y ) is c µ -strongly-convex w.r.t.
z , we have
[Equation]
[Equation]
To further bound the RHS of Eq.(9), recall from the ONS local update rule in Algorithm 2 that, for each client i ∈
[ N ] at the end of each time step s ∈ [ t last + 1 , t ] ,
[Equation]
Then due to the property of generalized projection (Lemma 8 of [10]), we have
[Equation]
By rearranging terms
[Equation]
we have
[Equation]
Note that ∇ l ( x (cid:62) s,i θ s − 1 ,i , y s,i ) =
x s,i [ µ ( x (cid:62) s,i θ s − 1 ,i ) − y s,i ] , so with the inequality above
[Equation]
we can further bound the RHS of Eq.(9):
[Equation]
Then summing over s ∈ [ t last + 1 , t ] , we have
[Equation]
where A t last ,i =
A t last , θ t last ,i = θ t last , ∀ i ∈
[ N ] due to the global update (line 15 in Algorithm 1).
We should note that the second term above itself essentially corresponds to a conﬁdence ellipsoid centered at the globally updated model θ t last , and its appearance in the upper bound for the loss difference (online regret) of local updates is because the local update is initialized by θ t last .
And based on Lemma A.5, with probability at least 1 − δ ,
[Equation]
Therefore, with probability at least 1 − δ ,
[Equation]
which ﬁnishes the proof for Lemma 4.1.
C Proof of Lemma 4.2 and Corollary 4.2.
Proof of Lemma 4.2.
Due to c µ -strongly convexity of l ( z, y ) w.r.t.
z , we have l ( x (cid:62) s,i θ, y s,i ) − l ( x (cid:62) s,i θ [Equation] , y s [Equation]
i [Equation] ≥ [Equation]
θ [Equation] θ [Equation]
) (cid:3) 2 .
Substituting this to the LHS of Eq.(6) and Eq.(7), we have
[Equation]
By rearranging the terms
[Equation]
we have
[Equation]
where the LHS is quadratic in θ (cid:63) .
For the RHS [Equation] we will further upper bound the second term as shown below.
• Upper Bound for (cid:80) ts = t last + [Table] x
[Table]
s,i
[Table]
θ s − 1 ,i − θ
[Table]
) (cid:3) 2 Note that x (cid:62) s,i
θ s − 1 ,i − θ (cid:63) ) is F s,imeasurable, and η s,i is F s +1 ,i -
measurable and conditionally R max sub
Gaussian
By apply-
ing Lemma A (Corollary of [2])
w.
r t client i ’s ﬁltration { F s,i } ∞
s = t last + , where F s,i = σ [Table]
[Table] η [Table] and taking union bound over all i ∈
[ N ] , with probability at least 1 − δ , for all t ∈
[ T ] , i ∈
[ N ] ,
[Table]
Therefore,
1 + t (cid:88) s
=
t
[Equation]
Then by applying Lemma 2 of [12], i.e., if q≤ a + fq (cid:113) log( q δ/N ) then q≤ 2 a + f 2 log( √ 4 a + f 4 / (4 δ 2 ) δ/N ) (for a, f ≥ 0 , q ≥ 1 ). And by setting q =
[Equation]
1 +
[Equation]
=
[Equation]
, we have
[Equation]
with probability at least 1 − δ .
• Upper Bound for (cid:80)
[Equation]
2 Note that θ t last depends on
all
data samples in { ( x s,i , y s,i ) } s ∈
[ t last ] as a result of the ofﬂine regression method, and therefore x (cid:62) s i ( θ t last − θ (cid:63) ) is no longer F s,i -measurable for s ∈ [1 , t last ) .
Hence, we cannot use Lemma A.2 as before.
Instead, we have
[Equation]
[Table]
with probability at least 1 − δ
[Table]
where the ﬁrst inequality is due to the matrix-weighted CauchySchwarz inequality in Lemma A.4, such that x (cid:62) A − 1 Ay ≤ (cid:112) x
(cid:62) A − 1 x · y (cid:62) A (cid:62) A {+− 
+}
Ay = (cid:112) x
(cid:62) A − 1 x · y (cid:62)
Ay for symmetric PD matrix A , and the second inequality is obtained by applying the self-normalized bound in Lemma A.1 w.r.t.
the ﬁltration {F s } s ∈{ t p } Bp =1 , where
F s = σ (cid:0) [ x k,j , η k,j ] k,j : k ≤ s − 1 ∩ j ≤ N , [ x k,j , η k,j ] k,j : k = s ∩ j ≤ N − 1 , x s,N (cid:1) and { t p } Bp =1 denotes the se- quence of time steps when global update happens, and B denotes the total number of global updates.
By substituting it back, we have
[Equation]
[Equation]
Then by applying the Proposition 9 of [2], i.e. if z 2 ≤ a + bz then z ≤ b + √ a (for a, b ≥ 0 ), and setting z = (cid:113)(cid:80)
[-x (cid:62) s,i ( θ t last − θ (cid:63) ) (cid:3) 
 + (cid:107) θ t last − θ (cid:63) (cid:107) 22 , a = (cid:107) θ t last − θ (cid:63) (cid:107) 22 + 2 c µ B 1 , b =2 R max c µ (cid:114) 2 log (cid:0) 1 δ (cid:113) det ( I + (cid:80) t last s =1 (cid:80) Ni =1 x s,i x (cid:62) s,i ) (cid:1) ,we have(cid:118)(cid:117)(cid:117)-][Equation]
[-(cid:116) t last (cid:88) s = N (cid:88) i = (cid:2) x (cid:62) s,i ( θ t last − θ (cid:63) ) (cid:3) 
 + (cid:107) θ t last − θ (cid:63) (cid:107) 2≤ 2 R max c µ (cid:118)(cid:117)(cid:117)(cid:117)(cid:116) 2 log (cid:0)δ (cid:118)(cid:117)(cid:117)(cid:116) det( I + t last (cid:88) s = N (cid:88) i = x s,i x (cid:62) s,i ) (cid:1) + (cid:113) (cid:107) θ t last − θ (cid:63) (cid:107) 22 + B(13)-][Table]
Taking square on both sides, and rearranging terms, we have
[Table]
[Equation]
Now putting everything together
[Equation]
we have the following conﬁdence region for θ (cid:63) ,
[Equation]
where
[Equation]
Denote X t,
[Equation]
can rewrite the inequality above as
[Equation]
where ˆ θ t,i =
A − [Equation] ,i X (cid:62) t,i z t,i denotes the Ridge regression estimator based on the predicted rewards given by the past sequence of model updates, and the regularization parameter is λc µ .
Note that by expanding ˆ θ t,i , we can show ˆ θ (cid:62) t,i
A i,t ˆ θ t,i =
z (cid:62) i,t X i,t ˆ θ t,i , and ˆ θ (cid:62) t,i
A i,t θ (cid:63) =
z (cid:62) i,t X i,t θ (cid:63) .
Therefore, we have
[Equation]
which ﬁnishes the proof of Lemma 4.2.
Proof of Corollary 4.2.1.
Under the condition that (cid:15) t last ≤
[Equation]
+
[Equation]
Note that ∇ l ( x (cid:62) s,i θ s − 1 ,i , y s,i ) =
x s,i [ µ ( x (cid:62) s,i θ s − 1 ,i ) − y s,i ] .
We can upper bound the squared prediction error by
[Equation]
where the ﬁrst inequality is due to AM-QM inequality, and the second inequality is due to the k µLipschitz continuity of µ ( · ) according to Assumption 1.
Since | η s,i | ≤ R max , (cid:2) µ ( x (cid:62) s,i θ s − 1 ,i ) − y s,i (cid:3) 2 ≤ k 2 µ S 2 + R 2max .
In addition, due to Lemma 11 of [1], i.e., (cid:80) ts = t last +1 (cid:107) [Equation] [Table] [Equation]
log
[Equation]
i ) det( λI ) ) Therefore,
[Equation]
so B [Equation] = O
[Equation] + R [Equation] ] (cid:1) .
Hence,
[Equation]
which ﬁnishes the proof
[Equation]
D Proof of Theorem 4.
Proof.
Since µ ( · ) is k µ -Lipschitz continuous [Equation] we have µ ( x [Equation]
t, [Equation]
Then we have the following upper bound on the instantaneous regret,
[Equation]
(cid:107)
A t − 1 ,i + (cid:107) x t,i (cid:107) A − 1 t − 1 ,i (cid:107) ˆ θ t − 1 ,i − θ (cid:63) (cid:107) A t − 1 ,i ≤ 2 α t − 1 ,i · (cid:107) x t,i (cid:107) A − 1 t − 1 ,i which holds for all i ∈
[ N ] , t ∈ [ T ] , with probability at least 1 − 2 δ .
And ˜ θ t − 1 .i denotes the optimistic estimate in the conﬁdence ellipsoid that maximizes the UCB score when client i selects arm at time step t .
Now consider an imaginary centralized agent that has direct access to all clients’ data, and we denote its covariance matrix as ˜
A t,i = λc µ I + (cid:80) t − 1 s =1 (cid:80) Nj =1 x s,j x s,j + (cid:80) ij =1 x t,j x (cid:62) t,j , i.e., ˜
A t,i is immediately updated after any client obtains a new data sample from the environment.
Then we can obtain the following upper bound for r t,i , which is dependent on the determinant ratio between the covariance matrix of the imaginary centralized agent and that of client i , i.e., det( ˜ A t − 1 ,i ) / det( A t − 1 ,i ) .
[Equation]
We refer to the time period in-between two consecutive global updates as an epoch, and denote the total number of epochs as B ∈ R , i.e., the p -th epoch refers to the period from t p −
+ 1 to t p , for p ∈ [ B ] , where t p denotes the time step when the p -th global update happens.
Then the p -th epoch is called a ‘good’ epoch if the determinant ratio det( A tp ) det( A t p − 1 ) ≤ 2 , where A t p is the aggregated sufﬁcient statistics computed at the p -th global update.
Otherwise, it is called a ‘bad’ epoch.
In the following, we bound the cumulative regret in ‘good’ and ‘bad’ epochs separately.
Suppose the p -th epoch is a good epoch, then for any client i ∈ [ N ] , and time step t ∈
[ t p − 1 + 1 , t p ] , we have det( ˜ A t − 1 ,i ) det( A t − 1 ,i ) ≤ det( A tp ) det( A
tp − 1 ) ≤ 2 , because A t − 1 ,i (cid:60)
A t p − 1 and ˜
A t − 1 ,i (cid:52) A t p .
Therefore, the instantaneous regret incurred by any client i at any time step t of a good epoch can be bounded by
[Equation]
with probability at least 1 − 2 δ .
Therefore, using standard arguments for UCB-type algorithms, e.g.,
Theorem 2 in [20], the cumulative regret for all the ‘good epochs’ is
[Equation]
which matches the regret upper bound of GLOC [12].
Now suppose the p -th epoch is bad.
Then the cumulative regret incurred by all N clients during this ‘bad epoch’ can be upper bounded by:
[Equation]
where the last inequality is due to the event-trigger design in Algorithm 1.
Following the same argument as [28], there can be at most R = (cid:100)
d log (1 + NTc µ λd ) (cid:101) =
O (cid:0) d log( NT ) (cid:1) ‘bad epochs’, because det( A t B ) ≤ det( ˜ A T,N ) ≤ ( λc µ + NTd ) d .
Therefore, the cumulative regret for all the ‘bad epochs’ is
[Equation]
Combining the regret upper bound for ‘good’ and ‘bad’ epochs, the cumulative regret
[Equation]
) (cid:19) .
To obtain upper bound for the communication cost C T , we ﬁrst upper bound the total number of epochs B .
Denote the length of an epoch, i.e., the number of time steps between two consecutive global updates, as α > 0 , so that there can be at most (cid:100)
Tα (cid:101) epochs with length longer than α .
For a particular epoch p with less than α time steps, we have t p − t p − 1 < α .
Moreover, due to the event-trigger design in Algorithm 1, we have ( t p − t p − 1 ) log det( A tp ) det( A tp − 1 ) >
D , which means log det( A tp ) det( A tp − 1 ) >
Dα .
Since (cid:80) Bp =1 log det( A tp ) det( A tp − 1 ) ≤ R , the number of epochs with less than α time steps is at most (cid:100) RαD (cid:101) .
Therefore, the total number of epochs.
[Equation]
which is minimized it by choosing α = (cid:113) DTR , so B ≤ (cid:113) TRD
=
O ( d 0 . 5 log 0 .
5 ( NT ) T 0 .
5 D − 0 . 5 ) .
At the end of each epoch, FedGLB-UCB has a global update step that executes AGD among all N clients.
As mentioned in Section 4.1, the number of iterations required by AGD has upper bound
J t ≤ 1 +
[Equation]
[Equation]
t −
[Equation]
MLE t (cid:107)2 (cid:15) t , and under the condition that (cid:15) t = 1 N 2 t 2 , ∀ t ∈
[ T ] , we have J t =
O (cid:0) √ NT log( NT ) (cid:1) , ∀ t ∈
[ T ] .
Moreover, each iteration of AGD involves communication with N clients, so the communication cost
[Equation]
In order to match the regret under centralized setting, we set the threshold D =
T Nd log( NT ) , which gives us R T
= O ( k µ ( k µ + R max )
c µ d √ NT log( NT )) , and C T
= O ( dN 2 √ T log 2 ( NT )) .
E Theoretical Analysis for Variants of FedGLB-UCB
In this section, we describe and analyze the variants of FedGLB-UCB listed in Table 1.
The ﬁrst variant, FedGLB-UCB 1 , completely disables local update, and we can see that it requires a linear communication cost in T to attain the O ( d √ NT log( NT )) regret.
As we mentioned in Section 4.1, this is because in the absence of local update, FedGLB-UCB 1 requires more frequent global updates, i.e., √ NT in total, to control the sub-optimality of the employed bandit model w.r.t the growing training set.
The second variant, denoted as FedGLB-UCB 2 , is exactly the same as FedGLB-UCB, except for its ﬁxed communication schedule.
This leads to additional d √ N global updates, as ﬁxed update schedule cannot adapt to the actual quality of collected data.
The third variant, denoted as
FedGLB-UCB 3 , uses ONS for both local and global update, such that only one round of gradient aggregation among N clients is performed for each global update, i.e., lazy ONS update over batched data.
It incurs the least communication cost among all variants, but its regret grows at a rate of ( NT ) 3 / 4 due to the inferior quality of its lazy ONS update.
E.1 FedGLB-UCB 1 : scheduled communication + no local update
Though many real-world applications are online problems in nature, i.e., the clients continuously collect new data samples from the users, standard federated/distributed learning methods do not provide a principled solution to adapt to the growing datasets.
A common practice is to manually set a ﬁxed global update schedule in advance, i.e., periodically update and deploy the model.
To demonstrate the advantage of FedGLB-UCB over this straightforward solution, we present and analyze the ﬁrst variant FedGLB-UCB 1 , which completely disables local update, and performs global update according to a ﬁxed schedule S
= { t 1 := (cid:98) TB (cid:99) , [Equation] = [Equation]
(cid:98) TB (cid:99)
[Equation]
[Equation] [Equation] (cid:98) TB (cid:99)} , where B is the total number of global updates up to time step T .
The description of FedGLB-UCB 1 is presented in Algorithm 4.
Initialize ∀ i ∈
N ] : θ 0 ,i = 0 ∈ R d , A 0 ,i = λc µ I ∈ R d × d , X 0 ,i = 0 ∈ R 0 × d , y 0 ,i = 0 ∈ R 0 , t last =3: for t = 1 , 2 , ..., T do 4: for client i = 1 , 2 , ..., N do 5:
Observe arm set A t,i for client i 6:
Select arm x t,i ∈
A t,i according to Eq.
and observe reward y t,i 7:
Update client i : X t,i =
(cid:20)
X t − 1 ,i x (cid:62) t,i (cid:21) , y t,i = (cid:20)
y t − 1 ,i y t,i (cid:21) 8: if t / ∈ S then 9:
Clients : set θ t,i
= θ t − 1 ,i , A t,i =
A t − 1 ,i , ∀ i ∈
[ N ] 10: else 11:
Clients : send { X (cid:62) t,i X t,i } i ∈
[ N ] to server 12:
Server compute A t = λc µ I + (cid:80) Ni =1 X (cid:62) t,i X t,i and send A t to all clients.
Clients : set A t,i =
A t , for i ∈ [ N ] 14:
Server update global model θ t = AGD-Update ( θ t last , J t ) , and set t last = t 15:
Clients set local models θ t,i
= θ t , ∀ i ∈ [ N ]
In FedGLB-UCB 1 , each client stores a local model θ t − 1 ,i , and the corresponding covariance matrix A t − 1 ,i . Note that { θ t − 1 ,i , A t − 1 ,i } i ∈ [ N ] are only updated at time steps t ∈ S , and remain unchanged for t / ∈ S .
At time t , client i selects the arm that maximizes the following UCB score:
[Equation]
(16) where α t − 1 ,i is given in Lemma A.5.
The regret and communication cost of FedGLB-UCB 1 is given in the following theorem.
Theorem E.1 (Regret and Communication Cost Upper Bound of FedGLB-UCB 1 ) .
Under the condition that (cid:15) t = 1 N 2 t 2 , and the total number of global synchronizations B = √ NT , the cumulative regret R T has upper bound
[Equation]
with probability at least 1 − δ .
The cumulative communication cost has upper bound
[Equation]
Proof.
First, based on Lemma A.5 and under the condition that (cid:15) t = 1 N 2 t 2 , we have
[Equation]
holds ∀ t , where α t = (cid:113)
[Equation]
O
[Equation]
d log
[Equation]
Nt )) , which matches the order in [20].
Similar to the proof of Theorem 4.3, we decompose all B epochs into ‘good’ and ‘bad’ epochs according to the log-determinant ratio: the p -th epoch, for p ∈
[ B ] , is a ‘good’ epoch if the determinant ratio det( A tp ) det( A tp − 1 ) ≤ 2 .
Otherwise, it is a ‘bad’ epoch.
In the following, we bound the cumulative regret in ‘good’ and ‘bad’ epochs separately.
Suppose epoch p is a good epoch, then for any client i ∈ [ N ] , and time step t ∈
[ t p − 1 + 1 , t p ] , we have det( ˜ A t − 1 ,i ) det( A t − 1 ,i ) ≤ det( A tp ) det( A tp − 1 ) ≤ 2 , because A t − 1 ,i =
A t p − 1 and ˜
A t − 1 ,i (cid:52) A t p .
Therefore, the instantaneous regret incurred by any client i at any time step t of a good epoch p can be bounded by
[Equation]
[Table] [Equation] [Table] By standard arguments [1, 20], the cumulative regret incurred in all good epochs can be upper bounded by O ( k µ R max c µ d √ NT log( NT/δ )) with probability at least 1 − δ .
By Assumption 1, µ ( · ) is Lipschitz continuous with constant k µ , i.e., | µ ( x (cid:62) θ 1 ) − µ ( x (cid:62) θ 2 ) | ≤ k µ | x (cid:62) ( θ 1 − θ 2 ) | , so the instantaneous regret r t,i is uniformly bounded ∀ t ∈
[ T ] , i ∈
[ N ] by 2 k µ S .
Now suppose epoch p is bad, then we can upper bound the cumulative regret in this bad epoch by 2 k µ S NTB , where NTB is the number of time steps in each epoch.
Since there can be at most
O ( d log NT ) bad epochs, the cumulative regret incurred in all bad epochs can be upper bounded by
O ( NT B k µ Sd log( NT )) .
Combining both parts together, the cumulative regret upper bound is
[Equation]
To recover the regret under centralized setting, we set B = √ NT , so
[Equation]
Note that FedGLB-UCB 1 has B = √ NT global updates in total, and during each global update, there are J t rounds of communications, for t ∈ S .
As mentioned earlier, for AGD to attain (cid:15) t = 1 N 2 t 2 sub-optimality, the required number of inner iterations
[Equation]
Therefore, the communication cost over time horizon T is
[Equation]
which ﬁnishes the proof.
E.2 FedGLB-UCB 2 : scheduled communication
For the second variant FedGLB-UCB 2 , we enabled local update on top of FedGLB-UCB 1 .
Therefore, compared with the original algorithm FedGLB-UCB, the only difference is that FedGLB-UCBuses scheduled communication instead of event-triggered communication.
Its description is given in
Algorithm 5.
Algorithm i , ∆ A t,i =
∆ A t − 1 ,i + x t,i x (cid:62) t,i 8: if t / ∈ S then 9:
Clients ∀ i ∈ [ N ] : θ t,i = ONS-Update ( θ t − 1 ,i , A t,i , ∇ l ( x (cid:62) t,i θ t − 1 ,i , y t,i )) , b t,i =
b t − 1 ,i + x t,i x (cid:62) t,i θ t − 1 ,i 10: else 11:
Clients ∀ i ∈ [ N ] : send ∆ A t,i to server, and reset ∆ A t,i =12:
Server compute A t =
A t last + (cid:80) Ni =
∆
A t,i 13:
Server perform global model update θ t = AGD-Update ( θ t last , J t ) (see Eq.(3) for choice of
J t ), b t =
b t last + (cid:80) {+Ni =
+} ∆ A t,i θ t , and set t last = t 14:
Clients ∀ i ∈ [ N ] : set θ t,i =
θ t , A t,i =
A t , b t,i =
b t
The regret and communication cost of FedGLB-UCB 2 is given in the following theorem.
Theorem E.2 (Regret and Communication Cost Upper Bound of FedGLB-UCB 2 ) .
Under the condition that (cid:15) t = 1 N 2 t 2 , and the total number of global synchronizations B = d 2 N log( NT ) , the cumulative regret R T has upper bound
[Equation]
with probability at least 1 − δ .
The cumulative communication cost has upper bound [Equation] Proof. Compared with the analysis for FedGLB-UCB, the main difference in the analysis for FedGLBUCB 2 is how we bound the regret incurred in ‘bad epochs’.
Using the same argument, the cumulative regret for the ‘good epochs’ is REG good
= O ( k µ ( k µ + R max )
c µ d √ NT log NT/δ ) .
Now consider a particular bad epoch
p ∈
[ B ] .
Then the cumulative regret incurred by all N clients during this ‘bad epoch’ can be upper bounded by:
[Equation]
where the last inequality is because all epochs has length TB as deﬁned by S .
Again, since there can be at most O ( d log NT ) ‘bad epochs’, the cumulative regret for the ‘bad epochs’ is upper bounded by
[Equation]
Combining the cumulative regret for both ‘good’ and ‘bad’ epochs, and setting B = d 2 N log( NT ) , we have
[Equation]
Now that FedGLB-UCB 2 has B = d 2 N log( NT ) global updates in total, and during each global update, there are J t
=
O ( √ NT log( NT )) rounds of communications, for t ∈ S .
Therefore, the communication cost over time horizon T is
[Equation]
which ﬁnishes the proof.
E.3 FedGLB-UCB 3 : scheduled communication + ONS for global update
The previous two variants both adopt iterative optimization method, i.e., AGD, for the global update, which introduces a √ NT log( NT ) factor in the communication cost.
In this section, we try to avoid this by studying the third variant FedGLB-UCB 3 that adopts ONS for both local and global update, such that only one step of ONS is performed (based on all new data samples N clients collected in this epoch).
It can be viewed as the ONS-GLM algorithm [12] with lazy batch update.
Recall that the update schedule is denoted as S
= { t 1 := (cid:98) TB (cid:99) , t 2 := 2 (cid:98) TB (cid:99) , . . .
, t q := q (cid:98) TB (cid:99) , . .
. , t B := B (cid:98) TB (cid:99)} , where B denotes the total number of global updates up to T .
Compared with [12], the main difference in our construction is that the loss function in the online regression problem may contain multiple data samples, i.e., for global update, or one single data sample, i.e., for local update.
Then for a client i ∈ [ N ] at time step t (suppose t is in the ( q + 1) -th epoch, so t ∈ [ t q + 1 , t q +1 ] ), the sequence of loss functions observed by the online regression estimator till time t is: t [Equation] s = N
[Equation]
[Table]
We can see that the ﬁrst q terms correspond to the global ONS updates that are computed using the whole batch of data collected by N clients in each epoch, and the remaining t − t q terms are local
1: Input: communication schedule S , regularization parameter λ > 0 , δ ∈ (0 , 1) and c µ 2: Initialize ∀ i ∈
[ N ] : θ 0 ,i = 0 ∈ R d , A 0 ,i = λ
I ∈ R
d × d , V 0 ,i = λ I ∈ R d × d , b 0 ,i = {+
 ∈ R+}
d ; θ
0 = 0 ∈ R d , A 0 = λ
I ∈ R
{+d × d , V 
+}
= λ
I ∈ R
d × d , b 0 = 0 ∈ R d , t last =3: for t = 1 , 2 , ..., T do 4: for client i = 1 , 2 , ..., N do 5:
Observe arm set A t,i for client i ∈ [ N ] 6:
Select arm x t,i = arg max x ∈A t,i x (cid:62) ˆ θ t − 1 ,i + α t − 1 ,i (cid:107) x (cid:107) V − 1 t − 1 ,i , where ˆ θ t − 1 ,i =
V − 1 t − 1 ,i b t − 1 ,i and α t − 1 ,i is given in Lemma E.4; and then observe reward y t,i 7:
Compute loss l ( z t,i , y t,i ) , where z t,i = x (cid:62) t,i θ t − 1 ,i 8:
Update client i :
A t,i = A t − 1 ,i + ∇ l ( z t,i , y t,i ) ∇ l ( z t,i , y t,i ) (cid:62) , V t,i = V t − 1 ,i + x t,i x (cid:62) t,i 9: if t / ∈ S then 10:
Clients ∀ i ∈ [ N ] : θ t,i = ONS-Update ( θ t − 1 ,i , A t,i , ∇ l ( z t,i , y t,i )) , b t,i =
b t − 1 ,i + x t,i z t,i 11: else 12:
Clients ∀ i ∈ [ N ] : send gradient ∇ F t,i ( θ t last ) =
(cid:80) ts = t last +1 ∇ l ( x (cid:62) s,
i θ t last , y s,i ) and ∆ V t,i =
V t,i − V t last ,i to server 13:
Server A t =
A t last + ( (cid:80) Ni =1 ∇ F t,i ( θ t last ))
( (cid:80) Ni =1 ∇ F t,i ( θ t last )) (cid:62) , V t =
V t last + (cid:80) Ni =1 ∆ V t,i , b t = b t last + (cid:80) Ni =1 ∆ V t,i θ t last , θ t = ONS-Update ( θ t last , A t , (cid:80) N i =1 ∇ F t,i ( θ t last )) 14:
Clients ∀ i ∈ [ N ] : θ t,i = θ t , A t,i =
A t , V t =
V t , b t,i =
b t 15:
Set t last = t
ONS updates that are computed using each new data sample collected by client i in the ( q + 1) -th epoch.
To facilitate further analysis, we introduce a new set of indices for the data samples, so that we can unify the notations for the loss functions above.
Imagine all the arm pulls are performed by an imaginary centralized agent, such that, in each time step t ∈ [ T ] , it pulls an arm for clients 1 , 2 , . . .
, N one by one.
Therefore, the sequence of data sample obtained by this imaginary agent can be denoted as ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x s , y s ) , . . .
, ( x NT , y NT ) .
Moreover, we denote n p as the total number of data samples collected by all N clients till the p -th ONS update (including both global and local ONS update), and denote the updated model as θ p , for p ∈
[ P ] .
Note that P denotes the total number of updates up to time t (total number of terms in the sequence above), such that P = q + t − t q .
Then this sequence of loss functions can be rewritten as:
[Equation]
where F p ( θ p − 1 ) = (cid:80)
n p s = n p −
1 + l ( x (cid:62) s θ p − 1 , y s ) , for p ∈
P
• Online regret upper bound for lazily-updated ONS To construct the conﬁdence ellipsoid based on this sequence of global and local ONS updates, we ﬁrst need to upper bound the online regret that ONS incurs on this sequence of loss functions, which is given in Lemma E.3.
Lemma E.3 (Online regret upper bound) .
Under the condition that the learning rate of ONS is set to γ = 12 min(
[Equation]
, then the cumulative online regret over P steps
[Equation]
where B P = 12 γ (cid:80)
Pp =1 ||∇ F p ( θ p − 1 )
||
2 A − 1 p + 2 γλS 2 .
Proof of Lemma E.3.
Recall from the proof of Corollary 4.2.1 that | µ ( x (cid:62) s θ p − 1 ) − y s | ≤ (cid:113) k
2 µ S 2 + R 2max :
= G, ∀ s .
First, we need to show that F p ( θ p − 1 )
=
(cid:80) n p s = n p − 1 +1 l ( x (cid:62) s θ p − 1 , y s ) is
[Equation]
(Lemma 4.2 in [9]).
Taking ﬁrst and second order derivative of F p ( θ p − 1 ) w.r.t.
θ p − 1 , we have ∇
[Equation]
where
[Equation]
∈
[Equation]
, and y p = [ y n p − 1 +1 , y n p − 1 +2 , . . , y n p ] (cid:62) ∈ R n p − n p − 1 .
Then due to Assumption 1, we have ∇ 2 F p ( θ p − 1 ) (cid:60) c µ (cid:80) n p s = n p − 1 +1 x s x (cid:62) s =
c µ X (cid:62) p X p .
For any vector u ∈ R d , we can show that,
[Equation]
where the ﬁrst inequality is due to Cauchy-Schwarz inequality [Equation] and the second inequality is because
[Equation]
µ
[Equation]
G 2 .
Therefore, X
[Equation]
which gives us
[Equation]
Then due to Lemma 4.3 of [9] [Equation] under the condition that γ p ≤ 12 min( 1 4 GS ,
[Equation]
[Equation]
[Equation]
we have
[Equation]
Then we start to upper bound the RHS of the inequality above.
Recall that the ONS update rule is:
[Equation]
where A p
[Equation]
, and γ is set to min p ∈
[ P ] γ p = 12 min( 1 4 GS ,
[Equation]
G
[Equation]
p ∈ [ P ]
[Equation]
n p − n p − 1 ) ) .
So we have
[Equation]
Then due to the property of the generalized projection, and by substituting into the update rule, we have
[Equation]
By rearranging terms,
[Equation]
After summing over P steps, we have P
[Equation]
The second term can be simpliﬁed,
[Equation]
which leads to
[Equation]
By rearranging terms, we have
[Equation]
Combining with Eq.(17) [Equation] we obtain the following upper bound for the P -step online regret
[Equation]
where A p = (cid:80) pρ =1 ∇ F ρ ( θ ρ − 1 ) ∇ F ρ ( θ ρ − 1 ) (cid:62) .
Corollary
E.3.
(Order of
B P ) .
Under the condition that γ =
[Equation]
the online regret upper bound
[Equation]
Proof of Corollary E.3.1.
Recall that A p = (cid:80) pρ =1 ∇ F
[Equation]
Therefore, we have
P (cid:88)
[Equation]
where the ﬁrst inequality is due to Lemma 11 of [1], and the second due to the determinanttrace inequality (Lemma 10 of [1]), i.e., det( λI + (cid:80) Pp
[Equation]
) ≤
[Equation]
(cid:1)
[Equation]
Also note that ∇ F p ( θ p − 1 ) =
[Equation]
y s (cid:3) , so we have
[Equation]
where the second inequality is due to Jensen’s inequality and the assumption that (cid:107) x s (cid:107) ≤ 1 , ∀ s .
Substituting this back gives us
[Equation]
where the equality is because max [Equation] dominates [Equation]
12 min(
[Equation]
GS ,
[Equation]
• Construct Conﬁdence Ellipsoid for FedGLB-UCB
With the online regret bound B P in Lemma E.3, the steps to construct the conﬁdence ellipsoid largely follows that of Theorem 1 in [12], with the main difference in our batch update.
We include the full proof here for the sake of completeness.
Lemma E.4 (Conﬁdence Ellipsoid for FedGLB-UCB 3 ) .
Under the condition that the learning rate of ONS γ = 12 min(
[Equation]
, we have ∀ t ∈
[ T ] , i ∈
[ N ]
[Equation]
with probability at least 1 − δ .
Proof of Lemma E.4.
Due to c µ -strongly convexity of l ( z, y ) w.r.t.
z , we have l ( x (cid:62) s θ p − 1 , y s ) − l
[Equation]
x
[Equation]
[Equation]
1 − θ (cid:63) ) (cid:3) 2 .
Therefore,
[Equation]
where η s is the R -sub-Gaussian noise in the reward y s .
Summing over P steps we have
[Equation]
[Table]
By rearranging terms, we have
[Equation]
Then as x
[Equation]
s ( θ p
[Equation]
1 − θ (cid:63) ) for s ∈ [ n p − 1 + 1 , n p ] is F s -measurable for lazily updated online estimator θ p − 1 , we can use Corollary 8 from [2], which leads to
[Equation]
Then we have
[Equation]
Then by applying Lemma 2 from [12], we have
[Equation]
Therefore
[Equation]
we have the following conﬁdence ellipsoid (regularized with parameter λ ):
[Equation]
And this can be rewritten as a ellipsoid centered at ridge regression estimator ˆ θ t,i =
V − 1 t,i b t,i , where V t,i =
λI + (cid:80)
Pp =1 (cid:80) n p s = n p −
+1 x s x (cid:62) s and b t,
i = (cid:80)
Pp =1 (cid:80) n p s = n p −
+
x
s z s (recall that ONS’s prediction at time s is denoted as z s = x (cid:62) s θ p − 1 ),
i.e., ∀ t ∈ [ T ] (cid:107) θ (cid:63) − ˆ θ t,i (cid:107) 2 V t,i ≤ λS 2 + 1 + 4 c µ [Equation] P [Equation]
R 2 max
[Equation]
[Equation]
(cid:115)
[Equation]
max
[Equation]
µ · 4 δ 2 )
[Equation]
+ ˆ θ (cid:62) t,i b t,i − n P (cid:88) s =
z 2 s with probability at least 1 − δ .
Then taking union bound over all N clients, we have, ∀ t ∈
[ T ] , i ∈
[ N ] (cid:107) θ (cid:63) − ˆ θ t,i (cid:107) 2 V t,i ≤ λS 2 + 1 + 4 c µ B P + 8 R 2 max c [Equation] log ( [Equation]
(cid:115)
[Equation]
max
[Equation]
µ ·
[Equation]
δ
[Equation]
[Equation]
θ (cid:62) t,i b t,i − n P (cid:88) s =
z 2 s with probability at least 1 − δ .
• Regret and Communication Upper Bounds for FedGLB-UCB
The regret and communication cost of FedGLB-UCB 3 is given in the following theorem.
Theorem E.5 (Regret and Communication Cost Upper Bound of FedGLB-UCB 3 ) .
Under the condition that the learning rate of ONS γ = 12 min( 1 4 S √ k 2 µ S 2 + R 2max , c µ ( k
[Equation]
S 2 + R 2max ) √ NT ) , and the total number of global synchronizations B = √ NT , the cumulative regret R T has upper bound
[Equation]
with probability at least 1 − δ .
The cumulative communication cost has upper bound
[Equation]
Proof.
Similar to the proof for the previous two variants of FedGLB-UCB, we divide the epochs into ‘good’ and ‘bad’ ones according to the determinant ratio, and then bound their cumulative regret separately.
Recall that the instantaneous regret r t,i incurred by client i ∈
[ N ] at time step t ∈ [ T ] has upper bound r t,
i
[Equation]
Note that due to the update schedule S , we have max
p ∈
[ P ] ( n p − n p − 1 ) = NTB .
Then based on
Corollary E.3.1, α t,i =
O
[Equation]
max
[Equation]
NTB )
, so we have, ∀ t ∈
[ T ] , i ∈
[ N ] ,
[Equation]
with probability at least 1 − δ .
Therefore, the cumulative regret for the ‘good epochs’ is
REG good =
O ( k µ ( k µ + R max ) c µ d NT [Equation] log( NT )) .
Using the same argument as in the proof for FedGLB-UCB 1 , the cumulative regret for each ‘bad ’ epoch is upper bounded by 2 k µ S NTB .
Since there can be at most O ( d log NT ) ‘bad epochs’, the cumulative regret for all the ‘bad epochs’ is upper bounded by
[Equation]
Combining the regret incurred in both ‘good’ and ‘bad’ epochs, we have
[Equation]
To recover the regret in centralized setting, we can B = NT , which leads to R T =
O ( k µ ( k µ + R max ) c µ d √ NT log( NT )) .
However, this incurs communication cost C T = N 2 T .
Alternatively, if we set B = √ NT , we have R T
= O ( k µ ( k µ + R max )
c µ dN 3 / 4 T 3 / 4 log( NT )) , and
C T
= O ( N 1 . 5 √ T ) .
F Additional Explanation about Figure
In Section 5, we used the scatter plots to present the experiment results.
Here we provide more explanation about how to interpret these ﬁgures.
As mentioned earlier, each dot in Figure 2 denotes the cumulative communication cost (x-axis) and regret (y-axis) that an algorithm (FedGLB-UCB, its variants, or DisLinUCB) with certain threshold value of D or B (labeled next to the dot) has obtained at iteration T .
Here, Figure 3 shows how the cumulative regret/reward and communication cost of ﬁve algorithms change over the course of federated bandit learning in our evaluations on synthetic dataset, (their ﬁnal results at iteration T are used to plot ﬁve dots in Figure 2(a)).
By carefully examining the relationship between their regret and communication cost, we can see that in Figure 3, FedGLB-UCB ( D = 5 . 0 ),
FedGLB-UCB 1 ( B = 10 . 0 ), FedGLB-UCB 2 ( B = 10 . 0 ), and FedGLB-UCB 3 ( B = 5000 . 0 ) incur similar total communication cost, but FedGLB-UCB ( D = 5 . 0 ) attains much smaller regret than the others.
Meanwhile, FedGLB-UCB ( D = 5000 . 0 ) attains almost the same regret as FedGLB-UCB( B = 10 . 0 ), but its communication cost is much lower.
Figure 3 also depicts how the communication was controlled in FedGLB-UCB under its event triggered protocol (e.g., generally a decreasing frequency of communication comparing to the scheduled updated in its variants).
This shows that FedGLB-UCB strikes the best regret/rewardcommunication trade-off among the algorithm instances in comparison.
However, this line chart can only accommodate a limited range of trade-off settings for these algorithms, to attain a reasonable visibility.
In comparison, the scatter plots in Figure 2(a) provide a much more thorough view of how well the algorithms balance regret/reward and communication cost, by covering a large range of trade-off settings.