Communication Efficient Distributed Learning for Kernelized Contextual Bandits
Abstract
We tackle the communication efficiency challenge of learning kernelized contextual bandits in a distributed setting.
Despite the recent advances in communicationefficient distributed bandit learning, existing solutions are restricted to simple models like multi-armed bandits and linear bandits, which hamper their practical utility.
In this paper, instead of assuming the existence of a linear reward mapping from the features to the expected rewards, we consider non-linear reward mappings, by letting agents collaboratively search in a reproducing kernel Hilbert space (RKHS).
This introduces significant challenges in communication efficiency as distributed kernel learning requires the transfer of raw data, leading to a communication cost that grows linearly w.r.t.
time horizon T .
We addresses this issue by equipping all agents to communicate via a common Nyström embedding that gets updated adaptively as more data points are collected.
We rigorously proved that our algorithm can attain sub-linear rate in both regret and communication cost.
Introduction
Contextual bandit algorithms have been widely used for a variety of real-world applications, including recommender systems [20], display advertisement [21] and clinical trials [11].
While most existing bandit solutions assume a centralized setting (i e., all the data reside in and all the actions are taken by a central server), there is increasing research effort on distributed bandit learning lately [30, 10, 24, 17, 19], where N clients, under the coordination of a central server, collaborate to minimize the overall cumulative regret incurred over a finite time horizon T .
In many distributed application scenarios, communication is the main bottleneck, e.g., communication in a network of mobile devices can be slower than local computation by several orders of magnitude [16].
Therefore, it is vital for distributed bandit learning algorithms to attain sub-linear rate (w.r t. time horizon T ) in both cumulative regret and communication cost.
However, prior works in this line of research are restricted to linear models [30], which could oversimplify the problem and thus leads to inferior performance in practice.
In centralized setting, kernelized bandit algorithms, e.g., KernelUCB [29] and IGP-UCB [6], are proposed to address this issue by modeling the unknown reward mapping as a non-parametric function lying in a reproducing kernel
Hilbert space (RKHS), i.e., the expected reward is linear w.r.t.
an action feature map of possibly infinite dimensions.
Despite the strong modeling capability of kernel method, collaborative exploration in the RKHS gives rise to additional challenges in designing a communication efficient bandit algorithm.
Specifically, unlike distributed linear bandit where the clients can simply communicate the d × d sufficient statistics [30], where d is the dimension of the input feature vector, the joint kernelized estimation of the unknown reward function requires communicating either 1) the p × p sufficient statistics in the RKHS, where p is the dimension of the RKHS that is possibly infinite, or 2) the set of input feature vectors that grows linearly w.r.t.
T .
Neither of them is practical due to the huge communication cost.
In this paper, we propose the first communication efficient algorithm for distributed kernel bandits, which tackles the aforementioned challenge via a low-rank approximation of the empirical kernel matrix.
In particular, we extended the Nyström method [22] to distributed learning for kernelized contextual bandits.
In this solution, all clients first project their local data to a finite RKHS spanned by a common dictionary, i.e., a small subset of the original dataset, and then they only need to communicate the embedded statistics for collaborative exploration.
To ensure effective regret reduction after each communication round, as well as ensuring the dictionary remains representative for the entire distributed dataset throughout the learning process, the frequency of dictionary update and synchronization of embedded statistics is controlled by measuring the amount of new information each client has gained since last communication.
We rigorously prove that the proposed algorithm incurs an O ( N 2 γ 3 NT ) communication cost, where γ NT is the maximum information gain that is known to be O (cid:0) log( NT ) (cid:1) for kernels with exponentially decaying eigenvalues, which includes the most commonly used Gaussian kernel, while attaining the optimal O ( √ NTγ NT ) cumulative regret.
Related Works
To balance exploration and exploitation in stochastic linear contextual bandits, LinUCB algorithm [20, 1] is commonly used, which selects arm optimistically w.r.t.
a constructed confidence set on the unknown linear reward function.
By using kernels and Gaussian processes, studies in [26, 29, 6] further extend UCB algorithms to non-parametric reward functions in RKHS, i.e., the feature map associated with each arm is possibly infinite.
Recent years have witnessed increasing research efforts in distributed bandit learning, i.e., multiple agents collaborate in pure exploration [15, 27, 7], or regret minimization [24, 30, 19].
They mainly differ in the relations of learning problems solved by the agents (i.e., homogeneous vs., heterogeneous) and the type of communication network (i.e., peer-to-peer (P2P) vs., star-shaped).
Most of these works assume linear reward functions, and the clients communicate by transferring the O ( d 2 ) sufficient statistics.
Korda et al.
[18] considered a peer-to-peer (P2P) communication network and assumed that the clients form clusters, i.e., each cluster is associated with a unique bandit problem.
Huang et al.
[17] considered a star-shaped communication network as in our paper, but their proposed phase-based elimination algorithm only works in the fixed arm set setting.
The closest works to ours are [30, 10, 19], which proposed event-triggered communication protocols to obtain sublinear communication cost over time for distributed linear bandits with a time-varying arm set.
In comparison, distributed kernelized contextual bandits still remain under-explored.
The only existing work in this direction [9] considered heterogeneous agents, where each agent is associated with an additional feature describing the task similarity between agents.
However, they assumed a local communication setting, where the agent immediately shares the new raw data point to its neighbors after each interaction, and thus the communication cost is still linear over time.
Another closely related line of works is kernelized bandits with approximation, where Nyström method is adopted to improve computation efficiency in a centralized setting.
Calandriello et al.
[3] proposed an algorithm named BKB, which uses Ridge Leverage Score sampling (RLS) to re-sample a new dictionary from the updated dataset after each interaction with the environment.
A recent work by Zenati et al.
[31] further improved the computation efficiency of BKB by adopting an online sampling method to update the dictionary.
However, both of them updated the dictionary at each time step to ensure the dictionary remains representative w.r.t.
the growing dataset, and therefore are not applicable to our problem.
This is because the dataset is stored cross clients in a distributed manner, and projecting the dataset to the space spanned by the new dictionary requires communication with all clients, which is prohibitively expensive in terms of communication.
Calandriello et al.
[4] also proposed a variant of BKB, named BBKB, for batched Gaussian process optimization.
BBKB only needs to update the dictionary occasionally according to an adaptive schedule, and thus partially addresses the issue mentioned above.
However, as BBKB works in a centralized setting, their adaptive schedule can be computed based on the whole batch of data, while in our decentralized setting, each client can only make the update decision according to the data that is locally available.
Moreover, in BBKB, all the interactions are based on a fixed model estimation over the whole batch, which is mentioned in their Appendix A.4 as a result of an inherent technical difficulty.
In comparison, our proposed method effectively addresses this difficulty with improved analysis, and thus allows each client to utilize newly collected data to update its model estimation on the fly.
Preliminaries
In this section, we first formulate the problem of distributed kernelized contextual bandits.
Then, as a starting point, we propose and analyze a naive UCB-type algorithm for distributed kernelized contextual bandit problem, named DisKernelUCB.
This demonstrates the challenges in designing a communication efficient algorithm for this problem, and also lays down the foundation for further improvement on communication efficiency in Section 4.
3.1 Distributed Kernelized Contextual Bandit Problem
Consider a learning system with 1) N clients that are responsible for taking actions and receiving feedback from the environment, and 2) a central server that coordinates the communication among the clients.
The clients cannot directly communicate with each other, but only with the central server, i.e., a star-shaped communication network.
Following prior works [30, 10], we assume the N clients interact with the environment in a round-robin manner for a total number of T rounds.
Specifically, at round l ∈ [ T ] , each client i ∈ [ N ] chooses an arm x t from a candidate set A t , and then receives the corresponding reward feedback y t
= f ( x t ) + η t ∈ R , where the subscript t := N ( l − 1)+ i indicates this is the t -th interaction between the learning system and the environment, and we refer to it as time step t 1 .
Note that A t is a time-varying subset of A ⊆ R d that is possibly infinite, f denotes the unknown reward function shared by all the clients, and η t denotes the noise.
Denote the sequence of indices corresponding to the interactions between client i and the environment up to time t as N t ( i ) = { 1 ≤ s ≤ t : i s =
i } (if s mod N = 0 , then i s =
N ; otherwise i s =
s mod N ) for t = 1 , 2 , . .
, NT .
By definition, |N
Nl ( i ) |
= l, ∀ l ∈ [ T ] , i.e., the clients have equal number of interactions at the end of each round l .
Kernelized Reward Function We consider an unknown reward function f that lies in a RKHS, denoted as H , such that the reward can be equivalently written as
[Equation]
where θ ⋆ ∈ H is an unknown parameter, and ϕ : R d → H is a known feature map associated with
H .
We assume η t is zero-mean R -sub-Gaussian conditioned on σ (cid:0) ( x s , η s ) s ∈N t − 1 ( i t ) , x t (cid:1) , ∀ t , which denotes the σ -algebra generated by client i t ’s previously pulled arms and the corresponding noise.
In addition, there exists a positive definite kernel k ( · , · ) associated with H , and we assume ∀ x ∈
A that, ∥ x ∥ k ≤ L and ∥ f ∥ k ≤ S for some L, S > 0 .
Regret and Communication Cost The goal of the learning system is to minimize the cumulative (pseudo) regret for all N clients, i.e., R NT
=
(cid:80) NTt =1 r t , where r t = max x ∈A t ϕ ( x ) ⊤ θ ⋆ − ϕ ( x t ) ⊤ θ ⋆ .
Meanwhile, the learning system also wants to keep the communication cost C NT low, which is measured by the total number of scalars being transferred across the system up to time step NT .
3.2 Distributed Kernel UCB
As a starting point to studying the communication efficient algorithm in Section 4 and demonstrate the challenges in designing a communication efficient distributed kernelized contextual bandit algorithm, here we first introduce and analyze a naive algorithm where the N clients collaborate on learning the exact parameters of kernel bandit, i.e., the mean and variance of estimated reward.
We name this algorithm Distributed Kernel UCB, or DisKernelUCB for short, and its description is given in
Algorithm 1.
Arm Selection For each round l ∈
[ T ] , when client i ∈ [ N ] interacts with the environment, i.e., the t -th interaction between the learning system and the environment where t = N ( l − 1)
+ i , it chooses arm x t ∈
A t based on the UCB of the mean estimator (line 5):
[Equation]
where ˆ µ t,i ( x ) and ˆ σ 2 t,i ( x ) denote client i ’s local estimated mean reward for arm x ∈ A and its variance, and α t − 1 ,i is a carefully chosen scaling factor to balance exploration and exploitation (see
Lemma 3.1 for proper choice).
To facilitate further discussion, for time step t ∈ [ NT ] , we denote the sequence of time indices for the data points that have been used to update client i ’s local estimate as D t ( i ) , which include both data points collected locally and those shared by the other clients.
If the clients never communicate,
D t ( i ) =
N t ( i ) , ∀ t, i ; otherwise, N t
( i ) ⊂
D t ( i ) ⊆
[ t ] , with D t ( i ) =
[ t ] recovering the centralized setting, i.e., each new data point collected from the environment immediately becomes available to all the clients in the learning system.
The design matrix and reward vector for client i at time step t are denoted by X D t ( i ) =
[ x s ] ⊤ s ∈D t
( i ) ∈ R |D t ( i ) |×
d , y t,i =
[ y s ] ⊤ s ∈D t ( i ) ∈
R |D t ( i ) | , respectively.
By applying the feature map ϕ ( · ) to each row of X D t ( i ) , we obtain Φ D t ( i
) ∈ R |D t ( i ) |× p , where p is the dimension of H and is possibly infinite.
Since the reward function is linear in H , client i can construct the Ridge regression estimator ˆ θ t,i = ( Φ ⊤D t ( i ) Φ D t ( i ) + λI ) − 1 Φ ⊤D t ( i ) y t,i , where λ >is the regularization coefficient.
This gives us the estimated mean reward and variance in primal form for any arm x ∈ A , i.e., ˆ µ t,i ( x ) =
ϕ
( x ) ⊤ A − 1 t,i b t,i and ˆ σ t,i ( x )
= (cid:113) ϕ
( x ) ⊤ A − 1 t,i ϕ ( x ) , where
A t,i =
Φ ⊤D t ( i ) Φ D t ( i )
+ λ
I and b t,i =
Φ ⊤D t ( i ) y
t,i .
Then using the kernel trick, we can obtain their equivalence in the dual form that only involves entries of the kernel matrix, and avoids directly working on H which is possibly infinite:
[Equation]
x ) where K D t ( i ) ( x ) =
Φ D t ( i ) ϕ
( x ) =
[ k ( x s , x )]
⊤ s ∈D t ( i ) ∈ R |D t ( i ) | , and K D t ( i ) , D t ( i ) =
Φ ⊤D t ( i ) Φ D t ( i ) =
[ k ( x s , x s ′ )] s,s ′ ∈D t
( i ) ∈ R |D t ( i ) |×|D t
( i ) | .
Communication Protocol To reduce the regret in future interactions with the environment,
the
N clients need to collaborate via communication, and a carefully designed communication protocol is essential in ensuring the communication efficiency.
In prior works like DisLinUCB [30], after each round of interaction with the environment, client i checks whether the event { ( |D t ( i ) | − |D t last ( i ) | ) log( det( A t,i ) det( A t last ,i ) ) >
D } is true, where t last denotes the time step of last global synchronization.
If true, a new global synchronization is triggered, such that the server will require all clients to upload their sufficient statistics since t last , aggregate them to compute { A t , b t } , and then synchronize the aggregated sufficient statistics with all clients, i.e., set { A t,i , b t,i } =
{ A t , b t } , ∀ i ∈ [ N ] .
Using kernel trick, we can obtain an equivalent event-trigger in terms of the kernel matrix,
U t
[Equation]
where D > 0 denotes the predefined threshold value.
If event U t ( D ) is true (line 7), a global synchronization is triggered (line 7-10), where the local datasets of all N clients are synchronized to { ( x s , y s ) } s ∈
[ t ] .
We should note that the transfer of raw data ( x s , y s ) is necessary for the update of the kernel matrix and reward vector in line 6 and line 10, which will be used for arm selection at line 5.
This is an inherent disadvantage of kernelized estimation in distributed settings, which, as we mentioned in Section 2, is also true for the existing distributed kernelized bandit algorithm [9].
Lemma 3.1 below shows that in order to obtain the optimal order of regret, DisKernelUCB incurs a communication cost linear in T (proof given in the appendix), which is expensive for an online learning problem.
Lemma 3.1 (Regret and Communication Cost of DisKernelUCB) .
With threshold D =
T Nγ NT , α t,i = √ λ
∥ θ ⋆ ∥
+ R (cid:112) 4 ln N/δ
+
ln det( I + K D t ( i ) , D t ( i ) /λ ) , we have
[Equation]
with probability at least 1 − δ , and
C NT
= O ( TN 2 d ) .
where γ NT := max D⊂A : |D| =
NT 12 log det( K D , D /λ + I ) is the maximum information gain after NT interactions [6].
It is problem-dependent and can be bounded for specific arm set A and kernel function k ( · , · ) .
For example, γ NT =
O
( d log( NT )) for linear kernel and γ NT = O (log( NT ) d +1 ) for Gaussian kernel.
Remark 1.
In the distributed linear bandit problem, to attain O ( d √ NT ln( NT )) regret, DisLinUCB [30] requires a total number of O ( N 0 .
5 d log( NT )) synchronizations, and DisKernelUCB matches this result under linear kernel, as it requires O ( N 0 .
5 γ NT ) synchronizations.
We should note that the communication cost for each synchronization in DisLinUCB is fixed, i.e., O ( Nd 2 ) to synchronize the sufficient statistics with all the clients, so in total C NT
= O ( N 1 .
5 d 3 ln( NT )) .
However, this is not the case for DisKernelUCB that needs to send raw data, because the communication cost for each synchronization in DisKernelUCB is not fixed, but depends on the number of unshared data points on each client.
Even if the total number of synchronizations is small, DisKernelUCB could still incur
C NT
= O ( TN 2 d ) in the worse case.
Consider the extreme case where synchronization only happens once, but it happens near NT , then we still have C NT
= O ( TN 2 d ) .
The time when synchronization gets triggered depends on {A t } t ∈ [ NT ] , which is out of the control of the algorithm.
Therefore, in the following section, to improve the communication efficiency of DisKernelUCB, we propose to let each client communicate embedded statistics in some small subspace during each global synchronization.
Approximated Distributed Kernel UCB
In this section, we propose and analyze a new algorithm that improves the communication efficiency of DisKernelUCB using the Nyström approximation, such that the clients only communicate the embedded statistics during event-triggered synchronizations.
We name this algorithm Approximated
Distributed Kernel UCB, or Approx-DisKernelUCB for short.
Its description is given in Algorithm 2.
4.1 Algorithm
Arm selection For each round l ∈
[ T ] , when client i ∈ [ N ] interacts with the environment, i.e., the t -th interaction between the learning system and the environment where t :
= N ( l − 1) + i , instead of using the UCB for the exact estimator in Eq (1), client i chooses arm x t ∈
A t that maximizes the
UCB for the approximated estimator (line 5):
[Equation]
where ˜ µ t − 1 ,i ( x ) and ˜ σ t − 1 ,i ( x ) are approximated using Nyeström method, and the statistics used to compute these approximations are much more efficient to communicate as they scale with the maximum information gain γ NT instead of T .
Specifically, Nyström method works by projecting some original dataset D to the subspace defined by a small representative subset S ⊆ D , which is called the dictionary.
The orthogonal projection matrix is defined as
[Equation]
1: Input: threshold D
0 , regularization parameter δ
1) and kernel
2: Initialize ˜ µ 0 ,i ( x ) = 0 , ˜ σ 0 ,i ( x ) = (cid:112) k
( x , x ) , N 0 ( i )
=
D
( i ) = ∅ , ∀ i ∈
[ N ] ; S 0 = ∅ , t last =3: for round l = 1 , 2 , ..., T do 4: for client i = 1 , 2 , ..., N do 5:
[Client i ] selects arm x t ∈
A t according to Eq (3) and observes reward y t , where t :=
N ( l − 1) + i 6: [Client i ] updates Z ⊤D t ( i ); S t last Z D t ( i ); S t last and Z ⊤D t ( i ); S t last y D t ( i ) using (cid:0)
z ( x t ; S t last ) , y t (cid:1) ; sets
N t ( i ) =
N t − 1 ( i ) ∪ { t } , and D t ( i ) =
D t − 1 ( i ) ∪ { t } //
Global Synchronization 7: if the event U t ( D ) defined in Eq (4) is true then 8: [Clients ∀ i ] sample S t,i = RLS ( N t ( i ) , ¯ q, ˜ σ 2 t last ,i ) , and send { ( x s , y s ) } s ∈S t,i to server 9: [Server] aggregates and sends { ( x s , y s ) } s ∈S t back to all clients,
where S t = ∪ i ∈
[ N ] S t,i 10: [Clients ∀ i ] compute and send { Z ⊤N t ( i ); S t Z N t ( i ); S t , Z ⊤N t ( i ); S t y N t ( i ) } to server 11:
[Server] aggregates (cid:80) Ni
=
Z ⊤N t ( i ); S t Z N t ( i ); S t , (cid:80)
Ni =
Z ⊤N t ( i ); S t y N t ( i ) and sends it back 12: [Clients ∀ i ] updates Z ⊤D t ( i );
S t Z D t ( i ); S t and Z ⊤D t ( i ); S t y D t ( i ) ; sets D t ( i ) =
∪ Ni
=1 N t
( i ) =
[ t ] and t last =
t
We then take eigen-decomposition of K S , S =
UΛU ⊤ to rewrite the orthogonal projection as
P S =
Φ ⊤S UΛ − 1 / 2 Λ − 1 / 2 U ⊤ Φ S , and define the Nyström embedding function
[Equation]
which maps the data point x from R d to R |S| .
Therefore, we can approximate the Ridge regression estimator in Section 3.2 as ˜ θ t,i = ˜
A − 1 t,i ˜ b t,i , where ˜
A t,i =
P S Φ ⊤D t ( i ) Φ D t ( i ) P S
+ λ I , and ˜ b t,i =
P S Φ ⊤D t ( i )
y D t ( i ) , and thus the approximated mean reward and variance in Eq (3) can be expressed as ˜ µ t,i ( x ) =
ϕ
( x ) ⊤ ˜
A − 1 t,i ˜ b t,i and ˜ σ t,i ( x ) = (cid:113) ϕ
( x ) ⊤ ˜
A − 1 t,i ϕ ( x ) , and their kernelized representation are (see appendix for detailed derivation)
[Equation]
; S [ Z ⊤D t ( i ); S Z D t ( i ); S
+ λ I ] − 1 z ( x |S ) where Z D t ( i ); S ∈ R |D t
( i ) |×|S| is obtained by applying z ( · ; S ) to each row of X D t ( i ) , i.e., Z D t ( i ); S =
Φ D t ( i ) P 1 / 2 S .
We can see that the computation of ˜ µ t,i ( x ) and ˜ σ t,i ( x ) only requires the embedded statistics:
matrix Z ⊤ D t ( i ); S Z D t ( i ); S ∈ R |S|×|S| and vector Z ⊤ D t ( i ); S y D t
( i ) ∈ R |S| , which, as we will show later, makes joint kernelized estimation among N clients much more efficient in communication.
After obtaining the new data point ( x t , y t ) , client i immediately updates both ˜ µ t − 1 ,i ( x ) and ˜ σ t − 1 ,i ( x ) using the newly collected data point ( x t , y t ) , i.e., by projecting x t to the finite dimensional RKHS spanned by Φ S t last (line 6).
Recall that, we use N t ( i ) to denote the sequence of indices for data collected by client i , and denote by D t ( i ) the sequence of indices for data that has been used to update client
i ’s model estimation ˜ µ t,i .
Therefore, both of them need to be updated to include time step t .
Communication Protocol With the approximated estimator, the size of message being communicated across the learning system is reduced.
However, a carefully designed event-trigger is still required to minimize the total number of global synchronizations up to time NT .
Since the clients can no longer evaluate the exact kernel matrices in Eq (2), we instead use the event-trigger in Eq (4), which can be computed using the approximated variance from last global synchronization as,
[Equation]
Similar to Algorithm 1, if Eq (4) is true, global synchronization is triggered, where both the dictionary and the embedded statistics get updated.
During synchronization, each client first samples a subset
S t ( i ) from N t ( i ) (line 8) using Ridge Leverage Score sampling (RLS) [3, 4], which is given in
Algorithm 3, and then sends { ( x s , y s ) } s ∈S t ( i ) to the server.
The server aggregates the received local subsets to construct a new dictionary { ( x s , y s ) } s ∈S t , where S t = ∪ Ni =1 S t ( i ) , and then sends it back to all N clients (line 9).
Finally, the N clients use this updated dictionary to re-compute the embedded statistics of their local data, and then synchronize it with all other clients via the server (line 10-12).
Intuitively, in Algorithm 2, the clients first agree upon a common dictionary S t that serves as a good representation of the whole dataset at the current time t , and then project their local data to the subspace spanned by this dictionary before communication, in order to avoid directly sending the raw data as in Algorithm 1.
Then using the event-trigger, each client monitors the amount of new knowledge it has gained through interactions with the environment from last synchronization.
When there is a sufficient amount of new knowledge, it will inform all the other clients to perform a synchronization.
As we will show in the following section, the size of S t scales linearly w.r.t.
the maximum information gain γ NT , and therefore it improves both the local computation efficiency on each client, and the communication efficiency during the global synchronization.
4.2 Theoretical Analysis
Denote the sequence of time steps when global synchronization is performed, i.e., the event U t ( D ) in Eq (4) is true, as { t p } Bp =1 , where B ∈ [ NT ] denotes the total number of global synchronizations.
Note that in Algorithm 2, the dictionary is only updated during global synchronization, e.g., at time t p , the dictionary { ( x s , y s ) } s ∈S tp is sampled from the whole dataset { ( x s , y s ) } s ∈
[ t p ] in a distributed manner, and remains fixed for all the interactions happened at t ∈ [ t p + 1 , t p +1 ] .
Moreover, at time t p , all the clients synchronize their embedded statistics, so that D t p ( i ) =
[ t p ] , ∀ i ∈
[ N ] .
Since Algorithm 2 enables local update on each client, for time step t ∈ [ t p + 1 , t p ] , new data points are collected and added into D t ( i ) ,
such that D t ( i ) ⊇ [ t p ] .
This decreases the approximation accuracy of S t p , as new data points may not be well approximated by S t p .
For example, in extreme cases, the new data could be orthogonal to the dictionary.
To formally analyze the accuracy of the dictionary, we adopt the definition of ϵ -accuracy from [5].
Denote by ¯ S t,i ∈ R
|D
t ( i ) |×|D t
( i ) | a diagonal matrix, with its s -th diagonal entry equal to 1 √ ˜ p
s if s ∈ S t p and 0 otherwise.
Then if
[Equation]
, we say the dictionary { ( x s , y s ) } s ∈S tp is ϵ t,i -accurate w.r.t.
dataset { ( x s , y s ) } s ∈D t
( i ) .
As shown below, the accuracy of the dictionary for Nyström approximation is essential as it affects the width of the confidence ellipsoid, and thus affects the cumulative regret.
Intuitively, in order to ensure its accuracy throughout the learning process, we need to 1) make sure the RLS procedure in line 8 of
Algorithm 2 that happens at each global synchronization produces a representative set of data samples, and 2) monitor the extent to which the dictionary obtained in previous global synchronization has degraded over time, and when necessary, trigger a new global synchronization to update it.
Compared with prior work that freezes the model in-between consecutive communications [4], the analysis of ϵ -accuracy for Approx-DisKernelUCB is unique to our paper and the result is presented below.
Lemma 4.1.
With ¯ q = 6 1+ ϵ 1 − ϵ log(4 NT/δ ) /ϵ 2 , for some ϵ ∈ [0 , 1) , and threshold D > 0 , Algorithm 2 guarantees that the dictionary is accurate with constant ϵ t,i := (cid:0) ϵ + 1 − 1 1+ 1+ ϵ 1 − ϵ D (cid:1) , and its size |S t | =
O ( γ NT ) for all t ∈
[ NT ] .
Based on Lemma 4.1, we can construct the following confidence ellipsoid for unknown parameter θ ⋆ .
Lemma 4.2 (Confidence Ellipsoid of Approximated Estimator) .
Under the condition that ¯ q = 6 1+ ϵ 1 − ϵ log(4 NT/δ ) /ϵ 2 , for some ϵ ∈ [0 , 1) , and threshold D > 0 , with probability at least 1 − δ , we have ∀ t, i that
[Equation]
Using Lemma 4.2, we obtain the regret and communication cost upper bound of ApproxDisKernelUCB, which is given in Theorem 4.3 below.
Theorem 4.3 (Regret and Communication Cost of Approx-DisKernelUCB) .
Under the same condition as Lemma 4.2, and by setting D = 1 N , ϵ < 13 , we have
[Equation]
with probability at least 1 − δ , and
[Equation]
Here we provide a proof sketch for Theorem 4.3, and the complete proof can be found in appendix.
Proof Sketch.
Similar to the analysis of DisKernelUCB in Section 3.2 and DisLinUCB from [30], the cumulative regret incurred by Approx-DisKernelUCB can be decomposed in terms of ‘good’ and ‘bad’ epochs, and bounded separately.
Here an epoch refers to the time period in-between two consecutive global synchronizations, e.g., the p -th epoch refers to [ t p − 1 + 1 , t p ] .
Now consider an imaginary centralized agent that has immediate access to each data point in the learning system, and denote by A t = (cid:80) ts =1 ϕ s ϕ ⊤ s for t ∈
[ NT ] the matrix constructed by this centralized agent.
We call the p -th epoch a good epoch if ln( det( I + λ − 1 K [ tp ] , [ tp ] ) det( I + λ − 1 K
[ tp − 1] , [ tp − 1] ) ) ≤ 1 , otherwise it is a bad epoch.
Note that ln( det( I + λ − 1 K [ t 1] , [ t 1] ) det( I )
[Equation]
ln(det( I + λ − 1 K
[ NT ] , [ NT ] )) ≤ 2 γ NT , where the last equality is due to the matrix determinant lemma, and the last inequality is by the definition of the maximum information gain γ NT in Lemma 3.1.
Then based on the pigeonhole principle, there can be at most 2 γ NT bad epochs.
By combining Lemma E.1 and Lemma 4.2, we can bound the cumulative regret incurred during all good epochs, i.e., R good = O ( √ NTγ NT ) , which matches the optimal regret attained by the KernelUCB algorithm in centralized setting.
Our analysis deviates from that of DisKernelUCB in the bad epochs, because of the difference in the event-trigger.
Previously, the event-trigger of DisKernelUCB directly bounds the cumulative regret each client incurs during a bad epoch, i.e., (cid:80) t ∈D tp ( i ) \D tp − 1 ( i ) ˆ σ t − 1 ,i ( x t ) ≤ (cid:113) ( |D t p ( i ) | − |D t p − 1 ( i ) | ) log (cid:0) det( I
[Equation]
λ −
[Equation]
K
[Equation] [Equation]
λ
[Equation]
K
[Equation]
\ ∆ D t
[Equation]
i t ) , D t
[Equation]
i t ) \ ∆
[Equation]
t
[Equation]
i t ) )
[Equation]
< √
D .
However, the event trigger of Approx-DisKernelUCB only bounds part of it, i.e.,
[Equation] [Equation]
i
[Equation] [Equation]
) D , which leads to
R bad =
O ( √ Tγ NT N √ D ) that is slightly worse than that of DisKernelUCB, i.e., a √ T factor in place of the √ γ NT factor.
By setting D = 1 /N , we have R NT
= O ( √ NTγ NT ) .
Note that, to make sure ϵ t,i = (cid:0) ϵ + 1 −
[Equation]
∈ [0 , 1) is still well-defined, we can set ϵ < 1 / 3 .
For communication cost analysis, we bound the total number of epochs B by upper bounding the total number of summations like (cid:80) t p s =
t
p − 1 +1 ˆ σ 2 t p − 1 ( x s ) , over the time horizon NT .
Using Lemma
E.1, our event-trigger in Eq (4) provides a lower bound (cid:80) t p s =
t p − 1 +1 ˆ σ 2 t p − 1 ( x s ) ≥ 1 − ϵ 1+ ϵ D .
Then in order to apply the pigeonhole principle, we continue to upper bound the summation over all epochs, (cid:80)
Bp =
[Equation]
(cid:80)
[Equation]
=
[Equation]
1 +1 ˆ σ
[Equation]
1 ( x s ) =
(cid:80) Bp =1 (cid:80)
[Equation]
p s = t p − 1 +1 ˆ σ
[Equation]
s
[Equation]
1 ( x s ) ˆ σ
[Equation]
( x
[Equation]
) ˆ σs − 1 ( x s ) by deriving a uniform bound for the ratio ˆ σ 2 tp − 1 ( x s ) ˆ σs − 1 ( x s [Equation] ≤ ˆ σ [Equation]
−
[Equation]
( x s ) ˆ σ
[Equation]
( x s ) ≤ 1 + (cid:80) t p s =
[Equation]
p − [Equation] +1 ˆ σ 2 t p − 1 ( x s ) ≤ 1 + 1+ ϵ 1 − ϵ (cid:80) t p s = t p − 1 +1 ˜ σ 2 t p − 1 ( x s ) in terms of the communication threshold D on each client.
This leads to the following upper bound about the total number of epochs B ≤ 1+ ϵ 1 − ϵ
[ 1 D + 1+ ϵ 1 − ϵ ( N + L 2 / ( λD ))]2 γ NT , and with D = 1 /N , we have C NT ≤ B · Nγ 2 NT
= O ( N 2 γ 3 NT ) , which completes the proof.
Remark 2.
Compared with DisKernelUCB’s O ( TN 2 d ) communication cost, Approx-DisKernelUCB removes the linear dependence on T , but introduces an additional γ 3 NT dependence due to the communication of the embedded statistics.
In situations where γ NT ≪ T 1 / 3 d 1 / 3 , DisKernelUCB is preferable.
As mentioned in Lemma 3.1, the value of γ NT , which affects how much the data can be compressed, depends on the specific arm set of the problem and the kernel function of the choice.
By Mercer’s Theorem, one can represent the kernel using its eigenvalues, and γ NT characterizes how fast its eigenvalues decay.
Vakili et al.
[28] showed that for kernels whose eigenvalues decay exponentially, i.e., λ m =
O (exp( − m β e )) , for some β e > 0 , γ NT =
O (log 1+ 1 βe ( NT )) .
In this case, Approx-DisKernelUCB is far more efficient than DisKernelUCB.
This includes Gaussian kernel, which is widely used for GPs and SVMs.
For kernels that have polynomially decaying eigenvalues, i.e., λ m =
O ( m − β p ) , for some β p > 1 ,
γ NT
= O ( T 1 βp log 1 − 1 βp ( NT )) .
Then as long as β p > 3 ,
Approx-DisKernelUCB still enjoys reduced communication cost.
Experiments
In order to evaluate Approx-DisKernelUCB’s effectiveness in reducing communication cost, we performed extensive empirical evaluations on both synthetic and real-world datasets, and the results (averaged over 3 runs) are reported in Figure 1, 2 and 3, respectively.
We included DisKernelUCB,
DisLinUCB [30], OneKernelUCB, and NKernelUCB [6] as baselines, where One-KernelUCB learns a shared bandit model across all clients’ aggregated data where data aggregation happens immediately after each new data point becomes available, and N-KernelUCB learns a separated bandit model for each client with no communication.
For all the kernelized algorithms, we used the Gaussian kernel k ( x, y ) =
exp( − γ ∥ x − y ∥ 2 ) .
We did a grid search of γ ∈ { 0 . 1 , 1 , 4 } for kernelized algorithms, and set D = 20 for DisLinUCB and DisKernelUCB, D = 5 for Approx-DisKernelUCB.
For all algorithms, instead of using their theoretically derived exploration coefficient α , we followed the convention [20, 32] to use grid search for α in { 0 . 1 , 1 , 4 } .
Due to space limit, here we only present the experiment results and discussions.
Details about the experiment setup are presented in appendix.
When examining the experiment results presented in Figure 1, 2 and 3, we can first look at the cumulative regret and communication cost of OneKernelUCB and NKernelUCB, which correspond to the two extreme cases where the clients communicate in every time step to learn a shared model, and each client learns its own model independently with no communication, respectively.
OneKernelUCB achieves the smallest cumulative regret in all experiments, while also incurring the highest communication cost, i.e., O ( TN 2 d ) .
This demonstrates the need of efficient data aggregation across clients for reducing regret.
Second, we can observe that DisKernelUCB incurs the second highest communication cost in all experiments due to the transfer of raw data, as we have discussed in Remark 1, which makes it prohibitively expensive for distributed setting.
On the other extreme, we can see that DisLinUCB incurs very small communication cost thanks to its closed-form solution, but fails to capture the complicated reward mappings in most of these datasets, e.g. in Figure 1(a), 2(b) and 3(a), it leads to even worse regret than NKernelUCB that learns a kernelized bandit model independently for each client.
In comparison, the proposed Approx-DisKernelUCB algorithm enjoys the best of both worlds in most cases, i.e., it can take advantage of the superior modeling power of kernels to reduce regret, while only requiring a relatively low communication cost for clients to collaborate.
[Equation]
[Figure] [Figure]
[Figure]
On all the datasets, Approx-DisKernelUCB achieved comparable regret with DisKernelUCB that maintains exact kernelized estimators, and sometimes even getting very close to OneKernelUCB, e.g., in Figure 1(b) and 2(a), but its communication cost is only slightly higher than that of DisLinUCB.
Conclusion
In this paper, we proposed the first communication efficient algorithm for distributed kernel bandits using Nyström approximation.
Clients in the learning system project their local data to a finite RKHS spanned by a shared dictionary, and then communicate the embedded statistics for collaborative exploration.
To ensure communication efficiency, the frequency of dictionary update and synchronization of embedded statistics are controlled by an event-trigger.
The algorithm is proved to incur
O ( N 2 γ 3 NT ) communication cost, while attaining the optimal O ( √ NTγ NT ) cumulative regret.
We should note that the total number of synchronizations required by Approx-DisKernelUCB is
Nγ NT , which is √ N worse than DisKernelUCB.
An important future direction of this work is to investigate whether this part can be further improved.
The lower bound analysis for the communication cost of distributed contextual bandits still remains an open problem, and is an important future direction.
To the best of our knowledge, the only applicable lower bound states that, in order to have smaller regret than the trivial O ( N √ T ) result, i.e., run N instances of optimal bandit algorithm with no communication, Ω( N ) communications is necessary [14].
In comparison, it is more interesting to know what the communication lower bound is in order to attain the optimal O ( √ NT ) regret.
It is also interesting to extend our algorithm to P2P setting, i.e., no central server to coordinate the update of the shared dictionary and the exchange of embedded statistics, which may require utilizing local structures in the network of clients, to approximate each block of the kernel matrix separately [25].
Acknowledgement
This work is supported
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] , see Remark(c)
Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] , in Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] , the complete proof is given in appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Technical Lemmas
Lemma A.1 (Lemma 12 of [1]) .
Let A , B and C be positive semi-definite matrices with finite dimension, such that A = B + C .
Then, we have that:
[Equation]
Lemma A.2 (Extension of Lemma A.1 to kernel matrix) .
Define positive definite matrices
A = λ
I + Φ ⊤ 1 Φ 1 + Φ ⊤ 2 Φ 2 and B =
λ
I + Φ ⊤ 1 Φ 1 , where Φ ⊤ 1 Φ 1 , Φ ⊤ 2 Φ 2 ∈ R p × p and p is possibly infinite.
Then, we have that:
[Equation]
where K A = (cid:20) ΦΦ [Equation] and K
[Equation]
Proof of Lemma A.2.
Similar to the proof of Lemma 12 of [1], we start from the simple case when Φ ⊤ 2 Φ 2 = mm ⊤ , where m ∈ R p . Using Cauchy-Schwartz inequality, we have [Equation] and thus,
[Equation]
so we have
[Equation]
for any ϕ .
Then using the kernel trick, e.g., see the derivation of Eq (27) in [31], we have
[Equation]
which finishes the proof of this simple case.
Now consider the general case where Φ ⊤ 2 Φ 2 =
m 1 m ⊤ 1 + m 2 m ⊤ 2 + · · · + m t − 1 m ⊤ t − 1 .
Let’s define V s = B + m 1 m ⊤ 1 + m
m ⊤ 2 + · · · + m s − 1 m ⊤ s −and the corresponding kernel matrix K V s =   Φ [Equation]
m ⊤ s
[Equation]

 (cid:2) Φ ⊤ 1 , m 1 , . . .
, m s − 1 (cid:3) , and note that ϕ ⊤ Aϕ ϕ ⊤ Bϕ = ϕ ⊤ V t ϕ ϕ
⊤ V t − 1 ϕ ϕ ⊤ V t − 1 ϕ ϕ ⊤ V t − 2 ϕ . . .
ϕ ⊤ V 2 ϕ ϕ ⊤ Bϕ .
Then we can apply the result for the simple case on each term in the product above, which gives us
[Equation]
which finishes the proof.
Lemma A.3 (Eq (26) and Eq (27) of [31]) .
Let { ϕ t } ∞ t =1 be a sequence in R p , V ∈ R p × p
a positive definite matrix, where p is possibly infinite, and define V t
= V + (cid:80) ts =1 ϕ s ϕ ⊤ s .
Then we have that [Equation] where K V t is the kernel matrix corresponding to V t as defined in Lemma A.2. Lemma A.4 (Lemma 4 of [4]) .
For t > t last , we have for any x ∈ R d
[Equation]
B Confidence Ellipsoid for DisKernelUCB
In this section, we construct the confidence ellipsoid for DisKernelUCB as shown in Lemma B.1.
Lemma B.1 (Confidence Ellipsoid for DisKernelUCB) .
Let δ ∈ (0 , 1) .
With probability at least 1 − δ , for all t ∈ [ NT ] , i ∈ [ N ] , we have
[Equation]
The analysis is rooted in [31] for kernelized contextual bandit, but with non-trivial extensions: we adopted the stopping time argument from [1] to remove a logarithmic factor in T (this improvement is hinted in Section 3.3 of [31] as well); and this stopping time argument is based on a special ‘batched filtration’ that is different for each client, which is required to address the dependencies due to the event-triggered distributed communication.
This problem also exists in prior works of distributed linear bandit, but was not addressed rigorously (see Lemma H.1.
of [30]).
Recall that the Ridge regression estimator
[Equation]
and thus, we have
[Equation]
where the first inequality is due to the triangle inequality, and the second is due to the property of
Rayleigh quotient, i.e., ∥ A − 1 / 2 t,
i
[Equation]
Difference from standard argument Note that the second term may seem similar to the ones appear in the self-normalized bound in previous works of linear and kernelized bandits [1, 6, 31].
However, a main difference is that D t ( i ) , i.e., the sequence of indices for the data points used to update client i , is constructed using the event-trigger as defined in Eq (2) .
The event-trigger is data-dependent, and thus it is a delayed and permuted version of the original sequence [ t ] .
It is delayed in the sense that the length |D
t ( i ) | < t unless t is the global synchronization step.
It is permuted in the sense that every client receives the data in a different order, i.e., before the synchronization, each client first updates using its local new data, and then receives data from other clients at the synchronization.
This prevents us from directly applying Lemma 3.1 of [31], and requires a careful construction of the filtration, as shown in the following paragraph.
Construction of filtration For some client i at time step t , the sequence of time indices in D t ( i ) is arranged in the order that client i receives the corresponding data points, which includes both data added during local update in each client, and data received from the server during global synchronization.
The former adds one data point at a time, while the latter adds a batch of data points, which, as we will see, break the assumption commonly made in standard self-normalized bound [1, 6, 31].
Specifically, we denote D t ( i )
[ k ] , for k ≤ |D t ( i ) | , as the k -th element in this sequence, i.e., ( x D t ( i )[ k ] , y D t ( i )
[ k ] ) is the k -th data point received by client i .
Then we denote B t ( i ) =
{ k 1 , k 2 , . . . }
as the sequence of k ’s that marks the end of each batch (a singel data point added by local update is also considered a batch).
We can see that if the k -th element is in the middle of a batch, i.e., k ∈ [ k q − 1 , k q ] , it has dependency all the way to the k q ’s element, since this index can only be determined until some client triggers a global synchronization at time step D t ( i )
[ k q ] .
Denote by F k,i = σ (cid:0) ( x s , η s ) s ∈D ∞
( i )[1: k − 1] , x D ∞ ( i )
[ k ] (cid:1) the σ -algebra generated by the sequence of data points up to the k -th element in D ∞ ( i ) .
As we mentioned, because of the dependency of the index on the future data points, for some k -th element that is inside a batch, i.e., k ∈
[ k q − 1 , k q ] , x D ∞ ( i )[ k ] is not F k,i -measurable and η D ∞ ( i )
[ k ] is not F k +1 ,i -measurable, which violate the as- sumption made in standard self-normalized bound [1, 6, 31].
However, they become measurable if we condition on F k q ,i .
In addition, recall that in Section 3.1 we assume η D ∞ ( i )
[ k ] is zeromean R -sub-Gaussian conditioned on σ (cid:0) ( x s , η s ) s ∈N D∞ ( i )
[ k ] − 1 ( i D∞ ( i )
[ k ] ) , x D ∞ ( i )
[ k ] (cid:1) , which is the σ -algebra generated by the sequence of local data collected by client i D ∞ ( i )
[ k ] .
We can see that as σ (cid:0) ( x s , η s ) s ∈N D∞ ( i )
[ k ] − 1 ( i D∞ ( i )
[ k ] ) , x D ∞ ( i )
[ k ] (cid:1) ⊆
F k,i , η D ∞ ( i )[ k ] is zero-mean R -sub-Gaussian conditioned on F k,i .
Basically, our assumption of R -sub-Gaussianity conditioned on local sequence instead of global sequence of data points, prevents the situation where the noise depends on data points that have not been communicated to the current client yet, i.e., they are not included in F k,i .
With our ‘batched filtration’ {F k,i } k ∈B ∞ ( i ) for each client i , we have everything we need to establish a time-uniform self-normalized bound that resembles Lemma 3.1 of [31], but with improved logarithmic factor using the stopping time argument from [1].
Then we can take a union bound over N clients to obtain the uniform bound over all clients and time steps.
Super-martingale & self-normalized bound
First, we need the following lemmas adopted from [31] to our ‘batched filtration’.
Lemma B.2.
Let υ ∈ R
d be arbitrary and consider for any k ∈ B ∞ ( i ) , i ∈
[ N ] , we have
[Equation]
is a F k +1 ,i -super-martingale, and E [ M υk,i ] ≤ exp( − 12 λυ ⊤ υ ) .
Let ˜ k be a stopping time w.r.t.
the filtration {F k,i } k ∈B ∞ ( i ) .
Then M υ ˜ k,i is almost surely well-defined and E [
M υ ˜ k,i ] ≤ 1 .
Proof of Lemma B.2.
To show that { M υk,i } k ∈B ∞ ( i ) is a super-martinagle, we denote
[Equation]
with D υ 0 ,i = exp( − 12 λυ ⊤ υ ) , and as we have showed earlier, η D ∞ ( i )[ k ] is R -sub-Gaussian conditioned on F k,i .
Therefore, E [ D υk,
i |F k,i ] ≤ 1 .
Moreover, D υk,i and M υk,i are F k +1 ,i -measurable.
Then we have
[Equation]
which shows { M υk,i } k ∈B ∞ ( i ) is a super-martinagle, with E [ M υk,i ] ≤ D υ 0 ,i = exp( − 12 λυ ⊤ υ ) .
Then using the same argument as Lemma 8 of [1], we have that M υ ˜ k,i is almost surely well-defined, and
E [ M υ ˜ k,i ] ≤ D υ 0 ,i = exp( − 12 λυ ⊤ υ ) .
Lemma B.3.
Let ˜ k be a stopping time w.r.t.
the filtration {F k,i } k ∈B ∞ ( i ) .
Then for δ > 0 , we have
P (cid:16) ∥ ( λ I + (cid:88)
[Equation]
≤ δ.
Proof of Lemma B.3.
Using m :
= ( λ I + (cid:80) s ∈D ∞
( i )[1: k ] X s X ⊤ s )
[Equation] [Equation]
s ∈D ∞
[Equation]
i )
[1: k ] X s η s ) , we can rewrite M υ ˜ k,i as
[Equation]
Then based on Lemma B.2, we have
[Equation]
Following the Laplace’s method as in proof of Lemma 3.1 of [31], we have
[Equation]
By applying Markov-Chernov bound, we finish the proof.
Proof of Lemma B.1 Now using the stopping time argument as in [1], and a union bound over clients, we can bound the second term in Eq (5).
First, define the bad event
[Equation]
and ˜ k ( ω ) = min( k ≥ 0 : ω ∈ B k ( δ )) , which is a stopping time.
Moreover, ∪ k ∈B ∞ ( i ) B k
( δ )
= { ω : ˜ k < ∞} .
Then using Lemma B.3, we have
[Equation]
Note that B ∞ ( i ) is the sequence of indices k in D ∞ ( i ) when client i gets updated.
Therefore, the result above is equivalent to
[Equation]
A − 1 / 2 t,i (cid:88) s ∈D t
( i ) ϕ
s η s ∥ ≤ R (cid:113) 2 ln(1 /δ )
+ ln(det( K D t ( i ) ,
D t
( i ) /λ
+ I ))
for all t ≥ 1 , with probability at least 1 − δ .
Then by taking a union bound over N clients, we finish the proof.
C Proof of Lemma 3.1: Regret and Communication Cost of DisKernelUCB
Based on Lemma B.1 and the arm selection rule in Eq (1), we have f ( x ⋆t ) ≤ ˆ µ t − 1 ,i t ( x ⋆t )
+ α t − 1 ,i t ˆ σ t − 1 ,i t ( x ⋆t )
≤ ˆ µ t − 1 ,i t ( x t ) + α t − 1 ,i t ˆ σ t − 1 ,i t ( x t ) , f ( x t ) ≥ ˆ µ t − 1 ,i t ( x t ) − α t − 1 ,i t ˆ σ t − 1 ,i t ( x t ) , and thus r t = f
( x ⋆t ) − f ( x t ) ≤ 2 α t − 1 ,i t ˆ σ t − 1 ,i t ( x t ) , for all t ∈
[ NT ] , with probability at least 1 − δ .
Then following similar steps as DisLinUCB of [30], we can obtain the regret and communication cost upper bound of DisKernelUCB.
C.1 Regret Upper Bound
We call the time period in-between two consecutive global synchronizations as an epoch, i.e., the p -th epoch refers to [ t p − 1 + 1 , t p ] , where p ∈ [ B ] and 0 ≤ B ≤ NT denotes the total number of global synchronizations.
Now consider an imaginary centralized agent that has immediate access to each data point in the learning system, and denote by A t = (cid:80) ts =1 ϕ s ϕ ⊤ s and K [ t ] , [ t ] for t ∈ [ NT ] the covariance matrix and kernel matrix constructed by this centralized agent.
Then similar to [30], we call the p -th epoch a good epoch if
[Equation]
otherwise it is a bad epoch.
Note that ln(det( I + λ − 1 K
[ NT ] , [ NT ] )) ≤ 2 γ NT by definition of γ NT , i.e., the maximum information gain.
Since ln( det( I + λ − 1 K [ t 1] , [ t 1] ) det( I )
[Equation]
= ln(det
[Equation]
I + λ − 1 K [ NT ] , [ NT ] )) ≤ 2 γ NT , and due to the pigeonhole principle, there can be at most 2 γ NT bad epochs.
If the instantaneous regret r t is incurred during a good epoch, we have
[Equation]
where the second inequality is due to
Lemma A.
[Equation]
and the last inequality is due to the definition of good epoch,
i
[Equation]
e.,
[Equation]
Define α NT :=
[Equation]
λ ∥ θ ⋆ ∥ + (cid:112) 2 ln( N/δ ) + ln(det( K [ NT ] , [ NT ] /λ + I )) .
Then using standard arguments, the cumulative regret incurred in all good epochs can be bounded by,
[Equation]
where the third inequality is due to Cauchy-Schwartz and Lemma A.3, and the forth is due to the definition of maximum information gain γ NT .
Then we look at the regret incurred during bad epochs.
Consider some bad epoch p
[Equation]
and the cumulative regret incurred during this epoch can be bounded by
[Equation]
where the last inequality is due to our event-trigger in Eq (2).
Since there can be at most 2 γ NT bad epochs, the cumulative regret incurred in all bad epochs
[Equation]
Combining cumulative regret incurred during both good and bad epochs, we have
[Equation]
C.2 Communication Upper Bound
For some α > 0 , there can be at most ⌈ NTα
⌉ epochs with length larger than α .
Based on our event-trigger design, we know that ( |D t p ( i t p ) | −
[Equation]
D t
[Equation]
− 1 ( i t p
[Equation]
| )
ln
[Equation]
det
[Equation]
) det
[Equation] [Equation]
] ) ) ≥
[Equation]
det
[Equation]
) det
[Equation]
K
[Equation]
1( itp ) , D tp − 1( itp ) ) ) ≥ D for any epoch p ∈
[ B ] , where i t p is the client who triggers the global synchronization at time step t p .
Then if the length of certain epoch p is smaller than α , i.e., t p − t p − 1 ≤ α , we have ln( det( I + λ − 1 K
[ tp ] , [ tp ] ) det( I
[Equation]
λ − 1 K [ tp − 1] , [ tp − 1] ) ) ≥ NDα .
Since
[Equation]
I + λ − 1 K [ t 1] , [ t 1] ) )
+ · · · + ln( det( I + λ − 1 K [ t B ] , [ t B ] ) det( I + λ − 1 K [ tB − 1] , [ tB − 1] ) ) ≤ ln(det( I + λ − 1 K
[ NT ] , [ NT ] )) ≤ 2 γ NT , the total number of such epochs is upper bounded by ⌈ 2 γ NT α ND ⌉ .
Combining the two terms, the total number of epochs can be bounded by,
[Equation]
where the LHS can be minimized using the AM-GM inequality, i.e., B ≤ (cid:113)
[Equation]
=
[Equation]
D .
To obtain the optimal order of regret, we set D
= O ( T Nγ NT ) , so that R NT
= O (cid:0) √ NT ( ∥ θ ⋆ ∥√ γ NT + γ NT ) (cid:1) .
And the total number of epochs B = O ( √ Nγ NT ) .
However, we should note that as
DisKernelUCB communicates all the unshared raw data at each global synchronization, the total communication cost mainly depends on when the last global synchronization happens.
Since the sequence of candidate sets {A t } t ∈ [ NT ] , which controls the growth of determinant, is an arbitrary subset of A , the time of last global synchronization could happen at the last time step t =
NT .
Therefore, C T
=
O
( N 2 Td ) in such a worst case.
D Derivation of the Approximated Mean and Variance in Section
For simplicity, subscript t is omitted in this section.
The approximated Ridge regression estimator for dataset { ( x s , y s ) } s ∈D is formulated as
[Equation]
where D denotes the sequence of time indices for data in the original dataset, S ⊆
D denotes the time indices for data in the dictionary, and P S ∈ R
p × p denotes the orthogonal projection matrix defined by S .
Then by taking derivative and setting it to zero, we have ( P S Φ ⊤D Φ D P S + λ I )˜ θ
=
P S Φ ⊤D y D , and thus ˜ θ =
˜
A − 1 ˜ b , where
˜ A =
P S Φ ⊤D Φ D P S
+ λ I and ˜
b = P S Φ ⊤D y D .
Hence, the approximated mean reward and variance for some arm x are
[Equation]
To obtain their kernelized representation, we rewrite
[Equation]
where ρ := 1 λ ( y D − Φ D P S ˜ θ ) = 1 λ ( y D − Φ D P S P S Φ ⊤D ρ ) .
Solving this equation, we get ρ =
( Φ D P S P S Φ ⊤D
+ λ I ) − 1 y D .
Note that P S P S = P S , since projection matrix P S is idempotent.
Moreover, we have ( Φ ⊤ Φ + λ I )
Φ ⊤ =
Φ ⊤ ( ΦΦ ⊤ + λ I ) , and ( Φ ⊤ Φ + λ I ) − 1 Φ ⊤ =
Φ ⊤ ( ΦΦ ⊤ + λ I ) − 1 .
Therefore, we can rewrite the approximated mean for some arm x as
[Equation]
To derive the approximated variance, we start from the fact that ( P S Φ ⊤D Φ D P S + λ I ) ϕ
( x ) =
P S Φ ⊤D Φ D P
[Equation]
ϕ ( x ) + λϕ ( x ) , so
[Equation]
Then we have ϕ ( x
[Equation]
⊤
[Equation]
( x
[Equation]
⊤D + λ I ) − 1 Φ D P S ϕ ( x ) + λ ( P S Φ ⊤D
Φ D P S + λ I ) − 1 ϕ
( x ) (cid:9) ⊤ (cid:8) P S Φ
[Equation]
Φ
[Equation]
S
[Equation]
S Φ
[Equation]
) − 1 Φ
[Equation]
P S ϕ
( x ) + λ ( P S Φ
[Equation]
Φ
[Equation]
P S + λ I ) − 1 ϕ
( x ) (cid:9)
=
ϕ ( x ) ⊤ P S Φ ⊤D
( Φ D P S P S Φ ⊤D
+ λ I ) − 1 Φ D P S P S Φ ⊤D ( Φ D P S P S Φ ⊤D
+ λ I ) − 1 Φ D P S ϕ
( x ) + 2 λϕ ( x ) ⊤ P S Φ ⊤D
( Φ D P S P S
[Equation]
⊤D + λ I ) − 1 Φ D P S ( P S Φ ⊤D Φ D P S
+ λ I ) − 1 ϕ
( x ) + λϕ ( x ) ⊤ ( P S Φ ⊤D Φ D P S
+ λ I ) − 1 λ
I ( P S Φ ⊤D Φ D P S
+ λ I ) − 1 ϕ
( x ) =
ϕ ( x ) ⊤ P S Φ ⊤D ( Φ D P S P S Φ ⊤D
+ λ I ) − 1 Φ D P S ϕ ( x ) + λϕ ( x ) ⊤ ( P S Φ ⊤D Φ D P S
+ λ I ) − 1 ϕ
( x )
By rearranging terms, we have
[Equation]
E Proof of Lemma 4.
In the following, we analyze the ϵ t,i -accuracy of the dictionary for all t, i .
At the time steps when global synchronization happens, i.e., t p for p ∈
[ B ] , S t p is sampled from [ t p ] =
D t p ( i ) using approximated variance ˜ σ 2 t p − 1 ,i .
In this case, the accuracy of the dictionary only depends on the RLS procedure, and Calandriello et al.
[4] have already showed that the following guarantee on the accuracy and size of dictionary holds ∀ t ∈ { t p } p ∈
[ B ] .
Lemma E.1 (Lemma 2 of [4]) .
Under the condition that ¯ q = 6 1+ ϵ 1 − ϵ log(4 NT/δ ) /ϵ 2 , for some ϵ ∈ [0 , 1) , with probability at least 1 − δ , we have ∀ t ∈ { t p } p ∈ [ B ] that the dictionary { ( x s , y s ) } s ∈S t is ϵ -accurate w.r.t.
{ ( x s , y s ) } s ∈D t
( i ) , and 1 − ϵ 1+ ϵ σ 2 t ( x ) ≤ ˜ σ 2 t ( x ) ≤ 1+ ϵ 1 − ϵ σ 2 t ( x ) , ∀ x ∈ A .
Moreover, the size of dictionary |S t | ≤ 3(1 + L 2 /λ ) {+
+}
+ ϵ 1 − ϵ ¯ q ˜ d , where ˜ d :=
Tr ( K [ NT ] , [ NT ] ( K [ NT ] , [ NT ] + λ I ) − 1 ) denotes the effective dimension of the problem, and it is known that ˜ d
= O ( γ NT ) [6].
Lemma E.1 guarantees that for all t ∈ { t p } p ∈ [ B ] , the dictionary has a constant accuracy, i.e., ϵ t,i =
ϵ, ∀ i .
In addition, since the dictionary is fixed for t / ∈ { t p } p ∈ [ B ] , its size S t
= O ( γ NT ) , ∀ t ∈ [ NT ] .
Then for time steps t / ∈ { t p } p ∈ [ B ] , due to the local update, the accuracy of the dictionary will degrade.
However, thanks to our event-trigger in Eq (4), the extent of such degradation can be controlled, i.e., a new dictionary update will be triggered before the previous dictionary becomes completely irrelevant.
This is shown in Lemma E.2 below.
Lemma E.2.
Under the condition that { ( x s , y s ) } s ∈S tp is ϵ -accurate w.r.t.
{ ( x s , y s ) } s ∈D tp ( i ) , ∀ t ∈ [ t p + 1 , t p +1 ] , i ∈
[ N ] , S t p is (cid:0) ϵ + 1 − 1 1+ 1+ ϵ 1 − ϵ D (cid:1)
-accurate w.r.t.
D t ( i ) .
Combining Lemma E.1 and Lemma E.2 finishes the proof.
Proof of Lemma E.2.
Similar to [3], we can rewrite the ϵ -accuracy condition of S t p w.r.t.
D t ( i ) for t ∈ [ t p + 1 , t p +1 ] as
[Equation]
t ( i ) Φ D
[Equation]
i
[Equation]
λ
I
[Equation]
∥ ≤ ϵ
[Equation]
where ψ s = ( Φ ⊤D t ( i ) Φ D t ( i )
+ λ I ) − 1 / 2 ϕ s .
Notice that the second term in the norm has weight −because the dictionary S t p is fixed after t p .
With triangle inequality, now it suffices to bound ∥ (cid:88)
[Equation]
We should note that the first term corresponds to the approximation accuracy of S t p w.r.t.
the dataset
D t p .
And under the condition that it is ϵ -accurate w.r.t.
D t p , we have ∥ (cid:80) s ∈D tp ( q s ˜ p s − 1) ψ s ψ ⊤ s ∥ ≤ ϵ .
The second term measures the difference between D t ( i ) compared with D t p , which is unique to our work.
We can bound it as follows.
[Equation]
We can further bound the term σ 2 tp,i ( x ) σ 2 t,i ( x ) using the threshold of the event-trigger in Eq (4).
For any x ∈ R d , σ
[Equation]
p
[Equation]
where the first inequality is due to Lemma A.4, the second is due to Lemma E.1, and the third is due to the event-trigger in Eq (4).
Putting everything together, we have that if S t p is ϵ -accurate w.r.t.
D t p , then it is (cid:0) ϵ + 1 − 1 1+ 1+ ϵ 1 − ϵ D (cid:1) -accurate w.r.t.
dataset D t ( i ) , which finishes the proof.
F Proof of Lemma 4.
To prove Lemma 4.
[Equation]
we need the following lemma.
Lemma F.1.
We have ∀ t, i that
[Equation]
with probability at least 1 − δ .
Proof of Lemma F
[Equation]
Recall that the approximated kernel Ridge regression estimator for θ ⋆ is defined as
[Equation]
where P S is the orthogonal projection matrix for the Nyström approximation
[Equation]
and ˜
A t
[Equation]
i =
P S Φ ⊤D t ( i ) Φ D t ( i ) P S
+ λ I .
Then our goal is to bound
[Equation]
˜
[Equation]
t,i −
[Equation]
⊤ ˜
[Equation]
t,i
[Equation]
Bounding the first term To bound the first term, we begin with rewriting
[Equation]
and by substituting this into the first term
[Equation]
we have
[Equation]
where the first inequality is due to Cauchy Schwartz
[Equation]
and the last inequality is because ∥ ˜
A − 1 / 2 t,
i
[Equation]
Bounding the second term By applying Cauchy-Schwartz inequality to the second term
[Equation]
we have
[Equation]
Note that P S
A t,i P S =
P S ( Φ ⊤D t ( i ) Φ D t ( i )
+ λ I ) P S =
˜
A t,i + λ ( P S − I ) and P S ⪯ I , so we have ∥
[Equation]
t
[Equation]
Then using the self-normalized bound derived for Lemma B.3, the term ∥
A − 1 / 2 t,i Φ ⊤D
[Equation]
( i
[Equation]
∥ Φ ⊤D
[Equation] [Equation]
,i can be bounded by
[Equation]
for ∀ t, i , with probability at least 1 − δ .
Combining everything finishes the proof.
Now we are ready to prove Lemma 4.2 by further bounding the term ∥ Φ D t ( i ) ( I − P S tp ) ∥ .
Proof of Lemma 4.2.
Recall that ¯ S t,i ∈
R |D t ( i ) |×|D t
( i ) | denotes the diagonal matrix, whose s -th diagonal entry equals to q s √ ˜ p s ,
where q s = 1 if s ∈ S t p and 0 otherwise (note that for s / ∈ S t p , we set ˜
p s = 1 , so q s / ˜ p s = 0 ).
Therefore, ∀ s ∈ D t ( i ) \ D t p , q s = 0 , as the dictionary is fixed after t p .
We can rewrite Φ ⊤D t ( i ) ¯ S ⊤ t,i ¯ S t,i Φ D t ( i ) =
(cid:80) s
∈D t
( i ) q s ˜ p s ϕ s ϕ ⊤ s , where ϕ s := ϕ ( x s ) .
Then by definition of the spectral norm ∥·∥ , and the properties of the projection matrix P S tp , we have ∥
[Equation]
(cid:1) .
Moreover, due to Lemma E.2, we know S t p is ϵ t,i -accurate w.r.t.
D t ( i ) for t ∈ [ t p + 1 , t p +1 ] , where ϵ t,i = (cid:0) ϵ + 1 − 1 1+ 1+ ϵ 1 − ϵ D (cid:1) , so we have I − P S tp ⪯ λ 1 − ϵ t,i ( Φ ⊤D t ( i ) Φ D t ( i )
+ λ I ) − 1 by the property of ϵ -accuracy (Proposition 10 of [3]).
Therefore, by substituting this back to Eq (6), we have
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
λ −
[Equation] [Equation]
which finishes the proof.
G Proof of Theorem 4.3: Regret and Communication Cost of Approx-DisKernelUCB G.1 Regret Analysis
Consider some time step t ∈
[ t p − 1 + 1 , t p ] , where p ∈ [ B ] .
Due to Lemma 4.2, i.e., the confidence ellipsoid for approximated estimator, and the fact that x t = arg max x ∈A t,i ˜ µ t − 1 ,i ( x )+ α t − 1 ,i ˜ σ t − 1 ,i ( x ) , we have
[Equation]
i ( x t ) , and thus r t = f ( x ⋆t ) − f ( x t ) ≤ 2 α t − 1 ,i ˜ σ t − 1 ,i ( x t ) , where
[Equation]
Note that, different from Appendix C, the α t − 1 ,i term now depends on the threshold D and accuracy constant ϵ , as a result of the approximation error.
As we will see in the following paragraphs, their values need to be set properly in order to bound α t − 1 ,i .
We begin the regret analysis of Approx-DisKernelUCB with the same decomposition of good and bad epochs as in Appendix C.1, i.e., we call the p -th epoch a good epoch if ln( det( I
[Equation]
[Equation] [Equation]
det
[Equation]
− 1] , [
[Equation]
− 1] ) ) ≤ 1 , otherwise it is a bad epoch.
Moreover, due to the pigeon-hold principle, there can be at most 2 γ NT bad epochs.
As we will show in the following paragraphs, using Lemma E.1, we can obtain a similar bound for the cumulative regret in good epochs as that in Appendix C.1, but with additional dependence on D and ϵ .
The proof mainly differs in the bad epochs, where we need to use the event-trigger in Eq (4) to bound the cumulative regret in each bad epoch.
Compared with Eq (2), Eq (4) does not contain the number of local updates on each client since last synchronization.
, and as mentioned in Section 4.2, this introduces a √ T factor in the regret bound for bad epochs in place of the √ γ NT term in
Appendix C.1.
Cumulative Regret in Good Epochs Let’s first consider some time step t in a good epoch p , i.e., t ∈ [ t p − 1 + 1 , t p ] , and we have the following bound on the instantaneous regret
[Equation]
where the second inequality is because the (approximated) variance is non-decreasing, the third inequality is due to Lemma E.1, the forth is due to Lemma A.2, and the last is because in a good epoch, we have
[Equation]
Therefore, the cumulative regret incurred in all good epochs, denoted by R good , is upper bounded by
[Equation]
where α NT := (cid:32) 1 (cid:114) − ϵ + 1 1+1+ ϵ 1 − ϵD +(cid:33) √ λ
∥
θ ⋆ ∥
+ R (cid:112) 4 ln N/δ + 2 ln det((
+ λ )
I + K
[ NT ] , [ NT ] ) .
Cumulative Regret in Bad Epochs
The cumulative regret incurred in this bad epoch is
[Equation]
[Table]
where the third inequality is due to the Cauchy-Schwartz inequality, the forth is due to our eventtrigger in Eq (4), the fifth is due to our assumption that clients interact with the environment in a round-robin manner, the sixth is due to the Cauchy-Schwartz inequality again, and the last is due to the fact that there can be at most 2 γ NT bad epochs.
Combining cumulative regret incurred during both good and bad epochs, we have
[Equation]
G.2 Communication Cost Analysis
Consider some epoch p .
We know that for the client i who triggers the global synchronization, we have
[Equation]
Then by summing over B epochs, we have
[Equation]
Now we need to bound the ratio σ
[Equation]
[Equation] [Equation]
s
[Equation]
for s ∈ [ t p
[Equation]
1 + 1 , t p ] .
[Equation]
Note that for the client who triggers the global synchronization, we have
[Equation]
x s ) <
D , i.e., one time step before it triggers the synchronization at time t p .
Due to the fact that the (approximated) posterior variance cannot exceed L 2 /λ , we have (cid:80) s ∈D tp
( i ) \D tp − 1 ( i ) ˜
σ 2 t p [Equation] 1 ( x s ) < D + L 2 /λ .
For the other N − 1 clients, we have
[Equation]
i )
[Equation]
2 t
[Equation]
( x s ) < D .
Summing them together, we have
[Equation]
for the p -th epoch.
By substituting this back, we have
[Equation]
Therefore,
[Equation]
and thus the total number of epochs B < 1+ ϵ 1 − ϵ [ 1 D + 1+ ϵ 1 − ϵ ( N
[Equation]
L
[Equation]
[Equation]
γ NT .
By setting D = 1 N , we have
[Equation]
because N ≥ 1 .
Moreover, to ensure − ϵ +
[Equation]
> 0 , we need to set the constant ϵ < 1 / 3 .
Therefore,
[Equation]
and the total number of global synchronizations B = O
( Nγ NT ) .
Since for each global synchronization, the communication cost is O
( Nγ 2 NT ) , we have
[Equation]
H Experiment Setup
Synthetic dataset We simulated the distributed bandit setting defined in Section 3.1, with d = 20 , T = 100 , N = 100 ( NT = 10 4 interactions in total).
In each round l ∈ [ T ] , each client i ∈ [ N ] (denote t = N ( l − 1) + i ) selects an arm from candidate set A t , where A t is uniformly sampled from a ℓ 2 unit ball, with |A t | = 20 .
Then the corresponding reward is generated using one of the following reward functions:
[Equation]
where the parameter θ ⋆ is uniformly sampled from a ℓ 2 unit ball.
UCI Datasets To evaluate Approx-DisKernelUCB’s performance in a more challenging and practical scenario, we performed experiments using real-world datasets: MagicTelescope, Mushroom and Shuttle from the UCI Machine Learning Repository [8].
To convert them to contextual bandit problems, we pre-processed these datasets following the steps in [12].
In particular, we partitioned the dataset in to 20 clusters using k-means, and used the centroid of each cluster as the context vector for the arm and the averaged response variable as mean reward (the response variable is binarized by associating one class as 1 , and all the others as 0 ).
Then we simulated the distributed bandit learning problem in Section 3.1 with |A t | = 20 , T = 100 and N = 100 ( NT = 10 4 interactions in total).
MovieLens and Yelp dataset Yelp dataset, which is released by the Yelp dataset challenge, consists of 4.7 million rating entries for 157 thousand restaurants by 1.18 million users.
MovieLens is a dataset consisting of 25 million ratings between 160 thousand users and 60 thousand movies [13].
Following the pre-processing steps in [2], we built the rating matrix by choosing the topusers and top 10000 restaurants/movies and used singular-value decomposition (SVD) to extract a 10-dimension feature vector for each user and restaurant/movie.
We treated rating greater than 2 as positive.
We simulated the distributed bandit learning problem in Section 3.1 with T = 100 and
N = 100 ( NT = 10 4 interactions in total).
In each time step, the candidate set A t (with |A t | = 20 ) is constructed by sampling an arm with reward 1 and nineteen arms with reward 0 from the arm pool, and the concatenation of user and restaurant/movie feature vector is used as the context vector for the arm (thus d = 20 ).
I Lower Bound for Distributed Kernelized Contextual Bandits
First, we need the following two lemmas
Lemma I.1 (Theorem 1 of [29]) .
There exists a constant C > 0 , such that for any instance of kernelized bandit with L = S = R = 1 , the expected cumulative regret for KernelUCB algorithm is upper bounded by E [ R T ] ≤ C √
Tγ T , where the maximum information gain γ T
= O (cid:0) (ln( T ))
d +1 (cid:1) for Squared Exponential kernels.
Lemma I.2 (Theorem 2 of [23]) .
There always exists a set of hard-to-learn instances of kernelized bandit with L = S = R = 1 , such that for any algorithm, for a uniformly random instance in the set, the expected cumulative regret E [ R T ] ≥ c (cid:112) T (ln( T )) d/ 2 for Squared Exponential kernels, with some constant c .
Then we follow a similar procedure as the proof for Theorem 2 of [30] and Theorem 5.3 of [14], to prove the following lower bound results for distributed kernelized bandit with Squared Exponential kernels.
Theorem I.3.
For any distributed kernelized bandit algorithm with expected communication cost less than O (
N (ln( T )) 0 .
25 d +0 . 5 ) , there exists a kernelized bandit instance with Squared Exponential kernel, and L = S = R = 1 , such that the expected cumulative regret for this algorithm is at least Ω( N (cid:112) T (ln( T )) d/ 2 ) .
Proof of Theorem I.3.
Here we consider kernelized bandit with Squared Exponential kernels.
The proof relies on the construction of a auxiliary algorithm, denoted by AuxAlg , based on the original distributed kernelized bandit algorithm, denoted by DisKernelAlg , as shown below.
For each agent i ∈ [ N ] , AuxAlg performs DisKernelAlg , until any communication happens between client i and the server, in which case, AuxAlg switches to the single-agent optimal algorithm, i.e., the KernelUCB algorithm that attains the rate in Lemma I.1.
Therefore, AuxAlg is a single-agent bandit algorithm, and the lower bound in Lemma I.2 applies: the cumulative regret that AuxAlg incurs for some agent i ∈
[ N ] is lower bounded by
[Equation]
and by summing over all N clients, we have
[Equation]
For each client i ∈ [ N ] , denote the probability that client i will communicate with the server as p i , and p :=
(cid:80) Ni =
1 p i .
Note that before the communication, the cumulative regret incurred by AuxAlg is the same as DisKernelAlg , and after the communication happens, the regret incurred by AuxAlg is the same as KernelUCB, whose upper bound is given in Lemma I.1.
Therefore, the cumulative regret that AuxAlg incurs for client i can be upper bounded by
[Equation]
and by summing over N clients, we have
[Equation]
Combining the upper and lower bounds for E [ R AuxAlg ] [Equation] we have
[Equation]
Therefore, for any DisKernelAlg with number of communications p ≤ N
[Equation]
O (
N (ln(
[Equation]
+0 . 5 ) , we have
[Equation]