Communication Efficient Distributed Learning for Kernelized Contextual Bandits
Anonymous Author(s)
AffiliationAddress email
Abstract
We tackle the communication efficiency challenge of learning kernelized contextualbandits in a distributed setting.
Despite the recent advances in communicationefficient distributed bandit learning, existing solutions are restricted to simplemodels like multi-armed bandits and linear bandits, which hamper their practicalutility.
In this paper, instead of assuming the existence of a linear reward mappingfrom the features to the expected rewards, we consider non-linear reward mappings, by letting agents collaboratively search in a reproducing kernel Hilbert space(RKHS).
This introduces significant challenges in communication efficiency asdistributed kernel learning requires the transfer of raw data, leading to a communication cost that grows linearly w.r t. time horizon T .
We addresses this issue byequipping all agents to communicate via a common Nyström embedding that getsupdated adaptively as more data points are collected.
We rigorously proved thatour algorithm can attain sub-linear rate in both regret and communication cost.
Introduction
Contextual bandit algorithms have been widely used for a variety of real-world applications, includingrecommender systems [20], display advertisement [21] and clinical trials [11].
While most existingbandit solutions assume a centralized setting (i.e., all the data reside in and all the actions aretaken by a central server), there is increasing research effort on distributed bandit learning lately[30, 10, 24, 17, 19], where N clients, under the coordination of a central server, collaborate tominimize the overall cumulative regret incurred over a finite time horizon T .
In many distributedapplication scenarios, communication is the main bottleneck, e.g., communication in a network ofmobile devices can be slower than local computation by several orders of magnitude [16].
Therefore,it is vital for distributed bandit learning algorithms to attain sub-linear rate (w.r.t.
time horizon T ) inboth cumulative regret and communication cost.
However, prior works in this line of research are restricted to linear models [30], which could oversimplify the problem and thus leads to inferior performance in practice.
In centralized setting, kernelizedbandit algorithms, e.g., KernelUCB [29] and IGP-UCB [6], are proposed to address this issue bymodeling the unknown reward mapping as a non-parametric function lying in a reproducing kernel
Hilbert space (RKHS), i.e., the expected reward is linear w.r.t.
an action feature map of possiblyinfinite dimensions.
Despite the strong modeling capability of kernel method, collaborative exploration in the RKHS gives rise to additional challenges in designing a communication efficient banditalgorithm.
Specifically, unlike distributed linear bandit where the clients can simply communicatethe d × d sufficient statistics [30], where d is the dimension of the input feature vector, the jointkernelized estimation of the unknown reward function requires communicating either 1) the p × psufficient statistics in the RKHS, where p is the dimension of the RKHS that is possibly infinite, or 2)the set of input feature vectors that grows linearly w.r.t.
T .
Neither of them is practical due to thehuge communication cost.
In this paper, we propose the first communication efficient algorithm for distributed kernel bandits,which tackles the aforementioned challenge via a low-rank approximation of the empirical kernelmatrix.
In particular, we extended the Nyström method [22] to distributed learning for kernelizedcontextual bandits.
In this solution, all clients first project their local data to a finite RKHS spannedby a common dictionary, i.e., a small subset of the original dataset, and then they only need tocommunicate the embedded statistics for collaborative exploration.
To ensure effective regretreduction after each communication round, as well as ensuring the dictionary remains representativefor the entire distributed dataset throughout the learning process, the frequency of dictionary updateand synchronization of embedded statistics is controlled by measuring the amount of new informationeach client has gained since last communication.
We rigorously prove that the proposed algorithmincurs an O ( N 2 γ 3 NT ) communication cost, where γ NT is the maximum information gain that isknown to be O (cid:0) log( NT ) (cid:1) for kernels with exponentially decaying eigenvalues, which includes themost commonly used Gaussian kernel, while attaining the optimal O ( √ NTγ NT ) cumulative regret.
Related Works
To balance exploration and exploitation in stochastic linear contextual bandits, LinUCB algorithm[20, 1] is commonly used, which selects arm optimistically w.r.t.
a constructed confidence set onthe unknown linear reward function.
By using kernels and Gaussian processes, studies in [26, 29, 6]further extend UCB algorithms to non-parametric reward functions in RKHS, i.e., the feature mapassociated with each arm is possibly infinite.
Recent years have witnessed increasing research efforts in distributed bandit learning, i.e., multipleagents collaborate in pure exploration [15, 27, 7], or regret minimization [24, 30, 19].
They mainlydiffer in the relations of learning problems solved by the agents (i.e., homogeneous vs., heterogeneous)and the type of communication network (i.e., peer-to-peer (P2P) vs., star-shaped).
Most of these worksassume linear reward functions, and the clients communicate by transferring the O ( d 2 ) sufficientstatistics.
Korda et al.
[18] considered a peer-to-peer (P2P) communication network and assumedthat the clients form clusters, i.e., each cluster is associated with a unique bandit problem.
Huanget al.
[17] considered a star-shaped communication network as in our paper, but their proposedphase-based elimination algorithm only works in the fixed arm set setting.
The closest worksto ours are [30, 10, 19], which proposed event-triggered communication protocols to obtain sublinear communication cost over time for distributed linear bandits with a time-varying arm set.
Incomparison, distributed kernelized contextual bandits still remain under-explored.
The only existingwork in this direction [9] considered heterogeneous agents, where each agent is associated with anadditional feature describing the task similarity between agents.
However, they assumed a localcommunication setting, where the agent immediately shares the new raw data point to its neighborsafter each interaction, and thus the communication cost is still linear over time.
Another closely related line of works is kernelized bandits with approximation, where Nyströmmethod is adopted to improve computation efficiency in a centralized setting.
Calandriello et al.
[3]proposed an algorithm named BKB, which uses Ridge Leverage Score sampling (RLS) to re-samplea new dictionary from the updated dataset after each interaction with the environment.
A recent workby Zenati et al.
[31] further improved the computation efficiency of BKB by adopting an onlinesampling method to update the dictionary.
However, both of them updated the dictionary at each timestep to ensure the dictionary remains representative w.r.t.
the growing dataset, and therefore are notapplicable to our problem.
This is because the dataset is stored cross clients in a distributed manner,and projecting the dataset to the space spanned by the new dictionary requires communication withall clients, which is prohibitively expensive in terms of communication.
Calandriello et al.
[4] alsoproposed a variant of BKB, named BBKB, for batched Gaussian process optimization.
BBKB onlyneeds to update the dictionary occasionally according to an adaptive schedule, and thus partiallyaddresses the issue mentioned above.
However, as BBKB works in a centralized setting, their adaptiveschedule can be computed based on the whole batch of data, while in our decentralized setting, eachclient can only make the update decision according to the data that is locally available.
Moreover,in BBKB, all the interactions are based on a fixed model estimation over the whole batch, which ismentioned in their Appendix A.4 as a result of an inherent technical difficulty.
In comparison, ourproposed method effectively addresses this difficulty with improved analysis, and thus allows eachclient to utilize newly collected data to update its model estimation on the fly.
Preliminaries
In this section, we first formulate the problem of distributed kernelized contextual bandits.
Then,as a starting point, we propose and analyze a naive UCB-type algorithm for distributed kernelizedcontextual bandit problem, named DisKernelUCB.
This demonstrates the challenges in designing acommunication efficient algorithm for this problem, and also lays down the foundation for furtherimprovement on communication efficiency in Section 4.
3.1 Distributed Kernelized Contextual Bandit Problem
Consider a learning system with 1) N clients that are responsible for taking actions and receivingfeedback from the environment, and 2) a central server that coordinates the communication amongthe clients.
The clients cannot directly communicate with each other, but only with the central server,i.e., a star-shaped communication network.
Following prior works [30, 10], we assume the N clientsinteract with the environment in a round-robin manner for a total number of T rounds.
Specifically, at round l ∈ [ T ] , each client i ∈ [ N ] chooses an arm x t from a candidate set A t ,and then receives the corresponding reward feedback y t
= f ( x t ) + η t ∈ R , where the subscriptt := N ( l − 1)+ i indicates this is the t -th interaction between the learning system and the environment,and we refer to it as time step t 1 .
Note that A t is a time-varying subset of A ⊆ R d that is possiblyinfinite, f denotes the unknown reward function shared by all the clients, and η t denotes the noise.
Denote the sequence of indices corresponding to the interactions between client i and the environmentup to time t as N t ( i ) = { 1 ≤ s ≤ t : i s =
i } (if s mod N = 0 , then i s =
N ; otherwise i s =
s mod N ) for t = 1 , 2 , . .
, NT .
By definition, |N
Nl ( i ) |
= l, ∀ l ∈ [ T ] , i.e., the clients have equalnumber of interactions at the end of each round l .
Kernelized Reward Function We consider an unknown reward function f that lies in a RKHS, denoted as H , such that the reward can be equivalently written as
[Equation]
where θ ⋆ ∈ H is an unknown parameter, and ϕ : R d → H is a known feature map associated with
H .
We assume η t is zero-mean R -sub-Gaussian conditioned on σ (cid:0) ( x s , η s ) s ∈N t − 1 ( i t ) , x t (cid:1) , ∀ t , whichdenotes the σ -algebra generated by client i t ’s previously pulled arms and the corresponding noise.
In addition, there exists a positive definite kernel k ( · , · ) associated with H , such that k ( x , x ′ )
=
ϕ ( x ) ⊤ ϕ ( x ′ ) , and we assume k ( · , · ) is bounded by κ 2 .
Regret and Communication Cost The goal of the learning system is to minimize the cumulative(pseudo) regret for all N clients, i.e., R NT
=
(cid:80) NTt =1 r t , where r t = max x ∈A t ϕ ( x ) ⊤ θ ⋆ − ϕ ( x t ) ⊤ θ ⋆ .
Meanwhile, the learning system also wants to keep the communication cost C NT low, which ismeasured by the total number of scalars being transferred across the system up to time step NT .
3.2 Distributed Kernel UCB
As a starting point to studying the communication efficient algorithm in Section 4 and demonstrate thechallenges in designing a communication efficient distributed kernelized contextual bandit algorithm,here we first introduce and analyze a naive algorithm where the N clients collaborate on learningthe exact parameters of kernel bandit, i.e., the mean and variance of estimated reward.
We namethis algorithm
Distributed Kernel UCB, or DisKernelUCB for short, and its description is given in
Algorithm 1.
Arm Selection For each round l ∈
[ T ] , when client i ∈ [ N ] interacts with the environment, i.e., thet -th interaction between the learning system and the environment where t = N ( l − 1)
+ i , it choosesarm x t ∈
A t based on the UCB of the mean estimator (line 5):
[Equation]
where ˆ µ t,i ( x ) and ˆ σ 2 t,i ( x ) denote client i ’s local estimated mean reward for arm x ∈ A and itsvariance, and α t − 1 ,i is a carefully chosen scaling factor to balance exploration and exploitation (see
Lemma 3.1 for proper choice).
To facilitate further discussion, for time step t ∈ [ NT ] , we denote the sequence of time indices forthe data points that have been used to update client i ’s local estimate as D t ( i ) , which include bothdata points collected locally and those shared by the other clients.
If the clients never communicate,
D t ( i ) =
N t ( i ) , ∀ t, i ; otherwise, N t
( i ) ⊂
D t ( i ) ⊆
[ t ] , with D t ( i ) =
[ t ] recovering the centralizedsetting, i.e., each new data point collected from the environment immediately becomes available toall the clients in the learning system.
The design matrix and reward vector for client i at time stept are denoted by X D t ( i ) =
[ x s ] ⊤ s ∈D t
( i ) ∈ R |D t ( i ) |×
d , y t,i =
[ y s ] ⊤ s ∈D t ( i ) ∈
R |D t ( i ) | , respectively.
Byapplying the feature map ϕ ( · ) to each row of X D t ( i ) , we obtain Φ D t ( i
) ∈ R |D t ( i ) |× p , where p isthe dimension of H and is possibly infinite.
Since the reward function is linear in H , client i canconstruct the Ridge regression estimator ˆ θ t,i = ( Φ ⊤D t ( i ) Φ D t ( i ) + λI ) − 1 Φ ⊤D t ( i )
y t,i , where λ > 0is the regularization coefficient.
This gives us the estimated mean reward and variance in primalform for any arm x ∈ A , i.e., ˆ µ t,i ( x ) =
ϕ ( x ) ⊤ A − 1 t,i b t,i and ˆ σ t,i ( x ) = (cid:113) ϕ ( x ) ⊤ A − 1 t,i ϕ ( x ) , where
A t,i =
Φ ⊤D t ( i ) Φ D t ( i )
+ λ
I and b t,i =
Φ ⊤D t ( i ) y
t,i .
Then using the kernel trick, we can obtain theirequivalence in the dual form that only involves entries of the kernel matrix, and avoids directlyworking on H which is possibly infinite:
[Equation]
⊤ (cid:0) K D t ( i ) , D t ( i ) + λI (cid:1) − 1 K D t ( i ) ( x ) where K D t ( i ) ( x ) =
Φ D t ( i ) ϕ
( x ) =
[ k ( x s , x )]
⊤ s ∈D t ( i ) ∈ R |D t ( i ) | , and K D t ( i ) , D t ( i ) =
Φ ⊤D t ( i ) Φ D t ( i ) =
[ k ( x s , x s ′ )] s,s ′ ∈D t
( i ) ∈ R |D t ( i ) |×|D t
( i ) | .
Communication Protocol To reduce the regret in future interactions with the environment,
the
N clients need to collaborate via communication, and a carefully designed communication protocol is essential in ensuring the communication efficiency.
In prior works like DisLinUCB[30], after each round of interaction with the environment, client i checks whether the event{ ( |D t ( i ) | − |D t last ( i ) | ) log( det( A t,i ) det( A t last ,i ) ) >
D } is true, where t last denotes the time step of lastglobal synchronization.
If true, a new global synchronization is triggered, such that the serverwill require all clients to upload their sufficient statistics since t last , aggregate them to compute { A t , b t } , and then synchronize the aggregated sufficient statistics with all clients, i.e., set{ A t,i , b t,i } =
{ A t , b t } , ∀ i ∈ [ N ] .
Using kernel trick, we can obtain an equivalent event-trigger in terms of the kernel matrix,
U
[Equation]
∆ D t ( i t ) ) (cid:19) > D (cid:27) .
where D > 0 denotes the predefined threshold value.
If event U t ( D ) is true (line 7), a globalsynchronization is triggered (line 7-10), where the local datasets of all N clients are synchronizedto { ( x s , y s ) } s ∈
[ t ] .
We should note that the transfer of raw data ( x s , y s ) is necessary for the updateof the kernel matrix and reward vector in line 6 and line 10, which will be used for arm selection atline 5.
This is an inherent disadvantage of kernelized estimation in distributed settings, which, aswe mentioned in Section 2, is also true for the existing distributed kernelized bandit algorithm [9].
Lemma 3.1 below shows that in order to obtain the optimal order of regret, DisKernelUCB incursa communication cost linear in T (proof given in the appendix), which is expensive for an onlinelearning problem.
Lemma 3.1 (Regret and Communication Cost of DisKernelUCB) .
With threshold D =
T Nγ NT ,α t,i = √ λ
∥ θ ⋆ ∥
+ R (cid:112) 4 ln N/δ
+
ln det( I + K D t ( i ) , D t ( i ) /λ ) , we have
[Equation]
with probability at least 1 − δ , and
C NT = O ( TN 2 d ) .
where γ NT := max D⊂A : |D| = NT 12 log det( K D , D /λ + I ) is the maximum information gain after 174 NT interactions [6].
It is problem-dependent and can be bounded for specific arm set A and
kernelfunction k ( · , · ) .
For example, γ NT =
O
( d log( NT )) for linear kernel and γ NT = O (log( NT ) d +1 )for Gaussian kernel.
Remark 1.
In the distributed linear bandit problem, to attain O ( d √ NT ln( NT )) regret, DisLinUCB[30] requires a total number of O ( N 0 .
5 d log( NT )) synchronizations, and DisKernelUCB matchesthis result under linear kernel, as it requires O ( N 0 .
5 γ NT ) synchronizations.
We should note that thecommunication cost for each synchronization in DisLinUCB is fixed, i.e., O ( Nd 2 ) to synchronize thesufficient statistics with all the clients, so in total C NT
= O ( N 1 .
5 d 3 ln( NT )) .
However, this is notthe case for DisKernelUCB that needs to send raw data, because the communication cost for eachsynchronization in DisKernelUCB is not fixed, but depends on the number of unshared data points oneach client.
Even if the total number of synchronizations is small, DisKernelUCB could still incur
C NT
= O ( TN 2 d ) in the worse case.
Consider the extreme case where synchronization only happensonce, but it happens near NT , then we still have C NT
= O ( TN 2 d ) .
The time when synchronizationgets triggered depends on {A t } t ∈ [ NT ] , which is out of the control of the algorithm.
Therefore, in thefollowing section, to improve the communication efficiency of DisKernelUCB, we propose to let eachclient communicate embedded statistics in some small subspace during each global synchronization.
Approximated Distributed Kernel UCB
In this section, we propose and analyze a new algorithm that improves the communication efficiencyof DisKernelUCB using the Nyström approximation, such that the clients only communicate theembedded statistics during event-triggered synchronizations.
We name this algorithm Approximated
Distributed Kernel UCB, or Approx-DisKernelUCB for short.
Its description is given in Algorithm 2.
4.1 Algorithm
Arm selection For each round l ∈
[ T ] , when client i ∈ [ N ] interacts with the environment, i.e., thet -th interaction between the learning system and the environment where t :=
N ( l − 1) + i , insteadof using the UCB for the exact estimator in Eq (1), client i chooses arm x t ∈
A t that maximizes the
UCB for the approximated estimator (line 5):
[Equation]
where ˜ µ t − 1 ,i ( x ) and ˜ σ t − 1 ,i ( x ) are approximated using Nyeström method, and the statistics usedto compute these approximations are much more efficient to communicate as they scale with themaximum information gain γ NT instead of T .
Specifically, Nyström method works by projecting some original dataset D to the subspace definedby a small representative subset S ⊆ D , which is called the dictionary.
The orthogonal projectionmatrix is defined as
[Equation]
1: Input: threshold D
0 , regularization parameter δ
1) and kernel
2: Initialize ˜ µ 0 ,i ( x ) = 0 , ˜ σ 0 ,i ( x ) = (cid:112) k
( x , x ) , N 0 ( i )
=
D
( i ) = ∅ , ∀ i ∈
[ N ] ; S 0 = ∅ , t last =3: for round l = 1 , 2 , ..., T do 4: for client i = 1 , 2 , ..., N do 5:
[Client i ] selects arm x t ∈
A t according to Eq (3) and observes reward y t , where t :=
N ( l − 1) + i 6: [Client i ] updates Z ⊤D t ( i ); S t last Z D t ( i ); S t last and Z ⊤D t ( i ); S t last y D t ( i ) using (cid:0)
z ( x t ; S t last ) , y t (cid:1) ; sets
N t ( i ) =
N t − 1 ( i ) ∪ { t } , and D t ( i ) =
D t − 1 ( i ) ∪ { t } //
Global Synchronization 7: if the event U t ( D ) defined in Eq (4) is true then 8: [Clients ∀ i ] sample S t,i = RLS ( N t ( i ) , ¯ q, ˜ σ 2 t last ,i ) , and send { ( x s , y s ) } s ∈S t,i to server 9: [Server] aggregates and sends { ( x s , y s ) } s ∈S t back to all clients,
where S t = ∪ i ∈
[ N ] S t,i 10: [Clients ∀ i ] compute and send { Z ⊤N t ( i ); S t Z N t ( i ); S t , Z ⊤N t ( i ); S t y N t ( i ) } to server 11:
[Server] aggregates (cid:80) Ni
=
Z ⊤N t ( i ); S t Z N t ( i ); S t , (cid:80)
Ni =
Z ⊤N t ( i ); S t y N t ( i ) and sends it back 12: [Clients ∀ i ] updates Z ⊤D t ( i );
S t Z D t ( i ); S t and Z ⊤D t ( i ); S t y D t ( i ) ; sets D t ( i ) =
∪ Ni
=1 N t
( i ) =
[ t ] and t last =
t
We then take eigen-decomposition of K S , S =
UΛU ⊤ to rewrite the orthogonal projection as
P S =
Φ ⊤S UΛ − 1 / 2 Λ − 1 / 2 U ⊤ Φ S , and define the Nyström embedding function
[Equation]
which maps the data point x from R d to R |S| .
Therefore, we can approximate the Ridge regression estimator in Section 3.2 as ˜ θ t,i = ˜
A − 1 t,i ˜ b t,i ,where ˜
A t,i =
P S Φ ⊤D t ( i ) Φ D t ( i ) P S
+ λ I , and ˜ b t,i =
P S Φ ⊤D t ( i )
y D t ( i ) , and thus the approximatedmean reward and variance in Eq (3) can be expressed as ˜ µ t,i ( x ) =
ϕ
( x ) ⊤ ˜
A − 1 t,i ˜ b t,i and ˜ σ t,i ( x ) = 212 (cid:113) ϕ
( x ) ⊤ ˜
A − 1 t,i ϕ ( x ) , and their kernelized representation are (see appendix for detailed derivation)
[Equation]
S Z D t ( i ); S [ Z ⊤D t ( i ); S Z D t ( i ); S
+ λ I ] − 1 z ( x |S ) where Z D t ( i ); S ∈ R |D t
( i ) |×|S| is obtained by applying z ( · ; S ) to each row of X D t ( i ) , i.e., Z D t ( i ); S =
Φ D t ( i ) P 1 / 2 S .
We can see that the computation of ˜ µ t,i ( x ) and ˜ σ t,i ( x ) only requires the embeddedstatistics: matrix Z ⊤ D t ( i );
S Z D t ( i ); S ∈ R |S|×|S| and vector Z ⊤ D t ( i ); S y D t ( i )
∈
R |S| , which, as we willshow later, makes joint kernelized estimation among N clients much more efficient in communication.
After obtaining the new data point ( x t , y t ) , client i immediately updates both ˜ µ t − 1 ,i ( x ) and ˜ σ t − 1 ,i ( x )using the newly collected data point ( x t , y t ) , i.e., by projecting x t to the finite dimensional RKHSspanned by Φ S t last (line 6).
Recall that, we use N t ( i ) to denote the sequence of indices for datacollected by client i , and denote by D t ( i ) the sequence of indices for data that has been used to updateclient i ’s model estimation ˜ µ t,i .
Therefore, both of them need to be updated to include time step t .
Communication Protocol With the approximated estimator, the size of message being communicated across the learning system is reduced.
However, a carefully designed event-trigger is stillrequired to minimize the total number of global synchronizations up to time NT .
Since the clientscan no longer evaluate the exact kernel matrices in Eq (2), we instead use the event-trigger in Eq (4),which can be computed using the approximated variance from last global synchronization as,
[Equation]
Similar to Algorithm 1, if Eq (4) is true, global synchronization is triggered, where both the dictionaryand the embedded statistics get updated.
During synchronization, each client first samples a subset
S t ( i ) from N t ( i ) (line 8) using Ridge Leverage Score sampling (RLS) [3, 4], which is given in
Algorithm 3, and then sends { ( x s , y s ) } s ∈S t ( i ) to the server.
The server aggregates the received localsubsets to construct a new dictionary { ( x s , y s ) } s ∈S t , where S t = ∪ Ni =1 S t ( i ) , and then sends it backto all N clients (line 9).
Finally, the N clients use this updated dictionary to re-compute the embeddedstatistics of their local data, and then synchronize it with all other clients via the server (line 10-12).
Intuitively, in Algorithm 2, the clients first agree upon a common dictionary S t that serves as agood representation of the whole dataset at the current time t , and then project their local data tothe subspace spanned by this dictionary before communication, in order to avoid directly sendingthe raw data as in Algorithm 1.
Then using the event-trigger, each client monitors the amount ofnew knowledge it has gained through interactions with the environment from last synchronization.
When there is a sufficient amount of new knowledge, it will inform all the other clients to perform asynchronization.
As we will show in the following section, the size of S t scales linearly w.r.t.
themaximum information gain γ NT , and therefore it improves both the local computation efficiency oneach client, and the communication efficiency during the global synchronization.
4.2 Theoretical Analysis
Denote the sequence of time steps when global synchronization is performed, i.e., the event U t ( D )in Eq (4) is true, as { t p } Bp =1 , where B ∈ [ NT ] denotes the total number of global synchronizations.
Note that in Algorithm 2, the dictionary is only updated during global synchronization, e.g., at timet p , the dictionary { ( x s , y s ) } s ∈S tp is sampled from the whole dataset { ( x s , y s ) } s ∈
[ t p ] in a distributedmanner, and remains fixed for all the interactions happened at t ∈ [ t p + 1 , t p +1 ] .
Moreover, at timet p , all the clients synchronize their embedded statistics, so that D t p ( i ) =
[ t p ] , ∀ i ∈
[ N ] .
Since Algorithm 2 enables local update on each client, for time step t ∈ [ t p + 1 , t p ] , new data pointsare collected and added into D t ( i ) , such that D t ( i ) ⊇ [ t p ] .
This decreases the approximation
accuracyof S t p , as new data points may not be well approximated by S t p .
For example, in extreme cases, thenew data could be orthogonal to the dictionary.
To formally analyze the accuracy of the dictionary,we adopt the definition of ϵ -accuracy from [5].
Denote by ¯ S t,i ∈ R
|D
t ( i ) |×|D t
( i ) | a diagonal matrix,with its s -th diagonal entry equal to 1 √ ˜ p
s if s ∈ S t p and 0 otherwise.
Then if
[Equation]
we say the dictionary { ( x s , y s ) } s ∈S tp is ϵ t,i -accurate w.r.t.
dataset { ( x s , y s ) } s ∈D t
( i ) .
As shown below, the accuracy of the dictionary for Nyström approximation is essential as it affects thewidth of the confidence ellipsoid, and thus affects the cumulative regret.
Intuitively, in order to ensureits accuracy throughout the learning process, we need to 1) make sure the RLS procedure in line 8 of
Algorithm 2 that happens at each global synchronization produces a representative set of data samples,and 2) monitor the extent to which the dictionary obtained in previous global synchronization hasdegraded over time, and when necessary, trigger a new global synchronization to update it.
Comparedwith prior work that freezes the model in-between consecutive communications [4], the analysis ofϵ -accuracy for Approx-DisKernelUCB is unique to our paper and the result is presented below.
Lemma 4.1.
With ¯ q = 6 1+ ϵ 1 − ϵ log(4 NT/δ ) /ϵ 2 , for some ϵ ∈ [0 , 1) , and threshold D > 0 , Algorithm2 guarantees that the dictionary is accurate with constant ϵ t,i := (cid:0) ϵ + 1 − 1 1+ 1+ ϵ 1 − ϵ D (cid:1) , and its size|S t | =
O ( γ NT ) for all t ∈
[ NT ] .
Based on Lemma 4.1, we can construct the following confidence ellipsoid for unknown parameter θ ⋆ .
Lemma 4.2 (Confidence Ellipsoid of Approximated Estimator) .
Under the condition that ¯ q =6 1+ ϵ 1 − ϵ log(4 NT/δ ) /ϵ 2 , for some ϵ ∈ [0 , 1) , and threshold D > 0 , with probability at least 1 − δ , wehave ∀ t, i that [Equation]
[Equation]
Using Lemma 4.2, we obtain the regret and communication cost upper bound of ApproxDisKernelUCB, which is given in Theorem 4.3 below.
Theorem 4.3 (Regret and Communication Cost of Approx-DisKernelUCB) .
Under the same condition as Lemma 4.2, and by setting D = 1 N , ϵ < 13 , we have
[Equation]
with probability at least 1 − δ , and
[Equation]
Here we provide a proof sketch for Theorem 4.3, and the complete proof can be found in appendix.
Proof Sketch.
Similar to the analysis of DisKernelUCB in Section 3.2 and DisLinUCB from [30],the cumulative regret incurred by Approx-DisKernelUCB can be decomposed in terms of ‘good’and ‘bad’ epochs, and bounded separately.
Here an epoch refers to the time period in-between twoconsecutive global synchronizations, e.g., the p -th epoch refers to [ t p − 1 + 1 , t p ] .
Now consider animaginary centralized agent that has immediate access to each data point in the learning system,and denote by A t = (cid:80) ts =1 ϕ s ϕ ⊤ s for t ∈
[ NT ] the matrix constructed by this centralized agent.
We call the p -th epoch a good epoch if ln( det( I + λ − 1 K [ tp ] , [ tp ] ) det( I + λ − 1 K
[ tp − 1] , [ tp − 1] ) ) ≤ 1 , otherwise it is a badepoch.
Note that ln( det( I + λ − 1 K [ t 1] , [ t 1] ) det( I )
[Equation]
ln(det( I + λ − 1 K
[ NT ] , [ NT ] )) ≤ 2 γ NT , where the last equality is due to the matrix determinant lemma,and the last inequality is by the definition of the maximum information gain γ NT in Lemma 3.1.
Then based on the pigeonhole principle, there can be at most 2 γ NT bad epochs.
By combining Lemma E.1 and Lemma 4.2, we can bound the cumulative regret incurred during all good epochs, i.e., R good = O ( √ NTγ NT ) , which matches the optimalregret attained by the KernelUCB algorithm in centralized setting.
Our analysis deviates from that of DisKernelUCB in the bad epochs, because of the difference in theevent-trigger.
Previously, the event-trigger of DisKernelUCB directly bounds the cumulative regret each client incurs during a bad epoch, i.e., (cid:80) t ∈D tp ( i ) \D tp − 1 ( i ) ˆ σ t − 1 ,i ( x t ) ≤(cid:113) ( |D t p ( i ) | − |D t p − 1 ( i ) | ) log (cid:0) det( I + λ − 1 K D t
( i t ) , D t ( i t ) ) / det( I + λ − 1 K D t
( i t ) \ ∆ D t ( i t ) , D t ( i t ) \ ∆ D t ( i t ) ) (cid:1) < 296 √
D .
However, the event trigger of Approx-DisKernelUCB only bounds part ofit, i.e.,
[Equation]
[Equation]
|
[Equation]
D
[Equation]
) D , which leads to
R bad =
O ( √ Tγ NT N √ D ) that is slightly worse than that of DisKernelUCB, i.e., a √ Tfactor in place of the √ γ NT factor.
By setting D = 1 /N , we have R NT
= O ( √ NTγ NT ) .
Notethat, to make sure ϵ t,i = (cid:0) ϵ + 1 − 1 1+
[Equation]
+
[Equation]
1 N (cid:1) ∈ [0 , 1) is still well-defined, we can set ϵ < 1 / 3 .
For communication cost analysis, we bound the total number of epochs B by upper bounding thetotal number of summations like (cid:80) t p s =
t
p − 1 +1 ˆ σ 2 t p − 1 ( x s ) , over the time horizon NT .
Using Lemma
E.1, our event-trigger in Eq (4) provides a lower bound (cid:80) t p s =
t p − 1 +1 ˆ σ 2 t p − 1 ( x s ) ≥ 1 − ϵ 1+ ϵ D .
Then inorder to apply the pigeonhole principle, we continue to upper bound the summation over all epochs,(cid:80)
Bp
[Equation]
(cid:80) t p s
[Equation]
p
[Equation]
ˆ σ
[Equation]
p − 1 ( x s )
= (cid:80) Bp =1 (cid:80) t p s =
t p − 1 +1 ˆ σ 2 s − 1 ( x s ) ˆ σ
[Equation]
[Equation]
x s )
[Equation]
s − 1 ( x s ) by deriving a uniform boundfor the ratio
[Equation]
in terms of the communication threshold D on each client.
This leads to the following upper boundabout the total number of epochs B ≤ 1+ ϵ 1 − ϵ
[ 1 D + 1+ ϵ 1 − ϵ ( N + κ 2 / ( λD ))]2 γ NT , and with D = 1 /N , wehave C NT ≤ B · Nγ 2 NT
= O ( N 2 γ 3 NT ) , which completes the proof.
Remark 2.
Compared with DisKernelUCB’s O ( TN 2 d ) communication cost, Approx-DisKernelUCBremoves the linear dependence on T , but introduces an additional γ 3 NT dependence due to thecommunication of the embedded statistics.
In situations where γ NT ≪ T 1 / 3 d 1 / 3 , DisKernelUCBis preferable.
As mentioned in Lemma 3.1, the value of γ NT , which affects how much the data canbe compressed, depends on the specific arm set of the problem and the kernel function of the choice.
By Mercer’s Theorem, one can represent the kernel using its eigenvalues, and γ NT characterizeshow fast its eigenvalues decay.
Vakili et al.
[28] showed that for kernels whose eigenvalues decayexponentially, i.e., λ m =
O (exp( − m β e )) , for some β e > 0 , γ NT =
O (log 1+ 1 βe ( NT )) .
In thiscase, Approx-DisKernelUCB is far more efficient than DisKernelUCB.
This includes Gaussian kernel,which is widely used for GPs and SVMs.
For kernels that have polynomially decaying eigenvalues,i.e., λ m =
O ( m − β p ) , for some β p > 1 ,
γ NT
= O ( T 1 βp log 1 − 1 βp ( NT )) .
Then as long as β p > 3 ,
Approx-DisKernelUCB still enjoys reduced communication cost.
Experiments
In order to evaluate Approx-DisKernelUCB’s effectiveness in reducing communication cost, weperformed extensive empirical evaluations on both synthetic and real-world datasets, and the results(averaged over 3 runs) are reported in Figure 1, 2 and 3, respectively.
We included DisKernelUCB,
DisLinUCB [30], OneKernelUCB, and NKernelUCB [6] as baselines, where One-KernelUCB learnsa shared bandit model across all clients’ aggregated data where data aggregation happens immediatelyafter each new data point becomes available, and N-KernelUCB learns a separated bandit model foreach client with no communication.
For all the kernelized algorithms, we used the Gaussian kernelk ( x, y ) =
exp( − γ ∥ x − y ∥ 2 ) .
We did a grid search of γ ∈ { 0 . 1 , 1 , 4 } for kernelized algorithms,and set D = 20 for DisLinUCB and DisKernelUCB, D = 5 for Approx-DisKernelUCB.
For allalgorithms, instead of using their theoretically derived exploration coefficient α , we followed theconvention [20, 32] to use grid search for α in { 0 . 1 , 1 , 4 } .
Due to space limit, here we only presentthe experiment results and discussions. Details about the experiment setup are presented in appendix.
When examining the experiment results presented in Figure 1, 2 and 3, we can first look at thecumulative regret and communication cost of OneKernelUCB and NKernelUCB, which correspondto the two extreme cases where the clients communicate in every time step to learn a shared model,and each client learns its own model independently with no communication, respectively. OneKernelUCB achieves the smallest cumulative regret in all experiments, while also incurring the highestcommunication cost, i.e., O ( TN 2 d ) . This demonstrates the need of efficient data aggregation acrossclients for reducing regret. Second, we can observe that DisKernelUCB incurs the second highestcommunication cost in all experiments due to the transfer of raw data, as we have discussed in Remark1, which makes it prohibitively expensive for distributed setting.
On the other extreme, we can seethat DisLinUCB incurs very small communication cost thanks to its closed-form solution, but fails tocapture the complicated reward mappings in most of these datasets, e.g. in Figure 1(a), 2(b) and 3(a),it leads to even worse regret than NKernelUCB that learns a kernelized bandit model independentlyfor each client. In comparison, the proposed Approx-DisKernelUCB algorithm enjoys the best ofboth worlds in most cases, i.e., it can take advantage of the superior modeling power of kernels toreduce regret, while only requiring a relatively low communication cost for clients to collaborate. [Figure] [Figure]
[Figure]
On all the datasets, Approx-DisKernelUCB achieved comparable regret with DisKernelUCB thatmaintains exact kernelized estimators, and sometimes even getting very close to OneKernelUCB, e.g.,in Figure 1(b) and 2(a), but its communication cost is only slightly higher than that of DisLinUCB.
Conclusion
In this paper, we proposed the first communication efficient algorithm for distributed kernel banditsusing Nyström approximation.
Clients in the learning system project their local data to a finite RKHSspanned by a shared dictionary, and then communicate the embedded statistics for collaborativeexploration.
To ensure communication efficiency, the frequency of dictionary update and synchronization of embedded statistics are controlled by an event-trigger.
The algorithm is proved to incur
O ( N 2 γ 3 NT ) communication cost, while attaining the optimal O ( √ NTγ NT ) cumulative regret.
We should note that the total number of synchronizations required by Approx-DisKernelUCB is
Nγ NT , which is √ N worse than DisKernelUCB.
An important future direction of this work is toinvestigate whether this part can be further improved.
The lower bound analysis for the communicationcost of distributed contextual bandits still remains an open problem, and is an important futuredirection.
To the best of our knowledge, the only applicable lower bound states that, in order to havesmaller regret than the trivial O ( √ NT ) result, i.e., run N instances of optimal bandit algorithm withno communication, Ω( N ) communications is necessary [14].
In comparison, it is more interestingto know what the communication lower bound is in order to attain the optimal O ( √ T ) regret.
It isalso interesting to extend our algorithm to P2P setting, i.e., no central server to coordinate the updateof the shared dictionary and the exchange of embedded statistics, which may require utilizing localstructures in the network of clients, to approximate each block of the kernel matrix separately [25]. Acknowledgement This work is supported by NSF grants IIS-2128019, IIS-1838615, IIS-1553568, IIS-2107304, CMMI1653435, AFOSR grant and ONR grant 1006977.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] , see Remark 2(c)
Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] , in Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] , the complete proof is given in appendix
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
N
A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A
A Technical Lemmas
Lemma A.1 (Lemma 11 of [1]) .
Let { x t } ∞ t =1 be a sequence in R d ,
V ∈ R d × d a positive definitematrix, and define V t
= V + (cid:80) ts =1 x s x ⊤ s .
Then we have that
[Equation]
If ∥ x t ∥ 2 ≤ L, ∀ t , and λ min ( V ) ≥ max(1 , L 2 ) , then
[Equation]
Lemma A.2 (Lemma 12 of [1]) .
Let A , B and C be positive semi-definite matrices with finitedimension, such that A = B + C . Then, we have that: [Equation]
Lemma A.3 (Extension of Lemma A.2 to infinite dimension) .
Define positive definite matrices A = λ I + Φ ⊤ 1 Φ 1 + Φ ⊤ 2 Φ 2 and B = λ
I + Φ ⊤ 1 Φ 1 , where Φ ⊤ 1 Φ 1 , Φ ⊤ 2 Φ 2 ∈ R p × p and p is possiblyinfinite.
Then, we have that:
[Equation]
where K A = (cid:20) ΦΦ(cid:21) (cid:2) Φ ⊤ 1 , Φ ⊤ 2 (cid:3) and K B =
Φ 1 Φ ⊤ 1 .
Proof of Lemma A.3.
Similar to the proof of Lemma 12 of [1], we start from the simple case whenΦ ⊤ 2 Φ 2 = mm ⊤ , where m ∈ R p , and thus, using Cauchy-Schwartz inequality, we have
[Equation]
for any ϕ .
Then using the kernel trick, i.e., see the derivation of Equation 27 in [31], we have
[Equation]
which finishes the proof of this simple case.
Now consider Φ ⊤ 2 Φ 2 =
m 1 m ⊤ 1 + m 2 m ⊤ 2 +· · · + m t − 1 m ⊤ t − 1 .
{+Let’s define V s = B + m 1 m ⊤ 1 + m 
+}
m ⊤ 2 + · · · + m s − 1 m ⊤ s − 1 and K V s = 509   Φm ⊤. . .
[Equation]

[Equation]
, m s
[Equation]
and note that ϕ ⊤ Aϕ ϕ ⊤ Bϕ = ϕ ⊤
[Equation]
⊤
[Equation]
⊤
[Equation]
⊤
[Equation]
Bϕ .
Then we canapply the result for the simple case on each term in the product above, which gives us
[Equation]
which finishes the proof.
Lemma A.4 (Lemma 4 of [4]) .
For t > t last , we have for any x ∈ R d
[Equation]
B Confidence Ellipsoid for DisKernelUCB
In this section, we construct the confidence ellipsoid for DisKernelUCB as shown in Lemma B.1.
Lemma B.1 (Confidence Ellipsoid for DisKernelUCB) .
Let δ ∈ (0 , 1) .
With probability at least1 − δ , for all t ∈ [ NT ] , i ∈ [ N ] , we have
[Equation]
The analysis is rooted in [31] for kernelized contextual bandit, but with non-trivial extensions: weadopted the stopping time argument from [1] to remove a logarithmic factor in T (this improvement ishinted in Section 3.3 of [31] as well); and this stopping time argument is based on a special ‘batchedfiltration’ that is different for each client, which is required to address the dependencies due to theevent-triggered distributed communication.
This problem also exists in prior works of distributedlinear bandit, but was not addressed rigorously (see Lemma H.1.
of [30]).
Recall that the Ridge regression estimator
[Equation]
and thus, we have
[Equation]
where the first inequality is due to the triangle inequality, and the second is due to the property of
Rayleigh quotient, i.e., ∥ A − 1 / 2 t,
i
[Equation]
Difference from standard argument Note that the second term may seem similar to the onesappear in the self-normalized bound in previous works of linear and kernelized bandits [1, 6, 31].
However, a main difference is that D t ( i ) , i.e., the sequence of indices for the data points used toupdate client i , is constructed using the event-trigger as defined in Eq (2) .
The event-trigger isdata-dependent, and thus it is a delayed and permuted version of the original sequence [ t ] .
It is delayedin the sense that the length |D
t ( i ) | < t unless t is the global synchronization step.
It is permutedin the sense that every client receives the data in a different order, i.e., before the synchronization,each client first updates using its local new data, and then receives data from other clients at thesynchronization.
This prevents us from directly applying Lemma 3.1 of [31], and requires a carefulconstruction of the filtration, as shown in the following paragraph.
Construction of filtration For some client i at time step t , the sequence of time indices in D t ( i )is arranged in the order that client i receives the corresponding data points, which includes bothdata added during local update in each client, and data received from the server during globalsynchronization.
The former adds one data point at a time, while the latter adds a batch of data points,which, as we will see, break the assumption commonly made in standard self-normalized bound[1, 6, 31].
Specifically, we denote D t ( i )
[ k ] , for k ≤ |D t ( i ) | , as the k -th element in this sequence, i.e.,( x D t ( i )[ k ] , y D t ( i )
[ k ] ) is the k -th data point received by client i .
Then we denote B t ( i ) =
{ k 1 , k 2 , . . . }
as the sequence of k ’s that marks the end of each batch (a singel data point added by local updateis also considered a batch).
We can see that if the k -th element is in the middle of a batch, i.e.,k ∈ [ k q − 1 , k q ] , it has dependency all the way to the k q ’s element, since this index can only bedetermined until some client triggers a global synchronization at time step D t ( i )
[ k q ] .
Denote by F k,i = σ (cid:0) ( x s , η s ) s ∈D ∞
( i )[1: k − 1] , x D ∞ ( i )
[ k ] (cid:1) the σ -algebra generated by the sequence ofdata points up to the k -th element in D ∞ ( i ) .
As we mentioned, because of the dependency of theindex on the future data points, for some k -th element that is inside a batch, i.e., k ∈
[ k q − 1 , k q ] ,x D ∞ ( i )[ k ] is not F k,i -measurable and η D ∞ ( i )
[ k ] is not F k +1 ,i -measurable, which violate the as- 553 sumption made in standard self-normalized bound [1, 6, 31].
However, they become measurableif we condition on F k q ,i .
In addition, recall that in Section 3.1 we assume η D ∞ ( i )
[ k ] is zeromean R -sub-Gaussian conditioned on σ (cid:0) ( x s , η s ) s ∈N D∞ ( i )
[ k ] − 1 ( i D∞ ( i )
[ k ] ) , x D ∞ ( i )
[ k ] (cid:1) , which is theσ -algebra generated by the sequence of local data collected by client i D ∞ ( i )[ k ] .
We can see thatas σ (cid:0) ( x s , η s ) s ∈N D∞ ( i )
[ k ] − 1 ( i D∞ ( i )
[ k ] ) , x D ∞ ( i )
[ k ] (cid:1) ⊆
F k,i , η D ∞ ( i )[ k ] is zero-mean R -sub-Gaussianconditioned on F k,i .
Basically, our assumption of R -sub-Gaussianity conditioned on local sequenceinstead of global sequence of data points, prevents the situation where the noise depends on datapoints that have not been communicated to the current client yet, i.e., they are not included in F k,i .
With our ‘batched filtration’ {F k,i } k ∈B ∞ ( i ) for each client i , we have everything we need to establish atime-uniform self-normalized bound that resembles Lemma 3.1 of [31], but with improved logarithmicfactor using the stopping time argument from [1].
Then we can take a union bound over N clients toobtain the uniform bound over all clients and time steps.
Super-martingale & self-normalized bound
First, we need the following lemmas adopted from[31] to our ‘batched filtration’.
Lemma B.2.
Let υ ∈ R
d be arbitrary and consider for any k ∈ B ∞ ( i ) , i ∈
[ N ] , we have
[Equation]
is a F k +1 ,i -super-martingale, and E [ M υk,i ] ≤ exp( − 12 λυ ⊤ υ ) .
Let ˜ k be a stopping time w.r.t.
thefiltration {F k,i } k ∈B ∞ ( i ) .
Then M υ ˜ k,i is almost surely well-defined and E [
M υ ˜ k,i ] ≤ 1 .
Proof of Lemma B.2.
To show that { M υk,i } k ∈B ∞ ( i ) is a super-martinagle, we denote
[Equation]
, with D υ 0 ,i = exp( − 12 λυ ⊤ υ ) , and as we have showed earlier, η D ∞ ( i )[ k ] is R -sub-Gaussian conditionedon F k,i .
Therefore, E [ D υk,
i |F k,i ] ≤ 1 .
Moreover, D υk,i and M υk,i are F k +1 ,i -measurable.
Then wehave
[Equation]
which shows { M υk,i } k ∈B ∞ ( i ) is a super-martinagle, with E [ M υk,i ] ≤ D υ 0 ,i = exp( − 12 λυ ⊤ υ ) .
Thenusing the same argument as Lemma 8 of [1], we have that M υ ˜ k,i is almost surely well-defined, and
[Equation]
Lemma B.3.
Let ˜ k be a stopping time w.r.t.
the filtration {F k,i } k ∈B ∞ ( i ) .
Then for δ > 0 , we have
P
[Equation]
I
[Equation]
≤ δ.
Proof of Lemma B.3.
Using m :
= ( λ I + (cid:80) s ∈D ∞
( i )[1: k ] X s X ⊤ s )
[Equation]
[Equation]
∈D ∞
[Equation]
i )
[1: k ] X s η s ) , we canrewrite M υ ˜ k,i as
[Equation]
Then based on Lemma B.2, we have
[Equation]
Following the Laplace’s method as in proof of Lemma 3.1 of [31], we have
[Equation]
By applying Markov-Chernov bound, we finish the proof.
Proof of Lemma B.1 Now using the stopping time argument as in [1], and a union bound overclients, we can bound the second term in Eq (5).
First, define the bad event
[Equation]
, and ˜ k ( ω ) = min( k ≥ 0 : ω ∈ B k ( δ )) , which is a stopping time.
Moreover, ∪ k ∈B ∞ ( i ) B k
( δ )
= { ω :˜ k < ∞} .
Then using Lemma B.3, we have
P [Equation] ∈B ∞
[Equation]
i
[Equation]
= P
[Equation]
≤ P
[Equation]
≤ δ
Note that B ∞ ( i ) is the sequence of indices k in D ∞ ( i ) when client i gets updated.
Therefore, theresult above is equivalent to
[Equation]
for all t ≥ 1 , with probability at least 1 − δ .
Then by taking a union bound over N clients, we finishthe proof.
C Proof of Lemma 3.1: Regret and Communication Cost of DisKernelUCB
Based on Lemma B.1 and the arm selection rule in Eq (1), we have
[Equation]
x
[Equation]
and thus r t = f ( x ⋆t ) − f ( x t ) ≤ 2 α t − 1 ,i t ˆ σ t − 1 ,i t ( x t ) , for all t ∈
[ NT ] , with probability at least 1 − δ .
Then following similar steps as DisLinUCB of [30], we can obtain the regret and communicationcost upper bound of DisKernelUCB.
C.1 Regret Upper Bound
We call the time period in-between two consecutive global synchronizations as an epoch, i.e., thep -th epoch refers to [ t p − 1 + 1 , t p ] , where p ∈ [ B ] and 0 ≤ B ≤ NT denotes the total number ofglobal synchronizations.
Now consider an imaginary centralized agent that has immediate access toeach data point in the learning system, and denote by A t = (cid:80) ts =1 ϕ s ϕ ⊤ s and K [ t ] , [ t ] for t ∈ [ NT ]the covariance matrix and kernel matrix constructed by this centralized agent.
Then similar to [30],we call the p -th epoch a good epoch if
[Equation]
otherwise it is a bad epoch. Note that ln(det( I + λ − 1 K [ NT ] , [ NT ] )) ≤ 2 γ NT by definition ofγ NT , i.e., the maximum information gain. Since ln( det( I + λ − 1 K [ t 1] , [ t
[Equation]
) det(
[Equation]
[Equation]
I
[Equation]
[ NT ]
[Equation]
[ NT ] )) ≤ 2 γ NT , and due to the pigeonholeprinciple, there can be at most 2 γ NT bad epochs.
If the instantaneous regret r t is incurred during a good epoch, we have
[Equation]
where the second inequality is due to Lemma A.3, and the last inequality is due to the definition of good epoch, i.e.,
[Equation]
Defineα NT : = √ λ
∥ θ ⋆ ∥ + (cid:112) 2 ln( N/δ ) + ln(det( K [ NT ] , [ NT ] /λ + I )) .
Then using standard arguments,the cumulative regret incurred in all good epochs can be bounded by,
[Equation]
where the third inequality is due to Cauchy-Schwartz, and the forth is due to Lemma A.1 anddefinition of maximum information gain γ NT .
Then we look at the regret incurred during bad epochs.
Consider some bad epoch p , and the cumulativeregret incurred during this epoch can be bounded by
[Equation]
where the last inequality is due to our event-trigger in Eq (2).
Since there can be at most 2 γ NT badepochs, the cumulative regret incurred in all bad epochs
[Equation]
Combining cumulative regret incurred during both good and bad epochs, we have
[Equation]
C.2 Communication Upper Bound
For some α > 0 , there can be at most ⌈ NTα
⌉ epochs with length larger than α .
Based onour event-trigger design, we know that ( |D t p ( i t p ) | −
|D t p − 1 ( i t p ) | ) ln( det( I + λ
[Equation]
1 K [ tp ] , [ tp ] ) det( I + λ − 1 K [ tp − 1] , [ tp − 1] ) ) ≥( |D t p
[Equation]
i
[Equation]
p ) | −
[Equation]
D
[Equation] ) | )
ln
[Equation]
det
[Equation]
K
[Equation]
itp
[Equation]
D tp
[Equation]
itp ) ) det( I + λ
[Equation]
1 K D tp
[Equation]
1( itp ) , D tp − 1( itp ) ) ) ≥ D for any epoch p ∈
[ B ] , wherei t p is the client who triggers the global synchronization at time step t p .
Then if the length of certainepoch p is smaller than α , i.e., t p − t p − 1 ≤ α , we have ln( det( I + λ − 1 K
[ tp ] , [ tp ] ) det( I + λ − 1 K
[ tp − 1] , [ tp − 1] ) ) ≥ NDα .
Sinceln( det( I + λ − 1 K
[ t 1] , [ t 1] ) det( I ) )
+ ln( det( I + λ − 1 K [ t 2] , [ t 2] ) det( I + λ − 1 K
[ t 1] , [ t 1] ) ) + · · · + ln( det( I + λ − 1 K [ t B ] , [ t B ] ) det( I + λ − 1 K [ tB − 1] , [ tB − 1] ) ) ≤ ln(det( I +λ − 1 K
[ NT ] , [ NT ] )) ≤ 2 γ NT , the total number of such epochs is upper bounded by ⌈ 2 γ NT α ND ⌉ .
Combining the two terms, the total number of epochs can be bounded by,
[Equation]
where the LHS can be minimized using the AM-GM inequality, i.e., B ≤ (cid:113) NTα 2 γ NT α ND =
(cid:113) 2 γ NT T D .
To obtain the optimal order of regret, we set D
= O ( T Nγ NT ) , so that R NT
= O (cid:0) √ NT ( ∥ θ ⋆ ∥√ γ NT +γ NT ) (cid:1) .
And the total number of epochs B = O ( √ Nγ NT ) .
However, we should note that as
DisKernelUCB communicates all the unshared raw data at each global synchronization, the totalcommunication cost mainly depends on when the last global synchronization happens.
Since thesequence of candidate sets {A t } t ∈ [ NT ] , which controls the growth of determinant, is an arbitrarysubset of A , the time of last global synchronization could happen at the last time step t
= NT .
Therefore, C T
=
O
( N 2 Td ) in such a worst case.
D Derivation of the Approximated Mean and Variance in Section
For simplicity, subscript t is omitted in this section.
The approximated Ridge regression estimator fordataset { ( x s , y s ) } s ∈D is formulated as
[Equation]
where D denotes the sequence of time indices for data in the original dataset, S ⊆
D denotes the timeindices for data in the dictionary, and P S ∈ R
p
×
p denotes the orthogonal projection matrix definedby S .
Then by taking derivative and setting it to zero, we have ( P S Φ ⊤D Φ D P S + λ I )˜ θ
=
P S Φ ⊤D y D ,and thus ˜ θ =
˜
A − 1 ˜ b , where
˜ A =
P S Φ ⊤D Φ D P S
+ λ I and ˜
b = P S Φ ⊤D y D .
Hence, the approximated mean reward and variance for some arm x are
[Equation]
To obtain their kernelized representation, we rewrite
[Equation]
where ρ := 1 λ ( y D − Φ D P S ˜ θ ) = 1 λ ( y D − Φ D P S P S Φ ⊤D ρ ) .
Solving this equation, we get ρ =( Φ D P S P S Φ ⊤D
+ λ I ) − 1 y D .
Note that P S P S = P S , since projection matrix P S is idempotent.
Moreover, we have ( Φ ⊤ Φ + λ I )
Φ ⊤ =
Φ ⊤ ( ΦΦ ⊤ + λ I ) , and ( Φ ⊤ Φ + λ I ) − 1 Φ ⊤ =
Φ ⊤ ( ΦΦ ⊤ +λ I ) − 1 .
Therefore, we can rewrite the approximated mean for some arm x as
[Equation]
To derive the approximated variance, we start from the fact that ( P S Φ ⊤D Φ D P S + λ I ) ϕ
( x ) =
P
[Equation]
, so
[Equation]
Then we haveϕ ( x ) ⊤ ϕ ( x ) = (cid:8)
[Equation]
By rearranging terms, we have
[Equation]
E Proof of Lemma 4.
In the following, we analyze the ϵ t,i -accuracy of the dictionary for all t, i .
At the time steps when global synchronization happens, i.e., t p for p ∈
[ B ] , S t p is sampled from[ t p ] =
D t p ( i ) using approximated variance ˜ σ 2 t p − 1 ,i .
In this case, the accuracy of the dictionary onlydepends on the RLS procedure, and Calandriello et al.
[4] have already showed that the followingguarantee on the accuracy and size of dictionary holds ∀ t ∈ { t p } p ∈
[ B ] .
Lemma E.1 (Lemma 2 of [4]) .
Under the condition that ¯ q = 6 1+ ϵ 1 − ϵ log(4 NT/δ ) /ϵ 2 , for someϵ ∈ [0 , 1) , with probability at least 1 − δ , we have ∀ t ∈ { t p } p ∈ [ B ] that the dictionary { ( x s , y s ) } s ∈S tis ϵ -accurate w.r.t.
{ ( x s , y s ) } s ∈D t
( i ) , and 1 − ϵ 1+ ϵ σ 2 t ( x ) ≤ ˜ σ 2 t ( x ) ≤ 1+ ϵ 1 − ϵ σ 2 t ( x ) , ∀ x ∈ A .
Moreover,the size of dictionary |S t | ≤ 3(1 + κ 2 /λ )
1+ ϵ 1 − ϵ ¯ q ˜ d , where ˜ d :=
Tr ( K [ NT ] , [ NT ] ( K [ NT ] , [ NT ] + λ I ) − 1 )denotes the effective dimension of the problem, and it is known that ˜ d
= O ( γ NT ) [6].
Lemma E.1 guarantees that for all t ∈ { t p } p ∈ [ B ] , the dictionary has a constant accuracy, i.e., ϵ t,i =
ϵ, ∀ i .
In addition, since the dictionary is fixed for t / ∈ { t p } p ∈ [ B ] , its size S t
= O ( γ NT ) , ∀ t ∈ [ NT ] .
Then for time steps t / ∈ { t p } p ∈ [ B ] , due to the local update, the accuracy of the dictionary will degrade.
However, thanks to our event-trigger in Eq (4), the extent of such degradation can be controlled, i.e., anew dictionary update will be triggered before the previous dictionary becomes completely irrelevant.
This is shown in Lemma E.2 below.
Lemma E.2.
Under the condition that { ( x s , y s ) } s ∈S tp is ϵ -accurate w.r.t.
{ ( x s , y s ) } s ∈D tp ( i ) , ∀ t ∈[ t p + 1 , t p +1 ] , i ∈
[ N ] , S t p is (cid:0) ϵ + 1 − 1 1+ 1+ ϵ 1 − ϵ D (cid:1)
-accurate w.r.t.
D t ( i ) .
Combining Lemma E.1 and Lemma E.2 finishes the proof.
Proof of Lemma E.2.
Similar to [3], we can rewrite the ϵ -accuracy condition of S t p w.r.t.
D t ( i ) fort ∈ [ t p + 1 , t p +1 ] as
[Equation]
Φ ⊤D
[Equation] i
[Equation]
+
[Equation]
≤
[Equation]
where ψ s = ( Φ ⊤D t ( i ) Φ D t ( i )
+ λ I ) − 1 / 2 ϕ s .
Notice that the second term in the norm has weight − 1because the dictionary S t p is fixed after t p .
With triangle inequality, now it suffices to bound∥ (cid:88)
[Equation]
∈D tp
[Equation]
We should note that the first term corresponds to the approximation accuracy of S t p w.r.t.
the dataset
D t p .
And under the condition that it is ϵ -accurate w.r.t.
D t p , we have ∥ (cid:80) s [Equation] tp [Equation]
s
[Equation] ≤ [Equation] The second term measures the difference between D t ( i ) compared with D t p , which is unique to ourwork.
We can bound it as follows.
[Equation]
We can further bound the term σ 2 tp,i ( x ) σ 2 t,i ( x ) using the threshold of the event-trigger in Eq (4).
For anyx ∈ R d ,
[Equation]
where the first inequality is due to Lemma A.4, the second is due to Lemma E.1, and the third is dueto the event-trigger in Eq (4).
Putting everything together, we have that if S t p is ϵ -accurate w.r.t.
D t p ,then it is (cid:0) ϵ + 1 − [Equation]
[Table]
F Proof of Lemma 4.2
To prove Lemma 4.2, we need the following lemma.
Lemma F.1.
We have ∀ t, i that
[Equation]
with probability at least 1 − δ .
Proof of Lemma F.1.
Recall that the approximated kernel Ridge regression estimator for θ ⋆ is definedas
[Equation]
where P S is the orthogonal projection matrix for the Nyström approximation, and
˜
A t,i =
P S Φ ⊤D t ( i ) Φ D t ( i ) P S
+ λ I .
Then our goal is to bound
[Equation]
Bounding the first term To bound the first term, we begin with rewriting
[Equation]
and by substituting this into the first term, we have
[Equation]
where the first inequality is due to Cauchy Schwartz
[Equation]
and the last inequality is because∥ ˜
A − 1 / 2 t,
i
[Equation]
Bounding the second term By applying Cauchy-Schwartz inequality to the second term
[Equation]
we have
[Equation]
Note that P S
A t,i P S =
P S ( Φ ⊤D t ( i ) Φ D t ( i )
+ λ I ) P S =
˜
A t,i + λ ( P S − I ) and P S ⪯ I , so we have∥
˜ A − 1 / 2 t,
i
[Equation]
Then using the self-normalized bound derived for Lemma B.3, the term ∥
A − 1 / 2 t,i Φ ⊤D t ( i ) η
[Equation] ) ∥ =∥ Φ ⊤D [Equation]
∥
A
[Equation]
1 t,i can be bounded by
[Equation]
for ∀ t, i , with probability at least 1 − δ .
Combining everything finishes the proof.
Now we are ready to prove Lemma 4.2 by further bounding the term ∥ Φ D t ( i ) ( I − P S tp ) ∥ .
Proof of Lemma 4.2.
Recall that ¯ S t,i ∈
R |D t ( i ) |×|D t
( i ) | denotes the diagonal matrix, whose s -thdiagonal entry equals to q s √ ˜ p s ,
where q s = 1 if s ∈ S t p and 0 otherwise (note that for s / ∈ S t p , we set˜ p s = 1 , so q s / ˜ p s = 0 ).
Therefore, ∀ s ∈ D t ( i ) \ D t p , q s = 0 , as the dictionary is fixed after t p .
Wecan rewrite Φ ⊤D t ( i ) ¯ S ⊤ t,i ¯ S t,i Φ D [Equation] ( i ) =
(cid:80) s
∈D t
( i ) q s ˜ p s ϕ s ϕ ⊤ s , where ϕ s := ϕ ( x s ) .
Then by definition ofthe spectral norm ∥·∥ , and the properties of the projection matrix P S tp , we have∥
[Equation]
Moreover, due to Lemma E.2, we know S t p is ϵ t,i -accurate w.r.t.
D t ( i ) for t ∈ [ t p + 1 , t p +1 ] , whereϵ t,i = (cid:0) ϵ + 1 − 1 1+
[Equation]
+
[Equation]
[Equation]
D (cid:1) , so we have I − P S tp ⪯ λ 1 − ϵ t,i ( Φ ⊤D t ( i ) Φ D t ( i ) + λ I ) − 1 by the property ofϵ -accuracy (Proposition 10 of [3]).
Therefore, by substituting this back to Eq (6), we have
[Equation]
λ
[Equation]
which finishes the proof.
G Proof of Theorem 4.3: Regret and Communication Cost ofApprox-DisKernelUCBG.1 Regret Analysis
Consider some time step t ∈
[ t p − 1 + 1 , t p ] , where p ∈ [ B ] .
Due to Lemma 4.2, i.e., the confidence ellipsoid for approximated estimator, and the fact that x t = arg max x ∈A t,i ˜ µ t − 1 ,i ( x )+ α t − 1 ,i ˜ σ t − 1 ,i ( x ) ,we have
[Equation]
i ( x t ) , f ( x t ) ≥ ˜ µ t − 1 ,i ( x t ) − α t − 1 ,i ˜ σ t − 1 ,i ( x t ) , and thus r t = f ( x ⋆t ) − f ( x t ) ≤ 2 α t − 1 ,i ˜ σ t − 1 ,i ( x t ) , whereα t − 1 ,i =
[Equation]
Note that, different from Appendix C, the α t − 1 ,i term now depends on the threshold D and accuracyconstant ϵ , as a result of the approximation error.
As we will see in the following paragraphs, theirvalues need to be set properly in order to bound α t − 1 ,i .
We begin the regret analysis of Approx-DisKernelUCB with the same decomposition of good and badepochs as in Appendix C.1, i.e., we call the p -th epoch a good epoch if ln( det( I + λ − 1 K [ tp ] , [ tp ] ) det( I + λ − 1 K [ tp − 1] , [ tp − 1] ) ) ≤1 , otherwise it is a bad epoch.
Moreover, due to the pigeon-hold principle, there can be at most 2 γ NTbad epochs.
As we will show in the following paragraphs, using Lemma E.1, we can obtain a similar bound forthe cumulative regret in good epochs as that in Appendix C.1, but with additional dependence on Dand ϵ .
The proof mainly differs in the bad epochs, where we need to use the event-trigger in Eq (4)to bound the cumulative regret in each bad epoch.
Compared with Eq (2), Eq (4) does not containthe number of local updates on each client since last synchronization.
, and as mentioned in Section4.2, this introduces a √ T factor in the regret bound for bad epochs in place of the √ γ NT term in
Appendix C.1.
Cumulative Regret in Good Epochs Let’s first consider some time step t in a good epoch p , i.e.,t ∈ [ t p − 1 + 1 , t p ] , and we have the following bound on the instantaneous regret
[Equation]
where the second inequality is because the (approximated) variance is non-decreasing, the thirdinequality is due to Lemma E.1, the forth is due to Lemma A.3, and the last is because in a goodepoch, we have
[Equation]
Therefore, the cumulative regret incurred in all good epochs, denoted by R good , is upper bounded by
[Equation]
where α NT := (cid:32) 1 (cid:114) − ϵ + 1 1+1+ ϵ 1 − ϵD +(cid:33) √ λ
∥
θ ⋆ ∥
+ R (cid:112) 4 ln N/δ + 2 ln det((
+ λ )
I + K
[ NT ] , [ NT ] ) .
Cumulative Regret in Bad Epochs
The cumulative regret incurred in this bad epoch is
B (cid:88) p =
[Equation] [Table]
where the third inequality is due to the Cauchy-Schwartz inequality, the forth is due to our eventtrigger in Eq (4), the fifth is due to our assumption that clients interact with the environment in around-robin manner, the sixth is due to the Cauchy-Schwartz inequality again, and the last is due tothe fact that there can be at most 2 γ NT bad epochs.
Combining cumulative regret incurred during both good and bad epochs, we have
[Equation]
G.2 Communication Cost Analysis
Consider some epoch p .
We know that for the client i who triggers the global synchronization, wehave
[Equation]
Then by summing over B epochs, we have
[Equation]
Now we need to bound the ratio σ 2 tp − 1 ( x s ) σ 2 s − 1 ( x s ) for s ∈ [ t p − 1 + 1 , t p ] .
[Equation]
Note that for the client who triggers the global synchronization, we have
[Equation]
∈D
[Equation]
i ) ˜
σ
[Equation]
p − 1 ( x s ) < D , i.e., one time step before it triggers the synchronization at time t p .
Due to the fact that the (approximated) posterior variance cannot exceed κ 2 /λ ,we have (cid:80) s ∈D tp ( i ) [Equation]
− [Equation] i [Equation]
˜ σ [Equation] x s ) < D + κ 2 /λ .
For the other N − 1 clients, we have
[Equation]
∈D
[Equation]
i )
[Equation]
i ) ˜
σ
[Equation]
x s ) < D .
Summing them together, we have
[Equation]
for the p -th epoch.
By substituting this back, we have
[Equation]
Therefore,
[Equation]
and thus the total number of epochs B < 1+ ϵ 1 − ϵ [ 1 D + 1+ ϵ 1 − ϵ ( N + κ 2 / ( λD ))]2 γ NT .
By setting D = 1 N , we have
[Equation]
because N [-+(cid:33) √ λ ∥ θ ⋆ ∥ + R (cid:113) 4 ln N/δ + 2 ln det((
-]≥ 1 .
Moreover, to ensure − ϵ + 1 1+
[-∥ θ ⋆ ∥ + R (cid:113) 4 ln N/δ + 2 ln det((
 + λ ) I + K-][Equation]
ϵ > 0 , we need to set the constant ϵ < 1 / 3 .
Therefore,
[Equation]
and the total number of global synchronizations B = O
( Nγ NT ) .
Since for each global synchronization, the communication cost is O
( Nγ 2 NT ) , we have
[Equation]
H Experiment Setup
Synthetic dataset We simulated the distributed bandit setting defined in Section 3.1, with d =20 , T = 100 , N = 100 ( NT = 10 4 interactions in total).
In each round l ∈ [ T ] , each client i ∈ [ N ](denote t = N ( l − 1) + i ) selects an arm from candidate set A t , where A t is uniformly sampledfrom a ℓ 2 unit ball, with |A t | = 20 .
Then the corresponding reward is generated using one of thefollowing reward functions:
[Equation]
where the parameter θ ⋆ is uniformly sampled from a ℓ 2 unit ball.
UCI Datasets To evaluate Approx-DisKernelUCB’s performance in a more challenging and practical scenario, we performed experiments using real-world datasets: MagicTelescope, Mushroomand Shuttle from the UCI Machine Learning Repository [8].
To convert them to contextual banditproblems, we pre-processed these datasets following the steps in [12].
In particular, we partitionedthe dataset in to 20 clusters using k-means, and used the centroid of each cluster as the context vectorfor the arm and the averaged response variable as mean reward (the response variable is binarized byassociating one class as 1 , and all the others as 0 ).
Then we simulated the distributed bandit learningproblem in Section 3.1 with |A t | = 20 , T = 100 and N = 100 ( NT = 10 4 interactions in total).
MovieLens and Yelp dataset Yelp dataset, which is released by the Yelp dataset challenge, consistsof 4.7 million rating entries for 157 thousand restaurants by 1.18 million users.
MovieLens is adataset consisting of 25 million ratings between 160 thousand users and 60 thousand movies [13].
Following the pre-processing steps in [2], we built the rating matrix by choosing the top 2000users and top 10000 restaurants/movies and used singular-value decomposition (SVD) to extract a10-dimension feature vector for each user and restaurant/movie.
We treated rating greater than 2 aspositive.
We simulated the distributed bandit learning problem in Section 3.1 with T = 100 and
N = 100 ( NT = 10 4 interactions in total).
In each time step, the candidate set A t (with |A t | = 20 )is constructed by sampling an arm with reward 1 and nineteen arms with reward 0 from the arm pool,and the concatenation of user and restaurant/movie feature vector is used as the context vector for thearm (thus d = 20 ).
I Lower Bound for Distributed Kernelized Contextual Bandits
First, we need the following two lemmas
Lemma I.1 (Theorem 1 of [29]) .
There exists a constant C > 0 , such that for any instance ofkernelized bandit with κ = σ = 1 , the expected cumulative regret for KernelUCB algorithm is upperbounded by E [ R T ] ≤ C √
Tγ T , where the maximum information gain γ T
= O (cid:0) (ln( T ))
d +1 (cid:1) for
Squared Exponential kernels.
Lemma I.2 (Theorem 2 of [23]) .
There always exists a set of hard-to-learn instances of kernelizedbandit with κ = σ = 1 , such that for any algorithm, for a uniformly random instance in the set, theexpected cumulative regret E [ R T ] ≥ c [Table] Squared someconstant c [Table]
Then we follow a similar procedure as the proof for Theorem 2 of [30] and Theorem 5.3 of [14], toprove the following lower bound results for distributed kernelized bandit with Squared Exponentialkernels.
Theorem I.3.
For any distributed kernelized bandit algorithm with expected communication costless than O (
N (ln( T )) 0 .
25 d +0 . 5 ) , there exists a kernelized bandit instance with Squared Exponentialkernel, and κ = σ = 1 , such that the expected cumulative regret for this algorithm is at leastΩ( N (cid:112) T (ln( T )) d/ 2 ) .
Proof of Theorem I.3.
Here we consider kernelized bandit with Squared Exponential kernels.
Theproof relies on the construction of a auxiliary algorithm, denoted by AuxAlg , based on the originaldistributed kernelized bandit algorithm, denoted by DisKernelAlg , as shown below.
For each agenti ∈ [ N ] , AuxAlg performs DisKernelAlg , until any communication happens between client i and theserver, in which case, AuxAlg switches to the single-agent optimal algorithm, i.e., the KernelUCBalgorithm that attains the rate in Lemma I.1.
Therefore, AuxAlg is a single-agent bandit algorithm,and the lower bound in Lemma I.2 applies: the cumulative regret that AuxAlg incurs for some agenti ∈ [ N ] is lower bounded by
[Equation]
and by summing over all N clients, we have
[Equation]
For each client i ∈ [ N ] , denote the probability that client i will communicate with the server as p i ,and p :=
(cid:80) Ni =
1 p i .
Note that before the communication, the cumulative regret incurred by AuxAlg isthe same as DisKernelAlg , and after the communication happens, the regret incurred by AuxAlg isthe same as KernelUCB, whose upper bound is given in Lemma I.1.
Therefore, the cumulative regretthat AuxAlg incurs for client i can be upper bounded by
[Equation]
and by summing over N clients, we have
[Equation]
Combining the upper and lower bounds for E [ R AuxAlg ] , we have
[Equation]
Therefore, for any DisKernelAlg with number of communications p ≤ N c
[Equation]
[Equation]
[Equation]
=
O (
N (ln( T ))
[Equation]
[Equation]
d +
[Equation]
5 ) , we have
[Equation]