Federated Online Clustering of Bandits
Abstract
Contextual multi-armed bandit (MAB) is an important sequential decision-making problem in recommendation systems.
A line of works, called the clustering of bandits (CLUB), utilize the collaborative effect over users and dramatically improve the recommendation quality.
Owing to the increasing application scale and public concerns about privacy, there is a growing demand to keep user data decentralized and push bandit learning to the local server side.
Existing CLUB algorithms, however, are designed under the centralized setting where data are available at a central server.
We focus on studying the federated online clustering of bandit (FCLUB) problem, which aims to minimize the total regret while satisfying privacy and communication considerations.
We design a new phase-based scheme for cluster detection and a novel asynchronous communication protocol for cooperative bandit learning for this problem.
To protect users’ privacy, previous differential privacy (DP) definitions are not very suitable, and we propose a new DP notion that acts on the user cluster level.
We provide rigorous proofs to show that our algorithm simultaneously achieves (clustered) DP, sublinear communication complexity and sublinear regret.
Finally, experimental evaluations show our superior performance compared with benchmark algorithms.
INTRODUCTION
Stochastic multi-armed bandit (MAB)
[Auer et al., 2002] is a well-known sequential decision-making problem, where a learner sequentially selects actions so as to maximize the cumulative rewards (or minimize the cumulative regret).
One fruitful application area of MAB is the online recommendation systems (RecSys)
[Chu et al., 2011, Abbasi-Yadkori et al., 2011, Gentile et al., 2014, Li et al., 2019, Zhang et al., 2020, Li et al., 2021], where MAB algorithms provide a principled way to handle the challenge of explorationexploitation trade-off [Lattimore and Szepesvári, 2020].
To advance the bandit algorithm for large-scale applications, contextual linear bandits add the simple yet effective linear structure assumptions on actions and reward functions
[Chu et al., 2011, Li et al., 2010, Abbasi-Yadkori et al., 2011].
One limitation, however, is that such a model mainly works in a content-dependent manner, ignoring the often used tool of collaborative filtering.
To address this issue, the clustering of bandits (CLUB) are proposed [Gentile et al., 2014, Li et al., 2016, Li and Zhang, 2018, Li et al., 2019].
The CLUB algorithms adaptively cluster similar users and utilize the collaborative information given by the cluster structure, which dramatically improves the recommendation quality.
While most existing bandit algorithms are designed under a centralized setting, in response to the increasing application scale and public concerns about privacy, there is a growing demand to keep user data decentralized and push the learning of bandit models to the client or the local server side .
This paradigm is now known as federated learning [Kairouz et al., 2021].
Owing to its overall applicability, there has been a surge of interest in studying federated
MAB [Dubey and Pentland, 2020, Zhu et al., 2021, Shi and
Shen, 2021], which promises cooperative bandit learning with larger amounts of data (across multiple local servers) while keeping the data decentralized.
This motivates us to study the CLUB problem to its federated counterpart, i.e., the federated clustering of bandits (FCLUB).
In FCLUB, each local server can conduct its own local clustering of bandit algorithms.
To enable the collaborative effects of users across different servers, the local server could also collaborate with other local servers under the coordination of a global server, whose communication needs to satisfy specific privacy and communication requirements.
The goal of this work is to design an federated online clustering of bandit framework, so as to minimize the T -round regret under the privacy protection requirements and communication cost considerations.
The key challenge of FCLUB is designing collaborative bandit learning procedures
and
cluster detection strategies to identify the overall cluster structures across different local servers, where each local server only holds part of the users with unknown interests.
Such a problem is more challenging due to the following privacy and communication cost requirements, which are two first-order requirements for any federated applications [Kairouz et al., 2021].
Privacy protection: To reduce the privacy leakage of each user, we expect local servers to only share user clusters’ data instead of individuals’ raw data.
In addition, we still need a mechanism to protect the uploaded (cluster) information against possible adversaries outside the local server, for which we adopt the solution concept of differential privacy (DP).
However, the off-the-shelf DP notion is defined on individual users, hence unsuitable for FCLUB.
It is challenging and unclear what is a suitable notion of privacy over the clustering of users and how to devise algorithms to guarantee the corresponding privacy requirements.
Communication: Communication is critical for collaborative learning, but may also be expensive or time-consuming.
For FCLUB, it is desired to minimize the total regret while keeping the communication costs (in terms of communication rounds between the global server and local servers) as low as possible.
Another requirement is to design an asynchronous communication protocol incorporating the randomly arriving users and possibly lagging servers, preventing commonly used synchronous protocols [Dubey and
Pentland, 2020].
1.1 OUR CONTRIBUTIONS
To address the aforementioned challenges, this paper makes four contributions.
Problem Formulation: We propose the setting of online clustering of bandits to its federated counterpart, which considers the privacy protection and communication requirements.
We also propose a novel cluster differential privacy (CDP) notion tailored for the FCLUB setting.
Algorithm Design: We propose a private and communication-efficient FCLUB-CDP algorithm.
For privacy protection, a tree-based privatizer is designed to guarantee our proposed CDP.
For communication efficiency, we follow the phase-based principle for cluster detection and propose the asynchronous communication protocol for delayed information sharing.
In particular, each local server maintains upload/download buffers and occasionally uploads/downloads the buffered information to/from the global server only if it finds the latest information deviates too far from the last update.
Theoretical Analysis: We prove that FCLUB-CDP achieves the O ( dL (cid:113) mT log(1 /δ )
ε log 1 .
5 T ) regret bounds, O ( dmL log T ) communication costs and ( ε, δ, L, m ) -CDP privacy guarantee, respectively.
Experiments:
We
conduct extensive
experiments over synthetic and real-world datasets to
validate
our
theoretical
analysis.
Empirical results show the superior performance of our algorithm over existing
algorithms.
1.2 RELATED WORK
Online Clustering of Bandits.
The online clustering bandits is first proposed by Gentile et al.
[2014] and shows its effeteness by accelerating the learning process of contextual bandits.
The key idea is to use a graph representing the user similarity and adaptively refine the user clusters for information sharing.
This work has been extended by a series of works considering the collaborative effects on both users and items [Li et al., 2016], the context-aware settings [Gentile et al., 2017], the cascading bandit setting [Li and Zhang, 2018] and the users with different user frequency [Li et al., 2019].
However, none of these works consider the privacy constraints and communication cost requirements imposed by the FL paradigm like the current work, and therefore cannot give guarantees on these two critical criteria.
Korda et al.
[2016] considers the peer-to-peer but non-private clustering of bandits, our work studies the private bandit setting under the orchestration of a global server, which requires different algorithms and analysis.
Federated and Distributed Bandits.
There has been growing interest in bandit learning with multiple players.
One line of research investigates the competitive agents with collisions [Anandkumar et al., 2011, Rosenski et al., 2016,
Bistritz and Leshem, 2018, Boursier and Perchet, 2019], in which the reward for an arm is zero if it is chosen by more than one agent.
The goal of these works is to minimize regret without communication, which is different from ours.
The cooperative distributed bandits are most related to our work, in which multiple agents collaborate to solve a bandit problem over certain communication networks, e.g., peer-to-peer networks [Korda et al., 2016] or client-server networks [Dubey and Pentland, 2020, Li and Wang, 2021].
Our work belongs to the client-server setting, but we differ from both Dubey and Pentland [2020] and Li and Wang [2021] since neither of them considers the clustering effects of users.
Differential Privacy.
Our work leverages on differential privacy , a rigorous mathematical framework of privacy first proposed by Dwork et al.
We utilize several useful techniques from the standard differential privacy to maintain our cluster differential privacy condition.
Most notably, we use a tree-based algorithm which is introduced in Chan et al.
[2011] to realize differential privacy for the continual release of statistics.
In the single-agent bandit setting, Shariff and Sheffet [2018] also utilizes this tree-based algorithm to achieve Joint DP, which is then extended by Dubey and
Pentland [2020] to the federated setting.
The closest work to ours is Dubey and Pentland [2020] and they study the simpler case where each local server only holds one user and all users are identical (with the same unknown preference vector), hence gives the different user-level DP definition with different privatizer and analysis.
To the best of our knowledge, this paper is the first to generalize the CLUB to its federated setting, which simultaneously achieves privacy protection and communication requirements.
PROBLEM SETTINGS
In this section, we formulate the setting of “Federated Clustering of Bandits” (FCLUB).
We use [ n ] to represent set { 1 , ..., n } .
We use boldface lowercase letters and boldface capitalized letters for column vectors and matrices, respectively.
For the norms, ∥ x ∥ denotes the ℓ 2 norm of vector x .
For any symmetric positive semi-definite (PSD) matrix M (i.e., x ⊤ Mx ≥ 0 , ∀ x ),
∥ x ∥ M = √ x ⊤ Mx denotes the matrix norm of x regarding matrix M .
At the global level, there are n users, denoted by the set
U = { u 1 , ..., u n } .
Each user i ∈
U has an unknown preference vector θ i ∈ R d and for simplicity we assume
∥ θ i ∥ ≤ 1 .
Since users may have the same/similar preference vector, we assume there exists m (unknown) different preference vectors, i.e., |{ θ 1 , ..., θ n }| = m .
Users with the same preference vector form an underlying cluster
and we denote these m (unrevealed) clusters by C
= { C 1 , ..., C m } .
Different from CLUB, users in FCLUB are distributed in L local servers denoted by { 1 , ..., L } .
At the local level, the local server ℓ contains n ℓ users U ℓ = { u ℓ 1 , ..., u ℓn ℓ } (with (cid:83) ℓ ∈
[ L ] U ℓ = U ) and similarly, these n ℓ users form local cluster C ℓ = { C ℓ 1 , ..., C ℓm ℓ } where m ℓ ≤ m .
The learning agent interacts with the bandit game as the follows.
At each time t , a user i t ∈ [ n ] randomly arrives with probability 1 /n .
Then K items are generated to form a item set D t , where the feature of each item x ∈ D t is drawn independently from a fixed but unknown distribution ρ over { x ∈ R d : ∥ x ∥ ≤ 1 } .
The learning agent identifies the local server ℓ t that i t belongs to and the current user cluster j t (detected by our algorithm) which i t lies in.
The local server then recommends an item x t ∈ D t to the user based on the aggregated information from cluster j t .
After i t receives the recommendation, the learning agent receives a random reward y t ∈ [0 , 1] .
Let
H t
= { i 1 , x 1 , y 1 , ..., i t − 1 , x t − 1 , y t − 1 , i t } be the historical information before time t .
We assume the expectation of reward y t is linear in the feature vector x ∈ D t and the unknown preference vector θ i t , i.e., E t [ y t | x ] =
θ ⊤ i t x , and { y t − θ ⊤ i t x } t =1 , 2 ,... have sub-Gaussian tails σ 20 .
Now we give some assumptions on preference vectors and item feature vectors.
Note that all the assumptions follow the previous works Gentile et al.
[2014, 2017], Li and Zhang [2018], Li et al.
Assumption 1 (Gap between preference vectors) .
For any two different preference vectors θ i 1 ̸
= θ i 2 , there is a fixed but unknown gap γ >
so that ∥ θ i 1 − θ i 2 ∥ ≥ γ .
Assumption 2 (Item regularity) .
For item distribution ρ , there exists a known λ x > 0 so that E x ∼ ρ [ xx ⊤ ] is full rank with minimal eigenvalue λ x .
Meanwhile, for all time t , for any fixed unit vector θ ∈ R d , ( θ ⊤ x ) 2 has sub-Gaussian tail with variance σ 2 ≤ λ x 8log(
[Equation]
Learning Efficiency.
The goal of the learning agent is to accumulate as much reward as possible.
Let the optimal item for user i t at time t be x ∗ i t = arg max x ∈ D t θ ⊤ i t x .
The learning performance is measured by the regret, defined as
[Equation]
where r t is the regret at time t and the expectation is taken over the randomness of the algorithm and the environment regarding the users
i 1 , ..., i T and the item sets D 1 , ..., D T .
In addition to the regret, privacy protection and the communication cost are two important criterion in federated learning.
In this work, we aim to ensure that the user data are protected under privacy constraints and the communication complexity is low.
Privacy Requirements.
To protect the user data, we introduce two privacy requirements.
First, we desire the local server only uploads the user clusters’ sufficient statistics (or clustered data) for the learning procedure, instead of individual users’ raw data.
Second, to protect the clustered data against third-party adversaries outside the local server, we adopt the notion of DP, which encodes the intuition that any observable output changes very little (in probability) when any input datum changes.
Since existing DP notions are defined on user-level data Shariff and Sheffet [2018],
Dubey and Pentland [2020], we introduce a new differential privacy requirement to protect the cluster-level data.
The contextual MAB problem involves two sets of variables against the adversaries outside the local server: the decision sets D t and the observed rewards y t .
Since the users only receive and store observations regarding the chosen action x t and the observed reward y t , it suffices to protect ( x t , y t ) t ∈
[ T ] to achieve DP requirements.
Let τ ℓ,j be time slots when user in cluster j appears at local server ℓ , we denote two sequences S ℓ,j =
( x t , y t ) t ∈ τ ℓ,j and S ′
ℓ,j =
( x ′ t , y ′ t ) t ∈ τ ℓ,j as t
-neighboring if ( x τ , y τ ) =
( x ′ τ , y ′ τ ) for τ ̸ = t ∈ τ ℓ,j .
Definition 1 (Cluster Differential Privacy) .
In the FCLUB setting with L servers and (at most) m clusters, a federated contextual bandit algorithm A
=
( A ℓ,j ) ℓ ∈ [ L ] ,j ∈ [ m ] is ( ε, δ, L, m ) -CDP, if for any ( ℓ, j ) , ( ℓ ′ , j ′ )
s.t. ( ℓ, j ) ̸
= ( ℓ ′ , j ′ ) , any t and the set of sequences ¯ S ℓ ′ ,j ′ =
∪ i ∈
[ L ] ,k ∈ [ m ] S i,k and ¯ S ′ ℓ ′ ,j ′ = ∪
i ∈
[ L ] \ ℓ ′ ,k ∈ [ m ] \ j ′ S i,k ∪ S ′ ℓ ′ ,j ′ s.t.
S ℓ ′ ,j ′ and S ′ ℓ ′ ,j ′ are t neighboring, and for any subset of actions a ℓ,j ⊂ (cid:81) τ ∈ τ
ℓ,j D τ of actions, it holds that
[Equation]
Note that our CDP notion formalizes the intuition that the action chosen by any local server ℓ (at the cluster level j ) must be sufficiently indistinguishable (in probability) to any single ( x, y ) pair from any other local cluster ( ℓ ′ , j ′ ) .
Such a notion does not require each cluster is private to its own observations, i.e., each cluster of users can be trusted with its own data, which is different from the local DP [Zheng et al., 2020] or Joint DP [Shariff and Sheffet, 2018] that assume even itself cannot be trusted.
Communication Complexity.
To evaluate the communication complexity, we count one upload operation (or one download operation) between any local server and the global server as one communication round.
Our communication complexity is the total number of communication rounds
C ( T ) over the time horizon T .
ALGORITHM
In this section, we introduce our phase-based federated clustering of bandit algorithms with CDP (FCLUB-CDP).
Identify the Underlying Cluster Structure.
To correctly identify the cluster structure, we design a phase-based clustering detection algorithm in Algorithm 2.
The high level idea is to first conduct local-level clustering of bandits on each local server and merge the local clusters on the global server.
At the local level, each server ℓ maintains a profile of ( V t,i , b t,i , T t,i ) for its own local users i ∈ U ℓ , where V t,i is Gramian matrix, b t,i is the moment vector of regressand by the regressors, and T t,i is the number of times that i has appeared up to time t .
At the beginning of s phase (or t = 2 s + 1 ), based on profiles ( V t,i , b t,i , T t,i ) i ∈U ℓ , each server ℓ maintains an undirected graph structure G ℓs , where nodes represent all local users U ℓ and a pair of users are connected by an edge if they are similar.
We initialize the graph by a complete graph G ℓ 0 and gradually delete edges at every beginning of each phase s .
Specifically, in line 3, we delete edge between any user i 1 and i 2 if the distance between their estimated preference vector are larger than the following threshold.
[Equation]
where ˆ θ t,i =
( λ I + V t,i ) − 1 b t,i and F ( x ) = (cid:113) 1+ln(
+ x
[Equation]
1+ x ) .
After the deletion, users in connected components j ∈
C ( G ℓs ) are grouped into local cluster j .
In line 5, server ℓ uploads the clustered information I s,ℓ
[Equation]
) to the global server, which contains clustered information for each cluster j ∈ C ( G ℓs ) .
Note that I s,ℓ are added with random perturbation to protect the users’ data, which will be introduced shortly after.
In line 7, when the global server receives the privatized clustered information from all servers, it performs a merge operation to merge clusters from different servers whose estimated clustered preference vectors are close into a global clusters according to the following inequality.
[Equation]
where ˆ θ ℓs,j = ( ˜ V ℓs,j ) − 1 ˜ b ℓs,j and F ( x
[Equation]
= (cid:113)
[Equation]
+ln(
[Equation]
+ x ) (
[Equation]
+ x ) .
P s denotes the set of m s global clusters.
Note that the local clusters in the same global cluster indexed by k ∈ [ m s ] will communicate and share protected clustered information with each other in an asynchronous manner.
At the beginning of phase s , if the new global cluster structure P s is different from P s − 1 at phase s − 1 , we will renew the shared global information ( S gt,k , u gt,k , T gt,k ) for k ∈ [ m s ] .
For local servers ( ℓ, j ) in the same global cluster P s,k , the local synchronized information ( S ℓt,j , u ℓt,j , T ℓt,j ) , the upload buffers (∆ S ℓt,j , ∆ u ℓt,j , ∆ T ℓt,j ) and download buffers (∆ S − ℓ t,j , ∆ u − ℓ t,j , ∆ T − ℓ t,j ) are renewed.
We also generate a new tree-based privatizer PVT ( ℓ, j ) for each cluster j at server ℓ , which will be introduced later on.
Input: Failure probability α deletion parameter α 1 > 0 , merge parameter α 2 > privacy parameters ε δ .
2: Local
∈
, set graph G ℓ= ( U ℓ , E ℓ 0 ) local information, upload buffers, download buffers: ( S ℓ 0 , 1 , u ℓ 0 , 1 , T ℓ 0 , 1 ) =
(∆ S ℓ 0 , 1 , ∆ u ℓ 0 , 1 , ∆ T ℓ 0 , 1 ) = (∆ S − ℓ 0 , 1 , ∆ u − ℓ 0 , 1 , ∆ T − ℓ 0 , 1 ) =
( 0 d × d , 0 d × 1 , 0) , perturbations ℓt
¯ ℓt, ℓt, ℓt, 1 )
0 d × d , 0 d × 1 ) .
P set global information
4: for
., do 5:
Detect and adjust clusters (Algorithm 2).
6: for τ
1 , ..., 2 s do 7:
Compute the total time step t
2 s − 2 + τ .
Advance all parameter, e.g., S ℓt,j = S ℓt − 1 ,j .
User i t at local server l t arrives and l t gets the local cluster j t that i t belongs to based on G ℓs .
Compute local β l t t,j t according to Lemma 1.
Local server receives feasible context set
D t and recommends item x t = arg max x ∈ D t x ⊤ ( S l t t,j t ) − 1 u l t t,j t + β l t t,
j t ∥ x ∥ ( S ltt,jt ) − 1 .
User i t receives feedback y t , update the user i t ’s information: ( T t,i t , V t,i t , b t,i t )
+= (1 , x
t x ⊤ t , y t x t ) and others unchanged.
Check upload event (Algorithm 3).
Check download event (Algorithm 4).
15: end for 16: end for
Asynchronous Communication Protocol.
In this work, we design a novel asynchronous communication protocol to incorporate the randomly arriving users.
To reduce the communication cost, our high-level idea is to use the delayed communication, where the feedback are temporarily stored in buffers and only if the stored information exceeds a threshold, the upload/download events are triggered.
Such a threshold will ensure that the local information will not diverge too far from the global information, which in turn will not diverge too far from the scenario when information are fully synchronized.
Also note that our communication is conducted in the asynchronous manner at the local cluster level.
In other words, all local clusters indexed by ( ℓ, j ) ∈ P s,k will establish connection with each other within the global cluster k .
For each local cluster ( ℓ, j ) , it stores a local copy of the sufficient statistics ( S ℓt,j , u ℓt,j , T ℓt,j ) and a upload buffer (∆ S ℓt,j ,
∆ u ℓt,j , ∆ T ℓt,j ) .
For the global server, it prepares for each local cluster ( ℓ, j ) a download buffer (∆ S − ℓ t,j , ∆ u − ℓ t,j , ∆ T − ℓ t,j ) , which are used to send other local servers’ information to the local cluster.
It also maintains the global statistics ( S gt,k , u gt,k , T gt,k ) to save the data uploaded from local clusters in global cluster k .
Our proposed communication framework consists of two components: the upload protocol (Algorithm 3) and the download protocol (Algorithm 4).
For the upload protocol, at each time step t , user i t visits the server l t and receives recommended item x t .
After the user i t interacts with the environment and observes feedback ( x t , y t ) , the local server updates the upload buffers in line 1 and checks the following condition to decide whether to upload the upload buffer:
[Equation]
where H l t t,j t and ¯ H l t t,j t are tentative and current perturbation for privacy protection, respectively.
If the condition is satisfied, the local server sends (∆ S l t t,j t , u l t t,j t , T l t t,j t ) to the global cluster k t .
The global server then merges the uploaded information into the global information in lineand also sends it to download buffers for other local clusters ( ℓ, j ) ̸
= ( l t , j t ) ∈
P s,k t in line 6.
For local cluster ( l t , j t ) itself, the local server updates the local statistics and initializes the upload buffer using the newly generated perturbation in lines 8 to 10.
For the download protocol, at each time step t , the global server will check the deviation between global statistics and the local statistics via following condition:
[Equation]
independently for local clusters ( ℓ, j ) in global cluster P
[Equation] [Equation]
t .
If any cluster ( ℓ, j ) satisfies such condition, the global server sends the information from other clusters to ( ℓ, j ) , which is used to update ( ℓ, j ) ’s local statistics.
Finally, the global server cleans the download buffer.
Tree-Based Privacy Protocol.
To ensure the uploaded information are privatized, we adopt the tree-based privatizer to generate random perturbations H ℓt,j and h ℓt,j whenever an upload event happens.
Note that the privatier subroutine is at the local cluster level and a new privatizer is created if the cluster structure changes at the start of any phase s .
Let x 1 , ..., x T be a (matrix-valued) sequence of length T , and s i
= (cid:80) it =
1 x i be the partial sum of the first i elements that will be realised privately.
Generally speaking, the treebased mechanism Dwork et al.
[2006] maintains a binary tree T of depth 1 + ⌈ log T ⌉ , where the leaf nodes contain the elements x i and the parent node maintains the sum of its children.
For each node with value n i , the tree-base mechanism protects privacy by adding noise h i to each node and release n i
+ h
i if queried.
The key advantage is that such a tree only accesses ν =
O (log T ) nodes to compute and release the partial sum
s i , which means the perturbation is at most O ( ν ) instead of O ( T ) .
Following this general idea, we implement the tree-based privatizer ( ℓ, j ) that satisfies the requirements of CDP.
Recall that we only need to protect the information uploaded to the global server, it suffices to maintain a tree T ℓj of depth ν
= O (1 + ⌈ log t c ⌉ ) for the upload event, where t c is the total number of uploads.
To make the partial sums private, we insert a random noise matrix to each node in T ℓj , similar to that of Shariff and Sheffet [2018] and Dubey and
Pentland [2020],.
Specifically, we sample a random matrix ¯
N ∈ R ( d +1) ×
( d +1) where each entry ¯ N p,q is drawn from i.i.d.
Gaussian distribution N (0 , σ noise ) and symmetrize it to get N = ( ¯ N ⊤ + ¯ N ) / √ 2 .
It follows that in order to ensure the whole tree is ( ε, δ ) -DP, each node should preserve ( ε/ (cid:112) 8 ν log(2 /δ ) , δ/ 2) -DP.
In other words, it suffices to set the variance σ noise = 64 ν log(2 /δ ) 2 /ϵ 2 for each tree node.
Note that at each upload round t , the total noise added to the partial sum is the summation of at most ν random matrices with size ( d + 1) ×
( d + 1) , where the top-left ( d × d ) - submatrix forms H ℓt,j and the first d elements from the right-most ( d + 1) × 1 vector forms
h ℓt,j .
By concentration of random matrices Tao [2011], we have with probability at least (1 − αmL ) , the operator norm of H ℓt,j is (cid:13)(cid:13)
H ℓ t
[Equation]
for any ℓ ∈ [ L ] , j ∈ [ m ] , t ∈
[ T ] .
Recommendation Procedure.
At each time step t , the recommended item x t for user i t is selected as follows.
When the current cluster is correct (which is guaranteed after
O (log T ) rounds and to be proved later), the estimated ˆ θ t
=
( S l t t,j t ) − 1 u l t t,j t is computed using the local information S l t t,j t and u l t t,j t .
Since by Lemma 1, (cid:13)(cid:13)(cid:13) θ i t − ˆ θ t (cid:13)(cid:13)(cid:13) 2 ≤ β l t t,j t , the confidence radius is β l t t,
j t ∥ x ∥ ( S ltt,jt ) − 1 , which characterizes the exploration bonus for item x ∈ D t .
Then the local server will recommend the item x t ∈ D t that maximizes the x ⊤ ˆ theta t plus the above exploration bonus.
Finally, the user will receive feedback y t and the system updates corresponding statistics for better decision in future rounds.
Lemma 1.
Under the setting of FCLUB and fix a local cluster j located at the server ℓ which shares the information with L ′ ≤ L clusters (including itself), let the true preference vector be θ ∗ and the true cluster be j ∗ , let ˆ θ ℓt,j = ( S ℓt,j ) − 1 u ℓt,j .
When all (global) clusters are correctly identified and partitioned, it holds with probability at least 1 − 2 α , (cid:13)(cid:13)(cid:13)
[Equation]
(8) where
[Equation]
RESULTS
Recall that perturbations ¯ H ℓt,j , ¯ H ℓt,j are designed to satisfy the ( ε, δ, L, m ) -CDP requirement.
In particular, the privacy
Algorithm
1: t = 2 s − 1 .
2: for ℓ ∈ [ L ] do 3:
Set G ℓs by deleting any edge ( i 1 , i 2 )
∈ G ℓs − 1 if Equation (3) holds.
For ℓ ∈ [ L ] , j ∈ C ( G ℓs ) , generate new perturbation
H ℓt,j , h ℓt,j using PVT ( ℓ, j ) in Algorithm 5 and set historical ¯ H ℓt,j = H ℓt,j , h ℓt,j = ¯ h ℓt,j .
For ℓ ∈ [ L ] , upload the local clustered information I s,ℓ =
( C ℓs,j , ˜ V
ℓs,j , ˜ b ℓs,j , ˜ T ℓs,j ) j ∈ C ( G ℓs ) to the global server, where ( ˜ V ℓs,j , ˜ b ℓs,j , ˜ T ℓs,j ) =
(2 ρ I +
H ℓt,j , h ℓt,j , 0)
+ (cid:80) i ∈
C ℓs,j ( V t,i , b t,i , T t,i ) .
6: end for 7: The global server does global merge based on I s and get m s global clusters P s = { P s, 1 , ..., P s,m s } , where the two local clusters C ℓ 1 t,j 1 , C ℓ 2 t,j 2 (with ℓ 1 ̸ = ℓ 2 ) are merged together in P s,k if Equation (4) holds.
8: if s = 0 or P s ̸
= P s − 1 then 9: //Renew
the cluster information.
10: for k ∈ [ m s ] do 11:
Set global gram matrix ( S gt,k , u gt,k , T gt,k ) =
(cid:80) ( ℓ,j )
∈
P s,k ( ˜ V ℓs,j , ˜ b ℓs,j , ˜ T ℓs,j ) .
12: for ( ℓ, j ) ∈ P s,k do 13:
Set ( S ℓt,j , b ℓt,j , T ℓt,j ) =
( S gt,k , b gt,k , T gt,k ) .
Create new perturbation H ℓt,j , h ℓt,j using
PVT ( ℓ, j ) in Algorithm 5.
Set new (∆ S ℓt,j , ∆ u ℓt,j , ∆ T ℓt,j ) =
(3 ρ I +
H ℓt,j − ¯ H ℓt,j , h ℓt,j − ¯ h ℓt,j , 0) 16:
Set new (∆ S − ℓ t,j , ∆ u − ℓ t,j , ∆ T − ℓ t,j ) =
17: end for 18: end for 19: end if budget ( ε, δ ) affects the regret and communication bounds via the following quantities ( ρ max , ρ min , κ ) , which can be treated as spectral bounds for ¯ H ℓt,j , ¯ H ℓt,j .
Let
˜ H ℓt,j = 2 ρ I +
[Equation]
,t I + ¯ H ℓt,j , where c ℓj,t is the number of uploads for local server ℓ and cluster j .
Definition 2 (Approximately-accurate ρ min , ρ max and κ ) .
The bounds 0 < ρ t, min ≤ ρ t, max and κ > 0 are ( α/ ( mL )) - accurate for ( ¯ H ℓt,j ) for any ℓ ∈
[ L ] , j ∈ [ m ] and t ∈ [ T ] :
[Equation]
(9) with probability at least (1 − αmL ) .
As will be shown later, our communication protocol ensures c ℓ j,t ∈ [0 , d log T/ log(min { U, D } )] , so ρ min = ρ , ρ max = 3 ρ + 3 ρd log T/ log(min { U, D } ) , and κ = (cid:13)(cid:13)
¯ h ℓt,j (cid:13)(cid:13) / √ ρ , where ρ ≜ 8 √ 2 ν log(4 /δ )
(4 √ d + 2 log(2 mL/α )) /ε is given by our privatizer.
In the following, we will give general regret and communi-
1: Update upload buffer (∆ S l t t,j t , ∆ u l t t,j t , ∆ T l t t,j t ) += ( x t x ⊤ t , y t x t , 1) .
2: if det ( S l t t,j t + ∆ S l t t,j t ) / det ( S l t t,j t ) ≥ U then 3:
The global cluster finds k t so that ( l t , j t ) ∈
P s,k t .
Update global information ( S gt,k t , u gt,k t , T gt,k t )
+= (∆ S l t t,j t , ∆ u l t t,j t , ∆ T l t t,j t ) .
5: for ( ℓ, j ) ̸
= ( l t , j t ) ∈
P s,k t do 6:
Global server updates other servers’ download buffer (∆ S − ℓ t,j , ∆ u − ℓ t,j , ∆ T − ℓ t,j ) += (∆ S l t t,j t , ∆ u l t t,j t , ∆ T l t t,j t ) .
7: end for 8:
Local server l t updates the local statistics: ( S l t t,j t , u l t t,j t , T l t t,j t ) += (∆ S l t t,j t , ∆ u l t t,j t , ∆ T l t t,j t ) .
Local server l t sets ( ¯ H l t t,j t , ¯ h l t t,j t ) =
( H l t t,j t , h l t t,j ) and creates new perturbation H ℓt,j t , h ℓt,j t using the tree-based privatizer PVT ( l t , j t ) .
Local server l t initializes the upload buffer using the new perturbation: (∆ S l t t,j t , ∆ u l t t,j t , ∆ T l t t,j t ) =
(3 ρ I + H l t t,j − H l t t,j t , h l t t,j t − ¯ h l t t,j t , 0) .
11: end if
1: for ( l, j ) ∈ P s,k t do 2: if det ( S gt,k t ) / det ( S ℓt,j ) ≥ D then 3:
Local server receives ( S ℓt,j , u ℓt,j , T ℓt,j ) += (∆ S − ℓ t,j , ∆ u − ℓ t,j , ∆ T − ℓ t,j ) .
Global server cleans the download buffer: (∆ S − ℓ t,j , ∆ u − ℓ t,j , ∆ T − ℓ t,j ) =
5: end if 6: end for cation bounds using ( ρ max , ρ min , κ ) and replace them with their exact values.
4.1 REGRET BOUND
We give the following theorem as our main result for the regret bound.
Theorem 1.
Suppose the cluster structure over the users and items satisfy the assumptions in Section 2 with gap parameter γ > 0 and item regularity parameter 1 ≥ λ x > 0 .
If the privatizer produces random perturbation that are (1 / (8 mLT ))
-accurate
as in Definition 2, with probability at least 1 − 1 /T , the regret is upper bounded by
[Equation]
We will give the proof sketch for the above theorem 1.
N ∈ R ( d +1) ×
( d +1) , where N = ( ¯ N + ¯ N ⊤ ) / √ 2 and ¯ N ∈ R ( d +1) ×
( d +1) with ¯ N p,q ∼ N (
, 64 ν log
Proof.
Our proof mainly consists of two parts.
The first part bounds the number of exploration rounds 2 T 0 after which the overall user clusters are correctly detected at the global server.
The second part is to bound the regret for the asynchronous contextual linear bandits after the clusters are partitioned correctly.
Different from standard online clustering bandits, the key technical challenge is to take care of the additional random
Gaussian noise produced by the privater, which perturbs the true observation that is needed for global cluster detection and the regret analysis for contextual linear bandits.
Moreover, such perturbed observation are also lagged behind the instant observation, since FCLUB-CDP adopts the "delayed" asynchronous communication where upload and download are triggered occasionally.
This makes standard contextual bandit analysis no longer works and requires new proof techniques to handle the gap between instant observation and the lagged (and perturbed) observation.
For the first cluster detection part, by the assumption of item regularity, we prove that after t ≥ O ( n ( log T λ 2 x + dσ 20 log T λ x γ 2 )) rounds, the local estimates are accurate enough so that the local clusters are correctly identified, similar to that of Li and
Zhang [2018].
Specifically, the 2-norm distance between local estimate ˆ θ t,i and the truth θ i for any user i is less than γ/ 4 .
Thus the local clusters are split correctly for all local servers.
Now for the global cluster detection, the global server receives the aggregated observation from correctly partitioned local clusters, in which random Gaussian noises are added.
Based on spectra property of Gaussian noise matrices (definition 2), the global server will spend additional
O ( n log(1 /δ )
log T λ x εγ) rounds so that the perturbed estimate ˆ θ ls,j are accurate enough at the beginning of phase s = ⌈ log 2 T 0 ⌉ , where T
= O ( n ( log T λ 2 x + dσ 20 log T λ x γ+ log(1 /δ )
log T λ x εγ)) .
Therefore, after t > 2 T 0 , the overall user clusters are partitioned correctly.
For the regret after 2 T 0 , we use the delayed update technique from [Abbasi-Yadkori et al., 2011, Section 5.1], which only recomputes the confidence radius only O (log T ) times and hence saves computation.
The same strategy can also be applied for the delayed communication.
The key analysis relies on using the upload and download condition in eq.
(5) and eq.
(6), so that the actually-used cluster confidence radius is at most Γ times larger than that if all local servers upload their perturbed observations in a fully synchronized manner, where Γ = (cid:112) D (1 + ( L − 1)
( U − 1))
+ U − 1 .
This will give a Γ R j ( T j ) regret for the second part, where
R j ( T j ) is the private-version regret for the cluster j if all observation are synchronized at each round.
The full proof is put in Appendix B.
■
4.2 COMMUNICATION COST
We give the following theorem to bound the total communication cost.
Theorem 2.
Under the CDP setting, the total communication cost satisfies:
[Equation]
Proof.
The total communication cost also has two parts: the upload at the beginning of each phase for global cluster detection and the asynchronous communication within each phase for information sharing.
For the first part, the algorithm has at most log T phases and at each phase, there are total mL local clusters uploading the clustered information, hence the total communication cost is O ( mL log T ) .
For the second part, recall that we adopt the delayed asynchronous communication protocol and the total number of uploads and downloads can be bounded by O ( dmL log T ) .
See Appendix C for the detailed proofs.
■
4.3 PRIVACY GUARANTEE
Theorem 3.
Algorithm 1 preserves ( ε, δ, L, m ) -CDP as defined in Definition 1.
Proof.
The CDP condition is satisfied by assigning the right amount of Gaussian noise in each tree node of our treebased privacy protocol in Section 3.
See Appendix D for details.
■
4.4 DISCUSSION AND COMPARISON
Discussion on the Regret Bounds.
For the regret bound, our result has two terms: the regret before the clusters are correctly partitioned n ( log T λ 2 x + σ
[Equation]
d log
[Equation] [Equation]
) and the regret after the clusters are correctly partition
O ( dL (cid:113) mT log(
[Equation]
log 1 .
5 T ) .
We will compare our results with several degenerate cases, given that we are the first work to study the federated clustering of bandits setting.
For these cases, the additional CDP causes at most
O ( (cid:113) log(1 /δ ) ε ) factor and asynchronous communication protocol causes at most O ( √ dL log T ) factor in general.
First, when m = 1 , L = 1 , our setting degenerates to the linear bandits with DP where all users share the same underlying parameter.
Compared to Shariff and Sheffet [2018] which gives a O ( (cid:113) d
[Equation]
√
[Equation]
5 T ) regret withcommunication, our bound has a O ( √ d ) additional factor (or more precisely O ( √ d log log T ) factor) for the second term, which stems from the larger perturbation in order to protect total O ( d log T ) communication rounds.
Second, when L = 1 , our setting reduces to the online clustering bandits with DP, Li and Zhang [2018] gives a
O ( n (
[Equation]
+ d
[Equation]
mT log T ) for the non-DP version.
Since CDP mechanism requires random perturbation, the clustering process suffers an additional n log T log(1 /δ )
λ x εγfor the first term and the second regret term now has a new (cid:113)
[Equation]
leading factor due to the CDP requirements.
Third, when m = 1 and if we consider the special case when each local server only has one user and all users come in a round-robin manner, our setting reduces to the distributed linear bandits with DP.
Dubey and Pentland [2020] provides a synchronized algorithm that achieves
O ( (cid:113)
[Equation] [Equation]
log 1 .
5 T ) , our second term has an additional √ dL factor because of different communication protocol, which enables asynchronous communication at the cost of the larger O ( dL log T ) (compared with O ( L log T ) ) communication rounds and an additional Γ = O ( √ L ) factor in the confidence radius.
Finally, there is a lower bound Ω( √ dmT ) , if we consider the case where the clustering structure is known, the communication and privacy budgets are unlimited and each cluster contains equal number of users.
In this case, it is equivalent to learn m independent linear bandits, each with expected rounds T/m and according to Dani et al.
[2008], the lower bound is Ω( (cid:80) i ∈
[ m ] (cid:112) dT/m )
=
Ω( √ dmT ) .
In other cases, the regret lower bound will be greater and the lower bound Ω( √ dmT ) still holds.
Our regret bound matches the lower bound up to a factor of O ( L (cid:113)
d log(1 /δ )
ε log 1 .
5 T ) .
Discussion on the Communication Cost.
Our communication cost also has two terms: the first O ( mL log T ) term for identifying clusters at the beginning of each phase and the leading O ( dmL log T log(min { U,D } ) term for our asynchronous communication protocol.
Compared with Dubey and Pentland [2020] when m =
and users come at the round-robin manner, our communication has an additional O ( d ) factor.
Due to the specialty of the user arrival, the same paper can achieve communication cost independent of T at the cost of
O (log( LT )) additional factor in the regret.
Though our total communication cost can not be reduced below O ( mL log T ) due to the first term, it will be interesting to consider whether the similar trade-off works for our asynchronous protocol in the future work.
[Figure] [Figure]
EXPERIMENTS
To validate our theoretical findings, we conduct experiments on a synthetic dataset and a real-world MovieLens dataset.
Algorithm 1 is denoted as CDP-FCLUB-DC and we also present its non-private version FCLUB-DC and its nonprivate, synchronized version FCLUB (by instantly uploading the observations).
The baselines include CLUB which uses a separate CLUB algorithm Gentile et al.
[2014] for each local server; SCLUB which uses a separate SCLUB algorithm Li et al.
[2019] for each server; and LinUCB which uses a separate LinUCB algorithm Abbasi-Yadkori et al.
[2011] for each user.
We also consider the synchronized and asynchronous version of Algorithm 1 (denoted as Homo and Homo-DC, respectively) by treating users are identical with the same preference vector.
Note that all results are averaged over ten random seeds, and we provide mean results with one unit of standard derivation for each curve.
Due to the space limit, we provide the detailed experiment settings (including data generation and processing) in Appendix E.1, the parameter study Appendix E.2, the communication cost
Appendix E.3 and running time results in Appendix E.4, respectively.
Synthetic Dataset.
We first conduct experiments on a synthetic dataset.
In Figure 2a, we compare our algorithm
CDP-FCLUB-DC with the baselines listed above.
The vertical axis indicates the cumulative regret and the horizontal axis indicates the round t .
In general, our algorithm CDPFCLUB-DC’s performance has a clear advantage over baseline SCLUB, CLUB, LinUCB, Homo and Homo-DC.
Since
Homo-DC and Homo assumes users are in the same cluster, they mistakenly merge different clusters and suffer linear regrets, indicating the correctness of cluster detection is essential to have small regrets.
Compared with SCLUB and
CLUB that only perform local clustering operations, we can verify the correctness of our algorithm’s clustering operations at the global level, which successfully leverages the collaborative effects across different local servers.
As expected, CDP-FCLUB-DC performs a little worse than
FCLUB and FCLUB-DC due to the delayed communication and cluster differential privacy requirements.
MovieLens Dataset.
In this section, we also compare our algorithm CDP-FCLUB-DC with the baselines listed above on movie recommendations with the MovieLens dataset.
The performances are shown in Figure 2b.
Our algorithm CDPFCLUB-DC’s performance has an advantage over baseline
SCLUB, CLUB, LinUCB, Homo and Homo-DC in general.
Figure 2b shows CDP-FCLUB-DC performs worse than
FCLUB and FCLUB-DC
due to the delay communication and cluster differential privacy (CDP) as we have explained in synthetic dataset part.
Different from the synthetic dataset, in the early stage, our algorithm needs more time to identify the underlying cluster structure.
But after all user clusters are correctly detected at the global server, our algorithm performs better than Homo/Homo-DC that assume users are homogeneous, CLUB/SCLUB on each local server and
LinUCB on each user.
CONCLUSION AND FUTURE WORK
In this paper, we formulate the federated online clustering of bandits problem, which generalizes the clustering of bandits problem to its federated counterpart.
To tackle this new problem, we propose a FCLUB-CDP algorithm, which simultaneously achieves sublinear regret, sublinear communication complexity and satisfies our newly-defined clustered differential privacy requirements.
Compared with benchmark algorithms, we show that FCLUB-CDP achieves superior performance regarding regret and communication cost.
There are many compelling directions for future study.
For example, it would be interesting to study our problem where local differential privacy is considered.
One could also study a more efficient protocol to further reduce the communication cost.
Acknowledgement
The corresponding author Shuai Li is supported by National
Natural Science Foundation of China (62006151).
This work is sponsored by Shanghai Sailing Program.
The work of
John C.S. Lui was supported in part by the RGC SRFS21224S02.