D ISTRIBUTED D IFFERENTIAL P RIVACY IN M ULTI -A RMED B ANDITS
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider the standard K -armed bandit problem under a distributed trust model of differential privacy (DP), which enables to guarantee privacy without a trustworthy server.
Under this trust model, previous work on private bandits largely focus on achieving privacy using a shufﬂe protocol, where a batch of users data are randomly permuted before sending to a central server.
This protocol achieves ( " ,  ) or approximate-DP guarantee by sacriﬁcing an additive O
[Equation]
◆ factor in T -step cumulative regret.
In contrast, the optimal privacy cost to achieve a stronger ( " , 0 ) or pure-DP guarantee under the widely used central trust model is only ⇥ ⇣
K log T " ⌘ , where, however, a trusted server is required.
In this work, we aim to obtain a pure-DP guarantee under distributed trust model while sacriﬁcing no more regret than that under the central trust model.
We achieve this by designing a generic bandit algorithm based on successive arm elimination, where privacy is guaranteed by corrupting rewards with an equivalent discrete Laplace noise ensured by a secure computation protocol.
We also show that our algorithm, when instantiated with Skellam noise and the secure protocol, ensures R´enyi differential privacy – a stronger notion than approximate DP – under distributed trust model with a privacy cost of O ⇣
K p log T " ⌘ .
Our theoretical ﬁndings are corroborated by numerical evaluations on both synthetic and real-world data.
I NTRODUCTION
The multi-armed bandit (MAB) problem provides a simple but powerful framework for sequential decision-making under uncertainty with bandit feedback, which has attracted a wide range of practical applications such as online advertising (Abe et al., 2003), product recommendations (Li et al., 2010), clinical trials (Tewari & Murphy, 2017), to name a few.
Along with its broad applicability, however, there is an increasing concern of privacy risk in MAB due to its intrinsic dependence on users’ feedback, which could leak users’ sensitive information (Pan et al., 2019).
To alleviate the above concern, the notion of differential privacy , introduced by Dwork et al.
(2006) in the ﬁeld of computer science theory, has recently been adopted to design privacy-preserving bandit algorithms (see, e.g., Mishra & Thakurta (2015); Tossou & Dimitrakakis (2016); Shariff & Sheffet (2018)).
Differential privacy (DP) provides a principled way to mathematically prove privacy guarantees against adversaries with arbitrary auxiliary information about users.
To achieve this, a differentially private bandit algorithm typically relies on a well-tuned random noise to obscure each user’s contribution to the output, depending on privacy levels " ,  – smaller values lead to stronger protection but also suffer worse utility (i.e., regret).
For example, the central server of a recommendation system can use random noise to perturb its statistics on each item after receiving feedback (i.e., clicks/ratings) from users.
This is often termed as central model (Dwork et al., 2014), since the central server has the trust of its users and hence has a direct access to their raw data.
Under this model, an optimal private MAB algorithm with a pure DP guarantee (i.e., when  = 0 ) is proposed in Sajed & Sheffet (2019), which only incurs an additive O ⇣
K log T " ⌘ term in the cumulative regret compared to the standard setting when privacy is not sought after (Auer, 2002).
However, this high trust model is not always feasible in practice since users may not be willing to share their raw data directly to the server.
This motivates to employ a local model (Kasiviswanathan
[Table]
et al., 2011) of trust
[Table]
where DP is achieved without a trusted server as each user perturbs her data prior to sharing with the server.
This ensures a stronger privacy protection
[Table]
but leads to a high cost in utility due to large aggregated noise from all users.
As shown in Ren et al.
(2020), under the local model, private MAB algorithms have to incur a multiplicative 1 / " 2 factor in the regret rather than the additive one in the central model
[Table]
In attempts to recover the same utility of central model while without a trustworthy server like the local model
[Table]
an intermediate DP trust model called distributed model has gained an increasing interest, especially in the context of (federated) supervised learning (Kairouz et al., 2021b; Agarwal et al., 2021; Kairouz et al., 2021a; Girgis et al., 2021; Lowy & Razaviyayn, 2021).
Under this model, each user ﬁrst perturbs her data via a local randomizer, and then sends the randomized data to a secure computation function.
This secure function can be leveraged to guarantee privacy through aggregated noise from distributed users.
There are two popular secure computation functions: secure aggregation (Bonawitz et al., 2017) and secure shufﬂing (Bittau et al., 2017).
The former often relies on cryptographic primitives to securely aggregate users’ data so that the central server only learns the aggregated result, while the latter securely shufﬂe users’ messages to hide their source.
To the best of our knowledge, distributed DP model is far less studied in online learning as compared to supervised learning, with only known results for standard K -armed bandits in Tenenbaum et al.
(2021), where secure shufﬂing is adopted.
Despite being pioneer work, the results obtained in this paper have several limitations: (i)
The privacy guarantee is obtained only for approximate DP (  > 0 ) – a stronger pure DP (  = 0 ) guarantee is not achieved; (ii) The cost of privacy is a multiplicative p log(1 /  ) factor away from that of central model, leading to a higher regret bound; (iii)
The secure protocol works only for binary rewards (or communication intensive for real rewards).
Our contributions.
In this work, we design the ﬁrst communication-efﬁcient MAB algorithm that satisﬁes pure DP in the distributed model while attaining the same regret bound as in the central model (see Table 1).
We overcome several key challenges that arise in the design and analysis of distributed DP algorithms for bandits.
We now list the challenges and our proposed solutions below.
(a) Private and communication efﬁcient algorithm design.
Secure aggregation (SecAgg) works only in the integer domain due to an inherent modular operation (Bonawitz et al., 2017).
Hence, leveraging this in bandits to achieve distributed DP with real rewards needs adopting data quantization, discrete privacy noise and modular summation arithmetic in the algorithm design.
To this end, we take a batch version of the successive arm elimination technique as a building block of our algorithm, and on top of it, employ a privacy protocol tailored to discrete privacy noise and modular operation (see Algorithm 1).
Instantiating the protocol at each user with P´olya random noise, we ensure that our algorithm satisﬁes pure DP in the distributed model.
Moreover, the communication bits per-user scale only logarithmicaly with the number of participating users in each batch.
(b) Regret analysis under pure DP with SecAgg.
While our pure DP guarantee exploits known results for discrete Laplace mechanism, the utility analysis gets challenging due to modular clipping of SecAgg.
In fact, in supervised learning, no known convergence rate exists for SGD under pure DP with SecAgg (although the same is well-known under central model).
This is because modular
clipping makes gradient estimates biased, and hence, standard convergence guarantees using unbiased estimates do not hold.
In bandits, however, we work with zeroth order observations to build estimates of arms’ rewards, and require high-conﬁdence tight tail bounds for the estimates to analyse convergence.
To this end, relying on tail properties of discrete Laplace and a careful analysis of modular operation, we prove a sublinear regret rate of our algorithm, which matches the optimal one in the central model, and thus, achieves the optimal rate under pure DP (see Theorem 1).
(c) Improved regret bound under RDP.
While our main focus were to design the ﬁrst bandit algorithm with pure distributed DP that achieves the same regret rate under central model, our template protocol is general enough to obtain different privacy guarantees by tuning the noise at each user.
We demonstrate this by achieving R´enyi differential privacy (RDP) (Mironov, 2017) using a Skellam random noise.
RDP is a weaker notion of privacy compared to pure DP, but it is still stronger than approximate DP.
It also provides a tighter privacy accounting for composition compared to approximate DP.
This is particularly useful for bandit algorithms, when users may participate in multiple rounds, necessitating the need for privacy composition.
Hence, we focus on RDP with SecAgg and and show that a tighter regret bound compared to pure DP can be achieved (see Theorem 2) by proving novel tail-bound for Skellam distribution.
We support our theoretical ﬁndings with extensive numerical evaluation over bandit instances generated from both synthetic and real-life data.
Finally, our analysis technique is also general enough to recover best-known regrets under central and local DP models while only using discrete privacy noise (see Appendix H).
This is important in practice since continuous Laplace noise might leak privacy on ﬁnite computers due to ﬂoating point arithmetic (Mironov, 2012), which is a drawback of existing central and local DP MAB algorithms.
P RELIMINARIES
In this section, we formally introduce the distributed differential privacy model in bandits.
Before that we recall the learning paradigm in multi-armed bandits and basic differential privacy deﬁnitions.
Learning model and regret in MAB.
At each time slot t 2 [ T ] := { 1 , . .
, T } , the agent (e.g., recommender system) selects an arm a 2 [ K ] (e.g., an advertisement), recommends it to a new user t and obtains an i.i.d reward r t (e.g., a rating indicating how much she likes it), which is sampled from a distribution over [0 , 1] with mean given by µ a .
Let a ⇤ :
= argmax a 2 [ K ] µ a be the arm with the highest mean and denote µ ⇤ :
= µ a ⇤ for simplicity.
Let  a := µ ⇤  µ a be the gap of the expected reward between the optimal arm a ⇤ and any other arm a .
Further, let N a ( t ) be the total number of times that arm a has been recommended to ﬁrst t users.
The goal of the agent is to maximize its total reward, or equivalently to minimize the cumulative expected pseudo-regret, deﬁned as
[Equation]
Differential privacy.
Let D =
[0 , 1] be the data universe, and n 2 N be the number of unique users.
we say D, D 0 2 D n are neighboring datasets if they only differ in one user’s reward D i for some i 2 [ n ] .
We have the following standard deﬁnition of differential privacy (Dwork et al., 2006).
Deﬁnition 1 (Differential Privacy) .
For " ,  > 0 , a randomized mechanism M satisﬁes ( " ,  ) -DP if for all neighboring datasets D, D 0 and all events E in the range of M , we have
[Equation]
The special case of ( " , 0) -DP is often referred to as pure differential privacy , whereas, for  > 0 , ( " ,  ) -DP is referred to as approximate differential privacy .
We also consider a related notion of privacy called R´enyi differential privacy (RDP) Mironov (2017), which allows for a tighter composition compared to approximate differential privacy.
Deﬁnition 2 (R´enyi Differential Privacy) .
For ↵ > 1 , a randomized mechanism M satisﬁes ( ↵ , " ( ↵ ))
-RDP if for all neighboring datasets D, D 0 , we have D ↵ ( M ( D ) , M ( D 0 ))  " ( ↵ ) , where
D ↵ ( P, Q ) is the R´enyi divergence (of order ↵ ) of the distribution P from the distribution Q , and is given by D ↵ ( P, Q ) :
= 1 ↵  1 log
[Equation]
Distributed differential privacy.
A distributed bandit learning protocol P =
( R , S , A ) consists of three parts: (i) a (local) randomizer
R at each user’s side, (ii) an intermediate secure protocol
S , and (iii) an analyzer A at the central server.
Each user i ﬁrst locally apply the randomizer R on its raw data (i.e., reward) D i , and sends the randomized data to a secure computation protocol S (e.g., secure aggregation or shufﬂing).
This intermediate secure protocol S takes a batch of users’ randomized data and generates inputs to the central server, which utilizes an analyzer A to compute the output (e.g., action) using received messages from S .
The secure computation protocol S has two main variations: secure shufﬂing and secure aggregation .
Both of them essentially work with a batch of users’ randomized data and guarantee that the central server cannot infer any individual’s data while the total noise in the inputs to the analyzer provides a high privacy level.
To adapt both into our MAB protocol, it is natural to divide participating users into batches.
For each batch b 2 [ B ] with n b users, the outputs of S is given by S  R n b ( D ) :
=
S ( R ( D 1 ) , . . .
, R ( D n b )) .
The goal is to guarantee that the the view of all B batches’ outputs satisfy DP.
To this end, we deﬁne a (composite) mechanism
[Equation]
where each individual mechanism S  R n b operates on n b users’ rewards, i.e., on a dataset from
D n b .
With this notation, we have the following deﬁnition of distributed differential privacy.
Deﬁnition 3 (Distributed DP) .
A protocol P =
( R , S , A ) is said to satisfy DP (or RDP) in the distributed model if the mechanism
M P satisﬁes Deﬁnition 1 (or Deﬁnition 2).
In the central DP model, the privacy burden lies with a central server (in particular, analyzer A ), which needs to inject necessary random noise to achieve privacy.
On the other hand, in the local
DP model, each user’s data is privatized by local randomizer R .
In contrast, in the distributed DP model, privacy without a trusted central server is achieved by ensuring that the inputs to the analyzer
A already satisfy differential privacy.
Speciﬁcally, by properly designing the intermediate protocol
S and the noise level in the randomizer R , one can ensure that the ﬁnal added noise in the aggregated data over a batch of users matches the noise that would have otherwise been added in the central model by the trusted server.
Through this, distributed DP model provides the possibility to achieve the same level of utility as the central model while without a trustworthy central server.
A G ENERIC A LGORITHM FOR P RIVATE B ANDITS
In this section, we propose a generic algorithmic framework (Algorithm 1) for multi-armed bandits under the distributed privacy model.
Batch-based successive arm elimination.
Our algorithm builds upon the classic idea of successive arm elimination (Even-Dar et al., 2006) with the additional incorporation of batches and a blackbox protocol P =
( R , S , A ) to achieve distributed differential privacy.
It divides the time horizon T into batches of exponentially increasing size and eliminates sub-optimal arms successively.
To this end, for each active arm a at batch b , it ﬁrst prescribes arm a to a batch of l ( b ) {+= 
 b+} new users.
After pulling the prescribed action a , each user applies the local randomizer R to her reward and sends the randomized reward to the intermediary function S , which runs a secure computation protocol (e.g., secure aggregation or secure shufﬂing) over the total l ( b ) number of randomized rewards.
Then, upon receiving the outputs of S , the server applies the analyzer A to compute the the sum of rewards for batch b when pulling arm a (i.e., R a ( b ) ), which in turn gives the new mean estimate b µ a ( b ) of arm a after being divided by the total pulls l ( b ) .
Then, upper and lower conﬁdence bounds, UCB a ( b ) and LCB a ( b ) , respectively, are computed around the mean estimate b µ a ( b ) with a properly chosen conﬁdence width  ( b ) .
Finally, after the iteration over all active arms in batch b (denoted by the set  ( b ) ), it adopts the standard arm elimination criterion to remove all obviously sub-optimal arms, i.e., it removes an arm a from  ( b ) if UCB a ( b ) falls below LCB a 0 ( b ) of any other arm a 0 2  ( b ) .
It now only remains to design a distributed DP protocol P , which is detailed as follows.
There is one key difference between our algorithm and the VB-SDP-AE algorithm in Tenenbaum et al.
At the start of a batch, VB-SDP-AE uses all the past data to compute reward estimates. In contrast, we adopt the idea of forgetting and use only the data of the last completed batch to design the estimates. This small but important tweak allows us to accumulate “less” noise in reward estimates compared to VB-SDP-AE, which is key to achieve optimal regret.
1: Parameters: # arms K , Time horizon T , privacy level " > 0 , Conﬁdence radii {  ( b ) } b 2: Initialize: Batch count b =1 , Active arm set  ( b )= { 1 , . . .
, K } , Estimate b µ a (1)=0 , 8 a
[ K ] 3: for batch b =1 , 2 , . . .
do 4:
Set batch size l ( b )=2 b 5: for each active arm a 2  ( b ) do 6: for each new user i from 1 to l ( b ) do 7:
Pull arm a and generate reward r ia ( b ) 8:
Send randomized data y ia ( b )
=
R ( r ia ( b )) to S // randomizer 9:
If total number of pulls reaches T , exit 10: end for 11:
Send messages b y a ( b )=
S ( { y ia ( b ) } 1  i  l ( b ) ) to A // secure computation 12:
Compute the sum of rewards R a ( b )
=
A ( b y a ( b )) // analyzer 13:
Compute mean estimate b µ a ( b )=
R a ( b ) /l
( b ) 14:
Compute conﬁdence bounds UCB a ( b )=
b µ a ( b )+  ( b ) and LCB a ( b )=
b µ a ( b )   ( b ) 15: end for 16:
Update active set of arms:  ( b +1)=  a 2  ( b ) :
UCB a ( b )  max a 0 2  ( b ) LCB a 0 ( b ) 17: end for 18: Subroutine: Local Randomizer R ( Input: x i 2 [0 , 1] , Output: y i ) 19: Require: precision g 2 N , modulo m 2 N , batch size n 2 N , privacy level " 20: Encode x i as
[Equation]
( x i g  b x i g c ) 21: Generate discrete noise ⌘ i (depending on n, " , g ) //
random noise generator 22:
Add noise and modulo clip y i
= ( b x i + ⌘ i ) mod m 23: Subroutine: Secure Aggregation S
( Input: y 1 , . . .
, y n , Output: b y ) 24: Require: modulo m 2 N 25: Securely compute b [Equation] = (
P
[Equation] mod [Equation] // black-box function 26: Subroutine: Analyzer A
( Input: b y , Output: z ) 27: Require: precision g 2 N , modulo m 2 N , batch size n 2 N , accuracy level ⌧ 2 R 28: if b y > ng + ⌧ then 29: set z = ( b y  m ) /g // correction for underﬂow 30: else set z = b y/g
Distributed DP protocol via discrete privacy noise. Inspired by Balle et al. (2020); Cheu & Yan (2021)
, we provide a general template protocol P for the distributed DP model, which relies only on discrete privacy noise .
Local randomizer R receives each user
i ’s real-valued data x i and encodes it as an integer via ﬁxed-point encoding with precision g > 0 and randomized rounding.
Then, it generates a discrete noise, which depends on the speciﬁc privacy-regret trade-off requirement (to be discussed later under speciﬁc mechanisms).
Next, it adds the random noise to the encoded reward, clips the sum with modulo m 2 N and sends the ﬁnal integer y
i as input to secure computation function S .
We mainly focus on secure aggregation (SecAgg) for S here.
3 SecAgg is treated as a black-box function as in previous work on supervised learning (Kairouz et al., 2021a), which implements the following procedure: given n users and their randomized messages y
i
Z m (i.e., integer in { 0 , 1 , . . .
, m  1 } ) obtained via R , the SecAgg function S securely computes the modular sum of the n messages, b y = (
P ni =1 y i ) mod m , while revealing no further information on individual messages to a potential attacker, ensuring that it is perfectly secure .
Details of engineering implementations of
SecAgg is beyond the scope of this paper, see Appendix G for a brief discussion on this.
The job of analyzer A is to compute the sum of rewards within a batch as accurately as possible.
It uses an accuracy parameter ⌧ 2 R and g to correct for possible underﬂow due to modular operation and bias due to encoding.
To sum it up, the end goal of our protocol P =
( R , S , A ) is to ensure that it provides the required privacy protection while guaranteeing an output z ⇡
P ni =1 x i with high probability, which is the key to our privacy and regret analysis in the following sections.
A CHIEVING P URE DP IN THE D ISTRIBUTED M ODEL
In this section, we show that Algorithm 1 achieves pure-DP in the distributed DP model via secure aggregation.
To do so, we need to carefully determine the amount of (discrete) noise in R so that the total noise in a batch provides ( " , 0) -DP.
One natural choice is the discrete Laplace noise.
Deﬁnition 4 (Discrete Laplace Distribution) .
Let b > 0 .
A random variable X has a discrete
Laplace distribution with scale parameter b , denoted by Lap Z ( b ) , if it has a p.m.f.
given by
[Equation]
A key property of discrete Laplace that we will use is its inﬁnite divisibility , which allows us to simulate it in a distributed way (Goryczka & Xiong, 2015, Theorem 5.1).
Fact 1 (Inﬁnite Divisibility of Discrete Laplace) .
A random variable X has a P´olya distribution with parameters r> 0 ,  2 [0 , 1] , denoted by P´olya ( r,  ) , if it has a p.m.f.
given by
[Equation]
Now, for any n 2 N , let {  + i ,   i } i 2 [ n ] be 2 n i.i.d samples 4 from P´olya (1 /n, e  1 /b ) , then the random variable
P ni =
(  + i    i ) is distributed as Lap Z ( b ) .
Armed with the above fact and the properties of discrete Laplace noise (see Fact 3 in Appendix K), we are able to obtain the following main theorem, which shows that the same regret as in the central model is achieved under the distributed model via SecAgg.
Theorem 1 (Pure-DP via SecAgg) .
Fix " > 0 and T 2 N .
For each batch b , let noise for the i -th user in the batch be ⌘ i =
+ i    i , where  + i ,   i i.i.d.
⇠ P´olya (1 /n, e  " /g ) , set n = l ( b ) , g = d " p n e , ⌧ =
d g " log(2 T ) e and m = ng + 2 ⌧
+ 1 .
Then, Algorithm 1 achieves ( " , 0) -DP in the distributed model.
Moreover, setting  ( b )=
O
[Equation]
enjoys
[Equation]
regret
[Equation]
Theorem 1 achieves optimal regret under pure DP.
Theorem 1 achieves the same regret bound as the one achieved in Sajed & Sheffet (2019) under the central trust model with continuous Laplace noise.
Moreover, it matches the lower bound obtained under pure DP in Shariff & Sheffet (2018), indicating the bound is indeed tight.
Note that, we achieve this rate under distributed trust model – a stronger notion of privacy protection than the central model – while using only discrete noise.
Communication bits.
Algorithm 1 needs to communicate O (log m ) bits per user to the secure protocol S , i.e., communicating bits scales logarithmically with the batch size.
In contrast, the number of communication bits required in existing distributed DP bandit algorithms that work with real-valued rewards (as we consider here) scale polynomially with the batch size (Chowdhury &
Zhou, 2022b; Garcelon et al., 2022).
Remark 1 (Pure DP via Secure Shufﬂing) .
It turns out that one can achieve same privacy and regret guarantees (orderwise) using a relaxed SecAgg protocol.
Building on this result, we also establish pure DP under shufﬂing while again maintaining the same regret bound as the central model (see Theorem 3 in Appendix D.2).
This improves the state-of-the-art result for MAB with shufﬂing (Tenenbaum et al., 2021) in terms of both privacy and regret.
A CHIEVING RDP IN THE D ISTRIBUTED M ODEL
A natural question to ask is whether one can get a better regret performance by sacriﬁcing a small amount of privacy.
We consider the notion of RDP (see Deﬁnition 2), which is a weaker notion of privacy than pure DP.
However, it avoids the possible catastrophic privacy failure in approximate
DP, and also provides a tighter privacy accounting for composition (Mironov, 2017).
To achieve RDP guarantee using discrete noise, we consider the Skellam distribution – which has recently been introduced in private federated learning (Agarwal et al., 2021).
A key challenge in the regret analysis of our bandit algorithm is to characterize the tail property of Skellam distribution.
This is different from federated learning, where characterizing the variance renders sufﬁcient.
In
Proposition 1, we prove that Skellam has sub-exponential tails, which not only is the key to our regret analysis, but could also be of independent interest.
Below is the formal deﬁnition of Skellam.
Deﬁnition 5 (Skellam Distribution) .
A random variable X has a Skellam distribution with mean µ and variance  2 , denoted by Sk ( µ,  2 ) , if it has a probability mass function given by
[Equation]
where I ⌫ ( · ) is the modiﬁed Bessel function of the ﬁrst kind.
To sample from Skellam distribution, one can rely on existing procedures for Poisson samples.
This is because if X = N 1  N 2 , where N 1 , N 2 i.i.d.
⇠ Poisson (  2 / 2) , then X is Sk (0 ,  2 ) distributed.
Moreover, due to this fact, Skellam is closed under summation, i.e., if X 1 ⇠ Sk ( µ 1 ,  21 ) and X 2 ⇠
Sk ( µ 2 ,  22 ) , then X 1 + X 2 ⇠
Sk ( µ 1 + µ 2 ,  21 +  22 ) .
Proposition 1 (Sub-exponential Tail of Skellam) .
Let X ⇠ Sk (0 ,  2 ) .
Then, X is (2  2 , p 22 ) -subexponential.
Hence, for any p 2 (0 , 1] , with probability at least 1  p ,
[Equation]

[Equation]
With the above result, we can establish the following privacy and regret guarantee of Algorithm 1.
Theorem 2 (RDP via SecAgg) .
Fix " > 0 , T 2 N and a scaling factor s  1 .
For each batch b , let noise for the i -th user be ⌘ i
⇠ Sk (0 , g 2 n " 2 ) , set n = l ( b ) , g
= d s " p n e , ⌧ =
d 2 g " p log(2 T ) + p 2 log(2 T ) e and m = ng + 2 ⌧
+ 1 .
Then, Algorithm 1 achieves ( ↵ , b " ( ↵ ))
-RDP in the distributed model for all ↵ = 2 , 3 , . . .
, with
[Equation]
Moreover, setting
[Equation]
O [Equation] enjoys the expected
[Equation]
Privacy-Regret-Communication Trade-off.
Observe that the scaling factor s allows us to achieve different trade-offs.
If s increases, both privacy and regret performances improve.
In fact, for a sufﬁciently large value of s , the third term in the regret bound becomes sufﬁciently small, and we obtain an improved regret bound compared to Theorem 1.
Moreover, the RDP privacy guarantee improves to b " ( ↵ ) ⇡ ↵" 2 2 , which is the standard RDP rate for Gaussian mechanism (Mironov, 2017).
However, a larger s leads to an increase of communicating bits per user, but only grows logarithmically, since
Algorithm 1 needs to communicate O (log m ) bits to the secure protocol S .
See Appendix B for further discussion on the communication cost.
RDP to Approximate DP.
To shed more insight on Theorem 2, we convert our RDP guarantee to approximate DP for a sufﬁciently large s .
It holds that under the setup of Theorem 2, for sufﬁciently large s , one can achieve ( O ( " ) ,  ) -DP with regret O
[Equation]
(via Lemma 10 in Appendix K).
Implication of this conversion is three-fold.
First , this regret bound is O
( p log T ) factor tighter than that achieved by Tenenbaum et al.
(2021) using a shufﬂe protocol.
Second , it yields a better regret performance compared to the bound achieved under ( " , 0) -DP in
Theorem 1 when the privacy budget  > 1 /T .
This observation is consistent with the fact that a weaker privacy guarantee typically warrants a better utility bound.
Third , this conversion via RDP also yields a gain of O ( p log(1 /  )) in the regret when dealing with privacy composition (e.g., when participating users across different batches are not unique ) as compared to Tenenbaum et al.
(2021) that only relies on approximate DP (see Appendix I for details).
This results from the fact that RDP provides a tighter composition compared to approximate DP.
Remark 2 (Achieving RDP with discrete Gaussian) .
One can also achieve RDP using discrete
Gaussian noise (Canonne et al., 2020).
Here, we work with Skellam noise since it is closed under summation and enjoys efﬁcient sampling procedure as opposed to discrete Gaussian (Agarwal et al., 2021).
Nevertheless, as a proof of ﬂexibility of our proposed framework, we show in Appendix F that Algorithm 1 with discrete Gaussian noise can guarantee RDP with a similar regret bound.
K EY T ECHNIQUES : O VERVIEW
Now, we provide an overview of the key techniques behind our privacy and regret guarantees.
We show that the results of Theorem 1 and 2 can be obtained via a clean generic analytical framework, which not only covers the analysis of distributed pure DP/RDP with SecAgg, but also offers a uniﬁed view of private MAB under central, local and distributed DP models.
As in many private learning algorithms, the key is to characterize the impact of added privacy noise on the utility.
In our case, this reduces to capturing the tail behavior of total noise n a ( b ) :
= R a ( b )
P l ( b ) i =1 r ia ( b ) added at each batch b for each active arm a .
The following lemma gives a generic regret bound of our algorithm under mild tail assumptions on n a ( b ) .
Lemma 1 (Generic regret) .
Let there exist constants  , h >
such that, with probability  1  p , | n a ( b ) |  N := O ⇣  p log( KT/p )+ h log( KT/p ) ⌘ for all b  1 , a 2 [ K ] .
Then, setting conﬁdence radius  ( b )=
O ⇣p log( KT/
[Equation]
/l ( b ) + N /l ( b ) ⌘ and p = 1 /T , Algorithm 1 enjoys expected regret
E [ Reg ( T )]
= O
[Equation]
X a
[Equation]
[ K ]:
[Equation]
a
[Equation]
log T
a + K
[Equation]
p log T + Kh log T ⌘ .
An acute reader may note that the bound N on the noise is the tail bound of sub-exponential distribution and it reduces to the bound for sub-Gaussian tail if h = 0 .
Our SecAgg protocol P with discrete
Laplace noise (as in Theorem 1) satisfy this bound with  = p 2 / " , h = 1 / " .
Similarly, our protocol with Skellam noise (as in Theorem 2) satisfy this bound with
= O (1 / " ) , h = 1 / ( s " ) .
Therefore, we can build on the above general result to directly obtain our regret bounds.
In the following, we present the high-level idea behind privacy and regret analysis in distributed DP model.
Privacy.
For distributed DP, by deﬁnition, the view of the server during the entire algorithm needs to be private.
Since each user only contributes once, 5 by parallel-composition of DP, it sufﬁces to ensure that each view b y a ( b ) (line 11 in Algorithm 1) is private.
To this end, under SecAgg, the distribution of b y a ( b ) can be simulated via (
P i y i ) mod m , which further reduces to (
P i b x i + ⌘ i ) mod m by the distributive property of modular sum.
Now, consider a mechanism M that accepts an input dataset { b x i } i and outputs
P i ( b x i + ⌘ i ) .
By post-processing, it sufﬁces to show that M satisﬁes pure
DP or RDP.
To this end, the variance  2 tot of the total noise
P i ⌘ i needs to scale with the sensitivity of
P i b x i .
Thus, each user within a batch only needs to add a proper noise with variance of  2 tot /n .
Finally, by the particular distribution properties of the noise, one can show that M is pure DP or
RDP, and hence, obtain the privacy guarantees.
Regret.
Thanks to Lemma 1, we only need to focus on the tail of n a ( b ) .
To this end, ﬁx any batch b and arm a .
We have b y
[Equation]
b y a ( b ) , x i = r ia ( b ) , n = l ( b ) for P in Algorithm 1 and we need to establish that with probability at least 1  p , for some  and h ,
[Equation]
To get the bound, inspired by Balle et al.
; Cheu & Yan (2021), we divide the LHS into
Term (i) = |A
( b y )
P i b x
i /g | and Term (ii) =
|
P i b x
i /g
P i x i | , where Term (i) captures the error due to privacy noise and modular operation, while Term (ii) captures the error due to random rounding.
In particular, Term (ii) can be easily bounded via sub-Gaussian tail since the noise is bounded.
Term (i) needs care for the possible underﬂow due to modular operation by considering two different cases (see line 28-30 in Algorithm 1).
In both cases, one can show that Term (i) is upper bounded by ⌧ /g with high probability, where ⌧ is the tail bound on the total privacy noise
P i ⌘ i .
Thus, depending on particular privacy noise and parameter choices, one can ﬁnd  and h such that equation 1 holds, and hence, obtain the corresponding regret bound by Lemma 1.
Remark 3.
As a by-product of our generic analysis technique, Algorithm 1 and privacy protocol P along with Lemma 1 provide a new and structured way to design and analyze private MAB algorithms under central and local models with discrete private noise (see Appendix H for details).
This enables us to reap the beneﬁts of working with discrete noise (e.g., ﬁnite-computer representations, bit communications) in all three trust models (central, local and distributed).
[Figure]
T i m e - a v e r a g
e R e g r e t
DP-SEDist-DP-SE Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) (e) " = 5 , K =0 .0 .0 .0 .0 .1 .
Round  100 .0 .0 .
T i m e - a v e r a g
e R e g r e t
DP-SEDist-DP-SE Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100)
(f) " = 10 , K =
S IMULATION R ESULTS
We empirically evaluate the regret performance of our successive elimination scheme with SecAgg protocol (Algorithm 1) under distributed trust model, which we abbreviate as Dist-DP-SE and DistRDP-SE when the randomizer R is instantiated with P´olya noise (for pure DP) and Skellam noise (for RDP), respectively.
We compare them with the DP-SE algorithm of Sajed & Sheffet (2019) that achieves optimal regret under pure DP in the central model, but works only with continuous Laplace noise.
We ﬁx conﬁdence level p = 0 .
1 and study comparative performances under varying privacy levels ( " < 1 for synthetic data, "  1 for real data).
We plot time-average regret Reg ( T ) /T
in
Figure 1 by averaging results over 20 randomly generated bandit instances.
Bandit instances.
In the top panel , similar to Vaswani et al.
(2020), we consider easy and hard
MAB instances
Observations.
We observe that as T becomes large, the regret performance of Dist-DP-SE matches the regret of DP-SE.
The slight gap in small T regime is the cost that we pay to achieve distributed privacy using discrete noise without access to a trusted server (for higher " value, this gap is even smaller).
In addition, we ﬁnd that a relatively small scaling factor ( s = 10 ) provides a considerable gain in regret under RDP compared to pure DP, especially when " is small (i.e., when the cost of privacy is not dominated by the non-private part of regret).
The experimental ﬁndings are consistent with our theoretical results.
Here, we note that our simulations are proof-of-concept only and we did not tune any hyperparameters.
More details and additional plots are given in Appendix J.
Concluding remarks.
We show that MAB under distributed trust model can achieve pure DP while maintaining the same regret under central model.
In addition, RDP is also achieved in MAB under distributed trust model for the ﬁrst time.
Both results are obtained via a uniﬁed algorithm design and performance analysis.
More importantly, our work also opens the door to a promising and interesting research direction – private online learning with distributed DP guarantees, including contextual bandits and reinforcement learning.