Distributed Differential Privacy in Multi-Armed Bandits
Abstract
We consider the standard K -armed bandit problem under a distributed trust model of differential privacy (DP), which enables to guarantee privacy without a trustworthy server.
Under this trust model, previous work largely focus on achieving privacy using a shuffle protocol, where a batch of users data are randomly permuted before sending to a central server.
This protocol achieves ( ε, δ ) or approximate-DP guarantee by sacrificing an additive O (cid:18) K log T √ log(1 /δ )
ε (cid:19) factor in T -step cumulative regret.
In contrast, the optimal privacy cost to achieve a stronger ( ε, 0 ) or pure-DP guarantee under the widely used central trust model is only Θ (cid:16) K log T ε (cid:17) , where, however, a trusted server is required.
In this work, we aim to obtain a pure-DP guarantee under distributed trust model while sacrificing no more regret than that under central trust model.
We achieve this by designing a generic bandit algorithm based on successive arm elimination, where privacy is guaranteed by corrupting rewards with an equivalent discrete Laplace noise ensured by a secure computation protocol.
We numerically simulate regret performance of our algorithm, which corroborates our theoretical findings.
Introduction
The multi-armed bandit (MAB) [1] problem provides a simple but powerful framework for sequential decision-making under uncertainty with bandit feedback, which has attracted a wide range of practical applications such as online advertising [2], product recommendations [3], clinical trials [4], to name a few.
Along with its broad applicability, however, there is an increasing concern of privacy risk in MAB due to its intrinsic dependence on users’ feedback, which could leak users’ sensitive information [5].
To alleviate the above concern, the notion of differential privacy , introduced by Dwork et al.
[6] in the field of computer science theory, has recently been adopted to design privacy-preserving bandit algorithms (see, e.g., [7–9]).
Differential privacy (DP) provides a principled way to mathematically prove privacy guarantees against adversaries with arbitrary auxiliary information about users.
To achieve this, a differentially private bandit algorithm typically relies on a well-tuned random noise to obscure each user’s contribution to the output, depending on privacy levels ε, δ – smaller values lead to stronger protection but also suffer worse utility (i.e., regret).
For example, the central server of a recommendation system can use random noise to perturb its statistics on each item after receiving feedback (i.e., clicks/ratings) from users.
This is often termed as central model [10], since the central server has the trust of its users and hence has a direct access to their raw data.
Under this model, an optimal private MAB algorithm with a pure DP guarantee (i.e., when δ = 0 ) is proposed in [11], which only incurs an additive O (cid:16) K log T ε (cid:17) term in the cumulative regret compared to the standard setting when privacy is not sought after [12].
However, this high trust model is not always feasible in practice since users may not be willing to share their raw data directly to the server.
This motivates
[Table]
to employ a local model [13] of trust, where DP is achieved without a trusted server as each user perturbs her data prior to sharing with the server.
This ensures a stronger privacy protection [Table] but leads to a high cost in utility due to large aggregated noise from all users.
As shown in [14], under the local model [Table] private MAB algorithms have to incur a multiplicative 1 /ε 2 factor in the regret rather than the additive one in the central model.
In attempts to recover the same utility of central model without a trustworthy server like the local model [Table] an intermediate DP trust model called distributed model has gained an increasing interest, especially in the context of (federated) supervised learning [15–19].
Under this model, each user first perturbs her data via a local randomizer
[Table]
and then sends the randomized data to a secure computation function.
This secure function can be leveraged to guarantee privacy through aggregated noise from distributed users.
There are two popular secure computation functions: secure aggregation (SecAgg) [20] and secure shuffling [21].
The former often relies on cryptographic primitives to securely aggregate users’ data so that the server only learns the aggregated result, while the latter securely shuffle users’ messages to hide their source.
To the best of our knowledge, distributed DP model is far less studied in online learning as compared to supervised learning, with only known results for standard K -armed bandits in [22] where only secure shuffling is adopted.
Despite being pioneer work, the results obtained in this paper have several limitations: (i)
The privacy guarantee is obtained only for approximate DP ( δ > 0 ) – a stronger pure DP ( δ = 0 ) guarantee is not {+achieved; (ii) The cost of privacy is a multiplicative (cid:
+}
) log(1 /δ ) factor away from that of central model, leading to a higher regret bound; (iii)
The secure protocol works only for binary rewards (or communication intensive for real rewards).
All of these lead to the following primary question:
Is there a communication-efficient MAB algorithm that satisfies pure DP in the distributed model while attaining the same regret bound as in the central model?
Our contributions.
We answer this in the affirmative (see Table 1) by overcoming several key challenges that arise in the distributed DP model for bandits.
First, although secure aggregation protocol offers a benefit in communication cost, it works only in the integer domain due to an inherent modular operation [20].
This immediately requires substantial changes to existing private bandit algorithms, including data quantization, distributed discrete privacy noise and modular summation arithmetic.
Second, compared to supervised learning where typically a bound on the noise variance is sufficient to analyse utility, regret analysis of private bandits require a tight tail bound.
This gets challenging due to aforementioned requirements and communication constraints in the distributed model.
We take a systematic approach to address these challenges, which is summarized below:
We propose a private bandit algorithm using a batch-variant of the successive arm elimination technique as a building block.
We ensure distributed DP via a private protocol P =
( R , S , A ) tailored to discrete noise and modular operation (see Algorithm 1).
It consists of a local randomizer R at each user, a generic secure computation function S , and an analyzer A at the server.
This template protocol enables us to achieve different privacy guarantees by tuning the noise
in R .
To achieve pure DP guarantee, we instantiate R at each user with an appropriate Pólya random noise so that the total noise seen by the server is a discrete Laplace.
Using tail properties of discrete
Laplace, we show that the cumulative regret of our algorithm matches the one in the central model, achieving the minimax rate under pure DP (see Theorem 1).
Moreover, the communication bits peruser scale only logarithmicaly with the number of participating users in each batch.
We numerically evaluate the regret performance of our algorithm, which corroborate our theoretical results.
Preliminaries
In this section, we formally introduce the distributed differential privacy model in bandits.
Before that we recall the learning paradigm in multi-armed bandits and basic differential privacy definitions.
Learning Model and Regret in MAB.
At each time slot t ∈ [ T ] := { 1 , . .
, T } , the agent (e.g., recommender system) selects an arm a ∈ [ K ] (e.g., an advertisement) and obtains an i.i.d reward r t from user t (e.g., a rating indicating how much she likes it), which is sampled from a distribution over [0 , 1] with mean given by µ a .
Let a ∗ := argmax a ∈ [ K ] µ a be the arm with the highest mean and denote µ ∗ :
= µ a ∗ for simplicity.
Let ∆ a := µ ∗ − µ a be the gap of the expected reward between the optimal arm a ∗ and any other arm a .
Further, let N a ( t ) be the total number of times that arm a has been played in the first t rounds.
The goal of the agent is to maximize its total reward, or equivalently to minimize the cumulative expected pseudo-regret, defined as
[Equation]
Differential Privacy.
Let D =
[0 , 1] be the data universe, and n ∈ N be the number of unique users. we say D,
D ′ ∈ D n are neighboring datasets if they only differ in one user’s reward D i for some i ∈ [ n ] .
With this, we have the following standard definition of differential privacy [6].
Definition 1 (Differential Privacy) .
For ε, δ > 0 , a randomized mechanism M satisfies ( ε, δ ) -DP if for all neighboring datasets D, D ′ and all events E in the range of M , we have
[Equation]
The special case of ( ε, 0) -DP is often referred to as pure differential privacy , whereas, for δ > 0 , ( ε, δ ) -DP is referred to as approximate differential privacy .
Distributed Differential Privacy.
A distributed bandit learning protocol P =
( R , S , A ) consists of three parts: (i) a (local) randomizer
R at each user’s side, (ii) an intermediate secure protocol S , and (iii) an analyzer
A at the central server.
Each user i first locally apply the randomizer R on its raw data (i.e., reward) D i , and sends the randomized data to a secure computation protocol S (e.g., secure aggregation or shuffling).
This intermediate secure protocol S takes a batch of users’ randomized data and generates inputs to the central server, which utilizes an analyzer A to compute the output (e.g., action) using received messages from S .
The secure computation protocol S has two main variations: secure shuffling and secure aggregation .
Both of them essentially work with a batch of users’ randomized data and guarantee that the central server cannot infer any individual’s data while the total noise in the inputs to the analyzer provides a high privacy level.
To adapt both into our MAB protocol, it is natural to divide participating users into batches.
For each batch b ∈ [ B ] with n b users, the outputs of S is given by S ◦ R n b ( D ) :=
S ( R ( D 1 ) , . . .
, R ( D n b )) .
The goal is to guarantee that the the view of all B batches’ outputs satisfy
DP.
To this end, we define a (composite) mechanism
[Equation]
where each individual mechanism S ◦ R n b operates on n b users’ rewards, i.e., on a dataset from D n b .
With this notation, we have the following definition of distributed differential privacy.
Definition 2 (Distributed DP) .
A protocol P =
( R , S , A ) is said to satisfy DP in the distributed model if the mechanism
M P satisfies Definition 1.
In the central DP model, the privacy burden lies with a central server (in particular, analyzer A ), which needs to inject necessary random noise to achieve privacy.
On the other hand, in the local
DP model, each user’s data is privatized by local randomizer R .
In contrast, in the distributed DP model, privacy without a trusted central server is achieved by ensuring that the inputs to the analyzer
A already satisfy differential privacy.
Specifically, by properly designing the intermediate protocol S and the noise level in the randomizer
R , one can ensure that the final added noise in the aggregated data over a batch of users matches the noise that would have otherwise been added in the central model by the trusted server.
Through this, distributed DP model provides the possibility to achieve the same level of utility as the central model without a trustworthy central server.
A Generic Algorithm for Private Bandits
In this section, we propose a generic algorithmic framework for multi-armed bandits under the distributed privacy model.
Batch-Based Successive Elimination Algorithm.
Our generic algorithm (Algorithm 1) builds upon the classic idea of successive arm elimination [23] with the additional incorporation of batches and a black-box protocol P =
( R , S , A ) to achieve distributed differential privacy.
More specifically, it divides the time horizon T into batches of exponentially increasing size and eliminates sub-optimal arms successively.
To this end, for each active arm a at batch b , it first prescribes arm a to a batch of l ( b ) new users.
After pulling the prescribed action a , each user applies the local randomizer R to her reward and sends the randomized reward to the intermediary function S , which runs a secure protocol (e.g., secure aggregation or secure shuffling) over the total l ( b ) number of randomized rewards.
Then, upon receiving the outputs of S , the server applies the analyzer A to compute the the sum of rewards for batch b when pulling arm a , which gives the new mean estimate (cid:98) µ a ( b ) of arm a after being divided by the total pulls l ( b ) .
Then, upper and lower confidence bounds, UCB a ( b ) and LCB a ( b ) , respectively, are computed around the mean estimate (cid:98) µ a ( b ) with a properly chosen confidence width β ( b ) .
Finally, after the iteration over all active arms in batch b (denoted by the set Φ( b ) ), it adopts the standard arm elimination criterion to remove all obviously sub-optimal arms, i.e., it removes an arm a from Φ( b ) if UCB a ( b ) falls below LCB a ′ ( b ) of any other arm
a ′ ∈
Φ( b ) .
It now remains to design a distributed DP protocol P , which will be explored at length in the next section.
Distributed DP Protocol via Discrete Privacy Noise.
In this section, inspired by [24, 25], we provide a general template protocol P for the distributed DP model, which rely only on discrete privacy noise.
The motivation behind using discrete noise is three-fold: (i) Practical secure aggregation (SecAgg) functions work only on the integer domain [20]; (ii) A real-value noise is often difficult to encode on finite computers in practice [26, 17] and a naive use of finite precision approximation may lead to a possible failure of privacy protection [27]; (iii) Discrete noise enables communication via bits rather than real numbers, hence reducing communication overheads.
The detail of our template protocol P =
( R , S , A ) for distributed DP model is as follows.
The local randomizer R receives user’s real-valued reward and encodes it as an integer via a fixed-point encoding with precision g > 0 and randomized rounding.
Then, it generates a discrete noise, which depends on the specific privacy-regret trade-off requirement (to be discussed later under specific mechanisms).
Next, it adds the random noise with encoded reward, modulo clips the sum and sends the final integer as input to S .
We leave S as a black-box function that implements secure aggregation.
Engineering implementations of this function is beyond the scope of this paper.
Instead, a high-level idea behind this technique is to ensure that after receiving messages from S , the server cannot distinguish each individual’s message.
Finally, the job of the analyzer A in our template protocol is to calculate the sum of rewards within a batch as accurately as possible.
To this end, it directly corrects for possible underflow due to modular operation and bias due to encoding by g .
To sum it up, the end goal of our protocol P is to ensure that it provides the required privacy protection while guaranteeing an output z ≈ (cid:80) ni
=1 x i with high probability, which is the key to our privacy and regret analysis.
Achieving Pure DP in the Distributed Model
In this section, we show that Algorithm 1 with a specific instantiation of the template protocol
P is able to achieve pure-DP in the distributed DP model via secure aggregation.
As mentioned before, we will treat SecAgg as a black-box function, which implements the following procedure: given n users and their randomized messages y i ∈
Z m (i.e., integer in { 0 , 1 , . . .
, m − 1 } ) obtained via R , the SecAgg function S faithfully computes the modular sum of the n messages, that is, (cid:98) y
=
( (cid:80) ni =1 y i )
mod m (i.e., it is perfectly correct), while revealing no further information (e.g., individual message) to a potential attacker (i.e., it is perfectly secure).
Now, to guarantee privacy in the distributed model, we need to carefully determine the amount of (discrete) noise in R so that the total noise in a batch provides ( ε, 0) -DP.
One natural choice is the discrete Laplace noise.
Definition 3 (Discrete Laplace Distribution) .
Let b > 0 .
A random variable X has a discrete Laplace distribution with scale parameter b , denoted by Lap Z ( b ) , if it has a probability mass function given by
[Equation]
A key property of discrete Laplace that we will use is its infinite divisibility , which allows us to simulate it in a distributed way [28, Theorem 5.1].
1: Parameters: # arms K , Time horizon T , privacy level ε > 0 , Confidence radii { β ( b ) } b
≥2: Initialize: Batch count b =1 , Active arm set Φ( b )= { 1 , . . .
, K } , Estimate (cid:98) µ a (1)=0 , ∀ a ∈ [ K ] 3: for batch b =1 , 2 , . . .
do 4:
Set batch size l ( b )=2 b 5: for each active arm a ∈ Φ( b ) do 6: for each new user i from 1 to l ( b ) do 7:
Pull arm a and generate reward r ia ( b ) 8:
Send randomized data y ia ( b )
=
R ( r ia ( b )) to S // randomizer 9:
If total number of pulls reaches T , exit 10: end for 11:
Send messages (cid:98)
y a ( b )=
S ( { y ia ( b ) } 1 ≤ i ≤ l ( b ) ) to A // secure computation 12:
Compute the sum of rewards R a ( b )
=
A ( (cid:98) y a ( b )) // analyzer 13:
Compute mean estimate (cid:98) µ a ( b )=
R a ( b ) /l
( b ) 14:
Compute confidence bounds UCB a ( b )= (cid:98) µ a ( b )+ β ( b ) and LCB a ( b )
= (cid:98) µ a ( b ) − β ( b ) 15: end for 16:
Update active set of arms: Φ( b +1)= (cid:8) a ∈ Φ( b ) :
UCB a ( b ) ≥ max a ′ ∈
Φ( b ) LCB a ′ ( b ) (cid:9) 17: end for 18: Subroutine: Local Randomizer R ( Input: x i ∈ [0 , 1] , Output: y i ) 19: Require: precision g ∈ N , modulo m ∈ N , batch size n ∈ N , privacy level ε 20:
Encode x i as (cid:98) x i =
⌊ x i g black- else
Fact 1 (Infinite Divisibility of Discrete Laplace) .
A random variable X has a Pólya distribution with parameters r> 0 , β ∈ [0 , 1] , denoted by Pólya ( r, β ) , if it has a probability mass function given by[-
-]
[Equation]
Now, for any n ∈ N , let { γ + i , γ − i } i ∈
[ n ] be 2 n i.i.d samples from Pólya (1 /n, e − 1 /b ) , then the random variable (cid:80) ni
=1 ( γ + i − γ − i ) is distributed as Lap Z ( b ) .
Further, to analyze privacy and regret, we will rely on the following fact on discrete Laplace [26].
Fact 2 (Discrete Laplace Mechanism) .
Let ∆ , ε > 0 .
Let q : D n → Z satisfy | q ( D ) − q ( D ′ ) | ≤ ∆ for all D, D ′ differing on a single user’s data.
Define a mechansim M :
[Equation] Z by M ( D ) =
q ( D ) + Y , where Y ∼ Lap Z (∆ /ε ) .
Then, M satisfies ( ε, 0) -DP.
Moreover, for all m ∈ N ,
[Equation]
Armed with these facts, we are able to obtain the following main theorem, which shows that the same regret as in the central model is achieved under the distributed model via SecAgg.
Theorem 1 (Pure-DP via SecAgg) .
Fix any ε > 0 .
Let P =
( R , S , A ) be a protocol such that the noise in R is given by η i = γ
+ i − γ −
i , where γ + i , γ − i ∼ i.i.d.
Pólya (1 /n, e − ε/g ) .
For each batch b ≥ 1 , choose n = l ( b ) , g = ⌈ ε √ n ⌉ , τ = ⌈ gε log(2 /p )
⌉ , p = 1 /T and m =
ng +2 τ +1 .
Then, Algorithminstantiated with protocol P and confidence radius β ( b ) =
O (cid:16)(cid:113) log( | Φ( b ) | b 2 /p ) 2 l
[Equation]
b )
[Equation]
[Figure]
achieves ( ε, 0) -DP in the distributed model.
Moreover, it enjoys the expected regret
[Equation]
Theorem 1 achieves optimal regret under pure DP.
Theorem 1 achieves the same regret bound as the one achieved in Sajed and Sheffet [11] under the central trust model with continuous Laplace noise.
Moreover, it also matches the lower bound obtained under pure DP [9], indicating the bound is indeed tight Note that, we achieve this rate under distributed trust model – a stronger notion of privacy protection than the central model – while using only discrete privacy noise.
Communication bits.
Algorithm 1 needs to communicate O (log m ) bits per user to the secure protocol S , i.e., communicating bits scales logarithmically with the batch size.
In contrast, the number of communication bits required in existing distributed DP bandit algorithms that work with real-valued rewards (as we consider here) scale polynomially with the batch size [29, 30].
MAB with distributed RDP guarantee.
In addition to pure DP, it is of natural interest to offer a slightly weaker privacy guarantee (but, still stronger than approximate DP) in the hope of gaining improvement in utility.
One such notion of privacy is Rényi differential privacy (RDP) [31].
To this end, we obtain RDP in the distributed model by instantiating R at each user with a Skellam random noise.
Proving a novel tail-bound for Skellam distribution, we show that a tighter regret bound compared to pure DP can be achieved (see Theorem 2)
Simulation Results
In this section, we empirically evaluate the regret performance of our successive elimination scheme with SecAgg protocol (Algorithm 1) under distributed trust model, which we abbreviate as Dist-DPSE and Dist-RDP-SE when the randomizer R is instantiated with Pólya noise (achieves pure DP) and Skellam noise (achieves RDP), respectively.
We compare them with the DP-SE algorithm of [11] that works only with continuous Laplace noise and achieves optimal regret under pure DP in the central model.
We fix confidence level p = 0 .
1 and study comparative performances under varying privacy levels ε < 1 .
Similar to [32], we consider easy and hard MAB instances: in the former, arm means are sampled uniformly in [0 . 25 , 0 .
75] , while in the latter, those are sampled in [0 . 45 , 0 .
We consider real rewards – sampled from Gaussian distribution with aforementioned means and projected to [0 , 1] .
We plot time-average regret Reg ( T ) /T
in Figure 1 by averaging results overrandomly generated bandit instances.
We observe that as T becomes large, the regret performance of
Dist-DP-SE matches the regret of DP-SE.
The slight gap in small T regime is the cost that we pay to achieve distributed privacy using discrete noise without access to a trusted server (for higher ε value, this gap is even smaller).
In addition, we find that a relatively small scaling factor ( s = 10 ) provides a considerable gain in regret under RDP compared to pure DP, especially when ε is small (i.e., when the cost of privacy is not dominated by the non-private part of regret).
The experimental findings are consistent with our theoretical results.
Here, we note that our simulations are proof-of-concept only and we did not tune any hyperparameters.
More details and additional plots are given in Appendix E.
Conclusion
We show that MAB under distributed trust model can achieve pure DP while maintaining the same regret under central model.
In addition, RDP is also achieved in MAB udner distributed trust model for the first time.
Both results are obtained via a unified algorithm design and performance analysis.
More importantly, our work also opens the door to a promising research direction – private online learning with distributed DP guarantees, including contextual bandits and reinforcement learning.
A A General Regret Bound of Algorithm
In this section, we first present a generic regret bound for private MAB, see Lemma 1 below.
Recall that n a ( b ) :
= R a ( b ) − (cid:80) l ( b ) i =1 r ia ( b ) is the total noise injected in the sum of rewards for arm a during batch b .
We consider the following tail property on n a ( b ) .
Assumption 1 (Concentration of Private Noise) .
Fix any p ∈ (0 , 1] , a ∈ [ K ] , b ≥ 1 , there exist non-negative constants σ, h (possibly depending on b ) such that, with probability at least 1 − 2 p ,
[Equation]
We remark that this assumption naturally holds for a single ( σ 2 , h ) -sub-exponential noise and a single σ 2 -sub-Gaussian noise where h
=
(cf.
Lemma 4 and Lemma 3 in Appendix D) with constants adjustment.
Lemma 1.
Let Assumption 1 hold and choose confidence radius
[Equation]
where | Φ( b ) | is the number of active arms in batch b .
Then, for any p ∈ (0 , 1] , with probability at least 1 − 3 p , the regret of Algorithm 1 satisfies
[Equation]
Taking p = 1 /T and assuming T ≥ K , yields the expected regret
[Equation]
Proof.
Let E b be the event that for all active arms | (cid:98) µ a ( b ) − µ a | ≤ β ( b ) and E = ∪ b ≥ 1 E b .
Then, we first show that with the choice of β ( b ) given by (1), we have P [
E ] ≥ 1 − 3 p for any p ∈ (0 , 1] under Assumption 1.
To see this, we note that
[Equation]
By Hoeffeding’s inequality (cf.
Lemma 5), we have for any p ∈ (0 , 1) , with probability at least 1 − p ,
[Equation]
Then, by the concentration of noise n a ( b ) in Assumption 1 and triangle inequality, we obtain for a given arm a and batch b , with probability at least 1 − 3 p
[Equation]
Thus, by the choice of β ( b ) and a union bound, we have P [ E ] ≥ 1 − 3 p .
In the following, we condition on the good event E .
We first show that the optimal arm a ∗ will always be active.
We show this by contradiction.
Suppose at the end of some batch b , a ∗ will be eliminated, i.e., UCB a ∗ ( b ) <
LCB a ′ ( b ) for some a ′ .
This implies that under good event E
[Equation]
which contradicts the fact that a ∗ is the optimal arm.
Then, we show that at the end of batch b , all arms such that ∆ a > 4 β ( b ) will be eliminated.
To show this, we have that under good event E
[Equation]
which implies that arm a will be eliminated by the rule.
Thus, for each sub-optimal arm a , let ˜ b a be the last batch that arm a is not eliminated.
By the above result, we have
[Equation]
Hence, we have for some absolute constants c 1 , c 2 , c 3 ,
[Equation]
Since the batch size doubles [Equation] we have N a ( T ) ≤ 4 l (˜ b a ) for each sub-optimal arm a .
Therefore,
Reg
[Equation]
[Equation]
[Equation]
l
[Equation]
˜ b a
[Equation]
∆ a .
Moreover, choose p = 1 /T and assume T ≥ K , we have that the expected regret satisfies
[Equation]
Remark 1.
In stead of a doubling batch schedule, one can also set l ( b ) =
η b for some absolute constant η > 1 while attaining the same order of regret bound.
B Appendix for Pure DP in Section
In this section, we provide proofs for Theorem 1, which show that pure DP can be achieved under the distributed model via SecAgg.
The result builds on the generic regret bound in Lemma 1.
B.1 Proof of Theorem
Proof.
Privacy: We need to show that the server’s view at each batch has already satisfies ( ε, 0) -DP, which combined with the fact of unique users and parallel composition, yields that Algorithmsatisfies ( ε, 0) -DP in the distributed model.
To this end, in the following, we fix a batch b and arm a , and hence x i = r ia ( b ) and n = l ( b ) .
Note that the server’s view for each batch is given by
[Equation]
where (a) holds by SecAgg function; (b) holds by the distributive property: ( a + b ) mod c = ( a mod c + b mod c ) mod c for any a, b, c ∈ Z .
Thus, the view of the server can be simulated as a post-processing of a mechanism H that accepts an input dataset { (cid:98) x
i } i and outputs (cid:80) i (cid:98) x
i + (cid:80) i η i .
Hence, it suffices to show that
H is ( ε, 0) -DP by post-processing of DP.
To this end, we note that the sensitivity of (cid:80) i (cid:98) x i is g , which, by Fact 2, implies that (cid:80) i
η i needs to be distributed as Lap Z ( g/ε ) in order to guarantee ε -DP.
Finally, by Fact 1, it suffices to generate η i = γ
+ i − γ −
i , where γ + i and γ − i are i.i.d samples from Pólya (1 /n, e − ε/g ) .
Regret: Thanks to the generic regret bound in Lemma 1, we only need to verify Assumption 1.
To this end, fix any batch b and arm a , we have (cid:98)
y
= (cid:98) y a ( b ) , x i =
r i a ( b ) and n = l ( b ) .
Then, in the following we will show that with probability at least 1 − 2 p
[Equation]
which implies that Assumption 1 holds with σ
= O (1 /ε ) and h =
O (1 /ε ) .
Inspired by [25, 24], we first divide the LHS of (3) as follows.
[Equation]
where Term (i) captures the error due to private noise and modular operation, and Term (ii) captures the error due to random rounding.
To start with, we will bound Term (ii).
More specifically, we will show that for any p ∈ (0 , 1] , with probability at least 1 − p ,
[Equation]
Let ¯ x i :=
⌊ x i · g ⌋ , then (cid:98) x i =
¯ x i + Ber
( x i g − ¯ x i ) =
x i g + ¯ x
i + Ber
( x i g − ¯ x i ) − x i g =
x i g + ι i , where ι i :=
¯ x i + Ber ( x i g − ¯ x i ) − x i g .
We have E [ ι i ] =
and ι i ∈ [ − 1 , 1] .
Hence, ι i is 1 -sub-Gaussian and as a result, 1 g (cid:80) i (cid:98) x i − (cid:80) i x i = 1 g (cid:80) i ι
i is n/g 2 -sub-Gaussian.
Therefore, by the concentration of sub-Gaussian (cf.
Lemma 3), we have
[Equation]
Hence, by the choice of g = ⌈ ε √ n ⌉ , we establish (4).
Now, we turn to bound Term (i).
Recall the choice of parameters: g = ⌈ ε √ n ⌉ , τ = ⌈ gε log(2 /p )
⌉ , and m = ng
+ 2 τ + 1 .
We would like to show that
[Equation]
which implies that for any p ∈ (0 , 1] , with probability at least 1 − p
[Equation]
To show (5), the key is to bound the error due to private noise and handle the possible underflow carefully.
First, we know that the total private noise (cid:80) i η i is distributed as Lap Z ( g/ε ) .
Hence, by the concentration of discrete Laplace (cf.
Fact 2), we have
P (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:88)
[Equation]
≤ p.
Let E noise denote the event that (cid:80) i (cid:98)
x
i + η i ∈ [ (cid:80) i (cid:98) x
i − τ, (cid:80) i (cid:98) x
i + τ ] , then by the above inequality, we have P [
E noise ] ≥ 1 − p .
In the following, we condition on the event of E noise to analyze the output
A ( (cid:98) y ) .
As already shown in (2), the input (cid:98) y
=
( (cid:80) i (cid:98) x i + η i ) mod m is already an integer and hence y = (cid:98) y .
We will consider two cases of y .
Case 1: y > ng + τ .
We argue that this happens only when (cid:80) i (cid:98) x
i + η i ∈ [ − τ, 0) , i.e., underflow.
This is because for all i ∈ [ n ] , (cid:98) x i ∈
[0 , g ] , m = ng + 2 τ +
and the total privacy noise is at most τ under E noise .
Therefore,
[Equation]
That is, y − m [Equation]
[Equation]
i + τ ] with high probability.
In other words, we have show that when y > ng + τ , A ( (cid:98) y ) =
y − mg satisfies (5).
Case 2: y ≤ ng + τ .
Here, we have noisy sum (cid:80) i (cid:98) x
i + η i ∈ [0 , ng + τ ] .
Hence, y = (cid:80) i (cid:98) x i + η i since m = ng
+ 2 τ + 1 , which implies that A ( (cid:98) y )
=
yg satisfies (5).
Hence, we have shown that the output of the analyzer under both cases satisfies (5), which implies (6).
Combined with the bound in (4), yields the bound in (3).
Finally, plugging in σ = O (1 /ε ) , h =
O (1 /ε ) into the generic regret bound in Lemma 1, yields the required regret bound and completes the proof.
C Appendix for RDP
In the section, we will present the details of achieving Rényi differential privacy (RDP) [31] in the distributed DP model.
Definition 4 (Rényi Differential Privacy) .
For α > 1 , a randomized mechanism M satisfies ( α, ε ( α )) -
RDP if for all neighboring datasets D, D ′ , we have D α ( M ( D ) , M ( D ′ ))
≤ ε ( α ) , where D α ( P, Q ) is the Rényi divergence (of order α ) of the distribution P from the distribution Q , and is given by
[Equation]
To achieve RDP guarantee using discrete noise, instead of discrete Laplace distribution in the previous section, we consider the Skellam distribution – which has recently been applied in federated learning [16].
In the multi-armed bandit setting, a key challenge in the regret analysis is to characterize the tail property of Skellam distribution.
This is different from [16], where characterizing the variance of Skellam distribution is sufficient.
In Proposition 1, we prove that Skellam has sub-exponential tails, which not only is the key to our regret analysis, but also could be of independent interest.
Definition 5 (Skellam Distribution) .
A random variable X has a Skellam distribution with mean µ and variance σ 2 , denoted by Sk ( µ, σ 2 ) , if it has a probability mass function given by
[Equation]
where I ν ( · ) is the modified Bessel function of the first kind.
To sample from Skellam distribution, one can rely on existing procedures for Poisson samples.
This is because if X = N 1 − N 2 , where N 1 , N 2 ∼ i.i.d.
Poisson ( σ 2 / 2) , then X is Sk (0 , σ 2 ) distributed.
Moreover, due to this fact, Skellam is closed under summation, i.e., if X 1 ∼ Sk ( µ 1 , σ 21 ) and
X 2 ∼ Sk ( µ 2 , σ 22 ) , then X 1 + X 2 ∼ Sk ( µ 1 + µ 2 , σ 21 + σ 22 ) .
Proposition 1 (Sub-exponential Tail of Skellam) .
Let X ∼ Sk
(0 , σ 2 ) .
Then, X is (2 σ 2 , √ 22 ) -subexponential.
Hence, for any p ∈ (0 , 1] , with probability at least 1 − p ,
[Equation]
With the above result, we can establish the following privacy and regret guarantee of Algorithm 1.
Theorem 2 (RDP via SecAgg) .
Fix any ε > 0 .
Let P =
( R , S , A ) be a protocol such that the noise in R is given by η i ∼ Sk (0 , g 2 nε 2 ) .
Fix a scaling factor s ≥ 1 .
For each batch b ≥ 1 , choose n = l ( b ) , g = ⌈ sε √ n ⌉ , τ = ⌈ 2 gε (cid:112) log(2 /p )
+ √ 2 log(2 /p )
⌉ , p = 1 /T and m = ng + 2 τ + 1 .
Then, Algorithm 1 instantiated with protocol P and confidence radius β ( b ) =
O (cid:16)(cid:113) log( | Φ( b ) | b 2 /p )
2 l ( b ) + (1+1 /s )log( | Φ( b ) | b 2 /p )
εl ( b ) (cid:17) , achieves ( α, (cid:98) ε
( α ))
-RDP in the distributed model for all α = 2 , 3 , . .
, with (cid:98)
[Equation]
Moreover, it enjoys the regret bound
[Equation]
Privacy-Regret-Communication Trade-off.
Observe that the scaling factor s allows us to achieve different trade-offs.
If s increases, both privacy and regret performances improve.
In fact, for a sufficiently large value of s , the third term in the regret bound becomes sufficiently small, and we obtain an improved regret bound compared to Theorem 1.
Moreover, the RDP privacy guarantee improves to (cid:98) ε ( α ) ≈
αε 2 2 , which is the standard RDP rate for Gaussian mechanism [31].
However, a larger s leads to an increase of communicating bits per user, but only grows logarithmically, since
Algorithm 1 needs to communicate O (log m ) bits to the secure protocol S .
RDP to Approximate DP.
To shed more insight on Theorem 2, we convert our RDP guarantee to approximate DP for a sufficiently large s .
It holds that under the setup of Theorem 2, for sufficiently large s , one can achieve ( O ( ε ) , δ ) -DP with regret O (cid:0) (cid:80) a ∈ [ K ]:∆ a > 0 log T ∆ a + K √ log T log(1 /δ )
ε (cid:1) (via
Lemma 7).
The implication of this conversion is three-fold.
First , this regret bound is O ( √ log T ) factor tighter than that achieved by Tenenbaum et al.
[22] using a shuffle protocol with same ( ε, δ ) -DP guarantee.
Second , it yields a better regret performance compared to the bound achieved under ( ε, 0) -DP in Theorem 1 when the privacy budget δ > 1 /T .
This observation is consistent with the fact that a weaker privacy guarantee typically warrants a better utility bound.
Third , this conversion via RDP also yields a gain of O ( (cid:112) log(1 /δ )) in the regret when dealing with privacy composition (e.g., when participating users across different batches are not unique ) compared to [22] that can only rely on approximate DP.
This results from the fact that RDP provides a tighter composition compared to approximate DP.
C.1 Proof of Proposition
Proof.
We first establish the following result.
Claim 1.
For all λ ∈ R , we have
[Equation]
To show this, by the infinite product representation of the hyperbolic cosine function, we have cosh( λ ) =
[Equation]
where (a) holds by the fact that 1 + x ≤ e x for all x ∈ R and (b) follows from (cid:80) ∞
[Equation]
[Equation]
Then, note that X ∼ Sk (0 , σ 2 ) , then its moment generating function (MGF) is given by E (cid:2)
e λX (cid:3) =
exp( σ 2 (cosh( λ ) − 1)) .
Hence, by the above claim, we have E (cid:2)
e λX (cid:3) ≤ exp( σ 2 ( e λ 2 / 2 − 1) .
Further, note that e x − 1 ≤ 2 x for x ∈ [0 , 1] .
Thus, for | λ | ≤ √ 2 , we have
[Equation]
Hence, by the definition of sub-exponential random variable (cf.
Lemma 4), X is (2 σ 2 , √ 22 ) -subexponential, which again by Lemma 4 implies the required concentration result, i.e., for any p ∈ (0 , 1] , with probability at least 1 − p ,
[Equation]
C.2 Proof of Theorem
We will leverage the following result in [16, Theorem 3.5 ] to prove privacy guarantee.
Lemma 2.
For α ∈ Z , α > 1 , let X ∼ Sk (0 , σ 2 ) .
Then, an algorithm M that adds X to a sensitivity- ∆ query satisfies ( α, ε ( α )) -RDP with ε ( α ) given by
[Equation]
Proof of Theorem 2.
Privacy: As before, we fix any batch b and arm a , for simplicity, we let x i = r ia
( b ) and n = l ( b ) .
Then, it suffices to show that the mechanism H that accepts an input dataset { (cid:98)
x i } i and outputs (cid:80) i (cid:98) x
i + (cid:80) i
η i is private.
To this end, since each local randomizer R generates noise η i ∼ Sk (0 , g 2 nε 2 ) and Skellam is closed under summation, we have that the total noise (cid:80) i η i ∼
Sk (0 , g 2 ε 2 ) .
Thus, by Lemma 2 with ∆ = g , we have that for each batch b with n = l ( b ) and g = ⌈ sε √ n ⌉ , Algorithm 1 is ( α, (cid:98) ε n ( α ))
-RDP with (cid:98) ε n ( α ) given by
[Equation]
Since n = l ( b ) > 1 , we have that for all batches b ≥ 1 ,
[Equation]
Regret: We will establish the following high probability bound so that we can apply our generic regret bound in Lemma[-
-]
[Equation]
where (cid:98)
y :
= (cid:98) y a ( b ) for each batch b and arm a .
We again divide the LHS of (7) into
[Equation]
In particular, Term (ii) can be bounded by using the same method before, i.e.,
[Equation]
Hence, by the choice of g = ⌈ sε √ n ⌉ , we establish that with high probability
Term (ii) ≤ O
[Equation]
[Equation]
For Term (i), as in the previous proof, the key is to show that
[Equation]
To this end, we will utilize our established result in Proposition 1.
Note that the total noise (cid:80) i η i ∼
Sk (0 , g 2 ε 2 ) , and hence by Proposition 1 and the choice of τ = ⌈ 2 gε (cid:112) log(2 /p )
+ √ 2 log(2 /p )
⌉ , we have the above result.
Following previous proof, this result implies that with high probability
[Equation]
Combining the bounds on Term (i) and Term (ii), we have that the private noise satisfies Assumptionwith constants σ = O (1 /ε ) and
h =
O ( 1 sε ) .
Hence, by the generic regret bound in Lemma 1, we have established the required result.
D Auxiliary Lemmas
In this section, we summarize some useful facts that have been used in the paper.
Lemma 3 (Concentration of sub-Gaussian) .
A mean-zero random variable X is σ 2 -sub-Gaussian if for all λ ∈ R
[Equation]
Then, it satisfies that for any p ∈ (0 , 1] , with probability at least 1 − p ,
[Equation]
Lemma 4 (Concentration of sub-exponential) .
A mean-zero random variable X is ( σ 2 , h ) -subexponential if for | λ | ≤ 1 /h
[Equation]
Then, we have
[Equation]
Thus, it satisfies that for any p ∈ (0 , 1] , with probability at least 1 − p ,
[Equation]
Lemma 5 (Hoeffding’s Inequality) .
Let X 1 , . . .
, X n be independent and identically distributed (i.i.d) random variables and X i ∈ [0 , 1] with probability one.
Then, for any p ∈ (0 , 1] , with probability at least 1 − p ,
[Equation]
Lemma 6 (Sum of sub-exponential) .
Let { X i } ni
=1 be independent zero-mean ( σ 2 i , h i )
-subexponential random variables.
Then, (cid:80) i
X i is ( (cid:80) i σ
i , h ∗ ) -sub-exponential, where h ∗ := max i h i .
Thus, we have
[Equation]
In other words, for any p ∈ (0 , 1] , if v ≥ max { (cid:112) 2 (cid:80) i σ 2 i log(2 /p ) , 2 h ∗ log(2 /p ) } , with probability at least 1 − p , | (cid:80) i
X i | ≤ v .
Lemma 7 (Conversion Lemma) .
If M satisfies ( α, ε ( α ))
-RDP, then for any δ ∈ (0 , 1) , M satisfies ( ε, δ ) -DP where
[Equation]
If M satisfies 12 ε 2 -CDP, then for for any δ ∈ (0 , 1) , M satisfies ( ε ′ , δ ) -DP where
[Equation]
Moreover, if M satisfies ( ε, 0) -DP, then it satisfies ( α, 12 ε 2 α ) -RDP simultaneously for all α ∈ (1 , ∞ ) .
Theorem 3 (Advanced composition) .
Given target privacy parameters ε ′ ∈ (0 , 1) and δ ′ > 0 , to ensure ( ε ′ , kδ + δ ′ )
-DP for the composition of k (adaptive) mechanisms, it suffices that each mechanism is ( ε, δ ) -DP with ε = [Equation]
E More Details on Simulations
We numerically compare the performance of Algorithm 1 under pure-DP and RDP guarantees in the distributed model (named Dist-DP-SE and Dist-RDP-SE, respectively) with the DP-SE algorithm of
Sajed and Sheffet [11], which achieves pure-DP under the central model.
We vary the privacy level as ε ∈ { 0 . 1 , 0 . 5 , 1 } , where a lower value of ε indicates higher level of privacy.
In Figure 2, we consider the easy instance, i.e., where arm means are sampled uniformly in [0 . 25 , 0 .
In Figure 3, we consider the hard instance, i.e., where arm means are sampled uniformly in [0 . 45 , 0 . 55] .
The sampled rewards are Gaussian distributed with the given means and truncated to [0 , 1] .
We plot results for K = 10 arms.
We see that, for higher value of time horizon T , the time-average regret of Dist-DP-SE is order-wise same to that of DP-SE, i.e., we are able to achieve similar regret performance in the distributed trust model as that is achieved in the central trust model.
As mentioned before, we observe a gap for small value of T , which is the price we pay for discrete privacy noise (i.e., additional data quantization error on the order of O ( 1 ε (cid:112) log(1 /p )) ) and not requiring a trusted central server.
Hence, if we lower the level of privacy (i.e., higher value of ε ), this gap becomes smaller, which indicates an inherent trade-off between privacy and utility.
We also observe that if we relax the requirement of privacy from pure-DP to RDP, then we can achieve a considerable gain in regret performance; more so when privacy level is high (i.e., ε is small).
This gain depends on the scaling factor s – the higher the scale, the higher the gain in regret.
In Figure 4, we compare regret achieved by our generic batch-based successive arm elimination algorithm (Algorithm 1) instantiated with different protocols P under different trust models and privacy guarantees: (i) central model with pure-DP (CDP-SE), (ii) local model with pure-DP (LDPSE), (iii) Distributed model with pure-DP (Dist-DP-SE), Renyi-DP (Dist-RDP-SE) and ConcentratedDP (Dist-CDP-SE).
First, consider the pure-DP algorithms.
We observe that regret performance of CDP-SE and Dist-DP-SE is similar (with a much better regret than LDP-SE).
Now, if we relax the pure-DP requirement, then we achieve better regret performance both for Dist-RDP-SE and
Dist-CDP-SE.
Furthermore, Dist-CDP-SE performs better in terms of regret than Dist-RDP-SE.
This is due to the fact that under CDP, we use discrete Gaussian noise (which has sub-Gaussian tails) as opposed to the Skellam noise (which has sub-exponential tails) used under RDP.
Experiment on Real Data We evaluate the performance of our algorithm on bandit instances generated from Microsoft Learning to Rank dataset MSLR-WEB10K [33].
The dataset consists of 1200192 rows and 138 columns, where each row corresponds to a query-url pair.
The first column is relevance label 0, 1, ... , 4 of the pair, which we take as rewards.
The second column denotes the query id, and the rest 136 columns denote contexts of a query-url pair.
We cluster the data by running
K-means algorithm with K = 50 .
We treat each cluster as a bandit arm with mean reward as the empirical mean of the individual ratings in the cluster.
This way, we obtain a bandit setting with number of arms K = 50 .
We average over 10 parallel runs and plot the results in Figure 5 for privacy levels ε ∈ { 1 , 5 , 10 } .
Similar to synthetic data experiments, we observe that as T becomes large, the regret performance of our algorithm (Dist-DP-SE) under distributed model matches the regret of the state-of-the-art algorithm (DP-SE) under central model.
Furthermore, consistent with observations on synthetic data, we also observe a considerable gain in regret under RDP compared to pure DP.
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100)
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100)
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
CDP-SE
LDP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-CDP-SE(s=10) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
CDP-SE
LDP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-CDP-SE(s=10) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .0 .0 .
T i m ea v e r a g e
R e g r e t
CDP-SE
LDP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-CDP-SE(s=10)
Round × 100 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10)
Dist-RDP-SE(s=100) 0 .0 .0 .0 .0 .1 .
Round × 100 .0 .0 .
T i m ea v e r a g e
R e g r e t
DP-SE
Dist-DP-SE
Dist-RDP-SE(s=10) Dist-RDP-SE(s=100)