Rényi Differential Privacy of Propose-Test-Release and Applications to Private and Robust Machine Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Propose-Test-Release (PTR) is a differential privacy framework that works withlocal sensitivity of functions, instead of their global sensitivity.
This framework istypically used for releasing robust statistics such as median or trimmed mean ina differentially private manner.
While PTR is a common framework introducedover a decade ago, using it in applications such as robust SGD where we needmany adaptive robust queries is challenging.
This is mainly due to the lack of
Rényi Differential Privacy (RDP) analysis, an essential ingredient underlying themoments accountant approach for differentially private deep learning.
In this work,we generalize the standard PTR and derive the ﬁrst RDP bound for it.
We show thatour RDP bound for PTR yields tighter DP guarantees than the directly analyzed( " ,  ) -DP.
We also derive the algorithm-speciﬁc privacy ampliﬁcation bound of PTRunder subsampling.
We show that our bound is much tighter than the general upperbound and close to the lower bound.
Our RDP bounds enable tighter privacy losscalculation for the composition of many adaptive runs of PTR.
As an applicationof our analysis, we show that PTR and our theoretical results can be used todesign differentially private variants for byzantine robust training algorithms thatuse robust statistics for gradients aggregation.
We conduct experiments on thesettings of label, feature, and gradient corruption across different datasets andarchitectures.
We show that PTR-based private and robust training algorithmsigniﬁcantly improves the utility compared with the baseline.
Introduction
Privacy is a major concern for deploying machine learning (ML).
In response, differential privacy(DP)
[DMNS06] has become the de-facto measure of privacy.
For a differentially private mechanism,the probability distribution of the mechanism’s outputs on a dataset should be close to the distributionof its outputs on the same dataset with any single individual’s data replaced.
A general recipe forreleasing the value of a function f on dataset S in a differentially private way is adding random noiseto f ( S ) (output perturbation), where noise magnitude should scale with f ’s global sensitivity.
However, it might be over-conservative to add noise scaled with global sensitivity.
There is a line ofresearch on whether we can do better (e.g., [NRS07, DL09, TS13, KNRS13]).
Propose-Test-Release(PTR)
[DL09] is a framework that improves the general recipe with the notion of local sensitivity .
The main idea of PTR is as follows: instead of adding noise with respect to global sensitivity, wepropose an amount of noise that is tolerable for many common queries.
When we receive the actualquery, we test (in a differentially private way) whether answering the query with the proposed amountof noise is enough for privacy.
If the proposed noise is too small for limiting the privacy loss from theactual query, we may refuse to answer the query or respond with a larger noise.
PTR works especiallywell for releasing robust statistics such as median or trimmed mean, as the robust statistics usuallyhave small local sensitivity on most common inputs.
While PTR is a basic framework that ages back to the early days after the introduction of DP, it has notbeen used for differentially private optimization before.
DP-SGD [ACG + 16] is the general backbonefor differentially private deep learning and optimization.
One major challenge of augmenting SGDwith PTR and training DP models is the calculation of privacy parameters after a large number ofadaptive compositions.
Rényi differential privacy (RDP) and Moment Accountant [Mir17, ACG + 16]enable us to calculate tighter privacy parameters for training DP models.
Without the RDP bound of
PTR, we need to calculate the privacy parameters using the advanced composition theorems [DR + 14]that can lead to signiﬁcantly looser privacy bounds.
Besides, we also need the bound of privacyampliﬁcation by subsampling for PTR, which is the other important support for training DP models.
It allows us to exploit the stochasticity of SGD for the interest of stronger privacy guarantees.
Technical Overview.
In this work, we derive the Rényi DP bound for PTR, as well as for its Poissonsubsampled variant.
Our bounds make it possible for us to use PTR framework in augmenting private
SGD.
PTR could be characterized by three mechanisms.
The ﬁrst mechanism is M 1 that determineswhich mechanism to run next.
Depending on the outcome of M 1 we then either run M 2 or M 0 2 .
It is often the case that the worst-case privacy loss of one of M 2 or M 0 2 is much larger than the 52 other.
Our bound exploits the fact that the worst-case will happen with small probability.
Speciﬁcally,instead of considering the worst-case privacy loss between M 2 and M 0 2 , and naively composing it 54 with M 1 , we show that for RDP we can tighten the bound by the average privacy loss of M 2 and
M 0 2 under the distribution imposed by M 1 .
Direct ( " ,  ) -DP analysis does not enjoy this beneﬁt;compared with directly analyzing the ( " ,  ) -DP bound of PTR, we show that by ﬁrst bounding the
RDP of PTR and then convert it to ( " ,  ) -DP can lead to better privacy guarantee.
Our proof couldserve as a general recipe for analyzing DP/RDP guarantees for composed mechanisms where theprivacy loss of each mechanism is adaptively determined.
Additionally, we extend our analysis to the
RDP of subsampled PTR.
Our algorithm-speciﬁc analysis (the “white-box bound”) allows us to gettighter privacy ampliﬁcation bounds, compared with the one obtained by general subsampled RDPbound that supports any mechanisms (the “black-box bound”)
[ZW19].
The proof tackled severaladditional difﬁculties compared with the analysis for simple Gaussian mechanism [MTZ19].
Bynumerical veriﬁcation, we show that our RDP bound for subsampled PTR is much tighter than theblack-box bound, and is close to the lower bound (Figure 2.).
Applications of PTR in ML.
PTR is especially suitable for improving the utility of privatizing robuststatistics such as trimmed mean, as these functions usually have a much smaller local sensitivitycompared to their global sensitivity.
A critical application of robust statistics for machine learningis defending against corrupted data.
The learning algorithm should be robust in the presence ofcorrupted data (referred to as Byzantine failure [LSP19]).
Since privacy and robustness are two majorconcerns for ML training, developing techniques that achieve both goals simultaneously is desirable.
We demonstrate the application of PTR in incorporating differential privacy with robust SGD methods.
A popular technique for robustifying SGD is to replace the mean with some robust statistics (e.g.,trimmed mean) for gradient aggregation [YCKB18, AHJ + 21, GLV21].
We use trimmed mean as anexample of showing how to augment robust SGD with DP through PTR.
We show that the augmented
SGD still maintains the robustness guarantee.
We conduct extensive experiments on defending againstthree kinds of attacks: label, feature, and gradient corruptions, and we show that PTR-based robust
SGD achieves much better utility than naively privatizing the robust SGD with global sensitivity.
Related Work
Since the seminal work of [DL09], the Propose-Test-Release has become a common DP frameworkmainly used in statistical inference [BAM20, LKO21].
To the best of our knowledge, this work is theﬁrst application of PTR in machine learning.
Several prior works (e.g., [LQS12, BBG18, WBK19, ZW19]) focus the general subsampled DP/RDPbound that supports any mechanisms.
[WBK19] derives a general RDP privacy ampliﬁcationfor “sampling without replacement” scheme, and [ZW19] obtains a similar result under Poissonsubsampling (i.e., including each data point independently at random with certain probability).
Onlya small body of recent works study algorithm-speciﬁc privacy ampliﬁcation bound by subsampling.
However, most of them focus on subsampled Gaussian mechanism [ACG + 16, BDRS18, MTZ19].
This work derives the ﬁrst subsampled RDP bound speciﬁc for ﬂexible algorithms such as PTR.
Designing machine learning and optimization algorithms that achieve both privacy and Byzantinerobustness is certainly an important direction.
Nevertheless, there are only few works on this line sofar.
[GGP + 21] considered the problem of achieving privacy and byzantine resilience in distributed
SGD with an untrusted server.
The privacy level they considered is essentially local differentialprivacy, which is orthogonal to our focus.
[HKJ20] and [SGA20] aim for both robustness andsecure multiparty computation instead of differential privacy.
[MZH19] naively applies differentialprivacy to defend against data poisoning attack.
However, they ﬁnd that DP alone cannot defendadversaries that poison a large fraction of training examples.
Aside from the setting of training
ML models, [LKKO21] propose a polynomial time algorithm that achieves both goals for meanestimation via privatizing ﬁlter-based robust mean estimator [DKK + 17].
[EMN21] develop a robustand differentially private mean estimator based on exponential mechanism.
However, both of theirapproaches become inefﬁcient (even in polynomial time asymptotically) in high dimensional settings.
Background
In this section, we introduce some background on differential privacy, Rényi differential privacy, andprivacy-ampliﬁcation by subsampling.
We will also introduce notations as we proceed.
Differential Privacy.
Differential privacy is a framework for protecting privacy when performingstatistical releases on a dataset with sensitive information about individuals (see the surveys [DR + 14,
Vad17]).
Speciﬁcally, for a differentially private mechanism, the probability distribution of themechanism’s outputs of a dataset should be close to the distribution of its outputs on the same datasetwith any single individual’s data replaced.
To formalize this, we call two datasets S , S 0 , each multisetsover a data universe X , adjacent if one can be obtained from the other by adding or removing a singleelement of X .
Further, we use d ( S, S 0 ) to denote the number of times of adding/removing of datapoints to transform S to S 0 .
So S and S 0 are adjacent if and only if d ( S, S 0 ) = 1 .
Deﬁnition 3.1 (Differential Privacy [DMNS06]) .
For " ,   0 , a randomized algorithm M : 114 MultiSets(
X ) ! Y is ( " ,  ) -differentially private if for every dataset pair S, S 0 2 MultiSets
( X ) 115 such that d ( S, S 0 )
= 1 , we have:
[Equation]
where the randomness is over the coin ﬂips of M .
Rényi Differential Privacy (RDP).
Rényi differential privacy (RDP) is a variant of the standard( " ,  ) -DP that uses Rényi-divergence as a distance metric between the output distributions of M ( S )and M ( S 0 ) , which is particularly useful in training differentially private machine learning models.
Deﬁnition 3.2 (Rényi Differential Privacy [Mir17]) .
We say that a mechanism M is ( ↵ , " ) -RDP withorder ↵ 2 (1 , 1 ) if for every dataset pair S, S 0 2 MultiSets( X ) such that d ( S, S 0 )
= 1 , we have:
[Equation]
2) where µ M ( · ) denotes the density function of M ’s distribution.
As we can see, ( ↵ , " ) -RDP is essentially an upper bound for the moment
E [Equation]
◆
[Equation]
 exp(( ↵  1) " ) for all adjacent S, S 0 , where " can be viewedas a degree of the privacy loss incurred by running M .
For convenience, in the following presentation,we denote the bound as a function f ↵ ( " ) = exp(( ↵  1) " ) .
A different ↵ typically leads to a differentprivacy bound " .
Following the convention of literature [ZW19], we view " as a function of ↵ , andthe notation " M ( ↵ ) means the algorithm M obeys ( ↵ , " M ( ↵ ))
-RDP.
We note that RDP reduces to( " , 0) -DP when we take ↵ = 1 .
The current tightest ( " ,  ) -DP/RDP transformation we are aware ofis by [ALC + 21].
A central property of DP/RDP is its behavior under composition.
If we run multipledistinct differentially private algorithms on the same dataset, the resulting composed algorithm isalso differentially private, with some degradation in the privacy parameters ( " ,  ) .
Speciﬁcally, if werun k sequentially chosen ( " ,  ) -DP algorithm on a dataset, the overall composed privacy parameteris ⇣ ˜ O
( p k " ) , k  + ⌘ -DP by Advanced composition theorem [DR + 14].
However, the Advancedcomposition theorem for ( " ,  ) -DP is loose.
On the contrary, the composition is trivial for RDP as" M 1  M 2 ( · ) =
" M 1 ( · ) + " M 2 ( · ) .
The Moment Accountant technique, which composes RDP and 137 then transforms to DP, is a much simpler approach and often produces much more favorable privacyparameters than directly composing ( " ,  ) -DP.
Therefore, RDP and Moment Accountant are widelyused to calculate the privacy guarantee in training differentially private deep learning models.
Privacy ampliﬁcation by subsampling.
“Privacy ampliﬁcation by subsampling” is the other boosterbesides RDP / moments accountant that drives much of the recent progress in differentially privatedeep learning.
Most of the existing works focus on Poisson subsampling [BBG18], which sampleseach data point independently with a given sampling probability q .
The tightest privacy ampliﬁcationbound we are aware of for general mechanism under Poisson Subsampling is from [ZW19].
Propose-Test-Release and Rényi Differential Privacy
In this section, we introduce a general version of the Propose-Test-Release framework, present ourmain results on its RDP bound, and its privacy ampliﬁcation bound under Poisson subsampling.
In our presentation, we use GS f = sup S,S 0 :
d ( S,S 0 )=
1 k f ( S )  f ( S 0 )
k to denote the global sensitivityof target function f (in ` 2 distance), and we use LS f ( S )
= sup S,S 0 : d ( S,S 0 )=
1 k f ( S )  f ( S 0 )
k todenote the local sensitivity of function f on dataset S .
Before we present our main results, we wouldlike to remind the readers about the deﬁnition of Laplace and Gaussian mechanisms, as well as their
DP/RDP bound, which will be referred to in our results.
Laplace Mechanism: If f ’s output is 1-dimensional, the Laplace mechanism M Lap ,b ( S ) =
f ( S ) + Lap(0 , b ) obeys ( " (˜ b ) Lap , 0) -DP for " (˜ b ) Lap =
e 1 / ˜ b , and obeys ( ↵ , " (˜ b )R  Lap ( ↵ ))
-RDP for" (˜ b ) R  Lap ( ↵ )
= 1 ↵  1 log ⇣ ↵ 2 ↵  1 exp ⇣ ↵  1 ˜ b ⌘ + ↵  1 2 ↵  1 exp ⇣
↵ ˜ b ⌘⌘ , where ˜ b = b/ GS f
(the “noiseto-sensitivity” ratio).
Gaussian Mechanism:
If f ’s output is d -dimensional, the Gaussian mechanism M N ,  ( S ) = f ( S )+ N (0 ,  2 1 d ) obeys ( " (˜  ) N (  ) ,  ) -DP for " (˜  ) N (  ) =
˜  p 2 log(1 .
25 /  ) , and obeys ( ↵ , " (˜  ) R  N ( ↵ )) -
RDP for " (˜  ) R  N ( ↵ ) =
↵ 2˜  2 , where ˜  =  / GS f
(the “noise-to-sensitivity” ratio).
4.1 Propose-Test-Release
input : S – dataset, f 1 – target function, f 2 – robust statistic, ⌧ – proposed local sensitivity bound of f 2 ,  1 ,  2 , b –
Gaussian/Laplace noise scales (  1 >  2 ),  0 – failure probability.
1  min ˜ S 2 { ˜ S : LS f 2 (˜ S ) > ⌧ } d ⇣ S, ˜ S ⌘ .
b   + Lap(0 , b ) .
3 if b   log(1 / (2  0 ))
b then return f 1 ( S ) + N (0 ,  21 1 d ) 5 else return f 2 ( S ) + N (0 ,  22 1 d )
Naive use of Laplace/Gaussian mechanismmay result in the poor utility of functionoutput, as the global sensitivity of the target function may be intolerably large dueto some extreme cases.
Meanwhile, robust statistics such as median, mode, InterQuantile Range (IQR) are quite insensitiveto single data addition/removal for datasetsthat are i.i.d.
drawn from natural distributions.
This means that robust statistics mayhave a small local sensitivity for most inputdatasets.
The seminal work of [DL09] introduced Propose-Test-Release frameworkto reduce the noise addition when the targetfunction has an approximation that is a robust statistic 1 .
Here, we introduce a moregeneral and useful version of PTR instantiated by Laplace and Gaussian mechanism.
Given a target function f 1 (e.g., mean) and its robustvariant f 2 (e.g., median), the PTR framework proceeds as follows: (1) Propose: a local sensitivitybound of the target function, ⌧ , is proposed; (2) Test: a safety margin  ( S ) , which is the minimumamount of data points that we need to replace for S to have local sensitivity larger than ⌧ , is computed.
A private version (via Laplace mechanism) of the safety margin, b  , is compared with a threshold;(3)
Release: if the safety margin is large enough, then the algorithm releases f 2 ( S ) via Gaussianmechanism with a smaller noise  2 (usually scaled with ⌧ ).
Otherwise, the algorithm release f 1 ( S )with a larger noise  1 (usually scaled with GS f 1 ).
The pseudocode is outlined in Algorithm 1.
We make several remarks on Algorithm 1.
Remark 4.1 .
(1) The traditional version of PTR in the textbooks [DR + 14, Vad17] simply refuse tooutput anything when the noisy safety margin b  is small.
Algorithm 1 is a more general version which output f 1 ( S )
+ N (0 ,  21 1 d ) when the sensitivity test is failed.
The textbook PTR can bethought of as a special case of Algorithm 1 where we set  1 ! 1 .
The mechanism used in the
Test and Release step can be other mechanisms instead of Laplace and Gaussian.
In this paper, wepresent our results for the PTR instantiated by these two mechanisms since we intend to apply it fordifferentially private SGD later.
Our proof can be easily extended to other mechanisms as well.
The threshold log(1 / (2  0 ))
b in Line 3 is chosen so that Pr[Lap(0 , b ) >
log(1 / (2  0 ))
b ] =  0 .
The global sensitivity of  is 1 for any functions.
This could be seen by noticing that, for any pair ofadjacent S, S 0 , we have d ( S, ˜ S )  d ( S 0 , ˜ S ) + 1 for any dataset ˜ S .
Direct DP Analysis of Propose-Test-Release.
We show the DP guarantee for the Propose-TestRelease framework.
To prove privacy, we need to ﬁnd the worst pair of adjacent datasets S and
S 0 that incurs the largest privacy loss.
It is clear that the worst possible scenario of PTR is whenf 2 ( S )  f 2 ( S 0 ) >
⌧ , while the algorithm still releases f 2 ( S ) with noise scaled with ⌧ .
However, notethat this worst possible scenario will only happen when both LS ( S ) and LS ( S 0 ) are greater than ⌧ .
In this case, however,  for both S and S 0 are 0 since there are no data points we need to changefor them to have local sensitivity larger than ⌧ , and thus there is no privacy loss from b  .
Moreover,when  = 0 , the probability that PTR will release f 2 ( S ) + N (0 ,  22 1 d ) is at most  0 , which couldbe simply added to the  term in ( " ,  ) -DP.
If one of LS ( S ) and LS ( S 0 ) is smaller than ⌧ , then weknow that f 2 ( S )  f 2 ( S 0 )  ⌧ , and the overall privacy parameters could be computed by Basic
Composition Theorem [DR + 14].
We obtain the following differential privacy guarantee for PTRbased on these observations.
We defer more details of the proof to the Appendix A.
Theorem 4.2 (Direct DP analysis for PTR) .
Suppose GS f 1 =
GS f 2 = 1 and  1 =  2 / ⌧ , then
Algorithm 1 is ( " ( b ) Lap + " (  1 ) N (  ) ,  0 +  ) -DP, where " ( b ) Lap =
e 1 /b is the DP guarantee for Laplacemechanism, and " (  1 )
N (  ) =
p 2 log(1 .
25 /  ) .
4.2 RDP of Propose-Test-Release
We then derive the RDP of Propose-Test-Release framework.
The major differences between theproofs of ( " ,  ) -DP and RDP bound is that, for ( " ,  ) -DP we can move the probability of runninginto the bad scenario to the  term, while for RDP we need to consider the “average-case” privacyloss.
This is in fact an advantage of RDP.
Consider a mechanism M that randomly selects one ofmechanisms M 1 and M 2 to run, each with probability 1   0 and  0 .
Suppose M 1 is ( " 1 ,  1 ) {+-DP 
+}
[-δ 1 + δ 
-]and
M 2 is ( " 2 ,  2 ) [--DP, or if we know ε 
-]-DP.
We can only claim that M is (max( " 1 , " 2 ) , (1   0 )  1 +  0  2 ) -DP, or ifwe know " 2  " 1 we can also move the probability  0 to the  term and obtain ( " 1 ,  0 +  1 )(which is the case for Theorem 4.2).
However, if we know the RDP guarantee of M 1 and M 2 as
E M 1 ( S ) h⇣ µ M 1( S 0 ) µ M 1( S ) ⌘ ↵ i  f ↵ ( " 1 ) and E M 2 ( S ) [-∥M 
-]h⇣ µ
[Equation]
0 ) µ M 2( S ) ⌘ ↵ i  f ↵ ( " 2 ) , then E M ( S ) h⇣ µ M ( S 0 ) µ M ( S ) ⌘ ↵ i can be simply bounded by (1   0 )
f ↵ ( " 1 ) + [-δ 
-] 0 f ↵ ( " 2 ) .
Compared with ( " ,  ) -DP analysis, there areno extra inequalities used in RDP analysis of M except for the RDP guarantee of M 1 and M 2 .
Thus, 224 the RDP bound is tighter than the DP bound, especially when  0 is close to the target  .
Theorem 4.3 (RDP analysis of PTR) .
Suppose GS f 1 =
GS f 2 = 1 and  1 =  2 / ⌧ .
Then for any↵ > 1 , Algorithm 1 is ( ↵ , " PTR ( ↵ ))
-RDP for
[Equation]
⌘ The above result implies that, with appropriately chosen  0 , the privacy loss of PTR is the same asdirectly adding up the privacy loss from b  and from releasing f 1 ( S ) + N (0 ,  21 1 d ) regardless of thevalue of b  .
Compared with naively releasing f 1 ( S ) + N (0 ,  21 1 d ) , PTR will only pay an extra costfor privatizing  , independent from the privacy cost for the case of releasing f 2 ( S ) + N (0 ,  22 1 d ) !
Thus, the algorithm could potentially add much smaller noise while introducing just a little extraprivacy loss.
In Figure 1, we convert this RDP bound to ( " ,  ) -DP via the RDP-DP conversionformula by [BBG + 20], and compare with the ( " ,  ) -DP bound obtained through direct analysis in
Theorem 4.2.
Although due to the loss in RDP-DP conversion, the " parameter by RDP is worse thanthe one by direct analysis when  0 is far smaller than the target  , it is tighter when  0 is close to  .
Alarger  0 means PTR has a greater chance to release f 2 ( S )
+ N (0 ,  22 1 d ) , which leads to a betterutility.
Thus, the privacy parameter converted from the RDP bound is more favorable. bound Theorem
[BBG + 20], and compare it with the the direct analysis in Theorem bound
RDP, we search for the optimal ↵ 2 [ when  0 is small, the RDP for PTR will take the term in
We would like to stress that in the theorem, most of the conditions on the parameters such as GS or 1 ,  2 can be easily relaxed.
The conditions in the theorems are mainly used to make the presentedbound more clean and interpretable.
4.3 RDP for Poisson Subsampled Propose-Test-Release
We further derive the RDP for Poisson subsampled PTR.
Here, “subsampled” means the inputdataset will be subsampled ﬁrst before feeding to PTR.
Poisson subsampling means each datapoint x will be included with probability q independently.
One way of obtaining the RDP for anysubsampled mechanism is by simply plugging in the RDP bound of the original algorithm into theprivacy ampliﬁcation formula for general mechanisms (e.g., [WBK19, ZW19]).
Here, we directlyderive the RDP for Poisson subsampled PTR in a white-box manner.
Denote M 0 as the randomvariable for PTR’s output on dataset S , and M 1 for PTR’s output on dataset S {+
+}
= S [ { x } , and
M = (1  q ) M 0 + q M 1 .
Compared with the analysis for subsampled Gaussian mechanism 251 by [MTZ19], there are two main difﬁculties for extending the arguments of quasi-convexity of
Rényi divergence for PTR: (1) the distribution of PTR’s output may not be centrally symmetric,thus we need to bound both D ↵ ( M||M 0 ) and D ↵ ( M 0 ||M ) .
(2) the conditional distribution M| b cannot be decomposed as (1  q ) M 0 | b  + q M 1 | b  .
A big part of our novelty in the proof is aboutaddressing these two challenges.
We defer details of the proof to Appendix A.
Theorem 4.4 (RDP analysis of sub-sampled PTR (the “white-box bound” )) .
Let q be the subsamplingprobability.
Suppose GS f 1 =
GS f 2 = 1 and  1 =  2 / ⌧ .
When q,  1 ,  2 , and ↵ satisfy certainconditions, we have
[Equation]
[Equation] where B 0 = 1 + 2 q 2 ↵ ( ↵  1)( 1   0  21 +  0  22 ) , B 1 =
R ( ↵ ) q + 2 ↵ ( ↵
[Equation]
[R ( ↵ ) q  2(1  q )R ( ↵  1) q
+(1  q ) 2 R ( ↵  2) q ] , B 2 = e
R ( ↵ ) q + 2 ↵ ( ↵  1)[ e R ( ↵ ) q  2(1  q ) e R ( ↵ +1) q + (1  q )e
R ( ↵ +2) q ] , with R ( ↵ ) q =
E s ⇠ µh⇣ µ ( s ) µ 0 ( s ) ⌘ ↵ i and e
R ( ↵ ) q = E s ⇠ µ h⇣ µ 0 ( s ) µ ( s ) ⌘ ↵ i for µ 0 ⇠ Lap(0 , b ) and µ ⇠ (1  q )Lap(0 , b ) +q Lap(1 , b ) .
Remark 4.5 .
Both R ( ↵ ) q and e R ( ↵ ) q can be computed easily since they either have closed form or canbe accurately computed via numerical integration in bounded range.
This bound may not be very interpretable, which is a typical feature for the privacy ampliﬁcationbounds as they are meant to be implemented in practice.
After all, constant matters for differentialprivacy practitioners!
To show the tightness of our bound, we plug in the RDP of original PTR(Theorem 4.3) to the current tightest privacy ampliﬁcation formula for general mechanisms derivedin [ZW19] (the “black-box bound” ), and compare it with our white-box bound in Theorem 4.4.
Aswe can see from Figure 2, Theorem 4.4 (orange curve) is much tighter than the black-box bound(blue curve).
Moreover, it is very close to the lower bound of privacy ampliﬁcation by [ZW19] (greenline), which means that Theorem 4.4 is near optimal.
In Figure 3, we illustrate the application ofsubsampled RDP bound in Moment Accountant.
We can see that the privacy parameters for the composed mechanism obtained based on our white-box bound (Theorem 4.4) is tighter than the one by black-box bound, as well as the one by directly composing Theorem 4.2 with strong compositiontheorem [KOV15].
We remark that Direct Analysis achieves lower privacy loss with very fewiterations since we set {+ 
+} = 10  8 to allow more iterations for Strong Composition of ( " ,  ) -DP,which leads to a better " for a single iteration (see Figure 1).
An exciting byproduct during our research on subsampled PTR is the privacy ampliﬁcation boundfor a M that outputs M 1 ( S ) , M 2 ( S ) sequentially and independently .
This result is important sincesuch a M is instantiated in many existing techniques in improving differentially private deep learning.
We defer this result to the Appendix.
Differentially Private and Robust SGD with Propose-Test-Release
In this section, we demonstrate the application of the RDP for (subsampled) PTR algorithm inprivatizing robust SGD algorithms.
Mini-batch SGD is the de-facto method for training deep neural networks.
Despite its strongconvergence properties in the standard settings, it is well known that even a small fraction ofcorrupted gradients can lead SGD to an arbitrarily poor solution.
This is typically being modeledas Byzantine failure [LSP19], meaning that some gradients can be arbitrarily corrupted , and thecorrupted gradients may be crafted in an adversarial way.
A popular class of defense strategies against Byzantine failure is to use robust aggregation rules such as trimmed-mean [GLV21], coordinate-wisemedian
[YCKB18], and geometric median [AHJ + 21].
Since both privacy and robustness are essentialfor ML applications, it is important to develop learning algorithms that achieve differential privacyand robustness simultaneously.
Baseline: privatizing robust SGD with Gaussian Mechanism.
A natural (but naive) way toaccomplish such a goal for SGD-based algorithms is to privatize each iteration of robust SGDthrough Gaussian mechanism.
Speciﬁcally, in each iteration, we add Gaussian noise to the aggregatedgradient, where noise magnitude scale with the global sensitivity of the robust aggregation function.
The privacy parameter for the entire process is computed through Moment Accountant.
The global sensitivity can be obtained by either upper bounding the Lipschitz constant of the objective function,or clipping each gradient according to a certain threshold.
However, this method could potentiallyadd over-conservative noise to the aggregated gradients, as the local sensitivity of robust aggregationfunctions is usually very low for most input gradient sets.
Privatizing robust SGD with PTR.
We improve the naive method with the PTR framework,where we instantiate it by showing how to privatize trimmed-mean SGD method from [GLV21].
We ﬁrst deﬁne trimmed-sum 2 aggregation function, and show that the  for trimmed-sum couldbe computed efﬁciently.
Given a dataset S
= { x 1 , . . .
, x m } , let x ( k ) denote the k th smallestdata point among S in ` 2 norm, i.e.,  x (1)    x (2)   . . .
  x ( m )  .
The F -trimmedsum of S is deﬁned as TSUM F ( S ) =
P m  F i =1 x ( i ) if m > F , or simply 0 if m  F .
Deﬁne LS ( r ) f ( S ) = sup S, ˜ S : d ( S, ˜ S )=
r LS f ( ˜ S ) , which is the maximum local sensitivity that can beachieved by adding/removing r elements from S .
Thus, LS (0) f ( S ) =
LS f ( S ) , and the safety margin ( S ) =
min { r : LS ( r ) f ( S ) > ⌧ } .
Therefore, as long as we can efﬁciently compute LS ( r ) f
( S ) , wecan efﬁciently compute  ( S ) in linear time by simply enumerate all r from 0 to m and terminateonce LS ( r ) f ( S ) > ⌧ .
In fact, LS ( r )TSUM F ( S ) can be computed in O (1) time.
Theorem 5.1.
LS ( r ) TSUM F ( S ) =
x ( m  F [-+
 +-]+1+ r )  if r  F  1 , or GS TSUM F if r > F  1 .
Back to the goal of designing differentially private and robust SGD.
Fix a positive integer F which isa potential upper bound for the number of corrupted gradients.
We instantiate a PTR-based gradientaggregation algorithm by instantiating f 1 ( · ) as SUM( S ) =
P x 2 S x , and f 2 ( · ) as TSUM F ( · ) .
We 319 simply plugging in this PTR algorithm as the gradient aggregation function for regular SGD.
We showthe convergence guarantee of this algorithm in the presence of at most F gradients being corruptedfor the case that the loss function is Lipschitz.
Theorem 5.2 (Informal) .
When the loss function is Lipschitz in model parameters, SGD with PTRinstantiated by SUM and TSUM provides convergence guarantee for update to F gradients beingcorrupted arbitrarily for any F < m/ 2 .
We make two remarks of our speciﬁc design choices for implementing PTR-based SGD in practice.
Remark 5.3 .
(1) In practice, F could be adjusted dynamically during the model training, based onthe value of b  .
That is, if b   B , it means that the current gradient batch has many outliers, thuswe increase F for the next iteration; if b  > B , it means that gradients are relatively concentratedand few of them are corrupted, thus we can decrease F afterwards.
(2) We set f 1 as SUM instead of
TSUM since F is usually set to be larger than the actual corrupted gradients; in this case, b   B 331 means that there are many benign gradients also have large norms.
These benign gradients usuallycorrespond to the data points that are misclassiﬁed by the partially trained models, which is importantfor improving model performance.
The full pseudocode for the algorithm is deferred to the Appendix A.
5.1 Evaluation
In this section, we empirically evaluate the performance of PTR-based private and robust SGD.
Speciﬁcally, we compare the utility-privacy tradeoff between the algorithm that privatize trimmedmean-based SGD with PTR ( TSGD + PTR ), and the baseline algorithm that privatize trimmedmean-based SGD with Gaussian mechanism ( TSGD + Gaussian ).
Experiment Settings.
We evaluate the utility-privacy tradeoff of the two robust and differentiallyprivate SGD algorithms under three common types of attack: label, feature, and communicatedgradient corruption.
For label corruption, we randomly ﬂip of label of certain amount of data points.
For feature corruption, we add Gaussian noise from N (0 , 100) directly to the corrupted images.
Forgradient corruption, we add Gaussian noise from N (0 , 100) to the true gradients.
[-δ 1 + δ 
-]
[-Alternatively, if we know ε 2 ≫ ε 1 we can also move the probability δ 0 to theδ term and obtain ( ε 1 , δ 
-]
We experimenton three classic datasets, MNIST, CIFAR10, and EMNIST.
[-∥ is the global sensitivity GS f 
-]
We vary different corruption ratios andprivacy parameter " .
Experiment details are deferred to Appendix.
[Table]
[-∥ is the global sensitivity GS f 
-]
best accuracy-privacy tradeoff point varied for datasets
. Results.
[-( m + r ) − F +
-]
The comparison of model test accuracy under given corruption settings and privacyparameters are summarized in Table 1.
We highlight the improvement of TSGD + PTR overthe baseline TSGD + Gaussian in red texts.
As we can see, for all settings, TSGD + PTRoutperforms and often works signiﬁcantly better than TSGD + Gaussian .
This demonstrates that,while TSGD + PTR may introduce extra privacy loss in the test step, the performance gain fromadding smaller noise in the release step overshadows it.
[-B.1 Experiment Settings for Table 
 B.1.1 Corruption Simulation. Following the literature in Byzantine robustness [YCKB18, XKG19, AHJ + 21, GLV21], we considerthree possible sources of Byzantine failures: corruption-]We also observe two interesting phenomena in the experiment: (1) a higher corruption ratio maynot necessarily lead to worse model performance for trimmed mean-based robust SGD, especiallyfor MNIST dataset.
This is because the high norm gradients can be either corrupted, or benign butthe partially-trained model misclassiﬁes the corresponding data points.
[-For TSGD+PTR , we set δ 
-]
The latter case is extremelyimportant for improving model performance compared with the gradients of data points are alreadybeing classiﬁed correctly.
[-Specifically, we use two famous,moderately large architecture ResNet
-]
When the [-model. We set gradient clipping bound R = 5 , batch size as 2048, learning rate 0.01. For TSGD+PTR , weset δ 
 = 10 − 8 , b = 1 , and τ = 3 .-]corruption ratio is small, more benign gradients are beingtrimmed, which may lead to worse model performance.
(2) For TSGD + Gaussian , a larger privacybudget may not lead to better model performance on MNIST dataset.
This is because when thetraining accuracy reaches the peak, there are many trimmed gradients whose corresponding trainingdata points are already correctly classiﬁed; continuing training without those data points and withlarge noise may result in catastrophic forgetting.
Limitation and Future Works
This work derives the Rényi Differential Privacy for propose-test-release framework as well asits subsampled version.
With the RDP bound for the PTR framework, this work demonstrate theapplication of PTR in training differentially private and robust models.
One limitation of PTR isthat it does not work well in privatizing coordinate-wise median in high-dimensional space.
Theglobal sensitivity of coordinate-wise median is far greater than the one for mean, which results inhuge privacy loss.
For future work, we would like to explore whether the robust gradient aggregationcould also improve the utility of training differentially private ML models, as they typically exhibitsmaller sensitivity than regular mean.