Optimal Membership Inference Bounds for Adaptive Composition of Sampled Gaussian Mechanisms
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Given a trained model and a data sample, membership-inference (MI) attackspredict whether the sample was in the model’s training set.
A common countermeasure against MI attacks is to utilize differential privacy (DP) during modeltraining to mask the presence of individual examples.
While this use of DP isa principled approach to limit the efﬁcacy of MI attacks, there is a gap betweenthe bounds provided by DP and the empirical performance of MI attacks.
In thispaper, we derive bounds for the advantage of an adversary mounting a MI attack,and demonstrate tightness for the widely-used Gaussian mechanism.
Our boundsare much stronger than those obtained by DP analysis. Our analysis shows thatalthough differential privacy guarantees might sometimes lead to large/vacuousmembership inference guarantees, mechanisms that obtain differential privacy canbe in fact much more secure against membership inference attacks. Previously, thisphenomenon was observed for DP-SGD and here, for the ﬁrst time, we prove it.
Finally, using our analysis, we provide MI metrics for models trained on CIFAR10dataset.
To the best of our knowledge, our analysis provides the state-of-the-artmembership inference bounds.
Introduction
The recent success of machine learning models makes them the go-to approach to solve a variety ofproblems, ranging from computer vision [Krizhevsky et al., 2012] to NLP [Sutskever et al., 2014],including applications to sensitive data such as health records or chatbots.
Access to a trainedmachine learning model, through a black-box API or a white-box access to a published model, canleak traces of information [Dwork et al., 2015] from the training data.
Researchers have tried tomeasure this information leakage through metrics such as membership inference [Shokri et al., 2017].
Membership inference is the task of guessing, from a trained model, whether it includes a givensample or not.
This task is interesting in its own right, as the participation of an individual in a datacollection can be sensitive information.
Furthermore, it also serves as the “most signiﬁcant bit” ofinformation: if membership inference fails, attacks revealing more information such as reconstructionattacks [Fredrikson et al., 2014, Carlini et al., 2020] will also fail.
In other words, defending againstmembership inference attacks also defends against more advanced attacks.
The standard approach to provably defeat these membership privacy attacks is differential privacy [Dwork et al., 2006].
The traditional variant of differential privacy deﬁnes a class of algorithmsthat apply on a database D and respect a privacy budget (cid:15) and a probability of failure δ .
Theseparameters control the divergence between the distribution of outcome of the algorithm when appliedon two “neighboring” databases that only differ in one location.
In particular, our mechanism ofinterest, the Sampled Gaussian Mechanism (SGM), makes an arbitrary function f : X ∗ → R ddifferentially private by ﬁrst sub-sampling a database of smaller size from the original database, using Poisson sampling, and then calculating f on the sub-sampled database, and ﬁnally adding Gaussian noise to the outcome.
Increasing the amount of injected noise will provide stronger privacyguarantees but it often lead to a trade-off between privacy utility of the trained model.
To measure theprivacy of a given algorithm, researchers have developed advanced mathematical tools and notionssuch as Renyi differential privacy [Mironov, 2017, Abadi et al., 2016] and advanced compositiontheorems [Dwork et al., 2010, Kairouz et al., 2015].
These tools allow us to calculate ( (cid:15), δ ) values forcarefully designed algorithms.
Previous work has shown that any differentially private algorithm will provably bound the accuracyof any membership inference adversary.
Speciﬁcally, starting with a given ( (cid:15), δ ) , Humphries et al.[2020] prove that any model trained with ( (cid:15), δ ) differential privacy will induce an upper boundon the accuracy of membership adversary and this upper bound depends only on (cid:15) and δ .
Theseupper bounds enable us to obtain provable defenses against membership inference attacks by usingdifferentially private algorithms.
In fact, there is a large gap between upper bounds proved for thepower of adversaries in performing membership inference, and the power of real adversaries that tryto attack differentially private models.
In this work, we develop a new proof technique to bypass DPanalysis and directly bound the advantage of MI attacks.
The most widely-used DP algorithm in machine learning applications is DP-SGD: it is a small changeof the classical stochastic gradient descent algorithm that only requires to clip per-sample gradients,average them and add Gaussian noise.
DP-SGD has been shown to be more accurate than otherdifferentially private training algorithms in the case of linear and convex models [van der Maatenand Hannun, 2020]. Each iteration of DP-SGD is an instance of the sampled Gaussian mechanism,which chooses a fraction q of a dataset and outputs a noisy sum of the desired quantity.
Ouranalysis mirrors the recent shift in the ﬁeld of empirical membership inference, from advantage (oraccuracy) metrics [Shokri et al., 2017, Yeom et al., 2018, Sablayrolles et al., 2019] to precision/recallmeasures [Watson et al., 2022, Carlini et al., 2020].
Differential privacy guarantees are known toyield tight true positive and false positive rates [Nasr et al., 2021].
Our paper is the ﬁrst to showequivalent results in the case of advantage.
Our analysis explains, from a theoretical point of view,why the precision and recall of membership inference attacks is stronger than the accuracy.
Our contributions in this work are as follows:
• Our main theorem in Section 4 bounds the membership inference advantage of any adversaryfor the composition of an arbitrary set of sampled Gaussian mechanisms that could adaptively depend on each other. Speciﬁcally, for any adaptive series of sampled Gaussian mechanism ( M 1 , . . .
, M T ) where M i has sensitivity 1 . 0 and standard deviation σ i and sub-sampling rate
q i , we show that the membership inference advantage of any adversary is bounded by thetotal variation distance between the mixture of two Gaussians deﬁned by σ = ( σ 1 , . . .
, σ T )and q = ( q 1 , . . . , q T ) . This bound is optimal as it reﬂects the membership inferenceadvantage for a real adversary on a particular series of Gaussian mechanisms.•
We propose a numerical way to calculate the total variation distance between mixtures ofGaussians (Section 4.3).
Our algorithm is computationally tractable and works in lineartime with respect to number of mechanisms and is independent from the dimension.
We useour numerical approach to obtain concrete bounds on membership inference for Gaussianmechanisms and compare our bounds to that of Humphries et al.
[2020].• Finally, to understand the practical implication of our bound for mainstream datasets, we useDP-SGD to train models on CIFAR10 and MNIST and calculate the membership inference bounds using our techniques.
Our approach allows us to achieve state-of-the-art provablemembership inference privacy for any given accuracy.
Background
In this section, we brieﬂy recall the main properties of KL divergence and total variation (TV).
Weconsider a random variable X (resp.
Y) with probability distribution µ (resp.
ν ) over a measureablespace Ω .
With a slight abuse of notation, we consider quantities as function of either the randomvariables X, Y or their probability distributions µ, ν . [Figure] [Figure] [Figure]
[Figure]
A d v a n t a g e Humphries et al. Ours (Pinsker) Ours (TV) Empirical 29.
[Figure]
1 Total variation, KL divergence, and Pinsker inequality
Total variation
[Equation]
TV properties.
We recall brieﬂy some properties of TV that will be useful in the remainder of thispaper.
Given a function g : E → F , applying g to X and Y can only decrease TV :
[Equation]
In particular if g is a bijective transform, we have TV ( g ( X ) , g ( Y ))
=
TV ( X, Y ) .
Kullback-Leibler (KL) divergence.
The KL divergence between X and Y is deﬁned as
[Equation]
Now we are ready to state Pinsker’s inequality that connects total variation to KL divergence:
Pinsker’s Inequality.
For any two probability distributions X and Y we have
[Equation]
2.2 Differential Privacy
Differential privacy.
A randomized algorithm M satisﬁes ( (cid:15), δ ) -differential privacy if, for any twodatasets D and D (cid:48) that differ in at most one sample, and any subset R of the output space, we have:
[Equation]
While the probability of failure δ is often chosen to be inversely proportional to the number ofsamples, there is no consensus in the literature over desirable values of (cid:15) .
DP-SGD with the subsampled Gaussian mechanism.
DP-SGD [Abadi et al., 2016] is a modiﬁcation of Stochastic Gradient Descent that makes it differentially private.
The core mechanism behindit is the subsampled Gaussian mechanism.
Given a function φ that operates on sets S with sensitivity1 (i.e. (cid:107) φ ( S ) − φ ( S (cid:48) )
(cid:107) 2 ≤ 1 ), the subsampled Gaussian mechanism selects sets S at random andadds noise to the output of φ .
Note that the mechanism is differentially private even if all intermediatesteps of the training process are revealed.
2.3 Membership inference attacks
Membership Inference is the task of predicting whether a given sample was in the training setof a given model.
Homer et al.
[2008] showed the ﬁrst proof of concept, and Shokri et al.
[2017] showed that a wide variety of machine learning models are vulnerable to such attacks.
Shokri et
al.[2017] train neural networks to attack machine learning models, and measure the success of the attackby the percentage of correctly predicted (train/test) samples, or equivalently the advantage [Yeomet al., 2018].
While Shokri et al. [2017] trained neural networks to attack machine learning models, itwas shown later that simple heuristics such as the loss [Yeom et al., 2018, Sablayrolles et al., 2019]provide a more accurate and robust measure of membership inference.
Recent works [Watson et al., 2022, Carlini et al., 2021, Rezaei and Liu, 2021] have proposed toevaluate membership inference by the precision/recall trade-off [Watson et al., 2022] or the precisionat low levels of recall [Carlini et al., 2021].
In particular, such works show that some setups whichwere thought to be private because the membership accuracy is signiﬁcantly less than 100% canactually reveal membership of a small group of samples with very high precision.
There is also a line of work developed to design algorithms to speciﬁcally defend against membershipinference attacks.
Although differential privacy would bound the membership inference, these empirical defenses are potentially able to achieve better utility while withstanding existing membershipinference attacks.
Membership inference and total variation
In this section, we deﬁne our security game for membership inference and show its connection to thenotion of total variation distance (or statistical distance) between probability distributions.
3.1 Security Game
We adopt the classical assumptions of membership inference [Yeom et al., 2018, Sablayrolles et al. ,2019, Humphries et al., 2020].
In particular, for the remainder of this paper, we assume that thedatasets differ by adding/removing one element.
Saeed: Talk about federated learning and “transcript"+ fact that the learner is adversarial
Adversary picks a datasets D
= { z 1 , . . .
, z n } and a data point z (cid:48)
Challenger samples a bit b uniformly at random and creates D (cid:48) =
(cid:26) D ∪ { z (cid:48) } if b =D if b =3.
Challenger runs the SGM on D (cid:48) for T steps with transcript s = ( s 1 , . . . , s T )4.
Adversary observes the transcript s and guesses a bit b (cid:48) .
Adversary wins if b (cid:48)
= b .
We deﬁne the advantage of adversary A on learning algorithm L as Adv ( L, A )
= 2 · Pr[ b = b (cid:48) ] − 1 .
We also use Adv ( L ) =
sup A Adv ( A, L ) to denote the advantage of any adversary against L .
Remark 1.
In Section 4 we prove bounds on the membership advantage for composition of Gaussianmechanisms with and without sub-sampling.
Both of these bounds are tight for the advantage deﬁnedbased on addition/removal.
However, for the notion of advantage deﬁned based on replacement, ifwe double the bounds, only the bound without replacement will remain tight.
We leave the questionof obtaining tight bounds on the advantage based on replacement for the sub-sampled Gaussian asan open question.
Let X := s | ( b = 0) and Y :=
s | ( b = 1) .
A deterministic adversary
A deﬁnes a region A and predicts that θ is sampled from µ if s ∈ A and from Y if s / ∈ A .
We use µ ( A ) and ν ( A ) to denote Pr[ X ∈ A ] and Pr[ Y ∈ A ] .
For such an adversary we have
[Equation]
Note that with a simple averaging argument we can show that the best adversarial strategy inmembership security game is a deterministic strategy.
Therefore, the advantage for the learningalgorithm L is then deﬁned as
[Equation]
where TV is the total variation distance.
Therefore, total variation distance gives us an upper boundon the advantage of any adversary.
However, it is not clear how to calculate the total variation distancein general. In next subsection we discuss an approximation of total variation distance that can actuallybe calculated using existing techniques for RDP accounting.
3.2 Bounding
membership inference using Pinsker’s inequality
Using Equation 3 and directly applying Pinsker’s inequality (1), we have:
[Equation]
where D α is the Renyi divergence at α .
Now one might ask why this bound is preferable to ourbound using total variation distance. The reason we state this bound is that we have techniques forcalculating the Renyi divergence of composition of adaptive and sampled Gaussian mechanisms. This enables us to calculate numerical upper bounds on the membership inference advantage of anyadversary against adaptive composition of sampled Gaussian mechanisms, e.g. DP-SGD.
To this end, we use RDP accounting to calculate the D α for an α > 1 .
We know that for any α > 1 ,
D α
( X, Y ) is greater than KL ( X (cid:107) Y ) because D α is increasing in α .
This means that for any α > {+
+}
D α will be a valid upper bound on the membership inference advantage and the bound becomesbetter as we decrease α .
In Figures 1(a) and 1(b) we calculate numerical upper bounds for membership inference for DP-SGDusing typical parameters.
We refer to this bound as the Pinsker bound.
The ﬁgure shows that thisbound obtains better numerical values compared to the bound of Humphries et al.
However,there are two main limitations with our Pinsker bound: 1)
The bound is not tight.
We are applying
Pinsker’s inequality which is not optimal for the Gaussian mechanism.
It provides vacuous boundsin cases where KL ( X (cid:107) Y ) > 2 .
In next section, we will optimally bound the membership inferencefor composition of adaptive and sampled gaussian mechanisms.
Membership inference bounds for composition of Gaussian mechanisms
In this section, we show a tighter upper-bound for the adversary’s advantage.
Our main resultsare Theorem 4 and Theorem 5. In particular, we will upper-bound the total variation between thetranscripts of the (subsampled) Gaussian mechanism, speciﬁcally the noisy gradients produced by the DP-SGD algorithm.
The upper bound on the result of the DP-SGD algorithm follows by applicationof post-processing.
The proof technique is the following: we show that the entire process of DP-SGD can be replacedby a process where each step is replaced by either N (0 , σ 2 ) or N ( r, σ 2 ) for the basic Gaussianmechanism, and replaced by (1 − q ) N (0 , σ 2 ) + q N ( r, σ 2 ) for the subsampled Gaussian mechanism.
Our proof technique relies on a modiﬁed version of TV , called TV a .
Deﬁnition 2.
For a > 0 deﬁne TV a ( X, Y ) = 1 2 (cid:82) Ω | P ( x ) − a · Y ( x ) | dx.
While we do not know an explicit form of the TV a divergence between Gaussians, we can still provethat it increases with (cid:107) u 1 − u 2 (cid:107) 2 , as formalized in the following lemma.
Lemma 3.
Let X ≡ N ( u 1 , σ · I d ) and Y ≡ N ( u 2 , σ · I d ) .
Then, for any a ∈ R + , TV a ( X, Y ) is only a function of (cid:107) u 1 − u 2 (cid:107) 2 and σ .
Moreover this function is monotonically increasing with respect to (cid:107) u 1 − u 2 (cid:107) 2 .
That is,
[Equation]
2 , σ )) . The proof is deferred to Appendix A.4.1 Warm up: Without sampling
Notation.
We use s = ( s 1 , . . .
, s T ) to denote the output (or transcript ) of a random process thatconsists of T adaptive steps (typically the subsampled Gaussian mechanism).
We use s ≤ t to denotethe ﬁrst t steps of the transcript of the random process.
The sampling rate q is the probability of including any sample in the random set S .
We use S i to denote the union of the support set of the i thstep of the mechanism M on all possible datasets.
That is S
i
=
{ s i ; ∃ D, s
i ∈ Supp ( M ( D ) i ) } .
Wealso deﬁne S ≤ i =
S 1 × . .
, S i .
Theorem 4 (Gaussian Composition without sub-sampling) .
Let M 1 , . . .
, M T be a series of adaptive
Gaussian Mechanisms with L 2 sensitivity r and Gaussian noise with standard deviation σ .
Themembership inference risk of the composition of M i ’s is at most as much as a single Gaussianmechanism with sensitivity √ T · r and standard deviation σ .
[Equation]
Proof.
Let M i
( D ) be the mechanism that works on D (resp.
D (cid:48) ) and consists of i Gaussians
N (0 , σ 2 ) (resp.
N ( r, σ 2 ) ) steps followed by T − i steps of DP-SGD applied to D (resp.
D (cid:48) ).
Speciﬁcally, the mechanism M 0 corresponds to DP-SGD, and M T to pure Gaussians.
We will showin this proof that
[Equation]
and hence
[Equation]
To this end, we will ﬁrst argue that the very ﬁnal step of the mechanism M i can be replaced with a
Gaussian step without increasing the total variation distance.
Then we can move this noise to thestart of the process without affecting the result, obtaining M i +1 .
Figure 2 illustrates this procedure.
Let us ﬁx a step
i
and let s =
( s 1 , . . .
, s T ) be a transcript.
Denoting X ∼ M
i
( D ) and Y ∼ M i
( D (cid:48) ) ,we have
[Equation]
We know that the last ( T ’th) step of M i ( D ) (resp.
M i ( D (cid:48) ) ) has two isotropic Gaussian distributionscentered around two points u 1 and u 2 such that (cid:107) u 1 − u 2 (cid:107) 2 ≤ r and with standard deviation σ .
Thesecenters could be chosen adaptively according to the history of the mechanism.
We use a ( s ≤ T − 1 ) todenote Pr[ Y ≤ T −
= s ≤ T − 1 ]
Pr[ X ≤ T − 1 = s ≤ T − 1 ] .
By Lemma 3 we have(cid:88)
[Equation]
T
[Equation]
Denoting by N i the mechanism that coincides with M i for the ﬁrst T − 1 steps and is replaced by a
Gaussian at the last ( T ’th) step, we thus have, by following Equation (7) in the reverse direction
[Equation]
Given that the last step of N i does not depend on the ﬁrst T − 1 steps, we can permute to put it in ﬁrstposition (see Figure 2), which shows that TV ( N i ( D ) , N i ( D (cid:48) ))
=
TV ( M i +1 ( D ) , M i +
( D (cid:48) )) .
4.2 With sampling
Notation.
We use B ( q ) to denote a Bernoulli random variable that is equal to 1 with probabilityq and 0 with probability 1 − q .
We use B ( q ) n to denote an n dimensional random variable whereeach coordinate is independent and distributed as B ( q ) .
For r ∈ R , we use r · B ( q ) n to denote therandom variable that is sampled by the following process: ﬁrst sample from B ( q ) n and then multiplyby r .
We also use N ( r · B ( q ) n , σ ) to denote an n dimensional random variable that is distributedaccording to a mixture of Gaussians which all have standard deviation σ and centers are chosen atrandom from r · B ( q ) n .
Theorem 5 (Gaussian Composition with sub-sampling) .
Let M 1 , . . .
, M T be a series of adaptive
Gaussian Mechanisms with L 2 sensitivity r and Gaussian noise with standard deviation σ and sub-sampling rate q .
The membership inference risk of the composition of M i ’s is at most
[Equation]
Lemma 6.
Let X (cid:48) ≡ (1 − q ) · Y + q · X .
Then we have
[Equation]
X
[Equation]
Y
[Equation]
Proof. We have [Equation] Proof of Theorem 5.
The proof steps are similar to Theorem 4.
First, we have2 TV ( X, Y ) = (cid:88) [Equation] But since X T and Y T are subsampled Gaussian mechanisms we have X T ≡ (1 − q ) Y T + qX (cid:48) T where
Y and X (cid:48) are mixtures of Gaussians. Therefore, by Lemma 3 and Lemma 6 we have TV ( X, Y ) = (cid:88)
[Equation] Pr [Equation] By Lemma 6)
[Equation]
[Equation] Lemma 6)
[Equation] Therefore, we can replace X T with a mixture of two Gaussians centered at 0 and r and Y T with a single Gaussian centered at 0 . Now we can use the same technique used in proof of Theoremand move X T and Y T to the ﬁrst round and repeat this process.
At the end, Y is replaced by a n -dimensional Gaussian centered at 0 and standard deviation σ , and X by a mixture of Gaussians with center randomly selected according to a n -dimensional Bernoulli distribution with probability q . That is, the advantage is bounded by
[Equation] Remark 7. Theorem 4 could be simply extended to composition of Gaussian mechanisms withvarying noise levels. However, if the noise levels are selected adaptively, the proof is not clear.
Weleave the composition of Gaussians with adaptive noise selection as an open question.
Remark 8. Although Theorem 4 is stated only for Gaussian mechanism, the Theorem extends toany mechanism that satisﬁes monotonicity under TV a according to some notion of sensitivity.
Forexample, if one can show that TV a ( L (0 , σ ) , L ( u, σ )) is monotonically increasing with respect to| u | 1 , then Theorem 4 extends to composition of Laplace mechanisms with bounded (cid:96) 1 sensitivity.
4.3 Numerical computation
In order to numerically approximate the upper-bound, we ﬁrst convert the notion of TV into anexpectation formulation as follows:
[Equation]
Note that this expectation is over distribution X . So we can sample a dataset from X and approximatethis expectation using empirical averaging (or Monte-Carlo sampling):
[Equation]
Where t i ∼ i.i.d. µ .
We know that Monte-Carlo estimation of this quantity can be very accurate, becauseit is bounded between 0 and 1 .
Note that in-order to calculate this, we need to calculate Y ( t i ) /X
( t i )and that is possible because we have the mathematical form of the probability distribution functionof for Y and X .
In Figures 1(a) and 1(b) we calculate the upper bound using this Monte-Carlosimulation for typical settings in DP-SGD.
[-For the CIFAR-10 experiments, we use 5-layer CNNs and trained them for 50 epochs with sampling rate 0 . 02 and learning rate 0 . 5 using the cross-entropy loss. We set the gradient clipping threshold to 10 . 
-]Proposition 9.
Algorithm 1 return a value η that with high probability is close to the optimalmembership inference bound η ∗ of Theorem 4:
[Equation]
Evaluation
In this section we perform experiments to calculate concrete trade-off between accuracy and membership inference on mainstream datasets.
Conclusion
In this paper, we directly analyzed membership inference bounds for adaptive composition of sampled
Gaussian mechanisms.
Our analysis enables us to obtain bounds that are better than bounds fromprior analyses of differential privacy.
Our analysis shows that although differential privacy guaranteesmight sometimes lead to large/vacuous membership inference guarantees, the mechanisms that obtain Require: sample rate vector q , number of iterations T , learning rate vector η , noise scale vector σ , gradient norm clip vector r , sample size S 1: for all j ∈ [ S ] do 2: η j ←3: for i ∈ [ T ] do 4: for j ∈ [ S ] do 5: sample t i,j ∼ N (0 , σ [ i ] 2 · r [ i ] 2 · I ) 6: η j ← η j +ln (cid:0) 1+ q [ i ]( e 2 ti,j/r
[ i ] − 1 2 σ [ i ] − 1) (cid:1) 7: η ←8: for j ∈ [ S ] do 9: η j = max( η j , 0) . 10: η ← η + 1 − e ηj S 11: return η Require: training dataset D , sample rate vector q , number of iterations T , learning rate vector η , noise scale vector σ , gradient norm clip vector r , loss function L 1: Initiate θ randomly 2: for i ∈ { T } do 3:
B i ← Sample batch via Poisson sampling with rate q
[ i ] 4: ∇ [ t ] ← (cid:126)5: for all
( x, y ) ∈ B t do 6: ∇ ( x,y ) ← gradient of L ( x, y ) 7: ∇ ( x,y )
← r [ t ] · ( ∇ ( x,y ) ) max( r [ t ] , (cid:107)∇ ( x,y ) )
(cid:107) 2 ) 8: ∇
[ t ] ← ∇ [ t ] + ∇ ( x,y ) 9: (cid:103) ∇ [ t ] ← ∇ [ t ] + N (0 , σ [ t ] 2 r [ t ] 2 I ) 10: θ ← θ − η (cid:103) ∇ [ t ] .
11: return θ differential privacy can be in fact much more secure against membership inference attacks.
Previously,this phenomenon was observed for DP-SGD and here, for the ﬁrst time, we prove it.
Our analysis is the ﬁrst to directly analyze membersihp inference bounds.
We limited our studyto membership inference attacks against sampled Gaussian mechanisms as DP-SGD is the mostwidely used differentially private learning algorithm.
Direct study of membership inference could bepotentially done for other mechanisms and algorithms as well.
We leave this for future work.
We also note that the hyper-parameters for DP-SGD that achieve optimal membership privacy versusutility might be different than that of differential privacy.
Our new analysis opens up the possibilityof a systematic search for the hyper-parameters and architectures (using recent advancements inprivacy preserving hyperparameter tuning and architecture search Papernot et al. [2020], Papernotand Steinke [2021], Chaudhuri and Vinterbo [2013]) that obtain optimal utility for a given upperbound on membership inference. Checklist The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] . You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description. For example: • Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[TODO] (b) Did you describe the limitations of your work? [TODO] (c) Did you discuss any potential negative societal impacts of your work?
[TODO]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[TODO]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[TODO]
(b) Did you include complete proofs of all theoretical results?
[TODO]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[TODO]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[TODO]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[TODO]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[TODO]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[TODO]
(b) Did you mention the license of the assets?
[TODO]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Table]
[TODO]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[TODO]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[TODO]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[TODO]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[TODO]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[TODO]
[Table]
A Proof of Lemma 3
Proof.
The ﬁrst part follows by the symmetry of isotropic Gaussian.
For the second part (monotonicity) we use the deﬁnition of TV a .
Without loss of generality we can assume a ∈ [0 , 1] as otherwisewe can work with TV a ( P, Q ) /a
=
TV 1 /a
( Q, P ) .
Let r = (cid:107) u 1 − u 2 (cid:107) 2 .
We can show that thederivative of the integral is always positive.
In the following calculations, we use c 1 , c 2 , c 3 and c 4 todenote positive constants that are independent of r .
First note that x ∗ =
r 2 − 2 σ 2 ln( a ) 2 r is a middle point where e − x 2 2 σ 2 − ae − ( x − r )2 2 σ 2 goes from positive tonegative as x increases.
By our assumption that a ∈ [0 , 1] , we have that x ∗ > 0 .
Recalling thaterf( z )
= 2 √ π (cid:82) z 0 exp( − t 2 ) dt , and that erf( ∞ )
{+= 
+}
{+so that (by symmetry) 2 √ π (cid:82) 
+}
−∞ exp( − t 2 ) dt = 1 ,we can write
[Equation]
Now, let f 1 ( r ) = erf (cid:16) r 2 − ln
( a ) σ 2 2 √ 2 σr (cid:17) and f 2 ( r ) = − a erf (cid:16) − r
[Equation]
− ln(
[Equation]
2 √ 2 σr (cid:17) .
Taking the derivative withrespect to r we have
[Equation]
Now note that we have e −
[Equation]
−
[Equation]
[Equation]
ln
[Equation]
a
[Equation]
σr (cid:19).
Therefore, we have
[Equation]
Now since a ∈ [0 , 1] , we have ln( a ) ≤ 0 and √ a − 1 < 0 , which means the term 1+ √ a 2 √ 2 σ + ln( a )( √ a − 1) σ 2 √ 2 r 2is positive.
This implies that the whole gradient is positive.
B Estimating KL with Rényi divergence
For two probability distributions µ and ν , the Rényi divergence of order α > 1 is
[Equation]
[Equation]
Rényi divergence is non-decreasing in α , and lim α → 1 D α ( P (cid:107) Q
[Equation]
= KL
[Equation]
P
[Equation]
Q ) .
RDP accounting for DP-SGD.
[Abadi et al., 2016, Mironov, 2017] propose methods to accountfor RDP for the Gaussian mechanism.
Implementations of DP-SGD such as Opacus make use ofthese accounting procedures.
This is important as we use these accounting methods to calculate oneof our bounds for membership inference for composition of sub-sampled Gaussian mechanims.
C Membership inference precision
In this section, we reﬁne the analysis of Sablayrolles et al.
[2019] for the accuracy of a membershipattack.
Upper-bound on precision.
Let us ﬁrst derive a bound on the precision of membership inference.
We assume that there are two datasets D and D (cid:48) and that a differentially-private mechanism M trainsa model represented by θ .
With probability (1 − δ ) over the choice of θ , we have:
[Equation]
Given that there is a balanced prior P ( D )
=
P ( D (cid:48) ) , using Bayes rule, we have:
[Equation]
[Equation]
with σ ( u ) = 1 / (1 + exp( − u )) the sigmoid function.
Hence the precision P ( D | θ ) is bounded between σ ( − (cid:15) ) and σ ( (cid:15) ) , as σ ( · ) is non decreasing.
Upper-bound on attack accuracy.
The accuracy of the Bayes classiﬁer is
[Equation]
and thus
[Equation]
This means that the attack accuracy is bounded by σ ( (cid:15) ) with probability 1 − δ .
Empirically, wesee that the sigmoid function closely matches the bound given by Humphries et al.
Simplystated, this derivation shows that the bound proven by Humphries et al.
[2020] actually holds withprobability 1 − δ instead of on average.