A General Framework for Auditing Differentially Private Machine Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We present a framework to statistically audit the privacy guarantee conferred by adifferentially private machine learner in practice.
While previous works have takensteps toward evaluating privacy loss through poisoning attacks or membershipinference, they have been tailored to speciﬁc models or have demonstrated lowstatistical power.
Our work develops a general methodology to empirically evaluatethe privacy of differentially private machine learning implementations, combiningimproved privacy search and veriﬁcation methods with a toolkit of inﬂuence-basedpoisoning attacks.
We demonstrate signiﬁcantly improved auditing power overprevious approaches on a variety of models including logistic regression, Naive
Bayes, and random forest.
Our method can be used to detect privacy violations dueto implementation errors or misuse.
When violations are not present, it can aid inunderstanding the amount of information that can be leaked from a given dataset,algorithm, and privacy speciﬁcation
Introduction
Machine learning (ML) is increasingly deployed in contexts where the privacy of the data used tobuild the model is of concern.
ML models are capable of ingesting enormous quantities of datain order to determine meaningful predictive patterns.
When such models are built on potentiallysensitive inputs such as healthcare [1] or ﬁnancial data [2], it becomes important to determine towhat extent information from the training dataset can be inferred from the model.
Such informationcan be of personal interest when the identity of a data contributor may be compromised based oninvolvement in the data collection process [3, 4], but it can also lead to regulatory concern aboutwhether a model trained on private data should itself be considered private [5].
Differentially private (DP) machine learning is a theoretically rigorous approach that provides a worstcase privacy guarantee for ML algorithms [6].
Speciﬁcally, it provides a mathematical guaranteethat the distribution of possible output models based on the input dataset is not signiﬁcantly alteredwhether an individual data point is included in the training set or not.
This is accomplished byrandomizing the output model in a speciﬁc manner so that the distributions of the model trainedwith or without any individual data point are statistically indistinguishable up to a level speciﬁed byparameters " ,  that must be set appropriately.
While DP mathematically certiﬁes the privacy of amechanism, the noise added is generally tailored to the worst case scenario.
Previous results attempting to assess the privacy leakage of such models, using attacks ranging frommembership inference to model inversion, have inferred that the actual detectable risk of a modelprocedure is lower than the theoretical bound [7, 8].
While a recent study on DP neural networksindicates that the privacy guarantee of the DP-SGD algorithm is tight under the strongest adversary[9], the privacy risk under realistic datasets and attackers without complete access to the trainingprocess still lies below the worst case bound.
Across other models, the question remains of whetherthe privacy violation of a mechanism is lower than speciﬁed or whether the attacks previously used are not strong enough.
Given that there are no guidelines on how to set " ,  in practice, it wouldbe useful for practitioners to estimate the actual risk they incur in practice and set the parametersaccordingly.
This is critical when DP for ML is still a nascent ﬁeld, with limited implementationsthat can be hard to verify.
Recent works have shown that simple DP mechanisms can be audited effectively [10, 11].
Thesemechanisms generally map an input vector to a perturbed output vector, and are often used as buildingblocks of more complicated DP algorithms such as machine learning models.
Examples include the
Laplace and exponential mechanisms which are respectively used in DP Naive Bayes and randomforest [12, 13].
The auditing procedure involves searching for optimal neighboring input sets a, a 0and sampling the DP outputs M ( a ) , M ( a 0 ) , to get a Monte Carlo estimate of " .
To extend thesetechniques to audit a machine learning model, the vector inputs a, a 0 are replaced with datasets
D, D 0 which differ by a single entry.
Working in this realm raises important challenges.
First,previously effective search methods for neighboring inputs involving enumeration or symbolic searchare impossible over large datasets, making it difﬁcult to ﬁnd optimal dataset pairs.
In addition,
Monte Carlo estimation requires costly model retraining thousands of times to bound " with highconﬁdence.
We propose solutions to these problems to accommodate the general auditing approach to machinelearning algorithms.
For the ﬁrst concern, we develop a set of data poisoning attacks which perturba single point of dataset D , approximating a worst-case neighboring dataset D 0 speciﬁc to M .
Toaddress the second, we present a statistical framework to estimate " for general ML procedures forsmaller sample sizes .
Our techniques involve improved estimation bounds and, more signiﬁcantly,learning probabilities over general model spaces , while previous works can only estimate probabilitiesover a single output (the predictive probability).
These novel contributions form our framework ML-Audit for auditing the DP of arbitrary machinelearning models.
Compared to prior works we can obtain orders of magnitude improvement in theestimation of " that are effective across a larger range of " , more datasets, and more model types.
Ourframework is compatible with prior model-speciﬁc approaches (small neural networks), obtainingsimilar or improved efﬁcacy.
Further still, we provide case studies where our framework detectspotential violations in existing implementations of DP machine learning.
We review these prior approaches in section 2, and introduce key DP concepts in section 3.
Thenwe present our framework in section 4, followed by results and discussion in section 5.
Finally weconclude in section 6.
Related work
Although differential privacy of a mechanism is established with mathematical proof, previouswork has shown that holes can exist, either in theory, such as the sparse vector mechanism (ofwhich erroneous versions have been proposed [10]), or implementation, for example samplingnon-uniformity in pseudo-random number generators [14].
Such occurrences point to the need forempirical veriﬁcation of the promised security of differential privacy mechanisms.
This is criticalas lapses in DP are likely to provide greater predictive performance, so choosing a model to obtainmaximal accuracy given a chosen level of privacy is likely to select these errant models.
Recent works propose efﬁcient solutions for auditing simple differential privacy mechanisms operatingon scalar or vector inputs [10, 11].
In such cases, the neighboring inputs can be enumerated tractably(e.g.
for a vector input, trying adding one to each input element).
For each neighboring pair, theappropriate output set is determined, and Monte Carlo probabilities are measured to determine privacy.
The sampling process in such mechanisms is vectorized to greatly reduce the runtime [11].
Few studies have attempted to audit DP machine learners.
We are aware of two works that measure theprivacy of neural networks trained with the DP-SGD mechanism [15].
Jagielski et al.
[16] develops apoisoning attack (ClipBKD) by constructing a data point along the axis of least variance in the dataset.
This causes the gradients of the point to be distinguishable, mitigating the effect of the DP-SGDclipping and noise. Since this attack is model-agnostic, we adopt it as a baseline for all our models.
We show considerably improved performance with our method and that the least variance approachsuffers as the sphericity of the dataset increases.
The second work analyzes DP-SGD through a series of increasingly white-box attacks Nasr et al.
They note that while the DP-SGD privacy guarantee is formulated for all neighboring datasets, the mechanism itself operates purely via gradient manipulation.
Therefore they are able to formulatestronger attacks which directly target the gradient or make use of adaptive poisoning.
While theseare unrealistic for general-purpose ML auditing, we implement their poisoning attack, based onadversarial perturbation, for our DP-SGD experiments.
We ﬁnd that this attack performs comparablywith ClipBKD on DP-SGD.
While their works are important progenitors to ours, they estimate the privacy loss by thresholding asingle scalar output, e.g. the loss on a speciﬁc poisoning point.
This is a major bottleneck limitingthe ability to audit other machine learning algorithms.
Instead of a scalar, our approach learnsdistributions over any vector representation of a model, allowing us to be the ﬁrst to propose a generalprocedure for auditing any DP learner.
Another predecessor established the complementary relationship between data poisoning and differential privacy, using a poisoning attack based on gradient ascent of logistic regression parameters ✓with respect to x [17] to reach a target ✓ ? .
The result is identical to a perturbation inﬂuence functionwhich we discuss [18, 19].
However, the aim of their study is to evaluate the effect of differentialprivacy on poisoning strength rather than vice versa.
In addition, their attack formulation requiresmanual gradient computation which does not readily extend to other models.
Differential privacy overview
We provide a background on key elements of differential privacy relevant to our work.
Deﬁnition 3.1.
A randomized learner M : D ⇥ B 7! ⇥ is a function mapping a training dataset 110 D 2 D and auxiliary noise b 2 B to an output model ✓ 2 ⇥ .
If we train the learner repeatedly on the same dataset, we can obtain a different model ✓ each timebased on the noise source b .
For simplicity we implicitly denote the learner as a randomized function
M ( D ) with a probability distribution P induced by b .
Deﬁnition 3.2.
A randomized learner M is considered ( " ,  ) -differentially private if for all datasets
D, D 0 2 D differing in a single entry and measurable sets S over the output space ⇥ ,
[Equation]
In this deﬁnition, the " term is the primary quantity of interest because it bounds the ratio betweenmodel output probabilities when the original dataset D is perturbed by a single row.
Setting " =
= 0would enforce that M ( D ) equals M ( D 0 ) in distribution, while a large " or  permits the distributionsto be arbitrarily different.
As " !
1 the model distribution converges to M 1 ( D ) , which in 120 most cases is identical to the standard non-private version of the model.
The  term permits someadditive slack in the probability bound, but in most purposes is set to a very small quantity below
O (1 /n ) .
Given " and  , M scales the noise distribution according to the sensitivity of the model function[20].
Deﬁnition 3.3.
The sensitivity of a function f : D ! R d is max D,D 0 k f ( D )  f ( D 0 )
k where D, D 0 126 differ by one row.
Our main goal is to audit the privacy of a known mechanism whose inner workings may be hidden,so we assume in our work that the user has access to the mechanism, the privacy parameters, and thedataset, but does not have control over the training process or the randomness b .
The ﬁnal tool which we will employ in our framework is group privacy , which iterates over Eq.
1 tobound the probabilities when D, D 0 differ by multiple points.
Lemma 3.4.
Suppose M is ( " ,  ) -private and D, D 0 differ by k rows.
Then for all measurable
S ⇢ ⇥ ,
[Equation]
Auditing framework
A DP mechanism calibrates its randomization to a user-speciﬁed " th representing the desired theoretical level of privacy.
The goal of DP auditing is to empirically assess the actual privacy enabled by
S S
Dataset
D
D’
M p e r t u r b a ti on
N samples
M ( D ) t
P ( M ( D 0 ) 2 S ) =
c
M ( D 0 ) p ( D | z ) the mechanism, " ? .
To do this, we determine a lower bound for " ? with high statistical conﬁdence.
Observe that rewriting the deﬁnition of DP gives
[Equation]
For any given ( D, D 0 , S ) we can estimate the quantity with Monte Carlo by retraining M to generatesamples and then compute a 95% conﬁdence interval.
The lower bound of the interval ˆ " lb is thedetected privacy violation, and with high probability, we state that " ?  ˆ " lb with ( D, D 0 , S ) as awitness.
By maximizing ˆ " lb over all possible witnesses, we increase the lower bound on " ? which wecan then compare against the promised " th .
Given a mechanism M , the auditing objective is then toﬁnd
[Equation]
that is, the maximum privacy loss ratio over neighboring datasets D, D 0 and measurable output set
S .
For all mechanisms considered in this work  < 10  4 which is not detectable, so we simplify bysetting  = 0 .
A successful auditing approach requires solutions to the two separate maximizations.
We maximizeover D, D 0 by developing a toolkit of poisoning attacks which have the largest inﬂuence on themechanism in question.
Once D and D 0 are obtained, we ﬁt a likelihood ratio-based optimization todetermine an appropriate S .
In the following, we discuss these procedures in detail.
4.1 Optimizing D, D
Given a dataset D =
( X, Y )
we deﬁne a valid poisoning attack to be an operation which selectsan existing data point ( x, y ) and perturbs x to any new point x ? within a constraint set C .
As weconsider classiﬁcation algorithms in this work, the class label y can also be switched.
The constraint
C is speciﬁc to each DP algorithm and is used to determine the max sensitivity bound from Def.
3.3(as otherwise a single data point can arbitrarily affect the model).
Without loss of generality, we setthe constraint to be the smallest containing the training data.
This is the only part of the framework which needs user design, as a strong attack against onemechanism may not be effective against another.
However we distill the steps for constructing suchattacks into a recipe:
The underlying model of M can be abstract, requiring a summary function ⌧ to embed M into
a vector space.
Since post-processing of DP outputs does not decrease privacy [6], we can select⌧ to preserve as much information about M as possible.
In logistic regression the transformationis natural: we deﬁne ⌧  M as the coefﬁcient vector ✓ (Table 1).
To reduce notation overload
we
implicitly apply ⌧ when we discuss empirical samples z ⇠ M ( D ) .
Obtain a non-private algorithm M 1 ( D ) as a surrogate.
Note that as long as E b [ M ( D, b )] ⇡M 1 ( D ) , meaning the DP model is on average equivalent to the non-private version, it is sufﬁcient
to ﬁnd the optimal poisoning with respect to the non-private model.
While not explored in thiswork, attacks can instead take into account the actual noise distribution by averaging over samples ofM ( D, b ) , at the cost of additional run-time.
Determine a poisoning ( x ? , y ? ) from ( x, y ) maximizing the distance between ⌧ ( M 1 ( D )) and⌧ ( M 1 ( D 0 )) .
This involves a selection and a perturbation step.
Bound C
Summary ⌧
Log.
reg.
L 2 -norm coefﬁcient ✓
NB hyper-rectangle µ yd ,  2 yd , ⇡ y
RF hyper-rectangle
Pr( x ? ) of each tree
DP-SGD
L 2 -norm of r ✓ L ( x, y ; ✓ )
Pr( x ? ) , Pr( ~ 0) , Pr( x testi )
Ahead we detail the poisoning attack design for each mechanism:
Logistic Regression.
DP logistic regression is achieved using one of three approaches: objective,output, or gradient perturbation.
We evaluate objective and output perturbation using the widely usedmethod developed by Chaudhuri and Monteleoni [21] and Chaudhuri et al.
For the attack we adopt the inﬂuence function, a technique which estimates the change in a modelwhen a speciﬁc data point is inﬁnitesimally upweighted in the training set [19, 23].
For a generalmodel with gradient and Hessian information, the effect of point x ? on the model parameters fora loss function L can be approximated as I ✓ ( x ? ) =
H  1 ✓ r ✓ L ( x ? , ˆ ✓ ) .
In generalized linearmodels with canonical link function where L is the negative log likelihood, this has a closedform involving the Fisher information as the expected Hessian [18], which for logistic regressionevaluates to I ✓ ( x ? ) =
( y ?  ˆ y ? )
( X > WX +  I )  1 x ? where W is a diagonal matrix with entries
W ii =
(ˆ y i (1  ˆ y i ))  1 , and  regularization.
We choose ⌧ to be the coefﬁcient vector ✓ , ✓ 0 of M 1 ( D ) , M 1 ( D 0 ) respectively.
We ﬁrst select 187 ( x, y ) closest to the corner of the hyper-rectangle containing the data (a heuristic for selecting a pointfar from where the separating hyperplane would likely be) and initialize ( x ? , y ? ) as the mean point inthe opposite class.
Since our goal is to increase the distance k ✓  ✓ 0 k 2 , we optimize the L 2 norm 190 of the inﬂuence function using projected gradient ascent.
The constraint set C is the L 2 norm ball 191 containing the training set, so after each iteration we clip x ? if needed.
Naive Bayes.
The Gaussian Naive Bayes model parameterizes the class-conditional distribution offeatures using independent Normal distributions.
The parameters involve a mean µ yd and variance 2 yd for each feature and class combination, as well as prior probabilities for each class ⇡ y .
To achievedifferential privacy, Laplace or Geometric noise are added to the maximum likelihood estimates foreach parameter [12].
The constraint set C is a hyper-rectangle set by the smallest and largest value ofeach feature in the dataset.
We choose ⌧ to be the vector of all { µ yd ,  2 yd , ⇡ y } .
The maximum inﬂuence a perturbation can haveon ˆ µ y of class y is by selecting a point on the corner of the hyper-rectangle C nearest the data pointsof class y and placing it on the corner furthest away.
On the other hand, the maximum inﬂuenceattack on ⇡ y is to ﬂip a class label.
We combine the two ideas by ﬂipping the class label of the pointclosest to a corner.
In our experiments we compare the power against only attacking µ y .
Random Forest.
The DP random forest mechanism uses unsupervised random splits of the featuresbased on the domain of the inputs.
Each tree is split to a pre-determined depth.
Then the trainingpoints are percolated through the forest, and the majority label of each leaf is sampled from the
Exponential mechanism [13].
Thus a perturbed training point has no impact on the actual treestructure besides the potential label of a single leaf in each tree.
As a result we measure only thechange in those leaves, choosing ⌧ to be the prediction of each tree on x ? .
The most detectable change in probability occurs when j = 1 and then the majority class is swappedby a class ﬂip.
For example, if a leaf has one positive and two negative points, we ﬂip the class ofone of the negatives.
Since each tree is random, there is no guarantee that any given point will bein such a situation where j = 1 , but to increase this chance we target points which are likely tobe solitary in each leaf.
Thus we select the point most distant from the rest of the training set asmeasured by L 1 -distance and ﬂip its label.
We provide further exposition and hyperparameter detailsin the Appendix.
DP-SGD.
For DP-SGD we evaluate two extent poisoning attacks using our framework.
The ClipBKDattack constructs a data point along the axis of least variance in the dataset [16].
This can be performed by taking the eigenvector of the covariance matrix with the smallest eigenvalue, and then projecting itto the median L2 norm of the training set.
This point is assigned the label with smallest predictedprobability.
The goal of this attack is to enable the gradients of the point to be distinguishable,mitigating the effect of the DP-SGD clipping and noise.
The second attack selects a random datum and maximizes the loss with gradient ascent.
The loss isdeﬁned with respect to a set of trained shadow models [9].
In both attacks, the outcome ⌧ ( M ( D ))being measured is the predicted probability of the perturbed point P M ( x ? = 1) , while ClipBKD ﬁrstsubtracts the zero vector’s predicted P M ( ~ 0) .
We also consider an updated ClipBKD with the zeroprediction encoded separately with additional test predictions in ⌧ : P M ( ~ 0) , P M ( x testi ) .
4.2 Optimizing S
Algorithm
Require: Datasets D, D 0 , DP mechanism M , min.
probability constraint r , conﬁdence ↵ , sample size N 1: Sample z 1 , . . .
, z N ⇠ M ( D ) , z 0 1 , . . .
, z 0 N ⇠
M ( D 0 ) 2: Generate labels y i
=
if z i , else y i =
if z {+
+} i 3: Train classiﬁer on { z, z 0 } , { y } to learn p ✓ ( D | z ) 4: p 1 , . .
, p N p ✓ ( D | z 1 ) , . . .
, p ✓ ( D | z N ) 5: p 0 1 , . . .
, p 0 N p ✓ ( D | z 0 1 ) , . .
. , p ✓ ( D | z 0 N ) 6: ˜ " lb 1 ; ˜ t 7: for t in sort( { p i } N 1 [ { p 0 i } N 1 ) do 8: n P i ( p i > t ) 9: n P i ( p 0 i > t ) 10: if n 0 /N <
r then . Avoid estimating probs < r 11: skip to next iter 12: " tlb Eps _ LB( n 1 , n 0 , N, ↵ / 2) .
Lemma A.
13: if " tlb > ˜ " lb then 14: ˜ " lb " tlb 15: ˜ t t 16: return S { z | p ✓ ( D | z ) > t }
Our approach for determining S draws from Bichsel et al.
[11], which derives the optimal region for distinguishing two vector distributions.
Given candidate datasets ( D, D 0 ) with DP distributions M ( D ) and M ( D 0 ) , we seek S maximizing P ( M ( D ) 2 S ) / P ( M ( D 0 ) 2 S ) . Let 234 f
M ( D ) and f M ( D 0 ) be their respective densities.
235 Selecting S to best distinguish two distributionsis analogous to identifying an optimal rejectionregion in hypothesis testing.
From the NeymanPearson Lemma (cf. Appendix), the maximumpower test at level c is deﬁned by rejection region S :=
{ z | f M ( D ) ( z ) /f
M ( D 0 ) ( z ) > k }for some k > 0 , where P ( M ( D 0 ) 2 S )
= c[24].
Intuitively, we have a likelihood ratiof M ( D ) ( z ) /f
M ( D 0 ) ( z ) and we are selecting 245 points to ﬁll S where D is more likely than
D 0 .
We start by adding points where D is mostlikely and work downwards until S is largeenough that there is a c chance
that M ( D 0 ) iserroneously in S .
Directly obtaining these densities is intractable.
Instead we can learn, given collected samples {M ( D ) } where D 2 { D, D 0 } , the posterior probability of the dataset D :
[Equation]
Let us consider the set L k :=
{ z | p ( D | z ) /p
( D 0 | z ) > k } .
From Bayes’
Theorem we know p ( D | z ) /
251 f
M ( D ) ( z ) ·
Pr( D = D ) .
Given equal samples of D from D and D 0 , then P ( D = D ) =
P ( D = D 0 )implying p ( D | z ) /p
( D 0 | z ) =
f M ( D ) ( z ) /f
M ( D 0 ) ( z ) .
Therefore S =
L k
and we use it as the rejection set.
Finally, we can further simplify S by observingthat p ( D | z ) + p ( D 0 | z ) = 1 , and so S =
{ z | p ( D | z ) > t } where t := k/ (1 + k ) .
What remains isto set t according to the level c so
that P ( M ( D 0 ) 2 S ) =
c .
As this is equivalent to P ( p ( D | z ) >t | z ⇠ M ( D 0 )) , we set t empirically to correspond to the (1  c ) -quantile of M ( D 0 ) .
Our approach introduces the following further adaptations on [11] to enable auditing on slower,potentially expensive machine learning mechanisms.
(The original work used around 10 8 samplesfor auditing simple mechanisms, which is infeasible for training machine learning models.
We use N = 10000 samples for all mechanisms except DP-SGD where N = 500 .)1)
We make the method ﬂexible by searching c empirically to maximize ˜ " lb , rather than settingc = 0 .
01 as in the original.
When N ! 1 , smaller values of c are optimal since the likelihood ratiois largest.
At smaller sample sizes, however, this is offset by increasing uncertainty in Monte Carloestimation of small probabilities.
We use group privacy to increase statistical distinguishability, a trick used also in [16].
Afterobtaining ( x ? , y ? ) , we insert k copies of the point.
From Lemma 3.4 the detected privacy ˜ " lb isreduced to ˜ " lb /k .
However, the extra copies causes M ( D 0 ) to be more distinguishable which makes P ( M ( D ) 2 S ) larger relative to P ( M ( D 0 ) 2 S ) .
This is also a tradeoff: At small true " the lowerbound ˜ " lb improves despite division by k , whereas at high " the distributions are sufﬁciently differentat k = 1 that it is not worth dividing ˜ " lb .
We include empirical analysis of k in our Appendix.3)
All previous auditing works (including below) use Clopper-Pearson intervals for ˆ p 1 and ˆ p 0 todetermine the lower bound.
This method is highly suboptimal because it separately bounds thenumerator and denominator.
We identify instead the Katz-log conﬁdence interval to directly boundthe ratio of binomial proportions [25], see Lemma A.3.
This has far better coverage properties in oursimulations (Appendix).
Our modiﬁed procedure is presented in Algorithm 1 and summarized visually in Fig.
Comparison to other approaches.
While our method allows M ( D ) to be any vector of information,the previous works auditing DP-SGD require M ( D ) to be a single value (e.g. prediction probabilityor loss) and search for a threshold dividing the score into S and S c [9, 16].
Our method can reduce tothis special case by choosing ⌧ to map to a scalar.
The approach can be interpreted as membershipinference with the bound FN + e " FP  1   from Kairouz et al.
[26], where {M ( D ) 2 S } are true 282 positives and {M ( D 0 ) 2 S } are false positives.
These works also note that ﬂipping the deﬁnition of 283 false positive and false negatives enables testing the complement P (
M ( D 0 ) 2 S c ) / P ( M ( D ) 2 S c ) .
As in those works we select S c instead of S if it gives higher ˜ " lb .
Corollary 4.0.1.
Given Algo.
1 and N samples, the highest detectable privacy risk is ln( N ) z ↵ /
[Equation]
Estimating ˆ " lb .
After ( D, D 0 , S, k ) are determined, the ﬁnal step is to draw N fresh samples of
M ( D ) and M ( D 0 ) for an independent Monte Carlo veriﬁcation.
As in the search phase, we computen 1 =[-(cid:80) i 
-]
P i ( M ( D ) 2 S ) , n 0 =
P i ( M ( D 0 ) 2 S ) and use the lower bound of the Katz log intervalto return our ﬁnal estimate ˆ " lb .
To avoid biasing the conﬁdence interval, it is important that the ﬁnal MC estimate is conducted on anindependent set of samples, as done in [11, 16].
This is especially the case when optimizing overthresholds in S .
Since we compute a 1  ↵ interval over ˆ " for each threshold, using the best resultfrom the same sample would lead to biased inference from multiple testing.
Experiments and Results
We evaluate the DP mechanisms over a range of " th : { 0 . 1 , 0 . 25 , 0 . 5 , 1 , 2 , 4 , 8 , 16 , 50 } .
At " th = 0 .
1the ratio of probabilities is bounded by e 0 . 1 ⇡ 1 .
1 giving nearly indistinguishable distributions,whereas at " th = 50 essentially no privacy is guaranteed.
For a given dataset D , we perturbk 2 { 1 , 2 , 4 , 8 } points to get D 0 and train N = 10000 times for each to determine the appropriateauditing set S .
Then we obtain N new samples to perform the ﬁnal Monte Carlo estimate and obtainthe lower bound ˆ " lb .
We use conﬁdence level ↵ = 0 .
05 throughout.
We assess Naive Bayes, logistic regression (output and objective perturbation), and random forest oncommon machine learning datasets: adult , credit , iris , breast-cancer , banknote , thoracic .
We usethe diffprivlib library [27] and implement output perturbation following [22].
Additionally, we test
DP-SGD on FMNIST and CIFAR10 (here with N = 500 ) using [28].
Refer to Appendix for moredetails.
Figure 2 directly compares, over three datasets, the ˆ " lb obtained by our poisoning attacks (ML-Audit),the ClipBKD attack, and a baseline where a random data point x is changed to a random point fromthe opposite class without changing labels (Swap-X).
(Note that while the original ClipBKD workuses DP-SGD, the poisoning itself is mechanism-agnostic, so it is an appropriate baseline.)
Wereplicated our procedure 3x and show the median ˆ " lb .
The Katz log interval was used for the lowerbounds even on the baselines.
As a reference we plot the theoretical privacy loss up to the maximumdetectable limit (Corollary 4.0.1).
Our attacks consistently and often dramatically outperform baselines across datasets, models, and" th .
In some cases, the privacy detection is within a small factor (1  2 ⇥ ) of optimal.
The Swap-Xbaseline shows the expected privacy loss of an attack which poisons a data point within the original [Figure] data distribution, rather than a worst-case perturbation (e.g. swapping two points in the trainingset).
The only case where Swap-X performs similarly to our tailored attacks is in output-perturbedlogistic regression on linearly separable datasets such as iris ; after ﬂipping a class label in iris ,
Swap-X becomes much weaker.
Our results validate previous ﬁndings showing that estimating "using membership inference-based bounds are weaker than poisoning attacks [9, 16].
We also triedthe loss-threshold attack of [7, 29] and found it ineffective.
On many models and datasets we obtain nearly optimal results, indicating that these datasets are closeto the maximal sensitivity of the mechanism.
We observe this for random forest at " th  2 but also aplateau in higher " th .
This is likely due to lack of resolution in the outputs: an average over m treedecisions only takes m unique probability values.
Our attack on Naive Bayes is also near-optimal,while varying among datasets.
Since the algorithm assigns " th / 3 to protect each of µ yd ,  2 yd , and ⇡ y ,a perturbation needs to achieve the sensitivity bound on all three to reach the theoretical privacy risk,with the bound over all pairs D, D 0 rather than conditioning on a starting D .
For example, to reachthe bound for µ , all points of one class must be in one corner of the hyper-rectangle and the poisonedpoint on the other corner, which is unrealistic.
For this reason, our choice of attacking the class prior⇡ is generally more effective than attacking the mean µ (Fig. 4).
Lastly, our DP-SGD experiments assess the two existing perturbation methods of [9, 16].
Ourﬁndings (Fig.
8) are consistent with what is reported in those works.
Interestingly, we ﬁnd that theshadow adversarial attack outperforms ClipBKD on FMNIST and vice versa on CIFAR10.
We alsoupdated ClipBKD to use additional information in ⌧ as detailed in Section 4 . 1 and ﬁnd a moderateimprovement on FMNIST.
Refer to the Appendix for further discussion.
Dataset-speciﬁc privacy.
Previous work hypothesized that privacy attacks on realistic datasets donot cause the model shift to reach the worst-case given by the sensitivity bound, except when theattacker can completely specify the dataset [9].
Based on our observations on the impact of thedataset distribution on attack strength, we agree that achievable privacy is dataset-dependent.
Webelieve theoretical analysis of datasets for attack risk to be useful for future work.
We brieﬂy investigate this phenomenon by comparing the performance of ClipBKD by datasetsphericity, as computed by ratio between largest and smallest singular values.
An attack on thedirection of least variance is likely ineffective when the dataset is spherical.
We observe a strong relationship between the non-sphericity of a dataset and the inﬂuence of the ClipBKD perturbed point(Fig. 3(a)).
In comparison our inﬂuence-based attack consistently ﬁnds a higher inﬂuence value (Fig.3(b)).
Furthermore, this difference in inﬂuence is directly linked to the auditing improvement of ourapproach compared to ClipBKD.
Detecting violations in privacy .
We discuss two case studies where privacy leaks occur in practiceand are detected by our method.
First, the Naive Bayes mechanism in diffprivlib exposes perturbedclass counts in the API.
However, the sum of class counts is enforced to be the dataset size.
Thismeans whenever ( D, D 0 ) differ by one row in length
no privacy is guaranteed.
When we include theexposed class counts as part of ⌧ , our framework detects maximal privacy loss at all " th (Fig. 4(a)).
We therefore recommend that only the class priors and not counts be made accessible.
As another example, in some works DP deﬁnes neighboring datasets can only have a single rowaddition or deletion, while others allow modiﬁying an existing point (equivalently deleting and thenadding a row).
This means that depending on the implementation, the actual privacy level may be halfor double what the user desires to obtain.
Our auditing framework correctly detected this discrepancyin the DP random forest, which deﬁnes " under the ﬁrst option.
Our estimated ˆ " using the seconddeﬁnition exceeded the theoretical level until the deﬁnitions were reconciled.
incremental improvements over baseline.
We reiterate that the advantages of our approach over precursors are (1) improved perturbation attackson D , and (2) optimizing a likelihood attack S on any summary function ⌧ of a model M .
Our ﬁnalablation (Fig. 4(b)) highlights these strengths by demonstrating the incremental improvement to
ClipBKD when either (1) or (2) are added.
Conclusion
We have proposed ML-Audit, a framework for estimating the differential privacy of a ML model.
MLAudit provides a recipe for devising audits against arbitrary models and is often orders of magnitudemore effective than existing approaches – sometimes  2 ⇥ of theoretical optimum.