A General Framework for Auditing Differentially Private Machine Learning
Abstract
We present a framework to statistically audit the privacy guarantee conferred by adifferentially private machine learner in practice.
While previous works have takensteps toward evaluating privacy loss through poisoning attacks or membershipinference, they have been tailored to specific models or have demonstrated lowstatistical power.
Our work develops a general methodology to empirically evaluatethe privacy of differentially private machine learning implementations, combiningimproved privacy search and verification methods with a toolkit of influence-basedpoisoning attacks.
We demonstrate significantly improved auditing power overprevious approaches on a variety of models including logistic regression, Naive
Bayes and random forest.
Our method can be used to detect privacy violations dueto implementation errors or misuse.
When violations are not present, it can aid inunderstanding the amount of information that can be leaked from a given dataset,algorithm, and privacy specification
Introduction
Machine learning (ML) is increasingly deployed in contexts where the privacy of the data used tobuild the model is of concern.
ML models are capable of ingesting enormous quantities of datain order to determine meaningful predictive patterns.
When such models are built on potentiallysensitive inputs such as healthcare [1] or financial data [2], it becomes important to determine towhat extent information from the training dataset can be inferred from the model.
Such informationcan be of personal interest when the identity of a data contributor may be compromised based oninvolvement in the data collection process [3, 4], but it can also lead to regulatory concern aboutwhether a model trained on private data should itself be considered private [5].
Differentially private (DP) machine learning is a theoretically rigorous approach that provides a worstcase privacy guarantee for ML algorithms [6].
Specifically, it provides a mathematical guaranteethat the distribution of possible output models based on the input dataset is not significantly alteredwhether an individual data point is included in the training set or not.
This is accomplished byrandomizing the output model in a specific manner so that the distributions of the model trainedwith or without any individual data point are statistically indistinguishable up to a level specified byparameters ε, δ that must be set appropriately.
While DP mathematically certifies the privacy of amechanism, the noise added is generally tailored to the worst case scenario.
Previous results attempting to assess the privacy leakage of such models, using attacks ranging frommembership inference to model inversion, have inferred that the actual detectable risk of a modelprocedure is lower than the theoretical bound [7, 8].
While a recent study on DP neural networksindicates that the privacy guarantee of the DP-SGD algorithm is tight under the strongest adversary[9], the privacy risk under realistic datasets and attackers without complete access to the trainingprocess still lies below the worst case bound.
Across other models, the question remains of whetherthe privacy violation of a mechanism is lower than specified or whether the attacks previously used are not strong enough.
Given that there are no guidelines on how to set ε, δ in practice, it wouldbe useful for practitioners to estimate the actual risk they incur in practice and set the parametersaccordingly.
This is critical when DP for ML is still a nascent field, with limited implementationsthat can be hard to verify.
Recent works have shown that simple DP mechanisms can be audited effectively [10, 11].
Thesemechanisms generally map an input vector to a perturbed output vector, and are often used as buildingblocks of more complicated DP algorithms such as machine learning models.
Examples include the
Laplace and exponential mechanisms which are respectively used in DP Naive Bayes and randomforest [12, 13].
The auditing procedure involves searching for optimal neighboring input sets a, a ′and sampling the DP outputs M ( a ) , M ( a ′ ) , to get a Monte Carlo estimate of ε .
To extend thesetechniques to audit a machine learning model, the vector inputs a, a ′ are replaced with datasets
D, D ′ which differ by a single entry.
Working in this realm raises important challenges.
First,previously effective search methods for neighboring inputs involving enumeration or symbolic searchare impossible over large datasets, making it difficult to find optimal dataset pairs.
In addition,
Monte Carlo estimation requires costly model retraining thousands of times to bound ε with highconfidence.
We propose solutions to these problems to accommodate the general auditing approach to machinelearning algorithms.
For the first concern, we develop a set of data poisoning attacks which perturba single point of dataset D , approximating a worst-case neighboring dataset D ′ specific to M .
Toaddress the second, we present a statistical framework combining elements of Bichsel et al. [11]and Jagielski et al. [14] to estimate ε for general ML procedures for smaller sample sizes .
Ourtechniques involve improved estimation bounds and, more significantly, learning probabilities overgeneral model spaces , while previous works can only estimate probabilities over a single output (thepredictive probability).
These novel contributions form our framework ML-Audit for auditing the DP of arbitrary machinelearning models.
Compared to prior works we can obtain orders of magnitude improvement in theestimation of ε that are effective across a larger range of ε , more datasets, and more model types.
Ourframework is compatible with prior model-specific approaches (small neural networks), obtainingsimilar or improved efficacy.
Further still, we provide case studies where our framework detectspotential violations in existing implementations of DP machine learning.
We review these prior approaches in section 2, and introduce key DP concepts in section 3.
Thenwe present our framework in section 4, followed by results and discussion in section 5.
Finally weconclude in section 6.
Related work
Although differential privacy of a mechanism is established with mathematical proof, previouswork has shown that holes can exist, either in theory, such as the sparse vector mechanism (ofwhich erroneous versions have been proposed [10]), or implementation, for example samplingnon-uniformity in pseudo-random number generators [15].
Such occurrences point to the need forempirical verification of the promised security of differential privacy mechanisms.
This is criticalas lapses in DP are likely to provide greater predictive performance, so choosing a model to obtainmaximal accuracy given a chosen level of privacy is likely to select these errant models.
Recent works propose efficient solutions for auditing simple differential privacy mechanisms operatingon scalar or vector inputs [10, 11].
In such cases, the neighboring inputs can be enumerated tractably(e.g.
for a vector input, trying adding one to each input element).
For each neighboring pair, theappropriate output set is determined, and Monte Carlo probabilities are measured to determine privacy.
The sampling process in such mechanisms is vectorized to greatly reduce the runtime [11].
Theseworks develop a rigorous statistical framework for DP auditing which serves as the basis of current
ML auditing approaches, including our work.
Few studies have attempted to audit DP machine learners.
We are aware of two works that measure theprivacy of neural networks trained with the DP-SGD mechanism [16].
Jagielski et al.
[14] develops apoisoning attack (ClipBKD) by constructing a data point along the axis of least variance in the dataset.
This causes the gradients of the point to
We show considerably improved performance with our method and that the least variance approachsuffers as the sphericity of the dataset increases.
The second work analyzes DP-SGD through a series of increasingly white-box attacks [9].
They notethat while the DP-SGD privacy guarantee is formulated for all neighboring datasets, the mechanismitself operates purely via gradient manipulation.
Therefore they are able to formulate stronger attackswhich directly target the gradient or make use of adaptive poisoning.
While these are unrealistic forgeneral-purpose ML auditing, we implement their poisoning attack, based on adversarial perturbation,for our DP-SGD experiments.
We find that this attack performs comparably with ClipBKD on DP-SGD.
While their works are important progenitors to ours, they estimate the privacy loss by thresholding asingle scalar output, e.g. the loss on a specific poisoning point.
This is a major bottleneck limitingthe ability to audit other machine learning algorithms.
Instead of a scalar, our approach learnsdistributions over any vector representation of a model, allowing us to be the first to propose a generalprocedure for auditing any DP learner.
Another predecessor established the complementary relationship between data poisoning and differential privacy, using a poisoning attack based on gradient ascent of logistic regression parameters θwith respect to x to reach a target θ ⋆ [17].
The result is identical to a perturbation influence functionwhich we discuss [18, 19].
However, the aim of their study is to evaluate the effect of differentialprivacy on poisoning strength rather than vice versa.
In addition, their attack formulation requiresmanual gradient computation which does not readily extend to other models.
Differential privacy overview
We provide a background on key elements of differential privacy relevant to our work.
Definition 3.1.
A randomized learner M : D × B (cid:55)→ Θ is a function mapping a training dataset D ∈ D and auxiliary noise b ∈ B to an output model θ ∈ Θ .
If we train the learner repeatedly on the same dataset, we can obtain a different model θ each timebased on the noise source b .
For simplicity we implicitly denote the learner as a randomized function
M ( D ) with a probability distribution P induced by b .
Definition 3.2.
A randomized learner M is considered ( ε, δ ) -differentially private if for all datasets
D, D ′ ∈ D differing in a single entry and measurable sets S over the output space Θ ,
[Equation]
In this definition, the ε term is the primary quantity of interest because it bounds the ratio betweenmodel output probabilities when the original dataset D is perturbed by a single row.
Setting ε =
δ = 0would enforce that M ( D ) equals M ( D ′ ) in distribution, while a large ε or δ permits the distributionsto be arbitrarily different.
As ε → ∞ the model distribution converges to M ∞ ( D ) , which inmost cases is identical to the standard non-private version of the model.
The δ term permits someadditive slack in the probability bound, but in most purposes is set to a very small quantity below
O (1 /n ) .
Given ε and δ , M calibrates the noise distribution to the sensitivity of the model function [20].
Definition 3.3.
The sensitivity of a function f : D → R d is max D,D ′ ∥ f ( D ) − f ( D ′ )
∥ where D, D ′differ by one row.
Our main goal is to audit the privacy of a known mechanism whose inner workings may be hidden,so we assume in our work that the user has access to the mechanism, the privacy parameters, and thedataset, but does not have control over the training process or the randomness b .
The final tool which we will employ in our framework is group privacy , which iterates over Eq.
1 tobound the probabilities when D, D ′ differ by multiple points.
Lemma 3.4.
Suppose M is ( ε, δ ) -private and D, D ′ differ by k rows.
Then for all measurable
S ⊂ Θ ,
[Equation]
S S
Dataset
D
D’
M p e r t u r b a ti on
N samples
M ( D ) t
P ( M ( D ′ ) ∈ S ) =
c
M ( D ′ ) p ( D | z )  Auditing framework
A DP mechanism calibrates its randomization to a user-specified ε th representing the desired theoretical level of privacy.
The goal of DP auditing is to empirically assess the actual privacy enabled bythe mechanism, ε ⋆ .
To do this, we determine a lower bound for ε ⋆ with high statistical confidence.
Observe that rewriting the definition of DP gives
[Equation]
For any given ( D, D ′ , S ) we can estimate the quantity with Monte Carlo by retraining M to generatesamples and then compute a 95% confidence interval.
The lower bound of the interval ˆ ε lb is thedetected privacy violation, and with high probability, we state that ε ⋆ ≥ ˆ ε lb with ( D, D ′ , S ) as awitness.
By maximizing ˆ ε lb over all possible witnesses, we increase the lower bound on ε ⋆ which wecan then compare against the promised ε th .
Given a mechanism M , the auditing objective is then tofind
[Equation]
that is, the maximum privacy loss ratio over neighboring datasets D, D ′ and measurable output set
S .
For all mechanisms considered in this work δ < 10 − 4 which is not detectable, so we simplify bysetting δ = 0 .
A successful auditing approach requires solutions to the two separate maximizations.
We maximizeover D, D ′ by developing a toolkit of poisoning attacks which have the largest influence on themechanism in question.
Once D and D ′ are obtained, we fit a likelihood ratio-based optimization todetermine an appropriate S .
In the following, we discuss these procedures in detail.
4.1 Optimizing D, D ′
Given a dataset D =
( X, Y )
we define a valid poisoning attack to be an operation which selectsan existing data point ( x, y ) and perturbs x to any new point x ⋆ within a constraint set C .
As weconsider classification algorithms in this work, the class label y can also be switched.
The constraint
C is specific to each DP algorithm and is used to determine the max sensitivity bound from Def.
3.3(as otherwise a single data point can arbitrarily affect the model).
Without loss of generality, we setthe constraint to be the smallest containing the training data.
This is the only part of the framework which needs user design, as a strong attack against onemechanism may not be effective against another.
However we distill the steps for constructing suchattacks into a recipe:
The underlying model of M can be abstract, requiring a summary function τ to embed M into
a vector space.
Since post-processing of DP outputs does not decrease privacy [6], we can select τ to preserve as much information about M as possible.
In logistic regression the transformation is natural: we define τ ◦ M as the coefficient vector θ (Table 1).
To reduce notation overload
we
implicitly apply τ when we discuss empirical samples z ∼ M ( D ) .
Obtain a non-private algorithm M ∞ ( D ) as a surrogate.
Note that as long as E b [ M ( D, b )] ≈M ∞ ( D ) , meaning the DP model is on average equivalent to the non-private version, it is sufficient
to find the optimal poisoning with respect to the non-private model.
While not explored in thiswork, attacks can instead take into account the actual noise distribution by averaging over samples of M ( D, b ) , at the cost of additional run-time.3.
Determine a poisoning ( x ⋆ , y ⋆ ) from ( x, y ) maximizing the distance between τ ( M ∞ ( D )) andτ ( M ∞ ( D ′ )) .
This involves a selection and a perturbation step.
Bound C
Summary τ
Log.
reg.
L 2 -norm coefficient θ
NB hyper-rectangle µ yd , σ 2 yd , π y
RF hyper-rectangle
Pr( x ⋆ ) of each tree
DP [Equation] SGD
L [Equation] norm of
[Equation] [Equation]
Ahead we detail the poisoning attack design for each mechanism:
Logistic Regression.
DP logistic regression is achieved using one of three approaches: objective,output, or gradient perturbation.
We evaluate objective and output perturbation using the widely usedmethod developed by Chaudhuri and Monteleoni [21] and Chaudhuri et al.
For the attack we adopt the influence function, a technique which estimates the change in a modelwhen a specific data point is infinitesimally upweighted in the training set [19, 23].
For a generalmodel with gradient and Hessian information, the effect of point x ⋆ on the model parameters fora loss function L can be approximated as I θ ( x ⋆ ) =
− H − 1 θ ∇ θ L ( x ⋆ , ˆ θ ) .
In generalized linearmodels with canonical link function where L is the negative log likelihood, this has a closedform involving the Fisher information as the expected Hessian [18], which for logistic regressionevaluates to I θ ( x ⋆ ) =
( y ⋆ − ˆ y ⋆ )
( X ⊤ WX + λI ) − 1 x ⋆ where W is a diagonal matrix with entries
W ii =
(ˆ y i (1 − ˆ y i )) − 1 , and λ regularization.
We choose τ to be the coefficient vector θ, θ ′ of M ∞ ( D ) , M ∞ ( D ′ ) respectively.
We first select( x, y ) closest to the corner of the hyper-rectangle containing the data (a heuristic for selecting a pointfar from where the separating hyperplane would likely be) and initialize ( x ⋆ , y ⋆ ) as the mean point inthe opposite class.
Since our goal is to increase the distance ∥ θ − θ ′ ∥ 2 , we optimize the L 2 normof the influence function using projected gradient ascent.
The constraint set C is the L 2 norm ballcontaining the training set, so after each iteration we clip x ⋆ if needed.
Naive Bayes.
The Gaussian Naive Bayes model parameterizes the class-conditional distribution offeatures using independent Normal distributions.
The parameters involve a mean µ yd and varianceσ 2 yd for each feature and class combination, as well as prior probabilities for each class π y .
To achievedifferential privacy, Laplace or Geometric noise are added to the maximum likelihood estimates foreach parameter [12].
The constraint set C is a hyper-rectangle set by the smallest and largest value ofeach feature in the dataset.
We choose τ to be the vector of all { µ yd , σ 2 yd , π y } .
The maximum influence a perturbation can haveon ˆ µ y of class y is by selecting a point on the corner of the hyper-rectangle C nearest the data pointsof class y and placing it on the corner furthest away.
On the other hand, the maximum influenceattack on π y is to flip a class label.
We combine the two ideas by flipping the class label of the pointclosest to a corner.
In our experiments we compare the power against only attacking µ y .
Random Forest.
The DP random forest mechanism uses unsupervised random splits of the featuresbased on the domain of the inputs.
Each tree is split to a pre-determined depth.
Then the trainingpoints are percolated through the forest, and the majority label of each leaf is sampled from the
Exponential mechanism [13].
Thus a perturbed training point has no impact on the actual treestructure besides the potential label of a single leaf in each tree.
As a result we measure only thechange in those leaves, choosing τ to be the prediction of each tree on x ⋆ .
The most detectable change in probability occurs when j = 1 and then the majority class is swappedby a class flip.
For example, if a leaf has one positive and two negative points, we flip the class ofone of the negatives.
Since each tree is random, there is no guarantee that any given point will bein such a situation where j = 1 , but to increase this chance we target points which are likely tobe solitary in each leaf.
Thus we select the point most distant from the rest of the training set as measured by L 1 -distance and flip its label.
We provide further exposition and hyperparameter detailsin the Appendix.
DP-SGD.
For DP-SGD we evaluate two extent poisoning attacks using our framework.
The ClipBKDattack constructs a data point along the axis of least variance in the dataset [14].
This can be performedby taking the eigenvector of the covariance matrix with the smallest eigenvalue, and then projecting itto the median L2 norm of the training set.
This point is assigned the label with smallest predictedprobability.
The goal of this attack is to enable the gradients of the point to be distinguishable,mitigating the effect of the DP-SGD clipping and noise.
The second attack selects a random datum and maximizes the loss with gradient ascent.
The loss isdefined with respect to a set of trained shadow models [9].
In both attacks, the outcome τ ( M ( D ))being measured is the predicted probability of the perturbed point P M ( x ⋆ = 1) , while ClipBKD firstsubtracts the zero vector’s predicted P M ( ⃗ 0) .
We also consider an updated ClipBKD with the zeroprediction encoded separately with additional test predictions in τ : P M ( ⃗ 0) , P M ( x testi ) .
4.2 Optimizing S
Require: Datasets D, D ′ , DP mechanism M , min.
probability constraint r , confidence α , sample size N 1: Sample z 1 , . .
, z N ∼ M ( D ) , z ′ 1 , . . .
, z ′ N ∼
M ( D ′ ) 2: Generate labels y i
=
if z i , else y i =
if z [-
-]′ i 3: Train classifier on { z, z ′ } , { y } to learn p θ ( D | z ) 4: p 1 , . .
, p N ← p θ ( D | z 1 ) , . .
, p θ ( D | z N ) 5: p ′ 1 , . . .
, p ′ N ← p θ ( D | z ′ 1 ) , . .
, p θ ( D | z ′ N ) 6: ˜ ε lb ← −∞ ; ˜ t ←7: for t in sort( { p i } N 1 ∪ { p ′ i } N 1 ) do 8: n 1 ← (cid:80) i 1 ( p i > t ) 9: n 0 ← (cid:80) i 1 ( p ′ i > t ) 10: if n 0 /N <
r then ▷ Avoid estimating probs < r 11: skip to next iter 12: ε tlb ← Eps _ LB( n 1 , n 0 , N, α/ 2) ▷
Lemma A.
13: if ε tlb > ˜ ε lb then 14: ˜ ε lb ← ε tlb 15: ˜ t ← t 16: return S ← { z | p θ ( D | z ) > t }
Our approach for determining S is based onthe framework of Bichsel et al.
[11], whichderives the optimal region for distinguishing two vector distributions.
Given candidate datasets ( D, D ′ ) with DP distributions M ( D ) and M ( D ′ ) , we seek S maximizing 236 P ( M ( D ) ∈ S ) / P ( M ( D ′ ) ∈ S ) .
Let f M ( D )and f M ( D ′ ) be their respective densities.
Selecting S to best distinguish two distributionsis analogous to identifying an optimal rejectionregion in hypothesis testing.
From the NeymanPearson Lemma (cf. Appendix), the maximumpower test at level c is defined by rejection region S :
= { z | f M ( D ) ( z ) /f
M ( D ′ ) ( z ) > k }for some k > 0 , where P ( M ( D ′ ) ∈ S )
= c[24].
Intuitively, we have a likelihood ratiof M ( D ) ( z ) /f
M ( D ′ ) ( z ) and we are selectingpoints to fill S where D is more likely than
D ′ .
We start by adding points where D is mostlikely and work downwards until S is largeenough that there is a c chance
that M ( D ′ ) iserroneously in S .
Directly obtaining these densities is intractable.
Instead we can learn, given collected samples {M ( D ) } where D ∈ { D, D ′ } , the posterior probability of the dataset D :
[Equation]
Let us consider the set L k :=
{ z | p ( D | z ) /p
( D ′ | z ) > k } .
From Bayes’
Theorem we know p ( D | z )
∝f M ( D ) ( z ) ·
Pr( D = D ) .
Given equal samples of D from D and D ′ , then P ( D = D ) =
P ( D = D ′ )implying p ( D | z ) /p
( D ′ | z ) =
f M ( D ) ( z ) /f
M ( D ′ ) ( z ) .
Therefore S =
L k
and we use it as the rejection set.
Finally, we can further simplify S by observingthat p ( D | z ) + p ( D ′ | z ) = 1 , and so S =
{ z | p ( D | z ) > t } where t := k/ (1 + k ) .
What remains isto set t according to the level c so
that P ( M ( D ′ ) ∈ S ) =
c .
As this is equivalent to P ( p ( D | z ) >t | z ∼ M ( D ′ )) , we set t empirically to correspond to the (1 − c ) -quantile of M ( D ′ ) .
Our approach introduces the following further adaptations on Bichsel et al. [11] to enable auditing onslower, potentially expensive machine learning mechanisms.
(The original work used around 10 8samples for auditing simple mechanisms, which is infeasible for training machine learning models.
We use N = 10000 samples for all mechanisms except DP-SGD where N = 500 .)
1) We make the method flexible by searching c empirically to maximize ˜ ε lb , rather than settingc = 0 .
01 as in the original.
When N → ∞ , smaller values of c are optimal since the likelihood ratiois largest.
At smaller sample sizes, however, this is offset by increasing uncertainty in Monte Carloestimation of small probabilities.2)
We use group privacy to increase statistical distinguishability, a trick used also in Jagielski et al.[14].
After obtaining ( x ⋆ , y ⋆ ) , we insert k copies of the point.
From Lemma 3.4 the detectedprivacy ˜ ε lb is reduced to ˜ ε lb /k .
However, the extra copies causes M ( D ′ ) to be more distinguishablewhich makes P ( M ( D ) ∈ S ) larger relative to P ( M ( D ′ ) ∈ S ) .
This is also a tradeoff: At smalltrue ε the lower bound ˜ ε lb improves despite division by k , whereas at high ε the distributions aresufficiently different at k = 1 that it is not worth dividing ˜ ε lb .
We include empirical analysis of k inour Appendix.3)
All previous auditing works (including below) use Clopper-Pearson intervals for ˆ p 1 and ˆ p 0 todetermine the lower bound.
This method is highly suboptimal because it separately bounds thenumerator and denominator.
We identify instead the Katz-log confidence interval to directly boundthe ratio of binomial proportions [25], see Lemma A.3.
This has far better coverage properties in oursimulations (Appendix).
Our modified procedure is presented in Algorithm 1 and summarized visually in Fig.
Comparison to other approaches.
While our method allows M ( D ) to be any vector of information,the previous works auditing DP-SGD require M ( D ) to be a single value (e.g. prediction probabilityor loss) and search for a threshold dividing the score into S and S c [9, 14].
Our method can reduce tothis special case by choosing τ to map to a scalar.
The approach can be interpreted as membershipinference with the bound FN + e ε FP ≤ 1 − δ from Kairouz et al.
[26], where {M ( D ) ∈ S } are truepositives and {M ( D ′ ) ∈ S } are false positives.
These works also note that flipping the definition offalse positive and false negatives enables testing the complement P ( M
( D ′ ) ∈ S c ) / P ( M ( D ) ∈ S c ) .
As in those works we select S c instead of S if it gives higher ˜ ε lb .
Corollary 4.0.1.
Given Algo.
1 and N samples, the highest detectable privacy risk is ln( N ) −z α/ 2 (cid:113) 1 − 1 N .
Estimating ˆ ε lb .
After ( D, D ′ , S, k ) are determined, the final step is to draw N fresh samples of
M ( D ) and M ( D ′ ) for an independent Monte Carlo verification.
As in the search phase, we computen 1 = {+(cid:80) i 
+}
( M ( D ) ∈ S ) , n 0 = (cid:80)
i 1 ( M ( D ′ ) ∈ S ) and use the lower bound of the Katz log intervalto return our final estimate ˆ ε lb .
To avoid biasing the confidence interval, it is important that the final MC estimate is conducted on anindependent set of samples, as done in [11, 14].
This is especially the case when optimizing overthresholds in S .
Since we compute a 1 − α interval over ˆ ε for each threshold, using the best resultfrom the same sample would lead to biased inference from multiple testing.
Experiments and Results
We evaluate the DP mechanisms over a range of ε th : { 0 . 1 , 0 . 25 , 0 .
At ε th = 0 .
1the ratio of probabilities is bounded by e 0 . 1 ≈ 1 .
1 giving nearly indistinguishable distributions,whereas at ε th = 50 essentially no privacy is guaranteed.
For a given dataset D , we perturbk ∈ { 1 , 2 , 4 , 8 } points to get D ′ and train N = 10000 times for each to determine the appropriateauditing set S .
Then we obtain N new samples to perform the final Monte Carlo estimate and obtainthe lower bound ˆ ε lb .
We use confidence level α = 0 .
05 throughout.
We assess Naive Bayes, logistic regression (output and objective perturbation), and random forest oncommon machine learning datasets: adult , credit , iris , breast-cancer , banknote , thoracic .
We usethe diffprivlib library [27] and implement output perturbation following [22].
Additionally, we test
DP-SGD on FMNIST and CIFAR10 (here with N = 500 ) using [28].
Refer to Appendix for moredetails.
Figure 2 directly compares, over three datasets, the ˆ ε lb obtained by our poisoning attacks (ML-Audit),the ClipBKD attack, and a baseline where a random data point x is changed to a random point fromthe opposite class without changing labels (Swap-X).
(Note that while the original ClipBKD workuses DP-SGD, the poisoning itself is mechanism-agnostic, so it is an appropriate baseline.)
Wereplicated our procedure 3x and show the median ˆ ε lb .
The Katz log interval was used for the lower [Figure] bounds even on the baselines.
As a reference we plot the theoretical privacy loss up to the maximumdetectable limit (Corollary 4.0.1).
Our attacks consistently and often dramatically outperform baselines across datasets, models, andε th .
In some cases, the privacy detection is within a small factor (1 − 2 × ) of optimal.
The Swap-Xbaseline shows the expected privacy loss of an attack which poisons a data point within the originaldata distribution, rather than a worst-case perturbation (e.g. swapping two points in the trainingset).
The only case where Swap-X performs similarly to our tailored attacks is in output-perturbedlogistic regression on separable datasets with high non-sphericity such as iris and breast-cancer . We believe these properties enable simple strategies to work well (see Fig. 3(a)).
We believethis is not a meaningful difference in performance as both methods are near the limit of what canbe theoretically detected.
Our results validate previous findings showing that estimating ε usingmembership inference-based bounds are weaker than poisoning attacks [9, 14].
We also tried theloss-threshold attack of [7, 29] and found it ineffective.
On many models and datasets we obtain nearly optimal results, indicating that these datasets are closeto the maximal sensitivity of the mechanism.
We observe this for random forest at ε th ≤ 2 but also aplateau in higher ε th .
This is likely due to lack of resolution in the outputs: an average over m treedecisions only takes m unique probability values.
Our attack on Naive Bayes is also near-optimal,while varying among datasets.
Since the algorithm assigns ε th / 3 to protect each of µ yd , σ 2 yd , and π y ,a perturbation needs to achieve the sensitivity bound on all three to reach the theoretical privacy risk,with the bound over all pairs D, D ′ rather than conditioning on a starting D .
For example, to reachthe bound for µ , all points of one class must be in one corner of the hyper-rectangle and the poisonedpoint on the other corner, which is unrealistic.
For this reason, our choice of attacking the class priorπ is generally more effective than attacking the mean µ (Fig. 4).
Lastly, our DP-SGD experiments assess the two existing perturbation methods of [9, 14].
Ourfindings (Fig.
8) are consistent with what is reported in those works.
Interestingly, we find that theshadow adversarial attack outperforms ClipBKD on FMNIST and vice versa on CIFAR10.
We also updated ClipBKD to use additional information in τ as detailed in Section 4 . 1 and find a moderateimprovement on FMNIST.
Refer to the Appendix for further discussion.
Dataset-specific privacy.
Previous work hypothesized that privacy attacks on realistic datasets donot cause the model shift to reach the worst-case given by the sensitivity bound, except when theattacker can completely specify the dataset [9].
Based on our observations on the impact of thedataset distribution on attack strength, we agree that achievable privacy is dataset-dependent.
Webelieve theoretical analysis of datasets for attack risk to be useful for future work. [Figure] A u d i t i n g i m p
r o v e m e n t We briefly investigate this phenomenon by comparing the performance of ClipBKD by datasetsphericity, as computed by ratio between largest and smallest singular values.
An attack on thedirection of least variance is likely ineffective when the dataset is spherical.
We observe a strongrelationship between the non-sphericity of a dataset and the influence of the ClipBKD perturbed point(Fig. 3(a)).
In comparison our influence-based attack consistently finds a higher influence value (Fig.3(b)).
Furthermore, this difference in influence is directly linked to the auditing improvement of ourapproach compared to ClipBKD.
Detecting violations in privacy .
We discuss two case studies where privacy leaks occur in practiceand are detected by our method.
First, the Naive Bayes mechanism in diffprivlib exposes perturbedclass counts in the API.
However, the sum of class counts is enforced to be the dataset size.
Thismeans whenever ( D, D ′ ) differ by one row in length
no privacy is guaranteed.
When we include theexposed class counts as part of τ , our framework detects maximal privacy loss at all ε th (Fig. 4(a)).
We therefore recommend that only the class priors and not counts be made accessible.
As another example, in some works DP defines neighboring datasets can only have a single rowaddition or deletion, while others allow modifiying an existing point (equivalently deleting and thenadding a row).
This means that depending on the implementation, the actual privacy level may be halfor double what the user desires to obtain.
Our auditing framework correctly detected this discrepancyin the DP random forest, which defines ε under the first option.
Our estimated ˆ ε using the seconddefinition exceeded the theoretical level until the definitions were reconciled.
[Figure]
We reiterate that the advantages of our approach over precursors are (1) improved perturbation attackson D and (2) optimizing a likelihood attack S on any summary function τ of a model M .
Our finalablation (Fig. 4(b)) highlights these strengths by demonstrating the incremental improvement to
ClipBKD when either (1) or (2) are added.
Conclusion
We have proposed ML-Audit, a framework for estimating the differential privacy of a ML model.
MLAudit provides a recipe for devising audits against arbitrary models and is often orders of magnitudemore effective than existing approaches – sometimes ≤ 2 × of theoretical optimum. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] As with any
auditing tool
, there is slight potential for misuse in probing security of existing systems.
However, the benefits to white-hat investigators surpasses the potential negatives, since
the tool would only detect if the system’s privacy is incorrectly implemented.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Previously published
proofs are cited to original work
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] In supplement
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Details in appendix
(c) Did you report error bars (e.g., with respect to the random seed after running exper-
iments multiple times)?
[No] Our work by definition runs experimental sampling
thousands of times and reports the lower bound.
While we replicated our entire proce-
dure 3x, by nature of the statistical lower bound, the results are extremely similar every
time.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] In Appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Code from diffprivlib
and previous models are cited
(b) Did you mention the license of the assets?
[No] We used the most widely-used open
source datasets
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] We used standard open-source data
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
No data used contains personally identifiable
or offensive information
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
No crowdsourcing or HSR
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
No crowdsourcing or HSR
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
No crowdsourcing or HSR
A Mathematical claims and proofs
Lemma A.1.
Suppose M is ( ε, δ ) -private and D, D ′ differ by k rows.
Then for all measurable
S ⊂ Θ ,
[Equation]
Proof.
Let D 1 , D 2 , . . .
, D k − 1 represent intermediate datasets between D and D ′ changing one rowat a time.
For each step Definition 3.2 holds.
For example
[Equation]
Continuing the iteration, we get
[Equation]
For ε > 0 , the final sum is a geometric series.
Applying the standard identity gives the final result.
Lemma A.2.
(Neyman-Pearson): Given observations of a variable X , consider a hypothesis testdistinguishing H 0 :
θ = θ 0 and H 1 : θ = θ 1 with respective densities f 0 ( x ) and f 1 ( x ) .
For a level ofsignificance α , there exists a test with rejection region R and t ≥ 0 such that
[Equation]
A test satisfying these conditions is a most powerful test at level α
(that is, P 1 ( X ∈ R ) is greater orequal to that of any other test with the same level).
Proof.
See [24].
Lemma A.3.
[30] Given two independent Binomial variables with underlying probability parameters p 1 , p 0 , number of trials N , and observed values n 1 and n 0 , a 1 − α confidence interval for the ratio
ln
( p 1 /p 0) is
[Equation]
where z α/ 2 is the critical value of the standard normal (e.g. 1.96 for α = 0 . 05 ).
Proof.
See [30]
Corollary A.3.1.
Given Algorithm 1 and N samples, the highest detectable privacy violation is
[Equation]
Proof.
Given N samples of distribution M 1 and N of distribution M 0 with Bernoulli probabilityp 1 , p 0 of being in set S , the largest value of ln( p 1 /p 0 ) is when all N of M 1 are accepted in S andonly one of M 0 is (otherwise the ratio is infinite).
That is, n 1 = N and n 0 = 1 .
So the maximalvalue of ln( p 1 /p 0 ) is ln( N ) .
Then apply Lemma A.3 to get the confidence lower bound, which is thebest case reportable by our algorithm.
Theorem A.4.
(DP-Sniper approximately optimal)
Given a minimum measured probability c as in
Algorithm 1, neighboring inputs ( D, D ′ ) , N > 0 samples, the DP-Sniper algorithm finds set ˆ S t with threshold defined by c .
Let ˜ ε ( D, D ′ , ˆ S t ) be the power of this attack, and ε ( D, D ′ , S ⋆ ) be the power of the best possible attack set S ⋆ .
Then
[Equation]
with probability at least 1 − α , where ρ = (cid:113) ln(2 /α ) 2 N , and ρc ≤ 12 .
Proof.
See [11]
B Additional Results
Auditing results on all datasets follow:
[Figure]
B.1 Empirical analysis of k to perturb
We highlight some results comparing k = { 1 , 2 , 4 , 8 } for the same attack, model, and dataset.
Asshown, small values of ε benefit from larger k , while at larger ε , k is detrimental because it requiresdividing the final privacy estimate.
In our final experiments we adopt k = 8 for all ε ≤ 2 , k = 2 for2 <
ε ≤ 8 , and k = 1 for ε ∈ { 16 , 50 } .
B.2 Coverage simulation
We compare the baseline Clopper-Pearson interval with the Katz-log interval specifically designedfor ratios of binomial variables.
At a 95% desired coverage ( α = 0 . 05 ), we find the Katz log intervalsto be precise while the Clopper-Pearson are highly conservative.
This is to be expected since the
Clopper-Pearson approach separately bounds the two binomial variables before taking the ratio: thelower bound of the numerator divided by the upper bound of the denominator.
Thus intuitively at aα = 0 .
05 level it actually computes a quantity that holds with probability on the order of α 2 .
[Figure]
[Table]
[Figure]
[Table]
C
DP-SGD Result
Specified 0.
M e a s u r e d
FMNIST
Specified 0.
M e a s u r e d
CIFAR
ClipBKD (orig)
Shadow adv.
ClipBKD (updated) theoretical
Our findings here are consistent with what is reported in the benchmark works.
The shadow adversarialattack here outperforms ClipBKD on FMNIST and vice versa on CIFAR10.
In general the auditingperformance of these attacks are weaker compared to the other models.
We believe this to be acombination of small sample size and resistance of DP-SGD to data perturbations: we found thatmore perturbations k
= 4 , 8 were required to induce detectable changes at high ε th .
D Additional experiment details
Our work considers binary classification so we restrict all datasets to classes 0 and 1.
Datasets whichare larger than 1000 points are subsampled to 1000.
For evaluating random forest, which scalespoorly with dataset size, we further subsample all datasets to 500 points.
Results do not changesignificantly with dataset size (it is currently a pure Python implementation), as the DP noise added iscalibrated to dataset statistics.
Hyperparameter details are in the Appendix.
Our experiments were run over an internal cluster of 240 CPUs over a week.
A single auditing run(40k retrainings, not DP-SGD) averaged 10-40 min depending on the dataset and model, while asingle DP-SGD audit took about 3 hrs for 2k retrainings.
D.1 DP-SGD
For FMNIST, we use a CNN with three 2D-conv layers and 2x2 max-pooling.
The number of filtersstarts at 4 and is doubled to 8 at the third convolution.
This is followed by a fully-connected layer.
For CIFAR10, the overall architecture is the same but the number of filters is doubled for all convlayers.
We set the clipping norm to 0.002 similar to previous work and use learning rate 1e-3 with RMSprop.
We train FMNIST for 10 epochs and CIFAR10 for 15.
For all settings tested, model accuracy is over98% so we believe our settings are reasonable.
Note that in [14] one- and two-layer fully connected networks are studied.
Thus our models havehigher capacity and are more realistic for the datasets studied.
Our results for both baselines areconsistent with reported.
During training our observations correlate with the comments in [14] about the normalization of thedataset vs impact of weight initialization.
For example, we find the strongest auditing performanceby far when FMNIST is kept in the original feature scaling of [0 , 255] , with a detected privacy risk ofaround 0.7 for ε = 50 .
In contrast, when normalized to [ − 1 , 1] or [0 , 1] , the highest detected privacyrisk drops to around 0.4.
We believe this to be the intersection of a few factors, including weightinitialization (affecting initial variance of the network embeddings), the max grad norm clippingvalue, and the learning rate.
We think this phenomenon worthy of further study to better understandthe nuances affecting privacy in networks trained with DP-SGD.
D.2 Random Forest
Derivation of attack: The class label y
( l ) of a leaf l is determined as
[Equation]
where u ( z, l ) is 1 if z is the majority label in l and 0 otherwise, and j is the number of excess pointsfor the majority.
We want a change of a single point that can induce a large change of this probability.
As j increases the probability rapidly converges to 1, so for larger
j the difference is statisticallyindistinguishable.
We can compare the ratio when j = 2 and j = 1 , and verify that the largestdifference is when j = 1 and the majority is flipped.
Note that this requires actively flipping the label of a point rather than just removing or adding a pointto make the majority an equality.
In the case of equal proportions of classes then the final probabilitybecomes 0.5 which is not as strong as if the majority flips.
We use m = 15 trees and depth 10 in our study.
We further note that the detectable privacy violation is hyperparameter-dependent.
By our analysis,growing the tree depth with n sufficiently such that each data point is solitary would guarantee theeffectiveness of the class flip attack.
However, the size of the tree increases exponentially and theruntime is too costly with the current available implementation.
As a compromise we cap our datasetsizes to 500 and disregard categorical features in the random splits, as they cannot be used to isolatepoints.
This provides additional evidence that practical privacy risk is dataset-specific.
We note thatgiven the goal to analyze the largest privacy risk in random forest (e.g. to assess whether there areimplementation errors), it makes sense to select parameters and dataset sizes to maximize privacyrisk.
D.3 Logistic regression
We found in the implementations of logistic regression that the performance of the model at certain εand λ combinations would lead to severe degradation of optimization, essentially giving degeneratecoefficients.
In such cases, nearly no privacy loss is detectable.
Our goal is to assess the models at areasonable performance level reflective of their actual use case on each dataset, so we adjusted theregularization λ to avoid these situations.
We found that λ = 5 · 10 − 4 for objective perturbation andλ = 5 · 10 − 3 for output perturbation worked most of the time.
E Adapting ML-Audit for δ > 0
In our current study all methods we evaluate use standard implementations with δ = 0 , with theexception of DP-SGD, where δ is very small.
Our method can be readily adapted for δ > 0 bykeeping the δ in the auditing objective of Section 4.
Then the objective splits into two quantities, thefirst of which is what we currently bound.
For the second we can use a binomial proportion CI on thedenominator while setting δ to be the level we wish to audit at.
If we don’t have prior information wecould also vary δ which results in a curve of estimated δ vs ˆ ε .
Nonetheless, in practical applications, δ ∈ o (1 /n ) yielding values which are exceedingly difficult tomeasure (and which essentially have no effect on our measurements).
For example, the minimumprobabilities which we can measure with confidence via Monte Carlo are around 0.01 while δ < 0 . 001for
a 1000-point dataset.
F Additional logistic regression baseline
We demonstrate an additional baseline for logistic regression, using the data poisoning attack from[17].
As discussed in our Related Works
[Figure]
their optimization objective parallels the influence functionapproach, with a different target.
Rather than strictly maximizing the influence function, a targetcoefficient can be specified.
To maximize the change in the model coefficients
[Figure]
we specify a randomtarget vector perpendicular (that is, with zero dot product) to the true coefficients
[Figure]
G Data-specific epsilon vs influence
As the ε of a DP ML mechanism holds for a worst-case pair of neighboring datasets, we note thatmany real-world datasets may have lower achievable maximum privacy leakage.
We can take themaximum ˆ ε estimated from our logistic regression auditing experiments as a proxy lower bound forthis quantity.
We let Y be this value relative to a true ε = 1 .
Additionally, we define X to be the norm of the influence function applied at our perturbation point,relative to the average influence function norm of each dataset.
The following figure compares X and
Y :
[Figure]