P RIVACY A UDITING OF M ACHINE L EARNING USING M EMBERSHIP I NFERENCE A TTACKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Membership inference attacks are used as an auditing tool to quantify the private information that a model leaks about the individual data points in its training set.
Membership inference attacks are inﬂuenced with different uncertainties that an attacker has to resolve about training data, training algorithm, and the underlying data distribution.
Thus attack success rates, of many attacks in the literature, do not precisely capture the information leakage of models about their data, as they also reﬂect other uncertainties that the attack algorithm has.
In this paper, we present a hypothesis testing framework that can explain the implicit assumptions and also the simpliﬁcations made in the prior work.
We also derive new attack algorithms from our framework that can achieve a high AUC score while also highlighting the different factors that affect their performance.
Our algorithms capture a very precise approximation of privacy loss in models, and can be used as a tool to perform an accurate and informed estimation of privacy risk in machine learning models.
We provide a thorough empirical evaluation of our attack strategies on various machine learning tasks trained on benchmark datasets.
I NTRODUCTION
Machine learning systems have come under intense scrutiny of the regulatory authorities in the past few years.
Veale et al.
(2018) argue that machine learning models could be considered personal data due to their susceptibility to inference attacks that can recover sensitive information about training data just from the models.
Membership inference attacks (Homer et al., 2008; Dwork et al., 2015; Shokri et al., 2017) and reconstruction attacks (Dinur & Nissim, 2003; Song et al., 2017; Carlini et al., 2020) are the main inference attacks that highlight, and can quantify, the privacy risk of releasing aggregate information computed on sensitive data (Dwork et al., 2017).
The focus of this paper will be on membership inference attacks for measuring privacy risk.
Organizations such as the ICO (UK) and NIST (US) have highlighted membership inference as a potential conﬁdentiality violation and privacy threat to the training data (Murakonda & Shokri, 2020).
This has lead to the development of open-source tools 1 and capabilities in widely-used ML libraries 2 for measuring privacy risk from machine learning models using membership inference attacks.
Although the approach of quantifying privacy risk through membership inference attacks is gaining traction, the attack success, as measured by a lot of works, cannot be completely attributed to information leakage from the models and hence their privacy risk.
Various factors such as the distribution of training data, difference in distributions of the train and test data may provide an over-estimate or under-estimate of the actual privacy risk from the model (Erlingsson et al., 2019; Humphries et al., 2020).
Theoretical analyses that connect the success of membership inference to privacy risk through the framework of differential privacy avoid this issue by slightly modifying how the attack performance is measured (Yeom et al., 2018; Jagielski et al., 2020;
Nasr et al., 2021; Malek et al., 2021).
Instead of measuring the leakage from a particular model, these works aim at measuring the worst-case leakage of the training algorithm (for a given model architecture), and construct multiple models with and without one training point and keep the rest of training set ﬁxed.
The performance of the attack (false positive and false negative errors) is then computed over these sampled models.
However,in practice and during auditing, the performance needs to be computed based on the points in the training set versus the population data (test set) for a given ﬁxed model.
Various works rely on measuring privacy risk through the performance of membership inference attacks, but the subtle differences in how the attack is formulated mean that they might associate different causes of attack success to privacy loss.
When auditing machine learning models, we need to pay attention to what exactly we are measuring and how it relates to the information leakage of the machine learning algorithm and not other factors such as the prior knowledge of the attacker. Previous membership inference attacks are designed for high overall performance, i.e. to succeed for most member (or non-member) data points of most target models.
With few exception, they train shadow (reference) models on datasets randomly sampled from a population, to mimic the general behavior of models on member (or non-member) data points. This general behavior, however, does not capture the behavior of speciﬁc models on speciﬁc samples.
As a result, the limited performance of shadow model attack does not precisely measure the model (sample)-speciﬁc information leakage. In this work, we focus on designing membership inference attacks that more precisely measure what a target model leaks about each individual training data, in a binary hypothesis testing framework. We start from the (constant) loss threshold attack, and study how to further design different attack strategy (loss threshold) for different target model (sample).
We derive multiple attack strategies (existing and new) from this hypothesis test formulation via different approximations for the null hypothesis.
We also formulate the performance of an attack inside the hypothesis testing framework, via the trade-off between its false positive rate and true positive rate.
Following the methodology, we not only derive and explain existing shadow (reference) model attacks (Attack S and R), but also design new attacks (Attack P and D) that offer more accurate privacy auditing for machine learning models, through model-dependent and (or) sample-dependent attack strategies. We empirically evaluate and compare the attack performance (TPR-FPR curve and AUC score) and computation cost of our new attacks (notably Attack D), and the prior attacks on multiple datasets.
A TTACK F RAMEWORK
Our objective is to design a framework that enables auditing the privacy loss of machine learning models in the black-box setting (where only model outputs —and not their parameters or internal computations— are observable).
This framework needs to have three elements: (i) the inference game as the evaluation setup; (ii) the indistinguishably metric to measure the privacy risk, and (iii) the construction of membership inference attack as hypothesis testing.
The notion of privacy underlying our framework is primarily based on differential privacy, and multiple pieces of this framework are generalizations of existing inference attacks against machine learning algorithms.
Instead of focusing on designing new attacks, we present the important design choices for constructing and evaluating membership inference attacks, for the purpose of having a precise privacy auditing.
We identify different sources of uncertainty that inﬂuence the error of inference attacks, and can lead to miscalculation of the privacy loss of a model.
We quantify privacy loss of a model in a hypothetical inference game between a challenger and an adversary.
We are given a private training set D , and a model θ which is trained on D using a training algorithm T .
The challenger samples a random data point ( x 1 , y 1 ) from the training set (a member), and a random data point ( x 0 , y 0 ) from the data population outside the training set (a non-member).
He then randomly selects one of the two (member or non-member) b ∼ { 0 , 1 } with probability 12 , and shares the selected data point ( x b , y b ) and the model’s output f ( x b ; θ ) with the adversary.
The adversary’s task is to determine if the data point is a member or not (i.e., guess b ).
privacy loss of an algorithm as, for example, its worst case privacy loss (as in differential privacy).
Differentially private algorithms enforce an upper bound on the privacy loss of an algorithm over all models with respect to all possible training data.
Thus, in the inference game for differential privacy, the privacy loss of the training algorithms , would be the worst case privacy loss over all choices of D , ( x 0 , y 0 ) , and ( x 1 , y 1 ) in our inference game for model privacy.
We use an indistinguishability measure (which is the basis of differential privacy) to deﬁne privacy of individual training data of a model.
According to this measure, the privacy loss of the model with respect to its training data is the adversary’s success in distinguishing between the two possibilities b = 0 vs b = 1 over multiple repetitions of the inference game.
Naturally, the inference attack is a hypothesis test , and adversary’s error is composed of the false positive (i.e., inferring a non-member as member) and false negative of the test.
In practice, the error of adversary in each round of the inference game depends of multiple factors:
• The true leakage of the model about the target data ( x b , y b ) when b = 1 .
• The uncertainty (belief or background knowledge) of attack algorithm about the population data • The adversary’s uncertainty about the training algorithm T • The uncertainty about all training data except the target data ( x b , y b ) • The attack dependency on the target data ( x b , y b ) , and the model θ
In the ideal setting, we only want the attack error to be dependent on the true leakage of the model about the target data (i.e., whether the same model trained with and without ( x b , y b ) are distinguishable from each other).
To this end, and to cancel out the effect of other uncertainties, an attack algorithm and the evaluation setup for the inference game need to be designed based on the following principle: The population data used for constructing the attack algorithm, and evaluating the inference game, need to be similar, in distribution, to the training data.
This is to minimize the impact of prior belief (what could have been sampled for the training set) in the performance of the inference attack.
This is not hard to achieve as all the process (of constructing the hypothesis testing attack, and evaluating it) is controlled by the auditor.
By violating this principle, we might overestimate the privacy loss (by making the test dependent on a distinct prior knowledge) or underestimate the privacy loss (by evaluating the inference attack on a population data distribution for which it was not constructed).
Another crucial requirement is that the privacy audit needs to output a detailed report, which captures the uncertainty of the attack.
Reporting one number as the accuracy of the attack, as it is mostly reported in the literature, is not an informative report.
Given the attack being a hypothesis test, the audit report needs to include the analysis of the error versus power of the test: if we can tolerate a certain level of false positive rate in the inference attack, how much would be the true positive rate of the attack, over the random samples from member and non-member data?
The area under the curve for such an analysis reﬂects the chance that the membership of a random data point from the population or the training set can be inferred correctly.
C ONSTRUCTING M EMBERSHIP I NFERENCE A TTACKS
The adversary in the membership inference game can observe the output of a target machine learning model θ , trained on unknown dataset D .
He also gets a precise target data point z as input, and is expected to output 0 or 1 to guess whether the sample z is in the dataset D or not.
We use likelihood ratio test (LRT) as the most powerful criterion for choosing among membership hypotheses, under the following assumptions.
The adversary knows, and can sample from the underlying data distribution π ( z ) over population z = ( x z , y z ) ∈ D pop . 2.
A randomized training algorithm T : D (cid:55)→ θ takes in a training dataset D , which consists of i.i.d.
samples from the data population, and produces a model θ that incurs a low loss (cid:80) ( x,y ) ∈ D (cid:96) ( θ, x, y ) on dataset D .
We denote P ( θ | D ) to be the posterior distribution of trained model θ given training dataset D .
Deﬁnition 3.1 (Approximated LRT for Membership Inference) Let ( θ, z ) be random samples from the joint distribution of target model and target data point, speciﬁed by one of the following membership hypotheses.
[Equation]
[Table]
[Equation] The likelihoods function of hypothesis H 0 and H 1 , given observed target model θ and target data point z , is as follows (detailed derivations are in the Appendix).
[Equation]
Now we follow the previous construction of Bayes-optimal membership inference attack Sablayrolles et al. (2019), and model the posterior distribution P ( θ | D ) of trained model as follows.
[Equation]
where T is a temperature constant that allows some randomness in the training algorithm T .
The equation 5 holds for Bayesian learning algorithms, such as stochastic gradient descent Polyak &
Juditsky (1992), deterministic MAP (Maximum A Posteriori) inference (for T → 0 ), and Bayesian posterior sampling Welling & Teh (2011) (for T = 1 ).
Therefore, the LRT statistics can be computed as follows (detailed derivations are in the Appendix).
[Equation]
The LRT hypothesis test rejects H 0 when the LRT statistic is small.
By equation 7, the rejection region { ( θ, z ) : LR ( θ, z ) ≤ c } can be approximated as follows.
[Equation]
The above approximated LRT strategy compares a constant threshold T · log c with the loss (cid:96) ( θ, x z , y z ) of the target model θ on the target data point z .
This recovers the commonly used (constant) loss threshold attacks in literature Nasr et al.
; Sablayrolles et al.
However, by derivations in the appendix (equation 48), a more accurate LRT attack strategy would compare the loss (cid:96) ( θ, x z , y z ) with a threshold function T · log (cid:16)
[Equation]
that depends on the target model θ .
Therefore, the attack in equation 8 with constant threshold c α can overly simplify the LRT, thus limiting its performance.
This motivates our design of attacks with model-dependent and sample-dependent thresholds, with the objective of improving the attack performance.
Our general template for attack construction.
Building on the approximate LRT equation 8, we derive the following variant of sample-dependent and (or) model dependent attack strategy.
[Equation]
where c α ( θ, x z , y z ) is a threshold function chosen by the attacker under α tolerance of false positive rate (FPR), i.e., c α ( θ, x z , y z ) satisﬁes the following equation which controls false positive rate.
[Equation]
In the following attacks, we approximate the joint distribution P H
( θ, z ) with empirical distribution over its samples.
This facilitates solving equation 10, which give valid attack threshold c
α
( θ, x z , y z )
[Equation]
3. A TTACK S: MIA VIA S HADOW MODELS
We ﬁrst formalize a shadow model membership inference attack S based on Shokri et al.
(2017), that effectively uses label-dependent attack threshold c α ( y z ) , as follows.
[Equation]
where the threshold function c α ( y z ) satisﬁes equation 10 and ensures false positive rate α .
To approximate the joint distribution P H
( θ, x z , y z ) in equation [Equation] the attacker samples the following set S of shadow models and shadow data points from the joint distribution P H
( θ, z ) .
[Equation]
The shadow models θ s =
θ i } i =1 , 2 , ··· trained on population datasets approximate the distribution E D ∼ π n [ P ( θ | D )] in equation 3, and the shadow points z
i 1 , z i 2 , · · · for the i -th shadow model approximate the population data distribution π ( z ) in equation 3.
Therefore, we approximate the joint distribution P H
( θ, z ) in equation 10 with the empirical distribution over set S , as follows.
[Equation]
One sufﬁcient condition for equation 15 to hold is that for any ﬁxed data label y ∈ Y , we have
[Equation]
By solving equation 16 for every y ∈ Y , we compute that c α ( y ) equals the the α -percentile for the histogram of loss values (cid:96) ( θ, x z , y z ) over samples { ( θ, z ) ∈ S :
y z = y } .
This recovers the class-dependent attack threshold function c
α ( y z ) that we use in Attack S.
3. A TTACK P: MODEL - DEPENDENT MIA VIA P OPULATION DATA
We design a new membership inference Attack P that uses attack threshold function
c α ( θ ) dependent on the target model θ .
The rationale for this design is to construct an inference attack which exploits the similar statistics as in Attack S, in a more accurate way by computing it on the target model, yet with less computations (without the need to train shadow models).
The hypothesis test is as follows.
[Equation]
where the threshold function c α ( θ ) satisﬁes equation 10 and ensures false positive rate α .
Using equation 23, we rewrite the false positive rate in equation 10 as follows.
[Equation]
One sufﬁcient condition for equation 18 to hold is that, for any ﬁxed target model θ , we have
[Equation]
To approximate the data distribution π ( z ) in equation 19, the attacker samples the following set P of population data points from distribution π
( z ) .
[Equation]
[Equation]
Therefore, we approximate π ( z ) in equation 19 with the empirical distribution over samples z ∈ P .
[Equation]
By solving equation 21 for every ﬁxed target model θ , we compute that c α ( θ ) equals the the α − percentile for the histogram of loss values (cid:96) ( θ, x z , y z ) over population data samples
z
=
( x z , y z ) ∈ P .
This recovers the model-dependent attack threshold c
α
( θ ) for Attack P.
3. A TTACK R: SAMPLE - DEPENDENT MIA VIA R EFERENCE MODELS
The privacy loss of the model with respect to the target data could be directly related to how susceptible the target data is to be memorized (e.g., being an outlier) Feldman (2020).
Based on this ﬁnding, we design the membership inference Attack R that uses attack threshold function
c α ( x z , y z ) which depends on the target data (both its input features x z and the label y z ).
This attack is very similar to the membership inference attacks designed for summary statistics, graphical models and machine learning, which use reference models to compute the probability of the null hypothesis Sankararaman et al.
; Murakonda et al.
[Equation]
; Long et al.
The hypothesis test is as follows.
[Equation]
where the threshold function c α ( x z , y z ) satisﬁes equation 10 and ensures false positive rate α .
Using the chain rule of joint distribution, and the independence of θ and z = ( x z , y z ) under null hypothesis H 0 in equation 1, we prove that
[Equation]
where P H
( θ ) =
E D ∼ π n [ P ( θ | D )] is the target model distribution speciﬁed by the null hypothesis
H 0 .
Using equation 23, we rewrite the false positive rate constraint in equation 10 as follows.
[Equation]
One sufﬁcient condition for equation 24 to hold is that, for any ﬁxed data point z
= ( x z , y z ) ,
[Equation]
To approximate the target model distribution P H
( θ ) in equation 25, the attacker then samples the following set R of reference models.
[Equation]
θ i ∼ T
[Equation]
D
[Equation]
Because the reference models θ r = { θ i } i =1 , 2 , ··· are trained on population datasets, they serve as samples from the distribution P H 0 ( θ ) =
E D ∼ π n [ P ( θ | D )] .
Therefore, we replace P H
( θ, z ) in equation 25 with the empirical distribution over samples θ i ∈ R , as follows.
[Equation]
By solving equation 27 for ﬁxed target sample z = ( x z , y z ) , we compute that c
α ( x z , y z ) equals the the α -percentile for the histogram of loss values (cid:96) ( θ i , x z , y z ) over reference models samples θ i ∈ R .
This computes the sample-dependent attack threshold
c α ( x z , y z ) that in Attack R.
3. A TTACK D: MODEL - DEPENDENT AND SAMPLE DEPENDENT MIA VIA D ISTILLATION
Can we design an attack that takes advantage of all the information available in the target model and the target data that can increase the chance of identifying the right hypothesis?
We design a membership inference Attack D whose threshold function c
α ( θ, x z , y z ) depends on both the target sample z and the target model θ , as follows.
[Equation]
where the threshold function c α ( θ, x z , y z ) satisﬁes equation 10 and ensures false positive rate α .
However, the degree of freedom in the threshold function c
α ( θ, x z , y z ) is still too large for us to directly solve equation 10.
Therefore, we restrict c α ( θ, x z , y z ) to take the following form.
[Equation]
where D θ
=
T − 1 ( θ ) is the training dataset for target model θ .
For the simplicity of derivation, let us ﬁrst assume that the randomized training algorithm T has a deterministic inverse mapping
T − 1 : θ → D , i.e. the training dataset for a given model θ is uniquely speciﬁed.
(Later we also show how to approximate the training datasets D θ for model θ when the training algorithm T is not invertible.)
Plugging equation 29 into equation 10, we rewrite the FPR constraint as follows.
[Equation]
[Equation]
By deterministic mapping from θ to D θ , the chain rule of joint distribution, and the independence between random variables θ and z = ( x z , y z ) under null hypothesis H 0 in equation 1, we prove that
[Equation]
where P ( θ | D θ ) is distribution of trained model under a ﬁxed training dataset D θ , as speciﬁed in equation 5.
Plugging equation 31 into equation
[Equation]
we rewrite the false positive rate constraint as
[Equation]
One sufﬁcient condition for equation 34 to hold is that, for any ﬁxed sample z and target model θ ,
[Equation]
where D θ is an implicitly ﬁxed training dataset for the target model θ , and the distribution P ( θ (cid:48)
| D θ ) captures retrained models on the training dataset D θ
=
T − 1 ( θ ) for given target model θ , as follows.
[Equation]
To approximate this distribution of retrained model T (cid:0) T − 1 ( θ )
[Equation]
∼ P ( θ
[Equation]
| D θ ) , the attacker generates the following set of self-distilled models using the target model θ .
[Equation]
These distilled models M = { θ i } i =1 , 2 , ··· approximate samples from the retrained model distribution T ( T − 1 ( θ )) ∼ P ( θ | D θ ) .
This is because θ i is trained on distillation dataset D i consisting of population data points which are soft-labeled with the target model θ .
This roughly recovers the target model θ trained on D θ , however without its potential dependence on z .
Therefore, the attacker approximate P ( θ | D θ ) in the false positive rate constraint equation 35 with the empirical distribution over distilled models samples in M , as follows.
[Equation]
By solving equation 38 for ﬁxed target model θ and target point z , we compute that c
α
( D θ , x z , y z ) equals the the α -percentile for the histogram of loss values (cid:96) ( θ i , x z , y z ) on distilled models θ i ∈ M .
This recovers the model-dependent and sample-dependent attack threshold c α ( x z , y z ) in Attack D.
3. A TTACK L: L EAVE ONE OUT ATTACK
An ideal attack, that removes the randomness over the training data (except the target data that could potentially be part of the training set) would be the leave one out attack.
In this attack, the adversary trains reference models θ (cid:48) on D \ { ( x b , y b ) } .
The attack would be in the same class of attacks as in Attack D, as it will be a model-dependent and data-dependent attack.
It also runs a similar hypothesis test, however the attack requires assuming the adversary already knows the n − 1 data records in D \ { ( x b , y b ) } .
This is a totally acceptable assumption in the setting of privacy auditing.
Note that Attack D aims at reproducing the results of the leave-one-out attack without assuming the knowledge of n − 1 data records in D \ { ( x b , y b ) } .
3. S UMMARY AND COMPARISON OF DIFFERENT ATTACKS
For identifying whether a data point z has been part of the training set of θ , here are the main underlying questions for the attacks we present in this section:
• How likely is the loss (cid:96) ( θ, z ) to be a sample from the distribution of loss of random samples from the population on (Attack P: the same model) (Attack S: models trained on the population data)?
Depending on the tolerable false positive rate α and the estimated distribution of loss, we reject the null hypothesis.
• How likely is the loss (cid:96) ( θ, z ) to be a sample from the distribution of loss of z on (Attack R: models trained on population data) (Attack D: models trained to be as close as possible to the target model, using distillation) (Attack L: models trained on n − 1 records from D excluding z )?
Depending on the tolerable false positive rate α and the estimated distribution of loss, we reject the null hypothesis.
Effectively, these questions cover different types of hypothesis tests that could be designed for performing membership inference attacks.
We expect these attacks to have a different error due to the uncertainties that can inﬂuence their performance.
Attacks S and P are of the same nature.
However, attack S could potentially have a higher error due to its imprecision in using other models to approximate the loss distribution of the target model on population data.
Attacks R, D, and L are also of the same nature.
However, we expect attacks D to have more conﬁdence in the tests due to reducing the uncertainty of other training data that can inﬂuence the model’s loss.
Thus, we expect attack D to be the closest to the strongest attack which is the leave-one-out attack. [Figure]
E XPERIMENTS
In this section, we empirically measure and study the performance of different attack strategies that we derived from our framework.
A detailed analysis of the internal mechanics of these attacks and how they differ from each other is provided in the Appendix A.2.
Evaluation of attacks performance Which one of the attacks S, P, R, D has the best performance?
How to evaluate the strength of attack besides attack accuracy?
We quantify the attacker’s performance using two metrics: its true positive rate (TPR), and its false positive rate (FPR), over the random samples from member and non-member data and random target models.
The ROC curve captures the tradeoff between the TPR and FPR of an attack, as its threshold c α is varied across different FPR tolerance α .
The AUC (area under the ROC curve) score measures the strength of an attack.
Therefore, we plot the ROC curves of all attacks on the Purchase100 dataset, and computes their AUC (area under the ROC curve) score in Figure 3 (in the appendix).
The attack with the highest AUC score on Purchase100 is Attack D, which has the least level of uncertainty, as discussed
Section 3.
Table 1 and Table 4 shows more detailed AUC score results for different attacks under more settings.
Comparison between the Attacks Besides attack strength, how differently are the attacks performing on random input target models and target points?
How do the different internal mechanics of attacks changing the attacker’s guess qualitatively?
Do the attacks succeed on similar or different samples of member data and target model?
Do the attacks have different level of conﬁdence on the same input?
How far away are the attack performance from the most ideal leave-one-out attacks described in Section 3?
Answers to these questions require understanding how and why the attacks perform differently, for which we do detailed comparisons between attacks as follows.
[Table]
• Similarity of attacks with each other.
The scatter plot comparing Attack S and P in Figureis roughly linearly centered around the diagonal, with slightly more points in NorthWest then in SouthEast.
This shows that Attack S and Attack P almost always guesses membership of train points similarly, while Attack P performs slightly better than Attack S.
Meanwhile, Attack D dominates Attack R for correctly guessing membership of train points because their comparison scatter plot is biased towards the NorthWest direction.
• Gap between attacks to the ground truth.
From Table 2, among all attacks, Attack D agrees with the ground truth the most on train points, while the least on test points.
This matches our observation that Attack D has a larger threshold under given α from Figure 1, which causes it to guess both more points as members.
Besides that, we see that the agreement rate between Attack S and Attack P is as high as 0 .
94 both on train points and test points, this matches their linear comparison scatter plot in Figure 5.
This may be because Attack P and Attack R both reduces one degree of uncertainty for the joint distribution under null hypothesis H 0 , as discussed in Section 3.
• Closeness of attacks to ideal leave-one-out attack.
One interesting observation from Tableis that, among all the attacks, Attack D agrees with the Attack L the most, with agreement rate 0 .on train points and 0 .
82 on test points.
We believe this is because Attack D is highly similar in nature with Attack L, by approximating the training dataset of a target model and performing retraining, as discussed in Section 3. GT L S
P
R D GT 0. L 0.
S 0.
P 0.
R 0.
[Table]
The experimental setup is Purchase100 Conﬁguration IIa, with α = 0 . 3 .
S UMMARY
We provide a framework for auditing the privacy risk from machine learning models through membership inference attacks.
The framework is used to derive attack strategies and also highlight the factors beyond leakage from the models that affect the attack performance.
We also empirically analyze the performance of these attack strategies against models trained on benchmark datasets.
A A PPENDIX A. D ETAILED DERIVATION OF APPROXIMATED LRT FOR MEMBERSHIP INFERENCE
We ﬁrst prove two useful approximation inequalities about the posterior distribution P ( θ | D ) as follows.
For arbitrary data point z =
( z x , z y ) , and arbitrary dataset D , we have
[Equation]
Let z 1 , z 2 , · · · , z n be i.i.d.
samples from the data distribution π ( z ) .
Then when n is large enough, for any model parameter θ , we have
[Equation]
We now offer details for deriving the approximated likelihood ratio test (LRT) for membership inference.
Deﬁnition A.1 (Approximated LRT for membership inference) Let ( θ, z ) be random samples from the joint distribution of target model and target data point, speciﬁed by one of the following membership hypotheses.
[Equation]
The likelihoods function of hypothesis H 0 and H 1 given observed target model θ and target data point z is as follows. [Equation]
[Equation]
Therefore the LRT statistics is
[Equation]
The LRT hypothesis test rejects H 0 when the LRT statistic is small.
By equation 6, the rejection region { ( θ, z ) : λ ( θ, z ) ≤ c } can be approximated as follows.
[Equation]
Datasets
• The Purchase100 dataset is based on Kaggle’s ”Acquire Valued Shoppers Challenge” that contains shopping histories for thousands of individuals 4 .
We use a simpliﬁed preprocessed purchase dataset provided in Shokri et al.
There are 197 , 324 data points in this dataset, where each data point has 600 binary features.
The data points are clustered into 100 classes to represent different shopping style.
• The CIFAR10 and CIFAR100 datasets are widely used benchmark datasets for image classiﬁcation.
The CIFAR10 dataset consists of 60 , 000 data points in 10 classes, whereas the CIFAR100 dataset consists of 60 , 000 data points in 100 classes.
Here each data point is a 32 × 32 color image, and there are 50 , 000 training images and 10 , 000 test images in total.
• The MNIST dataset is a widely used benchmark dataset for image classiﬁcation.
The MNIST dataset consists of 70 , 000 data points in 10 classes, where each data point is a 28 × 28 handwritten digit image.
In total, there are 60 , 000 training images and 10 , 000 test images.
Experiment setup
For each conﬁguration, we train up to n = 1000 models on random IID splits of the data.
we use layer units
For CIFAR100 and MNIST conﬁgurations, we use a 2 layer CNN with ﬁlters =
[32, 64] and max pooling.
For CIFAR10 conﬁgurations, we use AlexNet and a 3-block VGGNet.
For all target models and datasets, we use SGD as the optimization function and categorical crossentropy loss.
For Conﬁguration IV of Purchase100, we use gradient clipping while training the models.
For CIFAR10, we add momentum while training the models.
Target models: We use the ﬁrst 10 models as target models on each of the Purchase100, CIFAR10, CIFAR100 and MNIST datasets.
The performance of Attacks S, P, and R are evaluated on all 10 models, whereas Attack D is evaluated on the ﬁrst 3 models.
Shadow models for Attack S and Reference models for Attack R:
Given a target model, we use the remaining ( n − 1) = 999 models as shadow models or reference models for Attack S and Attack R respectively.
Distilled models for Attack D:
Given a target model, we use the same random IID splits used to train n = 1000 models to distill ( n − 1) = 999 models using the soft labels from the target model.
Using n < 1000 models: For Conﬁguration IV of Purchase100 and Conﬁguration III of CIFAR10, we report the attack performance results using n = 30 shadow, reference, and distilled models. For Conﬁgurations I and II of CIFAR10 we report the attack performance results using n = 400 shadow, reference, and distilled models.
A. A DDITIONAL E MPIRICAL E VALUATION R ESULTS
In this section we provide a detailed analysis of the internal workings of the attack strategies.
We speciﬁcally compare how the thresholds on loss values derived from these strategies differ on various target points and target models.
Illustration of Attack threshold How differently does the attack threshold in Attack S, P, R, D depend on the target model θ and the target data point z ?
How is the different dependence of threshold on θ and z in different attacks affecting their success?
To investigate these important questions, we plot the loss histograms that different attacks use to compute the thresholds, on different input θ 1 , θ 2 and z 1 , z 2 in Figure 1.
The loss histogram approximates the LRT statistics distribution under null hypothesis H 0 in equation 1, with different uncertainty for different attacks, as discussed in 3.
We see that model-dependence and sample-dependence of attack threshold reduces the uncertainty in the loss histogram.
The attacks (Attack D) with more concentrated loss histogram are more likely to succeed. [Table]
B R ELATED W ORK
The current work in this domain can be broadly grouped into three categories: 1) Empirical works for improving the existing attack strategies or adapting them to different settings and models 2)
Theoretically/empirically analyzing the privacy risk in various systems using the existing attack strategies 3) Exploring the connections with differential privacy and using them to establish lower bounds on leakage and/or select privacy parameters.
Below, we provide a brief summary of works in all the three categories.
[Figure]
[Table]
[Table]
[Figure] [Figure]
Empirical attack strategies for membership inference:
Shokri et al.
(2017) demonstrated the vulnerability of machine learning models to membership inference attacks in the black-box setting, where the adversary has only a query access to the target model.
The attack algorithm is based on the concept of shadow models, which are models trained on some attacker dataset that is similar to that of the training data.
Membership inference is modeled as a binary classiﬁcation task for an attack model that is trained on the predictions of shadow models on the attacker dataset.
A huge literature followed this work extending the attacks to different setting like white box access (Nasr et al., 2019;
Sablayrolles et al., 2019; Leino & Fredrikson, 2020), label-only access (Li & Zhang, 2020;
Choquette-Choo et al., 2021), federated learning (Melis et al., 2019), transfer learning (Zou et al., 2020) and different types of data, models such as aggregate location data (Pyrgelis et al., 2017), generative models (Hayes et al., 2019), language models (Carlini et al., 2019; Song & Shmatikov, 2019; Carlini et al., 2020), sentence embeddings (Song & Raghunathan, 2020), and speech recognition models (Shah et al., 2021).
Multiple works have looked at improving the attack methodology through a more ﬁne grained analysis or by reducing the background knowledge and the compute power required to execute the attack (Long et al., 2018; Song & Mittal, 2021; Salem et al., 2018).
See (Hu et al., 2021) for a more comprehensive list of membership inference attacks against machine learning models.
All these works follow the same attack framework for membership inference but they either exploit a slightly different signal that is correlated with membership of a point in the training set or ﬁnd an efﬁcient way to exploit the already known signals.
[Figure]
Privacy risk analysis with membership inference attacks:
Homer et al.
(2008) performed the ﬁrst membership inference attack on genome data to identify the presence of an individual’s genome in a mixture of genomes.
Sankararaman et al.
(2009) provided a formal analysis of this risk of detecting the presence of an individual from aggregate statistics computed on independent and binary attributes.
Murakonda et al.
(2021) extended this analysis to the case of releasing
Discrete
Bayesian networks learned from data with dependent attributes.
Backes et al.
(2016) perform an analysis similar to that of (Sankararaman et al., 2009) but for MicroRNA data (aggregate statistics computed on independent and continuous attributes).
Dwork et al.
(2015) provide a more extensive analysis when the released statistics are noisy and the attacker has only one reference sample to perform the attack.
The key results of all these works establish the privacy risk of releasing aggregate statistics by quantifying the success of membership inference attacks as a function of the number of statistics released and the number of individuals in the dataset.
Similar attempts were made to analyze the privacy risk of machine learning models through membership inference via the lens of mutual information (Farokhi & Kaafar, 2020) and generalization error (Yeom et al., 2018; Del Grosso et al., 2021).
Beyond these theoretical analyses, membership inference attacks are also used to empirically study the trade-offs between privacy and other desirable characteristics for machine learning models such as fairness (Chang & Shokri, 2020), robustness to adversarial examples (Song et al., 2019), and providing explanations (Shokri et al., 2021).
Differential privacy and Membership inference: The deﬁnitions of differential privacy (Dwork et al., 2006) and membership inference (Homer et al., 2008; Dwork et al., 2015; Shokri et al., 2017) are very closely connected and the hypothesis testing interpretation of differential privacy provides a clear view of the relationship between them.
Satisfying differential privacy is equivalent to imposing a bound on the ability to distinguish any two neighboring datasets that differ by the presence of one individual i.e., inferring about the presence/absence of the individual.
The bound is stated as a tradeoff between the type-I and type-II errors in distinguishing the two neighboring datasets.
Hypothesis testing interpretation of differential privacy is very useful in deriving tight compositions (Kairouz et al., 2015) and has even motivated a new relaxed notion of differential privacy called f-DP (Dong et al., 2019).
By deﬁnition, differentially private algorithms bound the success of membership inference attacks.
Multiple works (Yeom et al., 2018; Erlingsson et al., 2019; Humphries et al., 2020), each improving on the previous work, have provided upper bounds on the success of membership inference attacks as a function of the parameters in differential privacy.
Jayaraman & Evans (2019) evaluated the performance of membership inference attacks on machine learning models trained with different values of epsilon under different relaxed notions of differential privacy.
Rahman et al.
(2018) also use membership inference attacks to measure the privacy loss on models trained with differentially private algorithms.
The empirical performance of membership inference attacks has also been used to provide lower bounds on the privacy guarantees achieved by various differentially private algorithms (Jagielski et al., 2020; Nasr et al., 2021; Malek et al., 2021).
The key difference between the empirical analysis of membership inference in the previous three works and other works is that they simulate the exact adversary in differential privacy i.e., they train multiple models with and without one particular training point and keep the rest of training set ﬁxed.
The performance of the attack (type I and type II errors) is averaged over these models, whereas in the previous works the model is ﬁxed and the performance is computed as an average over points in the training and test sets.
This simulation of the DP adversary helps in removing the effects of other points in the dataset when measuring the leakage through the model about a particular point of interest.