Membership Inference Attacks via Adversarial Examples
Abstract
The rise of machine learning and deep learning has led to signiﬁcant improvements in several areas.
This change is supported by both the dramatic increase in computing power and the collection of large datasets.
Such massive datasets often contain personal data that can pose a threat to privacy.
Membership inference attacks are an emerging research direction that aims to recover training data used by a learning algorithm.
In this work, we develop a means to measure the leakage of training data leveraging a quantity appearing as a proxy of the total variation of a trained model near its training samples.
We extend our work by providing a novel defense mechanism.
Our contributions are supported by empirical evidence through convincing numerical experiments.
Introduction
At the forefront of artiﬁcial intelligence (AI) the beneﬁts and outstanding performance of modern machine learning cannot be overlooked.
Through the lens of supervised learning, computers can now compete with humans regarding vision [28, 30, 38, 61], object recognition [16] or medical image segmentation [39].
Unsupervised and semi-supervised learning techniques led to an impressive development of natural language processing [18, 36, 37, 76] or language understanding [12, 24, 81, 84].
More recently, reinforcement learning [70] has reached superhuman levels in a wide range of tasks such as games [54, 67], complex biological tasks [40] or even smart devices and smart cities [55].
Most of the aforementioned methods use Deep Learning [43] to achieve peak performance.
However, these methods usually require a large amount of data to achieve generalization [2, 22, 45, 75] and nowadays large data sets used in the era of so-called big data [83] can be easily collected.
Information Disclosure [44] describes the involuntary leakage of information from a provider ( e.g. a website database, an API or even a Deep Learning model).
Machine learning models used in production do not necessarily provide fairness [53, 78], security [4, 11] or privacy considerations [35].
Before claiming that leakage of sensitive information is not a bug, but a feature [26], privacy protection of data providers to machine learning models has become necessary.
Indeed, it is required under recent data regulation (see CCP [1], Villani et al.
[77] and Regulation [62], van der Burg et al.
It is a growing direction of research [17, 52].
Recent contributions from the ﬁelds of deep learning [6, 10, 63] and privacy and security [21, 47, 64] ∗ Equal contribution. show that neural networks and deep models are likely to behave like interpolators in high dimensions.
Considered benign for generalization [9, 49, 72], overﬁtting and thus overconﬁdent wrong predictions can be harmful [4, 64] and uncalibrated [60, 79].
Sharing data and model in a private way is a major barrier to the adoption of new AI technologies [58].
Therefore, ensuring that data-driven technologies respect important privacy constraints and government regulations is at the core of modern AI security [25].
In this way, this paper attempts to measure the extent of information disclosure or data leakage that a malicious attacker exploiting the vulnerabilities of a given model can potentially recover.
Defense mechanisms that prevent or minimize information leakage are desirable tools to ensure the conﬁdentiality of training data.
We propose a defense mechanism that prevents the leakage of training data with membership inference attacks based on adversarial strategies.
Contributions.
(i)
We describe a novel framework for membership inference attacks (MIA) using adversarial examples with general functionals in the output space; (ii) We present a particular method based on the total variation of the score function; (iii) We propose a defense mechanism against MIA leveraging adversarial examples deferred to the supplementary material.
Outline.
Section 2 introduces the relevant background for the analysis.
Section 3 provides our novel
MIA framework using counterexamples in the output space.
Section 4 gathers numerical experiments.
Finally, Section 5 concludes our work with a discussion of further avenue.
The proofs and relevant material are given in the Appendix.
Framework
Statistical Learning & Pattern Recognition.
Statistical learning [29] is a branch of machine learning that deals with the problem of statistical inference, i.e., constructing a prediction function from data.
In pattern recognition [14], a discrete random label Y valued in Y , with cardinality arbitrarily set to K > 1 , is to be predicted based on the observation of a random vector X valued in an input space
X using the classiﬁcation rule minimizing a loss function.
To perform pattern recognition, the empirical risk minimization (ERM) learning paradigm [74] suggests to solve
[Equation]
where the minimization is done over a class of classiﬁers G and the risk E { (cid:96) ( g ( X ) , Y ) } is associated with a loss (cid:96) measuring the error between the prediction g ( X ) and the label Y .
Since the joint distribution P X,Y is generally unknown, one may rely on a training dataset D train =
{ ( X i , Y i ) } ni = consisting of n ≥ 1 i.i.d copies of ( X, Y ) to minimize the empirical risk
[Equation]
In the case of deep neural networks, the empirical risk minimization can be rewritten as minimization over a set Θ corresponding to the set of weights associated with a given neural network architecture G .
In full generality, the minimization should occur over G × Θ , although for the sake of simplicity, it is assumed that the deep models’architecture is ﬁxed and well-designed.
A classiﬁer g θ ∈ G × Θ yields a score distribution on the label set P ( Y ) that hopefully approximates the probability distribution P ( Y | X = x ) [33].
Consistently with our notation, let (cid:98) g θ denote a minimizer over Θ of the risk (cid:98) R ( g θ ) .
Membership Inference Attack.
To measure the leakage of information of a model, the authors of [65] introduced the paradigm of Membership Inference Attacks (MIA).
The goal of MIA is to determine whether a given labeled sample ( x, y ) ∈
X × Y belongs to the set of training data D train of a classiﬁer (cid:98) g θ .
Without loss of generality, most MIA strategies determine whether a sample was used to train a particular model based on decision rules involving indicators functions.
Therefore, the MIA strategies can be written as follows
[Equation]
where s : X (cid:55)→ R is a score function and τ ∈ R is a threshold.
The score function s may rely on the loss value [65], the gradient norm [66, 82], or on a modiﬁed entropy measure [68, 69].
Adversarial Examples.
Evasion attacks [7, 8, 13, 51] of classiﬁers are referred to adversarial attacks when dealing with neural networks [13].
As a tool that contributes to a better understanding of the underlying estimators, adversarial strategies are attracting increasing interest in various areas of machine learning such as computer vision [3], reinforcement learning [42], and ranking data [31].
Two types of adversarial strategies can be considered: targeted attacks and untargeted attacks .
For a given sample x ∈ X labeled by a classiﬁer (cid:98) g θ as y ∈ Y , a targeted attacker ψ with target label (cid:101)
y (cid:54) =
y builds a carefully crafted noise ε ∈ R d such that x + ε is predicted as (cid:101) y .
On the other hand, regarding untargeted attacks, an attacker ψ will generate a noise ε ∈ R d for a given sample x ∈ X such that: arg max (cid:98) g θ ( x ) (cid:54) = arg max (cid:98) g θ ( x + ε ) .
Additional constraints on the noise ε are common to provide the smallest perturbation that gives the desired results.
In the numerical experiments of
Section 4, adversarial examples are built using the FAB adversarial strategy [19] from Torchattacks library [41].
One such scheme aims to provide the smallest perturbation ε to fool the target model (cid:98) g
(see Figure 5 in Appendix C.1)
Learning with noisy labels.
The authors of [56] introduced the formal framework of learning with noisy labels.
They studied the binary classiﬁcation problem in the presence of random classiﬁcation noise and provided a surrogate objective.
Instead of seeing the true label, a learner sees a sample with a label that has been ﬂipped independently.
To the best of our knowledge, there is no paper that relies on injecting voluntary noise into the labels to prevent deep models leakage with peaks of conﬁdence on training data samples.
MIA Methodology
This section provides insights about the methodology behind MIA with adversarial examples.
First, we motivate the strategy by using synthetic data to show that trained classiﬁers exhibit local peaks of conﬁdence around the training samples.
These peaks are then exploited to perform MIA using general functionals in the output space.
3.1 Motivations
[Figure]
Excess of conﬁdence.
The starting point of the method is based on the following conjecture: ”trained classiﬁers are over-conﬁdent on their training datasets because they exhibit local peaks of conﬁdence around the training samples”.
More precisely, we argue that the shape of the landscape of a trained binary classiﬁer consists of high hills and valleys around the training samples and ﬂatter curves for new, unseen data.
While Figure 1a depicts the landscape of a binary classiﬁer in a cut-away view, its general shape takes the form described in Figure 1b, where the scores are sharp around samples from the train set and ﬂat around the new unseen data.
To highlight such behavior with peaks of conﬁdence , we place ourselves in the classiﬁcation framework with synthethic data.
The goal here is to assess the following statement: for a trained classiﬁer, a small amount of noise is sufﬁcient to change the distribution of the score function on the training data, whereas a larger amount of noise is required to change the distribution of the score function on new unseen data.
Intuitively, more noise should be required near a training sample to fool a classiﬁer, as the sample must move farther away from the local excess of conﬁdence.
Synthetic framework.
Consider the multiclass classiﬁcation task with a dataset D consisting of n ≥ 1 observations X 1 , . .
X n ∈ R d along with their associated labels y 1 , . . .
, y n ∈ { 1 , . .
. , K } .
The data is split into a training set D train and a testing set D test .
For ease of notation, denote by D a with a ∈ { train , test } the different datasets.
For each class k ∈ { 1 , . . .
, K } , denote by D a ( k ) =
{ ( X, y ) ∈ D : X ∈ D a , y = k } the elements of D a which are labeled k and n a ( k ) =
(cid:80) ( X,y ) ∈D a 1 { y = k } its cardinal number.
For k ∈ { 1 , . . .
, K } , denote by ¯ X a ( k )
=
(cid:80) X ∈D a ( k ) X/n a ( k ) the barycenter of class k and consider the average pairwise intracluster distance δ a ( k ) deﬁned by δ a ( k )
=
(cid:80) n a ( k ) i,j =1 (cid:107)
X i − X j (cid:107)
(2( n a ( k ) − 1) n a ( k )) .
Adversarial examples.
Each sample X is gradually modiﬁed by a random noise ξ ( t ) and the distribution of the scores ρ X : t (cid:55)→ (cid:107) (cid:98) g θ
( X + ξ ( t ))
(cid:107) ∞ are to be compared.
The random noise is a
Gaussian anisotropic noise directed towards the closest barycenter of X that is differently labeled.
More precisely, given a sample X ∈ D a ( k ) , consider the barycenter ¯ X a ( l ) which is the closest to
X and has a different label, i.e. , l = arg min j (cid:54) =
k (cid:107) X − ¯ X a ( j ) (cid:107) 2 .
The underlying unit direction is v =
( X − ¯
X a ( l )) /
(cid:107) X
− ¯ X a ( l ) (cid:107)
and the sample X is modiﬁed as
[Equation]
where ε t ∼ N (0 , tδ a ( k )) controls the level of noise required to disturb the sample.
For each value of t , denote by D t train and D t test the corresponding datasets consisting of the modiﬁed samples X t .
For different values of t , univariate Kolmogorov-Smirnov (KS) tests are performed to compare the distributions of the scores on the initial data ρ X (0) = (cid:107) (cid:98) g
( X ) (cid:107) ∞ and their noisy counterparts ρ X ( t )
= (cid:107) (cid:98) g
θ ( X + ξ ( t ))
(cid:107) ∞ for both the train and test sets.
Furthermore, we also perform bivariate two sample KS tests (see [27]) directly on the data from the same pairs (cid:0) D train , D t train ) and (cid:0) D test , D t test ) to ensure that we do not disturb the data too much.
The p -values are averaged over the total number of classes in their respective pairs.
It is straightforward that as the value of t increases, the classiﬁer’s score on the initial data and the score on the noisy data are getting increasingly different, hence the p -value of the involved tests decreases.
Although, in the idea that trained classiﬁers present local conﬁdence peaks, it is expected that the p -values for the KS test on ( ρ X (0) , ρ X ( t )) for D train decrease faster than the respective ones for D test .
Results.
Figure 2 (Left) shows the evolution of p -values from the KS tests performed on ρ X (0) and ρ X ( t ) where the number of classes equals 4 .
As expected, the distributional gap between the original training data D train and its noisy counterpart D t train is more remarkable, the more noise is added compared to the test data D test and its noisy counterpart D t test , as suggested by the lower median p -values.
The boxplots in Figure 2 (Right) provide the evolution of the p -values from a bivariate KS test directly computed on the input data to ensure that the procedure does not excessively disrupt the data.
Such a test describes the evolution of the distributional inﬂuence of the noise level t .
Note that the decay rate of p -values with t is similar between ( D train , D t train ) and ( D test , D t test ) .
This behavior along with the left-hand boxplots conﬁrm that the changes in the output space do not depend on the perturbations made in the input data, but rather on the conﬁdence peaks in the training data.
As a consequence of this ﬁrst motivational experiment with some simplistic noise as adversarial strategy, the local excess of conﬁdence of a classiﬁer transcribes the underlying presence of training data.
As a result, the output manifold of the classiﬁer can be scrutinized to perform membership inference attacks..
3.2 Arc length & Total Variation
In order to measure what happens on the manifold of predicted values, we consider curve lengths along adversarial paths.
For any real-valued continuous function f , the total variation is a measure of the one-dimensional arc length of the curve with parametric equation t (cid:55)→ f ( t ) .
Deﬁnition 1 (Total variation for univariate function R ) Consider a real-valued function f deﬁned on an interval [ a, b ] ⊂ R and let Π denote the set of all partitions i.e. Π = { π =
[Figure] ( x , x
) s t = x 0 <
<
= b } of the [ a, b ] is given by V [ a,b ] ( f ) =
sup π ∈ Π (cid:80) nk =
1 | f ( x k ) − f ( x k − 1 ) | .
If f is differentiable, we have V [ a,b ] ( f ) =
(cid:82) b a | f (cid:48) ( t ) | d t.
Output space functionals.
Given a classiﬁer g ∈ G and a sample x ∈ X , standard MIA strategies determine whether x is part of the training set or not using 1 { s ( x ) ≥ τ } where s : X (cid:55)→ R is a score function.
This scheme can be extended to account for the information brought in by adversarial examples.
For any sample x ∈ X with associated adversarial example (cid:101) x ∈ X , a general MIA strategy may be written as
[Equation]
where ϕ : G × X × X → R is a functional to be speciﬁed.
When using (cid:96) p -norms in the input space as ϕ ( (cid:98) g θ , x, (cid:101) x ) = (cid:107) (cid:101) x − x (cid:107) p , the rule of Equation (1) recovers the framework of [21].
Such distances denoted as adversarial distances by [21] are only concerned by the input space X and do not directly take into account the behavior of the classiﬁer (cid:98) g θ and its predicted values.
Instead, we advocate for functionals ϕ : G × P ( Y ) × P ( Y ) that focus on distances in the output space.
Motivated by the illustration of Figure 1b, we argue that more attention is needed on the manifold of predicted values g ( X ) rather than the aforementioned adversarial distances .
Indeed, the landscape is more likely to capture the high conﬁdence levels associated to the samples used during the training phase.
Such behavior is depicted in Figure 1c below where the arc length of the adversarial path better transcribes the local excess of conﬁdence than the adversarial distance.
Consider x ∈ X and its associated minimal adversarial counterpart (cid:101) x
= x + ε .
The adversarial path is the path connecting the max scores of x and (cid:101) x along the manifold of predicted values.
These paths are characterized by the following family of parameterizations.
Deﬁnition 2 (Adversarial paths)
The family of adversarial paths is parameterized by ( γ x ) x ∈X where for all x ∈ X , γ x : [0 , 1] → [0 , 1] with γ x : t (cid:55)→ (cid:107) ˆ g θ ( x + tε ) (cid:107) ∞ .
Approximated total variation.
The associated arc length of an adversarial path is L ( γ x ) = (cid:82)0 | γ (cid:48) x
( t ) | d t .
Using deﬁnition 2, the MIA rule we propose is given by
[Equation]
However, the underlying score function in Equation (2) may be intractable in practice.
Hence, one needs to rely on an estimate version of such curvature length.
Consider N ∈ N \ { 0 } and the partition sequence ( t k ) k<N given by [-
-]0 =
t − 1 = t 0 < . . .
<
t N = 1 where t k = k/N .
We approximate the arc length L ( γ x ) by L N ( γ x ) = (cid:80) N − 1 k =0 | γ x ( t k ) − γ x
( t k − 1 ) | .
Accordingly, Algorithm 1 provides the pseudocode of our MIA strategy based on the adversarial path and its approximated curvature length.
Note that with Riemann sums, we have L N ( γ x ) →
L ( γ x ) as N → + ∞ .
More precisely we have the following upper bound whose proof is deferred to Appendix A.
Proposition 1 Assume γ x ∈ C 1 ([0 , 1]) and denote ( I j )
j ∈
J the intervals where γ x is strictly monotone.
If the subdivision ( t k ) Nk =1 satisﬁes sup Nk =1 ( t k − t k − 1 ) ≤ inf j ∈ J | I j | , then we have
[Equation]
Remark
If (cid:98) g θ is Lipschitz then the framework of [21] is covered by Equation (2) .
Numerical Experiments
Hereafter we focus on numerical experiments regarding MIA.
Further numerical experiments illustrating a defense mechanism are deferred to Appendix D.3.
4.1 Experimental Setting
MIA Baseline.
We place ourselves in the experimental framework of [21] and compare their MIA strategy to Sisyphos (Algorithm 1) as their method is essentially the main white box comparable adversarial example-based membership inference attack.
Adversarial examples are built with FAB adversarial strategy [19].
Experiments conducted in Appendix C and Appendix E illustrate and discuss broader experimental frameworks.
Target Dataset.
The real world dataset CIFAR10 consists of 60 , 000 colored images of size 32 × 32 associated with 10 different labels.
These data are divided into two subsets: the training set containing 50 , 000 images and the 10 , 000 remaining samples belonging to the test set which corresponds to the unseen data.
Target Neural Networks.
We consider the Resnet model [34] publicly available in the following repository 2 .
For the sake of comparison, we take the same weights as used in [21] to perform MIA.
4.2 Results
Table 1 and Figure 3 provide the comparison’s result of Sisyphos and [21].
Based on our MIA experiment, Sisyphos outperforms the method of [21] regarding both AUC and accuracy.
1.0 FPR 0. T P R adversarial_distancesisyphos MIA Strategy Resnet
AUC Accuracy [Table] [Table]
[Table] (Ours)
[Table]
Conclusion and Future Works
We addressed the problem of membership inference attacks based on adversarial examples relying on general functionals.
Classiﬁers’ excess of conﬁdence on their training examples poses a privacy threat.
We introduced a defense mechanism mitigating the risks of attacks using adversarial examples.
Future work will explore relevant privacy preserving means with gradient ﬂows [5, 50] to provide a better understanding of learning with privacy.
The MIA risks of classiﬁers partitioning the input space with hyperplanes [15, 57] will be further investigated.
Acknowledgment.
This research was supported by the ERC project Hypatia under the European Unions Horizon 2020 research and innovation program. Grant agreement No. S UPPLEMENTARY MATERIAL : M EMBERSHIP I NFERENCE A TTACKS VIA A DVERSARIAL E XAMPLES
Appendix A gathers some mathematical proofs.
Appendix B provides additional details concerning the synthetic framework presented in Section 3.
Appendix C collects a detailed comparison of different MIA strategies, especially regarding the construction and properties of the adversarial noise built in popular adversarial attacks libraries.
Appendix D presents our defense mechanism while
Appendix D.3 is a numerical illustration of the defense mechanism with the label noise injection.
Finally, Appendix E invites to MIA risks based on adversarial examples in a broader scope.
A Proofs
Appendix A gathers proofs deferred from the main paper.
A.1 Proof of Proposition
PROOF .
Since γ (cid:48) x is continuous, note that | J | < ∞ .
In addition, for any j ∈ J consider K j = { k ∈ { 1 , . . .
, N } : ]
t k − 1 , t k [ ⊂ I j } and deﬁne ¯ K = { 1 , . . .
, N } \ ( ∪ j ∈ J K j ) .
Since γ x is supposed in
C 1 ([0 , 1]) , using Taylor’s theorem we obtain γ x ( t k ) − γ x
( t k − 1 ) = (cid:90) t k t k −γ (cid:48) x ( t )d t.
Thus, we deduce that
L N ( γ x ) =
N (cid:88) k
= | (cid:90) t k t k −γ (cid:48)
x ( t )d t | = (cid:88) j ∈ J (cid:88) k ∈
K j | (cid:90) t k t k −γ (cid:48) x
( t )d t | + (cid:88) k ∈ ¯ K | (cid:90) t k t k −γ (cid:48) x ( t )d t | =
L ( γ x ) + (cid:88)
k ∈ ¯ K (cid:32) | (cid:90)
t k t k −γ (cid:48) x (
[Equation]
Hence, we have
[Equation]
For any k ∈ ¯ K , either ] t k − 1 , t k [ ∩ ( ∪ j ∈ J I j ) = ∅
but it yields (cid:82) t k t k − 1 | γ (cid:48)
x | = 0 ; or there exists j ∈ J such that ] t k − 1 , t k [ ∩ I j (cid:54) = ∅ and therefore t k − 1 < sup I j < t k , but using sup Nk =1 ( t k − t k − 1 ) ≤ inf j ∈ J | I j | ensures the uniqueness of j ∈ J satisfying ] t k − 1 , t k
[ ∩ I j (cid:54) = ∅ .
Hence, combining (3) with the previous line gives
[Equation]
which concludes the proof.
A.2 Proof of Remark
PROOF .
The decision rule of the proposed approach is D ( x ) = 1 { L ( γ x ) ≥ τ } .
Using that (cid:98) g θ is λ -Lipschitz, it is straightforward to make the link between the input and output space as follows
[Equation]
=
[Equation] = [Equation] which recovers the decision rule of [21] of the form x (cid:55)→ 1 {|| ε || ≥ τ (cid:48) } with τ (cid:48)
= τ/λ .
Note that in practice and in the experiments conducted in Section 4, the values of τ can be optimized through cross validation.
B Further Details on Motivational Experiments
[Figure]
In Section 3, the simulated data is generated with make classification function from scikit learn [59], the number of cluster per class is set to 1 as illustrated in Figure 4.
The number of sample is set to 2000 .
The class of classiﬁer G is the class of multi-layer perceptrons with 2 hidden layers of size 10.
The number of iterations is set to 5000 to guarantee convergence of the classiﬁer’s weights.
The boxplots are obtained over 50 iterations of the experiment.
C Comparing MIA based on Adversarial Strategies: an Ablation Study
In this section, we present the main difference of the MIA strategies mentioned in the paper, namely
Sisyphos (Algorithm 1) and [21].
First, we recall that [21]’s MIA strategy corresponds to a speciﬁc case of functional regarding the MIA framework introduced in this paper.
In a similar way as results depicted in [46], through an ablation study we analyze the inﬂuence of three elements.
First, we study in Section C.1, the inﬂuence of the adversarial strategy.
In Section C.2, we focus on the inﬂuence of the experimental framework.
Finally in Section C.3, the inﬂuence of the classiﬁer pre-trained model is discussed.
C.1 Inﬂuence of the Adversarial Strategy
Hereafter the difference between two adversarial strategies are depicted.
[21] leverages adversarial examples based on AutoAttack [20].
In Experiments from Section 4, the chosen adversarial strategy is FAB [19].
Let ( x i , y i ) denote an image sample x
i from CIFAR10 with label y
i .
Let ε i be some adversarial noise fooling Resnet neural network on x i obtained either with FAB or AutoAttack .
Figure 5 reports the results of Resnet’s accuracy (%) on the partially noisy samples x i + kN ε i .
It appears that the partial noise kN ε i is more likely to fool Resnet classiﬁer with ε i is obtained with
AutoAttack than FAB .
Hence one may prefer to work with FAB to obtain minimal noise required to perform some adversarial strategy.
[Figure]
p e r c e n t
FABAutoAttack
C.2 Inﬂuence of the Experimental Framework
To train neural network on images, padding or image rotation are common augmentation schemes [80].
Without loss of generalization, one may assume that some random input image to be classiﬁed -after the training phase- is not padded.
Figure 6 shows examples of CIFAR10 images without padding and their respective counterpart with padding or rotation.
[Figure]
Figure 7 and Figure 8 represent the adversarial noise obtained by an untargeted adversarial attack.
Note that some samples in Figure 7 and Figure 8 appear as minor noise ( i.e. dark sample) since no noise is required to change the class from the true class, it implies that the classiﬁer did not successfully classify the input image.
Experiments conducted in Section 4 reproduce the experimental framework of [21] 3 .
Figure 9 is the counterpart of Figure 6 in the framework where padding on targeted training data is removed, FAB remains the adversarial strategy for both MIA strategy.
Comparing the performance of both MIA strategies on both ﬁgures, it appears that padding and other preprocessing on train data may favor the success of membership inference attackers, more especially MIA strategy relying on adversarial examples inducing large change in the input (padded) data.
In both settings, Sysiphos (Algorithm 1) outperforms [21]. C
3 Inﬂuence of the classiﬁer’s pre- trained model
In this section, we assess the inﬂuence of Resnet classiﬁer’s pretraining and weights on the success of
MIA.
The PytorchCV public repository 4 provides the weights of Resnet neural networks which are obtained independently from the weights used in Section 4.
Figure 10 is the counterpart of Figure 3 where Resnet’s weights are replaced with the weights from
PytorchCV.
Comparing results from both ﬁgures, one can conclude that the target model pretraining has inﬂuence on the success of MIAs.
1.0 FPR 0.
T P R adversarial_distancesisyphos
D Towards a Defense Mechanism D.1 Privacy by Design
Most obfuscation mechanisms tend to add noise directly to the samples
X i to reduce the probability of identifying the training data [25].
Alternative means analyze the inﬂuence of regularization of neural network [48] or label smoothing [32].
Hereafter we design a mechanism preventing peaks of conﬁdence of the neural network, mitigating MIA related risks.
The strategy consists in reducing the vulnerability by adding controlled and voluntary noise to the one-hot encoded labels –counterparts of Y i – while training the deep model.
The core idea is the following: as the model’s generalization improves through the training phase, the model should less require simplistic labels.
Algorithmdescribes the defense mechanism.
Algorithm 2 Defense Mechanism
Require:
Training dataset D train =
{ ( X i , Y i ) } ni =1 , class of classiﬁer G , batch size 0 < m ≤ n , a non-decreasing non-negative function M valued in [0 , ( K − 1) /K ] .
Initialize (cid:98) g θ with random θ ∈ Θ Set L 1 =
L 2 = 0 .
Set (cid:101) Y i =
( 1 { j = Y i } ) Kj =1 .
Set 0 < ζ < ( K − 1) /K .
2: while θ not converged or L 1 ≤ L 2 do 3: Sample { ( X 1 , (cid:101) Y 1 ) , . . .
, ( X m
while 10:
The defense method described above do not change label from sample X i with some randomization to another label: it provides a mixture of labels in a similar way as label smoothing [71].
Remark 2 (Random noise injection)
The label noise may be applied on a random subset of a batch instead of the whole batch.
In this way, Appendix D.2 provides an alternative of Algorithm 2 with a randomized noise injection.
1.0 FPR 0.
T P R adversarial_distance sisyphos
D.2 Alternative Defense Mechanism
In Section D.1, we introduce the defense mechanism relying on noisy labels to build a classiﬁer (cid:98) g θ hopefully robust to MIA based on adversarial examples.
Algorithm 3 provides an alternative version of Algorithm 2 where the voluntary label noise is randomly injected in a subset of the batches of while training.
The experimental inﬂuence of the randomization of Algorithm 2 is depicted in Section D.3.
Algorithm 3 Randomized SISYPHOS
Require:
Training dataset D train =
{ ( X i , Y i ) } ni =1 , class of classiﬁer G , batch size 0 < m ≤ n , a non-decreasing non-negative function M valued in [0 , ( K − 1) /K ] , noise injection probability p ∈ ]0 , 1[ .
1: Initialize
Set L 1 =
L 2 = 0 .
Set (cid:101) Y i =
( 1 { j = Y i } ) Kj =1 .
Set 0 < ζ < ( K − 1) /K .
2: while θ not converged or L 1 ≤ L 2 do 3:
Sample from D train .
Update L 1 =
L 2 .
Update L 2 = 1 /m (cid:80) mi =1 (cid:96) ( (cid:101)
Y i , (cid:98) g
θ ( X i )) .
Update θ by descending L 2 .
7: for i in { 1 , . . .
, m } do 8:
With probability p update (cid:101) Y i , (cid:101) Y i = (cid:18)
[Equation]
9: end for 10:
Update
11: end while 12: Return (cid:98) g
D.3 Numerical Experiments regarding our Defense Mechanism
In this section, we present numerical experiments run to illustrate the defense mechanism depicted in
Section D and its alternative algorithm from Section D.2.
Dataset.
We work with MNIST dataset [23]: a database of handwritten digits with a training set of 60,000 examples and a test set of 10,000 examples.
The digits have been size-normalized and centered in a ﬁxed-size image.
The original black and white (bilevel) images from NIST were size normalized to ﬁt in a 20x20 pixel box while preserving their aspect ratio.
The resulting images contain gray levels as a result of the anti-aliasing technique used by the normalization algorithm.
The images were centered in a 28x28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28x28 ﬁeld.
Each training and test example is assigned to the corresponding handwritten digit between 0 and 9 .
Experimental framework.
The target model is a multi-layer perceptron and the defense mechanisms are Algorithm 2 and Algorithm 3.
For both defense mechanisms, the label noise is constant through training ( i.e. for any ζ, M ( ζ ) = ζ ) similarly to label smoothing.
We train a multi-layer perceptron with one layer containing 100 neurons.
[Figure]
Results.
Figure 11 provides the evolution of (cid:98) g θ ’s accuracy on the validation set when label noise is injected (with noise level ζ = 0 . 1 ).
One can observe similar accuracy between the two settings considered.
Figure 12 represents the loss values of (cid:98) g θ for the validation and train sets through training with noisy labels following Algorithm 2 or Algorithm 3.
[Figure]
Interestingly, one can observe in Figure 12 (Bottom Right) that when large noise ( ζ = 0 . 9 ) is added to the training labels, two antagonist forces seem to be present: ﬁrst, the loss minimization focuses on the learning task ( i.e. predicting the correct class for each sample with the best accuracy possible) and then starts overﬁtting to the voluntary injected noise in the remaining labels which thus decreases the maximum score granted to the predicted label.
In this way, such noise injection could be relevant to better assess early-stopping while training neural networks in addition to some improved robustness to MIAs.
[Figure]
Figure 13 describes the evolution of the excess of conﬁdence of (cid:98) g θ with varying noise levels for (Left)
Algorithm 2 and (Right) Algorithm 3.
We can see that as more noise is injected in the labels, the excess conﬁdence of (cid:98) g θ decreases, thus being more robust to MIAs based on adversarial examples.
E Broader MIA Experiments
The MIA framework in this paper targets to predict if some sample ( X i , Y i )
∈ D test belongs to from the training set D train , under the assumption that D test ∩ D train (cid:54) = ∅ .
In practice, access to the exact samples ( X i , Y i ) is unlikely.
In this section, we want to assess if classiﬁer’s excess of conﬁdence can still be noticed when one has solely access to ( (cid:101)
X i , Y i ) with (cid:101)
X i remaining close to X i .
The experiment detailed in this section suggests that the adversarial attack designed in this paper may be generalized to more complex and real world settings.
Dataset.
The Olivetti dataset contains a set of face images taken between April 1992 and Aprilat AT&T Laboratories Cambridge.
There are ten different images { X ji } 10 i =1 of each distinct subject j among the 40 subjects.
The label Y i associated to a sample X i consists in assessing if the person in
X i wears glasses ( i.e. Y i =
if and only if the input image
X i contains glasses).
For some subjects, the images were taken at different times, varying the lighting, facial expressions (open / closed eyes, smiling / not smiling) and facial details (glasses / no glasses).
All the images were taken against a dark homogeneous background with the subjects in an upright, frontal position (with tolerance for some side movement).
Figure 14 illustrates samples X j 1 , X j 2 (Left, Center) of subject j and X j (cid:48)(Right) of subject j (cid:48) .
Subject j wears no glasses in both presented images while subject j (cid:48) wears glasses in the presented image.
Experimental framework.
Let D train =
( X ji , Y i ) } j ≤ 39 i< 10 denote the training set used to build a classiﬁer (cid:98) g θ .
Let D test =
{ ( X j i , Y i ) } j ≤ 39 i =10 ∪ { ( X i , Y i ) } j =40 i ≤ 10 contain the remaining samples.
By construction, there is no sample X
i belonging to the intersection between D train and D test , although some subjects belong to both D train and D test .
D test contains 10 images of subject 40 unseen in the training set.
We compare the maximum scores of (cid:98) g θ for new samples of subjects seen in the training data and new samples of subject unseen in the training phase.
Results.
Boxplots in Figure 15 report the max score of (cid:98) g θ in two subsets of D test .
The mean of the scores is denoted in orange.
The median of the scores is denoted in dashed green.
Figureessentially suggests that exploiting the excess of conﬁdence of a classiﬁer to retrieve training data is possible even only with access to modiﬁed inputs of the training set.
The classiﬁer (cid:98) g θ shows larger conﬁdence on its prediction when subjects have already been encountered in the training phase.
Therefore, one may generalize MIA strategies with adversarial examples relying on functional in the output space of (cid:98) g θ to real world settings.
[Figure]