C LASSIFIER - TO -G ENERATOR A TTACK : E STIMATION OF T RAINING D ATA D ISTRIBUTION FROM C LASSIFIER
Anonymous authors
Paper under double-blind review
A BSTRACT
Suppose a deep classiﬁcation model is trained with samples that need to be kept private for privacy or conﬁdentiality reasons.
In this setting, can an adversary obtain the private samples if the classiﬁcation model is given to the adversary?
We call this reverse engineering against the classiﬁcation model the Classiﬁer-toGenerator (C2G) Attack .
This situation arises when the classiﬁcation model is embedded into mobile devices for ofﬂine prediction (e.g., object recognition for the automatic driving car and face recognition for mobile phone authentication).
For C2G attack, we introduce a novel GAN, PreImageGAN .
In PreImageGAN, the generator is designed to estimate the the sample distribution conditioned by the preimage of classiﬁcation model f , P ( X j f ( X ) =
y ) , where X is the random variable on the sample space and y is the probability vector representing the target label arbitrary speciﬁed by the adversary.
In experiments, we demonstrate PreImageGAN works successfully with hand-written character recognition and face recognition.
In character recognition, we show that, given a recognition model of hand-written digits, PreImageGAN allows the adversary to extract numeric images without knowing that the model is built for numeric images.
In face recognition, we show that, when an adversary obtains a face recognition model for a set of individuals, PreImageGAN allows the adversary to extract face images of speciﬁc individuals contained in the set, even when the adversary has no knowledge of the face of the individuals.
I NTRODUCTION
Recent rapid advances in deep learning technologies are expected to promote the application of deep learning to online services with recognition of complex objects.
Let us consider the face recognition task as an example.
The probabilistic classiﬁcation model f takes a face image x
and the model predicts the probability of which the given face image is associated with an individual t , f ( x ) ≃ Pr[ T = t
j X = x ] .
The following three scenarios pose situations that probabilistic classiﬁcation models need to revealed in public for online services in real applications:
Prediction with cloud environment: Suppose an enterprise provides an online prediction service with a cloud environment, in which the service takes input from a user and returns predictions to the user in an online manner.
The enterprise needs to deploy the model f into the cloud to achieve this.
Prediction with private information: Suppose an enterprise develops a prediction model f
(e.g., disease risk prediction) and a user wishes to have a prediction of the model with private input (e.g., personal genetic information).
The most straightforward way to preserve the user’s privacy entirely is to let the user download the entire model and perform prediction on the user side locally.
Ofﬂine prediction: Automatic driving cars or laptops with face authentication contain face/object recognition systems in the device.
Since these devices are for mobile use and need to work standalone, the full model f needs to be embedded in the device.
In such situations that classiﬁcation model f is revealed, we consider a reverse-engineering problem of models with deep architectures.
Let D tr and d X;T be a set of training samples and its underlying distribution, respectively.
Let f be a model trained with D tr .
In this situation, is it possible for an adversary to obtain the training samples D tr (or its underlying distribution d X;T ) if the classiﬁcation model is given to the adversary?.
If this is possible, this can cause serious problems, particularly when D tr or d X;T is private or conﬁdential information.
Privacy violation by releasing face authentication: Let us consider the face authentication task as an example again.
Suppose an adversary is given the classiﬁcation model f .
The adversary aims to estimate the data (face) distribution of a target individual t (cid:3) ; d X j T = t (cid:3) .
If this kind of reverseengineering works successfully, serious privacy violation arises because individual faces are private information.
Furthermore, once d X j T = t (cid:3) is revealed, the adversary can draw samples from d X j T = t (cid:3) , which would cause another privacy violation (say, the adversary can draw an arbitrary number of the target’s face images).
Conﬁdential information leakage by releasing object recognizer: Let us consider an object recognition system for automatic driving cars.
Suppose a model f takes as input images from car-mounted cameras and detect various objects such as trafﬁc signs or trafﬁc lights.
Given f , the reverse engineering reveals the sample distribution of the training samples, which might help adversaries having malicious intentions.
For example, generation of adversarial examples that make the recognition system confuse without being detected would be possible.
Also, this kind of attack allows exposure of hidden functionalities for privileged users or unexpected vulnerabilities of the system.
If this kind of attack is possible, it indicates that careful treatment is needed before releasing model f in public considering that publication of f might cause serious problems as listed above.
We name this type of reverse engineering classiﬁer-to-generator (C2G) attack .
In principle, estimation of labeled sample distributions from a classiﬁcation/recognition model of complex objects (e.g., face images) is a difﬁcult task because of the following two reasons.
First, estimation of generative models of complex objects is believed to be a challenging problem itself.
Second, model f often does not contain sufﬁcient information to estimate the generative model of samples.
In supervised classiﬁcation, the label space is always much more abstract than the sample space.
The classiﬁcation model thus makes use of only a limited amount of information in the sample space that is sufﬁcient to classify objects into the abstract label space.
In this sense, it is difﬁcult to estimate the sample distribution given only classiﬁcation model f .
To resolve the ﬁrst difﬁculty, we employ Generative Adversarial Networks (GANs).
GANs are a neural network architecture for generative models which has developed dramatically in the ﬁeld of deep learning.
Also, we exploit one remarkable property of GANs, the ability to interpolate latent variables of inputs.
With this interpolation, GANs can generate samples (say, images) that are not included in the training samples, but realistic samples 1 .
Even with this powerful generation ability of GANs, it is difﬁcult to resolve the second difﬁculty.
To overcome this for the C2G attack, we assume that the adversary can make use of unlabeled auxiliary samples D aux as background knowledge.
Suppose f be a face recognition model that recognizes
Alice and Bob, and the adversary tries to extract Alice’s face image from f .
It is natural to suppose that the adversary can use public face image samples that do not contain Alice’s and Bob’s face images as D aux .
PreImageGAN exploits unlabeled auxiliary samples to complement knowledge extracted from the model f .
1. O UR C ONTRIBUTION
The contribution of this study is summarized as follows.
(cid:15)
We formulate the Classiﬁer-to-Generator (C2G) Attack, which estimates the training sample distribution when a classiﬁcation model and auxiliary samples are given(Section 3) (cid:15)
We propose PreImageGAN as an algorithm for the C2G attack.
The proposed method estimates the sample generation model using the interpolation ability of GANs even when the auxiliary samples used by the adversary is not drawn from the same distribution as the training sample distribution (Section 4)
realizes the operation of latent vectors.
For example, by subtracting a latent vector of a man’s face from a face image of a man wearing glasses, and then adding a latent vector of a female’s face, then the GAN can generate the woman’s face image wearing glasses.
(cid:15)
We demonstrate the performance of C2G attack with PreImageGAN using EMNIST (alphanumeric image dataset) and FaceScrub (face image dataset).
Experimental results show that the adversary can estimate the sample distribution even when the adversary has no samples associated with the target label at all (Section 5)
G ENERATIVE A DVERSARIAL N ETWORKS
Generative Adversarial Networks (GANs) is a recently developed methodology for designing generative models proposed by Goodfellow et al.
Given a set of samples, GANs is an algorithm with deep architectures that estimates the sample-generating distribution.
One signiﬁcant property of GANs is that it is expected to be able to accurately estimate the sample distribution even when the sample space is in the high dimensional space, and the target distribution is highly complex, such as face images or natural images.
In this section, we introduce the basic concept of GANs and its variants.
The learning algorithm of GANs is formulated by minimax games consisting of two players, generator and discriminator (Goodfellow et al.
Generator G generates a fake sample G ( z ) using a random number z (cid:24)
d Z drawn from any distribution (say, uniform distribution).
Discriminator D is a supervised model and is trained so that it outputs 1 if the input is a real sample x (cid:24) d X drawn from the sample generating distribution d X ; it outputs 0 or (cid:0) 1 if the input is a fake sample G ( z ) .
The generator is trained so that the discriminator determines a fake sample as a real sample.
By training the generator under the setting above, we can expect that samples generated from G ( z ) for arbitrary z are indistinguishable from real samples x (cid:24) d X .
Letting Z be the random variable of d Z , G ( Z ) can be regarded as the distribution of samples generated by the generator.
Training of GANs is known to be reduced to optimization of G so that the distribution between G ( Z ) and the data generating distribution d X is minimized in a certain type of divergence (Goodfellow et al.
Training of GAN proposed by Goodfellow et al.
(2014) (VanillaGAN) is shown to be reduced to minimization e of Jensen Shannon (JS) divergence of G ( Z ) and d X .
Minimization of JS-divergence often suffers gradient explosion and mode collapse (Goodfellow et al.
(2014), Arjovsky & Bottou (2017)).
To overcome these problems, Wasserstein-GAN (WGAN), GAN that minimizes Wasserstein distance between G ( Z ) and d X , was proposed (Arjovsky et al.
(2017), Arjovsky & Bottou (2017)).
As a method to stabilize convergence behavior of WGAN, a method to add a regularization term called Gradient Penalty (GP) to the loss function of the discriminator was introduced (Gulrajani et al.
Given a set of labeled samples f ( x ; c ) ; (cid:1) (cid:1) (cid:1) g where c denotes the label, Auxiliary Classiﬁer GAN (ACGAN) was proposed as a GAN to estimate d X j C
=
c , sample distribution conditioned by label c (Odena et al.
Differently from VanillaGAN, the generator of ACGAN takes as input a random noise z and a label c .
Also, the discriminator of ACGAN is trained to predict a label of sample in addition to estimation of real or fake samples.
In the learning process of ACGAN, generator is trained so that discriminator predicts correctly the label of generated sample in addition.
The generator of ACGAN can generate samples with a label speciﬁed arbitrarily.
For example, when x corresponds to face images and c corresponds to age or gender, ACGAN can generate images with specifying the age or gender (Mirza & Osindero (2014), Gauthier (2014)).
In our proposed algorithm introduced in the latter sections, we employ WGAN and ACGAN as building blocks.
P ROBLEM F ORMULATION 3. P ROBABILISTIC D ISCRIMINATION M ODEL
We consider a supervised learning setting.
Let T be the label set, and X (cid:18)
R d be the sample domain where d denotes the sample dimension.
Let (cid:26) t be the distribution of samples in X with label t .
In face recognition, x 2 X and t 2 T correspond to a (face) image and an individual, respectively.
(cid:26) t thus denotes the distribution of face images of individual t .
We suppose the images contained in the training dataset are associated with a label subset T tr (cid:26) T .
Then, the training dataset is deﬁned as D tr = f ( x ; t ) j x 2 X ; t 2 T tr g .
We denote the random variables associated with ( x ; t ) by ( X tr ; T tr ) ． Then, the distribution of X tr is given by a mixture distribution
[Equation]
where ∑ t 2 T tr (cid:11) t = 1 ; (cid:11) t > 0 for all t 2 T tr .
In the face recognition task example again, a training sample consists of a pair of an individual t and his/her face image x , ( x ; t ) where x (cid:24) (cid:26) t .
Next, we deﬁne the probabilistic discrimination model we consider in our problem.
Let Y be a set of j
T tr j -dimension probability vector, ∆ j T tr j .
Given a training dataset D tr , a learning algorithm L gives a probabilistic discrimination model f : X !
Y as f = L ( D tr ) .
Here the t th element of the output ( f ( x )) t of f corresponds to the probability with which x has label t .
Letting T tr and X tr represents the random variable of T tr and d X tr , f is the approximation of Pr[ T tr j X tr ] .
3. C LASSIFIER - TO -G ENERATOR A TTACK
We deﬁne the Classiﬁer-to-Generator Attack (C2G Attack) in this section.
We consider two stakeholders, publisher and adversary in this attack.
The publisher holds training dataset D tr drawn from d X tr and a learning algorithm L .
She trains model
f = L ( D tr ) and publishes f to the adversary.
We suppose training dataset D tr and data generating distribution (cid:26) t for any t 2 T tr is private or conﬁdential information of the publisher and the publisher does not wish to leak them by publishing f .
Given f and T tr , the adversary aims to obtain (cid:26) t (cid:3) for any label t (cid:3) 2 T tr speciﬁed by the adversary.
We suppose the adversary can make use of an auxiliary dataset D aux drawn from underlying distribution d X aux as background knowledge.
D aux is a set of samples associated with labels in T aux (cid:26) T .
We remark that D aux is deﬁned as a set of samples associated with a speciﬁc set of labels, however, in our algorithm described in the following sections, we do not require that samples in D aux are labeled.
Then, the underlying distribution d X aux is deﬁned as follows:
[Equation]
where ∑ t 2 T aux (cid:11)
[Equation]
for all
[Equation]
The richness of the background knowledge can be determined by the relation between T tr and T aux .
When T tr = T aux , d X tr = d X aux holds.
That is, the adversary can make use of samples drawn from the distribution that is exactly same as that of the publisher.
In this sense, this setting is the most advantageous to the adversary.
If t (cid:3) = 2 T aux , the adversary cannot make use of samples with the target label t (cid:3) ; this setting is more advantageous to the publisher.
As the overlap between T tr and T aux increases, the situation becomes more advantageous to the adversary.
Discussions on the background knowledge of the adversary are given in 3.4 in detail.
The goal of the adversary is to learn the publisher’s private distribution (cid:26) t (cid:3) for any t (cid:3) 2 T tr speciﬁed by the adversary provided model f , target label t (cid:3) and auxiliary (unlabeled) samples D aux .
Let A be the adversary’s attack algorithm.
Then, the attack by the adversary can be formulated by
[Equation]
where the output of A is a distribution over X .
In the face recognition example of Alice and Bob again, when the target label of the adversary is t (cid:3) =
Alice, the objective of the adversary is to estimate the distribution of face images of Alice by A ( f; D aux ; t (cid:3) ) .
3. E VALUATION OF THE C2G A TTACK
The objective of the C2G attack to estimate (cid:26) t (cid:3) , the private data generating distribution of the publisher.
In principle, the measure of the success of the C2G attack is evaluated with the quasi-distance between the underlying distribution (cid:26) t (cid:3) and the estimated generative model A ( f; D aux ; t (cid:3) ) .
If the two distributions are close, we can conﬁrm that the adversary successfully estimates (cid:26) t (cid:3) .
However, (cid:26) t (cid:3) is unknown, and we cannot evaluate this quasi-distance directly.
Instead of evaluating the distance of the two distributions directly, we evaluate the attack algorithm empirically.
We ﬁrst prepare a classiﬁer f ′ that is trained with D tr using a learning algorithm different from f .
We then give samples drawn from A ( f; D aux ; t (cid:3) ) to f ′ and evaluate the probability of which the label of the given samples are predicted as t (cid:3) .
We expect that the classiﬁer f ′ would label samples drawn from A ( f; D aux ; t (cid:3) ) as t (cid:3) with high probability if A ( f; D aux ; t (cid:3) ) successfully estimates (cid:26) t (cid:3) .
Considering the possibility that A ( f; D aux ; t (cid:3) ) overﬁts to f , we employ another classiﬁer f ′ for this evaluation.
This evaluation criterion is the same as the inception accuracy introduced for
ACGAN by Odena et al. (2016).
In our setting, since our objective is to estimate the distribution concerning a speciﬁc label t (cid:3) , we employ the following inception accuracy:
[Equation]
We remark that the generated model with a high inception accuracy is not always a reasonable estimation of (cid:26) t (cid:3) .
Discrimination models with a deep architecture are often fooled with artifacts.
For example, Nguyen et al.
(2014) reported that images look like white noise for humans can be classiﬁed as a speciﬁc object with high probability.
For this reason, we cannot conclude that a model with a high inception accuracy always generates meaningful images.
To avoid this, the quality of generated images should be subjectively checked by humans.
The evaluation criterion of the C2G Attack we employed for this study is similar to those for GANs.
Since the objective of GANs and the C2G attack is to estimate unknown generative models, we cannot employ the pseudo distance between the underlying generating distribution and the estimated distribution.
The evaluation criterion of GANs is still an open problem, and subjective evaluation is needed for evaluation of GANs (Goodfellow (2017)).
In this study, we employ both the inception accuracy and subjective evaluation for performance evaluation of the C2G attack.
3. B ACKGROUND K NOWLEDGE OF A DVERSARY
The richness of the background knowledge of the adversary affects the performance of the C2G attack signiﬁcantly.
We consider the following three levels of the background knowledge.
In the following, let T aux be the set of labels of samples generated by the underlying distribution of the auxiliary data, d X aux .
Also, let T tr be the set of labels of samples generated by the underlying distribution of the training data.
(cid:15) Exact same: T tr = T aux
In this setting, we suppose T tr is exactly same as the T aux .
Since D aux follows d X tr , D aux contains samples with the target label.
That is, the adversary can obtain samples labeled with the target label.
The background knowledge of the adversary in this setting is the most powerful among the three settings.
(cid:15)
Partly same: t (cid:3) = 2 T aux ; T aux (cid:26) T tr
In this setting, T aux and T tr are overlapping.
However, T aux does not contain the target label.
That is, the adversary cannot obtain samples labeled with the target label.
In this sense, the background knowledge of the adversary in this setting is not as precise as that in the former setting.
(cid:15) Mutually exclusive: T aux \ T tr = ∅
In this setting, we suppose T aux and T tr are mutually exclusive, and the adversary cannot obtain samples labeled with the target label.
In this setting, the adversary cannot obtain any samples with labels used for training of model f .
In this sense, the background knowledge of the adversary in this setting is the poorest among the three settings.
P RE I MAGE GAN 4. O UTLINE OF P RE I MAGE GAN
In this section, we propose PreImageGAN that works as an algorithm that achieves C2G attack.
Given the model f ≃ d Y tr j X tr , the goal of PreImageGAN is to estimate ( d X tr j Y tr = y ( t (cid:3) ) )
= (cid:26) t (cid:3) for target label t (cid:3) speciﬁed by the adversary where y ( t (cid:3) ) is the one-hot vector in which the element corresponds to t (cid:3) is activated.
In the adversary can draw samples from d X tr j Y tr = y ( t (cid:3) ) , this generative model can be straightforwardly estimated with existing GANs.
However, in the setting of the C2G attack, what the adversary can utilize for the C2G attack is the probabilistic classiﬁcation model f that is expected to be similar to d Y tr j X tr and auxiliary samples that are drawn from auxiliary distribution
d X aux only.
So the target distribution cannot be estimated in a straightforward manner.
PreImageGAN ﬁrst gives labels to auxiliary samples with the given model as y = f ( x ) and estimates d X aux j Y aux using f ( x ; y )
j x 2 D aux g by optimizing the objective function deﬁned below.
If the sample distribution of the auxiliary samples is close to that of the true underlying distribution, we can expect that estimation of d X aux j Y aux can be used as an approximation of d X tr j Y tr .
More speciﬁcally, we can obtain the generative model of the target label d X aux j Y aux
=
y ( t (cid:3) ) by specifying the one-hot vector of the target label as the condition.
As we mentioned in Section 3.4, the sample generating distribution of the auxiliary samples is not necessarily equal or close to the true sample generating distribution.
In the ”partly same” setting or ”mutually exclusive” setting, D aux does not contain samples labeled with t (cid:3) at all.
It is well known that GANs can generate samples with interpolating latent variables (Berthelot et al.
(2017), Radford et al.
We expect that PreImageGAN generates samples with the target label by interpolation of latent variables of given samples without having samples of the target label.
More speciﬁcally, if latent variables of given auxiliary samples are diverse enough and d X aux j Y
aux
well approximates the true sample generating distribution, we expect that GAN can generate samples with the target label by interpolating obtained latent variables of auxiliary samples without having samples with the target label.
Figure 1 describes a conceptual illustration of generation of image ”9” when PreImageGAN takes images of alphabets (A-Z,a-z) as auxiliary samples and f ( x ) =
Pr [ number j x ] as a probabilistic classiﬁcation model.
The auxiliary samples do not contain images of numbers whereas PreImageGAN generates images close to ”9” by interpolating latent variables of alphabets close to ”9”, such as ”g” or ”Q”.
4. O BJECTIVE F UNCTION
In PreImageGAN, a conditional attribute y (cid:24) d Y aux in addition to noise z (cid:24) d Z is used to generate samples.
Similarly to other GANs, d Z can be arbitrarily determined by the learner (adversary).
Given f and D aux , d Y aux can be empirically estimated from the set of predictions f f ( x ) j x 2 D aux g .
Generator G : ( Z ; Y ) !
X of PreImageGAN generates fake samples x fake =
G ( z ; y ) using random draws of y and z .
After the learning process is completed, we expect generated fake samples x fake satisfy f ( x fake ) =
y .
On the other hand, discriminator D : X !
R takes as input a sample x and discriminates whether it is a generated fake sample x fake or a real sample x real 2 D aux .
With these requirements, the objective function of G and D is formulated as follows
[Equation]
where ∥ (cid:1) ∥
L (cid:20) 1 denotes (cid:11) -Lipschitz functions with (cid:11) (cid:20) 1 .
By maximizing the ﬁrst and second term concerning D , Wasserstein distance between the marginal of the generator ∫ G ( Z; Y aux ) dY aux and the generative distribution of auxiliary samples d X aux is minimized.
By maximizing the similarity between y and f ( G ( z ; y )) , G is trained so that samples generated from G ( z ; y ) satisfy f ( G ( z ; y ))
=
y .
(cid:13)
(cid:21) 0 works as a parameter adjusts the effect of this term.
For sample generation with PreImageGAN, G ( Z; y ( t (cid:3) ) ) is utilized as the estimation of (cid:26) t (cid:3) .
We here remark that model f is regarded as a constant in the learning process of GAN and used as it is.
E XPERIMENTS
In this section, we show that the proposed method enables to perform C2G attack with experiments.
We experimentally demonstrate that the adversary can successfully estimate (cid:26) t (cid:3) given classiﬁer f and the set of unlabeled auxiliary samples D aux =
f x j x 2 X g even in the partly same setting and the mutually exclusive setting. .
5. E XPERIMENTAL S ETUP
For demonstration, we consider a hand-written character classiﬁcation problem (MNIST/EMNIST) and a face recognition problem (FaceScrub).
We used the Adam optimizer ( (cid:11) =
2 e + 4 , (cid:12) 1 = 0 : 5 , (cid:12) 2 = 0 : 999 ) for the training of the generator and discriminator.
The batch size was set as 64.
We set the number of discriminator (critic) iterations per each generator iteration n cric = 5 .
To enforce the 1-Lipschitz continuity of the discriminator, we add a gradient penalty (GP) term to the loss function of the discriminator (Gulrajani et al.
and set the strength parameter of GP as (cid:21) = 10 .
We used 128-dim uniform random distribution [ (cid:0) 1 ; 1] 128 as d Z .
We estimated d Y aux empirically from f f ( x ) j x 2 D aux g using kernel density estimation where the bandwidth is 0 : 01 , and the Gaussian kernel was employed.
5. EMNIST: HAND - WRITTEN CHARACTERS CLASSIFIER
EMNIST consists of grayscale 28x28 pixel images from 62 alphanumeric characters (0-9A-Za-z).
We evaluated C2G attack with changing the richness of the adversary’s background knowledge as discussed in Section 3.4.
Table 1 represents the settings used for the C2G attack to alphanumeric classiﬁcation.
In all cases, the target label t (cid:3) was set as numeric characters ( t (cid:3) 2 f 0 ; 1 ; : : : ; 9 g ).
The alphanumeric classiﬁer given to the adversary in the exact same setting is trained on 10k samples for ten epochs and achieved test accuracy 0.8443.
The numeric classiﬁer given to the adversary in the partly same/mutually exclusive setting is trained on 116k samples for ten epochs and achieved test accuracy 0.9911.
In the training process of PreImageGAN, we trained the generator for 20k iterations.
We set the initial value of (cid:13) to 0, incremented gamma by 0.001 per generator iteration while (cid:13) is less than 10.
[Table]
Fig.
2 represents the results of the C2G attack.
In the exact same setting, we can conﬁrm that
PreImageGAN generated images that look like numeric images corresponding to each target label quite well.
In the partially same setting, some generated images are disﬁgured compared to images generated in the exact same setting (especially when t (cid:3) = 3 ; 7 ) while we can recognize most of the generated images as numeric images corresponding to each target label as well.
From these results, we can conclude that the C2G attack against character recognition works successfully in the exact same and partially same setting.
We note that the adversary could generate numeric images even when the adversary has no numeric images from the result in the partially same setting.
In the mutually exclusive setting, some digitlike images are generated while some samples still look like alphabets.
When the classiﬁer cannot distinguish alphabet images and numeric images well, PreImageGAN tends to generate disﬁgure images.
This observation indicates that when classiﬁcation models can classify samples without exploiting the feature of entire images, the attack does not necessarily work successfully.
We evaluate inception accuracy using f ′ where we employed ResNet-based network architecture (Table A).
As a baseline, we trained ACGAN with the same training samples D tr and evaluated the inception accuracy.
In the exact same setting, the inception accuracy is almost equal to ACGAN.
From the results, we can see that the inception scores drop as the background knowledge of the adversary becomes poorer.
This result indicates that the background knowledge of the adversary affects the result of the C2G attack signiﬁcantly.
5. F ACE S CRUB : F ACE R ECOGNITION
FaceScrub dataset consists of color face images (530 persons).
We resized images to 64x64 pixel images for experiments and evaluated the C2G attack in the mutually exclusive setting.
In detail, we picked up 100 persons as T tr and used remaining 430 persons as T aux .
If the adversary can generate faces of 100 persons in T tr by utilizing model f recognizing T tr and face images with labels in T aux , we can conﬁrm that the C2G attack works successfully.
D tr consisted of 13,650 images and D aux consists of 59,764 images.
f is trained on D tr for ten epochs and achieved test accuracy 0.8395.
In training PreImageGAN, we train generator for 130k iterations.
We set the initial value of (cid:13) to 0, incremented gamma by 0.0001 per generator iteration while (cid:13) is less than 10.
Fig.
3 represents the results of the C2G attack against face recognition.
Those samples are randomly generated without human selection.
The generated face images well capture the features of the face images in the training samples, and we can conclude that the C2G attack successfully extracts training samples from face recognition model without having training samples in the mutually exclusive setting.
As byproducts, we can learn what kind of features the model used for face recognition with the results of the C2G attack.
For example, all generated face images of Keanu Reeves wear the mustache.
This result implies that f exploits his mustache to recognize Keanu Reeves.
R ELATED W ORK
Fredrikson et al.
(2014) proposed the model inversion attack against machine learning algorithms that extract private input attributes from published predicted values.
Through a case study of personalized adjustment of Warfaline dosage, Fredrikson et al.
(2014) showed that publishing predicted dosage amount can cause leakage of private input attributes (e.g., personal genetic information) in generalized linear regression.
Fredrikson et al.
(2015) presented a model inversion attack that re-
[Table]
constructs face images from a face recognition model.
The signiﬁcant difference between the C2G attack and the model inversion attack is the goal of the adversary.
In the model inversion attack, the adversary tries to estimate a private input (or input attributes predicted values y using the predictor f .
Thus, the adversary’s goal is the private input x itself in the model inversion attack.
By contrast, in the C2G attack, the adversary’s goal is to obtain the training sample distribution.
Another difference is that the target network model.
The target model of Fredrikson et al.
(2015) was a shallow neural network model while ours is deep neural networks.
As the network ar- chitecture becomes deeper, it becomes more difﬁcult to extract information about the input because the output of the model tends to be more abstract (Hitaj et al.
Hitaj et al.
(2017) discussed leakage of training samples in collaborative learning based on the model inversion attack using the IcGAN (Perarnau et al.
In their setting, the adversary’s goal is not to estimate training sample distribution but to extract training samples.
Also, their demonstration is limited to small-scale datasets, such as MNIST dataset (hand-written digit grayscale images, 10 labels) and AT&T dataset (400 face grayscale images with 40 labels).
By contrast, our experiments are demonstrated with larger datasets, such as EMNIST dataset (62 labels) and FaceScrub dataset (530 labels, 100,000+ color images).
Hayes et al.
(2017) discussed the membership inference attack against a generative model trained by
BEGAN (Berthelot et al.
or
DCGAN (Radford et al.
In the membership inference attack, the adversary’s goal is to determine whether the sample is contained in the private training dataset; the problem and the goal are apparently different from ours.
Song et al.
(2017) discussed malicious regularizer to memorize private training dataset when the adversary can specify the learning algorithm and obtain the classiﬁer trained on the private training data.
Their experiments showed that the adversary can estimate training data samples from the classiﬁer when the classiﬁer is trained with malicious regularizer.
Since our setting does not assume that the adversary can specify the learning algorithm, the problem setting is apparently different from ours.
Mahendran & Vedaldi (2015) and Mahendran & Vedaldi (2016) consider the understanding representation of deep neural networks through reconstruction of input images from intermediate features.
Their studies are related to ours in the sense that the algorithm exploits intermediate features to attain the goal.
To the best of our knowledge, no attack algorithm has been presented to estimate private training sample distribution as the C2G attack achieves.
C ONCLUSION
As described in this paper, we formulated the Classiﬁer-to-Generator (C2G) Attack, which estimates the training sample distribution (cid:26) t (cid:3) from given classiﬁcation model f and auxiliary dataset D tr .
As an algorithm for C2G attack, we proposed PreImageGAN which is based on ACGAN and
WGAN.
The proposed method can estimate the sample generation model using the interpolation ability of GANs even if the adversary does not have samples with training data labels.
In experiments, we demonstrated the performance of C2G attack against handwritten character classiﬁer and face recognition model.
Experimental results show that the adversary can estimate the training sample distribution (cid:26) t (cid:3) even when the adversary does not have samples with training data labels.
A CKNOWLEDGMENTS
To be written.
A M ODEL A RCHITECTURE A. EMNIST
EMNIST:
Classiﬁer f
Input Domain : X = [0 ; 1] 28 (cid:2) 28 (cid:2)
Output Domain : Y =
∆
d
Kernel size
Stride
Output channels
Convolution 5x 2x
BatchNormalizationReluConvolution 5x 2x
BatchNormalizationRelu
Flatten
Dropout 0.
Linear
Dropout 0.
Linear d
Softmax
EMNIST:
Generator G
Input Domain : Z = [ (cid:0) 1 ; 1] 128 ; Y =
∆ d
Output Domain : X = [0 ; 1] 28 (cid:2) 28 (cid:2)
Kernel size
Stride
Output channels
Concat ( z and y )
Linear 7x7x
Reshape
Deconvolution 4x 2x
LeakyReLUDeconvolution 4x 2x
LeakyReLUDeconvolution 3x 1x
Sigmoid
EMNIST:
Discriminator D
Input Domain : X = [0 ; 1] 28 (cid:2) 28 (cid:2)
Output Domain : R
Kernel size
Stride
Output channels
Convolution 3x 1x
LeakyReLUConvolution 4x 2x
LeakyReLUConvolution 3x 1x
LeakyReLUConvolution 4x 2x
LeakyReLUConvolution 3x 1x
LeakyReLUFlattenLinear
EMNIST:
Alternative Classiﬁer f ′
Input Domain : X = [0 ; 1] 28 (cid:2)
Output Domain : Y =
∆
d
Kernel size
Resample
Output channels
Convolution 3x 1x
Bottleneck Residual Block 3x Bottleneck Residual Block 3x
Down 2x
Bottleneck Residual Block 3x Bottleneck Residual Block 3x
Down 2x
BatchNormalizationReluFlattenDropout .
Linear
Dropout 0.
Linear d
Softmax
A.
F ACE S CRUB
FaceScrub: Classiﬁer f
Input Domain : X
= [ (cid:0) 1 ; 1] 64 (cid:2) 64 (cid:2)
Output Domain : Y =
∆
d
Kernel size
Stride
Output channels
Convolution 5x 2x
BatchNormalizationReluConvolution 5x 2x BatchNormalizationReluConvolution 5x 2x
BatchNormalizationReluFlattenDropout .
Linear
Dropout 0.
Linear d
Softmax
EMNIST:
Generator G
Input Domain : Z = [ (cid:0) 1 ; 1] 128 ; Y =
∆ d
Output Domain : X
= [ (cid:0) 1 ; 1] 64 (cid:2) 64 (cid:2)
Kernel size
Stride
Output channels
Concat ( z and y )
Linear 4x4x
Reshape
Deconvolution 4x 2x
LeakyReLUDeconvolution 4x 2x LeakyReLUDeconvolution 4x 2x
LeakyReLUDeconvolution 4x 2x
LeakyReLUDeconvolution 3x 1x  tanh
[Table] [Table]
D
[Table]
Convolution 3x 1x
LeakyReLUConvolution 4x 2x
LeakyReLUConvolution 3x 1x
LeakyReLUConvolution 4x 2x
LeakyReLUConvolution 3x 1x
LeakyReLUConvolution 4x 2x
LeakyReLUConvolution 3x 1x
LeakyReLU
Convolution 4x 2x
LeakyReLUConvolution 3x 1x
LeakyReLUFlattenLinear