HODA: P ROTECTING DNN S A GAINST M ODEL E XTRACTION A TTACKS VIA H ARDNESS OF S AMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Model Extraction attacks exploit the target model’s prediction API to create a surrogate model in order to steal or reconnoiter the functionality of the target model in the black-box setting.
Several recent studies have shown that a data-limited adversary who has no or limited access to the samples from the target model’s training data distribution can use synthesis or semantically similar samples to conduct model extraction attacks.
As the training process of DNN-based classiﬁers is done in several epochs, we can consider this process as a sequence of subclassiﬁers so that each subclassiﬁer is created at the end of an epoch.
We use the sequence of subclassiﬁers to calculate the hardness degree of samples.
In this paper, we investigate the hardness degree of samples and demonstrate that the hardness degree histogram of a data-limited adversary’s sample sequences is distinguishable from the hardness degree histogram of benign users’ samples sequences, consisting of normal samples.
Normal samples come from the target classiﬁer’s training data distribution.
We propose Hardness-Oriented Detection Approach (HODA) to detect the sample sequences of model extraction attacks.
The results demonstrate that HODA can detect the sample sequences of model extraction attacks with a high success rate by only watching 100 samples of them.
I NTRODUCTION
Deep Neural Networks (DNNs) have shown impressive performance in various tasks in recent years that have encouraged the industry to deploy DNN-based models in a variety of real-world applications.
Since the training process of DNNs and collecting training data is an expensive and tedious process, models are considered the intellectual property of organizations, and they must be kept secure.
Therefore, models are often securely deployed on cloud servers, and only the creators can access the model parameters.
Users are only allowed to query the model via a prediction API and receive predictions.
Recent studies Tram`er et
al.
Papernot et al.
; Juuti et al.
Orekondy et al.
Jagielski et al.
(2020) demonstrate that an adversary can exploit the prediction API of a target model to create a surrogate model in order to steal or reconnoiter the functionality of the target model.
Such attacks are called model extraction attacks , and they violate the intellectual property of model owners.
Furthermore, the surrogate model can be leveraged to conduct other attacks on the target model in black-box setting, such as adversarial example attacks Papernot et al.
(2017); Juuti et al.
and membership inference attacks Shokri et al.
Most of the model extraction attacks use the target model’s prediction API to label an unlabeled dataset to create the surrogate model’s training set.
In most real-world settings, the adversary has no or limited access to samples from the target model’s training data distribution, which is called normal or in-distribution samples.
Hence, most proposed attacks in the previous studies use some form of out-of-distribution samples, such as synthesis Papernot et al.
; Juuti et al.
(2019) or semantically similar samples to the target model’s training set Orekondy et al.
; Pal et al.
(2020) to conduct model extraction attacks.
We focus on such attacks in this paper.
There are two main approaches to defend against model extraction attacks, manipulating the target model outputs to prevent adversary from producing high-quality surrogate model Lee et al.
; Orekondy et al.
(2020); Kariyappa & Qureshi (2020); Kariyappa et al.
(2021b)
and detecting the sample sequences of model extraction attacks Kesarwani et al.
Juuti et al.
To the best of our knowledge,
PRADA Juuti et al.
(2019) is the only approach attempting to detect the sample sequences of model extraction attacks on DNN-based classiﬁers.
We propose Hardness-Oriented Detection Approach (HODA), a new approach to detect sample sequences of model extraction attacks, which outperforms
PRADA by a large margin and has signiﬁcantly less computational overhead.
Generally, the training process of DNN-based classiﬁers is done in several epochs, and the resulted classiﬁer at the end of the last epoch is considered the ﬁnal classiﬁer.
We can consider the training process of DNN-based classiﬁers as a sequence of subclassiﬁers in which the i th subclassiﬁer is created at the end of the i th epoch.
We say a sample is learned in the i th epoch when the i th subclassiﬁer is the ﬁrst subclassiﬁer that agrees with all subsequent subclassiﬁers predicted labels.
We consider the index of the epoch in which a new sample is learned as the hardness degree of that sample.
Consequently, easy samples being learned in the early epochs have low hardness degree, and harder ones being learned in the last epochs have high hardness degree.
It is important to note that we must save subclassiﬁers in the training phase of a target classiﬁer in order to use their predictions to calculate the hardness degree of new samples.
It is indicated the accuracy of classiﬁers is reduced by increasing the hardness degree of samples.
We demonstrate that the hardness degree histogram of normal sample sequences is distinguishable from the hardness degree histogram of model extraction attack sample sequences, and HODA uses this property to detect sample sequences of model extraction attacks.
For each user, HODA calculates the distance between the hardness degree histograms of the user’s samples and normal samples, and if the distance is greater than a threshold, the user is detected as an adversary.
HODA can detect JBDA Papernot et al.
(2017), JBRAND Juuti et al.
(2019), and Knockoff Net Orekondy et al.
(2019) attacks with a high success rate by only watching 100 samples of attack.
We demonstrate that HODA is also highly effective when the target classiﬁer is trained using transfer learning.
Contributions .
(i)
We demonstrate that the hardness degree of a sample for a classiﬁer pertains to the training data distribution of that classiﬁer. (ii) We indicate that the hardness degree histogram of normal samples is distinct from the hardness degree histograms of model extraction attack samples.
(iii)
We propose HODA to detect the sample sequences of model extraction attacks.
R ELATED W ORK
Model Extraction Attacks : Primary model extraction attacks try to extract the exact value of parameters Lowd & Meek (2005); Tram`er et al.
and hyperparameters Wang & Gong (2018) of shallow models.
In recent years, the proposed attacks mainly aimed to steal or reconnoiter the functionality of deep neural networks by querying them in the black-box setting.
It is often sensibly assumed in the literature that the adversary has no or limited access to samples from the training set distribution of target classiﬁer.
In order to overcome this issue, attacks generally use some form of out-of-distribution samples, such as synthesis or semantically similar samples, to create the surrogate classiﬁer’s training set.
Knockoff Net Orekondy et al.
ActiveThief Pal et al.
(2020), and Copycat CNN da Silva et al.
(2018) use semantically similar datasets to the target model’s training set to train a surrogate classiﬁer.
In another line of studies, Papernot et al.
(2017), Juuti et al.
(2019), Yu et al.
(2020), Truong et al.
, Kariyappa et al.
(2021a), and Barbalau et al.
(2020) use synthetic data to create the surrogate classiﬁer’s training set.
Model Extraction Defenses : Existing defense methods against model extraction attacks generally distribute into two branches: perturbation-based and detection-based.
Perturbation-based defenses
Lee et al.
; Orekondy et al.
(2020); Kariyappa & Qureshi (2020) attempt to prevent adversaries from producing high-quality surrogate classiﬁers by adding perturbation to the target classiﬁer outputs. Recently, Kariyappa et al.
(2021b) proposed a new defense with the same goal as perturbation-based defenses, which does not perturb the target classiﬁer outputs.
Their approach employs an ensemble of diverse models to produce discontinuous predictions for out-of-distribution samples.
Detection-based defenses attempt to detect the occurrence of model extraction attacks by observing successive input queries to the target classiﬁer.
Kesarwani et al.
(2018) present a method to detect extraction attacks against Decision Tree models.
PRADA Juuti et al.
(2019) is the ﬁrst proposed detection-based defense for DNN models.
We propose a new defense detecting the sample sequences of model extraction attacks via hardness of samples.
M ODEL E XTRACTION A TTACKS
The model extraction attack is one of the most serious threats against machine learning-based classiﬁers on remote servers, such as Machine Learning as a Service (MLaaS).
The adversary’s goal is to create a surrogate classiﬁer f s that imitates a target classiﬁer f t on task T .
Most model extraction attacks exploit target model f t to label unlabeled samples to create the surrogate model’s training set.
The adversary sends sample x i to the target model and receives its output f t ( x i ) , and then she uses pair ( x i , f t ( x i )) to train surrogate classiﬁer f s .
The output type of target model can be label, label conﬁdence, top-k values in probability vector, or the entire probability vector.
We only consider label ¯ f t ( x i ) and the entire probability vector f t ( x i ) as the output type of target classiﬁers in our experiments.
There are two primary intents for adversaries to conduct model extraction attacks, stealing and reconnaissance .
Stealing : Producing a high performance classiﬁer is an expensive and time-consuming process and requires computational resources and experts.
Besides, given that DNNs need a large number of training samples, collecting data and labeling them is a complex and costly procedure for most realworld applications.
Therefore, adversaries are motivated to take advantage of a target classiﬁer to reduce the cost of creating a new classiﬁer.
The adversary’s goal in stealing is to maximize the accuracy of surrogate model on data distribution D T .
Hence, the adversary’s goal is:
[Equation]
Reconnaissance : The model extraction attacks can be used to conduct other attacks in the blackbox setting, such as adversarial example attacks Papernot et al.
; Goodfellow et al.
and membership inference attacks Shokri et al.
The adversary’s goal in reconnaissance is to maximize the ﬁdelity among surrogate and target classiﬁers in order to increase the success rate of black-box attacks.
Similar to Jagielski et al.
, we consider label agreement among surrogate and target classiﬁers as the ﬁdelity metric on data distribution D T .
Hence, the adversary’s goal is:
[Equation]
Proposed model Extraction attacks create the surrogate classiﬁer training set X s = { ( x i , f t ( x i )) }
Bi
= by various methods, where B is the attack budget.
The attack budget determines the number of samples that an adversary is allowed to send to the target classiﬁer and receive their associated predictions.
After creating X s , the adversary trains surrogate classiﬁer f s to minimize empirical loss on X s .
We suppose that the adversary knows the architecture and hyperparameters of the target classiﬁer and uses them to train the surrogate classiﬁer.
It is noteworthy that our proposed defense is independent of surrogate classiﬁers’ training process.
O UR P ROPOSAL : H ARDNESS -O RIENTED DETECTION A PPROACH
We ﬁrst introduce the hardness degree of samples and then show the hardness degree histogram of model extraction attack samples is distinguishable from the hardness degree histogram of normal samples.
Using this observation, we propose Hardness-Oriented Detection Approach (HODA) to detect the sample sequences of model extraction attacks.
4. H ARDNESS D EGREE OF S AMPLES
The training process of a DNN-based classiﬁer can be considered a sequence of subclassiﬁers so that each subclassiﬁer is created at the end of an epoch.
Suppose that classiﬁer f t is trained for m epochs.
The training process of classiﬁer f t can be represented as the following sequence of subclassiﬁers:
[Equation]
where subclassiﬁer f it is created at the end of the i th epoch.
We say sample x i is learned in epoch e when f et is the ﬁrst subclassiﬁer that its assigned label is equal to all subsequent subclassiﬁers’ predicted labels.
Generally, as the number of epochs is increased, the performance of classiﬁer f t is improved so that easier samples are learned in the early epochs, and harder ones are learned in the last epochs.
Therefore, the hardness degree of sample x i for classiﬁer f t , which is displayed by φ f t ( x i ) , directly relates to the epoch number that x i is learned by f t .
Hardness degree of sample x i for classiﬁer f t is deﬁned as follows:
[Equation]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
The hardness degree domain is dependent on the number of subclassiﬁers, and since we have m subclassiﬁers, the hardness degree of a sample is in the range [0 , m − 1] .
To determine the hardness degree of unseen samples, we need to save subclassiﬁers at the end of each or several epochs in the training phase of target classiﬁers. When a new sample arrives, it is fed to all subclassiﬁers, and using their predictions, the hardness degree of that sample is calculated.
We train three various types of classiﬁers, including
DenseNet
Huang et al.
(2017), ResNet18 He et al.
(2016), and MobileNet Sandler et al.
(2018), on CIFAR10 and CIFAR100 training sets for 100 epochs (details of datasets in Appendix A).
All classiﬁers are trained using stochastic gradient descent with momentum 0.9 and batch size 128.
The learning rate is 0 .
and it is scheduled to be decreased in each epoch by a constant factor 0 .
The accuracy of classiﬁers is presented in Table 1.
We save all 100 subclassiﬁers in the training phase of each classiﬁer and use them to calculate the hardness degree of samples.
Figure 1 shows the hardness degree histogram of CIFAR10 and CIFAR100 test samples for various classiﬁers.
The ﬁgure demonstrates that a large fraction of Cifar10 test samples are easy, and many samples are learned in the ﬁrst few epochs.
However, the learning of Cifar100 test samples is distributed over various epochs, and the number of hard samples is more than Cifar10.
[Figure]
We divide the hardness degree domain into ten hardness degree ranges and partition the CIFAR10 and CIFAR100 test samples based on their hardness degrees into ten groups so that each group consists of samples whose hardness degrees belong to the associated hardness degree range.
Figure 2 shows the accuracy of classiﬁers on samples in each range of hardness degrees.
More than 99% and 95% of samples being learned in the ﬁrstepochs (hardness degree < 30) are correctly classiﬁed in CIFAR10 and CIFAR100 test sets, respectively.
On the other side, less than 55% and 36% of samples being learned in the last 10 epochs (hardness degree ≥ 90) are correctly classiﬁed in Cifar10 and Cifar100 test sets, respectively.
The results demonstrate that as the hardness degree of samples is increased, the accuracy of classiﬁers is reduced.
As ResNet18 architecture achieves strong performance on both datasets at a reasonable computational cost, we use this architecture for target classiﬁers in the rest of the paper.
We conduct various model extraction attacks on two CIFAR10 and CIFAR target classiﬁers in the next subsection to depict the hardness degree histogram of their samples.
4. M ODEL E XTRACTION A TTACKS S ETUP
In line with prior works (Orekondy et al.
; Kariyappa & Qureshi (2020); Kariyappa et al.
(2021b)), we select JBDA Papernot et al.
(2017), JBRAND Juuti et al.
(2019), and Knockoff Net
Orekondy et al.
(2019) model extraction attacks to evaluate our defense method.
These attacks broadly represent two main strategies (synthesis or semantically similar samples) to conduct model extraction attacks.
Jacobian-Based Dataset Augmentation (JBDA) Papernot et al.
(2017) and its improvement (JBRAND)
Juuti et al.
(2019) assume that the adversary has access to a limited number of samples from the target classiﬁer’s training data distribution called seed samples, and they aim to augment seed samples using adversarial examples to increase the ﬁdelity of the surrogate classiﬁer to the target classiﬁer.
Orekondy et al.
(2019) propose Knockoff Net (K.Net) attack that uses large public datasets that is semantically similar to the target classiﬁer dataset to increase the accuracy of the surrogate classiﬁer.
We consider two versions of K.Net attack, K.Net CIFARX, and K.Net
TIN.
K.Net CIFARX attack uses CIFAR100 training set to extract CIFAR10 target classiﬁer and vice versa.
K.Net TIN employs TinyImageNet training set to extract target classiﬁers.
More details about attacks and their implementations are presented in Appendix C.
[Table]
To evaluate the performance of model extraction attacks, we use two
ResNet18 classiﬁers being trained on CIFAR10 and CIFAR100 training sets as the target classiﬁers and conduct all four attacks on them.
The default value of the attack budget in our experiments is 50000 (B=50K).
Table 2 shows the accuracy and the ﬁdelity of surrogate classiﬁers created by various model extraction attacks on CIFAR10 and CIFAR100 test samples.
The results demonstrate that K.Net attacks have signiﬁcantly better performance than jacobian-based attacks (JBDA and JBRAND), and when the output of target classiﬁers is probability vector, the performance of attacks is considerably increased.
4. H ARDNESS D EGREE OF M ODEL E XTRACTION A TTACK S AMPLES
Figure 3 depicts the hardness degree histogram of 50000 samples generated by various attacks for CIFAR10 and CIFAR100 target classiﬁers.
In this experiment, the architecture of target classiﬁers is ResNet18.
We also present the hardness degree histogram of attack samples when the architecture of target classiﬁers is Densenet121 in Appendix D. Figure 3 demonstrates that the samples generated by various attacks have a very small number of easy samples, and most samples have medium and high hardness degrees.
However, Figure 1 indicates that a high number of normal samples that are from the same distribution as the target classiﬁer’s training set are easy.
To investigate more on the hardness degree of attack and normal samples, Figure 4 displays twodimensional visualization of CIFAR10 test samples using t-SNE.
Figure 4a uses the logits of the
CIFAR10 classiﬁer to visualize CIFAR10 test samples, and the color of each sample is determined by its label.
This ﬁgure has ten sample clusters where most samples of each cluster are from one class.
Figure 4b illustrates the hardness degree of CIFAR10 test samples for CIFAR10 target classiﬁer and demonstrates that most of the easy samples are in the high-density regions inside clusters, and most of the hard samples are in the low-density regions at the borders of clusters.
Figure 4c is similar to Figure 4b, but the hardness degree of each sample is calculated via CIFAR100 target classiﬁer.
This ﬁgure demonstrates when the training data distribution of the classiﬁer being used to calculate the hardness degree of samples becomes different from the distribution of CIFAR10 test samples, [Figure] [Figure] the hardness degree of a high number of samples is changed.
Figure 4c shows hard and medium samples are distributed among clusters
[Figure]
and the number of easy samples is very small. Similar to Figure 4, we visualize CIFAR100 test samples and their hardness for CIFAR10 and CIFAR target classiﬁers in Appendix E. Figures 3 and 4 demonstrate that the hardness degree of a sample for a classiﬁer pertains to the training data distribution of that classiﬁer, and when an adversary uses samples from a different distribution than the target model’s training set distribution, the hardness degree histogram of the adversary’s samples is distinct from the hardness degree histogram of normal samples being used by benign users.
We use histogram rather than hardness degree histogram in the rest of the paper for simplicity.
4. H ARDNESS -O RIENTED D ETECTION A PPROACH
We propose Hardness-Oriented Detection Approach (HODA) to detect sample sequences of model extraction attacks.
HODA requires normal histogram H n representing the histogram of normal samples.
When a new sample x i from user u arrives, HODA calculates its hardness degree φ f t ( x i ) , and the histogram belongs to that user H u is updated.
After the number of samples sent by user u reaches a speciﬁc number num s , HODA calculates Pearson distance between the histograms of normal samples H n and user samples H u , and if the distance is greater than a threshold δ , the user u is detected as an adversary.
Pearson Distance (PD) between two random variable X and Y is deﬁned as follows:
[Equation]
where Cov ( X, Y ) is the covariance between random variables X and Y , and ρ X is the standard deviation of random variable X .
The output of Pearson distance is in the range [0,2].
To calculate the Pearson distance between two histograms, HODA ﬁrst transforms histograms into probability vectors by dividing the value of histogram bins by the total number of samples in the histogram ( H n /sum
( H n ) and H u /sum ( H u ) )
and then calculates the Pearson distance between them.
HODA uses normal sample set S HODA to create H n and calculate δ .
It randomly selects num seq sample sequences with size num s from the sample set S HODA and for each sample sequence, produces a histogram and adds it to the histogram set HistSet .
The normal histogram H n is the average of all histograms in HistSet , and δ is the maximum Pearson distance between H n and all histograms in HistSet .
Since δ is independent of attacks and only relies on normal samples, HODA is not dependent on any attacks.
Notably, HODA does not need to save samples of each user or their hardness degrees.
It only keeps a vector ( H u ) that indicates the values of histogram bins for each user.
Algorithm 1 in Appendix F describes HODA in details.
S ETUP AND E VALUATION
Two normal sample sets S HODA and S u are required to evaluate the performance of HODA.
S u is used to simulate benign users.
We randomly select 40% and 60% of test samples of each dataset for S HODA and S u , respectively.
We randomly select num seq = 40000 sequences with size num s from S HODA to create H n and calculate δ .
To evaluate the performance of HODA against model extraction attacks, we simulate 10000 benign users and 10000 adversaries for each attack.
Each benign user sends a sequence of num s samples randomly selected from S u , and each adversary sends a sequence of num s samples randomly selected from 50000 samples of attack in the order they were generated.
So far, we have used 100 subclassiﬁers to calculate the hardness degree of samples.
However, it may not be possible to classify each sample by a high number of subclassiﬁers in practice.
So in order to reduce the computational cost of HODA, we only use 11 subclassiﬁers to calculate the hardness degree of each sample, and these subclassiﬁers are saved in the training phase of target classiﬁer f t at the end of each 10 epochs < f 0 t , f 9 t , f 19 t , f 29 t , f 39 t , f 49 t , f 59 t , f 69 t , f 79 t , f 89 t , f 99 t > .
Since the hardness degree domain depends on the number of subclassiﬁers, the hardness degree of a sample in HODA is in the range [0,10].
[Table]
To the best of our knowledge,
PRADA Juuti et al.
(2019) is the only defense that is comparable to
HODA.
PRADA declares that the histogram of minimum L 2 distance between a new sample and all previous samples of a benign user follows a Gaussian distribution.
Hence, it uses the Shapiro-Wilk normality test to determine that a sample sequence belongs to a benign user or an adversary.
Similar to HODA, PRADA also uses threshold δ to detect sample sequences of model extraction attacks, and δ is the only parameter of PRADA.
Since PRADA needs to save each user’s samples and calculate L 2 distance between them, it has a high computational overhead.
Table 3 indicates the detection rate and False Positive Rate (PFR) of PRADA and HODA against four various model extraction attacks on CIFAR10 and CIFAR100 target classiﬁers.
We evaluate
HODA when it only watches 50 and 100 samples of each user ( num s = 50 and num s = 100 ), and since PRADA needs to watch more samples to detect attacks, we use larger num s to evaluate PRADA.
PRADA and HODA have very low false-positive rates.
False-Positive Rate (FPR) indicates the percentage of benign users’ sample sequences wrongly detected as an attack.
The results demonstrate that HODA is very effective against model extraction attacks, and it outperforms PRADA by
[Figure]
a large margin.
Since HODA does not rely on the distance between samples, it can detect knockoff Net attacks that use natural samples.
HODA also has better performance on jacobian-based attacks.
The runtime and the number of samples that need to be stored by PRADA depend on the attack.
Nevertheless, for num s = 500 and CIFAR10 target classiﬁer, the average runtime of PRADA for each user is 0.47 seconds on Tesla K80 GPU, and 471 samples are stored for each user on average.
For each user, the average runtime of HODA is 0.0012 seconds, and it only stores a vector with size 11 representing a hardness degree histogram.
Although HODA requires the predictions ofmodels to calculate the hardness degree of each sample, there is no sequential relationship between models, and they can predict parallelly, so HODA does not increase the prediction time of target models.
5. T RANSFER L EARNING
Transfer learning is a machine learning technique that initializes the parameters of the target task classiﬁer using the parameters of a pre-trained source task classiﬁer.
We train two new target classiﬁers on CUB200 and Caltech256 datasets using transfer learning (details of datasets in Appendix
A).
The training process of new target classiﬁers is the same as CIFAR10 and CIFAR100 target classiﬁers (Section 4.1).
We initialize the parameters of target classiﬁers from a pre-trained ImageNet
Deng et al.
(2009) classiﬁer and train all layers of target classiﬁers.
Orekondy et al.
(2020) indicate that jacobian-based model extraction attacks have very poor performance on high dimensional datasets.
Thereby, we only evaluate the performance of target classiﬁers against K.Net ILSVRC attack.
K.Net ILSVRC12 is the Knockoff Net attack that uses ILSVRC12 dataset as the surrogate classiﬁer’s training set.
The budget of K.Net ILSVRC12 is 50000, and the output of target classiﬁers is the entire probability vector.
The accuracy of CUB200 target classiﬁer and its surrogate classiﬁer is 73.7% and 59.3%, respectively, and the accuracy of Caltech256 target classiﬁer and its surrogate classiﬁer is 77.2% and 72.2%, respectively.
[Table]
[Figure]
Figure 5 depicts the hardness degree histogram of
CUB200 and Caltech256 test sets on the associated target classiﬁer and also the hardness degree histogram of
K.Net ILSVRC12 samples for both target classiﬁers.
The ﬁgure demonstrates that the majority number of K.Net
ILSVRC12 attack samples are hard (hardness degree > 70), and the number of easy samples (hardness degree < 30) is very small.
We replicate the experiment of the previous section to evaluate the performance of HODA against K.Net ILSVRC12 attack with the same parameters.
Table 4 shows the performance of HODA against K.Net ILSVRC12 attack on both target classiﬁers.
The results demonstrate that even the starting point of target classiﬁers’ parameters is not random, HODA is very effective in detecting K.Net ILSVRC12 attack.
[Figure]
D ISCUSSION ON A DAPTIVE A DVERSARY
An adaptive adversary who is aware of HODA must send her queries based on the hardness degree histogram of normal samples to evade HODA.
We consider two scenarios for an adaptive adversary to conduct model extraction attacks.
In the ﬁrst scenario, the adversary has no access to normal samples, and she only can use synthetic or semantically similar samples to extract the target model.
There are two reasons why such attacks are hard to conduct.
First, the adversary needs samples with various degrees of hardness; however, since the adversary has no access to the target classiﬁer, she can not determine the hardness degree of her samples for the target classiﬁer.
Second, the adversary has no access to the histogram of normal samples to generate her samples based on it.
In the second scenario, we assume the adversary has access to a limited number of normal samples, and she can use normal samples to make her hardness degree histogram more similar to the hardness degree histogram of normal samples.
To evaluate HODA in this scenario, we suppose that the adversary has access to 1000 normal samples from S user and she sends a sample sequence of which
P n % is ﬁlled by normal samples, and the rest is ﬁlled by model extraction attack samples.
Notably, when the number of normal samples in the sequence exceeds 1000, the adversary sends duplicate normal samples.
It is important to note that the cost of attack is increased by a factor of
[Table]
[Table]
) in this scenario.
Figure 6 shows the detection rate of HODA for various P n over different values of num s .
The false-positive rate of all experiments is less than 0.2%.
The ﬁgure demonstrates that increasing num s improves the detection rate of HODA.
Except for K.Net attacks on CIFAR10 target classiﬁer in P n = 90% , HODA can detect all attacks with a high success rate by increasing num s .
Due to the dataset limitation, we can not evaluate HODA for num s > 4000 .
However, we think the detection rate of HODA against K.Net attacks on CIFAR10 target classiﬁer in P n = 90% will be improved for num s > 4000 .
Altogether, we think the main challenge of an adaptive adversary to evade HODA is to collect easy samples, which are very rare in out-of-distribution samples based on our experiments.
C ONCLUSIONS
This paper demonstrates that the hardness degree of samples is important in trustworthy machine learning.
We investigated the hardness degree of samples and demonstrated that the hardness degree histogram of model extraction attack samples is different from the hardness degree histogram of normal samples.
Using this observation, we proposed Hardness-Oriented Detection Approach (HODA) to detect sample sequences of model extraction attacks.
HODA can detect the sample sequences of model extraction attacks with a high success rate by only watching 100 samples of attacks.
A D ATASETS
CIFAR10 Krizhevsky (2009): CIFAR-10 dataset consists of 60K 32 × 32 color images in 10 classes, including airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks.
It has 6K images per class, where 5K images is in the training set and 1K images is in the test set.
CIFAR100 Krizhevsky (2009): CIFAR100 dataset consists of 60K 32 × 32 color images inclasses.
It has 600 images per class, where 500 images is in the training set and 100 images is in the test set.
TinyImageNet Le & Yang (2015): TinyImageNet is a subset of ILSVRC12 Deng et al.
(2009) dataset, and contains 200 image classes.
It has 500 training samples and 50 test samples for each class.
The size of images is 64 × 64 .
We resize all images to 32 × 32 .
CUB200 Wah et al. (2011): CUB200 dataset contain 200 classes of bird categories.
It consists of about 6K training and about 6K test samples.
The size of images is 224 × 224 .
Caltech256 Grifﬁn et al.
(2007): Caltech256 dataset contain 256 classes of common objects categories.
It consists of about 24K training and about 6K test samples.
The size of images is 224 × 224 .
ILSVRC12 Deng et al.
(2009): ILSVRC12 uses a subset of ImageNet and consists of 1.2 million training images, 50,000 validation images, and 100,000 test images.
The dataset has 1000 classes and the size of images is 224 × 224 .
B H ARDNESS T RANSFERABILITY
[Table]
In this section, we indicate that the hardness of samples is relatively transferable among various classiﬁers.
We use three classiﬁers created in Section 4.1 and a new ResNet18 classiﬁer in this experiment.
Table 5 displays the Pearson correlation coefﬁcients between hardness degree of CIFAR10 and CIFAR100 test samples for various pairs of classiﬁers.
The results demonstrate a positive and strong correlation between the hardness degree of samples for various pairs of classiﬁers.
Therefore, the hardness of samples is relatively transferable between different classiﬁers.
On the other side, it implies that the hardness degree of samples is relatively independent of the architecture of classiﬁers.
C D ETAILS OF M ODEL E XTRACTION A TTACKS
Jacobian-Based Dataset Augmentation (JBDA) Papernot et al.
The goal of JBDA attack is to increase the ﬁdelity of the surrogate classiﬁer to the target classiﬁer in order to produce adversarial examples for the target classiﬁer in the black-box setting.
The authors assume that the adversary has access to a limited number of normal samples called seed samples.
JBDA augment seed samples using adversarial examples to improve the performance of surrogate model.
The augmentation process is conducted in multiple rounds.
In the ﬁrst round, surrogate training set X s is initialized by seed samples, and surrogate model f s is trained on X s .
In the next rounds, sample set
S with size κ is randomly selected from X s , and for each x ∈ S , adversarial example x (cid:48) is created using the following equation:
[Equation]
6) where λ is step size and J is the Jacobian function. Afterward, new adversarial examples are labeled by the target model, and they are added to X s .
Lastly, surrogate model f s is trained on X
[Figure]
Jacobian-Based Random Target (JBRAND)
Juuti et al.
The goal of JBRAND is to improve the performance of JBDA.
It perturbs each sample in multiple iterations to generate more powerful adversarial examples and generates targeted adversarial examples with random targets.
We generate three adversarial examples with random targets for each sample and use the same seed samples as JBDA.
Each sample is perturbed in ﬁve iterations with (cid:15) =
64 225 × 5 .
The attack is implemented with λ = 64255 and κ = 2000 .
Knockoff Net (K.Net)
Orekondy et al.
(2019): Knockoff Net attack uses large public datasets that are semantically similar to the target model’s training samples to create the surrogate model’s training set.
It has adaptive and random strategies to select the surrogate classiﬁer’s training set, which both use semantically similar samples.
Since the adaptive strategy has very marginal beneﬁts, we only consider the random strategy to implement this attack.
K. Net randomly selects a subset of a public dataset and labels them using the target classiﬁer to create X s .
Finally, it uses X s to train surrogate classiﬁer f s .
D D ENSE N ET 121 T ARGET C LASSIFIERS
Figure 7 shows the hardness degree histograms of samples of four model extraction attacks on
CIFAR10 and CIFAR100 target classiﬁers.
The architecture of target classiﬁers is DenseNet121.
As seen in Figure 7, the hardness degree histogram of model extraction attack samples is distinguishable from the hardness degree of normal samples (Figure 1) for DenseNet121 target classiﬁers.
E V ISUALIZATION OF CIFAR100 SAMPLES AND T HEIR H ARDNESS
Figure 8 displays a two-dimensional visualization of CIFAR100 test samples using t-SNE.
Figure 8a uses the logits of CIFAR100 classiﬁer to visualize CIFAR100 test samples, and the color of each sample is determined by its label.
Figures 8b and 8c show the hardness degree of CIFAR100 test samples for CIFAR100 and CIFAR10 target classiﬁers, respectively.
Eachcolorisforsamplesofoneclass.
(a)
80 HardnessDegree (b) 0 .10 .
80 HardnessDegree (c)
[Equation]
[Table] [Table]
[Table]
[Table]
[Table]
HD
[Table] [Table]
HistSet
[Table]
[Table] [Table]
15: for
[Table]
do 16:
[Table]
D IST ( H n ,Hist )) 17: δ ← Max ( DistList ) 18: return H n ,δ 19: end function 20: function HODA( NewQuery,UserID,H n ,δ,num s ,SubClfSeq ) 21: AttackAlarm ← False 22:
H u ← GetUserHisogram( UserID ) 23:
HD ← GetHardnessDegree( NewQuery,SubClfSeq ) 24:
H u [ HD ] + = 1 25: if Sum ( H u ) =
=
Num s then 26: if P EARSON D IST ( H n ,H u ) > δ then 27:
AttackAlarm ← True 28:
SaveUserHistogram( H u ,UserID ) 29: return AttackAlarm 30: end function
G A DVERSARIAL E XAMPLES (AE S )
Adversarial examples (AEs) are maliciously crafted inputs that cause the target classiﬁer to misclassify them.
There are numerous methods to generate adversarial examples such as L-BFGS Szegedy et al.
(2014), FGSM Goodfellow et al.
(2015), C&W Carlini & Wagner (2017), PGD Madry et al.
(2018), and AutoAttack (AA) Croce & Hein (2020).
To investigate the hardness of adversarial examples, we use FGSM ( (cid:15) = 0 . 1 ), C&W ( L 2 ), PGD ( (cid:15)
= 8 / 255 , α = 3 / 255 ), and AA ( L ∞ , (cid:15)
= 8 / 255 ) attacks to generate adversarial examples on CIFAR10 and CIFAR100 test sets.
The adversarial examples are created in the white-box setting, and they are untargeted.
Figure 9 indicates the hardness degree of adversarial examples on CIFAR10 and CIFAR100 target classiﬁers.
Although the [Figure] distance between normal samples and adversarial examples is very small, Figure 9 demonstrates that the hardness degree histograms of adversarial examples are very different from normal samples (Figure 1).
Most adversarial examples generated by FGSM and C&W are harder than adversarial examples generated by PGD and AA.
We think this is because the size of perturbations added by FGSM and C&W is larger than PGD and AA.
An intriguing observation is that almost all adversarial examples being generated by PGD are not hard (hardness degree > 70).
AA has relatively the same behavior [Figure] and the number of its hard adversarial examples is very small.
H P ERFORMANCE A NALYSIS OF M ODEL E XTRACTION A TTACKS
To give new insight into model extraction attacks, we investigate the performance of model extraction attacks on normal samples with various levels of hardness.
For this purpose, the test sets of CIFAR10 and CIFAR100 datasets are partitioned into 10 hardness groups based on hardness degree of samples.
The hardness group i consists of samples that their hardness degree is in range [ i × 10 , ( i + 1) × 10] .
Hence, the ﬁrst hardness group consists of the easiest samples, and the last hardness group consists of the hardest ones.
Figure 10 shows the accuracy and the ﬁdelity of attacks over 10 hardness groups when the output of target classiﬁer is the entire probability vector.
The results demonstrate that the accuracy and the ﬁdelity of all attacks are decreased as the hardness of samples is increased.
We know from Figure 2 that the accuracy of target classiﬁers is decreased by increasing the hardness of samples. Figure 10 indicates the surrogate classiﬁers also have the same behavior.
The results demonstrate that the distance between the accuracy of target classiﬁers and surrogate classiﬁers (specially K.Net attacks) is increased by increasing the hardness of samples in the ﬁrst hardness groups.
However, the accuracy of the K.Net surrogate classiﬁers approaches the accuracy of target classiﬁers on the last two hardness groups.
To investigate this observation, Figureshows the percentage of samples being classiﬁed correctly by both surrogate classiﬁer and target classiﬁer for all attacks over various hardness groups with dashed lines.
The results indicate that all samples correctly classiﬁed by surrogate classiﬁers are also correctly classiﬁed by target classiﬁers in the ﬁrst two hardness groups.
However, by increasing the hardness of samples, the surrogate classiﬁers correctly classify some samples that are not correctly classiﬁed by the target classiﬁer, and the number of such samples is increased by increasing the hardness of samples.
Jagielski et al.
(2020) demonstrate that labels from the target classiﬁer are more informative than dataset labels.
We think the information in the labels that come from the target model causes the surrogate classiﬁers to correctly classify hard samples that are not correctly classiﬁed by the target classiﬁer.
The ﬁdelity of all surrogate classiﬁers is decreased by increasing the hardness of samples, which means that the disagreement among surrogate classiﬁers and target classiﬁers is raised on harder
[Figure]
samples. An intriguing observation is that the ﬁdelity of surrogate classiﬁers to the target classiﬁers on correctly classiﬁed samples by target classiﬁers is much more than wrongly classiﬁed samples.
I C OMPREHENSIVE RELATED WORK
Some recent studies investigate the dynamic of DNNs training process. Hacohen et al.
(2020); Frankle et al.
; Mangalam & Prabhu (2019) show that DNNs learn samples that are learnable by shallow models in early epochs of training before learning harder ones.
Hacohen et al.
(2020) demonstrate that DNNs learn samples in both training and test sets in a similar order.
Pliushch et al.
(2021) discuss correlation between learning order of samples with image statistics like segment count, edge strengths, image intensity entropy, and DCT coefﬁcient matrix.
Frankle et al.
(2020) indicate that the DNN-based classiﬁers undergo substantial changes in the ﬁrst few SGD iterations.
In the following, we brieﬂy review the most prominent model extraction attacks and defenses presented so far.
I. M ODEL E XTRACTION A TTACKS
For the ﬁrst time, Lowd & Meek (2005) demonstrate the possibility of stealing simple linear machine learning models through only interaction with them.
Tram`er et
al.
(2016) show the feasibility of model extraction attacks on commercial MLaaS. Papernot et al.
(2017) and Juuti et al.
(2019) investigate stealing DNN-based classiﬁers and propose jacobian-based model extraction attacks for creating a surrogate classiﬁer in order to generate adversarial examples in the black-box setting.
Chandrasekaran et al.
(2020) explore the connection between active learning and model extraction attacks.
They implement two query synthesis active learning algorithms to extract machine learning models, such as decision trees.
Jagielski et al.
(2020) use semi-supervised learning methods to improve the performance of model extraction attacks.
Knockoff Net Orekondy et al.
ActiveThief Pal et al.
(2020), and Copycat CNN da Silva et al.
(2018) use a semantically similar dataset to the target classiﬁer’s training set to create the surrogate classiﬁer’s training set.
They employ different strategies for selecting samples from attack datasets to extract more information from the target classiﬁer.
Yu et al.
(2020) employ active learning, transfer learning, and a new method for generating adversarial examples to improve model extraction attacks efﬁciency.
A line of studies
Truong et al.
; Kariyappa et al.
(2021a)
; Barbalau et al.
(2020) use synthetic data to create the training set of surrogate classiﬁers.
Although their methods do not need to have access to natural samples, they send a high number of queries to the target classiﬁer, which makes their methods impractical.
For example, Truong et al.
(2021) and Kariyappa et al.
(2021a) send millions of queries to extract a CIFAR10 target classiﬁer.
While most model extraction attacks have focused on the vulnerabilities of image classiﬁers, recent studies demonstrate the vulnerability of NLP Krishna et al.
Graph DNN He et al.
(2021), and
Reinforcement learning Chen et al.
(2021) models against model extraction attacks.
Another type of model extraction attack uses hardware side-channel vulnerabilities to extract a target classiﬁer Zhu et al.
; Batina et al.
; Hong et al.
Yan et al.
However, these attacks have a very strong threat model and suppose the adversary has access to the hardware that hosts the target classiﬁer.
I. D EFENSES AGAINST M ODEL E XTRACTION A TTACKS
Existing defense methods against model extraction attacks generally distribute into two branches: perturbation-based and detection-based defenses.
Perturbation-based defenses Lee et al.
Orekondy et al.
(2020); Kariyappa & Qureshi (2020) attempt to prevent adversaries from producing high-quality surrogate classiﬁers by adding perturbation to the output of target classiﬁer.
These methods generate the perturbation with various strategies to minimize the accuracy of surrogate classiﬁers.
Recently, Kariyappa et al.
(2021b) proposed a new defense with the same goal as perturbation-based defenses, which does not perturb the output of target classiﬁers.
Their approach employs an ensemble of diverse models to produce discontinuous predictions for out-of-distribution samples.
Proposed detection-based defenses Kesarwani et al.
Juuti et al.
(2019) attempt to detect the occurrence of model extraction attacks by observing successive input queries to the target classiﬁer.
Kesarwani et al.
(2018) propose a method to measure adversary perceived knowledge from target classiﬁer, but this method only works for Decision Tree models.
PRADA Juuti et al.
(2019) is the ﬁrst proposed detection-based defense for DNN models.
PRADA uses the histogram of the minimum L 2 distance among a new sample and all previous samples to detect model extraction attacks.
Aside from its high computational overhead, it has been shown that Prada is unable to detect model extraction attacks when an adversary uses natural samples Pal et al.
Watermarking neural networks Jia et al.
; Zhang et al.
; Szyller et al.
; Adi et al.
(2018) is another type of defense against model extraction attacks.
These methods prove ownership of a surrogate classiﬁer instead of preventing the occurrence of model extraction attacks.