MExMI: Pool-based Active Model Extraction Crossover Membership Inference
Abstract
With increasing popularity of Machine Learning as a Service (MLaaS), ML models trained from public and proprietary data are deployed in the cloud and deliver prediction services to users.
However, as the prediction API becomes a new attack surface, growing concerns have arisen on the conﬁdentiality of ML models.
Existing literatures show their vulnerability under model extraction (ME) attacks, while their private training data is vulnerable to another type of attacks, namely, membership inference (MI).
In this paper, we show that ME and MI can reinforce each other through a chained and iterative reaction, which can signiﬁcantly boost
ME attack accuracy and improve MI by saving the query cost.
As such, we build a framework MExMI for pool-based active model extraction (PAME) to exploit MI through three modules: “MI Pre-Filter”, “MI Post-Filter”, and “semi-supervised boosting”.
Experimental results show that MExMI can improve up to 11 .
14% from the best known PAME attack and reach 94 .
07% ﬁdelity with only 16k queries.
Furthermore, the accuracy, precision and recall of the MI attack in MExMI are on par with state-of-the-art MI attack which needs 150k queries.
Introduction
Recent advances in machine learning (ML) has signiﬁcantly shifted the paradigm in all walks of life.
Thanks to cloud computing, every business and individual can host black-box ML models in the cloud (e.g., Microsoft Azure ML, Amazon AWS ML, and Google Cloud AI) to provide pay-per-query predictive services, known as Machine Learning as a Service (MLaaS).
Obviously, ML models are proprietary assets to the providers who spend great efforts training them [22].
However, by exploiting the correspondence between queries and prediction results from an MLaaS model, an adversary could learn the internals of that victim model to a large, or even full extent.
Such attack on model’s conﬁdentiality is known as model extraction (ME) [43].
It is a fundamental attack in adversarial machine learning because it enables follow-up attacks such as adversarial samples [19], model evasion [7], and model inversion attacks that infer private information of the training set [16, 48].
State-of-the-art ME attacks can be categorized into direct discovery [23] and active-learning-based
ME attacks [6, 34].
Active learning (AL) refers to those semi-supervised training methods that aim to ﬁnd the most informative training dataset with a limited query budget [17].
Since both AL and ME aim to train a model with as few queries as possible, AL has become increasingly popular in model extraction attacks.
Depending on the availability of real-life samples for querying, AL-based ME can be further divided into pool-based active model extraction (PAME) and query-synthesizing-based active model extraction (SAME).
The former assumes the presence of samples, from which query samples are iteratively selected using a pool-based or stream-based AL.
Classic PAMEs include
ActiveThief [34] and Knockoff [33].
The latter does not assume the presence of such samples and obtains them by generative methods [42, 6].
Classic SAMEs include black-box DNNs [35] and
PRADA [26].
However, a critical difference between pool-based AL and pool-based ME has been long neglected.
Since pool-based AL is essentially a training method, the data pool is where training samples are drawn, which means this pool must have the same feature distribution as the training dataset.
However, in an ME attack the adversary has no access to the victim model’s training dataset or even samples in the same problem domain [10].
Unfortunately, all previous PAME works ignore this difference and assume the samples in the pool are homogeneous to those in the training dataset.
To demonstrate how non-homogeneous datasets can affect the ME performance, we build a Wide-ResNet-based victim model using the ﬁrst 25k training image samples of CIFAR10.
Then we extract this model using ActiveThief [34], a state-of-the-art PAME, from three adversary data pools: (1) pool A consists of the second 25k training samples of CIFAR10, (2) pool B consists of the middle 25k training samples, and (3) pool C consists of the ﬁrst 25k training samples.
In other words, the victim’s training set does not overlap with A (non-homogeneous) but overlaps with half of B (quasi-homogeneous), and completely with C (fully homogeneous).
Table 1 shows the ﬁdelity (i.e., similarity to the victim model) of the extracted copy models from A , B and C .
We observe that homogeneous samples contributes more to the success of model extraction than non-homogeneous samples.
[Table]
In this paper, we address this issue by identifying homogeneous samples in the data pool and making full use of them for a “guided" model extraction.
We exploit membership inference (MI), an attack that infers the training samples from a given dataset [41], to select them and train a copy model.
In turn, the extracted model can enhance the accuracy of membership inference.
As such, we propose an iterative extraction framework MExMI where ME and MI reinforce each other through iterations.
As such, within limited query budget, the ﬁnal outcome consists of both a high-ﬁdelity copy model and an accurate set of training samples.
There are several challenges in the design of MExMI.
First, existing MI attacks do not consider query cost [41, 39], which is essential in ME attacks.
We exploit the training data property of the copy model to conduct MI attacks without consuming query budget.
Second, state-of-the-art MI attacks impose assumptions that are not available in MExMI or PAME in general.
For example, a shadow-model MI requires a labelled dataset drawn from the same distribution as the model’s training dataset with the same size [41].
In this paper, we propose a quality metric to evaluate and optimize a shadow model without the need of a large labelled dataset.
With this technique, we adapt both shadow-model MI and unsupervised MI [39, 47] to the MExMI framework.
Third, existing PAME attacks fail to utilize the training samples puriﬁed from the pool.
We designed three modules to facilitate PAME to make the most of it.
In summary, we make the following contributions:
• We propose a taxonomy (SAME/PAME) on active-learning-based model extraction attacks, and an iterative framework MExMI where PAME and MI reinforce each other. • We adapt both shadow-model MI and unsupervised MI to MExMI.
To boot-strap shadowmodel MI, we develop an indicative quality metric of shadow models and design a metricbased shadow-model training algorithm.
• We conduct extensive experiments to compare the effectiveness of MExMI with state-ofthe-art PAMEs [34].
MExMI outperforms the latter by 11 .
14% to 94 .
07% in terms of ME ﬁdelity, and its MI attack achieves 84 .
13% precision, on par with the state-of-the-art MI attack [39] ( 75 . 25% ) that assumes unlimited query budget.
The rest of the paper is organized as follows.
Section 2 introduces the background and deﬁnitions of ME and MI attacks.
Section 3 overviews the MExMI framework and its key components, and
Section 4 introduces adaptive MI algorithms for MExMI.
Section 5 presents the evaluation results of
MExMI through extensive experiments.
Section 6 reviews related works, and Section 7 concludes this work.
Background and Deﬁnition 2.1 Victim Models
Our target victim models are multi-class neural network (NN) classiﬁers, where the input domain is X ⊆ R d and the output is a probability vector of classes
Y ⊆
R K .
Here d denotes dimensionality of the input data, and K denotes the number of output classes.
A neural network model makes inference on the output class probabilities y based on the input x : y =
[Pr ( Y = 0 | x ) , . . .
, Pr ( Y = K | x )] .
To train such a model, we assume a supervised learning process.
2 Let D =
( x i , y i )
ni
=1 ⊆
X × Y denote the victim’s training dataset with n labelled samples, where x i is the i -th training sample, and y i is its one-hot format label vector.
2.2 Threat Model
We assume a well-trained black-box victim model F is deployed in a machine-learning-as-a-service (MLaaS) with a chargeable query interface.
For ease of presentation we do not consider the case when the victim employs anti-ME [26] or anti-MI defenses, such as perturbation on the output probability vector [52, 53].
We also assume that the adversary either only has black-box access to the MLaaS model, or knows its architecture (e.g. in AWS Marketplace [1] and Huawei AI Gallery [2]).
In addition, the adversary can collect a large number of unannotated public samples to construct an adversary data pool P .
2.3 Problem Deﬁnition
We assume the adversary performs ME attacks against F within a query budget b , with the aim to produce a copy model F (cid:48) .
The main objective of ME is to let F (cid:48) approximate the victim model F with high degree of resemblance, a.k.a., ﬁdelity [23], which is the proportion of label agreement of two models on an evaluation dataset D t .
Model Extraction Crossover Membership Inference
In this section, we present our iterative model extraction framework MExMI where ME and MI reinforce each other.
An MI attack aims to distinguish those individuals ˆ D from a population P that exist in the victim model F ’s training dataset [41].
As illustrated in Fig.
1 and pseudocode in
Appendix A, the input of MExMI is an adversary data pool P and the access to a black-box victim model F , and its outputs are the copy model F (cid:48) and the inferred training dataset ˆ D .
In the ﬁrst iteration, the adversary chooses k initial seed samples [ x 1 , . . .
, x k ] 0 from P without putting them back, where k is the query budget per iteration.
These samples are fed to the victim model F , which outputs a probability vector F ( x ) .
Then an MI attack model F MIA is constructed using the queried dataset [ { x 1 , F ( x 1 ) } , . .
0 or the copy model (see Section 4).
F MIA is used in both MI Post-Filter and MI Pre-Filter modules.
The queried dataset is then passed through the MI Post-Filter, which weighs them according to their probability of being a training sample of the victim model.
Then the weighted queried dataset is used to train the copy model F (cid:48) , which is then fed to the MI Pre-Filter to reﬁne the adversary data pool for AL sample selection of k queries in the next iteration.
The process is repeated until the total query budget b is depleted.
Thereafter, without consuming any query budget,
F MIA is used to launch MI attacks on F (cid:48) and the data pool to obtain the ﬁnal inferred training dataset ˆ D .
This dataset will be used in a semi-supervised learning on F (cid:48) to release the ﬁnal copy model F (cid:48) .
From Fig.
1, MExMI has three key modules on top of the basic PAME iterative framework [34], namely, MI Pre-Filter, MI Post-Filter and semi-supervised boosting that will be elaborated in rest of this section.
As will become clear, they are orthogonal to each other, so they can be turned on or off
[Figure] independently. For ease of presentation, the construction of the MI attack model F MIA , a key issue in the MExMI framework, will be back introduced in Section 4.
3.1 MI Pre-Filter
MI Pre-Filter works before AL sample selection.
The core idea is to use an MI attack model F MIA to choose from the adversary data pool P only those samples that are highly homogeneous to the victim’s training data D .
Training the copy model with them, both models can thus exhibit strong resemblance.
Ideally, F MIA should perform membership inference attack directly on the victim model F , which is truly trained from D .
However, since such training causes extra query budget on
F , F MIA attacks the copy model F (cid:48) instead as the latter exhibit similar training data property. 3.2 MI Post-Filter
MI Post-Filter works after querying a sample x from the victim model.
The rationale of this ﬁlter is that since the victim model returns the probability vectors F ( x ) , the adversary can infer if x belongs to the training dataset D by an MI attack model F MIA .
Obviously a negative membership result means this sample may not lead to a high-ﬁdelity copy model F (cid:48) , so its contribution to the training process should be lowered by reducing its weight in the training loss calculation.
We use a parameter ω ( ω > 1 ) to denote the weighted loss ratio of a positive membership sample to a negative one.
3.3 Semi-Supervised Boosting
The main idea of this module is that MExMI gain the results of its MI attack— an inferred training dataset ˆ D of the victim model.
Although this dataset is not labelled and there is no more query budget to label them in the end of MExMI, we can still train the copy model F (cid:48) on this dataset together with the queried set using semi-supervised learning algorithms.
Note that this module is unique in MExMI as other PAME methods cannot distinguish training samples from others in the data pool.
Nonetheless, this module is intended to improve model accuracy only, not ﬁdelity, because semi-supervised learning can divert the copy model’s ability to follow the same label distribution as the victim model’s training data.
Therefore, MExMI only applies semi-supervised boosting after the ﬁnal iteration and when higher accuracy is needed .
Adaptive Membership Inference
In MExMI an MI attack model F MIA is trained after the initial seeds query.
For MI to play a role in
MExMI framework in early stage iterations, F MIA must be accurate enough even when there are only few samples.
In this section we renovate existing MI algorithms to be adaptive to their training sample sizes and thus suitable for the MExMI framework.
We focus on two state-of-the-art black-box
MI attack paradigms: (1) shadow-model MI [41, 39], and (2) unsupervised MI [39].
4.1 Shadow-Model Membership Inference
The rationale of shadow-model MI is to obtain a shadow model similar to the victim model, so that their output probability vectors for training and non-training samples are also distinguishable in a similar manner.
As such, the adversary can build a binary MI classiﬁer from these samples instead of from those of the victim model.
In order to approximate the victim model, a shadow model should (1) draw its training dataset from the same distribution as in the victim model, (2) have roughly the same size of training set as the victim model, and (3) have the same training algorithm.
However, in
MExMI, neither (1) nor (2) holds because:
a) The queried samples may not be drawn from the same distribution as in the victim’s training samples.
b)
The number of queried samples is signiﬁcantly smaller than that of the victim’s training set size, especially in the beginning phase.
As such, the design principle of our adaptive shadow-model MI attack is to work under a limited number of labelled training samples in a different distribution from the victim model.
To start with, we need to know how to estimate the quality of a shadow model so that we can tell when it is good enough for MExMI.
Intuitively, the ﬁdelity of the shadow model against the victim model can serve as the performance indicator, but it is inaccessible from the adversary’s side, especially when
MExMI just starts.
As such, we need an easy-to-access performance metric.
4.1.1 Measuring Quality of Shadow-model MI
An MI attack works by distinguishing the output probability vector distribution Y of the victim model
F on training samples X from non-training samples.
A shadow-model MI estimates the above on a shadow model — it distinguishes the output probability vector distribution Y ( s ) of shadow models
F s on training samples X
( s ) from non-training samples.
As such, to improve the attack accuracy,
Y ( s ) should be as similar as possible to the victim
’s Y .
We measure the similarity by the bias of the expectation values between Y ( s ) and Y , denoted by b .
For each shadow model that targets at class j ∈ [1 , . . .
, K ] , b j is formally deﬁned as:
[Equation]
where n denotes the size of training set, and the superscript ( s ) denotes the shadow model.
The following theorem shows that the gap between the training loss ∆ l is positively correlated to the bias of expectations between Y ( s ) and Y .
Therefore, it can serve as a quality measurement for F s .
We can minimize it to enhance the quality of Y ( s ) approximating Y .
Theorem 1 ( Quality measurement for shadow-model MI ).
Given a shadow model F s which has the same model structure and hyper-parameters as the victim model F , the gap ∆ l between the training loss of F s and F is positively correlated to the bias of expectations between Y ( s ) and Y , i.e., ∆ l ∝ b .
See Appendix C for the proof.
4.1.2 Metric-based Shadow-model MI
To obtain ∆ l , the gap between the training loss of both models, we need them both.
However, the victim’s training loss is not available to the adversary.
Nonetheless, in practice a victim model is valuable for extraction mainly because this model accurately predicts the training data, or equivalently, its training loss is smaller than other models of the same training set size.
As such, we can replace ∆ l with the loss of the shadow model, denoted by l s , compensated by its training dataset size n ( s ) .
Furthermore, for the sake of comparing various shadow models, only the relative rather than the absolute value of ∆ l is needed.
So we propose the following metric Q as a relative value of ∆ l .
The larger the Q , the better the quality of a shadow model.
Formally,
[Equation]
where f ( · ) is a non-negative non-linear decreasing function, and a ∈ [0 , 1] scales down the impact of n ( s ) .
Note that Q does not require that shadows’ training data come from the same distribution as the victim’s.
From the above equation, there are two ways to increase the quality metric Q : (1) reducing l s using good training algorithms on shadow models, and (2) increasing n ( s ) by augmenting the training set.
Few-shot learning (FSL), a training paradigm to improve models’ accuracy with a limited number of examples [45], can serve both purposes.
For example, we can use FSL approaches, namely data augmentation [29] and transfer learning [21], to train shadow models.
This leads to two metric-based shadow-model MIs, which are elaborated in Appendix D.
4.2 Unsupervised Membership Inference
Recent works [39, 47] have shown the effectiveness of unsupervised learning on MI attack models.
In such models, the feature values are usually the top- m score, loss or entropy of the output probabilities, and the output value serves as the conﬁdence of membership inference — if the value is higher than an adversary-speciﬁed threshold c , the sample is inferred as in the training dataset and vice versa.
To set this threshold, the adversary ﬁrst gets a set of non-member samples and then query them to get corresponding top- m scores.
The top t percentile of these scores can serve as a threshold [39].
To save the query cost, the copy model instead of the victim model should be queried.
Experiment
In this section, we ﬁrst conduct experiments to validate the shadow model quality metric Q .
Then we evaluate the attack performance of four variants of MExMI — Pre-Filter only, Post-Filter only,
MExMI without semi-supervised boosting, and MExMI — against state-of-the-art ME and MI attacks. The codes are available at https://github.com/mexmi/mexmi-project .
5.1 Experiment Setup
Datasets .
We perform PAME attacks on two image datasets, namely CIFAR10 [28] and Street
View House Number (SVHN) [32], and a text dataset AG’S NEWS which contains corpus of AG’s news articles [11] (see Appendix E.1 for details).
Victim Model .
For the image classiﬁcation task, we use Wide-ResNet-28-10 [49] trained on
CIFAR10 as the victim model with an accuracy of 96 .
We also use a cloud MLaaS, ModelArts [3], to train an online victim model on SVHN that leads to 94 .
30% accuracy.
In the text classiﬁcation task, we use DPCNN [25] as the victim model which achieves an accuracy of 89 . 88% .
Adversarial data pool .
In the default CIFAR10 experiments, the pool consists of 50k training samples and 100k from the ImageNet32 [9].
In the default AG’S NEWS experiments, the pool has 50k training samples and 100k from Dbpedia [4].
Note that as with existing pool-based ME [34],
MExMI does not require the pool to contain training samples .
The main reason for such a mixed dataset composition is for us to evaluate the performance of MI [41] and show how much it can be enhanced by ME.
In Section 5.4, we evaluate the performance of MExMI when the pool has no training sample at all.
5.2 Metric-based Shadow-Model MI
Implementation Details .
Recall that the hyper-parameters (such as epoch, initial learning rate, and optimizer) of shadow models F s can be adjusted to maximize the metric Q .
Parameter a in Q is set as 0 .
05 and f ( · ) is set as − log 10 ( · ) .
The training dataset of F s , denoted by D ( s ) , is constructed by random sampling from P , and its size varies from the set { 2000 , 5000 } .
A non-training dataset, denoted by D ( s ) n , is also randomly sampled from P \ D ( s ) .
We implement two metric-based shadowmodel MI methods in Section 4: original shadow-model MI [41] and FSL shadow-model MI that utilizes transfer learning [21] and data augmentation respectively.
(See Appendix E.5 for experimental details.)
For each MI method, we vary their hyper-parameters and thus Q in various settings.
An ideal shadow-model MI is built as a reference by using the same training dataset as the victim model.
Results .
We plot the MI attack models’ accuracy with respect to shadow metric Q in Fig.
We observe that Q is positively correlated with the attack accuracy of shadow-model MI and therefore can guide the training process of the shadow models, whether using the original or FSL training algorithms.
The recall rate of all MI attacks is almost 100% , so it is omitted from the ﬁgure.
In addition, once the metric Q is large enough ( ≥ 6 ), metric-based shadow-model MI attack models can achieve almost the same accuracy as the ideal shadow-model MI [39] even for 2k samples. [Figure]
5.3 Overall Performance of MExMI 5.3.1 Implementation Details
We compare six PAME attacks, including four MExMI variants, namely, baseline ME without MI,
Pre-Filter only, Post-Filter only, MExMI without semi-supervised boosting, regular MExMI (which adopts Mix-Match semi-supervised methods [5] for image classiﬁers, and consistency regularization [38] semi-supervised methods for text classiﬁers), and the ideal ME attack.
The baseline ME is
ActiveThief [34], which is the state-of-the-art PAME attack.
The ideal ME uses the real training samples as its pool.
Moreover, to comprehensively evaluate MExMI, we added two additional state-of-the-art query-synthesizing-based ME baselines, including PRADA [26] and DFME [44] which share the same query budget with PAME attacks.
Each MExMI variant has a MI result.
In our experiments we compare our MI attacks that don’t cost additional query budget with the existing MI attacks [39] that assume inﬁnite query budget.
The AL algorithms used are entropy uncertainty [31], greedy k-center [40] and adversarial deepfool [14] (see Appendix B for a brief explanation).
The last algorithm is not evaluated on AG’S
NEWS since there is no trivial way to adapt it to text classiﬁcation.
The copy models in MExMI share the same architecture as the victim models.
The MI attack model F MIA is trained on initial seed samples.
The preset weights ratio ω in MI Post-Filter is 5 : 1 .
For CIFAR10 experiments, MExMI queries 2k samples in each round with a total of 8 rounds.
For AG’S NEWS experiments, there are 6 rounds, each with 5k samples.
All experimental results are the average measures of 5 trials.
See
Appendix E for more implementation details and Appendix F for supplementary experimental results.
5.3.2 Overall Results of MExMI
We use ﬁdelity and model’s accuracy obtained from the test datasets D t (CIFAR10 and AG’S NEWS test sets) to evaluate copy models against the victim model (see Section 2.2), and use accuracy, precision and recall of the inferred training datasets against ground truth to evaluate the MI accuracy.
Fig.
3 plots the ﬁdelity of various PAME attacks with respect to iterations on CIFAR10 and AG’S
NEWS (see more ﬁgures in Appendix F.1), respectively 5 , and Table 2 shows the ﬁnal results.
Fig.plots the accuracy, precision and recall of the MI attack of each MExMI variant.
The results indicate that MExMI greatly boosts the potential of AL algorithms in PAME, and breaks the curse on query budget of existing MI.
Overall, MExMI performs the best and achieves a ﬁdelity gain of 7 .
76% , 7 . 8% , and 11 .
14% on CIFAR10 over the baseline ME attack in all three AL methods.
A similar gain of 4 .
46% and 3 .
46% is observed on AG’S NEWS over the baseline PAME attacks in both two AL methods.
Without additional queries, the MI attacks of MExMI yield up to 83 . 20% accuracy, 84 .
13% precision and 75 .
93% recall on CIFAR10, and 68 .
77% accuracy, 71 . 73% precision and 82 .
53% recall on AG’S NEWS respectively, both on the par with existing MI attacks [39] that assume inﬁnite query budgets.
Impact of MI Post-Filter and MI Pre-Filter .
In Fig.
3, on CIFAR10 attacks with MI Post-Filter always outperform those without it, by up to a 1 .
64% increase on ﬁdelity in the ﬁnal results.
The gain is more eminent in the beginning iterations, and then gradually decreases.
On AG’S NEWS, MI art work that adopts model inversion to assist ME. We do not include it in the experiments for two reasons. First, its performance is similar to ActiveThief under our 16k-query budget setting. Second, it augments query selection from the data pool with query synthesis from the model, which can be considered orthogonal to our method.
[Figure] [Figure] Post-Filter also performs effectively, with a maximum boost of 1 .
57% on ﬁdelity over those without it.
As for MI Pre-Filter, except for the greedy k-center one in CIFAR10 experiments, attacks with
Pre-Filter always outperform those without it.
In addition, the gain does not decrease with more iterations, because the adversary data pool is much larger than the total query budget.
Interestingly, we ﬁnd that when MI Pre-ﬁlter and MI Post-ﬁlter work together, they can achieve a greater gain on ﬁdelity than the sum of individual gains when they work separately.
This suggests that the two ﬁlters truly reinforce each other in our MExMI framework. [Table] AG’S NEWS [Table]
PRADA [26]/DFME [44]
[Table]
Impact of Semi-Supervised Boosting .
In Fig.
3 and Table [Table] MExMI outperforms MExMI without semi-supervised boosting by at least 3 .
11% on CIFAR10 and 0 .
56% on AG’S NEWS in terms of accuracy, which indicates that MExMI does beneﬁt from effective MI attacks.
A ﬁdelity gain is observed in MExMI since the copy models’ accuracy is closer to that of the victim model.
MI Performance in MExMI .
The precision and recall of the adaptive shadow-model MI attacks of three MExMI variants are shown in Fig.
MExMI always performs the best and can achieve up to 83 .
20% accuracy and 84 .
13% precision on CIFAR10, and 68 . 77% and 71 .
73% precision on
AG’S NEWS.
This precision is even better than the state-of-the-art MI — ML-leaks [39] ( 75 . 25% on CIFAR10, 65 . 37% on AG’S NEWS) which assumes unlimited query budget.
Furthermore, our adaptive MI attacks have no additional cost when inferring training samples. Discussion About Potential Defenses. There are two potential defenses against MExMI. First,
MExMI is subject to MI-related defensive strategies that can reduce the accuracy of MI, such as using differential privacy [13], which in turn lower the ﬁdelity of MExMI.
Second, as with other model extraction attacks, MExMI can also be defended by model provenance techniques, such as watermark embedding [24], to detected copyright infringement from an extracted model.
5.4 Impact of Adversary Data Pool on PAME
In this experiment [Table] we study how the quality of adversary data pool affects the outcome of PAME attacks.
Since there are cases where the data pool does not contain any training data, the results of MI attack are not evaluated.
To be fair, we ﬁx the size of the pool to 150k samples and change the proportion of training data in it to vary its quality.
Since the total training samples are 50k, this proportion is capped at 1 / 3 .
The results on CIFAR10 are shown in Table 3.
We can see that the quality of the data pool greatly affects each PAME attack.
MExMI consistently outperforms the baseline irrespective of the quality, even in the complete absence of victim’s training data, i [Table] e., when the adversary has no access to the true training samples.
In such extreme case, we also observe that Pre-Filter only is outperformed by the baseline.
This is due to the fact that the Pre-Filter cannot ﬁnd any training data in the pool and thus excludes most of them for training.
Since the ﬁltered data pool is too small, active learning might not be effective.
[Table]
5.5 Impact of Output Access
We investigate the impact of output access granted to our PAME attacks.
We limit the output access to top-1 score and show the results on CIFAR10 in Table 4.
It indicates that even with limited output access, the three modules of MExMI still perform consistently well.
Among various PAME attacks,
MExMI always performs the best and can outperform the baseline by 11 .
47% on ﬁdelity.
[Table]
5.6 Case Study: Blackbox Attacks on MLaaS ModelArts
[Figure]
We use a real case study to show the feasibility and real-life impact of MExMI.
We target at ModelArts , the MLaaS provided by
Huawei Cloud [3] where developers can train and deploy their ML models in the cloud, and then access them via Web API (e.g., CURL).
We train and deploy a classiﬁcation model on ModelArts without the knowledge of its architecture using
SVHN and then perform various PAME attacks on it with VGG16.
See Appendix E.6 for more implementation details.
[Table]
The experimental results of the adversarial deep-fool
PAME are shown in Fig.
All four MExMI variants outperform the baseline.
MExMI achieves the highest 90 .
32% ﬁdelity and MExMI without semi-supervised boosting comes the second with 89 .
45% ﬁdelity.
The ﬁnal MI results of MExMI framework are shown in
Table 5.
We observe similar precision and recall of the three variants, all on the par with ML leaks [39], the state-of-the-art unsupervised MI attack that exhausts all pool data, which costs ten times higher.
Related Works
Model Extraction .
Model extraction (ME) on MLaaS models has drawn a lot of attention since
Tram ` e r, et al.
[43], mainly because it can be used for downstream attacks.
In terms of query data availability, ME can be categorized into query-synthesizing ME and query-acquiring ME.
The former is used when the adversary does not possess enough real data for query, which includes iterative active frameworks [35, 26] and minimax-game frameworks [44, 27].
Despite saving the data collection cost, it suffers from huge query budget demands.
The latter is used when the adversary spares extra cost to actively collect real data for query [10, 23, 34, 10].
We summarize the above ME attacks in Table 6, and highlight the data pool requirement of each work and their dependency on data selection and data synthesis.
Memership Inference .
The vulnerability of MLaaS models’ training data privacy has raised great conerns [51, 22] and various attacks against it are proposed. Of these, Membership inference (MI) which attempts to infer the training membership of some query data, shows a worrying breach of data privacy.
Existing works exploit different sources of information such as outputs [41, 39] and model parameters [30] in either black-box or white-box setting to infer the training data.
In black-box attack scenario, moet MI works follow shadow-model MI [41, 39] which cast the attack to a supervised learning problem. Another type of black-box MI is to utilize the unsupervised learning [39, 37] with different measure metrics, such as output entropy, predicted conﬁdence [39] and loss [37].
On the other hand, a white-box MI is proposed [30] which infers the training data and non-training data distribution characteristics via the victim model’s parameters.
More recent works focus on extending MI attacks to less favorable scenarios, for example when the model output is label only [8, 30], and extremely low query budget.
[Table]
Conclusion
In this paper, we proposed a PAME crossover MI framework called MExMI where the model and training data privacy can trigger a chain reaction to boost the performance of both attacks.
The framework is generic in that it can adopt various PAME and MI attacks.
In our experiments, MExMI improves the ﬁdelity of copy models to 94 .
07% , up from the basic PAME by 11 .
Meanwhile, the MI accuracy and precision can reach 83 .
20% and 84 .
13% without additional query budget, on par with state-of-the-art MI attack which requires about 10 times more queries. Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China (Grant No: 62072390, 62102334), the Research Grants Council, Hong Kong SAR, China (Grant No: 15222118, 15218919, 15203120, 15226221, 15225921, and C2004-21GF), and a Huawei research grant (TC20200831001).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contribu- tions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.4 and the discussion about potential defenses in Section 5.3.2.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 1 and Section 5.6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Appendix C. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix C. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 5.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 5.1, Section 5.2, Section 5.3.1, Section 5.6, Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Figure 3 and Figure 8 in Appendix F.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E.2.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Section(d)
Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
[N/A]
The data sets that I used are public.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable informa- tion or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Appendix A MExMI Algorithm
1: Input: F , P ; Parameters: k , b , ω ; 2: Output: F (cid:48) , ˆ D ; 3: [ x 1 , . . . , x k ] 0 ← RandomPick ( P ); 4: P r ← P \ [ x 1 , . . . , x k ] 0 , d ← d − k , i = 0 ; 5: D q ← [ { x 1 , F ( x 1 ) } , . . . ] 0 ; 6: F MIA ← MIUpdate ( D q , P ); (cid:46)
Section7: while b > = 0 do 8:
D q + , D q − ← F MIA ( D q ) ; 9:
AssignWeigtht ( D q + , D q − , ω ); 10:
F (cid:48)
i ← WeightLossTrain ( D q + , D q − ); 11:
P t ← F MIA ( F (cid:48)
i , P r ); 12: [ x 1 , . . .
, x k ] i +1 ← ActiveLearning ( P t , F (cid:48) i ); 13:
P r ← P r \ [ x 1 , . . .
, x k ] i +1 , 14:
D q ← [ { x 1 , F ( x 1 ) } , . .
i +1 ∪ D q , d ← d − k , i ← i + 1 ; 15: end while 16: ˆ D ← F MIA ( F (cid:48) , P ); 17: F (cid:48) ← SemiSupervisedTrain ( D q , ˆ D );
B Pool-based Active Learning Algorithms
MExMI can use various pool-based AL algorithms available in the literature.
In this paper, we focus on three AL algorithms based on different metrics of samples: uncertainty, diversity, and vulnerability to deep-fool perturbation.
As AL algorithms often calculate the distance between samples [50], MExMI also provides an encoding process to reduce dimension for high-dimensional feature space.
In essence, it uses F (cid:48)
i − 1 , the copy model trained in the previous iteration, as an encoder.
When inter-sample distances are required, we use inter-vector distances between the outputs of the encoder for calculation.
The distance is calculated using the output probability vectors of F (cid:48) whenever the calculation of the distance is needed.
B.1 Entropy Uncertainty
One of the most common ways to measure uncertainty is entropy [31].
The larger the entropy, the higher the un- certainty level is.
For a sample { x , ˆ y } in the data pool, where ˆ y = F (cid:48) ( x ) =
[Pr(1 | x ) , Pr(2 | x ) , . .
Pr( K | x )] T , its entropy is deﬁned as
[Equation]
where K is the number of class labels.
B.2 Greedy K-center
One classic diversity-based AL is the greedy k-center algorithm [40].
Let D q :=
[ { x q , F ( x q ) } , . .
denote the set of samples selected previously, and P :=
[ x p , . .
the data pool.
Greedy k-center algorithm sets [ F (cid:48)
( x q ) , . . . ]
as cluster centers and selects the sample x s that has the largest Euclidean distance from all existing centers.
Formally,
[Equation]
[Equation]
Then we query the selected sample x s and update D q by D q = D q ∪ { x s , y s } , where y s = F ( x s ) .
B.3 Adversarial Deep-Fool
Deep-Fool based AL (DFAL) uses sample perturbation attack in Deep Fool to calculate the distance between samples and decision boundaries, a.k.a., margin, and then selects those with the smallest perturbation distance to query [14].
The perturbation algorithm iteratively perturbs samples by adding linear noise until
{+Proof C.
 For a multi-classiﬁer with K labels and n training samples, the training loss l is measured using cross-entropy: [Equation]+}
where Y is the ground truth label variable, and ˆ Y is the predicted label variable.
Since the ground truth probability vectors are in one-hot format, the training loss in the j -th class, l j , can be rewritten as:
[Equation]
Therefore, the gap of loss in the j -th class between the distribution of the shadow model Y ( s ) and that of the victim model Y is:
[Equation]
where y j represents the j -th element of y .
On the other hand, We deﬁne b (cid:48)
jj as the log-bias between the two logarithm distributions for class j :
[Equation]
From Eqn.
7 and 8, we get:
[Equation]
where a j is the proportion of class j in both shadow and victim model training sets.
b j ∈ R K is a vector, and its element in dimension j is denoted by b jj .
To complete the proof, in Eqn.9 we need to replace b (cid:48)
jj with b jj , the j -th element in vector b j .
This replacement is correct because of the following two assumptions, whose validity will be veriﬁed experimentally in Section 5.
Assumption 1 .
The correlation coefﬁcient ρ between a distribution and its logarithm distribution is positive.
Formally,
[Equation] where j ∈ [1 , . . , K ] . Assumption 2 .
For two models with the same structure and hyper-parameters, for any class j , the correlation coefﬁcient ρ between the output vector distribution and its j -th element is positive.
Formally,
[Equation]
where j ∈ [1 , . .
, K ] .
Based on the above two assumptions, we replace b (cid:48)
jj with b jj in Eqn.9 and obtain ∆ l j ∝ b j .
To verify Assumptions 1 and 2 of Theorem 1, we calculated the bias mean as the mean of bias of all classes, i.e., (cid:80)
Kj =1 | b jj | , the log-bias mean as the mean of log-bias of all classes, i.e., (cid:80) Kj =1 | b (cid:48)
jj | , and the vector-bias mean as the mean of scalar bias (cid:80) dim (cid:80)
Kj =1 | b j | of all classes.
The results are shown in Fig.
6 and Fig.
7 respectively.
We observe that:
{+1 The bias mean is positively correlated with the log-bias mean, which justiﬁes Assumption 1. 
+}
The vector-bias mean is positively correlated with the bias mean, which justiﬁes Assumption 2.
D Few-shot Learning (FSL) Shadow-model MI
In MExMI, we use the following two FSL approaches to train shadow models, leading to two metric-based shadow-model MI.
Data-augmented shadow-model MI .
Data augmentation expands the training dataset to accelerate convergence by adding synthetic samples transformed from existing samples.
In image classiﬁcation, an image can be ﬂipped, panned or rotated to enrich the training set [29].
[Figure]
[Figure]
Transfer shadow-model MI .
Transfer learning shares the knowledge from a source domain to reduce the training complexity in a target domain.
In image classiﬁcation, NN models are suitable for transfer learning, because their shallow layers learn task-independent abstract features, and deep layers are more task-related [21].
As such, an adversary can transfer the shallow layers of a pre-trained NN model to initialize a shadow model’s parameters and to accelerate its convergence.
This technique is valid even if the pre-trained model’s problem domain is different from that of the shadow models.
E Implementation of Experiments E.1 Datasets
CIFAR10 .
CIFAR10 is an image dataset in color (with 3 channels) with 10 class labels, 50k training samples, and 10k test samples.
The image samples have a resolution of 32 and are evenly distributed intoclasses.
It is a well-known benchmark dataset to evaluate image classiﬁers.
SVHN .
SVHN is another 32-resolution benchmark for color image classiﬁcation.
It consists of street-view images of door numbers, which are labelled with digits from “0" to “9".
The dataset contains 73, images for training, and 26,032 images for testing.
AG’S NEWS .
AG’S NEWS is a benchmark for text classiﬁcation.
It consists of titles and descriptions of articles from 4 news classes, namely “World”, “Sports”, “Business” and “Sci/Tech”.
This dataset contains 120k training samples and 7.6k test samples.
E.2 Running Environment
Experiments are implemented with Python 3.7 on a desktop computer running Windows 10 with AMD Ryzen 7 2700X CPU and 64 GB RAM.
E.3 Implementation Details for ML-leaks [39] Membership Inference Attacks
In the overall evaluation experiments and case study experiments, our membership inference (MI) attacks on the copy models are compared with ML-leaks [39] MI direct attacks on the victim models.
For the shadow-model
MI attack in ML-leaks, we use one shadow model trained on the victim model’s training data set to build its MI attack model.
For unsupervised MI attack in ML-leaks, 1000 non-member noise samples are queried from the victim model.
The parameter tolerant percentage t is set to 0.06 in the threshold decision method.
E.4 Implementation Details for Iterative MExMI attacks.
The hyper-parameters are set as follows: initial learning rate ( lr )= 0 .
03 for image classiﬁers, lr = 0 .
01 for text classiﬁers, momentum = 0 .
5 , weight decay = 10 e − 4 , epochs = 150 .
The optimization method is SGD accelerated
[Figure]
by Nesterov Gradient Method [12].
The lr scheduler multiplies lr with the factor 0 .
1 at epoch 100 .
To be fair, all attacks, including the ideal one, use the same initial seed samples.
The adaptive shadow-model MI used in MI Pre-Filter and MI Post-Filter is trained on initial seed samples for 150 epochs.
E.5 Details of Metric-based Shadow-model MI Experiments
For transfer shadow-model MI, it needs prior knowledge of the source model.
In the experiment, we use a 5-block model with the same shallow structure as the victim model as the source model and train it on CIFAR100.
We then transfer the parameters of the three shallow blocks to initialize the shadow models.
For data-augmented shadow-model MI, we add augmented samples to the training dataset to double its size.
We adopt the same augmentation policy as in [49], which includes inverting, rotating, sharpness etc., but excludes those methods used to train models for a fair comparison.
The test dataset consists of 10k victim’s training images as positive samples and another 10k non-training images as negative samples.
E.6 Implementation Detail for Case Study
We train and deploy a classiﬁcation model on ModelArts without the knowledge of its architecture using SVHN and then perform various PAME attacks on it.
Our adversarial data pool consists of 2.5k SVHN and 5k
ImageNet32 images as we have to measure both ME results and MI results of MExMI.
Since ModelArts returns top-5 probabilities with three decimal places, we choose unsupervised MI as the MI module in MExMI.
The parameter tolerant percentage t is set to 0.06 in the threshold decision method [39].
The architecture of the copy model is VGG16 with batch normalization.
The query budget is 7k, and the size of initial seeds as well as the step are both 1k.
F Supplementary Experiments F.1 The Overall Performance MExMI on Adversarial Deep-fool
We present here a supplement to the performance of MExMI in Adversarial Deep-fool’s PAME during the iterative process, as shown in Fig.
It is a supplement to Fig. 3.
F.2 The Evaluation of MI Pre-Filter in the Iteration
To understand the underlying mechanism why MI Pre-Filter works, we track the ﬁltering results of Pre-Filter in each iteration of MExMI attack on CIFAR10.
The results are shown in Fig.
We observe that MI Pre-Filter can accurately ﬁnd victim’s training samples in the remaining pool, so the training set for copy model is gradually restored through iterations.
[Figure]
F.3 Transferability of Adversarial Attacks
We measured the transferability of adversarial samples obtained from the FSGM [19] adversarial attacks (at a rate of (cid:15) = 0 . 1 ) on the PAME copy models.
The transferability rate is the fraction of these samples misclassiﬁed by the victim model.
The results on CIFAR10 are shown in Table 7.
MExMI consistently has higher transferability rate than the baseline, indicating that our MExMI algorithms is also superior from this perspective.
[Table]
F.4 Impact of Weight Ratio in MI Post-Filter
In MI Post-Filter, we introduce a weight ratio ω between loss weights of the inferred training data and non- training data
[Table]
which has an effect on both Post-Filter only and MExMI without semi-supervised boosting variants.
In this experiment
[Table]
we vary ω in CIFAR10 experiments and show the results in Table.
We observe that ω has a very limited impact on the ﬁdelity and therefore our MExMI framework is robust to this parameter.
[Table]
F.5 The Ability of Evading PRADA Defence
For image classiﬁcation, PRADA[26] is the state-of-the-art method to detect ME attacks.
The detection is based on the distribution of consecutive query data, as PRADA believes that the adversary tends to issue query samples across an exceptional feature space.
To evaluate the ability of MExMI evading such detection, we measure the minimal L2 distance between query samples of MExMI and several benchmark ME attacks on CIFAR10.
The results are shown in Fig.
We can see that MExMI and the baseline attack are not different from benign queries (subjecting to Gaussian distribution) and both cannot be detected by PRADA.
In contrast, the distributions of
PRADA attack[26] and Black-box DNNs[35] have distinct traits, which are thus easier to be detected.
[Figure]
l
[Figure]
i z
[Figure]
F.6 Impact of ML Optimizations
As there are many emerging optimization methods in ML, in this subsection we investigate what impact they have on PAME attacks.
In particular, we focus on the following two methods:
• Data augmentation.
It is used in the training process to prevent overﬁtting.
This method has become increasingly popular in the domain of image classiﬁcation [36].
As shown in the experiments below, applying data augmentation in PAME can signiﬁcantly improve the ﬁdelity.
• Ensembles for neural networks.
Ensembling a set of models trained separately is well known for effectively reducing generalization error [20].
As shown in the experiments below, applying ensembles in PAME can improve the ﬁdelity.
[Table]
We use the performance results in Section 5.3 on CIFAR10, especially MExMI without semi-supervised boosting, as the baseline in this experiment.
We then use the transforming policy in [49] to perform a richer data augmentation and model averaging ensemble method.
The results are shown in Table 9, where “Data-Aug" denotes data augmentation.
Richer data augmentation improves baselines’ ﬁdelity by 1 . 18% , and the ensemble method further improves ﬁdelity by another 1 . 52% to 92 .
These results warn us that in a never-ending battle between model owners and model extractors, emerging technologies in ML may favour the latter rather than the former.