A CCURACY -P RIVACY T RADE - OFF IN D EEP E NSEMBLE : A M EMBERSHIP I NFERENCE P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep ensemble learning has been shown to improve accuracy by training multiple neural networks and fusing their outputs.
Ensemble learning has also been used to defend against membership inference attacks that undermine privacy.
In this paper, we empirically demonstrate a trade-off between these two goals, namely accuracy and privacy (in terms of membership inference attacks), in deep ensembles.
Using a wide range of datasets and model architectures, we show that the effectiveness of membership inference attacks also increases when ensembling improves accuracy.
To better understand this trade-off, we study the impact of various factors such as prediction conﬁdence and agreement between models that constitute the ensemble.
Finally, we evaluate defenses against membership inference attacks based on regularization and differential privacy.
We show that while these defenses can mitigate the effectiveness of the membership inference attack, they simultaneously degrade ensemble accuracy.
We illustrate similar trade-off in more advanced and state-of-the-art ensembling techniques, such as snapshot ensembles and diversiﬁed ensemble networks.
The source code is available in supplementary materials.
I NTRODUCTION
Ensemble learning has been shown to improve classiﬁcation accuracy of neural networks in particular, and machine learning classiﬁers in general Kondratyuk et al.
(2020); Kuncheva & Whitaker (2003);
Sagi & Rokach (2018).
The most commonly used approach in deep ensemble learning involves averaging the output of multiple neural networks (NN) that are independently trained on the same dataset with different random initialization Lobacheva et al.
Such a simple approach has been extensively used in practice to improve accuracy Lee et al.
; Wang et al.
Notably, a majority of the top performers in machine learning benchmarks, such as the ImageNet Large Scale
Visual Recognition Challenge Russakovsky et al.
(2015), have adopted some form of ensemble learning Lee et al.
; Szegedy et al.
; He et al.
Ensemble learning has also been used to defend against privacy-harming membership inference (MI) attacks, where the goal of an attacker is to infer whether a sample has been used to train a model–i.e., whether the sample belongs to the train set.
Membership inference attacks generally use the prediction conﬁdence of NN models to infer membership status of a sample Salem et al.
Shokri et al.
; Truex et al.
; Yeom et al.
by leveraging the insight that trained models may output higher prediction conﬁdence on train samples than non-train samples Choo et al.
The intuition behind using ensemble learning to defend against MI attacks is that training each model on a different subset of data makes the ensemble less prone to overﬁtting Salem et al.
This idea of using ensemble learning to defend against MI attacks has since been discussed in the literature Huang et al.
; Li et al.
; Rahimian et al.
; Yang et al.
In this paper, we demonstrate the trade-off between accuracy and privacy in deep ensemble learning.
Figure 1 illustrates this trade-off by plotting accuracy and membership inference attack effectiveness for ensembles comprising of varying number of base models (1, 2, 5, and 10) that are trained for different numbers of epochs (5, 45, and 90).
We make two key observations here.
First, there is an increase in both accuracy and MI attack effectiveness as we go from a single model to ensembles comprising of an increasing number of base models.
The trade-off is more noticeable for more accurate models trained for a larger number of epochs.
Second, we can adapt the design of ensembles to suitably navigate the trade-off between accuracy and privacy.
Speciﬁcally, the two extreme cases
[Figure]
are to: (1) maximize accuracy by using an ensemble of models at the same level overﬁtting but at the cost of worse privacy 1 (purple arrow); and (2) maximize privacy by intentionally using an ensemble of under-ﬁtted models instead of a single model but at the cost of accuracy (brown arrow).
To understand the root cause of this trade-off, we show that using ensemble learning to improve accuracy exacerbates its susceptibility to membership inference attacks by making train and non-train samples more distinguishable.
By analyzing the conﬁdence averaging mechanism of deep ensembles, we investigate potential factors that enable membership inference.
We show that the most inﬂuential factor is the level of correct agreement among models.
Simply put, the number of models that correctly classify a train sample is often larger than the ones that correctly classify a test sample.
This results in a wider conﬁdence gap between train and non-train samples, when conﬁdence values are averaged, enabling more effective membership inference attacks.
We further show that the difference in the level of correct agreement between train and non-train samples stems from models’ generalization gap.
Hence, a natural question to ask is "can deep ensembles that use less overﬁtted models mitigate privacy issues while achieving high accuracy?".
To answer this question, we study several regularization techniques, common membership inference defenses, and a few other ensembling approaches.
We again observe a privacy-accuracy trade-off pattern similar to that shown in Figure 1.
Summary of contributions: In this paper, we perform a systematic empirical study of MI attacks on deep ensemble models.
First, we show that when the ensemble improves accuracy, it also leads to a different distribution shift in the prediction conﬁdence of train and test samples, which in turn enables more effective membership inference.
Second, we analyze various factors that potentially cause the prediction conﬁdence of train and non-train samples to diverge: prediction conﬁdence of correctly classifying models, prediction conﬁdence of misclassifying models, and level of correct agreement.
Among these factors, we show that the most dominant factor is the level of correct agreement among models which indicates that more models in an ensemble agree on their prediction when a sample is a training sample.
Hence, the aggregation of their prediction yields higher conﬁdence output in comparison with non-train samples.
We show that common defense mechanisms in membership inference literature, including differential privacy, MMD+Mixup, L1 and L2 regularization, as well as other ensemble training approaches, such as bagging and partitioning, can be used to mitigate effectiveness of MI attacks but at the cost of accuracy. We observe similar trade-off in more advanced and state-of-the-art ensembling techniques, such as snapshot ensembles Huang et al. (2017) and diversiﬁed ensemble networks Zhang et al.
In summary, we provide empirical evidences and analyses to demonstrate the trade-off between accuracy and privacy in deep ensemble models.
S YSTEM M ODEL 2. E NSEMBLE L EARNING
Background. In literature, ensemble learning refers to various approaches that combine multiple models to make a prediction.
Models used to construct an ensemble are often called base learners.
There are two main factors to construct an ensemble Sagi & Rokach (2018): 1) how base learners are trained to ensure diversity, such as random initialization, bagging, partitioning, etc., and 2) how the output of base learners are fused to obtain the ﬁnal output, including majority voting, conﬁdence averaging, stacking, etc.
Unlike ensemble of traditional machine learning algorithms, in a deep ensemble, the main source of diversity often comes only from random initialization of base learners Fort et al.
In fact, other sources of diversity, such as bagging, have been shown to considerably degrade the overall accuracy of a deep ensemble Lee et al.
; Lakshminarayanan et al.
System Model. We focus on the most widely used deep ensemble Kondratyuk et al. (2020) unless otherwise speciﬁed. In this model, 1) base models are trained with random initialization on the same training dataset, and 2) their prediction conﬁdence are fused through averaging.
A less common approach is to average model logits which has been used in a few studies Webb et al.
; Wang et al. See Appendix A.3 for experimental evaluation of logit averaging. We also evaluate two state-of-the-art deep ensembling approaches, namely snapshot ensemble and diversiﬁed ensemble network, in Appendix A.4. Throughout our experiments, we observed similar accuracy-privacy trade-off with small variations over all these ensembling methods.
2. M EMBERSHIP I NFERENCE
Background. Membership inference is a form of privacy leakage where the goal is to determine if a sample was used during the training of a target model.
Samples used during training are often referred to as member or train samples, and other samples are referred to as non-member, non-train, or test samples.
The ﬁrst MI attack on neural networks was proposed in Shokri et al.
(2017) where the attacker trains an attack classiﬁer to predict the membership status.
The attack classiﬁer takes the prediction conﬁdence of a target model as an input.
Assuming that the attacker has access to a dataset with a similar distribution, she trains a set of shadow models to mimic the target model.
Since the membership status of the data with which the shadow models are trained are known to the attacker, she can use the data to train the attack classiﬁer.
Many papers use the same idea with different variations or less restrictive assumptions Salem et al.
Liu et al.
; Song et al.
(2019); Long et al.
; Truex et al.
Long et al.
; Yeom et al.
; Rezaei & Liu (2021); Zou et al.
; Li & Zhang (2020).
Most previous work built upon the idea of using prediction conﬁdence to infer the membership status, except for Rezaei & Liu (2021); Choo et al.
(2020); Rahimian et al.
In Rezaei & Liu (2021), the authors assumed white-box access to the target model and launched a series of MI attack based on conﬁdence values, distance to the decision boundary, gradient w.r.t model weight, and gradient w.r.t input.
In Choo et al.
, the authors proposed two attacks based on input transformation and distance to the boundary in a black-box setting.
Similarly, in Rahimian et al.
(2020), the attacker randomly perturbs an input to obtain a set of random transformations of the input and uses the predicted labels to infer membership status.
System Model. We focus on changes of conﬁdence values, used in most existing attacks, when using deep ensembles.
We show that when using deep ensembles the distribution of conﬁdence values becomes more distinguishable between train and non-train set in comparison with non-ensemble case. Consequently, any MI attack that relies on conﬁdence values would be more effective on deep ensembles.
Since our goal is to show a trade-off between accuracy and privacy, not to show which conﬁdence-based attack can slightly outperforms another, we focus on a conﬁdence-based attack proposed in Rezaei & Liu (2021) in both white-box and black-box settings.
Here, white-box means the attacker has access to base-learners’ output before aggregation, and black-box means the attacker has only access to the aggregated conﬁdence output.
Decision boundary-based attacks are extremely computational and query inefﬁcient and it is not trivial how to adopt them for ensemble learning where essentially an input is copied n times and then fed to n models. Gradient-based approach of Rezaei & Liu (2021) also needs full knowledge of the entire deep ensemble.
We consider adaptation of non-conﬁdence-based attacks for deep ensembles as future work.
2. M EMBERSHIP I NFERENCE D EFENSES
Background.
Defense mechanisms against membership inference attacks can be summarized into two categories Rahimian et al.
Generalization-based: Shokri Shokri et al.
(2017) was ﬁrst to correlate membership inference success with overﬁtting.
Since then, many standard regularization techniques have been used to alleviate overﬁtting, such as L1 regularization Choo et al.
(2020), L2 regularization Choo et al.
; Truex et al.
; Nasr et al.
; Jia et al.
Shokri et al.
(2017), differential privacy Choo et al.
Rahimian et al.
dropout Jia et al.
(2019), and adversarial training Nasr et al.
Interestingly, ensemble learning has also been proposed as a defense mechanism.
In Salem et al.
, they proposed a combination of partitioning and stacking as a defense mechanism.
The intuition is that training each model with different subset of data makes the entire ensemble model less prone to overﬁtting.
Note that these defense mechanisms often degrade the accuracy of the target model (see Section 4.2) Choo et al.
Conﬁdence-masking: These defense mechanisms aim to reduce the amount of information that can be obtained from the output of a target model by perturbing Jia et al.
or limiting the dimensionality of the output Shokri et al.
; Truex et al.
Choo et al.
Majority of these approaches are post-training methods and only work in black-box setting.
System Model. In this paper, we mainly focus on generalization-based defense mechanisms with the exception of MMD+Mixup Li et al. This is because most conﬁdence-masking approaches manipulate conﬁdence values post-training. As a result, the output values of these models do not reliably represent the "conﬁdence" of the model.
These approaches are built under the assumption that accurate prediction of conﬁdence is not needed. However, many applications require accurate estimation of conﬁdence. Moreover, if the accurate prediction of conﬁdence is not required, then the trivial MI defense would be to only output the class label and avoid these conﬁdence-masking defenses altogether.
H OW D OES E NSEMBLING I NCREASE M EMBERSHIP I NFERENCE E FFECTIVENESS ? 3. C ONFIDENCE D ISTRIBUTION S HIFT
Ensemble learning is only helpful when base learners disagree on some samples Kuncheva & Whitaker (2003); Sagi & Rokach (2018).
Otherwise, ensembling does not improve accuracy.
Furthermore, when deep ensemble is used, the conﬁdence values of multiple models are averaged to obtain the ﬁnal prediction.
Consequently, when ensembling improves accuracy, it averages the prediction conﬁdence of highly conﬁdent predictions (mostly from models which correctly classiﬁed the sample) and less conﬁdent predictions (mostly from models which misclassiﬁed the sample).
As a result, conﬁdence distribution shift is inevitable for both train and test samples.
This phenomenon can be observed as the distribution of Figure 2(a) shifts to that of 2(d).
This can be better observed by separating correctly classiﬁed samples which have signiﬁcantly higher prediction conﬁdence (Figure 2(e)) and misclassiﬁed samples which have lower prediction conﬁdence (Figure 2(f)).
One can clearly observe that both distributions shift more towards the center from Figure 2(b) and (c) to Figure 2(e) and (f).
However, the change in the distribution of train and test samples does not necessarily cause a more effective membership inference if the change has a similar effect on the conﬁdence distribution of both train and test samples.
In the next subsection, we analyze the potential factors that affect the distribution change and how they can change conﬁdence distribution of member and non-member sets differently. [Figure]
versus an ensemble of 10 models (bottom row).
3. E FFECT OF ENSEMBLING ON I NDIVIDUAL S AMPLES
We use y i to denote the conﬁdence value of the i th model in an ensemble of n models.
Hence, for a given sample x , the output of the ensemble is:
[Equation]
Given a single sample, we can further divide models in the ensemble into two groups: 1) models that correctly classiﬁed the sample denoted by y ci and 2) models that misclassiﬁed the sample by y mi .
For a given sample x , c models correctly classify it and m models misclassify, where c + m = n .
Note that the value of c and m depends on the sample 2 .
Based on the Eq.
(1), three major factors affect the ﬁnal conﬁdence value ( y el ) of a sample: y c , y m , and c .
As a result, if these values are different for train and test samples, the ensembling causes different shift in the distributions, and consequently, membership inference attack will be more effective.
These factors are as follows:
Conﬁdence value of correctly classifying models ( y c ): Since the majority of samples are supposed to be correctly classiﬁed by a practical model, any distinguishable conﬁdence difference between train and test samples can lead to a very effective membership inference attack.
However, as shown in Figure 2(b), we can observe that there is no signiﬁcant difference between train and test samples.
Conﬁdence value of misclassifying models ( y m ): Unlike correctly classiﬁed samples, there is a marginal difference between conﬁdence distribution of train and test samples of misclassiﬁed samples (see Figure 2(c)).
This may be exploited by membership inference attack to partially distinguish between train and test samples.
Level of correct agreement ( c ) among models: As shown in Figure 3(a), the number of models that correctly classify a sample [Figure] Appendix A.2 for more results).
Since prediction conﬁdence of correctly classiﬁed samples are higher than misclassiﬁed samples on average, i.e., y c > y m , and c is smaller for test samples, the ensemble conﬁdence of test samples ( y el ) becomes lower than train samples.
As a result, this factor can largely contribute to the effectiveness of membership inference attacks on deep ensembles.
We note that the ﬁrst two factors are not unique to ensembles and can be exploited by an attacker in a single model scenario as well.
As a result, these two factors have been extensively studied in Rezaei & Liu (2021) in a single model scenario across various image datasets and well-trained models.
They have shown that for deep models the ﬁrst factor ( y c ) is almost indistinguishable between train and test set and only the second factor ( y m ) is marginally distinguishable.
However, this marginal difference does not have a considerable impact on the different distribution shift in train and test sets.
On the other hand, the level of agreement has a big impact on different distribution shifts.
To better demonstrate this, we can analyze the distribution difference in each level of agreement separately.
As shown in Figure 3(b), the average conﬁdence of train and test samples are very close and indistinguishable when each level of agreement is drawn separately.
If the effect of the ﬁrst two factors were considerable, the two conﬁdence values for each level of agreement would have been more distinguishable.
Note that the average conﬁdence between train and test sets is more distinguishable for the ﬁrst two points in x-axis (where majority of models misclassify a sample).
However, these distributions only constitute a tiny portion of the training dataset, as shown with the ﬁrst two blue bins in Figure 3(a).
However, when all samples are combined, we can vividly observe that the average conﬁdence gap between train and test sets considerably widens, as shown in the last point in x-axis in Figure 3(b).
This clearly demonstrates that the major factor in different distribution shift between train and test sets is the level of agreement ( c ).
Note that, unlike the ﬁrst two factors, the third factor ( c ) only improves the effectiveness of membership inference attacks in ensemble scenario because it does not exist in a single model.
In other words, if a defence strategy eliminates the effect of the average level of correct agreement (i.e., it ensures that c is close between train and test samples), the membership inference attack is still possible on the ensemble, but only to the degree that it is possible on a single model 3 .
As shown in Figure 3(c), as the gap between y c of train and test sets (red lines) and the gap between y m of train and test sets (brown lines) increases, the attack on both single model (non-ensemble) and also the ensemble (EL-10) increases.
However, only when the average level of correct agreement gap between train and test (blue lines) widens, the membership inference attack on ensembles becomes more effective than on non-ensembles.
Another important observation from Figure 3(c) is that the minimum level of agreement gap between train and test occurs when models are relatively underﬁtted (i.e., the blue lines in ﬁrst few epochs).
This phenomena has also been partially observed in Fort et al.
(Figure 2(c)).
The main reason is that underﬁtted models often only learn the most common and generalizable features and, thus, they often agree on the features and predictions.
As a results, ensembling them also does not yield any signiﬁcant accuracy improvement.
As they move from underﬁtted region to overﬁtted region, they start to memorize features more speciﬁc to train samples Feldman (2020).
Consequently, the average gap of correct agreement between train and test set widens.
Hence, the wider generalization gap of base learners is, the more effective membership inference attack would be on ensemble.
3. W HY D OES IT O UTPERFORM G AP A TTACK S IGNIFICANTLY ?
Recently, several studies report a simple baseline attack called gap attack Choo et al.
(2020), also known as naive attack Rezaei & Liu (2021)
; Leino & Fredrikson (2020) that achieves similar performance as the conﬁdence-based attacks in most scenarios.
The gap attack predicts a sample as member if it is correctly classiﬁed by the target model, and as non-member otherwise. In other words, gap attack essentially reﬂects the generalization gap Rezaei & Liu (2021). In Rezaei & Liu (2021), authors extensively analyzed this phenomena in deep models and argued current membership inference attacks that barely outperform gap attacks are ineffective in practice because they only reﬂect the generalization gap and cannot infer the membership status of each individual sample accurately. Figure 4 shows that the effectiveness of membership inference attacks increases and outperforms the gap attack as ensemble is deployed.
This raises signiﬁcant privacy concern since the gap attack is often suggested as a baseline that is also hard to outperform in non-ensemble setting Rezaei &
Liu (2021); Leino & Fredrikson (2020); Choo et al.
Note that gap attack can be viewed as a metric directly reﬂecting the generalization gap rather than a reliable membership inference.
As suggested in Rezaei & Liu (2021), we can separate correctly classiﬁed samples and misclassiﬁed samples to understand why membership inference attacks can barely outperform gap attack.
As shown in Figure 2(b) and (c), the distributions of train and test samples are similar when separated into correctly classiﬁed and misclassiﬁed sets.
The reason why the distribution of all samples (Figure 2(a)) looks more distinguishable when correctly classiﬁed samples and misclassiﬁed samples are combined is that there are often more misclassiﬁed samples in the test set than the train set.
This is the information that gap attack exploits which essentially reﬂects the generalization gap.
In order for a membership inference attack to considerably outperform the gap attack, the distribution of correctly classiﬁed samples and misclassiﬁed samples should leak membership status information, which is not often the case as it is shown in Figure 2(b) and (c), and Rezaei & Liu (2021).
When ensembling is used, the distribution of conﬁdence values changes dramatically, as explained in Section 3.1.
By comparing the conﬁdence distribution of correctly classiﬁed samples in an ensemble (Figure 2(e)) with a non-ensemble scenario (Figure 2(b)), the distribution is clearly more distinguishable in ensemble case.
This is of signiﬁcant privacy concern because, as discussed in Rezaei & Liu (2021), majority of samples in practice belong to the correctly classiﬁed set.
Similar trend is also observable in misclassiﬁed samples (Figure 2(f)).
Hence, the conﬁdence values, that barely leak more information than generalization gap itself in a single model scenario, now considerably leak more membership information than just the generalization gap.
That is the reason why membership inference attacks are signiﬁcantly more effective in ensembles in comparison to the gap attack.
E XPERIMENTS R ESULTS 4. E XPERIMENTAL S ETUP
We explore a wide range of datasets that are often used in deep ensemble literature or membership inference literature: Adult 4 , Texas, Purchase Shokri et al.
(2017), MNIST LeCun et al.
(1998), FM- [Figure] NIST Xiao et al.
(2017), SVHN Netzer et al.
(2011), CIFAR10 Krizhevsky et al.
(2009), CIFAR
Krizhevsky et al.
(2009), and ImageNet Russakovsky et al.
For non-image datasets (Adult,
Purchase, and Texas), we use a fully connected neural network consisting of a hidden layer of size 128 and a Softmax layer.
All other training parameters for these datasets are set as suggested in Shokri et al.
For image datasets, we use a wide range of convolutional neural networks depending on the input dimension and the difﬁculty of the task.
We use the model implementations adopted in Nasr et al.
; Rezaei & Liu (2021) 5 .
We train 10 models for each dataset with random initialization and construct an ensemble of 2, 5 and 10 models, respectively.
We train each model for up toepochs.
For membership inference attack models, we consider two different threat models: black-box and white-box.
In the black-box setting, the attacker only access the output of the ensemble, which is the average over softmax output of individual models.
In the white-box setting, the attacker can see the output of each model in ensemble.
In the former case, the attack model takes the output of ensemble as input, while in the latter case, it takes all conﬁdence outputs of all models as input.
Attack models are NNs with three hidden layers of size 128, 128, and 64, respectively.
We consider a worst case scenario where 80% of the training dataset is given to the attacker and the goal is to infer the membership of the remaining samples, similar to Rezaei & Liu (2021).
All other training parameters are set as suggested in Rezaei & Liu (2021).
All experiments are implemented in Python 3.5, PyTorch 1.6 and are conducted on a server with Intel Xeon Processor 24-core 2.70GHz, 8 GeForce RTX
GPUs and 384GB memory.
See Appendix A.1 for full report.
The results of the snapshot ensembles and diversiﬁed ensemble networks are shown in Appendix A.4.
Figure 4 shows the results on all datasets.
For some datasets, such as Adult, Texas, and MNIST, ensemble learning barely changes the accuracy or privacy.
That is because the disagreement across models is insigniﬁcant in these datasets.
For all other datasets, ensemble learning improves the accuracy as well as the effectiveness of membership inference attacks.
As mentioned in Section 3.2, the most salient factor in membership inference effectiveness on deep ensembles is the accuracy gap between train and test set.
Figure 4 clearly shows that whenever this generalization gap is large for non-ensemble case, the attack improvement is signiﬁcant after ensembling.
It is worth noting that the ensembling can often reduce the generalization gap and the effectiveness of the gap attack (e.g., CIFAR10-DenseNet-100, CIFAR100-AlexNet, or CIFAR100-ResNet20).
However, due to the reasons explained in Section 3.3, the membership inference effectiveness still increases.
4. D EFENSE MECHANISMS
As discussed in previous sections, the main factor that causes more privacy leakage in ensemble learning is the generalization gap of the base models.
Therefore, any standard regularization technique can potentially work as a defense mechanism.
In this paper, we study L1 and L2 regularization, [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] DP-SGD 6 , and MMD+Mixup Li et al.
Furthermore, one can simply terminate training early to keep the model’s weights in less overﬁtted region.
Moreover, some ensembling techniques, such as bagging and partitioning, limits the access of models to all training samples, which can potentially reduce membership inference effectiveness on deep ensembles.
We train 10 AlexNet models with different initialization on CIFAR10 for each defense mechanism.
Here, we consider black-box membership inference attack to measure privacy.
Figure 5 shows the effect of defense mechanisms on ensemble learning.
We can observe a consistent trade-off between ensemble accuracy and privacy that resembles Pareto optimal points.
Note that privacy degradation rate is clearly not constant.
An ensemble of heavily regularized models or under-ﬁtted models barely causes more privacy leakage (e.g., L2 regularization with λ = 1 ).
On the other hand, an ensemble of overﬁtted models (e.g., non-regularized models trained for 90 epochs) results in large privacy leakage.
It is worth mentioning that for a given accuracy, some approaches provide more privacy.
For instance, the regular deep ensemble and the bagging of models trained for 30 epochs achieve similar accuracy.
However, bagging is in fact more prone to membership inference because each model in bagging is exposed to fewer training data samples and consequently is more overﬁtted.
C ONCLUSION & D ISCUSSION
In this paper, we investigate membership inference attacks in deep ensemble learning and demonstrate that there is a trade-off between accuracy and privacy.
We note some limitations of our empirical analysis and opportunities for future work.
First, privacy is a multi-faceted concept and can be deﬁned or quantiﬁed in several ways.
In this work, we quantiﬁed privacy leakage in terms of the effectiveness of membership inference attacks.
Future work can quantify privacy in terms of other relevant attacks such as model inversion Fredrikson et al.
; He et al.
, property inference Ateniese et al.
; Ganju et al.
(2018), and model stealing attack Tramèr et al.
(2016) as well as formally-provable measures such as differential privacy Dwork (2008).
Second, ensemble learning is an umbrella term covering a variety of methods to combine multiple base learners.
An arbitrary method of training and combining base learners can be construed as ensemble learning.
We mainly focused to ensemble averaging because of its prevalent use for ensembling deep models.
Given the privacy leakage of deep ensembles demonstrated here, future work can look into designing other fusion approaches to better navigate the trade-off between privacy and accuracy.
One solution can focus on training models sequentially and applying some non-trivial criteria during training of each model to force the distribution of correct agreement to be close for train and non-train samples.
How to achieve this is not trivial and needs further research.
A A PPENDIX A
RAINING
Figure 6 demonstrates the improvement of accuracy and MI attack over various training epochs.
For datasets that ensembling outperforms a single model, using an ensemble of underﬁtted models is less prone to MI attack.
However, it corresponds to lower accuracy.
This is consistent with all experiments across different datasets and models, shown in Figure 6.
A. L EVEL OF C ORRECT A GREEMENT
As discussed in section 3.2, overﬁtted models tend to disagree more on test samples than train samples.
In other word, the distribution of agreement for train and test sets becomes more distinguishable as models overﬁt.
This distinction is more clear for datasets, such as CIFAR10 and CIFAR100, which shows most improvement when ensembling is used, as shown in Figure 7, 8, and 9.
Furthermore, the level of agreement can reveal if an ensemble can actually improve prediction.
If all models correctly classify a sample or all models misclassify a sample, ensembling fails to outperform a single model.
This is the case for Adult, Texas, and MNIST, as shown in Figure 9.
A. A VERAGING L OGIT
In this section, we average logits (the output of a model before Softmax) of NN models instead of the conﬁdences.
We can still observe that ensembling leaks more membership status than non-ensemble scenario.
However, the MI attacks with average conﬁdence (Figure 6), in general, are slightly more effective than MI attacks with average logits (Figure 10).
The reason is that that conﬁdence values are normalized and, hence, when aggregated all models have the same contribution to the overall conﬁdence output of the ensemble.
However, when logit is used, the conﬁdence output of the ensemble is more inﬂuenced by highly activated neurons.
These highly activated neurons, which
[Figure] [Figure]
[Figure] [Figure]
[Figure]
(j) MNIST-LeNet (epoch=25)
[Figure]
(k) MNIST-LeNet (epoch=25)
[Figure]
[Figure]
often belong to the correctly classifying models, has signiﬁcantly more inﬂuence on the conﬁdence output of ensemble in comparison with lightly activated neurons of misclassifying models.
Hence, the conﬁdence output is heavily inﬂuenced by only a portion of models in ensemble that have high activation neurons.
In other words, it can be seen as an ensemble of only a portion of models, not all models in the ensemble.
Since ensembling with fewer models leaks less membership status, logit averaging of n models leak membership status than conﬁdence averaging of the same number of models.
Note that logit averaging is still prone with the same degree to membership leakage in a white-box attack since a white-box attacker has access to all conﬁdence values.
Note that the consequence of using logit in certain applications, such as conﬁdence estimation, that requires reliable conﬁdence estimation is out of the scope of this paper.
Moreover, one major drawback of using logit is that it can be arbitrary scaled Wang et al.
However, in scenarios where only accuracy is concerned and white-box access is unavailable to the attacker, averaging logits seems to have a better privacy protection of training data than averaging conﬁdences. [Figure]
A. M ORE A DVANCED E NSEMBLING A PPROACHES
In this section [Figure] we evaluate two state-of-the-art ensembling approaches, namely snapshot ensembles
Huang et al.
(2017) and diversiﬁed ensemble networks Zhang et al.
For snapshot ensemble, we train several models on several datasets for 500 epochs and restart the cycle every 50 epochs, similar to the original paper Huang et al.
Note that the goal of our evaluation is show the accuracy-privacy trade-off, not to achieve the highest accuracy possible.
Due to this reason and limited time we had, we did not perform an exhaustive hyper-parameter tuning.
Nevertheless, similar trade-off can be observed in Figure 11.
We also conduct the same experiment with diversiﬁed ensemble networks Zhang et al.
The original paper used pre-trained VGG and ResNet models.
We did not use pre-trained models for two reasons: 1) It make comparison with other approaches unfair, and 2)
It may interfere with the [Figure] [Figure] membership inference analysis.
We ﬁnd that by using randomly initialized models to start training,
L d varies signiﬁcantly and prevents the optimization to converge.
Therefore, we add a weight to the L d term to reduce its effect on the entire loss.
We use 0.01 for CIFAR10 and SVHN and 0.
for CIFAR100.
For the shared layer, we use a fully-connected layer of size 128 followed by batch normalization and Relu activation.
We use SGD to train models for 60 epochs while dropping the learning at each 20 epoch by 0.1.
We could not achieve the exact same results as reported in the paper for two main reasons: 1) we did not use pre-trained models in the ensemble, and 2) many hyper-parameters and implementation details are missed from the original paper.
We could not ﬁnd a set of hyper-parameters and conditions to consistently achieve higher accuracy when increasing the number of models.
This was also reported in the original paper where they found that more models in the diversiﬁed ensemble do not always improve accuracy.
One penitential reason is that training base models in diversiﬁed neural networks are not independent.
These models are trained during the training of the entire network. This means when the number of models in the diversiﬁed neural network is increased, there are signiﬁcantly more parameters to train, but the number of epochs are constant. So, it is expected that diversiﬁed neural network with less models to sometimes outperform diversiﬁed neural network with more models if the number of training epoch is ﬁxed. Nevertheless, as shown in Figure 12, in cases where accuracy increases, the membership inference attack effectiveness also increases.