In Differential Privacy, There is Truth: On Vote Leakage in Ensemble Private Learning
Abstract
When learning from sensitive data, care must be taken to ensure that training algorithms address privacy concerns.
The canonical Private Aggregation of Teacher
Ensembles, or PATE, computes output labels by aggregating the predictions of a(possibly distributed) collection of teacher models via a voting mechanism.
Themechanism adds noise to attain a differential privacy guarantee with respect tothe teachers’ training data.
In this work, we observe that this use of noise, whichmakes PATE predictions stochastic, enables new forms of leakage of sensitive information.
For a given input, our adversary exploits this stochasticity to extracthigh-fidelity histograms of the votes submitted by the underlying teachers.
Fromthese histograms, the adversary can learn sensitive attributes of the input such asrace, gender, or age.
Although this attack does not directly violate the differentialprivacy guarantee, it clearly violates privacy norms and expectations, and wouldnot be possible at all without the noise inserted to obtain differential privacy.
Infact, counter-intuitively, the attack becomes easier as we add more noise to provide stronger differential privacy.
We hope this encourages future work to considerprivacy holistically rather than treat differential privacy as a panacea.
Introduction
The canonical Private Aggregation of Teacher Ensembles, PATE, is a model-agnostic approach toobtaining differential privacy guarantees for the training data of ML models [1], [2], that is widelyapplied [3], [4] and adapted [5]–[7] due to its comparatively favorable trade-off between differentialprivacy, utility and ease of decentralization [6].
In PATE, one considers an ensemble of independently trained teacher models.
To generate a prediction, PATE first collects the predictions of theseteachers to form a histogram of votes.
It then adds Gaussian noise to the histogram and only revealsthe label achieving plurality.
This label can be used directly as a prediction, or to supervise thetraining of a student model—in a form of knowledge transfer.
Because PATE only reveals the labelreceiving the most votes, it comes with guarantees of differential privacy, i.e., the noisy voting mechanism allows us to bound how much information from the training data is potentially exposed [8].
But PATE does not explicitly protect from leakage of a key element in its inference procedure: thehistogram of votes submitted by teachers.
While the histogram is used internally and not directlyexposed to clients, a careful examination of PATE reveals that information about the histogram leaksto clients via query answers.
The histogram can contain highly sensitive information, not the least of which is membership inminority groups which, if revealed, can be used to discriminate against individuals.
We demonstratethis by showing how an attacker, using the vote histogram of a PATE ensemble trained to predictan individual’s income, can infer wholly different attributes such as their level of education, evenwhen the attacker’s instance does not contain any information related to education-level.
Why is thispossible?
At a high level, the histogram of votes can be interpreted as a relatively rich representationof the instance, that reveals attributes beyond what the ensemble was designed to predict.
Next, we ask, is this attack a realistic threat?
We answer this in the affirmative by designing anattack that extracts PATE histograms by repeatedly querying PATE, and showing that it reconstructsinternal histograms to near perfection.
Our attack builds on the fact that repeated executions ofthe same query produce the same internal histogram and a consistent distribution of PATE’s noisedanswers corresponding to this histogram.
Our adversary can thus sample this distribution manytimes via querying, and use it to reconstruct the histogram.
This implies that our attack relies on the stochasticity of PATE’s output, which is a product of Gaussian noise, the very mechanism that was intended to protect privacy.
In fact, we find that the largerthe variance of noise added to the histogram votes, the more successful our adversary is in reconstructing the histogram.
This is in sharp contrast with the known and expected effect in differentialprivacy, that higher noise scale generally leads to stronger privacy.
Put simply: differential privacymakes our attack possible.
An astute reader may observe that histogram leakage does not violate the differential privacy guarantee, which only protects individual users in the training data, which is not compromised here.
While it is absolutely true that our attack does not violate differential privacy, it clearly violatessocietal norms and user expectations that differential privacy is often incorrectly assumed to protect.
The fact that differential privacy enables the leakage we exploit nicely underscores the distinctionbetween technical definitions of privacy and common conceptions of privacy.
The attack is difficult to mitigate.
Particularly, we show that it is stealthy in the sense that PATE’sown accounting of “privacy cost” considers our attacker’s set of queries “cheap”, meaning that revealing their answers has a relatively small effect in terms of differential privacy.
Consequently,
PATE’s privacy-spending monitoring does not prevent our attack.
Our attack also performs onlya moderate number of queries in absolute numbers, the same number used by common legitimate
PATE clients, so a hard limit on queries would impede PATE’s utility.
We will discuss other mitigation approraches, which are not robust and/or not always usable.
To summarize, our contributions are as follows:
• We posit the novel threat of extracting PATE’s internal vote histograms.
We observe andshow that those contain sensitive information such as minority-group membership.
• We show that differential privacy is the cause for histogram-information leakage to PATE’squerying clients.
• We exploit this leakage to reconstruct the vote histogram.
We achieve this by minimizingthe difference between (a) the probability distribution of outcomes observed by repeatedlyquerying PATE and (b) an analytical counterpart that we derive.
• We experiment with standard PATE benchmarks, showing that the attack can recover highfidelity histograms while using a low number of queries that remain well within PATE’sbudget intended to control leakage.
Vote Histograms are Sensitive Information
We consider an ensemble’s vote histogram, such as those computed internally in PATE.
Clearly,such histograms contain a lot more information on PATE’s innerworkings than simply its revealeddecision, but it is important to clarify that there are common contexts in which this leakage canactually be used to hurt individuals as they contain sensitive information about them.
As a prominent example, minority-group membership often leaks via histograms, and can of coursebe used to discriminate against group members.
To understand this, let’s consider a minority groupthat is under-represented in the training data distributed across PATE’s teachers.
Each teacher observes some outliers and mis-representitive phenomena such as coincidental correlations or out-ofdistribution examples.
When data on group members is scarce, each model will tend to over-fit tothe outlier phenomena within its own data, creating inter-model inconsistencies and resulting in disagreement, or low consensus, when predicting on similar inputs at test time—which readily presentsitself on vote histograms.
Thus, we expect histograms to reveal members of minority group membersvia low consensus values .
Next, we illustrate this via a simple experiment.
Extracting sensitive attributes from UCI Adult-Income histograms.
We now simulate an attack that receives the vote histogram of a salary-predictor ensemble and uses it to detect a smallminority of the population, specifically, PhD holders.
Following the above observation, our attack will simply classify all highly-consensus (consensus > 75% ) predictions as non-PhD-holders,whereas low-consensus ( < 75% ) predictions will be classified as PhD holders.
This is a heuristicattack that relies on intuition rather than learning the ensemble’s behavior using a labeled dataset.
On one hand, it may underestimate the attacker’s ability to detect PhD holders; on the other hand, itdoes not require a labeled dataset and only assumes that the attacker sees the votes histogram.
We use UCI Adult-Income dataset [9], containing around 41000 data points with basic personalinformation on people such as age, work hours, weight, education, marital status, and more.
PhDholders form about 1% of this dataset.
We randomly selected 80% of the dataset for training, andheld out the rest for testing.
We randomly partitioned the data into 250 disjoint sets.
For each,we fitted a random forest model (using a hyperparameter grid search, see Appendix D) predictingwhether income is above or below $50000.
For both training and test data, we removed from thedata columns including education levels: that is, training and test individuals do not contain anyfeature that directly distinguishes PhD from non-PhD holders. [Figure]
Figure 1 shows the distribution of high-consensus and [Figure] low-consensus on the test set (to make the effect clearer, [Figure]
we balanced the minority and majority groups in the test [Figure] set by randomly removing most of the non-PhD samples). We observe that low consensus indeed indicates minoritygroup membership.
Our attacker’s precision is not particularly high (75% on the balanced set), but they can stilluse this signal to discriminate against minority groups.
End-to-end scenario and other attacks.
Appendix Epresents this attack in an end-to-end scenario where theattacker does not have direct access to the histogram, andhas to first query a PATE instance to infer it, using ourmethodology in Section 3.
More sophisticated attackerscan look for distinctive histogram patterns that characterize certain groups; the attack should become more accurate as more models are added to the ensemble, refiningthe attacker’s histogram measurement; and precision can be amplified if the attacker holds multiplesamples that are known to belong to the same group.
Further, we note that sensitive-attribute extraction is not the only example for when vote histograms leak sensitive information: an attack coulduse votes to try to infer dataset properties [10] or distinguish between different partitions of the dataassociated with the different teachers in the ensemble.
How to Extract PATE Histograms
Having established that vote histogram leakage poses a risk to privacy and fairness, we proceed toprovide a generic method for extracting vote histograms from PATE.
3.1 Problem Formulation and Attack Model
A primer on PATE.
The PATE framework begins by independently training an ensemble of models, called teachers, on partitions of the private data.
There is no particular requirement for thetraining procedure of each of these teacher models; the only constraint is that the partitions be disjoint.
Queries made by clients are answered as follows: (1) each teacher model predicts a label onthe instance, (2) the PATE aggregator builds a histogram of class votes submitted by teachers, (3)
Gaussian noise is added to this histogram, and (4) the client receives the noised histogram’s argmaxclass (henceforth result class ).
This noisy voting mechanism gives labels predicted by PATE differential privacy guarantee, in what is an application of the Gaussian mechanism [11].
To preserve differential privacy, PATE tracks the privacy cost of the set of past queries, and stopsanswering queries once the cost surpasses the privacy budget .
The cost computation is parameterizedby a size δ .
The key differential privacy guarantee of PATE can be stated as follows: for a given set of queries with cost ε , PATE is ( ε, δ ) differentially private.
Put succinctly, ε bounds an adversary’sability to distinguish between any adjacent training datasets, whereas δ bounds the (usually small)probability, over PATE’s randomness, of this bound not holding.
We defer additional details ondifferential privacy in PATE to the appendix.
[Figure]
Attacker’s motivation.
Our attacker’s goal is to
Formally, given N predictors { P 1 , ..., P N } and a target input a , our attackerwants to infer H ≡ Count ( P 1 ( a ) , . .
P N ( a )) =[
h 1 , . .
, h m ] where Count counts the number of appearances of each element in [ m ] .
Vote histogramscan be used to extract potentially-sensitive information about an instance, such as its race, gender, orreligion (see Section 2).
Attacker’s access and knowledge.
Our attackercan send queries to the aggregator and receive thelabel predicted by PATE (i.e., the output of the noisyvoting mechanism).
This may be possible becausethe aggregator willfully exposes the predictions of
PATE, e.g., through a MLaaS API.
Alternatively,fully-decentralized implementations of PATE havebeen proposed where the central aggregator is replaced with a cryptographic multi-party computationprotocol [6], and its output is exposed directly.
Figure 2 visualizes the workflow of our attack.
In PATE, the parameters (mean, variance) of the noise added during aggregation are public domain [2]; we therefore assume the attacker knows them.
We also assume the attacker knows thenumber of teacher models N , which may or may not be public.
This assumption is only necessaryto shift the attacker’s learned distribution by a constant to attain a low L 1 approximation error whenreconstructing histograms (Section 3).
We note that the attacker could just as easily exploit the leakage (e.g. to learn sensitive attributes or differentiate between training sets) without it but we choseto instead make this assumption to simplify result presentation and interpretation.
3.2 Our histogram reconstruction attack
The idea behind the attack, given in pseudo-code in Algorithm 2, is as follows: let Q be a functionthat computes output-class probability distribution of PATE given a vote histogram H .
First, ourattacker will sample PATE to find an estimate for this distribution ¯ q ≈ Q ( H ) .
Second, the attackerwill use gradient descent to find ˆ H that minimizes the Euclidean distance between Q ( ˆ H ) and ¯ q .
Finally, they shift the estimated histogram ˆ H by a constant to account for the number of teachers(this step assumes the number of teachers is known, but is done mostly for presentation purposes,see Section 3.1).
We now detail these 3 steps.
Step 1: Monte Carlo approximation.
The first step will sample PATE M times and estimate thedistribution over PATE’s outputs ¯ q ≈ Q ( H ) by setting each class probability as its Monte Carloestimated mean frequency, i.e. ¯ q i ← 1 M (cid:80) Mj
=1 q ji where q ji indicates whether class i was sampled inthe j th step.
By the law of large numbers, as M increases, ¯ q i converges to i ’s sampling probability[ Q ( H )]
i , and we can expect the attacker’s estimate produced in the next steps to be more accurate.
Our attacker would want to increase M as much as possible, until they exceed PATE’s privacybudget.
Indeed, in PATE, the privacy leakage expended by each individual query can then be composed overmultiple queries to obtain the total privacy cost ε needed to answer the set of queries
Once the total privacy cost ε exceeds a a maximum tolerable privacy budget, PATE must stop answering queries topreserve differential privacy.
Section 4 shows that the attack succeeds for values of M that remainwell below PATE’s privacy budget, and are also moderate in absolute value, as they are similar tothe query number of student models that use PATE. Algorithm Input: 1: N ∈ N ▷ total number of teachers (see
Section 3.1 for why this is needed) 2: O ▷ PATE instance 3: T, λ ▷ optimization termination threshold and learning rate
Output: ˆ H 4: S ← sample ( O , M )
▷ sampling PATE M times and storing into S ∈ 1 , ..., K M 5: for i = 1 , 2 , . .
, M do 6: for j = 1 , 2 , . . . , c do 7: q ij = int ( S i == j ) ▷ q ij = 1 if
S i =
j , 0 otherwise 8: end for 9: end for 10: ¯ q ← 0 c ▷ initialization of ¯ q with all 0’s first 11: for i = 1 , 2 , . .
, M do 12: ¯ q [ i ] ← 1 M (cid:80) Mj =1 q ji 13: end for 14: ˆ H {+← 0 c ▷ initialization of ˆ H , here we use an all-zero array of length c 15: while (cid:13)(cid:13)(cid:13) Q ( ˆ H ) − ¯ q (cid:13)(cid:13)(cid:13) 
+}
> T do 16: ˆ H ← ˆ H − λ ∇ ˆ H (cid:13)(cid:13)(cid:13) Q ( ˆ H ) − ¯ q (cid:13)(cid:13)(cid:13)17: end while 18: ˆ H ←
ˆ H + [ N − (cid:80) ˆ H c ]
c ▷ shift ˆ H to sum to N return ˆ H Step 2: constructing the optimization objective. Our attacker wants to find ˆ H suchthat (cid:13)(cid:13)(cid:13) Q
[Equation]
ˆ H ) −
[Equation]
q (cid:13)(cid:13)(cid:13) 2 is minimized where ∥·∥ 2 denotes Euclidean norm.
Given a (differentiable)closed-form expression for Q , it becomes natural to program and solve this with moderngradient-based optimization frameworks.
Theorem 1 provides a closed form expression; andour attacker will use a differentiable approximation of this expression, as explained below.
Theorem 1.
Let H =
[ H 1 , . .
, H c ] be thevote histogram for the c classes, and let PATE’s
Gaussian-mechanism function Agg ( H ) ≡argmax { H i + S i } where S =
[ S 1 , . . .
, S M ]is a vector of M samples from a zeromean normal distribution with variance σ 2 .
Then the probability that the randomizedaggregator outputs the class k is givenby [ Q ( H )]
k
[Equation]
Φ i ( α ) ϕ k ( α )
d α where Φ i ( · ) is thecumulative probability distribution (CDF) of N ( H i , σ 2 ) (normal distribution with mean H iand variance σ ) and ϕ k ( · ) is the probabilitydensity function (PDF) of N ( H k , σ 2 ) .
Proof.
Q ( H ) =
P ( Agg ( H ) = k ) , is the probability that H k + S k = max { Agg ( H ) } .
For any 219 k , H k + S k is a random variable that follows a 220 normal distribution with mean equal to H k andvariance equal to σ 2 .
Let g k
= H k
+ S i , theng k ∼ N ( H k , σ 2 ) .
[ Q ( H )] k is the probabilityof g k is greater than g j , ∀ j ∈ { 1 , . . .
k − 1 , k +1 , . . .
c }[ Q ( H )]
k = P ( Agg ( H ) =
k ) = P ( g k > g 1 , . . , g k > g k − 1 , g k > g k +1 , . .
. , g k > g c ) = (cid:90) ∞ −∞ c,i ̸ = k (cid:89) i = P ( g i < α | g k = α ) P ( g k = α )d α = (cid:90) ∞
−∞ c,i ̸
= k (cid:89) i =
Φ i ( α ) ϕ k ( α )d α The expression in Theorem 1 is not usable in automatic differentiation and optimization frameworks; we therefore use an approximation of the integral by the trapezoid formula.
We select points with higher probability and sum up their values to get an approximation of the integral with infinite bounds.
Then we decide what values to select.
In the integral (cid:82) ∞ −∞ (cid:81) m,i ̸
= k i = Φ i ( α ) ϕ
k
( α
)d α , α is the value of g k ∼ N ( H k , σ 2 ) .
Therefore α has the highest probability at H k , and has the higher probability closer to H k .
More specifically, properties of the normal distribution give us that µ ± 6 ∗ σ covers 99% of the values of Gaussian random variable z ∼ N ( µ, σ ) .
Therefore values of α between
[-H k ± 
-][Figure] e c o un [Figure] t o p v o t e c
[Figure] un t [Figure] H k ± 6 ∗ σ cover 99% of the integral area.
Therefore,
[Equation]
which is differentiable and is handled well by most automatic differentiation packages.
Step 3: accounting for the number of teachers.
The distribution estimate produced by our optimization may be skewed by a constant, so the attacker shifts each element of ˆ H by ( N − (cid:80) ˆ H )
/cso that the new histogram ˆ H sums up to (cid:80)
ˆ H + c ∗ ( N − (cid:80) ˆ H ) /c
= N .
Theorem 2 in the appendixprovides proof that shifting Q ( ˆ H ) by a constant does not affect Q ( ˆ H ) .
Evaluation
We evaluate our attack against instantiations of PATE on common benchmarks.
We will show theattack extracts histograms that only slightly differ from the true ones underlying PATE’s decision,despite the fact that the privacy cost considered by PATE is low, and remains well within budgets thatare enforced in common PATE instantiations.
We also quantify how the impact of the differentialprivacy noise scale, showing that higher noise values result in increased attack success for a givenquery number.
We offer an hypothesis to explain this ostensibly surprising observation.
4.1 Experimental Setup
Data.
We use the experimental results from Papernot et al.
[1] to simulate our attack environment.
Papernot et al.
released the histograms obtained by PATE using 250 teachers for two 10-classcomputer-vision benchmarks, MNIST [12] and SVHN [13].
There are 9,000 histograms generatedby MNIST experiments and 26,032 histograms generated by SVHN experiments, corresponding tothe sizes of these datasets’ test sets.
We define a histogram’s consensus as its maximum value, and divide each dataset into three equalsized groups corresponding to high consensus, medium consensus, and low consensus.
Figure 3illustrate this.
We sample five histograms randomly from each group, and mount our attack forvarious noise levels.
Attack parameterization.
We simulated attackers with two types of query limits: first, an attackerlimited by PATE’s canonical privacy budget; we used the parameterization from Papernot et al.
[1],i.e.
budgets of 1.97 and 4.96 for MNIST and SVHN and σ = 40 .
Second, an attacker with a hardlimit of 10 4 queries; this is a moderate number of queries for clients wishing to train their own“student” model using the aggregator’s labels (see [1], [2]).
We applied this attack against PATEinstantiations for MNIST and SVHN with noise levels σ ∈ { 40 , 60 , 80 , 100 } .
For optimization (see Section 3), we use an adaptive learning rate: at the beginning of training, weuse a learning rate of 10 ∥ ∇ ˆ H J ∥ 2 , where J = Q ( ˆ H ) − ¯ q is the optimization objective.
As the optimiza- tion starts to converge, 10 ∥ ∇ ˆ H J ∥ 2 becomes too large so we switch to a learning rate of 1 ∥ ∇ ˆ H J ∥ 2 .
Thisresults in changes to the histogram of the magnitude of one vote for each update.
We use 0.01 asa threshold on the loss to establish convergence, and thus when ∥ J ∥ 2 < 0 .
01 , we stop optimizing.
For the attacks against canonical settings, we stopped once estimated histograms started presentingnegative values, which we found to be a slightly better strategy.
(We could also try to constrain it toonly-positive values; we discuss improving this optimization procedure further in Section 5).
Metrics.
For every attack, we measured the error rate and privacy cost .
The error rate is definedas the normalized L 1 distance between the ground-truth histogram H =
[ H 1 , . .
, H c ] and ourattacker’s estimate ˆ H
=
[ ˆ H 1 , . .
, ˆ H c ] , i.e., (cid:80) i (cid:12)(cid:12)(cid:12) H
i − ˆ H i (cid:12)(cid:12)(cid:12) /
(2 (cid:80) i | H i | ) .
Note that this can beinterpreted as the number of mis-counted votes.
We define and compute privacy cost using established practices.
At a high level (see detailsin [1]), we model PATE as a R´enyi-differentially-private mechanism and leverage known privacypreserving-composition theorems for such mechanisms; we attain (non-R´enyi) differential privacyvia a known reduction from differential privacy to R´enyi differential privacy.
The parameter δ is set as 10 − 5 for MNIST and 10 − 6 for SVHN, following Papernot et al.
Implementation Our implementation is provided in Python and the optimization in Jax code,which we make available in the supplementary material.
We ran the optimization on an Intel Xeon
Processor E5-2630 v4; optimization takes about 2.5 hours.
4.2 Results
Our attack has high performance within canonical privacy budget We first evaluate our attackon canonical PATE from Papernot et al.
Figure 4a and 4b show our attacker’s error rates forthe different histograms, averaging 0.11 on the MNIST setup and 0.05 on the SVHN setup.
Our attack extracts high-fidelity histograms and has low privacy costs.
Figure 4 shows theprivacy-budget limited attack; Figures 6 and 7 show our hard-query-limit attacker’s error rate andquery costs for different noise levels, i.e. values of σ .
We observe that, across attacks, we attain verylow error rates, often as low as 0.03, translating to 3% of the votes being miscounted.
For the hardquery-limit attack, privacy costs roughly range between 1 to 12, which is the order of magnitude forthe budget one would plausibly use, for example to attain guarantees similar to Papernot et al.
[2](which uses budgets of up to 8 in a directly comparable setting to ours) or Abadi et al.
[14] (whichalso employs a (8 , 10 − 5 ) -differentially private mechanism for MNIST). [Figure] [Figure] [Figure]
[Figure]
[Figure] [Figure] [Figure] r i v a c y C o s t
[Figure] [Figure] Adding more noise helps the attacker. Perhaps the most surprising result in this work is that thehigher the noise scale, the lower the attacker’s error is.
This is not necessarily aligned with using upmore of the privacy budget.
In fact, in many cases, increasing the noise decreases both the attacker’sprivacy cost and their error; Figure 5 shows the correlation between cost and error [Figure] [Figure] attacks
This is counter-intuitive, as larger Gaussian scales σ usually correspond to tighter privacy guarantees,more expected protection againstattacks.
Specifically, our Monte Carlo estimation should be less accurate when higher-variance noiseis added, as convergence to themean is slower.
Nevertheless, ourattack actually performs better withhigher noise levels.
To explain this, consider aggregator’s output distribution.
When it isuniform, classes are sampled with equal probabilities, contributing equal information to each Monte Carlo estimator.
Conversely, when some classes have a lower probability than others, their estimator will receive less samples.
Sharp output distributions, for example, have a peak that essentially“eclipses” other classes.
To illustrate this, consider the case where no noise is added at all; here, theoutput is always the plurality vote, and a black-box querying adversary cannot learn anything aboutthe histogram except its top-voted class, which is already known after a single query.
Our results indicate that the mitigation of this eclipsing effect by increasing the noise, can be moredominant than the adverse effect that increasing noise has on Monte Carlo convergence.
Interestingly, this is not always reflected in PATE’s privacy-cost score, which is often lower for setups thatleak more on vote histograms.
Technically, there is no contradiction: privacy cost measures differential privacy, which does not necessarily translate to protection against vote-histogram leakage.
Discussion
Mitigation.
The possibility of this attack is inherent to PATE’s aggregation mechanism, as long asthe attacker can make multiple queries to PATE.
Our experiments in Section 4 show that (1) usingtighter privacy budgets does not necessarily mitigate the attack, as there is no strong correspondencebetween the privacy cost and the attack’s success, and (2) it would be hard to limit the numberof queries some other way without crippling PATE’s utility, because our attack is successful whileusing the same number of queries used in common scenarios from the literature.
Theoretically, the attack would be mitigated if PATE returned a consistent answer for each query.
PATE can thus try to cache answers to past queries and not recalculate them.
Unfortunately, thisdefense would be exposed to adversarial perturbations that try to evade the caching mechanismwithout affecting predictions, and would not be possible for settings that keep queries confidentialand/or include decentralized aggregation [6].
Finally, we can try to prevent sensitive information from leaking onto vote histograms.
Particularly,models that generalize well across subgroups will be more immune to an attacker inferring groupmembership via consensus.
This reduces to the problem of subgroup fairness , an active line of workwith many proposed approaches [15]–[19] but no silver-bullet solutions.
Limitations.
Empirical analysis of sensitive-attribute leakage onto vote histograms (Section 2)can be expanded to improve more sophisticated attackers, other scenarios, and also other formsof sensitive information that can leak onto histograms.
We instead focus our work on extractinghistograms from PATE, noting that this can be used as a foundation for various different attacks.
Our optimization procedure takes a noticeably long time (roughly 10 minutes for a single step and 13hours to convergence on a histogram), which prevented us from fully optimizing its hyper-parameterchoices.
This is however a limitation of our current experimental setup, not of the attack, bearingthe main consequence that we are potentially under-estimating our attack’s capabilities.
Related work.
PATE is a widely-adopted framework for differentially-private ML, with myriadapplications [3], [4] and extensions [5]–[7]; our attack is generally applicable to many of thoseframeworks, which inherit their privacy analysis from PATE.
Another prominent decentralized ML framework, Federated Learning (FL) [20], has been extensively investigated from a privacy perspective.
As we did for PATE in this work, prior work attacking
FL uncovered numerous forms of leakage.
For example, Hitaj et al.
[21] reconstructed the averagetraining set representation of each classes;
Geiping et al.
[22] reconstructed training data with highfidelity; Nasr et al.
[23] mounted a membership inference attack against the clients; Wang et al.[24]showed how a malicious server could distinguish multiple properties of data simultaneously; and
Melis et al.
[25] inferred the clients’ training data sensitive properties.
These prior efforts all focuson FL, and are orthogonal to ours.
We are the first to evaluate any attack against PATE.
Conclusion We are the first to audit the confidentiality of PATE from an adversarial perspective.
Our attack extracts histograms of votes, which can reveal attributes of the input such as race orgender, or help attackers characterize teacher partitions.
The attacker’s success is not highly correlated with their queries’ privacy cost which is monitored by PATE, and mitigations of this attackare nontrivial and/or significantly hinder prediction utility.
Particularly, using larger Gaussian noise,even when it fortifies the differential privacy guarantee, actually increases risk to the confidentialityof the vote histogram.
This surprising tension demonstrates that care must be taken to analyze theprotection differential privacy provides within a given threat model, rather than treat it as protectingagainst any form of leakage.
Broader Impact Our work studies information leakage in a widely-adopted system, thus promoting our understanding of its risks.
Our adversarial method can be used by developers and auditorsto evaluate the confidentiality and privacy promises of PATE-based frameworks.
Our observation that differential privacy does not prevent but rather enables the attack is the first ofits kind in that it reveals a discrepancy between differential privacy and societal norms of privacy.
Characterizing this distinction is essential to building technology that uses technical definitions ofprivacy as an instrument to protect privacy norms. A Differential Privacy
An algorithm is said to be differentially private if its outputs on adjacent inputs (in our case, datasets)are statistically indistinguishable.
Informally, the framework of differential privacy requires that theprobabilities of an algorithm making specific outputs be indistinguishible on two adjacent inputdatasets.
Two datasets are said to be adjacent if they only differ by at most one training record.
Thedegree of indistinguishibility is bounded by a parameter denoted ε .
The lower ε is, the stronger theprivacy guarantee is for the algorithm because it is harder for an adversary to distinguish adjacentdatasets given access to the algorithm’s predictions on these datasets.
In the variant of differentialprivacy we use, we can also tolerate that the guarantee not hold with probability δ .
This allows us toachieve higher utility.
B Shifting Distributions
In Section 3, we explain that we shift our histogram estimate by a constant to account for the numberof teachers known to the attacker.
The following theorem shows that the number of teachers doesnot affect the attacker’s computation
Theorem 2.
For two histograms, H 1 =
[ h 11 , . . .
, h 1 m ] and H 2 =
[ h 21 , . .
h 2 m ] , Q H 1 ,σ = Q H 2 ,σ ifh 1 i − h 2 i
=
{+h 
+}
j − h 2 j for all i, j = 1 , . . .
, m .
Proof.
let
d = h 1 i − h 2 i
=
{+h 
+}
j − h 2 j for all i, j = 1 , . . .
, m .
[Equation]
[Equation]
[Equation]
[Equation]
2 ) =
[Equation]
=
[Equation]
for all i, j = 1 , . . .
, m .
Q H
[Equation]
[Equation]
=
P ([ g
[Equation]
> g 11 , .
[Equation]
What Theorem 2 states is, if the difference between two histograms is uniform, then the probabilitydistribution of the outcomes is the same.
With the support of Theorem 2, H can be safely shifted bya constant amount to sums up to the number of teachers, N .
C Chosen histograms for evaluation
Table 1 shows the histograms we chose for evaluation in the 3 consensus-level categories.
[Table]
D Fittig Random Forests
Every one of our teachers in Section 2 fits a random forest classifier using the sklearn package; eachteacher performed a grid search over the following hyperparameters
[Table]
and picked the values that leadto the lowest training loss.
• max depth : the maximum number of levels that a tree has, an integer chosen between
and 11 inclusively;
• max features : the maximum number of features, while splitting a node, one of sqrt(number
of features)
, log(number of features), 0.1*(number of features), 0.2*(number of features),
0.3*(number of features), 0.4*(number of features), 0.5*(number of features), 0.6*(numberof features), 0.7*(number of features), 0.8*(number of features), 0.9*(number of features);
• n estimators : the number of trees that the forest has, an integer chosen between log(9.5)and log(300.5);
• criterion: the loss function, one of gini impurity and entropy;
• min samples split : the minimum number of instance for a node to split, one of 2, 5, 10;• bootstrap: one of True or False
E End-to-end sensitive-attribute inference
In Section 2, we showed that histograms leak by mounting an attack that classifies histograms tolow-consensus and high-consensus groups, which reveals information about minority-group membership.
In Section 4, we showed that we can extract histograms by querying PATE instances.
Now,we combine these two attacks, to extract minority-group membership information directly from a
PATE instance.
Our setting mirrors the setting from Section 2, but the attacker does not have direct access to histograms of individuals, and instead they extract them from PATE’s answers usingour methodology (Section 3).
We used the same ensemble from Section 2, but this time, the 250teachers’ vote histogram was noised, again using σ = 40 , δ = 0 .
00001 and a privacy budget of 1.9 as in [2].
We sampled 10 low-consensus and 10 high-consensus members of the test set, and ranthe attack on them: we queried PATE with each member’s data record until exhausting the privacybudget, computed the Monte Carlo estimators, ran the optimization to recover the vote histogram,and then classified it to low-consensus/high-consensus as in Section 2.
Results are given in Figure 8,and indeed, they mirror the results of the attack in Section 2.
[Figure]
F Edge values for noise
Here, our purpose is to evaluate our attack given extremely low and extremely high values of σ .
Werepeated the query-number-limited attack from Section 4.1 where adversaries perform 10 4 queries.
This time, we used a σ value approaching 0 and a very high one (400).
Figure 9 shows that whennoise is close to 0, the error rate is the highest, it then drops and climbs again as we increase theerror.
This is consistent with what we would expect: we know that when σ = 0 , the attacker cannotlearn anything but the argmax class, whereas if σ is infinitely large, PATE’s output distribution isuniform regardless of the underlying votes, and the attacker again cannot learn anything.
[Figure]
as the noise increases.